> ## Documentation Index
> Fetch the complete documentation index at: https://comis-feature-skill-archive-import.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.
# Logging
> Understanding and viewing Comis logs
Comis writes detailed logs about everything it does -- they are your primary tool
for understanding what is happening and diagnosing problems. Every message processed,
every agent execution, every health check, and every error is recorded.
## Where Logs Go
Comis writes logs to two places simultaneously:
* **Log file**: `~/.comis/logs/daemon.log` -- A structured JSON file that is automatically
rotated when it gets too large (10 MB by default). The daemon keeps up to 5
rotated copies before deleting the oldest.
* **Console output**: The same log data is also written to standard output. This is
what you see when using `pm2 logs` or `journalctl`.
Logs are in JSON format for machine readability. Each log line is a JSON object
with fields like `level`, `msg`, `module`, and timestamps. You typically view them
through pm2 or journalctl, which display them line by line.
## Log Levels
Comis uses Pino's standard log levels plus a custom `audit` level. From most
severe to least:
| Level | Pino Value | What It Means | Example |
| --------- | ---------- | --------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- |
| **fatal** | 60 | Unrecoverable process failure, imminent exit. The daemon is about to crash. | `"msg": "FATAL: Bootstrap failed"` |
| **error** | 50 | Something broke and needs attention. Always includes `hint` and `errorKind` fields. | `"msg": "Channel connection failed"` with `"hint": "Check bot token"`, `"errorKind": "auth"` |
| **warn** | 40 | Something is degraded but still working. Also includes `hint` and `errorKind` fields. | `"msg": "Gateway token auto-generated (ephemeral)"` with `"errorKind": "config"` |
| **audit** | 35 | Compliance and security operational events. Custom level between warn and info. | `"msg": "Secret accessed"` with `"agentId": "assistant"` |
| **info** | 30 | Normal operation boundary events. Startup, shutdown, request completed. This is the default level. Budget: 2-5 lines per request. | `"msg": "Comis daemon started"` |
| **debug** | 20 | Detailed internal operations. Individual LLM calls, tool executions, intermediate steps. Unbounded output. | `"msg": "Tool execution complete"` with `"toolName": "bash"` |
| **trace** | 10 | Most granular diagnostic level. Available but not used in the codebase by default. | (no standard examples -- enable for deep protocol debugging) |
The `silent` value exists at the logger API layer to suppress output, but the
`daemon.logLevels` configuration only accepts `trace`, `debug`, `info`, `warn`,
`error`, and `fatal`. To quiet a noisy module, raise its level (for example,
set it to `error`) rather than relying on `silent`.
Start with INFO (the default). Only switch to DEBUG when troubleshooting a specific
problem -- DEBUG produces significantly more output. Use TRACE only for deep protocol
debugging. The FATAL and AUDIT levels are used by the system automatically -- you
typically do not set them as module log levels.
## Viewing Logs
**View the last 50 lines (snapshot):**
```bash theme={null}
pm2 logs comis --lines 50 --nostream
```
**Follow logs in real time (press Ctrl+C to stop):**
```bash theme={null}
pm2 logs comis
```
pm2 keeps its own log files at `~/.pm2/logs/`. The `--lines` flag shows recent
lines from those files, and omitting `--nostream` follows them live.
**View the last 50 lines (snapshot):**
```bash theme={null}
journalctl -u comis -n 50 --no-pager
```
**Follow logs in real time (press Ctrl+C to stop):**
```bash theme={null}
journalctl -u comis -f
```
journalctl reads from the system journal, which captures everything the daemon
writes to standard output and standard error.
**View the last 50 lines of the log file:**
```bash theme={null}
tail -n 50 ~/.comis/logs/daemon.log
```
**Follow the log file in real time (press Ctrl+C to stop):**
```bash theme={null}
tail -f ~/.comis/logs/daemon.log
```
This reads directly from the daemon's log file. Since it is JSON, each line is
a complete log entry.
## Changing Log Levels
### Per-Module Overrides
Comis is organized into modules (daemon, agent, channels, scheduler, etc.), and you
can set a different log level for each one. This is useful when you want to debug a
specific part of the system without being overwhelmed by output from everything else.
Add a `logLevels` section under `daemon` in your configuration:
```yaml theme={null}
# config.yaml
daemon:
logLevels:
daemon: "info" # daemon lifecycle events
agent: "debug" # detailed agent execution (LLM calls, tool runs)
channels: "warn" # only show channel problems
scheduler: "info" # scheduled task events
skills: "info" # tool and skill processing
memory: "info" # database and embedding operations
gateway: "info" # HTTP/WebSocket server events
```
Each module only logs messages at or above its configured level. In the example
above, the `agent` module logs everything including DEBUG messages, while the
`channels` module only logs warnings and errors.
The most common use case is setting `agent: "debug"` to see exactly what your agent
is doing during a conversation -- which tools it calls, how long each LLM request
takes, and what context it retrieves from memory.
### Runtime Level Changes
You can change log levels without restarting the daemon by using the
`daemon.setLogLevel` RPC command. This is useful for temporarily enabling DEBUG
logging to investigate an issue, then switching back to INFO when you are done.
```bash theme={null}
# Enable debug logging for the agent module
curl -X POST http://localhost:4766/rpc \
-H "Content-Type: application/json" \
-H "Authorization: Bearer YOUR_TOKEN" \
-d '{"method":"daemon.setLogLevel","params":{"module":"agent","level":"debug"}}'
```
Runtime level changes take effect immediately and persist until the daemon restarts,
at which point the levels revert to whatever is in your configuration file.
## Error Classification
Every ERROR and WARN log entry includes two mandatory fields that help you
quickly understand and act on problems:
* **`hint`** -- A human-readable string explaining what to do about the error
* **`errorKind`** -- A classification tag from a fixed set of 9 categories
### ErrorKind Values
| Kind | Description | Typical Action |
| ------------ | --------------------------------------------------------------- | ------------------------------------------------------------------------ |
| `config` | Configuration parsing, missing keys, schema violations | Check config.yaml against the [Config Reference](/reference/config-yaml) |
| `network` | TCP/HTTP failures, DNS resolution, connection resets | Check network connectivity and firewall rules |
| `auth` | Authentication or authorization failures (401/403, bad token) | Verify API keys and tokens in SecretManager |
| `validation` | Input validation failures (bad request body, invalid params) | Check the request payload format |
| `timeout` | Operation exceeded deadline (LLM call, HTTP request, DB query) | Increase timeout or check provider status |
| `resource` | Resource exhaustion (OOM, disk full, file descriptor limit) | Free resources or increase limits |
| `dependency` | External service unavailable (LLM provider, embedding API) | Check provider status pages and retry |
| `internal` | Unexpected internal errors (assertion failures, logic bugs) | Report as a bug with the full log entry |
| `platform` | Chat platform API errors (Discord, Telegram, Slack rate limits) | Check platform status and rate limit headers |
Filter logs by `errorKind` to quickly find related problems. For example, to see
all authentication errors: `grep '"errorKind":"auth"' ~/.comis/logs/daemon.log`
## Memory and Recall Signals
The `memory` module (set its level under `daemon.logLevels.memory`, shown above)
emits two families of structured signal: step-tagged write/curation logs and
graceful-degradation WARNs.
### Step-tagged write and curation logs
Memory writes and the background curation jobs carry a `step` tag plus a
`durationMs` timing field, so you can filter the memory pipeline by stage:
| `step` | Stage |
| ------------- | -------------------------------------------------------------------------------------------- |
| `extract` | Extracting candidate memories from a turn |
| `store` | Persisting a memory entry (also surfaces as the INFO "Memory store complete" forensic event) |
| `link` | Resolving and linking entities during a memory-review pass |
| `consolidate` | Clustering and consolidating memories into observations |
```bash theme={null}
# Follow the memory write/curation pipeline by stage
jq 'select(.step == "consolidate")' ~/.comis/logs/daemon.log
```
The aggregate outcome of these passes is also emitted as the typed
`memory:entities_linked` and `memory:consolidated` events — see
[Observability → Memory & Recall Diagnostics](/operations/observability#15-memory-recall-diagnostics).
### Graceful-degradation WARNs (no silent degradation)
Recall degrades gracefully, and **every degradation path logs a WARN** carrying
the mandatory `errorKind` + `hint` pair (see [Error Classification](#error-classification)
above) — recall never drops to a cheaper path silently:
| Degradation | WARN signal |
| ---------------------------------------------------------------- | --------------------------------------------------------------------------------------- |
| Reranker unavailable / timed out → fusion-ranked fallback | WARN with `errorKind` + `hint`; corresponds to `memory:reranked.fellBack` / `.timedOut` |
| Vector index unavailable → FTS-only recall | WARN with `errorKind` + `hint`; corresponds to `memory:recalled.vectorCandidates: 0` |
| Candidate missing embedding, or invalid extraction/consolidation | WARN with `errorKind` + `hint` |
As with all Comis logs, these never log secrets, message bodies, or absolute
paths — the per-recall ranking detail lives only in the opt-in, full-sanitized
[recall-trace artifact](/operations/observability#15-memory-recall-diagnostics),
never in `daemon.log`.
```bash theme={null}
# Spot recall falling back to a cheaper path
grep '"errorKind"' ~/.comis/logs/daemon.log | grep -i rerank
```
For the full "why did recall pick X / why is recall slow?" diagnosis flow, see the
[Troubleshooting runbook](/operations/troubleshooting#memory-recall-issues).
## Field Dictionary
Every log entry is a JSON object. Beyond the standard Pino fields (`level`, `time`,
`msg`, `pid`, `hostname`), Comis uses 48 canonical fields organized by category.
These are the fields you will see when reading JSON log output.
### Core Identity
| Field | Type | When Present |
| ------------- | ------ | ------------------------------------------------------- |
| `agentId` | string | Agent-scoped operations |
| `traceId` | string | Auto-injected via AsyncLocalStorage on every request |
| `channelType` | string | Channel adapter logs (e.g., "telegram", "discord") |
| `module` | string | Always -- set via `logLevelManager.getLogger("module")` |
### Timing
| Field | Type | When Present |
| ---------------------- | ------ | -------------------------------------------------------- |
| `durationMs` | number | Any timed operation (LLM calls, tool execution, startup) |
| `startupDurationMs` | number | Daemon startup complete log |
| `shutdownDurationMs` | number | Daemon shutdown complete log |
| `connectionDurationMs` | number | WebSocket connection lifetime (on close) |
### Operation
| Field | Type | When Present |
| ----------- | ------ | ------------------------------- |
| `toolName` | string | Tool/skill execution logs |
| `method` | string | RPC/HTTP method name |
| `requestId` | string | Per-HTTP-request correlation ID |
### Error
| Field | Type | When Present |
| ----------- | --------- | --------------------------------------------------------------------- |
| `err` | object | Error objects. **Use `err`, not `error`** -- matches Pino serializer. |
| `hint` | string | Required on ERROR and WARN. Actionable guidance for the operator. |
| `errorKind` | ErrorKind | Required on ERROR and WARN. One of 9 classification values. |
### Token and Cost
| Field | Type | When Present |
| --------------------- | ------ | ------------------------------------------------- |
| `tokensIn` | number | After LLM call -- input token count |
| `tokensOut` | number | After LLM call -- output token count |
| `tokensTotal` | number | After LLM call -- total token count |
| `cacheReadTokens` | number | After LLM call -- tokens served from prompt cache |
| `cacheCreationTokens` | number | After LLM call -- tokens written to prompt cache |
| `estimatedCostUsd` | number | After LLM call -- estimated cost in USD |
### Pipeline
| Field | Type | When Present |
| ----------- | ------- | --------------------------------------------------------- |
| `step` | string | Pipeline step name during multi-step processing |
| `reason` | string | Why an action was taken (e.g., compaction trigger reason) |
| `inputLen` | number | Input length for processing steps |
| `outputLen` | number | Output length for processing steps |
| `itemCount` | number | Count of items processed |
| `success` | boolean | Whether a processing step succeeded |
### WebSocket
| Field | Type | When Present |
| ------------------- | ------ | ------------------------------- |
| `activeConnections` | number | WebSocket connection events |
| `closeCode` | number | WebSocket close events |
| `closeReason` | string | WebSocket close events |
| `closeType` | string | WebSocket close classification |
| `connectionId` | string | WebSocket connection identifier |
| `clientId` | string | WebSocket client identifier |
### Message
| Field | Type | When Present |
| ------------------ | ------- | -------------------------------------------- |
| `messageTruncated` | boolean | When a message was truncated before delivery |
| `messageLen` | number | Length of incoming message |
| `responseLen` | number | Length of outgoing response |
### Agent Execution
| Field | Type | When Present |
| --------------------- | ------ | ------------------------------------------------------------------- |
| `llmCalls` | number | Execution complete summary -- total LLM API calls |
| `toolCalls` | number | Execution complete summary -- total tool invocations |
| `prunedMessages` | number | After context compaction -- messages removed |
| `sessionMessageCount` | number | Current session message total |
| `stopReason` | string | Why execution ended (e.g., "end\_turn", "max\_tokens", "tool\_use") |
### System
| Field | Type | When Present |
| --------------- | ------ | ----------------------------------------------- |
| `instanceId` | string | Daemon instance identifier (unique per process) |
| `configName` | string | Config file being loaded |
| `shutdownOrder` | number | Component shutdown sequence number |
### Truncation Flags
| Field | Type | When Present |
| ----------------- | ------- | ------------------------------------------------- |
| `paramsTruncated` | boolean | When RPC parameters were truncated in log output |
| `queryTruncated` | boolean | When a database query was truncated in log output |
## Security
Comis automatically redacts sensitive information from log output. API keys, tokens,
passwords, and other credentials are replaced with `[REDACTED]` before being written
to the log file or console.
The redaction engine covers all credential field name patterns across multiple nesting depths (top-level
through 3 levels of nesting):
| Redacted Fields |
| ---------------------------------------------------------- |
| `apiKey`, `token`, `password`, `secret` |
| `authorization`, `accessToken`, `refreshToken`, `botToken` |
| `privateKey`, `credential`, `credentials`, `key` |
| `passphrase`, `connectionString`, `accessKey` |
| `cookie`, `webhookSecret` |
Nested paths are also redacted: `config.telegram.botToken`, `*.*.apiKey`, etc.,
up to 4 levels deep (`*.*.*.*`).
You do not need to worry about sensitive data in logs. Comis automatically detects
and hides API keys, tokens, and passwords using a compiled redaction engine. The
redaction rules are compiled once at startup for zero runtime overhead.
## Log Rotation
Comis automatically rotates all observability log streams to bound disk usage.
### `observability.logRotation` — Cross-Stream Policy
A single policy governs five log streams under `~/.comis/logs/`:
* `daemon.log` — daemon-wide structured JSON log
* `cache-trace.jsonl` — per-turn cache observability (cache hits, misses, tokens)
* `config-audit.jsonl` — config-write audit trail (all writes to config.yaml)
* `session-index.YYYY-MM-DD.jsonl` — append-only session index (date-rolled, one file per UTC day)
* `*.trajectory.jsonl` — per-session trajectory recordings (one file per agent session)
Configure the policy under `observability.logRotation` in your `config.yaml`:
```yaml theme={null}
# config.yaml
observability:
logRotation:
maxSizeBytes: 52428800 # 50 MB; pino-roll rolls when daemon.log crosses this
maxFiles: 5 # keep 5 rotated copies per stream (oldest deleted)
maxAgeDays: 30 # delete rotated files older than 30 days
compressAged: true # gzip rotated files (.log.gz / .jsonl.gz)
```
| Field | Default | What It Controls |
| -------------- | ------------------ | ------------------------------------------------------------------------------------------------------------------- |
| `maxSizeBytes` | `52428800` (50 MB) | Size at which daemon.log rolls. Honored by pino-roll for `daemon.log` and by per-stream caps for the JSONL streams. |
| `maxFiles` | `5` | Number of rotated copies retained per stream. Oldest deleted on rotation or startup sweep. |
| `maxAgeDays` | `30` | Files older than this are deleted on the daemon-startup sweep (mtime-based). |
| `compressAged` | `true` | Gzip rotated files after rename — saves \~5× disk space on JSON text. |
### How to Verify
Inspect the resolved policy, including any operator overrides, with:
```bash theme={null}
comis config get observability.logRotation
```
### Storage Budget
Worst-case storage: 5 streams × 5 files × 50 MB = **1.25 GB**.
With `compressAged: true` (the default), gzipped JSON text compresses approximately 5×, so realistic steady-state is **\~300 MB**.
Increase `maxFiles` or `maxAgeDays` if you need more history; reduce them on tight disks.
### Per-Stream Behavior
**`daemon.log`** — pino-roll handles size-based rotation in a worker thread; rotated files become `daemon.1.log`, `daemon.2.log`, etc. The daemon-startup sweep gzips any uncompressed rotated files and removes those beyond `maxFiles` or older than `maxAgeDays`.
**`cache-trace.jsonl`** — a per-file cap (50 MB) prevents a single cache-trace file from growing unbounded. The startup sweep prunes accumulated history against `maxFiles` and `maxAgeDays`.
**`config-audit.jsonl`** — a small (10 MB) rename-shift rotation runs on every append, keeping the active file write-friendly. The startup sweep gzips the resulting rotated copies and prunes them by count and age.
**`session-index.YYYY-MM-DD.jsonl`** — a new file starts each UTC day (date-roll). Old dated files are pruned by the startup sweep against `maxFiles` and `maxAgeDays`. Only today's dated file is treated as the active base and is never pruned.
**`*.trajectory.jsonl`** — one file per agent session, with a 10 MB soft cap and a 50 MB hard cap applied in-session. Accumulated session files across past sessions are pruned by the startup sweep against `maxFiles` and `maxAgeDays`.
### Per-File `daemon.logging` Keys
The `daemon.logging.maxSize` and `daemon.logging.maxFiles` keys (documented below) apply to `daemon.log` specifically. When both key families are set, `observability.logRotation` takes precedence for the cross-stream sweep, and `daemon.logging.maxSize` retains its role as the pino-roll size threshold when `observability.logRotation.maxSizeBytes` is unset.
Prefer `observability.logRotation` for a unified, consistent rotation policy
across all five log streams. The `daemon.logging` keys govern only `daemon.log`.
### `daemon.logging` Configuration
You can customize per-file daemon-log rotation with these keys:
```yaml theme={null}
# config.yaml
daemon:
logging:
filePath: "~/.comis/logs/daemon.log" # where to write the active log file
maxSize: "10m" # rotate when file reaches 10 MB
maxFiles: 5 # keep up to 5 rotated copies
```
| Setting | Default | What It Controls |
| ---------- | -------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `filePath` | `~/.comis/logs/daemon.log` | Path to the active log file. Supports `~` expansion. |
| `maxSize` | `10m` | Maximum file size before rotation. Supports `k`, `m`, `g` suffixes. When `observability.logRotation.maxSizeBytes` is set, it overrides this threshold for pino-roll. |
| `maxFiles` | 5 | Number of rotated files to keep. Older files are deleted. |
With the cross-stream policy active (the default), `observability.logRotation`
controls the 50 MB roll threshold and the 5-file / 30-day retention for all streams.
The `daemon.logging.maxSize` value still reaches pino-roll as the size threshold if
you have not set `observability.logRotation.maxSizeBytes`.
## Trace Files
In addition to the main log file, Comis can write per-agent JSONL trace files that
capture detailed execution data. These are useful for debugging specific agent
conversations.
```yaml theme={null}
# config.yaml
daemon:
logging:
tracing:
outputDir: "~/.comis/traces" # directory for trace files
maxSize: "5m" # rotate trace files at 5 MB
maxFiles: 3 # keep 3 rotated trace files per session
```
Trace files are created automatically when an agent processes messages. They contain
the full execution timeline: prompts sent, responses received, tools called, and
timing data.
How the daemon starts, runs, and shuts down.
Health checks that watch your system's vital signs.
Token tracking, cost estimation, and performance data.
Common problems and how to solve them.