> ## Documentation Index
> Fetch the complete documentation index at: https://comis-feature-skill-archive-import.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.
# Voice
> Speech-to-text transcription and text-to-speech auto-reply for voice messages
When someone sends a voice message to your agent, Comis can transcribe it to text
so the agent can read and respond. Optionally, the agent can reply with a voice
message of its own using text-to-speech. The entire flow -- from receiving a voice
note to sending a spoken reply -- can be fully automatic.
You don't need to understand the technical details to use this feature. The configuration examples below are copy-paste ready.
## Keyless by Default
Voice works with **zero credentials out of the box**. Speech-to-text defaults to
`auto` (keyless-first: a local whisper engine when one is available, otherwise an
honest "not available" hint) and text-to-speech defaults to `edge` (Microsoft Edge
TTS — free, no API key). A keyed cloud provider (OpenAI, Groq, Deepgram, ElevenLabs)
is used **only** when you set it explicitly **or** your agent's main provider already
supplies a usable audio key.
A **Codex / ChatGPT-subscription (OAuth) login cannot drive OpenAI audio endpoints.**
The OAuth bearer is scoped to the Codex Responses backend and cannot reach
`api.openai.com/v1/audio/*`. With the keyless-first resolver, a Codex-only agent is
steered to a keyless provider and — if none is available yet — gets an actionable
"not available" hint, **never a silent HTTP 401** from an empty-bearer request. To use
OpenAI audio, set a separate platform API key (`OPENAI_API_KEY`) and
`integrations.media.transcription.provider: openai` explicitly.
## How Voice Works
A user records a voice note in Telegram, Discord, WhatsApp, or any connected
platform.
The voice message is automatically converted to text using your configured
speech-to-text provider. The agent sees the written transcript.
Your agent processes the transcript like any text message and writes a reply.
If auto-TTS is enabled, Comis converts the text reply into audio and sends it
back as a voice message.
Voice also helps with mention detection. In group chats where the bot only responds
when mentioned, Comis can pre-transcribe voice messages to check if the user said
the bot's name.
## Transcription (Speech-to-Text)
The transcription `provider` is one of `auto` (the keyless-first default), `local`
(a keyless local whisper engine), or a keyed cloud provider (`openai`, `groq`,
`deepgram`). You only need one configured, and the default needs none — set up
multiple via `fallbackProviders` for automatic failover.
| Provider | Default Model | API Key | Notes |
| ---------- | ---------------------- | ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `auto` | -- | None needed | **Default.** Keyless-first: resolves to the local engine when available, else follows the agent's main provider key, else honest-unavailable |
| `local` | base | None needed | Keyless **in-process** whisper engine — auto-downloads a small model (default `base`) once to `~/.comis/models/whisper/` on first use, then reuses it. Or point `local.baseUrl` at an OpenAI-compatible local whisper server. |
| `openai` | gpt-4o-mini-transcribe | `OPENAI_API_KEY` | Explicit opt-in; fast |
| `groq` | whisper-large-v3-turbo | `GROQ_API_KEY` | Explicit opt-in; returns language detection and duration |
| `deepgram` | nova-3 | `DEEPGRAM_API_KEY` | Explicit opt-in; high accuracy, real-time streaming |
The default. Resolves a provider in priority order: a keyless **local** whisper
engine first (when available), then the agent's main-provider audio key **only if
one genuinely exists** (e.g. an `openai`/`groq` main with that key set), then any
configured `fallbackProviders`, and finally an honest "not available" hint naming
the exact remedy. An OAuth-only main (Codex) has no reusable audio key, so it is
steered straight to keyless / honest-unavailable — never an empty-bearer 401.
A keyless local whisper engine — no API key, no cloud round-trip. Two modes:
* **In-process (default, zero-config).** When `local.baseUrl` is unset, Comis runs
whisper in-process via [Transformers.js](https://huggingface.co/docs/transformers.js).
The engine lazy-loads on the first voice note: it auto-downloads a small model
(`transcription.local.model`, default `base`) once over TLS from the model hub to
the scoped cache `~/.comis/models/whisper/`, then reuses the cached model for the
life of the process — so the **first** transcription is slower (it pays the
download + load) and every subsequent one is fast. The engine runs on ONNX
Runtime, which the daemon's existing `--allow-addons` flag already covers — no new
permission flag is needed (see [Node Permissions](/reference/node-permissions#keyless-local-stt-whisper-engine)).
If the engine or model can't load, or ffmpeg is absent, transcription
honest-degrades to "not available" (so `auto` falls through to the next rung) —
it never crashes the inbound message.
* **`local.baseUrl` server (escape hatch).** Set `transcription.local.baseUrl` to an
OpenAI-compatible local whisper **server** (faster-whisper-server, Speaches,
whisper.cpp-server, …) and Comis talks to it keyless — no in-process model
download, fully offline-capable, and GPU-accelerated if the server is. A reachable
server makes `local` available without the in-process engine.
**`local.baseUrl` is SSRF-guarded (loopback-only by default).** The URL is validated
(with DNS-rebinding protection) before any request: only **loopback** (`127.0.0.1`,
`localhost`, `::1`) and explicitly-allowlisted hosts are permitted; public and
cloud-metadata addresses are denied. A rejected `baseUrl` is treated as
*unreachable* (the resolver falls through to the in-process engine or
honest-unavailable) — it never drives an outbound request. Point it at a whisper
server on the local host, not a public URL.
Explicit opt-in. Uses the gpt-4o-mini-transcribe model. Reliable and fast for
most use cases. Requires an OpenAI **platform** API key (`OPENAI_API_KEY`) — a
Codex/ChatGPT OAuth login cannot drive audio. Does not return language or
duration metadata in the response. Set `provider: openai` to use it.
Uses Whisper Large V3 Turbo. Returns rich metadata including the detected
language of the audio and total duration. Requires a Groq API key. A good
alternative if you want automatic language detection.
The language detection feature is especially useful for multilingual communities
where voice messages arrive in different languages.
Uses the Nova 3 model. Known for high accuracy and fast processing speeds.
Requires a Deepgram API key. Internally uses a different API format (raw binary
instead of form data), but this is handled automatically -- you just provide
the API key.
Deepgram is a good choice when transcription accuracy is your top priority,
especially for noisy audio or accented speech.
### Fallback Chains
If your primary provider fails (API error, timeout, rate limit), Comis tries the
next provider in your fallback list. This keeps voice transcription working even
when a single provider has issues.
```yaml theme={null}
integrations:
media:
transcription:
provider: auto # Keyless-first default
fallbackProviders: # Try these if the primary fails
- groq
- deepgram
```
In this example, Comis resolves `auto` first (keyless / follow-main). If that
resolution cannot serve, it tries Groq, then Deepgram. The first successful
transcription is used. (`auto` and `local` are also valid `fallbackProviders`
entries — `auto` is a benign no-op rung the resolver collapses.)
## Text-to-Speech (TTS)
The TTS `provider` is one of `edge` (the keyless default), `openai`, `elevenlabs`,
or `local`. You only need one configured, and the default needs none — the right
choice depends on voice quality, budget, and whether you need an API key.
| Provider | Default Model | Default Voice | API Key | Cost |
| ----------------- | ------------------------ | --------------------------- | -------------------- | ------------------------------------------------------------- |
| `edge` | -- | en-US-AvaMultilingualNeural | None needed | **Default.** Free (network-dependent) |
| `local` / `piper` | Xenova/mms-tts-eng | -- | None needed | Keyless **in-process** offline TTS — no network at synth time |
| `openai` | tts-1 | alloy | `OPENAI_API_KEY` | Paid (explicit opt-in) |
| `elevenlabs` | eleven\_multilingual\_v2 | Rachel | `ELEVENLABS_API_KEY` | Paid (explicit opt-in) |
**Keyless default:** TTS defaults to `edge`. Edge
needs no key, so auto-reply works out of the box. A Codex/OAuth-only main also
resolves to `edge`. Set `provider: openai` or `provider: elevenlabs` explicitly
(with the matching key) for the paid providers.
The default. Microsoft Edge text-to-speech — free, no API key required. Default
voice is en-US-AvaMultilingualNeural. Works out of the box for every install
(including Codex/OAuth-only mains). Limited customization compared to OpenAI and
ElevenLabs, but a great zero-config baseline. Network-dependent (it calls a
Microsoft Edge endpoint); for a fully offline voice, use `local`/`piper` below.
Keyless **offline** TTS. `provider: local` (or its alias `piper`) runs a small
single-speaker `text-to-audio` model (`Xenova/mms-tts-eng`) **in-process** via
`@huggingface/transformers` — the TTS twin of the local whisper STT engine. The
model auto-downloads once over TLS to `~/.comis/models/tts/` (a separate dir from
`models/whisper/`, already inside the daemon's `--allow-fs-write` scope), then
synthesizes with **no network at synth time**. A short/corrupt model download
fails closed (`model_load_failed`); if the engine or model is unavailable it
honest-degrades to a structured error (never crashes the reply). With TTS
`provider: auto`, the chain is `edge → local/piper → honest-unavailable`, so a
no-network install still speaks once the model is cached.
Explicit opt-in. Uses the tts-1 model with the "alloy" voice. Supports speed
adjustment from 0.25x to 4.0x. Requires an OpenAI **platform** API key
(`OPENAI_API_KEY`) — a Codex/ChatGPT OAuth login cannot drive audio. A good
balance of quality and speed.
OpenAI offers several voice options beyond "alloy." You can change the voice
in your configuration or per-reply using TTS directives.
Premium voice synthesis using the eleven\_multilingual\_v2 model. Default voice
is "Rachel." Offers advanced voice settings: stability, similarity boost, style,
and speaker boost. Requires an ElevenLabs API key. Best voice quality for
human-like speech.
ElevenLabs voices sound the most natural and expressive. The advanced settings
let you fine-tune how the voice sounds -- for example, increasing stability
makes the voice more consistent, while boosting style adds more expressiveness.
## Auto-Reply Modes
The `autoMode` setting controls when Comis automatically converts text replies
into voice messages.
| Mode | Behavior |
| ----------------- | ------------------------------------------------------------------------------ |
| `"off"` (default) | TTS only when the agent explicitly uses the `tts_synthesize` tool |
| `"always"` | Every text reply is automatically converted to a voice message |
| `"inbound"` | Reply with voice only when the user sent a voice message first |
| `"tagged"` | Reply with voice only when the agent's response contains a `[[tts]]` directive |
### Off
The default. Your agent's text replies are sent as text. The agent can still
generate voice on demand using the [tts\_synthesize](/agent-tools/media) tool,
but nothing is automatic.
### Always
Every text reply from the agent is also sent as a voice message. Useful for
accessibility or when users prefer listening. Replies that already contain media
attachments skip TTS to avoid sending double media.
### Inbound
The agent mirrors the user. If the user sends a voice message, the reply is a
voice message. If the user sends text, the reply is text. This creates natural
conversational behavior where the agent adapts to how the user is communicating.
### Tagged
The agent controls when to use voice. When the LLM includes `[[tts]]` in its
output, that reply is converted to speech. The tag is stripped from the final
text so the user never sees it.
## TTS Directives
In tagged mode, the agent can include directives to fine-tune how the voice
reply sounds. The `[[tts:...]]` syntax allows overriding the voice, provider,
format, and speed for individual replies.
```
[[tts]] # Use default voice and provider
[[tts:voice=nova]] # Override the voice
[[tts:voice=nova provider=openai]] # Override voice and provider
[[tts:voice=nova speed=1.5 format=mp3]] # Override multiple settings
```
Available directive parameters:
| Parameter | Description |
| ---------- | ------------------------------------ |
| `voice` | Voice identifier (provider-specific) |
| `provider` | TTS provider to use for this reply |
| `format` | Audio format (`opus`, `mp3`, etc.) |
| `speed` | Playback speed multiplier |
Directives work in `tagged` mode. In `always` or `inbound` mode, TTS happens
automatically using your default settings -- directives are not needed.
## Per-Channel Audio Formats
Different platforms work best with different audio formats. Comis automatically
converts audio to the best format for each platform so you do not have to worry
about compatibility. You can override these defaults in your configuration if
needed.
| Platform | Default Format | Why |
| -------- | -------------- | -------------------------------------------- |
| Telegram | opus | Native voice message format for Telegram |
| Discord | mp3 | Widely supported across Discord clients |
| WhatsApp | mp3 | Compatible with WhatsApp voice playback |
| Slack | mp3 | Standard audio format for Slack file uploads |
| Other | mp3 | Safe default for any platform |
The format conversion happens automatically. Your agent does not need to know
which platform the user is on -- Comis selects the right format based on the
channel type.
## Configuration
Full configuration reference for both transcription and text-to-speech:
```yaml theme={null}
integrations:
media:
transcription:
provider: auto # auto (keyless-first), local, openai, groq, deepgram
model: gpt-4o-mini-transcribe # Model override (provider-dependent)
local: # Keyless local whisper engine (used by auto/local)
model: base # In-process model (auto-downloads once to ~/.comis/models/whisper/)
baseUrl: "" # Set → OpenAI-compatible local whisper server (no in-process download)
# engine: "" # Engine selector (reserved for a later release)
maxFileSizeMb: 25 # Max audio file size
timeoutMs: 60000 # API timeout (1 min)
language: "" # BCP-47 hint (e.g., "en", "es") -- empty = auto-detect
autoTranscribe: true # Auto-transcribe voice messages
preflight: true # Pre-transcribe for mention detection
fallbackProviders: [] # Ordered fallback list (auto/local/openai/groq/deepgram)
tts:
provider: edge # edge (keyless default), openai, elevenlabs, local
voice: alloy # Voice identifier
format: opus # Default output format
model: tts-1 # Model override
autoMode: off # off, always, inbound, or tagged
maxTextLength: 4096 # Max text for synthesis
tagPattern: "\\[\\[tts(?::.*?)?\\]\\]" # Regex for tagged mode
outputFormats: # Per-channel format overrides
telegram: opus
discord: mp3
whatsapp: mp3
slack: mp3
default: mp3
elevenlabsSettings: # ElevenLabs-specific
stability: 0.5
similarityBoost: 0.75
style: 0.0
useSpeakerBoost: true
speed: 1.0
```
## What Happens Without a Provider
Voice features degrade gracefully — and the keyless defaults mean "no provider"
is increasingly rare. You do not need every provider configured for the system to
work.
**When transcription resolves to honest-unavailable** (STT `auto` when the keyless
local engine cannot serve — e.g. ffmpeg is absent or the host has no egress to the
model hub and no `local.baseUrl` server — and there is no reusable main-provider
audio key, which is also the case for a Codex/OAuth-only main with none of the
above), voice messages are not auto-transcribed and **no adapter is constructed**
(so there is no empty-bearer 401). Instead, your agent still receives the message
with a hint to use the [transcribe\_audio](/agent-tools/media) on-demand tool. The
honest-unavailable reason is logged once, naming the exact remedy (install ffmpeg,
set `local.baseUrl`, or set `GROQ_API_KEY` / `OPENAI_API_KEY`). The agent's text
reply is never blocked.
**TTS defaults to keyless Edge**, so auto-reply works without any key. If you set
a keyed TTS provider without its key, the resolution is honest-unavailable and text
replies are sent as text regardless of `autoMode`; the agent can still use
[tts\_synthesize](/agent-tools/media) if a provider key is available.
This means voice works out of the box with zero credentials — add a keyed cloud
provider later only if you want one.
## Walkthrough: voice → transcript → reply → TTS
Here is the full round-trip for a WhatsApp voice note when transcription
(OpenAI) and auto-TTS (`autoMode: inbound`) are both configured.
A user holds the microphone in WhatsApp and says: "Can you check whether
we have any meetings tomorrow before 10am?" — sends the audio.
The WhatsApp adapter pulls the audio, decrypts it via Baileys, and hands
it to the transcription factory. OpenAI's `gpt-4o-mini-transcribe`
returns: `Can you check whether we have any meetings tomorrow before 10am?`
The agent receives the text plus a metadata flag indicating the source
was a voice message. With `autoMode: inbound`, that flag tells Comis to
reply by voice as well.
After looking up the calendar via an MCP tool, the agent writes:
"You have one meeting before 10am tomorrow — the standup at 9:30. Nothing
else booked until 11."
Because auto-TTS fired, the OpenAI TTS provider renders the reply with
the `alloy` voice into an mp3. The WhatsApp adapter sends it as an
`audioMessage` so it appears as a native voice note on the user's phone.
Total round-trip on a normal connection: about 3–5 seconds.
If TTS were not configured, the reply would still be delivered — just as
text instead of audio. The `inbound` setting only triggers voice when the
user spoke first, so written messages always get written replies.
## Observability and cost
Every transcription and synthesis is fully instrumented so an operator can
reconstruct any voice turn from logs and the observability tools alone — no live
repro needed.
### Logs
Each transcription/synthesis emits **one INFO completion line** carrying
`provider`, `keyless` (a boolean), `model`, `durationMs`, `audioBytes`,
`costUsd`, and the resolved selection `source` (the rung `auto` picked —
`keyless-local`, `follow-main-key`, `fallback`, or `explicit`). Every failure
branch emits a **WARN/ERROR** line with an `errorKind` (one of
`no_keyless_engine`, `auth_required`, `model_load_failed`,
`model_download_failed`, `timeout`, `network`, `dependency`) and an actionable
`hint`. Secrets are never logged at any level — a provider's `baseUrl` is logged
host-only (path and query dropped), and `Bearer` tokens, API keys, and the
keyless-LLM sentinel are stripped before a message reaches a log line.
### `comis explain` — reconstruct one voice turn
`comis explain ` reconstructs a voice turn offline from the
session's trajectory. The report's `voice` block shows the `provider`, whether
it ran `keyless`, the `model`, `durationMs`, the `outcome` (`ok`/`failed`), the
resolved selection `source` rung (and, when `auto` skipped a higher rung, the
`onSkip` reasons that explain *why* it landed where it did), and — on a failure
— the `errorKind`. The block is content-free: it carries ids, labels, numbers,
and booleans only, never the audio, the transcript text, or a credential.
### `comis fleet` — the voice-health signal
`comis fleet` surfaces a `voice_health` finding beside `model_health` and
`config_posture` when transcriptions/syntheses have been degrading across the
window: the **count of degraded STT/TTS turns** and the **dominant voice
`errorKind`** (for example, `model_load_failed` pointing at the local whisper
model cache, or `auth_required` pointing at a missing audio key). Like every
fleet finding it carries **counts, a short code, and a static hint only — no raw
provider message and no secret — so it is safe to paste into a review.** Drill
into the worst session it points at with `comis explain`.
### Cost honesty
Voice **cost reporting is honest about what the providers actually return.**
Keyless paths (the `local` whisper engine, Edge TTS) record `costUsd: 0`
**explicitly**, so "free" is *visible* on the `voice` block rather than an
ambiguous absence. Keyed providers, however, **do not return a per-call cost**
for transcription or synthesis today — neither the STT nor the TTS port carries
a cost field — so a keyed voice turn shows **no `costUsd` at all**. That absence
is honest: Comis never fabricates a price it did not receive. (Token-based LLM
cost for the agent's *reply* is tracked separately, as usual; this note is only
about the STT/TTS calls themselves.)
Voice turns appear in the existing per-session trajectory — voice observability
adds no new on-disk artifact or data-directory path.
## Related
Image and video analysis
Document text extraction
transcribe\_audio and tts\_synthesize tool reference
Back to media overview