Voice - Comis

When someone sends a voice message to your agent, Comis can transcribe it to text so the agent can read and respond. Optionally, the agent can reply with a voice message of its own using text-to-speech. The entire flow — from receiving a voice note to sending a spoken reply — can be fully automatic.

You don’t need to understand the technical details to use this feature. The configuration examples below are copy-paste ready.

Keyless by Default

Voice works with zero credentials out of the box. Speech-to-text defaults to auto (keyless-first: a local whisper engine when one is available, otherwise an honest “not available” hint) and text-to-speech defaults to edge (Microsoft Edge TTS — free, no API key). A keyed cloud provider (OpenAI, Groq, Deepgram, ElevenLabs) is used only when you set it explicitly or your agent’s main provider already supplies a usable audio key.

A Codex / ChatGPT-subscription (OAuth) login cannot drive OpenAI audio endpoints. The OAuth bearer is scoped to the Codex Responses backend and cannot reach api.openai.com/v1/audio/*. With the keyless-first resolver, a Codex-only agent is steered to a keyless provider and — if none is available yet — gets an actionable “not available” hint, never a silent HTTP 401 from an empty-bearer request. To use OpenAI audio, set a separate platform API key (OPENAI_API_KEY) and integrations.media.transcription.provider: openai explicitly.

How Voice Works

User sends a voice message

A user records a voice note in Telegram, Discord, WhatsApp, or any connected platform.

Comis transcribes the audio

The voice message is automatically converted to text using your configured speech-to-text provider. The agent sees the written transcript.

Agent reads and responds

Your agent processes the transcript like any text message and writes a reply.

Response is converted to speech (optional)

If auto-TTS is enabled, Comis converts the text reply into audio and sends it back as a voice message.

Voice also helps with mention detection. In group chats where the bot only responds when mentioned, Comis can pre-transcribe voice messages to check if the user said the bot’s name.

Transcription (Speech-to-Text)

The transcription provider is one of auto (the keyless-first default), local (a keyless local whisper engine), or a keyed cloud provider (openai, groq, deepgram). You only need one configured, and the default needs none — set up multiple via fallbackProviders for automatic failover.

Provider	Default Model	API Key	Notes
`auto`	—	None needed	Default. Keyless-first: resolves to the local engine when available, else follows the agent’s main provider key, else honest-unavailable
`local`	base	None needed	Keyless in-process whisper engine — auto-downloads a small model (default `base`) once to `~/.comis/models/whisper/` on first use, then reuses it. Or point `local.baseUrl` at an OpenAI-compatible local whisper server.
`openai`	gpt-4o-mini-transcribe	`OPENAI_API_KEY`	Explicit opt-in; fast
`groq`	whisper-large-v3-turbo	`GROQ_API_KEY`	Explicit opt-in; returns language detection and duration
`deepgram`	nova-3	`DEEPGRAM_API_KEY`	Explicit opt-in; high accuracy, real-time streaming

auto (default, keyless-first)

The default. Resolves a provider in priority order: a keyless local whisper engine first (when available), then the agent’s main-provider audio key only if one genuinely exists (e.g. an openai/groq main with that key set), then any configured fallbackProviders, and finally an honest “not available” hint naming the exact remedy. An OAuth-only main (Codex) has no reusable audio key, so it is steered straight to keyless / honest-unavailable — never an empty-bearer 401.

local (keyless)

A keyless local whisper engine — no API key, no cloud round-trip. Two modes:

In-process (default, zero-config). When local.baseUrl is unset, Comis runs whisper in-process via Transformers.js. The engine lazy-loads on the first voice note: it auto-downloads a small model (transcription.local.model, default base) once over TLS from the model hub to the scoped cache ~/.comis/models/whisper/, then reuses the cached model for the life of the process — so the first transcription is slower (it pays the download + load) and every subsequent one is fast. The engine runs on ONNX Runtime, which the daemon’s existing --allow-addons flag already covers — no new permission flag is needed (see Node Permissions). If the engine or model can’t load, or ffmpeg is absent, transcription honest-degrades to “not available” (so auto falls through to the next rung) — it never crashes the inbound message.
local.baseUrl server (escape hatch). Set transcription.local.baseUrl to an OpenAI-compatible local whisper server (faster-whisper-server, Speaches, whisper.cpp-server, …) and Comis talks to it keyless — no in-process model download, fully offline-capable, and GPU-accelerated if the server is. A reachable server makes local available without the in-process engine.
local.baseUrl is SSRF-guarded (loopback-only by default). The URL is validated (with DNS-rebinding protection) before any request: only loopback (127.0.0.1, localhost, ::1) and explicitly-allowlisted hosts are permitted; public and cloud-metadata addresses are denied. A rejected baseUrl is treated as unreachable (the resolver falls through to the in-process engine or honest-unavailable) — it never drives an outbound request. Point it at a whisper server on the local host, not a public URL.

OpenAI

Explicit opt-in. Uses the gpt-4o-mini-transcribe model. Reliable and fast for most use cases. Requires an OpenAI platform API key (OPENAI_API_KEY) — a Codex/ChatGPT OAuth login cannot drive audio. Does not return language or duration metadata in the response. Set provider: openai to use it.

Groq

Uses Whisper Large V3 Turbo. Returns rich metadata including the detected language of the audio and total duration. Requires a Groq API key. A good alternative if you want automatic language detection.The language detection feature is especially useful for multilingual communities where voice messages arrive in different languages.

Deepgram

Uses the Nova 3 model. Known for high accuracy and fast processing speeds. Requires a Deepgram API key. Internally uses a different API format (raw binary instead of form data), but this is handled automatically — you just provide the API key.Deepgram is a good choice when transcription accuracy is your top priority, especially for noisy audio or accented speech.

Fallback Chains

If your primary provider fails (API error, timeout, rate limit), Comis tries the next provider in your fallback list. This keeps voice transcription working even when a single provider has issues.

integrations:
  media:
    transcription:
      provider: auto                     # Keyless-first default
      fallbackProviders:                 # Try these if the primary fails
        - groq
        - deepgram

In this example, Comis resolves auto first (keyless / follow-main). If that resolution cannot serve, it tries Groq, then Deepgram. The first successful transcription is used. (auto and local are also valid fallbackProviders entries — auto is a benign no-op rung the resolver collapses.)

Text-to-Speech (TTS)

The TTS provider is one of edge (the keyless default), openai, elevenlabs, or local. You only need one configured, and the default needs none — the right choice depends on voice quality, budget, and whether you need an API key.

Provider	Default Model	Default Voice	API Key	Cost
`edge`	—	en-US-AvaMultilingualNeural	None needed	Default. Free (network-dependent)
`local` / `piper`	Xenova/mms-tts-eng	—	None needed	Keyless in-process offline TTS — no network at synth time
`openai`	tts-1	alloy	`OPENAI_API_KEY`	Paid (explicit opt-in)
`elevenlabs`	eleven_multilingual_v2	Rachel	`ELEVENLABS_API_KEY`	Paid (explicit opt-in)

Keyless default: TTS defaults to edge. Edge needs no key, so auto-reply works out of the box. A Codex/OAuth-only main also resolves to edge. Set provider: openai or provider: elevenlabs explicitly (with the matching key) for the paid providers.

Edge TTS (default, keyless)

The default. Microsoft Edge text-to-speech — free, no API key required. Default voice is en-US-AvaMultilingualNeural. Works out of the box for every install (including Codex/OAuth-only mains). Limited customization compared to OpenAI and ElevenLabs, but a great zero-config baseline. Network-dependent (it calls a Microsoft Edge endpoint); for a fully offline voice, use local/piper below.

Local / Piper (offline, keyless)

Keyless offline TTS. provider: local (or its alias piper) runs a small single-speaker text-to-audio model (Xenova/mms-tts-eng) in-process via @huggingface/transformers — the TTS twin of the local whisper STT engine. The model auto-downloads once over TLS to ~/.comis/models/tts/ (a separate dir from models/whisper/, already inside the daemon’s --allow-fs-write scope), then synthesizes with no network at synth time. A short/corrupt model download fails closed (model_load_failed); if the engine or model is unavailable it honest-degrades to a structured error (never crashes the reply). With TTS provider: auto, the chain is edge → local/piper → honest-unavailable, so a no-network install still speaks once the model is cached.

OpenAI

Explicit opt-in. Uses the tts-1 model with the “alloy” voice. Supports speed adjustment from 0.25x to 4.0x. Requires an OpenAI platform API key (OPENAI_API_KEY) — a Codex/ChatGPT OAuth login cannot drive audio. A good balance of quality and speed.OpenAI offers several voice options beyond “alloy.” You can change the voice in your configuration or per-reply using TTS directives.

ElevenLabs

Premium voice synthesis using the eleven_multilingual_v2 model. Default voice is “Rachel.” Offers advanced voice settings: stability, similarity boost, style, and speaker boost. Requires an ElevenLabs API key. Best voice quality for human-like speech.ElevenLabs voices sound the most natural and expressive. The advanced settings let you fine-tune how the voice sounds — for example, increasing stability makes the voice more consistent, while boosting style adds more expressiveness.

Auto-Reply Modes

The autoMode setting controls when Comis automatically converts text replies into voice messages.

Mode	Behavior
`"off"` (default)	TTS only when the agent explicitly uses the `tts_synthesize` tool
`"always"`	Every text reply is automatically converted to a voice message
`"inbound"`	Reply with voice only when the user sent a voice message first
`"tagged"`	Reply with voice only when the agent’s response contains a `[[tts]]` directive

Off

The default. Your agent’s text replies are sent as text. The agent can still generate voice on demand using the tts_synthesize tool, but nothing is automatic.

Always

Every text reply from the agent is also sent as a voice message. Useful for accessibility or when users prefer listening. Replies that already contain media attachments skip TTS to avoid sending double media.

Inbound

The agent mirrors the user. If the user sends a voice message, the reply is a voice message. If the user sends text, the reply is text. This creates natural conversational behavior where the agent adapts to how the user is communicating.

Tagged

The agent controls when to use voice. When the LLM includes [[tts]] in its output, that reply is converted to speech. The tag is stripped from the final text so the user never sees it.

TTS Directives

In tagged mode, the agent can include directives to fine-tune how the voice reply sounds. The [[tts:...]] syntax allows overriding the voice, provider, format, and speed for individual replies.

[[tts]]                                    # Use default voice and provider
[[tts:voice=nova]]                         # Override the voice
[[tts:voice=nova provider=openai]]         # Override voice and provider
[[tts:voice=nova speed=1.5 format=mp3]]    # Override multiple settings

Available directive parameters:

Parameter	Description
`voice`	Voice identifier (provider-specific)
`provider`	TTS provider to use for this reply
`format`	Audio format (`opus`, `mp3`, etc.)
`speed`	Playback speed multiplier

Directives work in tagged mode. In always or inbound mode, TTS happens automatically using your default settings — directives are not needed.

Per-Channel Audio Formats

Different platforms work best with different audio formats. Comis automatically converts audio to the best format for each platform so you do not have to worry about compatibility. You can override these defaults in your configuration if needed.

Platform	Default Format	Why
Telegram	opus	Native voice message format for Telegram
Discord	mp3	Widely supported across Discord clients
WhatsApp	mp3	Compatible with WhatsApp voice playback
Slack	mp3	Standard audio format for Slack file uploads
Other	mp3	Safe default for any platform

The format conversion happens automatically. Your agent does not need to know which platform the user is on — Comis selects the right format based on the channel type.

Configuration

Full configuration reference for both transcription and text-to-speech:

integrations:
  media:
    transcription:
      provider: auto                  # auto (keyless-first), local, openai, groq, deepgram
      model: gpt-4o-mini-transcribe   # Model override (provider-dependent)
      local:                          # Keyless local whisper engine (used by auto/local)
        model: base                   # In-process model (auto-downloads once to ~/.comis/models/whisper/)
        baseUrl: ""                   # Set → OpenAI-compatible local whisper server (no in-process download)
        # engine: ""                  # Engine selector (reserved for a later release)
      maxFileSizeMb: 25               # Max audio file size
      timeoutMs: 60000                # API timeout (1 min)
      language: ""                    # BCP-47 hint (e.g., "en", "es") -- empty = auto-detect
      autoTranscribe: true            # Auto-transcribe voice messages
      preflight: true                 # Pre-transcribe for mention detection
      fallbackProviders: []           # Ordered fallback list (auto/local/openai/groq/deepgram)

    tts:
      provider: edge                  # edge (keyless default), openai, elevenlabs, local
      voice: alloy                    # Voice identifier
      format: opus                    # Default output format
      model: tts-1                    # Model override
      autoMode: off                   # off, always, inbound, or tagged
      maxTextLength: 4096             # Max text for synthesis
      tagPattern: "\\[\\[tts(?::.*?)?\\]\\]"  # Regex for tagged mode
      outputFormats:                  # Per-channel format overrides
        telegram: opus
        discord: mp3
        whatsapp: mp3
        slack: mp3
        default: mp3
      elevenlabsSettings:             # ElevenLabs-specific
        stability: 0.5
        similarityBoost: 0.75
        style: 0.0
        useSpeakerBoost: true
        speed: 1.0

What Happens Without a Provider

Voice features degrade gracefully — and the keyless defaults mean “no provider” is increasingly rare. You do not need every provider configured for the system to work. When transcription resolves to honest-unavailable (STT auto when the keyless local engine cannot serve — e.g. ffmpeg is absent or the host has no egress to the model hub and no local.baseUrl server — and there is no reusable main-provider audio key, which is also the case for a Codex/OAuth-only main with none of the above), voice messages are not auto-transcribed and no adapter is constructed (so there is no empty-bearer 401). Instead, your agent still receives the message with a hint to use the transcribe_audio on-demand tool. The honest-unavailable reason is logged once, naming the exact remedy (install ffmpeg, set local.baseUrl, or set GROQ_API_KEY / OPENAI_API_KEY). The agent’s text reply is never blocked. TTS defaults to keyless Edge, so auto-reply works without any key. If you set a keyed TTS provider without its key, the resolution is honest-unavailable and text replies are sent as text regardless of autoMode; the agent can still use tts_synthesize if a provider key is available. This means voice works out of the box with zero credentials — add a keyed cloud provider later only if you want one.

Walkthrough: voice → transcript → reply → TTS

Here is the full round-trip for a WhatsApp voice note when transcription (OpenAI) and auto-TTS (autoMode: inbound) are both configured.

User records a voice note

A user holds the microphone in WhatsApp and says: “Can you check whether we have any meetings tomorrow before 10am?” — sends the audio.

Comis downloads and transcribes

The WhatsApp adapter pulls the audio, decrypts it via Baileys, and hands it to the transcription factory. OpenAI’s gpt-4o-mini-transcribe returns: Can you check whether we have any meetings tomorrow before 10am?

The agent reads the transcript

The agent receives the text plus a metadata flag indicating the source was a voice message. With autoMode: inbound, that flag tells Comis to reply by voice as well.

The agent calls calendar tools and writes a reply

After looking up the calendar via an MCP tool, the agent writes: “You have one meeting before 10am tomorrow — the standup at 9:30. Nothing else booked until 11.”

Comis synthesises the reply

Because auto-TTS fired, the OpenAI TTS provider renders the reply with the alloy voice into an mp3. The WhatsApp adapter sends it as an audioMessage so it appears as a native voice note on the user’s phone.

The user listens to the reply

Total round-trip on a normal connection: about 3–5 seconds.

If TTS were not configured, the reply would still be delivered — just as text instead of audio. The inbound setting only triggers voice when the user spoke first, so written messages always get written replies.

Observability and cost

Every transcription and synthesis is fully instrumented so an operator can reconstruct any voice turn from logs and the observability tools alone — no live repro needed.

Logs

Each transcription/synthesis emits one INFO completion line carrying provider, keyless (a boolean), model, durationMs, audioBytes, costUsd, and the resolved selection source (the rung auto picked — keyless-local, follow-main-key, fallback, or explicit). Every failure branch emits a WARN/ERROR line with an errorKind (one of no_keyless_engine, auth_required, model_load_failed, model_download_failed, timeout, network, dependency) and an actionable hint. Secrets are never logged at any level — a provider’s baseUrl is logged host-only (path and query dropped), and Bearer tokens, API keys, and the keyless-LLM sentinel are stripped before a message reaches a log line.

`comis explain` — reconstruct one voice turn

comis explain <sessionKey> reconstructs a voice turn offline from the session’s trajectory. The report’s voice block shows the provider, whether it ran keyless, the model, durationMs, the outcome (ok/failed), the resolved selection source rung (and, when auto skipped a higher rung, the onSkip reasons that explain why it landed where it did), and — on a failure — the errorKind. The block is content-free: it carries ids, labels, numbers, and booleans only, never the audio, the transcript text, or a credential.

`comis fleet` — the voice-health signal

comis fleet surfaces a voice_health finding beside model_health and config_posture when transcriptions/syntheses have been degrading across the window: the count of degraded STT/TTS turns and the dominant voice errorKind (for example, model_load_failed pointing at the local whisper model cache, or auth_required pointing at a missing audio key). Like every fleet finding it carries counts, a short code, and a static hint only — no raw provider message and no secret — so it is safe to paste into a review. Drill into the worst session it points at with comis explain.

Cost honesty

Voice cost reporting is honest about what the providers actually return. Keyless paths (the local whisper engine, Edge TTS) record costUsd: 0 explicitly, so “free” is visible on the voice block rather than an ambiguous absence. Keyed providers, however, do not return a per-call cost for transcription or synthesis today — neither the STT nor the TTS port carries a cost field — so a keyed voice turn shows no costUsd at all. That absence is honest: Comis never fabricates a price it did not receive. (Token-based LLM cost for the agent’s reply is tracked separately, as usual; this note is only about the STT/TTS calls themselves.) Voice turns appear in the existing per-session trajectory — voice observability adds no new on-disk artifact or data-directory path.

Vision

Image and video analysis

Documents

Document text extraction

Media Tools

transcribe_audio and tts_synthesize tool reference

Media Overview

Back to media overview

​Keyless by Default

​How Voice Works

​Transcription (Speech-to-Text)

​Fallback Chains

​Text-to-Speech (TTS)

​Auto-Reply Modes

​Off

​Always

​Inbound

​Tagged

​TTS Directives

​Per-Channel Audio Formats

​Configuration

​What Happens Without a Provider

​Walkthrough: voice → transcript → reply → TTS

​Observability and cost

​Logs

​comis explain — reconstruct one voice turn

​comis fleet — the voice-health signal

​Cost honesty

​Related

Vision

Documents

Media Tools

Media Overview

Keyless by Default

How Voice Works

Transcription (Speech-to-Text)

Fallback Chains

Text-to-Speech (TTS)

Auto-Reply Modes

Off

Always

Inbound

Tagged

TTS Directives

Per-Channel Audio Formats

Configuration

What Happens Without a Provider

Walkthrough: voice → transcript → reply → TTS

Observability and cost

Logs

`comis explain` — reconstruct one voice turn

`comis fleet` — the voice-health signal

Cost honesty

Related