Voice API
Voice requires gateway orchestration and agents[].voice.enabled. See orchestration configuration.
Channel speech failures
Telegram, Discord, LINE and Slack deliver the response text before speech. If synthesis fails, the same response/destination gets one durable plain-text notice explaining that audio could not be created, with a normalized provider diagnostic. All provider failures use concise, provider-independent wording, retain the HTTP status when available, and suggest a relevant action or /voice off. Chat notices omit technical details and reference IDs; the failed delivery ID remains the log reference. Quota, rate limit, payment, authentication, permission, unavailable, timeout, network, language/model, invalid response and unknown errors each have short messages. Unknown future failures use a safe generic notice. The failure and pending notice are committed together; a restart does not duplicate the notice or retry paid synthesis. Gateway logs and the delivery outbox retain the normalized failure code. Raw provider messages and credentials are never included. A Google structured daily quota violation is identified as a daily limit; a bare HTTP 429 remains ambiguous between rate limiting and exhausted quota. Turning voice off or an uncertain channel send does not produce a misleading synthesis-failure notice.
Telegram voice notes
Voice-note failures retain normalized provider codes in voice_note_transcripts and return a readable provider diagnostic plus a Reference input ID in the original conversation. Gateway logs record the same reference, provider, model, HTTP status and category without raw provider messages or credentials. This applies to the shared voice-note flow across channels, including Gemini and Paxa errors and ElevenLabs batch HTTP errors. A failed note is not automatically retried; quota/payment/configuration failures should be resolved before resending.
For orchestration-enabled Telegram ingress, agents[].voice.enabled together with agents[].voice.notes.enabled enables transcription. Outgoing voice controls are available by default; set agents[].voice.notes.replyWithVoice: false to deny them for an Agent. The old gateway-level replyWithVoice value is ignored. Inbound transcription can be disabled independently. Voice synthesis requires a supported voice.tts provider (ElevenLabs, Cartesia, Gemini, PaxaLabs, or OpenRouter) and an optional voiceId (empty selects Auto), without a browser WebSocket. Use a supported direct, upstream, or managed route from the provider guide. Authorized users in private chats use /voice and its Always / Voice messages only / Off buttons, or /voice on, /voice auto and /voice off directly. The preference is stored per Agent/chat in telegram_voice_preferences, defaults to Off even when voice capability is enabled, and survives session switches/restarts. On enables bounded spoken summaries for both typed and voice-note replies and automatic Worker reports; Off keeps text-only responses. The English menu includes Dismiss, which closes it without changing the preference. Successful mode selection replaces the menu with a persistent confirmation. The menu never invokes inference or TTS. Pending audio checks the preference before synthesis and again before Telegram submission; a send already submitted cannot be recalled by this toggle.
Voice reply mode auto sends audio only for voice-origin inputs and their dependent task results; typed-only conversations remain text-only. Modes apply to Telegram, Discord, LINE and Slack. Preferences are stored in voice_reply_modes per Agent/chat, with old boolean preferences used as a fallback; old On/Off choices retain their meaning. Browser live voice remains independent.
The gateway synthesizes MP3 and submits Telegram sendVoice to the original bound chat/topic. Speech deliveries have their own durable outbox entry and provider message receipt, with the voice/model pinned when queued. Failed synthesis keeps the text reply available. Ambiguous sends remain unknown for reconciliation and are not blindly retried. Voice-enabled Agent turns use CLI JSON-schema validation for display_text and spoken_text, and consume structured_output from the terminal result. Short plain conversational replies (up to 600 characters, without code, URLs or structured data) can be spoken directly in their original language. Long reports require the explicit spoken summary and are never truncated into audio. Browser voice changes do not change the Agent-configured Telegram voice.
/voices uses the same provider-neutral catalog as web Voice controls. ElevenLabs and Cartesia adapters list real provider voices; Cartesia pages through its voice catalog API. Each chat’s choice is stored in telegram_tts_voices, scoped to the provider. Voice messages and a Telegram /call in that chat both speak with it; a /call reads it when the call socket starts, so a change applies from the next call or reconnect. API voice sessions keep voice.tts.voiceId or their own voice_id. Telegram menus first select a gender category, then show paginated voice names with ✅ on the selected voice and Back/Dismiss controls. Menus are bound to the chat/provider for five minutes. Selection does not enable auto voice replies. Changed providers require a new selection if the chat had explicitly selected an old provider’s voice. Missing catalogs report unavailable rather than inventing voices. Speech file output supports native MP3 through ElevenLabs and Cartesia; other providers need a catalog and file-synthesis adapter.
Telegram registers /tasks, /voice, /voices and /call only with orchestration enabled for Telegram. /cli is registered only with gateway.headless: false, and is hidden for app-agents and orchestration agents. Help and callback handling follow the same capability checks. Hot-reloading orchestration/headless mode refreshes receiver command registration without stopping Agent/Worker tasks.
Telegram /tasks is a direct gateway control for the active chat session. Its paginated list and detail callbacks check conversation membership; cancellation checks the task ID against the current session before calling the same user-cancellation service as /stop. Viewing/refreshing tasks does not interrupt Agent responses. Old task buttons cannot cancel work after switching to another session. Progress is the latest worker/runtime report, not an inferred percentage.
Live voice endpoints
For direct and upstream provider values, API-key locations, and per-Agent examples, see Setting up voice providers.
| Method | Path | Purpose |
|---|---|---|
GET | /api/v1/agents/:agentId/voice-settings | Effective voice settings and orchestration_enabled; requires access to this Agent |
PATCH | /api/v1/agents/:agentId/voice-settings | Persist a partial Agent voice override; requires write permission for this Agent |
GET | /api/v1/agents/:agentId/voice-settings/catalog?provider=upstream | Provider voice/model catalog for the settings UI; provider may also be elevenlabs, cartesia, deepgram, paxalabs, or upstream:paxalabs |
GET | /api/v1/agents/:agentId/voice-sessions/voices | Available voices: { voices: [{ id, name, gender? }], default_voice_id } |
POST | /api/v1/agents/:agentId/sessions/:sessionId/voice-sessions | Create a voice lease and single-use WebSocket ticket; body { "chat_id": "<chat-id>" } |
GET (WebSocket upgrade) | /api/v1/agents/:agentId/voice-sessions/:voiceSessionId/stream?ticket=<ticket> | Duplex audio/control stream |
DELETE | /api/v1/agents/:agentId/sessions/:sessionId/voice-sessions/:voiceSessionId | Close the owned voice lease; returns 204 |
These HTTP endpoints require an authorized API key; Telegram voice calls authenticate with Telegram initData instead. Voice must be enabled at agents[].voice.enabled, with gateway orchestration enabled. Ticket creation verifies the existing chat/session pair and principal. Only one voice lease is allowed per conversation. Tickets expire after 30 seconds and are consumed at upgrade; the WebSocket must use the same Origin as ticket creation. No public URL or voice.allowedOrigins setup is required. Direct browser requests use API-key authentication with non-credentialed CORS; cookie-authenticated products must enforce session ownership and credentialed CORS at their trusted proxy. The legacy allowedOrigins field is ignored. Disabled voice or an occupied lease returns 409; ownership/origin errors are rejected. Do not send the API key in the WebSocket URL.
Creation returns voice_session_id, ticket, expires_at (Unix milliseconds), stream_path, input/output formats and duplex/playback capabilities. Both audio directions currently use mono 16 kHz PCM signed 16-bit little-endian.
Voice settings writes validate the same voice configuration schema as startup, reject unknown fields, and merge nested stt, tts, notes, turns and playback overrides under a configuration write lock. live and narrate values, including the fields of live.vad, are saved only when they differ from the agent's current settings, so echoing the read payload back does not pin the defaults. They write only agents[].voice and preserve other agents and task settings. An empty new configuration stays disabled; enabling it requires selected speech models. Changes apply to new voice connections and subsequent replies; existing queued speech retains its pinned settings. Enabling Agent voice does not enable gateway orchestration. Browser clients do not need to save settings to register a URL. API keys are never accepted in this body. The catalog returns { provider, voices, models, warnings }. Model entries include can_do_text_to_speech, can_do_speech_to_text, realtime, voice_messages, and optional conversation capabilities so clients can combine connected providers into role-specific model pickers. Scribe STT entries come from supported transport IDs (the ElevenLabs model endpoint lists synthesis models); Cartesia currently lists the adapter-supported Sonic 3 model. Paxa and Deepgram model inventories are fetched from their provider APIs; unavailable credentials/catalogs return 503 without upstream credentials.
The upstream:elevenlabs provider (upstream is a legacy alias) uses the origin of ANTHROPIC_BASE_URL, ANTHROPIC_AUTH_TOKEN (falling back to ANTHROPIC_API_KEY, then CLAUDE_CODE_OAUTH_TOKEN), and the /v1/voice/elevenlabs/ relay endpoints. It requires a provider service implementing this relay; an ordinary Anthropic-compatible messages endpoint alone is insufficient. The relay resolves the user's ElevenLabs BYOK connection. Direct elevenlabs configuration remains supported with ELEVENLABS_API_KEY.
Voice settings language is a recognition hint for STT. TTS follows the language of the approved spoken response, including typed messages while the microphone is muted. Model and voice catalog permissions are independent: a missing model-list permission returns usable voices and supported synthesis defaults marked catalog_source: supported_defaults, with a warning to enable models_read for the complete provider catalog.
After connecting, send JSON { "type": "voice.start", "voice_id": "<catalog-id>" } (voice_id is optional). Include muted: true to restore a muted microphone without opening STT; the server reports state: "muted" and TTS remains available. Wait for voice.state with state: "listening", generation, epoch and utterance_id before sending microphone frames.
Binary audio uses a 44-byte header followed by PCM bytes: ASCII CGV1 (4 bytes), generation UUID (16 bytes), epoch (uint32 big-endian), sequence (uint32 big-endian), segment UUID (16 bytes). Microphone segment IDs are the current utterance ID; playback frames identify the segment from playback.start. Reject stale generations/epochs. Payloads must have an even byte length and are bounded to 32,000 bytes per frame.
Client JSON controls:
| Type | Fields / behavior |
|---|---|
voice.configure | voice_id; validated against the catalog, acknowledged by voice.configured |
speech.activity | Signals current microphone speech activity |
speech.started | epoch; signals a possible interruption |
speech.ended | last_audio_seq; signals an utterance boundary |
utterance.commit | last_audio_seq, optional final: true; finalize received audio |
voice.mute | muted boolean, policy: "commit" or "discard", last_audio_seq; commit pending input before muting or discard it. With commit, an open utterance (last_audio_seq > 0) or one waiting out the silence timer or merge window is answered at once; with nothing pending, no turn is created. Replies keep playing while muted |
playback.progress | epoch, sample_offset; samples actually played, not merely received |
playback.stop | epoch, request_id (positive integer); stop live output without cancelling agent work. The server advances beyond its current epoch even when the browser has not seen the latest start, clears output, then acknowledges with playback.stopped carrying the request ID, generation and authoritative epoch. Ignore start/audio events until this acknowledgement; only then begin explicit replay. |
playback.pause | epoch, paused boolean; preserve queued output while STT classifies a microphone candidate. Pause-aware backpressure is bounded by the configured utterance plus STT finalization budget; repeated pause messages do not extend it. Resume after empty STT or microphone cancellation, not a fixed classification timer. |
playback.clear.ack | Acknowledge that queued audio was cleared |
voice.stop | Close voice transport; does not cancel background tasks. Optional hangup boolean: true ends a Telegram call now instead of waiting for it to be rejoined |
Server controls include voice.state, voice.configured, stt.partial, stt.segment, stt.final, utterance.accepted, response.text, response.speech, playback.start, playback.end, playback.clear, playback.stopped, voice.notice and voice.error. Correlate chat/voice output using response_id and request_id when present; avoid rendering duplicate bubbles. Clear local queued audio on playback.clear. SPEECH_SUMMARY_UNAVAILABLE is a notice that approved spoken text is unavailable; retain the text response instead of reading the full report aloud.
If the STT provider connection ends unexpectedly, the gateway sends voice.notice with reconnect: true and retryable: true, then closes the WebSocket with code 1012. Obtain a fresh ticket and reconnect with backoff; never reuse the consumed ticket. Committed conversation work continues. Audio from an unfinished utterance is not replayed automatically; the user may need to repeat it. Authentication, billing and other non-retryable errors remain voice.error and must not trigger an automatic retry loop.
Voice errors include a safe message, category, retryable, optional httpStatus, and referenceId alongside code. Preview generation failures return the same diagnostic fields plus error. The reference correlates with server logs; raw provider bodies and credentials are never returned. Payment/authentication/permission/quota errors require corrective action before retrying. Explicit rate-limit, network, timeout and temporary-unavailability errors permit a later retry; this flag does not automatically retry or resubmit audio. A bare HTTP 429 remains quota_or_rate_limit unless a recognized provider code distinguishes the cause. Unknown errors remain non-retryable diagnostics rather than guessed billing or model failures. EMPTY_TRANSCRIPT (the recognizer heard sound but no words, such as a cough or room noise) has category no_speech and is retryable; a hands-free client keeps listening rather than showing it as an error.
A connected, same-principal voice session can speak typed chat responses while its microphone is muted, including a task acknowledgement and the later worker result. The chosen voice remains the vocal identity. Spoken summaries follow the response language unless the user explicitly requests different chat/audio languages.
Telegram voice calls
/call in a bot's private chat opens a Telegram Mini App that uses the same WebSocket protocol. It needs gateway orchestration serving telegram, agents[].voice.enabled, and an HTTPS gateway.publicUrl; it works without gateway.api.keys. Routes are relative to the gateway.publicUrl path:
| Method | Path | Purpose |
|---|---|---|
GET | /voice-call/:pairingId | The call page (Cache-Control: no-store). Loading it never claims the link. Once a ticket claimed it, the page renders "This call link was already used" and starts nothing unless it holds that call's token (a reload of the same call, which then rejoins). 404 with "This call link has expired" for an expired or unknown link |
GET | /voice-call/:pairingId/avatar[?v=<file_unique_id>] | The bot's profile photo (image/jpeg, image/png or image/webp); 404 (no-store) when the bot has no photo or the link expired. Without a known photo it waits for the lookup the page started; it never starts one itself. With v naming the current photo it is Cache-Control: private, max-age=3600, otherwise no-store. The page always links it for an agent with a Telegram bot, versioned by Telegram's file_unique_id once known, and shows the name's initial when it fails to load |
POST | /voice-call/:pairingId/ticket | Body { "initData": "<Telegram.WebApp.initData>", "call_token": "<optional>" }; returns the ticket fields above plus call_token, and call_elapsed_ms when this call is already in progress |
The pairing id in the link is not a credential. Each ticket request verifies the initData signature with the agent's bot token, requires the user the link was sent to, and re-checks that the user is still in the receiver's allowFrom and that dmPolicy is not disabled. The first ticket claims the link and returns call_token; later tickets for the same call, such as a reconnect after close code 1012, must present it. Opening a call needs initData at most one hour old; a claimed call that presents its call_token may reconnect with older initData, because Telegram does not refresh it during a session. A link opened elsewhere gets 409 with code: CALL_LINK_USED. Unclaimed links expire after three minutes; a claimed call can take new tickets for 30 minutes after its last ticket or the last time it was connected. If the chat switches sessions, the next utterance fails with voice.error code VOICE_SESSION_CHANGED and the user starts a new call.
| Status | Meaning |
|---|---|
401 | Missing, forged or expired initData |
403 | Another user's initData, or the user is no longer in allowFrom |
409 | code: VOICE_DISABLED; code: CALL_LINK_USED; or, without a code, another voice session holds this conversation |
410 | code: CALL_LINK_EXPIRED |
429 / 503 | Managed voice credit, as for API tickets |
Every refusal from these routes has a JSON body with error. Behind a proxy that requires its own sign-in (on a GetPod pod, every /gateway/* path except /gateway/shared/ and /gateway/webhooks), the call page follows the same flow as /cli: opening the link without a session redirects to the proxy's login page and back to the call link, keeping Telegram's initData in the URL fragment. The ticket request and the WebSocket then carry the proxy's session cookie, and the WebSocket's Origin is the page's own. If a ticket request is refused with 401 or 403 without a JSON error (the proxy session expired, or another account is signed in), the page ends the call and offers Sign in again, which reloads the page into that login redirect.
Tickets keep the API rules: 30-second lifetime, single use, bound to the request Origin. A ticket request with the call's own token is never blocked by that call's own earlier connection: an unused ticket (the page closed before connecting) is revoked, and a socket the gateway still holds (the network changed before its heartbeat noticed the old socket was gone) is closed with code 4001, so the page rejoins at once. That closed socket counts as a disconnect of the call. Another call in the same chat session still gets 409. The page builds the WebSocket URL from the public URL path and stream_path. Each utterance is submitted with the Telegram scope of the private chat (source: telegram, the chat's active session and topic, principal telegram:<userId>), so replies are delivered to that chat. Execution follows the Telegram rule (allow_tools !== false); live voice never writes memory. A Telegram call does not save browser voice intent. API-key reloads close API voice sessions with 1008 but leave Telegram calls open.
A text reply delivered to a channel for live-voice input starts with one 🎙 <transcript> line for each utterance spoken since the chat last received a reply, so a turn interrupted by barge-in is quoted with the next reply. Conversation history stores the reply without these lines. Channel voice replies (/voice on or auto) are not sent for replies the caller hears in the call: replies finished while the same user's call is connected (answers to spoken turns and to typed messages alike), and replies that already played in the call. A reply that finishes after the call ended, with none of it played, follows the chat's voice rule.
The page captures with an AudioWorklet, resamples to 16 kHz PCM16, and detects speech with an energy threshold over an adaptive noise floor. It sends speech.activity when speech starts, or speech.started with the next epoch while reply audio is audible, then speech.ended; the gateway commits after turns.silenceCommitMs. Words spoken before the next utterance opens are held and sent first. While reply audio is audible, speech must by default be twice as loud for 250 ms to interrupt it, so speaker echo does not. The agent's voice.live.vad settings (bargeInRatio, bargeInMs and the other detector fields) override these defaults. The page starts the call as it opens. Telegram's Android WebView blocks audio until the first touch, and some WebViews grant the microphone only from a tap; then the page shows one Start call button instead. On Telegram deactivated or a hidden page it sends voice.stop and closes the socket, so replies are delivered to the chat as without a call; on return it takes a new ticket with its call token. Hanging up sends voice.stop with hangup: true.
A call runs from its first socket that reaches listening until hang-up. Reconnects with the call token continue the same call; a ticket for a call in progress returns call_elapsed_ms, and the page's call clock resumes from it. A call that no socket rejoins within 60 seconds ends at its last disconnect. When a call ends, the gateway posts one system notice to the chat (and topic) the call joined, such as 📞 Call ended · 3:42. It is delivered like other orchestration notices, without inference. A page that never reached listening posts nothing, and a gateway restart drops the notice for calls still open. The page proposes the call token on its first ticket request, so a lost response can be retried. It calls getUserMedia once per call. Mute sends voice.mute with policy: "commit" and the open utterance's last sequence, so what the caller said is answered at once; words spoken after speech.ended wait for the next listening utterance, are sent there, and are then committed by the same mute. Each time /call sends its button, and again as the call page opens, the gateway looks the bot up with its token (getMe and getUserProfilePhotos, in parallel), so a new name or photo set in BotFather shows on the next call. The page shows the bot's Telegram name (the agent id if Telegram cannot be reached). The photo is downloaded (getFile) only when its file_unique_id changes; when Telegram cannot be reached, the last known name and photo stay. The page waits at most 1.5 seconds for this lookup; a slower name shows from the next call, while the photo request waits for the lookup. The Telegram file URL, which contains the token, never reaches the page. A /call sent in a private-chat topic binds the call to that topic (thread_key). The page plays reply audio through one AudioWorklet node that resamples the 16 kHz stream continuously to the device rate, so frame boundaries do not click. Each start waits for 200 ms of audio (or 200 ms after the first sample), and starts, underruns and barge-in fade over 4 ms. playback.progress reports the samples that node has played. The page creates its AudioContext after getUserMedia, because on iOS the microphone can change the audio session's rate, and logs both rates to the console once per call.
PaxaLabs speech providers
Agent voice settings accept paxalabs (gateway environment key) and upstream:paxalabs (BYOK proxy under ANTHROPIC_BASE_URL) for voice.tts.provider, voice.stt.provider, and voice.notes.provider. Models are paxa-tts-flash-v1 and paxa-stt-lite-v1-preview, respectively. GET /api/v1/agents/:agentId/voice-settings includes direct_providers, containing provider names only, never credentials.
The voice WebSocket voice.state listening event includes stt_mode: "batch" | "realtime". In batch mode, utterance.commit (including microphone mute with commit policy) finalizes the recording and starts transcription; stt.partial is not emitted. Clients should display a transcribing state until the final transcript is available. Paxa accepts only th/en language hints; omission enables detection.
Voice model catalog and preview
GET /api/v1/agents/:agentId/voice-settings/catalog?provider=upstream:gemini lists connected Gemini models and prebuilt voices. Gemini TTS uses generateContent; Gemini browser STT uses realtime partial transcripts with gemini-3.5-transcribe-live; other Gemini STT models transcribe recorded segments after pause/mute. Uploaded voice messages use a recorded-file model configured in voice.notes. Direct Gemini uses GEMINI_API_KEY; upstream Gemini reuses the pod provider credentials.
BYOK catalog IDs use <provider>/<native-model>, e.g. gemini/gemini-3.1-flash-tts-preview. Catalog entries include native_model_id, provider, source (byok or gateway), and metered: false. Legacy bare IDs still use their explicitly configured provider. A mismatched prefix is rejected. Managed catalogs use getpod-voice/<provider>/<native-model> with explicit managed:<provider> routing, source: managed, metered: true, and credit_multiplier. Missing BYOK credentials never switch payer.
POST /api/v1/agents/:agentId/voice-settings/preview requires agent write access. Body: { "provider": "upstream:gemini", "model": "gemini/gemini-3.1-flash-tts-preview", "voiceId": "Kore", "language": "en" }. Preview language defaults to English. Supported codes: ar, de, en, es, fr, hi, id, it, ja, ko, pt, ru, th, vi, zh; PaxaLabs accepts only en and th. Returns { "mime": "audio/wav", "audio": "<base64>" } for a short fixed preview, without saving settings or adding chat history. Generation uses the selected provider account.
Retained speech replay
GET /api/v1/agents/:agentId/sessions/:sessionId/voice-sessions/replays returns {response_ids: string[], replayable_response_ids: string[]}. response_ids preserves the retained-recording contract; replayable_response_ids also includes completed responses with approved spoken text, even when the original audio was interrupted.
GET /api/v1/agents/:agentId/sessions/:sessionId/voice-sessions/replays/:responseId returns the original synthesized audio as audio/wav, or 404 if missing/expired. This GET never invokes a TTS provider. Both endpoints use API authentication, Agent access and conversation membership; no bearer credential belongs in the URL. Recordings are retained for at most 30 days with a 64 MiB per-Agent budget, oldest first eviction, and a 16 MiB per-recording limit. Disabling new voice synthesis does not delete existing recordings.
POST /api/v1/agents/:agentId/sessions/:sessionId/voice-sessions/replays/:responseId returns retained WAV audio or generates and retains missing audio from the response's approved spoken text. It never falls back to arbitrary display text. Generation requires voice and orchestration enabled, session membership and managed voice credit where applicable. Missing speech returns 404; disabled synthesis returns 409; exhausted managed credit returns 429; generation failures return 503. Missing-audio synthesis uses the current configured TTS provider and can incur normal provider charges. Concurrent generation is shared for the same principal/session/response and bounded to four jobs, 60 seconds per synthesis and 16 MiB per recording. Stop live playback before starting replay and cancel obsolete browser downloads on navigation.
Agent model selection in a voice connection
Voice clients may include model in voice.start and update it during the connection with { "type": "voice.configure", "model": "model-id" }. The server acknowledges with voice.configured; subsequent utterance submissions use that model, including their new worker tasks. Existing running tasks retain their assigned model. This is the Agent model, independent of STT/TTS model selection.
Provider credentials
Upstream voice uses the same Claude provider settings as the Agent: the env block in $CLAUDE_CONFIG_DIR/settings.json (default ~/.claude/settings.json) takes precedence over inherited environment. Credentials are resolved as one group, so a settings identity cannot silently fall back to another exported token. Catalog, preview, STT and TTS resolve this connection when requested; a shell export is not required after restart. Model-catalog failures retain any available voices and report a provider diagnostic without changing saved selections.
Automatic voice selection
When TTS voiceId is empty, Auto selects a voice from the configured provider's catalog. An explicit session/chat choice takes precedence over the configured voice. Auto keeps its choice for the gateway process and provider/account; after restart it deterministically selects the lowest voice ID in the current catalog. Catalog changes can therefore change Auto after a restart. Set a voice explicitly to pin it across restarts. Catalog failures leave text chat available and report a recoverable TTS error instead of preventing gateway startup.
Managed voice wallet and startup recovery
The same reconnect protocol applies when opening the recognizer fails with a retryable provider error. Managed wallet checks honor cancellation. Each attempt has a five-second timeout, and a check that cannot finish is retried twice (after 0.5 s and 1.5 s, within 10 s in total). Only a definite empty wallet stops the request: remaining of 0 or less, HTTP 402, or code: MANAGED_VOICE_QUOTA_EXHAUSTED. A client error other than 408/429 is not retried. A check that still cannot finish lets the request through, because the provider refuses work without credit and that refusal is reported as usual. Its cause is recorded as MANAGED_VOICE_USAGE_TIMEOUT, MANAGED_VOICE_USAGE_NETWORK, MANAGED_VOICE_USAGE_HTTP_<status> or MANAGED_VOICE_USAGE_INVALID_BODY. These codes, and the older MANAGED_VOICE_USAGE_UNAVAILABLE, read as "Could not check the voice credit balance", never as an outage or exhausted credit.
Explicit managed:elevenlabs and managed:paxalabs routes use the authenticated provider service's /v1/voice/managed/usage wallet. A live voice reply is synthesized one piece of spoken text at a time and checks the wallet once, before its first piece; each new reply checks again, so an empty wallet is never remembered. Ticket creation returns 429 with code: MANAGED_VOICE_QUOTA_EXHAUSTED when that wallet is empty; if exhaustion happens after connection, the same code arrives in voice.error with retryable: false. Clients should pause voice until the wallet is replenished or reset, rather than retrying repeatedly. Channel voice-note transcription and synthesized replies report explicit managed exhaustion as a text notice; text replies remain available. The notices are fixed English strings sent by the gateway, not by the agent, and are never localized or translated, whatever the chat, agent or voice.language. A voice note gets Voice credit is used up, so this voice message could not be transcribed. Please type your message instead. Reference: <inputId>, and a voice reply is followed by Voice reply unavailable: voice credit is used up. Text replies continue until the credit resets or is topped up, or use /voice off.. Each chat gets at most one notice per kind (stt, tts) per hour, across sessions, stored in the voice_credit_notices table so restarts do not repeat it. Expired entries are pruned. A later successful managed transcription or delivered managed voice reply clears every chat's entries, since the agent's managed wallet is shared. A notice whose text delivery definitely failed releases its claim; an uncertain delivery (unknown) keeps it, so the notice is not repeated. Within that hour further voice notes end without a reply and further voice replies are dropped after their text, as before. MANAGED_VOICE_USAGE_* codes never use this wording. Saved voice preferences are unchanged, and the next request checks the current wallet again. Ordinary provider 429s and BYOK billing errors retain their normal diagnostics.
OpenRouter voice catalog
GET /api/v1/agents/:agentId/voice-settings/catalog?provider=upstream:openrouter discovers connected OpenRouter speech/transcription models and their supported voices. Direct openrouter uses OPENROUTER_API_KEY; upstream:openrouter uses the upstream account credentials. Select models by their catalog capabilities: TTS uses the speech endpoint, while browser segments and uploaded voice messages use the transcription endpoint. OpenRouter browser STT is recorded, not realtime.
Resume browser voice after navigation
A successful voice ticket saves voice intent for the authenticated principal and chat session. Closing the socket or deleting its lease releases microphone resources but preserves that intent. After reconnecting, the gateway queues approved spoken summaries of completed responses since voice was enabled, including replies completed while disconnected or after returning. It reuses canonical response IDs and the normal TTS queue; no full-report text fallback is introduced. Catch-up uses current voice settings and may incur TTS usage.
Clients must send accurate playback.progress receipts. Fully played responses and responses explicitly interrupted by barge-in are not replayed automatically. A partially heard response detached by navigation restarts from its beginning. Repeated catch-up checks and live notifications do not synthesize the same response twice within one connection. Provider failures remain visible and are not repeatedly retried on that connection.
Explicitly switching to text mode requires closing the current connection, then PUT /api/v1/agents/:agentId/sessions/:sessionId/voice-sessions/preference with {"enabled":false} (204). The endpoint requires API authentication, Agent access and conversation membership; enabling still requires the normal ticket and credit checks. Switching voice back on starts a new catch-up window. Navigation alone must not send this preference update. Browser clients should save mode and mute per authenticated pod/Agent/session, disable media tracks before connecting when muted, and wait for old lease deletion before opening a replacement.