{"asyncapi":"3.0.0","info":{"title":"PyAI Omni realtime API","version":"2.0.0","description":"The realtime WebSocket protocol for **Omni** voice agents (wire protocol v2). Omni is zero-state: the session is authorized by your key's org and the agent's behavior travels in a post-handshake `configure` frame, there is nothing to create first.\n\n**Frame-key asymmetry (read before you build):** every client → server control frame is keyed on `type` (`configure`, `dtmf`, `session_ending`). Every server → client JSON payload is keyed on `event` (`hello`, `session_started`, `configured`, `turn`, `flush`, `session_end`, `tool_call`). Do not mirror the server's `event` key on your outbound frames, an event-keyed `configure` is ACKed by the transparent gateway and then silently dropped by the engine (clean handshake, zero turns, no error). Source of truth: `docs/OMNI_PROTOCOL_V2.md`.\n\n**Direct-session admission:** a session without a managed Agent profile must send `configure` immediately after `hello` / `session_started`, wait for `configured`, and only then send non-silent caller audio. Audio sent before that acknowledgement may be discarded. Service readiness does not imply that an individual WebSocket session is configured; send digital silence when the media clock must run while waiting.\n","license":{"name":"Proprietary","url":"https://pyai.com"}},"servers":{"production":{"host":"api.pyai.com","pathname":"/v1/omni","protocol":"wss","description":"Canonical production Omni endpoint.","security":[{"$ref":"#/components/securitySchemes/subprotocolKey"},{"$ref":"#/components/securitySchemes/apiKeyQuery"}],"bindings":{"ws":{"bindingVersion":"0.1.0","query":{"type":"object","properties":{"format":{"type":"string","enum":["pcm16"],"default":"pcm16","description":"Audio sample format for both directions. Load-bearing."},"rate":{"type":"string","enum":["24000","16000","8000"],"default":"24000","description":"Caller-input sample rate (Hz). 24000 for browser/WebRTC, 16000 for wideband input, 8000 for telephony. Agent output is 24000 for 24000/16000 input sessions and 8000 for 8000 sessions; hello.audio_out is authoritative.\n"},"session_label":{"type":"string","description":"Optional opaque tag echoed to your kb_endpoint so you can branch per session. PyAI stores no per-agent state.\n"},"api_key":{"type":"string","description":"Server-side auth alternative to the subprotocol. Prefer the pyai-key subprotocol; do not put the key in any other query param.\n"}}}}}}},"channels":{"omni":{"address":"/v1/omni","title":"Omni session","description":"A full-duplex voice session. Every WebSocket message is binary and starts with a one-byte tag: 0x01 audio, 0x02 transcript UTF-8 text, or 0x03 control JSON.\n","messages":{"audioOut":{"$ref":"#/components/messages/ClientAudio"},"configure":{"$ref":"#/components/messages/Configure"},"dtmf":{"$ref":"#/components/messages/Dtmf"},"sessionEnding":{"$ref":"#/components/messages/SessionEnding"},"toolResult":{"$ref":"#/components/messages/ToolResult"},"audioIn":{"$ref":"#/components/messages/AgentAudio"},"hello":{"$ref":"#/components/messages/Hello"},"sessionStarted":{"$ref":"#/components/messages/SessionStarted"},"configured":{"$ref":"#/components/messages/Configured"},"turn":{"$ref":"#/components/messages/Turn"},"flush":{"$ref":"#/components/messages/Flush"},"transcript":{"$ref":"#/components/messages/Transcript"},"sessionEnd":{"$ref":"#/components/messages/SessionEnd"},"toolCall":{"$ref":"#/components/messages/ToolCall"},"toolConfirmationRequired":{"$ref":"#/components/messages/ToolConfirmationRequired"},"error":{"$ref":"#/components/messages/Error"},"callControl":{"$ref":"#/components/messages/CallControl"}}}},"operations":{"sendToAgent":{"action":"send","channel":{"$ref":"#/channels/omni"},"title":"Client → server","summary":"Microphone audio (binary PCM16) plus control frames keyed on `type`.","messages":[{"$ref":"#/channels/omni/messages/audioOut"},{"$ref":"#/channels/omni/messages/configure"},{"$ref":"#/channels/omni/messages/dtmf"},{"$ref":"#/channels/omni/messages/sessionEnding"},{"$ref":"#/channels/omni/messages/toolResult"}]},"receiveFromAgent":{"action":"receive","channel":{"$ref":"#/channels/omni"},"title":"Server → client","summary":"Agent audio, transcript text, plus control JSON keyed only on `event`.","messages":[{"$ref":"#/channels/omni/messages/audioIn"},{"$ref":"#/channels/omni/messages/hello"},{"$ref":"#/channels/omni/messages/sessionStarted"},{"$ref":"#/channels/omni/messages/configured"},{"$ref":"#/channels/omni/messages/turn"},{"$ref":"#/channels/omni/messages/flush"},{"$ref":"#/channels/omni/messages/transcript"},{"$ref":"#/channels/omni/messages/sessionEnd"},{"$ref":"#/channels/omni/messages/toolCall"},{"$ref":"#/channels/omni/messages/toolConfirmationRequired"},{"$ref":"#/channels/omni/messages/error"},{"$ref":"#/channels/omni/messages/callControl"}]}},"components":{"securitySchemes":{"subprotocolKey":{"type":"userPassword","description":"Use SDK subprotocol auth: `Sec-WebSocket-Protocol: pyai.v1, pyai-key.<credential>`. Browser apps use an ephemeral origin-locked token minted on the server, never a live API key. The edge validates the credential (and the `omni:session` scope) and swaps in the internal engine credential.\n"},"apiKeyQuery":{"type":"httpApiKey","in":"query","name":"api_key","description":"Legacy server-side alternative. Prefer SDK subprotocol auth; never put live keys in browser code or URLs."}},"messages":{"CallControl":{"name":"callControl","title":"Native call-control requests (server → client)","summary":"Binary 0x03 control frames requesting transport-owned phone actions. A request is not proof that the action completed. transfer_to_human uses a destination from trusted per-agent configuration. A browser-only client cannot perform a phone transfer; use managed telephony or the official telephony bridge with call-control credentials.\n","payload":{"oneOf":[{"type":"object","required":["event","call_id","destination"],"properties":{"event":{"const":"transfer_to_human"},"call_id":{"type":"string"},"destination":{"type":"string"}}},{"type":"object","required":["event","call_id","digits"],"properties":{"event":{"const":"send_dtmf"},"call_id":{"type":"string"},"digits":{"type":"string"}}},{"type":"object","required":["event","call_id"],"properties":{"event":{"const":"play_hold"},"call_id":{"type":"string"},"seconds":{"type":"number"}}},{"type":"object","required":["event","call_id"],"properties":{"event":{"const":"collect"},"call_id":{"type":"string"},"field":{"type":"string"},"kind":{"type":"string","enum":["speech","dtmf"]}}},{"type":"object","required":["event","call_id"],"properties":{"event":{"const":"end_call"},"call_id":{"type":"string"}}}]},"examples":[{"payload":{"event":"transfer_to_human","call_id":"example_call","destination":"+15551230000"}},{"payload":{"event":"send_dtmf","call_id":"example_call","digits":"123#"}},{"payload":{"event":"play_hold","call_id":"example_call","seconds":20}},{"payload":{"event":"collect","call_id":"example_call","field":"account_id","kind":"speech"}},{"payload":{"event":"end_call","call_id":"example_call"}}]},"ClientAudio":{"name":"clientAudio","title":"Caller audio (binary, 0x01-prefixed)","summary":"Microphone audio as binary frames in the negotiated format/rate (PCM16 LE), each prefixed with the `0x01` type tag. Send continuously; the engine handles turn detection and barge-in. The tag is MANDATORY: the engine demuxes on the first byte, so an untagged frame is dropped (clean handshake, zero transcripts, a deaf agent). The engine counts the drop internally but sends no error frame. Omni has no commit, EOF, or done frame; keep sending real-time silence during caller pauses.\n","payload":{"type":"string","format":"binary"}},"AgentAudio":{"name":"agentAudio","title":"Agent audio (binary, 0x01-prefixed)","summary":"The agent's speech as 0x01-prefixed PCM16 frames; strip the tag and play the body at hello.audio_out (24 kHz for 16/24 kHz input sessions, 8 kHz for 8 kHz sessions).\n","payload":{"type":"string","format":"binary"}},"Configure":{"name":"configure","title":"configure (client → server)","summary":"Sent once right after the upgrade; supplies the agent's behavior. Keyed on `type`.","payload":{"type":"object","required":["type"],"properties":{"type":{"type":"string","const":"configure"},"voice_id":{"type":"string","description":"Voice to speak with (stock / clone / designed id). Multilingual stock voices also accept permanent aliases es1-es3, fr1, de1, hi1-hi4, and staged Mandarin zh1. Omit for the language default; the configured ack reports the canonical stock id actually served.\n"},"endpointing_ms":{"type":"number","description":"Finite milliseconds; clamped to 50–5000. Omission inherits serving/language policy. Not total response latency."},"idle_check_in":{"type":"string","enum":["auto","patient","off"],"description":"Inline idle preset overrides the stored Agent; most roles default auto, support patient. Check effective acknowledgement."},"persona":{"type":"string","description":"System prompt / role + instructions for the brain."},"kb_endpoint":{"type":"string","format":"uri","description":"Customer-hosted URL the engine POSTs each turn for grounding (hard 300 ms timeout, fail-open)."},"kb_token":{"type":"string","description":"Bearer the engine presents to kb_endpoint."},"language":{"type":"string","enum":["en","fr","es","de","hi","zh"],"description":"Requested session language. Availability is staged; inspect the configured ack's language_active and language_fallback. Public serving is en/fr/es/hi; de and zh fall back to en unless explicitly enabled with qualified endpoints.\n"},"greeting":{"type":"string","maxLength":1000,"description":"Greeting message, first line spoken at turn 0. Optional when session_label matches a /v1/agents profile that already stores greeting.\n"},"consent_line":{"type":"string","maxLength":500,"description":"Recording disclosure spoken before recording when recordings_enabled is true on the agent profile or inline configure.\n"},"meta":{"type":"object","additionalProperties":true,"description":"Opaque customer object (≤16 keys / ≤4 KB). Echoed on post-call records and extraction webhooks; meta.external_id lifts to external_id.\n"},"tools":{"type":"array","description":"Function calling. Hosted catalog tools (by name), client-loop tools (engine emits tool_call; client replies with tool_result), or names of server tools already registered with POST /v1/tools and bound to the agent. An inline `endpoint` or `webhook_url` is rejected with event=error, code=unsupported_tool_transport; the configure is not applied and those tools are not counted.\n","items":{"type":"object","required":["name"],"properties":{"name":{"type":"string"},"description":{"type":"string"},"parameters":{"type":"object","description":"JSON Schema for arguments."},"execution":{"type":"string","enum":["hosted","server","client","engine"]}}}}},"examples":[{"type":"configure","voice_id":"stock_emma_en_gb","persona":"You are a warm, concise receptionist for Acme.","greeting":"Hi, thanks for calling Acme. How can I help?"},{"type":"configure","voice_id":"stock_emma_en_gb","persona":"You are a warm, concise receptionist for Acme.","meta":{"external_id":"twilio_CA123"}}]}},"Dtmf":{"name":"dtmf","title":"dtmf (client → server)","payload":{"type":"object","required":["type","digit"],"properties":{"type":{"type":"string","const":"dtmf"},"digit":{"type":"string"}}}},"SessionEnding":{"name":"sessionEnding","title":"session_ending (client → server)","summary":"Clean session close. Not a caller-turn commit and does not request a reply.","payload":{"type":"object","required":["type"],"properties":{"type":{"type":"string","const":"session_ending"}}}},"Hello":{"name":"hello","title":"hello (server → client)","summary":"Handshake; advertises protocol version + audio formats.","payload":{"type":"object","required":["event"],"properties":{"event":{"type":"string","const":"hello"},"protocol":{"type":"string","example":"v2"},"audio_in":{"type":"string"},"audio_out":{"type":"string"}}}},"SessionStarted":{"name":"sessionStarted","title":"session_started (server → client)","payload":{"type":"object","required":["event"],"properties":{"event":{"type":"string","const":"session_started"},"audio_in":{"type":"string"},"audio_out":{"type":"string"}}}},"Configured":{"name":"configured","title":"configured (server → client)","summary":"Ack for your configure frame; echoes the resolved voice_id. An ack does NOT mean every field was honored.","payload":{"type":"object","required":["event"],"properties":{"event":{"type":"string","const":"configured"},"voice_id":{"type":"string"},"language_active":{"type":"string","description":"Language actually served; authoritative for behavior and billing."},"language_fallback":{"type":"boolean","description":"True when the requested language was not served and the session uses English."},"audio_out":{"type":"string","description":"Effective PCM16 output format and rate, for example pcm16@24000."},"endpointing_ms":{"type":"number","minimum":50,"maximum":5000},"idle_check_in":{"type":"string","enum":["auto","patient","off"],"description":"Effective preset; omitted by older serving releases."},"idle_check_in_enabled":{"type":"boolean","description":"False when off or disabled by serving policy."},"idle_check_in_thresholds_s":{"type":"array","items":{"type":"number"},"description":"Effective cumulative silence thresholds in seconds; empty when disabled."},"conversation_controls":{"type":"object","properties":{"endpointing_ms":{"type":"boolean"},"idle_check_in":{"type":"boolean"},"barge_sensitivity":{"type":"boolean"},"custom_idle_schedule":{"type":"boolean"}},"description":"Runtime capability flags; absence means unreported."},"ignored_fields":{"type":"array","items":{"type":"string","enum":["barge_sensitivity","ack_mode"]},"description":"Deprecated controls sent inline and ignored. Not an exhaustive acknowledgement of unknown keys."}}}},"Turn":{"name":"turn","title":"turn (server → client)","summary":"Turn boundary (user/assistant speaking).","payload":{"type":"object","required":["event","role"],"properties":{"event":{"type":"string","const":"turn"},"role":{"type":"string","enum":["user","assistant"]}}}},"Flush":{"name":"flush","title":"flush (server → client)","summary":"The user interrupted; assistant audio is being cut. Stop local playback immediately.","payload":{"type":"object","required":["event"],"properties":{"event":{"type":"string","const":"flush"},"reason":{"type":"string","description":"Why queued playback is being cleared."},"cancelled_turn":{"type":"integer","minimum":1,"description":"Optional generation identifier revoked when the caller resumes before reply audio. This does not certify what the client heard; retain received audio and transcript evidence."}}}},"Transcript":{"name":"transcript","title":"transcript (server → client, 0x02-prefixed)","summary":"A non-empty plain UTF-8 text delta for the current caller turn. This payload is not JSON and carries no discriminator, role, or finality bit. Coalesce successive deltas; official SDKs normalize them for callers.\n","payload":{"type":"string","minLength":1,"maxLength":4000,"example":"Can you help me?"}},"SessionEnd":{"name":"sessionEnd","title":"session_end (server → client)","summary":"Session is closing; see the close code. 4401 bad key, 4403 scope, 4429 rate/concurrency (retry w/ backoff), 1011 transient (retry), 1000 normal.","payload":{"type":"object","required":["event"],"properties":{"event":{"type":"string","const":"session_end"}}}},"ToolCall":{"name":"toolCall","title":"tool_call (server → client)","summary":"The brain selected a tool. Run it in your client and reply with tool_result. Default per-tool budget ~5 s (load-bearing, unlike kb_endpoint grounding).\n","payload":{"type":"object","required":["event","call_id","name"],"properties":{"event":{"type":"string","const":"tool_call"},"call_id":{"type":"string"},"name":{"type":"string"},"arguments":{"type":"object"}}}},"ToolConfirmationRequired":{"name":"toolConfirmationRequired","title":"tool_confirmation_required (server → client)","summary":"A write tool is waiting for the caller to confirm. It has not run.\n","payload":{"type":"object","required":["event","name"],"properties":{"event":{"type":"string","const":"tool_confirmation_required"},"name":{"type":"string"},"reason":{"type":"string"},"call_id":{"type":"string"}}}},"Error":{"name":"error","title":"error (server → client)","summary":"Configure or session error. unsupported_tool_transport means an inline tool URL was sent; register via POST /v1/tools or use the client loop.\n","payload":{"type":"object","required":["event","code"],"properties":{"event":{"type":"string","const":"error"},"code":{"type":"string"},"message":{"type":"string"},"call_id":{"type":"string"}}}},"ToolResult":{"name":"toolResult","title":"tool_result (client → server)","summary":"Client-loop reply to a tool_call. Keyed on `type`.","payload":{"type":"object","required":["type","call_id"],"properties":{"type":{"type":"string","const":"tool_result"},"call_id":{"type":"string"},"result":{"description":"Tool output (mutually exclusive with error)."},"error":{"type":"string","description":"Error message if the tool failed."}}}}}},"x-pyai-agent-notes":{"guide":"https://pyai.com/agents/speech-calling.md","sdk":"Use pyai.omni.connect({rate, configure}) from @pyai/sdk. The SDK owns framing and subprotocol auth.","browser_auth":"Mint an ephemeral origin-locked token on your server with omni.createSession. Never ship a live key in a browser or URL.","frames":{"audio":{"tag":1,"encoding":"PCM16 little-endian"},"caller_transcript":{"tag":2,"encoding":"UTF-8 text delta, not JSON"},"control":{"tag":3,"encoding":"UTF-8 JSON","client_key":"type","server_key":"event"}},"admission":"Wait for configured before non-silent caller audio; send paced digital silence during pauses. Omni has no commit frame.","barge_in":"On flush or barge_in, clear queued playback as well as stopping the current chunk.","output_rate":"Read hello.audio_out; input and output rates can differ.","compatibility":"Lifecycle fields can be extended. Ignore unknown fields; do not infer unlisted capabilities or language availability."}}