Opal WebSocket Protocol

The Opal WebSocket Protocol uses Protocol Buffers for message encoding. All messages are wrapped in either ServiceBoundMessage (client to server) or ClientBoundMessage (server to client).

Messages are binary-encoded protobuf. JSON examples below are shown for readability. Download the proto file.

Need VAD-only without LLM inference? See the VAD WebSocket Protocol for a dedicated VAD API that streams voice activity events.

Connection

wss://app.deepslate.eu/api/v1/vendors/{vendorId}/organizations/{organizationId}/realtime

Authentication via Bearer token in the connection headers.

Client Messages

Messages sent from client to server, wrapped in ServiceBoundMessage.

InitializeSessionRequest

Must be the first message sent. Configures the session parameters.

Field	Type	Description
`input_audio_line`	AudioLineConfiguration	Input audio format configuration
`output_audio_line`	AudioLineConfiguration	Output audio format configuration
`vad_configuration`	VadConfiguration	Voice activity detection settings
`inference_configuration`	InferenceConfiguration	System prompt and model behavior
`tts_configuration`	TtsConfiguration	Optional TTS provider config (e.g., ElevenLabs)
`supports_playback_reporting`	bool	Set to `true` when the client intends to send `PlaybackPositionReport` messages. Defaults to `false`.
`enable_vad_frame_telemetry`	bool	When `true`, the server emits per-frame `VadAnalysisFrame` messages (~50 Hz). Off by default for cost and bandwidth reasons. `VadStateEvent` transitions are emitted either way. Experimental.
`experiments`	map<string, Value>	Optional experiments to enable, mapping experiment name to a parameter value whose meaning the experiment defines. No stability guarantees.

{
  "initializeSessionRequest": {
    "inputAudioLine": {
      "sampleRate": 16000,
      "channelCount": 1,
      "sampleFormat": "SIGNED_16_BIT"
    },
    "outputAudioLine": {
      "sampleRate": 16000,
      "channelCount": 1,
      "sampleFormat": "SIGNED_16_BIT"
    },
    "vadConfiguration": {
      "confidenceThreshold": 0.5,
      "minVolume": 0,
      "startDuration": { "seconds": 0, "nanos": 200000000 },
      "stopDuration": { "seconds": 0, "nanos": 500000000 },
      "backbufferDuration": { "seconds": 1, "nanos": 0 }
    },
    "inferenceConfiguration": {
      "systemPrompt": "You are a helpful assistant.",
      "temperature": 0.7
    },
    "supportsPlaybackReporting": true
  }
}

ReconfigureSessionRequest

Reconfigure an ongoing session. Useful for changing audio input settings or the system prompt on the fly. You can update either field or both.

Reconfiguration may not be seamless. There may be glitches or dropped audio during the transition.

Field	Type	Description
`input_audio_line`	AudioLineConfiguration	Updated input audio format configuration
`inference_configuration`	InferenceConfiguration	Updated system prompt and model behavior

{
  "reconfigureSessionRequest": {
    "inputAudioLine": {
      "sampleRate": 48000,
      "channelCount": 1,
      "sampleFormat": "SIGNED_16_BIT"
    },
    "inferenceConfiguration": {
      "systemPrompt": "You are now a billing specialist."
    }
  }
}

UserInput

User input data (audio or text).

Field	Type	Description
`packet_id`	uint64	Client-defined packet identifier for tracking
`mode`	InferenceTriggerMode	How to trigger inference for this input
`audio_data`	AudioData	Raw PCM audio bytes (one of audio_data or text_data)
`text_data`	TextData	Text input (one of audio_data or text_data)

Audio Input
Text Input

{
  "userInput": {
    "packetId": 1,
    "mode": "IMMEDIATE",
    "audioData": {
      "data": "<binary-pcm-data>"
    }
  }
}

{
  "userInput": {
    "packetId": 1,
    "mode": "IMMEDIATE",
    "textData": {
      "data": "What's the weather like today?"
    }
  }
}

UpdateToolDefinitionsRequest

Define or update available tools. Replaces all existing definitions.

Field	Type	Description
`tool_definitions`	ToolDefinition[]	List of tool definitions

Each ToolDefinition contains:

Field	Type	Description
`name`	string	Tool identifier used by the model
`description`	string	Purpose and functionality description
`parameters`	object	JSON Schema for tool parameters

{
  "updateToolDefinitionsRequest": {
    "toolDefinitions": [
      {
        "name": "get_weather",
        "description": "Get current weather for a location",
        "parameters": {
          "type": "object",
          "properties": {
            "location": { "type": "string" }
          },
          "required": ["location"]
        }
      }
    ]
  }
}

ToolCallResponse

Response to a tool call request from the server.

Field	Type	Description
`id`	string	Must match the `id` from `ToolCallRequest`
`result`	string	Tool execution result (any format the model understands)

Every ToolCallRequest must receive a corresponding ToolCallResponse, even if execution fails.

{
  "toolCallResponse": {
    "id": "call_abc123",
    "result": "{\"temperature\": 22, \"condition\": \"sunny\"}"
  }
}

TriggerInference

Manually trigger inference processing immediately, instead of waiting for natural pauses or end-of-input signals.

Primary use case is generating an initial greeting. Model behavior may be unpredictable if used directly after a model response.

Field	Type	Description
`extra_instructions`	string	Optional extra instructions to guide the inference
`flush_vad`	bool	When `true`, flush whatever the VAD pipeline has buffered and append it as a user audio message before triggering inference. Defaults to `false`.

Use flush_vad to commit speech the VAD has not released yet, when the trigger comes from an external signal such as a push-to-talk button or a wake word rather than from natural end-of-speech detection. It is a no-op when the buffer is empty.

{
  "triggerInference": {
    "extraInstructions": "Greet the user warmly and ask how you can help.",
    "flushVad": false
  }
}

ExportChatHistoryRequest

Request the full conversation history. The server responds with a ChatHistory message.

Field	Type	Description
`await_pending`	bool	When `true`, waits for all in-flight async operations (e.g. transcriptions) to complete before responding
`exclude_audio`	bool	When `true`, omits audio data from the response. Useful when you only need transcripts and want to avoid transferring large audio blobs. Defaults to `false`.

{
  "exportChatHistoryRequest": {
    "awaitPending": true,
    "excludeAudio": true
  }
}

PlaybackPositionReport

Reports how many audio bytes the client has played for a given turn.The server uses this to truncate the LLM context to exactly what the caller heard when they interrupt, and to pace ModelSpeechProgress. Without these reports it falls back to a server-side elapsed-time estimate.

Field	Type	Description
`bytes_played`	uint64	Number of audio bytes the client has played for `turn_id`
`turn_id`	uint32?	Which assistant turn these bytes belong to. Should always be set.

Omitting turn_id is not the same as sending 0. Turn IDs are 0-based, so a report without one is read as coming from a client written before turn correlation existed, and the server applies it to the current response instead of the turn you meant.

Set supports_playback_reporting in InitializeSessionRequest when you intend to send these reports.

{
  "playbackPositionReport": {
    "bytesPlayed": 32000,
    "turnId": 4
  }
}

DirectSpeech

Instructs the service to speak the given text via TTS immediately, bypassing the LLM. Any active inference is cancelled and the audio buffer is cleared before the text is spoken.

Field	Type	Description
`text`	string	Text to speak
`include_in_history`	bool	When `false`, the text is spoken but marked as ephemeral — the LLM won’t know it was spoken
`uninterruptable`	bool	When `true`, the utterance plays to completion and overlapping user speech is ignored until playback finishes. Defaults to `false`.

Use uninterruptable for compliance announcements, such as notifying the caller that they are speaking with an AI.

{
  "directSpeech": {
    "text": "Please hold, transferring your call.",
    "includeInHistory": false,
    "uninterruptable": false
  }
}

Ping

Liveness check. The server replies with Pong. Carries no fields.Use it to keep a session alive when nothing else is being sent, or to measure round-trip latency.

{
  "ping": {}
}

ConversationQuery

Runs a one-shot LLM inference over the current conversation history without modifying it. Useful for side tasks like summarization or classification. The server responds with a ConversationQueryResult.At least one of prompt or instructions must be provided.

Field	Type	Description
`prompt`	string	Replaces the system prompt for this one-shot call. If absent, uses the session’s current system prompt
`instructions`	string	Appended as instructions after the conversation turns. If absent, no extra instructions are appended

{
  "conversationQuery": {
    "prompt": "You are a sentiment analyzer.",
    "instructions": "This is the chat history between an assistant and a customer. Do not reply to the chat in any way; only create a brief summary of the entire chat and describe the key points as briefly as possible. 1-2 sentences without line breaks. Reply directly with the summary, no introductions like: 'Okay, here is a brief summary: ...' Example summary: 'Customer requested a callback; conversation positive.'"
  }
}

Server Messages

Messages sent from server to client, wrapped in ClientBoundMessage. Most of these messages carry a turn_id identifying the assistant turn they belong to. Turn IDs are 0-based, so 0 is a real turn rather than a missing value. ResponseBegin, ResponseEnd, ModelSpeechProgress and InferenceComplete always carry one, whereas ModelTextFragment and ModelAudioChunk carry an optional turn_id. When absent, the message belongs to the current turn. An assistant turn ends in three stages, each reported by a different message. They are not interchangeable.

Signal	Meaning
`InferenceComplete`	The LLM has finished generating. TTS may still be synthesizing.
`ResponseEnd`	All audio for the turn has been sent. Content can still change due to interruption.
`TurnSnapshot` with `is_final: true`	The turn’s content is final and will not change again.

SessionReady

Sent once the session is fully initialized and ready to accept input, after TTS warmup completes. Wait for this message before sending UserInput. Carries no fields.

{
  "sessionReady": {}
}

ModelTextFragment

Streamed text output as tokens arrive.ModelTextFragment is emitted for every assistant turn, whether or not TTS is configured, and carries text as the model generates it.When TTS is configured, these fragments run ahead of speech synthesis, so they reflect what the model has produced rather than what the caller has heard. ModelSpeechProgress reports the playback-paced view of the same text.

Field	Type	Description
`text`	string	Text content of this fragment
`turn_id`	uint32?	Optional. Session-local ID of the assistant turn this fragment belongs to. When absent, attribute the fragment to the current turn.

{
  "modelTextFragment": {
    "text": "Hello, how can I help you?",
    "turnId": 4
  }
}

ModelAudioChunk

TTS audio output when a TTS provider is configured.

Field	Type	Description
`audio`	AudioData	Audio bytes matching output_audio_line config
`turn_id`	uint32?	Optional. Session-local ID of the assistant turn this audio belongs to.

ModelAudioChunk carries audio only. For the text that goes with it, use ModelSpeechProgress.

{
  "modelAudioChunk": {
    "audio": {
      "data": "<binary-pcm-data>"
    },
    "turnId": 4
  }
}

ModelSpeechProgress

Reports how much of an assistant turn’s text has become audible. Emitted continuously while TTS audio is playing.Where ModelTextFragment gives you the text as the model produces it, ModelSpeechProgress gives you the same text paced to playback. Use it to display what the caller has actually heard.

Field	Type	Description
`turn_id`	uint32	Session-local ID of the assistant turn this progress belongs to
`text`	string	Text that became audible since the previous `ModelSpeechProgress` for this turn
`audio_bytes_played`	uint64	Audio bytes played so far, per the server’s estimate
`exact`	bool	Debug indicator of whether the segment aligns precisely with played audio. See note below.

Concatenating text in arrival order reproduces the same text as the turn’s ModelTextFragment messages, so the result is always a valid prefix of the generated text.

Treat exact as debug information only, never as a correctness gate.A segment with exact: false still contains the correct and complete text. It may just not correspond precisely to the audio played so far. The server keeps this drift as small as it can, but a small amount is unavoidable.

The server derives audio_bytes_played from its own estimate of playback. Send PlaybackPositionReport messages to replace that estimate with your client’s real position and tighten the whole stream.

{
  "modelSpeechProgress": {
    "turnId": 4,
    "text": "Hello, how can I",
    "audioBytesPlayed": 48000,
    "exact": true
  }
}

ToolCallRequest

Model requests to execute a tool.

Field	Type	Description
`id`	string	Unique identifier for this request
`name`	string	Name of the tool to call
`parameters`	object	Parameters matching the tool’s schema
`turn_id`	uint32?	Optional. Session-local ID of the assistant turn that issued this tool call.

{
  "toolCallRequest": {
    "id": "call_abc123",
    "name": "get_weather",
    "parameters": {
      "location": "Amsterdam"
    },
    "turnId": 4
  }
}

PlaybackClearBuffer

Notification to clear the audio playback buffer. Sent proactively when the user starts speaking, regardless of whether there is ongoing TTS playback.When received, immediately discard any buffered audio that hasn’t been played yet. This message may be sent multiple times if the user interrupts multiple times.

{
  "playbackClearBuffer": {}
}

ResponseBegin

Notification that the model has begun its response.

Field	Type	Description
`turn_id`	uint32	Session-local ID of the assistant turn this response begins

{
  "responseBegin": {
    "turnId": 4
  }
}

InferenceComplete

The LLM has finished producing this turn’s output.Generation is done, but the turn is not. TTS audio may still be synthesizing, and further events for the turn will still arrive.

Field	Type	Description
`turn_id`	uint32	Session-local ID of the assistant turn whose generation finished

{
  "inferenceComplete": {
    "turnId": 4
  }
}

ResponseEnd

Notification that no more audio is coming for this turn. Sent once all audio packets for the turn have been sent.

ResponseEnd does not mean the turn is finished. The turn’s content can still change afterwards, for example when an interruption shortens it.

Field	Type	Description
`turn_id`	uint32	Session-local ID of the assistant turn this response ends. Matches the `turn_id` from the preceding `ResponseBegin`.

{
  "responseEnd": {
    "turnId": 4
  }
}

TurnSnapshot

The server’s authoritative view of one turn’s content at a point in time.

Field	Type	Description
`message`	ChatMessage	The turn’s content as of this snapshot
`is_final`	bool	When `true`, the turn is finalized and no further snapshots for it will arrive

Snapshots are not guaranteed to be monotonic. A turn can produce several of them, and content can be rolled back, for example when an interruption means part of a response was never heard.

is_final means the turn’s content will not change again. It does not mean the turn stays visible forever since context truncation can still remove it from the model’s context later, which ContextTruncated reports.

{
  "turnSnapshot": {
    "message": {
      "role": "ASSISTANT",
      "turnId": 4,
      "deliveryStatus": "DELIVERY_INTERRUPTED",
      "ephemeral": false,
      "createdAt": "2026-09-01T14:23:10Z",
      "content": [
        {
          "textContent": {
            "text": "It's currently 3:45"
          }
        }
      ]
    },
    "isFinal": true
  }
}

ChatHistory

The full conversation history, returned in response to ExportChatHistoryRequest.

Field	Type	Description
`messages`	ChatMessage[]	Ordered list of conversation messages

{
  "chatHistory": {
    "messages": [
      {
        "role": "USER",
        "turnId": 3,
        "deliveryStatus": "DELIVERY_COMPLETE",
        "ephemeral": false,
        "content": [
          {
            "inputAudio": {
              "transcription": "What time is it?"
            }
          }
        ]
      },
      {
        "role": "ASSISTANT",
        "turnId": 4,
        "deliveryStatus": "DELIVERY_COMPLETE",
        "ephemeral": false,
        "content": [
          {
            "textContent": {
              "text": "It's currently 3:45 PM."
            }
          }
        ]
      }
    ]
  }
}

ContextTruncated

Messages were removed from the LLM’s context window to fit the token budget.Only turns newly truncated in this inference cycle are listed. Turns reported in an earlier ContextTruncated are not repeated.

Field	Type	Description
`truncated_turn_ids`	uint32[]	Turn IDs newly removed from the model’s context. The model can no longer see these messages.
`response_turn_id`	uint32	The assistant turn generated with the truncated context, correlating this event with that inference cycle

A truncated turn keeps its content and stays in an exported ChatHistory. It is the model’s view that shrinks, not yours. The turn’s ChatMessage.truncated_at_response_turn_id records the same relationship, so you can look up when it happened after the fact.

{
  "contextTruncated": {
    "truncatedTurnIds": [1, 2],
    "responseTurnId": 9
  }
}

VadStateEvent

Emitted on every VAD state-machine transition. Always sent, regardless of whether enable_vad_frame_telemetry is set.

Field	Type	Description
`session_time`	Duration	Time elapsed since the input pipeline started when the transition occurred
`from_state`	VadState	The state before the transition
`to_state`	VadState	The state after the transition
`packet_id`	uint64	The `packet_id` of the `UserInput` packet that triggered the transition

{
  "vadStateEvent": {
    "sessionTime": { "seconds": 2, "nanos": 340000000 },
    "fromState": "SILENCE",
    "toState": "SPEECH_STARTING",
    "packetId": 42
  }
}

VadAnalysisFrame

Per-frame VAD telemetry, emitted at ~50 Hz (20 ms frames on 16 kHz audio). Only sent when enable_vad_frame_telemetry: true was set in InitializeSessionRequest.

This message type is experimental and may be changed or removed without a major version bump. It is intended for debugging and telemetry. Do not rely on it for critical functionality.

Field	Type	Description
`frame_index`	uint64	Monotonic frame index for this session
`session_time`	Duration	Wall-clock time elapsed since the input pipeline started, at the end of this frame
`confidence`	float	Raw VAD confidence score from the underlying engine (0.0–1.0)
`volume`	float	Raw RMS volume of the audio frame (0.0–1.0)
`state`	VadState	The state-machine state at the end of this frame
`source_packet_ids`	uint64[]	`packet_id`s whose audio contributed to this frame. Usually one; multiple at packet boundaries or with very small packets.

{
  "vadAnalysisFrame": {
    "frameIndex": 117,
    "sessionTime": { "seconds": 2, "nanos": 340000000 },
    "confidence": 0.82,
    "volume": 0.31,
    "state": "SPEECH",
    "sourcePacketIds": [42]
  }
}

Pong

Reply to a client Ping. Carries no fields.

{
  "pong": {}
}

SessionErrorNotification

Structured error notification sent before the server closes the connection.

Field	Type	Description
`category`	SessionErrorCategory	Error category for programmatic handling
`message`	string	Human-readable message for logging or display
`trace_id`	string	Optional trace ID for correlating with server logs

{
  "error": {
    "category": "ERROR_CONFIGURATION",
    "message": "Invalid sample rate: must be between 8000 and 48000",
    "traceId": "abc-123-xyz"
  }
}

UserTranscriptionResult

Async transcription result for a completed user audio turn. Sent after the transcription worker finishes processing.

Field	Type	Description
`turn_id`	uint32	Identifies which conversation turn this transcription belongs to
`text`	string	Transcribed text
`language`	string	Detected language (ISO 639-1 code, e.g. `"en"`)

{
  "userTranscriptionResult": {
    "turnId": 3,
    "text": "What time is it?",
    "language": "en"
  }
}

ConversationQueryResult

Result of a ConversationQuery request.

Field	Type	Description
`text`	string	The LLM’s complete response text

{
  "conversationQueryResult": {
    "text": "positive"
  }
}

Type Definitions

AudioLineConfiguration

Field	Type	Description
`sample_rate`	uint32	Sample rate in Hz (e.g., 16000)
`channel_count`	uint32	Number of channels (typically 1 for mono)
`sample_format`	SampleFormat	Audio sample format

SampleFormat

Value	Description
`UNSIGNED_8_BIT`	8-bit unsigned integer samples
`SIGNED_16_BIT`	16-bit signed integer samples (recommended)
`SIGNED_32_BIT`	32-bit signed integer samples
`FLOAT_32_BIT`	32-bit floating point (0.0 to 1.0)
`FLOAT_64_BIT`	64-bit floating point (0.0 to 1.0)

VadConfiguration

Voice Activity Detection settings.

Field	Type	Description
`confidence_threshold`	float	Min confidence for speech detection (0.0-1.0)
`min_volume`	float	Min volume level for speech (0.0-1.0)
`start_duration`	Duration	Speech duration to trigger start
`stop_duration`	Duration	Silence duration to trigger end
`backbuffer_duration`	Duration	Audio buffer before speech start (recommended: 1s)

InferenceConfiguration

Field	Type	Description
`system_prompt`	string	System prompt to guide model behavior
`temperature`	double	Controls output randomness. Higher values produce more random output, lower values more deterministic output.

TtsConfiguration

Optional text-to-speech configuration. If omitted, raw text fragments are sent.

ElevenLabs
Hosted

Field	Type	Description
`api_key`	string	Your ElevenLabs API key
`voice_id`	string	Voice ID (e.g., “21m00Tcm4TlvDq8ikWAM”)
`model_id`	string	Optional model ID (e.g., “eleven_turbo_v2”)
`voice_settings`	ElevenLabsVoiceSettings	Optional voice fine-tuning settings
`location`	ElevenLabsLocation	Service location for data residency (default: `US`)

{
  "ttsConfiguration": {
    "elevenLabs": {
      "apiKey": "sk-...",
      "voiceId": "21m00Tcm4TlvDq8ikWAM",
      "modelId": "eleven_turbo_v2",
      "voiceSettings": {
        "stability": 0.5,
        "similarityBoost": 0.75,
        "style": 0.0,
        "useSpeakerBoost": true,
        "speed": 1.0
      },
      "location": "EU"
    }
  }
}

Uses Deepslate-hosted TTS. You can reference a hosted voice by ID, or provide an inline voice clone from reference audio and exact reference text for this realtime session.

Field	Type	Description
`voice_ref`	HostedVoiceRef	Reference to a hosted voice by ID. Mutually exclusive with `voice_clone_v1`.
`voice_clone_v1`	HostedVoiceCloneV1	Inline custom voice clone using reference audio plus exact transcript. Mutually exclusive with `voice_ref`.
`mode`	HostedTtsMode	Quality/latency trade-off mode (default: `HIGH_QUALITY`)

Hosted voice ID

{
  "ttsConfiguration": {
    "hosted": {
      "voiceRef": {
        "voiceId": "c3dfa73f-a1ab-4aad-b48a-0e9b9fe4a69f"
      },
      "mode": "HIGH_QUALITY"
    }
  }
}

Reference audio + text

{
  "ttsConfiguration": {
    "hosted": {
      "voiceCloneV1": {
        "audioData": "<base64-encoded-reference-audio>",
        "audioFormat": {
          "sampleRate": 16000,
          "channelCount": 1,
          "sampleFormat": "SIGNED_16_BIT"
        },
        "refText": "This is the exact transcript of the reference audio, including disfluencies."
      },
      "mode": "HIGH_QUALITY"
    }
  }
}

Use 20 to 25 seconds of clean speech for voice_clone_v1.audio_data. In JSON, protobuf bytes fields are base64-encoded; binary protobuf clients send the raw audio bytes.

ElevenLabsVoiceSettings

Fine-tuning settings for ElevenLabs voices.

Field	Type	Description
`stability`	double	Stability for the voice (0.0-1.0)
`similarity_boost`	double	Similarity boost for the voice (0.0-1.0)
`style`	double	Style setting for v2 models (0.0-1.0)
`use_speaker_boost`	bool	Whether to apply speaker boost
`speed`	double	Speed setting for the voice

ElevenLabsLocation

Controls which ElevenLabs regional endpoint is used. See ElevenLabs data residency docs for details.

Value	Description
`US`	United States (default) — accessed via https://elevenlabs.io/
`EU`	European Union — requires ElevenLabs enterprise access
`INDIA`	India — requires ElevenLabs enterprise access

HostedVoiceRef

Reference a Deepslate-hosted voice that is already available on the server.

Field	Type	Description
`voice_id`	string	ID of the hosted voice to synthesize with

HostedVoiceCloneV1

Provide an inline custom voice clone for hosted TTS using reference audio and its exact transcript.

Field	Type	Description
`audio_data`	bytes	Raw reference audio bytes. Use 20 to 25 seconds of clean speech.
`audio_format`	AudioLineConfiguration	Format of `audio_data`, including sample rate, channel count, and sample format
`ref_text`	string	Exact transcript of the reference audio, including disfluencies and false starts

HostedTtsMode

Controls the quality/latency trade-off for hosted TTS generation.

Value	Description
`HIGH_QUALITY`	Default. Higher quality output with still relatively low latency. Recommended for most use cases.
`LOW_LATENCY`	Fastest generation mode that takes next to no time to complete. Output quality may be significantly reduced.

VadState

State of the VAD debounce machine.

Value	Description
`SILENCE`	No speech detected
`SPEECH_STARTING`	Above-threshold frame seen; waiting for `start_duration` to confirm onset
`SPEECH`	Active speech confirmed
`SPEECH_ENDING`	Below-threshold frame seen; waiting for `stop_duration` to confirm offset

For the full state machine, including the debounce windows and the threshold rule, see VAD.

Duration

Field	Type	Description
`seconds`	uint64	Whole seconds
`nanos`	uint32	Nanoseconds component

// 500 milliseconds
{ "seconds": 0, "nanos": 500000000 }

// 1.5 seconds
{ "seconds": 1, "nanos": 500000000 }

InferenceTriggerMode

Controls how this input interacts with ongoing inference.

Value	Description
`NO_TRIGGER`	Don’t trigger inference from this input. Audio is buffered for VAD processing but won’t start inference on its own.
`QUEUE`	Queue inference to start after current inference completes (or immediately if idle).
`IMMEDIATE`	Interrupt any ongoing inference and start processing new input immediately. Recommended for streaming audio.

TextData

Text input wrapper.

Field	Type	Description
`data`	string	Raw text data

ChatMessage

A single message in the conversation history.

Field	Type	Description
`role`	ChatMessageRole	Role of the entity this message is attributed to
`content`	ChatMessageContent[]	Ordered content blocks of this message
`delivery_status`	ChatDeliveryStatus	Delivery status of this message
`ephemeral`	bool	`true` when the message was spoken via `DirectSpeech` with `include_in_history: false` — audible to the user but not in the LLM’s context
`created_at`	Timestamp	When the turn was created
`turn_id`	uint32?	Session-local ID of the assistant turn this message belongs to
`truncated_at_response_turn_id`	uint32?	If this turn was dropped from the LLM’s context window, the turn ID of the assistant response first generated without it. Absent while the turn is still in context.

ChatMessageRole

Value	Description
`SYSTEM`	System message (usually the system prompt)
`USER`	User message
`ASSISTANT`	Assistant message

ChatDeliveryStatus

Value	Description
`DELIVERY_IN_PROGRESS`	Turn is still being generated
`DELIVERY_COMPLETE`	All content was delivered to the client
`DELIVERY_INTERRUPTED`	User interrupted — content reflects what was actually delivered

ChatMessageContent

A single content block within a chat message. Contains one of:

Field	Type	Description
`text_content`	ChatTextContent	Text content, optionally with TTS-synthesized audio
`input_audio`	ChatAudioData	User input or model-output audio (not TTS-synthesized)
`thoughts`	string	Internal model reasoning / chain-of-thought
`tool_call`	ToolCallRequest	Tool call requested by the model
`tool_result`	ToolCallResponse	Tool execution result
`instructions`	string	Model instructions (e.g. directives injected via `TriggerInference`)

ChatTextContent

Text content from a conversation turn, with optional TTS audio. When TTS is active, each synthesized sentence becomes a ChatTextContent with both fields populated.

Field	Type	Description
`text`	string	The text content
`tts_audio`	ChatAudioData	TTS-synthesized audio for this text, if available

ChatAudioData

Self-describing audio data including format metadata so consumers can decode without out-of-band knowledge.

If you reconfigure the audio pipeline mid-conversation, the format may change. Always inspect the format field rather than assuming it matches the initial configuration.

Field	Type	Description
`audio`	AudioData	Raw audio bytes
`format`	AudioLineConfiguration	Audio format (sample rate, channels, sample format)
`transcription`	string	Transcription of the audio content. Populated asynchronously for user audio turns.

SessionErrorCategory

Broad error categories for programmatic handling of SessionErrorNotification.

Value	Description
`ERROR_UNKNOWN`	Unknown or unclassified error
`ERROR_SESSION`	Session lifecycle errors (not initialized, already initialized)
`ERROR_CONFIGURATION`	Configuration errors (invalid audio format, missing required fields)
`ERROR_PROTOCOL`	Protocol errors (malformed packets, unexpected message types)
`ERROR_INFERENCE`	Inference/AI processing errors (model unavailable, processing failed, timeout)
`ERROR_AUDIO`	Audio pipeline errors (codec failure, VAD errors)
`ERROR_TTS`	TTS synthesis errors
`ERROR_INTERNAL`	Internal service errors (catch-all for server-side issues)

Overview

Realtime

VAD

Call Event Hooks

Tool Webhooks

Assistants

Agents

Agent Tasks

Organizations

SIP Credentials

SIP Phone Numbers

LLM Generation Strategies

Opal WebSocket Protocol

Connection

Client Messages

Server Messages

Type Definitions

AudioLineConfiguration

SampleFormat

VadConfiguration

InferenceConfiguration

TtsConfiguration

ElevenLabsVoiceSettings

ElevenLabsLocation

HostedVoiceRef

HostedVoiceCloneV1

HostedTtsMode

VadState

Duration

InferenceTriggerMode

TextData

ChatMessage

ChatMessageRole

ChatDeliveryStatus

ChatMessageContent

ChatTextContent

ChatAudioData

SessionErrorCategory

​Connection

​Client Messages

​Server Messages

​Type Definitions

​AudioLineConfiguration

​SampleFormat

​VadConfiguration

​InferenceConfiguration

​TtsConfiguration

​ElevenLabsVoiceSettings

​ElevenLabsLocation

​HostedVoiceRef

​HostedVoiceCloneV1

​HostedTtsMode

​VadState

​Duration

​InferenceTriggerMode

​TextData

​ChatMessage

​ChatMessageRole

​ChatDeliveryStatus

​ChatMessageContent

​ChatTextContent

​ChatAudioData

​SessionErrorCategory

Connection

Client Messages

Server Messages

Type Definitions

AudioLineConfiguration

SampleFormat

VadConfiguration

InferenceConfiguration

TtsConfiguration

ElevenLabsVoiceSettings

ElevenLabsLocation

HostedVoiceRef

HostedVoiceCloneV1

HostedTtsMode

VadState

Duration

InferenceTriggerMode

TextData

ChatMessage

ChatMessageRole

ChatDeliveryStatus

ChatMessageContent

ChatTextContent

ChatAudioData

SessionErrorCategory