Opal WebSocket Protocol

The Opal WebSocket Protocol uses Protocol Buffers for message encoding. All messages are wrapped in either ServiceBoundMessage (client to server) or ClientBoundMessage (server to client).

Messages are binary-encoded protobuf. JSON examples below are shown for readability. Download the proto file.

Need VAD-only without LLM inference? See the VAD WebSocket Protocol for a dedicated VAD API that streams voice activity events.

Connection

wss://app.deepslate.eu/api/v1/vendors/{vendorId}/organizations/{organizationId}/realtime

Authentication via Bearer token in the connection headers.

Client Messages

Messages sent from client to server, wrapped in ServiceBoundMessage.

InitializeSessionRequest

Must be the first message sent. Configures the session parameters.

Field	Type	Description
`input_audio_line`	AudioLineConfiguration	Input audio format configuration
`output_audio_line`	AudioLineConfiguration	Output audio format configuration
`vad_configuration`	VadConfiguration	Voice activity detection settings
`inference_configuration`	InferenceConfiguration	System prompt and model behavior
`tts_configuration`	TtsConfiguration	Optional TTS provider config (e.g., ElevenLabs)
`supports_playback_reporting`	bool	Whether the client will send `PlaybackPositionReport` messages. When `true`, enables accurate context truncation on user interrupt. Defaults to `false`.

{
  "initializeSessionRequest": {
    "inputAudioLine": {
      "sampleRate": 16000,
      "channelCount": 1,
      "sampleFormat": "SIGNED_16_BIT"
    },
    "outputAudioLine": {
      "sampleRate": 16000,
      "channelCount": 1,
      "sampleFormat": "SIGNED_16_BIT"
    },
    "vadConfiguration": {
      "confidenceThreshold": 0.5,
      "minVolume": 0,
      "startDuration": { "seconds": 0, "nanos": 200000000 },
      "stopDuration": { "seconds": 0, "nanos": 500000000 },
      "backbufferDuration": { "seconds": 1, "nanos": 0 }
    },
    "inferenceConfiguration": {
      "systemPrompt": "You are a helpful assistant.",
      "temperature": 0.7
    },
    "supportsPlaybackReporting": true
  }
}

ReconfigureSessionRequest

Reconfigure an ongoing session. Useful for changing audio input settings or the system prompt on the fly. You can update either field or both.

Reconfiguration may not be seamless. There may be glitches or dropped audio during the transition.

Field	Type	Description
`input_audio_line`	AudioLineConfiguration	Updated input audio format configuration
`inference_configuration`	InferenceConfiguration	Updated system prompt and model behavior

{
  "reconfigureSessionRequest": {
    "inputAudioLine": {
      "sampleRate": 48000,
      "channelCount": 1,
      "sampleFormat": "SIGNED_16_BIT"
    },
    "inferenceConfiguration": {
      "systemPrompt": "You are now a billing specialist."
    }
  }
}

UserInput

User input data (audio or text).

Field	Type	Description
`packet_id`	uint64	Client-defined packet identifier for tracking
`mode`	InferenceTriggerMode	How to trigger inference for this input
`audio_data`	AudioData	Raw PCM audio bytes (one of audio_data or text_data)
`text_data`	TextData	Text input (one of audio_data or text_data)

Audio Input
Text Input

{
  "userInput": {
    "packetId": 1,
    "mode": "IMMEDIATE",
    "audioData": {
      "data": "<binary-pcm-data>"
    }
  }
}

{
  "userInput": {
    "packetId": 1,
    "mode": "IMMEDIATE",
    "textData": {
      "data": "What's the weather like today?"
    }
  }
}

UpdateToolDefinitionsRequest

Define or update available tools. Replaces all existing definitions.

Field	Type	Description
`tool_definitions`	ToolDefinition[]	List of tool definitions

Each ToolDefinition contains:

Field	Type	Description
`name`	string	Tool identifier used by the model
`description`	string	Purpose and functionality description
`parameters`	object	JSON Schema for tool parameters

{
  "updateToolDefinitionsRequest": {
    "toolDefinitions": [
      {
        "name": "get_weather",
        "description": "Get current weather for a location",
        "parameters": {
          "type": "object",
          "properties": {
            "location": { "type": "string" }
          },
          "required": ["location"]
        }
      }
    ]
  }
}

ToolCallResponse

Response to a tool call request from the server.

Field	Type	Description
`id`	string	Must match the `id` from `ToolCallRequest`
`result`	string	Tool execution result (any format the model understands)

Every ToolCallRequest must receive a corresponding ToolCallResponse, even if execution fails.

{
  "toolCallResponse": {
    "id": "call_abc123",
    "result": "{\"temperature\": 22, \"condition\": \"sunny\"}"
  }
}

TriggerInference

Manually trigger inference processing immediately, instead of waiting for natural pauses or end-of-input signals.

Primary use case is generating an initial greeting. Model behavior may be unpredictable if used directly after a model response.

Field	Type	Description
`extra_instructions`	string	Optional extra instructions to guide the inference

{
  "triggerInference": {
    "extraInstructions": "Greet the user warmly and ask how you can help."
  }
}

ExportChatHistoryRequest

Request the full conversation history. The server responds with a ChatHistory message.

Field	Type	Description
`await_pending`	bool	When `true`, waits for all in-flight async operations (e.g. transcriptions) to complete before responding
`exclude_audio`	bool	When `true`, omits audio data from the response. Useful when you only need transcripts and want to avoid transferring large audio blobs. Defaults to `false`.

{
  "exportChatHistoryRequest": {
    "awaitPending": true,
    "excludeAudio": true
  }
}

PlaybackPositionReport

Reports how many audio bytes the client has played. Only sent when the client declares supports_playback_reporting: true in InitializeSessionRequest.The server uses this data to accurately truncate the LLM context to exactly what the user heard when they interrupt. Without this, the server falls back to elapsed-time estimation.

Field	Type	Description
`bytes_played`	uint64	Cumulative number of audio bytes played by the client

{
  "playbackPositionReport": {
    "bytesPlayed": 32000
  }
}

DirectSpeech

Instructs the service to speak the given text via TTS immediately, bypassing the LLM. Any active inference is cancelled and the audio buffer is cleared before the text is spoken.

Field	Type	Description
`text`	string	Text to speak
`include_in_history`	bool	When `false`, the text is spoken but marked as ephemeral — the LLM won’t know it was spoken

{
  "directSpeech": {
    "text": "Please hold, transferring your call.",
    "includeInHistory": false
  }
}

ConversationQuery

Runs a one-shot LLM inference over the current conversation history without modifying it. Useful for side tasks like summarization or classification. The server responds with a ConversationQueryResult.At least one of prompt or instructions must be provided.

Field	Type	Description
`prompt`	string	Replaces the system prompt for this one-shot call. If absent, uses the session’s current system prompt
`instructions`	string	Appended as instructions after the conversation turns. If absent, no extra instructions are appended

{
  "conversationQuery": {
    "prompt": "You are a sentiment analyzer.",
    "instructions": "This is the chat history between an assistant and a customer. Do not reply to the chat in any way; only create a brief summary of the entire chat and describe the key points as briefly as possible. 1-2 sentences without line breaks. Reply directly with the summary, no introductions like: 'Okay, here is a brief summary: ...' Example summary: 'Customer requested a callback; conversation positive.'"
  }
}

Server Messages

Messages sent from server to client, wrapped in ClientBoundMessage.

ModelTextFragment

Streamed text output as tokens arrive.

ModelTextFragment is only sent in text-only mode - when no tts_configuration is present in InitializeSessionRequest. If TTS is configured, the server sends ModelAudioChunk messages instead and ModelTextFragment is never emitted.

Field	Type	Description
`text`	string	Text content of this fragment

{
  "modelTextFragment": {
    "text": "Hello, how can I help you?"
  }
}

ModelAudioChunk

TTS audio output when a TTS provider is configured.

Field	Type	Description
`audio`	AudioData	Audio bytes matching output_audio_line config
`transcript`	string?	Optional. When present, may be empty or contain the spoken text. See note below.

The transcript field is optional — it may be absent from a given chunk entirely.

Deepslate hosted TTS does not populate transcript.
Third-party TTS providers (e.g., ElevenLabs) may populate it to supply word-level alignment data alongside the audio.

{
  "modelAudioChunk": {
    "audio": {
      "data": "<binary-pcm-data>"
    },
    "transcript": "Hello, how can I help you?"
  }
}

ToolCallRequest

Model requests to execute a tool.

Field	Type	Description
`id`	string	Unique identifier for this request
`name`	string	Name of the tool to call
`parameters`	object	Parameters matching the tool’s schema

{
  "toolCallRequest": {
    "id": "call_abc123",
    "name": "get_weather",
    "parameters": {
      "location": "Amsterdam"
    }
  }
}

PlaybackClearBuffer

Notification to clear the audio playback buffer. Sent proactively when the user starts speaking, regardless of whether there is ongoing TTS playback.When received, immediately discard any buffered audio that hasn’t been played yet. This message may be sent multiple times if the user interrupts multiple times.

{
  "playbackClearBuffer": {}
}

ResponseBegin

Notification that the model has begun its response.

{
  "responseBegin": {}
}

ResponseEnd

Notification that the model has finished its response.

{
  "responseEnd": {}
}

ChatHistory

The full conversation history, returned in response to ExportChatHistoryRequest.

Field	Type	Description
`messages`	ChatMessage[]	Ordered list of conversation messages

{
  "chatHistory": {
    "messages": [
      {
        "role": "user",
        "delivery_status": "DELIVERY_COMPLETE",
        "ephemeral": false,
        "content": [
          {
            "type": "input_audio",
            "transcription": "What time is it?"
          }
        ]
      },
      {
        "role": "assistant",
        "delivery_status": "DELIVERY_COMPLETE",
        "ephemeral": false,
        "content": [
          {
            "type": "text",
            "text": "It's currently 3:45 PM.",
            "tts_audio": null
          }
        ]
      }
    ]
  }
}

SessionErrorNotification

Structured error notification sent before the server closes the connection.

Field	Type	Description
`category`	SessionErrorCategory	Error category for programmatic handling
`message`	string	Human-readable message for logging or display
`trace_id`	string	Optional trace ID for correlating with server logs

{
  "error": {
    "category": "ERROR_CONFIGURATION",
    "message": "Invalid sample rate: must be between 8000 and 48000",
    "traceId": "abc-123-xyz"
  }
}

UserTranscriptionResult

Async transcription result for a completed user audio turn. Sent after the transcription worker finishes processing.

Field	Type	Description
`turn_id`	uint32	Identifies which conversation turn this transcription belongs to
`text`	string	Transcribed text
`language`	string	Detected language (ISO 639-1 code, e.g. `"en"`)

{
  "userTranscriptionResult": {
    "turnId": 3,
    "text": "What time is it?",
    "language": "en"
  }
}

ConversationQueryResult

Result of a ConversationQuery request.

Field	Type	Description
`text`	string	The LLM’s complete response text

{
  "conversationQueryResult": {
    "text": "positive"
  }
}

Type Definitions

AudioLineConfiguration

Field	Type	Description
`sample_rate`	uint32	Sample rate in Hz (e.g., 16000)
`channel_count`	uint32	Number of channels (typically 1 for mono)
`sample_format`	SampleFormat	Audio sample format

SampleFormat

Value	Description
`UNSIGNED_8_BIT`	8-bit unsigned integer samples
`SIGNED_16_BIT`	16-bit signed integer samples (recommended)
`SIGNED_32_BIT`	32-bit signed integer samples
`FLOAT_32_BIT`	32-bit floating point (0.0 to 1.0)
`FLOAT_64_BIT`	64-bit floating point (0.0 to 1.0)

VadConfiguration

Voice Activity Detection settings.

Field	Type	Description
`confidence_threshold`	float	Min confidence for speech detection (0.0-1.0)
`min_volume`	float	Min volume level for speech (0.0-1.0)
`start_duration`	Duration	Speech duration to trigger start
`stop_duration`	Duration	Silence duration to trigger end
`backbuffer_duration`	Duration	Audio buffer before speech start (recommended: 1s)

InferenceConfiguration

Field	Type	Description
`system_prompt`	string	System prompt to guide model behavior
`temperature`	double	Controls output randomness. Higher values produce more random output, lower values more deterministic output.

TtsConfiguration

Optional text-to-speech configuration. If omitted, raw text fragments are sent.

ElevenLabs
Hosted

Field	Type	Description
`api_key`	string	Your ElevenLabs API key
`voice_id`	string	Voice ID (e.g., “21m00Tcm4TlvDq8ikWAM”)
`model_id`	string	Optional model ID (e.g., “eleven_turbo_v2”)
`voice_settings`	ElevenLabsVoiceSettings	Optional voice fine-tuning settings
`location`	ElevenLabsLocation	Service location for data residency (default: `US`)

{
  "ttsConfiguration": {
    "elevenLabs": {
      "apiKey": "sk-...",
      "voiceId": "21m00Tcm4TlvDq8ikWAM",
      "modelId": "eleven_turbo_v2",
      "voiceSettings": {
        "stability": 0.5,
        "similarityBoost": 0.75,
        "style": 0.0,
        "useSpeakerBoost": true,
        "speed": 1.0
      },
      "location": "EU"
    }
  }
}

Uses Deepslate-hosted TTS. You can reference a hosted voice by ID, or provide an inline voice clone from reference audio and exact reference text for this realtime session.

Field	Type	Description
`voice_ref`	HostedVoiceRef	Reference to a hosted voice by ID. Mutually exclusive with `voice_clone_v1`.
`voice_clone_v1`	HostedVoiceCloneV1	Inline custom voice clone using reference audio plus exact transcript. Mutually exclusive with `voice_ref`.
`mode`	HostedTtsMode	Quality/latency trade-off mode (default: `HIGH_QUALITY`)

Hosted voice ID

{
  "ttsConfiguration": {
    "hosted": {
      "voiceRef": {
        "voiceId": "c3dfa73f-a1ab-4aad-b48a-0e9b9fe4a69f"
      },
      "mode": "HIGH_QUALITY"
    }
  }
}

Reference audio + text

{
  "ttsConfiguration": {
    "hosted": {
      "voiceCloneV1": {
        "audioData": "<base64-encoded-reference-audio>",
        "audioFormat": {
          "sampleRate": 16000,
          "channelCount": 1,
          "sampleFormat": "SIGNED_16_BIT"
        },
        "refText": "This is the exact transcript of the reference audio, including disfluencies."
      },
      "mode": "HIGH_QUALITY"
    }
  }
}

Use 20 to 25 seconds of clean speech for voice_clone_v1.audio_data. In JSON, protobuf bytes fields are base64-encoded; binary protobuf clients send the raw audio bytes.

ElevenLabsVoiceSettings

Fine-tuning settings for ElevenLabs voices.

Field	Type	Description
`stability`	double	Stability for the voice (0.0-1.0)
`similarity_boost`	double	Similarity boost for the voice (0.0-1.0)
`style`	double	Style setting for v2 models (0.0-1.0)
`use_speaker_boost`	bool	Whether to apply speaker boost
`speed`	double	Speed setting for the voice

ElevenLabsLocation

Controls which ElevenLabs regional endpoint is used. See ElevenLabs data residency docs for details.

Value	Description
`US`	United States (default) — accessed via https://elevenlabs.io/
`EU`	European Union — requires ElevenLabs enterprise access
`INDIA`	India — requires ElevenLabs enterprise access

HostedVoiceRef

Reference a Deepslate-hosted voice that is already available on the server.

Field	Type	Description
`voice_id`	string	ID of the hosted voice to synthesize with

HostedVoiceCloneV1

Provide an inline custom voice clone for hosted TTS using reference audio and its exact transcript.

Field	Type	Description
`audio_data`	bytes	Raw reference audio bytes. Use 20 to 25 seconds of clean speech.
`audio_format`	AudioLineConfiguration	Format of `audio_data`, including sample rate, channel count, and sample format
`ref_text`	string	Exact transcript of the reference audio, including disfluencies and false starts

HostedTtsMode

Controls the quality/latency trade-off for hosted TTS generation.

Value	Description
`HIGH_QUALITY`	Default. Higher quality output with still relatively low latency. Recommended for most use cases.
`LOW_LATENCY`	Fastest generation mode that takes next to no time to complete. Output quality may be significantly reduced.

Duration

Field	Type	Description
`seconds`	uint64	Whole seconds
`nanos`	uint32	Nanoseconds component

// 500 milliseconds
{ "seconds": 0, "nanos": 500000000 }

// 1.5 seconds
{ "seconds": 1, "nanos": 500000000 }

InferenceTriggerMode

Controls how this input interacts with ongoing inference.

Value	Description
`NO_TRIGGER`	Don’t trigger inference from this input. Audio is buffered for VAD processing but won’t start inference on its own.
`QUEUE`	Queue inference to start after current inference completes (or immediately if idle).
`IMMEDIATE`	Interrupt any ongoing inference and start processing new input immediately. Recommended for streaming audio.

TextData

Text input wrapper.

Field	Type	Description
`data`	string	Raw text data

ChatMessage

A single message in the conversation history.

Field	Type	Description
`role`	ChatMessageRole	Role of the entity this message is attributed to
`content`	ChatMessageContent[]	Ordered content blocks of this message
`delivery_status`	ChatDeliveryStatus	Delivery status of this message
`ephemeral`	bool	`true` when the message was spoken via `DirectSpeech` with `include_in_history: false` — audible to the user but not in the LLM’s context

ChatMessageRole

Value	Description
`SYSTEM`	System message (usually the system prompt)
`USER`	User message
`ASSISTANT`	Assistant message

ChatDeliveryStatus

Value	Description
`DELIVERY_IN_PROGRESS`	Turn is still being generated
`DELIVERY_COMPLETE`	All content was delivered to the client
`DELIVERY_INTERRUPTED`	User interrupted — content reflects what was actually delivered

ChatMessageContent

A single content block within a chat message. Contains one of:

Field	Type	Description
`text_content`	ChatTextContent	Text content, optionally with TTS-synthesized audio
`input_audio`	ChatAudioData	User input or model-output audio (not TTS-synthesized)
`thoughts`	string	Internal model reasoning / chain-of-thought
`tool_call`	ToolCallRequest	Tool call requested by the model
`tool_result`	ToolCallResponse	Tool execution result
`instructions`	string	Model instructions (e.g. directives injected via `TriggerInference`)

ChatTextContent

Text content from a conversation turn, with optional TTS audio. When TTS is active, each synthesized sentence becomes a ChatTextContent with both fields populated.

Field	Type	Description
`text`	string	The text content
`tts_audio`	ChatAudioData	TTS-synthesized audio for this text, if available

ChatAudioData

Self-describing audio data including format metadata so consumers can decode without out-of-band knowledge.

If you reconfigure the audio pipeline mid-conversation, the format may change. Always inspect the format field rather than assuming it matches the initial configuration.

Field	Type	Description
`audio`	AudioData	Raw audio bytes
`format`	AudioLineConfiguration	Audio format (sample rate, channels, sample format)
`transcription`	string	Transcription of the audio content. Populated asynchronously for user audio turns.

SessionErrorCategory

Broad error categories for programmatic handling of SessionErrorNotification.

Value	Description
`ERROR_UNKNOWN`	Unknown or unclassified error
`ERROR_SESSION`	Session lifecycle errors (not initialized, already initialized)
`ERROR_CONFIGURATION`	Configuration errors (invalid audio format, missing required fields)
`ERROR_PROTOCOL`	Protocol errors (malformed packets, unexpected message types)
`ERROR_INFERENCE`	Inference/AI processing errors (model unavailable, processing failed, timeout)
`ERROR_AUDIO`	Audio pipeline errors (codec failure, VAD errors)
`ERROR_TTS`	TTS synthesis errors
`ERROR_INTERNAL`	Internal service errors (catch-all for server-side issues)

Overview

Realtime

VAD

Call Event Hooks

Tool Webhooks

Assistants

Agents

Agent Tasks

Organizations

SIP Credentials

SIP Phone Numbers

LLM Generation Strategies

Opal WebSocket Protocol

Connection

Client Messages

Server Messages

Type Definitions

AudioLineConfiguration

SampleFormat

VadConfiguration

InferenceConfiguration

TtsConfiguration

ElevenLabsVoiceSettings

ElevenLabsLocation

HostedVoiceRef

HostedVoiceCloneV1

HostedTtsMode

Duration

InferenceTriggerMode

TextData

ChatMessage

ChatMessageRole

ChatDeliveryStatus

ChatMessageContent

ChatTextContent

ChatAudioData

SessionErrorCategory

​Connection

​Client Messages

​Server Messages

​Type Definitions

​AudioLineConfiguration

​SampleFormat

​VadConfiguration

​InferenceConfiguration

​TtsConfiguration

​ElevenLabsVoiceSettings

​ElevenLabsLocation

​HostedVoiceRef

​HostedVoiceCloneV1

​HostedTtsMode

​Duration

​InferenceTriggerMode

​TextData

​ChatMessage

​ChatMessageRole

​ChatDeliveryStatus

​ChatMessageContent

​ChatTextContent

​ChatAudioData

​SessionErrorCategory

Connection

Client Messages

Server Messages

Type Definitions

AudioLineConfiguration

SampleFormat

VadConfiguration

InferenceConfiguration

TtsConfiguration

ElevenLabsVoiceSettings

ElevenLabsLocation

HostedVoiceRef

HostedVoiceCloneV1

HostedTtsMode

Duration

InferenceTriggerMode

TextData

ChatMessage

ChatMessageRole

ChatDeliveryStatus

ChatMessageContent

ChatTextContent

ChatAudioData

SessionErrorCategory