WebSocket

The WebSocket API provides low-level access to Deepslate Realtime for server-side integrations. Use this for telephony backends, SIP gateways, or custom voice pipelines.

This interface is for server-side use only. End users should connect through WebRTC or your application’s frontend. Never expose your API key to clients.

Prerequisites

A Deepslate API key
Node.js 18+ with the ws and protobufjs packages
The proto definition file

npm install ws protobufjs

Connect

Connect to the WebSocket endpoint with your API key in the headers:

import WebSocket from 'ws';
import protobuf from 'protobufjs';

const ws = new WebSocket('wss://app.deepslate.eu/api/v1/vendors/{vendorId}/organizations/{organizationId}/realtime', {
  headers: {
    'Authorization': 'Bearer YOUR_API_KEY'
  }
});

ws.binaryType = 'arraybuffer';

ws.on('open', () => {
  console.log('Connected');
  // Initialize session immediately after connecting
});

ws.on('message', (data) => {
  // Handle incoming protobuf messages
});

ws.on('close', (code, reason) => {
  console.log(`Disconnected: ${code}`);
});

Initialize Session

The first message must be an InitializeSessionRequest to configure audio format, VAD, and model behavior:

// Load protobuf definitions
const root = await protobuf.load('realtime.proto');
const ServiceBoundMessage = root.lookupType('eu.deepslate.realtime.speeq.ServiceBoundMessage');
const ClientBoundMessage = root.lookupType('eu.deepslate.realtime.speeq.ClientBoundMessage');

// Build initialization message
const initMessage = ServiceBoundMessage.create({
  initializeSessionRequest: {
    inputAudioLine: {
      sampleRate: 16000,
      channelCount: 1,
      sampleFormat: 1  // SIGNED_16_BIT
    },
    outputAudioLine: {
      sampleRate: 16000,
      channelCount: 1,
      sampleFormat: 1  // SIGNED_16_BIT
    },
    vadConfiguration: {
      confidenceThreshold: 0.5,
      minVolume: 0,
      startDuration: { seconds: 0, nanos: 300000000 },  // 300ms
      stopDuration: { seconds: 0, nanos: 700000000 },   // 700ms
      backbufferDuration: { seconds: 1, nanos: 0 }      // 1s
    },
    inferenceConfiguration: {
      systemPrompt: 'You are a helpful assistant.',
      temperature: 0.7
    },
    supportsPlaybackReporting: true  // This client will report playback positions
  }
});

// Send as binary protobuf
const buffer = ServiceBoundMessage.encode(initMessage).finish();
ws.send(buffer);

Set supportsPlaybackReporting: true when your client will report how many audio bytes it has played. Reporting gives you accurate context truncation when the caller interrupts mid-response. See Playback Position Reporting.

See the API Reference for all configuration options including TTS providers and tool definitions.

Send Audio

Stream audio as UserInput messages. Audio must match your inputAudioLine configuration:

let packetId = 0;

function sendAudio(pcmBuffer) {
  const message = ServiceBoundMessage.create({
    userInput: {
      packetId: packetId++,
      mode: 2,  // IMMEDIATE - interrupt ongoing inference
      audioData: {
        data: pcmBuffer  // Raw PCM bytes matching your config
      }
    }
  });

  const buffer = ServiceBoundMessage.encode(message).finish();
  ws.send(buffer);
}

The mode field controls how the input interacts with ongoing inference:

Value	Name	Behavior
`0`	`NO_TRIGGER`	Send audio without triggering inference.
`1`	`QUEUE`	Queue inference to run after any current inference completes, or immediately if idle.
`2`	`IMMEDIATE`	Interrupt any ongoing inference and start a new one immediately.

You can also send text input instead of audio by using textData in place of audioData.

Audio format must exactly match your session configuration. For 16-bit signed PCM at 16kHz mono, each sample is 2 bytes, little-endian.

Handle Responses

The server sends ClientBoundMessage with one of several payload types.

Text and audio output

ModelTextFragment — emitted for every assistant turn, whether or not TTS is configured. Contains streamed text tokens as the model generates them. When TTS is configured, they arrive ahead of speech synthesis.
ModelAudioChunk — sent only when a TTS provider is configured. Contains the synthesized audio. The text that goes with it arrives on ModelSpeechProgress.
ModelSpeechProgress — emitted continuously while TTS audio plays. Reports which part of the turn’s text has become audible.

Most response messages carry a turnId identifying the assistant turn they belong to. Turn IDs are 0-based, so 0 is a real turn, not a missing value. ResponseBegin, ResponseEnd, ModelSpeechProgress and InferenceComplete always carry one. On ModelTextFragment and ModelAudioChunk it is optional. When it is absent, attribute the message to the turn opened by the most recent ResponseBegin.

// Turn opened by the most recent ResponseBegin
let currentTurnId;

ws.on('message', (data) => {
  const message = ClientBoundMessage.decode(new Uint8Array(data));

  if (message.responseBegin) {
    // Model has started responding
    currentTurnId = message.responseBegin.turnId;
    console.log(`Response started (turn ${currentTurnId})`);
  }

  if (message.modelTextFragment) {
    // Streamed text for this turn, emitted whether or not TTS is configured
    const { text, turnId } = message.modelTextFragment;
    appendTurnText(turnId ?? currentTurnId, text);
  }

  if (message.modelAudioChunk) {
    // TTS mode: received when a TTS provider is configured
    const { turnId } = message.modelAudioChunk;
    const audioData = message.modelAudioChunk.audio.data;

    // Queue audio for playback, tagged with the turn it belongs to
    playAudio(audioData, turnId ?? currentTurnId);
  }

  if (message.modelSpeechProgress) {
    // Text the caller has actually heard, paced to playback
    const { turnId, text } = message.modelSpeechProgress;
    appendSpokenText(turnId, text);
  }

  if (message.turnSnapshot) {
    // Authoritative turn content
    const { message: turn, isFinal } = message.turnSnapshot;
    storeTurn(turn.turnId, turn, isFinal);
  }

  if (message.inferenceComplete) {
    // LLM finished generating; TTS may still be synthesizing
    console.log(`Generation done (turn ${message.inferenceComplete.turnId})`);
  }

  if (message.responseEnd) {
    // No more audio for this turn; content can still change
    console.log(`Audio ended (turn ${message.responseEnd.turnId})`);
  }

  if (message.playbackClearBuffer) {
    // User started speaking - clear any buffered audio immediately
    clearAudioQueue();
  }

  if (message.userTranscriptionResult) {
    // Async transcription for a completed user audio turn
    const { turnId, text, language } = message.userTranscriptionResult;
    console.log(`Turn ${turnId} transcribed (${language}): ${text}`);
  }

  if (message.error) {
    // Structured error notification sent before the server closes the connection
    const { category, message: msg, traceId } = message.error;
    console.error(`Session error [${category}]: ${msg}`, traceId ?? '');
  }
});

Handle Interruptions

When the user starts speaking, the server sends PlaybackClearBuffer proactively to ensure any ongoing playback is stopped. This is sent regardless of whether there is currently TTS playback. You should immediately discard any queued audio that hasn’t played yet:

let audioQueue = [];

function playAudio(data, turnId) {
  audioQueue.push({ data, turnId });
  // Process queue...
}

function clearAudioQueue() {
  audioQueue = [];
  // Also stop any currently playing audio
}

Trigger Inference

Use TriggerInference to make the model respond immediately without waiting for user speech. The primary use case is generating a greeting when the session opens.

function triggerGreeting() {
  const message = ServiceBoundMessage.create({
    triggerInference: {
      extraInstructions: 'Greet the user warmly and ask how you can help.'
    }
  });
  ws.send(ServiceBoundMessage.encode(message).finish());
}

ws.on('open', () => {
  // Initialize session first, then trigger a greeting
  send({ initializeSessionRequest: { /* ... */ } });
  triggerGreeting();
});

TriggerInference is designed for generating a greeting before any user input. Using it directly after a model response may produce unpredictable results.

Reconfigure Session

Use ReconfigureSessionRequest to update the input audio format or system prompt mid-session without reconnecting. You can update either field or both.

function reconfigure({ inputAudioLine, systemPrompt } = {}) {
  const message = ServiceBoundMessage.create({
    reconfigureSessionRequest: {
      ...(inputAudioLine && { inputAudioLine }),
      ...(systemPrompt && {
        inferenceConfiguration: { systemPrompt }
      })
    }
  });
  ws.send(ServiceBoundMessage.encode(message).finish());
}

// Example: switch system prompt mid-call
reconfigure({ systemPrompt: 'You are now a billing specialist.' });

Reconfiguration is not guaranteed to be seamless. There may be brief audio glitches or dropped audio around the transition.

Direct Speech

Use DirectSpeech to speak text via TTS immediately, bypassing the LLM. Any active inference is cancelled and the audio buffer is cleared before the text is spoken.

function speak(text, includeInHistory = true, uninterruptable = false) {
  const message = ServiceBoundMessage.create({
    directSpeech: {
      text,
      includeInHistory,  // false = ephemeral, LLM won't know it was spoken
      uninterruptable    // true = plays to completion, barge-in is ignored
    }
  });
  ws.send(ServiceBoundMessage.encode(message).finish());
}

// Speak a notice the LLM shouldn't know about
speak('Please hold, transferring your call.', false);

// Speak a compliance announcement the user cannot interrupt
speak('Please note that this call is handled by an AI and may be recorded.', true, true);

When includeInHistory is false, the message is marked as ephemeral in the chat history — it is audible to the user but invisible to the LLM’s context. When uninterruptable is true, the utterance plays to completion and overlapping user speech is ignored until playback finishes. It defaults to false (interruptible). Use it for compliance announcements, such as notifying the user that they are speaking with an AI.

Conversation Query

Use ConversationQuery to run a one-shot LLM inference over the current conversation history without modifying it. The result is returned as a ConversationQueryResult. This is useful for side tasks like summarization or classification that should not affect the ongoing conversation.

function queryConversation() {
  const message = ServiceBoundMessage.create({
    conversationQuery: {
      prompt: 'You are a sentiment analyzer.',       // Replaces system prompt for this query
      instructions: 'Rate the user sentiment so far as positive, neutral, or negative. Reply with one word.'
    }
  });
  ws.send(ServiceBoundMessage.encode(message).finish());
}

// Handle the result in your message handler
if (message.conversationQueryResult) {
  console.log('Sentiment:', message.conversationQueryResult.text);
}

At least one of prompt or instructions must be provided. If prompt is absent, the session’s current system prompt is used.

Playback Position Reporting

Send PlaybackPositionReport messages regularly as audio plays. This gives the server accurate data to truncate the LLM context to exactly what the caller heard when they interrupt. Counts are per turn, not per session. Use the turnId from the ModelAudioChunk the bytes came from, falling back to the current turn when the chunk omits it, and keep a separate total for each turn. turnId on the report is itself optional, and omitting it is not the same as sending 0. Turn IDs are 0-based, so a report without one is read as coming from a client that predates turn correlation, and the server applies it to the current response instead. Always set it.

const playedByTurn = new Map(); // turnId -> bytes played for that turn

function onAudioBytesPlayed(turnId, byteCount) {
  const bytesPlayed = (playedByTurn.get(turnId) ?? 0) + byteCount;
  playedByTurn.set(turnId, bytesPlayed);

  const message = ServiceBoundMessage.create({
    playbackPositionReport: { bytesPlayed, turnId }
  });
  ws.send(ServiceBoundMessage.encode(message).finish());
}

Set supportsPlaybackReporting: true in InitializeSessionRequest when you intend to send these. Without playback reporting, the server falls back to elapsed-time estimation for context truncation, which is less precise. The same estimate drives audio_bytes_played and the pacing of ModelSpeechProgress, so reporting real positions tightens that stream too.

Track Turn Content

TurnSnapshot carries the server’s authoritative view of a single turn.

const turns = new Map();

function storeTurn(turnId, turn, isFinal) {
  turns.set(turnId, { turn, isFinal });
}

Snapshots are not guaranteed to be monotonic. A turn can produce several, and content can be rolled back.
is_final means immutable, not permanent. Once a snapshot arrives with isFinal: true, that turn’s content will not change again and no further snapshots for it will arrive. The turn can still leave the model’s context later through truncation, which ContextTruncated reports separately.

Export Chat History

Use ExportChatHistoryRequest to retrieve the full conversation history at any point. Set awaitPending: true to wait for any in-flight transcriptions to finish before the history is returned.

function exportHistory(awaitPending = false, excludeAudio = false) {
  const message = ServiceBoundMessage.create({
    exportChatHistoryRequest: { awaitPending, excludeAudio }
  });
  ws.send(ServiceBoundMessage.encode(message).finish());
}

// Handle ChatHistory in your message handler
if (message.chatHistory) {
  for (const msg of message.chatHistory.messages) {
    console.log(msg.role, msg.content, msg.deliveryStatus);
  }
}

Each ChatMessage includes a role (SYSTEM, USER, or ASSISTANT), ordered content blocks, a deliveryStatus (DELIVERY_COMPLETE, DELIVERY_INTERRUPTED), and an ephemeral flag for messages spoken via DirectSpeech with includeInHistory: false. Audio content blocks (input_audio and tts_audio) include a transcription string that is populated asynchronously for user audio turns.

Tool Calling

Enable the model to call functions by defining tools and handling requests.

Define Tools

Send an UpdateToolDefinitionsRequest to register available tools. Each tool needs a name, description, and JSON Schema parameters:

function updateTools() {
  const message = ServiceBoundMessage.create({
    updateToolDefinitionsRequest: {
      toolDefinitions: [
        {
          name: 'get_weather',
          description: 'Get current weather for a location',
          parameters: {
            fields: {
              location: {
                kind: { stringValue: 'string' }
              }
            }
          }
        },
        {
          name: 'get_time',
          description: 'Get the current time',
          parameters: {}
        }
      ]
    }
  });

  ws.send(ServiceBoundMessage.encode(message).finish());
}

Calling UpdateToolDefinitionsRequest replaces all existing tools. Send an empty array to clear all tools.

Handle Tool Requests

When the model wants to use a tool, you receive a ToolCallRequest. You must respond with a ToolCallResponse:

ws.on('message', (data) => {
  const message = ClientBoundMessage.decode(new Uint8Array(data));

  if (message.toolCallRequest) {
    const { id, name, parameters } = message.toolCallRequest;

    // Execute the tool
    const result = executeToolCall(name, parameters);

    // Send the response (required for every request)
    const response = ServiceBoundMessage.create({
      toolCallResponse: {
        id: id,      // Must match the request ID
        result: result
      }
    });
    ws.send(ServiceBoundMessage.encode(response).finish());
  }
});

function executeToolCall(name, parameters) {
  switch (name) {
    case 'get_weather':
      const location = parameters?.fields?.location?.kind?.stringValue;
      return JSON.stringify({ temperature: 22, condition: 'sunny', location });

    case 'get_time':
      return JSON.stringify({ time: new Date().toISOString() });

    default:
      return JSON.stringify({ error: `Unknown tool: ${name}` });
  }
}

Every ToolCallRequest must receive a ToolCallResponse, even if the tool execution fails. The model waits for the response before continuing.

Complete Example

A Node.js client with microphone input. It configures no TTS provider, so the server returns text rather than speech. The client does not play audio and therefore does not report playback positions; see Playback Position Reporting for that side.

npm install ws protobufjs audify

import WebSocket from 'ws';
import protobuf from 'protobufjs';
import pkg from 'audify';
const { RtAudio, RtAudioFormat } = pkg;

async function main() {
  // Load protobuf types
  const root = await protobuf.load('realtime.proto');
  const ServiceBoundMessage = root.lookupType('eu.deepslate.realtime.speeq.ServiceBoundMessage');
  const ClientBoundMessage = root.lookupType('eu.deepslate.realtime.speeq.ClientBoundMessage');

  // Audio config
  const SAMPLE_RATE = 16000;
  const CHANNELS = 1;
  const FRAME_SIZE = 1600; // 100ms of audio

  // Set up audio I/O
  const rtAudio = new RtAudio();

  // State
  let packetId = 0;
  const spokenByTurn = new Map(); // turnId -> text the caller has heard
  const turns = new Map();        // turnId -> authoritative turn content
  let currentTurnId;              // turn opened by the most recent ResponseBegin

  // Connect
  const ws = new WebSocket('wss://app.deepslate.eu/api/v1/vendors/{vendorId}/organizations/{organizationId}/realtime', {
    headers: { 'Authorization': 'Bearer YOUR_API_KEY' }
  });
  ws.binaryType = 'arraybuffer';

  // Helper to send messages
  function send(payload) {
    const msg = ServiceBoundMessage.create(payload);
    ws.send(ServiceBoundMessage.encode(msg).finish());
  }

  ws.on('open', () => {
    // Initialize session
    send({
      initializeSessionRequest: {
        inputAudioLine: { sampleRate: SAMPLE_RATE, channelCount: CHANNELS, sampleFormat: 1 },
        outputAudioLine: { sampleRate: SAMPLE_RATE, channelCount: CHANNELS, sampleFormat: 1 },
        vadConfiguration: {
          confidenceThreshold: 0.5,
          minVolume: 0,
          startDuration: { seconds: 0, nanos: 300000000 },
          stopDuration: { seconds: 0, nanos: 700000000 },
          backbufferDuration: { seconds: 1, nanos: 0 }
        },
        inferenceConfiguration: {
          systemPrompt: 'You are a friendly and helpful assistant.'
        },
        supportsPlaybackReporting: false
      }
    });

    // Register tools
    send({
      updateToolDefinitionsRequest: {
        toolDefinitions: [{
          name: 'get_time',
          description: 'Get the current time',
          parameters: {}
        }]
      }
    });

    // Open microphone input
    rtAudio.openStream(
      null, // No output in this stream
      { nChannels: CHANNELS },
      RtAudioFormat.RTAUDIO_SINT16,
      SAMPLE_RATE,
      FRAME_SIZE,
      'deepslate-input',
      (pcm) => {
        send({
          userInput: {
            packetId: packetId++,
            mode: 2,  // IMMEDIATE
            audioData: { data: pcm }
          }
        });
      }
    );
    rtAudio.start();

    console.log('Listening... speak into your microphone');
  });

  ws.on('message', (data) => {
    const msg = ClientBoundMessage.decode(new Uint8Array(data));

    if (msg.responseBegin) {
      currentTurnId = msg.responseBegin.turnId;
      console.log(`\n[Response started: turn ${currentTurnId}]`);
    }

    if (msg.modelTextFragment) {
      // Emitted for every turn, whether or not TTS is configured
      process.stdout.write(msg.modelTextFragment.text);
    }

    if (msg.modelAudioChunk) {
      // Not reached: this example configures no TTS provider
    }

    if (msg.modelSpeechProgress) {
      // Text paced to playback: what the caller has heard so far
      const { turnId, text } = msg.modelSpeechProgress;
      spokenByTurn.set(turnId, (spokenByTurn.get(turnId) ?? '') + text);
    }

    if (msg.turnSnapshot) {
      // Authoritative turn content
      const { message: turn, isFinal } = msg.turnSnapshot;
      turns.set(turn.turnId, turn);
      if (isFinal) console.log(`\n[Turn ${turn.turnId} finalized]`);
    }

    if (msg.inferenceComplete) {
      console.log(`\n[Generation done: turn ${msg.inferenceComplete.turnId}]`);
    }

    if (msg.responseEnd) {
      console.log(`\n[Response ended: turn ${msg.responseEnd.turnId}]`);
    }

    if (msg.playbackClearBuffer) {
      // User started speaking. A client with playback would stop it here.
    }

    if (msg.toolCallRequest) {
      const { id, name, parameters } = msg.toolCallRequest;
      console.log(`Tool call: ${name}`, parameters);
      const result = handleTool(name, parameters);
      console.log(`Tool result: ${result}`);
      send({ toolCallResponse: { id, result } });
    }

    if (msg.userTranscriptionResult) {
      const { turnId, text, language } = msg.userTranscriptionResult;
      console.log(`[Transcription turn ${turnId} (${language})]: ${text}`);
    }

    if (msg.error) {
      const { category, message: errMsg, traceId } = msg.error;
      console.error(`Session error [${category}]: ${errMsg}`, traceId ?? '');
    }
  });

  ws.on('close', (code) => {
    rtAudio.stop();
    rtAudio.closeStream();
    console.log('Disconnected:', code);
  });

  ws.on('error', (err) => console.error('Error:', err));
}

function handleTool(name, parameters) {
  switch (name) {
    case 'get_time':
      return JSON.stringify({ time: new Date().toLocaleTimeString() });
    default:
      return JSON.stringify({ error: `Unknown tool: ${name}` });
  }
}

main();

Next Steps

API Reference

Full message schemas, all configuration options, and tool calling

WebRTC

Browser-based integration for end-user applications

LiveKit Plugin

LiveKit Agents plugin for Deepslate integration

Pipecat Plugin

Pipecat framework plugin for Deepslate integration

Getting started

Integration

SDK/Plugins

Voice Cloning

Prompt Engineering

Prerequisites

Connect

Initialize Session

Send Audio

Handle Responses

Handle Interruptions

Trigger Inference

Reconfigure Session

Direct Speech

Conversation Query

Playback Position Reporting

Track Turn Content

Export Chat History

Tool Calling

Define Tools

Handle Tool Requests

Complete Example

Next Steps

API Reference

WebRTC

LiveKit Plugin

Pipecat Plugin

​Prerequisites

​Connect

​Initialize Session

​Send Audio

​Handle Responses

​Handle Interruptions

​Trigger Inference

​Reconfigure Session

​Direct Speech

​Conversation Query

​Playback Position Reporting

​Track Turn Content

​Export Chat History

​Tool Calling

​Define Tools

​Handle Tool Requests

​Complete Example

​Next Steps

API Reference

WebRTC

LiveKit Plugin

Pipecat Plugin

Prerequisites

Connect

Initialize Session

Send Audio

Handle Responses

Handle Interruptions

Trigger Inference

Reconfigure Session

Direct Speech

Conversation Query

Playback Position Reporting

Track Turn Content

Export Chat History

Tool Calling

Define Tools

Handle Tool Requests

Complete Example

Next Steps