Optimizing Latency with Resemble AI Streaming Speech for Conversational Voicebots

In conversational voice applications—ranging from telephony customer service agents to real-time interactive AI assistants—response latency determines whether an interaction feels natural or awkwardly robotic. Human conversational turn-taking typically occurs within a 200 ms to 500 ms window; once end-to-end latency climbs past 800 ms, speakers begin talking over one another, and perceived system responsiveness degrades rapidly.

In a voicebot pipeline, latency accumulates across multiple components: voice activity detection (VAD), speech-to-text (STT), large language model (LLM) inference, text-to-speech (TTS), and audio transport. Because TTS sits directly on the critical path following the LLM's first generated tokens, optimizing speech synthesis is essential to achieving sub-second turnarounds.

Resemble AI provides a dedicated WebSocket streaming interface (wss://websocket.cluster.resemble.ai/stream) that enables real-time speech synthesis with sub-second time-to-first-byte (TTFB). By combining persistent WebSocket connections, optimized audio payloads, smart utterance chunking, and responsive interruption handling, developers can compress speech playback latency down to 200–350 ms.

1. The Voicebot Latency Budget

To evaluate where milliseconds are won or lost, examine the standard turn-taking sequence in a conversational voicebot:

Traditional REST TTS endpoints require rendering the entire sentence or paragraph before sending an audio file back over HTTP, introducing 1,500 ms to 3,000 ms of synthesis delay alone. In contrast, streaming TTS begins returning playable audio frames while downstream text is still being spoken, removing TTS duration from the end-to-end critical path.

2. Choosing the Right Synthesis Transport

Resemble AI provides three synthesis modes, but only one is suitable for conversational voicebots:

Synchronous REST API (POST /api/v2/projects/{project_uuid}/clips)

Renders the complete audio clip before responding. Ideal for asynchronous batch jobs, static IVR prompts, and video narration, but introduces excessive delay for conversational voicebots.

Chunked HTTP Streaming (POST https://f.cluster.resemble.ai/stream)

Returns chunked WAV data progressively over an HTTP connection. While it lowers perceived latency for long-form reading, opening a new HTTP connection for every conversational turn incurs repeated TCP handshakes and TLS negotiation overhead (adding 80 ms to 150 ms per turn).

WebSocket Streaming (wss://websocket.cluster.resemble.ai/stream)

Maintains a persistent, bidirectional WebSocket connection. Because the connection remains warm across conversational turns, requests incur zero transport handshake latency. Audio chunks stream back sequentially as binary or JSON frames alongside word-level timestamps.

Plan Requirement: Resemble AI's WebSocket streaming interface requires a Business plan or above. By default, clusters support up to 20 simultaneous sessions per cluster and 20 parallel connections per API key.

3. Payload and Audio Parameter Tuning

Synthesizing speech over WebSockets requires balancing fidelity against payload size and CPU decoding overhead. Resemble AI allows fine-grained configuration in the synthesis JSON payload:

{
  "voice_uuid": "YOUR_VOICE_UUID",
  "project_uuid": "YOUR_PROJECT_UUID",
  "data": "Hello! How can I help you today?",
  "binary_response": true,
  "request_id": 1,
  "output_format": "wav",
  "sample_rate": 16000,
  "precision": "PCM_16",
  "no_audio_header": true,
  "apply_custom_pronunciations": false
}

Key Parameter Optimizations

• binary_response: true
  By default (false), Resemble returns JSON frames containing Base64-encoded audio strings. Base64 encoding inflates network bandwidth by approximately 33% and demands CPU cycles to encode and decode on every chunk. Enabling binary_response: true streams raw audio bytes directly, reducing payload size and deserialization overhead.

• precision: "PCM_16"
  Resemble defaults to PCM_32 on certain endpoints. PCM_16 (16-bit linear PCM) cuts the transmitted byte volume in half compared to 32-bit audio without audible degradation in voice clarity. Furthermore, 16-bit PCM aligns natively with standard WebRTC audio engines and telephony gateways, eliminating intermediate resampling steps.

• sample_rate: 16000 (or 24000)
  While studio production benefits from 44.1 kHz or 48 kHz, conversational voicebots do not require full-band studio sampling. Telephony systems (Twilio, SIP trunks, G.711 / Opus) operate on 8 kHz or 16 kHz wideband audio. Requesting 16 kHz streams approximately one-third the byte volume of 48 kHz, reducing network serialization latency and packet fragmentation.

• no_audio_header: true
  When receiving binary chunks, setting no_audio_header: true strips the standard 44-byte RIFF/WAV header from the initial chunk. This delivers a continuous stream of pure PCM byte frames that can be piped directly into WebRTC tracks or telephony RTP packetizers without custom header stripping.

• apply_custom_pronunciations: false
  Unless your application relies on domain-specific vocabulary loaded into your Resemble pronunciation dictionary, keep this set to false to avoid dictionary lookup overhead on each synthesis pass.

4. Utterance Planning: The Asymmetric Chunking Strategy

When connecting an LLM's streaming token output to Resemble AI's TTS engine, developers face the chunking trade-off:

1. Token-by-token (Too Small): Pushing two or three words to the TTS produces choppy, unnatural intonation and floods the WebSocket with micro-requests.
2. Full Sentence / Full Turn (Too Large): Waiting for a full sentence or paragraph forces the user to wait for 15–30 LLM tokens, adding 800 ms to 1,500 ms of idle silence.

To resolve this, adopt an Asymmetric Utterance Planning strategy:

[ User Finishes Speaking ]
         │
         ▼
[ LLM Generates Tokens ]
         │
         ├─► Chunk 1 (Aggressive): Split at first comma, dash, or 4–6 words
         │         │
         │         ▼
         │   Send to Resemble AI WebSocket  ──► First audio plays at ~300 ms
         │
         └─► Chunk 2+ (Syntactic): Buffer to full sentence boundaries (. ? !)
                   │
                   ▼
             Send to Resemble AI WebSocket  ──► Arrives while Chunk 1 is still speaking

Chunk 1: Aggressive First Clause

Buffer only until the first clause punctuation mark (,, ;, :, —) or a threshold of 4 to 8 words (approximately 25–40 characters). Dispatch this opening fragment to Resemble AI immediately. Because speech takes roughly 1.5 to 2.5 seconds to speak even a short clause, this buys valuable execution time while the user begins hearing speech in 250–350 ms.

Chunk 2 and Later: Full Syntactic Sentences

While the user is listening to Chunk 1 playing out of the audio buffer, buffer incoming LLM tokens up to complete sentence terminators (., ?, !, or newline). Dispatching complete sentences ensures that Resemble Ultra generates pitch contours, natural breathing, and accurate cadence without jitter.

Text Normalization and Sanitization

LLMs frequently emit Markdown symbols (**bold**, *italics*, bullet points - , numbered lists 1. ), code blocks, or raw emojis. Feeding Markdown characters to TTS causes the model to either stumble, pause, or literally vocalize symbols (e.g., reading "asterisk asterisk"). Run a zero-allocation regex filter on the text buffer before pushing it to the WebSocket payload:

import re

CLEANUP_REGEX = re.compile(r"[*_#`~]|\[.*?\]\(.*?\)|\n\s*[-*]\s*")

def sanitize_for_tts(text: str) -> str:
    cleaned = CLEANUP_REGEX.sub(" ", text)
    return " ".join(cleaned.split())

5. Connection Pooling and Persistent Socket Architecture

Creating a new WebSocket connection for each conversational turn destroys the latency benefit. Opening a TLS socket requires DNS resolution, TCP handshake, and TLS session negotiation, which adds 100 ms to 250 ms depending on network distance.

Connection Architecture Best Practices

• Maintain Persistent Sessions: Keep the WebSocket connected throughout the duration of a voice call or chat session.
• Warm Socket Pool: In multi-tenant voicebot servers, pre-warm a pool of active WebSocket connections to wss://websocket.cluster.resemble.ai/stream. When a call connects via SIP or WebRTC, assign an idle connection from the pool.
• Correlate with request_id: Increment request_id for each utterance within the same connection. When Resemble returns audio_end with the matching request_id, your state machine transitions to awaiting the next user input.

import asyncio
import json
import websockets

class ResembleAudioStreamer:
    def __init__(self, api_token: str, voice_uuid: str, project_uuid: str):
        self.uri = "wss://websocket.cluster.resemble.ai/stream"
        self.api_token = api_token
        self.voice_uuid = voice_uuid
        self.project_uuid = project_uuid
        self.ws = None
        self.request_counter = 0

    async def connect(self):
        headers = {"Authorization": f"Bearer {self.api_token}"}
        self.ws = await websockets.connect(self.uri, extra_headers=headers)

    async def stream_utterance(self, text_chunk: str, audio_queue: asyncio.Queue):
        self.request_counter += 1
        req_id = self.request_counter
        
        payload = {
            "voice_uuid": self.voice_uuid,
            "project_uuid": self.project_uuid,
            "data": text_chunk,
            "binary_response": True,
            "request_id": req_id,
            "output_format": "wav",
            "sample_rate": 16000,
            "precision": "PCM_16",
            "no_audio_header": True
        }
        
        await self.ws.send(json.dumps(payload))
        
        while True:
            message = await self.ws.recv()
            if isinstance(message, bytes):
                # Raw PCM_16 bytes ready for audio track playback
                await audio_queue.put(message)
            else:
                data = json.loads(message)
                if data.get("type") == "audio_end" and data.get("request_id") == req_id:
                    break
                elif data.get("type") == "error":
                    raise RuntimeError(f"Resemble API Error: {data.get('message')}")

6. Barge-in and Interruption Handling

In human conversations, people frequently interrupt or speak over the bot. When your VAD detects user speech while the voicebot is actively playing synthesized audio, the pipeline must execute an instant barge-in routine:

1. Flush Playback Buffers Immediately: Clear the client-side speaker queue, WebRTC media buffer, or telephony jitter buffer so sound stops within 50 ms.
2. Discard Obsolete Frames: Increment an internal active_turn_id. Discard any subsequent audio chunks received from the Resemble WebSocket that carry an outdated request_id.
3. Do Not Terminate the WebSocket: Terminating and recreating the WebSocket adds reconnection overhead for the bot's upcoming response. Simply let the current in-flight stream complete or issue a cancellation frame while keeping the socket open for the next turn.

7. Ecosystem Integrations: LiveKit and Pipecat

Instead of building custom WebSocket state machines from scratch, modern voicebot architectures often rely on voice agent orchestration frameworks with native Resemble AI integrations:

LiveKit Agents (livekit-plugins-resemble)

LiveKit provides real-time WebRTC infrastructure with sub-100 ms transport latency. The official Resemble plugin connects directly into the AgentSession:

uv add "livekit-agents[resemble]~=1.8"

from livekit.agents import AgentSession
from livekit.plugins import resemble

session = AgentSession(
    tts=resemble.TTS(
        voice_uuid="YOUR_VOICE_UUID",
        # Configures streaming transport automatically
    ),
    # ... stt, llm, vad configurations
)

LiveKit handles jitter buffering, WebRTC packetization, and barge-in audio interruption natively, piping Resemble's synthesized audio straight to the participant's audio track.

Pipecat (pipecat-ai[resemble])

Pipecat's pipeline architecture operates with ResembleAITTSService. It manages asynchronous frames, word-level timestamp alignment for UI lip-sync, and automatic buffer clearing on user speech interrupts.

8. Summary Checklist for Sub-Second Voicebots

To guarantee sub-second latency with Resemble AI in production voicebots, verify each item in this implementation checklist:

• Protocol: Use wss://websocket.cluster.resemble.ai/stream over persistent connections; avoid synchronous REST endpoints.
• Audio Configuration: Request binary_response: true, precision: "PCM_16", sample_rate: 16000, and no_audio_header: true.
• Utterance Chunking: Dispatch the first clause (4–8 words or first comma) immediately, then chunk subsequent tokens at full sentence boundaries.
• Text Sanitization: Strip Markdown syntax and formatting symbols before text reaches the synthesis payload.
• Socket Reuse: Keep the WebSocket open across conversational turns; do not close sockets during user barge-in.
• Geographic Co-location: Deploy agent orchestration servers in the same cloud region as your LLM and TTS endpoints to minimize round-trip network hops.

References

• Streaming (WebSocket) | Resemble | Documentation
• Text-to-Speech | Resemble | Documentation
• Streaming (HTTP) | Resemble | Documentation
• Resemble AI TTS plugin guide | LiveKit Documentation
• Model Versions | Resemble | Documentation
• Rate Limits | Resemble | Documentation