Introduction
Voice assistants are becoming ubiquitous, but building a low‑latency, conversational AI pipeline from scratch is still a complex systems challenge. In this post, I'll share how I engineered a near‑real‑time voice AI system using WebSockets for bidirectional streaming, Faster‑Whisper for transcription, a large language model for response generation, and a TTS engine for speech synthesis.
The goal was to achieve end‑to‑end latency under 2 seconds while maintaining high accuracy and a natural conversational flow. Here's how we did it.
Why WebSockets for Voice?
Traditional HTTP request‑response is ill‑suited for voice interactions. WebSockets provide:
- Full‑duplex communication – simultaneous audio streaming and text/audio responses.
- Low overhead – persistent connection reduces handshake latency.
- Streaming support – we can send audio chunks incrementally and receive partial responses.
We chose WebSockets as the primary transport, with a fallback to HTTP polling for environments that block WebSockets (though rare).
Architecture Overview
The pipeline consists of several components connected via a central WebSocket server:
The server manages sessions, buffers incoming audio, invokes the STT model, calls the LLM, and streams back the TTS audio. All stages are asynchronous to avoid blocking.
Speech-to-Text with Faster-Whisper
We evaluated several STT models and settled on Faster‑Whisper (distilled version) for its speed and accuracy. With a tiny or base model, we achieved a transcription latency of ~300ms per utterance on a moderate GPU.
To handle continuous speech, we implemented a Voice Activity Detection (VAD) that segments audio into utterances. The VAD runs on the server, using webrtcvad to detect speech boundaries. Once an utterance is complete, we send it to Faster‑Whisper for transcription.
We also added a silence‑based auto‑submit – if the user pauses for more than 700ms, the system automatically sends the accumulated audio for transcription, mimicking the behaviour of modern assistants.
LLM Integration & Response Streaming
For the language model, we used a mix of Gemini‑2.5‑Flash for most queries and GPT‑4o for complex reasoning (via a router). The LLM is invoked with the transcribed text and optionally with conversation history (stored per session).
We leverage streaming from the LLM – we receive tokens incrementally and begin generating the TTS output as soon as the first few words are available. This overlaps the LLM's generation with TTS synthesis, reducing perceived latency.
To keep costs in check, we implemented a simple caching layer for frequently asked questions and used a smaller model for short, factual responses.
Text-to-Speech Synthesis
We integrated a cloud‑based TTS service (Google Cloud TTS) with support for streaming audio. The TTS engine receives text and outputs audio in chunks (MP3 or Opus) that are sent back over the WebSocket to the client for immediate playback.
To reduce latency, we pre‑warm the TTS connection and use SSML to control prosody and speed. We also experimented with local TTS (e.g., Piper) for offline use, but the quality was lower.
Latency Optimizations
We broke down the end‑to‑end latency into its components and optimised each:
- Network: Used a regional deployment (same cloud region as users) and WebSocket compression.
- STT: Switched from a larger Whisper model to the distilled version, and used GPU inference.
- LLM: Reduced the prompt size, used a faster model, and implemented streaming.
- TTS: Used a low‑latency TTS engine with chunked output.
- VAD: Fine‑tuned silence thresholds to avoid unnecessary round‑trips.
After optimisations, we achieved a p95 end‑to‑end latency of 1.8 seconds, which felt natural in user tests.
Deployment & Scaling
The system is containerised with Docker and orchestrated via Kubernetes. We deployed two types of nodes: one for the WebSocket server (stateless, easy to scale horizontally) and another for the inference pipeline (GPU‑heavy, scaled with a queue).
We used Redis for session state (caching conversation history and VAD state) to allow sticky sessions across WebSocket connections. For auto‑scaling, we set up a horizontal pod autoscaler based on CPU and custom metrics (e.g., queue length).
Monitoring is done via Prometheus and Grafana, tracking metrics like connection count, latency percentiles, and error rates.
Lessons Learned
Here are some practical takeaways from building this system:
- Start with the speech boundary detection. VAD tuning is critical for a good user experience – too sensitive and you get cut‑off, too conservative and you get long delays.
- Overlap inference with output. Streaming LLM responses while synthesizing TTS in parallel cuts perceived latency significantly.
- Fallbacks are essential. Have a backup STT or TTS provider in case the primary fails.
- Monitor cost per session. Voice AI can be expensive; set alerts and optimise model selection dynamically.
- Test with real‑world noise. Our initial tests were in quiet rooms; in production, background noise caused VAD to fail. We added a noise‑reduction pre‑processor.
Conclusion
Building a near‑real‑time voice AI system is a rewarding engineering challenge. WebSockets provide the ideal transport for bidirectional audio streaming, and with careful optimisation of each component, we achieved latency that feels instantaneous to users. The system is now in production, handling thousands of sessions per day.
If you're building a voice assistant, start with a simple prototype and iterate. Focus on latency and reliability first—accuracy can be improved over time with better models.
I'd love to hear about your own voice AI projects. Reach out on Twitter or GitHub.