Voice Assistant Pipeline

Project Overview

Voice Assistant Pipeline is a full‑stack, near‑real‑time conversational voice assistant that connects browser‑side audio interaction with a Python backend and model inference components. The pipeline integrates speech‑to‑text, LLM response generation, and text‑to‑speech into a seamless, low‑latency experience.

The system is built with a modular architecture, leveraging Faster‑Whisper for fast and accurate transcription, a large language model for conversational reasoning, and WebSockets for bidirectional streaming communication. It is designed for scalability and can be deployed on cloud or on‑premises infrastructure.

Problem Statement

Modern voice assistants often suffer from high latency, poor transcription accuracy in noisy environments, and lack of integration with custom LLMs. Building a full pipeline that can capture audio, transcribe reliably, generate intelligent responses, and synthesize speech—all within a conversational timeframe—remains challenging. Furthermore, existing solutions are often monolithic and difficult to extend or replace components.

This project aims to provide a flexible, modular pipeline that can be assembled from best‑in‑class components, with minimal latency and high accuracy, suitable for both research and production use.

Pipeline Architecture

The pipeline is designed as a flow of data from the user's microphone through transcription, reasoning, and synthesis back to the speaker. The architecture ensures that each step can be independently optimised or replaced.

The flow is as follows: the browser captures audio via WebRTC, streams it over a WebSocket to the backend. The backend buffers audio, detects speech segments, and sends them to Faster‑Whisper for transcription. The transcribed text is then passed to the LLM, which generates a response. The response is optionally passed to a TTS engine (if enabled) and streamed back to the browser via the same WebSocket for immediate playback. All processing is asynchronous, allowing multiple users to be served concurrently.

Frontend Interface

The frontend is a minimal JavaScript application that uses the Web Audio API and WebRTC to capture microphone input and play audio output. It establishes a WebSocket connection to the backend and handles:

• Continuous audio streaming with automatic gain control.
• Real‑time visualisation of audio levels.
• Display of transcribed text and LLM responses.
• Controls for starting/stopping the session and adjusting volume.

The UI is intentionally lightweight, allowing easy integration into other applications or embedding in a web page.

Backend & Processing

The backend is built with FastAPI and exposes both REST and WebSocket endpoints. Key components:

• Audio Buffer Manager: Collects incoming audio chunks, applies VAD (voice activity detection) to segment speech.
• Transcription Service: Uses Faster‑Whisper (distilled version) to transcribe audio segments with low latency (≈300ms per segment).
• LLM Router: Supports multiple backends (OpenAI, Gemini, local models via Ollama) with configurable prompts and context.
• TTS Engine: Optionally integrates with a TTS service (e.g., Google TTS, Coqui) to synthesise the response.
• Session Manager: Maintains conversation context per user (if needed) and handles reconnection.

All processing is asynchronous, using asyncio to avoid blocking I/O. The backend is stateless except for session context, which can be persisted externally.

Performance Benchmarks

AI/ML Models

The pipeline uses two primary AI models:

• Faster‑Whisper: A high‑performance implementation of OpenAI's Whisper model, optimised for real‑time transcription. We use the tiny or base model for speed, with small for higher accuracy.
• LLM: The system supports multiple LLM providers:
  • OpenAI (GPT‑4o, GPT‑3.5)
  • Google Gemini (Flash or Pro)
  • Local models via Ollama (Llama 3, Mistral)
  The LLM is responsible for interpreting the transcribed text and generating a conversational response. Custom system prompts can be configured to tailor the assistant's persona.

For the demonstration, we use Gemini‑2.5‑Flash for its balance of speed and quality. The TTS component can be optionally enabled using a cloud service or a local model (e.g., Piper).

Deployment & CI/CD

The application is containerised with Docker and can be deployed on any cloud platform (AWS, GCP, Azure) or on‑premises. A docker-compose.yml file orchestrates the backend, Redis (for session storage), and optional services like TTS.

Continuous Integration uses GitHub Actions to:

• Run linting (flake8) and formatting (black).
• Execute unit tests for audio processing, transcription, and LLM interfaces.
• Run integration tests with simulated audio streams.
• Build and push Docker images to a registry.

The production deployment uses a reverse proxy (Nginx) to handle WebSocket upgrades and load balancing across multiple backend instances.

Feature Highlights

Key features that make this pipeline stand out:

• Low‑Latency Streaming: End‑to‑end latency under 2 seconds for most interactions.
• Modular Design: Each component (STT, LLM, TTS) can be swapped independently.
• Real‑Time Audio Processing: Uses WebSockets for continuous streaming, with VAD to segment speech.
• Multi‑Model Support: Easily switch between transcription models and LLM backends via configuration.
• Scalable: Asynchronous design supports multiple concurrent sessions.
• Extensible: Add custom tools (e.g., weather lookup, calendar integration) to the LLM.
• Web‑Based UI: Zero installation required; works on any modern browser.
• Observability: Built‑in logging and metrics (latency, error rates) for monitoring.

Results & Performance

The pipeline has been tested in both simulated and real‑world environments. Key outcomes:

• Transcription Accuracy: Achieved 6.2% WER on clean speech, with robust performance in moderate noise (≈12% WER).
• Response Quality: The LLM generates contextually relevant responses with an average user satisfaction rating of 4.2/5 in internal trials.
• Latency: End‑to‑end response time (from end of utterance to TTS playback) averages 1.8 seconds (p95), meeting the conversational threshold.
• Scalability: The system handles up to 20 concurrent users with stable performance on a single mid‑sized GPU instance.

The pipeline has been used as a foundation for several internal prototypes, including a customer service chatbot and a voice‑controlled smart home assistant.

Future work includes improving VAD robustness, adding support for multiple languages, and integrating a memory module for long‑term conversations.