Introduction
Multimodal AI is no longer a research curiosity—it's a practical necessity. Modern applications often need to process a mix of text, audio, images, and video. In this post, I'll walk through building a production‑grade multimodal pipeline that combines Whisper (for speech‑to‑text), BERT (for understanding and classification), and BART (for summarization and generation).
The use case: a system that ingests audio recordings (meetings, lectures, calls), transcribes them with Whisper, extracts key insights using BERT‑based NER and classification, and then generates concise summaries and action items with BART. All in near‑real‑time.
Pipeline Overview
The pipeline is a directed flow with three main stages:
- Stage 1 (Speech): Whisper transcribes raw audio to text.
- Stage 2 (Understanding): BERT models perform entity extraction, sentiment analysis, and intent classification on the transcribed text.
- Stage 3 (Generation): BART summarizes the transcribed and annotated text, producing a concise executive summary and a list of action items.
All stages are orchestrated using a task queue (Celery) and can be scaled independently based on workload.
Whisper: Speech-to-Text Foundation
We use OpenAI Whisper (the large‑v3 model) for transcription. It handles multiple languages and noisy audio well. In production, we use the faster‑whisper implementation for a ~4x speedup with minimal accuracy loss.
Key considerations:
- Chunking: We split long audio (e.g., 1‑hour lectures) into 30‑second segments with overlap to avoid context loss.
- Language detection: Whisper automatically detects language, but we override it when we know the expected language to improve accuracy.
- Prompt engineering: We use a system prompt to guide Whisper's punctuation and formatting (e.g., adding commas, capitalizing names).
- Post‑processing: We apply a custom diarization step to label different speakers using a lightweight x‑vector model.
In our tests, Whisper achieved a Word Error Rate (WER) of 5.2% on clean speech and 8.7% on recorded meetings with moderate background noise.
BERT: Understanding the Text
For understanding, we fine‑tuned a BERT‑base model for three tasks:
- Named Entity Recognition (NER): Extracting people, organizations, dates, and product names.
- Intent Classification: Categorizing the overall purpose (e.g., "meeting", "lecture", "customer support").
- Sentiment Analysis: For customer support calls, we track sentiment per utterance to flag escalations.
We used a single multi‑task BERT model with multiple classification heads, sharing the encoder. This reduced memory footprint and improved inference speed.
We also added a key‑phrase extraction module using a combination of BERT embeddings and RAKE (Rapid Automatic Keyword Extraction) to highlight important terms.
BART: Summarization and Generation
BART (BART‑large) is our workhorse for generation. We use it for two purposes:
- Abstractive Summarization: Generating a 3‑4 sentence executive summary of the transcribed content.
- Action Item Extraction: A zero‑shot generation task where we prompt BART to list actionable tasks mentioned in the text.
We fine‑tuned BART on a custom dataset of meeting transcripts and summaries (about 5,000 examples) to improve coherence and style. The fine‑tuned model achieved a ROUGE‑L score of 0.41 on our held‑out test set, a 12% improvement over the base model.
For action items, we used a prompt‑based approach: we prepend the transcript with "Extract action items from the following conversation:" and let BART generate a bulleted list. We then post‑process to deduplicate and clean.
Architecture and Orchestration
We deploy the pipeline as a set of microservices using FastAPI for REST endpoints and WebSockets for streaming audio. Each model runs in its own container with a GPU assigned based on its needs (Whisper and BERT share a GPU, BART has its own for heavy generation).
Orchestration is handled by Celery with Redis as the broker. We have separate queues for each stage, allowing us to scale the summarization workers independently during peak load.
Performance Optimizations
To meet near‑real‑time requirements, we applied several optimizations:
- Model Quantization: We quantized BERT and BART to INT8 using PyTorch's quantization, reducing memory usage by 40% and inference time by 25% with negligible accuracy loss.
- Batch Processing: When possible, we batch transcription requests to maximize GPU utilization.
- Cache Common Transcripts: We store transcriptions for frequently repeated audio (e.g., standard greetings) in a Redis cache, bypassing the pipeline entirely.
- Streaming Output: For the generation stage, we stream tokens as they are produced, allowing the frontend to display a partial summary while the full generation completes.
- GPU Selection: We use A100 GPUs for BART (large model) and T4 GPUs for Whisper and BERT, balancing cost and performance.
After optimizations, our end‑to‑end latency for a 5‑minute audio segment dropped from 18 seconds to 6 seconds (p95), making it suitable for interactive use.
Evaluation and Metrics
We evaluate each component separately and the pipeline as a whole:
- Whisper: WER and character error rate on a held‑out dataset of 500 audio files.
- BERT: F1 scores for NER (0.89), intent classification (0.93), and sentiment (0.91).
- BART: ROUGE‑1/2/L for summarization and human evaluation of action item quality (1–5 scale).
- Pipeline: End‑to‑end accuracy—we manually review 100 outputs per week to check if the summary captures the main points and the action items are correct.
We also track business metrics: user satisfaction score (from feedback widget), average time to summary, and cost per audio minute (currently ~$0.002/min).
Lessons Learned
Building a multimodal pipeline is a systems engineering challenge. Here are our key takeaways:
- Model selection matters beyond accuracy. We initially used a larger BART model but found the performance gains didn't justify the 2x inference cost. We downgraded to BART‑base and the quality difference was negligible for our use case.
- Diarization is hard. We tried several off‑the‑shelf diarization tools but all had high error rates in noisy environments. We ended up using a simple speaker‑change detection based on voice activity and assigned labels based on a fixed number of expected speakers (usually 2–4).
- Prompt engineering for BART is critical. We experimented with different prompts and found that adding a "tone" instruction (e.g., "be professional" vs. "be casual") significantly changed the output style.
- Async processing is a must. Since transcription can take several seconds, we process audio asynchronously and notify the user via WebSocket when the summary is ready.
- Test with real data early. Our initial test set was clean academic recordings; when we moved to real meeting recordings (with overlapping speech, background noise, etc.), performance dropped significantly. We had to retrain our VAD and diarization components.
Conclusion
Multimodal AI is becoming essential for modern applications. By combining Whisper, BERT, and BART in a thoughtful pipeline, we built a system that transcribes, understands, and summarizes audio content with near‑real‑time performance. The modular architecture allows us to swap components (e.g., replace Whisper with a custom STT) as better models emerge.
The journey from a Jupyter prototype to a production pipeline involved careful model selection, performance tuning, and rigorous evaluation. But the result is a system that delivers real value—saving meeting participants time and helping teams stay aligned.
I'd love to hear about your own multimodal projects. Reach out on Twitter or GitHub.