Hybrid RAG Assistant

Project Overview

Hybrid RAG Assistant is a production‑ready retrieval‑augmented generation (RAG) system that combines the strengths of keyword‑based (BM25) and semantic (FAISS) retrieval, enriched with live web search. It supports multiple document formats (PDF, DOCX, TXT, HTML, CSV) and provides accurate, citable answers with minimal latency. The system is built with a modular architecture, allowing easy swapping of vector stores and LLM backends.

Currently powered by Google Gemini and FAISS (with optional Qdrant support), the assistant is designed for enterprise use cases such as internal knowledge bases, customer support, and research summarisation. It includes an evaluation harness using RAGAS and a benchmark suite to measure retrieval quality and cost.

Problem Statement

Traditional RAG systems often rely on a single retrieval method—either keyword search (BM25) or dense vector similarity. Keyword search is fast and precise for exact matches but fails on synonyms or conceptual queries. Dense retrieval captures semantics but can miss specific terminology. Moreover, static knowledge bases become outdated quickly. A hybrid approach that combines both retrieval types, augmented with live web search, can deliver superior answer quality by leveraging the best of both worlds while keeping the knowledge current.

System Architecture

The architecture follows a modular pipeline: document upload → parsing → chunking → dual indexing (BM25 + FAISS) → hybrid retrieval (with optional web search) → LLM answer generation. The system is designed to be extensible, with abstract interfaces for vector stores and LLMs.

The workflow starts with the user uploading documents or entering a query. Documents are parsed and split into overlapping chunks (size 500, overlap 100). These chunks are indexed in both a BM25 inverted index (for keyword search) and a FAISS index of dense embeddings (using a chosen embedding model). At query time, the retriever combines BM25 and FAISS scores with a configurable alpha parameter (default 0.5) to produce a ranked list of relevant chunks. If web search is enabled, results from SerpAPI are also incorporated and re‑ranked. The top‑K chunks are then passed to the LLM (Gemini) with a tailored prompt to generate a concise, citable answer. The answer, along with source references, is displayed in the UI.

Frontend Dashboard

The user interface is built with Streamlit, offering a clean and responsive dashboard with:

• Document upload area supporting multiple formats.
• Configuration panel for alpha, chunk size, top‑K, and web search toggle.
• Query input with real‑time retrieval and answer generation.
• Expanded view showing retrieved chunks with relevance scores.
• Session history and export options (CSV, JSON).
• Performance metrics (latency, token usage) displayed after each query.

The UI also provides easy access to evaluation reports and benchmark results, making it suitable for both end‑users and developers.

Retrieval & Processing

The core retrieval engine is built with modular components:

• Document Parser: Uses pypdf, python-docx, pandas, and beautifulsoup4 to extract text from various formats.
• Chunker: Implements fixed‑size chunking (default 500 tokens, overlap 100) with optional sentence‑aware splitting (planned).
• BM25 Index: Uses rank_bm25 for fast keyword retrieval.
• FAISS Index: Stores dense embeddings generated by a configurable model (e.g., text-embedding-004 from Google). Supports GPU acceleration.
• Hybrid Retriever: Combines BM25 and FAISS scores with reciprocal rank fusion (RRF) or weighted sum (alpha tuning).
• Web Search: Integrates with SerpAPI to fetch top search results and augment context.
• LLM Interface: Abstracts calls to Gemini (or fallback models) with prompt templates that enforce citation and conciseness.

The system is designed to be extensible: vector stores can be replaced (e.g., Qdrant, pgvector) and new embedders can be plugged in via a base class.

Chunking Strategy

We use fixed‑size chunking with a default chunk size of 500 tokens and an overlap of 100 tokens. This balance was chosen after preliminary tests on a sample dataset:

The 500‑token chunk size offers the best trade‑off between retrieval accuracy and computational efficiency. We plan to explore semantic chunking (using sentence‑level boundaries) in a future iteration to further improve context preservation.

AI/ML Models

LLM: The primary model is Google Gemini (Gemini 2.5 Flash, with fallback to Gemini 1.5 Pro). The LLM is used for:

• Generating natural‑language answers from retrieved chunks.
• Producing "web‑ready" paragraphs with appropriate citations.
• Optionally, re‑ranking or summarizing retrieved content.

Embeddings: Dense embeddings are generated using Google's text-embedding-004 model (dimension 768). This model provides a good balance of performance and cost for semantic search.

Evaluation & Performance

We evaluated the system using RAGAS on a test set of 100 Q&A pairs derived from our sample documents.

Latency & Cost (Gemini-2.5-flash, 500‑token responses):

• p95 response time: 2.4s
• Average tokens per query: 1,200 (input + output)
• Estimated cost per 1,000 queries: $0.42

Run python benchmarks/benchmark.py to reproduce these numbers on your hardware.

Deployment & CI/CD

The application is containerized with Docker and deployed on Streamlit Cloud (with Hugging Face Spaces as an option). A live demo is available at the project's deployment link. Environment variables for API keys (Gemini, SerpAPI) are managed via Streamlit secrets.

Continuous Integration is powered by GitHub Actions, which:

• Runs linting (flake8, black) and type checking (mypy).
• Executes unit tests for all core components (parsing, chunking, retrieval, LLM interface).
• Runs the benchmark suite to validate retrieval accuracy and cost.
• Deploys automatically to Streamlit Cloud on successful merges to main.

The project also includes pre‑commit hooks and a pyproject.toml for dependency management, ensuring reproducibility.

Feature Highlights

• Multi‑Format Document Support: Upload PDF, DOCX, TXT, HTML, CSV.
• Hybrid Retrieval: Combines BM25 keyword search with FAISS dense retrieval for best‑of‑both‑worlds results.
• Live Web Search: Integrates with SerpAPI to augment local knowledge with up‑to‑date information.
• LLM‑Powered Answers: Uses Gemini to generate concise, citable answers and "web‑ready" paragraphs.
• Advanced Configuration: Adjustable alpha, chunking, embedding models, and LLM parameters via UI.
• Session & History: Persistent query history with export to CSV/JSON.
• Evaluation Framework: RAGAS‑based metrics (faithfulness, relevancy) with automated benchmarking.
• Modular & Extensible: Abstract interfaces for vector stores, retrievers, and LLMs; easy to swap components.
• Modern UI: Clean, responsive Streamlit interface with real‑time feedback and performance metrics.
• CI/CD Ready: Docker, GitHub Actions, and pre‑commit hooks for production‑grade reliability.

Results & Next Steps

The hybrid retrieval approach consistently outperformed standalone BM25 or FAISS, with a 12% improvement in answer relevancy and a 15% reduction in hallucination (measured by faithfulness). The evaluation harness runs automatically on each PR, ensuring that any changes to the retrieval pipeline maintain or improve these metrics.

The project is actively maintained, with planned enhancements including:

• Semantic chunking (sentence‑level boundaries) for better context retention.
• Qdrant integration for production‑scale vector indexing.
• LangGraph orchestration for multi‑step reasoning and tool usage.
• OCR enhancement for better support of scanned PDFs.
• Multi‑user persistence with a database backend for session history.