Introduction
Retrieval-Augmented Generation (RAG) has become the de facto architecture for building LLM applications grounded in external knowledge. Yet most teams focus their evaluation efforts on retrieval metrics like Hit Rate or Mean Reciprocal Rank (MRR). While these are useful, they tell only part of the story.
In this post, I'll explain why retrieval metrics alone are insufficient and how to evaluate the full stack: grounding (faithfulness to sources), relevance (contextual appropriateness), and answer quality (completeness, correctness, and helpfulness). Drawing from real projects, I'll share a practical framework for RAG evaluation that goes well beyond "did we retrieve the right document?"
Why Retrieval Metrics Aren't Enough
Retrieval metrics tell us if the right documents are in the top‑K candidates. But a RAG system can pass retrieval with flying colours and still produce a terrible answer:
- Hallucination: The LLM ignores the retrieved context and generates a plausible but incorrect answer.
- Mis‑grounding: The answer references the retrieved documents but misinterprets the facts.
- Irrelevant context: The retrieved documents are topically relevant but don't actually answer the user's question.
- Incomplete answer: The LLM retrieves the right information but leaves out critical details.
In one of our projects, retrieval MRR was 0.92, yet end‑user satisfaction was below 60%. The gap was entirely in how the LLM used the retrieved context. We needed a richer evaluation framework.
Grounding: The Foundation of Trust
Grounding measures whether the LLM's response is faithful to the retrieved context. A grounded answer should only contain claims that are supported by one or more retrieved chunks.
We use a combination of automatic and human evaluation:
- Automatic: Using RAGAS and Faithfulness metrics, which compare the answer's statements against the context using an LLM-as-judge.
- Human: A small set of experts reviews answers for hallucination and logical consistency.
In practice, we found that automatic grounding scores correlate strongly with human judgement (ρ ≈ 0.85), making them a good proxy for detection.
We also implemented a citation extraction step that forces the LLM to annotate each factual claim with a citation number, which we then verify against the source chunks. This both improves grounding and gives users traceability.
Relevance: Not All Retrieved Chunks Are Equal
Relevance in RAG is two‑sided:
- Context Relevance: Are the retrieved chunks actually useful for answering the user's query? This goes beyond topical similarity—a chunk can be about the right topic but lack the specific answer.
- Answer Relevance: Does the final answer directly and completely address the user's question? A beautiful paragraph that misses the point scores zero.
We measure context relevance by running an evaluation on each retrieved chunk, asking: "Does this chunk contain information that helps answer the user's question?" This creates a per‑chunk relevance score that we average.
For answer relevance, we use a combination of:
- Semantic similarity between the answer and a "golden" answer (for test sets).
- LLM‑based scoring that evaluates if the answer directly addresses all aspects of the question.
- Keyword coverage – ensuring that expected terms from the gold standard are present.
Answer Quality: Completeness and Correctness
Answer quality is the most holistic metric, encompassing:
- Completeness: Does the answer cover all parts of the question?
- Correctness: Are the facts accurate? (This requires a ground‑truth reference.)
- Helpfulness: Is the answer presented in a clear, actionable way?
- Conciseness: Is the answer appropriately verbose—not too short, not too long?
In our evaluation pipeline, we combine automated scores with a lightweight human review process. For automatic scoring, we use:
- BLEU/ROUGE for surface‑level similarity (weak signal, but easy to compute).
- BERTScore for semantic similarity to gold answers.
- LLM‑as‑judge (using a separate, more capable model) to score completeness and correctness on a 1–5 scale.
# Simplified evaluation pipeline
def evaluate_answer(question, answer, gold_answer, context):
return {
'faithfulness': faithfulness(answer, context),
'context_relevance': context_relevance(context, question),
'answer_relevance': answer_relevance(answer, question),
'completeness': llm_score(f"Does this answer cover all aspects of the question? {question} | {answer}"),
'correctness': bertscore(answer, gold_answer),
'overall': weighted_average(...)
}Evaluation Frameworks and Tools
We evaluated several open‑source frameworks and settled on a hybrid approach:
We use RAGAS as the primary automatic evaluator because it's well‑documented and covers the core dimensions. For test sets, we augment with custom evaluators that check for domain‑specific requirements (e.g., citations are in APA format).
Practical Implementation Tips
Here's what we learned about implementing RAG evaluation in practice:
- Start with a small, curated test set. We began with 50 hand‑crafted questions covering different difficulty levels and retrieval scenarios. This set catches most regressions.
- Automate as much as possible. Run the evaluation suite on every PR—if metrics drop, the PR is flagged for review.
- Use LLM‑as‑judge cautiously. We found that using a different model (e.g., GPT‑4 to evaluate Gemini) reduced bias. Always validate with human raters on a subset.
- Track metadata. Log not just the score but also the question type, retrieved chunks, and the LLM's reasoning. This helps debug failures.
- Don't neglect edge cases. Include questions that test adversarial scenarios (e.g., ambiguous queries, out‑of‑domain questions) to measure robustness.
Building a Comprehensive Evaluation Suite
Our final evaluation suite includes three tiers:
- Tier 1 (Fast, Automatic): Retrieval metrics (MRR, Hit@5), RAGAS Faithfulness, Answer Relevance, Context Relevance. Runs on every commit.
- Tier 2 (Slower, Automatic): LLM‑as‑judge for completeness and correctness, BERTScore against gold answers. Runs nightly.
- Tier 3 (Human): Weekly review of 20 samples by domain experts, scoring on a 1–5 rubric. Used to calibrate automatic metrics.
This layered approach gives us rapid feedback with high confidence. When the automatic metrics drift, we have the human data to understand why.
Lessons Learned from Production RAG
Here are the most valuable lessons from operating RAG systems in production:
- Grounding is harder than retrieval. Even with perfect retrieval, LLMs can hallucinate. We added a "citation requirement" that forces the LLM to quote sources, which reduced hallucinations by 63%.
- Relevance is contextual. A chunk can be semantically similar but not answer the question. We improved this by re‑ranking with a cross‑encoder that conditions on the question.
- Answer quality depends on prompt design. Simple changes to the system prompt ("be concise", "use bullet points") had outsized effects on helpfulness scores.
- Evaluate with production traffic. We collect user feedback and run a shadow‑evaluation pipeline that re‑scores real queries with our latest metrics—this reveals gaps that test sets miss.
- Data drift is real. The distribution of user questions shifts over time; our test set must evolve too. We periodically sample production queries to refresh the evaluation set.
Conclusion
RAG evaluation is a multi‑dimensional problem. Focusing solely on retrieval metrics leads to a false sense of security. By expanding your evaluation to include grounding, relevance, and answer quality, you'll build systems that are truly trustworthy and useful.
Start with a curated test set, automate with frameworks like RAGAS, and complement with human evaluation. Iterate on your metrics just as you would on your model—they are the compass that guides your development.
I'd love to hear how you're evaluating RAG systems. Reach out on Twitter or GitHub.