Evaluating RAG: Datasets, Metrics, and LLM Judges

A Retrieval-Augmented Generation system can produce a fluent answer while failing in several independent ways. It may retrieve the wrong evidence, omit a required document, misread correct context, answer a different question, or invent a value that never appeared in any source. A single “accuracy” score cannot explain these failures or identify the component that needs to change.

Reliable evaluation treats RAG as a pipeline. Retrieval and generation receive separate metrics. Exact values are checked with deterministic code. Free-form answers are judged against explicit criteria. Every model, prompt, chunking, and ranking change is evaluated on the same dataset before release.

The objective is not to produce an impressive dashboard number. It is to create a repeatable decision system that answers three operational questions:

Did the system retrieve the evidence required to answer the question?
Did the generated answer use that evidence correctly?
Did a proposed change improve quality enough to justify its cost and latency?

Separate Retrieval from Generation

RAG has at least two primary failure surfaces.

The retrieval layer selects context. It can return irrelevant chunks, miss a required passage, or retrieve an outdated version. The generation layer interprets that context. It can ignore evidence, combine facts incorrectly, or add unsupported claims.

These failures require different fixes. If the correct invoice row never enters the context, changing the answer prompt will not repair retrieval. If the correct row is present but the model reports the wrong unit price, increasing vector top-k will not repair generation.

Evaluation should therefore preserve the boundary between these stages.

Retrieval Metrics

Context Precision

Context precision measures how much of the retrieved context is relevant to the question. Low precision means the prompt contains noise. Irrelevant chunks consume tokens and can distract the generator even if the correct evidence is also present.

Precision should also weigh rank order - a relevant chunk buried at position 8 while irrelevant chunks occupy positions 1-7 indicates a weaker retriever than raw relevance proportion suggests.

For a question about the return conditions of product PRD-482, a clause about warranty duration may be topically related but not answer the question. Retrieval evaluation must distinguish general similarity from answer-bearing evidence.

Context Recall

Context recall measures whether all evidence required for the answer was retrieved. This is especially important for questions that require multiple sources or table rows.

Suppose a request asks for the total cost of three line items. Retrieving two correct rows produces high-looking precision but incomplete recall. The generated total cannot be correct because one required input never reached the model.

Ranking Metrics

When the dataset identifies the expected evidence, ranking can also be evaluated with metrics such as recall at k, Mean Reciprocal Rank, or normalized discounted cumulative gain. These metrics reveal whether the correct chunk exists in the candidate set and how high it appears.

The value of k should reflect the actual pipeline. A correct result at rank 40 is useful only if the candidate generator passes at least 40 items to the next stage. If the application sends only the top 5 chunks to the model, recall at 5 is operationally more meaningful than recall at 100.

Generation Metrics

Faithfulness

Faithfulness asks whether claims in the answer are supported by the provided context. A response can be fluent, relevant, and still unfaithful if it introduces a price, date, condition, or conclusion that the retrieved evidence does not support.

Faithfulness should be evaluated claim by claim when possible. A multi-sentence answer may contain three supported statements and one unsupported conclusion. A single binary label hides that distinction.

Answer Relevancy

Answer relevancy asks whether the response addresses the user’s actual request. An answer may repeat correct information from the context while avoiding the requested comparison or calculation.

For example, a response that lists product descriptions does not answer a question about which product has the lower unit price. The content may be grounded but irrelevant to the requested decision.

Exact and Structured Checks

Not every answer needs a language-model judge. Invoice numbers, product codes, quantities, unit prices, dates, currencies, and totals should be normalized and compared through deterministic code.

Structured checks can verify:

Exact identifier match
Numeric equality within an explicit tolerance
Correct currency and unit
Correct number of returned line items
Correct association between product code and price
Arithmetic consistency
Required source references

Code-based evaluation is cheaper, faster, and more reproducible than asking a model to judge facts that already have a formal representation.

Open-source evaluation frameworks make these checks easier to operationalize. Ragas and DeepEval provide ready-made implementations of faithfulness, answer relevancy, context precision, and context recall; TruLens and Arize Phoenix expose closely related prebuilt RAG evaluators for groundedness or faithfulness, answer relevance, and retrieval relevance. Their defaults are useful starting points, but the judge model, rubric, and thresholds still need calibration against the target domain.

Metric Independence Matters

A matrix separating retrieval quality from generation quality metrics

Retrieval and generation metrics are not interchangeable. Context recall can be perfect while faithfulness is poor. The model may receive the correct invoice row and still report a value from its own prior knowledge or confuse two adjacent columns.

The reverse can also occur. A model may remain perfectly faithful to incomplete context. Its answer accurately reflects the retrieved chunks but omits a required product because retrieval missed it.

A useful diagnostic matrix is shown below. This is a simplified heuristic for diagnosis, not a formal or standardized framework.

Context quality	Answer quality	Likely problem
Low	Low	Chunking, retrieval, filtering, or indexing
High	Low	Prompt, context presentation, model, or validation
Low	Apparently high	Possible benchmark gap or unsupported lucky answer
High	High	Expected behavior

This separation turns evaluation from scorekeeping into diagnosis.

Designing a Golden Dataset

A golden dataset is a curated set of test cases with known expectations. Each example should contain more than a question and reference answer.

A practical record may include:

{
  "id": "price-lookup-014",
  "question": "What was the March unit price of PRD-482?",
  "expected_answer": {
    "product_code": "PRD-482",
    "unit_price": 125.50,
    "currency": "EUR"
  },
  "required_sources": [
    "invoice-2026-0148-line-04"
  ],
  "category": "exact_lookup",
  "answerable": true
}

The dataset should represent the real distribution of queries while deliberately including difficult cases:

Exact identifier lookups
Semantic questions
Multi-document comparisons
Multi-hop relationship questions
Aggregations
Ambiguous requests
Multi-turn references
Outdated and current document versions
Missing information
Access-controlled content

Categories allow regressions to be localized. A new embedding model may improve semantic questions while harming exact product-code retrieval. A single aggregate score can conceal that trade-off.

Testing the Ability to Refuse

Some golden questions should be intentionally unanswerable from the available corpus. The expected behavior is a clear statement that the information is unavailable, not a plausible completion.

Unanswerable cases test:

Whether retrieval confidence thresholds work
Whether the prompt permits abstention
Whether the model invents missing prices or policies
Whether citations actually support the response
Whether a tool failure is exposed rather than hidden

Negative examples are essential because a system evaluated only on answerable questions can appear accurate while remaining unsafe whenever evidence is missing.

Abstention can be measured as a classification task. Define abstention precision as correct abstentions / all abstentions, which measures how often a refusal was justified. Define abstention recall as correct abstentions / all unanswerable cases, which measures how many questions that should have been refused were actually refused. These should be reported with answer coverage, defined as answered cases / all cases, and accuracy on answered cases so that a system cannot improve its refusal score merely by declining every request.

Keeping the Dataset Alive

A continuous loop that turns reviewed failures into golden evaluation cases

A golden dataset is not a one-time benchmark. It must evolve with the corpus and production traffic.

New documents should introduce new questions. Otherwise the benchmark measures only whether old queries still work and cannot detect failures on recently added information. Production incidents and user-reported errors should become regression cases after the correct answer and source are verified.

Every test case needs ownership and provenance. Reference answers should be updated only through review, with a record of why the expectation changed. If the underlying business data changes, historical and current questions should remain distinct rather than silently overwriting one another.

The dataset should also avoid contamination. If the exact test questions are repeatedly used to tune prompts or train models, reported scores may stop representing general performance. A development set can guide iteration while a separate holdout set provides a more honest release check.

Human Feedback as an Evaluation Loop

Explicit feedback such as a negative rating should create a review item rather than remain an isolated analytics event. A reviewer can classify the cause:

Retrieval failure
Generation failure
Tool or calculation failure
Missing or stale source data
Ambiguous user request
Incorrect golden expectation

After the issue is corrected, the question, verified answer, and source become a regression test. This closes the loop from production failure to permanent coverage.

Human review can also annotate partial quality. A response may use the correct source but omit a requested field. Capturing structured failure reasons is more useful than storing only a thumbs-up or thumbs-down label.

Using LLM-as-Judge Carefully

Language-model judges are useful for qualities that are difficult to encode as exact rules, such as faithfulness, completeness, clarity, and relevance. They are not automatically reliable.

A judge requires:

A precise rubric
The question
The retrieved context
The generated answer
Clear score definitions
Examples of acceptable and unacceptable judgments
A requirement to explain the evidence behind the score

Before automated judging is trusted, a representative sample should be scored independently by humans and by the judge. Agreement should be measured by category, not only as one overall percentage. A judge may be reliable for faithfulness but inconsistent on writing quality or partial completeness.

Published results illustrate why metric-specific calibration matters. In the original RAGAS paper, the automated metrics agreed with resolved human preferences on 95% of WikiEval pairwise faithfulness comparisons and 78% of answer-relevance comparisons. These figures are accuracies on one 50-question evaluation setup, not universal guarantees for another domain, rubric, or judge model.

Low agreement indicates that the rubric, prompt, or score definitions need refinement. The judge should be recalibrated and tested again. Human review can then focus on uncertain cases, disagreements, high-risk categories, and samples used to monitor judge drift.

Deterministic values should remain outside this process. A code comparison should verify that 125.50 EUR matches the expected price. The judge can evaluate whether the surrounding explanation is faithful and relevant.

Experimenting with Quality, Latency, and Cost

The trade-off between answer quality, latency, and operating cost

Retrieval configuration is a multi-objective optimization problem. Increasing the number of retrieved chunks may improve recall but increase latency, token use, and noise. Adding a reranker can improve ordering but adds inference cost. A stronger generator may improve faithfulness while increasing response time.

Each candidate configuration should run on the same golden dataset and produce a comparison table:

The first row below is an illustrative, realistic example rather than a benchmark result. Production reports should replace it and every measured placeholder with observations from the target environment.

Configuration	Context recall	Context precision	Faithfulness	P95 latency	Cost/query
Dense, top 5 (illustrative)	0.81	0.74	0.88	340 ms	$0.004
Dense + BM25, top 10	measured	measured	measured	measured	measured
Hybrid + reranker	measured	measured	measured	measured	measured

The table should use observed values from the target environment. Decisions then follow explicit rules. If cost decreases while quality remains within the accepted range, the change can proceed. If quality drops below a category threshold, lower cost alone is not sufficient.

Routing can improve this trade-off. Simple lookups may use fewer retrieval stages and a smaller model. Complex comparisons may justify broader retrieval, reranking, and a stronger generator. Each route should have its own service-level and quality targets.

Debugging When the Correct Context Is Present

When retrieval evaluation confirms that the required evidence reached the generator but the answer remains wrong, debugging should move to the generation layer.

Grounding Instructions

The system prompt should explicitly require the model to use only the supplied evidence and to state when the evidence is insufficient. A vague request to “use the following information” does not clearly prohibit unsupported additions.

Context Position and Structure

Long prompts can cause relevant passages in the middle to receive less attention. The strongest evidence should be placed prominently, chunks should have clear boundaries, and source metadata should distinguish documents and table rows.

Model and Sampling

Repeating the test with lower temperature or a stronger model helps separate sampling variability from structural prompt problems. If the same error persists across models and deterministic decoding, the context representation or instructions are more likely to be responsible.

Citation Enforcement

Requiring source references for every factual claim makes unsupported content easier to detect. Citations should be verified programmatically: the referenced source must exist, and the cited passage must contain the value or claim.

Output Validation

A final faithfulness check can compare generated claims with the context. Structured answers should also pass schema and arithmetic validation. This layer is more expensive than fixing a prompt or context layout, so it should complement rather than replace earlier controls.

Release Gates

Evaluation becomes operational when it controls releases. Changes to chunking, embeddings, prompts, models, routing, or index parameters should trigger the relevant test suites.

A release gate can enforce:

Minimum retrieval recall by category
Minimum faithfulness and answer relevancy
Exact-match thresholds for structured answers
Maximum hallucination rate on unanswerable questions
P95 latency and cost budgets
No regression on critical test cases

Failed gates should block deployment until the regression is understood. The purpose is not to prevent every score fluctuation; it is to prevent unexamined quality loss from reaching users.

Risk-Based Thresholds and Score Aggregation

Not every query category has the same consequence. A minor wording difference in a product summary is not equivalent to an incorrect unit price or a missing contract restriction. Evaluation thresholds should reflect the risk of the output.

Critical exact-value routes can require deterministic equality, complete source coverage, and successful arithmetic validation. Semantic summaries may use graded relevance and faithfulness scores. Unanswerable questions can require a high abstention rate and zero unsupported numeric claims.

For example, a risk-based gate might require a 100% deterministic match on critical exact-value routes, including identifiers, units, and arithmetic. A lower-risk semantic-summary route might require faithfulness >= 0.85 and answer relevancy >= 0.80. These are example starting points, not universal defaults; they should be calibrated with human review, observed error costs, and score distributions from the actual application.

A release report should therefore present both aggregate and category-level results. Macro averages give every category equal influence, while traffic-weighted averages reflect current usage. Both can be misleading if used alone. A rare but critical calculation category can disappear inside a strong overall score.

Hard gates protect these cases:

No regression on designated critical queries
Exact product and invoice identifiers must match after normalization
Numeric answers must agree with deterministic tool output
Multi-source questions must retrieve every required source
Unsupported numeric claims are not permitted
Access-control tests must always pass

Confidence intervals or repeated runs are useful when generation is nondeterministic. A one-point change based on a small dataset may be noise rather than an improvement. Test-set size, category coverage, and run variance should accompany reported scores.

Evaluation should also detect improvements that merely move errors between categories. Increasing top-k may improve recall while reducing context precision and faithfulness. A stronger model may improve answer quality but violate latency targets. A new router may reduce cost while sending a small set of aggregate questions to the wrong tool.

The decision rule should be defined before examining results. For example, a candidate may require no critical regressions, a minimum quality improvement in its target category, and compliance with cost and latency budgets. Predefined rules reduce the temptation to justify a favored configuration after the experiment.

Quality thresholds are operational contracts. They translate abstract evaluation scores into explicit conditions under which a system is considered safe to release.

Evaluation as an Engineering Discipline

RAG evaluation is most useful when it connects measurements to actions. Low context recall points toward ingestion, chunking, embeddings, filters, or ranking. Low faithfulness with strong context points toward prompting, context presentation, model behavior, or validation. High latency with unchanged quality points toward unnecessary stages or oversized models.

Golden datasets, deterministic checks, calibrated language-model judges, and human feedback provide complementary evidence. Together they replace subjective debates with repeatable experiments and turn production mistakes into permanent tests.

A RAG system cannot be improved reliably until its failures are separated, labeled, and measured at the layer where they occur.

Evaluating RAG: From Golden Datasets to LLM-as-Judge

Separate Retrieval from Generation