RAG Evaluation Metrics for Interviews: RAGAS Explained With a Worked Example

If an interviewer asks how you'd evaluate a RAG system, "I'd use RAGAS" is where the questions start, not where they end. The useful answer is that RAG evaluation metrics split the pipeline in two: context precision and context recall judge what retrieval returned, faithfulness and answer relevancy judge what the model did with it. Each one is a ratio over yes/no verdicts, so you can compute it by hand, and this post does that on a toy example before turning to the questions.

The questions are practice questions I wrote for this site. None are from a real company's interview. I read the Ragas paper and the current Ragas documentation on 2026-10-09 (details at the end), and the numbers below come from a script I ran that day.

What did the RAGAS paper actually measure?

The paper, Ragas: Automated Evaluation of Retrieval Augmented Generation (Es, James, Espinosa-Anke and Schockaert), proposes reference-free evaluation of RAG pipelines, meaning no human-written ground truth answers. It defines three quality aspects:

• Faithfulness: the answer is grounded in the retrieved context. The LLM first extracts statements from the answer, then checks each against the context. The score is supported statements divided by all statements.
• Answer relevance: the answer addresses the question. An LLM generates candidate questions from the answer, and the score is the mean cosine similarity between their embeddings and the original question's.
• Context relevance: the context contains only what is needed. An LLM picks the sentences crucial to the question, and the score is extracted sentences over total sentences.

To test this, the authors built WikiEval: 50 Wikipedia pages about events after the model's training cutoff, with human judgments of the three aspects. In the paper's Table 1, agreement with humans (pairwise accuracy) was 0.95 for faithfulness, 0.78 for answer relevance and 0.70 for context relevance, against 0.72, 0.52 and 0.63 for a "GPT Score" baseline. The authors say context relevance was the hardest dimension, especially for long contexts. That's one paper's experiment on 50 pages with one judge model (gpt-3.5-turbo-16k at the time), so treat it as evidence that the method is plausible, not as a guarantee for your data.

How do the current RAGAS metrics differ from the paper?

The documentation has moved on. On the pages I read, the metrics you will meet are:

Two details worth knowing. The docs page for answer relevancy notes that the score usually falls between 0 and 1 but is not guaranteed to, since cosine similarity ranges from -1 to 1. And the docs mark the older SingleTurnSample API as legacy, to be deprecated in version 0.4 in favour of a ragas.metrics.collections API, so code from older tutorials may not match the version you install. I downloaded the 0.4.3 wheel and read the source for two formulas: faithfulness divides faithful statements by the number of statements and returns NaN when there are none, and context precision adds 1e-10 to its denominator, which is why the docs show 0.9999999999 instead of 1.0. I did not install or run Ragas, because it calls an LLM and I had no key here.

Can you compute the RAG metrics by hand?

Yes, once the judge's calls are fixed. In the real library an LLM extracts claims and decides, per claim or per chunk, yes or no. In this script I write those verdicts by hand, so the only thing under test is the arithmetic. The knowledge base is a made-up store policy:

Four cases: Q1 retrieves [c5, c1, c4] for a two-part question and misses c2, and its answer invents "within 3 days". Q2 is a clean success. Q3 retrieves the right chunk in second place. Q3b has the same retrieval as Q3, but the answer is a true sentence about the warehouse that ignores the question. Here is the script, trimmed to the parts that matter:

import math, re
from collections import Counter

STOP = {"a", "an", "the", "do", "i", "to", "of", "for", "is", "are", "my", "you",
        "your", "have", "how", "what", "when", "can", "will", "get"}

def context_precision(v):          # v = per-chunk verdicts in rank order
    hits, total = 0, 0.0
    for k, x in enumerate(v, 1):
        hits += x
        total += (hits / k) if x else 0
    return total / sum(v) if sum(v) else 0.0

def frac(flags):                   # supported claims / all claims
    return sum(flags) / len(flags) if flags else float("nan")

def toks(s):
    return [w.rstrip("s") for w in re.findall(r"[a-z0-9]+", s.lower()) if w not in STOP]

def cos(a, b):
    ca, cb = Counter(toks(a)), Counter(toks(b))
    dot = sum(ca[w] * cb[w] for w in ca)
    n = math.sqrt(sum(x*x for x in ca.values())) * math.sqrt(sum(x*x for x in cb.values()))
    return dot / n if n else 0.0

# name, question, chunk verdicts by rank, reference-claim flags, answer-claim flags,
# questions an LLM might regenerate from the answer (hand-written, illustrative)
CASES = [
    ("Q1 return window + refund timing",
     "How long do I have to return an item and when will I get my refund?",
     [0, 1, 0], [1, 0, 0], [1, 1, 0],
     ["How many days do I have to return unused items after delivery?",
      "How fast are refunds processed?", "What is the return window?"]),
    ("Q2 gift card returns", "Can I return a gift card?",
     [1, 0], [1, 1], [1, 1],
     ["Can gift cards be returned?", "Can a gift card be exchanged?", "Are gift cards refundable?"]),
    ("Q3 shipping time", "What are your shipping times?",
     [0, 1, 0], [1], [1],
     ["How long does standard shipping take?", "What are the shipping times?",
      "How many business days is shipping?"]),
    ("Q3b faithful but off-topic", "What are your shipping times?",
     [0, 1, 0], [1], [1],
     ["When is the warehouse closed?", "Is the warehouse open on public holidays?",
      "What days is the warehouse closed?"]),
]

print(f"{'case':32s} ctxP  ctxR faith ansRel")
for name, q, chunks, ref, ans, regen in CASES:
    rel = sum(cos(q, g) for g in regen) / len(regen)
    print(f"{name:32s} {context_precision(chunks):4.2f}  {frac(ref):4.2f}  {frac(ans):4.2f}  {rel:5.2f}")

print("relevant chunk first :", context_precision([1, 0]))
print("relevant chunk second:", context_precision([0, 1]))
print("no claims            :", frac([]))

Output from running it with Python 3 on 2026-10-09:

case                             ctxP  ctxR faith ansRel
Q1 return window + refund timing 0.50  0.33  0.67   0.30
Q2 gift card returns             1.00  1.00  1.00   0.61
Q3 shipping time                 0.50  1.00  1.00   0.56
Q3b faithful but off-topic       0.50  1.00  1.00   0.00
relevant chunk first : 1.0
relevant chunk second: 0.5
no claims            : nan

Read it row by row, because this is how you'd talk through it in an interview:

• Q1: context recall is 0.33 because only one of three reference claims (the 30-day window) is in the retrieved chunks. Nothing in the generation could have fixed that. Faithfulness is 0.67 because two of three answer claims are supported, and the unsupported one is the invented "3 days". Two separate failures, one in each half of the pipeline, and a single end-to-end score would blur them.
• Q3 vs Q2: the same retrieved chunk scores 1.00 in first position and 0.50 in second. This matches the Ragas docs, where an irrelevant chunk placed first halves the context utilization score. Recall is unaffected by order.
• Q3b: faithfulness is 1.00 and the answer is useless. Only answer relevancy catches it, and it drops to 0.00.
• The answer relevancy column is crude. I used bag-of-words cosine instead of an embedding model, so the absolute values (0.56 for a perfect shipping answer) are not what Ragas would print. The docs' own example prints 0.9165 for a short correct answer. Use my column for the ordering only.
• The empty case: with zero claims, the ratio is undefined. The Ragas faithfulness source I read returns NaN there, so your aggregation code needs to handle it.

With a longer version of the script (not shown), I also computed the classic id-based retrieval numbers for the first three cases, using hand-labelled relevant ids (c1 and c2 for Q1, c3 for Q2, c5 for Q3): mean recall@k was 0.83 and mean reciprocal rank (MRR) was 0.67. These need no LLM at all, and they're worth reporting alongside the LLM-judged metrics. How to build the question set is in building evaluation datasets.

How do you diagnose a low score?

A heuristic table, not a law:

Six RAGAS interview questions with answer outlines

1. What does faithfulness measure, and how is it different from correctness? Outline: claims in the answer checked against the retrieved context, as a ratio. It measures grounding, not truth. Give the example of an outdated chunk: faithful and wrong. Say that correctness needs a reference or a human. Follow-up: "Would you ship on faithfulness alone?" No, pair it with recall against a reference. Common wrong answer: "Faithfulness is whether the answer is factually correct."

2. Explain context precision and context recall to someone who knows classic precision and recall. Outline: recall asks whether the needed information was retrieved (via reference claims); precision, as documented, is rank-aware: it averages precision@k over the ranks where a relevant chunk appears. Show the order-sensitivity numbers above. Follow-up: "Which would you improve first?" Recall, since a generator cannot use what was never retrieved. Common wrong answer: "Precision is the fraction of retrieved chunks that are relevant," which ignores rank.

3. Can you trust an LLM judge? Outline: the verdicts are model calls, so they inherit model error and bias. Label a sample by hand and compare. Report agreement beyond chance. Fix the judge model and version when comparing runs. The paper itself notes that the judge struggled most on context relevance. The sibling post LLM-as-judge pitfalls goes further. Follow-up: "How many labels do you need?" There's no universal number; enough that a disagreement pattern shows up, and I'd start at 20 to 30. Common wrong answer: "GPT-4 level models are accurate enough that you can skip this."

4. Why does answer relevancy not catch hallucinations? Outline: it regenerates questions from the answer and compares them with the original. A fluent, confident, wrong answer to the right question scores well. Q3b is the reverse case: grounded but off-question. Follow-up: "What does it depend on that people forget?" The embedding model and the question-generating LLM.

5. How would you use these metrics in CI? Outline: freeze a labelled question set, pin the judge model, run before and after each change, and compare to a baseline rather than a fixed threshold, since I haven't found a source giving universal pass marks. Run more than once because judge calls are noisy. Gate on large drops, and read the failing rows. Common wrong answer: "Fail the build under 0.8."

6. Faithfulness is 0.95 but users say answers are wrong. What do you check? Outline: (a) the answers are faithful to the wrong chunks, so check context precision and recall against a reference; (b) the eval questions do not match real queries; (c) the corpus is stale or contradicts itself; (d) the judge is too lenient. Sample 20 real failures and classify them.

How do you check your own judge?

If you use an LLM judge for any of these, agree on a small hand-labelled sample first. This script (Python 3, run on 2026-10-09) uses invented labels purely to show the calculation:

def agreement(human, judge):
    n = len(human)
    observed = sum(h == j for h, j in zip(human, judge)) / n
    p_h, p_j = sum(human) / n, sum(judge) / n
    expected = p_h * p_j + (1 - p_h) * (1 - p_j)
    return observed, (observed - expected) / (1 - expected)

human = [1, 1, 0, 1, 0, 1, 1, 0, 1, 1]   # invented, illustrative
judge = [1, 1, 1, 1, 0, 1, 1, 1, 1, 1]
print(agreement(human, judge))

raw agreement = 0.80, Cohen's kappa = 0.41
judge says supported 9/10, human says 7/10

Eighty percent sounds fine until you see that the judge said "supported" nine times in ten. Kappa corrects for the agreement you'd get by chance, and 0.41 is a prompt to read the disagreements. Cohen's kappa is a standard statistic and not something Ragas defines.

How do you run the real thing?

The docs show a pattern like this for faithfulness (copied in shape from the documentation page, unrun by me, and the import path is the part most likely to change between versions):

from openai import AsyncOpenAI
from ragas.llms import llm_factory
from ragas.metrics.collections import Faithfulness

client = AsyncOpenAI()
llm = llm_factory("gpt-4o-mini", client=client)
scorer = Faithfulness(llm=llm)
result = await scorer.ascore(
    user_input="When was the first super bowl?",
    response="The first superbowl was held on Jan 15, 1967",
    retrieved_contexts=["The First AFL-NFL World Championship Game was played on January 15, 1967."],
)
print(result.value)

Before you do, write the 20 questions and hand-score five of them using the arithmetic above. If your hand scores and the library scores disagree wildly, you have found something to explain in the interview, which is better than a clean dashboard you can't defend. For where this fits in a full design answer, see the RAG chatbot system design walkthrough and the broader evaluation questions list. Classic metrics (precision, recall, F1) are covered in ML evaluation metrics for interviews.

What I could not verify

• I did not run Ragas, so I have no real library outputs. Library scores differ from my script because the real claim extraction and verdicts come from an LLM.
• The documentation pages I read show a footer date of December 9, 2025, and the docs may have changed since. Read via a local HTML-to-text conversion on 2026-10-09; check exact wording on the live pages.
• The 0.4.3 wheel was read, not installed. Newer versions may differ.
• The toy corpus, labels and regenerated questions are invented for illustration. None of the numbers say anything about real RAG systems.
• I give no accuracy figure for any judge model beyond the paper's WikiEval table.

Sources: the Ragas paper (arXiv 2309.15217v2) and the Ragas documentation pages for faithfulness, context precision, context recall and answer relevancy at docs.ragas.io, all read 2026-10-09.