An interviewer asks "your classifier has 99% accuracy, is that good?" and watches for one of two reactions. The weak one is "yes." The strong one is a question back: what's the prevalence, and what does each error cost? This post collects the evaluation-metric questions that show up between classic ML and LLM work, each with an answer outline, a follow-up and a common wrong answer, with every number below computed by a script I ran on 2026-10-09.
These are practice questions I wrote for this site. They aren't leaked and they aren't from any company. The data is simulated or hand-made, and I say so wherever it matters. For related rounds see AI engineer interview questions on RAG, agents and evals, and for building the labelled set these metrics need, building evaluation datasets.
Why is accuracy misleading, and what do precision and recall say instead?
Strong answer outline
- Precision is tp / (tp + fp): of what you flagged, how much was right. Recall is tp / (tp + fn): of what was there, how much you found. F1 is their harmonic mean.
- Accuracy mixes both classes, so a majority-class predictor can look excellent on rare-event data.
- Say the confusion matrix first, then pick the metric to match the cost: missing a fraud case (recall) versus drowning reviewers in false alarms (precision).
I simulated a detector with fixed score distributions (positives from a normal centred at 1.5, negatives centred at 0, both with standard deviation 1) and changed only the prevalence:
prevalence 0.50: positives=100348 ROC AUC=0.855 PR AUC(AP)=0.853
threshold 0.75: acc=0.773 precision=0.774 recall=0.772 f1=0.773
always-negative accuracy=0.498
prevalence 0.01: positives=1989 ROC AUC=0.861 PR AUC(AP)=0.107
threshold 0.75: acc=0.775 precision=0.034 recall=0.777 f1=0.064
always-negative accuracy=0.990
confusion (1% case): tp=1545 fp=44559 fn=444 tn=153452
The same detector has 77.5% accuracy in both worlds. At 1% prevalence its precision is 3.4%: 44,559 false alarms for 1,545 real hits. Meanwhile "always say negative" scores 99.0% accuracy and finds nothing.
Follow-up: Management wants 90% recall. What does it cost? Pick the threshold from the precision-recall curve. On the same 1% data, recall of 0.90 needed a threshold of 0.27 and gave precision of 0.022, which is about 44.6 alerts per true positive. That sentence, an alert budget, is what turns a metric into a decision.
Common wrong answer: "Use F1 because it balances both." F1 weights precision and recall equally, which may not match your costs, and it hides the threshold you chose.
ROC AUC or PR AUC?
Strong answer outline
- ROC AUC is the probability that a random positive scores above a random negative. It's about ranking and doesn't depend on prevalence.
- That is also its weakness on rare classes. In the simulation, ROC AUC was 0.855 at 50% prevalence and 0.861 at 1%, while average precision (the usual PR AUC estimate) fell from 0.853 to 0.107. Same ranking ability, very different queue quality.
- Use ROC AUC to compare scorers when you only care about ranking. Use precision-recall when positives are rare and you act on the top of the list.
- Neither tells you the operating threshold. You still choose that from costs.
Follow-up: Can a model have high ROC AUC and be useless? Yes, exactly the 1% case above.
Common wrong answer: "AUC of 0.86 means 86% accuracy." It doesn't. It's a ranking probability.
How do you evaluate retrieval for RAG: recall@k, MRR, nDCG?
Strong answer outline
- You need queries with labelled relevant documents (or chunks). Without that, none of these can be computed.
- Recall@k: relevant documents found in the top k, divided by all relevant documents. Right for RAG when the generator reads the whole top k.
- MRR: mean of 1 / rank of the first relevant result. Right when one good hit is enough.
- nDCG@k: graded relevance, discounted by
log2(rank + 1), divided by the ideal ordering's score. Right when order and grades matter, for example a reranked list. - State the gain formula you use. Two common conventions exist (linear gain, or
2^rel - 1).
Hand-made toy: four queries, top 5 results each, relevance 0, 1 or 2 per result. Query 3 retrieves nothing relevant, query 4 misses one of its three relevant documents.
recall@1 = 0.250
recall@3 = 0.458
recall@5 = 0.667
MRR = 0.458
nDCG@5 per query: [0.624, 1.0, 0.0, 0.467] mean 0.523
q1: by hand=0.624 sklearn=0.624
q2: by hand=1.000 sklearn=1.000
q4 ideal from retrieved only (sklearn, linear gain) = 0.544
q4 ideal from whole corpus (by hand, 2^rel-1 gain) = 0.467
I checked the by-hand function against scikit-learn's ndcg_score on queries 1 and 2, where they agree. The last two lines are the trap. For query 4, scikit-learn normalises against the documents you passed in, and my hand version normalises against all relevant documents in the corpus, using a different gain. Both are defensible, they give different numbers, and the second is more honest about misses. Whichever you choose, write it down so two runs are comparable.
I computed recall@k by hand from the lists (query 1: 1 of 2 relevant in the top 3, and so on), so you can reproduce each number on paper in an interview.
Follow-up: Recall@5 is 0.667. Is retrieval or generation the problem? If the answer isn't in the top 5, no prompt fixes it. Check retrieval first, then look at reranking, chunking and hybrid search.
Common wrong answer: "Higher recall@k is always better, so raise k." Raising k lifts recall and also feeds the generator more noise and tokens.
What are RAG-specific metrics like faithfulness and answer relevance?
Strong answer outline
- A RAG answer can fail in three separate places: retrieval didn't find the evidence, the model ignored or contradicted the evidence it was given, or the answer is true but doesn't address the question. A single score hides which one.
- The Ragas paper (Es et al., 2023; I read the arXiv abstract on 2026-10-09) frames evaluation along those lines: how well retrieval finds relevant, focused passages, whether the LLM uses them faithfully, and the quality of the generation, and it proposes metrics that don't require human reference answers.
- Reference-free metrics are usually computed by an LLM, which makes them judge metrics with the biases below. Validate them against a small human-labelled sample before trusting a dashboard.
- Keep one metric per failure location, so a regression points to a stage.
Follow-up: Faithfulness is 0.95 but users complain. Why? Faithful to retrieved text isn't the same as correct: if retrieval returned a stale document, a faithful answer is faithfully wrong. Add a retrieval metric and a freshness check.
Common wrong answer: "Faithfulness measures whether the answer is true." It measures support by the provided context, not truth.
Why not just use exact match, BLEU or ROUGE for generated text?
Strong answer outline
- Overlap metrics reward shared words, not shared meaning. They work for narrow tasks with short canonical answers (extractive QA, code that must match a fixed output) and fail for free-form answers.
- Here are three candidates against one reference, scored with exact match and a simple word-overlap F1 that I wrote (lowercased whitespace tokens, no stemming):
paraphrase, correct exact_match=0 token_F1=0.24
wrong number exact_match=0 token_F1=0.88
negation exact_match=0 token_F1=0.94
The reference was "the refund window is 30 days from delivery". The correct paraphrase scored 0.24 while a flipped meaning ("is not 30 days") scored 0.94. This is a deliberately chosen example, and it illustrates the failure mode, not how often it happens. Real BLEU and ROUGE implementations differ in details from my toy F1.
- Alternatives: task-specific checks (does the JSON parse, does the code pass tests), embedding similarity (with its own blind spots), or a rubric-based LLM judge. The right answer for most free-form tasks is a small human-labelled set plus checks you can automate.
Follow-up: When is exact match still the right choice? Closed-form outputs: a class label, a number, a SQL result set, a function-calling argument.
Common wrong answer: "Higher ROUGE means a better summary." It means more overlap with that reference.
How do you validate an LLM-as-a-judge?
Strong answer outline
- The MT-Bench and Chatbot Arena paper (Zheng et al., 2023; abstract read 2026-10-09) reports that strong LLM judges can reach over 80% agreement with human preferences, similar to human-human agreement, and names position, verbosity and self-enhancement biases. The site's LLM-as-a-judge pitfalls post goes deeper into testing for those.
- Label a sample by hand, run the judge on the same items, and compare. Report agreement, then Cohen's kappa, which discounts the agreement you'd get by chance.
- Look at the rare class. If failures are rare, a judge can look fine overall while missing them.
These labels are simulated (no model was called). Humans pass 90 percent of 400 items; the judge copies the human label 92 percent of the time at random:
raw agreement=0.907 Cohen kappa=0.595
always-pass judge: agreement=0.915 kappa=0.000
Look at the second line. A useless judge that always says "pass" has higher raw agreement (0.915) than the simulated judge that is right 92% of the time (0.907, within sampling noise of its true 92%), and a kappa of exactly zero. Raw agreement can't distinguish them, kappa can.
Follow-up: What kappa is good enough? There's no universal cutoff. Decide by cost: if the judge gates releases, you want to know its recall on real failures, and you read sample disagreements by hand.
Common wrong answer: "The judge agrees with humans 90% of the time, so it's validated." Not without checking class balance and the failing class.
How big does an eval set need to be to trust a difference?
Strong answer outline
- A score from n items has sampling error. The bootstrap gives an interval without distribution assumptions: resample the per-item results with replacement many times and take the 2.5th and 97.5th percentiles.
- Simulated system with true accuracy 0.80, three eval sizes:
n= 30: observed acc=0.667 95% bootstrap CI=[0.500, 0.833]
n=100: observed acc=0.750 95% bootstrap CI=[0.660, 0.830]
n=500: observed acc=0.800 95% bootstrap CI=[0.764, 0.834]
With 30 items, the observed 0.667 sat far from the true 0.80 and the interval spanned 33 points. With 500 the interval was about 7 points wide. One run per size, so the exact observed values are luck, but the shrinking width is the point.
- Consequence: don't call a prompt change an improvement on a 30-item set unless the effect is huge. For two systems on the same items, compare per-item (paired) results, which usually gives a tighter test than comparing two separate intervals. I didn't compute a paired test here.
Follow-up: Your eval is 40 items and the new prompt scores 5 points higher. Treat it as noise until you've added items, rerun with several seeds if the system is stochastic, or looked at which items flipped.
Common wrong answer: "Run it once, higher number wins." Especially with sampling temperature above zero.
Run it yourself
The pieces fit in about forty lines of NumPy and scikit-learn. This is the core that produced the retrieval numbers:
import numpy as np
def recall_at_k(rel, total_relevant, k):
return sum(1 for x in rel[:k] if x > 0) / total_relevant
def reciprocal_rank(rel):
for i, x in enumerate(rel, 1):
if x > 0:
return 1 / i
return 0.0
def dcg(rel, k):
return sum((2**x - 1) / np.log2(i + 2) for i, x in enumerate(rel[:k]))
def ndcg(rel, all_relevant_grades, k):
ideal = dcg(sorted(all_relevant_grades, reverse=True), k)
return dcg(rel, k) / ideal if ideal else 0.0
print(recall_at_k([0, 0, 2, 1, 0], 3, 3)) # 0.333...
print(reciprocal_rank([0, 1, 0, 0, 1])) # 0.5
print(round(ndcg([0, 1, 0, 0, 1], [1, 1], 5), 3)) # 0.624
Replace the toy lists with your own retrieval output and relevance labels, then compute the same three numbers per query before averaging. Looking at the per-query values will teach you more than the mean.
How to prepare for this round
- Be able to write the confusion matrix and derive precision, recall and F1 without notes, and say what each costs in a real scenario.
- Memorise one sentence on PR versus ROC under imbalance, with a number: "same ranking, ROC AUC 0.86 at both 50% and 1% prevalence, but average precision drops from 0.85 to 0.11".
- Practise stating the nDCG gain convention and what the ideal ranking is built from.
- Have a validation story for an LLM judge: labelled sample, kappa, failing-class recall, bias checks.
- Know the limits of a small eval set and say how you'd widen it.
What I could not verify
All data is simulated or hand-made. No real model, judge or retriever was run, so none of these numbers describe a real system. The Ragas and MT-Bench papers were read through their arXiv abstract pages (via a summarising fetch), not in full. I didn't test a paired significance test, calibration metrics such as expected calibration error, or perplexity, and I haven't compared my toy token F1 with a standard ROUGE package. The scripts ran with NumPy 2.5.3 and scikit-learn 1.9.1 on 2026-10-09.
