Contact centre QA usually means a reviewer listens to a sample of calls per agent and fills in a form, which means most calls are never heard. An LLM can read every transcript, which is tempting and dangerous in equal measure: a scorer that is quietly more lenient than your best reviewers will tell you everything is fine.
This post gives you a rubric, a judge prompt, and the part most guides skip: a calibration step that measures whether the model agrees with humans. The prompts are untested templates, since I had no model access with transcripts in this session. The calibration script was run, but on invented ratings for invented calls, so its numbers show how the method works and say nothing about any real model. Scores from your own calls are the only ones that count.
What belongs in a call QA rubric?
Keep to criteria you can point at in a transcript. "Showed empathy" is vague; "acknowledged the problem in the first two turns without blaming the customer" is checkable. A starting rubric for a support or booking call, with three levels per row:
| Criterion | 0 Fail | 1 Partial | 2 Pass |
|---|---|---|---|
| Opening | No greeting, no identification | Greeting but no name or no reason | Greeting, company, and name or AI disclosure |
| Problem understood | Agent answers a different question | Needs the customer to repeat once | Restates the issue correctly |
| Accuracy | Gives wrong or invented information | Vague or unverified information | Correct, and from an approved source |
| Process followed | Skips a required step (verification, consent) | Required step done late or unclear | All required steps done in order |
| Resolution or next step | None | Next step unclear | Clear outcome, owner and timing |
| Tone | Rude, dismissive or interrupts repeatedly | Flat or scripted | Polite, adapts to the customer |
| Close | Abrupt | Summary missing | Summary, check for other needs, goodbye |
Two rules make the table work. Each cell needs a short written anchor your reviewers agree on; the table above is a template, so edit the anchors to your process. And mark a few criteria as critical: for example, giving wrong information, skipping identity verification, or ignoring a request to stop recording. A call that fails any critical item fails overall regardless of total, which stops a friendly tone from hiding a serious error.
The judge prompt
This is an untested template. It forces evidence, a score per criterion, and an explicit "cannot tell" path, which cuts down on invented judgements.
You are a quality reviewer for customer calls. Score the transcript against the rubric. Use only what is in the transcript.
RUBRIC
[PASTE THE TABLE ABOVE WITH YOUR ANCHORS]
CRITICAL ITEMS (any 0 on these fails the call): Accuracy, Process followed.
RULES
- For each criterion give: score (0, 1 or 2), a verbatim quote from the transcript as evidence, and one sentence of reasoning.
- If the transcript does not contain evidence for a criterion, score "NA" and say why. Do not guess.
- Do not reward length or politeness over correctness.
- Do not score what the customer did.
- Mask nothing yourself; the transcript is already masked.
OUTPUT: JSON only, in this shape:
{"criteria": [{"name": "...", "score": 0|1|2|"NA", "evidence": "...", "reason": "..."}], "critical_fail": true|false, "summary": "two sentences"}
TRANSCRIPT
[TRANSCRIPT]
Use structured output or a schema validator so malformed JSON never reaches your dashboard. Requiring a verbatim quote lets you check each score mechanically: if the quote is not a substring of the transcript, reject that score and re-run. That one check catches a good share of the invented evidence you'd otherwise miss. For more failure modes of model graders, read LLM-as-judge pitfalls.
What does a scored call look like?
Here is an invented, masked transcript (illustrative, written by me, not a real call):
AGENT: Thanks for calling Greenleaf Dental, this is Meera. How can I help?
CUSTOMER: I want to move my appointment from Tuesday.
AGENT: Sure. Can I confirm your name and date of birth?
CUSTOMER: Ravi Kumar, fourth of March, 1990.
AGENT: Thanks. You'd like to move Tuesday's cleaning. Which day works instead?
CUSTOMER: Thursday afternoon.
AGENT: Thursday at four is free. Shall I book it?
CUSTOMER: Yes.
AGENT: Done. You'll get a text confirming Thursday at four. Anything else?
CUSTOMER: No, thanks.
AGENT: Thanks Ravi, goodbye.
A reasonable human score: Opening 2, Problem understood 2, Accuracy 2 (assuming the slot was real), Process 2 (verification done before changes), Resolution 2, Tone 1 (efficient but flat), Close 2. If a judge scores Tone 2 and Close 2 here, that's a defensible disagreement. If it scores Process 2 on a version of this call where the agent skipped verification, that's a miss that matters. The point of calibration is to find out how often the second kind happens.
How do you calibrate the model against human reviewers?
Take 30 to 50 calls, include plenty of bad ones, and have two reviewers score them independently on the final 0/1/2 per call (or per criterion). Then have the model score the same calls. Compare:
- Exact agreement: how often two raters give the same score.
- Cohen's kappa: agreement corrected for chance. Raw agreement flatters you when most calls are the same score.
- Quadratic-weighted kappa: treats a 0 versus 2 miss as worse than a 0 versus 1 miss.
- Bias: does the model score higher than humans on average? Leniency is the common failure.
Here is the script I ran on 2026-10-09 with Python 3. The ratings are invented for 12 calls to show the method.
from collections import Counter
def kappa(a, b, weighted=False):
n = len(a)
cats = [0, 1, 2]
w = lambda i, j: 1 - (abs(i - j) / 2) ** 2 if weighted else float(i == j)
po = sum(w(x, y) for x, y in zip(a, b)) / n
ca, cb = Counter(a), Counter(b)
pe = sum(w(i, j) * (ca[i] / n) * (cb[j] / n) for i in cats for j in cats)
return (po - pe) / (1 - pe), po
With three rating lists (human A, human B, model) of 12 calls each, it printed:
human A vs B: exact agreement 0.75, kappa 0.62, quadratic-weighted kappa 0.82
A vs model: exact agreement 0.67, kappa 0.47, quadratic-weighted kappa 0.75
B vs model: exact agreement 0.50, kappa 0.20, quadratic-weighted kappa 0.50
model minus human A (positive = model more lenient): 4
calls where model scored higher than both humans: [3, 6, 8, 11]
How to read it. The two humans agree better with each other than the model agrees with either, which is the normal pattern: your humans are the ceiling for the model, not the floor. Against human A the model looks acceptable at 0.67 exact agreement, but against human B it drops to 0.50, so the verdict depends on which reviewer you ask; that is a rubric-clarity problem as much as a model problem. And the model is more lenient: it totals 4 points above human A, and on calls 3, 6, 8 and 11 it scored higher than both humans, including call 8, where both humans gave a 0 and the model gave a 1. A lenient scorer that never fails a bad call is the dangerous kind. The list in the last line is your review queue: read those calls and decide who is right.
With 12 calls these figures are noisy; a single call changes them a lot. Use 30 to 50 minimum, and report the confusion matrix, not just one number.
How do you improve the scorer once you see disagreements?
Work from the disagreement list, not from a hunch.
- Read the calls where the model is higher than both humans. Find the shared pattern (often tone or process).
- Add the missing anchor to the rubric row, with a short real example, and rerun.
- If one human disagrees with everyone, fix the rubric wording, not the model.
- Run the model three times on the same calls and compare. Large run-to-run swings mean the criterion is underspecified.
- Keep a frozen set of calls and rerun after every prompt or model change. This is the same discipline as building golden test sets for any LLM feature.
Don't tune the prompt on the calibration calls and then report the agreement on them. Keep a held-out set you never look at while editing.
What should the scores be used for?
Be deliberate. Good uses: sorting which calls a human should listen to first, spotting training topics across many calls, and catching process misses (a critical item flagged on a call nobody would have sampled). Risky uses: ranking individual agents for pay, or disciplinary action, from model scores alone. Use human confirmation there, and tell your staff how calls are scored.
Privacy matters too. Transcripts carry personal data. Mask names, phone numbers, account numbers and addresses before sending them to a model, check your provider's retention and training terms, and make sure your callers were told calls are recorded and analysed. The legal side is in call recording consent and AI disclosure in India, and the answers there start with "ask a professional".
If you run an AI voice agent rather than human agents, the same rubric works with two changes: replace Tone with "stayed within scope and handed off when it should have", and add a critical item for "stated it was an AI". Scripts to score against are in voice agent call scripts for appointment booking. For India-specific support prompts, see AI prompts for customer support and BPO teams.
Rollout checklist
- Rubric with three levels, written anchors and two or three critical items.
- Judge prompt requiring a verbatim quote per score, NA allowed, JSON schema enforced.
- Reject any score whose quote is not found in the transcript.
- 30 to 50 double-scored calls, enriched with failures, plus a held-out set.
- Report exact agreement, kappa, weighted kappa and average bias per criterion.
- Human review queue built from calls where the model is more lenient than both humans.
- Masking, retention and notice settled before any real transcript is sent.
- Rerun calibration after every prompt or model change.
