LLM-as-a-Judge Pitfalls: Position, Verbosity and Self-Preference Bias

Picture this (hypothetical numbers): your eval says the new prompt wins 71% of head-to-head comparisons. Then you swap which answer goes first and it wins 52%. Nothing about the prompt changed. The judge did.

LLM-as-a-judge pitfalls are well documented and still routinely ignored. This post covers what the foundational paper measured (I read its HTML version on 2026-10-08), a judge prompt template, and a script that computes agreement with human labels, including order-consistency. The small worked example is illustrative: I made up 20 labels to show the arithmetic. It says nothing about any real judge.

What does the LLM-as-a-judge paper say about bias?

The paper is "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" by Zheng et al. (arXiv:2306.05685). The numbers below were read from the HTML version through a summarising fetch tool and re-checked a second time on 2026-10-09; the abstract's "over 80% agreement" matches. Verify against the paper before quoting. These are 2023 models (GPT-4, GPT-3.5, Claude-v1). Current models will differ, which is the reason to run your own checks.

On the positive side, they report GPT-4 agreeing with humans 85% of the time on MT-bench (setup S2, ties excluded), versus 81% human-to-human, and 87% on Chatbot Arena. "Same level of agreement as between humans" is the paper's framing. Those figures are averages over their datasets, not a guarantee for your task.

Mitigations they tried: swap positions and count a win only if it holds in both orders; few-shot examples in the judge prompt (GPT-4 consistency 65.0% to 77.5%, with longer prompts costing roughly 4x per call); chain-of-thought; and reference-guided grading where the judge sees a correct answer.

How do I write a judge prompt that avoids the obvious traps?

An untested template. I haven't run it on a live model; calibrate it before trusting it (next section).

You are grading two answers to the same question.

Question:
<question>{question}</question>

Reference answer (may be empty):
<reference>{reference}</reference>

<answer_1>{a}</answer_1>
<answer_2>{b}</answer_2>

Criteria, in order of importance:
1. Factual correctness against the reference or the question.
2. Whether it does what was asked.
3. Clarity. Length is NOT a quality signal. Do not reward extra words.

First write two or three sentences of reasoning that quote the specific
parts you are judging. Then output one line: WINNER: 1, WINNER: 2 or WINNER: TIE.

Why each part is there:

• Reference answer. The paper's biggest single improvement on math was reference-guided grading. Use it whenever you have ground truth.
• Explicit "length is not quality". A mitigation by instruction, not a cure. Test it with the padding check below.
• Reasoning before verdict. Cheap chain-of-thought, and it gives you text to audit when a verdict looks wrong.
• Neutral labels and tags. Avoid "Assistant A" being first-class; run both orders in code anyway.

Rubric scoring (rate 1 to 5 against stated criteria) avoids pairwise position bias, but introduces its own: judges cluster at 4 and drift between runs. Pairwise with swap is my default for "is B better than A" questions; rubric scores for tracking one prompt over time. For the wider test setup, see Promptfoo's LLM-as-judge assertions.

How do I test my judge for position, verbosity and self-preference?

Three cheap checks, each using data you already have.

1. Swap test. Run every pair in both orders. Report the share of pairs where the verdict flips. If it's high, your judge is mostly reading position.
2. Padding test. Take 20 good answers, append a fluff paragraph or a rephrased copy of part of the answer, and ask the judge to compare the original with the padded version. A fair judge should return ties or prefer the original. Count how often the padded one wins.
3. Self-preference test. Grade a set where you know which model wrote which answer, and compare the judge's win rate for its own model's answers against human labels on the same pairs. Needs a human-labelled set.

How do I calibrate a judge against human labels?

Label 50 to 100 pairs yourself, including hard ones, and compare. The script below takes rows of (human, judge_original_order, judge_swapped_order) where 1 means answer A is better. It computes agreement, order-consistency, agreement on consistent verdicts only, Cohen's kappa, and a Wilson 95% interval.

rows = [  # (human, judge_orig, judge_swapped) ILLUSTRATIVE, hand-made
    (1, 1, 0), (1, 1, 0), (0, 0, 1), (0, 1, 1), (1, 1, 0),
    (0, 0, 1), (1, 0, 0), (0, 0, 1), (1, 1, 0), (0, 1, 0),
    (1, 1, 0), (0, 0, 1), (1, 1, 1), (0, 0, 1), (1, 1, 0),
    (0, 1, 1), (1, 1, 0), (0, 0, 1), (1, 0, 1), (0, 0, 0),
]
n = len(rows)
agree = sum(h == j for h, j, _ in rows)
# in the swapped run the labels refer to presented positions, so flip back
agree_swap = sum(h == 1 - s for h, _, s in rows)
consistent = [(h, j) for h, j, s in rows if j == 1 - s]
cons_agree = sum(h == j for h, j in consistent)
p_o = agree / n
ja = sum(j for _, j, _ in rows) / n
ha = sum(h for h, _, _ in rows) / n
p_e = ja * ha + (1 - ja) * (1 - ha)
z = 1.96
c = (p_o + z*z/(2*n)) / (1 + z*z/n)
h = z * ((p_o*(1-p_o)/n + z*z/(4*n*n)) ** 0.5) / (1 + z*z/n)
print(f"agreement original: {agree}/{n} = {p_o:.0%}")
print(f"agreement swapped : {agree_swap}/{n} = {agree_swap/n:.0%}")
print(f"order-consistent  : {len(consistent)}/{n}")
print(f"agree when consistent: {cons_agree}/{len(consistent)} = {cons_agree/len(consistent):.0%}")
print(f"kappa: {(p_o - p_e)/(1 - p_e):.2f}")
print(f"Wilson 95% CI: {c-h:.2f} to {c+h:.2f}")

Output, run on 2026-10-08:

agreement original: 15/20 = 75%
agreement swapped : 16/20 = 80%
order-consistent  : 15/20
agree when consistent: 13/15 = 87%
kappa: 0.50
Wilson 95% CI: 0.53 to 0.89

Reading it (again, on made-up labels): the judge matches the human 75% of the time in one order and 80% in the other, but flips on 5 of 20 pairs. When it's consistent it agrees 87% of the time (13 of 15). Kappa of 0.50 is moderate once chance agreement is removed, since a coin flip already agrees about half the time. The interval, 0.53 to 0.89, shows why 20 labels can't separate a mediocre judge from a good one.

What I'd do with those numbers: use only order-consistent verdicts for decisions, send the flipped pairs to a human, and expand the label set before gating anything in CI. The golden test set guide covers building the labelled set, and prompt regression tests in CI shows where a judge fits in a pipeline.

When should I not use an LLM judge?

• Exact-match answers (dates, IDs, JSON fields): use code. It's free and has no bias.
• High-stakes decisions with no human check: a judge is a filter, not an authority.
• When the judge is the same model and prompt that generated the output and you have no human-labelled sample.
• Subtle factual claims without a reference: the judge can only be as right as its own knowledge, and the paper's math results show it can miss errors.

Checklist before you trust a judge score

• Both orders run, flip rate reported.
• Padding test done, length-controlled if needed.
• Judge from a different model family than the generator, or self-preference checked.
• Reference answers supplied where ground truth exists.
• At least 50 human labels, interval reported, hard cases included.
• Judge model and prompt version pinned, and re-calibrated when either changes.