Speech-to-Text Accuracy Test Protocol: Score Any Tool on Your Own Recordings

To compare speech-to-text tools fairly, record the same script yourself in a few conditions, run each tool on the same files, normalise both texts the same way, and score word error rate (WER) with a script. That takes about an hour for two tools. This post gives you the script to read aloud, the scoring code I ran, and the mistakes that make a home test misleading.

I am deliberately not publishing accuracy figures for Wispr Flow, Superwhisper, Talon, Whisper models or any other tool here. The only transcription I ran was one local model on three synthetic clips (shown below), and the WER sample pairs in the main example are invented to demonstrate the script. For a facts-only comparison of features, see the dictation tool comparison.

What should the recording script contain?

Generic reading passages hide the errors you care about. Write sentences that contain your real trouble spots, then read them naturally. Here is a starter script of invented sentences; replace the names and jargon with yours.

Block A: plain speech (control)
1. Please move the meeting to three thirty on Friday.
2. Call the vendor and ask about the refund.

Block B: names and places
3. Send the invoice to Priya before the end of the day.
4. The client is based in Coimbatore and the contact is Mr Venkataraghavan.

Block C: numbers, dates, money
5. The order total was twelve thousand four hundred and fifty rupees.
6. The release is on the fourteenth of November at nine fifteen.

Block D: your jargon
7. The Kubernetes pod keeps restarting after the deploy.
8. Add a unit test for the parse date function.

Block E: how you really talk
9. (Say one thing in your usual mix of languages, if you code-switch.)
10. (Say one sentence with a deliberate pause and a restart, "so the, no wait, the second one".)

Keep your reference text in a file exactly as you will read it. If you ad-lib, edit the reference to match what you actually said before scoring; otherwise the tool gets blamed for your wording.

How should you record so the test is fair?

• Record once, test many times. Save each recording as a file and run every tool on those files. Live dictation tests where you re-speak for each tool add your own variation to the result.
• Vary one condition at a time. Quiet room with a headset, quiet room with the laptop mic, noisy room with the headset. Three conditions times ten sentences is already 30 clips.
• Same audio format. Convert once, for example ffmpeg -i in.m4a -ar 16000 -ac 1 out.wav, so no tool gets a better file.
• Note the settings. Model name or app version, language setting, any custom dictionary. If one tool had your name in its dictionary and the other did not, write that down.
• Live dictation apps cannot always take files. If a tool only works live, play your recording through speakers near the mic and say so in your notes. It is a worse test, so do not mix those numbers with file-based ones.
• Privacy. Only record words you are happy to send to any cloud tool you test. Do not read real client data.

How do you normalise text before scoring?

Decide the rules first, write them in your notes, and apply them to every tool. My script uses these:

1. Lowercase everything.
2. Replace hyphens with spaces ("stand-up" becomes "stand up").
3. Drop punctuation except apostrophes.
4. Split on whitespace.

It does not convert digits to words. "Three thirty" against "3.30" therefore counts as two wrong words, which is the right call if you want the tool to write words the way you dictated them, and the wrong call if you only care that the time is right. Choose based on your use. If you want numbers treated as equal, add a number-to-words step and say so.

What is the scoring script?

This is a plain edit-distance implementation with no dependencies beyond Python 3.9+. It reads a CSV with columns id,reference,hypothesis and prints substitutions (S), deletions (D) and insertions (I) per row, plus a pooled total.

#!/usr/bin/env python3
"""Word error rate with a plain edit-distance implementation.
Usage: python wer.py pairs.csv     (columns: id, reference, hypothesis)
"""
import csv
import re
import sys


def normalise(text: str) -> list[str]:
    text = text.lower().replace("-", " ")
    text = re.sub(r"[^\w\s']", " ", text)
    return text.split()


def align(ref: list[str], hyp: list[str]) -> tuple[int, int, int]:
    """Return (substitutions, deletions, insertions) from a Levenshtein table."""
    n, m = len(ref), len(hyp)
    d = [[0] * (m + 1) for _ in range(n + 1)]
    for i in range(n + 1):
        d[i][0] = i
    for j in range(m + 1):
        d[0][j] = j
    for i in range(1, n + 1):
        for j in range(1, m + 1):
            cost = 0 if ref[i - 1] == hyp[j - 1] else 1
            d[i][j] = min(d[i - 1][j - 1] + cost, d[i - 1][j] + 1, d[i][j - 1] + 1)
    i, j, s, dele, ins = n, m, 0, 0, 0
    while i > 0 or j > 0:
        if i > 0 and j > 0 and d[i][j] == d[i - 1][j - 1] + (ref[i - 1] != hyp[j - 1]):
            s += ref[i - 1] != hyp[j - 1]
            i, j = i - 1, j - 1
        elif i > 0 and d[i][j] == d[i - 1][j] + 1:
            dele += 1
            i -= 1
        else:
            ins += 1
            j -= 1
    return s, dele, ins


def main(path: str) -> None:
    tot_s = tot_d = tot_i = tot_n = 0
    with open(path, newline="", encoding="utf-8") as f:
        for row in csv.DictReader(f):
            ref, hyp = normalise(row["reference"]), normalise(row["hypothesis"])
            s, d, i = align(ref, hyp)
            n = len(ref)
            print(f"{row['id']:<8} words={n:<3} S={s} D={d} I={i} WER={(s + d + i) / n:.3f}")
            tot_s, tot_d, tot_i, tot_n = tot_s + s, tot_d + d, tot_i + i, tot_n + n
    print(f"POOLED   words={tot_n} S={tot_s} D={tot_d} I={tot_i} WER={(tot_s + tot_d + tot_i) / tot_n:.3f}")


if __name__ == "__main__":
    main(sys.argv[1])

Report the pooled figure, not the average of per-sentence WERs. Averaging per sentence lets one short sentence with one error count as much as a long clean one.

What does the output look like? (invented pairs)

These five pairs are invented by me to show the script working. They are not output from any real tool, and the errors are only plausible guesses.

id,reference,hypothesis
a1,please move the meeting to three thirty on friday,please move the meeting to 3.30 on friday
a2,send the invoice to priya before the end of the day,send the invoice to pria before the end of day
a3,the kubernetes pod keeps restarting after the deploy,the cuban eddies pod keeps restarting after the deploy
a4,call the vendor and ask about the refund,call the vendor and ask about the refund
a5,add a unit test for the parse date function,add a unit test for the parts date function

Real output from running python wer.py invented_pairs.csv on 2026-10-09:

a1       words=9   S=2 D=0 I=0 WER=0.222
a2       words=11  S=1 D=1 I=0 WER=0.182
a3       words=8   S=1 D=0 I=1 WER=0.250
a4       words=8   S=0 D=0 I=0 WER=0.000
a5       words=9   S=1 D=0 I=0 WER=0.111
POOLED   words=45 S=5 D=1 I=1 WER=0.156

I cross-checked the pooled figure against the jiwer library on the same normalised text, and it also gave 0.156. Note what a3 shows: "Kubernetes" heard as "cuban eddies" is one substitution plus one insertion, so a single jargon failure costs two errors. Pooled WER of 0.156 on 45 words means 7 errors. That is a demonstration of arithmetic, not a measurement of anything.

What did a real run look like?

To get at least one real transcription through the scorer, I compared three synthetic clips (Piper text-to-speech reading known sentences) against base.en run through faster-whisper on a 4-core CPU, as in the Whisper batch script post. The reference is the text I fed to the speech synthesiser. Scored with jiwer:

memo_01 raw 0.278 normalised 0.222
memo_02 raw 0.15 normalised 0.0
memo_03 raw 0.059 normalised 0.0

Read this carefully. It is three clips, 55 reference words, of a clean computer voice. It says nothing about human speakers, noise, accents, or any other tool. What it does show is why normalisation matters: memo_02 went from 0.150 to 0.000 purely by lowercasing and stripping punctuation. memo_01 stayed at 0.222 because the model wrote "10.30" for "ten thirty" and "stand-up" for "standup", which is a formatting disagreement that you must decide how to treat.

How do you read the results?

Do not stop at one number. Build a small table per tool and condition.

Then do three checks before choosing.

1. Which errors would you actually fix? One wrong name in an email is worse than three dropped "the" words. Count error types by hand for the worst five sentences.
2. Does the gap survive a second recording? If tool A beats tool B by two points on one day and loses on the next, you have no winner.
3. What does the tool cost you after transcription? Latency, price, whether audio leaves your machine, and whether it writes numbers the way you want. Accuracy is one column. The voice coding workflow covers the cloud-processing trade-off for one dictation app.

What are the common ways this test goes wrong?

• Scoring a tool's cleanup as errors. Some apps remove filler words or fix grammar on purpose. Your reference says "um, so the, no wait", the tool outputs a clean sentence, and WER calls that wrong. Decide whether you score the spoken words or the intended text, and use separate references for each.
• Testing too few words. One sentence flips with a single error.
• Reading too carefully. Read at your real pace and with your real mistakes.
• Letting a custom dictionary hide differences. Test with it off and on.
• Cherry-picking the winner afterwards. Write the normalisation rules and the table before running anything.

If you want to make the notes useful after the test, keep the audio and pairs.csv in a folder. Rerunning the same files when a tool ships a new version is the cheapest regression test you will ever write.