Image Model Text Rendering Comparison: A Method You Can Rerun

If you need words inside an AI image (a shop sign, a poster headline, a slide), the question isn't which model wins a leaderboard but which tool gets your words right, most of the time, on your plan. This post is a protocol for finding out: eight fixed prompts, a scoring sheet, a script that tallies the sheet, and the rules that keep you from fooling yourself. You can finish a first pass in an afternoon.

What this post does not contain: results. I didn't run any image tools for it, so there is no ranking and no claim about any named tool. The only numbers below come from a tally script run on invented sample scores, and they are labelled as invented.

What does the protocol measure?

Three things, per tool:

• Exact match rate: the share of images where the text you read is identical to the target string.
• Character error rate (CER): the edit distance between what you read and the target, divided by the target's length. Zero is perfect; a dropped letter in a ten-letter word is 0.1.
• Group rates: exact match by prompt type (short words, longer phrases, numbers, non-Latin script, multi-line), because a tool can handle OPEN fine and fall apart on a phone number.

Exact match is the number that matters if you'll publish the image without editing it. CER tells you how far off the near misses are, which matters if you'll patch the text afterwards.

What is the prompt set?

Eight target strings, from easy to hard. Each prompt is the same wrapper plus the target text, so the only thing that changes is the text.

The wrapper (untested template; adapt the scene but keep it identical across tools):

A photo of a plain white wall with a hand-painted sign. The sign reads exactly: "[TARGET TEXT]". The text is large, centred and fully visible. No other text anywhere in the image.

Swap in your own words as extra rows. The built-in eight are a starting set; the ones that matter most are your own business name, your tagline and a price. Replace P6 with a number format you actually use, and P7 with your own language if it isn't Hindi. I chose these targets myself and did not validate them against any published benchmark.

What rules keep the comparison fair?

1. Same wrapper, same aspect ratio, same count. At least four images per prompt per tool, generated fresh.
2. No cherry-picking. Score every image generated, including ones you dislike. Don't regenerate a prompt until it works, because that is what you'd do in real use but it isn't what's being measured here. If you want that measure too, record it separately as "tries to first exact match".
3. Record the setup. Tool name, the model or mode as shown in the app, plan, date, resolution, and any style or "magic prompt" options. Tools change under the same name; a result without a date is half a result.
4. Turn off prompt rewriting if you can, or note that it's on. Some tools expand your prompt before generating. That is part of the tool, but it's a variable.
5. Transcribe before you look at the target. Cover the target column, read the image, type exactly what you see including mistakes, capitals and spacing. If there's no text, leave it empty. This avoids reading the right word into a wrong one.
6. Zoom to 100%. Check each character. Thumbnails hide swapped letters.
7. Shuffle the files. If someone else can rename the images randomly and give you the key afterwards, scoring is blinder. Alone, at least score by tool in separate sessions.
8. Treat text as separate from looks. A beautiful sign with a wrong word is a fail for this test.

What does the scoring sheet look like?

One row per image. Save it as scores.csv:

tool,prompt_id,run,transcribed
ToolA,P1,1,OPEN
ToolA,P1,2,OPEN
ToolA,P5,1,Buy two get one free on all notebooks this weekend onl

Columns you can add for your own notes (the script ignores extra columns): date, model_label, plan, notes. For the multi-line prompt, type line breaks as a real newline inside quotes, or use | between lines. The script converts newlines to | and compares against the target written with |.

Human-readable version of the same sheet, if you prefer to score on paper first:

What is the tally script?

Plain Python 3, no installs. It reads scores.csv, compares each transcription with the target, and prints per-tool exact match, a Wilson 95 percent interval for the exact-match rate, mean CER, and group breakdowns. Without an argument it runs on a tiny invented sample so you can see the format.

import csv
import sys
from collections import defaultdict

PROMPTS = {
    "P1": ("short", "OPEN"),
    "P2": ("short", "SALE 50%"),
    "P3": ("medium", "Fresh Coffee Roasters"),
    "P4": ("medium", "Mon-Fri 9am to 6pm"),
    "P5": ("long", "Buy two get one free on all notebooks this weekend only"),
    "P6": ("numbers", "Call 080 4123 9876"),
    "P7": ("hindi", "स्वागत है"),
    "P8": ("multi", "SUMMER MENU|Iced Tea 120|Cold Brew 160"),
}


def edit_distance(a, b):
    prev = list(range(len(b) + 1))
    for i, ca in enumerate(a, 1):
        cur = [i]
        for j, cb in enumerate(b, 1):
            cur.append(min(prev[j] + 1, cur[j - 1] + 1, prev[j - 1] + (ca != cb)))
        prev = cur
    return prev[-1]


def norm(s):
    return " ".join(s.replace("\n", "|").split())


def wilson(k, n, z=1.96):
    if n == 0:
        return (0.0, 0.0)
    p = k / n
    d = 1 + z * z / n
    c = p + z * z / (2 * n)
    m = z * ((p * (1 - p) / n + z * z / (4 * n * n)) ** 0.5)
    return ((c - m) / d, (c + m) / d)


def tally(rows):
    per_tool = defaultdict(list)
    per_group = defaultdict(lambda: defaultdict(list))
    for r in rows:
        group, target = PROMPTS[r["prompt_id"]]
        got, tgt = norm(r["transcribed"]), norm(target)
        exact = got == tgt
        cer = edit_distance(got, tgt) / max(len(tgt), 1)
        per_tool[r["tool"]].append((exact, cer))
        per_group[r["tool"]][group].append(exact)
    return per_tool, per_group


def report(rows):
    per_tool, per_group = tally(rows)
    print("tool | n | exact | 95% interval | mean CER")
    for tool, vals in sorted(per_tool.items()):
        n = len(vals)
        ex = sum(v[0] for v in vals)
        cer = sum(v[1] for v in vals) / n
        lo, hi = wilson(ex, n)
        print(f"{tool} | {n} | {ex}/{n} ({ex / n:.0%}) | {lo:.0%} to {hi:.0%} | {cer:.3f}")
    print()
    print("exact-match by group")
    for tool, groups in sorted(per_group.items()):
        parts = [f"{g}={sum(v)}/{len(v)}" for g, v in sorted(groups.items())]
        print(f"{tool}: " + ", ".join(parts))


SAMPLE = """tool,prompt_id,run,transcribed
ToolA,P1,1,OPEN
ToolA,P2,1,SALE 50%
ToolA,P3,1,Fresh Coffee Roasters
ToolA,P5,1,Buy two get one free on all notebooks this weekend onl
ToolA,P6,1,Call 080 4123 9876
ToolA,P7,1,स्वगत है
ToolB,P1,1,OPEN
ToolB,P2,1,SALE 5O%
ToolB,P3,1,Fresh Cofee Roastrs
ToolB,P5,1,Buy two get one fee on all notebooks this weekend only
ToolB,P6,1,Call 080 4123 9786
ToolB,P7,1,
"""

if __name__ == "__main__":
    if len(sys.argv) > 1:
        with open(sys.argv[1], newline="", encoding="utf-8") as f:
            report(list(csv.DictReader(f)))
    else:
        print("INVENTED SAMPLE DATA (not real results)")
        report(list(csv.DictReader(SAMPLE.strip().splitlines())))

Run it with python3 tally.py scores.csv. I ran the no-argument sample on 2026-10-09. The scores in it are invented to exercise the code (one 5O% with a letter O, a dropped final letter, an empty read). Real output:

INVENTED SAMPLE DATA (not real results)
tool | n | exact | 95% interval | mean CER
ToolA | 6 | 4/6 (67%) | 30% to 90% | 0.022
ToolB | 6 | 1/6 (17%) | 3% to 56% | 0.225

exact-match by group
ToolA: hindi=0/1, long=0/1, medium=1/1, numbers=1/1, short=2/2
ToolB: hindi=0/1, long=0/1, medium=0/1, numbers=0/1, short=1/2

ToolA and ToolB are fictional. Look at the intervals, though: even in this invented case, 4/6 against 1/6 gives overlapping intervals (30 to 90 percent against 3 to 56 percent). That is the point of printing them. With six images per tool you can't say much, and with 32 you can say a bit more. Note that the script's norm function is case-sensitive and counts a different capital letter as an error, which is what you want for signage.

One limit: edit distance counts characters, and for Devanagari a single visible syllable may be several code points (letters plus vowel signs). So a Hindi CER is not comparable with an English one; compare tools within the Hindi group only.

How do you read the results?

• Rank by exact match within a group first, not the overall average. An overall number mixes easy and hard prompts and rewards whoever is good at OPEN.
• Check the interval. If two tools' intervals overlap heavily, say "no clear difference" and add more runs for the groups you care about.
• Look at the failures by hand. Do errors cluster on numbers, on long strings, on lowercase, on a particular letter? That tells you how to prompt around them: shorter text, capitals, text added in an editor.
• Test your real strings next. The sample set is generic. Your name and price list are what count.
• Repeat after updates. Save scores.csv with the date and tool label, and rerun after a model change.

What if no tool passes?

That is a normal outcome for long or non-Latin strings. The fallback is the workflow in the carousel and infographic prompts post: generate the image with no text, then add the words in a design tool. For brand marks, the same reasoning appears in the logo and brand kit workflow. If you are testing tools to produce product imagery, AI product photos for ecommerce covers the label-text problem from the product side, and realistic product ad prompts has prompt formulas to put inside the wrapper.

If you use a vision model as a second reader, see Claude's vision API guide, and keep the human transcription as the record.

What would make this protocol weaker than it looks?

Honest limits, so you don't over-read your own results:

• Eight prompts is a small sample of all text you might want.
• Your transcription is a judgement. Two people can read a smudged letter differently.
• One wrapper scene favours some tools; try a second scene (a poster, a product label) before generalising.
• Tools may change behaviour by time of day, load or silent updates. Dates and reruns are your defence.
• Results describe the tool at the moment you tested it, on your plan, in your region.

Run it, keep the CSV, and make the decision on your own numbers.