Chunking Strategies Compared on Real Documents: Fixed, Paragraph and Heading-Aware, Measured

Fixed-size, paragraph and heading-aware chunking are the three splitting strategies most tutorials mention, and almost none of them show what you lose by picking the wrong one. I ran all three on two small document sets on 2026-10-08, scored them with recall@k, and the short result is this: heading-aware chunks led at recall@1 on both sets (and at recall@3 on the manuals, where the margin was large), the margin depended mostly on whether a chunk says what it is about, and the 32-question set is small enough that some of its gaps are one question wide.

If you haven't built a retrieval pipeline yet, start with the PDF chatbot walkthrough. This post is for the stage after that, when answers are mediocre and you're wondering whether the chunker is the reason.

What I tested, and how small it is

Set A (synthetic, mine). 22 Markdown files, 87,189 characters, about 14,500 words. Sixteen are product manuals for a made-up company (eight device types, two models each, for example LS-210 and LS-310 ultrasonic level sensors) with Overview, Specifications, Installation, Calibration, Maintenance, Troubleshooting, Spare parts, Firmware notes and Warranty sections. Six are generic policy files (warranty, returns, calibration service, safety, support, glossary) that deliberately reuse the manuals' vocabulary. A fixed seed generated them, so every fact is known. About half of the facts behind the paraphrase questions are written as the unit or the sensor instead of the model name, because real manuals do that.

Set B (real, public domain). The United States Constitution from Project Gutenberg (ebook 5), 25,937 characters in one file, with Article and Section turned into Markdown headings. The wording is untouched. I wrote 32 questions and copied the answer span from the text, checking by script that each span occurs exactly once.

Questions. Set A has 245: 176 paraphrased questions built from 11 templates, 5 hand-written ones answered by the policy files, and 64 identifier questions built from 4 templates. They are templated, so the effective variety is much smaller than 245 suggests. Set B has the 32 hand-written ones listed below.

Scoring. A chunk counts as a hit only if it comes from the right file and contains the complete gold sentence or span. That is stricter than "contains the right number", and it punishes chunkers that cut sentences. Recall@k is the share of questions with a hit in the top k. Retrieval is dense only: BAAI/bge-small-en-v1.5 through fastembed 0.9.0 (its cache folder was named bge-small-en-v1.5-onnx-Q, so it serves a quantized ONNX build), cosine similarity, Python 3.13, CPU. Chunk sizes are in characters.

How much to trust this: set A is a controlled stress test I built, and I designed the the unit sentences to make context matter, so set A shows the mechanism, not a prediction for your corpus. Set B is real text but one document with 32 questions, where one question is 3.1 points. I'd treat gaps under about 5 points on B as noise, and gaps on A as real for that corpus only.

The three chunkers and the evaluation script

This is the script I used (a trimmed copy of my scratch code, run unchanged on set B to produce the numbers below). It takes a folder of .md files and a JSON list of {question, doc, gold}.

import json, re, sys
from pathlib import Path
import numpy as np
from fastembed import TextEmbedding

norm = lambda s: re.sub(r"\s+", " ", s).strip()
SENT = re.compile(r"(?<=[.!?;])\s+")

def split_long(par, size):
    parts, cur = [], ""
    for s in SENT.split(par):
        if cur and len(cur) + 1 + len(s) > size:
            parts.append(cur); cur = s
        else:
            cur = f"{cur} {s}".strip()
    return parts + [cur] if cur else parts

def pack(pars, size):
    chunks, cur = [], ""
    for p in pars:
        for piece in (split_long(p, size) if len(p) > size else [p]):
            if cur and len(cur) + 1 + len(piece) > size:
                chunks.append(cur); cur = piece
            else:
                cur = f"{cur} {piece}".strip()
    return chunks + [cur] if cur else chunks

def chunk_fixed(docs, size):
    overlap, out = size // 6, []
    for name, raw in docs.items():
        t = norm(raw)
        for i in range(0, len(t), size - overlap):
            out.append({"doc": name, "text": t[i:i + size]})
    return out

def chunk_paragraph(docs, size):
    return [{"doc": n, "text": c} for n, raw in docs.items()
            for c in pack([norm(p) for p in re.split(r"\n\s*\n", raw) if p.strip()], size)]

def chunk_heading(docs, size):
    out = []
    for name, raw in docs.items():
        path, pars = [], []
        def flush():
            out.extend({"doc": name, "text": " > ".join(path) + "\n" + c} for c in pack(pars, size))
        for block in re.split(r"\n\s*\n", raw):
            block = block.strip()
            m = re.match(r"^(#+)\s+(.*)$", block)
            if m and "\n" not in block:          # a markdown heading line
                flush(); pars = []
                path = path[:len(m.group(1)) - 1] + [m.group(2)]
            elif block:
                pars.append(norm(block))
        flush()
    return out

def unit(v):
    return v / np.linalg.norm(v, axis=-1, keepdims=True)

if __name__ == "__main__":
    docs = {p.stem: p.read_text() for p in sorted(Path(sys.argv[1]).glob("*.md"))}
    qs = json.load(open(sys.argv[2]))
    model = TextEmbedding("BAAI/bge-small-en-v1.5")
    qvecs = unit(np.array([next(iter(model.query_embed(q["question"]))) for q in qs]))
    for name, fn in [("fixed", chunk_fixed), ("paragraph", chunk_paragraph), ("heading", chunk_heading)]:
        chunks = fn(docs, 600)
        vecs = unit(np.array(list(model.embed([c["text"] for c in chunks]))))
        hit = lambda c, q: c["doc"] == q["doc"] and norm(q["gold"]) in norm(c["text"])
        ceiling = np.mean([any(hit(c, q) for c in chunks) for q in qs])
        ranks = []
        for q, qv in zip(qs, qvecs):
            top = np.argsort(-(vecs @ qv))[:10]
            ranks.append(next((r + 1 for r, i in enumerate(top) if hit(chunks[i], q)), None))
        rec = lambda k: 100 * np.mean([r is not None and r <= k for r in ranks])
        print(f"{name:9s} chunks={len(chunks):4d} ceiling={100 * ceiling:5.1f}  "
              f"R@1={rec(1):5.1f} R@3={rec(3):5.1f} R@5={rec(5):5.1f}")

What differs between the three:

• Fixed slices the whitespace-normalised text every size - size/6 characters. It ignores everything about the document.
• Paragraph splits on blank lines, then packs whole paragraphs into a chunk up to size. A paragraph longer than size is split on sentence boundaries. Headings are just short paragraphs, so they end up glued to whatever follows.
• Heading-aware does the same packing but never crosses a heading, and prepends the heading path (Kestrel LS-310 Ultrasonic level sensor > Calibration) to every chunk. The size limit applies to the body, so the prefix adds extra characters (26 to 57 on set A, 48 on average).

The output on set B, run exactly as above:

fixed     chunks=  52 ceiling=100.0  R@1= 65.6 R@3= 81.2 R@5= 96.9
paragraph chunks=  57 ceiling=100.0  R@1= 78.1 R@3= 90.6 R@5= 93.8
heading   chunks=  60 ceiling=100.0  R@1= 84.4 R@3= 90.6 R@5= 90.6

Which chunking strategy retrieved the answer most often?

All sizes at 600 characters, dense retrieval, top-k recall in percent. "Avg chars" is the mean body length.

On set A the gap is large: 74.3 percent against 46.9 and 48.6 at recall@3. On set B the order is the same at recall@1, but at recall@3 paragraph and heading-aware tie, and at recall@5 fixed is actually highest (96.9 against 93.8 and 90.6, a difference of one or two questions). I would not read anything into that last cell.

Why heading-aware chunks won on the manuals

I split set A's paraphrase questions by whether the gold sentence names the model (Re-zero the LS-310 every 365 days) or says the unit. Dense recall@3 at 600 characters:

The middle row is the story. If the sentence that holds the answer doesn't say which device it describes, a chunk made from that sentence plus neighbours about "the unit" is nearly identical to the same chunk in fifteen other manuals. The embedding cannot tell LS-210 calibration from PH-310 calibration, because the text doesn't say. Prepending the heading path puts the product and section back into the chunk.

This is not a trick specific to my corpus. Any document where a fragment depends on its heading has the problem: "The limit is 30 days" under a returns heading and under a travel-expenses heading. Anthropic's contextual retrieval write-up tackles the same issue by having a model write a context line for each chunk. A heading path is the free version of that idea when your documents have headings. I did not test the model-written version.

The first row is a warning about my own test. Even when the sentence names the model, fixed and paragraph chunks scored 52.5 and 68.8, which means other chunks outscored them often enough. That is partly the cost of 22 near-duplicate documents, and partly why recall@1 is so low everywhere except for heading-aware chunks.

Does chunk size matter?

Dense recall@3 in percent, three sizes per chunker (the 600-character column repeats the table above):

Three observations, in order of how much I trust them.

1. For heading-aware chunking on set A, size barely mattered. Most sections in those manuals are under about 730 characters (median 532, longest 806), so a limit of 600 or 1200 mostly produces one chunk per section (average body 393 and 517 characters).
2. For fixed and paragraph chunking on set A, bigger chunks scored higher, and I'd be careful with that. My guess, which I did not isolate, is that a 1200-character chunk is more likely to contain a sentence that names the model, and there are only 88 to 93 of them to rank instead of about 184 at 600. You also pay for it downstream: three chunks of 1090 characters is 3,270 characters of context for the LLM, against 1,179 for three heading-aware chunks at 600.
3. On set B, smaller was not worse for heading-aware chunks at recall@3 (93.8 at 300, 81.2 at 1200), but that is 32 questions, and the 300-character setting has its own problem, below.

How to check that your chunker isn't cutting the answer

The ceiling column is the cheapest useful number in this post. It asks: for each question, does any chunk contain the complete answer sentence? If not, retrieval cannot succeed, whatever model you use.

Fixed-size chunks at 300 characters (overlap 50) had a ceiling of 93.5 percent on set A and 96.9 percent on set B. That is 16 of 245 and 1 of 32 questions with no chunk holding the whole sentence. At 600 characters the ceiling was 99.2 and 100.0, and the paragraph and heading-aware chunkers hit 100.0 at every size because they only cut on sentence boundaries when a paragraph is oversized. My gold sentences are short (median 68 characters on set A, 36 on set B), so this effect would be stronger for answers that span a whole paragraph. For your documents, check it with your own answers.

The 32 questions for set B

Each answer span is copied from the Gutenberg text and was checked by script to occur exactly once.

1. How old do you have to be to serve in the House of Representatives?
2. How long is a Senator's term?
3. Who runs a Senate trial when the President is the one being tried?
4. What size majority does the Senate need to convict someone who has been impeached?
5. If the President neither signs nor returns a bill, how long before it becomes law?
6. Which chamber must start bills about taxes?
7. Under what conditions may the right to challenge unlawful detention be suspended?
8. What is the youngest age at which someone can become President?
9. How many years must a presidential candidate have lived in the country?
10. How long does the President's term last?
11. Who is in charge of the armed forces?
12. How many witnesses are needed to convict someone of treason?
13. How many states had to agree before the Constitution took effect?
14. What share of the states must approve a proposed amendment?
15. Can the government hand out noble titles?
16. Who takes over if the President dies in office?
17. What is the longest period Congress can fund an army for at once?
18. Which body is allowed to make money and set its value?
19. On what day does Congress start its yearly session unless it picks another?
20. How must criminal cases be tried, other than impeachment?
21. How many members must be present for a chamber to do business?
22. Which laws rank above state laws?
23. Is a person's religion allowed to be a condition for holding public office?
24. What protection exists for authors and inventors?
25. How large may the national capital district be?
26. Are taxes on goods shipped out of a state permitted?
27. What kind of government does the federal level promise every state?
28. What age and citizenship period apply to Senators?
29. What does the Vice President do in the upper chamber?
30. Who picks the person who chairs the lower chamber?
31. Who nominates ambassadors and Supreme Court judges?
32. What does the oath of office for the President say he will do?

For set A, the paraphrase templates are of the form "How frequently must I recalibrate the LS-310 ultrasonic level sensor?" (gold: Re-zero the LS-310 every 365 days.), and the identifier templates are "What does error E-4417 indicate?", "What triggers E-2208?", "What is part KP-6120?" and "What did firmware 4.18.2 fix?". Those codes are examples of the format, not a particular manual's entries.

What to do with this

1. Count first, tune second. Run the ceiling check. A ceiling under 100 percent is a chunker bug, not a model problem.
2. If your documents have headings, carry them into every chunk. On the set where it mattered, it moved recall@3 by about 25 points. If they don't, find out whether your chunks can still say what they are about.
3. Pick a size from your chunk lengths, not from a blog post. Look at how long your sections are. If a section fits in the limit, one section per chunk is a sensible default.
4. Keep chunk size and k in view together. A bigger chunk scoring better at k=3 may just mean you are giving the model three times the text.
5. Build your own question set. Thirty questions with the answer sentence copied from the source take an afternoon, and building golden test sets describes how to keep them honest.

After chunking, the next two things worth measuring are adding a reranker and adding BM25 for exact identifiers. If you're choosing between embedding models, the embedding models comparison covers candidates, and how RAG works covers the pipeline around all of this.

What I did not test

• Semantic chunking (splitting where embedding similarity drops), model-written context lines, token-based sizes, and parent-child retrieval.
• Other embedding models. Everything here uses one small model, and a larger one may need context less.
• Extraction noise from real PDFs, tables, and scanned pages.
• Retrieval over more than 22 documents, or anything that needs an approximate-nearest-neighbour index.
• Answer quality from an LLM. These are retrieval numbers only.