Quantization Explained: What GGUF 4-bit and 8-bit Models Lose

Quantization stores a model's weights in fewer bits so it fits in less memory and runs faster. The price is accuracy, and the price is not constant: going from 16 to 8 bits costs almost nothing, 4 bits costs something you can measure, and 2 bits breaks small models. This post gives you a memory calculator, real file sizes, a perplexity experiment I ran, and a test harness to check what quantization costs on your own prompts.

How much memory will a quantized model need?

Multiply parameters by bits per weight, then divide by 8 for bytes. I generated this table with a script (decimal GB, weights only):

The formula is one line of Python: params * bits / 8. The table is a lower bound, because the files are bigger than the label says.

Why are GGUF files bigger than the bit width?

I pulled four builds of Llama 3.2 1B from Ollama and divided each file's bytes by the parameter count in its GGUF metadata (1,235,814,432):

So "4-bit" Q4_K_M costs 5.23 bits per weight in practice, and "2-bit" Q2_K costs 3.76. Part of the gap is block scales stored alongside the weights; part is that some tensors are kept at higher precision. I'm inferring that from the numbers rather than from reading the format spec, so treat the cause as likely, not proven. Small models also carry a big embedding table, which inflates the effect: on a 70B model the overhead is proportionally smaller.

Budget with the real file size from ollama list, not the nominal bits.

What about the KV cache?

The weights are not the only thing in RAM. The KV cache stores keys and values for every token in the context, and it grows linearly with context length. For Llama 3.2 1B (16 layers, 8 KV heads, head dimension 64, fp16 cache) the arithmetic is 2 x layers x kv_heads x head_dim x 2 bytes:

32768 bytes per token
ctx   2048:     67.1 MB
ctx   8192:    268.4 MB
ctx  32768:   1073.7 MB
ctx 131072:   4295.0 MB

At the model's full 131,072-token context, the cache alone (4.3 GB) is bigger than the 4-bit weights (0.8 GB). Quantizing the weights does nothing about this, so long-context use on small machines is limited by the cache. The layer and head counts come from the model's GGUF metadata, and I assumed an fp16 cache; check what your runtime actually uses.

What does quantization actually do to the weights?

Take a group of 32 weights, find the largest absolute value, divide everything by max / (2^(bits-1) - 1), round to the nearest integer, and store the integers plus one scale. To use the weights, multiply back. That is round-to-nearest (RTN) quantization with a group size of 32. Real GGUF types are more elaborate, so this is a simplified model of the idea.

I implemented exactly that and applied it to every transformer-block linear layer of SmolLM2-135M-Instruct, leaving embeddings in fp32. Then I measured perplexity on 4,096 tokens of text taken from three posts on this site (so the model has probably not memorised them). Perplexity is how surprised the model is by the text; lower is better. The core of the script:

def quantize_(w, bits, group=32):
    shape = w.shape
    g = w.reshape(-1, group)
    qmax = 2 ** (bits - 1) - 1
    scale = g.abs().amax(dim=1, keepdim=True) / qmax
    scale = torch.where(scale == 0, torch.ones_like(scale), scale)
    q = torch.clamp(torch.round(g / scale), -qmax, qmax)
    return (q * scale).reshape(shape)

Output, as printed:

fp32 perplexity: 17.78
8-bit group32 RTN: perplexity 17.80 (x1.001), relative weight error 0.6%
6-bit group32 RTN: perplexity 17.94 (x1.009), relative weight error 2.3%
5-bit group32 RTN: perplexity 18.92 (x1.064), relative weight error 4.7%
4-bit group32 RTN: perplexity 22.60 (x1.271), relative weight error 10.1%
3-bit group32 RTN: perplexity 95.30 (x5.359), relative weight error 23.4%
2-bit group32 RTN: perplexity 6872914.89 (x386481.699), relative weight error 66.1%

Three lessons. Eight bits is effectively free here. Four bits costs 27 percent more perplexity with this naive method, which is why practical 4-bit formats use better tricks than plain RTN. And below 4 bits it falls off a cliff: 2-bit with only three representable levels (-1, 0, 1) destroys the model.

Caveats that matter: this is a 135M model, which is much more fragile than a 7B one; it is one text sample; and RTN is not what llama.cpp does. I also had a bug in my first run (I shifted the labels twice, which gave a fp32 perplexity of 25,584); the numbers above are from the corrected script, and the sanity check was that 8-bit lands within 0.1 percent of fp32.

What does quality loss look like in practice?

Perplexity is abstract. Here is what the same four Llama 3.2 1B builds did on 15 tasks with automatic checks, at temperature 0 with a fixed seed. The checkers are strict about format. Note that n=15 is tiny and the machine was shared, so speeds are omitted.

Two things stand out. Q8_0 and Q4_K_M score the same as fp16 within noise, yet 4-bit changed 6 of 15 answers word for word. Different text is not the same as worse text: for example, Q4_K_M answered "Bonjour" where fp16 answered "Bonjour.". And a single task, counting the r's in "strawberry", flipped from wrong to right at Q4_K_M, which is luck, not a reason to prefer 4-bit.

Q2_K failed differently. It stopped following format instructions. Asked "What is the capital of Australia? Answer in one word." it replied "The capital of Australia is Canberra." (right, but my checker wanted one word). Asked for JSON it opened with "Dedicated Professionals" before the braces. Asked to sort 42, 7, 19, 3 it returned "3, 7, 19, 42, 19", adding a number. It also answered 253 for 17 times 23 (the answer is 391). So low-bit degradation shows up first as sloppy instruction following, then as wrong facts.

One more unexpected detail: Q2_K said 91 "is not a prime number", the correct answer, while every higher-precision build said yes. I'd call that a coin flip, not a win for 2-bit.

How do I test quantization on my own prompts?

Do not trust anyone's benchmark, including mine, for your task. Write 20 to 50 prompts from your real work, each with a checker function, and run them against two quantization levels. The harness I used is a small loop over the Ollama API:

import json, re, sys, urllib.request

def chat(model, prompt):
    body = json.dumps({
        "model": model,
        "messages": [{"role": "user", "content": prompt}],
        "stream": False,
        "options": {"temperature": 0, "seed": 7, "num_predict": 120},
    }).encode()
    req = urllib.request.Request("http://localhost:11434/api/chat", data=body,
                                 headers={"Content-Type": "application/json"})
    with urllib.request.urlopen(req) as r:
        return json.load(r)["message"]["content"]

TASKS = [
    ("mult", "What is 17 * 23? Answer with just the number.",
     lambda s: re.search(r"(?<![\d.])391(?![\d.])", s) is not None),
    ("sort", "Sort these numbers ascending as a comma-separated list, nothing else: 42, 7, 19, 3",
     lambda s: re.sub(r"\s", "", s).strip(".") == "3,7,19,42"),
]

for model in sys.argv[1:]:
    ok = sum(bool(check(chat(model, prompt))) for _, prompt, check in TASKS)
    print(model, f"{ok}/{len(TASKS)}")

Run it as python3 check.py llama3.2:1b-instruct-q8_0 llama3.2:1b-instruct-q4_K_M. My full version has 15 tasks and prints each answer. Practical advice for reading results:

1. Prefer checks that are exact (a regex, a JSON schema, a unit test) over eyeballing.
2. If 4-bit passes everything 8-bit passes on your set, take 4-bit and keep the memory.
3. If 4-bit fails tasks 8-bit passes, look at whether the failures are formatting (fixable with a clearer prompt or structured output) or reasoning (move up a precision level or a model size).
4. Re-run with a different seed or temperature 0.7 a few times; one run at temperature 0 can flatter or punish a build.

For picking a model size in the first place, the beginner Ollama guide walks through installing and sizing. For which models exist, see open-weight models in 2026.

So which quantization should I pick?

My rule, based on the numbers above and flagged as a judgement:

• Plenty of memory: Q8_0. Near-lossless, about half the size of fp16.
• Normal laptop: a Q4-family build of the largest model that fits. A bigger model at 4-bit generally beats a smaller one at 8-bit, though I did not test that across sizes here.
• Very tight memory: drop to a smaller model before dropping below 4 bits.
• Fine-tuning: train on a higher-precision base and quantize afterwards, or use adapters on a quantized base; see the LoRA walkthrough.

What I could not verify

I did not run models above 1.2B, did not measure real RSS or tokens per second on a quiet machine, and did not read the llama.cpp quantization specification, so the explanation of K-quants is hedged. The perplexity experiment uses simple RTN on a 135M model, not GGUF quantization; I re-ran that script on 2026-10-09 and it reproduced the table exactly. All runs were on 2026-10-08 with Ollama 0.40.1.