Model Distillation: Train a Small Model From a Big Model's Outputs

Your prompt works. A frontier model classifies support tickets perfectly, and the bill makes you wince at 50,000 tickets a day. Model distillation is the standard answer: let the big model label or write a pile of examples, then train a small model to imitate it. When it works, you get most of the quality at a fraction of the cost and latency. When it fails, it fails quietly.

I ran a small distillation experiment so the claims here are measured, not folklore. It is a toy (pure numpy, a two-dimensional task), and I'll be clear about what it can and cannot tell you about language models.

What is model distillation?

Distillation trains a small student model to reproduce a large teacher's behaviour. Hinton, Vinyals and Dean (arXiv 1503.02531, abstract read 2026-10-31) described compressing an ensemble's knowledge into a single model by training on the large model's outputs. For LLMs, two variants are common:

• Soft-label distillation. The student matches the teacher's probability distribution. This needs access to probabilities or logits, which most hosted APIs don't give you in full.
• Output distillation. The student is fine-tuned on text the teacher produced (labels, answers, reasoning). This is what most API users can do, and it is the same mechanism as training on synthetic data.

One paper worth knowing for the LLM case is "Distilling step-by-step" (Hsieh et al., arXiv 2305.02301, abstract read 2026-10-31). The authors extract rationales from a large model and use them as extra training signal. They report that on one benchmark a 770M-parameter T5 outperformed a few-shot prompted 540B PaLM using 80 percent of the available data. That is the authors' result on specific 2023 benchmarks. Don't assume it transfers to your task.

When is distillation worth doing?

Run this check before building anything.

Also try cheaper moves first: a smaller hosted model with the same prompt, routing, and prompt caching. The fine-tuning versus prompting post has the fuller decision tree. Distillation adds a training pipeline you must maintain, and it needs redoing when your task changes.

How does a student compare with real labels? A runnable experiment

The setup: two noisy interleaved spirals, two classes. A wide teacher MLP (two hidden layers of 128) is trained on 3,000 real labels. A small student (two hidden layers of 16) is trained under four conditions, and everything is scored on 4,000 held-out test points the student never saw. Five random seeds. The full script is about 100 lines of numpy; here is the core.

def run(seed):
    rng = np.random.default_rng(seed)
    Xte, yte = spirals(4000, 0.03, rng)       # held-out test
    Xlab, ylab = spirals(3000, 0.03, rng)     # teacher's real labels
    Xpool, _ = spirals(3000, 0.03, rng)       # unlabeled pool, labels unused
    Xfew, yfew = Xlab[:60], ylab[:60]         # student's own few real labels

    teacher = train(init([2, 128, 128, 2], rng), Xlab, onehot(ylab), 150, 0.1, rng)
    tprob = softmax(forward(teacher, Xpool)[-1])   # teacher soft labels
    thard = tprob.argmax(1)                        # teacher hard labels
    shape = [2, 16, 16, 2]
    student_few  = train(init(shape, rng), Xfew,  onehot(yfew), 150, 0.1, rng)
    student_hard = train(init(shape, rng), Xpool, onehot(thard), 150, 0.1, rng)
    student_soft = train(init(shape, rng), Xpool, tprob,         150, 0.1, rng)

Output from a real run on 2026-10-31:

condition                                       mean     min     max   (test accuracy, 5 seeds)
teacher                                        0.999   0.999   1.000
student, 60 real labels                        0.655   0.610   0.692
student, teacher hard labels (3000)            0.999   0.998   1.000
student, teacher soft labels (3000)            0.999   0.999   1.000
student, real labels (3000), ceiling           0.999   0.999   1.000
student, teacher wrong where x>0.5             0.784   0.778   0.790
  ...accuracy on x>0.5 region                  0.029   0.021   0.045

What this shows, and what it doesn't:

1. The core idea holds. A student with only 60 real labels reaches about 66 percent. The same student, given 3,000 inputs labelled by the teacher, reaches 99.9 percent, matching one trained on 3,000 real labels. The teacher turned unlabeled inputs into training data. That is the entire value proposition.
2. Soft and hard labels didn't differ here. The teacher is almost always near-certain on this task, so its probabilities carry little extra information. Don't read this as "soft labels never help". On tasks where the teacher is genuinely uncertain, they might. I didn't test that.
3. The student inherits blind spots. In the last two rows I made the teacher wrong in one region (I flipped its labels where x is above 0.5). The student copied that almost perfectly, scoring about 3 percent in that region. Its overall accuracy was still 78 percent, which is exactly the kind of number that looks acceptable in a dashboard. You find the problem only by slicing.
4. The task is trivially learnable. Two dimensions, clean boundary, plenty of inputs. Language tasks have messy inputs and fuzzy labels. The experiment demonstrates mechanics and failure modes, not expected accuracy for your problem.

How do you build the dataset from an LLM teacher?

Use the same shape as the toy: inputs you own, teacher outputs as targets, held-out test kept apart. This collection script is unrun, because I have no API key. The call_teacher function is a placeholder you must fill in.

import json, random

def call_teacher(text: str) -> str:
    raise NotImplementedError("send the same production prompt to your big model")

def build(inputs: list[str], out_path: str, test_frac: float = 0.2, seed: int = 0):
    rng = random.Random(seed)
    rng.shuffle(inputs)
    cut = int(len(inputs) * (1 - test_frac))
    train_in, test_in = inputs[:cut], inputs[cut:]
    with open(out_path + ".train.jsonl", "w") as f:
        for x in train_in:
            f.write(json.dumps({"input": x, "target": call_teacher(x)}) + "\n")
    with open(out_path + ".test_inputs.jsonl", "w") as f:
        for x in test_in:
            f.write(json.dumps({"input": x}) + "\n")

Notes that matter more than the code:

• Split before you call the teacher. Then no test input can leak into training through duplicates. Also dedupe near-identical inputs before the split.
• Label the test set by a human, not the teacher. If the same teacher labels train and test, you measure agreement with the teacher, not correctness. Even 100 hand-checked test items beat 5,000 teacher-labelled ones. The evaluation dataset post covers building that set.
• Filter the teacher's outputs. Drop malformed JSON, refusals and answers that fail your validator. Keep the rejected ones and read a sample, since they show where the teacher struggles.
• Cover the tails. The toy's failure came from a region. Sample inputs across categories, languages and lengths, not just the common case.

Then fine-tune. The LoRA walkthrough shows a small-model fine-tune end to end, and the small language models post covers candidates for the student.

What do vendor terms say about training on outputs?

This is where teams get hurt, so read the terms for your exact product and plan. What I could check on 2026-10-31:

Three cautions. These were summaries from a fetch tool, not my reading of the full contract, so confirm against the live page. "Compete" is undefined in what I saw, and a narrow internal classifier may or may not count. And open-weight teachers have their own licences, which also vary. This is not legal advice. If distillation is part of your product, ask a lawyer, and consider using a teacher whose licence plainly allows it.

What should you check before shipping a distilled model?

• Held-out test set labelled by humans, never seen in training
• Accuracy reported per slice (category, language, input length), not only overall
• A baseline: the teacher's accuracy and a few-shot prompted small model, on the same test
• A plan for the teacher changing: pin the teacher model ID and date in the dataset metadata
• Cost comparison including training, evaluation and maintenance, not only per-call price
• Terms for the teacher checked and written down with the date
• A fallback: route low-confidence student outputs back to the teacher

What I did not verify

• No LLM was used. The experiment is a pure-numpy toy, and its accuracy numbers say nothing about language tasks.
• The data-collection script is unrun.
• Vendor terms were read as fetch-tool summaries, and OpenAI's page was blocked (HTTP 403).
• The two papers were read as abstracts only.