LoRA Fine-Tuning Walkthrough on a Small Model, With Real Loss and Eval

LoRA fine-tuning trains a tiny set of extra weights on top of a frozen model, so you can teach a small model a format or style on a laptop-class machine. This walkthrough does that end to end on a 135M-parameter model, prints the real loss, scores the model before and after, and shows where the result falls apart. The honest summary: it worked on the phrasings it saw and only half-worked on one it hadn't. That second number is the one that matters.

Everything below was run on 2026-10-08 on a shared, CPU-only 4-core container.

When is LoRA worth it?

Before touching code, run the decision. LoRA earns its keep when all of these are true:

1. You need the same output shape every time (a fixed line format, a JSON schema, a house style).
2. A strong model with a good prompt already does it, but it's too slow or expensive at your volume, or you must run on a small local model.
3. You have at least a few hundred good examples, ideally from real traffic.
4. You can measure success automatically.

If the problem is missing knowledge (your docs, your prices), that's retrieval; see fine-tuning vs RAG vs prompting. If a few-shot prompt already works on the model you can afford, stop there. The longer argument is in fine-tuning vs prompting in 2026. For interview-style questions on rank, alpha and QLoRA, see LoRA and QLoRA interview questions.

What task am I teaching it?

Support-ticket triage into one strict line. Given a messy ticket, output exactly:

category=<refund|shipping|login|billing>; urgency=<high|low>; order=<number>

I generated the data synthetically with a script, so there's no real customer text and no privacy issue. The catch is that synthetic data is cleaner than reality. Each category has four phrasing templates and each ticket gets a random 5-digit order number and an urgent or relaxed closing sentence. Example generated ticket and target:

#84606 still hasn't arrived and the tracking page is stuck. This is urgent.
category=shipping; urgency=high; order=84606

The split matters more than the model:

• Train: 400 tickets using templates 0, 1 and 2 of each category.
• Seen test: 40 new tickets (fresh order numbers and combinations) from the same three templates per category.
• Unseen test: 40 tickets from template 3 of each category, a phrasing the model never trains on.

Without that third split I'd have reported a flattering number.

How do I set up LoRA with PEFT?

Libraries as installed: torch 2.14.1 (CPU), transformers 5.19.0, peft 0.21.2. The base model is HuggingFaceTB/SmolLM2-135M-Instruct. The adapter config:

from peft import LoraConfig, get_peft_model

cfg = LoraConfig(r=8, lora_alpha=16, lora_dropout=0.0,
                 target_modules=["q_proj", "v_proj"], task_type="CAUSAL_LM")
model = get_peft_model(model, cfg)
model.print_trainable_parameters()

It printed:

trainable params: 460,800 || all params: 134,975,808 || trainable%: 0.3414

Rank 8 on the query and value projections is a common starting point, not a tuned choice. I didn't try other ranks or target modules, so I can't tell you they'd do better or worse.

The training loop is plain PyTorch so there's no Trainer API to fight. The one detail that matters is masking the prompt so the loss counts only the answer tokens:

def encode(d):
    p = tok(prompt(d["text"]), add_special_tokens=False)["input_ids"]
    c = tok(" " + d["target"])["input_ids"] + [tok.eos_token_id]
    return torch.tensor(p + c), torch.tensor([-100] * len(p) + c)

opt = torch.optim.AdamW([p for p in model.parameters() if p.requires_grad], lr=2e-4)

Batch size 8, 2 epochs, 100 steps, learning rate 2e-4, padded batches with -100 labels on padding.

What did the training loss do?

Printed every 10 steps (the loss of the current batch, so it's noisy):

epoch 1 step 1 loss 1.7936
epoch 1 step 10 loss 1.0928
epoch 1 step 20 loss 0.4442
epoch 1 step 30 loss 0.2355
epoch 1 step 40 loss 0.1161
epoch 1 step 50 loss 0.0962
epoch 2 step 60 loss 0.0555
epoch 2 step 70 loss 0.0514
epoch 2 step 80 loss 0.0474
epoch 2 step 90 loss 0.0431
epoch 2 step 100 loss 0.0268
train seconds: 516

Loss fell from 1.79 to 0.03 in 100 steps and 516 seconds (3 CPU threads, shared machine). That looks great, and it tells you almost nothing about generalisation. A loss that low on a templated task means the model memorised the template space. It did not say whether the model would cope with something new, which is what the next section measures.

Did it actually work? Before and after

I scored three setups with the same checker: the full output line must match exactly (and I report each field separately). The base model got the instruction in its chat template. For the "3-shot" baseline I added three training examples as prior turns. n=40 per split, so one example is 2.5 points.

On seen phrasings the LoRA scored 100% on category, 82.5% on urgency and 100% on order number. Zero-shot, the base model never produced the format: for the unseen sample it just echoed "The courier is marked as delivered but my doorstep is empty."

The headline is real: a 1.9 MB adapter took exact-match from 17.5% to 82.5% on in-distribution tickets, against the only prompt baseline I tried (3-shot). Then the unseen row. On a phrasing it had never seen, category accuracy dropped from 100% to 45%, which is worse than you'd want from a four-way classification. Urgency and order number transferred much better (90% and 95%), because those cues (an urgent closing sentence, a number) look the same across templates. The category cue changes with the wording, and 3 templates per category weren't enough for the model to learn the concept instead of the sentences.

That's the central lesson: fine-tuning on narrow, templated data teaches the template. The fix is more varied data, not more epochs.

What are the limits of this experiment?

• Weak baseline. A 135M model with 3 examples is a hard baseline to be fair to. I did not test a larger model with a prompt, and for a simple classify-and-extract task a stronger model probably handles this with a few-shot prompt alone. If so, that's the right answer and LoRA wasn't needed.
• Synthetic, templated data. Real tickets are messier, so don't extrapolate these percentages.
• Small n. 40 per split, one seed, one run. The unseen result could move by 10 points or more on a rerun.
• Hyperparameters not tuned. One configuration, no sweep.
• Raw prompt for the tuned model. The adapter was trained and evaluated with the plain prompt format, the baselines used the chat template. That is the natural setup for each but it isn't a perfectly controlled comparison.

How should I evaluate before and after on my own data?

A checklist that I'd follow, in order:

1. Split first. Before generating or collecting training data, set aside tests that differ from the training distribution (new phrasings, new customers, new months).
2. Write the checker before the training script. Exact match, JSON schema validity, a regex or a unit test.
3. Score the base model with the best prompt you can write, including few-shot. That's your real baseline.
4. Score the tuned model on both an in-distribution and a shifted split. Report both.
5. Read 20 failures by eye. If they cluster on one cue, add data that varies that cue.
6. Only then consider rank, alpha, target modules and learning rate.

If you also want to check that the adapter didn't damage general ability, add a handful of unrelated prompts to the set and compare before and after. I didn't do that here.

When should I stop and use prompting or RAG?

Stop and switch if any of these show up:

• The prompt-only baseline on a model you can afford already passes your checker. Done.
• Your failures are factual (wrong price, missing policy). Add retrieval; see how RAG works.
• Your data changes weekly. Retraining every time costs more than editing a prompt or an index.
• You can't gather several hundred varied examples. Few-shot prompting needs ten.
• The tuned model must run on a laptop, offline, at volume, with a fixed format. That is where LoRA wins; to see how small you can go, read about quantization and the small-model landscape.

What to try next

Train the same setup with 12 templates per category instead of 3 and re-run the unseen split. My expectation, flagged as a hypothesis I did not test, is that the category gap closes substantially. Then run the same task with a larger model and a plain 3-shot prompt and compare cost and accuracy. That comparison, not the loss curve, is what tells you whether to ship an adapter.

What I could not verify

This is one run on one seed. I did not run a GPU, QLoRA, a larger base model or an alternative rank. I didn't merge the adapter into the base weights or export it to GGUF for Ollama, so I can't vouch for that path. The data is synthetic.