A fine-tuned model needs to be evaluated against the un-fine-tuned baseline, on held-out data, using metrics that reflect the actual goal — "it feels better in a few tries" isn't a substitute for a real, measured comparison.
The Core Comparison
baseline_results = evaluate(base_model, validation_set)
finetuned_results = evaluate(finetuned_model, validation_set)
compare(baseline_results, finetuned_results)
# Did fine-tuning actually improve the metric that matters?
# By how much? Is the improvement worth the fine-tuning cost
# and the added complexity of maintaining a custom model?Skipping this comparison means you can't actually confirm fine-tuning helped — a genuine risk, since fine-tuning can sometimes make a model worse at things it wasn't specifically trained on, even while improving the targeted behavior.
What to Measure
| Dimension | Why Check It |
|---|---|
| Target task performance | Did the specific behavior you fine-tuned for actually improve? |
| General capability regression | Did fine-tuning on a narrow task degrade the model's performance on other things it still needs to do? |
| Consistency across the validation set | Not just average performance, but whether it's reliably good across a range of inputs, not just the easy cases |
Watch for Overfitting
Training accuracy: 98%
Validation accuracy: 71%
→ A large gap suggests the model memorized training examples
rather than learning a generalizable pattern — a sign to
reconsider dataset size/diversity or training configuration,
not necessarily to train longer.Practical Use Case
Before deploying a fine-tuned model to replace a prompting-based system in production, a genuine before/after comparison on a held-out evaluation set — not just a handful of manual spot-checks — is the standard, responsible practice for confirming the investment actually paid off.
Common Mistakes
- Evaluating only on training data, producing an overly optimistic, unreliable quality signal
- Not comparing against the un-fine-tuned baseline, making it impossible to confirm the fine-tuning investment actually helped
- Only measuring the specific targeted behavior, missing regressions in other capabilities the model still needs
Interview Relevance
"How would you know if a fine-tuning project actually succeeded?" — a measured comparison against the baseline model on held-out data, checking both the target improvement and for regressions elsewhere, not a subjective impression.
Practice Question
A fine-tuned model shows 95% accuracy on its target classification task but is now noticeably worse at general conversation. What would you investigate, and what tradeoff does the team need to decide on?
