Model evaluation is how you find out whether a trained model is actually good — not on the data it was trained on, but on data it has never seen, using a metric that genuinely matches the problem you're solving.
The Evaluation Workflow
| Step | Purpose |
|---|---|
| 1. Split data into train/validation/test | Ensure evaluation reflects genuinely unseen data |
| 2. Train on the training set | Fit model parameters |
| 3. Tune using the validation set (or cross-validation) | Compare candidate models/hyperparameters without touching the test set |
| 4. Evaluate once, finally, on the test set | Report an honest, unbiased estimate of real-world performance |
Classification vs Regression — Different Metrics Entirely
| Classification | Regression | |
|---|---|---|
| What's being measured | How often predicted classes match actual classes | How close predicted numbers are to actual numbers |
| Core tool | Confusion Matrix | Residuals (actual − predicted) |
| Common metrics | Accuracy, Precision, Recall, F1, ROC-AUC | MSE, RMSE, MAE, R² |
Why "Just Check Accuracy" Isn't Enough
A single metric is often misleading in isolation — accuracy alone hides how a model performs on a rare but important class (see Imbalanced Data), and R² alone hides whether errors are evenly distributed or concentrated in a few bad predictions. Real evaluation looks at several complementary metrics together, chosen to match what actually matters for the specific business problem.
Practical Use Cases
- Comparing candidate models fairly before choosing one to deploy
- Deciding whether a model is good enough to ship, or needs more work
- Communicating a model's real-world reliability to stakeholders honestly
Common Mistakes
- Evaluating on the training set instead of held-out data — this measures memorization, not generalization.
- Picking a metric without considering the business cost of different error types (a missed fraud case vs a false alarm are rarely equally costly).
- Touching the test set more than once during model development, quietly turning it into a second validation set and inflating the final reported number.
Interview Relevance
Q: "How do you decide which metric to optimize for a new classification problem?" Start from the business cost of each error type — false positives and false negatives are rarely equally costly — and choose a metric (precision, recall, F1, or a custom cost-weighted score) that reflects that asymmetry, rather than defaulting to accuracy.
Practice Question
You're building a model to flag potential cancer cases for further testing. Would you prioritize precision or recall, and why?
