Splitting data into training and test sets is the foundational discipline that makes model evaluation honest — without it, you're only ever measuring how well a model memorized the data it already saw.
Why One Split Isn't Always Enough
A single train/test split answers "how did the model do on this specific random 20% of data?" — but that number can shift meaningfully depending purely on which rows happened to land in the test set, especially for smaller datasets. This variance is exactly why cross-validation exists — it averages performance across several different splits instead of trusting just one.
The Three-Way Split
| Set | Purpose | Touched How Often |
|---|---|---|
| Train | Fit model parameters | Every training run |
| Validation | Compare models/hyperparameters, tune decisions | Repeatedly, during development |
| Test | Final, honest performance estimate | Exactly once, at the very end |
See Train-Test Split in Python for the scikit-learn implementation, including stratify and random_state.
Practical Use Cases
- Every supervised ML project, without exception — this is the non-negotiable baseline of honest evaluation
Common Mistakes
- Repeatedly checking test-set performance while tuning, quietly turning it into a second validation set.
- Splitting after fitting any preprocessing step, leaking test-set information — see Data Leakage.
Interview Relevance
Q: "Why use a three-way split instead of just train/test?" Repeatedly checking model performance against the test set while tuning hyperparameters effectively turns the test set into part of training — a separate validation set lets you tune freely, keeping the test set's final number an honest, untouched estimate.
Practice Question
Explain why a single 80/20 train/test split on a dataset of only 50 rows might give a misleadingly optimistic or pessimistic accuracy estimate.
