K-fold cross-validation is the standard way to run cross-validation: split the data into \(k\) equal parts ("folds"), and rotate which fold serves as the test set — every row gets used for testing exactly once, and for training \(k-1\) times.
Graphical Intuition
Each row runs 5 times, using a different fifth of the data as its test set each time — the red block rotates position across all 5 runs.
Formula and Algorithm
| Step | What Happens |
|---|---|
| 1 | Split the data into \(k\) roughly equal folds |
| 2 | For each fold \(i\): train on all other \(k-1\) folds, test on fold \(i\), record the score |
| 3 | Average the \(k\) recorded scores |
Python Implementation
from sklearn.model_selection import KFold, cross_val_score
from sklearn.linear_model import LogisticRegression
import numpy as np
kf = KFold(n_splits=5, shuffle=True, random_state=42)
model = LogisticRegression(max_iter=1000)
scores = cross_val_score(model, X_train, y_train, cv=kf, scoring="accuracy")
print(scores)
print(f"Mean: {scores.mean():.3f}, Std: {scores.std():.3f}")
# Manually iterating over folds, for full control
for fold_num, (train_idx, test_idx) in enumerate(kf.split(X_train)):
print(f"Fold {fold_num+1}: {len(train_idx)} train, {len(test_idx)} test rows")Choosing k
| k Value | Tradeoff |
|---|---|
| Small (e.g. 3) | Faster, but each fold's training set is smaller and test set larger — higher variance in per-fold scores |
| Common default (5 or 10) | A reasonable, widely-used balance |
| Large (approaching \(n\), "leave-one-out") | Uses almost all data for training each time, but very slow — trains the model \(n\) separate times |
Practical Use Cases
- The standard method for reliably comparing models and hyperparameter choices
- Small-to-medium datasets, where a single held-out validation set would waste too much data
Common Mistakes
- Not shuffling the data before splitting into folds, when the original row order isn't already random (e.g. sorted by date or class) — this can create wildly unrepresentative folds.
- Using plain K-Fold on classification data with imbalanced classes instead of Stratified K-Fold.
Interview Relevance
Q: "Why does k-fold cross-validation use every row for both training and testing?" Rotating which fold serves as the test set means every row is tested on exactly once and trained on \(k-1\) times — this uses the full dataset for both purposes without ever testing on data the current model was trained on, maximizing data efficiency compared to a single fixed split.
Practice Question
With \(k=10\) on a dataset of 1000 rows, how many rows are in each fold's test set, and how many training runs does the full cross-validation process require?
