Data leakage happens when information that wouldn't be available at real prediction time somehow influences training — producing a model that looks great in evaluation and then quietly underperforms in production.
Three Common Types of Leakage
| Type | What Happens | Example |
|---|---|---|
| Preprocessing leakage | Fitting a scaler/encoder/imputer on the full dataset before splitting | scaler.fit_transform(X) before train_test_split |
| Target leakage | A feature that's only known after the outcome happens gets included as an input | Using "days_since_cancellation" to predict churn |
| Temporal leakage | Future information leaks into training for a time-based problem | Randomly splitting time-series data instead of splitting by date |
The Preprocessing Leakage Example, Concretely
# WRONG — scaler sees the test set's distribution before the split even happens
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X) # fit on EVERYTHING, including future test rows
X_train, X_test, y_train, y_test = train_test_split(X_scaled, y, test_size=0.2)
# RIGHT — split first, fit only on training data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train) # learns mean/std from train ONLY
X_test_scaled = scaler.transform(X_test) # applies it, doesn't relearnThe Target Leakage Example, Concretely
Predicting whether a customer will churn using a feature called "support_tickets_after_cancellation" is target leakage in disguise — that feature literally can't exist until after the customer has already churned. A model trained with it will show suspiciously high accuracy in testing, then fail completely on new customers where that field is naturally empty.
Why Leakage Is So Dangerous
Leakage doesn't cause an error — it causes a model that looks better than it actually is. Unrealistically high accuracy or R² on a test set is one of the strongest signals to actively suspect leakage, not a reason to celebrate. See Overfitting for a related but distinct failure mode.
How to Prevent It
- Always split into train/test before fitting any preprocessing step
- Use a scikit-learn Pipeline so preprocessing and modeling are bundled and can't accidentally be applied out of order
- For time-series data, split by time (train on the past, test on the future) — never shuffle randomly
- Ask of every feature: "would this value actually be available at the moment I need to make this prediction in production?"
Common Mistakes
- Doing feature selection (e.g. picking the "most correlated" features) using the full dataset before splitting.
- Cross-validating with a preprocessing step fit once on the whole dataset, instead of refitting it within each fold.
Interview Relevance
Q: "Your model gets 99% test accuracy on a problem that's normally hard. What's your first suspicion?" Data leakage — check whether preprocessing was fit before splitting, whether any feature encodes post-outcome information, and whether duplicate rows span the train/test split.
Practice Question
A hospital readmission model includes a feature "discharge_summary_length" — which tends to be much longer for patients who were readmitted. Explain why this could be a leakage risk.
