A decision guide tying together MSE, MAE, RMSE and R² — each already covered individually, this note is about which one (or which combination) to actually report for a given regression problem.
The Decision Table
| Situation | Best Metric | Why |
|---|---|---|
| Communicating typical error to a non-technical audience | MAE or RMSE | Both are in the target's original units, directly interpretable |
| Large errors are much worse than small ones | RMSE or MSE | Both penalize large errors disproportionately more |
| Data has legitimate outliers you don't want dominating the score | MAE | Scales linearly with error size, not quadratically |
| Comparing model quality across different targets/datasets | R² | Scale-independent — always between (roughly) 0 and 1 |
| Comparing models with different numbers of features | Adjusted R² | Penalizes features that don't genuinely improve the fit |
All Four, Computed Together
from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score
import numpy as np
y_true = [52, 58, 62, 68, 75]
y_pred = [51.8, 57.4, 63.0, 68.6, 74.2]
mse = mean_squared_error(y_true, y_pred)
rmse = np.sqrt(mse)
mae = mean_absolute_error(y_true, y_pred)
r2 = r2_score(y_true, y_pred)
print(f"MSE: {mse:.3f}, RMSE: {rmse:.3f}, MAE: {mae:.3f}, R²: {r2:.4f}")
# MSE: 0.480, RMSE: 0.693, MAE: 0.640, R²: 0.9924Reporting all four together — rather than picking just one — gives a complete picture: RMSE and MAE communicate typical error magnitude in interpretable units, their gap (0.693 vs 0.640) hints at error consistency, and R² places the result in scale-independent context.
A Worked Decision, End to End
# Business context: predicting delivery time in minutes. A 5-minute error is mildly
# annoying; a 60-minute error is a serious service failure -- large errors matter MORE
# This asymmetry argues for RMSE (or MSE) as the primary metric, not MAE,
# since RMSE will more heavily penalize the rare, serious 60-minute misses
# that MAE would treat as "just 12x a typical 5-minute error," proportionallyWhy R² Alone Is Rarely Sufficient
R² tells you relative improvement over a naive mean-baseline, but says nothing about whether the absolute error size is acceptable for the business. An R² of 0.95 sounds excellent, but if the remaining 5% of unexplained variance still translates to errors of ₹50 lakh on house price predictions, that may be completely unacceptable in practice — always pair R² with an absolute metric.
Common Mistakes
- Reporting only R² without any absolute error metric, hiding whether the actual error magnitude is business-acceptable.
- Choosing MSE/RMSE by default without considering whether MAE's outlier-robustness better fits the specific data and business context.
- Comparing RMSE values across models trained on differently-scaled or transformed targets (e.g. one on raw price, one on log-price) without converting back to a common scale first.
Interview Relevance
Q: "You have two regression models: one with lower MAE, another with lower RMSE. How do you decide which is actually better?" It depends on whether large errors are disproportionately costly for this specific problem — if a few big misses matter much more than many small ones, prioritize the lower-RMSE model; if you want a metric robust to occasional outliers and want a fair sense of typical error, prioritize the lower-MAE model. There's no universal answer independent of the business context.
Practice Question
You're evaluating a house price prediction model. Would you prioritize RMSE or MAE if the business specifically cares about avoiding rare, very large mispricing errors? Justify your choice.
