A decision guide, not a repeat lesson — with Accuracy, Precision, Recall, F1 and ROC-AUC already covered individually, this note is about choosing correctly between them for a specific problem.
The Decision Table
| Situation | Best Metric | Why |
|---|---|---|
| Roughly balanced classes, both errors similarly costly | Accuracy | Simple, intuitive, and not misleading here |
| False positives are expensive/disruptive | Precision | Directly measures how trustworthy a positive prediction is |
| False negatives are dangerous/costly | Recall | Directly measures how much of the real positive class is caught |
| Need one number balancing both errors | F1-Score | Harmonic mean punishes imbalance between precision and recall |
| Comparing models before choosing a threshold | ROC-AUC | Threshold-independent measure of ranking quality |
| Rare positive class (imbalanced data) | Precision-Recall / Average Precision | ROC-AUC can look misleadingly strong here |
| Care about calibrated probability quality | Log Loss | Rewards confident-and-correct, punishes confident-and-wrong |
A Worked Decision, End to End
# Business context: fraud detection where missing fraud costs $500 average per case,
# but a false alarm costs $5 in manual review time -- clearly recall matters more
from sklearn.metrics import classification_report
y_true = [1]*20 + [0]*80
y_pred = [1]*15 + [0]*5 + [1]*10 + [0]*70
print(classification_report(y_true, y_pred))
# Given the cost asymmetry above, prioritize the RECALL number for the positive class
# specifically -- 0.75 here means 25% of real fraud still slips through, which at
# $500/case average cost may justify lowering the classification threshold furtherMulti-Class Metric Averaging
For more than 2 classes, precision/recall/F1 need an averaging strategy:
| Averaging | What It Does |
|---|---|
macro | Average each class's metric equally, regardless of class size — good when all classes matter equally |
weighted | Average weighted by each class's frequency — reflects overall performance, dominated by common classes |
micro | Aggregate TP/FP/FN across all classes first, then compute — equivalent to accuracy for single-label multi-class |
from sklearn.metrics import f1_score
print(f1_score(y_true_multiclass, y_pred_multiclass, average="macro"))
print(f1_score(y_true_multiclass, y_pred_multiclass, average="weighted"))Common Mistakes
- Defaulting to accuracy without checking class balance and the relative cost of each error type first.
- Using
average="weighted"when a rare class is actually the one that matters most — this averaging strategy will mostly reflect performance on the common classes.
Interview Relevance
Q: "How do you decide which classification metric to optimize for a new business problem?" Start by estimating the relative cost of a false positive vs a false negative for this specific problem — that asymmetry (or lack of one), combined with the class balance, directly points to accuracy, precision, recall, F1, or a threshold-tuned approach via the precision-recall curve.
Practice Question
You're building a content moderation model to flag policy-violating posts. A missed violation (false negative) can cause real harm; a false positive means an innocent post gets extra review. Which metric would you prioritize, and would you lean toward macro or weighted averaging if there are multiple violation categories?
