ROC-AUC measures how well a classifier separates the two classes across every possible decision threshold at once — instead of evaluating one fixed threshold like precision/recall/F1 do.
Building the ROC Curve
The ROC (Receiver Operating Characteristic) curve plots the True Positive Rate against the False Positive Rate as the classification threshold sweeps from 1 (predict positive for almost nothing) down to 0 (predict positive for everything).
Worked Example — A Small ROC Curve by Hand
| Threshold | TP | FP | FN | TN | TPR | FPR |
|---|---|---|---|---|---|---|
| 0.9 (strict) | 5 | 1 | 15 | 79 | 0.25 | 0.0125 |
| 0.5 (default) | 15 | 10 | 5 | 70 | 0.75 | 0.125 |
| 0.1 (lenient) | 20 | 40 | 0 | 40 | 1.0 | 0.5 |
As the threshold loosens, TPR (catching more true positives) rises — but so does FPR (more false alarms). The ROC curve traces this entire tradeoff, and AUC (Area Under the Curve) condenses it into one number.
Graphical Intuition
The more the curve bows toward the top-left corner (high TPR, low FPR simultaneously), the better the classifier separates the classes — and the larger the area beneath it.
Python Implementation
from sklearn.metrics import roc_curve, roc_auc_score, RocCurveDisplay
import matplotlib.pyplot as plt
# probabilities, not hard labels, are needed for ROC-AUC
y_true = [0,0,0,1,1,0,1,0,1,1]
y_scores = [0.1,0.2,0.35,0.6,0.8,0.15,0.55,0.4,0.9,0.7]
fpr, tpr, thresholds = roc_curve(y_true, y_scores)
auc = roc_auc_score(y_true, y_scores)
print("AUC:", auc)
RocCurveDisplay.from_predictions(y_true, y_scores)
plt.plot([0,1],[0,1], linestyle="--", color="gray") # the random-guessing diagonal
plt.show()Reading the AUC Value
| AUC | Meaning |
|---|---|
| 1.0 | Perfect separation — every positive ranked above every negative |
| 0.5 | No better than random guessing |
| < 0.5 | Worse than random — predictions are systematically backwards |
A genuinely useful interpretation: AUC equals the probability that a randomly chosen positive example is ranked higher (given a higher predicted probability) than a randomly chosen negative example — a direct measure of ranking quality, independent of any specific threshold choice.
Practical Use Cases
- Comparing classifiers when the optimal decision threshold isn't yet known or might change later
- Ranking problems — lead scoring, risk ranking — where relative order matters more than a single hard cutoff
Common Mistakes
- Using ROC-AUC as the primary metric on severely imbalanced data — it can look deceptively good even when precision on the rare positive class is poor; see Precision-Recall Curve for the more appropriate alternative there.
- Passing hard class predictions (0/1) instead of probability scores to
roc_auc_score— this discards the ranking information ROC-AUC is actually designed to measure.
Interview Relevance
Q: "What does an AUC of 0.5 mean?" The classifier's predictions carry no more ranking information than random guessing — a randomly chosen positive example is exactly as likely to be ranked above a randomly chosen negative example as below it.
Practice Question
Using the worked threshold table above, verify the TPR and FPR calculation at threshold 0.5, matching the earlier confusion matrix numbers (TP=15, FP=10, FN=5, TN=70).
