ML Monitoring

ML monitoring continuously tracks a deployed model's health — not just whether the service is up, but whether its predictions are still trustworthy — since a model can fail silently in ways a standard uptime check would never catch.

What to Actually Monitor

Minimal Monitoring Implementation

import numpy as np
import logging

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("model_monitor")

def predict_with_monitoring(model, features, reference_stats):
    # Log basic input statistics for drift comparison later
    logger.info(f"Input feature means: {features.mean(axis=0)}")

    prediction = model.predict(features)
    probability = model.predict_proba(features)[:, 1]

    # Flag low-confidence predictions for review
    if probability.mean() < 0.55 and probability.mean() > 0.45:
        logger.warning("Batch has unusually low average confidence -- possible drift")

    return prediction, probability

The Delayed-Label Problem

Unlike system latency (known instantly), a model's true accuracy often can't be measured until real outcomes are known — a loan default might not be confirmed for months, a churn prediction's accuracy only resolves once you see whether the customer actually left. This delay is exactly why monitoring input and prediction distributions matters as an early warning signal — they're available immediately, well before ground-truth labels catch up.

Setting Alerting Thresholds

# A simple threshold-based alert -- more sophisticated systems use
# statistical drift tests (see Data Drift) instead of a fixed cutoff
baseline_mean_confidence = 0.78   # recorded from validation data

def check_confidence_drift(current_batch_confidence, threshold=0.10):
    drift = abs(current_batch_confidence - baseline_mean_confidence)
    if drift > threshold:
        send_alert(f"Confidence drift detected: {drift:.3f} change from baseline")

check_confidence_drift(current_batch_confidence=0.61)   # triggers -- 0.17 change > 0.10 threshold

Practical Use Cases

• Every production model, without exception — monitoring is what turns "deployed" into "reliably operated"
• Early detection of upstream data pipeline bugs, which often show up first as input distribution anomalies

Common Mistakes

• Only monitoring system-level metrics (uptime, latency) and assuming that's sufficient — a model can be perfectly "up" while making increasingly wrong predictions.
• Waiting for ground-truth labels before checking for any problems, when input/prediction distribution monitoring can surface issues much earlier.

Interview Relevance

Q: "Why isn't uptime monitoring enough for a production ML model?" A model can be fully "up" and responding instantly while its predictions have become systematically wrong due to drift or an upstream data bug — uptime monitoring only catches the service failing outright, not the model quietly making bad predictions while functioning normally.

Practice Question

Your model's true labels (loan defaults) aren't known for 6 months after prediction. Propose two monitoring signals you could check well before that delay resolves.

Related ML Notes

• Data Drift
• Model Drift
• ML Retraining