Experiment Tracking

Experiment tracking automatically logs every training run's parameters, metrics and artifacts — replacing "which of my 40 notebook runs actually produced the good result?" with a searchable, comparable record.

What Gets Logged, Every Run

Python Implementation — MLflow

import mlflow
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score, f1_score

mlflow.set_experiment("loan_approval_model")

for n_estimators in [100, 200, 300]:
    with mlflow.start_run(run_name=f"rf_n{n_estimators}"):
        model = RandomForestClassifier(n_estimators=n_estimators, random_state=42)
        model.fit(X_train, y_train)
        predictions = model.predict(X_test)

        mlflow.log_param("n_estimators", n_estimators)
        mlflow.log_metric("accuracy", accuracy_score(y_test, predictions))
        mlflow.log_metric("f1_score", f1_score(y_test, predictions))
        mlflow.sklearn.log_model(model, "model")

# Launch the UI to browse and compare every run visually:
# mlflow ui

Every run in this loop gets its own tracked record — no manual notes needed to remember which n_estimators value produced which score; the MLflow UI lets you sort, filter and compare runs directly.

Comparing Runs Programmatically

experiment = mlflow.get_experiment_by_name("loan_approval_model")
runs = mlflow.search_runs(experiment_ids=[experiment.experiment_id])

best_run = runs.sort_values("metrics.f1_score", ascending=False).iloc[0]
print(f"Best run: {best_run['run_id']}, F1: {best_run['metrics.f1_score']:.3f}")
print(f"Params: n_estimators={best_run['params.n_estimators']}")

Why This Matters Beyond Convenience

Without tracking, comparing hyperparameter choices means manually scrolling through notebook cell outputs or, worse, relying on memory. With tracking, "which configuration actually performed best, and why" becomes a query, not an archaeology project — and it directly feeds the model registry, since the best-tracked run is exactly what gets promoted to production.

Practical Use Cases

• Any project running more than a handful of training experiments — manual tracking stops scaling almost immediately
• Team environments, where multiple people need to see and compare each other's experiment results

Common Mistakes

• Relying on notebook cell outputs or personal notes instead of a proper tracking tool, losing the ability to systematically compare runs later.
• Logging metrics but not hyperparameters (or vice versa) — both are needed to understand why one run outperformed another.

Interview Relevance

Q: "How would you keep track of 50 different hyperparameter combinations tried during model development?" Use an experiment tracking tool like MLflow to automatically log every run's parameters, metrics and artifacts — this makes comparing, sorting, and retrieving the best run a direct query instead of manually reviewing scattered notebook outputs.

Practice Question

Modify the MLflow example above to also log a confusion matrix plot as an artifact for each run.

Related ML Notes

• Model Registry
• Model Versioning
• Hyperparameter Tuning