XGBoost

XGBoost (Extreme Gradient Boosting) is a highly optimized, regularized implementation of gradient boosting — for years the dominant algorithm in tabular data competitions and a common production default for structured data problems.

What XGBoost Adds Beyond Plain Gradient Boosting

Python Implementation

import xgboost as xgb
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
from sklearn.datasets import load_breast_cancer

data = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(
    data.data, data.target, test_size=0.2, random_state=42, stratify=data.target
)

model = xgb.XGBClassifier(
    n_estimators=200,
    learning_rate=0.1,
    max_depth=4,
    reg_lambda=1.0,       # L2 regularization strength
    random_state=42,
    eval_metric="logloss",
)
model.fit(X_train, y_train)

predictions = model.predict(X_test)
print(accuracy_score(y_test, predictions))

XGBoost's scikit-learn-compatible API (fit()/predict()/predict_proba()) has been stable for years, so it drops directly into a scikit-learn Pipeline or GridSearchCV exactly like any other estimator.

Using Early Stopping

model = xgb.XGBClassifier(n_estimators=1000, learning_rate=0.05, early_stopping_rounds=20, eval_metric="logloss")
model.fit(
    X_train, y_train,
    eval_set=[(X_test, y_test)],
    verbose=False,
)
print("Best iteration:", model.best_iteration)   # training stopped early once validation loss stopped improving

Early stopping directly addresses gradient boosting's core overfitting risk — instead of guessing the right n_estimators in advance, set it generously high and let the algorithm halt automatically once the held-out validation metric stops improving.

Reading Feature Importance

import pandas as pd

importances = pd.Series(model.feature_importances_, index=data.feature_names).sort_values(ascending=False)
print(importances.head(10))
# Same caveats about MDI-style importance apply here as in Random Forest Feature Importance

Practical Use Cases

• Structured/tabular data problems where maximizing predictive accuracy is the priority
• Datasets with missing values, thanks to XGBoost's native handling
• Kaggle-style competitions and production systems where the extra tuning effort is worth the accuracy gain over Random Forest

Common Mistakes

• Not tuning regularization (reg_lambda, reg_alpha) at all — XGBoost's regularization is a real, meaningful lever, not just a safety net left at defaults.
• Skipping early stopping and manually guessing n_estimators, wasting tuning effort that early stopping largely automates.
• Using an outdated code example — always check the currently installed XGBoost version's API before relying on older tutorials, since parameter names and defaults have shifted across major versions.

Interview Relevance

Q: "What does XGBoost add on top of plain gradient boosting?" Built-in L1/L2 regularization to directly control model complexity, use of second-order gradient (curvature) information for more efficient optimization, native missing-value handling, and substantial engineering optimizations (parallelized split-finding, efficient memory use) that make it dramatically faster in practice.

Practice Question

Explain why setting n_estimators=1000 combined with early_stopping_rounds=20 is generally safer than manually guessing a smaller fixed number of estimators.

Related ML Notes

• Gradient Boosting
• LightGBM
• Hyperparameter Tuning