CatBoost (Categorical Boosting) is a gradient boosting implementation built specifically to handle categorical features natively and reduce a subtle leakage problem called prediction shift that other boosting methods can suffer from.
Native Categorical Feature Handling
XGBoost and LightGBM generally still expect categorical features to be numerically encoded first (one-hot, target, or similar). CatBoost accepts raw categorical columns directly, using an internal, statistically-aware encoding technique — removing an entire preprocessing step and, more importantly, avoiding the target-leakage risks that come with naive target encoding (see Categorical Features).
from catboost import CatBoostClassifier
import pandas as pd
df = pd.DataFrame({
"city": ["Delhi","Mumbai","Delhi","Pune","Mumbai"],
"income": [45000, 62000, 51000, 39000, 71000],
"churned": [0, 0, 1, 1, 0],
})
X = df[["city", "income"]]
y = df["churned"]
model = CatBoostClassifier(iterations=200, learning_rate=0.1, cat_features=["city"], verbose=False)
model.fit(X, y)
print(model.predict(X))
# No manual one-hot or target encoding needed -- CatBoost handles "city" directlyOrdered Boosting — Fixing Prediction Shift
A subtle problem in standard gradient boosting: when computing residuals for training point \(i\), the model used to predict it was itself trained partly using point \(i\) in earlier rounds — a mild form of target leakage baked into the boosting process itself, called "prediction shift." CatBoost's ordered boosting uses a random permutation of the data and only computes each point's residual using a model trained on points that came before it in that ordering — directly preventing this leakage.
Comparison With XGBoost and LightGBM
| XGBoost | LightGBM | CatBoost | |
|---|---|---|---|
| Categorical features | Requires encoding first | Some native support | Fully native, most robust handling |
| Tree growth | Level-wise (default) | Leaf-wise | Symmetric (oblivious) trees |
| Prediction shift handling | Not addressed directly | Not addressed directly | Ordered boosting specifically fixes this |
| Default hyperparameter robustness | Good, some tuning helps | Good, some tuning helps | Often strong "out of the box," less tuning needed |
| Training speed on GPU | Fast | Very fast | Fast, competitive |
Symmetric (Oblivious) Trees
CatBoost's trees use the same split condition across an entire level, for every node at that depth — a more constrained, "symmetric" tree structure than either level-wise or leaf-wise growth. This makes prediction faster (the same decision logic applies uniformly) and provides a built-in regularization effect, at some cost to per-tree flexibility.
Practical Use Cases
- Datasets with many, especially high-cardinality, categorical features
- Projects where minimizing manual preprocessing and hyperparameter tuning effort matters — CatBoost's defaults are often unusually strong
Common Mistakes
- Manually one-hot or target encoding categorical features before passing them to CatBoost — this discards its main advantage; pass raw categorical columns via
cat_featuresinstead. - Assuming CatBoost is always the fastest option — LightGBM often still wins on raw training speed for very large, mostly-numeric datasets.
Interview Relevance
Q: "What is 'prediction shift,' and how does CatBoost address it?" A subtle leakage in standard gradient boosting where a training point's residual is computed using a model that was itself influenced by that same point in earlier rounds; CatBoost's ordered boosting uses a random permutation so each point's residual is only ever computed from a model trained on points before it, avoiding this leakage.
Practice Question
You have a dataset where 8 of 12 features are categorical, several with over 1,000 unique values. Which of the three boosting libraries would you reach for first, and why?
