ColumnTransformer

ColumnTransformer applies different preprocessing to different columns within a single fitted object — exactly what's needed the moment a dataset mixes numeric and categorical features, which is almost every real dataset.

The Problem It Solves

A single Pipeline applies its steps to the entire input uniformly — but you can't sensibly apply StandardScaler to a categorical "city" column, or OneHotEncoder to a numeric "age" column. ColumnTransformer routes different column subsets to different transformers, then concatenates the results back into one feature matrix.

Full Implementation

from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

numeric_features = ["age", "income"]
categorical_features = ["city", "payment_method"]

numeric_pipeline = Pipeline([
    ("imputer", SimpleImputer(strategy="median")),
    ("scaler", StandardScaler()),
])
categorical_pipeline = Pipeline([
    ("imputer", SimpleImputer(strategy="most_frequent")),
    ("encoder", OneHotEncoder(handle_unknown="ignore")),
])

preprocessor = ColumnTransformer([
    ("num", numeric_pipeline, numeric_features),
    ("cat", categorical_pipeline, categorical_features),
])

X_transformed = preprocessor.fit_transform(X_train)
print(X_transformed.shape)   # numeric columns + one-hot expanded categorical columns

The remainder Parameter — What Happens to Unlisted Columns

# By default, any column NOT listed in the transformers list is silently DROPPED
preprocessor_drop = ColumnTransformer([
    ("num", StandardScaler(), numeric_features),
], remainder="drop")   # this is the default -- easy to miss a column by accident

# Explicitly pass through unlisted columns unchanged instead
preprocessor_passthrough = ColumnTransformer([
    ("num", StandardScaler(), numeric_features),
], remainder="passthrough")

# Or apply a transformer to whatever's left over
preprocessor_remainder_transform = ColumnTransformer([
    ("num", StandardScaler(), numeric_features),
], remainder=SimpleImputer(strategy="most_frequent"))

The default (remainder="drop") is a genuine trap — any column you forget to list explicitly silently disappears from the output with no error or warning, which can cause confusing downstream bugs.

Tracking Feature Names After Transformation

# After one-hot encoding, column count changes -- get_feature_names_out()
# recovers meaningful names for every output column
feature_names = preprocessor.get_feature_names_out()
print(feature_names)
# e.g. ['num__age', 'num__income', 'cat__city_Delhi', 'cat__city_Mumbai', ...]

This is essential for later interpreting feature importance or SHAP values — without it, you're left with anonymous column indices instead of meaningful names.

Selecting Columns by Data Type Automatically

from sklearn.compose import make_column_selector

preprocessor_auto = ColumnTransformer([
    ("num", StandardScaler(), make_column_selector(dtype_include="number")),
    ("cat", OneHotEncoder(handle_unknown="ignore"), make_column_selector(dtype_include="object")),
])
# No need to hard-code column names -- selects by dtype automatically,
# useful when column sets change across datasets or over time

Practical Use Cases

• Any dataset mixing numeric and categorical features — essentially every real tabular dataset
• Applying entirely different preprocessing logic to different logical groups of columns (e.g. text columns vs numeric columns vs date columns)

Common Mistakes

• Leaving remainder="drop" (the default) and unintentionally losing columns that were never explicitly listed.
• Forgetting handle_unknown="ignore" on the categorical encoder, causing a crash when a new, unseen category appears in production.

Interview Relevance

Q: "You added a new column to your dataset, and it silently disappeared from your model's input. What's a likely cause?" The ColumnTransformer's default remainder="drop" behavior — any column not explicitly listed in one of the transformer tuples gets silently dropped with no warning; explicitly setting remainder="passthrough" (or listing the column) fixes this.

Practice Question

You have numeric, categorical, and a free-text column in one dataset. Sketch the three-branch ColumnTransformer structure you'd use to handle all of them appropriately.

Related ML Notes

• scikit-learn Pipeline
• Preprocessing Pipeline
• One-Hot Encoding