Feature Transformation

Feature transformation reshapes an existing feature's mathematical form — most often to fix skew, stabilize variance, or make a relationship more linear — without changing what real-world quantity the feature represents.

The Log Transform — The Workhorse of Feature Transformation

Adding 1 before taking the log (np.log1p) avoids an undefined result when \(x=0\). Log transforms compress large values much more than small ones, which is exactly what taming a right-skewed distribution (income, prices, view counts) requires.

import numpy as np
import pandas as pd

incomes = pd.Series([30000, 32000, 35000, 40000, 500000])

print(incomes.skew())                  # strongly positive -- right-skewed
log_incomes = np.log1p(incomes)
print(log_incomes.skew())               # much closer to 0 after the transform
print(log_incomes)

Other Common Transforms

Box-Cox — Letting the Data Choose the Transform

from scipy import stats

# Box-Cox requires strictly positive values
transformed, best_lambda = stats.boxcox(incomes)
print(best_lambda)      # the power that best normalizes THIS specific data
print(transformed)

Instead of guessing between square root, log, or reciprocal, Box-Cox searches for the single \(\lambda\) that makes the transformed data closest to normally distributed — a data-driven alternative to picking a transform by trial and error.

Why This Matters for ML

• Linear/logistic regression assumes roughly linear relationships — a log-transformed skewed feature often relates to the target far more linearly than the raw version
• Reduces the outsized influence of extreme values without discarding them entirely, unlike outright outlier removal
• Some statistical tests and confidence intervals assume roughly normal data — transformation can make that assumption more reasonable

Common Mistakes

• Applying a log transform to a feature that can be zero or negative without first shifting it appropriately (log1p handles zero, but not negative values).
• Transforming a feature but forgetting to inverse-transform predictions back to the original scale when the target itself was transformed (e.g. predicting log(price) but reporting results as if they were price directly).
• Applying transforms mechanically to every numeric feature without checking whether each one is actually skewed in the first place — unnecessary for already roughly-symmetric features.

Interview Relevance

Q: "You log-transformed your target variable to train a regression model. How do you get predictions back in the original units?" Apply the inverse transform (exponentiate, minus the offset used) to the model's predictions before reporting or evaluating them — forgetting this step is a common, easy-to-miss bug that silently reports predictions on the wrong scale.

Practice Question

A "number_of_purchases" feature is heavily right-skewed with many zeros. Which transform from the table above would you try first, and why?

Related ML Notes

• Skewness
• Numerical Features
• Normal Distribution