Polynomial features let a linear model fit curved, non-linear relationships — by expanding your features into their powers and products before training, rather than needing a fundamentally different, more complex algorithm.
The Core Idea
Linear regression can only fit a straight line (or flat plane, in higher dimensions) to the raw features you give it. But if you add \(x^2\) as a new feature alongside \(x\), linear regression can now fit \(y = b_0 + b_1x + b_2x^2\) — a curve — because from the model's perspective, it's still just fitting a straight-line relationship, only now against an expanded set of inputs that happen to include \(x^2\).
Formula
Numerical Example
from sklearn.preprocessing import PolynomialFeatures
import numpy as np
X = np.array([[2, 3]]) # x1=2, x2=3
poly = PolynomialFeatures(degree=2)
X_poly = poly.fit_transform(X)
print(poly.get_feature_names_out())
print(X_poly)
# ['1' 'x0' 'x1' 'x0^2' 'x0 x1' 'x1^2']
# [[1. 2. 3. 4. 6. 9.]] -- 1, x1=2, x2=3, x1²=4, x1·x2=6, x2²=9Six new columns from two original features — this is exactly why polynomial feature counts grow fast: for \(n\) original features at degree \(d\), the number of resulting columns grows combinatorially, not linearly.
Fitting a Curve — The Visual Payoff
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures
from sklearn.pipeline import make_pipeline
import numpy as np
X = np.array([[1], [2], [3], [4], [5]])
y = np.array([2, 5, 10, 17, 26]) # follows y = x^2 + 1, a curve, not a line
linear_model = LinearRegression().fit(X, y)
print(linear_model.predict([[6]])) # a straight-line model badly underfits this curved data
poly_model = make_pipeline(PolynomialFeatures(degree=2), LinearRegression())
poly_model.fit(X, y)
print(poly_model.predict([[6]])) # much closer to the true value of 37 (6^2 + 1)Practical Use Cases
- Extending linear/logistic regression to capture curved relationships without switching algorithms entirely
- Baseline non-linear modeling when a more complex algorithm isn't justified or interpretability still matters
Advantages
- Keeps the interpretability and simplicity of a linear model while capturing non-linear patterns
- Easy to implement, and composes cleanly with scikit-learn's Pipeline
Limitations
- Feature count explodes quickly with degree and number of original features — a high degree on many features can create an unmanageable number of columns
- High-degree polynomial features are prone to overfitting, especially combined with limited training data — regularization is almost always needed alongside them
Common Mistakes
- Using a high polynomial degree "to be safe" without checking how many columns it actually produces or whether it overfits on a validation set.
- Forgetting to scale features before applying polynomial expansion — squared/cubed terms of an already-large feature can produce enormous, poorly-scaled values.
- Applying polynomial features to tree-based models — trees can already model non-linear splits natively and gain little from this transform.
Interview Relevance
Q: "How can linear regression fit a curved relationship?" By expanding the input features to include their powers (like \(x^2\)) before fitting — the model is still linear in its parameters, but the added polynomial terms let it represent curves in terms of the original feature.
Practice Question
For a single feature \(x\) at degree 3, list all the columns PolynomialFeatures would generate (including the bias term).
