Feature selection decides which features actually earn a place in your model — dropping the redundant, irrelevant or noisy ones. Fewer, better-chosen features often outperform "throw everything in and let the model figure it out."
Why Fewer Features Can Mean a Better Model
- Less overfitting: more features means more opportunity for a model to fit noise instead of signal, especially with limited training data
- Faster training and inference: fewer inputs, less computation
- Easier interpretation: a 10-feature model is far easier to explain than a 200-feature one
- Removes redundancy: highly correlated features can destabilize linear model coefficients (multicollinearity)
The Three Families of Feature Selection
Filter is fastest but ignores the model; wrapper is most thorough but slowest; embedded is a practical middle ground.
| Family | Speed | Considers Feature Interactions? | Example |
|---|---|---|---|
| Filter Methods | Fast | No — scores each feature independently | Variance threshold, correlation with target |
| Wrapper Methods | Slow — retrains the model repeatedly | Yes | Recursive Feature Elimination (RFE) |
| Embedded Methods | Moderate — one training run | Yes, implicitly | Lasso (L1) regularization, tree-based importance |
A Practical Sequencing
In practice, these aren't mutually exclusive — a common workflow uses a fast filter method first to eliminate obviously useless features (near-zero variance, near-zero correlation with the target), then applies a more expensive wrapper or embedded method to fine-tune the remaining, more promising subset.
Practical Use Cases
- Reducing an initial 200-feature dataset down to the 20-30 that actually carry signal
- Improving a linear model's stability by removing multicollinear features
- Speeding up training and inference for latency-sensitive production systems
Common Mistakes
- Performing feature selection on the full dataset before splitting into train/test — leaks information about which features "look good" on data the model shouldn't have seen yet.
- Assuming a low individual filter-method score means a feature is useless — filter methods miss interaction effects a wrapper or embedded method (or the model itself) might still exploit.
Interview Relevance
Q: "What's the difference between filter, wrapper and embedded feature selection?" Filter methods score features using statistics alone, independent of any model (fast, but ignores interactions); wrapper methods search feature subsets by actually training and evaluating a model repeatedly (thorough, but slow); embedded methods perform selection as a byproduct of training a single model (like Lasso zeroing out coefficients).
Practice Question
You have 500 features and limited compute time. Describe a practical two-stage feature selection strategy combining a filter method and a more expensive method.
