PCA and feature selection both reduce the number of dimensions a model works with — but they do it in fundamentally different ways, with a real interpretability tradeoff that decides which one actually fits a given project.
The Core Difference
| PCA | Feature Selection | |
|---|---|---|
| What survives | New features — linear combinations of ALL original features | A subset of the ORIGINAL features, unchanged |
| Interpretability | Low — a principal component mixes many original features together | High — a selected feature is still exactly what it was (e.g. "age") |
| Uses the target variable? | No — purely based on feature variance, unsupervised | Can (wrapper/embedded methods) or can't (filter methods) |
| Removes multicollinearity? | Yes, inherently — components are orthogonal | Only if you explicitly select against correlated pairs |
| Reversible? | Approximately, via inverse_transform() (with some information loss) | Trivially — dropped features can simply be re-added |
A Concrete Illustration
# Feature selection: keep original, human-readable features
selected_features = ["age", "income", "credit_score"] # still directly meaningful
# PCA: create new features that mix everything together
# PC1 = 0.42*age + 0.61*income - 0.35*credit_score + 0.58*num_dependents + ...
# -- mathematically powerful, but "what does PC1 actually represent?" has no simple answerWhen to Choose Each
| Situation | Better Choice |
|---|---|
| Need to explain the model's reasoning in terms of real-world features | Feature selection |
| Features are highly correlated, and just want the smallest useful representation | PCA |
| Regulatory/compliance requirement to justify decisions per original feature | Feature selection |
| Preparing data purely for a downstream model with no interpretability requirement | Either — often PCA, for its multicollinearity-removal benefit |
| Visualizing high-dimensional data | PCA (or t-SNE/UMAP) |
They're Not Mutually Exclusive
A common real-world pipeline uses feature selection first (to remove obviously irrelevant or redundant raw features, keeping the rest interpretable) and then applies PCA only if the remaining feature count is still too high or too correlated for the intended downstream use — combining both when each one's strengths are actually needed.
Common Mistakes
- Reaching for PCA by default without considering whether the project actually needs interpretable, per-feature explanations — a compliance or medical context often can't accept "PC1 increased" as an acceptable justification.
- Assuming feature selection alone solves multicollinearity — it only does so if the selection method explicitly accounts for correlated pairs (see Filter Methods).
Interview Relevance
Q: "Why might a bank prefer feature selection over PCA for a loan approval model?" Regulatory requirements often demand explaining a credit decision in terms of specific, real-world factors ("denied due to low credit score and high debt-to-income ratio") — PCA's components are opaque linear combinations that can't be explained this way, while feature-selected original features remain directly interpretable.
Practice Question
You're building a model where accuracy is the only priority and no explanation is ever required. Would PCA's interpretability tradeoff matter here? Explain your reasoning.
