Wrapper Methods

Wrapper methods select features by actually training a model on different feature subsets and comparing performance — slower than filter methods, but able to capture feature interactions a purely statistical filter would miss.

Recursive Feature Elimination (RFE) — The Standard Wrapper Technique

Because the model is retrained after each removal, RFE naturally accounts for how remaining features' apparent importance shifts once a correlated feature is gone — something a one-shot filter method can't do.

Python Implementation

from sklearn.feature_selection import RFE
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_breast_cancer

data = load_breast_cancer()
X, y = data.data, data.target

model = LogisticRegression(max_iter=5000)
selector = RFE(estimator=model, n_features_to_select=10)
selector.fit(X, y)

selected_features = data.feature_names[selector.support_]
print(selected_features)
print(selector.ranking_)   # 1 = selected; higher numbers = eliminated earlier

Expected output: exactly 10 feature names, out of the original 30 — the specific 10 chosen because they were the last to survive repeated elimination rounds using this particular model.

RFE With Cross-Validation — Letting the Data Choose the Feature Count

from sklearn.feature_selection import RFECV

selector_cv = RFECV(estimator=LogisticRegression(max_iter=5000), cv=5, scoring="accuracy")
selector_cv.fit(X, y)

print(selector_cv.n_features_)         # the optimal number of features, chosen automatically
print(data.feature_names[selector_cv.support_])

Instead of guessing a target feature count upfront, RFECV uses cross-validation to find the number of features that actually maximizes validation performance — avoiding an arbitrary manual choice.

Why Wrapper Methods Are Slower — and Worth It Sometimes

With \(n\) features, RFE retrains the model roughly \(n\) times (once per elimination round) — for 100 features, that's 100 model fits just to select features, before you've even started hyperparameter tuning. This cost is exactly why filter methods are typically run first to cut the feature count down before a wrapper method is applied to the smaller remaining set.

Practical Use Cases

• Selecting features for a linear model where interaction and interdependency effects matter and a filter method's independence assumption would miss them
• Finding the smallest feature subset that still achieves near-best performance, useful for interpretability or latency-constrained deployment

Advantages

• Accounts for feature interactions and interdependencies, unlike filter methods
• Directly optimizes for the metric you actually care about (accuracy, F1, etc.), not a generic statistical proxy

Limitations

• Computationally expensive — impractical on very high-dimensional data without first filtering down
• Results are somewhat tied to the specific model used for selection — features chosen for a linear model may not be the ideal set for a tree-based model
• Risk of overfitting the selection itself to the specific data used, especially without cross-validation

Common Mistakes

• Running RFE on the full dataset (train + test) instead of the training set only — feature selection needs the same train/test discipline as any other fitted step.
• Using plain RFE with an arbitrary fixed n_features_to_select instead of RFECV, when the "right" number of features isn't actually known in advance.

Interview Relevance

Q: "Why is RFE more computationally expensive than a filter method, and when is that cost justified?" RFE retrains the model at every elimination round, roughly \(n\) times for \(n\) starting features, versus a filter method's single pass over statistics — worth the cost when feature interactions matter and you have the compute budget, typically after a cheaper filter pass has already reduced the feature count.

Practice Question

You have 40 features and want to find the smallest subset that maintains at least 95% of the full model's cross-validated accuracy. Which scikit-learn tool would you reach for, and why?

Related ML Notes

• Feature Selection
• Filter Methods
• Cross-Validation