Embedded Methods

Embedded methods perform feature selection as a natural byproduct of training a single model — no separate selection step is needed, because the algorithm itself decides which features matter while it learns.

Technique 1 — L1 (Lasso) Regularization

Ordinary linear regression assigns every feature a nonzero coefficient. L1 regularization adds a penalty proportional to the absolute value of each coefficient — and uniquely among common penalties, this specific penalty shape can drive some coefficients to exactly zero, effectively removing those features from the model entirely.

from sklearn.linear_model import Lasso
import numpy as np

X = np.random.rand(100, 20)   # 20 features, most randomly unrelated to y
y = 3 * X[:, 0] - 2 * X[:, 1] + np.random.normal(0, 0.1, 100)   # only features 0 and 1 actually matter

model = Lasso(alpha=0.1)
model.fit(X, y)

print(model.coef_)                       # many coefficients will be EXACTLY 0.0
selected_features = np.where(model.coef_ != 0)[0]
print(selected_features)                  # roughly [0, 1] -- Lasso found the truly relevant features

Expected behavior: because only features 0 and 1 actually generated y, Lasso's exact-zero property should zero out most or all of the other 18 unrelated coefficients — feature selection happening automatically as a side effect of training, not a separate step.

Technique 2 — Tree-Based Built-In Importance

from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_breast_cancer
import pandas as pd

data = load_breast_cancer()
model = RandomForestClassifier(random_state=42).fit(data.data, data.target)

importances = pd.Series(model.feature_importances_, index=data.feature_names).sort_values(ascending=False)
top_features = importances.head(10).index
print(list(top_features))

Every tree-based model computes feature importance as a natural output of training — using it to select the top-k features and discard the rest is embedded selection, with zero additional training runs beyond the one you already needed.

Why "Embedded" Sits Between Filter and Wrapper

Practical Use Cases

• Lasso regression when you want both a predictive model AND automatic feature selection from one training run
• Using a Random Forest or XGBoost's built-in importance to prune features before training a separate, simpler production model

Advantages

• No separate feature selection step required — selection is essentially "free," happening during normal training
• Accounts for interactions and correlations the model itself is sensitive to

Limitations

• The selected features are somewhat specific to the model used — features Lasso prunes might still matter to a tree-based model, and vice versa
• Choosing the right regularization strength (\(\alpha\) in Lasso) requires its own tuning, which affects how aggressively features get zeroed out

Common Mistakes

• Using an untuned, default regularization strength and assuming the resulting feature selection is optimal — too weak keeps everything, too strong can zero out genuinely useful features.
• Treating a feature Lasso zeroed out as definitively "useless" — a different model type or a different regularization strength might still find it valuable.

Interview Relevance

Q: "Why does L1 (Lasso) regularization produce exactly-zero coefficients, but L2 (Ridge) doesn't?" L1's penalty (sum of absolute values) has a geometric shape (a diamond in coefficient space) whose corners align with the axes, making it likely for the optimal solution to land exactly on an axis (a coefficient of zero); L2's penalty (sum of squares) is a smooth circle with no corners, so it shrinks coefficients toward zero without usually reaching it exactly. See L1 Regularization and L2 Regularization for the full geometric explanation.

Practice Question

You train a Lasso model and find that 35 out of 50 features have exactly zero coefficients. What does this tell you, and what would increasing the regularization strength (\(\alpha\)) further likely do?

Related ML Notes

• Feature Selection
• L1 Regularization
• Feature Importance