Feature Extraction

Feature extraction is the simplest, cheapest form of transfer learning: keep the entire pretrained model completely frozen, and train only a small new "head" (typically a single linear layer) on top of its outputs.

The Setup

The pretrained backbone acts purely as a fixed feature extractor — it processes the input exactly as it always would, producing rich, general representations, but none of its own weights ever get updated. Only the new head's (typically few) parameters are trained on your target task's data.

Why This Approach Is Attractive

When Feature Extraction Works Best

This approach works particularly well when the target task is reasonably similar to what the model was originally pretrained on — the pretrained features are already well-suited, and only a lightweight mapping from those features to your specific output categories needs to be learned. When the target task is meaningfully different from the pretraining domain, feature extraction alone may not adapt well enough, motivating the fine-tuning approaches covered in the next two notes.

Code

import torch
import torch.nn as nn
import torchvision.models as models

backbone = models.resnet50(weights="IMAGENET1K_V2")

# Freeze EVERY parameter in the pretrained backbone
for param in backbone.parameters():
    param.requires_grad = False

# Replace the final layer with a new, TRAINABLE head for your specific task
backbone.fc = nn.Linear(backbone.fc.in_features, 10)   # this new layer's params DO require grad by default

# Only the new head's parameters will actually receive gradient updates during training
optimizer = torch.optim.Adam(backbone.fc.parameters(), lr=0.001)   # note: only .fc.parameters(), not the whole model

Common Mistakes

• Forgetting to explicitly set requires_grad = False on the frozen backbone's parameters — without this, the backbone's weights would still receive gradients and be updated during training, defeating the entire point of feature extraction.
• Passing the whole model's parameters (including the frozen backbone) to the optimizer — even with requires_grad=False correctly set, it's cleaner and clearer to explicitly optimize only the new head's parameters.

Interview Relevance

Q: "When would feature extraction (fully frozen backbone) be preferable to fine-tuning any of the backbone's own weights?" When the target dataset is small (reducing overfitting risk from having fewer trainable parameters matters a lot), when the target task is reasonably similar to the original pretraining task (so the pretrained features are already well-suited without adjustment), or when training speed/compute is a significant constraint, since only the small new head needs gradient computation.

Practice Question

Why does feature extraction carry a much lower overfitting risk than full fine-tuning, when working with a very small target dataset?

Related DL Notes

• Freezing Layers
• Fine-Tuning
• Partial vs Full Fine-Tuning