ResNet

ResNet (2015) solved the exact problem that had limited how deep any CNN could practically go — vanishing gradients across many stacked layers — with one deceptively simple architectural addition: the residual (skip) connection.

The Problem It Solved

Before ResNet, simply stacking more layers eventually made networks harder to train, not easier — beyond a certain depth, accuracy would actually get worse, even on the training set (ruling out overfitting as the explanation). This was a direct manifestation of the vanishing gradient problem from Vanishing Gradient Problem: gradients shrinking multiplicatively across dozens or hundreds of layers, leaving early layers with almost no usable training signal.

Key Innovation: The Residual Connection

Instead of a block of layers learning the full desired output directly, ResNet has each block learn only the residual — the difference between its input and its desired output, \(F(\mathbf{x})\) — and then adds the original input \(\mathbf{x}\) back in directly. This "skip connection" provides an additive path for gradients to flow backward, completely bypassing the multiplicative chain-rule shrinkage that causes vanishing gradients — as flagged already in Vanishing Gradient Problem's solutions table.

Diagram

Code

import torch
import torch.nn as nn

class ResidualBlock(nn.Module):
    def __init__(self, channels):
        super().__init__()
        self.conv1 = nn.Conv2d(channels, channels, kernel_size=3, padding=1)
        self.conv2 = nn.Conv2d(channels, channels, kernel_size=3, padding=1)
        self.relu = nn.ReLU()

    def forward(self, x):
        identity = x
        out = self.relu(self.conv1(x))
        out = self.conv2(out)
        out += identity   # the residual/skip connection
        return self.relu(out)

Advantages and Limitations

Use Cases

ResNet variants (ResNet-18, 34, 50, 101, 152) remain extremely widely used today — as classification backbones, as pretrained feature extractors for transfer learning, and as the architectural inspiration for the residual connections used throughout modern Transformer architectures (see the Transformers category).

Common Mistakes

• Assuming residual connections only matter for very deep networks — even moderately deep networks often train faster and more reliably with them, since they generally improve gradient flow regardless of exact depth.
• Forgetting that a residual connection requires the input and the block's output to have matching shapes — architectures typically insert a small projection (often a 1×1 convolution) on the skip path whenever a block changes the number of channels or spatial resolution.

Interview Relevance

Q: "How do residual connections solve the vanishing gradient problem in very deep networks?" By adding the block's input directly to its output, residual connections provide an additive path for gradients to flow backward, bypassing the multiplicative chain-rule product that causes vanishing gradients in a purely sequential stack of layers. Even if a block's own gradient contribution shrinks toward zero, the identity path still carries gradient signal back to earlier layers essentially undiminished.

Practice Question

Why does a residual block require its convolutional path's output to match the shape of its input exactly, and what's the typical fix when a block needs to change the number of channels?

Related DL Notes

• Vanishing Gradient Problem
• DenseNet
• Residual Connections