Residual Connections

A residual connection wraps every single sublayer in a Transformer — every attention block and every feed-forward block — with a simple addition: the sublayer's input is added directly to its output, giving gradients an unobstructed path through an otherwise very deep stack of layers.

Formula

Rather than replacing \(\mathbf{x}\) entirely with whatever the sublayer computes, the sublayer's output is added on top of the original input — the sublayer only needs to learn the residual (the difference/adjustment), not reconstruct the entire representation from scratch.

Why This Matters for Very Deep Stacks

Recall the exact multiplicative-shrinkage mechanism from Vanishing Gradient Problem and its RNN-specific version in RNN Vanishing Gradient: gradients passing through many layers, each contributing a local derivative less than 1, shrink exponentially. A Transformer with, say, 24 or 48 stacked encoder/decoder layers is exactly the kind of deep stack where this would be a serious problem without mitigation.

The residual connection's addition has a local derivative of exactly 1 with respect to \(\mathbf{x}\) (since \(\frac{\partial}{\partial \mathbf{x}}(\mathbf{x}+\text{Sublayer}(\mathbf{x})) = 1 + \frac{\partial \text{Sublayer}}{\partial \mathbf{x}}\)) — providing a direct, undiminished gradient "highway" straight through every layer, alongside whatever the sublayer itself contributes. This is conceptually identical to the skip-connection idea in ResNet (covered in the CNN Architectures category), applied here to attention and feed-forward sublayers instead of convolutional ones.

Diagram

Code

import torch
import torch.nn as nn

class ResidualSublayer(nn.Module):
    def __init__(self, sublayer, d_model):
        super().__init__()
        self.sublayer = sublayer
        self.norm = nn.LayerNorm(d_model)

    def forward(self, x):
        return self.norm(x + self.sublayer(x))   # the residual connection: x + Sublayer(x), then normalized

attention_sublayer = nn.MultiheadAttention(embed_dim=512, num_heads=8, batch_first=True)
# wrapping requires a small adapter since MultiheadAttention needs (q,k,v) -- simplified for illustration
x = torch.randn(1, 10, 512)
attn_output, _ = attention_sublayer(x, x, x)
residual_output = x + attn_output   # the core residual pattern, directly
print(residual_output.shape)   # (1, 10, 512) -- unchanged shape, ready for the next sublayer

Common Mistakes

• Assuming residual connections are unique to CNNs (ResNet) — the exact same idea, applied to attention and feed-forward sublayers instead of convolutional ones, is what makes training very deep Transformer stacks (dozens of layers) practically feasible.
• Forgetting that the residual connection requires the sublayer's output to have the same shape as its input — this is precisely why every Transformer sublayer (attention, feed-forward) is designed to preserve the input's dimensionality exactly.

Interview Relevance

Q: "Why does a Transformer need residual connections around every attention and feed-forward sublayer?" Modern Transformer stacks are very deep (often dozens of layers), and without residual connections, gradients backpropagating through that many sublayers would be vulnerable to the same exponential vanishing-gradient mechanism that affects any very deep network. The residual connection's addition provides a direct gradient path with a local derivative of 1, letting gradients flow through the entire stack largely undiminished, alongside whatever each sublayer itself contributes.

Practice Question

If a specific sublayer's learned transformation happened to be exactly zero (contributing nothing), what would the residual connection's output equal, and why is that a safe, sensible fallback?

Related DL Notes

• Vanishing Gradient Problem
• ResNet
• Layer Normalization