A transformer is built from identical blocks, stacked one after another — each block combines multi-head self-attention with a feed-forward network, wrapped in residual connections and normalization that keep training stable at large scale.
Inside One Transformer Block
Input (token representations from the previous layer)
↓
Multi-head self-attention
↓
Add input back in (residual connection) + normalize
↓
Feed-forward network (applied independently to each token)
↓
Add previous result back in (residual connection) + normalize
↓
Output — passed to the next stacked blockThis single block is repeated N times (N = the model's "depth" — more blocks generally means more capability, at the cost of more compute; see LLM Parameters).
Why Residual Connections and Normalization Matter
Stacking dozens of layers naively tends to make training unstable — gradients (the training signal) can shrink or explode as they flow backward through many layers. Residual connections (adding a layer's input back to its output) give the training signal a more direct path through the network. Layer normalization keeps the scale of values consistent between layers. Together, these are what make training networks dozens of layers deep practically feasible, rather than a purely theoretical architecture.
The Feed-Forward Layer's Role
After attention lets tokens gather information from each other, the feed-forward network processes each token's representation independently — a straightforward transformation applied token-by-token. Attention handles "which tokens relate to which"; the feed-forward layer handles further processing of each token's own representation.
Practical Use Case
Understanding this structure explains real, practical model differences: two models with the same parameter count but different depth/width tradeoffs (more, thinner layers vs fewer, wider ones) can behave differently, and this is part of what varies across model families and sizes.
Common Mistakes
- Assuming attention is the entire transformer block — the feed-forward layer, residual connections, and normalization are equally structural, not minor implementation details
- Assuming "more layers" is a strictly better choice regardless of training data and compute — depth is one lever among several, with real tradeoffs
Interview Relevance
"What are the two main sub-layers inside a transformer block, and why are residual connections included?" — a solid mid-level interview question testing whether the candidate understands the architecture beyond just "attention."
Practice Question
Explain, in your own words, why removing residual connections from a very deep transformer would likely make it harder to train successfully.
