This note focuses specifically on the "block" as the fundamental unit of LLM scale — the exact same structure from Transformer Decoder, repeated \(N\) times, where \(N\) is one of the most consequential design choices in an LLM's architecture.
What "Number of Layers" Actually Means
Every time you read that a model has "96 layers" or "32 layers," this refers directly to how many times the same block structure (masked self-attention + feed-forward, each with residual connection and normalization) is stacked sequentially. Each block has its own independently learned weights — deeper stacking means more sequential transformation steps, not more parallel capacity within a single step.
Depth vs Width — Revisited at LLM Scale
Recall the depth/width tradeoff from Neural Network Architecture — it applies directly here. An LLM's total capacity is shaped by both its depth (number of blocks) and its width (\(d_{\text{model}}\), the hidden dimension carried through every block). Different model families make different tradeoffs between the two, and neither dimension alone determines a model's quality.
How Information Flows Through the Stack
Each block's residual connection (see Residual Connections) means the hidden state at block \(l\) is the hidden state at block \(l-1\) plus whatever that block's self-attention and feed-forward sublayers computed — information doesn't get replaced at each block so much as it gets progressively refined and added to, layer after layer, all the way through the stack.
Code — Inspecting the Block Stack Directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("gpt2")
print(len(model.transformer.h)) # 12 -- the number of stacked decoder blocks
for i, block in enumerate(model.transformer.h):
if i < 2: # just inspect the first couple
print(f"Block {i}:", block)Common Mistakes
- Assuming every block in a stack learns something "higher level" than the previous one, the way stacked convolutional layers progressively capture larger receptive fields — the relationship between depth and what each layer "learns" in a Transformer is less strictly hierarchical, since self-attention at every layer can already relate any two positions directly.
- Confusing the number of transformer blocks with the number of attention heads within a single block — these are two entirely separate hyperparameters, one controlling sequential depth, the other controlling parallel attention specialization within each block (see Multi-Head Attention).
Interview Relevance
Q: "If two LLMs have the same total parameter count but different depth/width tradeoffs, would you expect them to behave identically?" Not necessarily — depth and width represent different kinds of capacity (sequential transformation steps versus per-step representational richness), and empirical research has found the balance between them affects both training dynamics and downstream task performance, even at matched total parameter counts.
Practice Question
What's the difference between increasing an LLM's number of transformer blocks versus increasing its number of attention heads per block?
