Self-attention, by itself, has no built-in sense of word order — it treats input as a set, not a sequence. Positional encoding is what injects that missing order information back in.
Why This Is a Real Problem
"The dog bit the man" vs "The man bit the dog"
Both sentences contain the exact same tokens. Without any
positional information, self-attention's token-to-token
comparisons would treat these as equivalent — clearly wrong,
since word order completely changes the meaning.The Approach: Adding Position Information to Each Token
final_token_representation = token_embedding + positional_encoding
Token: "dog" (at position 2)
→ token_embedding (what "dog" means)
+ positional_encoding for position 2 (where it is)
= a representation that carries both meaning AND positionSinusoidal Positional Encoding (the Original Approach)
The original transformer paper used fixed sine and cosine functions at different frequencies to generate a unique positional pattern for each position — not learned, but mathematically constructed so that relative distances between positions are consistently represented. This isn't the only approach used today.
Modern Alternatives — Rotary Position Embeddings (RoPE)
Many current LLMs use RoPE (Rotary Position Embeddings) or similar techniques instead of the original sinusoidal approach — these encode relative position directly into the attention calculation itself, and have generally shown better performance on longer sequences. Exactly which technique a given model uses is an architecture-specific implementation detail that evolves over time; treat this as an active, evolving area rather than a single settled standard.
Practical Use Case
Positional encoding design directly affects how well a model handles long sequences and whether it can generalize to sequence lengths longer than what it was trained on — a real factor behind why some models handle very long context windows better than others.
Common Mistakes
- Assuming word order is captured by the sequence order tokens are processed in — self-attention itself processes all tokens in parallel with no inherent order; positional encoding is what supplies that information
- Assuming all modern LLMs use the exact same positional encoding scheme — it varies by model family and has evolved significantly since 2017
Interview Relevance
Q: "Why do transformers need positional encoding, when RNNs didn't?" — the expected answer: RNNs process tokens sequentially, so order is implicit in their processing; transformers process all tokens in parallel via attention, which has no inherent notion of order without it being explicitly added.
Practice Question
Explain why "positional encoding" would matter less for a bag-of-words style task (where word order is irrelevant) than for a translation task.
