Decoder-Only Transformers

Most modern general-purpose LLMs use a decoder-only architecture — a single stack of transformer blocks with causal (masked) self-attention, trained purely to predict the next token. No separate encoder, no cross-attention — just one simpler, unified structure.

Why "Decoder-Only" Is Simpler

Encoder-decoder: two stacks, two kinds of self-attention,
                 plus cross-attention between them

Decoder-only:    one stack, one kind of self-attention
                 (causal/masked), no cross-attention needed

There's no separate "input phase" and "output phase" architecturally — the entire prompt and the generated response live in the same single sequence, processed by the same stack.

How Prompting Fits Into a Decoder-Only Model

Everything — your prompt AND the model's generated response —
is just one continuous token sequence to a decoder-only model:

["Explain", " photosynthesis", " in", " one", " sentence", ":",
 " Photosynthesis", " converts", " sunlight", ...]
 └──────── your prompt ────────┘ └── model's generated tokens ──┘

The model doesn't structurally distinguish "prompt" from
"response" — it's simply predicting the next token given
everything before it, whether that's your input or its own
prior output.

This is also why the model's own generated tokens become part of the context for predicting the next generated token — autoregressive generation, one continuous sequence.

Causal Masking Is What Makes This Work for Generation

Because self-attention here is causal (each token can only attend to itself and earlier tokens — see Self-Attention), the model can be trained efficiently on full sequences in parallel (predicting every position's next token simultaneously during training) while still behaving correctly at generation time, where tokens genuinely don't exist yet.

Practical Use Case

This unified structure is exactly why the same decoder-only model can flexibly handle chat, completion, code generation, and more, purely through how you structure the prompt — there's no architectural distinction between "translation mode" and "chat mode" the way an encoder-decoder setup might imply.

Common Mistakes

• Assuming decoder-only means "worse" or "simpler in a limiting sense" — it's simpler structurally, but this design is what most flagship modern LLMs are built on, not a lesser alternative
• Assuming the model architecturally "knows" where your prompt ends and its response begins — it's all one sequence; role separation (system/user/assistant) is typically handled by formatting conventions in the prompt, not a hard architectural boundary

Interview Relevance

"Why do most modern LLMs use decoder-only architectures instead of encoder-decoder?" — simplicity, training/inference efficiency, and flexibility across many task types without needing a distinct "input vs output" structure are the key points.

Practice Question

Explain why, in a decoder-only model, the model's own previously generated tokens are treated the same way as the original prompt tokens during the next generation step.

Related Notes

• Encoder-Decoder
• LLM Architecture
• LLM Inference