For text generation specifically: a model is trained on enormous amounts of text to predict the next token, then that same next-token prediction is run repeatedly at inference time to generate a full response.
Step 1 — Training: Learning to Predict the Next Token
During training, the model is shown huge volumes of text and, for every position in every sentence, learns to predict what comes next:
Training example:
Input: "The quick brown fox jumps over the lazy"
Target: "dog"Repeated across billions of examples, the model's internal parameters (see Model Weights) adjust to capture grammar, facts, reasoning patterns, and style — all indirectly, purely from next-token prediction. Full detail in LLM Training.
Step 2 — Tokenization: Text Becomes Numbers
Models don't process raw text; input is first split into tokens (roughly word-pieces) and converted to numeric IDs. See Tokenization for the full mechanism.
Step 3 — Inference: Generating One Token at a Time
Prompt: "Write a haiku about the ocean"
Step 1: model predicts token 1 → "Waves"
Step 2: model predicts token 2, given prompt + "Waves" → " crash"
Step 3: model predicts token 3, given prompt + "Waves crash" → " softly"
...continues until a stop condition is reached...This is autoregressive generation — each new token is predicted based on everything generated so far, one step at a time. This is also why longer outputs take proportionally longer to generate — see LLM Inference.
Step 4 — Sampling: Not Always the Single "Best" Word
The model doesn't always pick the single most likely next token — parameters like temperature and top-p control how much randomness is introduced, which is why the same prompt can produce different outputs across runs.
Minimal API Example
# Conceptual, provider-agnostic — see LLM API for real syntax per provider
response = llm_client.generate(
prompt="Write a haiku about the ocean",
temperature=0.7,
max_tokens=50
)
print(response.text)Practical Use Case
Understanding this pipeline explains real, observable behavior: why longer prompts and outputs cost more (more tokens processed), why responses can vary between calls (sampling), and why the model can "run out of room" mid-response (see Context Window).
Common Mistakes
- Assuming the model "thinks ahead" and plans the whole response before writing — it generates one token at a time, using only what's been generated so far as context for the next step
- Expecting identical output on every run at a non-zero temperature — some randomness is often the intended, configurable behavior, not a bug
Interview Relevance
Q: "Explain, at a high level, how an LLM generates a response to a prompt." A strong answer covers: tokenization → autoregressive next-token prediction → sampling strategy → stopping condition — the four steps above, without necessarily needing transformer internals.
Practice Question
Explain why asking an LLM to generate a 2,000-word essay takes noticeably longer than asking it to answer "What is 2+2?" — in terms of the generation process, not just word count.
