An LLM works in two distinct phases: an expensive, one-time training phase that produces the model's weights, and a fast, repeated inference phase that uses those fixed weights to generate responses to new prompts.
Training vs Inference — The Distinction That Matters Most
| Training | Inference | |
|---|---|---|
| When it happens | Once (or periodically, for model updates), before the model is released | Every time someone sends a prompt |
| What it does | Adjusts millions/billions of weights to minimize prediction error | Uses the already-fixed weights to predict the next token, repeatedly |
| Cost | Extremely high — large compute clusters over weeks | Much lower per request, but scales with usage volume |
| Who does it | The model provider (or you, if fine-tuning) | Anyone calling the model via an API or running it locally |
When you call an LLM API, you are only ever doing inference — the weights don't change based on your conversation. See LLM Inference for the mechanics of that process.
Training Has Its Own Stages
Raw text (internet, books, code)
↓ pretraining
Base model (fluent, but not great at following instructions)
↓ instruction tuning
Instruction-following model
↓ alignment
Aligned, deployable model (what you interact with via chat)Each stage is covered separately: Pretraining, Instruction Tuning, Alignment.
Where Prompting and Fine-Tuning Fit In
| Approach | Changes the model's weights? | When to use |
|---|---|---|
| Prompting | No | Default choice — flexible, no training cost, works immediately |
| RAG | No — adds external context at inference time | When answers need to be grounded in specific, current, or private data |
| Fine-tuning | Yes — further trains weights on your own data | When you need consistent behavior/format/style that prompting alone can't reliably achieve — see Fine-Tuning vs Prompting |
Common Mistakes
- Assuming a conversation "teaches" the model anything permanent — nothing about an API conversation updates the underlying weights; the illusion of learning within a session comes entirely from context being re-sent each turn
- Reaching for fine-tuning to add knowledge the model doesn't have — that's what RAG is for; fine-tuning is much better suited to changing behavior/style than injecting facts
Interview Relevance
"Does chatting with an LLM change the model?" is a deceptively simple, very common screening question — the correct answer is no, and being able to explain the training/inference split clearly is a good signal.
Practice Question
A user says "I told the chatbot my name yesterday, why doesn't it remember today in a new conversation?" Explain the answer using the training/inference distinction.
