Temperature controls how much randomness is introduced when the model selects its next token — low temperature favors the most likely token consistently; higher temperature allows more variety and creativity, at the cost of predictability.
What's Actually Happening
Given the same partial sentence, the model computes a probability
for many possible next tokens:
"The weather today is" →
" sunny" (35% probability)
" cloudy" (25%)
" rainy" (15%)
" nice" (10%)
...
Temperature = 0: always pick the single highest-probability token
(deterministic-ish — "sunny", every time)
Temperature = 0.7: sample from the distribution, weighted by
probability — usually "sunny" or "cloudy",
occasionally something else
Temperature = 1.5: flatten the distribution further, giving lower-
probability tokens meaningfully more chance —
more varied, more unpredictable outputChoosing a Temperature for the Task
| Task Type | Typical Temperature | Why |
|---|---|---|
| Factual Q&A, data extraction, classification | Low (0-0.3) | Consistency and reliability matter more than variety |
| General conversation, drafting | Moderate (0.5-0.8) | Some natural variation is desirable without going off-topic |
| Creative writing, brainstorming | Higher (0.8-1.2+) | Variety and unexpected combinations are the actual goal |
Important: Temperature 0 Isn't Always Perfectly Deterministic
Even at temperature 0, some providers' systems can produce slightly different outputs across identical requests due to infrastructure-level factors (like floating-point computation differences across hardware). Treat temperature 0 as "highly consistent," not as an absolute deterministic guarantee — if you need true reproducibility for testing, check your specific provider's documented guarantees.
Practical Use Case
A data-extraction feature pulling structured fields from documents should use low temperature (consistency matters, creativity doesn't help). A marketing-copy generator benefits from higher temperature to avoid repetitive, formulaic output across multiple generations.
Common Mistakes
- Using a high default temperature for tasks needing consistency (classification, extraction), causing unpredictable, hard-to-test behavior
- Assuming temperature 0 guarantees bit-for-bit identical output every time, and building tests that assume perfect reproducibility
- Adjusting both temperature and top_p simultaneously without understanding they interact — see Top-p
Interview Relevance
"Why would you set temperature to 0 for a data extraction feature?" — consistency and predictability matter far more than variety for a task with one objectively correct output.
Practice Question
Recommend a temperature setting (with reasoning) for: (1) generating 5 varied ad headline options, (2) extracting a shipping address from an email.
