Top-p (nucleus sampling) is another way to control output randomness — instead of adjusting the whole distribution's "temperature," it restricts the model to sampling only from the smallest set of tokens whose combined probability reaches a threshold p.
How Top-p Works, Concretely
Token probabilities for "The weather today is":
" sunny" 35%
" cloudy" 25%
" rainy" 15%
" nice" 10%
" cold" 8%
" odd" 2%
... (many more, tiny probabilities)
top_p = 0.7:
Include tokens until cumulative probability ≥ 70%:
"sunny" (35%) + "cloudy" (25%) + "rainy" (15%, brings total to 75%)
→ only these 3 tokens are eligible; sample among them
→ everything else, including " odd", is excluded entirelyThis differs from temperature, which reshapes probabilities across the entire vocabulary without hard-excluding any token; top-p draws a hard cutoff, excluding low-probability tokens completely.
Temperature vs Top-p — Different Mechanisms, Similar Goal
| Temperature | Top-p | |
|---|---|---|
| Mechanism | Reshapes the probability distribution across all tokens | Restricts the candidate pool to a cumulative-probability threshold |
| Low-probability tokens | Still possible, just less likely | Can be excluded entirely if outside the threshold |
| Common practice | Adjust one or the other, not both aggressively at once | Same |
Most guidance suggests tuning one parameter at a time rather than adjusting both temperature and top-p simultaneously — combining aggressive changes to both makes output behavior harder to predict and tune deliberately.
Practical Use Case
Top-p is useful when you want to eliminate clearly implausible tokens entirely (avoiding rare, low-quality completions) while still allowing reasonable variety among the plausible options — a middle ground between the determinism of temperature 0 and the wide-open randomness of high temperature.
Common Mistakes
- Adjusting both temperature and top-p aggressively at the same time, making it hard to reason about which change caused a behavior shift
- Assuming top-p and temperature do the same thing — they're related but mechanically distinct approaches to controlling randomness
Interview Relevance
"What's the difference between temperature and top-p sampling?" — temperature reshapes the whole distribution; top-p restricts to a cumulative-probability-based candidate set, potentially excluding tokens entirely.
Practice Question
Explain what would likely happen to output diversity if top_p were set very low (e.g. 0.1) versus very high (e.g. 0.99), holding temperature constant.
