LLM Inference

Inference is what happens every time you send a prompt to an LLM — the trained, fixed model processes your input and generates a response, one token at a time.

Two Phases: Prefill and Decode

This is the technical reason a long output takes noticeably longer to generate than a long input takes to process — prefill scales efficiently, decode does not.

KV Caching — Why Generation Doesn't Get Slower Per Token

Without optimization, generating each new token would require re-processing the entire conversation so far from scratch. In practice, inference systems cache the intermediate attention calculations (the "key" and "value" vectors) from previous tokens, so generating token N only requires new computation for that one token, reusing the cached results for everything before it. This is a major reason modern LLM inference is practically usable at all for longer outputs.

Minimal Example — Provider-Agnostic

# Conceptual; see LLM API for real provider-specific syntax
response = llm_client.generate(
    prompt="Explain KV caching in one sentence.",
    max_tokens=50
)
print(response.text)
print(response.usage.output_tokens)  # generation cost/time scales with this

Practical Cost & Latency Implications

• Cost scales with total tokens processed (input + output) — see LLM API Cost
• Latency scales primarily with output length, since decode is sequential — a request asking for a long response will feel slower than one asking for a short one, even with an identical prompt length
• Streaming (see Streaming) doesn't make generation faster overall, but improves perceived latency by showing tokens as they're produced instead of waiting for the full response

Common Mistakes

• Assuming inference latency is dominated by input length — for most requests, output length matters more, since decode is the sequential bottleneck
• Not using streaming for user-facing long-form generation, resulting in a poor perceived-latency experience even when total generation time is unavoidable

Interview Relevance

Q: "Why is generating a 500-word response slower than generating a 50-word one, even from the same prompt?" — the expected answer covers autoregressive, sequential decoding — each token depends on every token before it, so more output tokens means more sequential steps.

Practice Question

Explain why increasing max_tokens on an API call doesn't guarantee a slower response, but a model that actually generates more tokens will be slower.

Related Notes

• Streaming
• LLM API Cost
• Context Window