LLM APIs are typically billed per token — usually a separate rate for input and output tokens, per model. Understanding this pricing model is the difference between a predictable AI feature budget and an unpleasant billing surprise.
The Basic Cost Formula
request_cost = (input_tokens × input_price_per_token)
+ (output_tokens × output_price_per_token)
# Illustrative only — actual rates vary by provider and model,
# and change over time; always check current provider pricing.Why Costs Scale Faster Than Expected
Example: a support chatbot handling 10,000 conversations/day,
averaging 8 turns each, with growing conversation history
resent on every turn.
Turn 1: ~500 input tokens
Turn 8: ~4,000 input tokens (full history resent)
Average input tokens per turn grows across the conversation —
so total daily cost isn't just "10,000 × one average request,"
it compounds with conversation length unless history is
managed (trimmed/summarized).Practical Cost Levers
| Lever | Effect |
|---|---|
| Model selection | Smaller models typically cost meaningfully less per token — see Model Size |
| Trimming conversation history | Directly reduces input tokens on later turns |
| Reducing retrieved context (RAG) | Fewer, more relevant chunks lowers input tokens per request |
| Capping max_tokens | Bounds worst-case output cost per request |
| Caching | Avoids paying for repeated identical/near-identical requests — see Semantic Caching |
Practical Use Case: Monitoring Before Optimizing
Before optimizing cost, you need visibility into where it's actually going — see LLM Cost Monitoring. Guessing which feature is expensive, without per-request token tracking, usually leads to optimizing the wrong thing.
Common Mistakes
- Estimating cost from a handful of manual tests instead of tracking real token usage in production across actual user behavior
- Not accounting for conversation history growth when projecting cost at scale — per-conversation cost compounds as history lengthens
- Defaulting to the most capable (and most expensive) model for every request type, regardless of whether the task actually needs it
Interview Relevance
"How would you estimate the monthly cost of a new LLM-powered feature before launch?" — a strong answer involves estimating typical input/output token counts per request, expected request volume, and multiplying against current provider pricing — not guessing.
Practice Question
A feature averages 1,500 input tokens and 300 output tokens per request, at 50,000 requests/day. Using illustrative rates of $3 per million input tokens and $15 per million output tokens, estimate the daily cost.
