LLM API costs are usually driven by token usage (see Token Cost) multiplied by request volume — but real production costs involve more moving parts than a simple per-token calculation, including retries, failed requests, and architecture choices.
The Full Cost Picture, Not Just Token Price
| Cost Factor | Often Overlooked Because... |
|---|---|
| Retried requests | A retried request is billed again — failures aren't free just because they didn't "succeed" |
| Conversation history growth | Input tokens per request grow across a long conversation, not staying flat |
| Retrieved context (RAG) | Often the largest input-token contributor, easy to underestimate during initial testing |
| Multi-step chains/agents | Each step is a separate billed call — a 5-step agent task isn't "one request's worth" of cost |
A More Realistic Cost Estimation Approach
# Naive (often wrong) estimate:
cost_per_request ≈ avg_tokens × price_per_token
monthly_cost ≈ cost_per_request × requests_per_month
# More realistic estimate accounts for:
# - retry rate (some requests are billed more than once)
# - conversation length distribution (not every request has the
# same input size)
# - actual measured token usage from a pilot/beta period, not
# just estimated averages from a handful of manual testsPractical Cost Controls
- Model selection matched to task complexity (see Model Selection)
- Prompt and context token efficiency (see Token Optimization)
- Caching repeated/similar requests where appropriate (see LLM Caching)
- Batch processing for non-time-sensitive bulk workloads (see Batch Processing)
- Ongoing cost monitoring rather than a one-time pre-launch estimate (see LLM Cost Monitoring)
Practical Use Case
Before launching a new LLM-powered feature, run a pilot with real (or realistic) usage patterns and measure actual token consumption — pre-launch estimates based on a handful of manual tests routinely underestimate real-world cost, especially for conversational or RAG-based features.
Common Mistakes
- Estimating cost from initial testing alone, without accounting for retries, conversation growth, or real usage distribution
- Not distinguishing input vs output token costs when they're priced differently, leading to inaccurate projections
- Setting cost estimates once at launch and never revisiting them as usage patterns evolve
Interview Relevance
"Your team estimated a feature would cost $500/month but it's costing $3,000. What would you investigate?" — retry rates, conversation history growth, and actual vs estimated token usage per request are the practical first places to look.
Practice Question
List three cost factors beyond simple "tokens × price" that a team should account for when estimating the cost of a new RAG-based chatbot feature.
