Rate Limits

Rate limits cap how many requests (and/or tokens) your application can send to an API within a given time window — a real, expected constraint that production systems need to design around, not an edge case to handle as an afterthought.

Common Rate Limit Dimensions

Exact limits vary by provider, account tier, and model — always check current, specific documentation rather than assuming a fixed number.

Handling a Rate Limit Response

import time

def call_with_rate_limit_handling(prompt, max_retries=3):
    for attempt in range(max_retries):
        response = llm_client.generate(prompt=prompt)
        if response.status_code == 429:  # "too many requests"
            wait_time = response.retry_after or (2 ** attempt)
            time.sleep(wait_time)
            continue
        return response
    raise Exception("Rate limit retries exhausted")

Respecting a provider's suggested "retry-after" value (when returned) is generally better than a fixed guess — see Retries for the broader retry strategy this fits into.

Designing Around Rate Limits Proactively

• Client-side throttling — deliberately pacing requests to stay under known limits rather than sending a burst and reacting to failures
• Queuing — for high-volume batch-style workloads, a queue that processes at a controlled rate rather than firing all requests at once
• Monitoring usage — tracking actual request/token rate against known limits before you hit them, not just reacting to 429 errors after the fact

Practical Use Case

A feature that suddenly goes viral or gets a traffic spike needs to handle rate limiting gracefully (queued, degraded, or delayed responses) rather than simply failing outright for a portion of users — this is a real production reliability concern, not a rare edge case.

Common Mistakes

• No handling at all for rate-limit responses, causing visible failures for users during traffic spikes
• Retrying immediately in a tight loop after hitting a rate limit, worsening the problem rather than backing off
• Not monitoring actual usage against limits until a limit is unexpectedly hit in production

Interview Relevance

"How would you design a system to gracefully handle LLM API rate limits under high load?" — throttling, queuing, exponential backoff on retries, and proactive monitoring are the expected building blocks of a strong answer.

Practice Question

Design a simple client-side throttling strategy for a batch job that needs to process 10,000 requests against an API with a 500 RPM limit.

Related Notes

• Retries
• API Error Handling
• Batch Processing