Retries

LLM API calls fail sometimes — rate limits, transient network issues, temporary provider-side errors. A deliberate retry strategy is what separates a resilient application from one that surfaces avoidable errors to users.

Exponential Backoff — The Standard Pattern

import time
import random

def call_with_retry(prompt, max_retries=4):
    for attempt in range(max_retries):
        try:
            return llm_client.generate(prompt=prompt)
        except TransientError as e:
            if attempt == max_retries - 1:
                raise  # out of retries — let the caller handle failure
            wait = (2 ** attempt) + random.uniform(0, 1)  # backoff + jitter
            time.sleep(wait)
    

Each retry waits longer than the last (exponential backoff), and adding a small random jitter prevents many clients from retrying at exactly the same moment and re-triggering the same rate limit together.

Not Every Error Should Be Retried

Blindly retrying every error type wastes time and can mask real bugs in your request construction.

Setting a Sensible Retry Limit

Unlimited retries risk making a failing request hang indefinitely and can compound load during an outage. A bounded number of retries (commonly 3-5) with a final clear failure/fallback path is the standard, sensible approach.

Practical Use Case

Any production feature calling an LLM API should have retry logic for transient failures — without it, a brief provider hiccup becomes a visible, avoidable user-facing error instead of a handled, invisible retry.

Common Mistakes

• Retrying non-retryable errors (like malformed requests) repeatedly, wasting time without ever succeeding
• No jitter in backoff timing, causing many clients to retry in synchronized bursts
• No maximum retry limit, risking requests that hang far longer than acceptable for the use case

Interview Relevance

"Should you retry every failed LLM API call the same way?" — no; a good answer distinguishes retryable (transient, rate-limit) from non-retryable (invalid request, auth) errors, and describes exponential backoff with jitter for the former.

Practice Question

Write pseudocode for a retry function that uses exponential backoff, respects a maximum of 4 attempts, and does NOT retry on a 400 (bad request) error.

Related Notes

• API Error Handling
• Rate Limits
• Retry Strategies (Agents)