TL;DR (July 2026): A widely-shared writeup this month asked the right question - "why did one day of AI cost more than a month of servers?" - and the answer was a retry storm. When a model call times out or returns a 5xx, well-meaning retry logic fires again, and again. Every attempt that reaches the provider is a fresh, fully-billed generation, even the ones your app throws away. A slow provider hour plus aggressive retries plus concurrency multiplies your token spend by the retry count - and unlike a server bill, there is no fixed ceiling. The fix is not "stop retrying"; it is retry with a cap, exponential backoff and jitter, a circuit breaker, and idempotency, plus a meter that counts attempts and not just successes so the storm shows up the same day instead of on the invoice.
Server cost has a ceiling: you provisioned N machines, and a bad day pins them at 100% but does not conjure more. LLM cost has no such ceiling, because every call is metered and every retry is a new call. That difference is the whole story. A retry storm turns a reliability event into a billing event, and the two do not look alike in your dashboards - the app may even appear healthy, because the user eventually got a good answer on attempt four.
How a retry storm bills you
The mechanics are mundane, which is why they bite. Consider a request that normally costs 8,000 tokens:
- The provider has a slow hour. Your client's timeout is 30 seconds; generations start taking 35.
- Every request times out and retries. Most APIs bill for work performed up to the point you disconnected, and each fresh attempt is a full new generation - so one logical request becomes 3 to 5 billed ones.
- Your retries have no jitter, so thousands of clients retry in lockstep, hammering an already-slow provider and making the slow hour slower - the classic thundering herd.
- Concurrency multiplies it: 500 in-flight requests each retrying 4 times is 2,000 billed generations for 500 answers.
None of these steps is a bug in isolation. Timeouts are correct. Retries are correct. Together, without limits, they produce a bill that is 4x normal for a day when nothing actually shipped more value. A related failure mode is retrying on a 429 rate-limit response, which can burn both your rate budget and, on some setups, real tokens - retry the wrong status codes and you pay to be told "slow down."
The five controls that cap it
- A hard retry cap. Two or three attempts, not "until success." Unbounded retries are how a slow hour becomes an unbounded bill.
- Exponential backoff with full jitter. Backoff spaces the attempts; jitter de-synchronizes your fleet so you do not thundering-herd the provider. Both matter - backoff without jitter still stampedes.
- A circuit breaker. After a threshold of failures, stop calling the provider for a cool-down window and fail fast. A tripped breaker converts a four-hour storm into a four-minute one.
- Retry only idempotent, retry-safe statuses. Retry timeouts and 5xx; do not blindly retry 400s or content-policy refusals - they will fail again and bill again.
- Idempotency keys. Where the provider supports them, an idempotency key lets a retried request return the first result instead of running a new generation - the difference between paying once and paying per attempt.
Why the meter has to count attempts
Here is the part most teams miss: if your cost tracking only records successful, returned generations, a retry storm is invisible until the provider invoice lands weeks later. The thrown-away attempts were billed but never logged on your side, so your internal "AI cost" number understates reality and the storm looks like nothing happened. To catch it the same day, the meter has to record every attempt - success, timeout, and discard alike - and reconcile that against the provider's number. That reconciliation is exactly where the gap shows up, and it is the same discipline behind why usage metering needs its own event store rather than a success-only log.
Two numbers make the storm legible once you meter attempts: attempts-per-success (should hover near 1.0; a spike to 3-4 is a storm in progress) and billed-tokens-per-delivered-answer (the true unit cost, retries included). Alert on the first, budget with the second.
The honest take
Retry storms are not exotic; they are the default outcome of correct-looking retry code meeting a metered API and a slow provider hour. The reason they surprise people is that server intuition says load has a ceiling and AI cost does not. Cap the retries, jitter the backoff, break the circuit, and meter the attempts. The goal is not zero retries - retries make your app reliable - it is making sure a reliability feature can never quietly become an unbounded line item. When one bad hour costs more than a month of servers, the fix was three config values and one meter that counted the calls you threw away.