TL;DR (May 2026): There is no single switch that cuts your LLM bill. There is a stack of six layers, ordered cheapest-effort-first, and each one multiplies against the ones below it. Layer 1 (prompt caching) cuts 60-90% off repeated context. Layer 2 (model routing) sends the easy 70% of traffic to a model that's 10-25x cheaper. Layer 3 (batching) takes another 50% off anything that isn't latency-sensitive. Layer 4 (context discipline) trims the tokens you never needed to send. Layer 5 (output control) attacks the tokens that cost 4-6x as much as input. Layer 6 (metering) is the one that makes the other five real, you cannot cut what you cannot see. Stacked, the worked example below goes from $200/month to $28/month on the same workload. Each layer with its own dollar math is below.
The most circulated AI-cost story of 2026 wasn't a model launch. It was a Hacker News thread titled "From $200 to $30: Five Layers of LLM Cost Optimization," and the reason it resonated is that it wasn't about a trick. It was about an order of operations. The author didn't find one clever hack. They applied a stack of boring, compounding interventions in the right sequence and watched a bill collapse by 85%.
That framing, layers, not hacks, is the correct one. Every "we cut our agent token cost by 95%" Reddit post and every "Practical Guide to Reducing AI Agent Token Costs" blog is describing some subset of the same six moves. What almost none of them do is order the moves by effort-to-payoff and show the dollar math at each step. That's what this playbook is.
The rule that makes the stack work: do them in order, and re-measure after each one. The savings multiply, not add. A 90% caching cut on top of a model that's already 10x cheaper from routing is not "90% plus 90%." It's the product. If you skip Layer 6 (metering) you'll apply Layers 1-5 blind and have no idea which one actually moved the invoice. We've watched teams "optimize" for a month and end up more expensive because they swapped to a cheaper model that doubled retries, and nobody was measuring per-request cost to catch it.
The price card every layer attaches to
All the math below uses the May 2026 flagship rates. These are the same numbers from our GPT-5.5 vs Gemini 3.1 Pro vs Claude Opus 4.7 pricing breakdown:
| Model | Input ($/1M) | Output ($/1M) | Cached input ($/1M) | Tier |
|---|---|---|---|---|
| GPT-5.5 | $5.00 | $30.00 | $0.50 | Flagship |
| Claude Opus 4.7 | $5.00 | $25.00 | $0.50 | Flagship |
| Gemini 3.1 Pro (≤200K) | $2.00 | $12.00 | $0.20 | Flagship (cheap) |
| GPT-5.2 | $1.75 | $14.00 | $0.18 | Mid |
| GPT-5-mini | $0.40 | ~$1.60 | $0.20 | Small |
| Gemini 3.1 Flash | $0.30 | ~$2.50 | $0.075 | Small |
Two structural facts to keep in front of you for the rest of this playbook. First, output tokens cost 4-6x input tokens on every flagship (Opus is $5 in / $25 out, 5x; GPT-5.5 is $5 / $30, 6x; Gemini 3.1 Pro is $2 / $12, 6x). That asymmetry is why Layer 5 exists. Second, the gap between a flagship and a small model is 10-25x on input (Opus $5 vs Gemini Flash $0.30 is ~17x). That gap is why Layer 2 exists. Everything else is exploiting one of those two facts.
The reference workload
To make the stacking visible, we'll carry one workload through all six layers. A B2B SaaS support-and-research agent:
- 50,000 requests/month.
- Each request: a 6,000-token system prompt (instructions + tone + tool reference), a 12,000-token RAG context block (retrieved docs), a ~400-token user question.
- Output: averaging ~1,200 tokens.
- Currently: everything goes to Claude Opus 4.7, uncached, no routing, no batching, full retrieved context, no output cap.
Starting cost: per request, 18,400 input tokens at $5/M = $0.092, plus 1,200 output at $25/M = $0.030. Total $0.122/request. At 50K requests: $6,100/month. (We'll express the per-1,000-requests figure too, $122 per 1K, so the percentages are easy to track. Scaled to the HN thread's hobby numbers it's the same $200-shaped story; the percentages are what transfer.)
Layer 1, Prompt caching (effort: hours, saving: 50-75% here)
Start here because it is the highest payoff per hour of work and requires zero behavioral change. The system prompt (6K tokens) and a large slice of the RAG context are identical or stable across requests within a session. Mark them cacheable.
This article won't re-derive the caching math, that's a whole topic with its own breakeven tables, TTL choices, and four classic mistakes, all worked out in our dedicated prompt-caching cost guide. The short version: Anthropic charges 0.10x for cache reads (a 90% discount), OpenAI auto-caches at ~50% off, Gemini reads at ~25% of base. Read that article before you ship caching to production.
For our workload, assume the 6K system prompt caches cleanly and ~8K of the 12K RAG block is reused within the 5-minute window (the stable corpus chunks; the query-specific chunks vary). That's 14K of 18.4K input tokens served from cache at $0.50/M instead of $5/M.
- Cached input: 14,000 * $0.50/M = $0.007
- Fresh input (4,400 tok): * $5/M = $0.022
- Output unchanged: $0.030
- Per request: $0.059 (down from $0.122)
New monthly cost: $2,950. That's a 52% cut from one configuration change, no model swap, no UX change. On a workload with a more stable prefix you'd see 70%+. Re-measure, then move down.
Layer 2, Model routing / cascade (effort: a day, saving: 40-60% of what's left)
The single biggest source of waste in 2026 is sending every request to a flagship when most requests don't need one. A support agent's traffic is not uniformly hard. A large fraction, "where do I find my invoice," "reset my password," "is feature X supported", is answerable by a model 10-25x cheaper with no quality loss a customer would ever notice.
The pattern is a cascade: route every request to a cheap model first (Gemini 3.1 Flash or GPT-5-mini), and escalate to the flagship only when a confidence signal says the cheap model is out of its depth. The confidence signal can be a classifier, a self-reported "I'm not sure" from the small model, a schema-validation failure, or a simple keyword/complexity heuristic. The Reddit "reduce Agentic AI token cost by 95%" posts are almost always describing an aggressive version of this.
Conservative split for our workload: 70% of requests resolve on Gemini 3.1 Flash, 30% escalate to Claude Opus 4.7. Caching (Layer 1) still applies on both tiers. Flash cached input is $0.075/M, output ~$2.50/M.
| Tier | Share | Per-request cost (cached) | Monthly |
|---|---|---|---|
| Gemini 3.1 Flash (easy 70%) | 35,000 req | ~$0.0046 | $161 |
| Claude Opus 4.7 (hard 30%) | 15,000 req | $0.059 | $885 |
| Escalation overhead (double-run on the 30%) | 15,000 req | ~$0.0046 | $69 |
The escalation overhead row is the honest cost of a cascade: when you escalate, you've already paid for the cheap attempt. Total: $1,115/month, a 62% cut from the post-caching number. The trap is over-escalating, if your confidence threshold is timid and 60% of traffic escalates, the double-run overhead eats the savings. Tune the threshold against measured quality, not vibes, and watch the escalation rate as a first-class metric.
Layer 3, Batching & async (effort: a day, saving: ~50% on eligible traffic)
Not every request needs an answer in 800ms. Overnight report generation, bulk document classification, embedding backfills, eval runs, and any "the user will check back later" workflow can go through a provider's batch API. OpenAI, Anthropic, and Google all offer a batch tier at roughly a 50% discount in exchange for asynchronous delivery within a window (typically up to 24 hours).
Suppose 25% of our 50K requests are async-eligible, nightly summaries, ticket-trend digests, re-classification jobs. Those 12,500 requests, currently mostly on the Opus tier from Layer 2's split, move to batch at half price. The 30% that were on Opus include these, so roughly 3,750 Opus requests/month shift to batch.
- 3,750 Opus requests at $0.059 → $0.0295 batched: saves ~$111/month.
- Plus the Flash-tier async jobs at half their tiny cost: a few more dollars.
New total: roughly $1,000/month. The percentage looks small here because batching only touches the async slice, but on workloads that are mostly async (data pipelines, content generation farms, nightly analytics) batching is a near-free 50% and jumps above Layer 2 in priority. Order your layers by your traffic shape: a real-time chat product reorders 2 and 3; a bulk-ETL product runs 3 first. Some providers also publish off-peak or committed-throughput discounts, worth checking your contract if your spend is large enough to negotiate.
Layer 4, Context discipline / RAG hygiene (effort: ongoing, saving: 20-40%)
This is where the "cutting LLM context costs 70%+" threads live, and it's the layer most teams skip because it's unglamorous. You are almost certainly sending tokens the model never uses.
The three biggest offenders:
- Over-retrieval. RAG pipelines default to top-k = 10 or 20 chunks "to be safe." Most answers use 2-3. Every unused chunk is input tokens you paid for and the model had to read past. Drop k, add a reranker, and trim each chunk to the relevant passage instead of shipping the whole document section.
- Bloated system prompts. The 6K-token system prompt in our example accreted over a year, every incident added a "never do X" line. Half of it is dead. A focused 2.5K-token prompt usually performs the same or better (less to dilute attention) and caches just as well.
- Full conversation history replay. Agents that re-send the entire transcript every turn balloon linearly. Summarize old turns, use a sliding window, or externalize the durable facts to a persistent memory layer like memnode so the agent recalls decisions by key instead of replaying the whole transcript. This is the single biggest lever on long agent loops.
For our workload, assume RAG hygiene trims the 12K context block to 7K (rerank + tighter chunks) and the system prompt drops from 6K to 3K. That's 8K fewer tokens per request. Because most of those tokens were cached reads at $0.50/M, the dollar saving on the cached portion is modest, but the fresh, query-specific tokens shrink too, and the smaller prompt means cheaper escalations on the Opus tier. Net effect on our stack: roughly 15-20% off the running total, landing around $820/month. On uncached workloads this layer is far bigger, trimming raw input tokens you pay full freight for is a direct 1:1 saving.
The discipline part matters more than the one-time cleanup: retrieval configs drift, prompts re-bloat, and a Layer-6 dashboard that graphs average input tokens per workload is how you catch the regression before it's a line item.
Layer 5, Output control (effort: hours, saving: 10-30%)
Output tokens cost 4-6x input. A flagship that costs $5/M to read costs $25-30/M to write. Yet output is the least-governed part of most pipelines, teams set max_tokens to 4096 "just in case" and let the model ramble.
The levers, all cheap to apply:
- Set a real
max_tokens. If your answers are 1-2 paragraphs, cap at 400, not 4096. This is a ceiling, not a target, it doesn't truncate good answers, it stops runaway ones. - Structured / JSON output. Asking for a tight schema instead of prose eliminates the "Certainly! Here's a detailed breakdown..." preamble and the closing summary the model loves to add. Structured output is routinely 30-50% fewer output tokens for the same information.
- Stop sequences. Cut generation the moment the useful part is done. If you only need the first code block, stop on the closing fence.
- Prompt for brevity explicitly. "Answer in under 3 sentences unless asked to elaborate" measurably shortens output and customers usually prefer it.
For our agent, tightening output from an average 1,200 tokens to ~700 (cap + brevity instruction + structured fields where applicable) cuts output cost by ~40% on every request. Since output is now a large fraction of the per-request cost (caching crushed the input side), this matters more than it looks. New total: roughly $640/month.
Layer 6, Metering & visibility (the one that makes the other five real)
Here is the uncomfortable truth that every cost-optimization thread eventually admits: the teams that successfully cut their bill are the teams that could see their bill, broken down to the unit that maps to a decision. The teams that failed were optimizing against a single number on a provider dashboard 30 days delayed.
You cannot cut what you cannot see. Concretely, you need cost broken out three ways:
- Per model. So when you flip Layer 2's router, you can confirm the cheap tier is actually absorbing 70% of traffic and not silently failing over to the flagship. We've seen a "router" that escalated 90% of requests because of a misconfigured confidence check, the bill went up and nobody noticed for three weeks.
- Per customer. So you find the 3% of customers generating 40% of token spend and either rate-limit them, move them to a higher plan, or fix whatever workflow is looping. Per-customer cost is also what makes usage-based billing and sane limits possible in the first place.
- Per feature. So you know whether the "AI summary" button is paying for itself or quietly burning more than the revenue it drives. Feature-level cost is the data that lets a PM kill a feature or a CFO approve more budget.
This is what UsageBox is for. A thin wrapper around your model calls records token counts, model name, customer ID, and feature tag on every request, and the dashboard turns that into cost-per-model, cost-per-customer, and cost-per-feature in real time. When you ship Layer 1 caching, you watch the cache-hit rate climb live instead of waiting for the invoice. When you tune Layer 2's router, you watch the tier split move. When a deploy accidentally disables caching or flips the router, you get an alert the same day, not a surprise at month-end. The five layers above are the levers; metering is the instrument panel that tells you which way to pull them and whether it worked. Build Layer 6 first, even though it's listed last, because it's the prerequisite for trusting every other number in this article.
The stacked result: $6,100 → $640
Tracking our reference workload down the stack:
| Layer | Intervention | Monthly cost | Cumulative cut |
|---|---|---|---|
| Baseline | All Opus, uncached, no controls | $6,100 | , |
| 1 | Prompt caching | $2,950 | -52% |
| 2 | Model routing / cascade | $1,115 | -82% |
| 3 | Batching async slice | $1,000 | -84% |
| 4 | Context / RAG hygiene | $820 | -87% |
| 5 | Output control | $640 | -89% |
From $6,100 to $640 a month, an 89% cut, with no model that a customer would call "worse" on the requests that needed quality, no architecture rewrite, and no new headcount. Scaled to the HN thread's hobbyist numbers it's the same shape: $200 to roughly $28. The percentages travel; the absolute numbers are whatever your traffic is.
Two honest caveats. First, these are illustrative round numbers, your prefix stability, your easy/hard split, and your async share will move every row. That's exactly why Layer 6 isn't optional. Second, the layers interact: aggressive Layer 4 trimming can hurt quality and push more traffic to escalate in Layer 2, raising cost. Optimize, measure, repeat. The stack is a loop, not a checklist.
The honest take
"How do I reduce my LLM API costs" almost always gets answered with a single tip, "use a cheaper model" or "turn on caching", and that's why most teams' bills stay high. The win isn't any one layer. It's applying all six in the right order, measuring between each, and treating the cheapest-effort moves (caching, output caps) as table stakes before reaching for the harder ones (routing, RAG re-architecture).
The teams that will have a sustainable AI roadmap in the back half of 2026 are the ones that put cost-per-customer and cost-per-feature on the same dashboard as revenue and DAU, then run this stack as a quarterly discipline. The teams that won't are the ones still reading the provider's month-end total and guessing. The bill is rarely unsustainable. The visibility usually is.
Related reading
- Cut Your AI API Bill 70-90% with Prompt Caching, Layer 1 in full: TTL breakeven math, the four mistakes, and per-provider caching mechanics
- GPT-5.5 vs Gemini 3.1 Pro vs Claude Opus 4.7 Pricing, the raw per-token rates this whole playbook attaches to, and which model to route to
- How to Cap AI Coding Costs per Engineer, the FinOps governance layer for internal developer-tool spend
- Claude API Billing in 2026, per-model limits and the metering schema behind Layer 6