TL;DR (May 2026): The $/M-token sticker price is the worst predictor of your actual LLM bill. Output costs 4-6x input across every flagship, so output-heavy workloads blow past the headline. The 90% caching discount you're probably not using, RAG context you pay for but never read, retries you don't count, the 50% batch discount you're leaving on the table, and tokenizer differences that make the same paragraph cost 20% more on one provider than another, these six gaps routinely turn a "$2/M" model into an effective $9-12/M. We work a full sticker-vs-real example below: a workload with a $1,400 headline lands at $3,900 actual. The fix isn't picking a cheaper model. It's measuring the real per-call cost.
Every LLM comparison opens with the same table: input price, output price, context window. Then someone picks the lowest number and calls it a procurement decision. The trending Hacker News thread, "Stop comparing price per million tokens: the hidden LLM API costs", got it right, and the follow-up comment got it sharper: most gateway comparisons are backwards. The answer isn't the cheapest sticker. It's the workload shape that determines what you actually pay.
Here is the uncomfortable part: two teams can run the identical model at the identical published rate and see a 3x difference in their monthly bill. The price card didn't lie to them. They just read the wrong column. This article is the six columns nobody reads, each with the dollar math worked out, and then a head-to-head where a $1,400 headline becomes a $3,900 invoice.
The price card everyone starts from
For grounding, here are the May 2026 flagship rates. These are the same numbers we use in the flagship pricing breakdown. Watch the output column, it does most of the damage in this article.
| Model | Input ($/1M) | Output ($/1M) | Output / input ratio | Cached input ($/1M) |
|---|---|---|---|---|
| GPT-5.5 | $5.00 | $30.00 | 6.0x | $0.50 |
| Claude Opus 4.7 | $5.00 | $25.00 | 5.0x | $0.50 |
| Gemini 3.1 Pro (≤200K ctx) | $2.00 | $12.00 | 6.0x | $0.20 |
| GPT-5.2 | $1.75 | $14.00 | 8.0x | $0.18 |
Notice that the "cheapest" model by input price (Gemini 3.1 Pro at $2/M) and the model with the gentlest output ratio (Claude Opus at 5.0x) aren't the same model. Which one is actually cheaper for you depends entirely on the shape of your traffic, and that's exactly the thing the sticker price hides. Let's take the hidden costs one at a time.
Hidden cost 1: output-token skew
Input and output are not priced the same. They're not even close. Output runs 4-6x the input rate on every flagship, $30/M vs $5/M on GPT-5.5, $25/M vs $5/M on Claude Opus, $12/M vs $2/M on Gemini. When people quote "the $5 model," they're quoting the cheap half and ignoring the expensive half.
This matters because the sticker price implies a symmetric world and your workload is asymmetric. A summarization job is input-heavy (cheap). A code-generation, long-form-writing, or reasoning-with-visible-chain job is output-heavy (expensive). The same nominal "$5 model" costs radically different amounts depending on which way your tokens flow.
Worked example. Take GPT-5.5 and two workloads, each 10,000 requests/month, each moving 4,000 total tokens per request:
- Input-heavy (3,500 in / 500 out): 35M input at $5/M = $175; 5M output at $30/M = $150. Total: $325/month.
- Output-heavy (500 in / 3,500 out): 5M input at $5/M = $25; 35M output at $30/M = $1,050. Total: $1,075/month.
Same model. Same token count per request. Same request volume. The bill is 3.3x higher purely because the tokens flow the other direction. If you sized your budget off "$5/M tokens" and your workload turned out to be output-heavy, you're already 2-3x over before any of the other five hidden costs even apply. Reasoning models make this worse: a visible chain-of-thought is billed as output, so a model that "thinks out loud" can quietly triple your output volume per answer.
Hidden cost 2: the caching discount you're not using
Cached input reads cost 10% of the base input rate on Anthropic and Gemini's explicit caches, and roughly 50% off on OpenAI's automatic cache. That's the single biggest lever on the bill, and most teams hit a cache-hit rate of 5-15% when 80-95% is achievable. We cover the full mechanics in the prompt-caching cost guide; here the point is narrower. The sticker price assumes you pay full input rate for every token, every time. If your workload re-sends a stable system prompt, tool definitions, or a RAG corpus across requests, and almost every production workload does, then the full-rate sticker is overstating your achievable cost, and your actual bill is overstating it right back because you didn't turn caching on.
Worked example. An agent with a 30K-token stable prefix (system prompt + tools + retrieved context), 50,000 calls/month, on Claude Opus 4.7:
- No caching: 50,000 × 30K × $5/M = $7,500/month on the prefix alone.
- With caching (90% of reads hit the warm cache at $0.50/M, 10% pay a $6.25/M write): roughly 45,000 reads × 30K × $0.50/M = $675, plus 5,000 writes × 30K × $6.25/M = $937. Total: ~$1,612/month.
That's a 79% cut on the prefix, from one config change with zero behavioral impact on users. The gap between $7,500 and $1,612 isn't a different model or a different rate, it's the same rate, read correctly. Every dollar of that $5,888/month delta was invisible on the price card.
Hidden cost 3: context and RAG bloat
You pay for every token you put in the context window, including the ones the model never uses. RAG pipelines are the worst offender: a naive top-k retrieval stuffs 8-12 chunks into the prompt to be safe, when the answer lived in one of them. You paid input rate for the other ten.
This is the hidden cost that scales with your "just add more context" instinct. Larger k, longer chunks, more documents, full conversation history replayed every turn, each is a token you're billed for whether or not it changed the answer. The sticker price has no opinion about retrieval quality; your invoice does.
Worked example. A RAG chatbot on Gemini 3.1 Pro, 100,000 queries/month, 500-token question + 1,000-token answer. Compare a tight retriever (3 chunks × 800 tokens = 2,400 context tokens) against a lazy one (12 chunks × 800 = 9,600):
- Tight (2,900 in / 1,000 out): 290M input at $2/M = $580; 100M output at $12/M = $1,200. Total: $1,780/month.
- Lazy (10,100 in / 1,000 out): 1,010M input at $2/M = $2,020; output $1,200. Total: $3,220/month.
The lazy retriever costs $1,440/month more, an 81% premium on the same answers, because nine of twelve chunks were dead weight. A reranker that trims k from 12 to 3 pays for itself many times over, but you'd never see the opportunity from a price-per-token comparison, because the per-token price is identical. The waste is in the token count, and that's a property of your pipeline, not the price card.
Hidden cost 4: retries, timeouts, and rate-limit overage
Every failed call you retry is billed. A 529/overloaded response that your client library silently retries three times bills you for the input on all three attempts (and any partial output streamed before the failure). Timeouts mid-generation are the cruelest: you pay for output tokens that were produced and then thrown away because the connection dropped before the full response landed.
The sticker price assumes one successful call per logical request. Real production sees retry multipliers of 1.05x to 1.4x depending on how aggressive your backoff config is and how close you run to your rate limits. Teams that set max_retries=5 with no jitter and run hot against their TPM ceiling routinely pay 20-30% more than their "happy path" cost model predicts.
Worked example. A workload with a $4,000/month happy-path cost, running near its rate limit so that 12% of calls hit a 429 and retry once before succeeding, plus a 3% timeout rate where ~60% of the output was already generated and billed before the drop:
- Retry overhead: 12% of calls re-send their full input. If input is ~40% of the bill (~$1,600), that's 12% × $1,600 = $192/month of pure waste.
- Timeout overhead: 3% of calls bill ~60% of output for nothing. If output is ~60% of the bill (~$2,400), that's 3% × 60% × $2,400 = ~$43/month, plus the cost of the full retry that follows.
- Total invisible overhead: roughly $250-350/month, or 6-9% on top of the sticker-derived budget, and it grows the closer you run to your limits.
None of this appears on a per-token comparison. It appears on your invoice as "more tokens than we sent," and it's almost always traceable to a backoff config and a rate-limit headroom decision, not the model price.
Hidden cost 5: batch vs real-time pricing
Both OpenAI and Anthropic offer a flat 50% discount on asynchronous batch processing, requests submitted to the Batch / Message Batches API and returned within 24 hours. Gemini offers a comparable batch discount on its async path. This is the inverse of a hidden cost: it's a hidden saving that the real-time sticker price quietly excludes, and most teams pay full freight for work that was never latency-sensitive in the first place.
Embedding backfills, nightly summarization, eval runs, content classification, document enrichment, none of these needs a sub-second response. Running them through the synchronous API at full price is a self-inflicted 2x. The sticker price you compared against is the real-time price; for any deferrable workload, the relevant number is half of it.
Worked example. A nightly classification job: 20M input + 5M output tokens/day on GPT-5.5, 30 days/month:
- Real-time: 600M input at $5/M = $3,000; 150M output at $30/M = $4,500. Total: $7,500/month.
- Batch (50% off): $3,750/month.
$3,750/month saved on a workload where nobody is waiting on the response. If you priced this job off the published real-time rate and ran it synchronously because that's how the SDK example was written, you doubled the bill for no benefit. The batch discount is the easiest money in this entire article, and it's exactly the column the sticker comparison omits.
Hidden cost 6: tokenizer differences between providers
"Price per million tokens" assumes a token is a token. It isn't. Each provider tokenizes text with its own scheme, so the same paragraph becomes a different token count depending on who's counting. The unit you're being billed in is provider-specific, which means the rates aren't directly comparable even before any of the above.
The divergence is largest on the content that doesn't look like clean English prose: code, JSON, non-Latin scripts, heavy whitespace, long numbers, and emoji. A block of minified JSON or a CJK-language document can land 10-30% more tokens on one tokenizer than another. So a model that looks 10% cheaper per token can be the more expensive choice once your specific text is tokenized, because it splits your content into more billable units.
Worked example (illustrative). Say a fixed corpus of source code measures 1,000,000 tokens on Provider A's tokenizer and, because its scheme fragments code more aggressively, 1,200,000 tokens on Provider B's, a 20% inflation that's entirely realistic for code-heavy input. Provider B advertises $4/M input, undercutting Provider A's $4.50/M by 11%:
- Provider A: 1.0M tokens × $4.50/M = $4.50 per corpus pass.
- Provider B: 1.2M tokens × $4.00/M = $4.80 per corpus pass.
The "cheaper" sticker is 7% more expensive on this content. The only way to know which provider is actually cheaper for your text is to tokenize a representative sample with each provider's tokenizer and multiply by the real rate, never to compare the headline $/M directly. Two prices in different units aren't comparable, and tokens are different units across vendors.
Sticker vs real: the full worked comparison
Put all six together on one realistic workload and watch the headline number detach from the invoice. The setup: a customer-facing RAG agent on Gemini 3.1 Pro, the "cheapest flagship" at $2/M input. 200,000 requests/month. The naive sticker estimate, built the way most teams build it:
The sticker estimate. "Average request is ~3,500 tokens. 200,000 × 3,500 = 700M tokens. At $2/M, that's $1,400/month." This is the number that goes in the planning doc.
Now the real shape, accounting for each hidden cost:
| Component | Real-world detail | Monthly cost |
|---|---|---|
| Output skew | Answers average 1,200 output tokens, not the assumed split, 240M output at $12/M | $2,880 |
| Stable prefix, uncached | 2,000-token system prompt + tools, never cached: 400M input at $2/M | $800 |
| RAG bloat | Lazy top-10 retrieval, ~8K context tokens/request when 2K would do: 1,600M input at $2/M | $3,200 |
| Question + misc input | ~300-token user question: 60M input at $2/M | $120 |
| Retries / timeouts | ~8% retry overhead near the rate limit | $560 |
| Real total (as-built) | $7,560 |
The $1,400 headline is a $7,560 invoice, 5.4x the planning number. The single biggest driver is the output skew the sticker ignored, with RAG bloat close behind. Now apply the fixes the hidden costs point to:
- Cache the 2K prefix (90% hit): $800 drops to ~$170.
- Rerank RAG from 10 chunks to 3 (~2.4K context): $3,200 drops to ~$960.
- Fix backoff and add rate-limit headroom: retry overhead drops from $560 to ~$120.
- Output skew stays, you can't cache or trim the answer the user asked for, only choose a model with a gentler output ratio if quality allows.
Tuned total: roughly $3,900/month. Still 2.8x the sticker, because the sticker was never real, but nearly half the as-built bill, recovered with zero model swaps and zero quality loss. The lesson the HN thread was circling: the model price was the least interesting number on the page. The pain was in the shape.
Estimate the shape, then meter the production quantity
Every fix above started the same way: someone saw the real per-call breakdown, input vs output, cached vs uncached, retries vs successes, instead of multiplying a headline rate by a guessed token count. You cannot tune what you don't measure, and the provider invoice arrives once a month with everything aggregated into a single line you can't decompose.
Keep that decomposition in your model-call telemetry: input/output/cache token fields, retry flags, model name and workload identity are exactly the dimensions you need for optimization. The LLM cost-per-task calculator lets you model those components before rollout. UsageBox has a narrower billing role: send one numeric usage or computed-cost value to the customer and explicit meter/product-item stream, then retain and roll up that quantity independently from the provider invoice.
That separation matters: your observability layer explains why the workload is expensive, while an independent meter preserves how much billable quantity the customer or usage stream produced. You need both if you want optimization detail without coupling the billing record to every telemetry field.
The honest take
Price per million tokens is a real number that answers a question almost nobody actually has. The question you have is "what will this workload cost me," and the answer is a function of six things the sticker doesn't mention: which direction your tokens flow, whether you cache, how much dead context you stuff in, how often you retry, whether the work can wait, and how the provider's tokenizer chops your specific text. Get those right and the cheapest sticker and the cheapest bill finally line up. Get them wrong and you'll do what the thread warned about, pick the lowest headline number and then wonder why the invoice is triple.
The move isn't to memorize a better price card. It's to instrument the real per-call cost, find the workload-specific waste, and let the data pick the model. The sticker is the start of the conversation. It was never the answer.
Related reading
- GPT-5.5 vs Gemini 3.1 Pro vs Claude Opus 4.7 Pricing, the raw per-token rates this article tears apart, with three workload shapes worked to the dollar
- Cut Your AI API Bill 70-90% with Prompt Caching, the full mechanics behind hidden cost #2, including the $720-to-$72 receipt
- Anthropic API Billing and Claude Limits, the rate-limit and per-model side of hidden cost #4
- Gemini API Free Tier (2026), the free-tier-after-billing gotcha that turns a "$0" estimate into a real invoice