Self-Hosted LLM vs API: Electricity and Bad Outputs (2026)

The short answer: Running a model locally can beat a hosted API at high, steady utilization on a workload where the local model's outputs are good enough. But the real comparison is not “free tokens on my GPU” versus $10 per million. It is GPU or server depreciation + idle and active electricity + cooling + operations + storage + failed-output recovery, divided by the number of actually successful tasks. And no, a model described as “almost as good as a frontier model” on a public leaderboard is not proof it can do your job.

An example, with every assumption exposed: A $3,000 self-hosted machine amortized over 36 months, 10,000 attempted tasks per month, 650 W during work, 110 W idle, 20¢/kWh, PUE 1.15, 2 hours of monthly maintenance, and a local model that passes 55% of your evaluations can appear inexpensive at about 3.7¢ of infrastructure per successful task. But if each failed task needs even 30 seconds of human triage at $40/hour, the effective cost becomes about 31¢ per successful task. An API model passing 85% of the same tasks could be about 14¢ per success at a 7¢ model-call cost. Those pass rates are hypothetical, not measured model results; the failure-cost sensitivity is the real lesson.

Use the quality-adjusted self-host vs API calculator included with this article. It takes your electricity price, measured wall power, inference throughput, hardware capex, work volume and task pass rates and compares infrastructure-only with real finished-work economics. It also flags when your GPU cannot complete the assumed workload in a month.

Electricity is real - and your GPU's TDP isn't a power bill

A GeForce RTX 5090 has 32 GB of VRAM and a listed total graphics power of 575 W; an RTX 4090 is listed at 450 W. These are product specifications, not measured whole-system power during your inference workload. The CPU, RAM, drives, fans, power-supply losses and cooling add more. A model that stalls on CPU offload can consume many watts while generating very few useful tokens. NVIDIA RTX 5090 · NVIDIA RTX 4090.

Record active and idle power at the wall with a meter. For a datacenter, use an appropriate measured PUE or facility overhead factor; for a home machine where total measured wall consumption already includes power-supply losses, don't add them a second time. The equation is:

monthly_energy_kwh =,  (active_watts * active_hours + idle_watts * idle_hours),  / 1000 * facility_overhead_factor,,monthly_power_usd = monthly_energy_kwh * electricity_usd_per_kwh

Illustrative local setup (not an independently measured 5090 system):

The 20¢/kWh tariff is a user-changeable illustration, not a claim about US, Polish or worldwide average electricity. Electricity rates differ by location, contract, tax, and consumption class; Eurostat publishes non-household electricity price data rather than one universal rate suitable for every household or inference lab.

Calculating the hours your GPU is actually busy

GPU occupancy depends on your real model, context length, batch size and concurrency, not model parameter count alone. A rough sequential estimate per attempt is:

seconds_per_attempt ≈ input_tokens / measured_prefill_tokens_per_sec,                    + output_tokens / measured_decode_tokens_per_sec,busy_hours_per_month ≈ attempts_per_month * seconds_per_attempt / 3600

This approximation is for serial, non-overlapped processing. In high-concurrency serving, continuous batching and prefill/decode overlap change wall time; use observed aggregate throughput instead of blindly applying the formula. vLLM's serve benchmark records serving throughput and latency (TTFT and output-token metrics) under stated request rates and prompt lengths. Do not extrapolate a single-request tokens/second benchmark to a production server saturated with 100 clients.

For an entirely hypothetical sample of 10,000 attempts, each with 20K input and 3K output tokens, at measured-like assumed throughput of 1,500 input tokens/s and 55 output tokens/s, a serial estimate is:

,
• Prefill: 20,000 / 1,500 = 13.33 seconds.
,
• Decode: 3,000 / 55 = 54.55 seconds.
,
• Total: 67.88 seconds per attempt, or 188.55 occupied hours/month.
,

At 730 hours/month this is about 25.8% occupancy, with a little under 541.5 idle hours. The assumed machine consumes about 182.12 kWh before overhead; after ×1.15 overhead and ×$0.20/kWh, electricity is about $41.89/month. Add $83.33/month depreciation and $80 monthly operations, and infrastructure costs approximately $205.22/month - before failed work, downtime, storage and networking.

The source of these throughput numbers is explicitly the worked example, not a GPU benchmark. Substitute actual vllm bench serve telemetry to make the estimate meaningful. If predicted busy hours exceed the available hours, you need additional capacity or you must lower workload/SLO expectations; a cheap-looking saturated GPU is not an infinite token faucet.

The chart nobody includes: cost per correct output

Run the same frozen test tasks on the local model and on an API model. Illustrative scenario, not observed benchmark data:

The API's sample 7¢ request cost corresponds to 20K input tokens at $2/M and 3K output tokens at $10/M, a rate card used by multiple Standard mid-tier models at the verified date. Those price rows are vendor list rates, not a statement that any such model achieved 85% accuracy. The success rates were chosen to demonstrate the economics of poor outputs.

Without human repair, the local model looks much cheaper: $205.22 / 5,500 ≈ $0.037 versus $700 / 8,500 ≈ $0.082. Adding just 30 seconds of reviewer time per unsuccessful output changes the result dramatically. This is why a local model that posts a “nearly frontier” number on one benchmark can still lose to a more expensive API on actual customer support, coding or data transformation.

Important accounting assumption: This compares one attempt per task and divides by first-pass successes. If an operator actually retries failures until they succeed, repeated model calls, limits on retries, changing success probabilities and possible error propagation must all be included. The widget compares per-attempt economics by default; it does not pretend a fail is magically transformed into a pass.

Why “almost as good” can be dangerously expensive

Benchmarks are not interchangeable. A coding model can rank highly on patch generation and still break your authentication flow, subtly change API behavior, hallucinate missing fields, ignore tool failures, or produce a perfectly shaped JSON blob with false values. A public leaderboard score also depends on sampling settings, model routing, harness implementation and contamination. EleutherAI's evaluation harness specifically documents contamination concerns. None of that means all open models are poor - it means you cannot infer your production pass rate from one public average.

A genuine quality test for self-hosting must keep these conditions constant:

,
• Same tasks, output contracts, validation tests and timeouts across local and API candidates.
,
• Same system instructions and tool access, adjusted only when a provider protocol requires it; document adjustments.
,
• Same realistic context length and concurrency, including cases where local GPU memory forces a shorter context.
,
• Same downstream validator, including negative tests and human spot checks for plausible-looking wrong answers.
,
• Same measurement of final usefulness, not merely syntactic completion.
,

Quantization can change results. Fitting a large model into less VRAM may change accuracy or require CPU offload that destroys throughput. A 32 GB consumer GPU cannot hold arbitrary frontier-scale weights just because the model uses mixture-of-experts active parameters; weight storage, KV cache and serving buffers matter. Test the exact model checkpoint, quantization, inference runtime and context window you will deploy. The fact that a model has open weights is not a hardware-fit promise.

The complete monthly cost equation

self_host_month = hardware_purchase / months_of_service,                + electricity_kwh * $/kwh,                + maintenance_hours * $/hour,                + storage_network_backup_and_facility_cost,                + failed_attempts * review_minutes_per_failure/60 * $/hour,,hosted_api_month = sum(each_actual_call_token_cost),                 + extra_tool_and_storage_charges,                 + failed_attempts * review_minutes_per_failure/60 * $/hour,,cost_per_verified_success = all_in_monthly_cost / verified_successes

If you own the machine already, depreciation is an economic allocation rather than a new cash outlay; you may substitute resale opportunity cost or the residual useful life. If you rent GPUs hourly, replace depreciation with the provider's billed instance hours (including standby and minimum-duration billing), add instance storage and egress, and do not add the datacenter's power a second time if it is already included in the rental rate.

Other easily missed line items: GPU failure or replacement, RAM and NVMe, OS or image maintenance, security patches, model downloads, disk snapshots, logging, cold starts, queuing, rate limits, monitoring, backups, staff incident response, lost work from downtime and per-token charges for hosted embeddings or rerankers you still use. Confidentiality and sovereignty can independently justify self-hosting even when it costs more; economics is not the only constraint.

When self-hosting genuinely wins

Self-hosting is attractive when (a) you have sustained usage to amortize hardware and idle hours, (b) the model fits on your available GPU memory at the required context/concurrency, (c) measured throughput meets your latency budget, (d) private evaluation shows sufficiently high success rates and (e) the team can operate the service efficiently.

Self-hosting is risky when traffic is intermittent, models are constantly changing, most requests need a premium coding/reasoning model, error recovery is expensive, or quantization/offload lowers quality and speed. A third option often gets missed: use an externally hosted inference endpoint for an open-weight model. That buys you the cheaper model without buying the GPU; it deserves its own row in procurement tests alongside local inference and frontier APIs.

A practical experiment: take 50 real tasks (plus a protected holdout), run at production-like concurrency on all three options, log energy using wall power, record successful tasks and elapsed time, and recompute the result with your actual regional electricity tariff and burdened labor rate. Repeat for a week to capture idle periods and real failures.

What to meter once you deploy

Keep independent records for model_attempt, input_tokens, output_tokens, gpu_busy_seconds, gpu_energy_kwh (measured or tagged estimate), test_passed, review_minutes, failed_retries and successful_job. Attach the workload, model checkpoint, quantization, runtime version and tenant/job ID in your own analytics so the counter can be reconciled with infrastructure telemetry.

UsageBox can record billable numeric quantities and retain retry-safe raw usage events for rollups; it does not run GPU benchmarking, measure electricity, adjudicate quality, set a dynamic model rate card or invoice customers. If you want a simple input/output/retry comparison, try the existing cost-per-task calculator. The new self-hosted TCO widget adds the missing hardware, power and quality dimensions.

Read this next / sources

,
• NVIDIA RTX 5090 specs and RTX 4090 specs: GPU memory and power ratings; not inference-time power measurements.
,
• Eurostat electricity prices, non-household: regional/contract price context.
,
• vLLM benchmark CLI: real serving throughput and latency measurement.
,
• EleutherAI evaluation harness and decontamination notes: why public test scores are not equivalent to private task quality.
,
• OpenAI API pricing, Anthropic pricing, Google Gemini API pricing: API rates used in scenario comparisons.
,
• UsageBox's earlier utilization break-even guide: the original GPU-hours vs tokens treatment. This article extends it with measured electricity, quantified human cleanup and quality-adjusted pass rates, so the two URLs should interlink rather than compete on identical headings.
,

Verified October 17, 2026. Power draw, hardware price, inference speed, pass rates and recovery time in this scenario are explicitly illustrative. No actual cross-model benchmark was run for this article. Edit the inputs, then check the numbers against measurements before making a purchasing decision.