Self Hosting LLM vs API Cost: A TCO Breakdown for 2026

The self hosting llm vs api cost question gets asked constantly and answered poorly. Most takes reduce it to a single spreadsheet: GPU rental rate divided by tokens per second, versus OpenAI’s per-token price. That math is real, but it prices only the compute. The actual TCO includes idle GPU time, engineering salaries, model update cycles, and monitoring infrastructure — and those terms dominate the outcome for most teams.

This is the number-by-number breakdown.

What API Access Actually Costs

The API tier is simple. You pay per million tokens, and the range is enormous.

GPT-4o sits at roughly $2.50 per million input tokens and $10 per million output tokens. GPT-4o Mini drops to $0.15 input / $0.60 output per million. DeepInfra’s hosted Llama 3.3 70B comes in around $0.12 per day for 1M tokens — a fraction of frontier model pricing.

That 80x spread within the API tier matters. Before comparing self-hosting against “API cost,” you need to be comparing against the right tier. Teams that benchmark self-hosting against GPT-4o pricing but actually need GPT-4o Mini quality are solving the wrong problem.

API costs are also perfectly elastic: you pay for what you call, and nothing for idle time. That matters a lot once you look at utilization.

What Self-Hosting Actually Costs

Self-hosting breaks into three layers: hardware or cloud GPU rental, inference stack operation, and engineering time.

Hardware. On-prem GPU purchases carry high upfront costs with uncertain amortization. An RTX 5090 has an MSRP of $1,999 but street prices in early 2026 ran $3,500–4,000 due to allocation constraints. A used RTX 4090 (24 GB VRAM, viable for 7B–13B parameter models) runs $1,600–2,000 on the secondary market. H100 SXM GPUs traded between $15,000–20,000 on the secondary market in 2025–2026, down from $25,000+ earlier.

Amortized over 36 months, a two-H100 setup contributes $800–1,100/month in hardware depreciation before electricity, rack, or networking.

Cloud GPU rental. Managed cloud GPU is the lower-friction path. H100 instances on Hyperbolic run around $1.49/hr; the same on Azure runs $6.98/hr. That 4.7x spread between providers for the same hardware class is not a rounding error — it directly determines where the breakeven falls.

Inference throughput and cost per token. Fin AI’s serving cost research benchmarked Qwen 3 14B at 70,000 tokens/second on a single A100 at $1.91/hr, and 135,000 tokens/second on an H100 at $3.90/hr. That puts sustained A100 inference at roughly $0.027 per million tokens at 100% load — cheaper than most budget API tiers.

Small open-weight models are genuinely cheap to serve at scale. The same research found Gemma 3 4B costs approximately 0.04x GPT-4.1 per token when self-hosted. But a 70B parameter model on the same methodology costs 1.70x GPT-4.1 — because model quality requires hardware that closes the gap.

GPU memory: weights are not the whole bill. The most common sizing mistake is budgeting VRAM for model weights alone. The KV cache holds the attention keys and values for every token in every in-flight request, so it grows with sequence length and concurrency rather than sitting at a fixed overhead. On long-context or high-concurrency workloads it can demand as much memory as the weights or more. Published GPU-memory analysis puts it bluntly: capacity planning for self-hosted LLMs is usually wrong because teams size for weights and forget the cache. A cluster sized that way does not fail on the spreadsheet, it fails under real concurrent load, and the emergency capacity you rent at retail rates afterward is the most expensive GPU time you will ever buy.

Serving-engine efficiency sets your real cost per token. The GPU’s hourly rate is the input; what your engine extracts from that GPU is the output. vLLM built its position on PagedAttention, which manages the KV cache in non-contiguous pages the way an operating system manages virtual memory, plus continuous batching, which admits new requests as slots free up instead of waiting for a whole batch to drain. Both raise throughput on fixed hardware, which moves cost per token directly. The vLLM-versus-TGI comparison from Alongside lands where most 2026 production guidance does: vLLM for throughput-sensitive serving, Hugging Face TGI mainly inside HF-centric workflows, and Ollama for local development rather than production. KV-cache quantization such as FP8 shrinks the cache footprint again on long-context work. The vLLM vs TGI serving comparison covers where those engines now diverge in more depth.

The Utilization Trap

This is the most common failure mode in self-hosting cost models: teams calculate cost at 100% GPU utilization, then run 10–20% in production.

The math is brutal. At 100% load, a well-tuned A100 inference stack costs around $0.013 per 1,000 tokens. At 10% load, you’re paying $0.13 per 1,000 tokens — a 10x multiplier — because the idle capacity still costs the same hourly rate.

APIs charge nothing for idle. Self-hosted infrastructure charges the full rate, 24/7, whether you’re serving traffic or waiting for it.

For teams with highly variable load profiles — spiky usage, overnight silence, batch jobs that run weekly — the effective utilization rate is often far below the peak-hours number used in the business case. Batch scheduling with Ray Serve or aggressive autoscaling can help, but it adds operational complexity that itself costs engineering time.

The Hidden Costs Nobody Budgets

Hardware and GPU rental are the visible line items. These are not.

Engineering overhead. Running a production LLM inference cluster is not a weekend deploy. DevOps and MLOps engineers cost roughly $145,000/year fully loaded in the US. Operational analysis puts the overhead multiplier at 3–5x raw GPU cost once you account for engineering time, monitoring, and incident response. A team scaling from 2M to 15M daily tokens can spend $38,000 in engineering time over six weeks on infrastructure work alone.

Model update cycles. Open-weight model releases move fast. A production Llama deployment that was current in January may need a full re-evaluation, re-quantization, and serving config update by March. Model update cycles cost approximately $12,000 per refresh in engineering time at a typical company cadence.

Observability stack. Self-hosted inference needs its own monitoring. You cannot see latency p95, throughput drops, or KV cache saturation without purpose-built instrumentation. Setting up Prometheus scraping, latency histograms, and token-throughput dashboards adds to the build cost. For teams moving to self-hosting, the MLOps observability stack at sentryml.com covers what production LLM monitoring needs to expose.

What the Cost Model Cannot Price

Three tradeoffs decide this question for a lot of teams and resist a dollar figure entirely.

Who owns reliability. With an API, the provider owns uptime, scaling, and on-call for the model serving layer, and you own your application. Self-hosting moves serving reliability onto your team: GPU node failures, out-of-memory events under load, autoscaling, capacity planning, and the 2 a.m. page when a traffic spike exhausts the KV cache. That on-call burden is a recurring cost no per-token comparison shows, and it lands on the same people you already counted as engineering overhead.

Model drift, inverted. A self-hosted model is a frozen artifact. It changes only when you change it. An API model can shift underneath you when the provider updates a non-pinned alias, which is the silent quality decay problem. Self-hosting removes that class of drift entirely, and for some regulated or high-stakes systems that alone decides the question regardless of cost. The flip side is that you now own every upgrade, including the security patches and capability improvements the API would have handed you for free.

Capability ceiling. The largest frontier models are generally API-only. Self-hosting means open-weight models, which clear the quality bar for many tasks and not for others. The honest question is task-specific: does an open-weight model you can actually run on hardware you can actually afford meet your quality bar? If only a frontier API model clears it, the path is chosen for you and the cost math never applies.

Where the Breakeven Actually Falls

Comprehensive cost analysis puts the financial breakeven at approximately $4,200/month in equivalent API spend — around 11 billion tokens per month — before self-hosting becomes cheaper after accounting for full engineering overhead. Below that, the API is typically cheaper even when the per-token GPU math looks favorable, because the GPU math ignores the engineer. The LLM cost and latency estimator projects monthly spend at your token volume across GPT, Claude, Gemini, and open-weight models, so the crossover question starts from a real number rather than a vendor’s example.

That number shifts down if you already have the infrastructure team, or up if your utilization is low. The direction is more reliable than the specific threshold: self-hosting is a fixed-cost structure that beats a variable-cost structure only once the variable costs are large enough to absorb the fixed base.

One exception that doesn’t depend on volume: data sovereignty and regulatory requirements. HIPAA, SOC 2 Type II, and government data handling requirements can make self-hosting necessary regardless of cost. Teams processing PHI or classified data in regulated environments face security constraints that API terms of service and shared-infrastructure models cannot satisfy. In those cases, self-hosting is a compliance decision, not a cost decision.

When the Math Favors Each Path

Self-hosting wins when:

• Monthly API spend exceeds ~$4,000–5,000 and utilization is high and predictable
• You need small open-weight models (7B–14B class) at high volume with stable load
• Data cannot leave your infrastructure for regulatory or contractual reasons
• Latency requirements demand p99 under 100ms with guaranteed allocation (no shared API queues)

API wins when:

• Volume is low or variable (utilization below 50%)
• You’re using frontier models (GPT-4o, Claude Sonnet-class) that require expensive hardware to match
• The team has no existing MLOps headcount to absorb operational load
• You need elastic scale without capacity planning

The Hybrid Most Teams Actually Run

Treating this as a binary is itself part of the mistake. Teams large enough to seriously consider self-hosting usually end up routing both paths behind one gateway, which is the same control-plane pattern the LLMOps best-practices guide recommends for cost governance generally:

• Predictable high-volume traffic goes to self-hosted open-weight models on vLLM, where steady load amortizes the fixed cost and you control the artifact.
• Spiky overflow traffic goes to an API, so you never provision GPUs for a peak you hit twice a week.
• Frontier-capability requests go to whichever API model clears the quality bar.
• Privacy-sensitive requests go to the self-hosted path by policy, regardless of what it costs.

A gateway is what makes this tractable. Routing lives in one control plane, application code stays agnostic about where a request lands, and you can shift traffic between self-hosted and API capacity without a redeploy. It also converts the decision from all-or-nothing into a line you can move: self-host the slice where the numbers clearly work, buy the rest, and revisit the split as volume and open-weight capability change.

The self hosting llm vs api cost comparison is not a single number — it’s a function of your utilization curve, model tier, team structure, and compliance requirements. The compute math is the easy part. The hard part is honestly pricing the engineering time, which most analyses discount to zero.

Once the decision to self-host is made, a second fork follows immediately: scale-to-zero serverless GPU containers or an always-warm dedicated instance. That one turns on duty cycle and cold-start behaviour rather than token volume, and it is worked through in serverless vs dedicated GPU for LLM hosting.

Related across the network

• Training Infrastructure Cost Control: Where ML Spend Goes — mlopsplatforms.com
• Implementing Rate Limiting and Abuse Detection for AI APIs — aidefense.dev
• Closing the Eval-Prod Gap: Online Evaluation as Observability — mlobserve.com
• End-to-End Tracing for LLM Applications: What Belongs in a Span — mlobserve.com
• LLM Cost & Latency Observability with OpenTelemetry — mlobserve.com