Self Hosting LLM vs API Cost: Private Hosting TCO

LLM API hosting services come in five price shapes, and utilization decides which is cheapest. At October 2026 list prices, gpt-oss-120B costs $0.15 per million input tokens and $0.60 per million output on Together AI and Fireworks, a dedicated H100 endpoint costs $5.49 to $8.00 per GPU-hour, and a self-managed Lambda H100 PCIe lists at $3.29.

Self hosting LLM vs API cost is a comparison between capacity you pay to operate and requests you pay a provider to process. The cost to host a private LLM includes more than model weights and a GPU quote: idle capacity, recovery headroom, storage, retrieval, and operating time all belong in the calculation.

No monthly-spend threshold makes self-hosting cheaper everywhere. Compare total cost at the same quality and latency target; the worked examples further down use labeled hypothetical inputs.

The engine decision is covered in vLLM vs TGI. For a private RAG application, include the retrieval choice from best vector database for RAG; buying generation through an API does not remove your retrieval bill.

LLM API hosting services compared: October 2026 prices

Each option bills in a different unit, so compare the unit before the number. These are list prices, excluding negotiated discounts and tax.

Billing details move the result as much as headline rates. Anthropic’s Batch API takes 50% off input and output tokens, and Fireworks bills batch inference at 50% of serverless prices. Fireworks prices unlisted dense open models by size: $0.20 per million tokens for 4B to 16B parameters, $0.90 above 16B. Together’s dedicated inference bills per minute while a deployment runs, regardless of request volume. Fireworks deployments are billed by GPU-second while replicas are active.

Self-hosting and API cost: what belongs in each column

Amazon Bedrock’s pricing page shows why the API contract matters: per-token On-Demand rates and hourly Provisioned Throughput are different cost structures. Use the rates offered to your deployment, and never apply one model’s input/output rates to another model.

On rented compute, a running instance bills per second with a one-minute minimum even when idle, and a stopped one still pays for its EBS volumes, per Amazon EC2’s lifecycle documentation. “No requests” and “no charges” are different states.

What one dedicated H100 must replace in API spend

Over a 730-hour month, one always-on H100 costs $2,401.70 self-managed on Lambda PCIe, $4,007.70 as a Together dedicated endpoint, and $5,840.00 as an always-on Fireworks deployment. That fixed bill is what the GPU has to displace in per-token spend.

Take a hypothetical request of 1,000 input and 250 output tokens. At the list prices above it costs $0.0003 on gpt-oss-120B, $0.0013 on Llama 3.3 70B through Together, and $0.0045 on Claude Sonnet 5.5. Dividing the monthly GPU bill by those figures gives the volume where spend matches:

Read the table as a floor. The GPU must sustain that average at your latency target, absorb peaks, and run a model that passes the same evaluation; the Sonnet row applies only if an open-weight model replaces it on your task. A second replica doubles the GPU line. Whether a model fits one 80 GB card is worked through in cost to host a private LLM.

Private LLM hosting: what each level of “private” costs

Private hosting shows up on different invoice lines depending on how private it is.

1. Public API with contractual controls. Residency can be a multiplier: Anthropic charges 1.1x on all token categories for US-only inference (inference_geo) on Claude 4.6 and later.
2. Managed API over a private network path. Bedrock interface VPC endpoints let a VPC reach Bedrock “without the use of an internet gateway, NAT device, VPN connection, or Direct Connect connection.” With private DNS enabled on com.amazonaws.{region}.bedrock-runtime, SDK calls route through the endpoint unchanged. At $0.01 per hour per network interface in AWS’s us-east-1 example, two Availability Zones add about $14.60 a month plus data processing. Claude on regional rather than global Bedrock endpoints carries a 10% premium.
3. Dedicated endpoint at a hosting provider. GPU time bills whether or not traffic arrives.
4. Self-managed in your VPC or data center. You control weights, network path, and logs, and carry every budget line below.

Run a private LLM API on your own GPUs

A self-hosted private LLM API is typically an OpenAI-compatible server behind your own gateway. vLLM’s defaults make it quick to start, but they are not a security boundary.

1. Choose the model with an evaluation set before choosing hardware.
2. Size GPUs from weights plus KV cache at your context length and concurrency.
3. Start the server. The vllm serve reference lists 8000 as the default port, and the online serving docs include /v1/chat/completions, /v1/completions, /v1/responses, and /v1/embeddings among the OpenAI-compatible routes.
4. Do not treat --api-key as access control. It only authenticates routes under /v1, /v2, and /inference; /invocations has no key check. Keep the port private and put authentication, rate limits, and serving-path guardrails in a gateway.
5. Keep a per-token API as overflow or fallback, gated by the same acceptance tests.

# OpenAI-compatible server on the default port 8000
vllm serve <model> --api-key "$VLLM_API_KEY"

Host LLM API latency: what each path lets you buy

On per-token APIs, latency is a priced tier. Anthropic’s fast mode for Claude Opus 5.5 costs $8 input and $40 output per million tokens, double the standard $4 / $20, and cannot be combined with batch. Fireworks Priority lists gpt-oss-120B at $0.18 / $0.72 against $0.15 / $0.60 Standard.

Amazon Bedrock’s latency-optimized inference is in preview, set per request with "performanceConfig": {"latency": "optimized"}, for Llama 3.1 70B and 405B and Claude 3.5 Haiku in US East (Ohio) and US West (Oregon). Past the optimized quota, requests fall back to standard and bill at standard rates.

On dedicated or self-hosted capacity, latency is a capacity decision. Scale-to-zero saves the idle bill but adds startup time; Fireworks returns a 503 while a deployment scales up. Interactive chat needs warm capacity or a fast tier, while offline jobs belong on batch pricing.

Build a request ledger before comparing providers

Record input and output tokens separately for every attempt. Preserve the requested model, resolved model when available, request outcome, cache billing category, feature, and application release.

For a simple token-priced API with rates quoted per million tokens:

api_generation_cost =
  (input_tokens × input_rate + output_tokens × output_rate) / 1,000,000

Extend it for each category on the provider’s rate card, such as cache reads, cache writes, batch, and tool use, counting each once so cached input is not billed twice in your model.

A user request may involve retrieval, reranking, retries, and multiple generations. Attach each attempt to the user-facing operation, then calculate cost per completed task. Token cost observability develops that accounting model.

Reconcile calculated usage against invoices; gaps expose missing attempts, credits, minimum commitments, or rate-card changes.

Cost to host a private LLM: five budget lines

Compute and redundancy

Obtain a quote for the whole instance or cluster, including the number of accelerators and the interconnect you need. Record whether the price is on demand, committed, interruptible, or otherwise conditional.

Budget replicas from the availability requirement: a plan that works only when every replica is healthy cannot absorb a failure. Show upgrade and recovery capacity explicitly rather than hiding it in a utilization assumption.

Owned hardware

For owned equipment, separate the purchase decision from the operating decision. A planning model can amortize purchase cost over an assumed useful life, then add electricity, hosting, networking, maintenance, and replacement reserves.

Useful life and resale value are assumptions. For hardware already owned, show incremental operating cost and a capital allocation, or self-hosting looks free because the purchase sits in another budget.

Model memory and runtime headroom

A basic dense-weight estimate is parameter count multiplied by bytes per stored parameter. It is only a starting point: quantization metadata, runtime allocations, and the attention KV cache also consume capacity.

The PagedAttention paper describes why KV-cache allocation matters for LLM serving. Request length and concurrent sequences affect the memory needed for in-flight work, so plan capacity from the intended workload rather than only checking whether weights load.

The vLLM optimization documentation covers controls such as chunked prefill, preemption, and parallelism, but a configuration value does not establish usable throughput. Weigh serving framework tradeoffs when picking the engine for that plan.

Storage, retrieval, and observability

Include artifact storage, logs, metrics, backups, and any vector database. For RAG, include embedding refreshes, ingestion, reranking, and index rebuilds. These items can change independently of generation volume.

Record retention and replication assumptions next to the compute quote; a short development run misses a month of retained traces and data refreshes.

Operating time

Estimate hours for routine upgrades, evaluation, release management, capacity changes, and incident response. Multiply by your own planning rate. Do not substitute an unsourced salary figure or a universal percentage of GPU cost.

Separate one-time migration effort from recurring work. Show both first-period cost and steady-state cost so a cheap steady-state estimate cannot conceal an expensive transition.

Utilization: paid hours versus useful work

Two different measures are often called utilization. Device activity describes how busy an accelerator is at a moment in time. Economic utilization describes how much of your paid capacity produces useful work across the billing period. Device activity and queue time come from different sources; LLM observability tools for serving and GPU cost maps which tool exposes each.

For a fixed workload, let:

g = total hourly cost of the allocated serving fleet
q = output tokens per second at the accepted latency and quality target
u = average fraction of that useful capacity consumed while the fleet bills

compute_cost_per_million_output_tokens =
  g × 1,000,000 / (3,600 × q × u)

Here q includes the input-processing load of the same workload, even though the denominator counts output tokens. It is not comparable to another workload with much shorter prompts. Keep the request mix fixed when comparing this metric.

For autoscaled fleets the rate is not constant: sum the cost of actual allocated intervals and divide by actual completed work.

Worked example: hypothetical hosting arithmetic

Assume a fleet costs $2 per billed hour and completes 100 output tokens per second at the required quality and latency on a fixed request mix. These are invented planning inputs, not a provider quote or a benchmark.

At full use, it produces 360,000 output tokens per hour. Its compute-only cost is therefore $2 / 0.36, or approximately $5.56 per million output tokens.

Over a hypothetical 720-hour period the fleet costs $1,440 however lightly it is used, before storage, operations, redundancy, and migration.

Calculate break-even without a universal token threshold

A simple planning model can separate a recurring fixed cost F from variable cost per completed task:

self_hosted_cost = F + tasks × variable_self_hosted_cost
api_cost = tasks × api_cost_per_task

break_even_tasks =
  F / (api_cost_per_task - variable_self_hosted_cost)

A crossover exists only when API cost per task exceeds the self-hosted variable cost, and only while the same fleet has capacity; another replica or accelerator is a step change in F.

For a hypothetical F of $1,800 per month, API cost of $0.020 per completed task, and variable self-hosted cost of $0.005, the crossover is 120,000 tasks per month. Those are illustrative inputs. They establish the arithmetic, not a market break-even claim.

Count each charge once: always-allocated compute belongs in F, genuinely usage-billed components in the variable term.

Quality and reliability can change the comparison

Establish the minimum acceptable task quality before evaluating cost. A less expensive model that fails the required task is not an equivalent replacement. Record whether results require retries, longer context, external tools, or human handling.

Pinned self-hosted weights do not stop production drift: inputs, retrieval data, engine configuration, and task requirements still change output quality.

Define reliability responsibilities explicitly. A provider operates its serving layer, while your application still owns timeouts, quotas, retries, and user-facing failure behavior. Self-hosting adds responsibility for the inference fleet. Compare both paths against the same recovery objective.

A private network or self-hosted model is a design choice, not automatic proof that contractual or regulatory requirements are met; assess that separately from cost.

When each cost structure deserves a closer look

An on-demand API is a useful candidate when request volume is uncertain, there is little tolerance for idle capacity, or the team wants to avoid operating the serving fleet. Its actual model capability, quotas, data terms, and rate card still need review.

Self-hosting deserves a closer look when a suitable model is available, demand is sustained enough to use allocated capacity, and the team can operate the required deployment.

A hybrid can reserve owned or rented capacity for predictable work and route overflow elsewhere. Include gateway behavior, fallback model quality, and any duplicated monitoring effort.

FAQ

how much does it cost to host a private llm

A private LLM on one always-on H100 costs about $2,402 a month at Lambda’s $3.29 H100 PCIe list rate, or $4,008 as a Together dedicated endpoint at $5.49 per GPU-hour, at October 2026 list prices. Add a second replica, storage, retrieval, and operating time; larger models need several GPUs. Get current deployment-specific quotes before committing.

can high token volume guarantee self-hosting savings

No. Volume only helps if the GPU stays busy at your latency target with a model that passes the same evaluation. Prompt length, output length, retries, failed tasks, and paid idle hours all move the result. Compare equivalent completed work, and recalculate whenever growth forces another replica or a larger GPU.

does serverless llm hosting remove idle cost

It depends on the service and configuration. Fireworks on-demand deployments bill per GPU-second while replicas are active and scale to zero after an hour unused by default, but requests get a 503 while capacity scales back up. Warm replicas and minimum billing rules change the equation; serverless vs dedicated GPU hosting covers that decision.

where should the cost calculation start

Start with the LLM cost and latency estimator to explore traffic assumptions, then replace its illustrative rates and latency figures with list or contract prices for your chosen deployment. Record the request mix, quality bar, and recovery target first, and keep the cost ledger alongside the LLMOps tools map used to assemble the stack.