Cost to Host Private LLM: Monthly GPU Math by Model Size

The cost to host private LLM endpoints on rented GPUs runs about $796 a month for an 8B model on one 48 GB card, $2,402 for a 4-bit 70B on one H100, and $11,943 for a BF16 70B on four H100s, at Lambda’s on-demand list prices. Add a second replica for availability and the bill doubles.

Those figures are the GPU line only: 730 billed hours at the per-GPU rates on Lambda’s pricing page as of October 2026. That page is a vendor price list, not a quote and not a benchmark. Storage, egress, tax and engineer time sit on top. The break-even against a hosted API is a separate calculation, worked through in self-hosting LLM vs API cost.

What does each model size cost per month?

A private 8B model costs roughly $796 a month on one 48 GB GPU; a 70B model costs $2,402 at 4-bit on one H100 PCIe, $6,117 at FP8 on two H100 SXM, or $11,943 at BF16 on four; a 405B model at FP8 needs a full 8× H100 node at $23,302. Every row assumes always-on, single-replica, on-demand capacity.

Lambda’s per-GPU H100 SXM rate falls as the count rises, from $4.29 for one GPU to $3.99 for eight, so the fleet column is not a clean multiple. The 405B row is FP8 because BF16 does not fit: the Llama 3 paper states that “Llama 3 405B does not fit in the GPU memory of a single machine with 8 Nvidia H100 GPUs” and serves BF16 across two machines (Grattafiori et al., 2024).

Low-traffic endpoints have a second option. Modal’s pricing page bills GPUs per second: $0.000222 for an L4 and $0.001097 for an H100, about $0.80 and $3.95 per active hour, plus separate CPU and memory charges. An internal tool busy 176 hours a month (8 hours, 22 weekdays) spends about $141 on L4 time. Cold starts and lost batching are the price; serverless vs dedicated GPU hosting works out the duty-cycle break-even.

Why VRAM, not parameter count, sets the bill

Weights are the easy part: parameters times bytes per parameter. Two bytes at BF16, one at FP8, roughly half a byte at 4-bit, plus the per-group scales that weight-only methods such as AWQ store alongside. What decides how many users a GPU serves is the KV cache left over after the weights load.

Per-token KV cache for a grouped-query-attention model is:

kv_bytes_per_token = 2 × layers × kv_heads × head_dim × bytes_per_element

The Llama 3 paper’s Table 3 supplies the inputs: 32 layers and 8 KV heads for 8B, 80 and 8 for 70B, 126 and 8 for 405B, all with a 128-wide head (model dimension divided by attention heads). At BF16 that is 128 KiB per token for 8B, 320 KiB for 70B and 504 KiB for 405B. A single 8K-token conversation on the 70B holds 2.5 GiB of cache.

This is where cheap-per-hour hardware goes wrong. On Lambda, a 24 GB A10 lists at $1.29 an hour and a 48 GB A6000 at $1.09. Load an 8B BF16 model on the A10 at a 0.90 memory fraction, subtract a 4 GB runtime allowance (an assumption; measure yours), and roughly 12K tokens of cache remain: one long request at a time. The A6000 leaves room for about 177K tokens and costs less per hour.

The same arithmetic applies to --kv-cache-dtype fp8 in vLLM’s engine arguments: one byte per element instead of two doubles the tokens in flight without renting another GPU. The quality side of that trade is covered in GPU cost optimization techniques.

The metric that matters: dollars per million tokens at SLO

Dollars per GPU-hour is the number on the invoice, and it is the wrong one to optimize. The number that compares deployments is cost per million output tokens served within your latency SLO:

usd_per_M_output_tokens = fleet_usd_per_hour × 1,000,000 / (3,600 × goodput_tokens_per_sec)

Goodput counts only tokens from requests that met the TTFT and inter-token latency targets. It beats GPU-hour price because a GPU that costs half as much per hour but sustains a third of the goodput costs 50% more per token. It beats peak tokens/sec from a vendor benchmark because peak throughput is usually measured at a concurrency your p99 TTFT will not tolerate. And it exposes idle time: at 3am tokens/sec falls, the metric climbs, and you see what always-on capacity really costs.

Wiring it up

Two pieces: a sizing script to pick hardware before you rent it, and a Prometheus recording rule that prices the tokens once it is live.

# size_private_llm.py: VRAM fit and monthly GPU bill for one replica.
# Architecture: Llama 3 paper, Table 3. Prices: lambda.ai/pricing, Oct 2026.
HOURS_PER_MONTH = 730

MODELS = {  # params (billions), layers, KV heads, head dim
    "llama-3.1-8b": (8, 32, 8, 128),
    "llama-3.1-70b": (70, 80, 8, 128),
    "llama-3.1-405b": (405, 126, 8, 128),
}


def kv_bytes_per_token(layers, kv_heads, head_dim, kv_bytes=2):
    return 2 * layers * kv_heads * head_dim * kv_bytes  # K and V


def plan(model, weight_bytes, gpus, vram_gb, usd_per_gpu_hour,
         mem_fraction=0.90, overhead_gb_per_gpu=4.0, kv_bytes=2):
    params_b, layers, kv_heads, head_dim = MODELS[model]
    weights_gb = params_b * weight_bytes  # ignores quantization scales
    kv_gb = gpus * (vram_gb * mem_fraction - overhead_gb_per_gpu) - weights_gb
    per_token = kv_bytes_per_token(layers, kv_heads, head_dim, kv_bytes)
    tokens = int(max(kv_gb, 0) * 1e9 // per_token)  # 0 means it does not fit
    return model, tokens, round(gpus * usd_per_gpu_hour * HOURS_PER_MONTH, 2)


for args in [
    ("llama-3.1-8b", 2.0, 1, 24, 1.29),   # A10 24 GB
    ("llama-3.1-8b", 2.0, 1, 48, 1.09),   # A6000 48 GB
    ("llama-3.1-70b", 0.5, 1, 80, 3.29),  # H100 PCIe, 4-bit weights
    ("llama-3.1-70b", 2.0, 4, 80, 4.09),  # 4x H100 SXM, BF16
]:
    print(plan(*args))

The four rows come out at roughly 12K, 177K, 100K and 403K tokens in flight. The overhead figure is a placeholder; once the server is up, replace the estimate with the KV cache capacity vLLM reports at startup.

# rules/private-llm-cost.yaml: price the tokens a vLLM deployment actually serves
groups:
  - name: private-llm-cost
    interval: 1m
    rules:
      - record: deployment:fleet_usd_per_hour
        expr: vector(16.36)  # 4x H100 SXM at $4.09/GPU-hour; update with the invoice
      - record: deployment:output_tokens_per_second:rate5m
        # model_name matches --served-model-name
        expr: sum(rate(vllm:generation_tokens_total{model_name="llama-3.1-70b"}[5m]))
      - record: deployment:usd_per_million_output_tokens
        expr: |
          scalar(deployment:fleet_usd_per_hour) * 1e6
            / (3600 * clamp_min(deployment:output_tokens_per_second:rate5m, 1))
      - record: deployment:ttft_seconds:p99_5m
        expr: |
          histogram_quantile(0.99,
            sum by (le) (rate(vllm:time_to_first_token_seconds_bucket[5m])))

vLLM exposes vllm:generation_tokens_total as a counter and vllm:kv_cache_usage_perc as a 0-to-1 gauge, per its metrics design notes. The cost rule measures throughput, not goodput, so put the TTFT p99 series on the same panel and you can see when the cheap tokens are late tokens. Do per-tenant attribution at the gateway rather than by adding tenant labels to engine metrics; token cost observability covers that ledger.

What you’ll see

Healthy. At peak, vllm:kv_cache_usage_perc moves with load but does not pin at 1.0, vllm:num_requests_waiting sits at zero most of the time, and the dollars-per-million line is flat through business hours, then rises overnight as tokens/sec falls. The clamp_min floor turns a fully idle fleet into a spike instead of a division error; that spike is your idle bill.

Undersized. Cache usage pinned near 1.0, a nonzero waiting queue, TTFT p99 climbing. The cost line looks its best right here because the GPU is saturated. That is the trap: the cheapest tokens arrive just before the SLO breaks. Add a replica or move to a larger-memory GPU.

Oversized. Cache usage stays low even at peak and cost per million sits high all day. Drop to 4-bit or FP8 on fewer GPUs, consolidate models onto shared hardware, or move the endpoint to per-second billing.

Caveats

• List prices are not quotes. Lambda describes its on-demand instances as self-serve, first-come access and says prices exclude sales tax, VAT and GST. Capacity at that price is not guaranteed, and hyperscaler rates for the same GPU differ. Rerun the arithmetic with your contract.
• Serverless has extra lines. Modal bills CPU at $0.0000131 per physical core-second and memory at $0.00000222 per GiB-second on top of GPU time.
• Quantization changes the model. The 4-bit and FP8 rows are cheaper because they run different numerics. Meta reports that FP8 inference raised 405B prefill throughput by up to 50%; that is Meta’s own measurement, not an independent benchmark, and it says nothing about your task quality. Gate the switch with a regression eval in CI.
• Owned hardware moves cost to power and depreciation. NVIDIA rates the H100 SXM at up to 700 W, so eight GPUs at max TDP draw 5.6 kW, about 4,088 kWh over a 730-hour month before CPUs, networking and cooling. Multiply by your utility rate and add amortized purchase cost.
• Staff time is not on the GPU invoice. Engine upgrades, on-call and capacity planning are real hours; price them at your own rate.

FAQ

how much does it cost to run llama 70b privately

Running Llama 3.1 70B privately costs about $2,402 a month at 4-bit on one H100 PCIe, $6,117 at FP8 on two H100 SXM, or $11,943 at BF16 on four H100 SXM, at Lambda’s on-demand list prices for 730 hours. Double it for two replicas, then add storage, tax and engineering time.

how much gpu memory do i need to host an llm

You need room for weights, KV cache and runtime overhead. Weights are parameters times bytes per parameter: 16 GB for an 8B model at BF16, about 35 GB for a 70B model at 4-bit. KV cache is 2 × layers × KV heads × head dimension × bytes, per token in flight, summed across concurrent requests.

is it cheaper to self host an llm than use an api

It is cheaper only when the GPUs stay busy. A rented GPU bills every hour whether it serves tokens or not, so cost per token rises as utilization falls. Compare cost per completed task at equal quality, including idle hours and a second replica; the self-hosting vs API breakdown gives the break-even formula.

does serverless gpu hosting make a private llm cheaper

It does when the endpoint sits idle most of the month. Per-second billing such as Modal’s charges about $0.80 per active L4 hour, so 176 busy hours costs about $141 in GPU time. Run it all month, though, and Modal’s H100 rate adds up to about $2,883, against $2,402 for a dedicated Lambda H100 PCIe.

Related across the network

• How Much VRAM for a 7B Model? Quantization and Context — lmstudioguide.com
• ML Model Deployment: Shipping Models That Stay Healthy — sentryml.com