Serverless vs Dedicated GPU for LLM Hosting

The self-host-versus-API question gets all the attention, but most teams that decide to self-host immediately hit a second fork that is less discussed and often more expensive to get wrong: run the model on serverless GPU containers that scale to zero, or on dedicated instances that stay warm.

The marketing answer is that serverless is cheaper because you only pay for what you use. The operational answer is that serverless charges you in cold starts and destroys the batching economics that make GPU inference efficient in the first place. Which one wins is decided by a single number — your duty cycle — and it is computable before you commit to either.

The three shapes, defined precisely

Managed API. Someone else owns the GPU. You pay per token, there is no idle cost, and there is no capacity to plan. This is the baseline everything else is measured against, and the crossover into self-hosting is worked through in self-hosting an LLM vs API cost.

Serverless GPU containers. Your container image and weights, someone else’s scheduler. Instances spin up on request and scale to zero after an idle timeout. Billing is per second of active compute, sometimes with a separate charge for the time spent loading.

Dedicated GPU instances. A reserved, always-on GPU running your serving stack continuously. You pay for wall-clock hours whether requests arrive or not, and you own the autoscaling policy.

One clarification worth making early, because it causes real confusion: “serverless inference” on a major cloud is not automatically GPU-backed. Amazon’s SageMaker Serverless Inference documentation describes CPU-only endpoints configured by memory size, which rules them out for anything but small embedding or classifier workloads. Serverless GPU is a distinct product category offered by a smaller set of providers, and the term is used loosely across vendor pages. Check which one you are actually buying before modelling costs against it.

Where each one wins

Dimension	Serverless GPU	Dedicated GPU
Idle cost	None after scale-down	Full hourly rate, always
Cold start	Seconds to minutes, weights-dominated	None once warm
Effective batching	Poor at low concurrency	Good, improves with load
Cost per token at high utilisation	Higher	Lower
Capacity guarantee	Provider pool, can be denied	Reserved
Ops burden	Low	Autoscaling, upgrades, drain policy
Best fit	Spiky, low duty cycle, batch jobs, dev and staging	Steady traffic, latency SLOs, high volume

Cold start is a weights problem, not a container problem

Serverless platforms describe cold start as container startup, which is misleading for LLM workloads. The container is not the slow part. Moving model weights into GPU memory is.

The arithmetic is simple and unforgiving. Weight size in bytes is approximately parameters multiplied by bytes per parameter: an 8B model at fp16 is roughly 16 GB, at int8 roughly 8 GB, at 4-bit roughly 4 GB. A 70B model at fp16 is roughly 140 GB. Divide by the effective throughput of whatever path the weights travel:

cold_start ≈ weight_bytes / transfer_rate + engine_init

8B fp16 (16 GB) from object storage at ~1 GB/s   ≈ 16 s + init
8B fp16 (16 GB) from local NVMe cache at ~5 GB/s ≈  3 s + init
70B fp16 (140 GB) from object storage at ~1 GB/s ≈ 140 s + init

Engine initialisation adds more. A vLLM start-up profiles available memory and preallocates the KV cache block pool before it serves the first token; the vLLM documentation covers the gpu-memory-utilization setting that governs how much is claimed. CUDA graph capture, when enabled, adds further seconds.

Three consequences follow. Quantisation is a cold-start lever, not only a cost lever, because it cuts the bytes that have to move — which is one more reason the techniques in LLM GPU cost optimization compound. Providers that cache weights on node-local storage are structurally faster than those pulling from object storage every time, so ask where the weights live between invocations. And any serverless plan for a 70B-class model needs a warm-pool floor, because a two-minute cold start is not a user-facing latency, it is an outage.

Knative’s scale-to-zero documentation exposes the relevant knobs directly — enable-scale-to-zero, the scale-to-zero-grace-period that bounds how long the system waits for scale-from-zero machinery before the last replica goes, and the scale-to-zero-pod-retention-period that keeps the last pod alive for a minimum time regardless. That last one is the setting to reach for first: extending it is a cheap way to absorb bursty traffic without paying for a full warm pool. The honest way to use the set for LLM serving is a non-zero minimum replica count during business hours and true zero only overnight.

Serverless quietly destroys your batching

This is the effect that surprises teams and it is worth stating plainly: GPU inference is efficient because many requests share one forward pass. Continuous batching merges arriving requests into an in-flight batch, and PagedAttention makes the KV cache non-contiguous so memory fragmentation stops capping batch size. The PagedAttention paper puts a number on what that is worth: under contiguous pre-allocation, effective KV-cache memory in the systems it measured fell as low as 20.4%, the rest lost to reserved slots and internal and external fragmentation. The paper’s claim for vLLM is near-zero waste in KV cache memory, and the batch size that memory buys is the whole efficiency argument.

All of that value is proportional to concurrency on a single replica. A serverless platform that routes each request to a freshly scaled instance, or that holds concurrency per container at one, is running batch size one. You get the worst tokens-per-second-per-dollar the hardware can produce, on the most expensive hardware you can rent.

So the serverless configuration knob that matters most is not the memory size or the timeout, it is maximum concurrent requests per container. Set it too low and you are paying dedicated-instance prices for a fraction of dedicated-instance goodput. Set it too high and tail latency degrades. The metric to watch while tuning it is goodput rather than raw throughput, which is defined and instrumented in the vLLM vs TGI serving comparison.

The duty-cycle break-even

The decision reduces to one comparison. Let H be the hours per month you actually need a GPU serving traffic, R_d the dedicated hourly rate, R_s the serverless active-compute rate, and O the overhead multiplier for cold starts and warm-pool floors:

dedicated_cost  = 730 × R_d
serverless_cost = H × R_s × O

serverless wins when   H < (730 × R_d) / (R_s × O)

Two observations make this practical. Serverless rates are typically a meaningful multiple of dedicated rates for the same GPU class, so the break-even lands well below full-time use — often somewhere in the region of a quarter to a half of the month, depending on the spread. And O is rarely 1: a warm-pool floor that keeps one replica alive through a sixteen-hour business day burns about 486 of the month’s 730 hours, two thirds of the wall-clock a dedicated instance would have billed — and because the serverless rate is the higher of the two, that floor alone costs more than two thirds of the dedicated bill before a single token is generated.

The blunt version: if your GPU would be busy most of the working day, dedicated is cheaper and faster. If your traffic is genuinely spiky — internal tools, batch enrichment, a feature used by a few hundred people a day — serverless is cheaper by a wide margin and the cold starts are tolerable. The middle ground is the trap.

To put real numbers into that comparison for a given traffic shape, the LLM cost and latency estimator projects monthly spend, p50 and p95 latency, and cost per user across hosted and open-weight model classes, so the break-even starts from a figure rather than a vendor example.

What to measure before and after the decision

Instrument these four, and prefer them over vendor dashboards:

• Duty cycle. Fraction of wall-clock time with at least one in-flight request. This is the input to the formula above and almost nobody measures it before choosing.
• Cold-start rate and p95 cold-start duration. Measured at the client, not the platform, so queueing before the container exists is included.
• Achieved batch size distribution. On vLLM this is visible from the running-sequences and cache-usage gauges. A distribution clustered at 1 means the concurrency setting is wrong.
• Cost per thousand tokens by feature. Rolled up through a gateway rather than reconstructed from cloud billing, using the approach in token cost observability.

Two adjustments change the answer for either shape. A semantic cache removes duplicate requests entirely, which lowers duty cycle and can push a borderline workload firmly into serverless territory — hit rates and failure modes are covered in semantic caching for LLM serving. And the serving engine itself sets the ceiling on how much batching you can extract from a given GPU, which is why the framework choice in the LLM serving frameworks roundup precedes the hosting choice rather than following it.

Caveats

Every rate in this analysis is a variable, not a constant. GPU pricing moves, providers change their cold-start architecture, and committed-use or spot pricing can shift a dedicated break-even by a factor of two or more in either direction. The transfer-rate figures used above are order-of-magnitude planning numbers for object storage and local NVMe paths, not measurements of any specific provider, and the only reliable way to get the real cold-start figure for a given model and platform is to load it once and time it. What does generalise is the structure: cold start scales with weight bytes, cost efficiency scales with achieved batch size, and the choice between the two hosting shapes is decided by duty cycle rather than by preference. The broader operating habits that make either option survivable are collected in LLMOps best practices.