LLM GPU cost optimization techniques are not a single dial you turn. They’re a stack: quantization shrinks memory footprint, KV cache management eliminates waste on what’s left, batching strategy determines how hard the GPU actually works, and parallelism decides whether you scale up or out. Stack them correctly and a 70B model that required two H100s can run profitably on one A6000. Whether to run it yourself at all is the question in self-hosting an LLM vs API cost. Stack them wrong and you pay cloud rates for low GPU utilization.
What follows is a technical breakdown of each layer, what metric it moves, and where it fails.
Quantization: Smaller Weights, Same Output Distribution
Model weights stored at FP16 or BF16 use 2 bytes per parameter. A 70B model at BF16 is roughly 140 GB — two H100-80G or four A100-40G just to fit it. The first question in any GPU cost conversation is whether you actually need that precision.
AWQ (Activation-aware Weight Quantization) uses activation statistics to find the roughly 1% of salient weight channels and protects them with an equivalent per-channel scaling before quantization, so every weight still ends up in INT4 and no mixed-precision kernel is needed. The result, per the MLSys 2024 Best Paper, is a 3x speedup over HuggingFace’s FP16 implementation on a TinyChat backend, with the 70B Llama-2 deployable on hardware that could not previously fit it. The AWQ method requires no backpropagation and is calibration-dataset-agnostic, which matters when your fine-tuned model has a narrow domain and generic calibration data would overfit.
GPTQ takes a second-order optimization approach, using Hessian information to minimize quantization error weight-by-weight. It tends to take 2-4x longer than AWQ at quantization time but ships a slightly different accuracy profile on certain model families.
FP8 sits between full precision and INT4. On NVIDIA Hopper GPUs (H100, H200) with native Transformer Engine support, FP8 operates at near-BF16 quality with roughly 50% memory reduction and measurable throughput gains from the hardware’s native FP8 matrix math units. For teams that cannot accept any accuracy regression from INT4, FP8 is the production-viable compromise.
The practical starting point: reach for AWQ INT4 on anything 13B and up if you need to fit the model on a smaller GPU class or maximize batch size on the GPU you have. Use FP8 when you’re on Hopper hardware and quality sensitivity is high.
KV Cache Management: The Hidden Memory Tax
The key-value cache stores intermediate attention computations for each token in a sequence. At inference time, this cache grows linearly with sequence length and batch size — it’s not a fixed overhead. For Llama-3-70B at a 4,096-token context, a single request occupies approximately 1.3 GB of KV cache. At batch size 32, that’s 41 GB before you’ve loaded the model weights.
Traditional serving frameworks pre-allocated a contiguous memory block per request, wasting 60-80% of that allocation on average due to fragmentation and over-reservation. PagedAttention, introduced with vLLM at SOSP 2023, applies an operating-system paging model to KV cache: memory is divided into fixed blocks, requests get blocks as they’re needed, and blocks are returned when requests complete. The measured waste drops to under 4%, and reported throughput gains over FasterTransformer and Orca range from 2-4x on production-representative workloads.
In vLLM, --gpu-memory-utilization sets the model executor’s total GPU-memory budget as a fraction of device memory, not a reservation solely for KV cache. Model weights and runtime allocations consume that budget; the remaining capacity determines the inferred KV-cache allocation. The vLLM optimization documentation provides the memory-tuning context. Lowering the budget can reduce cache capacity and increase preemption pressure, so compare cache capacity, out-of-memory failures, and latency under the intended workload.
A separate explicit KV-cache byte setting, --kv-cache-memory-bytes, specifies cache capacity directly in releases that support it. Check its precedence and supported syntax against the pinned release before combining it with the executor fraction. This guide does not assume a default value across vLLM versions. Record the release, model precision, context limit, and startup memory allocation with the deployment. Framework choice also affects the operating decision; see the LLM serving framework comparison.
Two extensions worth knowing: FP8 KV storage (switching KV tensors from BF16 to FP8) halves KV cache memory with negligible quality regression for most workloads. Chunked prefill breaks large prefills into smaller chunks batched alongside decode operations, improving both TTFT and inter-token latency on mixed workloads.
If your serving workload includes agentic workflows with repeated system prompts, SGLang’s RadixAttention variant caches shared prefix KV blocks across requests, eliminating redundant recomputation. This is the relevant optimization for chains that re-inject the same context window on every step. Monitoring production model behavior for drift when switching between quantization levels is a separate concern; ML observability tooling at sentryml.com covers drift detection for quantized models.
Batching: The Metric Is GPU Utilization, Not Request Count
Static batching collects a group of requests, runs the forward pass, and waits for all requests to complete before accepting new ones. At realistic inter-arrival times this leaves much of the GPU idle, because longer sequences block shorter ones from starting.
Continuous batching, implemented in vLLM and SGLang, inserts new requests into the decode loop at the iteration level: as soon as one sequence completes a token step and has room to exit, a waiting request enters. RunPod’s inference-optimization guide cites real-world figures of 60-85% GPU utilization under steady traffic with continuous batching, against the low utilization of naive serving.
The operational implication is that your p99 latency for short requests improves dramatically because they no longer queue behind long sequences. The tradeoff is that continuous batching makes batch composition less predictable, which complicates cost-per-token accounting. Track achieved batch size and GPU utilization with observability tools that expose queue and GPU signals.
Speculative decoding layers on top of batching: a small draft model generates candidate token sequences, the large target model verifies them in a single forward pass. On generation-heavy workloads (long completions, low rejection rate), this yields 2-3x effective throughput improvement. It fails when the draft model’s distribution diverges from the target — common on code or structured output tasks with narrow valid-token windows.
Parallelism: When One GPU Isn’t Enough
Tensor parallelism (TP) splits weight matrices across GPUs, reducing per-GPU memory and allowing models that exceed a single GPU’s VRAM capacity. vLLM exposes this as --tensor-parallel-size N. Communication overhead via NVLink is low on well-provisioned hardware; PCIe-only systems suffer a measurable throughput penalty.
Pipeline parallelism (PP) assigns model layers to different GPUs sequentially. It’s suited for very large models where TP communication would bottleneck. The latency cost is bubble time between stages — stages are never perfectly synchronized.
Expert parallelism (enable_expert_parallel=True in vLLM) is specific to Mixture-of-Experts architectures. It distributes experts across GPUs, enabling efficient serving of models like Mixtral without over-provisioning for the inactive experts.
For teams running at scale: CPU allocation is a non-obvious GPU efficiency lever. vLLM’s scheduling and input processing requires at minimum 2 + N physical cores, where N is the GPU count. Hyperthreading systems need double. Underprovisioning physical cores starves the scheduler and degrades GPU utilization even with the model fully loaded. The vLLM optimization documentation covers the full parallelism matrix.
Caveats
Quantization and reasoning models: INT4 quantization can inflate reasoning token counts on chain-of-thought-heavy models, which can offset the throughput gains from smaller weights. Benchmark your specific model on your actual task distribution before committing to INT4 in production.
KV cache FP8 and long contexts: FP8 KV precision loss accumulates over long sequences. Quality regression is typically negligible at 4k tokens; behavior at 32k-128k contexts requires empirical validation.
Continuous batching and SLA isolation: Dynamic batch composition means a single long request can inflate latency for short ones during preemption events. If you’re serving a mix of interactive and batch workloads, separate them into distinct vLLM instances rather than sharing a continuous-batch pool. Repeat traffic is worth intercepting before it reaches the GPU at all, which is what semantic caching does. Quantisation also shortens the weight-load path, which is the dominant term in serverless cold starts — see serverless vs dedicated GPU for LLM hosting for how that changes the hosting decision.
Security teams evaluating LLM serving stacks should also assess prompt injection surface across serving configurations; aisec.blog tracks active exploitation patterns against deployed inference endpoints.
