LLM GPU Cost Optimization Techniques That Move the Needle

LLM GPU cost optimization techniques are not a single dial you turn. They’re a stack: quantization shrinks memory footprint, KV cache management eliminates waste on what’s left, batching strategy determines how hard the GPU actually works, and parallelism decides whether you scale up or out. Stack them correctly and a 70B model that required two H100s can run profitably on one A6000. Whether to run it yourself at all is the question in self-hosting an LLM vs API cost. Stack them wrong and you pay cloud rates for 40% GPU utilization.

What follows is a technical breakdown of each layer, what metric it moves, and where it fails.

Quantization: Smaller Weights, Same Output Distribution

Model weights stored at FP16 or BF16 use 2 bytes per parameter. A 70B model at BF16 is roughly 140 GB — two H100-80G or four A100-40G just to fit it. The first question in any GPU cost conversation is whether you actually need that precision.

AWQ (Activation-aware Weight Quantization) compresses to INT4 by identifying the 1% of weight channels that activate most strongly under representative inputs and protecting them from aggressive quantization. Everything else drops to 4 bits. The result, per the MLSys 2024 Best Paper, is a 3x speedup over HuggingFace’s FP16 implementation on a TinyChat backend, with the 70B Llama-2 deployable on hardware that could not previously fit it. The AWQ method requires no backpropagation and is calibration-dataset-agnostic, which matters when your fine-tuned model has a narrow domain and generic calibration data would overfit.

GPTQ takes a second-order optimization approach, using Hessian information to minimize quantization error weight-by-weight. It tends to take 2-4x longer than AWQ at quantization time but ships a slightly different accuracy profile on certain model families.

FP8 sits between full precision and INT4. On NVIDIA Hopper GPUs (H100, H200) with native Transformer Engine support, FP8 operates at near-BF16 quality with roughly 50% memory reduction and measurable throughput gains from the hardware’s native FP8 matrix math units. For teams that cannot accept any accuracy regression from INT4, FP8 is the production-viable compromise.

The practical starting point: reach for AWQ INT4 on anything 13B and up if you need to fit the model on a smaller GPU class or maximize batch size on the GPU you have. Use FP8 when you’re on Hopper hardware and quality sensitivity is high.

KV Cache Management: The Hidden Memory Tax

The key-value cache stores intermediate attention computations for each token in a sequence. At inference time, this cache grows linearly with sequence length and batch size — it’s not a fixed overhead. For Llama-3-70B at a 4,096-token context, a single request occupies approximately 1.3 GB of KV cache. At batch size 32, that’s 41 GB before you’ve loaded the model weights.

Traditional serving frameworks pre-allocated a contiguous memory block per request, wasting 60-80% of that allocation on average due to fragmentation and over-reservation. PagedAttention, introduced with vLLM at SOSP 2023, applies an operating-system paging model to KV cache: memory is divided into fixed blocks (16 tokens per block by default in vLLM), requests get blocks as they’re needed, and blocks are returned when requests complete. The measured waste drops to under 4%, and reported throughput gains over FasterTransformer and Orca range from 2-4x on production-representative workloads.

In practice, vLLM’s --gpu-memory-utilization parameter (default 0.90) controls how much VRAM is reserved for the KV cache pool. Lowering it reduces available cache and increases preemption — requests get paused, recomputed, and resumed, adding latency. Framework choice changes how much headroom you get; see our LLM serving framework comparison. The correct value is the highest setting that keeps your OOM risk near zero.

Two extensions worth knowing: FP8 KV storage (switching KV tensors from BF16 to FP8) halves KV cache memory with negligible quality regression for most workloads. Chunked prefill, enabled by default in recent vLLM versions, breaks large prefills into smaller chunks batched alongside decode operations, improving both TTFT and inter-token latency on mixed workloads.

If your serving workload includes agentic workflows with repeated system prompts, SGLang’s RadixAttention variant caches shared prefix KV blocks across requests, eliminating redundant recomputation. This is the relevant optimization for chains that re-inject the same context window on every step. Monitoring production model behavior for drift when switching between quantization levels is a separate concern — ML observability tooling at sentryml.com covers drift detection for quantized models.

Batching: The Metric Is GPU Utilization, Not Request Count

Static batching collects a group of requests, runs the forward pass, and waits for all requests to complete before accepting new ones. At realistic inter-arrival times this produces 40-50% GPU utilization because longer sequences block shorter ones from starting.

Continuous batching, implemented in vLLM and SGLang, inserts new requests into the decode loop at the iteration level — as soon as one sequence completes a token step and has room to exit, a waiting request enters. RunPod’s benchmark of vLLM under steady traffic shows 60-85% GPU utilization with continuous batching, versus 40% with static.

The operational implication is that your p99 latency for short requests improves dramatically because they no longer queue behind long sequences. The tradeoff is that continuous batching makes batch composition less predictable, which complicates cost-per-token accounting.

Speculative decoding layers on top of batching: a small draft model generates candidate token sequences, the large target model verifies them in a single forward pass. On generation-heavy workloads (long completions, low rejection rate), this yields 2-3x effective throughput improvement. It fails when the draft model’s distribution diverges from the target — common on code or structured output tasks with narrow valid-token windows.

Parallelism: When One GPU Isn’t Enough

Tensor parallelism (TP) splits weight matrices across GPUs, reducing per-GPU memory and allowing models that exceed a single GPU’s VRAM capacity. vLLM exposes this as --tensor-parallel-size N. Communication overhead via NVLink is low on well-provisioned hardware; PCIe-only systems suffer a measurable throughput penalty.

Pipeline parallelism (PP) assigns model layers to different GPUs sequentially. It’s suited for very large models where TP communication would bottleneck. The latency cost is bubble time between stages — stages are never perfectly synchronized.

Expert parallelism (enable_expert_parallel=True in vLLM) is specific to Mixture-of-Experts architectures. It distributes experts across GPUs, enabling efficient serving of models like Mixtral without over-provisioning for the inactive experts.

For teams running at scale: CPU allocation is a non-obvious GPU efficiency lever. vLLM’s scheduling and input processing requires at minimum 2 + N physical cores, where N is the GPU count. Hyperthreading systems need double. Underprovisioning physical cores starves the scheduler and degrades GPU utilization even with the model fully loaded. The vLLM optimization documentation covers the full parallelism matrix.

Caveats

Quantization and reasoning models: INT4 quantization inflates reasoning token counts on chain-of-thought-heavy models. A recent arXiv analysis found this can offset the throughput gains from smaller weights — benchmark your specific model on your actual task distribution before committing to INT4 in production.

KV cache FP8 and long contexts: FP8 KV precision loss accumulates over long sequences. Quality regression is typically negligible at 4k tokens; behavior at 32k-128k contexts requires empirical validation.

Continuous batching and SLA isolation: Dynamic batch composition means a single long request can inflate latency for short ones during preemption events. If you’re serving a mix of interactive and batch workloads, separate them into distinct vLLM instances rather than sharing a continuous-batch pool. Repeat traffic is worth intercepting before it reaches the GPU at all, which is what semantic caching does. Quantisation also shortens the weight-load path, which is the dominant term in serverless cold starts — see serverless vs dedicated GPU for LLM hosting for how that changes the hosting decision.

Security teams evaluating LLM serving stacks should also assess prompt injection surface across serving configurations — aisec.blog tracks active exploitation patterns against deployed inference endpoints.