vLLM vs TGI Serving Comparison: Throughput, Latency, and EOL Risk

Every vLLM vs TGI serving comparison written before this year was really a benchmark argument: whose continuous batching squeezes more tokens per second out of an A100. That argument is mostly settled now, but the choice itself got more complicated, not less. The wider field, including SGLang and TensorRT-LLM, is covered in our LLM serving framework roundup. Hugging Face put Text Generation Inference into maintenance mode, and the project’s own GitHub repo carries a caution banner telling new deployments to look elsewhere. So the comparison that matters in 2026 isn’t just p99 latency at batch size 64 — it’s whether you’re picking a serving stack with a future or one you’ll be migrating off within a year.

The operational reality first

The TGI documentation and its GitHub repository both carry the same notice: TGI “is now in maintenance mode,” accepting only bug fixes and documentation edits going forward, with Hugging Face explicitly directing users to vLLM and SGLang for new work. If you’re running TGI in production today, that means no new model architecture support, no new quantization formats, and a shrinking window on security patches. That last part should worry you more than any throughput gap — a serving layer that stops receiving CVE fixes is a supply-chain problem, not a performance one. If you track AI-specific vulnerability disclosures, an incident tracker like ai-alert.org is a reasonable place to watch for anything landing against a frozen dependency like this.

Practically: greenfield deployments should default to vLLM. The only reason to reach for TGI now is an existing deployment you haven’t migrated yet, or a narrow workload where it still wins on hardware you already own.

Serving is one layer of a longer toolchain, and the same maintenance-mode question applies to the gateway, tracing, evaluation and guardrail projects sitting around it. The open-source LLMOps stack on GitHub maps those layers and the repo signals worth checking before you commit to any of them.

The metric that matters: goodput, not raw throughput

Raw tokens/sec is the number vendors lead with, and it’s the wrong one to optimize alone. What you actually want is goodput — throughput measured only over requests that meet a latency SLO, typically expressed as tokens/sec at a fixed time-to-first-token (TTFT) and inter-token latency (ITL) ceiling. A server that hits 12,000 tokens/sec by starving half its requests past your SLO isn’t actually serving 12,000 tokens/sec of usable traffic.

The formula in practice:

goodput = (tokens generated by requests where TTFT ≤ SLO_ttft AND ITL ≤ SLO_itl) / measurement_window

This beats plain throughput because it’s the number your on-call actually gets paged on. A dashboard that only shows aggregate tokens/sec will look fine while your p99 TTFT quietly blows past 2 seconds for a fifth of traffic. Pair it with token-cost observability so latency and spend move on the same dashboard.

The mechanism driving both engines’ throughput is the same idea, implemented differently. vLLM’s PagedAttention paper treats the KV cache like OS virtual memory — paged, non-contiguous, shareable across requests — which is what lets it push GPU memory utilization near 100% instead of leaving 60-80% fragmented and unusable, per the original SOSP paper’s own accounting. TGI implements continuous batching (iteration-level scheduling that merges new requests into an in-flight batch instead of waiting for a static batch to drain) plus FlashAttention, described in its own documentation, but without PagedAttention’s page-table-style memory scheme. An independent arXiv comparison of the two systems found vLLM’s throughput advantage widens sharply as concurrency increases, while TGI held a tail-latency edge in low-concurrency, single-user interactive scenarios — there’s no universal winner, only a workload-dependent one. Anyscale’s earlier benchmark work on continuous batching (a vendor writeup, not independent) reported up to a 23x throughput gain over naive batching once continuous batching and PagedAttention-style memory management are combined, which is the baseline both engines are now measured against.

Wiring it up

Both engines expose Prometheus-compatible /metrics endpoints out of the box, so the instrumentation story is nearly identical. Launch each with metrics enabled:

# vLLM OpenAI-compatible server, metrics on by default at /metrics
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.90 \
  --max-num-seqs 256

# TGI, same target endpoint shape
docker run --gpus all -p 8080:80 \
  ghcr.io/huggingface/text-generation-inference:latest \
  --model-id meta-llama/Llama-3.1-8B-Instruct \
  --num-shard 2 \
  --max-batch-total-tokens 16000

Scrape both the same way:

scrape_configs:
  - job_name: llm-serving
    metrics_path: /metrics
    static_configs:
      - targets: ["vllm-host:8000", "tgi-host:8080"]
    scrape_interval: 15s

vLLM exposes vllm:time_to_first_token_seconds, vllm:e2e_request_latency_seconds, and vllm:gpu_cache_usage_perc as histograms/gauges you can graph directly for goodput. TGI exposes an equivalent set (tgi_request_queue_duration, tgi_request_inference_duration, tgi_batch_current_size). If you’re already running a monitoring stack for model drift and data quality, extending the same Prometheus/Grafana pipeline to inference-serving metrics is the natural next step — see sentryml.com for the broader observability side of that stack, since serving latency and model-quality drift end up on the same on-call rotation in practice.

What you’ll see

Good: gpu_cache_usage_perc (vLLM) sitting in the 80-95% band under steady load, TTFT flat across concurrency until you hit your configured max-num-seqs ceiling, then a controlled queueing increase rather than a cliff. Bad: TTFT climbing linearly with concurrency well before you’ve saturated GPU memory — usually a sign your batch-size or max-batch-total-tokens config is too conservative for the hardware, or that KV cache fragmentation (more likely on TGI without paged memory) is capping effective batch size below what the raw VRAM would allow.

Caveats

Published vLLM-vs-TGI numbers vary 2-5x across writeups because model size, sequence length, quantization, and GPU generation aren’t held constant between them — treat any single benchmark chart as a starting hypothesis, not a production guarantee, and re-run on your own model and hardware before committing. TGI’s long-context prefill path has been reported faster than vLLM’s on some single-GPU, very-long-prompt configurations, so a workload dominated by long-document prefill rather than short-turn generation is the one place worth actually testing TGI before ruling it out. And remember that a maintenance-mode project can still be fast in a benchmark while being the wrong long-term choice — throughput charts don’t show you the CVE backlog. And if the real decision is build versus buy, start from self-hosting an LLM vs API cost; if it is already build, the next fork is serverless vs dedicated GPU hosting, where the batching behaviour described above is what makes scale-to-zero expensive.