The best LLM observability tools for a serving team are the ones that connect a slow or expensive request to the capacity and release configuration responsible. An application trace, an inference-server metric, and a GPU utilization series describe different parts of that problem.
This comparison focuses on serving-side latency, queueing, errors, and cost. For full application tracing, use end-to-end tracing for LLM applications. For convention definitions, use OpenTelemetry GenAI semantic conventions. The recommendations below are derived from official documentation, with no hands-on ratings.
LLM observability tools by serving signal
| Tool or signal source | Documented coverage | Serving question it helps answer | Additional evidence to collect |
|---|---|---|---|
| vLLM metrics with a Prometheus-compatible collector | Queue, request latency, token, and cache signals | Is the inference engine accumulating work? | Client timings, hardware metrics, and request mix |
| NVIDIA DCGM | GPU health and utilization/ profiling telemetry | What is happening on the accelerator? | Engine queue and application outcome |
| Langfuse | Generation usage and cost tracking | Which feature or generation accounts for spend? | Paid idle capacity and engine metrics |
| Arize Phoenix | LLM application tracing and evaluation | Which recorded step is associated with the request problem? | Server-side queue and GPU evidence |
| Datadog Agent Observability | Span-derived counts, errors, token usage, latency, and cost | How do instrumented application signals change together? | The actual engine and hardware integrations in the deployment |
The rows are complementary layers, not a score ranking. A GPU telemetry source cannot judge answer relevance, and an application cost estimate does not account for every idle GPU hour.
The vLLM production metrics guide and NVIDIA DCGM feature overview document the infrastructure side. Langfuse cost tracking, Phoenix documentation, and Datadog’s metric reference document the application side.
Start with the latency boundary
Define the user-visible interval you care about. Time to first token, time to completion, and time spent waiting in an engine queue are different intervals. A tool should make their boundaries explicit.
For a retrieval application, distinguish the vector query, reranking, prompt construction, generation, and gateway work. Some steps may overlap, so summing every child-span duration can overstate elapsed time. Inspect the critical path and client-visible duration rather than assuming all spans are sequential.
Use vLLM vs TGI to define serving acceptance criteria. Use the best vector database for RAG guide to make the retrieval workload part of the same latency budget.
Evaluate queue visibility before dashboard polish
A slow first token may be associated with queueing, prompt processing, a cold start, or another upstream delay. The observation alone does not prove which explanation is correct.
For a self-hosted engine, collect queue and latency metrics from the pinned release and correlate them with active requests and hardware signals. Confirm metric units and labels instead of copying a dashboard built for another version.
For a hosted API, mark unavailable server-side telemetry as unavailable. Client traces can establish the duration and result of a call, but do not reveal an internal provider queue unless the service exposes that information.
A useful acceptance check is a benign request fixture whose instrumented steps are known. Verify which timings appear, how errors are represented, and whether a failed attempt remains visible when another attempt succeeds.
Compare token cost with allocated capacity
Langfuse documents supplied and calculated usage/cost information. Datadog documents span-derived token and cost metrics. Preserve whether a value came from provider usage, a local estimate, or a configured rate calculation.
Then connect it to the application task. Multiple attempts can belong to one successful response. A final-span-only total can hide the work done by retries or fallbacks.
Self-hosted GPU economics require another record: the capacity that billed while requests were absent or waiting. Token cost observability explains that ledger, and self hosting LLM vs API cost adds storage, redundancy, and operations. Do not call a tool’s token dashboard a complete hosting TCO report.
Choose a deployment boundary you can operate
Compare the exact deployment and license tier under consideration. Self-hosting a telemetry service adds storage, upgrades, backup, and recovery responsibilities. A hosted service adds its own retention, ingestion, access, and pricing decisions.
Count events using the expected request graph. A task with several model calls generates more telemetry than a single-generation task. Include expected retries and retention when estimating ingestion and storage costs.
Check content capture separately from timing metadata. Full prompts and answers may not be necessary for a capacity dashboard. Verify the selected SDK’s actual defaults and configuration; a convention permitting an attribute does not mean every SDK captures it automatically.
The LLMOps tools on GitHub guide includes repository and deployment checks for the surrounding stack.
A serving-team acceptance checklist
Use the following as proposed selection criteria:
- Identify the service, model, release, and request cohort on each relevant signal.
- Observe client latency and engine latency as separate values with documented boundaries.
- Retain failed and retried attempts without double-counting completed tasks.
- Check exported token counts and pricing categories against a representative billing example.
- Confirm what GPU, queue, and retrieval signals require separate integrations.
- Verify retention, export, and application behavior when telemetry delivery fails.
- Estimate total telemetry operating cost at the intended request and retention volume.
These checks evaluate a deployment, not a vendor logo. Record gaps explicitly so the incident runbook does not expect a tool to produce evidence that was never collected.
When a dataset report is the better next step
Serving telemetry can show that latency and error rates stayed stable while answer quality declined. At that point, a scored request cohort and release comparison may be more useful than additional GPU charts.
The focused Arize vs Evidently comparison covers moving from an LLM incident to trace inspection and dataset reports. Framework and scoring methodology belongs with Evals ML’s framework comparison.
Practical selection
Start with the engine’s documented metrics if you operate inference capacity, then add hardware signals where they answer unresolved questions. Choose an application backend that preserves request and release context and makes its usage estimates identifiable.
Evaluate the full telemetry chain with your own fixtures and requirements. The useful result is a clear account of where requests wait, what work costs, and which evidence an operator can recover during an incident.
