Token cost observability should explain the economics of a completed LLM task: model calls, retries, cache categories, and any GPU capacity allocated to serve it. A trace helps identify where work happened. A cost ledger establishes what that work consumed and how it was priced.
This guide focuses on token and GPU economics. For instrumenting the request path, use LLM cost and latency observability with OpenTelemetry. For attribute definitions and their stability, use OpenTelemetry GenAI semantic conventions.
Separate usage, pricing, and allocation
Keep three records distinguishable:
| Record | What it contains | What it cannot establish alone |
|---|---|---|
| Usage event | Model attempt, input/output tokens, outcome, timestamp | The contract rate that applied |
| Rate card | Model, billing category, currency, effective period | Whether all attempts were observed |
| Capacity ledger | Allocated instances, billed duration, storage and operating costs | Which application feature consumed useful work |
Langfuse’s cost-tracking documentation distinguishes usage and cost information, including supplied values and calculated estimates. Preserve that provenance in your ledger. Estimated usage is useful, but should not be indistinguishable from a provider-reported count.
The Amazon Bedrock pricing page illustrates model-specific pricing and different billing modes. Store the effective rate rather than joining historical usage to whichever price is current when a dashboard loads.
Price attempts before aggregating user requests
A simple token-priced attempt can be modeled as:
attempt_cost =
sum(tokens_in_billing_category × rate_per_million_for_category)
/ 1,000,000Use mutually exclusive billing categories according to the provider’s definitions. If cached input is already included in total input tokens, subtract or partition it appropriately before applying separate rates. Do not add cached input on top of an inclusive input count.
For a hypothetical request with 2,000 input tokens at $1 per million and 500 output tokens at $4 per million, the generation cost is $0.004. Those are illustrative rates, not a model quote.
If the same complete call is repeated once and billed identically, the two attempts cost $0.008. A dashboard recording only the successful final attempt would show half that amount. Actual failed-attempt billing depends on the provider and failure point, so preserve supplied usage or reconcile the uncertainty rather than assuming all failures are free.
Attribute spend to features and releases
Connect attempts to a user-facing operation with a stable request identifier. Attach a feature, deployment environment, model route, and prompt/retrieval release where applicable.
Use opaque account identifiers where account allocation is needed, and restrict access to that ledger. Avoid exporting raw identities or full prompt content merely to obtain cost totals. Aggregate metric labels should remain bounded; per-request detail belongs in an event store suited to that cardinality.
Calculate cost per completed task as well as cost per request attempt. State what completion means: transport success, validated output, or a task-quality outcome. Track the quality result separately when it arrives after billing.
Compare features and releases under a consistent task mix. A feature serving longer documents can be more expensive for a legitimate reason. Investigate changes within comparable cohorts before applying a quota or switching models.
Cache savings need a counterfactual
Distinguish an application response cache from provider-side prefix caching. A response-cache hit may avoid a model attempt. A provider cache category may reduce a rate while the model still processes a generation request.
Record the actual cache category and actual bill first. Estimate avoided cost separately using a documented baseline, including the cost of embeddings or cache lookup if relevant.
For example, estimate savings for a response cache from the requests that would otherwise have required generation, then subtract the cache’s operating cost. Label that number as modeled savings. It is not an invoice credit.
A high hit rate alone does not establish success if the returned answer is stale or outside the request’s permissions. The semantic caching guide covers those controls. Cost optimization needs an unchanged acceptance bar.
GPU costs include paid idle time
A self-hosted model may have no external token invoice. Its economics still include allocated capacity. Amazon EC2’s instance-state documentation gives one billing example: a running instance and a stopped instance have different compute-charge behavior, while storage can continue to cost money.
Allocate the serving fleet’s cost over the same reporting period used for completed work:
allocated_cost_per_completed_task =
(compute + storage + allocated_operations_cost) / completed_tasksIf completed tasks are zero, report the idle allocation directly. Do not replace the undefined ratio with a reassuring zero.
Document how shared-fleet costs are apportioned. Token share, execution time, or reserved-capacity share are possible accounting policies; none should be described as exact per-request GPU billing unless the system actually supplies it. Keep unallocated overhead visible.
The self hosting LLM vs API cost guide provides the full TCO and break-even model. vLLM vs TGI explains why capacity comparisons must meet the same latency and quality objectives. The serving framework comparison covers engine choices that change achievable batch size, and with it cost per completed task.
Add retrieval and failed work to the budget
For RAG, price embedding refreshes, vector storage, retrieval requests, reranking, and the extra context sent to generation. Returning more chunks changes both retrieval work and input tokens.
The best vector database for RAG guide separates index capacity from raw-vector storage. Include rebuild and backup assumptions in the cost model rather than assigning the whole application bill to the generator.
Keep failed requests, canceled work, fallbacks, and repeated tool calls visible in the ledger. A successful final response can conceal several preceding attempts. Retain uncertainty markers when usage for an interrupted stream is unavailable.
Reconcile and alert on explainable changes
Reconcile the ledger against provider invoices and infrastructure bills at a defined cadence. Investigate missing attempts, duplicate events, rate changes, currency mismatches, credits, and minimum commitments before changing application behavior. Compare tools for cost observability against these accounting and reconciliation requirements.
Set feature-level budget alerts with minimum traffic and completeness checks. Compare cost per completed task, attempt count, cache mix, and quality together. An increase caused by a rollout needs the responsible release identifier, which the LLM/prompt registry pattern supplies.
Keep the response proportionate: inspect a rate-card mismatch, limit a repeated-call failure, or evaluate a cheaper route against the same task requirements. A global token cap can lower the invoice while making the application less useful.
Keep instrumentation and economics connected
The LLMOps tools on GitHub map covers the components that can emit these events. Use ML Observe’s end-to-end tracing guide for request structure and SentryML’s conventions coverage for schema definitions.
The economic contract remains the same across implementations: identifiable work, a dated price basis, explicit allocation, and a completed-task denominator. Without those, a cost chart can show a total while leaving the hosting decision unexplained.
