Concept drift detection in LLM production needs thresholds that connect output quality to a particular task and release. A changed prompt-length histogram or embedding distribution is an investigation signal; it does not by itself prove that the relationship between inputs and correct outcomes changed.
This guide focuses on LLM production thresholds and release decisions. The general distinction belongs in concept drift vs data drift explained. A discrepancy between training and serving transformations is a different investigation, covered by training-serving skew detection.
Start with three separate signal groups
Evidently’s drift documentation describes reference/current distribution comparisons. Its customization guide documents configurable methods and thresholds. Treat those settings as statistical configuration, not a universal production paging rule.
For an LLM service, use the following suggested separation:
| Signal group | Examples | What it supports |
|---|---|---|
| Request and context changes | Input length, retrieval counts, task mix | Investigate changes to the workload or available context |
| Output quality | Task pass rate, judged relevance, validated structured output | Assess whether the application still meets its task requirements |
| Serving health | Queue time, time to first token, failures | Investigate capacity or execution problems |
A serving failure can prevent a correct answer without constituting concept drift. A change in the mix of tasks can change aggregate scores even when each task cohort behaves the same. Keep those explanations available before deciding that a model needs retraining.
The third row needs instrumentation the first two do not. LLM observability tools for serving signals sets out which sources expose queue time, time to first token, and accelerator utilization, and why an application tracer alone will not show a saturated engine behind an unchanged quality score.
Define a threshold contract per task
Choose an operational action first: investigate, stop a rollout, revert a configuration, or collect more labels. Then document the evidence required for that action.
A useful threshold record includes the task cohort, reference period, sample minimum, metric definition, acceptable degradation, persistence requirement, and response owner. It should also include the release versions and scorer configuration.
For example, a team could require a quality decline to persist across two complete comparison windows before opening an investigation, while handling an invalid-output surge immediately through an availability runbook. These are illustrative policy choices, not recommended universal thresholds.
A small sample should produce an explicit insufficient-data state. It should not quietly pass as healthy or generate an alarm solely because one rare request failed. Choose the sample requirement and uncertainty treatment as part of the evaluation design.
For scoring methodology and harness selection, use LLM evaluation frameworks. The operational threshold should consume a defined metric rather than reinventing its meaning in a dashboard.
Compare matching cohorts and fixed revisions
Group requests by task and relevant application configuration before comparing periods. Keep prompt version, model route, retrieval revision, and scorer revision available as dimensions in the investigation record.
A prompt release that lengthens context can increase cost and latency without changing correctness. A new retrieval index can change answer evidence while the generation model stays fixed. These require different remediation from a change in the task’s expected answers.
The LLM/prompt registry pattern provides the version bundle. For serving metrics, vLLM vs TGI covers queue, latency, and cache signals that help separate capacity problems from quality changes.
Do not pool a new scorer with an old scorer and interpret the resulting jump as a production regression. Evaluate their agreement on shared examples and keep their histories distinguishable.
Delayed labels require provisional conclusions
If correctness is known only later, associate the eventual outcome with the original request and release. Compare completed label windows so recent requests with missing outcomes do not distort the result.
Before labels arrive, distribution changes and automated scores can identify candidates for review. Mark them as proxies. You cannot calculate a residual requiring an observed outcome before that outcome exists.
Evidently’s LLM evaluation quickstart documents descriptors applied to evaluation datasets, including checks and judge-based scoring. Those tools can support an investigation dataset; their scores still need an interpretation tied to the task.
Track missing-feedback rates as well as quality. A healthy-looking score with a shrinking labeled sample may reflect incomplete observation rather than stable performance.
A release-aware response sequence
Use this recommended investigation order:
- Confirm that telemetry and feedback are arriving and that comparison windows are complete.
- Check whether task mix, scorer configuration, or sample coverage changed.
- Locate changes to prompt, model, retrieval, serving engine, or routing.
- Compare quality and serving signals within the affected cohort.
- Choose a reversible response tied to the suspected layer, preserving evidence for validation.
For a RAG service, inspect the retrieval release before assuming the generator must change. The vector database comparison explains filtered retrieval and corpus-version checks.
For a capacity regression, evaluate the hosting plan and request mix. Self hosting LLM vs API cost connects capacity changes with completed-work cost. Increasing GPU allocation without identifying the bottleneck can change spend while leaving the quality problem intact.
Connect production findings to the next gate
A confirmed incident should add a representative case to the release dataset. Preserve the expected behavior, relevant dependency versions, and the reason the previous gate missed it.
Run the candidate fix through LLM eval pipelines in CI/CD, then assess its rollout against the same threshold contract. A deployment passing CI and a service meeting production requirements are separate observations.
Keep this page’s thresholds specific to LLM tasks. For statistical drift definitions and detector selection, follow the ML Monitoring Report drift pillar. The objective here is a clear operational decision with attributable evidence, not an additional general drift framework.
