LLM Serving Guardrails: Placement and Failure Policy

A serving policy must decide when a request may advance, what happens when a check is unavailable, and which bytes or actions may leave the service. This reference turns the five rail types described by NeMo Guardrails into a proposed operating policy. It uses the OWASP Top 10 for LLM Applications to identify risks at input, retrieval, dialog, output, and tool-execution boundaries.

The failure policies below are design recommendations, not a tested deployment. The per-check costs come from the cited vendor documentation, not from measurements on this site. For classifier selection, recall, false positives, and shadow evaluation, use prompt injection detection in production. Here the question is how to enforce a decision once a check returns, fails, or times out.

Serving-policy matrix

NeMo’s rail types supply both the placement vocabulary and the configuration surface, so each row names the config.yml key or flow location where that check is actually switched on. OWASP’s prompt-injection, sensitive-information-disclosure, improper-output-handling, and excessive-agency risks explain why permission to proceed must be checked at multiple boundaries.

A deny means the check completed and rejected the request. An error or timeout means no valid decision arrived. Keep these outcomes separate in telemetry even when they lead to the same refusal.

The five placements as one configuration

Nothing in the file below is application code. Placement, ordering, and release behavior are configuration, which is what makes them reviewable:

# config.yml
models:
  - type: main
    engine: openai
    model: gpt-4o

rails:
  input:
    flows:
      - self check input          # policy screen on the user turn
      - llama guard check input   # hazard-taxonomy classifier
  retrieval:
    flows:
      - screen retrieved chunks   # your Colang flow over retrieved context
  dialog:
    single_call:
      enabled: False              # keep the canonical-form step separate and inspectable
    user_messages:
      embeddings_only: False
  output:
    flows:
      - self check output
      - llama guard check output
    streaming:
      enabled: True               # evaluate buffers instead of holding the whole answer
      chunk_size: 200
      context_size: 50
      stream_first: False         # decide before the bytes leave, not after

All four input and output flows in this file call a model: self check input and self check output prompt the configured LLM, and the two llama guard flows run Llama Guard 3. Llama Guard 3 is a fine-tuned Llama 3 model that classifies a prompt or a response against a published hazard taxonomy, the MLCommons hazard categories plus a code-interpreter-abuse category, and returns a safe or unsafe label with the codes of any violated categories. That has two consequences for the matrix above. On the Output row, both flows read the generated answer as model input, so their cost scales with the length of the response and neither can be budgeted as a cheap post-filter; size them as model work on the critical path. On the Execution row, the code-interpreter-abuse category is the part of the taxonomy that speaks to tool execution, and recording the violated category code is better than recording a free-text reason, because the code stays stable across policy revisions and survives a prompt rewrite.

These are deliberately conservative failure policies. Each rail also needs a deadline, which the configuration above does not supply: derive it from the route’s latency budget and the measured duration of the check, and record expiry as an error rather than as an allow. Do not silently turn an unavailable required check into permission to proceed. A low-risk route may have a separately reviewed degraded mode, but specify its permitted outputs and disabled actions before an incident. Human approval time is outside the execution-check deadline; keep the action pending while approval is absent.

Treat each approved input, context set, and action as bound to the version of the policy that evaluated it. If retrieved content or tool arguments change after a check, require a new decision for the changed material. For output handling, pair content checks with schema validation and the escaping required by the downstream consumer; a classifier decision should not stand in for those controls.

Streaming: decide the release boundary first

OWASP’s disclosure and output-handling risks make the release boundary consequential. Choose one of these policies explicitly, and include it in the route’s acceptance criteria:

• Whole-response buffering. Hold the complete answer when a rule needs the complete answer to decide, such as a final schema or cross-sentence disclosure check. This is the behavior you get by leaving the rails.output.streaming block off. Release only after all required output checks pass. If generation or checking fails, discard the pending answer and return a controlled fallback. Model time to first token is then different from time to first user-visible content.
• Checked-buffer release. For a route whose rules can be evaluated incrementally, collect a bounded buffer, evaluate it with the required preceding context, and release only the approved portion. In the configuration above that is chunk_size for the buffer, context_size for the preceding text each buffer is judged against, and stream_first for whether a chunk leaves before or after its rail runs. Define all three, plus a maximum wait. Do not claim this enforces a whole-answer rule before the whole answer exists.
• Tool calls. Keep partial tool arguments internal. Assemble and validate the complete call, apply the execution-rail policy, and only then allow execution. Displaying streamed text must never itself authorize a tool.

Already released content cannot be withdrawn by a later check. If a later buffer fails, stop further release, discard pending buffers, and record the abort; do not expose the rejected text in the error message. A final check remains useful for review, but cannot replace a required pre-release decision. Exercise deny, timeout, and disconnect cases in the evaluation pipeline, including boundaries between buffers.

What each rail costs per request

Count model calls before assigning durations. For the config.yml above with whole-response buffering, a request that passes every check makes these calls:

That is the main generation plus up to four rail model calls, and three of the five calls go to the main LLM. The token-cost consequence follows from where each call lands. Both self-check calls are billed at the main model’s rates: self check input resends the user turn wrapped in the self-check prompt, and self check output sends the finished answer back to the same model as input, so every answer is paid for once as generated output and again as checked input, plus the prompt template each time. The two Llama Guard 3 calls bill wherever you host that model, as GPU time or per-token fees, and each one processes the checked text as input. A deny on input stops the request before generation, so a rejected request costs only the input-rail calls.

The counts assume whole-response buffering. With the rails.output.streaming block shown above, both output flows run on every chunk_size buffer, so an answer that spans several buffers multiplies the two output calls by the number of buffers.

Per-check durations from vendor documentation

Only some of these checks have a published duration. Treat the vendor figures as hardware-specific starting points and read the rest from your own traces:

The two Prompt Guard rows show the range for a dedicated classifier placed in front of the rails: per 512-token window, the 86M model takes close to five times as long as the 22M model, and text longer than one window needs several classifications. For a hypothetical retrieval batch of four 512-token chunks scored one after another, that is 4 × 92.4 = 369.6 ms with the 86M model and 4 × 19.3 = 77.2 ms with the 22M model. The detection guide covers choosing between them.

The model-based rows have no constant duration. A self-check call costs whatever the main LLM takes to read the self-check prompt and the checked text and return a verdict, and a Llama Guard 3 pass grows with the length of the text it classifies. This config.yml contains no rule-based output check; a schema validator or pattern filter you add would be the only output step cheap enough to treat as a near-constant. Budget self check output as one more LLM call alongside the Llama Guard pass, both on the critical path between generation and release.

Serial versus concurrent checks

If the two input flows run one after the other, the input stage costs t(self check input) + t(llama guard check input). If your serving layer runs them concurrently, it costs max(t(self check input), t(llama guard check input)); neither check is omitted, and both must approve before retrieval starts. The same reasoning applies to the two output flows on a buffered answer. Retrieval screening still follows retrieval, and output checking still follows generation. Do not overlap a required approval with the work it gates. An execution rail adds its own check to any request that runs a tool; the tool’s own duration and any human approval need separate budgets.

This is critical-path arithmetic. It excludes scheduling, network variance, and queueing, and it is not a p95 prediction: adding stage p95 values does not establish request p95. Measure the complete request distribution under the intended load, with each release mode reported separately.

Telemetry and rollout acceptance

Langfuse’s guardrails guidance connects safety controls with observability. Its primitives map onto the matrix directly: give each required stage its own span inside the request trace, so that a rail’s duration and outcome are attributable to the request that triggered it, and attach the check result to the trace as a score, so allow, deny, error, timeout, and skipped stay distinguishable in aggregate after the fact. Record the violated category code from a Llama Guard deny on the same span. A missing span is not evidence that a check passed.

Use bounded metric labels for stage, policy revision, and decision category. Keep detailed request correlation in access-controlled traces, and avoid putting raw prompts, retrieved secrets, or rejected output into metric labels. Track check duration and timeout rate alongside refusal rate, approved-context count, user-visible first-byte time, and stream aborts.

Before enforcing the policy, verify that a timeout at each boundary produces the documented result, no pending buffer is released after a denial, and no tool executes without an approved action decision. Use sampled review to assess false blocks and missed violations; an increase in classifier alerts alone does not prove an attack. The detection guide covers that calibration work.

For broader context, sentryml.com covers production tracing, and mlmonitoring.report covers quality and drift monitoring.