Prompt Injection Detection in Production: What to Alert On

The incident that forces the question usually looks like this: a RAG assistant with tool access retrieves a page, the page carries hidden instructions, and the model follows them instead of the system prompt. Nothing in the request logs looks unusual, because the attack rode in on a document, not a user message. Prompt injection detection in production is the discipline of catching that class of input before the model acts on it, and it is harder than the vendor landing pages suggest: the detectors are real and useful, but they are probabilistic classifiers with false-positive budgets, blind spots, and drift, and they need the same operational treatment as any model in the serving path.

OWASP LLM01:2025 splits the problem into direct injection (the user tries to override the system prompt) and indirect injection (instructions embedded in retrieved or third-party content). The indirect form is the one that scales against you. Greshake et al. demonstrated it against deployed LLM applications back in 2023, framing retrieved text as something close to arbitrary code execution for an LLM-integrated app, and everything agentic shipped since has widened that surface. If your pipeline feeds the model anything a third party can influence (web pages, support tickets, email bodies, PDF uploads), you have the exposure whether or not you have the detector.

Where detection sits in the request path

A detector is a small text classifier that scores input before it reaches the main model. In practice you screen at three points:

• User turns, for direct injection and jailbreak attempts.
• Retrieved chunks and tool outputs, for indirect injection. This is the placement most teams skip and the one that matters most for agents.
• Model output, as a tripwire: if the response contains your system prompt verbatim or an unexpected tool call, something upstream already failed.

The realistic options are a small self-hosted classifier or a managed API. Meta’s Llama Prompt Guard 2 is the reference point for the first camp: an 86M-parameter mDeBERTa classifier with a 512-token window that labels text benign or malicious, with a 22M variant when the latency budget is tight. Azure’s Prompt Shields is the managed version, and notably ships separate endpoints for user-prompt attacks and document attacks, which maps cleanly onto the direct/indirect split. Microsoft’s own docs are candid that the shields produce false positives and negatives and should not be the only layer.

Detection is one layer, not the defense. Least-privilege tool scopes, output filtering, and human confirmation on consequential actions do the load-bearing work when the classifier misses; the wider guardrail stack is covered in our piece on guardrails in the serving path, and https://guardml.io tracks the defensive tooling landscape in more depth.

The metric that matters: recall at a fixed false-positive rate

The obvious metrics, accuracy and AUC, will mislead you here, because the base rate is brutally skewed. On a normal day, well under one request in a thousand is an attack. A detector with a 1% false-positive rate on 1M requests per day generates roughly 10,000 false blocks per day, every one of them a legitimate user staring at a refusal. Accuracy hides this entirely, and AUC averages over thresholds you will never operate at.

The metric to tune against is recall at a fixed false-positive rate: pick the FPR your product can tolerate (usually well under 1%), find the score threshold that produces it on benign production traffic, and report the fraction of attacks caught at that threshold. This is the number vendors quote when they are being honest; the Prompt Guard 2 model card reports 97.5% recall at 1% FPR on English jailbreaks for the 86M model, against an AUC of .998 (vendor benchmark, on Meta’s own evaluation set). For independent evaluation, Lakera’s PINT benchmark is deliberately built so detectors cannot have trained on it, and over a fifth of its 4,314 inputs are hard negatives: benign text that talks about injection, quotes attack strings, or otherwise looks malicious. That composition is the point. Your worst false positives will come from users discussing prompt injection, and a security-focused product will hit them constantly.

Wiring it up: shadow mode first

Never ship a detector straight into blocking mode. Run it in shadow: score everything, block nothing, and export metrics until you know your real-world FPR at the candidate threshold.

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer
from prometheus_client import Counter, Histogram

MODEL_ID = "meta-llama/Llama-Prompt-Guard-2-86M"
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForSequenceClassification.from_pretrained(MODEL_ID).eval().to("cuda")

DETECTIONS = Counter(
    "pi_detections_total", "Inputs scored over threshold", ["source", "action"]
)
SCORES = Histogram(
    "pi_score", "Malicious-class probability", ["source"],
    buckets=[0.1, 0.5, 0.9, 0.95, 0.99],
)

@torch.inference_mode()
def injection_score(text: str) -> float:
    enc = tokenizer(text, truncation=True, max_length=512, return_tensors="pt")
    probs = torch.softmax(model(**enc.to(model.device)).logits, dim=-1)
    return probs[0, 1].item()

def screen(text: str, source: str, threshold: float = 0.98, enforce: bool = False) -> bool:
    score = injection_score(text)
    SCORES.labels(source=source).observe(score)
    if score >= threshold:
        DETECTIONS.labels(source=source, action="block" if enforce else "shadow").inc()
        return enforce
    return False

Call screen() per user turn and per retrieved chunk, with source set to user, retrieval, or tool_output. The label is what makes the dashboard useful: a spike in retrieval detections against a specific connector is an indirect-injection campaign; a spike in user is someone probing you by hand. Documents longer than 512 tokens must be chunked before scoring or the tail goes unscreened, and the model card’s latency figure (92.4 ms per 512-token classification on an A100, vendor number) tells you retrieval-side screening belongs on a batched async path, not inline on time-to-first-token.

What you’ll see on the chart

Healthy shadow mode looks bimodal: a large mass of scores near zero, a thin spike near one, and very little in between. Two patterns should trigger action. A slow rise in the mid-range (0.5 to 0.9) usually means input drift, a new document source, a new locale, a new user cohort, shifting the benign distribution toward the threshold; re-derive the threshold before it becomes a false-positive incident. A sudden cluster of high scores concentrated on one source label is an active campaign, and the offensive playbooks evolve fast enough that it is worth following the attack-side literature at https://aisec.blog to know what the cluster likely contains.

Caveats

• Hard negatives are your steady-state pain. Support tickets quoting attack strings, security documentation, and users pasting error logs all score high. Route high-score-but-blocked traffic to review, and keep an allow-path escape hatch.
• The 512-token window is an attack surface. An instruction split across chunk boundaries can score benign in every chunk. Overlap your chunks when screening.
• Classifiers catch injection-shaped text, not goal hijacking. A semantically clean instruction (“summarize this, then email it to…”) inside a document can pass every filter. Tool-scope restrictions catch what detectors cannot.
• Benchmark numbers travel badly. Vendor evals overstate production performance because public attack datasets leak into training sets; independent held-out sets like PINT exist precisely because of this.
• Multilingual coverage is thinner. Prompt Guard 2’s own card shows multilingual AUC below its English number; if you serve non-English traffic, evaluate on it before trusting the threshold.

Treat the detector like any other model you operate: shadow first, threshold from your own traffic, per-source metrics, and a periodic re-eval against a held-out attack set. That is the whole discipline, and it beats any single product decision.

Related across the network

• The AI Security Tools Directory: 40+ Tools Compared (2026) — aisecbench.com
• How to Detect Prompt Injection: Four Approaches Ranked — aiattacks.dev
• AI Defense Techniques for LLMs: A Practitioner’s Guide — aidefense.dev
• Jailbreak Detection for LLMs Explained: How Runtime Filters Work — aidefense.dev
• How to Detect Jailbreak Prompts: A Practitioner’s Guide — aimoderationtools.com