Judge
An LLM-based policy engine that reads traffic and decides block, sanitize or warn.
Judge runs admin-written, natural-language policies through an LLM to decide on gateway traffic and the server-side agent loop. Unlike Guardrails' deterministic checks, the decision comes from a model — which is exactly why when it gets called (the "tiered gate" below) is the product's single most important cost control.
#Four surfaces
| Surface | Inspects | Where it runs |
|---|---|---|
| request | The last user message | The proxy, before the request goes upstream |
| response | The assistant's final reply text | The proxy, after the reply arrives (streaming behavior depends on mode) |
| tool_call | A tool call's arguments | Two distinct sites: the proxy (when the client runs its own tools, before the reply reaches the client) and the agent loop (when the server runs the tool itself, before execution) |
| context | Text about to be fed back INTO the model — in the agent loop, chiefly a tool's result | Only the agent loop, right after a tool runs, before the result is handed to the model |
#Three actions
- block
- Stops the request/turn. There is no reroute — a blocked request is never redirected to another model or group.
- sanitize
- Rewrites the text and continues: `spans` masks the quoted fragments with a placeholder, `rewrite` replaces the whole text (only when the input wasn't truncated and the result stays a sane length). On the `tool_call` surface `rewrite` is always downgraded to `spans` — replacing the JSON going to a tool with free text is unsafe.
- warn
- The decision is logged; nothing is changed or stopped.
#Execution modes
| Mode | When the judge model gets called |
|---|---|
| always | On every matching request — the most expensive mode. |
| tiered | A cheap deterministic gate (keyword/regex/min length) runs first; if it doesn't match, the judge model is never called. |
| sample | Called at a fixed probability (`sample_rate`) — the rest of the requests are never evaluated. |
| async | Always passes the gate, but runs off the request's critical path in the background — so whatever action it's configured with, it can never block or sanitize, only ever produce a `warn`. |
#When the judge itself is unavailable
A policy's `fail_open` field (default: on/true) decides what happens when the judge call itself fails at the infrastructure level (timeout, malformed reply, no model configured…). With it on, the failure is only logged/flagged and the request proceeds normally — the judge's own trouble never reaches the user. With `fail_open=false` (closed), that same infrastructure failure is enforced like a violation: gateway surfaces (request/response/client-side tool_call) get a 422, agent-loop surfaces (tool_call/context) get a readable tool error fed back to the model.
#Clustering
Policies that share the same judge model (and aren't marked `isolated`) merge into a single LLM call — five policies on the same model make one call, not five. Token usage, latency and cost are written only to the cluster's first policy; the other members show these fields blank, which is not missing data — it's the mark of a shared call. Every verdict row's `cluster_id` ties together the rows that came from that one call.