05/Observability

Request Capture

Storing request and response bodies per scope: what is stored, how it is switched on, how long it stays.

Request Capture stores THREE bodies for a request that passes through the gateway: the ORIGINAL body the client sent, the FINAL body the gateway sent upstream, and the model's RESPONSE. The reason both request forms are kept is the DIFFERENCE between them: the system prompt, second brain memory, skills and guardrails injection is visible exactly there. That difference is what answers "why did the model answer like that" — with only one body you see the request, but not what the gateway added to it.

OFF by default, and bodies are sensitive data

With no configuration, no body is stored. When one is stored, what is stored is the user's real text: conversation content, injected second brain memory, skill bodies and system prompts. Only fields that LOOK like credentials are masked during capture; message content is not scanned and not masked. Captured bodies are readable only with an admin identity, and the panel deliberately offers no "copy"/"download" button.

#The three bodies

Client

request_originalfrozen, no injection

routerinjectionsystem prompt · skills · second brain · guardrails

request_finalsent upstream

Upstream model

response_bodystream teed, written at the end

The original is frozen BEFORE injection

Body	When it is taken
`request_original`	At the proxy entrance, before ANY injection. It is frozen by serialising it to text immediately — the steps below replace `data` wholesale, so holding a reference would store the final body under the "original" label.
`request_final`	Immediately before it is sent upstream — that is, AFTER the system prompt, skills, second brain, guardrails and judge sanitize.
`response_body`	The parsed body on a non-streaming response. On a stream the chunks are teed as they pass and reassembled after the stream ENDS, then written — so the stream itself is never delayed.

Capture is not limited to chat endpoints: endpoints such as embeddings that pass through the proxy are captured the same way. When a row is expanded in the panel, the default tab is the "Changes" diff rather than two blobs side by side — that difference is the whole reason both request forms are stored.

#Where it is switched on — scope

Capture is switched on per scope. The vocabulary is the same one used across the panel (System Prompts, Skills, Judge): `global`, `user`, `workspace`, `node`, `group`, `model`, `mapping`. Each configuration carries its own retention, its own per-body limit, and which of the three bodies it stores.

System Prompts stack EVERY matching record; here there is nothing to stack. A row is either written to ClickHouse or it is not, and if it is written it carries exactly one `retention_days` — a body cannot live for both 30 and 365 days. So the matches are reduced to a single winner: **the narrowest match wins**.

Narrowness order	What it covers
`mapping` → `model`	A single mapping or a single real model (narrowest).
`node` → `group`	The traffic of one node or one model group.
`workspace` → `user`	One workspace, or ALL of that user's traffic.
`global`	Everything, without distinction (broadest).

At equal narrowness, the disabled row wins

A disabled configuration is NOT filtered out before matching — it is selected, and it says "off". That is how you switch capture on globally and leave one user out; if disabled rows were dropped first, a narrow "do not capture this user" would lose to a broad "capture everything" and the off switch would silently do nothing. Two rows can match at the same narrowness — the `node` scope matches a node's name, code AND id, so `node:my-node` and `node:42` catch the same physical node at the same level — and there the DISABLED one wins: when in doubt, the body is not stored. Every remaining tie is broken by `(scope_type, scope_value)`, so the decision never depends on row order.

If nothing matches, the answer is off — that is also the behaviour of an empty table. A configuration that is enabled but has all three body flags off stores nothing, and the panel warns about exactly that. The configurations are cached, but every create/edit/delete invalidates the cache immediately: a capture you switch off stops on the next request, not minutes later.

#Retention

Each configuration carries a retention between **1 and 3650 days**; the default is 30. The row carries that number in its own column, so different scopes live for different durations in the same table. `0` and "unlimited" are not offered.

Why 3650 days — a measured limit

ClickHouse's TTL expression must produce a `DateTime`, and `DateTime` is 32-bit (its ceiling is 2106-02-07). Past that ceiling nothing errors — it SILENTLY wraps: `toDateTime('2020-01-01') + toIntervalDay(36500)` measures as **1983-10-31**. So a value meaning "keep for 100 years" makes the row deletable at the next merge — the exact opposite of the intent. The cap is not a preference; it is a data-loss guard.

The defence is deliberately layered, and both layers point the same way — TOWARD KEEPING THE DATA. The API REJECTS an out-of-range value rather than clamping it: a value clamped in silence would lie to the admin, who would believe "kept for 3650 days" while having typed 36500. The schema is the LAST line, and it works on rows the API never saw: for a row inserted by hand in SQL, or written before this rule existed, `retention_days` is clamped at the ceiling and a stray `0` is read as "the longest life", not "delete now". That is not a contradiction but a division of labour: the API refuses the input, and the schema refuses to turn what it could not refuse into lost data.

ClickHouse applies TTL during background merges (`merge_with_ttl_timeout`, 4 hours by default). An expired row is removed at the first eligible merge rather than on the second: retention is an UPPER BOUND, not a stopwatch. The way to stop storing bodies altogether is to switch the configuration off, not to shorten the TTL.

#Truncation

Every configuration has a per-body byte limit; the default is 128 KB, selectable between 16 KB and 256 KB in the panel — the largest preset below the API ceiling of roughly 341 KB, which is itself derived from the write path's 1 MiB budget per queued record. A body over the limit is **truncated and FLAGGED — never dropped silently**. In the panel a truncated body carries a "Truncated" badge; a body shown without that badge would be trusted further than it should be.

The write path has its own ceiling too: a single queued record (all three bodies together) may not exceed 1 MiB. So a setting above roughly 341 KB per body is truncated once more at write time — still flagged. When the two ceilings disagree, the gateway logs a warning at startup.

If all three bodies end up empty, no ClickHouse row is written at all: it would cost TTL, disk and panel noise while answering no question — the request's metadata already sits in the activity row in Postgres.

#What is NOT stored

Headersnever

The schema has NO header column; no request or response header is stored — that is the strongest masking available. Credential-shaped fields INSIDE the body (`authorization`, `api_key`, `cookie`, `access_token` and the like) are replaced with `***`, and proxy-internal `_`-prefixed keys are dropped entirely.

The gateway's own self-callsconsent

The MCP agent loop, judge evaluation, second brain distiller and content-router classifier are not captured. The reason is CONSENT, not noise: these calls authenticate as the service identity, so a user's own "do not capture me" row would not match them — storing them would leave a capture the user believes is off running under the admin name. Their content already sits inside the outer request's captured bodies.

Costelsewhere

The captured row deliberately leaves `cost_usd` empty. Cost is computed on the activity row in Postgres together with the price catalog; computing it a second time would create a place where two numbers could quietly drift apart.

Sampling100%

There is no sampling rate: all matching traffic is captured. Adding sampling later is easy; bringing the past back is impossible.

#Where the data lives

Bodies are not MOVED out of Postgres, they are placed NEXT to it. Billing and plan limits stay transactional in `user_activity_logs` and this feature does not touch them; the bodies plus a few denormalised fields for analysis go to ClickHouse. `request_uid` joins the two.

request_logs (ClickHouse): The three bodies (`ZSTD`-compressed), the truncation flags, `retention_days`, and which scope made the decision. `MergeTree`, partitioned by month, with the TTL read from the row's own retention column.
user_activity_logs (Postgres): The billing and usage record — UNCHANGED by this feature.
request_uid: The link between the two stores; the panel fetches the bodies by this key when a row is expanded.
CLICKHOUSE_URL / _USER / _PASSWORD / _DB: The store's address and credentials; the compose default database is `manifold_logs`.

The write path never slows a request down or fails one: all it does on the request side is a single `rpush` onto a BOUNDED (32 MiB) Redis queue, and a background batch writer sends it to ClickHouse (roughly every 2 seconds, over HTTP `JSONEachRow`). If ClickHouse goes down the batch is put back and retried with exponential backoff; if the queue's byte budget fills up the OLDEST records are dropped and counted — silent unbounded growth would kill Redis.

There is NO separate page. The "Body capture" card at the top of Observability → Request Logs manages the scopes, and expanding a request row shows that request's captured bodies and the difference between them. When the store cannot be reached, the panel says so separately rather than confusing it with "no record".