Skip to main content

Guardrails Architecture

The guardrail system runs every user message, the turn's output stream that processRequest() returns, and the streams that continue a turn after an external tool call, through a two-phase dispatcher. Input guardrails run before the orchestrator sees the message. Output guardrails run on the chunks of the turn's stream before they reach the client: every guardrail evaluates the chunks that carry isFinal: true (the FINAL_RESPONSE, an ERROR the turn yields), a guardrail with evaluateStreamingChunks: true also evaluates each TEXT_DELTA, and every other chunk passes through unevaluated. The ERROR that processRequest() yields when an exception reaches its own catch is not on the turn's stream and is not evaluated. Each guardrail returns one of four verdicts (ALLOW, SANITIZE, BLOCK, FLAG), and the dispatcher composes them in a fixed order. A guardrail that throws, or runs past its timeoutMs, is skipped with a logged warning; with failClosed: true in its config, it blocks instead. A guard the runtime config names in requiredGuardrails always runs fail-closed with the deadline given there, and an answer that is not one of the four verdicts blocks.

This page documents the internals. For recipe-style usage of the shipped guardrail packs (PII, ML classifiers, topicality, code safety, grounding), see Guardrails System.

Request Lifecycle​

Every user message passes through the input guardrails before the orchestrator sees it. The chunks of the turn's stream pass through the output guardrails before processRequest() yields them to the client. The streams that handleToolResult(), handleToolResults() and resumeExternalToolRequest() return, which continue a turn after an external tool call, pass through the same output guardrails. While a guard named in requiredGuardrails is not active, each of these calls yields one SYS_GUARDRAIL_REQUIRED_MISSING error chunk and reaches no provider; initialize() refuses to finish in that state.

The four possible verdicts are:

  • ALLOW — content passes through unchanged.
  • SANITIZE — content is modified in-place (e.g., PII replaced with [PERSON]) and the modified version continues downstream.
  • BLOCK — content is rejected. For input, an error response is returned immediately. For output, the dispatcher yields one ERROR chunk, whose code is the guardrail's reasonCode or GUARDRAIL_BLOCKED, and the stream ends; when the verdict carries replacementText, it yields a FINAL_RESPONSE holding that text, with the block in metadata.guardrail.output, in place of the ERROR chunk.
  • FLAG — content passes through unchanged, but metadata is attached for downstream logging and auditing.

Two-Phase Parallel Execution​

The dispatcher splits registered guardrails into two phases based on whether they can modify content (canSanitize). Sanitizers must run sequentially (each one's output feeds the next). Non-sanitizing guardrails run in parallel for maximum throughput.

Worst-wins aggregation: if any parallel guardrail returns BLOCK, the final verdict is BLOCK regardless of what the others returned. FLAG wins over ALLOW.

Streaming Chunk Lifecycle​

A guardrail with evaluateStreamingChunks: true evaluates each TEXT_DELTA as it arrives, up to its maxStreamingEvaluations for the stream; every guardrail evaluates each chunk that carries isFinal: true. The dispatcher counts streaming evaluations by the guardrail object's id property: sanitizers (canSanitize: true) without an id share one count, so one can use up the limit of another. Give each streaming sanitizer an id. A BLOCK verdict on an evaluated chunk ends the stream with an ERROR chunk (or the guard's replacement reply). TOOL_CALL_REQUEST, SYSTEM_PROGRESS, METADATA_UPDATE and every other chunk with isFinal: false pass through without evaluation.

In hold mode (guardrailOutputMode: 'hold', or a guard required on the output stage) no TEXT_DELTA reaches the client before the final guardrails have judged the whole reply: the dispatcher holds the deltas and sends them just before the FINAL_RESPONSE when the verdict allows or flags it, and drops them when it blocks or sanitizes it. An actionable TOOL_CALL_REQUEST closes the window: the held text is judged first as a FINAL_RESPONSE the dispatcher builds, then sent with the request. A stream that ends without a final chunk drops its held text.

Chunk Types​

TypeKey FieldsWhen It AppearsOutput guardrails
TEXT_DELTAtextDelta, isFinal: falseEach text delta the model streamsEvaluated by guardrails with evaluateStreamingChunks: true
FINAL_RESPONSEfinalResponseText, ragSources, usage, error, isFinal: trueThe turn's response, at the end of the streamEvaluated by every guardrail
TOOL_CALL_REQUESTtoolCalls: [{id, name, arguments}], executionMode, requiresExternalToolResultThe model asks for tools; processRequest() returns after a request the host must execute (executionMode: 'external')Not evaluated
TOOL_RESULT_EMISSIONtoolCallId, toolName, toolResult, isSuccess, errorMessageThe result of an external tool call, when the host returns it; results of tools the runtime runs itself are not on this streamNot evaluated (isFinal: false, not a text delta)
SYSTEM_PROGRESSmessage, progressPercentageStatus updatesNot evaluated
ERRORcode, message, isFinal: trueAn error, a guardrail block includedEvaluated by every guardrail when the turn yields it; not the ERROR a block produces, the one processRequest() yields from its own catch, nor SYS_GUARDRAIL_REQUIRED_MISSING

Memory Budget​

All models lazy-load on first use. Nothing is loaded until a guardrail actually evaluates content.

PackIdleActiveWhat Loads
PII Redaction0~115MBOpenRedaction + compromise + BERT NER
ML Classifiers0~98MBtoxic-bert + DeBERTa + PromptGuard
Topicality0~1.7MBTopic centroid embeddings
Code Safety~10KB~10KBCompiled regex (always loaded)
Grounding Guard0~40MBNLI cross-encoder
Combined~10KB~255MBOnly if ALL packs + ALL tiers active

References​

LLM safety surveys​

  • Wei, A., Haghtalab, N., & Steinhardt, J. (2023). Jailbroken: How does LLM safety training fail? NeurIPS 2023. — Foundational analysis of why prompt-injection and jailbreak attacks succeed; motivates the per-chunk evaluation model used here. arXiv:2307.02483
  • Greshake, K., Abdelnabi, S., Mishra, S., Endres, C., Holz, T., & Fritz, M. (2023). Not what you've signed up for: Compromising real-world LLM-integrated applications with indirect prompt injection. AISec '23. — Indirect prompt injection threat model the input-side guardrails defend against. arXiv:2302.12173

Toxicity + harm classification​

  • Hartvigsen, T., Gabriel, S., Palangi, H., Sap, M., Ray, D., & Kamar, E. (2022). ToxiGen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection. ACL 2022. — Training-data foundation for the ML classifier pack's toxicity probes. arXiv:2203.09509
  • Markov, T., Zhang, C., Agarwal, S., Eloundou, T., Lee, T., Adler, S., Jiang, A., & Weng, L. (2023). A holistic approach to undesired content detection in the real world. AAAI 2023. — OpenAI's content-classifier methodology; informs the multi-category labeling approach in the ml-classifiers pack. arXiv:2208.03274

PII detection​

  • Pilán, I., Lison, P., Øvrelid, L., Papadopoulou, A., Sánchez, D., & Batet, M. (2022). The Text Anonymization Benchmark (TAB): A dedicated corpus and evaluation framework for text anonymization. Computational Linguistics, 48(4), 1053–1101. — Evaluation methodology for the pii-redaction sanitizer. arXiv:2202.00443

Grounding / hallucination detection​

  • Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023). Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12), 1–38. — Survey of hallucination types the grounding-guard pack targets. arXiv:2202.03629
  • Min, S., Krishna, K., Lyu, X., Lewis, M., Yih, W.-t., Koh, P. W., Iyyer, M., Zettlemoyer, L., & Hajishirzi, H. (2023). FactScore: Fine-grained atomic evaluation of factual precision in long form text generation. EMNLP 2023. — Atomic-fact verification methodology behind grounding-guard's per-claim source-attribution check. arXiv:2305.14251

Streaming guardrails​

  • Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., Joseph, N., Kadavath, S., Kernion, J., Conerly, T., El-Showk, S., Elhage, N., Hatfield-Dodds, Z., Hernandez, D., Hume, T., ... Kaplan, J. (2022). Constitutional AI: Harmlessness from AI feedback. arXiv preprint. — Critique-and-revise pattern that informs the content-policy-rewriter pack's sanitize-rather-than-block design. arXiv:2212.08073

Implementation references​