Skip to main content

Guardrails Architecture

The guardrail system runs every user message, and the turn's output stream that processRequest() returns, through a two-phase dispatcher. Input guardrails run before the orchestrator sees the message. Output guardrails run on the chunks of the turn's stream before they reach the client: every guardrail evaluates the chunks that carry isFinal: true (the FINAL_RESPONSE, an ERROR the turn yields), a guardrail with evaluateStreamingChunks: true also evaluates each TEXT_DELTA, and every other chunk passes through unevaluated. The ERROR that processRequest() yields when an exception reaches its own catch is not on the turn's stream and is not evaluated. Each guardrail returns one of four verdicts (ALLOW, SANITIZE, BLOCK, FLAG), and the dispatcher composes them in a fixed order. A guardrail that throws, or runs past its timeoutMs, is skipped with a logged warning; with failClosed: true in its config, it blocks instead.

This page documents the internals. For recipe-style usage of the shipped guardrail packs (PII, ML classifiers, topicality, code safety, grounding), see Guardrails System.

Request Lifecycle​

Every user message passes through the input guardrails before the orchestrator sees it. The chunks of the turn's stream pass through the output guardrails before processRequest() yields them to the client. The streams that handleToolResult(), handleToolResults() and resumeExternalToolRequest() return, which continue a turn after an external tool call, do not pass through output guardrails.

The four possible verdicts are:

  • ALLOW — content passes through unchanged.
  • SANITIZE — content is modified in-place (e.g., PII replaced with [PERSON]) and the modified version continues downstream.
  • BLOCK — content is rejected. For input, an error response is returned immediately. For output, the dispatcher yields one ERROR chunk, whose code is the guardrail's reasonCode or GUARDRAIL_BLOCKED, and the stream ends.
  • FLAG — content passes through unchanged, but metadata is attached for downstream logging and auditing.

Two-Phase Parallel Execution​

The dispatcher splits registered guardrails into two phases based on whether they can modify content (canSanitize). Sanitizers must run sequentially (each one's output feeds the next). Non-sanitizing guardrails run in parallel for maximum throughput.

Worst-wins aggregation: if any parallel guardrail returns BLOCK, the final verdict is BLOCK regardless of what the others returned. FLAG wins over ALLOW.

Streaming Chunk Lifecycle​

A guardrail with evaluateStreamingChunks: true evaluates each TEXT_DELTA as it arrives, up to its maxStreamingEvaluations for the stream; every guardrail evaluates each chunk that carries isFinal: true. The dispatcher counts streaming evaluations by the guardrail object's id property: sanitizers (canSanitize: true) without an id share one count, so one can use up the limit of another. Give each streaming sanitizer an id. A BLOCK verdict on an evaluated chunk ends the stream with an ERROR chunk. TOOL_CALL_REQUEST, SYSTEM_PROGRESS, METADATA_UPDATE and every other chunk with isFinal: false pass through without evaluation.

Chunk Types​

TypeKey FieldsWhen It AppearsOutput guardrails
TEXT_DELTAtextDelta, isFinal: falseEach text delta the model streamsEvaluated by guardrails with evaluateStreamingChunks: true
FINAL_RESPONSEfinalResponseText, ragSources, usage, error, isFinal: trueThe turn's response, at the end of the streamEvaluated by every guardrail
TOOL_CALL_REQUESTtoolCalls: [{id, name, arguments}], executionMode, requiresExternalToolResultThe model asks for tools; processRequest() returns after a request the host must execute (executionMode: 'external')Not evaluated
TOOL_RESULT_EMISSIONtoolCallId, toolName, toolResult, isSuccess, errorMessageThe result of an external tool call, when the host returns it; results of tools the runtime runs itself are not on this streamNot evaluated: it arrives on the streams that continue the turn, which skip output guardrails
SYSTEM_PROGRESSmessage, progressPercentageStatus updatesNot evaluated
ERRORcode, message, isFinal: trueAn error, a guardrail block includedEvaluated by every guardrail when the turn yields it; not the ERROR a block produces, nor the one processRequest() yields from its own catch

Memory Budget​

All models lazy-load on first use. Nothing is loaded until a guardrail actually evaluates content.

PackIdleActiveWhat Loads
PII Redaction0~115MBOpenRedaction + compromise + BERT NER
ML Classifiers0~98MBtoxic-bert + DeBERTa + PromptGuard
Topicality0~1.7MBTopic centroid embeddings
Code Safety~10KB~10KBCompiled regex (always loaded)
Grounding Guard0~40MBNLI cross-encoder
Combined~10KB~255MBOnly if ALL packs + ALL tiers active

References​

LLM safety surveys​

  • Wei, A., Haghtalab, N., & Steinhardt, J. (2023). Jailbroken: How does LLM safety training fail? NeurIPS 2023. — Foundational analysis of why prompt-injection and jailbreak attacks succeed; motivates the per-chunk evaluation model used here. arXiv:2307.02483
  • Greshake, K., Abdelnabi, S., Mishra, S., Endres, C., Holz, T., & Fritz, M. (2023). Not what you've signed up for: Compromising real-world LLM-integrated applications with indirect prompt injection. AISec '23. — Indirect prompt injection threat model the input-side guardrails defend against. arXiv:2302.12173

Toxicity + harm classification​

  • Hartvigsen, T., Gabriel, S., Palangi, H., Sap, M., Ray, D., & Kamar, E. (2022). ToxiGen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection. ACL 2022. — Training-data foundation for the ML classifier pack's toxicity probes. arXiv:2203.09509
  • Markov, T., Zhang, C., Agarwal, S., Eloundou, T., Lee, T., Adler, S., Jiang, A., & Weng, L. (2023). A holistic approach to undesired content detection in the real world. AAAI 2023. — OpenAI's content-classifier methodology; informs the multi-category labeling approach in the ml-classifiers pack. arXiv:2208.03274

PII detection​

  • Pilán, I., Lison, P., Øvrelid, L., Papadopoulou, A., Sánchez, D., & Batet, M. (2022). The Text Anonymization Benchmark (TAB): A dedicated corpus and evaluation framework for text anonymization. Computational Linguistics, 48(4), 1053–1101. — Evaluation methodology for the pii-redaction sanitizer. arXiv:2202.00443

Grounding / hallucination detection​

  • Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023). Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12), 1–38. — Survey of hallucination types the grounding-guard pack targets. arXiv:2202.03629
  • Min, S., Krishna, K., Lyu, X., Lewis, M., Yih, W.-t., Koh, P. W., Iyyer, M., Zettlemoyer, L., & Hajishirzi, H. (2023). FactScore: Fine-grained atomic evaluation of factual precision in long form text generation. EMNLP 2023. — Atomic-fact verification methodology behind grounding-guard's per-claim source-attribution check. arXiv:2305.14251

Streaming guardrails​

  • Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., Joseph, N., Kadavath, S., Kernion, J., Conerly, T., El-Showk, S., Elhage, N., Hatfield-Dodds, Z., Hernandez, D., Hume, T., ... Kaplan, J. (2022). Constitutional AI: Harmlessness from AI feedback. arXiv preprint. — Critique-and-revise pattern that informs the content-policy-rewriter pack's sanitize-rather-than-block design. arXiv:2212.08073

Implementation references​