# Agent Work Runtime (/docs/ai/durable-agent-runtime) Ductor's agent runtime separates **accepted work** from **connected clients**. A chat request or workflow node first admits an immutable turn to an append-only journal. Only then can a runtime lease execute provider and tool work. Chat and workflow are host adapters around that same state machine. ```mermaid stateDiagram-v2 [*] --> Accepted: admit immutable turn Accepted --> Running: acquire lease Running --> Running: commit model delta / tool receipt Running --> NeedsInput: approval required NeedsInput --> Running: bound response accepted Running --> NeedsReconciliation: external outcome unknown NeedsReconciliation --> Running: reconciler response accepted Running --> Completed: final response committed Running --> Failed: terminal error / bound exceeded Completed --> [*] Failed --> [*] ``` ## Durable invariants [#durable-invariants] | Invariant | Operational effect | | ------------------------------ | ---------------------------------------------------------------------------------------------------------------------- | | Append-only journal | History is auditable; state is projected from committed events rather than overwritten blobs. | | Tenant-scoped session identity | Reads, cursors, input responses, and execution leases cannot cross tenant boundaries. | | Monotonic sequence | Every committed event has a stable replay cursor. A future cursor is a conflict, not an empty success. | | Immutable admission | Reusing a turn identity with different input, definition, manifest, policy, or budget is rejected. | | Idempotent execution | Duplicate admission or concurrent workers do not issue duplicate provider/tool work for the same committed transition. | | Detached accepted work | Browser cancellation closes delivery, not the admitted turn. | | Bounded loop | Iterations, provider requests, tool calls, and cost policy prevent an unbounded autonomous loop. | | Exact approval binding | Tenant, session, turn, request, sequence, tool version, schema, receipt, and arguments remain bound across a wait. | | Explicit model policy | Provider/model selection is pinned and explainable; a strict policy never silently falls back. | | Isolated delegation | Declared specialists inherit narrower tools and budgets without receiving the parent transcript or memory. | The journal stores normalized events: accepted turns, runtime leases and state changes, model deltas, tool invocations and redacted results, input requests and responses, usage, completion, and terminal failures. Raw provider reasoning and credential-bearing payloads do not belong in it. ## One runtime, three hosts [#one-runtime-three-hosts] ```mermaid flowchart LR C["Chat host"] --> R["Canonical runtime"] W["Workflow host"] --> R E["External conversation bridge"] --> R R --> J["Agent journal"] R --> I["Inference vertical"] R --> G["Governed tool broker"] C --> S["Resumable SSE"] W --> N["Node result / approval wait"] E --> P["Provider thread reply"] ``` * The [chat host](/docs/ai/chat-agent) admits an operator turn and projects committed journal events into SSE. Reconnect starts strictly after `Last-Event-ID`. * The [workflow host](/docs/ai/workflow-ai-steps) derives a stable agent session from run/node identity and maps runtime outcomes to node state. Approval requests become durable workflow waits. * The [external conversation bridge](/docs/ai/external-conversation-bridges) verifies provider requests and explicit participant bindings, durably orders each thread, admits turns to the same runtime, and sends the terminal result back through the governed connector action path. Neither host owns provider-specific parsing or tool execution. The inference vertical normalizes provider streams and accounts for usage; the tool broker resolves a pinned tool definition, enforces policy, executes through the connector registry, and returns a redacted receipt. ## Definitions and presets [#definitions-and-presets] At admission, the runtime resolves an agent definition: model, instructions, output schema, tools, limits, model policy, ordered processors, memory scope, delegation policy, and policy context. Tenant presets can be version-pinned. Tool definitions are immutable for the lifetime of a turn and include the canonical name, version, and schema hash. A later catalog change therefore causes an explicit mismatch instead of silently changing an in-flight agent. The model decision receipt records eligible candidates, the selected provider/model, policy version, scope, and reason. Every proxy request is strict: a missing or prohibited provider, or a response attributed to another provider, fails closed. An allowed provider transition must become a new durable decision; retries inside an adapter cannot hide a model-policy change. ## Context, processors, and memory [#context-processors-and-memory] The immutable transcript is audit history; the context projection is the bounded request sent to a model. Projection accounts for the selected model ceiling, reserved output tokens, tool schemas, complete assistant/tool batches, and a conservative token count. When compaction is needed, the summary records its source sequence range, generating model, prompt hash, token metadata, and supersession lineage. Definitions pin ordered input, request, output, and error processors. The default pipeline bounds input, records injection signals, redacts secrets—including values split across stream frames—prevents tool expansion, validates the pinned model decision, supports fail-closed moderation, checks structured output, and emits a stable error taxonomy. Processor receipts contain hashes and bounded signals, not redacted content. Durable memory is opt-in. The Postgres backend requires exact tenant, authenticated subject, entity type, and entity id; it returns only non-expired records marked `model_safe`, within item and byte ceilings. The projector rechecks scope and provenance and injects recall as explicitly untrusted user-role context. There is no public free-form memory insertion endpoint. ## Typed input and specialist delegation [#typed-input-and-specialist-delegation] Approval is one kind of a general durable input protocol. `approval`, `question`, `budget_extension`, `external_auth`, and `reconciliation` requests carry an exact tenant/session/turn/sequence binding, response JSON Schema hash, actor/role policy, expiry, and resume state. A valid response resolves once; its redacted form and hash are auditable across restart. A parent definition may declare named specialists with exact child definition refs, tool allowlists, per-child budgets, output schemas, and depth/fanout/concurrency ceilings. The runtime exposes a generated `delegate_` tool; the model supplies only bounded `{id,prompt}` tasks. Server code derives child identity, intersects tools, inherits deadlines, enforces the parent's remaining aggregate budget, admits children through the same journal/runtime, validates outputs, and joins results in stable task order. Ductor does not expose an arbitrary recursive agent-spawn path. ## Structured output and multi-tool turns [#structured-output-and-multi-tool-turns] A provider iteration may emit content, multiple tool calls, usage, and a terminal reason. Tool-call fragments are assembled before validation. Each complete call is authorized independently and committed in deterministic order. When a definition declares a JSON output schema, final output must validate before completion; malformed output fails the turn instead of being stored as a successful result. The schema is also sent to compatible providers as a strict structured-output request, including OpenAI-compatible endpoints and Gemini, but local validation remains authoritative. ## Recovery boundary [#recovery-boundary] The committed sequence is the recovery boundary. A subscriber can always replay from the journal without re-calling a provider. An execution worker must acquire the session lease before advancing state, and duplicate attempts observe the already committed transition. Each chat replica performs a bounded recovery scan at startup and every five seconds. The scan selects only non-terminal sessions with no live lease; every candidate still has to win the normal atomic claim. Safe persisted states resume. A recovered `model_running` or `tool_running` state is an ambiguous external outcome and parks as `needs_reconciliation` instead of repeating possible work. Tool authorization is itself a recovery artifact. The live authenticated context is checked before the authorization receipt is committed. The receipt binds the tenant, session, invocation, manifest, schema, argument hash, effective arguments, execution mode, limits, and redaction policy. A recovered worker can therefore execute that same authority without relying on the original request context or accepting a new caller identity. Subscriptions also preflight cursor continuity. A future cursor is a conflict. If an operator's external archive policy has removed a requested prefix, chat returns HTTP 410 `resync_required` with the requested, earliest, and latest sequences. The default schema does not prune agent events. Durable delivery and idempotent transitions protect reconnects, retries, and competing workers. Operators should still configure provider timeouts, deployment shutdown grace, journal retention, and an operational reconciliation process for ambiguous external outcomes. Validate those conditions in a staging environment that matches the production topology. ## Production limits [#production-limits] Set the narrowest useful model, tools, request/tool bounds, and tenant budget. Treat an exceeded bound as a typed operational result, not permission to retry the whole turn. Keep approval required for mutations and destructive actions. Keep read-only diagnostics in the default safe policy, with output redaction and byte limits. Apply the database migrations included with your Ductor release before enabling the runtime. ## Tracing, feedback, and evaluation [#tracing-feedback-and-evaluation] Content capture is off by default. The trace hierarchy restores a persisted trace id across worker restart and emits only durable identifiers, definition/manifest/policy ids, provider/model/tool names, approval class, usage, outcome, and stable error type. Authenticated feedback binds to tenant, turn, step, and trace; only the note hash is retained. The evaluation plane stores versioned datasets, trajectory assertions, scores, and experiments. Promotion policies can require schema, security, recovery, tool-trajectory, and isolation checks. Model-judge scores can supplement those checks but do not override them. ## Related [#related] HTTP admission, resumable event delivery, and input responses. Manifests, policy receipts, schema binding, and approval modes. Membership, controls, cursor conformance, verified folds, and safe upgrades. Progressive disclosure from immutable signed instruction bundles. Deterministic, reauthorized conditions and durable notification intent. Authenticated messaging ingress, ordered thread processing, and browser-bound approvals.