Agent Work Runtime

Ductor's agent runtime separates accepted work from connected clients. A chat request, workflow node, or RFQ pricing job first admits an immutable turn to an append-only journal. Only then can a runtime lease execute provider and tool work. Execution hosts are host adapters around that same state machine.

Durable invariants

The journal stores normalized events: accepted turns, runtime leases and state changes, model deltas, tool invocations and redacted results, input requests and responses, usage, completion, and terminal failures. Raw provider reasoning and credential-bearing payloads do not belong in it.

One runtime, explicit execution hosts

• The chat host admits an operator turn and projects committed journal events into SSE. Reconnect starts strictly after Last-Event-ID.
• The workflow host derives a stable agent session from run/node identity and maps runtime outcomes to node state. Approval requests become durable workflow waits.
• The external conversation bridge verifies provider requests and explicit participant bindings, durably orders each thread, admits turns through the chat host, and sends the terminal result back through the governed connector action path. A bridge is not a separate execution host.
• agentrfq_bidder runs the opt-in bounded discover/evaluate/submit loop for explicitly configured worker targets. Its runtime and admission planner are independent of chat configuration. A renewable pricing-job lease and fresh credential authority govern admission and recovery; generic recovery cannot substitute for them. See Clear AI work.

Each admission and runtime explicitly declare their host. The journal persists that ownership for the session's entire lifetime. Admission retries, external identity collisions, later turns, and compatibility upgrades cannot transfer it; cross-host reuse is refused. Clients cannot select a host in public payloads.

The hosts share provider parsing; chat and workflow can also execute governed tools. The inference vertical normalizes provider streams and accounts for usage; the tool broker resolves a pinned tool definition, enforces policy, executes through the connector registry, and returns a redacted receipt. The typed-decision plane can choose an exact enabled model before the first definition pin and can classify a hard-gate-clean attempt as close, continue, or review without becoming the execution authority.

Definitions and presets

At admission, the runtime resolves an agent definition: model, instructions, output schema, tools, limits, model policy, ordered processors, memory scope, delegation policy, and policy context. Tenant presets can be version-pinned. Tool definitions are immutable for the lifetime of a turn and include the canonical name, version, and schema hash. A later catalog change therefore causes an explicit mismatch instead of silently changing an in-flight agent.

The model decision receipt records eligible candidates, the selected provider/model, policy version, scope, and reason. Every proxy request is strict: a missing or prohibited provider, or a response attributed to another provider, fails closed. An allowed provider transition must become a new durable decision; retries inside an adapter cannot hide a model-policy change.

An optional typed-decision policy can rank an exact configured subset of enabled candidates before the definition's first pin. Policy version, candidates, evaluator model, projection/question versions, threshold, and evidence become part of the immutable definition. Existing sessions reuse that definition and do not make a fresh routing call.

After a model attempt passes hard tool, schema, authorization, and budget gates, an optional completion policy can classify it as close, continue, or review. The runtime commits the evaluation receipt before acting. Continue removes provisional assistant output and emits an ordered text_reset before a fresh projection; review retains the candidate response for explicit approval.

Bounded RFQ pricing

The bidder uses a separately published, exact version-and-hash pricing preset. That preset decides an offer or decline; the registered worker's original fulfilment preset remains the identity stamped on the bid and later executed by the award-bound workflow. Pricing source pins and the actual governed pricing execution definition are both retained, without replacing the pricing preset's own output schema or applying deployment-wide semantic defaults.

Pricing accepts no tools, skills, specialists, or enabled memory. Model selection is static, completion evaluation is off, and the tool executor refuses every call. Only a completed journal outcome that passes the pricing schema and strict proposal validation can become an offer. Partial text, malformed output, and non-completed outcomes never supply bid terms. A validated proposal is saved before canonical submission; worker ownership, eligibility, disclosure, and the original fulfilment pins are checked again before submission.

Before a new pricing job can spend, Postgres reserves its full model-request ceiling against the target's durable lifetime allowance. Reservation and unique job creation are atomic; the counter is never reset or refunded, including for declines and failed or uncertain attempts. The initial maximum can only be reduced. A lower configured cap cannot restore consumed slots, and a cap below already reserved requests prevents new jobs. Existing jobs retain their frozen bounds and deadline. Duplicate target entries cannot create a second job for the same tenant, canonical environment, market session, RFQ, and worker.

The original acceptance time, disclosed task, authority subject, budget, deadline, session/turn identity, exact admission input, and validated proposal survive restart. Database-clock leases fence stale owners. Recovery first checks for the exact already-submitted bid, including after a market closes; it does not re-price a saved proposal or extend a deadline. The machine API key is resolved from a tenant-default secret and authenticated afresh on each target pass and before paid admission, recovery, and submission. A stored subject is never used to impersonate current authority.

The bidder has a dedicated inference proxy with zero automatic retries; chat and workflow retry settings remain their own. Canonical usage accounting and configured monthly budget admission still apply. The request ceiling counts runtime requests and conservatively reserves slots; it is not a provider billing idempotency guarantee. The per-evaluation cost guard runs after a response and cannot promise a strict pre-call monetary ceiling or reserve the RFQ's purchase currency. Configure the request allowance and pricing deadline before enabling bounded bidding.

Public bidder observation reads persisted target allowances and all-state job history under the actual caller's RFQ-read permission, environment scope, and worker ownership. It can inspect prior work while automatic bidding is disabled or the worker is paused. Each page is an independent read-only snapshot. Reading never reserves slots, claims a lease, refreshes the target's machine credential, or resumes pricing.

The allowance counter records full request ceilings reserved for jobs, not actual model calls or cost. A saved proposal is distinct from a market bid; submitted records the runner's persisted submission outcome, not independent market or signed-receipt proof. A persisted target does not establish current configuration, runner health, or permission to start another evaluation. Use award observation and receipt retrieval for the downstream execution and settlement evidence.

Context, processors, and memory

The immutable transcript is audit history; the context projection is the bounded request sent to a model. Projection accounts for the selected model ceiling, reserved output tokens, tool schemas, complete assistant/tool batches, and a conservative token count. When compaction is needed, the summary records its source sequence range, generating model, prompt hash, token metadata, and supersession lineage.

Definitions pin ordered input, request, output, and error processors. The default pipeline bounds input, records injection signals, redacts secrets—including values split across stream frames—prevents tool expansion, validates the pinned model decision, supports fail-closed moderation, checks structured output, and emits a stable error taxonomy. Processor receipts contain hashes and bounded signals, not redacted content.

Durable memory is opt-in. The Postgres backend requires exact tenant, authenticated subject, entity type, and entity id; it returns only non-expired records marked model_safe, within item and byte ceilings. The projector rechecks scope and provenance and injects recall as explicitly untrusted user-role context. There is no public free-form memory insertion endpoint.

Typed input and specialist delegation

Approval is one kind of a general durable input protocol. approval, question, budget_extension, external_auth, and reconciliation requests carry an exact tenant/session/turn/sequence binding, response JSON Schema hash, actor/role policy, expiry, and resume state. A valid response resolves once; its redacted form and hash are auditable across restart.

A parent definition may declare named specialists with exact child definition refs, tool allowlists, per-child budgets, output schemas, and depth/fanout/concurrency ceilings. The runtime exposes a generated delegate_<name> tool; the model supplies only bounded {id,prompt} tasks. Server code derives child identity, intersects tools, inherits deadlines and the persisted parent's execution host, enforces the parent's remaining aggregate budget, admits children through the same journal/runtime, validates outputs, and joins results in stable task order. Ductor does not expose an arbitrary recursive agent-spawn path.

Structured output and multi-tool turns

A provider iteration may emit content, multiple tool calls, usage, and a terminal reason. Tool-call fragments are assembled before validation. Each complete call is authorized independently and committed in deterministic order. When a definition declares a JSON output schema, final output must validate before completion; malformed output fails the turn instead of being stored as a successful result. The schema is also sent to compatible providers as a strict structured-output request, including OpenAI-compatible endpoints and Gemini, but local validation remains authoritative.

Recovery boundary

The committed sequence is the recovery boundary. A subscriber can always replay from the journal without re-calling a provider. An execution worker must acquire the session lease before advancing state, and duplicate attempts observe the already committed transition.

Each chat replica performs a bounded recovery scan at startup and every five seconds. The scan selects only chat-owned non-terminal sessions with no live lease, excluding input and reconciliation waits. Host filtering happens before ordering and the result limit; workflow or bidder sessions cannot consume the scan's slots. Every candidate still has to win the normal atomic claim. Generic recovery is refused for workflow and bidder runtimes before a scan occurs. Workflow agents resume through the DAG's original run/step context; bidder recovery uses its separately authorized durable job path and original pricing deadline.

Runtime execution checks ownership before returning terminal content, claiming a lease, and renewing it. Steering, cancellation, input responses, and compatibility upgrades also require the correct host, alongside the existing actor authorization. Read-only tokens cannot delegate those mutations across hosts. Safe persisted states resume. A recovered model_running or tool_running state is an ambiguous external outcome and parks as needs_reconciliation instead of repeating possible work.

Tool authorization is itself a recovery artifact. The live authenticated context is checked before the authorization receipt is committed. The receipt binds the tenant, session, invocation, manifest, schema, argument hash, effective arguments, execution mode, limits, and redaction policy. A saved receipt preserves the authorization decision; it does not replace current credentials or grant a different caller permission to execute. Treat a credential or policy refusal as a refusal, and reconcile an uncertain external outcome before resuming work.

Historical sessions without proven ownership are stored as unassigned. They remain readable under the existing audit and membership permissions, but cannot execute, claim, renew, admit another turn, or enter recovery. Ownership is never inferred from a session name or assigned to chat by default. Snapshots bind the stored host; an older snapshot without that binding is quarantined and reads fall back to the canonical journal rather than inventing ownership.

Actor-authorized audit administration can still manage membership, legal holds, and terminal close/reopen across hosts. Reopening preserves terminal state and cannot make unassigned work executable. Retention maintenance remains host-neutral: it only removes eligible closed terminal sessions without a lease or legal hold, and never claims or resumes work.

Subscriptions also preflight cursor continuity. A future cursor is a conflict. If an operator's external archive policy has removed a requested prefix, chat returns HTTP 410 resync_required with the requested, earliest, and latest sequences. The default schema does not prune agent events.

Production limits

Set the narrowest useful model, tools, request/tool bounds, and tenant budget. Treat an exceeded bound as a typed operational result, not permission to retry the whole turn. Keep approval required for mutations and destructive actions. Keep read-only diagnostics in the default safe policy, with output redaction and byte limits.

Apply the database migrations included with your Ductor release before enabling the runtime.

Tracing, feedback, and evaluation

Content capture is off by default. The trace hierarchy restores a persisted trace id across worker restart and emits only durable identifiers, definition/manifest/policy ids, provider/model/tool names, approval class, usage, outcome, and stable error type. Authenticated feedback binds to tenant, turn, step, and trace; only the note hash is retained.

The evaluation plane stores versioned datasets, trajectory assertions, scores, and experiments. Promotion policies can require schema, security, recovery, tool-trajectory, and isolation checks. Model-judge scores can supplement those checks but do not override them.

Related