# Durable Work Data Streams (/docs/operations/workflow-data-streams)



<StoryFrame label="Stream invariant" title="Commit the record first. Treat every live signal as a reason to reread." accent="worker">
  <StoryFact label="Order">
    Strict ascending sequence with exclusive resume cursor
  </StoryFact>

  <StoryFact label="Pressure">
    Durable item and byte credit under a Postgres row lock
  </StoryFact>

  <StoryFact label="Placement">
    Immutable retention, redaction, residency, and egress pins
  </StoryFact>
</StoryFrame>

Workflow data streams carry progressive input or output payloads independently of
run-state subscriptions.

## Exact cursor rule [#exact-cursor-rule]

Every read returns records with `sequence > cursor` in ascending order. A live
subscription is established before backfill; boundary duplicates are dropped and a
live gap is repaired from Postgres. Notification is only a wake signal and never
precedes durable commit.

If retention removed `cursor + 1`, the API returns HTTP 410 with requested,
earliest, and latest sequences plus a resync reason. A future cursor is invalid input,
not an implicit reset.

<ProcessRail label="Continuity loop" title="Durability repairs reconnects and live-delivery gaps">
  <ProcessStep label="01" title="Subscribe">
    Establish the live wake channel before reading retained history.
  </ProcessStep>

  <ProcessStep label="02" title="Backfill" tone="gold">
    Return durable records with sequence strictly greater than the cursor.
  </ProcessStep>

  <ProcessStep label="03" title="Follow">
    Drop boundary duplicates and treat notifications only as wake signals.
  </ProcessStep>

  <ProcessStep label="04" title="Repair" tone="commit">
    Read any detected gap from Postgres before continuing live delivery.
  </ProcessStep>
</ProcessRail>

## HTTP surface [#http-surface]

The authenticated API is mounted below `/api/workflow-streams`:

\| Operation           | Relative path                                                      |
\| ------------------- | ------------------------------------------------------------------ | --------------------------------------------- |
\| Create definition   | `POST /runs/{runID}/streams`                                       |
\| Read definition     | `GET /runs/{runID}/streams/{streamID}`                             |
\| Append/read records | `POST                                                              | GET /runs/{runID}/streams/{streamID}/records` |
\| Subscribe with SSE  | `GET /runs/{runID}/streams/{streamID}/subscribe`                   |
\| Acknowledge         | `POST /runs/{runID}/streams/{streamID}/consumers/{consumerID}/ack` |
\| Close               | `POST /runs/{runID}/streams/{streamID}/close`                      |
\| Legal hold          | `PUT /runs/{runID}/streams/{streamID}/legal-hold`                  |

The `/objects` prepare, upload, commit, and abort operations implement recoverable
large-payload delivery. Inline records are capped at 64 KiB; object bodies at 32 MiB.

## Durable producer credit [#durable-producer-credit]

Credit is computed under the Postgres stream-row lock from unread durable items and
bytes after the minimum consumer acknowledgement. It never depends on process-local
subscriber counts.

* Within unread and total ceilings: append at the next sequence, then notify.
* Unread ceiling reached: block/retry or reject according to pinned policy.
* Total ceiling reached: return quota exhausted without appending.
* Closed stream: reject without appending.

Acknowledgements only advance. Regression, future sequence, or cross-scope ack fails.
Backpressure maps to HTTP 429 with `Retry-After`.

## Input and large-object recovery [#input-and-large-object-recovery]

Input records and their delivery outbox commit together. A dispatcher publishes a
deterministic event through the existing workflow wait service, so streams do not
create another resume authority.

Large objects follow `prepared → uploaded → committed`, or abort/expiry. Prepare pins
scope, idempotency key, digest, size, schema, authorization receipt, expiry, and a
server-owned key. A record is visible only after object evidence and metadata commit
agree. Duplicate commit returns the original sequence; orphan cleanup removes
uncommitted objects.

## Data handling [#data-handling]

Definitions immutably pin metadata home/replicas, object regions, client-egress
regions, log region, retention, redaction, and authorization. Compute movement cannot
relocate data. Legal hold prevents pruning and object deletion. Raw hidden model
reasoning is not an allowed stream content class.

<Callout type="warn" title="No default MCP payload tools">
  Arbitrary stream payloads may be sensitive. Ductor does not register general MCP read/append tools
  without a separate explicit tool policy and entitlement contract.
</Callout>

## Operations [#operations]

Migration 476 creates stream records, append receipts, consumers, object intents,
input delivery, deletion outboxes, and gap evidence. Workers persist claims and retry
state in Postgres, so restart or replica takeover does not lose delivery.