Durable Execution Workers

The single-writer execution model that keeps DAG run state correct under concurrency.

Ductor's runtime follows a coordinator-plus-worker model. It exists to answer one hard question — how do you run a workflow across many machines without two of them corrupting the same run state? — with a structural answer rather than careful locking everywhere.

This page is the working mental model — enough to reason about how a run makes progress. The internals it deliberately glosses over — the exact locking protocol, the table schema, and the crash-and-recovery windows — live in The Coordinator-Worker Model, in depth.

Two roles

The Coordinator (serial, single writer)

The Coordinator is the only component allowed to mutate a run's state row (eec_workflow_run). It runs one tick per wakeup, and a tick is the atomic unit of progress for a run.

Ductor's durable runtime tables share an eec_ prefix — short for Enterprise Eventing Core, its event, task, and workflow store (see the glossary).

Step Workers (parallel, many)

Step Workers execute the actual units of work — invoking connectors, evaluating rules, running your step logic — in parallel across the fleet. When a worker finishes, it writes its outcome as a row in eec_workflow_step_attempt. Workers never touch run state; they only append attempts that the Coordinator later reads and folds in.

Why split it this way?

Writers to shared state are where workflow engines quietly lose data. By making the Coordinator the sole writer and workers append-only, progress is always either committed by the Coordinator or visible as a recorded attempt — never silently dropped.

Anatomy of a tick

A tick is deliberately structured as load → compute (pure) → commit → dispatch. Keeping the middle pure — a function of loaded state with no side effects — is what makes retries and replays safe.

Load state. Read the mutable run state (status, per-node states, run context, record_version) from the store. If the run is already terminal, the tick is a no-op and returns immediately.

Drain one control. If an operator intent is pending (pause, resume, cancel, terminate, redrive, continue-as-new), handle it first — control is a first-class, coordinator-owned transition, not a side channel.

Load tick inputs. Gather everything the compute step needs: new attempt results from workers, pending signals, fired timers, child-run snapshots, credentials, and heartbeats.

Compute the payload (pure). Over a clone of the state — never the live row — run the tick phases in order:

Advance nodes — move ready nodes from pending to running, and apply the attempt results workers have reported.
Drive fan-out — fire scatter timers, and finalize scatter parents once their branches complete.
Absorb external input — drain signals and controls, and resolve expired timers and child waits.
Apply and schedule — apply effect patches, schedule newly runnable nodes, and detect completion or checkpoints.

The output is a single TickPayload stamped with ExpectedVersion = record_version — no durable state has been mutated yet.

Commit atomically. Persist the payload in one Postgres transaction: update the run row, insert new attempt rows, insert transitions, and mark applied signals — all conditional on the record_version still matching. If it moved, the commit is rejected and the tick is recomputed against fresh state (see Optimistic Locking).

Dispatch post-commit. Only after the commit succeeds, dispatch the newly runnable step tasks, spawn sub-workflows, fire defers, release or reacquire concurrency leases, and enqueue any coordinator self-wakeups.

Because a single writer owns the transition and every commit is version-checked, there is no read-modify-write race on run state — the invariant "only the coordinator mutates eec_workflow_run" is enforced, not hoped for.

Commit, then dispatch

Dispatch happens post-commit on purpose. If Ductor dispatched a task and then failed to persist that it did so, a crash could double-run work or lose it. By committing the intent first and dispatching afterward, a redelivery is always safe: the Coordinator re-derives what still needs doing from durable state. This is exactly the boundary that makes state progress exactly-once while side-effect dispatch is at-least-once — see Idempotency & Exactly-Once.

Attempts and stale workers

Workers write to eec_workflow_step_attempt; the Coordinator reads those rows on its next tick and folds the results into run state. Two safeguards keep this honest under retries:

Each attempt carries a stamp/token. Re-applying a result that was already applied is a no-op — it produces no new rows.
On retry, the node's attempt generation is bumped, so a stale worker from an older attempt cannot land its result on top of a newer one.

If a run is cancelled or terminated, any dangling running attempt rows are closed out — nothing is left half-open.

Beyond linear DAGs

The Coordinator detects and drives the richer patterns a real workflow needs. Each is covered in depth in The DAG Workflow Model:

ContinueAsNew — close out an oversized run and start a fresh successor, carrying forward the state that matters. Can be triggered explicitly or automatically at a transition-count threshold.
Scatter / gather — fan a step out into many parallel branches (inline or as child runs) and collect their results before proceeding.
Sub-workflows — a step that runs another workflow to a terminal state or a named checkpoint.
Signals, waits, approvals, and event sets — external events and timers that park a run and later wake it with new input.

Wakeups

Coordinators don't poll; they're driven by wakeups delivered through Redis — the zero-latency in-process SyncMatch channel or the durable tiered queue — one tick per wakeup. This keeps ticks cheap and lets many runs make progress concurrently while each individual run stays strictly serial. The exact dequeue-and-requeue wiring lives in the architecture deep-dive.

Because the tick is idempotent under record_version, wakeups are intentionally not deduplicated: a redundant wakeup produces a harmless no-op tick, and that's cheaper and safer than trying to suppress it. A missed wakeup would stall a run; a duplicate one costs almost nothing.

Where to go next

The DAG Workflow Model — the steps and patterns a tick drives.
Optimistic Locking — the concurrency primitive that makes the commit safe.
The Coordinator-Worker Model, in depth — the architectural view.

Durable Execution Workers

On this page