Durable Runs & Workflows

Workflow management has two distinct halves, and it's worth separating them up front:

• Definitions — the design of a workflow: its graph of steps and edges. You edit a definition as a draft, then publish it as an immutable, versioned artifact.
• Runs — a single execution of a published definition. You trigger a run and then operate it: pause, resume, cancel, terminate, redrive, signal, or continue-as-new.

The Coordinator is the sole writer of run state, so every run operation here is an intent the coordinator applies on its next tick — never a direct state mutation. That's what makes the run control surface safe to call concurrently.

Part 1 — Workflow definitions

The lifecycle: draft → publish → archive

A definition family (identified by a stable family_slug) moves through three states:

The core rule: you edit drafts, you run published versions. Publishing validates the draft and writes a brand-new immutable version row (version = previous max + 1 for that family); the draft is untouched and remains available for continued editing. Published versions are never mutated in place, so a run pinned to version 3 always executes exactly the graph that was version 3.

Create a draft

CreateWorkflowDefinition — POST /api/workflow-definitions (workflow_definition:write). You supply a family_slug and, optionally, the initial graph.

curl -s -X POST https://api.ductor.io/api/workflow-definitions \
  -H "Authorization: Bearer $DUCTOR_API_KEY" \
  -H "X-Tenant-ID: $DUCTOR_TENANT" \
  -H "Content-Type: application/json" \
  -d '{
    "family_slug": "lead-router",
    "title": "Lead Router",
    "description": "Enrich, score, and assign inbound leads",
    "tags": ["sales"]
  }'

The response is a WorkflowDefinitionDetail with status: "draft", version: 1, and a draft_revision you'll need for the next edit.

Edit the draft

UpdateWorkflowDraft — PATCH /api/workflow-definitions/{id}/draft (workflow_definition:write). Edits are expressed as graph operations and guarded by the draft's revision for optimistic concurrency.

curl -s -X PATCH https://api.ductor.io/api/workflow-definitions/$DEF_ID/draft \
  -H "Authorization: Bearer $DUCTOR_API_KEY" \
  -H "X-Tenant-ID: $DUCTOR_TENANT" \
  -H "Content-Type: application/json" \
  -d '{
    "expected_draft_revision": 3,
    "title": "Lead Router v2",
    "operations": [
      { "add_step": { "ref": "enrich", "type": "STEP_TYPE_ACTION", "action": "hubspot.enrich_contact" } }
    ]
  }'

• expected_draft_revision must equal the current revision. If another editor has changed the draft since you read it, the write fails (optimistic-lock conflict) and you re-read, reconcile, and retry.
• The response includes any non-fatal warnings alongside the updated detail.

A step (StepDefinitionProto) carries an id/ref, a type, and — for action steps — an action plus args. Step types include ACTION, WAIT, APPROVAL, SCATTER, GATHER, SUBWORKFLOW, RACE, and more; edges (EdgeDefinitionProto) connect step refs and carry a type (SUCCESS, ERROR, TRUE/FALSE, APPROVED/REJECTED, …) and an optional condition. Two event-oriented step types are worth calling out:

• STEP_TYPE_EVENT_SET — parks a run until a correlated set of external events arrives (not just one). It waits on a correlation_key derived from the trigger and requirement expressions, then resumes once the set is satisfied.
• STEP_TYPE_DIGEST — collapses N upstream events that share a digest_key into a single downstream arrival, with optional per-arrival dedup. Runs merged into a digest surface the terminal merged run status.

The full graph vocabulary is described in The DAG Workflow Model, and the event model behind event_set/digest in Events.

Saga: compensation and finally

A step can declare a compensation policy (domain/workflow/compensation_policy.go) so the workflow behaves like a saga — with inverse actions that undo committed work and finalizers that always run:

Compensation targets are connector actions (the first-class executable kind); a failed compensation is recorded per CompensationFailurePolicy (manual_repair / fail_compensation / ignore_with_audit) and never mutates the already-terminal parent run.

Scheduled workflows

A definition can carry a cron schedule so runs fire on a timetable rather than an external trigger. The schedule is a ScheduleSpec (domain/workflow/schedule):

OverlapPolicy in v1 supports exactly two values: skip (drop the new occurrence while a run is active) and allow_all (fire regardless). Anything else is rejected (ErrInvalidOverlapPolicy).

Publish

PublishWorkflowDefinition — POST /api/workflow-definitions/{id}/publish (workflow_definition:write). No body beyond the path id is needed.

curl -s -X POST https://api.ductor.io/api/workflow-definitions/$DEF_ID/publish \
  -H "Authorization: Bearer $DUCTOR_API_KEY" \
  -H "X-Tenant-ID: $DUCTOR_TENANT"

Publish runs strict validation — graph well-formedness, routing/reply/event CEL checks, entitlement gates — before it writes the version. If validation fails, you get a structured validation error and nothing is published. On success the response carries both the newly published version and a fresh draft to keep editing.

Other definition operations

ListWorkflowDefinitions filters by status, free-text q, and tags. DuplicateWorkflowDefinition clones the latest draft (or the published version when no draft exists) into a fresh draft under a new family_slug — the way to fork a workflow. Deleting a draft never affects published versions.

Part 2 — Triggering a run

Start a run with WorkflowTriggerService.TriggerWorkflow — POST /api/v2/workflow-definitions/{definition_id}:trigger (workflow:write). The definition_id must be a published definition.

curl -s -X POST "https://api.ductor.io/api/v2/workflow-definitions/$DEF_ID:trigger" \
  -H "Authorization: Bearer $DUCTOR_API_KEY" \
  -H "X-Tenant-ID: $DUCTOR_TENANT" \
  -H "Content-Type: application/json" \
  -d '{
    "subject_type": "lead",
    "subject_id": "lead-123",
    "input": { "email": "[email protected]", "source": "webinar" },
    "wait_for_checkpoint": false,
    "idempotency_key": "lead-123-signup",
    "external_deployment_id": "prod-us-east-1-wave-1",
    "correlation_labels": [
      { "key": "conversation", "id": "conversation-42" },
      { "key": "support_ticket", "id": "ticket-9182" }
    ]
  }'

{
  "run_id": "d1e2f3a4-...",
  "status": "running",
  "metadata": { "trace_id": "trace-...", "request_id": "req-...", "fetched_at": "2026-07-11T11:10:00Z" }
}

• input is the run's initial payload (arbitrary JSON).
• idempotency_key dedupes on (tenant, definition_id, idempotency_key) — retry the same trigger and you get the same run back, not a duplicate.
• external_deployment_id pins public admission to the server-declared live caller release when the production policy requires it. It is read only from the request body, never a generic header. See Workflow release pinning.
• correlation_labels attaches stable business identities to this run and every descendant run. Use them to find all work for one conversation, ticket, import, order, or other domain journey without overloading the input payload.
• wait_for_checkpoint: true (with checkpoint_timeout_seconds) blocks the call until the workflow reaches its response checkpoint and returns response_group_output inline — useful for synchronous request/response workflows.

Correlate a run tree

System lineage (root_run_id and journey identity) tells Ductor how runs are related internally. correlation_labels adds the bounded identities your application already understands. Labels are indexed in workflow visibility and propagate automatically through subworkflows and continue-as-new successors. Set them once on the root trigger instead of reattaching them at every step.

Ductor stores the canonical list under the reserved visibility attribute ductor.correlation_labels and indexes each value as ductor.correlation.<key>. Callers and workflow effects cannot write either namespace directly; send correlation_labels on the trigger so the runtime can validate and propagate them without allowing a step to spoof lineage.

Good keys describe a stable domain type (conversation, support_ticket, import, order). Keep the id opaque and non-sensitive. Do not use labels for secrets, prompt content, or arbitrary high-cardinality telemetry.

Part 3 — Operating a run

Run control is WorkflowRunControlService under /api/v2/workflow-runs. Every control RPC dispatches a coordinator wakeup carrying the intent; the coordinator applies the state change and writes the audit entry on its next tick. Handlers never touch eec_workflow_run directly.

The core controls

All require workflow_run:write and share a request shape: request_id (a caller-supplied idempotency token) and reason (an audit note).

curl -s -X POST "https://api.ductor.io/api/v2/workflow-runs/$RUN_ID:pause" \
  -H "Authorization: Bearer $DUCTOR_API_KEY" \
  -H "X-Tenant-ID: $DUCTOR_TENANT" \
  -H "Content-Type: application/json" \
  -d '{ "request_id": "op-8f2a-1", "reason": "manual hold pending review" }'

{ "status": "accepted", "run_id": "d1e2f3a4-...",
  "metadata": { "trace_id": "trace-...", "request_id": "req-...", "fetched_at": "2026-07-11T11:12:00Z" } }

A status of "accepted" means the intent was durably enqueued for the coordinator — including when a duplicate request_id is replayed idempotently. It does not mean the state has already changed; poll the run (below) to observe the applied transition.

continueAsNew additionally accepts seed_run_context (initial context for the successor) and discard_pending, and its response carries the new_run_id.

Node-level control

The same service exposes finer operator levers on individual steps of a run: PauseNode / ResumeNode / ResetNode (/api/v2/workflow-runs/{run_id}/nodes/{step_ref}:…), plus per-node retry-policy and timeout updates. GetRunSnapshot (GET /api/v2/workflow-runs/{run_id}/snapshot) and ListRunAttempts (/attempts) give you the debug view — every node's state and every recorded attempt.

Reset / replay

ResetRun (POST /api/v2/workflow-runs/{source_run_id}:reset) forks a run from a chosen transition sequence into a new run, optionally overriding step inputs and preserving selected step outputs — the tool for surgical replay after a bad deploy or data fix. ListResetTargets and PreviewReset let you see valid reset points and dry-run the outcome first.

Part 4 — Signals

Some steps park waiting for an external decision or event. Signals resume them. All signal RPCs are on WorkflowSignalService, take a request_id for idempotency, and return status: "accepted". Duplicate signals (same request_id, or an already-resolved step) are no-ops.

Approve/reject/resolve require workflow_run:write; publish-event requires workflow_events:publish.

# Approve a human-approval step
curl -s -X POST "https://api.ductor.io/api/v2/workflow-runs/$RUN_ID/steps/manager-approval:approve" \
  -H "Authorization: Bearer $DUCTOR_API_KEY" \
  -H "X-Tenant-ID: $DUCTOR_TENANT" \
  -H "Content-Type: application/json" \
  -d '{ "request_id": "sig-1", "approver_id": "[email protected]", "comment": "cleared", "payload": { "tier": "gold" } }'

Publishing an event is deliberately decoupled from run and step IDs — you publish by event_name + correlation_key, and the engine resolves which waiting runs match. event_id is the producer's idempotency token.

curl -s -X POST https://api.ductor.io/api/v2/workflow-events:publish \
  -H "Authorization: Bearer $DUCTOR_API_KEY" \
  -H "X-Tenant-ID: $DUCTOR_TENANT" \
  -H "Content-Type: application/json" \
  -d '{ "event_name": "order.paid", "correlation_key": "order-42", "event_id": "evt-abc", "payload": { "amount": 12900 } }'

The response reports matched — how many waiting pauses this event resolved. A duplicate event or one with no waiters returns matched: 0 (a safe no-op).

Part 5 — Inspecting runs

Run reads live on WorkflowService under /api/workflows (workflow:read). These are the canonical observability endpoints:

curl -s "https://api.ductor.io/api/workflows/runs?workflow_type=lead_router&status=running&active_only=true" \
  -H "Authorization: Bearer $DUCTOR_API_KEY" -H "X-Tenant-ID: $DUCTOR_TENANT"

Run status values

Where run state lives

Run state is a single row in eec_workflow_run — status, current step, node_states (per-step snapshots as JSONB), run_context, lineage (root_run_id, parent_run_id, nesting_depth), and the optimistic-lock counter db_record_version. Every worker attempt is a row in eec_workflow_step_attempt (input, output, error, heartbeat), which is what ListRunAttempts and the events endpoint surface.

The coordinator advances a run by reading pending attempts, computing the next tick, and committing with db_record_version = db_record_version + 1 guarded by the version it read — a mismatch means another tick won the race and the coordinator retries. This is the mechanism behind Ductor's exactly-once progress guarantee; as an operator you observe it as monotonic, never-lost run progress.

Where to go next