Define & publish a workflow

This guide is the executed stage of the clearing lifecycle, hands-on: a workflow definition is a DAG of steps that Ductor's coordinator executes durably once Work is routed to a Worker. Definitions can be authored as YAML or through the tenant API. Publishing freezes an immutable version that the runtime pins runs to. This guide uses an ordinary tenant API key to create, inspect, publish, trigger, and reuse a small workflow. The optional YAML check runs locally before any server mutation.

1. Author a connector-free definition

This example projects a literal greeting and then waits two seconds. It needs no connector credentials or paid provider. Save this optional YAML form as hello_world.yaml if you want to run the offline validator:

type: workflow_definition
definition:
  environment: default
  family_slug: hello_world
  title: Hello World
  steps:
    - ref: greet
      type: dataflow
      title: Build greeting
      args:
        operation: project
        input_expr: TRIGGER
        redaction_policy: none
        project:
          message: '"hello"'
    - ref: pause
      type: wait
      title: Short delay
      args:
        duration: 2s
  edges:
    - source_ref: greet
      target_ref: pause
      type: success

input_expr selects the run input. redaction_policy: none exposes this synthetic greeting for inspection; use the default hashed samples for sensitive data.

The YAML envelope contains a definition body. Optional triggers and checkpoints can be added for event entry points and named progress markers. The manual trigger endpoint below starts this example directly.

Key fields:

The 15 step types

action is one of 15 authorable step types. The rest cover control flow, fan-out, human-in-the-loop, AI, and cross-runtime calls:

See the DAG workflow model for the full semantics of each. The ai_action / ai_agent steps are covered in AI; bridge steps in SDKs.

Add a human approval gate

A native approval step requires "type": "STEP_TYPE_APPROVAL" and the top-level "action": "approval". Put its authorization audience and deadline in args; args.action is invalid. This separate example uses the same create, inspect, and publish API lifecycle below. Choose a unique family slug, then inspect its review and done nodes and send an approval decision as described below; the greeting-specific output checks apply only to the greeting example:

{
  "environment": "default",
  "family_slug": "review_before_work",
  "title": "Review before work",
  "steps": [
    {"ref": "review", "type": "STEP_TYPE_APPROVAL", "action": "approval",
     "args": {"audience": {"match_mode": "any", "roles": ["operator"]},
              "timeout_ms": 600000, "on_timeout": "fail"}},
    {"ref": "done", "type": "STEP_TYPE_DATAFLOW",
     "args": {"operation": "project", "input_expr": "TRIGGER",
              "redaction_policy": "none", "project": {"reviewed": "true"}}}
  ],
  "edges": [{"source_ref": "review", "target_ref": "done", "type": "EDGE_TYPE_APPROVED"}]
}

In YAML, use type: approval, action: approval, and edge types approved, rejected, or done. The API equivalents are EDGE_TYPE_APPROVED, EDGE_TYPE_REJECTED, and EDGE_TYPE_DONE; ordinary success and error edges are invalid for this step type.

audience is required. approver_role is only a display/audit label and grants no authority. A reviewer needs current workflow_run:write permission and must match the frozen audience. Inspect the run snapshot for review.status: waiting and a pending done node, then use the supported approve or reject operation. A refused decision must leave the gate waiting and downstream work unstarted.

timeout_ms accepts integers from 0 through 9223372036854. Zero, or omission, waits indefinitely; set an explicit deadline when unattended waiting is not acceptable. With a positive timeout, on_timeout accepts fail (the default), approve, or skip. Use approve only when automatically allowing work after the deadline is intentional. on_timeout without a positive timeout and a nonempty expires_at_cel are rejected. Publication also refuses a missing or different action with APPROVAL_ACTION_INVALID, or invalid typed arguments with APPROVAL_SPEC_INVALID, before creating a published version. Repair the editable draft, review it again, and publish a new version. HTTP refusals return 400 with top-level error_code: "INVALID_ARGUMENT". Each errors[] item includes the step reference in field, the validation code, and a corrective description.

Event triggers

An event trigger fires runs when a matching event arrives. Each trigger carries one or more patterns, and every pattern is either a literal event name (stripe.invoice.paid) or a suffix wildcard (stripe.invoice.*).

Steps that call connectors

Action steps can call a third-party system through a connector. Reference a connection and pass templated arguments (${{ ... }} expressions resolve at dispatch time against trigger payload, variables, and prior step outputs):

steps:
  - ref: create_invoice
    type: action
    action: stripe.invoices.create
    connection_ref: "${{ VARS.stripe_connection_ref }}"
    connection_ref_mode: direct
    args:
      customer_id: "${{ TRIGGER.payload.customer_id }}"
      amount: "${{ TRIGGER.payload.amount_cents }}"
      currency: "${{ TRIGGER.payload.currency }}"
    retry_max_attempts: 3
    retry_backoff_ms: 1000
    timeout_ms: 10000
    on_error: fail

Steps with external side effects can declare compensation — inverse actions that run on_failure or on_cancel — so a partially-completed run unwinds cleanly.

2. Validate YAML offline (optional)

With the Ductor binary installed, lint the saved YAML. workflow validate parses and validates it without touching the database. It exits nonzero when the definition is invalid or a reference cannot be proven by the offline catalog:

ductor workflow validate ./hello_world.yaml

The offline catalog checks linked action, strategy, and hook references. It checks graph structure and expressions without a running Ductor or Postgres. Deployment-dependent capabilities can remain unresolved; a successful offline check does not establish live credentials, permissions, budgets, or connector readiness. Publication performs the server-side validation. This command does not upload the YAML or create a tenant definition.

Definition size limits

Structural validation caps how large a single definition can be. These bounds are checked first, before any per-step graph analysis, and a definition over either limit fails validation with a single coded violation:

The edge cap keeps the per-step graph scans (predecessor lookup, dataflow, cycle detection) clear of the O(V·E) blow-up a dense graph would cause. At the 500-step ceiling it still allows an average fan-out of ten edges per step — well above real workflows, which typically run one to four. Real DAGs never approach either limit; hitting one almost always means a generated definition has looped.

3. Create and inspect a tenant draft

Use a running instance with the workflow runtime enabled, Bash, curl 7.76 or later, and jq. Have an administrator issue an ordinary tenant key with an appropriate role and workflow_definition:read, workflow_definition:write, workflow:read, workflow:write, and workflow_run:read scopes. Set DUCTOR_API_KEY to that key and DUCTOR_TENANT_ID to its tenant UUID. Administrative setup is covered in Getting Started and Issue & use API keys.

Run the remaining commands in one Bash session. The helper fails on HTTP errors; pipefail and checked identifiers prevent errors from turning into null ids:

set -euo pipefail
: "${DUCTOR_API_KEY:?Set your ordinary tenant API key}"
: "${DUCTOR_TENANT_ID:?Set your tenant UUID}"
BASE=${DUCTOR_API_URL:-http://localhost:8080}
ductor_api() {
  curl --fail-with-body --silent --show-error --max-time 30 \
    -H "Authorization: Bearer $DUCTOR_API_KEY" \
    -H "X-Tenant-ID: $DUCTOR_TENANT_ID" "$@"
}

The API accepts a JSON creation request, not the YAML envelope. Save the same two-step graph in its canonical request form:

environment is required and names an active, mutable environment that allows direct mutation, does not require promotion, and your key can write. Use its canonical key (default here), rather than its UUID. Changing a draft's environment requires current write authority in both environments; publication checks the saved binding again.

cat > hello_world.json <<'JSON'
{
  "environment": "default",
  "family_slug": "hello_world",
  "title": "Hello World",
  "steps": [
    {"ref": "greet", "type": "STEP_TYPE_DATAFLOW", "title": "Build greeting",
     "args": {"operation": "project", "input_expr": "TRIGGER",
              "redaction_policy": "none", "project": {"message": "\"hello\""}}},
    {"ref": "pause", "type": "STEP_TYPE_WAIT", "title": "Short delay",
     "args": {"duration": "2s"}}
  ],
  "edges": [{"source_ref": "greet", "target_ref": "pause", "type": "EDGE_TYPE_SUCCESS"}]
}
JSON

CREATED=$(ductor_api -X POST "$BASE/api/workflow-definitions" \
  -H "Content-Type: application/json" --data-binary @hello_world.json)
DRAFT_ID=$(echo "$CREATED" | jq -er '.data.id | strings | select(length > 0)')
ductor_api "$BASE/api/workflow-definitions/$DRAFT_ID" > hello_world.draft.json
jq -e '.data | select(.status == "DEFINITION_STATUS_DRAFT")' hello_world.draft.json
DRAFT_REVISION=$(jq -r '.data.draft_revision // 0' hello_world.draft.json)

If a draft already exists for hello_world, creation returns a conflict. Inspect that family through GET /api/workflow-definitions or choose a new family_slug; do not treat a failed creation as an empty workflow. Review the saved draft's steps and edges before publication.

4. Publish the reviewed revision

A freshly created or edited definition is a draft. Publication creates a new immutable row and returns its id at published.id; the draft id remains editable. Publish the exact revision you reviewed and capture the published id:

PUBLISHED=$(ductor_api -X POST "$BASE/api/workflow-definitions/$DRAFT_ID/publish" \
  -H "Content-Type: application/json" \
  -d "$(jq -nc --argjson revision "$DRAFT_REVISION" '{expected_draft_revision: $revision}')")
PUBLISHED_ID=$(echo "$PUBLISHED" | jq -er '.published.id | strings | select(length > 0)')
ductor_api "$BASE/api/workflow-definitions/$PUBLISHED_ID" > hello_world.published.json
jq -e '.data | select(.status == "DEFINITION_STATUS_PUBLISHED")' hello_world.published.json

If the draft changed after review, publication refuses the stale revision. Read and review the current draft before trying again. Other publication refusals require resolving their validation or authority error; no executable version should be assumed from an error response.

Re-publishing later creates another version under the same family_slug. In-flight runs keep the immutable version they started with.

5. Trigger and inspect the outcome

Use PUBLISHED_ID, not DRAFT_ID, to start a run:

TRIGGERED=$(ductor_api -X POST "$BASE/api/v2/workflow-definitions/$PUBLISHED_ID:trigger" \
  -H "Content-Type: application/json" \
  -d '{"subject_type": "demo", "subject_id": "1"}')
RUN_ID=$(echo "$TRIGGERED" | jq -er '.run_id | strings | select(length > 0)')
echo "run: $RUN_ID"

Read the status; repeat this read until it reaches a terminal state. The expected successful state is completed. A nonterminal response is not completion, and an HTTP error is a failed observation:

ductor_api "$BASE/api/workflows/runs/$RUN_ID" > hello_world.run.json
jq -e '.data.summary.status | strings' hello_world.run.json

Inspect the visible step evidence after completion:

ductor_api "$BASE/api/workflows/runs/$RUN_ID/steps" > hello_world.steps.json
jq . hello_world.steps.json

The attempt list need not contain in-coordinator dataflow steps. Read the run snapshot for their authoritative node status and output:

ductor_api "$BASE/api/v2/workflow-runs/$RUN_ID/snapshot" > hello_world.snapshot.json
jq -e '.nodes.greet.latest_output_json | fromjson | .result | select(.message == "hello")' \
  hello_world.snapshot.json

The output should be {"message":"hello"}. A successful run status alone does not establish that result. For a failed run, retain its status, snapshot, and step errors, resolve the cause, and use the supported workflow runtime controls for recovery.

For process recovery with intact persistence, follow the crash-resume example. A process restart is separate from restoring a database backup.

6. Reuse the workflow

Keep hello_world.json as the editable creation payload, and hello_world.published.json as the API snapshot of the exact published version. The snapshot includes server-owned fields and is not itself a creation request. Another authorized person can adapt the creation payload without source changes:

jq '.family_slug = "hello_world_copy" | .title = "Hello World copy"' \
  hello_world.json > hello_world_copy.json

They configure their own tenant credentials, create the adapted request, review its returned draft, then publish and trigger its published.id using the steps above. Share the payload and expected greeting, not an API key. Review connector, environment, spending, and approval requirements again when adapting it to real work; an earlier definition does not grant current execution authority.

Administrative YAML operations are separate

workflows.dir is a server bootstrap setting for global system definitions. ductor workflow export reads the configured database directly in that global namespace. Neither is the authenticated tenant creation or retrieval route used here. Use the tenant API to inspect and reuse this workflow; do not expect a global filesystem scan or export command to locate its tenant-owned version.

How this maps to the runtime

Once published and triggered, a run is owned by the serial Coordinator, which loads run state, applies pending attempt results, computes the next tick, persists atomically with optimistic locking, and then dispatches newly-runnable steps to parallel Step Workers. Your edges, checkpoints, and compensation blocks are exactly what the Coordinator evaluates on each tick.

Next steps