Production Guarantees

Things to get right before Ductor carries production traffic. Most of the security items below are exactly what the Helm production gate enforces — security_profile=enterprise refuses to start unless they are set. Setting them by hand (Docker/Dokploy) gets you the same posture without the chart. For the full enterprise walkthrough, see Enterprise activation.

Pin your version

Deploy an immutable release, never a moving tag. Reference a specific version — the container image tag or the git release tag you build from — so every environment runs a known build and every upgrade is a deliberate, reviewable change. Pulling latest (or redeploying master) leaves you unable to say what's actually running or reproduce it later. Release tags are listed on the releases page, and a running instance reports its stamped version with ductor version.

For public workflow triggers, also enable caller release pinning so the request body names the exact live deployment that issued the work. Container pinning tells operators what is deployed; caller and worker pins prevent version-skewed code from executing an already admitted run.

Run migrations explicitly

In production, keep database.auto_migrate off (its default) and run migrations as a deliberate step in your deploy pipeline:

ductor migrate up

The Helm chart does this for you via a pre-install/pre-upgrade Job. Migrations are backward-compatible by convention (no breaking column removes in the same release), so a new schema is safe for the currently running version during a rollout. Full detail: Database migrations.

Lock down authentication

• Set auth.enabled=true and configure either OIDC or DB-backed API keys. The API server only builds identities from OIDC / DB-backed API keys / SAML, so a locked deployment needs one of those wired.
• Never ship auth.allow_anonymous=true in production.
• Restrict api.allowed_hosts to the hostnames you actually serve — it is a DNS-rebinding guard with exact-match semantics.
• Set authz.allow_when_unconfigured=false so a missing authorizer fails closed (Connect/gRPC returns PermissionDenied, REST returns 503). The default is still true this release, and the enterprise profile requires it false.

See Security & auth.

Protect the connector master key

Connector credentials are AEAD-encrypted with a base64 32-byte master key. Never bake it into an image or commit it. You have two sources:

• connector.encryption_key — the raw key, injected from a secret manager.
• key_provider (BYOK/KEK) — set key_provider.enabled=true with a uri. Enterprise deployments must use awskms://, gcpkms://, or vault://; local:// is an in-process passthrough for dev only.

Plan rotation with connector.encryption_key_id plus the decrypt-only connector.rotation_keys keyring: add the new key, keep the previous key decrypt-only until all ciphertext has re-encrypted, then retire it. A key rollback alone cannot decrypt data written under a newer key — the additive keyring is what makes rotation safe. See Key management.

Harden the runtime (enterprise posture)

security_profile=enterprise will not start unless these are set explicitly. Configure them directly on any deployment that carries real traffic:

Non-zero workflow-runtime safety limits

The workflow_runtime.* bounds default to sensible non-zero values but can be set to 0 for local compatibility. In production keep them non-zero — the enterprise profile requires it — so no single run can monopolize a worker or grow unbounded:

• coordinator_tick_timeout (default 30s) — bounds one coordinator tick.
• continue_as_new_after_history / continue_as_new_after_wakeups — cap workflow history and wakeup churn.
• can_transition_soft_limit / can_transition_hard_limit / can_terminate_limit — transition-count safety net.
• max_step_output_size (default 100 MiB) and max_run_state_size (default 256 MiB) — hard caps on persisted state.
• span_id_seed_key — enterprise requires a deployment-specific value of at least 16 bytes.
• In distributed coordinator deployments, enable workflow_runtime.sharding.range_id_fencing so every live tick asserts shard ownership (the enterprise profile requires it whenever the runtime is enabled).

Size resources

The Helm defaults are a sensible starting point:

Memory scales with in-flight run state — the runtime caps a single run's state at workflow_runtime.max_run_state_size (256 MiB) and step output at max_step_output_size (100 MiB), so raise limits if you run large fan-outs. Tune the Postgres pool (database.pool.max_conns, default 25) to your database's connection budget across all replicas.

Scale the coordinator/worker model

Ductor's DAG runtime is a coordinator-plus-worker system, and it scales horizontally without special care because the invariants are structural, not lock-based:

• Run-state safety is per-run, not per-process. Only the Coordinator mutates a run's state row, guarded by optimistic locking on record_version. Adding replicas never introduces a write race — a losing commit is simply re-queued.
• Workers are stateless and parallel. They only append attempts, so you scale throughput by adding replicas/workers (router.queue.workers, router.queue.concurrency, router.queue.tenant_concurrency).
• Split roles if you need to. server.role (all/api/worker/executor) lets you run dedicated API and worker fleets from the same image.

Autoscaling (Helm autoscaling.*) drives replicas on CPU/memory; keep podDisruptionBudget.minAvailable ≥ 2 so rollouts never drain the coordinator pool entirely.

Back up state

PostgreSQL is the source of truth — back it up on a schedule and test restores. Redis/Dragonfly holds cache, flow counters, and queue state that is rebuildable but whose loss causes disruption. See Backup & restore.

Turn on observability before you need it

Enable tracing (tracing.enabled=true) and scrape metrics from the loopback metrics listener from day one, and alert on the readiness and coordinator-health signals. See Observability and Health checks.

Pre-flight checklist

• Deploying a pinned release tag (image/git), not latest or master
• Public workflow callers send their required live external_deployment_id
• ductor migrate up run; schema current before pods serve
• auth.enabled=true, OIDC or API keys configured, anonymous off
• authz.allow_when_unconfigured=false
• api.allowed_hosts restricted to real hostnames
• Connector master key from a secret manager / KMS / Vault
• egress.mode=enforce, server.exec_isolation_required=true, aggregator.backend=redis
• workflow_runtime.* safety limits non-zero
• Metrics bound to loopback and scraped via a secured proxy; tracing on
• Resource requests/limits and Postgres pool sized for replica count
• Postgres backups scheduled and a restore rehearsed