# Production Guarantees (/docs/deployment/production)



Things to get right before Ductor carries production traffic. Most of the
security items below are exactly what the Helm
[production gate](/docs/deployment/kubernetes-helm#the-production-gate) enforces —
`security_profile=enterprise` refuses to start unless they are set. Setting them
by hand (Docker/Dokploy) gets you the same posture without the chart. For the
full enterprise walkthrough, see
[Enterprise activation](/docs/operations/enterprise-activation).

## Pin your version [#pin-your-version]

Deploy an immutable release, never a moving tag. Reference a specific version —
the container image tag or the git release tag you build from — so every
environment runs a known build and every upgrade is a deliberate, reviewable
change. Pulling `latest` (or redeploying `master`) leaves you unable to say what's
actually running or reproduce it later. Release tags are listed on the
[releases page](https://github.com/ductor-io/ductor/releases), and a running
instance reports its stamped version with `ductor version`.

For public workflow triggers, also enable
[caller release pinning](/docs/deployment/release-pinning) so the request body
names the exact live deployment that issued the work. Container pinning tells
operators what is deployed; caller and worker pins prevent version-skewed code
from executing an already admitted run.

## Run migrations explicitly [#run-migrations-explicitly]

In production, keep `database.auto_migrate` **off** (its default) and run
migrations as a deliberate step in your deploy pipeline:

```bash
ductor migrate up
```

The Helm chart does this for you via a pre-install/pre-upgrade Job. Migrations
are backward-compatible by convention (no breaking column removes in the same
release), so a new schema is safe for the currently running version during a
rollout. Full detail: [Database migrations](/docs/operations/migrations).

## Lock down authentication [#lock-down-authentication]

* Set `auth.enabled=true` and configure either OIDC or DB-backed API keys. The
  API server only builds identities from OIDC / DB-backed API keys / SAML, so a
  locked deployment needs one of those wired.
* Never ship `auth.allow_anonymous=true` in production.
* Restrict `api.allowed_hosts` to the hostnames you actually serve — it is a
  DNS-rebinding guard with exact-match semantics.
* Set **`authz.allow_when_unconfigured=false`** so a missing authorizer fails
  closed (Connect/gRPC returns `PermissionDenied`, REST returns `503`). The
  default is still `true` this release, and the enterprise profile requires it
  `false`.

See [Security & auth](/docs/operations/security).

## Protect the connector master key [#protect-the-connector-master-key]

Connector credentials are AEAD-encrypted with a base64 32-byte master key. Never
bake it into an image or commit it. You have two sources:

* **`connector.encryption_key`** — the raw key, injected from a secret manager.
* **`key_provider`** (BYOK/KEK) — set `key_provider.enabled=true` with a `uri`.
  Enterprise deployments must use `awskms://`, `gcpkms://`, or `vault://`;
  `local://` is an in-process passthrough for dev only.

Plan rotation with `connector.encryption_key_id` plus the **decrypt-only**
`connector.rotation_keys` keyring: add the new key, keep the previous key
decrypt-only until all ciphertext has re-encrypted, then retire it. A key
rollback alone cannot decrypt data written under a newer key — the additive
keyring is what makes rotation safe. See
[Key management](/docs/operations/key-management).

## Harden the runtime (enterprise posture) [#harden-the-runtime-enterprise-posture]

`security_profile=enterprise` will not start unless these are set explicitly.
Configure them directly on any deployment that carries real traffic:

| Setting                                      | Production value       | Why                                                                                                                                            |
| -------------------------------------------- | ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| `authz.allow_when_unconfigured`              | `false`                | Fail closed when no authorizer is wired                                                                                                        |
| `egress.mode`                                | `enforce`              | Refuse outbound dials outside the allowlist (blocks private nets + the cloud metadata IP); `shadow` only logs                                  |
| `server.exec_isolation_required`             | `true`                 | Fail closed until a real isolated exec runner is wired — the default `in_process` runner is not an isolation boundary                          |
| `aggregator.backend`                         | `redis`                | Durable debounce/batch/singleton state across restarts and replicas; `memory` loses deferred slots on restart and can't coordinate across pods |
| `config.metricsHost` / `server.metrics_addr` | loopback (`127.0.0.1`) | Keep the metrics listener off the network; scrape via a secured in-pod proxy                                                                   |

### Non-zero workflow-runtime safety limits [#non-zero-workflow-runtime-safety-limits]

The `workflow_runtime.*` bounds default to sensible non-zero values but can be
set to `0` for local compatibility. In production keep them non-zero — the
enterprise profile requires it — so no single run can monopolize a worker or grow
unbounded:

* `coordinator_tick_timeout` (default `30s`) — bounds one coordinator tick.
* `continue_as_new_after_history` / `continue_as_new_after_wakeups` — cap
  workflow history and wakeup churn.
* `can_transition_soft_limit` / `can_transition_hard_limit` / `can_terminate_limit`
  — transition-count safety net.
* `max_step_output_size` (default 100 MiB) and `max_run_state_size` (default
  256 MiB) — hard caps on persisted state.
* `span_id_seed_key` — enterprise requires a deployment-specific value of at
  least 16 bytes.
* In distributed coordinator deployments, enable
  `workflow_runtime.sharding.range_id_fencing` so every live tick asserts shard
  ownership (the enterprise profile requires it whenever the runtime is enabled).

## Size resources [#size-resources]

The Helm defaults are a sensible starting point:

|        | Request | Limit   |
| ------ | ------- | ------- |
| Memory | `256Mi` | `1Gi`   |
| CPU    | `250m`  | `1000m` |

Memory scales with in-flight run state — the runtime caps a single run's state at
`workflow_runtime.max_run_state_size` (256 MiB) and step output at
`max_step_output_size` (100 MiB), so raise limits if you run large fan-outs. Tune
the Postgres pool (`database.pool.max_conns`, default `25`) to your database's
connection budget across all replicas.

## Scale the coordinator/worker model [#scale-the-coordinatorworker-model]

Ductor's DAG runtime is a [coordinator-plus-worker](/docs/concepts/coordinator-workers)
system, and it scales horizontally without special care because the invariants
are structural, not lock-based:

* **Run-state safety is per-run, not per-process.** Only the Coordinator mutates
  a run's state row, guarded by optimistic locking on `record_version`. Adding
  replicas never introduces a write race — a losing commit is simply re-queued.
* **Workers are stateless and parallel.** They only append attempts, so you scale
  throughput by adding replicas/workers (`router.queue.workers`,
  `router.queue.concurrency`, `router.queue.tenant_concurrency`).
* **Split roles if you need to.** `server.role` (`all`/`api`/`worker`/`executor`)
  lets you run dedicated API and worker fleets from the same image.

Autoscaling (Helm `autoscaling.*`) drives replicas on CPU/memory; keep
`podDisruptionBudget.minAvailable` ≥ 2 so rollouts never drain the coordinator
pool entirely.

## Back up state [#back-up-state]

PostgreSQL is the source of truth — back it up on a schedule and test restores.
Redis/Dragonfly holds cache, flow counters, and queue state that is rebuildable
but whose loss causes disruption. See
[Backup & restore](/docs/operations/backup-restore).

## Turn on observability before you need it [#turn-on-observability-before-you-need-it]

Enable tracing (`tracing.enabled=true`) and scrape metrics from the loopback
metrics listener from day one, and alert on the readiness and coordinator-health
signals. See [Observability](/docs/operations/observability) and
[Health checks](/docs/operations/health-checks).

## Pre-flight checklist [#pre-flight-checklist]

* [ ] Deploying a pinned release tag (image/git), not `latest` or `master`
* [ ] Public workflow callers send their required live `external_deployment_id`
* [ ] `ductor migrate up` run; schema current before pods serve
* [ ] `auth.enabled=true`, OIDC or API keys configured, anonymous off
* [ ] `authz.allow_when_unconfigured=false`
* [ ] `api.allowed_hosts` restricted to real hostnames
* [ ] Connector master key from a secret manager / KMS / Vault
* [ ] `egress.mode=enforce`, `server.exec_isolation_required=true`, `aggregator.backend=redis`
* [ ] `workflow_runtime.*` safety limits non-zero
* [ ] Metrics bound to loopback and scraped via a secured proxy; tracing on
* [ ] Resource requests/limits and Postgres pool sized for replica count
* [ ] Postgres backups scheduled and a restore rehearsed