Data Flow

Where run state, attempt results, capacity counters, caches, and events physically live — and how they move.

Ductor's correctness properties come from where each kind of data lives and how it moves between Postgres and Redis. This page is the map: for each kind of state, which store owns it, who writes it, and how changes propagate.

The two stores, and what each owns

Ductor uses Postgres as the durable source of truth and Redis (or Dragonfly) as the fast, coordinating layer. The division is deliberate. (The durable eec_-prefixed tables below belong to Ductor's Enterprise Eventing Core — its event, task, and workflow store.)

Data	Store	Writer	Why there
Workflow run state	Postgres `eec_workflow_run`	Coordinator only	Durable, version-locked source of truth
Attempt results	Postgres `eec_workflow_step_attempt`	Step Workers	Durable, append-only record of work
Config aggregates (pools, rules)	Postgres (event-sourced)	Command side	Auditable, replayable history
Idempotency records	Postgres `eec_routing_idempotency`	Routing plane	Durable dedup across retries
Capacity / concurrency counters	Redis (Lua-atomic)	Hot path	Fast, atomic, cross-pod without a lock
Pool / rule cache	Redis	Cache layer	Low-latency reads, pub/sub invalidation
Step tasks & wakeups	Redis (tiered queue)	Dispatcher	Fair, durable-enough delivery plane

The rule of thumb: anything that must survive a crash lives in Postgres; anything that must be fast and shared across pods lives in Redis. Redis is a coordinating accelerator, not the system of record.

Workflow execution flow

Run state and attempt results both live in Postgres, so the durable record of "where is this run" and "what did each step produce" survives any pod restart. Redis carries the coordination — wakeups and task delivery — which can be redriven if lost. See The Coordinator-Worker Model.

Routing decision flow

The routing pipeline reads cached pools and rules from Redis on the enrich stage, then commits its decision to Postgres and adjusts capacity counters in Redis atomically. The idempotency reservation in Postgres is what makes a retried request reuse its decision.

Config change and cache invalidation

Config changes fan out through the append-only bus:

The event is persisted before it's published, so a dropped pub/sub message degrades cache freshness (until the next read reconciles), never correctness. Every pod subscribes and invalidates its local cache, so a rule edit propagates fleet-wide without a redeploy.

Capacity counters

Concurrency and rate ceilings are Redis counters mutated by Lua scripts, so an increment-check-decrement is a single atomic step with no read-modify-write race across pods. This is the same primitive used by entitlements on the routing hot path and by the tiered queue at lease time — one mechanism, two call sites.

Postgres decides, Redis coordinates

If you remember one thing about Ductor's data flow: Postgres is the arbiter of truth and correctness; Redis makes it fast and fair. Every durable decision is a Postgres commit. Redis carries wakeups, task delivery, caches, and counters — all of which can be rebuilt or redriven from the durable state if they're lost.

Where to go next

Storage Model — how the Postgres side is structured (sqlc, CQRS, migrations).
The Tiered Fair Queue — the Redis delivery plane.
Idempotency & Exactly-Once — how the two stores combine for delivery guarantees.