# Data Flow (/docs/architecture/data-flow)



Ductor's correctness properties come from *where* each kind of data lives and
*how* it moves between Postgres and Redis. This page is the map: for each kind of
state, which store owns it, who writes it, and how changes propagate.

## The two stores, and what each owns [#the-two-stores-and-what-each-owns]

Ductor uses **Postgres** as the durable source of truth and **Redis (or
Dragonfly)** as the fast, coordinating layer. The division is deliberate. (The
durable `eec_`-prefixed tables below belong to Ductor's *Enterprise Eventing
Core* — its event, task, and workflow store.)

| Data                             | Store                                | Writer           | Why there                               |
| -------------------------------- | ------------------------------------ | ---------------- | --------------------------------------- |
| Workflow run state               | Postgres `eec_workflow_run`          | Coordinator only | Durable, version-locked source of truth |
| Attempt results                  | Postgres `eec_workflow_step_attempt` | Step Workers     | Durable, append-only record of work     |
| Config aggregates (pools, rules) | Postgres (event-sourced)             | Command side     | Auditable, replayable history           |
| Idempotency records              | Postgres `eec_routing_idempotency`   | Routing plane    | Durable dedup across retries            |
| Capacity / concurrency counters  | Redis (Lua-atomic)                   | Hot path         | Fast, atomic, cross-pod without a lock  |
| Pool / rule cache                | Redis                                | Cache layer      | Low-latency reads, pub/sub invalidation |
| Step tasks & wakeups             | Redis (tiered queue)                 | Dispatcher       | Fair, durable-enough delivery plane     |

The rule of thumb: **anything that must survive a crash lives in Postgres;
anything that must be fast and shared across pods lives in Redis.** Redis is a
coordinating accelerator, not the system of record.

## Workflow execution flow [#workflow-execution-flow]

```mermaid
flowchart TD
  T["Trigger"] --> CR["Create run<br/>(Postgres)"]
  CR --> W["Coordinator wakeup<br/>(Redis)"]
  W --> Tick
  subgraph Tick["Coordinator.Tick"]
    direction TB
    L["Load state (Postgres)"] --> Cp["Compute payload (pure)"]
    Cp --> Cm["Commit (Postgres, record_version)"]
    Cm --> D["Dispatch step tasks<br/>(Redis: SyncMatch / tiered queue)"]
  end
  Tick --> SW["Step Worker executes"]
  SW --> AR["Write attempt result<br/>(Postgres)"]
  AR --> W2["Coordinator wakeup (Redis)"]
  W2 -->|next tick folds in the result| Tick
```

Run state and attempt results both live in Postgres, so the durable record of
"where is this run" and "what did each step produce" survives any pod restart.
Redis carries the *coordination* — wakeups and task delivery — which can be
redriven if lost. See
[The Coordinator-Worker Model](/docs/architecture/coordinator-worker-model).

## Routing decision flow [#routing-decision-flow]

```mermaid
flowchart TD
  E["Routable event"] --> R["Reserve idempotency<br/>(Postgres)"]
  R --> V["Validate"]
  V --> En["Enrich<br/>(read cache: Redis)"]
  En --> F["Filter"]
  F --> S["Select"]
  S --> A["Assign"]
  A --> Fin["Finalize<br/>persist decision (Postgres)<br/>+ capacity counters (Redis Lua)<br/>+ emit events & metrics"]
```

The [routing pipeline](/docs/concepts/routing-pipeline) reads cached pools and
rules from Redis on the enrich stage, then commits its decision to Postgres and
adjusts capacity counters in Redis atomically. The idempotency reservation in
Postgres is what makes a retried request reuse its decision.

## Config change and cache invalidation [#config-change-and-cache-invalidation]

Config changes fan out through the [append-only bus](/docs/concepts/events#2-the-append-only-bus):

```mermaid
flowchart TD
  Ch["Pool / rule change"] --> Ev["Append event<br/>(Postgres)"]
  Ev --> PS["Redis pub/sub fanout"]
  PS --> Inv["Every pod's cache invalidator<br/>drops the stale entry"]
```

The event is **persisted before** it's published, so a dropped pub/sub message
degrades cache freshness (until the next read reconciles), never correctness.
Every pod subscribes and invalidates its local cache, so a rule edit propagates
fleet-wide without a redeploy.

## Capacity counters [#capacity-counters]

Concurrency and rate ceilings are Redis counters mutated by Lua scripts, so an
increment-check-decrement is a single atomic step with no read-modify-write race
across pods. This is the same primitive used by
[entitlements](/docs/concepts/entitlements) on the routing hot path and by the
[tiered queue](/docs/concepts/tiered-queue) at lease time — one mechanism, two
call sites.

<Callout title="Postgres decides, Redis coordinates">
  If you remember one thing about Ductor's data flow: &#x2A;*Postgres is the arbiter
  of truth and correctness; Redis makes it fast and fair.** Every durable
  decision is a Postgres commit. Redis carries wakeups, task delivery, caches,
  and counters — all of which can be rebuilt or redriven from the durable state
  if they're lost.
</Callout>

## Where to go next [#where-to-go-next]

* [Storage Model](/docs/architecture/storage-model) — how the Postgres side is structured (sqlc, CQRS, migrations).
* [The Tiered Fair Queue](/docs/concepts/tiered-queue) — the Redis delivery plane.
* [Idempotency & Exactly-Once](/docs/concepts/idempotency) — how the two stores combine for delivery guarantees.