Events & Event Sourcing

Ductor uses the word "event" for three genuinely different things. Conflating them is the fastest way to misunderstand the system, so this page keeps them separate: aggregate event sourcing, the append-only bus, and the workflow transition journal.

1. Aggregate event sourcing

Some domain aggregates — pools, rules, and similar configuration objects — are event-sourced: their current state is a fold over an append-only log of immutable events rather than a mutable row you overwrite.

An event carries an aggregate ID, an aggregate type, a type string (like pool.created), a sequential per-aggregate version, a timestamp, an opaque JSON data payload, and correlation/causation IDs. Events are appended, never edited.

The EventStore interface enforces optimistic concurrency at append time:

Append(ctx, aggregateID, expectedVersion, events)

If expectedVersion doesn't match the aggregate's current version, the append is rejected with a concurrency-conflict error. In Postgres this is enforced by a UNIQUE (aggregate_id, version) constraint on the domain_events table — two writers can't both claim version N. Reconstruction is a replay: load events from a version and Apply each in order. Snapshots (aggregate_snapshots) let you skip replaying from zero for long-lived aggregates.

This is the audit-trail-and-time-travel layer: you can reconstruct exactly what a pool or rule looked like at any point, and why it changed.

2. The append-only bus

The append-only bus is the canonical event plane — how a state change fans out to the rest of the system. Its defining property is in the name: it appends durably before it fans out.

When something publishes an event (for example, a pool config change), the bus:

1. Normalizes and validates the event envelope.
2. Appends it durably to Postgres (with a per-event idempotency key).
3. Then attempts the optional Redis pub/sub fanout to other pods.

The order matters. Because the durable append happens first, an event is never lost just because the fanout hop failed. A fanout policy decides what a fanout failure means:

• best_effort (default) — the durable append is authoritative; fanout errors are observed and swallowed.
• required — a fanout failure is returned to the caller, so a system that must fan out can fail closed.

This is the mechanism behind cache invalidation: a config change is persisted, then a Redis pub/sub message tells every pod's cache invalidator to drop the stale entry. If the message is dropped, the durable record is still there to reconcile against.

3. The workflow transition journal

A workflow run keeps its own history — a durable, append-then-compact transition journal on eec_workflow_run (the eec_ prefix marks Ductor's durable Enterprise Eventing Core tables) and its transition rows. This is not the aggregate event store; it's the coordinator's record of every state transition a run made, which underpins replay, debugging, and audit.

Because journal payloads can be large or sensitive, they're governed by two policies:

• Payload budget — each journal entry has a payload class (inline_redacted by default, inline_public, external_artifact_ref, summary_only, discarded_by_policy) and an overflow action. Oversized payloads are externalized to the archival backend rather than bloating the run row.
• Compaction — old transitions can be pruned, but never silently: compaction writes a durable CompactionReceipt recording the sequence range removed, the policy that removed it, the retained checkpoint, and an evidence artifact reference and hash. The receipt still anchors replay, so history stays reconstructable even after it's compacted.

Which one am I looking at?

The trigger plane

None of the three systems above is how an external event starts a workflow. That is the job of the trigger plane — the ingress boundary every external source passes through before it can spawn or wake a run. It is not part of event sourcing; it is the front door.

Every external ingest surface — webhook, schedule, event, signal, manual (API/ operator), and connector sources — normalizes what it received into one canonical envelope before anything downstream sees it:

type Envelope struct {
    ID         string          // caller-facing event ID
    Source     Source          // webhook | schedule | event | signal | manual | connector
    Name       string          // event name, e.g. "stripe.invoice.paid"
    TenantID   string
    Data       json.RawMessage // opaque payload
    ReceivedAt time.Time
    InternalID string          // deterministic internal ID
    DedupeKey  string          // optional, for spawn suppression
    ContextRef ContextRef      // {type, id} the event concerns
}

Two properties matter. First, Data is hard-capped at MaxDataBytes (1 MiB) at the ingress boundary — an oversized payload is rejected there, not carried into a run. Second, ingest persists a receipt for every accepted envelope, so external ingest is auditable independently of whatever run it did or didn't spawn.

Wildcard event routing

Event-name triggers can be literal or wildcard. When you publish a workflow whose event trigger declares a name pattern, the publish path upserts a row into an EventTriggerIndex keyed (tenantID, pattern, definitionID). Patterns are either a literal name (stripe.invoice.paid) or a suffix wildcard (stripe.invoice.*).

At ingest time the dispatcher calls pkg/eventpattern.Expand on the concrete event name — expanding stripe.invoice.paid into ["stripe.invoice.paid", "stripe.invoice.*", "stripe.*", "*"] — and walks the index to find every definition that should spawn. Fan-out to N subscribers is a single index lookup, not a scan.

Which one am I looking at (again)?

The trigger plane feeds the systems above — a spawned run then writes its own transition journal, and config changes it makes fan out over the bus. If you need a run to pause mid-flight and fan in a set of correlated events, that is the event_set step, which is part of the DAG model, not the trigger plane.

Where to go next

• Idempotency & Exactly-Once — how durable append plus dedup keys give effectively-once delivery.
• Data Flow — where events sit in the system.
• The DAG Workflow Model — what a run's transition journal records, and the event_set in-run wait.
• Managing Workflows — publishing triggers and enabling wildcard fan-out.