Delivery Reliability

Ductor treats provider delivery as an external side effect, not a request that is always safe to repeat. Every (tenant, transaction, workflow, subscriber, channel, target) attempt is claimed through a durable ledger before a provider can be called. Fenced claim tokens prevent stale workers from completing another worker's attempt, and retries are bounded by what Ductor can prove about the previous effect.

Attempt identity and lifecycle

Reusing a logical attempt with identical canonical inputs returns the existing record; reusing it with different inputs is a conflict. Delivery status moves through queued, running, and one terminal state: sent, failed, or skipped.

The failure_phase is what decides whether a retry may happen at all. Failures before the provider accepted anything are safely repeatable; a failure after the call, where the outcome is unknown, is deliberately not retried — sending again could duplicate a real notification, so it dead-letters for a human or a reconciliation pass instead:

Automatic retries occur only when the previous attempt is known not to have produced an unsafe duplicate effect. An ambiguous post-call failure is dead-lettered for reconciliation instead of being sent again automatically. Default retry policy is at most five attempts, exponential delay from 1 second to 1 minute, no more than 15 minutes elapsed, and a 30-second claim lease. Provider Retry-After may lengthen, but never shorten, the delay.

Bounded, resumable topic fan-out

A topic trigger first persists one encrypted confidential payload object and one fan-out manifest. Per-subscriber jobs carry the payload reference, not a copy of the message content. The manifest freezes the admission-time membership cutoff and uses stable keyset pages, deterministic emission keys, and committed progress so a worker restart resumes without starting the broadcast over.

Current safeguards are:

• 500 members per expansion page;
• 1,000,000 recipients per topic broadcast;
• 256 KiB maximum canonical payload;
• maximum JSON nesting depth of 32;
• seven-day default payload-object retention;
• 30-second manifest claim lease.

Topic fan-out has no synchronous fallback. If durable payload, manifest, or queue storage is unavailable, trigger admission fails before it reports success. Setting notification.fanout.queue_disabled=true therefore disables topic delivery while single-subscriber dispatch remains available. Enterprise configuration rejects that setting when notification dispatch is enabled.

Protected content and safe diagnostics

Provider content is materialized only on the execution path. Durable fan-out stores encrypted payload objects and validates tenant, hash, version, and expiry before hydration. Credential material remains in connector storage; notification records carry integration, connection, provider, and secret references rather than resolved secret values.

Delivery lists, dead letters, observations, logs, and audit events do not return raw message bodies, provider response bodies, or credentials. Operator-facing content views mask secret-like keys and values and fail closed beyond 12 levels, 2,048 keys, or 64 KiB of retained strings. Diagnostic strings are individually bounded and redacted.

Provider acceptance and observations

status=sent means the provider accepted the request. It does not prove that a recipient device or mailbox received it. Downstream state is independent: unknown, delivered, temporarily_failed, bounced, rejected, or complained. Engagement (opened or clicked) is recorded separately.

GET /api/v2/notifications/deliveries/{delivery_id}/observations

The response is an append-only, normalized, redacted history containing provider and event identifiers, state, engagement, provider code, safe summary, provider event time, observed time, and source. Duplicate authenticated provider events converge on their canonical identity.

Dead letters and audited redrive

List tenant-scoped dead letters with notification:read:

GET /api/v2/notifications/dead-letters?limit=50&offset=0

Each row is deliberately content-free: delivery id, typed failure phase/code, provider and integration ids, redacted safe summary, attempt number, effect generation, and created/resolved timestamps.

Redrive requires notification:admin, an operator reason of at most 512 characters, and one explicit effect mode:

curl -X POST "$DUCTOR_API/api/v2/notifications/dead-letters/$DEAD_LETTER_ID:redrive" \
  -H "Authorization: Bearer $DUCTOR_TOKEN" \
  -H "X-Tenant-ID: $DUCTOR_TENANT_ID" \
  -H "Content-Type: application/json" \
  -d '{
    "reason": "Provider confirmed the first request was not accepted",
    "effect_mode": "new_external_send"
  }'

Redrive fails closed if durable audit is unavailable. A successful call resolves the dead letter and resets the delivery ledger to queued with the chosen attempt semantics. It does not retain, reconstruct, or enqueue the original notification body. The operator or calling system must replay the original trigger using the same canonical business inputs and its own retained source data.

Operator runbook

1. Inspect the delivery record, failure phase/code, attempt numbers, and reconciliation_required.
2. Read append-only observations and verify provider-side evidence using the provider message id outside Ductor when necessary.
3. Prefer reconcile_replay while the external effect remains uncertain.
4. Select new_external_send only after establishing that a second effect is wanted.
5. Supply a useful, non-secret reason; Ductor writes it to durable audit in redacted form.
6. Replay the original trigger from the authoritative source payload after redrive.
7. Confirm the new ledger generation and downstream observation state.

Related