# Delivery Reliability (/docs/notifications/delivery-reliability)
Ductor treats provider delivery as an external side effect, not a request that is
always safe to repeat. Every `(tenant, transaction, workflow, subscriber, channel,
target)` attempt is claimed through a durable ledger before a provider can be called.
Fenced claim tokens prevent stale workers from completing another worker's attempt,
and retries are bounded by what Ductor can prove about the previous effect.
## Attempt identity and lifecycle [#attempt-identity-and-lifecycle]
Reusing a logical attempt with identical canonical inputs returns the existing
record; reusing it with different inputs is a conflict. Delivery status moves through
`queued`, `running`, and one terminal state: `sent`, `failed`, or `skipped`.
| Field | Meaning |
| -------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| `execution_attempt_number` | External-effect generation. An explicit new-send redrive increments it. |
| `provider_attempt_number` | Bounded provider calls within the current effect generation. |
| `failure_phase` | Where the effect failed: `before_call`, `request_not_accepted`, `accepted`, `after_call_ambiguous`, or `permanent`. |
| `failure_code` | Stable machine-readable provider or transport diagnosis. |
| `next_attempt_at` | Earliest time a proven-safe retry may run. |
| `reason_code` | Stable preference or skip explanation. |
| `resolver_version` | Preference algorithm version used for the decision. |
| `reconciliation_required` | The external outcome is ambiguous or contradicts later evidence. |
The `failure_phase` is what decides whether a retry may happen at all. Failures
*before* the provider accepted anything are safely repeatable; a failure *after*
the call, where the outcome is unknown, is deliberately not retried — sending
again could duplicate a real notification, so it dead-letters for a human or a
reconciliation pass instead:
```mermaid
flowchart TD
call[provider call] --> ph{failure phase}
ph -->|before_call| safe[safe to retry]
ph -->|request_not_accepted| safe
ph -->|after_call_ambiguous| dlq([dead letter, reconcile])
ph -->|permanent| stop([failed, no retry])
safe --> budget{attempts left?}
budget -->|yes| wait[backoff, honor Retry-After] --> call
budget -->|no| dlq
```
Automatic retries occur only when the previous attempt is known not to have produced
an unsafe duplicate effect. An ambiguous post-call failure is dead-lettered for
reconciliation instead of being sent again automatically. Default retry policy is at
most five attempts, exponential delay from 1 second to 1 minute, no more than 15
minutes elapsed, and a 30-second claim lease. Provider `Retry-After` may lengthen, but
never shorten, the delay.
Common reason codes include `enabled_default`, `disabled_global`,
`disabled_topic`, `disabled_workflow`, `disabled_subscriber`, `quiet_hours`,
`digest_deferred`, `condition_false`, `condition_error`, `schedule_error`,
`missing_endpoint`, `integration_unavailable`, `provider_unavailable`,
`pre_filtered`, `subscriber_disabled`, and `in_app_not_wired`. Clients should
branch on these codes instead of parsing human-readable text.
## Bounded, resumable topic fan-out [#bounded-resumable-topic-fan-out]
A topic trigger first persists one encrypted `confidential` payload object and one
fan-out manifest. Per-subscriber jobs carry the payload reference, not a copy of the
message content. The manifest freezes the admission-time membership cutoff and uses
stable keyset pages, deterministic emission keys, and committed progress so a worker
restart resumes without starting the broadcast over.
Current safeguards are:
* 500 members per expansion page;
* 1,000,000 recipients per topic broadcast;
* 256 KiB maximum canonical payload;
* maximum JSON nesting depth of 32;
* seven-day default payload-object retention;
* 30-second manifest claim lease.
Topic fan-out has no synchronous fallback. If durable payload, manifest, or queue
storage is unavailable, trigger admission fails before it reports success. Setting
`notification.fanout.queue_disabled=true` therefore disables topic delivery while
single-subscriber dispatch remains available. Enterprise configuration rejects that
setting when notification dispatch is enabled.
## Protected content and safe diagnostics [#protected-content-and-safe-diagnostics]
Provider content is materialized only on the execution path. Durable fan-out stores
encrypted payload objects and validates tenant, hash, version, and expiry before
hydration. Credential material remains in connector storage; notification records
carry integration, connection, provider, and secret references rather than resolved
secret values.
Delivery lists, dead letters, observations, logs, and audit events do not return raw
message bodies, provider response bodies, or credentials. Operator-facing content
views mask secret-like keys and values and fail closed beyond 12 levels, 2,048 keys,
or 64 KiB of retained strings. Diagnostic strings are individually bounded and
redacted.
## Provider acceptance and observations [#provider-acceptance-and-observations]
`status=sent` means the provider accepted the request. It does not prove that a
recipient device or mailbox received it. Downstream state is independent:
`unknown`, `delivered`, `temporarily_failed`, `bounced`, `rejected`, or `complained`.
Engagement (`opened` or `clicked`) is recorded separately.
```http
GET /api/v2/notifications/deliveries/{delivery_id}/observations
```
The response is an append-only, normalized, redacted history containing provider and
event identifiers, state, engagement, provider code, safe summary, provider event
time, observed time, and source. Duplicate authenticated provider events converge on
their canonical identity.
Current bundled providers do not expose a generic callback or polling route with
sufficient authenticated tenant attribution. Ductor therefore does not mount an
open provider-receipt endpoint. Observation ingestion must come through a trusted
adapter that can bind the event to a tenant, provider, and existing delivery.
## Dead letters and audited redrive [#dead-letters-and-audited-redrive]
List tenant-scoped dead letters with `notification:read`:
```http
GET /api/v2/notifications/dead-letters?limit=50&offset=0
```
Each row is deliberately content-free: delivery id, typed failure phase/code,
provider and integration ids, redacted safe summary, attempt number, effect
generation, and created/resolved timestamps.
Redrive requires `notification:admin`, an operator reason of at most 512 characters,
and one explicit effect mode:
| Effect mode | Use when |
| ------------------- | ----------------------------------------------------------------------------------------------------- |
| `reconcile_replay` | Re-evaluating an attempt without authorizing a new external send. |
| `new_external_send` | Operator has confirmed that a new provider effect is intended. This increments the effect generation. |
```bash
curl -X POST "$DUCTOR_API/api/v2/notifications/dead-letters/$DEAD_LETTER_ID:redrive" \
-H "Authorization: Bearer $DUCTOR_TOKEN" \
-H "X-Tenant-ID: $DUCTOR_TENANT_ID" \
-H "Content-Type: application/json" \
-d '{
"reason": "Provider confirmed the first request was not accepted",
"effect_mode": "new_external_send"
}'
```
Redrive fails closed if durable audit is unavailable. A successful call resolves the
dead letter and resets the delivery ledger to `queued` with the chosen attempt
semantics. It does **not** retain, reconstruct, or enqueue the original notification
body. The operator or calling system must replay the original trigger using the same
canonical business inputs and its own retained source data.
## Operator runbook [#operator-runbook]
1. Inspect the delivery record, failure phase/code, attempt numbers, and
`reconciliation_required`.
2. Read append-only observations and verify provider-side evidence using the provider
message id outside Ductor when necessary.
3. Prefer `reconcile_replay` while the external effect remains uncertain.
4. Select `new_external_send` only after establishing that a second effect is wanted.
5. Supply a useful, non-secret reason; Ductor writes it to durable audit in redacted
form.
6. Replay the original trigger from the authoritative source payload after redrive.
7. Confirm the new ledger generation and downstream observation state.
## Related [#related]
Subscriber, topic, preference, integration, layout, and trigger APIs.
Subscriber-visible in-app message lifecycle and live updates.
General queue inspection and recovery outside notification delivery.