Queue operations
The operator runbook for the tiered fair queue and its dead-letter queue — pause, resume, drain, admission control, and DLQ replay.
This is the operator's control plane over the tiered fair queue. That page explains how the queue schedules work fairly across tenants; this one is how you operate it when a pool needs to be paused, drained, inspected, or when failed items pile up in the dead-letter queue.
Queue control plane
Every pool has a routing queue you can pause, resume, drain, and inspect. These are deliberate, admin-gated operations — reach for them during an incident, a controlled cutover, or a backlog investigation.
| Operation | Endpoint | Notes |
|---|---|---|
| List all queues | GET /api/queues | Filter by tenant_id or paused_only; cursor-paginated |
| Get pool status | GET /api/queues/{pool_id}/status | Depth, pause state, oldest-item age, average wait, daily throughput |
| Pause | POST /api/queues/{pool_id}/pause | Items keep enqueuing but stop dequeuing — admin only |
| Resume | POST /api/queues/{pool_id}/resume | Queued items begin processing immediately — admin only |
| Drain | POST /api/queues/{pool_id}/drain | Removes up to N items without processing them — admin only |
Pause holds, drain discards
Pause stops dequeuing but keeps items safe — processing resumes on
resume. Drain removes items from the queue without routing them and
returns the drained count plus remaining depth. Drain is destructive to
in-flight work; use it to shed a poisoned backlog, not to pause traffic.
Check a pool before acting on it:
curl http://localhost:8080/api/queues/POOL_ID/status \
-H "X-Tenant-ID: $DUCTOR_TENANT_ID"Pause a pool during an incident, then resume when clear:
curl -X POST http://localhost:8080/api/queues/POOL_ID/pause \
-H "X-Tenant-ID: $DUCTOR_TENANT_ID"
curl -X POST http://localhost:8080/api/queues/POOL_ID/resume \
-H "X-Tenant-ID: $DUCTOR_TENANT_ID"Admission control
Beyond raw depth, the queue runs an admission-control subsystem that parks items behind singleton, debounce, throttle, and concurrency constraints. These endpoints let you predict, explain, and override admission decisions without touching the fair-scheduling machinery.
| Operation | Endpoint | Purpose |
|---|---|---|
| Preview admission | POST /api/queues/{pool_id}:preview-admission | Evaluate admission for a sample routable without enqueueing or consuming throttle tokens |
| Explain an item | GET /api/queues/{pool_id}/items/{queue_item_id}/admission-explanation | Current admission status for one item, with redacted group identity and retry / not-before hints |
| List admission groups | GET /api/queues/admission-groups | Redacted groups, filterable by environment, pool, constraint kind, and status |
| Get one group | GET /api/queues/{pool_id}/admission-groups/{constraint_kind}/{group_id} | One singleton, debounce, throttle, or concurrency group — raw key values never exposed |
| Clear a group | POST /api/queues/{pool_id}/admission-groups/{constraint_kind}/{group_id}:clear | Expires a singleton, debounce, or throttle group and records an operator override reason |
Why an item is stuck
If a pool has depth but nothing is draining, an item may be parked by
admission, not blocked by a pause. Use admission-explanation to see which
constraint holds it and the not-before hint, then preview-admission to
confirm a fix before you enqueue for real. :clear is the escape hatch — it
expires a singleton/debounce/throttle group and always records your override
reason.
Dead-letter queue runbook
Items that fail routing land in the pool's dead-letter queue. The DLQ is
where you triage failures, retry what's recoverable, quarantine what needs
inspection, and purge what's genuinely dead. All operations are scoped to one
pool under /api/dlq/{pool_id}.
| Operation | Endpoint | Notes |
|---|---|---|
| List items | GET /api/dlq/{pool_id} | Failed items, cursor-paginated |
| Stats | GET /api/dlq/{pool_id}/stats | Current depth and timestamp of the oldest failure |
| Retry specific items | POST /api/dlq/{pool_id}/retry | Re-enqueues named items; each item's retry counter increments |
| Retry all | POST /api/dlq/{pool_id}/retry-all | Re-enqueues everything; items past max retries are skipped |
| Quarantine | POST /api/dlq/{pool_id}/quarantine | Excludes items from normal retry/list flows until inspected |
| Purge | POST /api/dlq/{pool_id}/purge | Permanent removal; optional older_than filter; needs dlq:delete scope |
Assess the damage
curl http://localhost:8080/api/dlq/POOL_ID/stats \
-H "X-Tenant-ID: $DUCTOR_TENANT_ID"Depth and oldest-failure timestamp tell you whether this is a spike or a standing backlog. List the items to see what actually failed.
Fix the root cause first
Retrying before the underlying fault is fixed just re-fills the DLQ. Resolve the downstream issue (a bad connector credential, an unreachable recipient), then retry.
Retry, quarantine, or purge
Re-enqueue everything recoverable:
curl -X POST http://localhost:8080/api/dlq/POOL_ID/retry-all \
-H "X-Tenant-ID: $DUCTOR_TENANT_ID"Items past their max-retry ceiling are skipped automatically. Quarantine anything that needs a human before it re-enters the pipeline. Purge only what is genuinely dead — it cannot be recovered.
Purge is irreversible
Purged items cannot be recovered, and purge requires the dlq:delete scope.
Prefer quarantine when you are unsure — it takes items out of the retry and
list flows without destroying them, so you can inspect before deciding.