# Queue operations (/docs/operations/queue-operations)



This is the operator's control plane over the [tiered fair
queue](/docs/concepts/tiered-queue). That page explains *how* the queue
schedules work fairly across tenants; this one is *how you operate it* when a
pool needs to be paused, drained, inspected, or when failed items pile up in the
dead-letter queue.

## Queue control plane [#queue-control-plane]

Every pool has a routing queue you can pause, resume, drain, and inspect. These
are deliberate, admin-gated operations — reach for them during an incident, a
controlled cutover, or a backlog investigation.

| Operation       | Endpoint                            | Notes                                                               |
| --------------- | ----------------------------------- | ------------------------------------------------------------------- |
| List all queues | `GET /api/queues`                   | Filter by `tenant_id` or `paused_only`; cursor-paginated            |
| Get pool status | `GET /api/queues/{pool_id}/status`  | Depth, pause state, oldest-item age, average wait, daily throughput |
| Pause           | `POST /api/queues/{pool_id}/pause`  | Items keep enqueuing but stop dequeuing — admin only                |
| Resume          | `POST /api/queues/{pool_id}/resume` | Queued items begin processing immediately — admin only              |
| Drain           | `POST /api/queues/{pool_id}/drain`  | Removes up to N items **without processing them** — admin only      |

<Callout type="warn" title="Pause holds, drain discards">
  **Pause** stops dequeuing but keeps items safe — processing resumes on
  `resume`. **Drain** removes items from the queue *without routing them* and
  returns the drained count plus remaining depth. Drain is destructive to
  in-flight work; use it to shed a poisoned backlog, not to pause traffic.
</Callout>

Check a pool before acting on it:

```bash
curl http://localhost:8080/api/queues/POOL_ID/status \
  -H "X-Tenant-ID: $DUCTOR_TENANT_ID"
```

Pause a pool during an incident, then resume when clear:

```bash
curl -X POST http://localhost:8080/api/queues/POOL_ID/pause \
  -H "X-Tenant-ID: $DUCTOR_TENANT_ID"

curl -X POST http://localhost:8080/api/queues/POOL_ID/resume \
  -H "X-Tenant-ID: $DUCTOR_TENANT_ID"
```

## Admission control [#admission-control]

Beyond raw depth, the queue runs an **admission-control** subsystem that parks
items behind singleton, debounce, throttle, and concurrency constraints. These
endpoints let you predict, explain, and override admission decisions without
touching the fair-scheduling machinery.

| Operation             | Endpoint                                                                         | Purpose                                                                                          |
| --------------------- | -------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ |
| Preview admission     | `POST /api/queues/{pool_id}:preview-admission`                                   | Evaluate admission for a sample routable **without enqueueing** or consuming throttle tokens     |
| Explain an item       | `GET /api/queues/{pool_id}/items/{queue_item_id}/admission-explanation`          | Current admission status for one item, with redacted group identity and retry / not-before hints |
| List admission groups | `GET /api/queues/admission-groups`                                               | Redacted groups, filterable by environment, pool, constraint kind, and status                    |
| Get one group         | `GET /api/queues/{pool_id}/admission-groups/{constraint_kind}/{group_id}`        | One singleton, debounce, throttle, or concurrency group — raw key values never exposed           |
| Clear a group         | `POST /api/queues/{pool_id}/admission-groups/{constraint_kind}/{group_id}:clear` | Expires a singleton, debounce, or throttle group and records an operator override reason         |

<Callout type="info" title="Why an item is stuck">
  If a pool has depth but nothing is draining, an item may be *parked by
  admission*, not blocked by a pause. Use `admission-explanation` to see which
  constraint holds it and the not-before hint, then `preview-admission` to
  confirm a fix before you enqueue for real. `:clear` is the escape hatch — it
  expires a singleton/debounce/throttle group and always records your override
  reason.
</Callout>

## Dead-letter queue runbook [#dead-letter-queue-runbook]

Items that fail routing land in the pool's **dead-letter queue**. The DLQ is
where you triage failures, retry what's recoverable, quarantine what needs
inspection, and purge what's genuinely dead. All operations are scoped to one
pool under `/api/dlq/{pool_id}`.

| Operation            | Endpoint                             | Notes                                                                         |
| -------------------- | ------------------------------------ | ----------------------------------------------------------------------------- |
| List items           | `GET /api/dlq/{pool_id}`             | Failed items, cursor-paginated                                                |
| Stats                | `GET /api/dlq/{pool_id}/stats`       | Current depth and timestamp of the oldest failure                             |
| Retry specific items | `POST /api/dlq/{pool_id}/retry`      | Re-enqueues named items; each item's retry counter increments                 |
| Retry all            | `POST /api/dlq/{pool_id}/retry-all`  | Re-enqueues everything; items past max retries are skipped                    |
| Quarantine           | `POST /api/dlq/{pool_id}/quarantine` | Excludes items from normal retry/list flows until inspected                   |
| Purge                | `POST /api/dlq/{pool_id}/purge`      | **Permanent** removal; optional `older_than` filter; needs `dlq:delete` scope |

<Steps>
  <Step>
    ### Assess the damage [#assess-the-damage]

    ```bash
    curl http://localhost:8080/api/dlq/POOL_ID/stats \
      -H "X-Tenant-ID: $DUCTOR_TENANT_ID"
    ```

    Depth and oldest-failure timestamp tell you whether this is a spike or a
    standing backlog. List the items to see what actually failed.
  </Step>

  <Step>
    ### Fix the root cause first [#fix-the-root-cause-first]

    Retrying before the underlying fault is fixed just re-fills the DLQ. Resolve the
    downstream issue (a bad connector credential, an unreachable recipient), then
    retry.
  </Step>

  <Step>
    ### Retry, quarantine, or purge [#retry-quarantine-or-purge]

    Re-enqueue everything recoverable:

    ```bash
    curl -X POST http://localhost:8080/api/dlq/POOL_ID/retry-all \
      -H "X-Tenant-ID: $DUCTOR_TENANT_ID"
    ```

    Items past their max-retry ceiling are skipped automatically. **Quarantine**
    anything that needs a human before it re-enters the pipeline. **Purge** only
    what is genuinely dead — it cannot be recovered.
  </Step>
</Steps>

<Callout type="warn" title="Purge is irreversible">
  Purged items cannot be recovered, and purge requires the `dlq:delete` scope.
  Prefer `quarantine` when you are unsure — it takes items out of the retry and
  list flows without destroying them, so you can inspect before deciding.
</Callout>

## Related [#related]

<Cards>
  <Card title="The tiered fair queue" href="/docs/concepts/tiered-queue">
    The fair-scheduling concept this runbook operates.
  </Card>

  <Card title="Pools & recipients" href="/docs/management/pools-and-recipients">
    The pools whose queues you pause, drain, and triage.
  </Card>

  <Card title="Workflow runtime" href="/docs/operations/workflow-runtime">
    Reliability safety nets, including bulk cancellation and stuck-run recovery.
  </Card>
</Cards>