# Triggers & Polling (/docs/connectors/triggers-and-polling)



Actions are how a workflow reaches *out*; triggers are how the outside world
reaches *in*. A trigger is an event source a provider exposes — a new HubSpot
contact, a Stripe charge, a row added to Airtable — that starts or advances a
Ductor workflow. This page covers the three delivery modes, how inbound webhooks
are verified, how polling tracks its place, and how bursts are coalesced.

## The TriggerSpec [#the-triggerspec]

A `TriggerSpec` (`domain/connector/trigger.go`) describes one event source, much
as an `ActionSpec` describes one operation. Alongside identity
(`Key`, `Provider`, `DisplayName`) and schema (`OutputSchema`, `SampleData`,
`EventTypes`), it carries lifecycle hooks (`OnDeploy`, `Activate`, `Deactivate`,
`OnRenew`), a `DedupeStrategy`, and an optional `EventCoalescing` policy.

The load-bearing field is the **mode**, which decides how the trigger is
activated and how events arrive:

| Mode     | Value      | How it works                                                                                                                                                                            |
| -------- | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Instance | `instance` | Ductor mints a unique, per-pipeline webhook URL at publish time and hands it to the user or external system to register.                                                                |
| Shared   | `shared`   | A single app-level webhook is registered once per provider account; inbound deliveries are demultiplexed to the right connection via the provider's `ResolveAppWebhookConnection` hook. |
| Poll     | `poll`     | No webhook — Ductor runs the trigger's `Activate` on a schedule owned by the trigger activator.                                                                                         |

`DedupeStrategy` controls how repeat events are suppressed: `unique` (default,
by exact event id), `greatest` (only the largest id/timestamp), or `last` (the
most recent N).

## Source components [#source-components]

Newer providers model inbound events as **source components** — first-class
connector event *producers* that sit between provider setup and workflow trigger
bindings (`domain/connector`, `docs/connectors/source-components.md`). Three
records make up the surface:

* A **source component definition** is immutable provider metadata: key, props,
  mode, event schema, sample event, dedupe policy, state policy, runtime status,
  certification status, and provenance.
* A **source installation** owns the tenant/provider setup — provider webhook
  subscription refs, connection refs, resolved props, and durable state.
* A **workflow source binding** references one installation plus a workflow
  definition trigger ref.

Crucially, a binding does not copy provider subscription or checkpoint state, so
**one installation can feed multiple workflows**. Source emissions produce a
`ProviderEventEnvelope` (below) before any workflow fan-out, and run through the
same event-coalescing contract as triggers. Runtime status is fail-closed:
`executable`, `metadata_only`, `needs_runtime`, or `blocked_by_policy` — a source
that lacks a Ductor-owned runtime is visible but cannot emit. (External
corpus-intake candidates stay `blocked_by_policy`; Ductor never imports or
executes third-party source automatically.) The read-only MCP tools
`connector.source_component.list` and `connector.source_component.describe` expose
metadata without activating or executing anything.

## Trigger lifecycle [#trigger-lifecycle]

The `TriggerActivator` (`application/connector/trigger_activator.go`) owns
activation and runs a sweep every 60 seconds:

* **On publish** — extracts the trigger steps from a published definition, checks
  the install gate, upserts an installation row (status `active`), and for
  instance mode mints the URL and secret. It then calls `OnDeploy` and `Activate`
  asynchronously and records the returned provider-side `ExternalID`.
* **On archive** — synchronously marks installations `pending_uninstall`, then
  asynchronously calls `Deactivate` and clears any poll dedup state.
* **Sweep** — reconciles installs that never got an `external_id`, renews
  installations whose `OnRenew` is set, runs a poll tick for poll-mode triggers,
  and finishes pending uninstalls.

Installations are stored in `connector_trigger_installation`
(`infrastructure/connector/trigger/`), which also keeps a delivery-dedup table
(`connector_webhook_delivery`) and durable provider-event audit tables.

## Webhook ingress [#webhook-ingress]

Inbound webhooks land on a dedicated mux (`infrastructure/connector/ingress/`)
mounted **before** tenant auth — these requests authenticate by HMAC signature,
not by a tenant token:

* `POST /trigger/instance/{id}` — instance-mode deliveries.
* `POST /app-webhook/{provider}` — shared-mode deliveries.

The ingress package is deliberately isolated: a depguard rule forbids it from
importing `application/` or `transport/`, so it talks to the rest of the system
only through injected adapters.

### The instance-webhook flow [#the-instance-webhook-flow]

<Steps>
  <Step>
    Read the body (capped at 4 MB → `413` if larger).
  </Step>

  <Step>
    Read the `webhook-id`, `webhook-timestamp`, and `webhook-signature` headers.
  </Step>

  <Step>
    **Replay check** — the timestamp must be within ±5 minutes, else `400`.
  </Step>

  <Step>
    Look up the installation by instance id (`404` if missing, rejected if not
    `active`), and check the install gate.
  </Step>

  <Step>
    Decrypt the per-installation secret and **verify the signature** (`401` on
    mismatch).
  </Step>

  <Step>
    Record the delivery for idempotency — a duplicate returns `409` to sync
    callers, `200` to async ones.
  </Step>

  <Step>
    Apply a GCRA rate limit keyed on the installation (`429` if limited).
  </Step>

  <Step>
    Map the payload, apply coalescing, and start the run (`202` with the run id).
  </Step>
</Steps>

The shared/app-webhook flow is similar but first calls the provider's
`ResolveAppWebhookConnection` hook to attribute the delivery to a tenant and
connection, then fans out to every active installation on that connection
(bounded by a max fan-out), evaluating each installation's optional CEL filter.

### Signature verification [#signature-verification]

Verification follows the [Standard Webhooks](https://www.standardwebhooks.com/)
scheme (`modules/integrations/webhook/adapter.go`). The signed value is the
string `{msgID}.{timestamp}.{body}`, HMAC-SHA256 with the webhook secret, encoded
as `v1,<base64>`, and compared in constant time. Instance mode uses the decrypted
per-installation secret; shared mode uses the connection's `webhook_secret`;
generic providers use a per-connection secret resolver. Secret-like headers
(`authorization`, `cookie`, `signature`, `token`, `api-key`, …) are always
stripped from any metadata Ductor retains.

<Callout title="Two dedup layers plus replay protection">
  The fast idempotency ledger (`connector_webhook_delivery`) rejects duplicate
  deliveries; a durable `connector_provider_event` table is the audit record;
  and the ±5-minute timestamp window blocks replays. A webhook that arrives twice
  starts a workflow run at most once.
</Callout>

## Provider-webhook subscriptions [#provider-webhook-subscriptions]

**Provider webhook subscriptions** are Ductor-owned records for webhooks an
upstream provider sends *to* Ductor (`domain/connector/provider_webhook_subscription.go`).
They are distinct from the outbound, customer-facing subscriptions managed by
`WebhooksService` — those are covered under
[Outbound Integrations](/docs/connectors/outbound-integrations). A subscription
tracks a `DesiredState` (`active`, `paused`, `deleted`) against an observed
`ActualState` (`unknown`, `creating`, `active`, `paused`, `missing`, `drifted`,
`orphaned`, `delete_pending`, `deleted`, `error`).

Most providers fit one of three **registration models**, and how Ductor attributes
a delivery depends on which:

| Model (scope)              | What it is                                                      | Attribution                                                                  |
| -------------------------- | --------------------------------------------------------------- | ---------------------------------------------------------------------------- |
| `global` app webhook       | One provider registration sends events for many accounts.       | Ductor must attribute each delivery before any connection-scoped logic runs. |
| `provider_account` webhook | One registration belongs to a tenant/provider-account identity. | Often preserved across reconnects when identity locks agree.                 |
| `connection` webhook       | One registration belongs to one Ductor connection.              | Must be cleaned up before that connection can be safely deleted.             |

The subscription is supported by several record types: **callback endpoints** (a
stable Ductor callback URL identity plus callback/verification secret refs),
**route rules** (safe attribution rules mapping a delivery to a connection,
provider account, sync variant, or quarantine reason), **reconcile runs** (bounded
desired-vs-observed checks that record safe evidence), and **quarantine items**
(redacted deliveries that couldn't be safely verified or attributed).

<Callout type="warn" title="Plan before mutating provider state; store refs, never secrets">
  Always plan a registration before mutating provider-side state:
  `PlanProviderWebhookSubscription` / `connector.provider_webhook.plan` returns the
  expected callback endpoint, event types, provider operation target, blockers, and
  manual-setup status. Subscription rows store only **secret refs** and **redacted
  external displays** (`RedactProviderWebhookExternalID` yields
  `redacted:abcd...wxyz`) — never callback signing secrets, access tokens, raw
  provider payloads, or unredacted credentials. Rotate secrets through
  `RotateProviderWebhookSubscriptionSecret` rather than editing endpoint rows.
</Callout>

A default-off reconciler worker
(`cmd/ductor/fx_connector_provider_webhook_reconciler.go`), leader-elected via a
Postgres advisory lock, renews subscriptions whose provider-side lease is expiring
and reconciles desired-vs-observed state, detecting drift, missing, or orphaned
subscriptions. Deliveries that can't be attributed or verified are **quarantined**
rather than dropped silently, with a reason (`unknown_topic`, `failed_verification`,
`missing_subscription`, `ambiguous_attribution`, …) and a replay path.

## Polling [#polling]

Poll-mode triggers have no webhook; the activator calls `Activate` on a schedule
and tracks its place with a cursor. Two poll styles exist
(`infrastructure/connector/polling/`):

* **Timebased** — stores the last-seen timestamp and returns items newer than it.
* **Cursor** — stores the last item's id and returns items appearing after it.

The activator's poll tick (`trigger_activator_poll.go`) fetches via `Activate`,
splits a multi-record activation into per-record items, dedups each item, fans
out one run per new item, and only then persists the advanced cursor — so a crash
mid-tick re-polls rather than skipping records. Per-item dedup uses Redis with a
7-day retention window: `unique` uses a sorted-set `ZADD NX`, `greatest` compares
against the last-seen id. The cursor is carried as opaque `{"cursor": "..."}`
state in the installation's metadata.

<Callout type="info" title="Poll cursors are seeded from the sync checkpoint">
  For providers that also run a sync, the poll cursor is seeded from — and
  written back to — the sync cursor, so polling and syncing share one notion of
  "where we've read up to" instead of drifting apart.
</Callout>

## Event coalescing [#event-coalescing]

A chatty provider can fire dozens of webhooks for what is logically one change.
**Event coalescing** (`domain/connector/event_coalescing.go`) collapses a burst
of provider events into a single downstream event before starting a workflow,
publishing to the event bus, or invoking a connector function.

A `ConnectorEventCoalescingPolicy` defines:

* **Key sources** — up to 8 fields that identify "the same thing": `provider`,
  `event_type`, `connection_id`, `provider_account_id`, `subscription_id`,
  `delivery_id`, `normalized_object_id`, `safe_header`, `safe_query`, or a bounded
  `normalized_payload_path`. Their hash is the coalescing key.
* **A window** — `WindowMs`, capped at 15 minutes. The window opens on the first
  event and closes at `start + WindowMs`.
* **A take mode** — `latest`, `first`, `all`, or `summary_only`, deciding which
  of the collected events flows downstream.
* **Overflow and stale-fact behavior** — what to do when the window exceeds
  `MaxEntities`/`MaxBytes` (`reject_new`, `emit_and_open_next`, `summary_only`,
  `quarantine`) or when policy facts are stale (`fail_closed`, `bypass`,
  `quarantine`).

Windows are closed and emitted by a default-off **deadline worker**
(`cmd/ductor/fx_connector_coalescing_deadline_worker.go`). Each tick, per tenant,
it runs three passes: dispatch due workflow windows, dispatch due event-bus
windows, and reconcile windows that closed with emit-intent but no emission
evidence (recovering from a crash between close and run-start). The reconcile
pass is idempotent on each window's key and re-drives the coordinator wakeup
before marking a window delivered — so a coalesced event is emitted exactly once.

## The provider-event envelope [#the-provider-event-envelope]

Every webhook and poll delivery is normalized into a `ProviderEventEnvelope`
(`domain/connector/provider_event.go&#x60;) — the canonical workflow input and durable
audit record. Webhook and poll triggers start the workflow DAG with that envelope
mounted at &#x2A;*`TRIGGER.event`**: the top-level input keeps `trigger_ref` for step
routing, but everything else lives under `event`.

```json
{
  "trigger_ref": "crm_change",
  "event": {
    "event_id": "evt_...",
    "delivery_id": "del_...",
    "provider": "shopify",
    "trigger_key": "shopify.webhook",
    "mode": "shared",
    "event_type": "orders/create",
    "connection_id": "connection_uuid",
    "raw_payload": {},
    "normalized_payload": {},
    "normalized_model_key": "canonical.order",
    "schema_status": "unknown",
    "safe_headers": { "X-Shopify-Topic": "orders/create" }
  }
}
```

Raw provider bytes stay at `event.raw_payload`; mapper output (with model identity
and mapper execution metadata) is attached at `event.normalized_payload` — see
[Data Mapping](/docs/connectors/data-mapping). Instance webhooks attribute from the
trigger installation row; shared webhooks add connection attribution from the demux
hook; poll triggers wrap each raw item in the same envelope using the poll dedupe
key as the provider message id. Only allowlisted request metadata is retained —
authorization headers, cookies, signature headers, secret/token-like names, and
unallowlisted query params are stripped before persistence. Durable history is
written to `connector_provider_event` (the audit record), with
`connector_webhook_delivery` remaining the high-throughput dedupe table. The
envelope also carries a `ReconciliationHint` (`fetch_changed_object` or
`run_sync_checkpoint`) telling the workflow whether to re-fetch the authoritative
record.

## Where to go next [#where-to-go-next]

* [Architecture](/docs/connectors/architecture) — how a started run dispatches connector actions.
* [The DAG Workflow Model](/docs/concepts/dag-workflow-model) — how a triggered run is executed.
* [Building a Provider](/docs/connectors/building-a-provider) — declaring triggers on your own provider.