Triggers & Polling

Actions are how a workflow reaches out; triggers are how the outside world reaches in. A trigger is an event source a provider exposes — a new HubSpot contact, a Stripe charge, a row added to Airtable — that starts or advances a Ductor workflow. This page covers the three delivery modes, how inbound webhooks are verified, how polling tracks its place, and how bursts are coalesced.

The TriggerSpec

A TriggerSpec (domain/connector/trigger.go) describes one event source, much as an ActionSpec describes one operation. Alongside identity (Key, Provider, DisplayName) and schema (OutputSchema, SampleData, EventTypes), it carries lifecycle hooks (OnDeploy, Activate, Deactivate, OnRenew), a DedupeStrategy, and an optional EventCoalescing policy.

The load-bearing field is the mode, which decides how the trigger is activated and how events arrive:

DedupeStrategy controls how repeat events are suppressed: unique (default, by exact event id), greatest (only the largest id/timestamp), or last (the most recent N).

Source components

Newer providers model inbound events as source components — first-class connector event producers that sit between provider setup and workflow trigger bindings (domain/connector, docs/connectors/source-components.md). Three records make up the surface:

• A source component definition is immutable provider metadata: key, props, mode, event schema, sample event, dedupe policy, state policy, runtime status, certification status, and provenance.
• A source installation owns the tenant/provider setup — provider webhook subscription refs, connection refs, resolved props, and durable state.
• A workflow source binding references one installation plus a workflow definition trigger ref.

Crucially, a binding does not copy provider subscription or checkpoint state, so one installation can feed multiple workflows. Source emissions produce a ProviderEventEnvelope (below) before any workflow fan-out, and run through the same event-coalescing contract as triggers. Runtime status is fail-closed: executable, metadata_only, needs_runtime, or blocked_by_policy — a source that lacks a Ductor-owned runtime is visible but cannot emit. (External corpus-intake candidates stay blocked_by_policy; Ductor never imports or executes third-party source automatically.) The read-only MCP tools connector.source_component.list and connector.source_component.describe expose metadata without activating or executing anything.

Trigger lifecycle

The TriggerActivator (application/connector/trigger_activator.go) owns activation and runs a sweep every 60 seconds:

• On publish — extracts the trigger steps from a published definition, checks the install gate, upserts an installation row (status active), and for instance mode mints the URL and secret. It then calls OnDeploy and Activate asynchronously and records the returned provider-side ExternalID.
• On archive — synchronously marks installations pending_uninstall, then asynchronously calls Deactivate and clears any poll dedup state.
• Sweep — reconciles installs that never got an external_id, renews installations whose OnRenew is set, runs a poll tick for poll-mode triggers, and finishes pending uninstalls.

Installations are stored in connector_trigger_installation (infrastructure/connector/trigger/), which also keeps a delivery-dedup table (connector_webhook_delivery) and durable provider-event audit tables.

Webhook ingress

Inbound webhooks land on a dedicated mux (infrastructure/connector/ingress/) mounted before tenant auth — these requests authenticate by HMAC signature, not by a tenant token:

• POST /trigger/instance/{id} — instance-mode deliveries.
• POST /app-webhook/{provider} — shared-mode deliveries.

The ingress package is deliberately isolated: a depguard rule forbids it from importing application/ or transport/, so it talks to the rest of the system only through injected adapters.

The instance-webhook flow

The shared/app-webhook flow is similar but first calls the provider's ResolveAppWebhookConnection hook to attribute the delivery to a tenant and connection, then fans out to every active installation on that connection (bounded by a max fan-out), evaluating each installation's optional CEL filter.

Signature verification

Verification follows the Standard Webhooks scheme (modules/integrations/webhook/adapter.go). The signed value is the string {msgID}.{timestamp}.{body}, HMAC-SHA256 with the webhook secret, encoded as v1,<base64>, and compared in constant time. Instance mode uses the decrypted per-installation secret; shared mode uses the connection's webhook_secret; generic providers use a per-connection secret resolver. Secret-like headers (authorization, cookie, signature, token, api-key, …) are always stripped from any metadata Ductor retains.

Provider-webhook subscriptions

Provider webhook subscriptions are Ductor-owned records for webhooks an upstream provider sends to Ductor (domain/connector/provider_webhook_subscription.go). They are distinct from the outbound, customer-facing subscriptions managed by WebhooksService — those are covered under Outbound Integrations. A subscription tracks a DesiredState (active, paused, deleted) against an observed ActualState (unknown, creating, active, paused, missing, drifted, orphaned, delete_pending, deleted, error).

Most providers fit one of three registration models, and how Ductor attributes a delivery depends on which:

The subscription is supported by several record types: callback endpoints (a stable Ductor callback URL identity plus callback/verification secret refs), route rules (safe attribution rules mapping a delivery to a connection, provider account, sync variant, or quarantine reason), reconcile runs (bounded desired-vs-observed checks that record safe evidence), and quarantine items (redacted deliveries that couldn't be safely verified or attributed).

A default-off reconciler worker (cmd/ductor/fx_connector_provider_webhook_reconciler.go), leader-elected via a Postgres advisory lock, renews subscriptions whose provider-side lease is expiring and reconciles desired-vs-observed state, detecting drift, missing, or orphaned subscriptions. Deliveries that can't be attributed or verified are quarantined rather than dropped silently, with a reason (unknown_topic, failed_verification, missing_subscription, ambiguous_attribution, …) and a replay path.

Polling

Poll-mode triggers have no webhook; the activator calls Activate on a schedule and tracks its place with a cursor. Two poll styles exist (infrastructure/connector/polling/):

• Timebased — stores the last-seen timestamp and returns items newer than it.
• Cursor — stores the last item's id and returns items appearing after it.

The activator's poll tick (trigger_activator_poll.go) fetches via Activate, splits a multi-record activation into per-record items, dedups each item, fans out one run per new item, and only then persists the advanced cursor — so a crash mid-tick re-polls rather than skipping records. Per-item dedup uses Redis with a 7-day retention window: unique uses a sorted-set ZADD NX, greatest compares against the last-seen id. The cursor is carried as opaque {"cursor": "..."} state in the installation's metadata.

Event coalescing

A chatty provider can fire dozens of webhooks for what is logically one change. Event coalescing (domain/connector/event_coalescing.go) collapses a burst of provider events into a single downstream event before starting a workflow, publishing to the event bus, or invoking a connector function.

A ConnectorEventCoalescingPolicy defines:

• Key sources — up to 8 fields that identify "the same thing": provider, event_type, connection_id, provider_account_id, subscription_id, delivery_id, normalized_object_id, safe_header, safe_query, or a bounded normalized_payload_path. Their hash is the coalescing key.
• A window — WindowMs, capped at 15 minutes. The window opens on the first event and closes at start + WindowMs.
• A take mode — latest, first, all, or summary_only, deciding which of the collected events flows downstream.
• Overflow and stale-fact behavior — what to do when the window exceeds MaxEntities/MaxBytes (reject_new, emit_and_open_next, summary_only, quarantine) or when policy facts are stale (fail_closed, bypass, quarantine).

Windows are closed and emitted by a default-off deadline worker (cmd/ductor/fx_connector_coalescing_deadline_worker.go). Each tick, per tenant, it runs three passes: dispatch due workflow windows, dispatch due event-bus windows, and reconcile windows that closed with emit-intent but no emission evidence (recovering from a crash between close and run-start). The reconcile pass is idempotent on each window's key and re-drives the coordinator wakeup before marking a window delivered — so a coalesced event is emitted exactly once.

The provider-event envelope

Every webhook and poll delivery is normalized into a ProviderEventEnvelope (domain/connector/provider_event.go) — the canonical workflow input and durable audit record. Webhook and poll triggers start the workflow DAG with that envelope mounted at TRIGGER.event: the top-level input keeps trigger_ref for step routing, but everything else lives under event.

{
  "trigger_ref": "crm_change",
  "event": {
    "event_id": "evt_...",
    "delivery_id": "del_...",
    "provider": "shopify",
    "trigger_key": "shopify.webhook",
    "mode": "shared",
    "event_type": "orders/create",
    "connection_id": "connection_uuid",
    "raw_payload": {},
    "normalized_payload": {},
    "normalized_model_key": "canonical.order",
    "schema_status": "unknown",
    "safe_headers": { "X-Shopify-Topic": "orders/create" }
  }
}

Raw provider bytes stay at event.raw_payload; mapper output (with model identity and mapper execution metadata) is attached at event.normalized_payload — see Data Mapping. Instance webhooks attribute from the trigger installation row; shared webhooks add connection attribution from the demux hook; poll triggers wrap each raw item in the same envelope using the poll dedupe key as the provider message id. Only allowlisted request metadata is retained — authorization headers, cookies, signature headers, secret/token-like names, and unallowlisted query params are stripped before persistence. Durable history is written to connector_provider_event (the audit record), with connector_webhook_delivery remaining the high-throughput dedupe table. The envelope also carries a ReconciliationHint (fetch_changed_object or run_sync_checkpoint) telling the workflow whether to re-fetch the authoritative record.

Where to go next

• Architecture — how a started run dispatches connector actions.
• The DAG Workflow Model — how a triggered run is executed.
• Building a Provider — declaring triggers on your own provider.