# The Sync Engine (/docs/connectors/sync-engine)



Actions reach out on demand and triggers push events in, but some workflows need
a *local, queryable copy* of a provider's records — every HubSpot contact, every
Shopify order — kept fresh in the background. That is what the **sync engine**
does: it replicates provider records into schema-versioned, normalized
`SyncRecord`s that routing, unified-object reads, and strategy feature extraction
can read without a live provider call.

This page covers the control planes that make replication safe: variants,
scheduling, run exclusivity, cache retention, and realtime binding. For how
provider payloads become normalized records, see
[Data Mapping](/docs/connectors/data-mapping).

## Sync installations and variants [#sync-installations-and-variants]

A **sync installation** is a tenant-owned binding from one registered sync spec
(`hubspot.contacts`) to a connection. Underneath an installation, work runs in a
**sync variant** (`domain/connector`, `docs/connectors/sync-variants.md`) — a
named execution and cache namespace with its own cursor, checkpoint namespace,
record cache, run history, frequency override, and lifecycle status.

Variants let one installation replicate several independently-isolated scopes —
a mailbox, folder, market, repository, or segment — without collisions. Because
the record uniqueness key *includes* the variant, base and named variants can
hold the same upstream `external_id` side by side.

<Callout type="warn" title="The base variant is protected">
  Every installation has a `base` variant. Public API callers may omit
  `variant_key`; Ductor resolves the request to `base` and echoes
  `variant_key = "base"` back. Operators **cannot** create, rename, archive, or
  delete `base`. Named variants (`CreateSyncVariant` or
  `connector.sync_variant.create`) must be targeted **explicitly** — an omitted
  key never means "all variants," and deletion detection is variant-scoped so a
  prune for one variant must never touch a sibling's records.
</Callout>

Named variants can bind an active mapping profile (see
[Data Mapping](/docs/connectors/data-mapping)) so each scope replicates under an
explicit, versioned field contract. Pausing or resuming one variant does not
affect siblings; triggering a paused or archived variant fails closed.

## The sync scheduler [#the-sync-scheduler]

The **sync scheduler** (`connector.sync_schedule.*`,
`docs/connectors/sync-scheduler.md`) is the control plane that decides *when*
tenant/environment sync work becomes due. It sits above variants and run leases:
it materializes explicit due-work items, but execution must still acquire a lease
before mutating anything.

Every materialized due-work item carries an **idempotency key** and a
`required_lease` hash — values that are safe for operators and logs, and that
never contain credentials, tokens, or raw payloads. Scheduler policy controls
frequency, jitter, catch-up bounds, a concurrency hint, and a backpressure group,
and it fails closed: a policy can be blocked or deferred by paused/disabled state,
readiness, entitlement, quota, usage budget, stale dependency evidence, catch-up
caps, or queue backpressure.

<Callout type="info" title="Due work is intent, not permission to mutate">
  A due-work item is a durable *intent* to start one bounded sync scope. It is
  not proof a run may write. The sync run lease plane remains the sole authority
  for concurrent-run exclusion — so the scheduler can safely over-enqueue and let
  the lease plane reject conflicts.
</Callout>

## Sync run leases [#sync-run-leases]

A **sync run lease** (`docs/connectors/sync-run-leases.md`) is a *system-owned*
exclusivity lease over one sync execution scope — tenant, provider config,
connection, sync key, variant, and optional partition. It protects sync starts,
partition claims, cursor writes, record materialization, delete tracking, and
terminal cleanup from concurrent runs that would corrupt the same checkpoint.

These are **not** the SDK-owned [execution locks](/docs/connectors/architecture)
a connector function calls for a short critical section. A lease is a runtime
guard attached to a whole sync run and its lifecycle evidence.

The ordering is the guarantee. Notice the lease is acquired *before* the durable
running row exists and before any provider call — so a losing start can never
write a cursor or materialize a record under a stale owner:

```mermaid
sequenceDiagram
  participant S as Sync start
  participant L as Lease store
  participant P as Provider
  S->>L: acquire scope lease
  alt acquired
    L-->>S: granted with TTL
    S->>S: persist running row
    S->>P: fetch page
    S->>L: refresh before cursor write
    S->>L: release after final mutation
  else conflict
    L-->>S: denied, receipt recorded
    Note over S,P: no provider call, no cursor write
  end
```

Every start builds the lease request *before* persisting a durable running row.
If the lease can't be acquired, the start fails closed — no provider calls, cursor
writes, or record writes proceed under a stale owner. The active run refreshes the
lease before each guarded mutation and releases it only after the final one.
Conflict decisions are recorded as lease receipts with an explicit policy:

| Policy           | Behavior                                                                              |
| ---------------- | ------------------------------------------------------------------------------------- |
| `reject`         | Fail the start with a typed sync-lease conflict.                                      |
| `skip`           | Record a skipped decision.                                                            |
| `defer`          | Record a deferred decision; retry belongs to the scheduler.                           |
| `cancel_replace` | Record a replacement decision; cancellation goes through execution lifecycle control. |
| `allow_disjoint` | Allow only scopes differing by an explicitly safe disjoint partition key.             |

Receipts carry a `scope_hash`, owner hash, run id, and TTL — never credentials or
payload fragments. `connector.sync_run_lease.force_release` exists for break-glass
recovery of a genuinely stale owner, and requires an actor and reason.

## Record retention [#record-retention]

Sync record retention (`docs/connectors/sync-record-retention.md`) governs
Ductor's **cached** payloads. It never deletes data from the upstream provider.
Three states are deliberately distinct:

* **Payload prune** clears cached payload bytes while keeping record identity,
  external id, cursor, hashes, variant, and tombstone state. A pruned record can
  still be active in the provider — readers must treat `payload_available = false`
  as metadata-only, not as a source delete.
* **Hard delete** removes source-deleted cache rows after the persisted
  hard-delete window. It requires break-glass authority (admin / global-admin /
  `connector_retention:break_glass`) plus a single-use, exact approval reference
  whose metadata must match the receipt, scope hash, policy version, and mode.
* **Legal hold** always blocks both prune and hard delete. Retention approvals
  never override it.

<Callout type="warn" title="Retention is bounded and lease-guarded">
  Mutation acquires the exact canonical sync-run lease in `reject` mode and
  re-checks active runs inside the Postgres transaction — there is no caller
  override for an active lease. Prune requests are always bounded by `limit` and
  `batch_size`, an omitted `variant_key` means the protected `base` variant, and a
  re-sync only restores a pruned payload when a fresh provider write is applied.
  The scheduled sweep worker (`connector.sync_record_retention_worker`) is
  default-off and needs explicit `tenant_ids`, interval, limit, and reason.
</Callout>

## Realtime sync channels [#realtime-sync-channels]

**Realtime sync channels** (`connector.realtime_sync.channel.*`,
`docs/connectors/realtime-sync-channels.md`) bind normalized provider webhook
events directly to sync-record writes. They are for providers where webhook
delivery is fast but may be partial, duplicated, delayed, or eventually
consistent — the channel keeps the cache converging without waiting for the next
scheduled sync.

Channel controls never accept or return raw webhook payload bytes: use
`normalized_payload` for mapper-safe data and `raw_payload_ref` for an external
evidence reference. `connector.realtime_sync.preview` always runs with writes
disabled; `connector.realtime_sync.replay` requires an explicit `execute_writes`
flag and reuses the same sync-record write gates as live delivery. When a channel
can't derive an external id or normalized record, its reconcile policy either
schedules bounded repair work or quarantines the receipt — it never drops events
silently.

## Where to go next [#where-to-go-next]

* [Data Mapping](/docs/connectors/data-mapping) — how provider payloads become the normalized records a sync stores.
* [Triggers & Polling](/docs/connectors/triggers-and-polling) — the webhook and poll surfaces realtime channels and poll cursors build on.
* [Policies, Quotas & Audit](/docs/connectors/policies-quotas-audit) — quota facts and execution admission that gate sync starts.