The Sync Engine

Cached provider-record replication — sync installations, variants, the scheduler, run leases, retention, and realtime channels.

Actions reach out on demand and triggers push events in, but some workflows need a local, queryable copy of a provider's records — every HubSpot contact, every Shopify order — kept fresh in the background. That is what the sync engine does: it replicates provider records into schema-versioned, normalized SyncRecords that routing, unified-object reads, and strategy feature extraction can read without a live provider call.

This page covers the control planes that make replication safe: variants, scheduling, run exclusivity, cache retention, and realtime binding. For how provider payloads become normalized records, see Data Mapping.

Sync installations and variants

A sync installation is a tenant-owned binding from one registered sync spec (hubspot.contacts) to a connection. Underneath an installation, work runs in a sync variant (domain/connector, docs/connectors/sync-variants.md) — a named execution and cache namespace with its own cursor, checkpoint namespace, record cache, run history, frequency override, and lifecycle status.

Variants let one installation replicate several independently-isolated scopes — a mailbox, folder, market, repository, or segment — without collisions. Because the record uniqueness key includes the variant, base and named variants can hold the same upstream external_id side by side.

The base variant is protected

Every installation has a base variant. Public API callers may omit variant_key; Ductor resolves the request to base and echoes variant_key = "base" back. Operators cannot create, rename, archive, or delete base. Named variants (CreateSyncVariant or connector.sync_variant.create) must be targeted explicitly — an omitted key never means "all variants," and deletion detection is variant-scoped so a prune for one variant must never touch a sibling's records.

Named variants can bind an active mapping profile (see Data Mapping) so each scope replicates under an explicit, versioned field contract. Pausing or resuming one variant does not affect siblings; triggering a paused or archived variant fails closed.

The sync scheduler

The sync scheduler (connector.sync_schedule.*, docs/connectors/sync-scheduler.md) is the control plane that decides when tenant/environment sync work becomes due. It sits above variants and run leases: it materializes explicit due-work items, but execution must still acquire a lease before mutating anything.

Every materialized due-work item carries an idempotency key and a required_lease hash — values that are safe for operators and logs, and that never contain credentials, tokens, or raw payloads. Scheduler policy controls frequency, jitter, catch-up bounds, a concurrency hint, and a backpressure group, and it fails closed: a policy can be blocked or deferred by paused/disabled state, readiness, entitlement, quota, usage budget, stale dependency evidence, catch-up caps, or queue backpressure.

Due work is intent, not permission to mutate

A due-work item is a durable intent to start one bounded sync scope. It is not proof a run may write. The sync run lease plane remains the sole authority for concurrent-run exclusion — so the scheduler can safely over-enqueue and let the lease plane reject conflicts.

Sync run leases

A sync run lease (docs/connectors/sync-run-leases.md) is a system-owned exclusivity lease over one sync execution scope — tenant, provider config, connection, sync key, variant, and optional partition. It protects sync starts, partition claims, cursor writes, record materialization, delete tracking, and terminal cleanup from concurrent runs that would corrupt the same checkpoint.

These are not the SDK-owned execution locks a connector function calls for a short critical section. A lease is a runtime guard attached to a whole sync run and its lifecycle evidence.

The ordering is the guarantee. Notice the lease is acquired before the durable running row exists and before any provider call — so a losing start can never write a cursor or materialize a record under a stale owner:

Every start builds the lease request before persisting a durable running row. If the lease can't be acquired, the start fails closed — no provider calls, cursor writes, or record writes proceed under a stale owner. The active run refreshes the lease before each guarded mutation and releases it only after the final one. Conflict decisions are recorded as lease receipts with an explicit policy:

Policy	Behavior
`reject`	Fail the start with a typed sync-lease conflict.
`skip`	Record a skipped decision.
`defer`	Record a deferred decision; retry belongs to the scheduler.
`cancel_replace`	Record a replacement decision; cancellation goes through execution lifecycle control.
`allow_disjoint`	Allow only scopes differing by an explicitly safe disjoint partition key.

Receipts carry a scope_hash, owner hash, run id, and TTL — never credentials or payload fragments. connector.sync_run_lease.force_release exists for break-glass recovery of a genuinely stale owner, and requires an actor and reason.

Record retention

Sync record retention (docs/connectors/sync-record-retention.md) governs Ductor's cached payloads. It never deletes data from the upstream provider. Three states are deliberately distinct:

Payload prune clears cached payload bytes while keeping record identity, external id, cursor, hashes, variant, and tombstone state. A pruned record can still be active in the provider — readers must treat payload_available = false as metadata-only, not as a source delete.
Hard delete removes source-deleted cache rows after the persisted hard-delete window. It requires break-glass authority (admin / global-admin / connector_retention:break_glass) plus a single-use, exact approval reference whose metadata must match the receipt, scope hash, policy version, and mode.
Legal hold always blocks both prune and hard delete. Retention approvals never override it.

Retention is bounded and lease-guarded

Mutation acquires the exact canonical sync-run lease in reject mode and re-checks active runs inside the Postgres transaction — there is no caller override for an active lease. Prune requests are always bounded by limit and batch_size, an omitted variant_key means the protected base variant, and a re-sync only restores a pruned payload when a fresh provider write is applied. The scheduled sweep worker (connector.sync_record_retention_worker) is default-off and needs explicit tenant_ids, interval, limit, and reason.

Realtime sync channels

Realtime sync channels (connector.realtime_sync.channel.*, docs/connectors/realtime-sync-channels.md) bind normalized provider webhook events directly to sync-record writes. They are for providers where webhook delivery is fast but may be partial, duplicated, delayed, or eventually consistent — the channel keeps the cache converging without waiting for the next scheduled sync.

Channel controls never accept or return raw webhook payload bytes: use normalized_payload for mapper-safe data and raw_payload_ref for an external evidence reference. connector.realtime_sync.preview always runs with writes disabled; connector.realtime_sync.replay requires an explicit execute_writes flag and reuses the same sync-record write gates as live delivery. When a channel can't derive an external id or normalized record, its reconcile policy either schedules bounded repair work or quarantines the receipt — it never drops events silently.

Where to go next

Data Mapping — how provider payloads become the normalized records a sync stores.
Triggers & Polling — the webhook and poll surfaces realtime channels and poll cursors build on.
Policies, Quotas & Audit — quota facts and execution admission that gate sync starts.