Connectors
Explaineralso code

Connector Architecture

The registries, the ConnectionService, action dispatch, action semantics, and the durable attempt and async-job execution models.

The connector subsystem is the machinery behind the action inventory — the governed, typed actions the executed stage of the clearing lifecycle draws on. It is built from a small set of registries and one central executor. Everything else — workflow steps, the management API, routing hooks, and agent tool calls — is a caller that funnels into that executor. This page traces the machinery from registration through dispatch to the durable attempt model.

The two registries

A provider is described by two registries, both singletons wired via fx in cmd/ductor/fx_connector.go (ConnectorRegistryModule):

  • ProviderRegistry (application/connector/provider_registry.go) maps a providerKey → a Provider definition. Get(key) returns the provider or ErrProviderNotFound. Keys must match ^[a-z][a-z0-9_]*$. The registry keeps a byCategory index so the catalog can list providers by category.
  • ActionRegistry (application/connector/action_registry.go) maps a (providerKey, actionKey) pair → an ActionSpec. Get(providerKey, actionKey) returns a deep clone of the spec or ErrActionNotFound.

There are four more registries alongside these — TriggerRegistry, ModelRegistry, MapperRegistry, and ResolverRegistry — all built on the same generic specRegistry[S], whose keys allow multi-segment names like connector.sync.lookup_record.

Registration goes through one interface

Providers never touch the registries directly. They call the Registrar interface (application/connector/registrar.go), which has one method per registry — Provider, Action, Trigger, Model, Mapper, Resolver, plus bulk variants. ActionRegistry.Register validates the spec's semantics and stamps its derived annotations before storing it, so an invalid action fails loudly at startup, not at dispatch.

The Provider definition

A Provider (domain/connector/provider.go) is a compiled, versioned description. The load-bearing fields:

FieldPurpose
Key, DisplayName, DescriptionIdentity and catalog display.
BaseURLA redacted authoring template — no tokens, headers, or secrets. Used to build the generic API-call action.
Categories []CategoryCatalog taxonomy (see The Catalog).
AuthTypes []AuthTypeWhich auth types this provider supports; the first is the default the builder suggests.
CredentialSchemaThe connection form for secret-text / basic / custom auth (nil for OAuth2).
ActionKeys, TriggerKeys, ModelKeys, MapperKeysFully-qualified capability keys the provider registers.
Hooks ProviderHooksProvider-level Go callbacks: TestAuth, ExtractIdentity, RefreshToken, InstallAppWebhook, UninstallAppWebhook, ResolveAppWebhookConnection.
OAuth2Config, TwoStepConfig, OAuth1Config, JWTConfig, MCPConfigPer-auth-type config, non-nil only for the auth types the provider uses.
ReleaseStage, Deprecated, Compliance, RateLimit, QuotaLifecycle and governance metadata.

The ActionSpec

An ActionSpec (domain/connector/action.go) is "the complete, immutable description of one operation... registered once at startup, read-only thereafter." Key fields:

  • Key (fully-qualified, e.g. hubspot.create_contact), Provider, Auth.
  • Properties []Property — the input form (see Building a Provider).
  • OutputSchema json.RawMessage — a JSON Schema for the result.
  • Semantics ActionSemantics — the canonical effect/idempotency/retry contract.
  • Execute ActionExecuteFunc — the function that does the work.
  • DefaultTimeout, DefaultFailurePolicy, Pagination, RequiredScopes, Quota.

The execute signature is uniform across every action:

type ActionExecuteFunc func(ctx context.Context, in ActionInput) (ActionOutput, error)

ActionInput carries the resolved Auth *AuthContext, the caller Args, and the full execution context (TenantID, ExecutionID, RunID, StepRef, AttemptNo, ExecutionSurface, BudgetDeadline, …). ActionOutput carries a Result (json.RawMessage), an optional normalized result, and — critically — an EffectPatch.

Actions return effects, they never mutate shared state

An action's output "must contain an effect patch; must not mutate shared state directly." An EffectPatch is a list of EffectOp{Op, Path, Value} where Op is one of set, merge_object, emit_request, or search_attributes_merge. The workflow coordinator applies these ops transactionally when it folds the attempt result back into run state. This is what keeps dispatch idempotent and the coordinator the sole writer of run state.

Action semantics

Every action declares an ActionSemantics (domain/connector/action_semantics.go) — the machine-readable contract that governs whether the engine may retry it, whether it needs approval, and how to make it idempotent. The dimensions:

DimensionValues
EffectClassread, create, update, upsert, delete, send, custom_write, multi_step, unknown
MutationScopenone, single_record, collection, provider_account, external_delivery, custom, unknown
DestructiveClassnone, soft_delete, hard_delete, overwrite, irreversible_send, unknown_destructive
IdempotencyKindnaturally_idempotent, provider_idempotency_key, client_supplied_external_key, ductor_ledger_only, not_idempotent, unknown
RetrySafetysafe_retry, retry_before_provider_call, retry_with_idempotency_key, no_automatic_retry, manual_review

Validate() enforces the relationships between them — a read cannot be destructive and must be naturally idempotent and safe to retry; a destructive action requires approval; a non-idempotent action cannot be marked safe-retry. The shared gate ValidateActionExecutionPolicy(sem, surface, attemptNo, key, policy) runs before side effects on the mcp_tool, async_action_job, and unified_operation surfaces, so a retry of a not-idempotent action without a key is refused rather than silently duplicating a downstream write.

This is what makes the inventory governed rather than merely callable: because every action's mutation class, idempotency, and retry safety are declared, the runtime — and any worker it dispatches to, human or agent — knows what is safe to retry and what needs a human, without reading the provider's source.

Dispatch: the central executor

All three call surfaces converge on ActionExecutorService.ExecuteActionWithRequest (application/connector/action_executor.go), which takes an ActionExecutionRequest and does, in order:

  1. spec, err := actions.Get(providerKey, actionKey) — resolve the action, or fail closed.
  2. Simulator gate — a simulator-only provider is not routable unless explicitly allowed.
  3. Merge required scopes and resolve the execution budget/deadline.
  4. StartExecution on the execution ledger.
  5. The install gate (is the provider config actually set up?).
  6. Resolve auth via ConnectionService.Resolve (see below).
  7. Run the interceptor chain (rate limit, audit, circuit breaker, …).
  8. Call spec.Execute, then the output mapper projection.
  9. FinishExecution on the ledger.

The interceptor chain is contributed through the connector_interceptors fx option group, which is the same extension pattern routing strategies use.

Resolving the connection

ConnectionService (application/connector/connection_service.go) has two methods: Get (the raw Connection) and Resolve (a per-attempt, decrypted AuthContext). The executor calls Resolve at dispatch time. That flow — cache, decrypt, per-auth decoration, OAuth2 refresh — is detailed on the Authentication page.

The durable attempt model

The synchronous workflow path (application/workflow/dag/worker_connector_action.go, StepWorker.dispatchConnectorAction) calls the same ExecuteActionWithRequest with ExecutionSurface = workflow_action and returns a StepExecutionResult tagged with an AttemptID. The coordinator applies the returned EffectPatch when it commits the tick. Because dispatch is at-least-once, the same action can execute more than once across a crash; the effect patch and any provider-side idempotency key are what make that safe. See Coordinator & Step Workers and Idempotency & Exactly-Once.

The async action-job model

For work that shouldn't block a synchronous request or a workflow tick, Ductor has a separate durable job surface (ExecutionSurface = async_action_job), modelled by ActionJob (domain/connector/action_job.go) and driven by the ConnectorActionJobService (application/connector/action_job_service.go).

A job moves through an explicit state machine:

cancel_requested attempts exhausted queued running succeeded failed canceled expired dead_lettered

The mechanics that make it safe:

  • Idempotent submission. SubmitConnectorActionJob computes an IdempotencyScope from the tenant, provider, action, connection, and route, then looks up any existing job by (scope, idempotencyKey). A resubmission with the same input digest returns the existing job (Deduplicated=true); a resubmission with a different digest under the same key returns ErrConflict.
  • Retries require a key. A job with MaxAttempts > 1 is rejected at creation unless it carries an idempotency key, and ScheduleRetry errors without one. On a retryable failure the job returns to queued with a backoff; once Attempts >= MaxAttempts it is dead_lettered.
  • Lease fencing. A worker leases a due job (MarkLeased), executes it, and writes the terminal result through UpdateConnectorActionJobFenced(job, leaseOwner, running). If the fence fails — the lease was reclaimed and re-leased by another worker — the worker discards its result rather than double-finalizing.
  • Budget inheritance. The persisted DeadlineAt is stamped onto the execution context so a deferred provider call still respects the originating run's deadline.

The worker itself (cmd/ductor/fx_connector_action_job_worker.go) is default-off: it runs only when connector.async_action_job_worker.enabled is set with a positive interval, batch limit, lease duration, and a non-empty tenant list. Each tick reclaims expired leases, expires due jobs, then runs due jobs up to the batch limit.

Both surfaces, one executor

The synchronous workflow step and the async job are two execution surfaces over the same ExecuteActionWithRequest. The workflow path carries the coordinator's AttemptID; the async job carries its own Attempts count and idempotency key. Both emit EffectPatch ops and both flow through the same execution ledger, so observability is uniform regardless of how the action was dispatched.

Governed proxy requests

Not every provider call is worth modeling as a stable action. For one-off HTTP calls, ProxyConnector (docs/connectors/proxy-requests.md) runs the request through the full connector boundary — tenant provider policy, marketplace install gates, direct/route connection selection, auth injection, egress policy, circuit breakers, interceptors, metrics, audit, and request/response limits. It is not raw HTTP: endpoint must be a relative path beginning with /, and the legacy provider.custom_api_call action is now just an action-shaped wrapper over the same governed engine.

  • Direct mode (connection_mode: direct) names a connection_id.
  • Route mode (connection_mode: route, optional connection_route_provider) lets the routing interceptor select the concrete connection and stamp resolved_connection_id in the response.

base_url_override is denied by default

A proxy request cannot point itself at an arbitrary origin: base_url_override is denied unless a provider or connection policy explicitly allows it. This is what keeps a proxy request from degrading into arbitrary raw HTTP egress. Unlike custom_api_call, the proxy keeps non-2xx provider responses inside the response envelope so callers can inspect provider validation errors.

The connector execution runtime

Four domain models back the observability, control, and safety of every execution, across all surfaces:

  • Execution ledger (ExecutionRecord, docs/connectors/execution-ledger.md) is the canonical, redacted operational history for all connector activity — direct ExecuteAction, workflow actions, sync runs, routing hooks, proxy requests, lifecycle/MCP operations, and mapper executions. Each row has a stable execution_id, and secret-shaped keys (authorization, token, refresh_token, client_secret, …) are stripped at any nesting depth.

    The ledger is observational, not the dedupe boundary

    The execution ledger records what happened; it is not the idempotency boundary. Side-effect deduplication remains the workflow idempotency table eec_connector_action_invocation (the eec_ prefix marks Ductor's durable Enterprise Eventing Core tables). Never use the ledger as a dedupe key — ExecuteActionResponse and ProxyConnectorResponse expose execution_id only so you can pivot into the execution/log APIs for triage.

  • Execution lifecycle control (ConnectorExecutionLifecycle, docs/connectors/execution-lifecycle.md) is a per-attempt record for one connector execution unit (async attempt, function invocation, sync, mapper). Statuses are explicit: queued, starting, running, interrupting, abort_requested, aborting, cleanup_running, then a terminal succeeded, failed, cancelled, timed_out, killed, orphaned, or cleanup_failed. Cancel stops queued/running work cooperatively; interrupt finishes the current safe operation then stops; kill is reserved for deadline or worker-loss cases — none claims to undo provider side effects that already happened. Running attempts record heartbeats, and orphan sweeps mark an attempt orphaned when the lease expires without one.

  • Execution locks (ConnectorExecutionLock, docs/connectors/execution-locks.md) are short-lived, non-blocking critical-section leases a connector function acquires with ductor.tryAcquireLock({ key, ttlMs }) (TTL bounded 1s–15min). They protect a small provider-scoped mutation, cursor rebuild, or webhook-dedupe window — they are not routing admission, and connector code never chooses the lock owner (Ductor derives it from the trusted execution context). The runner calls releaseAllLocks() in cleanup after both success and failure.

  • Async action jobs (ActionJob) run long-running actions off the synchronous path with the same lifecycle control. The worker (cmd/ductor/fx_connector_action_job_worker.go) is default-off (connector.async_action_job_worker.enabled); see the async action-job model above.

Where to go next

Connectors

How Ductor reaches the outside world — providers, actions, triggers, and connections — the governed action inventory the execute stage draws on.

Action Authentication & Credentials

Every connector auth type, how AuthContext is resolved at dispatch, and how credentials are AEAD-encrypted at rest with XChaCha20-Poly1305, BYOK, and key rotation.