Operations

Release Verification

Verify a fixed Ductor source revision with full release gates, executed critical flows, and matching binary identity.

A release needs evidence for the exact source revision and artifact being promoted. Record what ran, whether it passed, and which environment it exercised. A package-level pass can coexist with skipped tests, and a focused regression run covers less than the full release gate.

Run the commands on this page from the Ductor Go repository, not the frontend workspace. Use the repository-managed cache through Make targets or scripts/dev-cache.sh; all local worktrees share that bounded cache.

Choose the right gate

CommandCoverage
make ciFast development gate for changed packages
make ci-releaseFull static analysis, generated/contract checks, unit-level race and coverage tests, and binary build
make test-integration-storageReal PostgreSQL storage and migration integration tests
make test-integration-workflowWorkflow integration tests
make test-e2e-criticalRequired routing, durable-event, replay, restart, and journey flows
make ci-release-fullci-release plus storage, workflow, and critical end-to-end gates

make validate-config, make docs-check, and make openapi-check verify configuration and documentation/API consistency. They do not replace the test, lint, integration, or build gates. Focused tests are useful during development; run the full gate again after the final source change before assigning release evidence to that revision.

Verify with isolated dependencies

Start from a clean checkout of the intended revision, with the repository's required toolchain installed and Docker running. The hermetic wrapper creates isolated TimescaleDB/PostgreSQL and Dragonfly containers, applies the schema baseline, supplies test connection settings, and removes its containers on exit.

git status --short
git rev-parse HEAD
make validate-config
./scripts/dev-cache.sh run -- ./scripts/run-hermetic-tests.sh make ci-release-full
make verify-build-identity
./bin/ductor version --output=json

Preserve the full command output, exit status, test and coverage artifacts, and the reported dependency/toolchain versions with the release record. A failed prerequisite or unavailable database is a failed verification run. Resolve the cause and rerun the affected gate; do not substitute a skipped suite or a lint exclusion for passing evidence.

make verify-build-identity builds a disposable binary and verifies that its embedded commit equals the full source revision. Separately inspect the bin/ductor artifact built by the release gate: its version JSON exposes version, commit, build_time, and dirty. The commit must match the verified revision and dirty must be false. Record the digest of the image or binary actually promoted as well; a matching commit does not identify an artifact on its own.

Critical tests must execute

make test-e2e-critical requires these ten test cases to run and pass:

Test packageRequired caseWhat it exercises
tests/e2eTestE2ESuite/TestDecisionRecordingPersisted routing decisions
tests/e2eTestE2ESuite/TestCatchallDelayE2EDelayed catchall behavior
tests/e2eTestE2ESuite/TestCatchallImmediateE2EImmediate catchall behavior
tests/e2eTestE2ESuite/TestCatchallWithPrimaryAvailableE2EPrimary routing when a catchall is present
tests/e2eTestE2ESuite/TestRouteEmitsDurableEventsPersisted routing request and assignment events with tenant isolation
tests/e2eTestE2ESuite/TestEventSourcedReplayEvent-backed replay without changing the live decision or its history
tests/e2e/restartmatrixTestRestartAfterRoutingLifecycleCommitDrainsEffectIntentsExactlyOnceRecovery of committed routing effects after restart
tests/e2e/restartmatrixTestJourneyE2E_MultiPool_CrashRestart_ExactlyOnceAcrossHopsMulti-pool journey recovery across a crash
tests/e2e/restartmatrixTestJourneyE2E_BudgetBreachEmitsTypedEffectIntentNotSpawnTyped budget-breach effects without spawning a child
tests/e2e/restartmatrixTestRoutingEnrichStep_RealPGConnectorLedgerReplayAndFilterCELVisibilityConnector enrichment, durable ledger replay, and filter visibility

The test runner consumes Go test events and matches the exact package and test name. It rejects a required case that is missing, skipped, failed, or unfinished, even when the underlying Go command exits successfully. A passing parent suite does not stand in for a required child test. Each successful requirement produces a REQUIRED PASS line with the number of completed runs.

For a focused investigation, the same runner accepts repeatable --require-test options before --. The package is the full import path and the required name is exact; the Go -run expression after -- selects what to run.

./scripts/dev-cache.sh run -- ./scripts/run-hermetic-tests.sh \
  go run ./tools/testwatch \
  --require-test 'github.com/ductor-io/ductor/tests/e2e::TestE2ESuite/TestRouteEmitsDurableEvents' \
  -- -race -count=1 -timeout=10m ./tests/e2e \
  -run '^TestE2ESuite$/^TestRouteEmitsDurableEvents$'

Use the Make target for the complete critical set. If a case is renamed, update both the selection expression and the exact requirement in the same change. The requirement list is part of the release contract.

Interpret durable-event and replay evidence

The routing checks use PostgreSQL-backed event persistence and read the stored history. They wait for routing's asynchronous completion work before asserting that request and assignment events exist and belong to the expected tenant. Observing only an in-memory callback does not establish durable delivery.

Replay checks load the recorded decision or routable, retrieve its event history, and run the simulation through an isolated routing graph. The simulation may write its own audit record, while the original decision and live routing event history remain unchanged. This verifies that flow's isolation; it does not establish that every historical dependency can be reconstructed at any timestamp. See Events & Event Sourcing.

Complete operational acceptance

Keep source verification and deployment observations together in the release record:

The local release suite proves the code paths and dependencies it exercised. Production credentials, external provider behavior, deployment topology, and recovery procedures require evidence from the intended environment. Keep failures and unsupported assumptions visible in the release decision until they are resolved and verified.