Release Verification
Verify a fixed Ductor source revision with full release gates, executed critical flows, and matching binary identity.
A release needs evidence for the exact source revision and artifact being promoted. Record what ran, whether it passed, and which environment it exercised. A package-level pass can coexist with skipped tests, and a focused regression run covers less than the full release gate.
Run the commands on this page from the Ductor Go repository, not the frontend
workspace. Use the repository-managed cache through Make targets or
scripts/dev-cache.sh; all local worktrees share that bounded cache.
Choose the right gate
| Command | Coverage |
|---|---|
make ci | Fast development gate for changed packages |
make ci-release | Full static analysis, generated/contract checks, unit-level race and coverage tests, and binary build |
make test-integration-storage | Real PostgreSQL storage and migration integration tests |
make test-integration-workflow | Workflow integration tests |
make test-e2e-critical | Required routing, durable-event, replay, restart, and journey flows |
make ci-release-full | ci-release plus storage, workflow, and critical end-to-end gates |
make validate-config, make docs-check, and make openapi-check verify
configuration and documentation/API consistency. They do not replace the test,
lint, integration, or build gates. Focused tests are useful during development;
run the full gate again after the final source change before assigning release
evidence to that revision.
Verify with isolated dependencies
Start from a clean checkout of the intended revision, with the repository's required toolchain installed and Docker running. The hermetic wrapper creates isolated TimescaleDB/PostgreSQL and Dragonfly containers, applies the schema baseline, supplies test connection settings, and removes its containers on exit.
git status --short
git rev-parse HEAD
make validate-config
./scripts/dev-cache.sh run -- ./scripts/run-hermetic-tests.sh make ci-release-full
make verify-build-identity
./bin/ductor version --output=jsonPreserve the full command output, exit status, test and coverage artifacts, and the reported dependency/toolchain versions with the release record. A failed prerequisite or unavailable database is a failed verification run. Resolve the cause and rerun the affected gate; do not substitute a skipped suite or a lint exclusion for passing evidence.
make verify-build-identity builds a disposable binary and verifies that its
embedded commit equals the full source revision. Separately inspect the
bin/ductor artifact built by the release gate: its version JSON exposes
version, commit, build_time, and dirty. The commit must match the verified
revision and dirty must be false. Record the digest of the image or binary
actually promoted as well; a matching commit does not identify an artifact on
its own.
Critical tests must execute
make test-e2e-critical requires these ten test cases to run and pass:
| Test package | Required case | What it exercises |
|---|---|---|
tests/e2e | TestE2ESuite/TestDecisionRecording | Persisted routing decisions |
tests/e2e | TestE2ESuite/TestCatchallDelayE2E | Delayed catchall behavior |
tests/e2e | TestE2ESuite/TestCatchallImmediateE2E | Immediate catchall behavior |
tests/e2e | TestE2ESuite/TestCatchallWithPrimaryAvailableE2E | Primary routing when a catchall is present |
tests/e2e | TestE2ESuite/TestRouteEmitsDurableEvents | Persisted routing request and assignment events with tenant isolation |
tests/e2e | TestE2ESuite/TestEventSourcedReplay | Event-backed replay without changing the live decision or its history |
tests/e2e/restartmatrix | TestRestartAfterRoutingLifecycleCommitDrainsEffectIntentsExactlyOnce | Recovery of committed routing effects after restart |
tests/e2e/restartmatrix | TestJourneyE2E_MultiPool_CrashRestart_ExactlyOnceAcrossHops | Multi-pool journey recovery across a crash |
tests/e2e/restartmatrix | TestJourneyE2E_BudgetBreachEmitsTypedEffectIntentNotSpawn | Typed budget-breach effects without spawning a child |
tests/e2e/restartmatrix | TestRoutingEnrichStep_RealPGConnectorLedgerReplayAndFilterCELVisibility | Connector enrichment, durable ledger replay, and filter visibility |
The test runner consumes Go test events and matches the exact package and test
name. It rejects a required case that is missing, skipped, failed, or unfinished,
even when the underlying Go command exits successfully. A passing parent suite
does not stand in for a required child test. Each successful requirement produces
a REQUIRED PASS line with the number of completed runs.
For a focused investigation, the same runner accepts repeatable
--require-test options before --. The package is the full import path and the
required name is exact; the Go -run expression after -- selects what to run.
./scripts/dev-cache.sh run -- ./scripts/run-hermetic-tests.sh \
go run ./tools/testwatch \
--require-test 'github.com/ductor-io/ductor/tests/e2e::TestE2ESuite/TestRouteEmitsDurableEvents' \
-- -race -count=1 -timeout=10m ./tests/e2e \
-run '^TestE2ESuite$/^TestRouteEmitsDurableEvents$'Use the Make target for the complete critical set. If a case is renamed, update both the selection expression and the exact requirement in the same change. The requirement list is part of the release contract.
Interpret durable-event and replay evidence
The routing checks use PostgreSQL-backed event persistence and read the stored history. They wait for routing's asynchronous completion work before asserting that request and assignment events exist and belong to the expected tenant. Observing only an in-memory callback does not establish durable delivery.
Replay checks load the recorded decision or routable, retrieve its event history, and run the simulation through an isolated routing graph. The simulation may write its own audit record, while the original decision and live routing event history remain unchanged. This verifies that flow's isolation; it does not establish that every historical dependency can be reconstructed at any timestamp. See Events & Event Sourcing.
Complete operational acceptance
Keep source verification and deployment observations together in the release record:
- The full source revision, clean build identity, artifact digest, commands, exit statuses, and test artifacts.
- The target environment's configuration and explicit migration result.
- Readiness, operational health, and applicable canary results after rollout.
- The backup and restore evidence and promotion or rollback procedure appropriate to the change.
The local release suite proves the code paths and dependencies it exercised. Production credentials, external provider behavior, deployment topology, and recovery procedures require evidence from the intended environment. Keep failures and unsupported assumptions visible in the release decision until they are resolved and verified.