Simulate a lead marketplace

A lead marketplace connects publishers to buyers with independent filters, budgets, and delivery requirements. Ductor's optional ping/post service can solicit bids using partial lead data, select an exclusive buyer or several shared buyers, and deliver the full lead to winners.

This guide describes a tested marketplace slice. Boberdoo's buyer matching and distribution logic provide the comparison use cases. A full clone also needs publisher and buyer portals, pricing and billing operations, reporting, consent handling, and delivery reconciliation.

Follow one lead through the marketplace

1. Receive a publisher's inquiry with stable source and lead references.
2. Match geography and product to a bounded buyer candidate set.
3. Exclude inactive buyers and buyers without enough remaining budget.
4. Ping candidates with partial data that excludes configured PII fields.
5. Collect qualifying bids and explicit declines before closing the auction.
6. Select one buyer for exclusive delivery or top-N buyers for shared delivery.
7. Post the full lead to winners and record each buyer's delivery result.
8. Reconcile rejection, asynchronous acceptance, and uncertain transport results.
9. Correlate later business outcomes to the original lead and buyer decision.

Keep matching, auction selection, delivery, and settlement as observable stages. An auction winner does not prove delivery, and delivery does not prove a closing. The routing-market model and its settlement surfaces are separate contracts; closing a ping/post auction does not by itself exercise those settlement guarantees.

Choose the distribution policy deliberately

Before selecting a policy, read eligibility, pool management, and outcome feedback. A buyer's qualification, budget, bid, and historical conversion performance describe different constraints.

Preserve uncertain delivery

Per-winner post results expose four states:

Authenticated buyer callbacks can resolve pending or unknown to a terminal result. The ordinary pending callback timeout does not automatically accept unknown. Existing charges remain until reconciliation according to the owning pricing flow; transport uncertainty does not justify either blind redelivery or an automatic refund.

Expose unknown results to operations with the lead, session, buyer, and delivery attempt references. A dedicated business reconciliation workflow still needs to establish what the buyer received. Do not label an unknown result as a successful delivery in a publisher portal.

Reproduce the simulation

From the Ductor backend repository:

./scripts/dev-cache.sh run -- go test -race ./modules/services/pingpost/... -count=1
./scripts/dev-cache.sh run -- go test ./modules/services/pingpost -run TestBoberdoo -bench BenchmarkBoberdoo -benchtime=10x -count=1 -v -benchmem

The fixture calls the production ping/post service with synthetic buyer adapters and in-memory repositories. It verifies active and funded buyer filtering, declines, exclusive and shared winner counts, ping PII suppression, buyer rejection, solicitation failure, and uncertain-delivery callbacks.

Interpret the scale result

The requested target was a 10,000-buyer catalog and 1,000 leads per second. The latest local measurements on an Apple M1 Ultra include durable attempt bookkeeping and canonical pricing lookups in the in-memory fixture:

The bounded run excludes production candidate matching, HTTP, PostgreSQL, Redis and ledger mutations. It uses in-memory durable-receipt semantics and synthetic buyer senders. It is not a production throughput certification. Cumulative allocation was 19.779 GB over the minute; the ending Go heap was 17.421 MB. Neither figure measures peak process memory or infrastructure cost.

Candidate IDs have a persisted sorted, distinct representation. Binary search replaces a linear scan for each submitted bid. The earlier membership-only optimization improved the 10,000-candidate lifecycle from 140.9 ms to 25.3 ms; the latest 63.245 ms includes delivery bookkeeping, scoped lead snapshots and canonical pricing lookups. Production currently reads pricing per candidate; these in-memory lookups do not establish database matching capacity. Nonfinite prices are rejected and duplicate bid conflicts remain observable.

Preserve attempts across recovery

Each recipient has a persisted attempt identity for ping and post. A worker records an uncertain attempt before external I/O. Another worker skips both completed and uncertain attempts, so a restart cannot authorize blind resend. Independent buyer evidence is needed to resolve uncertainty; these receipts do not establish exactly-once effects at an external buyer.

Real PostgreSQL regression tests separately exercise concurrent dispatch claimants, duplicate bids, budget races, stale attempt acknowledgments, immutable original charges, and rejection refunds. Result, refund and receipt updates commit together. Those correctness tests are separate from the scale measurement above.

Production capacity still requires real candidate selection, HTTP buyer latency, PostgreSQL/Redis load, and sustained saturation testing. Track offered, started, completed, failed and queued work independently so delayed admission cannot make latency appear artificially low.

Configure and verify real buyer delivery

Author buyer URLs through authenticated admin PUT and GET /api/pingpost/recipients/{recipient_id}/delivery-config. These settings are persisted independently of pricing. Paid auction admission reads saved buyer pricing, excludes disabled or underfunded buyers, and fails closed when that configuration cannot be read. Enable the production sender with its callback origin, private signing secret, timeout, and governed egress policy. Buyer requests carry the immutable attempt UUID as Idempotency-Key; they do not receive the service credential.

Operators can inspect GET /api/pingpost/sessions/{session_id}/deliveries and resolve uncertain delivery through POST /api/pingpost/sessions/{session_id}/deliveries/{recipient_id}/reconcile. Reconciliation requires the original attempt identity and independent buyer evidence. Public receipts omit callback tokens; rejection, refund, and the terminal receipt commit together.

The self-hosted laboratory passed seven functional scenarios using actual HTTP buyers and PostgreSQL: synchronous acceptance, asynchronous callback acceptance, synchronous rejection, and lost acknowledgments reconciled as accepted or rejected, timeout-only auction closure, and funded/disabled/underfunded buyer admission. Each auction created one charge; rejected outcomes created one refund. Excluded buyers received no full lead and incurred no charge. Duplicate terminal operations conflicted. All 14 buyer endpoint configurations survived a controlled service restart. This functional evidence is separate from the synthetic throughput result above.

Run python3 scripts/validate-pingpost-lab.py in the backend repository with an owned laboratory and a private, short-lived admin credential. Its report keeps fixture identities and assertion results without credentials or callback tokens. The backend's docs/developers/pingpost-live-validation.md describes setup and restart verification.