Outbox pattern

Purpose

When something that happened in one service must land in another (a payment settled, so credit is owed; a key was minted, so the gateway must honour it; a balance crossed a line, so a customer must be told), the obligation is written as a row in the same transaction as the thing that caused it, and delivered afterwards by a separate, signed, retried HTTP call. The invariant that makes it trustworthy is that after the commit the obligation is exactly as durable as the fact, nothing between the two services can lose it, and a delivery that will never succeed becomes a row a human can SELECT rather than silence.

This foundation fixes the three verbs (ENQUEUE, DELIVER, STUCK), the row shape every outbox shares, the retry policy, where idempotency lives (on the receiver, in its database) and what the caller may and may not do around the commit. It was measured on the three Labs outboxes that write into Prima's prepaid ledger and key store (decision D2: prepaid credit in Prima's ledger is the one AI ledger, Sparks are display) and on Prima's own outbox in the opposite direction, which is unmerged.

When to use

Include it from any backend recipe where a state change in this service obliges another service to change state, and the two can fail independently. Trigger phrases: "after the webhook confirms", "tell Prima", "the key has to work on the gateway", "the customer paid but has no credit", "retry later", "fire and forget", "what if Prima is down", "how do I find the ones that did not go through".

Use the neighbouring recipe instead when: the other side is a database you own in the same transaction (a plain write, no outbox); the call is a read with no obligation attached (a normal client with a timeout); the work is periodic rather than caused (_shared/scheduler-and-background-work); the receiver is a third party whose retry policy is theirs (Payrexx redelivers its own webhook; the settlement receiver in commerce-payments makes that path idempotent and then enqueues into this pattern). The signing of the delivery call is _shared/hmac-seam-pattern (S10) and is not repeated here.

Stack assumptions

Postgres with SELECT ... FOR UPDATE SKIP LOCKED and partial unique indexes; SQLAlchemy 2.0 ORM sessions on Labs (SQLAlchemy==2.0.43, backend/requirements.txt at fd8558f) and text() SQL on Prima (sqlalchemy==2.0.36 at cd2c248); httpx==0.28.1 on both; APScheduler==3.11.0 on Labs for the drain; pydantic==2.12.0 for Labs settings. Python 3.12 or newer.

Foundations assumed: S10 for the signed delivery call and its 4xx rule; _shared/scheduler-and-background-work (S12) for the leader-only scheduler the drains register on; _shared/alembic-migrations (S8) for the dated migration that creates the table; _shared/workspace-scoping (S6) for why tenant_id is mirrored and never inferred; _shared/test-layout (S9) for the per-seam test folders.

Environment names on the sender (Labs core/config.py), all measured 2026-09-11: PRIMA_INTERNAL_BASE_URL (line 752; unset means every row queues as pending and none is delivered, which is the intended polarity: a visible backlog beats a silently broken key or lost money), PRIMA_CREDIT_GRANT_SECRET and PRIMA_KEY_SYNC_SECRET (S10), PRIMA_CREDIT_GRANT_TIMEOUT_SECONDS and PRIMA_KEY_SYNC_TIMEOUT_SECONDS (default 10.0 each, lines 741 and 720), PRIMA_CREDIT_GRANT_MAX_ATTEMPTS and PRIMA_KEY_SYNC_MAX_ATTEMPTS (default 24 each, lines 742 and 721), PRIMA_KEY_SYNC_ISSUER (line 712; unset makes every mint raise at enqueue rather than queue a row that can only end stuck). The Sparks outbox reuses the credit-grant variables (sparks_grants.py lines 359 to 360, 383 and 425). On Prima's w2 branch: PRIMA_LOW_BALANCE_ALERT_SECRET (unset means the drain loop is not started and one ERROR is logged at boot), PRIMA_LABS_INTERNAL_BASE_URL, and the constants MAX_ATTEMPTS = 24, _TIMEOUT_SECONDS = 5.0 in labs_events.py.

Data model

Three tables on Labs, one on Prima, one shape. Transcribed from the models and the w2 migration. Column, UNIQUE, index and FK names are the database names; the CHECK names in the three Labs blocks are the model-level names, and the database prefixes them (the note under the block).

-- alchemy_labs/backend/models/prima_key_sync.py (migration 20260804_0001_prima_key_syncs.py)
-- one row per key, mutated in place; the row IS the key's desired state
CREATE TABLE prima_key_syncs (
  id              uuid PRIMARY KEY,
  api_key_id      uuid NOT NULL REFERENCES api_keys(id) ON DELETE RESTRICT,
  key_hash        varchar(64) NOT NULL,          -- sha256 hex; never a plaintext key
  key_prefix      varchar(32) NOT NULL,
  tenant_id       uuid NOT NULL,                 -- = api_keys.workspace_id, mirrored, no FK
  external_sub    varchar(255) NOT NULL,
  issuer          varchar(255) NOT NULL,
  scopes          jsonb NOT NULL,
  desired_active  boolean NOT NULL DEFAULT true,
  expires_at      timestamptz,
  sync_seq        bigint NOT NULL DEFAULT 1,
  status          varchar(16) NOT NULL DEFAULT 'pending',
  attempts        integer NOT NULL DEFAULT 0,
  next_attempt_at timestamptz,
  last_error      text,                          -- a class name or 'HTTP <n>', never a body
  created_at      timestamptz NOT NULL DEFAULT now(),
  updated_at      timestamptz NOT NULL DEFAULT now(),
  synced_at       timestamptz,
  CONSTRAINT prima_key_syncs_status_valid     CHECK (status IN ('pending','synced','stuck')),
  CONSTRAINT prima_key_syncs_key_hash_is_sha256 CHECK (key_hash ~ '^[0-9a-f]{64}$'),
  CONSTRAINT prima_key_syncs_seq_positive     CHECK (sync_seq > 0),
  CONSTRAINT uq_prima_key_syncs_api_key       UNIQUE (api_key_id)
);
CREATE INDEX ix_prima_key_syncs_due ON prima_key_syncs (status, next_attempt_at);

-- alchemy_labs/backend/models/prima_credit_grant.py (migration 20260803_0003_prima_credit_grants.py)
-- one row per obligation; a top-up and a reversal per order, ever
CREATE TABLE prima_credit_grants (
  id                 uuid PRIMARY KEY,
  order_id           uuid NOT NULL REFERENCES orders(id) ON DELETE RESTRICT,
  tenant_id          uuid NOT NULL,              -- the BUYER's workspace, never orders.workspace_id
  kind               varchar(16) NOT NULL DEFAULT 'topup',
  amount_cents       integer,                    -- cents; the micro-CHF bridge lives in Prima
  currency           varchar(10),
  bonus_credit_micro integer,
  provider           varchar(32),
  provider_event_id  varchar(255),               -- the PSP's own transaction id: the idempotency anchor
  status             varchar(16) NOT NULL DEFAULT 'pending',
  attempts           integer NOT NULL DEFAULT 0,
  next_attempt_at    timestamptz,
  last_error         text,
  created_at         timestamptz NOT NULL DEFAULT now(),
  updated_at         timestamptz NOT NULL DEFAULT now(),
  granted_at         timestamptz,
  CONSTRAINT kind_valid   CHECK (kind IN ('topup','reversal')),
  CONSTRAINT status_valid CHECK (status IN ('pending','granted','stuck')),
  CONSTRAINT amount_shape CHECK ((kind = 'reversal' AND amount_cents IS NULL) OR
                                 (kind = 'topup' AND amount_cents IS NOT NULL AND amount_cents > 0)),
  CONSTRAINT uq_prima_credit_grants_order_kind UNIQUE (order_id, kind)
);
CREATE INDEX ix_prima_credit_grants_due ON prima_credit_grants (status, next_attempt_at);

-- alchemy_labs/backend/models/sparks_billing.py (migrations 20260808_0001_sparks_billing.py, 20260824_0001_sparks_team_grants.py)
-- obligations with no order behind them; the event id is deterministic
CREATE TABLE sparks_grants (
  id                uuid PRIMARY KEY,
  tenant_id         uuid NOT NULL,               -- mirrored, no FK
  kind              varchar(16) NOT NULL,
  subscription_id   uuid REFERENCES subscriptions(id) ON DELETE SET NULL,
  tier              varchar(20),
  period_key        varchar(20),                 -- 'YYYY-MM'
  spark_amount      integer NOT NULL,
  amount_micro      integer NOT NULL,            -- 32-bit; the module refuses a grant that would overflow it
  expires_at        timestamptz,
  provider          varchar(32) NOT NULL DEFAULT 'sparks',
  provider_event_id varchar(255) NOT NULL,       -- 'sparks-allowance-<tenant>-<YYYY-MM>', 'sparks-welcome-<tenant>', 'sparks-team-<tenant>-<reference>'
  status            varchar(16) NOT NULL DEFAULT 'pending',
  attempts          integer NOT NULL DEFAULT 0,
  next_attempt_at   timestamptz,
  last_error        text,
  created_at        timestamptz NOT NULL DEFAULT now(),
  updated_at        timestamptz NOT NULL DEFAULT now(),
  granted_at        timestamptz,
  CONSTRAINT sparks_kind_valid   CHECK (kind IN ('allowance','welcome','team')),
  CONSTRAINT sparks_status_valid CHECK (status IN ('pending','granted','stuck')),
  CONSTRAINT sparks_shape_valid  CHECK ((kind IN ('welcome','team') AND tier IS NULL AND period_key IS NULL) OR
                                        (kind = 'allowance' AND tier IS NOT NULL AND period_key IS NOT NULL)),
  CONSTRAINT sparks_amount_positive CHECK (spark_amount > 0),
  CONSTRAINT uq_sparks_grants_provider_event UNIQUE (provider_event_id)
);
CREATE INDEX ix_sparks_grants_due ON sparks_grants (status, next_attempt_at);

-- prima/migrations/versions/20260910_0001_low_balance_events.py (w2, unmerged)
-- the same shape in the other direction, with the content-free rule as CHECKs
CREATE TABLE prima_labs_events (
  id              bigserial PRIMARY KEY,
  kind            varchar(32) NOT NULL,
  tenant_id       uuid NOT NULL REFERENCES prima_credit_accounts(tenant_id) ON DELETE RESTRICT,
  seq             bigint NOT NULL,               -- the crossing number; sent as crossing_seq, ignored by Labs at fd8558f (Gotchas)
  balance_micro   bigint NOT NULL,
  allowance_micro bigint NOT NULL,
  threshold_micro bigint NOT NULL,
  occurred_at     timestamptz NOT NULL DEFAULT now(),
  status          varchar(16) NOT NULL DEFAULT 'pending',
  attempts        integer NOT NULL DEFAULT 0,
  next_attempt_at timestamptz,
  last_error      varchar(64),
  delivered_at    timestamptz,
  CONSTRAINT uq_prima_labs_events_tenant_kind_seq UNIQUE (tenant_id, kind, seq),
  CONSTRAINT ck_prima_labs_events_kind   CHECK (kind IN ('low_balance')),
  CONSTRAINT ck_prima_labs_events_status CHECK (status IN ('pending','delivered','stuck')),
  CONSTRAINT ck_prima_labs_events_attempts_nonneg CHECK (attempts >= 0),
  CONSTRAINT ck_prima_labs_events_seq_positive    CHECK (seq >= 1),
  CONSTRAINT ck_prima_labs_events_threshold_positive CHECK (threshold_micro > 0),
  CONSTRAINT ck_prima_labs_events_amounts_nonneg  CHECK (balance_micro >= 0 AND allowance_micro >= 0),
  CONSTRAINT ck_prima_labs_events_kind_token       CHECK (kind ~ '^[A-Za-z0-9_.:/@+-]+$'),
  CONSTRAINT ck_prima_labs_events_status_token     CHECK (status ~ '^[A-Za-z0-9_.:/@+-]+$'),
  CONSTRAINT ck_prima_labs_events_last_error_token CHECK (last_error ~ '^[A-Za-z0-9_.:/@+-]+$')
);
CREATE INDEX ix_prima_labs_events_due ON prima_labs_events (status, next_attempt_at);

(The models also declare single-column indexes on key_hash, tenant_id, status and next_attempt_at; they are omitted above.)

The CHECK names Postgres holds for the three Labs tables are not the strings in the models. core/database.py line 24 puts the convention ck_%(table_name)s_%(constraint_name)s on Base.metadata, and Alembic builds every table in a migration on that same metadata, so kind_valid is stored as ck_prima_credit_grants_kind_valid, sparks_kind_valid as ck_sparks_grants_sparks_kind_valid, and the three key-sync CHECKs, whose model names already carry the table, as ck_prima_key_syncs_prima_key_syncs_key_hash_is_sha256 and so on (read 2026-09-11 from pg_constraint on the harness's scratch database at head, fd8558f). A violation message names the prefixed form; the migrations' drop_constraint('sparks_kind_valid', ...) calls work because Alembic applies the same convention on the way out. Prima's ck_prima_labs_events_* names are written in full in the w2 migration and are stored as written.

The shared shape, which a new outbox copies: a status with exactly three values (pending, delivered, stuck) as a CHECK, where delivered is the terminal name for a new outbox and granted (prima_credit_grants, sparks_grants) and synced (prima_key_syncs) are the legacy Labs spellings of the same state (Prima's prima_labs_events above already writes delivered), so S12's note that no Labs outbox writes delivered describes the reference and not the rule, and a console (F30) keys on delivered; attempts, next_attempt_at, last_error; a (status, next_attempt_at) index for the due query; a UNIQUE that makes a duplicate enqueue a no-op (api_key_id; (order_id, kind); provider_event_id; (tenant_id, kind, seq)); a foreign key to the causing row with ON DELETE RESTRICT, never CASCADE (the row is evidence that a customer paid or held a credential; deleting the cause must not delete the evidence); a mirrored tenant_id with no FK, because it is used verbatim as the other service's tenant id and a later change to how this service resolves a workspace must not move an already-queued row.

What is not stored, on purpose: the signature and nonce (re-made per attempt, S10); the receiver's response body (last_error is type(exc).__name__ or HTTP <status>; Prima's column is a 64-character token with a CHECK); a plaintext key (key_hash only, with its own CHECK); a balance or a micro-CHF figure on the credit row (cents, as captured and verified by the webhook; the one bridge to micro-CHF is Prima's credits.MICRO_PER_CENT); an amount on a reversal (Prima looks up what it granted and clamps the claw-back). Two idempotency anchors exist in the estate: the PSP's own transaction id for a paid top-up (order.payrexx_transaction_id, falling back to the gateway id, then to order-<id> for a provider-less path no webhook can replay), and a deterministic string built from the tenant and the occasion for everything else.

API surface

An outbox has no route of its own. What it calls, per row, is a signed S10 endpoint; what it exposes is a set of functions with a fixed contract and, until F30 exists, nothing over HTTP.

Outbox (Labs)	Delivers to (Prima)	Success answer treated as terminal	Enqueue callers	Inline callers (after commit)
prima_credit_grants	POST /internal/credits/grant (kind topup), POST /internal/credits/reverse (kind reversal)	200, including {"granted": false, "duplicate": true}	fulfillment_service.py 281 to 282 (enqueue_topup inside the paid transition), 348 to 349 (enqueue_reversal on refund or cancel)	commerce_router.py 1567 to 1597: the Payrexx webhook, deliver_for_order through asyncio.to_thread after db.commit()
sparks_grants	POST /internal/credits/grant with bucket: "allowance" (allowance) or bucket: "purchase" and purchase_sparks (welcome, team)	200 including duplicate: true	webhooks_router.py 847 (enqueue_allowance on a settled subscription charge), primaapi_router.py 1384 (enqueue_allowance inside POST /sparks/change when a tenant moves to the free tier: an expiring allowance with no charge behind it, a deviation dated in Gotchas), primaapi_router.py 1168 (enqueue_welcome when POST /sparks/checkout picks the free tier) and services/primaapi/identity.py 329 (enqueue_welcome at self-serve signup), scripts/grant_team_sparks.py (team, operator)	webhooks_router.py 866, primaapi_router.py 1170 and 1389, identity.py 381, the script line 150
prima_key_syncs	POST /internal/keys/sync (whole desired state, sync_seq)	200 including {"applied": false, "superseded": true}	developer_router.py 173 (enqueue_mint), 310 (enqueue_revocation), identity.py 358	developer_router.py 186 and 317, identity.py 368 (deliver_for_key)

Outbox (Prima, w2 82ba0d1)	Delivers to (Labs)	Terminal success	Enqueue	Inline
prima_labs_events	POST /internal/sparks/low-balance-alert	any 200 (deliver_one reads only the status). At fd8558f the receiver answers {"ok": true, "email_sent": <bool>}, dedups one mail per (tenant_id, period_key) in its own table and never reads crossing_seq; the duplicate: true answer keyed on (tenant_id, crossing_seq) is ADR-0036 §2, planned, unmerged (Gotchas)	prima/credits.py _record_low_balance_crossing (line 1019) runs _SQL_RECORD_LOW_BALANCE_CROSSING (746, the UPDATE ... RETURNING on the account) and then _SQL_INSERT_LABS_EVENT (788, INSERT INTO prima_labs_events at 789), both inside the debit's transaction, under the account row lock	none; the 5 s drain is the mechanism (labs_events.drain_once docstring)

Function contract, identical across the three Labs modules and the pattern a new one copies:

Function	Owns a transaction	Raises	Returns
enqueue_*(db, ...)	no; runs inside the caller's	yes, on anything it cannot establish (no tenant, malformed tenant, unknown tenant, non-positive amount, no issuer, no workspace on an ai key)	the row, or the existing row on a duplicate; the desired-state variant (enqueue_revocation) returns None when the key is not a gateway key and has no row, which is the normal case for an ordinary Labs key
deliver(db, row)	no; the caller commits	never, on transport or server failure; records the failure class on the row	True if the row reached its terminal success
drain_once(db, limit=50)	yes; commits per row	never; one bad row is logged and marked, the loop continues	{"due": n, "delivered": m}
deliver_for_<cause>(db, id)	yes; commits once	never past its own log line	True or False
stuck_*(db, older_than_minutes=...)	no	no	the rows a human must look at: stuck, plus pending older than the cutoff (5 minutes for keys, 15 for money)
reconcile_existing_keys(db, limit=500)	yes; commits once	no	{"queued", "skipped"}; deliberately not scheduled

The receiver's contract, which the outbox depends on and S10 documents: a duplicate is a 200, never a 409; a 4xx means the message itself is wrong; 5xx, 408 and 429 mean try later. Prima's three receivers say the duplicate with a flag (duplicate, superseded) that the sender logs; the Labs receiver at fd8558f says nothing and dedups by calendar period in sparks_low_balance_alerts (services/commerce/sparks_low_balance.py lines 9 to 12, models/sparks_billing.py line 123), which is enough for the sender because it treats any 200 as delivered.

Application logic

In commit order, for a paid top-up (the other two Labs outboxes follow the same steps with their own cause):

1. The cause is established and verified by its own recipe: the Payrexx webhook re-fetches the transaction, checks amount and currency, and moves the order pending to paid in one transaction (commerce-payments).

2. ENQUEUE, inside that transaction: enqueue_topup(db, order, product) resolves the buyer's tenant from the order's own metadata (never from order.workspace_id, which is the selling merchant), checks the tenant exists, checks the amount is positive, picks the PSP transaction id as provider_event_id, reads the bonus from the product's server-side metadata, and INSERTs the row with status='pending', attempts=0, next_attempt_at=now(). It does not commit and it does not call anyone.

3. If the enqueue cannot establish who is owed what, it RAISES, the webhook answers 500, and the provider redelivers into the idempotent settlement path. This is the opposite of every best-effort fulfilment hook (an email that fails must not roll back an order), and the reason is stated in fulfillment_service.py lines 276 to 280: swallowing the exception means the money was captured and the obligation never recorded.

4. If the INSERT collides on the outbox's UNIQUE (a redelivered webhook, a double-submitted form, two workers), the obligation is already recorded: return the existing row. The key-sync outbox does this under a savepoint (begin_nested(), prima_key_sync.py lines 211 to 232) so the collision costs only this INSERT and not the caller's whole transaction; the credit and Sparks outboxes answer with a bare db.rollback() (Gotchas).

5. COMMIT, by the caller and only the caller. After it, "we owe this customer credit" is as durable as "this customer paid".

   A state change on a desired-state row (the key outbox's enqueue_revocation, prima_key_sync.py lines 240 to 316) is the second enqueue verb and copies none of the INSERT path. In the caller's transaction, still without committing: (a) lock the key's one row with a plain FOR UPDATE (line 254), blocking, not SKIP LOCKED, so a revocation that races a drain waits for the drain to finish delivering the previous state instead of skipping it; (b) set the new state, desired_active = False; (c) sync_seq += 1, so the retry of the previous state can never overwrite this one at Prima; (d) reset the retry fields as a unit, status = 'pending', attempts = 0, last_error = None, next_attempt_at = now() (lines 303 to 310), because the row may be synced or stuck from its last life and a bumped sync_seq on a synced row is a revocation the drain never picks up: a revoked key that still works, the worst outcome this outbox exists to prevent; (e) flush. A gateway key with no row (minted before the seam, or backfilled) gets a desired_active = False row at sync_seq = 1 under the same savepoint as the mint (lines 258 to 296; Prima's backfilled rows carry seq 0, so 1 still wins); a key that is not a gateway key returns None and nothing is written.

6. INLINE, after the commit, on the request's own session: deliver_for_order(db, order.id, kind=...) is called with the same db the endpoint has just committed (commerce_router.py line 1588), locks the row FOR UPDATE SKIP LOCKED (which opens the session's next transaction), calls deliver, and commits itself (prima_credit_grants.py line 459), because the endpoint does not commit again and, with S4's get_session (no autocommit, expunge_all() on close), nothing else would carry the attempt's attempts and status writes to the database. The webhook runs it through asyncio.to_thread because httpx.post blocks and the endpoint is async (commerce_router.py lines 1585 to 1588). It is wrapped in a try that only logs: the row is durable, so a failure here costs a minute, not a customer.

7. DELIVER, one attempt: read base URL and credential from settings (unset: record GrantNotConfigured or SyncNotConfigured and return, row stays pending); serialise the payload compactly; sign with a fresh timestamp and nonce (S10); attempts += 1; httpx.post with the configured timeout. Then: 200 is terminal success (granted or synced on the Labs tables, delivered on a new outbox; next_attempt_at = NULL, last_error = NULL, a duplicate or superseded flag is logged and nothing else); a 4xx other than 408 and 429 is stuck immediately with last_error = 'HTTP <n>' (retrying a malformed body or a bad signature changes nothing); everything else, including a transport exception, goes to _fail.

8. _fail: last_error is the exception class name or HTTP <n>; if attempts >= MAX_ATTEMPTS the row is stuck with next_attempt_at = NULL, else status = 'pending' and next_attempt_at = now() + backoff(attempts) where the backoff table is indexed by attempt count and clamped at its last entry.

9. DRAIN, every minute from the scheduler (scheduled_jobs.py lines 316 to 364: three jobs, IntervalTrigger(minutes=1), coalesce=True, max_instances=1, replace_existing=True): select up to 50 pending rows whose next_attempt_at is NULL or past, oldest first, FOR UPDATE SKIP LOCKED; deliver each; commit per row, so one row that throws does not return the successful ones beside it to pending. The lock stops noise between two workers or a drain racing the inline attempt; correctness does not depend on it (the receiver refuses the duplicate).

10. STUCK is terminal only for the worker. Nothing in any outbox module deletes a row. A human reads stuck_grants() or stuck_syncs() (or the SQL in the module docstring), fixes the cause, and resets the row by hand: status = 'pending', attempts = 0, next_attempt_at = now(), last_error = NULL. F30 is the planned button for exactly that statement.

11. RECONCILE, once, on the day a new outbox ships: reconcile_existing_keys queues a row for every ai-scoped key that has none. Deliberately not scheduled: if it finds anything after that day, a human should notice rather than a loop paper over it.

Prima's outbox (w2) differs in two places and both are deliberate: the enqueue is a SQL statement inside the debit's transaction, under the balance row's lock, and nothing holding that lock may wait on a network, so there is no inline attempt at all; and the drain is an asyncio task every 5 s that runs the synchronous drain_once through to_thread, one row per claim, committing per row. Its receiver at fd8558f is behind the sender: _payload (labs_events.py lines 199 to 222) sends crossing_seq, the three micro amounts and occurred_at as additive fields, and LowBalanceAlertRequest (prima_inbound_router.py lines 40 to 47) reads only tenant_id and percent_remaining, ignores the rest, and answers {"ok": true, "email_sent": <bool>} (line 87). The sender is correct against both the shipped receiver and the ADR-0036 one because deliver_one (lines 313 to 355) reads only the status code.

Invariants

• The enqueue runs inside the caller's transaction and never commits; the caller commits exactly once. After that commit the obligation cannot be lost by a timeout, a 500, a deploy or the receiver being down.
• The enqueue raises when it cannot establish the obligation; a recorded wrong obligation and an unrecorded right one are both worse than a 500 the provider retries.
• The inline attempt is an optimisation and runs after the commit; removing it changes latency and nothing else (deliver_for_order docstring; drain_once on w2 is "the mechanism, not an optimisation").
• deliver never raises past itself on a transport or server failure; a failure is a row state, not an exception a caller must catch correctly.
• Every attempt carries the same idempotency anchor (provider_event_id, the whole desired state plus sync_seq, or crossing_seq on w2) and a fresh signature; the receiver, not the sender, decides whether it already happened, and says so with a 200. The flag on that 200 is informational: Prima's receivers send one, the Labs receiver at fd8558f sends none, and no sender branches on it.
• A duplicate answer is success. A 4xx other than 408 and 429 is stuck at once. 5xx, 408, 429 and transport errors retry on the backoff table until MAX_ATTEMPTS, then stuck.
• stuck rows are never deleted by code and keep every field needed to replay them; next_attempt_at is NULL on a stuck row so the due query cannot pick it up.
• The due query is status = 'pending' AND (next_attempt_at IS NULL OR next_attempt_at <= now()), oldest first, FOR UPDATE SKIP LOCKED, committed per row.
• One outbox table per obligation kind: a CHF obligation reconciles to a payment forever, a Sparks obligation to a tier and a period, a key obligation to a credential; they share the wire and the retry policy, not the table (sparks_grants.py docstring, lines 1 to 50).
• One row per key, mutated in place with a bumped sync_seq, never an append-only event log, because two rows for one key can be delivered out of order and a stale "active" after a revocation is a revoked key that works (models/prima_key_sync.py lines 35 to 55; ADR-0026).
• tenant_id on the row is the other service's tenant id verbatim: never re-keyed (Prima's usage hash chain starts with it), never order.workspace_id.
• last_error is content-free: a class name or an HTTP status. The thing on the other end is an inference gateway and its response bodies do not belong in a billing table.
• Every outbox has its drain registered on the scheduler and its stuck query named in code; an outbox with a drain_once nobody calls is the defect of 2026-08-08 to 2026-08-24 (Gotchas).
• Every outbox is rendered by an admin surface: F30 for all of them (planned), which is the rule "no headless outbox" from the inventory.

Frontend

• billing-credits/ui/billing-outbox-admin-console (F30, planned): one table per outbox (prima_key_syncs, prima_credit_grants, sparks_grants, and Prima's prima_labs_events once merged) with status, attempts, next_attempt_at, last_error, created_at, the causing row's id; a filter that is exactly the stuck_*() query; a reset action that is exactly the statement in step 10; read-only for non-admins. The inventory row F30 also lists prima_auto_topups and webhook_events; the first does not exist at fd8558f (Gotchas).
• Until F30 exists, the minimum debug surface is the two named functions per module and the SQL in each module's docstring, run from a database prompt on the app's own database. There is no page, no route and no monitor reading these tables at the measured commits.

Security & tenancy

• Fail direction: the enqueue fails CLOSED (raise, 500, provider retries); the delivery fails SAFE (a visible pending row, never an exception and never a drop); an unset base URL or credential is a stalled outbox, not a fallback. This is the money class of the inventory's rule (§2).
• Auth dependency: none on the outbox itself; the delivery call is signed per S10 and the receiver's dependency is the only gate. The enqueue callers carry their own gates: the webhook's provider signature, the developer session behind POST /api/developer/keys, the self-serve signup path, an operator at a shell for grant_team_sparks.py.
• Workspace derivation: the row's tenant_id comes from the cause and is written once: the buyer's workspace from the order's own metadata (a credits order is sold by the merchant workspace and bought by another; crediting order.workspace_id would credit the seller), api_keys.workspace_id for a key, the account id for a Sparks grant. An ai-scoped key with no workspace raises rather than guessing: attributing usage to the wrong tenant's hash chain is permanent.
• Secrets at rest: none on the row; the credential is in settings (S10). key_hash is a digest and holding it does not authenticate.
• Attacker-controlled inputs: none reach the row from a request body. Amounts come from the verified capture and the product's server-side metadata; the team-grant reference comes from an operator; the webhook's own body never writes the outbox directly.
• Content-free discipline: last_error is a class name or a status, the payloads carry UUIDs, integers, bounded tokens and timestamps, and the module docstrings state there is no ledger and no plaintext key in the file (prima_credit_grants.py lines 50 to 54, prima_key_sync.py lines 60 to 64).
• Rate limits: the backoff table is the rate limit toward the receiver, and the drain's limit=50 per minute bounds a burst; there is no per-tenant limit because a row exists only for a cause the caller established (a verified capture, a mint behind a session, an operator at a shell). The one enqueue that runs on a self-serve request with no verified payment is the plan change to the free tier (Gotchas); its UNIQUE on provider_event_id bounds it to one row per tenant per calendar month.
• Who may reset a stuck row: today anyone with a database prompt; F30 makes it an admin action. A reset never changes the payload, only the retry state, so the worst a wrong reset does is one more refused delivery.

Tests

Labs (backend/tests/, S9 layout: per-seam folders with their own conftest.py, run with --confcutdir so the shared database fixture that drops every table stays out):

• prima_credits/test_credit_grant_client.py (hermetic, no database): a successful delivery marks the row granted; test_a_duplicate_answer_is_success; test_the_wire_carries_cents_and_never_micro_chf; test_every_attempt_carries_the_same_psp_event_id; a transport failure leaves a pending row with a backoff; test_the_error_recorded_is_a_class_or_a_status_never_a_response_body; test_attempts_exhausted_becomes_STUCK_and_is_never_deleted; test_a_4xx_goes_straight_to_stuck_rather_than_burning_the_retries; test_the_two_retryable_4xx_are_still_retried (408, 429); an unconfigured receiver leaves a pending row, not an exception; test_a_reversal_sends_no_amount; test_the_backoff_grows_and_is_bounded.
• prima_credits/test_credit_topup_fulfilment.py (live, against a scratch database named by PRIMA_CREDITS_TEST_DATABASE_URL, skipped when unset): the enqueue inside the paid transition, the UNIQUE on (order_id, kind).
• prima_keys/test_prima_key_sync.py: test_the_payload_carries_state_not_an_event, test_backoff_is_front_loaded_and_clamped, test_exhausting_attempts_parks_the_row_as_stuck_and_never_deletes_it, test_last_error_is_a_class_or_a_status_never_a_response_body, test_a_200_marks_the_row_synced, test_a_superseded_answer_is_a_success, test_a_4xx_goes_straight_to_stuck_rather_than_retrying_forever, test_a_transient_status_is_retried, test_unconfigured_stalls_visibly_rather_than_raising, test_only_ai_scoped_keys_are_gateway_keys, test_the_wildcard_set_matches_primas.
• sparks/test_sparks_grant_client.py (hermetic): the allowance payload uses the allowance bucket and expiry; the welcome payload uses the purchase bucket and never expires; test_team_grant_rides_the_purchase_bucket_not_the_allowance_bucket; test_it_posts_to_the_same_grant_path_as_credits_topups; test_it_signs_with_the_same_scheme_as_prima_credit_sig; test_every_attempt_carries_the_same_deterministic_event_id; test_backoff_schedule_matches_prima_credit_grants.
• sparks/test_sparks_grants_db.py (live): a confirmed payment queues exactly one allowance grant; a retried webhook delivery does not double-grant the same period; a renewal next month is a new grant; the welcome grant is created once and a second call returns the same row; the seed is idempotent across runs.
• sparks/test_sparks_team_grant.py, sparks/test_sparks_concurrent_checkout.py, sparks/test_sparks_reaper_webhook_interleave.py (live): the enqueue under concurrent checkouts, with deliver_for_grant monkeypatched to False so nothing leaves the process.

Prima (tests/), two layers. The hermetic seam files pin the boundary decided before the database is touched, never the duplicate: test_credit_grant_seam.py (31 tests, signature, schema and bucket refusals) and auth/test_key_sync_seam.py (the unset-secret 503, the wrong-secret 401, the wildcard and non-ai scope refusals as terminal 4xx, the canonical-string vector; its docstring, lines 1 to 9, says why idempotency is not tested with a mock). The live files, against Postgres and skipped without it, pin duplicate-is-200: test_credit_grant_live.py test_a_replayed_payment_event_moves_the_balance_exactly_once (line 153), test_a_replayed_refund_claws_back_once (279), test_a_replayed_allowance_grant_lands_once (385) and test_a_replayed_welcome_grant_lands_once (512), each asserting duplicate is False the first time and True after; test_key_sync_live.py::test_the_same_message_twice_is_idempotent (182, superseded is True at 193); test_credits_live.py::test_one_order_can_only_ever_fund_one_topup (511) pins the bare ON CONFLICT DO NOTHING so a narrowed conflict target fails a test instead of producing a grant that retries forever. On w2: test_low_balance_event.py (a 200 including duplicate: true is delivered; a retryable status backs off; a non-retryable 4xx goes stuck immediately; the backoff table is indexed by attempt and clamps; the row goes stuck once attempts are exhausted; a transport error records the class and never the message; test_the_error_token_is_always_storable; an unconfigured deployment queues rather than drops; the drain loop is not started without the credential; test_the_drain_never_runs_on_the_event_loop; test_the_debit_path_cannot_reach_the_module_that_talks_to_labs, which walks the import closure of prima.credits; test_the_outbound_module_holds_no_route) and test_low_balance_event_live.py.

Known-answer values a new outbox pins: the backoff table (5, 15, 60, 300, 900, 3600) with _backoff(0) == _backoff(1) == 5 s and _backoff(99) == 3600 s; MAX_ATTEMPTS = 24; the exact provider_event_id string for each kind; the payload keys per kind. No Playwright gate until F30 exists; then one that loads the console, filters stuck, and resets one seeded row.

Deploy & ops

• Config keys per app heredoc (S13): on the Labs blocks, PRIMA_INTERNAL_BASE_URL, PRIMA_CREDIT_GRANT_SECRET, PRIMA_KEY_SYNC_SECRET, PRIMA_KEY_SYNC_ISSUER, the two *_TIMEOUT_SECONDS and the two *_MAX_ATTEMPTS; on the Prima block, the matching credentials and, once w2 merges, PRIMA_LOW_BALANCE_ALERT_SECRET and PRIMA_LABS_INTERNAL_BASE_URL. Same name on both sides, one master value (ADR-0036 §6).
• Migrations, in order: Labs 20260803_0003_prima_credit_grants.py, 20260804_0001_prima_key_syncs.py, 20260808_0001_sparks_billing.py, 20260824_0001_sparks_team_grants.py (widens the kind CHECK); Prima 0002_prima_credits.py (the ledger and its UNIQUE guards), 20260804_0001_key_sync_seq.py (sync_seq on prima_api_keys), 20260808_0001_two_bucket_sparks.py, and on w2 20260910_0001_low_balance_events.py. All additive; every added column nullable or defaulted; the downgrade of an outbox migration drops the table and with it every undelivered obligation, so it is not run on a database that has ever taken a payment.
• Scheduler jobs (S12): prima_key_sync_drain, prima_credit_grant_drain, sparks_grant_drain, each IntervalTrigger(minutes=1), coalesce=True, max_instances=1, on the leader-only APScheduler in services/scheduled_jobs.py. Prima w2: the _labs_event_drain_loop asyncio task at 5 s, started only when labs_events.is_configured() is true (prima/app.py lines 288 to 385).
• Argus checks: none exist at the measured commits. The minimum this recipe asks for, per outbox: a count of stuck rows (alert on any), a count of pending rows older than the stuck query's cutoff (5 minutes for keys, 15 for money), and a heartbeat of the drain job. ADR-0036 names the same need for the W1 table.
• Rollback: a code rollback on the sender leaves rows where they are; nothing is lost because nothing is deleted. A code rollback on the receiver that changes the wire is caught by the S10 vector test before deploy. After a credential rotation, rows that hit 401 during the window are stuck with last_error = 'HTTP 401' and are reset with the step-10 statement. The one-off after a new outbox ships is its reconcile_* function, run by hand, once.

Gotchas

• 2026-08-03 (measured, alchemy_labs b5446cf, wiki d4f5b95): the outbox exists because the alternative design, an HTTP call to Prima inside the webhook transaction, is the one where a timeout costs a customer their money with no symptom; the founder's ruling of 2026-08-03 that Prima's ledger is the only ledger (decision D2) is what made the outbox the only shape left. An earlier branch that put the ledger in Labs never created its tables in any database.
• 2026-08-04 (measured, alchemy_labs 24f16ef): the first live mint against the key-sync seam failed inside enqueue_mint because APIKey.id is a Python-side default applied at flush, so a caller that had only db.add()ed the key handed over a row with a NULL api_key_id. Fix in the same commit: db.flush() inside the enqueue when api_key.id is None (prima_key_sync.py lines 180 to 189), still inside the caller's transaction, still one commit.
• 2026-08-08 to 2026-08-24 (measured, git log -S): sparks_grants.drain_once shipped in 86e5d9f (2026-08-08) and no scheduled job called it until run_sparks_grant_drain was registered in 07d4da7 (2026-08-24); a welcome or allowance grant whose inline delivery failed sat pending with no retry while the module docstring and the seed script said the drain would pick it up. scheduled_jobs.py lines 347 to 355 record it. The rule "every outbox registers its drain" is this entry.
• 2026-09-11 (measured, fd8558f): the duplicate-enqueue branch differs between modules. prima_key_sync.enqueue_mint and enqueue_revocation use a savepoint (begin_nested(), lines 217 and 292) so a collision on uq_prima_key_syncs_api_key undoes only the outbox INSERT; enqueue_topup, enqueue_reversal, enqueue_allowance, enqueue_team_grant and enqueue_welcome answer the IntegrityError with a bare db.rollback() (prima_credit_grants.py 169 and 223; sparks_grants.py 141, 235, 283), which ends the caller's transaction. The key-sync comment at lines 211 to 216 names the consequence ("a harmless duplicate into your key vanished"). The recipe rule is the savepoint; the five bare rollbacks are a deviation to fix on the shipping branch.
• 2026-09-11 (measured): three backoff tables. Credit and Sparks: (5, 15, 60, 300, 900, 3600); key sync: (5, 15, 60, 180, 600), deliberately tighter because the thing waiting is a customer who cannot make a first call (prima_key_sync.py lines 95 to 101); Prima w2: the six-entry table. With 24 attempts the six-entry table spans about 18.4 hours and the five-entry table about 3.2 hours (computed 2026-09-11 from the tables). The inventory row S11 names only the six-entry table; the code is the rule and the key-sync exception is written.
• 2026-09-11 (measured, code against the wiki): the inventory row S11 says "1-minute APScheduler drain". True for the three Labs outboxes; Prima's w2 outbox drains every 5 s from an asyncio task through to_thread (prima/app.py line 288, 345) and has no inline attempt. The code wins; the reason is that its enqueue runs under a balance row lock and must not wait on a socket.
• 2026-09-11 (measured, fd8558f): models/sparks_billing.py line 41 GRANT_KINDS = ("allowance", "welcome") while the CHECK on line 90 and migration 20260824_0001 allow team. Nothing validates against the tuple; it is stale documentation and _payload handles team (lines 308 to 328).
• 2026-09-11 (measured, fd8558f): no admin surface, no route and no monitor reads any of the three tables; outside the modules, a stuck query is named only in scripts/grant_team_sparks.py line 158 (an operator hint) and in the drain-job docstrings at services/scheduled_jobs.py lines 651 and 727. F30 is planned. The inventory row F30 also lists a prima_auto_topups table; it belongs to the W1 branch, which is unmerged and failed verification (inventory B66), and does not exist at fd8558f.
• 2026-09-11 (measured, code against the wiki): inventory B63 says "allowance RESETS, purchase ADDS, a top-up sent as allowance destroys balance". The code agrees: sparks_grants._payload sends bucket: "allowance" only for kind allowance and bucket: "purchase" for welcome and team (lines 304 to 349), and Prima's credits.credit(bucket="allowance") resets the expiring bucket (GrantRequest docstring, internal.py lines 182 to 232). A new grant kind that means "add" goes in purchase.
• 2026-08-10 (measured, sparks_grants.py and the wiki page updated 2026-08-20): renew_free_tier_allowances was removed (the note at sparks_grants.py lines 499 to 507); the free budget is a one-time welcome grant in the persistent bucket and is never re-granted. provider_event_id = "sparks-welcome-<tenant>" is what makes a re-run of the seed a no-op.
• 2026-09-11 (measured, fd8558f): one caller still contradicts that rule. POST /sparks/change (primaapi_router.py line 1256) on a move to the free tier creates a 30-day subscription row and calls enqueue_allowance (line 1384) with the free tier's configured monthly allowance, bucket: "allowance", which resets Prima's expiring bucket, with no settled charge behind it and a provider_event_id of sparks-allowance-<tenant>-<YYYY-MM>; POST /sparks/checkout on the same tier calls enqueue_welcome (line 1168, the comment at 1163 to 1167 states the one-time rule). A tenant who downgrades in month N and again in month N+1 gets an expiring allowance each time. The recipe rule is the welcome grant; the plan-change path is a deviation to close on the shipping branch, and the "verified cause" sentence in Security & tenancy is qualified for it.
• 2026-09-11 (measured, fd8558f against w2 82ba0d1 and wiki 76c29d9): the Labs receiver behind Prima's outbox is the one that shipped in August, not the one ADR-0036 §2 describes. POST /internal/sparks/low-balance-alert reads tenant_id and percent_remaining, ignores every additive field including crossing_seq, dedups one mail per (tenant_id, period_key) through uq_sparks_low_balance_alerts_tenant_period, and answers {"ok": true, "email_sent": <bool>} (prima_inbound_router.py line 87); there is no duplicate field and no (tenant_id, crossing_seq) key anywhere on the branch. The duplicate: true answer and the PRIMARY KEY (tenant_id, crossing_seq) table are the W1 receiver on the ADR branch, unmerged and failed verification (inventory B66). The sender treats any 200 as delivered, so nothing breaks; but a crossing that fires twice in one month is one mail, and one that fires twice across a month boundary is two, until the ADR receiver merges.
• 2026-09-11 (measured, fd8558f): the webhook's inline delivery runs deliver_for_order through asyncio.to_thread (commerce_router.py lines 1582 to 1588) because the endpoint is async and httpx.post blocks; the mint endpoint is sync and calls deliver_for_key directly. A new async caller copies the first; a sync caller the second.
• 2026-09-11 (measured, w2 82ba0d1): prima_labs_events.last_error is varchar(64) with a token CHECK, so a delivery failure whose class name carries a space or a slash would fail the UPDATE; labs_events._error_token strips to the allowed alphabet and truncates before the write (lines 252 to 258). The Labs tables use text with no CHECK and rely on the module writing only a class name or a status.

Reference implementation

Paths are relative to each repo's root; commits as in measured_on and also_measured_on.

• alchemy_labs (d1a-token-source, fd8558f; the backend tree equals feat/service-desk b1089b6):
  • backend/services/commerce/prima_credit_grants.py: docstring (1 to 55), _BACKOFF_SECONDS and _backoff (82 to 87), enqueue_topup (95 to 184), enqueue_reversal (187 to 232), _payload (245 to 264), deliver (267 to 353), _fail (356 to 381), drain_once (384 to 428), deliver_for_order (431 to 460), stuck_grants (463 to 484).
  • backend/services/commerce/sparks_grants.py: period_key (85 to 90), enqueue_allowance (97 to 156), MAX_TEAM_GRANT_SPARKS and enqueue_team_grant (163 to 250), enqueue_welcome (253 to 293), _payload (304 to 349), deliver (351 to 420), _fail (423 to 439), drain_once (442 to 473), deliver_for_grant (476 to 494), the removal note (499 to 507), stuck_grants (509 to 525).
  • backend/services/primaapi/prima_key_sync.py: docstring with the stranded-key SQL (1 to 64), _BACKOFF_SECONDS (102), is_gateway_key (116 to 131), enqueue_mint (134 to 237), enqueue_revocation (240 to 316), _issuer (322 to 338), _payload (346 to 357), deliver (360 to 448), _fail (451 to 471), drain_once (474 to 516), deliver_for_key (519 to 547), reconcile_existing_keys (550 to 586), stuck_syncs (589 to 617).
  • backend/models/prima_credit_grant.py, backend/models/sparks_billing.py, backend/models/prima_key_sync.py: the tables above and their docstrings.
  • backend/services/scheduled_jobs.py: registrations (316 to 364), run_prima_key_sync_drain (631), run_prima_credit_grant_drain (670), run_sparks_grant_drain (708).
  • backend/services/commerce/fulfillment_service.py lines 265 to 290 (enqueue inside the paid transition, must raise) and 348 to 349; backend/api/routers/commerce_router.py lines 1565 to 1597 (commit, then inline); backend/api/routers/developer_router.py lines 160 to 192 and 305 to 322; backend/api/routers/webhooks_router.py lines 840 to 870; backend/api/routers/primaapi_router.py lines 1150 to 1172 (welcome at a free checkout) and 1359 to 1389 (allowance at a change to free, the deviation); backend/services/primaapi/identity.py lines 320 to 384; backend/scripts/grant_team_sparks.py.
  • backend/api/routers/prima_inbound_router.py lines 40 to 87 and backend/services/commerce/sparks_low_balance.py (the receiver for Prima's outbox as it ships); backend/models/sparks_billing.py line 123 (uq_sparks_low_balance_alerts_tenant_period).
  • backend/core/config.py lines 700 to 752; backend/core/database.py line 24 (the CHECK naming convention).
  • backend/migrations/versions/20260803_0003_prima_credit_grants.py, 20260804_0001_prima_key_syncs.py, 20260808_0001_sparks_billing.py, 20260824_0001_sparks_team_grants.py.
• prima (feature/per-model-pricing, cd2c248; the receivers): prima/api/internal.py (grant_credit 302 onward, the bare ON CONFLICT DO NOTHING argument at 344 to 373, _grant_response 390 to 432), prima/api/keys_sync.py (sync_key 339 to 543), prima/credits.py line 522 (ON CONFLICT DO NOTHING), prima/models/credits.py lines 223 and 262 to 277.
• prima (w2-balance-threshold-event, 82ba0d1, unmerged; the sender): prima/api/labs_events.py (BACKOFF_SECONDS, MAX_ATTEMPTS, _TIMEOUT_SECONDS and _RETRYABLE_STATUSES 123 to 139, _payload 199 to 222, _headers 225 to 249, _error_token 252 to 258, the claim, delivered, retry and stuck statements 271 to 311, deliver_one 313 to 363, _fail 366 to 409, drain_once 412 to 450), prima/credits.py _SQL_RECORD_LOW_BALANCE_CROSSING (746 to 760), _SQL_INSERT_LABS_EVENT (788 to 796) and _record_low_balance_crossing (1019 onward), prima/app.py lines 288 to 385, migrations/versions/20260910_0001_low_balance_events.py, tests/test_low_balance_event.py.
• Wiki philosophers_stone (main d4f5b95): operations/prima-modules-inventory.md §1.1 row S11, §1.6 rows B12, B59, B63, B65, B66 and F30, §2 ("idempotency belongs to the database"); operations/decisions/0026-labs-to-prima-key-propagation.md ("Durability by outbox"); platform/prima-sparks-pricing.md (updated 2026-08-20; the operator path and the one-time welcome budget). Branch adr-0036-prepaid-auto-topup (76c29d9): operations/decisions/0036-prepaid-credits-auto-topup-and-balance-events.md §2, §3 and Consequences ("idempotency is structural on both sides").

Deviations of the references from this recipe, all dated 2026-09-11: five bare db.rollback() duplicate branches where the rule is a savepoint; the key-sync backoff table differs from the other two; GRANT_KINDS in sparks_billing.py is stale; no admin surface and no monitor; Prima's outbox and the W1 tables are unmerged; the sparks drain was unregistered for sixteen days in August; POST /sparks/change grants an expiring allowance on a move to the free tier where the rule is the one-time welcome grant; the Labs receiver for Prima's outbox dedups by period and returns no duplicate flag where ADR-0036 §2 keys it on crossing_seq.

Acceptance checks

Run without the author at the commits named above. Expected observations are exact.

1. Labs, backend/: pytest -q tests/prima_credits/test_credit_grant_client.py -p no:cacheprovider --confcutdir=tests/prima_credits passes with no database; pytest -q tests/sparks/test_sparks_grant_client.py --confcutdir=tests/sparks and pytest -q tests/prima_keys/test_prima_key_sync.py -k "stuck or superseded or 4xx or transient or unconfigured" pass.
2. Labs: grep -n "prima_key_sync_drain\|prima_credit_grant_drain\|sparks_grant_drain" services/scheduled_jobs.py | grep -c "id=" prints 3; grep -c "IntervalTrigger(minutes=1)" services/scheduled_jobs.py prints at least 3.
3. Labs: grep -nE "\.delete\(|DELETE FROM" services/commerce/prima_credit_grants.py services/commerce/sparks_grants.py services/primaapi/prima_key_sync.py prints nothing; grep -c "skip_locked=True" services/commerce/prima_credit_grants.py services/commerce/sparks_grants.py services/primaapi/prima_key_sync.py prints 2 for each file.
4. Labs, the deviation: grep -n "db.begin_nested()" services/primaapi/prima_key_sync.py prints exactly two lines, 217 and 292 (the bare word also matches a comment on 215); grep -c "db.rollback()" services/commerce/prima_credit_grants.py services/commerce/sparks_grants.py prints 2 and 3. When the five become savepoints, the second command prints 0 and 0 and the Gotchas entry is dated closed.
5. Labs, ordering at the mint: awk '/prima_key_sync.enqueue_mint/{e=NR} /db.commit\(\)/{if(e&&!c)c=NR} /deliver_for_key/{if(c&&!d)d=NR} END{print (e<c && c<d)}' api/routers/developer_router.py prints 1 (enqueue, then commit, then the inline attempt).
6. Labs, backoff, inside the app's own environment (the tests carry it): pytest -q tests/prima_credits/test_credit_grant_client.py -k backoff --confcutdir=tests/prima_credits, pytest -q tests/sparks/test_sparks_grant_client.py -k backoff --confcutdir=tests/sparks and pytest -q tests/prima_keys/test_prima_key_sync.py -k backoff pass; they assert the credit table climbs to exactly 3600 s, the Sparks table equals the credit table, and the key-sync table starts at 5 s and clamps at its own last entry (600 s). grep -n "^_BACKOFF_SECONDS" services/commerce/prima_credit_grants.py services/commerce/sparks_grants.py services/primaapi/prima_key_sync.py prints the three tuples as written in Gotchas.
7. Labs, on a database at alembic head (the harness's scratch database from tooling/render/labs-local.sh up qualifies; PGGSSENCMODE=disable if psql tries GSSAPI first): SELECT conname FROM pg_constraint WHERE conrelid = 'prima_credit_grants'::regclass ORDER BY 1 prints ck_prima_credit_grants_amount_shape, ck_prima_credit_grants_kind_valid, ck_prima_credit_grants_status_valid, fk_prima_credit_grants_order_id_orders, pk_prima_credit_grants, uq_prima_credit_grants_order_kind; the same for sparks_grants includes ck_sparks_grants_sparks_kind_valid, ck_sparks_grants_sparks_shape_valid, ck_sparks_grants_sparks_amount_positive, uq_sparks_grants_provider_event; for prima_key_syncs includes ck_prima_key_syncs_prima_key_syncs_key_hash_is_sha256, ck_prima_key_syncs_prima_key_syncs_seq_positive, uq_prima_key_syncs_api_key. SELECT confdeltype FROM pg_constraint WHERE conrelid = 'prima_credit_grants'::regclass AND contype = 'f' prints r (RESTRICT); the same for sparks_grants prints n (SET NULL).
8. Labs, on that database, inside BEGIN; ... ROLLBACK;: INSERT INTO prima_credit_grants (id, order_id, tenant_id, kind, amount_cents, status) VALUES (gen_random_uuid(), <an order id>, <a workspace id>, 'topup', 0, 'pending') fails with violates check constraint "ck_prima_credit_grants_amount_shape" (the CHECK fires before the FK, so a random order_id gives the same message); a second row with the same order_id and kind and a positive amount fails on uq_prima_credit_grants_order_kind.
9. Prima: grep -n "ON CONFLICT DO NOTHING" prima/credits.py prints line 522; pytest -q tests/test_credit_grant_seam.py tests/auth/test_key_sync_seam.py -p no:cacheprovider passes with no database (51 tests on 2026-09-11; none of them is the duplicate case); with a database, pytest -q tests/test_credit_grant_live.py tests/test_credits_live.py -k "replayed or one_order" (eight tests) and pytest -q tests/test_key_sync_live.py -k same_message_twice pass, and those are the duplicate-is-200 checks.
10. Prima w2 (82ba0d1): grep -n "_LABS_EVENT_DRAIN_INTERVAL_SECONDS = 5" prima/app.py prints line 288; grep -nE "^from prima.api|^import prima.api|labs_events import" prima/credits.py prints nothing; pytest -q tests/test_low_balance_event.py -k "debit_path_cannot_reach or holds_no_route or never_runs_on_the_event_loop" passes.
11. Labs, the receiver as it ships: grep -n 'crossing_seq\|"duplicate"' api/routers/prima_inbound_router.py services/commerce/sparks_low_balance.py prints nothing and grep -n "email_sent" api/routers/prima_inbound_router.py prints line 87. When ADR-0036 §2 merges, the first command prints the dedup INSERT and the duplicate answer, and the Gotchas entry is dated closed.
12. Labs, the free-tier deviation: grep -n "sparks_grants.enqueue_allowance" api/routers/primaapi_router.py api/routers/webhooks_router.py prints primaapi_router.py:1384 and webhooks_router.py:847. When the plan-change path calls enqueue_welcome instead, only the webhook line prints and the Gotchas entry is dated closed.
13. On a new service, the known answers a rebuilt outbox reproduces (all from the credit outbox at fd8558f): _BACKOFF_SECONDS == (5, 15, 60, 300, 900, 3600); _backoff(0) and _backoff(1) are 5 s, _backoff(6) and _backoff(99) are 3600 s (the index is min(max(attempts - 1, 0), 5)); the attempts ceiling defaults to 24 (<OUTBOX>_MAX_ATTEMPTS); SELECT conname FROM pg_constraint WHERE conrelid = '<outbox>'::regclass ORDER BY 1 lists ck_<outbox>_status_valid, pk_<outbox>, one uq_<outbox>_... and one fk_<outbox>_..._<cause> named by the convention (S8) and the CHECK lists pending, delivered, stuck; the drain is registered with id="<outbox>_drain" and IntervalTrigger(minutes=1) (S12); grep -nE "\.delete\(|DELETE FROM" <the outbox module> prints nothing; a 200 with duplicate: true is delivered, a 408 or 429 stays pending with next_attempt_at in the future, any other 4xx is stuck at once with last_error = 'HTTP <n>', and a transport error stores the exception class name only.
14. Every written domain recipe that declares depends_on: [_shared/outbox-pattern] names billing-credits/ui/billing-outbox-admin-console (or its own admin table) in its Frontend section and its drain job in Deploy & ops; npm run check in this repo prints 0 problems.