Auto top-up charge-and-retry engine

Purpose

When Prima tells Labs that a tenant's balance crossed their own top-up line, this engine turns that one crossing into at most one card charge against the token the tenant banked, bounded by a monthly cap and a three-strike dunning limit, and never loses the crossing to a crash, a timeout or a second scheduler tick. The invariant that makes it trustworthy is that the reconcile marker (prima_auto_topups.pending_reconcile_ref, written and committed before Payrexx is ever contacted) outlives every other transition on the row, a process death, a read timeout, a racing scheduler run, a tenant switching the instruction off, none of them may null it, so a crash between "the PSP said yes" and "the Order is recorded" becomes a late success the retry job reconciles by looking the transaction up, never a second debit (module docstring, auto_topup_service.py 1 to 103).

When to use

Read it before touching charge_if_due, _charge_if_due, _set_marker/_clear_marker, _record_charge/_record_decline, reconcile_pending, run_auto_topup_retry, or _stall_if_overdue; before changing what counts as a PSP "answer" versus "no answer" (subscription_service._classify_charge_failure); and when someone asks "why did this tenant get charged twice", "why is this crossing stuck", "did the card actually get charged", or "why didn't the retry fire". Trigger phrases: "auto top-up", "low-balance-alert", "claim before send", "reconcile marker", "pending_reconcile_ref", "crossing_seq", "ADR-0036 §4a", "three declines", "cap_reached", "in_flight", "resolved_elsewhere", "stalled".

Not this recipe: PUT/GET/DELETE /credits/auto-topup in api/routers/primaapi_router.py (1302 to 1622), the tenant-facing standing-instruction CRUD (threshold, amount, cap, enable/disable), a separate recipe on the same table; the threshold-sync outbox that tells Prima where the line is (services/commerce/prima_threshold_sync.py, threshold_sync_* columns), this recipe only calls mark_pending/deliver_for_tenant on the third strike, it does not own that outbox; the credit-grant outbox this engine enqueues into on every successful charge (services/commerce/prima_credit_grants.enqueue_topup/deliver_for_order, _shared/outbox-pattern's reference impl); the save_card token-banking path off the checkout webhook (persist_token_from_order, auto_topup_service.py 177 to 302, read for context, not this recipe's subject); the German billing mail copy itself (services/email_service, W11 Phase 3 §2); Prima's own crossing-detection, hysteresis latch and re-arm (Prima's repo, out of scope for Labs); the shared tri-state charge primitive's OTHER caller, subscription_service._convert_one (400 to 492) and run_due_conversions/convert_now, this recipe only borrows charge_token and _classify_charge_failure from that file.

Stack assumptions

At af7a352: fastapi==0.118.2, synchronous handlers on a sqlalchemy.orm.Session (SQLAlchemy==2.0.43), APScheduler==3.11.0 for the retry job, httpx==0.28.1/requests==2.32.5 for the PSP and Prima calls, pydantic==2.12.0 for the webhook body, psycopg2-binary==2.9.10, alembic==1.16.5, Postgres 15 in prod (docker-compose.prod.yml 9) / 14 in the test harness (docker-compose.test.yml 6). Foundations assumed: S4 (fastapi-project-layout) for the app and mount_router; S8 (alembic-migrations) for the four chained migrations under Data model; S10 (hmac-seam-pattern) for the inbound signature scheme this seam's HMAC verification is a near-verbatim port of (prima_inbound_sig.py 30 to 33); S11 (outbox-pattern) for prima_credit_grants.enqueue_topup/deliver_for_order, which _record_charge calls directly, and for prima_threshold_sync.mark_pending/deliver_for_tenant, which _record_decline calls on the third strike; S12 (scheduler-and-background-work) for the APScheduler singleton run_auto_topup_retry (the job) registers on.

Environment, all in core/config.py: PRIMA_CREDITS_MERCHANT_WORKSPACE_ID (line 749; unset makes _merchant_workspace_id return None and every charge answer not_configured), PRIMA_LOW_BALANCE_ALERT_SECRET (780; unset makes the webhook answer 503 before the signature is even checked), PAYREXX_TOKEN_CHARGE_ECHOES_REFERENCE (610 to 613, default=False; governs whether an empty filtered transaction listing may ever be read as absent, see Gotchas). Fail behaviour: charge_if_due and reconcile_pending are public wrappers around _charge_if_due/_reconcile_pending that catch every exception, roll back, and return "skipped" / "unavailable" respectively (585 to 620, 1266 to 1305), the module docstring's guarantee 4 ("it never raises") is enforced at exactly these two seams, because the caller is a webhook receiver whose 200 tells Prima's outbox "delivered", and run_auto_topup_retry's own per-row try/except (1796 to 1801) so one bad tenant cannot stop the sweep.

Data model

prima_auto_topups, one row per tenant, the standing instruction AND the crossing it currently owes (models/prima_auto_topup.py 127 to 321; migrated across four chained revisions, see Deploy & ops):

prima_auto_topups (
  tenant_id                       uuid PRIMARY KEY REFERENCES workspaces(id) ON DELETE RESTRICT,
  enabled                         boolean NOT NULL DEFAULT false,
  threshold_sparks                integer NOT NULL DEFAULT 0,       -- display unit; µCHF computed at send time
  amount_cents                    integer NOT NULL DEFAULT 0,       -- display copy of the Product's price
  monthly_cap_cents               integer NOT NULL DEFAULT 0,
  payrexx_token_ref               varchar(255),                     -- the AUTHORIZED transaction id, never the confirmed capture
  card_brand                      varchar(32), card_last4 varchar(4),
  consecutive_failures            integer NOT NULL DEFAULT 0,
  disabled_reason                 text,                             -- declined_3x | cap_reached | tenant | no_token
  last_attempt_at                 timestamptz,                      -- WRITTEN ONLY BY THE CLAIM (the in-flight clock)
  last_charged_at                 timestamptz,
  charged_this_period_cents       integer NOT NULL DEFAULT 0,
  period_key                      varchar(7),                       -- 'YYYY-MM'
  product_id                      uuid REFERENCES products(id) ON DELETE SET NULL,
  last_crossing_seq               bigint NOT NULL DEFAULT 0,        -- idempotency second opinion, under the row lock
  threshold_sync_seq              bigint NOT NULL DEFAULT 0,
  threshold_sync_status           varchar(16) NOT NULL DEFAULT 'synced',   -- pending | synced | stuck
  threshold_sync_attempts         integer NOT NULL DEFAULT 0,
  threshold_sync_next_attempt_at  timestamptz,
  threshold_sync_last_error       text,
  -- ADR-0036 §4a: the crossing Labs still owes this tenant
  topup_status                    varchar(16) NOT NULL DEFAULT 'idle',     -- idle | pending | stalled
  pending_crossing_seq            bigint,                           -- NULL exactly when idle
  pending_since                   timestamptz,                      -- the 7-day clock, set once per shortfall
  retry_after                     timestamptz,
  last_block_reason               varchar(32),                      -- cooldown | cap_reached | tier_inactive | declined | unknown_outcome
  pending_reconcile_ref           varchar(64),                      -- ⚠️ THE ONLY EVIDENCE A CHARGE MAY HAVE TAKEN MONEY ⚠️, our Order id
  pending_reconcile_amount_cents  integer,                          -- the CLAIM's own snapshot of what it asked the PSP for
  pending_reconcile_currency      varchar(3),
  pending_reconcile_product_id    uuid REFERENCES products(id) ON DELETE SET NULL,
  created_at timestamptz NOT NULL DEFAULT now(), updated_at timestamptz NOT NULL DEFAULT now()
);
CHECK cap:              monthly_cap_cents >= amount_cents
CHECK sync_status:      threshold_sync_status IN ('pending','synced','stuck')
CHECK topup_status:     topup_status IN ('idle','pending','stalled')
CHECK block_reason:     last_block_reason IS NULL OR last_block_reason IN
                        ('cooldown','cap_reached','tier_inactive','declined','unknown_outcome')
CHECK pending_shape:    (topup_status='idle' AND pending_crossing_seq IS NULL AND pending_since IS NULL
                          AND (retry_after IS NULL OR pending_reconcile_ref IS NOT NULL))
                        OR (topup_status<>'idle' AND pending_crossing_seq IS NOT NULL AND pending_since IS NOT NULL)
CHECK marker_snapshot:  (pending_reconcile_ref IS NULL AND pending_reconcile_amount_cents IS NULL
                          AND pending_reconcile_currency IS NULL AND pending_reconcile_product_id IS NULL)
                        OR (pending_reconcile_ref IS NOT NULL AND pending_reconcile_amount_cents IS NOT NULL
                          AND pending_reconcile_currency IS NOT NULL)

prima_balance_events, the receipt log and the FIRST idempotency guard, PRIMARY KEY (tenant_id, crossing_seq) (324 to 358): balance_micro, allowance_micro, threshold_micro (all recorded evidence, never read back as a balance), occurred_at (Prima's clock), received_at. No FK to workspaces (the id is mirrored from Prima, same rule as prima_credit_grants.tenant_id). No text column, deliberately, the event is content-free by construction, so the table cannot accumulate a prompt or a request id.

What is NOT stored anywhere in this recipe: a balance, in any form (module docstring, both files: "NO BALANCE IS READ ANYWHERE IN THIS FILE"); a card number (payrexx_token_ref is an opaque PSP transaction id, useless without the API credential); a PSP response body on any error column (threshold_sync_last_error and every log line store a class name or an HTTP nnn, never a message, which can carry a cardholder name).

API surface

One HTTP route receives the crossing; everything else in this recipe is called from it or from the scheduler.

Route	Auth	Tenancy	Body	Codes	Notes
POST /internal/sparks/low-balance-alert (prima_inbound_router.py 112 to 156, LOW_BALANCE_ALERT_PATH = the literal path, signed into the canonical string)	X-Prima-Signature/X-Prima-Timestamp/X-Prima-Nonce, HMAC-SHA256 over timestamp\nnonce\nMETHOD\npath\nsha256(body) (prima_inbound_sig.verify, 70 to 100), fail closed	tenant_id from the signed body, mirrored, never inferred from a session, there is no Labs session on this seam (surface_exempt("public", ...), app_fastapi.py 892)	LowBalanceAlertRequest (75 to 109): tenant_id: UUID, percent_remaining: float [0,1], and four ADR-0036 optional fields all defaulting to None, crossing_seq: int ≥1, balance_micro: int (⚠️ no ge=0, a post-paid overdraft is legal, line 102), allowance_micro: int ≥0, threshold_micro: int >0, occurred_at: datetime	200 {"ok": true, "duplicate": bool, "email_sent": bool, "topup": "charged"|"declined"|"unknown"|"disabled"|"skipped"|"not_configured"}; 401 invalid/missing signature; 503 seam not configured	Steps 2 to 4 (record crossing, email, charge) run in a threadpool, never on the event loop (147 to 156), a blocking requests POST to zahls, an httpx PUT to Prima and up to two SMTP round trips; crossing_seq absent ⇒ legacy call, email only, never a charge (this recipe's whole safety property, module docstring 35 to 39); a duplicate (tenant_id, crossing_seq) answers 200 duplicate: true, never 409 (a 409 here parks Prima's outbox stuck, 41 to 45)

The three Python entrypoints this route (and the scheduler) call, not routes, but the actual surface a rebuild reimplements:

Function	Owns its own transactions	Never raises	Returns
charge_if_due(db, tenant_id, *, reason, crossing_seq=None, now=None) (585 to 1037)	yes	yes (585 to 620 wraps _charge_if_due)	"charged"|"declined"|"unknown"|"disabled"|"skipped"|"not_configured"
reconcile_pending(db, tenant_id, *, now=None) (1266 to 1553)	yes	yes (1266 to 1305 wraps _reconcile_pending)	"charged"|"declined"|"absent"|"unavailable"|"resolved_elsewhere"|"in_flight"|"skipped"
run_auto_topup_retry(db, *, limit=100, now=None) (1695 to 1805)	per row	yes (1796 to 1801, one bad tenant)	{"due", "charged", "declined", "unknown", "skipped", "stalled", "reconciled"}

Application logic

_charge_if_due (623 to 1037), in commit order, exactly mirroring the source's own numbered comments:

1. LOCK the row FOR UPDATE (644 to 649); no row ⇒ not_configured (650 to 652).
2. Before any guard: a retry (crossing_seq is None) with no pending_crossing_seq on the row is refused, there is nothing identifiable to charge for, and it must never fall back to last_crossing_seq (654 to 673, W1-fix 2); a retry whose retry_after is still in the future has lost a race to another resolver and touches nothing (675 to 689).
3. GUARDS, all under the lock, in this order: structural, not row.enabled (696 to 699), no payrexx_token_ref (701 to 712, latches enabled=False, disabled_reason="no_token", should be unreachable, the PUT endpoint refuses this combination), consecutive_failures >= MAX_CONSECUTIVE_FAILURES (714 to 717), each clears the pending crossing (_clear_pending) rather than parking it, because no retry count charges a tenant who structurally cannot be charged. Then the idempotency second opinion: crossing_seq <= last_crossing_seq ⇒ replay, skipped, row untouched (719 to 732). Then the marker gate: pending_reconcile_ref set ⇒ even a fresh crossing is parked behind the open question, never charged over it (734 to 762, the hole this recipe's threshold path used to have). Then transient blocks, each parking the crossing with a specific retry_after: cooldown (last_attempt_at + MIN_ATTEMPT_INTERVAL_MINUTES, 764 to 780), tier inactive / no active priced Product (782 to 799, retried hourly), cap reached this period (814 to 840, retried at _next_period_start), period roll happens first, under the lock (805 to 812).
4. CLAIM, then commit, before Payrexx is touched (842 to 922): mint order_id = uuid.uuid4() (870); _set_marker(row, order_id, amount_cents, currency, product_id) writes pending_reconcile_ref AND its snapshot together (876 to 879); last_attempt_at = now (880, the ONE column only the claim writes, the in-flight clock); last_crossing_seq advanced to the delivered crossing_seq or, for a retry, to max(last_crossing_seq, pending_crossing_seq) (881 to 891); retry_after = now + 10 min (904, a schedule, not the claim itself); db.commit() (922).
5. SEND: charge_token(token_ref, amount_cents, currency, px=PayrexxService.for_workspace(db, merchant), reference_id=str(order_id)) (927 to 931).
6. RECORD, re-locked with populate_existing() (936 to 945, so stale identity-map state cannot leak into the guards below). outcome = result.get("outcome") or ("charged" if result.get("ok") else "declined") (947, back-compat with the pre-tri-state shape). Ownership check: if the re-locked row's pending_reconcile_ref is no longer str(order_id), another resolver answered this exact reference while the request was in flight (949 to 1003), on outcome == "charged" it either finds the Order already recorded (says charged, adds nothing) or records it now because the PSP's own answer requires an Order to exist; on any other outcome the other resolver's answer stands and nothing here parks, strikes or records over it. If ownership holds: outcome == "unknown" parks the crossing with a 10-minute reconcile window and concludes nothing (1005 to 1022, no strike, no mail); outcome != "charged" goes to _record_decline; else _record_charge.

_record_charge (1060 to 1176): resolves the owner email (blank rather than skipping, the money already moved, 1072 to 1081); inserts the Order(status="paid", provider="payrexx", payrexx_transaction_id=<the NEW confirmed tx>, extra_metadata={kind:"auto_topup", credit_tenant_id, crossing_seq, reason}) (1082 to 1111); prima_credit_grants.enqueue_topup(db, order, product) in the SAME transaction (1119, raises on anything it cannot establish, not caught here, so a raise here rolls back through charge_if_due's outer catch and the crash-recovery path in step 4/6 above resolves it next tick); resets consecutive_failures = 0 and, only if the row is still enabled, clears a stale disabled_reason (1131 to 1138, a late success reconciled after the tenant switched off still records the money without silently re-enabling anything); clears the marker only if pending_reconcile_ref == str(order_id) (1142 to 1143, _clear_pending on its own never touches the marker); commits; then, best-effort and after the commit, an inline grant-delivery attempt and the receipt mail (1159 to 1176), neither failure undoes the charge.

_record_decline (1179 to 1252): only reached for a PSP that actually answered declined (never for unknown); ends the marker if it is the one being answered (1205 to 1206); increments consecutive_failures; at the third strike, disables the row, clears the pending crossing, queues the threshold-clear via prima_threshold_sync.mark_pending in the SAME transaction as the disable (1210 to 1234) and attempts an inline delivery after commit; below the third strike, parks the crossing with backoff DECLINE_RETRY_HOURS[failures-1] = 1h then 6h (1236 to 1252).

_reconcile_pending (1308 to 1553): locks the row; no marker ⇒ skipped (1315 to 1325). In-flight guard: if last_attempt_at (never retry_after, W1-fix 4) is younger than the 10-minute cooldown, another runner's charge is at the PSP right now, stand down, in_flight, nothing touched (1327 to 1379). Reads the marker's OWN snapshot (_snapshot_product, pending_reconcile_amount_cents/currency, never the instruction's live columns, W1-fix 5) before releasing the lock to call find_transaction_by_reference (1381 to 1408). Re-locks; if the reference changed under it, resolved_elsewhere (1419 to 1430, NOT skipped, that word means "go ahead and charge", W1-fix 2). unavailable (the lookup could not be made) reschedules and keeps the marker (1432 to 1441); absent (a successful lookup, no match, gated by PAYREXX_TOKEN_CHARGE_ECHOES_REFERENCE, see Gotchas) clears the marker and frees the ordinary retry (1443 to 1462). confirmed/completed: if the snapshot product resolves to None (tier hard-deleted) or the captured amount/currency disagrees with what the claim sent, the marker is KEPT for a human, MANUAL RECONCILIATION (1467 to 1504); otherwise _record_charge with reason="reconcile" and the PSP's own transaction time (1505 to 1519). declined/cancelled/expired/error: clears the marker; if the instruction is already off, clears the obligation with no strike (1521 to 1535); otherwise _record_decline with the snapshot's amount (1539 to 1542). Any other status: reschedule, ask again (1544 to 1553).

run_auto_topup_retry (1695 to 1805): due set is every row topup_status='pending' or carrying a pending_reconcile_ref, with retry_after <= now, ordered by retry_after (1738 to 1751, a stalled row's retry_after IS NULL takes it out for good). Per row, own try/except: reconcile_pending first; charged/declined count as reconciled and stop; unavailable/resolved_elsewhere/in_flight all count as skipped and stop, never fall through to charge_if_due on any of these three (1763 to 1784, each is a documented double-charge regression, W1-fix 2/3); only absent or skipped (no marker) falls through to charge_if_due(reason="retry"); then _stall_if_overdue (1756 to 1795).

_stall_if_overdue (1808 to 1861): a pending row whose pending_since is PENDING_STALL_DAYS (7) old moves to stalled, retry_after = NULL, one mail, unless it still carries a pending_reconcile_ref, in which case it is NOT stalled and stays in the due queue, loudly, because stalling would end the only mechanism that can learn whether a card was charged (1834 to 1846).

Invariants

• A crossing charges at most once, ever: enforced twice, independently, prima_balance_events' primary key (before this module runs at all) and last_crossing_seq under the row lock (module docstring 13 to 18; test_two_concurrent_deliveries_of_one_crossing_charge_once, test_the_balance_event_primary_key_refuses_a_replayed_crossing).
• pending_reconcile_ref is written and committed strictly before Payrexx is contacted, on every charge path, not only after an unknown answer (test_the_claim_writes_the_reference_before_the_card_is_touched).
• The marker survives everything except its own resolution: not a fresh crossing (parked instead), not a decline for a different reference, not a structural block, not enabled=false, not a DELETE (_clear_pending's docstring, 488 to 501; test_a_structural_block_drops_the_obligation_and_keeps_the_question, test_a_late_success_is_recorded_even_after_the_tenant_switched_off).
• "The PSP did not answer" never counts a strike and never sends "nothing was charged" (_classify_charge_failure; test_the_psp_never_answering_is_unknown_and_never_a_decline).
• Two runners can never answer the same question twice: an in-flight claim (last_attempt_at-clocked) stands down without asking the PSP; a charging runner that re-locks and finds its marker gone writes nothing over the other resolver's answer (test_two_runners_racing_on_a_marked_row_charge_once, test_a_crash_between_the_charge_and_the_record_never_re_charges).
• An empty filtered PSP listing is absent (safe to retry) only when PAYREXX_TOKEN_CHARGE_ECHOES_REFERENCE records that the echo was measured live; otherwise it is unavailable and the marker is kept (find_transaction_by_reference 1604 to 1631; test_an_unmeasured_empty_listing_keeps_the_marker_and_does_not_charge).
• A crossing Labs cannot charge right now is parked, never dropped, unless the block is structural, nothing is owed to a tenant who cannot be charged at all (_park_pending vs _clear_pending; test_a_cooldown_blocked_crossing_is_kept_not_dropped, test_a_structural_block_clears_the_obligation).
• The database itself refuses the illegal states: an idle row with an obligation, a pending/stalled row with none, a marker without its snapshot or a snapshot without a marker (ck_prima_auto_topups_pending_shape, ck_prima_auto_topups_marker_snapshot; test_the_database_refuses_an_idle_row_that_still_owes_a_crossing, test_the_database_refuses_a_marker_without_its_snapshot_and_a_snapshot_without_a_marker).

Frontend

The module that would render this, by id:

billing-credits/ui/subscription-manager   the tenant-facing standing-instruction panel this
                                                               engine's charges are the effect of (PUT/GET/DELETE
                                                               /credits/auto-topup, a neighbouring recipe); at
                                                               af7a352 nothing in this repo's frontend calls
                                                               charge_if_due, reconcile_pending or
                                                               run_auto_topup_retry directly, they are reached only
                                                               by the webhook and the scheduler, so this recipe has
                                                               no UI of its own to render, only an effect a
                                                               subscription-manager panel displays (card on file,
                                                               last charge, disabled reason)

Minimum admin and debug surface (no dedicated route exists at this commit, these are the queries and log lines a human runs by hand): SELECT tenant_id, topup_status, pending_reconcile_ref, last_block_reason, retry_after FROM prima_auto_topups WHERE topup_status <> 'idle' OR pending_reconcile_ref IS NOT NULL ORDER BY retry_after (the live queue run_auto_topup_retry will pick up); SELECT tenant_id, pending_reconcile_ref, last_attempt_at FROM prima_auto_topups WHERE pending_reconcile_ref IS NOT NULL AND last_attempt_at < now() - interval '24 hours' (rows past UNKNOWN_ESCALATE_HOURS, the human-needed list); the log lines auto top-up: tenant %s HAS BEEN CHARGED (referenceId=%s, tx=%s) ... MANUAL RECONCILIATION REQUIRED (1476, 1501, the two cases no code path can resolve automatically) and auto top-up: tenant %s has had an UNRESOLVED charge outcome ... since %s (1571 to 1573); SCHEDULER: auto top-up retry, {due, charged, declined, unknown, stalled} once per tick with due > 0 (scheduled_jobs.py 798 to 804).

Security & tenancy

Auth dependency: none in the Labs-session sense, the router is mounted surface_exempt("public", ...) (app_fastapi.py 892), and the only credential is X-Prima-Signature/X-Prima-Timestamp/X-Prima-Nonce verified by prima_inbound_sig.verify (constant-time HMAC-SHA256 compare, hmac.compare_digest, plus a bounded 300-second clock-skew window; 88 to 100). The nonce is mixed into the signed string but is never checked for uniqueness or replay by this module (grep -n nonce finds it only in the canonical string and the header binding), replay protection for the thing that actually costs money is (tenant_id, crossing_seq)'s primary key and the last_crossing_seq second opinion, not the nonce; a captured-and-replayed signature within the 300-second window reaches charge_if_due and is refused there, not at the signature layer. Workspace derivation: tenant_id is taken verbatim from the signed body, mirrored from Prima, never inferred from a session, same rule prima_credit_grants.tenant_id follows (module docstring, prima_auto_topup.py, and prima_inbound_sig.py header comment).

Fail direction: the signature check closed (401 on anything malformed or missing, verify never raises); the seam itself closed when unconfigured (503 before the body is even parsed if PRIMA_LOW_BALANCE_ALERT_SECRET is unset); charge_if_due/reconcile_pending closed on money, every ambiguous PSP answer (unknown, unavailable, an unmeasured empty listing) keeps the marker and refuses to free a retry rather than guessing "no charge happened"; the crossing-record step closed via the database's own primary key, not application logic (_record_crossing, IntegrityError caught and read as duplicate).

Secrets and PII at rest: payrexx_token_ref is a PSP transaction id, not a PAN (_card_details_from_gateway's ⚠️ block accepts a last-four only if it already arrives as exactly four digits, refusing to truncate a masked PAN, auto_topup_service.py 364 to 394, the PCI SAQ-A claim this recipe does not own but must not break); every stored error column and log line carries a class name or HTTP nnn, never a PSP response body. Rate limits: none dedicated to this seam; the webhook is reachable by whoever holds PRIMA_LOW_BALANCE_ALERT_SECRET, described in the router's own docstring as the least powerful credential on the estate (30 to 33), which is why steps 2 to 4 are pushed off the event loop rather than rate-limited. Attacker-controlled inputs, all behind the signature: tenant_id (UUID-typed by Pydantic); crossing_seq (≥1, and the ONLY field whose presence gates a charge at all); balance_micro (deliberately unbounded below zero, a 422 here would strand a tenant in overdraft with neither the mail nor the top-up, ADR-0036 §2 erratum); percent_remaining, allowance_micro, threshold_micro, occurred_at (all display/evidence only, never read back as money).

Tests

Run 2026-09-12 against a scratch database at postgresql://localhost:5432/prima_grants_scratch, migrated to head with alembic upgrade head per tests/prima_credits/conftest.py's own docstring (20 to 34), using the venv at alchemy_labs/backend/.venv against the w1-auto-topup worktree:

PRIMA_CREDITS_TEST_DATABASE_URL=postgresql://localhost:5432/prima_grants_scratch \
    python -m pytest tests/prima_credits/test_auto_topup.py tests/prima_credits/test_auto_topup_retry.py \
    tests/prima_credits/test_auto_topup_outcomes.py -q --no-cov -o addopts="" -p no:cacheprovider \
    --confcutdir=tests/prima_credits

prints 117 passed (0 failed, 160 deprecation warnings unrelated to this recipe). Without PRIMA_CREDITS_TEST_DATABASE_URL set, the same three files print 35 passed, 82 skipped, every scratch-database test in test_auto_topup.py and test_auto_topup_retry.py is gated by pytestmark/the engine fixture (conftest 118 to 137) and skips rather than fails; only test_auto_topup_outcomes.py's 24 tests need no database at all and always run.

Hermetic vs live: nothing here touches a real PSP, charge_token is monkeypatched in every scratch-database test (test_auto_topup.py 6 to 9); test_auto_topup_outcomes.py stubs PayrexxService._make_request directly via _Px. Known-answer vectors, subscription_service.charge_token/_classify_charge_failure (the shared primitive, test_auto_topup_outcomes.py 26 to 197): a parsed {"status":"success"} body is outcome="charged", ok=True; a parsed non-success body is outcome="declined"; ReadTimeout/ConnectTimeout/ConnectionError/SSLError, a 5xx, 408, 429, 401, 403, a missing API credential and an unparsable body are ALL outcome="unknown" (never a strike); a 4xx outside {401,403,408,429} is outcome="declined"; the charge always sends {"amount": amount_cents} and, only when given a reference_id, adds {"referenceId": str(reference_id)[:100]}.

Concurrency proof: test_two_concurrent_deliveries_of_one_crossing_charge_once runs two real threads and two real sessions against one Postgres row with a threading.Barrier, asserts sorted(outcomes) == ["charged", "skipped"] and len(charged) == 1; test_two_runners_racing_on_a_marked_row_charge_once nests a second session's reconcile_pending call inside the first runner's PSP-lookup stub to reproduce PROBE3 exactly at the window it exploits. Database-constraint tests (no application code path can reach these states): test_the_database_refuses_an_idle_row_that_still_owes_a_crossing, test_a_cap_below_one_charge_is_refused_by_the_database, test_the_database_refuses_a_marker_without_its_snapshot_and_a_snapshot_without_a_marker.

What a new app adds and this branch lacks: no dedicated Playwright/E2E spec exists for this seam (there is no browser surface, it is webhook plus scheduler); a hermetic test asserting the nonce is checked for replay independently of crossing_seq, if that guarantee is ever wanted at the signature layer rather than the application layer (Security & tenancy).

Deploy & ops

Config keys per app: PRIMA_CREDITS_MERCHANT_WORKSPACE_ID, PRIMA_LOW_BALANCE_ALERT_SECRET, PAYREXX_TOKEN_CHARGE_ECHOES_REFERENCE (all above). Migrations, in chain order (backend/migrations/versions/): 20260910_0001_prima_auto_topups.py (prima_auto_topups_001, down-revision subscription_cancel_otp_001, re-pointed at that head after a rebase, per the commit this recipe is measured on) creates the table and the standing-instruction columns; 20260910_0002_auto_topup_retry.py (prima_auto_topup_retry_001) adds the ADR-0036 §4a pending-crossing columns and ck_prima_auto_topups_pending_shape; 20260910_0003_auto_topup_reconcile_marker.py (prima_auto_topup_marker_001) adds pending_reconcile_ref; 20260911_0001_auto_topup_marker_snapshot.py (prima_auto_topup_snapshot_001) adds the three snapshot columns and ck_prima_auto_topups_marker_snapshot (W1-fix 5). All four apply cleanly to a fresh alembic upgrade head (verified 2026-09-12 against the scratch database above).

Scheduler: run_auto_topup_retry registered on its OWN IntervalTrigger(minutes=5), coalesce=True, max_instances=1 (scheduled_jobs.py 351 to 359), deliberately not sharing a timer with the one-minute credit-grant drain, because this job talks to a payment provider with a 30-second read timeout and must not delay (or be delayed by) a customer who has already paid (351 to 353, 761 to 764). The webhook's own inline charge_if_due call is the fast path; the scheduler is the durability path for everything the inline call could not resolve.

Argus-style checks: POST /internal/sparks/low-balance-alert with a correctly signed body and no crossing_seq returns 200 {"topup": "skipped"} (the legacy/email-only shape, proves the seam is reachable without risking a charge); SELECT count(*) FROM prima_auto_topups WHERE pending_reconcile_ref IS NOT NULL AND last_attempt_at < now() - interval '24 hours' is the number of tenants needing a human, expected 0 in steady state. Rollback: nothing deployable beyond the code and the four migrations; a prima_auto_topups row is never deleted by this engine (RESTRICT on tenant_id), so a rollback of the code does not need to unwind data, a marker left mid-question by a rolled-back deploy is still resolvable by the next deploy's reconcile.

Gotchas

• 2026-09-12 (measured at af7a352): the branch w1-auto-topup is not an ancestor of alchemy_labs origin/main (git merge-base --is-ancestor fails); maturity: unmerged reflects that, not a claim about code quality.
• 2026-09-12 (measured at af7a352, prima_inbound_sig.py 14 to 27): the sender of POST /internal/sparks/low-balance-alert does not exist yet on Prima's side. Nothing in the alchemy_labs repo calls this endpoint; it is reachable and fully tested from Labs' side, but no crossing has ever been delivered to it outside a test. A rebuild treats this engine as unexercised in production regardless of how green its suite is, until Prima's outbound half is built and the same agent settles both directions (docstring's own words).
• 2026-09-12 (measured at af7a352, core/config.py 610 to 613): PAYREXX_TOKEN_CHARGE_ECHOES_REFERENCE defaults to False. Until Step 10 of the billing runbook measures live that a zahls token charge echoes its referenceId on the filtered GET /Transaction/ listing, every unknown charge that later shows an empty filtered listing is answered unavailable, not absent, the marker is kept, a human is named after 24 hours, and the row simply cannot self-resolve to a retry by lookup alone. This is fail-closed by design (find_transaction_by_reference 1604 to 1631), not a bug, but it means the reconcile path is inert on money until that flag is flipped, and flipping it before the echo is actually measured live re-opens the exact double-debit the flag exists to prevent.
• 2026-09-12 (measured at af7a352): row.amount_cents is deliberately NOT refreshed on the cap_reached branch (814 to 826) even when the tier's price has risen, writing the new price there would violate ck_prima_auto_topups_cap on the way out (monthly_cap_cents >= amount_cents) and turn "this month is spent" into an IntegrityError. The display column stays stale until a charge actually succeeds.
• 2026-09-12 (measured at af7a352): the HMAC nonce on this seam is bound into the signature but never checked for replay independently, see Security & tenancy. The actual protection against a replayed signed request causing a second charge is (tenant_id, crossing_seq), not the nonce; a legacy call with no crossing_seq has no such protection beyond the 300-second skew window, which is acceptable only because a legacy call can never charge (module docstring 35 to 39).
• 2026-09-12 (measured at af7a352): _record_charge's call to prima_credit_grants.enqueue_topup (1119) is unguarded, it raises ValueError on an unresolvable tenant or a non-positive amount, and nothing in _record_charge catches it. The only reason this is safe is that charge_if_due's outer wrapper (585 to 620) catches everything, rolls back, and answers skipped; the claim from step 4 already survived that rollback (it was committed separately, before the send), so the next tick reconciles the late success instead of losing it. A rebuild that removes the outer catch-all without also making the claim durable on its own reintroduces the double-charge this recipe exists to prevent.

Reference implementation

Paths relative to the repo root; all at w1-auto-topup af7a35246dd4a009485bc67b3a13fea6e5a688cc unless noted.

• alchemy_labs/backend/services/commerce/auto_topup_service.py (1912 lines): module docstring (1 to 103); _merchant_workspace_id/resolve_credit_product (133 to 169); persist_token_from_order and the token/card extraction helpers (177 to 405, out of this recipe's scope, cited for context); _set_marker/_clear_marker/_snapshot_product (418 to 477); _clear_pending/_park_pending (480 to 556); charge_if_due/_charge_if_due (585 to 1037); sparks_for_grant (1040 to 1057); _record_charge (1060 to 1176); _record_decline (1179 to 1252); reconcile_pending/_reconcile_pending (1266 to 1553); _escalate_stale_marker (1556 to 1574); find_transaction_by_reference (1594 to 1687); run_auto_topup_retry (1695 to 1805); _stall_if_overdue (1808 to 1861); _owner_email/_send (1869 to 1912).
• alchemy_labs/backend/models/prima_auto_topup.py (358 lines): module docstring; the seven constant blocks (59 to 124); PrimaAutoTopup (127 to 321, all columns and CHECKs); PrimaBalanceEvent (324 to 358).
• alchemy_labs/backend/services/commerce/subscription_service.py: CHARGE_OUTCOMES (230); _classify_charge_failure (233 to 277); charge_token (280 to 338, the shared tri-state primitive); _convert_one (400 to 492, the sibling shape this recipe's claim-before-send mirrors, per its own module docstring line 24).
• alchemy_labs/backend/api/routers/prima_inbound_router.py (255 lines, whole file in scope): module docstring (1 to 49); LowBalanceAlertRequest (75 to 109); low_balance_alert (112 to 156); _alert_side_effects (159 to 199); _record_crossing (202 to 234); _sparks (237 to 254).
• alchemy_labs/backend/services/commerce/prima_inbound_sig.py: module docstring, including the "seam does not exist yet on Prima's side" note (1 to 34); canonical_string/sign (51 to 67); verify (70 to 100); DEFAULT_SKEW_SECONDS = 300 (46).
• alchemy_labs/backend/services/scheduled_jobs.py: the prima_auto_topup_retry job registration (347 to 359); run_auto_topup_retry() (755 to 804).
• alchemy_labs/backend/app_fastapi.py: the router mount, surface_exempt("public", ...) (888 to 892).
• alchemy_labs/backend/core/config.py: PAYREXX_TOKEN_CHARGE_ECHOES_REFERENCE (610 to 613); PRIMA_CREDITS_MERCHANT_WORKSPACE_ID (749); PRIMA_LOW_BALANCE_ALERT_SECRET (780).
• alchemy_labs/backend/migrations/versions/: 20260910_0001_prima_auto_topups.py, 20260910_0002_auto_topup_retry.py, 20260910_0003_auto_topup_reconcile_marker.py, 20260911_0001_auto_topup_marker_snapshot.py.
• Tests as listed under Tests: alchemy_labs/backend/tests/prima_credits/test_auto_topup.py (1216 lines), test_auto_topup_retry.py (2337 lines), test_auto_topup_outcomes.py (393 lines), and conftest.py for the scratch-database setup this recipe's Acceptance checks depend on.

Deviations of the reference from an ideal implementation, all dated 2026-09-12: the Prima-side sender is unbuilt, so this engine is fully unexercised outside tests; the reconcile's absent path is inert until PAYREXX_TOKEN_CHARGE_ECHOES_REFERENCE is measured and flipped; the nonce carries no independent replay check; _record_charge's call into the grant outbox is unguarded and relies on the outer catch-all plus the durable claim rather than its own try/except; no dedicated admin/debug route exists for the two SQL queries under Frontend.

Acceptance checks

Run from alchemy_labs/backend at af7a352 (or an app built to this recipe) with a scratch Postgres migrated to head. Expected observations are exact, as measured 2026-09-12.

1. createdb prima_grants_scratch (if not already present) then SYNC_DATABASE_URL=postgresql://localhost:5432/prima_grants_scratch DATABASE_URL=postgresql://localhost:5432/prima_grants_scratch ENVIRONMENT=development WEBAUTHN_RP_ID_BACKFILL=localhost python -m alembic upgrade head completes with no error and no output beyond the two INFO [alembic.runtime.migration] lines.
2. PRIMA_CREDITS_TEST_DATABASE_URL=postgresql://localhost:5432/prima_grants_scratch python -m pytest tests/prima_credits/test_auto_topup.py tests/prima_credits/test_auto_topup_retry.py tests/prima_credits/test_auto_topup_outcomes.py -q --no-cov -o addopts="" -p no:cacheprovider --confcutdir=tests/prima_credits prints 117 passed.
3. The same command with PRIMA_CREDITS_TEST_DATABASE_URL unset prints 35 passed, 82 skipped (every database-gated test skips cleanly rather than erroring).
4. grep -c "^def test_" tests/prima_credits/test_auto_topup.py tests/prima_credits/test_auto_topup_retry.py tests/prima_credits/test_auto_topup_outcomes.py prints 30, 49, 25 (104 def test_ across the three files); grep -c "@pytest.mark.parametrize" prints 0, 3, 5, eight defs carry a parametrize decorator (the exc/code/method/method,kwargs cases in test_auto_topup_outcomes.py, the mutate,expected/start cases in test_auto_topup_retry.py), which is what takes pytest's collected total for step 2 from 104 defs to 117 passed.
5. grep -n "^def charge_token" services/commerce/subscription_service.py prints line 280, and grep -n "from services.commerce.subscription_service import charge_token" services/commerce/auto_topup_service.py prints one line (121), confirms the shared primitive is imported, not re-implemented.
6. grep -c "^ row\.pending_reconcile_ref = None$" services/commerce/auto_topup_service.py prints 1 (inside _clear_marker alone, 446), no other function in the file nulls the marker directly, which is the guarantee _clear_marker's own docstring (436 to 449) makes.
7. grep -n "PAYREXX_TOKEN_CHARGE_ECHOES_REFERENCE" core/config.py | head -1 shows default=False on line 610.
8. git merge-base --is-ancestor w1-auto-topup origin/main; echo $? (run from a clone with both refs) prints 1, confirms maturity: unmerged.