Developer API keys and security policy

Purpose

A signed-in developer mints a key with the scopes it needs, sees the raw value exactly once, lists the prefixes of the keys they hold in the current workspace, and revokes one so that it stops working here and on Prima in the same transaction. The invariant that makes it trustworthy is that a key of record is a SHA-256 hash with an explicit, enforced scope list: lookup is by hash and never by prefix, permissions is never written empty, a scope the endpoint does not know is refused at mint, and the method matrix, the gateway prefix lock and the security policy (expiry, suspension, use cap, IP allow-list, anomaly score) are applied on every request by the one dependency every Labs route already uses (api/dependencies.py lines 53 to 157, read 2026-09-12 at fd8558f).

When to use

Use it for every self-service credential a person mints from a page: the settings card in the SPA, the developer portal's keys page, the per-session gateway key the chat server mints (B21 calls this route with scopes: ["ai"]). Trigger phrases: "create an API key", "which scopes", "the key is read-only", "revoke a key", "the key works in Labs and not on api.prima.li", "unscoped key", "X-API-Key returns 403 on POST", "the admin minted a key for a client".

Do not use it for the CLI's browser flow (identity-auth/cli-login-scoped-key, B9, which writes the same table with a fixed scope list and no page), for the outbox that carries the key to Prima (identity-auth/key-sync-outbox-labs-to-prima, B12; this recipe calls it and does not describe it), for how Prima validates the key it received (identity-auth/gateway-api-key-auth, B11), for the agent gateway keys (airiel, aurum, mcp:full), which are minted by scripts under review and never by these routes (admin_router.py lines 2768 to 2772; scripts/mint_airiel_key.py, scripts/mint_aurum_key.py), or for the mail relay key, which only the admin route mints (2772 to 2786).

Stack assumptions

Pinned as measured at fd8558f (2026-09-12): fastapi==0.118.2, pydantic==2.12.0, pydantic-settings==2.11.0 (billing_config.py builds a BaseSettings whose Field(env=...) arguments are ignored by v2; the field names happen to equal the variable names, so the binding still works). Foundations assumed: S4 (the router is a module-level APIRouter(prefix="/api/developer"), mounted with surfaces=ALL_SURFACES at app_fastapi.py line 827), S5 (get_current_user, _authenticate_via_api_key, request.state.surface), S6 (get_workspace_id(req) reads request.state.workspace_id that the middleware validated, middleware/workspace_middleware.py lines 393 to 402), S7 (a HTTPException becomes {error, code, request_id, timestamp} with AUTH_001 for 401, AUTHZ_001 for 403, RES_001 for 404, SRV_001 for anything unmapped including 400 and 500; app_fastapi.py lines 731 to 739).

Environment names read by this layer (core/billing_config.py, prefix-less, case-insensitive):

Name	Default	Behaviour
ENABLE_ANOMALY_DETECTION	true (line 175)	runs _check_anomalies on every API-key request; off skips it
FORCE_KEY_ROTATION_DAYS	90 (line 181)	read by nothing outside billing_config.py (grep 2026-09-12); the force_rotation_at column it describes has no Python writer, and a hand-run SQL migration installs a trigger that writes it, so whether rotation is live is a per-database fact (Gotchas)
TRUSTED_PROXY_COUNT (core/config.py line 47)	1	used by utils/net.py and the rate limiter, not by the security service, which reads the leftmost X-Forwarded-For hop (Gotchas)

Nothing here has a fail-closed switch of its own: the policy is code, and its only configuration is the two flags above.

Data model

-- models/api_key_model.py lines 14 to 61; migrations/versions/baseline_prod_schema_001.py 501 to 522,
-- FKs in deferred_fk_phantom_tables_001.py 169 to 179, UNIQUE at 3673.
api_keys (
  id                   uuid PRIMARY KEY,
  user_id              uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE,
  workspace_id         uuid NULL REFERENCES workspaces(id) ON DELETE CASCADE,
  key_hash             varchar(255) NOT NULL,             -- sha256 hex; CREATE UNIQUE INDEX ix_api_keys_key_hash
  key_prefix           varchar(20)  NOT NULL,             -- raw[:12] + "..."; display only
  name                 varchar(255) NOT NULL,
  description          text,
  permissions          jsonb NOT NULL DEFAULT '{"scopes": ["read"]}',   -- {"scopes": [...]}; a missing or empty list is unlimited
  rate_limit_tier      varchar(50) NOT NULL DEFAULT 'standard',          -- read by nothing in this layer
  last_used_at         timestamptz NULL,
  expires_at           timestamptz NULL,
  created_at           timestamptz NOT NULL,              -- model default datetime.utcnow; the dump's literal default only on raw SQL
  is_active            boolean NOT NULL DEFAULT true,     -- revocation is is_active = false, never DELETE
  max_uses             integer NULL,
  uses_count           integer NOT NULL DEFAULT 0,        -- +1 per accepted request
  allowed_ip_addresses jsonb NULL,                        -- a JSON list of exact IP strings; no CIDR
  last_ip_address      varchar(45) NULL,
  anomaly_score        double precision NOT NULL DEFAULT 0,
  suspended_until      timestamptz NULL,
  suspension_reason    text NULL,
  force_rotation_at    timestamptz NULL
);
CREATE INDEX idx_api_keys_user_active ON api_keys(user_id, is_active);
CREATE INDEX idx_api_keys_hash ON api_keys(key_hash);
CREATE INDEX idx_api_keys_workspace ON api_keys(workspace_id);

-- the usage table GET /usage reads (models/api_key_model.py 86 to 121); written by AI routes, not by this layer
token_usage (
  id uuid PRIMARY KEY, user_id uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE,
  api_key_id uuid NULL REFERENCES api_keys(id) ON DELETE SET NULL,
  endpoint varchar(255) NOT NULL, method varchar(10) NOT NULL,
  request_tokens integer, response_tokens integer, total_tokens integer, multiplied_tokens integer NOT NULL,
  model_used varchar(100), status_code integer, error_message text, metadata jsonb,
  created_at timestamptz NOT NULL, usage_source varchar(10) NOT NULL DEFAULT 'web', multiplier_applied double precision NOT NULL DEFAULT 3.0
);

Scope vocabulary, as three sets that must agree (measured 2026-09-12): self-service _SELF_SERVICE_SCOPES = {"read", "write", "delete", "ai"} with default ["read"] (developer_router.py 35 to 36); admin _MINTABLE_SCOPES = {"read", "write", "delete", "admin", "full", "*", "mail"} (admin_router.py 2786); enforcement in _authenticate_via_api_key: the prefix-locked _GATEWAY_SCOPES = {"airiel": "/api/airiel/", "aurum": "/api/aurum/", "ai": "/api/prima/v1/", "mail": "/api/mail/"} (70 to 96), the grant-all set {"admin", "full", "*"} (110), and the method matrix over read, write, delete (108 to 127). The settings card mirrors the four self-service scopes by hand (frontend/src/components/settings/ApiKeysCard.tsx lines 37 to 42).

What is not stored: the raw key after the mint response; a scope's meaning (it is code); the workspace on token_usage (that table is per user).

API surface

All routes on /api/developer, ALL_SURFACES, Depends(get_current_user) (a session or an X-API-Key; a key needs read for the GETs and write for POST, and a key carrying a gateway scope is refused on this prefix by the lock). Tenancy is S6's request.state.workspace_id. The 400s and 500s below render as S7's envelope with code: "SRV_001".

Route	Auth dep	Tenancy	Codes	Notes
POST /api/developer/keys body {name (1..255), description?, expires_in_days? (1..365), scopes?: [str]}	get_current_user	required: 400 "Workspace context required. Please provide X-Workspace-ID header." without a context (88 to 92); the key's workspace_id is that context	200 APIKeyResponse with key set once (194 to 205; no status_code, so 200, not 201); 400 "Unsupported scope(s): x. Allowed: ai, delete, read, write." (100 to 108); 403 "Keys minted from a Prima surface may carry the 'ai' scope only (requested: ...)." when request.state.surface is in PRIMA_SURFACES and any requested scope is not ai (113 to 126); 422 on the Pydantic bounds; 500 str(e) on any other exception after a rollback (208 to 211)	scopes are trimmed, lowercased, de-duplicated and sorted (97); an empty list falls back to ["read"] (98 to 99), so a Prima surface that sends [] or omits scopes is 403, and must send ["ai"] explicitly; enqueues the Prima sync inside the transaction when the key carries ai (171 to 175) and tries one inline delivery after the commit (185 to 191)
GET /api/developer/keys	get_current_user	optional: filtered to the context when present, else every key of the user (229 to 234)	200 [APIKeyResponse] with key: null and, at fd8558f, scopes omitted (240 to 253); 500 str(e)	ordered by created_at descending
DELETE /api/developer/keys/{key_id}	get_current_user	optional, same rule as GET (279 to 288; comment 267 to 278 dates the filter to 2026-08-04)	200 {"message": "API key revoked successfully"}; 404 "API key not found" for another user's key, another workspace's key when a context is present, or an unknown id; 500 str(e) (a non-UUID key_id reaches the database comparison and lands here)	soft delete: is_active = False, then enqueue_revocation in the same transaction (297 to 312), then one inline delivery (316 to 323)
GET /api/developer/usage	get_current_user	none (per user)	200 {total_tokens_used, tokens_remaining, current_plan, usage_this_month, cost_this_month}; 500 str(e)	a placeholder: sums token_usage.total_tokens since the first of the month, prices it at a literal 0.01 per 1000 tokens (354), and reads users.token_quota, tokens_used_current_period, subscription_tier (357 to 362); not the ledger (D2, the prepaid credit in Prima is the one truth)
POST /api/admin/users/{user_id}/api-key body {name, description?, expires_in_days?, scopes?}	AdminUser	the target user's first owned workspace or NULL (2863 to 2866)	200 {id, key, key_prefix, user_id, user_email, name, description, expires_at, scopes} (2904 to 2917); 400 "Invalid user ID"; 404 "User not found"; 422 from the validator on an empty list ("scopes was provided but empty...") or an unknown scope ("Unknown scope(s): ...", 2799 to 2823)	omitting scopes leaves the column default ["read"] (2879 to 2884); writes an admin_service.log_action("create_api_key") row; never enqueues a Prima sync because ai is not mintable here

The response APIKeyResponse (49 to 60): id, key?, key_prefix, name, description, created_at, expires_at, last_used_at, is_active, scopes?. key is present only in the POST response.

Refusals the consuming dependency produces for a key minted here, verbatim from _authenticate_via_api_key: 401 "Could not validate credentials" (unknown hash, the caller's uniform 401), 401 "API key is deactivated" (73 to 76), 403 "This API key is restricted to the <scope> agent API (<prefix>)." (99 to 104), 403 "API key scope does not permit <METHOD> on this resource." (122 to 126), 401 with the security service's message (139 to 146): "API key has expired. Please generate a new key.", "API key suspended: <reason>", "API key has reached maximum usage limit", "Access denied: IP address not allowed for this key", "API key requires rotation. Please generate a new key.", "User account is inactive", "API key suspended due to suspicious activity. Check your email for details."; 403 "User account is disabled" (148 to 152).

Application logic

Mint, create_api_key (developer_router.py 70 to 211), in commit order:

1. get_workspace_id(req); none is 400 before anything is generated.
2. Resolve scopes before minting: requested list or ["read"], normalised and sorted, empty falls back to ["read"], any scope outside _SELF_SERVICE_SCOPES is 400 (94 to 108).
3. Surface rule: if req.state.surface is a Prima surface and any scope is not ai, 403; otherwise the list is set to exactly ["ai"] (110 to 126). A labs session is not narrowed.
4. Generate key = "prima_pk_" + token_urlsafe(32), key_hash, key_prefix; compute expires_at = utcnow + days when given (128 to 136).
5. db.add(APIKey(... permissions={"scopes": scopes}, is_active=True)) (139 to 151).
6. prima_key_sync.enqueue_mint(db, new_api_key) inside the same transaction, not wrapped: a failure to record the obligation fails the mint (153 to 173). For a key without ai this returns None and writes nothing.
7. db.commit(), db.refresh (175 to 176).
8. After the commit, deliver_for_key inside a try; a failure is a warning and the drain retries (180 to 191).
9. Return the raw key once (193 to 205).

Revoke, revoke_api_key (258 to 331): select by id and user, and by workspace when a context is present; 404 when nothing matches; is_active = False; enqueue_revocation in the same transaction (bumps sync_seq on the key's existing outbox row, or creates a desired_active=false row for a pre-seam gateway key; B12); commit; inline delivery in a try.

Consume, _authenticate_via_api_key (dependencies.py 53 to 157), on every request that presents X-API-Key without a bearer token, in order: hash and select by key_hash (61 to 64); unknown returns None and the caller answers the uniform 401; inactive is 401 (70 to 76); prefix lock (77 to 104); method matrix (106 to 127); load the owner (129 to 133); check_api_key_security (139 to 146); owner inactive is 403 (148 to 152).

Policy, check_api_key_security (api_key_security_service.py 26 to 111), in order, each returning (False, message) on the first failure: expiry, with _suspend_key setting is_active = False permanently (55 to 60); suspended_until in the future refuses, in the past clears itself with a commit (62 to 70); max_uses reached deactivates permanently (72 to 77); allowed_ip_addresses set and the client IP not in it records a violation (anomaly_score += 0.5) and refuses (79 to 85); force_rotation_at in the past deactivates (87 to 94); owner inactive (96 to 98); anomaly detection when the flag is on (100 to 106: a country change adds 0.3, more than 100 token_usage rows in five minutes adds 0.5, a score of 1.0 or more suspends for 24 hours, a passing check decays the score by 0.1 and commits); then _update_key_usage bumps uses_count, last_used_at, last_ip_address and commits (108 to 109, 314 to 332). Every step that changes the row commits the caller's session inside the dependency.

Invariants

• The key of record is sha256(raw); lookup is WHERE key_hash = :h, never by prefix, in the mint's own dependency (61 to 64), in the gate (surface_gate.py 97 to 104) and on Prima.
• permissions is never written without a non-empty scopes list by any route: self-service resolves the list before the constructor (94 to 108, test_self_service_keys_are_never_unscoped), admin refuses an empty list at validation and omits the argument to keep the column default (test_empty_scope_list_is_refused, test_omitting_scopes_leaves_it_unset).
• Self-service never mints admin, full, *, airiel, aurum, mail or mcp:full (_SELF_SERVICE_SCOPES); admin never mints a gateway scope (test_gateway_scopes_are_not_mintable, parametrised over airiel, aurum, ai, mcp:full).
• A Prima-surface session mints ["ai"] and nothing else, and only when it asks for exactly that: any other requested scope, including the implicit default read, is 403 (110 to 126, test_api_only_session_cannot_mint_a_labs_api_key).
• A key that declares scopes has them enforced by method; a key carrying a gateway scope is refused outside its prefix; airiel alone is excluded from the enforceable list and aurum alone is not, so an aurum key fails every method rather than becoming unlimited (comment 113 to 119).
• The raw key is returned once, in the POST response, and no later read returns it (key=None at 243; to_dict(include_key=False) at api_key_model.py 66 to 83).
• Revocation is is_active = False plus the outbox row in one commit; the api_keys row is never deleted by this layer, and prima_key_syncs.api_key_id is ON DELETE RESTRICT so a hand DELETE of a synced gateway key fails at the database (B12).
• Every acceptance of a key writes: uses_count + 1, last_used_at, last_ip_address (314 to 332). A read-only endpoint is not read-only for the api_keys table.

Frontend

The frontend modules that render this layer, by id:

identity-auth/ui/api-keys-settings-and-developer-dashboard   /settings/api-keys: ApiKeysCard lists GET /keys, mints with a
                                                             four-scope picker (read, write, delete, ai; the same set as
                                                             _SELF_SERVICE_SCOPES, mirrored by hand), renders the raw key
                                                             into one node once, revokes with DELETE; /developer shows the
                                                             usage placeholder and the docs
identity-auth/ui/developer-portal-keys-page                  /schluessel on the portal: a prima_dev session mints with
                                                             scopes ["ai"] and X-Workspace-ID resolved from
                                                             GET /api/auth/workspaces through chooseTenant (one
                                                             membership or refuse, never a guess); the tenant then
                                                             comes off the stored key record, not the meter. Lists
                                                             prefixes, revokes; the raw key rendered into exactly one
                                                             DOM node (prima-developer w6-konto d5f9aa8,
                                                             src/lib/prima-api.ts lines 46 to 48, 590 to 636,
                                                             672 to 684; src/pages/schluessel.astro 350 to 356)
identity-auth/ui/chat-signin-and-workspace-picker            the chat server mints a per-session ai key through this
                                                             route (B21) and presents it as X-API-Key
identity-auth/ui/admin-users-console                         the admin mint, POST /api/admin/users/{id}/api-key

Minimum admin and debug surface: GET /api/developer/keys with the workspace header (prefix, active, expiry, last use), the log lines Created API key <id> for user <id> in workspace <id> (178) and Revoked API key <id> for user <id> (314), the WARNING lines the security service writes on suspension and violation (283, 305 to 308), the INFO line naming the prefix and the owner's email on every accepted key request (dependencies.py 154), and the admin_action_logs row per admin mint. The suspension fields have no page; SELECT key_prefix, suspended_until, suspension_reason, anomaly_score FROM api_keys WHERE suspended_until IS NOT NULL OR anomaly_score > 0 is the query.

Security & tenancy

Auth dependency: get_current_user on the four developer routes, AdminUser on the admin mint; the consuming dependency is S5's. Workspace derivation: S6's middleware, read through get_workspace_id; POST requires it, GET and DELETE filter by it when present so a client that never sends the header keeps listing and revoking its own keys across workspaces (267 to 278). Fail direction: closed on the mint (unknown scope, missing context, surface rule, outbox write); closed in the consuming dependency, where an exception inside check_api_key_security propagates as a 500 rather than being swallowed (139 to 146 has no try); open in the legacy in-Labs gateway copy, which wraps the same call in a bare except and continues (services/primaapi/identity.py 175 to 182; B11, D3).

Secrets at rest: key_hash only; the raw key is in the POST response body and nowhere else server-side. The admin route puts the raw key in a JSON response an admin sees once and logs the prefix in admin_action_logs, not the key (2887 to 2896).

Rate limits: none on these routes. The security service's rate spike rule counts token_usage rows per key over five minutes (222 to 256) and adds to the score rather than refusing outright.

Attacker-controlled inputs: X-API-Key (hashed before lookup; a wrong key is the uniform 401 and leaks nothing); scopes (validated against a closed set, lowercased, so AI is ai); name and description (bounded, stored verbatim, rendered by the SPA as text); X-Forwarded-For and X-Real-IP, which _get_client_ip (360 to 382) reads leftmost-first without consulting TRUSTED_PROXY_COUNT, so a client behind the estate's proxy can name any IP to the allow-list check and the geo check (Gotchas); key_id on DELETE (compared as a UUID; a malformed one is a 500 with the driver's message on the wire, Gotchas).

Tenancy of the credential: user plus one workspace, set at mint from the session's context; the gateway's tenant is workspace_id (B11), and S6 re-validates X-Workspace-ID for an API-key caller against the key owner's memberships. A key minted from prima_dev or prima_web reaches /api/prima/v1/ and nothing else (the lock plus the ai-only rule); a key minted from labs with read reaches every Labs GET and no gateway call (S5 Gotchas: GET /api/prima/v1/models with a read key is 403 naming ai).

Tests

Backend (alchemy_labs/backend/tests/, run 2026-09-12 at fd8558f):

• test_admin_api_key_scopes.py: 20 passed in 2.82s with no database (-p no:cacheprovider --no-cov -o addopts=""). Cases: test_omitting_scopes_leaves_it_unset (37), test_model_default_is_read_only (41), test_empty_scope_list_is_refused (50), test_whitespace_only_scopes_are_refused (55), test_gateway_scopes_are_not_mintable[airiel|aurum|ai|mcp:full] (62), test_mail_IS_mintable_because_it_fails_closed (70), test_a_mail_key_needs_write_to_actually_send (85), test_typo_is_refused_rather_than_silently_useless (93), test_scopes_are_normalised_and_deduplicated (100), test_every_advertised_scope_is_actually_accepted (106, parametrised over _MINTABLE_SCOPES), test_write_scope_survives_round_trip (111).
• test_surface_scoped_access.py, against the :5434 Postgres: test_api_only_session_cannot_mint_a_labs_api_key (424, a prima_dev token asking for read and write is 403) and test_self_service_keys_are_never_unscoped (440, a labs token with no scopes gets 200, scopes == ["read"], and the row's permissions->'scopes' is non-empty); -k "api_key or unscoped or mint" prints 3 passed.
• test_primaapi_identity.py: 10 passed in 5.82s; test_ai_key_rejected_outside_gateway (68), test_ai_key_not_prefix_blocked_on_own_gateway_path (76), test_read_scoped_key_still_rejected_on_write_elsewhere (84), test_default_read_scope_key_rejected (104) pin the lock and the matrix from the consuming side.
• test_aurum_gateway.py: the aurum prefix lock.
• No test exercises api_key_security_service.py directly (grep 2026-09-12: no test file imports it); the suspension, use-cap, IP and anomaly branches are untested. A rebuilt policy gets one hermetic test per branch with a SimpleNamespace key and a fake request, in the style of tests/prima_keys/test_prima_key_sync.py.

Known-answer vectors: sha256("prima_pk_" + "x" * 43) is 38e293003c756b7d6625c6b33e53b49d1a58ccef3ae8b89f5c9951f6c910b419 (computed 2026-09-12 on both sides); any two implementations that disagree on hashlib.sha256(raw.encode("utf-8")).hexdigest() cannot share a table (Prima's hash_key, prima/auth/keys.py 92 to 99, is the same expression). The scope normalisation vector: input [" Write", "READ", "read", ""] yields ["read", "write"] (line 97).

Hermetic versus live: the admin validator tests and the identity tests are hermetic; the surface tests need Postgres; nothing reaches a deployed host. Playwright gate: none for the card; frontend/e2e/ has no spec for /settings/api-keys at fd8558f.

Deploy & ops

Config keys per app heredoc (S13): ENABLE_ANOMALY_DETECTION (leave the default on, or set false on a host where token_usage is not written and the rate spike rule is noise), FORCE_KEY_ROTATION_DAYS (inert as a setting; the column it names may still be written by a trigger, Gotchas); plus B12's PRIMA_KEY_SYNC_ISSUER, PRIMA_KEY_SYNC_SECRET and PRIMA_INTERNAL_BASE_URL. Set the issuer explicitly on every host: it does not fail closed. _issuer (services/primaapi/prima_key_sync.py 329 to 331) falls back to JWT_ISSUER, which is a str with the default https://labs.alchemy.zuerich (core/config.py 113 to 116), so the raise at 332 to 337 fires only when JWT_ISSUER is itself empty, and the config docstring's promise that an unset issuer makes every mint raise (core/config.py 712) is not what the code does. On a host whose Prima federates a different issuer, an unset PRIMA_KEY_SYNC_ISSUER mints keys that all succeed and then park as stuck with HTTP 400, which is the stranded-key outcome this outbox exists to prevent (B12 Gotchas records the same reading at fd8558f). The other two are genuinely Optional with None defaults, and unset means the outbox queues forever.

Migrations order: api_keys and token_usage are baseline tables; 20260804_0001_prima_key_syncs.py must be applied before a mint with ai succeeds, because enqueue_mint inserts into it inside the mint's transaction. Scheduler jobs: none owned here (prima_key_sync_drain is B12's). Argus checks: none specific; a synthetic GET /api/developer/keys with a monitoring key and its workspace header answers 200 and is the cheapest end-to-end probe of the dependency, the middleware and the router.

Rollback: the routes are stateless apart from the rows; reverting to a build before 24f16ef (2026-08-04) removes the outbox write and reintroduces the stranded-key defect ADR-0026 closed. Revoking every key of a user is UPDATE api_keys SET is_active = false WHERE user_id = :u followed, for ai keys, by a revocation per key through the route or enqueue_revocation in a shell, because a hand UPDATE does not reach Prima.

Gotchas

• 2026-09-12 (measured at fd8558f, D8, fix needed): GET /api/developer/keys omits scopes from every item (240 to 253 pass nine fields and not the tenth), while ApiKeysCard.tsx line 241 renders (k.scopes ?? []).map(...) as badges. The list shows no scope for any key, and the raw-key panel shows scopes only because the POST response carries them (204). The fix is one line, scopes=(key.permissions or {}).get("scopes"), in the list comprehension.
• 2026-09-12 (measured at fd8558f, D8, fix needed): _get_client_ip (api_key_security_service.py 360 to 382) trusts the leftmost X-Forwarded-For value and then X-Real-IP, ignoring TRUSTED_PROXY_COUNT, which utils/net.py (lines 11, 33 to 40) exists to honour. A client can satisfy allowed_ip_addresses by sending the header. Prima's key door reads only the peer address and says why (prima/api/auth.py 91 to 105). The fix is to call utils/net.py's resolver.
• 2026-09-12 (measured at fd8558f, not resolved: the reading that decides it is a live one this wave did not take): force_rotation_at has no Python writer, but it has a candidate SQL one, and which of the two stories is true is a property of each database rather than of the tree. grep -rn force_rotation_at backend/ outside tests returns five files: the model (models/api_key_model.py 50), the baseline DDL (migrations/versions/baseline_prod_schema_001.py 522), the check (services/api_key_security_service.py 88), and migrations/add_api_security_fields.sql at 17, 32 and 102 to 110. That last file is a hand-run migration, in no Alembic chain (grep -rn add_api_security backend/ finds no reference), and lines 105 to 121 of it are CREATE OR REPLACE FUNCTION set_api_key_rotation() setting NEW.force_rotation_at := NEW.created_at + INTERVAL '90 days' followed by CREATE TRIGGER trigger_set_api_key_rotation BEFORE INSERT ON api_keys FOR EACH ROW. The baseline dump cannot rule it out: grep -c "CREATE TRIGGER" migrations/versions/baseline_prod_schema_001.py prints 0, so the dump captured tables, indexes and foreign keys and says nothing about triggers. Where that migration was applied, forced rotation is live and every key self-deactivates 90 days after creation, because the check calls _suspend_key(api_key, "Key rotation required ...", db) with no duration_hours (87 to 94) and _suspend_key with no duration sets is_active = False permanently (274 to 277); where it was not, the check is dead code. FORCE_KEY_ROTATION_DAYS has no reader outside billing_config.py either way, so the 90 days on the wire come from the trigger's literal and not from the setting. The reading that settles it is SELECT tgname FROM pg_trigger WHERE tgrelid = 'api_keys'::regclass AND NOT tgisinternal on each live database; take it before building anything on top of this column, and a rebuilt service writes the rotation date from application code so the answer is in the tree.
• 2026-09-12 (measured at fd8558f): the geo anomaly rule cannot fire: _get_country returns None for every address (190 to 220, the GeoIP call is commented out), so last_country and current_country is always false. The rate spike rule counts token_usage, which only AI routes write, so a key used for forms alone never trips it. Anomaly detection is on by default and does nothing but decay a score that nothing raises except an IP allow-list violation.
• 2026-09-12 (measured at fd8558f): check_api_key_security and _update_key_usage call db.commit() on the request's session inside the dependency (70, 186, 281, 312, 332). Any handler that relies on an open transaction from get_db at the moment the dependency returns has none; and every accepted key request is a write on api_keys, which is why uses_count is a request counter and why an IP allow-list violation is a row change.
• 2026-09-12 (measured at fd8558f): POST /api/developer/keys answers 200, not 201 (no status_code on the decorator, line 70; test_self_service_keys_are_never_unscoped asserts 200). S5's API table says 201 for this route; the code wins and S5's row is the stale one.
• 2026-09-12 (measured at fd8558f, D8): the three except Exception branches (208 to 211, 254 to 256, 328 to 331) put str(e) into a 500 detail, which S7 renders on the wire as error. A DELETE /api/developer/keys/not-a-uuid reaches the database comparison and answers 500 with the driver's message. A rebuilt router validates key_id: UUID and answers 404, and never echoes an exception.
• 2026-09-12 (measured at fd8558f): the default scope is read, so a developer who mints from the Labs card without ticking ai holds a key that authenticates on the gateway and is refused on every completion with 403 insufficient_scope (Prima) or "This API key does not carry the 'ai' scope required for PrimaAPI." (legacy Labs gateway). The card's fourth checkbox is the fix; the wiki page askedquestions/2026-08-20-primaapi-key-minting-needs-a-frontend.md is the record of the period when no page could tick it.
• 2026-09-12 (measured at fd8558f): the legacy prefix deap_pk_ still appears in scripts/mint_airiel_key.py (68) and scripts/mint_aurum_key.py (91), which mint agent keys with it, and in utils/security_utils.py (310), which redacts it; since 0b2519e (2026-08-03) every route in this recipe mints prima_pk_. Both prefixes validate, because validation is the hash.
• 2026-09-12 (measured at fd8558f): GET /api/developer/usage is a placeholder that prices tokens at a literal 0.01 per 1000 and reads per-user quota columns the ledger no longer governs (D2). The developer portal reads GET /api/prima/v1/usage on the key instead (prima-developer src/lib/prima-api.ts 53 for the endpoint map, 1173 for getUsage, which is getWithKey and not the session); the SPA's dashboard still calls this one. The portal reads nothing else from the meter: /usage answers the key's own tenant_id, and the tenant was already known before the mint (next Gotcha).
• 2026-09-12 (measured at prima-developer d5f9aa8; corrects inventory row F8, which is stale at 6322fa3 line 96): the row says the portal's "tenant id resolved from the meter" and takes "X-Workspace-ID from /usage". That was the build before A-2 (a), decided 2026-09-11. src/lib/prima-api.ts 596 to 600 names /usage as the thing that can no longer be the source: Prima refuses a Labs session Bearer there (503 identity_provider_unavailable) and the meter now runs on the per-session key, whose tenant must be known before the mint. The tenant comes from GET /api/auth/workspaces, the caller's own memberships and the same predicate Labs authorises the mint from (602 to 607, listWorkspaces 622 to 624), through chooseTenant, which refuses rather than picks when there is more than one membership; the mint order at 672 begins with that call. Once a key exists, resolveTenantId (631 to 633) is ensureGatewayKey().tenantId with no network call, and schluessel.astro 350 to 356 says the same in the page. F8's own recipe carries this correction; the inventory row needs it too, and this recipe does not write there.
• 2026-09-11 (S5 Gotchas, carried): a key whose permissions is NULL or {} is unlimited. No route at fd8558f writes one; the hazard is a raw INSERT, an ORM call passing permissions=None, or a future scope set that filters every scope out of _enforceable the way airiel is filtered.

Reference implementation

Paths are relative to the alchemy_labs repo root; commit fd8558f on d1a-token-source (backend identical to feat/service-desk), read 2026-09-12.

• backend/api/routers/developer_router.py: prefix import (18), _SELF_SERVICE_SCOPES and the default (35 to 36), CreateAPIKeyRequest (39 to 47), APIKeyResponse (49 to 60), create_api_key (70 to 211: context 84 to 92, scopes 94 to 108, surface rule 110 to 126, mint 128 to 151, outbox 153 to 176, inline delivery 180 to 191), list_api_keys (213 to 256), revoke_api_key (258 to 331), get_usage_stats (333 to 368).
• backend/api/dependencies.py: _authenticate_via_api_key (53 to 157: hash lookup 61 to 64, inactive 70 to 76, _GATEWAY_SCOPES 77 to 104, matrix 106 to 127, security call 139 to 146, disabled owner 148 to 152).
• backend/services/api_key_security_service.py: check_api_key_security (26 to 111), _check_anomalies (113 to 188), _get_country (190 to 220), _get_recent_usage_count (222 to 256), _suspend_key (258 to 283), _record_violation (285 to 312), _update_key_usage (314 to 332), _get_client_ip (360 to 382), the module instance (386).
• backend/models/api_key_model.py: APIKey (14 to 83), TokenUsage (86 to 124).
• backend/api/routers/admin_router.py: the scope note (2760 to 2786), AdminCreateAPIKeyRequest (2789 to 2824), admin_create_api_key_for_user (2827 to 2917).
• backend/core/billing_config.py: enable_anomaly_detection (175 to 179), force_key_rotation_days (181 to 185), the settings class (12, Config 212 to 214).
• backend/app_fastapi.py: the developer router mount (827), the code map (731 to 739).
• backend/migrations/versions/baseline_prod_schema_001.py (501 to 522, 3106 to 3118, 3673), deferred_fk_phantom_tables_001.py (169 to 179, 769 to 778).
• backend/tests/test_admin_api_key_scopes.py, tests/test_surface_scoped_access.py (424 to 464), tests/test_primaapi_identity.py.
• frontend/src/components/settings/ApiKeysCard.tsx: the scope list (37 to 42), list and mint calls (83, 104), revoke (123), the raw key node (150 to 170), the badges (241 to 243).
• prima-developer (w6-konto, d5f9aa8): src/lib/prima-api.ts (the endpoint map 46 to 48, the key's life 127 to 128, tenant resolution 590 to 636 with listWorkspaces at 622 to 624 and resolveTenantId at 631 to 633, ensureGatewayKey and the mint order 662 to 740, getUsage 1173), src/pages/schluessel.astro (350 to 356).
• Wiki philosophers_stone (6322fa3): operations/prima-modules-inventory.md rows B10, F3, F8; platform/architecture.md (the scope note verified 2026-09-04); platform/primaapi.md ("Self-serve signup mints a key Prima cannot see"); askedquestions/2026-08-20-primaapi-key-minting-needs-a-frontend.md.

Deviations of the reference from this recipe, all dated 2026-09-12: the list route drops scopes; the client IP reads an untrusted header; the geo anomaly rule is dead code and forced rotation is live or dead per database, depending on one hand-run migration nobody has read back; the dependency commits mid-request; exception text on the wire; the usage route is a placeholder outside D2's ledger; no test for the security service; 200 where S5 says 201.

Acceptance checks

Run from alchemy_labs/backend at fd8558f with the :5434 test Postgres up and the app on http://127.0.0.1:8000, or from a new service built to this recipe. $T is a labs access token, $W the caller's workspace id, $P a prima_dev access token. Expected observations are exact.

1. python -m pytest tests/test_admin_api_key_scopes.py -q --no-cov -o addopts="" -p no:cacheprovider prints 20 passed; python -m pytest tests/test_surface_scoped_access.py -q --no-cov -o addopts="" -p no:cacheprovider -k "api_key or unscoped or mint" prints 3 passed; python -m pytest tests/test_primaapi_identity.py -q --no-cov -o addopts="" -p no:cacheprovider prints 10 passed.
2. curl -s -X POST -H "Authorization: Bearer $T" -H "X-Workspace-ID: $W" -H 'Content-Type: application/json' -d '{"name":"probe"}' http://127.0.0.1:8000/api/developer/keys -o /dev/null -w '%{http_code}\n' prints 200; the body's key starts with prima_pk_ and is 52 characters; scopes is ["read"]. The same without X-Workspace-ID prints 400 and "Workspace context required. Please provide X-Workspace-ID header.".
3. ... -d '{"name":"probe","scopes":["read","root"]}' prints 400 with "Unsupported scope(s): root. Allowed: ai, delete, read, write."; ... -d '{"name":"probe","scopes":[" Write","READ","read"]}' returns "scopes":["read","write"].
4. curl -s -X POST -H "Authorization: Bearer $P" -H "X-Workspace-ID: $W" -H 'Content-Type: application/json' -d '{"name":"p"}' http://127.0.0.1:8000/api/developer/keys prints 403 with "Keys minted from a Prima surface may carry the 'ai' scope only (requested: read)."; -d '{"name":"p","scopes":["ai"]}' prints 200 with "scopes":["ai"], and psql -c "SELECT status FROM prima_key_syncs ORDER BY created_at DESC LIMIT 1" prints pending or synced (never nothing).
5. With $K the read key from check 2: curl -s -H "X-API-Key: $K" http://127.0.0.1:8000/api/developer/keys -o /dev/null -w '%{http_code}\n' prints 200; curl -s -X POST -H "X-API-Key: $K" -H "X-Workspace-ID: $W" -H 'Content-Type: application/json' -d '{"name":"x"}' http://127.0.0.1:8000/api/developer/keys prints 403 with "API key scope does not permit POST on this resource."; curl -s -H "X-API-Key: $K" http://127.0.0.1:8000/api/prima/v1/models is 403 naming the ai scope. With $A the ai key from check 4: curl -s -H "X-API-Key: $A" http://127.0.0.1:8000/api/developer/keys is 403 with "This API key is restricted to the ai agent API (/api/prima/v1).".
6. psql -c "SELECT uses_count, last_ip_address IS NOT NULL FROM api_keys WHERE key_hash = encode(sha256('$K'::bytea), 'hex')" after check 5 prints 1 | t or more: every accepted request writes the row.
7. With $ID the id from check 2: curl -s -X DELETE -H "Authorization: Bearer $T" -H "X-Workspace-ID: $W" http://127.0.0.1:8000/api/developer/keys/$ID prints {"message":"API key revoked successfully"}; the same DELETE again prints 200 again, not 404 (the filter does not exclude inactive rows, 282 to 288); curl -s -H "X-API-Key: $K" http://127.0.0.1:8000/api/developer/keys now prints 401 with "API key is deactivated".
8. psql -c "UPDATE api_keys SET expires_at = now() - interval '1 minute' WHERE key_hash = encode(sha256('$K2'::bytea), 'hex')" for a fresh key $K2, then one request with it: 401 "API key has expired. Please generate a new key.", and SELECT is_active, suspension_reason prints f | Key expired.
9. grep -n "scopes=" api/routers/developer_router.py | wc -l prints 1 at fd8558f (the POST response only); on a fixed build it prints 2.
10. psql -c "SELECT tgname FROM pg_trigger WHERE tgrelid = 'api_keys'::regclass AND NOT tgisinternal" on each database the service runs against prints nothing on a build where forced rotation is dead code, and trigger_set_api_key_rotation where migrations/add_api_security_fields.sql was applied, in which case every key deactivates 90 days after created_at (Gotchas). Not run 2026-09-12: it needs a live database, and this wave reads code only.
11. python -c "import hashlib; print(hashlib.sha256(('prima_pk_' + 'x'*43).encode()).hexdigest())" prints 38e293003c756b7d6625c6b33e53b49d1a58ccef3ae8b89f5c9951f6c910b419, and in prima python -c "from prima.auth.keys import hash_key; print(hash_key('prima_pk_' + 'x'*43))" prints the same value (both computed 2026-09-12); that agreement is what lets one key row serve both services.