Cloudflare Worker and estate deploy

Purpose

Getting a build from a laptop to the place it serves from, for the four shapes this stack ships: a Worker per client site on Cloudflare (one Worker, one D1, one R2), a container per app on the estate (a directory under /opt/alchemy/<app>/ with a compose file and a 0600 env file, a tagged image, a loopback port behind one host-networked Caddy), a static site as verbatim bytes (rsync into a directory Caddy already serves), and a static site as a Cloudflare Pages direct upload. The invariant that makes it trustworthy is that what runs is reconstructible and refusable: the artefact comes from tracked files or a tagged image, the configuration has exactly one writer, and a ship script that would publish the wrong thing (a bundle pointing at localhost, an empty dist/, a schema behind the code, an image of the wrong architecture) says so and stops before anything is uploaded.

This foundation is the deploy half that every recipe's Deploy & ops section points at. It fixes where config lives (S13's heredoc per app), the order of a backend release (migrate, then swap the tag), how a frontend release becomes live (atomic symlink, or a new Pages deployment), and what a rollback is for each shape.

When to use

Include it from any recipe whose Deploy & ops says "on the estate", "one Worker per site", "the ship script", "rsync the bytes", "a Pages project", "the env heredoc", or "a systemd timer". Trigger phrases: "how do I get this to staging", "add a config key", "the site needs a custom domain", "roll back", "the container will not boot", "the bundle points at localhost", "which host serves this".

Do not use it for what the running app does with its configuration (that is _shared/fastapi-project-layout, S4, and the pydantic validators that refuse a bad value at boot), for the migration files themselves (_shared/alembic-migrations, S8), for the release gate's test content (_shared/test-layout, S9, and the infra-deploy/labs-deploy-runbook-and-preflight recipe, B179), or for the Astro project's own build (_shared/astro-site-skeleton, S3). The SPA's build script and Caddy serving rules are in _shared/react-app-skeleton (S2, Deploy & ops); this recipe adds the ship and rollback around them and does not repeat them.

Stack assumptions

Estate hosts (alchemycloud/cloud-init/base.yaml at b3d34da): Debian 12 with Docker CE from the vendor repository plus the compose plugin, restic, fail2ban, unattended-upgrades, SSH password authentication off, /opt/alchemy created at first boot. Provisioned by OpenTofu from alchemycloud/modules/{project-base,vm} and one envs/<bucket>/ per isolation bucket (labs, commerce, medical); state files are gitignored (*.tfstate, .terraform/). Hosts are x86_64; the laptop that builds is Apple Silicon (see Application logic, image shape). Admin access is SSH over WireGuard with an estate-specific key (SSH_KEY, default ~/.ssh/alchemy_estate in every ship script); public SSH to the estate closed 2026-08-08 (wiki tooling/deployment.md).

Per app on a host: /opt/alchemy/<app>/compose.yml, /opt/alchemy/<app>/.env (mode 600, written only by distribute-env.sh), and for source-built apps src/ (Labs) or src-<tag>/ (Prima, chat). One Caddy container per host, caddy:2-alpine, network_mode: host; on the host its config is a mounted directory (changed by hand on 2026-08-04, wiki platform/alchemy-cloud.md trap 1) and the SPA webroot is a parent-directory mount, with certificates in the named volume caddy_data. The tracked template alchemy_labs/deploy/infomaniak/caddy-compose.staging.yml (fd8558f, last changed 2026-07-27 in 1850c65) predates that fix and still bind-mounts the Caddyfile as a single file (./Caddyfile:/etc/caddy/Caddyfile:ro, line 26); the compose that runs is in no repo. Cite the template for host networking, the parent-directory mount (line 39) and the volume names (lines 41 to 51), not for its Caddyfile line (see Gotchas). App containers publish on loopback only (127.0.0.1:<port>:<container port>) or run host-networked; Caddy reverse-proxies by hostname to the loopback port. Measured ports (wiki platform/alchemy-cloud.md, 2026-08-04 and 2026-09-01): Labs staging 8000, Prima 8090, Prima's own Postgres 5433 (Labs' is 5432), the chat app 4318; Labs prod 8001 (wiki tooling/deployment.md, 2026-09-04).

Config: alchemycloud/env/.env.example is the master sheet, 118 key lines: 18 tagged [AUTO], 37 [YOU] (paste from a vendor), 27 [LATER], 2 [DONE] and 34 with no tag (measured at b3d34da, counting tags on ^[A-Z_]+= lines; the legend on lines 8 to 12 carries one more occurrence of each tag, which is why a bare grep -c prints 19 for [AUTO]; the wiki page says "~95 keys, 17 AUTO" from 2026-07-27 and is behind the file). The filled copy env/.env is gitignored. scripts/distribute-env.sh <app> [--dry-run] renders one app's block and writes it over SSH; targets at b3d34da are nine app names across three hosts (line 21), of which the Prima-relevant ones are labs-staging, labs-prod, prima, prima-chat, aurum, agents. The developer portal is deliberately absent: it is a static build with its API base baked in and has no env (lines 23 to 25). Its ship script, prima-developer/deploy/ship.sh, is untracked in that repo on 2026-09-11 (git status shows ?? deploy/; the on-disk copy is dated 2026-08-04, one day after 4428767, and no commit on any branch carries it), so this recipe reads it from disk, cites its on-disk line numbers, and does not count it as part of the deploy until it is committed (see Deviations). The build guard it relies on is tracked: astro.config.mjs lines 42 to 56 at 4428767.

Cloudflare side: wrangler ^4.0.0 in the site's package.json (emdash-theme-agency at 3dd00a02), @astrojs/cloudflare adapter with output: "server", @emdash-cms/cloudflare d1() and r2() bindings; one wrangler.jsonc per site with compatibility_flags: ["nodejs_compat"], a d1_databases entry (binding: "DB") and an r2_buckets entry (binding: "MEDIA"). No kv_namespaces in emdash-theme-agency/wrangler.jsonc at 3dd00a02, nor in any other site wrangler.jsonc on the laptop (measured 2026-09-11, no count kept because the set of client checkouts on one machine is not a stable population); the wiki's architecture diagram shows a KV box (platform/emdash.md), and Labs' cloudflare_service.py has a create_kv_namespace method (line 184) with no route calling it. Pages: wrangler pages deploy dist --project-name <project> --branch main, a direct upload with no git integration and no build on Cloudflare's side (prima.li/scripts/deploy-index.sh at 03f4205; wiki platform/primaos.md for the marketing project, 2026-08-02). Cloudflare authentication is wrangler's own OAuth token in ~/.wrangler/config/default.toml; there is no CLOUDFLARE_API_TOKEN on the laptop (wiki tooling/deployment.md, re-recorded 2026-08-02). Labs' provisioning service does use CLOUDFLARE_API_TOKEN and CLOUDFLARE_ACCOUNT_ID from its own settings (core/config.py lines 625 to 626) and answers 503 naming the missing variable when either is unset (cloudflare_service.py lines 34 to 43).

Environment names that decide a deploy's behaviour, with fail behaviour:

Name	Where	When missing or wrong
SSH_KEY	every ship script	defaults to the estate key; a wrong key is a refused connection, nothing shipped
SHIP_HOST	prima/deploy/ship.sh, prima-chat/deploy/ship.sh	defaults to the estate VM's public address; set to the WireGuard address when the public one is filtered
PUBLIC_PRIMA_API_BASE, PRIMA_REQUIRE_API_BASE=1	prima-developer/astro.config.mjs lines 42 to 56; set on the build line of deploy/ship.sh (on-disk line 34, untracked)	with the require flag set and the base empty the Astro build throws [prima-api-base-guard] rather than baking a throwaway host; the ship script never produces that failure because it defaults the base to the production API host (line 24) and always passes both, so the guard fires only on a direct PRIMA_REQUIRE_API_BASE=1 npm run build with the base unset
PRIMA_SITE=index	prima.li npm run build:index	without it the build is the marketing site and deploy-index.sh refuses at the dist/pricing/ check
PRIMA_HOST_PORT	/opt/alchemy/prima/.env, read by compose.yml and ship.sh	defaults to 8090
WEB_CONCURRENCY, PORT	Labs Dockerfile CMD	default 2 workers on 8000
FORCE=1	scripts/predeploy_window_check.sh	overrides a busy deploy window; without an API key the check skips with exit 0
PROD_API_KEY, PROD_WORKSPACE_ID	scripts/postdeploy_smoke.sh	the forms and commerce gates cannot run

Data model

Nothing in a database; the deploy's state is on disk and in image tags:

Thing	Where	Written by
the app's configuration	/opt/alchemy/<app>/.env, mode 600	distribute-env.sh only; [AUTO] values are generated once and written back into env/.env so they are stable across deploys (lines 69 to 104)
what is running	the image tag in compose.yml (labs:staging, labs:prod, prima:<utc timestamp>), plus labs:prod-rollback-<ts> tags	the ship script (Prima rewrites the image: line, ship.sh line 72) or the operator (Labs prod, wiki 2026-09-05)
what was shipped	/opt/alchemy/<app>/src/.deployed_commit (Labs)	the operator by hand; no script writes it (measured 2026-09-11: zero hits in scripts/*.sh, prima/deploy/ship.sh, prima-chat/deploy/ship.sh)
the SPA releases	/opt/alchemy/labs-web/releases/r<ts>/ with current a relative symlink; prod under labs-web/prod/	ship-frontend-staging.sh (keeps the last 5, line 86)
the static portal	/opt/alchemy/prima-surfaces/developer/, backup <dir>.bak-<ts>	prima-developer/deploy/ship.sh (untracked on 2026-09-11, see Deviations)
a Worker's variables	Cloudflare, per Worker, set with wrangler's encrypted-variable subcommand (put <NAME>, value from stdin); never in the repo	the operator
a Worker's live version	npx wrangler deployments list	wrangler deploy
a Pages deployment	the project's deployment list; the custom domain attached to the project	wrangler pages deploy
a site's runtime origin	the D1 options row name='emdash:site_url', a JSON-encoded string	wrangler d1 execute <db> --remote --command ... at go-live

Docker on a host keeps every built image (no registry: the estate has none and the VMs hold no git credentials, prima/deploy/ship.sh lines 12 to 16), which is what makes rollback a retag. Backups (/opt/alchemy/backup/, nightly 02:30 pg_dump -Fc per database, 14-day local retention, restic to two object stores) are host-only scripts recorded in the wiki, not in a repo (inventory B185).

NOT stored anywhere: a plaintext credential in a repo, a compose environment: block with operational values (Prima's compose header says why: a value there "travels silently between environments", which is how a model override reached a multi-model host on 2026-08-02), Terraform state in git, wrangler's .wrangler/state/ (see Gotchas: one repo tracks it).

API surface

The deploy has no HTTP API of its own. Endpoints it calls or that gate it:

Call	Used by	Contract
GET /health on the loopback port	ship-infomaniak.sh line 65, the Labs HEALTHCHECK (Dockerfile line 104)	200 is "the process answers"; Labs prod answers 400 on 127.0.0.1 because of the trusted-host list, so the operator checks the public URL (wiki 2026-09-04)
GET /healthz, GET /readyz	prima/deploy/ship.sh line 84 (liveness), Caddyfile.snippet health_uri /readyz	readiness pulls a replica out of rotation on a database blip; liveness would restart it
GET /api/admin/deploy-windows/check with X-API-Key	predeploy_window_check.sh	{level, message}; busy exits non-zero
Cloudflare POST .../d1/database, POST .../r2/buckets, PUT .../workers/domains	cloudflare_service.py lines 58 to 110 and 157 to 172	bearer from settings; a non-success envelope raises CloudflareAPIError
Labs POST /api/emdash/sites/{project_id}/create-d1, create-r2, register, store-token, activate, domain, sync-brand	the provisioning flow (emdash_router.py lines 47 to 194)	tenant-scoped by WorkspaceID; the eight-step order is in the wiki platform/emdash.md
POST /pages/assets/check-missing, POST /pages/assets/upload, POST .../pages/projects/{project}/deployments	the curl fallback when wrangler's fetch stalls (wiki tooling/deployment.md, 2026-07-21)	files hashed as blake3(base64(bytes) + extension)[:32], manifest {"/path": hash}

Application logic

Backend app on the estate, source built on the host (Labs staging, scripts/ship-infomaniak.sh at fd8558f):

1. Refuse an unclean tree: git status --porcelain -- backend scripts Dockerfile must be empty (lines 35 to 40); what runs must be reconstructible from a commit.
2. git archive --format=tar HEAD backend scripts Dockerfile | gzip (line 48): tracked files only, so no virtualenvs, uploads or logs (a hand-rolled tar was 466 MB, the real artefact about 3 MB) and no macOS AppleDouble ._* files.
3. scp the tarball, unpack into /opt/alchemy/<app>/src/, docker build -q -t labs:staging . on the host (line 59).
4. docker compose up -d --force-recreate in /opt/alchemy/<app>/, sleep 25, print docker ps status and the loopback /health code (lines 61 to 66). Production is deliberately not a target of this script (lines 24 to 27).

Backend app on the estate, promoted to production (wiki tooling/deployment.md, verified 2026-09-05 and executed five times on 2026-09-04):

5. Gate: scripts/preflight.sh (Layer 1 hermetic backend E2E on a scratch Postgres, the frontend build, Layer 2 smoke, Layer 3 a real browser submitting every form, Layer 4 a real purchase) exits 0, then scripts/predeploy_window_check.sh exits 0.
6. Build labs:prod-new from the staging source tree that the gate validated; grep a content marker in the tree and inside the built image before going on.
7. Migrate before the swap: docker run --rm --network host --env-file /opt/alchemy/labs-prod/.env -w /app/backend labs:prod-new sh /app/scripts/migrate.sh. migrate.sh holds session advisory lock 8110731 around alembic upgrade head and exits non-zero on failure (the deploy aborts). Boot refuses a database behind the code's head (core/database.verify_schema_at_head(), Dockerfile lines 60 to 70), so a swap before the migration crash-loops.
8. docker tag labs:prod labs:prod-rollback-<ts>, docker tag labs:prod-new labs:prod, cd /opt/alchemy/labs-prod && docker compose up -d.
9. Write .deployed_commit; run scripts/postdeploy_smoke.sh against the public hosts (marketing 200, API /health, SPA shell mounts, forms gate and commerce buy tagged deploy-gate).

Backend app on the estate, tarball with a timestamp tag (Prima, prima/deploy/ship.sh at cd2c248; the chat app, prima-chat/deploy/ship.sh at e606308, is the same shape with rsync -az --delete and an explicit exclude list in place of the tar, lines 37 to 47 and 62 to 67, into a fresh src-<TAG>/ holding two trees; its header says why rsync, and wiki trap 3 prefers it):

10. COPYFILE_DISABLE=1 tar of prima migrations scripts alembic.ini requirements.txt Dockerfile deploy with ._*, .DS_Store, __pycache__, .git, .venv, .env excluded (lines 37 to 41), unpacked into /opt/alchemy/prima/src-<TAG>/, then find -name '._*' -delete again on the host.
11. docker build --build-arg PRIMA_BUILD_ID=<TAG> -t prima:<TAG> -t prima:latest . on the host (line 60).
12. Install deploy/estate/compose.yml over /opt/alchemy/prima/compose.yml and rewrite its image: line to the new tag (lines 70 to 72); the compose file is owned by the repo, the tag line by the ship.
13. Env gate: /opt/alchemy/prima/.env must exist and be mode 600 or the ship stops with the distribute-env.sh prima command to run (lines 76 to 79). .env is never in the tarball.
14. docker compose up -d, sleep 10, curl /healthz on the loopback port; on failure print the docker logs command and exit 1; on success print the rollback command shape with a <prev> placeholder where the tag goes (line 93); the candidate tags are the docker images lines the build step printed (line 65), and the operator fills one in. The chat ship goes further and retags the running image as rollback-<TAG> before the swap, then prints a command naming that tag (prima-chat/deploy/ship.sh lines 83 to 96 and 133 to 134); see Deviations. Prima's migrations are not run by the ship: make migrate (alembic upgrade head against prima_alembic_version) is a separate step and Prima has no boot-time head check (see Gotchas).

Image built on the laptop (Argus, argus/deploy/push.sh at e42e467):

15. docker build --platform linux/amd64 (line 114), then assert docker image inspect --format '{{.Architecture}}' is amd64 (lines 118 to 120) because a stale same-tagged local image would ship arm64 and crash-loop with exec format error.
16. docker save | gzip -1 | ssh 'docker load' (line 123); compose, units and scripts copied with sudo tee; .env installed at mode 600 root-owned from a local file, never passed as an argument (line 145).

Frontend, the SPA release (scripts/ship-frontend-staging.sh at fd8558f; the build itself is S2):

17. Build with scripts/build-frontend-staging.sh, unpack the tarball locally, count files containing localhost:8000 (must be 0) and files containing the API host (must be more than 0); refuse otherwise (lines 52 to 72).
18. Unpack into releases/r<ts>/, ln -sfn releases/r<ts> current (relative, because an absolute link does not resolve inside the Caddy container), prune to the last five (lines 80 to 86). No Caddy reload: the parent directory is the mount, so the symlink is live.
19. Verify what is served, not what was uploaded: fetch / through --resolve to the host, find the entry chunk, fetch it, and roll back (--rollback re-points current at the previous release) if the served bytes contain the forbidden host or never mention the API host (lines 89 to 110).

Frontend, the static portal (prima-developer/deploy/ship.sh, untracked on 2026-09-11 and read from the on-disk copy dated 2026-08-04; line numbers are that copy's; the build guard is tracked at 4428767, astro.config.mjs lines 42 to 56):

20. Build with the API base baked in and required (line 34: the base defaults to the production API host at line 24, and both variables are always passed, so the guard cannot fire from this script); refuse if dist/index.html is missing or empty (line 39), refuse if any localhost:<port>, 127.0.0.1 or the staging API host survives in dist/ (lines 43 to 47).
21. sudo cp -a the live directory to <dir>.bak-<ts>; rsync -az --delete --exclude='._*' --exclude='.DS_Store' dist/ into the directory Caddy already serves; chmod -R a+rX so the container's unprivileged Caddy can read it (lines 51 to 68). Rollback is the printed rsync from the backup.

Frontend, a Pages direct upload (prima.li/scripts/deploy-index.sh at 03f4205):

22. npm run build:index (PRIMA_SITE=index astro build), then check dist/robots.txt, dist/sitemap.xml, dist/index.html exist and dist/pricing/index.html does not (the index build carries no marketing pages).
23. Print, do not run, npx wrangler pages deploy dist --project-name <project> --branch main. The founder runs it (wave 0 decision: no deploy by an agent).

A Worker per client site (S3 Deploy & ops for the build; this recipe for the rest):

24. Provision D1 and R2 (Labs create-d1, create-r2, or npx wrangler d1 create / r2 bucket create), write their names and the real database_id into wrangler.jsonc (the template ships "local"), npm run deploy (astro build && wrangler deploy).
25. Set the Worker's encrypted variables (the email transport key at least, or magic-link sign-in fails with "Email is not configured"), redeploy.
26. Point emdash:site_url at the custom domain with a live D1 write (the value is a JSON string, quotes included); attach the custom domain; when the hostname is already a custom domain on an older Worker, the plain PUT /workers/domains fails with 100116 already in use and the flip is a PUT with {hostname, service, zone_id, override_existing_origin: true, override_existing_dns: true} (wiki platform/emdash.md, 2026-07-23).
27. After deploy: npx wrangler deployments list shows one live version; npx wrangler tail <worker> --format json is where an empty 404 shows its stack; verify every uploaded object is really in R2 by crawling the live pages (a wrangler r2 object put loop can report success for objects that did not land).

Host timers (aurum/deploy/systemd/aurum-collect.service and .timer at a17fce3): a Type=oneshot unit whose ExecStart is curl -fsS -K /etc/<app>-cron.curlrc --max-time <s> -X POST http://127.0.0.1:<port>/api/<job>; the bearer lives in the 0600 root-owned curlrc, never in ExecStart (visible in systemctl show, ps and the journal); SuccessExitStatus=0 so a failed run is red in systemctl list-timers; the timer has OnCalendar in Europe/Zurich, RandomizedDelaySec, Persistent=true so a reboot catches up instead of skipping a day.

Invariants

• Exactly one writer for /opt/alchemy/<app>/.env: distribute-env.sh, for the nine apps its usage line targets (scripts/distribute-env.sh line 21); Argus is not one of them and argus/deploy/push.sh installs /opt/alchemy/argus/.env itself (step 16, Deviations). A hand edit on the host is reverted by the next distribute, so a fix on a box is fixed in the heredoc in the same session (wiki, 2026-08-02, "that pair is the rule").
• Adding a key to env/.env changes nothing until the app's heredoc names it (wiki trap 2, 2026-08-04).
• [AUTO] values are minted once and persisted; production crypto keys are require()d and never minted (distribute-env.sh lines 49 to 67: a freshly generated Fernet key over existing ciphertext looks valid and makes every stored token undecryptable).
• A value shared by two apps (the key-sync and credit-grant seam material, the Ed25519 signing seed) is ensure()d under both apps from the same master variable; Prima refuses to boot when any two of its three seam values are equal (lines 139 to 179).
• The artefact is tracked files (git archive, Labs), an excluded-list tar with COPYFILE_DISABLE=1 (Prima), or an rsync with the same explicit exclude list (._*, .DS_Store, .env, .git/, the chat app) into a fresh src-<tag>/ directory; never a bare copy of a working directory, never a sync in place over the tree the running image was built from, and .env is excluded by every one of the three (ship-infomaniak.sh line 48, prima/deploy/ship.sh lines 37 to 41, prima-chat/deploy/ship.sh lines 37 to 47). The rsync path ships the tree as it is on disk and refuses nothing (see Deviations).
• Migrate before the tag swap, from a one-off container, never from the image CMD (migrate.sh header: N replicas would race the same DDL; Dockerfile lines 60 to 70).
• An image built on Apple Silicon for the estate is built with --platform linux/amd64 and its architecture is asserted after the build.
• A frontend ship checks the served bytes for the forbidden host and the required host, and rolls back on its own verdict; a check that only inspects the uploaded artefact is not a check.
• A ship script prints its rollback command on success.
• No deploy of production by an agent; the agent writes a self-verifying promote script and the operator runs it (wiki 2026-09-04; wave 0 blocker record).
• One Worker, one D1, one R2 per client site; the site's variables live in the Worker, the site's origin lives in D1's options, nothing of either in the repo.
• Caddy is reloaded, never restarted (a restart drops every vhost); its config is a mounted directory, never a single bind-mounted file (wiki trap 1, 2026-08-04).

Frontend

The surfaces that render this foundation, by inventory id:

• infra-deploy/ui/deploy-and-ops-cli-surfaces (F88, CLI): the operator's screen. preflight.sh layer verdicts; predeploy_window_check.sh level and message; ship-infomaniak.sh artefact size and /health code; ship-frontend-staging.sh file counts and "served output references "; prima/deploy/ship.sh tag, liveness, the last five image tags and the printed rollback shape; distribute-env.sh --dry-run with every value masked to six characters (line 803); deploy-index.sh page count and the printed deploy command; argus/deploy/push.sh architecture assertion. Every one prints observations, not intentions, never echoes a value, and uses its exit code as the contract.
• monitoring-ops/ui/argus-telegram-surface (F86, Telegram): scheduler.expected_jobs (are the host timers still installed), scheduler.clock_skew, hosts.timer_health, apps.caddy, apps.redis (argus/argus/checks/scheduler.py header, argus/argus/checks/apps.py, at e42e467).
• monitoring-ops/ui/admin-dashboard-and-system-logs (F85, SPA admin): /health exposes email_redirect_all_to (null in normal operation, health_router.py line 36); /health/detailed reports config presence only (email_service keyed on whether RESEND_API_KEY is set, line 103), not the transport in use.
• content-cms/ui/* and the Labs EmDash pages: the provisioning steps (create-d1, create-r2, register, activate, domain) as buttons whose outcome is the Cloudflare envelope rendered as copy.

Minimum admin surface for a new app: the ship script's own output (above), a /health that names its build (PRIMA_BUILD_ID is baked at build and reported as version in Prima's service_start event), and the .deployed_commit file until a script writes it.

Security & tenancy

Credentials at rest: on the laptop in env/.env (gitignored, backups env/.env.bak-* also gitignored); on a host in /opt/alchemy/<app>/.env at mode 600, one file per app so a host running two apps holds two disjoint sets (the chat app never sees the PSP values, the agents box never sees Labs' database password). The MEDICAL bucket reuses nothing from the sheet (env/.env.example header). Worker variables are per Worker on Cloudflare. Wrangler's OAuth token is in wrangler's own config file; an agent never reads that file.

Fail direction: every ship script fails closed on its content check (nothing uploaded) and on its env gate (nothing started); migrate.sh fails the deploy rather than the boot; boot fails closed on a schema behind head and on a bad credential (S4's validators), which on 2026-08-04 took the in-Labs gateway down with Labs, the reason Prima is its own container. Backups report a per-layer failure without aborting the other layers and refuse to report success on a zero-dump run (wiki 2026-08-04).

Network: app ports on loopback or host-networked, never published on a public interface; TLS and the allowlist are Caddy's; unmatched hostnames get a plain 404 so an unknown Host does not look healthy (Caddyfile.staging header). Security groups are allow-only and pg_hba is the real enforcement (inventory B180). Admin path is WireGuard by key, not by source address (wiki 2026-08-07: the ISP allow-list never converged).

Attacker-controlled inputs the deploy touches: a customer's hostname in add_custom_domain (sent to Cloudflare verbatim; the route is tenant-scoped and the site row must belong to the caller's workspace, emdash_router.py register after the 2026-09-03 fix); the X-API-Key on the deploy-window check (an admin key). Rate limits: Let's Encrypt's five identical certificates per week is the one that locks you out; the staging Caddyfile carries the ACME staging directory line commented out for rehearsals.

Tests

The deploy is tested by its gates, which are commands, not pytest:

• scripts/preflight.sh (S9 for the content): Layer 1 runs the backend E2E against a throwaway Postgres named preflight_scratch_<pid>; the build layer runs tsc and vite build; Layers 2 to 4 run Playwright against the staging hosts (frontend/e2e/staging-smoke.spec.ts, forms-gate.spec.ts, commerce-gate.spec.ts). Exit 0 is the only green. Both host defaults were re-pointed at the estate on 2026-09-01 (0c136f1) after the 2026-08-29 platform move had left them silently passing against a dead previous host that still answered 200 (comment at lines 29 to 33, defaults at 34 to 35).
• scripts/postdeploy_smoke.sh: the same forms and commerce gates against production, writing deploy-gate tagged rows; needs the prod key and workspace id.
• ship-frontend-staging.sh's served-output check is a test that runs on every ship and rolls back on failure.
• distribute-env.sh --dry-run is the hermetic test of a heredoc change: it renders the block with values masked and touches no host.
• sync-prod-to-staging.sh --from-staging (alchemycloud/scripts/) rehearses dump, scratch restore, alembic upgrade head in the app image and verify with no external credential; it never writes production (lines 17 and 22 to 27).
• Hermetic Worker test: astro build && wrangler dev against the local Miniflare D1 (.wrangler/state/), then the S3 launch checks; live: wrangler deployments list and a crawl of the custom domain.
• Known-answer vectors: the Pages file hash blake3(base64(bytes) + ext)[:32] for one fixture file (the wiki's curl path depends on it); the [AUTO] generators' shapes (token_urlsafe(36), a 32-byte url-safe base64 Fernet key, an unpadded 32-byte Ed25519 seed on one line, distribute-env.sh lines 37 to 46).

Deploy & ops

Config keys per app heredoc (distribute-env.sh, key names only; values never appear in a recipe): labs-prod (lines 215 to 361) carries the database as DB_HOST/DB_PORT/DB_NAME/DB_USER/DB_PASSWORD (not a DATABASE_URL; wiki trap 2 of 2026-07-27), REDIS_URL, ENVIRONMENT, the signing and encryption keys, FRONTEND_URL, API_BASE_URL, ALLOWED_ORIGINS, ALLOWED_HOSTS, the WebAuthn pair, OAuth client pairs, the email transport block (EMAIL_TRANSPORT explicit, ADR-0023), SMS, PSP, STORAGE_BACKEND with the object-store block, the AI provider tokens, the Telegram bot token, the Prima seam block (PRIMA_KEY_SYNC_*, PRIMA_INTERNAL_BASE_URL, PRIMA_CREDIT_GRANT_SECRET, PRIMA_CREDITS_*), JWT_SIGNING_KEY_SOURCE and JWT_ISSUER. labs-staging (362 to 551) is rendered from LABS_* master variables and is not the same shape: it renders no WEBAUTHN_* key at all (below). prima (552 to 675) carries its own PRIMA_DATABASE_URL on 5433, the upstream token mapped from the shared provider token, the three seam values, PRIMA_NODE_ID and PRIMA_NODE_PEERS. prima-chat (676 to 760) carries the gateway and sign-in origin and the git-hosting token. A new key is added in three places in one change: env/.env.example with its tag, the ensure/require block if it is [AUTO], and the heredoc of every app that reads it. S8 names WEBAUTHN_RP_ID_BACKFILL for the revision webauthn_rp_id_001; no app block renders it at b3d34da (grep -n WEBAUTHN_RP_ID_BACKFILL scripts/distribute-env.sh is empty), and that is not the gap. The revision resolves the backfill value in order: WEBAUTHN_RP_ID_BACKFILL, then WEBAUTHN_RP_ID, then settings.WEBAUTHN_RP_ID (alchemy_labs/backend/migrations/versions/20260803_0001_webauthn_rp_id.py at fd8558f, lines 17 to 21 and 75 to 76), and refuses only when the inferred value is empty or one of the placeholders localhost, 127.0.0.1, ::1, example.com, test (line 67). labs-prod renders WEBAUTHN_RP_ID and WEBAUTHN_ORIGIN (lines 254 to 255), so step 7, which runs with --env-file /opt/alchemy/labs-prod/.env, infers the production RP and proceeds on a fresh host. labs-staging renders neither key: grep -n -i webauthn scripts/distribute-env.sh hits 244, 254 and 255, all inside the labs-prod heredoc (215 to 361), and nothing between 362 and 551. On a staging host the app therefore runs on the config default WEBAUTHN_RP_ID=localhost (core/config.py line 140), and a never-migrated staging database refuses at webauthn_rp_id_001 because the inferred localhost is a placeholder. The estate's databases are past that revision, so the refusal is invisible there. The missing keys are WEBAUTHN_RP_ID and WEBAUTHN_ORIGIN in the labs-staging heredoc; recorded as a dated deviation below. WEBAUTHN_RP_ID_BACKFILL stays the explicit override S8 describes and is optional wherever WEBAUTHN_RP_ID is rendered.

Files a new estate app needs, in the shape the reference files have (prima/deploy/estate/compose.yml lines 28 to 72 at cd2c248; alchemy_labs/deploy/infomaniak/Caddyfile.staging lines 28 to 151 at fd8558f). Values never appear in either: the compose reads the 0600 env file, the site block names a loopback port. A compose service:

services:
  app-db:
    image: postgres:16-alpine
    restart: unless-stopped
    environment:
      POSTGRES_USER: app
      POSTGRES_DB: app
      POSTGRES_PASSWORD: ${APP_DB_PASSWORD:?APP_DB_PASSWORD missing, run distribute-env.sh <app>}
    ports: ["127.0.0.1:<db host port>:5432"]
    volumes: [app-pgdata:/var/lib/postgresql/data]
    healthcheck: { test: ["CMD-SHELL", "pg_isready -U app -d app"], interval: 5s, timeout: 3s, retries: 20 }
  app:
    image: app:latest              # the ship rewrites this line to the tag it built
    restart: unless-stopped
    depends_on: { app-db: { condition: service_healthy } }
    env_file: [/opt/alchemy/<app>/.env]
    ports: ["127.0.0.1:${APP_HOST_PORT:-<port>}:<container port>"]
    volumes: [app-state:/var/lib/app]
volumes: { app-pgdata: {}, app-state: {} }

No operational environment: entries on the app service (the :? on the database password is the one exception, and it names the command that fixes it); the port is loopback; state that must survive a restart is a named volume. The host Caddy site block for an API, in the Caddyfile directory the container mounts:

{
	# acme_ca https://acme-staging-v02.api.letsencrypt.org/directory   # uncomment for a rehearsal, then reload for the real certificate
}
api.<host> {
	encode {
		zstd
		gzip
		match {
			header Content-Type application/json*
			header Content-Type application/javascript*
			header Content-Type text/html*
			header Content-Type text/plain*
			header Content-Type text/css*
			header Content-Type image/svg+xml*
		}
	}
	request_body {
		max_size 100MB
	}
	reverse_proxy 127.0.0.1:<port>
	handle_errors {
		respond "upstream error {err.status_code} reaching <app>" {err.status_code}
	}
}

No header Access-Control-Allow-* and no security headers (the application emits them; a duplicate fails closed in the browser), no header_up Host (Caddy 2 forwards the client's Host), the encode list explicit so text/event-stream is never buffered, every block host-scoped so an unknown Host gets Caddy's own 404. A single upstream gets no health_uri (a passive check that trips turns a recoverable 502 into an outage, Caddyfile.staging lines 130 to 131); the replica case with health_uri /readyz is prima/deploy/Caddyfile.snippet lines 37 to 47. caddy reload inside the container after a change, never a restart.

Migrations order per shape: Labs, migrate.sh from a one-off container before the tag swap (never alembic upgrade head inside the image CMD; the 2026-08-25 crash loop was a shipped migration never applied); Prima, make migrate against prima_alembic_version with the Labs-tables guard, before ship.sh; a Worker, D1 schema comes from the CMS's own migrations at first request, and a seeded D1 is loaded with a custom UTF-8 dumper in dependency order (wiki 2026-07-23).

Scheduler jobs: none of the deploy's own; the app's scheduler restarts with the container (S12). Host timers: pg-backup.timer 02:30 on every host (wiki), the app timers under deploy/systemd/ per app, argus-watchdog.timer.

Argus checks that watch a deploy: apps.caddy, apps.redis, apps.scheduler_jobs, hosts.timer_health, scheduler.expected_jobs, backups.* (layer 3 asked directly). A deploy that stops a job or a timer shows up at the next digest, not at the ship.

Rollback per shape: Labs backend, in this order: (1) if a migration shipped, docker run --rm --network host --env-file /opt/alchemy/labs-prod/.env -w /app/backend labs:prod sh -c 'alembic downgrade <previous head>' from the new image, which is still tagged labs:prod at this point, because only that image's migrations/versions/ holds the revision and its downgrade(); the old image cannot run the downgrade, upgrade head or even current against a database ahead of its chain (S8 Application logic step 7, Acceptance check 5); (2) docker tag labs:prod-rollback-<ts> labs:prod, the backup tag step 8 made before the swap; (3) cd /opt/alchemy/labs-prod && docker compose up -d, and the old image's boot check now finds the head it expects. Retagging labs:prod to labs:prod-rollback-<ts> is step 8's pre-swap backup, not the rollback; run during an incident it re-tags the broken image and restarts it; Prima, rewrite the image: line to the previous timestamp tag (ship.sh prints the command with a <prev> placeholder and the last five tags; the chat ship prints it with the real rollback-<TAG>); SPA, ship-frontend-staging.sh --rollback (previous release directory); static portal, the printed rsync from <dir>.bak-<ts>; Worker, wrangler rollback to the previous deployment (D1 content is unaffected); Pages, re-deploy the previous dist/ or promote the previous deployment in the dashboard. Restarting the old Labs container between the migration and the swap fails too: its own head check now fails (wiki 2026-09-05).

Gotchas

• 2026-09-11 (measured at fd8558f, deploy/infomaniak/docker-compose.infomaniak.yml lines 33 to 45 and 85 to 89): the repo's compose template has a migrate one-shot service and gates the backend on service_completed_successfully; the compose file on the production host has no such service and just runs uvicorn (wiki 2026-09-04, "Migrations are manual"), and that file is in no repo. The wiki describes what runs; the template describes a plan. Until the host compose is tracked, step 7 is by hand.
• 2026-09-11 (measured at fd8558f): scripts/ship-infomaniak.sh runs docker compose up -d --force-recreate (line 63) and never calls migrate.sh; the 2026-08-25 staging outage (15 minutes of crash loop, /health answering 000 internally and 502 publicly) was a branch with a new migration shipped through exactly this path. Run the one-off migration container first, or add it to the script.
• 2026-09-11 (measured): no script writes .deployed_commit; ship-infomaniak.sh updates src/ and leaves the file stale (wiki 2026-09-04). Grep a content marker in the tree and in the built image; do not trust the file.
• 2026-09-11 (measured at b3d34da, distribute-env.sh lines 197 to 800): the heredocs are unquoted (cat <<EOF), so backticks or $(...) even inside a comment are executed at render time, and an unquoted value containing & is truncated by set -a; . file (wiki trap 2, 2026-08-04: a correct SMTP password arrived broken). env/.env values with shell metacharacters are quoted; a heredoc comment never contains a backtick.
• 2026-09-11 (measured at b3d34da, distribute-env.sh lines 69 to 104): ensure() once generated nothing because the trailing # [AUTO] comment counted as a value, and once killed the script because a missing key made grep exit 1 under pipefail; both are fixed in the function with the reason in comments. gen() deliberately has no | head -c N (SIGPIPE under pipefail killed the script intermittently, lines 33 to 36).
• 2026-09-11 (measured at b3d34da): scripts/rotate-infomaniak-token.sh is untracked in git (git status shows ??), and the wiki inventory (B183) already records it. A script that only exists on one laptop is not part of the deploy.
• 2026-09-11 (measured in the prima-developer working tree, main at 4428767): prima-developer/deploy/ship.sh is untracked too (git status --short shows ?? deploy/, git log --all -- deploy/ship.sh is empty, the file is dated 2026-08-04). The same rule applies: the portal's publish path is a laptop script until it is committed, the line numbers this recipe cites are the on-disk copy's, and a verifier without that laptop can run only the tracked half (the astro.config.mjs guard). Commit it on main and re-record the measured commit.
• 2026-09-11 (measured at b3d34da): env/.env.example has 118 key lines, 18 of them tagged [AUTO] (a bare grep -c says 19 because the legend line counts); the wiki's platform/alchemy-cloud.md (2026-07-27) says about 95 and 17. Code wins; the numbers in this recipe are the file's, counted on key lines.
• 2026-09-11 (measured at 3dd00a02): emdash-theme-agency has no .gitignore and tracks 16 files under .wrangler/ (deploy/config.json, the Miniflare D1 and cache sqlite files, 2026-09-11 working tree shows them modified). Local Worker state in git is noise at best and a leaked seed at worst; a site repo ignores .wrangler/ and dist/.
• 2026-09-11 (measured at fd8558f, cloudflare_service.py lines 157 to 172): Labs' add_custom_domain is a plain PUT with hostname, service, environment; when the hostname is already a custom domain on another Worker, Cloudflare answers 100116 already in use. The flip that moves a domain needs override_existing_origin and override_existing_dns (wiki 2026-07-23, done by hand with the API). The service cannot re-point a domain today.
• 2026-09-11 (measured at 03f4205, prima.li/scripts/deploy-index.sh lines 10 to 16): never run wrangler pages project create for a site in this repo without --force. Without it Cloudflare's Pages-to-Workers delegation takes the project, rewrites astro.config.mjs for the Workers adapter and empties dist/. The project already exists; if it ever has to be recreated, the exact command is in the script header.
• 2026-09-11 (wiki platform/primaos.md, 2026-08-02): two Pages projects on one account are easy to confuse, and --branch=main is required on wrangler pages deploy because a repo on a detached HEAD makes wrangler pick a stray branch and publish a preview URL instead of production. Check --project-name and --branch every time.
• 2026-09-11 (wave 0 record, inventory): index.prima.li was created and deployed three times and had its custom domain attached by an agent session before build:index existed, so the host served the marketing site as an indexable duplicate; the founder decides the remedy. The rule that follows is in Invariants: an agent prints the deploy command and does not run it.
• 2026-09-11 (wiki tooling/deployment.md, 2026-07-21): on at least one machine wrangler pages deploy stalls after about 18 API calls (Uploading... (18/27), fetch failed) while curl to the same endpoints is reliable; the seven-step curl path (refresh the OAuth token, create the project, hash, upload token, check-missing, upload in batches of about five, create the deployment with the manifest) is the workaround.
• 2026-09-11 (measured at cd2c248): prima/deploy/ship.sh has no migration step and Prima has no boot-time head check (git grep -n alembic cd2c248 -- 'prima/*.py' hits four files, config.py, content_free.py, db.py, schema_guard.py, all docstrings and message strings, no call); a shipped migration that is not applied fails at the first query that needs it, not at boot. make migrate is a separate operator step before ship.sh.
• 2026-09-11 (measured at cd2c248, deploy/Caddyfile.snippet header): the per-node job-poll routing in Caddy is an optimisation in a different repository from the code it accelerates; correctness is the application's forward, and a non-zero forwarded=true rate is the drift signal, not an outage.
• 2026-09-11 (wiki platform/alchemy-cloud.md, 2026-08-04, trap 1): Caddy's config was once bind-mounted as a single file; tools that unlink and recreate (install, mv, GNU sed -i) left the container reading an orphan inode while caddy reload exited 0 and served the old config for every vhost. The directory is mounted on the host now. Reload, never restart.
• 2026-09-11 (measured at fd8558f, deploy/infomaniak/caddy-compose.staging.yml line 26; the file's last change is 1850c65 of 2026-07-27): the tracked Caddy compose still carries ./Caddyfile:/etc/caddy/Caddyfile:ro, the single-file shape trap 1 describes, and its header calls that the current state. The 2026-08-04 fix was applied to the host's compose by hand and that file is untracked, so the template describes the state before the trap was fixed. Copying it reproduces the orphan-inode trap. A new host mounts the config directory and the template is corrected when it is next touched.
• 2026-09-11 (wiki 2026-08-04, trap 3, and prima/deploy/ship.sh header): a macOS tar without COPYFILE_DISABLE=1 shipped 36,232 AppleDouble ._* files and Alembic imported one as Python (source code string cannot contain null bytes, naming no file). Every tar in this stack sets it and excludes ._*, the chat rsync excludes them in its list (prima-chat/deploy/ship.sh line 40), and git archive never emits them.
• 2026-09-11 (wiki 2026-08-04, trap 5, and Dockerfile lines 48 to 58): chown -R after the code COPY rewrote ownership across a 4.9 GB image on every one-line change (15 to 20 minute rebuilds, a 24 GB build cache). The user is created before the copies and COPY --chown sets ownership inline; a code-change rebuild is seconds.
• 2026-09-11 (Dockerfile lines 112 to 117, wiki 2026-07-27 trap 1): uvicorn must run with backend/ as the working directory (WORKDIR /app/backend, uvicorn app_fastapi:app); uvicorn backend.app_fastapi:app from /app fails with No module named 'api'.
• 2026-09-11 (ship-frontend-staging.sh lines 57 to 62 and 94 to 99): two different grep -q under pipefail bugs rolled good releases back: the artefact check needed || true (grep exits 1 on a clean bundle), and the served-output check moved to case matching because grep -q closes the pipe early and the producer's SIGPIPE fails the pipeline exactly when the match exists. Grepping only the entry chunk for the API host also rolled a good release back to a month-old one; the host may legitimately sit in index.html.
• 2026-09-11 (Caddyfile.staging, the SPA vhost): /opt/alchemy/labs-web/current is staging's document root and production serves from labs-web/prod/current; a staging build written into prod/ silently replaces production (wiki 2026-09-01). The two webroots differ by one path segment.
• 2026-09-11 (Caddyfile.staging, the API vhost): CORS and the security headers are deliberately absent from Caddy because the application emits them; a duplicate Access-Control-Allow-Origin fails closed in the browser and looks like an outage. encode lists content types explicitly because the default matcher includes text/*, which would buffer text/event-stream. Both are in the file's comments with the code lines they cite.
• 2026-09-11 (wiki 2026-08-02 and distribute-env.sh line 433 onward): a provider token missing from the heredoc left every AI call in Labs failing with ProviderNotConfiguredError and no alert; the fix was applied on the host and in the template in the same session. A single-purpose host may set a provider-wide model override; a host that routes per request must not, or every alias collapses onto one model.
• 2026-09-11 (aurum/deploy/systemd/aurum-collect.service at a17fce3, the unit's own comment at lines 29 to 35): a timer whose ExecStart had no bearer 401'd silently for a full day (2026-07-28) and nothing was collected; the bearer moved to the 0600 curlrc. A timer unit is tested by systemctl start <unit> once and reading the journal, not by waiting for its calendar.
• 2026-09-11 (argus/deploy/push.sh lines 109 to 120): an image built on Apple Silicon without --platform linux/amd64 builds and ships cleanly and crash-loops on the host with exec format error, which reads as an application bug. The Labs and Prima paths avoid this by building on the host; the Argus path builds locally and asserts the architecture.

Reference implementation

Branches that ship: alchemycloud main at b3d34da (the working tree carries uncommitted Terraform edits and the untracked rotation script; the measured files are the committed ones); alchemy_labs feat/service-desk, read at fd8558f on the d1a-token-source worktree (backend and scripts identical to the line); prima feature/per-model-pricing at cd2c248 (the reference gateway per decision D3; the in-Labs primaapi copy is legacy and is not a deploy target of its own); prima-chat integration/redteam-fixes at e606308; prima-developer main at 4428767 (astro.config.mjs tracked there; deploy/ship.sh untracked, on one laptop, dated 2026-08-04); prima.li w8b-content-source at 03f4205 (unmerged; the deploy-notes commits recorded in the wave 0 report were reverted and the script header carries the record); emdash-theme-agency at 3dd00a02; argus at e42e467 (no remote); aurum week2-3-confirm-gate at a17fce3. Paths are relative to each repo's root.

• alchemycloud/scripts/distribute-env.sh (808 lines): usage and targets (1 to 31), the generators (33 to 46), require (49 to 67), ensure (69 to 104), warn_pasted (107 to 113), the per-app ensure/require blocks (117 to 194), render with one heredoc per app (197 to 800), --dry-run masking (802 to 805), the SSH write to mode 600 (807 to 808).
• alchemycloud/env/.env.example: the legend (lines 1 to 16), the shared block, one section per app.
• alchemycloud/cloud-init/base.yaml: packages, sshd hardening, Docker CE install, /opt/alchemy.
• alchemycloud/scripts/sync-prod-to-staging.sh: header (1 to 27), the pg_dump from a newer client image (78), the migration in the app image (126).
• alchemy_labs/Dockerfile: builder and runtime stages, useradd before COPY --chown (48 to 58), migrate.sh rationale (60 to 70), env tuning (76 to 88), HEALTHCHECK (104), WORKDIR and CMD (116 to 117).
• alchemy_labs/scripts/migrate.sh: the advisory lock around alembic upgrade head and the non-zero exit.
• alchemy_labs/scripts/ship-infomaniak.sh (staging only), scripts/ship-frontend-staging.sh (build, verify, ship, verify served, rollback), scripts/build-frontend-staging.sh (S2), scripts/preflight.sh, scripts/predeploy_window_check.sh, scripts/postdeploy_smoke.sh.
• alchemy_labs/deploy/infomaniak/Caddyfile.staging (two vhosts, every deliberate absence commented), caddy-compose.staging.yml (host network, the parent-directory mount, the volume names that hold the certificates; deviation: single-file Caddyfile bind mount at line 26, the host mounts the directory), docker-compose.infomaniak.yml (the template with the migrate service; deviation: not what runs).
• alchemy_labs/backend/services/cloudflare_service.py: D1 (58 to 89), R2 (91 to 110), custom domains (157 to 182), KV (184 onward, unrouted); backend/api/routers/emdash_router.py lines 47 to 194 for the provisioning routes.
• prima/deploy/ship.sh (tar 37 to 41, build and the tag listing 60 to 66, compose install 68 to 73, env gate 75 to 79, liveness 84 to 89, the rollback shape with its <prev> placeholder 93), prima/deploy/estate/compose.yml (the header on what it must never touch, POSTGRES_PASSWORD with :?, loopback port, the JWKS volume), prima/deploy/Caddyfile.snippet, prima/deploy/README.md (replicas, PRIMA_NODE_PEERS, 410 job_owner_retired).
• prima-chat/deploy/ship.sh: the rsync variant with two trees (the UI package resolves through the parent directory); exclude list (37 to 47), the two syncs (62 to 67), rollback-<TAG> retag of the running image (83 to 96), the .env gate that also refuses a shared gateway key (105 to 115). Deviation: no clean-tree refusal.
• prima-developer/astro.config.mjs lines 42 to 56: the [prima-api-base-guard] that throws when PRIMA_REQUIRE_API_BASE=1 and the base is empty. prima-developer/deploy/ship.sh (untracked, on-disk copy): the production default for the base (24), the build line (34), the empty-dist/ refusal (39), the content refusal (43 to 47), backup, rsync and chmod -R a+rX (51 to 68), the printed rollback (72). Deviation: not in any commit.
• prima.li/scripts/deploy-index.sh: build, check, print.
• emdash-theme-agency/wrangler.jsonc and astro.config.mjs lines 1 to 20: the bindings. Deviation: .wrangler/ tracked.
• argus/deploy/push.sh lines 95 to 150: platform, architecture assertion, docker save | docker load, root-owned files.
• aurum/deploy/systemd/aurum-collect.service and .timer: the curlrc bearer, SuccessExitStatus, Persistent.

Deviations reference versus recipe, all dated above: the host compose files are not tracked, and the tracked Caddy template still bind-mounts a single Caddyfile (line 26) where the host mounts the directory; the chat ship has no clean-tree refusal, so rsync ships whatever is on disk and what runs there is reconstructible from src-<tag>/ on the host, not from a commit; .deployed_commit has no writer; the staging ship does not migrate; Prima's ship does not migrate and its boot does not check the head; the domain flip through Labs cannot override; one site repo tracks .wrangler/; the rotation script is untracked; the portal ship script is untracked, so the static portal's publish path is not reconstructible from a commit; Prima's ship prints its rollback with a <prev> placeholder where the invariant asks for the command (the chat ship meets it with rollback-<TAG>); the labs-staging heredoc renders neither WEBAUTHN_RP_ID nor WEBAUTHN_ORIGIN (only labs-prod does, lines 254 to 255), so a staging host runs on the localhost RP default and a never-migrated staging database refuses at webauthn_rp_id_001; no heredoc renders WEBAUTHN_RP_ID_BACKFILL, which is optional wherever WEBAUTHN_RP_ID is rendered; argus/deploy/push.sh (e42e467, lines 44, 46 and 146) is a second writer of an /opt/alchemy/<app>/.env, because Argus is not one of distribute-env.sh's nine targets.

Acceptance checks

1. cd alchemycloud && git ls-files | grep -c tfstate prints 0; grep -cE "^[A-Z_]+=" env/.env.example prints 118 and grep -E "^[A-Z_]+=" env/.env.example | grep -c '\[AUTO\]' prints 18 (a bare grep -c '\[AUTO\]' prints 19, the legend included); git status --short scripts/ shows the rotation script as untracked until it is committed.
2. scripts/distribute-env.sh prima --dry-run prints the prima block with every value cut to six characters followed by an ellipsis and exits 0 without an SSH connection; with env/.env absent it exits 1 with missing ... copy env/.env.example.
3. Add a key to env/.env only and re-run the dry run: the key does not appear. Add it to the app's heredoc: it appears.
4. In alchemy_labs, git status --porcelain -- backend scripts Dockerfile non-empty makes scripts/ship-infomaniak.sh staging exit 1 before any network call with "uncommitted changes"; clean, --build-only builds labs:staging on the host and touches no container.
5. (Runnable locally, without the estate: the only check besides 11 that is.) docker run --rm --network host --env-file /opt/alchemy/<app>/.env -w /app/backend <image> sh /app/scripts/migrate.sh | tail -1 prints "database is at Alembic head" (exit 0) on a migrated database, and on a database behind head the command applies the missing revisions and prints the same line; on a failing revision it exits non-zero and the container never starts. The local form, from a laptop image against a scratch database: docker run --rm --network host -e DATABASE_URL=postgresql://<user>:<password>@127.0.0.1:<port>/<scratch db> -e ENVIRONMENT=development -w /app/backend <image> sh /app/scripts/migrate.sh | tail -1 prints the same line (on Apple Silicon the image is built with --platform linux/amd64 and runs under emulation; run 2026-09-12 by the rebuild against its own scratch database).
6. A frontend bundle built with npm run build instead of the staging mode makes scripts/ship-frontend-staging.sh exit 1 with "REFUSING TO SHIP: bundle points at localhost:8000" and uploads nothing; a correct build prints "served output references " and ends with a done line naming the public host and the release id (lines 107 and 111), and readlink /opt/alchemy/labs-web/current names that release; --rollback moves it to the previous one and the served index.html changes on the next request without a Caddy reload.
7. cd prima-developer && git status --short deploy/ prints ?? deploy/ until the script is committed (this check is then re-recorded). On a tracked-only tree (git archive 4428767 | tar -x -C <dir>, node_modules linked in; no .env, because the guard also reads one through loadEnv, astro.config.mjs lines 32 to 36), env -u PUBLIC_PRIMA_API_BASE PRIMA_REQUIRE_API_BASE=1 npm run build exits 1 and prints [prima-api-base-guard] PUBLIC_PRIMA_API_BASE is not set (measured 2026-09-11); the ship script itself cannot produce that failure. On the laptop that holds it, deploy/ship.sh --dry-run builds against the production API host, prints the rsync itemised change list and creates a <dir>.bak-<ts> on the host without changing the served directory.
8. prima.li/scripts/deploy-index.sh ends with the printed npx wrangler pages deploy dist --project-name <project> --branch main and exits 0; dist/pricing/index.html does not exist; run without PRIMA_SITE=index the build is the marketing site and the script exits 1 at that check.
9. In a site repo, cat wrangler.jsonc shows compatibility_flags: ["nodejs_compat"], one d1_databases entry with binding: "DB" and a real database_id, one r2_buckets entry with binding: "MEDIA"; git ls-files | grep -c '^\.wrangler/' prints 0; npx wrangler deployments list after npm run deploy shows the new version at the top.
10. npx wrangler d1 execute <db> --remote --command "SELECT value FROM options WHERE name='emdash:site_url'" returns the custom domain as a quoted JSON string, not a workers.dev host.
11. (Runnable locally.) docker image inspect <image> --format '{{.Architecture}}' prints amd64 for any image built on the laptop for the estate (docker build --platform linux/amd64, on Apple Silicon included). Every other check in this list needs the estate, a Cloudflare account or the alchemycloud checkout.
12. systemctl list-timers on a host lists the app's timer with a next-run time; systemctl start <unit>.service once and journalctl -u <unit> show a 2xx from the loopback call; grep -cE 'Bearer [A-Za-z0-9._~+/=-]{16,}' /etc/systemd/system/<unit>.service prints 0 (the word Bearer may appear in a comment, as it does in the reference unit at line 30; a value-shaped token after it may not) and grep -c curlrc on the same file prints at least 1.
13. curl -s -o /dev/null -w '%{http_code}' https://<unknown host on the estate>/ prints 404, not 200 with a placeholder body.