Skip to content

Deployment procedure

Three components, three unrelated deploy models. This page is the cross-repo view; each repo's own docs are authoritative for exact commands.

No single deploy

There is no orchestrator that ships the whole platform at once. Components deploy independently, so cross-component changes need an ordering decision — see Coordinated changes below.

Frontend

Apache + PHP-FPM. PHP is deployed from the repo; React assets are built in CI and fetched as artifacts rather than committed.

./scripts/deploy.sh <branch>   # default: main

The script:

  1. Looks for a React Build Check artifact named react-app-<tree-sha>, where <tree-sha> is git rev-parse HEAD:react-graph-standard.
  2. Dispatches the workflow and waits if no matching artifact exists.
  3. Atomically swaps files into public/assets/react-app/, keeping one assets.old.<ts> directory for rollback.

Keying the artifact on the tree SHA of the React subdirectory means a commit that doesn't touch the SPA reuses the existing build instead of rebuilding.

Requires a fine-grained GitHub PAT at ~/.config/verilib/gh-token to download artifacts. See React rewrite and Tech stack.

Rollback: restore the retained assets.old.<ts> directory, or re-run deploy.sh against the previous branch/tag.

Atomizer (ECS on EC2)

Two steps, both taking an environment name that selects deploy/config/<env>.env:

AWS_PROFILE=<profile> deploy/scripts/build_and_push.sh <env> all      # worker + probe images
AWS_PROFILE=<profile> deploy/scripts/build_and_push.sh <env> worker   # worker only
AWS_PROFILE=<profile> deploy/scripts/deploy.sh <env> all

Images go to private ECR; deploy.sh registers task definitions and updates the ECS services.

ECS on EC2, not Fargate — each atomize job runs probe containers via the host Docker socket (Docker-out-of-Docker), which Fargate does not permit. This constrains task sizing: CPU/memory are hardcoded in deploy/templates/*.taskdef.json because the atomize task must reserve enough RAM to cover the probe container it spawns.

Per-environment secrets and account values live in gitignored deploy/config/<env>.env (start from example.env). RabbitMQ credentials come from Secrets Manager via RABBITMQ_SECRET_ARN; S3 uses the task IAM role, not keys.

One-time AWS setup — ECR repos, Secrets Manager entry, IAM roles, the EC2-backed ECS cluster, the S3 bucket, security groups — is documented in upstream deploy/README.md. Do not copy account ids or secret values into this hub.

See ECR on ECS.

Upgrading a probe

Probes deploy separately from the worker:

  1. Build and push new probe image tags for the environment.
  2. Point deploy/config/<env>.env at the new PROBE_*_VERSION / image refs.
  3. Redeploy the atomize-processor service so tasks pick them up.

Validate the new probe's output against the JSON Schema before rolling — see Probe ecosystem and Probe upgrading.

Cert workers

Two independent processes:

uv sync
cp .env.example .env    # RABBITMQ_*, S3_*; no DB_* vars
uv run python scripts/run_validate_processor.py
uv run python scripts/run_promote_processor.py   # separate process

Exactly one promote_processor with mainnet enabled

promote_processor scans Certified events for a duplicate before submitting. Under at-least-once delivery, two instances can both pass the scan and both submit — a check-then-act race that spends real gas twice. Run a single instance when MAINNET_CERTIFY_ENABLED=1.

validate_processor has no such constraint and can be scaled horizontally.

Coordinated changes

Some changes are not safe to deploy one side at a time:

Change Constraint
Queue topology arguments (x-max-priority, x-message-ttl, x-dead-letter-exchange) Must land in both repos in the same release. Whichever peer declares second gets a 406 PRECONDITION_FAILED and dies.
Renaming a queue or exchange Both peers, same release.
Removing or renaming a message field Three-step rollout: publish the new field, consume it, then drop the old one. Additive fields are safe alone.
GITHUB_TOKEN_ENCRYPTION_KEY / JWT_KEY rotation Frontend and upload-processor must change together, or private clones fail.
Bucket renames S3_BUCKET / S3_CERT_BUCKET / S3_REPOS_BUCKET are the same buckets under different variable names in different repos.

Full list: Message contracts and Cross-repo contracts.

Database migrations

Numbered SQL files applied by hand — see Scheme → Migrations.

Production MySQL is Amazon RDS (Managed), so two things shape how a migration is applied:

  • Run the file against the RDS endpoint from a host whose security group is allowed inbound on 3306 — the instance is not publicly reachable.
  • Take a manual snapshot first. It is the cheapest rollback available and it outlives the automated retention window — see Database backup and restore.

Changing DB_* on a host is itself a deploy step: PHP reads the env file in immutable mode, so reload PHP-FPM and Apache and restart the DB-writing workers afterwards.

Unknowns

Who applies migrations to staging and production, and where in the deploy sequence, is not recorded in any repo. This is the largest undocumented gap in the deploy story — confirm with the team before applying anything to a shared environment.