Scalability¶
The processing tier scales on queue depth per consumer, not CPU. A verification job is long and I/O-shaped, so CPU utilization is a poor signal — a worker deep in a Verus run looks idle to an ECS CPU metric while a backlog builds behind it.
The autoscaling loop¶
flowchart LR
mq[RabbitMQ<br/>management API] --> lam[queue_depth_publisher<br/>Lambda]
lam --> cw[CloudWatch<br/>Verilib/RabbitMQ]
cw --> pol[Application Auto Scaling<br/>target tracking]
pol --> ecs[ECS service<br/>task count]
ecs --> mq- A Lambda (
deploy/lambda/queue_depth_publisher) polls the RabbitMQ management API on an EventBridge schedule (default every 1 minute, port 15672). - It publishes messages ready per consumer per queue to CloudWatch under the
Verilib/RabbitMQnamespace. - ECS Application Auto Scaling target-tracking policies drive the atomize and upload service task counts against that metric.
Target-tracking policies are generated from deploy/templates/autoscaling/atomize-target-tracking.json and upload-target-tracking.json.
Tuning¶
| Variable | Default | Meaning |
|---|---|---|
ATOMIZE_MIN_TASKS / ATOMIZE_MAX_TASKS |
1 / 10 |
Bounds for the atomize service |
UPLOAD_MIN_TASKS / UPLOAD_MAX_TASKS |
1 / 5 |
Bounds for the upload service |
ATOMIZE_TARGET_MSG_PER_CONSUMER |
0.5 |
Target backlog per worker |
UPLOAD_TARGET_MSG_PER_CONSUMER |
1 |
Uploads are short; a shallow queue is fine |
LAMBDA_SCHEDULE_RATE |
1 minute |
Metric resolution — and therefore the floor on reaction time |
The target must be below 1
Target tracking keeps the metric at the target, not below it. Setting ATOMIZE_TARGET_MSG_PER_CONSUMER=1 tells the autoscaler that one queued message per worker is the desired steady state — so it will not scale out until the backlog exceeds one message per worker.
Use a value < 1 for "scale out as soon as a second message waits". 0.5 is the tracked default. The value must be greater than 0; target tracking rejects 0.
Uploads and atomize scale independently because their shapes differ: an upload is a shallow clone plus an S3 mirror (seconds), while an atomize job is a probe run that can take many minutes.
What limits throughput¶
| Limit | Why it exists | Where to change it |
|---|---|---|
RABBITMQ_PREFETCH=1 |
One in-flight message per consumer, so a slow job can't starve siblings on the same connection. Concurrency comes from more tasks, not deeper prefetch. | broker/config.py default |
| ECS task memory | Atomize tasks must reserve enough RAM to cover the probe container they spawn via the host Docker socket. | Hardcoded in deploy/templates/*.taskdef.json |
| EC2 instance disk | Concurrent probe runs each need scratch space; upstream guidance is a large EBS root volume (≥ 100 GB) plus WORKDIR_ROOT. |
Instance user-data |
| ECS on EC2, not Fargate | Docker-out-of-Docker needs /var/run/docker.sock, which Fargate does not expose. Scaling means more EC2 capacity, not just more tasks. |
Cluster capacity |
CLI_TIMEOUT vs broker consumer_timeout |
A probe that outruns the broker timeout loses its channel mid-job. | See Performance |
Scaling out ECS tasks does nothing if the underlying EC2 cluster has no room. Both need headroom.
Components that must not scale out¶
| Component | Constraint |
|---|---|
promote_processor |
Exactly one instance when MAINNET_CERTIFY_ENABLED=1. Two instances can both pass the duplicate scan and both submit a mainnet transaction — a check-then-act race under at-least-once delivery, costing real gas. |
validate_processor, upload-processor, and atomize-processor are all safe to run in parallel — each message is addressed to a single repo_id / certificate_id and the frontend applies results idempotently.
Data-plane scaling¶
| Tier | Notes |
|---|---|
| S3 | Repo trees at s3://<bucket>/<repo_id>/, cert artifacts at cert/<repo_id>/. Effectively unbounded; the prefix-per-repo layout avoids hot partitions. |
| RabbitMQ | Retry queues hold a message for 50 s before dead-lettering back. A sustained failure loop therefore grows the retry queue, not the main queue — watch both. |
| MySQL | Amazon RDS for MySQL 8, frontend-owned — see Managed. A single primary writer: instance class, gp3 IOPS/throughput, and Multi-AZ are all adjustable, but none of them add write throughput. The atomizer writes atoms directly, so a large repo produces a write burst proportional to its atom count. No sharding or read replicas are configured. |
The heavy paths (atomize persist, certify clone) issue many statements per job, so they are sensitive to per-statement round-trip time as well as to instance size — a networked instance punishes chatty SQL. Read CPU, IOPS, and write latency together before resizing: duration that is high while both CPU and IOPS are low is round-trip bound, and the fix is fewer statements per job, not a bigger instance. Keep RDS in the same Region as the PHP writers.
Unknowns
There is no documented load testing, no published throughput target, and no autoscaling for the frontend or MySQL tiers. The production instance class and storage settings are not recorded in any repo — read them from the RDS console. Capacity planning beyond the queue-depth loop is not recorded in any repo.