Error codes and failure states¶
VeriLib has no single numeric error-code namespace. Failures surface in four different places, and knowing which one you are looking at is most of the diagnosis:
| Surface | Shape | Where you see it |
|---|---|---|
| HTTP | Standard status codes from the PHP app | Browser, CLI, api.ts |
| Queue responses | status: "error" + error.message string |
Repo / certificate rows, livelog UI |
| Broker-level | AMQP errors and dead-letter queues | RabbitMQ management UI, ops dashboards |
| DB status columns | Integer ids into the statuses lookup table |
repos.status_id, certificates.status |
HTTP¶
| Code | Cause | Notes |
|---|---|---|
403 |
Users::userCanCertify(...) returned false on POST repobrowser/certify |
The user lacks the Certifier task permission. See Permissions. |
401 / redirect |
No valid session; USE_AUTH_SYSTEM enabled |
/v2 endpoints assume cookie session auth (withCredentials: true). |
There is no single structured error-body schema across /v2; endpoint success schemas are defined in public/swagger/swagger.json. See API spec.
Queue failure tiers¶
Every pipeline classifies failures into three tiers. This is the model to reason with — see Message contracts for the full mechanics.
| Tier | Examples | Destination | Retried? | User-visible? |
|---|---|---|---|---|
| Transient | S3 throttling, broker hiccup, unexpected exception | <queue>_retry_exchange, redelivered after 50 s |
Yes, up to the retry budget | No |
| Permanent | git clone failure, probe non-zero exit, retry budget exhausted, validation error on a parseable envelope |
Response queue with status="error" |
No | Yes — error.message is surfaced |
| Poison | Payload has no extractable repo_id / certificate_id |
<queue>_error_exchange |
No | No — ops-only |
Retry budgets: UPLOAD_MAX_RETRIES, ATOMIZE_MAX_RETRIES (code default 5; the tracked deploy example sets atomize to 3), JOB_MAX_ATTEMPTS (default 5, cert workers).
A transient failure is never visible on the response queue — only the final outcome is published. If a job appears stuck, check the retry queue depth before concluding nothing happened.
Common permanent errors¶
error.message shape |
Pipeline | Root cause |
|---|---|---|
git failed (exit=128): ... Repository not found. |
upload | Bad URL, missing branch/subfolder, or missing auth. |
| Authorization required | upload | github_repo_visibility=private with no github_access_token on the message. |
| Token decryption failure | upload | GITHUB_TOKEN_ENCRYPTION_KEY / JWT_KEY differs between the frontend and the upload processor. |
git rev-parse HEAD failure |
upload | Repository has no commits. Empty repos cannot be uploaded. |
probe-verus extract failed (exit 1) |
atomize | Probe run failed inside the container. Check the repo_probe_extract livelog event. |
Missing .verilib/probes/*.json |
atomize | Probe produced no output at the expected path. See JSON mapping. |
| Schema mismatch on parse | atomize | Probe emitted a non-Schema-3.0 envelope after an upgrade. See Probe upgrading. |
missing required field: 'certificate_id' |
validate / promote | Unaddressable payload — goes to the error queue, no response is published. |
Manifest not found at $PROBE_OUTPUT_DIR/cert<repo_id>/probe-manifest-complete.json |
validate | Probe image ran but did not write the manifest. |
| Dedup scan could not confirm | promote | MAINNET_CERTIFY_DEDUP_REQUIRED=1 (the default) and the eth_getLogs scan failed — usually *_DEDUP_FROM_BLOCK is unset or earliest. Job retries. |
Broker-level errors¶
| Symptom | Cause | Fix |
|---|---|---|
406 PRECONDITION_FAILED, channel closes at startup |
Two peers declared the same queue/exchange with different arguments (x-max-priority, x-message-ttl, x-dead-letter-exchange) |
Align BaseQueue.php and broker/topology.py — change both in the same PR. This failure is intentional. |
| Everything connects, nothing is delivered | Peers are on different vhosts | Set RABBITMQ_VHOST explicitly and identically on both sides. Defaults differ per repo (/ vs verilib). |
| Consumer killed mid-probe | CLI_TIMEOUT exceeds the broker's consumer_timeout |
Keep CLI_TIMEOUT ≤ consumer_timeout. |
| Error queue filling | Poison messages | Drain and inspect <queue>_error_queue; the envelope preserves original_message for replay. |
Database status columns¶
Repos and certificates carry integer status ids into the shared statuses lookup table.
Certificates: pending on insert (PHP), then ready or an error state once PHP consumes validate.response. PHP is the sole writer.
Repos: the legacy DB-poll path in the atomizer's main.py treats repos.status_id as 1 = pending, 2 = completed, 3 = error.
The statuses id mapping is not consistent across sources
The frontend's dev seed (docker/seed-dev.sql) populates statuses as 1 = pending, 2 = cancelled, 3 = verified — which does not match the atomizer's legacy 2 = completed / 3 = error interpretation.
Treat the id → label mapping as environment data, not a contract, and read the deployed statuses table before relying on a specific id. The queue-based path does not depend on these ids; only the legacy poll mode does. See Data flow.
Livelog error events¶
The atomize pipeline streams progress to livelog_queue; status is one of info, success, error, warning. A repo_error event marks a fatal failure. The last successful event tells you how far the job got:
repo_start → repo_probe_extract → repo_parse → repo_extract_bodies → repo_persist_deps → repo_persist_atoms → repo_complete
CLI¶
verilib-cli exits non-zero on failure; the --check-only flags exist specifically so CI fails without rewriting files.
| Command | Non-zero when |
|---|---|
atomize --check-only |
Stubs do not match the enriched stubs.json |
specify --check-only |
A stub with specs is missing a certificate |
verify --check-only |
Any stub has status failure |
Auth and keyring troubleshooting: CLI config and files.