Skip to content

GraphQL VQ* operations

Runbook for the VQD, VQS, and VQP hosts that run verilib-graphql. Each host is an independent Docker Compose stack; there is no shared deploy script with the PHP frontend or ECS atomizer.

Quick reference

Check Command
SSH ssh VQD · ssh VQS · ssh VQP
Stack status cd verilib-graphql && docker compose ps
Relay health curl -s http://localhost:8081/healthz
Hasura (with API key) curl -s http://localhost:8092/v1/graphql -H 'content-type: application/json' -H 'x-hasura-api-key: …' -d '{"query":"{ extracts(limit:1){ repo_id } }"}'
Relay logs docker compose logs -f ingest-relay
Postgres tables docker compose exec postgres psql -U verilib -d verilib_graphql -c '\dt public.*'

Default published ports: 8092 (Hasura), 8081 (relay), 5432 (Postgres).

Deploy / upgrade

Code lives in ~/verilib-graphql on each host (rsync or git pull). Secrets stay in .env on the server — never commit them.

ssh VQD   # or VQS / VQP
cd verilib-graphql
git pull origin main
docker compose up -d --build
curl -s http://localhost:8081/healthz

Hasura applies hasura/migrations and hasura/metadata on container start. Postgres data persists in the pg_data volume.

Rollback: check out the previous git ref and re-run docker compose up -d --build. Postgres schema is forward-only via Hasura migrations; down-migrations exist in-repo but are rarely needed in ops.

Environment pairing

Each VQ* host must agree with its paired frontend on three contracts:

Contract VQ* .env Frontend .env
RabbitMQ broker RABBITMQ_HOST, RABBITMQ_* Same broker the PHP app publishes to
Queue topology GRAPHQL_INGEST_EXCHANGE, _QUEUE, _ROUTING_KEY Same GRAPHQL_INGEST_* vars
Probe bucket S3_BUCKET Same bucket atomize writes probe JSON to

On EC2 the relay uses the instance IAM role for S3 — leave AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY blank on VQ* hosts. The role needs read on the bucket plus s3:PutObject on <bucket>/*/.verilib/probes/extracts/*; without the write grant the relay falls back to recording the source probe URI instead of a full timestamped snapshot.

The Next.js app (verilib-web) reads the store through its own /api/vq route. Its container needs two more variables, set in the paired host's next.env:

Variable Value
VQ_GRAPHQL_URL http://<VQ* host>:8092/v1/graphql (Hasura is plain HTTP)
VQ_GRAPHQL_API_KEY The VQ* host's GRAPHQL_API_KEY. Server-side only; never expose it with a NEXT_PUBLIC_ prefix
VQ_ACL_API_BASE_URL Optional. PHP API used for the per-repo access check. Defaults to NEXT_PUBLIC_API_BASE_URL

Shared broker credentials

Do not rotate RabbitMQ admin or other shared accounts without coordinating all consumers (upload/atomize/validate workers and every VQ* relay). Prefer a dedicated least-privilege broker user for new services. See platform ops policy in internal runbooks.

Backfill

When Hasura is empty but S3 already holds probe JSON (new stack, missed publishes, or prod lag), re-ingest per repo:

curl -s -X POST http://localhost:8081/backfill \
  -H 'content-type: application/json' \
  -d '{"repo_id": 1234}'

The relay lists existing probe outputs for that repo in S3 and enqueues ingest work. Idempotency on content_sha prevents duplicates.

Prod note (2026-08): only repos with probe JSON in S3 can be backfilled — repos atomized before GraphQL ingest was enabled may have MySQL atoms but no S3 envelope to snapshot.

Troubleshooting

Relay unhealthy (/healthz not {"ok":true})

  1. docker compose ps — all four services should be Up.
  2. Relay logs — Postgres connection errors, RabbitMQ ACCESS_REFUSED, or S3 AccessDenied.
  3. Verify .env broker host matches the paired environment (DEV/STG/PROD hostname).
  4. From the host: nc -zv <RABBITMQ_HOST> 5672 (security group must allow 5672 from the VQ* instance).

406 PRECONDITION_FAILED on publish

Symptom: atomize succeeds but GraphQL stays stale; PHP logs graphql ingest publish failed or push returned false.

Cause: the frontend's GraphqlIngestQueue and the relay declared the same queue name with different arguments (for example PHP still using BaseQueue::setup() topic retry/error exchanges while the relay declares a plain direct queue).

Fix: ensure frontend ships GraphqlIngestQueue::setup() override (main exchange/queue/bind only). Redeploy PHP; optionally delete and recreate the queue on the broker only when no consumers are attached (coordinate with ops).

Hasura returns empty for a repo that atomized

  1. Confirm a probe JSON exists in S3 for that repo (s3_output_uri from atomize response).
  2. Check relay consumed the message: docker compose logs ingest-relay | tail.
  3. Query Postgres: SELECT repo_id, captured_at, atom_count FROM extracts WHERE repo_id = … ORDER BY captured_at DESC LIMIT 5;
  4. Run /backfill for that repo_id if S3 has data but Postgres does not.

API key rejected

Hasura returns authorization errors when x-hasura-api-key is missing or wrong. The auth webhook compares against GRAPHQL_API_KEY in .env. Admin console access uses a separate HASURA_GRAPHQL_ADMIN_SECRET.

Provisioning notes (reference)

Initial VQ* rollout (2026-08):

  • VQD provisioned first (t3.medium, us-east-2, verilib-ecs account); VQS and VQP cloned from an AMI snapshot.
  • Per-env IAM instance profiles with read on the paired bucket and write on the extracts/ snapshot prefix.
  • Hasura port 8092 open to the world; access gated by API key webhook.

For hostnames and frontend pairings see GraphQL extract store — environment hosts.