Daan MeijerandClaude Opus 5.5
652f278474
Every commit waited on the WAL in ./pg on the HDD (WALWrite; the HDD 100% busy, 56–70 ms a write): prober flushes and claims took most of each minute. A PG_WAL_DIR volume (default /var/lib/usenet-rebuild-wal) at /var/lib/postgresql/wal, with data/pg_wal a symlink to it (set up by hand once, with Postgres stopped). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
3.7 KiB
Deploying voyager 2
Voyager 2 (branch rebuild) runs on homelab as the compose project
usenet-rebuild, separate from voyager 1's (usenet-indexer, branch
master): its own Postgres, network, images and directory.
A push to rebuild that passes CI is deployed: the deploy job SSHes to
homelab with the deploy key, whose forced command
(~/bin/deploy-usenet-indexer) passes deploy-rebuild <sha> on to
~/bin/deploy-usenet-rebuild (this directory's deploy.sh).
What runs
| Service | Image | Does |
|---|---|---|
postgres |
postgres:17 | the catalog (tablespace nvme) and the WAL (PG_WAL_DIR, data/pg_wal a symlink to it) on the NVMe; the chunks (HDD) |
migrate |
usenet-rebuild (target app) |
migration:run, before every deploy |
web |
usenet-rebuild |
catalog API, Newznab, web interface; port 3334 (HTTP_ROUTES=public) |
node-api |
usenet-rebuild |
the edge workers' API (leases, streams, batches); port 3335 on the compose network only (HTTP_ROUTES=node) |
coordinator |
usenet-rebuild |
prefetching (all NNTP connections), planning, the tail, merging, settling |
prober |
usenet-rebuild |
phase 2: reads the probe queue's articles over PROBE_CONNECTIONS (6) connections, stores what they show |
merger |
usenet-rebuild |
the par2 merge: releases whose files one par2 set describes become one (no NNTP) |
edge |
usenet-rebuild-edge (target edge) |
the Rust worker, --threads $EDGE_THREADS (8) |
edge-ts |
usenet-rebuild |
the TypeScript worker, profile edge-ts, not started |
ace |
usenet-rebuild |
one-off ace commands, profile tools |
On homelab
/mnt/bulk/usenet-rebuild (/home is nearly full):
| Path | |
|---|---|
compose.yaml |
copied from src/deploy/compose.yaml on every deploy |
.env |
credentials and tuning, mode 600; not in git (below) |
src/ |
the code at the deployed commit (the build context) |
pg/ |
Postgres data (HDD) |
spool/ |
prefetched XOVER chunks |
corpus/ |
stored XOVER chunks of ranges marked capture, for the grouping sweeps (up to 300 GB) |
corpus-out/ |
the sweeps' replays, per edge image (python3 src/deploy/corpus_sweep.py) |
deployed.log |
one line per deploy: sha, time (local) |
.cache/repo.git |
bare clone the deploys fetch into |
app/ |
an older copy of the code with dev dependencies, for tools/*.ts (r.sh); not what runs |
The catalog's tablespace is /var/lib/usenet-rebuild-catalog on the NVMe
(PG_NVME_DIR). Keep 10 GB free on the root filesystem.
.env holds APP_KEY, DB_USER, DB_PASSWORD, DB_DATABASE, the NNTP_*
account, NODE_TOKEN (the edge worker's token, from nodes:create),
CHUNK_TABLESPACE=pg_default, TZ, LOG_LEVEL, SESSION_DRIVER and
optional tuning (NNTP_CONNECTIONS, EDGE_THREADS, PG_SHARED_BUFFERS,
anything in start/env.ts). Hosts, ports and paths are set in compose.yaml.
By hand
cd /mnt/bulk/usenet-rebuild
~/bin/deploy-usenet-rebuild <sha> # what CI runs
docker compose ps
docker compose logs -f --since 5m coordinator
docker compose run --rm ace releases:duplicates
docker compose stop edge coordinator # pause indexing (the worker first)
docker compose up -d --no-deps edge coordinator
- Recreate app services only, with
--no-deps; Postgres only on purpose, with the worker and coordinator stopped first. - Stop
edgebeforenode-api: a worker holding a stream keeps the server from exiting. - The NNTP account is shared with voyager 1 and downloads:
NNTP_CONNECTIONS(40) plus one plain connection, plus the prober'sPROBE_CONNECTIONS(6), out of 50. deploy.shdoesn't update itself: after changing it, install it again (install -m 755 src/deploy/deploy.sh ~/bin/deploy-usenet-rebuild).