Daan MeijerandClaude Opus 5.5
652f278474
Every commit waited on the WAL in ./pg on the HDD (WALWrite; the HDD 100% busy, 56–70 ms a write): prober flushes and claims took most of each minute. A PG_WAL_DIR volume (default /var/lib/usenet-rebuild-wal) at /var/lib/postgresql/wal, with data/pg_wal a symlink to it (set up by hand once, with Postgres stopped). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
74 lines
3.7 KiB
Markdown
74 lines
3.7 KiB
Markdown
# Deploying voyager 2
|
|
|
|
Voyager 2 (branch `rebuild`) runs on homelab as the compose project
|
|
`usenet-rebuild`, separate from voyager 1's (`usenet-indexer`, branch
|
|
`master`): its own Postgres, network, images and directory.
|
|
|
|
A push to `rebuild` that passes CI is deployed: the deploy job SSHes to
|
|
homelab with the deploy key, whose forced command
|
|
(`~/bin/deploy-usenet-indexer`) passes `deploy-rebuild <sha>` on to
|
|
`~/bin/deploy-usenet-rebuild` (this directory's `deploy.sh`).
|
|
|
|
## What runs
|
|
|
|
| Service | Image | Does |
|
|
|---|---|---|
|
|
| `postgres` | postgres:17 | the catalog (tablespace `nvme`) and the WAL (`PG_WAL_DIR`, `data/pg_wal` a symlink to it) on the NVMe; the chunks (HDD) |
|
|
| `migrate` | `usenet-rebuild` (target `app`) | `migration:run`, before every deploy |
|
|
| `web` | `usenet-rebuild` | catalog API, Newznab, web interface; port 3334 (`HTTP_ROUTES=public`) |
|
|
| `node-api` | `usenet-rebuild` | the edge workers' API (leases, streams, batches); port 3335 on the compose network only (`HTTP_ROUTES=node`) |
|
|
| `coordinator` | `usenet-rebuild` | prefetching (all NNTP connections), planning, the tail, merging, settling |
|
|
| `prober` | `usenet-rebuild` | phase 2: reads the probe queue's articles over `PROBE_CONNECTIONS` (6) connections, stores what they show |
|
|
| `merger` | `usenet-rebuild` | the par2 merge: releases whose files one par2 set describes become one (no NNTP) |
|
|
| `edge` | `usenet-rebuild-edge` (target `edge`) | the Rust worker, `--threads $EDGE_THREADS` (8) |
|
|
| `edge-ts` | `usenet-rebuild` | the TypeScript worker, profile `edge-ts`, not started |
|
|
| `ace` | `usenet-rebuild` | one-off ace commands, profile `tools` |
|
|
|
|
## On homelab
|
|
|
|
`/mnt/bulk/usenet-rebuild` (`/home` is nearly full):
|
|
|
|
| Path | |
|
|
|---|---|
|
|
| `compose.yaml` | copied from `src/deploy/compose.yaml` on every deploy |
|
|
| `.env` | credentials and tuning, mode 600; not in git (below) |
|
|
| `src/` | the code at the deployed commit (the build context) |
|
|
| `pg/` | Postgres data (HDD) |
|
|
| `spool/` | prefetched XOVER chunks |
|
|
| `corpus/` | stored XOVER chunks of ranges marked `capture`, for the grouping sweeps (up to 300 GB) |
|
|
| `corpus-out/` | the sweeps' replays, per edge image (`python3 src/deploy/corpus_sweep.py`) |
|
|
| `deployed.log` | one line per deploy: sha, time (local) |
|
|
| `.cache/repo.git` | bare clone the deploys fetch into |
|
|
| `app/` | an older copy of the code with dev dependencies, for `tools/*.ts` (`r.sh`); not what runs |
|
|
|
|
The catalog's tablespace is `/var/lib/usenet-rebuild-catalog` on the NVMe
|
|
(`PG_NVME_DIR`). Keep 10 GB free on the root filesystem.
|
|
|
|
`.env` holds `APP_KEY`, `DB_USER`, `DB_PASSWORD`, `DB_DATABASE`, the `NNTP_*`
|
|
account, `NODE_TOKEN` (the edge worker's token, from `nodes:create`),
|
|
`CHUNK_TABLESPACE=pg_default`, `TZ`, `LOG_LEVEL`, `SESSION_DRIVER` and
|
|
optional tuning (`NNTP_CONNECTIONS`, `EDGE_THREADS`, `PG_SHARED_BUFFERS`,
|
|
anything in `start/env.ts`). Hosts, ports and paths are set in compose.yaml.
|
|
|
|
## By hand
|
|
|
|
```sh
|
|
cd /mnt/bulk/usenet-rebuild
|
|
~/bin/deploy-usenet-rebuild <sha> # what CI runs
|
|
docker compose ps
|
|
docker compose logs -f --since 5m coordinator
|
|
docker compose run --rm ace releases:duplicates
|
|
docker compose stop edge coordinator # pause indexing (the worker first)
|
|
docker compose up -d --no-deps edge coordinator
|
|
```
|
|
|
|
- Recreate app services only, with `--no-deps`; Postgres only on purpose,
|
|
with the worker and coordinator stopped first.
|
|
- Stop `edge` before `node-api`: a worker holding a stream keeps the server
|
|
from exiting.
|
|
- The NNTP account is shared with voyager 1 and downloads: `NNTP_CONNECTIONS`
|
|
(40) plus one plain connection, plus the prober's `PROBE_CONNECTIONS` (6),
|
|
out of 50.
|
|
- `deploy.sh` doesn't update itself: after changing it, install it again
|
|
(`install -m 755 src/deploy/deploy.sh ~/bin/deploy-usenet-rebuild`).
|