Daan MeijerandClaude Opus 5.5
8bd6d6aa5f
probe:run (compose service prober, PROBE_CONNECTIONS, default 6) claims waiting candidates of settled releases (lone, then par2, then archive), fetches each article with BODY over connections of its own (no compression) and inspects it: the yEnc name and size, the content type from the first bytes, a par2 set's files, a RAR volume's entries and whether they're encrypted. Results per release and kind in release_inspections (CHUNK_TABLESPACE, the HDD); on the release passworded, inner_name (the largest content name) and inspected_at. 430 = missing, not yEnc = error, both final; connection trouble puts the candidate back. Fetch and parse in one process: an article takes ~0.1 s per connection, parsing a few ms (no queue in between). Parsers from voyager 1 (lib/yenc, lib/rar, lib/par2, lib/content sniff) with their tests; XoverClient.body; yencode. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
70 lines
3.2 KiB
Markdown
70 lines
3.2 KiB
Markdown
# Deploying voyager 2
|
|
|
|
Voyager 2 (branch `rebuild`) runs on homelab as the compose project
|
|
`usenet-rebuild`, separate from voyager 1's (`usenet-indexer`, branch
|
|
`master`): its own Postgres, network, images and directory.
|
|
|
|
A push to `rebuild` that passes CI is deployed: the deploy job SSHes to
|
|
homelab with the deploy key, whose forced command
|
|
(`~/bin/deploy-usenet-indexer`) passes `deploy-rebuild <sha>` on to
|
|
`~/bin/deploy-usenet-rebuild` (this directory's `deploy.sh`).
|
|
|
|
## What runs
|
|
|
|
| Service | Image | Does |
|
|
|---|---|---|
|
|
| `postgres` | postgres:17 | the catalog (tablespace `nvme`) and the chunks (HDD) |
|
|
| `migrate` | `usenet-rebuild` (target `app`) | `migration:run`, before every deploy |
|
|
| `web` | `usenet-rebuild` | node API, catalog API, Newznab, web interface; port 3334 |
|
|
| `coordinator` | `usenet-rebuild` | prefetching (all NNTP connections), planning, the tail, merging, settling |
|
|
| `prober` | `usenet-rebuild` | phase 2: reads the probe queue's articles over `PROBE_CONNECTIONS` (6) connections, stores what they show |
|
|
| `edge` | `usenet-rebuild-edge` (target `edge`) | the Rust worker, `--threads $EDGE_THREADS` (8) |
|
|
| `edge-ts` | `usenet-rebuild` | the TypeScript worker, profile `edge-ts`, not started |
|
|
| `ace` | `usenet-rebuild` | one-off ace commands, profile `tools` |
|
|
|
|
## On homelab
|
|
|
|
`/mnt/bulk/usenet-rebuild` (`/home` is nearly full):
|
|
|
|
| Path | |
|
|
|---|---|
|
|
| `compose.yaml` | copied from `src/deploy/compose.yaml` on every deploy |
|
|
| `.env` | credentials and tuning, mode 600; not in git (below) |
|
|
| `src/` | the code at the deployed commit (the build context) |
|
|
| `pg/` | Postgres data (HDD) |
|
|
| `spool/` | prefetched XOVER chunks |
|
|
| `deployed.log` | one line per deploy: sha, time (local) |
|
|
| `.cache/repo.git` | bare clone the deploys fetch into |
|
|
| `app/` | an older copy of the code with dev dependencies, for `tools/*.ts` (`r.sh`); not what runs |
|
|
|
|
The catalog's tablespace is `/var/lib/usenet-rebuild-catalog` on the NVMe
|
|
(`PG_NVME_DIR`). Keep 10 GB free on the root filesystem.
|
|
|
|
`.env` holds `APP_KEY`, `DB_USER`, `DB_PASSWORD`, `DB_DATABASE`, the `NNTP_*`
|
|
account, `NODE_TOKEN` (the edge worker's token, from `nodes:create`),
|
|
`CHUNK_TABLESPACE=pg_default`, `TZ`, `LOG_LEVEL`, `SESSION_DRIVER` and
|
|
optional tuning (`NNTP_CONNECTIONS`, `EDGE_THREADS`, `PG_SHARED_BUFFERS`,
|
|
anything in `start/env.ts`). Hosts, ports and paths are set in compose.yaml.
|
|
|
|
## By hand
|
|
|
|
```sh
|
|
cd /mnt/bulk/usenet-rebuild
|
|
~/bin/deploy-usenet-rebuild <sha> # what CI runs
|
|
docker compose ps
|
|
docker compose logs -f --since 5m coordinator
|
|
docker compose run --rm ace releases:duplicates
|
|
docker compose stop edge coordinator # pause indexing (the worker first)
|
|
docker compose up -d --no-deps edge coordinator
|
|
```
|
|
|
|
- Recreate app services only, with `--no-deps`; Postgres only on purpose,
|
|
with the worker and coordinator stopped first.
|
|
- Stop `edge` before `web`: a worker holding a stream keeps the server from
|
|
exiting.
|
|
- The NNTP account is shared with voyager 1 and downloads: `NNTP_CONNECTIONS`
|
|
(40) plus one plain connection, plus the prober's `PROBE_CONNECTIONS` (6),
|
|
out of 50.
|
|
- `deploy.sh` doesn't update itself: after changing it, install it again
|
|
(`install -m 755 src/deploy/deploy.sh ~/bin/deploy-usenet-rebuild`).
|