Rebuild: phase 2's prober reads the probe queue

probe:run (compose service prober, PROBE_CONNECTIONS, default 6) claims
waiting candidates of settled releases (lone, then par2, then archive),
fetches each article with BODY over connections of its own (no
compression) and inspects it: the yEnc name and size, the content type
from the first bytes, a par2 set's files, a RAR volume's entries and
whether they're encrypted. Results per release and kind in
release_inspections (CHUNK_TABLESPACE, the HDD); on the release
passworded, inner_name (the largest content name) and inspected_at.
430 = missing, not yEnc = error, both final; connection trouble puts
the candidate back.

Fetch and parse in one process: an article takes ~0.1 s per connection,
parsing a few ms (no queue in between). Parsers from voyager 1 (lib/yenc,
lib/rar, lib/par2, lib/content sniff) with their tests; XoverClient.body;
yencode.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

deploy/README.md

@@ -17,6 +17,7 @@ homelab with the deploy key, whose forced command
 | `migrate` | `usenet-rebuild` (target `app`) | `migration:run`, before every deploy |
 | `web` | `usenet-rebuild` | node API, catalog API, Newznab, web interface; port 3334 |
 | `coordinator` | `usenet-rebuild` | prefetching (all NNTP connections), planning, the tail, merging, settling |
+| `prober` | `usenet-rebuild` | phase 2: reads the probe queue's articles over `PROBE_CONNECTIONS` (6) connections, stores what they show |
 | `edge` | `usenet-rebuild-edge` (target `edge`) | the Rust worker, `--threads $EDGE_THREADS` (8) |
 | `edge-ts` | `usenet-rebuild` | the TypeScript worker, profile `edge-ts`, not started |
 | `ace` | `usenet-rebuild` | one-off ace commands, profile `tools` |
@@ -62,6 +63,7 @@ docker compose up -d --no-deps edge coordinator
 - Stop `edge` before `web`: a worker holding a stream keeps the server from
   exiting.
 - The NNTP account is shared with voyager 1 and downloads: `NNTP_CONNECTIONS`
-  (40) plus one plain connection, out of 50.
+  (40) plus one plain connection, plus the prober's `PROBE_CONNECTIONS` (6),
+  out of 50.
 - `deploy.sh` doesn't update itself: after changing it, install it again
   (`install -m 755 src/deploy/deploy.sh ~/bin/deploy-usenet-rebuild`).

deploy/compose.yaml

@@ -82,6 +82,16 @@ services:
       <<: *app-env
       NNTP_CONNECTIONS: ${NNTP_CONNECTIONS:-40}
 
+  # Phase 2's prober: reads the probe queue's articles over PROBE_CONNECTIONS
+  # connections of its own (next to the coordinator's 40 + 1; the account has 50).
+  prober:
+    <<: *app
+    command: ['node', 'ace', 'probe:run']
+    volumes: []
+    environment:
+      <<: *app-env
+      PROBE_CONNECTIONS: ${PROBE_CONNECTIONS:-6}
+
   # The Rust worker; it has no NNTP account and talks to web only. Stopped
   # before web (a worker holding a stream keeps the server from exiting).
   edge:

deploy/deploy.sh

@@ -45,10 +45,10 @@ docker compose run --rm --no-deps migrate
 # stream keeps the web server from exiting.
 echo "== recreating app services"
 docker compose stop edge
-docker compose up -d --no-deps web coordinator
+docker compose up -d --no-deps web coordinator prober
 docker compose up -d --no-deps edge
 echo "$SHA $(date -Is)" >> "$ROOT/deployed.log"
 # The images this replaced (321 MB each; the root filesystem is the NVMe, with a 10 GB floor).
 docker image prune -f --filter label=com.docker.compose.project=usenet-rebuild >/dev/null
-docker compose ps --format '{{.Service}} {{.Status}}' web coordinator edge
+docker compose ps --format '{{.Service}} {{.Status}}' web coordinator prober edge
 echo "== deployed $SHA"