The par2 merge: releases whose files one par2 set describes become one

The prober stores, per probed article that starts a file, the MD5 of its
first 16 KiB (file_hashes), and per par2 body the set's whole file list
(par2_files, par2_sets): FileDesc holds the same hash per file, whatever
the file was renamed to. releases:merge (the merger service, no NNTP)
merges every minute the releases of one group, posted within 48 hours of
a release holding the set's par2, whose files the set describes, into the
one with the most files: chunks, probe candidates, inspections, hashes and
counts move; the par2's shared name when none was readable; release_merges
keeps what went where. Obfuscated uploads whose files were each a release
of their own ("[9/20] - "14f5999a…"", hep: 13,232) become one.

Also: corpus_sweep.py counts the marked ranges without psql's command tag.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

deploy/README.md

@@ -19,6 +19,7 @@ homelab with the deploy key, whose forced command
 | `node-api` | `usenet-rebuild` | the edge workers' API (leases, streams, batches); port 3335 on the compose network only (`HTTP_ROUTES=node`) |
 | `coordinator` | `usenet-rebuild` | prefetching (all NNTP connections), planning, the tail, merging, settling |
 | `prober` | `usenet-rebuild` | phase 2: reads the probe queue's articles over `PROBE_CONNECTIONS` (6) connections, stores what they show |
+| `merger` | `usenet-rebuild` | the par2 merge: releases whose files one par2 set describes become one (no NNTP) |
 | `edge` | `usenet-rebuild-edge` (target `edge`) | the Rust worker, `--threads $EDGE_THREADS` (8) |
 | `edge-ts` | `usenet-rebuild` | the TypeScript worker, profile `edge-ts`, not started |
 | `ace` | `usenet-rebuild` | one-off ace commands, profile `tools` |

deploy/compose.yaml

@@ -108,6 +108,12 @@ services:
       <<: *app-env
       PROBE_CONNECTIONS: ${PROBE_CONNECTIONS:-6}
 
+  # The par2 merge: releases whose files one par2 set describes (what the prober read) become one. No NNTP.
+  merger:
+    <<: *app
+    command: ['node', 'ace', 'releases:merge']
+    volumes: []
+
   # The Rust worker; it has no NNTP account and talks to node-api only.
   # Stopped before node-api (a worker holding a stream keeps the server from
   # exiting).

deploy/corpus_sweep.py

@@ -42,7 +42,8 @@ def psql(sql):
         ['docker', 'compose', 'exec', '-T', 'postgres', 'psql', '-U', env('DB_USER'), '-d', env('DB_DATABASE'), '-At', '-F', '\t', '-v', 'ON_ERROR_STOP=1'],
         input=sql, capture_output=True, text=True, check=True,
     ).stdout
-    return [line.split('\t') for line in out.splitlines() if line]
+    # Without the command tag a RETURNING prints after its rows ("UPDATE 100").
+    return [line.split('\t') for line in out.splitlines() if line and not line.startswith(('UPDATE ', 'INSERT ', 'DELETE '))]
 
 
 def corpus_bytes():

deploy/deploy.sh

@@ -45,10 +45,10 @@ docker compose run --rm --no-deps migrate
 # stream keeps the node API from exiting.
 echo "== recreating app services"
 docker compose stop edge
-docker compose up -d --no-deps node-api web coordinator prober
+docker compose up -d --no-deps node-api web coordinator prober merger
 docker compose up -d --no-deps edge
 echo "$SHA $(date -Is)" >> "$ROOT/deployed.log"
 # The images this replaced (321 MB each; the root filesystem is the NVMe, with a 10 GB floor).
 docker image prune -f --filter label=com.docker.compose.project=usenet-rebuild >/dev/null
-docker compose ps --format '{{.Service}} {{.Status}}' node-api web coordinator prober edge
+docker compose ps --format '{{.Service}} {{.Status}}' node-api web coordinator prober merger edge
 echo "== deployed $SHA"