4 Commits

Include renames

Author	SHA1	Message	Date
Daan MeijerandClaude Opus 5.5	3d0dc6ca1f	The par2 merge: releases whose files one par2 set describes become one ... The prober stores, per probed article that starts a file, the MD5 of its first 16 KiB (file_hashes), and per par2 body the set's whole file list (par2_files, par2_sets): FileDesc holds the same hash per file, whatever the file was renamed to. releases:merge (the merger service, no NNTP) merges every minute the releases of one group, posted within 48 hours of a release holding the set's par2, whose files the set describes, into the one with the most files: chunks, probe candidates, inspections, hashes and counts move; the par2's shared name when none was readable; release_merges keeps what went where. Obfuscated uploads whose files were each a release of their own ("[9/20] - "14f5999a…"", hep: 13,232) become one. Also: corpus_sweep.py counts the marked ranges without psql's command tag. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-12 09:37:35 +02:00
Daan MeijerandClaude Opus 5.5	4ed7f89951	Prober: no packaging or release-name-plus-extension as inner name ... Lone candidates are often par2 volumes of random-named sets ("42507096….vol007+08.par2"): their yEnc name was taken as the inner name. Packaging names are skipped for lone files too, and a name that starts with the release's own name says nothing new. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-11 15:44:08 +02:00
Daan MeijerandClaude Opus 5.5	738402aefa	Prober: an inspection per candidate; no inner name that is the release's name ... A release can have several candidates of a kind (two RAR sets): keyed by (release, kind) both landed in one upsert ("ON CONFLICT DO UPDATE command cannot affect row a second time"), and the batch was lost. release_inspections is keyed by candidate now; the first run's results are cleared and probed again. A random-named file's yEnc name is often its random name: an inner name equal to the release's name is dropped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-11 15:37:32 +02:00
Daan MeijerandClaude Opus 5.5	8bd6d6aa5f	Rebuild: phase 2's prober reads the probe queue ... probe:run (compose service prober, PROBE_CONNECTIONS, default 6) claims waiting candidates of settled releases (lone, then par2, then archive), fetches each article with BODY over connections of its own (no compression) and inspects it: the yEnc name and size, the content type from the first bytes, a par2 set's files, a RAR volume's entries and whether they're encrypted. Results per release and kind in release_inspections (CHUNK_TABLESPACE, the HDD); on the release passworded, inner_name (the largest content name) and inspected_at. 430 = missing, not yEnc = error, both final; connection trouble puts the candidate back. Fetch and parse in one process: an article takes ~0.1 s per connection, parsing a few ms (no queue in between). Parsers from voyager 1 (lib/yenc, lib/rar, lib/par2, lib/content sniff) with their tests; XoverClient.body; yencode. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-11 15:32:49 +02:00