5 Commits

Include renames

Author	SHA1	Message	Date
Daan MeijerandClaude Opus 5.5	2823d42bfa	Merge obfuscated uploads by the random stem of their inner names ... hep, hephaestus and gyges post each file under a subject name of its own (31,000 one-file releases), but the yEnc name in each file's first article keeps the upload's random stem ("bX147zx8I0oEgj7nzFCiorCT0ps2jEcb.7z.001"), one stem per counter total. The merger's second pass joins releases whose lone inspections share a random stem: one group and poster, the same total, within 48 h, no more files than the total plus 10. An expression index over the stem; release_merges records the kind (via) and takes a stem. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-12 23:38:31 +02:00
Daan MeijerandClaude Opus 5.5	942c4901fa	Par2 merge: a set held by many releases, or a merge far larger than its set, isn't one upload ... Every kantodoNNN.bin test upload carries the same par2 set: 64 releases held it nearby and became one release of 631 files for a set of 10. A set held by more than 4 releases nearby isn't merged, nor members with more than 3x the set's files plus 10. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-12 22:29:48 +02:00
Daan MeijerandClaude Opus 5.5	c72bf68fd2	Par2 merge: a set held by two merged releases; one failed set doesn't stop a round ... A par2 index and its repost both hold a set: moving both rows to the target broke par2_sets' primary key, and the round failed on it every minute from 20:28 UTC (no merges at all). Duplicates among the others go first; a set that fails is logged and the round goes on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-12 22:28:50 +02:00
Daan MeijerandClaude Opus 5.5	6d1f2cc2a1	Par2 merge: a set file matching several releases nearby is ambiguous ... 131 uploads of 1 GiB test files ("kantodoNNN.bin", a.b.test) share their first 16 KiB and size, so all matched one set's single file and became one release of 1,284 files. A par2 file that matches more than one release in the anchor's group and time window is left out; a plain repost of the same file stays apart too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-12 10:15:32 +02:00
Daan MeijerandClaude Opus 5.5	3d0dc6ca1f	The par2 merge: releases whose files one par2 set describes become one ... The prober stores, per probed article that starts a file, the MD5 of its first 16 KiB (file_hashes), and per par2 body the set's whole file list (par2_files, par2_sets): FileDesc holds the same hash per file, whatever the file was renamed to. releases:merge (the merger service, no NNTP) merges every minute the releases of one group, posted within 48 hours of a release holding the set's par2, whose files the set describes, into the one with the most files: chunks, probe candidates, inspections, hashes and counts move; the par2's shared name when none was readable; release_merges keeps what went where. Obfuscated uploads whose files were each a release of their own ("[9/20] - "14f5999a…"", hep: 13,232) become one. Also: corpus_sweep.py counts the marked ranges without psql's command tag. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-12 09:37:35 +02:00