4 Commits

Include renames

Author	SHA1	Message	Date
Daan MeijerandClaude Opus 5.5	27a26c9614	Rebuild: a release key taken by another group's writer gets the next number ... Writers lock their group only, so two groups' batches can number the same identity key at once (a spam post crossposted to 67 groups: duplicate key ...:65, HTTP 500, the batch retried). Releases are now inserted with ON CONFLICT (release_key) DO NOTHING, and those skipped take the next number (up to 20 tries). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-10 23:22:49 +02:00
Daan MeijerandClaude Opus 5.5	0613984de9	Rebuild: re-uploads are releases of their own ... A poster re-uploading under the same name made one release of all uploads: in a sample, 24% of releases with 2+ files had a file name more than once (4.3% of files extra copies), 70% of those copies posted a day or more apart; their NZBs listed each copy. - File assembly keeps a re-upload's copy apart: a part already in every open copy, posted more than 3 h from them, starts another copy; a missing part fills the nearest copy (a re-send of what was missing), a re-sent part near a copy that has it is ignored as before. - A key's files in a batch split into uploads where posting pauses for more than 3 h, counted from the latest article posted so far (a file that takes hours to post doesn't split it); each upload is judged and chunked on its own (ReleaseChunk.upload, .postedLast; candidates carry their upload). - The catalog matches a chunk to the release of its key whose articles the range's group owns and whose posting window is within 3 h, else starts one (key, key:2, …): releases get identity_key, group_id, last_posted_at. Keys are locked in order (advisory locks), so concurrent batches neither deadlock nor both start one; the merger writes in key order too. The same upload multi-posted to other groups is a release per group. - The gap: 91% of uploads post within 3 h; 73% of re-uploaded copies come later (releases:upload-gaps). releases:duplicates measures the duplicate names. Both implementations; check.ts compares the windows and uploads too (identical on all captures; Rust tests mirror the TypeScript cases). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-09 13:31:35 +02:00
Daan MeijerandClaude Opus 5.5	2c14315f94	Rebuild: fragments only for files an edge cut; lone random articles are per-article ... Measured on two neighbouring boneless ranges (merging their fragments): - 99.8% of fragment files were single articles of a random-named 'file' of >100 parts: a random subject per article, which never completes from headers. They now count as per-article. - A file is kept as a fragment when its first part seen is above 1 and comes within the start window (the idle window: uploads in busy groups spread over minutes, and a 50k-article edge zone is ~30 s of boneless), or when it was still open at the range's end with later parts missing. Anything else missing parts is incomplete and only counted. Per range: 70,308 → 130 fragment files; merged, 47 of the 48 files met complete (57,558 segments, ~8% of the range's kept segments). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-09 02:17:22 +02:00
Daan MeijerandClaude Opus 5.5	20910647fa	Rebuild: the batch store (catalog upserts, write-once chunks, candidates, rule counts, fragments, resume point) ... A batch is stored in one transaction and only at its range's resume point, so a retried batch can't count twice. Garbage is judged before per-article: a dropped tool isn't worth reading bodies for either. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-09 00:21:17 +02:00