2 Commits

Include renames

Author	SHA1	Message	Date
Daan MeijerandClaude Opus 5.5	2c14315f94	Rebuild: fragments only for files an edge cut; lone random articles are per-article ... Measured on two neighbouring boneless ranges (merging their fragments): - 99.8% of fragment files were single articles of a random-named 'file' of >100 parts: a random subject per article, which never completes from headers. They now count as per-article. - A file is kept as a fragment when its first part seen is above 1 and comes within the start window (the idle window: uploads in busy groups spread over minutes, and a 50k-article edge zone is ~30 s of boneless), or when it was still open at the range's end with later parts missing. Anything else missing parts is incomplete and only counted. Per range: 70,308 → 130 fragment files; merged, 47 of the 48 files met complete (57,558 segments, ~8% of the range's kept segments). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-09 02:17:22 +02:00
Daan MeijerandClaude Opus 5.5	20910647fa	Rebuild: the batch store (catalog upserts, write-once chunks, candidates, rule counts, fragments, resume point) ... A batch is stored in one transaction and only at its range's resume point, so a retried batch can't count twice. Garbage is judged before per-article: a dropped tool isn't worth reading bodies for either. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-09 00:21:17 +02:00