4 Commits

Include renames

Author	SHA1	Message	Date
Daan MeijerandClaude Opus 5.5	5ab5399b43	Prober: the claim's random starting points drawn in code ... Postgres evaluated random() once for all 20 slices: every claim still came from one place in the queue (214 of 214 taken from a.b.zogg). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-11 16:00:23 +02:00
Daan MeijerandClaude Opus 5.5	18f5d27413	Prober: claim from random points of the queue, not in id order ... In id order the prober went group by group, and some groups' articles take 0.6-3 s each (a.b.zogg 2016; repeated fetches the same, pipelining 4 deep no faster), where a sample across all years takes 20-60 ms for most: 7 articles/s on 6 connections, against 67/s in its first minutes. A claim now takes 20 slices from random points (353 ms per 200). tools/body_timing.ts: BODY time for the oldest and newest candidates. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-11 15:55:23 +02:00
Daan MeijerandClaude Opus 5.5	738402aefa	Prober: an inspection per candidate; no inner name that is the release's name ... A release can have several candidates of a kind (two RAR sets): keyed by (release, kind) both landed in one upsert ("ON CONFLICT DO UPDATE command cannot affect row a second time"), and the batch was lost. release_inspections is keyed by candidate now; the first run's results are cleared and probed again. A random-named file's yEnc name is often its random name: an inner name equal to the release's name is dropped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-11 15:37:32 +02:00
Daan MeijerandClaude Opus 5.5	8bd6d6aa5f	Rebuild: phase 2's prober reads the probe queue ... probe:run (compose service prober, PROBE_CONNECTIONS, default 6) claims waiting candidates of settled releases (lone, then par2, then archive), fetches each article with BODY over connections of its own (no compression) and inspects it: the yEnc name and size, the content type from the first bytes, a par2 set's files, a RAR volume's entries and whether they're encrypted. Results per release and kind in release_inspections (CHUNK_TABLESPACE, the HDD); on the release passworded, inner_name (the largest content name) and inspected_at. 430 = missing, not yEnc = error, both final; connection trouble puts the candidate back. Fetch and parse in one process: an article takes ~0.1 s per connection, parsing a few ms (no queue in between). Parsers from voyager 1 (lib/yenc, lib/rar, lib/par2, lib/content sniff) with their tests; XoverClient.body; yencode. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-11 15:32:49 +02:00