4 Commits

Include renames

Author	SHA1	Message	Date
Daan MeijerandClaude Opus 5.5	03ca96b255	Fix findCounter infinite loop; corpus filter skips chunks past 256 MB ... lastIndexOf('(', -1) searches from 0 again, so a subject starting with a "(" that isn't a counter, without a quote, looped forever. The corpus filter hit it at 11:23 on Oct 24: the coordinator (and corpus:filter) spun at 100% CPU for 10 hours and no range finished. The Rust find_counter stops at 0 already. A chunk inflating past V8's longest string can't be filtered as text; such a chunk is now stored unfiltered instead of throwing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-24 21:08:22 +02:00
Daan MeijerandClaude Opus 5.5	a66b0980f4	Corpus: store chunks without the lines the worker only counts ... History fetched ahead into the corpus (bulk HDD) is ~60 TB raw for the 1,180B missing articles, and the indexer keeps only ~38% of headers. With CORPUS_FILTER (default on), Corpus.write leaves out the lines the edge worker drops on their own, decided per line as RangeIndexer.line does: a crosspost another indexed group owns, and a subject that names no file part (~48% of the indexed ranges' headers). Release-level rules (garbage, per-article, executable, poster) judge files across chunks and aren't applied. The chunk is compressed again as zlib (the worker takes it as before); beside it a .drops (gzip JSON) keeps the rule version, the owners' priorities and per rule the article numbers (as differences, crossposts by owner), and range.json says filtered. A test shows the worker makes the same releases, candidates and fragments of a filtered chunk. corpus:filter filters chunks stored raw (planned capture ranges; --all for every range). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-23 16:34:09 +02:00
Daan MeijerandClaude Opus 5.5	7b436f3f77	Idle connections fetch capture ranges into the corpus ... The spool only fetches what workers lease soon, in priority order, and the tail never ends: after a reset the 1,400 capture ranges waited behind it while 34 of 40 connections idled. Up to CORPUS_AHEAD (8) planned capture ranges whose chunks aren't all stored are now targets after everything else, fetched into the corpus only (no spool); a later lease reads them from disk. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-12 20:11:57 +02:00
Daan MeijerandClaude Opus 5.5	9ea2669b2a	Corpus: stored XOVER chunks for the grouping sweeps ... Ranges marked `capture` keep their XOVER chunks in CORPUS_DIR (corpus/<group>/<start>-<end>/, edge-bench's layout, range.json once complete; only chunks that inflate). The prefetcher reads a stored chunk instead of fetching it, so a reset re-indexes them without the news server; a chunk a worker reports corrupt goes from the corpus too. edge-bench gets --releases (release chunks without payload, with their first subjects) and ships in the edge image; deploy/corpus_sweep.py marks more ranges (capture), replays the corpus with the deployed build (replay) and reports suspects and the change since the previous build (report). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-12 09:28:22 +02:00