Corpus: stored XOVER chunks for the grouping sweeps
Test and deploy / test (push) Successful in 43s
Test and deploy / edge (push) Successful in 1m48s
Test and deploy / deploy (push) Successful in 1m23s

Ranges marked `capture` keep their XOVER chunks in CORPUS_DIR
(corpus/<group>/<start>-<end>/, edge-bench's layout, range.json once
complete; only chunks that inflate). The prefetcher reads a stored chunk
instead of fetching it, so a reset re-indexes them without the news
server; a chunk a worker reports corrupt goes from the corpus too.

edge-bench gets --releases (release chunks without payload, with their
first subjects) and ships in the edge image; deploy/corpus_sweep.py marks
more ranges (capture), replays the corpus with the deployed build (replay)
and reports suspects and the change since the previous build (report).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Daan MeijerandClaude Opus 5.5 committed 2026-10-12 09:28:22 +02:00
1 parent d43603a43f
commit 9ea2669b2a
13 files changed
+389 -11

No files matched your search

+13
View File
@@ -208,6 +208,19 @@ of other groups whose counts show crossposts skipped for it (part 5). Start:
the 1000 largest `alt.binaries.*` groups by LIST ACTIVE (~266 k ranges of
2 M articles; boneless alone 80 k).
## Corpus
The grouping sweeps' dataset (2026-10-12; LOG.md 2026-10-12 (12)): the XOVER
chunks of ranges marked `capture`, kept on the HDD in
`corpus/<group>/<start>-<end>/` (edge-bench's layout, `range.json` once
complete), up to 300 GB. The prefetcher stores them as it fetches and reads
them from there afterwards, so a reset re-indexes them without the news
server. `deploy/corpus_sweep.py` marks more ranges each sweep (`capture`,
spread over groups and eras, the top-100 groups first), replays every stored
range with the deployed build's edge-bench (`replay`, ~seconds a range) and
reports suspects and the change since the previous build (`report`). A rule
change is judged on the whole corpus before a reset.
## Phase 2
Built: the prober (2026-10-11; `probe:run`, LOG.md 2026-10-11 (2)). It