Corpus: stored XOVER chunks for the grouping sweeps
Ranges marked `capture` keep their XOVER chunks in CORPUS_DIR (corpus/<group>/<start>-<end>/, edge-bench's layout, range.json once complete; only chunks that inflate). The prefetcher reads a stored chunk instead of fetching it, so a reset re-indexes them without the news server; a chunk a worker reports corrupt goes from the corpus too. edge-bench gets --releases (release chunks without payload, with their first subjects) and ships in the edge image; deploy/corpus_sweep.py marks more ranges (capture), replays the corpus with the deployed build (replay) and reports suspects and the change since the previous build (report). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
d43603a43f
commit
9ea2669b2a
13 files changed
+389
-11
No files matched your search
+13
@@ -208,6 +208,19 @@ of other groups whose counts show crossposts skipped for it (part 5). Start:
|
||||
the 1000 largest `alt.binaries.*` groups by LIST ACTIVE (~266 k ranges of
|
||||
2 M articles; boneless alone 80 k).
|
||||
|
||||
## Corpus
|
||||
|
||||
The grouping sweeps' dataset (2026-10-12; LOG.md 2026-10-12 (12)): the XOVER
|
||||
chunks of ranges marked `capture`, kept on the HDD in
|
||||
`corpus/<group>/<start>-<end>/` (edge-bench's layout, `range.json` once
|
||||
complete), up to 300 GB. The prefetcher stores them as it fetches and reads
|
||||
them from there afterwards, so a reset re-indexes them without the news
|
||||
server. `deploy/corpus_sweep.py` marks more ranges each sweep (`capture`,
|
||||
spread over groups and eras, the top-100 groups first), replays every stored
|
||||
range with the deployed build's edge-bench (`replay`, ~seconds a range) and
|
||||
reports suspects and the change since the previous build (`report`). A rule
|
||||
change is judged on the whole corpus before a reset.
|
||||
|
||||
## Phase 2
|
||||
|
||||
Built: the prober (2026-10-11; `probe:run`, LOG.md 2026-10-11 (2)). It
|
||||
|
||||
Reference in new issue
Block a user