Corpus: store chunks without the lines the worker only counts
Test and deploy / test (push) Successful in 59s
Test and deploy / edge (push) Successful in 2m3s
Test and deploy / deploy (push) Successful in 1m20s

History fetched ahead into the corpus (bulk HDD) is ~60 TB raw for the
1,180B missing articles, and the indexer keeps only ~38% of headers.
With CORPUS_FILTER (default on), Corpus.write leaves out the lines the
edge worker drops on their own, decided per line as RangeIndexer.line
does: a crosspost another indexed group owns, and a subject that names no
file part (~48% of the indexed ranges' headers). Release-level rules
(garbage, per-article, executable, poster) judge files across chunks and
aren't applied. The chunk is compressed again as zlib (the worker takes
it as before); beside it a .drops (gzip JSON) keeps the rule version,
the owners' priorities and per rule the article numbers (as differences,
crossposts by owner), and range.json says filtered. A test shows the
worker makes the same releases, candidates and fragments of a filtered
chunk. corpus:filter filters chunks stored raw (planned capture ranges;
--all for every range).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Daan MeijerandClaude Opus 5.5 committed 2026-10-23 16:34:09 +02:00
1 parent c1946792b2
commit a66b0980f4
9 files changed
+285 -9

No files matched your search

+14
View File
@@ -230,6 +230,20 @@ range with the deployed build's edge-bench (`replay`, ~seconds a range) and
reports suspects and the change since the previous build (`report`). A rule
change is judged on the whole corpus before a reset.
Since 2026-10-23 also history fetched ahead of the indexing: while the
catalog is over its NVMe cap, ranges marked `capture` (priority >= 0, not
leased) are fetched into the corpus by idle loops, at most perGroup /
perRange of a group at a time, to be indexed from disk once there's room
(LOG.md 2026-10-23 (100)). Stored filtered (`CORPUS_FILTER`, default on):
without the lines the edge worker only counts, decided per line as it does
(a crosspost another indexed group owns, a subject that names no file part;
~48% of headers), the chunk compressed again as zlib; beside it a `.drops`
(gzip JSON: rule version, the owners' priorities, per rule the article
numbers as differences, crossposts by owner) and `filtered` in range.json.
The worker makes the same releases of a filtered chunk; rule_counts lack
those two rules' counts. `corpus:filter` filters chunks stored raw. A
corpus filtered this way no longer serves a sweep of those two rules.
## Phase 2
Built: the prober (2026-10-11; `probe:run`, LOG.md 2026-10-11 (2)). It