Corpus: store chunks without the lines the worker only counts
Test and deploy / test (push) Successful in 59s
Test and deploy / edge (push) Successful in 2m3s
Test and deploy / deploy (push) Successful in 1m20s

History fetched ahead into the corpus (bulk HDD) is ~60 TB raw for the
1,180B missing articles, and the indexer keeps only ~38% of headers.
With CORPUS_FILTER (default on), Corpus.write leaves out the lines the
edge worker drops on their own, decided per line as RangeIndexer.line
does: a crosspost another indexed group owns, and a subject that names no
file part (~48% of the indexed ranges' headers). Release-level rules
(garbage, per-article, executable, poster) judge files across chunks and
aren't applied. The chunk is compressed again as zlib (the worker takes
it as before); beside it a .drops (gzip JSON) keeps the rule version,
the owners' priorities and per rule the article numbers (as differences,
crossposts by owner), and range.json says filtered. A test shows the
worker makes the same releases, candidates and fragments of a filtered
chunk. corpus:filter filters chunks stored raw (planned capture ranges;
--all for every range).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Daan MeijerandClaude Opus 5.5 committed 2026-10-23 16:34:09 +02:00
1 parent c1946792b2
commit a66b0980f4
9 files changed
+285 -9

No files matched your search

+6
View File
@@ -131,6 +131,12 @@ export default {
* ranges waited behind it for hours (1,400 marked, none stored, 34 of 40 connections idle; LOG.md 2026-10-12 (27)).
*/
ahead: env.get('CORPUS_AHEAD', 16),
/**
* Stored without the lines the edge worker drops on their own (crossposts another indexed group owns, obfuscated
* subjects; ~48% of the indexed ranges' headers), what was left out in each chunk's `.drops`: twice the history
* in the same space (LOG.md 2026-10-23 (101)). Off: as the news server sent them.
*/
filter: env.get('CORPUS_FILTER', true),
},
spool: {
dir: env.get('SPOOL_DIR', app.tmpPath('spool')),