Corpus: store chunks without the lines the worker only counts
History fetched ahead into the corpus (bulk HDD) is ~60 TB raw for the 1,180B missing articles, and the indexer keeps only ~38% of headers. With CORPUS_FILTER (default on), Corpus.write leaves out the lines the edge worker drops on their own, decided per line as RangeIndexer.line does: a crosspost another indexed group owns, and a subject that names no file part (~48% of the indexed ranges' headers). Release-level rules (garbage, per-article, executable, poster) judge files across chunks and aren't applied. The chunk is compressed again as zlib (the worker takes it as before); beside it a .drops (gzip JSON) keeps the rule version, the owners' priorities and per rule the article numbers (as differences, crossposts by owner), and range.json says filtered. A test shows the worker makes the same releases, candidates and fragments of a filtered chunk. corpus:filter filters chunks stored raw (planned capture ranges; --all for every range). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
c1946792b2
commit
a66b0980f4
9 files changed
+285
-9
No files matched your search
@@ -131,6 +131,12 @@ export default {
|
||||
* ranges waited behind it for hours (1,400 marked, none stored, 34 of 40 connections idle; LOG.md 2026-10-12 (27)).
|
||||
*/
|
||||
ahead: env.get('CORPUS_AHEAD', 16),
|
||||
/**
|
||||
* Stored without the lines the edge worker drops on their own (crossposts another indexed group owns, obfuscated
|
||||
* subjects; ~48% of the indexed ranges' headers), what was left out in each chunk's `.drops`: twice the history
|
||||
* in the same space (LOG.md 2026-10-23 (101)). Off: as the news server sent them.
|
||||
*/
|
||||
filter: env.get('CORPUS_FILTER', true),
|
||||
},
|
||||
spool: {
|
||||
dir: env.get('SPOOL_DIR', app.tmpPath('spool')),
|
||||
|
||||
Reference in new issue
Block a user