Corpus: store chunks without the lines the worker only counts
Test and deploy / test (push) Successful in 59s
Test and deploy / edge (push) Successful in 2m3s
Test and deploy / deploy (push) Successful in 1m20s

History fetched ahead into the corpus (bulk HDD) is ~60 TB raw for the
1,180B missing articles, and the indexer keeps only ~38% of headers.
With CORPUS_FILTER (default on), Corpus.write leaves out the lines the
edge worker drops on their own, decided per line as RangeIndexer.line
does: a crosspost another indexed group owns, and a subject that names no
file part (~48% of the indexed ranges' headers). Release-level rules
(garbage, per-article, executable, poster) judge files across chunks and
aren't applied. The chunk is compressed again as zlib (the worker takes
it as before); beside it a .drops (gzip JSON) keeps the rule version,
the owners' priorities and per rule the article numbers (as differences,
crossposts by owner), and range.json says filtered. A test shows the
worker makes the same releases, candidates and fragments of a filtered
chunk. corpus:filter filters chunks stored raw (planned capture ranges;
--all for every range).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Daan MeijerandClaude Opus 5.5 committed 2026-10-23 16:34:09 +02:00
1 parent c1946792b2
commit a66b0980f4
9 files changed
+285 -9

No files matched your search

+1
View File
@@ -51,6 +51,7 @@ export default await Env.create(new URL('../', import.meta.url), {
NNTP_HEDGE_CONNECTIONS: Env.schema.number.optional(),
NNTP_CONNECTION_MAX_MINUTES: Env.schema.number.optional(),
NNTP_IDLE_CLOSE_SECONDS: Env.schema.number.optional(),
CORPUS_FILTER: Env.schema.boolean.optional(),
NNTP_STALL_KB_PER_MINUTE: Env.schema.number.optional(),
// Newznab API key (empty: no key needed)