Daan MeijerandClaude Opus 5.5
a66b0980f4
History fetched ahead into the corpus (bulk HDD) is ~60 TB raw for the 1,180B missing articles, and the indexer keeps only ~38% of headers. With CORPUS_FILTER (default on), Corpus.write leaves out the lines the edge worker drops on their own, decided per line as RangeIndexer.line does: a crosspost another indexed group owns, and a subject that names no file part (~48% of the indexed ranges' headers). Release-level rules (garbage, per-article, executable, poster) judge files across chunks and aren't applied. The chunk is compressed again as zlib (the worker takes it as before); beside it a .drops (gzip JSON) keeps the rule version, the owners' priorities and per rule the article numbers (as differences, crossposts by owner), and range.json says filtered. A test shows the worker makes the same releases, candidates and fragments of a filtered chunk. corpus:filter filters chunks stored raw (planned capture ranges; --all for every range). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
9 lines
689 B
TypeScript
9 lines
689 B
TypeScript
export { RangeIndexer, postedSpan, chunkOf, candidatesOf, releaseKeyOf, uploadsOf, UPLOAD_GAP_MS, type EdgeBatch, type ReleaseChunk, type DroppedChunk, type ProbeCandidate, type CandidateKind, type RuleCount, type Fragment, type RangeIndexerOptions } from './range_indexer.js'
|
|
export { encodeFrame, decodeFrames, isKeepalive, KEEPALIVE, type Frame } from './stream.js'
|
|
export { toWire, fromWire, type WireBatch } from './wire.js'
|
|
export type { Lease } from './lease.js'
|
|
export { mergeFragments } from './merge.js'
|
|
export type { EdgeSegment, FileMeta } from './scan.js'
|
|
export { throttle } from './throttle.js'
|
|
export { filterOverviews, undelta, type ChunkDrops } from './corpus_filter.js'
|