Rebuild: the live tail
coordinator:run plans the articles after each group's last range every
TAIL_EVERY_SECONDS (120): one LIST ACTIVE per name prefix gives every
group's newest article, and new articles become ranges at priority -1,
leased and prefetched before any history. A group gets a tail range once
it has TAIL_MIN_ARTICLES (100k, a chunk) new or its last planning is
TAIL_MAX_DELAY_MINUTES (15) ago, so quiet groups don't make a range every
round. History (planAll, hourly) is now only for groups not planned yet.
ranges:plan --tail plans one round.
Found on the way: with XFEATURE COMPRESS GZIP on, Eweka compresses LIST
ACTIVE's reply too ("215 NewsGroups Follow [COMPRESS=GZIP]"), and
listActive waited for a plain end that never came (groups:add --top too).
It reads the compressed reply now; the fake server in the test does both.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
df89045f98
commit
fd0f2bde75
8 files changed
+166
-7
No files matched your search
@@ -12,6 +12,20 @@ export default {
|
||||
* 2026-10-09).
|
||||
*/
|
||||
chunkArticles: env.get('CHUNK_ARTICLES', 100_000),
|
||||
/**
|
||||
* The live tail (REBUILD.md, coordinator): new articles become ranges
|
||||
* every `everySeconds` (one LIST ACTIVE per name prefix), leased and
|
||||
* fetched before any history. A group gets a tail range once it has
|
||||
* `minArticles` new (a chunk) or its oldest unplanned article waited
|
||||
* `maxDelayMinutes`, so a quiet group doesn't make a range every round.
|
||||
*/
|
||||
tail: {
|
||||
everySeconds: env.get('TAIL_EVERY_SECONDS', 120),
|
||||
minArticles: env.get('TAIL_MIN_ARTICLES', 100_000),
|
||||
maxDelayMinutes: env.get('TAIL_MAX_DELAY_MINUTES', 15),
|
||||
/** Before the history's 0 (oldest) … 1000 (newest). */
|
||||
priority: -1,
|
||||
},
|
||||
/** A leased range without a heartbeat for this long goes back to the queue. */
|
||||
leaseTtlSeconds: env.get('LEASE_TTL_SECONDS', 15 * 60),
|
||||
/** NNTP connections the coordinator fetches headers with. */
|
||||
|
||||
Reference in new issue
Block a user