Rebuild: the live tail

coordinator:run plans the articles after each group's last range every
TAIL_EVERY_SECONDS (120): one LIST ACTIVE per name prefix gives every
group's newest article, and new articles become ranges at priority -1,
leased and prefetched before any history. A group gets a tail range once
it has TAIL_MIN_ARTICLES (100k, a chunk) new or its last planning is
TAIL_MAX_DELAY_MINUTES (15) ago, so quiet groups don't make a range every
round. History (planAll, hourly) is now only for groups not planned yet.
ranges:plan --tail plans one round.

Found on the way: with XFEATURE COMPRESS GZIP on, Eweka compresses LIST
ACTIVE's reply too ("215 NewsGroups Follow [COMPRESS=GZIP]"), and
listActive waited for a plain end that never came (groups:add --top too).
It reads the compressed reply now; the fake server in the test does both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Daan MeijerandClaude Opus 5.5 committed 2026-10-09 10:41:16 +02:00
1 parent df89045f98
commit fd0f2bde75
8 files changed
+166 -7

No files matched your search

+8
View File
@@ -6,6 +6,7 @@ import Planner from '#services/planner'
import FragmentMerger from '#services/fragment_merger'
import Settler from '#services/settler'
import NntpPool from '#services/nntp_pool'
import coordinatorConfig from '#config/coordinator'
const TICK_MS = 2000
const PLAN_EVERY_MS = 60 * 60 * 1000
@@ -55,7 +56,9 @@ export default class CoordinatorRun extends BaseCommand {
await sleep(TICK_MS)
}
})()
// History for new groups hourly; the tail every few minutes.
let plannedAt = 0
let tailedAt = 0
let mergedAt = 0
while (!stopping) {
try {
@@ -66,6 +69,11 @@ export default class CoordinatorRun extends BaseCommand {
const settled = await Settler.run()
if (settled > 0) this.logger.info(`Settled ${settled} releases.`)
}
if (main && Date.now() - tailedAt >= coordinatorConfig.tail.everySeconds * 1000) {
tailedAt = Date.now()
const { groups, ranges } = await Planner.planTail()
if (ranges > 0) this.logger.info(`Tail: ${ranges} ranges for ${groups} groups.`)
}
if (main && Date.now() - mergedAt >= MERGE_EVERY_MS) {
mergedAt = Date.now()
// Rounds until a whole pass over the file keys decides nothing more, or MERGE_BUDGET_MS.