Seed plans fill every uncovered gap, with priorities; cheaper ingest and file storing
Test and deploy / test (push) Successful in 1m27s
Test and deploy / deploy (push) Failing after 33m7s

- seed:plan plans every part of a group's history no range covers (it
  started below the lowest known range and left a.b.ath's 13 most recent
  months out); seed_ranges.priority orders leases (seed:plan --priority).
- binaries.part_count, kept by the parts insert in the same statement:
  claimCompleted compares it instead of counting parts every batch.
- file_keys: 64-bit hashes of all file keys; FileStore.insert asks it
  before probing every files partition.
- worker:file stores 100 completed binaries per job with set-based queries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Daan MeijerandClaude Opus 5.5 committed 2026-10-06 14:18:01 +02:00
1 parent 61b683a71f
commit 693e6c3244
14 files changed
+473 -132

No files matched your search

+11 -5
View File
@@ -4,9 +4,10 @@ import NntpService from '#services/NntpService'
import SeedOrchestrator from '#services/SeedOrchestrator'
/**
* Plans seed ranges (CONCEPT.md D38) for groups, from below what live
* indexing, the backfill and earlier plans cover down to `--days` back (or
* the server's oldest article). A group's backfill stops where its plan
* Plans seed ranges (CONCEPT.md D38) for groups: every article from below
* what live indexing and the backfill cover down to `--days` back (or the
* server's oldest article) that no earlier range covers. Running it again
* plans only what's missing. A group's backfill stops where its plan
* starts. Ranges are interleaved across groups, newest first.
*
* node ace seed:plan alt.binaries.ath alt.binaries.mom --days 365
@@ -28,11 +29,16 @@ export default class SeedPlan extends BaseCommand {
@flags.number({ description: 'Articles per range', default: 2_000_000 })
declare size: number
@flags.number({ description: 'Leased before ranges of a lower priority (default 0)', default: 0 })
declare priority: number
public async run() {
try {
const summary = await SeedOrchestrator.plan(this.groups, { days: this.days, size: this.size })
const summary = await SeedOrchestrator.plan(this.groups, { days: this.days, size: this.size, priority: this.priority })
for (const s of summary) {
this.logger.info(`${s.group}: ${s.ranges.toLocaleString('en')} ranges, articles ${s.from.toLocaleString('en')}–${s.to.toLocaleString('en')}.`)
this.logger.info(
`${s.group}: ${s.ranges.toLocaleString('en')} new ranges in ${s.gaps} gap(s) not yet planned, articles ${s.from.toLocaleString('en')}–${s.to.toLocaleString('en')}.`
)
}
} finally {
await NntpService.shutdown()