Seed plans fill every uncovered gap, with priorities; cheaper ingest and file storing
- seed:plan plans every part of a group's history no range covers (it started below the lowest known range and left a.b.ath's 13 most recent months out); seed_ranges.priority orders leases (seed:plan --priority). - binaries.part_count, kept by the parts insert in the same statement: claimCompleted compares it instead of counting parts every batch. - file_keys: 64-bit hashes of all file keys; FileStore.insert asks it before probing every files partition. - worker:file stores 100 completed binaries per job with set-based queries. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
61b683a71f
commit
693e6c3244
14 files changed
+473
-132
No files matched your search
+11
-5
@@ -4,9 +4,10 @@ import NntpService from '#services/NntpService'
|
||||
import SeedOrchestrator from '#services/SeedOrchestrator'
|
||||
|
||||
/**
|
||||
* Plans seed ranges (CONCEPT.md D38) for groups, from below what live
|
||||
* indexing, the backfill and earlier plans cover down to `--days` back (or
|
||||
* the server's oldest article). A group's backfill stops where its plan
|
||||
* Plans seed ranges (CONCEPT.md D38) for groups: every article from below
|
||||
* what live indexing and the backfill cover down to `--days` back (or the
|
||||
* server's oldest article) that no earlier range covers. Running it again
|
||||
* plans only what's missing. A group's backfill stops where its plan
|
||||
* starts. Ranges are interleaved across groups, newest first.
|
||||
*
|
||||
* node ace seed:plan alt.binaries.ath alt.binaries.mom --days 365
|
||||
@@ -28,11 +29,16 @@ export default class SeedPlan extends BaseCommand {
|
||||
@flags.number({ description: 'Articles per range', default: 2_000_000 })
|
||||
declare size: number
|
||||
|
||||
@flags.number({ description: 'Leased before ranges of a lower priority (default 0)', default: 0 })
|
||||
declare priority: number
|
||||
|
||||
public async run() {
|
||||
try {
|
||||
const summary = await SeedOrchestrator.plan(this.groups, { days: this.days, size: this.size })
|
||||
const summary = await SeedOrchestrator.plan(this.groups, { days: this.days, size: this.size, priority: this.priority })
|
||||
for (const s of summary) {
|
||||
this.logger.info(`${s.group}: ${s.ranges.toLocaleString('en')} ranges, articles ${s.from.toLocaleString('en')}–${s.to.toLocaleString('en')}.`)
|
||||
this.logger.info(
|
||||
`${s.group}: ${s.ranges.toLocaleString('en')} new ranges in ${s.gaps} gap(s) not yet planned, articles ${s.from.toLocaleString('en')}–${s.to.toLocaleString('en')}.`
|
||||
)
|
||||
}
|
||||
} finally {
|
||||
await NntpService.shutdown()
|
||||
|
||||
Reference in new issue
Block a user