Daan MeijerandClaude Opus 5.5
8c36744088
- seed_nodes (hashed bearer tokens) and seed_ranges (planned ranges with lease, resume point and counts); migration …140. - SeedOrchestrator: lease (oldest plan first, FOR UPDATE SKIP LOCKED; a lease without a batch for 15 min passes to the next node and resumes at next_id), batch (stores files and fragments, moves next_id), finish, release, plan (from below live indexing / the backfill / earlier plans down to N days back; takes over the group's backfill), status. - Seed API under /api/v1/seed (lease, ranges/:id/batches|done|release, status) behind a node-token middleware; JSON bodies up to 64 MB after gzip. - seed:node runs a node: no database or Redis, outgoing HTTPS only, N connections, gzip batches, retries for network errors and 5xx, drops a range on 409, gives its ranges back on SIGTERM. seed-node/ has a compose file that builds the image from this repository. - seed:token, seed:plan, seed:status for the operator. Header parsing moved to lib/ingest and the range loop to lib/seed/runner, shared by seed:range and seed:node. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
42 lines
1.5 KiB
TypeScript
42 lines
1.5 KiB
TypeScript
import { BaseCommand, args, flags } from '@adonisjs/core/ace'
|
||
import { CommandOptions } from '@adonisjs/core/types/ace'
|
||
import NntpService from '#services/NntpService'
|
||
import SeedOrchestrator from '#services/SeedOrchestrator'
|
||
|
||
/**
|
||
* Plans seed ranges (CONCEPT.md D38) for groups, from below what live
|
||
* indexing, the backfill and earlier plans cover down to `--days` back (or
|
||
* the server's oldest article). A group's backfill stops where its plan
|
||
* starts. Ranges are interleaved across groups, newest first.
|
||
*
|
||
* node ace seed:plan alt.binaries.ath alt.binaries.mom --days 365
|
||
*/
|
||
export default class SeedPlan extends BaseCommand {
|
||
public static commandName = 'seed:plan'
|
||
public static description = 'Plans article ranges of groups for seeding nodes to lease.'
|
||
|
||
public static options: CommandOptions = {
|
||
startApp: true,
|
||
}
|
||
|
||
@args.spread({ description: 'Groups' })
|
||
declare groups: string[]
|
||
|
||
@flags.number({ description: 'How many days back (default: everything the server has)' })
|
||
declare days?: number
|
||
|
||
@flags.number({ description: 'Articles per range', default: 2_000_000 })
|
||
declare size: number
|
||
|
||
public async run() {
|
||
try {
|
||
const summary = await SeedOrchestrator.plan(this.groups, { days: this.days, size: this.size })
|
||
for (const s of summary) {
|
||
this.logger.info(`${s.group}: ${s.ranges.toLocaleString('en')} ranges, articles ${s.from.toLocaleString('en')}–${s.to.toLocaleString('en')}.`)
|
||
}
|
||
} finally {
|
||
await NntpService.shutdown()
|
||
}
|
||
}
|
||
}
|