Seed orchestrator: remote nodes lease ranges and post results over HTTPS
Test and deploy / test (push) Successful in 1m4s
Test and deploy / deploy (push) Successful in 1m37s

- seed_nodes (hashed bearer tokens) and seed_ranges (planned ranges with
  lease, resume point and counts); migration …140.
- SeedOrchestrator: lease (oldest plan first, FOR UPDATE SKIP LOCKED; a
  lease without a batch for 15 min passes to the next node and resumes at
  next_id), batch (stores files and fragments, moves next_id), finish,
  release, plan (from below live indexing / the backfill / earlier plans
  down to N days back; takes over the group's backfill), status.
- Seed API under /api/v1/seed (lease, ranges/:id/batches|done|release,
  status) behind a node-token middleware; JSON bodies up to 64 MB after
  gzip.
- seed:node runs a node: no database or Redis, outgoing HTTPS only, N
  connections, gzip batches, retries for network errors and 5xx, drops a
  range on 409, gives its ranges back on SIGTERM. seed-node/ has a compose
  file that builds the image from this repository.
- seed:token, seed:plan, seed:status for the operator. Header parsing moved
  to lib/ingest and the range loop to lib/seed/runner, shared by seed:range
  and seed:node.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Daan MeijerandClaude Opus 5.5 committed 2026-10-05 12:02:01 +02:00
1 parent 97bdf8e982
commit 8c36744088
24 files changed
+1198 -246

No files matched your search

+54
View File
@@ -348,6 +348,60 @@ export class ReleaseSchema extends BaseModel {
declare source: string | null
}
export class SeedNodeSchema extends BaseModel {
static $columns = ['createdAt', 'disabled', 'id', 'lastSeenAt', 'name', 'tokenHash'] as const
$columns = SeedNodeSchema.$columns
@column.dateTime({ autoCreate: true })
declare createdAt: DateTime
@column()
declare disabled: boolean
@column({ isPrimary: true })
declare id: number
@column.dateTime()
declare lastSeenAt: DateTime | null
@column()
declare name: string
@column()
declare tokenHash: string
}
export class SeedRangeSchema extends BaseModel {
static $columns = ['attempts', 'createdAt', 'doneAt', 'endId', 'error', 'files', 'fragments', 'groupName', 'headers', 'heartbeatAt', 'id', 'leasedAt', 'nextId', 'nodeId', 'startId', 'status'] as const
$columns = SeedRangeSchema.$columns
@column()
declare attempts: number
@column.dateTime({ autoCreate: true })
declare createdAt: DateTime
@column.dateTime()
declare doneAt: DateTime | null
@column()
declare endId: bigint | number
@column()
declare error: string | null
@column()
declare files: bigint | number
@column()
declare fragments: bigint | number
@column()
declare groupName: string
@column()
declare headers: bigint | number
@column.dateTime()
declare heartbeatAt: DateTime | null
@column({ isPrimary: true })
declare id: bigint | number
@column.dateTime()
declare leasedAt: DateTime | null
@column()
declare nextId: bigint | number
@column()
declare nodeId: number | null
@column()
declare startId: bigint | number
@column()
declare status: string
}
export class SfvEntrySchema extends BaseModel {
static $columns = ['crc', 'name', 'postedAt', 'poster', 'releaseId'] as const
$columns = SfvEntrySchema.$columns