Seeding node: gzip overviews, batched SFV lookups, fragments beside fetching
Test and deploy / test (push) Successful in 1m0s
Test and deploy / deploy (push) Successful in 1m43s

- lib/nntp/xover_client: a minimal one-connection client for seed:range
  with XFEATURE COMPRESS GZIP (a zlib stream after "224 … [COMPRESS=GZIP]",
  then a "." line; 420 for an empty range). About 6x less header traffic.
  Overviews come out with the field names nntp-js uses.
- SfvService.claimAll: one sfv_entries lookup per stored batch.
- Fragments go through the regular ingest path beside fetching (at most
  two batches in flight); a failure fails the run at the end.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Daan MeijerandClaude Opus 5.5 committed 2026-10-05 11:11:41 +02:00
1 parent 68eb9a44fd
commit 97bdf8e982
5 files changed
+329 -26

No files matched your search

+7 -8
View File
@@ -89,15 +89,14 @@ class SeedService {
// As worker:file does for a new file: an SFV posted earlier may list it (D34), then inspection.
const rowById = new Map(rows.map((row) => [fileIds.get(row.file_key), row]))
const candidates: { name: string; data: { fileId: number }; opts: { jobId: string } }[] = []
for (const file of inserted) {
const added = inserted.flatMap((file) => {
const row = rowById.get(file.id)
if (!row) continue
await SfvService.claim({ id: file.id, name: row.filename, poster: row.poster, date: row.date, release_id: row.release_id })
if (ContentService.isCandidate(row.filename)) {
candidates.push({ name: 'process-collection', data: { fileId: file.id }, opts: { jobId: `file-${file.id}` } })
}
}
return row ? [{ id: file.id, name: row.filename, poster: row.poster, date: row.date, release_id: row.release_id }] : []
})
await SfvService.claimAll(added)
const candidates = added
.filter((file) => ContentService.isCandidate(file.name))
.map((file) => ({ name: 'process-collection', data: { fileId: file.id }, opts: { jobId: `file-${file.id}` } }))
if (candidates.length > 0) await QueueService.collectionQueue.addBulk(candidates)
result.inspections = candidates.length
return result