Rebuild: first end-to-end runs; prefetch per chunk, executable rule, wire fixes
Test and deploy / test (push) Successful in 20s
Test and deploy / deploy (push) Skipped

End to end on the workstation (coordinator:run, the API, edge:node, real
NNTP): a.b.e-book.french and a.b.cd.image.playstation2.dvdiso ranges.

- Prefetcher: every connection takes the next missing chunk (leased
  ranges first, in order); one connection per range (~26k lines/s) kept
  the worker waiting
- executable rule: releases of bare executables (malware floods named
  after films and software) are dropped; they were 92% of the catalog rows
  of the test ranges, 21% of the segments
- fragments per file (details once) and gzipped batch bodies; the JSON
  limit (1 MB by default) raised to 256 MB, empty strings kept (a rule
  count's key is '')
- release_chunks.batch_from is a bigint (boneless article numbers)
- tests get their own database; edge:bench --store times BatchStore:
  1.3 s for a boneless range (1.24 ms per release chunk)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Daan MeijerandClaude Opus 5.5 committed 2026-10-09 01:07:11 +02:00
1 parent d5f168d8d2
commit cd06f96f85
15 files changed
+201 -99

No files matched your search

+8 -2
View File
@@ -69,7 +69,7 @@ class BatchStore {
const rows = releases.slice(i, i + 500).map((r) => ({
release_id: ids.get(r.key)!,
range_id: rangeId,
batch_seq: batch.from,
batch_from: batch.from,
files: r.files,
segments: r.segments,
payload: r.payload,
@@ -100,7 +100,13 @@ class BatchStore {
}
for (let i = 0; i < batch.fragments.length; i += 2000) {
const rows = batch.fragments.slice(i, i + 2000).map((f) => ({ range_id: rangeId, file_key: f.fileKey, article_number: f.articleNumber, segment: JSON.stringify(f) }))
const rows = batch.fragments.slice(i, i + 2000).map((f) => ({
range_id: rangeId,
file_key: f.fileKey,
article_number: Math.min(...f.segments.map((s) => s[3])),
segments: f.segments.length,
fragment: JSON.stringify(f),
}))
await trx.insertQuery().table('fragments').multiInsert(rows)
}