Rebuild: first end-to-end runs; prefetch per chunk, executable rule, wire fixes
End to end on the workstation (coordinator:run, the API, edge:node, real NNTP): a.b.e-book.french and a.b.cd.image.playstation2.dvdiso ranges. - Prefetcher: every connection takes the next missing chunk (leased ranges first, in order); one connection per range (~26k lines/s) kept the worker waiting - executable rule: releases of bare executables (malware floods named after films and software) are dropped; they were 92% of the catalog rows of the test ranges, 21% of the segments - fragments per file (details once) and gzipped batch bodies; the JSON limit (1 MB by default) raised to 256 MB, empty strings kept (a rule count's key is '') - release_chunks.batch_from is a bigint (boneless article numbers) - tests get their own database; edge:bench --store times BatchStore: 1.3 s for a boneless range (1.24 ms per release chunk) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
d5f168d8d2
commit
cd06f96f85
15 files changed
+201
-99
No files matched your search
@@ -69,7 +69,7 @@ class BatchStore {
|
||||
const rows = releases.slice(i, i + 500).map((r) => ({
|
||||
release_id: ids.get(r.key)!,
|
||||
range_id: rangeId,
|
||||
batch_seq: batch.from,
|
||||
batch_from: batch.from,
|
||||
files: r.files,
|
||||
segments: r.segments,
|
||||
payload: r.payload,
|
||||
@@ -100,7 +100,13 @@ class BatchStore {
|
||||
}
|
||||
|
||||
for (let i = 0; i < batch.fragments.length; i += 2000) {
|
||||
const rows = batch.fragments.slice(i, i + 2000).map((f) => ({ range_id: rangeId, file_key: f.fileKey, article_number: f.articleNumber, segment: JSON.stringify(f) }))
|
||||
const rows = batch.fragments.slice(i, i + 2000).map((f) => ({
|
||||
range_id: rangeId,
|
||||
file_key: f.fileKey,
|
||||
article_number: Math.min(...f.segments.map((s) => s[3])),
|
||||
segments: f.segments.length,
|
||||
fragment: JSON.stringify(f),
|
||||
}))
|
||||
await trx.insertQuery().table('fragments').multiInsert(rows)
|
||||
}
|
||||
|
||||
|
||||
Reference in new issue
Block a user