Rebuild: a lean overview scan for the edge worker
The worker reads the XOVER reply's text directly: fields by position, the owner from the Xref without arrays (crossposts leave before any subject parsing), a subject parsed once per file (cached without its segment counter), a file's details as one shared object, dates as numbers. Garbage collection had been ~22% of the time. Same results as parseOverviews + segmentFromHeader on 800k real lines. edge:bench (one core): boneless 137k → 193k headers/s (CPU 21 → 12.8 s), ath (2013) 143k → 303k; peak RSS 1.15 / 0.87 GB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
981bc01810
commit
638cfa1dcd
5 files changed
+292
-59
No files matched your search
@@ -4,7 +4,7 @@ import { mkdir, readFile, readdir, writeFile } from 'node:fs/promises'
|
||||
import { join } from 'node:path'
|
||||
import { unzipSync } from 'node:zlib'
|
||||
import env from '#start/env'
|
||||
import { XoverClient, parseOverviews } from '#lib/nntp/index'
|
||||
import { XoverClient } from '#lib/nntp/index'
|
||||
import { RangeIndexer } from '#lib/edge/index'
|
||||
import { DEFAULT_RULES, type Rules } from '#lib/rules/index'
|
||||
import db from '@adonisjs/lucid/services/db'
|
||||
@@ -117,11 +117,9 @@ export default class EdgeBench extends BaseCommand {
|
||||
t.inflate += performance.now() - at
|
||||
text += data.length
|
||||
at = performance.now()
|
||||
const overviews = parseOverviews(data)
|
||||
t.parse += performance.now() - at
|
||||
at = performance.now()
|
||||
const next = Number(file.split('-')[2].replace('.gz', '')) + 1
|
||||
indexer.add(overviews, next)
|
||||
// The worker's path: the reply's text straight into the indexer (parsing is part of "index").
|
||||
indexer.addText(data, next)
|
||||
const last = file === files.at(-1)
|
||||
if (last) indexer.finish()
|
||||
t.index += performance.now() - at
|
||||
|
||||
Reference in new issue
Block a user