daan/usenet-indexer
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
118 Commits 5 Branches 0 Tags
fd0f2bde75d4d64bcaf53c66ecce5a43eab9d741
Commit Graph
3 Commits
Author SHA1 Message Date
Daan MeijerandClaude Opus 5.5 73fc9aabdf Rebuild: strings that outlive their chunk are copied (worker 1.14 → 0.66 GB)
Test and deploy / test (push) Successful in 18s
Details
Test and deploy / deploy (push) Skipped
Details 
A slice of a chunk's text keeps the whole text alive (tens of MB) for as
long as the file it belongs to stays open: message-ids, file details and
cache keys are now copies. edge:bench on boneless: peak RSS 1,137 → ~660
MB, external memory 570 → 40–75 MB, the same speed (~190k headers/s);
the same results on 800k real lines. edge:bench prints the breakdown.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 02:36:34 +02:00
Daan MeijerandClaude Opus 5.5 db882f0b48 Rebuild: the segment counter by a backwards scan, one cache from poster+subject to the file 
edge:bench (one core): boneless 193k → 215k headers/s, ath 303k → 343k.
Same results on 800k real lines.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 01:16:04 +02:00
Daan MeijerandClaude Opus 5.5 638cfa1dcd Rebuild: a lean overview scan for the edge worker
Test and deploy / test (push) Successful in 22s
Details
Test and deploy / deploy (push) Skipped
Details 
The worker reads the XOVER reply's text directly: fields by position, the
owner from the Xref without arrays (crossposts leave before any subject
parsing), a subject parsed once per file (cached without its segment
counter), a file's details as one shared object, dates as numbers.
Garbage collection had been ~22% of the time.

Same results as parseOverviews + segmentFromHeader on 800k real lines.
edge:bench (one core): boneless 137k → 193k headers/s (CPU 21 → 12.8 s),
ath (2013) 143k → 303k; peak RSS 1.15 / 0.87 GB.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 01:13:41 +02:00