Rebuild spec: TypeScript vs Rust for the edge worker, measured
Test and deploy / test (push) Successful in 42s
Test and deploy / deploy (push) Skipped

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Daan MeijerandClaude Opus 5.5 committed 2026-10-09 09:38:03 +02:00
1 parent c7b4ebe470
commit b1a6f02de6
1 file changed
+13 -4
+13 -4
View File
@@ -99,10 +99,19 @@ dropping garbage; 28 TB of compressed overview traffic. Throughput goal:
settles). settles).
2. **Edge worker** (node, TypeScript reusing `lib/`). Measured (`edge:bench`, 2. **Edge worker** (node, TypeScript reusing `lib/`). Measured (`edge:bench`,
2026-10-09): 73 k headers/s per core on an a.b.ath range from 2013, 2026-10-09, after two rounds of work): ~200–215 k headers/s per core on
103 k/s on a recent a.b.boneless range (2 M headers each), ~1.2 GB peak recent a.b.boneless, ~320–340 k on a 2013 a.b.ath range, ~0.7 GB peak
per process; the 25 B lines/day goal needs ~3–4 cores over all nodes, per process. **Rust** (`tools/edge-bench-rs`, the same RangeIndexer path
so no Rust. Per lease: compressed chunks of 10–50 k lines, in order → over the same captures; `check.ts` finds every output identical, the
decoded payloads included): 3.5–3.8× less CPU (boneless 2.9 vs 11.1
CPU-s per 2 M headers, ath 2.15 vs 7.5) and a third of the memory
(0.20–0.24 GB). Its gains: no GC, hand-written matching instead of the
regexes, mimalloc, libdeflate (half of zlib's inflate time) and SIMD line
scanning. At the full target (~1.25 M/s) that is ~1.5–2 worker cores
instead of ~6–7; the speed is still set by the NNTP connections. The
cost: a second implementation of lib/headers, lib/release, lib/rules and
the chunk format, kept identical (check.ts) for as long as both exist.
Per lease: compressed chunks of 10–50 k lines, in order →
parse subjects → skip crossposts outside their owner group (of the parse subjects → skip crossposts outside their owner group (of the
indexed groups in the article's Xref, the one with the lowest priority indexed groups in the article's Xref, the one with the lowest priority
number; see the group list) → group articles into files → files number; see the group list) → group articles into files → files