Rebuild spec: TypeScript vs Rust for the edge worker, measured
Test and deploy / test (push) Successful in 42s
Test and deploy / deploy (push) Skipped

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Daan MeijerandClaude Opus 5.5 committed 2026-10-09 09:38:03 +02:00
1 parent c7b4ebe470
commit b1a6f02de6
1 file changed
+13 -4
+13 -4
View File
@@ -99,10 +99,19 @@ dropping garbage; 28 TB of compressed overview traffic. Throughput goal:
settles).
2. **Edge worker** (node, TypeScript reusing `lib/`). Measured (`edge:bench`,
2026-10-09): 73 k headers/s per core on an a.b.ath range from 2013,
103 k/s on a recent a.b.boneless range (2 M headers each), ~1.2 GB peak
per process; the 25 B lines/day goal needs ~3–4 cores over all nodes,
so no Rust. Per lease: compressed chunks of 10–50 k lines, in order →
2026-10-09, after two rounds of work): ~200–215 k headers/s per core on
recent a.b.boneless, ~320–340 k on a 2013 a.b.ath range, ~0.7 GB peak
per process. **Rust** (`tools/edge-bench-rs`, the same RangeIndexer path
over the same captures; `check.ts` finds every output identical, the
decoded payloads included): 3.5–3.8× less CPU (boneless 2.9 vs 11.1
CPU-s per 2 M headers, ath 2.15 vs 7.5) and a third of the memory
(0.20–0.24 GB). Its gains: no GC, hand-written matching instead of the
regexes, mimalloc, libdeflate (half of zlib's inflate time) and SIMD line
scanning. At the full target (~1.25 M/s) that is ~1.5–2 worker cores
instead of ~6–7; the speed is still set by the NNTP connections. The
cost: a second implementation of lib/headers, lib/release, lib/rules and
the chunk format, kept identical (check.ts) for as long as both exist.
Per lease: compressed chunks of 10–50 k lines, in order →
parse subjects → skip crossposts outside their owner group (of the
indexed groups in the article's Xref, the one with the lowest priority
number; see the group list) → group articles into files → files