From b1a6f02de61aeb398fbf28c679f9cfae128b5ca0 Mon Sep 17 00:00:00 2001 From: Daan Meijer Date: Fri, 9 Oct 2026 09:38:03 +0200 Subject: [PATCH] Rebuild spec: TypeScript vs Rust for the edge worker, measured Co-Authored-By: Claude Opus 5.5 --- REBUILD.md | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/REBUILD.md b/REBUILD.md index 730e40c..33926d4 100644 --- a/REBUILD.md +++ b/REBUILD.md @@ -99,10 +99,19 @@ dropping garbage; 28 TB of compressed overview traffic. Throughput goal: settles). 2. **Edge worker** (node, TypeScript reusing `lib/`). Measured (`edge:bench`, - 2026-10-09): 73 k headers/s per core on an a.b.ath range from 2013, - 103 k/s on a recent a.b.boneless range (2 M headers each), ~1.2 GB peak - per process; the 25 B lines/day goal needs ~3–4 cores over all nodes, - so no Rust. Per lease: compressed chunks of 10–50 k lines, in order → + 2026-10-09, after two rounds of work): ~200–215 k headers/s per core on + recent a.b.boneless, ~320–340 k on a 2013 a.b.ath range, ~0.7 GB peak + per process. **Rust** (`tools/edge-bench-rs`, the same RangeIndexer path + over the same captures; `check.ts` finds every output identical, the + decoded payloads included): 3.5–3.8× less CPU (boneless 2.9 vs 11.1 + CPU-s per 2 M headers, ath 2.15 vs 7.5) and a third of the memory + (0.20–0.24 GB). Its gains: no GC, hand-written matching instead of the + regexes, mimalloc, libdeflate (half of zlib's inflate time) and SIMD line + scanning. At the full target (~1.25 M/s) that is ~1.5–2 worker cores + instead of ~6–7; the speed is still set by the NNTP connections. The + cost: a second implementation of lib/headers, lib/release, lib/rules and + the chunk format, kept identical (check.ts) for as long as both exist. + Per lease: compressed chunks of 10–50 k lines, in order → parse subjects → skip crossposts outside their owner group (of the indexed groups in the article's Xref, the one with the lowest priority number; see the group list) → group articles into files → files