From 53bfea8d15af97c520b40766b0ad80662ebb37ac Mon Sep 17 00:00:00 2001 From: Daan Meijer Date: Fri, 9 Oct 2026 03:37:04 +0200 Subject: [PATCH] Rebuild spec: measurements after the second performance round Co-Authored-By: Claude Opus 5.5 --- REBUILD.md | 30 ++++++++++++++++++++---------- 1 file changed, 20 insertions(+), 10 deletions(-) diff --git a/REBUILD.md b/REBUILD.md index df96e5b..51aad43 100644 --- a/REBUILD.md +++ b/REBUILD.md @@ -14,17 +14,27 @@ prefetch spool, leases, rule sets, fragment merger, settling), the node API (`edge:node`, `--processes`), the catalog and chunk store, NZBs and a name search. Measured on the workstation (4 cores, 100 Mbit, Eweka): -- Edge worker, one core: ~190–215 k headers/s on recent a.b.boneless, - ~330 k on a 2013 a.b.ath range; ~0.66 GB peak per process. (Start: - 73–103 k/s, 1.0–1.2 GB.) -- NNTP: ~22–30 k overview lines/s per connection in chunks of 100 k - (Eweka takes ~1.5–2 s per command); connections don't slow each other. -- End to end (coordinator with 10 connections, web server, Postgres and 3 - worker processes on one 4-core machine): 244 k headers/s; the server side - used 10% (web) + 22% (coordinator) + 32% (Postgres) of a core. +- Edge worker, one core: ~210–215 k headers/s on recent a.b.boneless, + ~330–340 k on a 2013 a.b.ath range; ~0.7–0.8 GB peak per process. + (Start: 73–103 k/s, 1.0–1.2 GB.) What's left is spread out (the + per-file key ~12%, GC ~10%, zlib ~10%, the scan ~6%). +- NNTP: Eweka takes ~1.5–2 s per XOVER command; ~22–30 k lines/s per + connection in chunks of 100 k when the line allows; connections don't + slow each other. +- Bandwidth: compressed XOVER is ~69 B per article on recent boneless + (100 k-article chunks of ~7 MB). The workstation's 100 Mbit line carries + ~140–150 k articles/s, which is where its end-to-end runs now stop (10 + connections 90–99% busy, ~89 Mbit/s). +- End to end at that rate (one 4-core machine): workers about half busy + each; server side ~17% (coordinator) + ~8% (web) + ~3% (Postgres) of a + core. (Before this round's fixes Postgres was at 32%: the merger's + neighbour query read every range per file.) - Storage: chunks 17.9–20.3 B per segment (format v2); a release ~1.1 KB. -- For the whole target (528 B overview lines): ~4–6 days at 50 connections, - ~3 worker cores. +- For the whole target (528 B overview lines): ~36 TB of compressed XOVER + to download. At 50 connections (~1–1.25 M lines/s, ~0.6–0.7 Gbit/s) + ~5–6 days, ~5–6 worker cores, two coordinator shards (one process + handles ~60 MB/s), under a core for web and Postgres. The line is the + limit below ~0.7 Gbit/s: at 1 Gbit, ~3.5 days at best. Not built yet: re-running ranges for a retracted rule (the counts and rule versions are recorded), deployment (Dockerfile, compose), phase 2.