From 122a054eab38798c4fd308b0d5386b2e6c0bcc0c Mon Sep 17 00:00:00 2001 From: Daan Meijer Date: Fri, 9 Oct 2026 09:40:45 +0200 Subject: [PATCH] edge-bench-rs: README with the findings and how to compare the outputs Co-Authored-By: Claude Opus 5.5 --- tools/edge-bench-rs/README.md | 190 ++++++++++++++++++++++++++++++++++ 1 file changed, 190 insertions(+) create mode 100644 tools/edge-bench-rs/README.md diff --git a/tools/edge-bench-rs/README.md b/tools/edge-bench-rs/README.md new file mode 100644 index 0000000..b782881 --- /dev/null +++ b/tools/edge-bench-rs/README.md @@ -0,0 +1,190 @@ +# edge-bench-rs: the edge worker's indexing in Rust + +A measurement, not a product. It answers whether a Rust edge worker would be +worth a second implementation of the TypeScript one (`edge:node`, REBUILD.md +part 2). It ports the path that `edge:bench` replays over a captured range: + +- inflate each XOVER chunk; +- scan the overview lines (no object per line); +- apply the owner rule from the Xref; +- build segments (file key, release identity); +- aggregate segments into files, and cut off idle ones as fragments, + per-article or incomplete files; +- every N chunks, judge the completed releases (rules: executables, + readable names, garbage tools and signatures, per-article random + subjects), encode their chunks (format v2) and pick probe candidates. + +It does not port NNTP, the coordinator API, batch posting or anything on the +server side. + +## Findings (2026-10-09) + +Same machine (the workstation, otherwise idle), same captures, 2M headers +each: + +| Capture | TypeScript (`edge:bench`) | Rust (`edge-bench-rs`) | Less CPU | +|---|---|---|---| +| a.b.boneless 2026 (160950000000–160951999999) | 11.1 CPU-s (~215k headers/s), 0.70 GB peak | 2.9 CPU-s (~690k/s), 0.24 GB | 3.8× | +| a.b.boneless 2026 (160952000000–160953999999) | 11.8 CPU-s (~205k/s), 0.70 GB | 3.1 CPU-s (~640k/s), 0.24 GB | 3.8× | +| a.b.ath 2013 (2549992990–2551992989) | 7.5 CPU-s (~320k/s), 0.74 GB | 2.15 CPU-s (~930k/s), 0.20 GB | 3.5× | + +**Comparing the numbers:** +- **CPU seconds are the fair comparison.** `edge:bench`'s headers/s leaves + out the time Node's GC threads use; the Rust binary is single-threaded, so + its wall time is its CPU time. +- **Homelab's cores** (Xeon E5-1650 v4) run the TypeScript ~10% slower than + the workstation's. + +**The Rust optimizations,** measured on boneless 2026 (all checked identical +after each step): + +| Step | Headers/s | +|---|---| +| Straight port: regex crate for the patterns, chrono for dates, zlib-rs, std allocator | 360k | +| File stems and subject remainders by hand: their 12 + 5 regexes were most of a new file's description (333k per boneless range, many of them per-article random subjects) | | +| mimalloc for the many small strings | 495k | +| Upload tools, release-name checks and the common Date formats by hand | 500k | +| libdeflate: inflates a chunk whole in half of zlib's time (1.09 → 0.51 s per range) | | +| memchr (SIMD) for line ends and tabs | ~700k | +| Fewer allocations in `describe` (hashing the parts without joining them): no measurable gain, reverted | | + +Rows without a figure weren't measured on their own; each figure covers the +rows above it. + +**What's left** is spread out: scanning and building segments, two sha1s per +new file, aggregation, cut-off and chunk encoding. `perf` isn't available on +the workstation (`perf_event_paranoid` = 3), so these come from coarse +timers. The binary prints them as "of which aggregate …, evict+cutoff …, +judge+encode …" on stderr. + +**What it means for the full target** (~1.25M articles/s with 50 NNTP +connections): +- **Worker cores:** ~1.5–2 instead of ~6–7. Spreading workers over other + machines (batcave, poweredge-kantoor) becomes mostly unnecessary. +- **Total speed:** unchanged. The account's NNTP connections set it (~30k + articles/s each), not the worker. +- **The cost:** a second implementation of `lib/headers`, `lib/release`, + `lib/rules`, `lib/files` (aggregation, volumes) and the chunk format. + Every rule change has to land in both and be checked identical (below) + for as long as both exist. + +## Running it + +You need a captured range. `edge:bench --capture` saves a range's compressed +XOVER chunks once (it needs `NNTP_*` in the environment and one NNTP +connection): + +```sh +node ace edge:bench --capture --dir /path/to/capture --group alt.binaries.boneless --start 160950000000 +# 2M articles in chunks of 50k (--articles, --chunk), plus range.json +``` + +Benchmark both on it: + +```sh +node ace edge:bench --dir /path/to/capture # TypeScript: headers/s, CPU, inflate/index, peak RSS +cd tools/edge-bench-rs && cargo build --release +./target/release/edge-bench-rs /path/to/capture # Rust: the same summary +``` + +Both take results every 10 chunks (`--batch-chunks`) and give a file up after +5 idle chunks (`--idle`). Keep these equal on both sides when comparing. +The rules are `DEFAULT_RULES` with only the range's group indexed and no +dropped signatures, as in `edge:bench`. + +The two "kept: …" summary lines should already agree: release chunks, +files, segments, payload size, candidates, fragments and samples. The +summary can hide compensating differences, so compare the outputs +themselves. + +## Comparing the output + +```sh +./target/release/edge-bench-rs /path/to/capture --dump /tmp/out.jsonl +cd ../.. && node --import=@poppinss/ts-exec tools/edge-bench-rs/check.ts /path/to/capture /tmp/out.jsonl +``` + +`--dump` writes every output of every batch as JSON lines: + +| Line `t` | Contents | +|---|---| +| `b` | batch boundaries: from, next, headers | +| `r` | releases: every ReleaseChunk field, the payload in base64 | +| `s` | samples: dropped releases kept in full, with their rule | +| `c` | probe candidates | +| `n` | rule counts | +| `f` | fragments | + +`check.ts` runs the TypeScript `RangeIndexer` (from `lib/edge`, not a copy) +on the same capture with the same batching. It matches items by batch and +key: release key, candidate (release, kind, message-id), rule and key, +fragment file key. Then it compares them field by field: + +``` +same batches: 4 TypeScript, 4 Rust +same releases: 1055 TypeScript, 1055 Rust +same samples: 2 TypeScript, 2 Rust +same candidates: 1090 TypeScript, 1090 Rust +same rule counts: 34 TypeScript, 34 Rust +same fragments: 130 TypeScript, 130 Rust +payload bytes: 13728351 TypeScript, 13727247 Rust +``` + +It exits 0 only when everything matches. Otherwise each kind says how many +items are only on one side or differ, and shows the first differing fields. + +- **Payloads are compared decoded.** Both sides go through `decodeChunk` + from `lib/chunks`, and their files are compared, ordered by name and + subject. Node's built-in zstd and the zstd crate frame the same data a + little differently, so the compressed sizes differ by about 0.1%. The + checker prints them, but they don't count as a difference. Decoding the + Rust payloads with the TypeScript decoder also checks the Rust encoder. +- **Order doesn't count.** Order within a batch isn't compared (the two + hash maps iterate differently). Grouping into batches, the order of a + release's files and the order of a fragment's segments are compared, + because they're part of the values. + +Run it after any change on either side. All three captures above are +identical as of c7b4ebe. + +## JavaScript semantics the port keeps (`src/js.rs`, `src/names.rs`) + +Differences found with `check.ts` or the tests, kept as JavaScript has +them: + +- **Regexes without the `u` flag:** `\d` is [0-9], `\b` an ASCII word + boundary, `/i` ASCII-only. They run as `(?-u)` byte regexes, or by hand. + The hand-written versions (file stems, subject remainders, upload tools, + release-name and file-counter checks) are tested against the regex + versions, which are kept for that (`*_re`). +- **JS `\s` and `trim()`** include Unicode spaces (NBSP, U+2000–200A, + U+3000, BOM…); `split(/\s+/)` keeps leading and trailing empty words. +- **`lastIndexOf(c, position)`** finds a match at or before the position + (the owner rule, the counter and filename scans). **`.length`** is in + UTF-16 units (`len16`). **`parseInt`** reads leading digits. +- **`Date.parse`:** chrono follows RFC 5322 and rejects "UTC", which + `Date.parse` takes. It ends about a quarter of boneless's Date headers, so + it's mapped to GMT; check.ts found this. The common formats are parsed by + hand: + + ``` + [Day, ]D[D] Mon YY[YY] HH:MM:SS (GMT|UTC|UT|±hhmm) + ``` + + They ignore the weekday as `Date.parse` does (chrono rejects a weekday + that doesn't fit). Anything else goes to chrono. Rare forms `Date.parse` + takes but chrono doesn't would show up in check.ts as a `postedAt` + difference. +- **Invalid UTF-8** becomes U+FFFD on both sides (`toString('utf8')`, + `from_utf8_lossy`). + +## Files + +| File | Contents | +|---|---| +| `src/main.rs` | arguments, the replay loop, timing, `--dump` | +| `src/indexer.rs` | scan.ts (SegmentScanner, scanOverviewText), aggregator.ts, range_indexer.ts | +| `src/names.rs` | lib/release (identity, random), lib/files/volumes, lib/rules (upload tools, judge) | +| `src/codec.rs` | lib/chunks encodeChunk, format v2 | +| `src/js.rs` | JavaScript semantics, sha1 | +| `check.ts` | the comparison |