# edge-bench-rs: the edge worker's indexing in Rust

A measurement, not a product. It answers whether a Rust edge worker would be
worth a second implementation of the TypeScript one (`edge:node`, REBUILD.md
part 2). It ports the path that `edge:bench` replays over a captured range:

- inflate each XOVER chunk;
- scan the overview lines (no object per line);
- apply the owner rule from the Xref;
- build segments (file key, release identity);
- aggregate segments into files, and cut off idle ones as fragments,
  per-article or incomplete files;
- every N chunks, judge the completed releases (rules: executables,
  readable names, garbage tools and signatures, per-article random
  subjects), encode their chunks (format v2) and pick probe candidates.

It does not port NNTP, the coordinator API, batch posting or anything on the
server side.

## Findings (2026-10-09)

Same machine (the workstation, otherwise idle), same captures, 2M headers
each:

| Capture | TypeScript (`edge:bench`) | Rust (`edge-bench-rs`) | Less CPU |
|---|---|---|---|
| a.b.boneless 2026 (160950000000–160951999999) | 11.1 CPU-s (~215k headers/s), 0.70 GB peak | 2.9 CPU-s (~690k/s), 0.24 GB | 3.8× |
| a.b.boneless 2026 (160952000000–160953999999) | 11.8 CPU-s (~205k/s), 0.70 GB | 3.1 CPU-s (~640k/s), 0.24 GB | 3.8× |
| a.b.ath 2013 (2549992990–2551992989) | 7.5 CPU-s (~320k/s), 0.74 GB | 2.15 CPU-s (~930k/s), 0.20 GB | 3.5× |

**Comparing the numbers:**
- **CPU seconds are the fair comparison.** `edge:bench`'s headers/s leaves
  out the time Node's GC threads use; the Rust binary is single-threaded, so
  its wall time is its CPU time.
- **Homelab's cores** (Xeon E5-1650 v4) run the TypeScript ~10% slower than
  the workstation's.

**The Rust optimizations,** measured on boneless 2026 (all checked identical
after each step):

| Step | Headers/s |
|---|---|
| Straight port: regex crate for the patterns, chrono for dates, zlib-rs, std allocator | 360k |
| File stems and subject remainders by hand: their 12 + 5 regexes were most of a new file's description (333k per boneless range, many of them per-article random subjects) | |
| mimalloc for the many small strings | 495k |
| Upload tools, release-name checks and the common Date formats by hand | 500k |
| libdeflate: inflates a chunk whole in half of zlib's time (1.09 → 0.51 s per range) | |
| memchr (SIMD) for line ends and tabs | ~700k |
| Fewer allocations in `describe` (hashing the parts without joining them): no measurable gain, reverted | |

Rows without a figure weren't measured on their own; each figure covers the
rows above it.

**What's left** is spread out: scanning and building segments, two sha1s per
new file, aggregation, cut-off and chunk encoding. `perf` isn't available on
the workstation (`perf_event_paranoid` = 3), so these come from coarse
timers. The binary prints them as "of which aggregate …, evict+cutoff …,
judge+encode …" on stderr.

**What it means for the full target** (~1.25M articles/s with 50 NNTP
connections):
- **Worker cores:** ~1.5–2 instead of ~6–7. Spreading workers over other
  machines (batcave, poweredge-kantoor) becomes mostly unnecessary.
- **Total speed:** unchanged. The account's NNTP connections set it (~30k
  articles/s each), not the worker.
- **The cost:** a second implementation of `lib/headers`, `lib/release`,
  `lib/rules`, `lib/files` (aggregation, volumes) and the chunk format.
  Every rule change has to land in both and be checked identical (below)
  for as long as both exist.

## Running it

You need a captured range. `edge:bench --capture` saves a range's compressed
XOVER chunks once (it needs `NNTP_*` in the environment and one NNTP
connection):

```sh
node ace edge:bench --capture --dir /path/to/capture --group alt.binaries.boneless --start 160950000000
# 2M articles in chunks of 50k (--articles, --chunk), plus range.json
```

Benchmark both on it:

```sh
node ace edge:bench --dir /path/to/capture        # TypeScript: headers/s, CPU, inflate/index, peak RSS
cd tools/edge-bench-rs && cargo build --release
./target/release/edge-bench-rs /path/to/capture   # Rust: the same summary
```

Both take results every 10 chunks (`--batch-chunks`) and give a file up after
5 idle chunks (`--idle`). Keep these equal on both sides when comparing.
The rules are `DEFAULT_RULES` with only the range's group indexed and no
dropped signatures, as in `edge:bench`.

The two "kept: …" summary lines should already agree: release chunks,
files, segments, payload size, candidates, fragments and samples. The
summary can hide compensating differences, so compare the outputs
themselves.

## Comparing the output

```sh
./target/release/edge-bench-rs /path/to/capture --dump /tmp/out.jsonl
cd ../.. && node --import=@poppinss/ts-exec tools/edge-bench-rs/check.ts /path/to/capture /tmp/out.jsonl
```

`--dump` writes every output of every batch as JSON lines:

| Line `t` | Contents |
|---|---|
| `b` | batch boundaries: from, next, headers |
| `r` | releases: every ReleaseChunk field, the payload in base64 |
| `s` | samples: dropped releases kept in full, with their rule |
| `c` | probe candidates |
| `n` | rule counts |
| `f` | fragments |

`check.ts` runs the TypeScript `RangeIndexer` (from `lib/edge`, not a copy)
on the same capture with the same batching. It matches items by batch and
key: release key, candidate (release, kind, message-id), rule and key,
fragment file key. Then it compares them field by field:

```
same batches: 4 TypeScript, 4 Rust
same releases: 1055 TypeScript, 1055 Rust
same samples: 2 TypeScript, 2 Rust
same candidates: 1090 TypeScript, 1090 Rust
same rule counts: 34 TypeScript, 34 Rust
same fragments: 130 TypeScript, 130 Rust
payload bytes: 13728351 TypeScript, 13727247 Rust
```

It exits 0 only when everything matches. Otherwise each kind says how many
items are only on one side or differ, and shows the first differing fields.

- **Payloads are compared decoded.** Both sides go through `decodeChunk`
  from `lib/chunks`, and their files are compared, ordered by name and
  subject. Node's built-in zstd and the zstd crate frame the same data a
  little differently, so the compressed sizes differ by about 0.1%. The
  checker prints them, but they don't count as a difference. Decoding the
  Rust payloads with the TypeScript decoder also checks the Rust encoder.
- **Order doesn't count.** Order within a batch isn't compared (the two
  hash maps iterate differently). Grouping into batches, the order of a
  release's files and the order of a fragment's segments are compared,
  because they're part of the values.

Run it after any change on either side. All three captures above are
identical as of c7b4ebe.

## JavaScript semantics the port keeps (`src/js.rs`, `src/names.rs`)

Differences found with `check.ts` or the tests, kept as JavaScript has
them:

- **Regexes without the `u` flag:** `\d` is [0-9], `\b` an ASCII word
  boundary, `/i` ASCII-only. They run as `(?-u)` byte regexes, or by hand.
  The hand-written versions (file stems, subject remainders, upload tools,
  release-name and file-counter checks) are tested against the regex
  versions, which are kept for that (`*_re`).
- **JS `\s` and `trim()`** include Unicode spaces (NBSP, U+2000–200A,
  U+3000, BOM…); `split(/\s+/)` keeps leading and trailing empty words.
- **`lastIndexOf(c, position)`** finds a match at or before the position
  (the owner rule, the counter and filename scans). **`.length`** is in
  UTF-16 units (`len16`). **`parseInt`** reads leading digits.
- **`Date.parse`:** chrono follows RFC 5322 and rejects "UTC", which
  `Date.parse` takes. It ends about a quarter of boneless's Date headers, so
  it's mapped to GMT; check.ts found this. The common formats are parsed by
  hand:

  ```
  [Day, ]D[D] Mon YY[YY] HH:MM:SS (GMT|UTC|UT|±hhmm)
  ```

  They ignore the weekday as `Date.parse` does (chrono rejects a weekday
  that doesn't fit). Anything else goes to chrono. Rare forms `Date.parse`
  takes but chrono doesn't would show up in check.ts as a `postedAt`
  difference.
- **Invalid UTF-8** becomes U+FFFD on both sides (`toString('utf8')`,
  `from_utf8_lossy`).

## Files

| File | Contents |
|---|---|
| `src/main.rs` | arguments, the replay loop, timing, `--dump` |
| `src/indexer.rs` | scan.ts (SegmentScanner, scanOverviewText), aggregator.ts, range_indexer.ts |
| `src/names.rs` | lib/release (identity, random), lib/files/volumes, lib/rules (upload tools, judge) |
| `src/codec.rs` | lib/chunks encodeChunk, format v2 |
| `src/js.rs` | JavaScript semantics, sha1 |
| `check.ts` | the comparison |