usenet-indexer/edge/README.md

# edge: voyager 2's edge worker (Rust)

The product: `edge-node` (`src/bin/edge-node.rs`) is the worker that runs
on homelab (compose service `edge`, image target `edge`). It leases ranges
from the node API, reads their prefetched XOVER chunks, finds releases and
posts batches. `edge-bench` (`src/main.rs`) replays a captured range through
the same indexing (`src/lib.rs`) for timing and for comparing.

The TypeScript `RangeIndexer` (`lib/edge`, `node ace edge:node`) is the
reference: `check.ts` runs both on a capture and compares every output field
by field. An identity or rule change goes into both and has to come out
identical. CI (`.gitea/workflows/deploy.yaml`, job `edge`) runs `cargo test`
and `check-fixtures.sh`, which compares both on the committed captures in
`fixtures/` (25k overview lines each of a.b.models, a.b.ath and
a.b.boneless, in 5 chunks); a push to `rebuild` deploys only when they
match.

```sh
cargo test --release --manifest-path edge/Cargo.toml
edge/check-fixtures.sh                       # from the repository root
```

In production since 2026-10-09: 122M headers in 10 minutes with 0.37 cores
and 565 MB, next to the TypeScript workers' 24M with 0.61 cores and 11.3 GB
(8 threads vs 8 processes): about 8× less CPU per header and 20× less
memory. See "The worker" below.

It started as a measurement (moved from `tools/edge-bench-rs` on
2026-10-11): whether a Rust worker would be worth a second implementation.
The findings below are from then.

## Findings (2026-10-09)

Same machine (the workstation, otherwise idle), same captures, 2M headers
each:

| Capture | TypeScript (`edge:bench`) | Rust (`edge-bench`) | Less CPU |
|---|---|---|---|
| a.b.boneless 2026 (160950000000–160951999999) | 11.1 CPU-s (~215k headers/s), 0.70 GB peak | 2.9 CPU-s (~690k/s), 0.24 GB | 3.8× |
| a.b.boneless 2026 (160952000000–160953999999) | 11.8 CPU-s (~205k/s), 0.70 GB | 3.1 CPU-s (~640k/s), 0.24 GB | 3.8× |
| a.b.ath 2013 (2549992990–2551992989) | 7.5 CPU-s (~320k/s), 0.74 GB | 2.15 CPU-s (~930k/s), 0.20 GB | 3.5× |
| a.b.models (2017215–4017214; image sets, re-uploads) | 16.7 CPU-s (~139k/s) | 5.0 CPU-s (~400k/s) | 3.3× |

**Comparing the numbers:**
- **CPU seconds are the fair comparison.** `edge:bench`'s headers/s leaves
  out the time Node's GC threads use; the Rust binary is single-threaded, so
  its wall time is its CPU time.
- **Homelab's cores** (Xeon E5-1650 v4) run the TypeScript ~10% slower than
  the workstation's.

**The Rust optimizations,** measured on boneless 2026 (all checked identical
after each step):

| Step | Headers/s |
|---|---|
| Straight port: regex crate for the patterns, chrono for dates, zlib-rs, std allocator | 360k |
| File stems and subject remainders by hand: their 12 + 5 regexes were most of a new file's description (333k per boneless range, many of them per-article random subjects) | |
| mimalloc for the many small strings | 495k |
| Upload tools, release-name checks and the common Date formats by hand | 500k |
| libdeflate: inflates a chunk whole in half of zlib's time (1.09 → 0.51 s per range) | |
| memchr (SIMD) for line ends and tabs | ~700k |
| Fewer allocations in `describe` (hashing the parts without joining them): no measurable gain, reverted | |

Rows without a figure weren't measured on their own; each figure covers the
rows above it.

**What's left** is spread out: scanning and building segments, two sha1s per
new file, aggregation, cut-off and chunk encoding. `perf` isn't available on
the workstation (`perf_event_paranoid` = 3), so these come from coarse
timers. The binary prints them as "of which aggregate …, evict+cutoff …,
judge+encode …" on stderr.

**What it means for the full target** (~1.25M articles/s with 50 NNTP
connections):
- **Worker cores:** ~1.5–2 instead of ~6–7. Spreading workers over other
  machines (batcave, poweredge-kantoor) becomes mostly unnecessary.
- **Total speed:** unchanged. The account's NNTP connections set it (~30k
  articles/s each), not the worker.
- **The cost:** a second implementation of `lib/headers`, `lib/release`,
  `lib/rules`, `lib/files` (aggregation, volumes) and the chunk format.
  Every rule change has to land in both and be checked identical (below)
  for as long as both exist.

## Running it

You need a captured range. `edge:bench --capture` saves a range's compressed
XOVER chunks once (it needs `NNTP_*` in the environment and one NNTP
connection):

```sh
node ace edge:bench --capture --dir /path/to/capture --group alt.binaries.boneless --start 160950000000
# 2M articles in chunks of 50k (--articles, --chunk), plus range.json
```

Benchmark both on it:

```sh
node ace edge:bench --dir /path/to/capture        # TypeScript: headers/s, CPU, inflate/index, peak RSS
cd edge && cargo build --release
./target/release/edge-bench /path/to/capture   # Rust: the same summary
```

Both take results every 10 chunks (`--batch-chunks`) and give a file up after
5 idle chunks (`--idle`). Keep these equal on both sides when comparing.
The rules are `DEFAULT_RULES` with only the range's group indexed and no
dropped signatures, as in `edge:bench`.

The two "kept: …" summary lines should already agree: release chunks,
files, segments, payload size, candidates, fragments and samples. The
summary can hide compensating differences, so compare the outputs
themselves.

## Comparing the output

```sh
./target/release/edge-bench /path/to/capture --dump /tmp/out.jsonl
cd .. && node --import=@poppinss/ts-exec edge/check.ts /path/to/capture /tmp/out.jsonl
```

`--dump` writes every output of every batch as JSON lines:

| Line `t` | Contents |
|---|---|
| `b` | batch boundaries: from, next, headers |
| `r` | releases: every ReleaseChunk field, the payload in base64 |
| `s` | samples: dropped releases kept in full, with their rule |
| `c` | probe candidates |
| `n` | rule counts |
| `f` | fragments |

`check.ts` runs the TypeScript `RangeIndexer` (from `lib/edge`, not a copy)
on the same capture with the same batching. It matches items by batch and
key: release key, candidate (release, kind, message-id), rule and key,
fragment file key. Then it compares them field by field:

```
same batches: 4 TypeScript, 4 Rust
same releases: 1055 TypeScript, 1055 Rust
same samples: 2 TypeScript, 2 Rust
same candidates: 1090 TypeScript, 1090 Rust
same rule counts: 34 TypeScript, 34 Rust
same fragments: 130 TypeScript, 130 Rust
payload bytes: 13728351 TypeScript, 13727247 Rust
```

It exits 0 only when everything matches. Otherwise each kind says how many
items are only on one side or differ, and shows the first differing fields.

- **Payloads are compared decoded.** Both sides go through `decodeChunk`
  from `lib/chunks`, and their files are compared, ordered by name and
  subject. Node's built-in zstd and the zstd crate frame the same data a
  little differently, so the compressed sizes differ by about 0.1%. The
  checker prints them, but they don't count as a difference. Decoding the
  Rust payloads with the TypeScript decoder also checks the Rust encoder.
- **Order doesn't count.** Order within a batch isn't compared (the two
  hash maps iterate differently). Grouping into batches, the order of a
  release's files and the order of a fragment's segments are compared,
  because they're part of the values.

Run it after any change on either side. All four captures above are
identical as of cfe85ab. The captures are kept on homelab in
`/mnt/bulk/usenet-rebuild/bench/` (boneless-2026, boneless-2026b,
ath-2013, models-a). models-a is a bottom-500 group (rank 501–1000 by
span) with what the other three lack: image sets (38,630 release chunks
for 412k files), documents, and re-uploads (521 chunks of a second or
later upload of a key in a batch). Its chunk 26 is the plain fetch of a
range whose compressed reply Eweka cuts short (LOG.md 2026-10-09 (11)).

## JavaScript semantics the port keeps (`src/js.rs`, `src/names.rs`)

Differences found with `check.ts` or the tests, kept as JavaScript has
them:

- **Regexes without the `u` flag:** `\d` is [0-9], `\b` an ASCII word
  boundary, `/i` ASCII-only. They run as `(?-u)` byte regexes, or by hand.
  The hand-written versions (file stems, subject remainders, upload tools,
  release-name and file-counter checks) are tested against the regex
  versions, which are kept for that (`*_re`).
- **JS `\s` and `trim()`** include Unicode spaces (NBSP, U+2000–200A,
  U+3000, BOM…); `split(/\s+/)` keeps leading and trailing empty words.
- **`lastIndexOf(c, position)`** finds a match at or before the position
  (the owner rule, the counter and filename scans). **`.length`** is in
  UTF-16 units (`len16`). **`parseInt`** reads leading digits.
- **`Date.parse`:** chrono follows RFC 5322 and rejects "UTC", which
  `Date.parse` takes. It ends about a quarter of boneless's Date headers, so
  it's mapped to GMT; check.ts found this. The common formats are parsed by
  hand:

  ```
  [Day, ]D[D] Mon YY[YY] HH:MM:SS (GMT|UTC|UT|±hhmm)
  ```

  They ignore the weekday as `Date.parse` does (chrono rejects a weekday
  that doesn't fit). Anything else goes to chrono. Rare forms `Date.parse`
  takes but chrono doesn't would show up in check.ts as a `postedAt`
  difference.
- **Invalid UTF-8** becomes U+FFFD on both sides (`toString('utf8')`,
  `from_utf8_lossy`).

## Files

| File | Contents |
|---|---|
| `src/main.rs` | arguments, the replay loop, timing, `--dump` |
| `src/indexer.rs` | scan.ts (SegmentScanner, scanOverviewText), aggregator.ts, range_indexer.ts |
| `src/names.rs` | lib/release (identity, random), lib/files/volumes, lib/rules (upload tools, judge) |
| `src/codec.rs` | lib/chunks encodeChunk, format v2 |
| `src/js.rs` | JavaScript semantics, sha1 |
| `check.ts` | the comparison |

## The worker

```sh
cargo build --release
COORDINATOR_URL=http://coordinator:3334 NODE_TOKEN=… ./target/release/edge-node [--threads N] [--batch-articles 500000] [--ranges N] [--max-mbit M]
```

- **One range per thread** (default one per core): one thread reads the
  coordinator's stream and inflates a frame ahead, another posts the
  batches in order (one may wait), the indexer goes on meanwhile.
- **As `commands/edge_node.ts`:** retries for network trouble and 5xx;
  409 means the lease is lost (the range stops, isn't given back); a
  batch once 500k articles or 100k files wait (a picture group's batches
  went over the server's 256 MB limit); a chunk that doesn't inflate is
  reported (the coordinator fetches it plain next time); SIGTERM/SIGINT
  give each range back after its current chunk; `--max-mbit` is the
  node's download cap, shared by its threads. Plus a heartbeat when no
  batch went out for a minute.
- **Wire format:** `src/wire.rs` writes a batch as `toWire` does.
  `edge-bench --wire` and `check.ts --wire` compare it with the
  TypeScript's, field by field.
- **End to end:** both workers indexed the a.b.models capture (spooled
  as a range) through the web server; what the server stored was the
  same (releases with every column, chunks, candidates, rule counts,
  fragments with their JSON).
- **On homelab:** compose service `edge` (deploy/compose.yaml), the
  `edge` image target of the Dockerfile, `--threads $EDGE_THREADS` (8);
  it talks to `node-api`.