usenet-indexer/edge/README.md

# edge: voyager 2's edge worker (Rust)

The product: `edge-node` (`src/bin/edge-node.rs`) is the worker that runs
on homelab (compose service `edge`, image target `edge`). It leases ranges
from the node API, reads their prefetched XOVER chunks, finds releases and
posts batches. `edge-bench` (`src/main.rs`) replays a captured range through
the same indexing (`src/lib.rs`) for timing and for comparing.

Since voyager 3 (2026-10-25) the Rust indexer is the one implementation of
the window (REBUILD.md, "Voyager 3: window indexing"; `src/indexer.rs`):
files and releases are finished when the frontier (the median Date of the
last 1,024 plausibly dated articles) is 24 h past their last part, or at
the block's end (provisionally, as edges), and written once. The
TypeScript `RangeIndexer` (`lib/edge`) keeps the range path for the corpus
scanner and the tests; the field-by-field comparison (`check.ts`) went
with the parity. CI (`.gitea/workflows/deploy.yaml`, job `edge`) runs
`cargo test` and `bench-fixtures.sh`, which replays the committed captures
in `fixtures/` (25k overview lines each of a.b.ath, a.b.boneless, a.b.fta
and a.b.models, in 5 chunks).

```sh
cargo test --release --manifest-path edge/Cargo.toml
edge/bench-fixtures.sh                       # from the repository root
```

In production since 2026-10-09: 122M headers in 10 minutes with 0.37 cores
and 565 MB, next to the TypeScript workers' 24M with 0.61 cores and 11.3 GB
(8 threads vs 8 processes): about 8× less CPU per header and 20× less
memory. See "The worker" below.

It started as a measurement (moved from `tools/edge-bench-rs` on
2026-10-11): whether a Rust worker would be worth a second implementation.
The findings below are from then.

## Findings (2026-10-09)

Same machine (the workstation, otherwise idle), same captures, 2M headers
each:

| Capture | TypeScript (`edge:bench`) | Rust (`edge-bench`) | Less CPU |
|---|---|---|---|
| a.b.boneless 2026 (160950000000–160951999999) | 11.1 CPU-s (~215k headers/s), 0.70 GB peak | 2.9 CPU-s (~690k/s), 0.24 GB | 3.8× |
| a.b.boneless 2026 (160952000000–160953999999) | 11.8 CPU-s (~205k/s), 0.70 GB | 3.1 CPU-s (~640k/s), 0.24 GB | 3.8× |
| a.b.ath 2013 (2549992990–2551992989) | 7.5 CPU-s (~320k/s), 0.74 GB | 2.15 CPU-s (~930k/s), 0.20 GB | 3.5× |
| a.b.models (2017215–4017214; image sets, re-uploads) | 16.7 CPU-s (~139k/s) | 5.0 CPU-s (~400k/s) | 3.3× |

**Comparing the numbers:**
- **CPU seconds are the fair comparison.** `edge:bench`'s headers/s leaves
  out the time Node's GC threads use; the Rust binary is single-threaded, so
  its wall time is its CPU time.
- **Homelab's cores** (Xeon E5-1650 v4) run the TypeScript ~10% slower than
  the workstation's.

**The Rust optimizations,** measured on boneless 2026 (all checked identical
after each step):

| Step | Headers/s |
|---|---|
| Straight port: regex crate for the patterns, chrono for dates, zlib-rs, std allocator | 360k |
| File stems and subject remainders by hand: their 12 + 5 regexes were most of a new file's description (333k per boneless range, many of them per-article random subjects) | |
| mimalloc for the many small strings | 495k |
| Upload tools, release-name checks and the common Date formats by hand | 500k |
| libdeflate: inflates a chunk whole in half of zlib's time (1.09 → 0.51 s per range) | |
| memchr (SIMD) for line ends and tabs | ~700k |
| Fewer allocations in `describe` (hashing the parts without joining them): no measurable gain, reverted | |

Rows without a figure weren't measured on their own; each figure covers the
rows above it.

**What's left** is spread out: scanning and building segments, two sha1s per
new file, aggregation, cut-off and chunk encoding. `perf` isn't available on
the workstation (`perf_event_paranoid` = 3), so these come from coarse
timers. The binary prints them as "of which aggregate …, evict+cutoff …,
judge+encode …" on stderr.

**What it means for the full target** (~1.25M articles/s with 50 NNTP
connections):
- **Worker cores:** ~1.5–2 instead of ~6–7. Spreading workers over other
  machines (batcave, poweredge-kantoor) becomes mostly unnecessary.
- **Total speed:** unchanged. The account's NNTP connections set it (~30k
  articles/s each), not the worker.
- **The cost:** a second implementation of `lib/headers`, `lib/release`,
  `lib/rules`, `lib/files` (aggregation, volumes) and the chunk format.
  Every rule change has to land in both and be checked identical (below)
  for as long as both exist.

## Running it

You need a captured range. `edge:bench --capture` saves a range's compressed
XOVER chunks once (it needs `NNTP_*` in the environment and one NNTP
connection):

```sh
node ace edge:bench --capture --dir /path/to/capture --group alt.binaries.boneless --start 160950000000
# 2M articles in chunks of 50k (--articles, --chunk), plus range.json
```

Benchmark both on it:

```sh
node ace edge:bench --dir /path/to/capture        # TypeScript: headers/s, CPU, inflate/index, peak RSS
cd edge && cargo build --release
./target/release/edge-bench /path/to/capture   # Rust: the same summary
```

Both take results every 10 chunks (`--batch-chunks`) and give a file up after
5 idle chunks (`--idle`). Keep these equal on both sides when comparing.
The rules are `DEFAULT_RULES` with only the range's group indexed and no
dropped signatures, as in `edge:bench`.

The two "kept: …" summary lines should already agree: release chunks,
files, segments, payload size, candidates, fragments and samples. The
summary can hide compensating differences, so compare the outputs
themselves.

## Replaying a group through one window

A corpus group directory (`<corpus>/<group>/<start>-<end>/`, each range with
its `range.json`) is replayed in article order with the window carried from
range to range, the end finished after the last range (REBUILD.md, "The
replay harness"):

```sh
./target/release/edge-bench --group /corpus/alt.binaries.ath --rules rules.json --batch-chunks 5 --wire - > batches.jsonl
# --from <article> skips ranges ending before it, --to <article> ranges starting after it (one window of the
# group's history), --limit <n> stops after n ranges, --no-finish leaves the window open
```

`--rules` takes the lease's rules JSON (`src/lease.rs`: the catalog's rule
set and group priorities, as `replay:corpus` writes it); without it the
defaults with only the group indexed apply. `--wire -` writes each batch as
the worker posts it to stdout, one JSON per line, and the report to stderr;
`node ace replay:corpus` reads them into the batch store. Each range end
prints the frontier, the releases out and the releases and files still open.

## The dump

```sh
./target/release/edge-bench /path/to/capture --dump /tmp/out.jsonl
```

`--dump` writes every output of every batch as JSON lines:

| Line `t` | Contents |
|---|---|
| `b` | batch boundaries: from, next, headers |
| `r` | releases: every ReleaseChunk field, the payload in base64 |
| `s` | samples: dropped releases kept in full, with their rule |
| `c` | probe candidates |
| `n` | rule counts |
| `f` | fragments |

`check.ts` runs the TypeScript `RangeIndexer` (from `lib/edge`, not a copy)
on the same capture with the same batching. It matches items by batch and
key: release key, candidate (release, kind, message-id), rule and key,
fragment file key. Then it compares them field by field:

```
same batches: 4 TypeScript, 4 Rust
same releases: 1055 TypeScript, 1055 Rust
same samples: 2 TypeScript, 2 Rust
same candidates: 1090 TypeScript, 1090 Rust
same rule counts: 34 TypeScript, 34 Rust
same fragments: 130 TypeScript, 130 Rust
payload bytes: 13728351 TypeScript, 13727247 Rust
```

It exits 0 only when everything matches. Otherwise each kind says how many
items are only on one side or differ, and shows the first differing fields.

- **Payloads are compared decoded.** Both sides go through `decodeChunk`
  from `lib/chunks`, and their files are compared, ordered by name and
  subject. Node's built-in zstd and the zstd crate frame the same data a
  little differently, so the compressed sizes differ by about 0.1%. The
  checker prints them, but they don't count as a difference. Decoding the
  Rust payloads with the TypeScript decoder also checks the Rust encoder.
- **Order doesn't count.** Order within a batch isn't compared (the two
  hash maps iterate differently). Grouping into batches, the order of a
  release's files and the order of a fragment's segments are compared,
  because they're part of the values.

Run it after any change on either side. All four captures above are
identical as of cfe85ab. The captures are kept on homelab in
`/mnt/bulk/usenet-rebuild/bench/` (boneless-2026, boneless-2026b,
ath-2013, models-a). models-a is a bottom-500 group (rank 501–1000 by
span) with what the other three lack: image sets (38,630 release chunks
for 412k files), documents, and re-uploads (521 chunks of a second or
later upload of a key in a batch). Its chunk 26 is the plain fetch of a
range whose compressed reply Eweka cuts short (LOG.md 2026-10-09 (11)).

## JavaScript semantics the port keeps (`src/js.rs`, `src/names.rs`)

Differences found with `check.ts` or the tests, kept as JavaScript has
them:

- **Regexes without the `u` flag:** `\d` is [0-9], `\b` an ASCII word
  boundary, `/i` ASCII-only. They run as `(?-u)` byte regexes, or by hand.
  The hand-written versions (file stems, subject remainders, upload tools,
  release-name and file-counter checks) are tested against the regex
  versions, which are kept for that (`*_re`).
- **JS `\s` and `trim()`** include Unicode spaces (NBSP, U+2000–200A,
  U+3000, BOM…); `split(/\s+/)` keeps leading and trailing empty words.
- **`lastIndexOf(c, position)`** finds a match at or before the position
  (the owner rule, the counter and filename scans). **`.length`** is in
  UTF-16 units (`len16`). **`parseInt`** reads leading digits.
- **`Date.parse`:** chrono follows RFC 5322 and rejects "UTC", which
  `Date.parse` takes. It ends about a quarter of boneless's Date headers, so
  it's mapped to GMT; check.ts found this. The common formats are parsed by
  hand:

  ```
  [Day, ]D[D] Mon YY[YY] HH:MM:SS (GMT|UTC|UT|±hhmm)
  ```

  They ignore the weekday as `Date.parse` does (chrono rejects a weekday
  that doesn't fit). Anything else goes to chrono. Rare forms `Date.parse`
  takes but chrono doesn't would show up in check.ts as a `postedAt`
  difference.
- **Invalid UTF-8** becomes U+FFFD on both sides (`toString('utf8')`,
  `from_utf8_lossy`).

## Files

| File | Contents |
|---|---|
| `src/main.rs` | arguments, the replay loop, timing, `--dump` |
| `src/indexer.rs` | scan.ts (SegmentScanner, scanOverviewText), aggregator.ts, range_indexer.ts |
| `src/names.rs` | lib/release (identity, random), lib/files/volumes, lib/rules (upload tools, judge) |
| `src/codec.rs` | lib/chunks encodeChunk, format v2 |
| `src/js.rs` | JavaScript semantics, sha1 |
| `check.ts` | the comparison |

## The worker

```sh
cargo build --release
COORDINATOR_URL=http://coordinator:3334 NODE_TOKEN=… ./target/release/edge-node [--threads N] [--tail-threads T] [--batch-articles 500000] [--snapshot-minutes 30] [--groups N] [--max-mbit M]
```

- **Two lanes** (2026-10-27): `--tail-threads T` (EDGE_TAIL_THREADS; one of
  two or more threads by default) puts that many threads on the groups'
  *tail* lane (`?lane=tail` on the group routes): the tail planner's
  ranges, through a window and snapshot of their own
  (`<group>-tail.gz`). The other threads work the history lane: the
  windows and the history the coordinator's bounds hand out. A tail thread
  never wades through a group's history first, and a held history group's
  tail isn't stuck behind its days of ranges; the coordinator orders each
  lane's leases by alternating the group with the most planned articles
  and the longest unheld one.

- **One group per thread** (default one per core; REBUILD.md, "Voyager 3:
  window indexing", 1 and 5): a thread leases a group (`POST
  groups/lease`), takes its ranges in article order (`POST
  groups/:id/next`), streams each range's chunks (`GET ranges/:id/stream`:
  the corpus or the spool, the worker can't tell) through one carried
  window, posts a batch at least at every range end and finishes the
  range; one thread reads the stream and inflates a frame ahead, another
  posts the batches in order, the indexer goes on meanwhile. The window's
  snapshot (bincode, gzip) goes to the coordinator every
  `--snapshot-minutes` (EDGE_SNAPSHOT_MINUTES, 30) right after a batch,
  and when the group is given back: caught up (`next` answers 204: the
  tail), a stop, a lost lease or a failure; a stop or a failed stream
  posts the open batch first, so the snapshot stands at its resume point
  (a batch the coordinator refused can't, and that window is lost); the
  next holder restores it (`GET groups/:id/snapshot`) and continues at
  its article. `--groups N`
  stops after N groups; `--ranges N` is the old per-range lease
  (`POST lease`), one range per thread without a carried window.
- **As `commands/edge_node.ts`:** retries for network trouble and 5xx;
  409 means the lease is lost (the range stops, isn't given back); a
  batch once 500k articles or 100k files wait (a picture group's batches
  went over the server's 256 MB limit); a chunk that doesn't inflate is
  reported (the coordinator fetches it plain next time) after the batch
  before it went out, and a held group asks for the range again from
  that chunk, its window kept; SIGTERM/SIGINT
  give each range back after its current chunk; `--max-mbit` is the
  node's download cap, shared by its threads. Plus a heartbeat when no
  batch went out for a minute.
- **Wire format:** `src/wire.rs` writes a batch as `toWire` does.
  `edge-bench --wire` and `check.ts --wire` compare it with the
  TypeScript's, field by field.
- **End to end:** both workers indexed the a.b.models capture (spooled
  as a range) through the web server; what the server stored was the
  same (releases with every column, chunks, candidates, rule counts,
  fragments with their JSON).
- **On homelab:** compose service `edge` (deploy/compose.yaml), the
  `edge` image target of the Dockerfile, `--threads $EDGE_THREADS` (8),
  EDGE_TAIL_THREADS (1); it talks to `node-api`.
- **End to end:** `tests/e2e/edge_node.spec.ts` runs this binary
  (`edge/target/release/edge-node`, or EDGE_NODE_BIN) against the test
  server, database, spool and corpus: a chunk the corpus fill stores while
  the stream waits, a failed stream and the restart that restores the
  snapshot, a restart whose snapshot file is gone, a takeover. `node ace
  test e2e` after `cargo build --release`; CI builds the binary first.