usenet-indexer/edge

edge: voyager 2's edge worker (Rust)

The product: edge-node (src/bin/edge-node.rs) is the worker that runs on homelab (compose service edge, image target edge). It leases ranges from the node API, reads their prefetched XOVER chunks, finds releases and posts batches. edge-bench (src/main.rs) replays a captured range through the same indexing (src/lib.rs) for timing and for comparing.

The TypeScript RangeIndexer (lib/edge, node ace edge:node) is the reference: check.ts runs both on a capture and compares every output field by field. An identity or rule change goes into both and has to come out identical. CI (.gitea/workflows/deploy.yaml, job edge) runs cargo test and check-fixtures.sh, which compares both on the committed captures in fixtures/ (25k overview lines each of a.b.models, a.b.ath and a.b.boneless, in 5 chunks); a push to rebuild deploys only when they match.

cargo test --release --manifest-path edge/Cargo.toml
edge/check-fixtures.sh                       # from the repository root

In production since 2026-10-09: 122M headers in 10 minutes with 0.37 cores and 565 MB, next to the TypeScript workers' 24M with 0.61 cores and 11.3 GB (8 threads vs 8 processes): about 8× less CPU per header and 20× less memory. See "The worker" below.

It started as a measurement (moved from tools/edge-bench-rs on 2026-10-11): whether a Rust worker would be worth a second implementation. The findings below are from then.

Findings (2026-10-09)

Same machine (the workstation, otherwise idle), same captures, 2M headers each:

Capture	TypeScript (edge:bench)	Rust (edge-bench)	Less CPU
a.b.boneless 2026 (160950000000–160951999999)	11.1 CPU-s (~215k headers/s), 0.70 GB peak	2.9 CPU-s (~690k/s), 0.24 GB	3.8×
a.b.boneless 2026 (160952000000–160953999999)	11.8 CPU-s (~205k/s), 0.70 GB	3.1 CPU-s (~640k/s), 0.24 GB	3.8×
a.b.ath 2013 (2549992990–2551992989)	7.5 CPU-s (~320k/s), 0.74 GB	2.15 CPU-s (~930k/s), 0.20 GB	3.5×
a.b.models (2017215–4017214; image sets, re-uploads)	16.7 CPU-s (~139k/s)	5.0 CPU-s (~400k/s)	3.3×

Comparing the numbers:

• CPU seconds are the fair comparison. edge:bench's headers/s leaves out the time Node's GC threads use; the Rust binary is single-threaded, so its wall time is its CPU time.
• Homelab's cores (Xeon E5-1650 v4) run the TypeScript ~10% slower than the workstation's.

The Rust optimizations, measured on boneless 2026 (all checked identical after each step):

Step	Headers/s
Straight port: regex crate for the patterns, chrono for dates, zlib-rs, std allocator	360k
File stems and subject remainders by hand: their 12 + 5 regexes were most of a new file's description (333k per boneless range, many of them per-article random subjects)
mimalloc for the many small strings	495k
Upload tools, release-name checks and the common Date formats by hand	500k
libdeflate: inflates a chunk whole in half of zlib's time (1.09 → 0.51 s per range)
memchr (SIMD) for line ends and tabs	~700k
Fewer allocations in describe (hashing the parts without joining them): no measurable gain, reverted

Rows without a figure weren't measured on their own; each figure covers the rows above it.

What's left is spread out: scanning and building segments, two sha1s per new file, aggregation, cut-off and chunk encoding. perf isn't available on the workstation (perf_event_paranoid = 3), so these come from coarse timers. The binary prints them as "of which aggregate …, evict+cutoff …, judge+encode …" on stderr.

What it means for the full target (~1.25M articles/s with 50 NNTP connections):

• Worker cores: ~1.5–2 instead of ~6–7. Spreading workers over other machines (batcave, poweredge-kantoor) becomes mostly unnecessary.
• Total speed: unchanged. The account's NNTP connections set it (~30k articles/s each), not the worker.
• The cost: a second implementation of lib/headers, lib/release, lib/rules, lib/files (aggregation, volumes) and the chunk format. Every rule change has to land in both and be checked identical (below) for as long as both exist.

Running it

You need a captured range. edge:bench --capture saves a range's compressed XOVER chunks once (it needs NNTP_* in the environment and one NNTP connection):

node ace edge:bench --capture --dir /path/to/capture --group alt.binaries.boneless --start 160950000000
# 2M articles in chunks of 50k (--articles, --chunk), plus range.json

Benchmark both on it:

node ace edge:bench --dir /path/to/capture        # TypeScript: headers/s, CPU, inflate/index, peak RSS
cd edge && cargo build --release
./target/release/edge-bench /path/to/capture   # Rust: the same summary

Both take results every 10 chunks (--batch-chunks) and give a file up after 5 idle chunks (--idle). Keep these equal on both sides when comparing. The rules are DEFAULT_RULES with only the range's group indexed and no dropped signatures, as in edge:bench.

The two "kept: …" summary lines should already agree: release chunks, files, segments, payload size, candidates, fragments and samples. The summary can hide compensating differences, so compare the outputs themselves.

Comparing the output

./target/release/edge-bench /path/to/capture --dump /tmp/out.jsonl
cd .. && node --import=@poppinss/ts-exec edge/check.ts /path/to/capture /tmp/out.jsonl

--dump writes every output of every batch as JSON lines:

Line t	Contents
b	batch boundaries: from, next, headers
r	releases: every ReleaseChunk field, the payload in base64
s	samples: dropped releases kept in full, with their rule
c	probe candidates
n	rule counts
f	fragments

check.ts runs the TypeScript RangeIndexer (from lib/edge, not a copy) on the same capture with the same batching. It matches items by batch and key: release key, candidate (release, kind, message-id), rule and key, fragment file key. Then it compares them field by field:

same batches: 4 TypeScript, 4 Rust
same releases: 1055 TypeScript, 1055 Rust
same samples: 2 TypeScript, 2 Rust
same candidates: 1090 TypeScript, 1090 Rust
same rule counts: 34 TypeScript, 34 Rust
same fragments: 130 TypeScript, 130 Rust
payload bytes: 13728351 TypeScript, 13727247 Rust

It exits 0 only when everything matches. Otherwise each kind says how many items are only on one side or differ, and shows the first differing fields.

• Payloads are compared decoded. Both sides go through decodeChunk from lib/chunks, and their files are compared, ordered by name and subject. Node's built-in zstd and the zstd crate frame the same data a little differently, so the compressed sizes differ by about 0.1%. The checker prints them, but they don't count as a difference. Decoding the Rust payloads with the TypeScript decoder also checks the Rust encoder.
• Order doesn't count. Order within a batch isn't compared (the two hash maps iterate differently). Grouping into batches, the order of a release's files and the order of a fragment's segments are compared, because they're part of the values.

Run it after any change on either side. All four captures above are identical as of cfe85ab. The captures are kept on homelab in /mnt/bulk/usenet-rebuild/bench/ (boneless-2026, boneless-2026b, ath-2013, models-a). models-a is a bottom-500 group (rank 501–1000 by span) with what the other three lack: image sets (38,630 release chunks for 412k files), documents, and re-uploads (521 chunks of a second or later upload of a key in a batch). Its chunk 26 is the plain fetch of a range whose compressed reply Eweka cuts short (LOG.md 2026-10-09 (11)).

JavaScript semantics the port keeps (src/js.rs, src/names.rs)

Differences found with check.ts or the tests, kept as JavaScript has them:

• Regexes without the u flag: \d is [0-9], \b an ASCII word boundary, /i ASCII-only. They run as (?-u) byte regexes, or by hand. The hand-written versions (file stems, subject remainders, upload tools, release-name and file-counter checks) are tested against the regex versions, which are kept for that (*_re).

• JS \s and trim() include Unicode spaces (NBSP, U+2000–200A, U+3000, BOM…); split(/\s+/) keeps leading and trailing empty words.

• lastIndexOf(c, position) finds a match at or before the position (the owner rule, the counter and filename scans). .length is in UTF-16 units (len16). parseInt reads leading digits.

• Date.parse: chrono follows RFC 5322 and rejects "UTC", which Date.parse takes. It ends about a quarter of boneless's Date headers, so it's mapped to GMT; check.ts found this. The common formats are parsed by hand:

  [Day, ]D[D] Mon YY[YY] HH:MM:SS (GMT|UTC|UT|±hhmm)
  

  They ignore the weekday as Date.parse does (chrono rejects a weekday that doesn't fit). Anything else goes to chrono. Rare forms Date.parse takes but chrono doesn't would show up in check.ts as a postedAt difference.

• Invalid UTF-8 becomes U+FFFD on both sides (toString('utf8'), from_utf8_lossy).

Files

File	Contents
src/main.rs	arguments, the replay loop, timing, --dump
src/indexer.rs	scan.ts (SegmentScanner, scanOverviewText), aggregator.ts, range_indexer.ts
src/names.rs	lib/release (identity, random), lib/files/volumes, lib/rules (upload tools, judge)
src/codec.rs	lib/chunks encodeChunk, format v2
src/js.rs	JavaScript semantics, sha1
check.ts	the comparison

The worker

cargo build --release
COORDINATOR_URL=http://coordinator:3334 NODE_TOKEN=… ./target/release/edge-node [--threads N] [--batch-articles 500000] [--ranges N] [--max-mbit M]

• One range per thread (default one per core): one thread reads the coordinator's stream and inflates a frame ahead, another posts the batches in order (one may wait), the indexer goes on meanwhile.
• As commands/edge_node.ts: retries for network trouble and 5xx; 409 means the lease is lost (the range stops, isn't given back); a batch once 500k articles or 100k files wait (a picture group's batches went over the server's 256 MB limit); a chunk that doesn't inflate is reported (the coordinator fetches it plain next time); SIGTERM/SIGINT give each range back after its current chunk; --max-mbit is the node's download cap, shared by its threads. Plus a heartbeat when no batch went out for a minute.
• Wire format: src/wire.rs writes a batch as toWire does. edge-bench --wire and check.ts --wire compare it with the TypeScript's, field by field.
• End to end: both workers indexed the a.b.models capture (spooled as a range) through the web server; what the server stored was the same (releases with every column, chunks, candidates, rule counts, fragments with their JSON).
• On homelab: compose service edge (deploy/compose.yaml), the edge image target of the Dockerfile, --threads $EDGE_THREADS (8); it talks to node-api.