usenet-indexer/edge

edge: voyager 2's edge worker (Rust)

The product: edge-node (src/bin/edge-node.rs) is the worker that runs on homelab (compose service edge, image target edge). It leases ranges from the node API, reads their prefetched XOVER chunks, finds releases and posts batches. edge-bench (src/main.rs) replays a captured range through the same indexing (src/lib.rs) for timing and for comparing.

Since voyager 3 (2026-10-25) the Rust indexer is the one implementation of the window (REBUILD.md, "Voyager 3: window indexing"; src/indexer.rs): files and releases are finished when the frontier (the median Date of the last 1,024 plausibly dated articles) is 24 h past their last part, or at the block's end (provisionally, as edges), and written once. The TypeScript RangeIndexer (lib/edge) keeps the range path for the corpus scanner and the tests; the field-by-field comparison (check.ts) went with the parity. CI (.gitea/workflows/deploy.yaml, job edge) runs cargo test and bench-fixtures.sh, which replays the committed captures in fixtures/ (25k overview lines each of a.b.ath, a.b.boneless, a.b.fta and a.b.models, in 5 chunks).

cargo test --release --manifest-path edge/Cargo.toml
edge/bench-fixtures.sh                       # from the repository root

In production since 2026-10-09: 122M headers in 10 minutes with 0.37 cores and 565 MB, next to the TypeScript workers' 24M with 0.61 cores and 11.3 GB (8 threads vs 8 processes): about 8× less CPU per header and 20× less memory. See "The worker" below.

It started as a measurement (moved from tools/edge-bench-rs on 2026-10-11): whether a Rust worker would be worth a second implementation. The findings below are from then.

Findings (2026-10-09)

Same machine (the workstation, otherwise idle), same captures, 2M headers each:

Capture	TypeScript (edge:bench)	Rust (edge-bench)	Less CPU
a.b.boneless 2026 (160950000000–160951999999)	11.1 CPU-s (~215k headers/s), 0.70 GB peak	2.9 CPU-s (~690k/s), 0.24 GB	3.8×
a.b.boneless 2026 (160952000000–160953999999)	11.8 CPU-s (~205k/s), 0.70 GB	3.1 CPU-s (~640k/s), 0.24 GB	3.8×
a.b.ath 2013 (2549992990–2551992989)	7.5 CPU-s (~320k/s), 0.74 GB	2.15 CPU-s (~930k/s), 0.20 GB	3.5×
a.b.models (2017215–4017214; image sets, re-uploads)	16.7 CPU-s (~139k/s)	5.0 CPU-s (~400k/s)	3.3×

Comparing the numbers:

• CPU seconds are the fair comparison. edge:bench's headers/s leaves out the time Node's GC threads use; the Rust binary is single-threaded, so its wall time is its CPU time.
• Homelab's cores (Xeon E5-1650 v4) run the TypeScript ~10% slower than the workstation's.

The Rust optimizations, measured on boneless 2026 (all checked identical after each step):

Step	Headers/s
Straight port: regex crate for the patterns, chrono for dates, zlib-rs, std allocator	360k
File stems and subject remainders by hand: their 12 + 5 regexes were most of a new file's description (333k per boneless range, many of them per-article random subjects)
mimalloc for the many small strings	495k
Upload tools, release-name checks and the common Date formats by hand	500k
libdeflate: inflates a chunk whole in half of zlib's time (1.09 → 0.51 s per range)
memchr (SIMD) for line ends and tabs	~700k
Fewer allocations in describe (hashing the parts without joining them): no measurable gain, reverted

Rows without a figure weren't measured on their own; each figure covers the rows above it.

What's left is spread out: scanning and building segments, two sha1s per new file, aggregation, cut-off and chunk encoding. perf isn't available on the workstation (perf_event_paranoid = 3), so these come from coarse timers. The binary prints them as "of which aggregate …, evict+cutoff …, judge+encode …" on stderr.

What it means for the full target (~1.25M articles/s with 50 NNTP connections):

• Worker cores: ~1.5–2 instead of ~6–7. Spreading workers over other machines (batcave, poweredge-kantoor) becomes mostly unnecessary.
• Total speed: unchanged. The account's NNTP connections set it (~30k articles/s each), not the worker.
• The cost: a second implementation of lib/headers, lib/release, lib/rules, lib/files (aggregation, volumes) and the chunk format. Every rule change has to land in both and be checked identical (below) for as long as both exist.

Running it

You need a captured range. edge:bench --capture saves a range's compressed XOVER chunks once (it needs NNTP_* in the environment and one NNTP connection):

node ace edge:bench --capture --dir /path/to/capture --group alt.binaries.boneless --start 160950000000
# 2M articles in chunks of 50k (--articles, --chunk), plus range.json

Benchmark both on it:

node ace edge:bench --dir /path/to/capture        # TypeScript: headers/s, CPU, inflate/index, peak RSS
cd edge && cargo build --release
./target/release/edge-bench /path/to/capture   # Rust: the same summary

Both take results every 10 chunks (--batch-chunks) and give a file up after 5 idle chunks (--idle). Keep these equal on both sides when comparing. The rules are DEFAULT_RULES with only the range's group indexed and no dropped signatures, as in edge:bench.

The two "kept: …" summary lines should already agree: release chunks, files, segments, payload size, candidates, fragments and samples. The summary can hide compensating differences, so compare the outputs themselves.

Replaying a group through one window

A corpus group directory (<corpus>/<group>/<start>-<end>/, each range with its range.json) is replayed in article order with the window carried from range to range, the end finished after the last range (REBUILD.md, "The replay harness"):

./target/release/edge-bench --group /corpus/alt.binaries.ath --rules rules.json --batch-chunks 5 --wire - > batches.jsonl
# --from <article> skips ranges ending before it, --to <article> ranges starting after it (one window of the
# group's history), --limit <n> stops after n ranges, --no-finish leaves the window open

--rules takes the lease's rules JSON (src/lease.rs: the catalog's rule set and group priorities, as replay:corpus writes it); without it the defaults with only the group indexed apply. --wire - writes each batch as the worker posts it to stdout, one JSON per line, and the report to stderr; node ace replay:corpus reads them into the batch store. Each range end prints the frontier, the releases out and the releases and files still open.

The dump

./target/release/edge-bench /path/to/capture --dump /tmp/out.jsonl

--dump writes every output of every batch as JSON lines:

Line t	Contents
b	batch boundaries: from, next, headers
r	releases: every ReleaseChunk field, the payload in base64
s	samples: dropped releases kept in full, with their rule
c	probe candidates
n	rule counts
f	fragments

check.ts runs the TypeScript RangeIndexer (from lib/edge, not a copy) on the same capture with the same batching. It matches items by batch and key: release key, candidate (release, kind, message-id), rule and key, fragment file key. Then it compares them field by field:

same batches: 4 TypeScript, 4 Rust
same releases: 1055 TypeScript, 1055 Rust
same samples: 2 TypeScript, 2 Rust
same candidates: 1090 TypeScript, 1090 Rust
same rule counts: 34 TypeScript, 34 Rust
same fragments: 130 TypeScript, 130 Rust
payload bytes: 13728351 TypeScript, 13727247 Rust

It exits 0 only when everything matches. Otherwise each kind says how many items are only on one side or differ, and shows the first differing fields.

• Payloads are compared decoded. Both sides go through decodeChunk from lib/chunks, and their files are compared, ordered by name and subject. Node's built-in zstd and the zstd crate frame the same data a little differently, so the compressed sizes differ by about 0.1%. The checker prints them, but they don't count as a difference. Decoding the Rust payloads with the TypeScript decoder also checks the Rust encoder.
• Order doesn't count. Order within a batch isn't compared (the two hash maps iterate differently). Grouping into batches, the order of a release's files and the order of a fragment's segments are compared, because they're part of the values.

Run it after any change on either side. All four captures above are identical as of cfe85ab. The captures are kept on homelab in /mnt/bulk/usenet-rebuild/bench/ (boneless-2026, boneless-2026b, ath-2013, models-a). models-a is a bottom-500 group (rank 501–1000 by span) with what the other three lack: image sets (38,630 release chunks for 412k files), documents, and re-uploads (521 chunks of a second or later upload of a key in a batch). Its chunk 26 is the plain fetch of a range whose compressed reply Eweka cuts short (LOG.md 2026-10-09 (11)).

JavaScript semantics the port keeps (src/js.rs, src/names.rs)

Differences found with check.ts or the tests, kept as JavaScript has them:

• Regexes without the u flag: \d is [0-9], \b an ASCII word boundary, /i ASCII-only. They run as (?-u) byte regexes, or by hand. The hand-written versions (file stems, subject remainders, upload tools, release-name and file-counter checks) are tested against the regex versions, which are kept for that (*_re).

• JS \s and trim() include Unicode spaces (NBSP, U+2000–200A, U+3000, BOM…); split(/\s+/) keeps leading and trailing empty words.

• lastIndexOf(c, position) finds a match at or before the position (the owner rule, the counter and filename scans). .length is in UTF-16 units (len16). parseInt reads leading digits.

• Date.parse: chrono follows RFC 5322 and rejects "UTC", which Date.parse takes. It ends about a quarter of boneless's Date headers, so it's mapped to GMT; check.ts found this. The common formats are parsed by hand:

  [Day, ]D[D] Mon YY[YY] HH:MM:SS (GMT|UTC|UT|±hhmm)
  

  They ignore the weekday as Date.parse does (chrono rejects a weekday that doesn't fit). Anything else goes to chrono. Rare forms Date.parse takes but chrono doesn't would show up in check.ts as a postedAt difference.

• Invalid UTF-8 becomes U+FFFD on both sides (toString('utf8'), from_utf8_lossy).

Files

File	Contents
src/main.rs	arguments, the replay loop, timing, --dump
src/indexer.rs	scan.ts (SegmentScanner, scanOverviewText), aggregator.ts, range_indexer.ts
src/names.rs	lib/release (identity, random), lib/files/volumes, lib/rules (upload tools, judge)
src/codec.rs	lib/chunks encodeChunk, format v2
src/js.rs	JavaScript semantics, sha1
check.ts	the comparison

The worker

cargo build --release
COORDINATOR_URL=http://coordinator:3334 NODE_TOKEN=… ./target/release/edge-node [--threads N] [--tail-threads T] [--batch-articles 500000] [--snapshot-minutes 30] [--groups N] [--max-mbit M]

• Two lanes (2026-10-27): --tail-threads T (EDGE_TAIL_THREADS; one of two or more threads by default) puts that many threads on the groups' tail lane (?lane=tail on the group routes): the tail planner's ranges, through a window and snapshot of their own (<group>-tail.gz). The other threads work the history lane: the windows and the history the coordinator's bounds hand out. A tail thread never wades through a group's history first, and a held history group's tail isn't stuck behind its days of ranges; the coordinator orders each lane's leases by alternating the group with the most planned articles and the longest unheld one.

• One group per thread (default one per core; REBUILD.md, "Voyager 3: window indexing", 1 and 5): a thread leases a group (POST groups/lease), takes its ranges in article order (POST groups/:id/next), streams each range's chunks (GET ranges/:id/stream: the corpus or the spool, the worker can't tell) through one carried window, posts a batch at least at every range end and finishes the range; one thread reads the stream and inflates a frame ahead, another posts the batches in order, the indexer goes on meanwhile. The window's snapshot (bincode, gzip) goes to the coordinator every --snapshot-minutes (EDGE_SNAPSHOT_MINUTES, 30) right after a batch, and when the group is given back: caught up (next answers 204: the tail), a stop, a lost lease or a failure; a stop or a failed stream posts the open batch first, so the snapshot stands at its resume point (a batch the coordinator refused can't, and that window is lost); the next holder restores it (GET groups/:id/snapshot) and continues at its article. --groups N stops after N groups; --ranges N is the old per-range lease (POST lease), one range per thread without a carried window.

• As commands/edge_node.ts: retries for network trouble and 5xx; 409 means the lease is lost (the range stops, isn't given back); a batch once 500k articles or 100k files wait (a picture group's batches went over the server's 256 MB limit); a chunk that doesn't inflate is reported (the coordinator fetches it plain next time) after the batch before it went out, and a held group asks for the range again from that chunk, its window kept; SIGTERM/SIGINT give each range back after its current chunk; --max-mbit is the node's download cap, shared by its threads. Plus a heartbeat when no batch went out for a minute.

• Wire format: src/wire.rs writes a batch as toWire does. edge-bench --wire and check.ts --wire compare it with the TypeScript's, field by field.

• End to end: both workers indexed the a.b.models capture (spooled as a range) through the web server; what the server stored was the same (releases with every column, chunks, candidates, rule counts, fragments with their JSON).

• On homelab: compose service edge (deploy/compose.yaml), the edge image target of the Dockerfile, --threads $EDGE_THREADS (8), EDGE_TAIL_THREADS (1); it talks to node-api.

• End to end: tests/e2e/edge_node.spec.ts runs this binary (edge/target/release/edge-node, or EDGE_NODE_BIN) against the test server, database, spool and corpus: a chunk the corpus fill stores while the stream waits, a failed stream and the restart that restores the snapshot, a restart whose snapshot file is gone, a takeover. node ace test e2e after cargo build --release; CI builds the binary first.