usenet-indexer/REBUILD.md

Voyager 2: header-first indexer (concept)

The concept for what runs (since 2026-10-11; it started as the spec of a rebuild, then a draft for approval). Voyager 1's CONCEPT.md and its decisions D1–D42 are superseded; its domain model still holds, and its features not yet here are the phase 2 backlog below.

Branch rebuild; its successor is designed under Voyager 3 below (branch voyager-3). Phase 1 finds releases from headers and makes their NZBs. Everything learned from articles (names from yEnc headers, par2, archive listings, encryption, spots) is phase 2 and adds to what phase 1 stored; it never rewrites it. Sources: CONCEPT.md, LOG.md 2026-10-08 (13)–(14).

Status (2026-10-09)

Built on this branch (a fresh AdonisJS 7 app; phase 1 code imported from the current app by pipeline stage): coordinator (planner, NNTP pool, prefetch spool, leases, rule sets, fragment merger, settling), the node API (lease, framed stream, batches, finish/release/heartbeat), the edge worker (edge:node, --processes), the catalog and chunk store, NZBs and a name search. Measured on the workstation (4 cores, 100 Mbit, Eweka):

• Edge worker, one core: ~210–215 k headers/s on recent a.b.boneless, ~330–340 k on a 2013 a.b.ath range; ~0.7–0.8 GB peak per process. (Start: 73–103 k/s, 1.0–1.2 GB.) What's left is spread out (the per-file key ~12%, GC ~10%, zlib ~10%, the scan ~6%).
• NNTP: Eweka takes ~1.5–2 s per XOVER command; ~22–30 k lines/s per connection in chunks of 100 k when the line allows; connections don't slow each other.
• Bandwidth: compressed XOVER is ~69 B per article on recent boneless (100 k-article chunks of ~7 MB). The workstation's 100 Mbit line carries ~140–150 k articles/s, which is where its end-to-end runs now stop (10 connections 90–99% busy, ~89 Mbit/s).
• End to end at that rate (one 4-core machine): workers about half busy each; server side ~17% (coordinator) + ~8% (web) + ~3% (Postgres) of a core. (Before this round's fixes Postgres was at 32%: the merger's neighbour query read every range per file.)
• On homelab (12 cores, 1 Gbit; 2026-10-09, separate containers and database, /mnt/bulk/usenet-rebuild): 20 connections fetched ~630 k articles/s (~31 k per connection, ~3 s per 100 k chunk, ~250 Mbit/s at ~50 B per article on older boneless) and 8 worker processes indexed all of it, waiting for the spool (~366 M headers in ~12 minutes). Server side at that rate: coordinator 0.4–0.5 cores (1.3–1.7 before it stopped inflating replies to check them), web 0.3–0.45, Postgres (on the HDD) under 0.15. Worker per core there: ~195 k headers/s (boneless 2026), ~275 k (ath 2013).
• Storage: chunks 17.9–20.3 B per segment (format v2); a release ~1.1 KB.
• For the whole target (528 B overview lines): ~36 TB of compressed XOVER to download. At 50 connections (~1–1.25 M lines/s, ~0.6–0.7 Gbit/s) ~5–6 days, ~5–6 worker cores, two coordinator shards (one process handles ~60 MB/s), under a core for web and Postgres. The line is the limit below ~0.7 Gbit/s: at 1 Gbit, ~3.5 days at best.

Where workers run: total speed is set by the account's NNTP connections (~30 k articles/s each), not by worker CPU, so remote workers don't make it faster; they take CPU off homelab (at 1.25 M/s ~7 worker cores) at the cost of relaying the compressed XOVER (≤ ~0.7 Gbit/s up, homelab has 1 Gbit/s). Plan: coordinator, web, Postgres and a few workers on homelab, most workers on batcave, a few on poweredge-kantoor with edge:node --max-mbit (e.g. 50 of its 160 Mbit/s ≈ 100–125 k articles/s; the cap is shared by the node's processes).

Every group is tiled from article 1 (2026-10-28): one "ancient" range, status ancient, covers 1 … low watermark − 1, what had expired on the server before the group was planned. Never leased, fetched or filled — nothing selects it — and the status panel reports it apart ("Ancient: … articles below the low watermark"); migration 6700 added it for the groups planned before, Planner.historyRows for groups planned since.

Also built (2026-10-09): the live tail (new articles planned every 2 minutes from one LIST ACTIVE per name prefix, at priority −1: leased and fetched before history; since 2026-10-27 a round grows the group's open tail range, a planned one at the tail's priority nothing spooled or took, up to RANGE_ARTICLES before it adds rows, as a tail nothing leased had become a sliver a group a round, 127k rows of a median 14k articles; ranges:merge-tail packs what such rounds left); the catalog API, Newznab (/api) and the web interface, from voyager 1 (the current build), with categories from the name; voyager 1's 29 learned garbage signatures (rules:signatures, database/data/voyager1_signatures.tsv).

Deployment (2026-10-10): Dockerfile (targets app and edge), deploy/compose.yaml (project usenet-rebuild on homelab), deploy/deploy.sh; a push to rebuild that passes CI is deployed. See deploy/README.md.

Not built yet: re-running ranges for a retracted rule (the counts and rule versions are recorded), decision f, phase 2 (inspection of the probe queue: yEnc names, encryption, archives, par2; passwords; spots).

Target

7,000 days (all of Eweka's retention, back to 2008-08) of the 1000 largest alt.binaries.* groups. Estimated: 528 B overview lines; 435 B in their main group; 140 B kept segments in ~1.25 B files and ~115 M releases after dropping garbage; 28 TB of compressed overview traffic. Throughput goal: ≥ 25 B overview lines/day (~50 connections), so ~3 weeks.

Parts

1. Coordinator (server). Plans each group's history as ranges of 2 M articles, leases them to workers, tracks progress and retries. The group list (owner rule) and the garbage signature table go with every lease. Fetches XOVER for the workers (relay: workers have no NNTP account; the gzip stream is passed on as is).

   Prefetch: the coordinator fetches the next ranges ahead of the workers, in chunks of 100 k lines and in order, into a spool on disk (~106 MB per 2 M range at ~53 B per line; on the HDD, written and read sequentially). It keeps about two ranges ahead per active lease, capped by size (default 10 GB), and fetches with all its header connections all the time, independent of how fast workers are. A lease goes to a range that is already (partly) spooled, else to the next planned one (leased ranges are fetched first); a worker reads its chunks from the spool, waiting only for chunks still being fetched. A worker gets its whole range in one streaming request (GET /ranges/:id/stream): the spooled chunks in order, waiting for those still being fetched. It still parses chunk by chunk (2 M lines are ~560 MB of text) and posts results per batch, moving the range's resume point. Chunks matter on the NNTP side: a compressed XOVER reply can't be resumed, old ranges answer slowly, and chunks of one range can be fetched in parallel. Spooled chunks are deleted when their range's results are stored, and fetched again if a range is leased again later. Order: oldest era first (2008–2018 holds 63% of the readable files on 17% of the lines), then forward. One coordinator process handles ~60 MB/s of compressed XOVER (its event loop was 16.5% busy at ~10 MB/s); beyond that, prefetching is split over processes with coordinator:run --shard=i/n (ranges with id % n = i − 1, each with its own NNTP_CONNECTIONS; only shard 1 plans, merges and settles).

2. Edge worker (node, TypeScript reusing lib/). Measured (edge:bench, 2026-10-09, after two rounds of work): ~200–215 k headers/s per core on recent a.b.boneless, ~320–340 k on a 2013 a.b.ath range, ~0.7 GB peak per process. Rust (edge/, the same RangeIndexer path over the same captures; check.ts finds every output identical, the decoded payloads included): 3.5–3.8× less CPU (boneless 2.9 vs 11.1 CPU-s per 2 M headers, ath 2.15 vs 7.5) and a third of the memory (0.20–0.24 GB). Its gains: no GC, hand-written matching instead of the regexes, mimalloc, libdeflate (half of zlib's inflate time) and SIMD line scanning. At the full target (~1.25 M/s) that is ~1.5–2 worker cores instead of ~6–7; the speed is still set by the NNTP connections. The cost: a second implementation of lib/headers, lib/release, lib/rules and the chunk format, kept identical (check.ts) for as long as both exist. Findings, how to run both and how to compare their outputs: edge/README.md. Since 2026-10-09 the Rust worker (edge-node, same crate) is the one running on homelab: ~8× less CPU per header and 20× less memory than edge:node in production. Per lease: compressed chunks of 10–50 k lines, in order → parse subjects → skip crossposts outside their owner group (of the indexed groups in the article's Xref, the one with the lowest priority number; see the group list) → group articles into files → files into releases (lib/release rules) → drop garbage (random name + dropped signature or tool), bare executables, dropped posters, and the single-archive fakes (stub archives and nonsense pieces, below) → set aside per-article random subjects and obfuscated subjects (counted only; phase 2). Returns per batch: release chunks, edge fragments, probe candidates, counters. Holds no state beyond its lease.

   Probe candidates: per release, the articles phase 2 should read, as (release key, kind, message-id, file in its chunk): the first article of the first archive volume (.part01.rar, .rar, .7z.001), of the par2 index (smallest .par2), and of a lone random-named file; for a 7z set also its last volume (the listing is in the tail). At most a few per release (~115 M releases → ~150–250 M articles), not one per file. Per- article and obfuscated subjects get none (they need every article's body).

3. Chunk store (bulk, append-only). One record per release per batch: its files' names, sizes and segment lists, compactly encoded (template codec for message-ids, zstd). Written once, never updated. A batch is stored in one transaction and only when it starts at its range's resume point, so a batch sent twice or late is refused and the catalog's sums stay right (no per-file dedup); a recovery run for a retracted rule only adds what that rule had dropped. ~3 TB.

4. Release catalog (hot, Postgres on NVMe). One row per release: key, name, poster, groups, posted_at, file count, size, segments, expected file count ([i/N]), has-par2, readable-name flag, state. Updated with aggregates once per batch (one upsert per release per batch), plus a small name index for search. ~115 M rows, ~50–100 GB.

5. Rules and recovery. What a worker skips isn't stored (message-ids are the bulk: garbage alone would be ~2.5 TB); a skipped range is read again instead (headers are cheap: ~53 B per overview line compressed).

   • Per range, per rule and signature: how many articles and files the rule dropped (garbage per (tool, group), per-article random subjects, obfuscated subjects, and crossposts per owner group). A few rows per range.
   • Rule versions: every range records the version of the rule set it ran with. Retracting or changing a rule makes the ranges where it matched stale; the coordinator leases them again for that rule only, and the worker returns just what the rule had dropped.
   • Sample: ~1% of what each rule drops (by a hash of the release key) is kept with full file details and probe candidates, so a rule can be checked, and its losses measured in phase 2, before it is retracted.

6. Probe queue. The candidates, in the catalog's database: one small row each (release, kind, message-id, chunk reference), priority by the release's state (settled first) and kind (random names before readable ones). Phase 2 takes them from here; phase 1 only fills it.

7. Fragments. Files crossing a range edge: kept by range and article number; merged when the neighbouring range is done, then written as a chunk. Never staged per segment.

8. Settling. A release is settled when nothing was added to it for 6 hours (30 minutes once its last post is that old: history), and its completeness decided: complete when every file its [i/N] counter announced was found. Without a counter, by its par2 sets once their index is probed (par2_completeness): every file a set describes is there by name or by its first 16 KiB, or, renamed, as many files and bytes; decided again after a par2 merge and before the pruner drops a set's file list. Segments aren't compared. Phase 2 picks its work from settled releases.

9. NZB and API. NZB built from a release's chunks, with a known password in its meta; Newznab search (names, inner names) and get; a web interface. Categories (Newznab's) by a trigger on releases (migration 5200): from the name, else the inner name the prober found, else a spot's category (spot_category, set by the spot inspector), else the groups (erotica, sounds, ebooks, …). releases:recategorize recomputes old rows in batches.

Group list

Editable at any time: an added group gets its ranges planned, a removed one no new leases. Each group gets a priority number when it is added (the first list by size, later additions after it); a crosspost belongs to the indexed group in its Xref with the lowest number. So adding a group never takes articles from groups already done, and removing one re-reads only the ranges of other groups whose counts show crossposts skipped for it (part 5). Start: the 1000 largest alt.binaries.* groups by LIST ACTIVE (~266 k ranges of 2 M articles; boneless alone 80 k).

Corpus

The grouping sweeps' dataset (2026-10-12; LOG.md 2026-10-12 (12)): the XOVER chunks of ranges marked capture, kept on the HDD in corpus/<group>/<start>-<end>/ (edge-bench's layout, range.json once complete), up to 300 GB. The prefetcher stores them as it fetches and reads them from there afterwards, so a reset re-indexes them without the news server. deploy/corpus_sweep.py marks more ranges each sweep (capture, spread over groups and eras, the top-100 groups first), replays every stored range with the deployed build's edge-bench (replay, ~seconds a range) and reports suspects and the change since the previous build (report). A rule change is judged on the whole corpus before a reset.

Since 2026-10-23 also history fetched ahead of the indexing: while the catalog is over its NVMe cap, ranges marked capture (priority >= 0, not leased) are fetched into the corpus by idle loops, at most perGroup / perRange of a group at a time, to be indexed from disk once there's room (LOG.md 2026-10-23 (100)). Stored filtered (CORPUS_FILTER, default on): without the lines the edge worker only counts, decided per line as it does (a crosspost another indexed group owns, a subject that names no file part; ~48% of headers), the chunk compressed again as zlib; beside it a .drops (gzip JSON: rule version, the owners' priorities, per rule the article numbers as differences, crossposts by owner) and filtered in range.json. The worker makes the same releases of a filtered chunk; rule_counts lack those two rules' counts. corpus:filter filters chunks stored raw. A corpus filtered this way no longer serves a sweep of those two rules.

Windows by posting time (2026-10-25; voyager 3's test data, LOG.md (104)): ranges:dates samples when each active group's articles were posted (one XOVER of 32 articles every 500M articles of the group, 16 … 1000 samples, the median Date of each slice, kept monotone; group_dates; over plain connections: Eweka cuts a compressed reply at the first ".\r\n" in its output, which some small replies begin with, TruncatedReply, LOG.md 2026-10-25 (105)) and estimates every range's posting date from the curve (ranges.est_date; the edges of a window are off by up to one sample interval, three weeks in boneless). ranges:capture --from 2025-01-01 --to 2026-01-01 marks every active group's ranges posted in the window capture: planned ones not started at --priority (0), done ones as they are, both fetched into the corpus by the fill, which takes capture ranges in priority order, done ones too (they're fetched again, corpus only), and skips those complete (ranges.stored_at, set when it sees the range.json). A year of every group is the data to replay voyager 3's 30-day blocks on: marked 2026-10-25, 2016 (priority 0, 30B articles, ~0.56 TB) and 2021 (priority 1, 87B, ~1.6 TB); a recent year is ~200B articles, ~3.6 TB.

Phase 2

Built: the prober (2026-10-11; probe:run, LOG.md 2026-10-11 (2)). It reads the probe queue's first articles (lone, par2, archive) and stores per release whether it's passworded and its inner name, which search, the web interface and Newznab use.

Built: the par2 merge (2026-10-12; releases:merge, the merger service, LOG.md 2026-10-12 (13)). The prober stores per probed file start the MD5 of its first 16 KiB (file_hashes) and per par2 body its set's file list (par2_files, par2_sets), which holds the same hash per file whatever the file was renamed to. Every minute the merger merges the releases of one group, posted within 48 hours of a release holding the set's par2, whose files the set describes, into the one with the most files (chunks, candidates, inspections, counts; the par2's shared name when none was readable); release_merges keeps what went where. Obfuscated uploads whose files were each a release of their own (lone candidates) become one. Readable files have no hash: a name pass (2026-10-23) joins the releases of the set's poster in its group posted within 12 hours whose files the set all names (no case or punctuation; sidecars, par2 and images aside), as via = 'name': an album's tracks and its index, an index beside its differently spelled video. releases:merge-names for sets from before. Also merged: lone files whose inner names share a random stem (inner), and small sidecar-only releases into their upload (sidecar). It reads what's new an hour (created_at) at a time, its place kept after each, and merges only the sets held or matched by two releases or more; a file whose first 16 KiB and size more than 20 files share is a common one and matches nothing (LOG.md 2026-10-20 (85)). A set's file list is dropped once it can't merge anything more: complete (all its files in its holder), judged (every holder junk), or final (every holder inside a run of done ranges, those ranges' candidates probed, its hashes seen by the merger); par2_sets stays (LOG.md 2026-10-20 (87)).

Backlog: voyager 1's features not in voyager 2 (CONCEPT.md decisions; voyager 1's data exported to /mnt/bulk/usenet-rebuild/v1-export/ on homelab: voyager1-tables.dump with 2.76 M spots, upload signatures, signature probes, dropped releases and groups; voyager1-release-passwords.csv with 68,032 releases with a password):

Feature	Voyager 1	In voyager 2
Spotnet spots	D29	the spotter and the spot inspector (below); new spots only, the export's not imported
Posted NZBs read into releases (also NZBs inside archives)	D26, D28	not yet
Passwords (from spots, NZB meta, names)	D29, D26	from spots (title, text) and their NZBs' meta, in release_passwords; else only whether passworded (the prober); the export's 68 k not imported
SFV grouping and naming	D34	not yet (SFV files aren't probe candidates)
Complete RAR listings by walking volume headers	D40	no: the prober reads the first volume's first article only
7z listings	D19	yes (2026-10-23): the prober reads the 7z-tail (the last segment of the highest volume, or of an unsplit multi-segment .7z) and lib/sevenzip lists it from the archive's end header (LZMA-packed, decoded in TypeScript); an encrypted header (-mhe) means passworded without names. One release keeps only the tails of its highest volume (seven_zip_tails). ZIP: the first segment's local headers (first entries, encryption)
Names from yEnc bodies for obfuscated subjects	D13, D21, D36	no: obfuscated and per-article subjects are only counted (part 5); their message-ids aren't stored, so a body path needs those ranges read again
Garbage dropped after probing (no real name, no password)	D42 (its second filter)	the junk judge hides it (below); nothing dropped yet

Live indexing (the tail) runs since 2026-10-09.

Junk (voyager 1's second filter, D42). After the header pass's rules (garbage signatures, executables, per-article and obfuscated subjects), the judge (releases:judge, service judge) judges releases by what the prober, the merges and the spots found: every 2 minutes the releases whose facts changed (noted by triggers in judge_dirty), daily every release (for what time changes: a day after settling, ranges read beside it). In this order (the first rule wins): dead (every probed first article gone), fake (a film, episode or music name whose content is a program: archive entries all executables, or the file itself one by name or by its first bytes, sniffed exe), repeated (the same first 16 KiB and size in 20+ releases), locked (passworded, no password from a spot, its NZB or the title), unusable (random names, no inner name, no par2 or inner-name merge, no spot, a day after settling). Verdicts (release_verdicts, with the rule set's version) hide releases from search (judged shows them); a verdict whose rule no longer matches is taken back. Nothing is dropped yet.

Sidecars (the merger's third pass). An upload's .nfo, .sfv or .nzb that came apart from it (another name, another counter: "[00/80] … .nzb" beside "[01/94] … .part01.rar") was a release of its own. The settler queues small settled releases (≤ 3 files, < 1 MB) in fold_candidates; one whose files are all sidecars joins the poster's bigger release in the same group posted within 10 minutes: the one sharing a word of 4+ letters with its subject, else the one under the same counter total, else none (release_merges.via 'sidecar').

Spots (after voyager 1's D29; lib/spotnet ported as is). Two processes, one NNTP connection each:

• The spotter (spots:read, service spotter) reads free.pt's overviews after the last article it saw (spot_state; the first time the last SPOT_BACKFILL, 20,000, ~3 months), every SPOT_EVERY_SECONDS (120). For each spot it reads the full header: the signature is checked there as Spotweb does (Spotnet's keys 2–4; self-signed key 7 by hashcash), and the X-XML gives the description, the website and the NZB's articles. A password the title ({{pw}}) or the description names (passwordFromSpot) is kept with the spot, with its source. A delete/dispose message marks its target removed (a personal one only from the same spotter within 5 days) and takes its links and passwords back from the releases.
• The spot inspector (spots:inspect, service spot-inspector) looks inside each listed spot, newest first: it reads the compressed NZB (≤ SPOT_MAX_NZB_SEGMENTS articles in alt.binaries.ftd), lists its files in spot_files (named as the header pass names them, parseSubject), and matches each file to the release the header pass built for it: the same release identity (releaseIdentity over the file's subject and poster) in one of its groups, the release posted nearest the file within 48 h. Matched releases are linked in release_spots. Placeholder NZBs (dummy), gone articles (nzb-gone) and unreadable ones are noted on the spot. Unmatched spots are matched again every hour; after SPOT_STANDALONE_AFTER_HOURS (24) a spot becomes a release of its own (source spot), whose NZB is the spot's (kept gzipped in spots.nzb).
• Passwords are kept apart in release_passwords (release, password, source: nzb for the NZB's <meta type="password">, else spot-title or spot-text, and the spot), not on the catalog row; the catalog only says passworded.

Voyager 3: window indexing (design, 2026-10-25)

Branch voyager-3 (a push to it only runs the tests; rebuild keeps running on homelab until the reset). Goal: the whole history (~1,220 B articles, 3% indexed on 2026-10-25) indexed and searchable on a 4 TB NVMe and the 15 TB HDD. The catalog costs 3.9 B of NVMe per article today (144 GB for 36.5 B), which puts the full history at ~4.7 TB; the target is ≤ 2.5 B per article (~3 TB) with room for the WAL and bloat. Sources: LOG.md 2026-10-24 (102).

What is wrong today: a release is written as soon as one range sees it, updated by every later range that finds more parts, merged afterwards (fragments, par2, names) and settled by wall-clock quiet time, which means little in a backfill. That churn is most of the WAL (0.6 TB a day) and the 20 k write IOPS bursts, and leaves bloat. Half the rows are single-article releases, a third of them one poster's image dump (27.7 M rows, LOG.md (102)).

1. One cursor a group (amended 2026-10-26; was: blocks). A group's history is still fetched as ranges of 2 M articles in chunks of 100 k (the spool, the corpus, the prefetcher stay) and leased as ranges, but a group's ranges are indexed in article order by one worker at a time, and the window below is carried from range to range: the next range finishes what the previous one left open. The first design leased blocks of 30 days of posting time, ran a group's blocks in parallel and joined their edges afterwards (a block merger). The user asked why (LOG.md 2026-10-25 (110)): downloading stays parallel through the prefetcher and a Rust thread parses faster than a connection delivers, so one worker a group loses nothing but the end of a reset, when the last big group runs alone. The merger, the block cutting and the blocks lease go. Costs: one seam a group where the history meets the tail (the releases in flight then end as two incomplete halves; history runs forward in one pass a group, priority only between groups), and the carried state must survive a restart (5 below). Measured on six consecutive ranges of a.b.ath (2016): one range is 15 hours of posting, so without the carry no release would finish inside a range (LOG.md 2026-10-26 (111)). The group lease (2026-10-26, the user's question in LOG.md (117)): a worker thread leases a group, not a range, and asks the coordinator for the group's next range each time it finished one; the coordinator keeps one cursor a group (group_leases: the holder, its heartbeat, the next range in article order, the snapshot it stands at). The answer is the range (id, span, resume point), the rules version and the stream URL — nothing about where the chunks are: the stream serves a stored range's chunks from the corpus file and a live one's from the spool, so a worker needs no mount and no corpus layout, and a remote worker is the same binary with another URL. The overhead is a copy through the node API: the corpus is ~30 B an article compressed (2016; 16–18 for 2021 and 2025), eight local workers at the bench's rate are ~170 MB/s, which the HDD under the corpus bounds either way; a remote worker is bounded by its link (100 Mbit ≈ 400k articles/s). History therefore no longer waits for the corpus fill: a group's stored ranges are read from disk, an unstored one is fetched live and captured into the corpus on the way, and the prefetcher runs ahead of each held group's cursor (plus the capture work left). A group with no planned range left is the tail: the worker keeps the group and asks again every minute, the planner's tail ranges fall into its order. A lease whose holder went quiet for LEASE_TTL_SECONDS is taken over: the new holder downloads the group's snapshot (5) and redoes the ranges since it; the batch store's keys make the redo idempotent. The replay harness (8) stays for offline runs; the driver on homelab goes once the worker runs.

2. The window. The worker reads a block in article order and keeps its open files and releases in memory (counters, names, sizes; the segment lists spooled to a file per block on the worker's disk). The frontier is the median Date of the last 1,024 articles with a plausible date (within 7 days of the frontier; 1,024 such dates in a row that agree among themselves move it, the group having paused); Date headers alone aren't trusted (clock skew, bogus years). 1,024, not more: the median trails the newest article by half the ring, a day in a group of 500 articles a day. A release is finished when the frontier is 24 h past its last part (WINDOW_MS): written once, with its completeness decided then (every announced file there and no file short of parts; par2 decides later for those without a counter), never updated by the header pass again. Incomplete releases are kept (user, 2026-10-24). A fill posted after the window is a release of its own (the judge's repeated rule already knows reposts). At a lease's end (the tail's poll, a worker stopping; amended 2026-10-26, was: the block's end) the releases the window hasn't closed on are written provisionally (settled_at null, finished: false, flagged edge): a release being uploaded shows in the catalog, and the carried window finishes the same row later, since the release key is the identity, the group and the first article; the store's finish path adds the files not sent before. No merger: the carry does the joining. The settler goes: settled_at is the finish. The tail needs no path of its own: its lease ends leave such provisional releases, the next lease's continuation finishes them. One exception (2026-10-27): a provisional release that is complete (every file its counter announced found) and quiet for six hours is settled by the coordinator (Settler.settleComplete, every five minutes), as the finish would: no counted file can still come, and the continuation may be hours away or never (the first run's range leases left 109k provisional rows at their blocks' end edges, 606 of them complete, shown as still collecting; the user, release 10697).

3. Albums. Files of one poster in one owner group whose subjects differ only by a number sequence, or share a directory-like prefix, posted within the window, become one release (source = 'album'): its name the shared template (or the prefix), its files kept by name so each stays findable; at least two files, else the single stays a release. This is for subjects that name no release (today's post/stem singles); a poster's ten different cracks in a day stay ten releases. The stake, from voyager 2's final analysis (LOG.md (112)): 77% of its 83.7 M releases had one file, 50% one segment (0.3% of the segments); a.b.pictures.erotica.porn alone was 27.7 M single-file releases of one poster. A counter total shared by many stems of one poster within the window is a batch counter, not a file count (Yenc-PP-A&A's [n/68067]): file_total null for them, not 68,067 incomplete singles. Built 2026-10-26 (edge/src/indexer.rs, LOG.md (113)): a release the window closed on with one file named by its stem or post, the name with a number and no archive, volume, sidecar or random name, waits in the album of its poster and template (the name's digit runs as #, the extension apart: #_#.jpg.data) posted within WINDOW_MS of the album's first file, at most ALBUM_MAX_FILES (1,000) a album; the album closes a window after that span and is written as one release (source: album, no file total) named by the members' shared numbers (1481439079025_# .jpg.data), a lone member as the release it was. The batch store writes the members' names as release_texts kind 6 (member, the first 20). The directory-like prefix variant is not built (no case seen in the corpus samples). The batch counter: a total ≥ 100 under which one poster started 20+ releases within 100 batches gives none of them a file total. On six a.b.ath ranges of 2016: 46 albums of 790 files among 2,099 chunks, most of them a subject parser miss ([n/60] Shogun I (1980) Shogun I (1980).avi.012: the counter text taken into the name, a volume each) that the album rule glues back together.

4. A slimmer catalog. releases loses its text copies: poster_id → posters(id) (the table gets an id; 33 B → 4), group_set_id → group_sets (the distinct group combinations: 139,927 in voyager 2's 83.7 M releases, LOG.md (112); 85 B jsonb → 4), release_key and identity_key as 20-byte bytea (41 B each → 20), source a smallint. ~150 of ~430 B a row; the poster and key indexes shrink with them (~12 GB of 50 today); releases_category_newest_idx (5.3 GB, used once) goes, category browsing filters the newest index. Today's 85 GB would be ~55. Cold tables move to the HDD tablespace: release_inspections and probe_candidates (28 GB, written once, not read by search); probe_cache is there already.

5. Memory and disk at the edge. The busiest group posts ~50 M articles a day; at a file per ~50 articles that is ~1 M open files of ~100 B in memory per block, and their segment lists (~25 B each) in the block's spool file, read back when a release is finished. Per worker thread a few hundred MB at worst. The carried window (amended 2026-10-26) has to survive a restart (deploys restart the workers daily): its counters and names are small (kilobytes for most groups, ~100 MB for the busiest), the open files' segment lists are the bulk (~1.25 GB for a day of the busiest group). A snapshot at every 2 M range would be ~600 B/article of writes, so the state is snapshotted at a lease's end and on a timer, to a file on bulk, and a crash redoes the ranges since the last snapshot from the corpus or the prefetch cache: the deterministic keys make the redo idempotent. The snapshot goes through the API (2026-10-26): PUT /node/groups/:id/snapshot with the serialized window (the frontier, the open files and releases, the open albums, the upload numbers, the batch-counter table; gzip) and the article it stands at (the last batch's next), kept by the coordinator on bulk as <snapshot dir>/<group id>; GET gives it back to the next holder. A worker snapshots when it gives a group back (a stop) and every SNAPSHOT_MINUTES while it holds one, always right after a batch was stored, so the snapshot's article is a resume point. The replay harness (8) keeps a group's window in one process without one. Built 2026-10-26 (LOG.md (117)): migration 6500 group_leases; app/services/group_leases.ts (lease by the most urgent planned range within LEASE_MIN_PRIORITY..LEASE_MAX_PRIORITY, the longest free first; takeover after LEASE_TTL_SECONDS, the ranges from the snapshot's article on planned again; next; heartbeat; release; the snapshot streamed to SNAPSHOT_DIR/<group id>.gz); the node API's groups/lease, groups/:id/next|heartbeat|release|snapshot; the range stream serving a stored range's chunks from the corpus; the prefetcher ahead of held groups' cursors, skipping chunks the corpus has; RangeIndexer::for_group/begin_range/snapshot/restore (bincode, the files a table, SNAPSHOT_VERSION 1); edge-node holding groups by default (--ranges keeps the old path). Tests: the lease order, takeover and redo, the endpoints with a corpus-served stream and the snapshot through the API, the snapshot restoring a window between two ranges (the two-range release one, the album joined). Deployed and started 2026-10-26 22:37 (LOG.md (118)). A chunk that doesn't inflate (a live reply cut short; the spool doesn't check) is reported after the batch before it went out and the range asked for again from that chunk: the window stays (the first run lost it, and the open batch). The lease bounds alone don't select a window: the history planner's oldest buckets are priority 0–2 too, so the first run also took 2026 history at 0 (25k ranges in 40 groups, fetched live) while 1,739 of the 2016 corpus ranges sat at voyager 2's leftover priorities (done when the window was marked, planned again by the reset): hence LEASE_CAPTURE_ONLY (the lease, next and the prefetcher's run-ahead take capture ranges only) and the one-off repair of the priorities. The stream polls the corpus as well as the spool while it waits for a chunk: the corpus fill claims chunks of the same range ahead of the leased run (the prefetcher's claims are per range and chunk), stores them in the corpus only, and the run leaves them there; a stream that looked at the corpus once, seconds before that store, waited its whole hour on the spool and the worker lost the group and its open batch (three groups on the first morning). A failed stream now posts the open batch before the group goes back, like a stop does, so the window's snapshot survives it; only a batch the coordinator refused (lease lost, too large) still loses the window. Two lanes a group (2026-10-27; migration 6600, LOG.md (119)). A group's ranges in article order through one window meant a thread that leased a group for its tail range first waded through the group's history (older articles come first), and a held history group's tail waited behind days of its ranges; and the bounds that ran the 2016 window alone (0..0, capture only) shut the tail out entirely. Now group_leases has a row per group and lane: the history lane works the ranges the bounds hand out (the windows, the history), the tail lane the tail planner's (priority at or below the tail's, whatever the bounds), each with its own holder, window, snapshot (<group>-tail.gz) and takeover redo (the lane's ranges only, and only those the window did after the snapshot: leased, done since it, or read past it). The edge puts --tail-threads of its threads on the tail lane (EDGE_TAIL_THREADS; 1 of 2 or more by default, which is what homelab's 16 run with), the rest on history; the prefetcher runs ahead of every held cursor. The seam a group is the same one as before, once: where the history lane reaches the first range the tail lane did, it is caught up. The lease order within a lane and priority is a mix (the user, 2026-10-27): one lease starts from the group with the most planned articles, the next from the longest unheld one (never held first), alternating on a sequence (group_lease_turns), so a big group snapshotted at a stop doesn't wait behind hundreds of small never-held groups and a quiet group's tail is visited all the same. coordinator:status (app/services/status.ts, --json) prints the catalog's room against CATALOG_MAX_NVME_GB, the ranges by band (tail, raised, each capture window, the history by thousand), the last hour's indexed and stored articles, the spool, the tail backlog, the held groups with their snapshots' age and size, the snapshot files, and the phase-2 backlogs (provisional releases, probe candidates, the pending ladder, the judge's dirty marks). End to end (tests/e2e, suite e2e, CI builds the binary first): the Rust worker against the real node API, database, spool and corpus — the corpus-fill race, a failed stream and the restart that restores the snapshot, a restart whose snapshot file is gone, a takeover.

6. The corpus stays as it is. Measured on 300 unfiltered chunks (30 M lines, LOG.md (102)): 43% of lines lack a "(k/N)" counter, and of those 73% are random 32-hex tokens, 27% [PRiVATE] hashes, ~0.3% posts with a readable name and no counter. The filter (crosspost, counterless) loses nothing voyager 3 would index; albums work from filtered chunks (the image dump's subjects carry "(1/1)"). Indexing without a counter at all (obfuscated uploads, ~31% of all lines) would need bodies, and the raw corpus (2,328 ranges) for a sweep.

7. Where it lives. The window, blocks and albums are in the Rust edge (edge/), the batch store, the planner and leases; the TypeScript RangeIndexer keeps the scanner (the corpus filter, tests) and drops its parity with the Rust worker for the window (the check.ts comparison covered the range path, which goes). New migrations rebuild the schema from a reset, no in-place conversion of the 81 M rows.

8. Test before the reset. Replay the corpus (the raw ranges and the captures, several B articles) with the new edge into a scratch database on bulk: NVMe bytes per article (the one number that decides whether 4 TB is enough), the share of albums and incomplete releases, and a sample of releases against today's. Speed is measured after the reset, on the NVMe.

9. The reset. Tail first, then history by priority, read from the corpus where stored. Survives: probe_cache (HDD, by message-id), spots, groups and their priorities, nntp_shares, the corpus. Recomputed: releases, chunks, inspections, candidates, par2 data, verdicts, file hashes. Resets are paused (2026-10-12); this one waits for the user's go-ahead, and for the new NVMe.

Schema (built 2026-10-25; migrations 6000 and 6100)

Migration 6000 is the reset: it truncates every table the indexing recomputes and refuses to run on a catalog with releases unless VOYAGER3_RESET=1, so merging voyager-3 into rebuild is the go-ahead. Survives: groups, ranges (planned again from their start, with their priorities, capture, est_date, stored_at, plain_chunks and measured posted_low/high), the corpus, probe_cache, spots (unlinked from releases; the inspector matches them again), nntp_shares, nodes, users, the rule sets.

• releases: created anew, 8-byte columns first: id, posted_at, last_posted_at, first_article, last_article, segments, bytes, settled_at (the finish; null while a tail release is provisional), inspected_at, first_seen_at; group_id, group_set_id, poster_id, file_total, files, chunks, category; source smallint (lib/release/source.ts: 1 subject, 2 post, 3 stem, 4 spot, 5 nzb, 6 album); readable, has_par2, complete, passworded, sidecar_only, edge (open at its block's end); release_key and identity_key as 20-byte bytea; name, inner_name. Gone: poster, groups, last_seen_at. Indexes: pkey, release_key unique, (identity_key, group_id), newest, (poster_id, newest), passworded, (group_id, identity_key) WHERE edge for the block merger, id WHERE settled_at IS NULL for the provisional ones. No trigram indexes: search matches release_texts. The category trigger reads the groups from group_sets; the judge's dirty trigger watches poster_id instead of poster.
• pending_releases (migrations 6300 and 6400, the HDD): the unnamed releases outside the catalog (decision f): release_key (primary), identity_key, group_id, name (the random stem), poster text, the posting span, first_article, last_article, counts, file_total, partial, finished, tool, candidates jsonb (the probe targets), inspections jsonb (what each probe showed), named, status (0 waiting, 1 named and in the catalog, 2 discarded, 3 exhausted, 4 named awaiting the move, 9 taken), tries, created_at, probed_at; partial indexes on the waiting (by created_at) and the named (by probed_at).
• pending_signatures (6400): per (tool, group, poster; poster '' for the tool and group together) probed, named, dropped, updated_at.
• release_texts(release_id, kind smallint, text varchar(255)) (6200), primary key on all three, one trigram index (fastupdate): every string search matches against, appended by whoever learns one and never an update of the releases row. Kinds (lib/release/text_kind.ts): 1 name (the release name, stored once more so one index serves everything), 2 file (the prober's names inside an archive), 3 nfo (an NFO's title), 4 par2 (a set's name), 5 media (a container's title); at most 20 per release and kind (textsToStore). A repeat insert is a no-op; the judge's dirty mark follows an insert or delete, so a rule can react to a name arriving later. Search: the trigram index yields release ids, joined to releases by primary key (never an OR across the two tables, which the planner cannot serve from indexes). inner_name stays on releases as the one display string. Facets to filter on (resolution, codec, language) are not texts: columns, or a meta jsonb with a jsonb_path_ops index once one is proven useful.
• posters(id serial, poster unique) with its trigram index; group_sets(id serial, groups jsonb unique), a sorted array of names.
• blocks (6100): group_id, start_id, end_id, next_id (the resume point), posted_low/high, status, priority, node_id, attempts, rule_version, headers, releases, leased_at, heartbeat_at, done_at, error; indexes todo (priority, id) and leased. ranges.block_id points at the block; the ranges' own lease columns stay until the leases move (then a migration drops them).
• release_chunks, rule_counts, rule_samples: range_id → block_id.
• Gone: fragments, range_blocks, release_final() (final = finished: settled_at IS NOT NULL), the settler, the fragment merger, releases_category_newest_idx.
• release_inspections and probe_candidates on CHUNK_TABLESPACE (the HDD) with their indexes. Note LOG.md 2026-10-19 (81): the judge's fake rule read all of release_inspections every round, 17 min on the HDD; the judge must read it by release or keep what it needs on the NVMe.

Batch store (built 2026-10-25)

app/services/batch_store.ts: every release chunk of a batch is one release, inserted once. Its keys (lib/release/keys.ts): identity_key the identity's SHA-1 as 20 bytes; release_key the SHA-1 of the identity, the owner group and the release's first article, so it is deterministic (no numbering under a group lock, no catalog lookups before writing) and a repost, starting at another article, is a release of its own. Its poster and group set by id (app/services/catalog_refs.ts: insert what's new, read the ids, cache them per process; the tests forget the cache per test), its source as a code, its name into release_texts (kind 1). The chunk's own counts go in as the release's; complete is decided then from the counter. A finished release's probe candidates are claimable at once (ready, unreadable); a provisional tail release (finished: false on the chunk, settled_at null) is finished by a later chunk of the same release key, which adds the files not sent before (sums, one more chunk), sets settled_at and makes its candidates claimable; a finished release sent again is left as it is (ON CONFLICT … WHERE settled_at IS NULL). Gone: the join of a chunk into an earlier release of its key (the window keeps an upload together; the block merger joins what a block's end cut), the join across groups (FTD's spread uploads: a gap, to be handled by the block merger or the par2 merger across groups), the numbering of release keys, the group lock, fragments (ignored when sent; the Rust edge still sends them). The chunk type carries finished and edge (both optional; the Rust edge doesn't send them yet, so its releases are finished, not edges). Interim until blocks are leased: chunks, rule counts and samples are keyed by ranges.block_id, and a range without a block gets a block of its own in the store (blockOf).

Readers moved with it: search joins posters and group_sets, matches words in release_texts (r.id IN (SELECT release_id … ILIKE)), searches posters by id, filters source by code and returns the source name; the par2 merger, the sidecar folder and the judge's scraped rule use poster_id (release_merges.from_key holds the key as hex); the spot inspector matches by bytea identity keys and inserts its spot releases by id references with their title in release_texts; the spotter and the NZB service compare the source code. Tests insert fixtures through tests/functional/helpers/releases.ts (insertRelease: text keys, poster and groups by name, source by name). The unit and functional suites pass.

The window in the Rust edge (built 2026-10-25)

edge/src/indexer.rs: the scanner and the file aggregator as before; then the window instead of the range path. Frontier keeps a ring of the last FRONTIER_ARTICLES (1,024) plausible dates (within PLAUSIBLE_MS, 7 days, of the frontier; 1,024 implausible dates in a row that agree move it), recomputed once per chunk; established after 64 dated articles (settle at the block's end for smaller blocks); its first value is the block's start in posting time. Per chunk: completed files, then the files whose last part is before frontier − WINDOW_MS (24 h; undated ones idle for idle chunks), go into the open release of their key whose posting span is within UPLOAD_GAP_MS (3 h) of theirs, else start one (the upload number per key, distinct within a batch); releases whose last part is before frontier − WINDOW_MS are judged and written finished, as an edge when posted within WINDOW_MS of the block's start. At the end (finish) everything left is written: finished when the window closed on it, else finished: false, edge: true. A file short of parts stays in its release and sets partial (the store: complete = false). A lone article of a multi-part random file is still counted per-article, not written; before the block's end a scrambled name counts as random too (looks_scrambled: a random subject and poster per article, 160k of alt.binaries.storage's releases; LOG.md (122)).

Single-archive rules (built 2026-10-31; LOG.md (123)-(127)). The judge (names::judge, RuleSet.judge) gets each file's articles seen and their bytes, and whether the release is settled (the window closed on it: a par2 can still join an open one; the start edge doesn't count, since a par2 follows its rar and 7% of the catalog's releases are start edges). A settled release of one unsplit zip, rar or 7z and nothing else is dropped as a stub archive (stub-archive, key: the legs tripped) when posted to 2+ groups, under 100 MB (the fakes stop at ~90 MB; hash-named uploads, real albums and passworded videos above it don't), and (A) its articles are ≤ 256 KiB with the release ≥ 1 MB (Power-Post's 250 kB default; a one-article release always has small articles, which exempts e-books and sidecars), or (X) posted to 4+ groups, or (W) by a webmail poster. A ROT13 volume name (".cnegN.rar" is ".partN.rar": a.b.erotica's passworded videos, LOG.md (128)) is a volume, not a single archive. A release touching a music, e-book, comics, tabs or fonts group is exempt: those communities post real single small archives the same way (karaoke zips to a.b.sounds.karaoke + a.b.test with Power-Post's articles; 16 of 32,308 known fakes lost, LOG.md (127)). These are the exe-stub and recycled-installer floods under film and software names: 55% of the catalog on 2026-10-30, 99.3% of the known fakes, 0 known content in the 217k sample, verified on 1,500 decoded articles. A settled single archive of one article and 100 kB–1 MB under a release-like, unscrambled name without spaces and without an e-book format in it is dropped as a nonsense piece (nonsense-piece, key: the upload tool): the newznzb.* pieces (scene-styled or random with a "-XX" suffix), 9% of the catalog; under 100 kB are cue/log sidecars and adverts, and the names with spaces are books and albums. Below that, a settled single archive of one article and 2–100 kB under a release-like name (spaces allowed), outside the exempt groups and without an e-book, subtitle, crack/keygen/trainer, cover or sidecar word, is dropped as a tiny archive (tiny-archive, key: the upload tool; LOG.md (131), (133)): the 2015–17 flood's RARs holding "