usenet-indexer/REBUILD.md

# Voyager 2: header-first indexer (concept)

The concept for what runs (since 2026-10-11; it started as the spec of a
rebuild, then a draft for approval). Voyager 1's CONCEPT.md and its
decisions D1–D42 are superseded; its domain model still holds, and its
features not yet here are the phase 2 backlog below.

Branch `rebuild`; its successor is designed under *Voyager 3* below (branch
`voyager-3`). Phase 1 finds releases from headers and makes their NZBs.
Everything learned from articles (names from yEnc headers, par2, archive
listings, encryption, spots) is phase 2 and adds to what phase 1 stored; it
never rewrites it. Sources: CONCEPT.md, LOG.md 2026-10-08 (13)–(14).

## Status (2026-10-09)

Built on this branch (a fresh AdonisJS 7 app; phase 1 code imported from
the current app by pipeline stage): coordinator (planner, NNTP pool,
prefetch spool, leases, rule sets, fragment merger, settling), the node API
(lease, framed stream, batches, finish/release/heartbeat), the edge worker
(`edge:node`, `--processes`), the catalog and chunk store, NZBs and a name
search. Measured on the workstation (4 cores, 100 Mbit, Eweka):

- Edge worker, one core: ~210–215 k headers/s on recent a.b.boneless,
  ~330–340 k on a 2013 a.b.ath range; ~0.7–0.8 GB peak per process.
  (Start: 73–103 k/s, 1.0–1.2 GB.) What's left is spread out (the
  per-file key ~12%, GC ~10%, zlib ~10%, the scan ~6%).
- NNTP: Eweka takes ~1.5–2 s per XOVER command; ~22–30 k lines/s per
  connection in chunks of 100 k when the line allows; connections don't
  slow each other.
- Bandwidth: compressed XOVER is ~69 B per article on recent boneless
  (100 k-article chunks of ~7 MB). The workstation's 100 Mbit line carries
  ~140–150 k articles/s, which is where its end-to-end runs now stop (10
  connections 90–99% busy, ~89 Mbit/s).
- End to end at that rate (one 4-core machine): workers about half busy
  each; server side ~17% (coordinator) + ~8% (web) + ~3% (Postgres) of a
  core. (Before this round's fixes Postgres was at 32%: the merger's
  neighbour query read every range per file.)
- On homelab (12 cores, 1 Gbit; 2026-10-09, separate containers and
  database, `/mnt/bulk/usenet-rebuild`): 20 connections fetched ~630 k
  articles/s (~31 k per connection, ~3 s per 100 k chunk, ~250 Mbit/s at
  ~50 B per article on older boneless) and 8 worker processes indexed all of
  it, waiting for the spool (~366 M headers in ~12 minutes). Server side at
  that rate: coordinator 0.4–0.5 cores (1.3–1.7 before it stopped
  inflating replies to check them), web 0.3–0.45, Postgres (on the HDD)
  under 0.15. Worker per core there: ~195 k headers/s (boneless 2026),
  ~275 k (ath 2013).
- Storage: chunks 17.9–20.3 B per segment (format v2); a release ~1.1 KB.
- For the whole target (528 B overview lines): ~36 TB of compressed XOVER
  to download. At 50 connections (~1–1.25 M lines/s, ~0.6–0.7 Gbit/s)
  ~5–6 days, ~5–6 worker cores, two coordinator shards (one process
  handles ~60 MB/s), under a core for web and Postgres. The line is the
  limit below ~0.7 Gbit/s: at 1 Gbit, ~3.5 days at best.

Where workers run: total speed is set by the account's NNTP connections
(~30 k articles/s each), not by worker CPU, so remote workers don't make
it faster; they take CPU off homelab (at 1.25 M/s ~7 worker cores) at the
cost of relaying the compressed XOVER (≤ ~0.7 Gbit/s up, homelab has 1
Gbit/s). Plan: coordinator, web, Postgres and a few workers on homelab,
most workers on batcave, a few on poweredge-kantoor with
`edge:node --max-mbit` (e.g. 50 of its 160 Mbit/s ≈ 100–125 k articles/s;
the cap is shared by the node's processes).

Also built (2026-10-09): the live tail (new articles planned every 2
minutes from one LIST ACTIVE per name prefix, at priority −1: leased and
fetched before history; since 2026-10-27 a round grows the group's open
tail range, a planned one at the tail's priority nothing spooled or
took, up to RANGE_ARTICLES before it adds rows, as a tail nothing leased
had become a sliver a group a round, 127k rows of a median 14k articles;
`ranges:merge-tail` packs what such rounds left); the catalog API,
Newznab (`/api`) and the web
interface, from voyager 1 (the current build), with categories from the
name; voyager 1's 29 learned garbage signatures (`rules:signatures`,
`database/data/voyager1_signatures.tsv`).

Deployment (2026-10-10): `Dockerfile` (targets app and edge),
`deploy/compose.yaml` (project usenet-rebuild on homelab), `deploy/deploy.sh`;
a push to `rebuild` that passes CI is deployed. See `deploy/README.md`.

Not built yet: re-running ranges for a retracted rule (the counts and rule
versions are recorded), decision f, phase
2 (inspection of the probe queue: yEnc names, encryption, archives, par2;
passwords; spots).

## Target

7,000 days (all of Eweka's retention, back to 2008-08) of the 1000 largest
`alt.binaries.*` groups. Estimated: 528 B overview lines; 435 B in their
main group; 140 B kept segments in ~1.25 B files and ~115 M releases after
dropping garbage; 28 TB of compressed overview traffic. Throughput goal:
≥ 25 B overview lines/day (~50 connections), so ~3 weeks.

## Parts

1. **Coordinator** (server). Plans each group's history as ranges of 2 M
   articles, leases them to workers, tracks progress and retries. The
   group list (owner rule) and the garbage signature table go with every
   lease. Fetches XOVER for the workers (relay: workers have no NNTP
   account; the gzip stream is passed on as is).

   **Prefetch**: the coordinator fetches the next ranges ahead of the
   workers, in chunks of 100 k lines and in order, into a spool on disk
   (~106 MB per 2 M range at ~53 B per line; on the HDD, written and read
   sequentially). It keeps about two ranges ahead per active lease, capped
   by size (default 10 GB), and fetches with all its header connections all
   the time, independent of how fast workers are. A lease goes to a range
   that is already (partly) spooled, else to the next planned one (leased
   ranges are fetched first); a worker reads its chunks from the spool,
   waiting only for chunks still being fetched. A worker gets its
   whole range in one streaming request (`GET /ranges/:id/stream`): the
   spooled chunks in order, waiting for those still being fetched. It still
   parses chunk by chunk (2 M lines are ~560 MB of text) and posts results
   per batch, moving the range's resume point. Chunks matter on the NNTP
   side: a compressed XOVER reply can't be resumed, old ranges answer
   slowly, and chunks of one range can be fetched in parallel. Spooled chunks are
   deleted when their range's results are stored, and fetched again if a
   range is leased again later.
   Order: oldest era first (2008–2018 holds 63% of the readable files on
   17% of the lines), then forward.
   One coordinator process handles ~60 MB/s of compressed XOVER (its event
   loop was 16.5% busy at ~10 MB/s); beyond that, prefetching is split over
   processes with `coordinator:run --shard=i/n` (ranges with id % n = i − 1,
   each with its own NNTP_CONNECTIONS; only shard 1 plans, merges and
   settles).

2. **Edge worker** (node, TypeScript reusing `lib/`). Measured (`edge:bench`,
   2026-10-09, after two rounds of work): ~200–215 k headers/s per core on
   recent a.b.boneless, ~320–340 k on a 2013 a.b.ath range, ~0.7 GB peak
   per process. **Rust** (`edge/`, the same RangeIndexer path
   over the same captures; `check.ts` finds every output identical, the
   decoded payloads included): 3.5–3.8× less CPU (boneless 2.9 vs 11.1
   CPU-s per 2 M headers, ath 2.15 vs 7.5) and a third of the memory
   (0.20–0.24 GB). Its gains: no GC, hand-written matching instead of the
   regexes, mimalloc, libdeflate (half of zlib's inflate time) and SIMD line
   scanning. At the full target (~1.25 M/s) that is ~1.5–2 worker cores
   instead of ~6–7; the speed is still set by the NNTP connections. The
   cost: a second implementation of lib/headers, lib/release, lib/rules and
   the chunk format, kept identical (check.ts) for as long as both exist.
   Findings, how to run both and how to compare their outputs:
   `edge/README.md`. Since 2026-10-09 the Rust worker
   (`edge-node`, same crate) is the one running on homelab: ~8× less CPU
   per header and 20× less memory than `edge:node` in production.
   Per lease: compressed chunks of 10–50 k lines, in order →
   parse subjects → skip crossposts outside their owner group (of the
   indexed groups in the article's Xref, the one with the lowest priority
   number; see the group list) → group articles into files → files
   into releases (`lib/release` rules) → drop garbage (random name +
   dropped signature or tool) → set aside per-article random subjects and
   obfuscated subjects (counted only; phase 2). Returns per batch: release
   chunks, edge fragments, probe candidates, counters. Holds no state beyond
   its lease.

   **Probe candidates**: per release, the articles phase 2 should read, as
   (release key, kind, message-id, file in its chunk): the first article of
   the first archive volume (`.part01.rar`, `.rar`, `.7z.001`), of the par2
   index (smallest `.par2`), and of a lone random-named file; for a 7z set
   also its last volume (the listing is in the tail). At most a few per
   release (~115 M releases → ~150–250 M articles), not one per file. Per-
   article and obfuscated subjects get none (they need every article's body).

3. **Chunk store** (bulk, append-only). One record per release per batch:
   its files' names, sizes and segment lists, compactly encoded (template
   codec for message-ids, zstd). Written once, never updated. A batch is
   stored in one transaction and only when it starts at its range's resume
   point, so a batch sent twice or late is refused and the catalog's sums
   stay right (no per-file dedup); a recovery run for a retracted rule only
   adds what that rule had dropped. ~3 TB.

4. **Release catalog** (hot, Postgres on NVMe). One row per release: key,
   name, poster, groups, posted_at, file count, size, segments, expected
   file count (`[i/N]`), has-par2, readable-name flag, state. Updated with
   aggregates once per batch (one upsert per release per batch), plus a
   small name index for search. ~115 M rows, ~50–100 GB.

5. **Rules and recovery**. What a worker skips isn't stored (message-ids
   are the bulk: garbage alone would be ~2.5 TB); a skipped range is read
   again instead (headers are cheap: ~53 B per overview line compressed).
   - *Per range, per rule and signature*: how many articles and files the
     rule dropped (garbage per (tool, group), per-article random subjects,
     obfuscated subjects, and crossposts per owner group). A few rows per
     range.
   - *Rule versions*: every range records the version of the rule set it ran
     with. Retracting or changing a rule makes the ranges where it matched
     stale; the coordinator leases them again for that rule only, and the
     worker returns just what the rule had dropped.
   - *Sample*: ~1% of what each rule drops (by a hash of the release key) is
     kept with full file details and probe candidates, so a rule can be
     checked, and its losses measured in phase 2, before it is retracted.

6. **Probe queue**. The candidates, in the catalog's database: one small row
   each (release, kind, message-id, chunk reference), priority by the
   release's state (settled first) and kind (random names before readable
   ones). Phase 2 takes them from here; phase 1 only fills it.

7. **Fragments**. Files crossing a range edge: kept by range and article
   number; merged when the neighbouring range is done, then written as a
   chunk. Never staged per segment.

8. **Settling**. A release is settled when nothing was added to it for 6
   hours (30 minutes once its last post is that old: history), and its
   completeness decided: complete when every file its [i/N] counter
   announced was found. Without a counter, by its par2 sets once their index
   is probed (`par2_completeness`): every file a set describes is there by
   name or by its first 16 KiB, or, renamed, as many files and bytes; decided
   again after a par2 merge and before the pruner drops a set's file list.
   Segments aren't compared. Phase 2 picks its work from settled releases.

9. **NZB and API**. NZB built from a release's chunks, with a known
   password in its meta; Newznab search (names, inner names) and get; a web
   interface. Categories (Newznab's) by a trigger on releases (migration
   5200): from the name, else the inner name the prober found, else a spot's
   category (spot_category, set by the spot inspector), else the groups
   (erotica, sounds, ebooks, …). `releases:recategorize` recomputes old rows
   in batches.

## Group list

Editable at any time: an added group gets its ranges planned, a removed one
no new leases. Each group gets a priority number when it is added (the first
list by size, later additions after it); a crosspost belongs to the indexed
group in its Xref with the lowest number. So adding a group never takes
articles from groups already done, and removing one re-reads only the ranges
of other groups whose counts show crossposts skipped for it (part 5). Start:
the 1000 largest `alt.binaries.*` groups by LIST ACTIVE (~266 k ranges of
2 M articles; boneless alone 80 k).

## Corpus

The grouping sweeps' dataset (2026-10-12; LOG.md 2026-10-12 (12)): the XOVER
chunks of ranges marked `capture`, kept on the HDD in
`corpus/<group>/<start>-<end>/` (edge-bench's layout, `range.json` once
complete), up to 300 GB. The prefetcher stores them as it fetches and reads
them from there afterwards, so a reset re-indexes them without the news
server. `deploy/corpus_sweep.py` marks more ranges each sweep (`capture`,
spread over groups and eras, the top-100 groups first), replays every stored
range with the deployed build's edge-bench (`replay`, ~seconds a range) and
reports suspects and the change since the previous build (`report`). A rule
change is judged on the whole corpus before a reset.

Since 2026-10-23 also history fetched ahead of the indexing: while the
catalog is over its NVMe cap, ranges marked `capture` (priority >= 0, not
leased) are fetched into the corpus by idle loops, at most perGroup /
perRange of a group at a time, to be indexed from disk once there's room
(LOG.md 2026-10-23 (100)). Stored filtered (`CORPUS_FILTER`, default on):
without the lines the edge worker only counts, decided per line as it does
(a crosspost another indexed group owns, a subject that names no file part;
~48% of headers), the chunk compressed again as zlib; beside it a `.drops`
(gzip JSON: rule version, the owners' priorities, per rule the article
numbers as differences, crossposts by owner) and `filtered` in range.json.
The worker makes the same releases of a filtered chunk; rule_counts lack
those two rules' counts. `corpus:filter` filters chunks stored raw. A
corpus filtered this way no longer serves a sweep of those two rules.

Windows by posting time (2026-10-25; voyager 3's test data, LOG.md (104)):
`ranges:dates` samples when each active group's articles were posted (one
XOVER of 32 articles every 500M articles of the group, 16 … 1000 samples,
the median Date of each slice, kept monotone; `group_dates`; over plain
connections: Eweka cuts a compressed reply at the first ".\r\n" in its
output, which some small replies begin with, `TruncatedReply`, LOG.md
2026-10-25 (105)) and estimates
every range's posting date from the curve (`ranges.est_date`; the edges of
a window are off by up to one sample interval, three weeks in boneless).
`ranges:capture --from 2025-01-01 --to 2026-01-01` marks every active
group's ranges posted in the window `capture`: planned ones not started at
`--priority` (0), done ones as they are, both fetched into the corpus by the
fill, which takes capture ranges in priority order, done ones too (they're
fetched again, corpus only), and skips those complete (`ranges.stored_at`,
set when it sees the range.json). A year of every group is the data to
replay voyager 3's 30-day blocks on: marked 2026-10-25, 2016 (priority 0,
30B articles, ~0.56 TB) and 2021 (priority 1, 87B, ~1.6 TB); a recent year
is ~200B articles, ~3.6 TB.

## Phase 2

Built: the prober (2026-10-11; `probe:run`, LOG.md 2026-10-11 (2)). It
reads the probe queue's first articles (`lone`, `par2`, `archive`) and
stores per release whether it's passworded and its inner name, which search,
the web interface and Newznab use.

Built: the par2 merge (2026-10-12; `releases:merge`, the `merger` service,
LOG.md 2026-10-12 (13)). The prober stores per probed file start the MD5 of
its first 16 KiB (`file_hashes`) and per par2 body its set's file list
(`par2_files`, `par2_sets`), which holds the same hash per file whatever the
file was renamed to. Every minute the merger merges the releases of one
group, posted within 48 hours of a release holding the set's par2, whose
files the set describes, into the one with the most files (chunks,
candidates, inspections, counts; the par2's shared name when none was
readable); `release_merges` keeps what went where. Obfuscated uploads whose
files were each a release of their own (`lone` candidates) become one.
Readable files have no hash: a name pass (2026-10-23) joins the releases
of the set's poster in its group posted within 12 hours whose files the
set all names (no case or punctuation; sidecars, par2 and images aside),
as `via = 'name'`: an album's tracks and its index, an index beside its
differently spelled video. `releases:merge-names` for sets from before.
Also merged: lone files whose inner names share a random stem (`inner`),
and small sidecar-only releases into their upload (`sidecar`).
It reads what's new an hour (`created_at`) at a time, its place kept after
each, and merges only the sets held or matched by two releases or more; a
file whose first 16 KiB and size more than 20 files share is a common one and
matches nothing (LOG.md 2026-10-20 (85)).
A set's file list is dropped once it can't merge anything more: complete (all
its files in its holder), judged (every holder junk), or final (every holder
inside a run of done ranges, those ranges' candidates probed, its hashes seen
by the merger); `par2_sets` stays (LOG.md 2026-10-20 (87)).

**Backlog: voyager 1's features not in voyager 2** (CONCEPT.md decisions;
voyager 1's data exported to `/mnt/bulk/usenet-rebuild/v1-export/` on
homelab: `voyager1-tables.dump` with 2.76 M spots, upload signatures,
signature probes, dropped releases and groups; `voyager1-release-passwords.csv`
with 68,032 releases with a password):

| Feature | Voyager 1 | In voyager 2 |
|---|---|---|
| Spotnet spots | D29 | the spotter and the spot inspector (below); new spots only, the export's not imported |
| Posted NZBs read into releases (also NZBs inside archives) | D26, D28 | not yet |
| Passwords (from spots, NZB meta, names) | D29, D26 | from spots (title, text) and their NZBs' meta, in `release_passwords`; else only *whether* passworded (the prober); the export's 68 k not imported |
| SFV grouping and naming | D34 | not yet (SFV files aren't probe candidates) |
| Complete RAR listings by walking volume headers | D40 | no: the prober reads the first volume's first article only |
| 7z listings | D19 | yes (2026-10-23): the prober reads the `7z-tail` (the last segment of the highest volume, or of an unsplit multi-segment .7z) and `lib/sevenzip` lists it from the archive's end header (LZMA-packed, decoded in TypeScript); an encrypted header (`-mhe`) means passworded without names. One release keeps only the tails of its highest volume (`seven_zip_tails`). ZIP: the first segment's local headers (first entries, encryption) |
| Names from yEnc bodies for obfuscated subjects | D13, D21, D36 | no: obfuscated and per-article subjects are only counted (part 5); their message-ids aren't stored, so a body path needs those ranges read again |
| Garbage dropped after probing (no real name, no password) | D42 (its second filter) | the junk judge hides it (below); nothing dropped yet |

Live indexing (the tail) runs since 2026-10-09.

**Junk** (voyager 1's second filter, D42). After the header pass's rules
(garbage signatures, executables, per-article and obfuscated subjects), the
judge (`releases:judge`, service `judge`) judges releases by what the
prober, the merges and the spots found: every 2 minutes the releases whose
facts changed (noted by triggers in `judge_dirty`), daily every release (for
what time changes: a day after settling, ranges read beside it). In this
order (the first rule wins): *dead* (every probed first article gone), *fake* (a film, episode
or music name whose content is a program: archive entries all
executables, or the file itself one by name or by its first bytes, sniffed
`exe`), *repeated* (the same first 16 KiB and size in 20+ releases),
*locked* (passworded, no password from a spot, its NZB or the title),
*unusable* (random names, no inner name, no par2 or inner-name merge, no
spot, a day after settling). Verdicts (`release_verdicts`, with the rule
set's version) hide releases from search (`judged` shows them); a verdict
whose rule no longer matches is taken back. Nothing is dropped yet.

**Sidecars** (the merger's third pass). An upload's .nfo, .sfv or .nzb
that came apart from it (another name, another counter: "[00/80] … .nzb"
beside "[01/94] … .part01.rar") was a release of its own. The settler
queues small settled releases (≤ 3 files, < 1 MB) in `fold_candidates`;
one whose files are all sidecars joins the poster's bigger release in the
same group posted within 10 minutes: the one sharing a word of 4+ letters
with its subject, else the one under the same counter total, else none
(`release_merges.via` 'sidecar').

**Spots** (after voyager 1's D29; `lib/spotnet` ported as is). Two
processes, one NNTP connection each:

- **The spotter** (`spots:read`, service `spotter`) reads free.pt's
  overviews after the last article it saw (`spot_state`; the first time the
  last `SPOT_BACKFILL`, 20,000, ~3 months), every `SPOT_EVERY_SECONDS`
  (120). For each spot it reads the full header: the signature is checked
  there as Spotweb does (Spotnet's keys 2–4; self-signed key 7 by hashcash),
  and the X-XML gives the description, the website and the NZB's articles.
  A password the title (`{{pw}}`) or the description names
  (`passwordFromSpot`) is kept with the spot, with its source. A
  delete/dispose message marks its target `removed` (a personal one only
  from the same spotter within 5 days) and takes its links and passwords
  back from the releases.
- **The spot inspector** (`spots:inspect`, service `spot-inspector`) looks
  inside each listed spot, newest first: it reads the compressed NZB (≤
  `SPOT_MAX_NZB_SEGMENTS` articles in alt.binaries.ftd), lists its files
  in `spot_files` (named as the header pass names them, `parseSubject`),
  and matches each file to the release the header pass built for it: the
  same release identity (`releaseIdentity` over the file's subject and
  poster) in one of its groups, the release posted nearest the file within
  48 h. Matched releases are linked in `release_spots`. Placeholder NZBs
  (`dummy`), gone articles (`nzb-gone`) and unreadable ones are noted on the
  spot. Unmatched spots are matched again every hour; after
  `SPOT_STANDALONE_AFTER_HOURS` (24) a spot becomes a release of its own
  (source `spot`), whose NZB is the spot's (kept gzipped in `spots.nzb`).
- **Passwords are kept apart** in `release_passwords` (release, password,
  source: `nzb` for the NZB's `<meta type="password">`, else `spot-title` or
  `spot-text`, and the spot), not on the catalog row; the catalog only says
  `passworded`.

## Voyager 3: window indexing (design, 2026-10-25)

Branch `voyager-3` (a push to it only runs the tests; `rebuild` keeps
running on homelab until the reset). Goal: the whole history (~1,220 B
articles, 3% indexed on 2026-10-25) indexed and searchable on a 4 TB NVMe
and the 15 TB HDD. The catalog costs 3.9 B of NVMe per article today
(144 GB for 36.5 B), which puts the full history at ~4.7 TB; the target is
≤ 2.5 B per article (~3 TB) with room for the WAL and bloat. Sources:
LOG.md 2026-10-24 (102).

What is wrong today: a release is written as soon as one range sees it,
updated by every later range that finds more parts, merged afterwards
(fragments, par2, names) and settled by wall-clock quiet time, which means
little in a backfill. That churn is most of the WAL (0.6 TB a day) and the
20 k write IOPS bursts, and leaves bloat. Half the rows are single-article
releases, a third of them one poster's image dump (27.7 M rows, LOG.md
(102)).

1. **One cursor a group (amended 2026-10-26; was: blocks).** A group's
   history is still fetched as ranges of 2 M articles in chunks of 100 k
   (the spool, the corpus, the prefetcher stay) and leased as ranges, but
   a group's ranges are indexed in article order by one worker at a time,
   and the window below is *carried* from range to range: the next range
   finishes what the previous one left open. The first design leased
   blocks of 30 days of posting time, ran a group's blocks in parallel and
   joined their edges afterwards (a block merger). The user asked why
   (LOG.md 2026-10-25 (110)): downloading stays parallel through the
   prefetcher and a Rust thread parses faster than a connection delivers,
   so one worker a group loses nothing but the end of a reset, when the
   last big group runs alone. The merger, the block cutting and the
   `blocks` lease go. Costs: one seam a group where the history meets the
   tail (the releases in flight then end as two incomplete halves; history
   runs forward in one pass a group, priority only between groups), and
   the carried state must survive a restart (5 below). Measured on six
   consecutive ranges of a.b.ath (2016): one range is 15 hours of posting,
   so without the carry no release would finish inside a range (LOG.md
   2026-10-26 (111)).
   **The group lease (2026-10-26, the user's question in LOG.md (117)):**
   a worker thread leases a *group*, not a range, and asks the coordinator
   for the group's next range each time it finished one; the coordinator
   keeps one cursor a group (`group_leases`: the holder, its heartbeat,
   the next range in article order, the snapshot it stands at). The
   answer is the range (id, span, resume point), the rules version and the
   stream URL — nothing about where the chunks are: the stream serves a
   stored range's chunks from the corpus file and a live one's from the
   spool, so a worker needs no mount and no corpus layout, and a remote
   worker is the same binary with another URL. The overhead is a copy
   through the node API: the corpus is ~30 B an article compressed (2016;
   16–18 for 2021 and 2025), eight local workers at the bench's rate are
   ~170 MB/s, which the HDD under the corpus bounds either way; a remote
   worker is bounded by its link (100 Mbit ≈ 400k articles/s). History
   therefore no longer waits for the corpus fill: a group's stored ranges
   are read from disk, an unstored one is fetched live and captured into
   the corpus on the way, and the prefetcher runs ahead of each held
   group's cursor (plus the capture work left). A group with no planned
   range left is the tail: the worker keeps the group and asks again
   every minute, the planner's tail ranges fall into its order. A lease
   whose holder went quiet for LEASE_TTL_SECONDS is taken over: the new
   holder downloads the group's snapshot (5) and redoes the ranges since
   it; the batch store's keys make the redo idempotent. The replay
   harness (8) stays for offline runs; the driver on homelab goes once the
   worker runs.

2. **The window.** The worker reads a block in article order and keeps its
   open files and releases in memory (counters, names, sizes; the segment
   lists spooled to a file per block on the worker's disk). The *frontier*
   is the median Date of the last 1,024 articles with a plausible date
   (within 7 days of the frontier; 1,024 such dates in a row that agree
   among themselves move it, the group having paused); Date headers alone
   aren't trusted (clock skew, bogus years). 1,024, not more: the median
   trails the newest article by half the ring, a day in a group of 500
   articles a day. A release is **finished** when the frontier is 24 h past
   its last part (`WINDOW_MS`): written once, with its completeness decided
   then (every announced file there and no file short of parts; par2
   decides later for those without a counter), never updated by the header
   pass again. Incomplete releases are kept (user, 2026-10-24). A fill
   posted after the window is a release of its own (the judge's `repeated`
   rule already knows reposts). At a lease's end (the tail's poll, a worker
   stopping; amended 2026-10-26, was: the block's end) the releases the
   window hasn't closed on are written **provisionally** (`settled_at`
   null, `finished: false`, flagged `edge`): a release being uploaded
   shows in the catalog, and the carried window finishes the same row
   later, since the release key is the identity, the group and the first
   article; the store's finish path adds the files not sent before. No
   merger: the carry does the joining. The settler goes: `settled_at` is
   the finish. The **tail** needs no path of its own: its lease ends leave
   such provisional releases, the next lease's continuation finishes them.

3. **Albums.** Files of one poster in one owner group whose subjects differ
   only by a number sequence, or share a directory-like prefix, posted
   within the window, become one release (`source = 'album'`): its name the
   shared template (or the prefix), its files kept by name so each stays
   findable; at least two files, else the single stays a release. This is
   for subjects that name no release (today's `post`/`stem` singles); a
   poster's ten different cracks in a day stay ten releases. The stake,
   from voyager 2's final analysis (LOG.md (112)): 77% of its 83.7 M
   releases had one file, 50% one segment (0.3% of the segments);
   a.b.pictures.erotica.porn alone was 27.7 M single-file releases of one
   poster. A counter total shared by many stems of one poster within the
   window is a batch counter, not a file count (Yenc-PP-A&A's [n/68067]):
   `file_total` null for them, not 68,067 incomplete singles.
   **Built 2026-10-26** (edge/src/indexer.rs, LOG.md (113)): a release the
   window closed on with one file named by its stem or post, the name with
   a number and no archive, volume, sidecar or random name, waits in the
   album of its poster and template (the name's digit runs as `#`, the
   extension apart: `#_#.jpg.data`) posted within WINDOW_MS of the album's
   first file, at most ALBUM_MAX_FILES (1,000) a album; the album closes a
   window after that span and is written as one release (`source: album`,
   no file total) named by the members' shared numbers (`1481439079025_#
   .jpg.data`), a lone member as the release it was. The batch store writes
   the members' names as `release_texts` kind 6 (`member`, the first 20).
   The directory-like prefix variant is not built (no case seen in the
   corpus samples). The batch counter: a total ≥ 100 under which one poster
   started 20+ releases within 100 batches gives none of them a file
   total. On six a.b.ath ranges of 2016: 46 albums of 790 files among
   2,099 chunks, most of them a subject parser miss (`[n/60] Shogun I
   (1980) Shogun I (1980).avi.012`: the counter text taken into the name,
   a volume each) that the album rule glues back together.

4. **A slimmer catalog.** `releases` loses its text copies: `poster_id` →
   `posters(id)` (the table gets an id; 33 B → 4), `group_set_id` →
   `group_sets` (the distinct group combinations: 139,927 in voyager 2's
   83.7 M releases, LOG.md (112); 85 B jsonb → 4),
   `release_key` and `identity_key` as 20-byte `bytea` (41 B each → 20),
   `source` a smallint. ~150 of ~430 B a row; the poster and key indexes
   shrink with them (~12 GB of 50 today); `releases_category_newest_idx`
   (5.3 GB, used once) goes, category browsing filters the newest index.
   Today's 85 GB would be ~55. Cold tables move to the HDD tablespace:
   `release_inspections` and `probe_candidates` (28 GB, written once, not
   read by search); `probe_cache` is there already.

5. **Memory and disk at the edge.** The busiest group posts ~50 M articles
   a day; at a file per ~50 articles that is ~1 M open files of ~100 B in
   memory per block, and their segment lists (~25 B each) in the block's
   spool file, read back when a release is finished. Per worker thread a
   few hundred MB at worst. The carried window (amended 2026-10-26) has to
   survive a restart (deploys restart the workers daily): its counters and
   names are small (kilobytes for most groups, ~100 MB for the busiest),
   the open files' segment lists are the bulk (~1.25 GB for a day of the
   busiest group). A snapshot at every 2 M range would be ~600 B/article
   of writes, so the state is snapshotted at a lease's end and on a timer,
   to a file on bulk, and a crash redoes the ranges since the last snapshot
   from the corpus or the prefetch cache: the deterministic keys make the
   redo idempotent. **The snapshot goes through the API** (2026-10-26):
   `PUT /node/groups/:id/snapshot` with the serialized window (the
   frontier, the open files and releases, the open albums, the upload
   numbers, the batch-counter table; gzip) and the article it stands at
   (the last batch's `next`), kept by the coordinator on bulk as
   `<snapshot dir>/<group id>`; `GET` gives it back to the next holder. A
   worker snapshots when it gives a group back (a stop) and every
   SNAPSHOT_MINUTES while it holds one, always right after a batch was
   stored, so the snapshot's article is a resume point. The replay
   harness (8) keeps a group's window in one process without one.
   **Built 2026-10-26** (LOG.md (117)): migration 6500 `group_leases`;
   `app/services/group_leases.ts` (lease by the most urgent planned
   range within LEASE_MIN_PRIORITY..LEASE_MAX_PRIORITY, the longest free
   first; takeover after LEASE_TTL_SECONDS, the ranges from the snapshot's
   article on planned again; next; heartbeat; release; the snapshot
   streamed to `SNAPSHOT_DIR/<group id>.gz`); the node API's
   `groups/lease`, `groups/:id/next|heartbeat|release|snapshot`; the
   range stream serving a stored range's chunks from the corpus; the
   prefetcher ahead of held groups' cursors, skipping chunks the corpus
   has; `RangeIndexer::for_group/begin_range/snapshot/restore` (bincode,
   the files a table, SNAPSHOT_VERSION 1); `edge-node` holding groups by
   default (`--ranges` keeps the old path). Tests: the lease order,
   takeover and redo, the endpoints with a corpus-served stream and the
   snapshot through the API, the snapshot restoring a window between two
   ranges (the two-range release one, the album joined). Deployed and
   started 2026-10-26 22:37 (LOG.md (118)). A chunk that doesn't inflate
   (a live reply cut short; the spool doesn't check) is reported after
   the batch before it went out and the range asked for again from that
   chunk: the window stays (the first run lost it, and the open batch).
   The lease bounds alone don't select a window: the history planner's
   oldest buckets are priority 0–2 too, so the first run also took 2026
   history at 0 (25k ranges in 40 groups, fetched live) while 1,739 of
   the 2016 corpus ranges sat at voyager 2's leftover priorities (done
   when the window was marked, planned again by the reset): hence
   `LEASE_CAPTURE_ONLY` (the lease, next and the prefetcher's run-ahead
   take capture ranges only) and the one-off repair of the priorities.
   The stream polls the corpus as well as the spool while it waits for a
   chunk: the corpus fill claims chunks of the same range ahead of the
   leased run (the prefetcher's claims are per range and chunk), stores
   them in the corpus only, and the run leaves them there; a stream that
   looked at the corpus once, seconds before that store, waited its whole
   hour on the spool and the worker lost the group and its open batch
   (three groups on the first morning). A failed stream now posts the
   open batch before the group goes back, like a stop does, so the
   window's snapshot survives it; only a batch the coordinator refused
   (lease lost, too large) still loses the window.

6. **The corpus stays as it is.** Measured on 300 unfiltered chunks
   (30 M lines, LOG.md (102)): 43% of lines lack a "(k/N)" counter, and of
   those 73% are random 32-hex tokens, 27% `[PRiVATE]` hashes, ~0.3% posts
   with a readable name and no counter. The filter (crosspost, counterless)
   loses nothing voyager 3 would index; albums work from filtered chunks
   (the image dump's subjects carry "(1/1)"). Indexing without a counter at
   all (obfuscated uploads, ~31% of all lines) would need bodies, and the
   raw corpus (2,328 ranges) for a sweep.

7. **Where it lives.** The window, blocks and albums are in the Rust edge
   (`edge/`), the batch store, the planner and leases; the TypeScript
   RangeIndexer keeps the scanner (the corpus filter, tests) and drops its
   parity with the Rust worker for the window (the `check.ts` comparison
   covered the range path, which goes). New migrations rebuild the schema
   from a reset, no in-place conversion of the 81 M rows.

8. **Test before the reset.** Replay the corpus (the raw ranges and the
   captures, several B articles) with the new edge into a scratch database
   on bulk: NVMe bytes per article (the one number that decides whether
   4 TB is enough), the share of albums and incomplete releases, and a
   sample of releases against today's. Speed is measured after the reset,
   on the NVMe.

9. **The reset.** Tail first, then history by priority, read from the
   corpus where stored. Survives: `probe_cache` (HDD, by message-id),
   `spots`, `groups` and their priorities, `nntp_shares`, the corpus.
   Recomputed: releases, chunks, inspections, candidates, par2 data,
   verdicts, file hashes. Resets are paused (2026-10-12); this one waits
   for the user's go-ahead, and for the new NVMe.

### Schema (built 2026-10-25; migrations 6000 and 6100)

Migration 6000 *is* the reset: it truncates every table the indexing
recomputes and refuses to run on a catalog with releases unless
`VOYAGER3_RESET=1`, so merging `voyager-3` into `rebuild` is the go-ahead.
Survives: `groups`, `ranges` (planned again from their start, with their
priorities, `capture`, `est_date`, `stored_at`, `plain_chunks` and measured
`posted_low/high`), the corpus, `probe_cache`, `spots` (unlinked from
releases; the inspector matches them again), `nntp_shares`, `nodes`,
`users`, the rule sets.

- `releases`: created anew, 8-byte columns first: `id`, `posted_at`,
  `last_posted_at`, `first_article`, `last_article`, `segments`, `bytes`,
  `settled_at` (the finish; null while a tail release is provisional),
  `inspected_at`, `first_seen_at`; `group_id`, `group_set_id`, `poster_id`,
  `file_total`, `files`, `chunks`, `category`; `source smallint`
  (`lib/release/source.ts`: 1 subject, 2 post, 3 stem, 4 spot, 5 nzb, 6
  album); `readable`, `has_par2`, `complete`, `passworded`, `sidecar_only`,
  `edge` (open at its block's end); `release_key` and `identity_key` as
  20-byte `bytea`; `name`, `inner_name`. Gone: `poster`, `groups`,
  `last_seen_at`. Indexes: pkey, release_key unique, (identity_key,
  group_id), newest, (poster_id, newest), passworded, `(group_id,
  identity_key) WHERE edge` for the block merger, `id WHERE settled_at IS
  NULL` for the provisional ones. No trigram indexes: search matches
  `release_texts`. The category trigger reads the groups from
  `group_sets`; the judge's dirty trigger watches `poster_id` instead of
  `poster`.
- `pending_releases` (migrations 6300 and 6400, the HDD): the unnamed
  releases outside the catalog (decision f): `release_key` (primary),
  `identity_key`, `group_id`, `name` (the random stem), `poster` text, the
  posting span, `first_article`, `last_article`, counts, `file_total`,
  `partial`, `finished`, `tool`, `candidates` jsonb (the probe targets),
  `inspections` jsonb (what each probe showed), `named`, `status` (0
  waiting, 1 named and in the catalog, 2 discarded, 3 exhausted, 4 named
  awaiting the move, 9 taken), `tries`, `created_at`, `probed_at`; partial
  indexes on the waiting (by `created_at`) and the named (by `probed_at`).
- `pending_signatures` (6400): per (tool, group, poster; poster '' for the
  tool and group together) `probed`, `named`, `dropped`, `updated_at`.
- `release_texts(release_id, kind smallint, text varchar(255))` (6200),
  primary key on all three, one trigram index (fastupdate): every string
  search matches against, appended by whoever learns one and never an
  update of the releases row. Kinds (`lib/release/text_kind.ts`): 1 name
  (the release name, stored once more so one index serves everything),
  2 file (the prober's names inside an archive), 3 nfo (an NFO's title),
  4 par2 (a set's name), 5 media (a container's title); at most 20 per
  release and kind (`textsToStore`). A repeat insert is a no-op; the
  judge's dirty mark follows an insert or delete, so a rule can react to a
  name arriving later. Search: the trigram index yields release ids, joined
  to `releases` by primary key (never an OR across the two tables, which
  the planner cannot serve from indexes). `inner_name` stays on `releases`
  as the one display string. Facets to filter on (resolution, codec,
  language) are not texts: columns, or a `meta jsonb` with a
  `jsonb_path_ops` index once one is proven useful.
- `posters(id serial, poster unique)` with its trigram index;
  `group_sets(id serial, groups jsonb unique)`, a sorted array of names.
- `blocks` (6100): `group_id`, `start_id`, `end_id`, `next_id` (the resume
  point), `posted_low/high`, `status`, `priority`, `node_id`, `attempts`,
  `rule_version`, `headers`, `releases`, `leased_at`, `heartbeat_at`,
  `done_at`, `error`; indexes todo (priority, id) and leased.
  `ranges.block_id` points at the block; the ranges' own lease columns stay
  until the leases move (then a migration drops them).
- `release_chunks`, `rule_counts`, `rule_samples`: `range_id` → `block_id`.
- Gone: `fragments`, `range_blocks`, `release_final()` (final = finished:
  `settled_at IS NOT NULL`), the settler, the fragment merger,
  `releases_category_newest_idx`.
- `release_inspections` and `probe_candidates` on `CHUNK_TABLESPACE` (the
  HDD) with their indexes. Note LOG.md 2026-10-19 (81): the judge's fake
  rule read all of `release_inspections` every round, 17 min on the HDD;
  the judge must read it by release or keep what it needs on the NVMe.

### Batch store (built 2026-10-25)

`app/services/batch_store.ts`: every release chunk of a batch is one
release, inserted once. Its keys (`lib/release/keys.ts`): `identity_key`
the identity's SHA-1 as 20 bytes; `release_key` the SHA-1 of the identity,
the owner group and the release's first article, so it is deterministic
(no numbering under a group lock, no catalog lookups before writing) and a
repost, starting at another article, is a release of its own. Its poster
and group set by id (`app/services/catalog_refs.ts`: insert what's new,
read the ids, cache them per process; the tests forget the cache per
test), its source as a code, its name into `release_texts` (kind 1). The
chunk's own counts go in as the release's; `complete` is decided then from
the counter. A finished release's probe candidates are claimable at once
(`ready`, `unreadable`); a provisional tail release (`finished: false` on
the chunk, `settled_at` null) is finished by a later chunk of the same
release key, which adds the files not sent before (sums, one more chunk),
sets `settled_at` and makes its candidates claimable; a finished release
sent again is left as it is (`ON CONFLICT … WHERE settled_at IS NULL`).
Gone: the join of a chunk into an earlier release of its key (the window
keeps an upload together; the block merger joins what a block's end cut),
the join across groups (FTD's spread uploads: a gap, to be handled by the
block merger or the par2 merger across groups), the numbering of release
keys, the group lock, `fragments` (ignored when sent; the Rust edge still
sends them). The chunk type carries `finished` and `edge` (both optional;
the Rust edge doesn't send them yet, so its releases are finished, not
edges). Interim until blocks are leased: chunks, rule counts and samples
are keyed by `ranges.block_id`, and a range without a block gets a block
of its own in the store (`blockOf`).

Readers moved with it: search joins `posters` and `group_sets`, matches
words in `release_texts` (`r.id IN (SELECT release_id … ILIKE)`), searches
posters by id, filters `source` by code and returns the source name; the
par2 merger, the sidecar folder and the judge's scraped rule use
`poster_id` (`release_merges.from_key` holds the key as hex); the spot
inspector matches by bytea identity keys and inserts its spot releases by
id references with their title in `release_texts`; the spotter and the NZB
service compare the source code. Tests insert fixtures through
`tests/functional/helpers/releases.ts` (`insertRelease`: text keys, poster
and groups by name, source by name). The unit and functional suites pass.

### The window in the Rust edge (built 2026-10-25)

`edge/src/indexer.rs`: the scanner and the file aggregator as before; then
the window instead of the range path. `Frontier` keeps a ring of the last
`FRONTIER_ARTICLES` (1,024) plausible dates (within `PLAUSIBLE_MS`, 7
days, of the frontier; 1,024 implausible dates in a row that agree move
it), recomputed once per chunk; established after 64 dated articles
(`settle` at the block's end for smaller blocks); its first value is the
block's start in posting time. Per chunk: completed files, then the files
whose last part is before frontier − `WINDOW_MS` (24 h; undated ones idle
for `idle` chunks), go into the open release of their key whose posting
span is within `UPLOAD_GAP_MS` (3 h) of theirs, else start one (the
upload number per key, distinct within a batch); releases whose last part
is before frontier − `WINDOW_MS` are judged and written `finished`, as an
`edge` when posted within `WINDOW_MS` of the block's start. At the end
(`finish`) everything left is written: finished when the window closed on
it, else `finished: false, edge: true`. A file short of parts stays in
its release and sets `partial` (the store: `complete = false`). A lone
article of a multi-part random file is still counted `per-article`, not
written. Gone: fragments and the cut-off, `uploads_of` (online now),
`check.ts` and the fixture comparison (the parity with the TypeScript
indexer; CI runs `edge/bench-fixtures.sh` instead), the `incomplete` count
(incomplete releases are kept). The wire carries `finished`, `edge` and
`partial`; the store reads them (`partial` into `complete`). Memory: the
window holds a day of a group's segments in memory (the busiest group's
~50 M); the spool file per block of the design is not built yet.

### The replay harness (built 2026-10-26)

`node ace replay:corpus --group <name> [--dir CORPUS_DIR] [--from article]
[--to article] [--limit ranges] [--batch-chunks 5] [--no-finish]` (commands/
replay_corpus.ts; design 8): a group's complete corpus ranges in article
order through one window in the Rust edge (`edge-bench --group <dir>
--rules rules.json --wire -`: the window carried from range to range, the
end finished after the last range; `edge/src/lease.rs` has the lease JSON
the worker and the bench share) and the batch store into a scratch
catalog, as the worker would store them: each range leased to a node named
`replay`, its batches stored at its resume point (pulled from the bench's
stdout as the store keeps up), the range finished when its last batch is
in. It refuses a database not named `*_replay` or `*_scratch`.
The rules are the scratch catalog's current set and group priorities:
copy `groups` and `rule_sets` from the catalog for a faithful replay. The
report: releases finished, provisional, edges; complete, incomplete,
undecided; readable, with par2; files, segments as a share of the
headers; bytes per header of the NVMe tables (releases, release_texts,
posters, group_sets, rule counts and samples) and of those on
CHUNK_TABLESPACE (release_chunks, probe_candidates, release_inspections).
The app image carries `edge-bench` for it (Dockerfile); on homelab the
scratch catalog is a second database (`usenet_replay`, default tablespace
`nvme` like the catalog, the chunk tables on `pg_default`; migrated with
the voyager-3 image built from the deploy cache's repo, `groups` and
`rule_sets` copied over), the corpus mounted read-only.
`deploy/replay-window.sh` runs a window there: each group whose window
ranges are all stored is replayed once with the window's bounds as
`--from`/`--to` (the corpus holds other windows of the same group), 4
groups at a time, logs and `.done` markers in `.cache/replay/`, looping
until the window is done (LOG.md 2026-10-26 (111) has the first local
numbers, (112) the homelab run). **Deployed 2026-10-26** (LOG.md (116)):
voyager 3 runs on homelab without its edge worker (`.cache/no-edge`:
deploy.sh and the nightly resume leave it stopped, since range by range
it would split releases at every edge with no merger left), and history
comes from the corpus through this driver into the catalog itself
(`DB=usenet_rebuild OUT=.cache/replay-catalog`, the harness allowed by
`REPLAY_INTO_CATALOG=1`), the prober's ladder, the merger and the judge
working on it as it fills; the scratch run's numbers (63 groups) were
read before (LOG.md (115)). The tail waits for the window worker.

Next: the replay on homelab over the 2016 window (the NVMe bytes per
header for real), the window worker (design 1 and 5: group leases, the
stream serving stored chunks, the snapshot through the API; being built,
LOG.md (117)), the prober writing `release_texts` kind 2, the spot/NZB
match on pending keys (decision f), the tail's provisional writes at a
range end (the window's open releases shown while they upload: the files
sent provisionally marked, the finish sending the rest). Gone from the list: block
leases, the planner and the block merger (design 1, amended).

## Decisions

a. Chunk payload: a write-once `bytea` table in Postgres, partitioned by
   month, on the HDD; files on disk only if writes still limit. **Accepted.**
b. A separate database on the same server. **Accepted.**
c. Group list: editable, owner by priority number (above). **Accepted.**
d. Reposts: separate releases in phase 1, merged in phase 2. **Accepted.**
e. Headers through the coordinator's relay, with prefetch (part 1).
   **Accepted.**
f. **Nothing enters the catalog without a usable name** (amended
   2026-10-26, the user's direction; before: random-named releases keep
   no chunk but their rows). A release the window closed on whose files
   all have random names is no release yet: it goes to `pending_releases`
   (migration 6300, the HDD) with its keys, counts, article span, upload
   tool and probe targets, no row in `releases`, `release_texts` or
   `posters`, no chunk. The pending ladder works them **from least to most
   effort**: (0) no fetch: a signature verdict (tool and group, poster)
   learned from earlier probes, a spot or NZB that names the message-ids;
   (1) a par2 main file's first article (it names the files); (2) the first
   article of the first file (the yEnc name, an archive's listing in its
   first 16 KiB); (3) a 7z's tail; then nothing more. A probe that yields
   a usable name moves the release into the catalog, its segments read
   again from the corpus (the range stored) or the server; one that
   yields none counts against its signature, and a signature with
   ≥100 probes and ≤0.5% named (voyager 2's garbage learner, LOG.md
   2026-10-08 (5)) is **discarded** from then on, a sample of it still
   probed to notice a change. The reason, in the 2016 replay: one poster's
   hash-named pieces ("Bob", 11 aliases, 3.2 M releases in five small
   groups) were 76% of the rows and three quarters of the NVMe bytes for
   11% of the segments and nothing anyone can find; the full history has
   725 M random-named files (LOG.md 2026-10-09). Built 2026-10-26: the
   edge sends an unnamed release with an empty payload and its tool, the
   batch store writes `pending_releases` (LOG.md (114)); the ladder in the
   prober (`app/services/pending_ladder.ts`, LOG.md (115)): a claim takes
   the oldest waiting releases' next target (par2, archive, lone, 7z-tail
   in that order), at least PENDING_SHARE (25%) of each claim next to the
   catalog's own candidates, answered from `probe_cache` when the article
   was read before; `usableName` (lib/probe/inspect.ts) takes the largest
   named file of a par2 set or a listing, or a lone file's yEnc name, that
   is neither packaging nor random; a name sets status 4, no name the next
   target or status 3 (exhausted); each finished climb counts on
   `pending_signatures` for (tool, group, poster) and (tool, group), and a
   signature with PENDING_MIN_PROBES (100) climbs and at most
   PENDING_MAX_NAMED (0.5%) named is dropped: every five minutes the
   waiting releases of dropped signatures are discarded (status 2), the
   rules' `samplePercent` of them left to climb. The namer
   (`app/services/pending_namer.ts`, in the prober's process): a named
   release's article span read again from the corpus (the stored range's
   chunks) or by XOVER on a connection of its own, through the TypeScript
   indexer, written as a release under the pending key with `source`
   weak-stem, the name as `inner_name`, its chunk and its candidates (the
   prober answers those from `probe_cache`: inspections, par2 sets, hashes,
   the password flag). Not built: the spot/NZB match on pending keys (step
   0 of the ladder beyond signatures), and a signature by poster alone
   across groups.
g. The account's NNTP connections shared by priority (`nntp_shares`): the
   coordinator (headers) before the prober, each between its minimum and
   maximum, 5 kept for the spotter, the spot inspector, the plain
   connection and reconnects. A process grows only into connections the
   others closed, so the account never goes over its limit. Replaces the
   fixed split (38 + 8), under which the prober stayed at 8 while history
   was paused and the coordinator's connections idle (LOG.md 2026-10-18
   (75)). **Accepted** 2026-10-18.
h. Idle NNTP connections closed: one with nothing to do for
   `NNTP_IDLE_CLOSE_SECONDS` (60) is closed and opened again with work
   (the coordinator's pool on its next sync, the plain one too; a dry
   prober lane; the spotter between rounds; the spot inspector with no
   spots). Eweka dropped the quiet ones and the next command failed
   ("Spot inspection failed: NNTP connection closed"), and idle ones took
   the account's connections from downloads. **Accepted** 2026-10-22.