Deploying voyager 3 without its edge worker: the no-edge marker, history replayed into the catalog

deploy.sh and the nightly resume leave the edge stopped while .cache/no-edge
exists (range by range it would split releases at every edge with no merger
left). deploy/replay-window.sh takes DB and OUT and passes
REPLAY_INTO_CATALOG=1, which lets replay:corpus write a database not named
*_replay or *_scratch (never the test one).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

.gitea/workflows/deploy.yaml

@@ -83,8 +83,8 @@ jobs:
           echo "$HOME/.cargo/bin" >> "$GITHUB_PATH"
       - name: cargo test
         run: cargo test --release --manifest-path edge/Cargo.toml
-      - name: Rust and TypeScript identical on the fixtures
-        run: edge/check-fixtures.sh
+      - name: The Rust indexer over the fixtures
+        run: edge/bench-fixtures.sh
 
   deploy:
     needs: [test, edge]

Dockerfile

@@ -8,6 +8,15 @@
 # validate commands) throws "Invalid URL" on $ref, so no ace command starts.
 ARG NODE_VERSION=24.16
 
+# The Rust worker (before the app: its image carries edge-bench). Registry and target dir are build caches, so a rebuild only compiles what changed.
+FROM rust:1-bookworm AS edge-build
+WORKDIR /src
+COPY edge ./
+RUN --mount=type=cache,target=/usr/local/cargo/registry \
+    --mount=type=cache,target=/src/target \
+    cargo build --release --locked --bin edge-node --bin edge-bench \
+ && cp target/release/edge-node target/release/edge-bench /usr/local/bin/
+
 # The app, compiled, with production dependencies only.
 FROM node:${NODE_VERSION}-bookworm AS app-build
 WORKDIR /src
@@ -22,19 +31,12 @@ FROM node:${NODE_VERSION}-bookworm-slim AS app
 ENV NODE_ENV=production
 WORKDIR /app
 COPY --from=app-build /src/build ./
+# The Rust indexer's bench: `node ace replay:corpus` runs a group's corpus through it (the replay harness).
+COPY --from=edge-build /usr/local/bin/edge-bench /usr/local/bin/edge-bench
 USER node
 # Every service picks its own command, e.g. `node ace coordinator:run`.
 CMD ["node", "ace", "list"]
 
-# The Rust worker. Registry and target dir are build caches, so a rebuild only compiles what changed.
-FROM rust:1-bookworm AS edge-build
-WORKDIR /src
-COPY edge ./
-RUN --mount=type=cache,target=/usr/local/cargo/registry \
-    --mount=type=cache,target=/src/target \
-    cargo build --release --locked --bin edge-node --bin edge-bench \
- && cp target/release/edge-node target/release/edge-bench /usr/local/bin/
-
 FROM debian:bookworm-slim AS edge
 COPY --from=edge-build /usr/local/bin/edge-node /usr/local/bin/edge-node
 # edge-bench replays the corpus for the grouping sweeps (deploy/corpus_sweep.py).

REBUILD.md

@@ -5,7 +5,8 @@ rebuild, then a draft for approval). Voyager 1's CONCEPT.md and its
 decisions D1–D42 are superseded; its domain model still holds, and its
 features not yet here are the phase 2 backlog below.
 
-Branch `rebuild`. Phase 1 finds releases from headers and makes their NZBs.
+Branch `rebuild`; its successor is designed under *Voyager 3* below (branch
+`voyager-3`). Phase 1 finds releases from headers and makes their NZBs.
 Everything learned from articles (names from yEnc headers, par2, archive
 listings, encryption, spots) is phase 2 and adds to what phase 1 stored; it
 never rewrites it. Sources: CONCEPT.md, LOG.md 2026-10-08 (13)–(14).
@@ -371,6 +372,340 @@ processes, one NNTP connection each:
   `spot-text`, and the spot), not on the catalog row; the catalog only says
   `passworded`.
 
+## Voyager 3: window indexing (design, 2026-10-25)
+
+Branch `voyager-3` (a push to it only runs the tests; `rebuild` keeps
+running on homelab until the reset). Goal: the whole history (~1,220 B
+articles, 3% indexed on 2026-10-25) indexed and searchable on a 4 TB NVMe
+and the 15 TB HDD. The catalog costs 3.9 B of NVMe per article today
+(144 GB for 36.5 B), which puts the full history at ~4.7 TB; the target is
+≤ 2.5 B per article (~3 TB) with room for the WAL and bloat. Sources:
+LOG.md 2026-10-24 (102).
+
+What is wrong today: a release is written as soon as one range sees it,
+updated by every later range that finds more parts, merged afterwards
+(fragments, par2, names) and settled by wall-clock quiet time, which means
+little in a backfill. That churn is most of the WAL (0.6 TB a day) and the
+20 k write IOPS bursts, and leaves bloat. Half the rows are single-article
+releases, a third of them one poster's image dump (27.7 M rows, LOG.md
+(102)).
+
+1. **One cursor a group (amended 2026-10-26; was: blocks).** A group's
+   history is still fetched as ranges of 2 M articles in chunks of 100 k
+   (the spool, the corpus, the prefetcher stay) and leased as ranges, but
+   a group's ranges are indexed in article order by one worker at a time,
+   and the window below is *carried* from range to range: the next range
+   finishes what the previous one left open. The first design leased
+   blocks of 30 days of posting time, ran a group's blocks in parallel and
+   joined their edges afterwards (a block merger). The user asked why
+   (LOG.md 2026-10-25 (110)): downloading stays parallel through the
+   prefetcher and a Rust thread parses faster than a connection delivers,
+   so one worker a group loses nothing but the end of a reset, when the
+   last big group runs alone. The merger, the block cutting and the
+   `blocks` lease go. Costs: one seam a group where the history meets the
+   tail (the releases in flight then end as two incomplete halves; history
+   runs forward in one pass a group, priority only between groups), and
+   the carried state must survive a restart (5 below). Measured on six
+   consecutive ranges of a.b.ath (2016): one range is 15 hours of posting,
+   so without the carry no release would finish inside a range (LOG.md
+   2026-10-26 (111)).
+
+2. **The window.** The worker reads a block in article order and keeps its
+   open files and releases in memory (counters, names, sizes; the segment
+   lists spooled to a file per block on the worker's disk). The *frontier*
+   is the median Date of the last 1,024 articles with a plausible date
+   (within 7 days of the frontier; 1,024 such dates in a row that agree
+   among themselves move it, the group having paused); Date headers alone
+   aren't trusted (clock skew, bogus years). 1,024, not more: the median
+   trails the newest article by half the ring, a day in a group of 500
+   articles a day. A release is **finished** when the frontier is 24 h past
+   its last part (`WINDOW_MS`): written once, with its completeness decided
+   then (every announced file there and no file short of parts; par2
+   decides later for those without a counter), never updated by the header
+   pass again. Incomplete releases are kept (user, 2026-10-24). A fill
+   posted after the window is a release of its own (the judge's `repeated`
+   rule already knows reposts). At a lease's end (the tail's poll, a worker
+   stopping; amended 2026-10-26, was: the block's end) the releases the
+   window hasn't closed on are written **provisionally** (`settled_at`
+   null, `finished: false`, flagged `edge`): a release being uploaded
+   shows in the catalog, and the carried window finishes the same row
+   later, since the release key is the identity, the group and the first
+   article; the store's finish path adds the files not sent before. No
+   merger: the carry does the joining. The settler goes: `settled_at` is
+   the finish. The **tail** needs no path of its own: its lease ends leave
+   such provisional releases, the next lease's continuation finishes them.
+
+3. **Albums.** Files of one poster in one owner group whose subjects differ
+   only by a number sequence, or share a directory-like prefix, posted
+   within the window, become one release (`source = 'album'`): its name the
+   shared template (or the prefix), its files kept by name so each stays
+   findable; at least two files, else the single stays a release. This is
+   for subjects that name no release (today's `post`/`stem` singles); a
+   poster's ten different cracks in a day stay ten releases. The stake,
+   from voyager 2's final analysis (LOG.md (112)): 77% of its 83.7 M
+   releases had one file, 50% one segment (0.3% of the segments);
+   a.b.pictures.erotica.porn alone was 27.7 M single-file releases of one
+   poster. A counter total shared by many stems of one poster within the
+   window is a batch counter, not a file count (Yenc-PP-A&A's [n/68067]):
+   `file_total` null for them, not 68,067 incomplete singles.
+   **Built 2026-10-26** (edge/src/indexer.rs, LOG.md (113)): a release the
+   window closed on with one file named by its stem or post, the name with
+   a number and no archive, volume, sidecar or random name, waits in the
+   album of its poster and template (the name's digit runs as `#`, the
+   extension apart: `#_#.jpg.data`) posted within WINDOW_MS of the album's
+   first file, at most ALBUM_MAX_FILES (1,000) a album; the album closes a
+   window after that span and is written as one release (`source: album`,
+   no file total) named by the members' shared numbers (`1481439079025_#
+   .jpg.data`), a lone member as the release it was. The batch store writes
+   the members' names as `release_texts` kind 6 (`member`, the first 20).
+   The directory-like prefix variant is not built (no case seen in the
+   corpus samples). The batch counter: a total ≥ 100 under which one poster
+   started 20+ releases within 100 batches gives none of them a file
+   total. On six a.b.ath ranges of 2016: 46 albums of 790 files among
+   2,099 chunks, most of them a subject parser miss (`[n/60] Shogun I
+   (1980) Shogun I (1980).avi.012`: the counter text taken into the name,
+   a volume each) that the album rule glues back together.
+
+4. **A slimmer catalog.** `releases` loses its text copies: `poster_id` →
+   `posters(id)` (the table gets an id; 33 B → 4), `group_set_id` →
+   `group_sets` (the distinct group combinations: 139,927 in voyager 2's
+   83.7 M releases, LOG.md (112); 85 B jsonb → 4),
+   `release_key` and `identity_key` as 20-byte `bytea` (41 B each → 20),
+   `source` a smallint. ~150 of ~430 B a row; the poster and key indexes
+   shrink with them (~12 GB of 50 today); `releases_category_newest_idx`
+   (5.3 GB, used once) goes, category browsing filters the newest index.
+   Today's 85 GB would be ~55. Cold tables move to the HDD tablespace:
+   `release_inspections` and `probe_candidates` (28 GB, written once, not
+   read by search); `probe_cache` is there already.
+
+5. **Memory and disk at the edge.** The busiest group posts ~50 M articles
+   a day; at a file per ~50 articles that is ~1 M open files of ~100 B in
+   memory per block, and their segment lists (~25 B each) in the block's
+   spool file, read back when a release is finished. Per worker thread a
+   few hundred MB at worst. The carried window (amended 2026-10-26) has to
+   survive a restart (deploys restart the workers daily): its counters and
+   names are small (kilobytes for most groups, ~100 MB for the busiest),
+   the open files' segment lists are the bulk (~1.25 GB for a day of the
+   busiest group). A snapshot at every 2 M range would be ~600 B/article
+   of writes, so the state is snapshotted at a lease's end and on a timer,
+   to a file on bulk, and a crash redoes the ranges since the last snapshot
+   from the corpus or the prefetch cache: the deterministic keys make the
+   redo idempotent. Not built yet; the replay harness (8) keeps a group's
+   window in one process.
+
+6. **The corpus stays as it is.** Measured on 300 unfiltered chunks
+   (30 M lines, LOG.md (102)): 43% of lines lack a "(k/N)" counter, and of
+   those 73% are random 32-hex tokens, 27% `[PRiVATE]` hashes, ~0.3% posts
+   with a readable name and no counter. The filter (crosspost, counterless)
+   loses nothing voyager 3 would index; albums work from filtered chunks
+   (the image dump's subjects carry "(1/1)"). Indexing without a counter at
+   all (obfuscated uploads, ~31% of all lines) would need bodies, and the
+   raw corpus (2,328 ranges) for a sweep.
+
+7. **Where it lives.** The window, blocks and albums are in the Rust edge
+   (`edge/`), the batch store, the planner and leases; the TypeScript
+   RangeIndexer keeps the scanner (the corpus filter, tests) and drops its
+   parity with the Rust worker for the window (the `check.ts` comparison
+   covered the range path, which goes). New migrations rebuild the schema
+   from a reset, no in-place conversion of the 81 M rows.
+
+8. **Test before the reset.** Replay the corpus (the raw ranges and the
+   captures, several B articles) with the new edge into a scratch database
+   on bulk: NVMe bytes per article (the one number that decides whether
+   4 TB is enough), the share of albums and incomplete releases, and a
+   sample of releases against today's. Speed is measured after the reset,
+   on the NVMe.
+
+9. **The reset.** Tail first, then history by priority, read from the
+   corpus where stored. Survives: `probe_cache` (HDD, by message-id),
+   `spots`, `groups` and their priorities, `nntp_shares`, the corpus.
+   Recomputed: releases, chunks, inspections, candidates, par2 data,
+   verdicts, file hashes. Resets are paused (2026-10-12); this one waits
+   for the user's go-ahead, and for the new NVMe.
+
+### Schema (built 2026-10-25; migrations 6000 and 6100)
+
+Migration 6000 *is* the reset: it truncates every table the indexing
+recomputes and refuses to run on a catalog with releases unless
+`VOYAGER3_RESET=1`, so merging `voyager-3` into `rebuild` is the go-ahead.
+Survives: `groups`, `ranges` (planned again from their start, with their
+priorities, `capture`, `est_date`, `stored_at`, `plain_chunks` and measured
+`posted_low/high`), the corpus, `probe_cache`, `spots` (unlinked from
+releases; the inspector matches them again), `nntp_shares`, `nodes`,
+`users`, the rule sets.
+
+- `releases`: created anew, 8-byte columns first: `id`, `posted_at`,
+  `last_posted_at`, `first_article`, `last_article`, `segments`, `bytes`,
+  `settled_at` (the finish; null while a tail release is provisional),
+  `inspected_at`, `first_seen_at`; `group_id`, `group_set_id`, `poster_id`,
+  `file_total`, `files`, `chunks`, `category`; `source smallint`
+  (`lib/release/source.ts`: 1 subject, 2 post, 3 stem, 4 spot, 5 nzb, 6
+  album); `readable`, `has_par2`, `complete`, `passworded`, `sidecar_only`,
+  `edge` (open at its block's end); `release_key` and `identity_key` as
+  20-byte `bytea`; `name`, `inner_name`. Gone: `poster`, `groups`,
+  `last_seen_at`. Indexes: pkey, release_key unique, (identity_key,
+  group_id), newest, (poster_id, newest), passworded, `(group_id,
+  identity_key) WHERE edge` for the block merger, `id WHERE settled_at IS
+  NULL` for the provisional ones. No trigram indexes: search matches
+  `release_texts`. The category trigger reads the groups from
+  `group_sets`; the judge's dirty trigger watches `poster_id` instead of
+  `poster`.
+- `pending_releases` (migrations 6300 and 6400, the HDD): the unnamed
+  releases outside the catalog (decision f): `release_key` (primary),
+  `identity_key`, `group_id`, `name` (the random stem), `poster` text, the
+  posting span, `first_article`, `last_article`, counts, `file_total`,
+  `partial`, `finished`, `tool`, `candidates` jsonb (the probe targets),
+  `inspections` jsonb (what each probe showed), `named`, `status` (0
+  waiting, 1 named and in the catalog, 2 discarded, 3 exhausted, 4 named
+  awaiting the move, 9 taken), `tries`, `created_at`, `probed_at`; partial
+  indexes on the waiting (by `created_at`) and the named (by `probed_at`).
+- `pending_signatures` (6400): per (tool, group, poster; poster '' for the
+  tool and group together) `probed`, `named`, `dropped`, `updated_at`.
+- `release_texts(release_id, kind smallint, text varchar(255))` (6200),
+  primary key on all three, one trigram index (fastupdate): every string
+  search matches against, appended by whoever learns one and never an
+  update of the releases row. Kinds (`lib/release/text_kind.ts`): 1 name
+  (the release name, stored once more so one index serves everything),
+  2 file (the prober's names inside an archive), 3 nfo (an NFO's title),
+  4 par2 (a set's name), 5 media (a container's title); at most 20 per
+  release and kind (`textsToStore`). A repeat insert is a no-op; the
+  judge's dirty mark follows an insert or delete, so a rule can react to a
+  name arriving later. Search: the trigram index yields release ids, joined
+  to `releases` by primary key (never an OR across the two tables, which
+  the planner cannot serve from indexes). `inner_name` stays on `releases`
+  as the one display string. Facets to filter on (resolution, codec,
+  language) are not texts: columns, or a `meta jsonb` with a
+  `jsonb_path_ops` index once one is proven useful.
+- `posters(id serial, poster unique)` with its trigram index;
+  `group_sets(id serial, groups jsonb unique)`, a sorted array of names.
+- `blocks` (6100): `group_id`, `start_id`, `end_id`, `next_id` (the resume
+  point), `posted_low/high`, `status`, `priority`, `node_id`, `attempts`,
+  `rule_version`, `headers`, `releases`, `leased_at`, `heartbeat_at`,
+  `done_at`, `error`; indexes todo (priority, id) and leased.
+  `ranges.block_id` points at the block; the ranges' own lease columns stay
+  until the leases move (then a migration drops them).
+- `release_chunks`, `rule_counts`, `rule_samples`: `range_id` → `block_id`.
+- Gone: `fragments`, `range_blocks`, `release_final()` (final = finished:
+  `settled_at IS NOT NULL`), the settler, the fragment merger,
+  `releases_category_newest_idx`.
+- `release_inspections` and `probe_candidates` on `CHUNK_TABLESPACE` (the
+  HDD) with their indexes. Note LOG.md 2026-10-19 (81): the judge's fake
+  rule read all of `release_inspections` every round, 17 min on the HDD;
+  the judge must read it by release or keep what it needs on the NVMe.
+
+### Batch store (built 2026-10-25)
+
+`app/services/batch_store.ts`: every release chunk of a batch is one
+release, inserted once. Its keys (`lib/release/keys.ts`): `identity_key`
+the identity's SHA-1 as 20 bytes; `release_key` the SHA-1 of the identity,
+the owner group and the release's first article, so it is deterministic
+(no numbering under a group lock, no catalog lookups before writing) and a
+repost, starting at another article, is a release of its own. Its poster
+and group set by id (`app/services/catalog_refs.ts`: insert what's new,
+read the ids, cache them per process; the tests forget the cache per
+test), its source as a code, its name into `release_texts` (kind 1). The
+chunk's own counts go in as the release's; `complete` is decided then from
+the counter. A finished release's probe candidates are claimable at once
+(`ready`, `unreadable`); a provisional tail release (`finished: false` on
+the chunk, `settled_at` null) is finished by a later chunk of the same
+release key, which adds the files not sent before (sums, one more chunk),
+sets `settled_at` and makes its candidates claimable; a finished release
+sent again is left as it is (`ON CONFLICT … WHERE settled_at IS NULL`).
+Gone: the join of a chunk into an earlier release of its key (the window
+keeps an upload together; the block merger joins what a block's end cut),
+the join across groups (FTD's spread uploads: a gap, to be handled by the
+block merger or the par2 merger across groups), the numbering of release
+keys, the group lock, `fragments` (ignored when sent; the Rust edge still
+sends them). The chunk type carries `finished` and `edge` (both optional;
+the Rust edge doesn't send them yet, so its releases are finished, not
+edges). Interim until blocks are leased: chunks, rule counts and samples
+are keyed by `ranges.block_id`, and a range without a block gets a block
+of its own in the store (`blockOf`).
+
+Readers moved with it: search joins `posters` and `group_sets`, matches
+words in `release_texts` (`r.id IN (SELECT release_id … ILIKE)`), searches
+posters by id, filters `source` by code and returns the source name; the
+par2 merger, the sidecar folder and the judge's scraped rule use
+`poster_id` (`release_merges.from_key` holds the key as hex); the spot
+inspector matches by bytea identity keys and inserts its spot releases by
+id references with their title in `release_texts`; the spotter and the NZB
+service compare the source code. Tests insert fixtures through
+`tests/functional/helpers/releases.ts` (`insertRelease`: text keys, poster
+and groups by name, source by name). The unit and functional suites pass.
+
+### The window in the Rust edge (built 2026-10-25)
+
+`edge/src/indexer.rs`: the scanner and the file aggregator as before; then
+the window instead of the range path. `Frontier` keeps a ring of the last
+`FRONTIER_ARTICLES` (1,024) plausible dates (within `PLAUSIBLE_MS`, 7
+days, of the frontier; 1,024 implausible dates in a row that agree move
+it), recomputed once per chunk; established after 64 dated articles
+(`settle` at the block's end for smaller blocks); its first value is the
+block's start in posting time. Per chunk: completed files, then the files
+whose last part is before frontier − `WINDOW_MS` (24 h; undated ones idle
+for `idle` chunks), go into the open release of their key whose posting
+span is within `UPLOAD_GAP_MS` (3 h) of theirs, else start one (the
+upload number per key, distinct within a batch); releases whose last part
+is before frontier − `WINDOW_MS` are judged and written `finished`, as an
+`edge` when posted within `WINDOW_MS` of the block's start. At the end
+(`finish`) everything left is written: finished when the window closed on
+it, else `finished: false, edge: true`. A file short of parts stays in
+its release and sets `partial` (the store: `complete = false`). A lone
+article of a multi-part random file is still counted `per-article`, not
+written. Gone: fragments and the cut-off, `uploads_of` (online now),
+`check.ts` and the fixture comparison (the parity with the TypeScript
+indexer; CI runs `edge/bench-fixtures.sh` instead), the `incomplete` count
+(incomplete releases are kept). The wire carries `finished`, `edge` and
+`partial`; the store reads them (`partial` into `complete`). Memory: the
+window holds a day of a group's segments in memory (the busiest group's
+~50 M); the spool file per block of the design is not built yet.
+
+### The replay harness (built 2026-10-26)
+
+`node ace replay:corpus --group <name> [--dir CORPUS_DIR] [--from article]
+[--to article] [--limit ranges] [--batch-chunks 5] [--no-finish]` (commands/
+replay_corpus.ts; design 8): a group's complete corpus ranges in article
+order through one window in the Rust edge (`edge-bench --group <dir>
+--rules rules.json --wire -`: the window carried from range to range, the
+end finished after the last range; `edge/src/lease.rs` has the lease JSON
+the worker and the bench share) and the batch store into a scratch
+catalog, as the worker would store them: each range leased to a node named
+`replay`, its batches stored at its resume point (pulled from the bench's
+stdout as the store keeps up), the range finished when its last batch is
+in. It refuses a database not named `*_replay` or `*_scratch`.
+The rules are the scratch catalog's current set and group priorities:
+copy `groups` and `rule_sets` from the catalog for a faithful replay. The
+report: releases finished, provisional, edges; complete, incomplete,
+undecided; readable, with par2; files, segments as a share of the
+headers; bytes per header of the NVMe tables (releases, release_texts,
+posters, group_sets, rule counts and samples) and of those on
+CHUNK_TABLESPACE (release_chunks, probe_candidates, release_inspections).
+The app image carries `edge-bench` for it (Dockerfile); on homelab the
+scratch catalog is a second database (`usenet_replay`, default tablespace
+`nvme` like the catalog, the chunk tables on `pg_default`; migrated with
+the voyager-3 image built from the deploy cache's repo, `groups` and
+`rule_sets` copied over), the corpus mounted read-only.
+`deploy/replay-window.sh` runs a window there: each group whose window
+ranges are all stored is replayed once with the window's bounds as
+`--from`/`--to` (the corpus holds other windows of the same group), 4
+groups at a time, logs and `.done` markers in `.cache/replay/`, looping
+until the window is done (LOG.md 2026-10-26 (111) has the first local
+numbers, (112) the homelab run). **Deployed 2026-10-26** (LOG.md (116)):
+voyager 3 runs on homelab without its edge worker (`.cache/no-edge`:
+deploy.sh and the nightly resume leave it stopped, since range by range
+it would split releases at every edge with no merger left), and history
+comes from the corpus through this driver into the catalog itself
+(`DB=usenet_rebuild OUT=.cache/replay-catalog`, the harness allowed by
+`REPLAY_INTO_CATALOG=1`), the prober's ladder, the merger and the judge
+working on it as it fills; the scratch run's numbers (63 groups) were
+read before (LOG.md (115)). The tail waits for the window worker.
+
+Next: the replay on homelab over the 2016 window (the NVMe bytes per
+header for real), the ordered range leases with the carried window in
+the edge-node and its snapshot (design 1 and 5), the prober writing
+`release_texts` kind 2, the spot/NZB match on pending keys (decision f). Gone from the list: block
+leases, the planner and the block merger (design 1, amended).
+
 ## Decisions
 
 a. Chunk payload: a write-once `bytea` table in Postgres, partitioned by
@@ -380,11 +715,52 @@ c. Group list: editable, owner by priority number (above). **Accepted.**
 d. Reposts: separate releases in phase 1, merged in phase 2. **Accepted.**
 e. Headers through the coordinator's relay, with prefetch (part 1).
    **Accepted.**
-f. Random-named releases (no file with a readable name) keep no chunk in
-   the header pass: their file keys and probe candidates only, until
-   phase 2 names them (bodies) and fetches their segments again. The full
-   history is ~3 TB with them, ~1.2–1.4 TB without (725 M of the 1.25 B
-   kept files; LOG.md 2026-10-09). **Accepted** 2026-10-09, not built yet.
+f. **Nothing enters the catalog without a usable name** (amended
+   2026-10-26, the user's direction; before: random-named releases keep
+   no chunk but their rows). A release the window closed on whose files
+   all have random names is no release yet: it goes to `pending_releases`
+   (migration 6300, the HDD) with its keys, counts, article span, upload
+   tool and probe targets, no row in `releases`, `release_texts` or
+   `posters`, no chunk. The pending ladder works them **from least to most
+   effort**: (0) no fetch: a signature verdict (tool and group, poster)
+   learned from earlier probes, a spot or NZB that names the message-ids;
+   (1) a par2 main file's first article (it names the files); (2) the first
+   article of the first file (the yEnc name, an archive's listing in its
+   first 16 KiB); (3) a 7z's tail; then nothing more. A probe that yields
+   a usable name moves the release into the catalog, its segments read
+   again from the corpus (the range stored) or the server; one that
+   yields none counts against its signature, and a signature with
+   ≥100 probes and ≤0.5% named (voyager 2's garbage learner, LOG.md
+   2026-10-08 (5)) is **discarded** from then on, a sample of it still
+   probed to notice a change. The reason, in the 2016 replay: one poster's
+   hash-named pieces ("Bob", 11 aliases, 3.2 M releases in five small
+   groups) were 76% of the rows and three quarters of the NVMe bytes for
+   11% of the segments and nothing anyone can find; the full history has
+   725 M random-named files (LOG.md 2026-10-09). Built 2026-10-26: the
+   edge sends an unnamed release with an empty payload and its tool, the
+   batch store writes `pending_releases` (LOG.md (114)); the ladder in the
+   prober (`app/services/pending_ladder.ts`, LOG.md (115)): a claim takes
+   the oldest waiting releases' next target (par2, archive, lone, 7z-tail
+   in that order), at least PENDING_SHARE (25%) of each claim next to the
+   catalog's own candidates, answered from `probe_cache` when the article
+   was read before; `usableName` (lib/probe/inspect.ts) takes the largest
+   named file of a par2 set or a listing, or a lone file's yEnc name, that
+   is neither packaging nor random; a name sets status 4, no name the next
+   target or status 3 (exhausted); each finished climb counts on
+   `pending_signatures` for (tool, group, poster) and (tool, group), and a
+   signature with PENDING_MIN_PROBES (100) climbs and at most
+   PENDING_MAX_NAMED (0.5%) named is dropped: every five minutes the
+   waiting releases of dropped signatures are discarded (status 2), the
+   rules' `samplePercent` of them left to climb. The namer
+   (`app/services/pending_namer.ts`, in the prober's process): a named
+   release's article span read again from the corpus (the stored range's
+   chunks) or by XOVER on a connection of its own, through the TypeScript
+   indexer, written as a release under the pending key with `source`
+   weak-stem, the name as `inner_name`, its chunk and its candidates (the
+   prober answers those from `probe_cache`: inspections, par2 sets, hashes,
+   the password flag). Not built: the spot/NZB match on pending keys (step
+   0 of the ladder beyond signatures), and a signature by poster alone
+   across groups.
 g. The account's NNTP connections shared by priority (`nntp_shares`): the
    coordinator (headers) before the prober, each between its minimum and
    maximum, 5 kept for the spotter, the spot inspector, the plain

app/services/batch_store.ts

@@ -1,14 +1,19 @@
 import db from '@adonisjs/lucid/services/db'
 import { pruneSevenZipTails } from '#services/seven_zip_tails'
 import type { TransactionClientContract } from '@adonisjs/lucid/types/database'
-import { UPLOAD_GAP_MS, type EdgeBatch, type ProbeCandidate, type ReleaseChunk } from '#lib/edge/index'
+import type { EdgeBatch, ProbeCandidate, ReleaseChunk } from '#lib/edge/index'
 import ChunkPartitions from '#services/chunk_partitions'
+import CatalogRefs, { groupSetKey, posterText } from '#services/catalog_refs'
 import { decodeChunk } from '#lib/chunks/index'
 import { SIDECAR } from '#services/sidecar_folder'
+import { identityKey, releaseKey } from '#lib/release/keys'
+import { sourceCode } from '#lib/release/source'
+import { uploadTool } from '#lib/rules/index'
+import { TEXT_KINDS, TEXT_MAX, textsToStore } from '#lib/release/text_kind'
 
 /**
  * A small new release (at most 3 files, under 1 MB) whose files are all sidecars (.nfo, .sfv, .nzb, …): the sidecar
- * fold's candidates (releases.sidecar_only; the settler queues only these). Its file list decoded here, small as it is.
+ * fold's candidates (releases.sidecar_only). Its file list decoded here, small as it is.
  */
 function sidecarOnly(r: ReleaseChunk): boolean | null {
   if (r.files > 3 || r.bytes >= 1_000_000) return null
@@ -19,18 +24,34 @@ function sidecarOnly(r: ReleaseChunk): boolean | null {
 export class StaleBatch extends Error {}
 
 export interface StoredBatch {
+  /** Release chunks in the batch. */
   releases: number
+  /** Releases the batch started. */
   newReleases: number
+  /** Provisional (tail) releases the batch finished. */
+  finalized: number
   candidates: number
-  fragments: number
+  /** Unnamed releases written to the pending store instead of the catalog (decision f). */
+  pending: number
 }
 
+/** A chunk's slot in its batch: the key and the upload the edge numbered it with (ProbeCandidate points at it). */
+const slot = (key: string, upload: number) => `${key}\n${upload}`
+
 /**
- * Stores one batch of a leased range (REBUILD.md, parts 3–7), in one
- * transaction: release aggregates in the catalog, one chunk per release,
- * probe candidates, rule counts and samples, edge fragments, and the
- * range's resume point. A batch that doesn't start at the resume point was
+ * Stores one batch of a leased range (REBUILD.md, "Voyager 3: window indexing", 2 and 4) in one transaction: each
+ * release chunk is a whole release the edge's window finished, written once (its poster and group set by id, its
+ * keys as bytea, its name into release_texts), with one chunk of its files, its probe candidates, and the batch's
+ * rule counts and samples; then the range's resume point moves. A batch that doesn't start at the resume point was
  * sent twice or too late, and is refused (StaleBatch), so the sums stay right.
+ *
+ * A tail release comes twice: provisionally (finished: false, settled_at null, its candidates waiting) and then
+ * finished, under the same release key (the identity, the group and its first article: lib/release/keys.ts), with
+ * the files not sent before (a chunk's counts are its own; the finish adds them up); the finish is the one update a
+ * release gets from the header pass. A finished release sent again is left as it is.
+ *
+ * Interim, until blocks are leased (the next step): chunks, counts and samples are keyed by the range's block
+ * (ranges.block_id), and a range without one gets a block of its own here.
  */
 class BatchStore {
   async store(rangeId: number, nodeId: number, batch: EdgeBatch): Promise<StoredBatch> {
@@ -39,7 +60,9 @@ class BatchStore {
       // One batch of a range at a time, without holding the range's row for the whole transaction: the
       // prefetcher's and heartbeats' updates of it waited on it (0.5 s on average, NNTP connections idle;
       // LOG.md 2026-10-09 (13)). The resume point is checked again when it moves, at the end.
-      await trx.rawQuery(`SELECT pg_advisory_xact_lock(hashtextextended('range:' || ?::text, 0))`, [rangeId])
+      await trx.rawQuery(`SELECT pg_advisory_xact_lock(hashtextextended('range:' || ?::text, 0))`, [
+        rangeId,
+      ])
       const range = await trx.from('ranges').where('id', rangeId).first()
       if (!range || range.status !== 'leased' || Number(range.node_id) !== nodeId)
         throw new StaleBatch(`Range ${rangeId} is not leased to this node`)
@@ -47,35 +70,48 @@ class BatchStore {
         throw new StaleBatch(
           `Range ${rangeId} is at ${range.next_id}, the batch starts at ${batch.from}`
         )
+      const blockId = await this.blockOf(trx, range)
       const result: StoredBatch = {
         releases: batch.releases.length,
         newReleases: 0,
+        finalized: 0,
         candidates: 0,
-        fragments: batch.fragments.length,
+        pending: 0,
       }
 
+      // Nothing enters the catalog without a usable name (decision f): an unnamed release (no readable file name)
+      // goes to the pending store with its probe targets, the named ones to the catalog.
+      const named = batch.releases.filter((r) => r.readable)
+      const unnamed = batch.releases.filter((r) => !r.readable)
       const written = await this.writeReleases(
         trx,
-        rangeId,
+        { groupId: Number(range.group_id), blockId },
         batch.from,
-        batch.releases,
+        named,
         batch.candidates
       )
       result.newReleases = written.newReleases
+      result.finalized = written.finalized
       result.candidates = written.candidates
+      result.pending = await this.writePending(
+        trx,
+        Number(range.group_id),
+        unnamed,
+        batch.candidates
+      )
 
       if (batch.counts.length > 0) {
         await trx.rawQuery(
-          `INSERT INTO rule_counts (range_id, rule, key, articles, files)
+          `INSERT INTO rule_counts (block_id, rule, key, articles, files)
            SELECT ?, v.rule, v.key, v.articles, v.files FROM jsonb_to_recordset(?::jsonb) AS v(rule text, key text, articles bigint, files bigint)
-           ON CONFLICT (range_id, rule, key) DO UPDATE SET articles = rule_counts.articles + excluded.articles, files = rule_counts.files + excluded.files`,
-          [rangeId, JSON.stringify(batch.counts)]
+           ON CONFLICT (block_id, rule, key) DO UPDATE SET articles = rule_counts.articles + excluded.articles, files = rule_counts.files + excluded.files`,
+          [blockId, JSON.stringify(batch.counts)]
         )
       }
 
       for (const s of batch.samples) {
         await trx.insertQuery().table('rule_samples').insert({
-          range_id: rangeId,
+          block_id: blockId,
           rule: s.rule,
           rule_key: s.ruleKey,
           release_key: s.key,
@@ -86,18 +122,6 @@ class BatchStore {
         })
       }
 
-      for (let i = 0; i < batch.fragments.length; i += 2000) {
-        const rows = batch.fragments.slice(i, i + 2000).map((f) => ({
-          range_id: rangeId,
-          file_key: f.fileKey,
-          // Not Math.min(...): a big file's fragment has more segments than the call stack takes arguments.
-          article_number: f.segments.reduce((min, s) => (s[3] < min ? s[3] : min), Infinity),
-          segments: f.segments.length,
-          fragment: JSON.stringify(f),
-        }))
-        await trx.insertQuery().table('fragments').multiInsert(rows)
-      }
-
       const moved = await trx
         .from('ranges')
         .where('id', rangeId)
@@ -107,106 +131,138 @@ class BatchStore {
         .update({
           next_id: batch.next,
           headers: db.raw('headers + ?', [batch.headers]),
-          // The range's posting time span (release_final): the batches' trimmed spans, min and max (least and greatest skip nulls).
-          posted_low: db.raw('least(posted_low, to_timestamp(?::double precision / 1000))', [batch.postedLow ?? null]),
-          posted_high: db.raw('greatest(posted_high, to_timestamp(?::double precision / 1000))', [batch.postedHigh ?? null]),
+          // The range's posting time span: the batches' trimmed spans, min and max (least and greatest skip nulls).
+          posted_low: db.raw('least(posted_low, to_timestamp(?::double precision / 1000))', [
+            batch.postedLow ?? null,
+          ]),
+          posted_high: db.raw('greatest(posted_high, to_timestamp(?::double precision / 1000))', [
+            batch.postedHigh ?? null,
+          ]),
           heartbeat_at: new Date(),
         })
-      if (Number(moved) !== 1 && !(Array.isArray(moved) && moved.length === 1)) throw new StaleBatch(`Range ${rangeId} moved on meanwhile`)
+      if (Number(moved) !== 1 && !(Array.isArray(moved) && moved.length === 1))
+        throw new StaleBatch(`Range ${rangeId} moved on meanwhile`)
       return result
     })
   }
 
-  /**
-   * An upload spread over groups (FTD: par2 and part1 to a.b.music.misc,
-   * part2 to a.b.sounds.music, each article in one group only; LOG.md
-   * 2026-10-11 (9)): a chunk without a release in its own group joins one of
-   * its key in another group whose window is within UPLOAD_GAP_MS, when none
-   * of its file names are in that release yet. A post of the same files to
-   * another group (other message-ids, the same names) stays a release of its
-   * own, or the NZB would hold every file twice. Reads the candidate's file
-   * names from its chunks; candidates are few.
-   */
-  private async joinAcrossGroups(
-    trx: TransactionClientContract,
-    part: ReleaseChunk[],
-    windows: { idx: number; key: string; first: number | null | undefined; last: number | null | undefined }[],
-    groupId: number,
-    joined: Map<number, number>
-  ) {
-    const open = windows.filter((w) => !joined.has(w.idx) && w.first)
-    if (open.length === 0) return
-    const { rows: candidates } = await trx.rawQuery(
-      `SELECT v.idx, r.id
-       FROM jsonb_to_recordset(?::jsonb) AS v(idx int, key text, first bigint, last bigint)
-       JOIN releases r ON r.identity_key = v.key AND r.group_id <> ?
-       WHERE r.posted_at <= to_timestamp((v.last + ?) / 1000.0) AND coalesce(r.last_posted_at, r.posted_at) >= to_timestamp((v.first - ?) / 1000.0)
-       ORDER BY v.idx, r.last_posted_at DESC NULLS LAST, r.id DESC`,
-      [JSON.stringify(open), groupId, UPLOAD_GAP_MS, UPLOAD_GAP_MS]
-    )
-    if (candidates.length === 0) return
-    const ids: number[] = [...new Set<number>(candidates.map((c: { id: string }) => Number(c.id)))]
-    const names = new Map<number, Set<string>>()
-    const { rows: chunks } = await trx.rawQuery(`SELECT release_id, payload FROM release_chunks WHERE release_id = ANY(?)`, [ids])
-    for (const c of chunks) {
-      const set = names.get(Number(c.release_id)) ?? new Set<string>()
-      for (const f of decodeChunk(c.payload)) set.add(f.name)
-      names.set(Number(c.release_id), set)
-    }
-    for (const c of candidates as { idx: number; id: string }[]) {
-      if (joined.has(c.idx)) continue
-      const have = names.get(Number(c.id)) ?? new Set<string>()
-      const incoming = decodeChunk(part[c.idx].payload).map((f) => f.name)
-      if (incoming.every((name) => !have.has(name))) {
-        joined.set(c.idx, Number(c.id))
-        for (const name of incoming) have.add(name)
-        names.set(Number(c.id), have)
-      }
-    }
+  /** The range's block; a range without one (no planner yet) is a block of its own, leased as the range is. */
+  async blockOf(trx: TransactionClientContract, range: any): Promise<number> {
+    if (range.block_id) return Number(range.block_id)
+    const [block] = await trx
+      .table('blocks')
+      .insert({
+        group_id: range.group_id,
+        start_id: range.start_id,
+        end_id: range.end_id,
+        next_id: range.next_id,
+        status: 'leased',
+        priority: range.priority ?? 0,
+        node_id: range.node_id,
+        leased_at: range.leased_at ?? new Date(),
+        heartbeat_at: new Date(),
+      })
+      .returning('id')
+    await trx.from('ranges').where('id', range.id).update({ block_id: block.id })
+    return Number(block.id)
   }
 
   /**
-   * Release aggregates in the catalog, one write-once chunk per release, and
-   * probe candidates (a batch's, or files the fragment merger completed).
-   * A chunk joins the release of its key whose articles the range's group
-   * owns and whose posting window is within UPLOAD_GAP_MS of its own, else
-   * starts one: re-uploads are releases of their own (decision d). The
-   * group is locked, so concurrent writers don't both start one.
+   * The unnamed releases of a batch into pending_releases (migration 6300, the HDD): keys, counts, the upload
+   * tool and the probe targets, no chunk. The pending ladder (REBUILD.md decision f) probes them cheapest first,
+   * and a name moves one into the catalog; a provisional tail release sent again adds its counts up.
+   */
+  async writePending(
+    trx: TransactionClientContract,
+    groupId: number,
+    releases: ReleaseChunk[],
+    candidates: ProbeCandidate[]
+  ): Promise<number> {
+    if (releases.length === 0) return 0
+    const targets = new Map<string, { kind: string; messageId: string; file: string }[]>()
+    for (const c of candidates) {
+      const key = slot(c.releaseKey, c.upload ?? 0)
+      const list = targets.get(key) ?? []
+      list.push({ kind: c.kind, messageId: c.messageId, file: c.file })
+      targets.set(key, list)
+    }
+    let written = 0
+    for (let i = 0; i < releases.length; i += 1000) {
+      const rows = releases.slice(i, i + 1000).map((r) => {
+        const probes = targets.get(slot(r.key, r.upload)) ?? []
+        return {
+          release_key: releaseKey(r.key, groupId, r.firstArticle).toString('hex'),
+          identity_key: identityKey(r.key).toString('hex'),
+          name: r.name.slice(0, TEXT_MAX),
+          poster: r.poster || null,
+          first: r.postedAt,
+          last: r.postedLast ?? r.postedAt,
+          first_article: r.firstArticle,
+          last_article: r.lastArticle,
+          files: r.files,
+          segments: r.segments,
+          bytes: r.bytes,
+          partial: r.partial === true,
+          finished: r.finished !== false,
+          file_total: r.fileTotal,
+          tool: r.tool ?? (probes[0] ? uploadTool(probes[0].messageId) : 'unknown'),
+          candidates: probes,
+        }
+      })
+      const { rows: done } = await trx.rawQuery(
+        `INSERT INTO pending_releases (release_key, identity_key, group_id, name, poster, posted_at, last_posted_at, first_article, last_article,
+                                       files, segments, bytes, partial, finished, file_total, tool, candidates)
+         SELECT decode(v.release_key, 'hex'), decode(v.identity_key, 'hex'), ?, v.name, v.poster, to_timestamp(v.first / 1000.0), to_timestamp(v.last / 1000.0),
+                v.first_article, v.last_article, v.files, v.segments, v.bytes, v.partial, v.finished, v.file_total, v.tool, v.candidates
+         FROM jsonb_to_recordset(?::jsonb) AS v(release_key text, identity_key text, name text, poster text, first bigint, last bigint, first_article bigint, last_article bigint,
+                                               files int, segments bigint, bytes bigint, partial boolean, finished boolean, file_total int, tool text, candidates jsonb)
+         ON CONFLICT (release_key) DO UPDATE SET
+           files = pending_releases.files + excluded.files, segments = pending_releases.segments + excluded.segments, bytes = pending_releases.bytes + excluded.bytes,
+           last_posted_at = greatest(pending_releases.last_posted_at, excluded.last_posted_at), last_article = greatest(pending_releases.last_article, excluded.last_article),
+           partial = excluded.partial, finished = excluded.finished, file_total = coalesce(excluded.file_total, pending_releases.file_total),
+           candidates = CASE WHEN jsonb_array_length(pending_releases.candidates) = 0 THEN excluded.candidates ELSE pending_releases.candidates END
+         WHERE NOT pending_releases.finished
+         RETURNING release_key`,
+        [groupId, JSON.stringify(rows)]
+      )
+      written += done.length
+    }
+    return written
+  }
+
+  /**
+   * The releases of a batch into the catalog, one chunk each, and their probe candidates. Every chunk is one
+   * release: inserted, or (a provisional tail release sent again) finished.
    */
   async writeReleases(
     trx: TransactionClientContract,
-    rangeId: number,
+    where: { groupId: number; blockId: number },
     batchFrom: number,
     releases: ReleaseChunk[],
     candidates: ProbeCandidate[]
   ) {
-    const out = { newReleases: 0, candidates: 0 }
+    const out = { newReleases: 0, finalized: 0, candidates: 0 }
+    if (releases.length === 0) return out
+    /** Release ids by slot, for the chunks and candidates; a slot not here was a finished release sent again. */
     const ids = new Map<string, number>()
-    const slot = (key: string, upload: number) => `${key}\n${upload}`
-    const { group_id: groupId } = await trx.from('ranges').where('id', rangeId).select('group_id').firstOrFail()
-    // One writer per group at a time: a chunk only matches releases of its group, and a lock per key
-    // ran out of shared memory (a picture group's batch has 50k keys; LOG.md 2026-10-09 (12)).
-    await trx.rawQuery(`SELECT pg_advisory_xact_lock(hashtextextended('releases:' || ?::text, 0))`, [groupId])
+    const finalized = new Set<number>()
+
+    const posters = await CatalogRefs.posterIds(trx, releases.map((r) => r.poster).filter(Boolean))
+    const groupSets = await CatalogRefs.groupSetIds(
+      trx,
+      releases.map((r) => r.groups)
+    )
     const sorted = [...releases].sort((a, b) => a.key.localeCompare(b.key) || a.upload - b.upload)
     for (let i = 0; i < sorted.length; i += 1000) {
       const part = sorted.slice(i, i + 1000)
-      const windows = part.map((r, idx) => ({ idx, key: r.key, first: r.postedAt, last: r.postedLast ?? r.postedAt }))
-      const { rows: matches } = await trx.rawQuery(
-        `SELECT DISTINCT ON (v.idx) v.idx, r.id
-         FROM jsonb_to_recordset(?::jsonb) AS v(idx int, key text, first bigint, last bigint)
-         JOIN releases r ON r.identity_key = v.key AND r.group_id = ?
-         WHERE v.first IS NULL OR r.posted_at IS NULL
-            OR (r.posted_at <= to_timestamp((v.last + ?) / 1000.0) AND coalesce(r.last_posted_at, r.posted_at) >= to_timestamp((v.first - ?) / 1000.0))
-         ORDER BY v.idx, r.last_posted_at DESC NULLS LAST, r.id DESC`,
-        [JSON.stringify(windows), groupId, UPLOAD_GAP_MS, UPLOAD_GAP_MS]
-      )
-      const joined = new Map<number, number>(matches.map((m: { idx: number; id: string }) => [m.idx, Number(m.id)]))
-      await this.joinAcrossGroups(trx, part, windows, groupId, joined)
-      const row = (r: ReleaseChunk) => ({
+      const rows = part.map((r, idx) => ({
+        idx,
+        release_key: releaseKey(r.key, where.groupId, r.firstArticle).toString('hex'),
+        identity_key: identityKey(r.key).toString('hex'),
         name: r.name,
-        source: r.source,
-        poster: r.poster,
-        groups: r.groups,
+        source: sourceCode(r.source),
+        poster_id: r.poster ? (posters.get(posterText(r.poster)) ?? null) : null,
+        group_set_id: groupSets.get(groupSetKey(r.groups))!,
         first: r.postedAt,
         last: r.postedLast ?? r.postedAt,
         file_total: r.fileTotal,
@@ -217,89 +273,85 @@ class BatchStore {
         last_article: r.lastArticle,
         readable: r.readable,
         has_par2: r.hasPar2,
-      })
-      const updates = part.flatMap((r, idx) => (joined.has(idx) ? [{ id: joined.get(idx)!, ...row(r) }] : []))
-      if (updates.length > 0) {
+        finished: r.finished !== false,
+        edge: r.edge === true,
+        partial: r.partial === true,
+        sidecar_only: sidecarOnly(r),
+      }))
+      // Inserted, or a provisional release of the key finished (the chunk's files added up); a finished release
+      // sent again (no row returned) is left as it is.
+      const { rows: written } = await trx.rawQuery(
+        `INSERT INTO releases (release_key, identity_key, group_id, group_set_id, poster_id, name, source, posted_at, last_posted_at, file_total, files, segments, bytes, chunks,
+                               first_article, last_article, readable, has_par2, complete, edge, sidecar_only, settled_at)
+         SELECT decode(v.release_key, 'hex'), decode(v.identity_key, 'hex'), ?, v.group_set_id, v.poster_id, v.name, v.source,
+                to_timestamp(v.first / 1000.0), to_timestamp(v.last / 1000.0), v.file_total, v.files, v.segments, v.bytes, 1,
+                v.first_article, v.last_article, v.readable, v.has_par2,
+                CASE WHEN v.partial THEN false WHEN v.file_total IS NULL THEN NULL ELSE v.files >= v.file_total END,
+                v.edge, v.sidecar_only, CASE WHEN v.finished THEN now() END
+         FROM jsonb_to_recordset(?::jsonb) AS v(idx int, release_key text, identity_key text, name text, source int, poster_id int, group_set_id int, first bigint, last bigint,
+                                               file_total int, files int, segments bigint, bytes bigint, first_article bigint, last_article bigint, readable boolean, has_par2 boolean,
+                                               finished boolean, edge boolean, partial boolean, sidecar_only boolean)
+         ON CONFLICT (release_key) DO UPDATE SET
+           files = releases.files + excluded.files, segments = releases.segments + excluded.segments, bytes = releases.bytes + excluded.bytes,
+           chunks = releases.chunks + CASE WHEN excluded.files > 0 THEN 1 ELSE 0 END,
+           posted_at = least(releases.posted_at, excluded.posted_at), last_posted_at = greatest(releases.last_posted_at, excluded.last_posted_at),
+           file_total = coalesce(excluded.file_total, releases.file_total),
+           first_article = least(releases.first_article, excluded.first_article), last_article = greatest(releases.last_article, excluded.last_article),
+           readable = releases.readable OR excluded.readable, has_par2 = releases.has_par2 OR excluded.has_par2,
+           complete = CASE WHEN releases.complete IS FALSE AND releases.file_total IS NULL THEN false
+                           WHEN coalesce(excluded.file_total, releases.file_total) IS NULL THEN releases.complete
+                           ELSE releases.files + excluded.files >= coalesce(excluded.file_total, releases.file_total) END,
+           edge = excluded.edge, sidecar_only = coalesce(releases.sidecar_only, excluded.sidecar_only), settled_at = excluded.settled_at
+         WHERE releases.settled_at IS NULL
+         RETURNING id, encode(release_key, 'hex') AS release_key, (xmax = 0) AS inserted, settled_at`,
+        [where.groupId, JSON.stringify(rows)]
+      )
+      const byKey = new Map<string, { id: number; inserted: boolean; settled: boolean }>(
+        written.map(
+          (w: { id: string; release_key: string; inserted: boolean; settled_at: Date | null }) => [
+            w.release_key,
+            { id: Number(w.id), inserted: w.inserted, settled: w.settled_at !== null },
+          ]
+        )
+      )
+      const texts: { release_id: number; kind: number; text: string }[] = []
+      for (const row of rows) {
+        const w = byKey.get(row.release_key)
+        if (!w) continue
+        const r = part[row.idx]
+        ids.set(slot(r.key, r.upload), w.id)
+        if (w.inserted) out.newReleases++
+        else if (w.settled) {
+          out.finalized++
+          finalized.add(w.id)
+        }
+        texts.push({ release_id: w.id, kind: TEXT_KINDS.name, text: row.name.slice(0, TEXT_MAX) })
+        // An album's members stay findable by their own names (the first TEXTS_PER_KIND of them).
+        if (r.source === 'album' && r.files > 0) {
+          for (const text of textsToStore(decodeChunk(r.payload).map((f) => f.name))) {
+            texts.push({ release_id: w.id, kind: TEXT_KINDS.member, text })
+          }
+        }
+      }
+      // The name (and an album's members), searchable (release_texts; a text sent again is a no-op).
+      if (texts.length > 0) {
         await trx.rawQuery(
-          `WITH v AS (SELECT * FROM jsonb_to_recordset(?::jsonb) AS v(id bigint, first bigint, last bigint, file_total int, files int, segments bigint, bytes bigint, first_article bigint, last_article bigint, readable boolean, has_par2 boolean)),
-                a AS (SELECT id, sum(files) AS files, sum(segments) AS segments, sum(bytes) AS bytes, count(*) AS chunks, min(first) AS first, max(last) AS last,
-                             max(file_total) AS file_total, min(first_article) AS first_article, max(last_article) AS last_article, bool_or(readable) AS readable, bool_or(has_par2) AS has_par2
-                      FROM v GROUP BY id)
-           UPDATE releases r SET
-             files = r.files + a.files,
-             segments = r.segments + a.segments,
-             bytes = r.bytes + a.bytes,
-             chunks = r.chunks + a.chunks,
-             posted_at = least(r.posted_at, to_timestamp(a.first / 1000.0)),
-             last_posted_at = greatest(r.last_posted_at, to_timestamp(a.last / 1000.0)),
-             file_total = coalesce(r.file_total, a.file_total),
-             first_article = least(r.first_article, a.first_article),
-             last_article = greatest(r.last_article, a.last_article),
-             readable = r.readable OR a.readable,
-             has_par2 = r.has_par2 OR a.has_par2,
-             -- As it stands (the counter's files all found); settling makes it final. Without a counter, what its par2
-             -- set decided stays (app/services/par2_completeness.ts).
-             complete = CASE WHEN coalesce(r.file_total, a.file_total) IS NULL THEN r.complete ELSE r.files + a.files >= coalesce(r.file_total, a.file_total) END,
-             last_seen_at = now()
-           FROM a WHERE r.id = a.id`,
-          [JSON.stringify(updates)]
+          `INSERT INTO release_texts (release_id, kind, text)
+           SELECT v.release_id, v.kind, v.text FROM jsonb_to_recordset(?::jsonb) AS v(release_id bigint, kind int, text text) WHERE v.text <> ''
+           ON CONFLICT DO NOTHING`,
+          [JSON.stringify(texts)]
         )
-        part.forEach((r, idx) => joined.has(idx) && ids.set(slot(r.key, r.upload), joined.get(idx)!))
-      }
-      const fresh = part.filter((_, idx) => !joined.has(idx))
-      if (fresh.length > 0) {
-        // The first upload of a key keeps the key; later ones are key:2, key:3, … after the highest number: merges
-        // delete releases, and a count fell behind it (83 releases of a key up to key:129: 20 numbers tried, all
-        // taken, the range's batch failed for good; LOG.md 2026-10-20 (89)). Spot releases (spot:key) aren't numbered.
-        const { rows: counts } = await trx.rawQuery(
-          `SELECT identity_key, max(CASE WHEN release_key = identity_key THEN 1
-                                         WHEN left(release_key, length(identity_key) + 1) = identity_key || ':' AND substr(release_key, length(identity_key) + 2) ~ '^[0-9]{1,9}$'
-                                         THEN substr(release_key, length(identity_key) + 2)::int END) AS n
-           FROM releases WHERE identity_key = ANY(?) GROUP BY 1`,
-          [[...new Set(fresh.map((r) => r.key))]]
-        )
-        const taken = new Map<string, number>(counts.map((c: { identity_key: string; n: string }) => [c.identity_key, Number(c.n)]))
-        const numbered = (r: ReleaseChunk) => {
-          const n = (taken.get(r.key) ?? 0) + 1
-          taken.set(r.key, n)
-          return { release_key: n === 1 ? r.key : `${r.key}:${n}`, key: r.key, upload: r.upload, ...row(r), sidecar_only: sidecarOnly(r) }
-        }
-        // Their posters, for poster search (posters: one row per poster, searched by trigram; migration 4000).
-        const posters = [...new Set(fresh.map((r) => r.poster).filter((p): p is string => !!p))]
-        if (posters.length > 0) await trx.rawQuery(`INSERT INTO posters (poster) SELECT DISTINCT left(p, 200) FROM unnest(?::text[]) AS p ON CONFLICT DO NOTHING`, [posters])
-        let inserts = fresh.map(numbered)
-        // Writers of other groups number the same key at the same time (a spam post crossposted to 67
-        // groups; only the group is locked): a number taken meanwhile is skipped, and the next one tried.
-        for (let attempt = 0; inserts.length > 0; attempt++) {
-          if (attempt === 20) throw new Error(`No free release key for ${inserts[0].key}`)
-          const { rows } = await trx.rawQuery(
-            `INSERT INTO releases (release_key, identity_key, group_id, name, source, poster, groups, posted_at, last_posted_at, file_total, files, segments, bytes, chunks, first_article, last_article, readable, has_par2, complete, sidecar_only)
-             SELECT v.release_key, v.key, ?, v.name, v.source, v.poster, v.groups, to_timestamp(v.first / 1000.0), to_timestamp(v.last / 1000.0), v.file_total, v.files, v.segments, v.bytes, 1,
-                    v.first_article, v.last_article, v.readable, v.has_par2, CASE WHEN v.file_total IS NULL THEN NULL ELSE v.files >= v.file_total END, v.sidecar_only
-             FROM jsonb_to_recordset(?::jsonb) AS v(release_key text, key text, name text, source text, poster text, groups jsonb, first bigint, last bigint, file_total int, files int, segments bigint, bytes bigint, first_article bigint, last_article bigint, readable boolean, has_par2 boolean, sidecar_only boolean)
-             ON CONFLICT (release_key) DO NOTHING
-             RETURNING id, release_key`,
-            [groupId, JSON.stringify(inserts)]
-          )
-          const byReleaseKey = new Map<string, number>(rows.map((x: { id: string; release_key: string }) => [x.release_key, Number(x.id)]))
-          const missed: typeof inserts = []
-          for (const r of inserts) {
-            const id = byReleaseKey.get(r.release_key)
-            if (id === undefined) missed.push(r)
-            else ids.set(slot(r.key, r.upload), id)
-          }
-          out.newReleases += rows.length
-          const chunks = new Map(fresh.map((r) => [slot(r.key, r.upload), r]))
-          inserts = missed.map((r) => numbered(chunks.get(slot(r.key, r.upload))!))
-        }
       }
     }
 
-    // One write-once chunk per release.
+    // One write-once chunk per release (a finish adds the files not sent before).
     for (let i = 0; i < sorted.length; i += 500) {
-      const rows = sorted.slice(i, i + 500).map((r) => ({
+      const rows = sorted
+        .slice(i, i + 500)
+        .filter((r) => ids.has(slot(r.key, r.upload)) && r.files > 0)
+        .map((r) => ({
           release_id: ids.get(slot(r.key, r.upload))!,
-        range_id: rangeId,
+          block_id: where.blockId,
           batch_from: batchFrom,
           files: r.files,
           segments: r.segments,
@@ -308,16 +360,27 @@ class BatchStore {
       if (rows.length > 0) await trx.insertQuery().table('release_chunks').multiInsert(rows)
     }
 
+    // A finished release's candidates are claimable at once; a provisional one's wait for its finish.
+    const settled = new Set<number>()
+    for (const r of sorted) {
+      const id = ids.get(slot(r.key, r.upload))
+      if (id !== undefined && r.finished !== false) settled.add(id)
+    }
+    const chunks = new Map(sorted.map((r) => [slot(r.key, r.upload), r]))
     const kept = candidates.filter((c) => ids.has(slot(c.releaseKey, c.upload ?? 0)))
     for (let i = 0; i < kept.length; i += 2000) {
-      const rows = kept
-        .slice(i, i + 2000)
-        .map((c) => ({
-          release_id: ids.get(slot(c.releaseKey, c.upload ?? 0))!,
+      const rows = kept.slice(i, i + 2000).map((c) => {
+        const releaseId = ids.get(slot(c.releaseKey, c.upload ?? 0))!
+        const chunk = chunks.get(slot(c.releaseKey, c.upload ?? 0))!
+        return {
+          release_id: releaseId,
           kind: c.kind,
           message_id: c.messageId,
           file: c.file,
-        }))
+          ready: settled.has(releaseId),
+          unreadable: !chunk.readable,
+        }
+      })
       const added = await trx
         .knexQuery()
         .table('probe_candidates')
@@ -329,16 +392,15 @@ class BatchStore {
       // A later piece of a split 7z: the tails of lower volumes go (app/services/seven_zip_tails.ts).
       const tails = new Set(rows.filter((r) => r.kind === '7z-tail').map((r) => r.release_id))
       if (added.length > 0 && tails.size > 0) await pruneSevenZipTails(trx, [...tails])
-      // Files added to a settled release (a later range of its upload): the settler made its candidates claimable
-      // already and won't again (93,642 waited since 2026-10-13; LOG.md 2026-10-20 (87)).
-      if (added.length > 0) {
+    }
+    // The finish of a provisional release: its waiting candidates become claimable.
+    if (finalized.size > 0) {
       await trx.rawQuery(
         `UPDATE probe_candidates pc SET ready = true, unreadable = NOT r.readable FROM releases r
-           WHERE pc.id = ANY(?) AND r.id = pc.release_id AND r.settled_at IS NOT NULL`,
-          [added.map((a: { id: string | number }) => String(a.id))]
+         WHERE pc.release_id = ANY(?) AND r.id = pc.release_id AND pc.status = 'waiting' AND NOT pc.ready`,
+        [[...finalized]]
       )
     }
-    }
 
     return out
   }

app/services/catalog_refs.ts

@@ -0,0 +1,93 @@
+import type db from '@adonisjs/lucid/services/db'
+
+type Client = Pick<typeof db, 'rawQuery'>
+
+/** Posters are kept to this length (as the trigram index and poster search always did; a btree entry holds 2,704 bytes). */
+export const POSTER_MAX = 200
+/** Cached ids per process: group sets are ~2,600 in all; posters many more, so the cache is emptied when it grows past this. */
+const CACHE_MAX = 200_000
+
+const posterCache = new Map<string, number>()
+const groupSetCache = new Map<string, number>()
+
+/** A group list as group_sets stores it: sorted, without repeats, as JSON. */
+export function groupSetKey(groups: string[]): string {
+  return JSON.stringify([...new Set(groups)].sort())
+}
+
+/** A poster as posters stores it. */
+export function posterText(poster: string): string {
+  return poster.slice(0, POSTER_MAX)
+}
+
+async function ids(
+  client: Client,
+  cache: Map<string, number>,
+  table: 'posters' | 'group_sets',
+  column: 'poster' | 'groups',
+  cast: 'text' | 'jsonb',
+  values: string[]
+): Promise<Map<string, number>> {
+  const out = new Map<string, number>()
+  const missing: string[] = []
+  for (const v of values) {
+    const id = cache.get(v)
+    if (id === undefined) missing.push(v)
+    else out.set(v, id)
+  }
+  if (missing.length === 0) return out
+  // Insert what's new, then read every id (ON CONFLICT DO NOTHING returns nothing for the rows that existed).
+  await client.rawQuery(
+    `INSERT INTO ${table} (${column}) SELECT DISTINCT v::${cast} FROM unnest(?::text[]) AS v ON CONFLICT DO NOTHING`,
+    [missing]
+  )
+  const { rows } = await client.rawQuery(
+    `SELECT id, ${column}::text AS value FROM ${table} WHERE ${column} = ANY(?::${cast}[])`,
+    [missing]
+  )
+  if (cache.size + rows.length > CACHE_MAX) cache.clear()
+  for (const r of rows) {
+    // jsonb comes back in Postgres' spelling ("["a", "b"]"); the caller's key is the JSON we sent.
+    const key = cast === 'jsonb' ? JSON.stringify(JSON.parse(r.value)) : r.value
+    cache.set(key, Number(r.id))
+    out.set(key, Number(r.id))
+  }
+  return out
+}
+
+/**
+ * Ids in the catalog's lookup tables (voyager 3, migration 6000): a release's poster_id and group_set_id. New
+ * values are inserted; known ones come from a cache per process (the batch store sees the same posters and
+ * group sets batch after batch).
+ */
+export default {
+  /** Ids by poster (posterText applied), for a batch of posters. */
+  async posterIds(client: Client, posters: string[]): Promise<Map<string, number>> {
+    return ids(client, posterCache, 'posters', 'poster', 'text', [
+      ...new Set(posters.map(posterText)),
+    ])
+  },
+
+  async posterId(client: Client, poster: string): Promise<number> {
+    const found = await this.posterIds(client, [poster])
+    return found.get(posterText(poster))!
+  },
+
+  /** Ids by groupSetKey, for a batch of group lists. */
+  async groupSetIds(client: Client, sets: string[][]): Promise<Map<string, number>> {
+    return ids(client, groupSetCache, 'group_sets', 'groups', 'jsonb', [
+      ...new Set(sets.map(groupSetKey)),
+    ])
+  },
+
+  async groupSetId(client: Client, groups: string[]): Promise<number> {
+    const found = await this.groupSetIds(client, [groups])
+    return found.get(groupSetKey(groups))!
+  },
+
+  /** Forgets the cached ids (tests roll their transactions back, so the ids they saw are gone). */
+  forget() {
+    posterCache.clear()
+    groupSetCache.clear()
+  },
+}

app/services/corpus.ts

@@ -42,13 +42,13 @@ class Corpus {
     return coordinatorConfig.corpus.dir
   }
 
-  private rangeDir(range: CorpusRange) {
-    return join(this.dir!, range.group, `${range.start}-${range.end}`)
+  private rangeDir(range: CorpusRange, dir = this.dir!) {
+    return join(dir, range.group, `${range.start}-${range.end}`)
   }
 
-  private path(range: CorpusRange, span: ChunkSpan) {
+  private path(range: CorpusRange, span: ChunkSpan, dir = this.dir!) {
     const index = Math.floor((span.from - range.start) / coordinatorConfig.chunkArticles)
-    return join(this.rangeDir(range), `${String(index).padStart(4, '0')}-${span.from}-${span.to}.gz`)
+    return join(this.rangeDir(range, dir), `${String(index).padStart(4, '0')}-${span.from}-${span.to}.gz`)
   }
 
   /** Whether every chunk of the range is stored (its range.json). */
@@ -69,10 +69,10 @@ class Corpus {
     )
   }
 
-  /** A stored chunk, or null. */
-  async read(range: CorpusRange, span: ChunkSpan): Promise<Buffer | null> {
-    if (!this.dir) return null
-    return readFile(this.path(range, span)).catch(() => null)
+  /** A stored chunk, or null (`dir`: another corpus than CORPUS_DIR's, the namer's tests). */
+  async read(range: CorpusRange, span: ChunkSpan, dir = this.dir): Promise<Buffer | null> {
+    if (!dir) return null
+    return readFile(this.path(range, span, dir)).catch(() => null)
   }
 
   /**

app/services/fragment_merger.ts

@@ -1,153 +0,0 @@
-import db from '@adonisjs/lucid/services/db'
-import BatchStore from '#services/batch_store'
-import ChunkPartitions from '#services/chunk_partitions'
-import RuleSets from '#services/rule_sets'
-import { RuleSet } from '#lib/rules/index'
-import { candidatesOf, chunkOf, mergeFragments, releaseKeyOf, uploadsOf, type Fragment, type ProbeCandidate, type ReleaseChunk } from '#lib/edge/index'
-import type { AggregatedFile } from '#lib/files/index'
-import type { EdgeSegment } from '#lib/edge/index'
-
-export interface MergeResult {
-  files: number
-  completed: number
-  incomplete: number
-  waiting: number
-  /** This round reached the end of the file keys (the next one starts over). */
-  wrapped: boolean
-}
-
-/** File keys per round. */
-const BATCH = 20_000
-
-/**
- * Files cut by a range's edge (REBUILD.md, fragments), put together once
- * the ranges around them are done: a complete file is judged by the rules
- * like any other and written as a release chunk (with probe candidates); a
- * file still missing parts while every neighbouring range is done is
- * incomplete, and dropped (counted on its ranges). Others wait.
- */
-class FragmentMerger {
-  /** Where the next round goes on: waiting files must not take every round's places. */
-  private cursor = ''
-
-  async run(): Promise<MergeResult> {
-    const result: MergeResult = { files: 0, completed: 0, incomplete: 0, waiting: 0, wrapped: false }
-    const { rows: keys } = await db.rawQuery(
-      `SELECT DISTINCT f.file_key FROM fragments f JOIN ranges r ON r.id = f.range_id
-       WHERE r.status = 'done' AND f.file_key > ? ORDER BY f.file_key LIMIT ?`,
-      [this.cursor, BATCH]
-    )
-    // At the end of the keys: the next round starts over.
-    result.wrapped = keys.length < BATCH
-    this.cursor = result.wrapped ? '' : keys.at(-1).file_key
-    if (keys.length === 0) return result
-    const rules = new RuleSet(await RuleSets.current())
-    const rows = await db
-      .from('fragments as f')
-      .join('ranges as r', 'r.id', 'f.range_id')
-      .whereIn('f.file_key', keys.map((k: { file_key: string }) => k.file_key))
-      .select('f.id', 'f.file_key', 'f.range_id', 'f.fragment', 'r.group_id', 'r.start_id', 'r.end_id', 'r.status')
-    const byKey = new Map<string, typeof rows>()
-    for (const row of rows) {
-      const list = byKey.get(row.file_key)
-      if (list) list.push(row)
-      else byKey.set(row.file_key, [row])
-    }
-
-    const completed: { file: AggregatedFile<EdgeSegment>; rangeId: number; ids: number[] }[] = []
-    const incomplete: { rangeId: number; ids: number[]; segments: number }[] = []
-    const neighbours = await this.neighbourStatus([...byKey.values()])
-    for (const list of byKey.values()) {
-      result.files++
-      const file = mergeFragments(list.map((r) => r.fragment as Fragment))
-      const ids = list.map((r) => Number(r.id))
-      const rangeId = Math.max(...list.map((r) => Number(r.range_id)))
-      if (file) {
-        completed.push({ file, rangeId, ids })
-        continue
-      }
-      if (list.every((r) => r.status === 'done') && neighbours(list)) {
-        incomplete.push({ rangeId, ids, segments: list.reduce((n, r) => n + (r.fragment as Fragment).segments.length, 0) })
-      } else {
-        result.waiting++
-      }
-    }
-
-    // Completed files, grouped into their releases, judged and written like a batch's.
-    const byRelease = new Map<string, { files: AggregatedFile<EdgeSegment>[]; rangeId: number }>()
-    for (const { file, rangeId } of completed) {
-      const key = releaseKeyOf(file.segments[0])
-      const entry = byRelease.get(key)
-      if (entry) entry.files.push(file)
-      else byRelease.set(key, { files: [file], rangeId })
-    }
-    const releases: { rangeId: number; chunk: ReleaseChunk; candidates: ProbeCandidate[] }[] = []
-    for (const [key, { files, rangeId }] of byRelease) {
-      const verdict = rules.judge(key, files.map((f) => ({ name: f.segments[0].file.name, totalParts: f.segments[0].totalParts, groups: f.segments[0].file.groups, firstMessageId: f.segments[0].messageId, poster: f.segments[0].file.poster })))
-      if (verdict.rule !== null) continue
-      uploadsOf(files).forEach((list, upload) => releases.push({ rangeId, chunk: chunkOf(key, list, upload), candidates: candidatesOf(key, list, upload) }))
-    }
-    await ChunkPartitions.ensure()
-    await db.transaction(async (trx) => {
-      // In key order (batches lock their group; the merger only ever waits on one batch at a time).
-      releases.sort((a, b) => a.chunk.key.localeCompare(b.chunk.key) || a.chunk.upload - b.chunk.upload)
-      for (const { rangeId, chunk, candidates } of releases) await BatchStore.writeReleases(trx, rangeId, 0, [chunk], candidates)
-      const done = [...completed.flatMap((c) => c.ids), ...incomplete.flatMap((c) => c.ids)]
-      for (let i = 0; i < done.length; i += 5000) await trx.from('fragments').whereIn('id', done.slice(i, i + 5000)).delete()
-      for (const { rangeId, segments } of incomplete) {
-        await trx.rawQuery(
-          `INSERT INTO rule_counts (range_id, rule, key, articles, files) VALUES (?, 'incomplete', 'edge', ?, 1)
-           ON CONFLICT (range_id, rule, key) DO UPDATE SET articles = rule_counts.articles + excluded.articles, files = rule_counts.files + 1`,
-          [rangeId, segments]
-        )
-      }
-    })
-    result.completed = completed.length
-    result.incomplete = incomplete.length
-    return result
-  }
-
-  /**
-   * For each file's fragments: whether the ranges right before and after
-   * their ranges (where a missing part could be) are done, read in one
-   * query for the whole round (per file it was three, ~2,000 files a
-   * minute). A neighbour not planned yet counts as not done (newer articles
-   * come later), except before the group's oldest article.
-   */
-  private async neighbourStatus(lists: { group_id: number; start_id: number | string; end_id: number | string }[][]) {
-    const span = (list: (typeof lists)[number]) => ({
-      group: Number(list[0].group_id),
-      first: Math.min(...list.map((r) => Number(r.start_id))),
-      last: Math.max(...list.map((r) => Number(r.end_id))),
-    })
-    const spans = lists.map(span)
-    const groups = [...new Set(spans.map((s) => s.group))]
-    // The range holding the article before and the one after, each by the (group_id, start_id) index:
-    // "end_id = ? OR start_id = ?" read every range of the group per file (5.5 s a round; LOG.md 2026-10-09).
-    const { rows } = await db.rawQuery(
-      `SELECT s.group_id, s.before_end, s.after_start, b.status AS before, a.status AS after
-       FROM unnest(?::int[], ?::bigint[], ?::bigint[]) AS s(group_id, before_end, after_start)
-       LEFT JOIN LATERAL (SELECT r.status FROM ranges r WHERE r.group_id = s.group_id AND r.start_id <= s.before_end AND r.end_id >= s.before_end
-                          ORDER BY r.start_id DESC LIMIT 1) b ON true
-       LEFT JOIN LATERAL (SELECT r.status FROM ranges r WHERE r.group_id = s.group_id AND r.start_id <= s.after_start AND r.end_id >= s.after_start
-                          ORDER BY r.start_id DESC LIMIT 1) a ON true`,
-      [spans.map((s) => s.group), spans.map((s) => s.first - 1), spans.map((s) => s.last + 1)]
-    )
-    const byEnd = new Map<string, string>()
-    const byStart = new Map<string, string>()
-    for (const r of rows) {
-      if (r.before !== null) byEnd.set(`${r.group_id}:${r.before_end}`, r.before)
-      if (r.after !== null) byStart.set(`${r.group_id}:${r.after_start}`, r.after)
-    }
-    const lows = new Map((await db.from('groups').whereIn('id', groups).select('id', 'low')).map((g) => [Number(g.id), g.low === null ? null : Number(g.low)]))
-    return (list: (typeof lists)[number]) => {
-      const { group, first, last } = span(list)
-      const before = byEnd.get(`${group}:${first - 1}`)
-      const low = lows.get(group)
-      const beforeDone = before !== undefined ? before === 'done' : low === null || low === undefined || first <= low
-      return beforeDone && byStart.get(`${group}:${last + 1}`) === 'done'
-    }
-  }
-}
-
-export default new FragmentMerger()

app/services/nzb_service.ts

@@ -2,6 +2,7 @@ import { gunzipSync } from 'node:zlib'
 import db from '@adonisjs/lucid/services/db'
 import { decodeChunk, type ChunkFile } from '#lib/chunks/index'
 import { buildNzb } from '#lib/nzb/index'
+import { SOURCES } from '#lib/release/source'
 
 /** Where a known password came from, the most trusted first: the spot's NZB meta, its title, its text. */
 const PASSWORD_SOURCES = ['nzb', 'spot-title', 'spot-text']
@@ -35,7 +36,7 @@ class NzbService {
     if (!release) return null
     const password = (await this.passwords(releaseId))[0]?.password
     // A release a spot made of its own (no files matched the header pass's): the spot's NZB as read.
-    if (release.source === 'spot') {
+    if (Number(release.source) === SOURCES.spot) {
       const spot = await db.from('spots').where('release_id', releaseId).whereNotNull('nzb').select('nzb').first()
       if (spot) return { name: release.name, xml: withPassword(gunzipSync(spot.nzb).toString('utf8'), password) }
     }

app/services/par2_merger.ts

@@ -93,11 +93,12 @@ const HOLDERS = `SELECT c.release_id FROM (
 
 interface Member {
   id: number
-  release_key: string
   group_id: number
   files: number
   posted_at: Date | null
   readable: boolean
+  poster_id?: number | null
+  /** The poster's text (joined from posters) where a query needs it. */
   poster?: string | null
   file_total?: number | null
 }
@@ -295,7 +296,8 @@ export class Par2Merger {
    */
   async nameMatches(trx: any, setId: string): Promise<Map<string, Member> | null> {
     const { rows: all } = await trx.rawQuery(
-      `SELECT r.id, r.release_key, r.group_id, r.files, r.posted_at, r.readable, r.poster FROM par2_sets s JOIN releases r ON r.id = s.release_id
+      `SELECT r.id, r.group_id, r.files, r.posted_at, r.readable, r.poster_id, p.poster
+       FROM par2_sets s JOIN releases r ON r.id = s.release_id LEFT JOIN posters p ON p.id = r.poster_id
        WHERE s.set_id = ? ORDER BY r.posted_at DESC NULLS LAST, r.id`,
       [setId]
     )
@@ -310,12 +312,12 @@ export class Par2Merger {
     const names = new Set(described.map((d: { name: string }) => fileKey(d.name)))
     const at = new Date(anchor.posted_at).getTime()
     const { rows: candidates } = await trx.rawQuery(
-      `SELECT r.id, r.release_key, r.group_id, r.files, r.posted_at, r.readable, r.poster FROM releases r
-       WHERE left(r.poster, 200) = left(?, 200) AND r.poster = ? AND r.group_id = ? AND r.posted_at BETWEEN ? AND ? AND r.id <> ALL(?)
+      `SELECT r.id, r.group_id, r.files, r.posted_at, r.readable, r.poster_id FROM releases r
+       WHERE r.poster_id = ? AND r.group_id = ? AND r.posted_at BETWEEN ? AND ? AND r.id <> ALL(?)
          AND r.files <= ?
-       ORDER BY left(r.poster, 200), r.posted_at DESC NULLS LAST, r.id DESC LIMIT ?`,
+       ORDER BY r.posted_at DESC NULLS LAST, r.id DESC LIMIT ?`,
       // More files than the set describes, with its sidecars, can't all be named in it.
-      [anchor.poster, anchor.poster, anchor.group_id, new Date(at - NAME_SPREAD_MS), new Date(at + NAME_SPREAD_MS), holders.map((h: Member) => h.id), described.length + SLACK_FILES, NAME_CANDIDATES]
+      [anchor.poster_id, anchor.group_id, new Date(at - NAME_SPREAD_MS), new Date(at + NAME_SPREAD_MS), holders.map((h: Member) => h.id), described.length + SLACK_FILES, NAME_CANDIDATES]
     )
     const found = new Map<string, Member>(holders.map((h: Member) => [String(h.id), h]))
     let files = holders.reduce((n: number, h: Member) => n + Number(h.files), 0)
@@ -344,7 +346,7 @@ export class Par2Merger {
   async mergeStem(stem: string): Promise<number> {
     return db.transaction(async (trx) => {
       const { rows } = await trx.rawQuery(
-        `SELECT DISTINCT r.id, r.release_key, r.group_id, r.files, r.posted_at, r.readable, r.poster, r.file_total
+        `SELECT DISTINCT r.id, r.group_id, r.files, r.posted_at, r.readable, r.poster_id, r.file_total
          FROM release_inspections i JOIN releases r ON r.id = i.release_id
          WHERE i.kind = 'lone' AND ${INNER_STEM} = ?`,
         [stem]
@@ -356,7 +358,7 @@ export class Par2Merger {
       const total = (r: Member) => (r.file_total ? Number(r.file_total) : null)
       const near = (r: Member) =>
         r.group_id === anchor.group_id &&
-        (r.poster ?? '') === (anchor.poster ?? '') &&
+        (r.poster_id ?? null) === (anchor.poster_id ?? null) &&
         (total(r) === null || total(anchor) === null || total(r) === total(anchor)) &&
         (at(r) === null || at(anchor) === null || Math.abs(at(r)! - at(anchor)!) <= MAX_SPREAD_MS)
       const found = new Map<string, Member>()
@@ -374,7 +376,7 @@ export class Par2Merger {
   async mergeSet(setId: string): Promise<number> {
     return db.transaction(async (trx) => {
       const { rows: holders } = await trx.rawQuery(
-        `SELECT r.id, r.release_key, r.group_id, r.files, r.posted_at, r.readable FROM par2_sets s JOIN releases r ON r.id = s.release_id
+        `SELECT r.id, r.group_id, r.files, r.posted_at, r.readable FROM par2_sets s JOIN releases r ON r.id = s.release_id
          WHERE s.set_id = ? ORDER BY r.id`,
         [setId]
       )
@@ -383,7 +385,7 @@ export class Par2Merger {
       const at = (r: Member) => (r.posted_at ? new Date(r.posted_at).getTime() : null)
       const near = (r: Member) => r.group_id === anchor.group_id && (at(r) === null || at(anchor) === null || Math.abs(at(r)! - at(anchor)!) <= MAX_SPREAD_MS)
       const { rows: matches } = await trx.rawQuery(
-        `SELECT f.file_id, r.id, r.release_key, r.group_id, r.files, r.posted_at, r.readable
+        `SELECT f.file_id, r.id, r.group_id, r.files, r.posted_at, r.readable
          FROM par2_files f CROSS JOIN LATERAL (${HOLDERS}) h JOIN releases r ON r.id = h.release_id
          WHERE f.set_id = ?`,
         [setId]
@@ -426,7 +428,7 @@ export class Par2Merger {
 
       await trx.rawQuery(
         `INSERT INTO release_merges (from_id, from_key, into_id, set_id, files, via)
-         SELECT id, release_key, ?, ?, files, ? FROM releases WHERE id = ANY(?)`,
+         SELECT id, encode(release_key, 'hex'), ?, ?, files, ? FROM releases WHERE id = ANY(?)`,
         [target.id, key, via, others]
       )
       // A candidate of an article the target, or another member of lower id, has already (the same file in two releases)
@@ -462,13 +464,13 @@ export class Par2Merger {
            files = t.files + s.files, segments = t.segments + s.segments, bytes = t.bytes + s.bytes, chunks = t.chunks + s.chunks,
            first_article = least(t.first_article, s.first_article), last_article = greatest(t.last_article, s.last_article),
            posted_at = least(t.posted_at, s.posted_at), last_posted_at = greatest(t.last_posted_at, s.last_posted_at),
-           first_seen_at = least(t.first_seen_at, s.first_seen_at), last_seen_at = greatest(t.last_seen_at, s.last_seen_at),
+           first_seen_at = least(t.first_seen_at, s.first_seen_at),
            readable = t.readable OR s.readable, has_par2 = t.has_par2 OR s.has_par2, passworded = t.passworded OR s.passworded,
            file_total = greatest(t.file_total, s.file_total), inner_name = coalesce(t.inner_name, s.inner_name),
            name = coalesce(nullif(?, ''), t.name)
          FROM (SELECT sum(files) AS files, sum(segments) AS segments, sum(bytes) AS bytes, sum(chunks) AS chunks,
                  min(first_article) AS first_article, max(last_article) AS last_article, min(posted_at) AS posted_at,
-                 max(last_posted_at) AS last_posted_at, min(first_seen_at) AS first_seen_at, max(last_seen_at) AS last_seen_at,
+                 max(last_posted_at) AS last_posted_at, min(first_seen_at) AS first_seen_at,
                  bool_or(readable) AS readable, bool_or(has_par2) AS has_par2, bool_or(passworded) AS passworded,
                  max(file_total) AS file_total, max(inner_name) AS inner_name
                FROM releases WHERE id = ANY(?)) s

app/services/par2_pruner.ts

@@ -75,7 +75,7 @@ export class Par2Pruner {
                   WHERE o.set_id = b.set_id AND v.release_id IS NULL) THEN 'judged'
            WHEN NOT EXISTS (SELECT 1 FROM par2_sets s WHERE s.set_id = b.set_id AND s.created_at > ?)
             AND NOT EXISTS (SELECT 1 FROM par2_sets s JOIN releases r ON r.id = s.release_id WHERE s.set_id = b.set_id AND (
-                  release_final(r.group_id, r.first_article, r.last_article, r.posted_at, r.last_posted_at, r.complete) IS NOT TRUE
+                  r.settled_at IS NULL
                   OR EXISTS (SELECT 1 FROM recent x WHERE x.group_id = r.group_id
                     AND x.posted_low <= coalesce(r.last_posted_at, r.posted_at) + interval '6 hours' AND x.posted_high >= r.posted_at - interval '6 hours')))
             AND NOT EXISTS (SELECT 1 FROM par2_files f JOIN unseen u ON u.hash16k = f.hash16k AND u.size = f.length WHERE f.set_id = b.set_id)

app/services/pending_ladder.ts

@@ -0,0 +1,230 @@
+import db from '@adonisjs/lucid/services/db'
+import type { TransactionClientContract } from '@adonisjs/lucid/types/database'
+import coordinatorConfig from '#config/coordinator'
+import { explored } from '#lib/rules/index'
+import { usableName, type Inspection } from '#lib/probe/index'
+import RuleSets from '#services/rule_sets'
+
+/** pending_releases.status (migration 6400). */
+export const PENDING = {
+  waiting: 0,
+  named: 1,
+  discarded: 2,
+  exhausted: 3,
+  naming: 4,
+  taken: 9,
+} as const
+
+/** The probe targets of a pending release in the order of effort: a par2 names the set's files outright, an archive's start its entries, a lone file only its yEnc name, a 7z's tail is the costliest find. */
+export const EFFORT = ['par2', 'archive', 'lone', '7z-tail'] as const
+
+export interface PendingTarget {
+  kind: string
+  messageId: string
+  file: string
+}
+
+/** One probe the ladder hands the prober: the next target of a pending release. */
+export interface PendingProbe {
+  /** pending_releases.release_key as hex. */
+  releaseKey: string
+  kind: string
+  messageId: string
+  file: string
+  tool: string
+  groupId: number
+  poster: string
+  /** Targets probed before this one. */
+  tries: number
+  targets: number
+}
+
+export type PendingOutcome = {
+  probe: PendingProbe
+  status: 'done' | 'missing' | 'error' | 'waiting'
+  inspection?: Inspection
+}
+
+export function byEffort(targets: PendingTarget[]): PendingTarget[] {
+  const rank = (kind: string) => {
+    const i = (EFFORT as readonly string[]).indexOf(kind)
+    return i < 0 ? EFFORT.length : i
+  }
+  return [...targets].sort((a, b) => rank(a.kind) - rank(b.kind))
+}
+
+/**
+ * The pending ladder (REBUILD.md decision f): pending_releases probed from least to most effort, a usable name
+ * handing the release to the namer (app/services/pending_namer.ts), no name after the last target exhausting it;
+ * every finished climb counted on its signature (tool, group, poster; and the tool and group together), and a
+ * signature with enough probes and next to no names dropped: its waiting releases are discarded without a probe,
+ * a sample of them (the rules' samplePercent) still climbing so the rate keeps being measured.
+ */
+class PendingLadder {
+  /** A previous prober's claims back in the queue. */
+  async release(): Promise<number> {
+    const { rowCount } = await db.rawQuery(
+      `UPDATE pending_releases SET status = ? WHERE status = ?`,
+      [PENDING.waiting, PENDING.taken]
+    )
+    return rowCount ?? 0
+  }
+
+  /** The next target of up to `n` waiting releases with a target left, oldest first, claimed. */
+  async claim(n: number): Promise<PendingProbe[]> {
+    if (n <= 0) return []
+    const { rows } = await db.rawQuery(
+      `UPDATE pending_releases p SET status = ? FROM (
+         SELECT release_key FROM pending_releases
+         WHERE status = ? AND finished AND tries < jsonb_array_length(candidates)
+         ORDER BY created_at LIMIT ? FOR UPDATE SKIP LOCKED) c
+       WHERE p.release_key = c.release_key
+       RETURNING encode(p.release_key, 'hex') AS release_key, p.candidates, p.tries, p.tool, p.group_id, coalesce(p.poster, '') AS poster`,
+      [PENDING.taken, PENDING.waiting, n]
+    )
+    return rows.map((r: any) => {
+      const targets = byEffort(r.candidates as PendingTarget[])
+      const tries = Number(r.tries)
+      const target = targets[Math.min(tries, targets.length - 1)]
+      return {
+        releaseKey: r.release_key,
+        kind: target.kind,
+        messageId: target.messageId,
+        file: target.file,
+        tool: r.tool,
+        groupId: Number(r.group_id),
+        poster: r.poster,
+        tries,
+        targets: targets.length,
+      }
+    })
+  }
+
+  /**
+   * What the probes showed: a usable name moves the release to the namer (status 4), else the next target, or the
+   * end of the ladder (status 3); the finished climbs counted on their signatures.
+   */
+  async record(outcomes: PendingOutcome[], trx: TransactionClientContract | typeof db = db) {
+    if (outcomes.length === 0) return
+    const rows = outcomes.map((o) => {
+      const p = o.probe
+      if (o.status === 'waiting') {
+        return {
+          release_key: p.releaseKey,
+          status: PENDING.waiting,
+          tries: p.tries,
+          named: null,
+          inspection: null,
+          final: null,
+        }
+      }
+      const name = o.status === 'done' && o.inspection ? usableName(o.inspection, p.kind) : null
+      const tries = p.tries + 1
+      const inspection = {
+        kind: p.kind,
+        status: o.status,
+        result: o.inspection ? { ...o.inspection, par2: undefined } : null,
+      }
+      if (name)
+        return {
+          release_key: p.releaseKey,
+          status: PENDING.naming,
+          tries: 0,
+          named: name,
+          inspection,
+          final: 'named',
+        }
+      if (tries >= p.targets)
+        return {
+          release_key: p.releaseKey,
+          status: PENDING.exhausted,
+          tries,
+          named: null,
+          inspection,
+          final: 'exhausted',
+        }
+      return {
+        release_key: p.releaseKey,
+        status: PENDING.waiting,
+        tries,
+        named: null,
+        inspection,
+        final: null,
+      }
+    })
+    await trx.rawQuery(
+      `UPDATE pending_releases p SET status = v.status, tries = v.tries, named = coalesce(v.named, p.named),
+         inspections = CASE WHEN v.inspection IS NULL THEN p.inspections ELSE p.inspections || v.inspection END,
+         probed_at = CASE WHEN v.inspection IS NULL THEN p.probed_at ELSE now() END
+       FROM jsonb_to_recordset(?::jsonb) AS v(release_key text, status int, tries int, named text, inspection jsonb)
+       WHERE p.release_key = decode(v.release_key, 'hex')`,
+      [JSON.stringify(rows.map(({ final: _final, ...row }) => row))]
+    )
+    // The signatures: the poster's own and the tool's and group's together.
+    const counts = new Map<
+      string,
+      { tool: string; group_id: number; poster: string; probed: number; named: number }
+    >()
+    for (const [i, o] of outcomes.entries()) {
+      const final = rows[i].final
+      if (!final) continue
+      for (const poster of [o.probe.poster, '']) {
+        const key = `${o.probe.tool}\t${o.probe.groupId}\t${poster}`
+        const entry = counts.get(key) ?? {
+          tool: o.probe.tool,
+          group_id: o.probe.groupId,
+          poster,
+          probed: 0,
+          named: 0,
+        }
+        entry.probed++
+        if (final === 'named') entry.named++
+        counts.set(key, entry)
+      }
+    }
+    if (counts.size === 0) return
+    const { pendingMinProbes, pendingMaxNamed } = coordinatorConfig.probe
+    await trx.rawQuery(
+      `INSERT INTO pending_signatures (tool, group_id, poster, probed, named)
+       SELECT v.tool, v.group_id, v.poster, v.probed, v.named FROM jsonb_to_recordset(?::jsonb) AS v(tool text, group_id int, poster text, probed int, named int)
+       ON CONFLICT (tool, group_id, poster) DO UPDATE SET probed = pending_signatures.probed + excluded.probed, named = pending_signatures.named + excluded.named, updated_at = now()`,
+      [JSON.stringify([...counts.values()])]
+    )
+    await trx.rawQuery(
+      `UPDATE pending_signatures s SET dropped = (s.probed >= ?::int AND s.named::float <= s.probed * ?::float)
+       FROM jsonb_to_recordset(?::jsonb) AS v(tool text, group_id int, poster text)
+       WHERE s.tool = v.tool AND s.group_id = v.group_id AND s.poster = v.poster`,
+      [
+        pendingMinProbes,
+        pendingMaxNamed,
+        JSON.stringify(
+          [...counts.values()].map(({ tool, group_id, poster }) => ({ tool, group_id, poster }))
+        ),
+      ]
+    )
+  }
+
+  /** Waiting releases of a dropped signature discarded without a probe, the explored sample excepted; how many. */
+  async discard(limit = 10_000): Promise<number> {
+    const { samplePercent } = await RuleSets.current()
+    const { rows } = await db.rawQuery(
+      `SELECT encode(p.release_key, 'hex') AS release_key FROM pending_releases p
+       WHERE p.status = ? AND EXISTS (
+         SELECT 1 FROM pending_signatures s
+         WHERE s.dropped AND s.tool = p.tool AND s.group_id = p.group_id AND (s.poster = '' OR s.poster = coalesce(p.poster, '')))
+       LIMIT ?`,
+      [PENDING.waiting, limit]
+    )
+    const keys = rows
+      .map((r: any) => r.release_key as string)
+      .filter((key: string) => !explored(key, samplePercent))
+    if (keys.length === 0) return 0
+    const { rowCount } = await db.rawQuery(
+      `UPDATE pending_releases SET status = ? WHERE status = ? AND release_key = ANY(SELECT decode(k, 'hex') FROM unnest(?::text[]) AS k)`,
+      [PENDING.discarded, PENDING.waiting, keys]
+    )
+    return rowCount ?? 0
+  }
+}
+
+export default new PendingLadder()

app/services/pending_namer.ts

@@ -0,0 +1,222 @@
+import db from '@adonisjs/lucid/services/db'
+import logger from '@adonisjs/core/services/logger'
+import { setTimeout as sleep } from 'node:timers/promises'
+import { promisify } from 'node:util'
+import { unzip } from 'node:zlib'
+import env from '#start/env'
+import coordinatorConfig from '#config/coordinator'
+import { XoverClient } from '#lib/nntp/index'
+import { RangeIndexer, type ReleaseChunk } from '#lib/edge/index'
+import { identityKey } from '#lib/release/keys'
+import BatchStore from '#services/batch_store'
+import Corpus from '#services/corpus'
+import RuleSets from '#services/rule_sets'
+import { PENDING } from '#services/pending_ladder'
+
+const inflate = promisify(unzip)
+
+interface NamedRow {
+  release_key: Buffer
+  identity_key: Buffer
+  group_id: number
+  named: string
+  first_article: number
+  last_article: number
+  file_total: number | null
+  tries: number
+}
+
+/** The namer gives a release up after this many failed moves (its articles gone from the server and the corpus). */
+const MAX_MOVES = 3
+
+/**
+ * The move into the catalog (REBUILD.md decision f): a pending release a probe named (status 4) gets its segments
+ * read again, from the corpus when its range is stored, else from the server (XOVER over its article span, one
+ * connection of its own, opened for it and closed after), through the TypeScript indexer, and is written as a
+ * release with the name the probe found (source weak-stem, the name as inner name), its chunk and its probe
+ * candidates, which the prober answers from probe_cache. Runs inside the prober.
+ */
+export class PendingNamer {
+  private client: XoverClient | null = null
+  private usedAt = 0
+
+  constructor(private corpusDir: string | undefined = coordinatorConfig.corpus.dir) {}
+
+  async run(stopped: () => boolean) {
+    while (!stopped()) {
+      let rows: NamedRow[] = []
+      try {
+        rows = await this.take(20)
+        for (const row of rows) {
+          if (stopped()) break
+          try {
+            await this.move(row)
+          } catch (error: any) {
+            logger.warn(
+              `Naming ${row.release_key.toString('hex')} "${row.named}" failed: ${error.message}`
+            )
+            await db.rawQuery(
+              `UPDATE pending_releases SET tries = tries + 1, probed_at = now() WHERE release_key = ?`,
+              [row.release_key]
+            )
+          }
+        }
+      } catch (error: any) {
+        logger.error(`Naming: ${error.message}`)
+      }
+      if (this.client && Date.now() - this.usedAt > coordinatorConfig.nntpIdleCloseMs) {
+        this.client.close()
+        this.client = null
+      }
+      if (rows.length === 0)
+        for (let waited = 0; waited < 30_000 && !stopped(); waited += 1000) await sleep(1000)
+    }
+    this.client?.close()
+    this.client = null
+  }
+
+  /** Named releases awaiting their move, the longest waiting first; those failed MAX_MOVES times stay as they are. */
+  async take(n: number): Promise<NamedRow[]> {
+    const { rows } = await db.rawQuery(
+      `SELECT release_key, identity_key, group_id, named, first_article, last_article, file_total, tries
+       FROM pending_releases WHERE status = ? AND tries < ? ORDER BY probed_at LIMIT ?`,
+      [PENDING.naming, MAX_MOVES, n]
+    )
+    return rows.map((r: any) => ({
+      ...r,
+      group_id: Number(r.group_id),
+      first_article: Number(r.first_article),
+      last_article: Number(r.last_article),
+      tries: Number(r.tries),
+    }))
+  }
+
+  /** One named release into the catalog. */
+  async move(row: NamedRow): Promise<void> {
+    const group = await db.from('groups').where('id', row.group_id).firstOrFail()
+    const range = await db
+      .from('ranges')
+      .where('group_id', row.group_id)
+      .where('start_id', '<=', row.first_article)
+      .where('end_id', '>=', row.first_article)
+      .orderBy('id')
+      .first()
+    if (!range) throw new Error(`no range of ${group.name} holds article ${row.first_article}`)
+    const text = await this.overviewText(group.name, row)
+    const rules = await RuleSets.current()
+    const ix = new RangeIndexer({
+      group: group.name,
+      start: row.first_article,
+      end: row.last_article,
+      rules: { ...rules, samplePercent: 0 },
+      idleChunks: 0,
+      edgeArticles: 0,
+    })
+    ix.addText(text, row.last_article + 1)
+    ix.finish()
+    const batch = ix.take()
+    const found = batch.releases.find((r) => identityKey(r.key).equals(row.identity_key))
+    if (!found)
+      throw new Error(
+        `its articles ${row.first_article}–${row.last_article} of ${group.name} no longer show it`
+      )
+    // The catalog row under the pending key (the identity, the group, the first article the edge saw).
+    const chunk: ReleaseChunk = {
+      ...found,
+      name: row.named,
+      source: 'weak-stem',
+      readable: true,
+      finished: true,
+      edge: false,
+      firstArticle: row.first_article,
+      fileTotal: row.file_total ?? found.fileTotal,
+      tool: undefined,
+    }
+    const candidates = batch.candidates
+      .filter((c) => c.releaseKey === found.key)
+      .map((c) => ({ ...c, upload: found.upload }))
+    await db.transaction(async (trx) => {
+      const blockId = await BatchStore.blockOf(trx, range)
+      await BatchStore.writeReleases(
+        trx,
+        { groupId: row.group_id, blockId },
+        row.first_article,
+        [chunk],
+        candidates
+      )
+      await trx.rawQuery(
+        `UPDATE releases SET inner_name = ?, inspected_at = now() WHERE release_key = ? AND inner_name IS NULL`,
+        [row.named.slice(0, 255), row.release_key]
+      )
+      await trx.rawQuery(
+        `UPDATE pending_releases SET status = ?, probed_at = now() WHERE release_key = ?`,
+        [PENDING.named, row.release_key]
+      )
+    })
+  }
+
+  /** The overview lines of the release's article span: the stored corpus chunks when its range is stored, else XOVER. */
+  private async overviewText(groupName: string, row: NamedRow): Promise<string> {
+    if (this.corpusDir) {
+      const stored = await db
+        .from('ranges')
+        .where('group_id', row.group_id)
+        .whereNotNull('stored_at')
+        .where('start_id', '<=', row.last_article)
+        .where('end_id', '>=', row.first_article)
+        .orderBy('start_id')
+      const parts: string[] = []
+      let covered = true
+      for (const r of stored) {
+        const range = { group: groupName, start: Number(r.start_id), end: Number(r.end_id) }
+        const size = coordinatorConfig.chunkArticles
+        for (
+          let from =
+            range.start +
+            Math.floor((Math.max(row.first_article, range.start) - range.start) / size) * size;
+          from <= Math.min(row.last_article, range.end);
+          from += size
+        ) {
+          const raw = await Corpus.read(
+            range,
+            { from, to: Math.min(from + size - 1, range.end) },
+            this.corpusDir
+          )
+          if (!raw) {
+            covered = false
+            break
+          }
+          parts.push((await inflate(raw)).toString('latin1'))
+        }
+        if (!covered) break
+      }
+      if (
+        covered &&
+        parts.length > 0 &&
+        stored.length > 0 &&
+        Number(stored[0].start_id) <= row.first_article &&
+        Number(stored[stored.length - 1].end_id) >= row.last_article
+      ) {
+        return parts.join('')
+      }
+    }
+    this.client ??= await XoverClient.connect({
+      host: env.get('NNTP_HOST')!,
+      port: env.get('NNTP_PORT', 563),
+      user: env.get('NNTP_USER'),
+      password: env.get('NNTP_PASSWORD'),
+      secure: env.get('NNTP_SECURE', true),
+      compress: true,
+      timeoutMs: coordinatorConfig.nntpTimeoutMs,
+    })
+    this.usedAt = Date.now()
+    try {
+      await this.client.group(groupName)
+      return (await this.client.xoverRaw(row.first_article, row.last_article)).toString('latin1')
+    } catch (error) {
+      this.client.close()
+      this.client = null
+      throw error
+    }
+  }
+}

app/services/prober.ts

@@ -9,12 +9,16 @@ import { inspect, innerName, PROBE_VERSION, type Inspection } from '#lib/probe/i
 import { NntpShare } from '#services/nntp_shares'
 import { DiskSpace } from '#services/disk_space'
 import { decidePar2Completeness } from '#services/par2_completeness'
+import PendingLadder, { type PendingProbe } from '#services/pending_ladder'
+import { PendingNamer } from '#services/pending_namer'
 
 interface Candidate {
   id: number
   releaseId: number
   kind: string
   messageId: string
+  /** A pending release's target (the pending ladder), not a catalog candidate: id and releaseId are 0. */
+  pending?: PendingProbe
 }
 
 type Outcome = { candidate: Candidate; status: 'done' | 'missing' | 'error' | 'waiting'; inspection?: Inspection; cached?: boolean }
@@ -34,6 +38,8 @@ const TIERS: { kind: string; unreadable: boolean }[] = [
   { kind: '7z-tail', unreadable: false },
 ]
 const FLUSH_MS = 2000
+/** Between sweeps that discard the waiting releases of dropped signatures. */
+const SWEEP_MS = 5 * 60_000
 
 /**
  * JSON for a jsonb parameter, without NUL characters: jsonb can't hold \u0000 (a name read from an article had one),
@@ -72,13 +78,19 @@ export class Prober {
 
   /** Per minute: what was read, and where the lanes' time went (ms summed over lanes; flushes ms on their own). */
   private static freshStats() {
-    return { articles: 0, bytes: 0, done: 0, missing: 0, error: 0, cached: 0, waitMs: 0, idleMs: 0, fetchMs: 0, inspectMs: 0, flushMs: 0, claimMs: 0, since: Date.now() }
+    return { articles: 0, bytes: 0, done: 0, missing: 0, error: 0, cached: 0, pending: 0, discarded: 0, waitMs: 0, idleMs: 0, fetchMs: 0, inspectMs: 0, flushMs: 0, claimMs: 0, since: Date.now() }
   }
+  /** The pending ladder's namer: named pending releases into the catalog, alongside the lanes. */
+  private namer = new PendingNamer()
+  private sweptAt = 0
 
   async run() {
     // A previous run's claims: one prober at a time.
     const { rowCount } = await db.rawQuery(`UPDATE probe_candidates SET status = 'waiting' WHERE status = 'taken'`)
     if (rowCount) logger.info(`Probing: ${rowCount} candidates from a previous run back in the queue.`)
+    const released = await PendingLadder.release()
+    if (released) logger.info(`Probing: ${released} pending releases from a previous run back on the ladder.`)
+    const namer = this.namer.run(() => this.stopped).catch((error) => logger.error(`Naming failed: ${error.message}`))
     logger.info(`Probing with ${this.share.min}–${this.share.max} connections (its share of the account's).`)
     const flusher = (async () => {
       while (!this.stopped || this.outcomes.length > 0) {
@@ -108,6 +120,7 @@ export class Prober {
       for (let waited = 0; waited < coordinatorConfig.nntpShares.everyMs && !this.stopped; waited += 1000) await sleep(1000)
     }
     await Promise.all(this.lanes)
+    await namer
     await this.share.leave().catch(() => {})
     // What was claimed and not probed goes back.
     for (const candidate of this.queue.splice(0)) this.outcomes.push({ candidate, status: 'waiting' })
@@ -227,6 +240,37 @@ export class Prober {
   private async claimTiers() {
     // The queue empties, the lanes go dry and the share shrinks to its minimum until there's room again.
     if (await this.space.short()) return
+    await this.claimCatalog()
+    await this.claimPending()
+  }
+
+  /** The pending ladder's share of a claim: at least pendingShare of the batch, and whatever the catalog's tiers left. */
+  private async claimPending() {
+    if (Date.now() - this.sweptAt > SWEEP_MS) {
+      this.sweptAt = Date.now()
+      this.stats.discarded += await PendingLadder.discard()
+    }
+    const want = Math.max(Math.floor(this.batch * coordinatorConfig.probe.pendingShare), this.batch - this.queue.length)
+    const probes = await PendingLadder.claim(want)
+    if (probes.length === 0) return
+    const { rows: cached } = await db.rawQuery(`SELECT message_id, status, result FROM probe_cache WHERE message_id = ANY(?) AND version = ?`, [
+      probes.map((p) => p.messageId),
+      PROBE_VERSION,
+    ])
+    const known = new Map(cached.map((c: any) => [c.message_id, c]))
+    for (const probe of probes) {
+      const candidate: Candidate = { id: 0, releaseId: 0, kind: probe.kind, messageId: probe.messageId, pending: probe }
+      const hit: any = known.get(probe.messageId)
+      if (hit) {
+        this.stats.cached++
+        this.outcomes.push({ candidate, status: hit.status, inspection: hit.result ?? undefined, cached: true })
+      } else {
+        this.queue.push(candidate)
+      }
+    }
+  }
+
+  private async claimCatalog() {
     for (const { kind, unreadable } of TIERS) {
       const want = this.batch - this.queue.length
       if (want <= 0) return
@@ -277,18 +321,25 @@ export class Prober {
 
   /** The outcomes so far: candidates' status, inspections, and the releases' passworded and inner name. */
   private async flush() {
-    const outcomes = this.outcomes.splice(0)
-    if (outcomes.length === 0) return
-    for (const o of outcomes) if (o.status !== 'waiting') this.stats[o.status]++
+    const taken = this.outcomes.splice(0)
+    if (taken.length === 0) return
+    for (const o of taken) if (o.status !== 'waiting') this.stats[o.status]++
+    // The ladder's outcomes apart from the catalog's: the next target, a name for the namer, or the end of the climb.
+    const pending = taken.filter((o) => o.candidate.pending)
+    const outcomes = taken.filter((o) => !o.candidate.pending)
+    this.stats.pending += pending.filter((o) => o.status !== 'waiting').length
     await db.transaction(async (trx) => {
+      if (outcomes.length > 0) {
         await trx.rawQuery(
           `UPDATE probe_candidates pc SET status = v.status FROM jsonb_to_recordset(?::jsonb) AS v(id bigint, status text) WHERE pc.id = v.id`,
           [jsonb(outcomes.map((o) => ({ id: o.candidate.id, status: o.status })))]
         )
+      }
+      await PendingLadder.record(pending.map((o) => ({ probe: o.candidate.pending!, status: o.status, inspection: o.inspection })), trx)
       // Final answers read from the news server, kept by message-id for the next catalog (a reset).
       // One per message-id: the same article is a candidate of more than one release, and ON CONFLICT can't take a
       // row twice in one statement.
-      const fresh = [...new Map(outcomes.filter((o) => !o.cached && o.status !== 'waiting').map((o) => [o.candidate.messageId, o])).values()]
+      const fresh = [...new Map(taken.filter((o) => !o.cached && o.status !== 'waiting').map((o) => [o.candidate.messageId, o])).values()]
       if (fresh.length > 0) {
         await trx.rawQuery(
           `INSERT INTO probe_cache (message_id, status, result, version)
@@ -365,7 +416,8 @@ export class Prober {
     const { articles, bytes, done, missing, error } = this.stats
     logger.info(
       `Probed ${articles} articles in ${seconds.toFixed(0)} s: ${(articles / seconds).toFixed(1)}/s, ${((bytes * 8) / seconds / 1e6).toFixed(0)} Mbit/s; ` +
-        `${done} inspected, ${missing} missing, ${error} not yEnc; ${this.stats.cached} from the probe cache. ` +
+        `${done} inspected, ${missing} missing, ${error} not yEnc; ${this.stats.cached} from the probe cache; ` +
+        `the pending ladder ${this.stats.pending} probed, ${this.stats.discarded} discarded by signature. ` +
         `Lanes: fetching ${(this.stats.fetchMs / 1000).toFixed(0)} s, waiting for candidates ${(this.stats.waitMs / 1000).toFixed(0)} s, ` +
         `idle ${(this.stats.idleMs / 1000).toFixed(0)} s, inspecting ${(this.stats.inspectMs / 1000).toFixed(1)} s; ` +
         `claims ${(this.stats.claimMs / 1000).toFixed(1)} s, flushes ${(this.stats.flushMs / 1000).toFixed(1)} s.`

app/services/range_blocks.ts

@@ -1,31 +0,0 @@
-import db from '@adonisjs/lucid/services/db'
-
-/**
- * range_blocks (migration 3900): per group, each run of consecutive done ranges with its posting time span, read by
- * release_final(). A few thousand rows from ~650k ranges, rebuilt whole in one statement each time; a run breaks at a
- * range that isn't done or doesn't start where the one before ended.
- */
-class RangeBlocks {
-  async rebuild(): Promise<number> {
-    return db.transaction(async (trx) => {
-      await trx.rawQuery(`DELETE FROM range_blocks`)
-      const result = await trx.rawQuery(
-        `WITH r AS (
-           SELECT group_id, start_id, end_id, status = 'done' AS done, posted_low, posted_high,
-                  min(start_id) OVER (PARTITION BY group_id) AS group_first,
-                  lag(status = 'done') OVER w AS done_before, lag(end_id) OVER w AS end_before
-           FROM ranges WINDOW w AS (PARTITION BY group_id ORDER BY start_id)),
-         runs AS (
-           SELECT *, sum(CASE WHEN done IS DISTINCT FROM done_before OR end_before IS DISTINCT FROM start_id - 1 THEN 1 ELSE 0 END)
-                       OVER (PARTITION BY group_id ORDER BY start_id) AS run
-           FROM r)
-         INSERT INTO range_blocks (group_id, start_id, end_id, posted_low, posted_high, at_group_start)
-         SELECT group_id, min(start_id), max(end_id), min(posted_low), max(posted_high), min(start_id) = min(group_first)
-         FROM runs WHERE done GROUP BY group_id, run`
-      )
-      return result.rowCount ?? 0
-    })
-  }
-}
-
-export default new RangeBlocks()

app/services/release_judge.ts

@@ -1,5 +1,6 @@
 import db from '@adonisjs/lucid/services/db'
 import logger from '@adonisjs/core/services/logger'
+import { SOURCES } from '#lib/release/source'
 
 /** Raised when a rule changes: verdicts of an older version are judged again (a full round, right away). */
 export const JUDGE_VERSION = 1
@@ -88,10 +89,10 @@ export const RULES: Rule[] = [
     // regex on all of releases was 45 s, every round (LOG.md 2026-10-19 (80)).
     sql: (scoped) =>
       scoped
-        ? `SELECT r.id FROM releases r ${only(scoped, 'r.id')}
-          WHERE r.poster ~* '${SCRAPED_POSTERS}' AND r.source = 'subject' AND r.file_total IS NULL`
-        : `SELECT r.id FROM posters p JOIN releases r ON left(r.poster, 200) = p.poster
-          WHERE p.poster ~* '${SCRAPED_POSTERS}' AND r.poster ~* '${SCRAPED_POSTERS}' AND r.source = 'subject' AND r.file_total IS NULL`,
+        ? `SELECT r.id FROM releases r ${only(scoped, 'r.id')} JOIN posters p ON p.id = r.poster_id
+          WHERE p.poster ~* '${SCRAPED_POSTERS}' AND r.source = ${SOURCES.subject} AND r.file_total IS NULL`
+        : `SELECT r.id FROM posters p JOIN releases r ON r.poster_id = p.id
+          WHERE p.poster ~* '${SCRAPED_POSTERS}' AND r.source = ${SOURCES.subject} AND r.file_total IS NULL`,
   },
   {
     name: 'locked',
@@ -102,10 +103,10 @@ export const RULES: Rule[] = [
   {
     name: 'unusable',
     says: 'random names, no name inside, no par2 or inner-name merge, no spot, a day after settling, and final',
-    // Final (release_final): next to an unread range its par2 or the rest of its files may still come (LOG.md 2026-10-16 (67)).
+    // Finished (settled_at): a provisional tail release's par2 or the rest of its files may still come.
     sql: (scoped) => `SELECT r.id FROM releases r ${only(scoped, 'r.id')}
           WHERE r.settled_at < now() - interval '1 day' AND NOT r.readable AND r.inner_name IS NULL
-            AND release_final(r.group_id, r.first_article, r.last_article, r.posted_at, r.last_posted_at, r.complete)
+            AND r.settled_at IS NOT NULL
             AND NOT EXISTS (SELECT 1 FROM release_merges m WHERE m.into_id = r.id)
             AND NOT EXISTS (SELECT 1 FROM release_spots s WHERE s.release_id = r.id)
             AND NOT EXISTS (SELECT 1 FROM par2_sets p WHERE p.release_id = r.id)`,

app/services/search_service.ts

@@ -1,6 +1,7 @@
 import db from '@adonisjs/lucid/services/db'
 import logger from '@adonisjs/core/services/logger'
 import NzbService from '#services/nzb_service'
+import { SOURCES, sourceName } from '#lib/release/source'
 
 export type SortOrder = 'newest' | 'oldest' | 'largest' | 'files' | 'name'
 
@@ -51,7 +52,7 @@ export interface ReleaseSummary {
   size: number
   readable: boolean
   complete: boolean | null
-  /** The complete verdict is definite: done ranges cover the release's posting times and 6 hours either side (release_final). */
+  /** The complete verdict is definite: the release is finished (voyager 3: the window closed on it; a tail release is provisional until then). */
   final: boolean
   category: number
   settled: boolean
@@ -91,27 +92,32 @@ class SearchService {
     // Judged junk (app/services/release_judge.ts) is left out unless asked for.
     if (!params.judged) where.push('NOT EXISTS (SELECT 1 FROM release_verdicts v WHERE v.release_id = r.id)')
     const bindings: unknown[] = []
-    // Every word in the name or the inner name the prober found, through their trigram indexes; "The.Block.AU" and "the block au" alike.
+    // Every word in one of the release's texts (its name, the names the prober found inside; release_texts, by its
+    // trigram index, which yields ids: never an OR across the two tables); "The.Block.AU" and "the block au" alike.
     for (const word of (params.q ?? '').split(/[\s._-]+/).filter(Boolean).slice(0, 8)) {
-      where.push('(r.name ILIKE ? OR r.inner_name ILIKE ?)')
-      bindings.push(contains(word), contains(word))
+      where.push('r.id IN (SELECT t.release_id FROM release_texts t WHERE t.text ILIKE ?)')
+      bindings.push(contains(word))
     }
     if (params.readable) where.push('(r.readable OR r.inner_name IS NOT NULL)')
     if (params.complete) where.push('r.complete')
-    // Incomplete only when final: a release next to unread ranges may yet get its missing files (release_final).
-    if (params.notIncomplete) where.push('(r.complete IS NOT FALSE OR NOT release_final(r.group_id, r.first_article, r.last_article, r.posted_at, r.last_posted_at, r.complete))')
+    // Incomplete only when final: a provisional tail release may yet get its missing files.
+    if (params.notIncomplete) where.push('(r.complete IS NOT FALSE OR r.settled_at IS NULL)')
     if (params.unlocked) where.push('r.passworded IS NOT TRUE')
     if (params.source) {
+      const code = (SOURCES as Record<string, number>)[params.source]
+      if (code === undefined) where.push('false')
+      else {
         where.push('r.source = ?')
-      bindings.push(params.source)
+        bindings.push(code)
+      }
     }
     // A poster search: the matching posters first (posters, by trigram), each one's releases then by
     // releases_poster_newest_idx (below). "poster ILIKE" on releases walked all 50M rows newest first: 30+ min for a
     // poster with few releases; as a filter on the posters found the planner still walked them (LOG.md 2026-10-16 (70)).
-    let posters: string[] | null = null
+    let posters: number[] | null = null
     if (params.poster) {
-      const { rows: matched } = await db.rawQuery(`SELECT poster FROM posters WHERE poster ILIKE ? LIMIT ${MAX_POSTERS}`, [contains(params.poster)])
-      posters = matched.map((m: { poster: string }) => m.poster)
+      const { rows: matched } = await db.rawQuery(`SELECT id FROM posters WHERE poster ILIKE ? LIMIT ${MAX_POSTERS}`, [contains(params.poster)])
+      posters = matched.map((m: { id: string }) => Number(m.id))
     }
     if (params.categories?.length) {
       where.push('r.category = ANY(?)')
@@ -143,14 +149,14 @@ class SearchService {
     const whereSql = where.join(' AND ')
     const countLimit = offset + (params.countLimit ?? COUNT_LIMIT)
     if (posters !== null && posters.length === 0) return { page, perPage, offset, total: 0, totalCapped: false, releases: [] as ReleaseSummary[] }
-    // Per poster found (by first 200 characters, as posters and the index keep them; a btree entry holds at most 2,704
-    // bytes): its releases matching everything else, the first offset + perPage in the order asked; then merged.
+    // Per poster found: its releases matching everything else (releases_poster_newest_idx), the first offset + perPage
+    // in the order asked; then merged.
     const source =
       posters === null
         ? { from: 'releases r', where: whereSql, bindings: [] as unknown[] }
         : {
-            from: `unnest(?::text[]) AS p(poster) CROSS JOIN LATERAL (
-                     SELECT r.* FROM releases r WHERE left(r.poster, 200) = p.poster AND ${whereSql} ORDER BY ${order} LIMIT ${offset + perPage}) r`,
+            from: `unnest(?::int[]) AS p(id) CROSS JOIN LATERAL (
+                     SELECT r.* FROM releases r WHERE r.poster_id = p.id AND ${whereSql} ORDER BY ${order} LIMIT ${offset + perPage}) r`,
             where: 'true',
             bindings: [posters] as unknown[],
           }
@@ -158,16 +164,17 @@ class SearchService {
       posters === null
         ? db.rawQuery(`SELECT count(*) AS total FROM (SELECT 1 FROM releases r WHERE ${whereSql} LIMIT ${countLimit + 1}) x`, bindings)
         : db.rawQuery(
-            `SELECT count(*) AS total FROM (SELECT 1 FROM unnest(?::text[]) AS p(poster) CROSS JOIN LATERAL (
-               SELECT 1 FROM releases r WHERE left(r.poster, 200) = p.poster AND ${whereSql} LIMIT ${countLimit + 1}) x LIMIT ${countLimit + 1}) y`,
+            `SELECT count(*) AS total FROM (SELECT 1 FROM unnest(?::int[]) AS p(id) CROSS JOIN LATERAL (
+               SELECT 1 FROM releases r WHERE r.poster_id = p.id AND ${whereSql} LIMIT ${countLimit + 1}) x LIMIT ${countLimit + 1}) y`,
             [posters, ...bindings]
           )
     const [{ rows }, { rows: counted }] = await Promise.all([
       db.rawQuery(
-        `SELECT r.id, r.name, r.source, r.poster, r.groups, r.posted_at, r.file_total, r.files, r.bytes, r.readable, r.complete, r.category, r.settled_at, r.passworded, r.inner_name,
+        `SELECT r.id, r.name, r.source, po.poster, gs.groups, r.posted_at, r.file_total, r.files, r.bytes, r.readable, r.complete, r.category, r.settled_at, r.passworded, r.inner_name,
                 EXISTS (SELECT 1 FROM release_passwords p WHERE p.release_id = r.id) AS password_known,
-                release_final(r.group_id, r.first_article, r.last_article, r.posted_at, r.last_posted_at, r.complete) AS final
-         FROM ${source.from} WHERE ${source.where} ORDER BY ${order} LIMIT ? OFFSET ?`,
+                r.settled_at IS NOT NULL AS final
+         FROM ${source.from} LEFT JOIN posters po ON po.id = r.poster_id LEFT JOIN group_sets gs ON gs.id = r.group_set_id
+         WHERE ${source.where} ORDER BY ${order} LIMIT ? OFFSET ?`,
         [...source.bindings, ...bindings, perPage, offset]
       ),
       counting,
@@ -182,9 +189,9 @@ class SearchService {
       releases: (rows as any[]).map((r): ReleaseSummary => ({
         id: Number(r.id),
         name: r.name,
-        source: r.source,
-        poster: r.poster,
-        groups: r.groups,
+        source: sourceName(r.source),
+        poster: r.poster ?? null,
+        groups: r.groups ?? [],
         postedAt: r.posted_at,
         fileTotal: r.file_total,
         fileCount: Number(r.files),
@@ -203,20 +210,24 @@ class SearchService {
 
   /** A release with its files, read from its chunks. */
   async release(id: number) {
-    const r = await db.from('releases').where('id', id).first()
+    const {
+      rows: [r],
+    } = await db.rawQuery(
+      `SELECT r.*, po.poster, gs.groups FROM releases r LEFT JOIN posters po ON po.id = r.poster_id LEFT JOIN group_sets gs ON gs.id = r.group_set_id WHERE r.id = ?`,
+      [id]
+    )
     if (!r) return null
-    const [files, { rows: inspections }, { rows: finality }, passwords] = await Promise.all([
+    const [files, { rows: inspections }, passwords] = await Promise.all([
       NzbService.files(id),
       db.rawQuery(`SELECT kind, result, created_at FROM release_inspections WHERE release_id = ? ORDER BY candidate_id`, [id]),
-      db.rawQuery(`SELECT release_final(?, ?, ?, ?, ?, ?) AS final`, [r.group_id, r.first_article, r.last_article, r.posted_at, r.last_posted_at, r.complete]),
       NzbService.passwords(id),
     ])
     return {
       id: Number(r.id),
       name: r.name,
-      source: r.source,
-      poster: r.poster,
-      groups: r.groups,
+      source: sourceName(r.source),
+      poster: r.poster ?? null,
+      groups: r.groups ?? [],
       postedAt: r.posted_at,
       fileTotal: r.file_total,
       fileCount: Number(r.files),
@@ -224,11 +235,10 @@ class SearchService {
       segments: Number(r.segments),
       readable: r.readable,
       complete: r.complete,
-      final: finality[0].final,
+      final: r.settled_at !== null,
       hasPar2: r.has_par2,
       category: r.category,
       firstSeenAt: r.first_seen_at,
-      lastSeenAt: r.last_seen_at,
       settledAt: r.settled_at,
       passworded: r.passworded,
       /** Known passwords (spots, their NZBs), the most trusted first; the first is in the NZB. */

app/services/settler.ts

@@ -1,49 +0,0 @@
-import db from '@adonisjs/lucid/services/db'
-
-/** Releases unchanged for this long are settled (ranges are read in order; the merger adds within minutes). */
-const QUIET_HOURS = 6
-/**
- * History: a release whose last post is QUIET_HOURS old gets nothing new from the tail, only from the
- * neighbouring ranges' fragments, which the merger adds within minutes: settled once unchanged this long.
- * Waiting QUIET_HOURS after it was read held every history release back from phase 2 for six hours after
- * each reset (the hourly sweeps' resets kept the prober idle; LOG.md 2026-10-12 (13)).
- */
-const QUIET_HISTORY_MINUTES = 30
-const BATCH = 50_000
-
-/**
- * Settles releases (REBUILD.md, part 8): once nothing has been added for
- * QUIET_HOURS (QUIET_HISTORY_MINUTES once its last post is that old), its completeness is decided: complete when every file the
- * [i/N] counter announced was found, unknown without a counter (phase 2
- * decides those by their par2 sets: app/services/par2_completeness.ts).
- * Phase 2 picks its work from settled releases.
- */
-class Settler {
-  /** Up to `batches` batches of BATCH (all there are without). */
-  async run(batches = Infinity): Promise<number> {
-    let settled = 0
-    for (let batch = 0; batch < batches; batch++) {
-      // Their probe candidates become claimable (Prober.claim), the unreadable ones first.
-      const { rows } = await db.rawQuery(
-        `WITH settled AS (
-           UPDATE releases SET settled_at = now(), complete = CASE WHEN file_total IS NULL THEN complete ELSE files >= file_total END
-           WHERE id IN (SELECT id FROM releases WHERE settled_at IS NULL AND last_seen_at < now() - make_interval(mins => ?)
-             AND (last_seen_at < now() - make_interval(hours => ?) OR coalesce(last_posted_at, posted_at) < now() - make_interval(hours => ?)) LIMIT ?)
-           RETURNING id, readable, files, bytes, poster, sidecar_only),
-         marked AS (UPDATE probe_candidates pc SET ready = true, unreadable = NOT s.readable FROM settled s WHERE pc.release_id = s.id),
-         -- A small one of sidecars only (the batch store's sidecar_only) may be an upload's .nfo or .nzb that came apart
-         -- from it: the merger's sidecar fold looks at them.
-         queued AS (INSERT INTO fold_candidates (release_id) SELECT id FROM settled WHERE sidecar_only AND files <= 3 AND bytes < 1000000 AND poster IS NOT NULL
-           ON CONFLICT DO NOTHING)
-         SELECT count(*)::int AS n FROM settled`,
-        [QUIET_HISTORY_MINUTES, QUIET_HOURS, QUIET_HOURS, BATCH]
-      )
-      const n = rows[0].n
-      settled += n
-      if (n < BATCH) return settled
-    }
-    return settled
-  }
-}
-
-export default new Settler()

app/services/sidecar_folder.ts

@@ -56,7 +56,7 @@ export class SidecarFolder {
     // A candidate whose release went (merged, deleted) leaves with its batch: a separate scan for those over the whole
     // queue took 5+ min a round at 13M (LOG.md 2026-10-17 (72)).
     const { rows: queued } = await db.rawQuery(
-      `SELECT f.release_id AS queued_id, r.id, r.release_key, r.group_id, r.poster, r.posted_at, r.files, r.file_total, r.readable
+      `SELECT f.release_id AS queued_id, r.id, r.group_id, r.poster_id, r.posted_at, r.files, r.file_total, r.readable
        FROM fold_candidates f LEFT JOIN releases r ON r.id = f.release_id ORDER BY f.created_at LIMIT ?`,
       [BATCH]
     )
@@ -70,10 +70,10 @@ export class SidecarFolder {
       round.sidecars++
       if (!c.posted_at) continue
       const { rows: siblings } = await db.rawQuery(
-        `SELECT r.id, r.release_key, r.group_id, r.files, r.file_total, r.posted_at, r.readable, r.name FROM releases r
+        `SELECT r.id, r.group_id, r.files, r.file_total, r.posted_at, r.readable, r.name FROM releases r
          WHERE r.posted_at BETWEEN ?::timestamptz - make_interval(mins => ?) AND ?::timestamptz + make_interval(mins => ?)
-           AND r.group_id = ? AND r.poster = ? AND r.id <> ? AND r.files > ?`,
-        [c.posted_at, WINDOW_MINUTES, c.posted_at, WINDOW_MINUTES, c.group_id, c.poster, c.id, c.files]
+           AND r.group_id = ? AND r.poster_id = ? AND r.id <> ? AND r.files > ?`,
+        [c.posted_at, WINDOW_MINUTES, c.posted_at, WINDOW_MINUTES, c.group_id, c.poster_id, c.id, c.files]
       )
       const sibling = pickSibling(files, c.file_total ? Number(c.file_total) : null, siblings.map((s: any) => ({ ...s, id: Number(s.id), files: Number(s.files), file_total: s.file_total === null ? null : Number(s.file_total) })))
       if (!sibling) continue

app/services/spot_inspector.ts

@@ -9,6 +9,9 @@ import { decodeSpotBinary } from '#lib/spotnet/index'
 import { parseNzb, type ParsedNzb } from '#lib/nzb/index'
 import { parseSubject } from '#lib/headers/index'
 import { releaseIdentity } from '#lib/release/index'
+import { identityKey, releaseKey } from '#lib/release/keys'
+import { SOURCES } from '#lib/release/source'
+import CatalogRefs from '#services/catalog_refs'
 
 /** Spots claimed per round, newest first. */
 const BATCH = 20
@@ -251,8 +254,8 @@ export class SpotInspector {
     }
     if (wanted.length === 0) return 0
     const { rows } = await db.rawQuery(
-      `SELECT id, identity_key, group_id, posted_at FROM releases WHERE identity_key = ANY(?) AND group_id = ANY(?)`,
-      [[...new Set(wanted.map((w) => w.key))], [...new Set(wanted.flatMap((w) => w.groups))]]
+      `SELECT id, encode(identity_key, 'hex') AS identity_key, group_id, posted_at FROM releases WHERE identity_key = ANY(?::bytea[]) AND group_id = ANY(?)`,
+      [[...new Set(wanted.map((w) => w.key))].map((k) => identityKey(k)), [...new Set(wanted.flatMap((w) => w.groups))]]
     )
     const byKey = new Map<string, { id: number; group: number; at: number | null }[]>()
     for (const r of rows) {
@@ -275,7 +278,7 @@ export class SpotInspector {
     await db.transaction(async (trx) => {
       // A release of the spot's own goes now that the header pass built its files' releases.
       if (spot.release_id) {
-        const own = await trx.from('releases').where('id', spot.release_id).where('source', 'spot').select('id').first()
+        const own = await trx.from('releases').where('id', spot.release_id).where('source', SOURCES.spot).select('id').first()
         if (own) {
           await trx.from('release_spots').where('release_id', own.id).delete()
           await trx.from('release_passwords').where('release_id', own.id).delete()
@@ -335,16 +338,19 @@ export class SpotInspector {
     const groups = [...new Set(files.flatMap((f: any) => f.groups ?? []))]
     const groupId = groups.length > 0 ? await this.groupId(groups[0]) : null
     const key = createHash('sha1').update(`spot\n${spotId}`).digest('hex')
+    // Its poster and group set by id (voyager 3, migration 6000), as the batch store stores its releases'.
+    const posterId = spot.poster ? await CatalogRefs.posterId(db, spot.poster) : null
+    const groupSetId = await CatalogRefs.groupSetId(db, groups)
     const [row] = await db
       .table('releases')
       .insert({
-        release_key: `spot:${key}`,
-        identity_key: key,
+        release_key: releaseKey(key, groupId, null),
+        identity_key: identityKey(key),
         group_id: groupId,
+        group_set_id: groupSetId,
+        poster_id: posterId,
         name: spot.title.slice(0, 1000),
-        source: 'spot',
-        poster: spot.poster,
-        groups: JSON.stringify(groups),
+        source: SOURCES.spot,
         posted_at: spot.posted_at,
         last_posted_at: spot.posted_at,
         files: files.length,
@@ -363,8 +369,8 @@ export class SpotInspector {
     const releaseId = Number(row.id)
     await db.transaction(async (trx) => {
       await trx.table('release_spots').insert({ release_id: releaseId, spot_id: spotId, files: files.length }).onConflict(['release_id', 'spot_id']).ignore()
-      // Its poster for poster search, as the batch store adds its releases' (migration 4000).
-      if (spot.poster) await trx.rawQuery(`INSERT INTO posters (poster) VALUES (left(?, 200)) ON CONFLICT DO NOTHING`, [spot.poster])
+      // Its title, searchable (release_texts).
+      await trx.rawQuery(`INSERT INTO release_texts (release_id, kind, text) VALUES (?, 1, left(?, 255)) ON CONFLICT DO NOTHING`, [releaseId, spot.title])
       if (spot.password) {
         await trx.table('release_passwords').insert({ release_id: releaseId, password: spot.password, source: spot.password_source ?? 'spot-text', spot_id: spotId }).onConflict(['release_id', 'password']).ignore()
       }

app/services/spotter.ts

@@ -1,5 +1,6 @@
 import db from '@adonisjs/lucid/services/db'
 import logger from '@adonisjs/core/services/logger'
+import { SOURCES } from '#lib/release/source'
 import env from '#start/env'
 import coordinatorConfig from '#config/coordinator'
 import { MissingArticle, XoverClient } from '#lib/nntp/index'
@@ -183,7 +184,7 @@ export class Spotter {
       await trx.from('release_spots').where('spot_id', targetId).delete()
       await trx.from('release_passwords').where('spot_id', targetId).delete()
       // A release the spot made of its own goes with it.
-      if (target.release_id) await trx.from('releases').where('id', target.release_id).where('source', 'spot').delete()
+      if (target.release_id) await trx.from('releases').where('id', target.release_id).where('source', SOURCES.spot).delete()
     })
     logger.info(`Spot "${target.title}" removed by ${personal ? 'its poster' : 'a moderator'}.`)
   }

commands/coordinator_run.ts

@@ -3,9 +3,6 @@ import type { CommandOptions } from '@adonisjs/core/types/ace'
 import { setTimeout as sleep } from 'node:timers/promises'
 import { Prefetcher } from '#services/prefetcher'
 import Planner from '#services/planner'
-import FragmentMerger from '#services/fragment_merger'
-import Settler from '#services/settler'
-import RangeBlocks from '#services/range_blocks'
 import Leases from '#services/leases'
 import NntpPool from '#services/nntp_pool'
 import { NntpShare } from '#services/nntp_shares'
@@ -13,38 +10,38 @@ import coordinatorConfig from '#config/coordinator'
 
 const TICK_MS = 2000
 const PLAN_EVERY_MS = 60 * 60 * 1000
-const MERGE_EVERY_MS = 60 * 1000
-const MERGE_BUDGET_MS = 30 * 1000
 const STOP_WAIT_MS = 15 * 1000
-/**
- * Settling in rounds of at most SETTLE_BATCHES batches every few minutes: once an hour, ~3M releases rewrote their
- * rows and index entries in one go (2.65M in 30 min), the NVMe saturated and the batch stores queued on their locks
- * (LOG.md 2026-10-16 (67)).
- */
-const SETTLE_EVERY_MS = 5 * 60 * 1000
-const SETTLE_BATCHES = 4
-/** range_blocks (release_final) after ranges finish. */
-const BLOCKS_EVERY_MS = 5 * 60 * 1000
+/** Leases whose worker went quiet after reading everything are finished every few minutes. */
+const ABANDONED_EVERY_MS = 5 * 60 * 1000
 
 /**
- * The coordinator's background work (REBUILD.md, part 1): the prefetcher,
- * merging fragments of finished ranges every minute, and every hour
- * planning ranges for new articles and settling quiet releases. The API (leases, stream,
- * batches) runs in the web server.
+ * The coordinator's background work (REBUILD.md, part 1): the prefetcher, the tail's ranges every few minutes,
+ * new groups' history every hour, and abandoned leases. The API (leases, stream, batches) runs in the web server.
+ * Voyager 3: the settler and the fragment merger are gone (the window finishes a release once; the block merger
+ * joins edge releases).
  */
 export default class CoordinatorRun extends BaseCommand {
   static commandName = 'coordinator:run'
   static description = 'Prefetches ranges into the spool and plans new ones'
   static options: CommandOptions = { startApp: true, staysAlive: true }
 
-  @flags.string({ description: 'This process\'s share of the prefetching, "i/n" (ranges with id % n = i - 1); only 1/n plans, merges and settles' })
+  @flags.string({
+    description:
+      'This process\'s share of the prefetching, "i/n" (ranges with id % n = i - 1); only 1/n plans, merges and settles',
+  })
   declare shard?: string
 
   async run() {
     const [index, count] = (this.shard ?? '1/1').split('/').map(Number)
-    if (!(count >= 1 && index >= 1 && index <= count)) throw new Error(`--shard: "i/n" with 1 <= i <= n, not "${this.shard}"`)
+    if (!(count >= 1 && index >= 1 && index <= count))
+      throw new Error(`--shard: "i/n" with 1 <= i <= n, not "${this.shard}"`)
     const prefetcher = new Prefetcher({ index: index - 1, count })
-    const share = new NntpShare(count > 1 ? `coordinator ${index}/${count}` : 'coordinator', coordinatorConfig.nntpShares.coordinatorPriority, coordinatorConfig.nntpMinConnections, coordinatorConfig.nntpConnections)
+    const share = new NntpShare(
+      count > 1 ? `coordinator ${index}/${count}` : 'coordinator',
+      coordinatorConfig.nntpShares.coordinatorPriority,
+      coordinatorConfig.nntpMinConnections,
+      coordinatorConfig.nntpConnections
+    )
     const main = index === 1
     let stopping = false
     let stopped: () => void
@@ -56,7 +53,11 @@ export default class CoordinatorRun extends BaseCommand {
       prefetcher.stop()
       await done
     })
-    this.logger.info(count > 1 ? `Coordinator started (shard ${index}/${count}${main ? '' : ', prefetching only'}).` : 'Coordinator started.')
+    this.logger.info(
+      count > 1
+        ? `Coordinator started (shard ${index}/${count}${main ? '' : ', prefetching only'}).`
+        : 'Coordinator started.'
+    )
     // The pool's size: its share of the account's connections, the prober the rest (app/services/nntp_shares.ts).
     // Before the first connection: until then it's NNTP_CONNECTIONS.
     const first = await NntpPool.share(share)
@@ -69,11 +70,16 @@ export default class CoordinatorRun extends BaseCommand {
         } catch (error: any) {
           this.logger.error(`NNTP share failed: ${error.message}`)
         }
-        for (let waited = 0; waited < coordinatorConfig.nntpShares.everyMs && !stopping; waited += TICK_MS) await sleep(TICK_MS)
+        for (
+          let waited = 0;
+          waited < coordinatorConfig.nntpShares.everyMs && !stopping;
+          waited += TICK_MS
+        )
+          await sleep(TICK_MS)
       }
     })()
-    // Two loops: the prefetcher's ticks don't wait for merging or planning (a merge round
-    // of up to MERGE_BUDGET_MS left the connections idle once their targets were fetched).
+    // The prefetcher's ticks in a loop of their own: planning doesn't hold them (a merge round of up to 30 s once
+    // left the connections idle once their targets were fetched).
     const prefetching = (async () => {
       while (!stopping) {
         try {
@@ -84,24 +90,24 @@ export default class CoordinatorRun extends BaseCommand {
         await sleep(TICK_MS)
       }
     })()
-    // range_blocks in a loop of its own: a settling or merging round doesn't hold release_final back.
-    const blocks = (async () => {
+    const abandoned = (async () => {
       while (main && !stopping) {
         try {
-          const abandoned = await Leases.finishAbandoned()
-          if (abandoned > 0) this.logger.info(`Finished ${abandoned} ranges read to their end whose worker went quiet.`)
-          await RangeBlocks.rebuild()
+          const finished = await Leases.finishAbandoned()
+          if (finished > 0)
+            this.logger.info(
+              `Finished ${finished} ranges read to their end whose worker went quiet.`
+            )
         } catch (error: any) {
-          this.logger.error(`Range blocks failed: ${error.message}`)
+          this.logger.error(`Finishing abandoned leases failed: ${error.message}`)
         }
-        for (let waited = 0; waited < BLOCKS_EVERY_MS && !stopping; waited += TICK_MS) await sleep(TICK_MS)
+        for (let waited = 0; waited < ABANDONED_EVERY_MS && !stopping; waited += TICK_MS)
+          await sleep(TICK_MS)
       }
     })()
     // History for new groups hourly; the tail every few minutes.
     let plannedAt = 0
     let tailedAt = 0
-    let mergedAt = 0
-    let settledAt = 0
     while (!stopping) {
       try {
         if (main && Date.now() - plannedAt >= PLAN_EVERY_MS) {
@@ -109,45 +115,22 @@ export default class CoordinatorRun extends BaseCommand {
           const { groups, ranges } = await Planner.planAll()
           if (ranges > 0) this.logger.info(`Planned ${ranges} ranges over ${groups} groups.`)
         }
-        if (main && Date.now() - settledAt >= SETTLE_EVERY_MS) {
-          settledAt = Date.now()
-          const settled = await Settler.run(SETTLE_BATCHES)
-          if (settled > 0) this.logger.info(`Settled ${settled} releases.`)
-        }
         if (main && Date.now() - tailedAt >= coordinatorConfig.tail.everySeconds * 1000) {
           tailedAt = Date.now()
           const { groups, ranges } = await Planner.planTail()
           if (ranges > 0) this.logger.info(`Tail: ${ranges} ranges for ${groups} groups.`)
         }
-        if (main && Date.now() - mergedAt >= MERGE_EVERY_MS) {
-          mergedAt = Date.now()
-          // Rounds until a whole pass over the file keys decides nothing more, or MERGE_BUDGET_MS.
-          // (Going on while only waiting files were left kept Postgres busy all the time.)
-          const totals = { completed: 0, incomplete: 0, waiting: 0 }
-          let decidedThisPass = 0
-          while (!stopping && Date.now() - mergedAt < MERGE_BUDGET_MS) {
-            const merged = await FragmentMerger.run()
-            totals.completed += merged.completed
-            totals.incomplete += merged.incomplete
-            totals.waiting = merged.waiting
-            decidedThisPass += merged.completed + merged.incomplete
-            if (merged.wrapped) {
-              if (decidedThisPass === 0) break
-              decidedThisPass = 0
-            }
-          }
-          if (totals.completed || totals.incomplete) this.logger.info(`Fragments: ${totals.completed} files completed, ${totals.incomplete} incomplete, ${totals.waiting} waiting.`)
-        }
       } catch (error: any) {
         this.logger.error(`Coordinator round failed: ${error.message}`)
       }
       await sleep(TICK_MS)
     }
-    await Promise.all([prefetching, blocks, sharing])
+    await Promise.all([prefetching, abandoned, sharing])
     // The chunks in flight, for at most STOP_WAIT_MS (a connection the server holds back can take minutes); then closed.
     const waitUntil = Date.now() + STOP_WAIT_MS
     while (prefetcher.active > 0 && Date.now() < waitUntil) await sleep(200)
-    if (prefetcher.active > 0) this.logger.info(`Closing ${prefetcher.active} connections still fetching.`)
+    if (prefetcher.active > 0)
+      this.logger.info(`Closing ${prefetcher.active} connections still fetching.`)
     await NntpPool.close()
     await share.leave().catch(() => {})
     this.logger.info('Coordinator stopped.')

commands/edge_bench.ts

@@ -108,7 +108,7 @@ export default class EdgeBench extends BaseCommand {
     let text = 0
     let peakRss = 0
     let peakDetail = ''
-    const out = { headers: 0, releases: 0, files: 0, segments: 0, payload: 0, candidates: 0, fragments: 0, samples: 0 }
+    const out = { headers: 0, releases: 0, files: 0, segments: 0, payload: 0, candidates: 0, samples: 0 }
     const counts = new Map<string, number>()
     const keys = new Set<string>()
     let batches = 0
@@ -164,7 +164,6 @@ export default class EdgeBench extends BaseCommand {
         out.payload += r.payload.length
       }
       out.candidates += batch.candidates.length
-      out.fragments += batch.fragments.length
       out.samples += batch.samples.length
       for (const c of batch.counts) counts.set(`${c.rule} ${c.key}`.trim(), (counts.get(`${c.rule} ${c.key}`.trim()) ?? 0) + c.articles)
       const m = process.memoryUsage()
@@ -182,7 +181,7 @@ export default class EdgeBench extends BaseCommand {
     this.logger.info(`${out.headers.toLocaleString('en')} headers in ${(total / 1000).toFixed(1)} s (${Math.round(out.headers / (total / 1000)).toLocaleString('en')}/s on one core; CPU ${cpuS.toFixed(1)} s, wall ${((Date.now() - wall) / 1000).toFixed(1)} s)`)
     this.logger.info(`  inflate ${(t.inflate / 1000).toFixed(1)} s, parse ${(t.parse / 1000).toFixed(1)} s, index ${(t.index / 1000).toFixed(1)} s; peak RSS ${(peakRss / 1e6).toFixed(0)} MB (${peakDetail})`)
     this.logger.info(`  ${batches} batches of ${this.batchChunks} chunks: ${out.releases} release chunks for ${keys.size} releases (${(out.releases / Math.max(1, keys.size)).toFixed(2)} chunks per release)`)
-    this.logger.info(`  kept: ${out.releases} release chunks, ${out.files} files, ${out.segments} segments, payload ${(out.payload / 1e6).toFixed(1)} MB (${(out.payload / Math.max(1, out.segments)).toFixed(1)} B/segment); ${out.candidates} candidates, ${out.fragments} fragments, ${out.samples} samples`)
+    this.logger.info(`  kept: ${out.releases} release chunks, ${out.files} files, ${out.segments} segments, payload ${(out.payload / 1e6).toFixed(1)} MB (${(out.payload / Math.max(1, out.segments)).toFixed(1)} B/segment); ${out.candidates} candidates, ${out.samples} samples`)
     for (const [k, v] of [...counts].sort((a, b) => b[1] - a[1]).slice(0, 8)) this.logger.info(`  dropped ${k}: ${v.toLocaleString('en')} articles`)
     if (this.store) this.logger.info(`  stored ${batches} batches in ${(storeMs / 1000).toFixed(1)} s (${(storeMs / Math.max(1, out.releases)).toFixed(2)} ms per release chunk)`)
   }

commands/releases_regroup.ts

@@ -3,6 +3,7 @@ import type { CommandOptions } from '@adonisjs/core/types/ace'
 import db from '@adonisjs/lucid/services/db'
 import { decodeChunk } from '#lib/chunks/index'
 import { releaseIdentity } from '#lib/release/index'
+import { sourceCode } from '#lib/release/source'
 
 /**
  * What lib/release's identity makes of releases already in the catalog: a
@@ -27,7 +28,7 @@ export default class ReleasesRegroup extends BaseCommand {
   async run() {
     for (const name of this.group ?? []) {
       const { rows } = await db.rawQuery(
-        `SELECT DISTINCT c.release_id FROM release_chunks c JOIN ranges r ON r.id = c.range_id JOIN groups g ON g.id = r.group_id WHERE g.name = ?`,
+        `SELECT DISTINCT c.release_id FROM release_chunks c JOIN blocks b ON b.id = c.block_id JOIN groups g ON g.id = b.group_id WHERE g.name = ?`,
         [name]
       )
       const ids = rows.map((r: { release_id: string }) => r.release_id)
@@ -54,7 +55,7 @@ export default class ReleasesRegroup extends BaseCommand {
     }
     if (this.show?.length) return
     for (const source of ['subject', 'post', 'stem']) {
-      const rows = await db.from('releases').where('source', source).where('files', '>', 1).orderByRaw('random()').limit(this.sample).select('id', 'name', 'files')
+      const rows = await db.from('releases').where('source', sourceCode(source)).where('files', '>', 1).orderByRaw('random()').limit(this.sample).select('id', 'name', 'files')
       let split = 0
       let pieces = 0
       const examples: string[] = []

commands/replay_corpus.ts

@@ -0,0 +1,396 @@
+import { BaseCommand, flags } from '@adonisjs/core/ace'
+import type { CommandOptions } from '@adonisjs/core/types/ace'
+import { spawn } from 'node:child_process'
+import { access, readdir, readFile, writeFile } from 'node:fs/promises'
+import { tmpdir } from 'node:os'
+import { join, resolve } from 'node:path'
+import db from '@adonisjs/lucid/services/db'
+import coordinatorConfig from '#config/coordinator'
+import BatchStore from '#services/batch_store'
+import Leases from '#services/leases'
+import RuleSets from '#services/rule_sets'
+import { fromWire, type WireBatch } from '#lib/edge/index'
+
+/**
+ * Only a database named like one of these is written: the replay leases and finishes ranges and fills the catalog.
+ * Not the test database: the rows left behind failed 27 functional tests (LOG.md 2026-10-26 (111)). With
+ * REPLAY_INTO_CATALOG=1 any database but the test one: voyager 3 deployed, history replayed from the corpus into
+ * the catalog itself (deploy/replay-window.sh, LOG.md (116)).
+ */
+const SCRATCH_DB = /_(replay|scratch)$/
+const TEST_DB = /_test$/
+const NODE_NAME = 'replay'
+
+interface StoredRange {
+  id: number
+  start: number
+  end: number
+}
+
+/**
+ * The replay harness (REBUILD.md, "Voyager 3", 8): a group's stored corpus ranges in article order through one
+ * window in the Rust edge (`edge-bench --group`, the window carried from range to range) and the batch store into a
+ * scratch catalog, as the worker would store them: each range leased to a node named "replay", its batches stored
+ * at its resume point, the range finished when its last batch is in. Then a report: releases finished, complete,
+ * partial and provisional, and the catalog's bytes per header (the NVMe tables apart from those on CHUNK_TABLESPACE).
+ *
+ *   node ace replay:corpus --group alt.binaries.ath [--dir CORPUS_DIR] [--from A] [--to B] [--limit N] [--batch-chunks 5]
+ *
+ * Refuses a database not named *_replay or *_scratch. The rules are the catalog's current rule set
+ * (rule_sets, the groups' priorities): copy `groups` and `rule_sets` from the catalog for a faithful replay.
+ */
+export default class ReplayCorpus extends BaseCommand {
+  static commandName = 'replay:corpus'
+  static description =
+    "Replays a group's corpus ranges through the Rust edge and the batch store into a scratch catalog"
+  static options: CommandOptions = { startApp: true }
+
+  @flags.string({ description: 'The group (its directory under the corpus)', required: true })
+  declare group: string
+
+  @flags.string({ description: 'The corpus directory (default: CORPUS_DIR)' })
+  declare dir?: string
+
+  @flags.number({ description: 'Replay this many ranges at most (default: all stored)' })
+  declare limit?: number
+
+  @flags.number({ description: 'Skip ranges ending before this article (continue a replay)' })
+  declare from?: number
+
+  @flags.number({
+    description: "Skip ranges starting after this article (one window of the group's history)",
+  })
+  declare to?: number
+
+  @flags.number({
+    description: 'Chunks of 100k articles per batch, as the worker posts them',
+    default: 5,
+  })
+  declare batchChunks: number
+
+  @flags.string({
+    description:
+      'The edge-bench binary (default: EDGE_BENCH, edge/target/release/edge-bench, /usr/local/bin/edge-bench)',
+  })
+  declare bench?: string
+
+  @flags.boolean({
+    description:
+      'Leave the window open at the end (the last ranges provisional), to continue later',
+  })
+  declare noFinish: boolean
+
+  async run() {
+    const database = db.connection().getReadClient().client.config.connection.database as string
+    const intoCatalog = process.env.REPLAY_INTO_CATALOG === '1' && !TEST_DB.test(database)
+    if (!SCRATCH_DB.test(database) && !intoCatalog) {
+      this.logger.error(
+        `The replay writes the catalog: it only runs against a database named *_replay or *_scratch (this is "${database}"; REPLAY_INTO_CATALOG=1 allows it)`
+      )
+      this.exitCode = 1
+      return
+    }
+    const dir = this.dir ?? coordinatorConfig.corpus.dir
+    if (!dir) {
+      this.logger.error('No corpus directory: --dir or CORPUS_DIR')
+      this.exitCode = 1
+      return
+    }
+    const bench = await this.benchBinary()
+    if (!bench) {
+      this.logger.error(
+        'No edge-bench binary found: build it (cargo build --release --manifest-path edge/Cargo.toml) or set EDGE_BENCH'
+      )
+      this.exitCode = 1
+      return
+    }
+    const groupDir = join(dir, this.group)
+    const stored = await this.storedRanges(groupDir)
+    if (stored.length === 0) {
+      this.logger.error(`${groupDir}: no complete range`)
+      this.exitCode = 1
+      return
+    }
+    const nodeId = await this.nodeId()
+    const groupId = await this.groupId()
+    const ranges = await this.leaseRanges(groupId, nodeId, stored)
+    const rules = await RuleSets.current()
+    const rulesPath = join(tmpdir(), `replay-rules-${process.pid}.json`)
+    await writeFile(rulesPath, JSON.stringify(rules))
+    this.logger.info(
+      `${this.group}: ${ranges.length} ranges (${ranges[0].start}–${ranges[ranges.length - 1].end}) through ${bench}, rules v${rules.version}, into ${database}.`
+    )
+
+    // edge-bench walks the group directory in article order itself; --from, --to and --limit cut its list as ours.
+    const args = [
+      '--group',
+      groupDir,
+      '--rules',
+      rulesPath,
+      '--batch-chunks',
+      String(this.batchChunks),
+      '--wire',
+      '-',
+    ]
+    if (this.from) args.push('--from', String(this.from))
+    if (this.to) args.push('--to', String(this.to))
+    if (this.limit) args.push('--limit', String(this.limit))
+    if (this.noFinish) args.push('--no-finish')
+    const started = Date.now()
+    const child = spawn(bench, args, { stdio: ['ignore', 'pipe', 'inherit'] })
+    const totals = {
+      batches: 0,
+      headers: 0,
+      releases: 0,
+      newReleases: 0,
+      finalized: 0,
+      candidates: 0,
+      storeMs: 0,
+    }
+    let current: StoredRange | null = null
+    const storeLine = async (line: string) => {
+      const batch = fromWire(JSON.parse(line) as WireBatch)
+      if (!current || batch.from < current.start || batch.from > current.end) {
+        current = ranges.find((r) => batch.from >= r.start && batch.from <= r.end) ?? null
+        if (!current) throw new Error(`Batch at ${batch.from} is in no leased range`)
+      }
+      const at = Date.now()
+      const result = await BatchStore.store(current.id, nodeId, batch)
+      totals.storeMs += Date.now() - at
+      totals.batches++
+      totals.headers += batch.headers
+      totals.releases += result.releases
+      totals.newReleases += result.newReleases
+      totals.finalized += result.finalized
+      totals.candidates += result.candidates
+      if (batch.next > current.end) {
+        await Leases.finish(current.id, nodeId)
+        const seconds = (Date.now() - started) / 1000
+        this.logger.info(
+          `  range ${current.start}–${current.end} done: ${totals.headers.toLocaleString('en')} headers so far in ${seconds.toFixed(0)} s ` +
+            `(${Math.round(totals.headers / seconds).toLocaleString('en')}/s), ${totals.releases} releases stored (${totals.finalized} finished later), store ${(totals.storeMs / 1000).toFixed(0)} s`
+        )
+      }
+    }
+    // Lines pulled as the store keeps up: the child's stdout only moves when a chunk is read, so batches don't pile up.
+    let rest = ''
+    let failure: Error | null = null
+    try {
+      for await (const chunk of child.stdout as AsyncIterable<Buffer>) {
+        rest += chunk.toString('utf8')
+        let nl: number
+        while ((nl = rest.indexOf('\n')) >= 0) {
+          const line = rest.slice(0, nl)
+          rest = rest.slice(nl + 1)
+          if (line.length > 0) await storeLine(line)
+        }
+      }
+      if (rest.trim().length > 0) await storeLine(rest)
+    } catch (error: any) {
+      failure = error
+      child.kill('SIGTERM')
+    }
+    const code = await new Promise<number | null>((done) =>
+      child.exitCode !== null ? done(child.exitCode) : child.on('exit', done)
+    )
+    if (failure) {
+      this.logger.error(`Replay failed: ${failure.message}`)
+      this.exitCode = 1
+      return
+    }
+    if (code !== 0) {
+      this.logger.error(`edge-bench exited with ${code}`)
+      this.exitCode = 1
+      return
+    }
+    await this.report(groupId, ranges, Date.now() - started)
+  }
+
+  private async benchBinary(): Promise<string | null> {
+    const candidates = [
+      this.bench,
+      process.env.EDGE_BENCH,
+      resolve('edge/target/release/edge-bench'),
+      '/usr/local/bin/edge-bench',
+    ].filter((p): p is string => !!p)
+    for (const path of candidates) {
+      if (
+        await access(path).then(
+          () => true,
+          () => false
+        )
+      )
+        return path
+    }
+    return null
+  }
+
+  /** The group directory's complete ranges (a range.json), in article order, cut by --from, --to and --limit. */
+  private async storedRanges(groupDir: string): Promise<{ start: number; end: number }[]> {
+    const names = await readdir(groupDir).catch(() => [] as string[])
+    const ranges: { start: number; end: number }[] = []
+    for (const name of names) {
+      const m = /^(\d+)-(\d+)$/.exec(name)
+      if (!m) continue
+      const json = await readFile(join(groupDir, name, 'range.json'), 'utf8').catch(() => null)
+      if (!json) continue
+      ranges.push({ start: Number(m[1]), end: Number(m[2]) })
+    }
+    ranges.sort((a, b) => a.start - b.start)
+    let chosen = this.from ? ranges.filter((r) => r.end >= this.from!) : ranges
+    if (this.to) chosen = chosen.filter((r) => r.start <= this.to!)
+    if (this.limit) chosen = chosen.slice(0, this.limit)
+    return chosen
+  }
+
+  private async nodeId(): Promise<number> {
+    const row = await db.from('nodes').where('name', NODE_NAME).first()
+    if (row) return Number(row.id)
+    const [inserted] = await db.table('nodes').insert({ name: NODE_NAME }).returning('id')
+    return Number(inserted.id)
+  }
+
+  /** The group's row (inserted after the last priority when the scratch catalog doesn't have it). */
+  private async groupId(): Promise<number> {
+    const row = await db.from('groups').where('name', this.group).first()
+    if (row) return Number(row.id)
+    const { rows } = await db.rawQuery(`SELECT coalesce(max(priority), 0) + 1 AS next FROM groups`)
+    const [inserted] = await db
+      .table('groups')
+      .insert({ name: this.group, priority: Number(rows[0].next), active: true })
+      .returning('id')
+    this.logger.info(`Group ${this.group} added with priority ${rows[0].next}.`)
+    return Number(inserted.id)
+  }
+
+  /** The ranges as leased to the replay node, from their start (an earlier replay's rows are taken over). */
+  private async leaseRanges(
+    groupId: number,
+    nodeId: number,
+    stored: { start: number; end: number }[]
+  ): Promise<StoredRange[]> {
+    const out: StoredRange[] = []
+    const now = new Date()
+    for (const r of stored) {
+      const lease = {
+        status: 'leased',
+        node_id: nodeId,
+        next_id: r.start,
+        headers: 0,
+        posted_low: null,
+        posted_high: null,
+        leased_at: now,
+        heartbeat_at: now,
+        done_at: null,
+        error: null,
+        capture: true,
+      }
+      const existing = await db
+        .from('ranges')
+        .where('group_id', groupId)
+        .where('start_id', r.start)
+        .first()
+      if (existing) {
+        await db
+          .from('ranges')
+          .where('id', existing.id)
+          .update({ ...lease, end_id: r.end, attempts: Number(existing.attempts) + 1 })
+        out.push({ id: Number(existing.id), start: r.start, end: r.end })
+        continue
+      }
+      const [inserted] = await db
+        .table('ranges')
+        .insert({
+          ...lease,
+          group_id: groupId,
+          start_id: r.start,
+          end_id: r.end,
+          priority: 0,
+          attempts: 1,
+        })
+        .returning('id')
+      out.push({ id: Number(inserted.id), start: r.start, end: r.end })
+    }
+    return out
+  }
+
+  /** What the replay left in the catalog (this group's releases; the sizes are the whole catalog's), per article. */
+  private async report(groupId: number, ranges: StoredRange[], elapsedMs: number) {
+    const ids = ranges.map((r) => r.id)
+    const { rows: counts } = await db.rawQuery(
+      `SELECT count(*)::bigint AS releases,
+              count(settled_at)::bigint AS finished,
+              count(*) FILTER (WHERE settled_at IS NULL)::bigint AS provisional,
+              count(*) FILTER (WHERE edge)::bigint AS edges,
+              count(*) FILTER (WHERE complete)::bigint AS complete,
+              count(*) FILTER (WHERE complete = false)::bigint AS incomplete,
+              count(*) FILTER (WHERE complete IS NULL)::bigint AS undecided,
+              count(*) FILTER (WHERE readable)::bigint AS readable,
+              count(*) FILTER (WHERE has_par2)::bigint AS with_par2,
+              coalesce(sum(files), 0)::bigint AS files,
+              coalesce(sum(segments), 0)::bigint AS segments,
+              coalesce(sum(bytes), 0)::bigint AS bytes
+       FROM releases WHERE group_id = ?`,
+      [groupId]
+    )
+    const { rows: sources } = await db.rawQuery(
+      `SELECT source, count(*)::bigint AS n FROM releases WHERE group_id = ? GROUP BY source ORDER BY source`,
+      [groupId]
+    )
+    const { rows: pending } = await db.rawQuery(
+      `SELECT count(*)::bigint AS n, coalesce(sum(segments), 0)::bigint AS segments, count(*) FILTER (WHERE jsonb_array_length(candidates) = 0)::bigint AS unprobable
+       FROM pending_releases WHERE group_id = ?`,
+      [groupId]
+    )
+    const { rows: headerRows } = await db.rawQuery(
+      `SELECT coalesce(sum(headers), 0)::bigint AS headers FROM ranges WHERE id = ANY(?::bigint[])`,
+      [ids]
+    )
+    const headers = Number(headerRows[0].headers)
+    const { rows: sizes } = await db.rawQuery(
+      `SELECT CASE WHEN c.relname LIKE 'release_chunks%' THEN 'release_chunks' ELSE c.relname END AS name,
+              sum(pg_total_relation_size(c.oid))::bigint AS bytes
+       FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace
+       WHERE n.nspname = 'public' AND c.relkind = 'r'
+         AND (c.relname IN ('releases', 'release_texts', 'posters', 'group_sets', 'probe_candidates', 'rule_counts', 'rule_samples', 'blocks', 'pending_releases')
+              OR c.relname LIKE 'release_chunks%')
+       GROUP BY 1 ORDER BY 2 DESC`
+    )
+    // On CHUNK_TABLESPACE (the HDD) in the catalog: the chunks (migration 3200), inspections and candidates (6000).
+    const hdd = new Set([
+      'release_chunks',
+      'probe_candidates',
+      'release_inspections',
+      'pending_releases',
+    ])
+    let nvme = 0
+    let other = 0
+    const c = counts[0]
+    const fmt = (n: unknown) => Number(n).toLocaleString('en')
+    this.logger.info(
+      `Replay of ${ranges.length} ranges: ${fmt(headers)} headers in ${(elapsedMs / 1000).toFixed(0)} s (${fmt(Math.round(headers / Math.max(1, elapsedMs / 1000)))}/s).`
+    )
+    this.logger.info(
+      `Releases ${fmt(c.releases)}: finished ${fmt(c.finished)}, provisional ${fmt(c.provisional)}, edges ${fmt(c.edges)}; ` +
+        `complete ${fmt(c.complete)}, incomplete ${fmt(c.incomplete)}, undecided ${fmt(c.undecided)}; readable ${fmt(c.readable)}, with par2 ${fmt(c.with_par2)}; ` +
+        `${fmt(c.files)} files, ${fmt(c.segments)} segments (${((Number(c.segments) / Math.max(1, headers)) * 100).toFixed(1)}% of the headers), ${(Number(c.bytes) / 1e12).toFixed(2)} TB.`
+    )
+    this.logger.info(
+      `By source: ${sources.map((s: any) => `${s.source}: ${fmt(s.n)}`).join(', ')}.`
+    )
+    this.logger.info(
+      `Pending (unnamed, outside the catalog): ${fmt(pending[0].n)} releases, ${fmt(pending[0].segments)} segments, ${fmt(pending[0].unprobable)} without a probe target.`
+    )
+    for (const s of sizes) {
+      const bytes = Number(s.bytes)
+      if (hdd.has(s.name)) other += bytes
+      else nvme += bytes
+      this.logger.info(
+        `  ${s.name}: ${(bytes / 1e6).toFixed(1)} MB, ${(bytes / Math.max(1, headers)).toFixed(2)} B/header`
+      )
+    }
+    this.logger.info(
+      `NVMe tables: ${(nvme / 1e6).toFixed(1)} MB, ${(nvme / Math.max(1, headers)).toFixed(2)} B/header; HDD tables (CHUNK_TABLESPACE): ${(other / 1e6).toFixed(1)} MB, ${(other / Math.max(1, headers)).toFixed(2)} B/header (the whole catalog's sizes over this group's headers: only the first group's are its own).`
+    )
+  }
+}

config/coordinator.ts

@@ -103,6 +103,14 @@ export default {
     batch: 400,
     /** BODY commands in flight per connection (pipelined; XoverClient.bodies). */
     pipeline: env.get('PROBE_PIPELINE', 8),
+    /**
+     * The pending ladder (REBUILD.md decision f; app/services/pending_ladder.ts): the share of each claim taken
+     * from pending_releases at least (the catalog's own candidates come first, the rest of the claim is theirs).
+     */
+    pendingShare: env.get('PENDING_SHARE', 0.25),
+    /** A signature (tool, group, poster) with this many climbs and at most pendingMaxNamed of them named is dropped. */
+    pendingMinProbes: env.get('PENDING_MIN_PROBES', 100),
+    pendingMaxNamed: env.get('PENDING_MAX_NAMED', 0.005),
   },
   /**
    * The catalog's tablespace may grow to this many GB (app/services/catalog_space.ts); above it only the tail and

database/migrations/1770000006000_voyager3_catalog.ts

@@ -0,0 +1,177 @@
+import { BaseSchema } from '@adonisjs/lucid/schema'
+import env from '#start/env'
+
+/**
+ * Voyager 3's catalog (REBUILD.md, "Voyager 3: window indexing", 4 and 7), built from a reset, not converted in
+ * place. This migration IS the reset: it empties every table the indexing recomputes (releases, chunks,
+ * inspections, candidates, par2 data, file hashes, verdicts, merges, rule counts and samples; what survives:
+ * groups, ranges, the corpus, probe_cache, spots, nntp_shares, nodes, users), so it refuses to run on a catalog
+ * with releases unless VOYAGER3_RESET=1 (the user's go-ahead; resets are paused since 2026-10-12). Then:
+ * - posters get an id; group_sets holds each distinct group combination once (~140k); releases is created anew
+ *   without its text copies (poster_id, group_set_id, the keys as 20-byte bytea, source a smallint), 8-byte
+ *   columns first, with `edge` (still open at its block's end, for the block merger) and settled_at as the finish
+ *   (null: a tail release still provisional). The category and judge triggers follow the new columns;
+ *   releases_category_newest_idx (5.3 GB, used once) goes, and the trigram indexes: search matches release_texts
+ *   (migration 6200), which holds the name and every searchable string learned later.
+ * - fragments and range_blocks go with the fragment merger and the settler (the window finishes a release once);
+ *   release_final() goes: a release is final once finished.
+ * - release_chunks, rule_counts and rule_samples are keyed by the block that wrote them (block_id; migration 6100).
+ * - release_inspections and probe_candidates move to CHUNK_TABLESPACE (the HDD) when set: written once, read per
+ *   release, not by search.
+ */
+export default class extends BaseSchema {
+  async up() {
+    this.defer(async (db) => {
+      const { rows } = await db.rawQuery(`SELECT count(*)::int AS n FROM releases`)
+      if (rows[0].n > 0 && process.env.VOYAGER3_RESET !== '1') {
+        throw new Error(
+          `The voyager 3 catalog migration empties the catalog (${rows[0].n} releases): run it with VOYAGER3_RESET=1 after the go-ahead`
+        )
+      }
+    })
+    this.schema.raw(`
+      TRUNCATE releases, release_chunks, release_inspections, probe_candidates, file_hashes, par2_files, par2_sets,
+        release_verdicts, release_merges, release_spots, release_passwords, rule_counts, rule_samples, judge_dirty,
+        judge_full, fold_candidates, posters, worker_cursors RESTART IDENTITY`)
+    // Spots matched to releases that are gone: the spot inspector matches them again.
+    this.schema.raw(`UPDATE spots SET release_id = NULL WHERE release_id IS NOT NULL`)
+    this.schema.raw(`UPDATE spot_files SET release_id = NULL WHERE release_id IS NOT NULL`)
+
+    this.schema.raw(`DROP TABLE IF EXISTS fragments`)
+    this.schema.raw(`DROP TABLE IF EXISTS range_blocks`)
+    this.schema.raw(
+      `DROP FUNCTION IF EXISTS release_final(integer, bigint, bigint, timestamptz, timestamptz, boolean)`
+    )
+    this.schema.raw(
+      `DROP FUNCTION IF EXISTS release_final(integer, bigint, bigint, timestamptz, timestamptz)`
+    )
+
+    // Posters by id (a release's poster_id); searched by trigram as before (poster_search).
+    this.schema.raw(`DROP TABLE IF EXISTS posters`)
+    this.schema.raw(`
+      CREATE TABLE posters (
+        id serial PRIMARY KEY,
+        poster text NOT NULL UNIQUE
+      )`)
+    this.schema.raw(`CREATE INDEX posters_trgm ON posters USING gin (poster gin_trgm_ops)`)
+    // Each distinct combination of groups a release was posted to, once: a sorted jsonb array of names.
+    this.schema.raw(`
+      CREATE TABLE group_sets (
+        id serial PRIMARY KEY,
+        groups jsonb NOT NULL UNIQUE
+      )`)
+
+    this.schema.raw(`DROP TABLE IF EXISTS releases CASCADE`)
+    this.schema.raw(`
+      CREATE TABLE releases (
+        id bigserial PRIMARY KEY,
+        -- The upload's posting window (first and last article's Date).
+        posted_at timestamptz,
+        last_posted_at timestamptz,
+        first_article bigint,
+        last_article bigint,
+        segments bigint NOT NULL DEFAULT 0,
+        bytes bigint NOT NULL DEFAULT 0,
+        -- The finish: the window closed on it (or the block ended); null while a tail release is provisional.
+        settled_at timestamptz,
+        inspected_at timestamptz,
+        first_seen_at timestamptz NOT NULL DEFAULT now(),
+        -- The owner group (null: a spot's release), the set of groups it was posted to, its poster.
+        group_id integer,
+        group_set_id integer NOT NULL,
+        poster_id integer,
+        file_total integer,
+        files integer NOT NULL DEFAULT 0,
+        chunks integer NOT NULL DEFAULT 0,
+        category integer,
+        -- lib/release/source.ts: 1 subject, 2 post, 3 stem, 4 spot, 5 nzb, 6 album.
+        source smallint NOT NULL,
+        readable boolean NOT NULL DEFAULT false,
+        has_par2 boolean NOT NULL DEFAULT false,
+        complete boolean,
+        passworded boolean,
+        sidecar_only boolean,
+        -- Still open when its block ended: the block merger joins it with the neighbour block's part by identity.
+        edge boolean NOT NULL DEFAULT false,
+        -- SHA-1s (lib/release/identity.ts): the identity (poster, name, group), and the release (identity and upload).
+        release_key bytea NOT NULL,
+        identity_key bytea NOT NULL,
+        name text NOT NULL,
+        inner_name varchar(255)
+      )`)
+    this.schema.raw(`CREATE UNIQUE INDEX releases_release_key_unique ON releases (release_key)`)
+    this.schema.raw(`CREATE INDEX releases_identity_idx ON releases (identity_key, group_id)`)
+    // No trigram indexes here: search matches release_texts (migration 6200), where the name is stored once more.
+    this.schema.raw(
+      `CREATE INDEX releases_newest_idx ON releases (posted_at DESC NULLS LAST, id DESC)`
+    )
+    this.schema.raw(
+      `CREATE INDEX releases_poster_newest_idx ON releases (poster_id, posted_at DESC NULLS LAST, id DESC)`
+    )
+    this.schema.raw(`CREATE INDEX releases_passworded_idx ON releases (id) WHERE passworded`)
+    this.schema.raw(
+      `CREATE INDEX releases_edge_idx ON releases (group_id, identity_key) WHERE edge`
+    )
+    this.schema.raw(
+      `CREATE INDEX releases_provisional_idx ON releases (id) WHERE settled_at IS NULL`
+    )
+
+    // The category from the name, the inner name, or the groups (migration 5200), the groups now in group_sets.
+    this.schema.raw(`
+      CREATE OR REPLACE FUNCTION releases_set_category() RETURNS trigger LANGUAGE plpgsql AS $$
+      BEGIN
+        NEW.category := release_category_for(NEW.name, NEW.inner_name, (SELECT groups FROM group_sets WHERE id = NEW.group_set_id),
+                                             CASE WHEN TG_OP = 'UPDATE' THEN OLD.category ELSE NEW.category END);
+        RETURN NEW;
+      END $$`)
+    this.schema.raw(`
+      CREATE TRIGGER releases_category BEFORE INSERT OR UPDATE OF name, inner_name, group_set_id ON releases
+      FOR EACH ROW EXECUTE FUNCTION releases_set_category()`)
+    // The judge's dirty marks (migration 4900), on the columns its rules read.
+    this.schema.raw(`
+      CREATE TRIGGER judge_dirty_insert AFTER INSERT ON releases REFERENCING NEW TABLE AS new_rows
+      FOR EACH STATEMENT EXECUTE FUNCTION judge_dirty_releases()`)
+    this.schema.raw(`
+      CREATE TRIGGER judge_dirty_delete AFTER DELETE ON releases REFERENCING OLD TABLE AS old_rows
+      FOR EACH STATEMENT EXECUTE FUNCTION judge_dirty_releases()`)
+    this.schema.raw(`
+      CREATE TRIGGER judge_dirty_update AFTER UPDATE ON releases FOR EACH ROW
+      WHEN (old.name IS DISTINCT FROM new.name OR old.poster_id IS DISTINCT FROM new.poster_id OR old.source IS DISTINCT FROM new.source
+        OR old.file_total IS DISTINCT FROM new.file_total OR old.passworded IS DISTINCT FROM new.passworded
+        OR old.settled_at IS DISTINCT FROM new.settled_at OR old.readable IS DISTINCT FROM new.readable
+        OR old.inner_name IS DISTINCT FROM new.inner_name OR old.group_id IS DISTINCT FROM new.group_id
+        OR old.first_article IS DISTINCT FROM new.first_article OR old.last_article IS DISTINCT FROM new.last_article
+        OR old.posted_at IS DISTINCT FROM new.posted_at OR old.last_posted_at IS DISTINCT FROM new.last_posted_at
+        OR old.complete IS DISTINCT FROM new.complete)
+      EXECUTE FUNCTION judge_dirty_releases()`)
+
+    // Chunks, rule counts and samples by the block that wrote them.
+    this.schema.raw(`ALTER TABLE release_chunks RENAME COLUMN range_id TO block_id`)
+    this.schema.raw(
+      `ALTER INDEX IF EXISTS release_chunks_range_idx RENAME TO release_chunks_block_idx`
+    )
+    this.schema.raw(`ALTER TABLE rule_counts RENAME COLUMN range_id TO block_id`)
+    this.schema.raw(`ALTER TABLE rule_samples RENAME COLUMN range_id TO block_id`)
+    this.schema.raw(
+      `ALTER INDEX IF EXISTS rule_samples_range_id_index RENAME TO rule_samples_block_id_index`
+    )
+
+    // Cold tables to the HDD (CHUNK_TABLESPACE), with their indexes; empty now, so at once.
+    const hdd = env.get('CHUNK_TABLESPACE')
+    if (hdd) {
+      for (const table of ['release_inspections', 'probe_candidates']) {
+        this.schema.raw(`ALTER TABLE ${table} SET TABLESPACE ${hdd}`)
+        this.schema.raw(`
+          DO $$ DECLARE i record; BEGIN
+            FOR i IN SELECT indexname FROM pg_indexes WHERE schemaname = 'public' AND tablename = '${table}' LOOP
+              EXECUTE format('ALTER INDEX %I SET TABLESPACE %I', i.indexname, '${hdd}');
+            END LOOP;
+          END $$`)
+      }
+    }
+  }
+
+  async down() {
+    throw new Error('The voyager 3 catalog migration cannot be undone (it emptied the catalog)')
+  }
+}

database/migrations/1770000006100_blocks.ts

@@ -0,0 +1,61 @@
+import { BaseSchema } from '@adonisjs/lucid/schema'
+
+/**
+ * Blocks (REBUILD.md, "Voyager 3: window indexing", 1): what a worker leases. A block is a run of consecutive
+ * ranges of one group covering at least 30 days of posting time (planned from ranges.est_date and posted_low/high),
+ * read in article order through the 24 h window; blocks of a group run in parallel. Ranges stay the unit the
+ * prefetcher fetches (the spool, the corpus) and point at their block; the lease, resume point, heartbeat and
+ * counts move here. Part of the reset (migration 6000): every range is read again, from the corpus where stored.
+ */
+export default class extends BaseSchema {
+  async up() {
+    this.schema.raw(`
+      CREATE TABLE blocks (
+        id bigserial PRIMARY KEY,
+        group_id integer NOT NULL REFERENCES groups (id) ON DELETE CASCADE,
+        -- The first range's start and the last range's end.
+        start_id bigint NOT NULL,
+        end_id bigint NOT NULL,
+        -- The resume point: the first article not stored yet (batches are refused out of order, as ranges' were).
+        next_id bigint NOT NULL,
+        -- The posting time the block covers: estimated when planned, the worker's measured span once done.
+        posted_low timestamptz,
+        posted_high timestamptz,
+        -- planned, leased, done, failed.
+        status varchar(8) NOT NULL DEFAULT 'planned',
+        -- As ranges': the tail before everything, then the history by era.
+        priority integer NOT NULL DEFAULT 0,
+        node_id integer,
+        attempts integer NOT NULL DEFAULT 0,
+        rule_version integer,
+        headers bigint NOT NULL DEFAULT 0,
+        releases integer NOT NULL DEFAULT 0,
+        leased_at timestamptz,
+        heartbeat_at timestamptz,
+        done_at timestamptz,
+        error text,
+        created_at timestamptz NOT NULL DEFAULT now(),
+        UNIQUE (group_id, start_id)
+      )`)
+    this.schema.raw(
+      `CREATE INDEX blocks_todo_idx ON blocks (priority, id) WHERE status IN ('planned', 'leased')`
+    )
+    this.schema.raw(`CREATE INDEX blocks_leased_idx ON blocks (leased_at) WHERE status = 'leased'`)
+    this.schema.raw(
+      `ALTER TABLE ranges ADD COLUMN IF NOT EXISTS block_id bigint REFERENCES blocks (id) ON DELETE SET NULL`
+    )
+    this.schema.raw(`CREATE INDEX IF NOT EXISTS ranges_block_idx ON ranges (block_id)`)
+    // The reset: every range planned again from its start (the corpus serves the stored ones). Priorities, capture,
+    // est_date, stored_at, plain_chunks and the measured posted_low/high stay.
+    this.schema.raw(`
+      UPDATE ranges SET status = 'planned', next_id = start_id, node_id = NULL, attempts = 0, spooled_chunks = 0,
+        spooled_at = NULL, leased_at = NULL, heartbeat_at = NULL, done_at = NULL, headers = 0, error = NULL,
+        rule_version = NULL
+      WHERE status <> 'planned' OR next_id <> start_id OR spooled_chunks <> 0 OR node_id IS NOT NULL`)
+  }
+
+  async down() {
+    this.schema.raw(`ALTER TABLE ranges DROP COLUMN IF EXISTS block_id`)
+    this.schema.raw(`DROP TABLE IF EXISTS blocks`)
+  }
+}

database/migrations/1770000006200_release_texts.ts

@@ -0,0 +1,35 @@
+import { BaseSchema } from '@adonisjs/lucid/schema'
+
+/**
+ * release_texts (REBUILD.md, "Voyager 3 … Schema"): every string search matches against, one table, one trigram
+ * index. The release name goes in (kind 1) with whatever anyone learns later: the prober's file names inside an
+ * archive, NFO titles, par2 set names, media titles (lib/release/text_kind.ts). Appending is an insert, never an
+ * update of the wide releases row; a repeat insert is a no-op by the primary key; the judge's dirty mark follows
+ * (judge_dirty_by_release_id, migration 4900), so a rule can react to a name arriving later. On the NVMe: search
+ * reads it. releases keeps name and inner_name for display, without trigram indexes (migration 6000).
+ */
+export default class extends BaseSchema {
+  async up() {
+    this.schema.raw(`
+      CREATE TABLE release_texts (
+        release_id bigint NOT NULL,
+        -- lib/release/text_kind.ts: 1 name, 2 file, 3 nfo, 4 par2, 5 media.
+        kind smallint NOT NULL,
+        text varchar(255) NOT NULL,
+        PRIMARY KEY (release_id, kind, text)
+      )`)
+    this.schema.raw(
+      `CREATE INDEX release_texts_trgm ON release_texts USING gin (text gin_trgm_ops) WITH (fastupdate = on, gin_pending_list_limit = 65536)`
+    )
+    this.schema.raw(
+      `CREATE TRIGGER judge_dirty_insert AFTER INSERT ON release_texts REFERENCING NEW TABLE AS new_rows FOR EACH STATEMENT EXECUTE FUNCTION judge_dirty_by_release_id()`
+    )
+    this.schema.raw(
+      `CREATE TRIGGER judge_dirty_delete AFTER DELETE ON release_texts REFERENCING OLD TABLE AS old_rows FOR EACH STATEMENT EXECUTE FUNCTION judge_dirty_by_release_id()`
+    )
+  }
+
+  async down() {
+    this.schema.raw(`DROP TABLE IF EXISTS release_texts`)
+  }
+}

database/migrations/1770000006300_pending_releases.ts

@@ -0,0 +1,60 @@
+import { BaseSchema } from '@adonisjs/lucid/schema'
+import env from '#start/env'
+
+/**
+ * pending_releases (REBUILD.md decision f, 2026-10-26): nothing enters the catalog without a usable name. A release
+ * the window closed on whose files all have random names is written here instead of releases: its keys (as the
+ * catalog would store them), its counts and article span, the upload tool from its message-id (with the group its
+ * signature), and its probe targets by effort (the edge's candidates: a par2, an archive start, a lone file, a 7z
+ * tail), no chunk. The pending ladder probes them cheapest first and moves a named one into the catalog, reading its
+ * segments again from the corpus or the server; a signature that names nothing is discarded. On CHUNK_TABLESPACE
+ * (the HDD): written once, read by the ladder, never by search.
+ */
+export default class extends BaseSchema {
+  async up() {
+    this.schema.raw(`
+      CREATE TABLE pending_releases (
+        -- lib/release/keys.ts: the identity, the group and the first article; the identity alone.
+        release_key bytea PRIMARY KEY,
+        identity_key bytea NOT NULL,
+        group_id integer NOT NULL,
+        -- The stem or post the edge keyed it by (a random name: no release name, but the inner-stem merge's handle).
+        name varchar(255) NOT NULL,
+        poster text,
+        posted_at timestamptz,
+        last_posted_at timestamptz,
+        first_article bigint NOT NULL,
+        last_article bigint NOT NULL,
+        files integer NOT NULL DEFAULT 0,
+        segments bigint NOT NULL DEFAULT 0,
+        bytes bigint NOT NULL DEFAULT 0,
+        -- A file misses parts.
+        partial boolean NOT NULL DEFAULT false,
+        -- False: a tail release still open; a later chunk of its key adds its counts up.
+        finished boolean NOT NULL DEFAULT true,
+        -- lib/rules uploadTool: ngPost, nyuu, hex32, random, other, …
+        tool varchar(16) NOT NULL,
+        -- [{kind, messageId, file}] by effort (lib/edge ProbeCandidate); empty: nothing to read, only a signature.
+        candidates jsonb NOT NULL DEFAULT '[]',
+        -- 0 waiting, 1 named (in the catalog), 2 discarded (its signature's verdict, or every probe gave nothing).
+        status smallint NOT NULL DEFAULT 0,
+        tries smallint NOT NULL DEFAULT 0,
+        created_at timestamptz NOT NULL DEFAULT now(),
+        probed_at timestamptz
+      )`)
+    // The ladder's order: a group's and tool's waiting releases, oldest first.
+    this.schema.raw(
+      `CREATE INDEX pending_releases_ladder_idx ON pending_releases (group_id, tool, status, created_at)`
+    )
+    const hdd = env.get('CHUNK_TABLESPACE')
+    if (hdd) {
+      this.schema.raw(`ALTER TABLE pending_releases SET TABLESPACE ${hdd}`)
+      this.schema.raw(`ALTER INDEX pending_releases_pkey SET TABLESPACE ${hdd}`)
+      this.schema.raw(`ALTER INDEX pending_releases_ladder_idx SET TABLESPACE ${hdd}`)
+    }
+  }
+
+  async down() {
+    this.schema.raw(`DROP TABLE IF EXISTS pending_releases`)
+  }
+}

database/migrations/1770000006400_pending_ladder.ts

@@ -0,0 +1,49 @@
+import { BaseSchema } from '@adonisjs/lucid/schema'
+
+/**
+ * The pending ladder (REBUILD.md decision f, 2026-10-26): what the prober needs to work pending_releases cheapest
+ * first and to learn which signatures never name anything.
+ * - pending_releases: `file_total` (the counter's, for the catalog row once named), `inspections` (what each probe
+ *   showed, by kind), `named` (the usable name a probe found; status 4 until the namer moved it into the catalog).
+ *   Status: 0 waiting, 1 named and in the catalog, 2 discarded (its signature's verdict), 3 exhausted (every
+ *   target probed, no name), 4 named, awaiting the move, 9 taken by a running prober.
+ * - pending_signatures: per (tool, group, poster) how many pending releases were probed to the end and how many
+ *   got a name; `dropped` once probed ≥ PENDING_MIN_PROBES and named ≤ PENDING_MAX_NAMED of them (voyager 2's
+ *   garbage learner, LOG.md 2026-10-08 (5)). The poster '' row is the tool's and group's together.
+ */
+export default class extends BaseSchema {
+  async up() {
+    this.schema.raw(`ALTER TABLE pending_releases ADD COLUMN IF NOT EXISTS file_total integer`)
+    this.schema.raw(
+      `ALTER TABLE pending_releases ADD COLUMN IF NOT EXISTS inspections jsonb NOT NULL DEFAULT '[]'`
+    )
+    this.schema.raw(`ALTER TABLE pending_releases ADD COLUMN IF NOT EXISTS named varchar(255)`)
+    // The ladder's claim: the oldest waiting ones; the namer's: the named ones.
+    this.schema.raw(
+      `CREATE INDEX IF NOT EXISTS pending_releases_waiting_idx ON pending_releases (created_at) WHERE status = 0`
+    )
+    this.schema.raw(
+      `CREATE INDEX IF NOT EXISTS pending_releases_naming_idx ON pending_releases (probed_at) WHERE status = 4`
+    )
+    this.schema.raw(`
+      CREATE TABLE pending_signatures (
+        tool varchar(16) NOT NULL,
+        group_id integer NOT NULL,
+        poster text NOT NULL DEFAULT '',
+        probed integer NOT NULL DEFAULT 0,
+        named integer NOT NULL DEFAULT 0,
+        dropped boolean NOT NULL DEFAULT false,
+        updated_at timestamptz NOT NULL DEFAULT now(),
+        PRIMARY KEY (tool, group_id, poster)
+      )`)
+  }
+
+  async down() {
+    this.schema.raw(`DROP TABLE IF EXISTS pending_signatures`)
+    this.schema.raw(`DROP INDEX IF EXISTS pending_releases_naming_idx`)
+    this.schema.raw(`DROP INDEX IF EXISTS pending_releases_waiting_idx`)
+    this.schema.raw(`ALTER TABLE pending_releases DROP COLUMN IF EXISTS named`)
+    this.schema.raw(`ALTER TABLE pending_releases DROP COLUMN IF EXISTS inspections`)
+    this.schema.raw(`ALTER TABLE pending_releases DROP COLUMN IF EXISTS file_total`)
+  }
+}

deploy/compose.yaml

@@ -190,6 +190,21 @@ services:
       node-api:
         condition: service_started
 
+  # The replay harness into the scratch catalog usenet_replay on bulk (REBUILD.md, "The replay harness"):
+  # docker compose run --rm --no-deps replay --group alt.binaries.ath
+  replay:
+    <<: *app
+    profiles: [replay]
+    restart: 'no'
+    entrypoint: ['node', 'ace', 'replay:corpus']
+    command: ['--help']
+    environment:
+      <<: *app-env
+      DB_DATABASE: usenet_replay
+    volumes:
+      - ./corpus:/corpus:ro
+    depends_on: []
+
   # One-off ace commands: docker compose run --rm ace releases:duplicates
   ace:
     <<: *app

deploy/deploy.sh

@@ -51,7 +51,14 @@ if [ -e "$CACHE/paused" ]; then
   docker compose up -d --no-deps node-api web
 else
   docker compose up -d --no-deps node-api web coordinator prober merger judge spotter spot-inspector
+  if [ -e "$CACHE/no-edge" ]; then
+    # Voyager 3 (2026-10-26, LOG.md (116)): the worker indexes range by range, without the carried window, and
+    # would split releases at every range edge with nothing to join them; history comes from the corpus through
+    # deploy/replay-window.sh until the window worker is built. Remove the marker to start it again.
+    echo "== no-edge: the edge worker left stopped"
+  else
     docker compose up -d --no-deps edge
+  fi
 fi
 echo "$SHA $(date -Is)" >> "$ROOT/deployed.log"
 # The images this replaced (321 MB each; the root filesystem is the NVMe, with a 10 GB floor).

deploy/replay-window.sh

@@ -0,0 +1,74 @@
+#!/bin/bash
+# A corpus window replayed on homelab (REBUILD.md, "The replay harness"): every group whose window ranges (the
+# ranges marked `capture` at PRIORITY in the catalog, 0 for 2016) are all stored in the corpus is replayed once
+# into the scratch catalog usenet_replay through the voyager-3 app image, PARALLEL groups at a time, the
+# window's bounds passed as --from/--to (the corpus holds other windows of the same group). The group's log
+# ends in .cache/replay/<group>.log, a .done marker next to it (.failed when the replay failed: remove it to
+# retry). Loops until every group of the window is done (the corpus fill runs meanwhile), so run it detached
+# from the deployment root:
+#   setsid nohup deploy/replay-window.sh > .cache/replay/driver.log 2>&1 &
+# No new group starts while the nightly pause is on (.cache/paused); the running ones finish.
+# DB=usenet_rebuild replays into the catalog itself (voyager 3 deployed, 2026-10-26: history comes from the corpus
+# this way until the window worker is built); the harness then needs REPLAY_INTO_CATALOG=1, passed here. OUT keeps
+# a run's logs and markers apart (.cache/replay-catalog for the catalog's).
+set -u
+ROOT=${ROOT:-/mnt/bulk/usenet-rebuild}
+PRIORITY=${PRIORITY:-0}
+PARALLEL=${PARALLEL:-4}
+MEMORY=${MEMORY:-12g}
+IMAGE=${IMAGE:-usenet-voyager3:latest}
+DB=${DB:-usenet_replay}
+OUT=${OUT:-$ROOT/.cache/replay}
+mkdir -p "$OUT"
+cd "$ROOT" || exit 1
+
+say() { echo "$(date -Is) $*"; }
+catalog() { docker compose exec -T postgres psql -U rebuild -d usenet_rebuild -X -q -At -F ' ' -c "$1"; }
+
+# The window's groups: name, first article, last article, whether every range is stored; smallest first.
+window() {
+  catalog "SELECT g.name, min(r.start_id), max(r.end_id), (count(*) = count(r.stored_at))::int
+           FROM ranges r JOIN groups g ON g.id = r.group_id
+           WHERE r.capture AND r.priority = $PRIORITY
+           GROUP BY g.name ORDER BY sum(r.end_id - r.start_id + 1)"
+}
+
+replay() {
+  local group=$1 from=$2 to=$3
+  say "$group: replaying $from–$to"
+  if docker run --rm --name "replay-${group//./-}" --network usenet-rebuild --env-file .env \
+      -e NODE_ENV=production -e HOST=0.0.0.0 -e PORT=3334 -e APP_URL=http://web:3334 -e SPOOL_DIR=/spool -e CORPUS_DIR=/corpus \
+      -e DB_HOST=postgres -e DB_PORT=5432 -e DB_DATABASE="$DB" -e CHUNK_TABLESPACE=pg_default -e REPLAY_INTO_CATALOG=1 \
+      -v "$ROOT/corpus:/corpus:ro" --memory "$MEMORY" "$IMAGE" \
+      node ace replay:corpus --group "$group" --from "$from" --to "$to" > "$OUT/$group.log" 2>&1; then
+    touch "$OUT/$group.done"
+    say "$group: done ($(grep -o 'Releases [0-9,]*' "$OUT/$group.log" | tail -1))"
+  else
+    touch "$OUT/$group.failed"
+    say "$group: FAILED (see $OUT/$group.log)"
+  fi
+  rm -f "$OUT/$group.running"
+}
+
+while true; do
+  todo=0
+  while read -r group from to complete; do
+    [ -n "$group" ] || continue
+    [ -e "$OUT/$group.done" ] && continue
+    [ -e "$OUT/$group.failed" ] && continue
+    todo=$((todo + 1))
+    [ "$complete" = 1 ] || continue
+    [ -e "$OUT/$group.running" ] && continue
+    [ -e "$ROOT/.cache/paused" ] && continue
+    while [ "$(find "$OUT" -name '*.running' | wc -l)" -ge "$PARALLEL" ]; do sleep 30; done
+    touch "$OUT/$group.running"
+    replay "$group" "$from" "$to" &
+    sleep 5
+  done <<< "$(window)"
+  if [ "$todo" = 0 ]; then
+    wait
+    say "every group of the window at priority $PRIORITY is replayed"
+    exit 0
+  fi
+  sleep 300
+done

deploy/usenet-rebuild-resume.sh

@@ -7,5 +7,10 @@ flock 9
 rm -f .cache/paused
 date "+%F %T resuming"
 docker compose up -d --no-deps coordinator prober merger judge spotter spot-inspector
-docker compose up -d --no-deps edge
+if [ -e .cache/no-edge ]; then
+  # Voyager 3 without its window worker yet (deploy.sh, LOG.md (116)): the edge stays stopped.
+  echo "no-edge: the edge worker left stopped"
+else
+  docker compose up -d --no-deps edge
+fi
 docker compose ps --format "{{.Service}} {{.Status}}"

edge/Cargo.toml

@@ -2,7 +2,7 @@
 name = "edge"
 version = "0.1.0"
 edition = "2021"
-description = "Voyager 2's edge worker (edge-node) and its benchmark (edge-bench): the RangeIndexer in Rust, kept identical to lib/edge (check.ts)"
+description = "The edge worker (edge-node) and its benchmark (edge-bench): voyager 3's window indexing in Rust"
 
 [[bin]]
 name = "edge-bench"

edge/README.md

@@ -6,18 +6,21 @@ from the node API, reads their prefetched XOVER chunks, finds releases and
 posts batches. `edge-bench` (`src/main.rs`) replays a captured range through
 the same indexing (`src/lib.rs`) for timing and for comparing.
 
-The TypeScript `RangeIndexer` (`lib/edge`, `node ace edge:node`) is the
-reference: `check.ts` runs both on a capture and compares every output field
-by field. An identity or rule change goes into both and has to come out
-identical. CI (`.gitea/workflows/deploy.yaml`, job `edge`) runs `cargo test`
-and `check-fixtures.sh`, which compares both on the committed captures in
-`fixtures/` (25k overview lines each of a.b.models, a.b.ath and
-a.b.boneless, in 5 chunks); a push to `rebuild` deploys only when they
-match.
+Since voyager 3 (2026-10-25) the Rust indexer is the one implementation of
+the window (REBUILD.md, "Voyager 3: window indexing"; `src/indexer.rs`):
+files and releases are finished when the frontier (the median Date of the
+last 1,024 plausibly dated articles) is 24 h past their last part, or at
+the block's end (provisionally, as edges), and written once. The
+TypeScript `RangeIndexer` (`lib/edge`) keeps the range path for the corpus
+scanner and the tests; the field-by-field comparison (`check.ts`) went
+with the parity. CI (`.gitea/workflows/deploy.yaml`, job `edge`) runs
+`cargo test` and `bench-fixtures.sh`, which replays the committed captures
+in `fixtures/` (25k overview lines each of a.b.ath, a.b.boneless, a.b.fta
+and a.b.models, in 5 chunks).
 
 ```sh
 cargo test --release --manifest-path edge/Cargo.toml
-edge/check-fixtures.sh                       # from the repository root
+edge/bench-fixtures.sh                       # from the repository root
 ```
 
 In production since 2026-10-09: 122M headers in 10 minutes with 0.37 cores
@@ -110,11 +113,30 @@ files, segments, payload size, candidates, fragments and samples. The
 summary can hide compensating differences, so compare the outputs
 themselves.
 
-## Comparing the output
+## Replaying a group through one window
+
+A corpus group directory (`<corpus>/<group>/<start>-<end>/`, each range with
+its `range.json`) is replayed in article order with the window carried from
+range to range, the end finished after the last range (REBUILD.md, "The
+replay harness"):
+
+```sh
+./target/release/edge-bench --group /corpus/alt.binaries.ath --rules rules.json --batch-chunks 5 --wire - > batches.jsonl
+# --from <article> skips ranges ending before it, --to <article> ranges starting after it (one window of the
+# group's history), --limit <n> stops after n ranges, --no-finish leaves the window open
+```
+
+`--rules` takes the lease's rules JSON (`src/lease.rs`: the catalog's rule
+set and group priorities, as `replay:corpus` writes it); without it the
+defaults with only the group indexed apply. `--wire -` writes each batch as
+the worker posts it to stdout, one JSON per line, and the report to stderr;
+`node ace replay:corpus` reads them into the batch store. Each range end
+prints the frontier, the releases out and the releases and files still open.
+
+## The dump
 
 ```sh
 ./target/release/edge-bench /path/to/capture --dump /tmp/out.jsonl
-cd .. && node --import=@poppinss/ts-exec edge/check.ts /path/to/capture /tmp/out.jsonl
 ```
 
 `--dump` writes every output of every batch as JSON lines:

edge/bench-fixtures.sh

@@ -0,0 +1,11 @@
+#!/bin/sh
+# The Rust indexer over the committed captures (edge/fixtures: 25k overview lines each of a.b.ath, a.b.boneless,
+# a.b.fta and a.b.models, in 5 chunks): a smoke test of the window on real headers, exiting non-zero when the bench
+# fails on one. CI runs it after cargo test; run it from the repository root after a change to the indexer.
+set -eu
+cd "$(dirname "$0")/.."
+cargo build --release --quiet --manifest-path edge/Cargo.toml --bin edge-bench
+for capture in edge/fixtures/*/; do
+  echo "== $(basename "$capture")"
+  edge/target/release/edge-bench "$capture" --batch-chunks 2 --idle 2 | tail -1
+done

edge/check-fixtures.sh

@@ -1,18 +0,0 @@
-#!/bin/sh
-# The Rust and TypeScript RangeIndexers on the committed captures (edge/fixtures: 25k
-# overview lines each of a.b.models, a.b.ath and a.b.boneless, in 5 chunks), compared
-# by check.ts field by field: exits non-zero on any difference. CI runs it; run it from
-# the repository root after any change to either side.
-set -eu
-cd "$(dirname "$0")/.."
-cargo build --release --quiet --manifest-path edge/Cargo.toml --bin edge-bench
-out=$(mktemp -d)
-trap 'rm -rf "$out"' EXIT
-status=0
-for capture in edge/fixtures/*/; do
-  name=$(basename "$capture")
-  edge/target/release/edge-bench "$capture" --batch-chunks 2 --idle 2 --dump "$out/$name.jsonl" --wire "$out/$name.wire.jsonl" >/dev/null 2>&1
-  echo "== $name"
-  node --import=@poppinss/ts-exec edge/check.ts "$capture" "$out/$name.jsonl" --wire "$out/$name.wire.jsonl" --batch-chunks 2 --idle 2 || status=1
-done
-exit $status

edge/check.ts

@@ -1,139 +0,0 @@
-/**
- * Compares edge-bench's output (--dump) with the TypeScript RangeIndexer
- * on the same captured range, item by item: releases (every field, and the
- * payload decoded with lib/chunks), samples, probe candidates, rule counts,
- * fragments and batch boundaries.
- *
- *   node --import=@poppinss/ts-exec edge/check.ts <dir> <dump.jsonl> [--wire batches.jsonl] [--batch-chunks 10] [--idle 5]
- *
- * With --wire (edge-bench --wire): also each batch as the Rust worker
- * posts it, against toWire of the TypeScript batch, field by field.
- */
-import { readFile, readdir } from 'node:fs/promises'
-import { join } from 'node:path'
-import { unzipSync } from 'node:zlib'
-import { isDeepStrictEqual } from 'node:util'
-import { RangeIndexer, toWire, type EdgeBatch } from '#lib/edge/index'
-import { DEFAULT_RULES, type Rules } from '#lib/rules/index'
-import { decodeChunk } from '#lib/chunks/index'
-
-const [dir, dumpPath] = process.argv.slice(2)
-const flag = (name: string, fallback: number) => {
-  const i = process.argv.indexOf(name)
-  return i === -1 ? fallback : Number(process.argv[i + 1])
-}
-const batchChunks = flag('--batch-chunks', 10)
-const idle = flag('--idle', 5)
-
-const { group, start, end } = JSON.parse(await readFile(join(dir, 'range.json'), 'utf8'))
-const rules: Rules = { ...DEFAULT_RULES, priorities: { [group]: 1 }, droppedSignatures: [] }
-const indexer = new RangeIndexer({ group, start, end, rules, idleChunks: idle })
-const files = (await readdir(dir)).filter((f) => f.endsWith('.gz')).sort()
-const batches: EdgeBatch[] = []
-for (const [i, file] of files.entries()) {
-  const next = Number(file.split('-')[2].replace('.gz', '')) + 1
-  indexer.addText(unzipSync(await readFile(join(dir, file))).toString('utf8'), next)
-  const last = i === files.length - 1
-  if (last) indexer.finish()
-  if (last || (i + 1) % batchChunks === 0) batches.push(indexer.take())
-}
-
-/** Canonical lines per kind, in a stable order, from both sides. */
-type Item = Record<string, unknown>
-const sortFiles = (decoded: ReturnType<typeof decodeChunk>) => [...decoded].sort((a, b) => (a.name + a.subject < b.name + b.subject ? -1 : 1))
-const payloadBytes = { ts: 0, rs: 0 }
-let side: 'ts' | 'rs' = 'ts'
-const release = (r: Item, b: number) => {
-  const { payload, ...rest } = r as Item & { payload: Buffer }
-  // The compressed size isn't compared (node's zstd and the zstd crate frame alike data a little
-  // differently); what the payload decodes to is.
-  payloadBytes[side] += payload.length
-  return { b, ...rest, decoded: sortFiles(decodeChunk(payload)) }
-}
-const ts: Record<string, Item[]> = { b: [], r: [], s: [], c: [], n: [], f: [] }
-batches.forEach((batch, b) => {
-  ts.b.push({ b, from: batch.from, next: batch.next, headers: batch.headers, postedLow: batch.postedLow, postedHigh: batch.postedHigh })
-  for (const r of batch.releases) ts.r.push(release({ upload: r.upload, key: r.key, name: r.name, source: r.source, fileTotal: r.fileTotal, poster: r.poster, groups: r.groups, postedAt: r.postedAt, postedLast: r.postedLast, files: r.files, segments: r.segments, bytes: r.bytes, firstArticle: r.firstArticle, lastArticle: r.lastArticle, readable: r.readable, hasPar2: r.hasPar2, payload: r.payload }, b))
-  for (const s of batch.samples) ts.s.push({ b, key: s.key, upload: s.upload, rule: s.rule, ruleKey: s.ruleKey, segments: s.segments })
-  for (const c of batch.candidates) ts.c.push({ b, upload: c.upload, releaseKey: c.releaseKey, kind: c.kind, messageId: c.messageId, file: c.file })
-  for (const n of batch.counts) ts.n.push({ b, rule: n.rule, key: n.key, articles: n.articles, files: n.files })
-  for (const f of batch.fragments) ts.f.push({ b, fileKey: f.fileKey, name: f.name, totalParts: f.totalParts, postedAt: f.postedAt, segments: f.segments })
-})
-
-const rs: Record<string, Item[]> = { b: [], r: [], s: [], c: [], n: [], f: [] }
-side = 'rs'
-let b = -1
-for (const line of (await readFile(dumpPath, 'utf8')).split('\n')) {
-  if (!line) continue
-  const { t, ...item } = JSON.parse(line)
-  if (t === 'b') b++
-  if (t === 'r') rs.r.push(release({ ...item, payload: Buffer.from(item.payload, 'base64') }, b))
-  else if (t === 's') rs.s.push({ b, key: item.key, upload: item.upload, rule: item.rule, ruleKey: item.ruleKey, segments: item.segments })
-  else rs[t].push({ b, ...item })
-}
-
-const keyOf: Record<string, (i: Item) => string> = {
-  b: (i) => String(i.b),
-  r: (i) => `${i.b} ${i.key} ${i.upload}`,
-  s: (i) => `${i.b} ${i.key} ${i.upload}`,
-  c: (i) => `${i.b} ${i.releaseKey} ${i.upload} ${i.kind} ${i.messageId}`,
-  n: (i) => `${i.b} ${i.rule} ${i.key}`,
-  f: (i) => `${i.b} ${i.fileKey}`,
-}
-const names: Record<string, string> = { b: 'batches', r: 'releases', s: 'samples', c: 'candidates', n: 'rule counts', f: 'fragments' }
-let failed = false
-for (const kind of Object.keys(keyOf)) {
-  const a = new Map(ts[kind].map((i) => [keyOf[kind](i), i]))
-  const z = new Map(rs[kind].map((i) => [keyOf[kind](i), i]))
-  const missing = [...a.keys()].filter((k) => !z.has(k))
-  const extra = [...z.keys()].filter((k) => !a.has(k))
-  const differ = [...a.keys()].filter((k) => z.has(k) && !isDeepStrictEqual(a.get(k), z.get(k)))
-  const ok = missing.length === 0 && extra.length === 0 && differ.length === 0
-  failed ||= !ok
-  console.log(`${ok ? 'same' : 'DIFF'} ${names[kind]}: ${a.size} TypeScript, ${z.size} Rust${ok ? '' : `; ${missing.length} only in TypeScript, ${extra.length} only in Rust, ${differ.length} differ`}`)
-  for (const k of [...missing.slice(0, 2), ...extra.slice(0, 2)]) console.log(`  only one side: ${k}`)
-  for (const k of differ.slice(0, 2)) {
-    const x = a.get(k)!
-    const y = z.get(k)!
-    const fields = Object.keys(x).filter((f) => !isDeepStrictEqual(x[f], y[f]))
-    console.log(`  ${k}: ${fields.map((f) => `${f}: ${JSON.stringify(x[f]).slice(0, 160)} vs ${JSON.stringify(y[f]).slice(0, 160)}`).join('; ')}`)
-  }
-}
-console.log(`payload bytes: ${payloadBytes.ts} TypeScript, ${payloadBytes.rs} Rust`)
-
-const wireIndex = process.argv.indexOf('--wire')
-if (wireIndex !== -1) {
-  // Order within a batch doesn't count; payloads by what they decode to.
-  const by = (f: (x: any) => string) => (a: any, b: any) => (f(a) < f(b) ? -1 : f(a) > f(b) ? 1 : 0)
-  const normal = (w: any) => ({
-    ...w,
-    releases: w.releases.map((r: any) => ({ ...r, payload: sortFiles(decodeChunk(Buffer.from(r.payload, 'base64'))) })).sort(by((r) => `${r.key} ${r.upload}`)),
-    samples: w.samples.map((r: any) => ({ ...r, payload: sortFiles(decodeChunk(Buffer.from(r.payload, 'base64'))) })).sort(by((r) => `${r.key} ${r.upload}`)),
-    candidates: [...w.candidates].sort(by((c) => JSON.stringify(c))),
-    counts: [...w.counts].sort(by((c) => `${c.rule} ${c.key}`)),
-    fragments: [...w.fragments].sort(by((f) => `${f.fileKey} ${f.segments[0]?.[1]}`)),
-  })
-  const rust = (await readFile(process.argv[wireIndex + 1], 'utf8')).split('\n').filter(Boolean).map((l) => normal(JSON.parse(l)))
-  const typescript = batches.map((b) => normal(JSON.parse(JSON.stringify(toWire(b)))))
-  let wireSame = rust.length === typescript.length
-  const firstDiff = (x: any, y: any, path: string): string | null => {
-    if (isDeepStrictEqual(x, y)) return null
-    if (typeof x !== 'object' || typeof y !== 'object' || x === null || y === null) return `${path}: ${JSON.stringify(x)?.slice(0, 120)} vs ${JSON.stringify(y)?.slice(0, 120)}`
-    for (const k of new Set([...Object.keys(x), ...Object.keys(y)])) {
-      const d = firstDiff(x[k], y[k], `${path}.${k}`)
-      if (d) return d
-    }
-    return path
-  }
-  for (let i = 0; i < Math.max(rust.length, typescript.length); i++) {
-    const d = firstDiff(typescript[i], rust[i], `batch ${i}`)
-    if (d) {
-      wireSame = false
-      console.log(`DIFF wire: ${d}`)
-      break
-    }
-  }
-  if (wireSame) console.log(`same wire batches: ${typescript.length} TypeScript, ${rust.length} Rust`)
-  failed ||= !wireSame
-}
-process.exit(failed ? 1 : 0)

edge/src/bin/edge-node.rs

@@ -9,7 +9,7 @@
 //! (the next worker resumes at the last stored batch).
 
 use edge::indexer::{Batch, RangeIndexer};
-use edge::names::Rules;
+use edge::lease::LeaseJson;
 use edge::wire;
 use flate2::write::GzEncoder;
 use serde::Deserialize;
@@ -34,63 +34,6 @@ fn log(level: &str, message: &str) {
     println!("[ {level} ] {message}");
 }
 
-#[derive(Deserialize)]
-struct LeaseJson {
-    range: RangeJson,
-    rules: RulesJson,
-}
-
-#[derive(Deserialize)]
-struct RangeJson {
-    id: i64,
-    group: String,
-    start: i64,
-    end: i64,
-    next: i64,
-}
-
-#[derive(Deserialize)]
-#[serde(rename_all = "camelCase")]
-struct RulesJson {
-    version: i64,
-    priorities: HashMap<String, u32>,
-    #[serde(default)]
-    dropped_signatures: Vec<String>,
-    #[serde(default)]
-    dropped_tools: Option<Vec<String>>,
-    #[serde(default)]
-    per_article_max_parts: Option<u32>,
-    #[serde(default)]
-    sample_percent: Option<f64>,
-    #[serde(default)]
-    drop_executables: Option<bool>,
-    #[serde(default)]
-    dropped_posters: Option<Vec<String>>,
-}
-
-impl RulesJson {
-    fn rules(self) -> Rules {
-        let mut rules = Rules::defaults(self.priorities.into_iter().collect());
-        rules.version = self.version;
-        rules.dropped_signatures = self.dropped_signatures.into_iter().collect();
-        if let Some(tools) = self.dropped_tools {
-            rules.dropped_tools = tools.into_iter().collect();
-        }
-        if let Some(n) = self.per_article_max_parts {
-            rules.per_article_max_parts = n;
-        }
-        if let Some(p) = self.sample_percent {
-            rules.sample_percent = p;
-        }
-        if let Some(e) = self.drop_executables {
-            rules.drop_executables = e;
-        }
-        if let Some(posters) = self.dropped_posters {
-            rules.dropped_posters = posters;
-        }
-        rules
-    }
-}
 
 #[derive(Deserialize)]
 struct StoredJson {

edge/src/indexer.rs

@@ -1,6 +1,7 @@
-//! lib/edge/scan.ts (SegmentScanner, scanOverviewText), lib/files/aggregator.ts
-//! and lib/edge/range_indexer.ts (RangeIndexer, chunkOf, candidatesOf), in the
-//! same order of operations, so the outputs can be compared one to one.
+//! The edge's indexing: lib/edge/scan.ts (SegmentScanner, scanOverviewText) and lib/files/aggregator.ts as in the
+//! TypeScript, then voyager 3's window (REBUILD.md, "Voyager 3: window indexing", 2): files and releases are
+//! finished when the frontier (the median Date of the last FRONTIER_ARTICLES plausible ones) is WINDOW_MS past
+//! their last part, or at the block's end (provisionally, as edges), and written once.
 
 use crate::codec::{encode_chunk, ChunkFile};
 use crate::js::{self, sha1_hex};
@@ -46,6 +47,17 @@ pub struct ReleaseChunk {
     pub last_article: i64,
     pub readable: bool,
     pub has_par2: bool,
+    /// The window closed on it (false: still open at the block's end, written provisionally).
+    pub finished: bool,
+    /// Open at the block's end, or within the window of its start: the block merger's.
+    pub edge: bool,
+    /// A file misses parts.
+    pub partial: bool,
+    /// The upload tool from the first article's message-id (names::upload_tool), for an unnamed release: with the
+    /// group its signature, what the pending ladder learns verdicts by. None for a named one.
+    pub tool: Option<&'static str>,
+    /// The files' names and segments (codec); empty for an unnamed release (decision f: nothing in the catalog
+    /// without a usable name; the pending store keeps its keys and probe targets until a probe names it).
     pub payload: Vec<u8>,
     /// The first files' subjects (at most SUBJECTS): edge-bench's --releases dump for the grouping sweeps.
     pub subjects: Vec<String>,
@@ -61,18 +73,6 @@ pub struct Candidate {
     pub file: String,
 }
 
-pub struct Fragment {
-    pub file_key: String,
-    pub name: String,
-    pub poster: String,
-    pub subject: String,
-    pub total_parts: u32,
-    pub groups: Vec<String>,
-    pub release: Release,
-    pub posted_at: Option<i64>,
-    pub segments: Vec<(u32, String, i64, i64)>,
-}
-
 #[derive(Default)]
 pub struct Batch {
     pub from: i64,
@@ -86,7 +86,6 @@ pub struct Batch {
     pub candidates: Vec<Candidate>,
     /// (rule, key) → (articles, files)
     pub counts: Vec<(&'static str, String, u64, u64)>,
-    pub fragments: Vec<Fragment>,
 }
 
 struct Counter {
@@ -470,8 +469,118 @@ fn scan_overview_text(text: &str, mut line: impl FnMut(Option<i64>, &str, &str,
     }
 }
 
-/// A re-upload's copy of a file is kept apart (FileAggregator, uploadGapMs).
+/// A re-upload's copy of a file is kept apart (FileAggregator, uploadGapMs); a file joins the open release of its
+/// key posted within this gap of it, else starts one (uploadsOf, online).
 pub const UPLOAD_GAP_MS: i64 = 3 * 60 * 60 * 1000;
+/// The window (REBUILD.md, "Voyager 3: window indexing", 2): a file still missing parts, and a release, are finished
+/// once the frontier is this far past their last part.
+pub const WINDOW_MS: i64 = 24 * 60 * 60 * 1000;
+/// The frontier is the median Date of the last this many plausibly dated articles: enough that a minority of skewed
+/// dates can't move it, few enough that it trails the newest article by little even in a quiet group (half the
+/// ring: a day at 500 articles a day).
+pub const FRONTIER_ARTICLES: usize = 1024;
+/// A Date further than this from the frontier isn't believed (clock skew, bogus years); FRONTIER_ARTICLES of them
+/// in a row that agree among themselves move it (the group paused for longer).
+pub const PLAUSIBLE_MS: i64 = 7 * 24 * 60 * 60 * 1000;
+/// The frontier is established once this many dated articles were seen (a lone bogus date doesn't set it).
+const FRONTIER_MIN: usize = 64;
+/// Albums (REBUILD.md, "Voyager 3: window indexing", 3): a poster's single-file releases whose file names differ only
+/// by number sequences, posted within WINDOW_MS of the album's first, written as one release (`source: "album"`); an
+/// album holds at most this many files (a dump of 38k pictures a day is a few albums, not one).
+pub const ALBUM_MAX_FILES: usize = 1000;
+/// A counter total this large, shared by this many releases of one poster, is a batch counter (Yenc-PP-A&A's
+/// [n/68067] over thousands of different programs), not a file count: those releases get no file total.
+pub const BATCH_COUNTER_MIN_TOTAL: i64 = 100;
+pub const BATCH_COUNTER_RELEASES: u32 = 20;
+/// Batches after which a poster's counter total is forgotten (BATCH_COUNTER_RELEASES must come within them).
+const BATCH_COUNTER_BATCHES: u64 = 100;
+
+fn median(dates: &mut [i64]) -> i64 {
+    let mid = dates.len() / 2;
+    *dates.select_nth_unstable(mid).1
+}
+
+/// Where the block is in posting time: the median Date of the last FRONTIER_ARTICLES plausible ones. Date headers
+/// alone aren't trusted: a date more than PLAUSIBLE_MS from the frontier is set aside, and only FRONTIER_ARTICLES
+/// such dates in a row, agreeing among themselves, move the frontier there.
+#[derive(Default)]
+pub struct Frontier {
+    ring: Vec<i64>,
+    at: usize,
+    value: Option<i64>,
+    /// The frontier when first established: the block's start in posting time (releases within WINDOW_MS of it are edges).
+    start: Option<i64>,
+    /// Dates set aside since the last plausible one.
+    aside: Vec<i64>,
+    pending: usize,
+}
+
+impl Frontier {
+    pub fn add(&mut self, t: i64) {
+        if let Some(f) = self.value {
+            if (t - f).abs() > PLAUSIBLE_MS {
+                self.aside.push(t);
+                if self.aside.len() >= FRONTIER_ARTICLES {
+                    let m = median(&mut self.aside.clone());
+                    if self.aside.iter().filter(|&&d| (d - m).abs() <= PLAUSIBLE_MS).count() * 2 >= self.aside.len() {
+                        self.ring = self.aside.iter().copied().filter(|&d| (d - m).abs() <= PLAUSIBLE_MS).collect();
+                        self.ring.truncate(FRONTIER_ARTICLES);
+                        self.at = self.ring.len() % FRONTIER_ARTICLES;
+                        self.value = Some(m);
+                    }
+                    self.aside.clear();
+                }
+                return;
+            }
+            self.aside.clear();
+        }
+        if self.ring.len() < FRONTIER_ARTICLES {
+            self.ring.push(t);
+        } else {
+            self.ring[self.at] = t;
+        }
+        self.at = (self.at + 1) % FRONTIER_ARTICLES;
+        self.pending += 1;
+    }
+
+    /// Recomputes the frontier (once per chunk: a median of FRONTIER_ARTICLES isn't for every article).
+    pub fn update(&mut self) {
+        if self.pending == 0 || (self.value.is_none() && self.ring.len() < FRONTIER_MIN) {
+            return;
+        }
+        self.pending = 0;
+        let m = median(&mut self.ring.clone());
+        self.value = Some(m);
+        if self.start.is_none() {
+            self.start = Some(m);
+        }
+    }
+
+    /// At the block's end: whatever dates there are set the frontier (a small block never reached FRONTIER_MIN).
+    pub fn settle(&mut self) {
+        if self.value.is_none() && !self.ring.is_empty() {
+            self.pending = 1;
+            let m = median(&mut self.ring.clone());
+            self.value = Some(m);
+            self.start = Some(m);
+        } else {
+            self.update();
+        }
+    }
+
+    pub fn value(&self) -> Option<i64> {
+        self.value
+    }
+
+    pub fn start(&self) -> Option<i64> {
+        self.start
+    }
+
+    /// The posting time before which a last part means the window closed.
+    fn closed_before(&self) -> Option<i64> {
+        self.value.map(|f| f - WINDOW_MS)
+    }
+}
 
 struct OpenFile {
     parts: Vec<Seg>,
@@ -570,105 +679,197 @@ impl Aggregator {
         complete
     }
 
-    /// Files idle for `idle` batches (all when 0): their segments, per file key (its copies one after another).
-    fn evict(&mut self, idle: u64) -> Vec<Vec<Seg>> {
+    /// Files the window closed on, still missing parts: those whose last part is before `closed_before`, undated ones
+    /// idle for `idle` batches (all when `all`). Their segments, part order.
+    fn evict(&mut self, closed_before: Option<i64>, idle: u64, all: bool) -> Vec<Vec<Seg>> {
         let batch = self.batch;
         let mut out = Vec::new();
         let keys: Vec<String> = self.open.keys().cloned().collect();
         for key in keys {
             let copies = self.open.get_mut(&key).unwrap();
-            let mut gone: Vec<Seg> = Vec::new();
             let mut i = 0;
             while i < copies.len() {
-                if idle > 0 && batch - copies[i].touched < idle {
+                let c = &copies[i];
+                let closed = match c.latest {
+                    Some(l) => closed_before.is_some_and(|before| l < before),
+                    None => idle > 0 && batch - c.touched >= idle,
+                };
+                if !all && !closed {
                     i += 1;
                     continue;
                 }
-                gone.extend(copies.remove(i).parts);
+                let mut f = copies.remove(i);
+                f.parts.sort_by_key(|s| s.number);
+                out.push(f.parts);
             }
             if copies.is_empty() {
                 self.open.remove(&key);
             }
-            if !gone.is_empty() {
-                out.push(gone);
-            }
         }
         out
     }
 }
 
-/// uploadsOf: a key's files split where posting pauses for more than UPLOAD_GAP_MS after the latest article so far.
-fn uploads_of(files: Vec<Vec<Seg>>) -> Vec<Vec<Vec<Seg>>> {
-    if files.len() < 2 {
-        return vec![files];
+/// A release still open in the window: the files of one upload of its key (one after another, as they finished).
+struct OpenRelease {
+    upload: usize,
+    files: Vec<Vec<Seg>>,
+    first: Option<i64>,
+    last: Option<i64>,
+    touched: u64,
+}
+
+impl OpenRelease {
+    /// How far a file's posting span is from the release's (0 when either is undated).
+    fn distance(&self, first: Option<i64>, last: Option<i64>) -> i64 {
+        match (first, last, self.first, self.last) {
+            (Some(ff), Some(fl), Some(rf), Some(rl)) => 0.max(rf - fl).max(ff - rl),
+            _ => 0,
         }
-    let first: Vec<Option<i64>> = files.iter().map(|f| posted_of(f)).collect();
-    let last: Vec<Option<i64>> = files.iter().map(|f| f.iter().filter_map(|s| s.posted_at).max()).collect();
-    let mut dated: Vec<usize> = (0..files.len()).filter(|&i| first[i].is_some()).collect();
-    if dated.len() < 2 {
-        return vec![files];
     }
-    dated.sort_by_key(|&i| (first[i].unwrap(), i));
-    let mut upload = vec![0usize; files.len()];
-    let mut current = 0;
-    let mut end = last[dated[0]].unwrap();
-    for &i in &dated[1..] {
-        if first[i].unwrap() - end > UPLOAD_GAP_MS {
-            current += 1;
-            end = last[i].unwrap();
-        } else if last[i].unwrap() > end {
-            end = last[i].unwrap();
+}
+
+/// An album still open: the closed singles of one poster and name template (ALBUM_MAX_FILES at most), with their
+/// release keys, so a lone member is written as the release it was.
+struct OpenAlbum {
+    upload: usize,
+    members: Vec<(String, OpenRelease)>,
+    first: Option<i64>,
+    last: Option<i64>,
+    touched: u64,
+}
+
+/// The template a file name joins an album by: every digit run as "#" ("1481439079025_014.jpg.data" →
+/// "#_#.jpg.data"); none for a name without a number, a random one, an archive, a volume or a sidecar.
+pub fn album_template(name: &str) -> Option<String> {
+    let (stem, ext) = split_extension(name);
+    // A numbered extension is a volume ("acs5152s1.11"), and a stray suffix hides none of the above ("x.par2.temp").
+    let volume = |n: &str| names::is_par2(n) || names::is_archive_start(n) || names::is_later_volume(n);
+    if is_random_file(name) || names::is_sidecar(name) || volume(name) || volume(stem) || (ext.len() > 1 && ext.bytes().skip(1).all(|c| c.is_ascii_digit())) {
+        return None;
     }
-        upload[i] = current;
+    let mut out = String::with_capacity(name.len());
+    let mut runs = 0;
+    let mut in_digits = false;
+    for c in stem.chars() {
+        if c.is_ascii_digit() {
+            if !in_digits {
+                out.push('#');
+                runs += 1;
+                in_digits = true;
             }
-    if current == 0 {
-        return vec![files];
+        } else {
+            in_digits = false;
+            out.push(c);
         }
-    let mut out: Vec<Vec<Vec<Seg>>> = (0..=current).map(|_| Vec::new()).collect();
-    for (i, f) in files.into_iter().enumerate() {
-        out[upload[i]].push(f);
     }
+    out.push_str(ext);
+    (runs > 0).then_some(out)
+}
+
+/// A name's stem and its extension (a last ".xxx" of 1–5 letters and digits, "mp4"'s 4 no number sequence).
+fn split_extension(name: &str) -> (&str, &str) {
+    match name.rfind('.') {
+        Some(dot) if dot > 0 && (1..=5).contains(&(name.len() - dot - 1)) && name[dot + 1..].bytes().all(|c| c.is_ascii_alphanumeric()) => (&name[..dot], &name[dot..]),
+        _ => (name, ""),
+    }
+}
+
+/// A name's runs of digits and of other characters, in order.
+fn digit_runs(name: &str) -> Vec<(bool, &str)> {
+    let mut out = Vec::new();
+    let mut start = 0;
+    let b = name.as_bytes();
+    let mut i = 0;
+    while i < b.len() {
+        let digits = b[i].is_ascii_digit();
+        let mut j = i + 1;
+        while j < b.len() && b[j].is_ascii_digit() == digits {
+            j += 1;
+        }
+        out.push((digits, &name[start..j]));
+        start = j;
+        i = j;
+    }
+    out
+}
+
+/// An album's name: its members' shared name, with the numbers that vary among them as "#" and the ones they share
+/// kept ("1481439079025_#.jpg.data" for one timestamp's pictures).
+pub fn album_name(names: &[&str]) -> String {
+    let (stem, ext) = split_extension(names[0]);
+    let first = digit_runs(stem);
+    let mut varies = vec![false; first.len()];
+    for name in &names[1..] {
+        let runs = digit_runs(split_extension(name).0);
+        if runs.len() != first.len() {
+            return album_template(names[0]).unwrap_or_else(|| names[0].to_string());
+        }
+        for (i, (digits, text)) in runs.iter().enumerate() {
+            if *digits && first[i].1 != *text {
+                varies[i] = true;
+            }
+        }
+    }
+    let mut out = String::with_capacity(names[0].len());
+    for (i, (digits, text)) in first.iter().enumerate() {
+        if *digits && varies[i] {
+            out.push('#');
+        } else {
+            out.push_str(text);
+        }
+    }
+    out.push_str(ext);
     out
 }
 
 pub struct RangeIndexer {
     group: String,
     rules: Rules,
-    start: i64,
     end: i64,
     priorities: FxHashMap<String, u32>,
     idle: u64,
-    chunk_articles: i64,
     scanner: Scanner,
     aggregator: Aggregator,
+    frontier: Frontier,
+    /// Per release key its open uploads, oldest first.
+    open: FxHashMap<String, Vec<OpenRelease>>,
+    /// Per album key its open albums, oldest first.
+    albums: FxHashMap<String, Vec<OpenAlbum>>,
+    /// The next upload number per key (distinct within a batch: the store's slot is the key and the upload).
+    uploads: FxHashMap<String, usize>,
+    /// Per (poster, counter total ≥ BATCH_COUNTER_MIN_TOTAL) the releases started under it and the batch of the last.
+    batch_totals: FxHashMap<(String, i64), (u32, u64)>,
     batch: Batch,
     counts: FxHashMap<(&'static str, String), (u64, u64)>,
     count_order: Vec<(&'static str, String)>,
-    completed: FxHashMap<String, Vec<Vec<Seg>>>,
-    completed_order: Vec<String>,
     /// Posting times of this batch's segments (posted_span at take).
     dates: Vec<i64>,
+    pending_files: usize,
 }
 
 impl RangeIndexer {
-    /// `resume_at`: where this worker starts (a range taken over midway); its first batch starts there.
-    pub fn new(group: &str, start: i64, end: i64, resume_at: i64, rules: Rules, idle: u64) -> Self {
+    /// `resume_at`: where this worker starts (a range taken over midway); its first batch starts there. `idle`: batches
+    /// (chunks) after which an undated file or release is given up.
+    pub fn new(group: &str, _start: i64, end: i64, resume_at: i64, rules: Rules, idle: u64) -> Self {
         RangeIndexer {
             group: group.to_string(),
-            start,
             end,
             priorities: rules.priorities.clone(),
             rules,
             idle,
-            chunk_articles: 0,
             scanner: Scanner::default(),
             aggregator: Aggregator::default(),
+            frontier: Frontier::default(),
+            open: FxHashMap::default(),
+            albums: FxHashMap::default(),
+            uploads: FxHashMap::default(),
+            batch_totals: FxHashMap::default(),
             batch: Batch { from: resume_at, next: resume_at, ..Default::default() },
             counts: FxHashMap::default(),
             count_order: Vec::new(),
-            completed: FxHashMap::default(),
-            completed_order: Vec::new(),
             dates: Vec::new(),
+            pending_files: 0,
         }
     }
 
@@ -699,7 +900,10 @@ impl RangeIndexer {
             });
         }
         self.batch.headers += headers;
-        self.dates.extend(segments.iter().filter_map(|s| s.posted_at));
+        for t in segments.iter().filter_map(|s| s.posted_at) {
+            self.dates.push(t);
+            self.frontier.add(t);
+        }
         for owner in crossposts {
             self.count("crosspost", owner, 1, 0);
         }
@@ -714,31 +918,55 @@ impl RangeIndexer {
         let done = self.aggregator.add(segments);
         unsafe { PROF[3] += t.elapsed().as_nanos() as u64; }
         self.files(done);
-        if self.chunk_articles == 0 {
-            self.chunk_articles = next - self.batch.from;
-        }
+        self.frontier.update();
         let t = std::time::Instant::now();
-        let gone = self.aggregator.evict(self.idle);
-        self.cut_off(gone, false);
+        let gone = self.aggregator.evict(self.frontier.closed_before(), self.idle, false);
+        self.files(gone);
+        self.close(false);
         unsafe { PROF[4] += t.elapsed().as_nanos() as u64; }
         self.batch.next = next;
     }
 
+    /// The block's end: every file and release left is written, finished when the window closed on it, else
+    /// provisionally and as an edge (the block merger finishes it with the neighbour block's part).
     pub fn finish(&mut self) {
-        let gone = self.aggregator.evict(0);
-        self.cut_off(gone, true);
+        self.frontier.settle();
+        let gone = self.aggregator.evict(None, 0, true);
+        self.files(gone);
+        self.close(true);
         self.batch.next = self.end + 1;
     }
 
-    /// Files completed and fragments cut since the last take (RangeIndexer.pending).
+    /// The next range of a group starts after a gap (edge-bench --group, the corpus holding some ranges only): a fresh
+    /// batch starts there, not at the previous range's end; the window carries on. Nothing when the batch has anything.
+    pub fn skip_to(&mut self, article: i64) {
+        if self.batch.headers == 0 && self.batch.releases.is_empty() && self.batch.candidates.is_empty() && self.batch.from < article {
+            self.batch.from = article;
+            self.batch.next = article;
+        }
+    }
+
+    /// Files in the releases finished since the last take (RangeIndexer.pending).
     pub fn pending(&self) -> usize {
-        self.batch.fragments.len() + self.completed.values().map(|l| l.len()).sum::<usize>()
+        self.pending_files
+    }
+
+    /// Where the block is in posting time (None until FRONTIER_MIN dated articles were read).
+    pub fn frontier(&self) -> Option<i64> {
+        self.frontier.value()
+    }
+
+    /// Releases the window still holds (edge-bench's progress line).
+    pub fn open_releases(&self) -> usize {
+        self.open.values().map(|u| u.len()).sum::<usize>() + self.albums.values().map(|a| a.len()).sum::<usize>()
+    }
+
+    /// Files still short of parts, or complete and not yet in a release.
+    pub fn open_files(&self) -> usize {
+        self.aggregator.open.values().map(|f| f.len()).sum()
     }
 
     pub fn take(&mut self) -> Batch {
-        let t = std::time::Instant::now();
-        self.judge_releases();
-        unsafe { PROF[5] += t.elapsed().as_nanos() as u64; }
         self.scanner.files.clear();
         let next = self.batch.next;
         let mut out = std::mem::replace(&mut self.batch, Batch { from: next, next, ..Default::default() });
@@ -747,89 +975,231 @@ impl RangeIndexer {
             let (articles, files) = self.counts.remove(&id).unwrap();
             out.counts.push((id.0, id.1, articles, files));
         }
+        self.pending_files = 0;
+        // Upload numbers only have to be distinct within a batch: those of keys with nothing open go.
+        let (open, albums) = (&self.open, &self.albums);
+        self.uploads.retain(|key, _| open.contains_key(key) || albums.contains_key(key));
+        let batch = self.aggregator.batch;
+        self.batch_totals.retain(|_, (_, touched)| batch - *touched < BATCH_COUNTER_BATCHES);
         out
     }
 
+    /// Finished files (complete, or the window closed on them) into the open release of their key and upload.
     fn files(&mut self, files: Vec<Vec<Seg>>) {
+        let batch = self.aggregator.batch;
         for file in files {
+            if file.len() == 1 && (file[0].number != 1 || file[0].file.total_parts > 1) && is_random_file(&file[0].file.name) {
+                // One article of a multi-part random file (its upload per article: ngPost's, nyuu's): not a release.
+                let tool = names::upload_tool(&file[0].message_id);
+                self.count("per-article", tool.to_string(), 1, 0);
+                continue;
+            }
             let key = release_key_of(&file[0]);
-            match self.completed.get_mut(&key) {
-                Some(list) => list.push(file),
+            let first = posted_of(&file);
+            let last = file.iter().filter_map(|s| s.posted_at).max();
+            let uploads = self.open.entry(key.clone()).or_default();
+            let near = uploads.iter_mut().filter(|u| u.distance(first, last) <= UPLOAD_GAP_MS).min_by_key(|u| u.distance(first, last));
+            let release = match near {
+                Some(u) => u,
                 None => {
-                    self.completed_order.push(key.clone());
-                    self.completed.insert(key, vec![file]);
+                    let n = self.uploads.entry(key).or_insert(0);
+                    let upload = *n;
+                    *n += 1;
+                    if let Some(total) = file[0].file.release.file_total_truthy().filter(|&t| t >= BATCH_COUNTER_MIN_TOTAL) {
+                        let e = self.batch_totals.entry((file[0].file.poster.clone(), total)).or_insert((0, batch));
+                        e.0 += 1;
+                        e.1 = batch;
+                    }
+                    uploads.push(OpenRelease { upload, files: Vec::new(), first: None, last: None, touched: batch });
+                    uploads.last_mut().unwrap()
+                }
+            };
+            if let Some(f) = first {
+                if release.first.is_none_or(|r| f < r) {
+                    release.first = Some(f);
                 }
             }
+            if let Some(l) = last {
+                if release.last.is_none_or(|r| l > r) {
+                    release.last = Some(l);
+                }
+            }
+            release.touched = batch;
+            release.files.push(file);
         }
     }
 
-    fn judge_releases(&mut self) {
-        let order = std::mem::take(&mut self.completed_order);
-        let mut by_release = std::mem::take(&mut self.completed);
-        for key in order {
-            for (upload, list) in uploads_of(by_release.remove(&key).unwrap()).into_iter().enumerate() {
+    /// Releases the window closed on (their last part before the frontier less WINDOW_MS; undated ones idle for
+    /// `idle` batches): judged and written. At the end (`all`), the rest too: provisionally, as edges.
+    fn close(&mut self, all: bool) {
+        let before = self.frontier.closed_before();
+        let start = self.frontier.start();
+        let batch = self.aggregator.batch;
+        let idle = self.idle;
+        let mut done: Vec<(String, OpenRelease, bool)> = Vec::new();
+        let keys: Vec<String> = self.open.keys().cloned().collect();
+        for key in keys {
+            let uploads = self.open.get_mut(&key).unwrap();
+            let mut i = 0;
+            while i < uploads.len() {
+                let u = &uploads[i];
+                let closed = match u.last {
+                    Some(l) => before.is_some_and(|b| l < b),
+                    None => idle > 0 && batch - u.touched >= idle,
+                };
+                if !all && !closed {
+                    i += 1;
+                    continue;
+                }
+                done.push((key.clone(), uploads.remove(i), closed));
+            }
+            if uploads.is_empty() {
+                self.open.remove(&key);
+            }
+        }
+        // An edge: still open at the block's end, or posted within the window of the block's start (its other
+        // part may be in the neighbour block); undated ones can't be placed, so they are edges too.
+        let near_start = |first: Option<i64>| match (start, first) {
+            (Some(s), Some(f)) => f < s + WINDOW_MS,
+            _ => true,
+        };
+        // In posting order, so an album's members join it from its first file on (the keys' order is a hash's).
+        done.sort_by(|a, b| a.1.first.cmp(&b.1.first).then_with(|| a.0.cmp(&b.0)));
+        for (key, u, closed) in done {
+            // A single the window closed on waits in its album (a lone one is written as it is when that closes).
+            if closed {
+                if let Some(template) = album_candidate(&u) {
+                    self.album(key, u, template);
+                    continue;
+                }
+            }
+            let edge = !closed || near_start(u.first);
+            self.emit(&key, u, closed, edge);
+        }
+        // Albums: closed once every single that could join (posted within WINDOW_MS of the first) has closed too.
+        let mut done_albums: Vec<(String, OpenAlbum, bool)> = Vec::new();
+        let keys: Vec<String> = self.albums.keys().cloned().collect();
+        for key in keys {
+            let list = self.albums.get_mut(&key).unwrap();
+            let mut i = 0;
+            while i < list.len() {
+                let a = &list[i];
+                let closed = match a.first {
+                    Some(f) => before.is_some_and(|b| f + WINDOW_MS < b),
+                    None => idle > 0 && batch - a.touched >= idle,
+                };
+                if !all && !closed {
+                    i += 1;
+                    continue;
+                }
+                done_albums.push((key.clone(), list.remove(i), closed));
+            }
+            if list.is_empty() {
+                self.albums.remove(&key);
+            }
+        }
+        for (key, mut a, closed) in done_albums {
+            let edge = !closed || near_start(a.first);
+            if a.members.len() < 2 {
+                let (release_key, u) = a.members.remove(0);
+                self.emit(&release_key, u, closed, edge);
+                continue;
+            }
+            let names: Vec<&str> = a.members.iter().map(|(_, u)| u.files[0][0].file.name.as_str()).collect();
+            let name = album_name(&names);
+            let files: Vec<Vec<Seg>> = a.members.into_iter().flat_map(|(_, u)| u.files).collect();
+            self.write(&key, a.upload, files, closed, edge, Some(name));
+        }
+    }
+
+    /// A closed single into the open album of its poster and template posted within WINDOW_MS of its first file
+    /// (with room), else a new one.
+    fn album(&mut self, release_key: String, u: OpenRelease, template: String) {
+        let batch = self.aggregator.batch;
+        let poster = &u.files[0][0].file.poster;
+        let poster = if names::random_poster(poster) { "" } else { poster.as_str() };
+        let key = sha1_hex(&format!("album\n{poster}\n{template}"));
+        let albums = self.albums.entry(key.clone()).or_default();
+        let fits = |a: &OpenAlbum| {
+            a.members.len() < ALBUM_MAX_FILES
+                && match (a.first, u.first) {
+                    (Some(af), Some(f)) => f >= af && f < af + WINDOW_MS,
+                    _ => true,
+                }
+        };
+        let index = match albums.iter().position(fits) {
+            Some(i) => i,
+            None => {
+                let n = self.uploads.entry(key).or_insert(0);
+                let upload = *n;
+                *n += 1;
+                albums.push(OpenAlbum { upload, members: Vec::new(), first: None, last: None, touched: batch });
+                albums.len() - 1
+            }
+        };
+        let a = &mut albums[index];
+        if let Some(f) = u.first {
+            if a.first.is_none_or(|r| f < r) {
+                a.first = Some(f);
+            }
+        }
+        if let Some(l) = u.last {
+            if a.last.is_none_or(|r| l > r) {
+                a.last = Some(l);
+            }
+        }
+        a.touched = batch;
+        a.members.push((release_key, u));
+    }
+
+    fn emit(&mut self, key: &str, release: OpenRelease, finished: bool, edge: bool) {
+        self.write(key, release.upload, release.files, finished, edge, None);
+    }
+
+    /// A release (or an album, under its name) judged and written to the batch.
+    fn write(&mut self, key: &str, upload: usize, list: Vec<Vec<Seg>>, finished: bool, edge: bool, album: Option<String>) {
         let judged: Vec<JudgedFile> = list
             .iter()
             .map(|f| JudgedFile { name: &f[0].file.name, total_parts: f[0].file.total_parts, groups: &f[0].file.groups, first_message_id: &f[0].message_id, poster: &f[0].file.poster })
             .collect();
-                match names::judge(&key, &judged, &self.rules) {
+        match names::judge(key, &judged, &self.rules) {
             Verdict::Keep => {
-                        let candidates = candidates_of(&key, &list, upload);
-                        self.batch.releases.push(chunk_of(&key, &list, upload));
+                let candidates = candidates_of(key, &list, upload);
+                let mut chunk = chunk_of(key, &list, upload);
+                chunk.finished = finished;
+                chunk.edge = edge;
+                let album_none = album.is_none();
+                if let Some(name) = album {
+                    chunk.name = name;
+                    chunk.source = "album";
+                    chunk.file_total = None;
+                }
+                if !chunk.readable {
+                    // Unnamed: no chunk (its segments are read again once a probe names it), the tool for its signature.
+                    chunk.tool = Some(names::upload_tool(&list[0][0].message_id));
+                    chunk.payload = encode_chunk(&[]);
+                }
+                if chunk.readable {
+                    self.pending_files += chunk.files;
+                }
+                if let Some(total) = chunk.file_total.filter(|&t| album_none && t >= BATCH_COUNTER_MIN_TOTAL && (chunk.files as i64) < t) {
+                    // A batch counter: no file total (the release is as complete as its files are).
+                    if self.batch_totals.get(&(chunk.poster.clone(), total)).is_some_and(|(n, _)| *n >= BATCH_COUNTER_RELEASES) {
+                        chunk.file_total = None;
+                    }
+                }
+                self.batch.releases.push(chunk);
                 self.batch.candidates.extend(candidates);
             }
             Verdict::Drop { rule, key: rule_key, sample } => {
                 let segments: usize = list.iter().map(|f| f.len()).sum();
                 self.count(rule, rule_key.clone(), segments as u64, list.len() as u64);
                 if sample {
-                            self.batch.samples.push((chunk_of(&key, &list, upload), rule, rule_key));
+                    self.batch.samples.push((chunk_of(key, &list, upload), rule, rule_key));
                 }
             }
         }
     }
-        }
-    }
-
-    fn cut_off(&mut self, files: Vec<Vec<Seg>>, at_end: bool) {
-        let window = self.idle as i64 * self.chunk_articles.max(1);
-        for list in files {
-            if list.len() == 1 && is_random_file(&list[0].file.name) {
-                let tool = names::upload_tool(&list[0].message_id);
-                self.count("per-article", tool.to_string(), 1, 0);
-                continue;
-            }
-            let min_part = list.iter().map(|s| s.number).min().unwrap();
-            let max_part = list.iter().map(|s| s.number).max().unwrap();
-            let first_article = list.iter().map(|s| s.article).min().unwrap();
-            let cut_at_start = min_part > 1 && first_article < self.start + window;
-            let cut_at_end = at_end && max_part < list[0].file.total_parts;
-            if !cut_at_start && !cut_at_end {
-                self.count("incomplete", String::new(), list.len() as u64, 1);
-                continue;
-            }
-            let f = &list[0].file;
-            let mut fragment = Fragment {
-                file_key: f.file_key.clone(),
-                name: f.name.clone(),
-                poster: f.poster.clone(),
-                subject: f.subject.clone(),
-                total_parts: f.total_parts,
-                groups: f.groups.clone(),
-                release: f.release.clone(),
-                posted_at: None,
-                segments: Vec::with_capacity(list.len()),
-            };
-            for s in &list {
-                fragment.segments.push((s.number, s.message_id.clone(), s.bytes, s.article));
-                if let Some(t) = s.posted_at {
-                    if fragment.posted_at.is_none_or(|p| t < p) {
-                        fragment.posted_at = Some(t);
-                    }
-                }
-            }
-            self.batch.fragments.push(fragment);
-        }
-    }
 
     fn count(&mut self, rule: &'static str, key: String, articles: u64, files: u64) {
         match self.counts.get_mut(&(rule, key.clone())) {
@@ -845,6 +1215,19 @@ impl RangeIndexer {
     }
 }
 
+/// A closed release that may be an album's member: one file, named by its stem or its post (no release name in the
+/// subject), with a number in its name (album_template).
+fn album_candidate(u: &OpenRelease) -> Option<String> {
+    if u.files.len() != 1 {
+        return None;
+    }
+    let meta = &u.files[0][0].file;
+    if !matches!(meta.release.source, "stem" | "post") {
+        return None;
+    }
+    album_template(&meta.name)
+}
+
 fn release_key_of(s: &Seg) -> String {
     // A named file always has a release (releaseIdentity isn't null unless weak).
     s.file.release.key.clone()
@@ -904,6 +1287,10 @@ fn chunk_of(key: &str, files: &[Vec<Seg>], upload: usize) -> ReleaseChunk {
         last_article,
         readable: chunk_files.iter().any(|f| !is_random_file(f.name)),
         has_par2: chunk_files.iter().any(|f| names::is_par2(f.name)),
+        finished: true,
+        edge: false,
+        partial: files.iter().any(|f| (f.len() as u32) < f[0].file.total_parts),
+        tool: None,
         payload: encode_chunk(&chunk_files),
         subjects: chunk_files.iter().take(SUBJECTS).map(|f| f.subject.to_string()).collect(),
     }
@@ -968,6 +1355,165 @@ mod tests {
         assert_eq!(got, vec![(0, 2), (1, 1)]);
         assert_eq!(batch.releases[0].posted_last.unwrap() - batch.releases[0].posted_at.unwrap(), 245 * 60_000);
         assert_eq!(batch.releases[0].key, batch.releases[1].key);
+        // A day of posting in all: the window never closed on them, so at the end they are provisional edges.
+        assert!(batch.releases.iter().all(|r| !r.finished && r.edge && !r.partial));
+    }
+
+    /// `n` single-part filler files posted `minutes` apart from `start`: dated articles that move the frontier.
+    fn fillers(from: i64, n: i64, start: i64, minutes: i64) -> Vec<String> {
+        (0..n).map(|i| line(from + i, &format!(r#"Filler.{} [1/1] - "filler.{}.jpg" yEnc (1/1)"#, from + i, from + i), start + i * minutes)).collect()
+    }
+
+    #[test]
+    fn the_window_finishes_releases_and_keeps_incomplete_files() {
+        let mut priorities = FxHashMap::default();
+        priorities.insert(GROUP.to_string(), 1);
+        let mut ix = RangeIndexer::new(GROUP, 1, 1_000_000, 1, Rules::defaults(priorities), 5);
+        let day = 24 * 60;
+        // Chunk 1: a release at the block's start (two of three parts of its file), and 100 dated articles after it.
+        let mut lines = vec![
+            line(10, r#"Early.Show [1/1] - "Early.Show.mkv" yEnc (1/3)"#, 0),
+            line(11, r#"Early.Show [1/1] - "Early.Show.mkv" yEnc (2/3)"#, 1),
+        ];
+        lines.extend(fillers(100, 100, 2, 1));
+        ix.add_text(&lines.join("
+"), 1000);
+        assert!(ix.frontier().is_some());
+        let b1 = ix.take();
+        assert!(b1.releases.iter().all(|r| r.name.starts_with("Filler")));
+        // Chunk 2, three days on (more articles than before, so the median moves there): the window closed on the
+        // early show (its file still short a part) and on the first fillers.
+        let mut lines = vec![line(2000, r#"Later.Show [1/1] - "Later.Show.mkv" yEnc (1/1)"#, 3 * day)];
+        lines.extend(fillers(2100, 300, 3 * day + 1, 1));
+        ix.add_text(&lines.join("
+"), 3000);
+        let b2 = ix.take();
+        let early = b2.releases.iter().find(|r| r.name == "Early Show").expect("the early show is written");
+        assert!(early.finished && early.edge && early.partial, "finished {} edge {} partial {}", early.finished, early.edge, early.partial);
+        assert_eq!((early.files, early.segments), (1, 2));
+        assert!(!b2.releases.iter().any(|r| r.name == "Later Show"));
+        // Chunk 3, three more days: the later show is finished, and not an edge (a day past the block's start).
+        ix.add_text(&fillers(4000, 500, 6 * day, 1).join("
+"), 5000);
+        let b3 = ix.take();
+        let later = b3.releases.iter().find(|r| r.name == "Later Show").expect("the later show is written");
+        assert!(later.finished && !later.edge && !later.partial);
+        // The end: the last fillers are still inside the window, so they are provisional edges.
+        ix.finish();
+        let b4 = ix.take();
+        assert_eq!(b4.releases.len(), 500);
+        assert!(b4.releases.iter().all(|r| !r.finished && r.edge));
+        assert_eq!(b4.next, 1_000_001);
+    }
+
+    #[test]
+    fn singles_of_one_template_become_an_album() {
+        let mut priorities = FxHashMap::default();
+        priorities.insert(GROUP.to_string(), 1);
+        let mut ix = RangeIndexer::new(GROUP, 1, 1_000_000, 1, Rules::defaults(priorities), 5);
+        let day = 24 * 60;
+        // A dump's pictures of one timestamp (a stem each: ".data" is no image), two scans, a lone document, and dated fillers.
+        let mut lines: Vec<String> = (1..=14).map(|i| line(1000 + i, &format!(r#"<IMGBANK> [1/1] "1481439079025_{i:03}.jpg.data" yEnc (1/1)"#), i)).collect();
+        lines.push(line(2001, r#"[1/1] "Scan_001.pdf" yEnc (1/1)"#, 20));
+        lines.push(line(2002, r#"[1/1] "Scan_002.pdf" yEnc (1/1)"#, 21));
+        lines.push(line(2003, r#"[1/1] "Lonely_007.pdf" yEnc (1/1)"#, 22));
+        lines.extend(fillers(3000, 100, 23, 1));
+        ix.add_text(&lines.join("\n"), 4000);
+        let b1 = ix.take();
+        assert!(b1.releases.iter().all(|r| r.name.starts_with("Filler")));
+        // Three days on: the window closed on the singles, and on their albums.
+        ix.add_text(&fillers(5000, 300, 3 * day, 1).join("\n"), 6000);
+        let b2 = ix.take();
+        let mut albums: Vec<(&str, &str, usize, bool)> = b2.releases.iter().filter(|r| !r.name.starts_with("Filler")).map(|r| (r.name.as_str(), r.source, r.files, r.finished)).collect();
+        albums.sort();
+        assert_eq!(albums, vec![("1481439079025_#.jpg.data", "album", 14, true), ("Lonely_007", "stem", 1, true), ("Scan_#.pdf", "album", 2, true)]);
+        let album = b2.releases.iter().find(|r| r.source == "album" && r.files == 14).unwrap();
+        // Posted within the window of the block's start: an edge, like any release there.
+        assert!(album.file_total.is_none() && album.edge && !album.partial && album.readable);
+        assert_eq!(album.segments, 14);
+        assert_eq!(b2.candidates.len(), 0);
+        assert!(ix.albums.is_empty());
+    }
+
+    #[test]
+    fn an_unnamed_release_keeps_no_chunk_but_its_tool_and_probe_target() {
+        let mut priorities = FxHashMap::default();
+        priorities.insert(GROUP.to_string(), 1);
+        let mut ix = RangeIndexer::new(GROUP, 1, 10_000, 1, Rules::defaults(priorities), 5);
+        let lines: Vec<String> = (0..4).map(|i| line(100 + i, &format!(r#"[1/1] - "q3B9OrpKiCxUnzgtUGnp.part01.rar" yEnc ({}/4)"#, i + 1), i)).collect();
+        ix.add_text(&lines.join("\n"), 10_001);
+        ix.finish();
+        let batch = ix.take();
+        assert_eq!(batch.releases.len(), 1);
+        let r = &batch.releases[0];
+        assert!(!r.readable && r.tool == Some("other") && r.files == 1 && r.segments == 4);
+        assert_eq!(r.payload, encode_chunk(&[]));
+        assert_eq!(batch.candidates.iter().map(|c| c.kind).collect::<Vec<_>>(), vec!["archive"]);
+        assert_eq!(ix.pending(), 0);
+    }
+
+    #[test]
+    fn album_templates_and_names() {
+        assert_eq!(album_template("1481439079025_014.jpg.data").as_deref(), Some("#_#.jpg.data"));
+        assert_eq!(album_template("ActiCalc 1.06.rar"), None);
+        assert_eq!(album_template("Scan_001.vol00+01.par2"), None);
+        assert_eq!(album_template("Scan_001.nfo"), None);
+        assert_eq!(album_template("NoNumber.mp4"), None);
+        assert_eq!(album_template("acs5152s1.11"), None);
+        assert_eq!(album_template("Broforce.vol00+01.par2.temp"), None);
+        assert_eq!(album_template("Starship For Rent 3 (2024).epub").as_deref(), Some("Starship For Rent # (#).epub"));
+        assert_eq!(album_template("c3453cdd2a3645a8f3cf3d22319abe63"), None);
+        assert_eq!(album_name(&["1481439079025_001.jpg.data", "1481439079025_014.jpg.data"]), "1481439079025_#.jpg.data");
+        assert_eq!(album_name(&["1481439079025_001.jpg.data", "1481439080000_001.jpg.data"]), "#_001.jpg.data");
+        assert_eq!(album_name(&["Scan_001.pdf", "Scan_002.pdf", "Scan_003.pdf"]), "Scan_#.pdf");
+    }
+
+    #[test]
+    fn a_batch_counter_is_no_file_total() {
+        let mut priorities = FxHashMap::default();
+        priorities.insert(GROUP.to_string(), 1);
+        let mut ix = RangeIndexer::new(GROUP, 1, 1_000_000, 1, Rules::defaults(priorities), 5);
+        let day = 24 * 60;
+        // 25 different programs of one poster under one running counter, and one upload with a counter of its own.
+        let mut lines: Vec<String> = (0..25).map(|i| line(100 + i, &format!(r#"[{}/68067] - "Program.Number.{i}.v1.0.rar" yEnc (1/1)"#, 1000 + i), i)).collect();
+        lines.push(line(200, r#"[1/120] - "Real.Upload.part001.rar" yEnc (1/1)"#, 30));
+        lines.extend(fillers(300, 100, 31, 1));
+        ix.add_text(&lines.join("\n"), 1000);
+        ix.add_text(&fillers(2000, 300, 3 * day, 1).join("\n"), 3000);
+        let batch = ix.take();
+        let programs: Vec<&ReleaseChunk> = batch.releases.iter().filter(|r| r.name.starts_with("Program")).collect();
+        assert_eq!(programs.len(), 25);
+        assert!(programs.iter().all(|r| r.file_total.is_none() && r.files == 1));
+        let real = batch.releases.iter().find(|r| r.name.starts_with("Real")).unwrap();
+        assert_eq!(real.file_total, Some(120));
+    }
+
+    #[test]
+    fn the_frontier_ignores_implausible_dates_and_follows_a_pause() {
+        let hour = 3_600_000i64;
+        let mut f = Frontier::default();
+        for i in 0..200 {
+            f.add(i * hour / 10);
+        }
+        f.update();
+        let settled = f.value().unwrap();
+        assert!((settled - 10 * hour).abs() <= hour);
+        assert_eq!(f.start(), Some(settled));
+        // A bogus year, and one a year back: set aside.
+        for _ in 0..100 {
+            f.add(4_000_000_000_000);
+            f.add(-400 * 24 * hour);
+        }
+        f.update();
+        assert_eq!(f.value(), Some(settled));
+        // The group paused for a month: FRONTIER_ARTICLES agreeing dates move the frontier there.
+        let later = 30 * 24 * hour;
+        for i in 0..FRONTIER_ARTICLES as i64 {
+            f.add(later + i * 1000);
+        }
+        f.update();
+        assert!(f.value().unwrap() >= later);
+        assert_eq!(f.start(), Some(settled));
     }
 
     #[test]

edge/src/lease.rs

@@ -0,0 +1,65 @@
+//! A lease as the coordinator sends it (lib/edge/lease.ts): the range and the rules, JSON with the TypeScript
+//! names. The edge-node worker takes one per range; edge-bench's replay takes the rules alone (`--rules`).
+
+use crate::names::Rules;
+use serde::Deserialize;
+use std::collections::HashMap;
+
+#[derive(Deserialize)]
+pub struct LeaseJson {
+    pub range: RangeJson,
+    pub rules: RulesJson,
+}
+
+#[derive(Deserialize)]
+pub struct RangeJson {
+    pub id: i64,
+    pub group: String,
+    pub start: i64,
+    pub end: i64,
+    pub next: i64,
+}
+
+#[derive(Deserialize)]
+#[serde(rename_all = "camelCase")]
+pub struct RulesJson {
+    pub version: i64,
+    pub priorities: HashMap<String, u32>,
+    #[serde(default)]
+    pub dropped_signatures: Vec<String>,
+    #[serde(default)]
+    pub dropped_tools: Option<Vec<String>>,
+    #[serde(default)]
+    pub per_article_max_parts: Option<u32>,
+    #[serde(default)]
+    pub sample_percent: Option<f64>,
+    #[serde(default)]
+    pub drop_executables: Option<bool>,
+    #[serde(default)]
+    pub dropped_posters: Option<Vec<String>>,
+}
+
+impl RulesJson {
+    /// The rules with today's defaults under them (a field the stored set doesn't have keeps the default).
+    pub fn rules(self) -> Rules {
+        let mut rules = Rules::defaults(self.priorities.into_iter().collect());
+        rules.version = self.version;
+        rules.dropped_signatures = self.dropped_signatures.into_iter().collect();
+        if let Some(tools) = self.dropped_tools {
+            rules.dropped_tools = tools.into_iter().collect();
+        }
+        if let Some(n) = self.per_article_max_parts {
+            rules.per_article_max_parts = n;
+        }
+        if let Some(p) = self.sample_percent {
+            rules.sample_percent = p;
+        }
+        if let Some(e) = self.drop_executables {
+            rules.drop_executables = e;
+        }
+        if let Some(posters) = self.dropped_posters {
+            rules.dropped_posters = posters;
+        }
+        rules
+    }
+}

edge/src/lib.rs

@@ -5,5 +5,6 @@
 pub mod codec;
 pub mod indexer;
 pub mod js;
+pub mod lease;
 pub mod names;
 pub mod wire;

edge/src/main.rs

@@ -1,10 +1,11 @@
-//! The edge worker's indexing in Rust, on a range captured by `edge:bench
-//! --capture` (REBUILD.md, part 2: TypeScript or Rust). Same work as
-//! edge:bench's replay: inflate each chunk, scan it, aggregate files, cut off
-//! idle ones, and every N chunks judge and encode the releases. `--dump`
-//! writes every output as JSON lines for edge/check.ts.
+//! The Rust indexer over stored chunks (REBUILD.md, part 2 and "Voyager 3"): a range captured by `edge:bench
+//! --capture` or stored in the corpus, or a group's corpus ranges in order through one window (`--group`, the
+//! replay harness behind `replay:corpus`). Same work as the edge-node worker: inflate each chunk, scan it,
+//! aggregate files, finish releases through the window, and every N chunks judge and encode them. `--dump`
+//! writes every output as JSON lines; `--wire` the batches as the worker posts them (`-`: stdout).
 //!
-//!   edge-bench <dir> [--batch-chunks 10] [--idle 5] [--dump out.jsonl] [--wire batches.jsonl]
+//!   edge-bench <range dir> | --group <group dir> [--from article] [--to article] [--limit ranges] [--rules rules.json]
+//!              [--batch-chunks 10] [--idle 5] [--no-finish] [--dump out.jsonl] [--wire batches.jsonl|-] [--releases out.jsonl]
 
 use base64::Engine;
 use edge::indexer::{self, Batch, RangeIndexer};
@@ -123,7 +124,7 @@ fn dump_batch(batch: &Batch, out: &mut impl Write) {
         line.push_str(",\"postedLast\":");
         opt_num(r.posted_last, line);
         line.push_str(&format!(
-            ",\"files\":{},\"segments\":{},\"bytes\":{},\"firstArticle\":{},\"lastArticle\":{},\"readable\":{},\"hasPar2\":{},\"payload\":\"{}\"",
+            ",\"files\":{},\"segments\":{},\"bytes\":{},\"firstArticle\":{},\"lastArticle\":{},\"readable\":{},\"hasPar2\":{},\"finished\":{},\"edge\":{},\"partial\":{},\"payload\":\"{}\"",
             r.files,
             r.segments,
             r.bytes,
@@ -131,6 +132,9 @@ fn dump_batch(batch: &Batch, out: &mut impl Write) {
             r.last_article,
             r.readable,
             r.has_par2,
+            r.finished,
+            r.edge,
+            r.partial,
             b64.encode(&r.payload)
         ));
     };
@@ -162,76 +166,162 @@ fn dump_batch(batch: &Batch, out: &mut impl Write) {
         line.push_str(&format!(",\"articles\":{articles},\"files\":{files}}}"));
         emit(&mut line);
     }
-    for f in &batch.fragments {
-        line.push_str("{\"t\":\"f\",\"fileKey\":");
-        json_str(&f.file_key, &mut line);
-        line.push_str(",\"name\":");
-        json_str(&f.name, &mut line);
-        line.push_str(&format!(",\"totalParts\":{},\"postedAt\":", f.total_parts));
-        opt_num(f.posted_at, &mut line);
-        line.push_str(",\"segments\":[");
-        for (i, (n, id, bytes, article)) in f.segments.iter().enumerate() {
-            if i > 0 {
-                line.push(',');
+}
+
+/// A stored range: its directory and article span (range.json).
+struct Range {
+    dir: String,
+    start: i64,
+    end: i64,
+}
+
+/// The range in a directory with a range.json, and its group.
+fn range_in(dir: &str) -> Option<(String, Range)> {
+    let json = std::fs::read_to_string(format!("{dir}/range.json")).ok()?;
+    Some((json_field(&json, "group"), Range { dir: dir.to_string(), start: json_field(&json, "start").parse().ok()?, end: json_field(&json, "end").parse().ok()? }))
+}
+
+/// A group directory's complete ranges in article order (the corpus layout: `<group>/<start>-<end>/`).
+fn ranges_in(group_dir: &str) -> (String, Vec<Range>) {
+    let mut group = String::new();
+    let mut ranges: Vec<Range> = Vec::new();
+    for entry in std::fs::read_dir(group_dir).unwrap_or_else(|e| panic!("{group_dir}: {e}")) {
+        let path = entry.unwrap().path();
+        if !path.is_dir() {
+            continue;
         }
-            line.push_str(&format!("[{n},"));
-            json_str(id, &mut line);
-            line.push_str(&format!(",{bytes},{article}]"));
+        if let Some((g, range)) = range_in(path.to_str().unwrap()) {
+            group = g;
+            ranges.push(range);
         }
-        line.push_str("]}");
-        emit(&mut line);
+    }
+    ranges.sort_by_key(|r| r.start);
+    (group, ranges)
+}
+
+fn date(ms: Option<i64>) -> String {
+    match ms.and_then(chrono::DateTime::from_timestamp_millis) {
+        Some(t) => t.format("%Y-%m-%d %H:%M").to_string(),
+        None => "-".to_string(),
     }
 }
 
+/// Files waiting in finished releases after which a batch goes out whatever its chunks (edge-node's limit).
+const MAX_PENDING_FILES: usize = 100_000;
+
 fn main() {
     let args: Vec<String> = std::env::args().collect();
-    let dir = args.get(1).expect("usage: edge-bench <dir> [--batch-chunks N] [--idle N] [--dump file]").clone();
+    let usage = "usage: edge-bench <range dir> | --group <group dir> [--from article] [--to article] [--limit ranges] [--rules rules.json] [--batch-chunks N] [--idle N] [--no-finish] [--dump file] [--wire file|-] [--releases file]";
     let flag = |name: &str| args.iter().position(|a| a == name).and_then(|i| args.get(i + 1)).cloned();
+    let has = |name: &str| args.iter().any(|a| a == name);
     let batch_chunks: usize = flag("--batch-chunks").map(|v| v.parse().unwrap()).unwrap_or(10);
     let idle: u64 = flag("--idle").map(|v| v.parse().unwrap()).unwrap_or(5);
     let mut dump = flag("--dump").map(|p| std::io::BufWriter::new(std::fs::File::create(p).unwrap()));
-    // Each batch as the worker posts it (wire.rs), one JSON per line: check.ts compares it with toWire.
-    let mut wire_out = flag("--wire").map(|p| std::io::BufWriter::new(std::fs::File::create(p).unwrap()));
+    // Each batch as the worker posts it (wire.rs), one JSON per line: a file, or stdout (`-`) for replay:corpus.
+    let wire_path = flag("--wire");
+    let wire_stdout = wire_path.as_deref() == Some("-");
+    let mut wire_out: Option<Box<dyn Write>> = match wire_path.as_deref() {
+        Some("-") => Some(Box::new(std::io::BufWriter::with_capacity(1 << 20, std::io::stdout().lock()))),
+        Some(p) => Some(Box::new(std::io::BufWriter::new(std::fs::File::create(p).unwrap()))),
+        None => None,
+    };
     // One line per release chunk without its payload, with its first subjects (deploy/corpus_sweep.py).
     let mut releases_out = flag("--releases").map(|p| std::io::BufWriter::new(std::fs::File::create(p).unwrap()));
 
-    let range = std::fs::read_to_string(format!("{dir}/range.json")).unwrap();
-    let group = json_field(&range, "group");
-    let start: i64 = json_field(&range, "start").parse().unwrap();
-    let end: i64 = json_field(&range, "end").parse().unwrap();
-    let mut files: Vec<String> = std::fs::read_dir(&dir).unwrap().map(|e| e.unwrap().file_name().into_string().unwrap()).filter(|f| f.ends_with(".gz")).collect();
-    files.sort();
-
+    // One range, or a group's ranges in order through one window (the window carries over from range to range;
+    // the end is only finished after the last one, unless --no-finish: a replay to be continued).
+    let (group, ranges) = match flag("--group") {
+        Some(dir) => {
+            let (group, mut ranges) = ranges_in(&dir);
+            // --from: ranges ending before it are skipped (a replay continued); --to: ranges starting after it (one
+            // window of the group's history, the corpus holding others); --limit: this many at most.
+            if let Some(from) = flag("--from").map(|v| v.parse::<i64>().unwrap()) {
+                ranges.retain(|r| r.end >= from);
+            }
+            if let Some(to) = flag("--to").map(|v| v.parse::<i64>().unwrap()) {
+                ranges.retain(|r| r.start <= to);
+            }
+            if let Some(limit) = flag("--limit").map(|v| v.parse::<usize>().unwrap()) {
+                ranges.truncate(limit);
+            }
+            (group, ranges)
+        }
+        None => {
+            let positional = (1..args.len()).find(|&i| !args[i].starts_with("--") && !args[i - 1].starts_with("--")).map(|i| args[i].clone());
+            let dir = positional.expect(usage);
+            let (group, range) = range_in(&dir).unwrap_or_else(|| panic!("{dir}: no range.json"));
+            (group, vec![range])
+        }
+    };
+    if ranges.is_empty() {
+        eprintln!("no complete range found");
+        std::process::exit(2);
+    }
+    let finish = !has("--no-finish");
+    let rules = match flag("--rules") {
+        Some(path) => {
+            let json = std::fs::read_to_string(&path).unwrap_or_else(|e| panic!("{path}: {e}"));
+            serde_json::from_str::<edge::lease::RulesJson>(&json).unwrap_or_else(|e| panic!("{path}: {e}")).rules()
+        }
+        None => {
             let mut priorities = FxHashMap::default();
             priorities.insert(group.clone(), 1);
-    let mut indexer = RangeIndexer::new(&group, start, end, start, names::Rules::defaults(priorities), idle);
+            names::Rules::defaults(priorities)
+        }
+    };
+    // The report goes to stderr when the wire takes stdout.
+    let say = |line: String| {
+        if wire_stdout {
+            eprintln!("{line}");
+        } else {
+            println!("{line}");
+        }
+    };
+
+    let (start, end) = (ranges[0].start, ranges[ranges.len() - 1].end);
+    let mut indexer = RangeIndexer::new(&group, start, end, start, rules, idle);
 
     let (mut inflate_s, mut index_s) = (0.0, 0.0);
-    let (mut compressed, mut text_bytes) = (0usize, 0usize);
-    let mut out = (0u64, 0usize, 0usize, 0usize, 0usize, 0usize, 0usize, 0usize); // headers, releases, files, segments, payload, candidates, fragments, samples
+    let (mut compressed, mut text_bytes, mut chunks) = (0usize, 0usize, 0usize);
+    let mut out = (0u64, 0usize, 0usize, 0usize, 0usize, 0usize, 0usize); // headers, releases, files, segments, payload, candidates, samples
+    let mut finished = 0usize;
     let mut keys = std::collections::HashSet::new();
     let mut decompressor = libdeflater::Decompressor::new();
     let wall = Instant::now();
+    let mut since_batch = 0usize;
+    for (ri, range) in ranges.iter().enumerate() {
+        let range_wall = Instant::now();
+        let headers_before = out.0;
+        // After a gap between stored ranges the batch starts at this range (every range ends with a batch taken).
+        if ri > 0 {
+            indexer.skip_to(range.start);
+        }
+        let mut files: Vec<String> = std::fs::read_dir(&range.dir).unwrap().map(|e| e.unwrap().file_name().into_string().unwrap()).filter(|f| f.ends_with(".gz")).collect();
+        files.sort();
         for (i, file) in files.iter().enumerate() {
-        let raw = std::fs::read(format!("{dir}/{file}")).unwrap();
+            let raw = std::fs::read(format!("{}/{file}", range.dir)).unwrap();
             compressed += raw.len();
             let t = Instant::now();
             let data = inflate(&raw, &mut decompressor);
             inflate_s += t.elapsed().as_secs_f64();
             text_bytes += data.len();
+            chunks += 1;
             let t = Instant::now();
             let text = String::from_utf8_lossy(&data);
             let next: i64 = file.split('-').nth(2).unwrap().trim_end_matches(".gz").parse::<i64>().unwrap() + 1;
             indexer.add_text(&text, next);
-        let last = i + 1 == files.len();
-        if last {
+            since_batch += 1;
+            let range_end = i + 1 == files.len();
+            let last = range_end && ri + 1 == ranges.len();
+            if last && finish {
                 indexer.finish();
             }
-        if !last && (i + 1) % batch_chunks != 0 {
+            if !range_end && since_batch < batch_chunks && indexer.pending() < MAX_PENDING_FILES {
                 index_s += t.elapsed().as_secs_f64();
                 continue;
             }
             let batch = indexer.take();
+            since_batch = 0;
             index_s += t.elapsed().as_secs_f64();
             out.0 += batch.headers;
             out.1 += batch.releases.len();
@@ -240,44 +330,56 @@ fn main() {
                 out.2 += r.files;
                 out.3 += r.segments;
                 out.4 += r.payload.len();
+                finished += r.finished as usize;
             }
             out.5 += batch.candidates.len();
-        out.6 += batch.fragments.len();
-        out.7 += batch.samples.len();
+            out.6 += batch.samples.len();
             if let Some(d) = dump.as_mut() {
                 dump_batch(&batch, d);
             }
             if let Some(w) = wire_out.as_mut() {
                 w.write_all(&wire::to_json(&batch)).unwrap();
                 w.write_all(b"\n").unwrap();
+                w.flush().unwrap();
             }
             if let Some(w) = releases_out.as_mut() {
                 dump_releases(&batch, &group, w);
             }
         }
+        if ranges.len() > 1 {
+            eprintln!(
+                "{} {}–{}: {} headers in {:.1} s; frontier {}; {} releases out ({} finished), {} releases and {} files open; RSS {} MB",
+                group,
+                range.start,
+                range.end,
+                out.0 - headers_before,
+                range_wall.elapsed().as_secs_f64(),
+                date(indexer.frontier()),
+                out.1,
+                finished,
+                indexer.open_releases(),
+                indexer.open_files(),
+                peak_rss_mb()
+            );
+        }
+    }
     let total = inflate_s + index_s;
     #[allow(static_mut_refs)]
     let p = unsafe { indexer::PROF };
     eprintln!("  of which aggregate {:.2} s, evict+cutoff {:.2} s, judge+encode {:.2} s", p[3] as f64 / 1e9, p[4] as f64 / 1e9, p[5] as f64 / 1e9);
-    println!("{group} {start}–{end}: {} chunks, {} MB compressed, {} MB text", files.len(), compressed / 1_000_000, text_bytes / 1_000_000);
-    println!(
-        "{} headers in {:.2} s ({:.0}/s on one core; wall {:.2} s)",
-        out.0,
-        total,
-        out.0 as f64 / total,
-        wall.elapsed().as_secs_f64()
-    );
-    println!("  inflate {inflate_s:.2} s, index {index_s:.2} s; peak RSS {} MB", peak_rss_mb());
-    println!(
-        "  kept: {} release chunks ({} releases), {} files, {} segments, payload {:.1} MB ({:.1} B/segment); {} candidates, {} fragments, {} samples",
+    say(format!("{group} {start}–{end}: {} ranges, {chunks} chunks, {} MB compressed, {} MB text", ranges.len(), compressed / 1_000_000, text_bytes / 1_000_000));
+    say(format!("{} headers in {:.2} s ({:.0}/s on one core; wall {:.2} s)", out.0, total, out.0 as f64 / total, wall.elapsed().as_secs_f64()));
+    say(format!("  inflate {inflate_s:.2} s, index {index_s:.2} s; peak RSS {} MB", peak_rss_mb()));
+    say(format!(
+        "  kept: {} release chunks ({} releases, {} finished), {} files, {} segments, payload {:.1} MB ({:.1} B/segment); {} candidates, {} samples",
         out.1,
         keys.len(),
+        finished,
         out.2,
         out.3,
         out.4 as f64 / 1e6,
         out.4 as f64 / out.3.max(1) as f64,
         out.5,
-        out.6,
-        out.7
-    );
+        out.6
+    ));
 }

edge/src/names.rs

@@ -1237,6 +1237,11 @@ pub fn explored(key: &str, percent: f64) -> bool {
 }
 
 static SIDECAR: LazyLock<Regex> = LazyLock::new(|| re(r"(?i)\.(par2|nfo|sfv|txt|jpe?g|png|nzb)$"));
+
+/// A sidecar by its extension (the judge's SIDECAR): no release of its own, and no album's member.
+pub fn is_sidecar(name: &str) -> bool {
+    test(&SIDECAR, name)
+}
 static EXECUTABLE: LazyLock<Regex> = LazyLock::new(|| re(r"(?i)\.(exe|scr|com|pif|bat|cmd|vbs|lnk)SP*$"));
 
 pub struct JudgedFile<'a> {

edge/src/wire.rs

@@ -1,7 +1,7 @@
 //! A batch as the coordinator takes it (lib/edge/wire.ts toWire): JSON with
 //! the TypeScript names, chunk payloads in base64.
 
-use crate::indexer::{Batch, Candidate, Fragment, ReleaseChunk};
+use crate::indexer::{Batch, Candidate, ReleaseChunk};
 use base64::Engine;
 use serde::Serialize;
 
@@ -24,6 +24,11 @@ struct WireRelease<'a> {
     last_article: i64,
     readable: bool,
     has_par2: bool,
+    finished: bool,
+    edge: bool,
+    partial: bool,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    tool: Option<&'a str>,
     payload: String,
     #[serde(skip_serializing_if = "Option::is_none")]
     rule: Option<&'a str>,
@@ -49,30 +54,6 @@ struct WireCount<'a> {
     files: u64,
 }
 
-#[derive(Serialize)]
-#[serde(rename_all = "camelCase")]
-struct WireFragmentRelease<'a> {
-    key: &'a str,
-    name: &'a str,
-    source: &'a str,
-    #[serde(skip_serializing_if = "Option::is_none")]
-    file_total: Option<i64>,
-}
-
-#[derive(Serialize)]
-#[serde(rename_all = "camelCase")]
-struct WireFragment<'a> {
-    file_key: &'a str,
-    name: &'a str,
-    poster: &'a str,
-    subject: &'a str,
-    total_parts: u32,
-    groups: &'a [String],
-    release: Option<WireFragmentRelease<'a>>,
-    posted_at: Option<i64>,
-    segments: Vec<(u32, &'a str, i64, i64)>,
-}
-
 #[derive(Serialize)]
 #[serde(rename_all = "camelCase")]
 struct WireBatch<'a> {
@@ -85,7 +66,6 @@ struct WireBatch<'a> {
     samples: Vec<WireRelease<'a>>,
     candidates: Vec<WireCandidate<'a>>,
     counts: Vec<WireCount<'a>>,
-    fragments: Vec<WireFragment<'a>>,
 }
 
 fn release<'a>(r: &'a ReleaseChunk, rule: Option<(&'a str, &'a str)>) -> WireRelease<'a> {
@@ -106,6 +86,10 @@ fn release<'a>(r: &'a ReleaseChunk, rule: Option<(&'a str, &'a str)>) -> WireRel
         last_article: r.last_article,
         readable: r.readable,
         has_par2: r.has_par2,
+        finished: r.finished,
+        edge: r.edge,
+        partial: r.partial,
+        tool: r.tool,
         payload: base64::engine::general_purpose::STANDARD.encode(&r.payload),
         rule: rule.map(|(rule, _)| rule),
         rule_key: rule.map(|(_, key)| key),
@@ -116,21 +100,6 @@ fn candidate(c: &Candidate) -> WireCandidate<'_> {
     WireCandidate { release_key: &c.release_key, upload: c.upload, kind: c.kind, message_id: &c.message_id, file: &c.file }
 }
 
-fn fragment(f: &Fragment) -> WireFragment<'_> {
-    WireFragment {
-        file_key: &f.file_key,
-        name: &f.name,
-        poster: &f.poster,
-        subject: &f.subject,
-        total_parts: f.total_parts,
-        groups: &f.groups,
-        // A named file always has a release; fileTotal only when truthy, as fragmentOf.
-        release: Some(WireFragmentRelease { key: &f.release.key, name: &f.release.name, source: f.release.source, file_total: f.release.file_total_truthy() }),
-        posted_at: f.posted_at,
-        segments: f.segments.iter().map(|(n, id, bytes, article)| (*n, id.as_str(), *bytes, *article)).collect(),
-    }
-}
-
 /// toWire(batch) as JSON bytes.
 pub fn to_json(batch: &Batch) -> Vec<u8> {
     let wire = WireBatch {
@@ -143,7 +112,6 @@ pub fn to_json(batch: &Batch) -> Vec<u8> {
         samples: batch.samples.iter().map(|(r, rule, key)| release(r, Some((rule, key.as_str())))).collect(),
         candidates: batch.candidates.iter().map(candidate).collect(),
         counts: batch.counts.iter().map(|(rule, key, articles, files)| WireCount { rule, key, articles: *articles, files: *files }).collect(),
-        fragments: batch.fragments.iter().map(fragment).collect(),
     };
     serde_json::to_vec(&wire).unwrap()
 }

lib/edge/index.ts

@@ -2,7 +2,6 @@ export { RangeIndexer, postedSpan, chunkOf, candidatesOf, releaseKeyOf, uploadsO
 export { encodeFrame, decodeFrames, isKeepalive, KEEPALIVE, type Frame } from './stream.js'
 export { toWire, fromWire, type WireBatch } from './wire.js'
 export type { Lease } from './lease.js'
-export { mergeFragments } from './merge.js'
 export type { EdgeSegment, FileMeta } from './scan.js'
 export { throttle } from './throttle.js'
 export { filterOverviews, undelta, type ChunkDrops } from './corpus_filter.js'

lib/edge/merge.ts

@@ -1,35 +0,0 @@
-import type { AggregatedFile } from '../files/index.js'
-import type { Fragment } from './range_indexer.js'
-import type { EdgeSegment, FileMeta } from './scan.js'
-
-/**
- * The fragments of one file from neighbouring ranges, put together
- * (REBUILD.md, fragments): the complete file when every part from 1 to its
- * total is there (the first article seen of a part wins), else null.
- */
-export function mergeFragments(fragments: Fragment[]): AggregatedFile<EdgeSegment> | null {
-  if (fragments.length === 0) return null
-  const first = fragments[0]
-  const meta: FileMeta = {
-    fileKey: first.fileKey,
-    name: first.name,
-    poster: first.poster,
-    subject: first.subject,
-    totalParts: first.totalParts,
-    groups: first.groups,
-    release: first.release ? { ...first.release, source: first.release.source as any } : null,
-  }
-  const parts = new Map<number, EdgeSegment>()
-  for (const fragment of fragments) {
-    const postedAt = fragment.postedAt ?? Number.NaN
-    for (const [number, messageId, bytes, articleNumber] of fragment.segments) {
-      const known = parts.get(number)
-      if (!known || articleNumber < known.articleNumber) {
-        parts.set(number, { fileKey: meta.fileKey, number, totalParts: meta.totalParts, messageId, bytes, articleNumber, postedAt, file: meta })
-      }
-    }
-  }
-  if (parts.size < meta.totalParts) return null
-  for (let n = 1; n <= meta.totalParts; n++) if (!parts.has(n)) return null
-  return { fileKey: meta.fileKey, segments: [...parts.values()].sort((a, b) => a.number - b.number) }
-}

lib/edge/range_indexer.ts

@@ -26,6 +26,20 @@ export interface ReleaseChunk {
   /** A file with a readable name; a par2 file. */
   readable: boolean
   hasPar2: boolean
+  /**
+   * Voyager 3: the window closed on the release (default), or it is a tail release sent provisionally, finished by a
+   * later chunk of the same release key (the totals then are the release's; the payload only the files not sent yet).
+   */
+  finished?: boolean
+  /** Still open at its block's end, or posted within the window of its start: the block merger joins it with the neighbour block's part. */
+  edge?: boolean
+  /** A file of the release misses parts (the window closed on it): not complete, whatever the counter says. */
+  partial?: boolean
+  /**
+   * Voyager 3, an unnamed release (readable false): the upload tool from its first article's message-id, its
+   * signature with the group; the payload is empty (no chunk until a probe names it: pending_releases).
+   */
+  tool?: string
   payload: Buffer
 }
 
@@ -88,7 +102,8 @@ export interface EdgeBatch {
   samples: DroppedChunk[]
   candidates: ProbeCandidate[]
   counts: RuleCount[]
-  fragments: Fragment[]
+  /** The range path's cut-off files (the TypeScript indexer); the Rust edge's window has none, the store ignores them. */
+  fragments?: Fragment[]
 }
 
 export interface RangeIndexerOptions {
@@ -203,7 +218,7 @@ export class RangeIndexer {
 
   /** Files completed and fragments cut since the last take: how big the next batch is. */
   get pending(): number {
-    let files = this.batch.fragments.length
+    let files = this.batch.fragments!.length
     for (const list of this.completed.values()) files += list.length
     return files
   }
@@ -287,7 +302,7 @@ export class RangeIndexer {
         fragment.segments.push([s.number, s.messageId, s.bytes, s.articleNumber])
         if (Number.isFinite(s.postedAt) && (fragment.postedAt === null || s.postedAt < fragment.postedAt)) fragment.postedAt = s.postedAt
       }
-      this.batch.fragments.push(fragment)
+      this.batch.fragments!.push(fragment)
     }
   }
 

lib/probe/index.ts

@@ -1 +1 @@
-export { inspect, innerName, MAX_NAMES, PROBE_VERSION, type Inspection } from './inspect.js'
+export { inspect, innerName, usableName, MAX_NAMES, PROBE_VERSION, type Inspection } from './inspect.js'

lib/probe/inspect.ts

@@ -5,6 +5,7 @@ import { parseRarHeaders } from '#lib/rar/index'
 import { parseSevenZipTail } from '#lib/sevenzip/index'
 import { parseZipStart } from '#lib/zip/index'
 import { sniffContentType, type ContentType } from '#lib/content/index'
+import { fileStem, looksRandom } from '#lib/release/index'
 
 /** What one probed article showed (REBUILD.md, part 6; stored per release and kind). */
 export interface Inspection {
@@ -25,7 +26,10 @@ export interface Inspection {
    */
   hash16k?: string
   /** A par2 body's recovery set: every file it describes (not kept in release_inspections; par2_files). */
-  par2?: { setId: string; files: { fileId: string; hash16k: string; length: number; name: string }[] }
+  par2?: {
+    setId: string
+    files: { fileId: string; hash16k: string; length: number; name: string }[]
+  }
 }
 
 const HASH_BYTES = 16 * 1024
@@ -63,9 +67,16 @@ function largestFirst(items: { name: string; size: number }[]): string[] {
  */
 export function inspect(body: Buffer, kind?: string): Inspection {
   const segment = decodeBody(body)
-  const result: Inspection = { yencName: segment.meta.begin.name.slice(0, MAX_NAME_LENGTH), size: segment.meta.begin.size, type: sniffContentType(segment.data) }
+  const result: Inspection = {
+    yencName: segment.meta.begin.name.slice(0, MAX_NAME_LENGTH),
+    size: segment.meta.begin.size,
+    type: sniffContentType(segment.data),
+  }
   // The file's first 16 KiB, or all of a smaller file.
-  if (segment.offset === 0 && (segment.data.length >= HASH_BYTES || segment.data.length === result.size)) {
+  if (
+    segment.offset === 0 &&
+    (segment.data.length >= HASH_BYTES || segment.data.length === result.size)
+  ) {
     result.hash16k = createHash('md5').update(segment.data.subarray(0, HASH_BYTES)).digest('hex')
   }
   if (isPar2(segment.data)) {
@@ -76,7 +87,12 @@ export function inspect(body: Buffer, kind?: string): Inspection {
       result.names = largestFirst(set.files.map((f) => ({ name: f.name, size: f.length })))
       result.par2 = {
         setId: set.recoverySetId,
-        files: set.files.map((f) => ({ fileId: f.fileId, hash16k: f.md5_16k, length: f.length, name: f.name.slice(0, MAX_NAME_LENGTH) })),
+        files: set.files.map((f) => ({
+          fileId: f.fileId,
+          hash16k: f.md5_16k,
+          length: f.length,
+          name: f.name.slice(0, MAX_NAME_LENGTH),
+        })),
       }
     }
     return result
@@ -99,7 +115,10 @@ export function inspect(body: Buffer, kind?: string): Inspection {
     }
   }
   // 7z keeps its listing at the end: the last segment of the last volume (a 7z-tail), or a small archive whole.
-  if ((kind === '7z-tail' || result.type === '7z') && segment.offset + segment.data.length === result.size) {
+  if (
+    (kind === '7z-tail' || result.type === '7z') &&
+    segment.offset + segment.data.length === result.size
+  ) {
     const listing = parseSevenZipTail(segment.data)
     if (listing) {
       result.entries = listing.files.length
@@ -110,16 +129,43 @@ export function inspect(body: Buffer, kind?: string): Inspection {
   return result
 }
 
+/**
+ * The usable name one probe of an unnamed release found (the pending ladder, REBUILD.md decision f): the largest
+ * named file of a par2 set or an archive's listing, or a lone file's yEnc name, when it isn't packaging and isn't
+ * random itself (an obfuscated upload's par2 names its random volumes; its yEnc name is mostly the random name
+ * again). Null: this probe named nothing, the next target is tried.
+ */
+export function usableName(inspection: Inspection, kind: string): string | null {
+  const usable = (n: string | undefined) => {
+    if (!n) return null
+    const name = n.replace(/^.*[\\/]/, '').slice(0, 255)
+    return name && !PACKAGING.test(name) && !looksRandom(fileStem(name)) ? name : null
+  }
+  const named = (inspection.names ?? []).map(usable).find((n) => n)
+  if (named) return named
+  return kind === 'lone' || !inspection.names?.length ? usable(inspection.yencName) : null
+}
+
 /**
  * A release's best inner name from its inspections (by probe kind): the
  * largest named entry of an archive, else the largest file of a par2 set
  * that isn't packaging, else a lone file's yEnc name. Null when none says
  * more than the release's own name.
  */
-export function innerName(inspections: { kind: string; result: Inspection }[], releaseName = ''): string | null {
+export function innerName(
+  inspections: { kind: string; result: Inspection }[],
+  releaseName = ''
+): string | null {
   const of = (kind: string) => inspections.filter((i) => i.kind === kind).map((i) => i.result)
-  const content = (kind: string) => of(kind).flatMap((i) => i.names ?? []).find((n) => !PACKAGING.test(n))
-  const pick = content('archive') ?? content('7z-tail') ?? content('par2') ?? of('lone').find((i) => i.yencName && !PACKAGING.test(i.yencName))?.yencName
+  const content = (kind: string) =>
+    of(kind)
+      .flatMap((i) => i.names ?? [])
+      .find((n) => !PACKAGING.test(n))
+  const pick =
+    content('archive') ??
+    content('7z-tail') ??
+    content('par2') ??
+    of('lone').find((i) => i.yencName && !PACKAGING.test(i.yencName))?.yencName
   const name = pick?.replace(/^.*[\\/]/, '').slice(0, 255)
   // The release's own name with an extension says nothing new (a random-named file's yEnc name is mostly its random name).
   return name && !name.toLowerCase().startsWith(releaseName.toLowerCase() || '\0') ? name : null

lib/release/keys.ts

@@ -0,0 +1,34 @@
+import { createHash } from 'node:crypto'
+
+/**
+ * The catalog's keys as voyager 3 stores them (migration 6000): 20-byte SHA-1s in bytea columns.
+ * - identity_key: the identity's hash (lib/release/identity.ts gives it as 40 hex characters).
+ * - release_key: one upload of an identity, unique: the identity, the owner group and the first article the window
+ *   saw of it. The same release sent twice (a tail release provisionally, then finished) hits the same row; a repost
+ *   starts at another article and is a release of its own (decision d). Deterministic, so no numbering under a lock.
+ */
+const HEX40 = /^[0-9a-f]{40}$/i
+
+function sha1(text: string): Buffer {
+  return createHash('sha1').update(text).digest()
+}
+
+/** The identity key's bytes: 40 hex characters decoded, any other text (a spot's, a test's) hashed. */
+export function identityKey(key: string): Buffer {
+  return HEX40.test(key) ? Buffer.from(key, 'hex') : sha1(key)
+}
+
+/** The release key's bytes for an identity in a group, first seen at an article (null for releases without one: a spot's). */
+export function releaseKey(
+  identity: string,
+  groupId: number | null,
+  firstArticle: number | null
+): Buffer {
+  return sha1(`release\n${identity}\n${groupId ?? ''}\n${firstArticle ?? ''}`)
+}
+
+/** A stored key as hex, for logs, the API and release_merges.from_key. */
+export function keyHex(key: Buffer | Uint8Array | string | null | undefined): string | null {
+  if (key === null || key === undefined) return null
+  return typeof key === 'string' ? key : Buffer.from(key).toString('hex')
+}

lib/release/source.ts

@@ -0,0 +1,38 @@
+/**
+ * How a release got its name, as releases.source stores it (a smallint; voyager 3, migration 6000). The names are
+ * what the edge, the spot inspector and the API use; the numbers never change meaning once written.
+ */
+export const SOURCES = {
+  /** A name the subject announces ("name" yEnc (k/N)). */
+  subject: 1,
+  /** The post's files, named after the first file (no release name in the subject). */
+  post: 2,
+  /** The files' shared stem. */
+  stem: 3,
+  /** A Spotnet spot's files that matched no release. */
+  spot: 4,
+  /** An NZB's files. */
+  nzb: 5,
+  /** One poster's files that differ only by a number sequence, or share a prefix (voyager 3). */
+  album: 6,
+  /** An obfuscated post's stem, read from the yEnc body (identity.ts: weak). */
+  'weak-stem': 7,
+} as const
+
+export type Source = keyof typeof SOURCES
+
+const NAMES = Object.fromEntries(
+  Object.entries(SOURCES).map(([name, code]) => [code, name])
+) as Record<number, Source>
+
+/** The code of a source name; throws for an unknown one (a typo would write a wrong number for good). */
+export function sourceCode(name: string): number {
+  const code = SOURCES[name as Source]
+  if (code === undefined) throw new Error(`Unknown release source "${name}"`)
+  return code
+}
+
+/** The name of a stored code; "unknown" for one not listed (never written, but a row might be read by an older build). */
+export function sourceName(code: number | string | null | undefined): Source | 'unknown' {
+  return NAMES[Number(code)] ?? 'unknown'
+}

lib/release/text_kind.ts

@@ -0,0 +1,60 @@
+/**
+ * What a searchable string of a release is, as release_texts.kind stores it (a smallint; voyager 3, migration 6200).
+ * The table holds every string search matches against (the release name too), appended by whoever learns one: the
+ * batch store the name, the prober the file names inside an archive, later passes NFO titles, par2 set names,
+ * media titles. The numbers never change meaning once written.
+ */
+export const TEXT_KINDS = {
+  /** The release name (releases.name, once more for the one trigram index). */
+  name: 1,
+  /** A file name inside an archive (the prober's inner names). */
+  file: 2,
+  /** A title read from an NFO. */
+  nfo: 3,
+  /** A par2 set's name when it differs from the release name. */
+  par2: 4,
+  /** A title from a media container's metadata. */
+  media: 5,
+  /** An album's member, by its own file name (the edge's albums: one poster's files differing by a number sequence). */
+  member: 6,
+} as const
+
+export type TextKind = keyof typeof TEXT_KINDS
+
+/** The longest string stored (release_texts.text is a varchar(255)); longer ones are cut, not refused. */
+export const TEXT_MAX = 255
+/** The most strings one writer appends per release and kind (an archive of two thousand files is not two thousand rows). */
+export const TEXTS_PER_KIND = 20
+
+const NAMES = Object.fromEntries(
+  Object.entries(TEXT_KINDS).map(([name, code]) => [code, name])
+) as Record<number, TextKind>
+
+/** The code of a kind; throws for an unknown one (a typo would write a wrong number for good). */
+export function textKindCode(name: string): number {
+  const code = TEXT_KINDS[name as TextKind]
+  if (code === undefined) throw new Error(`Unknown release text kind "${name}"`)
+  return code
+}
+
+/** The name of a stored code; "unknown" for one not listed. */
+export function textKindName(code: number | string | null | undefined): TextKind | 'unknown' {
+  return NAMES[Number(code)] ?? 'unknown'
+}
+
+/**
+ * The strings a writer appends for one kind: trimmed, cut to TEXT_MAX, empty ones and repeats dropped, the first
+ * TEXTS_PER_KIND kept in the order given (the writer puts the most telling first).
+ */
+export function textsToStore(texts: Iterable<string | null | undefined>): string[] {
+  const seen = new Set<string>()
+  const out: string[] = []
+  for (const raw of texts) {
+    const text = (raw ?? '').trim().slice(0, TEXT_MAX)
+    if (!text || seen.has(text)) continue
+    seen.add(text)
+    out.push(text)
+    if (out.length === TEXTS_PER_KIND) break
+  }
+  return out
+}

start/env.ts

@@ -74,6 +74,9 @@ export default await Env.create(new URL('../', import.meta.url), {
   SPOT_STANDALONE_AFTER_HOURS: Env.schema.number.optional(),
   CORPUS_AHEAD: Env.schema.number.optional(),
   PROBE_PIPELINE: Env.schema.number.optional(),
+  PENDING_SHARE: Env.schema.number.optional(),
+  PENDING_MIN_PROBES: Env.schema.number.optional(),
+  PENDING_MAX_NAMED: Env.schema.number.optional(),
   SPOOL_AHEAD_PER_LEASE: Env.schema.number.optional(),
   SPOOL_MIN_AHEAD: Env.schema.number.optional(),
   SPOOL_MAX_GB: Env.schema.number.optional(),

tests/bootstrap.ts

@@ -8,6 +8,7 @@ import testUtils from '@adonisjs/core/services/test_utils'
 import { authApiClient } from '@adonisjs/auth/plugins/api_client'
 import { sessionApiClient } from '@adonisjs/session/plugins/api_client'
 import type { Registry } from '../.adonisjs/client/registry/schema.d.ts'
+import CatalogRefs from '#services/catalog_refs'
 
 /**
  * This file is imported by the "bin/test.ts" entrypoint file
@@ -51,6 +52,8 @@ export const runnerHooks: Required<Pick<Config, 'setup' | 'teardown'>> = {
  */
 export const configureSuite: Config['configureSuite'] = (suite) => {
   if (['browser', 'functional', 'e2e'].includes(suite.name)) {
+    // Each test's transaction is rolled back: the poster and group-set ids cached meanwhile are gone with it.
+    suite.onGroup((group) => group.each.setup(() => CatalogRefs.forget()))
     return suite.setup(() => testUtils.httpServer().start())
   }
 }

tests/functional/batch_store.spec.ts

@@ -2,19 +2,35 @@ import { test } from '@japa/runner'
 import db from '@adonisjs/lucid/services/db'
 import testUtils from '@adonisjs/core/services/test_utils'
 import BatchStore, { StaleBatch } from '#services/batch_store'
-import { RangeIndexer } from '#lib/edge/index'
-import { decodeChunk } from '#lib/chunks/index'
+import { RangeIndexer, type EdgeBatch, type ReleaseChunk } from '#lib/edge/index'
+import { decodeChunk, encodeChunk } from '#lib/chunks/index'
 import { DEFAULT_RULES } from '#lib/rules/index'
 import type { Overview } from '#lib/nntp/index'
 
 const GROUP = 'alt.binaries.test'
 let id = 0
 
-function post(at: number, name: string, parts: number, counter: string, tool: 'ngPost' | 'plain' = 'plain'): Overview[] {
+function post(
+  at: number,
+  name: string,
+  parts: number,
+  counter: string,
+  tool: 'ngPost' | 'plain' = 'plain'
+): Overview[] {
   return Array.from({ length: parts }, (_, i) => {
     id++
     const mid = tool === 'ngPost' ? `${id.toString(16).padStart(32, '0')}@ngPost` : `art${id}@test`
-    return [at + i, { 'subject': `${counter} - "${name}" yEnc (${i + 1}/${parts})`, 'from': 'p <p@x>', 'date': 'Mon, 05 Oct 2026 10:00:00 +0000', 'message-id': `<${mid}>`, ':bytes': '700000', 'xref': `news.example ${GROUP}:${at + i}` }] as Overview
+    return [
+      at + i,
+      {
+        'subject': `${counter} - "${name}" yEnc (${i + 1}/${parts})`,
+        'from': 'p <p@x>',
+        'date': 'Mon, 05 Oct 2026 10:00:00 +0000',
+        'message-id': `<${mid}>`,
+        ':bytes': '700000',
+        'xref': `news.example ${GROUP}:${at + i}`,
+      },
+    ] as Overview
   })
 }
 
@@ -23,73 +39,224 @@ async function leasedRange() {
   const [node] = await db.table('nodes').insert({ name: 'test-node' }).returning('id')
   const [range] = await db
     .table('ranges')
-    .insert({ group_id: group.id, start_id: 1, end_id: 1_000_000, next_id: 1, status: 'leased', node_id: node.id, rule_version: 1 })
+    .insert({
+      group_id: group.id,
+      start_id: 1,
+      end_id: 1_000_000,
+      next_id: 1,
+      status: 'leased',
+      node_id: node.id,
+      rule_version: 1,
+    })
     .returning('id')
   return { rangeId: Number(range.id), nodeId: Number(node.id) }
 }
 
 function indexer() {
-  return new RangeIndexer({ group: GROUP, start: 1, end: 1_000_000, rules: { ...DEFAULT_RULES, samplePercent: 0, droppedSignatures: [], priorities: { [GROUP]: 1 } }, edgeArticles: 1000 })
+  return new RangeIndexer({
+    group: GROUP,
+    start: 1,
+    end: 1_000_000,
+    rules: {
+      ...DEFAULT_RULES,
+      samplePercent: 0,
+      droppedSignatures: [],
+      priorities: { [GROUP]: 1 },
+    },
+    edgeArticles: 1000,
+  })
 }
 
 /** One file's articles, posted `minutes` apart from `start` (an ISO time), in `group`. */
-function upload(at: number, name: string, parts: number, counter: string, start: string, group = GROUP, minutes = 1): Overview[] {
+function upload(
+  at: number,
+  name: string,
+  parts: number,
+  counter: string,
+  start: string,
+  group = GROUP,
+  minutes = 1
+): Overview[] {
   return Array.from({ length: parts }, (_, i) => {
     id++
-    const date = new Date(Date.parse(start) + i * minutes * 60_000).toUTCString().replace('GMT', '+0000')
-    return [at + i, { 'subject': `${counter} - "${name}" yEnc (${i + 1}/${parts})`, 'from': 'p <p@x>', 'date': date, 'message-id': `<up${id}@test>`, ':bytes': '700000', 'xref': `news.example ${group}:${at + i}` }] as Overview
+    const date = new Date(Date.parse(start) + i * minutes * 60_000)
+      .toUTCString()
+      .replace('GMT', '+0000')
+    return [
+      at + i,
+      {
+        'subject': `${counter} - "${name}" yEnc (${i + 1}/${parts})`,
+        'from': 'p <p@x>',
+        'date': date,
+        'message-id': `<up${id}@test>`,
+        ':bytes': '700000',
+        'xref': `news.example ${group}:${at + i}`,
+      },
+    ] as Overview
   })
 }
 
 async function rangeIn(groupName: string, start: number, nodeId?: number) {
-  const g = (await db.from('groups').where('name', groupName).first()) ?? (await db.table('groups').insert({ name: groupName, priority: groupName === GROUP ? 1 : 2 }).returning('*'))[0]
-  const node = nodeId ?? Number((await db.table('nodes').insert({ name: `n${groupName}${start}` }).returning('id'))[0].id)
-  const [range] = await db.table('ranges').insert({ group_id: g.id, start_id: start, end_id: start + 99_999, next_id: start, status: 'leased', node_id: node, rule_version: 1 }).returning('id')
+  const g =
+    (await db.from('groups').where('name', groupName).first()) ??
+    (
+      await db
+        .table('groups')
+        .insert({ name: groupName, priority: groupName === GROUP ? 1 : 2 })
+        .returning('*')
+    )[0]
+  const node =
+    nodeId ??
+    Number(
+      (
+        await db
+          .table('nodes')
+          .insert({ name: `n${groupName}${start}` })
+          .returning('id')
+      )[0].id
+    )
+  const [range] = await db
+    .table('ranges')
+    .insert({
+      group_id: g.id,
+      start_id: start,
+      end_id: start + 99_999,
+      next_id: start,
+      status: 'leased',
+      node_id: node,
+      rule_version: 1,
+    })
+    .returning('id')
   return { rangeId: Number(range.id), nodeId: node }
 }
 
+function batchOf(groupName: string, start: number, overviews: Overview[]): EdgeBatch {
+  const ix = new RangeIndexer({
+    group: groupName,
+    start,
+    end: start + 99_999,
+    rules: {
+      ...DEFAULT_RULES,
+      samplePercent: 0,
+      droppedSignatures: [],
+      priorities: { [GROUP]: 1, 'alt.binaries.other': 2 },
+    },
+    edgeArticles: 1,
+  })
+  ix.add(overviews, start + 100_000)
+  ix.finish()
+  return ix.take()
+}
+
 async function storeIn(groupName: string, start: number, overviews: Overview[]) {
   const { rangeId, nodeId } = await rangeIn(groupName, start)
-  const ix = new RangeIndexer({ group: groupName, start, end: start + 99_999, rules: { ...DEFAULT_RULES, samplePercent: 0, droppedSignatures: [], priorities: { [GROUP]: 1, 'alt.binaries.other': 2 } }, edgeArticles: 1 })
-  ix.add(overviews, start + 100_000)
-  ix.finish()
-  await BatchStore.store(rangeId, nodeId, ix.take())
+  return BatchStore.store(rangeId, nodeId, batchOf(groupName, start, overviews))
 }
 
-const releasesOf = () => db.from('releases').orderBy('id').select('release_key', 'identity_key', 'files', 'posted_at', 'last_posted_at')
+const releasesOf = () =>
+  db
+    .from('releases')
+    .orderBy('id')
+    .select(
+      db.raw(
+        `encode(release_key, 'hex') AS release_key, encode(identity_key, 'hex') AS identity_key, files, chunks, posted_at, last_posted_at, settled_at`
+      )
+    )
 
 test.group('BatchStore', (group) => {
   group.each.setup(() => testUtils.db().withGlobalTransaction())
 
-  test('stores releases, their chunks, probe candidates, rule counts and fragments, and moves the resume point', async ({ assert }) => {
+  test('stores releases by id references, their texts, chunks, probe candidates and rule counts, and moves the resume point', async ({
+    assert,
+  }) => {
     const { rangeId, nodeId } = await leasedRange()
     const ix = indexer()
-    ix.add([
+    ix.add(
+      [
         ...post(100_000, 'Some.Show.S01E01.1080p.WEB.part1.rar', 2, 'Some.Show.S01E01 [1/2]'),
         ...post(100_010, 'Some.Show.S01E01.1080p.WEB.par2', 1, 'Some.Show.S01E01 [2/2]'),
         ...post(100_020, 'Hs7dKq2LmZ9xWv3RtYp1Qa.part01.rar', 3, '[1/1]', 'ngPost'),
-      ...post(10, 'Edge.Cut.2020.part1.rar', 3, 'Edge.Cut [1/1]').slice(1),
-    ], 500_000)
+      ],
+      500_000
+    )
     const first = ix.take()
     const stored = await BatchStore.store(rangeId, nodeId, first)
-    assert.include(stored, { releases: 1, newReleases: 1, candidates: 2, fragments: 0 })
+    assert.deepEqual(stored, {
+      releases: 1,
+      newReleases: 1,
+      finalized: 0,
+      candidates: 2,
+      pending: 0,
+    })
 
     const release = await db.from('releases').first()
-    assert.include(release, { files: 2, segments: '3', chunks: 1, readable: true, has_par2: true, file_total: 2 })
+    assert.include(release, {
+      files: 2,
+      segments: '3',
+      chunks: 1,
+      readable: true,
+      has_par2: true,
+      file_total: 2,
+      complete: true,
+      edge: false,
+      source: 1,
+    })
+    assert.isNotNull(release.settled_at)
+    // The poster and the groups by id, the keys 20 bytes, the name searchable.
+    assert.equal(
+      (await db.from('posters').where('id', release.poster_id).firstOrFail()).poster,
+      'p <p@x>'
+    )
+    assert.deepEqual(
+      (await db.from('group_sets').where('id', release.group_set_id).firstOrFail()).groups,
+      [GROUP]
+    )
+    assert.lengthOf(release.release_key, 20)
+    assert.lengthOf(release.identity_key, 20)
+    assert.deepEqual(await db.from('release_texts').select('release_id', 'kind', 'text'), [
+      { release_id: release.id, kind: 1, text: 'Some Show S01E01' },
+    ])
     const chunk = await db.from('release_chunks').where('release_id', release.id).first()
-    assert.deepEqual(decodeChunk(chunk.payload).map((f) => f.name).sort(), ['Some.Show.S01E01.1080p.WEB.par2', 'Some.Show.S01E01.1080p.WEB.part1.rar'])
-    assert.deepEqual((await db.from('probe_candidates').orderBy('kind')).map((c) => c.kind), ['archive', 'par2'])
-    assert.deepEqual((await db.from('rule_counts')).map((c) => [c.rule, c.key, Number(c.articles)]), [['garbage', 'ngPost alt.binaries.test', 3]])
-    assert.equal(Number((await db.from('ranges').where('id', rangeId).first()).next_id), 500_000)
+    assert.deepEqual(
+      decodeChunk(chunk.payload)
+        .map((f) => f.name)
+        .sort(),
+      ['Some.Show.S01E01.1080p.WEB.par2', 'Some.Show.S01E01.1080p.WEB.part1.rar']
+    )
+    // Claimable at once: the release is finished.
+    assert.deepEqual(
+      (await db.from('probe_candidates').orderBy('kind')).map((c) => [
+        c.kind,
+        c.ready,
+        c.unreadable,
+      ]),
+      [
+        ['archive', true, false],
+        ['par2', true, false],
+      ]
+    )
+    // Keyed by the range's block (a range without one is a block of its own, until blocks are leased).
+    const range = await db.from('ranges').where('id', rangeId).first()
+    assert.isNotNull(range.block_id)
+    assert.equal(Number(chunk.block_id), Number(range.block_id))
+    assert.deepEqual(
+      (await db.from('rule_counts')).map((c) => [
+        Number(c.block_id),
+        c.rule,
+        c.key,
+        Number(c.articles),
+      ]),
+      [[Number(range.block_id), 'garbage', 'ngPost alt.binaries.test', 3]]
+    )
+    assert.equal(Number(range.next_id), 500_000)
 
     // The same batch again (a retry after a lost reply) is refused, and changes nothing.
     await assert.rejects(() => BatchStore.store(rangeId, nodeId, first), StaleBatch)
     assert.equal((await db.from('releases').first()).files, 2)
 
-    // The end of the range: the cut-off file near the start edge is kept as a fragment.
+    // The end of the range.
     ix.finish()
-    const last = ix.take()
-    assert.include(await BatchStore.store(rangeId, nodeId, last), { fragments: 1 })
+    await BatchStore.store(rangeId, nodeId, ix.take())
     assert.equal(Number((await db.from('ranges').where('id', rangeId).first()).next_id), 1_000_001)
   })
 
@@ -97,52 +264,192 @@ test.group('BatchStore', (group) => {
     const { rangeId, nodeId } = await leasedRange()
     const ix = indexer()
     ix.add(post(100_000, 'A.Release.2020.mkv', 1, 'A [1/1]'), 200_000)
-    await assert.rejects(() => BatchStore.store(rangeId, nodeId + 1, ix.take()), /not leased to this node/)
+    await assert.rejects(
+      () => BatchStore.store(rangeId, nodeId + 1, ix.take()),
+      /not leased to this node/
+    )
   })
-  test('a re-upload is a release of its own: in one range, or a later range; a continuation joins', async ({ assert }) => {
-    // One range: the same file posted Monday and again Thursday.
+
+  test('every chunk is a release of its own: a re-upload, and a part read by another range (the block merger joins those)', async ({
+    assert,
+  }) => {
+    // One range: the same file posted Monday and again Thursday are two uploads.
     await storeIn(GROUP, 1, [
-      ...upload(10, 'Some.Show.S01E01.1080p.mkv', 2, 'Some.Show.S01E01.1080p [1/2]', '2026-10-05T10:00:00Z'),
-      ...upload(20, 'Some.Show.S01E01.1080p.par2', 1, 'Some.Show.S01E01.1080p [2/2]', '2026-10-05T10:05:00Z'),
-      ...upload(500, 'Some.Show.S01E01.1080p.mkv', 2, 'Some.Show.S01E01.1080p [1/2]', '2026-10-08T10:00:00Z'),
+      ...upload(
+        10,
+        'Some.Show.S01E01.1080p.mkv',
+        2,
+        'Some.Show.S01E01.1080p [1/2]',
+        '2026-10-05T10:00:00Z'
+      ),
+      ...upload(
+        20,
+        'Some.Show.S01E01.1080p.par2',
+        1,
+        'Some.Show.S01E01.1080p [2/2]',
+        '2026-10-05T10:05:00Z'
+      ),
+      ...upload(
+        500,
+        'Some.Show.S01E01.1080p.mkv',
+        2,
+        'Some.Show.S01E01.1080p [1/2]',
+        '2026-10-08T10:00:00Z'
+      ),
     ])
     let rows = await releasesOf()
-    assert.lengthOf(rows, 2)
-    assert.equal(rows[1].release_key, `${rows[0].release_key}:2`)
-    assert.deepEqual(rows.map((r) => [r.identity_key, r.files]), [[rows[0].release_key, 2], [rows[0].release_key, 1]])
+    assert.deepEqual(
+      rows.map((r) => r.files),
+      [2, 1]
+    )
+    assert.equal(rows[0].identity_key, rows[1].identity_key)
+    assert.notEqual(rows[0].release_key, rows[1].release_key)
 
-    // The next range: Thursday's par2 an hour later joins Thursday's release; a Saturday copy is a third.
-    await storeIn(GROUP, 100_001, upload(100_010, 'Some.Show.S01E01.1080p.par2', 1, 'Some.Show.S01E01.1080p [2/2]', '2026-10-08T11:00:00Z'))
-    await storeIn(GROUP, 200_001, upload(200_010, 'Some.Show.S01E01.1080p.mkv', 2, 'Some.Show.S01E01.1080p [1/2]', '2026-10-10T10:00:00Z'))
+    // The next range: Thursday's par2 an hour later is written as a release of its own (its key: the identity, the
+    // group and its first article); the window keeps such parts together, and the block merger joins the edges.
+    await storeIn(
+      GROUP,
+      100_001,
+      upload(
+        100_010,
+        'Some.Show.S01E01.1080p.par2',
+        1,
+        'Some.Show.S01E01.1080p [2/2]',
+        '2026-10-08T11:00:00Z'
+      )
+    )
     rows = await releasesOf()
-    assert.deepEqual(rows.map((r) => [r.release_key.split(':')[1] ?? '1', r.files]), [['1', 2], ['2', 2], ['3', 1]])
-    const chunks = await db.from('release_chunks').count('* as n').first()
-    assert.equal(Number(chunks!.n), 4)
+    assert.deepEqual(
+      rows.map((r) => r.files),
+      [2, 1, 1]
+    )
+    assert.lengthOf(new Set(rows.map((r) => r.release_key)), 3)
+    assert.lengthOf(new Set(rows.map((r) => r.identity_key)), 1)
+    assert.equal(Number((await db.from('release_chunks').count('* as n').first())!.n), 3)
   })
 
-  test('candidates added to a settled release (a later range of its upload) are claimable at once', async ({ assert }) => {
-    await storeIn(GROUP, 1, upload(10, 'Some.Show.S01E02.1080p.mkv', 2, 'Some.Show.S01E02.1080p [1/2]', '2026-10-05T10:00:00Z'))
-    // Settled before the next range came (the settler marks only what it settles).
-    await db.from('releases').update({ settled_at: new Date() })
-    await db.from('probe_candidates').update({ ready: true })
-    await storeIn(GROUP, 100_001, upload(100_010, 'Some.Show.S01E02.1080p.par2', 1, 'Some.Show.S01E02.1080p [2/2]', '2026-10-05T10:20:00Z'))
-    const candidates = await db.from('probe_candidates').orderBy('id').select('kind', 'ready')
-    assert.deepEqual(candidates.map((c) => c.kind), ['par2'])
-    assert.isTrue(candidates.every((c) => c.ready))
-    assert.lengthOf(await releasesOf(), 1)
+  test('a provisional tail release is finished once by a later chunk of its key, which adds the files not sent before', async ({
+    assert,
+  }) => {
+    const { rangeId, nodeId } = await rangeIn(GROUP, 1)
+    const batch = batchOf(
+      GROUP,
+      1,
+      upload(
+        10,
+        'Some.Show.S01E02.1080p.mkv',
+        2,
+        'Some.Show.S01E02.1080p [1/2]',
+        '2026-10-05T10:00:00Z'
+      )
+    )
+    assert.lengthOf(batch.releases, 1)
+    const provisional: ReleaseChunk = { ...batch.releases[0], finished: false }
+    const stored = await BatchStore.store(rangeId, nodeId, {
+      ...batch,
+      next: 50_000,
+      releases: [provisional],
+    })
+    assert.include(stored, { newReleases: 1, finalized: 0 })
+    let [row] = await releasesOf()
+    assert.isNull(row.settled_at)
+    assert.include(row, { files: 1, chunks: 1 })
+
+    // The finish: the par2 that came within the window, under the same key (the identity, the group, the first
+    // article), with its probe candidate: claimable at once, the release being finished.
+    const second = batchOf(
+      GROUP,
+      1,
+      upload(
+        30,
+        'Some.Show.S01E02.1080p.par2',
+        1,
+        'Some.Show.S01E02.1080p [2/2]',
+        '2026-10-05T10:20:00Z'
+      )
+    )
+    const [par2] = second.releases
+    const finish: ReleaseChunk = {
+      ...par2,
+      key: provisional.key,
+      upload: provisional.upload,
+      firstArticle: provisional.firstArticle,
+      finished: true,
+    }
+    const candidates = second.candidates.map((c) => ({
+      ...c,
+      releaseKey: provisional.key,
+      upload: provisional.upload,
+    }))
+    assert.deepEqual(
+      candidates.map((c) => c.kind),
+      ['par2']
+    )
+    const finished = await BatchStore.store(rangeId, nodeId, {
+      ...batch,
+      from: 50_000,
+      next: 60_000,
+      releases: [finish],
+      candidates,
+      counts: [],
+      samples: [],
+      fragments: [],
+    })
+    assert.include(finished, { releases: 1, newReleases: 0, finalized: 1, candidates: 1 })
+    ;[row] = await releasesOf()
+    assert.isNotNull(row.settled_at)
+    assert.include(row, { files: 2, chunks: 2 })
+    const release = await db.from('releases').firstOrFail()
+    assert.include(release, { complete: true, has_par2: true, last_article: '30' })
+    assert.deepEqual(await db.from('probe_candidates').select('kind', 'ready'), [
+      { kind: 'par2', ready: true },
+    ])
+    assert.deepEqual(
+      (await db.from('release_chunks').orderBy('id')).flatMap((c) =>
+        decodeChunk(c.payload).map((f) => f.name)
+      ),
+      ['Some.Show.S01E02.1080p.mkv', 'Some.Show.S01E02.1080p.par2']
+    )
+
+    // Sent once more, finished already: left as it is.
+    const again = await BatchStore.store(rangeId, nodeId, {
+      ...batch,
+      from: 60_000,
+      next: 70_000,
+      releases: [{ ...finish, payload: encodeChunk([]), files: 0, segments: 0, bytes: 0 }],
+      candidates: [],
+      counts: [],
+      samples: [],
+      fragments: [],
+    })
+    assert.include(again, { releases: 1, newReleases: 0, finalized: 0 })
+    assert.include((await releasesOf())[0], { files: 2, chunks: 2 })
+    assert.equal(Number((await db.from('release_chunks').count('* as n').first())!.n), 2)
   })
 
-  test('a split 7z read in pieces keeps the tail of its highest volume only; RAR gets none', async ({ assert }) => {
-    // As the live tail reads it while posted: each piece queues a tail for the last volume it saw so far.
+  test('a split 7z keeps the tail of its highest volume only; RAR gets none', async ({
+    assert,
+  }) => {
     await storeIn(GROUP, 1, [
       ...upload(10, 'TsIgNBewuFbuwWy.7z.001', 2, '[1/4]', '2026-10-23T08:24:00Z'),
       ...upload(20, 'TsIgNBewuFbuwWy.7z.002', 2, '[2/4]', '2026-10-23T08:26:00Z'),
+      ...upload(30, 'TsIgNBewuFbuwWy.7z.003', 2, '[3/4]', '2026-10-23T08:28:00Z'),
+      ...upload(40, 'TsIgNBewuFbuwWy.7z.004', 2, '[4/4]', '2026-10-23T08:30:00Z'),
+      ...upload(
+        50,
+        'Some.Show.S01E03.part01.rar',
+        2,
+        'Some.Show.S01E03 [1/2]',
+        '2026-10-23T09:00:00Z'
+      ),
+      ...upload(
+        60,
+        'Some.Show.S01E03.part02.rar',
+        2,
+        'Some.Show.S01E03 [2/2]',
+        '2026-10-23T09:02:00Z'
+      ),
     ])
-    await storeIn(GROUP, 100_001, upload(100_010, 'TsIgNBewuFbuwWy.7z.004', 2, '[4/4]', '2026-10-23T08:30:00Z'))
-    // A lower volume posted late: its tail goes at once.
-    await storeIn(GROUP, 200_001, upload(200_010, 'TsIgNBewuFbuwWy.7z.003', 2, '[3/4]', '2026-10-23T08:28:00Z'))
-    await storeIn(GROUP, 300_001, upload(300_010, 'Some.Show.S01E03.part01.rar', 2, 'Some.Show.S01E03 [1/2]', '2026-10-23T09:00:00Z'))
-    await storeIn(GROUP, 400_001, upload(400_010, 'Some.Show.S01E03.part02.rar', 2, 'Some.Show.S01E03 [2/2]', '2026-10-23T09:02:00Z'))
     const candidates = await db.from('probe_candidates').orderBy('file').select('kind', 'file')
     assert.deepEqual(
       candidates.map((c) => [c.kind, c.file]),
@@ -154,43 +461,120 @@ test.group('BatchStore', (group) => {
     )
   })
 
-  test('the same upload in another group (multi-posted) is a release of its own', async ({ assert }) => {
-    await storeIn(GROUP, 1, upload(10, 'Some.Show.S01E01.1080p.mkv', 2, 'Some.Show.S01E01.1080p [1/1]', '2026-10-05T10:00:00Z'))
-    await storeIn('alt.binaries.other', 1, upload(10, 'Some.Show.S01E01.1080p.mkv', 2, 'Some.Show.S01E01.1080p [1/1]', '2026-10-05T10:01:00Z', 'alt.binaries.other'))
+  test('the same upload in another group (multi-posted) is a release of its own', async ({
+    assert,
+  }) => {
+    await storeIn(
+      GROUP,
+      1,
+      upload(
+        10,
+        'Some.Show.S01E01.1080p.mkv',
+        2,
+        'Some.Show.S01E01.1080p [1/1]',
+        '2026-10-05T10:00:00Z'
+      )
+    )
+    await storeIn(
+      'alt.binaries.other',
+      1,
+      upload(
+        10,
+        'Some.Show.S01E01.1080p.mkv',
+        2,
+        'Some.Show.S01E01.1080p [1/1]',
+        '2026-10-05T10:01:00Z',
+        'alt.binaries.other'
+      )
+    )
     const rows = await releasesOf()
     assert.lengthOf(rows, 2)
     assert.equal(rows[0].identity_key, rows[1].identity_key)
+    assert.notEqual(rows[0].release_key, rows[1].release_key)
   })
 
-  test('an upload spread over groups (other files in another group) is one release', async ({ assert }) => {
-    await storeIn(GROUP, 1, upload(10, 'Some.Album-WEB-2026-FTD.part1.rar', 2, 'Some.Album-WEB-2026-FTD [2/3]', '2026-10-05T10:00:00Z'))
-    await storeIn('alt.binaries.other', 1, upload(10, 'Some.Album-WEB-2026-FTD.part2.rar', 2, 'Some.Album-WEB-2026-FTD [3/3]', '2026-10-05T10:00:30Z', 'alt.binaries.other'))
-    const rows = await releasesOf()
-    assert.lengthOf(rows, 1)
-    assert.equal(rows[0].files, 2)
+  test('an unnamed release goes to the pending store with its probe targets, not into the catalog', async ({
+    assert,
+  }) => {
+    const { rangeId, nodeId } = await rangeIn(GROUP, 1)
+    const batch = batchOf(GROUP, 1, [
+      ...upload(10, 'q3B9OrpKiCxUnzgtUGnp.part01.rar', 4, '[1/2]', '2026-10-05T10:00:00Z'),
+      ...upload(20, 'q3B9OrpKiCxUnzgtUGnp.par2', 1, '[2/2]', '2026-10-05T10:01:00Z'),
+    ])
+    assert.lengthOf(batch.releases, 1)
+    assert.isFalse(batch.releases[0].readable)
+    const stored = await BatchStore.store(rangeId, nodeId, batch)
+    assert.include(stored, { releases: 1, newReleases: 0, candidates: 0, pending: 1 })
+    assert.equal(Number((await db.from('releases').count('* as n').first())!.n), 0)
+    assert.equal(Number((await db.from('probe_candidates').count('* as n').first())!.n), 0)
+    assert.equal(Number((await db.from('release_chunks').count('* as n').first())!.n), 0)
+    const pending = await db.from('pending_releases').firstOrFail()
+    assert.include(pending, {
+      files: 2,
+      segments: '5',
+      partial: false,
+      finished: true,
+      tool: 'other',
+      status: 0,
+      poster: 'p <p@x>',
+    })
+    assert.lengthOf(pending.release_key, 20)
+    assert.deepEqual(
+      pending.candidates.map((c: { kind: string; file: string }) => [c.kind, c.file]).sort(),
+      [
+        ['archive', 'q3B9OrpKiCxUnzgtUGnp.part01.rar'],
+        ['par2', 'q3B9OrpKiCxUnzgtUGnp.par2'],
+      ]
+    )
+    // Sent again as a later chunk of a tail release: its counts add up while it is open, not once finished.
+    const again = await BatchStore.store(rangeId, nodeId, {
+      ...batch,
+      from: batch.next,
+      next: batch.next + 1000,
+      releases: [{ ...batch.releases[0], files: 1, segments: 5, bytes: 100 }],
+      candidates: [],
+    })
+    assert.include(again, { pending: 0 })
+    assert.include(await db.from('pending_releases').firstOrFail(), { files: 2, segments: '5' })
   })
 
-  test('a release key taken meanwhile by another group\'s writer is skipped for the next number', async ({ assert }) => {
-    await storeIn(GROUP, 1, upload(10, 'Some.Show.S01E01.1080p.mkv', 2, 'Some.Show.S01E01.1080p [1/1]', '2026-10-05T10:00:00Z'))
-    const [first] = await releasesOf()
-    // As if another group's batch numbered the key at the same moment: key:2 exists, not counted for this key.
-    const [g] = await db.table('groups').insert({ name: 'alt.binaries.third', priority: 3 }).returning('id')
-    await db.table('releases').insert({ release_key: `${first.identity_key}:2`, identity_key: 'someone-else', group_id: g.id, name: 'x', source: 'subject', poster: 'p', groups: JSON.stringify([]), files: 0, segments: 0, bytes: 0, chunks: 0 })
-    await storeIn('alt.binaries.other', 1, upload(10, 'Some.Show.S01E01.1080p.mkv', 2, 'Some.Show.S01E01.1080p [1/1]', '2026-10-05T10:01:00Z', 'alt.binaries.other'))
-    const keys = (await releasesOf()).filter((r) => r.identity_key === first.identity_key).map((r) => r.release_key)
-    assert.deepEqual(keys, [first.identity_key, `${first.identity_key}:3`])
-  })
-
-  test('a re-upload is numbered after the highest number, not into the holes merges left', async ({ assert }) => {
-    for (let day = 0; day < 4; day++) {
-      await storeIn(GROUP, 1 + day * 100_000, upload(10 + day * 100_000, 'Some.Show.S01E01.1080p.mkv', 2, 'Some.Show.S01E01.1080p [1/1]', `2026-10-0${day * 2 + 1}T10:00:00Z`))
+  test("an album's members stay findable by their own names (release_texts kind member)", async ({
+    assert,
+  }) => {
+    const { rangeId, nodeId } = await rangeIn(GROUP, 1)
+    // Two scans the TypeScript indexer puts under their subject; the Rust edge's album of them is the same chunk
+    // with its name and source.
+    const batch = batchOf(GROUP, 1, [
+      ...upload(10, 'Scan_001.dat', 1, 'Some Scans [1/2]', '2026-10-05T10:00:00Z'),
+      ...upload(20, 'Scan_002.dat', 1, 'Some Scans [2/2]', '2026-10-05T10:01:00Z'),
+    ])
+    assert.lengthOf(batch.releases, 1)
+    const album: ReleaseChunk = {
+      ...batch.releases[0],
+      name: 'Scan_#.dat',
+      source: 'album',
+      fileTotal: null,
     }
-    const [first] = await releasesOf()
-    // Merged away: key:2 and key:3 gone, key and key:4 left.
-    await db.from('releases').whereIn('release_key', [`${first.identity_key}:2`, `${first.identity_key}:3`]).delete()
-    await storeIn(GROUP, 400_001, upload(400_010, 'Some.Show.S01E01.1080p.mkv', 2, 'Some.Show.S01E01.1080p [1/1]', '2026-10-09T10:00:00Z'))
-    const keys = (await releasesOf()).map((r) => r.release_key.split(':')[1] ?? '1')
-    assert.deepEqual(keys, ['1', '4', '5'])
+    const stored = await BatchStore.store(rangeId, nodeId, { ...batch, releases: [album] })
+    assert.include(stored, { releases: 1, newReleases: 1 })
+    const release = await db.from('releases').firstOrFail()
+    assert.include(release, {
+      name: 'Scan_#.dat',
+      source: 6,
+      files: 2,
+      file_total: null,
+      complete: null,
+    })
+    assert.deepEqual(
+      (
+        await db.from('release_texts').where('release_id', release.id).orderBy(['kind', 'text'])
+      ).map((t) => [t.kind, t.text]),
+      [
+        [1, 'Scan_#.dat'],
+        [6, 'Scan_001.dat'],
+        [6, 'Scan_002.dat'],
+      ]
+    )
   })
 
   test('a small release of sidecars only is marked for the sidecar fold', async ({ assert }) => {
@@ -199,7 +583,12 @@ test.group('BatchStore', (group) => {
       ...upload(1_000_010, 'Some.Tool.v1.zip', 1, '[01/01]', '2026-10-05T10:00:00Z'),
       ...upload(1_000_020, 'Some.Show.S01E01.part01.rar', 2, '[02/30]', '2026-10-05T10:00:00Z'),
     ])
-    const flags = new Map((await db.from('releases').select('name', 'sidecar_only')).map((r) => [r.name, r.sidecar_only]))
+    const flags = new Map(
+      (await db.from('releases').select('name', 'sidecar_only')).map((r) => [
+        r.name,
+        r.sidecar_only,
+      ])
+    )
     assert.isTrue(flags.get('Murdoch Mysteries S8D1'))
     assert.isFalse(flags.get('Some.Tool.v1'))
     // Over 1 MB: not looked at.

tests/functional/final_completeness.spec.ts

@@ -1,95 +0,0 @@
-import { test } from '@japa/runner'
-import db from '@adonisjs/lucid/services/db'
-import testUtils from '@adonisjs/core/services/test_utils'
-import BatchStore from '#services/batch_store'
-import RangeBlocks from '#services/range_blocks'
-import SearchService from '#services/search_service'
-import { RangeIndexer, postedSpan } from '#lib/edge/index'
-import { DEFAULT_RULES } from '#lib/rules/index'
-import type { Overview } from '#lib/nntp/index'
-
-const GROUP = 'alt.binaries.final'
-const at = (iso: string) => new Date(iso)
-
-test.group('Final completeness', (group) => {
-  group.each.setup(() => testUtils.db().withGlobalTransaction())
-
-  test("a batch's posting times, without the earliest and latest 1%, become the range's span", async ({ assert }) => {
-    assert.deepEqual(postedSpan([]), [null, null])
-    const dates = Array.from({ length: 200 }, (_, i) => 1_000_000 + i)
-    dates.push(5, 9_999_999_999)
-    // 202 dates: 2 trimmed at each end, so the two wild ones go.
-    assert.deepEqual(postedSpan(dates), [1_000_001, 1_000_198])
-
-    const [g] = await db.table('groups').insert({ name: GROUP, priority: 1 }).returning('id')
-    const [node] = await db.table('nodes').insert({ name: 'final-node' }).returning('id')
-    const [range] = await db
-      .table('ranges')
-      .insert({ group_id: g.id, start_id: 1, end_id: 1000, next_id: 1, status: 'leased', node_id: node.id, rule_version: 1 })
-      .returning('id')
-    const overviews = Array.from({ length: 10 }, (_, i) => {
-      const date = new Date(Date.parse('2026-01-01T10:00:00Z') + i * 60_000).toUTCString().replace('GMT', '+0000')
-      return [1 + i, { 'subject': `[1/1] - "Some.Thing.2026.rar" yEnc (${i + 1}/10)`, 'from': 'p <p@x>', 'date': date, 'message-id': `<f${i}@x>`, ':bytes': '1000', 'xref': `x ${GROUP}:${1 + i}` }] as Overview
-    })
-    const ix = new RangeIndexer({ group: GROUP, start: 1, end: 1000, rules: { ...DEFAULT_RULES, samplePercent: 0, droppedSignatures: [], priorities: { [GROUP]: 1 } } })
-    ix.add(overviews, 11)
-    await BatchStore.store(Number(range.id), Number(node.id), ix.take())
-    const stored = await db.from('ranges').where('id', range.id).firstOrFail()
-    assert.equal(new Date(stored.posted_low).toISOString(), '2026-01-01T10:00:00.000Z')
-    assert.equal(new Date(stored.posted_high).toISOString(), '2026-01-01T10:09:00.000Z')
-  })
-
-  test('a release is final once one block of done ranges covers its posting times and 6 hours either side', async ({ assert }) => {
-    const [g] = await db.table('groups').insert({ name: GROUP, priority: 1 }).returning('id')
-    const range = (start: number, status: string, low: string | null, high: string | null) =>
-      db.table('ranges').insert({ group_id: g.id, start_id: start, end_id: start + 99_999, next_id: start, status, posted_low: low && at(low), posted_high: high && at(high) })
-    await range(1, 'done', '2026-01-01T00:00:00Z', '2026-01-01T12:00:00Z')
-    await range(100_001, 'done', '2026-01-01T12:00:00Z', '2026-01-01T23:00:00Z')
-    await range(200_001, 'planned', null, null)
-    await range(300_001, 'done', '2026-01-02T10:00:00Z', '2026-01-02T20:00:00Z')
-    assert.equal(await RangeBlocks.rebuild(), 2)
-
-    let n = 0
-    const release = async (first: number, posted: string, extra: Record<string, unknown> = {}) => {
-      n++
-      const key = `fc${n}`.padEnd(40, '0')
-      const [row] = await db
-        .table('releases')
-        .insert({ release_key: key, identity_key: key, name: `Final.Test.${n}`, source: 'stem', groups: '[]', group_id: g.id, files: 3, file_total: 5, complete: false, first_article: first, last_article: first + 10, posted_at: at(posted), last_posted_at: at(posted), readable: true, settled_at: new Date(), ...extra })
-        .returning('id')
-      return Number(row.id)
-    }
-    const final = async (id: number) => {
-      const r = await db.from('releases').where('id', id).firstOrFail()
-      const { rows } = await db.rawQuery(`SELECT release_final(?, ?, ?, ?, ?, ?) AS final`, [r.group_id, r.first_article, r.last_article, r.posted_at, r.last_posted_at, r.complete])
-      return rows[0].final as boolean
-    }
-
-    // Mid-block: final either way.
-    const middle = await release(90_000, '2026-01-01T11:00:00Z')
-    // In the first range: nothing older to wait for, although 6 hours before it isn't covered.
-    const first = await release(1_000, '2026-01-01T02:00:00Z')
-    // Near the block's end, the next range unread: its missing files may be there.
-    const edge = await release(195_000, '2026-01-01T22:00:00Z')
-    // In the island after the gap: the gap is before it.
-    const island = await release(310_000, '2026-01-02T12:00:00Z')
-    assert.isTrue(await final(middle))
-    assert.isTrue(await final(first))
-    assert.isFalse(await final(edge))
-    // Complete there: every file found, nothing to wait for.
-    assert.isTrue(await final(await release(196_000, '2026-01-01T22:30:00Z', { files: 5, complete: true, name: 'Complete.Edge' })))
-    assert.isFalse(await final(island))
-
-    // Search: an incomplete release that isn't final isn't left out as incomplete; the API says which is final.
-    const shown = await SearchService.search({ q: 'Final.Test', notIncomplete: true })
-    assert.sameMembers(shown.releases.map((r) => r.id), [edge, island])
-    assert.isTrue(shown.releases.every((r) => r.final === false))
-
-    // The gap read: one block, the edge release final (the island's 12:00 + 6 h is inside 20:00).
-    await db.from('ranges').where('start_id', 200_001).update({ status: 'done', posted_low: at('2026-01-01T23:00:00Z'), posted_high: at('2026-01-02T10:00:00Z') })
-    assert.equal(await RangeBlocks.rebuild(), 1)
-    assert.isTrue(await final(edge))
-    assert.isTrue(await final(island))
-    assert.equal((await SearchService.search({ q: 'Final.Test', notIncomplete: true })).total, 0)
-  })
-})

tests/functional/fragment_merger.spec.ts

@@ -1,65 +0,0 @@
-import { test } from '@japa/runner'
-import db from '@adonisjs/lucid/services/db'
-import testUtils from '@adonisjs/core/services/test_utils'
-import BatchStore from '#services/batch_store'
-import FragmentMerger from '#services/fragment_merger'
-import { RangeIndexer } from '#lib/edge/index'
-import { decodeChunk } from '#lib/chunks/index'
-import { DEFAULT_RULES } from '#lib/rules/index'
-import type { Overview } from '#lib/nntp/index'
-
-const GROUP = 'alt.binaries.test'
-const rules = { ...DEFAULT_RULES, samplePercent: 0, droppedSignatures: [], priorities: { [GROUP]: 1 } }
-
-function parts(name: string, total: number, numbers: number[], at: number): Overview[] {
-  return numbers.map((n, i) => [at + i, {
-    'subject': `Big.Upload.2020 [1/1] - "${name}" yEnc (${n}/${total})`, 'from': 'p <p@x>', 'date': 'Mon, 05 Oct 2026 10:00:00 +0000',
-    'message-id': `<${name}-${n}@test>`, ':bytes': '700000', 'xref': `news.example ${GROUP}:${at + i}`,
-  }] as Overview)
-}
-
-/** Indexes one range's overviews and stores them as the range's only batch; the range ends done. */
-async function indexRange(groupId: number, nodeId: number, start: number, end: number, overviews: Overview[]) {
-  const [r] = await db.table('ranges').insert({ group_id: groupId, start_id: start, end_id: end, next_id: start, status: 'leased', node_id: nodeId }).returning('id')
-  const indexer = new RangeIndexer({ group: GROUP, start, end, rules, edgeArticles: 100 })
-  indexer.add(overviews, end + 1)
-  indexer.finish()
-  await BatchStore.store(Number(r.id), nodeId, indexer.take())
-  await db.from('ranges').where('id', r.id).update({ status: 'done' })
-  return Number(r.id)
-}
-
-test.group('FragmentMerger', (group) => {
-  group.each.setup(() => testUtils.db().withGlobalTransaction())
-
-  test('a file cut by a range edge is completed from both sides once both ranges are done', async ({ assert }) => {
-    const [g] = await db.table('groups').insert({ name: GROUP, priority: 1, low: 1 }).returning('id')
-    const [n] = await db.table('nodes').insert({ name: 'merge-node' }).returning('id')
-    // Parts 1–2 at the end of the first range (still open there), 3–4 at the start of the next.
-    await indexRange(g.id, n.id, 1, 1000, parts('big.part1.rar', 4, [1, 2], 990))
-    assert.equal((await db.from('fragments').count('* as c'))[0].c, '1')
-    assert.include(await FragmentMerger.run(), { completed: 0, incomplete: 0, waiting: 1 })
-
-    await indexRange(g.id, n.id, 1001, 2000, parts('big.part1.rar', 4, [3, 4], 1001))
-    assert.include(await FragmentMerger.run(), { completed: 1, incomplete: 0 })
-    assert.equal((await db.from('fragments').count('* as c'))[0].c, '0')
-    const release = await db.from('releases').first()
-    assert.include(release, { files: 1, chunks: 1 })
-    const chunk = await db.from('release_chunks').where('release_id', release.id).first()
-    assert.deepEqual(decodeChunk(chunk.payload)[0].segments.map((s) => s[0]), [1, 2, 3, 4])
-    assert.deepEqual((await db.from('probe_candidates')).map((c) => c.kind), ['archive'])
-  })
-
-  test('a file still missing parts when all its neighbours are done is dropped', async ({ assert }) => {
-    const [g] = await db.table('groups').insert({ name: GROUP, priority: 1, low: 1 }).returning('id')
-    const [n] = await db.table('nodes').insert({ name: 'merge-node' }).returning('id')
-    await indexRange(g.id, n.id, 1, 1000, parts('gap.part1.rar', 5, [1, 2], 990))
-    await indexRange(g.id, n.id, 1001, 2000, parts('gap.part1.rar', 5, [4, 5], 1001))
-    // Its later neighbour isn't there yet: the missing part could still come.
-    assert.include(await FragmentMerger.run(), { completed: 0, incomplete: 0, waiting: 1 })
-    await indexRange(g.id, n.id, 2001, 3000, [])
-    assert.include(await FragmentMerger.run(), { completed: 0, incomplete: 1 })
-    assert.equal((await db.from('fragments').count('* as c'))[0].c, '0')
-    assert.lengthOf(await db.from('releases'), 0)
-  })
-})

tests/functional/helpers/releases.ts

@@ -0,0 +1,57 @@
+import db from '@adonisjs/lucid/services/db'
+import CatalogRefs from '#services/catalog_refs'
+import { identityKey, releaseKey } from '#lib/release/keys'
+import { sourceCode } from '#lib/release/source'
+
+/** A release as the tests describe one: text keys, a poster and groups by name, the source by name. */
+export interface ReleaseFixture {
+  /** Any text; the stored key is its hash (keyOf), or 40 hex characters decoded. Default: from the identity, group and first article. */
+  release_key?: string
+  /** Any text (default: release_key, else the name). */
+  identity_key?: string
+  name: string
+  source?: string
+  poster?: string | null
+  groups?: string[] | string
+  group_id?: number | null
+  first_article?: number | null
+  [column: string]: unknown
+}
+
+/** The bytes a text key is stored as (identityKey: 40 hex characters decoded, anything else hashed). */
+export const keyOf = identityKey
+
+/** Inserts releases with their poster, group set, keys and source as the catalog stores them; their ids in order. */
+export async function insertReleases(fixtures: ReleaseFixture[]): Promise<number[]> {
+  const ids: number[] = []
+  for (const f of fixtures) {
+    const { release_key: textKey, identity_key: textIdentity, source, poster, groups, ...rest } = f
+    const identity = textIdentity ?? textKey ?? f.name
+    const groupList: string[] = typeof groups === 'string' ? JSON.parse(groups) : (groups ?? [])
+    const [row] = await db
+      .table('releases')
+      .insert({
+        ...rest,
+        release_key: textKey
+          ? identityKey(textKey)
+          : releaseKey(identity, f.group_id ?? null, f.first_article ?? null),
+        identity_key: identityKey(identity),
+        source: sourceCode(source ?? 'stem'),
+        poster_id: poster ? await CatalogRefs.posterId(db, poster) : null,
+        group_set_id: await CatalogRefs.groupSetId(db, groupList),
+      })
+      .returning('id')
+    const id = Number(row.id)
+    await db.rawQuery(
+      `INSERT INTO release_texts (release_id, kind, text) VALUES (?, 1, left(?, 255)) ON CONFLICT DO NOTHING`,
+      [id, f.name]
+    )
+    ids.push(id)
+  }
+  return ids
+}
+
+export async function insertRelease(fixture: ReleaseFixture): Promise<number> {
+  const [id] = await insertReleases([fixture])
+  return id
+}

tests/functional/node_api.spec.ts

@@ -77,7 +77,9 @@ test.group('Node API', (group) => {
     // The same batch again: refused.
     ;(await client.post(`/api/v1/node/ranges/${rangeId}/batches`).bearerToken(token).json(batch)).assertStatus(409)
 
-    assert.deepEqual((await db.from('rule_counts').where('range_id', rangeId)).map((c) => [c.rule, c.key, Number(c.articles)]), [['obfuscated', '', 3]])
+    // Counts by the range's block (a range without one is a block of its own, until blocks are leased).
+    const { block_id: blockId } = await db.from('ranges').where('id', rangeId).firstOrFail()
+    assert.deepEqual((await db.from('rule_counts').where('block_id', blockId)).map((c) => [c.rule, c.key, Number(c.articles)]), [['obfuscated', '', 3]])
 
     ;(await client.post(`/api/v1/node/ranges/${rangeId}/finish`).bearerToken(token)).assertStatus(200)
     const range = await db.from('ranges').where('id', rangeId).first()

tests/functional/par2_completeness.spec.ts

@@ -45,23 +45,23 @@ test.group('par2 completeness', (group) => {
     assert.isFalse(par2Complete(set, [{ name: 'Qx81.bin', size: 500 }, { name: 'Zr02.bin', size: 12 }], []))
   })
 
-  test('decided once settled with a par2 set, and kept when more files come', async ({ assert }) => {
+  test('decided once finished, by the par2 set the release holds', async ({ assert }) => {
     await store(1, [...file('Some.Upload.mkv', 3, 10), ...file('Some.Upload.par2', 1, 20)])
     const release = await db.from('releases').firstOrFail()
     assert.isNull(release.file_total)
     await db.table('par2_files').multiInsert([setFile('Some.Upload.mkv', 2_000_000), setFile('Some.Upload.nfo', 100)].map((f, i) => ({ set_id: 's'.repeat(32), file_id: String(i).repeat(32), ...f })))
     await db.table('par2_sets').insert({ set_id: 's'.repeat(32), release_id: release.id, files: 2 })
-    // Not settled yet: left alone.
+    // A provisional tail release (not finished yet): left alone.
+    await db.from('releases').where('id', release.id).update({ settled_at: null })
     assert.equal(await decidePar2Completeness(db, [release.id]), 0)
     await db.from('releases').where('id', release.id).update({ settled_at: new Date() })
     assert.equal(await decidePar2Completeness(db, [release.id]), 1)
     assert.isFalse((await db.from('releases').where('id', release.id).firstOrFail()).complete)
-    // The nfo comes in a later range: complete, and the batch store's own update keeps it.
-    await store(1001, file('Some.Upload.nfo', 1, 1010))
-    await decidePar2Completeness(db, [release.id])
-    await store(2001, file('Some.Upload.sfv', 1, 2010))
-    const after = await db.from('releases').where('id', release.id).firstOrFail()
-    assert.equal(after.files, 4)
-    assert.isTrue(after.complete)
+    // Another upload with the nfo: complete.
+    await store(1001, [...file('Some.Upload.mkv', 3, 1010), ...file('Some.Upload.par2', 1, 1020), ...file('Some.Upload.nfo', 1, 1030)])
+    const whole = await db.from('releases').whereNot('id', release.id).firstOrFail()
+    await db.table('par2_sets').insert({ set_id: 's'.repeat(32), release_id: whole.id, files: 2 })
+    assert.equal(await decidePar2Completeness(db, [whole.id]), 1)
+    assert.isTrue((await db.from('releases').where('id', whole.id).firstOrFail()).complete)
   })
 })

tests/functional/par2_merger.spec.ts

@@ -4,6 +4,7 @@ import testUtils from '@adonisjs/core/services/test_utils'
 import { Par2Merger } from '#services/par2_merger'
 import { encodeChunk } from '#lib/chunks/index'
 import ChunkPartitions from '#services/chunk_partitions'
+import { insertRelease } from '#tests/functional/helpers/releases'
 
 const hash = (c: string) => c.repeat(32)
 
@@ -13,14 +14,11 @@ test.group('Par2Merger', (group) => {
   test('merges the one-file releases of an obfuscated upload by its par2 set, named by the par2', async ({ assert }) => {
     const at = new Date('2022-07-08T03:46:00Z')
     const release = async (key: string, posted: Date, groupId: number, files = 1) => {
-      const [row] = await db
-        .table('releases')
-        .insert({
+      const row = { id: await insertRelease({
           release_key: key.padEnd(40, '0'), identity_key: key.padEnd(40, '0'), name: key, source: 'stem', groups: '[]', group_id: groupId,
           files, segments: files * 10, bytes: files * 1000, first_article: 100, last_article: 200, posted_at: posted, readable: false, file_total: 20,
           settled_at: posted,
-        })
-        .returning('id')
+        }) }
       return Number(row.id)
     }
     // The par2 index, two data files of the upload, the same file reposted a week later, and one in another group.
@@ -95,10 +93,7 @@ test.group('Par2Merger', (group) => {
     const at = new Date('2022-07-08T03:46:00Z')
     const ids: number[] = []
     for (const key of ['dd000001', 'dd000002']) {
-      const [row] = await db
-        .table('releases')
-        .insert({ release_key: key.padEnd(40, '0'), identity_key: key.padEnd(40, '0'), name: key, source: 'stem', groups: '[]', group_id: 1, files: 1, posted_at: at, readable: false })
-        .returning('id')
+      const row = { id: await insertRelease({ release_key: key.padEnd(40, '0'), identity_key: key.padEnd(40, '0'), name: key, source: 'stem', groups: '[]', group_id: 1, files: 1, posted_at: at, readable: false }) }
       ids.push(Number(row.id))
     }
     // The par2 is held by the first; the second holds the file, and so do 20 releases elsewhere and years apart.
@@ -120,10 +115,7 @@ test.group('Par2Merger', (group) => {
     const at = new Date('2022-07-08T03:46:00Z')
     const ids: number[] = []
     for (const [key, files] of [['bb000001', 1], ['bb000002', 1], ['bb000003', 3]] as const) {
-      const [row] = await db
-        .table('releases')
-        .insert({ release_key: key.padEnd(40, '0'), identity_key: key.padEnd(40, '0'), name: key, source: 'stem', groups: '[]', group_id: 1, files, posted_at: at, readable: false })
-        .returning('id')
+      const row = { id: await insertRelease({ release_key: key.padEnd(40, '0'), identity_key: key.padEnd(40, '0'), name: key, source: 'stem', groups: '[]', group_id: 1, files, posted_at: at, readable: false }) }
       ids.push(Number(row.id))
     }
     const setId = hash('7')
@@ -145,10 +137,7 @@ test.group('Par2Merger', (group) => {
     const ids: number[] = []
     for (let i = 1; i <= 6; i++) {
       const key = `cc00000${i}`
-      const [row] = await db
-        .table('releases')
-        .insert({ release_key: key.padEnd(40, '0'), identity_key: key.padEnd(40, '0'), name: `kantodo${i}`, source: 'stem', groups: '[]', group_id: 1, files: 11, posted_at: at, readable: true })
-        .returning('id')
+      const row = { id: await insertRelease({ release_key: key.padEnd(40, '0'), identity_key: key.padEnd(40, '0'), name: `kantodo${i}`, source: 'stem', groups: '[]', group_id: 1, files: 11, posted_at: at, readable: true }) }
       ids.push(Number(row.id))
       await db.table('par2_sets').insert({ set_id: setId, release_id: row.id, files: 1 })
     }
@@ -163,10 +152,7 @@ test.group('Par2Merger', (group) => {
     const release = async (poster: string, posted: Date, total: number, inner: string | null, groupId = 1) => {
       n++
       const key = `dd${String(n).padStart(6, '0')}`
-      const [row] = await db
-        .table('releases')
-        .insert({ release_key: key.padEnd(40, '0'), identity_key: key.padEnd(40, '0'), name: `hex${n}`, source: 'stem', groups: '[]', group_id: groupId, files: 1, posted_at: posted, readable: false, file_total: total, poster })
-        .returning('id')
+      const row = { id: await insertRelease({ release_key: key.padEnd(40, '0'), identity_key: key.padEnd(40, '0'), name: `hex${n}`, source: 'stem', groups: '[]', group_id: groupId, files: 1, posted_at: posted, readable: false, file_total: total, poster }) }
       if (inner) {
         await db.table('release_inspections').insert({ release_id: row.id, candidate_id: 5000 + n, kind: 'lone', result: JSON.stringify({ yencName: inner, type: '', size: 1 }) })
       }
@@ -202,10 +188,7 @@ test.group('Par2Merger', (group) => {
     const ids: number[] = []
     for (let i = 1; i <= 3; i++) {
       const key = `ee00000${i}`
-      const [row] = await db
-        .table('releases')
-        .insert({ release_key: key.padEnd(40, '0'), identity_key: key.padEnd(40, '0'), name: key, source: 'stem', groups: '[]', group_id: 1, files: 10, posted_at: at, readable: false, file_total: 5, poster: 'p' })
-        .returning('id')
+      const row = { id: await insertRelease({ release_key: key.padEnd(40, '0'), identity_key: key.padEnd(40, '0'), name: key, source: 'stem', groups: '[]', group_id: 1, files: 10, posted_at: at, readable: false, file_total: 5, poster: 'p' }) }
       ids.push(Number(row.id))
       await db.table('release_inspections').insert({ release_id: row.id, candidate_id: 6000 + i, kind: 'lone', result: JSON.stringify({ yencName: `${stem}.7z.00${i}` }) })
     }
@@ -217,10 +200,7 @@ test.group('Par2Merger', (group) => {
     const at = new Date('2022-07-08T03:46:00Z')
     const ids: number[] = []
     for (const key of ['aa000001', 'aa000002', 'aa000003']) {
-      const [row] = await db
-        .table('releases')
-        .insert({ release_key: key.padEnd(40, '0'), identity_key: key.padEnd(40, '0'), name: key, source: 'stem', groups: '[]', group_id: 1, files: 1, posted_at: at, readable: false })
-        .returning('id')
+      const row = { id: await insertRelease({ release_key: key.padEnd(40, '0'), identity_key: key.padEnd(40, '0'), name: key, source: 'stem', groups: '[]', group_id: 1, files: 1, posted_at: at, readable: false }) }
       ids.push(Number(row.id))
     }
     // Two test uploads whose 1 GiB files start alike, and a set describing one such file, held by the third.
@@ -242,13 +222,10 @@ test.group('Par2Merger', (group) => {
     const release = async (name: string, files: number, total: number | null, posted: Date, chunkFiles: string[]) => {
       n++
       const key = `ff${String(n).padStart(6, '0')}`.padEnd(40, '0')
-      const [row] = await db
-        .table('releases')
-        .insert({ release_key: key, identity_key: key, name, source: 'stem', groups: '[]', group_id: 1, files, file_total: total, poster, posted_at: posted, readable: true, settled_at: posted })
-        .returning('id')
+      const row = { id: await insertRelease({ release_key: key, identity_key: key, name, source: 'stem', groups: '[]', group_id: 1, files, file_total: total, poster, posted_at: posted, readable: true, settled_at: posted }) }
       const id = Number(row.id)
       const payload = encodeChunk(chunkFiles.map((f, i) => ({ name: f, poster, subject: `${name} [${String(i).padStart(2, '0')}/${total ?? 0}] - "${f}" yEnc`, postedAt: posted.getTime(), groups: ['a.b.test'], segments: [[1, `${id}-${i}@x`, 1000]] })))
-      await db.table('release_chunks').insert({ release_id: id, range_id: 1, batch_from: 1, files: chunkFiles.length, segments: chunkFiles.length, payload })
+      await db.table('release_chunks').insert({ release_id: id, block_id: 1, batch_from: 1, files: chunkFiles.length, segments: chunkFiles.length, payload })
       await db.table('fold_candidates').insert({ release_id: id }).onConflict('release_id').ignore()
       return id
     }

tests/functional/par2_name_merge.spec.ts

@@ -48,7 +48,7 @@ test.group('Par2Merger / by file names', (group) => {
       ...file('Track One.flac', 100_060, 5, 'other <o@x>'),
     ])
     await store(file('Track Two.flac', 200_070, 24 * 60 + 10))
-    const ids = Object.fromEntries((await db.from('releases').select('id', 'name', 'poster', 'posted_at')).map((r) => [`${r.name}${r.poster.startsWith('other') ? ' (other)' : ''}${new Date(r.posted_at).getUTCDate() === 21 ? ' (later)' : ''}`, Number(r.id)]))
+    const ids = Object.fromEntries((await db.from('releases as r').leftJoin('posters as p', 'p.id', 'r.poster_id').select('r.id', 'r.name', 'p.poster', 'r.posted_at')).map((r) => [`${r.name}${r.poster.startsWith('other') ? ' (other)' : ''}${new Date(r.posted_at).getUTCDate() === 21 ? ' (later)' : ''}`, Number(r.id)]))
     const index = ids['Some.Album-WEB-2026']
     const setId = 'a'.repeat(32)
     await db.table('par2_files').multiInsert(
@@ -57,7 +57,7 @@ test.group('Par2Merger / by file names', (group) => {
     await db.table('par2_sets').insert({ set_id: setId, release_id: index, files: 3 })
 
     assert.equal(await new Par2Merger().mergeByName(setId), 3)
-    const left = (await db.from('releases').orderBy('name').select('name', 'files', 'poster')).map((r) => `${r.name} ${r.files}${r.poster.startsWith('other') ? ' (other)' : ''}`)
+    const left = (await db.from('releases as r').leftJoin('posters as p', 'p.id', 'r.poster_id').orderBy('r.name').select('r.name', 'r.files', 'p.poster')).map((r) => `${r.name} ${r.files}${r.poster.startsWith('other') ? ' (other)' : ''}`)
     assert.includeMembers(left, ['Another Song 1', 'Track One 1 (other)', 'Track Two 1'])
     assert.lengthOf(left, 4)
     const merged = await db.from('release_merges').where('via', 'name').count('* as n')

tests/functional/par2_pruner.spec.ts

@@ -2,6 +2,7 @@ import { test } from '@japa/runner'
 import db from '@adonisjs/lucid/services/db'
 import testUtils from '@adonisjs/core/services/test_utils'
 import { Par2Pruner } from '#services/par2_pruner'
+import { insertRelease } from '#tests/functional/helpers/releases'
 
 test.group('Par2Pruner', (group) => {
   group.each.setup(() => testUtils.db().withGlobalTransaction())
@@ -12,10 +13,7 @@ test.group('Par2Pruner', (group) => {
     const release = async () => {
       n++
       const key = `pp${String(n).padStart(6, '0')}`.padEnd(40, '0')
-      const [row] = await db
-        .table('releases')
-        .insert({ release_key: key, identity_key: key, name: key, source: 'stem', groups: '[]', group_id: 1, files: 2, readable: false })
-        .returning('id')
+      const row = { id: await insertRelease({ release_key: key, identity_key: key, name: key, source: 'stem', groups: '[]', group_id: 1, files: 2, readable: false }) }
       return Number(row.id)
     }
     let candidate = 5000
@@ -59,12 +57,12 @@ test.group('Par2Pruner', (group) => {
     const posted = new Date('2024-03-01T12:00:00Z')
     const [group] = await db.table('groups').insert({ name: 'alt.binaries.pruned', priority: 1 }).returning('id')
     const groupId = Number(group.id)
-    await db.from('releases').where('id', f).update({ group_id: groupId, posted_at: posted, last_posted_at: posted, first_article: 5000, last_article: 5100 })
+    // Finished (the window closed on it) long ago; its range done since.
+    await db.from('releases').where('id', f).update({ group_id: groupId, posted_at: posted, last_posted_at: posted, first_article: 5000, last_article: 5100, settled_at: old })
     const [range] = await db
       .table('ranges')
       .insert({ group_id: groupId, start_id: 1, end_id: 10000, next_id: 10001, status: 'done', done_at: old, posted_low: new Date('2024-02-28T00:00:00Z'), posted_high: new Date('2024-03-03T00:00:00Z') })
       .returning('id')
-    await db.table('range_blocks').insert({ group_id: groupId, start_id: 1, end_id: 10000, posted_low: new Date('2024-02-28T00:00:00Z'), posted_high: new Date('2024-03-03T00:00:00Z'), at_group_start: false })
     // A lone candidate unprobed since before the range was done: its hash may still match. Kept.
     await db.table('probe_candidates').insert({ id: 9001, release_id: b, kind: 'lone', message_id: '<9001@x>', file: 'x', status: 'waiting', created_at: new Date(old.getTime() - 3600_000) })
 
@@ -87,7 +85,7 @@ test.group('Par2Pruner', (group) => {
     // A hash of its files since the merger's last window: the merger hasn't matched it yet. Kept.
     const g = await release()
     const unseen = await set('g', g, 1, null)
-    await db.from('releases').where('id', g).update({ group_id: groupId, posted_at: posted, last_posted_at: posted, first_article: 6000, last_article: 6100 })
+    await db.from('releases').where('id', g).update({ group_id: groupId, posted_at: posted, last_posted_at: posted, first_article: 6000, last_article: 6100, settled_at: old })
     await db.table('worker_cursors').insert({ name: 'merger', at: new Date(Date.now() - 3600_000) })
     await db.table('probe_candidates').insert({ id: 9002, release_id: await release(), kind: 'lone', message_id: '<9002@x>', file: 'x', status: 'done' })
     await db.table('file_hashes').insert({ candidate_id: 9002, release_id: 1, hash16k: 'g0'.padEnd(32, '0'), size: 1000 })
@@ -105,7 +103,7 @@ test.group('Par2Pruner', (group) => {
     let candidate = 7000
     for (const letter of ['p', 'q', 'r']) {
       const key = `pr${letter}`.padEnd(40, '0')
-      const [row] = await db.table('releases').insert({ release_key: key, identity_key: key, name: key, source: 'stem', groups: '[]', group_id: 1, files: 1, readable: false }).returning('id')
+      const row = { id: await insertRelease({ release_key: key, identity_key: key, name: key, source: 'stem', groups: '[]', group_id: 1, files: 1, readable: false }) }
       const id = Number(row.id)
       const setId = letter.repeat(32)
       const h = `${letter}0`.padEnd(32, '0')

tests/functional/pending_ladder.spec.ts

@@ -0,0 +1,208 @@
+import { test } from '@japa/runner'
+import db from '@adonisjs/lucid/services/db'
+import testUtils from '@adonisjs/core/services/test_utils'
+import PendingLadder, { PENDING, byEffort, type PendingProbe } from '#services/pending_ladder'
+import { identityKey, releaseKey } from '#lib/release/keys'
+
+let groupId = 0
+let n = 0
+
+async function pending(
+  name: string,
+  candidates: { kind: string; messageId: string; file: string }[],
+  extra: Record<string, unknown> = {}
+) {
+  n++
+  const key = releaseKey(name, groupId, n)
+  await db.table('pending_releases').insert({
+    release_key: key,
+    identity_key: identityKey(name),
+    group_id: groupId,
+    name,
+    poster: 'Bob <bob@home.me>',
+    first_article: n,
+    last_article: n,
+    files: 1,
+    segments: 10,
+    bytes: 1000,
+    tool: 'other',
+    candidates: JSON.stringify(candidates),
+    ...extra,
+  })
+  return key.toString('hex')
+}
+
+const inspection = (yencName: string, names?: string[]) => ({
+  yencName,
+  size: 1000,
+  type: null,
+  names,
+  entries: names?.length,
+})
+
+test.group('Pending ladder', (group) => {
+  group.each.setup(async () => {
+    const rollback = await testUtils.db().withGlobalTransaction()
+    groupId = Number(
+      (
+        await db
+          .table('groups')
+          .insert({ name: 'alt.binaries.pending', priority: 1 })
+          .returning('id')
+      )[0].id
+    )
+    n = 0
+    return rollback
+  })
+
+  test('targets are climbed by effort: a par2 before an archive before a lone file', ({
+    assert,
+  }) => {
+    assert.deepEqual(
+      byEffort([
+        { kind: 'lone', messageId: 'l', file: 'l' },
+        { kind: '7z-tail', messageId: 't', file: 't' },
+        { kind: 'archive', messageId: 'a', file: 'a' },
+        { kind: 'par2', messageId: 'p', file: 'p' },
+      ]).map((t) => t.kind),
+      ['par2', 'archive', 'lone', '7z-tail']
+    )
+  })
+
+  test('a claim takes the oldest waiting releases, their cheapest target first; outcomes move them up or to the end', async ({
+    assert,
+  }) => {
+    const twoTargets = await pending('a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4', [
+      { kind: 'archive', messageId: '<a@x>', file: 'x.rar' },
+      { kind: 'par2', messageId: '<p@x>', file: 'x.par2' },
+    ])
+    const lone = await pending('f6e5d4c3b2a1f6e5d4c3b2a1f6e5d4c3', [
+      { kind: 'lone', messageId: '<l@x>', file: 'f6e5d4c3b2a1f6e5d4c3b2a1f6e5d4c3' },
+    ])
+    const none = await pending('00000000000000000000000000000001', [])
+    const provisional = await pending(
+      '00000000000000000000000000000002',
+      [{ kind: 'lone', messageId: '<v@x>', file: 'v' }],
+      { finished: false }
+    )
+
+    const probes = await PendingLadder.claim(10)
+    assert.deepEqual(
+      probes.map((p) => [p.releaseKey, p.kind, p.tries, p.targets]),
+      [
+        [twoTargets, 'par2', 0, 2],
+        [lone, 'lone', 0, 1],
+      ]
+    )
+    const statusOf = async (key: string) =>
+      (await db
+        .from('pending_releases')
+        .where('release_key', Buffer.from(key, 'hex'))
+        .firstOrFail()) as any
+    assert.equal((await statusOf(twoTargets)).status, PENDING.taken)
+    assert.equal((await statusOf(none)).status, PENDING.waiting)
+    assert.equal((await statusOf(provisional)).status, PENDING.waiting)
+
+    // The par2 names only random volumes: the next target. The lone file's yEnc name is its random name: exhausted.
+    await PendingLadder.record([
+      {
+        probe: probes[0],
+        status: 'done',
+        inspection: inspection('x.par2', ['a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4.part01.rar']),
+      },
+      {
+        probe: probes[1],
+        status: 'done',
+        inspection: inspection('f6e5d4c3b2a1f6e5d4c3b2a1f6e5d4c3'),
+      },
+    ])
+    const first = await statusOf(twoTargets)
+    assert.include(first, { status: PENDING.waiting, tries: 1, named: null })
+    assert.lengthOf(first.inspections, 1)
+    assert.include(await statusOf(lone), { status: PENDING.exhausted, tries: 1 })
+
+    // The archive's listing names it: for the namer.
+    const [second] = await PendingLadder.claim(10)
+    assert.deepEqual([second.releaseKey, second.kind, second.tries], [twoTargets, 'archive', 1])
+    await PendingLadder.record([
+      {
+        probe: second,
+        status: 'done',
+        inspection: inspection('x.rar', ['Some.Movie.2016.1080p.mkv']),
+      },
+    ])
+    assert.include(await statusOf(twoTargets), {
+      status: PENDING.naming,
+      named: 'Some.Movie.2016.1080p.mkv',
+      tries: 0,
+    })
+
+    // The signature counted both ways: the poster's own and the tool's in the group.
+    const signatures = await db.from('pending_signatures').orderBy('poster')
+    assert.deepEqual(
+      signatures.map((s) => [s.tool, s.poster, s.probed, s.named, s.dropped]),
+      [
+        ['other', '', 2, 1, false],
+        ['other', 'Bob <bob@home.me>', 2, 1, false],
+      ]
+    )
+  })
+
+  test('a signature that names nothing is dropped, and its waiting releases discarded without a probe', async ({
+    assert,
+  }) => {
+    // 100 climbs of the tool in the group, none named.
+    await db
+      .table('pending_signatures')
+      .insert({ tool: 'other', group_id: groupId, poster: '', probed: 99, named: 0 })
+    const last = await pending('00000000000000000000000000000009', [
+      { kind: 'lone', messageId: '<z@x>', file: 'z' },
+    ])
+    const [probe] = await PendingLadder.claim(1)
+    assert.equal(probe.releaseKey, last)
+    await PendingLadder.record([{ probe, status: 'missing' }])
+    const signature = await db
+      .from('pending_signatures')
+      .where({ tool: 'other', group_id: groupId, poster: '' })
+      .firstOrFail()
+    assert.include(signature, { probed: 100, named: 0, dropped: true })
+
+    const waiting = await pending('00000000000000000000000000000010', [
+      { kind: 'lone', messageId: '<w@x>', file: 'w' },
+    ])
+    const other = await pending(
+      '00000000000000000000000000000011',
+      [{ kind: 'lone', messageId: '<o@x>', file: 'o' }],
+      { tool: 'nyuu' }
+    )
+    // samplePercent 0 in the test rules: no explored sample.
+    await db.table('rule_sets').insert({
+      version: 99,
+      rules: JSON.stringify({ samplePercent: 0, droppedSignatures: [] }),
+      note: 'test',
+    })
+    assert.equal(await PendingLadder.discard(), 1)
+    const statusOf = async (key: string) =>
+      (
+        await db
+          .from('pending_releases')
+          .where('release_key', Buffer.from(key, 'hex'))
+          .firstOrFail()
+      ).status
+    assert.equal(await statusOf(waiting), PENDING.discarded)
+    assert.equal(await statusOf(other), PENDING.waiting)
+  })
+
+  test('a probe given back goes to the queue again at the same step', async ({ assert }) => {
+    const key = await pending('00000000000000000000000000000020', [
+      { kind: 'lone', messageId: '<g@x>', file: 'g' },
+    ])
+    const [probe] = await PendingLadder.claim(1)
+    await PendingLadder.record([{ probe: probe as PendingProbe, status: 'waiting' }])
+    assert.include(
+      await db.from('pending_releases').where('release_key', Buffer.from(key, 'hex')).firstOrFail(),
+      { status: PENDING.waiting, tries: 0 }
+    )
+    assert.lengthOf(await db.from('pending_signatures'), 0)
+  })
+})

tests/functional/pending_namer.spec.ts

@@ -0,0 +1,143 @@
+import { test } from '@japa/runner'
+import db from '@adonisjs/lucid/services/db'
+import testUtils from '@adonisjs/core/services/test_utils'
+import { mkdtemp, mkdir, writeFile, rm } from 'node:fs/promises'
+import { tmpdir } from 'node:os'
+import { join } from 'node:path'
+import { deflateSync } from 'node:zlib'
+import { PendingNamer } from '#services/pending_namer'
+import { PENDING } from '#services/pending_ladder'
+import { RangeIndexer } from '#lib/edge/index'
+import { DEFAULT_RULES } from '#lib/rules/index'
+import { decodeChunk } from '#lib/chunks/index'
+import { identityKey, releaseKey } from '#lib/release/keys'
+
+const GROUP = 'alt.binaries.namer'
+
+/** One overview line as XOVER gives it (the corpus stores the reply's text). */
+function line(n: number, subject: string, minutes: number) {
+  const date = new Date(Date.UTC(2016, 0, 4, 10, minutes)).toUTCString()
+  return `${n}\t${subject}\tp <p@x>\t${date}\t<u${n}@test>\t\t1000\t10\tXref: news.example ${GROUP}:${n}`
+}
+
+test.group('Pending namer', (group) => {
+  let dir = ''
+  group.each.setup(async () => {
+    const rollback = await testUtils.db().withGlobalTransaction()
+    dir = await mkdtemp(join(tmpdir(), 'namer-'))
+    return rollback
+  })
+  group.each.teardown(() => rm(dir, { recursive: true, force: true }))
+
+  test('a named pending release is read again from the corpus and written to the catalog with its chunk and candidates', async ({
+    assert,
+  }) => {
+    const groupId = Number(
+      (await db.table('groups').insert({ name: GROUP, priority: 1 }).returning('id'))[0].id
+    )
+    // The range stored in the corpus: one chunk with a random-named upload (4 parts and its par2) among others.
+    const lines = [
+      line(5, 'Some.Show [1/1] - "Some.Show.mkv" yEnc (1/1)', 0),
+      ...[1, 2, 3, 4].map((i) =>
+        line(9 + i, `[1/2] - "q3B9OrpKiCxUnzgtUGnp.part01.rar" yEnc (${i}/4)`, i)
+      ),
+      line(14, '[2/2] - "q3B9OrpKiCxUnzgtUGnp.par2" yEnc (1/1)', 5),
+      line(20, 'Other.Thing [1/1] - "Other.Thing.mkv" yEnc (1/1)', 6),
+    ]
+    await mkdir(join(dir, GROUP, '1-200000'), { recursive: true })
+    await writeFile(
+      join(dir, GROUP, '1-200000', '0000-1-100000.gz'),
+      deflateSync(Buffer.from(lines.join('\r\n') + '\r\n', 'latin1'))
+    )
+    await db.table('ranges').insert({
+      group_id: groupId,
+      start_id: 1,
+      end_id: 200_000,
+      next_id: 200_001,
+      status: 'done',
+      stored_at: new Date(),
+      capture: true,
+    })
+
+    // The key the edge gave it: the same identity rules, run here over the same lines.
+    const ix = new RangeIndexer({
+      group: GROUP,
+      start: 1,
+      end: 200_000,
+      rules: { ...DEFAULT_RULES, droppedSignatures: [], priorities: { [GROUP]: 1 } },
+    })
+    ix.addText(lines.join('\r\n'), 200_001)
+    ix.finish()
+    const edge = ix.take().releases.find((r) => !r.readable)!
+    assert.equal(edge.firstArticle, 10)
+    const key = releaseKey(edge.key, groupId, 10)
+    await db.table('pending_releases').insert({
+      release_key: key,
+      identity_key: identityKey(edge.key),
+      group_id: groupId,
+      name: 'q3B9OrpKiCxUnzgtUGnp',
+      poster: 'p <p@x>',
+      first_article: 10,
+      last_article: 14,
+      files: 2,
+      segments: 5,
+      bytes: 5000,
+      file_total: 2,
+      tool: 'other',
+      candidates: JSON.stringify([
+        { kind: 'archive', messageId: 'u10@test', file: 'q3B9OrpKiCxUnzgtUGnp.part01.rar' },
+      ]),
+      status: PENDING.naming,
+      named: 'Some.Movie.2016.1080p.BluRay.x264-GRP.mkv',
+    })
+
+    const namer = new PendingNamer(dir)
+    const [row] = await namer.take(5)
+    assert.equal(row.named, 'Some.Movie.2016.1080p.BluRay.x264-GRP.mkv')
+    await namer.move(row)
+
+    const release = await db.from('releases').where('release_key', key).firstOrFail()
+    assert.include(release, {
+      name: 'Some.Movie.2016.1080p.BluRay.x264-GRP.mkv',
+      inner_name: 'Some.Movie.2016.1080p.BluRay.x264-GRP.mkv',
+      source: 7,
+      readable: true,
+      files: 2,
+      segments: '5',
+      file_total: 2,
+      complete: true,
+      first_article: '10',
+      last_article: '14',
+    })
+    assert.isNotNull(release.settled_at)
+    const chunk = await db.from('release_chunks').where('release_id', release.id).firstOrFail()
+    assert.deepEqual(
+      decodeChunk(chunk.payload)
+        .map((f) => [f.name, f.segments.length])
+        .sort(),
+      [
+        ['q3B9OrpKiCxUnzgtUGnp.par2', 1],
+        ['q3B9OrpKiCxUnzgtUGnp.part01.rar', 4],
+      ]
+    )
+    assert.deepEqual(
+      (await db.from('probe_candidates').where('release_id', release.id).orderBy('kind')).map(
+        (c) => [c.kind, c.ready, c.unreadable]
+      ),
+      [
+        ['archive', true, false],
+        ['par2', true, false],
+      ]
+    )
+    assert.deepEqual(
+      await db.from('release_texts').where('release_id', release.id).select('kind', 'text'),
+      [{ kind: 1, text: 'Some.Movie.2016.1080p.BluRay.x264-GRP.mkv' }]
+    )
+    assert.equal(
+      (await db.from('pending_releases').where('release_key', key).firstOrFail()).status,
+      PENDING.named
+    )
+    // Nothing else of the chunk got in.
+    assert.equal(Number((await db.from('releases').count('* as n').first())!.n), 1)
+  })
+})

tests/functional/prober.spec.ts

@@ -2,15 +2,13 @@ import { test } from '@japa/runner'
 import db from '@adonisjs/lucid/services/db'
 import testUtils from '@adonisjs/core/services/test_utils'
 import { Prober } from '#services/prober'
+import { insertRelease } from '#tests/functional/helpers/releases'
 
 test.group('Prober', (group) => {
   group.each.setup(() => testUtils.db().withGlobalTransaction())
 
   test('a claim takes each tier: the readable tiers also take unreadable candidates, a kind with only one of them too', async ({ assert }) => {
-    const [row] = await db
-      .table('releases')
-      .insert({ release_key: 'pb'.padEnd(40, '0'), identity_key: 'pb'.padEnd(40, '0'), name: 'pb', source: 'subject', groups: '[]', files: 3 })
-      .returning('id')
+    const row = { id: await insertRelease({ release_key: 'pb'.padEnd(40, '0'), identity_key: 'pb'.padEnd(40, '0'), name: 'pb', source: 'subject', groups: '[]', files: 3 }) }
     const candidate = (kind: string, unreadable: boolean, ready = true) => ({
       release_id: row.id, kind, message_id: `<${kind}-${unreadable}-${ready}@x>`, file: kind, ready, unreadable,
     })
@@ -26,7 +24,7 @@ test.group('Prober', (group) => {
 
   test('a flush stores a name with a NUL in it, and one article that two releases have as a candidate', async ({ assert }) => {
     const release = async (key: string) =>
-      Number((await db.table('releases').insert({ release_key: key.padEnd(40, '0'), identity_key: key.padEnd(40, '0'), name: key, source: 'subject', groups: '[]', files: 1 }).returning('id'))[0].id)
+      insertRelease({ release_key: key.padEnd(40, '0'), identity_key: key.padEnd(40, '0'), name: key, source: 'subject', groups: '[]', files: 1 })
     const a = await release('fa')
     const b = await release('fb')
     const candidate = async (releaseId: number) =>

tests/functional/release_category.spec.ts

@@ -1,6 +1,7 @@
 import { test } from '@japa/runner'
 import db from '@adonisjs/lucid/services/db'
 import testUtils from '@adonisjs/core/services/test_utils'
+import { insertReleases } from '#tests/functional/helpers/releases'
 
 const categoryOf = async (name: string) => Number((await db.rawQuery(`SELECT release_category(?) AS c`, [name])).rows[0].c)
 
@@ -37,20 +38,21 @@ test.group('Release categories', (group) => {
   })
 
   test("the name's, else the inner name's, else a given one (a spot's), else the groups'", async ({ assert }) => {
-    const row = (key: string, extra: Record<string, unknown>) => ({ release_key: key.padEnd(40, '0'), identity_key: key.padEnd(40, '0'), source: 'stem', files: 1, groups: '["alt.binaries.misc"]', ...extra })
-    await db.table('releases').multiInsert([
+    const row = (key: string, extra: { name: string } & Record<string, unknown>) => ({ release_key: key.padEnd(40, '0'), identity_key: key.padEnd(40, '0'), source: 'stem', files: 1, groups: '["alt.binaries.misc"]', ...extra })
+    const [a, b, c, d] = await insertReleases([
       row('a', { name: 'Show.S01E02.1080p.WEB' }),
       row('b', { name: 'TsIgNBewuFbuwWy' }),
       row('c', { name: 'TsIgNBewuFbuwWy', groups: '["alt.binaries.pictures.erotica.puffies"]' }),
       // A spot's release of its own, inserted with the spot's category.
       row('d', { name: 'Some spot title', category: 2040 }),
     ])
-    const byName = async () => Object.fromEntries((await db.from('releases').select('release_key', 'category')).map((r) => [r.release_key[0], r.category]))
+    const letters = new Map([[a, 'a'], [b, 'b'], [c, 'c'], [d, 'd']])
+    const byName = async () => Object.fromEntries((await db.from('releases').select('id', 'category')).map((r) => [letters.get(Number(r.id)), r.category]))
     assert.deepEqual(await byName(), { a: 5040, b: 8010, c: 6060, d: 2040 })
     // The prober finds an inner name; a spot links a release (the spot inspector sets a Misc one's category).
-    await db.from('releases').where('release_key', 'b'.padEnd(40, '0')).update({ inner_name: 'Some.Movie.2020.1080p.BluRay.x264-GRP.mkv' })
-    await db.from('releases').where('release_key', 'c'.padEnd(40, '0')).update({ category: 5030 })
-    await db.from('releases').where('release_key', 'c'.padEnd(40, '0')).update({ inner_name: 'nothing.bin' })
+    await db.from('releases').where('id', b).update({ inner_name: 'Some.Movie.2020.1080p.BluRay.x264-GRP.mkv' })
+    await db.from('releases').where('id', c).update({ category: 5030 })
+    await db.from('releases').where('id', c).update({ inner_name: 'nothing.bin' })
     assert.deepEqual(await byName(), { a: 5040, b: 2040, c: 5030, d: 2040 })
   })
 

tests/functional/release_judge.spec.ts

@@ -3,32 +3,52 @@ import db from '@adonisjs/lucid/services/db'
 import testUtils from '@adonisjs/core/services/test_utils'
 import { ReleaseJudge } from '#services/release_judge'
 import SearchService from '#services/search_service'
-import RangeBlocks from '#services/range_blocks'
+import { insertRelease } from '#tests/functional/helpers/releases'
 
 test.group('ReleaseJudge', (group) => {
   group.each.setup(() => testUtils.db().withGlobalTransaction())
 
-  test('judges dead, fake, repeated, locked and unusable releases, and takes a verdict back', async ({ assert }) => {
+  test('judges dead, fake, repeated, locked and unusable releases, and takes a verdict back', async ({
+    assert,
+  }) => {
     let n = 0
     const settled = new Date(Date.now() - 3 * 86400_000)
     const release = async (name: string, extra: Record<string, unknown> = {}) => {
       n++
       const key = `jj${String(n).padStart(6, '0')}`.padEnd(40, '0')
-      const [row] = await db
-        .table('releases')
-        .insert({ release_key: key, identity_key: key, name, source: 'stem', groups: '[]', group_id: 1, files: 3, posted_at: settled, readable: true, settled_at: settled, ...extra })
-        .returning('id')
+      const row = { id: await insertRelease({
+          release_key: key,
+          identity_key: key,
+          name,
+          source: 'stem',
+          groups: '[]',
+          group_id: 1,
+          files: 3,
+          posted_at: settled,
+          readable: true,
+          settled_at: settled,
+          ...extra,
+        }) }
       return Number(row.id)
     }
     let candidate = 9000
     const probe = async (releaseId: number, status: string) => {
       candidate++
-      await db.table('probe_candidates').insert({ id: candidate, release_id: releaseId, kind: 'lone', message_id: `<${candidate}@x>`, file: 'x', status })
+      await db.table('probe_candidates').insert({
+        id: candidate,
+        release_id: releaseId,
+        kind: 'lone',
+        message_id: `<${candidate}@x>`,
+        file: 'x',
+        status,
+      })
       return candidate
     }
     const inspect = async (releaseId: number, kind: string, result: Record<string, unknown>) => {
       const id = await probe(releaseId, 'done')
-      await db.table('release_inspections').insert({ release_id: releaseId, candidate_id: id, kind, result: JSON.stringify(result) })
+      await db
+        .table('release_inspections')
+        .insert({ release_id: releaseId, candidate_id: id, kind, result: JSON.stringify(result) })
     }
 
     const dead = await release('Gone.Movie.2020.1080p')
@@ -39,7 +59,11 @@ test.group('ReleaseJudge', (group) => {
     await probe(halfDead, 'done')
 
     const fakeArchive = await release('Das.A-Team.S02E23.Das.Spiel.ist.aus')
-    await inspect(fakeArchive, 'archive', { type: 'rar4', names: ['Das.A-Team.S02E23.Das.Spiel.ist.aus.exe'], entries: 1 })
+    await inspect(fakeArchive, 'archive', {
+      type: 'rar4',
+      names: ['Das.A-Team.S02E23.Das.Spiel.ist.aus.exe'],
+      entries: 1,
+    })
     const fakeFile = await release('Some.Film.2021.1080p.WEB-DL')
     // Only "H.264" and "WEB-DL" say it's media here (a "?" in the pattern never matched).
     const fakeCodec = await release('Some.Show.H.264.WEB-DL')
@@ -54,34 +78,56 @@ test.group('ReleaseJudge', (group) => {
     for (let i = 0; i < 20; i++) {
       const id = await release(`kantodo${i}`)
       repeatedIds.push(id)
-      await db.table('file_hashes').insert({ candidate_id: 20000 + i, release_id: id, hash16k: hash, size: 1073741824 })
+      await db
+        .table('file_hashes')
+        .insert({ candidate_id: 20000 + i, release_id: id, hash16k: hash, size: 1073741824 })
     }
 
-    const scraped = await release('Jamie Lee Curtis Nude Celebrity Pictures', { poster: '@_CelebMovie_@', source: 'subject' })
-    const celeborn = await release('Some.Album.2010', { poster: 'celeborn@not.here (Celeborn)', source: 'subject' })
-    // Scraped posters are found through posters (the batch store adds each release's).
-    await db.table('posters').multiInsert([{ poster: '@_CelebMovie_@' }, { poster: 'celeborn@not.here (Celeborn)' }])
+    const scraped = await release('Jamie Lee Curtis Nude Celebrity Pictures', {
+      poster: '@_CelebMovie_@',
+      source: 'subject',
+    })
+    const celeborn = await release('Some.Album.2010', {
+      poster: 'celeborn@not.here (Celeborn)',
+      source: 'subject',
+    })
     const locked = await release('Locked.Thing', { passworded: true })
 
     const unlocked = await release('Known.Password', { passworded: true })
-    await db.table('release_passwords').insert({ release_id: unlocked, password: 'pw', source: 'spot-text' })
+    await db
+      .table('release_passwords')
+      .insert({ release_id: unlocked, password: 'pw', source: 'spot-text' })
 
-    // A block of done ranges around the unusable ones (release_final); one beside an unread range isn't judged yet.
-    await db.table('groups').insert({ id: 1, name: 'alt.binaries.judged', priority: 1 }).onConflict('id').ignore()
-    await db.table('ranges').multiInsert([
-      { group_id: 1, start_id: 1, end_id: 1000, next_id: 1001, status: 'done', posted_low: new Date(settled.getTime() - 86400_000), posted_high: new Date(settled.getTime() + 86400_000) },
-      { group_id: 1, start_id: 1001, end_id: 2000, next_id: 1001, status: 'planned' },
-    ])
-    await RangeBlocks.rebuild()
+    // Unusable once finished (the window closed on it); a provisional tail release isn't judged yet.
+    await db
+      .table('groups')
+      .insert({ id: 1, name: 'alt.binaries.judged', priority: 1 })
+      .onConflict('id')
+      .ignore()
     const placed = { first_article: 100, last_article: 110 }
     const unusable = await release('ab12CD34ef56GH78ij90', { readable: false, ...placed })
-    const pending = await release('pp12CD34ef56GH78ij90', { readable: false, first_article: 990, last_article: 1000, posted_at: new Date(settled.getTime() + 86400_000 - 3600_000) })
-    const fresh = await release('zz12CD34ef56GH78ij90', { readable: false, settled_at: new Date(), ...placed })
+    const pending = await release('pp12CD34ef56GH78ij90', {
+      readable: false,
+      first_article: 990,
+      last_article: 1000,
+      settled_at: null,
+      posted_at: new Date(settled.getTime() + 86400_000 - 3600_000),
+    })
+    const fresh = await release('zz12CD34ef56GH78ij90', {
+      readable: false,
+      settled_at: new Date(),
+      ...placed,
+    })
     const spotted = await release('yy12CD34ef56GH78ij90', { readable: false, ...placed })
     await db.table('release_spots').insert({ release_id: spotted, spot_id: 's@x', files: 1 })
 
     const round = await new ReleaseJudge().full()
-    const verdicts = new Map((await db.from('release_verdicts').select('release_id', 'rule')).map((v) => [Number(v.release_id), v.rule]))
+    const verdicts = new Map(
+      (await db.from('release_verdicts').select('release_id', 'rule')).map((v) => [
+        Number(v.release_id),
+        v.rule,
+      ])
+    )
     assert.equal(verdicts.get(dead), 'dead')
     assert.isUndefined(verdicts.get(halfDead))
     assert.equal(verdicts.get(fakeArchive), 'fake')
@@ -106,7 +152,9 @@ test.group('ReleaseJudge', (group) => {
     assert.equal((await SearchService.search({ q: 'Locked', judged: true })).total, 1)
 
     // A password found: the next round takes the verdict back.
-    await db.table('release_passwords').insert({ release_id: locked, password: 'found', source: 'nzb' })
+    await db
+      .table('release_passwords')
+      .insert({ release_id: locked, password: 'found', source: 'nzb' })
     const again = await new ReleaseJudge().full()
     assert.equal(again.rules.locked.removed, 1)
     // Verdicts that stay are not written again.
@@ -115,20 +163,31 @@ test.group('ReleaseJudge', (group) => {
     assert.isNull(await db.from('release_verdicts').where('release_id', locked).first())
   })
 
-  test('judges the releases whose facts changed, and those a hash of theirs tipped over', async ({ assert }) => {
+  test('judges the releases whose facts changed, and those a hash of theirs tipped over', async ({
+    assert,
+  }) => {
     let n = 0
     const settled = new Date(Date.now() - 3 * 86400_000)
     const release = async (name: string, extra: Record<string, unknown> = {}) => {
       n++
       const key = `jc${String(n).padStart(6, '0')}`.padEnd(40, '0')
-      const [row] = await db
-        .table('releases')
-        .insert({ release_key: key, identity_key: key, name, source: 'stem', groups: '[]', files: 3, posted_at: settled, readable: true, settled_at: settled, ...extra })
-        .returning('id')
+      const row = { id: await insertRelease({
+          release_key: key,
+          identity_key: key,
+          name,
+          source: 'stem',
+          groups: '[]',
+          files: 3,
+          posted_at: settled,
+          readable: true,
+          settled_at: settled,
+          ...extra,
+        }) }
       return Number(row.id)
     }
     const judge = new ReleaseJudge()
-    const verdict = async (id: number) => (await db.from('release_verdicts').where('release_id', id).first())?.rule
+    const verdict = async (id: number) =>
+      (await db.from('release_verdicts').where('release_id', id).first())?.rule
     const dirty = async () => Number((await db.from('judge_dirty').count('* as n'))[0].n)
 
     // New releases are noted; a passworded one is locked, until a password turns up.
@@ -138,20 +197,31 @@ test.group('ReleaseJudge', (group) => {
     assert.equal(await verdict(locked), 'locked')
     assert.equal(round.rules.locked.added, 1)
     assert.equal(await dirty(), 0)
-    await db.table('release_passwords').insert({ release_id: locked, password: 'pw', source: 'nzb' })
+    await db
+      .table('release_passwords')
+      .insert({ release_id: locked, password: 'pw', source: 'nzb' })
     round = await judge.changed()
     assert.isUndefined(await verdict(locked))
     assert.equal(round.rules.locked.removed, 1)
 
     // Probes: a release whose probed articles are all missing is dead once the prober's statuses come in.
     const dead = await release('Gone.Movie.2020.1080p')
-    const ids = await db.table('probe_candidates').multiInsert([
+    const ids = await db
+      .table('probe_candidates')
+      .multiInsert([
         { release_id: dead, kind: 'lone', message_id: '<jc1@x>', file: 'a', status: 'waiting' },
         { release_id: dead, kind: 'lone', message_id: '<jc2@x>', file: 'b', status: 'waiting' },
-    ]).returning('id')
+      ])
+      .returning('id')
     await judge.changed()
     assert.isUndefined(await verdict(dead))
-    await db.from('probe_candidates').whereIn('id', ids.map((r: { id: string }) => r.id)).update({ status: 'missing' })
+    await db
+      .from('probe_candidates')
+      .whereIn(
+        'id',
+        ids.map((r: { id: string }) => r.id)
+      )
+      .update({ status: 'missing' })
     await judge.changed()
     assert.equal(await verdict(dead), 'dead')
 
@@ -159,13 +229,19 @@ test.group('ReleaseJudge', (group) => {
     // comes first), though only the 20th changed.
     const hash = 'b'.repeat(32)
     const holders: number[] = []
-    for (let i = 0; i < 19; i++) holders.push(await release(`holder${i}`, i === 0 ? { passworded: true } : {}))
-    for (const [i, id] of holders.entries()) await db.table('file_hashes').insert({ candidate_id: 30000 + i, release_id: id, hash16k: hash, size: 5000 })
+    for (let i = 0; i < 19; i++)
+      holders.push(await release(`holder${i}`, i === 0 ? { passworded: true } : {}))
+    for (const [i, id] of holders.entries())
+      await db
+        .table('file_hashes')
+        .insert({ candidate_id: 30000 + i, release_id: id, hash16k: hash, size: 5000 })
     await judge.changed()
     assert.equal(await verdict(holders[0]), 'locked')
     assert.isUndefined(await verdict(holders[1]))
     const twentieth = await release('holder19')
-    await db.table('file_hashes').insert({ candidate_id: 30019, release_id: twentieth, hash16k: hash, size: 5000 })
+    await db
+      .table('file_hashes')
+      .insert({ candidate_id: 30019, release_id: twentieth, hash16k: hash, size: 5000 })
     round = await judge.changed()
     assert.equal(round.releases, 20)
     const rules = new Set()
@@ -173,7 +249,9 @@ test.group('ReleaseJudge', (group) => {
     assert.deepEqual([...rules], ['repeated'])
     // A 21st: the file was in 20 releases without it, so only it is judged.
     const twentyFirst = await release('holder20')
-    await db.table('file_hashes').insert({ candidate_id: 30020, release_id: twentyFirst, hash16k: hash, size: 5000 })
+    await db
+      .table('file_hashes')
+      .insert({ candidate_id: 30020, release_id: twentyFirst, hash16k: hash, size: 5000 })
     round = await judge.changed()
     assert.equal(round.releases, 1)
     assert.equal(await verdict(twentyFirst), 'repeated')

tests/functional/releases_api.spec.ts

@@ -94,6 +94,8 @@ test.group('Releases API', (group) => {
     // As the prober leaves them: the movie's archive is encrypted, the show holds a differently named file.
     await db.from('releases').where('id', movie.id).update({ passworded: true, inspected_at: new Date() })
     await db.from('releases').where('id', show.id).update({ passworded: false, inner_name: 'Hidden.Gem.2023.mkv', readable: false, inspected_at: new Date() })
+    // The name inside, searchable (release_texts, kind file: lib/release/text_kind.ts).
+    await db.table('release_texts').insert({ release_id: show.id, kind: 2, text: 'Hidden.Gem.2023.mkv' })
     await db.table('release_inspections').insert({ release_id: show.id, candidate_id: 1, kind: 'archive', result: JSON.stringify({ yencName: 'x.rar', size: 1, type: 'rar5', names: ['Hidden.Gem.2023.mkv'], entries: 1, passworded: false }) })
 
     const byInner = (await client.get('/api/v1/releases?q=hidden gem')).body() as { releases: { id: number; innerName: string }[] }

tests/functional/settler.spec.ts

@@ -1,39 +0,0 @@
-import { test } from '@japa/runner'
-import db from '@adonisjs/lucid/services/db'
-import testUtils from '@adonisjs/core/services/test_utils'
-import Settler from '#services/settler'
-
-test.group('Settler', (group) => {
-  group.each.setup(() => testUtils.db().withGlobalTransaction())
-
-  test('settles quiet releases: complete by the [i/N] counter, unknown without one', async ({ assert }) => {
-    const old = new Date(Date.now() - 7 * 3600 * 1000)
-    const row = (key: string, files: number, fileTotal: number | null, lastSeen: Date) => ({
-      release_key: key.padEnd(40, '0'), identity_key: key.padEnd(40, '0'), name: key, source: 'subject', groups: '[]', files, file_total: fileTotal, last_seen_at: lastSeen,
-    })
-    await db.table('releases').multiInsert([row('a', 10, 10, old), row('b', 8, 10, old), row('c', 3, null, old), row('d', 10, 10, new Date())])
-    assert.equal(await Settler.run(), 3)
-    const byName = Object.fromEntries((await db.from('releases').select('name', 'complete', 'settled_at')).map((r) => [r.name, [r.complete, r.settled_at !== null]]))
-    assert.deepEqual(byName, { a: [true, true], b: [false, true], c: [null, true], d: [null, false] })
-  })
-
-  test('settles history releases (posted long ago) once unchanged for half an hour', async ({ assert }) => {
-    const posted = new Date('2015-01-01T00:00:00Z')
-    const row = (key: string, lastSeen: Date, lastPosted: Date) => ({
-      release_key: key.padEnd(40, '0'), identity_key: key.padEnd(40, '0'), name: key, source: 'subject', groups: '[]', files: 1, last_seen_at: lastSeen, posted_at: lastPosted, last_posted_at: lastPosted,
-    })
-    const minutesAgo = (m: number) => new Date(Date.now() - m * 60_000)
-    await db.table('releases').multiInsert([row('old', minutesAgo(31), posted), row('fresh', minutesAgo(5), posted), row('tail', minutesAgo(31), minutesAgo(40))])
-    const ids = Object.fromEntries((await db.from('releases').select('id', 'name')).map((r) => [r.name, r.id]))
-    await db.table('probe_candidates').multiInsert([
-      { release_id: ids.old, kind: 'lone', message_id: '<a@x>', file: 'a' },
-      { release_id: ids.fresh, kind: 'lone', message_id: '<b@x>', file: 'b' },
-    ])
-    assert.equal(await Settler.run(), 1)
-    // The settled release's candidates are claimable now, as unreadable (readable defaults to false).
-    const ready = await db.from('probe_candidates').orderBy('message_id').select('ready', 'unreadable')
-    assert.deepEqual(ready.map((r) => [r.ready, r.unreadable]), [[true, true], [false, false]])
-    const settled = (await db.from('releases').whereNotNull('settled_at').select('name')).map((r) => r.name)
-    assert.deepEqual(settled, ['old'])
-  })
-})

tests/functional/seven_zip_tails.spec.ts

@@ -2,13 +2,14 @@ import { test } from '@japa/runner'
 import db from '@adonisjs/lucid/services/db'
 import testUtils from '@adonisjs/core/services/test_utils'
 import { pruneSevenZipTails } from '#services/seven_zip_tails'
+import { insertReleases } from '#tests/functional/helpers/releases'
 
 test.group('7z tails', (group) => {
   group.each.setup(() => testUtils.db().withGlobalTransaction())
 
   test("of a release's waiting tails, those below its highest volume go; the highest volume's all stay", async ({ assert }) => {
     const row = (key: string) => ({ release_key: key.padEnd(40, '0'), identity_key: key.padEnd(40, '0'), name: key, source: 'stem', groups: '[]', files: 4 })
-    const [merged, other] = (await db.table('releases').multiInsert([row('a'), row('b')]).returning('id')).map((r) => r.id)
+    const [merged, other] = await insertReleases([row('a'), row('b')])
     const tail = (releaseId: number, volume: string, mid: string, status = 'waiting') => ({ release_id: releaseId, kind: '7z-tail', message_id: mid, file: `x.7z.${volume}`, status })
     await db.table('probe_candidates').multiInsert([
       // Two merged members' tails, one of the highest volume probed already.

tests/functional/spots.spec.ts

@@ -5,6 +5,8 @@ import { Spotter } from '#services/spotter'
 import { SpotInspector } from '#services/spot_inspector'
 import { releaseIdentity } from '#lib/release/index'
 import { parseSubject } from '#lib/headers/index'
+import { SOURCES } from '#lib/release/source'
+import { insertRelease } from '#tests/functional/helpers/releases'
 
 // Message ids whose sha1('<id>') starts with 0000: self-signed spots (key 7) pass on that hashcash alone.
 const SPOT = 'spottest731@spot.net'
@@ -68,10 +70,7 @@ test.group('Spots', (group) => {
     // The release the header pass built for these files: the same identity, in the group.
     const parsed = parseSubject(subjects[0])!
     const identity = releaseIdentity({ poster, subject: parsed.base, name: parsed.name, weak: false })!
-    const [r] = await db
-      .table('releases')
-      .insert({ release_key: identity.key, identity_key: identity.key, name: identity.name, source: identity.source, poster, groups: '[]', group_id: groupId, files: 3, posted_at: posted, readable: true })
-      .returning('id')
+    const r = { id: await insertRelease({ release_key: identity.key, identity_key: identity.key, name: identity.name, source: identity.source, poster, groups: '[]', group_id: groupId, files: 3, posted_at: posted, readable: true }) }
     const releaseId = Number(r.id)
     const nzb = nzbXml(subjects, poster, posted.getTime() / 1000, 'nzbpw')
     await new Spotter().store(rawSpot(SPOT, 'Some.Movie.2024.1080p', 'Wachtwoord: geheim123'), SPOT, 10)
@@ -100,13 +99,14 @@ test.group('Spots', (group) => {
     const inspector = new SpotInspector()
     assert.deepEqual(await inspector.inspectNzb(await db.from('spots').where('message_id', SPOT).firstOrFail(), nzb), { status: 'inspected', matched: 0 })
     assert.isTrue(await inspector.standalone(SPOT))
-    const release = await db.from('releases').where('source', 'spot').firstOrFail()
+    const release = await db.from('releases').where('source', SOURCES.spot).firstOrFail()
     assert.equal(release.name, 'Lonely File')
     // Its title says nothing: the spot's category (01, a09: a movie, x264).
     assert.equal(release.category, 2040)
     assert.equal(release.files, 1)
-    assert.isNotNull(release.poster)
-    assert.isNotNull(await db.from('posters').where('poster', release.poster).first())
+    // Its poster (the spot's) by id, in posters for poster search.
+    assert.isNotNull(release.poster_id)
+    assert.isNotEmpty((await db.from('posters').where('id', release.poster_id).firstOrFail()).poster)
     const { default: NzbService } = await import('#services/nzb_service')
     assert.include((await NzbService.build(Number(release.id)))!.xml, 'lonely.file.mkv')
     // Not again while it has one; once the header pass builds the file's release, the spot moves there.
@@ -114,10 +114,7 @@ test.group('Spots', (group) => {
     const group = await db.from('groups').where('name', 'alt.binaries.test.spots').firstOrFail()
     const parsed = parseSubject('[1/1] - "lonely.file.mkv" yEnc (1/9)')!
     const identity = releaseIdentity({ poster: 'p', subject: parsed.base, name: parsed.name, weak: false })!
-    const [h] = await db
-      .table('releases')
-      .insert({ release_key: identity.key, identity_key: identity.key, name: identity.name, source: identity.source, poster: 'p', groups: '[]', group_id: group.id, files: 1, posted_at: new Date(STAMP * 1000), readable: true })
-      .returning('id')
+    const h = { id: await insertRelease({ release_key: identity.key, identity_key: identity.key, name: identity.name, source: identity.source, poster: 'p', groups: '[]', group_id: group.id, files: 1, posted_at: new Date(STAMP * 1000), readable: true }) }
     assert.equal(await inspector.match(SPOT), 1)
     assert.isNull(await db.from('releases').where('id', release.id).first())
     assert.equal(Number((await db.from('spots').where('message_id', SPOT).firstOrFail()).release_id), Number(h.id))

tests/unit/edge/range_indexer.spec.ts

@@ -83,7 +83,7 @@ test.group('edge / RangeIndexer', () => {
     indexer.add(post(999_990, 'End.Cut.2020.part1.rar', 3).slice(0, 2), 1_000_001)
     indexer.finish()
     const batch = indexer.take()
-    assert.deepEqual(batch.fragments.map((f) => [f.name, f.segments.map((s) => s[0])]).sort(), [['End.Cut.2020.part1.rar', [1, 2]], ['Start.Cut.2020.part1.rar', [2, 3]]])
+    assert.deepEqual(batch.fragments!.map((f) => [f.name, f.segments.map((s) => s[0])]).sort(), [['End.Cut.2020.part1.rar', [1, 2]], ['Start.Cut.2020.part1.rar', [2, 3]]])
     assert.deepEqual(batch.counts.map((c) => [c.rule, c.articles, c.files]), [['incomplete', 2, 1]])
     assert.lengthOf(batch.releases, 0)
   })

tests/unit/probe/inspect.spec.ts

@@ -4,7 +4,7 @@ import { readFileSync } from 'node:fs'
 import { crc32 } from 'node:zlib'
 import yencode from 'yencode'
 import { crc32Hex } from '#lib/yenc/index'
-import { inspect, innerName, MAX_NAMES } from '#lib/probe/index'
+import { inspect, innerName, MAX_NAMES, usableName } from '#lib/probe/index'
 
 /** A single-part yEnc article of `data`. */
 function article(data: Buffer, name: string) {
@@ -76,6 +76,19 @@ function rar5File(name: string, size: number, opts: { packed: number; headerFlag
   return rar5Block(2, opts.headerFlags ?? 0, fields, { extra, data: opts.packed })
 }
 
+test.group('probe / usable name (the pending ladder)', () => {
+  test('a par2 or listing that names a real file names the release; random volumes and packaging do not', ({ assert }) => {
+    const base = { size: 1000, type: null }
+    assert.equal(usableName({ ...base, yencName: 'x.par2', names: ['q3B9OrpKiCxUnzgtUGnp.part01.rar', 'Some.Movie.2016.mkv'] }, 'par2'), 'Some.Movie.2016.mkv')
+    assert.isNull(usableName({ ...base, yencName: 'q3B9OrpKiCxUnzgtUGnp.par2', names: ['q3B9OrpKiCxUnzgtUGnp.part01.rar'] }, 'par2'))
+    assert.equal(usableName({ ...base, yencName: 'x.rar', names: ['Folder/Real.Name.S01E01.mkv'] }, 'archive'), 'Real.Name.S01E01.mkv')
+    assert.isNull(usableName({ ...base, yencName: 'x.rar', names: [], passworded: true }, 'archive'))
+    assert.equal(usableName({ ...base, yencName: 'brothalovers.20.02.16.hazel.moore.mp4' }, 'lone'), 'brothalovers.20.02.16.hazel.moore.mp4')
+    assert.isNull(usableName({ ...base, yencName: 'c3453cdd2a3645a8f3cf3d22319abe63' }, 'lone'))
+    assert.isNull(usableName({ ...base, yencName: 'c3453cdd2a3645a8f3cf3d22319abe63.part01.rar' }, 'lone'))
+  })
+})
+
 test.group('probe / inspect', () => {
   test('a par2 index: its files, largest first', ({ assert }) => {
     const setId = Buffer.alloc(16, 3)

tests/unit/release/source.spec.ts

@@ -0,0 +1,19 @@
+import { test } from '@japa/runner'
+import { SOURCES, sourceCode, sourceName } from '#lib/release/source'
+
+test.group('release source codes', () => {
+  test('names and codes map both ways; the codes are fixed', ({ assert }) => {
+    assert.deepEqual(SOURCES, { 'subject': 1, 'post': 2, 'stem': 3, 'spot': 4, 'nzb': 5, 'album': 6, 'weak-stem': 7 })
+    for (const [name, code] of Object.entries(SOURCES)) {
+      assert.equal(sourceCode(name), code)
+      assert.equal(sourceName(code), name)
+      assert.equal(sourceName(String(code)), name)
+    }
+  })
+
+  test('an unknown name throws, an unknown code reads as unknown', ({ assert }) => {
+    assert.throws(() => sourceCode('subjekt'), /Unknown release source/)
+    assert.equal(sourceName(99), 'unknown')
+    assert.equal(sourceName(null), 'unknown')
+  })
+})

tests/unit/release/text_kind.spec.ts

@@ -0,0 +1,42 @@
+import { test } from '@japa/runner'
+import {
+  TEXT_KINDS,
+  TEXTS_PER_KIND,
+  TEXT_MAX,
+  textKindCode,
+  textKindName,
+  textsToStore,
+} from '#lib/release/text_kind'
+
+test.group('release text kinds', () => {
+  test('names and codes map both ways; the codes are fixed', ({ assert }) => {
+    assert.deepEqual(TEXT_KINDS, { name: 1, file: 2, nfo: 3, par2: 4, media: 5, member: 6 })
+    for (const [name, code] of Object.entries(TEXT_KINDS)) {
+      assert.equal(textKindCode(name), code)
+      assert.equal(textKindName(code), name)
+      assert.equal(textKindName(String(code)), name)
+    }
+  })
+
+  test('an unknown name throws, an unknown code reads as unknown', ({ assert }) => {
+    assert.throws(() => textKindCode('title'), /Unknown release text kind/)
+    assert.equal(textKindName(99), 'unknown')
+    assert.equal(textKindName(null), 'unknown')
+  })
+
+  test('textsToStore trims, cuts, drops empties and repeats, keeps the first TEXTS_PER_KIND', ({
+    assert,
+  }) => {
+    const long = 'x'.repeat(TEXT_MAX + 10)
+    assert.deepEqual(textsToStore([' a.mkv ', '', null, 'a.mkv', 'b.mkv', long]), [
+      'a.mkv',
+      'b.mkv',
+      'x'.repeat(TEXT_MAX),
+    ])
+    const many = Array.from({ length: TEXTS_PER_KIND + 5 }, (_, i) => `file${i}.rar`)
+    const kept = textsToStore(many)
+    assert.lengthOf(kept, TEXTS_PER_KIND)
+    assert.equal(kept[0], 'file0.rar')
+    assert.equal(kept[TEXTS_PER_KIND - 1], `file${TEXTS_PER_KIND - 1}.rar`)
+  })
+})