voyager 3: the batch store writes releases once, by id references and bytea keys
Test and deploy / test (push) Successful in 51s
Test and deploy / edge (push) Successful in 1m48s
Test and deploy / deploy (push) Skipped

Each release chunk is a release inserted once: release_key the SHA-1 of
the identity, the owner group and the first article (deterministic; no
numbering under a group lock), identity_key the identity's SHA-1 bytes,
poster_id and group_set_id through catalog_refs (cached per process),
source as a code (weak-stem added), the name into release_texts. A
finished release's candidates are claimable at once; a provisional tail
release is finished by a later chunk of its key, a finished one sent again
is left alone. Gone: joins into earlier releases and across groups, key
numbering, the group lock, the fragment merger. Interim: chunks, counts
and samples keyed by ranges.block_id, a range without a block gets one.

Readers follow: search (posters and group_sets joined, words through
release_texts, posters by id, source by code), the par2 merger, the
sidecar folder, the judge's scraped rule, the spot inspector, the spotter,
the NZB service, releases:regroup, edge:bench. Tests insert fixtures
through tests/functional/helpers/releases.ts; the suites pass again.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Daan MeijerandClaude Fable 5.1 committed 2026-10-25 21:04:03 +01:00
1 parent bc4e17e86f
commit 2580f20a43
32 files changed
+962 -497

No files matched your search

+45 -6
View File
@@ -531,12 +531,51 @@ releases; the inspector matches them again), `nntp_shares`, `nodes`,
rule read all of `release_inspections` every round, 17 min on the HDD;
the judge must read it by release or keep what it needs on the NVMe.
Next: the batch store (write-once releases, posters and group sets by id,
bytea keys, blocks, the name into `release_texts`), then the window and
albums in the Rust edge, block
leases and the block merger, the corpus replay harness. Until the batch
store is done the 38 functional tests that write the old columns fail on
`voyager-3`.
### Batch store (built 2026-10-25)
`app/services/batch_store.ts`: every release chunk of a batch is one
release, inserted once. Its keys (`lib/release/keys.ts`): `identity_key`
the identity's SHA-1 as 20 bytes; `release_key` the SHA-1 of the identity,
the owner group and the release's first article, so it is deterministic
(no numbering under a group lock, no catalog lookups before writing) and a
repost, starting at another article, is a release of its own. Its poster
and group set by id (`app/services/catalog_refs.ts`: insert what's new,
read the ids, cache them per process; the tests forget the cache per
test), its source as a code, its name into `release_texts` (kind 1). The
chunk's own counts go in as the release's; `complete` is decided then from
the counter. A finished release's probe candidates are claimable at once
(`ready`, `unreadable`); a provisional tail release (`finished: false` on
the chunk, `settled_at` null) is finished by a later chunk of the same
release key, which adds the files not sent before (sums, one more chunk),
sets `settled_at` and makes its candidates claimable; a finished release
sent again is left as it is (`ON CONFLICT … WHERE settled_at IS NULL`).
Gone: the join of a chunk into an earlier release of its key (the window
keeps an upload together; the block merger joins what a block's end cut),
the join across groups (FTD's spread uploads: a gap, to be handled by the
block merger or the par2 merger across groups), the numbering of release
keys, the group lock, `fragments` (ignored when sent; the Rust edge still
sends them). The chunk type carries `finished` and `edge` (both optional;
the Rust edge doesn't send them yet, so its releases are finished, not
edges). Interim until blocks are leased: chunks, rule counts and samples
are keyed by `ranges.block_id`, and a range without a block gets a block
of its own in the store (`blockOf`).
Readers moved with it: search joins `posters` and `group_sets`, matches
words in `release_texts` (`r.id IN (SELECT release_id … ILIKE)`), searches
posters by id, filters `source` by code and returns the source name; the
par2 merger, the sidecar folder and the judge's scraped rule use
`poster_id` (`release_merges.from_key` holds the key as hex); the spot
inspector matches by bytea identity keys and inserts its spot releases by
id references with their title in `release_texts`; the spotter and the NZB
service compare the source code. Tests insert fixtures through
`tests/functional/helpers/releases.ts` (`insertRelease`: text keys, poster
and groups by name, source by name). The unit and functional suites pass.
Next: the window and albums in the Rust edge (`finished`, `edge`, the
provisional tail and its finish), block leases and the planner (then the
store keys by the leased block and the lease columns leave `ranges`), the
block merger, the prober writing `release_texts` kind 2, the corpus replay
harness.
## Decisions