Rebuild: the batch store (catalog upserts, write-once chunks, candidates, rule counts, fragments, resume point)
Test and deploy / test (push) Successful in 20s
Test and deploy / deploy (push) Skipped

A batch is stored in one transaction and only at its range's resume
point, so a retried batch can't count twice. Garbage is judged before
per-article: a dropped tool isn't worth reading bodies for either.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Daan MeijerandClaude Opus 5.5 committed 2026-10-09 00:21:17 +02:00
1 parent 0714b2c81a
commit 20910647fa
6 files changed
+242 -7

No files matched your search

+5 -2
View File
@@ -65,8 +65,11 @@ dropping garbage; 28 TB of compressed overview traffic. Throughput goal:
3. **Chunk store** (bulk, append-only). One record per release per batch:
its files' names, sizes and segment lists, compactly encoded (template
codec for message-ids, zstd). Written once, never updated; a re-run of a
range replaces that range's chunks (no per-file dedup). ~3 TB.
codec for message-ids, zstd). Written once, never updated. A batch is
stored in one transaction and only when it starts at its range's resume
point, so a batch sent twice or late is refused and the catalog's sums
stay right (no per-file dedup); a recovery run for a retracted rule only
adds what that rule had dropped. ~3 TB.
4. **Release catalog** (hot, Postgres on NVMe). One row per release: key,
name, poster, groups, posted_at, file count, size, segments, expected