Corpus: stored XOVER chunks for the grouping sweeps
Test and deploy / test (push) Successful in 43s
Test and deploy / edge (push) Successful in 1m48s
Test and deploy / deploy (push) Successful in 1m23s

Ranges marked `capture` keep their XOVER chunks in CORPUS_DIR
(corpus/<group>/<start>-<end>/, edge-bench's layout, range.json once
complete; only chunks that inflate). The prefetcher reads a stored chunk
instead of fetching it, so a reset re-indexes them without the news
server; a chunk a worker reports corrupt goes from the corpus too.

edge-bench gets --releases (release chunks without payload, with their
first subjects) and ships in the edge image; deploy/corpus_sweep.py marks
more ranges (capture), replays the corpus with the deployed build (replay)
and reports suspects and the change since the previous build (report).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Daan MeijerandClaude Opus 5.5 committed 2026-10-12 09:28:22 +02:00
1 parent d43603a43f
commit 9ea2669b2a
13 files changed
+389 -11

No files matched your search

+8 -1
View File
@@ -2,6 +2,7 @@ import db from '@adonisjs/lucid/services/db'
import coordinatorConfig from '#config/coordinator'
import RuleSets from '#services/rule_sets'
import Spool from '#services/spool'
import Corpus from '#services/corpus'
import type { Lease } from '#lib/edge/index'
export class LeaseLost extends Error {}
@@ -66,7 +67,13 @@ class Leases {
async corrupt(rangeId: number, nodeId: number, from: number) {
const range = await this.leased(rangeId, nodeId)
const span = Spool.chunksOf({ start: Number(range.start_id), end: Number(range.end_id) }).find((s) => s.from === from)
if (span) await Spool.removeChunk(rangeId, span)
if (span) {
await Spool.removeChunk(rangeId, span)
if (range.capture) {
const group = await db.from('groups').where('id', range.group_id).select('name').first()
await Corpus.removeChunk({ group: group.name, start: Number(range.start_id), end: Number(range.end_id) }, span)
}
}
const plain: number[] = (range.plain_chunks ?? []).map(Number)
// Reported once already: the compressed copy may have been in flight (or the prefetcher's targets not
// refreshed yet) and spooled again (range 161820, LOG.md 2026-10-11), so a second try plain first.