daan/usenet-indexer
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

voyager 3: the batch store writes releases once, by id references and bytea keys
Test and deploy / test (push) Successful in 51s
Details
Test and deploy / edge (push) Successful in 1m48s
Details
Test and deploy / deploy (push) Skipped
Details

Browse Source 
Each release chunk is a release inserted once: release_key the SHA-1 of
the identity, the owner group and the first article (deterministic; no
numbering under a group lock), identity_key the identity's SHA-1 bytes,
poster_id and group_set_id through catalog_refs (cached per process),
source as a code (weak-stem added), the name into release_texts. A
finished release's candidates are claimable at once; a provisional tail
release is finished by a later chunk of its key, a finished one sent again
is left alone. Gone: joins into earlier releases and across groups, key
numbering, the group lock, the fragment merger. Interim: chunks, counts
and samples keyed by ranges.block_id, a range without a block gets one.

Readers follow: search (posters and group_sets joined, words through
release_texts, posters by id, source by code), the par2 merger, the
sidecar folder, the judge's scraped rule, the spot inspector, the spotter,
the NZB service, releases:regroup, edge:bench. Tests insert fixtures
through tests/functional/helpers/releases.ts; the suites pass again.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Daan MeijerandClaude Fable 5.1 committed 2026-10-25 21:04:03 +01:00
1 parent bc4e17e86f
commit 2580f20a43
32 files changed
+962 -497
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

app/services/batch_store.ts
+176 -188
View File
@@ -1,14 +1,18 @@
import db from '@adonisjs/lucid/services/db'
import { pruneSevenZipTails } from '#services/seven_zip_tails'
import type { TransactionClientContract } from '@adonisjs/lucid/types/database'
import { UPLOAD_GAP_MS, type EdgeBatch, type ProbeCandidate, type ReleaseChunk } from '#lib/edge/index'
import type { EdgeBatch, ProbeCandidate, ReleaseChunk } from '#lib/edge/index'
import ChunkPartitions from '#services/chunk_partitions'
import CatalogRefs, { groupSetKey, posterText } from '#services/catalog_refs'
import { decodeChunk } from '#lib/chunks/index'
import { SIDECAR } from '#services/sidecar_folder'
import { identityKey, releaseKey } from '#lib/release/keys'
import { sourceCode } from '#lib/release/source'
import { TEXT_KINDS, TEXT_MAX } from '#lib/release/text_kind'
/**
* A small new release (at most 3 files, under 1 MB) whose files are all sidecars (.nfo, .sfv, .nzb, …): the sidecar
* fold's candidates (releases.sidecar_only; the settler queues only these). Its file list decoded here, small as it is.
* fold's candidates (releases.sidecar_only). Its file list decoded here, small as it is.
*/
function sidecarOnly(r: ReleaseChunk): boolean | null {
if (r.files > 3 || r.bytes >= 1_000_000) return null
@@ -19,18 +23,32 @@ function sidecarOnly(r: ReleaseChunk): boolean | null {
export class StaleBatch extends Error {}
export interface StoredBatch {
/** Release chunks in the batch. */
releases: number
/** Releases the batch started. */
newReleases: number
/** Provisional (tail) releases the batch finished. */
finalized: number
candidates: number
fragments: number
}
/** A chunk's slot in its batch: the key and the upload the edge numbered it with (ProbeCandidate points at it). */
const slot = (key: string, upload: number) => `${key}\n${upload}`
/**
* Stores one batch of a leased range (REBUILD.md, parts 3–7), in one
* transaction: release aggregates in the catalog, one chunk per release,
* probe candidates, rule counts and samples, edge fragments, and the
* range's resume point. A batch that doesn't start at the resume point was
* Stores one batch of a leased range (REBUILD.md, "Voyager 3: window indexing", 2 and 4) in one transaction: each
* release chunk is a whole release the edge's window finished, written once (its poster and group set by id, its
* keys as bytea, its name into release_texts), with one chunk of its files, its probe candidates, and the batch's
* rule counts and samples; then the range's resume point moves. A batch that doesn't start at the resume point was
* sent twice or too late, and is refused (StaleBatch), so the sums stay right.
*
* A tail release comes twice: provisionally (finished: false, settled_at null, its candidates waiting) and then
* finished, under the same release key (the identity, the group and its first article: lib/release/keys.ts), with
* the files not sent before (a chunk's counts are its own; the finish adds them up); the finish is the one update a
* release gets from the header pass. A finished release sent again is left as it is.
*
* Interim, until blocks are leased (the next step): chunks, counts and samples are keyed by the range's block
* (ranges.block_id), and a range without one gets a block of its own here.
*/
class BatchStore {
async store(rangeId: number, nodeId: number, batch: EdgeBatch): Promise<StoredBatch> {
@@ -39,7 +57,9 @@ class BatchStore {
// One batch of a range at a time, without holding the range's row for the whole transaction: the
// prefetcher's and heartbeats' updates of it waited on it (0.5 s on average, NNTP connections idle;
// LOG.md 2026-10-09 (13)). The resume point is checked again when it moves, at the end.
await trx.rawQuery(`SELECT pg_advisory_xact_lock(hashtextextended('range:' || ?::text, 0))`, [rangeId])
await trx.rawQuery(`SELECT pg_advisory_xact_lock(hashtextextended('range:' || ?::text, 0))`, [
rangeId,
])
const range = await trx.from('ranges').where('id', rangeId).first()
if (!range || range.status !== 'leased' || Number(range.node_id) !== nodeId)
throw new StaleBatch(`Range ${rangeId} is not leased to this node`)
@@ -47,35 +67,37 @@ class BatchStore {
throw new StaleBatch(
`Range ${rangeId} is at ${range.next_id}, the batch starts at ${batch.from}`
)
const blockId = await this.blockOf(trx, range)
const result: StoredBatch = {
releases: batch.releases.length,
newReleases: 0,
finalized: 0,
candidates: 0,
fragments: batch.fragments.length,
}
const written = await this.writeReleases(
trx,
rangeId,
{ groupId: Number(range.group_id), blockId },
batch.from,
batch.releases,
batch.candidates
)
result.newReleases = written.newReleases
result.finalized = written.finalized
result.candidates = written.candidates
if (batch.counts.length > 0) {
await trx.rawQuery(
`INSERT INTO rule_counts (range_id, rule, key, articles, files)
`INSERT INTO rule_counts (block_id, rule, key, articles, files)
SELECT ?, v.rule, v.key, v.articles, v.files FROM jsonb_to_recordset(?::jsonb) AS v(rule text, key text, articles bigint, files bigint)
ON CONFLICT (range_id, rule, key) DO UPDATE SET articles = rule_counts.articles + excluded.articles, files = rule_counts.files + excluded.files`,
[rangeId, JSON.stringify(batch.counts)]
ON CONFLICT (block_id, rule, key) DO UPDATE SET articles = rule_counts.articles + excluded.articles, files = rule_counts.files + excluded.files`,
[blockId, JSON.stringify(batch.counts)]
)
}
for (const s of batch.samples) {
await trx.insertQuery().table('rule_samples').insert({
range_id: rangeId,
block_id: blockId,
rule: s.rule,
rule_key: s.ruleKey,
release_key: s.key,
@@ -95,106 +117,75 @@ class BatchStore {
.update({
next_id: batch.next,
headers: db.raw('headers + ?', [batch.headers]),
// The range's posting time span (release_final): the batches' trimmed spans, min and max (least and greatest skip nulls).
posted_low: db.raw('least(posted_low, to_timestamp(?::double precision / 1000))', [batch.postedLow ?? null]),
posted_high: db.raw('greatest(posted_high, to_timestamp(?::double precision / 1000))', [batch.postedHigh ?? null]),
// The range's posting time span: the batches' trimmed spans, min and max (least and greatest skip nulls).
posted_low: db.raw('least(posted_low, to_timestamp(?::double precision / 1000))', [
batch.postedLow ?? null,
]),
posted_high: db.raw('greatest(posted_high, to_timestamp(?::double precision / 1000))', [
batch.postedHigh ?? null,
]),
heartbeat_at: new Date(),
})
if (Number(moved) !== 1 && !(Array.isArray(moved) && moved.length === 1)) throw new StaleBatch(`Range ${rangeId} moved on meanwhile`)
if (Number(moved) !== 1 && !(Array.isArray(moved) && moved.length === 1))
throw new StaleBatch(`Range ${rangeId} moved on meanwhile`)
return result
})
}
/**
* An upload spread over groups (FTD: par2 and part1 to a.b.music.misc,
* part2 to a.b.sounds.music, each article in one group only; LOG.md
* 2026-10-11 (9)): a chunk without a release in its own group joins one of
* its key in another group whose window is within UPLOAD_GAP_MS, when none
* of its file names are in that release yet. A post of the same files to
* another group (other message-ids, the same names) stays a release of its
* own, or the NZB would hold every file twice. Reads the candidate's file
* names from its chunks; candidates are few.
*/
private async joinAcrossGroups(
trx: TransactionClientContract,
part: ReleaseChunk[],
windows: { idx: number; key: string; first: number | null | undefined; last: number | null | undefined }[],
groupId: number,
joined: Map<number, number>
) {
const open = windows.filter((w) => !joined.has(w.idx) && w.first)
if (open.length === 0) return
const { rows: candidates } = await trx.rawQuery(
`SELECT v.idx, r.id
FROM jsonb_to_recordset(?::jsonb) AS v(idx int, key text, first bigint, last bigint)
JOIN releases r ON r.identity_key = v.key AND r.group_id <> ?
WHERE r.posted_at <= to_timestamp((v.last + ?) / 1000.0) AND coalesce(r.last_posted_at, r.posted_at) >= to_timestamp((v.first - ?) / 1000.0)
ORDER BY v.idx, r.last_posted_at DESC NULLS LAST, r.id DESC`,
[JSON.stringify(open), groupId, UPLOAD_GAP_MS, UPLOAD_GAP_MS]
)
if (candidates.length === 0) return
const ids: number[] = [...new Set<number>(candidates.map((c: { id: string }) => Number(c.id)))]
const names = new Map<number, Set<string>>()
const { rows: chunks } = await trx.rawQuery(`SELECT release_id, payload FROM release_chunks WHERE release_id = ANY(?)`, [ids])
for (const c of chunks) {
const set = names.get(Number(c.release_id)) ?? new Set<string>()
for (const f of decodeChunk(c.payload)) set.add(f.name)
names.set(Number(c.release_id), set)
}
for (const c of candidates as { idx: number; id: string }[]) {
if (joined.has(c.idx)) continue
const have = names.get(Number(c.id)) ?? new Set<string>()
const incoming = decodeChunk(part[c.idx].payload).map((f) => f.name)
if (incoming.every((name) => !have.has(name))) {
joined.set(c.idx, Number(c.id))
for (const name of incoming) have.add(name)
names.set(Number(c.id), have)
}
}
/** The range's block; a range without one (no planner yet) is a block of its own, leased as the range is. */
private async blockOf(trx: TransactionClientContract, range: any): Promise<number> {
if (range.block_id) return Number(range.block_id)
const [block] = await trx
.table('blocks')
.insert({
group_id: range.group_id,
start_id: range.start_id,
end_id: range.end_id,
next_id: range.next_id,
status: 'leased',
priority: range.priority ?? 0,
node_id: range.node_id,
leased_at: range.leased_at ?? new Date(),
heartbeat_at: new Date(),
})
.returning('id')
await trx.from('ranges').where('id', range.id).update({ block_id: block.id })
return Number(block.id)
}
/**
* Release aggregates in the catalog, one write-once chunk per release, and
* probe candidates (a batch's, or files the fragment merger completed).
* A chunk joins the release of its key whose articles the range's group
* owns and whose posting window is within UPLOAD_GAP_MS of its own, else
* starts one: re-uploads are releases of their own (decision d). The
* group is locked, so concurrent writers don't both start one.
* The releases of a batch into the catalog, one chunk each, and their probe candidates. Every chunk is one
* release: inserted, or (a provisional tail release sent again) finished.
*/
async writeReleases(
trx: TransactionClientContract,
rangeId: number,
where: { groupId: number; blockId: number },
batchFrom: number,
releases: ReleaseChunk[],
candidates: ProbeCandidate[]
) {
const out = { newReleases: 0, candidates: 0 }
const out = { newReleases: 0, finalized: 0, candidates: 0 }
if (releases.length === 0) return out
/** Release ids by slot, for the chunks and candidates; a slot not here was a finished release sent again. */
const ids = new Map<string, number>()
const slot = (key: string, upload: number) => `${key}\n${upload}`
const { group_id: groupId } = await trx.from('ranges').where('id', rangeId).select('group_id').firstOrFail()
// One writer per group at a time: a chunk only matches releases of its group, and a lock per key
// ran out of shared memory (a picture group's batch has 50k keys; LOG.md 2026-10-09 (12)).
await trx.rawQuery(`SELECT pg_advisory_xact_lock(hashtextextended('releases:' || ?::text, 0))`, [groupId])
const finalized = new Set<number>()
const posters = await CatalogRefs.posterIds(trx, releases.map((r) => r.poster).filter(Boolean))
const groupSets = await CatalogRefs.groupSetIds(
trx,
releases.map((r) => r.groups)
)
const sorted = [...releases].sort((a, b) => a.key.localeCompare(b.key) || a.upload - b.upload)
for (let i = 0; i < sorted.length; i += 1000) {
const part = sorted.slice(i, i + 1000)
const windows = part.map((r, idx) => ({ idx, key: r.key, first: r.postedAt, last: r.postedLast ?? r.postedAt }))
const { rows: matches } = await trx.rawQuery(
`SELECT DISTINCT ON (v.idx) v.idx, r.id
FROM jsonb_to_recordset(?::jsonb) AS v(idx int, key text, first bigint, last bigint)
JOIN releases r ON r.identity_key = v.key AND r.group_id = ?
WHERE v.first IS NULL OR r.posted_at IS NULL
OR (r.posted_at <= to_timestamp((v.last + ?) / 1000.0) AND coalesce(r.last_posted_at, r.posted_at) >= to_timestamp((v.first - ?) / 1000.0))
ORDER BY v.idx, r.last_posted_at DESC NULLS LAST, r.id DESC`,
[JSON.stringify(windows), groupId, UPLOAD_GAP_MS, UPLOAD_GAP_MS]
)
const joined = new Map<number, number>(matches.map((m: { idx: number; id: string }) => [m.idx, Number(m.id)]))
await this.joinAcrossGroups(trx, part, windows, groupId, joined)
const row = (r: ReleaseChunk) => ({
const rows = part.map((r, idx) => ({
idx,
release_key: releaseKey(r.key, where.groupId, r.firstArticle).toString('hex'),
identity_key: identityKey(r.key).toString('hex'),
name: r.name,
source: r.source,
poster: r.poster,
groups: r.groups,
source: sourceCode(r.source),
poster_id: r.poster ? (posters.get(posterText(r.poster)) ?? null) : null,
group_set_id: groupSets.get(groupSetKey(r.groups))!,
first: r.postedAt,
last: r.postedLast ?? r.postedAt,
file_total: r.fileTotal,
@@ -205,107 +196,105 @@ class BatchStore {
last_article: r.lastArticle,
readable: r.readable,
has_par2: r.hasPar2,
})
const updates = part.flatMap((r, idx) => (joined.has(idx) ? [{ id: joined.get(idx)!, ...row(r) }] : []))
if (updates.length > 0) {
await trx.rawQuery(
`WITH v AS (SELECT * FROM jsonb_to_recordset(?::jsonb) AS v(id bigint, first bigint, last bigint, file_total int, files int, segments bigint, bytes bigint, first_article bigint, last_article bigint, readable boolean, has_par2 boolean)),
a AS (SELECT id, sum(files) AS files, sum(segments) AS segments, sum(bytes) AS bytes, count(*) AS chunks, min(first) AS first, max(last) AS last,
max(file_total) AS file_total, min(first_article) AS first_article, max(last_article) AS last_article, bool_or(readable) AS readable, bool_or(has_par2) AS has_par2
FROM v GROUP BY id)
UPDATE releases r SET
files = r.files + a.files,
segments = r.segments + a.segments,
bytes = r.bytes + a.bytes,
chunks = r.chunks + a.chunks,
posted_at = least(r.posted_at, to_timestamp(a.first / 1000.0)),
last_posted_at = greatest(r.last_posted_at, to_timestamp(a.last / 1000.0)),
file_total = coalesce(r.file_total, a.file_total),
first_article = least(r.first_article, a.first_article),
last_article = greatest(r.last_article, a.last_article),
readable = r.readable OR a.readable,
has_par2 = r.has_par2 OR a.has_par2,
-- As it stands (the counter's files all found); settling makes it final. Without a counter, what its par2
-- set decided stays (app/services/par2_completeness.ts).
complete = CASE WHEN coalesce(r.file_total, a.file_total) IS NULL THEN r.complete ELSE r.files + a.files >= coalesce(r.file_total, a.file_total) END,
last_seen_at = now()
FROM a WHERE r.id = a.id`,
[JSON.stringify(updates)]
finished: r.finished !== false,
edge: r.edge === true,
sidecar_only: sidecarOnly(r),
}))
// Inserted, or a provisional release of the key finished (the chunk's files added up); a finished release
// sent again (no row returned) is left as it is.
const { rows: written } = await trx.rawQuery(
`INSERT INTO releases (release_key, identity_key, group_id, group_set_id, poster_id, name, source, posted_at, last_posted_at, file_total, files, segments, bytes, chunks,
first_article, last_article, readable, has_par2, complete, edge, sidecar_only, settled_at)
SELECT decode(v.release_key, 'hex'), decode(v.identity_key, 'hex'), ?, v.group_set_id, v.poster_id, v.name, v.source,
to_timestamp(v.first / 1000.0), to_timestamp(v.last / 1000.0), v.file_total, v.files, v.segments, v.bytes, 1,
v.first_article, v.last_article, v.readable, v.has_par2, CASE WHEN v.file_total IS NULL THEN NULL ELSE v.files >= v.file_total END,
v.edge, v.sidecar_only, CASE WHEN v.finished THEN now() END
FROM jsonb_to_recordset(?::jsonb) AS v(idx int, release_key text, identity_key text, name text, source int, poster_id int, group_set_id int, first bigint, last bigint,
file_total int, files int, segments bigint, bytes bigint, first_article bigint, last_article bigint, readable boolean, has_par2 boolean,
finished boolean, edge boolean, sidecar_only boolean)
ON CONFLICT (release_key) DO UPDATE SET
files = releases.files + excluded.files, segments = releases.segments + excluded.segments, bytes = releases.bytes + excluded.bytes,
chunks = releases.chunks + CASE WHEN excluded.files > 0 THEN 1 ELSE 0 END,
posted_at = least(releases.posted_at, excluded.posted_at), last_posted_at = greatest(releases.last_posted_at, excluded.last_posted_at),
file_total = coalesce(excluded.file_total, releases.file_total),
first_article = least(releases.first_article, excluded.first_article), last_article = greatest(releases.last_article, excluded.last_article),
readable = releases.readable OR excluded.readable, has_par2 = releases.has_par2 OR excluded.has_par2,
complete = CASE WHEN coalesce(excluded.file_total, releases.file_total) IS NULL THEN releases.complete
ELSE releases.files + excluded.files >= coalesce(excluded.file_total, releases.file_total) END,
edge = excluded.edge, sidecar_only = coalesce(releases.sidecar_only, excluded.sidecar_only), settled_at = excluded.settled_at
WHERE releases.settled_at IS NULL
RETURNING id, encode(release_key, 'hex') AS release_key, (xmax = 0) AS inserted, settled_at`,
[where.groupId, JSON.stringify(rows)]
)
const byKey = new Map<string, { id: number; inserted: boolean; settled: boolean }>(
written.map(
(w: { id: string; release_key: string; inserted: boolean; settled_at: Date | null }) => [
w.release_key,
{ id: Number(w.id), inserted: w.inserted, settled: w.settled_at !== null },
]
)
part.forEach((r, idx) => joined.has(idx) && ids.set(slot(r.key, r.upload), joined.get(idx)!))
)
const texts: { release_id: number; text: string }[] = []
for (const row of rows) {
const w = byKey.get(row.release_key)
if (!w) continue
const r = part[row.idx]
ids.set(slot(r.key, r.upload), w.id)
if (w.inserted) out.newReleases++
else if (w.settled) {
out.finalized++
finalized.add(w.id)
}
texts.push({ release_id: w.id, text: row.name.slice(0, TEXT_MAX) })
}
const fresh = part.filter((_, idx) => !joined.has(idx))
if (fresh.length > 0) {
// The first upload of a key keeps the key; later ones are key:2, key:3, … after the highest number: merges
// delete releases, and a count fell behind it (83 releases of a key up to key:129: 20 numbers tried, all
// taken, the range's batch failed for good; LOG.md 2026-10-20 (89)). Spot releases (spot:key) aren't numbered.
const { rows: counts } = await trx.rawQuery(
`SELECT identity_key, max(CASE WHEN release_key = identity_key THEN 1
WHEN left(release_key, length(identity_key) + 1) = identity_key || ':' AND substr(release_key, length(identity_key) + 2) ~ '^[0-9]{1,9}$'
THEN substr(release_key, length(identity_key) + 2)::int END) AS n
FROM releases WHERE identity_key = ANY(?) GROUP BY 1`,
[[...new Set(fresh.map((r) => r.key))]]
// The name, searchable (release_texts; a name sent again is a no-op).
if (texts.length > 0) {
await trx.rawQuery(
`INSERT INTO release_texts (release_id, kind, text)
SELECT v.release_id, ?, v.text FROM jsonb_to_recordset(?::jsonb) AS v(release_id bigint, text text) WHERE v.text <> ''
ON CONFLICT DO NOTHING`,
[TEXT_KINDS.name, JSON.stringify(texts)]
)
const taken = new Map<string, number>(counts.map((c: { identity_key: string; n: string }) => [c.identity_key, Number(c.n)]))
const numbered = (r: ReleaseChunk) => {
const n = (taken.get(r.key) ?? 0) + 1
taken.set(r.key, n)
return { release_key: n === 1 ? r.key : `${r.key}:${n}`, key: r.key, upload: r.upload, ...row(r), sidecar_only: sidecarOnly(r) }
}
// Their posters, for poster search (posters: one row per poster, searched by trigram; migration 4000).
const posters = [...new Set(fresh.map((r) => r.poster).filter((p): p is string => !!p))]
if (posters.length > 0) await trx.rawQuery(`INSERT INTO posters (poster) SELECT DISTINCT left(p, 200) FROM unnest(?::text[]) AS p ON CONFLICT DO NOTHING`, [posters])
let inserts = fresh.map(numbered)
// Writers of other groups number the same key at the same time (a spam post crossposted to 67
// groups; only the group is locked): a number taken meanwhile is skipped, and the next one tried.
for (let attempt = 0; inserts.length > 0; attempt++) {
if (attempt === 20) throw new Error(`No free release key for ${inserts[0].key}`)
const { rows } = await trx.rawQuery(
`INSERT INTO releases (release_key, identity_key, group_id, name, source, poster, groups, posted_at, last_posted_at, file_total, files, segments, bytes, chunks, first_article, last_article, readable, has_par2, complete, sidecar_only)
SELECT v.release_key, v.key, ?, v.name, v.source, v.poster, v.groups, to_timestamp(v.first / 1000.0), to_timestamp(v.last / 1000.0), v.file_total, v.files, v.segments, v.bytes, 1,
v.first_article, v.last_article, v.readable, v.has_par2, CASE WHEN v.file_total IS NULL THEN NULL ELSE v.files >= v.file_total END, v.sidecar_only
FROM jsonb_to_recordset(?::jsonb) AS v(release_key text, key text, name text, source text, poster text, groups jsonb, first bigint, last bigint, file_total int, files int, segments bigint, bytes bigint, first_article bigint, last_article bigint, readable boolean, has_par2 boolean, sidecar_only boolean)
ON CONFLICT (release_key) DO NOTHING
RETURNING id, release_key`,
[groupId, JSON.stringify(inserts)]
)
const byReleaseKey = new Map<string, number>(rows.map((x: { id: string; release_key: string }) => [x.release_key, Number(x.id)]))
const missed: typeof inserts = []
for (const r of inserts) {
const id = byReleaseKey.get(r.release_key)
if (id === undefined) missed.push(r)
else ids.set(slot(r.key, r.upload), id)
}
out.newReleases += rows.length
const chunks = new Map(fresh.map((r) => [slot(r.key, r.upload), r]))
inserts = missed.map((r) => numbered(chunks.get(slot(r.key, r.upload))!))
}
}
}
// One write-once chunk per release.
// One write-once chunk per release (a finish adds the files not sent before).
for (let i = 0; i < sorted.length; i += 500) {
const rows = sorted.slice(i, i + 500).map((r) => ({
release_id: ids.get(slot(r.key, r.upload))!,
range_id: rangeId,
batch_from: batchFrom,
files: r.files,
segments: r.segments,
payload: r.payload,
}))
const rows = sorted
.slice(i, i + 500)
.filter((r) => ids.has(slot(r.key, r.upload)) && r.files > 0)
.map((r) => ({
release_id: ids.get(slot(r.key, r.upload))!,
block_id: where.blockId,
batch_from: batchFrom,
files: r.files,
segments: r.segments,
payload: r.payload,
}))
if (rows.length > 0) await trx.insertQuery().table('release_chunks').multiInsert(rows)
}
// A finished release's candidates are claimable at once; a provisional one's wait for its finish.
const settled = new Set<number>()
for (const r of sorted) {
const id = ids.get(slot(r.key, r.upload))
if (id !== undefined && r.finished !== false) settled.add(id)
}
const chunks = new Map(sorted.map((r) => [slot(r.key, r.upload), r]))
const kept = candidates.filter((c) => ids.has(slot(c.releaseKey, c.upload ?? 0)))
for (let i = 0; i < kept.length; i += 2000) {
const rows = kept
.slice(i, i + 2000)
.map((c) => ({
release_id: ids.get(slot(c.releaseKey, c.upload ?? 0))!,
const rows = kept.slice(i, i + 2000).map((c) => {
const releaseId = ids.get(slot(c.releaseKey, c.upload ?? 0))!
const chunk = chunks.get(slot(c.releaseKey, c.upload ?? 0))!
return {
release_id: releaseId,
kind: c.kind,
message_id: c.messageId,
file: c.file,
}))
ready: settled.has(releaseId),
unreadable: !chunk.readable,
}
})
const added = await trx
.knexQuery()
.table('probe_candidates')
@@ -317,15 +306,14 @@ class BatchStore {
// A later piece of a split 7z: the tails of lower volumes go (app/services/seven_zip_tails.ts).
const tails = new Set(rows.filter((r) => r.kind === '7z-tail').map((r) => r.release_id))
if (added.length > 0 && tails.size > 0) await pruneSevenZipTails(trx, [...tails])
// Files added to a settled release (a later range of its upload): the settler made its candidates claimable
// already and won't again (93,642 waited since 2026-10-13; LOG.md 2026-10-20 (87)).
if (added.length > 0) {
await trx.rawQuery(
`UPDATE probe_candidates pc SET ready = true, unreadable = NOT r.readable FROM releases r
WHERE pc.id = ANY(?) AND r.id = pc.release_id AND r.settled_at IS NOT NULL`,
[added.map((a: { id: string | number }) => String(a.id))]
)
}
}
// The finish of a provisional release: its waiting candidates become claimable.
if (finalized.size > 0) {
await trx.rawQuery(
`UPDATE probe_candidates pc SET ready = true, unreadable = NOT r.readable FROM releases r
WHERE pc.release_id = ANY(?) AND r.id = pc.release_id AND pc.status = 'waiting' AND NOT pc.ready`,
[[...finalized]]
)
}
return out