Pending ladder by class, twins by content, dropped classes at the edge
The prober names 6% of the unnamed releases it finishes and the pending queue (44M) grows eleven times faster than it probes. A sample of 44,530 finished pending rows shows the headers predict a name: size bucket, name shape, completeness, tool and group separate 0% classes from 95% ones, while the learner keyed on tool and group alone (LOG.md (130)). - lib/release/classes.ts, names::pending_class, SQL pending_class(): a release's class (size × shape × partial, 1..70), the three in step. - Migration 6800: pending_releases.klass and target_bytes (the first target's article; candidates carry bytes now), pending_classes with each class's climbs, names, waiting and verdict, the claim's index. - The ladder claims the classes with the best record first (a prior for the unexplored, at most a quarter of a claim from one), the cheapest target first within one, then the oldest of the rest; records classes like signatures and drops them at the same thresholds; discards their waiting releases. Claimed rows sorted in code: an UPDATE returns none. - The rule set's droppedClasses: the edge keeps unnamed releases of a dropped class out of the pending store (counted as garbage), the explored sample excepted. - hash_twins.ts: a probed file whose first 16 KiB and size match a named catalog release's names a pending release the probe didn't, and sets the inner name of a catalog release without one. - pending:classes classes the rows from before and seeds the record from the finished ones; rules:classes publishes the dropped classes; releases:readable also marks the rows' candidates unreadable. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
1 parent
81a5dce45a
commit
44c0c73b05
20 files changed
+927
-55
No files matched your search
+12
-1
@@ -9,6 +9,7 @@ import { inspect, innerName, PROBE_VERSION, type Inspection } from '#lib/probe/i
|
||||
import { NntpShare } from '#services/nntp_shares'
|
||||
import { DiskSpace } from '#services/disk_space'
|
||||
import { decidePar2Completeness } from '#services/par2_completeness'
|
||||
import { twinKey, twinNames } from '#services/hash_twins'
|
||||
import PendingLadder, { type PendingProbe } from '#services/pending_ladder'
|
||||
import { PendingNamer } from '#services/pending_namer'
|
||||
|
||||
@@ -392,10 +393,20 @@ export class Prober {
|
||||
entry.all.push({ kind: r.kind, result: r.result })
|
||||
byRelease.set(id, entry)
|
||||
}
|
||||
// A twin by content names a release its own inspections didn't (hash_twins.ts; LOG.md (130)).
|
||||
const twins = await twinNames(
|
||||
done.filter((o) => o.inspection!.hash16k && o.inspection!.size).map((o) => ({ hash16k: o.inspection!.hash16k!, size: o.inspection!.size })),
|
||||
trx
|
||||
)
|
||||
const twinOf = new Map<number, string>()
|
||||
for (const o of done) {
|
||||
const twin = o.inspection!.hash16k ? twins.get(twinKey(o.inspection!.hash16k, o.inspection!.size)) : undefined
|
||||
if (twin && twin !== byRelease.get(o.candidate.releaseId)?.name) twinOf.set(o.candidate.releaseId, twin)
|
||||
}
|
||||
const updates = [...byRelease].map(([id, { name, all }]) => ({
|
||||
id,
|
||||
passworded: all.some((i) => i.result.passworded === true),
|
||||
inner_name: innerName(all, name),
|
||||
inner_name: innerName(all, name) ?? twinOf.get(id) ?? null,
|
||||
}))
|
||||
await trx.rawQuery(
|
||||
`UPDATE releases r SET passworded = v.passworded, inner_name = v.inner_name, inspected_at = now()
|
||||
|
||||
Reference in new issue
Block a user