Daan MeijerandClaude Fable 5.1
44c0c73b05
The prober names 6% of the unnamed releases it finishes and the pending queue (44M) grows eleven times faster than it probes. A sample of 44,530 finished pending rows shows the headers predict a name: size bucket, name shape, completeness, tool and group separate 0% classes from 95% ones, while the learner keyed on tool and group alone (LOG.md (130)). - lib/release/classes.ts, names::pending_class, SQL pending_class(): a release's class (size × shape × partial, 1..70), the three in step. - Migration 6800: pending_releases.klass and target_bytes (the first target's article; candidates carry bytes now), pending_classes with each class's climbs, names, waiting and verdict, the claim's index. - The ladder claims the classes with the best record first (a prior for the unexplored, at most a quarter of a claim from one), the cheapest target first within one, then the oldest of the rest; records classes like signatures and drops them at the same thresholds; discards their waiting releases. Claimed rows sorted in code: an UPDATE returns none. - The rule set's droppedClasses: the edge keeps unnamed releases of a dropped class out of the pending store (counted as garbage), the explored sample excepted. - hash_twins.ts: a probed file whose first 16 KiB and size match a named catalog release's names a pending release the probe didn't, and sets the inner name of a catalog release without one. - pending:classes classes the rows from before and seeds the record from the finished ones; rules:classes publishes the dropped classes; releases:readable also marks the rows' candidates unreadable. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
58 lines
3.0 KiB
TypeScript
58 lines
3.0 KiB
TypeScript
import { BaseSchema } from '@adonisjs/lucid/schema'
|
||
|
||
/**
|
||
* The pending ladder by class (REBUILD.md decision f; LOG.md (130)): a pending release's class (lib/release/classes.ts:
|
||
* size bucket × name shape × partial, 1..70; 0 for rows from before) and the bytes of its first target, so the
|
||
* claim takes the classes with the best record first and the cheapest article within one; pending_classes per
|
||
* (tool, group, class) how many climbed to the end and how many got a name, `waiting` how many wait (kept by the
|
||
* batch store, the claim and the discard; recounted by pending:classes), `dropped` as pending_signatures.
|
||
* The SQL pending_class() is the TypeScript's pendingClass, for the batch store's insert and the backfill.
|
||
*/
|
||
export default class extends BaseSchema {
|
||
static disableTransactions = true
|
||
|
||
async up() {
|
||
this.schema.raw(`
|
||
CREATE OR REPLACE FUNCTION pending_class(name text, bytes bigint, partial boolean) RETURNS smallint LANGUAGE sql IMMUTABLE AS $$
|
||
SELECT ((CASE WHEN bytes < 1000000 THEN 0 WHEN bytes < 10000000 THEN 1 WHEN bytes < 100000000 THEN 2 WHEN bytes < 1000000000 THEN 3 ELSE 4 END) * 14
|
||
+ (CASE WHEN s ~ '^[0-9a-fA-F]{16,}$' THEN 1
|
||
WHEN s ~ '^[0-9]{6,}$' THEN 2
|
||
WHEN s ~ '^[a-z2-7]{20,}$' AND s ~ '[2-7]' THEN 3
|
||
WHEN s ~ '^[A-Za-z]+$' THEN 4
|
||
WHEN s ~ '^[A-Za-z0-9]+$' THEN 5
|
||
WHEN s ~ '^[A-Za-z0-9._-]+$' THEN 6
|
||
ELSE 7 END - 1) * 2
|
||
+ (CASE WHEN partial THEN 1 ELSE 0 END) + 1)::smallint
|
||
FROM (SELECT regexp_replace(name, '\\.[A-Za-z0-9]{1,4}$', '') AS s) x
|
||
$$`)
|
||
this.schema.raw(
|
||
`ALTER TABLE pending_releases ADD COLUMN IF NOT EXISTS klass smallint NOT NULL DEFAULT 0`
|
||
)
|
||
this.schema.raw(`ALTER TABLE pending_releases ADD COLUMN IF NOT EXISTS target_bytes bigint`)
|
||
this.schema.raw(`
|
||
CREATE TABLE IF NOT EXISTS pending_classes (
|
||
tool varchar(16) NOT NULL,
|
||
group_id integer NOT NULL,
|
||
klass smallint NOT NULL,
|
||
probed integer NOT NULL DEFAULT 0,
|
||
named integer NOT NULL DEFAULT 0,
|
||
waiting integer NOT NULL DEFAULT 0,
|
||
dropped boolean NOT NULL DEFAULT false,
|
||
updated_at timestamptz NOT NULL DEFAULT now(),
|
||
PRIMARY KEY (tool, group_id, klass)
|
||
)`)
|
||
// The claim by class: the cheapest target first within one, the oldest among equals.
|
||
this.schema.raw(
|
||
`CREATE INDEX CONCURRENTLY IF NOT EXISTS pending_releases_class_idx ON pending_releases (tool, group_id, klass, target_bytes, created_at) WHERE status = 0`
|
||
)
|
||
}
|
||
|
||
async down() {
|
||
this.schema.raw(`DROP INDEX CONCURRENTLY IF EXISTS pending_releases_class_idx`)
|
||
this.schema.raw(`DROP TABLE IF EXISTS pending_classes`)
|
||
this.schema.raw(`ALTER TABLE pending_releases DROP COLUMN IF EXISTS target_bytes`)
|
||
this.schema.raw(`ALTER TABLE pending_releases DROP COLUMN IF EXISTS klass`)
|
||
this.schema.raw(`DROP FUNCTION IF EXISTS pending_class(text, bigint, boolean)`)
|
||
}
|
||
}
|