Daan MeijerandClaude Opus 5.5
cd06f96f85
End to end on the workstation (coordinator:run, the API, edge:node, real NNTP): a.b.e-book.french and a.b.cd.image.playstation2.dvdiso ranges. - Prefetcher: every connection takes the next missing chunk (leased ranges first, in order); one connection per range (~26k lines/s) kept the worker waiting - executable rule: releases of bare executables (malware floods named after films and software) are dropped; they were 92% of the catalog rows of the test ranges, 21% of the segments - fragments per file (details once) and gzipped batch bodies; the JSON limit (1 MB by default) raised to 256 MB, empty strings kept (a rule count's key is '') - release_chunks.batch_from is a bigint (boneless article numbers) - tests get their own database; edge:bench --store times BatchStore: 1.3 s for a boneless range (1.24 ms per release chunk) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
35 lines
1.3 KiB
TypeScript
35 lines
1.3 KiB
TypeScript
import { BaseSchema } from '@adonisjs/lucid/schema'
|
|
import env from '#start/env'
|
|
|
|
/**
|
|
* The chunk store (REBUILD.md, part 3; decision a): one write-once row per
|
|
* release per batch, its files encoded by lib/chunks. Partitioned by month
|
|
* of writing, in CHUNK_TABLESPACE (the HDD) when set; ChunkPartitions keeps
|
|
* the next months' partitions ready. A re-run of a range replaces its chunks
|
|
* (by range_id).
|
|
*/
|
|
export default class extends BaseSchema {
|
|
async up() {
|
|
const tablespace = env.get('CHUNK_TABLESPACE')
|
|
const where = tablespace ? ` TABLESPACE ${tablespace}` : ''
|
|
this.schema.raw(`
|
|
CREATE TABLE release_chunks (
|
|
id bigserial,
|
|
release_id bigint NOT NULL,
|
|
range_id bigint NOT NULL,
|
|
batch_from bigint NOT NULL,
|
|
files integer NOT NULL,
|
|
segments integer NOT NULL,
|
|
payload bytea NOT NULL,
|
|
created_at timestamptz NOT NULL DEFAULT now(),
|
|
PRIMARY KEY (id, created_at)
|
|
) PARTITION BY RANGE (created_at)${where}`)
|
|
this.schema.raw(`CREATE INDEX release_chunks_release_idx ON release_chunks (release_id)`)
|
|
this.schema.raw(`CREATE INDEX release_chunks_range_idx ON release_chunks (range_id)`)
|
|
}
|
|
|
|
async down() {
|
|
this.schema.raw(`DROP TABLE IF EXISTS release_chunks CASCADE`)
|
|
}
|
|
}
|