Daan MeijerandClaude Opus 5.5
cd06f96f85
End to end on the workstation (coordinator:run, the API, edge:node, real NNTP): a.b.e-book.french and a.b.cd.image.playstation2.dvdiso ranges. - Prefetcher: every connection takes the next missing chunk (leased ranges first, in order); one connection per range (~26k lines/s) kept the worker waiting - executable rule: releases of bare executables (malware floods named after films and software) are dropped; they were 92% of the catalog rows of the test ranges, 21% of the segments - fragments per file (details once) and gzipped batch bodies; the JSON limit (1 MB by default) raised to 256 MB, empty strings kept (a rule count's key is '') - release_chunks.batch_from is a bigint (boneless article numbers) - tests get their own database; edge:bench --store times BatchStore: 1.3 s for a boneless range (1.24 ms per release chunk) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
27 lines
825 B
TypeScript
27 lines
825 B
TypeScript
import { BaseSchema } from '@adonisjs/lucid/schema'
|
|
|
|
/**
|
|
* Files cut off by a range's edge (REBUILD.md, fragments): one row per file
|
|
* and range, with that range's segments; merged when the neighbouring range
|
|
* is done, then written as a chunk.
|
|
*/
|
|
export default class extends BaseSchema {
|
|
async up() {
|
|
this.schema.createTable('fragments', (table) => {
|
|
table.bigIncrements('id')
|
|
table.bigInteger('range_id').notNullable()
|
|
table.string('file_key', 40).notNullable()
|
|
// The file's first article in this range.
|
|
table.bigInteger('article_number').notNullable()
|
|
table.integer('segments').notNullable()
|
|
table.jsonb('fragment').notNullable()
|
|
table.index(['file_key'])
|
|
table.index(['range_id'])
|
|
})
|
|
}
|
|
|
|
async down() {
|
|
this.schema.dropTable('fragments')
|
|
}
|
|
}
|