Rebuild: first end-to-end runs; prefetch per chunk, executable rule, wire fixes
Test and deploy / test (push) Successful in 20s
Test and deploy / deploy (push) Skipped

End to end on the workstation (coordinator:run, the API, edge:node, real
NNTP): a.b.e-book.french and a.b.cd.image.playstation2.dvdiso ranges.

- Prefetcher: every connection takes the next missing chunk (leased
  ranges first, in order); one connection per range (~26k lines/s) kept
  the worker waiting
- executable rule: releases of bare executables (malware floods named
  after films and software) are dropped; they were 92% of the catalog rows
  of the test ranges, 21% of the segments
- fragments per file (details once) and gzipped batch bodies; the JSON
  limit (1 MB by default) raised to 256 MB, empty strings kept (a rule
  count's key is '')
- release_chunks.batch_from is a bigint (boneless article numbers)
- tests get their own database; edge:bench --store times BatchStore:
  1.3 s for a boneless range (1.24 ms per release chunk)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Daan MeijerandClaude Opus 5.5 committed 2026-10-09 01:07:11 +02:00
1 parent d5f168d8d2
commit cd06f96f85
15 files changed
+201 -99

No files matched your search

@@ -17,7 +17,7 @@ export default class extends BaseSchema {
id bigserial,
release_id bigint NOT NULL,
range_id bigint NOT NULL,
batch_seq integer NOT NULL,
batch_from bigint NOT NULL,
files integer NOT NULL,
segments integer NOT NULL,
payload bytea NOT NULL,
@@ -1,9 +1,9 @@
import { BaseSchema } from '@adonisjs/lucid/schema'
/**
* Segments of files cut off by a range's edge (REBUILD.md, fragments): kept
* by range, merged when the neighbouring range is done, then written as a
* chunk.
* Files cut off by a range's edge (REBUILD.md, fragments): one row per file
* and range, with that range's segments; merged when the neighbouring range
* is done, then written as a chunk.
*/
export default class extends BaseSchema {
async up() {
@@ -11,8 +11,10 @@ export default class extends BaseSchema {
table.bigIncrements('id')
table.bigInteger('range_id').notNullable()
table.string('file_key', 40).notNullable()
// The file's first article in this range.
table.bigInteger('article_number').notNullable()
table.jsonb('segment').notNullable()
table.integer('segments').notNullable()
table.jsonb('fragment').notNullable()
table.index(['file_key'])
table.index(['range_id'])
})
+6 -4
View File
@@ -33,18 +33,20 @@ export class AuthAccessTokenSchema extends BaseModel {
}
export class FragmentSchema extends BaseModel {
static $columns = ['articleNumber', 'fileKey', 'id', 'rangeId', 'segment'] as const
static $columns = ['articleNumber', 'fileKey', 'fragment', 'id', 'rangeId', 'segments'] as const
$columns = FragmentSchema.$columns
@column()
declare articleNumber: bigint | number
@column()
declare fileKey: string
@column()
declare fragment: any
@column({ isPrimary: true })
declare id: bigint | number
@column()
declare rangeId: bigint | number
@column()
declare segment: any
declare segments: number
}
export class GroupSchema extends BaseModel {
@@ -165,10 +167,10 @@ export class RangeSchema extends BaseModel {
}
export class ReleaseChunkSchema extends BaseModel {
static $columns = ['batchSeq', 'createdAt', 'files', 'id', 'payload', 'rangeId', 'releaseId', 'segments'] as const
static $columns = ['batchFrom', 'createdAt', 'files', 'id', 'payload', 'rangeId', 'releaseId', 'segments'] as const
$columns = ReleaseChunkSchema.$columns
@column()
declare batchSeq: number
declare batchFrom: bigint | number
@column.dateTime({ autoCreate: true })
declare createdAt: DateTime
@column()