Daan MeijerandClaude Opus 5.5
7df5a638ca
Two processes, one NNTP connection each (the coordinator 40 -> 38): - spots:read (service spotter): free.pt's overviews after the last article seen, each spot's full header verified (lib/spotnet from voyager 1, as Spotweb), its description, website and NZB articles; the password its title or text names; delete/dispose messages applied. - spots:inspect (service spot-inspector): each spot's compressed NZB read, its files listed (spot_files), each matched to the header pass's release by the same release identity in one of its groups within 48 h (release_spots); unmatched spots matched again hourly, after 24 h a release of their own (source spot) with the spot's NZB. - release_passwords: per release and password, its source (nzb, spot-title, spot-text) and spot; the catalog keeps only passworded. XoverClient.head(); PROBE_CONNECTIONS default 10. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
96 lines
3.7 KiB
TypeScript
96 lines
3.7 KiB
TypeScript
import { BaseSchema } from '@adonisjs/lucid/schema'
|
|
|
|
/**
|
|
* Spotnet spots (REBUILD.md, spots; after voyager 1's D29), in two processes:
|
|
* - the spotter (`spots:read`) reads free.pt's headers, verifies each spot's
|
|
* signature from its full header (X-XML) and keeps it in `spots`, with a
|
|
* password its title or description names;
|
|
* - the spot inspector (`spots:inspect`) reads each spot's compressed NZB
|
|
* (alt.binaries.ftd), keeps its files in `spot_files`, and matches them to
|
|
* the releases the header pass built (the same release identity), linking
|
|
* them in `release_spots`. A spot whose files match no release becomes a
|
|
* release of its own (source `spot`) once a day has passed.
|
|
* The NZB itself is kept (gzip) for such a release's downloads.
|
|
* Passwords are kept apart from the catalog in `release_passwords`, one row
|
|
* per release and password, with where it came from (the NZB's meta, the
|
|
* spot's title or its text); the catalog only says `passworded`.
|
|
*/
|
|
export default class extends BaseSchema {
|
|
async up() {
|
|
this.schema.raw(`
|
|
CREATE TABLE spots (
|
|
message_id text PRIMARY KEY,
|
|
article_number bigint,
|
|
title text NOT NULL,
|
|
tag text,
|
|
poster text NOT NULL,
|
|
category integer,
|
|
subcats jsonb,
|
|
size bigint,
|
|
posted_at timestamptz,
|
|
key_id integer,
|
|
spotter_id text,
|
|
description text,
|
|
website text,
|
|
nzb_segments text[],
|
|
-- The NZB as read (gzip): a release the spot makes of its own has no chunks to build one from.
|
|
nzb bytea,
|
|
password text,
|
|
password_source varchar(16),
|
|
-- listed: verified, waiting for the inspector; inspected, no-nzb, dummy, nzb-gone, nzb-unreadable: after it;
|
|
-- moderation: a delete/dispose message; removed: disposed of by a moderator or its spotter.
|
|
status varchar(16) NOT NULL DEFAULT 'listed',
|
|
files integer,
|
|
matched integer,
|
|
release_id bigint,
|
|
error text,
|
|
created_at timestamptz NOT NULL DEFAULT now(),
|
|
inspected_at timestamptz
|
|
)`)
|
|
this.schema.raw(`CREATE INDEX spots_listed_idx ON spots (posted_at DESC) WHERE status = 'listed'`)
|
|
this.schema.raw(`CREATE INDEX spots_unmatched_idx ON spots (inspected_at) WHERE status = 'inspected' AND matched = 0 AND release_id IS NULL`)
|
|
this.schema.raw(`
|
|
CREATE TABLE spot_files (
|
|
spot_id text NOT NULL,
|
|
idx integer NOT NULL,
|
|
name text NOT NULL,
|
|
subject text NOT NULL,
|
|
poster text,
|
|
groups text[],
|
|
posted_at timestamptz,
|
|
bytes bigint NOT NULL,
|
|
segments integer NOT NULL,
|
|
release_id bigint,
|
|
PRIMARY KEY (spot_id, idx)
|
|
)`)
|
|
this.schema.raw(`CREATE INDEX spot_files_release_idx ON spot_files (release_id) WHERE release_id IS NOT NULL`)
|
|
this.schema.raw(`
|
|
CREATE TABLE release_spots (
|
|
release_id bigint NOT NULL,
|
|
spot_id text NOT NULL,
|
|
files integer NOT NULL,
|
|
created_at timestamptz NOT NULL DEFAULT now(),
|
|
PRIMARY KEY (release_id, spot_id)
|
|
)`)
|
|
this.schema.raw(`CREATE INDEX release_spots_spot_idx ON release_spots (spot_id)`)
|
|
this.schema.raw(`
|
|
CREATE TABLE release_passwords (
|
|
release_id bigint NOT NULL,
|
|
password text NOT NULL,
|
|
source varchar(16) NOT NULL,
|
|
spot_id text,
|
|
created_at timestamptz NOT NULL DEFAULT now(),
|
|
PRIMARY KEY (release_id, password)
|
|
)`)
|
|
this.schema.raw(`
|
|
CREATE TABLE spot_state (
|
|
group_name text PRIMARY KEY,
|
|
last_article bigint NOT NULL
|
|
)`)
|
|
}
|
|
|
|
async down() {
|
|
this.schema.raw(`DROP TABLE IF EXISTS spots, spot_files, release_spots, release_passwords, spot_state`)
|
|
}
|
|
}
|