A seeding node fetches a range over one NNTP connection, parses headers
with the live rules, groups them into files in memory (lib/aggregate) and
stores per complete file its release, the file and a completed binary
(SeedService), instead of one parts row per article. What doesn't complete
within the window goes through the regular ingest path as fragments.
`--dry-run` only fetches and aggregates, to measure a node.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first version matched extensions in search_text, which was wrong and
slow: file names enter search_text as stems without extension (so plainly
posted .mkv files were missed), and matching the TOASTed text took 3-4 s
for common types.
- Migration …130: releases.extensions text[] (two to four characters with
a letter; .7z.001 and .r00 count as 7z and rar), kept current by triggers
on files (name, release) and on releases (archive listing), filled with
one sequential pass over files, with a GIN index.
- Search: ext matches with `extensions && ?`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- API: min_size and max_size in MB, ext as a comma-separated list. An
extension matches as a whole word in search_text (file and entry names,
dots as spaces), so "iso" finds .iso files but not "ISOLATION", which a
plain search word matched as a substring. The trigram index serves it.
- Web UI: a size range and an extension field in the filter row, kept in
the URL like the other filters.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Of 5,000 waiting inspection jobs, 3,212 were queued only because their
stem looked random, and 2,044 of those were later RAR volumes
(xyz.part47.rar, .r00) or par2 recovery volumes (.vol03+04.par2). Their
first segment adds nothing: the set is listed from its first volume and
named from its index .par2, as for readable names already. Each one cost a
~700 KB download; inspection pulled ~85 MB/s and fell behind the backfill.
- isCandidate() rejects later volumes (isLaterVolume) before the random-
name rule.
- worker:file only queues files that are candidates, so the ~30% that
inspection skips anyway no longer take queue slots.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Crossposts, reposts and backfill ranges keep bringing articles of files
that worker:file already created. Ingest inserted their header parts again,
and GC's copied-parts pass (which searched all binaries for parts on every
batch) removed only 150k-450k per sweep: parts grew to 68M rows, 56M of
them belonging to finished files, and the 23 GB table paused the backfill.
- Ingest skips header parts (no per-part CRC) for binaries with a file_id.
- GC drops copied parts with a cursor over binaries, 2000 per statement.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Spotnet (CONCEPT D29):
- Index free.pt fully and import spots with their descriptions, categories
and NZBs; a spot renames its release, since Spotnet names are more
trustworthy than names derived from subjects.
- Find passwords in spot descriptions too, and tell "needs a password we
have" from "needs a password we don't have".
Content inspection:
- List the contents of 7z (.7z.001) archives, SFV sets, nicovideo uploads
and other archive-like files; mark yEncBin .bin containers as encrypted.
- Take real names from yEnc headers for randomly named files, and merge
uploads that were split across posts.
Storage (D31, D32):
- Store segment lists zstd-compressed behind the FileStore accessor.
- group:backfill indexes older headers at low priority and pauses when
the NVMe or HDD part of the database gets too large.
Grouping (D35, D36):
- Lone header segments wait in staged_segments until a second segment of
their file arrives, which drops the per-article junk binaries.
- Counter grouping joins random-subject uploads by poster, [i/M] counter,
part count and day.
Search and maintenance (D33, D34):
- A trigram-indexed search_text column, smaller indexes, estimated stats.
- Passes that clean up nonsense names; weekly index maintenance; GC budgets
per kind.
- Web UI: grey out stale results while a search runs and cancel superseded
requests. Count only up to 1,000 matches for the page, which made filter
clicks take seconds (Newznab keeps the 10,000 cap).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
