A seeding node that waited over two minutes for the seed API (restarting)
had its idle NNTP connection closed by its own timeout, and gave its range
back. Now the timeout runs only while a reply is awaited.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- lib/nntp/xover_client: a minimal one-connection client for seed:range
with XFEATURE COMPRESS GZIP (a zlib stream after "224 … [COMPRESS=GZIP]",
then a "." line; 420 for an empty range). About 6x less header traffic.
Overviews come out with the field names nntp-js uses.
- SfvService.claimAll: one sfv_entries lookup per stored batch.
- Fragments go through the regular ingest path beside fetching (at most
two batches in flight); a failure fails the run at the end.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A seeding node fetches a range over one NNTP connection, parses headers
with the live rules, groups them into files in memory (lib/aggregate) and
stores per complete file its release, the file and a completed binary
(SeedService), instead of one parts row per article. What doesn't complete
within the window goes through the regular ingest path as fragments.
`--dry-run` only fetches and aggregates, to measure a node.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Of 5,000 waiting inspection jobs, 3,212 were queued only because their
stem looked random, and 2,044 of those were later RAR volumes
(xyz.part47.rar, .r00) or par2 recovery volumes (.vol03+04.par2). Their
first segment adds nothing: the set is listed from its first volume and
named from its index .par2, as for readable names already. Each one cost a
~700 KB download; inspection pulled ~85 MB/s and fell behind the backfill.
- isCandidate() rejects later volumes (isLaterVolume) before the random-
name rule.
- worker:file only queues files that are candidates, so the ~30% that
inspection skips anyway no longer take queue slots.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Spotnet (CONCEPT D29):
- Index free.pt fully and import spots with their descriptions, categories
and NZBs; a spot renames its release, since Spotnet names are more
trustworthy than names derived from subjects.
- Find passwords in spot descriptions too, and tell "needs a password we
have" from "needs a password we don't have".
Content inspection:
- List the contents of 7z (.7z.001) archives, SFV sets, nicovideo uploads
and other archive-like files; mark yEncBin .bin containers as encrypted.
- Take real names from yEnc headers for randomly named files, and merge
uploads that were split across posts.
Storage (D31, D32):
- Store segment lists zstd-compressed behind the FileStore accessor.
- group:backfill indexes older headers at low priority and pauses when
the NVMe or HDD part of the database gets too large.
Grouping (D35, D36):
- Lone header segments wait in staged_segments until a second segment of
their file arrives, which drops the per-article junk binaries.
- Counter grouping joins random-subject uploads by poster, [i/M] counter,
part count and day.
Search and maintenance (D33, D34):
- A trigram-indexed search_text column, smaller indexes, estimated stats.
- Passes that clean up nonsense names; weekly index maintenance; GC budgets
per kind.
- Web UI: grey out stale results while a search runs and cancel superseded
requests. Count only up to 1,000 matches for the page, which made filter
clicks take seconds (Newznab keeps the 10,000 cap).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
