Two processes, one NNTP connection each (the coordinator 40 -> 38):
- spots:read (service spotter): free.pt's overviews after the last article
seen, each spot's full header verified (lib/spotnet from voyager 1, as
Spotweb), its description, website and NZB articles; the password its
title or text names; delete/dispose messages applied.
- spots:inspect (service spot-inspector): each spot's compressed NZB read,
its files listed (spot_files), each matched to the header pass's release
by the same release identity in one of its groups within 48 h
(release_spots); unmatched spots matched again hourly, after 24 h a
release of their own (source spot) with the spot's NZB.
- release_passwords: per release and password, its source (nzb, spot-title,
spot-text) and spot; the catalog keeps only passworded.
XoverClient.head(); PROBE_CONNECTIONS default 10.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The rebuild branch starts over (REBUILD.md): the API starter kit, with
Postgres instead of SQLite (its own database, DB_SEARCH_PATH for a
shared one in development) and the #lib alias. Phase 1's code is
imported from the current app module by module.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
passwordFromSpot finds 30 more passwords in the stored descriptions (480
→ 510): words between keyword and separator, trailing "!!", the value on
the next line, "wachtwoord is x", "… wachtwoord. Deze is: x", "WW x" as a
whole line.
Listed spots were read from their overview only, so the NZB's own
password and the description were unknown for 2.27M of them.
SpotService.readListing reads the spot's article and NZB without
importing files: details, the NZB password (over a spot-text one), the
spot-text password, placeholder NZBs. worker:spots does a batch when
idle; spot:postprocess works through the history (nzb_read_at).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Spotnet (CONCEPT D29):
- Index free.pt fully and import spots with their descriptions, categories
and NZBs; a spot renames its release, since Spotnet names are more
trustworthy than names derived from subjects.
- Find passwords in spot descriptions too, and tell "needs a password we
have" from "needs a password we don't have".
Content inspection:
- List the contents of 7z (.7z.001) archives, SFV sets, nicovideo uploads
and other archive-like files; mark yEncBin .bin containers as encrypted.
- Take real names from yEnc headers for randomly named files, and merge
uploads that were split across posts.
Storage (D31, D32):
- Store segment lists zstd-compressed behind the FileStore accessor.
- group:backfill indexes older headers at low priority and pauses when
the NVMe or HDD part of the database gets too large.
Grouping (D35, D36):
- Lone header segments wait in staged_segments until a second segment of
their file arrives, which drops the per-article junk binaries.
- Counter grouping joins random-subject uploads by poster, [i/M] counter,
part count and day.
Search and maintenance (D33, D34):
- A trigram-indexed search_text column, smaller indexes, estimated stats.
- Passes that clean up nonsense names; weekly index maintenance; GC budgets
per kind.
- Web UI: grey out stale results while a search runs and cancel superseded
requests. Count only up to 1,000 matches for the page, which made filter
clicks take seconds (Newznab keeps the 10,000 cap).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
