Spots: the spotter and the spot inspector, passwords kept apart
Test and deploy / test (push) Successful in 50s
Test and deploy / edge (push) Successful in 2m1s
Test and deploy / deploy (push) Successful in 1m28s

Two processes, one NNTP connection each (the coordinator 40 -> 38):
- spots:read (service spotter): free.pt's overviews after the last article
  seen, each spot's full header verified (lib/spotnet from voyager 1, as
  Spotweb), its description, website and NZB articles; the password its
  title or text names; delete/dispose messages applied.
- spots:inspect (service spot-inspector): each spot's compressed NZB read,
  its files listed (spot_files), each matched to the header pass's release
  by the same release identity in one of its groups within 48 h
  (release_spots); unmatched spots matched again hourly, after 24 h a
  release of their own (source spot) with the spot's NZB.
- release_passwords: per release and password, its source (nzb, spot-title,
  spot-text) and spot; the catalog keeps only passworded.
XoverClient.head(); PROBE_CONNECTIONS default 10.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Daan MeijerandClaude Opus 5.5 committed 2026-10-13 12:10:10 +02:00
1 parent 3fd08b598a
commit 7df5a638ca
20 files changed
+1782 -8

No files matched your search

+33 -2
View File
@@ -247,9 +247,9 @@ with 68,032 releases with a password):
| Feature | Voyager 1 | In voyager 2 |
|---|---|---|
| Spotnet spots | D29 | not yet; 2.76 M spots in the export |
| Spotnet spots | D29 | the spotter and the spot inspector (below); new spots only, the export's not imported |
| Posted NZBs read into releases (also NZBs inside archives) | D26, D28 | not yet |
| Passwords (from spots, NZB meta, names) | D29, D26 | only *whether* passworded (the prober); the export's 68 k passwords not imported |
| Passwords (from spots, NZB meta, names) | D29, D26 | from spots (title, text) and their NZBs' meta, in `release_passwords`; else only *whether* passworded (the prober); the export's 68 k not imported |
| SFV grouping and naming | D34 | not yet (SFV files aren't probe candidates) |
| Complete RAR listings by walking volume headers | D40 | no: the prober reads the first volume's first article only |
| 7z listings | D19 | no: `7z-tail` candidates are queued, not probed |
@@ -258,6 +258,37 @@ with 68,032 releases with a password):
Live indexing (the tail) runs since 2026-10-09.
**Spots** (after voyager 1's D29; `lib/spotnet` ported as is). Two
processes, one NNTP connection each:
- **The spotter** (`spots:read`, service `spotter`) reads free.pt's
overviews after the last article it saw (`spot_state`; the first time the
last `SPOT_BACKFILL`, 20,000, ~3 months), every `SPOT_EVERY_SECONDS`
(120). For each spot it reads the full header: the signature is checked
there as Spotweb does (Spotnet's keys 2–4; self-signed key 7 by hashcash),
and the X-XML gives the description, the website and the NZB's articles.
A password the title (`{{pw}}`) or the description names
(`passwordFromSpot`) is kept with the spot, with its source. A
delete/dispose message marks its target `removed` (a personal one only
from the same spotter within 5 days) and takes its links and passwords
back from the releases.
- **The spot inspector** (`spots:inspect`, service `spot-inspector`) looks
inside each listed spot, newest first: it reads the compressed NZB (≤
`SPOT_MAX_NZB_SEGMENTS` articles in alt.binaries.ftd), lists its files
in `spot_files` (named as the header pass names them, `parseSubject`),
and matches each file to the release the header pass built for it: the
same release identity (`releaseIdentity` over the file's subject and
poster) in one of its groups, the release posted nearest the file within
48 h. Matched releases are linked in `release_spots`. Placeholder NZBs
(`dummy`), gone articles (`nzb-gone`) and unreadable ones are noted on the
spot. Unmatched spots are matched again every hour; after
`SPOT_STANDALONE_AFTER_HOURS` (24) a spot becomes a release of its own
(source `spot`), whose NZB is the spot's (kept gzipped in `spots.nzb`).
- **Passwords are kept apart** in `release_passwords` (release, password,
source: `nzb` for the NZB's `<meta type="password">`, else `spot-title` or
`spot-text`, and the spot), not on the catalog row; the catalog only says
`passworded`.
## Decisions
a. Chunk payload: a write-once `bytea` table in Postgres, partitioned by