Spots: the spotter and the spot inspector, passwords kept apart
Two processes, one NNTP connection each (the coordinator 40 -> 38): - spots:read (service spotter): free.pt's overviews after the last article seen, each spot's full header verified (lib/spotnet from voyager 1, as Spotweb), its description, website and NZB articles; the password its title or text names; delete/dispose messages applied. - spots:inspect (service spot-inspector): each spot's compressed NZB read, its files listed (spot_files), each matched to the header pass's release by the same release identity in one of its groups within 48 h (release_spots); unmatched spots matched again hourly, after 24 h a release of their own (source spot) with the spot's NZB. - release_passwords: per release and password, its source (nzb, spot-title, spot-text) and spot; the catalog keeps only passworded. XoverClient.head(); PROBE_CONNECTIONS default 10. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
3fd08b598a
commit
7df5a638ca
20 files changed
+1782
-8
No files matched your search
+33
-2
@@ -247,9 +247,9 @@ with 68,032 releases with a password):
|
||||
|
||||
| Feature | Voyager 1 | In voyager 2 |
|
||||
|---|---|---|
|
||||
| Spotnet spots | D29 | not yet; 2.76 M spots in the export |
|
||||
| Spotnet spots | D29 | the spotter and the spot inspector (below); new spots only, the export's not imported |
|
||||
| Posted NZBs read into releases (also NZBs inside archives) | D26, D28 | not yet |
|
||||
| Passwords (from spots, NZB meta, names) | D29, D26 | only *whether* passworded (the prober); the export's 68 k passwords not imported |
|
||||
| Passwords (from spots, NZB meta, names) | D29, D26 | from spots (title, text) and their NZBs' meta, in `release_passwords`; else only *whether* passworded (the prober); the export's 68 k not imported |
|
||||
| SFV grouping and naming | D34 | not yet (SFV files aren't probe candidates) |
|
||||
| Complete RAR listings by walking volume headers | D40 | no: the prober reads the first volume's first article only |
|
||||
| 7z listings | D19 | no: `7z-tail` candidates are queued, not probed |
|
||||
@@ -258,6 +258,37 @@ with 68,032 releases with a password):
|
||||
|
||||
Live indexing (the tail) runs since 2026-10-09.
|
||||
|
||||
**Spots** (after voyager 1's D29; `lib/spotnet` ported as is). Two
|
||||
processes, one NNTP connection each:
|
||||
|
||||
- **The spotter** (`spots:read`, service `spotter`) reads free.pt's
|
||||
overviews after the last article it saw (`spot_state`; the first time the
|
||||
last `SPOT_BACKFILL`, 20,000, ~3 months), every `SPOT_EVERY_SECONDS`
|
||||
(120). For each spot it reads the full header: the signature is checked
|
||||
there as Spotweb does (Spotnet's keys 2–4; self-signed key 7 by hashcash),
|
||||
and the X-XML gives the description, the website and the NZB's articles.
|
||||
A password the title (`{{pw}}`) or the description names
|
||||
(`passwordFromSpot`) is kept with the spot, with its source. A
|
||||
delete/dispose message marks its target `removed` (a personal one only
|
||||
from the same spotter within 5 days) and takes its links and passwords
|
||||
back from the releases.
|
||||
- **The spot inspector** (`spots:inspect`, service `spot-inspector`) looks
|
||||
inside each listed spot, newest first: it reads the compressed NZB (≤
|
||||
`SPOT_MAX_NZB_SEGMENTS` articles in alt.binaries.ftd), lists its files
|
||||
in `spot_files` (named as the header pass names them, `parseSubject`),
|
||||
and matches each file to the release the header pass built for it: the
|
||||
same release identity (`releaseIdentity` over the file's subject and
|
||||
poster) in one of its groups, the release posted nearest the file within
|
||||
48 h. Matched releases are linked in `release_spots`. Placeholder NZBs
|
||||
(`dummy`), gone articles (`nzb-gone`) and unreadable ones are noted on the
|
||||
spot. Unmatched spots are matched again every hour; after
|
||||
`SPOT_STANDALONE_AFTER_HOURS` (24) a spot becomes a release of its own
|
||||
(source `spot`), whose NZB is the spot's (kept gzipped in `spots.nzb`).
|
||||
- **Passwords are kept apart** in `release_passwords` (release, password,
|
||||
source: `nzb` for the NZB's `<meta type="password">`, else `spot-title` or
|
||||
`spot-text`, and the spot), not on the catalog row; the catalog only says
|
||||
`passworded`.
|
||||
|
||||
## Decisions
|
||||
|
||||
a. Chunk payload: a write-once `bytea` table in Postgres, partitioned by
|
||||
|
||||
Reference in new issue
Block a user