Rebuild: phase 2's prober reads the probe queue
probe:run (compose service prober, PROBE_CONNECTIONS, default 6) claims waiting candidates of settled releases (lone, then par2, then archive), fetches each article with BODY over connections of its own (no compression) and inspects it: the yEnc name and size, the content type from the first bytes, a par2 set's files, a RAR volume's entries and whether they're encrypted. Results per release and kind in release_inspections (CHUNK_TABLESPACE, the HDD); on the release passworded, inner_name (the largest content name) and inspected_at. 430 = missing, not yEnc = error, both final; connection trouble puts the candidate back. Fetch and parse in one process: an article takes ~0.1 s per connection, parsing a few ms (no queue in between). Parsers from voyager 1 (lib/yenc, lib/rar, lib/par2, lib/content sniff) with their tests; XoverClient.body; yencode. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
d72fe2f13f
commit
8bd6d6aa5f
30 files changed
+1666
-7
No files matched your search
@@ -65,6 +65,16 @@ export default {
|
||||
* under 2 minutes old as on ones over 10 (~25–35%; LOG.md 2026-10-09 (16)).
|
||||
*/
|
||||
nntpConnectionMaxMs: env.get('NNTP_CONNECTION_MAX_MINUTES', 0) * 60 * 1000,
|
||||
probe: {
|
||||
/**
|
||||
* NNTP connections the prober reads articles with, besides the
|
||||
* coordinator's: the account has 50, the coordinator 40 + 1 plain, and a
|
||||
* couple stay free for downloads. ~7 MB/s each (~10 articles/s).
|
||||
*/
|
||||
connections: env.get('PROBE_CONNECTIONS', 6),
|
||||
/** Candidates claimed at a time. */
|
||||
batch: 200,
|
||||
},
|
||||
spool: {
|
||||
dir: env.get('SPOOL_DIR', app.tmpPath('spool')),
|
||||
/** Ranges kept fetched ahead, per active lease. */
|
||||
|
||||
Reference in new issue
Block a user