Commit Graph
60 Commits
Author SHA1 Message Date
Daan MeijerandClaude Opus 5.5 cfe85abc3e Rebuild: chunks the server's compression cuts short are fetched plain
Test and deploy / test (push) Successful in 23s
Test and deploy / deploy (push) Skipped
Eweka sends some ranges' compressed XOVER truncated, the same bytes
every time, while the plain reply is whole (a.b.models 3317215-3367214:
1,675,600 bytes that don't inflate, twice; 50,000 lines plain;
tools/false_end.ts). Not a false end of reply. The worker's corrupt
report removed the chunk and the range was fetched again, with the same
bytes: one range was reported 540 times. (Before this morning's change
the inflate check waited for more data until the 10-minute timeout,
like voyager 1's 'chunk never answered' ranges.)

A reported chunk's start goes into ranges.plain_chunks; the prefetcher
fetches those over one connection without XFEATURE COMPRESS GZIP
(NntpPool.usePlain, opened on demand) and deflates them here. A plain
chunk reported again sets the range aside (status failed, error). The
capture (edge:bench --capture) fetches such a chunk plain too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 13:51:25 +02:00
Daan MeijerandClaude Opus 5.5 0613984de9 Rebuild: re-uploads are releases of their own
Test and deploy / test (push) Successful in 26s
Test and deploy / deploy (push) Skipped
A poster re-uploading under the same name made one release of all
uploads: in a sample, 24% of releases with 2+ files had a file name more
than once (4.3% of files extra copies), 70% of those copies posted a day
or more apart; their NZBs listed each copy.

- File assembly keeps a re-upload's copy apart: a part already in every
  open copy, posted more than 3 h from them, starts another copy; a
  missing part fills the nearest copy (a re-send of what was missing),
  a re-sent part near a copy that has it is ignored as before.
- A key's files in a batch split into uploads where posting pauses for
  more than 3 h, counted from the latest article posted so far (a file
  that takes hours to post doesn't split it); each upload is judged and
  chunked on its own (ReleaseChunk.upload, .postedLast; candidates carry
  their upload).
- The catalog matches a chunk to the release of its key whose articles
  the range's group owns and whose posting window is within 3 h, else
  starts one (key, key:2, …): releases get identity_key, group_id,
  last_posted_at. Keys are locked in order (advisory locks), so
  concurrent batches neither deadlock nor both start one; the merger
  writes in key order too. The same upload multi-posted to other groups
  is a release per group.
- The gap: 91% of uploads post within 3 h; 73% of re-uploaded copies
  come later (releases:upload-gaps). releases:duplicates measures the
  duplicate names.

Both implementations; check.ts compares the windows and uploads too
(identical on all captures; Rust tests mirror the TypeScript cases).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 13:31:35 +02:00
Daan MeijerandClaude Opus 5.5 14758a5b7a Rebuild: image sets, documents and audio albums as releases
Test and deploy / test (push) Successful in 22s
Test and deploy / deploy (push) Skipped
Picture groups made a release per image: 501k single-file releases in
the first 24 ranges of the bottom 500 groups (a.b.models 385k), ~120M
for all of them. Counters written out ('File 027 of 132', '[12of42]',
'104 of 472', '[2239-6389]', '02/17', '0097-1574'; only when the first
number is at most the second, so '[1948-12]' stays) and per-file sizes
('76.88 MB', '335555 Bytes') kept every subject unique.

- subjectRemainder removes those counters and sizes, and empty brackets.
- A subject-named release is bounded by its counter's total
  (counterTotal: the first form found): uploads a poster names alike
  stay apart, an upload's files (par2, nfo) together.
- Images and audio tracks don't take the poster-tag rule: the subject
  names the set or album. An image set without a counter: the image's
  name up to its last number, without numbers.
- Documents and comics (pdf, epub, mobi, azw3, djvu, cbr, cbz, cb7, chm,
  lit, doc, docx) are a release each: collections post hundreds under
  one subject ('cw-horror&mystery').

releases:regroup --group re-keys a group's indexed releases together:
cheerleaders 55,741 -> 3,165 releases, models 385,298 -> 40,313, German
audiobooks 4,106 -> 1,724, playstation 3,309 -> 2,345; comics and
e-books a release per item; boneless unchanged. Ported to edge-bench-rs;
check.ts: identical on all captures.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 12:21:00 +02:00
Daan MeijerandClaude Opus 5.5 59481037fb Rebuild: a poster's constant tag no longer makes one release of all their uploads
Test and deploy / test (push) Successful in 28s
Test and deploy / deploy (push) Skipped
Release 145637 held four albums: the poster puts '[KiLL.dA.FuCkIn.
EaStErRaBbIt]----' before every upload, and the text outside the quotes
(when it names a release) was the key with the poster. The same made
one release of a poster's albums under '(PowBeertje80 Posted)' (a post
id, with the same [i/M]), of masterposts' films, and of one volume of
every set in a DVD9 post.

Now, when the file's stem describes something itself (12+ letters and
digits, two words of 3+ letters) and the text outside the quotes shares
no word with it (4+ letters as part of the stem, 3-letter words and
numbers whole; boilerplate like www/net/posted not counted; no 5+ digit
per-post id), that text is taken for the poster's tag and the stem
says which release the file is. Albums of track files ('FTA … Ketchup'),
'DIE_ZEIT_35' with '35_China_dl', 'FTD769997' posts, scene names with
abbreviated file names ('grp-sm') and generic sidecars ('file_id.diz')
keep their grouping.

releases:regroup re-keys a sample of the catalog with the current rule:
on homelab's test data 145637 → 4 albums, 111566 → 17 albums of 27
files, the rest of the subject releases that change mostly lose an ad
image ('Flyer-download-2-Day.gif'); stem releases unchanged. Ported to
edge-bench-rs: check.ts still finds all captures identical (ath 2013:
16 → 76 releases on both sides).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 11:23:33 +02:00
Daan MeijerandClaude Opus 5.5 57dfa3d749 Rebuild: rules:signatures, and voyager 1's 29 learned garbage signatures
rules:signatures publishes a rule set version whose dropped signatures
come from a file of tool<TAB>group lines (--add to add to the current
ones). database/data/voyager1_signatures.tsv: voyager 1's dropped ones
(>=100 probes, <=0.5% named), with their probe counts; the two builds
name the posting tools the same way.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 10:42:00 +02:00
Daan MeijerandClaude Opus 5.5 fd0f2bde75 Rebuild: the live tail
coordinator:run plans the articles after each group's last range every
TAIL_EVERY_SECONDS (120): one LIST ACTIVE per name prefix gives every
group's newest article, and new articles become ranges at priority -1,
leased and prefetched before any history. A group gets a tail range once
it has TAIL_MIN_ARTICLES (100k, a chunk) new or its last planning is
TAIL_MAX_DELAY_MINUTES (15) ago, so quiet groups don't make a range every
round. History (planAll, hourly) is now only for groups not planned yet.
ranges:plan --tail plans one round.

Found on the way: with XFEATURE COMPRESS GZIP on, Eweka compresses LIST
ACTIVE's reply too ("215 NewsGroups Follow [COMPRESS=GZIP]"), and
listActive waited for a plain end that never came (groups:add --top too).
It reads the compressed reply now; the fake server in the test does both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 10:41:16 +02:00
Daan MeijerandClaude Opus 5.5 42b6cac3ee Rebuild: edge:node --max-mbit caps a node's download
Test and deploy / test (push) Successful in 38s
Test and deploy / deploy (push) Skipped
For workers on a shared line (poweredge-kantoor: 160 Mbit/s): the range
stream is read at most at the cap, TCP holds the coordinator back. The
cap is the node's, split over its processes (also EDGE_MAX_MBIT).
REBUILD.md: where workers run (connections set the speed; remote workers
take CPU off homelab).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 09:01:57 +02:00
Daan MeijerandClaude Opus 5.5 a979b5d11c Rebuild: the coordinator spools replies without inflating them
On homelab (1 Gbit, 20 connections, ~630k articles/s) the coordinator
used 1.3-1.7 cores, mostly GC of the text it inflated only to know a
compressed reply was complete, plus a copy of every reply. A raw reply
is now complete when it ends in ".\r\n" (no copy): 0.4-0.5 cores at
the same rate.

".\r\n" can also end a network read inside the compressed data (~1 in
16M reads, a few dozen times over the whole target). Such a chunk
doesn't inflate on the worker: it reports it (POST ranges/:id/corrupt),
the spooled chunk is removed and the range goes back to the queue; the
prefetcher fetches it again. The rest of that reply breaks the
connection's next command, so the connection is replaced.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 07:40:06 +02:00
Daan MeijerandClaude Opus 5.5 69f1cc63b2 Rebuild: coordinator:run --shard=i/n splits prefetching over processes
The coordinator's event loop was 16.5% busy at ~10 MB/s of compressed
XOVER (socket reads, buffers, GC, spool writes; nothing stands out), so
one process tops out around 60 MB/s: less than 50 connections bring
(~1M articles/s, ~70-86 MB/s). A shard prefetches the ranges with
id % n = i - 1 with its own NNTP_CONNECTIONS; only shard 1 plans, merges
and settles. Two shards of 5 connections fetched what one of 10 did.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 03:36:24 +02:00
Daan MeijerandClaude Opus 5.5 462da5c225 Rebuild: a stopping worker notices it on keep-alive frames too
It checked only after a data frame, so a worker waiting for its range's
spool (keep-alive frames only) never stopped on SIGTERM.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 03:26:20 +02:00
Daan MeijerandClaude Opus 5.5 883d85e91d Rebuild: the coordinator stops on SIGTERM
It waited for every chunk in flight, and a connection the server held
back (after killed coordinators left sessions open) waits up to the
10-minute reply timeout: it never seemed to stop. It now waits at most
15 s, then closes all connections, busy ones too; the greeting and login
time out after 30 s. And run() ends with terminate(), as edge:node does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 03:20:11 +02:00
Daan MeijerandClaude Opus 5.5 168605148f Rebuild: the worker joins stream pieces once per frame, gzips posts off its thread
decodeFrames joined each network piece (~64 KB) onto the frame so far:
a chunk of MBs copied over and over, 5.4% of a worker's time in an
end-to-end profile (the bench reads files, so it never showed). Pieces
are now joined once a frame is complete. A batch post's gzip (1.6%) runs
on a thread-pool thread.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 03:06:17 +02:00
Daan MeijerandClaude Opus 5.5 bd1c846f27 Rebuild: prefetching no longer waits for merging, nor stalls on finished ranges
The coordinator ran the prefetch tick in the same loop as the merger
(up to 30 s a minute): its connections went idle once their targets
were fetched, and the first leases after a start found nothing spooled.
The prefetcher now ticks in a loop of its own.

The prefetcher took the first planned ranges with nothing spooled, even
ones with nothing left to fetch (next_id past the end): two such ranges
(edge:bench's, set back to planned) kept it from fetching anything. It
and the lease query now skip those, and edge:bench marks its range done.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 03:04:00 +02:00
Daan MeijerandClaude Opus 5.5 821de668d0 Rebuild: the merger stops when a pass over the file keys decides nothing
Test and deploy / test (push) Successful in 21s
Test and deploy / deploy (push) Skipped
Waiting files counted as work, so the merge loop re-read them for its
whole 30 s budget every minute: Postgres used ~a core even without
workers. Now ~4% at rest, and storing a boneless range costs Postgres
~0.7–1.9 s of CPU (it was 33 s with the loop running). Chunk payloads
are stored EXTERNAL (zstd already; no pglz attempt).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 02:49:45 +02:00
Daan MeijerandClaude Opus 5.5 73fc9aabdf Rebuild: strings that outlive their chunk are copied (worker 1.14 → 0.66 GB)
Test and deploy / test (push) Successful in 18s
Test and deploy / deploy (push) Skipped
A slice of a chunk's text keeps the whole text alive (tens of MB) for as
long as the file it belongs to stays open: message-ids, file details and
cache keys are now copies. edge:bench on boneless: peak RSS 1,137 → ~660
MB, external memory 570 → 40–75 MB, the same speed (~190k headers/s);
the same results on 800k real lines. edge:bench prints the breakdown.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 02:36:34 +02:00
Daan MeijerandClaude Opus 5.5 80c920f678 Rebuild: settling quiet releases
Test and deploy / test (push) Successful in 21s
Test and deploy / deploy (push) Skipped
Hourly in coordinator:run: a release unchanged for 6 hours is settled,
complete when every file its [i/N] counter announced was found, unknown
without a counter (phase 2 reads par2 sets for those). Partial index on
unsettled releases.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 02:33:47 +02:00
Daan MeijerandClaude Opus 5.5 e9592dfd33 Rebuild: chunk format v2 — nyuu and hex message-ids as bits
Test and deploy / test (push) Successful in 20s
Test and deploy / deploy (push) Skipped
nyuu's 12 capital+small letter pairs in 15 bytes (10 bits a pair), its
timestamp as a delta within the file, the suffix once per file;
ngPost-style 32 hex digits in 16 bytes; other ids as zstd text as before.
Recent boneless: 25.2 → 20.3 B per segment (and 25.0 → 19.5 on the next
range); a 2013 ath range stays at 17.9 (JBinUp ids are text). Every chunk
of three real ranges decodes to itself (edge:bench --verify). Encoding is
no slower than v1 (less text for zstd). Version 1 chunks stay readable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 02:31:59 +02:00
Daan MeijerandClaude Opus 5.5 31493a98be Rebuild: merger rounds with one neighbour query, a cursor and a time budget; planning skips failing groups
Test and deploy / test (push) Successful in 22s
Test and deploy / deploy (push) Skipped
The neighbour check was three queries per file (~2,000 files/minute); now
one per round of 20k files, with a cursor so waiting files don't take
every round's places, and rounds until nothing is left or 30 s. On the
dev backlog (old-style fragments): ~18k files/minute in the 30 s budget;
at full throughput the new fragment rule leaves ~1,200 files/minute.
A group that fails GROUP (gone from the server) no longer stops planning
the others; edge:bench's groups are inactive.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 02:25:45 +02:00
Daan MeijerandClaude Opus 5.5 b862201991 Rebuild: the fragment merger
Test and deploy / test (push) Successful in 19s
Test and deploy / deploy (push) Skipped
Files cut by a range's edge are put together once the ranges around them
are done (coordinator:run, every minute): a complete file is judged by the
rules and written as a release chunk with probe candidates; one still
missing parts while both neighbouring ranges are done is dropped and
counted (incomplete/edge). A neighbour not planned yet counts as not done,
except before the group's oldest article. BatchStore.writeReleases is the
shared write path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 02:19:53 +02:00
Daan MeijerandClaude Opus 5.5 5ba93f558c Rebuild: XOVER chunks of 100k lines; batches by articles
Test and deploy / test (push) Successful in 21s
Test and deploy / deploy (push) Skipped
Eweka takes ~1.5–2 s per XOVER command whatever its size. One connection
on recent a.b.boneless: 9–15k lines/s in chunks of 25k, ~20k in 50k,
~30k in 100k–200k. Four connections at once: 15k each at 50k, 22k each
at 100k (the same as one connection alone at that moment): connections
don't slow each other. CHUNK_ARTICLES defaults to 100k; edge:node posts a
batch every --batch-articles (500k) instead of every N chunks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 02:09:04 +02:00
Daan MeijerandClaude Opus 5.5 ead2790515 Rebuild: edge:node --processes (one worker per core, supervised)
Test and deploy / test (push) Successful in 19s
Test and deploy / deploy (push) Skipped
On the 4-core workstation (shared with coordinator, web server and
database): 3 processes indexed 6 boneless ranges (12M headers) in 37 s,
~324k headers/s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 01:32:37 +02:00
Daan MeijerandClaude Opus 5.5 fa7d4f3394 Rebuild: end-to-end fixes — spool accounting, keep-alive frames, resume point, overlapped worker
Test and deploy / test (push) Successful in 21s
Test and deploy / deploy (push) Skipped
End to end (workstation, recent a.b.boneless ranges from the spool): a 2M
range in 10–12 s, 166k–202k headers/s per worker process (edge:bench on
one core: 215k).

- Prefetcher: spooled_at was set against a total counted from the range's
  resume point, which shrinks as the worker advances: the last chunks were
  never fetched and the worker waited. Now counted against the whole
  range, only new chunks counted, and leased ranges are always checked
  against the files. Planned ranges ahead are bounded (it had spooled ever
  more); the spool size is measured every 30 s (a stat per file).
- The stream sends an empty frame every 20 s while waiting for a chunk
  (Node's fetch gave up on a quiet body after 5 minutes).
- A range taken over midway: the first batch starts at the resume point
  (it started at the range's start and was refused).
- edge:node inflates the next chunk on the thread pool while indexing the
  current one, and doesn't wait for a post before indexing on (posts stay
  in order; a failed post stops the range).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 01:31:19 +02:00
Daan MeijerandClaude Opus 5.5 638cfa1dcd Rebuild: a lean overview scan for the edge worker
Test and deploy / test (push) Successful in 22s
Test and deploy / deploy (push) Skipped
The worker reads the XOVER reply's text directly: fields by position, the
owner from the Xref without arrays (crossposts leave before any subject
parsing), a subject parsed once per file (cached without its segment
counter), a file's details as one shared object, dates as numbers.
Garbage collection had been ~22% of the time.

Same results as parseOverviews + segmentFromHeader on 800k real lines.
edge:bench (one core): boneless 137k → 193k headers/s (CPU 21 → 12.8 s),
ath (2013) 143k → 303k; peak RSS 1.15 / 0.87 GB.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 01:13:41 +02:00
Daan MeijerandClaude Opus 5.5 cd06f96f85 Rebuild: first end-to-end runs; prefetch per chunk, executable rule, wire fixes
Test and deploy / test (push) Successful in 20s
Test and deploy / deploy (push) Skipped
End to end on the workstation (coordinator:run, the API, edge:node, real
NNTP): a.b.e-book.french and a.b.cd.image.playstation2.dvdiso ranges.

- Prefetcher: every connection takes the next missing chunk (leased
  ranges first, in order); one connection per range (~26k lines/s) kept
  the worker waiting
- executable rule: releases of bare executables (malware floods named
  after films and software) are dropped; they were 92% of the catalog rows
  of the test ranges, 21% of the segments
- fragments per file (details once) and gzipped batch bodies; the JSON
  limit (1 MB by default) raised to 256 MB, empty strings kept (a rule
  count's key is '')
- release_chunks.batch_from is a bigint (boneless article numbers)
- tests get their own database; edge:bench --store times BatchStore:
  1.3 s for a boneless range (1.24 ms per release chunk)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 01:07:11 +02:00
Daan MeijerandClaude Opus 5.5 d5f168d8d2 Rebuild: coordinator (planner, NNTP pool, spool, prefetcher, leases, rule sets), node API and the edge:node worker
Test and deploy / test (push) Successful in 21s
Test and deploy / deploy (push) Skipped
- groups:add / ranges:plan / nodes:create / coordinator:run commands
- the node API: lease, the range's chunks as one framed stream from the
  spool, batches, heartbeat, finish, release (node guard, token per node)
- edge:node: needs COORDINATOR_URL and NODE_TOKEN only (placeholders for
  the rest), posts a batch every 10 chunks, gives its range back on SIGTERM
- releases are gathered per batch: 1.04 chunk rows per release at 10
  chunks per batch (1.35 when built per XOVER chunk)
- partitions are checked per batch instead of remembered

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 00:31:06 +02:00
Daan MeijerandClaude Opus 5.5 a18ead666b Rebuild: edge:bench, and its result in the spec (TypeScript is fast enough)
Test and deploy / test (push) Successful in 19s
Test and deploy / deploy (push) Skipped
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 00:17:26 +02:00
Daan MeijerandClaude Opus 5.5 96f0ade10e Rebuild: fresh AdonisJS 7 API app on Postgres
The rebuild branch starts over (REBUILD.md): the API starter kit, with
Postgres instead of SQLite (its own database, DB_SEARCH_PATH for a
shared one in development) and the #lib alias. Phase 1's code is
imported from the current app module by module.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 00:05:24 +02:00
Daan MeijerandClaude Opus 5.5 fad1716d3a Content type and archive listing move from files to file_details
Test and deploy / test (push) Successful in 1m26s
Test and deploy / deploy (push) Successful in 1m54s
Every probe rewrote its file's row on the HDD (content_type, archive):
a new row version on a random page and an entry in each of the five
indexes. They now live in file_details on the NVMe; files rows are
inserted once. files:move-details copies the existing values after
the deploy; a later migration drops the old columns.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 19:36:36 +02:00
Daan MeijerandClaude Opus 5.5 ee898be8c8 Spots: more password phrasings; read listed spots' articles and NZBs
Test and deploy / test (push) Successful in 57s
Test and deploy / deploy (push) Successful in 4m18s
passwordFromSpot finds 30 more passwords in the stored descriptions (480
→ 510): words between keyword and separator, trailing "!!", the value on
the next line, "wachtwoord is x", "… wachtwoord. Deze is: x", "WW x" as a
whole line.

Listed spots were read from their overview only, so the NZB's own
password and the description were unknown for 2.27M of them.
SpotService.readListing reads the spot's article and NZB without
importing files: details, the NZB password (over a spot-text one), the
spot-text password, placeholder NZBs. worker:spots does a batch when
idle; spot:postprocess works through the history (nzb_read_at).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 13:46:32 +02:00
Daan MeijerandClaude Opus 5.5 211820067a Drop garbage uploads early (D42)
Test and deploy / test (push) Successful in 1m0s
Test and deploy / deploy (push) Successful in 2m33s
Random-named uploads that never get a real name or password were ~90% of
what seeding stores. Two filters:

- before storing (seeding, ingest): random-named uploads of a dropped
  signature (posting tool from the Message-ID, first group) or of a
  release dropped after its probe aren't stored; 2% of a dropped
  signature's releases are still kept and probed. Stored files of dropped
  signatures aren't probed.
- after a probe: a random-named release that is still unnamed, without
  password, names from files or spot, with all its probes done, and
  encrypted, a container or missing its first article, is deleted and
  tombstoned in dropped_releases.

Signatures are learned by gc:scheduler from a sample of probed releases
(signature_probes); garbage:learn --seed fills it from existing files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 11:21:01 +02:00
Daan MeijerandClaude Opus 5.5 35d0b2da65 Inspection backlog in Postgres; BullMQ gets a working set
Test and deploy / test (push) Successful in 2m14s
Test and deploy / deploy (push) Successful in 2m51s
5.4M waiting inspection jobs took 3.3 GB of Redis and most of its single
thread. Producers now add file ids to inspection_queue; index-scheduler
feeds up to INSPECTION_FEED_MAX (50k) into BullMQ. inspection:from-redis
moves the existing backlog over once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 22:12:13 +02:00
Daan MeijerandClaude Opus 5.5 f8ee7e8662 files partitions: no deadlock, four ahead, and a failure no longer stops the GC sweep
Test and deploy / test (push) Successful in 1m59s
Test and deploy / deploy (push) Successful in 2m37s
Creating a partition deadlocked with file inserts 11 times in an hour, and
each failure ended the GC sweep before its deletes. The parent is now
locked first (lock_timeout 10 s), four partitions are kept ahead, and an
error is logged while the sweep goes on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 15:17:53 +02:00
Daan MeijerandClaude Opus 5.5 e3e3ecc93d Scheduler and worker loops survive a database restart
Test and deploy / test (push) Successful in 1m0s
Test and deploy / deploy (push) Successful in 2m52s
index-scheduler's loop ended when Postgres restarted (the active-groups
query was outside its try), leaving live indexing stopped for ~3 hours
while the process stayed up. Each step's error is now caught and retried
next round; worker:spots and worker:deferred wait 30 s and claim again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 13:02:08 +02:00
Daan MeijerandClaude Opus 5.5 23bbdcc5ee Seed batches take file ids from the insert; merge split uploads hourly
Test and deploy / test (push) Successful in 1m29s
Test and deploy / deploy (push) Successful in 1m58s
SeedService looked up every file of a batch across all files partitions
after inserting it (15% of Postgres' active time); the merge job read
every release on each GC sweep.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 23:09:04 +02:00
Daan MeijerandClaude Opus 5.5 b2cc75fdd0 Keep binaries until inspected; seeding writes binaries only for inspection
Test and deploy / test (push) Successful in 1m24s
Test and deploy / deploy (push) Successful in 6m43s
Par2 naming needs the fingerprint inspection stores on a binary, and the
GC deleted seeded binaries (6 h unfingerprinted, 48 h stale) long before
the inspection queue reached them. binaries.awaiting_inspection protects
them for up to 7 days; the GC deletes use partial indexes so they don't
read past waiting rows. Seeding no longer creates binaries for files that
aren't inspected (~85% of a.b.ath), which only churned the table.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 20:03:43 +02:00
Daan MeijerandClaude Opus 5.5 7893876435 seed:node splits a batch the seed API finds too large
Test and deploy / test (push) Successful in 1m18s
Test and deploy / deploy (push) Successful in 1m54s
Dense recent a.b.ath ranges produced batches over the 64 MB body limit
(413). The node now posts such a batch in halves; only the second moves
the range's resume point.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 15:21:41 +02:00
Daan MeijerandClaude Opus 5.5 f0ff8e200f file_keys: fill in the background, use it once complete
Test and deploy / test (push) Successful in 1m37s
Test and deploy / deploy (push) Successful in 2m23s
Copying all existing keys in the migration read the files rows from the
HDD for over half an hour and held up the deploy. gc:scheduler now copies
them in slices (FILE_KEYS_FILL_SECONDS per sweep) and FileStore checks
every partition until the fill is complete.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 14:54:12 +02:00
Daan MeijerandClaude Opus 5.5 693e6c3244 Seed plans fill every uncovered gap, with priorities; cheaper ingest and file storing
Test and deploy / test (push) Successful in 1m27s
Test and deploy / deploy (push) Failing after 33m7s
- seed:plan plans every part of a group's history no range covers (it
  started below the lowest known range and left a.b.ath's 13 most recent
  months out); seed_ranges.priority orders leases (seed:plan --priority).
- binaries.part_count, kept by the parts insert in the same statement:
  claimCompleted compares it instead of counting parts every batch.
- file_keys: 64-bit hashes of all file keys; FileStore.insert asks it
  before probing every files partition.
- worker:file stores 100 completed binaries per job with set-based queries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 14:18:01 +02:00
Daan MeijerandClaude Opus 5.5 be2112f4ad Par2 size matching 100x faster; release bookkeeping once per statement
Test and deploy / test (push) Successful in 1m33s
Test and deploy / deploy (push) Failing after 1m29s
matchPar2Releases checked every same-size release_files row's set and
date (18 s per batch of body-derived binaries); now an index on
release_files (release_id, size) drops the repeated sizes first: 0.16 s.

File inserts updated their release in four row triggers; one statement
trigger now updates each release once per INSERT. Completeness is noted
in release_dirty and recomputed by gc:scheduler every 30 minutes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 13:48:59 +02:00
Daan MeijerandClaude Opus 5.5 288a6bab8d seed:node spreads its ranges and inspections over worker processes
Test and deploy / test (push) Successful in 2m51s
Test and deploy / deploy (push) Successful in 2m27s
One process ran everything on one core. SEED_PROCESSES (default: the
number of cores) worker processes now each run seed:node with a share of
the slots; stop signals are passed on and crashed workers restarted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 13:25:00 +02:00
Daan MeijerandClaude Opus 5.5 4d17c5a8e8 Release completeness check without reading every binary; SEED_RANGES
Test and deploy / test (push) Successful in 2m15s
Test and deploy / deploy (push) Successful in 2m26s
release_complete() matched each par2-listed file through a correlated
subquery that scanned all binaries per listed file, run twice per
inserted release_files row: one insert of a few thousand rows ran for
20+ minutes. It now reads the release's files once by release id, and
release_files refreshes once per INSERT statement.

seed:node's SEED_CONNECTIONS/--connections is now SEED_RANGES/--ranges.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 12:10:57 +02:00
Daan MeijerandClaude Opus 5.5 2dbca9b152 Content inspection and archive walks on seeding nodes (relay)
Test and deploy / test (push) Successful in 1m49s
Test and deploy / deploy (push) Successful in 2m15s
Inspection is split into prepare (database), probe (lib/content/probe:
downloads, decoding and parsing, no database) and apply (database).
worker:collection runs all three; seed:node with SEED_INSPECTORS leases
jobs from the same queues through the seed API (BullMQ manual leases),
fetches articles through the seed API and posts the probes back.

A node now needs only SEED_URL and SEED_TOKEN: the settings the app
requires but a node never uses get placeholders for seed:node.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 11:17:02 +02:00
Daan MeijerandClaude Opus 5.5 51975fe603 Hidden archive entries no longer name releases
Test and deploy / test (push) Successful in 1m44s
Test and deploy / deploy (push) Successful in 2m6s
One poster adds the same 22 GB hidden '.1917-….mkv' decoy to every 7z
upload; the largest-entry rule named 840 releases after it. Entries with
a hidden path part are skipped; release:rename --hidden renames the
existing ones from their stored listing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 10:29:05 +02:00
Daan MeijerandClaude Opus 5.5 f4ded062df Complete archive listings by walking RAR headers, at low priority
Test and deploy / test (push) Successful in 1m24s
Test and deploy / deploy (push) Successful in 1m58s
Inspection lists only the entries whose headers are in the first segment
of a first volume. A walk hops from header to header, fetching one segment
per entry and volume, in its own queue (one job per collection worker) on
NNTP connections inspection isn't waiting for. content:walk-archives queues
releases inspected earlier (CONCEPT D40).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 23:32:52 +02:00
Daan MeijerandClaude Opus 5.5 8d86f7d489 Seed fragments: keep only those near the range edges, unstaged
Test and deploy / test (push) Successful in 1m5s
Test and deploy / deploy (push) Successful in 1m48s
About 5% of seeded headers are fragments. They went through staging,
where they waited 30 minutes for parts that were never coming (the
neighbouring range is worked on by another node, often hours later) and
expired: the staging table grew to 22 GB and the NVMe part of the database
reached its 60 GB limit, which stopped leasing and the backfill. Files cut
by a range edge can still be completed by the neighbouring range, so
fragments within 50k articles of an edge go straight into binaries and
parts; the rest, files with articles missing for good, are dropped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 20:30:24 +02:00
Daan MeijerandClaude Opus 5.5 f1773744c0 Partition files by age (id)
Test and deploy / test (push) Successful in 54s
Test and deploy / deploy (push) Successful in 7m42s
New rows went into free space scattered through the 9.6 GB heap on the HDD
(left behind by inspection's updates): one random HDD read per insert, and
seed batches took a median 13 s to store. The existing table becomes the
first partition as it is (a CHECK scan, no rewrite; indexes, its foreign
key and the triggers are taken over by the partitioned parent); new
partitions of 4M ids go in the same tablespace and stay in memory while
they fill.

- Every partition has a unique file_key index; FileStore.insert looks keys
  up in all partitions first (a key can't be unique across partitions).
- FilePartitions.ensure() (gc:scheduler) keeps two partitions ahead.
- Stats count files from the partitions; seed:node waits ~20 minutes for
  the API before giving a range up.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 15:57:51 +02:00
Daan MeijerandClaude Opus 5.5 41859636d3 Seed relay: nodes get their headers through the seed API
Test and deploy / test (push) Successful in 1m28s
Test and deploy / deploy (push) Successful in 2m5s
The seed API keeps a pool of NNTP connections (SEED_NNTP_CONNECTIONS) and
serves GET /api/v1/seed/ranges/:id/chunk?from=&to= for leased ranges:
the gzip stream the news server sent, passed on unopened. seed:node fetches
that way by default (SEED_FETCH=relay), so a node needs only a token and
SEED_CONNECTIONS; SEED_FETCH=direct keeps its own NNTP connections. All
fetching then comes from the indexer's IP (the account allows two).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 15:24:39 +02:00
Daan MeijerandClaude Opus 5.5 3d5cfb2527 Seeding nodes lease first, connect for the lease, close when idle
Test and deploy / test (push) Successful in 1m36s
Test and deploy / deploy (push) Successful in 1m58s
A failed NNTP login gives the range back uncounted (no attempt used), and a
node with nothing to lease holds no connection. Replaces log-in-before-lease,
which kept idle connections and logged in again every minute while idle.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 15:18:38 +02:00
Daan MeijerandClaude Opus 5.5 70036847cf Seeding nodes log in before leasing; TLS through tunnels
Test and deploy / test (push) Successful in 1m29s
Test and deploy / deploy (push) Successful in 1m59s
- seed:node logs in to NNTP before taking a range and keeps the connection
  across ranges (replaced after 2 minutes idle or a failed range). A node
  that can't reach its news server takes no range: two ranges were set
  aside as failed while an office node's tunnel was being set up
  (ECONNREFUSED 127.0.0.1:1563, five leases each). After an IP-address
  limit refusal it waits 5 minutes.
- NNTP_TLS_SERVERNAME: the hostname the TLS certificate is checked
  against when NNTP_HOST is a tunnel ("Hostname/IP does not match
  certificate's altnames: Host: localhost").
- seed-node/README: the IP-address limit and tunnelling.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 14:14:15 +02:00
Daan MeijerandClaude Opus 5.5 00f6187b8f Seed leases respect the backfill's disk limits
Test and deploy / test (push) Successful in 1m8s
Test and deploy / deploy (push) Successful in 1m37s
No new range while the database's NVMe part or the hdd tablespace is over
BACKFILL_MAX_NVME_GB / BACKFILL_MAX_HDD_GB; nodes ask again a minute later.
seed:status says when leasing is paused. The NVMe part was at 47 GB of 50
with a plan for all of a.b.ath queued.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 13:30:46 +02:00