Rebuild: XOVER chunks of 100k lines; batches by articles
Test and deploy / test (push) Successful in 21s
Test and deploy / deploy (push) Skipped

Eweka takes ~1.5–2 s per XOVER command whatever its size. One connection
on recent a.b.boneless: 9–15k lines/s in chunks of 25k, ~20k in 50k,
~30k in 100k–200k. Four connections at once: 15k each at 50k, 22k each
at 100k (the same as one connection alone at that moment): connections
don't slow each other. CHUNK_ARTICLES defaults to 100k; edge:node posts a
batch every --batch-articles (500k) instead of every N chunks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Daan MeijerandClaude Opus 5.5 committed 2026-10-09 02:09:04 +02:00
1 parent 08c876b690
commit 5ba93f558c
2 files changed
+17 -10

No files matched your search

+7 -2
View File
@@ -5,8 +5,13 @@ import app from '@adonisjs/core/services/app'
export default {
/** Articles per range. */
rangeArticles: env.get('RANGE_ARTICLES', 2_000_000),
/** Articles per XOVER chunk (spool files, stream frames). */
chunkArticles: env.get('CHUNK_ARTICLES', 50_000),
/**
* Articles per XOVER chunk (spool files, stream frames). Eweka takes
* ~1.5–2 s per command whatever its size: one connection read 9–15k
* lines/s in chunks of 25k, ~20k in 50k, ~30k in 100k and 200k (LOG.md
* 2026-10-09).
*/
chunkArticles: env.get('CHUNK_ARTICLES', 100_000),
/** A leased range without a heartbeat for this long goes back to the queue. */
leaseTtlSeconds: env.get('LEASE_TTL_SECONDS', 15 * 60),
/** NNTP connections the coordinator fetches headers with. */