Rebuild: XOVER chunks of 100k lines; batches by articles
Eweka takes ~1.5–2 s per XOVER command whatever its size. One connection on recent a.b.boneless: 9–15k lines/s in chunks of 25k, ~20k in 50k, ~30k in 100k–200k. Four connections at once: 15k each at 50k, 22k each at 100k (the same as one connection alone at that moment): connections don't slow each other. CHUNK_ARTICLES defaults to 100k; edge:node posts a batch every --batch-articles (500k) instead of every N chunks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
08c876b690
commit
5ba93f558c
2 files changed
+17
-10
No files matched your search
@@ -5,8 +5,13 @@ import app from '@adonisjs/core/services/app'
|
||||
export default {
|
||||
/** Articles per range. */
|
||||
rangeArticles: env.get('RANGE_ARTICLES', 2_000_000),
|
||||
/** Articles per XOVER chunk (spool files, stream frames). */
|
||||
chunkArticles: env.get('CHUNK_ARTICLES', 50_000),
|
||||
/**
|
||||
* Articles per XOVER chunk (spool files, stream frames). Eweka takes
|
||||
* ~1.5–2 s per command whatever its size: one connection read 9–15k
|
||||
* lines/s in chunks of 25k, ~20k in 50k, ~30k in 100k and 200k (LOG.md
|
||||
* 2026-10-09).
|
||||
*/
|
||||
chunkArticles: env.get('CHUNK_ARTICLES', 100_000),
|
||||
/** A leased range without a heartbeat for this long goes back to the queue. */
|
||||
leaseTtlSeconds: env.get('LEASE_TTL_SECONDS', 15 * 60),
|
||||
/** NNTP connections the coordinator fetches headers with. */
|
||||
|
||||
Reference in new issue
Block a user