Skip to content

feat: Parallelize chunking - #295

Merged
Pringled merged 7 commits into
mainfrom
perf/parallel-chunking
Oct 11, 2026
Merged

Pringled merged 7 commits into
mainfrom
perf/parallel-chunking

Conversation

@Pringled

@Pringled Pringled commented Oct 11, 2026 •

Copy link
Copy Markdown
Member

Chunking and BM25 tokenization now run in worker processes when an index build has 200+ files to (re)chunk. The CLI and MCP server use 4 workers by default. There's a SEMBLE_INDEX_WORKERS var to overwrite this, and setting it to 0 turns it off.

Cold build timing on a large repo of 27k files:

main new
27k files ~120s ~84s

@codecov

codecov Bot commented Oct 11, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

Files with missing lines Coverage Δ
src/semble/cli.py 100.00% <100.00%> (ø)
src/semble/index/create.py 100.00% <100.00%> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@greptile-apps

greptile-apps Bot commented Oct 11, 2026 •

Copy link
Copy Markdown

RetriggerConfidence Score: 5/5

[High impact] The PR appears safe to merge; no new blocking issue remains.

Reviews (2) · Last reviewed commit: "Drop the Windows stdin workaround now th..." · Reviewed by Greptile

Comment thread src/semble/index/create.py
@Pringled
Pringled requested a review from stephantul October 11, 2026 10:57
@Pringled
Pringled merged commit 0e5e028 into main Oct 11, 2026
17 checks passed
@Pringled
Pringled deleted the perf/parallel-chunking branch October 11, 2026 11:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants