Skip to content

ci: overlap independent steps with background/parallel - #6199

Open
henryiii wants to merge 2 commits into
pybind:masterfrom
henryiii:ci/parallel-steps
Open

henryiii wants to merge 2 commits into
pybind:masterfrom
henryiii:ci/parallel-steps

Conversation

@henryiii

@henryiii henryiii commented Oct 8, 2026 •

Copy link
Copy Markdown
Collaborator

I found out about this in pypa/build#1213. Based on this, I've added a skill to henryiii/skills to do this in other repos.

I think we need to trim our CI a bit, but making jobs faster without trimming is always a win if it works. :)

🤖 AI text below 🤖

Description

Use the new GitHub Actions background/wait/parallel step keywords to run independent steps at the same time:

  • reusable-standard.yml: install Boost in the background during Python/uv setup. After the build, run the Python tests, the interface test, and the setuptools helpers test in parallel. Pytest is single-threaded, so the compile-heavy tests use the idle cores. The Python tests now call python -m pytest directly, because two concurrent cmake --build calls on one build tree are not safe.

Measured against the previous master run, the parallel test group saves a median of 40 s per job on Linux, 48 s on macOS, and 94 s on Windows. Pytest alone is 30-56 s slower because it shares the CPU, but the group is still shorter. The Boost change saves only 2-6 s.

I also tried to download the NVHPC package during the disk cleanup. It saved only about 17 s because the download and the cleanup competed for disk I/O, so I removed it.


📚 Documentation preview 📚: https://pybind11--6199.org.readthedocs.build/

Assisted-by: ClaudeCode:claude-opus-5-5
Overlapping the cleanup and download saved only ~17 s, and the
download competed with the cleanup for disk I/O.

Assisted-by: ClaudeCode:claude-opus-5-5

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant