[PranjalManhgaye]

Summary

Parallelize system tests via a per-tutorial GitHub Actions matrix derived from tests.yaml.

Approach: A prepare-matrix job reads release, groups cases by tutorial, and emits one job per tutorial with comma-separated per-case suite names (not bare tutorial suite names, so cases outside release are not pulled in).

Example: aste-turbine => 1 case, elastic-tube-1d => 6 cases. All 40 jobs together cover full release (67 cases), no duplicates.

Manual workflow:

• release => serial by default (single job)
• release + parallel: true => per-tutorial matrix (~40 jobs)
• any other suites => single job (unchanged)

Nightly: still runs release as one serial job (unchanged).

Closes #789. Replaces the earlier shard-based approach in this PR.

Test plan

• prepare-matrix job on ubuntu-latest
• Manual release serial (default)
• Manual release with parallel: true on self-hosted VM (#33298363903)

[PranjalManhgaye]

@MakisH follow-ups (i think later, not this PR) :

• same matrix for manual / latest-components workflows
• more shards if we get more runners
• docker build sharing only if ci shows we need it
• option 2 (systemtests.py parallel) => skipped for now

[MakisH]

Nice to see a first prototype towards parallization!

While this is a valid and easy-to-implement approach for parallelization, I think it mainly adds a layer to the current approach.

Ideally, we should end up with the individual test suites (the ones per tutorial) as shards, so that the jobs also get meaningful names. Issues there will be:

• race conditions in building the same Docker layers, if a runner picks more than one shard at the same time,
• that some too long tests are currently excluded from the release_test (see extra), as these take too long. We could then just define these directly in the extra, instead of referring to the ones defined at the tutorial level with anchors.

Nevertheless, I could create another runner and use this PR to test if the parallelism with multiple custom runners makes sense.

[MakisH]

@PranjalManhgaye #842 introduced some conflicts in system-tests-latest-components.yml that should be easy to resolve.

[MakisH]

I triggered a test run: https://github.com/precice/tutorials/actions/runs/27600082238

[MakisH]

While the matrix is a good approach, I think we should drop the shards and instead reuse the per-tutorial test suites (elastic-tube-1d, elastic-tube-3d, ...). This should also give clearer and faster output, and these test suites are by definition not overlapping with each other.

What I am not sure at the moment is what is the right way to get these inputs to avoid duplication.

[PranjalManhgaye]

@MakisH what do you think ; add a small prepare-matrix job in the workflow that reads tests.yaml , or should we extend systemtests.py tooling to expose this list somehow ??

[MakisH]

Maybe related: I always have to count how many tutorials and tutorial cases we have, both for reporting and now for ensuring that everything is tested. We could collect this information from all the metadata.yaml (assuming that they are complete, or somehow checking with another script if these are complete, checking all available directories).

We could then use that information as input for this matrix.

But in general, a prepare-matrix job would work. Note, however, that tests.yaml includes some duplicate information. We could define the release as all tutorials and read that. The question is then what do we do with the extra that take longer to run.

[PranjalManhgaye]

@MakisH i am happy to follow this and drop the artificial shards , and i can implement it by adding a small prepare-matrix step that reads tests.yaml so the workflow doesn’t duplicate suite lists, but to confirm before i do, do you want the matrix at tutorial-suite level (not per-case), keep release as the source of truth there for now, and keep extra out of the default latest-components run (manual or less frequent) ??

[MakisH]

Overall, I would say that the whole feature is still in the exploration phase and would need some research / brainstorming.

do you want the matrix at tutorial-suite level (not per-case)

We could also split per-case, as long as we don't run a case multiple times. That would give even more refined output. But I think it would be simpler to split per-tutorial, which would also give simpler names. It would also give fewer jobs, which might be easier to manage for the runners, and will share some of the setup overhead.

keep release as the source of truth there for now, and keep extra out of the default latest-components run (manual or less frequent) ??

Yes, I would keep this as-is for now, unless it makes sense to change it.

[PranjalManhgaye]

@MakisH Thanks for testing,

• Parallel checkbox only applies to exact release; quickstart staying serial is correct.
• Extra grey run_system_tests jobs are skipped reusable-workflow placeholders in the UI, not duplicate runs.
• Parallel run failures were artifact ETIMEDOUT, not test failures - we can re-run those jobs or disable _full uploads for matrix runs.

and yes ofc that strange thing in ui is coz of the manual workflow has 3 conditional jobs that all call the same reusable workflow - only the one that actually runs does real work ; I guess the possible fix could be : merge serial + custom into one job (2 jobs total) so fewer skipped branches appear in the sidebar, what do you think abt it ?

ready to tweak anything else when you revisit the remaining points.

[MakisH]

* Parallel run failures were artifact `ETIMEDOUT`, not test failures - we can re-run those jobs or `disable _full uploads` for matrix runs.

Starting the failing jobs again, the issue does not seem to be persistent: https://github.com/precice/tutorials/actions/runs/33298363903 Still, one reason to keep the parallel matrix optional.

I am not sure if this or making a separate workflow would be the best option. I lean towards the separate workflow, but I need to have a closer look.

[MakisH]

I added a second runner for testing, in the same VM, which now has 16 cores instead of 4 (same CPU). Results:

• With two runners, the workflow takes 52min total, but does not run all cases: https://github.com/precice/tutorials/actions/runs/36560048377 (starting from cached components)
• With a single runner, running in the serial mode, the workflow takes 78min: https://github.com/precice/tutorials/actions/runs/36568031720

In the parallel run, the following tests fail:

• Channel transport reaction (fluid-fenics, chemical-fenics) - times out at 3m 10s
• Flow over heated plate (fluid-openfoam, solid-fenics) - fails to run due to deleted network
• Wolf-sheep-grass soil creep (soil-creep-fenicsx, wolf-sheep-grass-mesa) - fails to run due to deleted network

seemingly unrelated:

• Channel transport particles (fluid-nutils, particles-mercurydpm) - runs in 2m 47s, fails comparisons
• Elastic tube 1D (fluid-python, solid-python) - runs in 1m 41s, fails comparisons
• Flow over heated plate (fluid-openfoam, solid-nutils) - runs in 1m 13s, fails comparisons
• Water hammer (fluid1d-left-nutils, fluid3d-right-openfoam) - runs in 2m 10s, fails comparisons

In the serial run, the following tests run but fail comparisons:

• Channel transport particles (fluid-nutils, particles-mercurydpm) - runs in 1m 26s
• Elastic tube 1D (fluid-python, solid-python) - runs in 1m 13s
• Flow over heated plate (fluid-openfoam, solid-nutils) - runs in 0m 53s
• Water hammer (fluid1d-left-nutils, fluid3d-right-openfoam) - runs in 1m 35s

No clear speedup conclusion can be drawn at this point, as some cases did not run in the parallel case, but the speedup is definitely below 33%. I suspect that some cases run matrix operations on all available cores, and having other active processes in the same VM slows that down. If that theory holds, then adding more runners would slow that down even further. Having both runners in the same VM seems to be beneficial for reusing the same build cache, but not necessarily for running. I need to try with a separate VM as well.

Further observations:

• Some jobs are failing because they don't find the docker network, which was probably deleted by another job running docker network prune. This step was introduced because there is a limit on the number of networks, but there are probably better ways to cleanup.
• Some jobs are failing at the results comparison step, which I did not expect. This also happens in the serial run. We got more CPU cores, but the CPU is the same, so I assume this is only a dependency issue. Running the water hammer with everything from develop (which worked yesterday in the old runner configuration) also fails, but with differences in the order of 1e-14. I will look closer and update the reference results.
• I still don't really like that there are inactive and disconnected jobs shown when I run the system tests in either mode, but I am not sure what the best way to make this straightforward would be. I would expect one workflow, where the pre-processing step always runs but creates a single job when running in serial, and then the system tests run on a matrix that might only have a single job to run.

Suggestions:

• Make a single workflow that always prepares the matrix, but the serial job only runs on a matrix with one job.
• I think we need a clearer overview (job summary), similarly to the serial job. Maybe a post-processing step that constructs a prominent job summary could help. A helpful feature would be an overview of which jobs failed, at which step, with an easy way to discover the log previews. Every job summary seems to have its own anchor, so we could link to that.
• Instead of parallelizing specifically per tutorial, we could generalize the parallelization and partition any given list of tests using some default shard size (1 or more tests, to reduce the setup time), assuming that all tests take comparable time to run. It should be the responsibility of the user to make sure that there are no duplicates. But maybe something for another PR.

[PranjalManhgaye]

@MakisH Thanks for the two-runner testing and for fixing the comparison failures in #935.

Next steps:

• Rebase this PR onto latest develop (includes Update reference results for updated runner #935 reference updates).

• Refactor the manual workflow so prepare-matrix always runs, and system tests always use a matrix — serial = one - - matrix job (full release), parallel = per-tutorial jobs - to remove the inactive/skipped jobs in the UI.

• Fix the docker network prune race so parallel jobs don’t delete networks other jobs still need.

• Add a post-processing summary job that lists failed matrix jobs/steps with links to their summaries.

• Re-run parallel tests on the VM after these changes for your review.

Question: Do you prefer this unified workflow approach, or a separate workflow for parallel runs? I’ll follow whichever you think is easier to maintain.

[MakisH]

@PranjalManhgaye more results need updating; doing that right now, I will write again here once ready.

Regarding the network prune, this already affects unrelated jobs running on the same machine. I can fix it in another PR (unless you prefer to do that).

I think the cleanest solution is to move the prepare-matrix: job into the system-tests-manual.yml/system-tests-nightly.yml and drop the system-tests-release-matrix.yml. A matrix with one job should still produce one regular run.

[MakisH]

I have updated more results in #937 and improved the network cleanup in #938.

[MakisH]

Triggered a new test in parallel: https://github.com/precice/tutorials/actions/runs/36749249241

I expect everything to work, besides the Channel transport reaction (fluid-fenics, chemical-fenics), which timed out in the previous run.

[MakisH]

@PranjalManhgaye all tests pass now, so this is ready for the discussed refactoring.

In principle, the prepare-matrix should work for any set of tests, not only for the release. What makes it release-specific?

[PranjalManhgaye]

@MakisH Refactored :

• Moved prepare-matrix into system-tests-manual.yml and system-tests-nightly.yml

• Dropped system-tests-release-matrix.yml

• One matrix path now: serial = 1 job, parallel = per-tutorial

• prepare_matrix.py now takes any suite via --suites (not release-only)

you can re-run manual release (serial + parallel) on the VM whenever you want to check CI.

[MakisH]

Thanks for the quick updates, @PranjalManhgaye. It looks much better now, and it seems to work as expected, also with an arbitrary list of cases. Even when asking for one specific case (e.g., channel-transport_fluid-openfoam_transport-openfoam), it splits per tutorial (channel-transport), but only runs the specific case. Nice!

I think that the parallel_matrix.py can be simplified and documented a bit more and then integrated directly into prepare_matrix.py. It should be rather easy; please let me know if some suggestions do not make sense.

[PranjalManhgaye]

@MakisH

• I have merged parallel_matrix.py into prepare_matrix.py, added module/docs with naming examples,
• unified suite lookup, simplified step summary, and shortened the parallel input description.
• Tested locally with serial/parallel release, quickstart, and a single case.

Let me know if you'd like the external lookup handled differently in tests.yaml.

[MakisH]

I think this looks good now. Thank you for all the iterations and for sticking that far, @PranjalManhgaye!

I have also tested this with various edge cases, and I think it is now good to go. Eventually, this could become the default, if more (isolated) runners are available, or after we solve some related issues (e.g., #936).

In the meantime, I would like to have this available to make measurements and evaluate whether splitting per tutorial is an approach that actually helps in practice (both in runtime and in fine-grained rerunning of failed cases).