Copy only set parameters for YAML inputs, share the reform-free tree, and run test batches in forked workers

[anth-volk]

Fixes #615
Fixes #618

Important

This PR needs both #569 and #570 to merge first. Merge in this order: #569, then #570, then this PR.

• Stop the YAML test runner keeping every case's simulation and reform system #569: this PR's base is Stop the YAML test runner keeping every case's simulation and reform system #569's branch (fix-test-runner-memory), and the code builds on Stop the YAML test runner keeping every case's simulation and reform system #569's reform cache.
• Resolve parameter nodes at instant on first read #570: this branch merges Resolve parameter nodes at instant on first read #570, so the diff shows Resolve parameter nodes at instant on first read #570's changes until Resolve parameter nodes at instant on first read #570 lands. Resolve parameter nodes at instant on first read #570 isn't needed for the code to work, but without it each forked worker uses about twice the memory (see below).
• CI: pr.yaml only runs on pull requests into master, so this PR gets no checks until Stop the YAML test runner keeping every case's simulation and reform system #569 merges and GitHub retargets it to master.

Problem

The YAML test runner uses too much memory and CPU for policyengine-us to run its YAML suites without 21 CI shards. #569 and #570 fix two causes. This PR fixes three more:

• each distinct combination of dotted parameter inputs cloned the whole tax-benefit system, costing about 4.5 s and 1.2 GB;
• each reform system held two copies of the parameter tree;
• parallel batches could not share one built system.

Change

• Dotted parameter inputs (parameters/path_copy.py, _ParameterInputReform in tools/test_runner.py). A case that sets parameters runs on a reform of its own. It copies only the nodes on the paths to the parameters it sets and shares the rest of the tree with the system it is built on. These systems are no longer cached, so inline values no longer key the reform cache. Reform gets a _copy_parameters hook, which by default still clones the whole tree.
• Reform systems are built on the cached reform-free system instead of a clone of the baseline, so each one holds a single copy of the parameter tree. This applies when every reform is a Reform subclass or a dotted name, because those copy what they change. A reform given as a function still gets a clone. On a cache miss, the cache now evicts before it builds, so it never holds reform_cache_size + 1 systems at once.
• --read-only-arrays (options["read_only_arrays"], InMemoryStorage.read_only_arrays). The runner stores arrays read-only, so a formula that writes into an array it has read raises an error. Without this, the bad write would silently change the cached value.
• policyengine_core.tools.forked_test_runner.run_batches_in_forks:
  • builds the reform-free system once, then runs gc.collect() and gc.freeze();
  • forks up to workers children, each running one batch with run_tests and exiting with os._exit;
  • enforces a per-batch timeout (SIGTERM, then SIGKILL) and reports each child's exit code, peak RSS and output;
  • children share the parent's system copy-on-write;
  • it requires os.fork, so it is POSIX only, and callers should cap BLAS threads at 1.
• modify_parameters (modify_parameters reads stale cached values after in-place edits, and Reform.modify_parameters deep-copies a tree the reform already owns #618).
  • Reform no longer overrides modify_parameters. It inherits TaxBenefitSystem.modify_parameters, which gives the modifier the reform's own tree: the copy Reform.__init__ already makes. A reform built as an instance therefore behaves the same as one applied with apply_reform_set.
  • That method now also clears every node's cached instants after the modifier runs, with Resolve parameter nodes at instant on first read #570's ParameterNode.clear_instant_caches. Without this, a modifier that edits values, brackets or children in place, like the country template's modify_social_security_taxation, read the values the tree had cached. The runner hit that once reforms were built on the reform-free system other cases had read.
  • For reform instances, changes made earlier in apply() are now kept, and successive modify_parameters calls build on each other.
  • Building such a reform no longer deep-copies the tree, which took 5.7 s for policyengine-us's.
  • modify_parameters reads stale cached values after in-place edits, and Reform.modify_parameters deep-copies a tree the reform already owns #618 has the history and a survey of downstream usage.
  • Let vectorial parameter nodes be copied and pickled #584's reform test asserted that the copy kept the baseline's cached vectorial node. It now reads the values before checking the node, and passes with and without this change.
• Docs and changelog. docs/engineering/skills/testing.md is updated, and changelog.d/test-runner-perf.added.md is added.

Test outcomes are unchanged. A Hypothesis test checks that path-copied parameters read the same as a full clone at every instant. test_parameter_inputs.py checks that dotted inputs give the same results as the old full-clone reform, with and without a named reform. I also built each of the 152 distinct reform lists in policyengine-us's YAML tests on the reform-free system: none of them changed its variables, formulas, entities or any parameter value. That audit checked only whether a reform changes the shared system. It did not check whether a reform still reads its own changes; the modify_parameters fix covers that case.

Measurements

#569 and #570 are now rebased onto master (b909d5a), so this branch includes it. Its code is what I measured plus three later changes: #570's pickling test, the modify_parameters cache fix and the modify_parameters change for #618. With all three, the whole policyengine-us YAML suite (34,843 cases) passes forked. So do the 29 policyengine-us Python test files that build or apply reforms: 472 passed, 4 skipped. policyengine-us applies its reforms in place, so these runs clear the cache on every system it builds.

The policyengine-us side is a change to test_batched.py: --fork --workers 4 uses run_batches_in_forks with a reform cache of 1 and read-only arrays. It will be opened as its own PR after a core release.

Same macOS host. "Before" is policyengine-us main on core 3.33.3. Every case passed in every run:

policyengine-us shard	Before: wall, CPU, peak	This stack, 4 forked workers: wall, CPU
Baseline states 2/4, 6,724 cases	2,088 s, 1,548 s, 10.4 GB	97 s, 284 s
Contrib states 4/4, 215 cases	3,401 s, 2,560 s, 12.0 GB	56 s, 156 s

The host was loaded, so CPU seconds are the more reliable figure. macOS also counts pages shared between forked workers once for each worker, so its after-peaks overstate memory.

The memory figures come from a Linux container capped at 7.5 GB, using cgroup memory.peak:

• The whole policyengine-us YAML suite (34,843 cases) passed forked, with and without --read-only-arrays, peaking at 6.6 GB with 2 workers.
• Each contrib worker uses about 2.5 GB of private memory, so policyengine-us runs contrib with 3 workers.

Two figures were measured before the reform-system change, which mainly affects contrib:

• Four baseline-states workers peaked at 6.36 GB together.
• Without Resolve parameter nodes at instant on first read #570, a two-worker baseline-states run peaked at 8.05 GB, and one worker (4.3 GB private) was killed by the OOM killer. With Resolve parameter nodes at instant on first read #570 the same run peaked at 4.16 GB, with 2.3 GB for the largest worker, and its CPU time fell from 1,029 s to 412 s.

Tests

On this head (b8aa6e6a), which merges #569's and #570's October 15 fixes, on macOS with Python 3.14:

• pytest tests: 2,662 passed, 4 skipped, 3 xfailed.
• The whole policyengine-us YAML suite passes forked (34,843 cases).
• The 29 policyengine-us Python test files that build or apply reforms, or change parameter trees, pass: 472 passed, 4 skipped.
• A differential check of 1,200 runner requests (40 random sequences of 30, mixing reforms, dotted inputs, cache sizes 0 to 2 and partial reads) matches freshly built systems.
• ruff format --check . and ruff check . pass.
• The country template YAML suite passes with default options, with --read-only-arrays and with --reform-cache-size 1.

Correction. An earlier version of this description said the country template YAML suite passed with and without --read-only-arrays. That run came before the reform-system change. After that change, 3 of its 39 cases failed (the modify_social_security_taxation cases) until the modify_parameters fix.

New and changed tests:

• tests/core/parameters/test_path_copy.py
• tests/core/tools/test_runner/: test_parameter_inputs.py, test_read_only_arrays.py, test_forked_test_runner.py and test_reform_system_size.py
• In tests/core/test_reforms.py, these fail on master:
  • test_modify_parameters_reads_its_changes_after_the_baseline_cached_them
  • test_an_in_place_reform_reads_its_changes_after_the_system_cached_them
  • test_modify_parameters_keeps_what_apply_changed_before_it
  • test_a_reform_instance_and_an_in_place_reform_give_the_same_parameters
• test_modify_parameters_leaves_the_baseline_unchanged passes on both.
• test_reform_cache_property.py now checks that no reform is ever applied to the shared reform-free system.
• Let vectorial parameter nodes be copied and pickled #584's test_reform_that_modifies_parameters_keeps_the_cached_vectorial_node is renamed ..._reads_the_vectorial_node, as described above.
• Docstrings in test_tracing_parameter_isolation.py and test_vectorial_parameter_node_copy.py no longer say modify_parameters deep-copies the tree.

Not run: make documentation, and anything in real CI (see the note at the top).

🤖 Generated with Claude Code

[anth-volk]

Closing without merging. This PR relied on #569 and #570, which are parked, and its own changes also depend on unwritten rules about parameter caches: per-case path copies share untouched subtrees and their cached nodes with the reform-free system. #621 proposes those rules, and the runner changes here (path copies, reform systems built on the reform-free system, forked workers) can be revisited on top of them. #618, which this PR also fixed, stays open; #621 covers it.