Skip to content

Fix a divergent partition function shipping as a rate coefficient, plus restart, submission and AutoTST fixes - #1069

Open
alongd wants to merge 13 commits into
mainfrom
paper-demo-fixes
Open

alongd wants to merge 13 commits into
mainfrom
paper-demo-fixes

Conversation

@alongd

@alongd alongd commented Oct 27, 2026

Copy link
Copy Markdown
Member

Summary

Fixes found while running two reactions end to end (n-C3H7OO 1,5-H shift; CH3 + C2H6 H-abstraction) for the ARC methods paper. One commit per fix:

statmech/arkane.py     discard k(T) when Arkane's own conformer has a mode < 5 cm^-1
plotter.py             "No rotors considered" note decided from rotors_dict, not a substring
scheduler.py  restart  skip IRC-endpoint rotors
                       spawn never-scanned rotors (success None, no scan_path)
                       run per-rotor scans when pipe_planner does not exist yet
                       drop superseded conf_opt names from running_jobs
                       skip restored jobs whose species are not in the project
                       skip running-job names missing from job_dict when saving
job/ssh.py             PBS "Maximum number of jobs already in queue" handled like Slurm's
                       AssocMaxSubmitJobLimit: one retry, return its result
                       unparsable job ID from a known scheduler -> errored job, not ValueError
autotst                script exits non-zero with a traceback on failure
                       subprocess runs without the caller's PYTHONPATH
devtools/install_autotst.sh  register AutoTST via autotst.pth in tst_env, not ~/.bashrc

The Arkane fix in one line: a TS whose 21 cm⁻¹ imaginary torsion reached Arkane as 0.0 cm⁻¹ produced a harmonic partition function of ~1e9 and a rate coefficient 18× above the collision limit, which ARC wrote to the library as converged. The check reads Arkane's output, not ARC's input, because Arkane re-derives frequencies from the Hessian when rotors are present.

 run_arkane
   parse output.py -> rxn.kinetics
+  if any conformer frequency in output.py < MIN_PHYSICAL_FREQ (5 cm^-1)
+    log the frequencies, note them in the TS description
+    rxn.kinetics = None

The PYTHONPATH strip keeps a source RMG-Py checkout on the caller's PYTHONPATH from shadowing tst_env's rmgpy. With it, the installer's default mode (a PYTHONPATH line appended to ~/.bashrc) would leave AutoTST unimportable, so the installer now writes autotst.pth into tst_env's site-packages after the env is created, in both modes. --conda hooks still survive: conda run fires activate.d after the strip (arc/job/env_run.py).

Evidence

Every new test fails on official/main and passes here.

Case Before (official/main) After
TS conformer HarmonicOscillator([0.0, 3.2, …]) kinetics kept, A = 5e20 s⁻¹ rxn.kinetics is None, reason in TS description
Same reaction, all modes ≥ 85 cm⁻¹ (control) kept kept
TS with rotors_dict = {0: {'success': True}} "No rotors considered for this TS." note absent
Restart with an IRC-endpoint rotor a scan spawned per IRC rotor none
Restart with a hand-added rotor never run run
Restart with a rotor to spawn AttributeError: ... no attribute 'pipe_planner' per-rotor scan
Restart restoring a job of rxn_label_TS KeyError skipped with a warning
Saving with an untracked conformer job KeyError: 5 skipped with a warning
PBS queue full, then 123456.pbs-server ValueError: Unrecognized cluster software: PBS ('running', '123456')
PBS queue full twice ValueError ('errored', 0) after one sleep
Unknown cluster software LSF ValueError ValueError (unchanged)
AutoTST raises inside the script exit 0, empty stderr exit 1, traceback on stderr
AutoTST subprocess env caller's PYTHONPATH passed strip_pythonpath=True
pytest -n 0 arc/statmech/arkane_test.py arc/plotter_test.py arc/scheduler_test.py \
       arc/job/ssh_test.py arc/job/adapters/ts/autotst_test.py \
       arc/job/adapters/scripts/autotst_script_test.py
337 passed, 15 subtests passed

bash -n and shellcheck pass on the installer. write_pth_file was run against a temporary directory, and the site-packages lookup was run against a real tst_env. The full installer was not run end to end.

Under the default xdist run, arkane_test.py and plotter_test.py fail intermittently on official/main as well, because tests share directories. This PR does not address that.

Merge Danger

Door: two-way

Each fix is its own commit and can be reverted alone.

Blast Radius: moderate

  • A reaction whose TS or species has a mode below 5 cm⁻¹ now gets no rate coefficient in the library instead of a wrong one. Users who relied on that number will see it disappear, along with the reason.
  • If a cluster's submission reply is not recognised, ARC now marks that one job as errored instead of stopping the whole run.
  • Existing AutoTST installs made in the default path mode need make install-autotst rerun to get autotst.pth. Until then AutoTST guesses fail loudly rather than silently. --conda installs are unaffected.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Critical restart-handling defects leave invalid kinetics available on disk and orphaned job entries that can cause crashes or infinite scheduling loops.

3 open issues
What changed in this PR

This PR prevents ARC from exporting rate coefficients derived from near-zero vibrational frequencies and improves restart, cluster submission, and AutoTST reliability.

Changes:

  • Reject suspect Arkane kinetics and derive TS rotor notes from rotor data.
  • Handle incomplete restart state and retry queue-limit submission failures once.
  • Isolate AutoTST’s Python environment, report subprocess failures, and register its installation through a .pth file.
File Description
docs/​source/​TS_search.rst Document AutoTST registration and environment isolation.
devtools/​install_autotst.sh Register AutoTST through .pth; retain optional activation hooks.
arc/​statmech/​arkane.py Reject kinetics involving near-zero conformer frequencies.
arc/​statmech/​arkane_test.py Test frequency rejection and valid-frequency controls.
arc/​scheduler.py Improve restart scans and stale-job handling.
arc/​scheduler_test.py Add restart and job-tracking regression tests.
arc/​plotter.py Determine TS rotor notes from successful rotor data.
arc/​plotter_test.py Test rotor descriptions in generated libraries.
arc/​job/​ssh.py Bound queue-limit retries and handle unrecognized submission replies.
arc/​job/​ssh_test.py Test submission failures and retry results.
arc/​job/​adapters/​ts/​autotst_ts.py Strip caller PYTHONPATH from AutoTST subprocesses.
arc/​job/​adapters/​ts/​autotst_test.py Verify AutoTST environment-isolation requests.
arc/​job/​adapters/​scripts/​autotst_script.py Report initialization failures with a traceback and nonzero exit.
arc/​job/​adapters/​scripts/​autotst_script_test.py Test subprocess failure reporting with an AutoTST stub.

Review effort: Balanced

Comment thread arc/scheduler.py Resolved
Comment thread arc/scheduler.py Resolved
Comment thread arc/statmech/arkane.py Resolved
@codecov

codecov Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 67.26%. Comparing base (6da9d99) to head (5e760aa).

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #1069      +/-   ##
==========================================
+ Coverage   67.15%   67.26%   +0.10%     
==========================================
  Files         123      123              
  Lines       43675    43768      +93     
  Branches    11153    11170      +17     
==========================================
+ Hits        29332    29442     +110     
+ Misses      11203    11169      -34     
- Partials     3140     3157      +17     
Flag Coverage Δ
functionaltests 67.26% <ø> (+0.10%) ⬆️
unittests 67.26% <ø> (+0.10%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 14 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

A harmonic oscillator at ~0 cm^-1 has a divergent partition function, so a
rate coefficient computed from a conformer carrying one can be off by many
orders of magnitude, even above the collision limit, while the geometry, the
imaginary-frequency check and the Arrhenius fit all look sound. Arkane's
rotor projection keeps the n_vib largest Hessian eigenvalues without checking
that they are vibrations, so a TS with a second, small imaginary mode (which
LOWEST_MAJOR_TS_FREQ tolerates) retains a residual ~0 eigenvalue.

After parsing each reaction's kinetics, scan the Arkane conformer blocks of
its reactants, products and TS for frequencies below MIN_PHYSICAL_FREQ
(5 cm^-1). On a hit, log the offenders, set the reaction's kinetics to None,
and note the discard in the TS's long_thermo_description.

Co-Authored-By: Claude <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Repeated saves can duplicate or retain stale rotor notes, while saved convergence causes newly added, unscanned rotors to be skipped.

2 open issues
3 resolved since last review

Review effort: Balanced

Comment thread arc/plotter.py Resolved
Comment thread arc/scheduler.py Resolved
alongd and others added 2 commits October 31, 2026 14:22
save_kinetics_lib appended "No rotors considered for this TS." whenever the
word "rotors" was absent from the TS's long_thermo_description. A TS that
Arkane modelled with a hindered rotor was therefore described as having none,
and a description that merely mentioned rotors suppressed the note for a TS
that had none. Decide it from rotors_dict instead, using the same
success-is-truthy filter the Arkane adapter applies when it builds the model.

Co-Authored-By: Claude <noreply@anthropic.com>
IRC endpoint species are diagnostic, not thermo/kinetics targets, and the
normal flow never scans their rotors. They still carry a populated
rotors_dict without a scan_path, so the restart loop spawned a scan per IRC
rotor. Exclude them both as the restarted species and in the scan over all
species.

Co-Authored-By: Claude <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The restart logic can misclassify queued or running pipe rotors as unstarted and submit duplicate standalone scans before existing pipe runs are restored and reconciled.

1 open issue
2 resolved since last review

Review effort: Balanced

Comment thread arc/scheduler.py Resolved
alongd and others added 10 commits October 31, 2026 16:43
On restart only rotors with success=False were respawned. A rotor with no
result and no scan_path, such as one added to restart.yml by hand or
determined after the previous session died, was silently never run and the
species was reported as converged while missing a mode.

A rotor queued or running in a pipe run carries that same state, and pipe
runs are not restored on restart: the coordinator starts with no active
pipes and is constructed after the restart loop, so has_pending_pipe_work()
cannot see one. Such a rotor is therefore skipped unless its staged pipe
task has reached a terminal state, which keeps the restart from submitting
a standalone scan that duplicates a job still in the queue.

Co-Authored-By: Claude <noreply@anthropic.com>
The restart loop in __init__ calls run_scan_jobs(), but the pipe planner is
only constructed after that loop, so a restart with a rotor to spawn died
with AttributeError. Pipe batching is an optimization; without a planner,
scan each rotor individually.

Co-Authored-By: Claude <noreply@anthropic.com>
run_ts_conformer_jobs() wipes job_dict['conf_opt'], which orphans any
conf_opt name restored into running_jobs from a previous session.
save_restart_dict() maps each such name back through job_dict and raised
KeyError on the first one that was gone. The jobs are superseded by the
conformers spawned right there, so stop tracking them.

Co-Authored-By: Claude <noreply@anthropic.com>
A restored job may reference a species that no longer exists in the session,
for example a TS adapter job carrying an auto-generated '<rxn>_TS' label.
Restoring it raised KeyError and killed the run over one stale job; warn and
skip it instead.

Co-Authored-By: Claude <noreply@anthropic.com>
save_restart_dict() looked up every conf_opt, conf_sp and tsg name in
running_jobs directly in job_dict and raised KeyError for a name with no
entry, killing the run over one untracked job. Warn and leave it out of the
restart file instead.

Co-Authored-By: Claude <noreply@anthropic.com>
… job

When a recognized scheduler (OGE/SGE, Slurm, PBS, HTCondor) answered a
submission with nothing parsable on stdout, submit_job fell through to
ValueError('Unrecognized cluster software: ...'). That killed the whole ARC
process over one rejected job, and blamed the scheduler name rather than the
submission. Log stdout and stderr and return an errored job instead, so the
scheduler's troubleshooting decides whether to resubmit. A cluster software
ARC has no parser for still raises.

Co-Authored-By: Claude <noreply@anthropic.com>
…ts result

PBS refuses qsub with "Maximum number of jobs already in queue" when a user
is over the queue cap. Treat it like Slurm's AssocMaxSubmitJobLimit: wait
five minutes and resubmit. Two defects in that retry path are fixed with it:
the resubmission's result was discarded (the outer call parsed its own empty
stdout), and a resubmission that was refused again could sleep and recurse
again while the whole scheduler waited. Return the retry's (status, job ID),
and give up as an errored job after the single retry.

Co-Authored-By: Claude <noreply@anthropic.com>
autotst_script.py swallowed any exception from building or labeling the
AutoTST reaction with a bare `return None`. The process exited 0 and wrote no
output file, so ARC logged nothing and treated the adapter as having found no
TS guess: a broken AutoTST setup looked like a chemical failure. Print the
traceback to stderr and exit 1, so run_in_conda_env logs it and the adapter's
return-code warning fires.

Co-Authored-By: Claude <noreply@anthropic.com>
An RMG-Py source checkout on ARC's PYTHONPATH, compiled for arc_env's
Python, shadows tst_env's installed rmgpy inside the AutoTST subprocess and
breaks it. Pass strip_pythonpath=True to run_in_conda_env, as the KinBot
adapter already does. AutoTST must then be importable from tst_env itself,
via a .pth file or the activation hook that install_autotst.sh --conda
writes; a clone that is only on PYTHONPATH through ~/.bashrc is no longer
seen.

Co-Authored-By: Claude <noreply@anthropic.com>
…ad of ~/.bashrc

The AutoTST subprocess now runs without the caller's PYTHONPATH, so the
bashrc PYTHONPATH line written by the default install mode no longer made
AutoTST importable. Write autotst.pth into tst_env's site-packages after the
environment exists, leave ~/.bashrc alone and note a leftover line.

Co-Authored-By: Claude <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants