diff --git a/ChangeLog b/ChangeLog index 156d3ba2..722bf46f 100644 --- a/ChangeLog +++ b/ChangeLog @@ -42,6 +42,9 @@ ## Technical changes and Optimizations +- preserve historical local validation helpers, probes and experimental source + snapshots as an inactive, checksummed documentation archive + - require ViennaRNA >= 2.7.0 and its logger/prepared-window APIs at configure time; install the logger lock for every supported version (PR #267) @@ -138,6 +141,13 @@ energies, restricted partition sums, and trackers. ################################################################################ ################################################################################ +261018 Alexander Mitrofanov + * doc/local-validation-archive/, doc/Makefile.am + + recover local validation/reproduction scripts and experimental source copies + + record original paths and byte hashes without activating archived code + * package the corpus as a verified ZIP to preserve long paths in distributions + * document why validation artifacts remained outside earlier production commits + 261015 Alexander Mitrofanov * configure.ac, conda-build-env.yml : * require ViennaRNA >= 2.7.0, the first release with vrna_log_lock_set() diff --git a/doc/Makefile.am b/doc/Makefile.am index cb78b081..d1a8b72c 100644 --- a/doc/Makefile.am +++ b/doc/Makefile.am @@ -4,6 +4,9 @@ ################################################################ EXTRA_DIST = \ + local-validation-archive/README.md \ + local-validation-archive/manifest.json \ + local-validation-archive/sources.zip \ conda.txt \ api-documentation.md \ build-api.sh \ diff --git a/doc/local-validation-archive/.gitattributes b/doc/local-validation-archive/.gitattributes new file mode 100644 index 00000000..a2612612 --- /dev/null +++ b/doc/local-validation-archive/.gitattributes @@ -0,0 +1 @@ +files/** -whitespace diff --git a/doc/local-validation-archive/README.md b/doc/local-validation-archive/README.md new file mode 100644 index 00000000..d644b05d --- /dev/null +++ b/doc/local-validation-archive/README.md @@ -0,0 +1,87 @@ +# Recovered local validation artifacts + +This archive preserves local research and validation source recovered before +removing the IntaRNA workspace. It is a historical corpus for review, rather +than an installed tool or a new prediction implementation. The files are not +wired into the application build, its tests, or CI. + +## Contents and original purpose + +The `files/` tree preserves the original workspace-relative layout. It includes +standalone public-header/installed-library consumers, benchmark drivers, +configure/dependency probes, archived configure and CI fixtures, native-output comparisons, plotting/report +helpers, instrumentation patches and alternative source implementations used +in local measurements. Source snapshots include the abandoned C++17 work, +a matrix prototype, hash/dense partition experiments, an uncached kinetic +control, and profiling instrumentation. They are archived rather than +substituted for current production files. + +The source distribution carries these files in `sources.zip`, whose members +use the original workspace-relative paths. This keeps long historical names +intact despite the existing distribution tar format's 99-character limit. +The individual files remain directly reviewable in Git. + +`manifest.json` lists every stored file with its original path, byte count, +SHA-256 and Git blob hash. Copies already reachable from published repository +refs are omitted. The C++23 audit is proposed separately, as is the six-file +probability-memory integration port. Delivery payloads, generated build files, +downloaded dependencies, private correspondence, binaries and large raw data +are outside this source archive. A separate local recovery archive preserves +the remaining unique workspace material. + +## Why these files were not committed before + +These files lived outside the tracked application checkouts, in issue-specific +validation and profiling directories. They supported earlier PR verification, +benchmarking and exploratory comparisons, while selected production changes +were committed separately. Several are deliberately alternate controls or +instrumented implementations. No file-specific decision record exists for +every helper; their location and content establish their validation role, +not that each was an overlooked production fix. They are committed now so +Martin can inspect that material and the original folder can be removed. + +The copied `issue-260-validation/debug-master` header is a pre-finalization +validation snapshot that omits two includes present in PR #261. Its apparent +missing documentation files came from sharing a copied `.git` pointer with +the real worktree; those deletions are not proposed here. The preserved source +snapshot is inactive and is not a header fix for current master. + +## Using the archive + +Scripts retain their original paths, assumptions and revision-specific APIs. +Reconstruct the relevant source revision and dependency environment before +running them, and use a separate workspace for scripts that edit source files. +The manifest preserves original bytes, including historical formatting. Its +scoped Git attribute avoids treating unified-diff context and copied formatting +as new production whitespace changes. + +Python and shell files were checked for syntax when archived. This does not +certify old experiments against current IntaRNA or reproduce their recorded +performance. The active memory port has its own release/debug validation PR. +Historical reports retain their original measurement dates and limitations. + +## Original directories + +| Directory | Preserved files | +| --- | ---: | +| `issue-184-validation` | 4 | +| `issue-212-validation` | 11 | +| `issue-244-validation` | 10 | +| `issue-245-validation` | 6 | +| `issue-246-validation` | 17 | +| `issue-257-validation` | 4 | +| `issue-258-review-validation` | 3 | +| `issue-258-validation` | 13 | +| `issue-259-revision-validation` | 8 | +| `issue-260-validation` | 3 | +| `issue-262-validation` | 4 | +| `issue-264-validation` | 5 | +| `issue-265-validation` | 8 | +| `issue-267-validation` | 13 | +| `issue-268-validation` | 8 | +| `issue-270-anchored-validation` | 4 | +| `issue-270-validation` | 10 | +| `issue-274-validation` | 2 | +| `kinetic-validation` | 9 | +| `profiling` | 17 | +| `profiling-default-20260921` | 14 | diff --git a/doc/local-validation-archive/files/issue-184-validation/build.sh b/doc/local-validation-archive/files/issue-184-validation/build.sh new file mode 100644 index 00000000..ad19d902 --- /dev/null +++ b/doc/local-validation-archive/files/issue-184-validation/build.sh @@ -0,0 +1,20 @@ +#!/usr/bin/env bash +set -euo pipefail +validation=/home/alex/CodexProjects/IntaRNA/issue-184-validation +dependencies=/home/alex/CodexProjects/IntaRNA/issue-246-validation/toolchain +compiler=/home/alex/CodexProjects/IntaRNA/issue-246-validation/toolchain-gcc14 +source_dir=$1 +mode=$2 +export PATH="$compiler/bin:$PATH" +export PKG_CONFIG_PATH="$dependencies/lib/pkgconfig" +export OPENBLAS_NUM_THREADS=1 +cd "$source_dir" +bash autotools-init.sh > "$validation/logs/$mode-bootstrap.log" 2>&1 +./configure CXX="$compiler/bin/x86_64-conda-linux-gnu-g++" \ + CC="$compiler/bin/x86_64-conda-linux-gnu-gcc" CPPFLAGS="-I$dependencies/include" \ + LDFLAGS="-L$dependencies/lib -Wl,-rpath,$dependencies/lib" \ + --with-boost="$dependencies" --with-boost-libdir="$dependencies/lib" \ + --with-vrna="$dependencies" --prefix="$validation/install-$mode" \ + --enable-debug > "$validation/logs/$mode-configure.log" 2>&1 +make -j2 > "$validation/logs/$mode-build.log" 2>&1 +make tests -j2 > "$validation/logs/$mode-tests.log" 2>&1 diff --git a/doc/local-validation-archive/files/issue-184-validation/check-install.sh b/doc/local-validation-archive/files/issue-184-validation/check-install.sh new file mode 100644 index 00000000..4213f8fd --- /dev/null +++ b/doc/local-validation-archive/files/issue-184-validation/check-install.sh @@ -0,0 +1,39 @@ +#!/usr/bin/env bash +set -euo pipefail +validation=/home/alex/CodexProjects/IntaRNA/issue-184-validation +dependencies=/home/alex/CodexProjects/IntaRNA/issue-246-validation/toolchain +compiler=/home/alex/CodexProjects/IntaRNA/issue-246-validation/toolchain-gcc14/bin/x86_64-conda-linux-gnu-g++ +prefix="$validation/install-release" +if test "${1:-all}" != consumer; then +for header in "$prefix/include/IntaRNA/"*.h; do + printf '#include \nint main() { return 0; }\n' "$(basename "$header")" > "$validation/header.cpp" + "$compiler" -std=c++23 -I"$prefix/include" -I"$dependencies/include" \ + -fsyntax-only "$validation/header.cpp" +done +echo 'All installed public headers compiled independently' +fi +cat > "$validation/consumer.cpp" <<'CPP' +#include +#include +#include +#include +INITIALIZE_EASYLOGGINGPP +int main() { + using namespace IntaRNA; + RnaSequence t("t", "GG"), q("q", "CC"); + AccessibilityDisabled a(t, 0, nullptr), b(q, 0, nullptr); + ReverseAccessibility reversed(b); + InteractionEnergyBasePair energy(a, reversed); + OutputHandlerInteractionList output(OutputConstraint(), 10); + PredictorEvalOnly eval(energy, output, nullptr, + PredictorEvalOnly::parseInteractions("1||&1||", t, q)); + eval.predict(); + return output.reported() == 1 && (*output.begin())->energy == -200 ? 0 : 1; +} +CPP +export PKG_CONFIG_PATH="$prefix/lib/pkgconfig:$dependencies/lib/pkgconfig" +"$compiler" -std=c++23 -DELPP_FEATURE_PERFORMANCE_TRACKING -DELPP_NO_DEFAULT_LOG_FILE "$validation/consumer.cpp" -I"$dependencies/include" \ + -L"$dependencies/lib" -Wl,-rpath,"$dependencies/lib" \ + $(pkg-config --cflags --libs IntaRNA) -leasylogging -o "$validation/consumer" +"$validation/consumer" +echo 'Installed headers and pkg-config evaluation consumer passed' diff --git a/doc/local-validation-archive/files/issue-184-validation/consumer.cpp b/doc/local-validation-archive/files/issue-184-validation/consumer.cpp new file mode 100644 index 00000000..5defb613 --- /dev/null +++ b/doc/local-validation-archive/files/issue-184-validation/consumer.cpp @@ -0,0 +1,17 @@ +#include +#include +#include +#include +INITIALIZE_EASYLOGGINGPP +int main() { + using namespace IntaRNA; + RnaSequence t("t", "GG"), q("q", "CC"); + AccessibilityDisabled a(t, 0, nullptr), b(q, 0, nullptr); + ReverseAccessibility reversed(b); + InteractionEnergyBasePair energy(a, reversed); + OutputHandlerInteractionList output(OutputConstraint(), 10); + PredictorEvalOnly eval(energy, output, nullptr, + PredictorEvalOnly::parseInteractions("1||&1||", t, q)); + eval.predict(); + return output.reported() == 1 && (*output.begin())->energy == -200 ? 0 : 1; +} diff --git a/doc/local-validation-archive/files/issue-184-validation/header.cpp b/doc/local-validation-archive/files/issue-184-validation/header.cpp new file mode 100644 index 00000000..2b45fa6d --- /dev/null +++ b/doc/local-validation-archive/files/issue-184-validation/header.cpp @@ -0,0 +1,2 @@ +#include +int main() { return 0; } diff --git a/doc/local-validation-archive/files/issue-212-validation/build.sh b/doc/local-validation-archive/files/issue-212-validation/build.sh new file mode 100644 index 00000000..202984ac --- /dev/null +++ b/doc/local-validation-archive/files/issue-212-validation/build.sh @@ -0,0 +1,20 @@ +#!/usr/bin/env bash +set -euo pipefail +validation=/home/alex/CodexProjects/IntaRNA/issue-212-validation +dependencies=/home/alex/CodexProjects/IntaRNA/issue-246-validation/toolchain +compiler=/home/alex/CodexProjects/IntaRNA/issue-246-validation/toolchain-gcc14 +source_dir=$1 +mode=$2 +export PATH="$compiler/bin:$PATH" +export PKG_CONFIG_PATH="$dependencies/lib/pkgconfig" +export OPENBLAS_NUM_THREADS=1 +cd "$source_dir" +bash autotools-init.sh > "$validation/logs/$mode-bootstrap.log" 2>&1 +./configure CXX="$compiler/bin/x86_64-conda-linux-gnu-g++" \ + CC="$compiler/bin/x86_64-conda-linux-gnu-gcc" CPPFLAGS="-I$dependencies/include" \ + LDFLAGS="-L$dependencies/lib -Wl,-rpath,$dependencies/lib" \ + --with-boost="$dependencies" --with-boost-libdir="$dependencies/lib" \ + --with-vrna="$dependencies" --prefix="$validation/install-$mode" \ + > "$validation/logs/$mode-configure.log" 2>&1 +make -j2 > "$validation/logs/$mode-build.log" 2>&1 + diff --git a/doc/local-validation-archive/files/issue-212-validation/check-small.py b/doc/local-validation-archive/files/issue-212-validation/check-small.py new file mode 100644 index 00000000..86df3830 --- /dev/null +++ b/doc/local-validation-archive/files/issue-212-validation/check-small.py @@ -0,0 +1,10 @@ +import csv, io, subprocess, json +b='/home/alex/CodexProjects/IntaRNA/pr-worktrees/issue-212/src/bin/IntaRNA' +def run(args): + p=subprocess.run([b,'--threads=1','--outMode=C','--outCsvCols=start1,end1,start2,end2,E','-n','100','--outDeltaE=100']+args,capture_output=True,text=True) + return p.returncode,p.stdout,p.stderr +for t,q in [('CCCCAAUU','CCGGGGUU'),('CCCAAU','CGGGUU'),('CCCAAUU','CGGGUU'),('CCCAAU','GGGUU'),('CCAAU','GGUU')]: + for mode in ('H','M'): + for ov in ('N','T','Q'): + code,out,err=run(['-t',t,'-q',q,'--seedBP=2','--energy=B','--acc=N','--mode='+mode,'--outOverlap='+ov]) + print(t,q,mode,ov,out.replace('\n',' / '),err) diff --git a/doc/local-validation-archive/files/issue-212-validation/distribution-doc/Makefile.am b/doc/local-validation-archive/files/issue-212-validation/distribution-doc/Makefile.am new file mode 100644 index 00000000..acfa3bd7 --- /dev/null +++ b/doc/local-validation-archive/files/issue-212-validation/distribution-doc/Makefile.am @@ -0,0 +1,17 @@ + +################################################################ +# extra files to be added to distribution +################################################################ + +EXTRA_DIST = \ + analysis/out-overlap.md \ + analysis/out-overlap/reproduce.py \ + conda.txt \ + doxygen.cfg \ + latex-deps/adjcalc.sty \ + latex-deps/adjustbox.sty \ + latex-deps/collectbox.sty \ + latex-deps/tc-pgf.def \ + latex-deps/tocloft.sty \ + latex-deps/trimclip.sty \ + latex-deps/xtab.sty diff --git a/doc/local-validation-archive/files/issue-212-validation/distribution-doc/analysis/out-overlap.md b/doc/local-validation-archive/files/issue-212-validation/distribution-doc/analysis/out-overlap.md new file mode 100644 index 00000000..1887c21e --- /dev/null +++ b/doc/local-validation-archive/files/issue-212-validation/distribution-doc/analysis/out-overlap.md @@ -0,0 +1,414 @@ +# Output overlap analysis for issue 212 + +This report examines `--outOverlap=B,N,T,Q` at master revision +`afede020fe7d1c44cf686684dc7eb40927f91583` (2026-10-02). It answers the +[request for analysis and consultation](https://github.com/BackofenLab/IntaRNA/issues/212#issuecomment-5927769865). +It proposes changes but implements none of them. + +There are reproducible correctness bugs beyond the documented enumeration +heuristic. Region merging can violate the requested overlap rule, and some +heuristic predictors bypass output validation or use invalid energies for later +results. Separately, even exact prediction retains only one right extension per +left boundary for non-overlapping output. Consequently, a missing suboptimal +result does not establish that no compatible interaction exists. + +## Findings and proposed priority + +| Finding | Scope | Assessment | Proposed priority | +| --- | --- | --- | --- | +| Independent region results can overlap on a forbidden sequence | `N`, `T`, `Q`, with multiple region combinations | Correctness bug under the default output per sequence pair | High | +| Exhausted seed-free heuristic ensemble enumeration emits an invalid energy and repeats a site | `P/H --noSeed`, `N/T/Q`, requesting more results than available | Undefined numeric conversion | High | +| Later seed-free heuristic ensemble results omit accessibility contributions | `P/H --noSeed`, `N/T/Q` | Incorrect reported energy, ranking and energy filtering | High | +| Later heuristic MFE results can ignore `outNoGUend` | Confirmed for `S/H`, seeded and seed-free, with `T` | Output constraint violation | High | +| Region merging does not apply a global `outDeltaE` threshold | All four overlap modes, multiple regions | Same local versus global scope problem | Medium | +| Valid non-overlapping alternatives can be discarded before selection | `N/T/Q`, including `mode=M` | Documented heuristic with a substantial completeness limitation | Consult on semantics and cost | + +These priorities are recommendations, not implementation decisions. The +ensemble-specific findings also explain a single-region overlap violation; +regional merging is not the only way to obtain one. + +## What the four settings mean + +Overlap concerns inclusive interaction **intervals**, including unpaired bases +between the outermost intermolecular pairs. It does not mean shared base pairs +or only shared paired nucleotides. Coordinates below use the normal one-based +forward target and query indices. + +| Setting | Target overlap allowed | Query overlap allowed | Reason to reject a later interaction | +| --- | --- | --- | --- | +| `B` | Yes | Yes | Neither interval is excluded by this option | +| `T` | Yes | No | Its query interval intersects an already selected query interval | +| `Q` | No | Yes | Its target interval intersects an already selected target interval | +| `N` | No | No | Either interval intersects a selected interval on that sequence | + +The CLI mapping and `PredictorMfe::reportOptima()` implement this interpretation +correctly within their local selection state. The internally reversed query +coordinates are converted through the energy interface. There is no evidence +here of a simple swapped `T`/`Q` flag or reversed-query indexing error. + +The [README](../../README.md#subopts) promises **up to** `outNumber` results, +including the optimum. `B` does not enumerate every structure: exact prediction +normally retains the best structure for each boundary pair, and heuristic +predictors can consider fewer candidates. Allowing an overlap does not require +one. `N` does not search for the best *set* of simultaneously occupied sites; +the current algorithm chooses an optimum first and then compatible alternatives. + +## Reproduction setup + +Build IntaRNA using the [repository instructions](../../AGENTS.md), then run +from the repository root: + +```bash +python3 doc/analysis/out-overlap/reproduce.py ./src/bin/IntaRNA \ + --output /tmp/intarna-overlap-observations.json +``` + +The script captures the arguments, exit status, output, and forbidden interval +overlaps for 118 invocations. It checks 64 non-exhausting controls: all four +overlap settings for 16 CLI predictor configurations (including RIblast mode). +It independently enumerates every antiparallel structure for three tiny +seed-free examples and verifies their exact `B` site energies. Known defects +are recorded as observations, not installed as regression-test expectations. + +All commands below use a Bash helper to keep the examples short: + +```bash +rna() { + ./src/bin/IntaRNA --threads=1 --outMode=C \ + --outCsvCols=start1,end1,start2,end2,E "$@" +} +``` + +Except where stated, examples use `energy=B` (minus one kcal/mol per pair) and +`acc=N` (zero accessibility penalty) to make energies directly inspectable. These +are demonstrations of program behavior, not biological predictions. Explicit +small seeds and seed-free examples are intentional. `--outDeltaE=100` makes +the energy window non-limiting in these examples. + +## 1 Regional merging permits forbidden overlaps + +```bash +rna -t CCAACC -q GG --energy=B --acc=N --seedBP=2 \ + --tRegion=1-2,5-6 --outOverlap=N -n 10 --outDeltaE=100 +``` + +Actual output: + +```text +start1;end1;start2;end2;E +1;2;1;2;-2 +5;6;1;2;-2 +``` + +The query interval `1..2` is reused. With the default `outPerRegion=false`, +`N` should permit at most one of these two interactions. `T` also produces this +violation; `Q` and `B` permit these rows. Conversely: + +```bash +rna -t CC -q GGAAGG --energy=B --acc=N --seedBP=2 \ + --qRegion=1-2,5-6 --outOverlap=Q -n 10 --outDeltaE=100 +``` + +reports target `1..2` with query `1..2` and `5..6`, violating `Q` and, if selected, +`N`. The permitted cases are `T` and `B`. + +Automatic decomposition reaches the same path: + +```bash +rna -t UAUCGGCC -q GG --energy=B --acc=C --seedBP=2 \ + --tRegionLenMax=4 --outOverlap=N -n 10 --outDeltaE=100 +``` + +This reports `(3..4,1..2,-2)` and `(7..8,1..2,-2)`. With `-v`, the prediction +messages show the two target regions. The problem also occurs with ViennaRNA +energies: replace the first example's target/query with `CCCACCC`/`GGG`, use +`--tRegion=1-3,5-7`, and omit `--energy=B`. It reports `-4.2` and `-3` kcal/mol +while reusing query `1..3`. + +**Cause.** [IntaRNA.cpp](../../src/bin/IntaRNA.cpp) creates a fresh predictor +for every region/window combination. Each predictor clears its own +`reportedInteractions` lists. +[OutputHandlerInteractionList::add](../../src/IntaRNA/OutputHandlerInteractionList.cpp) +merges, sorts, deduplicates and truncates their results but never enforces +`reportOverlap` across combinations. Disjoint target regions do not prevent +their interactions from sharing a query region, and vice versa. + +**Proposed change.** For default output per sequence pair, apply overlap +selection in a common sequence-pair context using original sequence +coordinates. Preserve independent selection when users explicitly request +`outPerRegion=true`, and document that scope. The reproduction with +`outPerRegion=true -n 1` intentionally produces one result per region, so its +combined overlaps are not evidence of a violation of that independent scope. + +A final overlap filter alone can ensure validity, but cannot ensure the best +available compatible output or fill the requested count. A region's local +winner may suppress a local alternative and later be rejected against a better +winner from another region. That suppressed alternative can now be admissible. +Selection therefore needs unpruned candidate streams, or a way to request or +recompute alternatives after global exclusions. Applying a fixed top-k cap +before compatibility filtering has the same shortfall. + +Window mode is explicitly different: the CLI rejects `outNumber>1` with +`N/T/Q` when `windowWidth` is set. The reproduction script checks that guard. +Manual and automatic regions currently lack equivalent protection or global +selection. `--rri` evaluation intentionally ignores overlap and other prediction +filters and is outside this report's prediction contract. + +## 2 Exhausted ensemble enumeration converts infinity to an integer + +```bash +rna -t CC -q GG --energy=B --acc=N --model=P --mode=H --noSeed \ + --outOverlap=N -n 2 --outDeltaE=100 +``` + +Actual output on the checked GCC build: + +```text +start1;end1;start2;end2;E +1;2;1;2;-2.14748e+07 +1;2;1;2;-2 +``` + +Only the `-2` row is valid. The corrupted row is sorted ahead of the real +optimum by the collector. `T` and `Q` reproduce the problem; exact `mode=M` +returns only the valid row. `B` follows a different enumeration path and does +not encounter this exhaustion conversion. + +**Cause.** In +[PredictorMfeEns2dHeuristic::getNextBest](../../src/IntaRNA/PredictorMfeEns2dHeuristic.cpp), +`curBestCellE` has type `Z_type` and starts at `Z_INF`. With no eligible cell, +the function assigns floating-point infinity to the integer `curBest.energy`. +It leaves the previous coordinates intact. The invalid converted energy can +pass the next report-loop checks. This is undefined behavior; the exact printed +number and even the failure mode are not portable. + +A focused rebuild of this unmodified translation unit with GCC 14.4 and +`-fsanitize=undefined,float-cast-overflow -fno-sanitize-recover=all` confirmed: + +```text +PredictorMfeEns2dHeuristic.cpp:291:19: runtime error: +inf is outside the range of representable values of type 'int' +``` + +**Proposed change.** Represent candidate energies with `E_type` and its +`E_INF` sentinel, and explicitly return exhaustion before updating coordinates +or emitting another result. Test zero, one and several remaining candidates +for all restricted overlap modes, with both boundary-only and traced output. +Reusing the final validated candidate path described next would remove the +need for this independent raw-matrix enumeration. + +## 3 Later ensemble results use the wrong energy + +This is distinct from exhaustion; `-n 2` below stops while a real second +candidate exists. + +```bash +rna -t AGAGC -q GAUUC --energy=B --acc=C --model=P --mode=H --noSeed \ + --outOverlap=N -n 2 --outDeltaE=100 --outMaxE=0 +``` + +Actual output: + +```text +start1;end1;start2;end2;E +1;2;3;4;-2 +5;5;1;1;-1 +``` + +The second site has `ED1=0`, `ED2=1.31`, and total energy `0.31`. The same +predictor reports `0.31` for that site with `--outOverlap=B -n 100 +--outMaxE=100`; include `ED1,ED2` in `outCsvCols` to inspect the penalties. +There is only one pair in the site, so no alternative interior structure +explains this difference. It should be excluded by `outMaxE=0`. + +**Cause.** The heuristic ensemble `getNextBest()` uses +`energy.getE(curCell->val)`, the hybrid score from its recursion matrix. +The first-result path instead calls `updateOptimaUsingZ()` on finalized site +partition values and `updateOptima(..., isHybridE=true)`, which adds the site +energy contributions. Thus changing the overlap option changes the energy +definition for later rows. This also invalidates their ranking and their +comparison with `outDeltaE`/`outMaxE`. + +**Proposed change.** Enumerate later results from the same finalized, filtered +site energies used for the first result. Adding ED to a raw matrix score alone +would not establish equivalence with finalized per-site partition aggregation. +Test the energy of identical boundaries across overlap modes, including +nonzero ED and energy cutoffs. Seeded `P/H` uses a different predictor class; +this reproducer and the raw-matrix diagnosis concern `P/H --noSeed`. + +## 4 Later heuristic results bypass terminal pair filtering + +```bash +rna -t UUGA -q CAUU --energy=B --acc=N --model=S --mode=H --noSeed \ + --outNoGUend --outNoLP=false --outOverlap=T -n 10 --outDeltaE=100 +``` + +Actual output: + +```text +start1;end1;start2;end2;E +1;3;1;2;-2 +3;4;3;4;-2 +``` + +The second row pairs target G at position 3 with query U at position 4. This +terminal GU pair violates `outNoGUend`. The row is absent with `B` and with +exact `S/M`. Replacing `--noSeed` by `--seedBP=2 --seedMaxUP=2` reproduces the +same violation in seeded `S/H`. The corresponding seeded `X/H` control does +not reproduce it. + +**Cause.** [PredictorMfe::updateOptima](../../src/IntaRNA/PredictorMfe.cpp) +checks terminal GU and maximum ED before recording output candidates. The +specialized `getNextBest()` functions in +[PredictorMfe2dHeuristic](../../src/IntaRNA/PredictorMfe2dHeuristic.cpp) and +[PredictorMfe2dHeuristicSeed](../../src/IntaRNA/PredictorMfe2dHeuristicSeed.cpp) +read recursion cells directly and bypass that validation. The raw state can +legitimately contain intermediate extensions that are unsuitable as complete +reported interactions. + +**Proposed change.** Share complete-candidate validation and the final energy +calculation between initial and subsequent selection. Audit the analogous +helix and ensemble overrides for all output constraints, including maximum ED; +the GU reproducer above establishes the two `S/H` paths, not every possible +constraint failure in every override. Merely rejecting the bad final row may +still hide a valid alternative at its starting position, so candidate retention +must also be considered. + +## 5 Energy windows are local to regions + +```bash +rna -t CCAAC -q GG --energy=B --acc=N --model=S --noSeed \ + --tRegion=1-2,5-5 --outOverlap=B -n 10 --outDeltaE=0 +``` + +Actual output: + +```text +start1;end1;start2;end2;E +1;2;1;2;-2 +5;5;1;1;-1 +5;5;2;2;-1 +``` + +The pair's best energy is `-2`. The `-1` rows are outside a zero-width energy +window around it. They survive because each predictor uses its own local MFE +as the reference and the collector does not apply `deltaE` again. The script +reproduces this in all four overlap settings; restricted modes change how many +of the local rows survive, not the scope of the threshold. + +**Proposed change.** Under default output per sequence pair, evaluate the +energy window relative to that pair's best eligible candidate when combining +regions. Under explicit output per region, retain and document local windows. +This is coupled to the global selector in finding 1; `B` needs the global +energy-window check even though it needs no overlap rejection. + +## 6 A valid right extension can be discarded before selection + +```bash +rna -t CCCAAU -q GGGUU --energy=B --acc=N --seedBP=2 \ + --model=X --mode=M --outOverlap=N -n 100 --outDeltaE=100 +``` + +Only `(1..3,1..3,-3)` is returned. Yet `(4..5,4..5,-2)`, a two-pair AU seed, +is disjoint on both sequences. It appears with `--outOverlap=B`. It is also +returned by rerunning this particular example with +`--tAccConstr=b:1-3 --qAccConstr=b:1-3`. `mode=H` shows the same omission; +changing `N` to `T` does too. + +**Cause.** [PredictorMfe::updateMfe4leftEnd](../../src/IntaRNA/PredictorMfe.cpp) +stores just one best right end for each `(i1,i2)`. At target start 4 and forward +query end 5, it retains the better `(4..6,1..5,-3)` extension. That extension +overlaps the first result on the query. The shorter `(4..5,4..5,-2)` extension +has already been discarded, so `getNextBest()` cannot recover it. Several +heuristic predictors instead scan matrices with the same one-extension limit. + +The following still smaller seed-free examples establish the limitation for +each restricted mode. The script verifies their complete exact `B` site +energies against an independent structure enumerator. Use `model=S`, `mode=M`, +`noSeed`, `outNoLP=false`, `energy=B`, `acc=N`, `outDeltaE=100`, and `-n 1000`. + +| Mode | Target | Query | Only reported site and energy | Omitted compatible site and energy | +| --- | --- | --- | --- | --- | +| `N` | `CCAU` | `CGGU` | `(1..2,2..3,-2)` | `(3..3,4..4,-1)` | +| `T` | `GUA` | `GUA` | `(1..2,1..2,-2)` | `(2..2,3..3,-1)` | +| `Q` | `GCGC` | `AGUG` | `(2..4,2..4,-3)` | `(1..1,3..3,-1)` | + +This behavior is already acknowledged in the README's warning about +non-overlapping output. It is therefore a documented completeness limitation, +not evidence that every missing suboptimal row is a newly introduced bug. +Calling `mode=M` exact can nevertheless mislead users unless the distinction +between exact candidate calculation and heuristic restricted enumeration is +explicit. These examples are small witnesses; no global minimality claim is +made. + +**Proposed choices for consultation.** Keep and clearly document the heuristic, +or support complete greedy selection by retaining more endpoints or recomputing +the best candidate after excluding previously selected intervals. For `N`, +recomputation must cover all combinations of remaining target and query +intervals; for `T` or `Q`, only the forbidden-overlap sequence is split. Avoid +changing the underlying accessibility model when excluding interaction spans. +Retaining all boundaries can require quartic storage, whereas repeated +prediction trades memory for time. Do not promise exact completeness for a +heuristic predictor merely because output selection has improved. + +Blocking paired nucleotides is not a general replacement for interval exclusion. +For example, `-t CCGG -q CCGG --model=S --mode=M --noSeed --energy=B --acc=N +--tAccConstr=b:2-3 --outOverlap=N -n 2 --outDeltaE=100` still produces target +interval `1..4`, with the blocked positions inside a loop. A span-based selector +must reject crossings over an excluded interval, not just forbid pairs at its +positions. + +## Decisions and verification needed before implementation + +The recommended contract is: choose the best eligible interaction first, then +greedily choose the best eligible interaction compatible with every previously +selected interval, within a documented sequence-pair or region scope. This +preserves the current interpretation of ranked alternatives. Optimizing the +number of interactions or the total energy of a compatible set is a different +objective that can discard the single best interaction; it should not be +introduced implicitly as an overlap bug fix. + +Agree on that scope, the desired completeness/runtime tradeoff, and tie-breaking +before changing enumeration. `Interaction::operator<`, the base map scan, and +specialized reverse matrix scans currently use different tie rules. Equal-energy +choices can change which later intervals remain available. This report does +not classify that unspecified ordering as a separate bug, but a common selector +needs an explicit rule in original sequence coordinates. + +Suggested implementation order: + +1. Repair the invalid ensemble sentinel and use common validated final energies + for all emitted candidates. Add exhaustion, ED and terminal-pair regressions. +2. Enforce sequence-pair overlap and energy-window scope across regions, with + explicit independent behavior for `outPerRegion`. Include alternatives that + become available when a local winner is globally rejected. +3. Decide whether to retain the documented one-extension heuristic or introduce + more complete restricted enumeration, with measured memory/runtime costs. + +Future regressions should check interval validity, energy identity, output +constraints, maximum count and completeness separately. Cover `B/N/T/Q`, +multiple regions on either sequence, automatic decomposition, nonzero region +offsets, shifted output indices, tied energies, seeds, terminal GU constraints, +nonzero accessibility and exhaustion. Existing overlap fixtures cap interaction +lengths at two bases, so they cannot detect a discarded longer/shorter extension +like finding 6. The independent toy enumerator is a useful oracle for those +regressions; the known-bug observations themselves are not correct expected +output. + +## Validation and limits + +The analysis used a fresh release build of the stated master revision with +GCC 14.4, bundled Kokkos mdspan, ViennaRNA 2.7.2 and Boost 1.85.0 on Linux. +The 118-run script passed all 64 control checks and three independent site +oracles. All 20 existing fixtures passed via `INTARNABINPATH="$PWD" +bash tests/runIntaRNA.sh` when run from `tests` with `INTARNABINPATH` set to +the repository root (the full API suite was not rerun for this analysis-only +change). The focused sanitizer probe instrumented the ensemble heuristic +translation unit, not the entire application. No production source, CLI +behavior, or existing test expectations were modified. + +These controls establish the reported examples and shared code paths, not +exhaustive correctness of every predictor, energy model or constraint +combination. In particular, the successful `B` controls do not prove exhaustive +structure enumeration, and the platform-specific numeric manifestation of +undefined behavior must not be used as a portable regression expectation. diff --git a/doc/local-validation-archive/files/issue-212-validation/explore-decomposition.py b/doc/local-validation-archive/files/issue-212-validation/explore-decomposition.py new file mode 100644 index 00000000..79051b1a --- /dev/null +++ b/doc/local-validation-archive/files/issue-212-validation/explore-decomposition.py @@ -0,0 +1,11 @@ +import subprocess,random,json,csv,io +b='/home/alex/CodexProjects/IntaRNA/pr-worktrees/issue-212/src/bin/IntaRNA' +rng=random.Random(212) +for length in range(8,21): + for n in range(100): + t=''.join(rng.choices('ACGU',k=length));q='GG' + cmd=[b,'-t',t,'-q',q,'--energy=B','--acc=C','--seedBP=2','--outOverlap=N','-n','10','--outMode=C','--outCsvCols=start1,end1,start2,end2,E','--tRegionLenMax=4','--outDeltaE=100','--threads=1','--outMaxE=100','-v'] + p=subprocess.run(cmd,capture_output=True,text=True) + rows=list(csv.DictReader(io.StringIO('\n'.join(x for x in p.stdout.splitlines() if not x.startswith('#'))),delimiter=';')) + if len(rows)>1: + print(json.dumps({'t':t,'q':q,'rows':rows,'stderr':p.stderr,'cmd':cmd}),flush=True);raise SystemExit diff --git a/doc/local-validation-archive/files/issue-212-validation/explore-ensemble.py b/doc/local-validation-archive/files/issue-212-validation/explore-ensemble.py new file mode 100644 index 00000000..c4ed25d2 --- /dev/null +++ b/doc/local-validation-archive/files/issue-212-validation/explore-ensemble.py @@ -0,0 +1,18 @@ +import subprocess,csv,io,random,json +b='/home/alex/CodexProjects/IntaRNA/pr-worktrees/issue-212/src/bin/IntaRNA' +base=[b,'--energy=B','--acc=C','--model=P','--mode=H','--noSeed','--outMode=C','--outCsvCols=start1,end1,start2,end2,E,ED1,ED2','--outNumber=1000','--outDeltaE=100','--outMaxE=100','--threads=1','--outNoLP=false'] +def run(t,q,ov): + p=subprocess.run(base+['-t',t,'-q',q,'--outOverlap='+ov],capture_output=True,text=True) + if p.returncode: raise Exception(p.stderr) + return list(csv.DictReader(io.StringIO(p.stdout),delimiter=';')) +rng=random.Random(213) +found={} +for length in range(3,9): + for n in range(200): + t=''.join(rng.choices('ACGU',k=length));q=''.join(rng.choices('ACGU',k=length)); bb=run(t,q,'B') + match={tuple(r[x] for x in ('start1','end1','start2','end2')):r for r in bb} + for ov in 'NTQ': + rows=run(t,q,ov) + mismatches=[(r,match.get(tuple(r[x] for x in ('start1','end1','start2','end2')))) for r in rows if tuple(r[x] for x in ('start1','end1','start2','end2')) in match and float(r['E'])>-1000 and r['E']!=match[tuple(r[x] for x in ('start1','end1','start2','end2'))]['E']] + if mismatches: + print(json.dumps({'t':t,'q':q,'ov':ov,'rows':rows,'mismatch':mismatches}),flush=True);raise SystemExit diff --git a/doc/local-validation-archive/files/issue-212-validation/explore-filters.py b/doc/local-validation-archive/files/issue-212-validation/explore-filters.py new file mode 100644 index 00000000..f52eed93 --- /dev/null +++ b/doc/local-validation-archive/files/issue-212-validation/explore-filters.py @@ -0,0 +1,19 @@ +import subprocess,csv,io,random,json +binary='/home/alex/CodexProjects/IntaRNA/pr-worktrees/issue-212/src/bin/IntaRNA' +base=[binary,'--energy=B','--acc=N','--model=S','--mode=H','--noSeed','--outMode=C','--outCsvCols=start1,end1,start2,end2,E,hybridDB','--outNumber=10','--outDeltaE=100','--threads=1','--outNoLP=false','--outNoGUend'] +rng=random.Random(212) +for length in range(3,10): + found=False + for n in range(300): + t=''.join(rng.choices('ACGU',k=length));q=''.join(rng.choices('ACGU',k=length)) + for ov in 'NTQ': + cmd=base+['-t',t,'-q',q,'--outOverlap='+ov] + p=subprocess.run(cmd,capture_output=True,text=True) + if p.returncode: raise Exception(p.stderr) + rows=list(csv.DictReader(io.StringIO(p.stdout),delimiter=';')) + invalid=[r for r in rows if any(t[int(r[a])-1]+q[int(r[b])-1] in ('GU','UG') for a,b in [('start1','end2'),('end1','start2')])] + if invalid: + print(json.dumps({'t':t,'q':q,'ov':ov,'rows':rows,'invalid':invalid,'cmd':cmd}),flush=True) + found=True;break + if found:break + if found:break diff --git a/doc/local-validation-archive/files/issue-212-validation/explore.py b/doc/local-validation-archive/files/issue-212-validation/explore.py new file mode 100644 index 00000000..a5f4b0aa --- /dev/null +++ b/doc/local-validation-archive/files/issue-212-validation/explore.py @@ -0,0 +1,28 @@ +import subprocess, csv, io, random, json +from pathlib import Path +binary='/home/alex/CodexProjects/IntaRNA/pr-worktrees/issue-184/src/bin/IntaRNA' +base=[binary,'--energy=B','--acc=N','--model=S','--mode=M','--noSeed','--outMode=C','--outCsvCols=start1,end1,start2,end2,E,hybridDB','--outNumber=1000','--outDeltaE=100','--threads=1','--outNoLP=false'] +def run(t,q,ov,extra=()): + p=subprocess.run(base+['-t',t,'-q',q,'--outOverlap='+ov]+list(extra),capture_output=True,text=True) + if p.returncode: raise Exception(p.stderr) + return list(csv.DictReader(io.StringIO(p.stdout),delimiter=';')) +def overlap(a,b,axis): + return max(int(a['start'+axis]),int(b['start'+axis]))<=min(int(a['end'+axis]),int(b['end'+axis])) +def allowed(a,b,ov): return all(not overlap(a,b,axis) for axis in {'N':'12','T':'2','Q':'1','B':''}[ov]) +rng=random.Random(212) +found={} +for length in range(3,10): + for n in range(200): + t=''.join(rng.choices('ACGU',k=length)); q=''.join(rng.choices('ACGU',k=length)) + b=run(t,q,'B') + if len(b)<2: continue + for ov in 'NTQ': + if ov in found: continue + a=run(t,q,ov) + missing=[x for x in b if all(allowed(x,y,ov) for y in a)] + if missing: + found[ov]={'t':t,'q':q,'actual':a,'missing':missing[:5]} + print(ov,json.dumps(found[ov]),flush=True) + if len(found)==3: break + if len(found)==3: break +Path('/home/alex/CodexProjects/IntaRNA/issue-212-validation/explore-results.json').write_text(json.dumps(found,indent=2)) diff --git a/doc/local-validation-archive/files/issue-212-validation/review-debug-build.sh b/doc/local-validation-archive/files/issue-212-validation/review-debug-build.sh new file mode 100644 index 00000000..163dc110 --- /dev/null +++ b/doc/local-validation-archive/files/issue-212-validation/review-debug-build.sh @@ -0,0 +1,18 @@ +#!/usr/bin/env bash +set -euo pipefail +validation=/home/alex/CodexProjects/IntaRNA/issue-212-validation +dependencies=/home/alex/CodexProjects/IntaRNA/issue-246-validation/toolchain +compiler=/home/alex/CodexProjects/IntaRNA/issue-246-validation/toolchain-gcc14 +export PATH="$compiler/bin:$PATH" +export PKG_CONFIG_PATH="$dependencies/lib/pkgconfig" +export OPENBLAS_NUM_THREADS=1 +cd "$validation/review-debug" +bash autotools-init.sh > "$validation/logs/review-debug-bootstrap.log" 2>&1 +./configure CXX="$compiler/bin/x86_64-conda-linux-gnu-g++" \ + CC="$compiler/bin/x86_64-conda-linux-gnu-gcc" CPPFLAGS="-I$dependencies/include" \ + CXXFLAGS="-O0 -g -fsanitize=undefined,float-cast-overflow -fno-sanitize-recover=all" \ + LDFLAGS="-L$dependencies/lib -Wl,-rpath,$dependencies/lib -fsanitize=undefined,float-cast-overflow" \ + --with-boost="$dependencies" --with-boost-libdir="$dependencies/lib" \ + --with-vrna="$dependencies" --prefix="$validation/install-review-debug" \ + --enable-debug > "$validation/logs/review-debug-configure.log" 2>&1 +make tests -j2 > "$validation/logs/review-debug-tests.log" 2>&1 diff --git a/doc/local-validation-archive/files/issue-212-validation/review-debug/tests/PredictorMfeHeuristicCellState_test.cpp b/doc/local-validation-archive/files/issue-212-validation/review-debug/tests/PredictorMfeHeuristicCellState_test.cpp new file mode 100644 index 00000000..fe0235aa --- /dev/null +++ b/doc/local-validation-archive/files/issue-212-validation/review-debug/tests/PredictorMfeHeuristicCellState_test.cpp @@ -0,0 +1,303 @@ +#include "catch.hpp" + +#undef NDEBUG + +#include "IntaRNA/AccessibilityDisabled.h" +#include "IntaRNA/InteractionEnergyBasePair.h" +#include "IntaRNA/OutputHandlerInteractionList.h" +#include "IntaRNA/PredictorMfe2dHeuristic.h" +#include "IntaRNA/PredictorMfe2dHeuristicSeed.h" +#include "IntaRNA/PredictorMfeEns2dHeuristic.h" +#include "IntaRNA/PredictorMfe2dHelixBlockHeuristic.h" +#include "IntaRNA/PredictorMfe2dHelixBlockHeuristicSeed.h" +#include "IntaRNA/SeedHandlerMfe.h" +#include "IntaRNA/ReverseAccessibility.h" +#include "IntaRNA/RnaSequence.h" +#include "IntaRNA/SeedConstraint.h" +#include "IntaRNA/SeedHandlerNoBulge.h" + +#include +#include +#include + +using namespace IntaRNA; + +namespace { + +class InspectableMfeEns2dHeuristic : public PredictorMfeEns2dHeuristic { +public: + InspectableMfeEns2dHeuristic(const InteractionEnergy & energy, + OutputHandler & output) + : PredictorMfeEns2dHeuristic(energy, output, NULL) + {} + + Z_type getBoundaryZ(const size_t i1, const size_t j1, + const size_t i2, const size_t j2) const + { + const auto entry = Z_partition.find(Interaction::Boundary(i1, j1, i2, j2)); + return entry == Z_partition.end() ? Z_type(0) : entry->second; + } +}; + +void requireThreePairStack(const OutputHandlerInteractionList & output) { + REQUIRE_FALSE(output.empty()); + const Interaction & interaction = **output.begin(); + REQUIRE(interaction.energy == Ekcal_2_E(-3.0)); + REQUIRE(interaction.basePairs.front() == Interaction::BasePair(0, 3)); + REQUIRE(interaction.basePairs.back() == Interaction::BasePair(2, 1)); +} + +} // namespace + +TEST_CASE("heuristic cells reset their incumbent energy", "[PredictorMfeHeuristicCellState]") { + + #include "testEasyLoggingSetup.icc" + + // In reversed query coordinates, the diagonal interaction is GC-GU-GC. + // The GU pair is internal and therefore valid when terminal GU pairs are + // filtered. With stacking-only loops, its energy is exactly -3 kcal/mol. + RnaSequence target("target", "GGG"); + RnaSequence query("query", "CCUC"); + AccessibilityDisabled targetAcc(target, 0, NULL); + AccessibilityDisabled queryAcc(query, 0, NULL); + ReverseAccessibility reverseQueryAcc(queryAcc); + InteractionEnergyBasePair energy(targetAcc, reverseQueryAcc, 0, 0); + OutputConstraint constraint(1, OutputConstraint::OVERLAP_BOTH, + E_INF, E_INF, false, false, true, true, true); + + SECTION("unseeded MFE retains an internal GU continuation") { + OutputHandlerInteractionList output(constraint, 1); + PredictorMfe2dHeuristic predictor(energy, output, NULL); + + predictor.predict(); + + requireThreePairStack(output); + REQUIRE((**output.begin()).basePairs.size() == 3); + REQUIRE((**output.begin()).basePairs.at(1) + == Interaction::BasePair(1, 2)); + } + + SECTION("ensemble heuristic retains the internal GU boundary partition") { + OutputHandlerInteractionList output(constraint, 1); + PredictorMfeEns2dHeuristic predictor(energy, output, NULL); + + predictor.predict(); + + requireThreePairStack(output); + // Ensemble traceback intentionally reports boundaries only. + REQUIRE((**output.begin()).basePairs.size() == 2); + } + + SECTION("seeded MFE can extend a seed through an internal GU cell") { + // Only the leading GC-GC seed is admissible. Its right extension is + // GC-GU-GC in the unseeded matrix, so the seed cannot bypass the + // poisoned cell via a later seed start. + RnaSequence seedTarget("seedTarget", "GGGG"); + RnaSequence seedQuery("seedQuery", "CCUCC"); + AccessibilityDisabled seedTargetAcc(seedTarget, 0, NULL); + AccessibilityDisabled seedQueryAcc(seedQuery, 0, NULL); + ReverseAccessibility reverseSeedQueryAcc(seedQueryAcc); + InteractionEnergyBasePair seedEnergy( + seedTargetAcc, reverseSeedQueryAcc, 0, 0); + IndexRangeList targetSeedRange; + targetSeedRange.push_back(IndexRange(0, 1)); + IndexRangeList querySeedRange; + querySeedRange.push_back(IndexRange(0, 1)); + SeedConstraint seedConstraint(2, 0, 0, 0, + E_INF, Accessibility::ED_UPPER_BOUND, E_INF, + targetSeedRange, querySeedRange, "", false, false, true); + OutputConstraint seedOutputConstraint(1, OutputConstraint::OVERLAP_BOTH, + E_INF, E_INF, false, true, true, true, true); + OutputHandlerInteractionList output(seedOutputConstraint, 1); + PredictorMfe2dHeuristicSeed predictor(seedEnergy, output, NULL, + new SeedHandlerNoBulge(seedEnergy, seedConstraint)); + + predictor.predict(); + + REQUIRE_FALSE(output.empty()); + const Interaction & interaction = **output.begin(); + REQUIRE(interaction.energy == Ekcal_2_E(-4.0)); + REQUIRE(interaction.basePairs.size() == 4); + REQUIRE(interaction.basePairs.front() == Interaction::BasePair(0, 4)); + REQUIRE((**output.begin()).basePairs.at(1) + == Interaction::BasePair(1, 3)); + REQUIRE((**output.begin()).basePairs.at(2) + == Interaction::BasePair(2, 2)); + REQUIRE(interaction.basePairs.back() == Interaction::BasePair(3, 1)); + } +} + +TEST_CASE("ensemble noLP heuristic keeps valid non-direct extensions", + "[PredictorMfeHeuristicCellState][PredictorMfeEns2dHeuristic]") { + + #include "testEasyLoggingSetup.icc" + + SECTION("a GU mandatory stack can be internal to non-GU boundaries") { + // Reversed query CUCC gives the unique best GC-GU-GC path at + // internal boundary (0,2,0,2). The mandatory second pair is GU, + // but the actual interaction right end is the following GC pair. + RnaSequence target("target", "GGG"); + RnaSequence query("query", "CCUC"); + AccessibilityDisabled targetAcc(target, 0, NULL); + AccessibilityDisabled queryAcc(query, 0, NULL); + ReverseAccessibility reverseQueryAcc(queryAcc); + InteractionEnergyBasePair energy(targetAcc, reverseQueryAcc, 0, 0); + OutputConstraint constraint(1, OutputConstraint::OVERLAP_BOTH, + E_INF, E_INF, false, true, true, true, true); + OutputHandlerInteractionList output(constraint, 1); + InspectableMfeEns2dHeuristic predictor(energy, output); + + predictor.predict(); + + const Z_type pathZ = std::exp(3.0); + const Z_type expectedZall = Z_type(2) * std::exp(2.0) + pathZ; + REQUIRE(predictor.getBoundaryZ(0, 2, 0, 2) + == Approx(pathZ).epsilon(1e-12)); + REQUIRE(predictor.getZall() == Approx(expectedZall).epsilon(1e-12)); + REQUIRE_FALSE(output.empty()); + const Interaction & interaction = **output.begin(); + REQUIRE(interaction.energy == Ekcal_2_E(-3.0)); + REQUIRE(interaction.basePairs.size() == 2); + REQUIRE(interaction.basePairs.front() == Interaction::BasePair(0, 3)); + REQUIRE(interaction.basePairs.back() == Interaction::BasePair(2, 1)); + } + + SECTION("an absent direct extension does not suppress a later bulge") { + // The first GC-GC block cannot continue directly because A-C at + // internal (2,2) is impossible. It can still cross the target A + // bulge to the second GC-GC block via loop offset (2,1). + RnaSequence target("target", "GGAGG"); + RnaSequence query("query", "CCCC"); + AccessibilityDisabled targetAcc(target, 0, NULL); + AccessibilityDisabled queryAcc(query, 0, NULL); + ReverseAccessibility reverseQueryAcc(queryAcc); + InteractionEnergyBasePair energy(targetAcc, reverseQueryAcc, 1, 1); + OutputConstraint constraint(1, OutputConstraint::OVERLAP_BOTH, + E_INF, E_INF, false, true, true, true, true); + OutputHandlerInteractionList output(constraint, 1); + InspectableMfeEns2dHeuristic predictor(energy, output); + + predictor.predict(); + + const Z_type pathZ = std::exp(4.0); + const Z_type expectedZall = Z_type(6) * std::exp(2.0) + pathZ; + REQUIRE(predictor.getBoundaryZ(0, 4, 0, 3) + == Approx(pathZ).epsilon(1e-12)); + REQUIRE(predictor.getZall() == Approx(expectedZall).epsilon(1e-12)); + REQUIRE_FALSE(output.empty()); + const Interaction & interaction = **output.begin(); + REQUIRE(interaction.energy == Ekcal_2_E(-4.0)); + REQUIRE(interaction.basePairs.size() == 2); + REQUIRE(interaction.basePairs.front() == Interaction::BasePair(0, 3)); + REQUIRE(interaction.basePairs.back() == Interaction::BasePair(4, 0)); + } +} + +namespace { + +std::unique_ptr makeOutputFilterPredictor(const InteractionEnergy & energy, + OutputHandler & output, const bool seeded, const bool helix) { + static const SeedConstraint seed(2, 2, 2, 2, E_INF, Accessibility::ED_UPPER_BOUND, E_INF, + IndexRangeList(), IndexRangeList(), "", false, false, true); + static const HelixConstraint helixConstraint(2, 4, 2, Accessibility::ED_UPPER_BOUND, E_INF, false); + if (helix) { + if (seeded) { + return std::make_unique( + energy, output, nullptr, helixConstraint, new SeedHandlerMfe(energy, seed)); + } + return std::make_unique(energy, output, nullptr, helixConstraint); + } + if (seeded) { + return std::make_unique(energy, output, nullptr, + new SeedHandlerMfe(energy, seed)); + } + return std::make_unique(energy, output, nullptr); +} + +// A small accessibility penalty confined to one end of the sequence. The +// second disjoint two-pair site is favorable but exceeds an ED limit of zero. +class EndPenaltyAccessibility : public AccessibilityDisabled { +public: + EndPenaltyAccessibility(const RnaSequence & sequence, const bool atStart) + : AccessibilityDisabled(sequence, 0, nullptr), atStart(atStart) {} + + E_type getED(const size_t from, const size_t to) const override { + const E_type base = AccessibilityDisabled::getED(from, to); + return base + ((atStart ? from < 2 : to >= 5) ? Ekcal_2_E(0.25) : 0); + } +private: + const bool atStart; +}; + +} // namespace + +TEST_CASE("heuristic suboptimals respect terminal GU constraints", "[PredictorMfeHeuristicCellState][Overlap]") { + #include "testEasyLoggingSetup.icc" + + for (bool seeded : {false, true}) { + for (bool helix : {false, true}) { + for (bool trace : {false, true}) { + for (size_t offset : {size_t(0), size_t(1)}) { + RnaSequence target("target", offset ? "NUUGAN" : "UUGA"); + RnaSequence query("query", offset ? "NCAUUN" : "CAUU"); + AccessibilityDisabled targetAcc(target, 0, nullptr); + AccessibilityDisabled queryAcc(query, 0, nullptr); + ReverseAccessibility reverseQueryAcc(queryAcc); + InteractionEnergyBasePair energy(targetAcc, reverseQueryAcc); + for (auto overlap : {OutputConstraint::OVERLAP_NONE, OutputConstraint::OVERLAP_SEQ1, + OutputConstraint::OVERLAP_SEQ2, OutputConstraint::OVERLAP_BOTH}) { + CAPTURE(seeded, helix, trace, offset, overlap); + OutputConstraint constraint(10, overlap, 0, Ekcal_2_E(100), + false, false, true, false, trace); + OutputHandlerInteractionList output(constraint, 10); + auto predictor = makeOutputFilterPredictor(energy, output, seeded, helix); + predictor->predict(IndexRange(offset, offset+3), IndexRange(offset, offset+3)); + REQUIRE_FALSE(output.empty()); + for (const Interaction * interaction : output) { + REQUIRE(interaction->isValid()); + for (auto bp : {interaction->basePairs.front(), interaction->basePairs.back()}) { + const char t = target.asString().at(bp.first), q = query.asString().at(bp.second); + REQUIRE_FALSE((t == 'G' && q == 'U')); + REQUIRE_FALSE((t == 'U' && q == 'G')); + } + } + } + } + } + } + } +} + +TEST_CASE("heuristic suboptimals respect complete-site accessibility limits", "[PredictorMfeHeuristicCellState][Overlap]") { + #include "testEasyLoggingSetup.icc" + + RnaSequence target("target", "CCCAACC"); + RnaSequence query("query", "GGAAGGG"); + EndPenaltyAccessibility targetAcc(target, false), queryAcc(query, true); + ReverseAccessibility reverseQueryAcc(queryAcc); + InteractionEnergyBasePair energy(targetAcc, reverseQueryAcc, 0, 0); + for (bool seeded : {false, true}) { + for (bool helix : {false, true}) { + for (bool trace : {false, true}) { + for (auto overlap : {OutputConstraint::OVERLAP_NONE, + OutputConstraint::OVERLAP_SEQ1, OutputConstraint::OVERLAP_SEQ2}) { + CAPTURE(seeded, helix, trace, overlap); + OutputConstraint constraint(10, overlap, 0, Ekcal_2_E(100), + false, false, false, false, trace, 0); + OutputHandlerInteractionList output(constraint, 10); + auto predictor = makeOutputFilterPredictor(energy, output, seeded, helix); + predictor->predict(); + REQUIRE(std::distance(output.begin(), output.end()) == 1); + for (const Interaction * interaction : output) { + REQUIRE(interaction->isValid()); + REQUIRE(targetAcc.getED(interaction->basePairs.front().first, + interaction->basePairs.back().first) == 0); + REQUIRE(queryAcc.getED(interaction->basePairs.back().second, + interaction->basePairs.front().second) == 0); + REQUIRE(interaction->energy == Ekcal_2_E(-3)); + } + } + } + } + } +} diff --git a/doc/local-validation-archive/files/issue-212-validation/sentinel.cpp b/doc/local-validation-archive/files/issue-212-validation/sentinel.cpp new file mode 100644 index 00000000..5c17984b --- /dev/null +++ b/doc/local-validation-archive/files/issue-212-validation/sentinel.cpp @@ -0,0 +1,18 @@ +#include "IntaRNA/AccessibilityDisabled.h" +#include "IntaRNA/InteractionEnergyBasePair.h" +#include "IntaRNA/OutputHandlerInteractionList.h" +#include "IntaRNA/PredictorMfeEns2dHeuristic.h" +#include "IntaRNA/ReverseAccessibility.h" +#include "IntaRNA/RnaSequence.h" +INITIALIZE_EASYLOGGINGPP +using namespace IntaRNA; +int main() { + RnaSequence target("target", "CC"), query("query", "GG"); + AccessibilityDisabled targetAcc(target,0,nullptr), queryAcc(query,0,nullptr); + ReverseAccessibility reverse(queryAcc); + InteractionEnergyBasePair energy(targetAcc,reverse); + OutputConstraint constraint(2,OutputConstraint::OVERLAP_NONE,0,10000); + OutputHandlerInteractionList output(constraint,2); + PredictorMfeEns2dHeuristic predictor(energy,output,nullptr); + predictor.predict(); +} diff --git a/doc/local-validation-archive/files/issue-244-validation/council-consensus.md b/doc/local-validation-archive/files/issue-244-validation/council-consensus.md new file mode 100644 index 00000000..eb494413 --- /dev/null +++ b/doc/local-validation-archive/files/issue-244-validation/council-consensus.md @@ -0,0 +1,18 @@ +# Issue #244 consensus record + +Source audited: `0a4568ad6e3da52f935221ff0da98a4a171e9833`. + +Three independent reviews covered technical requirements, scientific and +performance evidence, and maintenance/support policy. All recommended +retaining C++23 as the current implementation baseline, while distinguishing +purposeful use from scientific necessity or demonstrated performance gains. + +The technical review requested a precision change from “Lowering only the +configure requirement would break these uses” to “Changing only the configure +requirement would not make the current source compatible with earlier +standards.” The final text includes that correction. + +All three reviewers explicitly approved the exact final wording saved in +`issue-comment.md`. The root reviewer also approved it. The public comment +does not mention the review process or agents. No source/configuration change +or issue closure is part of the decision. diff --git a/doc/local-validation-archive/files/issue-244-validation/issue-comment.md b/doc/local-validation-archive/files/issue-244-validation/issue-comment.md new file mode 100644 index 00000000..8141b9db --- /dev/null +++ b/doc/local-validation-archive/files/issue-244-validation/issue-comment.md @@ -0,0 +1,7 @@ +**Decision: retain C++23 as the current implementation baseline; no change to `configure.ac`.** + +The code uses `std::string::resize_and_overwrite` to construct ViennaRNA constraint strings directly in owned storage without redundant initialization, and `std::string::contains` for sequence checks. Changing only the configure requirement would not make the current source compatible with earlier standards. + +This is an implementation and maintenance choice consistent with the [GCC 14 / Apple Clang support policy](https://github.com/BackofenLab/IntaRNA/discussions/232). It is not a scientific requirement: stream views need only C++20, and the ownership and partition-map improvements use earlier standards. The [recorded benchmarks](https://github.com/BackofenLab/IntaRNA/blob/0a4568ad6e3da52f935221ff0da98a4a171e9833/doc/refactor/2-c%2B%2B23.md#substantive-c23-follow-up-benchmark) preserve canonical outputs but do not demonstrate a material speedup attributable specifically to C++23. + +A lower language baseline would require a separate compatibility change and validation. diff --git a/doc/local-validation-archive/files/issue-244-validation/source-gcc11-debug/.github/workflows/build.yml b/doc/local-validation-archive/files/issue-244-validation/source-gcc11-debug/.github/workflows/build.yml new file mode 100644 index 00000000..ee4a0059 --- /dev/null +++ b/doc/local-validation-archive/files/issue-244-validation/source-gcc11-debug/.github/workflows/build.yml @@ -0,0 +1,132 @@ +name: C/C++ CI + +on: + push: + branches: [master, refactoring, 'refactor-to-c\+\+23'] + pull_request: + branches: [master, refactoring, 'refactor-to-c\+\+23'] + +defaults: + run: + shell: bash -el {0} + +jobs: + build: + name: Build and test (${{ matrix.name }}) + runs-on: ${{ matrix.os }} + strategy: + fail-fast: false + matrix: + include: + - name: GCC 11 release + os: ubuntu-22.04 + compiler: gcc + gcc_version: 11 + configure_debug: --disable-debug + - name: GCC 14 release + os: ubuntu-24.04 + compiler: gcc + gcc_version: 14 + configure_debug: --disable-debug + - name: GCC 14 debug + os: ubuntu-24.04 + compiler: gcc + gcc_version: 14 + configure_debug: --enable-debug + - name: Apple Clang release + os: macos-15 + compiler: apple-clang + configure_debug: --disable-debug + + steps: + - name: Check out sources + uses: actions/checkout@v7 + + - name: Set up conda environment + uses: conda-incubator/setup-miniconda@v4 + with: + miniconda-version: latest + environment-file: conda-build-env.yml + activate-environment: intarna-build-env + + - name: Install and select GCC ${{ matrix.gcc_version }} + if: matrix.compiler == 'gcc' + run: | + sudo apt-get update + sudo apt-get install --yes gcc-${{ matrix.gcc_version }} g++-${{ matrix.gcc_version }} + echo "CC=gcc-${{ matrix.gcc_version }}" >> "$GITHUB_ENV" + echo "CXX=g++-${{ matrix.gcc_version }}" >> "$GITHUB_ENV" + echo "LDFLAGS=-L${CONDA_PREFIX}/lib -Wl,-rpath,${CONDA_PREFIX}/lib" >> "$GITHUB_ENV" + + - name: Verify GCC ${{ matrix.gcc_version }} + if: matrix.compiler == 'gcc' + run: | + "$CC" --version + "$CXX" --version + test "$("$CC" -dumpfullversion | cut -d. -f1)" = ${{ matrix.gcc_version }} + test "$("$CXX" -dumpfullversion | cut -d. -f1)" = ${{ matrix.gcc_version }} + + - name: Set up Apple Clang with OpenMP + if: matrix.compiler == 'apple-clang' + run: | + brew install libomp + libomp_prefix="$(brew --prefix libomp)" + echo "CC=/usr/bin/clang" >> "$GITHUB_ENV" + echo "CXX=/usr/bin/clang++" >> "$GITHUB_ENV" + echo "CPPFLAGS=-I${libomp_prefix}/include" >> "$GITHUB_ENV" + echo "CXXFLAGS=-Xpreprocessor -fopenmp" >> "$GITHUB_ENV" + echo "LDFLAGS=-L${libomp_prefix}/lib -Wl,-rpath,${libomp_prefix}/lib -L${CONDA_PREFIX}/lib -Wl,-rpath,${CONDA_PREFIX}/lib" >> "$GITHUB_ENV" + echo "LIBS=-lomp" >> "$GITHUB_ENV" + vrna_libs="$(pkg-config --libs RNAlib2)" + vrna_libs="${vrna_libs//-Wl,-fno-lto/}" + echo "VRNA_LIBS=${vrna_libs}" >> "$GITHUB_ENV" + + - name: Verify Apple Clang + if: matrix.compiler == 'apple-clang' + run: | + "$CC" --version + "$CXX" --version + "$CXX" --version | grep -q "Apple clang" + + - name: Bootstrap and configure + run: | + export CXXFLAGS="${CXXFLAGS:-} -pedantic-errors" + echo "CXXFLAGS=$CXXFLAGS" >> "$GITHUB_ENV" + bash autotools-init.sh + ./configure \ + --prefix="$RUNNER_TEMP/intarna-install" \ + --with-vrna="$CONDA_PREFIX" \ + --with-boost="$CONDA_PREFIX" \ + --with-boost-libdir="$CONDA_PREFIX/lib" \ + --with-zlib="$CONDA_PREFIX" \ + ${{ matrix.configure_debug }} + grep -Eq '^CXX = .* -std=c\+\+17([[:space:]]|$)' Makefile + grep -q '^#define INTARNA_MULITHREADING 1' src/config.h + + - name: Build, install, and run full test suite + run: | + make -j 2 + make install + make tests -j 2 + "$RUNNER_TEMP/intarna-install/bin/IntaRNA" -h + + - name: Compile every installed public header independently + run: | + header_probe="$RUNNER_TEMP/intarna-header-probe.cpp" + for header in "$RUNNER_TEMP/intarna-install/include/IntaRNA/"*.h; do + printf '#include \nint main() { return 0; }\n' "$(basename "$header")" > "$header_probe" + "$CXX" -std=c++17 ${CPPFLAGS:-} ${CXXFLAGS:-} \ + -I"$RUNNER_TEMP/intarna-install/include" \ + -I"$CONDA_PREFIX/include" \ + -fsyntax-only "$header_probe" + done + + - name: Compile and link an installed pkg-config consumer + run: | + consumer="$RUNNER_TEMP/intarna-consumer.cpp" + printf '#include \nint main() { IntaRNA::RnaSequence rna("test", "ACGU"); return rna.size() == 4 ? 0 : 1; }\n' > "$consumer" + export PKG_CONFIG_PATH="$RUNNER_TEMP/intarna-install/lib/pkgconfig" + "$CXX" -std=c++17 ${CPPFLAGS:-} ${CXXFLAGS:-} "$consumer" \ + ${LDFLAGS:-} $(pkg-config --cflags --libs IntaRNA) \ + -o "$RUNNER_TEMP/intarna-consumer" + "$RUNNER_TEMP/intarna-consumer" diff --git a/doc/local-validation-archive/files/issue-244-validation/source-gcc11-debug/README.md b/doc/local-validation-archive/files/issue-244-validation/source-gcc11-debug/README.md new file mode 100644 index 00000000..050edfc8 --- /dev/null +++ b/doc/local-validation-archive/files/issue-244-validation/source-gcc11-debug/README.md @@ -0,0 +1,2239 @@ + +# IntaRNA [![releases](https://img.shields.io/github/tag/BackofenLab/IntaRNA.svg)](https://github.com/BackofenLab/IntaRNA/releases) [![Bioconda](https://anaconda.org/bioconda/intarna/badges/version.svg)](https://anaconda.org/bioconda/intarna) + +**Efficient RNA-RNA interaction prediction incorporating accessibility and +seeding of interaction sites** + +During the last few years, several new small regulatory RNAs +(sRNAs) have been discovered in bacteria. Most of them act as post-transcriptional +regulators by base pairing to a target mRNA, causing translational repressionex +or activation, or mRNA degradation. Numerous sRNAs have already been identified, +but the number of experimentally verified targets is considerably lower. +Consequently, computational target prediction is in great demand. Many existing +target prediction programs neglect the accessibility of target sites and the +existence of a seed, while other approaches are either specialized to certain +types of RNAs or too slow for genome-wide searches. + +IntaRNA, developed by +[Prof. Backofen's bioinformatics group at Freiburg University](http://www.bioinf.uni-freiburg.de), +is a general and fast approach to the +prediction of RNA-RNA interactions incorporating both the accessibility of +interacting sites +as well as the existence of a user-definable seed interaction. We successfully applied +IntaRNA to the prediction of bacterial sRNA targets and determined the exact +locations of the interactions with a higher accuracy than competing programs. + +For testing or ad hoc use of IntaRNA, you can use its webinterface at the + +**==> [Freiburg RNA tools IntaRNA webserver](http://rna.informatik.uni-freiburg.de/IntaRNA/) <==** + + +## Contribution + +Feel free to contribute to this project by writing +[Issues](https://github.com/BackofenLab/IntaRNA/issues) +with feature requests, bug reports, or just contact messages. + +## Citation + +If you use IntaRNA, please cite our respective articles + +#### Method + +- [IntaRNA 2.0: enhanced and customizable prediction of RNA-RNA interactions](https://doi.org/10.1093/nar/gkx279) + Martin Mann, Patrick R. Wright, and Rolf Backofen, + Nucleic Acids Research, 45 (W1), W435–W439, 2017, DOI:[10.1093/nar/gkx279](https://doi.org/10.1093/nar/gkx279). +- [IntaRNA: efficient prediction of bacterial sRNA targets incorporating target site accessibility and seed regions](https://doi.org/10.1093/bioinformatics/btn544) + Anke Busch, Andreas S. Richter, and Rolf Backofen, + Bioinformatics, 24 no. 24 pp. 2849-56, 2008, DOI:[10.1093/bioinformatics/btn544](https://doi.org/10.1093/bioinformatics/btn544). + +#### Features and Application + +- [Integration of accessibility data from structure probing into RNA–RNA interaction prediction](https://doi.org/10.1093/bioinformatics/bty1029) + Milad Miladi, Soheila Montaseri, Rolf Backofen, Martin Raden, + Bioinformatics, 2019, DOI:[10.1093/bioinformatics/bty1029](https://doi.org/10.1093/bioinformatics/bty1029). +- [IntaRNAhelix - Composing RNA-RNA interactions from stable inter-molecular helices boosts bacterial sRNA target prediction](https://doi.org/10.1142/S0219720019400092) + Rick Gelhausen, Sebastian Will, Ivo L. Hofacker, Rolf Backofen, and Martin Raden, + Journal of Bioinformatics and Computational Biology, 2019, 17(5), 1940009, DOI:[10.1142/S0219720019400092](https://doi.org/10.1142/S0219720019400092). +- [CopraRNA and IntaRNA: predicting small RNA targets, networks and interaction domains](https://doi.org/10.1093/nar/gku359) + Patrick R. Wright, Jens Georg, Martin Mann, Dragos A. Sorescu, Andreas S. Richter, Steffen Lott, Robert Kleinkauf, Wolfgang R. Hess, and Rolf Backofen, + Nucleic Acids Research, 42 (W1), W119-W123, 2014, DOI:[10.1093/nar/gku359](https://doi.org/10.1093/nar/gku359). +- [The impact of various seed, accessibility and interaction constraints on sRNA target prediction- a systematic assessment](https://doi.org/10.1186/s12859-019-3143-4). + Martin Raden, Teresa Müller, Stefan Mautner, Rick Gelhausen, and Rolf Backofen, + BMC Bioinformatics, 21 (15), 2020, DOI:[10.1186/s12859-019-3143-4](https://doi.org/10.1186/s12859-019-3143-4). + + +



+ + +# Documentation + +## Overview + +The following topics are covered by this documentation: + +- [Installation](#install) + - [IntaRNA via conda](#instconda) + - [IntaRNA docker container](#instdocker) + - [Dependencies](#deps) + - [Cloning from github](#instgithub) + - [Source code distribution](#instsource) + - [Microsoft Windows installation](#instwin) + - [OS X installation with homebrew](#instosx) +- [Usage and Parameters](#usage) +- [Just run ...](#defaultRun) + - [Multi-threading and parallelized computation](#multithreading) + - [Load arguments from file](#parameterFile) +- [General things you should know](#generalInformation) + - [Interaction Model](#interactionModel) + - [Single-site, loop-based RNA-RNA interaction](#interactionModel-ssUnconstraintMfe) + - [Single-site, ensemble-based RNA-RNA interaction](#interactionModel-ssProbability) + - [Single-site, helix-based RNA-RNA interaction](#interactionModel-ssHelixBlockMfe) + - [Prediction modes](#predModes) + - [Emulating other RNA-RNA interaction prediction tools](#predEmulateTools) + - [Limiting memory consumption - window-based prediction](#predWindowBased) + - [IntaRNA's multiple personalities](#personality) + - [IntaRNA - fast, heuristic RNA-RNA interaction prediction](#IntaRNA) + - [IntaRNAhelix - helix-based predictions](#IntaRNAhelix) + - [IntaRNAexact - exact predictions like RNAup](#IntaRNAexact) + - [IntaRNAduplex - hybrid-only optimization like RNAduplex](#IntaRNAduplex) + - [IntaRNAsTar - optimized for sRNA-target prediction](#IntaRNAsTar) + - [IntaRNAseed - identifys and reports seed interactions only](#IntaRNAseed) + - [IntaRNAens - ensemble-based prediction and partition function computation](#IntaRNAens) +- [How to constrain predicted interactions](#constraintSetup) + - [Interaction restrictions](#interConstr) + - [Seed constraints](#seed) + - [Explicit seed input](#seedExplicit) + - [Helix constraints](#helix) + - [SHAPE reactivity data to enhance accessibility computation](#shape) +- [Output Setup](#outputSetup) + - [Output modes](#outmodes) + - [Pairwise vs. all-vs-all](#outpairwise) + - [Sequence indexing](#idxPos0) + - [Suboptimal RNA-RNA interaction prediction and output restrictions](#subopts) + - [Energy parameters and temperature](#energy) + - [Additional output files](#outFiles) + - [Minimal energy profiles](#profileMinE) + - [Minimal energy for all intermolecular index pairs](#pairMinE) + - [Spot probability profiles](#profileSpotProb) using partition functions + - [Interaction probabilities for interaction spots of interest](#spotProb) + - [Accessibility and unpaired probabilities](#accessibility) + - [Local versus global unpaired probabilities](#accLocalGlobal) + - [Constrain regions to be accessible or blocked](#accConstraints) + - [Scaling factors for partition function computation for accessibility estimation](#accPfScale) + - [Read/write accessibility from/to file or stream](#accFromFile) +- [Library for integration in external tools](#lib) +- [Auxiliary R scripts for output visualization etc.](/R) +- [Auxiliary python scripts for IntaRNA-based pipelines](/python) + + +



+ + +# Installation + +

+ + +## IntaRNA via conda (bioconda channel) + +The most easy way to locally install IntaRNA is via conda using the +[bioconda](https://bioconda.github.io/) +channel (linux only). This way, you will install a pre-built IntaRNA binary along +with all dependencies. +Follow +[![install with bioconda](https://img.shields.io/badge/install%20with-bioconda-brightgreen.svg?style=flat-square)](http://bioconda.github.io/recipes/intarna/README.html) +to get detailed information or run the following command to install it to the currently activated environment. +```bash +conda install -c conda-forge -c bioconda intarna +``` +**Note**: Conda is available for Windows, MacOs and Linux. We recommend the [Miniconda installation](https://docs.anaconda.com/miniconda/install/#quick-command-line-install) + +**Note further**: *Windows user without Powershell or Commandline experience* might [consider to first install a Linux App via WSL](https://documentation.ubuntu.com/wsl/en/latest/howto/install-ubuntu-wsl2/) and use and install conda therein since most related scripts will be tailored for Linux BASH shells or similar systems. + +[![up](doc/figures/icon-up.28.png) back to overview](#overview) +

+ + +## IntaRNA docker container (via QUAY.IO) + +An [IntaRNA docker container](https://quay.io/repository/biocontainers/intarna) +([?](https://www.docker.com/)) is provided from the bioconda package via +[Quay.io](https://quay.io/). This gives +you with an encapsulated IntaRNA installation. + +**Note** The `biocontainer` builds do not support the `latest` tag, such that you have to specify a version to download! So best + +- open the [quay.io IntaRNA tag page](https://quay.io/repository/biocontainers/intarna?tab=tags) +- identify the tag of the container version you want to install, e.g. `3.4.1--pl5321h077b44d_3` +- use this tag in your container pull command as an appendix, e.g. + - `podman pull quay.io/biocontainers/intarna:3.4.1--pl5321h077b44d_3` or + - `docker pull quay.io/biocontainers/intarna:3.4.1--pl5321h077b44d_3` + + +[![up](doc/figures/icon-up.28.png) back to overview](#overview) +

+ + +## Dependencies + +If you are going to compile IntaRNA from source, ensure you meet the following +dependencies: + +- compiler and standard library supporting C++17, with OpenMP for multithreading +- [boost C++ library](http://www.boost.org/) version >= 1.50.0 + (ensure the following libraries are installed for development (not just runtime libraries!); or install all e.g. in Ubuntu via package `libboost-all-dev`) + - libboost_regex + - libboost_program_options + - libboost_filesystem + - libboost_system +- [Vienna RNA package](http://www.tbi.univie.ac.at/RNA/) version >= 2.4.14 +- `pkg-config` for detailed version checks of dependencies +- if [cloning from github](#instgithub): GNU autotools (automake, autoconf, ..) + +Also used by IntaRNA, but already part of the source code distribution (and thus +not needed to be installed separately): + +- [Catch](https://github.com/philsquared/Catch) test framework +- [Easylogging++](https://github.com/easylogging/easyloggingpp) logging framework + + + +[![up](doc/figures/icon-up.28.png) back to overview](#overview) +

+ + +## Cloning *Source code* from github (or downloading ZIP-file) + +The data provided within the github repository +(or within the `Source code` archives provided at the +[IntaRNA release page](https://github.com/BackofenLab/IntaRNA/releases/latest)) +is no complete distribution and +lacks all system specifically generated files. Thus, in order to get started with +a fresh clone of the IntaRNA source code repository you have to run the GNU autotools +to generate all needed files for a proper `configure` and `make`. To this end, +we provide the helper script `autotools-init.sh` that can be run as shown in the following. +```bash +# call aclocal, automake, autoconf +bash ./autotools-init.sh +``` +Afterwards, you can continue as if you would have downloaded an +[IntaRNA package distribution](#instsource). + + +[![up](doc/figures/icon-up.28.png) back to overview](#overview) +

+ + +## IntaRNA package distribution (e.g. `intaRNA-2.0.0.tar.gz`) + +When downloading an IntaRNA package distribution (e.g. `intaRNA-2.0.0.tar.gz`) from the +[IntaRNA release page](https://github.com/BackofenLab/IntaRNA/releases/latest), you should +first ensure, that you have all [dependencies](#deps) installed. If so, you can +simply run the following (assuming `bash` shell). +```bash +# generate system specific files (use -h for options) +./configure +# compile IntaRNA from source +make +# run tests to ensure all went fine +make tests +# install (use 'configure --prefix=XXX' to change default install directory) +make install +# (alternatively) install to directory XYZ +make install prefix=XYZ +``` + +If you installed one of the dependencies in a non-standard directory, you have +to use the according `configure` options: +- `--with-vrna` : the prefix where the Vienna RNA package is installed +- `--with-boost` : the prefix where the boost library is installed + +Note, the latter is for instance the case if your `configure` call returns an +error message as follows: +```[bash] +checking whether the Boost::System library is available... yes +configure: error: Could not find a version of the library! +``` +In that case your boost libraries are most likely installed to a non-standard +directory that you have to specify either using `--with-boost` or just the +library directory via `--with-boost-libdir`. + +[![up](doc/figures/icon-up.28.png) back to overview](#overview) +

+ + +## Microsoft Windows installation + +### ... from source + +IntaRNA can be compiled, installed, and used on a Microsoft Windows system when +e.g. using [Cygwin](https://www.cygwin.com/) as 'linux emulator'. Just install +Cygwin with the following packages: + +- *Devel*: + - make + - gcc-g++ + - autoconf + - automake + - pkg-config +- *Libs*: + - libboost-devel +- *Perl*: + - perl + +and follow either [install from github](#instgithub) or +[install from package](#instsource). + +*Note*, the source code comes without any waranties or what-so-ever +(see licence information)! + +### ... using pre-compiled binaries + +For some releases, we also provide precompiled binary packages for Microsoft Windows at the +[IntaRNA release page](https://github.com/BackofenLab/IntaRNA/releases/latest) +that enable 'out-of-the-box' usage. If you +want to use them: +- [download](https://github.com/BackofenLab/IntaRNA/releases/latest) the according ZIP archive and extract +- open a [Windows command prompt](https://www.lifewire.com/how-to-open-command-prompt-2618089) +- [run IntaRNA](#usage) + +*Note*, these binaries come without any waranties, support or what-so-ever! +They are just an offer due to user requests. + +If you do not want to work within the IntaRNA directory or don't want to provide +the full installation path with every IntaRNA call, you should add the installation +directory to your [`Path` System variable](http://www.computerhope.com/issues/ch000549.htm) +(using a semicolon `;` separator). + + + +[![up](doc/figures/icon-up.28.png) back to overview](#overview) +

+ + +## macOS installation with Homebrew + +The [Bioconda package](https://anaconda.org/bioconda/intarna) is the simplest +installation route. To build locally with the system Apple Clang compiler, +install the dependencies and the OpenMP runtime: + +```bash +brew install autoconf automake boost libtool libomp pkg-config viennarna +``` + +Apple Clang needs the Homebrew `libomp` headers, library and preprocessing flag. +The following keeps the normal optimized, multithreaded build: + +```bash +LIBOMP_PREFIX="$(brew --prefix libomp)" +CPPFLAGS="-I${LIBOMP_PREFIX}/include" \ +CXXFLAGS="-Xpreprocessor -fopenmp" \ +LDFLAGS="-L${LIBOMP_PREFIX}/lib -Wl,-rpath,${LIBOMP_PREFIX}/lib" \ +LIBS="-lomp" \ +./configure CC=clang CXX=clang++ +make -j2 +make tests -j2 +make install +``` + +Use `--prefix=INSTALLPATH` with `./configure` when a system-wide installation +is not desired. + + +[![up](doc/figures/icon-up.28.png) back to overview](#overview) + + +



+ + +# Usage and parameters + +IntaRNA comes with a vast variety of ways to tune or enhance *YOUR* RNA-RNA prediction. +To this end, different [prediction modes](#predModes) and +[interaction models](#interactionModel) +are implemented that allow +to balance predication quality and runtime requirement. Furthermore, it is +possible to define +[interaction restrictions](#interConstr), +[seed constraints](#seed), +[explicit seed information](#seedExplicit), +[SHAPE reactivity constraints](#shape), +[output modes](#outmodes), +[suboptimal enumeration](#subopts), +[energy parameters, temperature](#energy), +and the [accessibility](#accessibility) handling. If you are doing high-throughput +computations, you might also want to consider [multi-threading support](#multithreading). + +For ad hoc usage you can use the +[Freiburg RNA tools IntaRNA webserver](http://rna.informatik.uni-freiburg.de/IntaRNA/) +(with limited parameterization). + + + +[![up](doc/figures/icon-up.28.png) back to overview](#overview) +

+ + +# Just run ... + +If you just want to start and are fine with the default parameters set, +you only have to provide two RNA sequences, +a (long) target RNA (using `-t` or `--target`) and a (short) query RNA +(via `-q` or `--query`), in +[IUPAC RNA encoding](#https://en.wikipedia.org/wiki/Nucleic_acid_notation). +You can either directly input the sequences +```bash +# running IntaRNA with direct sequence input +# call : IntaRNA -t CCCCCCCCGGGGGGGGGGGGGG -q AAAACCCCCCCUUUU + +target + 9 15 + | | + 5'-CCCCCCCC GGGGGGG-3' + GGGGGGG + +++++++ + CCCCCCC + 3'-UUUU AAAA-5' + | | + 11 5 +query + +interaction energy = -10.85 kcal/mol + +``` + +In case you need specific RNA names in your output, you can provide ID strings +for each RNA using e.g. `--tId="mRNA with GC"` or `--qId="sRNA-example"`. + +Multiple sequences can be provided in [FASTA-format](#https://en.wikipedia.org/wiki/FASTA_format). +It is possible to use either file input or to read the FASTA input from the +`STDIN` stream. + +```bash +# running IntaRNA with FASTA files +IntaRNA -t myTargets.fasta -q myQueries.fasta +# reading query FASTA input from stream via pipe +cat myQueries.fasta | IntaRNA -q STDIN -t myTargets.fasta +``` + +If you are working with large FASTA input files, e.g. covering a whole +transcriptome, you can restrict the prediction to a subset of the input +sequences using the `--qSet` or `--tSet` parameter as shown in the following. + +```bash +# restrict prediction to the second load of 100 target sequences +IntaRNA -t myTranscriptome.fasta --tSet=101-200 -q myQuery.fasta +``` + +Furthermore, also gzip-compressed file input is supported and automatically +decompressed if the file name ends in `.gz`. + +Nucleotide encodings different from `ACGUT` are rewritten as `N` and the respective +positions are not considered to form base pairs (and thus ignored). +Thymine `T` encodings are replaced by uracil `U`, since we apply an RNA-only +energy model. + +For a list of general program argument run `-h` or `--help`. For a complete +list covering also more sophisticated options, run `--fullhelp`. + + +[![up](doc/figures/icon-up.28.png) back to overview](#overview) + +

+ + +## Multi-threading and parallelized computation + +IntaRNA supports the parallelization of the target-query-combination processing. +The maximal number of threads to be used can be specified using the `--threads` parameter. +If `--threads=k != 1`, than *k* predictions are processed in parallel. A value of +`0` requests the maximally available number of threads for this machine. + +When using parallelization, you should have the following in mind: + +- The memory consumption will be multiplied by the number of threads, + since each thread runs an independent + prediction (with according memory consumption). Thus, ensure you have enough + RAM available when using many threads of memory-demanding + [prediction modes](#predModes). You might consider [window-based prediction](#predWindowBased) + to limit the required RAM. + +- Parallelization is enabled hierarchically, ie. only one of the following input + sets is processed in parallel: + - target sequences (if more than one) + - if only one target: query sequences (if more than one) + - if only one target and query: [window combinations](#predWindowBased) (if enabled) + +The support for multi-threading can be completely disabled before compilation +using `configure --disable-multithreading`. + + + + + +[![up](doc/figures/icon-up.28.png) back to overview](#overview) +
+ + +## Load arguments from file + +If you are using IntaRNA with similar command line arguments (parameters), you +might want to reduce the call via the definition of a *parameter file*. To this +end prepare a file that defines the common parameters in a the following form + +```[python] +################################################################# +# my parameter.cfg file to simulate RNAup predictions +################################################################# +# slow but exact predictions +mode = M +# no seed constraint +noSeed = true +# full global accessibility computation +accW = 0 +accL = 0 +``` + +where you use the long parameter names without the leading `--`. +Boolean arguments (like `--noSeed`) have to be set to `true`|`1` or `false`|`0` +to be enabled or disabled, respectively. Given this, +you only have to pass the file name via `--parameterFile=...` and IntaRNA will +parse the additional parameters from your file. + +*Note:* parameters specified via the command line take precedence over arguments +from a parameter file. Thus, you can (silently) overwrite parameters that +you have specified within the file. + +*Note further:* parameter parsing from parameter file is (in contrast to the +command line parsing) **case sensitive**! + + + + +[![up](doc/figures/icon-up.28.png) back to overview](#overview) +

+ + +# General things you should know + + + +

+ + +## RNA-RNA interaction models + +IntaRNA supports various models how RNA-RNA interactions are represented. +The model selection has direct consequences for the interaction patterns that +can be predicted by IntaRNA. +Before elaborating the supported models, first terms needed for understanding +and representation: + +We denote with a **single site** an interaction pattern of two respective RNA +subsequences Qi..Qk and Tj..Tl that + +- form a *base pair on each end*, i.e. (Qi,Tl) and (Qk,Tj) are pairing, and +- there are *no intra-molecular base pairs* within the two subsequences, i.e. +the subsequences form only inter-molecular base pairs. + +Given that we can classify single-site RNA-RNA interactions based on the +structural context of the respective subsequences, which are + +- *exterior* - not enclosed by any base pair +- *hairpin loop* - directly enclosed by a base pair +- *non-hairpin loop* - subsequence enclosed by two loops forming a bulge, interior or multi-loop + +The following figure shows an RNA structure depiction with context annotations +(abbreviated by resp. first letter) of unpaired regions that can form RNA-RNA interactions. + +![depiction of RNA-RNA interaction pattern](doc/figures/interaction-context.svg) + +IntaRNA can predict single-site interactions within any structural context of the respective subsequences. + +| context | exterior | hairpin loop | non-hairpin loop | +| :-------------: | :------: | :----------: | :--------------: | +| **exterior** | ![yes](doc/figures/icon-yes.39.png) | ![yes](doc/figures/icon-yes.39.png) | ![yes](doc/figures/icon-yes.39.png) | +| **hairpin** | ![yes](doc/figures/icon-yes.39.png) | ![yes](doc/figures/icon-yes.39.png) | ![yes](doc/figures/icon-yes.39.png) | +| **non-hairpin loop** | ![yes](doc/figures/icon-yes.39.png) | ![yes](doc/figures/icon-yes.39.png) | ![yes](doc/figures/icon-yes.39.png) | + +Note, *concatenation-based approaches* as implemented in UNAfold, NUPACK or RNAcofold can +*only predict exterior-exterior context* combinations (shown by (b) in the figure above) +and are thus not capable +to investigate e.g. common loop-exterior or kissing-hairpin-loop +interaction patterns that are depicted by (c) and (d) in the figure from above, respectively! + +A detailed discussion about different prediction approaches and predictable +interaction pattern is available in our publications + +- [Interactive implementations of thermodynamics-based RNA structure and RNA-RNA interaction prediction approaches for example-driven teaching.](https://doi.org/10.1371/journal.pcbi.1006341) and +- [Structure and interaction prediction in prokaryotic RNA biology.](https://doi.org/10.1128/microbiolspec.RWR-0001-2017) + + + +[![up](doc/figures/icon-up.28.png) back to overview](#overview) +

+ + +### Single-site, loop-based RNA-RNA interaction with minimal free energy + +This *default model* of IntaRNA (`--model=X`) predicts the single-site interaction `I` with +minimal free energy. That is, it minimizes +``` + arg min ( E_hybrid(I) + ED1(I) + ED2(I) ) + I +``` +where `E_hybrid` represents all energy terms of intermolecular base pairs and +`ED` corresponds to the energy needed to make the respective subsequences +accessible for inter-molecular base pairing, i.e. removing any possible intra-molecular +base pairs. + +The model considers inter-molecular base pair patterns (so called *loops*) that correspond to +(helical) stackings, bulges or interior loops, which are depicted in figure (b) from above. +Since intra-molecular base pairs are not explicitely represented, any structural +context of single-site interactions is considered/possible within IntaRNA +predictions. + +This model is used e.g. by the [IntaRNA](#IntaRNA) and [IntaRNAexact](#IntaRNAexact) +personalities. + +Note, the underlying computational model implements a seed-extension strategy +(`--model=X`, version 3.0+), which first identifies putative seeds that are +subsequently extended to identify the optimal interactions. +Older versions of IntaRNA (version 1.* and 2.*) used a slower optimization +strategy (accessible via `--model=S`). + + + +[![up](doc/figures/icon-up.28.png) back to overview](#overview) +

+ + + +### Single-site, ensemble-based RNA-RNA interaction with minimal free ensemble energy + +IntaRNA supports using `--model=P` an ensemble based interaction prediction that +is based on partition function computation. To this end, the model computes among all +possible interaction sites `S`, defined by two interacting subsequences while the ends of the subsequences +form a base pair, the one with maximal partition function `Z(S)`, i.e. +``` + arg max Z(S) = ( exp(-ED1(S)/RT) * exp(-ED2(S)/RT) * sum ( exp(-E_hybrid(I)/RT) ) ) + S I of S +``` +which sums the Boltzmann weights of all interactions `I` (hybridization terms only) +for the given site `S` multiplied with the Boltzmann weights of the respective +accessibility penalties `ED1` and `ED2`. +Here, `R` denotes the gas constant and `T` the temperature of the system. +At `T=37` degree Celsius, the product `RT` is about `RT=0.6163173043012`. +The site's partition function can be +used to compute the ensemble energy of all interactions of a given site via +`E(S) = -RT log(Z(S))` (`log` = natural logarithm), which is reported for the optimal site +(see [CSV output](#outModeCsv)). +This abstracts from individual inter-molecular base pairing and incorporates the +dynamics and flexibility of the interactions formed by two regions. + +The overall partition function is given by +`Zall = sum Z(S)`, i.e. summation over all possible sites, +and enables the computation of the probabilities `P_E(I) = exp(-E(I)/RT)/Zall` +that a certain interaction `I` is formed. +Both can be accessed via [CSV output](#outModeCsv). + +Note, all computations are *only based on the considered interactions* that are +conform to the currently applied constraints (eg. on seed, accessibility, +interaction width, etc.) and do *not* take *all possible interactions* into account! +Thus, if you are predicting interactions for a subregion only, the results are +based on the respective subset of interactions! + +This model is used by the [IntaRNAens](#IntaRNAens) personality. + + + +[![up](doc/figures/icon-up.28.png) back to overview](#overview) +

+ + +### Helix-based single-site RNA-RNA interaction with minimal free energy + +The formation of multiple base pair stackings, i.e. helix formation, requires +a 'winding' of the respective subsequences. +Depending on the structural context, such winding might be sterically and kinetically +hindered by the necessary unwinding of intra-molecular structural elements. + +The helix-based prediction model aims to incorporate such +effects into the predictions of IntaRNA. +This is done by restricting the maximum length of inter-molecular helices to a +specified number of (stacked) base pairs. +That way, 'wound up' subhelices are interspaced by flexible interior loops that +will allow for a more flexible 3D arrangement of the overall helix. + +The following figure depicts the effect of the maximum helix length constraints +that only allows for helices up to a specified length. +That way, long interactions (left) are avoided and replaced by a more flexible +model composed of short inter-molecular helices (right). +The blue boxes represent the length-bound helices and the red boxes depict +the interspacing unpaired regions (interior loops). + +![helixbased](./doc/figures/helixbased.svg) + +Note, IntaRNA's helix-block-based model `--model=B` implements a heuristic, which +only considers for each intermolecular base pair the most stable helix (block) extending +to the right. This results in reduced runtimes while keeping or even improving +the prediction quality in genome wide screens. IntaRNA offers various +[helix constraints](#helix) to guide which helices are considered for interaction +prediction. + +This model is used by the [IntaRNAhelix](#IntaRNAhelix) personality. + +For further details on the model and the underlying algorithm, please refer to our respective publication + +- [IntaRNAhelix - Composing RNA-RNA interactions from stable inter-molecular helices boosts bacterial sRNA target prediction](https://doi.org/10.1142/S0219720019400092) + + + + +[![up](doc/figures/icon-up.28.png) back to overview](#overview) +

+ + +## Prediction modes + +For the prediction of *minimum free energy interactions*, the following modes +and according features are supported and can be set via the `--mode` parameter. +The time and space complexities are given for the prediction of two sequences +of equal length *n*. + +| Features | Heuristic `--mode=H` | Exact `--mode=M` | Seed-only `--mode=S` | +| -------- | :------------------: | :-----------------: | :--------------: | +| Time complexity (prediction only) | O(*n*^2) | O(*n*^4) | O(*n*^2) | +| Space complexity | O(*n*^2) | O(*n*^2) | O(*n*^2) | +| [Seed constraint](#seed) | ![yes](doc/figures/icon-yes.39.png) | ![yes](doc/figures/icon-yes.39.png) | ![yes](doc/figures/icon-yes.39.png) | +| [Explicit seeds](#seedExplicit) | ![yes](doc/figures/icon-yes.39.png) | ![yes](doc/figures/icon-yes.39.png) | ![yes](doc/figures/icon-yes.39.png) | +| [SHAPE reactivity constraint](#shape) | ![yes](doc/figures/icon-yes.39.png) | ![yes](doc/figures/icon-yes.39.png) | ![yes](doc/figures/icon-yes.39.png) | +| No [seed constraint](#seed) | ![yes](doc/figures/icon-yes.39.png) | ![yes](doc/figures/icon-yes.39.png) | ![no](doc/figures/icon-no.39.png) | +| Minimum free energy interaction | not guaranteed | ![yes](doc/figures/icon-yes.39.png) | ![yes](doc/figures/icon-yes.39.png) | +| Overlapping [suboptimal interactions](#subopts) | ![yes](doc/figures/icon-yes.39.png) | ![yes](doc/figures/icon-yes.39.png) | ![yes](doc/figures/icon-yes.39.png) | +| Non-overlapping [suboptimal interactions](#subopts) | ![yes](doc/figures/icon-yes.39.png) | ![no](doc/figures/icon-no.39.png) | ![yes](doc/figures/icon-yes.39.png) | + +Note, due to the low run-time requirement of the heuristic prediction mode +(`--mode=H`), heuristic IntaRNA interaction predictions are widely used to screen +for interaction in a genome-wide scale. If you are more interested in specific +details of an interaction site or of two relatively short RNA molecules, you +should investigate the exact prediction mode (`--mode=M`) providing the global +minimum free energy interaction. +Putative seed interactions (used by the `H` and `M` mode) can be enumerated +and studied using the `S` mode. + + +[![up](doc/figures/icon-up.28.png) back to overview](#overview) +

+ + +### Emulating other RNA-RNA interaction prediction tools + +Given these features, we can emulate and extend a couple of RNA-RNA interaction +tools using IntaRNA. + +**TargetScan**, **RNAhybrid** and **RNAduplex** are approaches that predict the interaction hybrid with +minimal interaction energy without consideration whether or not the interacting +subsequences are probably involved involved in intramolecular base pairings. Furthermore, +no seed constraint is taken into account. +This prediction result can be emulated (depending on the used prediction mode) +by running IntaRNA when disabling both the seed constraint +as well as the accessibility integration using +```bash +# prediction results similar to TargetScan/RNAhybrid +IntaRNA [..] --noSeed --acc=N +``` +We *add seed-constraint support to TargetScan/RNAhybrid-like computations* by removing the +`--noSeed` flag from the above call. +See [IntaRNAexact personality](#IntaRNAexact) for additional information. + + +**RNAup** was one of the first RNA-RNA interaction prediction approaches that took the +accessibility of the interacting subsequences into account while not considering the seed feature. +IntaRNA's exact prediction mode is eventually an alternative implementation when disabling +seed constraint incorporation. Furthermore, the unpaired probabilities used by RNAup to score +the accessibility of subregions are covering the respective overall structural ensemble for each +interacting RNA, such that we have to disable accessibility computation based on local folding (RNAplfold). +Finally, RNAup only predicts interactions for subsequences of length 25. +All this can be setup using +```bash +# prediction results similar to 'RNAup -b' (incorporating accessibility of both RNAs) +IntaRNA --mode=M --noSeed --accW=0 --accL=0 --intLenMax=25 +``` +We *add seed-constraint support to RNAup-like computations* by removing the +`--noSeed` flag from the above call. +See [IntaRNAexact personality](#IntaRNAexact) for additional information. + + + +[![up](doc/figures/icon-up.28.png) back to overview](#overview) +

+ + +### Limiting memory consumption - window-based prediction + +The memory requirement of IntaRNA grows quadratically with lengths of the input +sequences. Thus, for very long input RNAs, the requested memory can exceed the +available RAM of smaller computers. + +This can be circumvented by using a window-based prediction where the input +sequences are decomposed in overlapping subsequences (windows) that are +processed individually. That way, the maximal memory consumption is defined by +the (shorter) window length rather the length of the input sequence, resulting +in a user guided memory/RAM consumption. + +The window-based computation is enabled by setting the following parameters + +- `--windowWidth` : length of the windows/subsequences (value of 0 disables window-based computations) +- `--windowOverlap` : overlap of the windows, which has to be larger than the maximal interaction length (see [`--q|tIntLenMax`](#interConstr)) + +Note, window-based computation produces a computational overhead due to +redundant consideration of the overlapping subsequences. Thus, the runtime is +increased proportionally to the ratio of window overlap and length. + +If only one query and target are given, window-based computation can be +[parallelized](#multithreading), which typically remedies the computational overhead. + + + +[![up](doc/figures/icon-up.28.png) back to overview](#overview) +

+ + +## IntaRNA's multiple personalities + +IntaRNA comes as a powerful tool for RNA-RNA interaction prediction supporting +various [interaction models](#interactionModel), [prediction modi](#predModes) +and constraints. Thus, it is hard for a non-expert user to find and setup the +best parameter set for the task at hand. + +To provide a better guide how to use IntaRNA, we provide different preset parameter +combinations, where each defines one of IntaRNA's multiple *personalities*. +In the following, we list available personalities and respective use cases. + +Each personality can be enabled by either replacing `IntaRNA` with the respective +personality when calling or by setting the input parameter `--personality=...`. + +Thus, depending on how you call IntaRNA, it will react differently.. :smile: + + +[![up](doc/figures/icon-up.28.png) back to overview](#overview) + +### IntaRNA + +The standard personality of IntaRNA implements the prediction approach of +[(Busch et al., 2008)](https://doi.org/10.1093/bioinformatics/btn544) +with the extension from +[(Mann et al., 2017)](https://doi.org/10.1093/nar/gkx279). Thus it uses a fast +heuristic [prediction mode](#predModes) in combination with local +[accessibility](#accLocalGlobal) estimates. It predicts minimum free energy +RNA-RNA interactions for single-site unconstraint +[interaction models](#interactionModel-ssUnconstraintMfe). + +Thus, IntaRNA provides (typically) optimal predictions on a single inter-molecular +base pair level. Output is per default restricted to the interaction site and +base pairs, since intra-molecular structure is implicitely considered via the +accessibility penalties represented by the ED terms of the energy calculation +(see detailed output). + +#### IntaRNA1 and IntaRNA2 + +For backward compatibility, we also provide the personality *IntaRNA1* and *IntaRNA2*, +which emulate the settings of the respective version with the currently +available parameters. +Both personalities make use of the slower `--model=S` +[prediction strategy](#interactionModel-ssUnconstraintMfe) first +introduced for IntaRNA version 1.0. + +Note, IntaRNA v1.* used the old `Turner99` energy parameters, such that the +`--energyVRNA=Turner99` [energy set](#energy) is preset, when using *IntaRNA1*. + + +[![up](doc/figures/icon-up.28.png) back to overview](#overview) + +### IntaRNAexact + +**RNAup** was the first accessibility-based RNA-RNA interaction prediction approach +and it implements methods to identify the optimal minimum free energy interaction +for unconstraint, single-site [interaction models](#interactionModel-ssUnconstraintMfe). +IntaRNA also implements analogous methods when exact [prediction modes](#predModes) +are used, and thus can provide [RNAup-like predictions](#predEmulateTools). + +**IntaRNAexact** therefore + +- uses exact [prediction mode](#predModes), +- restricts the maximal [interaction length](#interConstr) to 60, +- uses global [accessibility](#accLocalGlobal) values (reenable local computation respectively if needed, e.g. for mRNAs or genomic subsequences) and +- reports suboptimal interactions that can [overlap in both RNAs](#subopts). + +In contrast to RNAup, it also + +- enforces [seed constraints](#seed), +- uses a much faster [seed-extension-based computation model](#predModel), +- allows longer interaction length, and +- enables much more flexible output options. + +If needed, you can disables these features. + +Thus, *IntaRNAexact* produces RNAup-like predictions with the extensive output +and constraint options of IntaRNA. +Since exact computations are computationally much more demanding, IntaRNAexact is +slower than IntaRNA. +Therefore, you should use *IntaRNAexact* if you want to investigate the details of a +single or few interactions unbiased by the heuristics applied in normal IntaRNA. +If you are using long RNAs, you should constraint predictions to the [regions of +interest](#interConstr) or [constraint the seed regions](#seed). + +In summary, the following calls are equivalent. + +```sh +IntaRNAup ... +IntaRNA --personality=IntaRNAup ... +IntaRNA --model=X --mode=M --accW=0 --accL=0 --intLenMax=60 --outOverlap=B ... +``` + +[![up](doc/figures/icon-up.28.png) back to overview](#overview) + +### IntaRNAhelix + +**IntaRNAhelix** provides helix-based RNA-RNA interaction prediction described +in [(Gelhausen et al., 2019)](https://doi.org/10.1142/S0219720019400092). +It therefore enables per default + +- the [helix-based single-site interaction model](#interactionModel-ssHelixBlockMfe) + +All other parameters are kept from the normal IntaRNA personality, such that +*IntaRNAhelix* predicts per default interactions heuristically based on optimal +helix blocks, which is faster than the normal mode but applies more constraints. +Thus, you should use *IntaRNAhelix* if you want to focus predictions on stable +subinteractions (helices) and need to do it fast. + +In summary, the following calls are equivalent. + +```sh +IntaRNAhelix ... +IntaRNA --personality=IntaRNAhelix ... +IntaRNA --model=B ... +``` + +[![up](doc/figures/icon-up.28.png) back to overview](#overview) + +### IntaRNAduplex + +**TargetScan**, **RNAhybrid** and **RNAduplex** are approaches that predict the +RNA-RNA interaction with +minimal free energy without consideration whether or not the interacting +subsequences are accessible. Furthermore, no seed constraint is taken into account. + +**IntaRNAduplex** provides analogous predictions by + +- disabling [accessibility](#accessibility) consideration. + +In constrast to other approaches, it + +- uses a fast [heuristic prediction mode](#predModes), and +- enforces [seed constraints](#seed). + +Thus, *IntaRNAduplex* is useful if you are not interested in the structuredness +of the interacting molecules, either since they are very short or very long. +The latter makes accessibility prediction difficult, since it is not only governed +by thermodynamics. + +In summary, the following calls are equivalent. + +```sh +IntaRNAduplex ... +IntaRNA --personality=IntaRNAduplex ... +IntaRNA --qAcc=N --tAcc=N ... +``` + +[![up](doc/figures/icon-up.28.png) back to overview](#overview) + +### IntaRNAsTar + +**IntaRNAsTar** provides optimized parameters for large scale (genome-wide) sRNA +target prediction identified via the benchmark introduced in our publication +[(Raden et al., 2020)](https://doi.org/10.1186/s12859-019-3143-4). This covers + +- [no GU base pairs in seeds](#seed) +- [minimal unpaired probability of 0.001 of seed regions](#seed) +- [maximal interaction length of 60](#interConstr) +- [maximal interior loop size of 8](#energy) +- [minimal unpaired probability of 0.001 of interacting regions](#interConstr) + +Furthermore, it ensures +- [no GU base pairs at helix or interaction ends](#interConstr) +- [no lonely base pairs](#interConstr) + + +In summary, the following calls are equivalent. + +```sh +IntaRNAsTar ... +IntaRNA --personality=IntaRNAsTar ... +IntaRNA --intLenMax=60 --intLoopMax=8 --seedNoGU --seedMinPu=0.001 --outMinPu=0.001 --outNoLP --outNoGUend --outOverlap=Q --outMode=C --outCsvCols=id1,id2,start1,end1,start2,end2,E ... +``` + +[![up](doc/figures/icon-up.28.png) back to overview](#overview) + +### IntaRNAseed + +**IntaRNAseed** only considers and reports putative seed interactions. +To this end, it + +- uses [seed-only prediction mode](#predModes). + + +In summary, the following calls are equivalent. + +```sh +IntaRNAseed ... +IntaRNA --personality=IntaRNAseed ... +IntaRNA --mode=S ... +``` + + +[![up](doc/figures/icon-up.28.png) back to overview](#overview) + +### IntaRNAens + +**IntaRNAens** provides [ensemble-based predictions](#interactionModel-ssProbability), +i.e. identifies an optimal site rather than an individual optimal interaction. +This abstracts from individual base pairing and incorporates the base pairing +flexibility of interacting regions. +To do so, it + +- uses [ensemble-based prediction model](#interactionModel-ssProbability). + +In summary, the following calls are equivalent. + +```sh +IntaRNAens ... +IntaRNA --personality=IntaRNAens ... +IntaRNA --model=P ... +``` + + + +[![up](doc/figures/icon-up.28.png) back to overview](#overview) +

+ + +# How to constrain predicted interactions + + + + +

+ + +## Interaction restrictions + +The predicted RNA-RNA interactions can be enhanced if additional +knowledge is available. To this end, IntaRNA provides different options to +restrict predicted interactions. + +One of the most general restriction is the maximal energy (inversely related to +stability) an RNA-RNA interaction is allowed to have. Per default, a reported +interaction should have a negative energy (<0) to be energetically favorable. +This report barrier can be altered using `--outMaxE`. For suboptimal interaction +restriction, please refer to [suboptimal interaction prediction](#subopts) section. + +Another stability constraint is `--outNoLP`, which forbids lonely, i.e. non-stacked, +inter-molecular base pairs. These are typically not contributing much to the +overall stability and can lead to instable subinteractions when e.g. enclosed +by two large interior loops. + +In addition, with `--outNoGUend` one can prohibit weak `GU` base pairs at +interaction ends and within interior loops. +That way, only stable inter-molecular helix ends are considered. + +If you are only interested in predictions for highly accessible regions, i.e. +with a high probability to be unpaired, you can use the `--outMinPu` parameter. +If given, each individual position of the interacting subsequences has to have +an unpaired probability reaching at least the given value. This significantly +increases prediction time but will exclude predictions where the formation of +the interaction (intermolecular base pairing) replaces intramolecular base +pairing (where the latter will cause low unpaired probabilities for the +respective positions). + +Furthermore, the region where interactions are supposed to occur can be restricted +for target and query independently. To this end, a list of according +subregion-defining index pairs +can be provided using `--qRegion` and `--tRegion`, respectively. The indexing +starts with 1 (or the value set via `--qIdxPos0` and `--tIdxPos0`) +and should be in the format `from1-end1,from2-end2,..` using +integers. +Note, if you want to have predictions individually for each region +combination (rather than just the best for each query-target combination) you +want to add `--outPerRegion` to the call. + +If you are dealing with very long sequences it might be useful to use the +*automatic identification of accessible regions*, which dramatically reduces +runtime and memory consumption of IntaRNA since predictions are only done for +individual regions and not for the whole sequence. Here, we use a +heuristic approach that finds and ignores subregions that are unlikely to form +an interaction, resulting in a decomposition of the full sequence range into +intervals of accessible regions. It can be enabled by providing the maximal +length of the resulting intervals via the parameters `--qRegionLenMax` and +`--tRegionLenMax`.
+More specifically, starting from the full +sequence's index range, the algorithm iteratively identifies in every too-long +range the window with highest ED value (penalty for non-accessibility). To +this end, it uses windows of length `--seedBP` to find subsequences where it is +most unlikely that a seed might be formed. This window is removed from the range, +which results in two shorter ranges. If a range is shorter than `--seedBP`, it +is completely removed. + +Finally, it is possible to restrict the overall length an interaction is allowed +to have via `--intLenMax`. This can be done independently for the query and target sequence using +`--qIntLenMax` and `--tIntLenMax`, respectively. By setting to 0 (default), +the smaller of the full sequence length and the maximal accessibility-window +size is used (see `--accW`, `--tAccW`, or `--qAccW`). + + + + + + + +[![up](doc/figures/icon-up.28.png) back to overview](#overview) +

+ + +## Seed constraints + +For different types of RNA-RNA interactions it was found that experimentally +verified interactions were showing a short and compact subinteraction of high +stability (= low energy). It was hypothesized that these regions are among the +first formed parts of the full RNA-RNA interaction, and thus considered as the +*seed* of the overall interaction. + +Based on this observation, RNA-RNA interaction predictors were enhanced by +incorporating such seed constraints into their prediction pipeline, i.e. a +reported interaction has to feature at least one seed. Typically, +a seed is defined as a short subinteraction of 5-8 consecutive base pairs that +are not enclosing any unpaired nucleotides (or if so only very few). + +IntaRNA supports the definition of such seed constraints and adds further +options to even more constrain the seed selection. The list of options is given +by + +- `--seedBP` : the number of base pairs within the seed +- `--seedMaxUP` : the maximal overall number of unpaired bases within the seed +- `--seedQMaxUP` : the maximal number of unpaired bases within the query's seed region +- `--seedTMaxUP` : the maximal number of unpaired bases within the target's seed region +- `--seedMaxE` : the maximal overall energy of the seed (to exclude unlikely seed interactions) +- `--seedMaxEhybrid` : the maximal hybridization energy (including E_init) of the seed (to exclude weak seed hybridizations likely to fall off again) +- `--seedMinPu` : the minimal unpaired probability of each seed region in query and target +- `--seedQRange` : a list of index intervals where a seed in the query is allowed +- `--seedTRange` : a list of index intervals where a seed in the target is allowed +- `--seedNoGU` : if present, no GU base pairs are allowed within seeds +- `--seedNoGUend` : if present, no GU base pairs are allowed at seed ends + +Alternatively, you can set + +- `--seedTQ` : to specify [explicit seed interactions](#seedExplicit) + +Seed constraint usage can be globally disabled using the `--noSeed` flag. + + + +[![up](doc/figures/icon-up.28.png) back to overview](#overview) +

+ + +### Explicit seed input + +Some experiments provide hints or explicit knowledge about the seed or +even provide details about some intermolecular base pairs formed between two RNAs. +This information can be incorporated into IntaRNA predictions by providing +*explicit seed information*. To this end, the `--seedTQ` parameter can be used. +It takes a comma-separated list of seed string encodings in the format +`startTbpsT&startQbpsQ`, which is in the same format as the IntaRNA `hybridDB` +output (see [below](#outModeCsv)), i.e. e.g. `--seedTQ='4|||.|&7||.||'` +(ensure you quote the seed encoding to avoid a shell interpretation of the pipe symbol '|') +to encode a seed interaction like +the following +```bash +target + 4 8 + | | + 5'-AAAC C UGGUUUGG-3' + AC C C + || | | + UG G G + 3'-GGUU U CCCACAAA-5' + | | + 11 7 +query +``` +If several or alternative seeds are known, you can provide all as a +comma-separated list and IntaRNA will consider all interactions that cover at +least one of them. + +[![up](doc/figures/icon-up.28.png) back to overview](#overview) +

+ + +## Helix constraints + +For [helix-based interaction models](#interactionModel-ssHelixBlockMfe), +IntaRNA provides various constraints +for the helices considered for interaction prediction: + +- `--helixMinBP` : minimal number of base pairs inside a helix (min|default=2) +- `--helixMaxBP` : maximal number of base pairs inside a helix +- `--helixMaxE` : maximal energy of each helix; any helix with higher + energy is not considered for interaction prediction. Note, this constraint is + useful to restrict the prediction to more stable interactions. + See also `--helixFullE`. +- `--helixFullE` if present, the overall helix energy (including E_init, ED, + dangling end contributions, etc.) is used for energy checks (`--helixMaxE`). + Otherwise, only the loop-terms are considered. +- `--helixMinPu` : minimal unpaired probability (per helix subsequence); thus, one can + constrain prediction to helices in accessible regions. +- `--helixMaxIL` : maximal size for each internal loop size in a helix; that way + you can relax the helix definition and allow for a given number of unpaired bases + between consecutive base pairs within a helix. Note, increasing this value reduces + the impact of the helix-based model for prediction. + + +[![up](doc/figures/icon-up.28.png) back to overview](#overview) + +

+ + +## SHAPE reactivity data to enhance accessibility computation + +For some RNA sequences, experimental reactivity data is available that can be +used to guide/help the structure and thus accessibility prediction for the RNA +molecule. IntaRNA supports such data by interfacing the Vienna RNA package +capabilities for SHAPE reactivity data incorporation, see +Lorenz et al. ([2015](https://doi.org/10.1093/bioinformatics/btv523), [2016](https://dx.doi.org/10.1186%2Fs13015-016-0070-z)) or the +[RNAfold manpage](https://www.tbi.univie.ac.at/RNA/RNAfold.1.html). +To use this feature, ViennaRNA-based accessibility computation is needed, +i.e. `--t|qAcc=C`, which is the default mode (no action required). + +The SHAPE reactivity data can be provided via file using `--qShape` or +`--tShape` for query or target sequence, respectively. The input format is a +space-separated 3-column format providing position, nucleotide and reactivity value as +exmemplified below. + +```[bash] + 1 G 0.134 + 2 C 0.044 + 3 C 0.057 + 4 G 0.114 + 5 U 0.094 + 6 C 0.035 + 7 G 0.909 + 8 C 0.224 + 9 C 0.529 + 10 A 1.475 +``` + +Independently for each sequence, it is possible +to define the methods to be used to convert the data into pseudo energies and +pairing probabilities. The respective IntaRNA arguments are +`--qShapeMethod`|`--tShapeMethod` +and `--qShapeConversion`|`--tShapeConversion`, which mimics the according +tool arguments in the Vienna RNA package (see e.g. the +[RNAfold manpage](https://www.tbi.univie.ac.at/RNA/RNAfold.1.html)). + +An example call that shows the effect of SHAPE data incorporation is given below (assuming the SHAPE data from above is stored in +the file `data-SHAPE.txt`). +```[bash] +IntaRNA -q GCCGUCGCCA -t GCCGUCGCCA --tShape=data-SHAPE.txt --out=qAcc:STDOUT --out=tAcc:STDERR --out=/dev/null +``` + + +For further details, please refer to our respective publication + +- [Integration of accessibility data from structure probing into RNA-RNA interaction prediction.](https://doi.org/10.1093/bioinformatics/bty1029) + + + + +[![up](doc/figures/icon-up.28.png) back to overview](#overview) + + + +

+ + +# Output setup + + +

+ + +## Output modes + +The RNA-RNA interactions predicted by IntaRNA can be provided in different +formats. The style is set via the argument `--outMode` and the different modes +will be discussed below. + +Furthermore, it is possible to define *where to output*, i.e. using `--out` +you can either name a file or one of the stream names `STDOUT`|`STDERR`. Note, +any string not matching one of the two stream names is considered a file name. +The file will be overwritten by IntaRNA! + +If a file name ends in `.gz`, gzip-compressed binary output is generated. + +Besides interaction output, you can set the verbosity of computation information +using the `-v` or `--verbose` arguments. To reduce the output to a minimum, you +can redirect all logging output of user information, warnings or verbose output +to a specific file using `--default-log-file=LOGFILENAME` (no gzip compression +supported). +If you are not interested in any logging output, redirect it to nirvana via +`--default-log-file=/dev/null`. Note, error output is not redirected and always +given on standard output streams. + + +[![up](doc/figures/icon-up.28.png) back to overview](#overview) + + + +### Standard RNA-RNA interaction output with ASCII chart + +The standard output mode `--outMode=N` provides a detailed ASCII chart of the +interaction together with its overall interaction energy. +For an example see the [Just run ...](#defaultRun) section. + + +[![up](doc/figures/icon-up.28.png) back to overview](#overview) + + + +### Detailed RNA-RNA interaction output with ASCII chart + +Using `--outMode=D`, a detailed ASCII chart of the interaction together with +various interaction details will be provided. An example is given below. + +```bash +# call: IntaRNA -t AAACACCCCCGGUGGUUUGG -q AAACACCCCCGGUGGUUUGG --outMode=D --seedBP=4 + +target + 5 16 + | | + 5'-AAAC C U UUGG-3' + ACC CCGG GGU + ||| ++++ ||| + UGG GGCC CCA + 3'-GGUU U C CAAA-5' + | | + 16 5 +query + +interaction seq1 = 5--16 +interaction seq2 = 5--16 + +interaction energy = -6.76 kcal/mol + = E(init) = 4.1 + + E(loops) = -18.8 + + E(dangleLeft) = -0.6 + + E(dangleRight) = -0.6 + + E(endLeft) = 0.5 + + E(endRight) = 0.5 + : E(hybrid) = -14.9 + + ED(seq1) = 4.07 + : Pu(seq1) = 0.00135534 + + ED(seq2) = 4.07 + : Pu(seq2) = 0.00135534 + +seed seq1 = 9--12 +seed seq2 = 9--12 +seed energy = -1.42 +seed ED1 = 2.66 +seed ED2 = 2.66 +seed Pu1 = 0.0133541 +seed Pu2 = 0.0133541 +``` +Note, within this output `seq1` and respective values refer to the target +sequence and `seq2` etc. to the query RNA. +Position annotations start indexing with 1 at the 5'-end of each RNA +(or the respective value set via `--qIdxPos0` and `--tIdxPos0`). +`ED` values are the energy penalties for reduced [accessibility](#accessibility) +and `Pu` denote unpaired probabilities of the respective interacting subsequences. +Note, `Pu` values are recomputed from rounded `ED` values and are thus not equal +to the RNAplfold values from which the ED values are derived from! +Base pairs that are part of putative seed interactions are represented by +`+` symbols within the interaction chart. Note, an interaction can contain more +than one putative seed and all respective base pairs are highlighted. +In addition, the seed information (given at the end of the output) will provide +information for all possible seeds sorted by increasing energy (ie. the most +stable and thus most likely seed is always given left most). +If you are only interested in the most stable seed, you should add +`--outBestSeedOnly` to the call. + + + +[![up](doc/figures/icon-up.28.png) back to overview](#overview) + + + +### Customizable CSV RNA-RNA interaction output + +IntaRNA provides via `--outMode=C` a flexible interface to generate RNA-RNA +interaction output in CSV format (using `;` as separator). Note, target sequence +information is listed with index `1` while query sequence information is given +by index `2`. + +```bash +# call: /IntaRNA -t AAACACCCCCGGUGGUUUGG -q AAACACCCCCGGUGGUUUGG --outMode=C --seedBP=4 --outOverlap=B -n 3 +id1;start1;end1;id2;start2;end2;subseqDP;hybridDP;E +target;5;16;query;5;16;ACCCCCGGUGGU&ACCCCCGGUGGU;(((.((((.(((&))).)))).)));-6.76 +target;6;16;query;5;15;CCCCCGGUGGU&ACCCCCGGUGG;((.((((.(((&))).)))).));-5.56 +target;7;16;query;5;15;CCCCGGUGGU&ACCCCCGGUGG;((((((.(((&))).)))).));-5.55 +``` +For each prediction, a row in the CSV is generated. The column separator within +the tabular CSV output can be changed using `--outSep`, e.g. to produce tab-separated +`.tsv` output. + +Using the argument `--outCsvCols`, the user can specify what columns are +printed to the output using a comma-separated list of colIds. Available colIds +are + +- `id1` : id of first sequence (target) +- `id2` : id of second sequence (query) +- `seq1` : full first sequence +- `seq2` : full second sequence +- `subseq1` : interacting subsequence of first sequence +- `subseq2` : interacting subsequence of second sequence +- `subseqDP` : hybrid subsequences compatible with hybridDP +- `subseqDB` : hybrid subsequences compatible with hybridDB +- `start1` : start index of hybrid in seq1 +- `end1` : end index of hybrid in seq1 +- `start2` : start index of hybrid in seq2 +- `end2` : end index of hybrid in seq2 +- `hybridDP` : hybrid in VRNA dot-bracket notation (interaction sites only) +- `hybridDPfull` : hybrid in VRNA dot-bracket notation (full sequence length) +- `hybridDB` : hybrid in dot-bar notation (interactin sites only) +- `hybridDBfull` : hybrid in dot-bar notation (full sequence length) +- `bpList` : list of hybrid base pairs, e.g. '(4,3):(5,2):(7,1)' +- `E` : overall interaction energy = E_hybrid + ED1 + ED2 +- `ED1` : ED value of seq1 +- `ED2` : ED value of seq2 +- `Pu1` : probability to be accessible for seq1 +- `Pu2` : probability to be accessible for seq2 +- `E_init` : initiation energy +- `E_loops` : sum of loop energies (excluding E_init) +- `E_dangleL` : dangling end contribution of base pair (start1,end2) +- `E_dangleR` : dangling end contribution of base pair (end1,start2) +- `E_endL` : penalty of closing base pair (start1,end2) +- `E_endR` : penalty of closing base pair (end1,start2) +- `E_hybrid` : energy of hybridization only = E - ED1 - ED2 +- `E_norm` : length normalized energy = E / ln(length(seq1)*length(seq2)) +- `E_hybridNorm` : length normalized energy of hybridization only = E_hybrid / ln(length(seq1)*length(seq2)) +- `E_add` : user defined energy correction term incorporated into `E` +- `w` : Boltzmann weight of `E`, e.g. used for partition function computation +- `seedStart1` : start index of the seed in seq1 (* see below) +- `seedEnd1` : end index of the seed in seq1 (* see below) +- `seedStart2` : start index of the seed in seq2 (* see below) +- `seedEnd2` : end index of the seed in seq2 (* see below) +- `seedE` : overall energy of the seed only (including seedED etc) (* see below) +- `seedED1` : ED value of seq1 of the seed only (excluding rest) (* see below) +- `seedED2` : ED value of seq2 of the seed only (excluding rest) (* see below) +- `seedPu1` : probability of seed region to be accessible for seq1 (* see below) +- `seedPu2` : probability of seed region to be accessible for seq2 (* see below) +- `Eall` : ensemble energy of all considered interactions (-RT*log(`Zall`)) +- `Zall` : partition function of all considered interactions +- `Eall1` : ensemble energy of all considered intra-molecular structures of seq1 (given its accessibility constraints) +- `Eall2` : ensemble energy of all considered intra-molecular structures of seq2 (given its accessibility constraints) +- `EallTotal` : total ensemble energy of all considered interactions including the ensemble energies of intra-molecular structure formation (`Eall+Eall1+Eall2`) +- `Etotal` : total energy of an interaction including the ensemble energies of intra-molecular structure formation (`E+Eall1+Eall2`) +- `Zall1` : partition function represented by `Eall1` (exp(-`Eall1`/RT)) +- `Zall2` : partition function represented by `Eall2` (exp(-`Eall2`/RT)) +- `P_E` : probability of an interaction (site) within the considered ensemble +- `RT` : normalized temperature used for Boltzmann weight computation + +(*) Note, since an interaction can cover more than one seed, all `seed*` columns +might contain multiple entries separated by ':' symbols. In order to print only +the most stable (lowest seed energy) seed information add `--outBestSeedOnly` +to the call. + +Note further, `Pu` values are recomputed from rounded `ED` values and are thus not equal +to the RNAplfold values from which the ED values are derived from! + +Using `--outCsvCols '*'`, all available columns are added to the output. + +Energies are provided in unit *kcal/mol*, probabilities in the interval [0,1]. +Position annotations start indexing with 1 (or the values set via `--qIdxPos0` and `--tIdxPos0`). + +The `hybridDP` format is a dot-bracket notation as e.g. generated by **RNAup**. +Here, for each target sequence position within the interaction, +a '.' represents a position not involved +in the interaction while a '(' marks an interacting position. For the query +sequence this is done analogously but using a ')' for interacting positions. +Both resulting strings are concatenated by a separator '&' to yield a single +string encoding of the interaction's base pairing details. + +The `hybridDB` format is similar to the `hybridDP` but also provides site information. +Here, a bar '|' is used in both base pairing encodings (which makes it a 'dot-bar encoding'). +Furthermore, each interaction string is prefixed +with the start position of the respective interaction site. + +In the following, an altered CSV output for the example from above is generated. +```bash +# call: IntaRNA --outCsvCols=Pu1,Pu2,subseqDB,hybridDB -t AAACACCCCCGGUGGUUUGG -q AAACACCCCCGGUGGUUUGG --outMode=C --seedBP=4 --outOverlap=B -n 3 +Pu1;Pu2;subseqDB;hybridDB +0.00135534;0.00135534;5ACCCCCGGUGGU&5ACCCCCGGUGGU;5|||.||||.|||&5|||.||||.||| +0.00135534;0.00135534;6CCCCCGGUGGU&5ACCCCCGGUGG;6||.||||.|||&5|||.||||.|| +0.00137751;0.00135534;7CCCCGGUGGU&5ACCCCCGGUGG;7||||||.|||&5|||.||||.|| +``` + + +You can produce a *sorted CSV output* using the argument `--outCsvSort=..`, +which provides the colId from the CSV output to be used for sorting. +Below, you find the `hybridDB`-sorted output from above. +```bash +# call: IntaRNA --outCsvCols=Pu1,Pu2,subseqDB,hybridDB --outCsvSort=hybridDB -t AAACACCCCCGGUGGUUUGG -q AAACACCCCCGGUGGUUUGG --outMode=C --seedBP=4 --outOverlap=B -n 3 +Pu1;Pu2;subseqDB;hybridDB +0.00135534;0.00135534;5ACCCCCGGUGGU&5ACCCCCGGUGGU;5|||.||||.|||&5|||.||||.||| +0.00135534;0.00135534;6CCCCCGGUGGU&5ACCCCCGGUGG;6||.||||.|||&5|||.||||.|| +0.00137751;0.00135534;7CCCCGGUGGU&5ACCCCCGGUGG;7||||||.|||&5|||.||||.|| +``` + + +[![up](doc/figures/icon-up.28.png) back to overview](#overview) + + + +### Ensemble output of considered RNA-RNA interactions + +The output mode `--outMode=E` provides information on the ensemble of all +considered RNA-RNA interactions (e.g. compatible with the current seed constraints). +This covers key-value pairs of the following information using +column labels introduced for the CVS output: + +- `id1` : id of first sequence (target) +- `id2` : id of second sequence (query) +- `RT`: the scaled temperature used for Boltzmann-weight computation +- `Eall` : ensemble energy of all considered interactions (-RT*log(Zall)) +- `Eall1` : ensemble energy of all considered intra-molecular structures of seq1 (given its accessibility constraints) +- `Eall2` : ensemble energy of all considered intra-molecular structures of seq2 (given its accessibility constraints) +- `EallTotal` : total ensemble energy of all considered interactions including the ensemble energies of intra-molecular structure formation (`Eall+Eall1+Eall2`) + +Note, `Zall` depends on the selected +[prediction mode](#predModes) and +[RNA-RNA interaction model](#interactionModel). +It holds + +- `Zall(--model=S) <= Zall(--model=P)` as well as +- `Zall(--mode=H --model=P) <= Zall(--mode=M --model=P)`. + +Thus, most accurate results are computed using +``` +IntaRNA --model=P --mode=M --outMode=E ... +``` + +BUT: the **seed-extension strategy in `--model=X` does not allow for correct `Zall` computation**. Thus, avoid `Eall` or `Zall` output in this mode! + + +[![up](doc/figures/icon-up.28.png) back to overview](#overview) + + + +

+ + +## Pairwise vs. all-vs-all + +When multiple query and target sequences are provided, IntaRNA predicts interactions for +all pairs of query-target combinations, i.e. **all-vs-all**. + +Alternatively, you can enforce **pairwise** processing using `--outPairwise`. +When given, sequences are combined based on their input order, i.e. the 5th target is +(only) considered for interaction prediction with the 5th query sequence. Thus, you +have to provide the same number of query and target sequences. + + +

+ + +## Sequence indexing + +Per default, IntaRNA assumes sequence indexing to be natural, i.e. starting with +index 1 at the RNAs' 5'-ends. + +If needed, you can alter indexing (independently for query and target) using the +`--qIdxPos0` and `--tIdxPos0` parameters, respectively. That way you can + +- *start indexing at an arbitrary position*, e.g. for subsequences of long RNAs or genomes, +- get a *0-based indexing* if needed by your down-stream processing, or +- get *relative indexing*, e.g. relative to the start codon within an mRNA sequence. + +The latter, i.e. relative sequencing, starts increasing indexing with the given +first (negative) index but omits the '0' index. Thus, position '-1' is followed by '+1'. +An example for a start-codon-related indexing is given below, where the start codon +within the target is located at position 9. + +``` +# call: IntaRNA -t gacugguaAUGggac --tIdxPos0=-8 -q UCUUACCGUGAGUC --seedBP=5 + +target + -8 1 + | | + 5'- --- UGGGAC-3' + GACU GGUAA + :||| +++++ + CUGA CCAUU + 3'- GUG CU-5' + | | + 14 3 +query + +interaction energy = -6.39 kcal/mol + +``` + + + + +[![up](doc/figures/icon-up.28.png) back to overview](#overview) + + + +

+ + +## Suboptimal RNA-RNA interaction prediction and output restrictions + +Besides the identification of the optimal (e.g. minimum-free-energy) RNA-RNA +interaction, IntaRNA enables the enumeration of suboptimal interactions. To this +end, the argument `-n N` or `--outNumber=N` can be used to generate up to `N` +interactions for each query-target pair (including the optimal one). + +*Note*: suboptimal interaction enumeration is not exhaustive! That is, for each +interaction site (defined by the left- and right-most intermolecular base pair) +only the best interaction is reported! In heuristic prediction mode (default +mode of IntaRNA), this is even less exhaustive, since only for each left-most +interaction boundary one interaction is reported! + +Furthermore, it is possible to *restrict (sub)optimal enumeration* using + +- `--outMaxE` : maximal energy for any interaction reported +- `--outDeltaE` : maximal energy difference of suboptimal interactions' energy + to the minimum free energy interaction +- `--outOverlap` : defines if and where overlapping of reported interaction sites + is allowed: + - 'N' : no overlap neither in target nor query allowed for reported interactions + - 'B' : overlap allowed for interacting subsequences for both target and query + - 'T' : overlap allowed for interacting subsequences in target only + - 'Q' : overlap allowed for interacting subsequences in query only + +*Note*: non-overlapping output (i) is heuristic by considering for each left +interaction site only the best right extension for overlap computation and +(ii) increases runtime. To get optimized results of non-overlapping suboptimals, +rerun IntaRNA and mark the optimal (mfe) interaction region as +[blocked](#accConstraints). + + + + +[![up](doc/figures/icon-up.28.png) back to overview](#overview) + +

+ + +## Energy parameters and temperatures + +The selection of the correct temperature and energy parameters is cruicial for +a correct RNA-RNA interaction prediction. To this end, various settings are +supported by IntaRNA. + +The temperature can be set via `--temperature=C`to set a temperature `C` in +degree Celsius. Note, this is important especially for predictions within plants +etc., since the default temperature is 37C. + +The energy model used can be specified using the `--energy` parameters using + +- 'B' for base pair maximization similar to the Nussinov intramolecular structure prediction. + Here, each base pair contributes an energy term of `-1` independently of its + structural or sequence context. This mode is mainly useful for study or teaching + purpose. +- 'V' enables *Nearest Neighbor Model* energy computation similar to the Zuker + intramolecular structure prediction using the Vienna RNA package routines. + Within this model, the energy contribution of a base + pair depends on its directly enclosed (neighbored) basepair and the subsequence(s) + involved. Different energy parameter sets have been experimentally derived + in the last decades. Since IntaRNA makes use of the energy evaluation routines + of the Vienna RNA package, all parameter sets from the Vienna RNA package are + available for RNA-RNA interaction prediction. Per default, the default parameter + set of the linked Vienna RNA package version is used. You can change the parameter + set using the `--energyVRNA` parameter as explained below. + +If Vienna RNA package is used for energy computation (`--energy=V`), per default +the default parameter set of the linked Vienna RNA package is used (e.g. the +`Turner04` set for VRNA 3.0.0). If you want to use a different parameter set, you +can provide an according parameter file via `--energyVRNA=MyParamFile`. The +following example shows how to run an IntaRNA-v1-like prediction (default for the +`IntaRNA1` personality) using the old `Turner99` parameter set (as used by IntaRNA v1.*). +```bash +# IntaRNA v1.* like energy parameter setup +IntaRNA --energyVRNA=Turner99 +# alternative IntaRNA v1.* like energy parameter setup with explicit file +IntaRNA --energyVRNA=/usr/local/share/Vienna/rna_turner1999.par +``` +IntaRNA provides the following predefined `--energyVRNA` values that load respective +parameter sets from the Vienna RNA package without explicit file specification: +- `Turner04` = `rna_turner2004.par` +- `Turner99` = `rna_turner1999.par` +- `Andronescu07` = `rna_andronescu2007.par` +If no value for `--energyVRNA` is provided, the default model of the underlying +Vienna RNA package is used (see respective documentation). + +To increase prediction quality and to reduce the computational complexity, the +number of unpaired bases between intermolecular base pairs is restricted via +`--intLoopMax` (similar to internal loop length restrictions in single RNA folding algorithm). The +upper bound can be set independently for the query and target sequence via +`--qIntLoopMax` and `--tIntLoopMax`, respectively, and defaults to 16. + +Note, you can shift the energy spectrum using `--energyAdd=..`. The given value +is always added when the overall energy of an interaction is computed. That way, +you can correct for conditional predictions based on given +[accessibility constraints](#accConstraints). + +If lonely (non-stacked) inter-molecular base pairs are of no interest, since they +can lead to instable subinteraction when enclosed by two large interior loops, +one can disable their prediction using `--outNoLP`. + +For some applications it is needed to ignore dangling end energy contributions +that reflect minor stabilizing (stacking) contributions of unpaired bases +adjacent to a helix-closing base pair. +They can be ignored via `--energyNoDangles`. + + + +[![up](doc/figures/icon-up.28.png) back to overview](#overview) + +

+ + +## Additional output files + +IntaRNA enables the generation of various additional information in dedicated +files/streams. The generation of such output is guided by an according (repeated) +definition of the `--out` argument in combination with one of the following +argument prefixes (case insensitive) that have to be colon-separated to the +targeted file/stream name: + +- `qSpotProb:`/`tSpotProb:` [query/target's spot probability profile](#profileSpotProb) (CSV format), respectively +- `spotProb:` [all spot probabilities](#spotProb) (CSV format) +- `qMinE:`/`tMinE:` [the query/target's minimal interaction energy profile](profileMinE) (CSV format), respectively +- `pMinE:` [minimal interaction energy for all query-target index pairs](pairMinE) (CSV format) +- `qAcc:`/`tAcc:` the [query/target's ED accessibility values](#accessibility) (RNAplfold-like format), respectively +- `qPu:`/`tPu:` the [query/target's unpaired probabilities](#accessibility) (RNAplfold format; rounded!!), respectively + +Note, for *multiple sequences* in FASTA input, the provided file names are suffixed by +- `-t#q#` : for `*spotProb` and `*minE` output, and +- `-s#` : for `*Acc` and `*Pu` output, +where `#` denotes the according target/query sequence number +within the input where numbering starts with 1. + +The column separator within tabular CSV output (defaulting to `;`) can be changed +using `--outSep`, e.g. to produce tab-separated `.tsv` output. + +Note further, `qPu:`|`tPu:` will report unpaired probability values based on rounded accessibility (ED) values. +Thus, these values will most likely differ from values eg. produced by the program RNAplfold. +We therefore strongly recommend to store `qAcc:`|`tAcc:` values when you want to use them +as input for subsequent IntaRNA calls! + +As for normal output, if the specified file name ends in `.gz`, gzip-compressed +binary output is generated. This is especially useful for large output data like +accessibility or unpaired probability information as well as pairwise energy +or spot probability profiles. + + +[![up](doc/figures/icon-up.28.png) back to overview](#overview) + +
+ + +### Minimal energy profiles + +To get a more global view of possible interaction sites for a pair of interacting +RNAs, one can generate the *minimal energy profile* for each sequence (independently). + +For instance, to generate the target's profile, add the following to your IntaRNA +call: `--out=tMinE:MYPROFILEFILE.csv`. For the query's profile, use `--out=qMinE:..` respectively. +This will produce an according CSV-file (`;` separated) with the according minimal +energy profile data that can be visualized with any program of your liking. + +In the following, such an output was visualized using R: +```R +d <- read.table("MYPROFLEFILE.csv", header=T, sep=";"); +plot( d[,1], d[,3], xlab="sequence index", ylab="minimal energy", type="l", col="blue", lwd=2) +abline(h=0, col="red", lty=2, lwd=2) +``` + +![Minimal interaction energy profile of an RNA](./doc/figures/profile-minE.png) + +This plot reveals two less but still stable (*E* below 0) interaction sites beside the +mfe interaction close to the 5'-end of the molecule. + + +[![up](doc/figures/icon-up.28.png) back to overview](#overview) + +
+ + +### Minimal energy for all intermolecular index pairs + +To investigate how stable RNA-RNA interactions are distributed for a given pair +of RNAs, you can also generate the minimal energy for all intermolecular index +pairs using `--out=pMinE:MYPAIRMINE.csv`. This generates a CSV file (`;`separated) +holding for each index pair the minimal energy of any interaction covering this +index combination or `NA` if no covers it at all. + +This information can be visualized with your preferred program. In the following, +the provided R call is used to generate a heatmap visualization of the +RNA-RNA interaction possibilities. + +```R +# read data, skip first column, and replace NA and E>0 values with 0 +d <- read.table("MYPAIRMINE.csv",header=T,sep=";"); +d <- d[,2:ncol(d)]; +d[is.na(d)] = 0; +d[d>0] = 0; +# plot +image( 1:nrow(d), 1:ncol(d), as.matrix(d), col = heat.colors(100), xlab="index in sequence 1", ylab="index in sequence 2"); +box(); +``` + +The following plot (for the [minimal energy profile](#profileMinE) example from +above) reveals, that the alternative stable (*E*<0) interactions all involve the +mfe-site in the second sequence and are thus less likely to occure. + +![Minimal interaction energy index pair information](./doc/figures/pair-minE.png) + +A more sophisticated plot can be done using `ggplot2` from the `tidyverse` package. + +```R +library(tidyverse) + +read_delim("pairMinE-t1q1.csv", delim=";") %>% + replace(is.na(.), 0) %>% + rename( target = minE ) %>% + pivot_longer( cols=-target, names_to = "query", values_to = "E" ) %>% + separate( query, into = c("qSeq","qIdx"), sep = "_") %>% + separate( target, into = c("tSeq","tIdx"), sep = "_") %>% + mutate(across(c(tIdx,qIdx,E),as.numeric)) %>% + mutate(E = ifelse( E>0, 0, E)) %>% + ggplot(aes(x=tIdx,y=qIdx, fill=E)) + + geom_tile() + + scale_fill_gradient(low="blue",high="red") +``` + + +[![up](doc/figures/icon-up.28.png) back to overview](#overview) + +
+ + +### Spot probability profiles + +Similarly to [minimal energy profiles](#profileMinE), it is also possible to +compute position-wise probabilities how likely a position is covered by an +interaction, i.e. its *spot probability*. To the end, we compute for each +position *i* the partition function *Zi* of all interactions covering *i*. +Given the overall partition function *Z* including all possible interactions, +the position-speficit spot probability for *i* is given by *Zi/Z*. + +Such profiles can be generated using `--out=qSpotProb:MYPROFILEFILE.csv` or +`--out=tSpotProb:...` for the query/target sequence respectively and independently. + +Note, instead of a file you can also write the profile to stream using `STDOUT` +or `STDERR` instead of a file name. + + +[![up](doc/figures/icon-up.28.png) back to overview](#overview) + +
+ + +### Interaction probabilities for interaction spots of interest + +For some research questions, putative regions of interactions are known from +other sources and it is of interest to study the effect of competitive binding +or other scenarios that might influence the accessibility of the interacting +RNAs (e.g. refer to [SHAPE data](#shape) or +[structure/accessibility constraints](#accConstraints)). + +To this end, one can specify the spots of interest by intermolecular index pairs, +e.g. using `5&67` to encode the fifth target RNA position (first number of the +encoding) and the 67th query RNA +position (second number of the encoding). Note, indexing starts with 1 + (or the values set via `--qIdxPos0` and `--tIdxPos0`). +Multiple spots can be provided as comma-separated list. The list in +concert with an output stream/file name (colon-separated) can be passed via the +`--out` argument using the `spotProb:` prefix, e.g. + +```[bash] +IntaRNA ... --out=spotProb:5&67,33&12:mySpotProbFile.csv +``` + +The reported probability is the ratio of according partition functions. That is, +for each interaction `I` that respects all input constraints and has an energy +below 0 (or set `--outMaxE` value) the respective Boltzmann weight `bw(I)` +is computed by `bw(I) = exp( - E(I) / RT )`. This weight is added to the +`overallZ` partition function. Furthermore, we add the weight to a respective +spot associated partition function `spotZ`, if the interaction `I` spans the spot, ie. +the spot's indices are within the interaction subsequences of `I`. If none of +the spots if spanned by `I`, the `noSpotZ` partition function is increased by +`bw(I)`. The final probability of a spot is than given by `spotZ/overallZ` and +the probability of interactions not covering any of the tracked spots is +computed by `noSpotZ/overallZ` and reported for the pseudo-spot encoding `0&0` +(if indexing starts with 1) or (`--qIdxPos0`-1)`&`(`--tIdxPos0`-1) if the respective +parameters for index shifting are set. + +*NOTE* and be aware that the probabilities are *only estimates* since IntaRNA +is not considering (in default prediction mode) all possible interactions due +to its heuristic (see [discussion about suboptimal interactions](#subopts)). +Nevertheless, since the Boltzmann probabilities are dominated by the low(est) +energy interactions, we consider the probability estimates as meaningful! + +[![up](doc/figures/icon-up.28.png) back to overview](#overview) + +#### Spot probabilities for all intermolecular index pairs + +If interested in all intermolecular index pair combinations (all spots), you +can exclude the list from the call and only specify the output file/stream by +`--out="spotProb:MYSPOTPROBFILE.csv"`. The resulting semicolon-separated table provides the spot +probability for each index pair combination, as shown below. + +```[bash] +# call : IntaRNA --out="spotProb:STDOUT" -t AAACACCCCCGGUGGUUUGG -q AAACACCCCCGGUGGUUUGG --energy=B -m M --seedBP=3 --out=/dev/null +spotProb;A_1;A_2;A_3;C_4;A_5;C_6;C_7;C_8;C_9;C_10;G_11;G_12;U_13;G_14;G_15;U_16;U_17;U_18;G_19;G_20 +A_1;1.14168e-05;3.62052e-05;6.5358e-05;0.000125994;0.000196552;0.000328442;0.000553431;0.000824511;0.00120974;0.00133627;0.00134247;0.00135501;0.00140472;0.000700197;0.000700197;0.000700197;0.000488841;0.000264782;0;0 +A_2;3.62052e-05;0.000114814;0.000229225;0.00039447;0.000706437;0.00109931;0.00184489;0.00275296;0.00395164;0.00437528;0.00440057;0.00444726;0.0046156;0.00229778;0.00229778;0.00229778;0.00160785;0.000873259;0;0 +A_3;6.5358e-05;0.000229225;0.000508878;0.000924219;0.00175717;0.00281323;0.00496666;0.00757395;0.0106558;0.0116963;0.0117524;0.0118769;0.0122887;0.00552015;0.00552015;0.00552015;0.00356421;0.00147568;0;0 +C_4;0.000125994;0.00039447;0.000924219;0.00202415;0.00404467;0.00714106;0.0130565;0.0190823;0.0252285;0.0267963;0.0268554;0.0269861;0.0249678;0.0181993;0.00847111;0.0061239;0.00416796;0.00207943;0.000603754;0.000139873 +A_5;0.000196552;0.000706437;0.00175717;0.00404467;0.00820796;0.013931;0.0259249;0.0418595;0.0596996;0.0666565;0.0668406;0.0675258;0.067224;0.0307559;0.0210277;0.0186805;0.00878539;0.00352443;0.000603754;0.000139873 +C_6;0.000328442;0.00109931;0.00281323;0.00714106;0.013931;0.0247509;0.0449983;0.069822;0.101313;0.110441;0.110632;0.111336;0.0991588;0.0626907;0.0334127;0.0198717;0.00997658;0.00471562;0.00179495;0.000446343 +C_7;0.000553431;0.00184489;0.00496666;0.0130565;0.0259249;0.0449983;0.0786128;0.119909;0.175115;0.191833;0.192042;0.192809;0.159914;0.123446;0.0559382;0.0212691;0.011374;0.00611305;0.00319238;0.000889846 +C_8;0.000824511;0.00275296;0.00757395;0.0190823;0.0418595;0.069822;0.119909;0.184088;0.278008;0.309829;0.310077;0.310961;0.24368;0.207212;0.0711069;0.0228361;0.012941;0.00768002;0.00475935;0.00121992 +C_9;0.00120974;0.00395164;0.0106558;0.0252301;0.0597011;0.101316;0.175118;0.278011;0.442224;0.540951;0.541838;0.548287;0.314152;0.27952;0.134106;0.0307786;0.0390998;0.134187;0.131266;0.127646 +C_10;0.00133627;0.00437528;0.0116963;0.0267979;0.066658;0.110445;0.191836;0.309832;0.540951;0.639678;0.640565;0.549795;0.314804;0.280171;0.134243;0.0307985;0.0391198;0.134207;0.131286;0.127653 +G_11;0.00134247;0.00440057;0.0117524;0.026857;0.0668422;0.110635;0.192045;0.310081;0.541838;0.640565;0.641452;0.550681;0.31569;0.280274;0.134346;0.0309011;0.0391607;0.134221;0.131286;0.127653 +G_12;0.00135501;0.00444726;0.0118769;0.0269877;0.0675273;0.111339;0.192813;0.310965;0.548287;0.549795;0.550681;0.459911;0.244312;0.207938;0.0838498;0.0269663;0.0368367;0.0322818;0.0296791;0.0269458 +U_13;0.00140472;0.0046156;0.0122887;0.0249678;0.067224;0.0991588;0.159914;0.24368;0.314149;0.3148;0.315687;0.244309;0.149036;0.12523;0.0603534;0.0204676;0.0169076;0.0134034;0.0115843;0.00969977 +G_14;0.000700197;0.00229778;0.00552015;0.0181993;0.0307559;0.0626907;0.123446;0.207212;0.279517;0.280168;0.280271;0.207934;0.12523;0.0989915;0.0415884;0.013814;0.0112185;0.00810799;0.00642706;0.00506415 +G_15;0.000700197;0.00229778;0.00552015;0.00847111;0.0210277;0.0334127;0.0559382;0.0711069;0.134103;0.13424;0.134343;0.0838467;0.0603534;0.0415884;0.0198644;0.0078437;0.00499886;0.00303726;0.00193873;0.00115273 +U_16;0.000700197;0.00229778;0.00552015;0.0061239;0.0186805;0.0198717;0.0212691;0.0228361;0.0307754;0.0307954;0.030898;0.0269632;0.0204676;0.013814;0.0078437;0.00372948;0.00238196;0.0014955;0.000994685;0.000636301 +U_17;0.000488841;0.00160785;0.00356421;0.00416796;0.00878539;0.00997658;0.011374;0.012941;0.0390967;0.0391166;0.0391576;0.0368336;0.0169076;0.0112185;0.00499886;0.00238196;0.00153295;0.000941024;0.000584357;0.000391474 +U_18;0.000264782;0.000873259;0.00147568;0.00207943;0.00352443;0.00471562;0.00611305;0.00768002;0.134184;0.134204;0.134217;0.0322786;0.0134034;0.00810799;0.00303726;0.0014955;0.000941024;0.000539355;0.000275861;0.000181808 +G_19;0;0;0;0.000603754;0.000603754;0.00179495;0.00319238;0.00475935;0.131263;0.131283;0.131283;0.0296759;0.0115843;0.00642706;0.00193873;0.000994685;0.000584357;0.000275861;6.54789e-05;2.81631e-05 +G_20;0;0;0;0.000139873;0.000139873;0.000446343;0.000889846;0.00121992;0.127643;0.12765;0.12765;0.0269426;0.00969977;0.00506415;0.00115273;0.000636301;0.000391474;0.000181808;2.81631e-05;5.22605e-06 +``` + +This data can be visualized in heatmaps as discussed for the [minimal energy heatmap](#pairMinE). + + +[![up](doc/figures/icon-up.28.png) back to overview](#overview) + +
+ + +### Accessibility and unpaired probabilities + +Accessibility describes the availability of an RNA subsequence for intermolecular +base pairing. It can be expressed in terms of the probability of the subsequence +to be unpaired (its *unpaired probability* *Pu*). + +A limited accessibility, i.e. a low unpaired probability, can be incorporated into +the RNA-RNA interaction prediction by adding according energy penalties. +These so called *ED* values are transformed unpaired probabilities, i.e. the +penalty for a subsequence partaking in an interaction is given by *ED=-RT log(Pu)*, +where *Pu* denotes the unpaired probability of the subsequence. Within the +IntaRNA energy model, *ED* values for both interacting subsequences are considered. + +To globally turn off accessibility consideration, set `--acc=N`. +Accessibility incorporation can be disabled separately for query or target sequences using +`--qAcc=N` or `--tAcc=N`, respectively. + +A setup of `--acc=C` (default, as for `--qAcc=C` and `--tAcc=C`) enables accessibility computation +using the selected [energy model](#energy) for query or target sequences, respectively. + +Using `--accNoLP` and `--accNoGUend`, the consideration of lonely base pairs +and GU-helix ends can be disabled for accessibility computation (for default +[energy model](#energy) `V`). + + +[![up](doc/figures/icon-up.28.png) back to overview](#overview) + + + +#### Local versus global unpaired probabilities + +Exact computation of unpaired probabilities (*Pu* terms) is considers all possible +structures the sequence can adopt (the whole structure ensemble). This is referred +to as *global unpaired probabilities* as computed e.g. by **RNAup**. + +Since global probability computation is (a) computationally demanding and (b) not +reasonable for long sequences, local RNA folding was suggested, which also enables +according *local unpaired probability* computation, as e.g. done by **RNAplfold**. +Here, a folding window of a defined length 'screens' along the RNA and computes +unpaired probabilities within the window (while only intramolecular base pairs +within the window are considered). + +IntaRNA enables both global as well as local unpaired probability computation. +To this end, the sliding window length `--accW` and the maximal base pair span +`--accL` (<= `--accW`) have to be specified in order to enable/disable +local folding. + +##### Use case examples global/local unpaired probability computation + +The use of global or local accessibilities can be defined independently +for query and target sequences using `--qAccW|L` and `--tAccW|L`, respectively. +Here, `--?AccW` defines the sliding window length (0 sets it to the whole sequence length) +and `--?AccL` defines the maximal length of considered intramolecular base pairs, +i.e. the maximal number of positions enclosed by a base pair +(0 sets it to the whole sequence length). Both can be defined +independently while respecting `AccL <= AccW`. +```bash +# using global accessibilities for target and query +IntaRNA [..] --accW=0 --accL=0 +# using global accessibilities for query and local ones for target +IntaRNA [..] --qAccW=0 --qAccL=0 --tAccW=150 --qAccL=100 +``` + +[![up](doc/figures/icon-up.28.png) back to overview](#overview) + + + +#### Constraints for accessibility computation + +For some RNAs additional accessibility information is available. For instance, +it might be known from experiments that some subsequence is unpaired or already +bound by some other factor. The first case (unpaired) makes such regions +especially interesting for interaction prediction and should result in no ED +penalties for these regions. In the second case (blocked) the region should be +excluded from interaction prediction. + +To incorporate such information, IntaRNA provides the possibility to constrain +the accessibility computation using the `--qAccConstr` and `--tAccConstr` +parameters. Both take a string encoding for each sequence position whether it is + +- `.` unconstrained +- `x` for sure accessible (unpaired) +- `p` paired intramolecularly with some other position of this RNA +- `b` blocked by some other interaction (implies single-strandedness) + +Note, *blocked* regions are currently assumed to be bound single-stranded by some +other factor and thus are *treated as unpaired* for ED computation. + +```bash +# constraining some central query positions to be blocked by some other molecules +IntaRNA [..] --query="GGGGGGGCCCCCCC" \ + --qAccConstr="...bbbb......." +``` + +It is also possible to provide a more compact index-range-based encoding of the +constraints, which is especially useful for longer sequences or if you have only +a few constrained regions. To this end, one can provide a comma-separated list +of index ranges that are prefixed with the according constraint letter from +above and a colon. Best check the following examples, which should give a good +idea how to use. Note, indexing is supposed to be based on a minimal index of 1 +and all positions not covered by the encoding are assumed to be unconstrained +(which must not to be encoded explicitely). + +```bash +# applying the same constraints by different encodings to query and target +# example 1 +IntaRNA [..] --qAccConstr="...bbbb....." --tAccConstr="b:4-7" +# example 2 +IntaRNA [..] --qAccConstr="..bb..xxp.bb" --tAccConstr="b:3-4,11-12,x:7-8,p:9-9" +``` + +When constraining the accessibility computation, all predictions are conditional +predictions for the given accessibility constraints. +If you are still interested in non-conditional results/energies, you will have to +correct the energy computation by providing a respective energy shift to be +applied. By setting `--energyAdd=..`, all energy evaluations will add your given +term and thus IntaRNA will use and provide corrected energy terms. + + +[![up](doc/figures/icon-up.28.png) back to overview](#overview) + + + + + + +#### Scaling factors for partition function computation for accessibility estimation + +To compute the unpaired probabilities, which are the base for accessibility estimation, +one needs to calculate so called partition functions of structure ensembles. The value +of these numbers can be very large for longer sequences, since the RNA structure space +growths exponentially with sequence length. Thus, the respective methods from the +Vienna RNA package employ mathematical tricks to scale the partition function values +in order to keep them within ranges that can be handled with normal data structures. + +If you are investigating very long (or special) sequences, these tricks might be insufficient +and the accessibility computation will fail with an error similar to the one shown below. + +![pf_scale problem](doc/figures/pf_scale.problem.png) + +This error, raised by the underlying routine from the Vienna RNA package, shows that +the value of the partition function `Q` exceeds the possible range for such numbers. + +This problem can be tackled, when supporting the partition function scaling with a +sequence specific scaling factor (the `pf_scale` mentioned within the error above). +You can set this factor using `--qPfScale`|`--tPfScale` for query|target sequence, +respectively. The larger the value, the higher `Q` values can be dealt with but with the +cost of precision. Too high values will render the accessibility computation useless, +since the precision is so low that all unpaired probabilities are dropping to zero. +Thus, be careful when setting the factor. The +[RNAplfold manual](https://www.tbi.univie.ac.at/RNA/RNAplfold.1.html) +recommends values between 1 and 2. But higher values might be needed for very long +sequences. + + +[![up](doc/figures/icon-up.28.png) back to overview](#overview) + + + + +#### Read/write accessibility from/to file or stream + +It is possible to read precomputed accessibility values from file or stream to +avoid their runtime demanding computation. To this end, we support the following +formats + +| Input format | produced by | +| ---- | --- | +| RNAplfold unpaired probabilities | `RNAplfold -u` or `IntaRNA --out=*Pu:` | +| RNAplfold-styled ED values | `IntaRNA --out=*Acc:` | +| ---- | --- | +| .. with gzip-compression | `IntaRNA --out=*:*.gz` | + +The **RNAplfold** format is a table encoding of a banded upper triangular matrix +with band width l. First row contains a header comment on the data starting with +`#`. Second line encodes the column headers, i.e. the window width per column. +Every successive line starts with the index (starting from 1) of the window end +followed by a tabulator separated list for each windows value in increasing +window length order. That is, column 2 holds values for window length 1, column +3 for length 2, ... . The following provides a short output/input +example for a sequence of length 5 with a maximal window length of 3. + +``` +#unpaired probabilities + #i$ l=1 2 3 +1 0.9949492 NA NA +2 0.9949079 0.9941056 NA +3 0.9554214 0.9518663 0.9511048 +4 0.9165814 0.9122866 0.9090283 +5 0.998999 0.915609 0.9117766 +6 0.8549929 0.8541667 0.8448852 + +``` + + +##### Use case examples for read/write accessibilities and unpaired probabilities + +If you have precomputed data, e.g. the file `plfold_lunp` with unpaired probabilities +computed by **RNAplfold**, you can run + +```bash +# fill accessibilities from RNAplfold unpaired probabilities +IntaRNA [..] --qAcc=P --qAccFile=plfold_lunp +# fill accessibilities from RNAplfold unpaired probabilities via pipe +cat plfold_lunp | IntaRNA [..] --qAcc=P --qAccFile=STDIN +``` + +Another option is to store the accessibility data computed by IntaRNA for +successive calls using + +```bash +# storing and reusing compressed (target) accessibility (Pu) data for successive IntaRNA calls +IntaRNA [..] --out=tPu:intarna.target.pu.gz +IntaRNA [..] --tAcc=P --tAccFile=intarna.target.pu.gz +# piping (target) accessibilities (ED values) between IntaRNA calls +IntaRNA [..] --out=tAcc:STDOUT | IntaRNA [..] --tAcc=E --tAccFile=STDIN +``` + + +Note, for *multiple sequences* in FASTA input, one can also load the +accessibilities (for *all* sequencces) from file. To this end, the file names +have to be suffixed with with `-s#`, where `#` denotes the respective sequence's number (where indexing +starts with 1) within the FASTA input using a common suffix after the index. +The file name without `-s#` is to be provided to the according `--?AccFile` argument. +The files generated by `--out=?Acc:...` are already conform to this requirement, +such that you can use the use case examples from above also for multi-sequence +FASTA input. +Note, this is not supported for a piped setup (e.g. via `--out=tAcc:STDOUT` +as shown above), since this does not produce the according output files! + + + + + +[![up](doc/figures/icon-up.28.png) back to overview](#overview) + +



+ + +# Library for integration in external tools + +The IntaRNA package also comes with a C++ library `libIntaRNA.a` containing the core classes +and functionalities used within the IntaRNA tool. The whole library comes with +an `IntaRNA` namespace and exhaustive class and member API documentation that is +processed using doxygen to generate html/pdf versions. + +When IntaRNA is build while `pkg-config` is present, according pkg-config +information is generated and installed too. + +IntaRNA's supported compilation baseline is C++17. External consumers must therefore +select `-std=c++17` (or a newer standard) in their own build; the installed +pkg-config metadata supplies IntaRNA and dependency flags but deliberately +does not override the consuming project's language mode. + +## Mandatory `Easylogging++` initalization ! + +Since IntaRNA makes heavy use of the `Easylogging++` library, you have to add (and adapt) +the following code to your central code that includes the `main()` function: +```[c++] +// get central IntaRNA-lib definitions and includes +#include +// initialize logging for binary +INITIALIZE_EASYLOGGINGPP + +[...] + +int main(int argc, char **argv){ + +[...] + // set overall logging style + el::Loggers::reconfigureAllLoggers(el::ConfigurationType::Format, std::string("# %level : %msg")); + // no log file output + el::Loggers::reconfigureAllLoggers(el::ConfigurationType::ToFile, std::string("false")); + el::Loggers::reconfigureAllLoggers(el::ConfigurationType::ToStandardOutput, std::string("true")); + // set additional logging flags + el::Loggers::addFlag(el::LoggingFlag::DisableApplicationAbortOnFatalLog); + el::Loggers::addFlag(el::LoggingFlag::LogDetailedCrashReason); + el::Loggers::addFlag(el::LoggingFlag::AllowVerboseIfModuleNotSpecified); + + // setup logging with given parameters + START_EASYLOGGINGPP(argc, argv); +[...] +} +``` + +Note further, to get the library correctly working the following compiler +flags are used within the IntaRNA configuration: +```[bash] + CXXFLAGS=" -DELPP_FEATURE_PERFORMANCE_TRACKING -DELPP_NO_DEFAULT_LOG_FILE " +``` + +[![up](doc/figures/icon-up.28.png) back to overview](#overview) + +### Attributions + +[![yes](doc/figures/icon-yes.39.png)](https://www.freepik.com/free-vector/icons-collection_1638275.htm) +[![no](doc/figures/icon-no.39.png)](https://www.freepik.com/free-vector/icons-collection_1638275.htm) +[![up](doc/figures/icon-up.38.png)](https://www.freepik.com/free-vector/colored-arrows_794372.htm) +Designed by Freepik + diff --git a/doc/local-validation-archive/files/issue-244-validation/source-gcc11-debug/configure.ac b/doc/local-validation-archive/files/issue-244-validation/source-gcc11-debug/configure.ac new file mode 100644 index 00000000..4f371a56 --- /dev/null +++ b/doc/local-validation-archive/files/issue-244-validation/source-gcc11-debug/configure.ac @@ -0,0 +1,505 @@ + +AC_PREREQ([2.69]) +# 5 argument version only available with aclocal >= 2.64 +AC_INIT([IntaRNA], [3.4.1], [], [intaRNA], [http://www.bioinf.uni-freiburg.de] ) + +# minimal required version of the boost library +BOOST_REQUIRED_VERSION=1.50.0 +# minimal required version of the Vienna RNA library +VRNA_REQUIRED_VERSION=2.4.14 +# minimal required version of python interpreter (to install intarnapvalue) +PYTHON_REQUIRED_VERSION=3.6 + + +AC_CANONICAL_HOST +AC_CONFIG_AUX_DIR([.]) +AC_CONFIG_SRCDIR([src/easylogging++.h]) +AC_CONFIG_HEADERS([src/config.h]) + +# AC_CONFIG_MACRO_DIR([m4]) +m4_include([m4/m4-ax_cxx_compile_stdcxx.m4]) +m4_include([m4/m4-ax_openmp.m4]) +m4_include([m4/m4-ax_boost_base.m4]) +m4_include([m4/m4-ax_boost_system.m4]) +m4_include([m4/m4-ax_boost_filesystem.m4]) +m4_include([m4/m4-ax_boost_program_options.m4]) +m4_include([m4/m4-ax_boost_regex.m4]) +m4_include([m4/m4-ax_boost_iostreams.m4]) +m4_include([m4/m4-ax_check_zlib.m4]) + +lt_enable_auto_import="" +case "$host_os" in + cygwin* | mingw* | cegcc*) + AM_LDFLAGS="-Wl,--enable-auto-import $AM_LDFLAGS" +esac + + +# check for C++ compiler +# store current compiler flags to avoid default setup via AC_PROG_CXX and *_CC +OLD_CXXFLAGS=$CXXFLAGS +OLD_CFLAGS=$CFLAGS +# Checks for programs. +AC_PROG_CXX +AC_PROG_CC +AC_PROG_LN_S +AC_PROG_RANLIB +AC_PROG_MKDIR_P +# reset compiler flags to initial flags +CXXFLAGS=$OLD_CXXFLAGS +CFLAGS=$OLD_CFLAGS + +# automake initialisation (mandatory) and check for minimal automake API version +AM_INIT_AUTOMAKE([1.11]) + +# use the C++ compiler for the following checks +AC_LANG([C++]) + +# Structured bindings and unordered_map::try_emplace require C++17. +AX_CXX_COMPILE_STDCXX([17], [noext], [mandatory]) + +# Check the library as well: accepting -std=c++17 does not guarantee that +# the compiler is paired with a library providing unordered_map::try_emplace. +AC_MSG_CHECKING([for required C++17 standard library features]) +intarna_saved_CXXFLAGS=$CXXFLAGS +CXXFLAGS="$CXXFLAGS -pedantic-errors" +AC_COMPILE_IFELSE( + [AC_LANG_PROGRAM( + [[#include ]], + [[std::unordered_map partitions; +auto [entry, inserted] = partitions.try_emplace(1, 2.0); +if (!inserted) { + entry->second += 2.0; +}]])], + [AC_MSG_RESULT([yes])], + [AC_MSG_RESULT([no]) + AC_MSG_ERROR([A C++17 standard library providing std::unordered_map::try_emplace is required. Upgrade libstdc++/libc++ or select a newer C++17 toolchain.])]) +CXXFLAGS=$intarna_saved_CXXFLAGS + +# check if python is available +AM_PATH_PYTHON([$PYTHON_REQUIRED_VERSION],, [:]) +AM_CONDITIONAL([HAVE_PYTHON], [test "$PYTHON" != ":"]) +AM_COND_IF([HAVE_PYTHON], [HAVE_PYTHON_yesno=yes], [HAVE_PYTHON_yesno=no]) + +############################################################################### +############################################################################### + +############ DEFAULT MACROS IF PKG-CONFIG MISSING ############### + +m4_define_default([PKG_PROG_PKG_CONFIG], + [AC_MSG_CHECKING([pkg-config]) + AC_MSG_RESULT([no])]) + +PKG_PROG_PKG_CONFIG + +# If the pkg-config autoconf support isn't installed, define its +# autoconf macro to disable any packages depending on it. +# +m4_define_default([PKG_CHECK_MODULES], + [AC_MSG_CHECKING([$1]) + AC_MSG_RESULT([no]) + $4]) + + +############ SILENT (MINIMAL) MAKEFILE OUTPUT ################### + +# show full make output via `make V=1` +m4_ifdef( + [AM_SILENT_RULES], + [AM_SILENT_RULES([yes])] +) + +############ PARAMETERS ######################################## + +############################################################################### +# PKG-CONFIG SETUP +############################################################################### + +AC_MSG_CHECKING([whether to use and provide pkg-config information]) +pkgconfigEnabled=yes +AC_ARG_ENABLE([pkg-config], + [AS_HELP_STRING([--disable-pkg-config], + [disable pkg-config support (def=enabled)])], + [pkgconfigEnabled="$enableval"], + [pkgconfigEnabled=yes]) +AC_MSG_RESULT([$pkgconfigEnabled]) +AS_IF([test x"$pkgconfigEnabled" = x"yes"], [ + # Checks for pkg-config + AC_CHECK_PROG([HAVE_PKG_CONFIG],[pkg-config],[yes],[no]) +], [ + HAVE_PKG_CONFIG=no +]) + +############################################################################### +# DEBUG SUPPORT SETUP +############################################################################### + +AC_MSG_CHECKING([whether to build with debug information]) +debuger=no +AC_ARG_ENABLE([debug], + [AS_HELP_STRING([--enable-debug], + [enable debug data generation (def=disabled)])], + [debuger="$enableval"]) +AC_MSG_RESULT([$debuger]) + +AS_IF([test x"$debuger" = x"yes"], [ + AC_DEFINE([_DEBUG], [1], [Run in DEBUG mode with additional assertions and debug output]) + AM_CXXFLAGS="$AM_CXXFLAGS -g -O0 -Wno-uninitialized -Wno-deprecated -DELPP_DEBUG_ERRORS" # -Wall" +], [ + AC_DEFINE([NDEBUG], [1], [Run in normal mode with minimal assertions]) + AM_CXXFLAGS="$AM_CXXFLAGS -O3 -fno-strict-aliasing -Wno-uninitialized -Wno-deprecated" +]) + +############################################################################### +# STATIC LINKING SUPPORT +############################################################################### + +AC_MSG_CHECKING([whether to build statically linked binaries]) +AC_ARG_ENABLE([static-linking], + [AS_HELP_STRING([--enable-static-linking], + [enables static linking of the executables (default=disabled)])], + [enable_static_linking=$enableval], + [enable_static_linking=no]) +AC_MSG_RESULT([$enable_static_linking]) + +AS_IF([test x"$enable_static_linking" = x"yes"], [ + AM_LDFLAGS="-static $AM_LDFLAGS" +]) + +############################################################################### +# MULTI-THREADING SUPPORT SETUP +############################################################################### + +AC_MSG_CHECKING([whether to enable multi-threading support]) +multithreadingEnabled=yes +AC_ARG_ENABLE([multithreading], + [AS_HELP_STRING([--disable-multithreading], + [disable multi-threading support (def=enabled)])], + [multithreadingEnabled="$enableval"], + [multithreadingEnabled=yes]) +AC_MSG_RESULT([$multithreadingEnabled]) +AS_IF([test x"$multithreadingEnabled" = x"yes"], [ + AC_DEFINE([INTARNA_MULITHREADING], [1], [Enabling multi-threading support]) + AC_SUBST([INTARNA_MULITHREADING],[1]) + # ensure OPENMP can be used + AX_OPENMP([],[AC_MSG_ERROR([OPENMP support is mandatory for multi-threading compilation])]) + AM_CXXFLAGS="$AM_CXXFLAGS $OPENMP_CXXFLAGS" +], [ + AC_DEFINE([INTARNA_MULITHREADING], [0], [Disabling multi-threading support]) + AC_SUBST([INTARNA_MULITHREADING],[0]) + AM_CXXFLAGS="$AM_CXXFLAGS -pthread" +]) + +############################################################################### +# MULTIPRECISION SUPPORT (via GNU quadmath library) +############################################################################### + +AC_MSG_CHECKING([whether to use a multi-precision partition function data type (quadmath)]) +enable_multiprecision=no +AC_ARG_ENABLE([multiprecision], + [AS_HELP_STRING([--enable-multiprecision], + [multi-precision partition function data type (default=disabled)])], + [enable_multiprecision=$enableval], + [enable_multiprecision=no]) +AC_MSG_RESULT([$enable_multiprecision]) + +AS_IF([test x"$enable_multiprecision" = x"yes"], [ + LIBS="$LIBS -lquadmath" + AC_DEFINE([INTARNA_MULTIPRECISION], [1], [Enabling multiprecision partition function (using quadmath)]) + AC_SUBST([INTARNA_MULTIPRECISION],[1]) +], [ + AC_DEFINE([INTARNA_MULTIPRECISION], [0], [Disabling multiprecision partition function]) + AC_SUBST([INTARNA_MULTIPRECISION],[0]) +]) + +############################################################################### +# Vienna RNA package library path support, if not installed in usual directories +############################################################################### +AC_ARG_WITH([vrna], + [AS_HELP_STRING( + [--with-vrna=PREFIX], + [alternative prefix path to Vienna RNA library] + )], + [RNAPATHSET=1], + [RNAPATHSET=0] +) +# handle user-defined path +AS_IF([test $RNAPATHSET = 1 ], [ + AS_IF([test "x$HAVE_PKG_CONFIG" = "xyes"], [ + # use pkg-config data if available + export PKG_CONFIG_PATH="$with_vrna/lib/pkgconfig/:$PKG_CONFIG_PATH" + ], [ + # guess compiler and linker flags if needed + AM_CXXFLAGS="-I$with_vrna/include $AM_CXXFLAGS" + AM_LDFLAGS="-L$with_vrna/lib $AM_LDFLAGS" + ]) +]) + +############################################################################### +dnl generate doxygen documentation in Doxy +############################################################################### + +AC_CHECK_PROG([HAVE_DOXYGEN],[doxygen],[yes],[no]) + +AS_IF([test "x$HAVE_DOXYGEN" = "xyes"], [ + # loading m4 macro + m4_include([m4/doxygen.m4]) + # setup doxygen documentation to build + DX_HTML_FEATURE(ON) + DX_CHM_FEATURE(OFF) + DX_CHI_FEATURE(OFF) + DX_MAN_FEATURE(OFF) + DX_RTF_FEATURE(OFF) + DX_XML_FEATURE(OFF) + DX_PDF_FEATURE(ON) + DX_PS_FEATURE(OFF) + # generate according options etc. + DX_INIT_DOXYGEN($PACKAGE_NAME, [doc/doxygen.cfg], [doxygen-doc]) +]) + +############################################################################### +# EASYLOGGING++ +############################################################################### + +# enable easylogging++ performance tracking +AM_CXXFLAGS="$AM_CXXFLAGS -DELPP_FEATURE_PERFORMANCE_TRACKING" +# disable default log file creation +AM_CXXFLAGS="$AM_CXXFLAGS -DELPP_NO_DEFAULT_LOG_FILE" + +############################# +# COLORING SETUP +############################# + +AC_MSG_CHECKING([whether to enable coloring of console log output]) +logColoring=yes +AC_ARG_ENABLE([log-coloring], + [AS_HELP_STRING([--disable-log-coloring], + [disable console log coloring (def=enabled)])], + [logColoring="$enableval"]) +AC_MSG_RESULT([$logColoring]) +AS_IF([test x"$logColoring" = x"yes"], [ + AC_DEFINE([INTARNA_LOG_COLORING], [1], [Enabling console log coloring]) + AC_SUBST([INTARNA_LOG_COLORING],[1]) +], [ + AC_DEFINE([INTARNA_LOG_COLORING], [0], [Disabling console log coloring]) + AC_SUBST([INTARNA_LOG_COLORING],[0]) +]) + + + +############################# +# P-VALUE SCRIPT +############################# + +AC_MSG_CHECKING([whether to compile and install 'intarnapvalue' python module]) +intarnapvalue=no +AC_ARG_ENABLE([intarnapvalue], + [AS_HELP_STRING([--enable-intarnapvalue], + [build and install the intarnapvalue python3 module])], + [AM_COND_IF([HAVE_PYTHON], [intarnapvalue="$enableval"], [intarnapvalue="no (python not available)"])]) +AC_MSG_RESULT([$intarnapvalue]) + +AM_CONDITIONAL([INSTALL_INTARNAPVALUE],[test x"$intarnapvalue" = x"yes"]) + + + +############################################################################### + +############################################################################### +# BOOST CHECK +############################################################################### + +AX_BOOST_BASE([$BOOST_REQUIRED_VERSION], [FOUND_BOOST=1;], [FOUND_BOOST=0;]) +AS_IF([test "x$want_boost" != "xno" && test "x$FOUND_BOOST" = "x1"], [ + AX_BOOST_SYSTEM + AX_BOOST_FILESYSTEM + AX_BOOST_PROGRAM_OPTIONS + AX_BOOST_REGEX + AX_BOOST_IOSTREAMS +]) +AX_CHECK_ZLIB([], []) + + +############ CHECKS ############################################ + +# Checks for typedefs, structures, and compiler characteristics. +AC_TYPE_SIZE_T + +########################################################################## +# check boost test results +########################################################################## + +# FOUND_BOOST is only defined if want_boost is "yes" +AS_IF([test $want_boost != "no" && test $FOUND_BOOST != 1], [ + AC_MSG_NOTICE([]) + AC_MSG_NOTICE([The Boost Library was not found!]) + AC_MSG_NOTICE([ -> If installed in a non-standard path, please use '--with-boost=PREFIX'.]) + AC_MSG_NOTICE([]) + DEPENDENCYNOTFOUND=1; +], [ + AM_CXXFLAGS="$BOOST_CPPFLAGS $AM_CXXFLAGS" + AM_LDFLAGS="$BOOST_LDFLAGS $AM_LDFLAGS" + LIBS="$LIBS -lboost_regex -lboost_program_options -lboost_filesystem -lboost_system -lboost_iostreams -lz" +]) + + +############################################################################### +# BEGIN VIENNA CHECK +############################################################################### +# check for Vienna RNA headers + +RNANOTFOUND=0 +AS_IF([test x"$HAVE_PKG_CONFIG" = x"yes"], [ + AC_MSG_NOTICE([check if VRNA version >= $VRNA_REQUIRED_VERSION]) + PKG_CHECK_MODULES([VRNA], [RNAlib2 >= $VRNA_REQUIRED_VERSION], [RNANOTFOUND=0], [RNANOTFOUND=1]) +]) + +AS_IF([test "$RNANOTFOUND" = "0"], [ + AC_MSG_CHECKING([for compilation with Vienna RNA package headers]) + OLD_CPPFLAGS=$CPPFLAGS + OLD_CXXFLAGS=$CXXFLAGS + OLD_LDFLAGS=$LDFLAGS + AC_LANG_PUSH([C]) + CPPFLAGS="$CPPFLAGS $AM_CXXFLAGS" + LDFLAGS="$LDFLAGS $AM_LDFLAGS" + AS_IF([test "x$HAVE_PKG_CONFIG" = "xyes"], [ + CPPFLAGS="$CPPFLAGS $VRNA_CFLAGS" + LDFLAGS="$LDFLAGS $VRNA_LIBS" + ]) + AC_COMPILE_IFELSE( + [AC_LANG_PROGRAM([[#include ]],[[vrna_md_t tmp; vrna_md_copy(&tmp,&tmp);]])], + [ + AC_MSG_RESULT([yes]) + RNANOTFOUND=0; + ], + [ + AC_MSG_RESULT([no]) + AC_MSG_NOTICE([DEBUG : used CPPFLAGS = $CPPFLAGS]) + AC_MSG_NOTICE([DEBUG : used LDFLAGS = $LDFLAGS]) + RNANOTFOUND=1; + ] + ) + AC_LANG_POP([C]) + CPPFLAGS=$OLD_CPPFLAGS + LDFLAGS=$OLD_LDFLAGS +]) + +# error output if ViennaRNA not found +AS_IF([test "$RNANOTFOUND" = "1"], [ + AC_MSG_NOTICE() + AC_MSG_NOTICE([The Vienna RNA C library version >= $VRNA_REQUIRED_VERSION is required.]) + AC_MSG_NOTICE([ -> It can be obtained from http://www.tbi.univie.ac.at/.]) + AC_MSG_NOTICE() + AS_IF([test "$RNAPATHSET" = "1"], [ + AC_MSG_NOTICE([ -> Can't find the Vienna RNA library in given path '$with_vrna'.]) + ], [ # else + AC_MSG_NOTICE([ -> If installed in a non-standard path, please use '--with-vrna=PREFIX'.]) + ]) + DEPENDENCYNOTFOUND=1; +],[ # else + # register Vienna RNA lib for linking + AS_IF([test "x$HAVE_PKG_CONFIG" = "xyes"], [ + # apply pkg-config options for VRNA package for further build + AM_CXXFLAGS="$AM_CXXFLAGS $VRNA_CFLAGS" + LIBS="$LIBS $VRNA_LIBS" + ], [ # else + # guess that only lib information missing + LIBS="$LIBS -lRNA" + ]) +]) + + +############################################################################### +# END VIENNA CHECK +############################################################################### + +############################################################################### +# FINAL DEPENDENCY CHECK AND EXIT IF NEEDED +############################################################################### + +# error ABORT if on of the libraries was not found +AS_IF([test "$DEPENDENCYNOTFOUND" = "1"], [ + AC_MSG_NOTICE() + AC_MSG_ERROR([Some dependency was not met! See above for errors and relate to './configure --help'.]) +]) + +########################################################################## + +# get available personalities +AC_SUBST([PERSONALITIES],[`grep 'return.*"IntaRNA..*"' ./src/bin/CommandLineParsing.h | grep -o '".*"' | tr -d '"' | tr "\n" " "`]) +########################################################################## + +# Keep strict diagnostics focused on IntaRNA sources. Boost and ViennaRNA +# headers are external dependencies and can contain compiler extensions. +INTARNA_SYSTEM_CPPFLAGS= +for intarna_cppflag in $BOOST_CPPFLAGS $VRNA_CFLAGS; do + case "$intarna_cppflag" in + -I?*) + intarna_include_path=${intarna_cppflag#-I} + INTARNA_SYSTEM_CPPFLAGS="$INTARNA_SYSTEM_CPPFLAGS -isystem $intarna_include_path" + ;; + esac +done + + +# distribute additional compiler and linker flags +# --> set these variables instead of CXXFLAGS or LDFLAGS +AC_SUBST([AM_CXXFLAGS]) +AC_SUBST([AM_LDFLAGS]) +AC_SUBST([INTARNA_SYSTEM_CPPFLAGS]) +AC_SUBST([LIBS]) +AM_CONDITIONAL(enable_pkg_config, [test "x$HAVE_PKG_CONFIG" = "xyes"]) +AM_CONDITIONAL(enable_doxygen, [test "x$HAVE_DOXYGEN" = "xyes"]) + + +# files to generate via autotools (.am or .in source files) +AC_CONFIG_FILES([Makefile]) +AC_CONFIG_FILES([src/Makefile]) +AC_CONFIG_FILES([src/IntaRNA/Makefile]) +AC_CONFIG_FILES([src/IntaRNA/intarna_config.h]) +AC_CONFIG_FILES([src/bin/Makefile]) +AC_CONFIG_FILES([perl/Makefile]) +AC_CONFIG_FILES([R/Makefile]) +AC_CONFIG_FILES([python/Makefile]) +AC_CONFIG_FILES([python/bin/Makefile]) +AC_CONFIG_FILES([python/intarnapvalue/Makefile]) +AC_CONFIG_FILES([python/copomus/Makefile]) +AC_CONFIG_FILES([tests/Makefile]) +AC_CONFIG_FILES([tests/data/Makefile]) +AC_CONFIG_FILES([doc/Makefile]) +AC_CONFIG_FILES([IntaRNA.pc]) + +# generate the final Makefile etc. +AC_OUTPUT + +# get absolute directory paths +eval _prefixdir=$(eval printf "%s" $prefix) +eval _bindir=$(eval printf "%s" $bindir) +eval _libdir=$(eval printf "%s" $libdir) +eval _includedir=$(eval printf "%s" $includedir) + +AC_MSG_NOTICE([ + +====================================== + IntaRNA $PACKAGE_VERSION +====================================== + +Features +-------- + * log coloring : $logColoring + * OpenMP multi-threading : $multithreadingEnabled + * intarnapvalue : $intarnapvalue + +Install Directories for given 'prefix' +-------------------------------------- + * Install prefix : $_prefixdir + * Executables : $_bindir + * Libraries : $_libdir + * Header files : $_includedir + * Python modules : $_pydir + +You can run 'make', 'make tests' and 'make install' now! + +Run 'make install prefix=XYZ' for installation in an alternative directory. + +====================================== +]) diff --git a/doc/local-validation-archive/files/issue-244-validation/source-gcc11-debug/doc/refactor/2-c++23.md b/doc/local-validation-archive/files/issue-244-validation/source-gcc11-debug/doc/refactor/2-c++23.md new file mode 100644 index 00000000..5ce46b43 --- /dev/null +++ b/doc/local-validation-archive/files/issue-244-validation/source-gcc11-debug/doc/refactor/2-c++23.md @@ -0,0 +1,515 @@ +# Phase 2: C++23 migration and performance roadmap + +The C++23 baseline below records the phase-2 migration. The later +[issue #244 audit](5-c++17.md) lowers the current requirement to C++17; +use the README and `configure.ac` for the current build contract. + +This document implements the scope of phase 2 in +[discussion #232](https://github.com/BackofenLab/IntaRNA/discussions/232). +C++23 is a compilation baseline here, not a reason to rewrite working +scientific code. The migration must retain the architecture and modularity +described in `1-current-state.md`, preserve every successful phase-1 test, and +run with both GCC 14 on Linux and the Apple Clang toolchain used for macOS +Bioconda builds. + +The performance-candidate section is a roadmap. The reproducible three-reference +baseline is complete below; each optimization is implemented later on its own +phase-3 branch and measured independently against its scientific parent. + +## Exact scope + +Phase 2 requires: + +1. Work on `refactor-to-c++23`, branched from `refactoring`, and maintain + this file as the change and decision record. +2. Adapt the build and source to strict C++23 using only facilities supported + by GCC 14/libstdc++ and the relevant Apple Clang/libc++ combination. +3. Keep the existing library families and extension points intact, and keep + the CLI/binary-program interface and output schema stable. This is not a + promise of C++ ABI compatibility across different compilers or standard + libraries. +4. Preserve all tests that pass on `refactoring`; a standard-independent bug + found during migration belongs on `refactoring` with a regression first, + then is merged forward. +5. Analyse performance opportunities, distinguish architecture-preserving + from architecture-breaking work, prioritize the preserving work, and define + its benchmark gates. +6. Commit the test-stable migration and open its pull request against + `refactoring`. + +This phase does not change recurrences, energy models, coordinate conventions, +CLI behavior, output schemas, or partition definitions. It also does not +import implementation code from alternate IntaRNA trees; they may be consulted +only as independent lookup material. + +## Baseline and final phase-2 evidence + +The migration originally branched from `refactoring` at `70920c6`, whose last +source fix was `a975c4d`. Standard-independent defects found subsequently were +fixed with regressions on `refactoring` and merged forward. Consequently, the +final phase-1 scientific reference is `ba5f9eb41229b5aa47339b6ee954858ee3d6b93b`, +and the final phase-2 executable source tree is +`7a9517b0587dd308659b61f81b26373ac7a5d0b5`. A later documentation-only commit +does not change that benchmarked executable. + +| Gate | Result | +| --- | --- | +| Initial API suite at `70920c6` | 3,895 assertions in 31 test cases passed | +| Final GCC 14 API suite | 4,281 assertions in 36 test cases passed | +| Command-line golden suite | all 20 cases passed | +| Configured language floor | C++11, strict `noext` mode | +| Final language contract | mandatory strict C++23, `noext` mode | +| Original GCC 14 strict-C++23 build and tests | passed at the compiler-baseline source: GCC 14.2 release and debug on Ubuntu 24.04 | +| Original Apple Clang strict-C++23 build and tests | passed at the compiler-baseline source: Apple Clang 17.0/libomp release on macOS 15 ARM64 | +| Original cross-toolchain output comparison | passed through the same API and 20-case CLI golden suites | +| Original phase-2 performance benchmark | complete at benchmark commit `565fc8617184565d3a33e7c02df63f750f4d26ea` | +| Substantive C++23 follow-up | exact library-feature probe, scoped ownership, allocation/copy removal and one-lookup boundary accumulation | +| Follow-up GCC 14 clean build/test and benchmark | passed; evidence commit `69174eb00354e24427a6ee37c98d4363930114ca` | +| Follow-up PR compiler matrix | GCC 14 release/debug and Apple Clang/libc++ release must be green before merge | + +The bundled `m4/m4-ax_cxx_compile_stdcxx.m4` at the branch point recognizes +only C++11, C++14 and C++17. The existing Linux CI is a single +`ubuntu-latest` Conda build and does not establish either required phase-2 +compiler. These are build-contract gaps, not source-performance findings. + +## Focused C++23 migration contract + +### Build contract + +- Replace the C++11 configure requirement with a mandatory strict C++23 + language check equivalent to + `AX_CXX_COMPILE_STDCXX([23], [noext], [mandatory])`, followed by a compile + probe for the exact library overloads used by IntaRNA: `string::contains`, + `string::resize_and_overwrite` with a `noexcept` callback and + `basic_stringstream::view` including direct stream insertion. +- Update or replace the vendored Autoconf probe so it understands C++23 and + the compiler spellings it tests; prefer a maintained upstream probe over an + ad-hoc local feature list. Merely adding `-std=c++23` to a Makefile is + insufficient because configure must fail clearly on an unsupported compiler. +- Regenerate the repository's Autotools outputs and verify a fresh-clone + `autotools-init.sh -> configure -> make -> make tests -> make install` + path. +- Compile in `-std=c++23`, not `gnu++23`. Source that accidentally relies + on GNU language extensions must be corrected or isolated behind an existing + platform boundary. +- Compile IntaRNA production translation units with `-pedantic-errors`. + Dependency include paths are system includes so an extension in a supported + Boost or ViennaRNA release is not promoted to an IntaRNA build failure. +- Treat GCC 14 with libstdc++ and Apple Clang with libc++ as separate targets. + Compiler acceptance alone is not enough; included Boost and ViennaRNA + headers must compile and link with the same C++ runtime. +- Keep OpenMP feature detection. The required default multithreaded build is + tested with GCC/libgomp and Apple Clang/libomp. A + `--disable-multithreading` build remains a useful supplemental check, not a + second phase-2 compatibility target. +- GNU quadmath remains an optional GCC-only configuration. It is not a + requirement for the Apple Clang gate; the common scientific baseline uses + `Z_type=double`. + +### Source adaptation rules + +1. Make the build accept C++23 first, with no algorithmic change in the same + patch. +2. Apply the smallest compatibility fixes for removed or newly diagnosed + constructs. Keep any required compatibility edits to bundled Catch or + easylogging code isolated from IntaRNA changes. +3. Add explicit standard headers where the old build relied on transitive + includes. Do not combine this with renaming or broad formatting. +4. Preserve public class signatures and ownership semantics. Local RAII is + welcome where it removes a proven lifetime hazard, but a factory-wide + ownership redesign is not part of the migration. +5. Use a C++23 library facility only when both required standard libraries + support it and it makes the code or measured hot path better. Modules, + header units, `std::print`, `std::mdspan`, parallel standard algorithms + and other unevenly implemented facilities are not migration dependencies. +6. Do not replace Boost components solely to advertise newer C++. Replacing a + dependency is separate work with its own compatibility and performance + evidence. +7. Do not enable `-ffast-math`, reassociation, native-only instruction flags, + or other options that relax floating-point or portability semantics. + +The result may still look much like the phase-1 code. That is intentional: +language migration and performance experiments must remain independently +reviewable and benchmarkable. + +### Implemented C++23 modernization + +The follow-up at `7a9517b0587dd308659b61f81b26373ac7a5d0b5` uses the +new language baseline at measured or ownership-sensitive boundaries rather +than spreading syntactic churn through the scientific kernels. + +| Change | Reason and invariant | +| --- | --- | +| `string::resize_and_overwrite` for ViennaRNA constraint buffers | fills the exact requested byte range without value-initializing and then overwriting it; the constraint is resolved before the `noexcept` callback and every byte is written | +| `basic_stringstream::view` at completed-output insertion sites | avoids an intermediate owning `string`; formatted bytes and insertion order are unchanged | +| `string::contains` in RNA alphabet validation | replaces a find/sentinel spelling without changing accepted characters | +| Local `unique_ptr` ownership for ViennaRNA compounds, parameters and ES matrices, and for CLI factory products | closes constructor-throw and early-exit leaks while preserving public factory signatures, destruction dependencies and OpenMP topology | +| Direct sequence `c_str()` use at ViennaRNA calls | removes redundant C allocation and copying; ViennaRNA receives the same null-terminated bases for the duration of each call | +| `try_emplace` for exact-ensemble boundary accumulation | performs one associative lookup; an existing entry is incremented only after insertion fails, preserving the original floating-point addition order | + +Focused regressions cover short-string constraint storage, constrained +RNAplfold and ES paths, reversed constraints and repeated boundary +accumulation. No recurrence, energy model, coordinate mapping, partition, +filter, output schema or public library signature changes. The installed +headers require consumers to select C++23 or newer explicitly; pkg-config does +not force or downgrade a consumer's language dialect. + +The council rejected facilities without a portable or attributable benefit: +`mdspan` is not jointly available in the required GCC 14/libstdc++ and Apple +Clang/libc++ targets; modules would reorganize the public build; `format` and +`print` risk byte-level output changes; and parallel ranges or recurrence +rewrites would mix a scientific algorithm change into the language phase. +Measured matrix-layout, workspace, virtual-dispatch and parallel-aggregation +experiments remain separate performance phases. + +## Scientific and numerical neutrality + +The following contracts are unchanged by C++23. + +| Area | Invariant | Required evidence | +| --- | --- | --- | +| Coordinates | sequence ranges are closed and zero-based internally; reversed-query coordinates remain bijective | existing range/offset tests plus exact interaction coordinates in CLI output | +| MFE arithmetic | `E_type` remains integer centi-kcal/mol; `E_INF/E_MAX` retain headroom and are never used as ordinary energies | exact API assertions and exact MFE/output energy fields | +| Ensemble arithmetic | independent Boltzmann factors multiply and alternative structures add exactly once | ensemble, noLP, filter and seed-extension regressions | +| Floating-point evaluation | recurrence traversal and addition/multiplication order remain unchanged during migration | same-toolchain before/after partition comparison using existing `Z_equal`; every textual difference reported | +| Accessibility | `ED=-RT log(Pu)`, threshold comparison is inclusive, and inaccessible positions never enter returned ranges | accessibility, `outMinPu`, range split and ViennaRNA tests | +| ViennaRNA model | temperature, parameter set, constraints and maximum base-pair span reach the same ViennaRNA calls | ViennaRNA accessibility/ES tests and golden outputs | +| Seeds and tracebacks | seed feasibility, noLP rules, lexicographic tie-breaking and reported base pairs are unchanged | seed-handler, seed-ordering and predictor traceback tests | +| Predictor lifecycle | repeated prediction cannot reuse stale MFE, partition or boundary state | phase-1 seed-extension reuse regression and any new per-family reuse tests | +| Filters and output | `noGUend`, `maxED`, overlap, top-k, tracker and CLI-column behavior remain consistent | API filter tests and all CLI golden cases | +| Parallel execution | thread count may change scheduling, not the accepted scientific result | single-thread versus multithread differential corpus | + +For one compiler and dependency set, deterministic integer, coordinate, +structure, ordering and formatted-output fields must be identical before and +after migration. Cross-platform math libraries can differ in final +floating-point bits; such differences must be parsed, quantified and reported, +not hidden by normalizing the whole output. The in-memory API gate retains the +project's current `Z_equal` contract. A changed interaction set, rank, MFE, +seed, boundary probability, infinity state or filter decision is a failure +even if a scalar partition difference is small. + +No recurrence should be reordered to make a compiler warning disappear. If a +C++23 diagnostic exposes undefined behavior in scientific code, first add a +focused regression and fix the behavior on `refactoring`. + +## Portability validation + +Every job must record the OS and architecture, compiler and standard-library +versions, Boost and ViennaRNA versions, configure arguments, OpenMP runtime, +commit hash, and whether multiprecision is enabled. + +| Environment | Configuration | Required checks | Status | +| --- | --- | --- | --- | +| Ubuntu 24.04, GCC 14.2/libstdc++ | optimized, default OpenMP, strict C++23 | fresh configure, build, complete API/CLI suites, install and consumer checks | passed | +| Ubuntu 24.04, GCC 14.2/libstdc++ | debug, default OpenMP, strict C++23 | build, complete API/CLI suites, install and consumer checks | passed | +| macOS 15 ARM64, Apple Clang 17.0/libc++ and libomp | optimized, default multithreading, strict C++23 | configure, build, complete API/CLI suites, install and consumer checks | passed | + +The Apple Clang version is the C++23-capable version in the Bioconda target +environment and must be recorded rather than inferred from the unversioned +`clang++` name. Both macOS architectures should be covered when Bioconda +publishes both; otherwise the current supported target is the mandatory gate. + +Warnings from IntaRNA sources should be reviewed under both compilers, but +`-Werror` over vendored headers is not a portability criterion. Sanitizer +runs are useful supplementary evidence; they do not replace either required +compiler job. + +## Performance classification + +An optimization is architecture-preserving when the current sequence, +accessibility, energy, seed/helix, predictor, tracker and output boundaries +remain recognizable and extensible; public behavior and recurrence mathematics +do not change. Private storage and loop-local implementation may change. + +An optimization is architecture-breaking when it changes recurrence +ownership, public library interfaces, aggregation semantics, execution +topology, or the way predictor and energy modules compose. Architecture +breaking does not authorize a CLI change: phase 4 still requires the binary +interface to remain stable. + +### Architecture-preserving candidates + +| Priority | Candidate | Expected effect | Main scientific gate | +| --- | --- | --- | --- | +| AP-1 | Hoist loop invariants and cache repeated energy, ED, complementarity, size and offset lookups inside existing DP loops | fewer virtual calls and duplicate table accesses in the dominant nested loops | identical traversal, arithmetic order, MFE/traceback and partition output | +| AP-2 | Reuse and reserve predictor workspaces within one predictor/window configuration; clear only the active matrix bands | fewer allocations, initializations and cache misses | exhaustive reuse tests, because phase 1 found stale seed-extension partition state | +| AP-3 | Replace private general-purpose matrices with contiguous banded/flat storage one predictor at a time | lower memory, better locality and simpler bounds in hot cells | cell-by-cell small-instance differential and unchanged traceback | +| AP-4 | Precompute immutable pair feasibility, reversed-index mappings and decomposed accessibility ranges once per applicable sequence/configuration | removes repeated work and parser-owned mutation; enables safe read sharing | asymmetric target/query, offset, constraint and multi-pair tests | +| AP-5 | Accumulate top-k candidates in thread/window-local buffers and merge deterministically | shorter OpenMP critical sections and better scaling on output-heavy runs | exact ordering/tie behavior and one-thread versus many-thread equality; do not merge overlapping `Zall` domains | +| AP-6 | Trial ViennaRNA window APIs or narrower ES work only where requested ranges permit it | may reduce ViennaRNA preprocessing time and memory | full ED/ES table differential across constraints, span and temperature; discard if ViennaRNA algorithms change results | +| AP-7 | Replace measured Boost/string-conversion or allocation hotspots with portable standard facilities | lower dependency or conversion overhead if profiling proves it material | byte-stable parsing/formatting and both standard libraries | + +AP-1 is intentionally first: it is local, cheap to test and targets the +repeated calls identified in phase 1. Static inspection suggests AP-2 and AP-3 +could address substantial predictor costs, but profiling must establish their +priority; both require stronger lifecycle and traceback gates. AP-4 and +AP-5 prepare safe parallel scaling without redefining the algorithms. AP-6 and +AP-7 are experiments, not assumed improvements. + +`std::ranges`, smart pointers, `constexpr`, `noexcept`, `std::span` or +other modern constructs are not performance work by themselves. They belong +in an optimization branch only when they enable a measured change or remove a +specific correctness/lifetime risk. + +### Architecture-breaking candidates + +| Candidate | Why it breaks the current architecture | Potential benefit | +| --- | --- | --- | +| Stream or factor exact-ensemble boundary ownership instead of retaining all four-boundary values | changes when the recurrence finalizes and exposes boundary partitions | removes the practical `O(n1^2*n2^2)` memory growth | +| Use a private sparse keyed container while retaining every boundary value | may remain preserving if ownership, values and iteration semantics are unchanged; otherwise it is staged here | reduces memory only when the valid boundary set is sparse | +| Introduce scaled or log-domain partition accumulation | changes the numeric representation and algebra used by ensemble recurrences | prevents release-build overflow and extends the usable sequence/energy range | +| Define disjoint ownership for multiple regions and overlapping windows | changes global partition/tracker aggregation semantics | permits scientifically valid global `Zall/Eall` with windowing | +| Introduce an immutable run configuration and task graph for target/query/window work | moves mutable factory state and parallel ownership across CLI/library boundaries | coarse task parallelism, less contention and safer caching | +| Specialize predictor/energy kernels rather than dispatching through virtual interfaces in inner loops | changes the principal modular composition mechanism | inlining, vectorization and removal of repeated virtual dispatch | +| Build a shared tiled/wavefront recurrence engine or accelerator backend | reorganizes DP storage, scheduling and traceback ownership | better locality and possible SIMD/GPU execution | + +These belong after the preserving roadmap and benchmark infrastructure. Any +selected design must still make it straightforward to add predictors, output +handlers, input types and energy/accessibility models. + +## Benchmark gate + +Before the first optimization, create and commit the separate private +repository `BackofenLab/IntaRNA-refactor-benchmark` required by discussion +#232. It must contain the runner, a versioned and extensible/prunable corpus +manifest, schemas, raw JSON artifacts and the generated Markdown comparison. +A GitHub Action must validate and regenerate the report whenever benchmark +JSON is added or changed. The first batch establishes comparable results for +the last commits on `master`, `refactoring` and `refactor-to-c++23`. + +Before AP-1, phase 2 adds small exhaustive/reference oracles for exact MFE, +exact partition enumeration and seeded prediction. They cover LP/noLP, +terminal GU, direct `maxED` partition filtering, bulged and overlapping seeds, +predictor reuse, reversed offsets and exact traceback coordinates. Together +with the phase-1 regressions and the benchmark's one-/many-thread cases, these +form the scientific gate for optimization work. + +The committed corpus is small enough for routine use and is stratified across: + +- short exact MFE and ensemble cases with independently checkable outputs; +- default heuristic seed-extension workloads; +- noLP, terminal-GU, `maxED`, seed-free and constrained cases; +- disabled, base-pair, stream and ViennaRNA accessibility; +- single pair, many target/query pairs, explicit regions and windowing; +- one and multiple threads; and +- output-light prediction, top-k output and tracker-heavy output. + +The initial routine corpus uses independently checkable tiny inputs, individual +biological sRNA/target pairs, a deterministic stratified 64-target throughput +subset and the complete 4,319-target scaling case. Timed cases use one warm-up +and three to seven fresh-process repetitions according to cost, fixed release +flags, explicit thread counts and GNU `time` wall/user/system/maximum-RSS +measurements. Candidate branches add the smallest counter or stage timer needed +to attribute their claimed improvement; the baseline harness does not require +instrumenting the scientific executable. + +For every case the JSON must contain the exact commit and environment metadata, +arguments, raw repetitions, summary statistics with dispersion, peak resident +memory, exit status, raw and canonical output hashes, and a semantic divergence +report. The corpus and build inputs are checksum-pinned. The aggregation action +produces the Markdown comparison requested in the discussion. + +An optimization branch passes only when: + +1. the full compiler/test portability gate remains green; +2. MFE, coordinates, structures, ordering, error behavior and categorical + fields are identical; +3. every floating or ensemble difference satisfies the predeclared + relative/ULP/induced-`Eall` case bounds and leaves ranks, filters, + structures and infinity states unchanged; +4. repeated timing improvement is larger than observed run-to-run noise on a + relevant workload, using the same machine, dependencies and thread count; +5. peak memory does not regress without a documented and worthwhile speed + tradeoff; and +6. results are stored for the exact branch commit. + +Output identity is judged against the branch's scientific parent. Differences +from `master` to `refactoring` caused by the documented phase-1 bug fixes are +expected but remain visible and classified; they are not optimization +regressions. No fixed percentage is declared before measuring the benchmark +noise. A change that merely moves cost between corpus cases is reported as a +tradeoff, not as a general speedup. + +The required private repository is +[BackofenLab/IntaRNA-refactor-benchmark](https://github.com/BackofenLab/IntaRNA-refactor-benchmark). +Evidence commit `565fc8617184565d3a33e7c02df63f750f4d26ea` contains the +checksum-pinned corpus, runner, strict result schema, deterministic aggregator, +GitHub workflows, 23 passing harness tests, raw compressed outputs and the +[generated report](https://github.com/BackofenLab/IntaRNA-refactor-benchmark/blob/master/REPORT.md). +The recorded source references are `master` at +`3b14bc09a7387d85faf7829d31a5839db5ca721c`, `refactoring` at +`ba5f9eb41229b5aa47339b6ee954858ee3d6b93b` and `refactor-to-c++23` at +`8bd1676d1e38ec6461c60e1b832ff3ace9ff77bc`. + +The same-host batch used GCC 14.4.0, Boost 1.85.0 and ViennaRNA 2.7.2 with +environment fingerprint +`52fb1f3e9e0f6e3806593ed89ca02aea5ea8fa2f2fa9544e9dfd53f07bdbd68e`. +Wall times are means plus/minus sample standard deviation; RSS entries are the +three mean peak values for master/refactoring/C++23 in KiB. The three tiny +scientific-oracle cases round to 0.000000 seconds and remain output-hash gates, +so they are omitted from this timing table. + +| Case | master wall (s) | refactoring wall (s) | C++23 wall (s) | mean peak RSS: master/refactoring/C++23 (KiB) | +| --- | ---: | ---: | ---: | ---: | +| `gcvb-phob-exact` | 1.043333 +/- 0.005774 | 1.040000 +/- 0.017321 | 1.050000 +/- 0.026458 | 15,862.7 / 15,920.0 / 15,904.0 | +| `gcvb-st-ilve-seed-exact` | 0.680000 +/- 0.000000 | 0.666667 +/- 0.020817 | 0.703333 +/- 0.061101 | 15,894.7 / 15,584.0 / 15,726.7 | +| `nc000913-stratified64-t1` | 6.000000 +/- 0.085440 | 6.136667 +/- 0.005774 | 6.243333 +/- 0.020817 | 12,944.0 / 12,844.0 / 13,034.7 | +| `nc000913-stratified64-t4` | 1.860000 +/- 0.010000 | 1.883333 +/- 0.015275 | 1.886667 +/- 0.041633 | 25,165.3 / 25,113.3 / 25,154.7 | +| `oxys-fhla-default` | 0.034000 +/- 0.005477 | 0.032000 +/- 0.004472 | 0.030000 +/- 0.000000 | 10,609.6 / 10,638.4 / 10,716.0 | + +The original compiler-baseline migration deliberately contained no algorithmic +optimization, and those measurements do not establish one: changes are within +a few percent and/or the observed dispersion and timer granularity. The +peak-memory envelope is stable. +All deterministic C++23 raw and canonical outputs are identical to +`refactoring`. The four-thread throughput case has schedule-dependent raw row +order (two hashes for `refactoring`, three for C++23), while every run has the +same canonical SHA-256 +`4950ee7d9eb2f9291fc66921b475972717d5bf86cde980404309e3145d5b8624`. +The only differences from `master` are the two expected phase-1 scientific +fixes: heuristic noLP ensemble direct-continuation de-duplication and seed-free +`outMinPu` accessible-range retention. C++23 is identical to `refactoring` for +both corrected cases. + +### Substantive C++23 follow-up benchmark + +The focused modernization was compared on the same host at the immutable +references `321364c56e914e9eff56ddba4846852297fe5c55` (the previous PR head) +and `7a9517b0587dd308659b61f81b26373ac7a5d0b5`, with the former declared as +the scientific parent. Both references were freshly configured, built and +fully tested by +[harness commit `932d5fd210a751d76d0a1fe267049c7642201e77`](https://github.com/BackofenLab/IntaRNA-refactor-benchmark/commit/932d5fd210a751d76d0a1fe267049c7642201e77) +before measurement. The pinned environment used +GCC 14.4.0, Boost 1.85.0 and ViennaRNA 2.7.2 and has fingerprint +`1cfd42e4452d1f83cf2e0aeaac0c41969e63df8c6893f4b8484d72e12f5add53`. +The schema-validated JSON and compressed representative outputs are published +at benchmark evidence commit +[`69174eb00354e24427a6ee37c98d4363930114ca`](https://github.com/BackofenLab/IntaRNA-refactor-benchmark/commit/69174eb00354e24427a6ee37c98d4363930114ca) +and [benchmark PR #2](https://github.com/BackofenLab/IntaRNA-refactor-benchmark/pull/2). + +| Core case | Previous head wall (s) | Modernized wall (s) | Ratio, previous/modernized | +| --- | ---: | ---: | ---: | +| `gcvb-phob-exact` | 1.246249 +/- 0.034565 | 1.240830 +/- 0.030188 | 1.004x | +| `gcvb-st-ilve-seed-exact` | 0.799654 +/- 0.065985 | 0.785018 +/- 0.010079 | 1.019x | +| `nc000913-stratified64-t1` | 6.592438 +/- 0.129549 | 6.450065 +/- 0.060261 | 1.022x | +| `nc000913-stratified64-t4` | 2.045925 +/- 0.024168 | 2.066979 +/- 0.053633 | 0.990x | +| `oxys-fhla-default` | 0.043098 +/- 0.003640 | 0.043656 +/- 0.003142 | 0.987x | + +All eight core cases have identical canonical output hashes. Raw outputs are +byte-identical for the deterministically ordered cases; the four-thread +throughput rows have the expected schedule-dependent raw order on both +references while retaining canonical hash +`4950ee7d9eb2f9291fc66921b475972717d5bf86cde980404309e3145d5b8624`. +Peak RSS is near-stable. The small, mixed timing changes do not prove a general +speedup or a material regression; they show that the allocation/copy and +lookup reductions were obtained without a material performance tradeoff +visible in the core suite. + +The core comparison batch is `cpp23-modernization-20260826`; its parent and +candidate benchmark IDs are respectively +`c8feae0672613c32bec5ee55ce276fc419c815663d9c44e2311fcf4944d6de3e` +and `d0325aad441ff1961daf047b1501578129f4fbd9ba313c239dbd61f89296cd74`. +The targeted `cpp23-output-view-20260826` AP5 batch exercises the output-heavy +four-thread path changed to stream views: + +| AP5 case | Previous head wall (s) | Modernized wall (s) | Ratio | Mean peak RSS, previous/modernized (KiB) | +| --- | ---: | ---: | ---: | ---: | +| `ilve-gcvbst-window-output-heavy-t4` | 0.163162 +/- 0.008610 | 0.164213 +/- 0.004132 | 0.994x | 7,603.2 / 7,526.4 | + +Its parent/candidate benchmark IDs are +`b6a2269b95a187d22d44b1db4790e7225ed6e1b3391f1ca17825fc90a44677d4` +and `87fbe3d084f03c4c7e3607f0d24916cabb4c0d74b2f1a2bdf40c7f4bc63f9ad2`. +Both references have identical raw output hash +`69ecbb617671f863b77301f130b50cdc7718467a4e439e587b7b4a944e28ae54` +and canonical hash +`d286fc91193a02e964fce7cc6e936aeb45fcb12bd401a0405423d4dcff2bf417`. +The 0.6% timing difference is below run-to-run dispersion and establishes +neither a speedup nor a regression; RSS does not regress. + +## Prioritized implementation sequence + +### Executed phase-2 migration + +1. Froze `70920c6` test outputs and environment metadata. +2. Committed the C++23 Autoconf/build requirement and regenerated build + artifacts separately from source adaptations. +3. Committed minimal project and vendored-test compatibility fixes without + changing scientific algorithms. +4. Ran the GCC 14 release and debug matrix with the default OpenMP build. +5. Ran the Apple Clang/libc++ release build with libomp. +6. Compared all API, CLI and installed-binary outputs with the phase-1 + reference; fixed standard-independent defects on `refactoring` first and + merged them forward. +7. Recorded the exact toolchains, tests and same-host benchmark results and + opened the pull request to `refactoring`. +8. Re-audited the result with an independent C++23 language, architecture and + portability council; implemented the focused facilities above; repeated + the clean build, complete tests, local installed-consumer probes and + immutable-ref performance comparison. + +Items 2 and 3 may be split further when a compatibility fix is independently +testable. No performance candidate is folded into those commits. + +### Phase-3 preserving work + +Subject to profiling, the provisional order is AP-1 inner-loop lookup removal, +AP-2 workspace reuse, AP-3 private matrix layout, AP-4 immutable preprocessing, +AP-5 deterministic local aggregation, then the AP-6/AP-7 experiments. Each +candidate gets its own branch rooted at `refactor-to-c++23`, or a documented +subbranch when it depends on a preceding level, and its own benchmark JSON. + +The order is deliberately revisable after the first profiles: measured runtime +and memory decide which hotspot is next. Architecture-breaking candidates are +deferred to phase 4 and must be staged so their scientific and benchmark impact +remains attributable. + +## Verification and change record + +The initial C++23 compatibility migration was intentionally mechanical. The +substantive follow-up is equally narrow but now uses the selected standard +facilities at concrete allocation, copy and lookup boundaries: + +- `configure.ac` now requests mandatory strict C++23 and uses non-obsolete + Autoconf forms; the vendored compiler probe is pinned to Autoconf Archive + serial 25 at upstream commit + `ce7e1be1c4b114ba5eb6dfa8b8253f5ca8522c65`; +- the test-only Catch 1.5.7 header is replaced by official Catch2 2.13.10, + whose pinned header SHA-256 is + `3725c0f0a75f376a5005dde31ead0feb8f7da7507644c201b814443de8355170`; +- project sources use `resize_and_overwrite`, stream `view` and `contains` at + the boundaries listed above, with local RAII for factory/ViennaRNA products, + direct sequence buffers and one-lookup boundary accumulation; +- installed pkg-config metadata now supplies its actual prefix, IntaRNA + library and the configured Boost/ViennaRNA public-header paths; and +- CI defines GCC 14 release/debug and Apple Clang/libomp release jobs, each + with build, install, full tests, CLI smoke, isolated installed-header + compilation and an external pkg-config consumer. + +Two independent exhaustive test oracles were added before optimization work: +the exact unseeded oracle at `1ee6aa0` contributes 227 assertions and the +seeded oracle at `3a83472` contributes 114. +Late standard-independent fixes were regression-tested on `refactoring` and +merged forward: `8fb6304` carries the heuristic-cell state, noLP continuation +and bulged-loop corrections, while `8bd1676` carries the +`OutputHandlerHub::add` signature and maximum-reported-count corrections. + +`autotools-init.sh` completes without obsolete-macro warnings. The original +[compiler matrix](https://github.com/BackofenLab/IntaRNA/actions/runs/31863660953) +passes with GCC 14.2.0 on Ubuntu 24.04 in release and debug configurations and +with Apple Clang 17.0.0 on macOS 15 ARM64 in a release/libomp configuration. +All three original jobs passed build, installation, both API/CLI test programs, +65 standalone installed-header probes and the installed pkg-config consumer. +The shared CLI golden suite supplies a byte-stable cross-toolchain output gate. + +A supplemental +GCC 13.3 run using the provisional strict `-std=c++23` flag passed all 3,895 +API assertions in 31 cases and all 20 CLI cases, but GCC 13 advertises a +pre-final C++23 dialect and is not counted as the required gate. Production +translation units also pass a strict-C++23 syntax scan with GCC 16.1. After the +two exhaustive oracles and forward-merged phase-1 regressions, the final GCC 14 +API gate passes 4,281 assertions in 36 cases; all 20 CLI golden cases pass. +The benchmark above supplies the separate same-host timing, memory and output +evidence; no performance result is inferred from compiler CI. The substantive +follow-up must repeat the required GCC 14 release/debug and Apple Clang/libc++ +jobs on PR #234 before merge; this record does not treat the original matrix as +a substitute for those new-commit checks. diff --git a/doc/local-validation-archive/files/issue-244-validation/source-gcc11-debug/src/IntaRNA/AccessibilityVrna.cpp b/doc/local-validation-archive/files/issue-244-validation/source-gcc11-debug/src/IntaRNA/AccessibilityVrna.cpp new file mode 100644 index 00000000..d2493a8f --- /dev/null +++ b/doc/local-validation-archive/files/issue-244-validation/source-gcc11-debug/src/IntaRNA/AccessibilityVrna.cpp @@ -0,0 +1,507 @@ + +#include "IntaRNA/AccessibilityVrna.h" + +#include +#include +#include +#include +#include +#include +#include + +//// constraint-based ED filling +//extern "C" { +// #include +// #include +// #include +// #include +//} +// +//// RNAup-like ED filling +//extern "C" { +// #include +// #include +// #include +// #include +// #include +//} + +// RNAplfold-like ED filling +extern "C" { + #include + #include + #include + #include + #include + #include + #include + #include +} + +namespace { + +struct VrnaFoldCompoundDeleter { + void operator()( vrna_fold_compound_t * foldCompound ) const noexcept { + if (foldCompound != nullptr) { + vrna_fold_compound_free(foldCompound); + } + } +}; + +using VrnaFoldCompoundPtr = std::unique_ptr; + +std::string +getVrnaConstraint( const IntaRNA::Accessibility & acc ) +{ + const std::size_t sequenceLength = acc.getSequence().size(); + const IntaRNA::AccessibilityConstraint & constraintSpec = acc.getAccConstraint(); + std::string constraint(sequenceLength, '.'); + for (std::size_t i = 0; i < sequenceLength; ++i) { + constraint[i] = constraintSpec.getVrnaDotBracket(i); + } + return constraint; +} + +} + +namespace IntaRNA { + +///////////////////////////////////////////////////////////////////////////// + +AccessibilityVrna::AccessibilityVrna( + const RnaSequence& seq + , const size_t maxLength + , const AccessibilityConstraint * const accConstraint + , const VrnaHandler & vrnaHandler + , const size_t plFoldW + , const double pfScale + ) + : + Accessibility( seq, maxLength, accConstraint ), + edValues( getSequence().size(), getSequence().size(), 0, getMaxLength() ) +{ + // if sequence shows minimal length + if (seq.size() > 4) { + // window-based accessibility computation + fillByRNAplfold(vrnaHandler + , (plFoldW==0? getSequence().size() : std::min(plFoldW,getSequence().size())) + , getAccConstraint().getMaxBpSpan() + , pfScale + ); + } else { + // init ED values for short sequences + for (auto row = edValues.begin1(); row != edValues.end1(); row++) { + for (auto ed = row.begin(); ed != row.end(); ed++) { + *ed = 0; + } + } + } + +} + +///////////////////////////////////////////////////////////////////////////// + +AccessibilityVrna::~AccessibilityVrna() +{ +} + +/////////////////////////////////////////////////////////////////////////////// + +void +AccessibilityVrna:: +callbackForStorage(FLT_OR_DBL *pr, + int pr_size, + int j, // right 3'-end of interval (indexing starting with 1) + int max, + unsigned int type, + void *data) +{ + // check if expected data + if (type & (VRNA_PROBS_WINDOW_UP | VRNA_ANY_LOOP)) { + + // access the storage data + std::pair< AccessibilityVrna*, FLT_OR_DBL > storageRT = *((std::pair< AccessibilityVrna*, FLT_OR_DBL >*)data); + // direct data access for computation + const FLT_OR_DBL RT = storageRT.second; + EdMatrix & edValues = storageRT.first->edValues; + const AccessibilityConstraint & accConstr = storageRT.first->getAccConstraint(); + + // copy unpaired data for all available interval lengths + // but ensure interval does not contain blocked positions + const bool rightEndBlocked = accConstr.isMarkedBlocked(j-1); + for (int l = std::min(j,std::min(pr_size,std::min(max,(int)storageRT.first->getMaxLength()))); l>=1; l--) { + // get unpaired probability + FLT_OR_DBL prob_unpaired = pr[l]; +// TODO: check for [0,1] range and correct if needed (print WARNING) + // get left interval boundary index + int i = j - l + 1; + // check if interval ends are blocked positions + // check if zero before computing its log-value + if (rightEndBlocked || accConstr.isMarkedBlocked(i-1) || (prob_unpaired == 0.0) ) { + // ED value = ED_UPPER_BOUND + edValues(i-1,j-1) = ED_UPPER_BOUND; + } else { + // compute ED value = E(unstructured in [i,j]) - E_all + edValues(i-1,j-1) = std::max( 0., Z_2_E( -RT*Z_type(std::log(prob_unpaired) ))); + } + } + + } else { +#if INTARNA_IN_DEBUG_MODE + LOG( DEBUG ) <<"AccessibilityVrna::callbackForStorage() : getting unexpected data for type " << type; +#endif + } +} + + +/////////////////////////////////////////////////////////////////////////////// + +void +AccessibilityVrna:: +addConstraints( vrna_fold_compound_t & fold_compound, const Accessibility & acc ) +{ + // setup folding constraints + if ( ! acc.getAccConstraint().isEmpty() ) { + + // Build the exact pseudo dot-bracket byte sequence in owned C++ storage. + const std::string structure = getVrnaConstraint(acc); + + // Adding hard constraints from pseudo dot-bracket + unsigned int constraint_options = VRNA_CONSTRAINT_DB_DEFAULT; + // enforce constraints + constraint_options |= VRNA_CONSTRAINT_DB_ENFORCE_BP; + + // add constraint information to the fold compound object + vrna_constraints_add(&fold_compound, structure.c_str(), constraint_options); + + // check if SHAPE reactivity data available + if (!acc.getAccConstraint().getShapeFile().empty()) { + // add SHAPE reactivity data + // add SHAPE data as soft constraints + vrna_constraints_add_SHAPE(&fold_compound, + acc.getAccConstraint().getShapeFile().c_str(), + acc.getAccConstraint().getShapeMethod().c_str(), // shape_method, + acc.getAccConstraint().getShapeConversion().c_str(), // shape_conversion, + 0, // verbose, + VRNA_OPTION_PF | VRNA_OPTION_WINDOW ); + } + } + +} + +/////////////////////////////////////////////////////////////////////////////// + + +void +AccessibilityVrna:: +fillByRNAplfold( const VrnaHandler &vrnaHandler + , const size_t plFoldW + , const size_t plFoldL + , const double pfScale ) +{ +#if INTARNA_MULITHREADING + #pragma omp critical(intarna_omp_logOutput) +#endif + { VLOG(2) <<"computing accessibility via plfold routines...";} + // time logging + TIMED_FUNC_IF(timerObj, VLOG_IS_ON(9)); + +#if INTARNA_IN_DEBUG_MODE + if (plFoldW < 3) { + throw std::runtime_error("AccessibilityVrna::fillByRNAplfold() : plFoldW < 3"); + } +#endif + + // get parameter-specific model + vrna_md_t curModel = vrnaHandler.getModel( plFoldL, plFoldW, pfScale ); + + // RnaSequence owns stable, null-terminated storage for the complete call. + const std::string & sequence = getSequence().asString(); + + // setup folding data + VrnaFoldCompoundPtr foldCompoundOwner( + vrna_fold_compound( sequence.c_str(), &curModel, VRNA_OPTION_PF | VRNA_OPTION_WINDOW ) ); + vrna_fold_compound_t * const fold_compound = foldCompoundOwner.get(); + if (fold_compound == nullptr) { + throw std::runtime_error("AccessibilityVrna::fillByRNAplfold() : vrna_fold_compound() failed"); + } + + // add accessibility constraints + addConstraints( *fold_compound, *this ); + + // provide access to this object to be filled by the callback + // and the normalized temperature for the Boltzmann weight computation + std::pair< AccessibilityVrna*, FLT_OR_DBL > storageRT(this, (FLT_OR_DBL)vrnaHandler.getRT()); + + // call folding and unpaired prob calculation + int retVal = vrna_probs_window( fold_compound, plFoldW, VRNA_PROBS_WINDOW_UP, &callbackForStorage, (void*)(&storageRT)); + // check if computations went fine + if (retVal == 0) { + throw std::runtime_error("AccessibilityVrna::fillByRNAplfold() : vrna_probs_window() returned 0 status ..."); + } + +} + + +/////////////////////////////////////////////////////////////////////////////// +// ALTERNATIVES +/////////////////////////////////////////////////////////////////////////////// +// +// +//E_type +//AccessibilityVrna:: +//calc_ensemble_free_energy( const int start_unfold, const int end_unfold, vrna_exp_param_s * partFoldParams ) +//{ +//#if INTARNA_IN_DEBUG_MODE +// if (start_unfold >= 0 && end_unfold >= 0) { +// checkIndices((size_t)start_unfold, (size_t)end_unfold); +// } else { +// if (start_unfold != -1 || end_unfold != -1) { +// throw std::runtime_error("AccessibilityVienna::calc_ensemble_free_energy : range ["+toString(start_unfold)+","+toString(end_unfold)+"] not allowed"); +// } +// } +//#endif +// +// +// // get sequence length +// int len = (int)getSequence().size(); +// +// // generate structure constraint +// // ('.' = 'unconstrained' and 'x' = 'unstructured/unpaired') +// char c_structure[len+1]; +// c_structure[len] = '\0'; +// if (start_unfold < 0) { +// for (int i=0; i set to upper bound +// edValues(i,j) = ED_UPPER_BOUND; +// } else { +// // compute ED value = E(unstructured in [i,j]) - E_all +// edValues(i,j) = std::max(E_type(0),(calc_ensemble_free_energy(i,j, partFoldParams) - E_all)); +// } +// +// } +// } +// +//} +// +/////////////////////////////////////////////////////////////////////////////// +// +// +//void +//AccessibilityVrna:: +//fillByRNAup( const VrnaHandler &vrnaHandler +// , const size_t plFoldL ) +//{ +//#if INTARNA_MULITHREADING +// #pragma omp critical(intarna_omp_logOutput) +//#endif +// { VLOG(2) <<"computing accessibility via RNAup routines..."; } +// // time logging +// TIMED_FUNC_IF(timerObj, VLOG_IS_ON(9)); +// +// const int seqLength = (int)getSequence().size(); +// +// // get model +// vrna_md_t curModel = vrnaHandler.getModel( plFoldL, seqLength ); +// +// // make model parameters accessible in VRNA2-API +// vrna_md_defaults_reset( &curModel ); +// fold_constrained=1; +// tetra_loop=1; +// noLonelyPairs = curModel.noLP; +// noGU = curModel.noGU; +// no_closingGU = curModel.noGUclosure; +// energy_set = curModel.energy_set; +// +// //////// RNAup-like (VRNA2-API) unpaired probability calculation /////// +// +// char * sequence = (char *) vrna_alloc(sizeof(char) * (seqLength + 1)); +// char * structure = (char *) vrna_alloc(sizeof(char) * (seqLength + 1)); +// for (int i=0; i0; i--) { +// const bool leftSideBlocked = getAccConstraint().isMarkedBlocked(i-1); +// // compute only for region lengths (j-i+1) <= maxLength +// for(int j=i; j<=std::min((int)seqLength,unstr_out->w);j++) +// { +// // check if region ends are blocked +// if (leftSideBlocked || getAccConstraint().isMarkedBlocked(j-1)) { +// // region ends blocked --> set to upper bound +// edValues(i-1,j-1) = ED_UPPER_BOUND; +// } else { +// // compute overall unpaired probability +// double prob_unpaired = +// unstr_out->H[i][j-i]+ +// unstr_out->I[i][j-i]+ +// unstr_out->M[i][j-i]+ +// unstr_out->E[i][j-i]; +// // check if zero before computing its log-value +// if ( prob_unpaired == 0.0 ) { +// // ED value = ED_UPPER_BOUND +// edValues(i-1,j-1) = ED_UPPER_BOUND; +// } else { +// // compute ED value = E(unstructured in [i,j]) - E_all +// edValues(i-1,j-1) = std::max( 0., Z_2_E( -RT*Z_type(std::log(prob_unpaired) ))); +// } +// } +// } +// } +// +// // free data +// free_pu_contrib( unstr_out ); +// +//} +// +///////////////////////////////////////////////////////////////////////////// + +} // namespace diff --git a/doc/local-validation-archive/files/issue-244-validation/source-gcc11-debug/src/IntaRNA/InteractionEnergyVrna.cpp b/doc/local-validation-archive/files/issue-244-validation/source-gcc11-debug/src/IntaRNA/InteractionEnergyVrna.cpp new file mode 100644 index 00000000..23c73868 --- /dev/null +++ b/doc/local-validation-archive/files/issue-244-validation/source-gcc11-debug/src/IntaRNA/InteractionEnergyVrna.cpp @@ -0,0 +1,245 @@ + +#include "IntaRNA/InteractionEnergyVrna.h" +#include "IntaRNA/AccessibilityVrna.h" + +#include +#include +#include +#include +#include +#include + +// ES computation +extern "C" { + #include + #include + #include + #include + #include + #include +} + + +namespace { + +struct VrnaFoldCompoundDeleter { + void operator()( vrna_fold_compound_t * foldCompound ) const noexcept { + if (foldCompound != nullptr) { + vrna_fold_compound_free(foldCompound); + } + } +}; + +struct VrnaParamDeleter { + void operator()( vrna_param_t * params ) const noexcept { + std::free(params); + } +}; + +typedef std::unique_ptr VrnaFoldCompoundPtr; +typedef std::unique_ptr VrnaParamPtr; + +std::string +getVrnaConstraint( const IntaRNA::Accessibility & acc ) +{ + const std::size_t sequenceLength = acc.getSequence().size(); + const IntaRNA::AccessibilityConstraint & constraintSpec = acc.getAccConstraint(); + std::string constraint(sequenceLength, '.'); + for (std::size_t i = 0; i < sequenceLength; ++i) { + constraint[i] = constraintSpec.getVrnaDotBracket(i); + } + return constraint; +} + +} + + + +namespace IntaRNA { + +//////////////////////////////////////////////////////////////////////////// + +InteractionEnergyVrna::InteractionEnergyVrna( + const Accessibility & accS1 + , const ReverseAccessibility & accS2 + , VrnaHandler &vrnaHandler + , const size_t maxInternalLoopSize1 + , const size_t maxInternalLoopSize2 + , const bool initES + , const E_type energyAdd + , const bool energyWithDangles + , const bool internalLoopGU + ) + : + InteractionEnergy(accS1, accS2, maxInternalLoopSize1, maxInternalLoopSize2, energyAdd, energyWithDangles, internalLoopGU) +// get final VRNA folding parameters + , foldModel( vrnaHandler.getModel() ) + , foldParams( vrna_params( &foldModel ) ) + , RT(vrnaHandler.getRT()) + , bpCG( BP_pair[RnaSequence::getCodeForChar('C')][RnaSequence::getCodeForChar('G')] ) + , bpGC( BP_pair[RnaSequence::getCodeForChar('G')][RnaSequence::getCodeForChar('C')] ) + , esValues1(NULL) + , esValues2(NULL) + , Eall1(E_INF) + , Eall2(E_INF) +{ + // Until construction succeeds, local guards own every raw member whose + // class destructor would otherwise not run on an exception. + VrnaParamPtr foldParamsOwner(foldParams); + + vrna_md_defaults_reset( &foldModel ); + + // init ES values if needed + if (initES) { +// 23.11.2017 : should not be relevant anymore +//#if INTARNA_MULITHREADING +// #pragma omp critical(intarna_omp_callingVRNA) +//#endif +// { + // Compute into local owners and publish both matrices atomically only + // after both ViennaRNA calls have completed successfully. + std::unique_ptr esValues1Owner = std::make_unique(); + std::unique_ptr esValues2Owner = std::make_unique(); + // fill ES container + computeES( accS1, *esValues1Owner ); + computeES( accS2, *esValues2Owner ); + esValues1 = esValues1Owner.release(); + esValues2 = esValues2Owner.release(); +// } // omp critical(intarna_omp_callingVRNA) + } + + // The fully constructed object resumes ownership through its unchanged raw + // member; its destructor retains the public/header-level ownership contract. + foldParamsOwner.release(); +} + +//////////////////////////////////////////////////////////////////////////// + +InteractionEnergyVrna::~InteractionEnergyVrna() +{ + // garbage collection + if (foldParams != NULL) { + free(foldParams); + foldParams = NULL; + } + INTARNA_CLEANUP(esValues1); + INTARNA_CLEANUP(esValues2); + +} + + +//////////////////////////////////////////////////////////////////////////// + +void +InteractionEnergyVrna:: +computeES( const Accessibility & acc, InteractionEnergyVrna::EsMatrix & esToFill ) +{ + + // prepare container + esToFill.resize( acc.getSequence().size(), acc.getSequence().size() ); + + // sequence length + const int seqLength = (int)acc.getSequence().size(); + const Z_type RT = getRT(); + + // RnaSequence owns stable, null-terminated storage for the complete call. + const std::string & sequence = acc.getSequence().asString(); + const std::string structureConstraint = getVrnaConstraint(acc); + // prepare folding data + vrna_md_t curModel; + vrna_md_copy( &curModel, &foldModel ); + // set maximal base pair span + curModel.max_bp_span = acc.getAccConstraint().getMaxBpSpan(); + if (curModel.max_bp_span >= (int)acc.getSequence().size()) { + curModel.max_bp_span = -1; + } + // TODO check if VRNA_OPTION_WINDOW reasonable to speedup + VrnaFoldCompoundPtr foldDataOwner( vrna_fold_compound( sequence.c_str(), &curModel, VRNA_OPTION_PF) ); + vrna_fold_compound_t * const foldData = foldDataOwner.get(); + if (foldData == nullptr) { + throw std::runtime_error("InteractionEnergyVrna::computeES() : vrna_fold_compound() failed"); + } + + // Adding hard constraints from pseudo dot-bracket + unsigned int constraint_options = VRNA_CONSTRAINT_DB_DEFAULT; + // enforce constraints + constraint_options |= VRNA_CONSTRAINT_DB_ENFORCE_BP; + + vrna_constraints_add( foldData, structureConstraint.c_str(), constraint_options); + + // compute correct partition function scaling via mfe + FLT_OR_DBL min_free_energy = vrna_mfe( foldData, NULL ); + vrna_exp_params_rescale( foldData, &min_free_energy); + + // compute partition functions + const FLT_OR_DBL ensembleE = vrna_pf( foldData, NULL ); + + if (foldData->exp_matrices == NULL) { + throw std::runtime_error("InteractionEnergyVrna::computeES() : partition functions after computation not available"); + } + if (foldData->exp_matrices->qm == NULL) { + throw std::runtime_error("InteractionEnergyVrna::computeES() : partition functions Qm after computation not available"); + } + // copy ensemble energies of multi loop parts = ES values + FLT_OR_DBL qm_val = 0.0; + const int minLoopSubseqLength = foldModel.min_loop_size + 2; + for (int i=0; iexp_matrices->qm[foldData->iindx[i+1]-j+1]; + if ( Z_equal(Z_type(qm_val), Z_type(0)) ) { + esToFill(i,j) = E_INF; + } else { + // ES energy = -RT*log( Qm ) + esToFill(i,j) = Z_2_E( - RT* Z_type( std::log(qm_val) + +((FLT_OR_DBL)(j-i+1))*std::log(foldData->exp_params->pf_scale))); + } + } + } + } +} + +//////////////////////////////////////////////////////////////////////////// + +E_type +InteractionEnergyVrna:: +computeIntraEall( const Accessibility & acc ) const +{ + + vrna_md_t curModel = foldModel; + // avoid computation of base pair probabilities + curModel.compute_bpp = 0; + + // RnaSequence owns stable, null-terminated storage for the complete call. + const std::string & sequence = acc.getSequence().asString(); + + // setup folding data + VrnaFoldCompoundPtr foldCompoundOwner( vrna_fold_compound( sequence.c_str(), &curModel, VRNA_OPTION_DEFAULT ) ); + vrna_fold_compound_t * const fold_compound = foldCompoundOwner.get(); + if (fold_compound == nullptr) { + throw std::runtime_error("InteractionEnergyVrna::computeIntraEall() : vrna_fold_compound() failed"); + } + + // add accessibility constraints + AccessibilityVrna::addConstraints( *fold_compound, acc ); + + // rescale parameters for Boltzmann factors + vrna_exp_params_rescale(fold_compound, NULL); + + // call PF function to get ensemble energy + E_type ensE = Ekcal_2_E(vrna_pf(fold_compound, NULL)); + + // get partition function from ensemble energy + return ensE; +} + +//////////////////////////////////////////////////////////////////////////// + + +} // namespace diff --git a/doc/local-validation-archive/files/issue-244-validation/source-gcc11-debug/src/IntaRNA/RnaSequence.h b/doc/local-validation-archive/files/issue-244-validation/source-gcc11-debug/src/IntaRNA/RnaSequence.h new file mode 100644 index 00000000..ea1a680b --- /dev/null +++ b/doc/local-validation-archive/files/issue-244-validation/source-gcc11-debug/src/IntaRNA/RnaSequence.h @@ -0,0 +1,655 @@ + +#ifndef INTARNA_RNASEQUENCE_H_ +#define INTARNA_RNASEQUENCE_H_ + +#include +#include +#include + +#include "IntaRNA/general.h" + + +#ifndef VIENNA_RNA_PAIR_MAT_H +#define VIENNA_RNA_PAIR_MAT_H +extern "C" { + #include +} +#endif + +namespace IntaRNA { + +/** + * Represents an RNA sequence within IntaRNA and holds all necessary utility + * functions. + * + * The integer sequence encoding uses the Vienna RNA package encoding functions. + * + * @author Martin Mann 2014 + */ +class RnaSequence { + +public: + + //! type for sequence string representation + typedef std::string String_type; + + //! type for integer encoding of a single sequence letter + //! (currently same as in Vienna package) + typedef short Code_type; + + //! type for sequence integer encoded representation + typedef std::vector CodeSeq_type; + + + /** + * Allowed nucleotide single letter character alphabet according to IUPAC + * codes and their lower case variants. + */ + const static std::string SequenceAlphabetIUPAC; + + /** + * Allowed nucleotide single letter character alphabet (uppercase only) + */ + const static std::string SequenceAlphabet; + + //! constant that serves as a placeholder for the last position within a + //! sequence independent of its real size + static const size_t lastPos; + +public: + + /** + * Creates a sequence. + * @param id the name of the sequence + * @param seqString the nucleotide string encoding the sequence + * @param idxPos0 input/output index of the first sequence position + * @param seqNumber index of the sequence among the respective sequence set in this call. + * relevant for accessibility parsing + */ + RnaSequence(const std::string& id + , const std::string & seqString + , const long idxPos0 = 1 + , const size_t seqNumber = 1 ); + + /** + * Destruction and garbage collection + */ + virtual ~RnaSequence(); + + + ///////////////////// DATA ACCESS ////////////////////////////// + + /** + * Access to the sequence's ID. + * @return the ID of the sequence. + */ + const std::string& + getId() const; + + /** + * Access to the sequence's index among the respective input sequence set of the call arguments + * @return the sequence index among the input sequence set (1-based) + */ + const size_t + getSeqNumber() const; + + /** + * Access to the length of the sequence. + * @return the length of the sequence. + */ + size_t + size() const; + + /** + * Provides the input/output index of a given sequence position. + * @param i the position of interest + * @return the input/output index shifted by idxPos0 + */ + long + getInOutIndex( const size_t i ) const; + + /** + * Provides the internal sequence position of an input/output index given + * that the first position represents idxPos0. + * @param i the input/output index of interest + * @return the internal position, i.e. i shifted by idxPos0 + */ + size_t + getIndex( const long i ) const; + + /** + * Provides the reverse index of a given sequence position. + * @param i the index of interest + * @return the reverse index, i.e. (seq.size()-i-1) + */ + size_t + getReversedIndex( const size_t i ) const; + + /** + * Access to the sequence in character encoding. + * @return the character encoding of the sequence + */ + const String_type& + asString() const; + + /** + * Access to the sequence in integer coding using the Vienna RNA package + * encoding. + * @return the integer encoding of the sequence + */ + const CodeSeq_type& + asCodes() const; + + /** + * Whether or not the sequence contains ambiguous nucleotide encodings. + * @return true if the sequence contains ambiguous nucleotide encodings; + * false otherwise + */ + bool + isAmbiguous() const; + + /** + * Whether or not a specific sequence position shows an ambiguous nucleotide + * encoding. + * @param i the sequence position of interest + * @return true if the sequence contains an ambiguous nucleotide encoding + * at position i; + * false otherwise + */ + bool + isAmbiguous( const size_t i ) const; + + /** + * Checks for equality + * @param rna2 the RnaSequence to compare to + * @return true if both are describing the same sequence with the same identifier. + */ + const bool operator == ( const RnaSequence &rna2 ) const; + + /** + * prints the sequence id and the sequence to stream + * @param out the ostream to write to + * @param rna the RnaSequence object to add + * @return the altered stream out + */ + friend std::ostream& operator<<(std::ostream& out, const RnaSequence& rna); + + + +public: + + ///////////////////// STATIC UTILITY ////////////////////////////// + + /** + * Utility function that converts a string into the internal sequence's + * string representation. This covers an upper case conversion. + * @param seqString sequence string + * @return the internally used sequence string representation + */ + static + String_type + getUpperCase( const std::string & seqString ); + + + /** + * Utility function that converts a sequence's string representation into + * its integer encoding + * + * NOTE: encoding covers only upper case characters! Otherwise an exception + * is raised. + * + * @param seqString the string to encode + * @return the integer encoding + * @throw std::runtime_error if an unsupported nucleotide character is given + */ + static + CodeSeq_type + getCodeForString( const String_type& seqString ); + + /** + * Utility function that converts a nucleotide's char representation into + * its integer encoding. + * + * NOTE: encoding covers only upper case characters! Otherwise an exception + * is raised. + * + * @param nucleotide the char to encode + * @return the integer encoding + * @throw std::runtime_error if an unsupported nucleotide character is given + */ + static + Code_type + getCodeForChar( const char nucleotide ); + + + /** + * Utility function that tests whether or not a given sequence is a valid + * RNA sequence + */ + static + bool + isValidSequence( const std::string& sequence ); + + + /** + * Utility function that tests whether or not a given sequence is a valid + * RNA sequence according to IUPAC encoding + */ + static + bool + isValidSequenceIUPAC( const std::string& sequence ); + + + /** + * Whether or not two positions of two RNAs are complementary, ie. can + * form a base pair + * @param s1 first RNA + * @param s2 second RNA + * @param p1 position within s1 + * @param p2 position within s2 + * @return true if s1[p1] can form a base pair with s2[p2]; false otherwise + */ + static + bool + areComplementary( const RnaSequence & s1, const RnaSequence & s2, + const size_t p1, const size_t p2 ); + + /** + * Whether or not two positions of two RNAs are forming a GU base pair + * @param s1 first RNA + * @param s2 second RNA + * @param p1 position within s1 + * @param p2 position within s2 + * @return true if (s1[p1]=G and s2[p2]==U) or (s1[p1]=U and s2[p2]==G); + * false otherwise + */ + static + bool + isGU( const RnaSequence & s1, const RnaSequence & s2, + const size_t p1, const size_t p2 ); + +protected: + + ///////////////////// DATA MEMBERS ////////////////////////////// + + + //! the locale to use for integer encoding + static std::locale codeLocale; + + //! codes for G and U to check for GU base pairs + static int bpGUcodes[]; + + //! ID of this sequence + std::string id; + + //! The sequence's string representation. + String_type seqString; + + //! Integer encoding of the sequence. + CodeSeq_type seqCode; + + //! Whether or not the sequence contains ambiguous nucleotide encodings + bool ambiguous; + + //! Input/output index of the first sequence position + long idxPos0; + + //! index of the sequence within the overall input (1-based). + //! relevant for reading accessibility files + size_t seqNumber; + +}; + + +///////////////////////////////////////////////////////////////////////////// +///////////////////////////////////////////////////////////////////////////// +///////////////////////////////////////////////////////////////////////////// + + +inline +RnaSequence::RnaSequence( + const std::string & id + , const std::string & seqString + , const long idxPos0 + , const size_t seqNumber ) + : + id(id) + , seqString(getUpperCase(seqString)) + , seqCode(getCodeForString(this->seqString)) + , ambiguous(this->seqString.find('N') != std::string::npos) + , idxPos0(idxPos0) + , seqNumber(seqNumber) +{ +#if INTARNA_IN_DEBUG_MODE + if (id.size() == 0) { + throw std::runtime_error("RnaSequence::RnaSequence : id empty"); + } + if (seqString.size() == 0) { + throw std::runtime_error("RnaSequence::RnaSequence : seqString empty"); + } +#endif + +} + +///////////////////////////////////////////////////////////////////////////// + +inline +RnaSequence::~RnaSequence() +{ +} + +///////////////////////////////////////////////////////////////////////////// + +inline +const std::string& +RnaSequence:: +getId() const +{ + return id; +} + +///////////////////////////////////////////////////////////////////////////// + +inline +const size_t +RnaSequence:: +getSeqNumber() const +{ + return seqNumber; +} + +///////////////////////////////////////////////////////////////////////////// + +inline +size_t +RnaSequence:: +size() const +{ + return seqString.size(); +} + +///////////////////////////////////////////////////////////////////////////// + +inline +long +RnaSequence:: +getInOutIndex( const size_t i ) const +{ +#if INTARNA_IN_DEBUG_MODE + if (i >= size()) { + throw std::runtime_error("RnaSequence::getInOutIndex : index "+toString(i)+" >= length "+toString(size())); + } +#endif + // get in/out index + long p = idxPos0 + (long)i; + // check for -+1 index transition + if (idxPos0<0 && p>=0) { + p++; + } + // final in/out index + return p; +} + +///////////////////////////////////////////////////////////////////////////// + +inline +size_t +RnaSequence:: +getIndex( const long i ) const +{ + // check lower bounds + if (i < idxPos0) { + throw std::runtime_error("RnaSequence::getIndex : index "+toString(i)+" < idxPos0 "+toString(idxPos0)); + } + // shift to internal indexing + size_t p = (size_t)(i - idxPos0); + // check for -+1 index transition + if (idxPos0<0 && i>=0) { + assert(i>0); + p--; + } + // check upper bound + if (p >= size()) { + throw std::runtime_error("RnaSequence::getIndex : index "+toString(i)+" relates to "+toString(p)+" >= length "+toString(size())); + } + // return internal index + return p; +} + +///////////////////////////////////////////////////////////////////////////// + +inline +size_t +RnaSequence:: +getReversedIndex( const size_t i ) const +{ +#if INTARNA_IN_DEBUG_MODE + if (i >= size()) { + throw std::runtime_error("RnaSequence::getReversedIndex : index "+toString(i)+" >= seq.length "+toString(size())); + } +#endif + + return this->size() -i -1; +} + +///////////////////////////////////////////////////////////////////////////// + +inline +const +RnaSequence:: +String_type& +RnaSequence:: +asString() const +{ + return seqString; +} + +///////////////////////////////////////////////////////////////////////////// + +inline +const +RnaSequence:: +CodeSeq_type& +RnaSequence:: +asCodes() const +{ + return seqCode; +} + +///////////////////////////////////////////////////////////////////////////// + +inline +bool +RnaSequence:: +isAmbiguous() const +{ + return ambiguous; +} + +///////////////////////////////////////////////////////////////////////////// + +inline +bool +RnaSequence:: +isAmbiguous( const size_t i ) const +{ + // check for ambiguous nucleotide encoding + return this->seqString.at(i) == 'N'; +} + +///////////////////////////////////////////////////////////////////////////// + +inline +RnaSequence:: +String_type +RnaSequence:: +getUpperCase( const std::string & seqString ) +{ +#if INTARNA_IN_DEBUG_MODE + if (!isValidSequenceIUPAC(seqString)) throw std::runtime_error("RnaSequence::getUpperCase() : the given sequence contains non-IUPAC codes : '"+seqString+"'"); +#endif + + // create container to fill + String_type seqRet(seqString.size(),'_'); + + for (size_t i=0; i=s1.size() || p2>=s2.size()) + throw std::runtime_error("RnaSequence::areComplementary : index positions p1/p2 (" + + toString(p1)+"/"+toString(p2) + + ") are out of bounds s1/s2 (" + + toString(s1.size())+"/"+toString(s2.size()) + +")" + ); +#endif + + // check via VRNA util + return BP_pair[s1.seqCode.at(p1)][s2.seqCode.at(p2)] > 0; +} + +///////////////////////////////////////////////////////////////////////////// + +inline +bool +RnaSequence:: +isGU( const RnaSequence & s1, const RnaSequence & s2, + const size_t p1, const size_t p2 ) +{ +#if INTARNA_IN_DEBUG_MODE + // check if valid positions + if (p1>=s1.size() || p2>=s2.size()) + throw std::runtime_error("RnaSequence::areComplementary : index positions p1/p2 (" + + toString(p1)+"/"+toString(p2) + + ") are out of bounds s1/s2 (" + + toString(s1.size())+"/"+toString(s2.size()) + +")" + ); +#endif + + // get bp code + int bpCode = BP_pair[s1.seqCode.at(p1)][s2.seqCode.at(p2)]; + + // check if GU pair + return bpCode == bpGUcodes[0] || bpCode == bpGUcodes[1] ; +} + +///////////////////////////////////////////////////////////////////////////// + +inline +const bool +RnaSequence:: +operator == ( const RnaSequence &rna2 ) const +{ + return // check if same object (pointer) + this == &rna2 + || ( + // ensure same lengths + this->size() == rna2.size() + // ids are identical (most likely shorter, i.e. faster check) + && this->id == rna2.id + // sequences identical + && this->seqString == rna2.seqString + ); +} + +///////////////////////////////////////////////////////////////////////////// + +} // namespace + + +#endif /* RNASEQUENCE_H_ */ diff --git a/doc/local-validation-archive/files/issue-244-validation/source-gcc11-debug/tests/RnaSequence_test.cpp b/doc/local-validation-archive/files/issue-244-validation/source-gcc11-debug/tests/RnaSequence_test.cpp new file mode 100644 index 00000000..d11842b1 --- /dev/null +++ b/doc/local-validation-archive/files/issue-244-validation/source-gcc11-debug/tests/RnaSequence_test.cpp @@ -0,0 +1,135 @@ + +#include "catch.hpp" + +#undef NDEBUG + +#include "IntaRNA/RnaSequence.h" + +using namespace IntaRNA; + +TEST_CASE( "RnaSequence", "[RNAsequence]" ) { + + // setup easylogging++ stuff if not already done + #include "testEasyLoggingSetup.icc" + + + SECTION( "sequence with non-ACGU" ) { + bool exceptionRaised = ! RnaSequence::isValidSequenceIUPAC("ACUGAC_error"); + REQUIRE( exceptionRaised ); + } + + SECTION( "alphabet normalization and ambiguity at sequence boundaries" ) { + RnaSequence canonical("canonical", "acgt"); + REQUIRE( canonical.asString() == "ACGU" ); + REQUIRE_FALSE( canonical.isAmbiguous() ); + + RnaSequence ambiguous("ambiguous", "nacgtRyn"); + REQUIRE( ambiguous.asString() == "NACGUNNN" ); + REQUIRE( ambiguous.isAmbiguous() ); + REQUIRE( ambiguous.isAmbiguous(0) ); + REQUIRE_FALSE( ambiguous.isAmbiguous(1) ); + REQUIRE( ambiguous.isAmbiguous(ambiguous.size()-1) ); + REQUIRE( ambiguous.asCodes().front() == RnaSequence::getCodeForChar('N') ); + REQUIRE( ambiguous.asCodes().at(1) == RnaSequence::getCodeForChar('A') ); +#if INTARNA_IN_DEBUG_MODE + REQUIRE_THROWS_AS( RnaSequence::getCodeForChar('?'), std::runtime_error ); +#endif + } + + SECTION( "getter", "[RNAsequence]" ) { + + std::string id= "test", seq="AAAUUUGGGCCC"; + + RnaSequence rna(id, seq); + + // check data access + REQUIRE( rna.size() == seq.size() ); + REQUIRE( rna.getId() == id ); + REQUIRE( rna.asString() == seq ); + REQUIRE( rna.asCodes().size() == seq.size() ); + } + + + SECTION( "areComplementary()" ) { + + std::string id= "test", seq="AAAUUUGGGCCC"; + + RnaSequence rna(id, seq); + + // check complementarity + REQUIRE( RnaSequence::areComplementary( rna, rna, 0, 3) ); + REQUIRE_FALSE( RnaSequence::areComplementary( rna, rna, 0, 6) ); + REQUIRE_FALSE( RnaSequence::areComplementary( rna, rna, 0, 9) ); + REQUIRE( RnaSequence::areComplementary( rna, rna, 3, 6) ); + REQUIRE_FALSE( RnaSequence::areComplementary( rna, rna, 3, 9) ); + REQUIRE( RnaSequence::areComplementary( rna, rna, 6, 9) ); + + } + + SECTION( "isGU()" ) { + + std::string id= "test", seq="AAAUUUGGGCCC"; + + RnaSequence rna(id, seq); + + // check complementarity + REQUIRE( RnaSequence::isGU( rna, rna, 3, 6) ); + REQUIRE( RnaSequence::isGU( rna, rna, 7, 4) ); + REQUIRE_FALSE( RnaSequence::isGU( rna, rna, 0, 3) ); + REQUIRE_FALSE( RnaSequence::isGU( rna, rna, 3, 0) ); + REQUIRE_FALSE( RnaSequence::isGU( rna, rna, 0, 6) ); + REQUIRE_FALSE( RnaSequence::isGU( rna, rna, 6, 0) ); + REQUIRE_FALSE( RnaSequence::isGU( rna, rna, 0, 9) ); + REQUIRE_FALSE( RnaSequence::isGU( rna, rna, 9, 0) ); + REQUIRE_FALSE( RnaSequence::isGU( rna, rna, 3, 9) ); + REQUIRE_FALSE( RnaSequence::isGU( rna, rna, 9, 3) ); + REQUIRE_FALSE( RnaSequence::isGU( rna, rna, 6, 9) ); + REQUIRE_FALSE( RnaSequence::isGU( rna, rna, 9, 6) ); + + } + + SECTION( "operator==()" ) { + + std::string id= "test", seq="AAAUUUGGGCCC"; + + RnaSequence rna(id, seq); + + // check equality + REQUIRE( rna == rna ); + REQUIRE( rna == RnaSequence(id,seq) ); + REQUIRE( RnaSequence(id,seq) == rna ); + REQUIRE_FALSE( rna == RnaSequence(id,seq+"A") ); + REQUIRE_FALSE( RnaSequence(id,seq+"A") == rna ); + REQUIRE_FALSE( rna == RnaSequence(id+"2",seq) ); + REQUIRE_FALSE( RnaSequence(id+"2",seq) == rna ); + REQUIRE_FALSE( rna == RnaSequence(id+"2",seq+"A") ); + REQUIRE_FALSE( RnaSequence(id+"2",seq+"A") == rna ); + } + + SECTION( "getInOutIndex() + getIndex()" ) { + + std::string id= "test", seq="AAAUUUGGGCCC"; + + RnaSequence rna(id, seq, 0); + // check index conversion + REQUIRE( rna.getInOutIndex(0) == 0 ); + REQUIRE( rna.getIndex(rna.getInOutIndex(0)) == 0 ); + REQUIRE( rna.getInOutIndex(3) == 3 ); + REQUIRE( rna.getIndex(rna.getInOutIndex(3)) == 3 ); + + rna = RnaSequence(id, seq, -3); + // check index conversion + REQUIRE( rna.getInOutIndex(0) == -3 ); + REQUIRE( rna.getIndex(rna.getInOutIndex(0)) == 0 ); + REQUIRE( rna.getInOutIndex(3) == 1 ); + REQUIRE( rna.getIndex(rna.getInOutIndex(3)) == 3 ); + + rna = RnaSequence(id, seq, 3); + // check index conversion + REQUIRE( rna.getInOutIndex(0) == 3 ); + REQUIRE( rna.getIndex(rna.getInOutIndex(0)) == 0 ); + REQUIRE( rna.getInOutIndex(3) == 6 ); + REQUIRE( rna.getIndex(rna.getInOutIndex(3)) == 3 ); + } + +} diff --git a/doc/local-validation-archive/files/issue-245-validation/benchmark.cpp b/doc/local-validation-archive/files/issue-245-validation/benchmark.cpp new file mode 100644 index 00000000..a6bc245a --- /dev/null +++ b/doc/local-validation-archive/files/issue-245-validation/benchmark.cpp @@ -0,0 +1,55 @@ +// Compare compressed text and binary accessibility I/O after one ViennaRNA fold. +// Usage: accessibility-benchmark SEQUENCE_LENGTH EXISTING_OUTPUT_DIRECTORY +#include +#include +#include +#include +#include +#include + +INITIALIZE_EASYLOGGINGPP + +int main(int argc, char **argv) { + if (argc != 3) return 2; + el::Loggers::reconfigureAllLoggers(el::ConfigurationType::Enabled, "false"); + const size_t n = std::stoull(argv[1]); + std::string sequence; + sequence.reserve(n); + unsigned state = 245; + for (size_t i=0;i> 30]; + } + IntaRNA::RnaSequence rna("benchmark", sequence); + IntaRNA::VrnaHandler vrna(37, "Turner04", false, false); + IntaRNA::AccessibilityVrna source(rna, 100, nullptr, vrna, 150); + using Clock = std::chrono::steady_clock; + for (int run=0;run<3;++run) { + for (bool binary : {false, true}) { + const auto path = std::string(argv[2]) + (binary ? "/cache.agz" : "/cache.txt.gz"); + auto start=Clock::now(); + auto *out=IntaRNA::newOutputStream(path); + if (binary) source.writeBinary(*out); else source.writeRNAplfold_ED_text(*out); + IntaRNA::deleteOutputStream(out); + auto wrote=Clock::now(); + auto *in=IntaRNA::newInputStream(path); + IntaRNA::AccessibilityFromStream read(rna,100,nullptr,*in, + binary ? IntaRNA::AccessibilityFromStream::IntaRNA_Binary : IntaRNA::AccessibilityFromStream::ED_RNAplfold_Text, 1.0); + IntaRNA::deleteInputStream(in); + auto loaded=Clock::now(); + if (read.getMaxLength()!=source.getMaxLength()) return 3; + size_t differences = 0; + IntaRNA::E_type maxDifference = 0; + for(size_t i=0;i(wrote-start).count() << ',' + << std::chrono::duration(loaded-wrote).count() << ',' + << std::filesystem::file_size(path) << ',' << differences << ',' << maxDifference << std::endl; + } + } +} diff --git a/doc/local-validation-archive/files/issue-245-validation/build.sh b/doc/local-validation-archive/files/issue-245-validation/build.sh new file mode 100644 index 00000000..76ac21b3 --- /dev/null +++ b/doc/local-validation-archive/files/issue-245-validation/build.sh @@ -0,0 +1,18 @@ +#!/bin/bash +set -euo pipefail +validation=/home/alex/CodexProjects/IntaRNA/issue-245-validation +toolchain=/home/alex/CodexProjects/IntaRNA/issue-246-validation/toolchain +source_dir=$1 +label=$2 +shift 2 +export PATH="$toolchain/bin:$PATH" +export LD_LIBRARY_PATH="$toolchain/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}" +export PKG_CONFIG_PATH="$toolchain/lib/pkgconfig" +export OPENBLAS_NUM_THREADS=1 +cd "$source_dir" +if test -f autotools-init.sh; then + bash autotools-init.sh > "$validation/logs/$label-autotools.log" 2>&1 +fi +./configure CXX="$toolchain/bin/x86_64-conda-linux-gnu-g++" CC="$toolchain/bin/x86_64-conda-linux-gnu-gcc" CPPFLAGS="-I$toolchain/include" LDFLAGS="-L$toolchain/lib -Wl,-rpath,$toolchain/lib" --with-boost="$toolchain" --with-boost-libdir="$toolchain/lib" --with-vrna="$toolchain" --prefix="$validation/install-$label" "$@" > "$validation/logs/$label-configure.log" 2>&1 +make -j4 > "$validation/logs/$label-build.log" 2>&1 +make tests -j2 > "$validation/logs/$label-check.log" 2>&1 diff --git a/doc/local-validation-archive/files/issue-245-validation/check-headers.sh b/doc/local-validation-archive/files/issue-245-validation/check-headers.sh new file mode 100644 index 00000000..12efce57 --- /dev/null +++ b/doc/local-validation-archive/files/issue-245-validation/check-headers.sh @@ -0,0 +1,9 @@ +#!/bin/bash +set -euo pipefail +validation=/home/alex/CodexProjects/IntaRNA/issue-245-validation +toolchain=/home/alex/CodexProjects/IntaRNA/issue-246-validation/toolchain +for header in Accessibility AccessibilityFromStream; do + printf '#include \nint main() { return 0; }\n' "$header" > "$validation/header.cpp" + "$toolchain/bin/x86_64-conda-linux-gnu-g++" -std=c++23 -Wno-deprecated-declarations \ + -I"$validation/install-release/include" -I"$toolchain/include" -fsyntax-only "$validation/header.cpp" +done diff --git a/doc/local-validation-archive/files/issue-245-validation/debug/doc/benchmarks/accessibility.md b/doc/local-validation-archive/files/issue-245-validation/debug/doc/benchmarks/accessibility.md new file mode 100644 index 00000000..da24fbf4 --- /dev/null +++ b/doc/local-validation-archive/files/issue-245-validation/debug/doc/benchmarks/accessibility.md @@ -0,0 +1,29 @@ +# Accessibility I/O benchmark + +This benchmark isolates writing and reading the same accessibility matrix as +RNAplfold-style gzip-compressed ED text and as a binary `.agz` archive. It folds +one deterministic pseudorandom RNA with ViennaRNA (37 C, Turner04, folding window +150, interaction length 100), then alternates formats for three trials. Folding +and verification are outside the timed sections. Every loaded ED cell, including +the dangling-end band, is compared to the computed matrix. Binary input must +match exactly; text conversion differences are reported. + +Build and install IntaRNA, then compile against that installation: + +```bash +export PKG_CONFIG_PATH=/path/to/intarna-install/lib/pkgconfig +c++ -std=c++23 -O3 doc/benchmarks/accessibility.cpp \ + $(pkg-config --cflags --libs IntaRNA) -leasylogging \ + -o accessibility-benchmark +mkdir -p accessibility-benchmark-output +./accessibility-benchmark 100000 accessibility-benchmark-output +``` + +Use the same compiler, dependency include/library paths and runtime library +paths as the IntaRNA build. With dependencies outside system paths, additional +`-L` and `-Wl,-rpath` flags may be necessary. Output columns are +`trial,format,write_seconds,read_seconds,compressed_bytes,different_cells,max_ED_difference`. +ED differences are measured in internal units (hundredths of kcal/mol). Files are overwritten +between trials; the output directory must already exist. These timings include +compression and stream closing, but not an `fsync` to durable storage. Results +depend on sequence, bandwidth, compressor, filesystem and machine load. diff --git a/doc/local-validation-archive/files/issue-245-validation/header.cpp b/doc/local-validation-archive/files/issue-245-validation/header.cpp new file mode 100644 index 00000000..e1a55c22 --- /dev/null +++ b/doc/local-validation-archive/files/issue-245-validation/header.cpp @@ -0,0 +1,2 @@ +#include +int main() { return 0; } diff --git a/doc/local-validation-archive/files/issue-245-validation/run-benchmark.sh b/doc/local-validation-archive/files/issue-245-validation/run-benchmark.sh new file mode 100644 index 00000000..7cdca824 --- /dev/null +++ b/doc/local-validation-archive/files/issue-245-validation/run-benchmark.sh @@ -0,0 +1,11 @@ +#!/bin/bash +set -euo pipefail +validation=/home/alex/CodexProjects/IntaRNA/issue-245-validation +toolchain=/home/alex/CodexProjects/IntaRNA/issue-246-validation/toolchain +export LD_LIBRARY_PATH="$toolchain/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}" +export PKG_CONFIG_PATH="$validation/install-release/lib/pkgconfig" +"$toolchain/bin/x86_64-conda-linux-gnu-g++" -std=c++23 -O3 \ + "$validation/benchmark.cpp" -L"$toolchain/lib" -Wl,-rpath,"$toolchain/lib" \ + $(pkg-config --cflags --libs IntaRNA) -L"$validation/install-release/lib" -leasylogging \ + -o "$validation/benchmark" +"$validation/benchmark" 100000 "$validation" > "$validation/benchmark.csv" diff --git a/doc/local-validation-archive/files/issue-246-validation/align-matrix-style.py b/doc/local-validation-archive/files/issue-246-validation/align-matrix-style.py new file mode 100644 index 00000000..fb76e159 --- /dev/null +++ b/doc/local-validation-archive/files/issue-246-validation/align-matrix-style.py @@ -0,0 +1,95 @@ +from pathlib import Path +import re +import subprocess +p=Path('/home/alex/CodexProjects/IntaRNA/pr-worktrees/issue-246/src/IntaRNA/Matrix.h') +s=subprocess.check_output(["git","show","HEAD:src/IntaRNA/Matrix.h"],cwd=p.parents[2],text=True) +common={ + 'size1': 'Number of logical rows.\n@return logical row count', + 'size2': 'Number of logical columns.\n@return logical column count', + 'storageSize': 'Number of allocated elements, including any band padding.\n@return stored element count', + 'clear': 'Reset stored cells to default values without changing the shape.', + 'swap': 'Exchange shape and owned storage.\n@param other matrix to exchange with', + 'operator=': 'Take ownership of another matrix; self-move leaves this matrix unchanged.\n@param other matrix whose storage is moved; emptied on a non-self move\n@return this matrix', +} +for cls in ('Matrix','UpperTriangularMatrix','UpperBandedMatrix'): + begin=s.index('template\nclass '+cls+' {') + end=s.index('\n};',begin)+3 + block=s[begin:end] + pattern=r'^\t(?:static )?(?:std::size_t |const T &|T &|void |'+cls+r' &)?(?:'+cls+r'|operator\(\)|operator=|size1|size2|storageSize|clear|swap|resize|offset|count|width)\(' + definitions=[] + edits=[] + for match in re.finditer(pattern,block,re.M): + start=match.start() + # For operator(), the first parentheses belong to the operator name. + open_paren=block.index('(',start) + if 'operator()' in match.group(): open_paren=block.index('(',open_paren+2) + depth=1 + i=open_paren+1 + while depth: + depth += (block[i]=='(')-(block[i]==')') + i+=1 + tail=block[i:] + if tail.lstrip().startswith('= default;'): continue + body_start=block.index('{',i) + # A semicolon before the next body indicates a declaration/defaulted method. + if ';' in block[i:body_start]: continue + depth=1 + j=body_start+1 + while depth: + depth += (block[j]=='{')-(block[j]=='}') + j+=1 + head=block[start:body_start].strip() + body="\n".join(line[1:] if line.startswith("\t") else line for line in block[body_start:j].split("\n")) + constructor=head.startswith(cls+'(') + declaration=re.split(r'(?::'+cls+'(',1) + else: + if head.startswith(cls+' &'): head=head.replace(cls+' &',cls+' &',1) + head=head.replace(name+'(',cls+'::'+name+'(',1) + definitions.append('template\ninline\n'+head+'\n'+body+'\n') + assert len(definitions)==({'Matrix':11,'UpperTriangularMatrix':13,'UpperBandedMatrix':12}[cls]),(cls,len(definitions)) + for a,b,replacement in reversed(edits): block=block[:a]+replacement+block[b:] + block=block.replace('\t'+cls+'() = default;', '\t/** Construct an empty matrix. */\n\t'+cls+'() = default;') + block=block.replace('\t'+cls+'(const '+cls+' &) = default;', '\t/** Copy the shape and values into independent storage. */\n\t'+cls+'(const '+cls+' &) = default;') + block=block.replace('\t'+cls+' &operator=(const '+cls+' &) = default;', '\t/** Copy the shape and values into independent storage.\n\t * @return this matrix\n\t */\n\t'+cls+' &operator=(const '+cls+' &) = default;') + s=s[:begin]+block+'\n\n'+'\n'.join(definitions)+s[end:] +s=s.replace('\tstd::vector values;', '\t//! Owned contiguous storage.\n\tstd::vector values;') +s=s.replace('\tstd::size_t rows = 0, columns = 0;', '\t//! Logical dimensions.\n\tstd::size_t rows = 0, columns = 0;') +s=s.replace('\tstd::size_t n = 0;', '\t//! Logical row and column count.\n\tstd::size_t n = 0;') +s=s.replace('\tMatrix band;', '\t//! Physical rows of the upper band, including row-end padding.\n\tMatrix band;') +s=s.replace('\tstd::size_t columns = 0;', '\t//! Logical column count, independent of stored band width.\n\tstd::size_t columns = 0;') +s=s.replace('\tusing value_type = T;', '\t//! Type of one stored cell.\n\tusing value_type = T;') +p.write_text(s) diff --git a/doc/local-validation-archive/files/issue-246-validation/audit-distribution/intaRNA-3.4.1/doc/Makefile.am b/doc/local-validation-archive/files/issue-246-validation/audit-distribution/intaRNA-3.4.1/doc/Makefile.am new file mode 100644 index 00000000..0e139915 --- /dev/null +++ b/doc/local-validation-archive/files/issue-246-validation/audit-distribution/intaRNA-3.4.1/doc/Makefile.am @@ -0,0 +1,19 @@ + +################################################################ +# extra files to be added to distribution +################################################################ + +EXTRA_DIST = \ + mdspan-storage.md \ + mdspan-storage-samples.tsv \ + handson/fhlA.fasta \ + handson/OxyS.fasta \ + conda.txt \ + doxygen.cfg \ + latex-deps/adjcalc.sty \ + latex-deps/adjustbox.sty \ + latex-deps/collectbox.sty \ + latex-deps/tc-pgf.def \ + latex-deps/tocloft.sty \ + latex-deps/trimclip.sty \ + latex-deps/xtab.sty diff --git a/doc/local-validation-archive/files/issue-246-validation/build-gcc14-clean.sh b/doc/local-validation-archive/files/issue-246-validation/build-gcc14-clean.sh new file mode 100644 index 00000000..37c52b91 --- /dev/null +++ b/doc/local-validation-archive/files/issue-246-validation/build-gcc14-clean.sh @@ -0,0 +1,19 @@ +#!/bin/bash +set -euo pipefail +validation=/home/alex/CodexProjects/IntaRNA/issue-246-validation +compiler=$validation/toolchain-gcc14 +deps=$validation/toolchain +source_dir=$1 +label=$2 +shift 2 +export PATH="$compiler/bin:$deps/bin:$PATH" +export LD_LIBRARY_PATH="$deps/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}" +export PKG_CONFIG_PATH="$deps/lib/pkgconfig" +export OPENBLAS_NUM_THREADS=1 +cd "$source_dir" +bash autotools-init.sh > "$validation/logs/$label-autotools.log" 2>&1 +./configure CXX="$compiler/bin/x86_64-conda-linux-gnu-g++" CC="$compiler/bin/x86_64-conda-linux-gnu-gcc" CPPFLAGS="-I$deps/include" LDFLAGS="-L$deps/lib -Wl,-rpath,$deps/lib" --with-boost="$deps" --with-boost-libdir="$deps/lib" --with-vrna="$deps" --prefix="$validation/install-$label" "$@" > "$validation/logs/$label-configure.log" 2>&1 +make clean > "$validation/logs/$label-clean.log" 2>&1 +make -j4 > "$validation/logs/$label-build.log" 2>&1 +make -j4 check > "$validation/logs/$label-check.log" 2>&1 +make install > "$validation/logs/$label-install.log" 2>&1 diff --git a/doc/local-validation-archive/files/issue-246-validation/build-gcc14.sh b/doc/local-validation-archive/files/issue-246-validation/build-gcc14.sh new file mode 100644 index 00000000..a427906b --- /dev/null +++ b/doc/local-validation-archive/files/issue-246-validation/build-gcc14.sh @@ -0,0 +1,18 @@ +#!/bin/bash +set -euo pipefail +validation=/home/alex/CodexProjects/IntaRNA/issue-246-validation +compiler=$validation/toolchain-gcc14 +deps=$validation/toolchain +source_dir=$1 +label=$2 +shift 2 +export PATH="$compiler/bin:$deps/bin:$PATH" +export LD_LIBRARY_PATH="$deps/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}" +export PKG_CONFIG_PATH="$deps/lib/pkgconfig" +export OPENBLAS_NUM_THREADS=1 +cd "$source_dir" +bash autotools-init.sh > "$validation/logs/$label-autotools.log" 2>&1 +./configure CXX="$compiler/bin/x86_64-conda-linux-gnu-g++" CC="$compiler/bin/x86_64-conda-linux-gnu-gcc" CPPFLAGS="-I$deps/include" LDFLAGS="-L$deps/lib -Wl,-rpath,$deps/lib" --with-boost="$deps" --with-boost-libdir="$deps/lib" --with-vrna="$deps" --prefix="$validation/install-$label" "$@" > "$validation/logs/$label-configure.log" 2>&1 +make -j4 > "$validation/logs/$label-build.log" 2>&1 +make -j4 check > "$validation/logs/$label-check.log" 2>&1 +make install > "$validation/logs/$label-install.log" 2>&1 diff --git a/doc/local-validation-archive/files/issue-246-validation/build.sh b/doc/local-validation-archive/files/issue-246-validation/build.sh new file mode 100644 index 00000000..61fb6c1c --- /dev/null +++ b/doc/local-validation-archive/files/issue-246-validation/build.sh @@ -0,0 +1,16 @@ +#!/bin/bash +set -euo pipefail +validation=/home/alex/CodexProjects/IntaRNA/issue-246-validation +toolchain=$validation/toolchain +source_dir=$1 +label=$2 +shift 2 +export PATH="$toolchain/bin:$PATH" +export LD_LIBRARY_PATH="$toolchain/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}" +export PKG_CONFIG_PATH="$toolchain/lib/pkgconfig" +export OPENBLAS_NUM_THREADS=1 +cd "$source_dir" +bash autotools-init.sh > "$validation/logs/$label-autotools.log" 2>&1 +./configure CXX="$toolchain/bin/x86_64-conda-linux-gnu-g++" CC="$toolchain/bin/x86_64-conda-linux-gnu-gcc" CPPFLAGS="-I$toolchain/include" LDFLAGS="-L$toolchain/lib -Wl,-rpath,$toolchain/lib" --with-boost="$toolchain" --with-boost-libdir="$toolchain/lib" --with-vrna="$toolchain" --prefix="$validation/install-$label" "$@" > "$validation/logs/$label-configure.log" 2>&1 +make -j4 > "$validation/logs/$label-build.log" 2>&1 +make -j4 check > "$validation/logs/$label-check.log" 2>&1 diff --git a/doc/local-validation-archive/files/issue-246-validation/check-mdspan-configure.py b/doc/local-validation-archive/files/issue-246-validation/check-mdspan-configure.py new file mode 100644 index 00000000..a6a99488 --- /dev/null +++ b/doc/local-validation-archive/files/issue-246-validation/check-mdspan-configure.py @@ -0,0 +1,39 @@ +from pathlib import Path +import os +import subprocess + +root=Path(__file__).resolve().parent +source=root/'mdspan-probe-source' +deps=root/'toolchain' +env=dict(os.environ,PKG_CONFIG_PATH=str(deps/'lib/pkgconfig'),LD_LIBRARY_PATH=str(deps/'lib'),OPENBLAS_NUM_THREADS='1') +cases=[('gcc14-auto','toolchain-gcc14','auto',0),('gcc14-std','toolchain-gcc14','std',None),('gcc16-auto','toolchain','auto',1),('gcc16-kokkos','toolchain','kokkos',0),('invalid-backend','toolchain','invalid',None)] +for label,compiler,backend,expected in cases: + build=root/('probe-'+label) + build.mkdir() + args=[str(source/'configure'),f'CXX={root/compiler}/bin/x86_64-conda-linux-gnu-g++',f'CC={root/compiler}/bin/x86_64-conda-linux-gnu-gcc',f'CPPFLAGS=-I{deps}/include',f'LDFLAGS=-L{deps}/lib -Wl,-rpath,{deps}/lib',f'--with-boost={deps}',f'--with-boost-libdir={deps}/lib',f'--with-vrna={deps}',f'--with-mdspan={backend}'] + log=root/'logs'/f'probe-{label}.log' + with log.open('w') as output: + result=subprocess.run(args,cwd=build,env=env,stdout=output,stderr=subprocess.STDOUT) + if expected is None: + assert result.returncode!=0, label + assert ('--with-mdspan=std requires' if backend=='std' else '--with-mdspan must be') in log.read_text(), label + else: + assert result.returncode==0,(label,log) + for header in ('src/config.h','src/IntaRNA/intarna_config.h'): + assert f'#define INTARNA_USE_STD_MDSPAN {expected}' in (build/header).read_text(),(label,header) + print(f'{label}: expected outcome confirmed',flush=True) +# Confirm the fallback probe rejects an unusable bundled tree with its own diagnostic. +headers=source/'src/mdspan' +hidden=source/'src/mdspan-unavailable' +headers.rename(hidden) +try: + build=root/'probe-missing-bundled' + build.mkdir() + args[-1]='--with-mdspan=kokkos' + log=root/'logs/probe-missing-bundled.log' + with log.open('w') as output: + result=subprocess.run(args,cwd=build,env=env,stdout=output,stderr=subprocess.STDOUT) + assert result.returncode!=0 and 'bundled Kokkos mdspan headers do not compile' in log.read_text() + print('missing-bundled: expected diagnostic confirmed',flush=True) +finally: + hidden.rename(headers) diff --git a/doc/local-validation-archive/files/issue-246-validation/consumer.cpp b/doc/local-validation-archive/files/issue-246-validation/consumer.cpp new file mode 100644 index 00000000..e77a7627 --- /dev/null +++ b/doc/local-validation-archive/files/issue-246-validation/consumer.cpp @@ -0,0 +1,11 @@ +#include +#include +#include +int main() { + IntaRNA::Matrix dense(2, 3, 7); + IntaRNA::UpperBandedMatrix band(3, 3, 0, 1); + band(1, 2) = dense(1, 2); + IntaRNA::NussinovHandler::Z2dMatrix triangle(3, 3); + triangle(0, 2) = band(1, 2); + return triangle(0, 2) == 7 ? 0 : 1; +} diff --git a/doc/local-validation-archive/files/issue-246-validation/debug/src/IntaRNA/Matrix.h b/doc/local-validation-archive/files/issue-246-validation/debug/src/IntaRNA/Matrix.h new file mode 100644 index 00000000..1ee1854a --- /dev/null +++ b/doc/local-validation-archive/files/issue-246-validation/debug/src/IntaRNA/Matrix.h @@ -0,0 +1,218 @@ +#ifndef INTARNA_MATRIX_H_ +#define INTARNA_MATRIX_H_ + +#include +#include +#include +#include +#include +#include +#include +#include + +namespace IntaRNA { + +namespace matrix_detail { +inline std::size_t product(std::size_t rows, std::size_t columns) { + if (columns != 0 && rows > std::numeric_limits::max() / columns) + throw std::length_error("matrix dimensions overflow"); + return rows * columns; +} +inline std::size_t triangle(std::size_t n) { + if (n == std::numeric_limits::max()) + throw std::length_error("matrix dimensions overflow"); + return n % 2 == 0 ? product(n / 2, n + 1) : product(n, (n + 1) / 2); +} +} + +/** Owning row-major matrix. Views are constructed on access, so copies, moves + * and resizes cannot leave a cached mdspan pointing into another allocation. + * resize preserves the overlapping rectangle unless preserve=false is passed. + * New cells are value-initialized; clear resets cells without changing shape. + */ +template +class Matrix { + std::vector values; + std::size_t rows = 0, columns = 0; + using Extents = std::dextents; +public: + using value_type = T; + Matrix() = default; + Matrix(std::size_t rows, std::size_t columns, const T &value = T{}) + : values(matrix_detail::product(rows, columns), value), rows(rows), columns(columns) {} + Matrix(const Matrix &) = default; + Matrix &operator=(const Matrix &) = default; + Matrix(Matrix &&other) noexcept : values(std::move(other.values)), + rows(std::exchange(other.rows, 0)), columns(std::exchange(other.columns, 0)) {} + Matrix &operator=(Matrix &&other) noexcept { + if (this != &other) { + Matrix moved(std::move(other)); + swap(moved); + } + return *this; + } + std::size_t size1() const noexcept { return rows; } + std::size_t size2() const noexcept { return columns; } + std::size_t storageSize() const noexcept { return values.size(); } + T &operator()(std::size_t i, std::size_t j) { + assert(i < rows && j < columns); + return std::mdspan(values.data(), rows, columns)[i, j]; + } + const T &operator()(std::size_t i, std::size_t j) const { + assert(i < rows && j < columns); + return std::mdspan(values.data(), rows, columns)[i, j]; + } + void clear() { std::fill(values.begin(), values.end(), T{}); } + void swap(Matrix &other) noexcept { + values.swap(other.values); + std::swap(rows, other.rows); + std::swap(columns, other.columns); + } + void resize(std::size_t newRows, std::size_t newColumns, bool preserve = true) { + if (rows == newRows && columns == newColumns) return; + if (!preserve) { + values.resize(matrix_detail::product(newRows, newColumns)); + rows = newRows; + columns = newColumns; + return; + } + Matrix next(newRows, newColumns); + for (std::size_t i = 0; i < std::min(rows, newRows); ++i) + for (std::size_t j = 0; j < std::min(columns, newColumns); ++j) + next(i, j) = (*this)(i, j); + swap(next); + } +}; + +/** Square upper-triangular matrix with exactly n*(n+1)/2 stored cells. + * Rows are packed consecutively. A one-dimensional mdspan addresses the + * packed storage because triangular row lengths are not an affine 2D layout. + * Const access below the diagonal returns zero; writes require i <= j. + */ +template +class UpperTriangularMatrix { + std::vector values; + std::size_t n = 0; + using Extents = std::dextents; + std::size_t offset(std::size_t i, std::size_t j) const noexcept { + const auto remaining = n - i; + // Constructor has already checked that every triangular size fits. + const auto tail = remaining % 2 == 0 + ? (remaining / 2) * (remaining + 1) : remaining * ((remaining + 1) / 2); + return values.size() - tail + j - i; + } + static std::size_t count(std::size_t rows, std::size_t columns) { + if (rows != columns) throw std::invalid_argument("upper-triangular matrix must be square"); + return matrix_detail::triangle(rows); + } +public: + using value_type = T; + UpperTriangularMatrix() = default; + UpperTriangularMatrix(std::size_t rows, std::size_t columns) + : values(count(rows, columns)), n(rows) {} + UpperTriangularMatrix(const UpperTriangularMatrix &) = default; + UpperTriangularMatrix &operator=(const UpperTriangularMatrix &) = default; + UpperTriangularMatrix(UpperTriangularMatrix &&other) noexcept + : values(std::move(other.values)), n(std::exchange(other.n, 0)) {} + UpperTriangularMatrix &operator=(UpperTriangularMatrix &&other) noexcept { + if (this != &other) { + UpperTriangularMatrix moved(std::move(other)); + swap(moved); + } + return *this; + } + std::size_t size1() const noexcept { return n; } + std::size_t size2() const noexcept { return n; } + std::size_t storageSize() const noexcept { return values.size(); } + T &operator()(std::size_t i, std::size_t j) { + assert(i <= j && j < n); + return std::mdspan(values.data(), values.size())[offset(i, j)]; + } + const T &operator()(std::size_t i, std::size_t j) const { + assert(i < n && j < n); + static const T zero{}; + return i > j ? zero : std::mdspan(values.data(), values.size())[offset(i, j)]; + } + void clear() { std::fill(values.begin(), values.end(), T{}); } + void swap(UpperTriangularMatrix &other) noexcept { + values.swap(other.values); + std::swap(n, other.n); + } + void resize(std::size_t rows, std::size_t columns, bool preserve = true) { + const auto cells = count(rows, columns); + if (rows == n) return; + if (!preserve) { + values.resize(cells); + n = rows; + return; + } + UpperTriangularMatrix next(rows, columns); + for (std::size_t i = 0; i < std::min(n, rows); ++i) + for (std::size_t j = i; j < std::min(n, rows); ++j) + next(i, j) = (*this)(i, j); + swap(next); + } +}; + +/** Upper band including the diagonal and 'upper' superdiagonals. + * Stores rows * min(columns, upper+1) cells, never a dense square for a + * narrow band. Logical (i,j) maps to mdspan[i,j-i] in the owning Matrix. + * Const access outside the band returns zero; writes must be in the band. + */ +template +class UpperBandedMatrix { + Matrix band; + std::size_t columns = 0; + static std::size_t width(std::size_t columns, std::size_t lower, std::size_t upper) { + if (lower != 0) throw std::invalid_argument("upper-banded matrix requires lower=0"); + return upper >= columns ? columns : upper + 1; + } +public: + using value_type = T; + UpperBandedMatrix() = default; + UpperBandedMatrix(std::size_t rows, std::size_t columns, std::size_t lower, std::size_t upper) + : band(rows, width(columns, lower, upper)), columns(columns) {} + UpperBandedMatrix(const UpperBandedMatrix &) = default; + UpperBandedMatrix &operator=(const UpperBandedMatrix &) = default; + UpperBandedMatrix(UpperBandedMatrix &&other) noexcept + : band(std::move(other.band)), columns(std::exchange(other.columns, 0)) {} + UpperBandedMatrix &operator=(UpperBandedMatrix &&other) noexcept { + if (this != &other) { + UpperBandedMatrix moved(std::move(other)); + swap(moved); + } + return *this; + } + std::size_t size1() const noexcept { return band.size1(); } + std::size_t size2() const noexcept { return columns; } + std::size_t storageSize() const noexcept { return band.storageSize(); } + T &operator()(std::size_t i, std::size_t j) { + assert(i < size1() && j < columns && i <= j && j - i < band.size2()); + return band(i, j - i); + } + const T &operator()(std::size_t i, std::size_t j) const { + assert(i < size1() && j < columns); + static const T zero{}; + return i > j || j - i >= band.size2() ? zero : band(i, j - i); + } + void clear() { band.clear(); } + void swap(UpperBandedMatrix &other) noexcept { + band.swap(other.band); + std::swap(columns, other.columns); + } + void resize(std::size_t rows, std::size_t newColumns, std::size_t lower, + std::size_t upper, bool preserve = true) { + UpperBandedMatrix next(rows, newColumns, lower, upper); + if (preserve) { + for (std::size_t i = 0; i < std::min(size1(), rows); ++i) + for (std::size_t d = 0; d < std::min(band.size2(), next.band.size2()) + && i < std::min(columns, newColumns) + && d < std::min(columns, newColumns) - i; ++d) + next.band(i, d) = band(i, d); + } + swap(next); + } +}; + +} // namespace IntaRNA +#endif diff --git a/doc/local-validation-archive/files/issue-246-validation/debug/tests/benchmark/compare-matrix-storage.py b/doc/local-validation-archive/files/issue-246-validation/debug/tests/benchmark/compare-matrix-storage.py new file mode 100644 index 00000000..e4dd9e83 --- /dev/null +++ b/doc/local-validation-archive/files/issue-246-validation/debug/tests/benchmark/compare-matrix-storage.py @@ -0,0 +1,121 @@ +#!/usr/bin/env python3 +"""Compare two identically built IntaRNA binaries; save every sample and output. + +Requires Python 3 and Linux /usr/bin/time. No optional Python packages. +Run after builds/tests finish, on an otherwise idle machine. +""" +import argparse +import hashlib +import json +import os +from pathlib import Path +import platform +import random +import statistics +import subprocess +import time + + +def sha(data): + return hashlib.sha256(data).hexdigest() + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("baseline", type=Path) + parser.add_argument("candidate", type=Path) + parser.add_argument("output", type=Path) + parser.add_argument("--repetitions", type=int, default=7) + parser.add_argument("--cpu", type=int) + args = parser.parse_args() + if args.repetitions < 1: + parser.error("repetitions must be positive") + if args.cpu is not None: + os.sched_setaffinity(0, {args.cpu}) + out = args.output.resolve() + out.mkdir(parents=True, exist_ok=False) + binaries = {"baseline": args.baseline.resolve(), "candidate": args.candidate.resolve()} + env = dict(os.environ, LC_ALL="C", OPENBLAS_NUM_THREADS="1", OMP_DYNAMIC="FALSE") + root = Path(__file__).resolve().parents[2] + rng = random.Random(246) + target = "".join(rng.choices("ACGU", k=1000)) + query = "".join(rng.choices("ACGU", k=120)) + inputs = {} + for name, seq in [("t1000", target), ("t300", target[:300]), ("t80", target[:80]), + ("q120", query), ("q80", query[:80])]: + path = out / (name + ".fa") + path.write_text(">" + name + "\n" + seq + "\n") + inputs[name] = path + for name in ("fhlA", "OxyS"): + path = out / (name + ".fa") + path.write_bytes((root / "doc/handson" / (name + ".fasta")).read_bytes()) + inputs[name] = path + common = ["--threads=1", "--outMode=C", "--outCsvCols=id1,id2,start1,end1,start2,end2,E,bpList", + "--default-log-file=/dev/null"] + cases = [] + + def case(name, target_name, query_name, *options): + cases.append({"name": name, "args": ["--target=" + str(inputs[target_name]), + "--query=" + str(inputs[query_name]), *common, *options]}) + + case("biological-default", "fhlA", "OxyS") + case("banded-default", "t1000", "q120") + case("banded-narrow", "t1000", "q120", "--accL=30", "--accW=100") + case("dense-no-accessibility", "t1000", "q120", "--acc=N") + case("seed-bulges", "t300", "q120", "--seedMaxUP=2") + case("helix-block", "t300", "q120", "--model=B") + case("exact-mfe", "t80", "q80", "--mode=M", "--model=S", "--noSeed", "--intLenMax=30") + case("exact-ensemble", "t80", "q80", "--mode=M", "--model=P", "--noSeed", "--intLenMax=30") + case("seed-extension", "t300", "q120", "--mode=H", "--model=P", "--intLenMax=40") + case("triangular-base-pair", "t80", "q80", "--energy=B") + metadata = {"platform": platform.platform(), "affinity": sorted(os.sched_getaffinity(0)), + "repetitions": args.repetitions, "warmups": 1, "seed": 246, + "binaries": {name: {"path": str(path), "sha256": sha(path.read_bytes()), + "version": subprocess.check_output([str(path), "--version"], env=env).decode()} + for name, path in binaries.items()}, + "inputs": {name: sha(path.read_bytes()) for name, path in inputs.items()}, + "cases": cases, "environment": {key: env[key] for key in ("LC_ALL", "OPENBLAS_NUM_THREADS", "OMP_DYNAMIC")}} + (out / "metadata.json").write_text(json.dumps(metadata, indent=2) + "\n") + expected = {} + samples = [] + for repetition in range(args.repetitions + 1): + schedule = [(case, variant) for case in cases for variant in binaries] + rng.shuffle(schedule) + for current, variant in schedule: + stem = out / f"{current['name']}.{variant}.{repetition}" + command = ["/usr/bin/time", "-f", "%U %S %M", "-o", str(stem) + ".time", + str(binaries[variant]), *current["args"]] + started = time.perf_counter() + result = subprocess.run(command, env=env, capture_output=True, timeout=180) + wall = time.perf_counter() - started + Path(str(stem) + ".stdout").write_bytes(result.stdout) + Path(str(stem) + ".stderr").write_bytes(result.stderr) + if result.returncode: + raise RuntimeError(f"{stem.name} failed: {result.stderr.decode()}") + digest = sha(result.stdout) + if digest != expected.setdefault(current["name"], digest): + raise RuntimeError(f"output mismatch: {stem.name}") + user, system, rss = map(float, Path(str(stem) + ".time").read_text().split()) + row = {"case": current["name"], "variant": variant, "repetition": repetition, + "warmup": repetition == 0, "wall_s": wall, "user_s": user, "system_s": system, + "max_rss_kib": int(rss), "sha256": digest} + samples.append(row) + with (out / "samples.jsonl").open("a") as handle: + handle.write(json.dumps(row) + "\n") + print(f"Completed round {repetition}/{args.repetitions}", flush=True) + summary = [] + for current in cases: + row = {"case": current["name"], "output_identical": True} + for variant in binaries: + selected = [x for x in samples if not x["warmup"] and x["case"] == current["name"] and x["variant"] == variant] + row[variant] = {key: statistics.median(x[key] for x in selected) for key in ("wall_s", "user_s", "max_rss_kib")} + row[variant]["wall_min_s"] = min(x["wall_s"] for x in selected) + row[variant]["wall_max_s"] = max(x["wall_s"] for x in selected) + row["wall_change_percent"] = 100 * (row["candidate"]["wall_s"] / row["baseline"]["wall_s"] - 1) + summary.append(row) + (out / "summary.json").write_text(json.dumps(summary, indent=2) + "\n") + print(json.dumps(summary, indent=2)) + + +if __name__ == "__main__": + main() diff --git a/doc/local-validation-archive/files/issue-246-validation/installed-header-probe.cpp b/doc/local-validation-archive/files/issue-246-validation/installed-header-probe.cpp new file mode 100644 index 00000000..2b45fa6d --- /dev/null +++ b/doc/local-validation-archive/files/issue-246-validation/installed-header-probe.cpp @@ -0,0 +1,2 @@ +#include +int main() { return 0; } diff --git a/doc/local-validation-archive/files/issue-246-validation/kokkos-distribution-probe.cpp b/doc/local-validation-archive/files/issue-246-validation/kokkos-distribution-probe.cpp new file mode 100644 index 00000000..a1f94c97 --- /dev/null +++ b/doc/local-validation-archive/files/issue-246-validation/kokkos-distribution-probe.cpp @@ -0,0 +1,8 @@ +#include +static_assert(INTARNA_USE_STD_MDSPAN == 0); +int main() { + IntaRNA::Matrix dense(2,3); dense(1,2)=42; + IntaRNA::UpperTriangularMatrix triangle(3,3); triangle(1,2)=7; + IntaRNA::UpperBandedMatrix band(3,3,0,1); band(1,2)=9; + return dense(1,2)==42 && triangle(1,2)==7 && band(1,2)==9 ? 0 : 1; +} diff --git a/doc/local-validation-archive/files/issue-246-validation/kokkos-distribution/intaRNA-3.4.1/doc/mdspan-storage.md b/doc/local-validation-archive/files/issue-246-validation/kokkos-distribution/intaRNA-3.4.1/doc/mdspan-storage.md new file mode 100644 index 00000000..21829303 --- /dev/null +++ b/doc/local-validation-archive/files/issue-246-validation/kokkos-distribution/intaRNA-3.4.1/doc/mdspan-storage.md @@ -0,0 +1,275 @@ +# Vector/mdspan storage evaluation (issue #246) + +The migration preserves scientific output and compact storage. In the measured +single-host experiment, performance is effectively neutral: median wall-time +changes range from 1.7% faster to 1.2% slower, with similar peak memory use. +This is a storage modernization rather than an established speedup. GCC 14 +support is retained through the bundled Kokkos implementation when native +mdspan is unavailable. The original measurements below used native mdspan +with GCC 16; they do not measure the Kokkos fallback. + +## Scope and representation + +All production `boost::numeric::ublas` storage aliases have been replaced by +owning containers in `IntaRNA/Matrix.h`. The dynamic programming recurrences, +traversal order, energy types and floating-point operations are unchanged. +`boost::multi_array` seed/helix recurrence tensors are outside this issue's +uBLAS scope and remain unchanged. + +| Container | Logical shape | Stored elements | mdspan access | +| --- | --- | --- | --- | +| `Matrix` | rows by columns | rows * columns | 2D row-major `[i,j]` | +| `UpperBandedMatrix` | upper band including diagonal | rows * min(columns, upper+1) | 2D row-major `[i,j-i]` | +| `UpperTriangularMatrix` | square upper triangle | n * (n+1) / 2 | 1D packed offset | + +The triangular container uses a one-dimensional view because its packed row +lengths vary; a standard affine 2D layout would require quadratic rectangular +storage or a custom mapping. The band contains row-end padding, as did uBLAS, +and clamps superdiagonals that exceed the number of columns. In particular, +10,000 rows with ten superdiagonals take 110,000 elements, not 100 million. +The accessibility constructors still request their existing extra +superdiagonals for dangling-end probabilities. + +Each vector owns its data. A short-lived mdspan is created at access time, +so there is no view pointer to repair after copying, moving or resizing. +Default resize preserves the overlapping logical cells; preserving resize +initializes new cells to zero/default values. Callers of non-preserving resize +must overwrite cells before reading. Const access outside a triangle/band +returns zero. Mutable access must address a stored cell, checked by assertions +in debug builds. Allocation-size overflow is rejected before allocating. +The intentionally small interface supports the operations used by IntaRNA, +not the full uBLAS expression/iterator interface. Triangular matrices must be +square and banded matrices must have lower bandwidth zero. + +## Build requirements + +The existing C++23/OpenMP requirement is unchanged: GCC 14 and Apple Clang +remain supported. Configure first compiles actual mutable 2D and const 1D +`std::mdspan` accesses over vector storage. If the native API is unavailable, +it compiles the same accesses using the bundled Kokkos implementation. +`INTARNA_USE_STD_MDSPAN` is set to `1` for native mdspan or `0` for Kokkos in +both the build configuration and the installed `IntaRNA/intarna_config.h`. +`Matrix.h` includes that public configuration and selects the corresponding +header and namespace; Kokkos types are not injected into `std`. + +The default `--with-mdspan=auto` chooses native mdspan when supported. +`--with-mdspan=std` requires the native API and reports an actionable error +if it is unavailable. `--with-mdspan=kokkos` selects the bundled headers even +on a toolchain with native support, allowing reproducible backend comparisons. +Both implementations use the same owning containers and storage tests. + +The vendored headers are copied unchanged from the latest +[Kokkos mdspan stable revision `8989f70749e28f337e6f7aa210db88659dba6f2f`](https://github.com/kokkos/mdspan/tree/8989f70749e28f337e6f7aa210db88659dba6f2f) +(as of 2026-09-29). Only upstream `include/mdspan` and `include/experimental` +headers are stored in `src/mdspan` and `src/experimental`. Both folders contain +an exact copy of the upstream `LICENSE` (Apache-2.0 WITH LLVM-exception). +No upstream build system, tests, benchmarks, or generated artifacts are +imported. Header-only use needs no additional upstream resources. + +Source distributions contain both complete header trees and licenses. +Installation preserves their relative include layout under +`include/IntaRNA/mdspan` and `include/IntaRNA/experimental`, avoiding global +installation into another project's `mdspan` or `experimental` include tree. +Installed consumers use the configured backend and the existing IntaRNA +include path; no additional mdspan package or include flag is needed. + +Linux CI retains its GCC 14 release/debug layout and checks automatic Kokkos +selection. Apple Clang CI checks native selection. Both compile every installed +public header independently and link/run an installed matrix consumer. +Boost remains a dependency for other project components and for the independent +reference containers in the storage regression tests. + +## Reproduction + +Build the parent revision `c5823e0` and this change in separate directories +with the same compiler, dependencies and configure flags. For each checkout: + +```sh +bash autotools-init.sh +./configure CXX=/path/to/g++ CC=/path/to/gcc \ + --with-boost=/path/to/deps --with-boost-libdir=/path/to/deps/lib \ + --with-vrna=/path/to/deps \ + CPPFLAGS=-I/path/to/deps/include \ + LDFLAGS='-L/path/to/deps/lib -Wl,-rpath,/path/to/deps/lib' +make -j4 +make -j4 check +``` + +Then, with no build or other benchmark running: + +```sh +python3 tests/benchmark/compare-matrix-storage.py \ + /path/to/parent/src/bin/IntaRNA /path/to/candidate/src/bin/IntaRNA \ + /path/to/new-results-directory --repetitions=7 --cpu=2 +``` + +The script generates seeded synthetic inputs, copies the repository's biological +fhlA/OxyS inputs, randomizes execution order, excludes one warmup per binary and +case, and retains stdout, stderr and each timing sample. It compares stdout +bytes for every execution, failing immediately on a mismatch or process error. +Each execution has a 180-second timeout that terminates both GNU time and its +IntaRNA child, retaining any captured stdout and stderr. Interrupting the +benchmark also terminates the process group. The biological input +files are included in source distributions so the script also runs from an +extracted release archive. +It records binary/input hashes, exact arguments, versions, affinity and controlled +environment. Wall time includes startup, loading and output; peak RSS comes from +GNU time. The test cases cover accessibility bands, dense predictors, seed +bulges, helix blocks, exact MFE, exact ensemble, seed extension and Nussinov +triangular storage. This is a bounded, single-host performance experiment. + +## Backend-selection validation on 2026-09-29 + +- GCC 14.4 selects bundled Kokkos automatically; the release build passes + 33,264 assertions in 41 API cases and all 20 CLI golden cases. +- Native GCC 16.2 and bundled Kokkos/GCC 14.4 each pass the 28,923 storage + assertions with AddressSanitizer and UndefinedBehaviorSanitizer. LeakSanitizer is disabled because of the sandbox's process-tracing limit. +- Configure checks cover GCC 14 automatic fallback, GCC 16 native selection, + forced Kokkos on GCC 16, and actionable failures for forced native on GCC 14, + an invalid backend option, and a missing bundled header tree. +- An installed GCC 14 consumer includes only installed IntaRNA/dependency + headers, exercises all three matrix types, and links/runs successfully. +- All 30 imported headers and both license copies match pinned upstream bytes. + +## Original native-backend validation on 2026-09-28 + +- Parent release: 4,341 assertions in 37 API cases; all 20 CLI golden cases pass. +- Candidate release and debug (`--enable-debug`): 33,264 assertions in 41 API + cases; all 20 CLI golden cases pass in each build. +- Standalone storage tests: 28,923 assertions in four cases, passing with + AddressSanitizer and UndefinedBehaviorSanitizer (`-O0 -g1 + -fsanitize=address,undefined -fno-omit-frame-pointer`). LeakSanitizer was + disabled because the execution sandbox does not support its process tracing; + no claim of leak-sanitizer coverage is made. +- The original native-only configure probe passed on GCC 16.2 and rejected + a simulated missing ``. The current automatic mode instead selects + Kokkos when native support is unavailable, as checked above. +- `make install` installs `Matrix.h`; a separate C++23 consumer including the + installed accessibility, Nussinov and seed-extension headers compiles and runs. +- Benchmark process regression checks pass, including timeout cleanup of GNU + time's child process and cleanup on interruption. Both checks reject runners + without the corresponding cleanup. +- The benchmark runs from an extracted `make dist` archive: all ten workloads + complete with matching outputs (one warmup and one measured run per binary, + 40 executions total). This is a packaging smoke check; the performance table + below retains the original seven-repetition measurements. + +The storage cases cover empty and rectangular shapes, logical preservation on +resize, zero-initialization of new cells, const structural zeros, the last +stored superdiagonal, packed triangular boundaries, non-scalar cells, +copy/move/swap independence, non-preserving resize, and allocation-size overflow. +Reference values come from uBLAS. New uBLAS primitive cells must be initialized +explicitly before comparison; its banded preserving resize is also unsuitable +as an oracle for some rectangular/empty transitions, so that reference is +rebuilt explicitly from overlapping cells. + +## Original native-backend performance results (GCC 16) + +Baseline: upstream `c5823e0`. Candidate: the storage migration in this change. + +Both are optimized builds using conda-forge GCC/libstdc++ 16.2.0, Boost 1.85.0 +and ViennaRNA 2.7.2 on Linux x86-64, AMD Ryzen 5 7530U. Configure adds +`-O3 -fno-strict-aliasing`; OpenMP is enabled with `--threads=1`. +Runs are pinned to CPU 2 with `LC_ALL=C`, `OPENBLAS_NUM_THREADS=1`, +`OMP_DYNAMIC=FALSE`. No build or sanitizer runs overlap these timings. + +One warmup plus seven measured executions per binary/case: 160 executions +total, all stdout results byte-identical within and between variants. +The table shows median wall seconds and median peak RSS in MiB. A negative +time change means faster. RSS is a process-level measurement and includes +the executable, shared libraries and ViennaRNA allocations. + +| Workload | uBLAS time (s) | mdspan time (s) | Change | uBLAS / mdspan RSS (MiB) | +| --- | ---: | ---: | ---: | ---: | +| biological-default | 0.0538 | 0.0542 | +0.8% | 16.80 / 16.67 | +| banded-default | 1.0801 | 1.0784 | -0.2% | 19.89 / 20.01 | +| banded-narrow | 0.7524 | 0.7592 | +0.9% | 17.71 / 17.59 | +| dense-no-accessibility | 3.9499 | 3.9981 | +1.2% | 15.05 / 15.05 | +| seed-bulges | 1.0204 | 1.0173 | -0.3% | 23.75 / 23.75 | +| helix-block | 0.1953 | 0.1975 | +1.1% | 18.38 / 18.26 | +| exact-mfe | 0.6855 | 0.6920 | +0.9% | 16.25 / 16.25 | +| exact-ensemble | 0.9845 | 0.9869 | +0.3% | 16.25 / 16.12 | +| seed-extension | 0.1306 | 0.1313 | +0.5% | 18.10 / 18.09 | +| triangular-base-pair | 2.4305 | 2.3897 | -1.7% | 14.25 / 14.38 | + +Measured wall-time ranges (min–max across the seven measured samples): + +| Workload | uBLAS (s) | mdspan (s) | +| --- | ---: | ---: | +| biological-default | 0.0534–0.0721 | 0.0529–0.0569 | +| banded-default | 1.0723–1.0850 | 1.0621–1.1033 | +| banded-narrow | 0.7463–0.7617 | 0.7476–0.7638 | +| dense-no-accessibility | 3.9071–4.1057 | 3.9527–4.0275 | +| seed-bulges | 1.0123–1.0264 | 1.0078–1.0262 | +| helix-block | 0.1949–0.1980 | 0.1934–0.2185 | +| exact-mfe | 0.6802–0.7005 | 0.6849–0.7092 | +| exact-ensemble | 0.9778–0.9980 | 0.9760–1.0010 | +| seed-extension | 0.1297–0.1322 | 0.1295–0.1320 | +| triangular-base-pair | 2.4121–2.5025 | 2.3744–2.4138 | + +All samples (including explicitly marked warmups) and output hashes are in +[mdspan-storage-samples.tsv](mdspan-storage-samples.tsv). Full invocation +arguments and deterministic input generation are in the benchmark script. + +Binary SHA-256 values: + +- baseline: `e82b2031d59d2a5c07620cb7361ca1cb42eb5d0c052752762ee131c35d2ef0bb` +- candidate: `76799b89844581511e1010b72af10bcf2e28c47fe1111c21171733491e66e1c8` + +These measurements describe the selected workloads on one host and one +toolchain. They do not establish a universal speedup. Short biological +runs include substantial startup overhead; differences near the observed +spread should be treated as inconclusive. Clang/macOS and other CPU +architectures still need their own measurements. + +## Bundled Kokkos performance results (GCC 14, 2026-09-29) + +The parent `c5823e0` and current candidate were rebuilt cleanly with the same +GCC 14.4.0 compiler and libstdc++ headers, Boost 1.85.0, ViennaRNA 2.7.2, +`-O3 -fno-strict-aliasing`, and OpenMP. Both link to the same dependency-prefix +libstdc++ 16.2 runtime; mdspan availability is determined by the GCC 14 headers. +The candidate automatically selects the pinned Kokkos backend. The host, CPU 2 +affinity, workload arguments, random seed, environment controls, one warmup and +seven measured runs are as described above. No builds, header compilation or +sanitizer checks overlap these timings. + +All 160 executions produce matching stdout for their respective workloads. +Median wall-time changes range from -5.0% to +4.4%. +These observations apply to this host and workload set; they do not establish +a universal performance improvement. + +| Workload | uBLAS time (s) | Kokkos time (s) | Change | uBLAS / Kokkos RSS (MiB) | +| --- | ---: | ---: | ---: | ---: | +| biological-default | 0.0719 | 0.0728 | +1.2% | 16.42 / 16.30 | +| banded-default | 1.5545 | 1.5099 | -2.9% | 19.63 / 19.63 | +| banded-narrow | 1.1298 | 1.1367 | +0.6% | 17.33 / 17.21 | +| dense-no-accessibility | 5.8946 | 6.1527 | +4.4% | 14.68 / 14.68 | +| seed-bulges | 1.5445 | 1.4732 | -4.6% | 23.37 / 23.50 | +| helix-block | 0.2665 | 0.2666 | +0.0% | 18.00 / 18.00 | +| exact-mfe | 1.0318 | 1.0234 | -0.8% | 15.87 / 15.75 | +| exact-ensemble | 1.4179 | 1.4298 | +0.8% | 15.75 / 15.75 | +| seed-extension | 0.1764 | 0.1782 | +1.0% | 17.84 / 17.72 | +| triangular-base-pair | 3.5270 | 3.3494 | -5.0% | 14.00 / 13.88 | + +Measured wall-time ranges across the seven samples: + +| Workload | uBLAS (s) | Kokkos (s) | +| --- | ---: | ---: | +| biological-default | 0.0662–0.1053 | 0.0721–0.0936 | +| banded-default | 1.4561–2.2961 | 1.4573–1.5641 | +| banded-narrow | 1.0549–1.2632 | 1.0636–1.2372 | +| dense-no-accessibility | 5.6979–8.0424 | 5.9377–6.3798 | +| seed-bulges | 1.4834–2.1358 | 1.3990–1.5722 | +| helix-block | 0.2456–0.2855 | 0.2433–0.2901 | +| exact-mfe | 1.0026–1.1652 | 0.9963–1.0786 | +| exact-ensemble | 1.3865–1.5064 | 1.3904–1.4736 | +| seed-extension | 0.1670–0.1934 | 0.1621–0.1911 | +| triangular-base-pair | 3.3628–4.6398 | 3.2633–5.2531 | + +All samples, warmups, and output hashes are in +[mdspan-storage-gcc14-samples.tsv](mdspan-storage-gcc14-samples.tsv). + +Binary SHA-256 values: + +- baseline: `f2d3293f7598980f4e539e925e1cac4bba35b08af688735d5d67250733329571` +- candidate: `b4842a322fa67d61990a3e4a623e19e3c5a049f2bf4d4db3213615c18b900b17` diff --git a/doc/local-validation-archive/files/issue-246-validation/kokkos-installed-consumer.cpp b/doc/local-validation-archive/files/issue-246-validation/kokkos-installed-consumer.cpp new file mode 100644 index 00000000..d9ac7989 --- /dev/null +++ b/doc/local-validation-archive/files/issue-246-validation/kokkos-installed-consumer.cpp @@ -0,0 +1,13 @@ +#include +#include +#include +static_assert(INTARNA_USE_STD_MDSPAN == 0); +int main() { + IntaRNA::RnaSequence rna("test", "ACGU"); + IntaRNA::Matrix dense(2,3); dense(1,2)=42; + IntaRNA::UpperTriangularMatrix triangle(4,4); triangle(2,3)=7; + IntaRNA::UpperBandedMatrix band(4,4,0,1); band(2,3)=9; + return rna.size()==4 && dense(1,2)==42 && triangle(2,3)==7 + && band(2,3)==9 && std::as_const(triangle)(3,2)==0 + && std::as_const(band)(1,3)==0 ? 0 : 1; +} diff --git a/doc/local-validation-archive/files/issue-246-validation/matrix-random-audit.cpp b/doc/local-validation-archive/files/issue-246-validation/matrix-random-audit.cpp new file mode 100644 index 00000000..112c3d93 --- /dev/null +++ b/doc/local-validation-archive/files/issue-246-validation/matrix-random-audit.cpp @@ -0,0 +1,97 @@ +#include +#include +#include +#include +#include +#include + +using Key = std::pair; +std::uint64_t checks = 0; +void require(bool ok) { ++checks; if (!ok) throw std::runtime_error("audit mismatch at check " + std::to_string(checks)); } +struct Oracle { + int kind; + std::size_t rows=0, cols=0, upper=0; + std::map cells; + bool stored(std::size_t i, std::size_t j) const { + return kind==0 || (i<=j && (kind==1 || j-i<=upper)); + } + int get(std::size_t i, std::size_t j) const { + auto pos=cells.find({i,j}); + return pos==cells.end() ? 0 : pos->second; + } + void reshape(std::size_t r, std::size_t c, std::size_t u, bool preserve) { + rows=r; cols=c; upper=u; + for (auto it=cells.begin();it!=cells.end();) { + const auto [i,j]=it->first; + if (!preserve || i>=r || j>=c || !stored(i,j)) it=cells.erase(it); + else ++it; + } + } +}; +template void verify(const M& m, const Oracle& o) { + require(m.size1()==o.rows); require(m.size2()==o.cols); + for(std::size_t i=0;i void fill(M& m, Oracle& o, std::mt19937& rng) { + for(std::size_t i=0;i& m, const Oracle& o, bool preserve) { m.resize(o.rows,o.cols,preserve); } +void resize(IntaRNA::UpperTriangularMatrix& m, const Oracle& o, bool preserve) { m.resize(o.rows,o.cols,preserve); } +void resize(IntaRNA::UpperBandedMatrix& m, const Oracle& o, bool preserve) { m.resize(o.rows,o.cols,0,o.upper,preserve); } +template void audit(int kind) { + std::mt19937 rng(246+kind); + M m; + Oracle oracle{kind}; + for(int step=0;step<10000;++step) { + switch(rng()%7) { + case 0: case 1: { + auto r=rng()%41, c=kind==1 ? r : rng()%41, u=rng()%61; + bool preserve=rng()%2; + oracle.reshape(r,c,u,preserve); + resize(m,oracle,preserve); + if(!preserve) fill(m,oracle,rng); + break; + } + case 2: fill(m,oracle,rng); break; + case 3: m.clear(); oracle.cells.clear(); break; + case 4: { + M copied(m); verify(copied,oracle); + Oracle changed=oracle; + fill(copied,changed,rng); + verify(m,oracle); verify(copied,changed); + M assigned; assigned=copied; + copied.clear(); verify(assigned,changed); + m=assigned; oracle=changed; + break; + } + case 5: { + M moved(std::move(m)); verify(moved,oracle); + require(m.size1()==0 && m.size2()==0); + resize(m,oracle,false); m.clear(); + verify(moved,oracle); + m=std::move(moved); + require(moved.size1()==0 && moved.size2()==0); + auto* alias=&m; m=std::move(*alias); + break; + } + case 6: { + Oracle other{kind}; + auto r=rng()%41, c=kind==1 ? r : rng()%41; + other.reshape(r,c,rng()%61,false); + M swapped; resize(swapped,other,false); fill(swapped,other,rng); + m.swap(swapped); verify(swapped,oracle); oracle=other; + break; + } + } + verify(m,oracle); + } +} +int main() { + audit>(0); + audit>(1); + audit>(2); + std::cout << "Passed 30000 seeded operations, " << checks << " independent reference checks\n"; +} diff --git a/doc/local-validation-archive/files/issue-246-validation/mdspan-probe-source/configure.ac b/doc/local-validation-archive/files/issue-246-validation/mdspan-probe-source/configure.ac new file mode 100644 index 00000000..ba431ed4 --- /dev/null +++ b/doc/local-validation-archive/files/issue-246-validation/mdspan-probe-source/configure.ac @@ -0,0 +1,572 @@ + +AC_PREREQ([2.69]) +# 5 argument version only available with aclocal >= 2.64 +AC_INIT([IntaRNA], [3.4.1], [], [intaRNA], [http://www.bioinf.uni-freiburg.de] ) + +# minimal required version of the boost library +BOOST_REQUIRED_VERSION=1.50.0 +# minimal required version of the Vienna RNA library +VRNA_REQUIRED_VERSION=2.4.14 +# minimal required version of python interpreter (to install intarnapvalue) +PYTHON_REQUIRED_VERSION=3.6 + + +AC_CANONICAL_HOST +AC_CONFIG_AUX_DIR([.]) +AC_CONFIG_SRCDIR([src/easylogging++.h]) +AC_CONFIG_HEADERS([src/config.h]) + +# AC_CONFIG_MACRO_DIR([m4]) +m4_include([m4/m4-ax_cxx_compile_stdcxx.m4]) +m4_include([m4/m4-ax_openmp.m4]) +m4_include([m4/m4-ax_boost_base.m4]) +m4_include([m4/m4-ax_boost_system.m4]) +m4_include([m4/m4-ax_boost_filesystem.m4]) +m4_include([m4/m4-ax_boost_program_options.m4]) +m4_include([m4/m4-ax_boost_regex.m4]) +m4_include([m4/m4-ax_boost_iostreams.m4]) +m4_include([m4/m4-ax_check_zlib.m4]) + +lt_enable_auto_import="" +case "$host_os" in + cygwin* | mingw* | cegcc*) + AM_LDFLAGS="-Wl,--enable-auto-import $AM_LDFLAGS" +esac + + +# check for C++ compiler +# store current compiler flags to avoid default setup via AC_PROG_CXX and *_CC +OLD_CXXFLAGS=$CXXFLAGS +OLD_CFLAGS=$CFLAGS +# Checks for programs. +AC_PROG_CXX +AC_PROG_CC +AC_PROG_LN_S +AC_PROG_RANLIB +AC_PROG_MKDIR_P +# reset compiler flags to initial flags +CXXFLAGS=$OLD_CXXFLAGS +CFLAGS=$OLD_CFLAGS + +# automake initialisation (mandatory) and check for minimal automake API version +AM_INIT_AUTOMAKE([1.11]) + +# use the C++ compiler for the following checks +AC_LANG([C++]) + +# ensure we are using the C++23 standard +AX_CXX_COMPILE_STDCXX([23], [noext], [mandatory]) + +# Check the C++23 library surface used by IntaRNA. A compiler accepting +# -std=c++23 is not sufficient when paired with an older standard library. +AC_MSG_CHECKING([for required C++23 standard library features]) +intarna_saved_CXXFLAGS=$CXXFLAGS +CXXFLAGS="$CXXFLAGS -pedantic-errors" +AC_COMPILE_IFELSE( + [AC_LANG_PROGRAM( + [[#include +#include +#include +#include ]], + [[std::string text{"IntaRNA"}; +const bool has_rna = text.contains("RNA") && text.contains('R'); +text.resize_and_overwrite(8, [](char * data, std::size_t size) noexcept { + for (std::size_t i = 0; i < size; ++i) + data[i] = 'x'; + return size; +}); +std::stringstream stream; +stream << text; +const std::string_view stream_view = stream.view(); +std::stringstream sink; +sink << stream.view(); +(void)has_rna; +(void)stream_view;]])], + [AC_MSG_RESULT([yes])], + [AC_MSG_RESULT([no]) + AC_MSG_ERROR([A C++23 standard library providing std::string::contains, std::string::resize_and_overwrite, and std::stringstream::view is required. Upgrade libstdc++/libc++ or select a newer C++23 toolchain.])]) +CXXFLAGS=$intarna_saved_CXXFLAGS + +# Prefer the standard implementation, falling back to bundled Kokkos headers. +# The selected backend is also recorded in the installed public configuration. +AC_ARG_WITH([mdspan], + [AS_HELP_STRING([--with-mdspan=auto|std|kokkos], + [select mdspan implementation (def=auto: prefer std, otherwise Kokkos)])], + [intarna_mdspan="$withval"], [intarna_mdspan=auto]) +AS_CASE([$intarna_mdspan], [auto|std|kokkos], [], + [AC_MSG_ERROR([--with-mdspan must be auto, std, or kokkos])]) + +INTARNA_USE_STD_MDSPAN=0 +AS_IF([test x"$intarna_mdspan" != x"kokkos"], [ +AC_MSG_CHECKING([for C++23 std::mdspan over std::vector]) +AC_COMPILE_IFELSE( + [AC_LANG_PROGRAM( + [[#include +#include +#ifndef __cpp_lib_mdspan +#error C++23 std::mdspan is unavailable +#endif]], + [[std::vector storage(6); +std::mdspan> matrix(storage.data(), 2, 3); +matrix[1, 2] = 42; +std::mdspan> packed(storage.data(), 6); +return packed[5] != 42;]])], + [AC_MSG_RESULT([yes]) + INTARNA_USE_STD_MDSPAN=1], + [AC_MSG_RESULT([no]) + AS_IF([test x"$intarna_mdspan" = x"std"], + [AC_MSG_ERROR([--with-mdspan=std requires a C++23 standard library providing std::mdspan. Use --with-mdspan=auto or --with-mdspan=kokkos for the bundled implementation.])])]) +]) + +AS_IF([test "$INTARNA_USE_STD_MDSPAN" = 0], [ + AC_MSG_CHECKING([for bundled Kokkos mdspan over std::vector]) + intarna_saved_CPPFLAGS=$CPPFLAGS + CPPFLAGS="$CPPFLAGS -I$srcdir/src" + AC_COMPILE_IFELSE( + [AC_LANG_PROGRAM( + [[#include "mdspan/mdspan.hpp" +#include +namespace intarna_mdspan = MDSPAN_IMPL_STANDARD_NAMESPACE;]], + [[std::vector storage(6); +intarna_mdspan::mdspan> matrix(storage.data(), 2, 3); +matrix[1, 2] = 42; +intarna_mdspan::mdspan> packed(storage.data(), 6); +return packed[5] != 42;]])], + [AC_MSG_RESULT([yes])], + [AC_MSG_RESULT([no]) + AC_MSG_ERROR([The bundled Kokkos mdspan headers do not compile with this C++23 toolchain. See config.log and ensure src/mdspan and src/experimental are complete.])]) + CPPFLAGS=$intarna_saved_CPPFLAGS +]) +AC_DEFINE_UNQUOTED([INTARNA_USE_STD_MDSPAN], [$INTARNA_USE_STD_MDSPAN], + [Use standard mdspan (1) or the bundled Kokkos implementation (0)]) +AC_SUBST([INTARNA_USE_STD_MDSPAN]) + +# check if python is available +AM_PATH_PYTHON([$PYTHON_REQUIRED_VERSION],, [:]) +AM_CONDITIONAL([HAVE_PYTHON], [test "$PYTHON" != ":"]) +AM_COND_IF([HAVE_PYTHON], [HAVE_PYTHON_yesno=yes], [HAVE_PYTHON_yesno=no]) + +############################################################################### +############################################################################### + +############ DEFAULT MACROS IF PKG-CONFIG MISSING ############### + +m4_define_default([PKG_PROG_PKG_CONFIG], + [AC_MSG_CHECKING([pkg-config]) + AC_MSG_RESULT([no])]) + +PKG_PROG_PKG_CONFIG + +# If the pkg-config autoconf support isn't installed, define its +# autoconf macro to disable any packages depending on it. +# +m4_define_default([PKG_CHECK_MODULES], + [AC_MSG_CHECKING([$1]) + AC_MSG_RESULT([no]) + $4]) + + +############ SILENT (MINIMAL) MAKEFILE OUTPUT ################### + +# show full make output via `make V=1` +m4_ifdef( + [AM_SILENT_RULES], + [AM_SILENT_RULES([yes])] +) + +############ PARAMETERS ######################################## + +############################################################################### +# PKG-CONFIG SETUP +############################################################################### + +AC_MSG_CHECKING([whether to use and provide pkg-config information]) +pkgconfigEnabled=yes +AC_ARG_ENABLE([pkg-config], + [AS_HELP_STRING([--disable-pkg-config], + [disable pkg-config support (def=enabled)])], + [pkgconfigEnabled="$enableval"], + [pkgconfigEnabled=yes]) +AC_MSG_RESULT([$pkgconfigEnabled]) +AS_IF([test x"$pkgconfigEnabled" = x"yes"], [ + # Checks for pkg-config + AC_CHECK_PROG([HAVE_PKG_CONFIG],[pkg-config],[yes],[no]) +], [ + HAVE_PKG_CONFIG=no +]) + +############################################################################### +# DEBUG SUPPORT SETUP +############################################################################### + +AC_MSG_CHECKING([whether to build with debug information]) +debuger=no +AC_ARG_ENABLE([debug], + [AS_HELP_STRING([--enable-debug], + [enable debug data generation (def=disabled)])], + [debuger="$enableval"]) +AC_MSG_RESULT([$debuger]) + +AS_IF([test x"$debuger" = x"yes"], [ + AC_DEFINE([_DEBUG], [1], [Run in DEBUG mode with additional assertions and debug output]) + AM_CXXFLAGS="$AM_CXXFLAGS -g -O0 -Wno-uninitialized -Wno-deprecated -DELPP_DEBUG_ERRORS" # -Wall" +], [ + AC_DEFINE([NDEBUG], [1], [Run in normal mode with minimal assertions]) + AM_CXXFLAGS="$AM_CXXFLAGS -O3 -fno-strict-aliasing -Wno-uninitialized -Wno-deprecated" +]) + +############################################################################### +# STATIC LINKING SUPPORT +############################################################################### + +AC_MSG_CHECKING([whether to build statically linked binaries]) +AC_ARG_ENABLE([static-linking], + [AS_HELP_STRING([--enable-static-linking], + [enables static linking of the executables (default=disabled)])], + [enable_static_linking=$enableval], + [enable_static_linking=no]) +AC_MSG_RESULT([$enable_static_linking]) + +AS_IF([test x"$enable_static_linking" = x"yes"], [ + AM_LDFLAGS="-static $AM_LDFLAGS" +]) + +############################################################################### +# MULTI-THREADING SUPPORT SETUP +############################################################################### + +AC_MSG_CHECKING([whether to enable multi-threading support]) +multithreadingEnabled=yes +AC_ARG_ENABLE([multithreading], + [AS_HELP_STRING([--disable-multithreading], + [disable multi-threading support (def=enabled)])], + [multithreadingEnabled="$enableval"], + [multithreadingEnabled=yes]) +AC_MSG_RESULT([$multithreadingEnabled]) +AS_IF([test x"$multithreadingEnabled" = x"yes"], [ + AC_DEFINE([INTARNA_MULITHREADING], [1], [Enabling multi-threading support]) + AC_SUBST([INTARNA_MULITHREADING],[1]) + # ensure OPENMP can be used + AX_OPENMP([],[AC_MSG_ERROR([OPENMP support is mandatory for multi-threading compilation])]) + AM_CXXFLAGS="$AM_CXXFLAGS $OPENMP_CXXFLAGS" +], [ + AC_DEFINE([INTARNA_MULITHREADING], [0], [Disabling multi-threading support]) + AC_SUBST([INTARNA_MULITHREADING],[0]) + AM_CXXFLAGS="$AM_CXXFLAGS -pthread" +]) + +############################################################################### +# MULTIPRECISION SUPPORT (via GNU quadmath library) +############################################################################### + +AC_MSG_CHECKING([whether to use a multi-precision partition function data type (quadmath)]) +enable_multiprecision=no +AC_ARG_ENABLE([multiprecision], + [AS_HELP_STRING([--enable-multiprecision], + [multi-precision partition function data type (default=disabled)])], + [enable_multiprecision=$enableval], + [enable_multiprecision=no]) +AC_MSG_RESULT([$enable_multiprecision]) + +AS_IF([test x"$enable_multiprecision" = x"yes"], [ + LIBS="$LIBS -lquadmath" + AC_DEFINE([INTARNA_MULTIPRECISION], [1], [Enabling multiprecision partition function (using quadmath)]) + AC_SUBST([INTARNA_MULTIPRECISION],[1]) +], [ + AC_DEFINE([INTARNA_MULTIPRECISION], [0], [Disabling multiprecision partition function]) + AC_SUBST([INTARNA_MULTIPRECISION],[0]) +]) + +############################################################################### +# Vienna RNA package library path support, if not installed in usual directories +############################################################################### +AC_ARG_WITH([vrna], + [AS_HELP_STRING( + [--with-vrna=PREFIX], + [alternative prefix path to Vienna RNA library] + )], + [RNAPATHSET=1], + [RNAPATHSET=0] +) +# handle user-defined path +AS_IF([test $RNAPATHSET = 1 ], [ + AS_IF([test "x$HAVE_PKG_CONFIG" = "xyes"], [ + # use pkg-config data if available + export PKG_CONFIG_PATH="$with_vrna/lib/pkgconfig/:$PKG_CONFIG_PATH" + ], [ + # guess compiler and linker flags if needed + AM_CXXFLAGS="-I$with_vrna/include $AM_CXXFLAGS" + AM_LDFLAGS="-L$with_vrna/lib $AM_LDFLAGS" + ]) +]) + +############################################################################### +dnl generate doxygen documentation in Doxy +############################################################################### + +AC_CHECK_PROG([HAVE_DOXYGEN],[doxygen],[yes],[no]) + +AS_IF([test "x$HAVE_DOXYGEN" = "xyes"], [ + # loading m4 macro + m4_include([m4/doxygen.m4]) + # setup doxygen documentation to build + DX_HTML_FEATURE(ON) + DX_CHM_FEATURE(OFF) + DX_CHI_FEATURE(OFF) + DX_MAN_FEATURE(OFF) + DX_RTF_FEATURE(OFF) + DX_XML_FEATURE(OFF) + DX_PDF_FEATURE(ON) + DX_PS_FEATURE(OFF) + # generate according options etc. + DX_INIT_DOXYGEN($PACKAGE_NAME, [doc/doxygen.cfg], [doxygen-doc]) +]) + +############################################################################### +# EASYLOGGING++ +############################################################################### + +# enable easylogging++ performance tracking +AM_CXXFLAGS="$AM_CXXFLAGS -DELPP_FEATURE_PERFORMANCE_TRACKING" +# disable default log file creation +AM_CXXFLAGS="$AM_CXXFLAGS -DELPP_NO_DEFAULT_LOG_FILE" + +############################# +# COLORING SETUP +############################# + +AC_MSG_CHECKING([whether to enable coloring of console log output]) +logColoring=yes +AC_ARG_ENABLE([log-coloring], + [AS_HELP_STRING([--disable-log-coloring], + [disable console log coloring (def=enabled)])], + [logColoring="$enableval"]) +AC_MSG_RESULT([$logColoring]) +AS_IF([test x"$logColoring" = x"yes"], [ + AC_DEFINE([INTARNA_LOG_COLORING], [1], [Enabling console log coloring]) + AC_SUBST([INTARNA_LOG_COLORING],[1]) +], [ + AC_DEFINE([INTARNA_LOG_COLORING], [0], [Disabling console log coloring]) + AC_SUBST([INTARNA_LOG_COLORING],[0]) +]) + + + +############################# +# P-VALUE SCRIPT +############################# + +AC_MSG_CHECKING([whether to compile and install 'intarnapvalue' python module]) +intarnapvalue=no +AC_ARG_ENABLE([intarnapvalue], + [AS_HELP_STRING([--enable-intarnapvalue], + [build and install the intarnapvalue python3 module])], + [AM_COND_IF([HAVE_PYTHON], [intarnapvalue="$enableval"], [intarnapvalue="no (python not available)"])]) +AC_MSG_RESULT([$intarnapvalue]) + +AM_CONDITIONAL([INSTALL_INTARNAPVALUE],[test x"$intarnapvalue" = x"yes"]) + + + +############################################################################### + +############################################################################### +# BOOST CHECK +############################################################################### + +AX_BOOST_BASE([$BOOST_REQUIRED_VERSION], [FOUND_BOOST=1;], [FOUND_BOOST=0;]) +AS_IF([test "x$want_boost" != "xno" && test "x$FOUND_BOOST" = "x1"], [ + AX_BOOST_SYSTEM + AX_BOOST_FILESYSTEM + AX_BOOST_PROGRAM_OPTIONS + AX_BOOST_REGEX + AX_BOOST_IOSTREAMS +]) +AX_CHECK_ZLIB([], []) + + +############ CHECKS ############################################ + +# Checks for typedefs, structures, and compiler characteristics. +AC_TYPE_SIZE_T + +########################################################################## +# check boost test results +########################################################################## + +# FOUND_BOOST is only defined if want_boost is "yes" +AS_IF([test $want_boost != "no" && test $FOUND_BOOST != 1], [ + AC_MSG_NOTICE([]) + AC_MSG_NOTICE([The Boost Library was not found!]) + AC_MSG_NOTICE([ -> If installed in a non-standard path, please use '--with-boost=PREFIX'.]) + AC_MSG_NOTICE([]) + DEPENDENCYNOTFOUND=1; +], [ + AM_CXXFLAGS="$BOOST_CPPFLAGS $AM_CXXFLAGS" + AM_LDFLAGS="$BOOST_LDFLAGS $AM_LDFLAGS" + LIBS="$LIBS -lboost_regex -lboost_program_options -lboost_filesystem -lboost_system -lboost_iostreams -lz" +]) + + +############################################################################### +# BEGIN VIENNA CHECK +############################################################################### +# check for Vienna RNA headers + +RNANOTFOUND=0 +AS_IF([test x"$HAVE_PKG_CONFIG" = x"yes"], [ + AC_MSG_NOTICE([check if VRNA version >= $VRNA_REQUIRED_VERSION]) + PKG_CHECK_MODULES([VRNA], [RNAlib2 >= $VRNA_REQUIRED_VERSION], [RNANOTFOUND=0], [RNANOTFOUND=1]) +]) + +AS_IF([test "$RNANOTFOUND" = "0"], [ + AC_MSG_CHECKING([for compilation with Vienna RNA package headers]) + OLD_CPPFLAGS=$CPPFLAGS + OLD_CXXFLAGS=$CXXFLAGS + OLD_LDFLAGS=$LDFLAGS + AC_LANG_PUSH([C]) + CPPFLAGS="$CPPFLAGS $AM_CXXFLAGS" + LDFLAGS="$LDFLAGS $AM_LDFLAGS" + AS_IF([test "x$HAVE_PKG_CONFIG" = "xyes"], [ + CPPFLAGS="$CPPFLAGS $VRNA_CFLAGS" + LDFLAGS="$LDFLAGS $VRNA_LIBS" + ]) + AC_COMPILE_IFELSE( + [AC_LANG_PROGRAM([[#include ]],[[vrna_md_t tmp; vrna_md_copy(&tmp,&tmp);]])], + [ + AC_MSG_RESULT([yes]) + RNANOTFOUND=0; + ], + [ + AC_MSG_RESULT([no]) + AC_MSG_NOTICE([DEBUG : used CPPFLAGS = $CPPFLAGS]) + AC_MSG_NOTICE([DEBUG : used LDFLAGS = $LDFLAGS]) + RNANOTFOUND=1; + ] + ) + AC_LANG_POP([C]) + CPPFLAGS=$OLD_CPPFLAGS + LDFLAGS=$OLD_LDFLAGS +]) + +# error output if ViennaRNA not found +AS_IF([test "$RNANOTFOUND" = "1"], [ + AC_MSG_NOTICE() + AC_MSG_NOTICE([The Vienna RNA C library version >= $VRNA_REQUIRED_VERSION is required.]) + AC_MSG_NOTICE([ -> It can be obtained from http://www.tbi.univie.ac.at/.]) + AC_MSG_NOTICE() + AS_IF([test "$RNAPATHSET" = "1"], [ + AC_MSG_NOTICE([ -> Can't find the Vienna RNA library in given path '$with_vrna'.]) + ], [ # else + AC_MSG_NOTICE([ -> If installed in a non-standard path, please use '--with-vrna=PREFIX'.]) + ]) + DEPENDENCYNOTFOUND=1; +],[ # else + # register Vienna RNA lib for linking + AS_IF([test "x$HAVE_PKG_CONFIG" = "xyes"], [ + # apply pkg-config options for VRNA package for further build + AM_CXXFLAGS="$AM_CXXFLAGS $VRNA_CFLAGS" + LIBS="$LIBS $VRNA_LIBS" + ], [ # else + # guess that only lib information missing + LIBS="$LIBS -lRNA" + ]) +]) + + +############################################################################### +# END VIENNA CHECK +############################################################################### + +############################################################################### +# FINAL DEPENDENCY CHECK AND EXIT IF NEEDED +############################################################################### + +# error ABORT if on of the libraries was not found +AS_IF([test "$DEPENDENCYNOTFOUND" = "1"], [ + AC_MSG_NOTICE() + AC_MSG_ERROR([Some dependency was not met! See above for errors and relate to './configure --help'.]) +]) + +########################################################################## + +# get available personalities +AC_SUBST([PERSONALITIES],[`grep 'return.*"IntaRNA..*"' ./src/bin/CommandLineParsing.h | grep -o '".*"' | tr -d '"' | tr "\n" " "`]) +########################################################################## + +# Keep strict diagnostics focused on IntaRNA sources. Boost and ViennaRNA +# headers are external dependencies and can contain compiler extensions. +INTARNA_SYSTEM_CPPFLAGS= +#for intarna_cppflag in $BOOST_CPPFLAGS $VRNA_CFLAGS; do # ORIG +for intarna_cppflag in $VRNA_CFLAGS; do + case "$intarna_cppflag" in + -I?*) + intarna_include_path=${intarna_cppflag#-I} + INTARNA_SYSTEM_CPPFLAGS="$INTARNA_SYSTEM_CPPFLAGS -isystem $intarna_include_path" + ;; + esac +done + + +# distribute additional compiler and linker flags +# --> set these variables instead of CXXFLAGS or LDFLAGS +AC_SUBST([AM_CXXFLAGS]) +AC_SUBST([AM_LDFLAGS]) +AC_SUBST([INTARNA_SYSTEM_CPPFLAGS]) +AC_SUBST([LIBS]) +AM_CONDITIONAL(enable_pkg_config, [test "x$HAVE_PKG_CONFIG" = "xyes"]) +AM_CONDITIONAL(enable_doxygen, [test "x$HAVE_DOXYGEN" = "xyes"]) + + +# files to generate via autotools (.am or .in source files) +AC_CONFIG_FILES([Makefile]) +AC_CONFIG_FILES([src/Makefile]) +AC_CONFIG_FILES([src/IntaRNA/Makefile]) +AC_CONFIG_FILES([src/IntaRNA/intarna_config.h]) +AC_CONFIG_FILES([src/bin/Makefile]) +AC_CONFIG_FILES([perl/Makefile]) +AC_CONFIG_FILES([R/Makefile]) +AC_CONFIG_FILES([python/Makefile]) +AC_CONFIG_FILES([python/bin/Makefile]) +AC_CONFIG_FILES([python/intarnapvalue/Makefile]) +AC_CONFIG_FILES([python/copomus/Makefile]) +AC_CONFIG_FILES([tests/Makefile]) +AC_CONFIG_FILES([tests/data/Makefile]) +AC_CONFIG_FILES([doc/Makefile]) +AC_CONFIG_FILES([IntaRNA.pc]) + +# generate the final Makefile etc. +AC_OUTPUT + +# get absolute directory paths +eval _prefixdir=$(eval printf "%s" $prefix) +eval _bindir=$(eval printf "%s" $bindir) +eval _libdir=$(eval printf "%s" $libdir) +eval _includedir=$(eval printf "%s" $includedir) + +AC_MSG_NOTICE([ + +====================================== + IntaRNA $PACKAGE_VERSION +====================================== + +Features +-------- + * log coloring : $logColoring + * OpenMP multi-threading : $multithreadingEnabled + * intarnapvalue : $intarnapvalue + +Install Directories for given 'prefix' +-------------------------------------- + * Install prefix : $_prefixdir + * Executables : $_bindir + * Libraries : $_libdir + * Header files : $_includedir + * Python modules : $_pydir + +You can run 'make', 'make tests' and 'make install' now! + +Run 'make install prefix=XYZ' for installation in an alternative directory. + +====================================== +]) diff --git a/doc/local-validation-archive/files/issue-246-validation/record-gcc14-benchmark.py b/doc/local-validation-archive/files/issue-246-validation/record-gcc14-benchmark.py new file mode 100644 index 00000000..ee094e64 --- /dev/null +++ b/doc/local-validation-archive/files/issue-246-validation/record-gcc14-benchmark.py @@ -0,0 +1,39 @@ +from pathlib import Path +import csv +import json + +repo=Path('/home/alex/CodexProjects/IntaRNA/pr-worktrees/issue-246') +root=Path(__file__).resolve().parent/'benchmark-gcc14-kokkos' +summary=json.loads((root/'summary.json').read_text()) +metadata=json.loads((root/'metadata.json').read_text()) +samples=[json.loads(line) for line in (root/'samples.jsonl').read_text().splitlines()] +assert len(summary)==10 and len(samples)==160 and all(row['output_identical'] for row in summary) +columns=['case','variant','repetition','warmup','wall_s','user_s','system_s','max_rss_kib','sha256'] +with (repo/'doc/mdspan-storage-gcc14-samples.tsv').open('w') as out: + writer=csv.DictWriter(out,fieldnames=columns,delimiter='\t',lineterminator='\n') + writer.writeheader() + writer.writerows(samples) +changes=[x['wall_change_percent'] for x in summary] +text='''\n## Bundled Kokkos performance results (GCC 14, 2026-09-29)\n\nThe parent `c5823e0` and current candidate were rebuilt cleanly with the same\nGCC 14.4.0 compiler and libstdc++ headers, Boost 1.85.0, ViennaRNA 2.7.2,\n`-O3 -fno-strict-aliasing`, and OpenMP. Both link to the same dependency-prefix\nlibstdc++ 16.2 runtime; mdspan availability is determined by the GCC 14 headers.\nThe candidate automatically selects the pinned Kokkos backend. The host, CPU 2\naffinity, workload arguments, random seed, environment controls, one warmup and\nseven measured runs are as described above. No builds, header compilation or\nsanitizer checks overlap these timings.\n\nAll 160 executions produce matching stdout for their respective workloads.\n''' +text+=f'Median wall-time changes range from {min(changes):+.1f}% to {max(changes):+.1f}%.\n' +text+='These observations apply to this host and workload set; they do not establish\na universal performance improvement.\n\n' +text+='| Workload | uBLAS time (s) | Kokkos time (s) | Change | uBLAS / Kokkos RSS (MiB) |\n| --- | ---: | ---: | ---: | ---: |\n' +for row in summary: + b,c=row['baseline'],row['candidate'] + text+=f"| {row['case']} | {b['wall_s']:.4f} | {c['wall_s']:.4f} | {row['wall_change_percent']:+.1f}% | {b['max_rss_kib']/1024:.2f} / {c['max_rss_kib']/1024:.2f} |\n" +text+='\nMeasured wall-time ranges across the seven samples:\n\n' +text+='| Workload | uBLAS (s) | Kokkos (s) |\n| --- | ---: | ---: |\n' +for row in summary: + b,c=row['baseline'],row['candidate'] + text+=f"| {row['case']} | {b['wall_min_s']:.4f}–{b['wall_max_s']:.4f} | {c['wall_min_s']:.4f}–{c['wall_max_s']:.4f} |\n" +text+='\nAll samples, warmups, and output hashes are in\n[mdspan-storage-gcc14-samples.tsv](mdspan-storage-gcc14-samples.tsv).\n\nBinary SHA-256 values:\n\n' +for name in ('baseline','candidate'): + text+=f"- {name}: `{metadata['binaries'][name]['sha256']}`\n" +p=repo/'doc/mdspan-storage.md' +s=p.read_text() +assert '## Bundled Kokkos performance results' not in s +p.write_text(s+text) +p=repo/'doc/Makefile.am' +s=p.read_text().replace('mdspan-storage-samples.tsv \\\n','mdspan-storage-samples.tsv \\\n\t\t\tmdspan-storage-gcc14-samples.tsv \\\n') +p.write_text(s) +print(f'Added 160 samples; median wall changes {min(changes):+.1f}% to {max(changes):+.1f}%') diff --git a/doc/local-validation-archive/files/issue-246-validation/summarize.py b/doc/local-validation-archive/files/issue-246-validation/summarize.py new file mode 100644 index 00000000..e770a2b7 --- /dev/null +++ b/doc/local-validation-archive/files/issue-246-validation/summarize.py @@ -0,0 +1,46 @@ +from pathlib import Path +import csv, json, statistics +validation=Path(__file__).resolve().parent +repo=validation.parent/'pr-worktrees/issue-246' +results=validation/'benchmark' +summary=json.loads((results/'summary.json').read_text()) +metadata=json.loads((results/'metadata.json').read_text()) +rows=[json.loads(line) for line in (results/'samples.jsonl').read_text().splitlines()] +fields=['case','variant','repetition','warmup','wall_s','user_s','system_s','max_rss_kib','sha256'] +with (repo/'doc/mdspan-storage-samples.tsv').open('w') as handle: + writer=csv.DictWriter(handle,fieldnames=fields,delimiter='\t',lineterminator='\n') + writer.writeheader();writer.writerows(rows) +lines=['\n## Same-host performance results\n', + 'Baseline: upstream `c5823e0`. Candidate: the storage migration in this change.\n', + 'Both are optimized builds using conda-forge GCC/libstdc++ 16.2.0, Boost 1.85.0\n' + 'and ViennaRNA 2.7.2 on Linux x86-64, AMD Ryzen 5 7530U. Configure adds\n' + '`-O3 -fno-strict-aliasing`; OpenMP is enabled with `--threads=1`.\n' + 'Runs are pinned to CPU 2 with `LC_ALL=C`, `OPENBLAS_NUM_THREADS=1`,\n' + '`OMP_DYNAMIC=FALSE`. No build or sanitizer runs overlap these timings.\n', + 'One warmup plus seven measured executions per binary/case: 160 executions\n' + 'total, all stdout results byte-identical within and between variants.\n' + 'The table shows median wall seconds and median peak RSS in MiB. A negative\n' + 'time change means faster. RSS is a process-level measurement and includes\n' + 'the executable, shared libraries and ViennaRNA allocations.\n', + '| Workload | uBLAS time (s) | mdspan time (s) | Change | uBLAS / mdspan RSS (MiB) |', + '| --- | ---: | ---: | ---: | ---: |'] +for row in summary: + b,c=row['baseline'],row['candidate'] + lines.append(f"| {row['case']} | {b['wall_s']:.4f} | {c['wall_s']:.4f} | {row['wall_change_percent']:+.1f}% | {b['max_rss_kib']/1024:.2f} / {c['max_rss_kib']/1024:.2f} |") +lines += ['\nMeasured wall-time ranges (min–max across the seven measured samples):\n', + '| Workload | uBLAS (s) | mdspan (s) |','| --- | ---: | ---: |'] +for row in summary: + b,c=row['baseline'],row['candidate'] + lines.append(f"| {row['case']} | {b['wall_min_s']:.4f}–{b['wall_max_s']:.4f} | {c['wall_min_s']:.4f}–{c['wall_max_s']:.4f} |") +lines += ['\nAll samples (including explicitly marked warmups) and output hashes are in\n' + '[mdspan-storage-samples.tsv](mdspan-storage-samples.tsv). Full invocation\n' + 'arguments and deterministic input generation are in the benchmark script.\n', + 'Binary SHA-256 values:\n'] +for name,item in metadata['binaries'].items(): lines.append(f"- {name}: `{item['sha256']}`") +lines += ['\nThese measurements describe the selected workloads on one host and one\n' + 'toolchain. They do not establish a universal speedup. Short biological\n' + 'runs include substantial startup overhead; differences near the observed\n' + 'spread should be treated as inconclusive. Clang/macOS and other CPU\n' + 'architectures still need their own measurements.\n'] +with (repo/'doc/mdspan-storage.md').open('a') as handle: handle.write('\n'.join(lines)) +print('\n'.join(lines)) diff --git a/doc/local-validation-archive/files/issue-257-validation/build.sh b/doc/local-validation-archive/files/issue-257-validation/build.sh new file mode 100644 index 00000000..c65da403 --- /dev/null +++ b/doc/local-validation-archive/files/issue-257-validation/build.sh @@ -0,0 +1,11 @@ +#!/usr/bin/env bash +set -euo pipefail +v=/home/alex/CodexProjects/IntaRNA/issue-257-validation +deps=/home/alex/CodexProjects/IntaRNA/issue-246-validation/toolchain +cc=/home/alex/CodexProjects/IntaRNA/issue-246-validation/toolchain-gcc14 +export PATH="$cc/bin:$PATH" PKG_CONFIG_PATH="$deps/lib/pkgconfig" OPENBLAS_NUM_THREADS=1 +cd "$1" +bash autotools-init.sh > "$v/logs/$2-bootstrap.log" 2>&1 +./configure CC="$cc/bin/x86_64-conda-linux-gnu-gcc" CXX="$cc/bin/x86_64-conda-linux-gnu-g++" CPPFLAGS="-I$deps/include" LDFLAGS="-L$deps/lib -Wl,-rpath,$deps/lib" --with-vrna="$deps" --with-boost="$deps" --with-boost-libdir="$deps/lib" --prefix="$v/install-$2" "${@:3}" > "$v/logs/$2-configure.log" 2>&1 +make -j2 > "$v/logs/$2-build.log" 2>&1 +make tests -j2 > "$v/logs/$2-tests.log" 2>&1 diff --git a/doc/local-validation-archive/files/issue-257-validation/debug/doc/analysis/seeded-probabilities-validation.md b/doc/local-validation-archive/files/issue-257-validation/debug/doc/analysis/seeded-probabilities-validation.md new file mode 100644 index 00000000..cf5f4ef0 --- /dev/null +++ b/doc/local-validation-archive/files/issue-257-validation/debug/doc/analysis/seeded-probabilities-validation.md @@ -0,0 +1,78 @@ +# Exact seeded probabilities: implementation and validation + +PR #258 implements the seven steps of the reviewed plan in separate commits. +The library ships the specialized suffix automaton; maximal stacks remain a +correctness/performance comparison in the tests. See the [kernel decision](seeded-kernel-comparison.md) +for measured tradeoffs and exact cell-storage counts. + +## Delivered contract + +- Stack-only capability is geometric and forwarded through index offsets. Active + occurrences retain each handler's admission and per-start selection policies; + both endpoints and inclusive span limits are checked. +- Seed-free suffix lengths and two seeded run-length states count ordered chains + once. Whole-interaction noLP forbids departure/termination of singleton runs. + End GU and full-span ED filters affect termination, not intermediate states. +- Initiation, successive-pair loop energies and the complete boundary coefficient + own their contributions once. The complete coefficient is shared with outside + initialization and virtual `updateZ` dispatch. Overrides can observe/delegate; + reweighting belongs in `exactBoundaryWeight`, which serves both passes. Changing + the hybrid mass through an old update override is rejected. +- The pair sink is non-owning. Each prediction creates private region state and + converts internal pairs once through `getBasePair`. The accumulator rejects + intersecting rectangles, validates every merge before mutation, and requires + explicit finalization. CLI ownership spans all regions of one sequence pair. + Cancellation/failure guards apply before output, including the OpenMP path. +- `bpProb` is independent of coverage trackers and ranked reporting. Pair-only + output does not retain `Z_partition`. Simultaneous legacy trackers retain their + existing callbacks/map and native energy rounding semantics. + +## Numerical limits + +Arithmetic uses `Z_type`, exact zero and checked nonnegative sums/products, +exponentials, division and region merges. Structural forbidden energies become +zero before exponentiation. NaN, infinity, positive underflow and positive +subnormals are errors. No scaling is implemented. Ordinary rounding loss in a +sum with widely different terms remains possible. Final row/column exclusivity +and pair mass bounds are checked; endpoint correction is limited to +`512 * epsilon(Z_type) * max(1, targetLength + queryLength)` above one. + +Independent tests use a 2e-12 relative tolerance for raw double-precision totals, +boundaries and masses against long-double enumeration and 50-digit theory gold +values. The CLI base-pair-energy oracle also uses a 2e-14 absolute tolerance for +printed zeros/small values. Native integer ED is preserved and is not compared +against unrounded theoretical ED as though they were identical models. + +## Memory + +Beyond the active suffix grids documented in the kernel comparison, pair output +holds one full target-by-query raw matrix, the private regional outside matrix, +and briefly a regional matrix in original coordinates. Explicit finalization and +writing allocate a normalized full matrix; writing also buffers the complete CSV +block. These phases are sequential, not all simultaneous. Native accessibility +and retained seed preprocessing are additional costs. Legacy trackers add their +own coverage matrices and complete-boundary map. No global run tape or global +boundary-weight table is retained. Without pair output, no pair matrix or reverse +state is allocated. + +## Reproducible checks + +Use GCC 14 with C++23, Boost, ViennaRNA and the bundled Kokkos mdspan fallback: + +```sh +bash autotools-init.sh +./configure --disable-debug # dependency prefixes as in AGENTS.md when needed +make -j2 +make tests -j2 +# Repeat in a clean source/build directory with --enable-debug. +``` + +The focused tags are `[SeededPartitionFunction]`, `[StackSeedDomain]`, +`[BasePairProbabilities]`, `[PredictorSeedOracle]` and `[PredictorMfeEns]`. +`tests/runBasePairProbabilities.sh` checks CLI behavior with a separate Python +chain enumerator. `tests/tools/export-seeded-theory.py` reproduces all ten +fixture extractions from the cited theory revision. No legacy CLI golden file +is regenerated for this change. + +The full release/debug results, installed-header checks and native measurements +are recorded below after the final validation run. diff --git a/doc/local-validation-archive/files/issue-257-validation/header-probe.cpp b/doc/local-validation-archive/files/issue-257-validation/header-probe.cpp new file mode 100644 index 00000000..2b45fa6d --- /dev/null +++ b/doc/local-validation-archive/files/issue-257-validation/header-probe.cpp @@ -0,0 +1,2 @@ +#include +int main() { return 0; } diff --git a/doc/local-validation-archive/files/issue-257-validation/oracle-check.cpp b/doc/local-validation-archive/files/issue-257-validation/oracle-check.cpp new file mode 100644 index 00000000..e2189ae8 --- /dev/null +++ b/doc/local-validation-archive/files/issue-257-validation/oracle-check.cpp @@ -0,0 +1,4 @@ +#include "SeededTheoryFixtures.h" +#include +#include +int main() { for(const auto & f:seeded_oracle::theoryFixtures()) { auto r=seeded_oracle::enumerate(f.model); assert(std::abs(r.z-f.expected.z)<2e-12L*f.expected.z); for(auto [p,m]:f.expected.mass) assert(std::abs(r.mass[p]-m)<2e-12L*std::max(1.L,m)); std::cout< +#include +#include +#include +#include +INITIALIZE_EASYLOGGINGPP + +class ObservedSeeds : public IntaRNA::SeedHandlerNoBulge { +public: + using SeedHandlerNoBulge::SeedHandlerNoBulge; + mutable size_t secondLengthCalls=0; + bool mismatch=false; + size_t getSeedLength2(size_t i,size_t j) const override { + ++secondLengthCalls; + return SeedHandlerNoBulge::getSeedLength2(i,j)+(mismatch?1:0); + } +}; + +int main(int argc,char **) { + using namespace IntaRNA; + RnaSequence target("target","GGG"),query("query","CCC"); + AccessibilityDisabled at(target,0,nullptr),aq(query,0,nullptr); + ReverseAccessibility ar(aq); + InteractionEnergyBasePair energy(at,ar); + SeedConstraint constraint(2,0,0,0,E_INF,Accessibility::ED_UPPER_BOUND,E_INF, + IndexRangeList(),IndexRangeList(),"",false,false,false); + ObservedSeeds handler(energy,constraint); + handler.fillSeed(0,2,0,2); + handler.secondLengthCalls=0; + handler.mismatch=argc>1; + StackSeedDomain domain(handler,3,3,3,3); + std::cout << "length=" << domain.maxSeedLength() + << " secondLengthCalls=" << handler.secondLengthCalls << '\n'; + return domain.maxSeedLength()==2?0:1; +} diff --git a/doc/local-validation-archive/files/issue-258-review-validation/header-probe.cpp b/doc/local-validation-archive/files/issue-258-review-validation/header-probe.cpp new file mode 100644 index 00000000..28acc345 --- /dev/null +++ b/doc/local-validation-archive/files/issue-258-review-validation/header-probe.cpp @@ -0,0 +1,2 @@ +#include +int main() {} diff --git a/doc/local-validation-archive/files/issue-258-review-validation/validate-installed.py b/doc/local-validation-archive/files/issue-258-review-validation/validate-installed.py new file mode 100644 index 00000000..9fb9a24c --- /dev/null +++ b/doc/local-validation-archive/files/issue-258-review-validation/validate-installed.py @@ -0,0 +1,45 @@ +from pathlib import Path +import os, resource, shlex, subprocess + +root=Path('/home/alex/CodexProjects/IntaRNA') +validation=root/'issue-258-review-validation' +deps=root/'issue-246-validation/toolchain' +compiler=root/'issue-246-validation/toolchain-gcc14/bin/x86_64-conda-linux-gnu-g++' +source=root/'pr-worktrees/issue-257-plan' + +def run(args, **kwargs): + return subprocess.run([str(x) for x in args],check=True,**kwargs) + +def no_core(): + resource.setrlimit(resource.RLIMIT_CORE,(0,0)) + +for mode,build in [('release',source),('debug',validation/'debug')]: + prefix=root/f'issue-258-validation/install-review-{mode}' + with (validation/f'logs/{mode}-install.log').open('w') as log: + run(['make','install'],cwd=build,stdout=log,stderr=subprocess.STDOUT) + env=os.environ.copy() + env['PKG_CONFIG_PATH']=str(prefix/'lib/pkgconfig') + flags=shlex.split(subprocess.check_output(['pkg-config','--cflags','--libs','IntaRNA'],env=env,text=True)) + link=[f'-L{deps}/lib',f'-Wl,-rpath,{deps}/lib'] + if mode=='release': + for header in ['StackSeedDomain.h','SeededPartitionFunction.h']: + probe=validation/'header-probe.cpp' + probe.write_text(f'#include \nint main() {{}}\n') + run([compiler,'-std=c++23',f'-I{prefix}/include',f'-I{deps}/include','-fsyntax-only',probe]) + print(f'PASS installed standalone header {header}',flush=True) + consumer=validation/'installed-probabilities-example' + run([compiler,'-std=c++23',source/'doc/seeded-probabilities-example.cpp',*link,*flags,'-o',consumer]) + result=subprocess.check_output([consumer],text=True) + assert 'bpProb;' in result + (validation/'installed-example.csv').write_text(result) + print('PASS installed pkg-config probability consumer',flush=True) + executable=validation/f'check-invariant-{mode}' + run([compiler,'-std=c++23','-DHAVE_CONFIG_H',f'-I{build}/src',validation/'check-invariant.cpp',*link,*flags,'-o',executable]) + valid=subprocess.check_output([executable],text=True) + assert 'secondLengthCalls=0' in valid if mode=='release' else 'secondLengthCalls=4' in valid + invalid=subprocess.run([executable,'mismatch'],text=True,capture_output=True,preexec_fn=no_core) + if mode=='release': + assert invalid.returncode==0 and 'secondLengthCalls=0' in invalid.stdout + else: + assert invalid.returncode==-6 and 'len == handler.getSeedLength2(i,j)' in invalid.stderr + print(f'PASS {mode} invariant: {valid.strip()}; mismatch exit={invalid.returncode}',flush=True) diff --git a/doc/local-validation-archive/files/issue-258-validation/BasePairProbabilities-header.cpp b/doc/local-validation-archive/files/issue-258-validation/BasePairProbabilities-header.cpp new file mode 100644 index 00000000..b3ecedee --- /dev/null +++ b/doc/local-validation-archive/files/issue-258-validation/BasePairProbabilities-header.cpp @@ -0,0 +1,2 @@ +#include +int main() {} diff --git a/doc/local-validation-archive/files/issue-258-validation/BasePairProbabilityWriter-header.cpp b/doc/local-validation-archive/files/issue-258-validation/BasePairProbabilityWriter-header.cpp new file mode 100644 index 00000000..d87241cb --- /dev/null +++ b/doc/local-validation-archive/files/issue-258-validation/BasePairProbabilityWriter-header.cpp @@ -0,0 +1,2 @@ +#include +int main() {} diff --git a/doc/local-validation-archive/files/issue-258-validation/audit-performance/compare-fixed-long.py b/doc/local-validation-archive/files/issue-258-validation/audit-performance/compare-fixed-long.py new file mode 100644 index 00000000..3f0b91eb --- /dev/null +++ b/doc/local-validation-archive/files/issue-258-validation/audit-performance/compare-fixed-long.py @@ -0,0 +1,52 @@ +#!/usr/bin/env python3 +import hashlib,json,os,pathlib,random,statistics,subprocess,time +ROOT=pathlib.Path('/home/alex/CodexProjects/IntaRNA') +OUT=ROOT/'issue-258-validation/audit-performance' +FIXED=OUT/'fixed-long';FIXED.mkdir(exist_ok=True) +original_commands=json.loads((OUT/'commands.json').read_text()) +ENV=dict(os.environ,OMP_NUM_THREADS='1',OPENBLAS_NUM_THREADS='1',OMP_DYNAMIC='FALSE') +versions={'before':OUT/'IntaRNA-before','fixed':ROOT/'pr-worktrees/issue-258-bpsvg/src/bin/IntaRNA'} +records=[];checks=[];commands={};rng=random.Random(259) +def digest(path): + h=hashlib.sha256() + with open(path,'rb') as stream: + while data:=stream.read(1048576):h.update(data) + return h.hexdigest() +def run(case,version,mode,rep): + cmd=original_commands[f'{case}/head/{mode}'].copy();cmd[3]=str(versions[version]) + cmd=[arg.replace(str(OUT),str(FIXED)) if arg.startswith('--out=') else arg for arg in cmd] + commands[f'{case}/{version}/{mode}']=cmd + start=time.perf_counter() + p=subprocess.run(['/usr/bin/time','-f','%e,%U,%S,%M','-o',str(FIXED/'time.txt')]+cmd,stdout=subprocess.PIPE,stderr=subprocess.PIPE,env=ENV) + duration=time.perf_counter()-start + if p.returncode:raise RuntimeError(p.stderr.decode()) + elapsed,user,system,rss=map(float,(FIXED/'time.txt').read_text().strip().split(',')) + if rep>=0: + records.append(dict(case=case,version=version,mode=mode,rep=rep,wall_seconds=duration,time_seconds=elapsed,user_seconds=user,system_seconds=system,rss_kib=int(rss),svg_bytes=(FIXED/'plot.svg').stat().st_size)) + (FIXED/'raw.json').write_text(json.dumps(records,indent=2)+'\n') + return dict(ensemble=digest(FIXED/'ensemble.txt'),svg=digest(FIXED/'plot.svg'),csv=digest(FIXED/'pairs.csv') if mode=='both' else None) +cases=['sparse1000'];configs=[(version,mode) for version in versions for mode in ['svg','both']] +for case in cases: + print('START',case,flush=True) + warm={} + for version,mode in configs: + warm[(version,mode)]=run(case,version,mode,-1) + for mode in ['svg','both']: + assert warm['before',mode]==warm['fixed',mode],(case,mode,warm) + checks.append({'case':case,'identical_before_fixed':True,'hashes':warm['fixed','both']}) + for rep in range(5): + order=configs.copy();rng.shuffle(order) + for version,mode in order:run(case,version,mode,rep) + for version,mode in configs: + rs=[r for r in records if (r['case'],r['version'],r['mode'])==(case,version,mode)] + print(case,version,mode,'median seconds',round(statistics.median(r['wall_seconds'] for r in rs),4),'max RSS KiB',max(r['rss_kib'] for r in rs),flush=True) +summary=[] +for case in cases: + for version,mode in configs: + rs=[r for r in records if (r['case'],r['version'],r['mode'])==(case,version,mode)] + summary.append(dict(case=case,version=version,mode=mode,wall_median=statistics.median(r['wall_seconds'] for r in rs),wall_min=min(r['wall_seconds'] for r in rs),wall_max=max(r['wall_seconds'] for r in rs),rss_max_kib=max(r['rss_kib'] for r in rs),rss_median_kib=statistics.median(r['rss_kib'] for r in rs),svg_bytes=rs[0]['svg_bytes'])) +(FIXED/'summary.json').write_text(json.dumps(summary,indent=2)+'\n') +(FIXED/'checks.json').write_text(json.dumps(checks,indent=2)+'\n') +(FIXED/'commands.json').write_text(json.dumps(commands,indent=2)+'\n') +(FIXED/'binaries.json').write_text(json.dumps({v:dict(path=str(p),sha256=digest(p)) for v,p in versions.items()},indent=2)+'\n') +print('DONE',flush=True) diff --git a/doc/local-validation-archive/files/issue-258-validation/audit-performance/compare-fixed.py b/doc/local-validation-archive/files/issue-258-validation/audit-performance/compare-fixed.py new file mode 100644 index 00000000..cadf0622 --- /dev/null +++ b/doc/local-validation-archive/files/issue-258-validation/audit-performance/compare-fixed.py @@ -0,0 +1,52 @@ +#!/usr/bin/env python3 +import hashlib,json,os,pathlib,random,statistics,subprocess,time +ROOT=pathlib.Path('/home/alex/CodexProjects/IntaRNA') +OUT=ROOT/'issue-258-validation/audit-performance' +FIXED=OUT/'fixed';FIXED.mkdir(exist_ok=True) +original_commands=json.loads((OUT/'commands.json').read_text()) +ENV=dict(os.environ,OMP_NUM_THREADS='1',OPENBLAS_NUM_THREADS='1',OMP_DYNAMIC='FALSE') +versions={'before':OUT/'IntaRNA-before','fixed':ROOT/'pr-worktrees/issue-258-bpsvg/src/bin/IntaRNA'} +records=[];checks=[];commands={};rng=random.Random(259) +def digest(path): + h=hashlib.sha256() + with open(path,'rb') as stream: + while data:=stream.read(1048576):h.update(data) + return h.hexdigest() +def run(case,version,mode,rep): + cmd=original_commands[f'{case}/head/{mode}'].copy();cmd[3]=str(versions[version]) + cmd=[arg.replace(str(OUT),str(FIXED)) if arg.startswith('--out=') else arg for arg in cmd] + commands[f'{case}/{version}/{mode}']=cmd + start=time.perf_counter() + p=subprocess.run(['/usr/bin/time','-f','%e,%U,%S,%M','-o',str(FIXED/'time.txt')]+cmd,stdout=subprocess.PIPE,stderr=subprocess.PIPE,env=ENV) + duration=time.perf_counter()-start + if p.returncode:raise RuntimeError(p.stderr.decode()) + elapsed,user,system,rss=map(float,(FIXED/'time.txt').read_text().strip().split(',')) + if rep>=0: + records.append(dict(case=case,version=version,mode=mode,rep=rep,wall_seconds=duration,time_seconds=elapsed,user_seconds=user,system_seconds=system,rss_kib=int(rss),svg_bytes=(FIXED/'plot.svg').stat().st_size)) + (FIXED/'raw.json').write_text(json.dumps(records,indent=2)+'\n') + return dict(ensemble=digest(FIXED/'ensemble.txt'),svg=digest(FIXED/'plot.svg'),csv=digest(FIXED/'pairs.csv') if mode=='both' else None) +cases=['OxyS-fhlA','sparse400','sparse1000'];configs=[(version,mode) for version in versions for mode in ['svg','both']] +for case in cases: + print('START',case,flush=True) + warm={} + for version,mode in configs: + warm[(version,mode)]=run(case,version,mode,-1) + for mode in ['svg','both']: + assert warm['before',mode]==warm['fixed',mode],(case,mode,warm) + checks.append({'case':case,'identical_before_fixed':True,'hashes':warm['fixed','both']}) + for rep in range(5): + order=configs.copy();rng.shuffle(order) + for version,mode in order:run(case,version,mode,rep) + for version,mode in configs: + rs=[r for r in records if (r['case'],r['version'],r['mode'])==(case,version,mode)] + print(case,version,mode,'median seconds',round(statistics.median(r['wall_seconds'] for r in rs),4),'max RSS KiB',max(r['rss_kib'] for r in rs),flush=True) +summary=[] +for case in cases: + for version,mode in configs: + rs=[r for r in records if (r['case'],r['version'],r['mode'])==(case,version,mode)] + summary.append(dict(case=case,version=version,mode=mode,wall_median=statistics.median(r['wall_seconds'] for r in rs),wall_min=min(r['wall_seconds'] for r in rs),wall_max=max(r['wall_seconds'] for r in rs),rss_max_kib=max(r['rss_kib'] for r in rs),rss_median_kib=statistics.median(r['rss_kib'] for r in rs),svg_bytes=rs[0]['svg_bytes'])) +(FIXED/'summary.json').write_text(json.dumps(summary,indent=2)+'\n') +(FIXED/'checks.json').write_text(json.dumps(checks,indent=2)+'\n') +(FIXED/'commands.json').write_text(json.dumps(commands,indent=2)+'\n') +(FIXED/'binaries.json').write_text(json.dumps({v:dict(path=str(p),sha256=digest(p)) for v,p in versions.items()},indent=2)+'\n') +print('DONE',flush=True) diff --git a/doc/local-validation-archive/files/issue-258-validation/audit-performance/control-fixed.py b/doc/local-validation-archive/files/issue-258-validation/audit-performance/control-fixed.py new file mode 100644 index 00000000..4a759fc7 --- /dev/null +++ b/doc/local-validation-archive/files/issue-258-validation/audit-performance/control-fixed.py @@ -0,0 +1,27 @@ +#!/usr/bin/env python3 +import hashlib,json,os,pathlib,random,statistics,subprocess,time +ROOT=pathlib.Path('/home/alex/CodexProjects/IntaRNA') +OUT=ROOT/'issue-258-validation/audit-performance';D=OUT/'fixed' +commands=json.loads((OUT/'commands.json').read_text()) +ENV=dict(os.environ,OMP_NUM_THREADS='1',OPENBLAS_NUM_THREADS='1',OMP_DYNAMIC='FALSE') +versions={'before':OUT/'IntaRNA-before','fixed':ROOT/'pr-worktrees/issue-258-bpsvg/src/bin/IntaRNA'} +records=[];rng=random.Random(260);configs=[(v,m) for v in versions for m in ['none','csv']];hashes={} +for rep in range(-1,5): + order=configs.copy();rng.shuffle(order) + for version,mode in order: + cmd=commands[f'dense40/head/{mode}'].copy();cmd[3]=str(versions[version]) + cmd=[arg.replace(str(OUT),str(D)) if arg.startswith('--out=') else arg for arg in cmd] + start=time.perf_counter();p=subprocess.run(['/usr/bin/time','-f','%e,%U,%S,%M','-o',str(D/'time.txt')]+cmd,stdout=subprocess.PIPE,stderr=subprocess.PIPE,env=ENV);duration=time.perf_counter()-start + if p.returncode:raise RuntimeError(p.stderr.decode()) + elapsed,user,system,rss=map(float,(D/'time.txt').read_text().strip().split(',')) + if rep<0: + hashes[(version,mode)]=(hashlib.sha256((D/'ensemble.txt').read_bytes()).hexdigest(),hashlib.sha256((D/'pairs.csv').read_bytes()).hexdigest() if mode=='csv' else None) + else: + records.append(dict(case='dense40',version=version,mode=mode,rep=rep,wall_seconds=duration,rss_kib=int(rss))) +for mode in ['none','csv']:assert hashes['before',mode]==hashes['fixed',mode] +summary=[] +for version,mode in configs: + r=[x for x in records if (x['version'],x['mode'])==(version,mode)] + x=dict(case='dense40',version=version,mode=mode,wall_median=statistics.median(x['wall_seconds'] for x in r),rss_max_kib=max(x['rss_kib'] for x in r));summary.append(x);print(x) +(D/'control-raw.json').write_text(json.dumps(records,indent=2)+'\n');(D/'control-summary.json').write_text(json.dumps(summary,indent=2)+'\n') +print('Dense controls matched byte-for-byte. DONE') diff --git a/doc/local-validation-archive/files/issue-258-validation/audit-performance/run.py b/doc/local-validation-archive/files/issue-258-validation/audit-performance/run.py new file mode 100644 index 00000000..01bb042c --- /dev/null +++ b/doc/local-validation-archive/files/issue-258-validation/audit-performance/run.py @@ -0,0 +1,73 @@ +#!/usr/bin/env python3 +import collections, hashlib, json, os, pathlib, random, statistics, subprocess, time, xml.etree.ElementTree as ET +ROOT=pathlib.Path('/home/alex/CodexProjects/IntaRNA') +OUT=ROOT/'issue-258-validation/audit-performance' +BASE=ROOT/'pr-worktrees/issue-257-plan' +HEAD=ROOT/'pr-worktrees/issue-258-bpsvg' +ENV=dict(os.environ,OMP_NUM_THREADS='1',OPENBLAS_NUM_THREADS='1',OMP_DYNAMIC='FALSE') +CPU=min(os.sched_getaffinity(0)); NS={'s':'http://www.w3.org/2000/svg'} +common=['--model=P','--mode=M','--seedBP=4','--intLenMax=30','--qIntLoopMax=3','--tIntLoopMax=3','--threads=1','--outNoLP=false','--outNoGUend=false','--outNumber=0','--outMode=E','--default-log-file=/dev/null'] +cases={} +cases['OxyS-fhlA']=[f'--target={HEAD}/doc/handson/fhlA.fasta',f'--query={HEAD}/doc/handson/OxyS.fasta'] +for name,n in [('dense40',40),('sparse400',400),('sparse1000',1000)]: + t='G'*n if name.startswith('dense') else 'A'*(n//2)+'G'*8+'A'*(n-n//2-8) + q='C'*n if name.startswith('dense') else 'A'*(n//2)+'C'*8+'A'*(n-n//2-8) + for label,seq in [('target',t),('query',q)]: + (OUT/f'{name}-{label}.fa').write_text(f'>{label}\n{seq}\n') + cases[name]=[f'--target={OUT}/{name}-target.fa',f'--query={OUT}/{name}-query.fa','--tAcc=N','--qAcc=N'] +configs=[('base','none'),('head','none'),('base','csv'),('head','csv'),('head','svg'),('head','both')] +commands={}; records=[]; checks=[]; rng=random.Random(258) +def run(name,version,mode,rep): + tree=BASE if version=='base' else HEAD + cmd=['taskset','-c',str(CPU),str(tree/'src/bin/IntaRNA')]+common+cases[name]+[f'--out={OUT}/ensemble.txt'] + if mode in ('csv','both'):cmd += [f'--out=bpProb:{OUT}/pairs.csv'] + if mode in ('svg','both'):cmd += [f'--out=bpsvg:{OUT}/plot.svg'] + commands[f'{name}/{version}/{mode}']=cmd + begin=time.perf_counter() + proc=subprocess.run(['/usr/bin/time','-f','%e,%U,%S,%M','-o',str(OUT/'time.txt')]+cmd,stdout=subprocess.PIPE,stderr=subprocess.PIPE,env=ENV) + duration=time.perf_counter()-begin + if proc.returncode:raise RuntimeError(f'{name}/{version}/{mode}: {proc.stderr.decode()}') + elapsed,user,system,rss=map(float,(OUT/'time.txt').read_text().strip().split(',')) + data=dict(case=name,version=version,mode=mode,rep=rep,wall_seconds=duration,time_seconds=elapsed,user_seconds=user,system_seconds=system,rss_kib=int(rss),csv_bytes=(OUT/'pairs.csv').stat().st_size if mode in ('csv','both') else 0,svg_bytes=(OUT/'plot.svg').stat().st_size if mode in ('svg','both') else 0) + if rep>=0: + records.append(data) + (OUT/'raw.json').write_text(json.dumps(records,indent=2)+'\n') + return (OUT/'ensemble.txt').read_bytes(),(OUT/'pairs.csv').read_bytes() if mode in ('csv','both') else None +for name in cases: + print('START',name,flush=True) + baseline_ens=None;baseline_csv=None + # Warm each configuration and independently compare full old/new outputs. + for version,mode in configs: + ensemble,csv=run(name,version,mode,-1) + if baseline_ens is None:baseline_ens=ensemble + assert ensemble==baseline_ens,(name,version,mode,'ensemble differs') + if csv is not None: + if baseline_csv is None:baseline_csv=csv + assert csv==baseline_csv,(name,version,mode,'CSV differs') + if mode=='both': + # Stream parsing bounds verifier memory separately from timing. + rows=[row.split(';')[1:] for row in csv.decode().strip().splitlines()[1:]] + count=seed_count=0 + for event,el in ET.iterparse(OUT/'plot.svg',events=['end']): + if el.get('data-type')=='base-pair': + i=count//len(rows[0]);j=count%len(rows[0]);assert float(el.get('data-probability'))==float(rows[i][j]);count+=1 + if el.tag.endswith('rect') and el.get('class')=='seed' and el.get('data-target') is not None:seed_count+=1 + el.clear() + assert count==len(rows)*len(rows[0]) + checks.append(dict(case=name,cells=count,seed_cells=seed_count,ensemble=ensemble.decode(),csv_sha256=hashlib.sha256(csv).hexdigest(),csv_bytes=len(csv),svg_bytes=(OUT/'plot.svg').stat().st_size)) + # Interleaved randomized order limits drift between base and candidate. + for rep in range(7): + order=configs.copy();rng.shuffle(order) + for version,mode in order:run(name,version,mode,rep) + for version,mode in configs: + rs=[r for r in records if (r['case'],r['version'],r['mode'])==(name,version,mode)] + print(name,version,mode,'median seconds',round(statistics.median(r['wall_seconds'] for r in rs),4),'max RSS KiB',max(r['rss_kib'] for r in rs),flush=True) + (OUT/'checks.json').write_text(json.dumps(checks,indent=2)+'\n') + (OUT/'commands.json').write_text(json.dumps(commands,indent=2)+'\n') +summary=[] +for name in cases: + for version,mode in configs: + rs=[r for r in records if (r['case'],r['version'],r['mode'])==(name,version,mode)] + summary.append(dict(case=name,version=version,mode=mode,wall_median=statistics.median(r['wall_seconds'] for r in rs),wall_min=min(r['wall_seconds'] for r in rs),wall_max=max(r['wall_seconds'] for r in rs),rss_max_kib=max(r['rss_kib'] for r in rs),rss_median_kib=statistics.median(r['rss_kib'] for r in rs),csv_bytes=rs[0]['csv_bytes'],svg_bytes=rs[0]['svg_bytes'])) +(OUT/'summary.json').write_text(json.dumps(summary,indent=2)+'\n') +print('DONE',flush=True) diff --git a/doc/local-validation-archive/files/issue-258-validation/audit-performance/serializer-control.py b/doc/local-validation-archive/files/issue-258-validation/audit-performance/serializer-control.py new file mode 100644 index 00000000..fa3d89aa --- /dev/null +++ b/doc/local-validation-archive/files/issue-258-validation/audit-performance/serializer-control.py @@ -0,0 +1,19 @@ +import json,os,pathlib,random,statistics,subprocess,time +R=pathlib.Path('/home/alex/CodexProjects/IntaRNA');P=R/'issue-258-validation/audit-performance';D=P/'serializer-control';D.mkdir(exist_ok=True) +versions={'before':P/'IntaRNA-before','fixed':R/'pr-worktrees/issue-258-bpsvg/src/bin/IntaRNA'} +cmd0=json.loads((P/'commands.json').read_text())['sparse400/head/svg'] +env=dict(os.environ,OMP_NUM_THREADS='1',OPENBLAS_NUM_THREADS='1',OMP_DYNAMIC='FALSE');rng=random.Random(261);records=[];commands={} +for rep in range(-1,7): + order=list(versions);rng.shuffle(order) + for version in order: + cmd=cmd0.copy();cmd[3]=str(versions[version]);cmd=[('--out=bpsvg:/dev/null' if x.startswith('--out=bpsvg:') else '--out=STDOUT' if x.startswith('--out=') else x) for x in cmd];commands[version]=cmd + start=time.perf_counter();r=subprocess.run(['/usr/bin/time','-f','%e,%U,%S,%M','-o',str(D/'time.txt')]+cmd,stdout=subprocess.DEVNULL,stderr=subprocess.PIPE,env=env);wall=time.perf_counter()-start + if r.returncode:raise RuntimeError(r.stderr.decode()) + elapsed,user,system,rss=map(float,(D/'time.txt').read_text().strip().split(',')) + if rep>=0:records.append(dict(version=version,rep=rep,wall=wall,user=user,system=system,cpu=user+system,rss=int(rss))) +summary=[] +for version in versions: + a=[x for x in records if x['version']==version] + summary.append(dict(version=version,median=statistics.median(x['wall'] for x in a),minimum=min(x['wall'] for x in a),maximum=max(x['wall'] for x in a),cpu_median=statistics.median(x['cpu'] for x in a),cpu_min=min(x['cpu'] for x in a),cpu_max=max(x['cpu'] for x in a),user_median=statistics.median(x['user'] for x in a),system_median=statistics.median(x['system'] for x in a),rss_max=max(x['rss'] for x in a))) +for name,data in [('raw',records),('summary',summary),('commands',commands)]: (D/f'{name}.json').write_text(json.dumps(data,indent=2)+'\n') +print(json.dumps(summary,indent=2)) diff --git a/doc/local-validation-archive/files/issue-258-validation/audit-performance/summarize-fixed.py b/doc/local-validation-archive/files/issue-258-validation/audit-performance/summarize-fixed.py new file mode 100644 index 00000000..786024dc --- /dev/null +++ b/doc/local-validation-archive/files/issue-258-validation/audit-performance/summarize-fixed.py @@ -0,0 +1,23 @@ +import json,pathlib,statistics +P=pathlib.Path('/home/alex/CodexProjects/IntaRNA/issue-258-validation/audit-performance') +r=json.loads((P/'fixed/raw.json').read_text())+json.loads((P/'fixed-long/raw.json').read_text()) +summary=[] +for case in ['OxyS-fhlA','sparse400','sparse1000']: + for version in ['before','fixed']: + for mode in ['svg','both']: + a=[x for x in r if (x['case'],x['version'],x['mode'])==(case,version,mode)] + assert len(a)==5,(case,version,mode,len(a)) + summary.append(dict(case=case,version=version,mode=mode,wall_median=statistics.median(x['wall_seconds'] for x in a),wall_min=min(x['wall_seconds'] for x in a),wall_max=max(x['wall_seconds'] for x in a),cpu_median=statistics.median(x['user_seconds']+x['system_seconds'] for x in a),rss_max_kib=max(x['rss_kib'] for x in a),svg_bytes=a[0]['svg_bytes'])) +(P/'summary-fixed-all.json').write_text(json.dumps(summary,indent=2)+'\n') +controls=json.loads((P/'fixed/control-summary.json').read_text()) +lines=['Performance audit after buffer-transfer and accessibility fixes (2026-10-09)', '', 'Direct comparison: retained fa2f84d binary vs final rebuilt candidate. Same environment/flags/input options as initial audit; five fresh-process timed repeats per configuration, deterministic shuffled/interleaved order, CPU0, one thread.', 'One warmup/check per configuration. Whole SVG/CSV/ensemble output matched byte-for-byte on native-energy cases.', 'The initial follow-up process received an unexplained SIGTERM after completing both shorter cases; sparse1000 was resumed in a separate process. Only complete five-run groups are reported; partial long-case warmups were discarded.', '', 'case | output | before/fixed wall median sec | before/fixed CPU median sec | before/fixed max RSS KiB | RSS reduction'] +for case in ['OxyS-fhlA','sparse400','sparse1000']: + for mode in ['svg','both']: + before=next(x for x in summary if (x['case'],x['version'],x['mode'])==(case,'before',mode));fixed=next(x for x in summary if (x['case'],x['version'],x['mode'])==(case,'fixed',mode)) + lines.append(f"{case} | {mode} | {before['wall_median']:.4f}/{fixed['wall_median']:.4f} | {before['cpu_median']:.2f}/{fixed['cpu_median']:.2f} | {before['rss_max_kib']}/{fixed['rss_max_kib']} | {100*(1-fixed['rss_max_kib']/before['rss_max_kib']):.1f}%") +lines+=['', 'Dense40 control after fixes (same before/fixed binary pair):'] +for mode in ['none','csv']: + b=next(x for x in controls if (x['version'],x['mode'])==('before',mode));f=next(x for x in controls if (x['version'],x['mode'])==('fixed',mode));lines.append(f"{mode}: {b['wall_median']:.4f}s -> {f['wall_median']:.4f}s ({100*(f['wall_median']/b['wall_median']-1):+.2f}%), peak RSS {b['rss_max_kib']} -> {f['rss_max_kib']}KiB; ensemble/CSV byte-identical.") +lines+=['', 'Conclusion: transferring output buffers eliminates an avoidable full XML/CSV copy. On 1000x1000 output, SVG peak RSS drops by approximately 36% while preserving all bytes. No material existing-mode regression found. File-write elapsed times fluctuate on this nearly full filesystem, so avoid a speedup claim; process CPU and RSS support the improvement.', 'Full matrix SVG cost remains proportional to all target-by-query cells: 1000x1000 emits 220,685,460 bytes. This is expected from per-cell SVG elements and hover metadata.', '', 'Reproduce: compare-fixed.py (all SVG comparisons); compare-fixed-long.py (resumed sparse1000); control-fixed.py (dense control). Raw timed results: fixed/raw.json, fixed-long/raw.json, fixed/control-raw.json. Exact long-case commands and binary hashes: fixed-long/commands.json and fixed-long/binaries.json. Initial base-vs-feature audit: report-before.txt.', 'No tracked source files edited by the performance agent; root applied the verified fix.'] +(P/'report-fixed.txt').write_text('\n'.join(lines)+'\n') +print('\n'.join(lines)) diff --git a/doc/local-validation-archive/files/issue-258-validation/build.sh b/doc/local-validation-archive/files/issue-258-validation/build.sh new file mode 100644 index 00000000..984b5cd7 --- /dev/null +++ b/doc/local-validation-archive/files/issue-258-validation/build.sh @@ -0,0 +1,11 @@ +#!/usr/bin/env bash +set -euo pipefail +v=/home/alex/CodexProjects/IntaRNA/issue-258-validation +deps=/home/alex/CodexProjects/IntaRNA/issue-246-validation/toolchain +cc=/home/alex/CodexProjects/IntaRNA/issue-246-validation/toolchain-gcc14 +export PATH="$cc/bin:$PATH" PKG_CONFIG_PATH="$deps/lib/pkgconfig" OPENBLAS_NUM_THREADS=1 +cd "$1" +bash autotools-init.sh > "$v/logs/$2-bootstrap.log" 2>&1 +./configure CC="$cc/bin/x86_64-conda-linux-gnu-gcc" CXX="$cc/bin/x86_64-conda-linux-gnu-g++" CPPFLAGS="-I$deps/include" LDFLAGS="-L$deps/lib -Wl,-rpath,$deps/lib" --with-vrna="$deps" --with-boost="$deps" --with-boost-libdir="$deps/lib" --prefix="$v/install-$2" "${@:3}" > "$v/logs/$2-configure.log" 2>&1 +make -j2 > "$v/logs/$2-build.log" 2>&1 +make tests -j2 > "$v/logs/$2-tests.log" 2>&1 diff --git a/doc/local-validation-archive/files/issue-258-validation/header-probe.cpp b/doc/local-validation-archive/files/issue-258-validation/header-probe.cpp new file mode 100644 index 00000000..d87241cb --- /dev/null +++ b/doc/local-validation-archive/files/issue-258-validation/header-probe.cpp @@ -0,0 +1,2 @@ +#include +int main() {} diff --git a/doc/local-validation-archive/files/issue-258-validation/installed-svg-example.cpp b/doc/local-validation-archive/files/issue-258-validation/installed-svg-example.cpp new file mode 100644 index 00000000..5c143f5c --- /dev/null +++ b/doc/local-validation-archive/files/issue-258-validation/installed-svg-example.cpp @@ -0,0 +1,29 @@ +// Standalone installed-library example; see README.md#bpProb. +#include +#include +#include +#include +#include +#include +#include + +INITIALIZE_EASYLOGGINGPP + +int main() { + using namespace IntaRNA; + RnaSequence target("target", "GGAGGG"), query("query", "CCCC"); + AccessibilityDisabled targetAcc(target, 0, nullptr), queryAcc(query, 0, nullptr); + ReverseAccessibility reversed(queryAcc); + InteractionEnergyBasePair energy(targetAcc, reversed); + SeedConstraint seeds(2, 0, 0, 0, E_INF, Accessibility::ED_UPPER_BOUND, + E_INF, IndexRangeList(), IndexRangeList(), "", false, false, false); + OutputConstraint constraints(1, OutputConstraint::OVERLAP_BOTH, E_INF, E_INF, + false, false, false, true, false); // needZall=true; needBPs=false + OutputHandlerInteractionList output(constraints, 1); + BasePairProbabilities probabilities(target.size(), query.size(), true); + PredictorMfeEns2dSeedExtension predictor(energy, output, nullptr, + new SeedHandlerNoBulge(energy, seeds), &probabilities); + predictor.predict(); + probabilities.finalize(); + BasePairProbabilityWriter::writeSvg(std::cout, probabilities, energy, 1, 1); +} diff --git a/doc/local-validation-archive/files/issue-258-validation/review-diagrams/build_probability_recursions.py b/doc/local-validation-archive/files/issue-258-validation/review-diagrams/build_probability_recursions.py new file mode 100644 index 00000000..5a4d0dff --- /dev/null +++ b/doc/local-validation-archive/files/issue-258-validation/review-diagrams/build_probability_recursions.py @@ -0,0 +1,135 @@ +from pathlib import Path +import xml.etree.ElementTree as ET +import cairosvg + +# Reuse only the generator's drawing primitives; never execute its old outputs. +primitives = Path('/home/alex/CodexProjects/IntaRNA/issue-258-validation/step9/build_recursions.py').read_text().split('# 1. Fixed-right-boundary')[0] +exec(primitives) +OUT = Path('/home/alex/CodexProjects/IntaRNA/issue-258-validation/review-diagrams') + +s = SVG('IntaRNA.PredictorMfeEns2d.outside.svg', + 'Unseeded pair probabilities · fixed-right outside pass', + 'PredictorMfeEns2d | --model=P --mode=M --noSeed | reverse the existing hybrid recurrence', 1730) +s.lines(48,185,[ + 'A pair i=(i₁,i₂) uses local energy coordinates; RNA 2 is reversed. i ≺ k increases both coordinates.', + 'Fix right pair j. Hⱼ(i) is the forward hybrid weight; Aⱼ(i)=∂Z/∂Hⱼ(i) is its outside adjoint.', + 'M(i) sums complete-interaction weights containing actual pair i, over every right boundary j.' +],18) +s.eq(48,285,'d=BW(E_init); t(i,k)=BW(E_interLeft(i,k)); i⁺=i+(1,1); s=t(i,i⁺)',23) +s.box(48,322,1344,170,'1 Initialize the same completed-boundary objective') +s.eq(73,389,'Aⱼ(i) ← B(i,j); Z += B(i,j)Hⱼ(i)',27) +s.lines(73,431,[ + 'B(i,j)=BW(energy.getE(i₁,j₁,i₂,j₂,0)) only when !Z_equal(Hⱼ(i),0) and isValidOutputSite() pass.', + 'Otherwise B=0. Complete ED / GU-end filters and native terminal terms match updateZ().' +],18) +s.box(48,520,1344,180,'2 Reverse i in increasing order; credit its explicit pair',fill='#edf6f2',color=TEAL) +s.eq(73,588,'M(i) += Aⱼ(i)Hⱼ(i)',29) +s.lines(73,631,[ + 'Every represented suffix contains its left pair i exactly once. Process only nonzero Aⱼ(i) and Hⱼ(i).', + 'Then propagate Aⱼ(i) through each forward term below; later suffix states receive all enclosing mass.' +],18) +s.box(48,728,645,257,'3a Lonely pairs allowed') +s.eq(73,795,'Hⱼ(j)=d',25) +s.eq(73,840,'Aⱼ(k) += Aⱼ(i)t(i,k)',25) +s.lines(73,883,[ + 'Direct terminal: k=j; its pair is counted at j.', + 'Intermediate k: i ≺ k ≺ j, allowed loop,', + 'and !Z_equal(Hⱼ(k),0) as in the forward sum.' +],18) +s.box(720,728,672,257,'3b noLP: direct stacking',fill='#fff7ed',color=ORANGE) +s.eq(743,795,'Aⱼ(i⁺) += Aⱼ(i)s if i⁺ ≠ j',25) +s.eq(743,840,'M(j) += Aⱼ(i)s d if i⁺ = j',25) +s.lines(743,883,[ + 'A noLP terminal has Hⱼ(j)=0.', + 'Its second pair j is implicit in the two-pair', + 'initialization and is credited separately.' +],18) +s.box(48,1014,1344,297,'3c noLP: stack, then a nonstack loop',fill='#fff7ed',color=ORANGE) +s.rna(96,1115,431,[(0,'i'),(.14,'i⁺'),(.58,'k'),(1,'j')],shade=(.58,1,'Hⱼ(k)')) +s.eq(584,1096,'a = Aⱼ(i)s t(i⁺,k)',25) +s.eq(584,1139,'Aⱼ(k) += a',25) +s.eq(584,1182,'M(i⁺) += a Hⱼ(k)',25) +s.lines(73,1249,[ + 'Require complementary i⁺; i⁺ ≺ k ≺ j; k ≠ i⁺+(1,1); the same loop limits and epsilon gate on Hⱼ(k).', + 'Here i⁺ is an actual pair skipped by the DP-state jump. Unpaired loop positions receive no mass.' +],18) +s.box(48,1340,1344,210,'4 Commit successful regions and normalize only once',fill='#edf6f2',color=TEAL) +s.eq(73,1409,'P(i,j) = ∑ M_region(i,j) / ∑ Z_region',27) +s.lines(73,1451,[ + 'Check the accumulated objective equals the unchanged Zall; map pairs with energy.getBasePair().', + 'Original target/query orientation is stored in the sink. Z=0 → NA; failed arithmetic → no result.', + 'Conditional on an allowed interaction; no unbound-state weight. No outside buffers when output is disabled.' +],18) +s.text(48,1590,'Exact for the existing forward recurrence, including its numerical-zero gates; no alternative ensemble is substituted.',18) +s.footer('PredictorMfeEns2d.cpp: fillHybridZ(), accumulatePairMasses(), predictRegion()', + 'PredictorMfeEns.cpp: addPartitionContribution(); BasePairProbabilities: addRegion(), finalize()') +s.save() + +s = SVG('IntaRNA.PredictorMfeEns2dHeuristic.probabilities.svg', + 'Unseeded heuristic probabilities · candidate flow', + 'PredictorMfeEns2dHeuristic | --model=P --mode=H --noSeed | approximate retained ensemble',1800) +s.lines(48,185,[ + 'Each retained cell C(i) represents one selected pair chain, with hybrid weight vᵢ and right boundary jᵢ.', + 'Pair coordinates increase on both strands internally; RNA 2 is reversed. Set i⁺=i+(1,1).', + 'd=BW(E_init), t(i,k)=BW(E_interLeft(i,k)), and B(i,j)=BW(energy.getE(i₁,j₁,i₂,j₂,0)).' +],18) +s.box(48,294,1344,172,'Approximate means the continuation ensemble is pruned',fill='#fff7ed',color=ORANGE) +s.lines(73,357,[ + 'Visit i in decreasing order. Generate candidates from already selected child chains C(k).', + 'Every admitted candidate contributes to Z; only the candidate with the lowest complete site energy', + 'is retained as C(i) for later extensions. Strict improvement wins; native energy rounding and ties are preserved.' +],18) +s.box(48,494,1344,358,'1 Record which new pairs each candidate owns') +s.text(74,559,'Candidate',19,BLUE,bold=True) +s.text(570,559,'Hybrid weight v',19,BLUE,bold=True) +s.text(929,559,'Owned pairs S',19,BLUE,bold=True) +s.text(1190,559,'Child k',19,BLUE,bold=True) +rows=[ + ('LP: initial pair','d','{i}','none'), + ('LP: loop / stack to k','t(i,k)vₖ','{i}','k'), + ('noLP: initial stack','s d, s=t(i,i⁺)','{i, i⁺}','none'), + ('noLP: direct stacking','s vᵢ⁺','{i}','i⁺'), + ('noLP: nonstack loop to k','s t(i⁺,k)vₖ','{i, i⁺}','k'), +] +for row,y in zip(rows,range(608,809,44)): + s.text(74,y,row[0],19) + s.eq(570,y,row[1],22) + s.eq(929,y,row[2],22) + s.eq(1190,y,row[3],22) +s.text(73,830,'Native pair, loop/span and right-end GU constraints apply; noLP loops exclude the adjacent child k=i⁺+(1,1).',17) +s.box(48,882,1344,207,'2 Initialize mass and flow from every admitted candidate',fill='#edf6f2',color=TEAL) +s.eq(73,950,'w = B(i,j)v; Z += w; M(p) += w for p ∈ S; F(k) += w if a child exists',23) +s.lines(73,995,[ + 'M starts at zero; F is accumulated incoming weight for a retained child chain. This includes unselected candidates.', + 'Use exactly updateZ() admission: !Z_equal(v,0), complete-site GU / ED filters, and finite arithmetic.', + 'The current candidate owns S; every remaining pair belongs to its selected child continuation.' +],18) +s.box(48,1119,1344,249,'3 Propagate incoming mass through selected chains') +s.rna(95,1221,368,[(0,'i'),(.48,'k'),(1,'jₖ')],shade=(.48,1,'C(k)')) +s.lines(521,1195,[ + 'Sweep i in increasing order. Let Sᵢ and kᵢ describe C(i).', + 'M(p) += F(i) for every p ∈ Sᵢ;', + 'F(kᵢ) += F(i) when C(i) has a child.', + 'Every edge increases both indices, so all parents arrive first.' +],21,step=36) +s.text(73,1333,'No division by vᵢ is needed: a retained cell represents one chain, so its full incoming weight follows every pair.',18) +s.box(48,1397,1344,223,'4 Publish marginals of the retained candidate ensemble',fill='#edf6f2',color=TEAL) +s.eq(73,1465,'P(i,j) = ∑ M_region(i,j) / ∑ Z_region',27) +s.lines(73,1510,[ + 'Map energy coordinates back to original strand order, merge disjoint regions, then validate and normalize.', + 'The denominator must equal the unchanged heuristic Zall. Mark the result approximate, including empty results.', + 'Missing continuation chains are not recovered. These marginals differ from the exact --mode=M ensemble.' +],18) +s.text(48,1660,'Only actual paired nucleotides receive mass; positions merely enclosed by a retained interaction remain unmarked.',18) +s.footer('PredictorMfeEns2dHeuristic.cpp: fillHybridZ(), recordProbabilityCandidate(), commitProbabilityRegion()', + 'BasePairProbabilities: markApproximate(), addRegion(), finalize(); PredictorMfeEns.cpp: updateZ()') +s.save() + +for name in records: + path=DEST/name + doc=ET.parse(path) + assert b'\r' not in path.read_bytes() + for node in doc.iter(): + assert node.tag.rsplit('}',1)[-1] not in ('image','script') + cairosvg.svg2png(url=str(path),write_to=str(OUT/(name+'.png'))) +print('\n'.join(records)) diff --git a/doc/local-validation-archive/files/issue-258-validation/step9/build_recursions.py b/doc/local-validation-archive/files/issue-258-validation/step9/build_recursions.py new file mode 100644 index 00000000..8b312dfc --- /dev/null +++ b/doc/local-validation-archive/files/issue-258-validation/step9/build_recursions.py @@ -0,0 +1,305 @@ +from pathlib import Path +from html import escape +import json +from PIL import ImageFont +from fontTools.ttLib import TTFont +SERIF_GLYPHS = TTFont("/usr/share/fonts/truetype/dejavu/DejaVuSerif.ttf").getBestCmap() + +DEST = Path('/home/alex/CodexProjects/IntaRNA/pr-worktrees/issue-258-bpsvg/doc/recursions') +BLUE, TEAL, ORANGE, INK = '#275f8c', '#23796a', '#bd6518', '#203244' +records = [] + + +class SVG: + def __init__(self, name, title, subtitle, height=1150): + self.name, self.height = name, height + self.parts = [f'', + f'{escape(title)}', + f'{escape(subtitle)}. Internal recurrence documentation for step 9 of PR 258. Equations describe the current implementation; RNA 2 uses reversed energy coordinates.', + '', + '', + ''] + self.text(48, 48, 'INTARNA / RECURRENCE NOTES', 14, BLUE, bold=True) + self.text(48, 91, title, 31, bold=True) + self.text(48, 125, subtitle, 18) + self.line(48, 148, 1392, 148) + + def text(self, x, y, text, size=19, color=INK, anchor='start', bold=False, math=False): + family = 'DejaVu Serif' if math else 'DejaVu Sans' + font = ImageFont.truetype('/usr/share/fonts/truetype/dejavu/'+('DejaVuSerif' if math else 'DejaVuSans')+('-Bold' if bold else '')+'.ttf', size) + width = font.getlength(text) + left = x - (width/2 if anchor == 'middle' else width if anchor == 'end' else 0) + assert left >= 16 and left + width <= 1424, (self.name, text, left, width) + assert 15 <= y <= self.height-12, (self.name, text, y) + rendered = ''.join(escape(c) if not math or ord(c) in SERIF_GLYPHS else f'{escape(c)}' for c in text) + self.parts.append(f'{rendered}') + + def lines(self, x, y, lines, size=19, step=29, **kw): + for i, line in enumerate(lines): + self.text(x, y+i*step, line, size, **kw) + + def line(self, x1, y1, x2, y2, color='#c4d0da', width=1.2, arrow=False, dash=False): + self.parts.append(f'') + + def box(self, x, y, w, h, title=None, fill='#f3f7fa', color=BLUE): + self.parts.append(f'') + if title: + self.text(x+22, y+34, title, 21, color, bold=True) + + def eq(self, x, y, text, size=25): + self.text(x, y, text, size, math=True) + + def rna(self, x, y, w, labels, shade=None, seed=None): + # Pair positions along antiparallel strands; both internal indices increase. + if shade: + a,b,label=shade + self.parts.append(f'') + self.text(x+w*(a+b)/2, y+37, label, 24, BLUE, anchor='middle', math=True) + if seed: + a,b=seed + self.parts.append(f'') + self.line(x, y, x+w, y, INK, 2) + self.line(x, y+60, x+w, y+60, INK, 2) + for fraction,label in labels: + px=x+w*fraction + self.line(px,y,px,y+60,ORANGE if seed and seed[0]<=fraction<=seed[1] else INK,2.4) + if label: + self.text(px,y-12,label,17,anchor='middle',math=True) + + def footer(self, *sources): + self.line(48, self.height-95, 1392, self.height-95) + self.text(48,self.height-71,'Implementation anchors (src/IntaRNA/) · BW(E)=exp(−E/RT) after native energy-unit conversion',14,BLUE,bold=True) + for i,source in enumerate(sources): + self.text(48,self.height-48+21*i,source,13) + + def save(self): + (DEST/self.name).write_text('\n'.join(self.parts+[''])+'\n') + records.append(self.name) + + +# 1. Fixed-right-boundary unseeded partition. +s=SVG('IntaRNA.PredictorMfeEns2d.svg','Unseeded ensemble · fixed right boundary','PredictorMfeEns2d | --model=P --mode=M --noSeed',1190) +s.lines(48,184,['Pair indices i=(i₁,i₂), k and j increase on both strands; RNA 2 is reversed.', + 'i ≺ k means strict increase in both coordinates. d=BW(E_init), t(i,k)=BW(E_interLeft(i,k)).']) +s.eq(48,253,'Hⱼ(i) = hybrid partition for all admitted chains from i to a fixed j',25) +for x,title in [(48,'Initiation'),(500,'Direct pair-to-pair step'),(952,'Recursive continuation')]: + s.box(x,285,440,225,title) +s.rna(106,364,260,[(.5,'i = j')]);s.eq(268,477,'d',28) +s.rna(553,364,330,[(0,'i'),(1,'j')]);s.eq(607,477,'t(i,j) · d',27) +s.rna(1001,364,330,[(0,'i'),(.4,'k'),(1,'j')],shade=(.4,1,'Hⱼ(k)'));s.eq(1001,477,'t(i,k) · Hⱼ(k)',25) +s.text(48,554,'Lonely pairs allowed',23,BLUE,bold=True) +s.eq(48,599,'Hⱼ(j)=d; Hⱼ(i)=t(i,j)d + ∑ t(i,k)Hⱼ(k) (i ≺ k ≺ j)',25) +s.lines(48,639,['All other or noncomplementary states are zero. Sum only loop-size-admissible steps.', + 'For every valid right pair j, fill i in decreasing order within the span limits.']) +s.box(48,697,1344,231,'noLP: the left boundary must stack to i⁺ = i + (1,1)',fill='#edf6f2',color=TEAL) +s.eq(72,776,'Hⱼ(j)=0; Hⱼ(i)=t(i,i⁺) · [ (d if i⁺=j, else Hⱼ(i⁺)) + ∑ t(i⁺,k)Hⱼ(k) ]',24) +s.lines(72,821,['The sum requires i⁺ ≺ k ≺ j and k ≠ i⁺+(1,1): the direct stack is already in Hⱼ(i⁺).', + 'Require a complementary i⁺. Unreachable suffixes are zero; the adjacent two-pair case contributes t(i,j)d.', + 'Thus initialization is a stack, and each nonstack continuation starts another stacked suffix.'],18) +s.eq(48,978,'Zall += B(i,j) · Hⱼ(i); E_site = energy.getE(i,j, energy.getE(Hⱼ(i)))',24) +s.lines(48,1017,['B(i,j)=BW(energy.getE(i₁,j₁,i₂,j₂,0)) for admitted boundaries; otherwise B=0.', + 'This adds full-interval ED and terminal terms once; updateCompleteZ submits each boundary once.'],18) +s.footer('PredictorMfeEns2d.cpp: predict(), fillHybridZ()', 'PredictorMfeEns.cpp: updateCompleteZ(), addPartitionContribution()');s.save() + +# 2. One successor retained at each left boundary. +s=SVG('IntaRNA.PredictorMfeEns2dHeuristic.svg','Unseeded ensemble · heuristic continuation','PredictorMfeEns2dHeuristic | --model=P --mode=H --noSeed',1140) +s.lines(48,185,['C(i)=(vᵢ,jᵢ) retains one hybrid weight and right boundary for each left pair i.', + 'Use d=BW(E_init), t(i,k)=BW(E_interLeft(i,k)); all pair coordinates increase internally.']) +s.text(48,241,'B(i,j)=BW(energy.getE(i₁,j₁,i₂,j₂,0)) is the complete-boundary Boltzmann factor.',17) +s.box(48,254,1344,247,'Enumerate candidates at i, visiting i in decreasing order') +s.rna(82,349,375,[(0,'i'),(.4,'k'),(1,'jₖ')],shade=(.4,1,'vₖ')) +s.lines(505,319,['Lonely pairs allowed:',' initial candidate (d, i), only if !noGUend or i is not GU', + ' continuation (t(i,k) · vₖ, jₖ) for each reachable k ≻ i', + ' within loop limits and the complete span limits.'],21,step=38) +s.line(712,510,712,561,arrow=True,width=2) +s.box(48,581,645,178,'Accumulate every considered candidate',fill='#edf6f2',color=TEAL) +s.eq(71,659,'updateZ(i,j,v,true)',26) +s.lines(71,697,['For admitted complete sites: add B(i,j)v to Zall', + 'and v to the boundary map; otherwise discard the update.'],18) +s.box(720,581,672,178,'Retain only the best continuation at i') +s.eq(743,659,'C(i) ← argmin E_full(i,j,v)',25) +s.lines(743,697,['Compare complete energy for each candidate.', + 'Strict improvement replaces vᵢ and jᵢ.'],18) +s.box(48,795,1344,188,'noLP candidate set',fill='#fff7ed',color=ORANGE) +s.lines(72,859,['Let i⁺=i+(1,1) be complementary and s=t(i,i⁺).', + 'Initialize (s·d, i⁺) only if !noGUend or i⁺ is not GU; direct continuation is (s·vᵢ⁺, jᵢ⁺).', + 'Other candidates are (s·t(i⁺,k)·vₖ, jₖ), with k ≻ i⁺ and k ≠ i⁺+(1,1).', + 'The adjacent continuation is excluded from the loop sum because it was handled directly.'],18) +s.lines(48,1020,['The boundary map sums contributions, but the DP cell keeps one right continuation: this is a restricted ensemble.'],18) +s.footer('PredictorMfeEns2dHeuristic.cpp: fillHybridZ()', 'PredictorMfeEns.cpp: updateZ(), updateOptimaUsingZ()');s.save() + +# 3. Seed-only mode is enumeration, not an all-seed-substructure partition. +s=SVG('IntaRNA.PredictorMfeEnsSeedOnly.svg','Seed-only ensemble · retained seed enumeration','PredictorMfeEnsSeedOnly | --model=P --mode=S',990) +s.box(48,181,1344,160,'1 Fill the seed handler for the active region') +s.lines(72,246,['Retain the handler-selected seed at each admitted start a; its end is e.', + 'Both endpoints must lie in the region. The handler supplies seed pairs and hybrid energy E_seed(a).', + 'A filled handler with no admitted seeds reports an empty result.'],19) +s.line(720,353,720,402,arrow=True,width=2) +s.box(48,420,1344,240,'2 One complete contribution per retained seed',fill='#fff7ed',color=ORANGE) +s.rna(89,524,391,[(0,'a'),(.25,''),(.5,''),(.75,''),(1,'e')],seed=(0,1)) +s.eq(540,522,'H_seed = E_seed(a) + E_init',26) +s.eq(540,566,'E_full = energy.getE(a₁,e₁,a₂,e₂,H_seed)',24) +s.eq(540,605,'Zall += BW(E_full)',23) +s.text(540,637,'Only when needZall is enabled and complete-site filters pass.',18) +s.box(48,702,1344,154,'3 Report and trace the retained seed') +s.lines(72,765,['updateOptima(..., isHybridE=true, incrementZall=true) applies the complete-site filters and ranking.', + 'traceBackSeed() supplies its interior pairs; initiation is charged exactly once.', + 'There is no extension recurrence, no sum over alternative seed structures at the same start, and no bpProb output.'],18) +s.footer('PredictorMfeEnsSeedOnly.cpp: predict(), traceBack()', 'PredictorMfe.cpp: updateOptima(); Predictor.h: updateZall() (inherited update path)');s.save() + +# 4. Exact stack-seed adapter and ownership, including legacy dispatch. +s=SVG('IntaRNA.PredictorMfeEns2dSeedExtension.svg','Seeded ensemble · exact stack-seed adapter','PredictorMfeEns2dSeedExtension | --model=P --mode=M with stack-only seed capability',1190) +s.box(48,177,645,128,'SeedHandler: guaranteesStackOnlySeeds()') +s.lines(71,238,['true → predictStackSeeds(): unique seeded-chain partition', + 'false → legacy anchor extensions; no pair-probability sink'],18) +s.box(720,177,672,128,'Capability is geometric',fill='#fff7ed',color=ORANGE) +s.lines(743,238,['Computed or explicit seeds; mixed lengths and singletons.', + 'Preserve handler admission and per-start selection rules.'],18) +s.box(48,344,1344,223,'1 Compile active seed occurrences and eligible left boundaries') +s.rna(95,438,398,[(0,'p'),(.40,'a'),(.57,''),(.72,'b'),(1,'q')],seed=(.4,.72)) +s.eq(546,433,'max(0, bₖ − Wₖ + 1) ≤ pₖ ≤ aₖ (k=1,2)',24) +s.lines(546,478,['Admitted stack a…b must fit both the region and span limits W.', + 'Enumerate the union of these p-boxes once, then require a valid p.', + 'At p, clip the working grid to both inclusive spans.'],18) +s.box(48,605,1344,227,'2 Give the suffix kernel one consistent energy objective',fill='#edf6f2',color=TEAL) +s.eq(73,677,'d(p)=BW(E_init); t(r,q)=BW(E_interLeft(r₁,q₁,r₂,q₂))',25) +s.eq(73,726,'B(p,q)=BW(energy.getE(p₁,q₁,p₂,q₂,0)) if the complete site is admitted',24) +s.lines(73,766,['Invalid / inaccessible / noncomplementary vertices and forbidden transitions cannot contribute.', + 'B owns complete-span ED, native dangles, terminal penalties and energyAdd. Seed energies only admit seeds.'],18) +s.box(48,870,1344,173,'3 Commit a completed region') +s.eq(73,937,'Z_region = ∑ B(p,q) · T₁,p(q)',27) +s.lines(580,915,['Complete boundary callback and outside initialization share exactly the same B.', + 'updateExactCompleteZ preserves the partition objective.', + 'Pair masses and optional seed masks map through energy.getBasePair().', + 'Only a successful computation reaches BasePairProbabilities::addRegion().'],17,step=29) +s.footer('PredictorMfeEns2dSeedExtension.cpp: predict(), predictStackSeeds(), exactBoundaryWeight()', 'StackSeedDomain.cpp; SeededPartitionFunction.cpp; PredictorMfeEns.cpp: updateExactCompleteZ()');s.save() + +# 5. Forward state machine used in production. +s=SVG('IntaRNA.SeededPartitionFunction.inside.svg','Exact seeded inside recurrence · suffix automaton','SeededPartitionFunction::compute() | fixed left boundary p; visit q in increasing pair order',1475) +s.lines(48,185,['C=max(2, longest admitted seed length). ℓ is the current consecutive stack-suffix length, capped at C.', + 'e(q)=shortest admitted seed ending at q, or 0. A run accepts a seed iff 0 < e(q) ≤ ℓ.']) +for x,w,title,lines in [(48,436,'Uℓ: still seed-free',['ℓ=1…C; no seed has occurred.', 'Stack increments ℓ; a loop resets it.']), + (502,436,'S₁: seeded, singleton suffix',['A seed has occurred; current run is length 1.', 'Under noLP, must stack before leaving.']), + (956,436,'S₂: seeded, usable suffix',['Under noLP: current run has length ≥2.', 'Otherwise: all seeded suffixes use S₂.'])]: + s.box(x,250,w,147,title,fill='#f3f7fa' if x==48 else '#fff7ed',color=BLUE if x==48 else ORANGE) + s.lines(x+20,316,lines,16,step=28) +s.line(266,397,266,422,BLUE,1.6) +s.line(266,422,1174,422,BLUE,1.6) +s.line(1174,422,1174,397,BLUE,1.6,arrow=True) +s.text(720,417,'first seed accepted on a stack',16,BLUE,anchor='middle') +s.text(720,377,'stack → S₂',16,ORANGE,anchor='middle') +s.parts.append('') +s.box(48,437,1344,221,'1 Initialize and extend every state') +s.eq(73,505,'H(p,c₀)=d(p); other H=0. c₀ = accept(p,1) ? (noLP ? S₁ : S₂) : U₁',23) +s.eq(73,553,'Stack r=q−(1,1): H(q,D(c,stack)) += t(r,q) · H(r,c)',25) +s.lines(73,599,['For Uℓ: let ℓ′=min(C,ℓ+1); D=Uℓ′ unless the ending seed is accepted, then D=S₂.', + 'For S₁ or S₂: D=S₂. Once seeded, a later seed never creates a second copy of the path.'],18) +s.box(48,697,1344,260,'2 Add nonstack arrivals, then aggregate eligible departures',fill='#edf6f2',color=TEAL) +s.eq(73,764,'For each nonstack r→q and b∈{0,1}: H(q,D(b,loop)) += t(r,q) · T_b(r)',24) +s.eq(73,809,'T₀(q)=∑ H(q,Uℓ) over ℓ≥(noLP ? 2 : 1)',25) +s.eq(73,854,'T₁(q)=H(q,S₂) + (noLP ? 0 : H(q,S₁))',25) +s.lines(73,891,['T_b(r) is already computed. Nonstack: strict increase within loop limits, excluding r=q−(1,1).', + 'Reset the suffix to 1; remain seeded if b=1, or accept a singleton seed at q if b=0.'],18) +s.box(48,995,1344,174,'3 Complete seeded paths at every q',fill='#fff7ed',color=ORANGE) +s.eq(73,1064,'Z += B(p,q) · T₁(q)',29) +s.lines(532,1049,['Seed-free states never terminate an interaction.', + 'With noLP, singleton runs cannot terminate or cross a nonstack edge.', + 'Such states remain available for a later stack.'],18) +s.lines(48,1214,['Loop destinations: seeded → S₁ under noLP, otherwise S₂; unseeded → U₁ unless a singleton seed accepts.', + 'The Boolean seeded state counts each ordered pair chain once, including paths with several seed witnesses.', + 'Termination-only GU / ED filters belong to B; they do not discard prefixes that can still extend.'],18) +s.text(48,1305,'d, t and B are the initiation, transition and complete-boundary factors supplied by Weights.',17) +s.parts.append('') +s.footer('SeededPartitionFunction.cpp: accepts, dest, nonstack, h, total; StackSeedDomain::seedEndingLength()', 'Code states: Uℓ ↔ c=ℓ−1; S₁ ↔ c=cap; S₂ ↔ c=cap+1. Checked nonnegative arithmetic throughout.');s.save() + +# 6. Reverse differentiation and regional normalization. +s=SVG('IntaRNA.SeededPartitionFunction.outside.svg','Actual base-pair probabilities · outside recurrence','Reverse differentiation of the suffix inside computation; same fixed p and same boundary weights B',1610) +s.lines(48,185,['H and T are forward state/aggregate weights. Their adjoints are A=∂Z/∂H and G=∂Z/∂T.', + 'M(q) is the raw mass of complete chains containing actual pair q; it is not site-coverage mass.']) +s.rna(110,275,710,[(0,'p'),(.4,'r'),(.65,'q'),(1,'terminal')],shade=(0,.4,'H(r,c)')) +s.line(571.5,275,571.5,335,ORANGE,5) +s.lines(920,288,['Reverse the incoming edge r → q.', 'Only actual pair q receives this edge’s mass.'],18) +s.parts.append('') +s.box(48,248,1344,139,'1 Initialize the reverse objective') +s.eq(73,313,'A(q,S₂)=B(p,q); A(q,S₁)=B(p,q) only without noLP; other A and G = 0',23) +s.text(73,354,'Initialize at reachable positive seeded totals; a zero terminal coefficient contributes nothing.',18) +s.box(48,423,1344,197,'2 Reverse q in decreasing order; first scatter aggregate adjoints',fill='#edf6f2',color=TEAL) +s.eq(73,489,'A(q,Uℓ) += G₀(q) for the same departure-eligible ℓ as in the forward sum',24) +s.eq(73,534,'A(q,S₂) += G₁(q); A(q,S₁) += G₁(q) only without noLP',24) +s.text(73,576,'This scatter must precede reversing the incoming edges at q.',18) +s.box(48,659,645,274,'3a Reverse a stack edge r→q') +s.eq(73,724,'a = A(q,D(c,stack)) · t(r,q)',24) +s.eq(73,770,'A(r,c) += a',27) +s.eq(73,816,'M(q) += a · H(r,c)',27) +s.lines(73,860,['Repeat for every forward source state c.', + 'The transition marker belongs to destination q.'],18) +s.box(720,659,672,274,'3b Reverse a nonstack edge r→q') +s.eq(743,724,'a = A(q,D(b,loop)) · t(r,q)',24) +s.eq(743,770,'G_b(r) += a',27) +s.eq(743,816,'M(q) += a · T_b(r)',27) +s.lines(743,860,['Repeat for b=0 and b=1 using the same forward totals.', + 'Do not mark the unpaired nucleotides in the loop.'],18) +s.box(48,970,1344,130,'4 Mark the source pair after the reverse sweep',fill='#fff7ed',color=ORANGE) +s.eq(73,1043,'M(p) += A(p,c₀) · d(p)',28) +s.text(650,1043,'Every other pair was marked when its incoming edge was reversed.',18) +s.box(48,1139,1344,179,'5 Merge successful disjoint search regions before normalization') +s.eq(73,1212,'Z_total = ∑ Z_region; M_total(i,j) = ∑ M_region(i,j); P(i,j)=M_total(i,j)/Z_total',23) +s.lines(73,1255,['Map regional pairs with energy.getBasePair() into original target/query coordinates, then finalize explicitly.', + 'Z_total=0 → NA. Failed/cancelled computation → no matrix. No unbound-state weight is added.'],18) +s.text(48,1356,'Seed admission and B are fixed while differentiating. Masses sum across every eligible left boundary p.',18) +s.parts.append('') +s.footer('SeededPartitionFunction.cpp: dh (A), dt (G), mass and initial-pair contribution', 'PredictorMfeEns2dSeedExtension::predictStackSeeds(); BasePairProbabilities::addRegion(), finalize(), probabilities()');s.save() + +# 7. Legacy extension kernels remain shared with the heuristic seeded predictor. +s=SVG('IntaRNA.PredictorMfeEns2dSeedExtension.legacy.svg','Legacy seeded extensions · anchor factorization','Bulged-seed fallback in mode M; also the extension kernels used by the seeded mode H predictor',1410) +s.lines(48,185,['For one retained anchor seed a…b, use S=BW(E_seed(a)), d=BW(E_init), t(i,k)=BW(E_interLeft(i,k)).', + 'Lₐ(i) extends left to a (includes initiation); R_b(j) extends right from b (excludes initiation).']) +s.rna(120,278,1180,[(0,'i'),(.33,'a'),(.4,''),(.5,''),(.59,''),(.66,'b'),(1,'j')],shade=(0,.33,'Lₐ(i)'),seed=(.33,.66)) +s.parts.append('') +s.rna(899,278,401,[(0,''),(1,'')],shade=(0,1,'R_b(j)')) +s.eq(530,379,'hybrid contribution = Lₐ(i) · S · R_b(j)',25) +s.box(48,421,1344,178,'Lonely pairs allowed') +s.eq(73,488,'Lₐ(a)=d; Lₐ(i)=∑ t(i,k)Lₐ(k) − Cₐ(i) (i ≺ k ≼ a)',25) +s.eq(73,538,'R_b(b)=1; R_b(j)=∑ R_b(k)t(k,j) (b ≼ k ≺ j)',25) +s.text(73,577,'Only nonzero, complementary states and loop-size-admissible terms participate. Sums use componentwise bounds.',17) +s.box(48,638,1344,245,'noLP: outward boundary stacks are explicit',fill='#edf6f2',color=TEAL) +s.eq(73,707,'Lₐ(i)=t(i,i⁺) · [Lₐ(i⁺) + ∑ t(i⁺,k)Lₐ(k)] − Cₐ(i) (i⁺ ≺ k ≼ a)',24) +s.eq(73,762,'R_b(j)=t(j⁻,j) · [1{j⁻=b}R_b(b) + ∑ R_b(k)t(k,j⁻)] (b ≼ k ≺ j⁻)',24) +s.lines(73,808,['i⁺=i+(1,1), j⁻=j−(1,1); require the extra stacked pair. Anchor bases still have Lₐ(a)=d and R_b(b)=1.', + 'These are the legacy loop bounds. The exact stack-seed path instead uses the suffix-state recurrence.'],18) +s.box(48,923,1344,212,'Left-seed correction Cₐ(i)',fill='#fff7ed',color=ORANGE) +s.lines(73,988,['At a competing seed start i: process it if its end is left of a, or it loop-overlaps the anchor.', + 'If a leftmost loop-overlapping seed h is found, subtract BW(getNonOverlappingEnergy(i,h)) · Lₐ(h).', + 'Otherwise subtract its seed weight times the continuation at its end; noLP adds explicit loop continuations.', + 'Apply the implementation’s overlap/loop guards, then clamp a negative corrected L value to zero.'],18,step=30) +s.eq(48,1185,'updateZ(i,j, Lₐ(i) · S · R_b(j), true) for each nonzero span-admissible combination',23) +s.lines(48,1231,['Complete-site filters and B(i,j) are applied by updateZ. The anchor’s hybrid seed energy is charged once.', + 'This legacy subtraction-based path has no actual-pair outside pass; do not substitute it for the exact seeded union.'],18) +s.footer('PredictorMfeEns2dSeedExtension.cpp: predict() legacy branch, fillHybridZ_left(), fillHybridZ_right()', 'PredictorMfeEns2dSeedExtension.cpp: getNonOverlappingEnergy(); PredictorMfeEns.cpp: updateZ()');s.save() + +# 8. Restriction to one optimum right extension per anchor seed. +s=SVG('IntaRNA.PredictorMfeEns2dHeuristicSeedExtension.svg','Seeded ensemble · one selected right extension','PredictorMfeEns2dHeuristicSeedExtension | --model=P --mode=H',1200) +s.lines(48,185,['For anchor seed a…b: S=BW(E_seed), d=BW(E_init), L=Lₐ(i), R=R_b(j).', + 'L and R use the inherited legacy extension recurrences, even when the seed handler is stack-only.']) +s.box(48,249,1344,198,'1 Compute R, submit true right extensions, and select j*') +s.eq(73,317,'updateZ(a,j, d · S · R_b(j), true) for j ≻ b',25) +s.eq(73,366,'j* = argmin E_full(a,j, d · S · R_b(j)) over right-end-admissible candidates',24) +s.text(73,411,'updateOptRightZ compares complete energy and the right GU-end restriction; ties retain the first candidate.',18) +s.line(720,458,720,504,arrow=True,width=2) +s.box(48,524,1344,288,'2 Compute corrected L and submit left-side combinations',fill='#edf6f2',color=TEAL) +s.rna(92,621,495,[(0,'i'),(.33,'a'),(.45,''),(.60,'b'),(1,'j*')],shade=(0,.33,'L'),seed=(.33,.6)) +s.eq(650,603,'updateZ(i,b, Lₐ(i) · S, true)',25) +s.text(650,643,'Includes the seed-only case i=a.',18) +s.eq(650,695,'updateZ(i,j*, Lₐ(i) · S · R_b(j*), true)',23) +s.lines(650,736,['Only for a true left extension and a true selected right extension,', + 'with both complete spans within their limits.'],17) +s.box(48,852,1344,185,'Candidate family per anchor',fill='#fff7ed',color=ORANGE) +s.lines(73,916,['All true right-only extensions + all corrected left-only extensions + left extensions paired with just j*.', + 'The second phase runs if noGUend is off, or a valid right optimum exists, or anchor end b is not GU.', + 'Initialize j*=b and E_right_opt=∞. A positive selected R is required before combining both sides.', + 'No full Cartesian product of left and right boundaries is retained, and bpProb / bpsvg are unsupported.'],18) +s.text(48,1082,'Every update is filtered and accumulated by PredictorMfeEns::updateZ; ranked reporting is a separate step.',18) +s.footer('PredictorMfeEns2dHeuristicSeedExtension.cpp: predict(), fillHybridZ_right(), fillHybridZ_left()', 'PredictorMfeEns2dHeuristicSeedExtension.h: updateOptRightZ(); inherited legacy fillHybridZ kernels');s.save() + +Path(__file__).with_name('new-svg-files.json').write_text(json.dumps(records,indent=2)+'\n') +print(f'Wrote {len(records)} standalone SVG recurrence depictions') diff --git a/doc/local-validation-archive/files/issue-259-revision-validation/PredictorMfeEns2d-consumer.cpp b/doc/local-validation-archive/files/issue-259-revision-validation/PredictorMfeEns2d-consumer.cpp new file mode 100644 index 00000000..de61a053 --- /dev/null +++ b/doc/local-validation-archive/files/issue-259-revision-validation/PredictorMfeEns2d-consumer.cpp @@ -0,0 +1,31 @@ +// Standalone installed-library example; see README.md#bpProb. +#include +#include +#include +#include +#include +#include +#include + +INITIALIZE_EASYLOGGINGPP + +int main() { + using namespace IntaRNA; + RnaSequence target("target", "GGAGGG"), query("query", "CCCC"); + AccessibilityDisabled targetAcc(target, 0, nullptr), queryAcc(query, 0, nullptr); + ReverseAccessibility reversed(queryAcc); + InteractionEnergyBasePair energy(targetAcc, reversed); + SeedConstraint seeds(2, 0, 0, 0, E_INF, Accessibility::ED_UPPER_BOUND, + E_INF, IndexRangeList(), IndexRangeList(), "", false, false, false); + OutputConstraint constraints(1, OutputConstraint::OVERLAP_BOTH, E_INF, E_INF, + false, false, false, true, false); // needZall=true; needBPs=false + OutputHandlerInteractionList output(constraints, 1); + BasePairProbabilities probabilities(target.size(), query.size()); + PredictorMfeEns2d predictor(energy, output, nullptr, + &probabilities); + predictor.predict(); + probabilities.finalize(); + if(probabilities.isApproximate() != false) return 2; + BasePairProbabilityWriter::write(std::cout, probabilities, target, query); + BasePairProbabilityWriter::writeSvg(std::cerr, probabilities, energy, 1, 1); +} diff --git a/doc/local-validation-archive/files/issue-259-revision-validation/PredictorMfeEns2dHeuristic-consumer.cpp b/doc/local-validation-archive/files/issue-259-revision-validation/PredictorMfeEns2dHeuristic-consumer.cpp new file mode 100644 index 00000000..30000920 --- /dev/null +++ b/doc/local-validation-archive/files/issue-259-revision-validation/PredictorMfeEns2dHeuristic-consumer.cpp @@ -0,0 +1,31 @@ +// Standalone installed-library example; see README.md#bpProb. +#include +#include +#include +#include +#include +#include +#include + +INITIALIZE_EASYLOGGINGPP + +int main() { + using namespace IntaRNA; + RnaSequence target("target", "GGAGGG"), query("query", "CCCC"); + AccessibilityDisabled targetAcc(target, 0, nullptr), queryAcc(query, 0, nullptr); + ReverseAccessibility reversed(queryAcc); + InteractionEnergyBasePair energy(targetAcc, reversed); + SeedConstraint seeds(2, 0, 0, 0, E_INF, Accessibility::ED_UPPER_BOUND, + E_INF, IndexRangeList(), IndexRangeList(), "", false, false, false); + OutputConstraint constraints(1, OutputConstraint::OVERLAP_BOTH, E_INF, E_INF, + false, false, false, true, false); // needZall=true; needBPs=false + OutputHandlerInteractionList output(constraints, 1); + BasePairProbabilities probabilities(target.size(), query.size()); + PredictorMfeEns2dHeuristic predictor(energy, output, nullptr, + &probabilities); + predictor.predict(); + probabilities.finalize(); + if(probabilities.isApproximate() != true) return 2; + BasePairProbabilityWriter::write(std::cout, probabilities, target, query); + BasePairProbabilityWriter::writeSvg(std::cerr, probabilities, energy, 1, 1); +} diff --git a/doc/local-validation-archive/files/issue-259-revision-validation/PredictorMfeEns2dHeuristicSeedExtension-consumer.cpp b/doc/local-validation-archive/files/issue-259-revision-validation/PredictorMfeEns2dHeuristicSeedExtension-consumer.cpp new file mode 100644 index 00000000..ecc322ed --- /dev/null +++ b/doc/local-validation-archive/files/issue-259-revision-validation/PredictorMfeEns2dHeuristicSeedExtension-consumer.cpp @@ -0,0 +1,31 @@ +// Standalone installed-library example; see README.md#bpProb. +#include +#include +#include +#include +#include +#include +#include + +INITIALIZE_EASYLOGGINGPP + +int main() { + using namespace IntaRNA; + RnaSequence target("target", "GGAGGG"), query("query", "CCCC"); + AccessibilityDisabled targetAcc(target, 0, nullptr), queryAcc(query, 0, nullptr); + ReverseAccessibility reversed(queryAcc); + InteractionEnergyBasePair energy(targetAcc, reversed); + SeedConstraint seeds(2, 0, 0, 0, E_INF, Accessibility::ED_UPPER_BOUND, + E_INF, IndexRangeList(), IndexRangeList(), "", false, false, false); + OutputConstraint constraints(1, OutputConstraint::OVERLAP_BOTH, E_INF, E_INF, + false, false, false, true, false); // needZall=true; needBPs=false + OutputHandlerInteractionList output(constraints, 1); + BasePairProbabilities probabilities(target.size(), query.size()); + PredictorMfeEns2dHeuristicSeedExtension predictor(energy, output, nullptr, + new SeedHandlerNoBulge(energy, seeds), &probabilities); + predictor.predict(); + probabilities.finalize(); + if(probabilities.isApproximate() != true) return 2; + BasePairProbabilityWriter::write(std::cout, probabilities, target, query); + BasePairProbabilityWriter::writeSvg(std::cerr, probabilities, energy, 1, 1); +} diff --git a/doc/local-validation-archive/files/issue-259-revision-validation/PredictorMfeEns2dSeedExtension-consumer.cpp b/doc/local-validation-archive/files/issue-259-revision-validation/PredictorMfeEns2dSeedExtension-consumer.cpp new file mode 100644 index 00000000..5837ef1e --- /dev/null +++ b/doc/local-validation-archive/files/issue-259-revision-validation/PredictorMfeEns2dSeedExtension-consumer.cpp @@ -0,0 +1,31 @@ +// Standalone installed-library example; see README.md#bpProb. +#include +#include +#include +#include +#include +#include +#include + +INITIALIZE_EASYLOGGINGPP + +int main() { + using namespace IntaRNA; + RnaSequence target("target", "GGAGGG"), query("query", "CCCC"); + AccessibilityDisabled targetAcc(target, 0, nullptr), queryAcc(query, 0, nullptr); + ReverseAccessibility reversed(queryAcc); + InteractionEnergyBasePair energy(targetAcc, reversed); + SeedConstraint seeds(2, 0, 0, 0, E_INF, Accessibility::ED_UPPER_BOUND, + E_INF, IndexRangeList(), IndexRangeList(), "", false, false, false); + OutputConstraint constraints(1, OutputConstraint::OVERLAP_BOTH, E_INF, E_INF, + false, false, false, true, false); // needZall=true; needBPs=false + OutputHandlerInteractionList output(constraints, 1); + BasePairProbabilities probabilities(target.size(), query.size()); + PredictorMfeEns2dSeedExtension predictor(energy, output, nullptr, + new SeedHandlerNoBulge(energy, seeds), &probabilities); + predictor.predict(); + probabilities.finalize(); + if(probabilities.isApproximate() != false) return 2; + BasePairProbabilityWriter::write(std::cout, probabilities, target, query); + BasePairProbabilityWriter::writeSvg(std::cerr, probabilities, energy, 1, 1); +} diff --git a/doc/local-validation-archive/files/issue-259-revision-validation/benchmark_revision.py b/doc/local-validation-archive/files/issue-259-revision-validation/benchmark_revision.py new file mode 100644 index 00000000..426ab8d4 --- /dev/null +++ b/doc/local-validation-archive/files/issue-259-revision-validation/benchmark_revision.py @@ -0,0 +1,118 @@ +#!/usr/bin/env python3 +"""Compare existing prediction modes and measure new outside/output overhead.""" +import hashlib +import json +import os +from pathlib import Path +import random +import shutil +import statistics +import subprocess +import time +import xml.etree.ElementTree as ET + +ROOT = Path('/home/alex/CodexProjects/IntaRNA') +REPO = ROOT/'pr-worktrees/issue-258-bpsvg' +HERE = ROOT/'issue-259-revision-validation' +OUT = HERE/'performance' +OUT.mkdir(exist_ok=True) +shutil.copy2(REPO/'src/bin/IntaRNA', OUT/'IntaRNA-final') +BINARIES = {'before': HERE/'IntaRNA-before', 'final': OUT/'IntaRNA-final'} +ENV = dict(os.environ, OMP_NUM_THREADS='1', OPENBLAS_NUM_THREADS='1', OMP_DYNAMIC='FALSE') +CPU = min(os.sched_getaffinity(0)) +COMMON = ['--model=P', '--seedBP=4', '--intLenMax=30', '--qIntLoopMax=3', '--tIntLoopMax=3', '--threads=1', '--outNoGUend=false', '--outNumber=0', '--outMode=E', '--default-log-file=/dev/null', '--out=STDOUT'] +PREDICTORS = {'noSeed-M': ['--mode=M', '--noSeed'], 'noSeed-H': ['--mode=H', '--noSeed'], 'seeded-H': ['--mode=H']} +CASES = {'OxyS-fhlA': [f'--target={REPO}/doc/handson/fhlA.fasta', f'--query={REPO}/doc/handson/OxyS.fasta', '--outNoLP=false']} +for name, length, no_lp in [('dense40', 40, False), ('dense32-noLP', 32, True)]: + for strand, nucleotide in [('target', 'G'), ('query', 'C')]: + (OUT/f'{name}-{strand}.fa').write_text(f'>{strand}\n{nucleotide*length}\n') + CASES[name] = [f'--target={OUT}/{name}-target.fa', f'--query={OUT}/{name}-query.fa', '--tAcc=N', '--qAcc=N', f'--outNoLP={str(no_lp).lower()}'] +CONFIGS = [('before', 'none'), ('final', 'none'), ('final', 'csv'), ('final', 'svg')] +records, checks, commands = [], [], {} +randomizer = random.Random(259) + + +def command(case, predictor, version, output, check=False): + cmd = ['taskset', '-c', str(CPU), str(BINARIES[version])] + COMMON + PREDICTORS[predictor] + CASES[case] + if output != 'none': + prefix, filename = ('bpProb', 'pairs.csv') if output == 'csv' else ('bpsvg', 'plot.svg') + destination = str(OUT/filename) if check else '/dev/null' + cmd.append(f'--out={prefix}:{destination}') + return cmd + + +def run(case, predictor, version, output, repetition): + check = repetition < 0 + cmd = command(case, predictor, version, output, check) + if not check: + commands[f'{case}/{predictor}/{version}/{output}'] = cmd + before = time.perf_counter() + completed = subprocess.run(['/usr/bin/time', '-f', '%e,%U,%S,%M', '-o', str(OUT/'time.txt')] + cmd, + stdout=subprocess.PIPE if check else subprocess.DEVNULL, stderr=subprocess.PIPE, env=ENV) + wall = time.perf_counter()-before + if completed.returncode: + raise RuntimeError(f'{case}/{predictor}/{version}/{output}: {completed.stderr.decode()}') + elapsed, user, system, rss = map(float, (OUT/'time.txt').read_text().strip().split(',')) + if not check: + records.append(dict(case=case, predictor=predictor, version=version, output=output, repetition=repetition, + wall_seconds=wall, user_seconds=user, system_seconds=system, cpu_seconds=user+system, rss_kib=int(rss))) + (OUT/'raw.json').write_text(json.dumps(records, indent=2)+'\n') + return completed.stdout + + +def check_plot(csv_data): + rows = [line.split(';') for line in csv_data.decode().splitlines()] + expected = {(row[0].rsplit('_', 1)[1], query.rsplit('_', 1)[1]): value for row in rows[1:] for query, value in zip(rows[0][1:], row[1:])} + count = 0 + for _, element in ET.iterparse(OUT/'plot.svg', events=['end']): + if element.get('data-type') == 'base-pair': + pair = (element.get('data-target'), element.get('data-query')) + actual, value = element.get('data-probability'), expected[pair] + assert (actual == value == 'NA') or float(actual) == float(value), (pair, actual, value) + count += 1 + element.clear() + assert count == len(expected), (count, len(expected)) + return count + + +for case in CASES: + for predictor in PREDICTORS: + print(f'Checking {case}/{predictor}', flush=True) + reference = None + csv_data = None + for version, output in CONFIGS: + ensemble = run(case, predictor, version, output, -1) + if reference is None: + reference = ensemble + assert ensemble == reference, (case, predictor, version, output, ensemble, reference) + if output == 'csv': + csv_data = (OUT/'pairs.csv').read_bytes() + assert csv_data.startswith(b'bpProbApproximate' if predictor.endswith('H') else b'bpProb;') + if output == 'svg': + count = check_plot(csv_data) + checks.append(dict(case=case, predictor=predictor, ensemble_sha256=hashlib.sha256(reference).hexdigest(), + ensemble=reference.decode(), cells=count, csv_bytes=len(csv_data), svg_bytes=(OUT/'plot.svg').stat().st_size)) + for repetition in range(7): + order = CONFIGS.copy() + randomizer.shuffle(order) + for version, output in order: + run(case, predictor, version, output, repetition) + (OUT/'checks.json').write_text(json.dumps(checks, indent=2)+'\n') + (OUT/'commands.json').write_text(json.dumps(commands, indent=2)+'\n') + print(f'Completed {case}/{predictor}', flush=True) + +summary = [] +for case in CASES: + for predictor in PREDICTORS: + for version, output in CONFIGS: + group = [row for row in records if (row['case'], row['predictor'], row['version'], row['output']) == (case, predictor, version, output)] + summary.append(dict(case=case, predictor=predictor, version=version, output=output, + wall_median=statistics.median(row['wall_seconds'] for row in group), + wall_min=min(row['wall_seconds'] for row in group), wall_max=max(row['wall_seconds'] for row in group), + cpu_median=statistics.median(row['cpu_seconds'] for row in group), rss_max_kib=max(row['rss_kib'] for row in group))) +(OUT/'summary.json').write_text(json.dumps(summary, indent=2)+'\n') +metadata = dict(cpu=CPU, repeats=7, binaries={name: dict(path=str(binary), sha256=hashlib.sha256(binary.read_bytes()).hexdigest()) for name, binary in BINARIES.items()}, + compiler='Same configured GCC 14 release toolchain; retained pre-revision executable versus final worktree executable.', + timing='Fresh processes; seven deterministically interleaved repetitions; one thread; CSV/SVG and ensemble sent to /dev/null for timings. Separate file-output warmups compare ensemble bytes and every CSV/SVG cell.') +(OUT/'metadata.json').write_text(json.dumps(metadata, indent=2)+'\n') +print(json.dumps(summary, indent=2), flush=True) diff --git a/doc/local-validation-archive/files/issue-259-revision-validation/check_legacy_seed_polynomials.py b/doc/local-validation-archive/files/issue-259-revision-validation/check_legacy_seed_polynomials.py new file mode 100644 index 00000000..61044ba4 --- /dev/null +++ b/doc/local-validation-archive/files/issue-259-revision-validation/check_legacy_seed_polynomials.py @@ -0,0 +1,82 @@ +#!/usr/bin/env python3 +"""Independent unit-weight chain-polynomial check of legacy left correction. + +Every polynomial coefficient is an integer multiplicity for an explicit pair +chain. This exploratory check is separate from the C++ reverse trace; it does +not prove general correctness or replace its API/oracle regressions. +""" +import json +from collections import defaultdict +from pathlib import Path +from random import Random + + +def add(out, src, prefix=(), scale=1): + for chain, weight in src.items(): + out[tuple(sorted(set(chain).union(prefix)))] += weight * scale + + +def calculate(anchor, seeds, length, no_lp, loop): + table = {} + for x in range(anchor[0], -1, -1): + for y in range(anchor[1], -1, -1): + start = (x, y) + current = defaultdict(int) + if start == anchor: + current[()] = 1 + elif x < anchor[0] and y < anchor[1]: + if no_lp: + add(current, table[x+1, y+1], [start]) + for k in range(x+1+no_lp, min(anchor[0], x+1+no_lp+loop)+1): + for l in range(y+1+no_lp, min(anchor[1], y+1+no_lp+loop)+1): + prefix = [start] + ([(x+1, y+1)] if no_lp else []) + add(current, table[k, l], prefix) + + def overlaps(a, b): + return 0 < b[0]-a[0] <= length-2 and b[0]-a[0] == b[1]-a[1] + + if start in seeds and ((x+length-1 <= anchor[0] and y+length-1 <= anchor[1]) or overlaps(start, anchor)): + overlapping = [p for p in seeds if p[0] <= anchor[0] and p[1] <= anchor[1] and overlaps(start, p)] + if overlapping: + child = min(overlapping) + add(current, table[child], [(x+d, y+d) for d in range(child[0]-x)], -1) + else: + end = (x+length-1, y+length-1) + add(current, table[end], [(x+d, y+d) for d in range(length-1)], -1) + if no_lp: + for k in range(end[0]+1, min(anchor[0], end[0]+loop+1)+1): + for l in range(end[1]+1, min(anchor[1], end[1]+loop+1)+1): + add(current, table[k, l], [(x+d, y+d) for d in range(length)], -1) + if sum(current.values()) < 0: + current = defaultdict(int) + table[start] = dict(current) + return table + + +def check(table): + for start, polynomial in table.items(): + total = sum(polynomial.values()) + masses = defaultdict(int) + for chain, coefficient in polynomial.items(): + for pair in chain: + masses[pair] += coefficient + if total > 0: + assert all(0 <= mass <= total for mass in masses.values()), (start, total, dict(masses), polynomial) + + +def main(): + random = Random(259) + for trial in range(500): + anchor = (4, 4) + length = random.choice([2, 3, 4]) + no_lp = random.choice([False, True]) + loop = random.choice([0, 1, 2, 4]) + seeds = {anchor} | {(i, j) for i in range(5) for j in range(5) if random.random() < 0.3} + check(calculate(anchor, seeds, length, no_lp, loop)) + result = dict(random_seed=259, trials=500, domain="5x5 all-complementary unit-weight pairs", seed_lengths=[2, 3, 4], loop_limits=[0, 1, 2, 4], no_lp=[False, True], result="All positive-Z cells had pair marginals in [0,Z].", limitation="Exploratory unit-weight polynomial check, not a proof or a C++ trace test.") + Path(__file__).with_suffix('.json').write_text(json.dumps(result, indent=2)+'\n') + print(json.dumps(result, indent=2)) + + +if __name__ == '__main__': + main() diff --git a/doc/local-validation-archive/files/issue-259-revision-validation/header-probe.cpp b/doc/local-validation-archive/files/issue-259-revision-validation/header-probe.cpp new file mode 100644 index 00000000..7e0238c2 --- /dev/null +++ b/doc/local-validation-archive/files/issue-259-revision-validation/header-probe.cpp @@ -0,0 +1,2 @@ +#include +int main() {return 0;} diff --git a/doc/local-validation-archive/files/issue-259-revision-validation/validate-installed.py b/doc/local-validation-archive/files/issue-259-revision-validation/validate-installed.py new file mode 100644 index 00000000..2acf0449 --- /dev/null +++ b/doc/local-validation-archive/files/issue-259-revision-validation/validate-installed.py @@ -0,0 +1,31 @@ +from pathlib import Path +import os,shlex,subprocess +root=Path('/home/alex/CodexProjects/IntaRNA') +repo=root/'pr-worktrees/issue-258-bpsvg' +work=root/'issue-259-revision-validation' +prefix=root/'issue-258-validation/install-release' +deps=root/'issue-246-validation/toolchain' +cxx=root/'issue-246-validation/toolchain-gcc14/bin/x86_64-conda-linux-gnu-g++' +common=[str(cxx),'-std=c++23','-fopenmp','-I'+str(prefix/'include'),'-I'+str(deps/'include')] +probe=work/'header-probe.cpp' +headers=sorted((prefix/'include/IntaRNA').glob('*.h')) +for header in ([] if os.environ.get("SKIP_HEADERS") else headers): + probe.write_text('#include \nint main() {return 0;}\n') + subprocess.run(common+['-fsyntax-only',str(probe)],check=True) +print(f'{len(headers)} installed headers passed',flush=True) +env=dict(os.environ,PKG_CONFIG_PATH=str(prefix/'lib/pkgconfig')) +flags=shlex.split(subprocess.check_output(['pkg-config','--cflags','--libs','IntaRNA'],env=env,text=True)) +original=(repo/'doc/seeded-probabilities-example.cpp').read_text() +for name in ['PredictorMfeEns2dSeedExtension','PredictorMfeEns2d','PredictorMfeEns2dHeuristic','PredictorMfeEns2dHeuristicSeedExtension']: + text=original.replace('PredictorMfeEns2dSeedExtension',name) + if not name.endswith('SeedExtension'): + text=text.replace('new SeedHandlerNoBulge(energy, seeds), &probabilities','&probabilities') + text=text.replace('probabilities.finalize();','probabilities.finalize();\n\tif(probabilities.isApproximate() != '+str('Heuristic' in name).lower()+') return 2;') + text=text.replace('BasePairProbabilityWriter::write(std::cout, probabilities, target, query);','BasePairProbabilityWriter::write(std::cout, probabilities, target, query);\n\tBasePairProbabilityWriter::writeSvg(std::cerr, probabilities, energy, 1, 1);') + source=work/(name+'-consumer.cpp'); source.write_text(text) + binary=work/(name+'-consumer') + subprocess.run(common+[str(source),'-L'+str(deps/'lib'),'-Wl,-rpath,'+str(deps/'lib'),*flags,'-o',str(binary)],check=True) + result=subprocess.run([str(binary)],capture_output=True,check=True) + assert result.stdout.startswith(b'bpProbApproximate;' if 'Heuristic' in name else b'bpProb;') + assert b' +#include +#include + +namespace IntaRNA { + +/** + * Generic Predictor interface for ensemble-based MFE interaction computation + * + * @author Martin Raden + * + */ +class PredictorMfeEns : public PredictorMfe { + + +public: + + /** + * Construction call the super constructor + * + * @param energy the interaction energy handler + * @param output the output handler to report optimal interactions to + * @param predTracker the prediction tracker to be used or NULL if no + * tracking is to be done; if non-NULL, the tracker gets deleted + * on this->destruction. + */ + PredictorMfeEns( const InteractionEnergy & energy + , OutputHandler & output + , PredictionTracker * predTracker ); + + virtual ~PredictorMfeEns(); + +protected: + + //! data container to encode a site with respective partition function + struct ZPartition { + size_t i1; + size_t j1; + size_t i2; + size_t j2; + Z_type partZ; + }; + + //! access to the interaction energy handler of the super class + using PredictorMfe::energy; + + //! access to the output handler of the super class + using PredictorMfe::output; + + //! access to the prediction tracker of the super class + using PredictorMfe::predTracker; + + //! map storing the partition of Zall for all considered interaction sites + std::unordered_map Z_partition; + + //! whether updateZ() currently receives one complete boundary partition + bool updateZisComplete; + + + //! a bucket supplies j1, so its key needs only the other three coordinates + struct FrontierBoundary { + size_t i1, i2, j2; + bool operator==(const FrontierBoundary &) const = default; + struct Hash { + size_t operator()(const FrontierBoundary & boundary) const; + }; + }; + + //! partial sums grouped by target right boundary; used only without trackers + std::vector> Z_frontier; + //! first right-boundary bucket that has not yet been finalized + size_t Z_frontierNext = 0; + + + /** + * Initializes the hybridization partition functions. + * Will be called by predict(). + */ + virtual + void + initZ(); + + /** + * Updates the local hybridization partition functions as well as Zall. + * + * Note: avoid other Zall updates via updateOptima(.., incrementZall=false)! + * + * Note: if called multiple time for the same boundaries then the + * reported partition functions have to represent disjoint interaction sets! + * + * @param i1 the index of the first sequence interacting with i2 + * @param j1 the index of the first sequence interacting with j2 + * @param i2 the index of the second sequence interacting with i1 + * @param j2 the index of the second sequence interacting with j1 + * @param partFunct the partition function of the interaction + * @param isHybridZ whether or not the given Z is only covering + * hybridization energy terms (init+loops) or the total + * interaction energy + */ + virtual + void + updateZ( const size_t i1, const size_t j1 + , const size_t i2, const size_t j2 + , const Z_type partFunct + , const bool isHybridZ ); + + /** + * Consumes a complete partition for one interaction boundary. Exact 2D + * prediction finalizes each boundary once and therefore does not need to + * retain it until reportOptima(). The virtual updateZ() hook is called so + * subclasses can observe or alter the update; overrides that delegate to the + * base implementation inherit streaming. Tracker-enabled predictions retain + * the partition so their established callback timing and ordering stay intact. + */ + void + updateCompleteZ( const size_t i1, const size_t j1 + , const size_t i2, const size_t j2 + , const Z_type partFunct + , const bool isHybridZ ); + + /** Enable partial-partition buckets by target right boundary. Tracker runs + * keep the original map to preserve callback ordering. Call advanceZFrontier + * only when no remaining seed can start below the supplied target index. + * @param size1 number of target positions in the current prediction region + */ + void beginZFrontier(size_t size1); + + /** Consume completed boundary sums without adding their mass to Zall again. + * @param minimumSeedStart smallest target start of any remaining seed + */ + void advanceZFrontier(size_t minimumSeedStart); + + /** + * Calls for the stored Z_partition information updateOptima() before + * calling reportOptima() from its super class. + */ + virtual + void + reportOptima(); + + /** + * Reports interaction boundaries only (no base pair tracking) + * @param interaction the interaction to be traced + */ + virtual + void + traceBack( Interaction & interaction ); + +private: + + /** + * Applies output filters, converts the partition representation and adds + * the accepted contribution to Zall. Returns the partition without ED. + */ + bool + addPartitionContribution( const size_t i1, const size_t j1 + , const size_t i2, const size_t j2 + , const Z_type partFunct + , const bool isHybridZ + , Z_type & partZ_noED ); + + /** + * Calls updateOptima() for each buffered entry of Z_partition. This hook is + * still called by reportOptima(), but complete tracker-free partitions were + * already consumed by updateZ() and are therefore not present in the map. + */ + virtual + void + updateOptimaUsingZ(); + +}; + +inline +size_t +PredictorMfeEns::FrontierBoundary::Hash::operator()(const FrontierBoundary & boundary) const +{ + size_t key=0; + boost::hash_combine(key,boundary.i1); + boost::hash_combine(key,boundary.i2); + boost::hash_combine(key,boundary.j2); + return key; +} + +} // namespace + +#endif /* INTARNA_PREDICTORMFEENS_H_ */ diff --git a/doc/local-validation-archive/files/issue-260-validation/installed-header.cpp b/doc/local-validation-archive/files/issue-260-validation/installed-header.cpp new file mode 100644 index 00000000..2237521d --- /dev/null +++ b/doc/local-validation-archive/files/issue-260-validation/installed-header.cpp @@ -0,0 +1,17 @@ +#include +#include +#include +#include +#include +INITIALIZE_EASYLOGGINGPP +int main() { + using namespace IntaRNA; + RnaSequence t("t","GGGGGGGG"), q("q","CCCC"); + AccessibilityDisabled at(t,4,nullptr), aq(q,4,nullptr); + ReverseAccessibility ar(aq); InteractionEnergyBasePair e(at,ar); + OutputConstraint oc(1,OutputConstraint::OVERLAP_BOTH,E_INF,E_INF,false,false,false,true,false); + OutputHandlerInteractionList out(oc,1); + SeedConstraint sc(2,0,0,0,E_INF,Accessibility::ED_UPPER_BOUND,E_INF,IndexRangeList(),IndexRangeList(),"",false,false,false); + PredictorMfeEns2dHeuristicSeedExtension p(e,out,nullptr,new SeedHandlerNoBulge(e,sc)); + p.predict(); return out.empty() || p.getZall()<=0; +} diff --git a/doc/local-validation-archive/files/issue-260-validation/shared.patch b/doc/local-validation-archive/files/issue-260-validation/shared.patch new file mode 100644 index 00000000..d8e84319 --- /dev/null +++ b/doc/local-validation-archive/files/issue-260-validation/shared.patch @@ -0,0 +1,124 @@ +diff --git a/src/IntaRNA/PredictorMfeEns.cpp b/src/IntaRNA/PredictorMfeEns.cpp +index 08b2e2d..d006a1a 100644 +--- a/src/IntaRNA/PredictorMfeEns.cpp ++++ b/src/IntaRNA/PredictorMfeEns.cpp +@@ -33,6 +33,8 @@ initZ() + { + // reset storage + Z_partition.clear(); ++ Z_frontier.clear(); ++ Z_frontierNext = 0; + } + + //////////////////////////////////////////////////////////////////////////// +@@ -109,7 +111,8 @@ updateZ( const size_t i1, const size_t j1 + + // store partial Z (without ED) + Interaction::Boundary key(i1,j1,i2,j2); +- auto [keyEntry, inserted] = Z_partition.try_emplace(key, partZ_noED); ++ auto & partitions = Z_frontier.empty() ? Z_partition : Z_frontier.at(j1); ++ auto [keyEntry, inserted] = partitions.try_emplace(key, partZ_noED); + if (!inserted) { + // update entry + keyEntry->second += partZ_noED; +@@ -154,6 +157,7 @@ void + PredictorMfeEns:: + updateOptimaUsingZ() + { ++ advanceZFrontier(Z_frontier.size()); + for (auto it = Z_partition.begin(); it != Z_partition.end(); ++it) + { + // if partition function is > 0 +@@ -163,6 +167,28 @@ updateOptimaUsingZ() + } + } + ++void PredictorMfeEns::beginZFrontier(size_t size1) ++{ ++ if (predTracker == nullptr) Z_frontier.resize(size1); ++} ++ ++void PredictorMfeEns::advanceZFrontier(size_t minimumSeedStart) ++{ ++ const size_t end = std::min(minimumSeedStart,Z_frontier.size()); ++ while (Z_frontierNext < end) { ++ auto & partitions = Z_frontier[Z_frontierNext]; ++ for (const auto & [boundary,partition] : partitions) { ++ if (Z_isNotINF(partition) && partition > 0) { ++ PredictorMfe::updateOptima(boundary.i1,boundary.j1,boundary.i2,boundary.j2, ++ energy.getE(partition),true,false); ++ } ++ } ++ // Free both nodes and bucket storage as soon as this boundary is complete. ++ decltype(Z_partition){}.swap(partitions); ++ ++Z_frontierNext; ++ } ++} ++ + //////////////////////////////////////////////////////////////////////////// + + void +diff --git a/src/IntaRNA/PredictorMfeEns.h b/src/IntaRNA/PredictorMfeEns.h +index 2037d52..c1f83b1 100644 +--- a/src/IntaRNA/PredictorMfeEns.h ++++ b/src/IntaRNA/PredictorMfeEns.h +@@ -9,6 +9,7 @@ + + #include + #include ++#include + + namespace IntaRNA { + +@@ -65,6 +66,12 @@ protected: + bool updateZisComplete; + + ++ //! partial sums grouped by target right boundary; used only without trackers ++ std::vector Z_frontier; ++ //! first right-boundary bucket that has not yet been finalized ++ size_t Z_frontierNext = 0; ++ ++ + /** + * Initializes the hybridization partition functions. + * Will be called by predict(). +@@ -111,6 +118,18 @@ protected: + , const Z_type partFunct + , const bool isHybridZ ); + ++ /** Enable partial-partition buckets by target right boundary. Tracker runs ++ * keep the original map to preserve callback ordering. Call advanceZFrontier ++ * only when no remaining seed can start below the supplied target index. ++ * @param size1 number of target positions in the current prediction region ++ */ ++ void beginZFrontier(size_t size1); ++ ++ /** Consume completed boundary sums without adding their mass to Zall again. ++ * @param minimumSeedStart smallest target start of any remaining seed ++ */ ++ void advanceZFrontier(size_t minimumSeedStart); ++ + /** + * Calls for the stored Z_partition information updateOptima() before + * calling reportOptima() from its super class. +diff --git a/src/IntaRNA/PredictorMfeEns2dSeedExtension.h b/src/IntaRNA/PredictorMfeEns2dSeedExtension.h +index 92107da..8d91a62 100644 +--- a/src/IntaRNA/PredictorMfeEns2dSeedExtension.h ++++ b/src/IntaRNA/PredictorMfeEns2dSeedExtension.h +@@ -23,6 +23,15 @@ namespace IntaRNA { + class PredictorMfeEns2dSeedExtension: public PredictorMfeEns { + + protected: ++ //! seed coordinates in handler order and the minimum remaining target start ++ struct SeedStart { size_t i1, i2, minimumRemaining; }; ++ ++ /** Collect seed starts without changing traversal or partition summation order. ++ * @param size1 target region length ++ * @param size2 reversed-query region length ++ * @return starts with a safe lower bound for all remaining target seed starts ++ */ ++ std::vector getSeedStarts(size_t size1,size_t size2) const; + + //! matrix type to hold the partition functions for interaction site starts + typedef Matrix Z2dMatrix; diff --git a/doc/local-validation-archive/files/issue-262-validation/hash-prototype/SeededPartitionFunction.cpp b/doc/local-validation-archive/files/issue-262-validation/hash-prototype/SeededPartitionFunction.cpp new file mode 100644 index 00000000..66803898 --- /dev/null +++ b/doc/local-validation-archive/files/issue-262-validation/hash-prototype/SeededPartitionFunction.cpp @@ -0,0 +1,134 @@ +#include "IntaRNA/SeededPartitionFunction.h" +#include "IntaRNA/PartitionArithmetic.h" +#include +#include +#include + +// Seed-free suffix lengths plus two absorbing seeded states (singleton/run). +// Nonstack edges operate on aggregate totals, independently of seed length. +namespace IntaRNA { +SeededPartitionFunction::Result +SeededPartitionFunction:: +compute( + const IntaRNA::SeededPartitionFunction::Domain & d, + const IntaRNA::StackSeedDomain & seeds, + const IntaRNA::SeededPartitionFunction::Weights & w, bool pairs) +{ + using namespace IntaRNA; + using A=PartitionArithmetic; + using Pair=SeededPartitionFunction::Pair; + SeededPartitionFunction::Result out; + // Empty seed families or zero spans admit no interaction, even with pair output. + if (pairs) out.mass=Matrix(d.n,d.m,0); + if (!seeds.maxSeedLength() || !d.span1 || !d.span2) return out; + // At least two length states are necessary for noLP even with singleton seeds. + const size_t cap=std::max(size_t(2),seeds.maxSeedLength()), s1=cap, s2=cap+1, states=cap+2; + seeds.forEachStart([&](size_t p1,size_t p2) { + // Each eligible left boundary owns a separate forward and reverse problem. + const Pair p{p1,p2}; if (!w.valid(p)) return; + const size_t n=std::min(d.span1,d.n-p1),m=std::min(d.span2,d.m-p2); + // Store all suffix states together for each admitted local base pair. The + // state count depends on the runtime seed length, so std::array cannot fit. + // Pair output appends reverse states to the same allocation; partition-only + // calls allocate only the forward states. Sequence 2 remains reversed. + std::unordered_map,Pair::Hash> state; + const size_t valuesPerPair=matrix_detail::product(states,pairs?2:1); + Matrix> total(n,m),dt; + Matrix valid(n,m); + Matrix ending(n,m); + if (pairs) dt.resize(n,m); + // Cache vertex admission and seed endings in this start's cropped domain. + size_t admitted=0; + for (size_t i=0;i=cap) return stacked || !d.noLP?s2:s1; + const size_t len=stacked?std::min(cap,c+2):1; + return accepts(i,j,len)?(len>=2 || !d.noLP?s2:s1):len-1; + }; + // Enumerate loop predecessors separately from the direct stacking edge. + auto nonstack=[&](size_t i,size_t j,const auto & visit) { + for(size_t u=1;u<=i && u-1<=d.loop1;++u) for(size_t v=1;v<=j && v-1<=d.loop2;++v) + if((u!=1 || v!=1) && valid(i-u,j-v)) visit(i-u,j-v); + }; + // Initiation belongs to the leftmost pair exactly once, including singletons. + const size_t initial=accepts(0,0,1)?(d.noLP?s1:s2):0; + const Z_type init=A::check(w.initiation(p)); state.at({0,0})[initial]=init; + // Forward pass: extend prefixes and finalize every admitted right boundary. + for(size_t i=0;i0) { + // Share the same boundary coefficient with reporting and the outside pass. + const Z_type b=A::check(w.boundary(p,{p1+i,p2+j})); + out.z=A::add(out.z,A::multiply(b,total(i,j)[1])); + if(pairs) { h[states+s2]=b;if(!d.noLP) h[states+s1]=b; } + if(w.complete) w.complete(p,{p1+i,p2+j},total(i,j)[1],b); + } + } + if(!pairs) return; + // Reverse pass: propagate boundary derivatives back through the same edges. + for(size_t ii=n;ii>0;--ii) for(size_t jj=m;jj>0;--jj) { + const size_t i=ii-1,j=jj-1; if(!valid(i,j)) continue; + Z_type * const dh=state.at({i,j}).data()+states; + // Distribute derivatives of the two aggregate totals to their suffix states. + for(size_t c=d.noLP?1:0;c +#include +#include +#include +#include +#include +#include +#include + +using Pair = IntaRNA::Interaction::BasePair; +using Z = IntaRNA::Z_type; +struct Counter { size_t calls = 0, bytes = 0; }; +std::array requested; + +template +struct CountingAllocator { + using value_type = T; + CountingAllocator() = default; + template CountingAllocator(const CountingAllocator &) {} + T *allocate(size_t n) { + T *result = std::allocator{}.allocate(n); + // libstdc++ rebinds the map allocator to nodes and bucket pointers. + auto &counter = requested[std::is_same_v ? 0 : std::is_pointer_v ? 1 : 2]; + ++counter.calls; + counter.bytes += n * sizeof(T); + return result; + } + void deallocate(T *p, size_t n) { std::allocator{}.deallocate(p, n); } + template bool operator==(const CountingAllocator &) const { return true; } +}; + +int main(int argc, char **argv) { + const size_t admitted = argc > 1 ? std::stoull(argv[1]) : 10000; + if (!admitted) return 1; + using Values = std::vector>; + using Entry = std::pair; + using Map = std::unordered_map, CountingAllocator>; + using NativeMap = std::unordered_map, Pair::Hash>; + static_assert(sizeof(Values) == sizeof(std::vector)); + static_assert(sizeof(Entry) == sizeof(NativeMap::value_type)); + static_assert(sizeof(Map) == sizeof(NativeMap)); + std::cout << "Z_bytes=" << sizeof(Z) << " Pair_bytes=" << sizeof(Pair) + << " vector_bytes=" << sizeof(Values) << " map_value_bytes=" << sizeof(Entry) + << " map_object_bytes=" << sizeof(Map) << '\n'; + std::cout << "admitted,states,pairs,buckets,value_allocations,value_bytes,bucket_allocations,bucket_bytes,node_allocations,node_bytes,state_bytes_per_pair,requested_hash_bytes_per_pair,break_even_density\n"; + for (size_t states : {size_t(4), size_t(9), size_t(26), size_t(66)}) { + for (bool pairs : {false, true}) { + requested = {}; + Map map; + map.reserve(admitted); + for (size_t i=0; i','#include \n#include \n#include ') +s=s.replace('\t\tMatrix h(matrix_detail::product(n,m),states),dh;', '''\t\t// Store all suffix states together for each admitted local base pair. The +\t\t// state count depends on the runtime seed length, so std::array cannot fit. +\t\t// Pair output appends reverse states to the same allocation; partition-only +\t\t// calls allocate only the forward states. Sequence 2 remains reversed. +\t\tstd::unordered_map,Pair::Hash> state; +\t\tconst size_t valuesPerPair=matrix_detail::product(states,pairs?2:1);''') +s=s.replace('\t\tif (pairs) { dh.resize(n*m,states); dt.resize(n,m); }', '\t\tif (pairs) dt.resize(n,m);') +s=s.replace('// Cache vertex admission and seed endings in this start\'s cropped domain.\n', '// Cache vertex admission and seed endings in this start\'s cropped domain.\n\t\tsize_t admitted=0;\n') +s=s.replace('\t\t\tvalid(i,j)=w.valid({p1+i,p2+j});','\t\t\tvalid(i,j)=w.valid({p1+i,p2+j});\n\t\t\tadmitted+=valid(i,j)!=0;') +s=s.replace('\t\tauto accepts=', '''\t\t// Reserve exactly the admitted count and finish insertion before using any +\t\t// state pointers. Traverse coordinates, never hash iteration order. +\t\tstate.reserve(admitted); +\t\tfor (size_t i=0;i=energy.getAccessibility1().getSequence().size() +- || r2.from>=energy.getAccessibility2().getSequence().size())) ++ || r2.from>=energy.getAccessibility2().getSequence().size()) + throw std::runtime_error("PredictorMfeEns2d::predict("+toString(r1)+","+toString(r2)+") is not sane"); + #endif + diff --git a/doc/local-validation-archive/files/issue-264-validation/core-benchmark/build.py b/doc/local-validation-archive/files/issue-264-validation/core-benchmark/build.py new file mode 100644 index 00000000..073692f8 --- /dev/null +++ b/doc/local-validation-archive/files/issue-264-validation/core-benchmark/build.py @@ -0,0 +1,24 @@ +#!/usr/bin/env python3 +from pathlib import Path +import hashlib +import json +import subprocess + +root=Path(__file__).resolve().parent +repo=root.parents[1]/'pr-worktrees'/'issue-264' +toolchain=root.parents[1]/'issue-246-validation'/'toolchain-gcc14' +dependencies=root.parents[1]/'issue-246-validation'/'toolchain' +compiler=toolchain/'bin'/'x86_64-conda-linux-gnu-g++' +commands=[] +for variant in ('base','candidate'): + command=[str(compiler),'-std=c++23','-O3','-DNDEBUG','-ffunction-sections','-fdata-sections', + '-I'+str(root/variant),'-I'+str(repo/'src'),'-I'+str(dependencies/'include'), + str(root/'driver.cpp'),str(root/variant/'IntaRNA'/'SeededPartitionFunction.cpp'),str(root/'StackSeedDomain.cpp'), + '-Wl,--gc-sections','-Wl,-rpath,'+str(toolchain/'lib'),'-o',str(root/variant/'benchmark')] + subprocess.run(command,check=True) + commands.append(command) +manifest={'baseline':'cb1c9d419f96c7d2a10bae864d922dcb0234f4d4', + 'compiler':subprocess.check_output([str(compiler),'--version'],text=True).splitlines()[0], + 'commands':commands,'sha256':{str(p.relative_to(root)):hashlib.sha256(p.read_bytes()).hexdigest() + for p in [root/'driver.cpp',root/'StackSeedDomain.cpp',*(root/variant/'IntaRNA'/name for variant in ('base','candidate') for name in ('SeededPartitionFunction.cpp','SeededPartitionFunction.h'))]}} +(root/'manifest.json').write_text(json.dumps(manifest,indent=2)+'\n') diff --git a/doc/local-validation-archive/files/issue-264-validation/core-benchmark/run.py b/doc/local-validation-archive/files/issue-264-validation/core-benchmark/run.py new file mode 100644 index 00000000..107ae964 --- /dev/null +++ b/doc/local-validation-archive/files/issue-264-validation/core-benchmark/run.py @@ -0,0 +1,78 @@ +#!/usr/bin/env python3 +from pathlib import Path +import csv +import json +import os +import statistics +import subprocess + +root=Path(__file__).resolve().parent +env=dict(os.environ,OMP_NUM_THREADS='1') +cases=[('dense','many',40),('dense','few',40),('band','many',60),('band','few',60)] +samples=[] +checks=[] + +def call(variant,case,pairs,dump='-'): + topology,density,n=case + command=[str(root/variant/'benchmark'),topology,density,str(n),str(pairs),dump] + rss=root/'last-rss.txt' + result=subprocess.check_output(['/usr/bin/time','-f','%M','-o',str(rss),*command],text=True,env=env) + seconds,z,moment,boundaries,checksum,seeds=result.strip().split('\t') + return {'variant':variant,'case':f'{topology}{n}-{density}','pairs':pairs, + 'seconds':float(seconds),'rss_kib':int(rss.read_text().strip()),'z':float(z), + 'pair_moment':float(moment),'boundaries':int(boundaries),'checksum':float(checksum),'seeds':int(seeds)} + +def read_dump(path): + values={} + for line in path.read_text().splitlines(): + fields=line.split() + if fields[0]=='b': + values[tuple(fields[:5])]=tuple(map(float,fields[5:])) + else: + values[tuple(fields[:-1])]=(float(fields[-1]),) + return values + +for case in cases: + name=f'{case[0]}{case[2]}-{case[1]}' + paths={variant:root/f'{name}-{variant}.txt' for variant in ('base','candidate')} + for variant,path in paths.items(): + call(variant,case,1,str(path)) + before,after=(read_dump(paths[variant]) for variant in ('base','candidate')) + assert before.keys()==after.keys(),name + maximum=0.0 + for key,values in before.items(): + for a,b in zip(values,after[key]): + relative=abs(a-b)/max(abs(a),abs(b)) if a or b else 0.0 + maximum=max(maximum,relative) + assert relative<1e-12,(name,key,a,b,relative) + checks.append({'case':name,'compared_entries':len(before),'max_relative_error':maximum}) + for pairs in (0,1): + for repetition in range(3): + # Alternate order to reduce a consistent order bias. + variants=('base','candidate') if repetition%2==0 else ('candidate','base') + for variant in variants: + sample=call(variant,case,pairs) + sample['repetition']=repetition + samples.append(sample) + print(name,checks[-1],flush=True) + +with (root/'samples.tsv').open('w') as stream: + writer=csv.DictWriter(stream,fieldnames=list(samples[0]),delimiter='\t') + writer.writeheader();writer.writerows(samples) +summary=[] +for case in cases: + name=f'{case[0]}{case[2]}-{case[1]}' + for pairs in (0,1): + groups={variant:[row for row in samples if row['case']==name and row['pairs']==pairs and row['variant']==variant] + for variant in ('base','candidate')} + result={'case':name,'pairs':pairs,'seeds':groups['base'][0]['seeds']} + for variant,rows in groups.items(): + result[variant+'_median_seconds']=statistics.median(row['seconds'] for row in rows) + result[variant+'_min_seconds']=min(row['seconds'] for row in rows) + result[variant+'_max_seconds']=max(row['seconds'] for row in rows) + result[variant+'_median_rss_kib']=statistics.median(row['rss_kib'] for row in rows) + result['runtime_ratio']=result['candidate_median_seconds']/result['base_median_seconds'] + result['rss_ratio']=result['candidate_median_rss_kib']/result['base_median_rss_kib'] + summary.append(result) +(root/'results.json').write_text(json.dumps({'checks':checks,'summary':summary},indent=2)+'\n') +print(json.dumps(summary,indent=2),flush=True) diff --git a/doc/local-validation-archive/files/issue-264-validation/debug/doc/Makefile.am b/doc/local-validation-archive/files/issue-264-validation/debug/doc/Makefile.am new file mode 100644 index 00000000..101c22b5 --- /dev/null +++ b/doc/local-validation-archive/files/issue-264-validation/debug/doc/Makefile.am @@ -0,0 +1,27 @@ + +################################################################ +# extra files to be added to distribution +################################################################ + +EXTRA_DIST = \ + conda.txt \ + analysis/base-pair-probabilities-plan.md \ + analysis/seeded-kernel-comparison.md \ + analysis/seeded-probabilities-validation.md \ + analysis/seed-extension-partition.md \ + seeded-probabilities-example.cpp \ + api-documentation.md \ + build-api.sh \ + check-api.py \ + doxygen.cfg \ + doxygen/mainpage.md \ + doxygen/header.html \ + doxygen/footer.html \ + doxygen/intarna.css \ + latex-deps/adjcalc.sty \ + latex-deps/adjustbox.sty \ + latex-deps/collectbox.sty \ + latex-deps/tc-pgf.def \ + latex-deps/tocloft.sty \ + latex-deps/trimclip.sty \ + latex-deps/xtab.sty diff --git a/doc/local-validation-archive/files/issue-264-validation/debug/doc/analysis/seed-extension-partition.md b/doc/local-validation-archive/files/issue-264-validation/debug/doc/analysis/seed-extension-partition.md new file mode 100644 index 00000000..706e279d --- /dev/null +++ b/doc/local-validation-archive/files/issue-264-validation/debug/doc/analysis/seed-extension-partition.md @@ -0,0 +1,240 @@ +# Seed-extension partition functions and actual-pair probabilities + +This document describes the nonnegative stack-seed extension formulation for +[issue #264](https://github.com/BackofenLab/IntaRNA/issues/264). Its reference is +the IntaRNA-probabilities branch +[`theory-seed-extension`](https://github.com/BackofenLab/IntaRNA-probabilities/tree/theory-seed-extension), +reviewed at revision +[`e904b65101294f782b0938590335f3e23631a1aa`](https://github.com/BackofenLab/IntaRNA-probabilities/tree/e904b65101294f782b0938590335f3e23631a1aa), +especially the +[seed-extension chapter](https://github.com/BackofenLab/IntaRNA-probabilities/blob/e904b65101294f782b0938590335f3e23631a1aa/theory/seed-extension.md) +and its `SE-LEFT-REC`, `SE-FIRST-HIT`, and `SE-PAIR-FIRST-HIT` equations. +The historical subtraction formulas discussed in that chapter are not the +recurrences used here. + +## Ensemble and weight ownership + +An interaction is an ordered nonempty chain of complementary intermolecular +pairs. Coordinates below are local, zero-based, and increase on both strands; +the second strand has already been reversed. A legal edge joins successive +pairs and respects the interaction loop limits. An admitted seed is a concrete +stack occurrence retained by the active seed handler, after its energy, +accessibility, position, and seed-specific pairing filters. + +For an interaction with outer pairs `p` and `r`, the hybrid weight is its +initiation weight multiplied by one transition weight for each successive pair. +The complete boundary coefficient contains accessibility and the native +boundary energy corrections. It is evaluated for the full outer intervals and +multiplied once. A seed's stored energy is used by seed admission; it is not an +additional factor in the exact chain weight. This avoids counting initiation, +anchor pairs, or their internal loops twice. + +The exact ensemble contains every legal chain with an admitted seed, once, +regardless of how many overlapping or disjoint seed occurrences it contains. +Interaction span limits are inclusive on both strands. Whole-interaction +restrictions such as `noLP` are part of the recurrence; endpoint restrictions +such as `noGUend` and complete-span accessibility are applied to the full +boundary. Neither ranked-result limits nor display cutoffs select the ensemble. + +## Unique first acceptance + +Scan a chain from its left boundary. Its canonical anchor is the first admitted +seed to finish. Before that completion, the prefix contains no admitted seed; +after it, the remaining chain has no further seed requirement. This event is +unique even when later seeds overlap the anchor. + +Strict stack occurrences ending at the same pair can have different lengths +when explicit seeds are used. Choose the shortest admitted occurrence ending +there as canonical owner. Every chain containing a longer occurrence ending at +that pair also contains this shorter suffix, so these simultaneous matches form +one acceptance event. Ownership is chosen before applying the heuristic +selector described below. + +To detect an event, the nonaccepting state retains the terminal strict-stack +run length, capped at the longest relevant seed length. A new pair accepts when +its terminal run is at least the shortest admitted seed length ending there. +The run can exceed that length because earlier potential seed windows may have +been filtered out. A filtered-out window neither accepts nor resets the run. +A nonstack transition resets the run to one pair. + +The empty declared seed family produces an empty seeded ensemble. A singleton +explicit seed can first accept at the initial pair or at a later appended pair; +these alternatives are disjoint. + +## Seed-free left prefixes, first hits, and right extensions + +Fix the exact left boundary `p`. Let `N[p,h,a]` be the hybrid weight of prefixes +ending at pair `h` in nonaccepting state `a`. The initial prefix has the sole +initiation factor. Its weight is assigned to `N` only when the initial pair is +not itself a declared singleton seed. Each legal transition appends its weight +to `N` only if the resulting state remains nonaccepting. + +Let `F[p,q,c]` hold first-hit weights ending at pair `q`, with structural state +`c` needed by the right extension. It receives: + +- the initiation weight for a singleton source hit at `p = q`; or +- `N[p,h,a] * transition(h,q)` for an accepting edge from a seed-free prefix. + +An accepting contribution leaves `N`; it is not retained in both families. +The left prefix owns initiation and all pairs before `q`. The accepting edge +owns the new pair `q`. Thus `F` already includes the complete anchor and must +not be multiplied by another standalone seed weight. + +The theory writes a normalized right continuation `B` with value one for an +empty continuation. In the absence of additional structural state, its +first-hit contraction is + +```text +Hseed[p,r] = sum_q F[p,q] * B[q,r]. +``` + +The implementation can share the work of all anchors through an inhomogeneous +right-extension recurrence: + +```text +T[p,q,c] = F[p,q,c] + + sum_(h,b -> q,c) T[p,h,b] * transition(h,q). +``` + +Here the edge relation includes any structural-state transition. The right +recurrence has no seed state, seed test, or second initiation factor. Expanding +it gives exactly the sum of first-hit weights multiplied by their unrestricted +right continuations. Combining injections before extending is valid by +distributivity: each complete chain still has its unique first-hit event. +This shares continuation work without enumerating complete interactions or +subtracting overlapping seed classes. + +Sum terminally admissible `T` states to obtain the seeded hybrid partition for +each `(p,r)`. Apply the complete boundary coefficient and publish that boundary +once, after all anchor contributions have been combined. Keeping `p` throughout +the calculation is necessary because accessibility and span restrictions can +couple the two outer ends. + +## The `noLP` structural state + +Seed acceptance does not remove the requirement that every pair have a stacked +neighbor. Before acceptance, a terminal run of length one needs support from a +future stack. A run of length at least two is already supported. Nonstack edges +may leave only supported states. + +After acceptance, the right extension retains these two structural states: + +- **Pending:** the terminal pair has no stacked neighbor yet. +- **Supported:** the terminal pair already has a stacked neighbor. + +A stack from either state enters the supported state. A nonstack edge can leave +only the supported state and enters the pending state. Only the supported +state may terminate. A singleton source hit therefore starts pending and can +become a valid interaction through a right stack. The same rule handles a +singleton first hit following a nonstack edge. + +Consequently the right extension is unrestricted with respect to seeds, but +requires structural state when `noLP` is enabled. Using a scalar unconstrained +continuation in that case would admit lonely pairs. + +## A defined heuristic subset + +The stack-seed heuristic uses the same canonical first-hit ownership as the +exact computation. For canonical anchor `S`, let `start(S)` and `end(S)` be its +outer pairs, and let `selectedRight(S)` be the deterministic endpoint selected +by the heuristic's right-extension policy. Select the endpoint before the +partition and outside calculations; marking a pair must not change it. + +The default predictor preserves its existing right-extension selection hooks. +It enumerates strict right extensions in increasing first-strand endpoint, +then increasing reversed-second-strand endpoint. Candidates pass the existing +right-partition numerical-zero test. Their scores are the full seed-plus-right +ensemble energies, including initiation and the complete boundary energy for +`start(S)` and the candidate endpoint. `noGUend` excludes GU right endpoints. +An endpoint replaces the current choice only for a strictly lower score, so +equal scores retain the first visited endpoint. If no candidate qualifies, +the choice remains `end(S)`. Custom overrides of the right-extension hooks +remain responsible for deterministic choices. The default heuristic predictor +retains its existing rejection of singleton seeds, although the shared kernel +and exact predictor support them. + +Keep a complete chain with outer boundaries `(p,r)` precisely when it satisfies +the ordinary interaction constraints and at least one of these equalities: + +```text +p == start(S) OR r == end(S) OR r == selectedRight(S). +``` + +The first alternative retains every right extension with no left extension. +The second retains every left extension with no right extension. The third +retains both-sided extensions ending at the selected right endpoint. The +selector is a union: a chain satisfying several alternatives is counted once. +All legal right paths to an admitted endpoint are retained, rather than only +one representative path. The canonical owner is fixed independently of whether +a later seed would offer a different selected endpoint. + +For fixed `p`, anchors with `p == start(S)` can share an unrestricted right pass. +For the remaining anchors, add the zero-right-extension contribution directly +at `end(S)`, and group the other first-hit injections by their selected right +endpoint. Each group shares one right recurrence and contributes only at that +group's terminal endpoint. If the selected endpoint equals the anchor end, +the direct contribution already covers it. Anchors admitted by the unrestricted +case must not also contribute through the restricted cases. + +Grouping is valid because the right recurrence is linear and independent of +seed history. Keeping separate selected-endpoint groups preserves the selector +for each canonical anchor; merging all groups into one unconstrained pass would +incorrectly admit other endpoints. Sum the retained contributions for each +complete boundary before reporting it or initializing its outside coefficient. + +With unchanged nonnegative weights, this defines a subset of the exact seeded +ensemble, hence `0 <= Zheuristic <= Zexact`. The heuristic probabilities are +conditional on that retained subset. They need not equal the exact seeded +probabilities, and this definition does not promise numerical identity with +the historical signed-overlap heuristic. + +## Actual-pair probabilities by the outside pass + +The raw numerator for pair `k` is the sum of complete interaction weights whose +chains actually contain `k`. It is not the sum of weights of boundary rectangles +covering `k`; the latter is a spot probability. + +Initialize outside derivatives from the same complete boundary coefficients +used by forward accumulation. Reverse the right-extension passes, transfer +their derivatives into the corresponding first-hit injections, and then +reverse the accepting edges and seed-free prefix recurrence. A transition +owns its destination pair; the initialization owns the first pair. Accumulating +each owned pair's forward weight times its outside coefficient counts that +pair exactly once in every retained complete chain, including anchor pairs. + +For heuristic calculations, reverse the same grouped passes and frozen +endpoint selector. The outside calculation must not independently recompute a +different ensemble or normalize each anchor separately. Merge raw region pair +numerators and denominators before final normalization. An empty ensemble has +undefined conditional probabilities; a numerical failure is an error, not an +empty ensemble. + +## Scope, checks, and resource considerations + +The run-length automaton supports admitted stack occurrences, including +different-length and singleton explicit seeds. A general gapped seed requires +a different prefix-matching automaton. The existing legacy gapped-seed fallback +is retained; the stack-seed proof and nonnegative formulation do not establish +its correctness or expand its supported probability modes. + +The production weights use native IntaRNA energies and `Z_type` arithmetic. +"Exact" describes the ensemble and recurrence, not exact real-number +arithmetic. Checked partition operations must report nonfinite values and +unsupported numerical ranges rather than silently discard positive terms. + +The independent tests should compare complete boundary partitions and raw +pair numerators, not only rounded final probabilities. Small path-enumeration +oracles should cover overlapping and disjoint seeds, filtered windows, +co-ending different-length seeds, singleton source and edge hits, both span +limits, and `noLP` states on both sides of acceptance. Heuristic checks must use +the same frozen selector, verify each retained chain once, and compare its +partition with the exact ensemble. + +The seed-free table still retains run-length state for each active left +boundary. Sharing right extensions avoids repeating identical continuation +work for every anchor, but does not imply a lower asymptotic bound for the +whole calculation. Heuristic cost also depends on the number of distinct +selected endpoints. Pair output adds outside storage and work. Performance or +memory improvements require measurements against the previous implementation +with identical inputs, options, build configuration, and thread count; the +factorization alone is not a performance claim. diff --git a/doc/local-validation-archive/files/issue-264-validation/native-heuristic-timing.py b/doc/local-validation-archive/files/issue-264-validation/native-heuristic-timing.py new file mode 100644 index 00000000..174d921d --- /dev/null +++ b/doc/local-validation-archive/files/issue-264-validation/native-heuristic-timing.py @@ -0,0 +1,38 @@ +from pathlib import Path +import hashlib +import json +import os +import shutil +import subprocess +import time + +root=Path(__file__).resolve().parent +repo=root.parent/'pr-worktrees'/'issue-264' +old=root.parent/'pr-worktrees'/'issue-257-plan' +directory=root/'native-heuristic-timing' +directory.mkdir(exist_ok=True) +for name,source in [('base',old),('candidate',repo)]: + shutil.copy2(source/'src/bin/IntaRNA',directory/name) +common=['--model=P','--mode=H','--seedBP=4','--intLenMax=30','--qIntLoopMax=3','--tIntLoopMax=3', + '--threads=1','--outNoLP=false','--outNoGUend=false','--outNumber=0','--outMode=E', + '--default-log-file=/dev/null','--out=STDOUT'] +cases={'OxyS-fhlA':['--target='+str(repo/'doc/handson/fhlA.fasta'),'--query='+str(repo/'doc/handson/OxyS.fasta')], + 'dense40':['--target='+'G'*40,'--query='+'C'*40,'--tAcc=N','--qAcc=N']} +env=dict(os.environ,OMP_NUM_THREADS='1',OPENBLAS_NUM_THREADS='1',OMP_DYNAMIC='FALSE') +records=[] +for case,arguments in cases.items(): + for repeat in range(3): + for variant in (('base','candidate') if repeat%2==0 else ('candidate','base')): + command=[str(directory/variant),*arguments,*common] + rss=directory/'rss.txt' + start=time.perf_counter() + result=subprocess.run(['/usr/bin/time','-f','%M','-o',str(rss),*command],env=env,text=True,capture_output=True,timeout=60) + elapsed=time.perf_counter()-start + result.check_returncode() + records.append({'case':case,'variant':variant,'repeat':repeat,'seconds':elapsed, + 'rss_kib':int(rss.read_text()),'output':result.stdout.strip()}) + print(records[-1],flush=True) +manifest={'baseline':'7a522358274f8ae636634bd89e51e2d8d7ee86b6','notes':'Exploratory native timing; older baseline, intentionally different heuristic ensembles, no pair output, build/test activity may overlap.', + 'commands':{case:args+common for case,args in cases.items()},'records':records, + 'binary_sha256':{variant:hashlib.sha256((directory/variant).read_bytes()).hexdigest() for variant in ('base','candidate')}} +(directory/'results.json').write_text(json.dumps(manifest,indent=2)+'\n') diff --git a/doc/local-validation-archive/files/issue-265-validation/benchmark/benchmark.py b/doc/local-validation-archive/files/issue-265-validation/benchmark/benchmark.py new file mode 100644 index 00000000..59ce68b8 --- /dev/null +++ b/doc/local-validation-archive/files/issue-265-validation/benchmark/benchmark.py @@ -0,0 +1,67 @@ +#!/usr/bin/env python3 +"""Reproducible end-to-end accessibility benchmark; unsafe base runs are compared to base serial.""" +import argparse, hashlib, json, os, pathlib, random, shutil, statistics, subprocess, time + +p=argparse.ArgumentParser() +p.add_argument('--base',type=pathlib.Path,required=True) +p.add_argument('--candidate',type=pathlib.Path,required=True) +p.add_argument('--output',type=pathlib.Path,required=True) +p.add_argument('--repetitions',type=int,default=3) +p.add_argument('--cpus',default='8-11') +a=p.parse_args() +a.output.mkdir(parents=True,exist_ok=True) +for label, binary in [('base',a.base),('candidate',a.candidate)]: + shutil.copy2(binary,a.output/('IntaRNA-'+label)) +rng=random.Random(265) +queries=a.output/'queries.fa' +queries.write_text(''.join('>query%02d\n%s\n' % (i+1,''.join(rng.choice('ACGU') for j in range(400))) for i in range(32))) +target=a.output/'target.fa' +target.write_text('>target\nGCGCGC\n') +options=['--query='+str(queries.resolve()),'--target='+str(target.resolve()),'--model=X','--mode=K','--seedBP=3','--qIntLenMax=8','--tIntLenMax=6','--qAccW=120','--qAccL=120','--tAccW=6','--tAccL=6','--outMode=C','--outCsvCols=id1,start1,end1,id2,start2,end2,subseqDP,hybridDP,E,ED1,ED2'] +env=dict(os.environ,OMP_DYNAMIC='FALSE',OMP_PROC_BIND='TRUE',OMP_PLACES='cores',OPENBLAS_NUM_THREADS='1') + +def hashes(directory): + lines=(directory/'predictions.csv').read_text().splitlines() + norm=lines[0]+'\n'+'\n'.join(sorted(lines[1:]))+'\n' + result={'predictions.csv':hashlib.sha256(norm.encode()).hexdigest()} + for f in sorted(directory.glob('query*.ed')): + result[f.name]=hashlib.sha256(f.read_bytes()).hexdigest() + if len(result)!=33: + raise RuntimeError('Expected 32 query ED files and CSV: '+str(directory)) + return result + +def run(label,threads,iteration,warmup=False): + name=('%s-t%d-%s%d' % (label,threads,'warmup' if warmup else 'run',iteration)) + directory=a.output/name + directory.mkdir() + command=['/usr/bin/time','-f','%e %U %S %M','-o',str((directory/'time.txt').resolve()),'taskset','-c',a.cpus,str((a.output/('IntaRNA-'+label)).resolve()),*options,'--threads='+str(threads),'--out='+str((directory/'predictions.csv').resolve()),'--out=qAcc:'+str((directory/'query.ed').resolve())] + with (directory/'stdout.txt').open('w') as stdout,(directory/'stderr.txt').open('w') as stderr: + start=time.perf_counter() + completed=subprocess.run(command,env=env,stdout=stdout,stderr=stderr) + elapsed=time.perf_counter()-start + if completed.returncode: + raise RuntimeError('Failed '+name+': '+(directory/'stderr.txt').read_text()) + wall,user,system,rss=(directory/'time.txt').read_text().split() + result={'name':name,'label':label,'threads':threads,'wall_s':float(wall),'monotonic_wall_s':elapsed,'user_s':float(user),'system_s':float(system),'peak_rss_kib':int(rss),'command':command,'hashes':hashes(directory)} + print(name,wall,rss,flush=True) + return result + +warmups=[run(label,threads,0,True) for label,threads in [('base',1),('candidate',1),('base',4),('candidate',4)]] +reference=warmups[0]['hashes'] +results=[] +for i in range(a.repetitions): + configs=[('base',1),('candidate',1),('base',4),('candidate',4)] + if i%2: configs.reverse() + for label,threads in configs: results.append(run(label,threads,i+1)) +for result in warmups+results: + result['matches_base_serial']=result['hashes']==reference + result['different_files']=[k for k,v in result['hashes'].items() if reference.get(k)!=v] +summary=[] +for label,threads in [('base',1),('candidate',1),('base',4),('candidate',4)]: + selected=[r for r in results if r['label']==label and r['threads']==threads] + summary.append({'label':label,'threads':threads,'median_wall_s':statistics.median(r['wall_s'] for r in selected),'min_wall_s':min(r['wall_s'] for r in selected),'max_wall_s':max(r['wall_s'] for r in selected),'median_peak_rss_kib':statistics.median(r['peak_rss_kib'] for r in selected),'matches_base_serial':all(r['matches_base_serial'] for r in selected)}) +meta={'cpu_affinity':a.cpus,'environment':{key:env[key] for key in ['OMP_DYNAMIC','OMP_PROC_BIND','OMP_PLACES','OPENBLAS_NUM_THREADS']},'binary_hashes':{label:hashlib.sha256((a.output/('IntaRNA-'+label)).read_bytes()).hexdigest() for label in ['base','candidate']},'input_hashes':{f.name:hashlib.sha256(f.read_bytes()).hexdigest() for f in [queries,target]},'base_version':subprocess.check_output([str((a.output/'IntaRNA-base').resolve()),'--version'],text=True),'candidate_version':subprocess.check_output([str((a.output/'IntaRNA-candidate').resolve()),'--version'],text=True),'summary':summary,'warmups':warmups,'results':results} +(a.output/'results.json').write_text(json.dumps(meta,indent=2)+'\n') +print(json.dumps(summary,indent=2),flush=True) +if not all(r['matches_base_serial'] for r in warmups+results): + raise SystemExit('Scientific output mismatch found; inspect results.json') diff --git a/doc/local-validation-archive/files/issue-265-validation/benchmark/serial_followup.py b/doc/local-validation-archive/files/issue-265-validation/benchmark/serial_followup.py new file mode 100644 index 00000000..ac9b610a --- /dev/null +++ b/doc/local-validation-archive/files/issue-265-validation/benchmark/serial_followup.py @@ -0,0 +1,26 @@ +#!/usr/bin/env python3 +import hashlib,json,os,pathlib,statistics,subprocess +root=pathlib.Path('/home/alex/CodexProjects/IntaRNA/issue-265-validation/benchmark/run') +original=json.loads((root/'results.json').read_text()) +env=dict(os.environ,**original['environment']) +reference=original['warmups'][0]['hashes'] +results=[] +for rep in range(4): + for label in (['base','candidate'] if rep%2==0 else ['candidate','base']): + template=next(r for r in original['results'] if r['label']==label and r['threads']==1) + directory=root/('%s-t1-idle%d' %(label,rep+1));directory.mkdir() + source_directory=root/template['name'] + command=[arg.replace(str(source_directory),str(directory)) for arg in template['command']] + command[command.index('-c')+1]='8' + with (directory/'stdout.txt').open('w') as stdout,(directory/'stderr.txt').open('w') as stderr: + subprocess.run(command,env=env,stdout=stdout,stderr=stderr,check=True) + wall,user,system,rss=(directory/'time.txt').read_text().split() + lines=(directory/'predictions.csv').read_text().splitlines();norm=lines[0]+'\n'+'\n'.join(sorted(lines[1:]))+'\n' + hashes={'predictions.csv':hashlib.sha256(norm.encode()).hexdigest()} + for f in sorted(directory.glob('query*.ed')):hashes[f.name]=hashlib.sha256(f.read_bytes()).hexdigest() + result={'label':label,'repetition':rep+1,'wall_s':float(wall),'user_s':float(user),'system_s':float(system),'peak_rss_kib':int(rss),'matches_base_serial':hashes==reference,'command':command} + results.append(result);print(label,rep+1,wall,user,system,rss,result['matches_base_serial'],flush=True) +summary=[{'label':label,'threads':1,'median_wall_s':statistics.median(r['wall_s'] for r in results if r['label']==label),'min_wall_s':min(r['wall_s'] for r in results if r['label']==label),'max_wall_s':max(r['wall_s'] for r in results if r['label']==label),'median_peak_rss_kib':statistics.median(r['peak_rss_kib'] for r in results if r['label']==label)} for label in ['base','candidate']] +(root/'serial-followup.json').write_text(json.dumps({'cpu_affinity':'8','note':'All four repetitions occurred after build and test processes completed. Same immutable binary snapshots, inputs/options and output checks as original benchmark.','results':results,'summary':summary},indent=2)+'\n') +print(json.dumps(summary,indent=2),flush=True) +if not all(r['matches_base_serial'] for r in results):raise SystemExit('Output mismatch') diff --git a/doc/local-validation-archive/files/issue-265-validation/debug/doc/analysis/vienna-accessibility-thread-safety.md b/doc/local-validation-archive/files/issue-265-validation/debug/doc/analysis/vienna-accessibility-thread-safety.md new file mode 100644 index 00000000..49d8ca65 --- /dev/null +++ b/doc/local-validation-archive/files/issue-265-validation/debug/doc/analysis/vienna-accessibility-thread-safety.md @@ -0,0 +1,174 @@ +# ViennaRNA accessibility thread-safety audit + +This audit accompanies [issue #265](https://github.com/BackofenLab/IntaRNA/issues/265). +The former critical section covered only `vrna_probs_window()`. That boundary +did not protect parameter construction: independent fold compounds can race +while ViennaRNA initializes them. Conversely, the sliding-window recurrence +uses compound-owned state and can run concurrently after synchronized setup. + +## Reviewed upstream sources + +The review covers the minimum supported ViennaRNA release, **2.4.14**, and the +**2.7.2** release used for the runtime experiments. Source links below are +pinned to immutable commits: + +| Release | Commit | Scope | +| --- | --- | --- | +| 2.4.14 | [`0b8ca4c6604e7a8bea366af62d82f8f40cea49bf`](https://github.com/ViennaRNA/ViennaRNA/tree/0b8ca4c6604e7a8bea366af62d82f8f40cea49bf) | Minimum supported API and implementation | +| 2.7.2 | [`1ffec79f5e258896160f7362ced8263450f371dc`](https://github.com/ViennaRNA/ViennaRNA/tree/1ffec79f5e258896160f7362ced8263450f371dc) | Installed runtime and current parameter cache implementation | + +This is a review of the paths IntaRNA calls, not a blanket claim that RNAlib, +its deprecated interfaces, or every future release is thread-safe. In +particular, compiling RNAlib with OpenMP does not make all its static state +thread-private. + +## Call path and ownership + +The relevant accessibility path is: + +```text +AccessibilityVrna::fillByRNAplfold + VrnaHandler::getModel -> vrna_md_copy + vrna_fold_compound(PF | WINDOW) + add_params -> vrna_params + vrna_params_prepare -> vrna_exp_params + sanitize_bp_span, sequence/constraint/matrix setup + addConstraints + vrna_constraints_add -> vrna_hc_add_from_db + vrna_constraints_add_SHAPE, when requested + vrna_fold_compound_prepare(PF | WINDOW) + vrna_probs_window + vrna_fold_compound_prepare(PF | WINDOW) + allocate local auxiliary arrays + update compound-owned hard/soft constraints and DP matrices + callbackForStorage -> this accessibility object's ED matrix + vrna_fold_compound_free +``` + +The explicit preparation before `vrna_probs_window()` is significant. The +window routine calls `vrna_fold_compound_prepare()` internally, and parameter +preparation can replace `exp_params` when its model differs from `params`. +Construction sanitizes span/window settings after initially constructing +parameters. Thus the synchronization rule must cover preparation as well as +the constructor. See the +[2.7.2 compound constructor and preparation](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/fold_compound.c#L200-L276), +[preparation call chain](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/fold_compound.c#L519-L626), +and [parameter replacement condition](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/params/params.c#L427-L456). + +After explicit preparation, the second preparation is idempotent with respect +to parameter construction for this path: the two parameter models match, +constraints do not change them, and no other worker accesses this compound. +Hard/soft constraint preparation and matrix scaling operate on its owned +storage. This reasoning assumes IntaRNA's single-sequence model, with no +external callbacks or custom grammar that mutate model settings. See +[hard constraints](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/constraints/hard.c#L276-L310), +[soft constraints](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/constraints/soft.c#L304-L356), +and [matrix preparation](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/datastructures/dp_matrices.c#L299-L390). + +## Shared state and required synchronization + +| State or operation | Evidence | Consequence | +| --- | --- | --- | +| Parameter caches `p_pre`, `pf_pre`, and their initialization flags | [2.7.2 `params.c`](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/params/params.c#L88-L217) enables `SPEEDUP_PARAMS`; both constructors rewrite cached model members and copy whole cached structures. Only `id` and `pf_id` are declared OpenMP thread-private. | Guard every IntaRNA entry to parameter construction with one shared mutex. Separate locks for accessibility and energy construction would not protect the same caches. | +| Parameter identifiers in older RNAlib | [2.4.14 declarations](https://github.com/ViennaRNA/ViennaRNA/blob/0b8ca4c6604e7a8bea366af62d82f8f40cea49bf/src/ViennaRNA/params/params.c#L88-L98), [identifier increment](https://github.com/ViennaRNA/ViennaRNA/blob/0b8ca4c6604e7a8bea366af62d82f8f40cea49bf/src/ViennaRNA/params/params.c#L552) | 2.4.14 lacks the newer caches, but its identifier is shared in builds without OpenMP. The same setup lock remains useful. | +| Global model defaults and legacy model variables | [2.7.2 `vrna_md_defaults_reset`](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/model.c#L346-L432), [2.4.14 implementation](https://github.com/ViennaRNA/ViennaRNA/blob/0b8ca4c6604e7a8bea366af62d82f8f40cea49bf/src/ViennaRNA/model.c#L311-L384) | Passing a model to `vrna_md_defaults_reset()` changes process-wide defaults; it does not merely reset that local object. Remove the worker-side call when every fold already receives its explicit model. Guard parameter-file loading/default reads during handler setup. | +| `vrna_md_copy` | [2.7.2 implementation](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/model.c#L196-L232) copies the supplied structure and its arrays. | Copying a stable handler model into worker-local storage needs no global lock. | +| Sliding-window DP, auxiliary arrays, and constraint updates | [2.7.2 window implementation](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/partfunc/pf_window.c#L681-L958), [2.4.14 implementation](https://github.com/ViennaRNA/ViennaRNA/blob/0b8ca4c6604e7a8bea366af62d82f8f40cea49bf/src/ViennaRNA/LPfold.c#L669-L941) obtain arrays and models from the compound or allocate local helpers. | Independent prepared compounds can compute concurrently. Keep callback storage distinct and alive until each call returns. | +| Window compatibility state | [2.7.2 declarations](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/partfunc/pf_window.c#L91-L110), [legacy wrapper](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/partfunc/pf_window.c#L2170-L2243) | `backward_compat_compound` and `backward_compat` belong to the legacy wrapper path. The direct `vrna_probs_window()` call does not set them. Do not replace the direct API with a legacy wrapper under the same concurrency assumptions. | +| SHAPE D, Z, and W input | [2.7.2 implementation](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/probing/SHAPE.c#L332-L404), [2.4.14 implementation](https://github.com/ViennaRNA/ViennaRNA/blob/0b8ca4c6604e7a8bea366af62d82f8f40cea49bf/src/ViennaRNA/constraints/SHAPE.c#L61-L120) | Setup loads input into the compound's soft constraints. W reads supplied pseudoenergies; this API does not perform stochastic optimization. Include constraint setup in the guarded preparation boundary. | +| Destruction | [2.7.2 `vrna_fold_compound_free`](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/fold_compound.c#L117-L197) frees compound-owned arrays, constraints, and auxiliary data. | Ordinary accessibility destruction does not mutate the parameter caches. Each compound must have one owner. | +| Global PF compatibility pointer `pr` | [2.7.2 `vrna_pf`](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/partfunc/partfunc.c#L208-L222), [2.4.14 implementation](https://github.com/ViennaRNA/ViennaRNA/blob/0b8ca4c6604e7a8bea366af62d82f8f40cea49bf/src/ViennaRNA/part_func.c#L129-L142) still write the process-global legacy pointer. | The accessibility conclusion does not justify unlocking global PF calls. Serialize the energy-model global-fold paths with the same mutex. | +| Lazy MFE SIMD dispatch | [2.7.2 dispatcher](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/utils/higher_order_functions.c#L55-L128) replaces a shared function pointer on first use. | Keep global MFE under synchronization as well; this is a different path from the accessibility recurrence. | +| Diagnostics | [2.7.2 logger](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/utils/log.c#L489-L552) lazily allocates callback storage; its default lock depends on the library's pthread build configuration. | OpenMP support alone is not proof for diagnostic paths. Install a dedicated logger lock through the public callback API before concurrent calls. | + +## Boundary and limitations + +The synchronization boundary must use a process-wide mutex shared by +`VrnaHandler`, `AccessibilityVrna`, and `InteractionEnergyVrna`; an OpenMP +critical section limited to one call site cannot cover the shared setup +state. RAII locking permits normal C++ exception unwinding and works when +IntaRNA itself is built without OpenMP. + +The parallel boundary is **create, constrain, and prepare while +locked; compute the window recurrence with exclusive ownership of the +compound**. Keep global MFE/PF calls synchronized. RNAlib builds without +OpenMP use a conservative full-call fallback. Do not concurrently mutate +the handler model, reload global parameter tables through external RNAlib +calls, modify the same compound, or replace callback state while a call is +running. IntaRNA's mutex cannot coordinate direct RNAlib calls made by other +libraries in an embedding application. + +For ViennaRNA 2.7 and later, handler initialization installs a process-lifetime +mutex callback once through `vrna_log_lock_set()`. This replaces the logger's +locking hook while preserving its output streams, levels, and message +callbacks. The logger mutex is distinct from the setup mutex because guarded +ViennaRNA operations can emit messages themselves. This protects diagnostics +even if RNAlib was built without pthread support. Embedding applications must +not reset or replace that hook while IntaRNA folds are running; callbacks must +not reenter folding. Earlier reviewed logging uses stdio directly rather than +the newer lazily initialized callback array. See the +[public locking callback](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/utils/log.h#L61-L70) +and [hook setter](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/utils/log.c#L284-L290). + +The 2.4.14 source review establishes API availability and identifies its +state, but does not substitute for testing that release. Runtime comparisons +on 2.7.2 do not prove every intermediate release or build configuration safe. +Repeat the audit when changing the supported RNAlib range, adding model +mutations after preparation, changing SHAPE integration, or adding callbacks. + +## Runtime evidence and reproduction + +The diagnostic compares every unpaired probability returned by the callback +against a serial baseline, using independent compounds and callback storage. +It distinguishes fully concurrent lifecycles, serial creation followed by +parallel window calls, and synchronized preparation followed by parallel +window calls. Varying models matters: a shared cache can appear harmless when +all workers request identical parameters. + +The committed [standalone diagnostic](vrna-window-threads.cpp) uses 24 +deterministic sequences of varied lengths, windows, spans, and hard/soft +constraints. Its default profile also varies temperature and model flags. +`shared-model` fixes temperature to 37 degrees Celsius and uses the same +`noLP`/`noGUclosure` settings across cases, demonstrating that varying window +dimensions alone is sufficient to expose the cache race. + +Build with a C++23 compiler, OpenMP, and RNAlib's pkg-config metadata available: + +```bash +c++ -std=c++23 -O2 -fopenmp doc/analysis/vrna-window-threads.cpp \ + -o /tmp/vrna-window-threads $(pkg-config --cflags --libs RNAlib2) +/tmp/vrna-window-threads setup-locked 8 12 +/tmp/vrna-window-threads setup-locked 8 30 shared-model +/tmp/vrna-window-threads window-only 8 12 +``` + +The following modes intentionally exercise unsynchronized RNAlib state. Run +them as separate diagnostic processes, never as ordinary CI pass conditions: + +```bash +/tmp/vrna-window-threads setup-only 8 30 shared-model +/tmp/vrna-window-threads window-locked 8 12 +/tmp/vrna-window-threads unsafe 8 30 shared-model +``` + +Observed results used ViennaRNA 2.7.2's static library with OpenMP enabled, +GCC 14.4.0 (conda-forge build 14.4.0-7), and Linux x86-64 with CPU affinity +0-11. Runs requested eight workers; other build activity could share these +CPUs. No ViennaRNA 2.4.14 runtime was installed for the experiment. + +| Mode/profile | Completed repetitions | Exact probability comparisons | Result | +| --- | ---: | ---: | --- | +| `setup-locked`, mixed model | 12 | 4,046,016 | No parameter, probability, or callback metadata mismatch | +| `setup-locked`, shared model | 30 | 10,115,040 | No parameter, probability, or callback metadata mismatch | +| `window-only`, mixed model | 12 | 4,046,016 | No parameter, probability, or callback metadata mismatch | +| `setup-only`, shared model | 1 | Not applicable | 17 compounds with mismatched parameter snapshots | +| `window-locked`, mixed model | 2 | 674,336 | 6,120 probability mismatches, one parameter mismatch, one metadata mismatch | +| `unsafe`, shared model | 1 | 337,168 | 6,120 probability mismatches, one parameter mismatch, one metadata mismatch | + +A one-worker `unsafe` control completed three repetitions with 1,011,504 +exact comparisons and no mismatches. Five fresh `setup-only` processes also +each reproduced parameter corruption. The failing modes stop at the first +iteration containing a mismatch, so repetition counts and mismatch counts +will vary with scheduling. These experiments locate a concrete failure +outside the original recurrence-only lock; finite passing stress runs are +not a race-detector proof or a performance measurement. diff --git a/doc/local-validation-archive/files/issue-265-validation/debug/src/IntaRNA/AccessibilityVrna.h b/doc/local-validation-archive/files/issue-265-validation/debug/src/IntaRNA/AccessibilityVrna.h new file mode 100644 index 00000000..1972843d --- /dev/null +++ b/doc/local-validation-archive/files/issue-265-validation/debug/src/IntaRNA/AccessibilityVrna.h @@ -0,0 +1,190 @@ + +#ifndef INTARNA_ACCESSIBILITYVIENNA_H_ +#define INTARNA_ACCESSIBILITYVIENNA_H_ + +#include "IntaRNA/Accessibility.h" +#include "IntaRNA/VrnaHandler.h" + +#include "IntaRNA/Matrix.h" + +#include + + +extern "C" { + #include + #include +} + + +namespace IntaRNA { + +/** + * Computes accessibility energy values for _all_ _regions_ using the free + * energies of structure ensembles based on partition function computations + * via the Vienna RNA package. + * Independent objects can be constructed concurrently once global energy + * parameter loading is complete. Window folding runs in parallel when the + * ViennaRNA library was built with OpenMP support; otherwise it is serialized. + * + * @author Martin Mann 2014 + */ +class AccessibilityVrna : public Accessibility { + +public: + + + /** + * Construction + * @param sequence the sequence the accessibility data belongs to + * @param maxLength the maximal window size of accessible regions to be + * considered. 0 defaults to the sequence's length. + * @param accConstraint if not NULL, accessibility constraint that enforces some regions + * to be unstructured both in sequence and interaction + * @param vrnaHandler the VRNA parameter handler to be used + * @param plFoldW the sliding window size to be used for plFold computations + * @oaram pfScale can be used to explicitly set pf_scale of VRNA partition function computation. + * Note: only used for values >= 1.0, all other values are ignored. + */ + AccessibilityVrna( const RnaSequence& sequence + , const size_t maxLength + , const AccessibilityConstraint * const accConstraint + , const VrnaHandler & vrnaHandler + , const size_t plFoldW = 0 + , const double pfScale = VrnaHandler::getPfScaleDefault() + ); + + /** + * destruction + */ + virtual ~AccessibilityVrna(); + + /** + * Returns the accessibility energy value for the given range in the + * sequence, i.e. the energy difference (ED) to make the region accessible. + * + * @param from the start index of the regions (from <= to) + * @param to the end index of the regions (to < seq.length) + * + * @return the ED value if (j-1+1) <= maxLength or ED_UPPER_BOUND otherwise + * + * @throw std::runtime_error in case it does not hold 0 <= from <= to < seq.length + */ + virtual + E_type + getED( const size_t from, const size_t to ) const; + + + /** + * Adds constraints to the VRNA fold compound that correspond to the present + * AccessibilityConstraints + * The caller must hold VrnaHandler::getSetupMutex() during concurrent use of + * ViennaRNA, including preparation of SHAPE soft constraints. + * @param fold_compound INOUT the object to extend + * @param acc the accessibility handler to get constraint information from + */ + static + void + addConstraints( vrna_fold_compound_t & fold_compound + , const Accessibility & acc ); + + + /** + * Write the stored ED matrix directly, without copying rows or calling + * getED() for unconstrained data. Constraints retain the generic getED() + * path. Subclasses changing getED() semantics must also override this method. + * @param out binary output stream; compression is supplied by the stream + */ + void writeBinary( std::ostream & out ) const override; + +protected: + + //! type for the ED value matrix (upper triangular matrix banded by maxLength) + typedef UpperBandedMatrix EdMatrix; + + //! the ED values for the given sequence + EdMatrix edValues; + + + /** + * Use RNAplfold-like style to fill ED-values + * + * @param vrnaHandler the VRNA handler to be used + * @param plFoldW the sliding window size to be used or 0 for full length + * @param plFoldL the maximal base pair span to be used or 0 for plFoldW + * @oaram pfScale can be used to explicitly set pf_scale of VRNA partition function computation. + * Note: only used for values >= 1.0, all other values are ignored. + */ + void + fillByRNAplfold( const VrnaHandler &vrnaHandler + , const size_t plFoldW + , const size_t plFoldL + , const double pfScale); + + /** + * callback function used when calling vrna_probs_window() + * + * This function will be called for each probability data set in the sliding + * window probability computation implementation of vrna_probs_window(). + * The argument @a type specifies the type of probability that is passed to + * this function. + * + * @see vrna_probs_window() + * + * @param pr An array of probabilities + * @param pr_size The length of the probability array + * @param j The j-position (3'-end) of the probability intervals (indexing starting with 1) + * @param max The (theoretical) maximum length of the probability array + * @param type The type of probability that is passed to this function + * @param storageRT Auxiliary data: should hold a pair(AccessibilityVrna*,double RT) + * + */ + static + void + callbackForStorage( FLT_OR_DBL *pr, + int pr_size, + int j, + int max, + unsigned int type, + void *storageRT); + +}; + + +///////////////////////////////////////////////////////////////////////////// +///////////////////////////////////////////////////////////////////////////// +///////////////////////////////////////////////////////////////////////////// + +inline +void +AccessibilityVrna::writeBinary( std::ostream & out ) const +{ + Accessibility::writeBinary(out, &edValues); +} + +inline +E_type +AccessibilityVrna:: +getED( const size_t from, const size_t to ) const +{ + // input range check + checkIndices(from,to); + + if ((to-from+1) <= getMaxLength() +1 ) { // "getMaxLength()+1" to allow for dangling-end probability computation + // check for constrained end positions + if (!getAccConstraint().isAccessible(from) || !getAccConstraint().isAccessible(to)) { + // end position blocked --> omit accessibility + return ED_UPPER_BOUND; + } + // return according ED value from the precomputed matrix + return edValues (from,to); + } else { + // region length exceeds maximally allowed length -> no value + return ED_UPPER_BOUND; + } +} + +///////////////////////////////////////////////////////////////////////////// + +} // namespace + +#endif /* ACCESSIBILITYVIENNA_H_ */ diff --git a/doc/local-validation-archive/files/issue-265-validation/fallback/override/ViennaRNA/vrna_config.h b/doc/local-validation-archive/files/issue-265-validation/fallback/override/ViennaRNA/vrna_config.h new file mode 100644 index 00000000..37f9f677 --- /dev/null +++ b/doc/local-validation-archive/files/issue-265-validation/fallback/override/ViennaRNA/vrna_config.h @@ -0,0 +1,2 @@ +#include "/home/alex/CodexProjects/IntaRNA/issue-246-validation/toolchain/include/ViennaRNA/vrna_config.h" +#undef VRNA_WITH_OPENMP diff --git a/doc/local-validation-archive/files/issue-265-validation/fallback/probe.cpp b/doc/local-validation-archive/files/issue-265-validation/fallback/probe.cpp new file mode 100644 index 00000000..4a101ff3 --- /dev/null +++ b/doc/local-validation-archive/files/issue-265-validation/fallback/probe.cpp @@ -0,0 +1,72 @@ +#include "IntaRNA/AccessibilityVrna.h" +#include "IntaRNA/InteractionEnergyVrna.h" +#include "IntaRNA/ReverseAccessibility.h" +#include +#include +#include +#include +#include +#include +#include +INITIALIZE_EASYLOGGINGPP +static_assert(INTARNA_MULITHREADING == 0); +#ifdef _OPENMP +#error This probe must compile without OpenMP enabled in IntaRNA. +#endif +std::atomic active{0}, maximum{0}; +extern "C" int __real_vrna_probs_window(vrna_fold_compound_t *, int, unsigned int, vrna_probs_window_callback *, void *); +extern "C" int __wrap_vrna_probs_window(vrna_fold_compound_t *fc, int u, unsigned int o, vrna_probs_window_callback *cb, void *data) { + const int now = ++active; + int seen = maximum.load(); + while (seen < now && !maximum.compare_exchange_weak(seen, now)) {} + const int result = __real_vrna_probs_window(fc, u, o, cb, data); + --active; + return result; +} +int main() { + constexpr int count = 16; + IntaRNA::VrnaHandler handler; + std::vector sequences; + for (int i = 0; i < count; ++i) { + std::string seq; + for (int j = 0; j < 128 + i; ++j) seq += "GGGAAACCCUGUA"[(j + i) % 13]; + sequences.emplace_back(std::to_string(i), seq); + } + auto calculate = [&](int i) { + auto &seq = sequences[i]; + IntaRNA::AccessibilityConstraint constraint(seq.size(), 55 + i % 4, "", "", ""); + IntaRNA::AccessibilityVrna acc(seq, 24, &constraint, handler, 72 + i % 4); + std::vector result; + for (size_t j = 0; j < seq.size(); ++j) + for (size_t k = j; k < seq.size() && k < j + 25; ++k) result.push_back(acc.getED(j,k)); + IntaRNA::ReverseAccessibility reverse(acc); + IntaRNA::InteractionEnergyVrna energy(acc, reverse, handler, 16, 16, true); + result.push_back(energy.getEall1()); + result.push_back(energy.getEall2()); + result.push_back(energy.getES1(0, seq.size()-1)); + return result; + }; + std::vector> expected(count), actual(count); + for (int i = 0; i < count; ++i) expected[i] = calculate(i); + maximum = 0; + for (int repetition = 0; repetition < 3; ++repetition) { + std::atomic next{0}; + std::barrier start(4); + std::vector workers; + for (int t = 0; t < 4; ++t) workers.emplace_back([&] { + start.arrive_and_wait(); + for (int i; (i = next++) < count;) actual[i] = calculate(i); + }); + for (auto &worker: workers) worker.join(); + for (int i = 0; i < count; ++i) if (actual[i] != expected[i]) return 2; + } + std::cout << "INTARNA_MULITHREADING=" << INTARNA_MULITHREADING + << " threads=4 cases=16 repetitions=3 all_energy_values_equal=1 max_window_calls=" << maximum << '\n'; +#ifdef VRNA_WITH_OPENMP + std::cout << "RNAlib header OpenMP enabled (control)\n"; + return maximum > 1 ? 0 : 3; +#else + std::cout << "RNAlib header OpenMP disabled by test override; underlying RNAlib remains OpenMP-enabled\n"; + return maximum == 1 ? 0 : 4; +#endif +} diff --git a/doc/local-validation-archive/files/issue-265-validation/fallback/run.py b/doc/local-validation-archive/files/issue-265-validation/fallback/run.py new file mode 100644 index 00000000..0a3ff603 --- /dev/null +++ b/doc/local-validation-archive/files/issue-265-validation/fallback/run.py @@ -0,0 +1,20 @@ +from pathlib import Path +import subprocess, shlex, os +p=Path(__file__).resolve().parent +repo=Path('/home/alex/CodexProjects/IntaRNA/pr-worktrees/issue-265') +deps=Path('/home/alex/CodexProjects/IntaRNA/issue-246-validation/toolchain') +cxx='/home/alex/CodexProjects/IntaRNA/issue-246-validation/toolchain-gcc14/bin/x86_64-conda-linux-gnu-g++' +flags=[cxx,'-std=c++23','-O2','-DNDEBUG','-DINTARNA_MULITHREADING=0','-pthread','-DELPP_FEATURE_PERFORMANCE_TRACKING','-DELPP_NO_DEFAULT_LOG_FILE','-Wno-deprecated','-I'+str(repo/'src'),'-I'+str(deps/'include'),'-I'+str(deps/'include/ViennaRNA')] +log=(p/'commands.log').open('w') +def run(args): + print(shlex.join(args),file=log,flush=True) + subprocess.run(['taskset','-c','0-7']+args,stdout=log,stderr=subprocess.STDOUT,check=True) +for name in ['VrnaHandler','InteractionEnergyVrna']: + run(flags+['-c',str(repo/'src/IntaRNA'/ (name+'.cpp')),'-o',str(p/(name+'.o'))]) +for mode in ['fallback','control']: + f=flags[:] + if mode=='fallback': f[1:1]=['-I'+str(p/'override')] + run(f+['-c',str(repo/'src/IntaRNA/AccessibilityVrna.cpp'),'-o',str(p/('AccessibilityVrna-'+mode+'.o'))]) + run(f+[str(p/'probe.cpp'),str(p/'VrnaHandler.o'),str(p/'InteractionEnergyVrna.o'),str(p/('AccessibilityVrna-'+mode+'.o')),str(repo/'src/IntaRNA/libIntaRNA.a'),str(repo/'src/libeasylogging.a'),'-L'+str(deps/'lib'),'-Wl,-rpath,'+str(deps/'lib'),'-Wl,--wrap=vrna_probs_window','-lboost_regex','-lboost_iostreams','-lboost_serialization','-lRNA','-lgomp','-lgsl','-lgslcblas','-lmpfr','-lgmp','-lz','-o',str(p/('probe-'+mode))]) + run([str(p/('probe-'+mode))]) +print('Both branch probes passed') diff --git a/doc/local-validation-archive/files/issue-265-validation/probe/vrna_window_threads.cpp b/doc/local-validation-archive/files/issue-265-validation/probe/vrna_window_threads.cpp new file mode 100644 index 00000000..0052a8df --- /dev/null +++ b/doc/local-validation-archive/files/issue-265-validation/probe/vrna_window_threads.cpp @@ -0,0 +1,140 @@ +// Standalone diagnostic for IntaRNA #265. This is a stress probe, not proof of thread safety. +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +extern "C" { +#include +#include +#include +#include +#include +#include +} + +struct Case { + std::string sequence; + std::string constraint; + vrna_md_t model; + int window; +}; +struct Result { + int length, window, status = 0, callbacks = 0; + bool invalid = false; + std::vector values; + std::vector seen; + Result(int n, int w): length(n), window(w), values((n+1)*(w+1)), seen(values.size()) {} +}; +using Compound = std::unique_ptr; +Compound compound(const Case &c) { + vrna_md_t model = c.model; + Compound fc(vrna_fold_compound(c.sequence.c_str(), &model, VRNA_OPTION_PF | VRNA_OPTION_WINDOW), &vrna_fold_compound_free); + if (!fc) { std::cerr << "fold compound allocation failed\n"; std::exit(3); } + if (!c.constraint.empty()) vrna_constraints_add(fc.get(), c.constraint.c_str(), VRNA_CONSTRAINT_DB_DEFAULT | VRNA_CONSTRAINT_DB_ENFORCE_BP); + return fc; +} +void capture(FLT_OR_DBL *pr, int pr_size, int j, int max, unsigned int type, void *data) { + auto &r = *static_cast(data); + if (!(type & VRNA_PROBS_WINDOW_UP)) return; + ++r.callbacks; + if (j < 1 || j > r.length) { r.invalid = true; return; } + for (int l = 1; l <= std::min({j, pr_size, max, r.window}); ++l) { + const size_t pos = j*(r.window+1)+l; + r.values[pos] = pr[l]; r.seen[pos] = 1; + if (!std::isfinite(pr[l]) || pr[l] < 0 || pr[l] > 1.00000001) r.invalid = true; + } +} +void calculate(vrna_fold_compound_t *fc, const Case &c, Result &r) { + r.status = vrna_probs_window(fc, c.window, VRNA_PROBS_WINDOW_UP, capture, &r); +} +std::vector cases() { + std::vector cs; + uint32_t rng = 0x265; + for (int id=0; id<24; ++id) { + Case c; + const int n = 77 + id*13; + for (int i=0; i>17; rng ^= rng<<5; c.sequence += "ACGU"[rng%4]; } + // Include stable stem motifs among pseudorandom backgrounds. + if (id%4==0) c.sequence.replace(10, 20, "GGGGGGGAAAAAACCCCCCC"); + c.window = std::min(n, 32+(id%7)*16); + vrna_md_set_default(&c.model); + c.model.window_size = c.window; + c.model.max_bp_span = (id%3==0 ? c.window-8 : c.window); + c.model.temperature = 20 + (id%5)*8.5; + c.model.noLP = id%2; + c.model.noGUclosure = (id/2)%2; + c.model.dangles = 2; + c.model.gquad = 0; + c.model.compute_bpp = 1; + if (id%3==1) { c.constraint.assign(n, '.'); c.constraint[0] = c.constraint[n-1] = 'x'; } + cs.push_back(c); + } + return cs; +} +int main(int argc, char **argv) { + const int threads = argc>1 ? std::atoi(argv[1]) : 4; + const int repetitions = argc>2 ? std::atoi(argv[2]) : 10; + const bool lifecycle = argc>3 && std::string(argv[3])=="lifecycle"; + const bool lock = argc>4 && std::string(argv[4])=="lock"; + const bool setupLock = argc>4 && std::string(argv[4])=="setup-lock"; + if (threads<1 || repetitions<1) return 2; + omp_set_dynamic(0); omp_set_num_threads(threads); + vrna_params_load_RNA_Turner2004(); + const auto inputs = cases(); + std::vector reference; + for (const auto &c: inputs) { + reference.emplace_back(c.sequence.size(), c.window); + auto fc=compound(c); calculate(fc.get(), c, reference.back()); + if (!reference.back().status || reference.back().invalid) { std::cerr<<"invalid serial baseline\n"; return 3; } + } + size_t compared = 0, mismatches = 0; + const auto start = std::chrono::steady_clock::now(); + for (int rep=0; rep observed; + std::vector compounds; + for (const auto &c: inputs) { + observed.emplace_back(c.sequence.size(), c.window); + compounds.emplace_back(lifecycle ? nullptr : compound(c).release(), &vrna_fold_compound_free); + } + #pragma omp parallel for schedule(dynamic, 1) + for (int id=0; id(inputs.size()); ++id) { + if (lifecycle) { + if (setupLock) { + #pragma omp critical(probe_setup) + compounds[id] = compound(inputs[id]); + } else compounds[id] = compound(inputs[id]); + } + if (lock) { + #pragma omp critical(probe_probs_window) + calculate(compounds[id].get(), inputs[id], observed[id]); + } else calculate(compounds[id].get(), inputs[id], observed[id]); + if (lifecycle) compounds[id].reset(); + } + for (size_t id=0; id "$validation/logs/vrna-2.7.0-configure.log" 2>&1 +# ViennaRNA 2.7.0 omits utils CPPFLAGS when SVM support is disabled. +# Supply its ordinary project flags without changing the upstream sources. +make -C src/ViennaRNA -j2 'libRNA_utils_la_CPPFLAGS=$(AM_CPPFLAGS)' > "$validation/logs/vrna-2.7.0-build.log" 2>&1 +make -C src/ViennaRNA install 'libRNA_utils_la_CPPFLAGS=$(AM_CPPFLAGS)' > "$validation/logs/vrna-2.7.0-install.log" 2>&1 +make install-pkgconfigDATA >> "$validation/logs/vrna-2.7.0-install.log" 2>&1 diff --git a/doc/local-validation-archive/files/issue-267-validation/check-installed.sh b/doc/local-validation-archive/files/issue-267-validation/check-installed.sh new file mode 100644 index 00000000..c8c8a78c --- /dev/null +++ b/doc/local-validation-archive/files/issue-267-validation/check-installed.sh @@ -0,0 +1,32 @@ +#!/usr/bin/env bash +set -euo pipefail +task_root=/home/alex/CodexProjects/IntaRNA +task_validation="$task_root/issue-267-validation" +task_deps="$task_root/issue-246-validation/toolchain" +task_cxx="$task_root/issue-246-validation/toolchain-gcc14/bin/x86_64-conda-linux-gnu-g++" +cd "$task_root/pr-worktrees/issue-267" +make install > "$task_validation/logs/install-release.log" 2>&1 +cat > "$task_validation/installed-consumer.cpp" <<'CPP' +#include +#include +#include +INITIALIZE_EASYLOGGINGPP +int main() { + IntaRNA::VrnaHandler handler; + IntaRNA::RnaSequence sequence("test", "GGGGAAAACCCC"); + IntaRNA::AccessibilityVrna accessibility(sequence, 6, nullptr, handler, 10); + return accessibility.getED(0, 2) < IntaRNA::E_INF ? 0 : 1; +} +CPP +export PKG_CONFIG_PATH="$task_validation/install/lib/pkgconfig:$task_deps/lib/pkgconfig" +"$task_cxx" -std=c++23 -fopenmp -DELPP_FEATURE_PERFORMANCE_TRACKING -DELPP_NO_DEFAULT_LOG_FILE \ + "$task_validation/installed-consumer.cpp" \ + -L"$task_deps/lib" -Wl,-rpath,"$task_deps/lib" \ + $(pkg-config --cflags --libs IntaRNA) -leasylogging -o "$task_validation/installed-consumer" +"$task_validation/installed-consumer" +for task_header in "$task_validation/install/include/IntaRNA/"*.h; do + printf '#include \nint main() { return 0; }\n' "$(basename "$task_header")" > "$task_validation/header-probe.cpp" + "$task_cxx" -std=c++23 -I"$task_validation/install/include" -I"$task_deps/include" \ + -fsyntax-only "$task_validation/header-probe.cpp" +done +printf 'Installed consumer and %s standalone public headers passed.\n' "$(find "$task_validation/install/include/IntaRNA" -maxdepth 1 -name '*.h' | wc -l)" diff --git a/doc/local-validation-archive/files/issue-267-validation/configure-fixtures/missing-logger-declaration/include/ViennaRNA/utils/log.h b/doc/local-validation-archive/files/issue-267-validation/configure-fixtures/missing-logger-declaration/include/ViennaRNA/utils/log.h new file mode 100644 index 00000000..36533012 --- /dev/null +++ b/doc/local-validation-archive/files/issue-267-validation/configure-fixtures/missing-logger-declaration/include/ViennaRNA/utils/log.h @@ -0,0 +1 @@ +/* Test fixture: headers omit the required logger declaration. */ diff --git a/doc/local-validation-archive/files/issue-267-validation/configure-fixtures/missing-logger-symbol/include/ViennaRNA/utils/log.h b/doc/local-validation-archive/files/issue-267-validation/configure-fixtures/missing-logger-symbol/include/ViennaRNA/utils/log.h new file mode 100644 index 00000000..ca0897ba --- /dev/null +++ b/doc/local-validation-archive/files/issue-267-validation/configure-fixtures/missing-logger-symbol/include/ViennaRNA/utils/log.h @@ -0,0 +1,2 @@ +#define vrna_log_lock_set intarna_test_unavailable_logger_lock_set +#include "/home/alex/CodexProjects/IntaRNA/issue-246-validation/toolchain/include/ViennaRNA/utils/log.h" diff --git a/doc/local-validation-archive/files/issue-267-validation/configure-fixtures/old-headers/include/ViennaRNA/vrna_config.h b/doc/local-validation-archive/files/issue-267-validation/configure-fixtures/old-headers/include/ViennaRNA/vrna_config.h new file mode 100644 index 00000000..ee61111f --- /dev/null +++ b/doc/local-validation-archive/files/issue-267-validation/configure-fixtures/old-headers/include/ViennaRNA/vrna_config.h @@ -0,0 +1,91 @@ +#ifndef VIENNA_RNA_PACKAGE_CONFIG_H +#define VIENNA_RNA_PACKAGE_CONFIG_H + +/* version number */ +#define VRNA_VERSION "2.6.4" + +#define VRNA_VERSION_MAJOR 2 +#define VRNA_VERSION_MINOR 6 +#define VRNA_VERSION_PATCH 4 + +/* + * The following pre-processor definitions specify whether + * or not certain features were activated upon build-time + */ + +/* + * Build with deactivated C11 Features + * + * It this feature is missing, the next line defines + * 'VRNA_DISABLE_C11_FEATURES' + */ + + +/* + * Build with OpenMP support + * + * If this feature is present, the next line defines + * 'VRNA_WITH_OPENMP' + */ +#define VRNA_WITH_OPENMP + +/* + * Build with single precision partition function + * + * If this feature is present, the next line defines + * 'USE_FLOAT_PF' + */ + + +/* + * Build with JSON input/output support + * + * If this feature is present, the next line defines + * 'VRNA_WITH_JSON_SUPPORT' + * + * Note: ViennaRNA >= 2.6.3 is always built with JSON support. + * This feature is still present to keep third-party code relying on it working. + */ +#define VRNA_WITH_JSON_SUPPORT + +/* + * Build with Support Vector Machine (SVM) Z-score feature in RNALfold + * + * If this feature is present, the next line defines + * 'VRNA_WITH_SVM' + */ +#define VRNA_WITH_SVM + +/* + * Build with GSL minimizers + * + * If this feature is present, the next line defines + * 'VRNA_WITH_GSL' + */ +#define VRNA_WITH_GSL + +/* + * Build with colored TTY output + * + * If this feature is missing, the next line defines + * 'VRNA_WITHOUT_TTY_COLORS' + */ + + +/* + * Build with Link Time Optimization support + * + * If this feature is enabled, the next line defines + * 'VRNA_WITH_LTO' + */ + + +/* + * Build with Naview Layout algorithm of Bruccoleri 1988 + * + * If this feature is enabled, the next line defines + * 'VRNA_WITH_NAVIEW_LAYOUT' + */ +#define VRNA_WITH_NAVIEW_LAYOUT + +#endif diff --git a/doc/local-validation-archive/files/issue-267-validation/debug/doc/analysis/vienna-accessibility-thread-safety.md b/doc/local-validation-archive/files/issue-267-validation/debug/doc/analysis/vienna-accessibility-thread-safety.md new file mode 100644 index 00000000..3fdb218c --- /dev/null +++ b/doc/local-validation-archive/files/issue-267-validation/debug/doc/analysis/vienna-accessibility-thread-safety.md @@ -0,0 +1,311 @@ +# ViennaRNA accessibility thread-safety audit + +This audit accompanies [issue #265](https://github.com/BackofenLab/IntaRNA/issues/265). +The former critical section covered only `vrna_probs_window()`. That boundary +did not protect parameter construction: independent fold compounds can race +while ViennaRNA initializes them. Conversely, the sliding-window recurrence +uses compound-owned state and can run concurrently after synchronized setup. + +## Reviewed upstream sources + +IntaRNA requires **ViennaRNA 2.7.0 or later**. The review covers that release's +required logging API and the **2.7.2** implementation used for the runtime +experiments. The earlier **2.4.14** review remains as historical evidence; +that release is no longer supported. Source links below are pinned to immutable +commits: + +| Release | Commit | Scope | +| --- | --- | --- | +| 2.4.14 | [`0b8ca4c6604e7a8bea366af62d82f8f40cea49bf`](https://github.com/ViennaRNA/ViennaRNA/tree/0b8ca4c6604e7a8bea366af62d82f8f40cea49bf) | Historical implementation comparison; unsupported | +| 2.7.0 | [`219394580aec203a9d6f0d5450021e22642d5a83`](https://github.com/ViennaRNA/ViennaRNA/tree/219394580aec203a9d6f0d5450021e22642d5a83) | Minimum supported release and public logger locking API | +| 2.7.2 | [`1ffec79f5e258896160f7362ced8263450f371dc`](https://github.com/ViennaRNA/ViennaRNA/tree/1ffec79f5e258896160f7362ced8263450f371dc) | Installed runtime and current parameter cache implementation | + +ViennaRNA 2.7.0 introduced the flexible logger, as recorded in its +[release notes](https://github.com/ViennaRNA/ViennaRNA/releases/tag/v2.7.0). +Its public [`vrna_log_lock_set()` interface](https://github.com/ViennaRNA/ViennaRNA/blob/219394580aec203a9d6f0d5450021e22642d5a83/src/ViennaRNA/utils/log.h#L379-L398) +lets IntaRNA protect diagnostics even when RNAlib has no built-in pthread +locking. This interface determines the minimum supported version. Configure +checks the header version and links the required logger and prepared-window +APIs, including when pkg-config is disabled; the version check cannot be +bypassed to build without the logger lock. + +This is a review of the paths IntaRNA calls, not a blanket claim that RNAlib, +its deprecated interfaces, or every future release is thread-safe. In +particular, compiling RNAlib with OpenMP does not make all its static state +thread-private. + +## Call path and ownership + +The relevant accessibility path is: + +```text +AccessibilityVrna::fillByRNAplfold + VrnaHandler::getModel -> vrna_md_copy + vrna_fold_compound(PF | WINDOW) + add_params -> vrna_params + vrna_params_prepare -> vrna_exp_params + sanitize_bp_span, sequence/constraint/matrix setup + addConstraints + vrna_constraints_add -> vrna_hc_add_from_db + vrna_constraints_add_SHAPE, when requested + vrna_fold_compound_prepare(PF | WINDOW) + vrna_probs_window + vrna_fold_compound_prepare(PF | WINDOW) + allocate local auxiliary arrays + update compound-owned hard/soft constraints and DP matrices + callbackForStorage -> this accessibility object's ED matrix + vrna_fold_compound_free +``` + +The explicit preparation before `vrna_probs_window()` is significant. The +window routine calls `vrna_fold_compound_prepare()` internally, and parameter +preparation can replace `exp_params` when its model differs from `params`. +Construction sanitizes span/window settings after initially constructing +parameters. Thus the synchronization rule must cover preparation as well as +the constructor. See the +[2.7.2 compound constructor and preparation](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/fold_compound.c#L200-L276), +[preparation call chain](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/fold_compound.c#L519-L626), +and [parameter replacement condition](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/params/params.c#L427-L456). + +After explicit preparation, the second preparation is idempotent with respect +to parameter construction for this path: the two parameter models match, +constraints do not change them, and no other worker accesses this compound. +Hard/soft constraint preparation and matrix scaling operate on its owned +storage. This reasoning assumes IntaRNA's single-sequence model, with no +external callbacks or custom grammar that mutate model settings. See +[hard constraints](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/constraints/hard.c#L276-L310), +[soft constraints](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/constraints/soft.c#L304-L356), +and [matrix preparation](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/datastructures/dp_matrices.c#L299-L390). + +## Shared state and required synchronization + +| State or operation | Evidence | Consequence | +| --- | --- | --- | +| Parameter caches `p_pre`, `pf_pre`, and their initialization flags | [2.7.2 `params.c`](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/params/params.c#L88-L217) enables `SPEEDUP_PARAMS`; both constructors rewrite cached model members and copy whole cached structures. Only `id` and `pf_id` are declared OpenMP thread-private. | Guard every IntaRNA entry to parameter construction with one shared mutex. Separate locks for accessibility and energy construction would not protect the same caches. | +| Parameter identifiers in historical RNAlib | [2.4.14 declarations](https://github.com/ViennaRNA/ViennaRNA/blob/0b8ca4c6604e7a8bea366af62d82f8f40cea49bf/src/ViennaRNA/params/params.c#L88-L98), [identifier increment](https://github.com/ViennaRNA/ViennaRNA/blob/0b8ca4c6604e7a8bea366af62d82f8f40cea49bf/src/ViennaRNA/params/params.c#L552) | The unsupported 2.4.14 release lacks the newer caches, but its identifier is shared in builds without OpenMP. This explains why removing the setup lock was not safe under the former version requirement. | +| Global model defaults and legacy model variables | [2.7.2 `vrna_md_defaults_reset`](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/model.c#L346-L432), [2.4.14 implementation](https://github.com/ViennaRNA/ViennaRNA/blob/0b8ca4c6604e7a8bea366af62d82f8f40cea49bf/src/ViennaRNA/model.c#L311-L384) | Passing a model to `vrna_md_defaults_reset()` changes process-wide defaults; it does not merely reset that local object. Remove the worker-side call when every fold already receives its explicit model. Guard parameter-file loading/default reads during handler setup. | +| `vrna_md_copy` | [2.7.2 implementation](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/model.c#L196-L232) copies the supplied structure and its arrays. | Copying a stable handler model into worker-local storage needs no global lock. | +| Sliding-window DP, auxiliary arrays, and constraint updates | [2.7.2 window implementation](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/partfunc/pf_window.c#L681-L958), [2.4.14 implementation](https://github.com/ViennaRNA/ViennaRNA/blob/0b8ca4c6604e7a8bea366af62d82f8f40cea49bf/src/ViennaRNA/LPfold.c#L669-L941) obtain arrays and models from the compound or allocate local helpers. | Independent prepared compounds can compute concurrently. Keep callback storage distinct and alive until each call returns. | +| Window compatibility state | [2.7.2 declarations](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/partfunc/pf_window.c#L91-L110), [legacy wrapper](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/partfunc/pf_window.c#L2170-L2243) | `backward_compat_compound` and `backward_compat` belong to the legacy wrapper path. The direct `vrna_probs_window()` call does not set them. Do not replace the direct API with a legacy wrapper under the same concurrency assumptions. | +| SHAPE D, Z, and W input | [2.7.2 implementation](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/probing/SHAPE.c#L332-L404), [2.4.14 implementation](https://github.com/ViennaRNA/ViennaRNA/blob/0b8ca4c6604e7a8bea366af62d82f8f40cea49bf/src/ViennaRNA/constraints/SHAPE.c#L61-L120) | Setup loads input into the compound's soft constraints. W reads supplied pseudoenergies; this API does not perform stochastic optimization. Include constraint setup in the guarded preparation boundary. | +| Destruction | [2.7.2 `vrna_fold_compound_free`](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/fold_compound.c#L117-L197) frees compound-owned arrays, constraints, and auxiliary data. | Ordinary accessibility destruction does not mutate the parameter caches. Each compound must have one owner. | +| Global PF compatibility pointer `pr` | [2.7.2 `vrna_pf`](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/partfunc/partfunc.c#L208-L222), [2.4.14 implementation](https://github.com/ViennaRNA/ViennaRNA/blob/0b8ca4c6604e7a8bea366af62d82f8f40cea49bf/src/ViennaRNA/part_func.c#L129-L142) still write the process-global legacy pointer. | The accessibility conclusion does not justify unlocking global PF calls. Serialize the energy-model global-fold paths with the same mutex. | +| Lazy MFE SIMD dispatch | [2.7.2 dispatcher](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/utils/higher_order_functions.c#L55-L128) replaces a shared function pointer on first use. | Keep global MFE under synchronization as well; this is a different path from the accessibility recurrence. | +| Diagnostics | [2.7.2 logger](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/utils/log.c#L489-L552) lazily allocates callback storage; its default lock depends on the library's pthread build configuration. | OpenMP support alone is not proof for diagnostic paths. Install a dedicated logger lock through the public callback API before concurrent calls. | + +## Boundary and limitations + +The synchronization boundary must use a process-wide mutex shared by +`VrnaHandler`, `AccessibilityVrna`, and `InteractionEnergyVrna`; an OpenMP +critical section limited to one call site cannot cover the shared setup +state. RAII locking permits normal C++ exception unwinding and works when +IntaRNA itself is built without OpenMP. + +The parallel boundary is **create, constrain, and prepare while +locked; compute the window recurrence with exclusive ownership of the +compound**. Keep global MFE/PF calls synchronized. RNAlib builds without +OpenMP use conservative full-call serialization. This is a property of the +RNAlib build configuration and remains necessary independently of the minimum +version. Do not concurrently mutate +the handler model, reload global parameter tables through external RNAlib +calls, modify the same compound, or replace callback state while a call is +running. IntaRNA's mutex cannot coordinate direct RNAlib calls made by other +libraries in an embedding application. + +This guarantee concerns the ViennaRNA boundary. Native-thread library users +must separately provide thread-safe IntaRNA/easylogging++ diagnostics, or leave +verbose/performance logging disabled. The CLI's OpenMP logging synchronization +is not a general native-thread logging contract. + +For every supported ViennaRNA version, handler initialization installs a +process-lifetime mutex callback once through `vrna_log_lock_set()`. This +replaces the logger's +locking hook while preserving its output streams, levels, and message +callbacks. The logger mutex is distinct from the setup mutex because guarded +ViennaRNA operations can emit messages themselves. This protects diagnostics +even if RNAlib was built without pthread support. Embedding applications must +not reset or replace that hook while IntaRNA folds are running; callbacks must +not reenter folding. There is no version-dependent path that omits the logger +lock. See the +[public locking callback](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/utils/log.h#L61-L70) +and [hook setter](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/utils/log.c#L284-L290). + +The 2.7.0 source review establishes the required public logging API, but +runtime comparisons below were performed on 2.7.2. Neither the historical +2.4.14 review nor those comparisons establish runtime behavior for every +supported release or build configuration. +Repeat the audit when changing the supported RNAlib range, adding model +mutations after preparation, changing SHAPE integration, or adding callbacks. + +## Runtime evidence and reproduction + +The diagnostic compares every unpaired probability returned by the callback +against a serial baseline, using independent compounds and callback storage. +It distinguishes fully concurrent lifecycles, serial creation followed by +parallel window calls, and synchronized preparation followed by parallel +window calls. Varying models matters: a shared cache can appear harmless when +all workers request identical parameters. + +The committed [standalone diagnostic](vrna-window-threads.cpp) uses 24 +deterministic sequences of varied lengths, windows, spans, and hard/soft +constraints. Its default profile also varies temperature and model flags. +`shared-model` fixes temperature to 37 degrees Celsius and uses the same +`noLP`/`noGUclosure` settings across cases, demonstrating that varying window +dimensions alone is sufficient to expose the cache race. + +Build with a C++23 compiler, OpenMP, and RNAlib's pkg-config metadata available: + +```bash +c++ -std=c++23 -O2 -fopenmp doc/analysis/vrna-window-threads.cpp \ + -o /tmp/vrna-window-threads $(pkg-config --cflags --libs RNAlib2) +/tmp/vrna-window-threads setup-locked 8 12 +/tmp/vrna-window-threads setup-locked 8 30 shared-model +/tmp/vrna-window-threads window-only 8 12 +``` + +The following modes intentionally exercise unsynchronized RNAlib state. Run +them as separate diagnostic processes, never as ordinary CI pass conditions: + +```bash +/tmp/vrna-window-threads setup-only 8 30 shared-model +/tmp/vrna-window-threads window-locked 8 12 +/tmp/vrna-window-threads unsafe 8 30 shared-model +``` + +Observed results used ViennaRNA 2.7.2's static library with OpenMP enabled, +GCC 14.4.0 (conda-forge build 14.4.0-7), and Linux x86-64 with CPU affinity +0-11. Runs requested eight workers; other build activity could share these +CPUs. These recorded experiments did not test ViennaRNA 2.7.0 or 2.4.14 +runtimes. + +| Mode/profile | Completed repetitions | Exact probability comparisons | Result | +| --- | ---: | ---: | --- | +| `setup-locked`, mixed model | 12 | 4,046,016 | No parameter, probability, or callback metadata mismatch | +| `setup-locked`, shared model | 30 | 10,115,040 | No parameter, probability, or callback metadata mismatch | +| `window-only`, mixed model | 12 | 4,046,016 | No parameter, probability, or callback metadata mismatch | +| `setup-only`, shared model | 1 | Not applicable | 17 compounds with mismatched parameter snapshots | +| `window-locked`, mixed model | 2 | 674,336 | 6,120 probability mismatches, one parameter mismatch, one metadata mismatch | +| `unsafe`, shared model | 1 | 337,168 | 6,120 probability mismatches, one parameter mismatch, one metadata mismatch | + +A one-worker `unsafe` control completed three repetitions with 1,011,504 +exact comparisons and no mismatches. Five fresh `setup-only` processes also +each reproduced parameter corruption. The failing modes stop at the first +iteration containing a mismatch, so repetition counts and mismatch counts +will vary with scheduling. These experiments locate a concrete failure +outside the original recurrence-only lock; finite passing stress runs are +not a race-detector proof or a performance measurement. + +## Integrated regressions and existing SHAPE limitations + +For the original synchronization change, GCC 14 release and `--enable-debug` +builds each passed `make tests -j4`: all seven test programs/scripts passed, +including 75,901 API assertions in 69 test cases. The source base was `c5f1d51` +on `master` (the repository has no +`main` branch). `git diff --check` and shell syntax validation also passed. + +`tests/AccessibilityVrna_test.cpp` compares all retained ED values (including +the extra band used for dangling ends), ES matrices, initiation energies and +both global ensemble energies against serial references. Twelve input/model +combinations vary sequence length, temperature, window, span, scaling and +hard constraints. Four native threads and four OpenMP workers each perform +48 jobs; assertions and exception reporting happen after workers join. + +`tests/runAccessibilityThreads.sh` compares sorted predictions and every +per-sequence accessibility export at one and two workers, with SHAPE D and W +as well as unconstrained input. Unconstrained and D cases also request Eall +output, exercising overlapping target accessibility and global folding. + +Two pre-existing serial failures in the installed ViennaRNA 2.7.2 path limit +SHAPE coverage. Z input crashes even without Eall output; W input crashes +when reported interactions request Eall. Both occur with the unmodified base +binary and the corrected binary at one thread. They are not introduced by +parallel window calculation, and are outside this synchronization change. +In particular, the existing `addConstraints()` passes window-specific options +also when called for global Eall, which is inappropriate for W's +`vrna_sc_set_up()` path. The regressions therefore exercise W without Eall and +do not claim working Z support. + +A minimal serial Z reproduction, using either binary, is: + +```bash +for i in $(seq 1 16); do printf '%s 0.5\n' "$i"; done > /tmp/reactivities.shape +IntaRNA --query=GGGGAAAACCCCUAGC --target=GGGGAAAACCCCUAGC --seedBP=3 \ + --qIntLenMax=8 --tIntLenMax=8 --qAccW=14 --tAccW=14 --qAccL=10 --tAccL=10 \ + --qShape=/tmp/reactivities.shape --tShape=/tmp/reactivities.shape \ + --qShapeMethod=Z --tShapeMethod=Z --outMode=C --outCsvCols=id1,id2,E,ED1,ED2 \ + --outMaxE=100 --threads=1 --default-log-file=/dev/null +``` + +An additional branch probe compiled the three changed implementation units +with `INTARNA_MULITHREADING=0` and without `-fopenmp`. Four native workers +matched serial ED, ES and Eall values for 48 jobs. A linker wrapper observed +four simultaneous `vrna_probs_window()` calls with the normal header and one +with an isolated header override removing `VRNA_WITH_OPENMP`. This verifies +both the parallel and serialized branches. It is **not** a runtime test of a +non-OpenMP RNAlib: the linked library remained the installed OpenMP-enabled +2.7.2, and unchanged IntaRNA objects came from the release archive. + +## End-to-end measurement + +The base and corrected binaries used the same GCC 14.4 release toolchain, +ViennaRNA 2.7.2 and options. The workload has 32 deterministic 400-nt queries, +a 6-nt target, and 120-nt accessibility windows. Kinetic interaction prediction +keeps this workload focused on accessibility. Every run exports all query ED +values as well as interaction structures, coordinates and energies. + +On a Ryzen 5 7530U, affinity CPUs 8-11 exposes four logical CPUs on two physical +cores. One warmup preceded three alternating measured runs per configuration. +Other compilation activity ran on CPUs 0-7 during this first measurement, so +these numbers are workload observations rather than general speedup claims. + +| Workers | Binary | Median wall time (range), s | Median peak RSS, KiB | +| ---: | --- | ---: | ---: | +| 1 | Base | 4.15 (4.11-5.07) | 18,376 | +| 1 | Corrected | 4.76 (4.51-5.92) | 18,392 | +| 4 | Base | 6.24 (4.71-6.48) | 30,396 | +| 4 | Corrected | 2.01 (1.98-2.03) | 30,564 | + +The four-worker median improved by 3.10x with comparable peak memory. To +check the noisy single-worker result, four further alternating runs pinned +to CPU 8 after compilation finished measured 3.865 s (3.77-4.09) for the base +and 3.890 s (3.84-3.95) for the correction. That 0.025-s difference lies within +the observed run variation; it does not establish zero overhead. + +All 24 runs matched the **base serial** reference exactly: sorted CSV output +(32 interaction rows) and all 32 ED exports (114,048 values per run). The +unmodified multiworker binary also happened to match on these inputs; its +passing benchmark does not negate the separate race reproducer. + +Generate the inputs with Python 3: + +```bash +mkdir -p /tmp/intarna-265-benchmark +python3 - <<'PY' +from pathlib import Path +from random import Random +out = Path('/tmp/intarna-265-benchmark') +rng = Random(265) +out.joinpath('queries.fa').write_text(''.join( + '>query%02d\n%s\n' % (i + 1, ''.join(rng.choice('ACGU') for _ in range(400))) + for i in range(32))) +out.joinpath('target.fa').write_text('>target\nGCGCGC\n') +PY +``` + +Use the following command with each binary, worker count, and a fresh output +directory for each repetition. Set `binary`, `threads`, and `out_dir` first; +adjust the CPU affinity to the local machine. The serial followup uses `8` +instead of `8-11`. + +```bash +mkdir -p "$out_dir" +export OMP_DYNAMIC=FALSE OMP_PROC_BIND=TRUE OMP_PLACES=cores OPENBLAS_NUM_THREADS=1 +/usr/bin/time -f '%e %U %S %M' -o "$out_dir/time.txt" \ + taskset -c 8-11 "$binary" \ + --query=/tmp/intarna-265-benchmark/queries.fa \ + --target=/tmp/intarna-265-benchmark/target.fa \ + --model=X --mode=K --seedBP=3 --qIntLenMax=8 --tIntLenMax=6 \ + --qAccW=120 --qAccL=120 --tAccW=6 --tAccL=6 --threads="$threads" \ + --outMode=C --outCsvCols=id1,start1,end1,id2,start2,end2,subseqDP,hybridDP,E,ED1,ED2 \ + --out="$out_dir/predictions.csv" --out="qAcc:$out_dir/query.ed" +``` diff --git a/doc/local-validation-archive/files/issue-267-validation/header-probe.cpp b/doc/local-validation-archive/files/issue-267-validation/header-probe.cpp new file mode 100644 index 00000000..2b45fa6d --- /dev/null +++ b/doc/local-validation-archive/files/issue-267-validation/header-probe.cpp @@ -0,0 +1,2 @@ +#include +int main() { return 0; } diff --git a/doc/local-validation-archive/files/issue-267-validation/installed-consumer.cpp b/doc/local-validation-archive/files/issue-267-validation/installed-consumer.cpp new file mode 100644 index 00000000..dbf5ddd7 --- /dev/null +++ b/doc/local-validation-archive/files/issue-267-validation/installed-consumer.cpp @@ -0,0 +1,10 @@ +#include +#include +#include +INITIALIZE_EASYLOGGINGPP +int main() { + IntaRNA::VrnaHandler handler; + IntaRNA::RnaSequence sequence("test", "GGGGAAAACCCC"); + IntaRNA::AccessibilityVrna accessibility(sequence, 6, nullptr, handler, 10); + return accessibility.getED(0, 2) < IntaRNA::E_INF ? 0 : 1; +} diff --git a/doc/local-validation-archive/files/issue-267-validation/minimum/doc/analysis/vienna-accessibility-thread-safety.md b/doc/local-validation-archive/files/issue-267-validation/minimum/doc/analysis/vienna-accessibility-thread-safety.md new file mode 100644 index 00000000..ad12a3dd --- /dev/null +++ b/doc/local-validation-archive/files/issue-267-validation/minimum/doc/analysis/vienna-accessibility-thread-safety.md @@ -0,0 +1,308 @@ +# ViennaRNA accessibility thread-safety audit + +This audit accompanies [issue #265](https://github.com/BackofenLab/IntaRNA/issues/265). +The former critical section covered only `vrna_probs_window()`. That boundary +did not protect parameter construction: independent fold compounds can race +while ViennaRNA initializes them. Conversely, the sliding-window recurrence +uses compound-owned state and can run concurrently after synchronized setup. + +## Reviewed upstream sources + +IntaRNA requires **ViennaRNA 2.7.0 or later**. The review covers that release's +required logging API and the **2.7.2** implementation used for the runtime +experiments. The earlier **2.4.14** review remains as historical evidence; +that release is no longer supported. Source links below are pinned to immutable +commits: + +| Release | Commit | Scope | +| --- | --- | --- | +| 2.4.14 | [`0b8ca4c6604e7a8bea366af62d82f8f40cea49bf`](https://github.com/ViennaRNA/ViennaRNA/tree/0b8ca4c6604e7a8bea366af62d82f8f40cea49bf) | Historical implementation comparison; unsupported | +| 2.7.0 | [`219394580aec203a9d6f0d5450021e22642d5a83`](https://github.com/ViennaRNA/ViennaRNA/tree/219394580aec203a9d6f0d5450021e22642d5a83) | Minimum supported release and public logger locking API | +| 2.7.2 | [`1ffec79f5e258896160f7362ced8263450f371dc`](https://github.com/ViennaRNA/ViennaRNA/tree/1ffec79f5e258896160f7362ced8263450f371dc) | Installed runtime and current parameter cache implementation | + +ViennaRNA 2.7.0 introduced the flexible logger, as recorded in its +[release notes](https://github.com/ViennaRNA/ViennaRNA/releases/tag/v2.7.0). +Its public [`vrna_log_lock_set()` interface](https://github.com/ViennaRNA/ViennaRNA/blob/219394580aec203a9d6f0d5450021e22642d5a83/src/ViennaRNA/utils/log.h#L379-L398) +lets IntaRNA protect diagnostics even when RNAlib has no built-in pthread +locking. This interface determines the minimum supported version. Configure +checks the header version and links the required logger and prepared-window +APIs. Using `--disable-pkg-config` follows the same requirement. + +This is a review of the paths IntaRNA calls, not a blanket claim that RNAlib, +its deprecated interfaces, or every future release is thread-safe. In +particular, compiling RNAlib with OpenMP does not make all its static state +thread-private. + +## Call path and ownership + +The relevant accessibility path is: + +```text +AccessibilityVrna::fillByRNAplfold + VrnaHandler::getModel -> vrna_md_copy + vrna_fold_compound(PF | WINDOW) + add_params -> vrna_params + vrna_params_prepare -> vrna_exp_params + sanitize_bp_span, sequence/constraint/matrix setup + addConstraints + vrna_constraints_add -> vrna_hc_add_from_db + vrna_constraints_add_SHAPE, when requested + vrna_fold_compound_prepare(PF | WINDOW) + vrna_probs_window + vrna_fold_compound_prepare(PF | WINDOW) + allocate local auxiliary arrays + update compound-owned hard/soft constraints and DP matrices + callbackForStorage -> this accessibility object's ED matrix + vrna_fold_compound_free +``` + +The explicit preparation before `vrna_probs_window()` is significant. The +window routine calls `vrna_fold_compound_prepare()` internally, and parameter +preparation can replace `exp_params` when its model differs from `params`. +Construction sanitizes span/window settings after initially constructing +parameters. Thus the synchronization rule must cover preparation as well as +the constructor. See the +[2.7.2 compound constructor and preparation](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/fold_compound.c#L200-L276), +[preparation call chain](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/fold_compound.c#L519-L626), +and [parameter replacement condition](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/params/params.c#L427-L456). + +After explicit preparation, the second preparation is idempotent with respect +to parameter construction for this path: the two parameter models match, +constraints do not change them, and no other worker accesses this compound. +Hard/soft constraint preparation and matrix scaling operate on its owned +storage. This reasoning assumes IntaRNA's single-sequence model, with no +external callbacks or custom grammar that mutate model settings. See +[hard constraints](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/constraints/hard.c#L276-L310), +[soft constraints](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/constraints/soft.c#L304-L356), +and [matrix preparation](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/datastructures/dp_matrices.c#L299-L390). + +## Shared state and required synchronization + +| State or operation | Evidence | Consequence | +| --- | --- | --- | +| Parameter caches `p_pre`, `pf_pre`, and their initialization flags | [2.7.2 `params.c`](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/params/params.c#L88-L217) enables `SPEEDUP_PARAMS`; both constructors rewrite cached model members and copy whole cached structures. Only `id` and `pf_id` are declared OpenMP thread-private. | Guard every IntaRNA entry to parameter construction with one shared mutex. Separate locks for accessibility and energy construction would not protect the same caches. | +| Parameter identifiers in historical RNAlib | [2.4.14 declarations](https://github.com/ViennaRNA/ViennaRNA/blob/0b8ca4c6604e7a8bea366af62d82f8f40cea49bf/src/ViennaRNA/params/params.c#L88-L98), [identifier increment](https://github.com/ViennaRNA/ViennaRNA/blob/0b8ca4c6604e7a8bea366af62d82f8f40cea49bf/src/ViennaRNA/params/params.c#L552) | The unsupported 2.4.14 release lacks the newer caches, but its identifier is shared in builds without OpenMP. This explains why removing the setup lock was not safe under the former version requirement. | +| Global model defaults and legacy model variables | [2.7.2 `vrna_md_defaults_reset`](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/model.c#L346-L432), [2.4.14 implementation](https://github.com/ViennaRNA/ViennaRNA/blob/0b8ca4c6604e7a8bea366af62d82f8f40cea49bf/src/ViennaRNA/model.c#L311-L384) | Passing a model to `vrna_md_defaults_reset()` changes process-wide defaults; it does not merely reset that local object. Remove the worker-side call when every fold already receives its explicit model. Guard parameter-file loading/default reads during handler setup. | +| `vrna_md_copy` | [2.7.2 implementation](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/model.c#L196-L232) copies the supplied structure and its arrays. | Copying a stable handler model into worker-local storage needs no global lock. | +| Sliding-window DP, auxiliary arrays, and constraint updates | [2.7.2 window implementation](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/partfunc/pf_window.c#L681-L958), [2.4.14 implementation](https://github.com/ViennaRNA/ViennaRNA/blob/0b8ca4c6604e7a8bea366af62d82f8f40cea49bf/src/ViennaRNA/LPfold.c#L669-L941) obtain arrays and models from the compound or allocate local helpers. | Independent prepared compounds can compute concurrently. Keep callback storage distinct and alive until each call returns. | +| Window compatibility state | [2.7.2 declarations](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/partfunc/pf_window.c#L91-L110), [legacy wrapper](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/partfunc/pf_window.c#L2170-L2243) | `backward_compat_compound` and `backward_compat` belong to the legacy wrapper path. The direct `vrna_probs_window()` call does not set them. Do not replace the direct API with a legacy wrapper under the same concurrency assumptions. | +| SHAPE D, Z, and W input | [2.7.2 implementation](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/probing/SHAPE.c#L332-L404), [2.4.14 implementation](https://github.com/ViennaRNA/ViennaRNA/blob/0b8ca4c6604e7a8bea366af62d82f8f40cea49bf/src/ViennaRNA/constraints/SHAPE.c#L61-L120) | Setup loads input into the compound's soft constraints. W reads supplied pseudoenergies; this API does not perform stochastic optimization. Include constraint setup in the guarded preparation boundary. | +| Destruction | [2.7.2 `vrna_fold_compound_free`](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/fold_compound.c#L117-L197) frees compound-owned arrays, constraints, and auxiliary data. | Ordinary accessibility destruction does not mutate the parameter caches. Each compound must have one owner. | +| Global PF compatibility pointer `pr` | [2.7.2 `vrna_pf`](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/partfunc/partfunc.c#L208-L222), [2.4.14 implementation](https://github.com/ViennaRNA/ViennaRNA/blob/0b8ca4c6604e7a8bea366af62d82f8f40cea49bf/src/ViennaRNA/part_func.c#L129-L142) still write the process-global legacy pointer. | The accessibility conclusion does not justify unlocking global PF calls. Serialize the energy-model global-fold paths with the same mutex. | +| Lazy MFE SIMD dispatch | [2.7.2 dispatcher](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/utils/higher_order_functions.c#L55-L128) replaces a shared function pointer on first use. | Keep global MFE under synchronization as well; this is a different path from the accessibility recurrence. | +| Diagnostics | [2.7.2 logger](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/utils/log.c#L489-L552) lazily allocates callback storage; its default lock depends on the library's pthread build configuration. | OpenMP support alone is not proof for diagnostic paths. Install a dedicated logger lock through the public callback API before concurrent calls. | + +## Boundary and limitations + +The synchronization boundary must use a process-wide mutex shared by +`VrnaHandler`, `AccessibilityVrna`, and `InteractionEnergyVrna`; an OpenMP +critical section limited to one call site cannot cover the shared setup +state. RAII locking permits normal C++ exception unwinding and works when +IntaRNA itself is built without OpenMP. + +The parallel boundary is **create, constrain, and prepare while +locked; compute the window recurrence with exclusive ownership of the +compound**. Keep global MFE/PF calls synchronized. RNAlib builds without +OpenMP use conservative full-call serialization. This is a property of the +RNAlib build configuration and is retained independently of the minimum +version. Do not concurrently mutate the handler model, reload global parameter +tables through external RNAlib calls, modify the same compound, or replace +callback state while a call is running. IntaRNA's mutex cannot coordinate +direct RNAlib calls made by other libraries in an embedding application. + +This guarantee concerns the ViennaRNA boundary. Native-thread library users +must separately provide thread-safe IntaRNA/easylogging++ diagnostics, or leave +verbose/performance logging disabled. The CLI's OpenMP logging synchronization +is not a general native-thread logging contract. + +For every supported ViennaRNA version, handler initialization installs a +process-lifetime mutex callback once through `vrna_log_lock_set()`. This +replaces the logger's locking hook while preserving its output streams, levels, +and message callbacks. The logger mutex is distinct from the setup mutex +because guarded ViennaRNA operations can emit messages themselves. This +protects diagnostics even if RNAlib was built without pthread support. +Embedding applications must not reset or replace that hook while IntaRNA folds +are running; callbacks must not reenter folding. There is no version-dependent +path that omits the logger lock. See the +[public locking callback](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/utils/log.h#L61-L70) +and [hook setter](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/utils/log.c#L284-L290). + +The 2.7.0 source review establishes the required public logging API, but +runtime comparisons below were performed on 2.7.2. Neither the historical +2.4.14 review nor those comparisons establish runtime behavior for every +supported release or build configuration. +Repeat the audit when changing the supported RNAlib range, adding model +mutations after preparation, changing SHAPE integration, or adding callbacks. + +## Runtime evidence and reproduction + +The diagnostic compares every unpaired probability returned by the callback +against a serial baseline, using independent compounds and callback storage. +It distinguishes fully concurrent lifecycles, serial creation followed by +parallel window calls, and synchronized preparation followed by parallel +window calls. Varying models matters: a shared cache can appear harmless when +all workers request identical parameters. + +The committed [standalone diagnostic](vrna-window-threads.cpp) uses 24 +deterministic sequences of varied lengths, windows, spans, and hard/soft +constraints. Its default profile also varies temperature and model flags. +`shared-model` fixes temperature to 37 degrees Celsius and uses the same +`noLP`/`noGUclosure` settings across cases, demonstrating that varying window +dimensions alone is sufficient to expose the cache race. + +Build with a C++23 compiler, OpenMP, and RNAlib's pkg-config metadata available: + +```bash +c++ -std=c++23 -O2 -fopenmp doc/analysis/vrna-window-threads.cpp \ + -o /tmp/vrna-window-threads $(pkg-config --cflags --libs RNAlib2) +/tmp/vrna-window-threads setup-locked 8 12 +/tmp/vrna-window-threads setup-locked 8 30 shared-model +/tmp/vrna-window-threads window-only 8 12 +``` + +The following modes intentionally exercise unsynchronized RNAlib state. Run +them as separate diagnostic processes, never as ordinary CI pass conditions: + +```bash +/tmp/vrna-window-threads setup-only 8 30 shared-model +/tmp/vrna-window-threads window-locked 8 12 +/tmp/vrna-window-threads unsafe 8 30 shared-model +``` + +Observed results used ViennaRNA 2.7.2's static library with OpenMP enabled, +GCC 14.4.0 (conda-forge build 14.4.0-7), and Linux x86-64 with CPU affinity +0-11. Runs requested eight workers; other build activity could share these +CPUs. These recorded experiments did not test ViennaRNA 2.7.0 or 2.4.14 +runtimes. + +| Mode/profile | Completed repetitions | Exact probability comparisons | Result | +| --- | ---: | ---: | --- | +| `setup-locked`, mixed model | 12 | 4,046,016 | No parameter, probability, or callback metadata mismatch | +| `setup-locked`, shared model | 30 | 10,115,040 | No parameter, probability, or callback metadata mismatch | +| `window-only`, mixed model | 12 | 4,046,016 | No parameter, probability, or callback metadata mismatch | +| `setup-only`, shared model | 1 | Not applicable | 17 compounds with mismatched parameter snapshots | +| `window-locked`, mixed model | 2 | 674,336 | 6,120 probability mismatches, one parameter mismatch, one metadata mismatch | +| `unsafe`, shared model | 1 | 337,168 | 6,120 probability mismatches, one parameter mismatch, one metadata mismatch | + +A one-worker `unsafe` control completed three repetitions with 1,011,504 +exact comparisons and no mismatches. Five fresh `setup-only` processes also +each reproduced parameter corruption. The failing modes stop at the first +iteration containing a mismatch, so repetition counts and mismatch counts +will vary with scheduling. These experiments locate a concrete failure +outside the original recurrence-only lock; finite passing stress runs are +not a race-detector proof or a performance measurement. + +## Integrated regressions and existing SHAPE limitations + +For the original synchronization change, GCC 14 release and `--enable-debug` +builds each passed `make tests -j4`: all seven test programs/scripts passed, +including 75,901 API assertions in 69 test cases. The source base was `c5f1d51` +on `master` (the repository has no `main` branch). `git diff --check` and shell +syntax validation also passed. + +`tests/AccessibilityVrna_test.cpp` compares all retained ED values (including +the extra band used for dangling ends), ES matrices, initiation energies and +both global ensemble energies against serial references. Twelve input/model +combinations vary sequence length, temperature, window, span, scaling and +hard constraints. Four native threads and four OpenMP workers each perform +48 jobs; assertions and exception reporting happen after workers join. + +`tests/runAccessibilityThreads.sh` compares sorted predictions and every +per-sequence accessibility export at one and two workers, with SHAPE D and W +as well as unconstrained input. Unconstrained and D cases also request Eall +output, exercising overlapping target accessibility and global folding. + +Two pre-existing serial failures in the installed ViennaRNA 2.7.2 path limit +SHAPE coverage. Z input crashes even without Eall output; W input crashes +when reported interactions request Eall. Both occur with the unmodified base +binary and the corrected binary at one thread. They are not introduced by +parallel window calculation, and are outside this synchronization change. +In particular, the existing `addConstraints()` passes window-specific options +also when called for global Eall, which is inappropriate for W's +`vrna_sc_set_up()` path. The regressions therefore exercise W without Eall and +do not claim working Z support. + +A minimal serial Z reproduction, using either binary, is: + +```bash +for i in $(seq 1 16); do printf '%s 0.5\n' "$i"; done > /tmp/reactivities.shape +IntaRNA --query=GGGGAAAACCCCUAGC --target=GGGGAAAACCCCUAGC --seedBP=3 \ + --qIntLenMax=8 --tIntLenMax=8 --qAccW=14 --tAccW=14 --qAccL=10 --tAccL=10 \ + --qShape=/tmp/reactivities.shape --tShape=/tmp/reactivities.shape \ + --qShapeMethod=Z --tShapeMethod=Z --outMode=C --outCsvCols=id1,id2,E,ED1,ED2 \ + --outMaxE=100 --threads=1 --default-log-file=/dev/null +``` + +An additional branch probe compiled the three changed implementation units +with `INTARNA_MULITHREADING=0` and without `-fopenmp`. Four native workers +matched serial ED, ES and Eall values for 48 jobs. A linker wrapper observed +four simultaneous `vrna_probs_window()` calls with the normal header and one +with an isolated header override removing `VRNA_WITH_OPENMP`. This verifies +both the parallel and serialized branches. It is **not** a runtime test of a +non-OpenMP RNAlib: the linked library remained the installed OpenMP-enabled +2.7.2, and unchanged IntaRNA objects came from the release archive. + +## End-to-end measurement + +The base and corrected binaries used the same GCC 14.4 release toolchain, +ViennaRNA 2.7.2 and options. The workload has 32 deterministic 400-nt queries, +a 6-nt target, and 120-nt accessibility windows. Kinetic interaction prediction +keeps this workload focused on accessibility. Every run exports all query ED +values as well as interaction structures, coordinates and energies. + +On a Ryzen 5 7530U, affinity CPUs 8-11 exposes four logical CPUs on two physical +cores. One warmup preceded three alternating measured runs per configuration. +Other compilation activity ran on CPUs 0-7 during this first measurement, so +these numbers are workload observations rather than general speedup claims. + +| Workers | Binary | Median wall time (range), s | Median peak RSS, KiB | +| ---: | --- | ---: | ---: | +| 1 | Base | 4.15 (4.11-5.07) | 18,376 | +| 1 | Corrected | 4.76 (4.51-5.92) | 18,392 | +| 4 | Base | 6.24 (4.71-6.48) | 30,396 | +| 4 | Corrected | 2.01 (1.98-2.03) | 30,564 | + +The four-worker median improved by 3.10x with comparable peak memory. To +check the noisy single-worker result, four further alternating runs pinned +to CPU 8 after compilation finished measured 3.865 s (3.77-4.09) for the base +and 3.890 s (3.84-3.95) for the correction. That 0.025-s difference lies within +the observed run variation; it does not establish zero overhead. + +All 24 runs matched the **base serial** reference exactly: sorted CSV output +(32 interaction rows) and all 32 ED exports (114,048 values per run). The +unmodified multiworker binary also happened to match on these inputs; its +passing benchmark does not negate the separate race reproducer. + +Generate the inputs with Python 3: + +```bash +mkdir -p /tmp/intarna-265-benchmark +python3 - <<'PY' +from pathlib import Path +from random import Random +out = Path('/tmp/intarna-265-benchmark') +rng = Random(265) +out.joinpath('queries.fa').write_text(''.join( + '>query%02d\n%s\n' % (i + 1, ''.join(rng.choice('ACGU') for _ in range(400))) + for i in range(32))) +out.joinpath('target.fa').write_text('>target\nGCGCGC\n') +PY +``` + +Use the following command with each binary, worker count, and a fresh output +directory for each repetition. Set `binary`, `threads`, and `out_dir` first; +adjust the CPU affinity to the local machine. The serial followup uses `8` +instead of `8-11`. + +```bash +mkdir -p "$out_dir" +export OMP_DYNAMIC=FALSE OMP_PROC_BIND=TRUE OMP_PLACES=cores OPENBLAS_NUM_THREADS=1 +/usr/bin/time -f '%e %U %S %M' -o "$out_dir/time.txt" \ + taskset -c 8-11 "$binary" \ + --query=/tmp/intarna-265-benchmark/queries.fa \ + --target=/tmp/intarna-265-benchmark/target.fa \ + --model=X --mode=K --seedBP=3 --qIntLenMax=8 --tIntLenMax=6 \ + --qAccW=120 --qAccL=120 --tAccW=6 --tAccL=6 --threads="$threads" \ + --outMode=C --outCsvCols=id1,start1,end1,id2,start2,end2,subseqDP,hybridDP,E,ED1,ED2 \ + --out="$out_dir/predictions.csv" --out="qAcc:$out_dir/query.ed" +``` diff --git a/doc/local-validation-archive/files/issue-267-validation/probe-source/doc/analysis/vienna-accessibility-thread-safety.md b/doc/local-validation-archive/files/issue-267-validation/probe-source/doc/analysis/vienna-accessibility-thread-safety.md new file mode 100644 index 00000000..ad12a3dd --- /dev/null +++ b/doc/local-validation-archive/files/issue-267-validation/probe-source/doc/analysis/vienna-accessibility-thread-safety.md @@ -0,0 +1,308 @@ +# ViennaRNA accessibility thread-safety audit + +This audit accompanies [issue #265](https://github.com/BackofenLab/IntaRNA/issues/265). +The former critical section covered only `vrna_probs_window()`. That boundary +did not protect parameter construction: independent fold compounds can race +while ViennaRNA initializes them. Conversely, the sliding-window recurrence +uses compound-owned state and can run concurrently after synchronized setup. + +## Reviewed upstream sources + +IntaRNA requires **ViennaRNA 2.7.0 or later**. The review covers that release's +required logging API and the **2.7.2** implementation used for the runtime +experiments. The earlier **2.4.14** review remains as historical evidence; +that release is no longer supported. Source links below are pinned to immutable +commits: + +| Release | Commit | Scope | +| --- | --- | --- | +| 2.4.14 | [`0b8ca4c6604e7a8bea366af62d82f8f40cea49bf`](https://github.com/ViennaRNA/ViennaRNA/tree/0b8ca4c6604e7a8bea366af62d82f8f40cea49bf) | Historical implementation comparison; unsupported | +| 2.7.0 | [`219394580aec203a9d6f0d5450021e22642d5a83`](https://github.com/ViennaRNA/ViennaRNA/tree/219394580aec203a9d6f0d5450021e22642d5a83) | Minimum supported release and public logger locking API | +| 2.7.2 | [`1ffec79f5e258896160f7362ced8263450f371dc`](https://github.com/ViennaRNA/ViennaRNA/tree/1ffec79f5e258896160f7362ced8263450f371dc) | Installed runtime and current parameter cache implementation | + +ViennaRNA 2.7.0 introduced the flexible logger, as recorded in its +[release notes](https://github.com/ViennaRNA/ViennaRNA/releases/tag/v2.7.0). +Its public [`vrna_log_lock_set()` interface](https://github.com/ViennaRNA/ViennaRNA/blob/219394580aec203a9d6f0d5450021e22642d5a83/src/ViennaRNA/utils/log.h#L379-L398) +lets IntaRNA protect diagnostics even when RNAlib has no built-in pthread +locking. This interface determines the minimum supported version. Configure +checks the header version and links the required logger and prepared-window +APIs. Using `--disable-pkg-config` follows the same requirement. + +This is a review of the paths IntaRNA calls, not a blanket claim that RNAlib, +its deprecated interfaces, or every future release is thread-safe. In +particular, compiling RNAlib with OpenMP does not make all its static state +thread-private. + +## Call path and ownership + +The relevant accessibility path is: + +```text +AccessibilityVrna::fillByRNAplfold + VrnaHandler::getModel -> vrna_md_copy + vrna_fold_compound(PF | WINDOW) + add_params -> vrna_params + vrna_params_prepare -> vrna_exp_params + sanitize_bp_span, sequence/constraint/matrix setup + addConstraints + vrna_constraints_add -> vrna_hc_add_from_db + vrna_constraints_add_SHAPE, when requested + vrna_fold_compound_prepare(PF | WINDOW) + vrna_probs_window + vrna_fold_compound_prepare(PF | WINDOW) + allocate local auxiliary arrays + update compound-owned hard/soft constraints and DP matrices + callbackForStorage -> this accessibility object's ED matrix + vrna_fold_compound_free +``` + +The explicit preparation before `vrna_probs_window()` is significant. The +window routine calls `vrna_fold_compound_prepare()` internally, and parameter +preparation can replace `exp_params` when its model differs from `params`. +Construction sanitizes span/window settings after initially constructing +parameters. Thus the synchronization rule must cover preparation as well as +the constructor. See the +[2.7.2 compound constructor and preparation](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/fold_compound.c#L200-L276), +[preparation call chain](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/fold_compound.c#L519-L626), +and [parameter replacement condition](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/params/params.c#L427-L456). + +After explicit preparation, the second preparation is idempotent with respect +to parameter construction for this path: the two parameter models match, +constraints do not change them, and no other worker accesses this compound. +Hard/soft constraint preparation and matrix scaling operate on its owned +storage. This reasoning assumes IntaRNA's single-sequence model, with no +external callbacks or custom grammar that mutate model settings. See +[hard constraints](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/constraints/hard.c#L276-L310), +[soft constraints](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/constraints/soft.c#L304-L356), +and [matrix preparation](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/datastructures/dp_matrices.c#L299-L390). + +## Shared state and required synchronization + +| State or operation | Evidence | Consequence | +| --- | --- | --- | +| Parameter caches `p_pre`, `pf_pre`, and their initialization flags | [2.7.2 `params.c`](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/params/params.c#L88-L217) enables `SPEEDUP_PARAMS`; both constructors rewrite cached model members and copy whole cached structures. Only `id` and `pf_id` are declared OpenMP thread-private. | Guard every IntaRNA entry to parameter construction with one shared mutex. Separate locks for accessibility and energy construction would not protect the same caches. | +| Parameter identifiers in historical RNAlib | [2.4.14 declarations](https://github.com/ViennaRNA/ViennaRNA/blob/0b8ca4c6604e7a8bea366af62d82f8f40cea49bf/src/ViennaRNA/params/params.c#L88-L98), [identifier increment](https://github.com/ViennaRNA/ViennaRNA/blob/0b8ca4c6604e7a8bea366af62d82f8f40cea49bf/src/ViennaRNA/params/params.c#L552) | The unsupported 2.4.14 release lacks the newer caches, but its identifier is shared in builds without OpenMP. This explains why removing the setup lock was not safe under the former version requirement. | +| Global model defaults and legacy model variables | [2.7.2 `vrna_md_defaults_reset`](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/model.c#L346-L432), [2.4.14 implementation](https://github.com/ViennaRNA/ViennaRNA/blob/0b8ca4c6604e7a8bea366af62d82f8f40cea49bf/src/ViennaRNA/model.c#L311-L384) | Passing a model to `vrna_md_defaults_reset()` changes process-wide defaults; it does not merely reset that local object. Remove the worker-side call when every fold already receives its explicit model. Guard parameter-file loading/default reads during handler setup. | +| `vrna_md_copy` | [2.7.2 implementation](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/model.c#L196-L232) copies the supplied structure and its arrays. | Copying a stable handler model into worker-local storage needs no global lock. | +| Sliding-window DP, auxiliary arrays, and constraint updates | [2.7.2 window implementation](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/partfunc/pf_window.c#L681-L958), [2.4.14 implementation](https://github.com/ViennaRNA/ViennaRNA/blob/0b8ca4c6604e7a8bea366af62d82f8f40cea49bf/src/ViennaRNA/LPfold.c#L669-L941) obtain arrays and models from the compound or allocate local helpers. | Independent prepared compounds can compute concurrently. Keep callback storage distinct and alive until each call returns. | +| Window compatibility state | [2.7.2 declarations](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/partfunc/pf_window.c#L91-L110), [legacy wrapper](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/partfunc/pf_window.c#L2170-L2243) | `backward_compat_compound` and `backward_compat` belong to the legacy wrapper path. The direct `vrna_probs_window()` call does not set them. Do not replace the direct API with a legacy wrapper under the same concurrency assumptions. | +| SHAPE D, Z, and W input | [2.7.2 implementation](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/probing/SHAPE.c#L332-L404), [2.4.14 implementation](https://github.com/ViennaRNA/ViennaRNA/blob/0b8ca4c6604e7a8bea366af62d82f8f40cea49bf/src/ViennaRNA/constraints/SHAPE.c#L61-L120) | Setup loads input into the compound's soft constraints. W reads supplied pseudoenergies; this API does not perform stochastic optimization. Include constraint setup in the guarded preparation boundary. | +| Destruction | [2.7.2 `vrna_fold_compound_free`](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/fold_compound.c#L117-L197) frees compound-owned arrays, constraints, and auxiliary data. | Ordinary accessibility destruction does not mutate the parameter caches. Each compound must have one owner. | +| Global PF compatibility pointer `pr` | [2.7.2 `vrna_pf`](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/partfunc/partfunc.c#L208-L222), [2.4.14 implementation](https://github.com/ViennaRNA/ViennaRNA/blob/0b8ca4c6604e7a8bea366af62d82f8f40cea49bf/src/ViennaRNA/part_func.c#L129-L142) still write the process-global legacy pointer. | The accessibility conclusion does not justify unlocking global PF calls. Serialize the energy-model global-fold paths with the same mutex. | +| Lazy MFE SIMD dispatch | [2.7.2 dispatcher](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/utils/higher_order_functions.c#L55-L128) replaces a shared function pointer on first use. | Keep global MFE under synchronization as well; this is a different path from the accessibility recurrence. | +| Diagnostics | [2.7.2 logger](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/utils/log.c#L489-L552) lazily allocates callback storage; its default lock depends on the library's pthread build configuration. | OpenMP support alone is not proof for diagnostic paths. Install a dedicated logger lock through the public callback API before concurrent calls. | + +## Boundary and limitations + +The synchronization boundary must use a process-wide mutex shared by +`VrnaHandler`, `AccessibilityVrna`, and `InteractionEnergyVrna`; an OpenMP +critical section limited to one call site cannot cover the shared setup +state. RAII locking permits normal C++ exception unwinding and works when +IntaRNA itself is built without OpenMP. + +The parallel boundary is **create, constrain, and prepare while +locked; compute the window recurrence with exclusive ownership of the +compound**. Keep global MFE/PF calls synchronized. RNAlib builds without +OpenMP use conservative full-call serialization. This is a property of the +RNAlib build configuration and is retained independently of the minimum +version. Do not concurrently mutate the handler model, reload global parameter +tables through external RNAlib calls, modify the same compound, or replace +callback state while a call is running. IntaRNA's mutex cannot coordinate +direct RNAlib calls made by other libraries in an embedding application. + +This guarantee concerns the ViennaRNA boundary. Native-thread library users +must separately provide thread-safe IntaRNA/easylogging++ diagnostics, or leave +verbose/performance logging disabled. The CLI's OpenMP logging synchronization +is not a general native-thread logging contract. + +For every supported ViennaRNA version, handler initialization installs a +process-lifetime mutex callback once through `vrna_log_lock_set()`. This +replaces the logger's locking hook while preserving its output streams, levels, +and message callbacks. The logger mutex is distinct from the setup mutex +because guarded ViennaRNA operations can emit messages themselves. This +protects diagnostics even if RNAlib was built without pthread support. +Embedding applications must not reset or replace that hook while IntaRNA folds +are running; callbacks must not reenter folding. There is no version-dependent +path that omits the logger lock. See the +[public locking callback](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/utils/log.h#L61-L70) +and [hook setter](https://github.com/ViennaRNA/ViennaRNA/blob/1ffec79f5e258896160f7362ced8263450f371dc/src/ViennaRNA/utils/log.c#L284-L290). + +The 2.7.0 source review establishes the required public logging API, but +runtime comparisons below were performed on 2.7.2. Neither the historical +2.4.14 review nor those comparisons establish runtime behavior for every +supported release or build configuration. +Repeat the audit when changing the supported RNAlib range, adding model +mutations after preparation, changing SHAPE integration, or adding callbacks. + +## Runtime evidence and reproduction + +The diagnostic compares every unpaired probability returned by the callback +against a serial baseline, using independent compounds and callback storage. +It distinguishes fully concurrent lifecycles, serial creation followed by +parallel window calls, and synchronized preparation followed by parallel +window calls. Varying models matters: a shared cache can appear harmless when +all workers request identical parameters. + +The committed [standalone diagnostic](vrna-window-threads.cpp) uses 24 +deterministic sequences of varied lengths, windows, spans, and hard/soft +constraints. Its default profile also varies temperature and model flags. +`shared-model` fixes temperature to 37 degrees Celsius and uses the same +`noLP`/`noGUclosure` settings across cases, demonstrating that varying window +dimensions alone is sufficient to expose the cache race. + +Build with a C++23 compiler, OpenMP, and RNAlib's pkg-config metadata available: + +```bash +c++ -std=c++23 -O2 -fopenmp doc/analysis/vrna-window-threads.cpp \ + -o /tmp/vrna-window-threads $(pkg-config --cflags --libs RNAlib2) +/tmp/vrna-window-threads setup-locked 8 12 +/tmp/vrna-window-threads setup-locked 8 30 shared-model +/tmp/vrna-window-threads window-only 8 12 +``` + +The following modes intentionally exercise unsynchronized RNAlib state. Run +them as separate diagnostic processes, never as ordinary CI pass conditions: + +```bash +/tmp/vrna-window-threads setup-only 8 30 shared-model +/tmp/vrna-window-threads window-locked 8 12 +/tmp/vrna-window-threads unsafe 8 30 shared-model +``` + +Observed results used ViennaRNA 2.7.2's static library with OpenMP enabled, +GCC 14.4.0 (conda-forge build 14.4.0-7), and Linux x86-64 with CPU affinity +0-11. Runs requested eight workers; other build activity could share these +CPUs. These recorded experiments did not test ViennaRNA 2.7.0 or 2.4.14 +runtimes. + +| Mode/profile | Completed repetitions | Exact probability comparisons | Result | +| --- | ---: | ---: | --- | +| `setup-locked`, mixed model | 12 | 4,046,016 | No parameter, probability, or callback metadata mismatch | +| `setup-locked`, shared model | 30 | 10,115,040 | No parameter, probability, or callback metadata mismatch | +| `window-only`, mixed model | 12 | 4,046,016 | No parameter, probability, or callback metadata mismatch | +| `setup-only`, shared model | 1 | Not applicable | 17 compounds with mismatched parameter snapshots | +| `window-locked`, mixed model | 2 | 674,336 | 6,120 probability mismatches, one parameter mismatch, one metadata mismatch | +| `unsafe`, shared model | 1 | 337,168 | 6,120 probability mismatches, one parameter mismatch, one metadata mismatch | + +A one-worker `unsafe` control completed three repetitions with 1,011,504 +exact comparisons and no mismatches. Five fresh `setup-only` processes also +each reproduced parameter corruption. The failing modes stop at the first +iteration containing a mismatch, so repetition counts and mismatch counts +will vary with scheduling. These experiments locate a concrete failure +outside the original recurrence-only lock; finite passing stress runs are +not a race-detector proof or a performance measurement. + +## Integrated regressions and existing SHAPE limitations + +For the original synchronization change, GCC 14 release and `--enable-debug` +builds each passed `make tests -j4`: all seven test programs/scripts passed, +including 75,901 API assertions in 69 test cases. The source base was `c5f1d51` +on `master` (the repository has no `main` branch). `git diff --check` and shell +syntax validation also passed. + +`tests/AccessibilityVrna_test.cpp` compares all retained ED values (including +the extra band used for dangling ends), ES matrices, initiation energies and +both global ensemble energies against serial references. Twelve input/model +combinations vary sequence length, temperature, window, span, scaling and +hard constraints. Four native threads and four OpenMP workers each perform +48 jobs; assertions and exception reporting happen after workers join. + +`tests/runAccessibilityThreads.sh` compares sorted predictions and every +per-sequence accessibility export at one and two workers, with SHAPE D and W +as well as unconstrained input. Unconstrained and D cases also request Eall +output, exercising overlapping target accessibility and global folding. + +Two pre-existing serial failures in the installed ViennaRNA 2.7.2 path limit +SHAPE coverage. Z input crashes even without Eall output; W input crashes +when reported interactions request Eall. Both occur with the unmodified base +binary and the corrected binary at one thread. They are not introduced by +parallel window calculation, and are outside this synchronization change. +In particular, the existing `addConstraints()` passes window-specific options +also when called for global Eall, which is inappropriate for W's +`vrna_sc_set_up()` path. The regressions therefore exercise W without Eall and +do not claim working Z support. + +A minimal serial Z reproduction, using either binary, is: + +```bash +for i in $(seq 1 16); do printf '%s 0.5\n' "$i"; done > /tmp/reactivities.shape +IntaRNA --query=GGGGAAAACCCCUAGC --target=GGGGAAAACCCCUAGC --seedBP=3 \ + --qIntLenMax=8 --tIntLenMax=8 --qAccW=14 --tAccW=14 --qAccL=10 --tAccL=10 \ + --qShape=/tmp/reactivities.shape --tShape=/tmp/reactivities.shape \ + --qShapeMethod=Z --tShapeMethod=Z --outMode=C --outCsvCols=id1,id2,E,ED1,ED2 \ + --outMaxE=100 --threads=1 --default-log-file=/dev/null +``` + +An additional branch probe compiled the three changed implementation units +with `INTARNA_MULITHREADING=0` and without `-fopenmp`. Four native workers +matched serial ED, ES and Eall values for 48 jobs. A linker wrapper observed +four simultaneous `vrna_probs_window()` calls with the normal header and one +with an isolated header override removing `VRNA_WITH_OPENMP`. This verifies +both the parallel and serialized branches. It is **not** a runtime test of a +non-OpenMP RNAlib: the linked library remained the installed OpenMP-enabled +2.7.2, and unchanged IntaRNA objects came from the release archive. + +## End-to-end measurement + +The base and corrected binaries used the same GCC 14.4 release toolchain, +ViennaRNA 2.7.2 and options. The workload has 32 deterministic 400-nt queries, +a 6-nt target, and 120-nt accessibility windows. Kinetic interaction prediction +keeps this workload focused on accessibility. Every run exports all query ED +values as well as interaction structures, coordinates and energies. + +On a Ryzen 5 7530U, affinity CPUs 8-11 exposes four logical CPUs on two physical +cores. One warmup preceded three alternating measured runs per configuration. +Other compilation activity ran on CPUs 0-7 during this first measurement, so +these numbers are workload observations rather than general speedup claims. + +| Workers | Binary | Median wall time (range), s | Median peak RSS, KiB | +| ---: | --- | ---: | ---: | +| 1 | Base | 4.15 (4.11-5.07) | 18,376 | +| 1 | Corrected | 4.76 (4.51-5.92) | 18,392 | +| 4 | Base | 6.24 (4.71-6.48) | 30,396 | +| 4 | Corrected | 2.01 (1.98-2.03) | 30,564 | + +The four-worker median improved by 3.10x with comparable peak memory. To +check the noisy single-worker result, four further alternating runs pinned +to CPU 8 after compilation finished measured 3.865 s (3.77-4.09) for the base +and 3.890 s (3.84-3.95) for the correction. That 0.025-s difference lies within +the observed run variation; it does not establish zero overhead. + +All 24 runs matched the **base serial** reference exactly: sorted CSV output +(32 interaction rows) and all 32 ED exports (114,048 values per run). The +unmodified multiworker binary also happened to match on these inputs; its +passing benchmark does not negate the separate race reproducer. + +Generate the inputs with Python 3: + +```bash +mkdir -p /tmp/intarna-265-benchmark +python3 - <<'PY' +from pathlib import Path +from random import Random +out = Path('/tmp/intarna-265-benchmark') +rng = Random(265) +out.joinpath('queries.fa').write_text(''.join( + '>query%02d\n%s\n' % (i + 1, ''.join(rng.choice('ACGU') for _ in range(400))) + for i in range(32))) +out.joinpath('target.fa').write_text('>target\nGCGCGC\n') +PY +``` + +Use the following command with each binary, worker count, and a fresh output +directory for each repetition. Set `binary`, `threads`, and `out_dir` first; +adjust the CPU affinity to the local machine. The serial followup uses `8` +instead of `8-11`. + +```bash +mkdir -p "$out_dir" +export OMP_DYNAMIC=FALSE OMP_PROC_BIND=TRUE OMP_PLACES=cores OPENBLAS_NUM_THREADS=1 +/usr/bin/time -f '%e %U %S %M' -o "$out_dir/time.txt" \ + taskset -c 8-11 "$binary" \ + --query=/tmp/intarna-265-benchmark/queries.fa \ + --target=/tmp/intarna-265-benchmark/target.fa \ + --model=X --mode=K --seedBP=3 --qIntLenMax=8 --tIntLenMax=6 \ + --qAccW=120 --qAccL=120 --tAccW=6 --tAccL=6 --threads="$threads" \ + --outMode=C --outCsvCols=id1,start1,end1,id2,start2,end2,subseqDP,hybridDP,E,ED1,ED2 \ + --out="$out_dir/predictions.csv" --out="qAcc:$out_dir/query.ed" +``` diff --git a/doc/local-validation-archive/files/issue-267-validation/run-configure-probes.py b/doc/local-validation-archive/files/issue-267-validation/run-configure-probes.py new file mode 100644 index 00000000..7a97e25b --- /dev/null +++ b/doc/local-validation-archive/files/issue-267-validation/run-configure-probes.py @@ -0,0 +1,74 @@ +from pathlib import Path +import json, os, re, shutil, subprocess +root=Path('/home/alex/CodexProjects/IntaRNA') +source=root/'pr-worktrees/issue-267' +validation=root/'issue-267-validation' +probe_source=validation/'probe-source' +deps=root/'issue-246-validation/toolchain' +gcc14=root/'issue-246-validation/toolchain-gcc14/bin' +logs=validation/'logs' +files=subprocess.check_output(['git','ls-files','-z','--cached','--others','--exclude-standard'],cwd=source).decode().split('\0') +for name in files: + if not name: continue + target=probe_source/name + target.parent.mkdir(parents=True,exist_ok=True) + shutil.copy2(source/name,target) +with (logs/'bootstrap-probes.log').open('w') as log: + subprocess.run(['bash','autotools-init.sh'],cwd=probe_source,stdout=log,stderr=subprocess.STDOUT,check=True) +fixtures=validation/'configure-fixtures' +old=fixtures/'old-headers/include/ViennaRNA' +old.mkdir(parents=True,exist_ok=True) +oldconfig=(deps/'include/ViennaRNA/vrna_config.h').read_text() +oldconfig=re.sub(r'(#define VRNA_VERSION\s+)"[^"]+"',r'\1"2.6.4"',oldconfig) +oldconfig=re.sub(r'(#define VRNA_VERSION_MINOR\s+)\d+',r'\g<1>6',oldconfig) +oldconfig=re.sub(r'(#define VRNA_VERSION_PATCH\s+)\d+',r'\g<1>4',oldconfig) +(old/'vrna_config.h').write_text(oldconfig) +oldpc=fixtures/'old-metadata/lib/pkgconfig' +oldpc.mkdir(parents=True,exist_ok=True) +metadata=(deps/'lib/pkgconfig/RNAlib2.pc').read_text().replace('Version: 2.7.2','Version: 2.4.14') +(oldpc/'RNAlib2.pc').write_text(metadata) +missing_decl=fixtures/'missing-logger-declaration/include/ViennaRNA/utils' +missing_decl.mkdir(parents=True,exist_ok=True) +(missing_decl/'log.h').write_text('/* Test fixture: headers omit the required logger declaration. */\n') +missing_symbol=fixtures/'missing-logger-symbol/include/ViennaRNA/utils' +missing_symbol.mkdir(parents=True,exist_ok=True) +(missing_symbol/'log.h').write_text('#define vrna_log_lock_set intarna_test_unavailable_logger_lock_set\n#include "'+str(deps/'include/ViennaRNA/utils/log.h')+'"\n') +common=[str(probe_source/'configure'), + 'CC='+str(gcc14/'x86_64-conda-linux-gnu-gcc'), + 'CXX='+str(gcc14/'x86_64-conda-linux-gnu-g++'), + 'LDFLAGS=-L'+str(deps/'lib')+' -Wl,-rpath,'+str(deps/'lib'), + '--with-boost='+str(deps),'--with-boost-libdir='+str(deps/'lib'), + '--with-zlib='+str(deps),'--disable-debug'] +static_libs='LIBS=-lgomp -lgsl -lgslcblas -lpthread -lmpfr -lgmp -lstdc++ -lm' +cases=[ + ('no-pkg-config',0,[],['--disable-pkg-config',static_libs],deps,None), + ('old-pkg-metadata',1,[],[],fixtures/'old-metadata','Package dependency requirement'), + ('old-headers-no-pkg',1,[old.parent],['--disable-pkg-config',static_libs],deps,'ViennaRNA 2.7.0 or newer is required'), + ('old-headers-current-pkg',1,[old.parent],[],deps,'ViennaRNA 2.7.0 or newer is required'), + ('missing-logger-declaration',1,[missing_decl.parents[1]],[],deps,'vrna_log_lock_set'), + ('missing-logger-symbol',1,[missing_symbol.parents[1]],[],deps,'undefined reference to `intarna_test_unavailable_logger_lock_set'), +] +results=[] +for name,expected,inc,extra,vrna,diagnostic in cases: + build=validation/'configure-probes'/name + build.mkdir(parents=True,exist_ok=True) + cpp='CPPFLAGS='+' '.join('-I'+str(path) for path in inc+[deps/'include']) + args=common+[cpp,'--with-vrna='+str(vrna)]+extra + env=os.environ.copy() + for variable in ('VRNA_CFLAGS','VRNA_LIBS','PKG_CONFIG_PATH','PKG_CONFIG_LIBDIR'): + env.pop(variable,None) + with (logs/('configure-'+name+'.log')).open('w') as log: + completed=subprocess.run(args,cwd=build,env=env,stdout=log,stderr=subprocess.STDOUT) + config=(build/'config.log').read_text() + output=(logs/('configure-'+name+'.log')).read_text() + passed=(completed.returncode==0 if expected==0 else completed.returncode!=0) + if expected: + passed=passed and 'Vienna RNA C library version >= 2.7.0 is required.' in output + if diagnostic: + passed=passed and diagnostic in config + result={'case':name,'expected':'success' if expected==0 else 'rejection','exit_code':completed.returncode,'passed':passed,'command':args,'log':str(logs/('configure-'+name+'.log'))} + results.append(result) + print(json.dumps({key:result[key] for key in ('case','expected','exit_code','passed')}),flush=True) +(validation/'configure-probes/results.json').write_text(json.dumps(results,indent=2)+'\n') +if not all(r['passed'] for r in results): + raise SystemExit(1) diff --git a/doc/local-validation-archive/files/issue-267-validation/run-debug.sh b/doc/local-validation-archive/files/issue-267-validation/run-debug.sh new file mode 100644 index 00000000..cd00f744 --- /dev/null +++ b/doc/local-validation-archive/files/issue-267-validation/run-debug.sh @@ -0,0 +1,23 @@ +#!/usr/bin/env bash +set -euo pipefail +root=/home/alex/CodexProjects/IntaRNA +validation="$root/issue-267-validation" +deps="$root/issue-246-validation/toolchain" +gcc14="$root/issue-246-validation/toolchain-gcc14/bin" +cd "$validation/debug" +bash autotools-init.sh > "$validation/logs/bootstrap-debug.log" 2>&1 +./configure \ + CC="$gcc14/x86_64-conda-linux-gnu-gcc" \ + CXX="$gcc14/x86_64-conda-linux-gnu-g++" \ + CPPFLAGS="-I$deps/include" \ + LDFLAGS="-L$deps/lib -Wl,-rpath,$deps/lib" \ + --with-vrna="$deps" \ + --with-boost="$deps" \ + --with-boost-libdir="$deps/lib" \ + --with-zlib="$deps" \ + --enable-debug \ + --prefix="$validation/install-debug" > "$validation/logs/configure-debug.log" 2>&1 +make -j2 > "$validation/logs/build-debug.log" 2>&1 +make tests -j2 > "$validation/logs/tests-debug.log" 2>&1 +printf 'Debug configure, build, and full tests passed\n' +tail -18 "$validation/logs/tests-debug.log" diff --git a/doc/local-validation-archive/files/issue-267-validation/run-minimum.sh b/doc/local-validation-archive/files/issue-267-validation/run-minimum.sh new file mode 100644 index 00000000..c52d51f5 --- /dev/null +++ b/doc/local-validation-archive/files/issue-267-validation/run-minimum.sh @@ -0,0 +1,39 @@ +#!/bin/bash +set -euo pipefail +root=/home/alex/CodexProjects/IntaRNA +validation="$root/issue-267-validation" +minimum="$validation/vrna-2.7.0" +deps="$root/issue-246-validation/toolchain" +compiler="$root/issue-246-validation/toolchain-gcc14" +export PATH="$compiler/bin:$deps/bin:$PATH" +export LD_LIBRARY_PATH="$deps/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}" +export PKG_CONFIG_PATH="$minimum/lib/pkgconfig:$deps/lib/pkgconfig" +export OPENBLAS_NUM_THREADS=1 +cd "$validation/minimum" +pkg-config --modversion RNAlib2 > "$validation/logs/minimum-dependency-version.log" +printf '#include \nVRNA_VERSION\n' | \ + "$compiler/bin/x86_64-conda-linux-gnu-g++" -x c++ -E \ + -isystem "$minimum/include" -isystem "$deps/include" - \ + >> "$validation/logs/minimum-dependency-version.log" +bash autotools-init.sh > "$validation/logs/bootstrap-minimum.log" 2>&1 +./configure \ + CC="$compiler/bin/x86_64-conda-linux-gnu-gcc" \ + CXX="$compiler/bin/x86_64-conda-linux-gnu-g++" \ + CPPFLAGS="-isystem $minimum/include -isystem $deps/include" \ + LDFLAGS="-L$minimum/lib -L$deps/lib -Wl,-rpath,$deps/lib" \ + --with-vrna="$minimum" --with-boost="$deps" --with-boost-libdir="$deps/lib" \ + --with-zlib="$deps" --prefix="$validation/install-minimum" \ + > "$validation/logs/configure-minimum.log" 2>&1 +# Put the selected ViennaRNA archive ahead of the other dependency prefix. +# configure otherwise prepends the Boost library directory to AM_LDFLAGS. +link_flags="-L$minimum/lib -L$deps/lib -Wl,-rpath,$deps/lib" +make -j4 AM_LDFLAGS="$link_flags" > "$validation/logs/build-minimum.log" 2>&1 +make tests -j4 AM_LDFLAGS="$link_flags" > "$validation/logs/tests-minimum.log" 2>&1 +"$compiler/bin/x86_64-conda-linux-gnu-g++" -std=c++23 -O2 -fopenmp \ + -isystem "$minimum/include" -isystem "$deps/include" \ + doc/analysis/vrna-window-threads.cpp -o "$validation/vrna-window-threads-2.7.0" \ + -L"$minimum/lib" -L"$deps/lib" -Wl,-rpath,"$deps/lib" \ + $(pkg-config --libs RNAlib2) \ + > "$validation/logs/window-threads-minimum-build.log" 2>&1 +"$validation/vrna-window-threads-2.7.0" setup-locked 4 3 \ + > "$validation/logs/window-threads-minimum.log" 2>&1 diff --git a/doc/local-validation-archive/files/issue-268-validation/check-installed.py b/doc/local-validation-archive/files/issue-268-validation/check-installed.py new file mode 100644 index 00000000..097e7830 --- /dev/null +++ b/doc/local-validation-archive/files/issue-268-validation/check-installed.py @@ -0,0 +1,26 @@ +from pathlib import Path +import os, shlex, subprocess +root=Path(__file__).resolve().parent +build=root/'release';install=root/'install-release' +deps=root.parent/'issue-246-validation/toolchain' +cxx=root.parent/'issue-246-validation/toolchain-gcc14/bin/x86_64-conda-linux-gnu-g++' +log=root/'logs/installed-api.log' +with log.open('w') as stream: + def run(command,**kwargs): + print(shlex.join(map(str,command)),file=stream,flush=True) + return subprocess.run(command,check=True,stdout=stream,stderr=subprocess.STDOUT,**kwargs) + run(['make','-C',str(build),'install']) + for name in ('StackSeedDomain.h','SeededPartitionFunction.h'): + probe=root/('installed-'+name+'.cpp') + probe.write_text('#include \nint main() { return 0; }\n') + run([str(cxx),'-std=c++23','-I'+str(install/'include'),'-I'+str(deps/'include'),'-fsyntax-only',str(probe)]) + env=dict(os.environ,PKG_CONFIG_PATH=str(install/'lib/pkgconfig')) + flags=shlex.split(subprocess.check_output(['pkg-config','--cflags','--libs','IntaRNA'],env=env,text=True)) + exe=root/'installed-probabilities-example' + run([str(cxx),'-std=c++23',str(build/'doc/seeded-probabilities-example.cpp'), + '-L'+str(deps/'lib'),'-Wl,-rpath,'+str(deps/'lib'),*flags,'-o',str(exe)]) + result=subprocess.check_output([str(exe)],text=True,env=dict(env,OMP_NUM_THREADS='1')) + (root/'installed-probabilities-example.csv').write_text(result) + assert result.startswith('bpProb;'),result + print('Standalone headers and installed pkg-config probability consumer passed',file=stream,flush=True) +print('Installed API checks passed:',log) diff --git a/doc/local-validation-archive/files/issue-268-validation/compare-native-results.py b/doc/local-validation-archive/files/issue-268-validation/compare-native-results.py new file mode 100644 index 00000000..50389d06 --- /dev/null +++ b/doc/local-validation-archive/files/issue-268-validation/compare-native-results.py @@ -0,0 +1,28 @@ +from pathlib import Path +import csv, json, math +root=Path(__file__).resolve().parent/'native-comparison' +checks=[] +for before in sorted(root.glob('*-base.csv')): + after=before.with_name(before.name.replace('-base.csv','-candidate.csv')) + old=list(csv.reader(before.open(), delimiter=';')) + new=list(csv.reader(after.open(), delimiter=';')) + assert len(old)==len(new),(before,'row count') + maximum=0.; count=0 + for a,b in zip(old,new): + assert len(a)==len(b),(before,'column count') + for x,y in zip(a,b): + try: u,v=float(x),float(y) + except ValueError: + assert x==y,(before,x,y) + continue + assert math.isfinite(u) and math.isfinite(v),(before,x,y) + relative=abs(u-v)/max(abs(u),abs(v)) if u or v else 0. + maximum=max(maximum,relative);count+=1 + assert relative<2e-12,(before,x,y,relative) + assert before.with_suffix('.stdout').read_bytes()==after.with_suffix('.stdout').read_bytes(),(before,'native predictions differ') + assert before.with_suffix('.stderr').read_bytes()==after.with_suffix('.stderr').read_bytes(),(before,'diagnostics differ') + checks.append(dict(case=before.stem.removesuffix('-base'),probabilities=count, + max_relative_error=maximum,byte_identical=before.read_bytes()==after.read_bytes())) +assert len(checks)==12, len(checks) +(root/'results.json').write_text(json.dumps(checks,indent=2)+'\n') +print(json.dumps(checks,indent=2)) diff --git a/doc/local-validation-archive/files/issue-268-validation/installed-SeededPartitionFunction.h.cpp b/doc/local-validation-archive/files/issue-268-validation/installed-SeededPartitionFunction.h.cpp new file mode 100644 index 00000000..fa6f256c --- /dev/null +++ b/doc/local-validation-archive/files/issue-268-validation/installed-SeededPartitionFunction.h.cpp @@ -0,0 +1,2 @@ +#include +int main() { return 0; } diff --git a/doc/local-validation-archive/files/issue-268-validation/installed-StackSeedDomain.h.cpp b/doc/local-validation-archive/files/issue-268-validation/installed-StackSeedDomain.h.cpp new file mode 100644 index 00000000..cf3c73c9 --- /dev/null +++ b/doc/local-validation-archive/files/issue-268-validation/installed-StackSeedDomain.h.cpp @@ -0,0 +1,2 @@ +#include +int main() { return 0; } diff --git a/doc/local-validation-archive/files/issue-268-validation/leftmost-benchmark/candidate-batch128/IntaRNA/SeededPartitionFunction.cpp b/doc/local-validation-archive/files/issue-268-validation/leftmost-benchmark/candidate-batch128/IntaRNA/SeededPartitionFunction.cpp new file mode 100644 index 00000000..e37e851c --- /dev/null +++ b/doc/local-validation-archive/files/issue-268-validation/leftmost-benchmark/candidate-batch128/IntaRNA/SeededPartitionFunction.cpp @@ -0,0 +1,576 @@ +#include "IntaRNA/SeededPartitionFunction.h" +#include "IntaRNA/PartitionArithmetic.h" +#include +#include +#include +#include + +namespace IntaRNA { +namespace { +using Kernel=SeededPartitionFunction; +using Pair=Kernel::Pair; +using A=PartitionArithmetic; +using Support=std::array; +using RightStates=std::array; + +/** Exact left-most-anchor factorization, with bounded boundary publication. */ +Kernel::Result +computeAnchored(const Kernel::Domain & d,const StackSeedDomain & seeds, + const Kernel::Weights & w,bool pairs) +{ + Kernel::Result out; + if(pairs) out.mass=Matrix(d.n,d.m,0); + if(!seeds.maxSeedLength() || !d.span1 || !d.span2) return out; + + // Containing seeds add no paths to the exact union. In the remaining family, + // start order equals end order along each diagonal, including overlapping + // seeds. Thus excluding an earlier seed while extending left is sufficient; + // no earlier seed can first finish beyond the anchor's right end. + const auto anchors=seeds.minimalOccurrences(); + std::unordered_map anchorIndex; + anchorIndex.reserve(anchors.size()); + size_t cap=2; + for(size_t a=0;a left(cells,cap),leftTotal(maxN,maxM),rightTotal(maxN,maxM); + Matrix right(maxN,maxM); + Matrix dLeft,dLeftTotal; + Matrix dRight; + if(pairs) { + dLeft.resize(cells,cap,false); dLeftTotal.resize(maxN,maxM,false); + dRight.resize(maxN,maxM,false); + } + Matrix valid(d.n,d.m); + for(size_t i=0;i hybrid(batchSize,cells),boundary; + if(pairs) boundary.resize(batchSize,cells,false); + std::vector starts; + std::vector initiation; + std::vector candidates; + std::vector included(anchors.size(),0); + starts.reserve(batchSize); initiation.reserve(batchSize); + candidates.reserve(anchors.size()); + auto reset=[](Matrix & matrix,size_t rows,size_t columns) { + if(columns==matrix.size2()) std::fill_n(&matrix(0,0),rows*columns,T{}); + else for(size_t i=0;i & matrix,size_t rows,size_t columns) { + // Left states use a fixed maxM row stride and a contiguous stack axis. + for(size_t i=0;isecond].length; + }; + auto nonstack=[&](size_t i,size_t j,const auto & visit) { + for(size_t u=1;u<=i && u-1<=d.loop1;++u) + for(size_t v=1;v<=j && v-1<=d.loop2;++v) + if(u!=1 || v!=1) visit(i-u,j-v); + }; + auto fill=[&](size_t a) { + const auto anchor=anchors[a]; + s={anchor.i,anchor.j}; length=anchor.length; + t={s.first+length-1,s.second+length-1}; + singleton=d.noLP && length==1; + supportBegin=singleton?0:1; + // Crop both extension rectangles to the batch's possible left boundaries. + // Paths to these sources cannot visit positions beyond their leftmost + // coordinates; their mandatory left offsets also bound the right side. + nL=0; mL=0; + size_t minLeft1=maxN,minLeft2=maxM; + for(const Pair p:starts) { + if(p.first>s.first || p.second>s.second) continue; + const size_t u=s.first-p.first,v=s.second-p.second; + if(u>maxN-length || v>maxM-length) continue; + nL=std::max(nL,u+1); mL=std::max(mL,v+1); + minLeft1=std::min(minLeft1,u); minLeft2=std::min(minLeft2,v); + } + if(!nL || !mL) { seedWeight=0; return; } + nR=std::min(d.n-t.first,maxN-length-minLeft1+1); + mR=std::min(d.m-t.second,maxM-length-minLeft2+1); + resetLeft(left,nL,mL); reset(leftTotal,nL,mL); + reset(right,nR,mR); reset(rightTotal,nR,mR); + // Seed energy consists of its actual stack transitions. Initiation is + // attached later to the complete interaction's left boundary exactly once. + seedWeight=1; + for(size_t k=0;ks.first || p.second>s.second) continue; + const size_t u=s.first-p.first,v=s.second-p.second; + if(u0 || leftTotal(u,v)[1]>0)) { reachable=true; break; } + } + if(!reachable) { seedWeight=0; return; } + right(0,0)[singleton?0:3]=1; + for(size_t i=0;is.first || p.second>s.second) continue; + const size_t u=s.first-p.first,v=s.second-p.second; + if(u>=nL || v>=mL || (leftTotal(u,v)[0]==0 && leftTotal(u,v)[1]==0)) continue; + const Z_type source=A::multiply(initiation[b],seedWeight); + for(size_t i=0;i0;--ii) for(size_t jj=mL;jj>0;--jj) { + const size_t i=ii-1,j=jj-1; const Pair p=leftPair(i,j); + if(!valid(p.first,p.second)) continue; + const size_t forbidden=earlierLength(p); + for(size_t c=d.noLP && !(singleton && i==0 && j==0)?1:0;c0;--ii) for(size_t jj=mR;jj>0;--jj) { + const size_t i=ii-1,j=jj-1; const Pair q=rightPair(i,j); + if(!valid(q.first,q.second)) continue; + for(size_t c=supportBegin*2;c<4;++c) if(right(i,j)[c]==0) dRight(i,j)[c]=0; + if(i || j) for(size_t c=supportBegin*2;c<4;++c) out.mass(q.first,q.second)=A::add( + out.mass(q.first,q.second),A::multiply(right(i,j)[c],dRight(i,j)[c])); + if(i && j) { + const Pair p=rightPair(i-1,j-1); + if(valid(p.first,p.second)) { + const Z_type edge=A::check(w.transition(p,q)); + for(size_t h=supportBegin;h<2;++h) { + const size_t support=singleton && i==1 && j==1?1:h; + if(dRight(i,j)[support*2+1]==0) continue; + const Z_type adj=A::multiply(edge,dRight(i,j)[support*2+1]); + for(size_t c=0;c<2;++c) if(right(i-1,j-1)[h*2+c]>0) + dRight(i-1,j-1)[h*2+c]=A::add(dRight(i-1,j-1)[h*2+c],adj); + } + } + } + nonstack(i,j,[&](size_t u,size_t v) { + const Pair p=rightPair(u,v); if(!valid(p.first,p.second)) return; + if((dRight(i,j)[d.noLP?0:1]==0 && dRight(i,j)[2+(d.noLP?0:1)]==0) + || (rightTotal(u,v)[0]==0 && rightTotal(u,v)[1]==0)) return; + const Z_type edge=A::check(w.transition(p,q)); + for(size_t h=supportBegin;h<2;++h) { + const size_t frontier=singleton && u==0 && v==0?0:1; + auto & adj=dRight(u,v)[h*2+frontier]; + adj=A::add(adj,A::multiply(edge,dRight(i,j)[h*2+(d.noLP?0:1)])); + } + }); + } + }; + + auto processBatch=[&]() { + reset(hybrid,starts.size(),cells); if(pairs) reset(boundary,starts.size(),cells); + candidates.clear(); initiation.clear(); + // Query only anchor starts inside the batch's span boxes, then restore + // deterministic seed order independently of hash-table iteration order. + for(const Pair p:starts) { + initiation.push_back(A::check(w.initiation(p))); + for(size_t i=p.first;isecond]) continue; + const auto & anchor=anchors[found->second]; + if(anchor.length>maxN-(i-p.first) || anchor.length>maxM-(j-p.second)) continue; + included[found->second]=1; candidates.push_back(found->second); + } + } + std::sort(candidates.begin(),candidates.end()); + for(const size_t a:candidates) { fill(a); contract(false); } + for(size_t b=0;b; // pending lonely pair / stack-supported end + Result out; + if (pairs) out.mass=Matrix(d.n,d.m,0); + if (!seeds.maxSeedLength() || !d.span1 || !d.span2) return out; + + // Freeze heuristic choices before either pass. A first hit is owned by the + // shortest admitted seed ending here, including coincident variable lengths. + std::unordered_map selected; + if (w.selectRight) for(size_t i=0;i left(cells,cap),leftTotal(maxN,maxM),hybrid(maxN,maxM); + Matrix hit(maxN,maxM),right(maxN,maxM); + Matrix valid(maxN,maxM); + Matrix ending(maxN,maxM),byEnd; + Matrix boundary,dLeft,dTotal; + Matrix dHit,adjoint; + if(w.selectRight) byEnd.resize(maxN,maxM,false); + if(pairs) { + boundary.resize(maxN,maxM,false); + dLeft.resize(cells,cap,false); dTotal.resize(maxN,maxM,false); + dHit.resize(maxN,maxM,false); adjoint.resize(maxN,maxM,false); + } + auto reset=[](Matrix & matrix,size_t rows,size_t columns) { + // Matrix storage is row-major. Clear contiguous full-width regions in + // one pass; narrower rectangles leave the inactive row tails untouched. + if(columns==matrix.size2()) std::fill_n(&matrix(0,0),rows*columns,T{}); + else for(size_t i=0;i groups; + std::vector anchors; + groups.reserve(w.selectRight?cells+1:1); anchors.reserve(cells); + auto addAnchor=[&](Group & group,Pair q) { + anchors.push_back({q,group.first}); group.first=anchors.size(); + }; + auto forAnchors=[&](const Group & group,const auto & visit) { + for(size_t a=group.first;a;a=anchors[a-1].next) visit(anchors[a-1].pair); + }; + seeds.forEachStart([&](size_t p1,size_t p2) { + const Pair p{p1,p2}; if (!w.valid(p)) return; + const size_t n=std::min(d.span1,d.n-p1),m=std::min(d.span2,d.m-p2); + reset(left,n*m,cap); reset(leftTotal,n,m); reset(hit,n,m); + reset(hybrid,n,m); + if(w.selectRight) reset(byEnd,n,m); + if(pairs) { reset(boundary,n,m); reset(dHit,n,m); } + for(size_t i=0;i0) extendLeft(i,j,std::min(cap,c+2),A::multiply(prefix,t)); + } + } + nonstack(i,j,[&](size_t u,size_t v) { + if(leftTotal(u,v)>0) extendLeft(i,j,1,A::multiply(leftTotal(u,v),transition(i,j,u,v))); + }); + // A nonstack edge may leave only a supported end under noLP. + for(size_t c=d.noLP?1:0;c=n || end.second-p2>=m) continue; + const Pair local{end.first-p1,end.second-p2}; + auto & groupIndex=byEnd(local.first,local.second); + if(!groupIndex) { + groupIndex=groups.size(); groups.push_back({local,false}); + } + addAnchor(groups[groupIndex],{i,j}); + } + auto fillRight=[&](const Group & group) { + reset(right,group.end.first+1,group.end.second+1); + forAnchors(group,[&](Pair q) { right(q.first,q.second)=hit(q.first,q.second); }); + // B is seed-unconstrained: no seed lookup or subtraction occurs here. + for(size_t i=0;i<=group.end.first;++i) for(size_t j=0;j<=group.end.second;++j) { + if(!valid(i,j)) continue; + if(i && j && valid(i-1,j-1)) { + const auto & prev=right(i-1,j-1); + if(prev[0]>0 || prev[1]>0) + right(i,j)[1]=A::add(right(i,j)[1],A::multiply(A::add(prev[0],prev[1]),transition(i,j,i-1,j-1))); + } + nonstack(i,j,[&](size_t u,size_t v) { + if(right(u,v)[1]>0) { + auto & value=right(i,j)[d.noLP?0:1]; + value=A::add(value,A::multiply(right(u,v)[1],transition(i,j,u,v))); + } + }); + } + }; + for(const auto & group:groups) { + if(!group.first) continue; + fillRight(group); + if(group.all) { + for(size_t i=0;i0) { + const Pair q{p1+i,p2+j}; + const Z_type b=A::check(w.boundary(p,q)); + if(pairs) boundary(i,j)=b; + out.z=A::add(out.z,A::multiply(hybrid(i,j),b)); + if(w.complete) w.complete(p,q,hybrid(i,j),b); + } + if(!pairs) return; + + // Reverse the retained right contractions. Recompute one group at a time + // to bound memory independently of the number of heuristic choices. + for(size_t g=0;g0;--ii) for(size_t jj=group.end.second+1;jj>0;--jj) { + const size_t i=ii-1,j=jj-1; if(!valid(i,j)) continue; + auto & a=adjoint(i,j); + if(group.all || (i==group.end.first && j==group.end.second)) a[1]=A::add(a[1],boundary(i,j)); + // A vertex is owned by its most recent pair: T*dT counts every + // right-side pair, including the accepting pair, exactly once. + auto & mass=out.mass(p1+i,p2+j); + for(size_t c=0;c<2;++c) mass=A::add(mass,A::multiply(right(i,j)[c],a[c])); + if(i && j && valid(i-1,j-1)) { + const Z_type value=A::multiply(a[1],transition(i,j,i-1,j-1)); + for(size_t c=0;c<2;++c) adjoint(i-1,j-1)[c]=A::add(adjoint(i-1,j-1)[c],value); + } + nonstack(i,j,[&](size_t u,size_t v) { + adjoint(u,v)[1]=A::add(adjoint(u,v)[1],A::multiply(a[d.noLP?0:1],transition(i,j,u,v))); + }); + } + forAnchors(group,[&](Pair q) { + for(size_t c=0;c<2;++c) + dHit(q.first,q.second)[c]=A::add(dHit(q.first,q.second)[c],adjoint(q.first,q.second)[c]); + }); + } + if(w.selectRight) for(size_t i=0;i0;--ii) for(size_t jj=m;jj>0;--jj) { + const size_t i=ii-1,j=jj-1,q=i*m+j; if(!valid(i,j)) continue; + for(size_t c=d.noLP?1:0;c +#include +#include +#include + +namespace IntaRNA { +namespace { +using Kernel=SeededPartitionFunction; +using Pair=Kernel::Pair; +using A=PartitionArithmetic; +using Support=std::array; +using RightStates=std::array; + +/** Exact left-most-anchor factorization, with bounded boundary publication. */ +Kernel::Result +computeAnchored(const Kernel::Domain & d,const StackSeedDomain & seeds, + const Kernel::Weights & w,bool pairs) +{ + Kernel::Result out; + if(pairs) out.mass=Matrix(d.n,d.m,0); + if(!seeds.maxSeedLength() || !d.span1 || !d.span2) return out; + + // Containing seeds add no paths to the exact union. In the remaining family, + // start order equals end order along each diagonal, including overlapping + // seeds. Thus excluding an earlier seed while extending left is sufficient; + // no earlier seed can first finish beyond the anchor's right end. + const auto anchors=seeds.minimalOccurrences(); + std::unordered_map anchorIndex; + anchorIndex.reserve(anchors.size()); + size_t cap=2; + for(size_t a=0;a left(cells,cap),leftTotal(maxN,maxM),rightTotal(maxN,maxM); + Matrix right(maxN,maxM); + Matrix dLeft,dLeftTotal; + Matrix dRight; + if(pairs) { + dLeft.resize(cells,cap,false); dLeftTotal.resize(maxN,maxM,false); + dRight.resize(maxN,maxM,false); + } + Matrix valid(d.n,d.m); + for(size_t i=0;i hybrid(batchSize,cells),boundary; + if(pairs) boundary.resize(batchSize,cells,false); + std::vector starts; + std::vector initiation; + std::vector candidates; + std::vector included(anchors.size(),0); + starts.reserve(batchSize); initiation.reserve(batchSize); + candidates.reserve(anchors.size()); + auto reset=[](Matrix & matrix,size_t rows,size_t columns) { + if(columns==matrix.size2()) std::fill_n(&matrix(0,0),rows*columns,T{}); + else for(size_t i=0;i & matrix,size_t rows,size_t columns) { + // Left states use a fixed maxM row stride and a contiguous stack axis. + for(size_t i=0;isecond].length; + }; + auto nonstack=[&](size_t i,size_t j,const auto & visit) { + for(size_t u=1;u<=i && u-1<=d.loop1;++u) + for(size_t v=1;v<=j && v-1<=d.loop2;++v) + if(u!=1 || v!=1) visit(i-u,j-v); + }; + auto fill=[&](size_t a) { + const auto anchor=anchors[a]; + s={anchor.i,anchor.j}; length=anchor.length; + t={s.first+length-1,s.second+length-1}; + singleton=d.noLP && length==1; + supportBegin=singleton?0:1; + // Crop both extension rectangles to the batch's possible left boundaries. + // Paths to these sources cannot visit positions beyond their leftmost + // coordinates; their mandatory left offsets also bound the right side. + nL=0; mL=0; + size_t minLeft1=maxN,minLeft2=maxM; + for(const Pair p:starts) { + if(p.first>s.first || p.second>s.second) continue; + const size_t u=s.first-p.first,v=s.second-p.second; + if(u>maxN-length || v>maxM-length) continue; + nL=std::max(nL,u+1); mL=std::max(mL,v+1); + minLeft1=std::min(minLeft1,u); minLeft2=std::min(minLeft2,v); + } + if(!nL || !mL) { seedWeight=0; return; } + nR=std::min(d.n-t.first,maxN-length-minLeft1+1); + mR=std::min(d.m-t.second,maxM-length-minLeft2+1); + resetLeft(left,nL,mL); reset(leftTotal,nL,mL); + reset(right,nR,mR); reset(rightTotal,nR,mR); + // Seed energy consists of its actual stack transitions. Initiation is + // attached later to the complete interaction's left boundary exactly once. + seedWeight=1; + for(size_t k=0;ks.first || p.second>s.second) continue; + const size_t u=s.first-p.first,v=s.second-p.second; + if(u0 || leftTotal(u,v)[1]>0)) { reachable=true; break; } + } + if(!reachable) { seedWeight=0; return; } + right(0,0)[singleton?0:3]=1; + for(size_t i=0;is.first || p.second>s.second) continue; + const size_t u=s.first-p.first,v=s.second-p.second; + if(u>=nL || v>=mL || (leftTotal(u,v)[0]==0 && leftTotal(u,v)[1]==0)) continue; + const Z_type source=A::multiply(initiation[b],seedWeight); + for(size_t i=0;i0;--ii) for(size_t jj=mL;jj>0;--jj) { + const size_t i=ii-1,j=jj-1; const Pair p=leftPair(i,j); + if(!valid(p.first,p.second)) continue; + const size_t forbidden=earlierLength(p); + for(size_t c=d.noLP && !(singleton && i==0 && j==0)?1:0;c0;--ii) for(size_t jj=mR;jj>0;--jj) { + const size_t i=ii-1,j=jj-1; const Pair q=rightPair(i,j); + if(!valid(q.first,q.second)) continue; + for(size_t c=supportBegin*2;c<4;++c) if(right(i,j)[c]==0) dRight(i,j)[c]=0; + if(i || j) for(size_t c=supportBegin*2;c<4;++c) out.mass(q.first,q.second)=A::add( + out.mass(q.first,q.second),A::multiply(right(i,j)[c],dRight(i,j)[c])); + if(i && j) { + const Pair p=rightPair(i-1,j-1); + if(valid(p.first,p.second)) { + const Z_type edge=A::check(w.transition(p,q)); + for(size_t h=supportBegin;h<2;++h) { + const size_t support=singleton && i==1 && j==1?1:h; + if(dRight(i,j)[support*2+1]==0) continue; + const Z_type adj=A::multiply(edge,dRight(i,j)[support*2+1]); + for(size_t c=0;c<2;++c) if(right(i-1,j-1)[h*2+c]>0) + dRight(i-1,j-1)[h*2+c]=A::add(dRight(i-1,j-1)[h*2+c],adj); + } + } + } + nonstack(i,j,[&](size_t u,size_t v) { + const Pair p=rightPair(u,v); if(!valid(p.first,p.second)) return; + if((dRight(i,j)[d.noLP?0:1]==0 && dRight(i,j)[2+(d.noLP?0:1)]==0) + || (rightTotal(u,v)[0]==0 && rightTotal(u,v)[1]==0)) return; + const Z_type edge=A::check(w.transition(p,q)); + for(size_t h=supportBegin;h<2;++h) { + const size_t frontier=singleton && u==0 && v==0?0:1; + auto & adj=dRight(u,v)[h*2+frontier]; + adj=A::add(adj,A::multiply(edge,dRight(i,j)[h*2+(d.noLP?0:1)])); + } + }); + } + }; + + auto processBatch=[&]() { + reset(hybrid,starts.size(),cells); if(pairs) reset(boundary,starts.size(),cells); + candidates.clear(); initiation.clear(); + // Query only anchor starts inside the batch's span boxes, then restore + // deterministic seed order independently of hash-table iteration order. + for(const Pair p:starts) { + initiation.push_back(A::check(w.initiation(p))); + for(size_t i=p.first;isecond]) continue; + const auto & anchor=anchors[found->second]; + if(anchor.length>maxN-(i-p.first) || anchor.length>maxM-(j-p.second)) continue; + included[found->second]=1; candidates.push_back(found->second); + } + } + std::sort(candidates.begin(),candidates.end()); + for(const size_t a:candidates) { fill(a); contract(false); } + for(size_t b=0;b; // pending lonely pair / stack-supported end + Result out; + if (pairs) out.mass=Matrix(d.n,d.m,0); + if (!seeds.maxSeedLength() || !d.span1 || !d.span2) return out; + + // Freeze heuristic choices before either pass. A first hit is owned by the + // shortest admitted seed ending here, including coincident variable lengths. + std::unordered_map selected; + if (w.selectRight) for(size_t i=0;i left(cells,cap),leftTotal(maxN,maxM),hybrid(maxN,maxM); + Matrix hit(maxN,maxM),right(maxN,maxM); + Matrix valid(maxN,maxM); + Matrix ending(maxN,maxM),byEnd; + Matrix boundary,dLeft,dTotal; + Matrix dHit,adjoint; + if(w.selectRight) byEnd.resize(maxN,maxM,false); + if(pairs) { + boundary.resize(maxN,maxM,false); + dLeft.resize(cells,cap,false); dTotal.resize(maxN,maxM,false); + dHit.resize(maxN,maxM,false); adjoint.resize(maxN,maxM,false); + } + auto reset=[](Matrix & matrix,size_t rows,size_t columns) { + // Matrix storage is row-major. Clear contiguous full-width regions in + // one pass; narrower rectangles leave the inactive row tails untouched. + if(columns==matrix.size2()) std::fill_n(&matrix(0,0),rows*columns,T{}); + else for(size_t i=0;i groups; + std::vector anchors; + groups.reserve(w.selectRight?cells+1:1); anchors.reserve(cells); + auto addAnchor=[&](Group & group,Pair q) { + anchors.push_back({q,group.first}); group.first=anchors.size(); + }; + auto forAnchors=[&](const Group & group,const auto & visit) { + for(size_t a=group.first;a;a=anchors[a-1].next) visit(anchors[a-1].pair); + }; + seeds.forEachStart([&](size_t p1,size_t p2) { + const Pair p{p1,p2}; if (!w.valid(p)) return; + const size_t n=std::min(d.span1,d.n-p1),m=std::min(d.span2,d.m-p2); + reset(left,n*m,cap); reset(leftTotal,n,m); reset(hit,n,m); + reset(hybrid,n,m); + if(w.selectRight) reset(byEnd,n,m); + if(pairs) { reset(boundary,n,m); reset(dHit,n,m); } + for(size_t i=0;i0) extendLeft(i,j,std::min(cap,c+2),A::multiply(prefix,t)); + } + } + nonstack(i,j,[&](size_t u,size_t v) { + if(leftTotal(u,v)>0) extendLeft(i,j,1,A::multiply(leftTotal(u,v),transition(i,j,u,v))); + }); + // A nonstack edge may leave only a supported end under noLP. + for(size_t c=d.noLP?1:0;c=n || end.second-p2>=m) continue; + const Pair local{end.first-p1,end.second-p2}; + auto & groupIndex=byEnd(local.first,local.second); + if(!groupIndex) { + groupIndex=groups.size(); groups.push_back({local,false}); + } + addAnchor(groups[groupIndex],{i,j}); + } + auto fillRight=[&](const Group & group) { + reset(right,group.end.first+1,group.end.second+1); + forAnchors(group,[&](Pair q) { right(q.first,q.second)=hit(q.first,q.second); }); + // B is seed-unconstrained: no seed lookup or subtraction occurs here. + for(size_t i=0;i<=group.end.first;++i) for(size_t j=0;j<=group.end.second;++j) { + if(!valid(i,j)) continue; + if(i && j && valid(i-1,j-1)) { + const auto & prev=right(i-1,j-1); + if(prev[0]>0 || prev[1]>0) + right(i,j)[1]=A::add(right(i,j)[1],A::multiply(A::add(prev[0],prev[1]),transition(i,j,i-1,j-1))); + } + nonstack(i,j,[&](size_t u,size_t v) { + if(right(u,v)[1]>0) { + auto & value=right(i,j)[d.noLP?0:1]; + value=A::add(value,A::multiply(right(u,v)[1],transition(i,j,u,v))); + } + }); + } + }; + for(const auto & group:groups) { + if(!group.first) continue; + fillRight(group); + if(group.all) { + for(size_t i=0;i0) { + const Pair q{p1+i,p2+j}; + const Z_type b=A::check(w.boundary(p,q)); + if(pairs) boundary(i,j)=b; + out.z=A::add(out.z,A::multiply(hybrid(i,j),b)); + if(w.complete) w.complete(p,q,hybrid(i,j),b); + } + if(!pairs) return; + + // Reverse the retained right contractions. Recompute one group at a time + // to bound memory independently of the number of heuristic choices. + for(size_t g=0;g0;--ii) for(size_t jj=group.end.second+1;jj>0;--jj) { + const size_t i=ii-1,j=jj-1; if(!valid(i,j)) continue; + auto & a=adjoint(i,j); + if(group.all || (i==group.end.first && j==group.end.second)) a[1]=A::add(a[1],boundary(i,j)); + // A vertex is owned by its most recent pair: T*dT counts every + // right-side pair, including the accepting pair, exactly once. + auto & mass=out.mass(p1+i,p2+j); + for(size_t c=0;c<2;++c) mass=A::add(mass,A::multiply(right(i,j)[c],a[c])); + if(i && j && valid(i-1,j-1)) { + const Z_type value=A::multiply(a[1],transition(i,j,i-1,j-1)); + for(size_t c=0;c<2;++c) adjoint(i-1,j-1)[c]=A::add(adjoint(i-1,j-1)[c],value); + } + nonstack(i,j,[&](size_t u,size_t v) { + adjoint(u,v)[1]=A::add(adjoint(u,v)[1],A::multiply(a[d.noLP?0:1],transition(i,j,u,v))); + }); + } + forAnchors(group,[&](Pair q) { + for(size_t c=0;c<2;++c) + dHit(q.first,q.second)[c]=A::add(dHit(q.first,q.second)[c],adjoint(q.first,q.second)[c]); + }); + } + if(w.selectRight) for(size_t i=0;i0;--ii) for(size_t jj=m;jj>0;--jj) { + const size_t i=ii-1,j=jj-1,q=i*m+j; if(!valid(i,j)) continue; + for(size_t c=d.noLP?1:0;c\nint main() {{ return 0; }}\n') + subprocess.run([str(cxx),'-std=c++23',f'-I{prefix}/include',f'-I{deps}/include','-fsyntax-only',str(probe)],check=True) + print('Standalone installed header:',name,flush=True) +env=dict(os.environ,PKG_CONFIG_PATH=str(prefix/'lib/pkgconfig')) +flags=shlex.split(subprocess.check_output(['pkg-config',f'--define-variable=prefix={prefix}','--cflags','--libs','IntaRNA'],env=env,text=True)) +consumer=root/'installed-probabilities' +subprocess.run([str(cxx),'-std=c++23',str(repo/'doc/seeded-probabilities-example.cpp'),f'-Wl,-rpath,{prefix}/lib',f'-Wl,-rpath,{deps}/lib',*flags,'-o',str(consumer)],check=True) +result=subprocess.check_output([str(consumer)]) +assert b'bpProb;' in result +(root/'installed-probabilities.csv').write_bytes(result) +print('Installed pkg-config probability consumer passed',flush=True) diff --git a/doc/local-validation-archive/files/issue-270-anchored-validation/compare-native.py b/doc/local-validation-archive/files/issue-270-anchored-validation/compare-native.py new file mode 100644 index 00000000..1340202f --- /dev/null +++ b/doc/local-validation-archive/files/issue-270-anchored-validation/compare-native.py @@ -0,0 +1,44 @@ +from pathlib import Path +import json,os,subprocess +root=Path(__file__).resolve().parent +out=root/'native-comparison';out.mkdir(exist_ok=True) +commands=[] +for original in json.loads((root.parent/'issue-268-validation/native-comparison/commands.json').read_text()): + variant='base' if '/baseline/' in original[0] else 'candidate' + binary=root.parent/'issue-268-validation/release/src/bin/IntaRNA' if variant=='base' else root/'release/src/bin/IntaRNA' + command=[str(binary),*[x.replace('issue-268-validation/native-comparison','issue-270-anchored-validation/native-comparison') for x in original[1:]]] + commands.append(command) + name=next(Path(x.split(':',1)[1]).stem for x in command if x.startswith('--out=bpProb:')) + result=subprocess.run(command,capture_output=True,check=True,env=dict(os.environ,OMP_NUM_THREADS='1')) + (out/(name+'.stdout')).write_bytes(result.stdout) + (out/(name+'.stderr')).write_bytes(result.stderr) + print(name,flush=True) +(out/'commands.json').write_text(json.dumps(commands,indent=2)+'\n') +from pathlib import Path +import csv, json, math +root=Path(__file__).resolve().parent/'native-comparison' +checks=[] +for before in sorted(root.glob('*-base.csv')): + after=before.with_name(before.name.replace('-base.csv','-candidate.csv')) + old=list(csv.reader(before.open(), delimiter=';')) + new=list(csv.reader(after.open(), delimiter=';')) + assert len(old)==len(new),(before,'row count') + maximum=0.; count=0 + for a,b in zip(old,new): + assert len(a)==len(b),(before,'column count') + for x,y in zip(a,b): + try: u,v=float(x),float(y) + except ValueError: + assert x==y,(before,x,y) + continue + assert math.isfinite(u) and math.isfinite(v),(before,x,y) + relative=abs(u-v)/max(abs(u),abs(v)) if u or v else 0. + maximum=max(maximum,relative);count+=1 + assert relative<2e-12,(before,x,y,relative) + assert before.with_suffix('.stdout').read_bytes()==after.with_suffix('.stdout').read_bytes(),(before,'native predictions differ') + assert before.with_suffix('.stderr').read_bytes()==after.with_suffix('.stderr').read_bytes(),(before,'diagnostics differ') + checks.append(dict(case=before.stem.removesuffix('-base'),probabilities=count, + max_relative_error=maximum,byte_identical=before.read_bytes()==after.read_bytes())) +assert len(checks)==12, len(checks) +(root/'results.json').write_text(json.dumps(checks,indent=2)+'\n') +print(json.dumps(checks,indent=2)) diff --git a/doc/local-validation-archive/files/issue-270-anchored-validation/header-probe.cpp b/doc/local-validation-archive/files/issue-270-anchored-validation/header-probe.cpp new file mode 100644 index 00000000..e9ddbedb --- /dev/null +++ b/doc/local-validation-archive/files/issue-270-anchored-validation/header-probe.cpp @@ -0,0 +1,2 @@ +#include +int main() { return 0; } diff --git a/doc/local-validation-archive/files/issue-270-anchored-validation/record-evidence.py b/doc/local-validation-archive/files/issue-270-anchored-validation/record-evidence.py new file mode 100644 index 00000000..fba0e8ec --- /dev/null +++ b/doc/local-validation-archive/files/issue-270-anchored-validation/record-evidence.py @@ -0,0 +1,34 @@ +from pathlib import Path +import hashlib,json,shutil,subprocess +root=Path(__file__).resolve().parent +repo=root.parent/'pr-worktrees/issue-270-anchored' +dest=repo/'doc/benchmarks/seed-extension' +def digest(p):return hashlib.sha256(p.read_bytes()).hexdigest() +def read(p):return json.loads(p.read_text()) +for label,folder in [('native','native-runtime'),('pairs','native-pairs'),('exact','kernel-exact'),('heuristic','kernel-heuristic')]: + for name in ('results.json','samples.tsv'): + shutil.copy2(root/folder/name,dest/f'anchored-{label}-{name}') +source_names=subprocess.check_output(['git','diff','--name-only','--','src','tests'],cwd=repo,text=True).splitlines() +flags={} +for variant,folder in [('base',root.parent/'issue-268-validation/release'),('candidate',root/'release')]: + flags[variant]={} + for line in (folder/'src/IntaRNA/Makefile').read_text().splitlines(): + if any(line.startswith(key+' =') for key in ('AM_CXXFLAGS','CPPFLAGS','CXX','CXXFLAGS','LDFLAGS')): + key,value=line.split(' =',1);flags[variant][key]=value.strip() + flags[variant]['configure']=subprocess.check_output([str(folder/'config.status'),'--config'],cwd=folder,text=True).strip() +metadata={ + 'baseline':'3866c62d10846fab1adc38b937d0c9b96cc06e51', + 'date':'2026-10-16', + 'candidate_source_sha256':{name:digest(repo/name) for name in source_names}, + 'native_build_flags':flags, + 'source_verification':read(root/'source-verification.json'), + 'kernel_build':read(root/'kernel/manifest.json'), + 'kernel_binary_sha256':{v:digest(root/'kernel'/v/'benchmark') for v in ('base','candidate')}, + 'kernel_dump_sha256':{str(p.relative_to(root)):digest(p) for mode in ('exact','heuristic') for p in sorted((root/f'kernel-{mode}-verify').glob('*.txt')) if p.name!='last-rss.txt'}, + 'native_manifests':{label:read(root/folder/'manifest.json') for label,folder in [('ordinary','native-runtime'),('pair_output','native-pairs')]}, + 'native_verification':{'commands':read(root/'native-comparison/commands.json'),'checks':read(root/'native-comparison/results.json')}, + 'validation':{'release':'make tests -j2: 7/7 drivers; 115 API cases; 457827 assertions','debug':'make tests -j2: 7/7 drivers; 115 API cases; 457828 assertions','kernel_exact':read(root/'kernel-exact-verify/verification.json'),'kernel_heuristic':read(root/'kernel-heuristic-verify/verification.json')}, + 'measurement_conditions':{'repetitions':5,'threads':1,'ordering':'alternating baseline/candidate fresh processes','concurrent_project_build_test_install':False,'other_workstation_activity_cpu_affinity_frequency':'uncontrolled','platform_validation':'Linux GCC14.4; macOS/Apple Clang not run'}, +} +(dest/'anchored-source-hashes.json').write_text(json.dumps(metadata,indent=2)+'\n') +print('Recorded benchmark summaries, samples and provenance') diff --git a/doc/local-validation-archive/files/issue-270-validation/compare-native.py b/doc/local-validation-archive/files/issue-270-validation/compare-native.py new file mode 100644 index 00000000..2de79dbf --- /dev/null +++ b/doc/local-validation-archive/files/issue-270-validation/compare-native.py @@ -0,0 +1,25 @@ +from pathlib import Path +import json,os,subprocess +root=Path(__file__).resolve().parent +out=root/'native-comparison';out.mkdir(exist_ok=True) +commands=[] +for original in json.loads((root.parent/'issue-268-validation/native-comparison/commands.json').read_text()): + variant='base' if '/baseline/' in original[0] else 'candidate' + binary=root.parent/'issue-268-validation/release/src/bin/IntaRNA' if variant=='base' else root/'release/src/bin/IntaRNA' + command=[str(binary),*[x.replace('issue-268-validation/native-comparison','issue-270-validation/native-comparison') for x in original[1:]]] + commands.append(command) + name=next(Path(x.split(':',1)[1]).stem for x in command if x.startswith('--out=bpProb:')) + result=subprocess.run(command,capture_output=True,check=True,env=dict(os.environ,OMP_NUM_THREADS='1')) + (out/(name+'.stdout')).write_bytes(result.stdout) + (out/(name+'.stderr')).write_bytes(result.stderr) + print(name,flush=True) +(out/'commands.json').write_text(json.dumps(commands,indent=2)+'\n') +checks=[] +for base in sorted(out.glob('*-base.csv')): + candidate=base.with_name(base.name.replace('-base','-candidate')) + for suffix in ['.csv','.stdout','.stderr']: + assert base.with_suffix(suffix).read_bytes()==candidate.with_suffix(suffix).read_bytes(),(base,suffix) + checks.append(base.name) +assert len(checks)==12 +(out/'results.json').write_text(json.dumps({'byte_identical_cases':checks},indent=2)+'\n') +print('All 12 native prediction/pair files identical') diff --git a/doc/local-validation-archive/files/issue-270-validation/kernel/candidate/IntaRNA/SeededPartitionFunction.cpp b/doc/local-validation-archive/files/issue-270-validation/kernel/candidate/IntaRNA/SeededPartitionFunction.cpp new file mode 100644 index 00000000..0a0eeb6a --- /dev/null +++ b/doc/local-validation-archive/files/issue-270-validation/kernel/candidate/IntaRNA/SeededPartitionFunction.cpp @@ -0,0 +1,605 @@ +#include "IntaRNA/SeededPartitionFunction.h" +#include "IntaRNA/PartitionArithmetic.h" +#include +#include +#include +#include + +namespace IntaRNA { +namespace { +using Kernel=SeededPartitionFunction; +using Pair=Kernel::Pair; +using A=PartitionArithmetic; +using Support=std::array; +using RightStates=std::array; + +/** Exact left-most-anchor factorization, with bounded boundary publication. */ +Kernel::Result +computeAnchored(const Kernel::Domain & d,const StackSeedDomain & seeds, + const Kernel::Weights & w,bool pairs) +{ + Kernel::Result out; + if(pairs) out.mass=Matrix(d.n,d.m,0); + if(!d.n || !d.m || !seeds.maxSeedLength() || !d.span1 || !d.span2) return out; + + // Containing seeds add no paths to the exact union. In the remaining family, + // start order equals end order along each diagonal, including overlapping + // seeds. Thus excluding an earlier seed while extending left is sufficient; + // no earlier seed can first finish beyond the anchor's right end. + const auto anchors=seeds.minimalOccurrences(); + std::unordered_map anchorIndex; + anchorIndex.reserve(anchors.size()); + size_t cap=2; + for(size_t a=0;a left(cells,cap),leftTotal(maxN,maxM),rightTotal(maxN,maxM); + Matrix right(maxN,maxM); + Matrix dLeft,dLeftTotal; + Matrix dRight; + if(pairs) { + dLeft.resize(cells,cap,false); dLeftTotal.resize(maxN,maxM,false); + dRight.resize(maxN,maxM,false); + } + Matrix valid(d.n,d.m); + for(size_t i=0;i hybrid(batchSize,cells),boundary; + if(pairs) boundary.resize(batchSize,cells,false); + std::vector starts; + std::vector initiation; + std::vector candidates; + std::vector included(anchors.size(),0); + starts.reserve(batchSize); initiation.reserve(batchSize); + candidates.reserve(anchors.size()); + auto reset=[](Matrix & matrix,size_t rows,size_t columns) { + if(columns==matrix.size2()) std::fill_n(&matrix(0,0),rows*columns,T{}); + else for(size_t i=0;i & matrix,size_t rows,size_t columns) { + // Left states use a fixed maxM row stride and a contiguous stack axis. + for(size_t i=0;isecond].length; + }; + auto nonstack=[&](size_t i,size_t j,const auto & visit) { + for(size_t u=1;u<=i && u-1<=d.loop1;++u) + for(size_t v=1;v<=j && v-1<=d.loop2;++v) + if(u!=1 || v!=1) visit(i-u,j-v); + }; + auto fill=[&](size_t a) { + const auto anchor=anchors[a]; + s={anchor.i,anchor.j}; length=anchor.length; + t={s.first+length-1,s.second+length-1}; + singleton=d.noLP && length==1; + supportBegin=singleton?0:1; + // Crop both extension rectangles to the batch's possible left boundaries. + // Paths to these sources cannot visit positions beyond their leftmost + // coordinates; their mandatory left offsets also bound the right side. + nL=0; mL=0; + size_t minLeft1=maxN,minLeft2=maxM; + for(const Pair p:starts) { + if(p.first>s.first || p.second>s.second) continue; + const size_t u=s.first-p.first,v=s.second-p.second; + if(u>maxN-length || v>maxM-length) continue; + nL=std::max(nL,u+1); mL=std::max(mL,v+1); + minLeft1=std::min(minLeft1,u); minLeft2=std::min(minLeft2,v); + } + if(!nL || !mL) { seedWeight=0; return; } + nR=std::min(d.n-t.first,maxN-length-minLeft1+1); + mR=std::min(d.m-t.second,maxM-length-minLeft2+1); + resetLeft(left,nL,mL); reset(leftTotal,nL,mL); + reset(right,nR,mR); reset(rightTotal,nR,mR); + // Seed energy consists of its actual stack transitions. Initiation is + // attached later to the complete interaction's left boundary exactly once. + seedWeight=1; + for(size_t k=0;ks.first || p.second>s.second) continue; + const size_t u=s.first-p.first,v=s.second-p.second; + if(u0 || leftTotal(u,v)[1]>0)) { reachable=true; break; } + } + if(!reachable) { seedWeight=0; return; } + right(0,0)[singleton?0:3]=1; + for(size_t i=0;is.first || p.second>s.second) continue; + const size_t u=s.first-p.first,v=s.second-p.second; + if(u>=nL || v>=mL || (leftTotal(u,v)[0]==0 && leftTotal(u,v)[1]==0)) continue; + const Z_type source=A::multiply(initiation[b],seedWeight); + Support sourceLeft{}; + for(size_t h=supportBegin;h<2;++h) sourceLeft[h]=A::multiply(source,leftTotal(u,v)[h]); + for(size_t i=0;i0;--ii) for(size_t jj=mL;jj>0;--jj) { + const size_t i=ii-1,j=jj-1; const Pair p=leftPair(i,j); + if(!valid(p.first,p.second)) continue; + const size_t forbidden=earlierLength(p); + for(size_t c=d.noLP && !(singleton && i==0 && j==0)?1:0;c0;--ii) for(size_t jj=mR;jj>0;--jj) { + const size_t i=ii-1,j=jj-1; const Pair q=rightPair(i,j); + if(!valid(q.first,q.second)) continue; + for(size_t c=supportBegin*2;c<4;++c) if(right(i,j)[c]==0) dRight(i,j)[c]=0; + if(i || j) for(size_t c=supportBegin*2;c<4;++c) out.mass(q.first,q.second)=A::add( + out.mass(q.first,q.second),A::multiply(right(i,j)[c],dRight(i,j)[c])); + if(i && j) { + const Pair p=rightPair(i-1,j-1); + if(valid(p.first,p.second)) { + const Z_type edge=A::check(w.transition(p,q)); + for(size_t h=supportBegin;h<2;++h) { + const size_t support=singleton && i==1 && j==1?1:h; + if(dRight(i,j)[support*2+1]==0) continue; + const Z_type adj=A::multiply(edge,dRight(i,j)[support*2+1]); + for(size_t c=0;c<2;++c) if(right(i-1,j-1)[h*2+c]>0) + dRight(i-1,j-1)[h*2+c]=A::add(dRight(i-1,j-1)[h*2+c],adj); + } + } + } + nonstack(i,j,[&](size_t u,size_t v) { + const Pair p=rightPair(u,v); if(!valid(p.first,p.second)) return; + if((dRight(i,j)[d.noLP?0:1]==0 && dRight(i,j)[2+(d.noLP?0:1)]==0) + || (rightTotal(u,v)[0]==0 && rightTotal(u,v)[1]==0)) return; + const Z_type edge=A::check(w.transition(p,q)); + for(size_t h=supportBegin;h<2;++h) { + const size_t frontier=singleton && u==0 && v==0?0:1; + auto & adj=dRight(u,v)[h*2+frontier]; + adj=A::add(adj,A::multiply(edge,dRight(i,j)[h*2+(d.noLP?0:1)])); + } + }); + } + }; + + auto processBatch=[&]() { + reset(hybrid,starts.size(),cells); if(pairs) reset(boundary,starts.size(),cells); + candidates.clear(); initiation.clear(); + // Query only anchor starts inside the batch's span boxes, then restore + // deterministic seed order independently of hash-table iteration order. + for(const Pair p:starts) { + initiation.push_back(A::check(w.initiation(p))); + for(size_t i=p.first;isecond]) continue; + const auto & anchor=anchors[found->second]; + if(anchor.length>maxN-(i-p.first) || anchor.length>maxM-(j-p.second)) continue; + included[found->second]=1; candidates.push_back(found->second); + } + } + std::sort(candidates.begin(),candidates.end()); + for(const size_t a:candidates) { fill(a); contract(false); } + for(size_t b=0;b; // pending lonely pair / stack-supported end + Result out; + if (pairs) out.mass=Matrix(d.n,d.m,0); + if (!seeds.maxSeedLength() || !d.span1 || !d.span2) return out; + + // Freeze heuristic choices before either pass. A first hit is owned by the + // shortest admitted seed ending here, including coincident variable lengths. + std::unordered_map selected; + if (w.selectRight) for(size_t i=0;i left(cells,cap),leftTotal(maxN,maxM),hybrid(maxN,maxM); + Matrix hit(maxN,maxM),right(maxN,maxM); + Matrix valid(maxN,maxM); + Matrix ending(maxN,maxM),byEnd; + Matrix boundary,dLeft,dTotal; + Matrix dHit,adjoint; + if(w.selectRight) byEnd.resize(maxN,maxM,false); + if(pairs) { + boundary.resize(maxN,maxM,false); + dLeft.resize(cells,cap,false); dTotal.resize(maxN,maxM,false); + dHit.resize(maxN,maxM,false); adjoint.resize(maxN,maxM,false); + } + auto reset=[](Matrix & matrix,size_t rows,size_t columns) { + // Matrix storage is row-major. Clear contiguous full-width regions in + // one pass; narrower rectangles leave the inactive row tails untouched. + if(columns==matrix.size2()) std::fill_n(&matrix(0,0),rows*columns,T{}); + else for(size_t i=0;i groups; + std::vector anchors; + groups.reserve(w.selectRight?cells+1:1); anchors.reserve(cells); + auto addAnchor=[&](Group & group,Pair q) { + if(!group.first) group.begin=q; + anchors.push_back({q,group.first}); group.first=anchors.size(); + group.begin.first=std::min(group.begin.first,q.first); + group.begin.second=std::min(group.begin.second,q.second); + }; + auto forAnchors=[&](const Group & group,const auto & visit) { + for(size_t a=group.first;a;a=anchors[a-1].next) visit(anchors[a-1].pair); + }; + seeds.forEachStart([&](size_t p1,size_t p2) { + const Pair p{p1,p2}; if (!w.valid(p)) return; + const size_t n=std::min(d.span1,d.n-p1),m=std::min(d.span2,d.m-p2); + reset(left,n*m,cap); reset(leftTotal,n,m); reset(hit,n,m); + reset(hybrid,n,m); + if(w.selectRight) reset(byEnd,n,m); + if(pairs) { reset(boundary,n,m); reset(dHit,n,m); } + for(size_t i=0;i0) extendLeft(i,j,std::min(cap,c+2),A::multiply(prefix,t)); + } + } + nonstack(i,j,[&](size_t u,size_t v) { + if(leftTotal(u,v)>0) extendLeft(i,j,1,A::multiply(leftTotal(u,v),transition(i,j,u,v))); + }); + // A nonstack edge may leave only a supported end under noLP. + for(size_t c=d.noLP?1:0;c=n || end.second-p2>=m) continue; + const Pair local{end.first-p1,end.second-p2}; + auto & groupIndex=byEnd(local.first,local.second); + if(!groupIndex) { + groupIndex=groups.size(); groups.push_back({local,false}); + } + addAnchor(groups[groupIndex],{i,j}); + } + // A group has no paths above or left of all its injected anchors. Keep + // both recurrences inside that rectangle, including predecessor lookups, + // so reused storage outside it cannot leak into a later group. + auto resetGroup=[&](Matrix & matrix,const Group & group) { + for(size_t i=group.begin.first;i<=group.end.first;++i) + std::fill_n(&matrix(i,group.begin.second),group.end.second-group.begin.second+1,Ends{}); + }; + auto nonstackRight=[&](const Group & group,size_t i,size_t j,const auto & visit) { + for(size_t u=1;u<=i-group.begin.first && u-1<=d.loop1;++u) + for(size_t v=1;v<=j-group.begin.second && v-1<=d.loop2;++v) + if((u!=1 || v!=1) && valid(i-u,j-v)) visit(i-u,j-v); + }; + auto fillRight=[&](const Group & group) { + resetGroup(right,group); + forAnchors(group,[&](Pair q) { right(q.first,q.second)=hit(q.first,q.second); }); + // B is seed-unconstrained: no seed lookup or subtraction occurs here. + for(size_t i=group.begin.first;i<=group.end.first;++i) + for(size_t j=group.begin.second;j<=group.end.second;++j) { + if(!valid(i,j)) continue; + if(i>group.begin.first && j>group.begin.second && valid(i-1,j-1)) { + const auto & prev=right(i-1,j-1); + if(prev[0]>0 || prev[1]>0) + right(i,j)[1]=A::add(right(i,j)[1],A::multiply(A::add(prev[0],prev[1]),transition(i,j,i-1,j-1))); + } + nonstackRight(group,i,j,[&](size_t u,size_t v) { + if(right(u,v)[1]>0) { + auto & value=right(i,j)[d.noLP?0:1]; + value=A::add(value,A::multiply(right(u,v)[1],transition(i,j,u,v))); + } + }); + } + }; + for(const auto & group:groups) { + if(!group.first) continue; + fillRight(group); + if(group.all) { + for(size_t i=group.begin.first;i0) { + const Pair q{p1+i,p2+j}; + const Z_type b=A::check(w.boundary(p,q)); + if(pairs) boundary(i,j)=b; + out.z=A::add(out.z,A::multiply(hybrid(i,j),b)); + if(w.complete) w.complete(p,q,hybrid(i,j),b); + } + if(!pairs) return; + + // Reverse the retained right contractions. Recompute one group at a time + // to bound memory independently of the number of heuristic choices. + for(size_t g=0;ggroup.begin.first;--ii) + for(size_t jj=group.end.second+1;jj>group.begin.second;--jj) { + const size_t i=ii-1,j=jj-1; if(!valid(i,j)) continue; + if(right(i,j)[0]==0 && right(i,j)[1]==0) continue; + auto & a=adjoint(i,j); + if(group.all || (i==group.end.first && j==group.end.second)) a[1]=A::add(a[1],boundary(i,j)); + // Nonnegative forward zeros have no contributing ancestors. Only + // differentiate states reached by an injection in this group. + for(size_t c=0;c<2;++c) if(right(i,j)[c]==0) a[c]=0; + // A vertex is owned by its most recent pair: T*dT counts every + // right-side pair, including the accepting pair, exactly once. + auto & mass=out.mass(p1+i,p2+j); + for(size_t c=0;c<2;++c) mass=A::add(mass,A::multiply(right(i,j)[c],a[c])); + if(a[1]>0 && i>group.begin.first && j>group.begin.second && valid(i-1,j-1) + && (right(i-1,j-1)[0]>0 || right(i-1,j-1)[1]>0)) { + const Z_type value=A::multiply(a[1],transition(i,j,i-1,j-1)); + for(size_t c=0;c<2;++c) if(right(i-1,j-1)[c]>0) + adjoint(i-1,j-1)[c]=A::add(adjoint(i-1,j-1)[c],value); + } + if(a[d.noLP?0:1]>0) nonstackRight(group,i,j,[&](size_t u,size_t v) { + if(right(u,v)[1]>0) + adjoint(u,v)[1]=A::add(adjoint(u,v)[1],A::multiply(a[d.noLP?0:1],transition(i,j,u,v))); + }); + } + forAnchors(group,[&](Pair q) { + for(size_t c=0;c<2;++c) + dHit(q.first,q.second)[c]=A::add(dHit(q.first,q.second)[c],adjoint(q.first,q.second)[c]); + }); + } + if(w.selectRight) for(size_t i=0;i0;--ii) for(size_t jj=m;jj>0;--jj) { + const size_t i=ii-1,j=jj-1,q=i*m+j; if(!valid(i,j)) continue; + for(size_t c=d.noLP?1:0;c(),n,m,span1,span2); + // Freeze heuristic choices before computing the shared forward/outside objective. ++ const auto profileBegin=std::chrono::steady_clock::now(); + prepareStackSeedExtensions(n,m,seeds); ++ const auto profilePrepared=std::chrono::steady_clock::now(); + // Supply native weights once per initiation, transition and complete boundary. + using K=SeededPartitionFunction; + const K::Domain domain{n,m,span1,span2,energy.getMaxInternalLoopSize1(),energy.getMaxInternalLoopSize2(),output.getOutputConstraint().noLP}; +@@ -377,11 +381,16 @@ + stackSeedRightSelector() + }; + const auto result=K::compute(domain,seeds,weights,pairProbabilities!=nullptr); ++ const auto profileComputed=std::chrono::steady_clock::now(); + // Custom updateZ overrides can change the objective even in release builds. + PartitionArithmetic::check(Zall); + if (result.z!=Zall) throw std::logic_error("exact seeded updateZ override did not preserve the partition objective"); + output.setExactPartition(true); + reportOptima(); ++ const auto profileReported=std::chrono::steady_clock::now(); ++ std::cerr << "#profile_predictor prepare_s=" << std::chrono::duration(profilePrepared-profileBegin).count() ++ << " compute_s=" << std::chrono::duration(profileComputed-profilePrepared).count() ++ << " report_s=" << std::chrono::duration(profileReported-profileComputed).count() << '\n'; + if (pairProbabilities) { + // Restore original strand orientation and offsets before merging this region. + const auto first=energy.getBasePair(0,0), last=energy.getBasePair(n-1,m-1); +--- a/src/IntaRNA/SeededPartitionFunction.cpp ++++ b/src/IntaRNA/SeededPartitionFunction.cpp +@@ -1,3 +1,6 @@ ++#include ++#include ++#include + #include "IntaRNA/SeededPartitionFunction.h" + #include "IntaRNA/PartitionArithmetic.h" + #include +@@ -408,9 +411,13 @@ + auto forAnchors=[&](const Group & group,const auto & visit) { + for(size_t a=group.first;a;a=anchors[a-1].next) visit(anchors[a-1].pair); + }; ++ uint64_t profileSources=0,profileLeftCells=0,profileGroups=0,profileAllGroups=0,profileRightCells=0,profileCroppedCells=0,profileAnchors=0; ++ double profileLeftSeconds=0,profileRightSeconds=0; + seeds.forEachStart([&](size_t p1,size_t p2) { + const Pair p{p1,p2}; if (!w.valid(p)) return; + const size_t n=std::min(d.span1,d.n-p1),m=std::min(d.span2,d.m-p2); ++ ++profileSources; profileLeftCells+=n*m; ++ const auto profileLeftBegin=std::chrono::steady_clock::now(); + reset(left,n*m,cap); reset(leftTotal,n,m); reset(hit,n,m); + reset(hybrid,n,m); + if(w.selectRight) reset(byEnd,n,m); +@@ -452,6 +459,7 @@ + for(size_t c=d.noLP?1:0;c(std::chrono::steady_clock::now()-profileLeftBegin).count(); + // Retained right extensions share first-hit injections by linearity: + // T[p,r] = sum_q F[p,q] B[q,r]. Injections share work only when they + // have the same retained endpoint (or retain all endpoints). +@@ -477,6 +485,12 @@ + addAnchor(groups[groupIndex],{i,j}); + } + auto fillRight=[&](const Group & group) { ++ const auto profileRightBegin=std::chrono::steady_clock::now(); ++ ++profileGroups; if(group.all) ++profileAllGroups; ++ size_t profileMin1=group.end.first,profileMin2=group.end.second; ++ forAnchors(group,[&](Pair q) { ++profileAnchors; profileMin1=std::min(profileMin1,q.first); profileMin2=std::min(profileMin2,q.second); }); ++ profileRightCells+=(group.end.first+1)*(group.end.second+1); ++ profileCroppedCells+=(group.end.first-profileMin1+1)*(group.end.second-profileMin2+1); + reset(right,group.end.first+1,group.end.second+1); + forAnchors(group,[&](Pair q) { right(q.first,q.second)=hit(q.first,q.second); }); + // B is seed-unconstrained: no seed lookup or subtraction occurs here. +@@ -494,6 +508,7 @@ + } + }); + } ++ profileRightSeconds+=std::chrono::duration(std::chrono::steady_clock::now()-profileRightBegin).count(); + }; + for(const auto & group:groups) { + if(!group.first) continue; +@@ -576,6 +591,11 @@ + }); + } + }); ++ std::cerr << "#profile_kernel sources=" << profileSources << " left_cells=" << profileLeftCells ++ << " fillRight_calls=" << profileGroups << " unrestricted_groups=" << profileAllGroups ++ << " right_rectangle_cells=" << profileRightCells << " minima_cropped_cells=" << profileCroppedCells ++ << " firsthit_anchors=" << profileAnchors << " prefix_s=" << profileLeftSeconds ++ << " fillRight_s=" << profileRightSeconds << '\n'; + return out; + } + } diff --git a/doc/local-validation-archive/files/issue-270-validation/profile-analysis/first-probe/run-profile.py b/doc/local-validation-archive/files/issue-270-validation/profile-analysis/first-probe/run-profile.py new file mode 100644 index 00000000..e31364d5 --- /dev/null +++ b/doc/local-validation-archive/files/issue-270-validation/profile-analysis/first-probe/run-profile.py @@ -0,0 +1,33 @@ +from pathlib import Path +import subprocess +import shlex +import json +import hashlib +import time +import argparse +parser=argparse.ArgumentParser(description='Run one seeded M/H attribution sample on the OxyS/fhlA issue reproduction.') +parser.add_argument('--binary',type=Path,required=True) +parser.add_argument('--output',type=Path,required=True) +parser.add_argument('--reference',type=Path,help='optional directory with M-seed.stdout and H-seed.stdout') +args=parser.parse_args() +out=args.output.resolve() +binary=args.binary.resolve() +out.mkdir(parents=True,exist_ok=True) +query='GAAACGGAGCGGCACCUCUUUUAACCCUUGAAGUCACUGCCCGUUUCGAGAGUUUCUCAACUCGAAUAACUAAAGCCAACGUGAACUUUUGCGGAUCUCCAGGAUCCGC' +target='AGUUAGUCAAUGACCUUUUGCACCGCUUUGCGGUGCUUUCCUGGAACAACAAAAUGUCAUAUACACCGAUGAGUGAUCUCGGACAACAAGGGUUGUUCGACAUCACUCGGACA' +commands=[] +for mode in ['M','H']: + command=[str(binary),'--qId=OxyS','--query='+query,'--tId=fhlA','--target='+target,'--model=P','--mode='+mode,'--threads=1'] + commands.append(shlex.join(command)) + before=time.perf_counter() + with (out/(mode+'-seed.stdout')).open('w') as stdout, (out/(mode+'-seed.stderr')).open('w') as stderr: + subprocess.run(['/usr/bin/time','-v','-o',str(out/(mode+'-seed.time')),*command],stdout=stdout,stderr=stderr,check=True) + print(mode, time.perf_counter()-before, flush=True) + if args.reference: + reference=(args.reference/(mode+'-seed.stdout')).read_bytes() + actual=(out/(mode+'-seed.stdout')).read_bytes() + print(mode,'stdout_exact_match',actual==reference,flush=True) + if actual!=reference: + raise RuntimeError(mode+' instrumented output differs from reference') +(out/'commands.txt').write_text('\n'.join(commands)+'\n') +(out/'binary-sha256.txt').write_text(hashlib.sha256(binary.read_bytes()).hexdigest()+' '+str(binary)+'\n') diff --git a/doc/local-validation-archive/files/issue-270-validation/profile-analysis/instrumentation.patch b/doc/local-validation-archive/files/issue-270-validation/profile-analysis/instrumentation.patch new file mode 100644 index 00000000..cd5df5d9 --- /dev/null +++ b/doc/local-validation-archive/files/issue-270-validation/profile-analysis/instrumentation.patch @@ -0,0 +1,121 @@ +--- a/src/IntaRNA/PredictorMfeEns2dSeedExtension.cpp ++++ b/src/IntaRNA/PredictorMfeEns2dSeedExtension.cpp +@@ -1,3 +1,5 @@ ++#include ++#include + + #include "IntaRNA/PredictorMfeEns2dSeedExtension.h" + #include "IntaRNA/SeededPartitionFunction.h" +@@ -363,7 +365,16 @@ + StackSeedDomain seeds=seedCount ? StackSeedDomain(seedHandler,n,m,span1,span2) + : StackSeedDomain(std::vector(),n,m,span1,span2); + // Freeze heuristic choices before computing the shared forward/outside objective. ++ std::cerr << "#profile_seeds"; ++ size_t profileSeedCount=0; ++ for(size_t i=0;i +@@ -408,17 +411,27 @@ + auto forAnchors=[&](const Group & group,const auto & visit) { + for(size_t a=group.first;a;a=anchors[a-1].next) visit(anchors[a-1].pair); + }; ++ uint64_t profileSources=0,profileLeftCells=0,profileBoundedLeftCells=0,profileGroups=0,profileAllGroups=0,profileRightCells=0,profileCroppedCells=0,profileAnchors=0; ++ double profileLeftSeconds=0,profileRightSeconds=0; + seeds.forEachStart([&](size_t p1,size_t p2) { + const Pair p{p1,p2}; if (!w.valid(p)) return; + const size_t n=std::min(d.span1,d.n-p1),m=std::min(d.span2,d.m-p2); ++ ++profileSources; profileLeftCells+=n*m; ++ const auto profileLeftBegin=std::chrono::steady_clock::now(); + reset(left,n*m,cap); reset(leftTotal,n,m); reset(hit,n,m); + reset(hybrid,n,m); + if(w.selectRight) reset(byEnd,n,m); + if(pairs) { reset(boundary,n,m); reset(dHit,n,m); } ++ size_t profilePrefixN=0,profilePrefixM=0; + for(size_t i=0;i(std::chrono::steady_clock::now()-profileLeftBegin).count(); + // Retained right extensions share first-hit injections by linearity: + // T[p,r] = sum_q F[p,q] B[q,r]. Injections share work only when they + // have the same retained endpoint (or retain all endpoints). +@@ -477,6 +491,12 @@ + addAnchor(groups[groupIndex],{i,j}); + } + auto fillRight=[&](const Group & group) { ++ const auto profileRightBegin=std::chrono::steady_clock::now(); ++ ++profileGroups; if(group.all) ++profileAllGroups; ++ size_t profileMin1=group.end.first,profileMin2=group.end.second; ++ forAnchors(group,[&](Pair q) { ++profileAnchors; profileMin1=std::min(profileMin1,q.first); profileMin2=std::min(profileMin2,q.second); }); ++ profileRightCells+=(group.end.first+1)*(group.end.second+1); ++ profileCroppedCells+=(group.end.first-profileMin1+1)*(group.end.second-profileMin2+1); + reset(right,group.end.first+1,group.end.second+1); + forAnchors(group,[&](Pair q) { right(q.first,q.second)=hit(q.first,q.second); }); + // B is seed-unconstrained: no seed lookup or subtraction occurs here. +@@ -494,6 +514,7 @@ + } + }); + } ++ profileRightSeconds+=std::chrono::duration(std::chrono::steady_clock::now()-profileRightBegin).count(); + }; + for(const auto & group:groups) { + if(!group.first) continue; +@@ -576,6 +597,11 @@ + }); + } + }); ++ std::cerr << "#profile_kernel sources=" << profileSources << " left_cells=" << profileLeftCells ++ << " bounded_prefix_cells=" << profileBoundedLeftCells << " fillRight_calls=" << profileGroups << " unrestricted_groups=" << profileAllGroups ++ << " right_rectangle_cells=" << profileRightCells << " minima_cropped_cells=" << profileCroppedCells ++ << " firsthit_anchors=" << profileAnchors << " prefix_s=" << profileLeftSeconds ++ << " fillRight_s=" << profileRightSeconds << '\n'; + return out; + } + } diff --git a/doc/local-validation-archive/files/issue-270-validation/profile-analysis/run-profile.py b/doc/local-validation-archive/files/issue-270-validation/profile-analysis/run-profile.py new file mode 100644 index 00000000..6c3dc70b --- /dev/null +++ b/doc/local-validation-archive/files/issue-270-validation/profile-analysis/run-profile.py @@ -0,0 +1,33 @@ +from pathlib import Path +import subprocess +import shlex +import hashlib +import time +import argparse +parser=argparse.ArgumentParser(description='Run one seeded M/H attribution sample on the OxyS/fhlA issue reproduction.') +parser.add_argument('--binary',type=Path,required=True) +parser.add_argument('--output',type=Path,required=True) +parser.add_argument('--modes',nargs='+',choices=['M','H'],default=['M','H']) +parser.add_argument('--reference',type=Path,help='optional directory with M-seed.stdout and H-seed.stdout') +args=parser.parse_args() +out=args.output.resolve() +binary=args.binary.resolve() +out.mkdir(parents=True,exist_ok=True) +query='GAAACGGAGCGGCACCUCUUUUAACCCUUGAAGUCACUGCCCGUUUCGAGAGUUUCUCAACUCGAAUAACUAAAGCCAACGUGAACUUUUGCGGAUCUCCAGGAUCCGC' +target='AGUUAGUCAAUGACCUUUUGCACCGCUUUGCGGUGCUUUCCUGGAACAACAAAAUGUCAUAUACACCGAUGAGUGAUCUCGGACAACAAGGGUUGUUCGACAUCACUCGGACA' +commands=[] +for mode in args.modes: + command=[str(binary),'--qId=OxyS','--query='+query,'--tId=fhlA','--target='+target,'--model=P','--mode='+mode,'--threads=1'] + commands.append(shlex.join(command)) + before=time.perf_counter() + with (out/(mode+'-seed.stdout')).open('w') as stdout, (out/(mode+'-seed.stderr')).open('w') as stderr: + subprocess.run(['/usr/bin/time','-v','-o',str(out/(mode+'-seed.time')),*command],stdout=stdout,stderr=stderr,check=True) + print(mode, time.perf_counter()-before, flush=True) + if args.reference: + reference=(args.reference/(mode+'-seed.stdout')).read_bytes() + actual=(out/(mode+'-seed.stdout')).read_bytes() + print(mode,'stdout_exact_match',actual==reference,flush=True) + if actual!=reference: + raise RuntimeError(mode+' instrumented output differs from reference') +(out/'commands.txt').write_text('\n'.join(commands)+'\n') +(out/'binary-sha256.txt').write_text(hashlib.sha256(binary.read_bytes()).hexdigest()+' '+str(binary)+'\n') diff --git a/doc/local-validation-archive/files/issue-270-validation/profile/src/IntaRNA/PredictorMfeEns2dSeedExtension.cpp b/doc/local-validation-archive/files/issue-270-validation/profile/src/IntaRNA/PredictorMfeEns2dSeedExtension.cpp new file mode 100644 index 00000000..beb96377 --- /dev/null +++ b/doc/local-validation-archive/files/issue-270-validation/profile/src/IntaRNA/PredictorMfeEns2dSeedExtension.cpp @@ -0,0 +1,746 @@ +#include +#include + +#include "IntaRNA/PredictorMfeEns2dSeedExtension.h" +#include "IntaRNA/SeededPartitionFunction.h" +#include "IntaRNA/PartitionArithmetic.h" +#include +#include + +namespace IntaRNA { + +// The heuristic objective contains signed seed-overlap corrections. Trace the +// implemented arithmetic directly, retaining pair ownership on every term. +// This is allocated only for requested probability output; ordinary prediction +// follows its original arithmetic and storage path. +struct PredictorMfeEns2dSeedExtension::ExtensionProbabilityTrace { + using Pair=Interaction::BasePair; + struct Edge { size_t child; Z_type weight; std::vector pairs; }; + struct Tape { + size_t columns=0; + std::vector> edges; + std::vector adjoint; + void reset(size_t n,size_t m); + void add(size_t p1,size_t p2,size_t c1,size_t c2,Z_type weight,std::vector pairs); + void reverse(const Matrix & values,ExtensionProbabilityTrace & trace); + }; + Matrix mass,absoluteMass; + Matrix seeds; + Tape left,right; + std::vector anchor; + size_t si1=0,si2=0,sj1=0,sj2=0; + Z_type seedZ=0,z=0; + explicit ExtensionProbabilityTrace(size_t n,size_t m); + static Z_type finite(Z_type x); + static Z_type product(Z_type a,Z_type b); + void addMass(const Pair & bp,Z_type value); +}; + +void +PredictorMfeEns2dSeedExtension::ExtensionProbabilityTrace::Tape::reset(size_t n,size_t m) +{ + columns=m; + edges.assign(n*m,{}); + adjoint.assign(n*m,0); +} + +void +PredictorMfeEns2dSeedExtension::ExtensionProbabilityTrace::Tape::add( + size_t p1,size_t p2,size_t c1,size_t c2,Z_type weight,std::vector pairs) +{ + finite(weight); + edges.at(p1*columns+p2).push_back({c1*columns+c2,weight,std::move(pairs)}); +} + +void +PredictorMfeEns2dSeedExtension::ExtensionProbabilityTrace::Tape::reverse( + const Matrix & values,ExtensionProbabilityTrace & trace) +{ + for(size_t parent=edges.size();parent-- >0;) { + finite(values(parent/columns,parent%columns)); + for(const auto & edge:edges[parent]) { + const Z_type a=product(adjoint[parent],edge.weight); + adjoint[edge.child]=finite(adjoint[edge.child]+a); + const Z_type mass=product(a,values(edge.child/columns,edge.child%columns)); + for(const auto & bp:edge.pairs) trace.addMass(bp,mass); + } + } +} + +PredictorMfeEns2dSeedExtension::ExtensionProbabilityTrace::ExtensionProbabilityTrace(size_t n,size_t m) + : mass(n,m,0),absoluteMass(n,m,0),seeds(n,m,0) +{} + +Z_type +PredictorMfeEns2dSeedExtension::ExtensionProbabilityTrace::finite(Z_type x) +{ + if (!std::isfinite(x)) throw std::range_error("heuristic seeded probabilities: nonfinite arithmetic"); + return x; +} + +Z_type +PredictorMfeEns2dSeedExtension::ExtensionProbabilityTrace::product(Z_type a,Z_type b) +{ + finite(a); finite(b); + const Z_type value=finite(a*b); + if(a!=0 && b!=0 && (value==0 || std::abs(value)::min())) + throw std::range_error("heuristic seeded probabilities: product underflow"); + return value; +} + +void +PredictorMfeEns2dSeedExtension::ExtensionProbabilityTrace::addMass(const Pair & bp,Z_type value) +{ + mass(bp.first,bp.second)=finite(mass(bp.first,bp.second)+value); + absoluteMass(bp.first,bp.second)=finite(absoluteMass(bp.first,bp.second)+std::abs(value)); +} + +void PredictorMfeEns2dSeedExtension::initExtensionProbabilities(size_t n,size_t m) +{ + if(pairProbabilities) extensionProbabilityTrace=std::make_unique(n,m); +} + +std::vector +PredictorMfeEns2dSeedExtension::extensionSeedPrefix(size_t i1,size_t i2,size_t before) const +{ + Interaction seed(energy.getAccessibility1().getSequence(),energy.getAccessibility2().getAccessibilityOrigin().getSequence()); + seed.basePairs.push_back(energy.getBasePair(i1,i2)); + seedHandler.traceBackSeed(seed,i1,i2); + seed.basePairs.push_back(energy.getBasePair(i1+seedHandler.getSeedLength1(i1,i2)-1, + i2+seedHandler.getSeedLength2(i1,i2)-1)); + std::vector pairs; + for(const auto & bp:seed.basePairs) { + const size_t p=energy.getIndex1(bp),q=energy.getIndex2(bp); + if(p::epsilon()*Z_type(n+m+1); + if(std::abs(Zall-trace.z)>tolerance*std::max(std::abs(trace.z),std::abs(Zall))) + throw std::logic_error("heuristic seeded probabilities: outside objective differs from reported partition"); + const auto first=energy.getBasePair(0,0),last=energy.getBasePair(n-1,m-1); + const IndexRange target(first.first,last.first),query(last.second,first.second); + Matrix original(n,m,0); + Matrix annotations(n,m,0); + for(size_t i=0;iaddRegion(target,query,trace.z,original,&annotations); + extensionProbabilityTrace.reset(); +} + +////////////////////////////////////////////////////////////////////////// + +PredictorMfeEns2dSeedExtension:: +PredictorMfeEns2dSeedExtension( + const InteractionEnergy & energy + , OutputHandler & output + , PredictionTracker * predTracker + , SeedHandler * seedHandlerInstance + , BasePairProbabilities * pairProbabilities ) + : + PredictorMfeEns(energy,output,predTracker) + , pairProbabilities(pairProbabilities) + , seedHandler(seedHandlerInstance) + , hybridZ_left( 0,0 ) + , hybridZ_right( 0,0 ) +{ + if (pairProbabilities && !output.getOutputConstraint().needZall) + throw std::invalid_argument("pair probabilities require needZall"); +} + +////////////////////////////////////////////////////////////////////////// + +PredictorMfeEns2dSeedExtension:: +~PredictorMfeEns2dSeedExtension() +{ +} + +////////////////////////////////////////////////////////////////////////// + +void +PredictorMfeEns2dSeedExtension:: +predict( const IndexRange & r1, const IndexRange & r2 ) +{ + if (seedHandler.guaranteesStackOnlySeeds()) { + try { predictStackSeeds(r1,r2); } + catch (...) { if (pairProbabilities) pairProbabilities->fail(); throw; } + return; + } + if (pairProbabilities) { + pairProbabilities->fail(); + throw std::invalid_argument("exact bulged-seed pair probabilities are unsupported"); + } + if (seedHandler.getConstraint().getBasePairs()<2) + throw std::invalid_argument("legacy bulged seed extension requires at least two seed pairs"); +#if INTARNA_MULITHREADING + #pragma omp critical(intarna_omp_logOutput) +#endif + { VLOG(2) <<"predicting ensemble mfe interactions with seed in O(n^2) space and O(n^4) time..."; } + // measure timing + TIMED_FUNC_IF(timerObj,VLOG_IS_ON(9)); + +#if INTARNA_IN_DEBUG_MODE + // check indices + if (!(r1.isAscending() && r2.isAscending()) ) + throw std::runtime_error("PredictorMfeEns2dSeedExtension::predict("+toString(r1)+","+toString(r2)+") is not sane"); +#endif + + // setup index offset + energy.setOffset1(r1.from); + energy.setOffset2(r2.from); + seedHandler.setOffset1(r1.from); + seedHandler.setOffset2(r2.from); + + const size_t range_size1 = std::min( energy.size1() + , (r1.to==RnaSequence::lastPos?energy.size1()-1:r1.to)-r1.from+1 ); + const size_t range_size2 = std::min( energy.size2() + , (r2.to==RnaSequence::lastPos?energy.size2()-1:r2.to)-r2.from+1 ); + + // compute seed interactions for whole range + // and check if any seed possible + if (seedHandler.fillSeed( 0, range_size1-1, 0, range_size2-1 ) == 0) { + // trigger empty interaction reporting + initOptima(); + initZ(); + reportOptima(); + // stop computation + return; + } + + // initialize mfe interaction for updates + initOptima(); + // initialize overall partition function for updates + initZ(); + + size_t si1 = RnaSequence::lastPos, si2 = RnaSequence::lastPos; + while( seedHandler.updateToNextSeed(si1,si2 + , 0, range_size1+1-seedHandler.getConstraint().getBasePairs() + , 0, range_size2+1-seedHandler.getConstraint().getBasePairs()) ) + { + // get Z and boundaries of seed + const Z_type seedZ = energy.getBoltzmannWeight( seedHandler.getSeedE(si1, si2) ); + const size_t sl1 = seedHandler.getSeedLength1(si1, si2); + const size_t sl2 = seedHandler.getSeedLength2(si1, si2); + const size_t sj1 = si1+sl1-1; + const size_t sj2 = si2+sl2-1; + // check if seed fits into interaction range + if (sj1 > range_size1 || sj2 > range_size2) + continue; + const size_t maxMatrixLen1 = energy.getAccessibility1().getMaxLength()-sl1+1; + const size_t maxMatrixLen2 = energy.getAccessibility2().getMaxLength()-sl2+1; + + // ER + hybridZ_right.resize( std::min(range_size1-sj1, maxMatrixLen1), std::min(range_size2-sj2, maxMatrixLen2) ); + fillHybridZ_right(sj1, sj2); + + // EL + hybridZ_left.resize( std::min(si1+1, maxMatrixLen1), std::min(si2+1, maxMatrixLen2) ); + fillHybridZ_left(si1, si2); + + // updateZ for all boundary combinations + for (size_t l1 = 0; l1= energy.getAccessibility1().getMaxLength()) break; + // iterate extension right of seed in seq 2 + for (size_t r2 = 0; r2 < hybridZ_right.size2() ; r2++) { + // ensure max interaction length in seq 2 + if (sj2+r2-si2+l2 >= energy.getAccessibility2().getMaxLength()) break; + // check complementarity of boundary + if (Z_equal(hybridZ_right(r1,r2),0.0)) continue; + // compute partition function given the current seed + updateZ(si1-l1, sj1+r1, si2-l2, sj2+r2, hybridZ_left(l1,l2) * seedZ * hybridZ_right(r1,r2), true); + + } // r2 + } // l2 + } // r1 + } // l1 + + } // si1 / si2 + + // report mfe interaction + reportOptima(); + +} + +////////////////////////////////////////////////////////////////////////// + +Z_type +PredictorMfeEns2dSeedExtension::exactBoundaryWeight(size_t i1,size_t j1,size_t i2,size_t j2) const +{ + if (!isValidOutputSite(i1,j1,i2,j2)) return 0; + const E_type e=energy.getE(i1,j1,i2,j2,E_type(0)); + return E_isINF(e)?Z_type(0):PartitionArithmetic::exp(-E_2_Z(e)/energy.getRT()); +} + +void +PredictorMfeEns2dSeedExtension::prepareStackSeedExtensions(size_t,size_t,const StackSeedDomain &) +{} + +std::function +PredictorMfeEns2dSeedExtension::stackSeedRightSelector() const +{ + return {}; +} + +void +PredictorMfeEns2dSeedExtension::predictStackSeeds(const IndexRange & r1,const IndexRange & r2) +{ + // Validate public regions before translating both strands to local coordinates. + const size_t size1=energy.getAccessibility1().getSequence().size(); + const size_t size2=energy.getAccessibility2().getSequence().size(); + if (!r1.isAscending() || !r2.isAscending() || r1.from>=size1 || r2.from>=size2) + throw std::invalid_argument("exact seeded prediction: invalid region"); + energy.setOffset1(r1.from); energy.setOffset2(r2.from); + seedHandler.setOffset1(r1.from); seedHandler.setOffset2(r2.from); + const size_t n=r1.to==RnaSequence::lastPos?energy.size1():std::min(energy.size1(),r1.to-r1.from+1); + const size_t m=r2.to==RnaSequence::lastPos?energy.size2():std::min(energy.size2(),r2.to-r2.from+1); + // Compile admitted, contained seeds using the active spans for this prediction. + initOptima(); initZ(); Zall=0; + const size_t seedCount=seedHandler.fillSeed(0,n-1,0,m-1); + const size_t span1=std::min(n,energy.getAccessibility1().getMaxLength()); + const size_t span2=std::min(m,energy.getAccessibility2().getMaxLength()); + StackSeedDomain seeds=seedCount ? StackSeedDomain(seedHandler,n,m,span1,span2) + : StackSeedDomain(std::vector(),n,m,span1,span2); + // Freeze heuristic choices before computing the shared forward/outside objective. + std::cerr << "#profile_seeds"; + size_t profileSeedCount=0; + for(size_t i=0;i si1p ) + throw std::runtime_error("PredictorMfeEns2dSeedExtension::getNonOverlappingEnergy( si "+toString(si1)+","+toString(si2)+", sip "+toString(si1p)+","+toString(si2p)+",..) si1 > sj1 !"); + // check if loop-overlapping (i.e. share at least one loop) + if( !seedHandler.areLoopOverlapping(si1,si2,si1p,si2p) ) { + throw std::runtime_error("PredictorMfeEns2dSeedExtension::getNonOverlappingEnergy( si "+toString(si1)+","+toString(si2)+", sip "+toString(si1p)+","+toString(si2p)+",..) are not loop overlapping"); + } +#endif + + // identity check + if( si1 == si1p ) { + return E_type(0); + } + + // trace seed at (si1,si2) + Interaction interaction = Interaction(energy.getAccessibility1().getSequence(), energy.getAccessibility2().getAccessibilityOrigin().getSequence()); + interaction.basePairs.push_back( energy.getBasePair(si1, si2) ); + seedHandler.traceBackSeed( interaction, si1, si2 ); + + E_type fullE = 0; + size_t k1old = si1, k2old = si2; + for (size_t i = 1; i < interaction.basePairs.size(); i++) { + // get index of current base pair + size_t k1 = energy.getIndex1(interaction.basePairs[i]); + // check if overlap done + if (k1 > si1p) break; + size_t k2 = energy.getIndex2(interaction.basePairs[i]); + // add hybridization energy + fullE += energy.getE_interLeft(k1old,k1,k2old,k2); + + // store + k1old = k1; + k2old = k2; + } + return fullE; +} + +//////////////////////////////////////////////////////////////////////////// + +void +PredictorMfeEns2dSeedExtension:: +fillHybridZ_left( const size_t si1, const size_t si2 ) +{ + if(extensionProbabilityTrace) extensionProbabilityTrace->left.reset(hybridZ_left.size1(),hybridZ_left.size2()); + // temporary access + const OutputConstraint & outConstraint = output.getOutputConstraint(); +#if INTARNA_IN_DEBUG_MODE + // check indices + if (!energy.areComplementary(si1,si2) ) + throw std::runtime_error("PredictorMfeEns2dSeedExtension::fillHybridZ_left("+toString(si1)+","+toString(si2)+",..) are not complementary"); +#endif + + // global vars to avoid reallocation + size_t i1,i2,k1,k2; + + // determine whether or not lonely base pairs are allowed or if we have to + // ensure a stacking to the right of the left boundary (i1,i2) + const size_t noLpShift = outConstraint.noLP ? 1 : 0; + Z_type iStackZ = Z_type(1); + + // iterate over all window starts i1 (seq1) and i2 (seq2) + for (size_t l1=0; l1 < hybridZ_left.size1(); l1++) { + for (size_t l2=0; l2 < hybridZ_left.size2(); l2++) { + i1 = si1-l1; + i2 = si2-l2; + + // referencing cell access + Z_type & curZ = hybridZ_left(si1-i1,si2-i2); + + // init current cell (0 if not just right-most (j1,j2) base pair) + curZ = (i1==si1 && i2==si2) ? energy.getBoltzmannWeight(energy.getE_init()) : 0.0; + + // check if complementary (use global sequence indexing) + if( i1left.add(l1,l2,l1-noLpShift,l2-noLpShift,iStackZ,{{i1,i2}}); + } + + // check all combinations of decompositions into (i1,i2)..(k1,k2)-(j1,j2) + for (k1=i1+noLpShift; k1++ < si1; ) { + // ensure maximal loop length + if (k1-i1-noLpShift > energy.getMaxInternalLoopSize1()+1) break; + for (k2=i2+noLpShift; k2++ < si2; ) { + // ensure maximal loop length + if (k2-i2-noLpShift > energy.getMaxInternalLoopSize2()+1) break; + // check if (k1,k2) are valid left boundary + if ( ! Z_equal(hybridZ_left(si1-k1,si2-k2), 0.0) ) { + curZ += (iStackZ + * energy.getBoltzmannWeight(energy.getE_interLeft(i1+noLpShift,k1,i2+noLpShift,k2)) + * hybridZ_left(si1-k1,si2-k2)); + if(extensionProbabilityTrace) { + std::vector pairs{{i1,i2}}; + if(noLpShift) pairs.emplace_back(i1+1,i2+1); + extensionProbabilityTrace->left.add(l1,l2,si1-k1,si2-k2, + iStackZ*energy.getBoltzmannWeight(energy.getE_interLeft(i1+noLpShift,k1,i2+noLpShift,k2)),std::move(pairs)); + } + } + } // k2 + } // k1 + + // correction for left seeds + if ( i1 i1 && seedHandler.areLoopOverlapping(i1, i2, sj1, sj2)) { + // update left-most loop-overlapping seed + if (sj1 < si1overlap) { + si1overlap = sj1; + si2overlap = sj2; + } + } + } + + // if we found an overlapping seed + if (si1overlap <= si1) { + // check if right side is non-empty + if ( ! E_equal(hybridZ_left( si1-si1overlap, si2-si2overlap),0) ) { + // compute Energy of loop S \ S' + E_type nonOverlapE = getNonOverlappingEnergy(i1, i2, si1overlap, si2overlap); + // subtract energy.getBoltzmannWeight( nonOverlapE ) * hybridZ_left( si1overlap, si2overlap up to anchor seed [==1 if equal]) + Z_type correctionTerm = energy.getBoltzmannWeight( nonOverlapE ) + * hybridZ_left( si1-si1overlap, si2-si2overlap); + curZ -= correctionTerm; + if(extensionProbabilityTrace) extensionProbabilityTrace->left.add(l1,l2,si1-si1overlap,si2-si2overlap, + -energy.getBoltzmannWeight(nonOverlapE),extensionSeedPrefix(i1,i2,si1overlap)); + // sanity insurance + if (curZ < 0) { + curZ = Z_type(0.0); + if(extensionProbabilityTrace) extensionProbabilityTrace->left.edges[l1*extensionProbabilityTrace->left.columns+l2].clear(); + } + } + } else { + // get data for seed to be removed + const Z_type seedZ_rm = energy.getBoltzmannWeight(seedHandler.getSeedE(i1, i2)); + const size_t sj1_rm = i1+seedHandler.getSeedLength1(i1,i2)-1; + const size_t sj2_rm = i2+seedHandler.getSeedLength2(i1,i2)-1; + // if no S' + // substract seedZ * hybridZ_left(right end seed up to anchor seed) + Z_type correctionTerm = seedZ_rm + * hybridZ_left( si1-sj1_rm + , si2-sj2_rm ); + if(extensionProbabilityTrace) extensionProbabilityTrace->left.add(l1,l2,si1-sj1_rm,si2-sj2_rm, + -seedZ_rm,extensionSeedPrefix(i1,i2,sj1_rm)); + // if noLP : handle explicit loop right of current seed + if (outConstraint.noLP) { + for (k1=sj1_rm; k1++ < si1; ) { + // ensure maximal loop length + if (k1-sj1_rm > energy.getMaxInternalLoopSize1()+1) break; + for (k2=sj2_rm; k2++ < si2; ) { + // ensure at least one unpaired base in interior loop following the seed to be removed + if (sj1_rm-k1 + sj2_rm-k2 == 2) {continue;} + // ensure maximal loop length + if (k2-sj2_rm > energy.getMaxInternalLoopSize2()+1) break; + // check if (k1,k2) are valid left boundary + if ( ! Z_equal(hybridZ_left(si1-k1,si2-k2), 0.0) ) { + correctionTerm += (seedZ_rm + * energy.getBoltzmannWeight(energy.getE_interLeft(sj1_rm,k1,sj2_rm,k2)) + * hybridZ_left(si1-k1,si2-k2) ); + if(extensionProbabilityTrace) extensionProbabilityTrace->left.add(l1,l2,si1-k1,si2-k2, + -seedZ_rm*energy.getBoltzmannWeight(energy.getE_interLeft(sj1_rm,k1,sj2_rm,k2)),extensionSeedPrefix(i1,i2,k1)); + } + } // k2 + } // k1 + + } + curZ -= correctionTerm; + // sanity insurance + if (curZ < 0) { + curZ = Z_type(0.0); + if(extensionProbabilityTrace) extensionProbabilityTrace->left.edges[l1*extensionProbabilityTrace->left.columns+l2].clear(); + } + } + } // substractThisSeed + } + + } // complementary + + } // i2 + } // i1 + +} + +//////////////////////////////////////////////////////////////////////////// + +void +PredictorMfeEns2dSeedExtension:: +fillHybridZ_right( const size_t sj1, const size_t sj2 ) +{ + if(extensionProbabilityTrace) extensionProbabilityTrace->right.reset(hybridZ_right.size1(),hybridZ_right.size2()); + // temporary access + const OutputConstraint & outConstraint = output.getOutputConstraint(); +#if INTARNA_IN_DEBUG_MODE + // check indices + if (!energy.areComplementary(sj1,sj2) ) + throw std::runtime_error("PredictorMfeEns2dSeedExtension::fillHybridZ_right("+toString(sj1)+","+toString(sj2)+",..) are not complementary"); +#endif + + // global vars to avoid reallocation + size_t j1,j2,k1,k2; + + // determine whether or not lonely base pairs are allowed or if we have to + // ensure a stacking to the right of the left boundary (i1,i2) + const size_t noLpShift = outConstraint.noLP ? 1 : 0; + Z_type iStackZ = Z_type(1); + + // iterate over all window ends j1 (seq1) and j2 (seq2) + for (j1=sj1; j1-sj1 < hybridZ_right.size1(); j1++ ) { + for (j2=sj2; j2-sj2 < hybridZ_right.size2(); j2++ ) { + + // referencing cell access + Z_type & curZ = hybridZ_right(j1-sj1,j2-sj2); + + // init partition function for current cell -> (i1,i2) are complementary per definition + curZ = sj1==j1 && sj2==j2 ? energy.getBoltzmannWeight(0.0) : 0.0; + + // check if complementary free base pair + if( sj1right.add(j1-sj1,j2-sj2,0,0,iStackZ,{{j1,j2}}); + } + } + + // check all combinations of decompositions into (i1,i2)..(k1,k2)-(j1,j2) + for (k1=j1-noLpShift; k1-- > sj1; ) { + // ensure maximal loop length + if (j1-noLpShift-k1 > energy.getMaxInternalLoopSize1()+1) break; + for (k2=j2-noLpShift; k2-- > sj2; ) { + // ensure maximal loop length + if (j2-noLpShift-k2 > energy.getMaxInternalLoopSize2()+1) break; + // check if (k1,k2) are valid left boundary + if ( ! Z_equal(hybridZ_right(k1-sj1,k2-sj2), 0.0) ) { + // update partition function + curZ += ( hybridZ_right(k1-sj1,k2-sj2) + * energy.getBoltzmannWeight(energy.getE_interLeft(k1,j1-noLpShift,k2,j2-noLpShift)) + * iStackZ ); + if(extensionProbabilityTrace) { + std::vector pairs{{j1,j2}}; + if(noLpShift) pairs.emplace_back(j1-1,j2-1); + extensionProbabilityTrace->right.add(j1-sj1,j2-sj2,k1-sj1,k2-sj2, + energy.getBoltzmannWeight(energy.getE_interLeft(k1,j1-noLpShift,k2,j2-noLpShift))*iStackZ,std::move(pairs)); + } + } + } // k2 + } // k1 + } + } + } + +} + +//////////////////////////////////////////////////////////////////////////// + +void +PredictorMfeEns2dSeedExtension:: +traceBack( Interaction & interaction ) +{ + + // forward tracing + PredictorMfeEns::traceBack(interaction); + + // add seeds in region + seedHandler.addSeeds( interaction ); + +} + +//////////////////////////////////////////////////////////////////////////// + + +} // namespace diff --git a/doc/local-validation-archive/files/issue-270-validation/profile/src/IntaRNA/SeededPartitionFunction.cpp b/doc/local-validation-archive/files/issue-270-validation/profile/src/IntaRNA/SeededPartitionFunction.cpp new file mode 100644 index 00000000..60be84ee --- /dev/null +++ b/doc/local-validation-archive/files/issue-270-validation/profile/src/IntaRNA/SeededPartitionFunction.cpp @@ -0,0 +1,607 @@ +#include +#include +#include +#include "IntaRNA/SeededPartitionFunction.h" +#include "IntaRNA/PartitionArithmetic.h" +#include +#include +#include +#include + +namespace IntaRNA { +namespace { +using Kernel=SeededPartitionFunction; +using Pair=Kernel::Pair; +using A=PartitionArithmetic; +using Support=std::array; +using RightStates=std::array; + +/** Exact left-most-anchor factorization, with bounded boundary publication. */ +Kernel::Result +computeAnchored(const Kernel::Domain & d,const StackSeedDomain & seeds, + const Kernel::Weights & w,bool pairs) +{ + Kernel::Result out; + if(pairs) out.mass=Matrix(d.n,d.m,0); + if(!d.n || !d.m || !seeds.maxSeedLength() || !d.span1 || !d.span2) return out; + + // Containing seeds add no paths to the exact union. In the remaining family, + // start order equals end order along each diagonal, including overlapping + // seeds. Thus excluding an earlier seed while extending left is sufficient; + // no earlier seed can first finish beyond the anchor's right end. + const auto anchors=seeds.minimalOccurrences(); + std::unordered_map anchorIndex; + anchorIndex.reserve(anchors.size()); + size_t cap=2; + for(size_t a=0;a left(cells,cap),leftTotal(maxN,maxM),rightTotal(maxN,maxM); + Matrix right(maxN,maxM); + Matrix dLeft,dLeftTotal; + Matrix dRight; + if(pairs) { + dLeft.resize(cells,cap,false); dLeftTotal.resize(maxN,maxM,false); + dRight.resize(maxN,maxM,false); + } + Matrix valid(d.n,d.m); + for(size_t i=0;i hybrid(batchSize,cells),boundary; + if(pairs) boundary.resize(batchSize,cells,false); + std::vector starts; + std::vector initiation; + std::vector candidates; + std::vector included(anchors.size(),0); + starts.reserve(batchSize); initiation.reserve(batchSize); + candidates.reserve(anchors.size()); + auto reset=[](Matrix & matrix,size_t rows,size_t columns) { + if(columns==matrix.size2()) std::fill_n(&matrix(0,0),rows*columns,T{}); + else for(size_t i=0;i & matrix,size_t rows,size_t columns) { + // Left states use a fixed maxM row stride and a contiguous stack axis. + for(size_t i=0;isecond].length; + }; + auto nonstack=[&](size_t i,size_t j,const auto & visit) { + for(size_t u=1;u<=i && u-1<=d.loop1;++u) + for(size_t v=1;v<=j && v-1<=d.loop2;++v) + if(u!=1 || v!=1) visit(i-u,j-v); + }; + auto fill=[&](size_t a) { + const auto anchor=anchors[a]; + s={anchor.i,anchor.j}; length=anchor.length; + t={s.first+length-1,s.second+length-1}; + singleton=d.noLP && length==1; + supportBegin=singleton?0:1; + // Crop both extension rectangles to the batch's possible left boundaries. + // Paths to these sources cannot visit positions beyond their leftmost + // coordinates; their mandatory left offsets also bound the right side. + nL=0; mL=0; + size_t minLeft1=maxN,minLeft2=maxM; + for(const Pair p:starts) { + if(p.first>s.first || p.second>s.second) continue; + const size_t u=s.first-p.first,v=s.second-p.second; + if(u>maxN-length || v>maxM-length) continue; + nL=std::max(nL,u+1); mL=std::max(mL,v+1); + minLeft1=std::min(minLeft1,u); minLeft2=std::min(minLeft2,v); + } + if(!nL || !mL) { seedWeight=0; return; } + nR=std::min(d.n-t.first,maxN-length-minLeft1+1); + mR=std::min(d.m-t.second,maxM-length-minLeft2+1); + resetLeft(left,nL,mL); reset(leftTotal,nL,mL); + reset(right,nR,mR); reset(rightTotal,nR,mR); + // Seed energy consists of its actual stack transitions. Initiation is + // attached later to the complete interaction's left boundary exactly once. + seedWeight=1; + for(size_t k=0;ks.first || p.second>s.second) continue; + const size_t u=s.first-p.first,v=s.second-p.second; + if(u0 || leftTotal(u,v)[1]>0)) { reachable=true; break; } + } + if(!reachable) { seedWeight=0; return; } + right(0,0)[singleton?0:3]=1; + for(size_t i=0;is.first || p.second>s.second) continue; + const size_t u=s.first-p.first,v=s.second-p.second; + if(u>=nL || v>=mL || (leftTotal(u,v)[0]==0 && leftTotal(u,v)[1]==0)) continue; + const Z_type source=A::multiply(initiation[b],seedWeight); + Support sourceLeft{}; + for(size_t h=supportBegin;h<2;++h) sourceLeft[h]=A::multiply(source,leftTotal(u,v)[h]); + for(size_t i=0;i0;--ii) for(size_t jj=mL;jj>0;--jj) { + const size_t i=ii-1,j=jj-1; const Pair p=leftPair(i,j); + if(!valid(p.first,p.second)) continue; + const size_t forbidden=earlierLength(p); + for(size_t c=d.noLP && !(singleton && i==0 && j==0)?1:0;c0;--ii) for(size_t jj=mR;jj>0;--jj) { + const size_t i=ii-1,j=jj-1; const Pair q=rightPair(i,j); + if(!valid(q.first,q.second)) continue; + for(size_t c=supportBegin*2;c<4;++c) if(right(i,j)[c]==0) dRight(i,j)[c]=0; + if(i || j) for(size_t c=supportBegin*2;c<4;++c) out.mass(q.first,q.second)=A::add( + out.mass(q.first,q.second),A::multiply(right(i,j)[c],dRight(i,j)[c])); + if(i && j) { + const Pair p=rightPair(i-1,j-1); + if(valid(p.first,p.second)) { + const Z_type edge=A::check(w.transition(p,q)); + for(size_t h=supportBegin;h<2;++h) { + const size_t support=singleton && i==1 && j==1?1:h; + if(dRight(i,j)[support*2+1]==0) continue; + const Z_type adj=A::multiply(edge,dRight(i,j)[support*2+1]); + for(size_t c=0;c<2;++c) if(right(i-1,j-1)[h*2+c]>0) + dRight(i-1,j-1)[h*2+c]=A::add(dRight(i-1,j-1)[h*2+c],adj); + } + } + } + nonstack(i,j,[&](size_t u,size_t v) { + const Pair p=rightPair(u,v); if(!valid(p.first,p.second)) return; + if((dRight(i,j)[d.noLP?0:1]==0 && dRight(i,j)[2+(d.noLP?0:1)]==0) + || (rightTotal(u,v)[0]==0 && rightTotal(u,v)[1]==0)) return; + const Z_type edge=A::check(w.transition(p,q)); + for(size_t h=supportBegin;h<2;++h) { + const size_t frontier=singleton && u==0 && v==0?0:1; + auto & adj=dRight(u,v)[h*2+frontier]; + adj=A::add(adj,A::multiply(edge,dRight(i,j)[h*2+(d.noLP?0:1)])); + } + }); + } + }; + + auto processBatch=[&]() { + reset(hybrid,starts.size(),cells); if(pairs) reset(boundary,starts.size(),cells); + candidates.clear(); initiation.clear(); + // Query only anchor starts inside the batch's span boxes, then restore + // deterministic seed order independently of hash-table iteration order. + for(const Pair p:starts) { + initiation.push_back(A::check(w.initiation(p))); + for(size_t i=p.first;isecond]) continue; + const auto & anchor=anchors[found->second]; + if(anchor.length>maxN-(i-p.first) || anchor.length>maxM-(j-p.second)) continue; + included[found->second]=1; candidates.push_back(found->second); + } + } + std::sort(candidates.begin(),candidates.end()); + for(const size_t a:candidates) { fill(a); contract(false); } + for(size_t b=0;b; // pending lonely pair / stack-supported end + Result out; + if (pairs) out.mass=Matrix(d.n,d.m,0); + if (!seeds.maxSeedLength() || !d.span1 || !d.span2) return out; + + // Freeze heuristic choices before either pass. A first hit is owned by the + // shortest admitted seed ending here, including coincident variable lengths. + std::unordered_map selected; + if (w.selectRight) for(size_t i=0;i left(cells,cap),leftTotal(maxN,maxM),hybrid(maxN,maxM); + Matrix hit(maxN,maxM),right(maxN,maxM); + Matrix valid(maxN,maxM); + Matrix ending(maxN,maxM),byEnd; + Matrix boundary,dLeft,dTotal; + Matrix dHit,adjoint; + if(w.selectRight) byEnd.resize(maxN,maxM,false); + if(pairs) { + boundary.resize(maxN,maxM,false); + dLeft.resize(cells,cap,false); dTotal.resize(maxN,maxM,false); + dHit.resize(maxN,maxM,false); adjoint.resize(maxN,maxM,false); + } + auto reset=[](Matrix & matrix,size_t rows,size_t columns) { + // Matrix storage is row-major. Clear contiguous full-width regions in + // one pass; narrower rectangles leave the inactive row tails untouched. + if(columns==matrix.size2()) std::fill_n(&matrix(0,0),rows*columns,T{}); + else for(size_t i=0;i groups; + std::vector anchors; + groups.reserve(w.selectRight?cells+1:1); anchors.reserve(cells); + auto addAnchor=[&](Group & group,Pair q) { + anchors.push_back({q,group.first}); group.first=anchors.size(); + }; + auto forAnchors=[&](const Group & group,const auto & visit) { + for(size_t a=group.first;a;a=anchors[a-1].next) visit(anchors[a-1].pair); + }; + uint64_t profileSources=0,profileLeftCells=0,profileBoundedLeftCells=0,profileGroups=0,profileAllGroups=0,profileRightCells=0,profileCroppedCells=0,profileAnchors=0; + double profileLeftSeconds=0,profileRightSeconds=0; + seeds.forEachStart([&](size_t p1,size_t p2) { + const Pair p{p1,p2}; if (!w.valid(p)) return; + const size_t n=std::min(d.span1,d.n-p1),m=std::min(d.span2,d.m-p2); + ++profileSources; profileLeftCells+=n*m; + const auto profileLeftBegin=std::chrono::steady_clock::now(); + reset(left,n*m,cap); reset(leftTotal,n,m); reset(hit,n,m); + reset(hybrid,n,m); + if(w.selectRight) reset(byEnd,n,m); + if(pairs) { reset(boundary,n,m); reset(dHit,n,m); } + size_t profilePrefixN=0,profilePrefixM=0; + for(size_t i=0;i0) extendLeft(i,j,std::min(cap,c+2),A::multiply(prefix,t)); + } + } + nonstack(i,j,[&](size_t u,size_t v) { + if(leftTotal(u,v)>0) extendLeft(i,j,1,A::multiply(leftTotal(u,v),transition(i,j,u,v))); + }); + // A nonstack edge may leave only a supported end under noLP. + for(size_t c=d.noLP?1:0;c(std::chrono::steady_clock::now()-profileLeftBegin).count(); + // Retained right extensions share first-hit injections by linearity: + // T[p,r] = sum_q F[p,q] B[q,r]. Injections share work only when they + // have the same retained endpoint (or retain all endpoints). + groups.clear(); anchors.clear(); + groups.push_back({{n-1,m-1},true}); + for(size_t i=0;i=n || end.second-p2>=m) continue; + const Pair local{end.first-p1,end.second-p2}; + auto & groupIndex=byEnd(local.first,local.second); + if(!groupIndex) { + groupIndex=groups.size(); groups.push_back({local,false}); + } + addAnchor(groups[groupIndex],{i,j}); + } + auto fillRight=[&](const Group & group) { + const auto profileRightBegin=std::chrono::steady_clock::now(); + ++profileGroups; if(group.all) ++profileAllGroups; + size_t profileMin1=group.end.first,profileMin2=group.end.second; + forAnchors(group,[&](Pair q) { ++profileAnchors; profileMin1=std::min(profileMin1,q.first); profileMin2=std::min(profileMin2,q.second); }); + profileRightCells+=(group.end.first+1)*(group.end.second+1); + profileCroppedCells+=(group.end.first-profileMin1+1)*(group.end.second-profileMin2+1); + reset(right,group.end.first+1,group.end.second+1); + forAnchors(group,[&](Pair q) { right(q.first,q.second)=hit(q.first,q.second); }); + // B is seed-unconstrained: no seed lookup or subtraction occurs here. + for(size_t i=0;i<=group.end.first;++i) for(size_t j=0;j<=group.end.second;++j) { + if(!valid(i,j)) continue; + if(i && j && valid(i-1,j-1)) { + const auto & prev=right(i-1,j-1); + if(prev[0]>0 || prev[1]>0) + right(i,j)[1]=A::add(right(i,j)[1],A::multiply(A::add(prev[0],prev[1]),transition(i,j,i-1,j-1))); + } + nonstack(i,j,[&](size_t u,size_t v) { + if(right(u,v)[1]>0) { + auto & value=right(i,j)[d.noLP?0:1]; + value=A::add(value,A::multiply(right(u,v)[1],transition(i,j,u,v))); + } + }); + } + profileRightSeconds+=std::chrono::duration(std::chrono::steady_clock::now()-profileRightBegin).count(); + }; + for(const auto & group:groups) { + if(!group.first) continue; + fillRight(group); + if(group.all) { + for(size_t i=0;i0) { + const Pair q{p1+i,p2+j}; + const Z_type b=A::check(w.boundary(p,q)); + if(pairs) boundary(i,j)=b; + out.z=A::add(out.z,A::multiply(hybrid(i,j),b)); + if(w.complete) w.complete(p,q,hybrid(i,j),b); + } + if(!pairs) return; + + // Reverse the retained right contractions. Recompute one group at a time + // to bound memory independently of the number of heuristic choices. + for(size_t g=0;g0;--ii) for(size_t jj=group.end.second+1;jj>0;--jj) { + const size_t i=ii-1,j=jj-1; if(!valid(i,j)) continue; + auto & a=adjoint(i,j); + if(group.all || (i==group.end.first && j==group.end.second)) a[1]=A::add(a[1],boundary(i,j)); + // A vertex is owned by its most recent pair: T*dT counts every + // right-side pair, including the accepting pair, exactly once. + auto & mass=out.mass(p1+i,p2+j); + for(size_t c=0;c<2;++c) mass=A::add(mass,A::multiply(right(i,j)[c],a[c])); + if(i && j && valid(i-1,j-1)) { + const Z_type value=A::multiply(a[1],transition(i,j,i-1,j-1)); + for(size_t c=0;c<2;++c) adjoint(i-1,j-1)[c]=A::add(adjoint(i-1,j-1)[c],value); + } + nonstack(i,j,[&](size_t u,size_t v) { + adjoint(u,v)[1]=A::add(adjoint(u,v)[1],A::multiply(a[d.noLP?0:1],transition(i,j,u,v))); + }); + } + forAnchors(group,[&](Pair q) { + for(size_t c=0;c<2;++c) + dHit(q.first,q.second)[c]=A::add(dHit(q.first,q.second)[c],adjoint(q.first,q.second)[c]); + }); + } + if(w.selectRight) for(size_t i=0;i0;--ii) for(size_t jj=m;jj>0;--jj) { + const size_t i=ii-1,j=jj-1,q=i*m+j; if(!valid(i,j)) continue; + for(size_t c=d.noLP?1:0;cenergy != -400; +} diff --git a/doc/local-validation-archive/files/kinetic-validation/consumer.cpp b/doc/local-validation-archive/files/kinetic-validation/consumer.cpp new file mode 100644 index 00000000..2a554b72 --- /dev/null +++ b/doc/local-validation-archive/files/kinetic-validation/consumer.cpp @@ -0,0 +1,22 @@ +#include +#include +#include +#include +#include + +INITIALIZE_EASYLOGGINGPP + +int main() { + using namespace IntaRNA; + RnaSequence target("target", "GGGG"), query("query", "CCCC"); + AccessibilityDisabled a(target, 0, nullptr), b(query, 0, nullptr); + ReverseAccessibility reverse(b); + InteractionEnergyBasePair energy(a, reverse, 1, 1); + SeedConstraint seeds(2, 0, 0, 0, E_INF, Accessibility::ED_UPPER_BOUND, + E_INF, IndexRangeList(""), IndexRangeList(""), "", false, false, true); + OutputHandlerInteractionList output(OutputConstraint(), 1); + PredictorSeedExtensionKinetic predictor(energy, output, nullptr, + new SeedHandlerNoBulge(energy, seeds)); + predictor.predict(); + return output.empty() || (*output.begin())->energy != -400; +} diff --git a/doc/local-validation-archive/files/kinetic-validation/header-probe-kix.cpp b/doc/local-validation-archive/files/kinetic-validation/header-probe-kix.cpp new file mode 100644 index 00000000..e4bb2529 --- /dev/null +++ b/doc/local-validation-archive/files/kinetic-validation/header-probe-kix.cpp @@ -0,0 +1,2 @@ +#include +int main() { return 0; } diff --git a/doc/local-validation-archive/files/kinetic-validation/header-probe-review.cpp b/doc/local-validation-archive/files/kinetic-validation/header-probe-review.cpp new file mode 100644 index 00000000..e4bb2529 --- /dev/null +++ b/doc/local-validation-archive/files/kinetic-validation/header-probe-review.cpp @@ -0,0 +1,2 @@ +#include +int main() { return 0; } diff --git a/doc/local-validation-archive/files/kinetic-validation/header-probe.cpp b/doc/local-validation-archive/files/kinetic-validation/header-probe.cpp new file mode 100644 index 00000000..66d40f79 --- /dev/null +++ b/doc/local-validation-archive/files/kinetic-validation/header-probe.cpp @@ -0,0 +1,2 @@ +#include +int main() { return 0; } diff --git a/doc/local-validation-archive/files/kinetic-validation/uncached-comparison/PredictorSeedExtensionKinetic.cpp b/doc/local-validation-archive/files/kinetic-validation/uncached-comparison/PredictorSeedExtensionKinetic.cpp new file mode 100644 index 00000000..8d4233df --- /dev/null +++ b/doc/local-validation-archive/files/kinetic-validation/uncached-comparison/PredictorSeedExtensionKinetic.cpp @@ -0,0 +1,409 @@ +#include "IntaRNA/PredictorSeedExtensionKinetic.h" + +#include +#include +#include +#include + +#include + +namespace IntaRNA { +namespace { + +SeedHandler * checkedSeedHandler(SeedHandler * handler) +{ + if (handler == NULL) { + throw std::invalid_argument("PredictorSeedExtensionKinetic requires a seed handler"); + } + return handler; +} + +// Do not add infinity sentinels or overflow the internal integer energy type. +E_type addEnergy(const E_type first, const E_type second) +{ + if (E_isINF(first) || E_isINF(second)) { + return E_INF; + } + const std::int64_t sum = std::int64_t(first) + std::int64_t(second); + return sum >= E_INF || sum < std::numeric_limits::min() + ? E_INF : static_cast(sum); +} + +} // namespace + +////////////////////////////////////////////////////////////////////////// + +PredictorSeedExtensionKinetic::PredictorSeedExtensionKinetic( + const InteractionEnergy & energy, OutputHandler & output, + PredictionTracker * predTracker, SeedHandler * seedHandlerInstance, + const char score) + : PredictorMfe(energy, output, predTracker) + , seedHandler(checkedSeedHandler(seedHandlerInstance)) + , score(score) + , interactions() +{ + if (score != 'A' && score != 'B' && score != 'C') { + throw std::invalid_argument("PredictorSeedExtensionKinetic score must be A, B or C"); + } + if (output.getOutputConstraint().needZall) { + throw std::invalid_argument("PredictorSeedExtensionKinetic does not compute an equilibrium partition function"); + } +} + +////////////////////////////////////////////////////////////////////////// + +PredictorSeedExtensionKinetic::~PredictorSeedExtensionKinetic() +{ +} + +////////////////////////////////////////////////////////////////////////// + +void +PredictorSeedExtensionKinetic::predict(const IndexRange & r1, const IndexRange & r2) +{ + const size_t size1 = energy.getAccessibility1().getSequence().size(); + const size_t size2 = energy.getAccessibility2().getSequence().size(); + if (!r1.isAscending() || !r2.isAscending() + || r1.from >= size1 || r2.from >= size2) { + throw std::invalid_argument("PredictorSeedExtensionKinetic::predict(): invalid sequence range"); + } + + energy.setOffset1(r1.from); + energy.setOffset2(r2.from); + seedHandler.setOffset1(r1.from); + seedHandler.setOffset2(r2.from); + const size_t last1 = std::min(r1.to, size1 - 1) - r1.from; + const size_t last2 = std::min(r2.to, size2 - 1) - r2.from; + interactions.clear(); + initOptima(); + + if (seedHandler.fillSeed(0, last1, 0, last2) != 0) { + size_t i1 = RnaSequence::lastPos, i2 = RnaSequence::lastPos; + while (seedHandler.updateToNextSeed(i1, i2, 0, last1, 0, last2)) { + const size_t length1 = seedHandler.getSeedLength1(i1, i2); + const size_t length2 = seedHandler.getSeedLength2(i1, i2); + if (length1 == 0 || length2 == 0 + || length1 - 1 > last1 - i1 || length2 - 1 > last2 - i2 + || length1 > energy.getAccessibility1().getMaxLength() + || length2 > energy.getAccessibility2().getMaxLength()) { + continue; + } + const size_t j1 = i1 + length1 - 1; + const size_t j2 = i2 + length2 - 1; + if (E_isINF(seedHandler.getSeedE(i1, i2))) { + continue; + } + + Interaction interaction(energy.getAccessibility1().getSequence(), + energy.getAccessibility2().getAccessibilityOrigin().getSequence()); + interaction.basePairs.push_back(energy.getBasePair(i1, i2)); + seedHandler.traceBackSeed(interaction, i1, i2); + if (i1 != j1 || i2 != j2) { + interaction.basePairs.push_back(energy.getBasePair(j1, j2)); + } + interaction.sort(); + const E_type hybrid = addEnergy(seedHandler.getSeedE(i1, i2), energy.getE_init()); + interaction.energy = energy.getE(i1, j1, i2, j2, hybrid); + if (E_isINF(interaction.energy)) { + continue; + } + interaction.setSeedRange(interaction.basePairs.front(), + interaction.basePairs.back(), interaction.energy); + extendSeed(interaction, hybrid, last1, last2); + } + } + + // Reduce identical boundaries before updating optima: an earlier, inferior + // path must never be traced back using a later replacement's base pairs. + for (const auto & entry : interactions) { + const Boundary & b = entry.first; + updateOptima(b[0], b[1], b[2], b[3], entry.second.energy, false, false); + } + // The generic reporter assumes a nonempty optimum list. Zero reports can + // still be useful for prediction trackers and must not dereference it. + if (output.getOutputConstraint().reportMax != 0) { + reportOptima(); + } +} + +////////////////////////////////////////////////////////////////////////// + +PredictorSeedExtensionKinetic::Boundary +PredictorSeedExtensionKinetic::getBoundary(const Interaction & interaction) const +{ + return Boundary{energy.getIndex1(interaction.basePairs.front()), + energy.getIndex1(interaction.basePairs.back()), + energy.getIndex2(interaction.basePairs.front()), + energy.getIndex2(interaction.basePairs.back())}; +} + +////////////////////////////////////////////////////////////////////////// + +bool +PredictorSeedExtensionKinetic::prune(const Candidate &, const Boundary &) const +{ + return false; +} + +////////////////////////////////////////////////////////////////////////// + +void +PredictorSeedExtensionKinetic::extendSeed(Interaction & interaction, + E_type hybrid, const size_t last1, const size_t last2) +{ + std::array sides; + Boundary bounds = getBoundary(interaction); + buildCandidates(sides[0], bounds, true, last1, last2); + buildCandidates(sides[1], bounds, false, last1, last2); + while (true) { + retain(interaction); + const Candidate * left = updateCandidates(sides[0], bounds, hybrid, interaction.energy); + const Candidate * right = updateCandidates(sides[1], bounds, hybrid, interaction.energy); + if (left == NULL && right == NULL) { + break; + } + const Candidate best = left != NULL && (right == NULL || isBetter(*left, *right)) ? *left : *right; + const Interaction::BasePair close = energy.getBasePair(best.close1, best.close2); + if (best.left) { + interaction.basePairs.insert(interaction.basePairs.begin(), close); + if (best.macro) { + interaction.basePairs.insert(interaction.basePairs.begin(), + energy.getBasePair(best.bounds[0], best.bounds[2])); + } + } else { + interaction.basePairs.push_back(close); + if (best.macro) { + interaction.basePairs.push_back(energy.getBasePair(best.bounds[1], best.bounds[3])); + } + } + hybrid = best.hybrid; + interaction.energy = best.total; + bounds = best.bounds; + // The opposite end keeps its geometry, pair checks and loop energies. + // Its full energy must still be refreshed (ED and BOTH dangles change). + buildCandidates(sides[0], bounds, true, last1, last2); + buildCandidates(sides[1], bounds, false, last1, last2); + } +} + +////////////////////////////////////////////////////////////////////////// + +void +PredictorSeedExtensionKinetic::buildCandidates(SideCandidates & side, + const Boundary & bounds, const bool left, const size_t last1, const size_t last2) const +{ + side.moves.clear(); + const size_t space1 = std::min(energy.getAccessibility1().getMaxLength() - (bounds[1]-bounds[0]+1), + left ? bounds[0] : last1-bounds[1]); + const size_t space2 = std::min(energy.getAccessibility2().getMaxLength() - (bounds[3]-bounds[2]+1), + left ? bounds[2] : last2-bounds[3]); + if (space1 == 0 || space2 == 0) { + return; + } + const bool stackOnly = (output.getOutputConstraint().noGUend || !energy.isInternalLoopGUallowed()) + && energy.isGU(bounds[left ? 0 : 1], bounds[left ? 2 : 3]); + const size_t maxGap1 = space1 < 2 || stackOnly ? 0 : std::min(energy.getMaxInternalLoopSize1(), space1-2); + const size_t maxGap2 = space2 < 2 || stackOnly ? 0 : std::min(energy.getMaxInternalLoopSize2(), space2-2); + side.columns = maxGap2+2; + side.complementary.assign((maxGap1+2)*side.columns, -1); + const auto append = [&](size_t s1, size_t s2, bool macro) { + Candidate c; + c.left = left; c.s1 = s1; c.s2 = s2; c.macro = macro; + c.bounds = bounds; + const size_t pairs = macro ? 2 : 1; + if (left) { + c.close1 = bounds[0]-s1-1; c.close2 = bounds[2]-s2-1; + c.bounds[0] -= s1+pairs; c.bounds[2] -= s2+pairs; + } else { + c.close1 = bounds[1]+s1+1; c.close2 = bounds[3]+s2+1; + c.bounds[1] += s1+pairs; c.bounds[3] += s2+pairs; + } + side.moves.push_back(c); + }; + append(0, 0, false); + if (space1 >= 2 && space2 >= 2) { + for (size_t s1 = 0; s1 <= maxGap1; ++s1) { + for (size_t s2 = 0; s2 <= maxGap2; ++s2) { + append(s1, s2, true); + } + } + } +} + +////////////////////////////////////////////////////////////////////////// + +const PredictorSeedExtensionKinetic::Candidate * +PredictorSeedExtensionKinetic::updateCandidates(SideCandidates & side, + const Boundary & bounds, const E_type hybrid, const E_type total) const +{ + // Phase one: each position pair is tested at most once per unchanged end, + // even when it is the closing pair of one move and outer pair of another. + size_t stopGap2 = std::numeric_limits::max(); + for (Candidate & c : side.moves) { + c.bounds[c.left ? 1 : 0] = bounds[c.left ? 1 : 0]; + c.bounds[c.left ? 3 : 2] = bounds[c.left ? 3 : 2]; + c.active = c.bounds[1]-c.bounds[0]+1 <= energy.getAccessibility1().getMaxLength() + && c.bounds[3]-c.bounds[2]+1 <= energy.getAccessibility2().getMaxLength(); + if (c.active && c.macro) { + if (c.s2 >= stopGap2) { + c.active = false; + } else if (prune(c, bounds)) { + // A suffix bound rejects this rectangle of larger gaps without + // further ED, complementarity or loop-energy lookups. + stopGap2 = c.s2; + c.active = false; + } + } + if (!c.active || c.topologyKnown) { + continue; + } + const auto complementary = [&](size_t s1, size_t s2) { + signed char & cached = side.complementary[s1*side.columns+s2]; + if (cached < 0) { + cached = energy.areComplementary(c.left ? bounds[0]-s1-1 : bounds[1]+s1+1, + c.left ? bounds[2]-s2-1 : bounds[3]+s2+1); + } + return cached != 0; + }; + c.topologyKnown = true; + c.topologyAllowed = complementary(c.s1, c.s2) + && (!c.macro || complementary(c.s1+1, c.s2+1)); + if (c.topologyAllowed && output.getOutputConstraint().noGUend && (c.s1 != 0 || c.s2 != 0)) { + c.topologyAllowed = !energy.isGU(c.close1, c.close2); + } + } + const Candidate * best = NULL; + for (Candidate & c : side.moves) { + if (!c.active || !c.topologyAllowed) { + continue; + } + if (!c.localKnown) { + c.localKnown = true; + c.local = c.left ? energy.getE_interLeft(c.close1, bounds[0], c.close2, bounds[2]) + : energy.getE_interLeft(bounds[1], c.close1, bounds[3], c.close2); + if (c.macro && E_isNotINF(c.local)) { + c.local = addEnergy(c.local, c.left + ? energy.getE_interLeft(c.bounds[0], c.close1, c.bounds[2], c.close2) + : energy.getE_interLeft(c.close1, c.bounds[1], c.close2, c.bounds[3])); + } + } + c.hybrid = addEnergy(hybrid, c.local); + if (E_isINF(c.hybrid)) { + continue; + } + c.total = energy.getE(c.bounds[0], c.bounds[1], c.bounds[2], c.bounds[3], c.hybrid); + if (E_isINF(c.total)) { + continue; + } + c.delta = std::int64_t(c.total)-std::int64_t(total); + if (c.delta < 0 && (best == NULL || isBetter(c, *best))) { + best = &c; + } + } + return best; +} + +////////////////////////////////////////////////////////////////////////// + +bool +PredictorSeedExtensionKinetic::isBetter(const Candidate & candidate, const Candidate & best) const +{ + // A 32-bit energy difference times a 65-bit gap denominator fits in + // 128 bits, including for public-API loop limits beyond the CLI limits. + using Wide = boost::multiprecision::int128_t; + const auto denominator = [this](const Candidate & c) -> Wide { + if (score == 'B') { + return Wide(1) + Wide(c.s1) + Wide(c.s2); + } + if (score == 'C') { + return Wide(1) + 2 * Wide(std::max(c.s1, c.s2)); + } + return Wide(1); + }; + const Wide lhs = Wide(candidate.delta) * denominator(best); + const Wide rhs = Wide(best.delta) * denominator(candidate); + if (lhs != rhs) { + return lhs < rhs; + } + if (candidate.left != best.left) { + return candidate.left; + } + const Wide size = Wide(candidate.s1) + Wide(candidate.s2); + const Wide bestSize = Wide(best.s1) + Wide(best.s2); + if (size != bestSize) return size < bestSize; + if (candidate.s1 != best.s1) return candidate.s1 < best.s1; + // Identical shape/score: retain the shorter move first. + return !candidate.macro && best.macro; +} + +////////////////////////////////////////////////////////////////////////// + +void +PredictorSeedExtensionKinetic::retain(const Interaction & interaction) +{ + const Boundary b = getBoundary(interaction); + const auto & constraint = output.getOutputConstraint(); + if (interaction.energy >= E_MAX + || (constraint.noGUend && (energy.isGU(b[0], b[2]) || energy.isGU(b[1], b[3]))) + || energy.getED1(b[0], b[1]) > constraint.maxED + || energy.getED2(b[2], b[3]) > constraint.maxED) { + return; + } + auto existing = interactions.find(b); + if (existing == interactions.end()) { + interactions.emplace(b, interaction); + } else if (interaction.energy < existing->second.energy + || (interaction.energy == existing->second.energy + && interaction.basePairs < existing->second.basePairs)) { + existing->second = interaction; + } +} + +////////////////////////////////////////////////////////////////////////// + +void +PredictorSeedExtensionKinetic::traceBack(Interaction & interaction) +{ + if (interaction.basePairs.empty()) { + return; + } + const auto path = interactions.find(getBoundary(interaction)); + if (path == interactions.end() || path->second.energy != interaction.energy) { + throw std::runtime_error("PredictorSeedExtensionKinetic::traceBack(): no matching greedy path"); + } + interaction = path->second; + seedHandler.addSeeds(interaction); +} + +////////////////////////////////////////////////////////////////////////// + +void +PredictorSeedExtensionKinetic::getNextBest(Interaction & interaction) +{ + const E_type previousEnergy = interaction.energy; + const Interaction * best = NULL; + for (const auto & entry : interactions) { + const Boundary & b = entry.first; + const Interaction & candidate = entry.second; + if (candidate.energy < previousEnergy + || reportedInteractions.first.overlaps(IndexRange(b[0], b[1])) + || reportedInteractions.second.overlaps(IndexRange(b[2], b[3]))) { + continue; + } + if (best == NULL || candidate < *best) { + best = &candidate; + } + } + if (best == NULL) { + interaction.clear(); + interaction.energy = E_INF; + return; + } + interaction = *best; + const Interaction::BasePair right = interaction.basePairs.back(); + interaction.basePairs.resize(interaction.basePairs.size() == 1 ? 1 : 2); + interaction.basePairs.back() = right; + INTARNA_CLEANUP(interaction.seed); +} + +} // namespace IntaRNA diff --git a/doc/local-validation-archive/files/kinetic-validation/verify-installed-kix.py b/doc/local-validation-archive/files/kinetic-validation/verify-installed-kix.py new file mode 100644 index 00000000..cfcb21c6 --- /dev/null +++ b/doc/local-validation-archive/files/kinetic-validation/verify-installed-kix.py @@ -0,0 +1,29 @@ +import os +from pathlib import Path +import shlex +import subprocess +root = Path('/home/alex/CodexProjects/IntaRNA') +validation = root/'kinetic-validation' +prefix = validation/'kix-stage'/str(validation/'install-release').lstrip('/') +toolchain = root/'issue-246-validation/toolchain' +cxx = root/'issue-246-validation/toolchain-gcc14/bin/x86_64-conda-linux-gnu-g++' +env = dict(os.environ, PKG_CONFIG_PATH=str(prefix/'lib/pkgconfig')+':'+str(toolchain/'lib/pkgconfig')) +pc = ['pkg-config', '--define-variable=prefix='+str(prefix)] +cflags = shlex.split(subprocess.check_output([*pc,'--cflags','IntaRNA'],env=env,text=True)) +libs = shlex.split(subprocess.check_output([*pc,'--libs','IntaRNA'],env=env,text=True)) +assert not (prefix/'include/IntaRNA/PredictorSeedExtensionKineticPruned.h').exists() +assert (prefix/'bin/IntaRNAkix').is_symlink() +for header in ['PredictorSeedExtensionKinetic.h','InteractionEnergyVrna.h']: + source = validation/'header-probe-kix.cpp' + source.write_text('#include \nint main() { return 0; }\n') + subprocess.run([str(cxx),'-std=c++23','-fsyntax-only',*cflags,str(source)],check=True) +subprocess.run([str(cxx),'-std=c++23',*cflags,str(validation/'consumer.cpp'),'-L'+str(toolchain/'lib'), + '-Wl,-rpath,'+str(toolchain/'lib'),*libs,'-leasylogging','-o',str(validation/'consumer-kix')],check=True) +subprocess.run([str(validation/'consumer-kix')],check=True) +args = ['--target=GGGGGG','--query=CCCCCC','--energy=B','--acc=N', + '--seedTQ=3||&3||','--outNumber=10','--outMode=C','--outCsvCols=hybridDB,E','--default-log-file=/dev/null'] +actual = subprocess.check_output([str(prefix/'bin/IntaRNAkix'),*args],text=True) +assert actual == 'hybridDB;E\n1||||||&1||||||;-6\n1||||&3||||;-4\n3||&3||;-2\n', actual +helptext = subprocess.check_output([str(prefix/'bin/IntaRNAkix'),'--fullhelp'],text=True) +assert 'IntaRNAkix' in helptext +print('Installed K/Vrna headers compile independently; installed K consumer links and predicts -400; installed IntaRNAkix link reports the expected prefixes; removed header absent.') diff --git a/doc/local-validation-archive/files/kinetic-validation/verify-installed-review.py b/doc/local-validation-archive/files/kinetic-validation/verify-installed-review.py new file mode 100644 index 00000000..42fba534 --- /dev/null +++ b/doc/local-validation-archive/files/kinetic-validation/verify-installed-review.py @@ -0,0 +1,22 @@ +import os +from pathlib import Path +import shlex +import subprocess +root = Path('/home/alex/CodexProjects/IntaRNA') +validation = root/'kinetic-validation' +prefix = validation/'install-release' +toolchain = root/'issue-246-validation/toolchain' +cxx = root/'issue-246-validation/toolchain-gcc14/bin/x86_64-conda-linux-gnu-g++' +env = dict(os.environ, PKG_CONFIG_PATH=str(prefix/'lib/pkgconfig')+':'+str(toolchain/'lib/pkgconfig')) +cflags = shlex.split(subprocess.check_output(['pkg-config','--cflags','IntaRNA'],env=env,text=True)) +libs = shlex.split(subprocess.check_output(['pkg-config','--libs','IntaRNA'],env=env,text=True)) +for header in ['PredictorSeedExtensionKinetic.h','PredictorSeedExtensionKineticPruned.h','InteractionEnergyVrna.h']: + source = validation/'header-probe-review.cpp' + source.write_text('#include \nint main() { return 0; }\n') + subprocess.run([str(cxx),'-std=c++23','-fsyntax-only',*cflags,str(source)],check=True) +source = validation/'consumer-review.cpp' +source.write_text((validation/'consumer.cpp').read_text().replace('PredictorSeedExtensionKinetic','PredictorSeedExtensionKineticPruned')) +subprocess.run([str(cxx),'-std=c++23',*cflags,str(source),'-L'+str(toolchain/'lib'), + '-Wl,-rpath,'+str(toolchain/'lib'),*libs,'-leasylogging','-o',str(validation/'consumer-review')],check=True) +subprocess.run([str(validation/'consumer-review')],check=True) +print('Three installed headers compile standalone; installed pruned predictor links and predicts -400.') diff --git a/doc/local-validation-archive/files/profiling-default-20260921/scripts/audit.py b/doc/local-validation-archive/files/profiling-default-20260921/scripts/audit.py new file mode 100644 index 00000000..d753c2cd --- /dev/null +++ b/doc/local-validation-archive/files/profiling-default-20260921/scripts/audit.py @@ -0,0 +1,115 @@ +#!/usr/bin/env python3 +"""Independently audit saved evidence; never runs IntaRNA or rewrites measurements.""" +import csv,hashlib,io,json,math,statistics +from collections import Counter +from datetime import datetime,timezone +from pathlib import Path +ROOT=Path(__file__).resolve().parents[1] +RESULT=ROOT/'results' +errors=[];warnings=[];findings=[] +def expect(condition,message): + if not condition:errors.append(message) +def digest(data):return hashlib.sha256(data).hexdigest() +def load(path):return json.loads(path.read_text()) +def records(path):return [json.loads(line) for line in path.read_text().splitlines()] +def normalized(data):return b'\n'.join(sorted(x for x in data.splitlines() if x and not x.startswith(b'#') and not x.startswith(b'Creating experiment directory ')))+b'\n' +def equal(a,b):return math.isclose(a,b,rel_tol=1e-10,abs_tol=1e-10) if isinstance(a,(int,float)) and isinstance(b,(int,float)) else a==b + +def main(): + plan=load(ROOT/'study-plan.json');cases=load(ROOT/'cases.json');env=load(ROOT/'environment.json');runs=records(RESULT/'runs.jsonl');summary=load(RESULT/'summary.json') + definitions={x['name']:x for x in cases};lookup={x['case']:x for x in summary} + expect(len(definitions)==plan['cases']==18,'Case coverage') + expect(env['revision']==plan['revision'],'Revision provenance') + binary=ROOT/'source/src/bin/IntaRNA' + if binary.exists():expect(digest(binary.read_bytes())==env['binary_sha256'],'Executable SHA-256') + else:warnings.append('Compiled executable omitted; recorded per-run binary hashes are checked.') + expect(env['environment_controlled']==plan['controlled_environment'],'Controlled environment') + expect(set(plan['scientific_defaults'])=={'--model=X','--mode=H','--acc=C','--accW=150','--accL=100','--windowWidth=0','--seedBP=7','--seedMaxUP=0','--intLenMax=0'},'Requested default-mode scope') + allowed={'target','query','model','mode','acc','accW','accL','windowWidth','seedBP','seedMaxUP','intLenMax','threads','outMode','outCsvCols'} + for case in cases: + flags=[a.split('=',1)[0][2:] for a in case['args']] + expect(len(flags)==len(set(flags)) and set(flags)==allowed,case['name']+' parameter set') + expect(set(plan['scientific_defaults']).issubset(case['args']),case['name']+' scientific defaults') + checked_inputs=0 + for name,entry in load(ROOT/'inputs/manifest.json').items(): + expect(digest((ROOT/'inputs'/name).read_bytes())==entry['sha256'],'Input hash '+name);checked_inputs+=1 + expected={(c['name'],phase,rep) for c in cases for phase,rep in [('warmup',0)]+[('measure',n) for n in range(1,6)]} + observed=Counter((r['case'],r['phase'],r['repetition']) for r in runs) + expect(set(observed)==expected and all(v==1 for v in observed.values()),'Exactly one warmup plus five measured records per case') + end=None;out_hash={};structure_hashes={} + for r in runs: + name=f"{r['case']}.{r['phase']}{r['repetition']}";data=(RESULT/(name+'.stdout')).read_bytes();key=(r['case'],r['phase'],r['repetition']) + expect(r['exit_code']==0 and not r['timeout'],name+' exit status') + expect(r['binary_sha256']==env['binary_sha256'],name+' binary provenance') + expect(r['command'][6:]==definitions[r['case']]['args'],name+' command definition') + expect(digest(data)==r['output_sha256'],name+' stdout hash') + norm=digest(normalized(data));out_hash[key]=norm + expect(norm==r['canonical_sha256'],name+' canonical hash') + cells=list(csv.reader(io.StringIO('\n'.join(x for x in data.decode().splitlines() if x and not x.startswith('#'))),delimiter=';')) + expect(bool(cells) and cells[0]==['id1','id2','start1','end1','start2','end2','E','bpList'] and all(len(c)==8 for c in cells),name+' full-structure CSV') + expect(r['csv_valid'] and len(cells)-1==r['output_rows'],name+' derived metadata') + structure_hashes[key]=digest(json.dumps(sorted(tuple(row[:6]+row[7:]) for row in cells[1:])).encode()) + expect(not (RESULT/(name+'.stderr')).read_bytes(),name+' empty stderr') + vals=list(map(float,(RESULT/(name+'.time')).read_text().strip().split('\t'))) + for field,value in zip(['wall_s','user_s','system_s','max_rss_kib','major_faults','minor_faults','voluntary_context_switches','involuntary_context_switches','fs_inputs','fs_outputs','time_exit_code'],vals):expect(equal(value,r[field]),name+' GNU time '+field) + expect(r['elapsed_s']>0 and -0.02<=r['elapsed_s']-r['wall_s']<0.1,name+' monotonic/GNU wall agreement') + start=datetime.fromisoformat(r['started_utc']);stop=datetime.fromisoformat(r['ended_utc']) + expect(stop>=start and (end is None or start>=end),name+' sequential timing');end=stop + for case in cases: + group=[r for r in runs if r['case']==case['name']];measured=[r for r in group if r['phase']=='measure'];times=[r['elapsed_s'] for r in measured];saved=lookup[case['name']] + stable_all=len({out_hash[(r['case'],r['phase'],r['repetition'])] for r in group})==1 + if not stable_all:findings.append({'kind':'within_case_output_instability','case':case['name'],'records':[f"{r['case']}.{r['phase']}{r['repetition']}.stdout" for r in group]}) + expected_values={'n':5,'median_s':statistics.median(times),'min_s':min(times),'max_s':max(times),'range_over_median_pct':100*(max(times)-min(times))/statistics.median(times),'median_user_s':statistics.median(r['user_s'] for r in measured),'median_system_s':statistics.median(r['system_s'] for r in measured),'median_rss_mib':statistics.median(r['max_rss_kib']/1024 for r in measured),'peak_rss_mib':max(r['max_rss_kib']/1024 for r in measured),'stable_output':len({r['canonical_sha256'] for r in measured})==1,'stable_bpList':len({structure_hashes[(r['case'],r['phase'],r['repetition'])] for r in measured})==1,'reported_rows':measured[0]['output_rows'],'case':case['name'],'category':case['category'],'threads':case['threads'],'predictions':case['predictions']} + expect(set(saved)==set(expected_values),case['name']+' summary fields') + for k,v in expected_values.items():expect(equal(saved[k],v),case['name']+' summary '+k) + csv_rows=list(csv.DictReader((RESULT/'summary.csv').open())) + expect(len(csv_rows)==len(summary),'CSV summary coverage') + for row in csv_rows: + for k,v in lookup[row['case']].items():expect(row[k]==str(v),row['case']+' CSV '+k) + thread_equality={};thread_structure_equality={} + for rna in ['gcvb','chix']: + keys=[key for key in out_hash if key[0] in {f'batch_{rna}_{t}t' for t in [1,6,12]}] + thread_equality[rna]=len({out_hash[key] for key in keys})==1 + thread_structure_equality[rna]=len({structure_hashes[key] for key in keys})==1 + if not thread_equality[rna]:findings.append({'kind':'thread_count_output_mismatch','query':rna,'coordinates_and_bpList_equal':thread_structure_equality[rna]}) + diag=load(ROOT/'diagnostics/checks.json') + expect(all(r['explicit_defaults_equal_implicit_defaults_bpList'] for r in diag['implicit_default_checks']),'Explicit vs implicit defaults') + for r in records(ROOT/'diagnostics/commands.jsonl'): + data=(ROOT/'diagnostics'/(r['name']+'.stdout')).read_bytes() + expect(digest(data)==r['stdout_sha256'] and digest(normalized(data))==r['canonical_sha256'],r['name']+' diagnostic hashes') + for name,entry in diag['ed_files'].items():expect(digest((ROOT/'diagnostics/ed'/(name+'.ed')).read_bytes())==entry['sha256'],name+' ED hash') + profile_checks=load(ROOT/'profiles/checks.json') + for name,entry in load(ROOT/'profiles/input-manifest.json').items(): + expect(digest((ROOT/'inputs'/name).read_bytes())==entry['sha256'],'Profile input hash '+name) + expect(Counter(r['kind'] for r in profile_checks)=={'cpu':4,'heap':2,'phases':3},'Profile coverage') + expect(all(r['output_equal_bpList'] for r in profile_checks),'All profiler/phase outputs retain bpList') + for r in records(ROOT/'profiles/commands.jsonl'): + expect(r['exit_code']==0 and datetime.fromisoformat(r['started_utc'])>=end,r['name']+' profile after clean timing') + # Independently re-check all nine instrumentation outputs against preserved raw references. + for r in profile_checks: + name=r['profile'];kind=r['kind'];suffix={'cpu':'cpu','heap':'heap','phases':'phases'}[kind] + if name=='chix384':reference=(ROOT/'profiles/chix384_reference.stdout').read_bytes() + else: + case={'gcvb':'batch_gcvb_1t','gcvb_repeat':'batch_gcvb_1t'}.get(name,name) + reference=(RESULT/(case+'.measure1.stdout')).read_bytes() + actual=(ROOT/'profiles'/(name+'_'+suffix+'.stdout')).read_bytes() + expect(normalized(actual)==normalized(reference),name+' independent '+kind+' output check') + parallel=load(ROOT/'diagnostics/parallel/summary.json') + repeat_rows=records(ROOT/'diagnostics/parallel/checks.jsonl') + expect(Counter(r['threads'] for r in repeat_rows)=={1:3,6:5,12:20},'Follow-up repeat coverage') + repeat_reference=normalized((RESULT/'batch_chix_1t.measure1.stdout').read_bytes()) + for r in repeat_rows: + data=(ROOT/'diagnostics/parallel'/(r['name']+'.stdout')).read_bytes() + expect(digest(data)==r['stdout_sha256'] and r['binary_sha256']==env['binary_sha256'],r['name']+' repeat provenance') + expect(r['output_equal']==(normalized(data)==repeat_reference and r['exit_code']==0),r['name']+' repeat comparison') + for threads in [1,6,12]: + expected_counts={'runs':sum(r['threads']==threads for r in repeat_rows),'different':sum(r['threads']==threads and not r['output_equal'] for r in repeat_rows)} + expect(parallel['counts'][str(threads)]==expected_counts,'Repeat counts '+str(threads)) + for r in load(ROOT/'diagnostics/numeric-commands.json'): + expect(r['exit_code']==0 and digest((ROOT/'diagnostics'/(r['name']+'.stdout')).read_bytes())==r['stdout_sha256'],r['name']+' energy-component evidence') + if parallel['observed_differences']:findings.append({'kind':'parallel_output_instability_reproduced','counts':parallel['counts'],'observed_differences':[{k:r[k] for k in ['name','threads','differences']} for r in parallel['observed_differences']]}) + if not diag['matched_149_99_comparison']['full_output_equal']:findings.append({'kind':'matched_ED_output_mismatch','comparison':diag['matched_149_99_comparison']}) + report={'audited_utc':datetime.now(timezone.utc).isoformat(),'status':('pass_with_findings' if findings else 'pass') if not errors else 'fail','integrity_status':'pass' if not errors else 'fail','scientific_output_status':'differences_observed' if findings else 'all_checked_equal','scientific_findings':findings,'errors':errors,'warnings':warnings,'cases':len(cases),'measured_runs':sum(r['phase']=='measure' for r in runs),'warmups':sum(r['phase']=='warmup' for r in runs),'input_hashes_checked':checked_inputs,'profile_checks':len(profile_checks),'thread_equality_full_output':thread_equality,'thread_equality_coordinates_and_bpList':thread_structure_equality,'default_equivalence_cases':len(diag['implicit_default_checks']),'additional_repeat_checks':len(repeat_rows)} + (RESULT/'audit.json').write_text(json.dumps(report,indent=2)+'\n');print(json.dumps(report,indent=2)) + raise SystemExit(bool(errors)) +if __name__=='__main__':main() diff --git a/doc/local-validation-archive/files/profiling-default-20260921/scripts/benchmark.py b/doc/local-validation-archive/files/profiling-default-20260921/scripts/benchmark.py new file mode 100644 index 00000000..f09517e9 --- /dev/null +++ b/doc/local-validation-archive/files/profiling-default-20260921/scripts/benchmark.py @@ -0,0 +1,131 @@ +#!/usr/bin/env python3 +"""Default-mode study: fixed scientific defaults, sequential randomized timing.""" +import argparse,csv,hashlib,io,json,os,platform,random,signal,statistics,subprocess,time,threading +from datetime import datetime,timezone +from pathlib import Path +ROOT=Path(__file__).resolve().parents[1] +BIN=ROOT/'source/src/bin/IntaRNA' +INPUT=ROOT/'inputs';RESULT=ROOT/'results' +REVISION='0a4568ad6e3da52f935221ff0da98a4a171e9833' +CONTROL={'LC_ALL':'C','OPENBLAS_NUM_THREADS':'1','OMP_DYNAMIC':'FALSE','OMP_PROC_BIND':'close','OMP_PLACES':'cores'} +ENV=dict(os.environ,**CONTROL) +DEFAULTS=['--model=X','--mode=H','--acc=C','--accW=150','--accL=100','--windowWidth=0','--seedBP=7','--seedMaxUP=0','--intLenMax=0'] +HEADER=['id1','id2','start1','end1','start2','end2','E','bpList'] +TIME_KEYS=['wall_s','user_s','system_s','max_rss_kib','major_faults','minor_faults','voluntary_context_switches','involuntary_context_switches','fs_inputs','fs_outputs','time_exit_code'] +def sha(p):return hashlib.sha256(Path(p).read_bytes()).hexdigest() +def dump(p,x):p.parent.mkdir(parents=True,exist_ok=True);p.write_text(json.dumps(x,indent=2)+'\n') +def fasta(p): + records=[] + for line in Path(p).read_text().splitlines(): + if line.startswith('>'):records.append([line[1:],'']) + elif line.strip():records[-1][1]+=line.strip().upper().replace('T','U') + return records + +def write_fasta(name,records): + p=INPUT/(name+'.fa');p.write_text(''.join('>'+n+'\n'+s+'\n' for n,s in records));return p + +def canonical(data):return b'\n'.join(sorted(line for line in data.splitlines() if line and not line.startswith(b'#') and not line.startswith(b'Creating experiment directory ')))+b'\n' +def validate(data): + lines=[l for l in data.decode().splitlines() if l and not l.startswith('#') and not l.startswith('Creating experiment directory ')] + rows=list(csv.reader(lines,delimiter=';')) + return bool(rows) and rows[0]==HEADER and all(len(r)==len(HEADER) for r in rows),max(0,len(rows)-1) + +def structure_hash(data): + text='\n'.join(x for x in data.decode().splitlines() if x and not x.startswith('#')) + rows=csv.DictReader(io.StringIO(text),delimiter=';') + values=sorted(tuple(r[k] for k in HEADER if k!='E') for r in rows) + return hashlib.sha256(json.dumps(values).encode()).hexdigest() + +def prepare(): + INPUT.mkdir(exist_ok=True);RESULT.mkdir(exist_ok=True) + docs=ROOT/'source/doc/handson' + for name in ['fhlA','OxyS','phoB','GcvB','GcvB.ST','ilvE','ChiX_NC_000913']: + write_fasta(name,fasta(docs/(name+'.fasta'))) + records=fasta(docs/'NC_000913.fa');indices=[i*(len(records)-1)//47 for i in range(48)] + write_fasta('promoters48_systematic',[records[i] for i in indices]) + dump(INPUT/'selection.json',{'source':'doc/handson/NC_000913.fa','source_sha256':sha(docs/'NC_000913.fa'),'total_records':len(records),'indices_zero_based':indices,'rule':'48 evenly spaced record indices including endpoints; no outcome-based selection','note':'Bounded sample, not the full target collection; differs from the first-48 sample used on 2026-09-19.'}) + rng=random.Random(20260921) + target=''.join(rng.choices('ACGU',k=10000));query=''.join(rng.choices('ACGU',k=800)) + for n in [100,300,1000,3000,10000]:write_fasta(f't{n}',[(f'synthetic_target_{n}',target[:n])]) + for n in [100,200,400,800]:write_fasta(f'q{n}',[(f'synthetic_query_{n}',query[:n])]) + cases=[] + def add(name,category,t,q,threads=1): + args=['--target='+str(INPUT/(t+'.fa')),'--query='+str(INPUT/(q+'.fa'))]+DEFAULTS+['--threads='+str(threads),'--outMode=C','--outCsvCols='+','.join(HEADER)] + cases.append({'name':name,'category':category,'target':t,'query':q,'threads':threads,'predictions':len(fasta(INPUT/(t+'.fa')))*len(fasta(INPUT/(q+'.fa'))),'args':args}) + for t,q in [('fhlA','OxyS'),('phoB','GcvB'),('ilvE','GcvB.ST')]:add('bio_'+t.lower(),'single_biological',t,q) + for n in [100,300,1000,3000,10000]:add(f'length_t{n}_q100','target_length',f't{n}','q100') + for n in [200,400,800]:add(f'length_t300_q{n}','query_length','t300',f'q{n}') + add('long_pair','long_pair','t3000','q800') + for query,name in [('GcvB','gcvb'),('ChiX_NC_000913','chix')]: + for threads in [1,6,12]:add(f'batch_{name}_{threads}t','biological_batch','promoters48_systematic',query,threads) + dump(ROOT/'cases.json',cases) + dump(INPUT/'manifest.json',{p.name:{'sha256':sha(p),'records':[{'id':n,'length':len(s),'gc_fraction':(s.count('G')+s.count('C'))/len(s)} for n,s in fasta(p)]} for p in sorted(INPUT.glob('*.fa'))}) + dump(ROOT/'study-plan.json',{'revision':REVISION,'cases':len(cases),'measured_repetitions':5,'excluded_warmups_per_case':1,'random_seed':20260921,'scientific_defaults':DEFAULTS,'controlled_environment':CONTROL,'output':'CSV including full bpList, retaining traceback; output formatting differs from CLI default','profiles':['batch_gcvb_1t','batch_chix_1t repeated 8x as one batch','long_pair'],'limitations':['One host; bounded biological sample','No comparison with historical revision','No algorithm/window/accessibility-off sweep','Diagnostics and sampling run after clean timings']}) + print('Prepared',len(cases),'cases',flush=True) + +def capture(cmd): + p=subprocess.run(cmd,capture_output=True,text=True,env=ENV);return {'returncode':p.returncode,'output':p.stdout+p.stderr} + +def environment(): + deps=Path(os.environ.get('INTARNA_DEPS','/home/alex/CursorProjects/IntaRNA_optimization/intaRNA_legacy/.conda-env')) + limits={} + cg=Path('/sys/fs/cgroup'+Path('/proc/self/cgroup').read_text().split('::',1)[1].strip()) + for parent in [cg,*cg.parents]: + if not parent.is_relative_to('/sys/fs/cgroup'):continue + for n in ['cpu.max','memory.max','cpuset.cpus.effective']: + p=parent/n + if p.exists():limits[str(p)]=p.read_text().strip() + dump(ROOT/'environment.json',{'recorded_utc':datetime.now(timezone.utc).isoformat(),'revision':REVISION,'binary_sha256':sha(BIN),'platform':platform.platform(),'version':capture([str(BIN),'--version']),'cpu':capture(['lscpu']),'compiler':capture([str(deps/'bin/x86_64-conda-linux-gnu-g++'),'--version']),'linked_libraries':capture(['ldd',str(BIN)]),'environment_controlled':CONTROL,'affinity':sorted(os.sched_getaffinity(0)),'load_average':os.getloadavg(),'memory':Path('/proc/meminfo').read_text(),'governors':{str(p):p.read_text().strip() for p in Path('/sys/devices/system/cpu').glob('cpu*/cpufreq/scaling_governor')},'perf_event_paranoid':Path('/proc/sys/kernel/perf_event_paranoid').read_text().strip(),'cgroup_limits':limits}) + +def run(case,phase,repetition,timeout=240): + assert all(x in case['args'] for x in DEFAULTS) + dest=RESULT/f"{case['name']}.{phase}{repetition}" + if dest.with_suffix(dest.suffix+'.stdout').exists():raise RuntimeError('Refusing to overwrite '+str(dest)) + output=Path(str(dest)+'.stdout');err=Path(str(dest)+'.stderr');timing=Path(str(dest)+'.time') + cmd=['/usr/bin/time','-f','%e\t%U\t%S\t%M\t%F\t%R\t%w\t%c\t%I\t%O\t%x','-o',str(timing),str(BIN)]+case['args'] + start=time.perf_counter();stamp=datetime.now(timezone.utc).isoformat();load=os.getloadavg();timed_out=False + with output.open('wb') as out,err.open('wb') as stderr: + p=subprocess.Popen(cmd,env=ENV,stdout=out,stderr=stderr,start_new_session=True) + def expire(): + nonlocal timed_out + if p.poll() is None: + timed_out=True + try:os.killpg(p.pid,signal.SIGKILL) + except ProcessLookupError:pass + watchdog=threading.Timer(timeout,expire);watchdog.daemon=True;watchdog.start() + p.wait();watchdog.cancel() + elapsed=time.perf_counter()-start + row={'case':case['name'],'category':case['category'],'phase':phase,'repetition':repetition,'command':cmd,'started_utc':stamp,'ended_utc':datetime.now(timezone.utc).isoformat(),'elapsed_s':elapsed,'exit_code':p.returncode,'timeout':timed_out,'load_before':load,'load_after':os.getloadavg(),'binary_sha256':sha(BIN),'output_sha256':sha(output),'canonical_sha256':hashlib.sha256(canonical(output.read_bytes())).hexdigest()} + values=timing.read_text().strip().splitlines() + if values and len(values[-1].split('\t'))==len(TIME_KEYS):row.update(dict(zip(TIME_KEYS,map(float,values[-1].split('\t'))))) + row['csv_valid'],row['output_rows']=validate(output.read_bytes()) + with (RESULT/'runs.jsonl').open('a') as f:f.write(json.dumps(row)+'\n') + print(f"{phase}{repetition} {case['name']}: {elapsed:.3f}s rc={p.returncode} RSS={row.get('max_rss_kib',0)/1024:.2f}MiB",flush=True) + if p.returncode or not row['csv_valid']:raise RuntimeError('Run failed '+str(dest)) + return row + +def summarize(): + records=[json.loads(line) for line in (RESULT/'runs.jsonl').read_text().splitlines()];summary=[] + for case in json.loads((ROOT/'cases.json').read_text()): + rows=[r for r in records if r['case']==case['name'] and r['phase']=='measure'];values=[r['elapsed_s'] for r in rows] + if not rows:continue + structural={structure_hash((RESULT/f"{r['case']}.{r['phase']}{r['repetition']}.stdout").read_bytes()) for r in rows} + summary.append({'case':case['name'],'category':case['category'],'threads':case['threads'],'predictions':case['predictions'],'n':len(rows),'median_s':statistics.median(values),'min_s':min(values),'max_s':max(values),'range_over_median_pct':100*(max(values)-min(values))/statistics.median(values),'median_user_s':statistics.median(r['user_s'] for r in rows),'median_system_s':statistics.median(r['system_s'] for r in rows),'median_rss_mib':statistics.median(r['max_rss_kib']/1024 for r in rows),'peak_rss_mib':max(r['max_rss_kib']/1024 for r in rows),'stable_output':len({r['canonical_sha256'] for r in rows})==1,'stable_bpList':len(structural)==1,'reported_rows':rows[0]['output_rows']}) + dump(RESULT/'summary.json',summary) + with (RESULT/'summary.csv').open('w',newline='') as f: + writer=csv.DictWriter(f,fieldnames=summary[0].keys(),lineterminator='\n');writer.writeheader();writer.writerows(summary) + return summary + +def main(): + p=argparse.ArgumentParser();p.add_argument('action',choices=['prepare','run','summarize','environment']);a=p.parse_args() + if a.action=='prepare':return prepare() + if a.action=='environment':return environment() + if a.action=='summarize':return summarize() + cases=json.loads((ROOT/'cases.json').read_text());environment() + for case in cases:run(case,'warmup',0) + rng=random.Random(20260921) + for repetition in range(1,6): + order=cases[:];rng.shuffle(order) + for case in order:run(case,'measure',repetition) + summarize() +if __name__=='__main__':main() diff --git a/doc/local-validation-archive/files/profiling-default-20260921/scripts/build.sh b/doc/local-validation-archive/files/profiling-default-20260921/scripts/build.sh new file mode 100644 index 00000000..f409d785 --- /dev/null +++ b/doc/local-validation-archive/files/profiling-default-20260921/scripts/build.sh @@ -0,0 +1,23 @@ +#!/usr/bin/env bash +set -euo pipefail +profile_root=$(cd "$(dirname "$0")/.." && pwd) +repo_root=${INTARNA_REPO:-"$profile_root/../intarna_optimization_supervised"} +dependency_root=${INTARNA_DEPS:-/home/alex/CursorProjects/IntaRNA_optimization/intaRNA_legacy/.conda-env} +revision=${INTARNA_REVISION:-0a4568ad6e3da52f935221ff0da98a4a171e9833} +if [[ -e "$profile_root/source" ]]; then + echo 'source already exists; use a new profiling directory to preserve evidence' >&2 + exit 1 +fi +mkdir -p "$profile_root/source" "$profile_root/logs" +git -C "$repo_root" archive "$revision" | tar -x -C "$profile_root/source" +cd "$profile_root/source" +autoreconf -fi > ../logs/autoreconf.log 2>&1 +./configure --with-boost="$dependency_root" --with-boost-libdir="$dependency_root/lib" \ + --with-vrna="$dependency_root" --prefix="$profile_root/install" \ + CXX="$dependency_root/bin/x86_64-conda-linux-gnu-g++" \ + CC="$dependency_root/bin/x86_64-conda-linux-gnu-gcc" \ + CXXFLAGS='-g -fno-omit-frame-pointer' CPPFLAGS="-I$dependency_root/include" \ + LDFLAGS="-L$dependency_root/lib -Wl,-rpath,$dependency_root/lib" \ + PKG_CONFIG_PATH="$dependency_root/lib/pkgconfig" > ../logs/configure.log 2>&1 +make -j4 V=1 > ../logs/build.log 2>&1 +make -j4 check > ../logs/check.log 2>&1 diff --git a/doc/local-validation-archive/files/profiling-default-20260921/scripts/diagnostics.py b/doc/local-validation-archive/files/profiling-default-20260921/scripts/diagnostics.py new file mode 100644 index 00000000..14c747d2 --- /dev/null +++ b/doc/local-validation-archive/files/profiling-default-20260921/scripts/diagnostics.py @@ -0,0 +1,51 @@ +#!/usr/bin/env python3 +"""Untimed default-equivalence checks and an ED writer/reader reproduction.""" +import json,subprocess,re,hashlib,csv,io +from pathlib import Path +from benchmark import ROOT,BIN,ENV,DEFAULTS,canonical,validate,dump,sha +DEST=ROOT/'diagnostics' +def compare_outputs(first,second): + def rows(name): + text='\n'.join(line for line in (DEST/(name+'.stdout')).read_text().splitlines() if line and not line.startswith('#')) + return sorted(csv.DictReader(io.StringIO(text),delimiter=';'),key=lambda row:(row['id1'],row['id2'],row['start1'],row['start2'])) + a,b=rows(first),rows(second) + differences=[{'row':i,'fields':{key:{'computed':x[key],'loaded':y[key]} for key in x if x[key]!=y[key]}} for i,(x,y) in enumerate(zip(a,b)) if x!=y] + structural=['id1','id2','start1','end1','start2','end2','bpList'] + return {'full_output_equal':a==b,'same_row_count':len(a)==len(b),'coordinates_and_bpList_equal':len(a)==len(b) and all(all(x[k]==y[k] for k in structural) for x,y in zip(a,b)),'differences':differences} +def execute(name,args): + command=[str(BIN)]+args + p=subprocess.run(command,env=ENV,capture_output=True,timeout=240) + (DEST/(name+'.stdout')).write_bytes(p.stdout);(DEST/(name+'.stderr')).write_bytes(p.stderr) + valid,rows=validate(p.stdout) + row={'name':name,'command':command,'exit_code':p.returncode,'csv_valid':valid,'reported_rows':rows,'stdout_sha256':hashlib.sha256(p.stdout).hexdigest(),'canonical_sha256':hashlib.sha256(canonical(p.stdout)).hexdigest()} + with (DEST/'commands.jsonl').open('a') as f:f.write(json.dumps(row)+'\n') + p.check_returncode();assert valid,name + return row + +def main(): + DEST.mkdir(exist_ok=True) + cases={x['name']:x for x in json.loads((ROOT/'cases.json').read_text())} + checks=[] + for name in ['bio_fhla','bio_phob','bio_ilve']: + row=execute(name+'_implicit',[a for a in cases[name]['args'] if a not in DEFAULTS]) + reference=ROOT/'results'/(name+'.measure1.stdout') + equal=row['canonical_sha256']==hashlib.sha256(canonical(reference.read_bytes())).hexdigest() + checks.append({'case':name,'explicit_defaults_equal_implicit_defaults_bpList':equal}) + assert equal,name + cache=DEST/'ed';cache.mkdir(exist_ok=True) + base=cases['length_t1000_q100']['args'] + generated=execute('ed_write',base+['--out=tAcc:'+str(cache/'target.ed'),'--out=qAcc:'+str(cache/'query.ed'),'--verbose']) + load_base=[a for a in base if a!='--acc=C']+['--tAcc=E','--tAccFile='+str(cache/'target.ed'),'--qAcc=E','--qAccFile='+str(cache/'query.ed')] + loaded=execute('ed_reload',load_base+['--verbose']) + rows={} + for name in ['target','query']: + p=cache/(name+'.ed');lines=p.read_text().splitlines();header=lines[1];maxlen=int(header.split()[-1]);data=[line.split() for line in lines[2:] if line.strip()] + rows[name]={'sha256':sha(p),'header':header,'last_header_length':maxlen,'data_rows':len(data),'fields_per_row':sorted({len(r) for r in data}),'numeric_final_column_rows':sum(r[-1] not in ['NA','nan','NaN'] for r in data),'first_valid_last_column':next({'row':int(r[0]),'value':r[-1]} for r in data if r[-1] not in ['NA','nan','NaN']),'last_row_final_column':data[-1][-1]} + matchedflags=['--tIntLenMax=149','--qIntLenMax=99'] + computed=execute('ed_matched_computed',[a for a in base if a!='--intLenMax=0']+matchedflags) + reloaded=execute('ed_matched_reloaded',[a for a in load_base if a!='--intLenMax=0']+matchedflags) + messages=[line for line in (DEST/'ed_reload.stdout').read_text().splitlines() if 'reducing maximal interaction length' in line] + assert len(messages)==2 and any('to 149' in line for line in messages) and any('to 99' in line for line in messages) + dump(DEST/'checks.json',{'implicit_default_checks':checks,'ed_files':rows,'reader_messages':messages,'unmatched_comparison':compare_outputs('ed_write','ed_reload'),'matched_149_99_comparison':compare_outputs('ed_matched_computed','ed_matched_reloaded'),'note':'ED loading is a separate diagnostic requested in the review, not part of default-mode performance timings; coordinates/base pairs and energy equality are checked separately.'}) + print(json.dumps(json.loads((DEST/'checks.json').read_text()),indent=2)) +if __name__=='__main__':main() diff --git a/doc/local-validation-archive/files/profiling-default-20260921/scripts/ed_conversion_probe.cpp b/doc/local-validation-archive/files/profiling-default-20260921/scripts/ed_conversion_probe.cpp new file mode 100644 index 00000000..cc932a4c --- /dev/null +++ b/doc/local-validation-archive/files/profiling-default-20260921/scripts/ed_conversion_probe.cpp @@ -0,0 +1,12 @@ +#include "IntaRNA/general.h" +#include +#include +#include +int main() { + using namespace IntaRNA; + for (const char* text : {"4.760000e+00", "4.890000e+00", "4.770000e+00", "9.800000e+00", "1.086000e+01"}) { + double value = 0; // AccessibilityFromStream reads into double curVal. + std::istringstream(text) >> value; + std::cout << text << '\t' << std::setprecision(17) << value*100 << '\t' << Ekcal_2_E(value) << '\n'; + } +} diff --git a/doc/local-validation-archive/files/profiling-default-20260921/scripts/ed_conversion_probe.sh b/doc/local-validation-archive/files/profiling-default-20260921/scripts/ed_conversion_probe.sh new file mode 100644 index 00000000..9389e7c5 --- /dev/null +++ b/doc/local-validation-archive/files/profiling-default-20260921/scripts/ed_conversion_probe.sh @@ -0,0 +1,9 @@ +#!/usr/bin/env bash +set -euo pipefail +profile_root=$(cd "$(dirname "$0")/.." && pwd) +dependency_root=${INTARNA_DEPS:-/home/alex/CursorProjects/IntaRNA_optimization/intaRNA_legacy/.conda-env} +"$dependency_root/bin/x86_64-conda-linux-gnu-g++" -std=c++23 -O2 \ + -I"$profile_root/source/src" -I"$dependency_root/include" \ + "$profile_root/scripts/ed_conversion_probe.cpp" -o "$profile_root/diagnostics/ed_conversion_probe" +"$profile_root/diagnostics/ed_conversion_probe" > "$profile_root/diagnostics/ed-conversion.tsv" +cat "$profile_root/diagnostics/ed-conversion.tsv" diff --git a/doc/local-validation-archive/files/profiling-default-20260921/scripts/ed_numeric.py b/doc/local-validation-archive/files/profiling-default-20260921/scripts/ed_numeric.py new file mode 100644 index 00000000..b7e3ffb1 --- /dev/null +++ b/doc/local-validation-archive/files/profiling-default-20260921/scripts/ed_numeric.py @@ -0,0 +1,18 @@ +#!/usr/bin/env python3 +"""Separate untimed follow-up locating the observed ED-round-trip energy delta.""" +import csv,hashlib,io,json,subprocess +from benchmark import ROOT,BIN,ENV,dump +DEST=ROOT/'diagnostics' +commands=[json.loads(x) for x in (DEST/'commands.jsonl').read_text().splitlines()] +columns='id1,id2,start1,end1,start2,end2,E,bpList,ED1,ED2,E_init,E_loops,E_dangleL,E_dangleR,E_endL,E_endR,E_hybrid' +result={};records=[] +for name in ['ed_matched_computed','ed_matched_reloaded']: + original=next(r['command'] for r in commands if r['name']==name) + command=[str(BIN)]+[('--outCsvCols='+columns) if a.startswith('--outCsvCols=') else a for a in original[1:]] + p=subprocess.run(command,env=ENV,capture_output=True,timeout=60);p.check_returncode() + (DEST/(name+'_components.stdout')).write_bytes(p.stdout);(DEST/(name+'_components.stderr')).write_bytes(p.stderr) + result[name]=list(csv.DictReader(io.StringIO(p.stdout.decode()),delimiter=';')) + assert len(result[name])==1 + records.append({'name':name+'_components','command':command,'exit_code':p.returncode,'stdout_sha256':hashlib.sha256(p.stdout).hexdigest()}) +dump(DEST/'energy-components.json',result);dump(DEST/'numeric-commands.json',records) +print(json.dumps(result,indent=2)) diff --git a/doc/local-validation-archive/files/profiling-default-20260921/scripts/package.py b/doc/local-validation-archive/files/profiling-default-20260921/scripts/package.py new file mode 100644 index 00000000..8ca338e0 --- /dev/null +++ b/doc/local-validation-archive/files/profiling-default-20260921/scripts/package.py @@ -0,0 +1,21 @@ +#!/usr/bin/env python3 +"""Create a checksummed evidence archive; exclude binaries and raw experiments.""" +import hashlib,json,shutil,zipfile +from pathlib import Path +ROOT=Path(__file__).resolve().parents[1] +for name in ['runApiTests.log','runIntaRNA.log','test-suite.log']: + shutil.copyfile(ROOT/'source/tests'/name,ROOT/'logs'/('tests-'+name)) +selected=[] +for name in ['README.md','REPORT.md','ED-READER-EVIDENCE.md','OUTPUT-STABILITY.md','environment.json','cases.json','study-plan.json','review-requirements.json']: + selected.append(ROOT/name) +for folder in ['scripts','inputs','results','diagnostics','logs']: + selected.extend(p for p in (ROOT/folder).rglob('*') if p.is_file() and '__pycache__' not in p.parts and p.name!='ed_conversion_probe') +selected.extend(p for p in (ROOT/'profiles').iterdir() if p.is_file()) +selected=sorted(set(selected));manifest={str(p.relative_to(ROOT)):{'bytes':p.stat().st_size,'sha256':hashlib.sha256(p.read_bytes()).hexdigest()} for p in selected} +archive=ROOT/'default-mode-evidence.zip' +with zipfile.ZipFile(archive,'w',compression=zipfile.ZIP_DEFLATED) as z: + for p in selected:z.write(p,'default-mode/'+str(p.relative_to(ROOT))) + z.writestr('default-mode/manifest.json',json.dumps(manifest,indent=2)+'\n') +meta={'file':archive.name,'bytes':archive.stat().st_size,'sha256':hashlib.sha256(archive.read_bytes()).hexdigest(),'files':len(selected),'source_revision':'0a4568ad6e3da52f935221ff0da98a4a171e9833','exclusions':'Compiled source/build artifacts and raw gprofng experiment directories; sources remain pinned in upstream Git.'} +(ROOT/'bundle-info.json').write_text(json.dumps(meta,indent=2)+'\n');(ROOT/'SHA256SUMS').write_text(meta['sha256']+' '+archive.name+'\n') +print(json.dumps(meta,indent=2)) diff --git a/doc/local-validation-archive/files/profiling-default-20260921/scripts/parallel_check.py b/doc/local-validation-archive/files/profiling-default-20260921/scripts/parallel_check.py new file mode 100644 index 00000000..f87587b9 --- /dev/null +++ b/doc/local-validation-archive/files/profiling-default-20260921/scripts/parallel_check.py @@ -0,0 +1,28 @@ +#!/usr/bin/env python3 +"""Post-profile diagnostic repeats for the observed ChiX energy instability.""" +import csv,hashlib,io,json,random,subprocess,time +from pathlib import Path +from benchmark import ROOT,BIN,ENV,dump,sha +DEST=ROOT/'diagnostics/parallel';DEST.mkdir(exist_ok=True) +cases={x['name']:x for x in json.loads((ROOT/'cases.json').read_text())} +def rows(data): + text='\n'.join(x for x in data.decode().splitlines() if x and not x.startswith('#')) + return {(r['id1'],r['id2']):r for r in csv.DictReader(io.StringIO(text),delimiter=';')} +reference=rows((ROOT/'results/batch_chix_1t.measure1.stdout').read_bytes()) +schedule=[(t,n) for t,count in [(1,3),(6,5),(12,20)] for n in range(1,count+1)] +random.Random(202609212).shuffle(schedule);checks=[] +for threads,n in schedule: + name=f't{threads}_repeat{n:02d}';command=[str(BIN)]+cases[f'batch_chix_{threads}t']['args'] + if (DEST/(name+'.stdout')).exists():raise RuntimeError('Refusing to overwrite '+name) + p=subprocess.run(command,env=ENV,capture_output=True,timeout=60) + (DEST/(name+'.stdout')).write_bytes(p.stdout);(DEST/(name+'.stderr')).write_bytes(p.stderr) + actual=rows(p.stdout);differences=[] + for key in reference.keys()|actual.keys(): + a,b=reference.get(key),actual.get(key) + if a!=b:differences.append({'id':key,'fields':{k:{'reference':a.get(k),'observed':b.get(k)} for k in a.keys()|b.keys() if a.get(k)!=b.get(k)}} if a and b else {'id':key,'reference':a,'observed':b}) + row={'name':name,'threads':threads,'command':command,'exit_code':p.returncode,'binary_sha256':sha(BIN),'stdout_sha256':hashlib.sha256(p.stdout).hexdigest(),'output_equal':not differences and p.returncode==0,'differences':differences} + checks.append(row) + with (DEST/'checks.jsonl').open('a') as f:f.write(json.dumps(row)+'\n') + print(name,'equal' if row['output_equal'] else json.dumps(differences),flush=True) +summary={'scope':'Untimed post-profile repeat check, not additional performance measurements','reference':'results/batch_chix_1t.measure1.stdout','random_seed':202609212,'counts':{str(t):{'runs':sum(r['threads']==t for r in checks),'different':sum(r['threads']==t and not r['output_equal'] for r in checks)} for t in [1,6,12]},'observed_differences':[r for r in checks if not r['output_equal']]} +dump(DEST/'summary.json',summary);print(json.dumps(summary,indent=2)) diff --git a/doc/local-validation-archive/files/profiling-default-20260921/scripts/phase_summary.py b/doc/local-validation-archive/files/profiling-default-20260921/scripts/phase_summary.py new file mode 100644 index 00000000..c568dd75 --- /dev/null +++ b/doc/local-validation-archive/files/profiling-default-20260921/scripts/phase_summary.py @@ -0,0 +1,26 @@ +#!/usr/bin/env python3 +"""Parse diagnostic phase logs, retaining integer-duration truncation bounds.""" +import json,re +from pathlib import Path +ROOT=Path(__file__).resolve().parents[1] +rows=[] +for path in sorted((ROOT/'profiles').glob('*_phases.stdout')): + data=path.read_text() + groups={k:dict(calls=0,lower_s=0,upper_s=0) for k in ['accessibility','prediction_including_seed','seed_nested']} + detailed=[] + for fun,n,unit in re.findall(r'Executed \[(.*?)\] in \[(\d+) ([^\]]+)\]',data): + scale={'ms':.001,'seconds':1,'minutes':60,'hours':3600}.get(unit) + if scale is None:raise ValueError(unit) + lo=int(n)*scale;hi=lo+scale + if 'AccessibilityVrna::' in fun:group='accessibility' + elif '::predict(' in fun:group='prediction_including_seed' + elif '::fillSeed(' in fun:group='seed_nested' + else:group=None + if group: + groups[group]['calls']+=1;groups[group]['lower_s']+=lo;groups[group]['upper_s']+=hi + detailed.append(dict(function=fun,displayed=f'{n} {unit}',lower_s=lo,upper_s=hi,group=group)) + rows.append(dict(profile=path.stem.removesuffix('_phases'),groups=groups,phases=detailed, + valid_seed_count_sum=sum(map(int,re.findall(r'valid seeds = (\d+)',data))))) +(ROOT/'profiles/phase-summary.json').write_text(json.dumps(rows,indent=2)+'\n') +for row in rows: + print(row['profile'],row['groups']) diff --git a/doc/local-validation-archive/files/profiling-default-20260921/scripts/plots.py b/doc/local-validation-archive/files/profiling-default-20260921/scripts/plots.py new file mode 100644 index 00000000..d24ba323 --- /dev/null +++ b/doc/local-validation-archive/files/profiling-default-20260921/scripts/plots.py @@ -0,0 +1,36 @@ +#!/usr/bin/env python3 +"""Static scientific figure, generated only from saved summaries.""" +import json +from pathlib import Path +import matplotlib +matplotlib.use('Agg') +import matplotlib.pyplot as plt +import numpy as np +ROOT=Path(__file__).resolve().parents[1] +S={x['case']:x for x in json.loads((ROOT/'results/summary.json').read_text())} +CPU=json.loads((ROOT/'profiles/cpu-summary.json').read_text()) +PH=json.loads((ROOT/'profiles/phase-summary.json').read_text()) +plt.rcParams.update({'font.family':'DejaVu Sans','font.size':9,'axes.spines.top':False,'axes.spines.right':False,'axes.titleweight':'bold','axes.titlelocation':'left','axes.grid':True,'grid.alpha':.18}) +fig,axs=plt.subplots(2,2,figsize=(12,8),layout='constrained') +blue='#175577';green='#237c70';orange='#be6a2e';gray='#cbd4d9' +a=axs[0,0];ns=[100,300,1000,3000,10000];rows=[S[f'length_t{n}_q100'] for n in ns] +med=np.array([r['median_s'] for r in rows]);lo=med-np.array([r['min_s'] for r in rows]);hi=np.array([r['max_s'] for r in rows])-med +a.errorbar(ns,med,yerr=[lo,hi],color=blue,marker='o',capsize=4) +a.set(xscale='log',yscale='log',xlabel='Target length (nt); query fixed at 100 nt',ylabel='Wall time (s)',title='A Length scaling within the measured range') +a.set_xticks(ns,[f'{n:,}' for n in ns]);a.grid(True,which='minor',alpha=.06) +a=axs[0,1] +for rna,color,label in [('gcvb',blue,'GcvB (201 nt)'),('chix',green,'ChiX (84 nt)')]: + ts=[1,6,12] if rna=='gcvb' else [1,6];base=S[f'batch_{rna}_1t']['median_s'];a.plot(ts,[base/S[f'batch_{rna}_{t}t']['median_s'] for t in ts],marker='o',color=color,label=label) + if rna=='chix':a.scatter([12],[base/S['batch_chix_12t']['median_s']],marker='x',color=orange,s=60,label='ChiX 12t: energy mismatch') +a.set(xlabel='OpenMP threads (6 physical cores / 12 hardware threads)',ylabel='Observed time ratio relative to one thread',xticks=[1,6,12],title='B 48-target biological batch throughput');a.legend(frameon=False) +a=axs[1,0];left=np.zeros(4);short=['GcvB run 1','GcvB run 2','ChiX (384 pairs)','Long pair'] +for text,needle,color in [('Loop-energy evaluation','::getE_interLeft(',blue),('Left extension','PredictorMfe2dHeuristicSeedExtension::fillHybridE_left(',green),('Right extension','PredictorMfe2dHeuristicSeedExtension::fillHybridE_right(',orange)]: + values=np.array([sum(f['self_pct'] for f in r['self_functions'] if needle in f['function']) for r in CPU]);a.barh(short,values,left=left,color=color,label=text);left+=values +a.barh(short,100-left,left=left,color=gray,label='Other');a.invert_yaxis();a.set(xlim=(0,100),xlabel='Exclusive sampled CPU (%)',title='C Default-mode hotspots (100 ms sampling)');a.legend(frameon=False,fontsize=8,loc='lower center',bbox_to_anchor=(.5,-.45),ncol=2) +a=axs[1,1];short=[];acc=[];pred=[] +for name,label in [('batch_gcvb_1t','GcvB 48'),('batch_chix_1t','ChiX 48'),('long_pair','Long pair')]: + r=next(x for x in PH if x['profile']==name);g=r['groups'];aa=g['accessibility']['lower_s'];pp=g['prediction_including_seed']['lower_s'];short.append(label);acc.append(aa);pred.append(pp) +a.bar(short,acc,color=green,label='Local accessibility');a.bar(short,pred,bottom=acc,color=blue,label='Predictor, including seed') +a.set(ylabel='Logged phase time (s; lower truncation bound)',title='D Phase attribution, measured separately');a.legend(frameon=False,fontsize=8) +fig.suptitle('IntaRNA default mode: seed-extension heuristic + local accessibility\nPrediction windowing disabled | master 0a4568a | 21 September 2026',fontsize=14,fontweight='bold') +fig.savefig(ROOT/'results/overview.png',dpi=180);fig.savefig(ROOT/'results/overview.svg') diff --git a/doc/local-validation-archive/files/profiling-default-20260921/scripts/profile.py b/doc/local-validation-archive/files/profiling-default-20260921/scripts/profile.py new file mode 100644 index 00000000..bd329eaa --- /dev/null +++ b/doc/local-validation-archive/files/profiling-default-20260921/scripts/profile.py @@ -0,0 +1,49 @@ +#!/usr/bin/env python3 +"""Separate CPU, diagnostic-phase, and heap runs of the requested default mode.""" +import json,subprocess,time,hashlib +from datetime import datetime,timezone +from benchmark import ROOT,BIN,ENV,INPUT,fasta,write_fasta,canonical,validate,dump,sha +DEST=ROOT/'profiles' +def execute(name,command,timeout=420): + if (DEST/(name+'.stdout')).exists():raise RuntimeError('Refusing to overwrite '+name) + wrapped=['/usr/bin/time','-f','%e\t%U\t%S\t%M','-o',str(DEST/(name+'.time'))]+command + started=datetime.now(timezone.utc).isoformat() + with (DEST/(name+'.stdout')).open('wb') as out,(DEST/(name+'.stderr')).open('wb') as err: + p=subprocess.run(wrapped,env=ENV,stdout=out,stderr=err,timeout=timeout) + row={'name':name,'command':command,'exit_code':p.returncode,'started_utc':started,'ended_utc':datetime.now(timezone.utc).isoformat(),'resources':(DEST/(name+'.time')).read_text().strip()} + with (DEST/'commands.jsonl').open('a') as f:f.write(json.dumps(row)+'\n') + print(name,p.returncode,flush=True);p.check_returncode() + return (DEST/(name+'.stdout')).read_bytes() + +def main(): + DEST.mkdir(exist_ok=True) + cases={x['name']:x for x in json.loads((ROOT/'cases.json').read_text())};checks=[] + records=fasta(INPUT/'promoters48_systematic.fa') + repeated=write_fasta('profile_chix384',[(f'repeat{k}_{n}',seq) for k in range(8) for n,seq in records]) + dump(DEST/'input-manifest.json',{'profile_chix384.fa':{'sha256':sha(repeated),'records':384,'construction':'Eight copies of the same systematically selected 48 targets, with unique IDs; no genome-wide claim.'}}) + cpu=[('gcvb',cases['batch_gcvb_1t']['args']),('gcvb_repeat',cases['batch_gcvb_1t']['args']),('chix384',[('--target='+str(repeated)) if a.startswith('--target=') else a for a in cases['batch_chix_1t']['args']]),('long_pair',cases['long_pair']['args'])] + for name,args in cpu: + if name=='chix384':reference=execute(name+'_reference',[str(BIN)]+args) + else:reference=(ROOT/'results'/(({'gcvb':'batch_gcvb_1t','gcvb_repeat':'batch_gcvb_1t','long_pair':'long_pair'}[name])+'.measure1.stdout')).read_bytes() + exp=DEST/(name+'.er') + actual=execute(name+'_cpu',['gprofng','collect','app','-p','100','-a','off','-o',str(exp),str(BIN)]+args) + valid,rows=validate(actual);equal=canonical(actual)==canonical(reference) + checks.append({'profile':name,'kind':'cpu','csv_valid':valid,'output_equal_bpList':equal,'reported_rows':rows});assert valid and equal,name + for kind,sort in [('self','e.totalcpu'),('inclusive','i.totalcpu')]: + execute(name+'_'+kind,['gprofng','display','text','-metrics','e.%totalcpu:i.%totalcpu:name','-sort',sort,'-limit','50','-functions',str(exp)]) + execute(name+'_statistics',['gprofng','display','text','-header','-statistics',str(exp)]) + execute(name+'_lines',['gprofng','display','text','-metrics','e.%totalcpu:i.%totalcpu:name','-sort','e.totalcpu','-limit','30','-lines',str(exp)]) + if name in ['gcvb','long_pair']: + execute(name+'_loop_source',['gprofng','display','text','-metrics','e.%totalcpu:i.%totalcpu:name','-source','IntaRNA::InteractionEnergyVrna::getE_interLeft(unsigned long, unsigned long, unsigned long, unsigned long) const',str(exp)]) + for name in ['batch_gcvb_1t','batch_chix_1t','long_pair']: + actual=execute(name+'_phases',[str(BIN)]+cases[name]['args']+['--verbose']) + equal=canonical(actual)==canonical((ROOT/'results'/(name+'.measure1.stdout')).read_bytes()) + checks.append({'profile':name,'kind':'phases','output_equal_bpList':equal});assert equal,name + for name in ['batch_gcvb_1t','long_pair']: + exp=DEST/(name+'_heap.er') + actual=execute(name+'_heap',['gprofng','collect','app','-p','off','-H','on','-a','off','-o',str(exp),str(BIN)]+cases[name]['args']) + equal=canonical(actual)==canonical((ROOT/'results'/(name+'.measure1.stdout')).read_bytes()) + checks.append({'profile':name,'kind':'heap','output_equal_bpList':equal});assert equal,name + execute(name+'_heap_summary',['gprofng','display','text','-limit','30','-heapstat','-allocs','-leaks',str(exp)]) + dump(DEST/'checks.json',checks) +if __name__=='__main__':main() diff --git a/doc/local-validation-archive/files/profiling-default-20260921/scripts/report.py b/doc/local-validation-archive/files/profiling-default-20260921/scripts/report.py new file mode 100644 index 00000000..74cc28b4 --- /dev/null +++ b/doc/local-validation-archive/files/profiling-default-20260921/scripts/report.py @@ -0,0 +1,102 @@ +#!/usr/bin/env python3 +"""Render review reports from audited observations, without changing evidence.""" +import json +from pathlib import Path +ROOT=Path(__file__).resolve().parents[1] +load=lambda name:json.loads((ROOT/name).read_text()) +S=load('results/summary.json');D={x['case']:x for x in S};CPU=load('profiles/cpu-summary.json');HEAP=load('profiles/heap-summary.json');PHASE=load('profiles/phase-summary.json');CHECK=load('diagnostics/checks.json');AUDIT=load('results/audit.json') +assert AUDIT['integrity_status']=='pass',AUDIT +REV='0a4568ad6e3da52f935221ff0da98a4a171e9833' +SRC='https://github.com/BackofenLab/IntaRNA/blob/'+REV+'/' +link=lambda file,lines:SRC+file+'#L'+lines +names={'gcvb':'GcvB 48, run 1','gcvb_repeat':'GcvB 48, run 2','chix384':'ChiX, 48 targets repeated 8x','long_pair':'Synthetic 3000 x 800 nt'} +shares=lambda needle:[sum(f['self_pct'] for f in p['self_functions'] if needle in f['function']) for p in CPU] +loop=shares('::getE_interLeft(') +rows=['# Default-mode IntaRNA profiling: review follow-up','', +'Profiled 21 September 2026 at current upstream master `'+REV+'`. **18 configurations; 90 measured runs plus 18 excluded warmups; four CPU profiles, two heap traces, and three separate phase runs.**', +'', 'This follows [Martin\'s requested mode](https://github.com/BackofenLab/IntaRNA/pull/240#issuecomment-5762928155): **seed-extension heuristic, no prediction windowing, computed local accessibility**. [ED-READER-EVIDENCE.md](ED-READER-EVIDENCE.md) answers his separate request for writer/reader evidence and source links.', +'', '## Main result', '', +'**Two performance regimes appear within the requested default mode.** In the separate diagnostic runs, GcvB spends about 31.3 s in prediction versus 3.82 s in local accessibility; ChiX spends about 1.30 s in prediction versus 3.80 s in local accessibility. Thus prediction accounts for about 89% of GcvB logged phase time, while local accessibility accounts for about 74–75% for ChiX. These are ratios of the logged major phases, not exact wall-time fractions.', '', +f'Internal-loop energy evaluation accounts for **{min(loop):.1f}–{max(loop):.1f}%** of exclusive sampled CPU across the default-mode profiles. The independently repeated GcvB profile gives **{loop[0]:.1f}% and {loop[1]:.1f}%**. These samples identify a CPU optimization target; they do not measure an optimization or promise a speedup.', +'', '![Default-mode profiling overview](results/overview.png)', +'', '**Correctness finding:** one measured 12-thread ChiX run changed an energy without changing coordinates/base pairs. A further 20 twelve-thread diagnostic runs reproduced an energy difference once on another target. See [OUTPUT-STABILITY.md](OUTPUT-STABILITY.md). The timing data are preserved, but the affected thread setting is not validated as an output-preserving speedup.', +'', 'Focus seed-extension/loop-energy work on prediction-heavy cases, while retaining an accessibility-heavy case in the acceptance suite. Local accessibility remains enabled. Exact/ensemble modes, prediction tiling, disabling accessibility, and saved-ED workflows do not substitute for this requested performance target.', +'', '## Configuration and provenance', '', +'Every timed case uses `--model=X --mode=H --acc=C --accW=150 --accL=100 --windowWidth=0 --seedBP=7 --seedMaxUP=0 --intLenMax=0`. These are the scientific defaults. The 150-nt local folding window and 100-nt base-pair span remain enabled; `windowWidth=0` disables prediction tiling. [Source mapping and protocol](README.md#requested-mode-and-recorded-source) explain the distinction.', +'', 'Output is CSV including full `bpList`; unlike the earlier seven-column output, this retains traceback. It changes formatting from the default text UI, not the prediction algorithm. Explicit versus implicit defaults match full output on three biological examples. Coordinates and base-pair lists match across the tested thread counts, but the 12-thread ChiX case has the energy mismatch described above.', +'', 'Fresh GCC 16.1.0 C++23 `-O3` build with debug symbols/frame pointers, OpenMP, ViennaRNA 2.7.2, and Boost 1.85.0. Host: AMD Ryzen 5 7530U, six cores/twelve hardware threads, Linux 6.8.0-139, performance governor. BLAS is limited to one worker; OpenMP uses `OMP_DYNAMIC=FALSE`, `OMP_PLACES=cores`, `OMP_PROC_BIND=close`. [Environment and binary hash](environment.json) and [exact case arguments](cases.json) are retained.', +'', 'The biological batch uses 48 evenly spaced record indices across the 4,319-record repository FASTA, without selecting on outcomes. [Indices](inputs/selection.json) and [input hashes/lengths](inputs/manifest.json) are supplied. This bounded sample is not a full-genome scan. Synthetic sequences use seed `20260921` and nested prefixes.', +'', 'The source revision includes later upstream changes, including the accepted feasibility optimization. Inputs and output configuration also differ from 19 September; the two studies are **not a before/after benchmark**.', +'', '## Clean timing results', '', +'Each row contains five measured fresh-process runs after one excluded warmup. Rounds run sequentially in deterministic randomized order. Wall time includes startup, input, accessibility, prediction, traceback, and output. Peak RSS is the median of process high-water marks in MiB, not traced heap allocation. Min/max are observed ranges, not confidence intervals.', +'', '| Case | Threads | Median wall s | Min–max s | Median user / system s | Peak RSS MiB | Reported interactions |','| --- | ---: | ---: | ---: | ---: | ---: | ---: |'] +for r in S:rows.append(f"| `{r['case']}` | {r['threads']} | {r['median_s']:.4f} | {r['min_s']:.4f}–{r['max_s']:.4f} | {r['median_user_s']:.3f} / {r['median_system_s']:.3f} | {r['median_rss_mib']:.2f} | {r['reported_rows']} |") +rows += ['', 'Machine-readable [CSV](results/summary.csv) and [JSON](results/summary.json) include maximum observed RSS and timing scatter. The length sweeps describe these sequences and ranges; they do not establish asymptotic complexity or population averages.', '', '### Biological batch throughput', '', '| Query | Threads | Median wall s | Predictions/s | Speedup | Peak RSS MiB |', '| --- | ---: | ---: | ---: | ---: | ---: |'] +for query in ['gcvb','chix']: + baseline=D[f'batch_{query}_1t']['median_s'] + for t in [1,6,12]: + r=D[f'batch_{query}_{t}t'];rows.append(f"| {query.upper()} | {t} | {r['median_s']:.3f} | {48/r['median_s']:.2f} | {baseline/r['median_s']:.2f}x | {r['median_rss_mib']:.2f} |") +rows += ['', '**ChiX at 12 threads: energy output is not stable.** Its table entry is an observed timing ratio, not an accepted correctness-preserving speedup. Throughput counts all 48 evaluated target/query pairs, including pairs with no reportable interaction. These numbers describe batched throughput, not acceleration of a single unwindowed pair.', '', '## CPU attribution in the requested mode', '', '| Profile | Samples | Sampled CPU s | Loop energy self % | Heuristic left self % | Heuristic right self % |', '| --- | ---: | ---: | ---: | ---: | ---: |'] +left=shares('PredictorMfe2dHeuristicSeedExtension::fillHybridE_left(');right=shares('PredictorMfe2dHeuristicSeedExtension::fillHybridE_right(') +for i,p in enumerate(CPU):rows.append(f"| {names[p['profile']]} | {p['samples']} | {p['sampled_cpu_s']:.1f} | {loop[i]:.2f} | {left[i]:.2f} | {right[i]:.2f} |") +rows += ['', f"{sum(p['samples'] for p in CPU):,} total nominal 100-ms samples. The ChiX sampling workload repeats the same 48 targets eight times with unique IDs to lengthen observation; it is not 384 independent biological targets. The second GcvB collection is an independent repeat of the same workload.", '', +'The source path is: [per-seed extension](%s), then the [right](%s) and [left](%s) recurrence calls into [`InteractionEnergyVrna::getE_interLeft`](%s), which checks the internal-loop boundaries, retrieves sequence/base-pair codes, and evaluates the ViennaRNA loop energy.' % (link('src/IntaRNA/PredictorMfe2dHeuristicSeedExtension.cpp','71-L101'),link('src/IntaRNA/PredictorMfe2dHeuristicSeedExtension.cpp','165-L183'),link('src/IntaRNA/PredictorMfe2dHeuristicSeedExtension.cpp','258-L275'),link('src/IntaRNA/InteractionEnergyVrna.h','377-L407')), +'', 'These are exclusive/self percentages. Inclusive caller rows overlap their children and must not be added. Optimized call ancestry and line attribution can be incomplete. [CPU summary](profiles/cpu-summary.json), original function tables, top source-line tables, annotated source, and collector statistics are preserved in the evidence archive.', +'', '### Collector checks and limits', '', +f"The largest absolute difference between sampled CPU and the collector's process user+system CPU totals is {max(abs(p['sampled_minus_process_cpu_s']) for p in CPU):.3f} s. The collector warning about its interval timer is retained in every affected profile; matching CPU totals and a repeated workload are supporting checks, not proof of perfect sampling. Use these results for coarse hotspot ranking, not precise speedup forecasts or fine differences between percentages.", +'', 'Hardware counter access was unavailable under the recorded host permissions (`perf_event_paranoid=4`); no IPC, cache-miss, branch-miss, or hardware-stall claims are made. [GNU gprofng documentation](https://sourceware.org/binutils/docs/gprofng.html) describes the function and annotated-source views used here.', +'', '## Local accessibility versus prediction phases', '', +'Built-in verbose timers run separately from clean timing. Predictor time includes seed filling and reporting. Seed time below is nested and must not be added again. Timers truncate their displayed unit; each cell retains the resulting lower/upper sum bounds.', '', '| Workload | Accessibility calls; seconds [lower, upper) | Predictor calls; seconds [lower, upper) | Nested seed calls; seconds [lower, upper) |', '| --- | --- | --- | --- |'] +for p in PHASE: + entries=[] + for key in ['accessibility','prediction_including_seed','seed_nested']: + g=p['groups'][key];entries.append(f"{g['calls']}; [{g['lower_s']:.3f}, {g['upper_s']:.3f})") + rows.append('| '+p['profile']+' | '+' | '.join(entries)+' |') +rows += ['', 'These are diagnostic phase totals, not subtractions of runs with different scientific models. Removing accessibility would change the prediction problem and is not used to estimate its cost.', '', '## Heap behavior', '', '| Workload | Allocation calls | Cumulative bytes allocated | Peak tracked heap bytes | Outstanding at exit bytes |', '| --- | ---: | ---: | ---: | ---: |'] +for p in HEAP:rows.append(f"| {p['profile']} | {p['allocation_calls']:,} | {p['cumulative_allocated_bytes']:,} | {p['peak_tracked_heap_bytes']:,} | {p['outstanding_at_exit_bytes']:,} |") +rows += ['', 'Cumulative allocation traffic, peak live traced heap, and resident memory measure different things. Outstanding-at-exit allocations alone do not prove a leak. Allocation stacks and statistics are retained in the archive.', '', +'Extension workspaces are [resized inside the seed loop](%s), with dimensions capped by the default interaction/accessibility limits. Allocation volume is a possible implementation cost, but a simple switch to non-preserving resize was already [tested and rejected for lacking a useful end-to-end gain](%s). This study does not reverse that result.' % (link('src/IntaRNA/PredictorMfe2dHeuristicSeedExtension.cpp','80-L101'),link('doc/refactor/3-performance.md','150-L173')), +'', '## Optimization priorities and acceptance criteria', '', +'1. **Resolve the observed parallel energy instability before accepting that configuration as a correctness-preserving speedup.** Both occurrences, all raw outputs, and a repeat script are supplied. The root cause is not established; no fix is claimed in this profiling PR.', +'2. **For prediction-heavy inputs, start in repeated loop-energy evaluation within default seed extension.** Investigate reuse/hoisting of sequence codes, base-pair types, and loop-bound invariants at the recurrence callsites. Preserve thermodynamic terms, boundary checks, target/query asymmetry, offsets, and traceback. A hotspot is a place to test a change, not proof that a particular cache or rewrite helps.', +'3. **For accessibility-heavy inputs, retain the current local folding model and inspect the accessibility path separately.** ChiX shows substantial ViennaRNA window/probability work and about 74–75% of logged phase time in accessibility. Any API or preprocessing change needs full ED-table/numerical equivalence checks. A gain on an accessibility-disabled case would not satisfy this requested mode.', +'4. **Respect existing negative experiments.** The [prior phase-3 record](%s) rejected ED-lookup caching, non-preserving scratch resize, and skipping unused partition merges. A different proposal needs a distinct mechanism and fresh same-parent evidence; do not treat those old candidates as newly validated.' % (SRC+'doc/refactor/3-performance.md'), +'5. **Benchmark each candidate against this exact parent on the same host/build protocol.** Require unchanged energies, coordinates, full base-pair output, seed/tie/constraint behavior, and regression tests. Use both biological queries and the long pair; measure five or more repeated candidate/parent runs in interleaved order, report dispersion and peak RSS, and accept only a useful gain larger than observed noise. No candidate code or optimization speedup is measured in this PR.', +'', '## Validation and uncertainty', '', +'- Fresh build: **4,334 assertions in 37 API cases**, plus **20 CLI golden cases**, all passed.', +'- Evidence integrity audit: **PASS, 18 cases / 108 runs**. Scientific output checks: **differences observed**, explicitly retained in the audit rather than treated as equality. [Machine-readable audit](results/audit.json).', +'- All four CPU profiles, two heap traces, and three phase runs match full prediction output. GcvB matches across 1/6/12 threads; ChiX coordinates/base pairs match, but twelve-thread energies vary. Implicit/explicit defaults match in three biological examples.', +f"- The largest full measured range is **{max(S,key=lambda x:x['range_over_median_pct'])['range_over_median_pct']:.2f}% of the median**, in `{max(S,key=lambda x:x['range_over_median_pct'])['case']}`. Five repetitions support descriptive statistics, not tight tail estimates or population confidence intervals.", +'- One laptop, one source revision, a bounded biological sample and a few synthetic sequences. Warm caches, dynamic frequency, shared desktop activity, compiler flags, and dependencies affect applicability. No cross-version, biological-accuracy, or genome-wide performance claim follows.', +'', 'See [README.md](README.md) for reproducible build, timing, profiling, and archive-verification commands. The prior report remains historical; this follow-up supplies the requested default-mode baseline.', ''] +(ROOT/'REPORT.md').write_text('\n'.join(rows)) + +ed=['# ED reader: source and reproduced evidence for Martin','', +'This addresses [the request for evidence and source links](https://github.com/BackofenLab/IntaRNA/pull/240#issuecomment-5763250539). The observation is reproduced at current master `'+REV+'`. It is separate from the default-mode performance matrix; cached ED is not offered as the optimization target.', '', +'## Observation and source path', '', +'1. The [writer](%s) labels lengths through `min(sequence size, getMaxLength())` and writes valid values through that limit; it does not append an extra length column.' % link('src/IntaRNA/Accessibility.cpp','68-L104'), +'2. The [reader](%s) obtains the last header label, subtracts one, and assigns that reduced value to `availMaxLength` if it is below the requested maximum. The adjacent comment explicitly says the reserve is for dangling-end treatment.' % link('src/IntaRNA/AccessibilityFromStream.cpp','87-L105'), +'3. [`getMaxLength()`](%s) returns `availMaxLength`, and [`getED()`](%s) returns its upper-bound sentinel for regions longer than that effective maximum.' % (link('src/IntaRNA/AccessibilityFromStream.h','158-L164'),link('src/IntaRNA/AccessibilityFromStream.h','135-L153')), +'4. The [computed-accessibility constructor](%s) caps default maximum length using sequence length and the accessibility window. On the 1,000/100-nt inputs here, those computed limits are 150/100. Reading the self-generated files exposes 149/99.' % link('src/bin/CommandLineParsing.cpp','1955-L1965'), +'', '**This is a reproduced writer/reader limit asymmetry, not a claim that the documented dangling-end policy is necessarily wrong.** The open design question is whether self-generated ED reload is intended to reduce the effective interaction search limit, or whether the export/read contract should preserve it.', +'', '## Saved files contain their last advertised columns', '', +'| File | Last header length | Data rows | Fields per row, including row index | Numeric entries in last column | First valid last-column value (kcal/mol) |', '| --- | ---: | ---: | --- | ---: | --- |'] +for name,r in CHECK['ed_files'].items():ed.append(f"| `{name}.ed` | {r['last_header_length']} | {r['data_rows']} | {r['fields_per_row']} | {r['numeric_final_column_rows']} | row {r['first_valid_last_column']['row']}: {r['first_valid_last_column']['value']} |") +ed += ['', 'Thus the observation is not caused by a missing or truncated final ED column. The complete files and their hashes are in the evidence archive under `default-mode/diagnostics/ed/`; extracted header/value excerpts are [available directly](diagnostics/ed-header-excerpts.txt).', '', '## Reader messages from the fresh reproduction', '', 'Exact messages in [ed_reload.stdout](diagnostics/ed_reload.stdout):', '', '```text']+CHECK['reader_messages']+['```','', +'The uncapped CLI diagnostic requests default interaction limits (`--intLenMax=0`); the cached reader initializes from sequence lengths before applying the header-derived limits. Its diagnostic can therefore mention the 1,000-nt target even though computed local accessibility is capped at 150. The source paths above distinguish those two stages.', '', +'On this input, **coordinates and base-pair lists match, but energy does not**, including after both variants are explicitly capped at target/query **149/99**: computed **-14.64**, loaded **-14.66 kcal/mol**. Matching the limits removes the search-space confound but does not establish a numerically transparent round trip. No cache-latency claim is made in this follow-up.', '', +'## Separate numeric round-trip observation', '', +'[Energy-component output](diagnostics/energy-components.json) localizes the difference: `ED1=4.76` and `ED2=9.8` agree, while left/right dangling contributions change from `-0.21/-0.49` to `-0.22/-0.50` kcal/mol. The target ED entries for the one-base extensions at positions 649–677 and 650–678 are written as `4.890000e+00` and `4.770000e+00`.', '', +'The reader [parses into `double curVal` and calls `Ekcal_2_E`](%s); the [conversion macro](%s) multiplies by 100 and casts to the integer energy type. A standalone C++ probe using the actual header and the reader\'s `double` type produces **488** and **476** for those two text values, rather than 489 and 477. The [probe output](diagnostics/ed-conversion.tsv) and source/build script are in the archive. The [conditional dangling probabilities](%s) use precisely these one-base-extended ED regions. This is an additional numeric conversion problem to review separately from the header-limit policy; no application fix is included.' % (link('src/IntaRNA/AccessibilityFromStream.cpp','135-L157'),link('src/IntaRNA/general.h','103-L113'),link('src/IntaRNA/InteractionEnergy.h','883-L949')), '', +'## Reproduce', '', +'Use the pinned executable and the extracted archive inputs. From a writable directory, with `BIN` pointing to that executable and `DATA` to the extracted `default-mode` directory:', '', '```bash', +'"$BIN" --target="$DATA/inputs/t1000.fa" --query="$DATA/inputs/q100.fa" \\\n --model=X --mode=H --acc=C --accW=150 --accL=100 --windowWidth=0 \\\n --intLenMax=0 --threads=1 --out=tAcc:target.ed --out=qAcc:query.ed', +'"$BIN" --target="$DATA/inputs/t1000.fa" --query="$DATA/inputs/q100.fa" \\\n --model=X --mode=H --accW=150 --accL=100 --windowWidth=0 \\\n --intLenMax=0 --threads=1 --tAcc=E --tAccFile=target.ed \\\n --qAcc=E --qAccFile=query.ed --verbose', '```', '', +'The exact collection commands, including the machine-readable output settings, are [recorded here](diagnostics/commands.jsonl). `scripts/diagnostics.py` in the archive automates file inspection, checks the two reduction messages, and compares full output before/after explicit limit matching. Its JSON result is [diagnostics/checks.json](diagnostics/checks.json). No files were manually extended or truncated, and no application source was modified.', ''] +(ROOT/'ED-READER-EVIDENCE.md').write_text('\n'.join(ed)) +excerpts=[] +for name,r in CHECK['ed_files'].items(): + lines=(ROOT/'diagnostics/ed'/(name+'.ed')).read_text().splitlines() + excerpts += [name+'.ed',lines[0],lines[1].rstrip(),f"First valid final column: row {r['first_valid_last_column']['row']}, value {r['first_valid_last_column']['value']}",f"Last row final column: {r['last_row_final_column']}",''] +(ROOT/'diagnostics/ed-header-excerpts.txt').write_text('\n'.join(excerpts)) +print(ROOT/'REPORT.md');print(ROOT/'ED-READER-EVIDENCE.md') diff --git a/doc/local-validation-archive/files/profiling-default-20260921/scripts/summarize_profiles.py b/doc/local-validation-archive/files/profiling-default-20260921/scripts/summarize_profiles.py new file mode 100644 index 00000000..db11ddf1 --- /dev/null +++ b/doc/local-validation-archive/files/profiling-default-20260921/scripts/summarize_profiles.py @@ -0,0 +1,23 @@ +#!/usr/bin/env python3 +"""Parse gprofng text exports without inventing missing symbols or call ancestry.""" +import json,re +from pathlib import Path +ROOT=Path(__file__).resolve().parents[1];DEST=ROOT/'profiles' +rows=[] +for name in ['gcvb','gcvb_repeat','chix384','long_pair']: + text=(DEST/(name+'_self.stdout')).read_text();stats=(DEST/(name+'_statistics.stdout')).read_text();functions=[] + for line in text.splitlines(): + match=re.match(r'\s*([\d.]+)\s+([\d.]+)\s+([\d.]+)\s+([\d.]+)\s+(.+)',line) + if match: + a,b,c,d,fun=match.groups();functions.append({'self_s':float(a),'self_pct':float(b),'inclusive_s':float(c),'inclusive_pct':float(d),'function':fun}) + total=next(f for f in functions if f['function']=='')['self_s'] + user=float(re.search(r'User CPU:\s+([\d.]+)',stats).group(1));system=float(re.search(r'System CPU:\s+([\d.]+)',stats).group(1)) + rows.append({'profile':name,'nominal_sampling_period_ms':100,'sampled_cpu_s':total,'samples':round(total/.1),'process_user_s':user,'process_system_s':system,'sampled_minus_process_cpu_s':total-user-system,'collector_warnings':[line for line in stats.splitlines() if 'Warning:' in line],'self_functions':functions}) +(DEST/'cpu-summary.json').write_text(json.dumps(rows,indent=2)+'\n') +heaps=[] +for name in ['batch_gcvb_1t','long_pair']: + text=(DEST/(name+'_heap_summary.stdout')).read_text() + totals=re.findall(r'^Total bytes\s+(\d+)',text,re.M) + heaps.append({'profile':name,'peak_tracked_heap_bytes':int(re.search(r'Heap size bytes\s+(\d+)',text).group(1)),'allocation_calls':int(re.search(r'Total allocations\s+(\d+)',text).group(1)),'cumulative_allocated_bytes':int(totals[0]),'outstanding_at_exit_bytes':int(totals[1]),'note':'Tracked heap metrics differ from RSS. Outstanding at exit does not by itself establish an application leak.'}) +(DEST/'heap-summary.json').write_text(json.dumps(heaps,indent=2)+'\n') +print(json.dumps({'cpu':[{'profile':r['profile'],'samples':r['samples'],'sampled_minus_process_cpu_s':r['sampled_minus_process_cpu_s']} for r in rows],'heap':heaps},indent=2)) diff --git a/doc/local-validation-archive/files/profiling/README.md b/doc/local-validation-archive/files/profiling/README.md new file mode 100644 index 00000000..9b3a7578 --- /dev/null +++ b/doc/local-validation-archive/files/profiling/README.md @@ -0,0 +1,46 @@ +# IntaRNA profiling artifacts + +Profiling target: commit `1b8377989ba79a0f1b8a450163b71bd8ed7c3e45` from the main `refactor-to-c++23` checkout. This directory contains an isolated source archive and fresh optimized executable. The working checkout and its pre-existing executables are unchanged. + +Read [REPORT.md](REPORT.md) for conclusions, measurements, limitations and optimization priorities. [results/overview.png](results/overview.png) is the visual overview. All timings refer to this machine and the recorded build, not a cross-version comparison. + +`intarna-profile-evidence.zip` contains the report, figures, scripts, inputs, text profile outputs, audited timing evidence and C/C++ sources for inspecting findings. Large build artifacts and raw gprofng experiment directories remain in this local folder. The archive includes a SHA-256 manifest of its evidence files. + +## Evidence + +- `environment.json`: CPU, compiler, linked libraries, revision and executable hash, relevant environment, affinity, governor and cgroup limits. +- `cases.json`: exact benchmark argument arrays, workload categories, thread counts and supplementary environment overrides. +- `inputs/manifest.json`: deterministic input hashes, actual lengths and GC fractions. Biological inputs come from the repository's handson examples. Synthetic generator seed is `20260919`. +- `results/runs.jsonl`: original per-run commands, monotonic elapsed time, GNU time resource metrics, timestamps, exit codes and output hashes. Individual `.stdout`, `.stderr` and `.time` files are preserved. +- `results/validated-runs.jsonl`: derived schema and hash validation after filtering IntaRNA's `# INFO` lines. Original run records are retained unchanged. +- `results/summary.csv` and `.json`: medians, min/max, CPU time, peak RSS and output stability for measured repetitions only. `failures.json` lists unsuccessful/invalid runs. +- `profiles/`: independent CPU sampling, heap tracing and verbose phase measurements, plus original capability probes. `probe*` uses the old executable solely to test tooling; these files do not supply final performance claims. +- `council/`: three agents' planning discussion, methodological and source reviews, and final audits. +- `logs/`: complete build/test logs, benchmark progress, cache-generation cost, and profile orchestration. + +## Reproduce + +Keep the current evidence intact. Copy `scripts/` into a new writable directory, then run the commands below there. The build requires the recorded conda dependency environment; another toolchain constitutes a different experiment. `INTARNA_DEPS` can point to an equivalent local dependency environment. + +```bash +INTARNA_REPO=/home/alex/CodexProjects/IntaRNA/intarna_optimization_supervised bash scripts/build.sh +python3 scripts/benchmark.py prepare +python3 scripts/benchmark.py run +python3 scripts/supplement.py +python3 scripts/structural_checks.py +python3 scripts/profile.py +python3 scripts/profile_audit.py +python3 scripts/phase_summary.py +bash scripts/stream_cleanup_probe.sh +python3 scripts/benchmark.py summarize +python3 scripts/audit.py +python3 scripts/plots.py +``` + +Run the stages sequentially and avoid concurrent compute-intensive work. The harness refuses to overwrite a run's output. Main benchmarks use one excluded warmup and three sequential, randomized measured rounds. Supplementary checks use a separately seeded randomized order. Compilation, testing, CPU sampling, verbose logs and heap tracing do not overlap clean timing. + +The controlled environment sets `LC_ALL=C`, `OMP_DYNAMIC=FALSE`, `OMP_PROC_BIND=close`, `OMP_PLACES=cores` and `OPENBLAS_NUM_THREADS=1`. Supplementary BLAS cases explicitly vary the last setting. The default host CPU affinity covers logical CPUs 0–11; it is not an isolated machine. Timings measure complete process latency including startup and output. RSS is process peak resident memory, in MiB (1024² bytes); it is distinct from traced heap allocations. + +CPU profiling uses gprofng's nominal 100 ms user-space sampling, with its timer warning preserved and discussed in the report. Hardware counters and syscall tracing were unavailable under the current host permissions. Heap tracing runs independently from the timing suite. + +The structural-check stage first runs the two explicit 149/99-cap cache controls, then checks complete base-pair lists. Original cache-generation files must retain their 150/100 columns. The reader's one-column limit reduction and the confirmed input-stream cleanup defect are documented in the report. The isolated stream probe links the unmodified freshly built library; its process exits after reporting descriptor counts. diff --git a/doc/local-validation-archive/files/profiling/REPORT.md b/doc/local-validation-archive/files/profiling/REPORT.md new file mode 100644 index 00000000..bc031086 --- /dev/null +++ b/doc/local-validation-archive/files/profiling/REPORT.md @@ -0,0 +1,224 @@ +# IntaRNA profiling report + +Profiled 2026-09-19. **56 workloads, 224 clean runs, four CPU profiles, four heap traces, and separate phase and structural checks.** The fresh build passed its regression tests; all timed commands and the independent evidence audit passed. The original checkout was left unchanged. + +The strongest findings are a **5.08× batch speedup at 12 threads**, a default-mode CPU hotspot in internal-loop energy evaluation, and a substantial time/memory tradeoff from windowing. **An input-stream cleanup defect is confirmed: 100 open/cleanup calls leave 100 file descriptors open.** Reusing accessibility data saved **39.0% latency** in a matched test, but the current ED reader lowers the available interaction limit by one nucleotide. Exact and ensemble modes have different hotspot distributions and need separate optimization work. + +![Measured performance overview](results/overview.png) + +## Scope and experimental controls + +The council comprised workload, hotspot/tooling, and experimental-methodology reviewers. They exchanged proposals and challenged stale build provenance, thread scaling, accessibility-limit confounding, sampling validity, output equivalence and memory interpretation. Their recommendations are retained in [workloads](council/workloads.md), [hotspots](council/hotspots.md), and [methodology](council/methodology.md). An independent [final audit](council/final-review.md) checked the actual evidence. + +Target: `refactor-to-c++23`, commit `1b8377989ba79a0f1b8a450163b71bd8ed7c3e45`, reporting IntaRNA 3.4.1. A clean Git archive was built in `profiling/source`; the pre-existing executable was used only for tooling probes. Build: GCC 16.1.0, C++23, release `-O3`, `-g -fno-omit-frame-pointer`, OpenMP, ViennaRNA 2.7.2, Boost 1.85.0. This is an optimized build with profiling-friendly stack frames, not a debug `-O0` build. Binary SHA-256: `3a082dd37a02b7fb07039565e33c0bbed9ee13c0763d6c2828459cd7a6c442b4`. + +Host: AMD Ryzen 5 7530U, six physical cores/twelve logical CPUs, approximately 30.7 GiB RAM, Linux 6.8.0-139, performance governor. All 12 logical CPUs were available; recorded cgroup ancestors imposed no CPU quota or memory maximum. Dynamic frequency and other desktop activity were not disabled. See [environment.json](environment.json) for exact provenance and linked dependencies. + +Each case had one excluded warmup and three measured runs. Runs were sequential, with deterministic randomized order in each round; compilation, tests, CPU sampling, heap tracing and verbose logs were separate. The main matrix and later controls have separate randomization blocks. Wall time covers process startup, input, computation and output. GNU time records user/system CPU, process peak RSS, faults, context switches and filesystem I/O. Medians and min–max ranges are reported; three repetitions do not support strong tail-latency or population confidence claims. + +The baseline fixes `OPENBLAS_NUM_THREADS=1`, `OMP_DYNAMIC=FALSE`, `OMP_PLACES=cores`, `OMP_PROC_BIND=close`, and `LC_ALL=C`. The BLAS experiment explicitly varies that setting. The linked dependency stack otherwise starts extra BLAS workers independently of IntaRNA's thread option. + +Inputs include repository biological examples, the first 48 real 300-nt target records in `NC_000913.fa`, and deterministic synthetic sequences (seed `20260919`). That FASTA is a collection of targets, not one contiguous genome. Length sweeps use nested sequence prefixes. GC-distribution and repeat cases are individual controlled examples, not a representative population of all RNAs. Actual lengths, composition and hashes are in [input manifest](inputs/manifest.json); every command is in [cases.json](cases.json). + +## Runtime and scaling + +### Biological examples + +| Case | Median wall s | Min–max s | Median user / system s | Median peak RSS MiB | Output rows | +|---|---:|---:|---:|---:|---:| +| `bio_fhla_oxys` | 0.053 | 0.053–0.053 | 0.04 / 0.00 | 16.77 | 1 | +| `bio_phob_gcvb` | 0.905 | 0.904–0.910 | 0.89 / 0.00 | 18.29 | 1 | +| `bio_ilve_gcvb` | 1.118 | 1.110–1.120 | 1.11 / 0.00 | 18.16 | 1 | + +These inputs are respectively 112×108, 299×201 and 299×200 nt. The last two have almost identical lengths yet different timing, illustrating sequence dependence. + +With a fixed 100-nt query and default local accessibility settings, target lengths 100, 300, 1,000, 3,000 and 10,000 nt take 0.043, 0.204, 0.810, 2.657, 8.401 seconds. The 1,000→10,000 range increases time 10.37× for 10× target length. This describes the measured regime with bounded folding/interaction windows; it is not an asymptotic complexity proof. + +At 300×100 nt, the composition cases span 0.040 seconds for nonpairing all-A sequences to 2.982 seconds for GC repeats. GC repeats take 11.9× the random 50%-GC-distribution case. Seed density and placement matter in addition to sequence length. Query-length tests intentionally include queries longer than the target; no orientation-invariance claim is made. + +### Algorithm choices + +On the same 100×100 synthetic input, default X/H takes 0.044 s, exact X/M 0.187 s, heuristic S/H 0.095 s, exact S/M 6.417 s, and helix B/H 0.053 s. Exact S is 147× the default on this case. These modes implement different prediction guarantees and models; their timing differences are not evidence of interchangeable answers. + +The suite also covers ensemble P/H and P/M on bounded 40×40 input, seed-only mode, gapped seeds, no lonely pairs, loop limits and suboptimal output. `--noSeed` changes X to S and is labelled accordingly in the council review. Requests for 10 and 100 outputs both return the same ten interactions here; their similar cost does not establish that arbitrary output enumeration is cheap. Seven fixed CSV columns avoid accidentally requesting extra ensemble computations through `--outCsvCols=*`. + +### Parallel throughput + +| Threads | Median s | Speedup | Evaluated pairs/s | Peak RSS MiB | +|---:|---:|---:|---:|---:| +| 1 | 4.944 | 1.00× | 9.71 | 18.71 | +| 2 | 2.556 | 1.93× | 18.78 | 22.98 | +| 4 | 1.457 | 3.39× | 32.94 | 31.36 | +| 6 | 1.104 | 4.48× | 43.50 | 39.95 | +| 12 | 0.974 | 5.08× | 49.29 | 65.25 | + +The denominator is **48 evaluated target–query pairs**, with ChiX length 84; 46 have reportable interactions. All five thread counts produce the same reported coordinates and energies after sorting. A separate comparison including complete base-pair lists confirms equality at one and twelve threads. + +Six threads deliver 88.2% of twelve-thread throughput using 61.2% of its RSS. Twelve was fastest among the tested settings. This is throughput from independent targets; a single unwindowed pair does not have the same parallelism. Source scheduling prioritizes targets, then queries, then windows. + +## Accessibility and memory + +For the 1,000×100 input with an explicit 150-nt interaction cap, local accessibility takes 0.811 s and 20.0 MiB. Global accessibility takes 4.143 s and 86.4 MiB. Disabling accessibility takes 1.235 s, despite removing folding. Accessibility changes energies, admissible boundaries and prediction work; subtracting these runtimes would not isolate folding cost. Separate phase logs provide that attribution. + +Reducing accessibility window/span to 75/75 also changes the scientific model. A separate 75-nt interaction-cap control retains the default 150/100 folding settings. Both are recorded so their differences cannot be mistaken for a pure storage optimization. + +### Reusing saved accessibility data + +The matched target/query limits 149/99 give 0.805 s when computing accessibility and 0.492 s when reloading ED: **1.64× speedup**, with RSS 19.9→14.6 MiB. Coordinates, energies and base-pair lists match. The separately recorded one-time generation took 0.847 s including a complete prediction and serialization; amortize that cost when planning reuse. + +**Cache semantic caveat:** the writer emits columns through its configured maximum, while `AccessibilityFromStream.cpp:90–100` advertises one less than the final header length, with a source comment about dangling ends. The saved files contain their last columns; they are not truncated. Original 100/150-limit round-trip runs therefore become 99/149. Their output happens to agree on this example, but only the explicitly matched controls support the cache timing claim. Do not assume this round trip preserves every requested interaction limit. See the [source and file review](council/accessibility_cache_review.md). No application fix was made. + +### Prediction windowing + +| Mode / 3,000×800 nt input | Whole input seconds | Window300 seconds | Whole peak MiB | Window300 peak MiB | +|---|---:|---:|---:|---:| +| X/H | 41.343 | 71.173 | 24.16 | 24.29 | +| S/H | 21.380 | 59.708 | 127.50 | 24.14 | + +Overlap is 150 and the effective interaction cap is 150. X/H's extension matrices are already bounded by interaction length (at most 144×144 per side with the seven-pair seed here). Whole-sequence accessibility still contributes to peak RSS. Windowing processes 95 target/query window combinations and repeat work, explaining the 1.72× time without useful RSS reduction. + +S/H keeps two matrices covering the full target/query ranges, so windowing reduces RSS by **81.1%**, while taking 2.79× longer. All tested window comparisons preserve the seven reported CSV fields; full structural equivalence was not separately tested for these window cases. The [window review](council/workload_results_review.md) links these observations to source. + +## CPU hotspots and allocations + +CPU workloads are a 48-target biological batch with GcvB; a 3,000×400 default-X pair with accessibility disabled and cap 150; 128 repeated 100-nt targets for exact X; and 32 repeated 60-nt targets for exact ensemble P without seeds. Repeated copies make sampling long enough to resolve coarse rankings. These profiles characterize distinct paths, not mutually comparable biological workloads. + +The user-space collector emits a timer-reset warning on this host. A nominal 100 ms interval gave sampled totals close to recorded process CPU; shorter nominal intervals failed that calibration in tooling probes. Percentages remain coarse sampled evidence. Optimized stack unwinding misses some ancestors, so inclusive rows must not be added or interpreted as exhaustive phase fractions. The complete tables and warning text remain in `profiles/`. + +### CPU hotspots differ by algorithm + +Four separate 100 ms sampling runs captured 1,182 samples. All four prediction outputs matched their clean references after removing the collector banner. Sampled CPU totals closely matched the profiler's independently recorded process CPU time. + +| Profile | Samples | Sampled CPU | Dominant exclusive CPU functions | +|---|---:|---:|---| +| 48 biological promoters × GcvB, default X/H | 368 | 36.8 s | `getE_interLeft` 57.6%; left extension 12.2%; right extension 7.9% | +| Synthetic 3000 × 400, X/H, accessibility off, interaction cap 150 | 296 | 29.6 s | `getE_interLeft` 72.0%; left extension 8.4%; right extension 8.4% | +| 128 repeated 100 × 100 pairs, exact X/M | 211 | 21.1 s | `getBoltzmannWeight` 19.4%; accessibility lookup `getED` 18.5%; full-energy `getE` 9.5%; unnamed libm routine 9.5% | +| 32 repeated 60 × 60 pairs, ensemble P/M without seed | 307 | 30.7 s | `getE_interLeft` 31.6%; ensemble DP `fillHybridZ` 22.8%; unnamed libm routine 12.1%; `updateZ` 8.1% | + +The default predictor spends most of its CPU evaluating internal-loop/stacking energies inside repeated seed-extension recurrences. Exact X/M instead spends most inclusive CPU in candidate scoring (`updateOptima` 83.4%, including its children). Its four boundary-combination loops repeatedly retrieve accessibility energies and compute probability-weighted dangling contributions. The exponential calls are part of the MFE energy model, not evidence of an unnecessary partition-function calculation. Ensemble mode adds Boltzmann-weighted DP and a boundary-keyed partition map; its insertion, reporting and destruction also appear in the profile. + +These are exclusive function shares, not additive call-tree phase percentages. Optimized call ancestry is incomplete: for example, `main` contains only 87% of biological samples. The unnamed libm routines remain unnamed; attribution should not be more specific than the symbols support. + +### Phase measurements + +Built-in timers were collected in separate diagnostic runs. Predictor time includes nested seed filling and reporting. + +| Diagnostic workload | Accessibility calls, summed | Predictor calls, summed | Nested seed filling | +|---|---:|---:|---:| +| Biological batch | 3.815 s, 49 calls | 32.848 s, 48 calls | 0.024 s | +| Exact X/M repeated batch | 1.253 s, 129 calls | 19.694 s, 128 calls | 0.015 s | +| Ensemble P/M repeated batch | 0.079 s, 33 calls | 29.244 s, 32 calls | No seed | +| Hybridization without accessibility | Disabled | Displayed “29 seconds”: approximately 29–30 s | 0.009 s | + +The first three rows sum millisecond-formatted durations; each individual call loses less than a millisecond through formatting. Longer timers truncate to whole seconds/minutes, so the last row is a range. Among the biological batch's logged major phases, about 10% is accessibility and 90% prediction. Its equal-length promoters still take 435–1,145 ms per prediction with 26–63 valid seeds. Seed count and predictor duration correlate strongly in this sample (Pearson r = 0.899), consistent with repeated extension DP per seed. + +### Heap allocation and memory representation + +Each heap trace matched its corresponding clean prediction output. Byte counts below are profiler-tracked allocations, not resident memory. + +| Heap workload | Allocation calls | Cumulative allocated bytes | Peak tracked heap bytes | Outstanding at shutdown | +|---|---:|---:|---:|---:| +| phoB/GcvB, default X/H | 10,997 | 13,796,710 | 4,554,589 | 14,174 bytes / 20 allocations | +| 100 × 100, exact X/M | 6,392 | 4,250,556 | 1,693,265 | 13,672 bytes / 18 allocations | +| 3000 × 800, default X/H | 64,100 | 238,886,646 | 10,750,986 | 26,554 bytes / 29 allocations | +| 3000 × 800, S/H | 61,660 | 181,619,663 | 118,003,656 | 27,802 bytes / 29 allocations | + +The default 3000 × 800 case allocates about 172.5 MB cumulatively at two extension-matrix callsites, across 1,231 allocations at each site: 72% of total allocated bytes. Source inspection identifies per-seed `resize` calls with Boost's default `preserve=true`, which allocates a temporary matrix and copies overlapping contents. This confirms substantial allocation churn, although allocation itself is not a leading sampled CPU hotspot. + +S/H instead makes two 57,600,000-byte matrix allocations. Each represents 3000 × 800 cells with a 24-byte energy/right-boundary record; together they explain most of its 118 MB peak tracked heap. This is a structural memory difference between algorithms, not a leak. + +### Confirmed input-stream cleanup defect + +All four traces retain 9,384 bytes in input-stream allocation stacks. Source `src/IntaRNA/general.cpp:136` casts an input stream to `boost::iostreams::filtering_ostream`, although `newInputStream` creates `filtering_istream`. The cast fails, cleanup is skipped, and the pointer is discarded. + +An isolated probe linked against the freshly built IntaRNA library confirmed resource retention: 100 open/read/delete-input-stream iterations increased open file descriptors from **4 to 104**. The probe and result are saved in `scripts/stream_cleanup_probe.cpp`, `scripts/stream_cleanup_probe.sh`, and `results/stream-cleanup-probe.json`. The OS reclaims these resources when the CLI exits; repeated use within a long-lived process accumulates them. Application source was not changed. + +Other outstanding allocations include a 4,096-byte stdout buffer, 192 bytes in OpenMP, and ViennaRNA allocation chains. Their presence at shutdown alone does not establish additional application leaks. + +## Recommended fixes and optimization priorities + +1. **Fix input-stream cleanup.** Correct the stream-type handling in [general.cpp](source/src/IntaRNA/general.cpp) at line 136 and add a repeated-open/cleanup regression check. The supplied probe confirms retained descriptors against the freshly built library; this is a concrete resource defect, separate from speculative performance improvements. +2. **Default X/H: reduce repeated internal-loop energy work.** `InteractionEnergyVrna::getE_interLeft` is the strongest measured CPU hotspot. Inspect repeated sequence-code and base-pair-type lookups, invariant loop bounds, and validation in the nested extension loops before changing the algorithm. Keep exact energy and structural regression checks; no speedup from such a change has been measured here. +3. **Exact X: reduce repeated boundary-energy and dangling-probability work.** Repeated ED lookups and exponentials have a different cost profile from default X. Evaluate caching or hoisting only with a numerical-correctness check and a measured memory tradeoff; these calculations contribute to the thermodynamic score. +4. **Reuse work across predictions.** Batched targets demonstrably parallelize; matched precomputed ED reuse helps repeated runs. Address or explicitly control the reader/writer limit asymmetry before treating caching as transparent. +5. **Ensemble P: measure recurrence and partition-map changes separately.** `fillHybridZ`, loop energy evaluation, exponentials and boundary-map maintenance contribute distinct costs. Preserve the ensemble definition and numerical behavior when evaluating changes. +6. **Treat matrix storage and windowing by algorithm.** The measured RSS benefit is strong for S/H and negligible for this bounded X/H workload. Heap traces and source review identify allocation reuse candidates, but allocation traffic is not itself proof of a dominant runtime bottleneck. +7. **Keep the benchmark harness as a regression baseline.** Compare proposed patches against the same binary provenance, cases, environment, warmups, repetitions and output checks. Report algorithm changes separately from implementation speedups. + +## Validation and limits + +- Fresh regression tests: 4,281 assertions in 36 API cases, plus all 20 CLI golden cases. No tracked source was changed. Original untracked executables and backup remain intact. +- All 56 benchmark cases completed their warmup and three measured runs. Every case's output was stable. The independent [audit](results/audit.md) verifies 224 records, original stdout/input/binary hashes, resource data, commands, recalculated summaries and structural comparisons. +- Four original no-seed records contain a normal informational log line on stdout. Derived validation excludes that line; the raw evidence is retained. Profiler output comparison similarly excludes only the collector's recognized experiment-creation banner and IntaRNA log lines. +- Excluding the two intentionally varied BLAS cases, the largest min–max range is 4.33% of the median. BLAS-unset and BLAS-12 are noisy: their ranges are 0.081–0.147 s and 0.083–0.132 s, versus 0.052 s with BLAS 1. They demonstrate startup sensitivity, not a precise universal penalty. +- Hardware cycles, IPC, cache misses and branch misses were unavailable: `perf_event_paranoid=4` blocked perf. Syscall tracing was denied; gprofng I/O tracing failed its initialization probe. GNU time resource counters and process CPU evidence are available, but no detailed disk/network/syscall bottleneck claim is made. +- Heap tracing reports allocated bytes, tracked high-water usage and allocations outstanding at process exit. It is not a memory-safety or leak proof. RSS includes additional process mappings and allocator state. Instrumented runs supply attribution, never the clean timing medians. +- This is one build on one laptop-class CPU, with a bounded workload suite and warm page caches. It excludes cross-PR/version comparisons, a full 4,319-target scan, all scientific parameter combinations, NUMA/multi-host scaling, energy-consumption measurement and biological prediction-accuracy benchmarking. Large exact/ensemble runs were deliberately bounded. No universal worst-case or tail-latency claim is made. + +## Complete clean timing results + +Every row below has three measured repetitions; warmups are excluded. "Peak RSS" is the median of the process high-water marks; maximum observed values are also in [summary.csv](results/summary.csv). Exact arguments and experimental controls are in [cases.json](cases.json). + +| Case | Median wall s | Min–max s | Median user / system s | Median peak RSS MiB | Output rows | +|---|---:|---:|---:|---:|---:| +| `bio_fhla_oxys` | 0.053 | 0.053–0.053 | 0.04 / 0.00 | 16.77 | 1 | +| `bio_phob_gcvb` | 0.905 | 0.904–0.910 | 0.89 / 0.00 | 18.29 | 1 | +| `bio_ilve_gcvb` | 1.118 | 1.110–1.120 | 1.11 / 0.00 | 18.16 | 1 | +| `length_t100_q100` | 0.043 | 0.043–0.043 | 0.03 / 0.00 | 16.51 | 1 | +| `length_t300_q100` | 0.204 | 0.204–0.205 | 0.19 / 0.00 | 18.01 | 1 | +| `length_t1000_q100` | 0.810 | 0.807–0.815 | 0.79 / 0.00 | 20.02 | 1 | +| `length_t3000_q100` | 2.657 | 2.653–2.658 | 2.64 / 0.01 | 23.90 | 1 | +| `length_t10000_q100` | 8.401 | 8.389–8.425 | 8.38 / 0.01 | 37.43 | 1 | +| `length_t300_q50` | 0.103 | 0.103–0.104 | 0.09 / 0.00 | 18.30 | 1 | +| `length_t300_q200` | 0.573 | 0.569–0.574 | 0.56 / 0.00 | 18.25 | 1 | +| `length_t300_q400` | 1.210 | 1.209–1.221 | 1.20 / 0.00 | 18.50 | 1 | +| `length_t300_q800` | 3.368 | 3.352–3.369 | 3.35 / 0.01 | 19.38 | 1 | +| `composition_gc20` | 0.199 | 0.197–0.204 | 0.19 / 0.00 | 18.26 | 1 | +| `composition_gc50` | 0.251 | 0.250–0.253 | 0.24 / 0.00 | 18.14 | 1 | +| `composition_gc80` | 0.426 | 0.423–0.426 | 0.41 / 0.00 | 18.26 | 1 | +| `au_repeat` | 0.120 | 0.119–0.121 | 0.11 / 0.00 | 18.26 | 0 | +| `gc_repeat` | 2.982 | 2.963–3.048 | 2.96 / 0.00 | 18.26 | 1 | +| `no_pair` | 0.040 | 0.039–0.040 | 0.03 / 0.00 | 18.14 | 0 | +| `algorithm_default` | 0.044 | 0.043–0.044 | 0.03 / 0.00 | 16.64 | 1 | +| `algorithm_exact_x` | 0.187 | 0.186–0.189 | 0.17 / 0.00 | 16.38 | 1 | +| `algorithm_heuristic_s` | 0.095 | 0.095–0.097 | 0.09 / 0.00 | 16.51 | 1 | +| `algorithm_exact_s` | 6.417 | 6.403–6.464 | 6.41 / 0.00 | 16.64 | 1 | +| `algorithm_helix` | 0.053 | 0.053–0.054 | 0.04 / 0.00 | 16.63 | 1 | +| `algorithm_no_seed` | 0.077 | 0.077–0.077 | 0.06 / 0.00 | 16.63 | 1 | +| `algorithm_gapped_seed` | 0.071 | 0.071–0.073 | 0.06 / 0.00 | 18.04 | 1 | +| `algorithm_seed_only` | 0.032 | 0.032–0.033 | 0.02 / 0.00 | 16.38 | 1 | +| `algorithm_no_lonely_pairs` | 0.035 | 0.034–0.035 | 0.02 / 0.00 | 16.61 | 1 | +| `algorithm_loop0` | 0.033 | 0.032–0.033 | 0.02 / 0.00 | 16.63 | 1 | +| `algorithm_loop30` | 0.066 | 0.066–0.066 | 0.05 / 0.00 | 16.63 | 1 | +| `algorithm_out10` | 0.043 | 0.043–0.043 | 0.03 / 0.00 | 16.63 | 10 | +| `algorithm_out100` | 0.044 | 0.044–0.044 | 0.03 / 0.00 | 16.63 | 10 | +| `ensemble_H` | 0.024 | 0.024–0.025 | 0.01 / 0.00 | 16.11 | 1 | +| `ensemble_M` | 0.126 | 0.125–0.128 | 0.11 / 0.00 | 19.11 | 1 | +| `accessibility_computed` | 0.811 | 0.811–0.812 | 0.80 / 0.00 | 20.03 | 1 | +| `accessibility_none` | 1.235 | 1.229–1.241 | 1.22 / 0.00 | 14.43 | 1 | +| `accessibility_global` | 4.143 | 4.085–4.160 | 4.10 / 0.03 | 86.39 | 1 | +| `accessibility_window75` | 0.582 | 0.580–0.585 | 0.57 / 0.00 | 16.84 | 1 | +| `accessibility_cap75_control` | 0.577 | 0.572–0.597 | 0.56 / 0.00 | 19.63 | 1 | +| `window_300` | 3.435 | 3.432–3.441 | 3.42 / 0.00 | 23.77 | 1 | +| `window_600` | 2.874 | 2.858–2.935 | 2.86 / 0.00 | 23.90 | 1 | +| `throughput_1t` | 4.944 | 4.929–5.006 | 4.85 / 0.08 | 18.71 | 46 | +| `throughput_2t` | 2.556 | 2.527–2.557 | 4.98 / 0.08 | 22.98 | 46 | +| `throughput_4t` | 1.457 | 1.441–1.465 | 5.49 / 0.09 | 31.36 | 46 | +| `throughput_6t` | 1.104 | 1.103–1.130 | 6.03 / 0.10 | 39.95 | 46 | +| `throughput_12t` | 0.974 | 0.970–0.997 | 9.99 / 0.21 | 65.25 | 46 | +| `memory_t3000_q800` | 41.343 | 41.024–41.381 | 41.33 / 0.01 | 24.16 | 1 | +| `memory_window300` | 71.173 | 71.009–71.297 | 71.15 / 0.01 | 24.29 | 1 | +| `cache_computed_control` | 0.821 | 0.806–0.825 | 0.81 / 0.00 | 20.02 | 1 | +| `cache_reused` | 0.490 | 0.490–0.494 | 0.48 / 0.00 | 14.62 | 1 | +| `blas_one_control` | 0.052 | 0.052–0.053 | 0.04 / 0.00 | 16.77 | 1 | +| `blas_unset` | 0.147 | 0.081–0.147 | 1.48 / 0.00 | 18.03 | 1 | +| `blas_twelve` | 0.086 | 0.083–0.132 | 0.89 / 0.00 | 18.02 | 1 | +| `memory_s` | 21.380 | 21.343–21.418 | 21.31 / 0.06 | 127.50 | 1 | +| `memory_s_window300` | 59.708 | 59.652–59.799 | 59.56 / 0.15 | 24.14 | 1 | +| `cache_matched_computed` | 0.805 | 0.805–0.808 | 0.79 / 0.00 | 19.89 | 1 | +| `cache_matched_reused` | 0.492 | 0.491–0.494 | 0.48 / 0.00 | 14.62 | 1 | + +## Reproduction and evidence + +See [README.md](README.md) for the sequential build/benchmark/profile commands and artifact map. The machine-readable [summary](results/summary.json), [validated runs](results/validated-runs.jsonl), [raw runs](results/runs.jsonl), [CPU/heap output checks](profiles/validated-outputs.json), [phase duration bounds](profiles/phase-summary.json), and [council reviews](council/) preserve the basis for the findings. The application source and fresh binaries are retained locally under `source/`. diff --git a/doc/local-validation-archive/files/profiling/scripts/audit.py b/doc/local-validation-archive/files/profiling/scripts/audit.py new file mode 100644 index 00000000..5df6119f --- /dev/null +++ b/doc/local-validation-archive/files/profiling/scripts/audit.py @@ -0,0 +1,420 @@ +#!/usr/bin/env python3 +"""Audit existing IntaRNA profiling evidence; never runs a benchmark. + +Run after benchmark.py, supplement.py, and structural_checks.py finish: + python3 profiling/scripts/audit.py +The default expectation is 47 original + 9 supplementary cases, each with one +warmup and three measurements. --allow-incomplete permits partial coverage, but +does not suppress evidence-integrity failures. All findings are preserved in +results/audit.json and results/audit.md. Python standard library only. +""" +import argparse +import csv +import hashlib +import io +import json +import math +import statistics +import sys +from collections import Counter, defaultdict +from datetime import datetime, timezone +from pathlib import Path + +from benchmark import canonical + +SUPPLEMENTARY = { + 'cache_computed_control', 'cache_reused', 'blas_one_control', 'blas_unset', + 'blas_twelve', 'memory_s', 'memory_s_window300', + 'cache_matched_computed', 'cache_matched_reused', +} +HEADER = ['id1', 'id2', 'start1', 'end1', 'start2', 'end2', 'E'] +CORRECTED_FIELDS = {'csv_valid', 'canonical_sha256', 'output_rows'} +TIME_KEYS = ['wall_s', 'user_s', 'system_s', 'max_rss_kib', 'major_faults', + 'minor_faults', 'voluntary_context_switches', + 'involuntary_context_switches', 'fs_inputs', 'fs_outputs', + 'time_exit_code'] + + +def digest(data): + return hashlib.sha256(data).hexdigest() + + +def identity(row): + return (row.get('case'), row.get('phase'), row.get('repetition')) + + +def same(a, b): + if isinstance(a, (int, float)) and not isinstance(a, bool): + return isinstance(b, (int, float)) and math.isclose(a, b, rel_tol=1e-11, abs_tol=1e-11) + return a == b + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('--root', type=Path, default=Path(__file__).resolve().parents[1]) + parser.add_argument('--expected-original', type=int, default=47) + parser.add_argument('--expected-supplementary', type=int, default=9) + parser.add_argument('--repetitions', type=int, default=3) + parser.add_argument('--allow-incomplete', action='store_true') + args = parser.parse_args() + root = args.root.resolve() + result = root / 'results' + errors, warnings, incompleteness = [], [], [] + + def problem(message, incomplete=False): + (incompleteness if incomplete else errors).append(message) + + def read_json(path, default): + try: + return json.loads(path.read_text()) + except (OSError, ValueError) as exc: + problem(f'{path.relative_to(root)}: {exc}', incomplete=not path.exists()) + return default + + def read_jsonl(path): + try: + lines = path.read_text().splitlines() + except OSError as exc: + problem(f'{path.relative_to(root)}: {exc}', incomplete=True) + return [] + rows = [] + for number, line in enumerate(lines, 1): + try: + rows.append(json.loads(line)) + except ValueError as exc: + problem(f'{path.name}:{number}: invalid JSON: {exc}') + return rows + + cases = read_json(root / 'cases.json', []) + definitions = {case['name']: case for case in cases} + if len(definitions) != len(cases): + problem('Duplicate names in cases.json') + original = [name for name in definitions if name not in SUPPLEMENTARY] + supplementary = [name for name in definitions if name in SUPPLEMENTARY] + for label, observed, expected in [('original', len(original), args.expected_original), + ('supplementary', len(supplementary), args.expected_supplementary)]: + if observed != expected: + problem(f'Case definition coverage: {observed} {label}, expected {expected}', incomplete=True) + + raw = read_jsonl(result / 'runs.jsonl') + validated = read_jsonl(result / 'validated-runs.jsonl') + summaries = read_json(result / 'summary.json', []) + failures_file = read_json(result / 'failures.json', []) + environment = read_json(root / 'environment.json', {}) + expected_binary = environment.get('binary_sha256') + binary_path = root / 'source/src/bin/IntaRNA' + if binary_path.exists(): + if digest(binary_path.read_bytes()) != expected_binary: + problem('Current executable SHA-256 differs from environment.json') + else: + warnings.append('Current executable unavailable; saved per-run provenance still audited') + + manifest_checks = [] + for manifest_path in [root / 'inputs/manifest.json', root / 'inputs/cache/manifest.json']: + manifest = read_json(manifest_path, {}) + for name, metadata in manifest.items(): + path = manifest_path.parent / name + valid = path.exists() and digest(path.read_bytes()) == metadata.get('sha256') + manifest_checks.append({'path': str(path.relative_to(root)), 'hash_valid': valid}) + if not valid: + problem(f'Input manifest SHA-256 mismatch/missing file: {path.relative_to(root)}') + + raw_keys = Counter(identity(row) for row in raw) + validated_keys = Counter(identity(row) for row in validated) + for key, count in raw_keys.items(): + if count != 1: + problem(f'Duplicate raw record {key}: {count}') + if raw_keys != validated_keys: + problem('validated-runs.jsonl does not contain exactly the raw run identities') + validated_by_key = {identity(row): row for row in validated} + corrected = [] + corrections = [] + stderr_nonempty = [] + timing_discrepancies = [] + for record in raw: + key = identity(record) + name, phase, repetition = key + label = f'{name}.{phase}{repetition}' + if name not in definitions: + problem(f'Unknown case in raw evidence: {name}') + elif record.get('command', [])[-len(definitions[name]['args']):] != definitions[name]['args']: + problem(f'{label}: recorded command differs from case definition') + if record.get('env_overrides', {}) != definitions.get(name, {}).get('env_overrides', {}): + problem(f'{label}: environment override differs from case definition') + if record.get('binary_sha256') != expected_binary: + problem(f'{label}: binary SHA-256 differs from saved environment') + if not isinstance(record.get('elapsed_s'), (int, float)) or record.get('elapsed_s', 0) <= 0: + problem(f'{label}: invalid elapsed time') + try: + started = datetime.fromisoformat(record['started_utc']) + ended = datetime.fromisoformat(record['ended_utc']) + if ended < started: + problem(f'{label}: end timestamp precedes start') + except (KeyError, ValueError, TypeError): + problem(f'{label}: missing/invalid UTC timestamps') + + stdout_path = result / (label + '.stdout') + try: + output = stdout_path.read_bytes() + except OSError as exc: + problem(f'{label}: missing stdout: {exc}') + continue + if digest(output) != record.get('output_sha256'): + problem(f'{label}: raw stdout SHA-256 mismatch') + normalized = canonical(output) + try: + rows = list(csv.reader(io.StringIO(normalized.decode()), delimiter=';')) + csv_valid = bool(rows) and rows[0] == HEADER and all(len(row) == len(HEADER) for row in rows) + numeric_valid = csv_valid and all( + all(int(value) != 0 for value in row[2:6]) and math.isfinite(float(row[6])) + for row in rows[1:]) + except (ValueError, UnicodeError): + rows, csv_valid, numeric_valid = [], False, False + derived = dict(record, canonical_sha256=digest(normalized), csv_valid=csv_valid, + output_rows=max(0, len(rows) - 1)) + corrected.append(derived) + delta = {field: {'raw': record.get(field), 'derived': derived[field]} + for field in CORRECTED_FIELDS if record.get(field) != derived[field]} + if delta: + corrections.append({'case': name, 'phase': phase, 'repetition': repetition, + 'fields': delta, 'comment_lines': sum(line.startswith(b'#') for line in output.splitlines())}) + if record.get('exit_code') == 0 and not numeric_valid: + problem(f'{label}: successful command produced invalid CSV schema or numeric fields') + if key in validated_by_key and validated_by_key[key] != derived: + problem(f'{label}: validated record differs from independent derivation (or altered immutable fields)') + + stderr_path = result / (label + '.stderr') + if not stderr_path.exists(): + problem(f'{label}: missing stderr file') + elif stderr_path.stat().st_size: + stderr_nonempty.append({'run': label, 'bytes': stderr_path.stat().st_size, + 'preview': stderr_path.read_text(errors='replace')[:600]}) + time_path = result / (label + '.time') + if not time_path.exists(): + problem(f'{label}: missing GNU time file') + continue + values = time_path.read_text().strip().splitlines() + if values and len(values[-1].split('\t')) == len(TIME_KEYS): + try: + time_fields = dict(zip(TIME_KEYS, map(float, values[-1].split('\t')))) + for field, value in time_fields.items(): + if not same(value, record.get(field)): + problem(f'{label}: {field} differs from GNU time evidence') + if record.get('exit_code') == 0 and time_fields['time_exit_code'] != 0: + problem(f'{label}: GNU time and harness exit codes disagree') + difference = record['elapsed_s'] - time_fields['wall_s'] + timing_discrepancies.append({'run': label, 'python_minus_gnu_s': difference}) + if difference < -0.02: + problem(f'{label}: monotonic wall time implausibly below GNU time') + except (ValueError, TypeError): + problem(f'{label}: invalid GNU time data') + elif record.get('exit_code') == 0: + problem(f'{label}: successful run lacks parseable GNU time resources') + + by_case = defaultdict(list) + for row in corrected: + by_case[row['case']].append(row) + expected_slots = {('warmup', 0)} | {('measure', i) for i in range(1, args.repetitions + 1)} + coverage = [] + scatter = [] + recomputed = {} + for name, definition in definitions.items(): + records = by_case[name] + observed = Counter((row['phase'], row['repetition']) for row in records) + missing = sorted(expected_slots - observed.keys()) + extra = sorted(observed.keys() - expected_slots) + successful = [row for row in records if row['exit_code'] == 0] + measured = [row for row in successful if row['phase'] == 'measure'] + status = {'case': name, 'warmups': sum(row['phase'] == 'warmup' for row in records), + 'measurements': sum(row['phase'] == 'measure' for row in records), + 'successful_measurements': len(measured), 'missing': missing, 'extra': extra, + 'failed_runs': [list(identity(row)) for row in records if row['exit_code'] != 0], + 'timeouts': sum(bool(row.get('timeout')) for row in records), + 'stable_all_successful_outputs': len({row['canonical_sha256'] for row in successful}) <= 1} + coverage.append(status) + if missing: + problem(f'{name}: missing slots {missing}', incomplete=True) + if extra: + problem(f'{name}: unexpected slots {extra}') + if not status['stable_all_successful_outputs']: + problem(f'{name}: successful warmup/measurement outputs differ') + if not measured: + continue + times = [row['elapsed_s'] for row in measured] + med = statistics.median(times) + entry = {'case': name, 'category': definition['category'], 'n': len(measured), + 'median_s': med, 'min_s': min(times), 'max_s': max(times), + 'stable_output': len({row['canonical_sha256'] for row in measured}) == 1, + 'rows': measured[0]['output_rows'], 'canonical_sha256': measured[0]['canonical_sha256']} + for source, target, divisor, aggregate in [ + ('user_s', 'median_user_s', 1, statistics.median), + ('system_s', 'median_system_s', 1, statistics.median), + ('max_rss_kib', 'median_rss_mib', 1024, statistics.median), + ('max_rss_kib', 'peak_rss_mib', 1024, max)]: + if all(source in row for row in measured): + entry[target] = aggregate(row[source] / divisor for row in measured) + recomputed[name] = entry + scatter.append({'case': name, 'n': len(times), 'times_s': times, 'median_s': med, + 'range_s': max(times) - min(times), + 'range_over_median_pct': 100 * (max(times) - min(times)) / med, + 'sample_cv_pct': 100 * statistics.stdev(times) / statistics.mean(times) if len(times) > 1 else None, + 'max_deviation_from_median_pct': 100 * max(abs(t - med) for t in times) / med, + 'median_cpu_per_wall': statistics.median((r.get('user_s', 0) + r.get('system_s', 0)) / r['elapsed_s'] for r in measured)}) + + saved_summary = {row['case']: row for row in summaries} + if len(saved_summary) != len(summaries): + problem('Duplicate cases in summary.json') + if saved_summary.keys() != recomputed.keys(): + problem('summary.json case coverage differs from successful measured runs') + for name, entry in recomputed.items(): + for field, value in entry.items(): + if not same(value, saved_summary.get(name, {}).get(field)): + problem(f'{name}: summary field {field} differs from independent recomputation') + csv_path = result / 'summary.csv' + if csv_path.exists(): + csv_summary = list(csv.DictReader(csv_path.open())) + if len(csv_summary) != len(summaries): + problem('summary.csv and summary.json row counts differ') + for row in csv_summary: + reference = saved_summary.get(row.get('case'), {}) + for field, value in reference.items(): + if row.get(field) != str(value): + problem(f'{row.get("case")}: summary.csv field {field} differs from JSON summary') + else: + problem('summary.csv missing', incomplete=True) + failed = [row for row in corrected if row['exit_code'] != 0 or not row['csv_valid']] + if Counter(identity(row) for row in failed) != Counter(identity(row) for row in failures_file): + problem('failures.json differs from independently derived failures') + if failed: + warnings.append(f'{len(failed)} command/schema failures retained; successful-only summaries require coverage caveat') + + groups = [] + def equality(label, names, required=False, performance_comparison_valid=True, caveat=None): + present = {name: sorted({r['canonical_sha256'] for r in by_case[name] + if r['exit_code'] == 0 and r['csv_valid']}) for name in names if name in definitions} + complete = len(present) == len(names) and all(present.values()) + equal = complete and len({value for hashes in present.values() for value in hashes}) == 1 + groups.append({'group': label, 'cases': names, 'hashes': present, + 'complete': complete, 'equal': equal, 'required': required, + 'performance_comparison_valid': performance_comparison_valid, + 'caveat': caveat}) + if complete and not equal: + (errors if required else warnings).append(f'{label}: canonical outputs differ') + equality('thread_scaling', ['throughput_1t', 'throughput_2t', 'throughput_4t', 'throughput_6t', 'throughput_12t'], True) + equality('prediction_windows', ['length_t3000_q100', 'window_300', 'window_600']) + equality('memory_windows_model_x', ['memory_t3000_q800', 'memory_window300']) + equality('memory_windows_model_s', ['memory_s', 'memory_s_window300']) + equality('cached_accessibility_original_observational', + ['accessibility_computed', 'cache_computed_control', 'cache_reused'], + performance_comparison_valid=False, + caveat='ED reload reduces maximum available length by one for dangling-end treatment ' + '(AccessibilityFromStream.cpp:90). Original computed/reused effective caps differ ' + '(target/query 150/100 versus 149/99). Equal observed outputs do not remove this confound.') + equality('cached_accessibility_matched_caps', ['cache_matched_computed', 'cache_matched_reused'], True, + caveat='Both commands explicitly cap target/query interaction lengths at 149/99; ' + 'cache generation remains a separate one-time cost.') + equality('blas_environment', ['bio_fhla_oxys', 'blas_one_control', 'blas_unset', 'blas_twelve'], True) + equality('duplicate_default_case', ['length_t100_q100', 'algorithm_default'], True) + + matched_cases = [definitions.get(name) for name in ['cache_matched_computed', 'cache_matched_reused']] + for case in matched_cases: + if case is not None: + if any(arg.startswith('--intLenMax=') for arg in case['args']) or not {'--qIntLenMax=99', '--tIntLenMax=149'}.issubset(case['args']): + problem(f"{case['name']}: matched cache comparison lacks explicit common 149/99 caps") + + structural = read_json(result / 'structural/checks.json', {}) + structural_hashes = {} + for row in structural.get('runs', []): + path = result / 'structural' / (row['case'] + '.stdout') + if not path.exists(): + problem(f"Structural output missing for {row['case']}") + continue + data = canonical(path.read_bytes()) + structural_hashes[row['case']] = digest(data) + if digest(data) != row.get('canonical_sha256'): + problem(f"Structural stdout hash mismatch for {row['case']}") + if row.get('exit_code') != 0: + problem(f"Structural command failed for {row['case']}") + cells = list(csv.reader(io.StringIO(data.decode()), delimiter=';')) + if not cells or cells[0] != HEADER + ['bpList'] or any(len(cell) != 8 for cell in cells): + problem(f"Structural CSV invalid for {row['case']}") + structural_comparisons = {} + for label, first, second, required in [ + ('threads_1_vs_12', 'throughput_1t', 'throughput_12t', True), + ('cached_vs_computed', 'cache_computed_control', 'cache_reused', False), + ('cache_matched_caps', 'cache_matched_computed', 'cache_matched_reused', True)]: + equal = first in structural_hashes and second in structural_hashes and structural_hashes[first] == structural_hashes[second] + structural_comparisons[label] = {'equal': equal, 'required': required, + 'performance_comparison_valid': label != 'cached_vs_computed'} + if structural and structural.get('comparisons', {}).get(label) != equal: + problem(f'Structural comparison summary mismatch: {label}') + if required and first in structural_hashes and second in structural_hashes and not equal: + problem(f'Structural base-pair outputs differ: {label}') + + thread_scaling = [] + baseline = recomputed.get('throughput_1t', {}).get('median_s') + if baseline: + for threads in [1, 2, 4, 6, 12]: + item = recomputed.get(f'throughput_{threads}t') + if item: + speedup = baseline / item['median_s'] + thread_scaling.append({'threads': threads, 'median_s': item['median_s'], + 'speedup': speedup, 'parallel_efficiency': speedup / threads, + 'predictions_per_s': 48 / item['median_s']}) + + errors = list(dict.fromkeys(errors)) + warnings = list(dict.fromkeys(warnings)) + incompleteness = list(dict.fromkeys(incompleteness)) + report = {'audited_at_utc': datetime.now(timezone.utc).isoformat(), + 'status': 'fail' if errors or (incompleteness and not args.allow_incomplete) else ('partial' if incompleteness else 'pass'), + 'expected_original_cases': args.expected_original, + 'expected_supplementary_cases': args.expected_supplementary, + 'defined_cases': len(cases), 'raw_records': len(raw), 'validated_records': len(validated), + 'expected_records': (args.expected_original + args.expected_supplementary) * (args.repetitions + 1), + 'errors': errors, 'warnings': warnings, 'incompleteness': incompleteness, + 'input_hash_checks': manifest_checks, 'coverage': coverage, + 'raw_metadata_corrections': corrections, 'failures': failed, + 'nonempty_stderr': stderr_nonempty, 'output_equality_groups': groups, + 'structural_comparisons': structural_comparisons, + 'scatter': sorted(scatter, key=lambda x: x['range_over_median_pct'], reverse=True), + 'thread_scaling': thread_scaling, + 'largest_wrapper_time_differences': sorted(timing_discrepancies, key=lambda x: abs(x['python_minus_gnu_s']), reverse=True)[:10], + 'notes': ['Canonicalization uses the current benchmark.canonical and ignores # comment lines.', + 'Raw records and benchmark evidence are never edited by this audit.', + 'Window equality is observed, not presumed; cache/BLAS/thread equality is checked explicitly.', + 'Only the matched 149/99 cache pair supports a cache performance comparison; original cache timings are confounded.', + 'Three samples support descriptive scatter, not tight confidence intervals.', + 'Throughput uses 48 attempted target-query predictions, not reported CSV row count.']} + result.mkdir(exist_ok=True) + (result / 'audit.json').write_text(json.dumps(report, indent=2) + '\n') + lines = ['# Profiling evidence audit', '', f"Status: **{report['status'].upper()}**", '', + f"{len(cases)} case definitions; {len(raw)}/{report['expected_records']} raw records; " + f"{len(validated)} validated records; {len(failed)} command/schema failures.", '', + f'{len(corrections)} raw collection-time metadata records corrected from preserved stdout. ' + 'Original raw records remain unchanged.', '', '## Findings', ''] + lines += [f'- ERROR: {message}' for message in errors] + lines += [f'- INCOMPLETE: {message}' for message in incompleteness] + lines += [f'- Note: {message}' for message in warnings] + if not errors and not incompleteness and not warnings: + lines.append('- No integrity, coverage, or equality issues found.') + lines += ['', '## Output comparisons', '', '| Comparison | Complete | Equal | Required | Performance comparison valid |', '|---|---|---|---|---|'] + lines += [f"| {g['group']} | {g['complete']} | {g['equal']} | {g['required']} | {g['performance_comparison_valid']} |" for g in groups] + lines += ['', 'Original cached/computed timings are observational: their effective maximum interaction lengths differ by one. ' + 'Only the explicit target/query 149/99 matched pair supports the cache performance comparison. ' + 'The JSON companion separately checks base-pair-list output equality.', ''] + lines += ['', '## Largest relative timing ranges', '', '| Case | n | Median s | Min–max s | Range / median |', '|---|---:|---:|---:|---:|'] + for item in report['scatter'][:15]: + lines.append(f"| {item['case']} | {item['n']} | {item['median_s']:.6f} | " + f"{min(item['times_s']):.6f}–{max(item['times_s']):.6f} | {item['range_over_median_pct']:.1f}% |") + lines += ['', '## Coverage', '', '| Case | Warmup | Measured | Successful measured | Timeouts | Stable outputs |', '|---|---:|---:|---:|---:|---|'] + lines += [f"| {c['case']} | {c['warmups']} | {c['measurements']} | {c['successful_measurements']} | {c['timeouts']} | {c['stable_all_successful_outputs']} |" for c in coverage] + lines += ['', 'The JSON companion contains detailed provenance, corrections, scatter, and file checks.', ''] + (result / 'audit.md').write_text('\n'.join(lines)) + print(f"Audit {report['status']}: {len(cases)} cases, {len(raw)} records, {len(errors)} integrity errors, " + f'{len(incompleteness)} incomplete-coverage findings, {len(warnings)} warnings') + print(result / 'audit.md') + return 1 if report['status'] == 'fail' else 0 + + +if __name__ == '__main__': + sys.exit(main()) diff --git a/doc/local-validation-archive/files/profiling/scripts/benchmark.py b/doc/local-validation-archive/files/profiling/scripts/benchmark.py new file mode 100644 index 00000000..93fcbe9d --- /dev/null +++ b/doc/local-validation-archive/files/profiling/scripts/benchmark.py @@ -0,0 +1,217 @@ +#!/usr/bin/env python3 +"""Reproducible sequential IntaRNA benchmarks; no third-party Python dependencies.""" +import argparse, csv, hashlib, json, os, platform, random, shutil, signal +import statistics, subprocess, time, threading +from datetime import datetime, timezone +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[1] +BIN = ROOT / 'source/src/bin/IntaRNA' +INPUT = ROOT / 'inputs' +RESULT = ROOT / 'results' +ENV = dict(os.environ, LC_ALL='C', OMP_DYNAMIC='FALSE', OMP_PROC_BIND='close', + OMP_PLACES='cores', OPENBLAS_NUM_THREADS='1') + +def sha(path): + return hashlib.sha256(Path(path).read_bytes()).hexdigest() + +def fasta(path): + records = [] + for line in Path(path).read_text().splitlines(): + if line.startswith('>'): + records.append([line[1:], '']) + elif line.strip(): + records[-1][1] += line.strip().upper().replace('T', 'U') + return records + +def write_fasta(name, records): + path = INPUT / (name + '.fa') + path.write_text(''.join(f'>{name}\n{seq}\n' for name, seq in records)) + return str(path) + +def prepare(): + INPUT.mkdir(exist_ok=True) + docs = ROOT / 'source/doc/handson' + for name in ['fhlA', 'OxyS', 'phoB', 'GcvB', 'GcvB.ST', 'ilvE', 'ChiX_NC_000913']: + write_fasta(name, fasta(docs / (name + '.fasta'))) + write_fasta('promoters48', fasta(docs / 'NC_000913.fa')[:48]) + rng = random.Random(20260919) + target = ''.join(rng.choices('ACGU', k=10000)) + query = ''.join(rng.choices('ACGU', k=800)) + for n in [40, 60, 100, 300, 1000, 3000, 10000]: + write_fasta(f't{n}', [(f'synthetic_target_{n}', target[:n])]) + for n in [40, 50, 60, 100, 200, 400, 800]: + write_fasta(f'q{n}', [(f'synthetic_query_{n}', query[:n])]) + for gc in [20, 50, 80]: + weights = [(100-gc)/2, gc/2, gc/2, (100-gc)/2] + write_fasta(f'gc{gc}t', [(f'gc{gc}target', ''.join(rng.choices('ACGU', weights=weights, k=300)))]) + write_fasta(f'gc{gc}q', [(f'gc{gc}query', ''.join(rng.choices('ACGU', weights=weights, k=100)))]) + for alphabet, name in [('AU', 'au_repeat'), ('GC', 'gc_repeat'), ('A', 'no_pair')]: + write_fasta(name+'t', [(name+'target', (alphabet*300)[:300])]) + write_fasta(name+'q', [(name+'query', (alphabet*100)[:100])]) + cases = [] + def add(name, category, t, q, extra=(), threads=1): + args = ['--target='+str(INPUT/(t+'.fa')), '--query='+str(INPUT/(q+'.fa')), + '--outMode=C', '--outCsvCols=id1,id2,start1,end1,start2,end2,E', + '--threads='+str(threads)] + list(extra) + cases.append(dict(name=name, category=category, args=args, threads=threads)) + add('bio_fhla_oxys', 'biological', 'fhlA', 'OxyS') + add('bio_phob_gcvb', 'biological', 'phoB', 'GcvB') + add('bio_ilve_gcvb', 'biological', 'ilvE', 'GcvB.ST') + for n in [100, 300, 1000, 3000, 10000]: + add(f'length_t{n}_q100', 'target_length', f't{n}', 'q100') + for n in [50, 200, 400, 800]: + add(f'length_t300_q{n}', 'query_length', 't300', f'q{n}') + for gc in [20, 50, 80]: + add(f'composition_gc{gc}', 'composition', f'gc{gc}t', f'gc{gc}q') + for name in ['au_repeat', 'gc_repeat', 'no_pair']: + add(name, 'composition', name+'t', name+'q') + for name, args in [ + ('default', []), ('exact_x', ['--mode=M']), ('heuristic_s', ['--model=S']), + ('exact_s', ['--mode=M', '--model=S']), ('helix', ['--model=B']), + ('no_seed', ['--noSeed']), ('gapped_seed', ['--seedMaxUP=2']), + ('seed_only', ['--mode=S']), ('no_lonely_pairs', ['--outNoLP']), + ('loop0', ['--intLoopMax=0']), ('loop30', ['--intLoopMax=30']), + ('out10', ['--outNumber=10']), ('out100', ['--outNumber=100'])]: + add('algorithm_'+name, 'algorithm', 't100', 'q100', args) + for mode in ['H', 'M']: + add('ensemble_'+mode, 'ensemble', 't40', 'q40', ['--model=P', '--mode='+mode, '--noSeed']) + for name, args in [ + ('computed', ['--intLenMax=150']), ('none', ['--intLenMax=150', '--acc=N']), + ('global', ['--intLenMax=150', '--accW=0', '--accL=0']), + ('window75', ['--intLenMax=75', '--accW=75', '--accL=75']), + ('cap75_control', ['--intLenMax=75'])]: + add('accessibility_'+name, 'accessibility', 't1000', 'q100', args) + for width in [300, 600]: + add(f'window_{width}', 'prediction_windows', 't3000', 'q100', + ['--intLenMax=150', '--windowWidth='+str(width), '--windowOverlap=150']) + for threads in [1, 2, 4, 6, 12]: + add(f'throughput_{threads}t', 'thread_scaling', 'promoters48', 'ChiX_NC_000913', threads=threads) + add('memory_t3000_q800', 'memory_stress', 't3000', 'q800') + add('memory_window300', 'memory_stress', 't3000', 'q800', + ['--intLenMax=150', '--windowWidth=300', '--windowOverlap=150']) + (ROOT/'cases.json').write_text(json.dumps(cases, indent=2)+'\n') + (ROOT/'inputs/manifest.json').write_text(json.dumps({p.name: dict(sha256=sha(p), + records=[dict(id=i, length=len(s), gc_fraction=round((s.count('G')+s.count('C'))/len(s),5)) for i,s in fasta(p)]) + for p in sorted(INPUT.glob('*.fa'))}, indent=2)+'\n') + print(f'Prepared {len(cases)} cases', flush=True) + +def canonical(data): + lines = [line for line in data.splitlines() if not line.startswith(b'#')] + return b'\n'.join(lines[:1]+sorted(lines[1:])) + b'\n' + +def run(case, phase, repetition, timeout): + dest = RESULT / f"{case['name']}.{phase}{repetition}" + if Path(str(dest)+'.stdout').exists(): + raise RuntimeError('Refusing to overwrite prior evidence: '+str(dest)) + cmd = ['/usr/bin/time', '-f', '%e\t%U\t%S\t%M\t%F\t%R\t%w\t%c\t%I\t%O\t%x', + '-o', str(dest)+'.time', str(BIN)] + case['args'] + started_utc=datetime.now(timezone.utc).isoformat() + binary_hash=sha(BIN) + run_env=ENV.copy() + for key,value in case.get('env_overrides',{}).items(): + if value is None: + run_env.pop(key,None) + else: + run_env[key]=value + start = time.perf_counter() + with open(str(dest)+'.stdout', 'wb') as out, open(str(dest)+'.stderr', 'wb') as err: + proc = subprocess.Popen(cmd, stdout=out, stderr=err, env=run_env, start_new_session=True) + timed_out = False + def expire(): + nonlocal timed_out + if proc.poll() is None: + timed_out = True + try: + os.killpg(proc.pid, signal.SIGKILL) + except ProcessLookupError: + pass + watchdog = threading.Timer(timeout, expire) + watchdog.start() + rc = proc.wait() # blocking wait avoids timeout polling's ~50 ms timing bias + watchdog.cancel() + if timed_out: + rc = 124 + elapsed = time.perf_counter()-start + csv_rows=list(csv.reader([line for line in Path(str(dest)+'.stdout').read_text().splitlines() + if not line.startswith('#')],delimiter=';')) + csv_valid=bool(csv_rows) and csv_rows[0]==['id1','id2','start1','end1','start2','end2','E'] and all(len(r)==7 for r in csv_rows) + vals = Path(str(dest)+'.time').read_text().strip().splitlines() + row = dict(case=case['name'], category=case['category'], phase=phase, + repetition=repetition, elapsed_s=elapsed, exit_code=rc, timeout=timed_out, + started_utc=started_utc, ended_utc=datetime.now(timezone.utc).isoformat(), + binary_sha256=binary_hash, csv_valid=csv_valid, + env_overrides=case.get('env_overrides',{}), + command=cmd, load_average=os.getloadavg(), + output_sha256=sha(str(dest)+'.stdout'), + canonical_sha256=hashlib.sha256(canonical(Path(str(dest)+'.stdout').read_bytes())).hexdigest(), + output_rows=max(0,len(csv_rows)-1)) + if vals and len(vals[-1].split('\t')) == 11: + keys=['wall_s','user_s','system_s','max_rss_kib','major_faults','minor_faults', + 'voluntary_context_switches','involuntary_context_switches','fs_inputs','fs_outputs','time_exit_code'] + row.update(dict(zip(keys, map(float, vals[-1].split('\t'))))) + with (RESULT/'runs.jsonl').open('a') as f: + f.write(json.dumps(row)+'\n') + print(f"{phase} {repetition} {case['name']}: {elapsed:.3f}s rc={rc} RSS={row.get('max_rss_kib', 0)/1024:.1f}MiB", flush=True) + return row + +def summarize(): + rows = [json.loads(line) for line in (RESULT/'runs.jsonl').read_text().splitlines()] + # Re-derive schema and equality from preserved stdout, ignoring IntaRNA's # INFO lines. + # Original run records remain unchanged as evidence of collection-time parsing. + for row in rows: + path=RESULT/f"{row['case']}.{row['phase']}{row['repetition']}.stdout" + data=canonical(path.read_bytes()) + cells=list(csv.reader(data.decode().splitlines(),delimiter=';')) + row['canonical_sha256']=hashlib.sha256(data).hexdigest() + row['csv_valid']=bool(cells) and cells[0]==['id1','id2','start1','end1','start2','end2','E'] and all(len(r)==7 for r in cells) + row['output_rows']=max(0,len(cells)-1) + (RESULT/'validated-runs.jsonl').write_text(''.join(json.dumps(r)+'\n' for r in rows)) + summary = [] + for name in dict.fromkeys(r['case'] for r in rows): + rr = [r for r in rows if r['case']==name and r['phase']=='measure' and r['exit_code']==0] + if not rr: + continue + times=[r['elapsed_s'] for r in rr] + summary.append(dict(case=name, category=rr[0]['category'], n=len(rr), + median_s=statistics.median(times), min_s=min(times), max_s=max(times), + median_user_s=statistics.median(r['user_s'] for r in rr), + median_system_s=statistics.median(r['system_s'] for r in rr), + median_rss_mib=statistics.median(r['max_rss_kib']/1024 for r in rr), + peak_rss_mib=max(r['max_rss_kib']/1024 for r in rr), + stable_output=len(set(r['canonical_sha256'] for r in rr))==1, + rows=rr[0]['output_rows'], canonical_sha256=rr[0]['canonical_sha256'])) + (RESULT/'summary.json').write_text(json.dumps(summary, indent=2)+'\n') + (RESULT/'failures.json').write_text(json.dumps([r for r in rows if r['exit_code'] or not r['csv_valid']],indent=2)+'\n') + with (RESULT/'summary.csv').open('w') as f: + writer=csv.DictWriter(f,fieldnames=list(summary[0])) + writer.writeheader(); writer.writerows(summary) + +def main(): + parser=argparse.ArgumentParser() + parser.add_argument('action', choices=['prepare','run','summarize']) + parser.add_argument('--only', default='') + parser.add_argument('--repetitions', type=int, default=3) + parser.add_argument('--timeout', type=float, default=90) + args=parser.parse_args() + RESULT.mkdir(exist_ok=True) + if args.action=='prepare': + prepare(); return + if args.action=='summarize': + summarize(); return + cases=json.loads((ROOT/'cases.json').read_text()) + if args.only: + names=set(args.only.split(',')); cases=[c for c in cases if c['name'] in names] + eligible=[] + for case in cases: + if run(case,'warmup',0,args.timeout)['exit_code']==0: + eligible.append(case) + rng=random.Random(7312026) + for repeat in range(1,args.repetitions+1): + order=eligible[:]; rng.shuffle(order) + for case in order: + run(case,'measure',repeat,args.timeout) + summarize() + +if __name__=='__main__': + main() diff --git a/doc/local-validation-archive/files/profiling/scripts/brief_martin.py b/doc/local-validation-archive/files/profiling/scripts/brief_martin.py new file mode 100644 index 00000000..f93e2887 --- /dev/null +++ b/doc/local-validation-archive/files/profiling/scripts/brief_martin.py @@ -0,0 +1,131 @@ +#!/usr/bin/env python3 +"""Create Martin's concise briefing and an unsent email with verified attachments.""" +import hashlib,json,shutil +from email.message import EmailMessage +from email.policy import SMTP +from pathlib import Path +from xml.sax.saxutils import escape +from reportlab.lib import colors +from reportlab.lib.enums import TA_LEFT +from reportlab.lib.pagesizes import A4 +from reportlab.lib.styles import ParagraphStyle +from reportlab.pdfbase import pdfmetrics +from reportlab.pdfbase.ttfonts import TTFont +from reportlab.platypus import SimpleDocTemplate,Paragraph,Spacer,Table,TableStyle,PageBreak,KeepTogether + +ROOT=Path(__file__).resolve().parents[1] +OUT=ROOT.parent/'output/pdf';OUT.mkdir(parents=True,exist_ok=True) +DELIVERY=ROOT/'martin';DELIVERY.mkdir(exist_ok=True) +DATA={r['case']:r for r in json.loads((ROOT/'results/summary.json').read_text())} +audit=json.loads((ROOT/'results/audit.json').read_text()) +assert audit['status']=='pass' and audit['raw_records']==224 +assert all(r['output_equal'] for r in json.loads((ROOT/'profiles/validated-outputs.json').read_text())['checks']) +probe=json.loads((ROOT/'results/stream-cleanup-probe.json').read_text());assert probe['fd_growth']==100 +def seconds(n):return DATA[n]['median_s'] +def rss(n):return DATA[n]['median_rss_mib'] +blue=colors.HexColor('#174D68'); ink=colors.HexColor('#1C2933');muted=colors.HexColor('#52636D') +light=colors.HexColor('#F0F5F7');line=colors.HexColor('#D5DFE4');amber=colors.HexColor('#8B451B') +fonts=Path('/usr/share/fonts/truetype/dejavu') +for name,file in [('Brief','DejaVuSans.ttf'),('Brief-Bold','DejaVuSans-Bold.ttf'),('Brief-Italic','DejaVuSans-Oblique.ttf')]: + pdfmetrics.registerFont(TTFont(name,str(fonts/file))) +pdfmetrics.registerFontFamily('Brief',normal='Brief',bold='Brief-Bold',italic='Brief-Italic',boldItalic='Brief-Bold') +styles={ + 'title':ParagraphStyle('title',fontName='Brief-Bold',fontSize=24,leading=28,textColor=blue,spaceAfter=7), + 'kicker':ParagraphStyle('kicker',fontName='Brief-Bold',fontSize=8,leading=11,textColor=muted,spaceAfter=8), + 'deck':ParagraphStyle('deck',fontName='Brief',fontSize=11,leading=15,textColor=ink,spaceAfter=14), + 'h2':ParagraphStyle('h2',fontName='Brief-Bold',fontSize=12,leading=16,textColor=blue,spaceBefore=9,spaceAfter=6), + 'body':ParagraphStyle('body',fontName='Brief',fontSize=9.2,leading=13,textColor=ink,spaceAfter=5), + 'small':ParagraphStyle('small',fontName='Brief',fontSize=8,leading=11,textColor=muted,spaceAfter=5), + 'cell':ParagraphStyle('cell',fontName='Brief',fontSize=8.5,leading=11.7,textColor=ink), + 'head':ParagraphStyle('head',fontName='Brief-Bold',fontSize=8.5,leading=11.7,textColor=colors.white), +} +def P(text,style='body'):return Paragraph(text,styles[style]) +W=A4[0]-88 +def grid(rows,widths): + table=Table([[P(str(v),'head' if i==0 else 'cell') for v in row] for i,row in enumerate(rows)],colWidths=widths,repeatRows=1,hAlign='LEFT') + table.setStyle(TableStyle([ + ('BACKGROUND',(0,0),(-1,0),blue),('VALIGN',(0,0),(-1,-1),'TOP'), + ('LEFTPADDING',(0,0),(-1,-1),8),('RIGHTPADDING',(0,0),(-1,-1),8), + ('TOPPADDING',(0,0),(-1,-1),6),('BOTTOMPADDING',(0,0),(-1,-1),6), + ('ROWBACKGROUNDS',(0,1),(-1,-1),[colors.white,light]), + ('LINEBELOW',(0,-1),(-1,-1),.5,line)])) + return table +PDF=OUT/'IntaRNA-profiling-brief-for-Martin.pdf' +doc=SimpleDocTemplate(str(PDF),pagesize=A4,rightMargin=44,leftMargin=44,topMargin=38,bottomMargin=44, + title='IntaRNA profiling: findings and proposed next steps',author='IntaRNA profiling',subject='Technical briefing for Martin, revision 1b83779') +story=[] +story += [P('ENGINEERING BRIEF / 19 SEPTEMBER 2026','kicker'),P('IntaRNA profiling','title'), + P('Findings and proposed next steps for Martin','deck'), + P('Tested revision: refactor-to-c++23 / 1b8377989ba7. This is a profile of one recorded build; no comparison with an earlier release or another PR was performed.','small')] +story += [P('Recommendation','h2'),P('Fix the confirmed input-stream cleanup defect first. Then review the accessibility-file limit behavior and benchmark narrowly scoped CPU optimizations, separately for default, exact and ensemble modes.')] +story += [P('Confirmed defect: input streams are not cleaned up','h2'), + P('Reproduction: 100 open/read/cleanup cycles against the freshly built library increased open file descriptors from 4 to 104. All four heap traces also retained 9,384 bytes in input-stream allocation stacks.'), + P('Cause: src/IntaRNA/general.cpp:136 casts the input stream to filtering_ostream, although it was created as filtering_istream. The cast fails and cleanup is skipped. Resources accumulate in repeated library use; the OS reclaims them when a CLI process exits. Introduction history was not investigated.','body')] +story += [P('Performance findings worth acting on','h2')] +rows=[['Finding','Measured evidence','Interpretation / boundary'], + ['Default X/H CPU', 'Loop-energy evaluation: 57.6% of 368 CPU samples in the biological GcvB batch.', 'First optimization target: repeated internal-loop energy work. Sample share is not a promised speedup.'], + ['Batch parallelism', f'48 targets × ChiX: {seconds("throughput_1t"):.3f} → {seconds("throughput_12t"):.3f} s at 1 → 12 threads (5.08×).', 'Peak RSS: 18.7 → 65.3 MiB. Batch throughput; not single-pair acceleration.'], + ['Accessibility reuse', '0.805 → 0.492 s (39% lower latency); same base-pair list.', 'Matched target/query caps: 149/99. One-time generation: 0.847 s, excluded from cached-call time.'], + ['Windowing, S/H', '127.5 → 24.1 MiB (-81%); 21.38 → 59.71 s (2.79× time).', '3,000 × 800 nt; window 300, overlap 150. Default X/H gets no useful RSS reduction and takes 1.72× longer.']] +story.append(grid(rows,[92,206,W-298])) +story += [P('Proposed sequence','h2'), + P('1. Isolate the cleanup fix and add repeated plain/gzip stream tests. 2. Confirm intended ED-file limit semantics with Martin. 3. Agree on a production workload, then measure an energy/scoring optimization against this baseline. Proposed changes have not been implemented or assigned an expected speedup.')] +story.append(PageBreak()) +story += [P('EVIDENCE AND INTERPRETATION','kicker'),P('How to read the results','title')] +story += [P('Reproducible baseline','h2'), + P('56 configurations: 168 measured repetitions plus 56 excluded warmups (224 CLI runs). Three measured repetitions per case, deterministic randomized order, sequential execution. Medians are shown; full ranges and raw records are supplied. CPU sampling, heap tracing and diagnostic logging were separate from clean timings.'), + P('Host/build: Ryzen 5 7530U (6 cores / 12 hardware threads), Linux 6.8.0-139; GCC 16.1.0, C++23, -O3 with symbols/frame pointers; ViennaRNA 2.7.2, Boost 1.85.0. OPENBLAS_NUM_THREADS=1, OMP_DYNAMIC=FALSE, OMP_PLACES=cores and OMP_PROC_BIND=close. Exact commands, hashes, dependencies and input sequences accompany the report.')] +story += [P('Batch scaling: choosing a thread count','h2')] +rows=[['Threads','Median wall s','Observed range s','Peak RSS MiB','Speedup']] +for n in [1,2,4,6,12]: + r=DATA[f'throughput_{n}t'];rows.append([n,f"{r['median_s']:.3f}",f"{r['min_s']:.3f}–{r['max_s']:.3f}",f"{r['median_rss_mib']:.1f}",f"{seconds('throughput_1t')/r['median_s']:.2f}×"]) +story.append(grid(rows,[55,94,134,124,W-407])) +story += [P('48 evaluated 300-nt targets × ChiX (84 nt); 46 reportable interactions. Six threads supply 88% of twelve-thread throughput at 61% of its RSS. Peak RSS values are medians of per-process high-water marks.','small')] +story += [P('Two interpretation traps','h2'), + P('Cached ED is not automatically a transparent replacement. The writer emits columns through its maximum; the reader advertises one nucleotide less for dangling-end treatment (AccessibilityFromStream.cpp:90). We therefore compared computed and loaded ED at the same 149/99 limits. Whether this is intended needs review before generalizing the saving.'), + P('Mode changes are not implementation speedups. Exact X/M is dominated by repeated scoring terms (Boltzmann weights 19.4%, ED lookup 18.5% of samples); ensemble mode adds partition-function recurrence and map costs. Disabling accessibility changes the scientific model and was slower in one test. Windowing and algorithm substitutions need their own output checks.')] +story += [P('Validation and uncertainty','h2'), + P('All timed cases and the automated evidence audit passed; 4,281 assertions in 36 API tests and all 20 CLI regression cases passed. Base-pair lists matched for 1 vs 12 threads and the matched ED comparison. Window comparisons matched coordinates and energy; full structure equality was not separately checked there.'), + P('This is one laptop, three measured repetitions per case and a bounded workload suite. No release-wide regression or biological-accuracy claim follows. CPU profiles total 1,182 samples at 100 ms. A collector timer warning and incomplete optimized stack ancestry limit precision, although CPU totals were consistent. Hardware counters were blocked; no cache-miss or IPC claims are made.')] +story += [P('Evidence and reporting conventions','h2'), + P('Attached evidence archive: start with profiling/REPORT.md. Inspect results/summary.csv, results/audit.json, results/stream-cleanup-probe.json, cases.json, and the reproduction scripts. Original data and hashes are retained.','small'), + P('Reporting follows full configuration disclosure and explicit warmup/repetition statistics, as illustrated by SPEC reporting rules and the Google Benchmark guide. This custom IntaRNA study does not claim SPEC certification.','small')] +def footer(c,d): + c.saveState();c.setFont('Brief',7.5);c.setFillColor(muted) + c.drawString(44,25,'IntaRNA | profiling date 2026-09-19 | revision 1b83779') + c.drawRightString(A4[0]-44,25,f'{d.page}');c.restoreState() +doc.build(story,onFirstPage=footer,onLaterPages=footer) + +subject='IntaRNA profiling (1b83779): confirmed cleanup defect and optimization priorities' +body='''Hi Martin, + +We completed a profiling pass on refactor-to-c++23 at commit 1b83779: 56 configurations, 168 measured repetitions plus 56 warmups, and separate CPU and heap profiles. The attached two-page brief gives the findings and proposed next steps; the evidence archive contains commands, inputs, raw results, audits and reproduction scripts. + +The main findings are: + +- Confirmed input-stream cleanup defect. Against the freshly built library, 100 open/read/cleanup calls increased open file descriptors from 4 to 104. general.cpp:136 checks for filtering_ostream although newInputStream creates filtering_istream. This accumulates resources in repeated library use; the OS reclaims them when a CLI process exits. I suggest treating this as the first, isolated fix. +- Default-mode optimization target. Internal-loop energy evaluation accounted for about 58% of CPU samples in the biological GcvB batch. Exact and ensemble modes have different scoring/partition-function hotspots, so optimization work should be measured separately by mode. +- Measured operating tradeoffs. The 48-target ChiX batch fell from 4.944 s on one thread to 0.974 s on twelve (5.08x), with RSS increasing from 18.7 to 65.3 MiB. In S mode, windowing reduced RSS by 81% but took 2.79x longer; default X mode showed no useful memory saving from windowing on that input. +- ED reuse requires a semantic check. With explicitly matched target/query limits of 149/99, loading ED reduced latency from 0.805 to 0.492 s and preserved the base-pair list. The reader currently lowers the available maximum length by one, so the saving should not be generalized to an unqualified round trip. + +All regression tests and the evidence audit passed. These are measurements of one build on one laptop, with three measured repetitions per case; they do not establish a cross-version improvement or a universal speedup. Hardware counters were unavailable, and the sampling limitations are documented. + +I propose an isolated cleanup fix first, then a review of the ED-file limit behavior, followed by a focused CPU optimization against an agreed production workload. Could you confirm whether the ED limit reduction is intended and which workload should be our acceptance benchmark? + +Best, +Alex +''' +(DELIVERY/'email-to-Martin.md').write_text('Subject: '+subject+'\n\n'+body) +message=EmailMessage(policy=SMTP) +message['Subject']=subject;message['X-Unsent']='1' +message.set_content(body) +message.add_attachment(PDF.read_bytes(),maintype='application',subtype='pdf',filename=PDF.name) +archive=ROOT/'intarna-profile-evidence.zip' +message.add_attachment(archive.read_bytes(),maintype='application',subtype='zip',filename=archive.name) +(DELIVERY/'email-to-Martin.eml').write_bytes(message.as_bytes()) +(DELIVERY/'delivery-status.json').write_text(json.dumps(dict(status='prepared_not_sent', + recipient=None,channel=None,subject=subject,pdf=str(PDF),draft=str(DELIVERY/'email-to-Martin.eml'), + attachments=[dict(name=p.name,bytes=p.stat().st_size,sha256=hashlib.sha256(p.read_bytes()).hexdigest()) for p in [PDF,archive]], + reason='Recipient and delivery channel require confirmation; no email integration is connected.'),indent=2)+'\n') +print(PDF) +print(DELIVERY/'email-to-Martin.eml') diff --git a/doc/local-validation-archive/files/profiling/scripts/build.sh b/doc/local-validation-archive/files/profiling/scripts/build.sh new file mode 100644 index 00000000..82a51cc5 --- /dev/null +++ b/doc/local-validation-archive/files/profiling/scripts/build.sh @@ -0,0 +1,23 @@ +#!/usr/bin/env bash +set -euo pipefail +profile_root=$(cd "$(dirname "$0")/.." && pwd) +repo_root=${INTARNA_REPO:-"$profile_root/../intarna_optimization_supervised"} +dependency_root=${INTARNA_DEPS:-/home/alex/CursorProjects/IntaRNA_optimization/intaRNA_legacy/.conda-env} +revision=${INTARNA_REVISION:-1b8377989ba79a0f1b8a450163b71bd8ed7c3e45} +if [[ -e "$profile_root/source" ]]; then + echo 'source already exists; use a new profiling directory to preserve evidence' >&2 + exit 1 +fi +mkdir -p "$profile_root/source" "$profile_root/logs" +git -C "$repo_root" archive "$revision" | tar -x -C "$profile_root/source" +cd "$profile_root/source" +autoreconf -fi > ../logs/autoreconf.log 2>&1 +./configure --with-boost="$dependency_root" --with-boost-libdir="$dependency_root/lib" \ + --with-vrna="$dependency_root" --prefix="$profile_root/install" \ + CXX="$dependency_root/bin/x86_64-conda-linux-gnu-g++" \ + CC="$dependency_root/bin/x86_64-conda-linux-gnu-gcc" \ + CXXFLAGS='-g -fno-omit-frame-pointer' CPPFLAGS="-I$dependency_root/include" \ + LDFLAGS="-L$dependency_root/lib -Wl,-rpath,$dependency_root/lib" \ + PKG_CONFIG_PATH="$dependency_root/lib/pkgconfig" > ../logs/configure.log 2>&1 +make -j4 V=1 > ../logs/build.log 2>&1 +make -j4 check > ../logs/check.log 2>&1 diff --git a/doc/local-validation-archive/files/profiling/scripts/cache_match.py b/doc/local-validation-archive/files/profiling/scripts/cache_match.py new file mode 100644 index 00000000..6ea94622 --- /dev/null +++ b/doc/local-validation-archive/files/profiling/scripts/cache_match.py @@ -0,0 +1,21 @@ +#!/usr/bin/env python3 +"""Match effective caps after discovering the ED reload's one-column reduction.""" +import json,random +from benchmark import ROOT,run,summarize + +def main(): + cases=json.loads((ROOT/'cases.json').read_text()) + by_name={c['name']:c for c in cases} + new=[] + for base,name in [('cache_computed_control','cache_matched_computed'),('cache_reused','cache_matched_reused')]: + new.append(dict(by_name[base],name=name,args=[a for a in by_name[base]['args'] if not a.startswith('--intLenMax=')] + +['--qIntLenMax=99','--tIntLenMax=149'])) + (ROOT/'cases.json').write_text(json.dumps(cases+new,indent=2)+'\n') + eligible=[c for c in new if run(c,'warmup',0,90)['exit_code']==0] + rng=random.Random(7312028) + for rep in range(1,4): + order=eligible[:];rng.shuffle(order) + for c in order:run(c,'measure',rep,90) + summarize() + +if __name__=='__main__':main() diff --git a/doc/local-validation-archive/files/profiling/scripts/package.py b/doc/local-validation-archive/files/profiling/scripts/package.py new file mode 100644 index 00000000..c68e6e64 --- /dev/null +++ b/doc/local-validation-archive/files/profiling/scripts/package.py @@ -0,0 +1,24 @@ +#!/usr/bin/env python3 +"""Bundle report and portable evidence, leaving large build/profile binaries local.""" +import hashlib,json,shutil,zipfile +from pathlib import Path +ROOT=Path(__file__).resolve().parents[1] +for name in ['runApiTests.log','runIntaRNA.log','test-suite.log']: + shutil.copyfile(ROOT/'source/tests'/name,ROOT/'logs'/('tests-'+name)) +files=[] +for name in ['README.md','REPORT.md','environment.json','cases.json']: + files.append(ROOT/name) +for dirname in ['scripts','inputs','results','council','logs']: + for p in (ROOT/dirname).rglob('*'): + if p.is_file() and '__pycache__' not in p.parts and p.name!='stream_cleanup_probe':files.append(p) +files += [p for p in (ROOT/'profiles').iterdir() if p.is_file() and not p.name.startswith('probe')] +files += [p for p in (ROOT/'source/src').rglob('*') if p.is_file() and p.suffix in {'.cpp','.cc','.h','.icc'}] +manifest={str(p.relative_to(ROOT)):dict(bytes=p.stat().st_size,sha256=hashlib.sha256(p.read_bytes()).hexdigest()) for p in sorted(set(files))} +dest=ROOT/'intarna-profile-evidence.zip' +with zipfile.ZipFile(dest,'w',compression=zipfile.ZIP_DEFLATED) as z: + for p in sorted(set(files)):z.write(p,'profiling/'+str(p.relative_to(ROOT))) + z.writestr('profiling/bundle-manifest.json',json.dumps(manifest,indent=2)+'\n') +(ROOT/'bundle-info.json').write_text(json.dumps(dict(file=dest.name,bytes=dest.stat().st_size, + sha256=hashlib.sha256(dest.read_bytes()).hexdigest(),files=len(manifest), + note='Large source build artifacts, compiled probe, gprofng experiment directories and old-binary capability probes remain local and are excluded.'),indent=2)+'\n') +print(dest, dest.stat().st_size, 'bytes') diff --git a/doc/local-validation-archive/files/profiling/scripts/phase_summary.py b/doc/local-validation-archive/files/profiling/scripts/phase_summary.py new file mode 100644 index 00000000..c568dd75 --- /dev/null +++ b/doc/local-validation-archive/files/profiling/scripts/phase_summary.py @@ -0,0 +1,26 @@ +#!/usr/bin/env python3 +"""Parse diagnostic phase logs, retaining integer-duration truncation bounds.""" +import json,re +from pathlib import Path +ROOT=Path(__file__).resolve().parents[1] +rows=[] +for path in sorted((ROOT/'profiles').glob('*_phases.stdout')): + data=path.read_text() + groups={k:dict(calls=0,lower_s=0,upper_s=0) for k in ['accessibility','prediction_including_seed','seed_nested']} + detailed=[] + for fun,n,unit in re.findall(r'Executed \[(.*?)\] in \[(\d+) ([^\]]+)\]',data): + scale={'ms':.001,'seconds':1,'minutes':60,'hours':3600}.get(unit) + if scale is None:raise ValueError(unit) + lo=int(n)*scale;hi=lo+scale + if 'AccessibilityVrna::' in fun:group='accessibility' + elif '::predict(' in fun:group='prediction_including_seed' + elif '::fillSeed(' in fun:group='seed_nested' + else:group=None + if group: + groups[group]['calls']+=1;groups[group]['lower_s']+=lo;groups[group]['upper_s']+=hi + detailed.append(dict(function=fun,displayed=f'{n} {unit}',lower_s=lo,upper_s=hi,group=group)) + rows.append(dict(profile=path.stem.removesuffix('_phases'),groups=groups,phases=detailed, + valid_seed_count_sum=sum(map(int,re.findall(r'valid seeds = (\d+)',data))))) +(ROOT/'profiles/phase-summary.json').write_text(json.dumps(rows,indent=2)+'\n') +for row in rows: + print(row['profile'],row['groups']) diff --git a/doc/local-validation-archive/files/profiling/scripts/plots.py b/doc/local-validation-archive/files/profiling/scripts/plots.py new file mode 100644 index 00000000..de3c7cf9 --- /dev/null +++ b/doc/local-validation-archive/files/profiling/scripts/plots.py @@ -0,0 +1,61 @@ +#!/usr/bin/env python3 +import json,os +from pathlib import Path +os.environ.setdefault('MPLCONFIGDIR','/tmp/intarna-matplotlib') +import matplotlib +matplotlib.use('Agg') +import matplotlib.pyplot as plt +from matplotlib.ticker import ScalarFormatter +ROOT=Path(__file__).resolve().parents[1] +D={r['case']:r for r in json.loads((ROOT/'results/summary.json').read_text())} +plt.rcParams.update({'font.family':'DejaVu Sans','font.size':10,'axes.spines.top':False,'axes.spines.right':False, + 'axes.titleweight':'bold','axes.titlesize':12,'figure.facecolor':'#fafaf8','axes.facecolor':'#fafaf8'}) +blue='#226e8b'; orange='#b86132'; green='#448065' +fig,axes=plt.subplots(2,3,figsize=(17,10),layout='constrained') + +def line(ax,xx,names,color,label): + yy=[D[n]['median_s'] for n in names] + lo=[D[n]['median_s']-D[n]['min_s'] for n in names] + hi=[D[n]['max_s']-D[n]['median_s'] for n in names] + ax.errorbar(xx,yy,yerr=[lo,hi],fmt='o-',color=color,capsize=3,label=label) + +ax=axes[0,0]; ns=[100,300,1000,3000,10000] +line(ax,ns,[f'length_t{n}_q100' for n in ns],blue,'Query = 100 nt') +ax.set(xscale='log',yscale='log',xlabel='Target length (nt)',ylabel='Wall time (seconds)',title='Target length: fixed query and folding window') +ax.set_xticks(ns,labels=[str(n) for n in ns]);ax.grid(alpha=.15) + +ax=axes[0,1]; ts=[1,2,4,6,12]; base=D['throughput_1t']['median_s'] +speed=[base/D[f'throughput_{n}t']['median_s'] for n in ts] +ax.plot(ts,speed,'o-',color=blue,label='Measured speedup') +ax.plot(ts,ts,'--',color='#aaaaaa',label='Ideal');ax.set(xlabel='IntaRNA threads',ylabel='Speedup vs one thread',title='48 biological targets × ChiX');ax.set_xticks(ts);ax.legend(frameon=False) + +ax=axes[0,2]; names=['algorithm_default','algorithm_exact_x','algorithm_heuristic_s','algorithm_exact_s','algorithm_helix','algorithm_no_seed'] +labels=['X heuristic','X exact','S heuristic','S exact','B heuristic','S / no seed'] +values=[D[n]['median_s'] for n in names]; bars=ax.barh(labels[::-1],values[::-1],color=blue) +ax.set(xscale='log',xlabel='Wall time (seconds; log scale)',title='Algorithm choices: same 100 × 100 nt input') +for bar,v in zip(bars,values[::-1]):ax.text(v*1.06,bar.get_y()+bar.get_height()/2,f'{v:.3f}',va='center',fontsize=9) +ax.set_xlim(right=max(values)*2.5) + +ax=axes[1,0]; names=['no_pair','composition_gc20','composition_gc50','composition_gc80','au_repeat','gc_repeat'] +labels=['All A (no pairing)','20% GC distribution','50% GC distribution','80% GC distribution','AU repeat','GC repeat'] +v=[D[n]['median_s'] for n in names]; bars=ax.barh(labels[::-1],v[::-1],color=green) +ax.set(xlabel='Wall time (seconds)',title='Sequence composition: 300 × 100 nt') +for bar,x in zip(bars,v[::-1]):ax.text(x+.035,bar.get_y()+bar.get_height()/2,f'{x:.3f}',va='center',fontsize=9) +ax.set_xlim(right=max(v)*1.2) + +ax=axes[1,1]; names=['accessibility_computed','accessibility_global','accessibility_none','cache_matched_reused'];labels=['Local (default)','Global','Disabled','Reused ED*'] +v=[D[n]['median_s'] for n in names];bars=ax.bar(labels,v,color=[blue,orange,'#829ca6',green]) +ax.set(ylabel='Wall time (seconds)',title='Accessibility: 1000 × 100 nt; interaction cap 150') +for bar,x in zip(bars,v):ax.text(bar.get_x()+bar.get_width()/2,x+.06,f'{x:.3f}',ha='center') +ax.text(.02,.97,'Global/disabled alter the model. *ED caps = 149/99.',transform=ax.transAxes,va='top',fontsize=9,color='#555555') +ax.set_ylim(top=max(v)*1.25) + +ax=axes[1,2]; names=['memory_t3000_q800','memory_window300','memory_s','memory_s_window300'];labels=['X whole','X window 300','S whole','S window 300'] +v=[D[n]['median_rss_mib'] for n in names if n in D] +if len(v)==4: + bars=ax.bar(labels,v,color=[blue,'#88b5c6',orange,'#d5a38a']) + for bar,x in zip(bars,v):ax.text(bar.get_x()+bar.get_width()/2,x+max(v)*.015,f'{x:.1f}',ha='center') +ax.set(ylabel='Peak resident memory (MiB)',title='Windowing memory effect: 3000 × 800 nt');ax.tick_params(axis='x',labelrotation=12) +fig.suptitle('IntaRNA profiling • commit 1b83779 • Ryzen 5 7530U\nMedians of 3 runs; clean timings, separate from instrumentation',fontsize=18,fontweight='bold') +out=ROOT/'results';fig.savefig(out/'overview.png',dpi=150);fig.savefig(out/'overview.svg');plt.close(fig) +print(out/'overview.png') diff --git a/doc/local-validation-archive/files/profiling/scripts/profile.py b/doc/local-validation-archive/files/profiling/scripts/profile.py new file mode 100644 index 00000000..fc03c027 --- /dev/null +++ b/doc/local-validation-archive/files/profiling/scripts/profile.py @@ -0,0 +1,66 @@ +#!/usr/bin/env python3 +"""Separate intrusive profiles from benchmark runs. Execute only with host idle.""" +import hashlib, json, os, subprocess, time +from pathlib import Path +from benchmark import ROOT, BIN, ENV, INPUT, canonical, write_fasta, fasta + +DEST=ROOT/'profiles' + +def profile_output(data): + return canonical(b'\n'.join(line for line in data.splitlines() + if not line.startswith(b'Creating experiment directory '))) + +def execute(name, cmd, timeout=180): + print(name, flush=True) + start=time.perf_counter() + wrapped=['/usr/bin/time','-f','%e\t%U\t%S\t%M','-o',str(DEST/(name+'.time'))]+cmd + with (DEST/(name+'.stdout')).open('wb') as out, (DEST/(name+'.stderr')).open('wb') as err: + p=subprocess.run(wrapped, stdout=out, stderr=err, env=ENV, timeout=timeout) + row=dict(name=name,command=cmd,exit_code=p.returncode,wall_s=time.perf_counter()-start) + vals=(DEST/(name+'.time')).read_text().strip().splitlines() + if vals and len(vals[-1].split('\t'))==4: + row.update(dict(zip(['gnu_wall_s','user_s','system_s','max_rss_kib'],map(float,vals[-1].split('\t'))))) + with (DEST/'commands.jsonl').open('a') as f:f.write(json.dumps(row)+'\n') + return row + +def main(): + DEST.mkdir(exist_ok=True) + cases={c['name']:c for c in json.loads((ROOT/'cases.json').read_text())} + exact_targets=write_fasta('profile_exact_targets',[(f'exact_repeat_{i}',fasta(INPUT/'t100.fa')[0][1]) for i in range(128)]) + ensemble_targets=write_fasta('profile_ensemble_targets',[(f'ensemble_repeat_{i}',fasta(INPUT/'t60.fa')[0][1]) for i in range(32)]) + (DEST/'input-manifest.json').write_text(json.dumps({str(p):dict(sha256=hashlib.sha256(Path(p).read_bytes()).hexdigest(), + records=len(fasta(p))) for p in [exact_targets,ensemble_targets]},indent=2)+'\n') + specs=[ + ('biological_batch', ['--target='+str(INPUT/'promoters48.fa'),'--query='+str(INPUT/'GcvB.fa'), + '--threads=1','--outMode=C','--outCsvCols=id1,id2,start1,end1,start2,end2,E']), + ('hybrid_noaccess', [('--query='+str(INPUT/'q400.fa')) if a.startswith('--query=') else a + for a in cases['memory_t3000_q800']['args']]+['--acc=N','--intLenMax=150']), + ('exact_x', [('--target='+exact_targets) if a.startswith('--target=') else a for a in cases['algorithm_exact_x']['args']]), + ('ensemble_m', ['--target='+ensemble_targets,'--query='+str(INPUT/'q60.fa'), + '--threads=1','--model=P','--mode=M','--noSeed','--outMode=C', + '--outCsvCols=id1,id2,start1,end1,start2,end2,E']), + ] + for name,args in specs: + clean=execute(name+'_reference',[str(BIN)]+args) + exp=str(DEST/(name+'.er')) + prof=execute(name+'_cpu',['gprofng','collect','app','-p','100','-a','off','-o',exp,str(BIN)]+args) + for kind,sort in [('self','e.totalcpu'),('inclusive','i.totalcpu')]: + execute(name+'_'+kind,['gprofng','display','text','-metrics','e.%totalcpu:i.%totalcpu:name', + '-sort',sort,'-limit','40','-functions',exp]) + execute(name+'_statistics',['gprofng','display','text','-header','-statistics',exp]) + execute(name+'_phases',[str(BIN)]+args+['--verbose']) + ref=(DEST/(name+'_reference.stdout')).read_bytes() + sampled=(DEST/(name+'_cpu.stdout')).read_bytes() + with (DEST/'validation.jsonl').open('a') as f: + f.write(json.dumps(dict(name=name,output_equal=profile_output(ref)==profile_output(sampled), + clean_s=clean['wall_s'],sampled_s=prof['wall_s'], + note='collector startup/postprocessing included in sampled wall time'))+'\n') + for name in ['bio_phob_gcvb','algorithm_exact_x','memory_t3000_q800','memory_s']: + args=cases[name]['args']; exp=str(DEST/(name+'_heap.er')) + execute(name+'_heap',['gprofng','collect','app','-p','off','-H','on','-a','off','-o',exp,str(BIN)]+args) + execute(name+'_heap_summary',['gprofng','display','text','-limit','30','-heapstat','-allocs','-leaks',exp]) + for name in ['bio_phob_gcvb','accessibility_computed','accessibility_none','accessibility_global', + 'length_t3000_q100','window_300','throughput_1t','throughput_6t']: + execute(name+'_phases',[str(BIN)]+cases[name]['args']+['--verbose']) + +if __name__=='__main__':main() diff --git a/doc/local-validation-archive/files/profiling/scripts/profile_audit.py b/doc/local-validation-archive/files/profiling/scripts/profile_audit.py new file mode 100644 index 00000000..f1f27448 --- /dev/null +++ b/doc/local-validation-archive/files/profiling/scripts/profile_audit.py @@ -0,0 +1,21 @@ +#!/usr/bin/env python3 +import csv,hashlib,json +from pathlib import Path +from benchmark import ROOT,canonical +from profile import profile_output +dest=ROOT/'profiles' +rows=[] +for name in ['biological_batch','hybrid_noaccess','exact_x','ensemble_m']: + a=profile_output((dest/(name+'_reference.stdout')).read_bytes()) + b=profile_output((dest/(name+'_cpu.stdout')).read_bytes()) + table=list(csv.reader(b.decode().splitlines(),delimiter=';')) + valid=bool(table) and table[0]==['id1','id2','start1','end1','start2','end2','E'] and all(len(r)==7 for r in table) + rows.append(dict(name=name,kind='cpu',schema_valid=valid,output_equal=a==b, + rows=len(table)-1,sha256=hashlib.sha256(b).hexdigest())) +for name in ['bio_phob_gcvb','algorithm_exact_x','memory_t3000_q800','memory_s']: + a=canonical((ROOT/'results'/(name+'.measure1.stdout')).read_bytes()) + b=profile_output((dest/(name+'_heap.stdout')).read_bytes()) + rows.append(dict(name=name,kind='heap',output_equal=a==b,sha256=hashlib.sha256(b).hexdigest())) +(dest/'validated-outputs.json').write_text(json.dumps(dict(note='Only recognized gprofng creation banner and IntaRNA # log lines excluded; original files/validation retained.',checks=rows),indent=2)+'\n') +print(json.dumps(rows,indent=2)) +if not all(r['output_equal'] and r.get('schema_valid',True) for r in rows):raise SystemExit(1) diff --git a/doc/local-validation-archive/files/profiling/scripts/report.py b/doc/local-validation-archive/files/profiling/scripts/report.py new file mode 100644 index 00000000..ed17957a --- /dev/null +++ b/doc/local-validation-archive/files/profiling/scripts/report.py @@ -0,0 +1,159 @@ +#!/usr/bin/env python3 +"""Render the report from audited measurements and reviewed profile interpretation.""" +import json,re,statistics +from pathlib import Path +ROOT=Path(__file__).resolve().parents[1] +S=json.loads((ROOT/'results/summary.json').read_text());D={r['case']:r for r in S} +ENV=json.loads((ROOT/'environment.json').read_text()) +RUNS=[json.loads(x) for x in (ROOT/'results/validated-runs.jsonl').read_text().splitlines()] +def t(name):return D[name]['median_s'] +def rss(name):return D[name]['median_rss_mib'] +def table(names): + lines=['| Case | Median wall s | Min–max s | Median user / system s | Median peak RSS MiB | Output rows |', + '|---|---:|---:|---:|---:|---:|'] + for name in names: + r=D[name] + lines.append(f"| `{name}` | {r['median_s']:.3f} | {r['min_s']:.3f}–{r['max_s']:.3f} | {r['median_user_s']:.2f} / {r['median_system_s']:.2f} | {r['median_rss_mib']:.2f} | {r['rows']} |") + return '\n'.join(lines) +thread=['| Threads | Median s | Speedup | Evaluated pairs/s | Peak RSS MiB |','|---:|---:|---:|---:|---:|'] +for n in [1,2,4,6,12]: + name=f'throughput_{n}t';thread.append(f'| {n} | {t(name):.3f} | {t("throughput_1t")/t(name):.2f}× | {48/t(name):.2f} | {rss(name):.2f} |') +cpu=['| Profile | Nominal 100 ms samples | Largest exclusive CPU samples |','|---|---:|---|'] +for name in ['biological_batch','hybrid_noaccess','exact_x','ensemble_m']: + text=(ROOT/'profiles'/f'{name}_self.stdout').read_text() + parsed=[] + for line in text.splitlines(): + m=re.match(r'\s*([\d.]+)\s+([\d.]+)\s+([\d.]+)\s+([\d.]+)\s+(.+)',line) + if m:parsed.append(m.groups()) + total=float(parsed[0][0]);top=[] + for own,pct,incl,ipct,fun in parsed[1:4]: + fun=fun.split('(')[0].replace('IntaRNA::','') + top.append(f'`{fun}` {float(pct):.1f}%') + cpu.append(f'| {name} | {round(total/.1)} | '+ '; '.join(top)+' |') +review=(ROOT/'council/profile-results-review.md').read_text() +review=review.split('## Optimization priorities supported by the profiles')[0].strip() +review=review.replace('## ', '### ') +doc=f'''# IntaRNA profiling report + +Profiled 2026-09-19. **56 workloads, 224 clean runs, four CPU profiles, four heap traces, and separate phase and structural checks.** The fresh build passed its regression tests; all timed commands and the independent evidence audit passed. The original checkout was left unchanged. + +The strongest findings are a **{t('throughput_1t')/t('throughput_12t'):.2f}× batch speedup at 12 threads**, a default-mode CPU hotspot in internal-loop energy evaluation, and a substantial time/memory tradeoff from windowing. **An input-stream cleanup defect is confirmed: 100 open/cleanup calls leave 100 file descriptors open.** Reusing accessibility data saved **{100*(1-t('cache_matched_reused')/t('cache_matched_computed')):.1f}% latency** in a matched test, but the current ED reader lowers the available interaction limit by one nucleotide. Exact and ensemble modes have different hotspot distributions and need separate optimization work. + +![Measured performance overview](results/overview.png) + +## Scope and experimental controls + +The council comprised workload, hotspot/tooling, and experimental-methodology reviewers. They exchanged proposals and challenged stale build provenance, thread scaling, accessibility-limit confounding, sampling validity, output equivalence and memory interpretation. Their recommendations are retained in [workloads](council/workloads.md), [hotspots](council/hotspots.md), and [methodology](council/methodology.md). An independent [final audit](council/final-review.md) checked the actual evidence. + +Target: `refactor-to-c++23`, commit `1b8377989ba79a0f1b8a450163b71bd8ed7c3e45`, reporting IntaRNA 3.4.1. A clean Git archive was built in `profiling/source`; the pre-existing executable was used only for tooling probes. Build: GCC 16.1.0, C++23, release `-O3`, `-g -fno-omit-frame-pointer`, OpenMP, ViennaRNA 2.7.2, Boost 1.85.0. This is an optimized build with profiling-friendly stack frames, not a debug `-O0` build. Binary SHA-256: `{ENV['binary_sha256']}`. + +Host: AMD Ryzen 5 7530U, six physical cores/twelve logical CPUs, approximately 30.7 GiB RAM, Linux 6.8.0-139, performance governor. All 12 logical CPUs were available; recorded cgroup ancestors imposed no CPU quota or memory maximum. Dynamic frequency and other desktop activity were not disabled. See [environment.json](environment.json) for exact provenance and linked dependencies. + +Each case had one excluded warmup and three measured runs. Runs were sequential, with deterministic randomized order in each round; compilation, tests, CPU sampling, heap tracing and verbose logs were separate. The main matrix and later controls have separate randomization blocks. Wall time covers process startup, input, computation and output. GNU time records user/system CPU, process peak RSS, faults, context switches and filesystem I/O. Medians and min–max ranges are reported; three repetitions do not support strong tail-latency or population confidence claims. + +The baseline fixes `OPENBLAS_NUM_THREADS=1`, `OMP_DYNAMIC=FALSE`, `OMP_PLACES=cores`, `OMP_PROC_BIND=close`, and `LC_ALL=C`. The BLAS experiment explicitly varies that setting. The linked dependency stack otherwise starts extra BLAS workers independently of IntaRNA's thread option. + +Inputs include repository biological examples, the first 48 real 300-nt target records in `NC_000913.fa`, and deterministic synthetic sequences (seed `20260919`). That FASTA is a collection of targets, not one contiguous genome. Length sweeps use nested sequence prefixes. GC-distribution and repeat cases are individual controlled examples, not a representative population of all RNAs. Actual lengths, composition and hashes are in [input manifest](inputs/manifest.json); every command is in [cases.json](cases.json). + +## Runtime and scaling + +### Biological examples + +{table(['bio_fhla_oxys','bio_phob_gcvb','bio_ilve_gcvb'])} + +These inputs are respectively 112×108, 299×201 and 299×200 nt. The last two have almost identical lengths yet different timing, illustrating sequence dependence. + +With a fixed 100-nt query and default local accessibility settings, target lengths 100, 300, 1,000, 3,000 and 10,000 nt take {', '.join(f"{t('length_t'+str(n)+'_q100'):.3f}" for n in [100,300,1000,3000,10000])} seconds. The 1,000→10,000 range increases time {t('length_t10000_q100')/t('length_t1000_q100'):.2f}× for 10× target length. This describes the measured regime with bounded folding/interaction windows; it is not an asymptotic complexity proof. + +At 300×100 nt, the composition cases span {t('no_pair'):.3f} seconds for nonpairing all-A sequences to {t('gc_repeat'):.3f} seconds for GC repeats. GC repeats take {t('gc_repeat')/t('composition_gc50'):.1f}× the random 50%-GC-distribution case. Seed density and placement matter in addition to sequence length. Query-length tests intentionally include queries longer than the target; no orientation-invariance claim is made. + +### Algorithm choices + +On the same 100×100 synthetic input, default X/H takes {t('algorithm_default'):.3f} s, exact X/M {t('algorithm_exact_x'):.3f} s, heuristic S/H {t('algorithm_heuristic_s'):.3f} s, exact S/M {t('algorithm_exact_s'):.3f} s, and helix B/H {t('algorithm_helix'):.3f} s. Exact S is {t('algorithm_exact_s')/t('algorithm_default'):.0f}× the default on this case. These modes implement different prediction guarantees and models; their timing differences are not evidence of interchangeable answers. + +The suite also covers ensemble P/H and P/M on bounded 40×40 input, seed-only mode, gapped seeds, no lonely pairs, loop limits and suboptimal output. `--noSeed` changes X to S and is labelled accordingly in the council review. Requests for 10 and 100 outputs both return the same ten interactions here; their similar cost does not establish that arbitrary output enumeration is cheap. Seven fixed CSV columns avoid accidentally requesting extra ensemble computations through `--outCsvCols=*`. + +### Parallel throughput + +{chr(10).join(thread)} + +The denominator is **48 evaluated target–query pairs**, with ChiX length84; 46 have reportable interactions. All five thread counts produce the same reported coordinates and energies after sorting. A separate comparison including complete base-pair lists confirms equality at one and twelve threads. + +Six threads deliver {100*t('throughput_12t')/t('throughput_6t'):.1f}% of twelve-thread throughput using {100*rss('throughput_6t')/rss('throughput_12t'):.1f}% of its RSS. Twelve was fastest among the tested settings. This is throughput from independent targets; a single unwindowed pair does not have the same parallelism. Source scheduling prioritizes targets, then queries, then windows. + +## Accessibility and memory + +For the 1,000×100 input with an explicit 150-nt interaction cap, local accessibility takes {t('accessibility_computed'):.3f} s and {rss('accessibility_computed'):.1f} MiB. Global accessibility takes {t('accessibility_global'):.3f} s and {rss('accessibility_global'):.1f} MiB. Disabling accessibility takes {t('accessibility_none'):.3f} s, despite removing folding. Accessibility changes energies, admissible boundaries and prediction work; subtracting these runtimes would not isolate folding cost. Separate phase logs provide that attribution. + +Reducing accessibility window/span to75/75 also changes the scientific model. A separate75-nt interaction-cap control retains the default150/100 folding settings. Both are recorded so their differences cannot be mistaken for a pure storage optimization. + +### Reusing saved accessibility data + +The matched target/query limits149/99 give {t('cache_matched_computed'):.3f} s when computing accessibility and {t('cache_matched_reused'):.3f} s when reloading ED: **{t('cache_matched_computed')/t('cache_matched_reused'):.2f}× speedup**, with RSS {rss('cache_matched_computed'):.1f}→{rss('cache_matched_reused'):.1f} MiB. Coordinates, energies and base-pair lists match. The separately recorded one-time generation took0.847 s including a complete prediction and serialization; amortize that cost when planning reuse. + +**Cache semantic caveat:** the writer emits columns through its configured maximum, while `AccessibilityFromStream.cpp:90–100` advertises one less than the final header length, with a source comment about dangling ends. The saved files contain their last columns; they are not truncated. Original100/150-limit round-trip runs therefore become99/149. Their output happens to agree on this example, but only the explicitly matched controls support the cache timing claim. Do not assume this round trip preserves every requested interaction limit. See the [source and file review](council/accessibility_cache_review.md). No application fix was made. + +### Prediction windowing + +| Mode / 3,000×800 nt input | Whole input seconds | Window300 seconds | Whole peak MiB | Window300 peak MiB | +|---|---:|---:|---:|---:| +| X/H | {t('memory_t3000_q800'):.3f} | {t('memory_window300'):.3f} | {rss('memory_t3000_q800'):.2f} | {rss('memory_window300'):.2f} | +| S/H | {t('memory_s'):.3f} | {t('memory_s_window300'):.3f} | {rss('memory_s'):.2f} | {rss('memory_s_window300'):.2f} | + +Overlap is150 and the effective interaction cap is150. X/H's extension matrices are already bounded by interaction length (at most144×144 per side with the seven-pair seed here). Whole-sequence accessibility still contributes to peak RSS. Windows add95 target/query window combinations and repeat work, explaining the {t('memory_window300')/t('memory_t3000_q800'):.2f}× time without useful RSS reduction. + +S/H keeps two matrices covering the full target/query ranges, so windowing reduces RSS by **{100*(1-rss('memory_s_window300')/rss('memory_s')):.1f}%**, while taking {t('memory_s_window300')/t('memory_s'):.2f}× longer. All tested window comparisons preserve the seven reported CSV fields; full structural equivalence was not separately tested for these window cases. The [window review](council/workload_results_review.md) links these observations to source. + +## CPU hotspots and allocations + +CPU workloads are a48-target biological batch with GcvB; a3,000×400 default-X pair with accessibility disabled and cap150; 128 repeated100-nt targets for exact X; and32 repeated60-nt targets for exact ensemble P without seeds. Repeated copies make sampling long enough to resolve coarse rankings. These profiles characterize distinct paths, not mutually comparable biological workloads. + +The user-space collector emits a timer-reset warning on this host. A nominal100ms interval gave sampled totals close to recorded process CPU; shorter nominal intervals failed that calibration in tooling probes. Percentages remain coarse sampled evidence. Optimized stack unwinding misses some ancestors, so inclusive rows must not be added or interpreted as exhaustive phase fractions. The complete tables and warning text remain in `profiles/`. + +PROFILE_REVIEW_INSERT + +## Recommended fixes and optimization priorities + +1. **Fix input-stream cleanup.** Correct the stream-type handling in [general.cpp](source/src/IntaRNA/general.cpp) at line 136 and add a repeated-open/cleanup regression check. The supplied probe confirms retained descriptors against the freshly built library; this is a concrete resource defect, separate from speculative performance improvements. +2. **Default X/H: reduce repeated internal-loop energy work.** `InteractionEnergyVrna::getE_interLeft` is the strongest measured CPU hotspot. Inspect repeated sequence-code and base-pair-type lookups, invariant loop bounds, and validation in the nested extension loops before changing the algorithm. Keep exact energy and structural regression checks; no speedup from such a change has been measured here. +3. **Exact X: reduce repeated boundary-energy and dangling-probability work.** Repeated ED lookups and exponentials have a different cost profile from default X. Evaluate caching or hoisting only with a numerical-correctness check and a measured memory tradeoff; these calculations contribute to the thermodynamic score. +4. **Reuse work across predictions.** Batched targets demonstrably parallelize; matched precomputed ED reuse helps repeated runs. Address or explicitly control the reader/writer limit asymmetry before treating caching as transparent. +5. **Ensemble P: measure recurrence and partition-map changes separately.** `fillHybridZ`, loop energy evaluation, exponentials and boundary-map maintenance contribute distinct costs. Preserve the ensemble definition and numerical behavior when evaluating changes. +6. **Treat matrix storage and windowing by algorithm.** The measured RSS benefit is strong for S/H and negligible for this bounded X/H workload. Heap traces and source review identify allocation reuse candidates, but allocation traffic is not itself proof of a dominant runtime bottleneck. +7. **Keep the benchmark harness as a regression baseline.** Compare proposed patches against the same binary provenance, cases, environment, warmups, repetitions and output checks. Report algorithm changes separately from implementation speedups. + +## Validation and limits + +- Fresh regression tests:4,281 assertions in36 API cases, plus all20 CLI golden cases. No tracked source was changed. Original untracked executables and backup remain intact. +- All56 benchmark cases completed their warmup and three measured runs. Every case's output was stable. The independent [audit](results/audit.md) verifies224 records, original stdout/input/binary hashes, resource data, commands, recalculated summaries and structural comparisons. +- Four original no-seed records contain a normal informational log line on stdout. Derived validation excludes that line; the raw evidence is retained. Profiler output comparison similarly excludes only the collector's recognized experiment-creation banner and IntaRNA log lines. +- Excluding the two intentionally varied BLAS cases, the largest min–max range is4.33% of the median. BLAS-unset and BLAS-12 are noisy: their ranges are {D['blas_unset']['min_s']:.3f}–{D['blas_unset']['max_s']:.3f} s and {D['blas_twelve']['min_s']:.3f}–{D['blas_twelve']['max_s']:.3f} s, versus {t('blas_one_control'):.3f} s with BLAS1. They demonstrate startup sensitivity, not a precise universal penalty. +- Hardware cycles, IPC, cache misses and branch misses were unavailable: `perf_event_paranoid=4` blocked perf. Syscall tracing was denied; gprofng I/O tracing failed its initialization probe. GNU time resource counters and process CPU evidence are available, but no detailed disk/network/syscall bottleneck claim is made. +- Heap tracing reports allocated bytes, tracked high-water usage and allocations outstanding at process exit. It is not a memory-safety or leak proof. RSS includes additional process mappings and allocator state. Instrumented runs supply attribution, never the clean timing medians. +- This is one build on one laptop-class CPU, with a bounded workload suite and warm page caches. It excludes cross-PR/version comparisons, a full4,319-target scan, all scientific parameter combinations, NUMA/multi-host scaling, energy-consumption measurement and biological prediction-accuracy benchmarking. Large exact/ensemble runs were deliberately bounded. No universal worst-case or tail-latency claim is made. + +## Complete clean timing results + +Every row below has three measured repetitions; warmups are excluded. "Peak RSS" is the median of the process high-water marks; maximum observed values are also in [summary.csv](results/summary.csv). Exact arguments and experimental controls are in [cases.json](cases.json). + +{table([r['case'] for r in S])} + +## Reproduction and evidence + +See [README.md](README.md) for the sequential build/benchmark/profile commands and artifact map. The machine-readable [summary](results/summary.json), [validated runs](results/validated-runs.jsonl), [raw runs](results/runs.jsonl), [CPU/heap output checks](profiles/validated-outputs.json), [phase duration bounds](profiles/phase-summary.json), and [council reviews](council/) preserve the basis for the findings. The application source and fresh binaries are retained locally under `source/`. +''' +doc=doc.replace('PROFILE_REVIEW_INSERT',review) +for old,new in { + 'ChiX length84':'ChiX length 84', 'to75/75':'to 75/75', 'separate75-nt':'separate 75-nt', + 'default150/100':'default 150/100', 'limits149/99':'limits 149/99', 'took0.847':'took 0.847', + 'Original100/150':'Original 100/150', 'become99/149':'become 99/149', 'Overlap is150':'Overlap is 150', + 'cap is150':'cap is 150', 'most144×144':'most 144×144', 'add95':'add 95', + 'a48-target':'a 48-target', 'a3,000':'a 3,000', 'cap150':'cap 150', 'and32':'and 32', + 'repeated100-nt':'repeated 100-nt', 'repeated60-nt':'repeated 60-nt', 'nominal100ms':'nominal 100 ms', + 'tests:4,281':'tests: 4,281', 'in36 API':'in 36 API', 'all20 CLI':'all 20 CLI', + 'All56':'All 56', 'verifies224':'verifies 224', 'is4.33%':'is 4.33%', 'BLAS1':'BLAS 1', + 'full4,319':'full 4,319', + 'Windows add 95 target/query window combinations':'Windowing processes 95 target/query window combinations', +}.items():doc=doc.replace(old,new) +(ROOT/'REPORT.md').write_text(doc) +print(ROOT/'REPORT.md') diff --git a/doc/local-validation-archive/files/profiling/scripts/stream_cleanup_probe.cpp b/doc/local-validation-archive/files/profiling/scripts/stream_cleanup_probe.cpp new file mode 100644 index 00000000..db5637dc --- /dev/null +++ b/doc/local-validation-archive/files/profiling/scripts/stream_cleanup_probe.cpp @@ -0,0 +1,29 @@ +#include +#include +#include +namespace IntaRNA { +std::istream* newInputStream(const std::string&); +void deleteInputStream(std::istream*&); +} +std::size_t descriptor_count() { + std::size_t count=0; + for (const auto& entry: std::filesystem::directory_iterator("/proc/self/fd")) { + (void)entry; ++count; + } + return count; +} +int main(int argc,char** argv) { + if (argc!=2) return 2; + auto before=descriptor_count(); + for (int i=0;i<100;++i) { + auto* input=IntaRNA::newInputStream(argv[1]); + if (!input) return 3; + std::string line; + std::getline(*input,line); + IntaRNA::deleteInputStream(input); + if (input!=nullptr) return 4; + } + auto after=descriptor_count(); + std::cout << "{\"iterations\":100,\"fd_before\":" << before + << ",\"fd_after\":" << after << ",\"fd_growth\":" << after-before << "}\n"; +} diff --git a/doc/local-validation-archive/files/profiling/scripts/stream_cleanup_probe.sh b/doc/local-validation-archive/files/profiling/scripts/stream_cleanup_probe.sh new file mode 100644 index 00000000..1d37d3ce --- /dev/null +++ b/doc/local-validation-archive/files/profiling/scripts/stream_cleanup_probe.sh @@ -0,0 +1,13 @@ +#!/usr/bin/env bash +set -euo pipefail +profile_root=$(cd "$(dirname "$0")/.." && pwd) +dependency_root=${INTARNA_DEPS:-/home/alex/CursorProjects/IntaRNA_optimization/intaRNA_legacy/.conda-env} +"$dependency_root/bin/x86_64-conda-linux-gnu-g++" -std=c++23 -O2 \ + "$profile_root/scripts/stream_cleanup_probe.cpp" \ + "$profile_root/source/src/IntaRNA/libIntaRNA.a" \ + -L"$dependency_root/lib" -Wl,-rpath,"$dependency_root/lib" \ + -lboost_regex -lboost_iostreams -lz -fopenmp \ + -o "$profile_root/results/stream_cleanup_probe" +"$profile_root/results/stream_cleanup_probe" "$profile_root/inputs/t100.fa" \ + > "$profile_root/results/stream-cleanup-probe.json" +cat "$profile_root/results/stream-cleanup-probe.json" diff --git a/doc/local-validation-archive/files/profiling/scripts/structural_checks.py b/doc/local-validation-archive/files/profiling/scripts/structural_checks.py new file mode 100644 index 00000000..6e9069a0 --- /dev/null +++ b/doc/local-validation-archive/files/profiling/scripts/structural_checks.py @@ -0,0 +1,23 @@ +#!/usr/bin/env python3 +"""Untimed correctness checks including exact base-pair lists.""" +import json,subprocess,hashlib +from benchmark import ROOT,BIN,ENV,canonical +from cache_match import main as run_matched_cache_cases +# This stage follows supplement.py and precedes profiling; add the newly discovered +# effective-cap control while clean timing is still isolated from instrumentation. +run_matched_cache_cases() +cases={c['name']:c for c in json.loads((ROOT/'cases.json').read_text())} +dest=ROOT/'results/structural';dest.mkdir(exist_ok=True) +rows=[] +for name in ['throughput_1t','throughput_12t','cache_computed_control','cache_reused','cache_matched_computed','cache_matched_reused']: + args=[('--outCsvCols=id1,id2,start1,end1,start2,end2,E,bpList') if a.startswith('--outCsvCols=') else a for a in cases[name]['args']] + cmd=[str(BIN)]+args + p=subprocess.run(cmd,env=ENV,capture_output=True,timeout=120) + (dest/(name+'.stdout')).write_bytes(p.stdout);(dest/(name+'.stderr')).write_bytes(p.stderr) + rows.append(dict(case=name,command=cmd,exit_code=p.returncode,canonical_sha256=hashlib.sha256(canonical(p.stdout)).hexdigest())) +lookup={r['case']:r for r in rows} +comparisons={label:lookup[a]['exit_code']==lookup[b]['exit_code']==0 and lookup[a]['canonical_sha256']==lookup[b]['canonical_sha256'] + for label,a,b in [('threads_1_vs_12','throughput_1t','throughput_12t'),('cached_vs_computed','cache_computed_control','cache_reused'), + ('cache_matched_caps','cache_matched_computed','cache_matched_reused')]} +(dest/'checks.json').write_text(json.dumps(dict(runs=rows,comparisons=comparisons),indent=2)+'\n') +print(json.dumps(comparisons)) diff --git a/doc/local-validation-archive/files/profiling/scripts/supplement.py b/doc/local-validation-archive/files/profiling/scripts/supplement.py new file mode 100644 index 00000000..917c1394 --- /dev/null +++ b/doc/local-validation-archive/files/profiling/scripts/supplement.py @@ -0,0 +1,43 @@ +#!/usr/bin/env python3 +"""Run after benchmark.py: matched cache/BLAS and full-matrix memory checks.""" +import json, subprocess, time, hashlib +from pathlib import Path +from benchmark import ROOT, BIN, ENV, INPUT, run, summarize + +def main(): + cases=json.loads((ROOT/'cases.json').read_text()) + by_name={c['name']:c for c in cases} + cache=ROOT/'inputs/cache';cache.mkdir(exist_ok=True) + source=by_name['accessibility_computed'] + cmd=[str(BIN)]+source['args']+['--out=tAcc:'+str(cache/'target.ed'),'--out=qAcc:'+str(cache/'query.ed')] + start=time.perf_counter() + p=subprocess.run(cmd,env=ENV,capture_output=True) + (ROOT/'logs/cache-generation.stdout').write_bytes(p.stdout) + (ROOT/'logs/cache-generation.stderr').write_bytes(p.stderr) + (ROOT/'logs/cache-generation.json').write_text(json.dumps(dict(command=cmd,wall_s=time.perf_counter()-start,exit_code=p.returncode, + note='One-time ED generation includes a complete prediction and text serialization.'),indent=2)+'\n') + p.check_returncode() + new=[] + def variant(base,name,extra=(),**kw): + c=dict(by_name[base],name=name,args=by_name[base]['args']+list(extra),**kw) + new.append(c) + variant('accessibility_computed','cache_computed_control') + variant('accessibility_computed','cache_reused',['--tAcc=E','--tAccFile='+str(cache/'target.ed'), + '--qAcc=E','--qAccFile='+str(cache/'query.ed')]) + variant('bio_fhla_oxys','blas_one_control') + variant('bio_fhla_oxys','blas_unset',env_overrides={'OPENBLAS_NUM_THREADS':None}) + variant('bio_fhla_oxys','blas_twelve',env_overrides={'OPENBLAS_NUM_THREADS':'12'}) + variant('memory_t3000_q800','memory_s',['--model=S']) + variant('memory_window300','memory_s_window300',['--model=S']) + cases+=new + (ROOT/'cases.json').write_text(json.dumps(cases,indent=2)+'\n') + eligible=[c for c in new if run(c,'warmup',0,120)['exit_code']==0] + import random + rng=random.Random(7312027) + for n in range(1,4): + order=eligible[:];rng.shuffle(order) + for c in order:run(c,'measure',n,120) + summarize() + (cache/'manifest.json').write_text(json.dumps({p.name:dict(bytes=p.stat().st_size,sha256=hashlib.sha256(p.read_bytes()).hexdigest()) for p in cache.glob('*.ed')},indent=2)+'\n') + +if __name__=='__main__':main() diff --git a/doc/local-validation-archive/manifest.json b/doc/local-validation-archive/manifest.json new file mode 100644 index 00000000..49fe8de6 --- /dev/null +++ b/doc/local-validation-archive/manifest.json @@ -0,0 +1,1382 @@ +{ + "scope": "Historical local validation artifacts; inactive in the build and CI.", + "files": [ + { + "original_path": "issue-259-revision-validation/header-probe.cpp", + "archived_path": "doc/local-validation-archive/files/issue-259-revision-validation/header-probe.cpp", + "bytes": 59, + "sha256": "76f2239b274293c80d974530baf744d99521e969a286afc1c41d6c95cedf8e5d", + "git_blob_sha1": "7e0238c29287273e6a86af79f638e93b062efbe9" + }, + { + "original_path": "issue-259-revision-validation/PredictorMfeEns2d-consumer.cpp", + "archived_path": "doc/local-validation-archive/files/issue-259-revision-validation/PredictorMfeEns2d-consumer.cpp", + "bytes": 1408, + "sha256": "a1c96726a96896d591d2623690a983dd3024a6bda8f2cdf1e8211d07dea1fcb2", + "git_blob_sha1": "de61a0539aa10075d9a9ddec378bac2aed91ea13" + }, + { + "original_path": "issue-259-revision-validation/PredictorMfeEns2dHeuristicSeedExtension-consumer.cpp", + "archived_path": "doc/local-validation-archive/files/issue-259-revision-validation/PredictorMfeEns2dHeuristicSeedExtension-consumer.cpp", + "bytes": 1490, + "sha256": "93b44bfe9f4f3a70e60c973df98a3184c268e1812f5005828b6019004462e5ff", + "git_blob_sha1": "ecc322ede2c297aba6eaeb4884b29f2dca0cee73" + }, + { + "original_path": "issue-259-revision-validation/validate-installed.py", + "archived_path": "doc/local-validation-archive/files/issue-259-revision-validation/validate-installed.py", + "bytes": 2310, + "sha256": "fbea014b8f9e0307eef078425dd9a91e4fbe6f3eb0efdfdb51d358264b6c3cc3", + "git_blob_sha1": "2acf044974369e4819a36a597f287e5a1ee72e8b" + }, + { + "original_path": "issue-259-revision-validation/PredictorMfeEns2dHeuristic-consumer.cpp", + "archived_path": "doc/local-validation-archive/files/issue-259-revision-validation/PredictorMfeEns2dHeuristic-consumer.cpp", + "bytes": 1425, + "sha256": "96377cf38f9915d414d47d17352bcf220f9e43919c31cb6f4ec6354cca5c2f72", + "git_blob_sha1": "300009200644b19a02e87b632e337ca8482a2046" + }, + { + "original_path": "issue-259-revision-validation/benchmark_revision.py", + "archived_path": "doc/local-validation-archive/files/issue-259-revision-validation/benchmark_revision.py", + "bytes": 6865, + "sha256": "726264d93ee33993e9ccef3d448135095306f5d81cd3739e593816da6fc43504", + "git_blob_sha1": "426ab8d4de9fe627f057669439986e3c7c134d65" + }, + { + "original_path": "issue-259-revision-validation/check_legacy_seed_polynomials.py", + "archived_path": "doc/local-validation-archive/files/issue-259-revision-validation/check_legacy_seed_polynomials.py", + "bytes": 3737, + "sha256": "f14acf905126f1c2fbc6de3c1f29b947d058d526d9b6302a07712937d411ec4f", + "git_blob_sha1": "61044ba4ce50f8cf590ea7df1885170c08f660f7" + }, + { + "original_path": "issue-259-revision-validation/PredictorMfeEns2dSeedExtension-consumer.cpp", + "archived_path": "doc/local-validation-archive/files/issue-259-revision-validation/PredictorMfeEns2dSeedExtension-consumer.cpp", + "bytes": 1473, + "sha256": "472995bc857cc234475c6fcb5253d7256b347aabb0b3a0808fa357af78928361", + "git_blob_sha1": "5837ef1e5d23ed67103509684b4a5bb1a2e41665" + }, + { + "original_path": "issue-262-validation/logs/hash-debug-prerequisite.patch", + "archived_path": "doc/local-validation-archive/files/issue-262-validation/logs/hash-debug-prerequisite.patch", + "bytes": 596, + "sha256": "bc9fdd38fc7b227026e0c2e2ebe6f6ab2b30f3751a72a7220427b3c43366a1a3", + "git_blob_sha1": "aac355978ed18e912277260c51cb3aebae0f4fed" + }, + { + "original_path": "issue-262-validation/hash-prototype/transform.py", + "archived_path": "doc/local-validation-archive/files/issue-262-validation/hash-prototype/transform.py", + "bytes": 3636, + "sha256": "f74a2f83835266403e8a9b9efb40a42a9c844bffbeaff26e4a38fc6e388f5ce5", + "git_blob_sha1": "cb1254568035a961214ed4706022f987639c47a1" + }, + { + "original_path": "issue-262-validation/hash-prototype/SeededPartitionFunction.cpp", + "archived_path": "doc/local-validation-archive/files/issue-262-validation/hash-prototype/SeededPartitionFunction.cpp", + "bytes": 6499, + "sha256": "d529d8f13d5b615417e57e4b53835db1a7d64a08de1ede3180fa62c4d4f60431", + "git_blob_sha1": "6680389815f124453875597dacef910c6b5bcdfb" + }, + { + "original_path": "issue-262-validation/hash-prototype/allocation-probe.cpp", + "archived_path": "doc/local-validation-archive/files/issue-262-validation/hash-prototype/allocation-probe.cpp", + "bytes": 2891, + "sha256": "f190e4009779dd817617888a3ad208dfa5d13e76429ebaeab67507e9520a66cc", + "git_blob_sha1": "da23e5e5f5babc22bfe254949b3b557db687bfe1" + }, + { + "original_path": "issue-244-validation/issue-comment.md", + "archived_path": "doc/local-validation-archive/files/issue-244-validation/issue-comment.md", + "bytes": 1080, + "sha256": "1f65280046eb19c805b8806696316eb211dbb6d4494c5a4a4b14c3f5876de315", + "git_blob_sha1": "8141b9db6ae4ead1ce74548c638dd55d5276d0d2" + }, + { + "original_path": "issue-244-validation/council-consensus.md", + "archived_path": "doc/local-validation-archive/files/issue-244-validation/council-consensus.md", + "bytes": 941, + "sha256": "161e1b262e9acf521a81933d360382f1d72283913300f8b350400d115a048968", + "git_blob_sha1": "eb494413958500e9a9950e1cad706d7217b602fe" + }, + { + "original_path": "issue-244-validation/source-gcc11-debug/README.md", + "archived_path": "doc/local-validation-archive/files/issue-244-validation/source-gcc11-debug/README.md", + "bytes": 101395, + "sha256": "f3e95de8a90921a1703467be6d4a9eaff4b1771c1ac97a8ceb57b5027a7e592a", + "git_blob_sha1": "050edfc84758d3470573028cc4d0ce4c2fb8c727" + }, + { + "original_path": "issue-244-validation/source-gcc11-debug/src/IntaRNA/RnaSequence.h", + "archived_path": "doc/local-validation-archive/files/issue-244-validation/source-gcc11-debug/src/IntaRNA/RnaSequence.h", + "bytes": 15738, + "sha256": "e475285cbc18fb8507372e0ecf515cfda0a1b60877e32f5182bb3c14a34da1ed", + "git_blob_sha1": "ea1a680b34b088e4f32935e5571a8e9d8848e993" + }, + { + "original_path": "issue-244-validation/source-gcc11-debug/src/IntaRNA/AccessibilityVrna.cpp", + "archived_path": "doc/local-validation-archive/files/issue-244-validation/source-gcc11-debug/src/IntaRNA/AccessibilityVrna.cpp", + "bytes": 16400, + "sha256": "3684ac112e7088e85416707473e1441a0b4e48f0f28f8262843f8a9343c44e48", + "git_blob_sha1": "d2493a8ff58feda8813a756d13dbd886bee217bb" + }, + { + "original_path": "issue-244-validation/source-gcc11-debug/src/IntaRNA/InteractionEnergyVrna.cpp", + "archived_path": "doc/local-validation-archive/files/issue-244-validation/source-gcc11-debug/src/IntaRNA/InteractionEnergyVrna.cpp", + "bytes": 7766, + "sha256": "94226bc45e16b8fd02fd924d793230aa8f53fabe68db0ea26e7ccc911db249c9", + "git_blob_sha1": "23c7386862012ac5fb74ae0ace27988b342010da" + }, + { + "original_path": "issue-244-validation/source-gcc11-debug/doc/refactor/2-c++23.md", + "archived_path": "doc/local-validation-archive/files/issue-244-validation/source-gcc11-debug/doc/refactor/2-c++23.md", + "bytes": 33964, + "sha256": "e31e9e9822d2e5f7bc17070b0ee404f37861f8e6f6c9cd9fd07ebd743e45ea64", + "git_blob_sha1": "5ce46b43062278ba1a4b117dc89a3e18fab3e1ec" + }, + { + "original_path": "issue-244-validation/source-gcc11-debug/tests/RnaSequence_test.cpp", + "archived_path": "doc/local-validation-archive/files/issue-244-validation/source-gcc11-debug/tests/RnaSequence_test.cpp", + "bytes": 4150, + "sha256": "ce20a52fc4e7ff23b937842a4b404429bd61610936355d020338ed69c061c34c", + "git_blob_sha1": "d11842b172868afdb8194d30de1271de376a85f1" + }, + { + "original_path": "profiling/README.md", + "archived_path": "doc/local-validation-archive/files/profiling/README.md", + "bytes": 4649, + "sha256": "053510f755e971474291a193e23d842d04458e0c9ad79fa6e182453d897577cd", + "git_blob_sha1": "9b3a7578f8506aa4ef882201e9add54048ca3542" + }, + { + "original_path": "profiling/REPORT.md", + "archived_path": "doc/local-validation-archive/files/profiling/REPORT.md", + "bytes": 25220, + "sha256": "37e49062f9261ff276ea84cb92e8854d5385380b2a0e93d68451f6355ae0f357", + "git_blob_sha1": "bc031086e3c3f6d07dfe012d781aebf053cc84df" + }, + { + "original_path": "profiling/scripts/cache_match.py", + "archived_path": "doc/local-validation-archive/files/profiling/scripts/cache_match.py", + "bytes": 931, + "sha256": "6b69d659d40148eb12573e1ad7f1ffb71470c18362152d10e5bf39c61d322e2a", + "git_blob_sha1": "6ea94622aca1d1581dd785899971834bdac2f00d" + }, + { + "original_path": "profiling/scripts/build.sh", + "archived_path": "doc/local-validation-archive/files/profiling/scripts/build.sh", + "bytes": 1274, + "sha256": "1f7b6c3c85b2418e4f4f78dccba4b0c8834b810fbf18b290f9056fa431d52314", + "git_blob_sha1": "82a51cc5c3712af97d0fcf5abe42d0977540baa6" + }, + { + "original_path": "profiling/scripts/benchmark.py", + "archived_path": "doc/local-validation-archive/files/profiling/scripts/benchmark.py", + "bytes": 11426, + "sha256": "ff89fc15f75e0a3b8181dfdc96c623d73534e069cfb19b65ba5af3d461af221d", + "git_blob_sha1": "93fcbe9d7d0ebfc4b7bb9c1879903b529268ed78" + }, + { + "original_path": "profiling/scripts/stream_cleanup_probe.sh", + "archived_path": "doc/local-validation-archive/files/profiling/scripts/stream_cleanup_probe.sh", + "bytes": 723, + "sha256": "849a43f438f15436008bae004ca9a38c062c2c98a3ca2cad0cb9a7ba6284d7e8", + "git_blob_sha1": "1d37d3ce3ac1f7a04021c827d9ef4d84ebf83615" + }, + { + "original_path": "profiling/scripts/plots.py", + "archived_path": "doc/local-validation-archive/files/profiling/scripts/plots.py", + "bytes": 4239, + "sha256": "fbfaed395cb123d5c5bc83b057d0712206dc8968f3b8d8c56ac4649ab56b922a", + "git_blob_sha1": "de3c7cf97bbe2ff428730b0884be8d995c3b6dda" + }, + { + "original_path": "profiling/scripts/profile.py", + "archived_path": "doc/local-validation-archive/files/profiling/scripts/profile.py", + "bytes": 4279, + "sha256": "13ae4b92c2ef3e8d3ed4c74158ee486beddd222901bf390b96dc0771b572f3bd", + "git_blob_sha1": "fc03c02791fdc6fa4fc5313b16eab554a254372b" + }, + { + "original_path": "profiling/scripts/report.py", + "archived_path": "doc/local-validation-archive/files/profiling/scripts/report.py", + "bytes": 19285, + "sha256": "dd0b1545deacf7db3db10688240ec108bf99cadc7d2ae225afed4113ea8e0ad6", + "git_blob_sha1": "ed17957a8c82a1e14fa6e62bb93ff355a9d8976c" + }, + { + "original_path": "profiling/scripts/brief_martin.py", + "archived_path": "doc/local-validation-archive/files/profiling/scripts/brief_martin.py", + "bytes": 13649, + "sha256": "6148f49c6a61743bbb976897d04fdf1c091c663267913b04e4688ffd5f37631c", + "git_blob_sha1": "f93e28872ce5b5d6779d30d3ac93f291d9e9df57" + }, + { + "original_path": "profiling/scripts/package.py", + "archived_path": "doc/local-validation-archive/files/profiling/scripts/package.py", + "bytes": 1685, + "sha256": "97a98290e5360531f7af5b5a668d63c4d989a9296233846199093aa1708814b5", + "git_blob_sha1": "c68e6e642f73e19530afb334fea9e2fc6d69ba44" + }, + { + "original_path": "profiling/scripts/profile_audit.py", + "archived_path": "doc/local-validation-archive/files/profiling/scripts/profile_audit.py", + "bytes": 1363, + "sha256": "9e34dba192b914dd9ef693d605b1118ae3919946fa63bd3f3bbc42021a5525bd", + "git_blob_sha1": "f1f27448625cdca197e42768c7f2c45b0effbd78" + }, + { + "original_path": "profiling/scripts/supplement.py", + "archived_path": "doc/local-validation-archive/files/profiling/scripts/supplement.py", + "bytes": 2305, + "sha256": "0d889927878de305d3855c96bc2a51fcc33813b9fdb777f7e6bfcc2148e1a39a", + "git_blob_sha1": "917c1394e7223330e892baf501fecca4da30aa7a" + }, + { + "original_path": "profiling/scripts/structural_checks.py", + "archived_path": "doc/local-validation-archive/files/profiling/scripts/structural_checks.py", + "bytes": 1677, + "sha256": "2d64c49d9b483744786bb740a8a141b368a0dbf6d4057da40694fd3c36521c99", + "git_blob_sha1": "6e9069a0d43ccf1f2b6e8a7161179d267886bafb" + }, + { + "original_path": "profiling/scripts/audit.py", + "archived_path": "doc/local-validation-archive/files/profiling/scripts/audit.py", + "bytes": 24858, + "sha256": "1ec9ca9bd4f96efaa313a87c4c5085dc1eab14e5cf6263237fc99584e7a543d4", + "git_blob_sha1": "5df6119fcbaffe67b5b3580021fe310d4f0ffea6" + }, + { + "original_path": "profiling/scripts/stream_cleanup_probe.cpp", + "archived_path": "doc/local-validation-archive/files/profiling/scripts/stream_cleanup_probe.cpp", + "bytes": 919, + "sha256": "397946ea954a5b681c804339a566bfeb99f7dab26352c13c6665a308287daab3", + "git_blob_sha1": "db5637dc2c579262a831d41d58e2686e4c6ce01e" + }, + { + "original_path": "profiling/scripts/phase_summary.py", + "archived_path": "doc/local-validation-archive/files/profiling/scripts/phase_summary.py", + "bytes": 1397, + "sha256": "fc2cb29c50ed583535bc91afa81ca793693291d60e7af5165cdfc46201109575", + "git_blob_sha1": "c568dd75bd288b45bb00103687b3ac5600e7b51d" + }, + { + "original_path": "issue-245-validation/build.sh", + "archived_path": "doc/local-validation-archive/files/issue-245-validation/build.sh", + "bytes": 999, + "sha256": "763efb10ef2ddbb9d263e059138394d20c7ca690b55f6e777c1e9d48ae0833e8", + "git_blob_sha1": "76ac21b364ce362f90f5ff1591409ee9d901be01" + }, + { + "original_path": "issue-245-validation/header.cpp", + "archived_path": "doc/local-validation-archive/files/issue-245-validation/header.cpp", + "bytes": 70, + "sha256": "db0f095822977591f2757f5b8f6cfd7c826589122a083101ff8e745fce928d3c", + "git_blob_sha1": "e1a55c222946d501eb3d47c3fc7b365c41ba481a" + }, + { + "original_path": "issue-245-validation/run-benchmark.sh", + "archived_path": "doc/local-validation-archive/files/issue-245-validation/run-benchmark.sh", + "bytes": 649, + "sha256": "edcfb36e4ac1df882a2b2fb436ea8ea6d8527c756c44c9935f7b1c3db43e5183", + "git_blob_sha1": "7cdca8248d5a42c3f05b1e805582ef29c3dfe504" + }, + { + "original_path": "issue-245-validation/check-headers.sh", + "archived_path": "doc/local-validation-archive/files/issue-245-validation/check-headers.sh", + "bytes": 524, + "sha256": "e1fb960abe43e51998da3970c8bfc067623af0ed9654622ca0883f4736f5d638", + "git_blob_sha1": "12efce57a50f2f02a45e57042e6ec066f0fcb820" + }, + { + "original_path": "issue-245-validation/benchmark.cpp", + "archived_path": "doc/local-validation-archive/files/issue-245-validation/benchmark.cpp", + "bytes": 2602, + "sha256": "cb146b799b1984b1924d82da6d677ebaed6cc3f9aa59dccc2469b9d26e8729b6", + "git_blob_sha1": "a6bc245adb17a296bfd938d8f486c28676954574" + }, + { + "original_path": "issue-245-validation/debug/doc/benchmarks/accessibility.md", + "archived_path": "doc/local-validation-archive/files/issue-245-validation/debug/doc/benchmarks/accessibility.md", + "bytes": 1591, + "sha256": "12e77dc1af0f7ea64b43be07822848ce9ea53d625a923c9e0c2279a369ca9448", + "git_blob_sha1": "da24fbf4ae2cecdf9d7ec81e446de222407f1716" + }, + { + "original_path": "issue-246-validation/record-gcc14-benchmark.py", + "archived_path": "doc/local-validation-archive/files/issue-246-validation/record-gcc14-benchmark.py", + "bytes": 3155, + "sha256": "3ca6c6d8d481435cc53639bbe1491f98204f7e05fe9dfc67918e58f2255a3d46", + "git_blob_sha1": "ee094e644377bc8e14241f9f2d02d9837dc9d6da" + }, + { + "original_path": "issue-246-validation/build.sh", + "archived_path": "doc/local-validation-archive/files/issue-246-validation/build.sh", + "bytes": 917, + "sha256": "ee5dec5d22aec981c16ddab613fab07d6533982fcc79842178f166d19e0d2ab1", + "git_blob_sha1": "61fb6c1cf16d54471e750a812d90123d4aff1b54" + }, + { + "original_path": "issue-246-validation/kokkos-installed-consumer.cpp", + "archived_path": "doc/local-validation-archive/files/issue-246-validation/kokkos-installed-consumer.cpp", + "bytes": 543, + "sha256": "5efaa5c0c30f7ed8ea763d2af718fce3adc98ede58dca85b5bb2a918ae4bef13", + "git_blob_sha1": "d9ac7989e414f7325239f7546dc1461feef22973" + }, + { + "original_path": "issue-246-validation/check-mdspan-configure.py", + "archived_path": "doc/local-validation-archive/files/issue-246-validation/check-mdspan-configure.py", + "bytes": 2213, + "sha256": "3f46899ca62e7cdc9de0fb34a6063b3e616bd98d1486a9cac60bb7d9ef6e35fb", + "git_blob_sha1": "a6a994883cdefabda0f08088f82de6198b0a307a" + }, + { + "original_path": "issue-246-validation/align-matrix-style.py", + "archived_path": "doc/local-validation-archive/files/issue-246-validation/align-matrix-style.py", + "bytes": 7702, + "sha256": "09225501d082d9903bfa303f69ce8058b563aa5a9453f07572100d5bf4653cad", + "git_blob_sha1": "fb76e159193fa688b0e9da62b99f602879b1a6ca" + }, + { + "original_path": "issue-246-validation/matrix-random-audit.cpp", + "archived_path": "doc/local-validation-archive/files/issue-246-validation/matrix-random-audit.cpp", + "bytes": 3884, + "sha256": "46158e4cb0aacc0b7990553cc7625ff4692bcf1fb3e0c98db347cc0a7bd3b024", + "git_blob_sha1": "112c3d93c16d42da948071837a6dbc6e7ca1082d" + }, + { + "original_path": "issue-246-validation/summarize.py", + "archived_path": "doc/local-validation-archive/files/issue-246-validation/summarize.py", + "bytes": 3274, + "sha256": "a1b07b62540c882f5f8567c95af67146393a14aaed507ecfde770e93d21b4bf9", + "git_blob_sha1": "e770a2b7d24ae49a162e9f948620ed036207a6de" + }, + { + "original_path": "issue-246-validation/installed-header-probe.cpp", + "archived_path": "doc/local-validation-archive/files/issue-246-validation/installed-header-probe.cpp", + "bytes": 61, + "sha256": "f1f68a746f2f97d3625ee6203a615f640dd6d0cd27909f64a05af3fef159e07d", + "git_blob_sha1": "2b45fa6d675bdd7443cd8705b5406a1df2147649" + }, + { + "original_path": "issue-246-validation/build-gcc14-clean.sh", + "archived_path": "doc/local-validation-archive/files/issue-246-validation/build-gcc14-clean.sh", + "bytes": 1028, + "sha256": "9899bad4182675e71995da7b158270cdc6dcf5775de82c50d2e8f73039c63aa8", + "git_blob_sha1": "37c52b91fd08c424aecd6a1bc29b65ac35a69ae6" + }, + { + "original_path": "issue-246-validation/consumer.cpp", + "archived_path": "doc/local-validation-archive/files/issue-246-validation/consumer.cpp", + "bytes": 397, + "sha256": "045ae708892283fa3acf79b91d8df9059ab67fa50222b468216d2b98d1160288", + "git_blob_sha1": "e77a762700666d3a57591d99fccd168da35085d0" + }, + { + "original_path": "issue-246-validation/kokkos-distribution-probe.cpp", + "archived_path": "doc/local-validation-archive/files/issue-246-validation/kokkos-distribution-probe.cpp", + "bytes": 346, + "sha256": "b67c433b592ace3e1797384309093d6946ddba3bdab412e3e1bb407ab14b838a", + "git_blob_sha1": "a1f94c970d79b626555e4371d32d0c40f8b98f63" + }, + { + "original_path": "issue-246-validation/build-gcc14.sh", + "archived_path": "doc/local-validation-archive/files/issue-246-validation/build-gcc14.sh", + "bytes": 974, + "sha256": "c797600bdf320b4b195b1112a27c2be12796817046ec3218e8dc8679f04450ae", + "git_blob_sha1": "a427906ba3d2617a7847f9dd9e9dc3eab98deeff" + }, + { + "original_path": "issue-246-validation/kokkos-distribution/intaRNA-3.4.1/doc/mdspan-storage.md", + "archived_path": "doc/local-validation-archive/files/issue-246-validation/kokkos-distribution/intaRNA-3.4.1/doc/mdspan-storage.md", + "bytes": 15159, + "sha256": "c4a1e5343ca3d99f774bec54f5ea2539bc4b7eae9699626a92971ec1b94ef681", + "git_blob_sha1": "21829303c0a274c2a9da97d5bfa492e7bbfb1ead" + }, + { + "original_path": "issue-246-validation/debug/src/IntaRNA/Matrix.h", + "archived_path": "doc/local-validation-archive/files/issue-246-validation/debug/src/IntaRNA/Matrix.h", + "bytes": 8148, + "sha256": "063566a98b84c9f705bd034e31ee808170497dffebe7a325f477dd83428dea54", + "git_blob_sha1": "1ee1854adbd6b1f343acd1237f86d1b540c541c2" + }, + { + "original_path": "issue-246-validation/debug/tests/benchmark/compare-matrix-storage.py", + "archived_path": "doc/local-validation-archive/files/issue-246-validation/debug/tests/benchmark/compare-matrix-storage.py", + "bytes": 6098, + "sha256": "816d56d72c8032702cfc5cdff20a40e5dc9f63b5bf766c3752c491ac7e701d1e", + "git_blob_sha1": "e4dd9e83d49373e2dea60b566a0d261bb0af72ca" + }, + { + "original_path": "kinetic-validation/header-probe.cpp", + "archived_path": "doc/local-validation-archive/files/kinetic-validation/header-probe.cpp", + "bytes": 76, + "sha256": "fcbc6c199cb09c30ed7c109fbf17cc43b1db56f834ffbef89c39a04b6e0898e8", + "git_blob_sha1": "66d40f790c1c48f1e46629fa0cf3227dd56037cb" + }, + { + "original_path": "kinetic-validation/verify-installed-review.py", + "archived_path": "doc/local-validation-archive/files/kinetic-validation/verify-installed-review.py", + "bytes": 1522, + "sha256": "8e9775db37cdeab89eaa469af2c4a4b83fb9ccb897177715fbbc95a19bbb1fa6", + "git_blob_sha1": "42fba5345807d2b720847cc678c4312451ad7fd6" + }, + { + "original_path": "kinetic-validation/verify-installed-kix.py", + "archived_path": "doc/local-validation-archive/files/kinetic-validation/verify-installed-kix.py", + "bytes": 2078, + "sha256": "4a26218968bd6d7404c0a9bfc24a20176c6436fc8ea6c9b41a538c48f765e9b1", + "git_blob_sha1": "cfcb21c67b74a98f3b8d5b4e6fed34a3772e5d54" + }, + { + "original_path": "kinetic-validation/build-uncached.py", + "archived_path": "doc/local-validation-archive/files/kinetic-validation/build-uncached.py", + "bytes": 1755, + "sha256": "8263163b09474287696a8a89ada7743679bebc83cd2ae57cc398b2811f25554a", + "git_blob_sha1": "0d982f41df4fae7e421549270d5e9970b3e9201c" + }, + { + "original_path": "kinetic-validation/consumer.cpp", + "archived_path": "doc/local-validation-archive/files/kinetic-validation/consumer.cpp", + "bytes": 965, + "sha256": "316354315760b4fb478ad106acc36550824c6121f472bbbb24a68a51614cf0e0", + "git_blob_sha1": "2a554b72ad1b6dad91e5f08a61566110717f9ef3" + }, + { + "original_path": "kinetic-validation/header-probe-review.cpp", + "archived_path": "doc/local-validation-archive/files/kinetic-validation/header-probe-review.cpp", + "bytes": 68, + "sha256": "0cc195056aa894fc6fb3b7b231c2c5a6b3451ccec2840fcd94cd69703099867e", + "git_blob_sha1": "e4bb252979fe29831734fe585aef3e36f00f70cb" + }, + { + "original_path": "kinetic-validation/consumer-review.cpp", + "archived_path": "doc/local-validation-archive/files/kinetic-validation/consumer-review.cpp", + "bytes": 977, + "sha256": "806b3e196c51de1a4343f95243760b87f28e8daac0af4f90ac532905faf3f5df", + "git_blob_sha1": "fe65413dfa6a2324583a81f837e703de4e4882f3" + }, + { + "original_path": "kinetic-validation/header-probe-kix.cpp", + "archived_path": "doc/local-validation-archive/files/kinetic-validation/header-probe-kix.cpp", + "bytes": 68, + "sha256": "0cc195056aa894fc6fb3b7b231c2c5a6b3451ccec2840fcd94cd69703099867e", + "git_blob_sha1": "e4bb252979fe29831734fe585aef3e36f00f70cb" + }, + { + "original_path": "kinetic-validation/uncached-comparison/PredictorSeedExtensionKinetic.cpp", + "archived_path": "doc/local-validation-archive/files/kinetic-validation/uncached-comparison/PredictorSeedExtensionKinetic.cpp", + "bytes": 14548, + "sha256": "ee42112e3130e93a360c792272a56354d53339752fafcb89a52b8eff7781249d", + "git_blob_sha1": "8d4233df87e79a015a7fbd2e8c197e6e3a257b24" + }, + { + "original_path": "issue-274-validation/header-probe.cpp", + "archived_path": "doc/local-validation-archive/files/issue-274-validation/header-probe.cpp", + "bytes": 121, + "sha256": "49ad3e039c027b886b17e8e106d5d349f1558ed259b43cc5cc9ff39055b104a7", + "git_blob_sha1": "2023293bb06f0821eb5b31e94053bf2613a10653" + }, + { + "original_path": "issue-274-validation/record-source.py", + "archived_path": "doc/local-validation-archive/files/issue-274-validation/record-source.py", + "bytes": 1856, + "sha256": "becf25ee761d3cfab4713f263e70047d6a50f48b79186e55e4926073901646ee", + "git_blob_sha1": "3768ef5eb28c2ff9bf662e4836ac58e2f356aa88" + }, + { + "original_path": "issue-264-validation/native-heuristic-timing.py", + "archived_path": "doc/local-validation-archive/files/issue-264-validation/native-heuristic-timing.py", + "bytes": 2203, + "sha256": "bf60ef26ce02fe88818dd251a8123fb38f71fe4fc7873644b5e988b58da7810b", + "git_blob_sha1": "174d921ddc985d42dff1aa3f1e429125a98c08dc" + }, + { + "original_path": "issue-264-validation/debug/doc/analysis/seed-extension-partition.md", + "archived_path": "doc/local-validation-archive/files/issue-264-validation/debug/doc/analysis/seed-extension-partition.md", + "bytes": 13257, + "sha256": "62b360f65b41e1cbb09318f8427ce651639174108003bd0e7b994576c9b847c0", + "git_blob_sha1": "706e279d21e44e4b54f076b2f519c5c5d70faf57" + }, + { + "original_path": "issue-264-validation/core-benchmark/run.py", + "archived_path": "doc/local-validation-archive/files/issue-264-validation/core-benchmark/run.py", + "bytes": 3653, + "sha256": "f971d362f6e90b98e87d69d204f870bfe1a69041308580c0309826c5634b9378", + "git_blob_sha1": "107ae964dfab9ac07ee17c1ad7a0a182a0cb10bb" + }, + { + "original_path": "issue-264-validation/core-benchmark/build.py", + "archived_path": "doc/local-validation-archive/files/issue-264-validation/core-benchmark/build.py", + "bytes": 1454, + "sha256": "c3c8c514e3247eeeb5c7b551ecdd32ea0b8f248fa202e04373b2b735516d7850", + "git_blob_sha1": "073692f8d43a69c4ccb32ea3551375bafc22232c" + }, + { + "original_path": "issue-265-validation/benchmark/serial_followup.py", + "archived_path": "doc/local-validation-archive/files/issue-265-validation/benchmark/serial_followup.py", + "bytes": 2396, + "sha256": "0f80e5396b5b44c6b092e611a38511f65bbf5389ae4ba224e6547d87fc8f05d9", + "git_blob_sha1": "ac9b610a009cb174b4b5048afaa81abbc169094c" + }, + { + "original_path": "issue-265-validation/benchmark/benchmark.py", + "archived_path": "doc/local-validation-archive/files/issue-265-validation/benchmark/benchmark.py", + "bytes": 4946, + "sha256": "b51b9113dd24b3afb9857025e38f738e646a858bc2207b8dc3f1040cddea0248", + "git_blob_sha1": "59ce68b8007219ba84461efca79722a728ff5852" + }, + { + "original_path": "issue-265-validation/probe/vrna_window_threads.cpp", + "archived_path": "doc/local-validation-archive/files/issue-265-validation/probe/vrna_window_threads.cpp", + "bytes": 5793, + "sha256": "79986e92d91007418a72f87ad37124f59d5bdf4a665297697a73cdbb2d124d35", + "git_blob_sha1": "0052a8df18109b030d7034741a1676c41836cbc1" + }, + { + "original_path": "issue-265-validation/debug/src/IntaRNA/AccessibilityVrna.h", + "archived_path": "doc/local-validation-archive/files/issue-265-validation/debug/src/IntaRNA/AccessibilityVrna.h", + "bytes": 6228, + "sha256": "e5247e1642a474b117e1b07af29e920fa498d26cbd26210762e80cac15f73b76", + "git_blob_sha1": "1972843d914081b776223b6801aa4c3198d852cb" + }, + { + "original_path": "issue-265-validation/debug/doc/analysis/vienna-accessibility-thread-safety.md", + "archived_path": "doc/local-validation-archive/files/issue-265-validation/debug/doc/analysis/vienna-accessibility-thread-safety.md", + "bytes": 14588, + "sha256": "fab3b9438e7a35106ef56a9e32d6189f057c6352436d85b043b2039315989436", + "git_blob_sha1": "49d8ca65393d4b94b6764836b1093da5e1f6dab6" + }, + { + "original_path": "issue-265-validation/fallback/run.py", + "archived_path": "doc/local-validation-archive/files/issue-265-validation/fallback/run.py", + "bytes": 1608, + "sha256": "194c23460b764381698e61a779d14b7c44bb6e852d179098b76a9acf587c05cc", + "git_blob_sha1": "0a3ff603304c4edff79d44dfb3954eb061e5c6f1" + }, + { + "original_path": "issue-265-validation/fallback/probe.cpp", + "archived_path": "doc/local-validation-archive/files/issue-265-validation/fallback/probe.cpp", + "bytes": 3158, + "sha256": "5caeaabae060a6a8565dca5980244c36e8dc5f0864f7805f5201ffa210856e42", + "git_blob_sha1": "4a101ff39adef2efb2394f83f097f4e6154e2823" + }, + { + "original_path": "issue-265-validation/fallback/override/ViennaRNA/vrna_config.h", + "archived_path": "doc/local-validation-archive/files/issue-265-validation/fallback/override/ViennaRNA/vrna_config.h", + "bytes": 131, + "sha256": "b4985947d1d22cfb3a1a6d61c6770150d66ebdf853761995559d7ca08be5dd60", + "git_blob_sha1": "37f9f67718b9331a8ffb0daf82325078cc6c4572" + }, + { + "original_path": "issue-268-validation/sync-candidate.py", + "archived_path": "doc/local-validation-archive/files/issue-268-validation/sync-candidate.py", + "bytes": 679, + "sha256": "f9ce47d0aa6207ee60bd34b2d7ba067241e8e30da8ff514d28865fbca7077b66", + "git_blob_sha1": "77c7a30c5f84f4b8b91bf6caf12338f24069e844" + }, + { + "original_path": "issue-268-validation/installed-SeededPartitionFunction.h.cpp", + "archived_path": "doc/local-validation-archive/files/issue-268-validation/installed-SeededPartitionFunction.h.cpp", + "bytes": 70, + "sha256": "f02aa93aaaca9764b81b28057d597691f700b2a116e762cc3bd2ee7bfc3ed373", + "git_blob_sha1": "fa6f256cc12c5b34185c78ab95b902e8029ac098" + }, + { + "original_path": "issue-268-validation/compare-native-results.py", + "archived_path": "doc/local-validation-archive/files/issue-268-validation/compare-native-results.py", + "bytes": 1505, + "sha256": "be140deee3127b08447a52752b560cc8ee3ce6808a5f99bd8f6dd75baab71cc8", + "git_blob_sha1": "50389d06b747a3251f9bd3b2fc56287f2444ecdd" + }, + { + "original_path": "issue-268-validation/check-installed.py", + "archived_path": "doc/local-validation-archive/files/issue-268-validation/check-installed.py", + "bytes": 1646, + "sha256": "7c5281d72538396e8ca11fa715d54303c356346aa4bf89baa94625332c5f5c1a", + "git_blob_sha1": "097e7830720801fd23d7ba25be0fc693544d9863" + }, + { + "original_path": "issue-268-validation/native-compare.py", + "archived_path": "doc/local-validation-archive/files/issue-268-validation/native-compare.py", + "bytes": 1714, + "sha256": "fa9427e3f8afec91cf2a4523b6f49f4234563ea0f0a45c401a9d514755f3150f", + "git_blob_sha1": "743df7f05e12ded23d4ca1a50e48b2686a5d1b3e" + }, + { + "original_path": "issue-268-validation/installed-StackSeedDomain.h.cpp", + "archived_path": "doc/local-validation-archive/files/issue-268-validation/installed-StackSeedDomain.h.cpp", + "bytes": 62, + "sha256": "13d5fdf05bb2cfe017967f8d0ee1cc14b520e9d6d5dda3a45877e5c0f8228b93", + "git_blob_sha1": "cf3c73c97538378c02ec28c61ab3e54e641f8c3b" + }, + { + "original_path": "issue-268-validation/leftmost-benchmark/candidate-batch256/IntaRNA/SeededPartitionFunction.cpp", + "archived_path": "doc/local-validation-archive/files/issue-268-validation/leftmost-benchmark/candidate-batch256/IntaRNA/SeededPartitionFunction.cpp", + "bytes": 26196, + "sha256": "6237b44f408e59a094b4399dcd90bcc6daf276fb71b1c22fa6b0edc360df9bcb", + "git_blob_sha1": "5bb328bfcd683f7a3e8d845b2a12bd27efc83fed" + }, + { + "original_path": "issue-268-validation/leftmost-benchmark/candidate-batch128/IntaRNA/SeededPartitionFunction.cpp", + "archived_path": "doc/local-validation-archive/files/issue-268-validation/leftmost-benchmark/candidate-batch128/IntaRNA/SeededPartitionFunction.cpp", + "bytes": 26196, + "sha256": "651099079f519284e932e932319c1c605368d0d6f97f22b0b10b85d12f838fed", + "git_blob_sha1": "e37e851c8fc625d4039039b7e06184341a818ced" + }, + { + "original_path": "issue-257-validation/header-probe.cpp", + "archived_path": "doc/local-validation-archive/files/issue-257-validation/header-probe.cpp", + "bytes": 61, + "sha256": "f1f68a746f2f97d3625ee6203a615f640dd6d0cd27909f64a05af3fef159e07d", + "git_blob_sha1": "2b45fa6d675bdd7443cd8705b5406a1df2147649" + }, + { + "original_path": "issue-257-validation/build.sh", + "archived_path": "doc/local-validation-archive/files/issue-257-validation/build.sh", + "bytes": 778, + "sha256": "9d84b71a90b08fcace9f59f5801df54d0901f3401d693db778116e87d0958c82", + "git_blob_sha1": "c65da40312a03a07963e15033084c3b1baae10a4" + }, + { + "original_path": "issue-257-validation/oracle-check.cpp", + "archived_path": "doc/local-validation-archive/files/issue-257-validation/oracle-check.cpp", + "bytes": 363, + "sha256": "802aa2bde769005921feffbb692dcf7fb2b886461493c9bc0877a7897d257576", + "git_blob_sha1": "e2189ae81c23bdd0212b240f8a3b2a3ab5b8d813" + }, + { + "original_path": "issue-257-validation/debug/doc/analysis/seeded-probabilities-validation.md", + "archived_path": "doc/local-validation-archive/files/issue-257-validation/debug/doc/analysis/seeded-probabilities-validation.md", + "bytes": 4323, + "sha256": "4b8b5113cc8e1f32d65f0a0978c6ee73e99d70e45328fb8955a8409a87b9ae1f", + "git_blob_sha1": "cf5f4ef097b7876846f4cefa641df521b4000a5b" + }, + { + "original_path": "issue-267-validation/header-probe.cpp", + "archived_path": "doc/local-validation-archive/files/issue-267-validation/header-probe.cpp", + "bytes": 61, + "sha256": "f1f68a746f2f97d3625ee6203a615f640dd6d0cd27909f64a05af3fef159e07d", + "git_blob_sha1": "2b45fa6d675bdd7443cd8705b5406a1df2147649" + }, + { + "original_path": "issue-267-validation/check-installed.sh", + "archived_path": "doc/local-validation-archive/files/issue-267-validation/check-installed.sh", + "bytes": 1736, + "sha256": "414c34966907304dfcb4ea3cb59c17a843c1a0ab29dea4fc34ed70082ce44349", + "git_blob_sha1": "c8c8a78c60e5ccfb2ce62c2d6f19c0b3f059f247" + }, + { + "original_path": "issue-267-validation/run-configure-probes.py", + "archived_path": "doc/local-validation-archive/files/issue-267-validation/run-configure-probes.py", + "bytes": 4441, + "sha256": "dabd53afeac31c98ee41774d4a2df2a900fb54d1a17474456fbed9a5fb8d58e9", + "git_blob_sha1": "7a97e25b15d1c0259bfd78f2db036d77eea92969" + }, + { + "original_path": "issue-267-validation/run-minimum.sh", + "archived_path": "doc/local-validation-archive/files/issue-267-validation/run-minimum.sh", + "bytes": 2150, + "sha256": "e66b08258aa9aeea2286d5b1455fe3256b83f538d29633e1cda3a93bbaee51b7", + "git_blob_sha1": "c52d51f520b138a2182a659b65243bda6806d906" + }, + { + "original_path": "issue-267-validation/build-vrna-2.7.0.sh", + "archived_path": "doc/local-validation-archive/files/issue-267-validation/build-vrna-2.7.0.sh", + "bytes": 1489, + "sha256": "8ba277dfcb8f7d9b22b092296a5f8629fb9ac2b01f4cf28c6aa40e00f0464801", + "git_blob_sha1": "5b99df7c91024ac309679c758287f139eb3d802a" + }, + { + "original_path": "issue-267-validation/installed-consumer.cpp", + "archived_path": "doc/local-validation-archive/files/issue-267-validation/installed-consumer.cpp", + "bytes": 382, + "sha256": "552c0e6e59466efbe3c58626876306e32f6cd06ce7ef0d3d209d644042a3a8c5", + "git_blob_sha1": "dbf5ddd721567eb113444d22ed902471372559e1" + }, + { + "original_path": "issue-267-validation/run-debug.sh", + "archived_path": "doc/local-validation-archive/files/issue-267-validation/run-debug.sh", + "bytes": 908, + "sha256": "d7dcb65b9746aef77ee983d139e06f6bb59e2f64cb204575c3ac5579734947d2", + "git_blob_sha1": "cd00f7444da6e9d24a8994c492d311cceb8a2f12" + }, + { + "original_path": "issue-267-validation/probe-source/doc/analysis/vienna-accessibility-thread-safety.md", + "archived_path": "doc/local-validation-archive/files/issue-267-validation/probe-source/doc/analysis/vienna-accessibility-thread-safety.md", + "bytes": 21861, + "sha256": "657ea649f1b6fc2e6defdb7b816d4651e677cfb77da7f84716d835dcbf1ec7ec", + "git_blob_sha1": "ad12a3dd00e117c5a8594e8d81168b284ed2e9c3" + }, + { + "original_path": "issue-267-validation/configure-fixtures/missing-logger-symbol/include/ViennaRNA/utils/log.h", + "archived_path": "doc/local-validation-archive/files/issue-267-validation/configure-fixtures/missing-logger-symbol/include/ViennaRNA/utils/log.h", + "bytes": 172, + "sha256": "b3e09508eee4a00b5b54415fa48c03c52faf48b1bd983cea6f2f541920a18e47", + "git_blob_sha1": "ca0897baaa8ae1717e35c33e4a28c96e0a897330" + }, + { + "original_path": "issue-267-validation/configure-fixtures/missing-logger-declaration/include/ViennaRNA/utils/log.h", + "archived_path": "doc/local-validation-archive/files/issue-267-validation/configure-fixtures/missing-logger-declaration/include/ViennaRNA/utils/log.h", + "bytes": 66, + "sha256": "8877a9e91a28449887fabf830a3435ee19f884da20ef64c829453dca15369835", + "git_blob_sha1": "365330125f4436b49471b90e4babcc31c94c7a7e" + }, + { + "original_path": "issue-267-validation/configure-fixtures/old-headers/include/ViennaRNA/vrna_config.h", + "archived_path": "doc/local-validation-archive/files/issue-267-validation/configure-fixtures/old-headers/include/ViennaRNA/vrna_config.h", + "bytes": 1814, + "sha256": "2550dc33fc30a6304513e3a0e91f17905b3c348c59028f6f397c983acf02b3e1", + "git_blob_sha1": "ee61111f1fdb6a18b5ac4118dc3acb5f28b59e8c" + }, + { + "original_path": "issue-267-validation/minimum/doc/analysis/vienna-accessibility-thread-safety.md", + "archived_path": "doc/local-validation-archive/files/issue-267-validation/minimum/doc/analysis/vienna-accessibility-thread-safety.md", + "bytes": 21861, + "sha256": "657ea649f1b6fc2e6defdb7b816d4651e677cfb77da7f84716d835dcbf1ec7ec", + "git_blob_sha1": "ad12a3dd00e117c5a8594e8d81168b284ed2e9c3" + }, + { + "original_path": "issue-267-validation/debug/doc/analysis/vienna-accessibility-thread-safety.md", + "archived_path": "doc/local-validation-archive/files/issue-267-validation/debug/doc/analysis/vienna-accessibility-thread-safety.md", + "bytes": 21918, + "sha256": "7c858a51f085fbf080bd65b5a893e1d89b2793512bd349f351725714c69fb767", + "git_blob_sha1": "3fdb218cd1800fa58c77754b85c796290dbd9902" + }, + { + "original_path": "issue-260-validation/installed-header.cpp", + "archived_path": "doc/local-validation-archive/files/issue-260-validation/installed-header.cpp", + "bytes": 881, + "sha256": "af5590919fd50bcaa7ee4ca82526a0ac854aa3fbaea038a164d99417d1242dc1", + "git_blob_sha1": "2237521d74bace8757b64901e6141988b2ef4660" + }, + { + "original_path": "issue-260-validation/shared.patch", + "archived_path": "doc/local-validation-archive/files/issue-260-validation/shared.patch", + "bytes": 4287, + "sha256": "732a7ad70b8e987764568bf660aa3f8105a7d7ea493322209cc375c44814c76d", + "git_blob_sha1": "d8e84319d56c920776867f188e8eac5717c97539" + }, + { + "original_path": "issue-260-validation/debug-master/src/IntaRNA/PredictorMfeEns.h", + "archived_path": "doc/local-validation-archive/files/issue-260-validation/debug-master/src/IntaRNA/PredictorMfeEns.h", + "bytes": 5736, + "sha256": "1c7980ad8890bafe1e6c2d3f198374061451d9c548be5ceee7c1ce80f0ff8ffb", + "git_blob_sha1": "4f4a8ddfd521817a6ea5f36aad88fd59ad58a249" + }, + { + "original_path": "profiling-default-20260921/scripts/build.sh", + "archived_path": "doc/local-validation-archive/files/profiling-default-20260921/scripts/build.sh", + "bytes": 1274, + "sha256": "43b58b9bcbdf73cd12141923ece64a4e5d088d30aae15bece2d843b66120b148", + "git_blob_sha1": "f409d785a4fa900ac3872a86b5f1c2a73aa4cb79" + }, + { + "original_path": "profiling-default-20260921/scripts/benchmark.py", + "archived_path": "doc/local-validation-archive/files/profiling-default-20260921/scripts/benchmark.py", + "bytes": 10689, + "sha256": "7751a440a57f22b9d13b2a93cdac622132dc08d02c0a1ee7aee2bdaf541ac23f", + "git_blob_sha1": "f09517e9783910dfc7a30ba5ba7419e757642fc9" + }, + { + "original_path": "profiling-default-20260921/scripts/ed_conversion_probe.sh", + "archived_path": "doc/local-validation-archive/files/profiling-default-20260921/scripts/ed_conversion_probe.sh", + "bytes": 567, + "sha256": "0f0438c0b3cefc615d0616818f40e5dc88877a8a2a361da571b1de5118c21df6", + "git_blob_sha1": "9389e7c591d08e6e84adc7005845c9616f54fcf9" + }, + { + "original_path": "profiling-default-20260921/scripts/plots.py", + "archived_path": "doc/local-validation-archive/files/profiling-default-20260921/scripts/plots.py", + "bytes": 3603, + "sha256": "7635a00a17a5a8da89ea80e8c1b95085b3fa80e638c028357a5cdefdfee9baaa", + "git_blob_sha1": "d24ba3230f9000e9ea993093897dfe303d36d818" + }, + { + "original_path": "profiling-default-20260921/scripts/profile.py", + "archived_path": "doc/local-validation-archive/files/profiling-default-20260921/scripts/profile.py", + "bytes": 4247, + "sha256": "a081cf9ca9a72a5b96799fa715eb596e993ba1992e857b051a572d0d64201e4c", + "git_blob_sha1": "bd329eaadabe2564c71ab45346710995b7910fec" + }, + { + "original_path": "profiling-default-20260921/scripts/report.py", + "archived_path": "doc/local-validation-archive/files/profiling-default-20260921/scripts/report.py", + "bytes": 21449, + "sha256": "a1a84911ede751a890f96ef25302f0a60bf5dc98f410616fb9327b2b52e4d9cf", + "git_blob_sha1": "74cc28b48fe84ff0450023279ce72f03b43de8a4" + }, + { + "original_path": "profiling-default-20260921/scripts/ed_numeric.py", + "archived_path": "doc/local-validation-archive/files/profiling-default-20260921/scripts/ed_numeric.py", + "bytes": 1259, + "sha256": "eb55347f0123f3853ad8af48f096ee3acc73fea67cbaef4d3c7c9904f17419b1", + "git_blob_sha1": "b7e3ffb132b48778128fbca54f03996195866002" + }, + { + "original_path": "profiling-default-20260921/scripts/parallel_check.py", + "archived_path": "doc/local-validation-archive/files/profiling-default-20260921/scripts/parallel_check.py", + "bytes": 2369, + "sha256": "e0077fdcca9a4550a118a60ad1cd622e351102eea29c09b6a03909dc5ee36728", + "git_blob_sha1": "f87587b935f32a00afa40c3c60ba96583fc1df0b" + }, + { + "original_path": "profiling-default-20260921/scripts/package.py", + "archived_path": "doc/local-validation-archive/files/profiling-default-20260921/scripts/package.py", + "bytes": 1759, + "sha256": "92293ce61f7c3c95cdcb15c76dc0ea393c86121ce6ed35f64c8cb2e7feca4870", + "git_blob_sha1": "8ca338e0aec5e0234e01d81dee70e50b38913cfc" + }, + { + "original_path": "profiling-default-20260921/scripts/diagnostics.py", + "archived_path": "doc/local-validation-archive/files/profiling-default-20260921/scripts/diagnostics.py", + "bytes": 4288, + "sha256": "391061daa680aac234c3e76d59a5e3f895e5b9454464b313b78540be0d5c28f0", + "git_blob_sha1": "14c747d2ec8d9d1f69d27d580890bb0d07c9f63e" + }, + { + "original_path": "profiling-default-20260921/scripts/ed_conversion_probe.cpp", + "archived_path": "doc/local-validation-archive/files/profiling-default-20260921/scripts/ed_conversion_probe.cpp", + "bytes": 480, + "sha256": "46cdc4961e033b306a60aca30685012486b9d2bd5670ec2af5e447da67d15c82", + "git_blob_sha1": "cc932a4c6553db4df0b10e0829802451c3dd220c" + }, + { + "original_path": "profiling-default-20260921/scripts/summarize_profiles.py", + "archived_path": "doc/local-validation-archive/files/profiling-default-20260921/scripts/summarize_profiles.py", + "bytes": 2098, + "sha256": "252f27c6822ac1e1a32544cce4f8141c6a76f7cd87ad3232a4b646b99ae59f6e", + "git_blob_sha1": "db11ddf1f2410863421b6e914e30a6207cfbfccc" + }, + { + "original_path": "profiling-default-20260921/scripts/audit.py", + "archived_path": "doc/local-validation-archive/files/profiling-default-20260921/scripts/audit.py", + "bytes": 11323, + "sha256": "217d26d65c776f131dd55b580ccaad9e002a6207b094f6b423f188dda98c9389", + "git_blob_sha1": "d753c2cd75a6fdc186c8d56b9828cb9a4cd2293a" + }, + { + "original_path": "profiling-default-20260921/scripts/phase_summary.py", + "archived_path": "doc/local-validation-archive/files/profiling-default-20260921/scripts/phase_summary.py", + "bytes": 1397, + "sha256": "fc2cb29c50ed583535bc91afa81ca793693291d60e7af5165cdfc46201109575", + "git_blob_sha1": "c568dd75bd288b45bb00103687b3ac5600e7b51d" + }, + { + "original_path": "issue-212-validation/explore-ensemble.py", + "archived_path": "doc/local-validation-archive/files/issue-212-validation/explore-ensemble.py", + "bytes": 1235, + "sha256": "271c04b55c7d565c2589ea20c2bb017a5fa355dddccb0d4f0a19416576096276", + "git_blob_sha1": "c4ed25d247009b8bbff99f1454a2fb7bb3d10a46" + }, + { + "original_path": "issue-212-validation/build.sh", + "archived_path": "doc/local-validation-archive/files/issue-212-validation/build.sh", + "bytes": 945, + "sha256": "3c7a0ef549d2ccaefbceb7c7fa97bdb8f7c2104ab934ed660fb67a8be58c50de", + "git_blob_sha1": "202984acf697a98ae24185b332547bf02f0e1422" + }, + { + "original_path": "issue-212-validation/review-debug-build.sh", + "archived_path": "doc/local-validation-archive/files/issue-212-validation/review-debug-build.sh", + "bytes": 1116, + "sha256": "c0446410361fa04b893022d6b3fd851ca21ec3a2b275c10a5bb6234790984b84", + "git_blob_sha1": "163dc110c2e8b2418ee73c0fd2182869ba7de948" + }, + { + "original_path": "issue-212-validation/sentinel.cpp", + "archived_path": "doc/local-validation-archive/files/issue-212-validation/sentinel.cpp", + "bytes": 769, + "sha256": "27abe8d687489a426d3dd16ac8e84842013061469c5239c84429ed0bd58745f6", + "git_blob_sha1": "5c17984b1057e93b6b22fe02878edbbfce89b37b" + }, + { + "original_path": "issue-212-validation/check-small.py", + "archived_path": "doc/local-validation-archive/files/issue-212-validation/check-small.py", + "bytes": 642, + "sha256": "b73f3f33c93c23e5806210273e3ceed95ba3753e3743bb7e44a57aff4c640741", + "git_blob_sha1": "86df3830188b3cc821112e49d1c7ab14495123ca" + }, + { + "original_path": "issue-212-validation/explore-filters.py", + "archived_path": "doc/local-validation-archive/files/issue-212-validation/explore-filters.py", + "bytes": 1136, + "sha256": "6786a72b18c488203d70a947ece7d9435ae3c1cfffb9d0d35c9dba81b3a26427", + "git_blob_sha1": "f52eed93f26fc8b8d98c54b21cfb3c3caeb52fa5" + }, + { + "original_path": "issue-212-validation/explore.py", + "archived_path": "doc/local-validation-archive/files/issue-212-validation/explore.py", + "bytes": 1549, + "sha256": "1b52321ae7b972d1703ab741cb3321708d98679ecb44151664716e75be89b594", + "git_blob_sha1": "a5f4b0aa2925f6455f4a1a208d790426ea854a33" + }, + { + "original_path": "issue-212-validation/explore-decomposition.py", + "archived_path": "doc/local-validation-archive/files/issue-212-validation/explore-decomposition.py", + "bytes": 750, + "sha256": "79c91b4556d8a404bfb6b3688c7ee771cb0ad11383626f9bd024e7c831c68cba", + "git_blob_sha1": "79051b1a4f0c668d5d9dd19dca63f40098949d44" + }, + { + "original_path": "issue-212-validation/review-debug/tests/PredictorMfeHeuristicCellState_test.cpp", + "archived_path": "doc/local-validation-archive/files/issue-212-validation/review-debug/tests/PredictorMfeHeuristicCellState_test.cpp", + "bytes": 12544, + "sha256": "36b6d9d323aeffdcddb9e6b5dbff6cfcbf1492de80edf0c4ee6fd4486c046750", + "git_blob_sha1": "fe0235aad93232af9b78efb4704f80abc840892d" + }, + { + "original_path": "issue-212-validation/distribution-doc/analysis/out-overlap.md", + "archived_path": "doc/local-validation-archive/files/issue-212-validation/distribution-doc/analysis/out-overlap.md", + "bytes": 20413, + "sha256": "dd2e1b6c01f9f3a0c36a71f9cdb08d74b0c9cb5cb0873e6f9b5624bd5f722099", + "git_blob_sha1": "1887c21eb106f689f95f4f63d203614e25707ced" + }, + { + "original_path": "issue-270-validation/sync-candidate.py", + "archived_path": "doc/local-validation-archive/files/issue-270-validation/sync-candidate.py", + "bytes": 654, + "sha256": "a71fd2cccc6c27e3cfe993e8ccce3efe244bb904980e65f1a593fe4580313f40", + "git_blob_sha1": "33031fa5bfb0fee591391c23b45f8352204b1170" + }, + { + "original_path": "issue-270-validation/compare-native.py", + "archived_path": "doc/local-validation-archive/files/issue-270-validation/compare-native.py", + "bytes": 1515, + "sha256": "8517361ec8fb3494673db8d85c7226f4066e1cf60f15f18dc5b102c283348190", + "git_blob_sha1": "2de79dbf613138fc47a4168b58112e91c6222549" + }, + { + "original_path": "issue-270-validation/kernel/candidate/IntaRNA/SeededPartitionFunction.cpp", + "archived_path": "doc/local-validation-archive/files/issue-270-validation/kernel/candidate/IntaRNA/SeededPartitionFunction.cpp", + "bytes": 27994, + "sha256": "41d3370da91be6f0a172807b7c413975c2d7a0ff57ea597031cba95e776990db", + "git_blob_sha1": "0a0eeb6acd93cb8db5d94f61729f9ea3476f652d" + }, + { + "original_path": "issue-270-validation/profile/src/IntaRNA/SeededPartitionFunction.cpp", + "archived_path": "doc/local-validation-archive/files/issue-270-validation/profile/src/IntaRNA/SeededPartitionFunction.cpp", + "bytes": 28462, + "sha256": "c9120ff740297d845544722a2093dd640b0697901b45ff14f20c8efbb5e59194", + "git_blob_sha1": "60be84eeb5bfe24caf09a735ab44879677c04490" + }, + { + "original_path": "issue-270-validation/profile/src/IntaRNA/PredictorMfeEns2dSeedExtension.cpp", + "archived_path": "doc/local-validation-archive/files/issue-270-validation/profile/src/IntaRNA/PredictorMfeEns2dSeedExtension.cpp", + "bytes": 30326, + "sha256": "1599a9ee4415e4416ffa355725ec16cf09679f4b2944e9dba7e390c652f7443f", + "git_blob_sha1": "beb963778400009038601d8bcd0ee78e8985b1eb" + }, + { + "original_path": "issue-270-validation/profile-analysis/run-profile.py", + "archived_path": "doc/local-validation-archive/files/issue-270-validation/profile-analysis/run-profile.py", + "bytes": 1903, + "sha256": "fc64c9b0ee33f4606422181b2e04f03b14d267fba58242cd7e01c2f31e6bda9b", + "git_blob_sha1": "6c3dc70b76e1924b1e782c25cec075fb83b927db" + }, + { + "original_path": "issue-270-validation/profile-analysis/instrumentation.patch", + "archived_path": "doc/local-validation-archive/files/issue-270-validation/profile-analysis/instrumentation.patch", + "bytes": 6203, + "sha256": "f7f2a4c13d606a03c63bf6bb259834b88fa92e7bea4e0c30ab712d4a471a1525", + "git_blob_sha1": "cd5df5d905630d9c9b5db5fe24602f904fac7aad" + }, + { + "original_path": "issue-270-validation/profile-analysis/first-probe/run-profile.py", + "archived_path": "doc/local-validation-archive/files/issue-270-validation/profile-analysis/first-probe/run-profile.py", + "bytes": 1837, + "sha256": "f7c2bc62d7663b8d371209b744d565a778d06c9afd19fda1aa8833bf13726618", + "git_blob_sha1": "e31364d52671f1bff647208ad5232461612dbafc" + }, + { + "original_path": "issue-270-validation/profile-analysis/first-probe/instrument.py", + "archived_path": "doc/local-validation-archive/files/issue-270-validation/profile-analysis/first-probe/instrument.py", + "bytes": 5204, + "sha256": "1acced9ec28106055dfc576039b22a8d152e28388c7ea0e7c5788393778efdd2", + "git_blob_sha1": "7bbd9522e979a7bd5cb5d9811398a6b94537cfc9" + }, + { + "original_path": "issue-270-validation/profile-analysis/first-probe/instrumentation.patch", + "archived_path": "doc/local-validation-archive/files/issue-270-validation/profile-analysis/first-probe/instrumentation.patch", + "bytes": 5030, + "sha256": "550c762d0689d10a30fcb9c021ab95a851b1d421d3d0760919ff672a53f06103", + "git_blob_sha1": "7b4412f38341ac531d4a6022ee4b5d20cd34bd44" + }, + { + "original_path": "issue-258-review-validation/header-probe.cpp", + "archived_path": "doc/local-validation-archive/files/issue-258-review-validation/header-probe.cpp", + "bytes": 59, + "sha256": "909c144f54b7408095c16e2e6babb530a17bc27eeea1fc0fef34c5b7806cfde5", + "git_blob_sha1": "28acc3452eca7c2ba804bd192b72035f00211680" + }, + { + "original_path": "issue-258-review-validation/check-invariant.cpp", + "archived_path": "doc/local-validation-archive/files/issue-258-review-validation/check-invariant.cpp", + "bytes": 1336, + "sha256": "b5593ebabaf6b480194a91e11604b1572694a05590bed06b9b329e45880f5e0c", + "git_blob_sha1": "8aafd0ed5aa701e26a4e4aead076d411eea531f3" + }, + { + "original_path": "issue-258-review-validation/validate-installed.py", + "archived_path": "doc/local-validation-archive/files/issue-258-review-validation/validate-installed.py", + "bytes": 2572, + "sha256": "a0d91fcac3fb194d6079faab24b29ea335a2e73e4fd1d926961c4b6bae74b8c6", + "git_blob_sha1": "9fb9a24ce8358e6000821ece71688148d83c62c2" + }, + { + "original_path": "issue-258-validation/header-probe.cpp", + "archived_path": "doc/local-validation-archive/files/issue-258-validation/header-probe.cpp", + "bytes": 61, + "sha256": "d6eab99d044f1893a6dd4920d51053d564e9594da148fa039094a19948c62b62", + "git_blob_sha1": "d87241cba8e749a62d848e05ef946a93c97c03c7" + }, + { + "original_path": "issue-258-validation/build.sh", + "archived_path": "doc/local-validation-archive/files/issue-258-validation/build.sh", + "bytes": 778, + "sha256": "fb755b3c8ea2621ea279678dd3cd632166818bcd457b9f14190df05ce1f47ec5", + "git_blob_sha1": "984b5cd724e6b1b174b816a15e8223db3af2fafd" + }, + { + "original_path": "issue-258-validation/BasePairProbabilityWriter-header.cpp", + "archived_path": "doc/local-validation-archive/files/issue-258-validation/BasePairProbabilityWriter-header.cpp", + "bytes": 61, + "sha256": "d6eab99d044f1893a6dd4920d51053d564e9594da148fa039094a19948c62b62", + "git_blob_sha1": "d87241cba8e749a62d848e05ef946a93c97c03c7" + }, + { + "original_path": "issue-258-validation/installed-svg-example.cpp", + "archived_path": "doc/local-validation-archive/files/issue-258-validation/installed-svg-example.cpp", + "bytes": 1349, + "sha256": "d1728ba369ebba751ff4c8ea6a21be227588421ed4a5c58ab94e94623e93dfea", + "git_blob_sha1": "5c143f5c4c9d90aea3f93fb70cbcdba8e435eb66" + }, + { + "original_path": "issue-258-validation/BasePairProbabilities-header.cpp", + "archived_path": "doc/local-validation-archive/files/issue-258-validation/BasePairProbabilities-header.cpp", + "bytes": 57, + "sha256": "c08a17d79dd1789cea975223193f9f1899696d957348db9936847d1369120fab", + "git_blob_sha1": "b3ecedee0b42d308766eb277610b1d4616fc3c56" + }, + { + "original_path": "issue-258-validation/audit-performance/run.py", + "archived_path": "doc/local-validation-archive/files/issue-258-validation/audit-performance/run.py", + "bytes": 5397, + "sha256": "0dea17815948f9258f6048cbc5e38983d0e7a4baafa0dc334f54ee18fa9c4843", + "git_blob_sha1": "01bb042c470873928edc540b74091dc6fd4a0f45" + }, + { + "original_path": "issue-258-validation/audit-performance/control-fixed.py", + "archived_path": "doc/local-validation-archive/files/issue-258-validation/audit-performance/control-fixed.py", + "bytes": 2079, + "sha256": "9e9b90b0587388c460772da4fffb844d1803e11acc8e949848d3310a8a2ed75d", + "git_blob_sha1": "4a759fc76c4c88f60f51faf598d4e76d58e36a67" + }, + { + "original_path": "issue-258-validation/audit-performance/serializer-control.py", + "archived_path": "doc/local-validation-archive/files/issue-258-validation/audit-performance/serializer-control.py", + "bytes": 1918, + "sha256": "502cec33c9e431aad0392d13c5d61cf9376c2cccd76817ed10f65547612b028d", + "git_blob_sha1": "fa3d89aa37231fa021788ad66c076fe4fe57b50a" + }, + { + "original_path": "issue-258-validation/audit-performance/compare-fixed.py", + "archived_path": "doc/local-validation-archive/files/issue-258-validation/audit-performance/compare-fixed.py", + "bytes": 3519, + "sha256": "4e38f8fbc0f8743bcc4fddd8ba672666bb44e183571c500cdb8cff20fc46eff3", + "git_blob_sha1": "cadf06221d1d83d87b718fec254f4f0b73c16eae" + }, + { + "original_path": "issue-258-validation/audit-performance/compare-fixed-long.py", + "archived_path": "doc/local-validation-archive/files/issue-258-validation/audit-performance/compare-fixed-long.py", + "bytes": 3500, + "sha256": "786c87340bf2e4f73350de2274c947eef13046abd26fdac6e6ed5c68c862969b", + "git_blob_sha1": "3f0b91eb3ff545e6c90f921319d727eb467cb433" + }, + { + "original_path": "issue-258-validation/audit-performance/summarize-fixed.py", + "archived_path": "doc/local-validation-archive/files/issue-258-validation/audit-performance/summarize-fixed.py", + "bytes": 3838, + "sha256": "1ce0170cccfb383417b2f460bd11ab74f8695130f2ba80e50c67b59c410388ee", + "git_blob_sha1": "786024dcbdc1042dd787acc603688a609d2bbcb6" + }, + { + "original_path": "issue-258-validation/review-diagrams/build_probability_recursions.py", + "archived_path": "doc/local-validation-archive/files/issue-258-validation/review-diagrams/build_probability_recursions.py", + "bytes": 8092, + "sha256": "29e5bb8f7ccb550f7c250e4f2b79ec54b86d854bd4d361913ed7e04c865e5f85", + "git_blob_sha1": "5a4d0dffe84a4010563032bc868275df7b77d240" + }, + { + "original_path": "issue-258-validation/step9/build_recursions.py", + "archived_path": "doc/local-validation-archive/files/issue-258-validation/step9/build_recursions.py", + "bytes": 25434, + "sha256": "a2c90388c528bc2b4a23a69ac693f576d51948e4a4deb4cefe1fbed503a922e7", + "git_blob_sha1": "8b312dfc3b6d759001b740e0d306aceb9127e220" + }, + { + "original_path": "issue-270-anchored-validation/header-probe.cpp", + "archived_path": "doc/local-validation-archive/files/issue-270-anchored-validation/header-probe.cpp", + "bytes": 86, + "sha256": "d25809e8518b026422695fd22cc56b462c697e428f2397e9a523d351204976ff", + "git_blob_sha1": "e9ddbedb0f8c34ad3884e429b714e1c1dcefa43e" + }, + { + "original_path": "issue-270-anchored-validation/check-install.py", + "archived_path": "doc/local-validation-archive/files/issue-270-anchored-validation/check-install.py", + "bytes": 1419, + "sha256": "76eec50b15002d4a5ee9fc10ec8aceea53ee89736ca2b813e88a7eba9abe6256", + "git_blob_sha1": "24cd8b32bb477cd91d411d122ccfed3facdacc6a" + }, + { + "original_path": "issue-270-anchored-validation/compare-native.py", + "archived_path": "doc/local-validation-archive/files/issue-270-anchored-validation/compare-native.py", + "bytes": 2548, + "sha256": "b707fd1d9cc7d8cc28e8a295e2cbe5ee33a25a008dfea7c16a013f661d648fd9", + "git_blob_sha1": "1340202fbce884774d0371849500050194e4b224" + }, + { + "original_path": "issue-270-anchored-validation/record-evidence.py", + "archived_path": "doc/local-validation-archive/files/issue-270-anchored-validation/record-evidence.py", + "bytes": 2843, + "sha256": "4fd39ba3c1459e5369bbd74b34db43f17092521ed26f506bead1e6ca570f45fa", + "git_blob_sha1": "fba0e8ec8d397fad4ba9a1a68812467b828ba1f5" + }, + { + "original_path": "issue-184-validation/build.sh", + "archived_path": "doc/local-validation-archive/files/issue-184-validation/build.sh", + "bytes": 1016, + "sha256": "446048d1dc57c1c3b9b84e12cc386d62ee5f9d9aa1b8ab4c74d875c18c365f27", + "git_blob_sha1": "ad19d90279a0de6d69cae9c62eeef444a80e3cd2" + }, + { + "original_path": "issue-184-validation/header.cpp", + "archived_path": "doc/local-validation-archive/files/issue-184-validation/header.cpp", + "bytes": 61, + "sha256": "f1f68a746f2f97d3625ee6203a615f640dd6d0cd27909f64a05af3fef159e07d", + "git_blob_sha1": "2b45fa6d675bdd7443cd8705b5406a1df2147649" + }, + { + "original_path": "issue-184-validation/consumer.cpp", + "archived_path": "doc/local-validation-archive/files/issue-184-validation/consumer.cpp", + "bytes": 724, + "sha256": "7f6b7c04d84ed3a8e7696143399ee0ab2234c98e7c32734d818ab5c8b4afb56b", + "git_blob_sha1": "5defb6138c72ac80ba79abe83c1d0a559bd6595b" + }, + { + "original_path": "issue-184-validation/check-install.sh", + "archived_path": "doc/local-validation-archive/files/issue-184-validation/check-install.sh", + "bytes": 1932, + "sha256": "166a09446967baa931612a212868a763975769421c494dfaed73d1d73462b0eb", + "git_blob_sha1": "4213f8fda1820eb557b42e0880ad32ec67e35260" + }, + { + "original_path": "issue-244-validation/source-gcc11-debug/configure.ac", + "archived_path": "doc/local-validation-archive/files/issue-244-validation/source-gcc11-debug/configure.ac", + "bytes": 17223, + "sha256": "0745bbf666828fddb48f0f57c2efd46439047c04cc42c935a0ff7c19183691cf", + "git_blob_sha1": "4f371a56b2435102bab6556969da75e4112cb609" + }, + { + "original_path": "issue-244-validation/source-gcc11-debug/.github/workflows/build.yml", + "archived_path": "doc/local-validation-archive/files/issue-244-validation/source-gcc11-debug/.github/workflows/build.yml", + "bytes": 5062, + "sha256": "f6f9fce0c17dbe86261a4fec42a708f0b436b6e82af0c75747cc74118e0d46e7", + "git_blob_sha1": "ee4a00592f7292a222f4695e58f82e0195367fa5" + }, + { + "original_path": "issue-246-validation/audit-distribution/intaRNA-3.4.1/doc/Makefile.am", + "archived_path": "doc/local-validation-archive/files/issue-246-validation/audit-distribution/intaRNA-3.4.1/doc/Makefile.am", + "bytes": 520, + "sha256": "9d70e4612646ea0d394da900aec36cc9809545fd9efc8cfaf0921343939eada2", + "git_blob_sha1": "0e139915c21528e8adcaa216e52cce5a222c45cb" + }, + { + "original_path": "issue-246-validation/mdspan-probe-source/configure.ac", + "archived_path": "doc/local-validation-archive/files/issue-246-validation/mdspan-probe-source/configure.ac", + "bytes": 19972, + "sha256": "9f0f054b36ceaacdad91d9158d33f9a7ae9b2441a4f7bb98f8d5d4afd1460977", + "git_blob_sha1": "ba431ed45ddc311af9e070c54ac3661ff24d1a6a" + }, + { + "original_path": "issue-264-validation/debug/doc/Makefile.am", + "archived_path": "doc/local-validation-archive/files/issue-264-validation/debug/doc/Makefile.am", + "bytes": 796, + "sha256": "49312594272e284b5a5a6d2451ee35d3ad0e04f5c59be4d7a0c6ec69695c04f9", + "git_blob_sha1": "101c22b521305cc76200a0f7c5e3b040d1bd1a00" + }, + { + "original_path": "issue-212-validation/distribution-doc/Makefile.am", + "archived_path": "doc/local-validation-archive/files/issue-212-validation/distribution-doc/Makefile.am", + "bytes": 485, + "sha256": "1b976a9e9ae60af6109439ce4d9ea1eb5c7ee24670df462964fb757a0f7ab5af", + "git_blob_sha1": "acfa3bd779bf41ac7e925d8e81e8831fa7e032d9" + } + ], + "other_coverage": [ + { + "path": "MIGRATION-AND-CHANGELOG.md", + "reason": "delivery/operational record retained in the external recovery archive" + }, + { + "path": "issue-259-revision-validation/oct12-pr-body.md", + "reason": "delivery/operational record retained in the external recovery archive" + }, + { + "path": "issue-259-revision-validation/revised-pr-body.md", + "reason": "delivery/operational record retained in the external recovery archive" + }, + { + "path": "issue-273-validation/pr-body.md", + "reason": "delivery/operational record retained in the external recovery archive" + }, + { + "path": "issue-262-validation/pr-body.md", + "reason": "delivery/operational record retained in the external recovery archive" + }, + { + "path": "issue-244-validation/current-cxx23-library-probe.cpp", + "reason": "content already reachable from a published Git ref" + }, + { + "path": "issue-244-validation/set-aside-cxx17.patch", + "reason": "content already reachable from a published Git ref" + }, + { + "path": "issue-244-validation/set-aside-cxx17-notes.md", + "reason": "content already reachable from a published Git ref" + }, + { + "path": "issue-244-validation/compiler-probe.cpp", + "reason": "content already reachable from a published Git ref" + }, + { + "path": "profiling/martin/email-to-Martin.md", + "reason": "delivery/operational record retained in the external recovery archive" + }, + { + "path": "profiling/martin/pull-request-body.md", + "reason": "delivery/operational record retained in the external recovery archive" + }, + { + "path": "issue-266-validation/pr-body.md", + "reason": "delivery/operational record retained in the external recovery archive" + }, + { + "path": "issue-245-validation/pr-body.md", + "reason": "delivery/operational record retained in the external recovery archive" + }, + { + "path": "issue-246-validation/pr-body.md", + "reason": "delivery/operational record retained in the external recovery archive" + }, + { + "path": "issue-246-validation/pr-body-current.md", + "reason": "delivery/operational record retained in the external recovery archive" + }, + { + "path": "kinetic-validation/pr-body.md", + "reason": "delivery/operational record retained in the external recovery archive" + }, + { + "path": "kinetic-validation/pr-body-kix.md", + "reason": "delivery/operational record retained in the external recovery archive" + }, + { + "path": "kinetic-validation/pr-body-review.md", + "reason": "delivery/operational record retained in the external recovery archive" + }, + { + "path": "issue-274-validation/pr-body.md", + "reason": "delivery/operational record retained in the external recovery archive" + }, + { + "path": "issue-264-validation/pr-body.md", + "reason": "delivery/operational record retained in the external recovery archive" + }, + { + "path": "issue-265-validation/pr-body.md", + "reason": "delivery/operational record retained in the external recovery archive" + }, + { + "path": "issue-268-validation/pr-body.md", + "reason": "delivery/operational record retained in the external recovery archive" + }, + { + "path": "issue-257-validation/pr-body-implementation.md", + "reason": "delivery/operational record retained in the external recovery archive" + }, + { + "path": "issue-267-validation/pr-body.md", + "reason": "delivery/operational record retained in the external recovery archive" + }, + { + "path": "issue-260-validation/pr-body.md", + "reason": "delivery/operational record retained in the external recovery archive" + }, + { + "path": "issue-260-validation/feat-fixed/src/IntaRNA/PredictorMfeEns2dHeuristicSeedExtension.cpp", + "reason": "promoted in the separate probability-memory port PR" + }, + { + "path": "issue-260-validation/feat-fixed/src/IntaRNA/PredictorMfeEns.h", + "reason": "promoted in the separate probability-memory port PR" + }, + { + "path": "issue-260-validation/feat-fixed/src/IntaRNA/PredictorMfeEns.cpp", + "reason": "promoted in the separate probability-memory port PR" + }, + { + "path": "issue-260-validation/feat-fixed/src/IntaRNA/PredictorMfeEns2dSeedExtension.cpp", + "reason": "promoted in the separate probability-memory port PR" + }, + { + "path": "issue-260-validation/feat-fixed/src/IntaRNA/PredictorMfeEns2dSeedExtension.h", + "reason": "promoted in the separate probability-memory port PR" + }, + { + "path": "issue-260-validation/feat-fixed/tests/PredictorMfeEnsRegression_test.cpp", + "reason": "promoted in the separate probability-memory port PR" + }, + { + "path": "profiling-default-20260921/delivery/pr-body.md", + "reason": "delivery/operational record retained in the external recovery archive" + }, + { + "path": "profiling-default-20260921/delivery/review-response.md", + "reason": "delivery/operational record retained in the external recovery archive" + }, + { + "path": "issue-212-validation/pr-body.md", + "reason": "delivery/operational record retained in the external recovery archive" + }, + { + "path": "issue-212-validation/pr-body-review.md", + "reason": "delivery/operational record retained in the external recovery archive" + }, + { + "path": "issue-270-validation/pr-body.md", + "reason": "delivery/operational record retained in the external recovery archive" + }, + { + "path": "issue-258-validation/pr-body.md", + "reason": "delivery/operational record retained in the external recovery archive" + }, + { + "path": "issue-258-validation/audit-pr-body.md", + "reason": "delivery/operational record retained in the external recovery archive" + }, + { + "path": "issue-258-validation/step9/pr-body.md", + "reason": "delivery/operational record retained in the external recovery archive" + }, + { + "path": "issue-270-anchored-validation/pr-body.md", + "reason": "delivery/operational record retained in the external recovery archive" + }, + { + "path": "issue-184-validation/pr-body.md", + "reason": "delivery/operational record retained in the external recovery archive" + } + ] +} diff --git a/doc/local-validation-archive/sources.zip b/doc/local-validation-archive/sources.zip new file mode 100644 index 00000000..a9784529 Binary files /dev/null and b/doc/local-validation-archive/sources.zip differ