Repository navigation
Action: GPU speedups and gradients, scaling fixes, portable build option - #248
Merged
Merged
Conversation
…ocate too much memory.
- Build with CUDA 13: drop the removed thrust/async/for_each.h include, include <iostream> and <thrust/tuple.h> explicitly. - Double precision: sumRootLikelihoods, justMaximize and cuda_max_l1_norm reduced in float; template them on T (logL now matches CPU action). - Multi-GPU: GPUInterface records the device ordinal and the action plugin calls cudaSetDevice before runtime/cuBLAS/cuSPARSE/Thrust work, so devices other than 0 work. - Block 1-norm estimator (GPUnormest3): correct row-wise max kernel (the reduce_by_key version could not reduce rows and wrote 25 elements), 1-norm over all t columns, Z = (A^T)^p S instead of A^T S, column history as in normest1, skip the unused final-iteration tail. BEAGLE_DEBUG_NORMEST prints GPU vs CPU vs exact ||B^p||_1. - (m,s): nCol is the number of patterns; since all d_p are precomputed, always refine the ||B||_1 choice with them, as the CPU code does. - Leaks: keep spMM buffer reallocations, free old CSR buffers on nnz change, free estimator buffers, fix scratch-space double free on move. - Memory: in action mode the base class skips the dense S x S transition matrices and the large pinned matrix cache; two scratch buffers replace the per-node copy of every partials buffer. S=16384 no longer OOMs. Dense-only entry points (pre-order, partitions, edge likelihoods, transition-matrix get/set) return BEAGLE_ERROR_NO_IMPLEMENTATION instead of computing on unallocated or unset matrices. Validated on A100: GPU logL matches the CPU action plugin to ~1e-15 relative on the MM (M=1/8/32) and IGC workloads; timings within ~2.5% of before, setQ slightly faster. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Stream: all action work (cuBLAS, cuSPARSE, kernels, copies) runs on the GPUInterface's stream 0, where the base class issues its own copies and kernels. For S > 4 that is a created blocking stream, which is not ordered with a second stream; the old host-synchronous Thrust calls hid this race (e.g. the async device-to-host copy of the logL). - Per node: one kernel for the child product, per-pattern max, reciprocal rescale and log/cumulative scalers; the root likelihood is two kernels (per-pattern log likelihood, deterministic weighted sum). No Thrust and no host synchronization on the hot path. - Taylor loop: A = edgeMultiplier * B is never formed (the multiplier goes into alpha; cacheAMatrices and its copies are gone), terms alternate between two buffers shared by both children, F accumulates in place, and F *= eta is folded into the last term. The SpMM buffer is sized once per Q. For small work (nnz * columns < 3e6) one fused column-major kernel per term replaces cuSPARSE's partition + beta fill + SpMM + axpy. - setSparseMatrix: one batched block 1-norm estimator for all powers (chained products, top-t kernel, B^T via csr2csc, one copy to the host): ~50 instead of ~700 GPU operations. B = Q - mu*I is built in CSR directly from the triplets instead of through Eigen. - Tip partials: update the cuSPARSE descriptor when the pointer changes. A100, vs Phase 0 (ms per evaluation incl. setQ): MM M=8, 64 patterns 6.0 -> 1.3; IGC, 1 pattern 16.7 -> 4.8; IGC, 4,096 patterns 1,136 -> 1,034; setQ 3.5-8.1 -> 0.24-2.1. logL matches the CPU action plugin to <= 1.5e-14 relative on the MM/IGC workloads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
macos-latest no longer has Xcode 16, so setup-xcode failed and fail-fast cancelled every other job. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ccache keys on the literal -march=native flag, not on the runner's CPU, so objects built on one runner were reused on another with an older CPU and synthetictest died with 'Illegal instruction' (ubuntu-gcc-13). Make BEAGLE_OPTIMIZE_FOR_NATIVE_ARCH an option (still ON by default) and turn it off in the build-and-test workflow. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…=native Turning -march=native off breaks the SSE plugin (_mm_dp_pd needs SSE4.1), so this reverts the previous commit. The 'Illegal instruction' came from ccache reusing -march=native objects built on a runner with a newer CPU; including the CPU model in the cache key prevents that. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tching
- Store action partials pattern-fastest, [category][state][pattern].
setTipPartials/setPartials upload a transposed copy; getPartials
transposes back. The node product/rescale and root kernels read the
new layout.
- Replace cuSPARSE SpMM + axpy with one custom kernel per Taylor term,
evaluating the polynomial by Horner's rule (3 memory passes per term).
- Wide kernel (>= 32 patterns): one warp per row, 1-4 columns per
lane, grid ordered so the blocks in flight share a column panel.
- Narrow kernel (< 32 patterns): lanes cover (group, pattern), with
the groups' work buffers interleaved so gathers coalesce.
- Batch all categories of both children, and all independent operations
of a dependency level, into one launch per term. Each group keeps its
own (m, s). The products of a level are batched too.
- Override accumulateScaleFactors/removeScaleFactors: the base kernels
read past the scaling buffers when patterns are not padded to their
block size, which the action layout never does.
A100, ms per evaluation (original -> now): IGC 4,096 patterns
1107 -> 227, IGC 64 32.1 -> 5.8, MM M=32 3,052 patterns 152 -> 34.8,
MM M=8 64 patterns 6.6 -> 0.75. logL matches the CPU action plugin to
<= 1.5e-14; compute-sanitizer memcheck/racecheck/synccheck clean.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…norm estimator) - ShiftedCsrBuilder remembers where each triplet of Q lands in B = Q - mu*I, and the structure of B^T. When Q has the same triplet pattern as last time (only the rates change), it only scatters the values, adding them in the same order as a full build (bit-identical). Only B's values go to the device; B^T's values come from one gather instead of csr2csc. - The block 1-norm estimator works on row-major matrices with lanes over the columns of a row, so the gathers are coalesced. Column sums use one block per column, and the unit-vector selection computes each row's maximum once. X0 has the same entries as before; the d_p are unchanged. A100, setQ per call: IGC 1.57 -> 0.62 ms, MM M=32 1.68 -> 0.57 ms, MM M=8 0.44 -> 0.26 ms. IGC at 1 pattern: 2.36 -> 1.41 ms per evaluation. logL unchanged; compute-sanitizer memcheck/racecheck clean. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- updatePrePartials: dest = exp(t Q^T) (exp(t_sib Q) sibling o parent), scheduled in dependency levels and batched in the Horner kernels like post-order. Q^T actions use the device B^T with its own norm estimates. Only a write scaling index rescales, as in the CPU plugin. - Sibling actions: once pre-order is used, post-order keeps each child's exp(t Q) partials in a per-buffer cache. A cached action is used only if the buffer, its matrix (edge length, eigen index) and Q are unchanged, tracked by generation counters, so a rejected proposal undone by index flips or re-sent edge lengths cannot leave a stale action (the CPU plugin's reject/restore bug). Otherwise it is recomputed. - setRootPrePartials, setSparseDifferentialMatrix. - calculateEdgeDerivatives: fused D*post, pre and per-pattern reduction, with D = Q (B + mu I) or a differential matrix; ratio and weighted sums on the device, one copy back. - calculateCrossProducts: SDDMM restricted to Q's sparsity pattern, accumulated into the caller's dense S x S array (the base-class version would read the new partials layout wrongly). - Product kernels accept right == nullptr (rescale only). Matches the CPU action plugin to <= 2e-13 (pre-order partials, dlogL/dt, differential-matrix derivatives, cross products on Q's pattern); 0 wrong gradients in the reject/restore test; compute-sanitizer clean. Gradient (pre-order + derivatives) is 6-33x faster than the CPU plugin on 10 threads for MM M=8..32 and IGC. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With a read scaling index (BEAST's default dynamic rescaling between recomputations), the standard plugins divide each pattern's new partials by the stored scale factor. The CPU and GPU action plugins skipped that while the root still added the factors back, so every evaluation after the first was wrong (BEAST: two clock moves gave -2491.08 instead of -1905.15). - CPU: applyFixedScaling() after post-order and pre-order operations with a read index, over the operation's pattern range. - GPU: the product kernels take optional fixed scale factors (exp'd for log scalers); post-order and pre-order pass the read index through, and the scheduler orders a read after any write of the same scaling buffer. Checked in BEAST on the CPU plugin (dynamic = always = none = from scratch) and with preorder_check on CPU and GPU: reusing the stored factors matches recomputing them to <= 2e-16. compute-sanitizer clean; post-order validation unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Runners with the same CPU model name can expose different instruction-set extensions, so -march=native objects cached on one could still be reused on another (Illegal instruction in synthetictest on ubuntu-gcc-12). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
-march=native was hard-coded, so binaries could crash with "Illegal instruction" on any CPU other than the build machine: GitHub runners differ, and ccache (keyed on the literal flag) reused objects across them. The same happens to BEAST CI, which caches one BEAGLE build for all its runners. - BEAGLE_OPTIMIZE_FOR_NATIVE_ARCH is now an option (default ON, as before). - The SSE plugin's x86 code uses SSE4.1 intrinsics (_mm_dp_pd), which it got from -march=native; it now gets -msse4.1 itself (only the x86_64 slice of macOS universal builds; arm64 uses sse2neon.h). Its load-time check now requires SSE4.1 as well as SSE2. - CI builds with the option OFF, so the per-CPU ccache key is gone. Checked on hmm (x86-64 Linux) and an Apple-silicon Mac (universal, both slices): portable and native builds pass synthetictest, hmctest and actiontest, the SSE plugin loads and gives the same logL as the non-vector plugin, and the portable CPU plugin contains no AVX instructions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings
ji-group/beagle-libactionup to date with the action-based likelihood work since #219: 421 commits, which include the CPU action plugin work (on-demand d_p, threading and partition fixes, the Eigen download) and the GPU work summarized below.GPU action plugin (CUDA)
setSparseMatrix.setSparseMatrix.setRootPrePartials;calculateEdgeDerivatives, including differential matrices;Fixes that also affect the CPU action plugin
Build and CI
BEAGLE_OPTIMIZE_FOR_NATIVE_ARCHis now a CMake option. The default is still ON, so builds are unchanged. OFF gives portable binaries, e.g. for CI caches and distribution;-march=nativeobjects reused across GitHub runners crashed with "Illegal instruction".-msse4.1itself (it relied on-march=native), and checks for SSE4.1 when it loads.macos-15.Validation
🤖 Generated with Claude Code