diff --git a/ALGORITHMS.md b/ALGORITHMS.md index 7529a12..0d6eec3 100644 --- a/ALGORITHMS.md +++ b/ALGORITHMS.md @@ -1,13 +1,14 @@ # OpenStrap Analytics — Algorithms -Every metric here is a **published, peer-reviewed method**, computed by a **pure -deterministic function**, running on the 1 Hz substrate a WHOOP 4.0 actually exposes: +This index includes published methods and internal rules. Unpublished rules are +identified below as ESTIMATE with low confidence. Computations are pure and +deterministic, running on the 1 Hz substrate a WHOOP 4.0 actually exposes: beat-to-beat RR (pulse-derived — this is **PRV, not ECG-HRV**, and that distinction matters more than it sounds like it should), 1 Hz heart rate, 1 Hz tri-axial accel, and relative-only ADC channels (skin temp, SpO2, ambient light). This file is the index — what's implemented, where it lives, and who published it. If a -row here doesn't have a real citation next to it, that's a bug in this file, go fix it. +row describes an internal rule, its component citations support only those components. See `docs/ALGORITHM_CATALOG_1HZ.md` for the underlying literature review and what's deliberately not shipped. @@ -30,8 +31,8 @@ A few multi-day/list outputs (`illnessCusum`, `multivariateAnomaly`, confidence/tier that applies across a whole list. **Tiers**: `AUTH` = directly measured/definitional. `HIGH` = strong literature support on -this substrate. `ESTIMATE` = published, but estimate-grade once run on a wrist 1 Hz -signal instead of a lab setup. `RELATIVE` = meaningful only as a deviation from the +this substrate. `ESTIMATE` = estimate-grade on a wrist 1 Hz signal, or an internal rule without +published validation. `RELATIVE` = meaningful only as a deviation from the user's own baseline (skin temp, SpO2 — never an absolute unit). **Absent input → `null` + confidence `0`, always.** A baseline-gated metric that doesn't @@ -47,24 +48,24 @@ Grouped by family (subdirectory under `lib/src/onehz/`). File paths are relative ### `foundations/` — shared math every family builds on | Function | File | Method | |---|---|---| -| `correctRr` | `foundations/rr_correction.dart` | Lipponen & Tarvainen 2019 RR artifact correction (dRR/mRR/sRR beat classification, Kubios-style) | +| `correctRr` | `foundations/rr_correction.dart` | Lipponen & Tarvainen 2019 RR artifact correction (dRR/mRR/sRR classification); Peltola 2012 correction policy; unpublished artifact-run cap is an internal rule, ESTIMATE with low confidence | | `Baselines` (Winsorized-EWMA) | `foundations/ewma_baselines.dart` | Winsorized exponentially-weighted moving baseline — the rolling personal reference most other metrics compare against | | inverse-variance fusion | `foundations/fusion.dart` | Standard inverse-variance weighting for combining multiple noisy estimates of the same quantity | ### `clinical/` — Tier-1 cardiac/autonomic metrics | Function | File | Method | Citation | |---|---|---|---| -| `hrvTime` | `clinical/hrv_time.dart` | RMSSD/SDNN/pNNx | standard time-domain HRV | +| `hrvTime` | `clinical/hrv_time.dart` | RMSSD/SDNN/pNNx | Task Force 1996 | | `nocturnalRmssd` | `clinical/hrv_time.dart` | median-of-5-min-window nightly RMSSD | — | | `sleepSessionWindowedRmssd` / `sleepSessionRmssdDetail` | `clinical/hrv_time.dart` | mean-of-5-min-window RMSSD; windows need ≥ 20 clean successive differences (ultra-short RMSSD literature); Malik 20 % ectopic rejection | Malik et al.; Munoz et al. 2015; Baek et al. 2015 | | `rrCoverage` | `clinical/hrv_time.dart` | Σ plausible RR ÷ wall-clock span; > 1.10 means duplicated or interleaved beats, and `hrvTime` / `nocturnalRmssd` / `nightHrvShape` / the sleep-session headline refuse RMSSD; the headline also drops any 5-min window holding more beat-time than its length | — (integrity check; contiguous runs measure 0.963–1.001) | | `hrvFreq` | `clinical/hrv_freq.dart` | LF/HF via Lomb-Scargle periodogram on native (unevenly-sampled) beat times | Laguna, Moody & Mark 1998; Bigger 1992 | | `decelerationCapacity` / `accelerationCapacity` | `clinical/prsa.dart` | Phase-rectified signal averaging (DC/AC) | Bauer et al. 2006 | -| `nocturnalRhr` / `hrDip` | `clinical/nocturnal.dart` | nocturnal resting HR + dip classification | — | +| `nocturnalRhr` / `hrDip` | `clinical/nocturnal.dart` | nocturnal resting HR + internal HR-dip classification rule, ESTIMATE with low confidence | Avram 2019; Dial 2025 support nocturnal RHR only; HR-dip bands are unvalidated | | `illnessCusum` | `clinical/illness_cusum.dart` | Online CUSUM state machine (green/yellow/red) over RHR — "NightSignal" | Alavi et al. 2022; Mishra et al. 2020 | | `readinessLnRmssd` | `clinical/readiness_lnrmssd.dart` | ln(RMSSD) z-scored against a rolling prior-nights baseline | Plews et al. 2013 | | `cosinor` | `clinical/cosinor.dart` | Cosinor rhythmometry (MESOR/amplitude/acrophase) | Halberg & Nelson 1979 | -| `banisterTrimp` | `clinical/load_trimp.dart` | Training impulse from HR-reserve | Banister 1991 | +| `banisterTrimp` | `clinical/load_trimp.dart` | Training impulse from HR-reserve | Morton, Fitz-Clarke & Banister 1990; sex-specific coefficients retained from Banister 1991 | | `dailyQuietWakingHrr` | `clinical/load_trimp.dart` | one day's quiet-waking level: median per-minute %HRR over waking minutes; null above 40 % HRR | Karvonen 1957; ACSM moderate-intensity floor | | `personalQuietWakingHrr` | `clinical/load_trimp.dart` | THIS user's quiet-waking level: median of the trailing 28 prior days' `dailyQuietWakingHrr`; abstains (need_baseline) below 3 days, "calibrating" below 7 | — | | `netTrimpAboveQuiet` | `clinical/load_trimp.dart` | Banister TRIMP above the quiet level, split at 40 % HRR so quiet minutes never offset exercise minutes (living block floored at 0) | Banister 1991; Morton 1990; ACSM 40 % HRR | @@ -72,9 +73,9 @@ Grouped by family (subdirectory under `lib/src/onehz/`). File paths are relative | `strainCurveFromSeries` | `clinical/load_trimp.dart` | cumulative 0-21 strain per minute, same arithmetic; last point == headline | — | | `strainScoreMetric` | `clinical/load_trimp.dart` | LUMP form (TRIMP minus wake-minutes × quiet cost) log-squashed onto 0-21; can debit exercise, kept only for rescaling a stored TRIMP with no per-minute series | — | | `trimpStrain` | `clinical/load_trimp.dart` | TRIMP → 0-100 strain, honesty-wrapped (absent without real HRmax/RHR anchors) | — | -| `ctlAtlTsb` | `clinical/load_trimp.dart` | Fitness-Fatigue-Form: EWMA CTL (42d) / ATL (7d) / TSB = CTL-ATL | Banister impulse-response model | +| `ctlAtlTsb` | `clinical/load_trimp.dart` | Fitness-Fatigue-Form: EWMA CTL (42d) / ATL (7d) / TSB = CTL-ATL | Banister et al. 1975; Morton et al. 1990; 42/7-day defaults and priming are implementation choices | | `baevskyStressIndex` | `clinical/stress_si.dart` | Baevsky Stress Index, 5-min sliding RR-histogram | Baevsky & Berseneva 2008 | -| `irregularBeatScreen` | `clinical/irregular_rhythm.dart` | Poincaré SD1/SD2 + pNN70 irregular-rhythm screen | — (screen, not a diagnosis; no ECG) | +| `irregularBeatScreen` | `clinical/irregular_rhythm.dart` | Poincaré SD1/SD2 + pNN70 irregular-rhythm screen | Internal rule, ESTIMATE with low confidence; Task Force 1996 supports successive-difference components only; SD1/SD2 thresholds and sustained-window voting are unvalidated | | `cardiacCoherence` | `clinical/cardiac_coherence.dart` | peak-band vs. remaining spectral power ratio, live RR during guided breathing | McCraty & Zayas 2014; McCraty et al. 2009 | ### `sleep/` — staging, structure, circadian @@ -85,11 +86,13 @@ Grouped by family (subdirectory under `lib/src/onehz/`). File paths are relative | `detectNaps` | `sleep/nap.dart` | **the only nap source.** z-angle immobility bouts on the complement of the main sleep window, corroborated by an HR dip vs the user's *awake daytime* baseline. Reports TST and TIB separately; makes **no stage claim**. Deliberately NOT `AdvancedSleepStager`, whose `minSleepMin=60` + 90-min daytime guard exist to reject naps | van Hees et al. | | `segmentSleep` (`SleepSegmentation`) | `sleep/segment.dart` | **single source of truth** for sleep windowing; delegates staging to `cardioStager` | | `cardioStager` | `sleep/cardio_stager.dart` | Webster/Cole-Kripke actigraphy + HRV fusion — explicitly replaces Walch 2019 (documented WAKE over-call bias) | +| `CausalStager.observe` | `sleep/causal_stager.dart` | **online** wake/NREM/REM for live decisions: `cardioStager`'s features, REM weights and cutoffs on past-only (trailing) windows and expanding baselines; returns stage + confidence + evidence age + a serialisable next state, and **abstains with a reason** (stale/off-wrist/missing HR or accel/thin coverage/warm-up) rather than guess. No look-ahead, deterministic, replacement-tolerant. Not PSG, not a replacement for `cardioStager` — see `docs/CAUSAL_STAGER.md` | `cardioStager`'s basis (linear-discriminant cardiorespiratory staging: Long 2016; Fonseca 2015/2018; weights measured on DREAMT); the causal adaptation is an unpublished internal rule, ESTIMATE with low confidence | +| `stageMinuteRanges` | `sleep/stage_ranges.dart` | per-stage [lo, hi] minutes from `cardioStager`'s `epochEvidence`: uncovered epochs could be any stage, scores within `kStageScoreMargin` of a cutoff count toward both neighbours. A bound, not a confidence interval | Internal margin/bound calibration rule, ESTIMATE with low confidence; underlying staging: Long 2016; Fonseca 2015/2018 | | `AdvancedSleepStager` | `sleep/advanced_stager.dart` | AASM-style 4-class staging + hypnogram metrics (TIB/TST/SOL/WASO/REM-latency); `StagingMethod.cardio` is the wired production default, v1/v2 kept for regression coverage only | | `sleepAccounting` | `sleep/accounting.dart` | TIB/TST/WASO/efficiency accounting | — | | `cardiopulmonaryCoupling` | `sleep/cpc.dart` | cardiopulmonary coupling | — | | `sleepCyclesMetric` | `sleep/cycles.dart` | fractal ultradian sleep-cycle detection | Rosenblum et al. 2024 | -| `circadianNonparametric` | `sleep/circadian_np.dart` | IS/IV/RA/L5/M10 nonparametric circadian indices | — | +| `circadianNonparametric` | `sleep/circadian_np.dart` | IS/IV/RA/L5/M10 nonparametric circadian indices | Witting 1990; van Someren 1999 | | `phillipsSri` | `sleep/sri.dart` | true Phillips epoch-agreement Sleep Regularity Index | Phillips et al. 2017 | ### `respiration/` @@ -97,7 +100,7 @@ Grouped by family (subdirectory under `lib/src/onehz/`). File paths are relative |---|---|---|---| | `rsaRespRate` | `respiration/resp_rate.dart` | respiratory sinus arrhythmia — HF spectral peak of the RR series: Lomb–Scargle on native beat times in 300 s Welch sub-windows (≥ 80 % covered by clean beats), median across them with an agreement gate; beat-rate Nyquist from the median NN | Welch 1967; Lomb 1976; Scargle 1982; Press & Rybicki 1989; DeBoer, Karemaker & Strackee 1984 | | `riivRespRate` / `fuseRespRate` | `respiration/resp_rate.dart` | respiration-induced intensity variation, fused with the RSA estimate | Pimentel et al. (multi-grid RIIV fusion) | -| `cvhrApnea` / `cvhrApneaScreen` | `respiration/cvhr_apnea.dart` | cyclic-variation-in-HR apnea screening | — | +| `cvhrApnea` / `cvhrApneaScreen` | `respiration/cvhr_apnea.dart` | cyclic-variation-in-HR apnea screening | Hayano 2011 | | `relativeOdi` | `respiration/relative_odi.dart` | ratio-of-ratios relative desaturation index — **never an absolute SpO2 claim** | — | | `breathingRateVariability` | `respiration/brv_trend.dart` | within-user breathing-rate variability trend | — | @@ -116,6 +119,7 @@ Grouped by family (subdirectory under `lib/src/onehz/`). File paths are relative | `hrRecovery` | `workout/hr_recovery.dart` | HRR — HR drop N seconds post-peak | Cole/Lauer 1999-style HRR | | `Calories.dailyEnergy` / `estimateBoutCalories` | `workout/calories.dart` | Keytel HR→kcal regression + Harris-Benedict/Mifflin BMR | Keytel et al. 2005 | | `Calories.metFromCadenceSpm` | `workout/calories.dart` | walking METs from measured cadence, for sub-flex-gate minutes (100/110/120/130 spm ↔ 3/4/5/6 METs, linear between, clamped) | Tudor-Locke et al. 2019 (CADENCE-Adults) | +| `Calories.minuteEnergy` / `hourlyRollup` | `workout/calories.dart` | the `dailyEnergy` computation exposed per minute (same gate, same billing; folds back to the daily `active`/`walking` bit-for-bit) plus an hourly rollup with per-hour coverage. A minute with no usable HR and no billable cadence abstains (null + reason); nothing is filled or interpolated | Keytel et al. 2005; Tudor-Locke et al. 2019 | ### `wellness/` | Function | File | Method | Citation | @@ -138,6 +142,45 @@ Grouped by family (subdirectory under `lib/src/onehz/`). File paths are relative | `sleepNeed` / `strainTarget` / `recommendedBedtime` / `recommendedWake` / `sleepPerformance` | `human/coaching.dart` | deterministic coaching layer over the metrics above (naps come from `sleep/nap.dart`, not here) | — | | `journalCorrelations` | `human/coaching.dart` | per-tag mean-difference correlation vs. logged outcomes, on-device, personal | — | +### Mergeable and incremental forms + +Paths below are relative to `lib/src/onehz/`. Structural combinators, codecs, +checkpoints and caches add no physiological estimator. Tolerances in the adjacent +code comments describe parity tests, not universal numerical guarantees. + +| Function | File | Method | Citation | +|---|---|---|---| +| `cosinor / cosinorAggregator` | `clinical/cosinor_aggregator.dart` | Centered least-squares sufficient statistics; relative singularity gate | Halberg & Nelson 1979 | +| `hrvTimeAggregator` | `clinical/hrv_time_aggregator.dart` | Mergeable time-domain statistics; timestamp behavior requires gapAware:true | Task Force 1996; internal jitter rule is ESTIMATE with low confidence; Welch 1967; Hirsch & Bishop 1981 | +| `RMSSD jitter gate / JitterEvidenceAggregator` | `clinical/rmssd_gate.dart; clinical/hrv_jitter_evidence.dart` | Shared spectral evidence; internal rejection/exemption threshold rule, ESTIMATE with low confidence | Welch 1967; Hirsch & Bishop 1981 | +| `banisterTrimpAggregator` | `clinical/load_trimp_aggregator.dart` | Mergeable per-minute Banister TRIMP sum | Morton, Fitz-Clarke & Banister 1990; sex-specific coefficients retained from Banister 1991 | +| `ctlAtlTsbAggregator` | `clinical/load_trimp_aggregator.dart` | Bounded-prefix priming and affine EWMA composition | Banister et al. 1975; Morton et al. 1990; 42/7-day defaults and priming are implementation choices | +| `prsaAggregator` | `clinical/prsa_aggregator.dart` | Mergeable PRSA anchor/profile sums | Bauer et al. 2006 | +| `readinessLnRmssdFromSuffix / readinessLnRmssdAggregator` | `clinical/readiness_lnrmssd.dart; clinical/readiness_lnrmssd_aggregator.dart` | Shared readiness arithmetic over the required suffix; positional aggregator history | Plews et al. 2013 | +| `circadianNpAggregator` | `sleep/circadian_np_aggregator.dart` | Mergeable IS/IV/RA/L5/M10; near-tie start epochs can differ | Witting 1990; van Someren 1999 | +| `bestCircularWindow` | `sleep/circadian_window.dart` | Shared circular L5/M10 search | Witting 1990; van Someren 1999 | +| `sriAggregator` | `sleep/sri_aggregator.dart` | Integer adjacent-day epoch agreement counts | Phillips et al. 2017 | +| `HypnogramAggregator / AdvancedSleepStager.hypnogramMetrics` | `sleep/hypnogram_aggregator.dart` | TST sums sleep stages; SOL from session start; efficiency = TST/session duration, capped at 1. Session bounds substitute for lights-off/on; inferred stages, explicit wake only between first and final sleep, no gap imputation or segment clipping | Berry et al., AASM Manual; application boundary conventions differ as stated | +| `correctRr / RrCapWindow / RrCorrector` | `foundations/rr_correction.dart; foundations/rr_correction_kernel.dart; foundations/rr_correction_stream.dart` | Shared capped batch/stream correction; internal artifact-run cap rule is ESTIMATE with low confidence | Lipponen & Tarvainen 2019; Peltola 2012 | +| `RrSandwich / NnColumn` | `foundations/rr_sandwich.dart` | Re-corrects raw seams, preserves source timestamps; uncapped long-run limitation | Lipponen & Tarvainen 2019; Peltola 2012; inner method attribution applies | +| `EpochRows / EpochRowsMemo / CardioRowCache / cardioEpochFeatures / classifyCardioEpochs` | `sleep/epoch_rows.dart; sleep/cardio_row_cache.dart; sleep/cardio_stager.dart` | Session-anchored feature rows and dependency-checked reuse; internal fitted weights and calibration rules are ESTIMATE with low confidence | van Hees 2013; Long 2016; Fonseca 2015/2018 | +| `HrvFreqStream` | `clinical/hrv_freq_stream.dart` | Append-only batch-equivalent Welch band processing | Laguna, Moody & Mark 1998; Bigger 1992; Welch 1967 | +| `RsaRespStream` | `respiration/resp_rate.dart` | Append-only RSA windows and exact median-NN ceiling | Welch 1967; Lomb 1976; Scargle 1982; Press & Rybicki 1989; DeBoer, Karemaker & Strackee 1984 | +| `CvhrStream` | `respiration/cvhr_apnea.dart` | Shared batch/stream CVHR screening | Hayano 2011 | +| `irregularBeatScreenDetailed / irregularWindowVerdict / IrregularScreenState` | `clinical/irregular_rhythm.dart; clinical/irregular_window.dart; clinical/irregular_rhythm_state.dart; clinical/irregular_diagnostics.dart` | Internal SD1/SD2, pNNx and sustained-window rule, ESTIMATE with low confidence | Task Force 1996 supports successive-difference HRV components only, not classifier calibration | +| `RunningMoments` | `incremental_core.dart` | Welford updates and algebraic removal with nonnegative M2 clamp; partition merge | Welford 1962; Chan, Golub & LeVeque 1983 | +| `IntHistogram` | `incremental_core.dart` | Exact expanded-list percentileSorted order statistics, Hyndman-Fan type 7 | Hyndman & Fan 1996 | +| `IncrementalLombScargle` | `incremental_core.dart` | Fixed-grid sufficient sums for Lomb-Scargle | Press & Rybicki 1989 | +| `IncrementalHrvTime` | `incremental_hrv.dart` | Incremental HRV with batch jitter fallback | Task Force 1996; internal jitter rule is ESTIMATE with low confidence; Welch 1967; Hirsch & Bishop 1981 | +| `IncrementalEnmoSeries` | `incremental_enmo.dart` | Explicit-reference motion updates; automatic calibration uses batch | van Hees 2013; Vähä-Ypyä 2015; internal dynAmp adaptation | +| `IncrementalMinuteMetrics` | `incremental_minutes.dart` | Keyed batch TRIMP and energy contributions | Banister 1991; Morton 1990; Keytel 2005; Mifflin-St Jeor 1990; Tudor-Locke 2019 | +| `HeartRateZones.timeInZone / timeInZoneAggregator` | `workout/time_in_zone_aggregator.dart` | Internal cadence-validation, median-capped interval and tail-credit rule, ESTIMATE with low confidence; exact gap histograms preserve it | Karvonen 1957 supports the HR-reserve zone option only | +| `hrDipFromDayTotals` | `clinical/nocturnal.dart` | Internal HR-dip band and minimum-sample rule, ESTIMATE with low confidence; sample-order day totals and all-valid-sample night mean | Avram 2019; Dial 2025 support nocturnal RHR only, not HR-dip classification | +| `Calories.minuteEnergyPricer / assembleMinuteEnergySeries` | `workout/calories.dart` | Shared dailyEnergy minute billing and ordered assembly | Keytel et al. 2005; Mifflin-St Jeor 1990; Tudor-Locke et al. 2019 | +| `slidingMedians` | `util.dart` | Same clipped-window median via maintained sorted values; percentileSorted uses type 7, zero-based rank (n - 1) * p / 100 | Hyndman & Fan 1996 | +| `lombScargle` | `util.dart` | First-sample time origin reduces argument error; mathematically time-shift invariant | Press & Rybicki 1989 | +| `Sum / SumOf / SumSq / Moments / Ewma` | `../aggregate/primitives.dart` | Compensated Sum/SumOf and ordinary SumSq; centered moments; affine EWMA composition, algebraically equal to the recurrence: f_y(f_x(c)) = a*a*c + a*lambda*x + lambda*y, a = 1-lambda | Welford 1962 and Chan, Golub & LeVeque 1983 for Moments only; sums are arithmetic, EWMA composition follows algebra | + --- ## What this deliberately doesn't do @@ -160,3 +203,13 @@ al. 2017 · Pimentel et al. · Brage et al. 2004 · Keytel et al. 2005 · Tudor- 2019 · Hopkins 2004 · Mahalanobis 1936 · Killick et al. 2012 · Wittmann & Roenneberg 2006 · Pietilä et al. 2018 · Malik et al. + + +References added for the methods above: + +- Welford, B. P. (1962). Note on a method for calculating corrected sums of squares and products. Technometrics 4(3):419-420. +- Chan, T. F., Golub, G. H., LeVeque, R. J. (1983). Algorithms for computing the sample variance: analysis and recommendations. The American Statistician 37(3):242-247. +- Hyndman, R. J., Fan, Y. (1996). Sample quantiles in statistical packages. The American Statistician 50(4):361-365. +- Banister, E. W., Calvert, T. W., Savage, M. V., Bach, T. (1975). A systems model of training for athletic performance. Australian Journal of Sports Medicine 7:57-61. +- Morton, R. H., Fitz-Clarke, J. R., Banister, E. W. (1990). Modeling human performance in running. Journal of Applied Physiology 69(3):1171-1177. +- Berry, R. B., et al. The AASM Manual for the Scoring of Sleep and Associated Events: Rules, Terminology and Technical Specifications. American Academy of Sleep Medicine. Used for duration terminology; application conventions are listed in the hypnogram row. diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..448cbbd --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,137 @@ +# Changelog + +## Unreleased + +### Added +- `stageMinuteRanges`: a [lo, hi] range of minutes per sleep stage (wake, + light, deep, REM, and NREM = light + deep). `lo` counts epochs that can only + be that stage: HR-covered, labelled so, and every deciding score at least + `kStageScoreMargin` (0.1) from its cutoff. `hi` adds epochs that could still + be that stage: uncovered or unlabelled epochs (any stage), and epochs with a + score inside the margin (REM score: REM and its NREM substage; deep score: + light and deep). An honest bound, not a confidence interval: the stager has + no per-epoch probabilities. Null for no epochs. +- `CardioStagerResult.epochEvidence` (with `StageEpochEvidence`): per final + epoch, the label, deep flag, HR coverage and the REM/deep scores the classify + loop computed. `CardioStagerResult` carries `remScores`, `deepScores` and + `hrCovered` for it. Stage labels are unchanged. +- `irregularBeatScreenDetailed` and `IrregularScreenState.evaluateDetailed`: the + irregular-rhythm screen's verdict plus its evidence (`IrregularDiagnostics`) — + beats in / kept after the [300, 2000] ms filter, the corrector's raw / + corrected / dropped counts (passed in as `RrCleaningCounts`), artifact + fraction, window counts (total, voting, flagged, the final OPEN window and + what became of it, sustained share seen vs required) and the thresholds — for + a screen that ran and for one that abstained (with the reason). Detailed + diagnostics preserve the wrapper's result for the same screen calculation. + `IrregularScreenState` checkpoints are version 3, carrying diagnostic counters + and the previous kept beat's time. Versions 1 and 2, and incomplete version 3 + states, are refused. A NaN beat no longer makes the checkpoint unencodable. + `IrregularDiagnostics.artifactFraction` is null (absent) when the corrector was + handed no beats: a share of nothing has no value, and the `1 - cleanFraction` + of an empty series (1.0) must not read as "every beat is an artifact". + Output change: timestamp-gap pairs no longer contribute successive differences, + so Poincaré statistics, pNNx, window votes and screen verdicts can change. +- Incremental states for centered moments, fixed-grid Lomb–Scargle, time-domain + HRV, motion with a measured gravity reference, and keyed minute TRIMP/energy. + Appends reuse retained contributions; edits, removals and changed dependencies + update or rebuild them. Batch APIs remain available. `phillipsSri` uses + `sriAggregator` for normal inputs, retaining the legacy short-validity-mask + path; `AdvancedSleepStager.computeMetrics` uses `HypnogramAggregator`. + The batch functions listed under Changed also run their aggregators. Other + batch implementations remain separate, sometimes sharing kernels/scoring + helpers with incremental forms. +- Bounded dependency-aware `CalculationCache` and explicit `CalculationMode`. + Only `periodicAwake` permits reuse. Sleep, heavy and forced runs use full math. +- Minute summary mode avoids constructing per-minute energy records when only + totals are needed. See `docs/INCREMENTAL_USAGE.md` for contracts and limitations. +- Parameterized incremental parity tests, checked-in capture fixtures, checkpoint + tests, work counters and a deterministic timing comparison tool. +- `hrDipFromDayTotals`: `hrDip` from the day side's running count and sum, for + callers that keep the waking day as totals. `hrDip` shares its scoring, so + both give the same result for the same samples. +- `IncrementalMinuteMetrics.sync` takes `dayMinutes` as an `int`, as + `Calories.dailyEnergy` does; a fractional day can no longer be passed. +- Long-run drift tests: 30 000 Lomb–Scargle appends and 100 000-step sliding + moments stay within 1e-9 of the batch results. +- `IntHistogram`: exact, mergeable order statistics for integer-valued series + (heart rate), kept as a sorted `value -> count` list. `percentile` and `median` + return the same doubles as `percentileSorted` on the expanded sorted list, + and null when empty. Not wired into any caller yet; see + `docs/INCREMENTAL_USAGE.md` for which batch callers could adopt it. +- Pinned Nix devshell with Flutter 3.41.6 / Dart 3.11.4 and native build tooling. +- `Calories.minuteEnergy` — per-minute energy (basal, active, walking, total, + and an abstention reason) from the same computation `Calories.dailyEnergy` + does. A minute with no usable HR and no billable cadence abstains: all values + null, `abstained: MinuteEnergyAbstain.noHr`. Nothing is filled or interpolated. + `MinuteEnergySeries.active` / `.walking` are `==` `dailyEnergy`'s, and + `basalKcalPerMin * dayMinutes` is its `basal`. +- `Calories.hourlyRollup` — hourly sums of covered minutes with per-hour + coverage; an hour with no covered minute is null, not zero. + +### Changed +- **Output change (combined bump; edge must bump kAlgoVersion when pinning + this): capped RR artifact runs become gaps; downstream metrics may change.** + `correctRr(maxArtifactRun:)` and `RrCorrector(maxArtifactRun:)` (default + `kMaxArtifactRun` = 32; null = uncapped) split over-cap runs and re-correct + each side. Without a cap split the result equals uncapped correction. + Staging retains the pre-bump session-onset 30 s grid and labels, except for + separately documented RR or spectral changes. After an accelerometer gap + exceeding the 60 s carry-forward limit, the next usable run re-anchors at + its first sample, as before. `EpochRows` accumulates these feature rows; + `EpochRowsMemo` caches rows by grid phase and recomputes an unavailable phase. + `RrSandwich` / `RawRr` / `NnColumn` feed capped corrected NN into aggregators, + retaining source-beat absolute timestamps via `nnSourceIndex`. + Registry counters `found.rr_correction` and `sleep.stage` remain at 2. +- **Output change: aggregator adoption (7483030).** `hrvTime`, `banisterTrimp`, + `ctlAtlTsb`, `cosinor`, `HeartRateZones.timeInZone` and + `circadianNonparametric` now return their aggregators' one-slice run. + Ordinary finite-input outputs move only in the last digits within tested + tolerances: HRV fields 1e-9 absolute plus 1e-9 relative; TRIMP 1e-12 relative; + CTL/ATL/TSB 1e-9 absolute and relative; zone seconds 1e-9 absolute / 1e-12 + relative; circadian indices 1e-12 relative on integer fixtures and 1e-9 on + real fixtures. Circadian near-ties can select different L5/M10 start epochs. + Cosinor mesor/amplitude use 1e-9 relative, with a near-zero amplitude floor + of 1e-12 * max(1, |mesor|); near-zero mesor has a 1e-12 data-scale floor. + Circular phase error uses 1e-9 * max(1, |phaseRad|) in radians and + 1e-9 * periodHours in hours; phase is not compared for rounding-noise + amplitudes. R², adjusted R² and confidence use 1e-7 absolute. These are checked + parity tolerances, not universal bounds. + SDANN/SDNN-index pins moved to 1e-9 relative. +- **Output change: near-singular cosinor designs.** The relative determinant + test replaces the old batch solver's absolute 1e-12 pivot test. Insufficient + phase coverage returns a null-valued `Metric` with the singular-design note; + the two tests can disagree about whether a fit is identifiable. +- Registry counters `found.core`, `hrv.freq`, `night.rhythm` and `activity.load` + go to 2 for the spectral origin, HRV, rhythm and load changes. Dependent + groups re-derive through `deps`; see `docs/COMBINED_BUMP_STATUS.md`. +- `IrregularScreenState` checkpoints are version 3: they carry the previous kept + beat's time, because a pair across a time hole is no longer successive. + Versions 1 and 2, and a version 3 checkpoint without `prevT`, are refused. +- `IncrementalHrvTime` follows `hrvTime`'s jitter gate under the ACF floor + (respiratory-line exemption, breathing-line fallback): when either could + apply it hands the night to the batch, so the incremental output equals + `hrvTime` where it used to refuse RMSSD. Checkpoint format unchanged. +- `hrvTimeAggregator` follows `hrvTime`'s jitter gate under the ACF floor + (respiratory-line exemption, breathing-line fallback) from mergeable + evidence (O(5-min windows), never the beats). Its checkpoint state is now + `[core, evidence]`; a checkpoint without the evidence is refused whole. +- Output change for invalid timestamps: `hrvTime`, `nocturnalRmssd`, + `sleepSessionRmssdDetail`, `rsaRespRate` and `cvhrApneaScreen` return absent + metrics for non-finite beat times. Streaming folds refuse invalid times + before mutation; snapshots with invalid provisional times abstain. + Finite-input formulas are unchanged by these guards. +- **Output change: all `lombScargle` callers can move; edge must bump + kAlgoVersion when pinning this.** Time is now measured from the first sample. + The old error reached approximately 5e-6 relative per frequency; integrated + metric changes depend on the input and were approximately 1e-14 on the + checked night. Threshold decisions can therefore change. Callers include + `hrvFreq`, `cardiacCoherence`, respiration RSA/peak rate and the cardio + stager's LF/HF. `IncrementalLombScargle` already used first-sample time and + is unchanged; it now agrees with batch on epoch seconds within 1e-9 in the + precision test, which saw 9e-6 before the change. + +### Unchanged +- `Calories.dailyEnergy` outputs are bit-for-bit identical. Its loop body moved + into a helper shared with `minuteEnergy`; golden tests in + `test/onehz/minute_energy_test.dart` pin the pre-change values with `==`. + No `kAlgoVersion` bump is needed for this change. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 82abddb..447291f 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -68,7 +68,14 @@ There is **one** sleep source (`segmentSleep`) and **one** headline readiness ## Tests +On Linux, `nix develop` supplies the pinned Flutter 3.41.6 / Dart 3.11.4 +toolchain used by Edge, plus Git, Clang, CMake, Ninja, and pkg-config. +The interactive shell enters an FHS environment for native build tools. +For a command without an interactive shell, use `nix develop -c dart test`; +`nix develop -c analytics-fhs ` also supplies the FHS environment. + ```bash +nix develop dart pub get dart analyze dart test diff --git a/README.md b/README.md index 5edee80..521bbca 100644 --- a/README.md +++ b/README.md @@ -85,6 +85,10 @@ if (hrv.value != null) { Every metric function in the package follows this shape: plain `List` (or a small typed input class for the composite ones) in, `Metric` out. +For repeated updates, [incremental calculation states](docs/INCREMENTAL_USAGE.md) +retain contributions while checking the complete current inputs for revisions. +The [math audit](docs/INCREMENTAL_MATH.md) explains the algebra and rebuild rules. + ## What's actually in here Eight families, each its own subdirectory with its own sub-barrel, built on two shared @@ -100,9 +104,9 @@ foundation layers: reference; the 0-100 `trimpStrain` effort score is raw TRIMP with no such subtraction), a Poincaré irregular-rhythm screen, cosinor circadian fitting, and real-time cardiac coherence for guided breathing sessions. - **`sleep/`** — van Hees z-angle segmentation feeding a cardiac/motion stager (the - single source of truth for sleep staging), AASM hypnogram metrics, cardiopulmonary - coupling, fractal sleep-cycle detection, circadian non-parametric indices - (IS/IV/RA/L5/M10). + single source of truth for sleep staging), a causal (online, abstaining) wake/NREM/REM + stager for live decisions, AASM hypnogram metrics, cardiopulmonary coupling, fractal + sleep-cycle detection, circadian non-parametric indices (IS/IV/RA/L5/M10). - **`respiration/`** — RSA-derived respiratory rate fused with motion-modulated RIIV, CVHR-based apnea screening, a relative (never absolute) oxygen-desaturation ratio. - **`motion/`** — ENMO/MAD activity metrics, a 100 Hz AN-2554-derived live pedometer @@ -138,8 +142,9 @@ assertion out. ## Validation -`tool/` has four harnesses that score shipped detectors against labelled corpora, not -synthetic fixtures — run one before touching the logic it covers: +`tool/` has five harnesses that score shipped detectors against labelled corpora (or, for +the causal stager, the one recorded night in-tree) rather than synthetic fixtures — run one +before touching the logic it covers: - `dart run tool/oxwalk_validate.dart ` — the pedometer against OxWalk (Oxford, CC BY): 39 subjects, camera-annotated heel strikes. @@ -147,6 +152,10 @@ synthetic fixtures — run one before touching the logic it covers: against a PSG-labelled corpus (e.g. DREAMT), reporting Cohen's kappa. - `dart run tool/nap_harness.dart ` — the nap detector against hand-labelled days. +- `dart run tool/causal_stager_validate.dart` — replays the in-tree real night through the + causal (online) stager and scores agreement, false- and late-trigger behaviour for the + "wake during REM" use case against the offline stager. There is no PSG-labelled night + in-tree, so this is agreement, not accuracy; see `docs/CAUSAL_STAGER.md`. - `dart run tool/whoop_proportions.dart ` — sweeps sleep-stage cutoffs against normative stage proportions on real device captures. diff --git a/docs/AGGREGATOR_DESIGN.md b/docs/AGGREGATOR_DESIGN.md new file mode 100644 index 0000000..2beab00 --- /dev/null +++ b/docs/AGGREGATOR_DESIGN.md @@ -0,0 +1,234 @@ +# Aggregators: mergeable summaries without a rigid monoid + +Goal: make it cheap to turn an analytics calculation into something that can +be computed in chunks (checkpoint/resume, segment recompute, parallel work) +and merged, with one generic law check instead of hand-written chunking tests. + +## Why not a plain monoid + +A textbook monoid on summary values asks for three things that cost speed or +honesty here: + +1. Immutable values combined one element at a time. Folding 86,400 samples a + day through `combine(summary, singleton(x))` allocated per sample: about + 180,000 µs against a 141 µs hand loop + (edge.research/experiments/dart-monoid/REPORT.md). +2. Structural equality of states. Two states can differ (float rounding + order, a field `finish` never reads) and still give the same metric. +3. One context-free operation. Several calculations depend on context known + before any data arrives: session bounds, a bin origin, heart-rate anchors. + +## The contract + +An `Aggregator` is a dictionary of four operations over slices of a +column batch `B`: + +| op | role | +|---|---| +| `S empty()` | fresh identity state | +| `S addSlice(S s, B batch, int start, int end)` | fold rows `[start, end)`; may mutate `s` and return it | +| `S combine(S left, S right)` | merge two adjacent chunks, `left` first; may mutate and return `left`; consumes both | +| `R finish(S s)` | state to result; must not mutate `s` | + +`Batch` is a run of rows stored column-wise (`length`, `copyRange`, +`join`). Missing doubles are NaN, so filtering happens inside tight loops +instead of through per-row closures. `ListBatch` wraps a plain list for +calculations over objects (hypnogram segments); `DoubleColumn` wraps one +`Float64List`. A module defines its own batch when it needs several columns +(`Beats`, `Epochs`, `TimedValues`). + +What this relaxes, each one on purpose: + +- **Slices, not elements.** The unit of `add` is a slice. Generic dispatch + then costs once per slice and the per-row work is a plain loop. +- **Mutation and consumption.** `addSlice` and `combine` reuse their inputs. + The laws are stated on fresh folds, so nothing reuses a consumed state. +- **Equality after `finish`.** Laws compare finished results with a + caller-supplied equality (exact for integer outputs, a tolerance for + floats). State layout is free. +- **Order matters.** `combine` joins a chunk with the one that follows it, so + a state can carry bounded edge context across a seam. No commutativity. +- **Context-indexed instances.** An aggregator may be built from context + (`HypnogramAggregator(sessionStart, sessionEnd)`, + `hrvTimeAggregator(originMs: …)`, `banisterTrimpAggregator(restingHr: …)`). + The laws hold per instance. + +## Laws + +`checkLaws(agg, cases, equals: …)` (lib/src/aggregate/laws.dart) checks, all +through `finish` and `equals`: + +1. left identity: `combine(empty, fold xs) ≈ fold xs` +2. right identity: `combine(fold xs, empty) ≈ fold xs` +3. split: for split points k (every k for short cases), + `combine(fold [0,k), fold [k,n)) ≈ fold xs` +4. associativity on three adjacent slices +5. singletons: left-combining one-row folds ≈ `fold xs` +6. slicing: one state fed random consecutive sub-slices ≈ `fold xs` (the + `addSlice`-after-data path, separate from `combine`) +7. finishPure: finishing a state twice gives the same result, and the state + keeps working afterwards +8. roundTrip (persistable aggregators only): see Checkpoints + +Law 3 is what the app needs: any chunking gives the same metric. The harness +tests itself against deliberately broken aggregators +(test/aggregate/laws_harness_test.dart), so a lawful verdict means something. +In tests, `expectLawful(agg, cases, equals: …)` is the one-liner. + +## Checkpoints + +A state can be stored and resumed. `encodeState` / `decodeState` turn a +state into JSON and back; `persistable` says whether an aggregator supports +it (a composite is persistable when all its children are). Doubles round-trip +exactly; NaN and ±Infinity are written as the strings `"NaN"`, `"Infinity"`, +`"-Infinity"`. Decoders throw `FormatException` on anything malformed. + +`encodeCheckpoint(agg, state, kind:, version:)` wraps a state with a kind and +version. `decodeCheckpoint` returns null, and never guesses, when the kind or +version differs or the payload is damaged; the caller then recomputes from +raw data. Bump the version whenever a module's state layout or arithmetic +changes. + +When an aggregator is persistable, `checkLaws` adds the law `roundTrip`: a state that went through JSON combines and continues exactly +like the original. So every `expectConversion` call also checks persistence. + +## Building blocks + +All lawful, so anything assembled from them is lawful too: + +- counts and sums: `Count`, `CountFinite`, `CountWhere`, `Sum` + (Neumaier-compensated), `SumSq`, `SumOf(col, f)` (sum of `f(x)`, NaN skips) +- extremes and edges: `Min`, `Max`, `First`, `Last` +- moments: `Moments` (wraps `RunningMoments`) +- composition: `both(a, b)` (mutable pair state, no per-row allocation), + `.map(f)` on the result +- grouping: `GroupBy(key, inner)` (consecutive equal keys reach `inner` as one + slice; per-key order kept) +- suffixes: `LastRows(k)` (the last k rows of the whole input, kept + verbatim) +- prefixes: `Prefix(k, head, tail)` (the first k rows of the whole input go + to `head`, kept verbatim, at most k rows; every later row goes to `tail`) +- recurrences: `Ewma(col, λ)` finishes as an `Affine` map (see below) +- neighbour context: `Lookback(k, inner)`. `inner` processes every length-k + window inside each slice it is given. `Lookback` keeps the first and last + k−1 rows and hands `inner` the seam rows on `combine` and on + `addSlice`-after-data, so each window is counted exactly once under any + chunking. `Lookback(2)` gives successive differences, `Lookback(3)` lag + pairs, and `Lookback(epochsPerDay + 1)` the SRI day-to-day comparison. + +Where no block fits, a small hand-written inner (an `Aggregator` with a tight +loop) is fine. It is still checked by the same laws. + +## Two ways to make an "unmergeable" calculation mergeable + +Earlier experiments filed some calculations as not mergeable. Two +techniques moved them across: + +- **Summarise a function, not a value.** An EWMA step + `c ← (1 − λ)c + λx` is an affine map of the running value. Affine maps + compose associatively, so a chunk's summary is the composed map + `(scale, offset)`, and `finish` applies it to the seed. CTL/ATL/TSB is + `both(Count, Prefix(prime, mean, both(Ewma, Ewma)))`. +- **Summarise a distribution exactly when its support is small.** + Time-in-zone credits each sample with `min(gap, m)`, where `m` is the + median gap of the whole stream. The state keeps exact histograms of gap + values, which are bounded by the number of distinct gaps (a handful on a + 1 Hz stream), and the median and crediting move to `finish`. The median is + bit-identical to the batch code. + +## Converting a module: recipe + +1. Pick the batch: an existing one, or a small `Batch` subclass with one typed + column per input. +2. List the sufficient statistics `finish` needs. If they're sums, counts, + extremes or moments, use the blocks; if they need neighbours, wrap them in + `Lookback(k)`; if they're per bin or per day, wrap them in `GroupBy`. +3. Write `finish` (`.map`) from the old function's formulas, including its + absent cases and notes. +4. Tests: one `expectConversion(build, cases, reference)` call checks parity + with the old function on every evaluation path (whole, chunked, + right-nested, copies, sliced, singletons) and every law, including + `roundTrip` once the aggregator is persistable. Add hand-worked value and + absent-case tests as usual. +5. Integer outputs: rewire the old function in place to run the aggregator, + and keep a frozen copy of the old body in the test. Float outputs: keep + the aggregator additive beside the old function, so no published value + moves and no kAlgoVersion bump is needed. + +## Results so far + +| module | how | parity | shape | +|---|---|---|---| +| hypnogram metrics | rewired in place | exact | hand inner over `ListBatch`; wake spans kept as a list (clipped at finish) | +| HRV time domain | additive | 1e-9 | `Lookback(2)`, `Lookback(3)`, `Moments`, `GroupBy` over `Beats` | +| Phillips SRI | rewired in place | exact | `Lookback(epochsPerDay + 1)` + `GroupBy` per day pair over `Epochs` | +| Banister TRIMP | additive | 1e-12 | `both(Count, SumOf)` | +| cosinor | additive | 1e-9 (R² 1e-7) | one inner computing cos/sin once per row; centred co-moments | +| time in zone | additive | 1e-12 | `Lookback(2)` + exact gap histograms; median at finish | +| CTL/ATL/TSB | additive | 1e-9 | `Count`, `Prefix`, `Sum`, two `Ewma` maps | +| readiness lnRMSSD | rewired in place | exact | `Count` + `LastRows(windowDays + 1)` | +| PRSA DC/AC | additive | 1e-9 | `Count` + `Lookback(2l + 1)` with a profile-sum inner | +| circadian IS/IV/M10/L5/RA | additive | 1e-12 int data, 1e-9 real | `Moments`, `GroupBy`, `Lookback(2)` + one 15-line inner | + +Every module above is persistable and resumes from a mid-stream checkpoint. +The circadian conversion, done by following the recipe, took a subagent +about nine tool calls: a 134-line aggregator and a 166-line test. + +Speed, AOT, Dart 3.11.4 on this box (`tool/bench_aggregate.dart`): + +| 30,000 beats, HRV time domain | µs | +|---|---:| +| `hrvTime()` (current batch code) | 4,956 | +| hand loop, same statistics | 639 | +| composed, one slice | 1,281 | +| composed, 24 slices + combine | 1,209 | +| composed, one-row slices | 10,106 | +| per-element composed (earlier API) | ~24,800 | + +The composed version is about 2× a hand loop and about 4× faster than the +current batch function. Chunking costs nothing extra. (It was 1.7× before the +review fixes switched the lag-1 ACF to centred co-moments, which cost one +division per row and removed a cancellation bug.) + +## Proposed next step: corrected HRV end to end + +Every HRV aggregator here starts from already-corrected NN intervals. RR +artifact correction is the upstream blocker the earlier experiments named. +The package already has a streaming `RrCorrector` +(lib/src/onehz/foundations/rr_correction_stream.dart): a left-to-right fold +whose output for a beat is final about 2H beats (H = windowBeats ~/ 2) plus +the current artifact run plus two normal beats behind the newest beat, and +whose checkpoint holds that window, not the series. + +That suggests a mergeable "sandwich" summary for raw RR: + +- raw head: the first beats of a chunk whose correction still depends on the + chunk before it (window, carried class, left normal pair); +- settled middle: corrected beats, folded straight into a downstream + aggregator (HRV time, PRSA, …); +- raw tail: the last beats whose correction still depends on the chunk after. + +`combine` re-runs correction on `left.tail ++ right.head` only, feeds the +newly settled beats to the downstream aggregator, and keeps the remainder as +the new head/tail. That is `Lookback` with a transducer in front of the +inner aggregator, and it is bounded only if artifact runs are bounded. So it +needs a cap on run length (a longer run becomes a gap: honest absence), which +changes output and needs a kAlgoVersion bump. Owner decision. + +## Known limits + +- `Lookback` copies k−1 rows per seam. With large k and tiny slices this + dominates: SRI over 7 days at 1,440 epochs/day takes 0.2 ms as one slice, + 0.5 ms in hourly slices, 17.7 ms in one-epoch slices. Feed slices of at + least k rows, or give `Lookback` an appendable edge buffer if a caller + can't. +- Still batch-only: exact medians and quantiles over floats, pairwise + statistics (sample entropy), Viterbi backtrack, and anything whose context + is unbounded (current RR artifact correction). These need a bounded variant + behind a kAlgoVersion bump, or stay batch. +- Still hard: `nocturnalRhr` slides a TIME window whose qualifying rule + depends on the stream-wide median cadence. A mergeable version would keep, + per possible valid-sample count, the lowest window mean (bounded by the + window length) plus a time-based lookback block, which doesn't exist yet. +- Float results are equal within a tolerance, not bit-identical, across + chunkings. That is why float modules stay additive. diff --git a/docs/CAUSAL_STAGER.md b/docs/CAUSAL_STAGER.md new file mode 100644 index 0000000..d2a457a --- /dev/null +++ b/docs/CAUSAL_STAGER.md @@ -0,0 +1,235 @@ +# Causal sleep stager + +`CausalStager.observe` in `lib/src/onehz/sleep/causal_stager.dart`, exported +from `package:openstrap_analytics/onehz.dart`. + +It answers one question: **right now, using only what the band has reported so +far, is the sleeper awake, in NREM, or in REM — and if it cannot tell, why not?** + +## What it is + +A live, past-only version of `cardioStager`. It uses the same per-epoch +features, the same REM axis weights, the same REM cutoff and the same wake, +atonia and HR-floor rules. Where `cardioStager` looks at both sides of an epoch, +this looks only backwards. + +It is built for decisions that cannot wait for the night to end, such as waking +someone during estimated REM. + +## What it is not + +- **Not polysomnography.** It is a wrist autonomic estimate (tier `ESTIMATE`). + It never labels N1/N2/N3 and never diagnoses anything. +- **Not more accurate than `cardioStager`.** It has fewer smoothing passes. The + retrospective stager itself scores kappa 0.13 against PSG on the held-out + DREAMT split. Agreement with the offline hypnogram below is a consistency + check, not an accuracy figure. +- **Not a sleep-onset detector.** Start it at in-bed time (`prior == null`). If + you feed it the waking afternoon, its baselines come from waking physiology. +- **Not a probability.** `confidence` measures how much evidence there was (HR + and RR coverage, capped at 0.6). It does not say how likely the stage is to be + right. + +## API + +```dart +import 'package:openstrap_analytics/onehz.dart'; + +final obs = CausalStager.observe(window, priorState); // priorState == null at night start +obs.stage; // CausalStage.wake | nrem | rem | absent +obs.confidence; // 0 when absent, else 0.15 .. 0.6 +obs.evidenceAgeMs; // now minus the end of the newest second with valid HR + accel +obs.abstentionReason; // CausalAbstention?, non-null exactly when stage == absent +obs.runSec; // how long this stage has held (0 when absent) +obs.epochStartMs; // the 30 s epoch the stage describes +obs.note; // 'warmup:have=H,need=N' while warming up +obs.trace; // plain-data decision trace (features, thresholds, counters) +obs.nextState; // pass this to the next call +``` + +`CausalSampleWindow({required double nowMs, List hr, List accel, RrSeries? rr})` +uses the package's existing sample types. `nowMs` is the decision time, supplied +by the caller (the package has no clock). `hr == 0` means off-skin. + +### Rules the contract enforces + +- **No look-ahead.** Samples stamped at or after `nowMs` are ignored and counted + in `trace['ignoredFuture']`. An epoch is staged only once it has closed. +- **Deterministic.** The same sequence of windows gives byte-identical + observations and states. Chunking does not matter (30 s windows and one big + catch-up window give the same answer) as long as a single catch-up window + spans no more than `historyEpochs`. +- **Replacement-tolerant.** A window is authoritative for the time span it + covers, per channel. Re-sent, overlapping and corrected windows converge on the + state a clean feed would have produced. Samples more than 600 s behind the + newest stored sample are ignored (`trace['ignoredStale']`); their features are + frozen. Send whole seconds. +- **Never carries a stage across an abstention.** An epoch that cannot be staged + is `absent`; the next answer comes from the next good epoch. + +### Latency + +A stage describes the newest *closed* 30 s epoch, and the RR features use +trailing 3 to 5 minute windows whose centres sit 1.5 to 2.5 minutes behind the +epoch. That is the cost of not looking ahead. Measured against the offline +hypnogram, agreement peaks at a lag of only 30 s and falls off slowly after that, +so one night cannot pin the lag down more precisely than "about a minute or two". + +## State + +`CausalStagerState` is plain data: `toJson()` and `CausalStagerState.fromJson()` +(returns `null` for anything it did not write, so the caller restarts and pays a +warm-up instead of getting a guessed state). It holds: + +| field | contents | +|---|---| +| `config` | `warmupEpochs` (40 = 20 min), `maxEvidenceAgeSec` (120), `historyEpochs` (360 = 3 h), `remScoreCut` (the shipped 0.5) | +| `lastNowMs` | latest decision time; an earlier `nowMs` is a clock regression | +| `lastUsableSec` | newest second with valid on-skin HR and valid accel | +| `hrTail`, `accelTail`, `rrTail` | the last 15 minutes of raw samples (`[sec,bpm]`, `[sec,x,y,z]`, `[ts_ms,rr_ms]`) | +| `rows` | one feature row per closed 30 s epoch (up to `historyEpochs`): motion, HR, HR SD, SDNN, R(k), LF/HF, coverage counts, and the raw stage decided when the epoch closed | + +It is about 160 KB of JSON for a long night, and it crosses an isolate boundary +as is. A call costs about 5 ms on the VM; a 3 hour catch-up in one window takes +about 1 s. Run it in an isolate, not on the UI thread. + +## Abstention reasons + +`absent` always carries one of these. They are checked roughly in this order. + +| reason | meaning | +|---|---| +| `clockRegressed` | `nowMs` is earlier than a time already seen, or not finite. State is returned untouched. | +| `noEvidence` | no sample of any kind has ever arrived | +| `staleEvidence` | nothing at all arrived within `maxEvidenceAgeSec` of `nowMs` (disconnect, out of range) | +| `offWrist` | the newest closed epoch is mostly `hr == 0` | +| `missingHr` | no valid HR in the newest closed epoch (absent, NaN, negative) | +| `missingAccel` | no valid accel in it. HR is checked first, so an epoch with neither says `missingHr`. | +| `lowCoverage` | HR or accel covers under half the epoch (`minHrCoverage`), or recent history is mostly unusable | +| `warmup` | fewer than `warmupEpochs` usable epochs so far; `note` says `warmup:have=H,need=N` | + +The 20-minute warm-up is an engineering floor (enough epochs for the robust-z +baselines and the local HR gates to exist), not a physiological claim. Epochs +that were off-skin or too thin never count toward it. For a Natural Wake window +of `N` minutes, start collecting at least `N` plus the warm-up before the window +opens, plus margin for the lag above. + +## How it differs from `cardioStager` + +| | `cardioStager` | `CausalStager` | +|---|---|---| +| RR windows (SDNN, R(k), LF/HF) | centred, ±150 s and ±90 s | trailing, the same total lengths (300 s, 180 s) | +| 1 g reference for motion | centred 330 s median | **trailing 120 s median** (see below) | +| local HR gates | ±90 min | past 90 min | +| baselines (motion MAD, robust z) | whole night | expanding, up to 3 h | +| Webster rescore, bout consolidation | yes | **no**, they need the future side of a bout | +| mode filter | centred 3-epoch | trailing 2-of-3 vote | +| deep overlay, personal profile | yes | no | + +Same on purpose: motion/HR/SDNN/R(k)/LF-HF definitions, the physiologic RR gate, +`kRemWeight*` (the measured DREAMT effect sizes), `kDefaultRemScoreCut`, the wake +gate (`hr` above local median plus max(6, SD), or big movement with an HR lift +or a big-move predecessor), the atonia and local-p25 HR-floor REM preconditions, +`minHrCoverage`, and the 0.6 confidence ceiling. The shared pieces were lifted +out of `cardio_stager.dart` (`weightedAxisScore`, `cleanRrBeatsBetween`, +`remFeaturesFromBeats`, `kRemWeight*`) so there is one copy, and that refactor is +bit-identical: `cardioStager` output on the real night (stages, deep flags, +confidence, at two cutoffs and with no RR) is unchanged. + +**The 120 s motion reference is the one number chosen on data.** A trailing +window the same length as the centred one must wait for more than half of itself +to be the new posture, so 150 s after every static posture change reads as +movement and the wake gate fires. I tried 300, 180, 120 and 90 s on the one +in-tree night: wake over-call against the offline stager fell 104, 83, 62, 52 min +and kappa rose 0.305, 0.335, 0.369, 0.372. 120 s is the middle of the plateau. +It was selected on the same night it is validated on, so the validation below is +a little optimistic by exactly that much. + +## Validation + +`dart run tool/causal_stager_validate.dart` replays the in-tree real night +(`test/onehz/fixtures/real_night_2026_07_*.csv`, 8 h 54 min) through both stagers. + +**What exists to validate against.** There is no PSG-labelled set in this repo +(DREAMT, used by `tool/stager_harness.dart`, lives outside it). The one real night +has only Apple Watch whole-night minutes, no epoch labels. So everything below is +agreement with the offline stager, on one night from one person. + +Stage minutes: offline wake 26 / NREM 358 / REM 150; causal (staged epochs) +wake 62.5 / NREM 374 / REM 78; Apple Watch summary wake 3 / REM 162 / light 330 +/ deep 38. The causal stager staged 1029 of 1068 epochs; the 39 it declined were +all `warmup`. Replay was deterministic. + +Per-epoch agreement with the offline hypnogram (3 classes): + +| causal(k) against offline(k-d) | agreement | kappa | +|---|---|---| +| d = 0 | 70.3 % | 0.369 | +| d = 1 (best) | 71.3 % | 0.391 | +| d = 4 (2 min) | 69.2 % | 0.343 | +| d = 8 (4 min) | 67.6 % | 0.307 | + +Against the offline stage at the same epoch: wake recall 54 %, precision 22 %; +NREM recall 87 %, precision 79 %; REM recall 36 %, precision 69 %. The wake +precision is low because the causal stager does not bridge short arousals the way +Webster rescoring does after the fact. A short wake run is not yet known to be +short. Read `runSec` before acting on a wake. + +Warm-up length barely moves agreement (kappa 0.369 at 10 and 20 min, 0.365 at +30, 0.350 at 60) and costs staged coverage (98 %, 96 %, 95 %, 89 %). + +### False and late triggers for "wake during REM" + +A trigger is `stage == rem && runSec >= X`. The reference is the 16 REM bouts the +offline stager emits (several are under 5 minutes). Chance level for a trigger +landing on offline REM is 29 %, and within 5 minutes of it 47 %, so the "no false +trigger" test is only as strict as that. + +| X | trigger epochs | on offline REM | within 5 min of it | far from REM | bouts caught (of 16) | median lateness | tail epochs after a bout | +|---|---|---|---|---|---|---|---| +| 60 s | 116 | 70.7 % | 92.2 % | 9 | 14 | 1.0 min | 25 | +| 120 s | 69 | 71.0 % | 100 % | 0 | 11 | 1.5 min | 20 | +| 180 s | 46 | 67.4 % | 100 % | 0 | 7 | 2.5 min | 15 | +| 300 s | 26 | 57.7 % | 100 % | 0 | 3 | 0.5 min | 11 | + +What this says: + +- **False triggers are rare but not absent at short persistence.** At X = 60 s, + 8 % of trigger epochs sit more than 5 minutes from any offline REM. At 120 s and + above there were none. +- **Late triggers are real.** Trailing windows keep the tail of a REM bout in + view, so between a fifth (X = 60 s) and two fifths (X = 300 s) of trigger epochs + fall in the five minutes after a bout ended. For X up to 180 s the first trigger + inside a bout arrives a median 1 to 2.5 minutes after the bout starts. +- **It misses REM, and X = 300 s misses most of it.** REM recall per epoch is + 36 %. The five-minute rule catches 3 of 16 bouts. Two minutes catches 11 of 16. + The cause is mostly a flickery causal REM label (no bout consolidation) plus + the lag. If Natural Wake needs the five-minute rule it will abstain in most + windows on a night like this one. +- **Window simulation** (fire once, at the first trigger in `[T-N, T)`, over every + must-be-up time with a fully observable window): at X = 120 s the trigger fires + in 38 % of 15-minute windows, 56 % of 30-minute, 80 % of 60-minute and every + 120-minute window. All fires landed within 5 minutes of offline REM; 77 to 87 % landed exactly + on it. At X = 300 s: 14 %, 27 %, 43 % and 77 %. The remaining windows contained + offline REM and did not fire (missed), or contained none. + +These are one night. The offline stager is itself an estimate. Do not turn them +into a promise to users. + +## Open caveats + +- One recorded night, no epoch labels, no PSG. The agreement figures are a + consistency check against another estimate. +- The 120 s motion reference was picked on the validation night. +- Wake is over-called relative to the offline stager (62 vs 26 min on the real + night) because Webster bridging is retrospective. +- REM recall is low and arrives late; the persistence threshold is a direct trade + between false triggers and missed ones. +- Baselines assume the stream starts at in-bed time. There is no sleep-onset + detector inside. +- A single catch-up window longer than `historyEpochs` (3 h) gives run lengths + that differ slightly from a live feed of the same data. The final stage and + state agree. +- The decision rules are re-implemented (the offline ones are inline in + `classifyCardioEpochs`). They are covered by the real-night agreement test, not + by a line-for-line equivalence test. diff --git a/docs/COMBINED_BUMP_STATUS.md b/docs/COMBINED_BUMP_STATUS.md new file mode 100644 index 0000000..b17cdf6 --- /dev/null +++ b/docs/COMBINED_BUMP_STATUS.md @@ -0,0 +1,235 @@ +The RR cap, the epoch accumulator, the RR sandwich and the onset-anchored grid +(option 2, see "Grid") are implemented. + +Implemented APIs: + +- `correctRr(..., maxArtifactRun: K)` defaults to 32. `null` retains the + uncapped implementation. A hard artifact run over K is cut before its + neighbours are classified. Longer detected soft runs also split the input. + Each side has separate correction windows; dropped gap beats keep an input + class. Runs under the cap use the original batch implementation verbatim. +- `RrCorrector(maxArtifactRun:)` persists K and uses a bounded raw window for + the capped path. The uncapped path retains the original incremental kernel. + Settled output plus the snapshot equals the capped batch in the cap tests, + including one-beat folds, JSON restarts, and 10,000 junk beats. +- `NnColumn(inner)` delegates fold, combine, finish, and persistence to a + `DoubleColumn` aggregator over `Beats.nnMs`. +- `EpochRows` implements the anchored grid (below), fold, + final/provisional/all rows, `newestMs`, JSON restore (checkpoint version 4), + and `stageWindow`. It keeps final features and a + raw time tail. The provisional watermark is four minutes behind the newest + sample. Gravity references use a sliding sorted time window. Partial HR + coverage is retained so sparse cardiac evidence cannot appear complete + merely because each epoch has a mean. Carry-only epochs also remain in the + classifier's coverage denominator. +- The default advanced stager delegates to these rows. Feature + context extends beyond the requested session. The existing bounded accel + coverage and unobserved-run semantics remain in use. +- Registry counters `found.core`, `found.rr_correction`, `sleep.stage`, + `hrv.freq`, `night.rhythm` and `activity.load` are 2. A group's own algorithm + changes, including rounding and abstention, require a bump; dependency-only + changes propagate through `deps`. One bump per group covers this PR: + - `found.core`: Lomb-Scargle now subtracts the first timestamp. + - `found.rr_correction`: the artifact-run cap, already bumped. + - `sleep.stage`: retain the existing bump; its direct staging path also + computes adopted HRV statistics. Session-onset/post-gap anchoring is retained. + - `hrv.freq`: adopted HRV rounding and non-finite-time abstention. + - `night.rhythm`: adopted cosinor/circadian statistics, the cosinor singularity + gate, timestamp-gap irregular-screen pairs and invalid-time refusals. + - `activity.load`: adopted TRIMP, CTL/ATL and time-in-zone rounding. + Other counters remain 1. `found.energy` preserves ordered minute billing; + `found.staging_helpers`, `found.baseline_ewma` and `found.calibration` have no + direct output changes. `day.blocks`, `readiness.stress_illness`, + `body.spo2_temp_coverage` and `human.insights` have no direct algorithm change; + changed upstream inputs propagate through dependencies. In particular, + importing the cosinor file in a source closure does not mean an entry point + calls it: `nightlySkinTemp` does not call `tempCircadian`/`cosinor`. + Hypnogram integer accounting, SRI counts, readiness suffix scoring and + sample-order HR-dip totals preserve their existing arithmetic. + +The corrector checkpoint is version 2. Version 1 is refused, including a +pre-cap checkpoint whose already-settled output could conflict with a later +cap decision. K, including null, is explicit. The capped path's conservative +settle delay is **4H + 2K + 4 = 248 beats**, plus up to a short artifact run. +This exceeds the brief's approximately 124-beat target. It needs further work +before the latency requirement can be called satisfied. + +The LF/HF grid is unchanged. Complete, identical per-row beat windows do not +need a fixed spectral grid to give exact chunk parity. Probing fixed lower +frequencies of 1/180 Hz and 0.04 Hz did not change this night's stage agreement. + +Sandwich (owner decision 2026-10-22: absolute timestamps). `RrSandwich` +hands its inner aggregator `Beats(nnMs, timeMs)` where each NN's time is the +ABSOLUTE `rrTsMs` of the input beat it came from (the beat itself, or for a +spline-corrected beat its own timestamp), not `correctRr`'s first-beat-relative +clock. `RrCorrectionResult.nnSourceIndex` (new, optional, default empty on a +hand-built result) says which input index each NN came from; no existing field +changed. With absolute times a seam needs no clock translation, which removes +the counterexample that blocked the first attempt (an inner aggregator keyed on +one fixed time). + +State: a chunk of m beats keeps the first and last W = 2 * reach raw beats +(reach = 4H + 3K + 4 = 280 at the defaults; H = windowBeats ~/ 2, K = the cap) +and the inner state over the NN of beats [reach, m - reach), which no beat +outside the chunk can change. `combine` re-corrects only `left.tail ++ +right.head` (via `correctRr`, `nnSourceIndex` selecting the seam beats) and +feeds the newly settled NN, so state is head + tail + whatever the inner keeps. +`finish` corrects the head and tail windows (clipped exactly as the whole +series is) and combines them around a copy of the settled middle. +`addSlice` is `combine(state, fold(slice))`. A one-beat append therefore costs +a ~1,100-beat correction; use larger chunks when streaming. + +The reach is the streaming corrector's proven settle horizon (4H + 2K + 4) plus +one K for a run straddling an edge. It is not a proof for every adversarial +input: an artifact pattern alternating normal and compensatory-pair beats could +in principle carry the previous-class dependency further than one run. Tests do +not detect a reach down to about 2H (they fail at reach 6), so the margin is +conservative and unmeasured. With `maxArtifactRun: null` the reach is sized as +for the default cap and a run longer than that is not bounded-state exact. + +Grid (owner decision 2026-10-23, option 2: onset-anchored; replaces the +gap-anchored grid of 2026-10-22). The stager puts its 30 s epochs at the +detected session onset (phase 0), as before the bump. The onset is the first +valid accelerometer sample at or after the session start. A run that follows a +dropout longer than the carry (more than 61 s between valid samples) re-anchors +at its own first sample, as the pre-bump stager did. The stager reads nothing +outside [start, end): it clips samples to the session, so its labels equal the +frozen pre-bump stager (`test/onehz/support/pre_bump_stager.dart`). + +- `EpochRows({anchorMs})`: the first run's grid passes through `anchorMs`. Rows + start at the anchor, or at the epoch holding the first sample when the anchor + lies before it. `EpochRows.phaseMs` is the anchor mod 30 s. Without + `anchorMs` the first sample anchors, as before. +- `EpochRowsMemo({maxPhases = 2})`: rows keyed by phase. `rowsFor(anchorMs, ...)` + takes cumulative input. A held phase folds only samples it has not consumed. + An unheld phase, or an anchor earlier than the held rows were built for, is + built from scratch (exact). Least recently used phases beyond `maxPhases` are + dropped. `toJson` / `fromJson` carry each phase's rows and consumed counts. +- Carry-forward matches the pre-bump dense array: for features, `EpochRows` + repeats the last valid vector each second for up to 60 s (until the next real + sample). Epochs that start after a run's last real sample are classified but + hidden from `allRows` / `finalRows`. `fold(throughMs:)` states that data is + complete through a time; the stager passes the session end. +- `stageWindow` classifies whole epochs only; a sub-epoch remainder joins the + last one, as before the bump. +- Checkpoint version 4 (versions 1 to 3 are refused; 81b9b4b changed row + semantics, so version 3 rows are wrong). Version 3 added `phase` and the + pending `anchor`. A phase its rows contradict is refused. Accel is kept 61 s + longer so carry from a dropped sample still reaches a live window. +- The memo does not reuse interior rows of a larger window for a smaller + session. The stager builds rows from the clipped session, which is exact and + costs what the pre-bump stager cost. A caller that wants reuse must hand the + memo input already clipped to the session. + +Measured: frozen-night agreement is 1.0 (window starts at the first sample). +The advanced-stager still-night efficiency guard (> 0.85) passes again (phase 0 +gave 0.8588 in the earlier control). + +Tests changed: + +- `test/onehz/incremental/rr_corrector_test.dart`: expect checkpoint v2 and + reject v1; extend the capped settle-horizon bounds to the implemented delay. +- `test/properties/rr_irregular_laws_test.dart`: run the existing tests of the + uncapped pipeline layout, stored first difference, and exact 2H class delay + with `maxArtifactRun: null`. Update invalid checkpoint versions for v2. + New capped behavior remains covered by `rr_artifact_cap_test.dart`. +- `test/onehz/cardio_row_cache_test.dart`: assert the actual union of cache + keys. Wall-clock rows shared by adjoining windows now deduplicate at a seam. +- `test/onehz/sleep_honesty_test.dart`: back to the original expectations (600 + s wake, 15,000 s longest sleep run); the wall-clock grid had moved them and + the gap-anchored grid does not. +- `test/onehz/epoch_rows_test.dart`: B1 rewritten for the gap-anchored grid + (anchor at the first sample, finality of the anchor, re-anchor after a gap + of 61 s but not 60 s, gap-only clipping with an unaligned gap, the seeded + `continuingAnchorMs`); B3 boundaries are checked against the run's grid, plus + a session that starts mid-run and a hole that re-anchors. B2 is unchanged + except `whole.length >= 100` (3,000 s of data is exactly 100 epochs from the + first sample; the wall-clock grid had 101). +- `test/onehz/wallclock_agreement_test.dart`: the agreement loop steps 30 s + from the first sample (the frozen hypnogram's own grid) instead of from the + next wall-clock boundary; the boundary test checks the run grid, with run + anchors recomputed independently from the valid accel timestamps. The 0.97 + guard is unchanged. +- `test/aggregate/rr_sandwich_test.dart`: the oracle is `correctRr(..., + maxArtifactRun: K)`'s NN paired with the absolute timestamps of its + `nnSourceIndex` beats (was `nnTimesMs`); the A3 HRV/PRSA references and the + HRV origin use the same absolute times. Added: series longer than the + windows (settled middle in play; random cuts, many even chunk sizes around + W, one-beat chunks, a dense-artifact property), a bounded-size check across + series length, a longer law case, and long A3 cases. The insertion of a + definite dropout into the seam fixture stays. +- `test/onehz/rr_artifact_cap_test.dart`: `nnSourceIndex` tests (length, + order, value, dropped count, over-cap runs, tiny inputs, spline-corrected beat + is its own source). +- `test/onehz/rr_artifact_cap_test.dart`: add a regression that folds and + restores through a settled gap into a longer clean tail. It caught a lost + segment boundary after compaction; the state now persists that boundary and + whether the active gap is physiological or detected by the soft classifier. + +The existing advanced-stager efficiency assertion and the 0.97 agreement +assertion have not been weakened. With option 2 the former passes again. + +The capped latency is above target, the still-night efficiency guard fails +(see Grid), and the sandwich reach and the adversarial soft-run influence +bounds still need a proof beyond the passing prefix/oracle cases. These are +open limitations, not completed acceptance criteria. + +Benchmarks below used the supplied Dart 3.11.4 executable in JIT mode. They +are local measurements, not device timings. The new probe is reproducible via +`dart run tool/combined_bump_probe.dart [bench|rr-bench]`. + +| Workload | Accumulated/current | Batch/reference | +| --- | ---: | ---: | +| Real night, EpochRows fold + allRows versus cardioEpochFeatures | 1,937.511 ms | 1,954.044 ms | +| Same night, all 15-minute prefixes, total feature work | 2,446.084 ms | 38,535.495 ms | +| Real night, warmed capped versus uncapped correctRr | 37.871 ms | 37.873 ms | +| bench_phase0 cardioStager, current versus frozen reference | 1,735.504 ms | 1,762.683 ms | +| bench_night_nn seed 1, streaming RR versus repeated batch, 32 passes | 61.78 ms | 547.87 ms | +| Same tool, one whole-day corrector fold + snapshot versus batch | 210.42 ms | 174.75 ms | + +The one-shot corrector is about 20% slower than batch on that cold whole-day +measurement, so the blanket no-performance-regression requirement is not +satisfied, although repeated night passes are faster. + +The phase-0 benchmark reports exact frozen-reference parity. The night-NN +benchmark reports bit identity for all 32 passes. The real-night EpochRows +checkpoint is 188,121 JSON characters. The 96,712-beat corrector test measured +an 8,114-character maximum checkpoint. The 10,000-junk-beat bound passes. +No RrSandwich benchmark was taken. The benchmark rows above were measured +before the grid change (EpochRows has the same per-fold work, plus a run +lookup). + +Final full-suite summary: + +```text ++2596 ~7 -1: Some tests failed. +``` + +That is 2,596 passed, 7 skipped, and 1 failed: the epoch_rows_test B3 test +"staged on ITS OWN start", whose `checked > 2` count cannot hold for window ++777..+2111 s (the pre-bump labels there have only two boundaries). The +sandwich tests, the frozen-night agreement guard (1.0 against 0.97), the RR cap, +checkpoint regression, batch oracles, epoch-row parity, cache behavior, +registry closure, and counter tests pass. `dart analyze` reports no issues. +No commits, checkouts, or stashes were run. + +Files changed: + +- `lib/src/onehz/foundations/rr_correction.dart` +- `lib/src/onehz/foundations/rr_correction_kernel.dart` +- `lib/src/onehz/foundations/rr_correction_stream.dart` +- `lib/src/onehz/foundations/rr_sandwich.dart` +- `lib/src/onehz/sleep/advanced_stager.dart` +- `lib/src/onehz/sleep/epoch_rows.dart` +- `lib/src/onehz/registry/group_registry.dart` +- `CHANGELOG.md` +- `test/aggregate/rr_sandwich_test.dart` +- `test/onehz/cardio_row_cache_test.dart` +- `test/onehz/epoch_rows_test.dart` +- `test/onehz/incremental/rr_corrector_test.dart` +- `test/onehz/rr_artifact_cap_test.dart` +- `test/onehz/sleep_honesty_test.dart` (back to its original content) +- `test/onehz/wallclock_agreement_test.dart` +- `test/properties/rr_irregular_laws_test.dart` +- `tool/combined_bump_probe.dart` +- `docs/COMBINED_BUMP_STATUS.md` diff --git a/docs/INCREMENTAL_INTEGRATION_PLAN.md b/docs/INCREMENTAL_INTEGRATION_PLAN.md new file mode 100644 index 0000000..434bbda --- /dev/null +++ b/docs/INCREMENTAL_INTEGRATION_PLAN.md @@ -0,0 +1,168 @@ +# Incremental calculations: integration plan + +Hand-off for getting the incremental work into the app (`edge`). Read +[INCREMENTAL_MATH.md](INCREMENTAL_MATH.md) for the derivations and +[INCREMENTAL_USAGE.md](INCREMENTAL_USAGE.md) for the API. + +## Where the work is + +| Repo | Branch | Location | State | +|---|---|---|---| +| analytics | `feat/minute-energy` | this checkout | `4bd76f7` committed; later changes below uncommitted | +| edge | `feat/incremental-analytics` | `../edge.worktrees/incremental-analytics` (git worktree of `../edge`) | all incremental work uncommitted on top of `687b5bd0` | + +`687b5bd0` is a snapshot of the scheduler work that was uncommitted in +`../edge` when this started. That work has since been committed there as +`561f2838` on `ecg-taps-one-clock` (about 100 lines differ), and that branch +has moved on to `f88d230c`. `../edge` itself also has uncommitted perf tests +(`test/perf/p3_*`). Leave that checkout alone. + +The edge worktree builds against this checkout through a gitignored +`pubspec_overrides.yaml`. `pubspec.lock` is modified only by that override: +never commit it. + +Run everything through the devshell in this repo: + +```sh +nix develop /path/to/openstrap-analytics -c flutter test # in the edge worktree +nix develop -c dart test # here +``` + +Run edge tests with `TZ=UTC`. The two-device golden fails under any other +offset. + +## What is in place + +Every incremental result is tested against the existing batch function as +its oracle. In this repo, exact paths must agree within a relative error of +1e-8 (`test/onehz/support/incremental_compare.dart`). In edge, the helper is +`test/support/incremental_compare.dart`: `kExactRelTol = 1e-9` for exact +reuse, and `kApproxRelTol = 0.02` reserved for a future method that is +approximate by design. No such method exists yet. + +**Used by the app** + +- `CalculationCache`, through `DayCalculationState.evaluate`. It holds the + finished night's results (about 25 entries) and the completed windows of + the rolling HRV and breathing-rate curves. These all hit on every periodic + awake pass. +- `IncrementalMinuteMetrics`: TRIMP, strain, and daily calories, with each + minute priced once. +- `IncrementalEnmoSeries`: motion minutes. Full passes calibrate the gravity + reference exactly as `enmoSeries` does. Awake passes keep that reference. + Every field the app reads is still the batch value; only + `MotionMinute.enmo` uses the kept reference, and nothing in edge reads it. +- Edge-side running summaries (`lib/compute/day_activity_state.dart`): wake + active minutes, the activity curve, wear runs, per-minute wake HR, the + day's HR stats (smoothed max/min, mean), and the day side of HR dip + (`hrDipFromDayTotals` here). Each one adds in sample order, so it is + bit-identical to the batch reader it replaces. +- Mode selection: `PeriodicCalculationPolicy` allows reuse only while the + phone is unplugged and the causal stager reports fresh, confident `wake`. + Sleep, heavy, forced, charging and missing or stale evidence all run in + full. + +Benchmark (`INCREMENTAL_BENCH=1 flutter test test/incremental_cache_benchmark_test.dart`, +16 h awake day plus an 8 h night, six 5-minute appends): 86.2 s recomputing +everything against 2.2 s with reuse. A full pass is also no slower than +before the work: about 350 ms compute per derive in +`sleep_override_blanks_night_test`, against 445 ms at baseline. + +**Implemented here, not used by the app** + +| Incremental | Why it is unused | Where it would pay | +|---|---|---| +| `IncrementalLombScargle` | Its callers (`hrvFreq`, `cardiacCoherence`) run on the finished night or on short sessions; the night is cached whole. | A live coherence/breathing session that recomputes its spectrum as beats arrive. | +| `IntHistogram` | No existing incremental state recomputes an HR median per pass. | A rolling or trailing HR median/percentile over a long window (see the caller table in INCREMENTAL_USAGE.md). | +| `RunningMoments` | No caller recomputes a long mean/SD per pass. Night SDNN is cached whole. | Rolling windows over the awake day (daytime HRV, rolling HR variability) or baselines kept as running state. | +| `IncrementalHrvTime` | Removed from the app: the night's beats do not change while awake, so the cached result already covers it. | Daytime HRV (`_daytimeHrv`) if it becomes expensive. | + +## Steps + +1. **Land this repo's changes.** Uncommitted: + - `hrDipFromDayTotals`, which `hrDip` now shares its scoring with + - `IncrementalMinuteMetrics.sync` taking `int dayMinutes` (as + `Calories.dailyEnergy` does) + - day-duration tests + - `drift_test.dart` + - `IntHistogram` + - the first-sample time shift in `lombScargle` and its precision test + - this plan + + Commit them, open a PR, and note the new commit SHA. The incremental work + changes no algorithm output, but the `lombScargle` shift does (about 1e-6 + relative, worst case), so edge must bump `kAlgoVersion` when it pins a commit + that contains it. + +2. **Move the edge work onto the current edge branch.** Commit the worktree's + changes on `feat/incremental-analytics`, then rebase onto + `ecg-taps-one-clock`, dropping `687b5bd0` (its content is already in + `561f2838`). Expect conflicts in: + - `lib/compute/derivation_engine.dart`: `run()` gained + `calculationMode`, `_derivePreparedDay` now publishes state after + persistence, and the activity helpers were changed. + - `lib/state/app_state.dart`: `_deriveRun` asks the policy before each + light pass. `f88d230c` changed about 230 lines here. + - `lib/compute/onehz_pipeline.dart`: memoized night components and the HR + summary. + + Keep both sides' behaviour. Keep `changedOnly` from forcing a full mode: + the automatic light pass uses `changedOnly`, so mapping it to `forced` + silently disables all reuse. `test/incremental_engine_run_test.dart` + covers this path end to end. + +3. **Pin the analytics dependency.** Replace the local override with the new + analytics SHA in `pubspec.yaml`, run `flutter pub get`, and update the SHA + that `db_serve_version_and_reads_test.dart` checks. That test already + fails on this branch because the pins and `kAlgoVersion` disagree. + +4. **Verify.** In edge, the incremental tests are + `test/incremental_*_test.dart`, 8 files including the opt-in benchmark. + Then run the full suite with `TZ=UTC`. Known failures that also fail + without this work: + - `db_serve_version_and_reads_test` (pins) + - three Today-tab tests in `health/health_h2_tabs_test.dart` + - `setUpAll` of the four proof-view tests + (`proof/affected_views`, `proof/chart_key_views`, `proof/phase8_views`, + `sources/proof_views`) + - the two-device golden comparison + + `sleep_blank_review_test` and `sleep_override_blanks_night_test` can hit + the 30 s timeout under full-suite load. They pass alone. + +5. **Device check.** On a phone, during the day and unplugged, the derive log + should show light passes with low compute times. Plugged in, or overnight, + should show full passes. `DerivationEngine.debugCalculationState(day)` + exposes hits and work counters. + +## Follow-ups + +- **Medians from compacted histograms (done).** `IntHistogram` is an exact + `value -> count` summary with `percentile` and `median` bit-identical to + `percentileSorted` on the expanded list, tested against the real-night HR and + randomized data. HR has about 45 distinct values in a real night + (`test/onehz/fixtures/real_night_2026_07_onehz.csv`). No existing incremental + state recomputed an HR median per pass, so nothing was switched; candidate + callers are listed in INCREMENTAL_USAGE.md. The largest medians in a day are + the two inside `calibrateGRef`, over every valid accelerometer magnitude + (up to 86,400 values, 30,848 distinct of 32,041 in the same capture), so a + histogram does not help there. Those are off the awake path because the + reference is kept. If they ever need to be incremental, use an + order-statistic structure, not a histogram. +- **Observed max HR (pending, edge-side).** The all-time ceiling is already a + cached comparison: `LocalDb.observedHrCeiling` takes the max over one stored + value per day. The per-day value comes from `sessionHrCeiling` over each + session's samples. A finished session's ceiling never changes, so cache it + per session and sample revision. Only an open session needs its hold-window + state carried forward. +- **Batch Lomb–Scargle precision (done here; edge must bump `kAlgoVersion`).** + `lombScargle` now shifts times by the first sample. On raw epoch seconds the + trig arguments reached about 1e10 radians and the periodogram was off by up + to 4.6e-6 relative against an 80-bit reference. This changes `hrvFreq`, + `cardiacCoherence`, the respiration RSA rate and the cardio stager's LF/HF at + the 1e-6 level worst case, so edge must bump `kAlgoVersion` when it pins this + commit (and the pin must contain it; see AGENTS.md invariants 4 and 5). +- Remaining batch calculations and their constraints are listed in + INCREMENTAL_MATH.md. Several (sleep staging, naps, cycles, rank-based + statistics) cannot be made exactly incremental, because new data revises + old decisions. diff --git a/docs/INCREMENTAL_MATH.md b/docs/INCREMENTAL_MATH.md new file mode 100644 index 0000000..a72a94e --- /dev/null +++ b/docs/INCREMENTAL_MATH.md @@ -0,0 +1,513 @@ +# Incremental calculation audit + +Audited on 2026-10-03 against analytics `7334289` and the local sibling +`edge` checkout `a613d8d8`. This is a math audit and implementation design. +It does not change calculation code or scheduling. + +The best first target is the awake-day pipeline: retain minute summaries, +replace changed minutes, and accumulate their contributions. Most headline +daytime calculations then cost work proportional to new or changed data. +Nighttime processing, heavy passes, and explicit recalculation should keep +running the complete pipeline and rebuild the incremental state afterward. + +## What runs today + +This package owns deterministic math. The app owns reads, scheduling, +isolate execution, and persistence. These sibling source links assume the +local checkout layout used for this audit. + +- [`DeriveScheduler`](../../edge/lib/compute/derive_scheduler.dart) queues light + work after stored data and heavy work after capture settles. It serializes + jobs and holds them during capture, workouts, and iOS background execution. +- [`DerivationEngine.run`](../../edge/lib/compute/derivation_engine.dart) selects + one freshness-critical day for a light pass and the pending span for a heavy + pass. Both use the same per-day derivation, followed by baseline refresh and + the cross-day pipeline when a day completes. “Light” currently limits days, + not calculations within a day. +- [`onehz_pipeline.dart`](../../edge/lib/compute/onehz_pipeline.dart) recomputes + sleep RR correction, HRV, resting HR, respiration, stress, whole-day rhythm + screening, readiness, awake-minute TRIMP, calories, curves, and other detail. + The coordinator's second isolate computes additional day blocks. +- [`crossday_pipeline.dart`](../../edge/lib/compute/crossday_pipeline.dart) builds + history-dependent results. A daytime update can therefore trigger work on + unchanged nightly history too. + +The current input stream is not append-only. The app stores canonical readings +with replacement by timestamp. History can arrive late, sources can change, +and sleep bounds can be edited. A timestamp cursor alone cannot prove that +cached contributions are still correct. + +## Calculation inventory + +These tables group computational entry points and their shared math. Data +classes, serialization, and barrel exports are excluded. “Incremental” means +the same mathematical result, subject to floating-point differences and the +same validity gates. It does not promise bit-for-bit identical accumulation. + +### Foundations and shared statistics + +| Calculation and source | Current math | Incremental design | +|---|---|---| +| Mean, SD, population SD, ordinary z; [`util.dart`](../lib/src/onehz/util.dart) | `mean = sum/n`; sample variance `sum((x-mean)^2)/(n-1)`; population variance divides by `n`; `z=(x-mean)/SD` | Count, mean, centered sum of squares. Append, remove, and merge summaries. Preserve each divisor and null case. | +| Percentiles, median, MAD, robust z; [`util.dart`](../lib/src/onehz/util.dart), [`baseline.dart`](../lib/src/onehz/foundations/baseline.dart) | Sort; interpolate rank `p*(n-1)/100`; scaled MAD `1.4826*median(abs(x-median(x)))`; MDC `1.96*sqrt(2)*typicalError` | Exact ordered values or histograms. MAD changes when the median moves. Recompute small daily baselines only when their membership changes. | +| Sampling cadence; [`util.dart`](../lib/src/onehz/util.dart) | Median positive timestamp gap, maximum supported cadence, and fraction of gaps within a factor of two of the median | Ordered gap counts can update; a changed cadence or eligibility can invalidate all duration-dependent results. | +| OLS slope/fit; [`util.dart`](../lib/src/onehz/util.dart) | `slope=sum((x-mx)*(y-my))/sum((x-mx)^2)`; `intercept=my-slope*mx` | Bivariate centered moments, or additive sums with careful numerical handling. | +| Theil–Sen slope; [`util.dart`](../lib/src/onehz/util.dart) | Median of all valid pairwise slopes | One new point adds up to `n` slopes. Exact storage grows quadratically; keep batch evaluation on small aggregate series. | +| Lomb–Scargle; [`util.dart`](../lib/src/onehz/util.dart) | Mean-subtracted sinusoidal projection at every grid frequency, phase rotation, one-sided PSD normalization, then band integration | Exact additive trigonometric sums per fixed frequency. Detailed derivation below. | +| Average ranks, normal p-values, Benjamini–Hochberg q-values; [`util.dart`](../lib/src/onehz/util.dart) | Tie-averaged ranks; numerical normal-tail approximation; sorted p-values with reverse running minimum | New values can change old ranks; a changed test family can change every q-value. Cache unchanged inputs, then rebuild the affected ranking/test family. | +| `correctRr`; [`rr_correction.dart`](../lib/src/onehz/foundations/rr_correction.dart) | Successive RR differences; centered local medians and quartile thresholds; beat classification; compensatory-pair reconciliation; spline correction of isolated artifacts, dropping artifact runs; cumulative/reanchored beat times | Cache stable cleaned regions only after proving their dependencies. New beats change centered neighborhoods and unresolved spline anchors. Recompute the affected suffix; retain raw beats and clock checkpoints. | +| `Baselines.update/deviation`; [`ewma_baselines.dart`](../lib/src/onehz/foundations/ewma_baselines.dart) | Winsorized EWMA center, EWMA absolute deviation against the old center, early adaptation, outlier/absence/staleness gates; z/delta/ratio | Already has a serializable one-night state update. Save state before the provisional night so reruns replace that night rather than folding it twice. | +| `inverseVarianceFuse`; [`fusion.dart`](../lib/src/onehz/foundations/fusion.dart) | `sum(value/variance)/sum(1/variance)`; fused variance `1/sum(1/variance)` over trusted inputs | Two sums and provenance. Usually only a few channels, so direct reevaluation is already cheap. | + +### Cardiac, load, and workout calculations + +| Calculation and source | Current math | Incremental design | +|---|---|---| +| `hrvTime`, `nnDiffAcf1`; [`hrv_time.dart`](../lib/src/onehz/clinical/hrv_time.dart) | RMSSD `sqrt(sum(dRR^2)/pairs)`; pNN50 `100*count(abs(dRR)>50)/pairs`; SDNN; SDANN as SD of 5-minute means; SDNN-index as mean of 5-minute SDs; pooled difference lag-1 autocorrelation controls jitter refusal | NN moments, difference moments, threshold counts, pair-product summaries, last contiguous beat/difference, and per-bin summaries. Correct seams and mutable bins explicitly. | +| `nocturnalRmssd`, `sleepSessionWindowedRmssd`; same file | First takes median of eligible 5-minute RMSSDs and a pooled jitter gate; second takes mean of locally cleaned, gap-aware 5-minute RMSSDs | Cache per-window outputs under fixed bounds and masks; replace the open/changed window. Preserve median versus mean and each cleaner's seam rule. | +| `hrvFreq`; [`hrv_freq.dart`](../lib/src/onehz/clinical/hrv_freq.dart) | Welch averages of band-integrated Lomb–Scargle spectra, 50% overlap; segment length `10/lowestBandHz`; LF/HF and normalized units; artifact/length gates | Cache completed segments per band, sum powers and count accepted segments. Global quality gates still apply at publication. | +| `decelerationCapacity`, `accelerationCapacity`; [`prsa.dart`](../lib/src/onehz/clinical/prsa.dart) | Select capped RR-change anchors; average their aligned profiles; capacity `[X(0)+X(1)-X(-1)-X(-2)]/4` | Sum profiles and anchor count. Delay finalizing anchors until all required right-hand context and the current loop's eligibility boundary have arrived. | +| `nocturnalRhr`, `hrDip`; [`nocturnal.dart`](../lib/src/onehz/clinical/nocturnal.dart) | Lowest eligible 30-minute rolling HR mean and p1 HR; dip `100*(dayMean-nightMean)/dayMean` | RHR already uses a sliding sum inside a batch call. Retain its queue, counts, best completed window, and percentile structure across calls. Dip needs two sums/counts. Cadence and sleep-bound changes invalidate dependencies. | +| `baevskyStressIndex`; [`stress_si.dart`](../lib/src/onehz/clinical/stress_si.dart) | 50 ms RR histogram; `SI=AMoPct/(2*modeSeconds*rangeSeconds)` in up-to-256-beat windows stepped by 128; median SI and component summaries | Cache stable full windows, recompute the partial tail, maintain histogram/min/max and ordered window outputs. Preserve mode tie ordering. | +| `irregularBeatScreen`; [`irregular_rhythm.dart`](../lib/src/onehz/clinical/irregular_rhythm.dart) | `SD1=SD(dRR)/sqrt(2)`; `SD2=sqrt(2*SDNN^2-SD1^2)`; pNNx; aggregate gates plus fraction of independently positive time windows | Moments and difference counts per valid input run/window, plus scored-window counts. Preserve this function's actual adjacency contract, which differs from `hrvTime`. | +| `cardiacCoherence`; [`cardiac_coherence.dart`](../lib/src/onehz/clinical/cardiac_coherence.dart) | Find spectral peak; power around peak divided by remaining power; score `100*ratio/(1+ratio)` | Cached spectral sums if grid fixed. Current low grid endpoint depends on span, so rebuild when its frequency grid changes. | +| `banisterTrimp`, `StrainScorer`, `trimpStrain`; [`load_trimp.dart`](../lib/src/onehz/clinical/load_trimp.dart) | `sum(duration*x*c*exp(b*x))`, `x=clamp((HR-RHR)/(HRmax-RHR),0,1)`; coefficients differ by sex; legacy 0–100 strain is a log map | Cache raw TRIMP contributions under fixed anchors. Minute-input and timestamp-input paths have different duration semantics. Replace the last sample's provisional duration. | +| `baselineTrimp`, `strainScore/Metric`, `dailyQuietWakingHrr`; same file | Quiet overhead `wakeMinutes*q*c*exp(b*q)`; log-map net TRIMP onto 0–21; quiet level is median HRR subject to coverage/range gates | Accumulate raw TRIMP and wake-minute count, then reevaluate the scalar map. Changing quiet level changes the baseline; changing HR anchors changes every TRIMP contribution. | +| `ctlAtlTsb`; same file | Seed with first `primeDays` mean, then CTL/ATL exponential recurrences; TSB `CTL-ATL` | Persist state and seed count. Replace today's load from yesterday's checkpoint. A history edit requires replay after the edit. | +| `illnessCusum`; [`illness_cusum.dart`](../lib/src/onehz/clinical/illness_cusum.dart) | Prior-calendar-day robust RHR z; `C=max(0,C+z-k)`; persistence/recovery state and gap resets | Save prior-day state and rolling baseline values. Score each new day once; revisions replay forward. | +| `readinessLnRmssd`; [`readiness_lnrmssd.dart`](../lib/src/onehz/clinical/readiness_lnrmssd.dart) | Prior-night lnRMSSD mean/SD standardization and short rolling summaries | Small rolling moments under the exact history convention. Usually cheaper to memoize by history revision. | +| `cosinor`; [`cosinor.dart`](../lib/src/onehz/clinical/cosinor.dart) | OLS on `y=M+beta*cos(w*t)+gamma*sin(w*t)`; amplitude/phase; residual R² and adjusted R² | Additive 3×3 normal matrix, 3-vector, `sum(y^2)`, and count. Refit a constant-size system. | +| Heart-rate zones; [`hr_zones.dart`](../lib/src/onehz/workout/hr_zones.dart) | Threshold bands; timestamp-based time credit capped by measured median cadence | Per-zone durations, last timestamp/label, gap statistics. Reclassify on zone-bound changes; replace tail credit. | +| Observed HR ceiling; [`observed_max_hr.dart`](../lib/src/onehz/workout/observed_max_hr.dart) | Highest sustained window minimum HR, confirmed by a short motion burst; gap and bounded-span guards | Retain bounded candidates and sliding minima/motion summaries. Preserve motion corroboration and tie behavior; a plain maximum is insufficient. | +| `Calories.dailyEnergy/minuteEnergy/hourlyRollup/estimateBoutCalories`; [`calories.dart`](../lib/src/onehz/workout/calories.dart) | Affine Keytel HR energy rate with caps; HRR flex gate; surplus above basal rate; below gate, measured walking cadence gives `(MET-1)*basal`; basal total prorated by caller duration | Cache each minute's bill and provenance; sum active, walking, coverage and hourly contributions. Reprice affected contributions when profile, anchors, cadence spans, or sleep mask change. | +| `Calories.metFromCadenceSpm`; same file | Below 100 spm absent; otherwise `min(6,3+0.1*(spm-100))` | Already constant work. | +| `autoDetectWorkouts`, `AutoWorkoutDetector`; [`auto_detect.dart`](../lib/src/onehz/workout/auto_detect.dart) | Sustained HR elevation, brief-dip/gap rules, merging, motion confirmation, saved-span exclusion; bout mean/max | Keep the active candidate and mergeable tail; finalize only once merge look-ahead is resolved. Saved-session edits invalidate overlap decisions. | +| `hrRecovery`; [`hr_recovery.dart`](../lib/src/onehz/workout/hr_recovery.dart) | Peak near end minus median near +60 s; candidate-tau least-squares fits `a+b*exp(-t/tau)` with residual and boundary gates | Small event window. Cache fixed-tau basis sums if repeated fitting matters; otherwise run once when tail settles. Refit if end time changes. | +| `vo2maxSubmaxEstimate`; [`vo2max.dart`](../lib/src/onehz/clinical/vo2max.dart) | ACSM speed/grade submax VO2; `VO2max=3.5+(VO2submax-3.5)/HRR` with bout gates | Already scalar. Cache/revise upstream bout summaries. | +| Sport classification; [`sport.dart`](../lib/src/onehz/workout/sport.dart) | Injectable classifier; default returns the untyped detected label | Cache by settled bout and classifier-input revision, not by day timestamp. | + +### Motion, sleep, and respiration + +| Calculation and source | Current math | Incremental design | +|---|---|---| +| `enmoSeries`, `calibrateGRef`; [`enmo.dart`](../lib/src/onehz/motion/enmo.dart) | Magnitude `sqrt(ax²+ay²+az²)`; auto gravity from medians of locally still magnitudes; ENMO mean `max(0,magnitude-gRef)`; minute mean absolute deviation; dynamic magnitude after trailing per-axis mean subtraction | Retain the trailing gravity queue/sums and minute summaries. Closed-minute MAD can run once on its samples. Auto-calibrated `gRef` is a changing dependency; it cannot silently become a fixed constant. | +| `relativeIntensityBands`; same file | Moving-value p50/p75/p90 thresholds or caller-provided frozen cuts, then class counts | Fixed cuts permit incremental labeling/counts. Whole-record cuts can change every old label. | +| `pedometer`, `calcSteps`, `livePedometer`; [`steps.dart`](../lib/src/onehz/motion/steps.dart) | High-rate moving average, local extrema, adaptive thresholds, candidate timing/regulation, calibrated count/cadence and confidence | Reuse existing independent minute/buffer boundaries first. Arbitrary streaming must preserve filter/extremum/regulation state and the whole-input dynamic reference; chunk totals are not freely additive. | +| `calibrateCadence`, floor helpers, `dailyActiveMinutes`; same file | Calibration update; robust dynamic-floor history; count gate-passing covered minutes in clock-consecutive runs of minimum length | Freeze only through existing floor policy. Cache minute gates and current run; credit the whole run when it first qualifies, then one minute per extension. | +| `staticTilt`, `positionSeries`; [`orientation.dart`](../lib/src/onehz/motion/orientation.dart) | Gravity direction gives pitch/roll; mean rotation and magnitude jitter determine stillness/coverage | Cache completed fixed bins and carry the boundary pair. A changed average orientation changes angles, so calculate once per affected bin. | +| `branchedEnergyFusion`; [`energy_fusion.dart`](../lib/src/onehz/motion/energy_fusion.dart) | Normalize acceleration/HR; branch by motion and HR validity/intensity; integrate relative load over time | Cache per-point contributions and boundary duration with fixed anchors and calibration. | +| `immobilityMask`, `vanHeesSleepWindow`; [`van_hees.dart`](../lib/src/onehz/sleep/van_hees.dart) | z-angle `atan2(z,sqrt(x²+y²))`; centered median smoothing; forward sustained angle-change test; bout merging and main-window selection | Intermediate angles and stable windows can cache, but forward windows revise the tail and main-window selection can change. Keep full sleep passes. | +| `segmentSleep`, `AdvancedSleepStager`, `cardioStager`, legacy staging wrappers; [`segment.dart`](../lib/src/onehz/sleep/segment.dart), [`advanced_stager.dart`](../lib/src/onehz/sleep/advanced_stager.dart), [`cardio_stager.dart`](../lib/src/onehz/sleep/cardio_stager.dart), [`stager.dart`](../lib/src/onehz/sleep/stager.dart) | Candidate/window rules; local cardiac/motion features; whole-night robust distributions; weighted stage axes and temporal consolidation; TIB/TST/SOL/WASO/REM-latency accounting | Cache finished-night results when inputs are unchanged. Changing distributions or session bounds can relabel old epochs. Stage counts alone do not make the detector incremental. | +| `hrLedSleepWindow`; [`hr_fallback.dart`](../lib/src/onehz/sleep/hr_fallback.dart) | Sustained nocturnal HR dip against waking/reference HR with confirmation guards | Cache intermediate bins/candidates, but changed baseline or window selection invalidates decisions. Full sleep mode. | +| `CausalStager.observe`; [`causal_stager.dart`](../lib/src/onehz/sleep/causal_stager.dart) | Past-only features, expanding references, quality/age gates, serializable next state | Existing online interface. Keep its separate estimate contract; substituting it for offline staging changes results. | +| `detectNaps`; [`nap.dart`](../lib/src/onehz/sleep/nap.dart) | Immobility bouts outside main sleep, masks and bout chaining, HR dip against awake baseline, independent TST/TIB and quality | Cache primitives, but changed main sleep, awake baseline, or new bout completion can revise older nap candidates. Batch until dependencies are stable. | +| `detectSleepCycles`, `sleepCyclesMetric`; [`cycles.dart`](../lib/src/onehz/sleep/cycles.dart) | Per-minute filtered RMSSD, centered smoothing, whole-record population z-score, prominence peaks with tallest-first distance pruning | Cache minute RMSSD. Whole-record z/prominence and competition between peaks can revise old cycles. Keep complete cycle detection. | +| `nightHrvShape`; [`night_hrv_shape.dart`](../lib/src/onehz/sleep/night_hrv_shape.dart) | Per-bin gap-aware RMSSD and sampling bands; means of first/last third and their ratio | Cache bins, reevaluate thirds when night length changes. | +| `circadianNonparametric`; [`circadian_np.dart`](../lib/src/onehz/sleep/circadian_np.dart) | IS phase-profile variance / total variance; IV successive-difference energy / variance; circular M10/L5 windows; RA `(M10-L5)/(M10+L5)` | Overall moments, adjacency energy, phase sums/counts; recompute the small average-day profile and its circular windows. | +| `phillipsSri`; [`sri.dart`](../lib/src/onehz/sleep/sri.dart) | `200*validEpochAgreement/validCases-100` across adjacent clock-aligned days | Cache each adjacent-day pair's agreement/cases. Revising day `d` changes pairs `d-1,d` and `d,d+1`. Thin pairs still enter the total even if omitted from the displayed breakdown. | +| `rsaRespRate`, `riivRespRate`, `fuseRespRate`; [`resp_rate.dart`](../lib/src/onehz/respiration/resp_rate.dart) | Window spectra/peaks, sampling-ceiling and quality gates, consensus, then trusted estimate fusion | Cache completed window spectra/peaks. RSA adapts segment length on short records and grid to each segment's span/Nyquist; only fixed completed windows have reusable fixed-grid state. | +| `breathingRateVariability`; [`brv_trend.dart`](../lib/src/onehz/respiration/brv_trend.dart) | Mean, sample SD, coefficient of variation, Theil–Sen slope over resolved respiratory windows | Moments update cheaply; exact robust slope adds pairwise work. Small series can stay batch. | +| `cvhrApneaScreen`, `cvhrPersonalDistribution`; [`cvhr_apnea.dart`](../lib/src/onehz/respiration/cvhr_apnea.dart) | Gap-separated resampling, local smoothing/detrending, prominence/width/spacing cycle rules, cycles per analyzed hour; across-night weighted rates/quantiles | Cache stable segments and weighted count/duration sums; prominence references and unfinished cycles still need reevaluation. Keep nightly detection batch initially. | +| `relativeOdi`; [`relative_odi.dart`](../lib/src/onehz/respiration/relative_odi.dart) | Relative red/IR AC-to-DC ratio, rolling baseline and duration-gated drops per analyzed hour | Rolling windows/event state are possible in principle. The audited app permanently refuses this output for its channel data. Do not reactivate it as an optimization. | +| `cardiopulmonaryCoupling`; [`cpc.dart`](../lib/src/onehz/sleep/cpc.dart) | Always absent; withdrawn because no independent respiration channel is supplied | No expensive calculation to optimize. | + +### Wellness and human-history calculations + +| Calculation and source | Current math | Incremental design | +|---|---|---| +| `readinessComposite`; [`readiness_composite.dart`](../lib/src/onehz/wellness/readiness_composite.dart) | Robust z or ordinary-z fallback, quantization/input gates, weighted signed z, `100/(1+exp(-weightedZ/weightSum))` | Cache prior-night baseline summaries by date/membership revision, then recompute the small scalar blend. Never fold today into its own reference. | +| `multivariateAnomaly`; [`anomaly.dart`](../lib/src/onehz/wellness/anomaly.dart) | Robust feature centers/scales, correlation matrix with ridge, Mahalanobis distance `sqrt(zᵀR⁻¹z)`, significance/persistence gates | Reuse unchanged baseline results. Pearson correlation is invariant to the positive affine standardization, so raw complete-row covariance summaries can replace repeated standardized-row work; retain feature eligibility and robust centers/scales. | +| `cusumChangePoints`, `segmentChangePoints`; [`changepoint.dart`](../lib/src/onehz/wellness/changepoint.dart) | Online robust pre-regime z with two accumulators/resets; offline greedy binary segmentation using prefix-sum SSE, global-variance/log-length penalty and effect gates | Online state plus exact robust baseline; replay revisions. Offline prefix sums extend cheaply, but best splits and penalty change globally, so rerun segmentation. | +| `nightlySkinTemp`, `tempCircadian`; [`temp_circadian.dart`](../lib/src/onehz/wellness/temp_circadian.dart) | Settled mean relative to night's median/family band; motion/ambient masking, median centering, cosinor and missing-aware phase variance/adjacent differences | Cache raw bins and masks; changing median can change settled membership. Cosinor sums and centered variance can update algebraically under unchanged masks. Preserve device units and missing epochs. | +| `tempIllnessFlag`, `menstrualCoverline`; [`temp_health.dart`](../lib/src/onehz/wellness/temp_health.dart) | Prior-night robust z and consecutive-night/luteal rules; coverline is prior-six maximum followed by three above-threshold nights | Checkpoints and small queues. Coverline is currently uncalled in the app and needs a defensible sensor-unit threshold before use. | +| `cycleLengthSeries`; [`cycle_lengths.dart`](../lib/src/onehz/wellness/cycle_lengths.dart) | Sort/deduplicate logged onsets, interval lengths, absolute successive length differences and extremes | Append updates last interval and difference. Inserting/editing an old date revises neighboring intervals and extrema. | +| `sleepDebt`; [`sleep_regularity.dart`](../lib/src/onehz/human/sleep_regularity.dart) | `p75(freeNightSleep)-median(recentSleep)`; absent debt without free nights | Ordered short histories. This implementation is a signed comparison, not a cumulative sum of nightly deficits. | +| `socialJetlag`, `chronotype`; [`circadian_lifestyle.dart`](../lib/src/onehz/human/circadian_lifestyle.dart) | Circular medians via resultant-anchored unwrapping; shortest-arc free/work difference; MSF sleep-duration correction | Sin/cos sums update circular mean anchor; median still needs ordered/reunwrapped history when anchor changes. Cache small nightly histories. | +| `alcoholNightFlag`, `roughNight`; [`event_detection.dart`](../lib/src/onehz/human/event_detection.dart) | Robust baseline deltas/MDC comparisons, count directional signs, derive state and optional hypothesis band | Cache reference summaries; constant-size reevaluation when nightly inputs change. | +| `percentileOfYou`, `personalRecord`; [`percentile_of_you.dart`](../lib/src/onehz/human/percentile_of_you.dart) | Rank `100*(below+0.5*equal)/n`; prior extreme plus MDC | Ordered values and extrema; preserve ties and history exclusion. | +| `glassBoxReadiness`; [`readiness_glassbox.dart`](../lib/src/onehz/human/readiness_glassbox.dart) | Deprecated baseline/percentile driver breakdown | Memoize by input revision; do not introduce a second canonical readiness score. | +| `sleepNeed`, `sleepPerformance`, `recommendedBedtime/Wake`, `strainTarget`; [`coaching.dart`](../lib/src/onehz/human/coaching.dart) | Need `clamp(base+debt+45min*strain/21-naps,6h,11h)`; performance ratio; bed/wake arithmetic with efficiency; recovery-band/load-ratio rules | Already constant-size arithmetic. Trigger only on changed upstream values. | +| `journalCorrelations`, `journalNumericCorrelations`; same file | Tag-group mean differences/pooled SD/effect size; numeric rank correlation and robust slopes; deterministic permutation significance and family-wide q-values | Group moments can update; ranks, permutation distributions, and q-values can change globally. Cache the full result under input revision and rerun on relevant log/outcome changes. | +| `scanAssociations`; [`associations.dart`](../lib/src/onehz/human/associations.dart) | Calendar/lag alignment, weekday-median adjustment, ranks, Spearman correlations, block permutations, contrast medians, FDR and redundancy filtering | Keep batch evaluation. New rows change alignment, adjusted old values, ranks and the test family. Reuse unchanged scan inputs/results rather than incrementing a final rho. | +| `weekdayEffect`; [`weekday_effect.dart`](../lib/src/onehz/human/weekday_effect.dart) | Weekday medians, tied-rank Kruskal–Wallis statistic, deterministic permutation tests and peak/trough effect gate | Small daily histories; recompute when the underlying nightly series changes. | +| `sessionMorningEffects`; [`session_cost.dart`](../lib/src/onehz/human/session_cost.dart) | Following morning minus its prior robust baseline, per-session-type median deltas and MDC | Cache paired morning contributions and per-type ordered values; history/session edits invalidate affected pairs. Preserve the current positional lookback. | +| `overreachingConjunction`; [`overreaching_conjunction.dart`](../lib/src/onehz/human/overreaching_conjunction.dart) | Acute/chronic load ratio and count of recent RHR values above baseline plus half-scale gate | Cache short history summaries; reevaluate the conjunction when its inputs change. | +| `alertnessForecast`; [`alertness_forecast.dart`](../lib/src/onehz/human/alertness_forecast.dart) | Exponential sleep/wake process, circadian/ultradian cosines, decaying inertia; normalize forecast by whole-horizon extrema and select lowest 2-hour window | Exact process recurrence exists; the forecast is small and can be cached as a whole by sleep/nap/phase inputs. Horizon changes can rescale every published point. | + +The old algorithm index contains stale entries: `sleepAccounting` is gone, +`accounting.dart` supplies the shared enum, CPC is withdrawn, and the implemented +sleep-debt calculation is the percentile-minus-median comparison above. + +## Exact update rules + +### 1. Cache contributions before nonlinear output transforms + +For `A = sum(f(x_i; theta))`, where `theta` is unchanged: + +```text +A_new = A_old + sum(f(new inputs; theta)) +A_replaced = A_old - f(old input; theta) + f(replacement; theta) +``` + +Cache raw contributions, counts, and quality evidence. Reevaluate logarithms, +ratios, clamps and publication gates from those summaries. Adding rounded or +clamped scores loses information. + +If input is a minute's mean HR, calculate the minute mean first: +`f(mean(HR samples))` generally differs from `mean(f(HR samples))`. +A partially filled minute therefore replaces one contribution as it grows. + +### 2. Stable means and variance + +Retain `n`, mean `mu`, and `M2 = sum((x-mu)^2)`. On adding `x`: + +```text +n' = n + 1 +d = x - mu +mu' = mu + d/n' +M2' = M2 + d*(x-mu') +``` + +On removing `x`, for `n > 1`: + +```text +n' = n - 1 +mu' = mu + (mu-x)/n' +M2' = M2 - (x-mu)*(x-mu') +``` + +Reset when removing the only point. Sample SD is `sqrt(M2/(n-1))`; +population SD is `sqrt(M2/n)`. Preserve absence for insufficient counts. +These centered updates avoid cancellation in `sum(x²)-sum(x)²/n`. + +For exact medians/percentiles, an ordered multiset with subtree counts supports +insertion/removal and rank selection without sorting the whole history. The +baseline median absolute deviation still needs distances from the *current* +median; storing only yesterday's median and MAD loses that information. +Small 14–28-night histories are cheap enough to rebuild when changed. + +For disjoint summaries A and B, let `d=muB-muA`, `n=nA+nB`: + +```text +mu = muA + d*nB/n +M2 = M2A + M2B + d²*nA*nB/n +``` + +This permits merging fixed bins. Pair-dependent metrics additionally need +boundary state; merging variance summaries alone does not recover RMSSD. + +### 3. RMSSD, pNNx, and the autocorrelation gate + +For each eligible successive difference `d`, cache difference count `p`, +`D1=sum(d)`, `D2=sum(d²)`, and threshold-exceedance counts. + +```text +RMSSD = sqrt(D2/p) +pNNx = 100*over/p +RMSSD' = sqrt((p*RMSSD²+d²)/(p+1)) +``` + +Across chunk boundaries, add a difference only when the two beats satisfy +the caller's actual adjacency/gap rule. Across 5-minute bins, apply the +function's bin policy; do not add a cross-bin pair to an estimator that +currently excludes it. + +The lag-1 difference autocorrelation can also avoid rescanning old differences. +For adjacent differences within the same run retain: + +```text +K = number of adjacent-difference pairs +P = sum(d_previous*d_current) +E = sum(d_previous+d_current) +m = D1/p +ACF1 = (P-m*E+K*m²) / (D2-D1²/p) +``` + +This algebra matches `nnDiffAcf1`, including its use of a global mean across +runs and a numerator that omits seam pairs. Retain the 30-difference minimum +and constant-series null result. Use centered/compensated accumulation or +rebuilding where cancellation matters. Reevaluate refusal and confidence +after updates; an old accepted RMSSD can become refused. + +### 4. TRIMP and strain + +With fixed resting HR `R`, maximum `H`, coefficients `c,b`, and time weight `dt`: + +```text +x = clamp((HR-R)/(H-R), 0, 1) +f(x) = c*x*exp(b*x) +TRIMP = sum(dt*f(x)) +baseline = wakeMinutes*f(q) +net = TRIMP-baseline +u = clamp(net/400, 0, 1) +strain = 21*log(1+14*u)/log(15) +``` + +For net at or below zero, current strain is zero. Awake minute count is part +of the state because additional quiet time changes the subtraction too. +The separate legacy 0–100 map remains a separate API. + +For timestamp-weighted scoring, the last sample provisionally gets median +cadence. Arrival of the next sample replaces that duration with the capped +actual gap, then gives the new tail its provisional credit. Updating the tail +can require subtraction. A changed median cadence can affect all durations. +The minute-input TRIMP path used by the audited day pipeline treats each +supplied minute as one minute and has no such timestamp-tail correction. + +Calculus identifies why anchor changes require reevaluation. Inside the +unclamped HRR interval: + +```text +f'(x) = c*exp(b*x)*(1+b*x) +dx/dR = (HR-H)/(H-R)² +dx/dH = -(HR-R)/(H-R)² +``` + +An anchor change affects old samples, and clamp crossings change which +derivative applies. A first-order Taylor update is an approximation. Keep +exact recalculation for changed anchors. A discrete HR or minute-mean +histogram can reprice contributions exactly by unique value when weights +and masks are also retained. + +Likewise, inside the strain map's unclamped interval, for `a=14/400`: + +```text +ds/dnet = 21*a / (log(15)*(1+a*net)) +``` + +Use this to understand sensitivity, not to accumulate score increments. +Reevaluating one logarithm is already constant work. + +### 5. Calories and zones + +For fixed profile and anchors: + +```text +flexHR = RHR + 0.40*(HRmax-RHR) +B = max(0, 10*kg + 6.25*cm - 5*age + sexConstant)/1440 +K(HR) = 60*max(0, cHR*min(HR,HRmax)+cWeight*kg+cAge*age+c0)/251.04 +``` + +`B` is basal kcal/minute. A minute at/above flex bills `max(0,K-B)`. +Otherwise, eligible measured cadence bills `(MET-1)*B`; otherwise it adds +no active surplus, with the existing per-minute abstention/provenance. +Walking is included in active total, so do not add it a second time. +Daily total is caller-prorated basal plus active. + +Retain per-minute contributions, hourly sums, and coverage counts. Zones +similarly sum time by the assigned zone. The final display is cheap; +profile/gate/zone/source changes require repricing affected minutes. + +### 6. Motion without rescanning a day + +The trailing gravity remover already updates per-axis sums inside one call: +`axisSum += incoming - expired`. Retain its queue between calls. Its output +depends on real timestamps and excludes samples at/older than the window +boundary; preserve that boundary exactly. + +For fixed gravity reference `g`, ENMO sum is additive. With an ordered set +of minute magnitudes, reference changes can also be handled exactly: + +```text +sum(max(0,m-g)) = sum(m where m>g) - g*count(m>g) +``` + +The minute MAD here means *mean* absolute deviation, unlike the baseline +MAD that means *median* absolute deviation. Because its center moves on +every append, absolute deviations cannot simply be added. If `mu=sum(m)/n`, +`k=count(m<=mu)` and `L=sum(m where m<=mu)`: + +```text +meanAbsoluteDeviation = 2*(k*mu-L)/n +``` + +An ordered structure with prefix counts/sums supports this query. For roughly +60 samples per minute, retaining the minute's magnitudes and recomputing only +that minute is simpler and likely preferable to a new tree. + +For movement bouts of minimum length `L`, retain current run length `r`. +Credit nothing below `L`, credit all `L` minutes when the run reaches `L`, +then one minute per extension. Missing/uncovered/nonpassing minutes break +the run. Replacing a past minute may split/merge a run, so recompute the +connected affected region rather than adjusting one boolean count. + +### 7. Lomb–Scargle can update exactly on a fixed grid + +This is the largest algebraic opportunity beyond simple sums. For each fixed +frequency `f`, set `w=2*pi*f`. Retain: + +```text +C = sum(cos(w*t)) S = sum(sin(w*t)) +CC = sum(cos(w*t)²) SS = sum(sin(w*t)²) +CS = sum(cos(w*t)*sin(w*t)) +YC = sum(y*cos(w*t)) YS = sum(y*sin(w*t)) +``` + +Also retain global count, `sum(y)`, variance state, and time extrema. Every +new point adds to these sums. The current implementation's changing mean +and phase can be recovered without visiting old samples: + +```text +mu = sum(y)/n +phi = 0.5*atan2(2*CS, CC-SS) # phi = w*tau +a = cos(phi) +b = sin(phi) +Uc = YC-mu*C +Us = YS-mu*S + +cNum = a*Uc+b*Us +sNum = a*Us-b*Uc +cDen = a²*CC+b²*SS+2*a*b*CS +sDen = a²*SS+b²*CC-2*a*b*CS + +PSD(f) = span/(n-1) * (cNum²/cDen+sNum²/sDen) +``` + +Preserve the source's zero-frequency, zero-denominator, too-few-points, +zero-variance and zero-span cases. PSD still has the source's physical units. +Band integration remains the same rectangular sum over its exact grid. + +For `F` frequencies, adding `deltaN` samples costs `O(deltaN*F)` and rendering +the spectrum costs `O(F)`, instead of `O(N*F)` each pass. Cached state costs +`O(F)` per active spectral window. A sliding window also needs outgoing +contributions or the raw samples to subtract them. + +`hrvFreq` already uses fixed-size, overlapping Welch segments per band. +Compute completed segments once, retain accepted power sums/counts, and +process only newly complete segments. This can save most repeated nightly +spectral work even without implementing streaming spectral sums. + +RSA and cardiac coherence have adaptive grids. Preserve each existing grid +choice: rebuild when it changes, or cache completed-window outputs. Choosing +a new universal fixed grid would change the estimator and needs separate +accuracy evaluation. Use a stable time origin; reanchoring times requires +rotating/rebuilding the trigonometric summaries, not relabeling timestamps. + +### 8. Linear fits and exponential processes + +Cosinor is linear in its three fitted coefficients. With design vector +`v=[1,cos(w*t),sin(w*t)]`, retain: + +```text +A = sum(v*vᵀ) +b = sum(v*y) +Q = sum(y²) +beta = solve(A,b) +SSE = Q-2*betaᵀ*b+betaᵀ*A*beta +SST = Q-sum(y)²/n +``` + +At the exact least-squares solution, SSE simplifies to `Q-betaᵀ*b`. +The matrix has fixed size. Amplitude, phase, R², adjusted R² and their gates +are then scalar calculations. Use centered sums or incremental QR if +conditioning/cancellation warrants it; compare to the current residual pass. + +Each recovery-fit tau candidate similarly uses fixed basis `e=exp(-t/tau)`. +Retain `n,sum(e),sum(e²),sum(y),sum(e*y),sum(y²)` and solve the two-parameter +fit. Its complete residual sum follows from those sums. Keep the current +candidate grid, tie selection, amplitude, boundary and residual gates. + +The alertness process solves `dS/dt=-(S-target)/tau`: + +```text +S(t+dt) = target + (S(t)-target)*exp(-dt/tau) +``` + +This is an exact recurrence over a constant-target interval, not an Euler +approximation. Follow the current time-step/nap membership policy to preserve +results. Normalizing the final forecast by its whole-horizon extrema remains +a global dependency, so caching the small complete forecast is preferable. + +CTL/ATL and EWMA baselines are already recurrences. Persist their checkpoint +states instead of replaying unchanged history. Winsor/outlier logic is +state-dependent, so revised historical values require forward replay. + +## Cache validity and full-pass policy + +The app should pass an explicit calculation mode. Only an awake periodic +update with proven unchanged dependencies qualifies for incremental execution. +During sleep, on heavy/finalization passes, and on forced recalculation, use +the complete current pipeline, then replace the cache with rebuilt state. +If sleep eligibility is uncertain, the safe initial policy is a complete pass. +The audited light/heavy job kind alone does not encode this distinction. + +Cache keys need calculation/state version, device family, profile/anchors, +source-selection revision, sleep bounds/masks, bin origin, timezone/day bounds, +baseline membership, and relevant manual edits. Include all parameters that +the function actually reads. Finalized results and provisional tail state +must have different lifetimes. + +Store sufficient statistics at full precision. Store per-bin contributions +so edits can remove old values before adding replacements. Do not reconstruct +state from rounded `Metric.toJson` scalars. + +The existing app fingerprint includes profile and previous-day inputs, but its +raw day component is `MAX(rec_ts):COUNT(*)`. That detects many insertions; it +does not detect replacement of a value at an existing timestamp with unchanged +count/max. An incremental cache requires a mutation revision/change log or +another proof that covers replacements, RR changes, and source/mask changes. +`MAX:COUNT` alone is insufficient. + +Publish cache state and derived output for the same captured input revision. +Commit together, or associate them with an identical validated revision and +discard mismatches. A failed/cancelled calculation must not advance the +checkpoint. Retrying an identical revision must be idempotent. Missing or +incompatible state falls back to a complete calculation. + +Late data invalidates its affected bins and downstream dependencies. A changed +NN beat also affects neighbor differences and downstream clock state where +times are accumulated. Centered filters compose their support widths; the RR +corrector's spline-anchor search and run reconciliation can extend dependency +reach. A fixed “last 91 beats” replay rule is not established by this audit. +Start with complete RR correction until stable-prefix equivalence is proved. + +Preserve absence, quality gates, coverage denominators, confidence, notes, +drivers and provenance. Updates are allowed to lower confidence or withdraw +a previously present metric. Valid data in a gap must never be inferred from +an accumulator's previous state. + +## Implementation order and validation + +1. Add mode/dependency revisions in `edge` and plain explicit accumulator states + in analytics. Keep existing batch entry points as the comparison reference. +2. Cache minute HR/motion summaries and their nonlinear TRIMP, energy and zone + contributions. Replace open/changed minutes. Reuse unchanged completed-night + results; keep every sleep-time calculation complete. +3. Cache baseline summaries and whole cross-day results by the inputs they read. + Daytime HR changes should not rerun unchanged nightly association analyses. + Advance daily recurrent states from a prior-day checkpoint, replacing today. +4. Cache completed Welch windows and shared cleaned RR/intermediate features. + The app currently invokes `detectSleepCycles` directly and again through + `sleepCyclesMetric`; share that result while preserving the envelope. +5. Add fixed-grid spectral accumulators if measurements justify their state + size and complexity. Prove RR stable-prefix boundaries before incremental + correction; keep global sleep-stage and retrospective scans complete. + +Compare incremental results against the current batch functions at every +prefix and varied chunk boundary, including one sample at a time. Exercise +duplicates, out-of-order insertion, value replacement, gaps, off-wrist data, +mutable partial minutes, changed anchors, cadence-mode changes, source edits, +sleep edits, midnight/DST, baseline expiration, interruption and restart. +Match publication gates and discrete decisions exactly; measure numerical +differences before selecting tolerances. Near a decision threshold, floating +differences can change absence or labels and need explicit handling. + +Real-capture comparisons should cover every device family with fixtures and +the existing sleep/causal/pedometer validation paths when those paths change. +Instrument reads, decoded rows, processed samples/windows, CPU time, isolate +payload/heap and serialized cache size. Accumulator improvements will not +remove full-day DB reads, decoding, or bundle serialization on their own. + +No performance speedup was measured in this audit. As a work-count example, +a 5-minute update at the end of 12 hours has 300 new seconds versus 43,200 +old-plus-new seconds: about 144× less scanning in an eligible append-only +component. That is not an app-level speedup estimate. + +The scratch algebra check ran 3,031 comparisons of append/remove variance, +pooled multi-run ACF1, direct versus cached Lomb–Scargle PSD, minute mean +absolute deviation, changed-reference ENMO, and nonlinear contribution +replacement. All passed at relative tolerance `2e-9` / absolute tolerance +`2e-7`; maximum absolute spectral difference was about `1.02e-10` on those +synthetic inputs. This checks the derivations, not a Dart implementation or +real-world speed. The scratch script was not added to the runtime library. +It is available in this session at `/tmp/openstrap_incremental_math_check.py`. diff --git a/docs/INCREMENTAL_USAGE.md b/docs/INCREMENTAL_USAGE.md new file mode 100644 index 0000000..2fbb4fa --- /dev/null +++ b/docs/INCREMENTAL_USAGE.md @@ -0,0 +1,90 @@ +# Incremental calculations + +Import `package:openstrap_analytics/onehz.dart`. The batch functions remain +available and are the reference algorithms used by the tests. + +| State | Cached math | Rebuild conditions | +|---|---|---| +| `RunningMoments` | Centered count, mean and sum of squared deviations; add, remove and merge | Caller owns membership when removing a value | +| `IntHistogram` | Sorted `value -> count` list of an integer-valued series; add, remove and merge | Caller owns membership when removing a value; whole numbers only | +| `IncrementalLombScargle` | Seven trigonometric sums per frequency, centered moments and time span | Earlier point edits, removals, a changed origin, or `force: true` | +| `IncrementalHrvTime` | NN moments, contiguous differences, lag-one products and five-minute bins | Earlier beat/time edits, removals, timing-mode changes, or force | +| `IncrementalEnmoSeries` | Trailing gravity sums and motion-minute buckets | Earlier sample edits, parameter changes, automatic gravity calibration, or force | +| `IncrementalMinuteMetrics` | TRIMP and energy contributions indexed by minute key | Changed minutes are repriced; changed HR anchors, sex or profile reprice all minutes | +| `CalculationCache` | Results with complete, owned dependency snapshots | Changed dependencies, explicit full evaluation, or eviction | + +`IntHistogram.percentile(p)` and `.median` return exactly what +`percentileSorted` returns for the expanded sorted list (same interpolation +between the two neighbouring order statistics, found from cumulative counts), +and null when empty. `add` takes a whole number (`70` or `70.0`; anything else +throws), `remove` throws if the value is absent, and `p` outside 0 to 100 +throws. Use it only for integral data: heart rate is whole bpm with about 45 +distinct values in a real night, but accelerometer magnitudes are almost all +distinct (30,848 of 32,041), so a histogram there is as long as the data. If +`calibrateGRef` ever needs to be incremental, use an order-statistic structure. + +Batch callers that take a median or percentile of heart rate over a long or +sliding window, and what adopting it would need: + +| Caller | Window | Notes | +|---|---|---| +| `_rollingMedianValidOnly` (`sleep/hr_fallback.dart`) | Centered, per sample | Best fit: a sliding add/remove. Input is `List`, so adopt only where the source is whole bpm. | +| `StrainScorer.estimateHRmax` (`clinical/load_trimp.dart`) | Trailing HR history | A high percentile of a long history; a running histogram avoids the re-sort per call. | +| `nocturnalRhr` p1 (`clinical/nocturnal.dart`) | The night's on-skin HR | One percentile per call, rebuilt each time. | +| `causal_stager` / `cardio_stager` local gates | ±180 epochs, p25 and p50 | Epoch HR is a mean, not a whole number. Not eligible unless the epoch value is made integral, which would change output. | +| `hrBaseline` (`sleep/advanced_stager.dart`) | Whole record | Median of all HR samples; a single call. | + +No existing incremental state recomputes an HR median per pass, so none was +switched; the histogram has no caller in the app yet. + +Pass the complete current input to `sync`. Each state checks that retained data +still matches, so replacing an old timestamp or value cannot silently leave a +stale result. Input comparison and output construction still visit retained +data. The work counters count samples whose math was evaluated, not every +array read or allocation. + +Minute keys must be unique and aligned with HR and optional cadence. Day duration +must be a finite whole number of minutes, matching `Calories.dailyEnergy`. +Changing day duration or the quiet-waking strain reference changes the summary +without repricing minute contributions. + +Use `includeMinuteSeries: false` when a caller needs only TRIMP, strain and +energy totals. It avoids rebuilding the minute output list. Turning that view +back on exposes the same retained contributions without repricing them. + +ENMO with an automatic gravity reference uses the full `enmoSeries` algorithm. +That reference comes from the whole record's median and can change earlier +minutes. Pass an explicit measured reference to use the incremental gravity +queue. The RR corrector also remains a full calculation because new beats can +change which earlier beats survive correction. + +HRV preserves time gaps, partially filled five-minute bins, the beat-timing +jitter refusal, confidence and notes. Ill-conditioned autocorrelation sums and +values near the refusal threshold fall back to the batch calculation. + +State checkpoints use `toJson` / `fromJson`, with version and type checks. +They retain double precision before public metric formatting. An unsupported +or malformed checkpoint throws; discard it and perform a full rebuild. + +Cache dependencies should be JSON-compatible maps, lists and scalars. Cached +results should be immutable objects or JSON-compatible collections. The cache +owns copies of collection results and dependencies and evicts the least +recently used entries at its configured limit. + +`CalculationMode.periodicAwake` is the only mode that permits reuse. Sleep, +heavy and forced runs must evaluate the full algorithms. The app owns awake +evidence, phone charging state and when a successfully persisted result may +replace its previous cache. These are orchestration decisions, so the math +package has no clock, battery API or database dependency. + +Run the deterministic timing comparison with: + +```sh +nix develop -c dart run tool/incremental_benchmark.dart +``` + +The benchmark reports local elapsed times and math-work counts. The parameterized +tests compare values, absent results, confidence, notes and provenance against +the batch algorithms after appends, revisions, deletions and checkpoint restores. +They also read subsets of the checked-in July 2026 overnight capture. Missing +capture files fail these tests rather than skipping them. diff --git a/flake.lock b/flake.lock new file mode 100644 index 0000000..8000e13 --- /dev/null +++ b/flake.lock @@ -0,0 +1,44 @@ +{ + "nodes": { + "nixpkgs": { + "locked": { + "lastModified": 1790046670, + "narHash": "sha256-MYiI+CzL0tuWgRPjGsKCDHqYs2T3OzMlMQWOYWG0qso=", + "owner": "NixOS", + "repo": "nixpkgs", + "rev": "6774f7bc253789b113a4f39285dc0fa100abeacc", + "type": "github" + }, + "original": { + "owner": "NixOS", + "ref": "nixos-unstable", + "repo": "nixpkgs", + "type": "github" + } + }, + "nixpkgs-flutter": { + "locked": { + "lastModified": 1774840819, + "narHash": "sha256-V9TPFnbTQx9bDyVKuHpYi/0fC5qTgxDeHkST0RNAuTk=", + "owner": "NixOS", + "repo": "nixpkgs", + "rev": "27dfede99da61fd1ead9d4a2fa92bc9c242e83d2", + "type": "github" + }, + "original": { + "owner": "NixOS", + "repo": "nixpkgs", + "rev": "27dfede99da61fd1ead9d4a2fa92bc9c242e83d2", + "type": "github" + } + }, + "root": { + "inputs": { + "nixpkgs": "nixpkgs", + "nixpkgs-flutter": "nixpkgs-flutter" + } + } + }, + "root": "root", + "version": 7 +} diff --git a/flake.nix b/flake.nix new file mode 100644 index 0000000..62def2a --- /dev/null +++ b/flake.nix @@ -0,0 +1,52 @@ +{ + description = "OpenStrap analytics Dart and Flutter development environment"; + + inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable"; + # Same Flutter/Dart release as ../edge and its CI. + inputs.nixpkgs-flutter.url = + "github:NixOS/nixpkgs/27dfede99da61fd1ead9d4a2fa92bc9c242e83d2"; + + outputs = { nixpkgs, nixpkgs-flutter, ... }: + let + systems = [ "x86_64-linux" "aarch64-linux" ]; + forAllSystems = nixpkgs.lib.genAttrs systems; + in { + devShells = forAllSystems (system: + let + pkgs = import nixpkgs { inherit system; }; + flutterPkgs = import nixpkgs-flutter { + inherit system; + config.allowUnfree = true; + }; + flutter = flutterPkgs.flutterPackages.v3_41; + tools = with pkgs; [ + flutter git ripgrep jq python3 gnumake unzip which + clang cmake ninja pkg-config + ]; + fhs = pkgs.buildFHSEnv { + name = "analytics-fhs"; + targetPkgs = p: tools ++ (with p; [ + bashInteractive zlib libcxx ncurses5 gtk3 fontconfig libGL + ]); + profile = '' + export IN_ANALYTICS_FHS=1 + ''; + runScript = pkgs.writeShellScript "analytics-fhs-run" '' + if [[ $# -eq 0 ]]; then exec bash; else exec "$@"; fi + ''; + }; + in { + default = pkgs.mkShell { + packages = tools ++ [ fhs ]; + shellHook = '' + # Let the packaged Flutter wrapper select its complete SDK, + # including prebuilt engine/test artifacts. + unset FLUTTER_ROOT + if [[ -z "$IN_ANALYTICS_FHS" && $- == *i* ]]; then + exec analytics-fhs + fi + ''; + }; + }); + }; +} diff --git a/lib/aggregate.dart b/lib/aggregate.dart new file mode 100644 index 0000000..ad2d63d --- /dev/null +++ b/lib/aggregate.dart @@ -0,0 +1,17 @@ +/// Mergeable summaries and the building blocks used to compose them. +library aggregate; + +export 'src/aggregate/aggregator.dart'; +export 'src/aggregate/checkpoint.dart'; +export 'src/aggregate/laws.dart'; +export 'src/aggregate/primitives.dart'; +export 'src/onehz/clinical/hrv_time_aggregator.dart' hide hrvTimeOneSlice; +export 'src/onehz/clinical/load_trimp_aggregator.dart'; +export 'src/onehz/clinical/cosinor_aggregator.dart'; +export 'src/onehz/sleep/hypnogram_aggregator.dart'; +export 'src/onehz/sleep/circadian_np_aggregator.dart'; +export 'src/onehz/clinical/prsa_aggregator.dart'; +export 'src/onehz/foundations/rr_sandwich.dart'; +export 'src/onehz/clinical/readiness_lnrmssd_aggregator.dart'; +export 'src/onehz/sleep/sri_aggregator.dart'; +export 'src/onehz/workout/time_in_zone_aggregator.dart'; diff --git a/lib/onehz.dart b/lib/onehz.dart index 308538b..389c351 100644 --- a/lib/onehz.dart +++ b/lib/onehz.dart @@ -18,6 +18,7 @@ export 'src/onehz/device.dart'; // Layer 0: foundations. export 'src/onehz/foundations/rr_correction.dart'; +export 'src/onehz/foundations/rr_correction_stream.dart'; export 'src/onehz/foundations/baseline.dart'; export 'src/onehz/foundations/ewma_baselines.dart'; export 'src/onehz/foundations/fusion.dart'; @@ -28,11 +29,15 @@ export 'src/onehz/clinical/hrv_freq.dart'; export 'src/onehz/clinical/prsa.dart'; export 'src/onehz/clinical/nocturnal.dart'; export 'src/onehz/clinical/illness_cusum.dart'; -export 'src/onehz/clinical/readiness_lnrmssd.dart'; +export 'src/onehz/clinical/readiness_lnrmssd.dart' + hide readinessLnRmssdFromSuffix; export 'src/onehz/clinical/cosinor.dart'; export 'src/onehz/clinical/load_trimp.dart'; export 'src/onehz/clinical/stress_si.dart'; export 'src/onehz/clinical/irregular_rhythm.dart'; +export 'src/onehz/clinical/irregular_rhythm_state.dart'; +export 'src/onehz/clinical/irregular_diagnostics.dart' + hide diagnosticArtifactFraction; export 'src/onehz/clinical/cardiac_coherence.dart'; export 'src/onehz/clinical/vo2max.dart'; @@ -43,3 +48,8 @@ export 'src/onehz/motion/motion.dart'; export 'src/onehz/workout/workout.dart'; export 'src/onehz/wellness/wellness.dart'; export 'src/onehz/human/human.dart'; + +export 'src/onehz/incremental.dart'; + +// Calculation-group registry (design 06): the groups this package declares. +export 'src/onehz/registry/group_registry.dart'; diff --git a/lib/src/aggregate/aggregator.dart b/lib/src/aggregate/aggregator.dart new file mode 100644 index 0000000..7763f97 --- /dev/null +++ b/lib/src/aggregate/aggregator.dart @@ -0,0 +1,271 @@ +import 'dart:typed_data'; + +import 'json_support.dart'; + +/// A run of rows stored column-wise. +abstract class Batch> { + int get length; + B copyRange(int start, int end); + B join(B other); + + /// A JSON-encodable copy of the rows. Throws [UnsupportedError] unless the + /// batch type implements it. + Object? toJson() => + throw UnsupportedError('$runtimeType does not support toJson'); +} + +/// How a [ListBatch] writes and reads one row. [decode] must throw +/// [FormatException] on a malformed row. +class RowCodec { + const RowCodec(this.encode, this.decode); + + final Object? Function(A row) encode; + final A Function(Object? json) decode; + + /// Reads a coded batch from the JSON list written by [ListBatch.toJson]. + ListBatch batchFromJson(Object? json) => + ListBatch.coded([for (final row in decodeList(json)) decode(row)], this); +} + +/// A batch backed by a normal Dart list. +class ListBatch extends Batch> { + ListBatch(this.rows) : _codec = null; + + /// A batch whose rows can be written with [codec] (see [toJson]). + ListBatch.coded(this.rows, RowCodec codec) : _codec = codec; + + final List rows; + final RowCodec? _codec; + + @override + int get length => rows.length; + + @override + ListBatch copyRange(int start, int end) => + ListBatch._(rows.sublist(start, end), _codec); + + @override + ListBatch join(ListBatch other) => + ListBatch._([...rows, ...other.rows], _codec ?? other._codec); + + ListBatch._(this.rows, this._codec); + + /// The encoded rows. Throws [UnsupportedError] for a batch without a codec. + @override + Object? toJson() { + final codec = _codec; + if (codec == null) { + throw UnsupportedError('ListBatch has no row codec; use ListBatch.coded'); + } + return [for (final row in rows) codec.encode(row)]; + } +} + +/// A batch containing one double column. NaN represents a missing row. +class DoubleColumn extends Batch { + DoubleColumn(this.values); + + final Float64List values; + + @override + int get length => values.length; + + @override + DoubleColumn copyRange(int start, int end) { + final copied = Float64List(end - start) + ..setRange(0, end - start, values, start); + return DoubleColumn(copied); + } + + @override + DoubleColumn join(DoubleColumn other) { + final joined = Float64List(length + other.length); + joined.setRange(0, length, values); + joined.setRange(length, joined.length, other.values); + return DoubleColumn(joined); + } + + /// The values as a JSON list; NaN and infinities become strings. + @override + Object? toJson() => encodeDoubles(values); + + factory DoubleColumn.fromJson(Object? json) => DoubleColumn(decodeDoubles(json)); +} + +/// A mergeable summary over slices of a [Batch]. +abstract class Aggregator, R> { + const Aggregator(); + + S empty(); + S addSlice(S state, B batch, int start, int end); + S combine(S left, S right); + R finish(S state); + + /// Whether [encodeState] and [decodeState] are implemented. A composite is + /// persistable only when every child is. + bool get persistable => false; + + /// A JSON-encodable form of [state] (null, bool, num, String, List, or a + /// Map with String keys). Throws [UnsupportedError] unless [persistable]. + Object? encodeState(S state) => + throw UnsupportedError('$runtimeType is not persistable'); + + /// Inverse of [encodeState]. Throws [FormatException] on malformed input, + /// and [UnsupportedError] unless [persistable]. + S decodeState(Object? json) => + throw UnsupportedError('$runtimeType is not persistable'); +} + +/// Convenience operations and result adapters for [Aggregator]. +extension AggregatorOps, R> on Aggregator { + S fold(B batch, [int start = 0, int? end]) { + final actualEnd = end ?? batch.length; + return addSlice(empty(), batch, start, actualEnd); + } + + R run(B batch) => finish(fold(batch)); + + Aggregator map(R2 Function(R result) transform) => + _MappedAggregator(this, transform); +} + +/// Element-world convenience for aggregators over [ListBatch]. +extension ListAggregatorOps on Aggregator, R> { + R runList(List rows) => run(ListBatch(rows)); +} + +class _MappedAggregator, R, R2> + extends Aggregator { + const _MappedAggregator(this.inner, this.transform); + + final Aggregator inner; + final R2 Function(R result) transform; + + @override + S empty() => inner.empty(); + + @override + S addSlice(S state, B batch, int start, int end) => + inner.addSlice(state, batch, start, end); + + @override + S combine(S left, S right) => inner.combine(left, right); + + @override + R2 finish(S state) => transform(inner.finish(state)); + + @override + bool get persistable => inner.persistable; + + @override + Object? encodeState(S state) => inner.encodeState(state); + + @override + S decodeState(Object? json) => inner.decodeState(json); +} + +/// Mutable state for [both]. +class Pair { + Pair(this.first, this.second); + + A first; + B second; +} + +/// Runs two aggregators over each same slice. +Aggregator, B, (R1, R2)> + both, R1, R2>( + Aggregator first, + Aggregator second, +) => + _BothAggregator(first, second); + +/// Adds module-specific consistency checks to a persistable checkpoint. +/// +/// [validate] runs after [inner] has decoded each child state. It receives the +/// encoded state so a composed module can compare child values whose concrete +/// state types are private to their aggregators. Encoding and hot paths are +/// otherwise unchanged. +Aggregator validateCheckpoint, R>( + Aggregator inner, + void Function(Object? json) validate, +) => + _CheckpointValidatedAggregator(inner, validate); + +class _CheckpointValidatedAggregator, R> + extends Aggregator { + const _CheckpointValidatedAggregator(this.inner, this.validate); + + final Aggregator inner; + final void Function(Object? json) validate; + + @override + S empty() => inner.empty(); + + @override + S addSlice(S state, B batch, int start, int end) => + inner.addSlice(state, batch, start, end); + + @override + S combine(S left, S right) => inner.combine(left, right); + + @override + R finish(S state) => inner.finish(state); + + @override + bool get persistable => inner.persistable; + + @override + Object? encodeState(S state) => inner.encodeState(state); + + @override + S decodeState(Object? json) { + final state = inner.decodeState(json); + validate(json); + return state; + } +} + +class _BothAggregator, R1, R2> + extends Aggregator, B, (R1, R2)> { + const _BothAggregator(this.first, this.second); + + final Aggregator first; + final Aggregator second; + + @override + Pair empty() => Pair(first.empty(), second.empty()); + + @override + Pair addSlice(Pair state, B batch, int start, int end) { + state.first = first.addSlice(state.first, batch, start, end); + state.second = second.addSlice(state.second, batch, start, end); + return state; + } + + @override + Pair combine(Pair left, Pair right) { + left.first = first.combine(left.first, right.first); + left.second = second.combine(left.second, right.second); + return left; + } + + @override + (R1, R2) finish(Pair state) => + (first.finish(state.first), second.finish(state.second)); + + @override + bool get persistable => first.persistable && second.persistable; + + @override + Object? encodeState(Pair state) => + [first.encodeState(state.first), second.encodeState(state.second)]; + + @override + Pair decodeState(Object? json) { + final list = decodeList(json); + if (list.length != 2) throw const FormatException('Expected two states'); + return Pair( + first.decodeState(list[0]), second.decodeState(list[1])); + } +} diff --git a/lib/src/aggregate/checkpoint.dart b/lib/src/aggregate/checkpoint.dart new file mode 100644 index 0000000..b27dbb7 --- /dev/null +++ b/lib/src/aggregate/checkpoint.dart @@ -0,0 +1,39 @@ +import 'aggregator.dart'; + +/// Wraps the JSON form of [state] in a versioned envelope: +/// {'kind', 'version', 'state'}. Throws [UnsupportedError] if [agg] is not +/// persistable. +Map encodeCheckpoint( + Aggregator agg, + S state, { + required String kind, + required int version, +}) { + if (!agg.persistable) { + throw UnsupportedError('${agg.runtimeType} is not persistable'); + } + return {'kind': kind, 'version': version, 'state': agg.encodeState(state)}; +} + +/// Restores a state written by [encodeCheckpoint]. Returns null, never a +/// guess, when the kind or version differ or the envelope or state is +/// malformed; the caller then recomputes from raw data. +S? decodeCheckpoint( + Aggregator agg, + Object? json, { + required String kind, + required int version, +}) { + if (json is! Map || + json['kind'] != kind || + json['version'] is! int || + json['version'] != version || + !json.containsKey('state')) { + return null; + } + try { + return agg.decodeState(json['state']); + } on FormatException { + return null; + } +} diff --git a/lib/src/aggregate/json_support.dart b/lib/src/aggregate/json_support.dart new file mode 100644 index 0000000..911521b --- /dev/null +++ b/lib/src/aggregate/json_support.dart @@ -0,0 +1,64 @@ +// JSON helpers shared by the aggregate persistence codecs. Not exported. +// +// Non-finite doubles are written as the strings 'NaN', 'Infinity' and +// '-Infinity' (the same strings incremental_core.dart reads). Every reader +// throws FormatException, never TypeError or RangeError. + +import 'dart:typed_data'; + +/// A double as a JSON value: finite stays a number, the rest become strings. +Object encodeDouble(double value) { + if (value.isFinite) return value; + if (value.isNaN) return 'NaN'; + return value > 0 ? 'Infinity' : '-Infinity'; +} + +/// Inverse of [encodeDouble]; an int is accepted for a double. +double decodeDouble(Object? value) { + if (value is num) return value.toDouble(); + if (value == 'NaN') return double.nan; + if (value == 'Infinity') return double.infinity; + if (value == '-Infinity') return double.negativeInfinity; + throw const FormatException('Invalid number'); +} + +double? decodeDoubleOrNull(Object? value) => + value == null ? null : decodeDouble(value); + +/// A non-negative int. +int decodeCount(Object? value) { + if (value is! int || value < 0) throw const FormatException('Invalid count'); + return value; +} + +List decodeList(Object? value) { + if (value is! List) throw const FormatException('Expected a list'); + return value; +} + +/// A JSON object with string keys. +Map decodeMap(Object? value) { + if (value is! Map) throw const FormatException('Expected an object'); + for (final key in value.keys) { + if (key is! String) throw const FormatException('Expected string keys'); + } + return value.cast(); +} + +/// The value at [key]; the key must be present (null is a value). +Object? field(Map map, String key) { + if (!map.containsKey(key)) throw FormatException('Missing field $key'); + return map[key]; +} + +List encodeDoubles(Float64List values) => + [for (final value in values) encodeDouble(value)]; + +Float64List decodeDoubles(Object? value) { + final list = decodeList(value); + final result = Float64List(list.length); + for (var i = 0; i < list.length; i++) { + result[i] = decodeDouble(list[i]); + } + return result; +} diff --git a/lib/src/aggregate/laws.dart b/lib/src/aggregate/laws.dart new file mode 100644 index 0000000..ba92b23 --- /dev/null +++ b/lib/src/aggregate/laws.dart @@ -0,0 +1,343 @@ +import 'dart:convert'; +import 'dart:math' as math; + +import 'aggregator.dart'; + +typedef ResultEquals = bool Function(R left, R right); + +class LawViolation { + const LawViolation(this.law, this.detail); + final String law; + final String detail; +} + +/// Checks aggregator merge and slicing laws on each batch in [cases]. +List checkLaws, R>( + Aggregator aggregator, + List cases, { + ResultEquals? equals, + int maxSplits = 64, + int seed = 0, +}) { + final equal = equals ?? (R left, R right) => left == right; + final violations = []; + + R whole(B batch) => aggregator.finish(aggregator.fold(batch)); + R joined(B batch, int split) => aggregator.finish(aggregator.combine( + aggregator.fold(batch, 0, split), + aggregator.fold(batch, split, batch.length))); + + for (var caseIndex = 0; caseIndex < cases.length; caseIndex++) { + final batch = cases[caseIndex]; + final detail = 'case $caseIndex: $batch'; + final expected = whole(batch); + final leftIdentity = aggregator.finish( + aggregator.combine(aggregator.empty(), aggregator.fold(batch))); + if (!equal(leftIdentity, expected)) { + violations.add(LawViolation('leftIdentity', detail)); + } + final rightIdentity = aggregator.finish( + aggregator.combine(aggregator.fold(batch), aggregator.empty())); + if (!equal(rightIdentity, expected)) { + violations.add(LawViolation('rightIdentity', detail)); + } + + final splits = _splitPoints(batch.length, maxSplits, seed + caseIndex); + for (final split in splits) { + if (!equal(joined(batch, split), expected)) { + violations.add(LawViolation('split', '$detail; split $split')); + } + } + for (var i = 0; i < splits.length; i++) { + for (var j = i; j < splits.length; j++) { + final first = splits[i]; + final second = splits[j]; + final left = aggregator.finish(aggregator.combine( + aggregator.combine(aggregator.fold(batch, 0, first), + aggregator.fold(batch, first, second)), + aggregator.fold(batch, second, batch.length))); + final right = aggregator.finish(aggregator.combine( + aggregator.fold(batch, 0, first), + aggregator.combine(aggregator.fold(batch, first, second), + aggregator.fold(batch, second, batch.length)))); + if (!equal(left, right)) { + violations.add( + LawViolation('associativity', '$detail; splits $first, $second')); + } + } + } + + var singletonState = aggregator.empty(); + for (var i = 0; i < batch.length; i++) { + singletonState = aggregator.combine( + singletonState, aggregator.fold(batch, i, i + 1)); + } + if (!equal(aggregator.finish(singletonState), expected)) { + violations.add(LawViolation('singletons', detail)); + } + + var sliced = aggregator.empty(); + final slicePoints = _splitPoints(batch.length, maxSplits, seed ^ caseIndex); + for (var i = 0; i + 1 < slicePoints.length; i++) { + sliced = aggregator.addSlice(sliced, batch, slicePoints[i], slicePoints[i + 1]); + } + if (!equal(aggregator.finish(sliced), expected)) { + violations.add(LawViolation('slicing', detail)); + } + + // finish must not consume or otherwise close a state. Keep separate + // states for the add and combine continuations: combine is allowed to + // consume either input. + final continuationStart = batch.length ~/ 2; + var afterFinish = aggregator.fold(batch, 0, continuationStart); + final firstFinish = aggregator.finish(afterFinish); + final secondFinish = aggregator.finish(afterFinish); + var finishIsPure = equal(firstFinish, secondFinish); + + afterFinish = aggregator.addSlice( + afterFinish, batch, continuationStart, batch.length); + var addReference = aggregator.fold(batch, 0, continuationStart); + addReference = aggregator.addSlice( + addReference, batch, continuationStart, batch.length); + if (!equal(aggregator.finish(afterFinish), aggregator.finish(addReference))) { + finishIsPure = false; + } + + var combinedAfterFinish = aggregator.fold(batch, 0, continuationStart); + aggregator.finish(combinedAfterFinish); + aggregator.finish(combinedAfterFinish); + combinedAfterFinish = aggregator.combine( + combinedAfterFinish, + aggregator.fold(batch, continuationStart, batch.length)); + var combineReference = aggregator.fold(batch, 0, continuationStart); + combineReference = aggregator.combine( + combineReference, + aggregator.fold(batch, continuationStart, batch.length)); + if (!equal(aggregator.finish(combinedAfterFinish), + aggregator.finish(combineReference))) { + finishIsPure = false; + } + if (!finishIsPure) { + violations.add(LawViolation('finishPure', detail)); + } + + if (aggregator.persistable) { + // A state written to JSON text and read back must behave like the + // original: alone, on either side of a merge, and when extended. + S roundTrip(S state) => aggregator.decodeState( + jsonDecode(jsonEncode(aggregator.encodeState(state)))); + void check(String what, R Function() actual) { + try { + if (!equal(actual(), expected)) { + violations.add(LawViolation('roundTrip', '$detail; $what')); + } + } catch (error) { + violations.add(LawViolation('roundTrip', '$detail; $what: $error')); + } + } + + check('whole state', () => aggregator.finish(roundTrip(aggregator.fold(batch)))); + for (final split in splits) { + check( + 'combine after split $split (restored left)', + () => aggregator.finish(aggregator.combine( + roundTrip(aggregator.fold(batch, 0, split)), + aggregator.fold(batch, split, batch.length)))); + check( + 'combine after split $split (restored right)', + () => aggregator.finish(aggregator.combine( + aggregator.fold(batch, 0, split), + roundTrip(aggregator.fold(batch, split, batch.length))))); + check( + 'combine after split $split (restored both)', + () => aggregator.finish(aggregator.combine( + roundTrip(aggregator.fold(batch, 0, split)), + roundTrip(aggregator.fold(batch, split, batch.length))))); + check( + 'addSlice after split $split', + () => aggregator.finish(aggregator.addSlice( + roundTrip(aggregator.fold(batch, 0, split)), + batch, + split, + batch.length))); + } + } + } + return violations; +} + +class ConversionFailure { + const ConversionFailure(this.path, this.caseIndex, this.detail); + final String path; + final int caseIndex; + final String detail; +} + +/// Checks that an aggregator reproduces [reference] on every evaluation path. +/// +/// Paths: 'whole', 'chunked' (left-to-right combine of ranges of the original +/// batch), 'chunkedRight' (same bounds, right-nested combine), 'copies' (each +/// chunk cut with copyRange first), 'sliced' (one state, consecutive slices) +/// and 'singletons'. With [laws], checkLaws violations appear as 'law:'. +/// Deterministic per [seed]. +List checkConversion, R>( + Aggregator Function(B input) build, + List cases, + R Function(B input) reference, { + ResultEquals? equals, + int chunkings = 8, + int seed = 0, + bool laws = true, +}) { + final equal = equals ?? (R left, R right) => left == right; + final failures = []; + + for (var caseIndex = 0; caseIndex < cases.length; caseIndex++) { + final batch = cases[caseIndex]; + final aggregator = build(batch); + final expected = reference(batch); + final seen = {}; + + void check(String path, R Function() actual, [String where = '']) { + if (seen.contains(path)) return; + final got = actual(); + if (equal(got, expected)) return; + seen.add(path); + failures.add(ConversionFailure(path, caseIndex, + 'case $caseIndex: $batch$where; got $got, expected $expected')); + } + + check('whole', () => aggregator.finish(aggregator.fold(batch))); + + final rng = _Lcg(seed * 7919 + caseIndex); + for (var c = 0; c < chunkings; c++) { + final bounds = _chunkBounds(batch.length, rng); + final where = '; bounds $bounds'; + List states({bool copy = false}) => [ + for (var i = 0; i + 1 < bounds.length; i++) + copy + ? aggregator.fold(batch.copyRange(bounds[i], bounds[i + 1])) + : aggregator.fold(batch, bounds[i], bounds[i + 1]), + ]; + S leftToRight(List parts) { + var acc = parts.first; + for (var i = 1; i < parts.length; i++) { + acc = aggregator.combine(acc, parts[i]); + } + return acc; + } + + check('chunked', () => aggregator.finish(leftToRight(states())), where); + check('chunkedRight', () { + final parts = states(); + var acc = parts.last; + for (var i = parts.length - 2; i >= 0; i--) { + acc = aggregator.combine(parts[i], acc); + } + return aggregator.finish(acc); + }, where); + check('copies', + () => aggregator.finish(leftToRight(states(copy: true))), where); + check('sliced', () { + var state = aggregator.empty(); + for (var i = 0; i + 1 < bounds.length; i++) { + state = aggregator.addSlice(state, batch, bounds[i], bounds[i + 1]); + } + return aggregator.finish(state); + }, where); + } + + check('singletons', () { + var state = aggregator.empty(); + for (var i = 0; i < batch.length; i++) { + state = aggregator.combine(state, aggregator.fold(batch, i, i + 1)); + } + return aggregator.finish(state); + }); + + if (laws) { + for (final v in checkLaws(aggregator, [batch], + equals: equals, seed: seed + caseIndex)) { + failures.add(ConversionFailure( + 'law:${v.law}', caseIndex, v.detail.replaceFirst('case 0', 'case $caseIndex'))); + } + } + } + return failures; +} + +class _Lcg { + _Lcg(int seed) : _state = seed & 0x7fffffff; + int _state; + int next(int bound) { + _state = (_state * 1103515245 + 12345) & 0x7fffffff; + return (_state >> 8) % bound; + } +} + +/// Sorted chunk bounds from 0 to [length]: always cuts at 1 and length - 1 and +/// a duplicated cut (an empty chunk), plus a few random cuts. +List _chunkBounds(int length, _Lcg rng) { + final cuts = [ + 0, + length, + math.min(1, length), + math.max(0, length - 1), + for (var i = rng.next(4); i > 0; i--) rng.next(length + 1), + ]; + cuts.add(cuts[2 + rng.next(cuts.length - 2)]); + return cuts..sort(); +} + +List _splitPoints(int length, int maxSplits, int seed) { + if (length == 0) return [0]; + final wanted = math.max(2, maxSplits); + if (length + 1 <= wanted) return [for (var i = 0; i <= length; i++) i]; + final points = {0, length}; + var state = seed & 0x7fffffff; + while (points.length < wanted) { + state = (state * 1103515245 + 12345) & 0x7fffffff; + points.add(1 + state % (length - 1)); + } + return points.toList()..sort(); +} + +/// Compares the listed fields pairwise; numbers use the tolerances and lists +/// compare element by element with the same rules. +ResultEquals fieldsEqual( + List Function(R value) fields, { + double relTol = 0, + double absTol = 0, +}) => + (left, right) { + final a = fields(left); + final b = fields(right); + if (a.length != b.length) return false; + for (var i = 0; i < a.length; i++) { + if (!_fieldEqual(a[i], b[i], relTol, absTol)) return false; + } + return true; + }; + +bool _fieldEqual(Object? left, Object? right, double relTol, double absTol) { + // Converting two ints to double loses distinctions above 2^53. + if (left is int && right is int) return left == right; + if (left is List || right is List) { + if (left is! List || right is! List || left.length != right.length) { + return false; + } + for (var i = 0; i < left.length; i++) { + if (!_fieldEqual(left[i], right[i], relTol, absTol)) return false; + } + return true; + } + if (left == right) return true; + if (left == null || right == null) return false; + if (left is! num || right is! num) return false; + if (left.isNaN || right.isNaN) return left.isNaN && right.isNaN; + if (!left.isFinite || !right.isFinite) return false; + final a = left.toDouble(); + final b = right.toDouble(); + final tolerance = absTol + relTol * math.max(a.abs(), b.abs()); + return (a - b).abs() <= tolerance; +} diff --git a/lib/src/aggregate/primitives.dart b/lib/src/aggregate/primitives.dart new file mode 100644 index 0000000..d2ddb26 --- /dev/null +++ b/lib/src/aggregate/primitives.dart @@ -0,0 +1,885 @@ +// Mergeable sums, centered moments and affine EWMA composition reformulate +// the corresponding batch arithmetic. Composition is algebraically equal; +// floating-point evaluation order can differ. Count/min/max and structural +// wrappers are definitional operations, not additional physiological estimators. + +import 'dart:typed_data'; + +import '../onehz/incremental_core.dart' show RunningMoments; + +import 'aggregator.dart'; +import 'json_support.dart'; + +/// Counts every row, including rows whose column value is NaN. +class Count> extends Aggregator { + const Count(); + @override + int empty() => 0; + @override + int addSlice(int state, B batch, int start, int end) => state + end - start; + @override + int combine(int left, int right) => left + right; + @override + int finish(int state) => state; + @override + bool get persistable => true; + @override + Object? encodeState(int state) => state; + @override + int decodeState(Object? json) => decodeCount(json); +} + +/// Counts non-missing values in a double column. +class CountFinite> extends Aggregator { + const CountFinite(this.column); + final Float64List Function(B batch) column; + @override + int empty() => 0; + @override + int addSlice(int state, B batch, int start, int end) { + final values = column(batch); + var count = state; + for (var i = start; i < end; i++) { + if (!values[i].isNaN) count++; + } + return count; + } + @override + int combine(int left, int right) => left + right; + @override + int finish(int state) => state; + @override + bool get persistable => true; + @override + Object? encodeState(int state) => state; + @override + int decodeState(Object? json) => decodeCount(json); +} + +/// Counts non-missing values in a double column which pass [test]. +class CountWhere> extends Aggregator { + const CountWhere(this.column, this.test); + final Float64List Function(B batch) column; + final bool Function(double value) test; + @override + int empty() => 0; + @override + int addSlice(int state, B batch, int start, int end) { + final values = column(batch); + var count = state; + for (var i = start; i < end; i++) { + final value = values[i]; + if (!value.isNaN && test(value)) count++; + } + return count; + } + @override + int combine(int left, int right) => left + right; + @override + int finish(int state) => state; + @override + bool get persistable => true; + @override + Object? encodeState(int state) => state; + @override + int decodeState(Object? json) => decodeCount(json); +} + +/// Neumaier-compensated sum of non-missing double column values. +/// +/// Chunk-invariant results are supported while all partial sums are finite. +/// Once a partial sum overflows, IEEE-754 arithmetic can make the result +/// depend on chunking. +class Sum> extends Aggregator<_SumState, B, double> { + const Sum(this.column); + final Float64List Function(B batch) column; + @override + _SumState empty() => _SumState(); + @override + _SumState addSlice(_SumState state, B batch, int start, int end) { + final values = column(batch); + for (var i = start; i < end; i++) { + final value = values[i]; + if (!value.isNaN) state.add(value); + } + return state; + } + @override + _SumState combine(_SumState left, _SumState right) { + left.add(right.sum); + left.add(right.correction); + return left; + } + @override + double finish(_SumState state) => state.sum + state.correction; + @override + bool get persistable => true; + + @override + Object? encodeState(_SumState state) => state.toJson(); + + @override + _SumState decodeState(Object? json) => _SumState.fromJson(json); +} + +/// Neumaier-compensated sum of a function of non-missing double column values. +/// +/// A NaN source value is skipped without calling [transform]. A NaN result from +/// [transform] is also skipped. +/// Chunk-invariant results are supported while all partial sums are finite. +/// Once a partial sum overflows, IEEE-754 arithmetic can make the result +/// depend on chunking. +class SumOf> extends Aggregator<_SumState, B, double> { + const SumOf(this.column, this.transform); + + final Float64List Function(B batch) column; + final double Function(double value) transform; + + @override + _SumState empty() => _SumState(); + + @override + _SumState addSlice(_SumState state, B batch, int start, int end) { + final values = column(batch); + for (var i = start; i < end; i++) { + final value = values[i]; + if (value.isNaN) continue; + final transformed = transform(value); + if (!transformed.isNaN) state.add(transformed); + } + return state; + } + + @override + _SumState combine(_SumState left, _SumState right) { + left.add(right.sum); + left.add(right.correction); + return left; + } + + @override + double finish(_SumState state) => state.sum + state.correction; + @override + bool get persistable => true; + + @override + Object? encodeState(_SumState state) => state.toJson(); + + @override + _SumState decodeState(Object? json) => _SumState.fromJson(json); +} + +class _SumState { + _SumState(); + + factory _SumState.fromJson(Object? json) { + final map = decodeMap(json); + return _SumState() + ..sum = decodeDouble(field(map, 'sum')) + ..correction = decodeDouble(field(map, 'correction')); + } + + double sum = 0.0; + double correction = 0.0; + + Map toJson() => + {'sum': encodeDouble(sum), 'correction': encodeDouble(correction)}; + + void add(double value) { + final next = sum + value; + if (!next.isFinite) { + sum = next; + return; + } + if (sum.abs() >= value.abs()) { + correction += (sum - next) + value; + } else { + correction += (value - next) + sum; + } + sum = next; + } +} + +/// Sum of squares of non-missing double column values. +class SumSq> extends Aggregator { + const SumSq(this.column); + final Float64List Function(B batch) column; + @override + double empty() => 0.0; + @override + double addSlice(double state, B batch, int start, int end) { + final values = column(batch); + var sum = state; + for (var i = start; i < end; i++) { + final value = values[i]; + if (!value.isNaN) sum += value * value; + } + return sum; + } + @override + double combine(double left, double right) => left + right; + @override + double finish(double state) => state; + @override + bool get persistable => true; + @override + Object? encodeState(double state) => encodeDouble(state); + @override + double decodeState(Object? json) => decodeDouble(json); +} + +/// Smallest non-missing value in a double column. +class Min> extends Aggregator { + const Min(this.column); + final Float64List Function(B batch) column; + @override + double? empty() => null; + @override + double? addSlice(double? state, B batch, int start, int end) { + final values = column(batch); + var result = state; + for (var i = start; i < end; i++) { + final value = values[i]; + if (!value.isNaN && (result == null || value < result)) result = value; + } + return result; + } + @override + double? combine(double? left, double? right) { + if (left == null || right == null) return left ?? right; + return right < left ? right : left; + } + @override + double? finish(double? state) => state; + @override + bool get persistable => true; + @override + Object? encodeState(double? state) => + state == null ? null : encodeDouble(state); + @override + double? decodeState(Object? json) => decodeDoubleOrNull(json); +} + +/// Largest non-missing value in a double column. +class Max> extends Aggregator { + const Max(this.column); + final Float64List Function(B batch) column; + @override + double? empty() => null; + @override + double? addSlice(double? state, B batch, int start, int end) { + final values = column(batch); + var result = state; + for (var i = start; i < end; i++) { + final value = values[i]; + if (!value.isNaN && (result == null || value > result)) result = value; + } + return result; + } + @override + double? combine(double? left, double? right) { + if (left == null || right == null) return left ?? right; + return right > left ? right : left; + } + @override + double? finish(double? state) => state; + @override + bool get persistable => true; + @override + Object? encodeState(double? state) => + state == null ? null : encodeDouble(state); + @override + double? decodeState(Object? json) => decodeDoubleOrNull(json); +} + +// Delegates updates/removal to Welford 1962 and partition merging to Chan, +// Golub & LeVeque 1983; full references and clamp details: RunningMoments. +/// Centered moments of non-missing double column values. +class Moments> + extends Aggregator { + const Moments(this.column); + final Float64List Function(B batch) column; + @override + RunningMoments empty() => RunningMoments(); + @override + RunningMoments addSlice(RunningMoments state, B batch, int start, int end) { + final values = column(batch); + for (var i = start; i < end; i++) { + final value = values[i]; + if (!value.isNaN) state.add(value); + } + return state; + } + @override + RunningMoments combine(RunningMoments left, RunningMoments right) { + left.merge(right); + return left; + } + @override + RunningMoments finish(RunningMoments state) => state; + @override + bool get persistable => true; + @override + Object? encodeState(RunningMoments state) => state.toJson(); + @override + RunningMoments decodeState(Object? json) => + RunningMoments.fromJson(decodeMap(json)); +} + +class _EdgeState> { + B? value; +} + +Object? _encodeEdge>( + _EdgeState state, B Function(Object?)? batchFromJson) { + if (batchFromJson == null) { + throw UnsupportedError('Pass batchFromJson to persist this block'); + } + return {'value': state.value?.toJson()}; +} + +_EdgeState _decodeEdge>( + Object? json, B Function(Object?)? batchFromJson) { + if (batchFromJson == null) { + throw UnsupportedError('Pass batchFromJson to persist this block'); + } + final raw = field(decodeMap(json), 'value'); + final state = _EdgeState(); + if (raw != null) { + final row = batchFromJson(raw); + if (row.length != 1) throw const FormatException('Expected one row'); + state.value = row; + } + return state; +} + +/// The first row as a one-row copy, or null for no rows. +class First> extends Aggregator<_EdgeState, B, B?> { + /// Persistable only when [batchFromJson] is given (the inverse of + /// [Batch.toJson] for [B]). + const First({this.batchFromJson}); + final B Function(Object? json)? batchFromJson; + @override + _EdgeState empty() => _EdgeState(); + @override + _EdgeState addSlice(_EdgeState state, B batch, int start, int end) { + if (state.value == null && start < end) { + state.value = batch.copyRange(start, start + 1); + } + return state; + } + @override + _EdgeState combine(_EdgeState left, _EdgeState right) { + if (left.value == null) left.value = right.value; + return left; + } + @override + B? finish(_EdgeState state) => state.value; + @override + bool get persistable => batchFromJson != null; + @override + Object? encodeState(_EdgeState state) => _encodeEdge(state, batchFromJson); + @override + _EdgeState decodeState(Object? json) => _decodeEdge(json, batchFromJson); +} + +/// The last row as a one-row copy, or null for no rows. +class Last> extends Aggregator<_EdgeState, B, B?> { + /// Persistable only when [batchFromJson] is given (the inverse of + /// [Batch.toJson] for [B]). + const Last({this.batchFromJson}); + final B Function(Object? json)? batchFromJson; + @override + _EdgeState empty() => _EdgeState(); + @override + _EdgeState addSlice(_EdgeState state, B batch, int start, int end) { + if (start < end) state.value = batch.copyRange(end - 1, end); + return state; + } + @override + _EdgeState combine(_EdgeState left, _EdgeState right) { + if (right.value != null) left.value = right.value; + return left; + } + @override + B? finish(_EdgeState state) => state.value; + @override + bool get persistable => batchFromJson != null; + @override + Object? encodeState(_EdgeState state) => _encodeEdge(state, batchFromJson); + @override + _EdgeState decodeState(Object? json) => _decodeEdge(json, batchFromJson); +} + +class _RowsState> { + B? rows; +} + +/// The last [k] rows of the whole input as a copy (fewer when the input is +/// shorter), or null for no rows. +class LastRows> extends Aggregator<_RowsState, B, B?> { + /// Persistable only when [batchFromJson] is given (the inverse of + /// [Batch.toJson] for [B]). + const LastRows(this.k, {this.batchFromJson}) : assert(k >= 1); + final int k; + final B Function(Object? json)? batchFromJson; + + B _tail(B rows) => + rows.length <= k ? rows : rows.copyRange(rows.length - k, rows.length); + + @override + _RowsState empty() => _RowsState(); + @override + _RowsState addSlice(_RowsState state, B batch, int start, int end) { + if (start >= end) return state; + final from = end - start > k ? end - k : start; + final piece = batch.copyRange(from, end); + final rows = state.rows; + state.rows = rows == null ? piece : _tail(rows.join(piece)); + return state; + } + @override + _RowsState combine(_RowsState left, _RowsState right) { + final l = left.rows; + final r = right.rows; + if (r == null) return left; + left.rows = l == null ? r : _tail(l.join(r)); + return left; + } + @override + B? finish(_RowsState state) => state.rows; + @override + bool get persistable => batchFromJson != null; + @override + Object? encodeState(_RowsState state) { + if (batchFromJson == null) { + throw UnsupportedError('Pass batchFromJson to persist this block'); + } + return {'rows': state.rows?.toJson()}; + } + @override + _RowsState decodeState(Object? json) { + final fromJson = batchFromJson; + if (fromJson == null) { + throw UnsupportedError('Pass batchFromJson to persist this block'); + } + final raw = field(decodeMap(json), 'rows'); + final state = _RowsState(); + if (raw != null) { + final rows = fromJson(raw); + if (rows.length < 1 || rows.length > k) { + throw const FormatException('Row count out of range'); + } + state.rows = rows; + } + return state; + } +} + +/// Routes consecutive equal keys to [inner] as one slice per run. +class GroupBy, S, R> + extends Aggregator, B, Map> { + const GroupBy(this.key, this.inner); + final int Function(B batch, int row) key; + final Aggregator inner; + @override + Map empty() => {}; + @override + Map addSlice(Map state, B batch, int start, int end) { + if (start == end) return state; + var runStart = start; + var runKey = key(batch, start); + for (var i = start + 1; i < end; i++) { + final nextKey = key(batch, i); + if (nextKey != runKey) { + _addRun(state, runKey, batch, runStart, i); + runStart = i; + runKey = nextKey; + } + } + _addRun(state, runKey, batch, runStart, end); + return state; + } + void _addRun(Map state, int runKey, B batch, int start, int end) { + final previous = state.containsKey(runKey) ? state[runKey] as S : inner.empty(); + state[runKey] = inner.addSlice(previous, batch, start, end); + } + @override + Map combine(Map left, Map right) { + for (final entry in right.entries) { + if (left.containsKey(entry.key)) { + left[entry.key] = inner.combine(left[entry.key] as S, entry.value); + } else { + left[entry.key] = entry.value; + } + } + return left; + } + @override + Map finish(Map state) => { + for (final entry in state.entries) entry.key: inner.finish(entry.value), + }; + @override + bool get persistable => inner.persistable; + + @override + Object? encodeState(Map state) => { + for (final entry in state.entries) + entry.key.toString(): inner.encodeState(entry.value), + }; + + @override + Map decodeState(Object? json) { + final map = decodeMap(json); + final result = {}; + for (final entry in map.entries) { + final key = int.tryParse(entry.key); + if (key == null || key.toString() != entry.key) { + throw FormatException('Invalid group key ${entry.key}'); + } + result[key] = inner.decodeState(entry.value); + } + return result; + } +} + +/// Makes an inner length-[k] window aggregator invariant to chunk boundaries. +class Lookback, S, R> + extends Aggregator<_LookbackState, B, R> { + /// Persistable only when [batchFromJson] is given (the inverse of + /// [Batch.toJson] for [B]) and [inner] is persistable. + Lookback(this.k, this.inner, {this.batchFromJson}) : assert(k >= 1); + final int k; + final Aggregator inner; + final B Function(Object? json)? batchFromJson; + int get _edge => k - 1; + @override + _LookbackState empty() => _LookbackState(inner.empty()); + @override + _LookbackState addSlice( + _LookbackState state, B batch, int start, int end) { + if (start == end) return state; + if (state.length > 0 && _edge > 0) { + final seam = state.last!.join(batch.copyRange(start, _prefixEnd(start, end))); + state.innerState = inner.addSlice(state.innerState, seam, 0, seam.length); + } + state.innerState = inner.addSlice(state.innerState, batch, start, end); + _appendEdges(state, batch, start, end); + return state; + } + int _prefixEnd(int start, int end) => + start + (end - start < _edge ? end - start : _edge); + void _appendEdges(_LookbackState state, B batch, int start, int end) { + final sliceLength = end - start; + if (_edge == 0) { + state.length += sliceLength; + return; + } + if (state.length == 0) { + state.first = batch.copyRange(start, _prefixEnd(start, end)); + state.last = batch.copyRange( + end - (sliceLength < _edge ? sliceLength : _edge), end); + } else { + if (state.length < _edge) { + final needed = _edge - state.length; + final take = sliceLength < needed ? sliceLength : needed; + if (take > 0) state.first = state.first!.join(batch.copyRange(start, start + take)); + } + if (sliceLength >= _edge) { + state.last = batch.copyRange(end - _edge, end); + } else { + final joined = state.last!.join(batch.copyRange(start, end)); + final keepStart = joined.length > _edge ? joined.length - _edge : 0; + state.last = joined.copyRange(keepStart, joined.length); + } + } + state.length += sliceLength; + } + @override + _LookbackState combine( + _LookbackState left, _LookbackState right) { + if (left.length > 0 && right.length > 0 && _edge > 0) { + final seam = left.last!.join(right.first!); + left.innerState = inner.addSlice(left.innerState, seam, 0, seam.length); + } + left.innerState = inner.combine(left.innerState, right.innerState); + _combineEdges(left, right); + return left; + } + void _combineEdges(_LookbackState left, _LookbackState right) { + if (right.length == 0) return; + if (left.length == 0) { + left.length = right.length; + left.first = right.first; + left.last = right.last; + return; + } + if (_edge > 0) { + if (left.length < _edge) { + final needed = _edge - left.length; + final take = right.length < needed ? right.length : needed; + if (take > 0) left.first = left.first!.join(right.first!.copyRange(0, take)); + } + if (right.length >= _edge) { + left.last = right.last; + } else { + final joined = left.last!.join(right.last!); + final keepStart = joined.length > _edge ? joined.length - _edge : 0; + left.last = joined.copyRange(keepStart, joined.length); + } + } + left.length += right.length; + } + @override + R finish(_LookbackState state) => inner.finish(state.innerState); + + @override + bool get persistable => batchFromJson != null && inner.persistable; + + @override + Object? encodeState(_LookbackState state) { + _requireBatchCodec(); + return { + 'length': state.length, + 'first': state.first?.toJson(), + 'last': state.last?.toJson(), + 'inner': inner.encodeState(state.innerState), + }; + } + + @override + _LookbackState decodeState(Object? json) { + final fromJson = _requireBatchCodec(); + final map = decodeMap(json); + final length = decodeCount(field(map, 'length')); + // Edges hold the first/last min(length, k - 1) rows. + final held = length < _edge ? length : _edge; + B? edge(String name) { + final raw = field(map, name); + if (raw == null) { + if (held != 0) throw FormatException('Missing $name rows'); + return null; + } + if (held == 0) throw FormatException('Unexpected $name rows'); + final rows = fromJson(raw); + if (rows.length != held) { + throw FormatException('Expected $held $name rows'); + } + return rows; + } + + return _LookbackState(inner.decodeState(field(map, 'inner'))) + ..length = length + ..first = edge('first') + ..last = edge('last'); + } + + B Function(Object?) _requireBatchCodec() => + batchFromJson ?? + (throw UnsupportedError('Pass batchFromJson to persist Lookback')); +} + +class _LookbackState, S> { + _LookbackState(this.innerState); + S innerState; + int length = 0; + B? first; + B? last; +} + +/// An affine map x -> scale * x + offset. Composition is associative, so a +/// run of affine steps can be summarised by one map and merged in any grouping. +class Affine { + const Affine(this.scale, this.offset); + + final double scale; + final double offset; + + double apply(double x) => scale * x + offset; + + /// The map that applies this one first, then [next]. + Affine then(Affine next) => Affine( + next.scale * scale, next.scale * offset + next.offset); +} + +class _AffineState { + double scale = 1; + double offset = 0; +} + +// Algebraically equal to the EWMA recurrence: f_x(c)=a*c+lambda*x, a=1-lambda; +// f_y(f_x(c))=a*a*c+a*lambda*x+lambda*y. Floating-point order may differ. +/// An exponentially weighted moving average, as the composed affine map of the +/// running value: each row x applies c -> (1 - lambda) * c + lambda * x. +/// +/// NaN rows are skipped (they act as the identity map), and no rows finish as +/// the identity. ±Infinity input is unsupported because scale can underflow to +/// zero across a seam and yield `0*Infinity`. Apply the result to the seed +/// value to get the final average. +class Ewma> extends Aggregator<_AffineState, B, Affine> { + const Ewma(this.column, this.lambda); + + final Float64List Function(B batch) column; + final double lambda; + + @override + _AffineState empty() => _AffineState(); + + @override + _AffineState addSlice(_AffineState state, B batch, int start, int end) { + final values = column(batch); + final keep = 1 - lambda; + var scale = state.scale; + var offset = state.offset; + for (var i = start; i < end; i++) { + final x = values[i]; + if (x.isNaN) continue; + scale *= keep; + offset = keep * offset + lambda * x; + } + state.scale = scale; + state.offset = offset; + return state; + } + + @override + _AffineState combine(_AffineState left, _AffineState right) { + // Right's map runs after left's. + left.offset = right.scale * left.offset + right.offset; + left.scale = right.scale * left.scale; + return left; + } + + @override + Affine finish(_AffineState state) => Affine(state.scale, state.offset); + @override + bool get persistable => true; + + @override + Object? encodeState(_AffineState state) => { + 'scale': encodeDouble(state.scale), + 'offset': encodeDouble(state.offset), + }; + + @override + _AffineState decodeState(Object? json) { + final map = decodeMap(json); + return _AffineState() + ..scale = decodeDouble(field(map, 'scale')) + ..offset = decodeDouble(field(map, 'offset')); + } +} + +class _PrefixState, S2> { + _PrefixState(this.tail); + + /// The first rows of the input seen so far (at most k), null before any. + B? rows; + int held = 0; + S2 tail; +} + +/// Sends the first [k] rows of the whole input to [head] and every later row +/// to [tail], in order, however the input is chunked. +/// +/// The state keeps the prefix rows (a copy, at most k) and the tail state. +/// The head result is computed at finish by running [head] over the stored +/// rows, so [head] never needs a working combine. With fewer than k rows in +/// total, head sees all of them and the tail sees none. +class Prefix, S1, R1, S2, R2> + extends Aggregator<_PrefixState, B, (R1, R2)> { + /// Persistable only when [batchFromJson] is given (the inverse of + /// [Batch.toJson] for [B]) and [tail] is persistable. [head] keeps no state, + /// so it does not matter. + const Prefix(this.k, this.head, this.tail, {this.batchFromJson}); + + final int k; + final Aggregator head; + final Aggregator tail; + final B Function(Object? json)? batchFromJson; + + @override + _PrefixState empty() => _PrefixState(tail.empty()); + + /// Appends [piece] to the stored prefix rows. + void _hold(_PrefixState state, B piece) { + final rows = state.rows; + state.rows = rows == null ? piece : rows.join(piece); + state.held += piece.length; + } + + @override + _PrefixState addSlice( + _PrefixState state, B batch, int start, int end) { + final room = k - state.held; + final take = room < end - start ? room : end - start; + if (take > 0) _hold(state, batch.copyRange(start, start + take)); + if (start + take < end) { + state.tail = tail.addSlice(state.tail, batch, start + take, end); + } + return state; + } + + @override + _PrefixState combine( + _PrefixState left, _PrefixState right) { + final rightRows = right.rows; + var rest = right.tail; + if (rightRows != null) { + // Right's leading rows fill whatever room left's prefix still has; the + // remainder are tail rows that come before right's own tail. + final room = k - left.held; + final take = room < right.held ? room : right.held; + if (take > 0) _hold(left, rightRows.copyRange(0, take)); + if (take < right.held) { + final extra = + tail.addSlice(tail.empty(), rightRows, take, right.held); + rest = tail.combine(extra, rest); + } + } + left.tail = tail.combine(left.tail, rest); + return left; + } + + @override + (R1, R2) finish(_PrefixState state) { + final rows = state.rows; + final headState = rows == null + ? head.empty() + : head.addSlice(head.empty(), rows, 0, rows.length); + return (head.finish(headState), tail.finish(state.tail)); + } + + @override + bool get persistable => batchFromJson != null && tail.persistable; + + @override + Object? encodeState(_PrefixState state) { + _requireBatchCodec(); + return { + 'rows': state.rows?.toJson(), + 'tail': tail.encodeState(state.tail), + }; + } + + @override + _PrefixState decodeState(Object? json) { + final fromJson = _requireBatchCodec(); + final map = decodeMap(json); + final raw = field(map, 'rows'); + final result = _PrefixState(tail.decodeState(field(map, 'tail'))); + if (raw != null) { + final rows = fromJson(raw); + if (rows.length < 1 || rows.length > k) { + throw const FormatException('Invalid prefix row count'); + } + result.rows = rows; + result.held = rows.length; + } + return result; + } + + B Function(Object?) _requireBatchCodec() => + batchFromJson ?? + (throw UnsupportedError('Pass batchFromJson to persist Prefix')); +} diff --git a/lib/src/onehz/clinical/cosinor.dart b/lib/src/onehz/clinical/cosinor.dart index aeda025..cfc0037 100644 --- a/lib/src/onehz/clinical/cosinor.dart +++ b/lib/src/onehz/clinical/cosinor.dart @@ -10,9 +10,11 @@ // the daytime peak — callers choose the period and supply trough-anchored // times; this engine just fits. R² is reported so a poor fit is visible. -import 'dart:math' as math; +import 'dart:typed_data'; +import '../../aggregate/aggregator.dart'; import '../types.dart'; import '../util.dart'; +import 'cosinor_aggregator.dart'; class CosinorFit { final double mesor; // rhythm-adjusted mean @@ -64,120 +66,18 @@ Metric cosinor( double periodHours = 24, int minPoints = cosinorMinPoints, }) { - const inputs = ['signal_timeseries']; - final n = y.length; - if (n < math.max(4, minPoints) || tHours.length != n || periodHours <= 0) { + // Runs [cosinorAggregator] over the whole series in one slice. Mismatched + // lengths cannot be a [TimedValues] batch; they are the same "too few + // points" refusal as before. + if (tHours.length != y.length) { return const Metric.absent( tier: Tier.high, - inputs_used: inputs, + inputs_used: ['signal_timeseries'], note: 'too few points for cosinor', ); } - final w = 2 * math.pi / periodHours; - final c = [for (final t in tHours) math.cos(w * t)]; - final s = [for (final t in tHours) math.sin(w * t)]; - - // Normal equations for [M, β, γ] of y = M + β·c + γ·s. - // Build 3×3 symmetric system. - var sC = 0.0, sS = 0.0, sCC = 0.0, sSS = 0.0, sCS = 0.0; - var sY = 0.0, sYC = 0.0, sYS = 0.0; - for (var i = 0; i < n; i++) { - sC += c[i]; - sS += s[i]; - sCC += c[i] * c[i]; - sSS += s[i] * s[i]; - sCS += c[i] * s[i]; - sY += y[i]; - sYC += y[i] * c[i]; - sYS += y[i] * s[i]; - } - // Matrix A = [[n, sC, sS],[sC, sCC, sCS],[sS, sCS, sSS]], b=[sY,sYC,sYS]. - final a = [ - [n.toDouble(), sC, sS], - [sC, sCC, sCS], - [sS, sCS, sSS], - ]; - final b = [sY, sYC, sYS]; - final sol = _solve3(a, b); - if (sol == null) { - return const Metric.absent( - tier: Tier.high, - inputs_used: inputs, - note: 'singular cosinor design (insufficient phase coverage)', - ); - } - final mesor = sol[0]; - final beta = sol[1]; - final gamma = sol[2]; - final amplitude = math.sqrt(beta * beta + gamma * gamma); - final acro = math.atan2(-gamma, beta); // peak phase - // Convert acrophase to clock-hours within the period (0..period). - var phaseHours = -acro / w; // time of peak relative to origin - phaseHours = phaseHours % periodHours; - if (phaseHours < 0) phaseHours += periodHours; - - // R². - final yMean = sY / n; - var ssTot = 0.0, ssRes = 0.0; - for (var i = 0; i < n; i++) { - final fit = mesor + beta * c[i] + gamma * s[i]; - ssTot += (y[i] - yMean) * (y[i] - yMean); - ssRes += (y[i] - fit) * (y[i] - fit); - } - final double r2 = ssTot == 0 ? 0.0 : (1 - ssRes / ssTot).clamp(0, 1); - // Adjusted for the 3 fitted parameters (M, β, γ). Confidence MUST come from - // the adjusted value: the raw R² of a 3-parameter fit is upward-biased - // (E[R²] = 2/(n−1) under the null), so a handful of noise points used to - // score confidence 0.95 at tier HIGH. - final double r2Adj = (1 - (1 - r2) * (n - 1) / (n - 3)).clamp(0, 1); - - final conf = r2Adj.clamp(0.1, 0.95); - return Metric( - value: CosinorFit( - mesor: mesor, - amplitude: amplitude, - acrophaseRad: acro, - acrophaseHours: phaseHours, - periodHours: periodHours, - r2: r2, - r2Adj: r2Adj, - ), - confidence: conf, - tier: Tier.high, - inputs_used: inputs, - note: 'single-component cosinor; anchor circadian phase on the nocturnal ' - 'trough, not the daytime peak', - ); + return cosinorAggregator(periodHours: periodHours, minPoints: minPoints) + .run(TimedValues( + Float64List.fromList(tHours), Float64List.fromList(y))); } -/// Solve a 3×3 linear system by Gaussian elimination with partial pivoting. -/// Returns null if singular. -List? _solve3(List> a, List b) { - final m = [ - [a[0][0], a[0][1], a[0][2], b[0]], - [a[1][0], a[1][1], a[1][2], b[1]], - [a[2][0], a[2][1], a[2][2], b[2]], - ]; - for (var col = 0; col < 3; col++) { - var piv = col; - for (var r = col + 1; r < 3; r++) { - if (m[r][col].abs() > m[piv][col].abs()) piv = r; - } - if (m[piv][col].abs() < 1e-12) return null; - final tmp = m[col]; - m[col] = m[piv]; - m[piv] = tmp; - final d = m[col][col]; - for (var j = col; j < 4; j++) { - m[col][j] /= d; - } - for (var r = 0; r < 3; r++) { - if (r == col) continue; - final f = m[r][col]; - for (var j = col; j < 4; j++) { - m[r][j] -= f * m[col][j]; - } - } - } - return [m[0][3], m[1][3], m[2][3]]; -} diff --git a/lib/src/onehz/clinical/cosinor_aggregator.dart b/lib/src/onehz/clinical/cosinor_aggregator.dart new file mode 100644 index 0000000..59923df --- /dev/null +++ b/lib/src/onehz/clinical/cosinor_aggregator.dart @@ -0,0 +1,259 @@ +import 'dart:math' as math; +import 'dart:typed_data'; + +import '../../aggregate/aggregator.dart'; +import '../../aggregate/json_support.dart'; +import '../types.dart'; +import 'cosinor.dart'; + +/// Sample times and values for a cosinor fit. +class TimedValues extends Batch { + TimedValues(this.tHours, this.y) : assert(tHours.length == y.length); + + factory TimedValues.fromJson(Object? json) { + final map = decodeMap(json); + final times = decodeDoubles(field(map, 'tHours')); + final values = decodeDoubles(field(map, 'y')); + if (times.length != values.length) { + throw const FormatException('TimedValues columns differ in length'); + } + return TimedValues(times, values); + } + + final Float64List tHours; + final Float64List y; + + @override + Object? toJson() => + {'tHours': encodeDoubles(tHours), 'y': encodeDoubles(y)}; + + @override + int get length => y.length; + + @override + TimedValues copyRange(int start, int end) => TimedValues( + Float64List(end - start)..setRange(0, end - start, tHours, start), + Float64List(end - start)..setRange(0, end - start, y, start), + ); + + @override + TimedValues join(TimedValues other) { + final times = Float64List(length + other.length); + final values = Float64List(length + other.length); + times + ..setRange(0, length, tHours) + ..setRange(length, times.length, other.tHours); + values + ..setRange(0, length, y) + ..setRange(length, values.length, other.y); + return TimedValues(times, values); + } +} + +class _CosinorState { + var n = 0; + var meanC = 0.0; + var meanS = 0.0; + var meanY = 0.0; + var mCC = 0.0; + var mSS = 0.0; + var mYY = 0.0; + var mCS = 0.0; + var mCY = 0.0; + var mSY = 0.0; + + void addCentered(double c, double s, double y) { + final nextN = n + 1; + final dc = c - meanC; + final ds = s - meanS; + final dy = y - meanY; + final nextMeanC = meanC + dc / nextN; + final nextMeanS = meanS + ds / nextN; + final nextMeanY = meanY + dy / nextN; + mCC += dc * (c - nextMeanC); + mSS += ds * (s - nextMeanS); + mYY += dy * (y - nextMeanY); + mCS += dc * (s - nextMeanS); + mCY += dc * (y - nextMeanY); + mSY += ds * (y - nextMeanY); + meanC = nextMeanC; + meanS = nextMeanS; + meanY = nextMeanY; + n = nextN; + } + + void mergeCentered(_CosinorState other) { + if (other.n == 0) return; + if (n == 0) { + meanC = other.meanC; + meanS = other.meanS; + meanY = other.meanY; + mCC = other.mCC; + mSS = other.mSS; + mYY = other.mYY; + mCS = other.mCS; + mCY = other.mCY; + mSY = other.mSY; + n = other.n; + return; + } + final total = n + other.n; + final factor = n * other.n / total; + final dc = other.meanC - meanC; + final ds = other.meanS - meanS; + final dy = other.meanY - meanY; + mCC += other.mCC + dc * dc * factor; + mSS += other.mSS + ds * ds * factor; + mYY += other.mYY + dy * dy * factor; + mCS += other.mCS + dc * ds * factor; + mCY += other.mCY + dc * dy * factor; + mSY += other.mSY + ds * dy * factor; + meanC += dc * other.n / total; + meanS += ds * other.n / total; + meanY += dy * other.n / total; + n = total; + } +} + +class _CosinorStats + extends Aggregator<_CosinorState, TimedValues, _CosinorState> { + const _CosinorStats(this.periodHours); + + final double periodHours; + + @override + _CosinorState empty() => _CosinorState(); + + @override + _CosinorState addSlice( + _CosinorState state, TimedValues batch, int start, int end) { + final w = 2 * math.pi / periodHours; + for (var i = start; i < end; i++) { + final c = math.cos(w * batch.tHours[i]); + final s = math.sin(w * batch.tHours[i]); + final y = batch.y[i]; + state.addCentered(c, s, y); + } + return state; + } + + @override + _CosinorState combine(_CosinorState left, _CosinorState right) { + left.mergeCentered(right); + return left; + } + + @override + _CosinorState finish(_CosinorState state) => state; + + @override + bool get persistable => true; + + @override + Object? encodeState(_CosinorState state) => { + 'n': state.n, + 'meanC': encodeDouble(state.meanC), + 'meanS': encodeDouble(state.meanS), + 'meanY': encodeDouble(state.meanY), + 'mCC': encodeDouble(state.mCC), + 'mSS': encodeDouble(state.mSS), + 'mYY': encodeDouble(state.mYY), + 'mCS': encodeDouble(state.mCS), + 'mCY': encodeDouble(state.mCY), + 'mSY': encodeDouble(state.mSY), + }; + + @override + _CosinorState decodeState(Object? json) { + final map = decodeMap(json); + return _CosinorState() + ..n = decodeCount(field(map, 'n')) + ..meanC = decodeDouble(field(map, 'meanC')) + ..meanS = decodeDouble(field(map, 'meanS')) + ..meanY = decodeDouble(field(map, 'meanY')) + ..mCC = decodeDouble(field(map, 'mCC')) + ..mSS = decodeDouble(field(map, 'mSS')) + ..mYY = decodeDouble(field(map, 'mYY')) + ..mCS = decodeDouble(field(map, 'mCS')) + ..mCY = decodeDouble(field(map, 'mCY')) + ..mSY = decodeDouble(field(map, 'mSY')); + } +} + +// Halberg & Nelson 1979 single-component cosinor. Centered co-moments solve the +// same least-squares model. cosinor() now runs this aggregator in one slice. +// Floating-point order differs from the former batch solver; the relative +// determinant gate replaces its absolute 1e-12 pivot gate. See +// test/aggregate/cosinor_aggregator_test.dart for numerical tolerances and phase +// degeneracies. +/// Builds a mergeable single-component cosinor fit over [TimedValues]. +Aggregator<_CosinorState, TimedValues, Metric> cosinorAggregator({ + double periodHours = 24, + int minPoints = cosinorMinPoints, +}) => + _CosinorStats(periodHours).map((state) => + _finishCosinor(state, periodHours: periodHours, minPoints: minPoints)); + +Metric _finishCosinor( + _CosinorState state, { + required double periodHours, + required int minPoints, +}) { + const inputs = ['signal_timeseries']; + final n = state.n; + if (n < math.max(4, minPoints) || periodHours <= 0) { + return const Metric.absent( + tier: Tier.high, + inputs_used: inputs, + note: 'too few points for cosinor', + ); + } + final determinant = state.mCC * state.mSS - state.mCS * state.mCS; + // The determinant is relative to the diagonal co-moments. A design needs + // at least 1e-12 of that scale after removing its means to identify both + // cosine coefficients; below that, numerical noise decides the fit. + if (determinant <= 1e-12 * state.mCC * state.mSS) { + return const Metric.absent( + tier: Tier.high, + inputs_used: inputs, + note: 'singular cosinor design (insufficient phase coverage)', + ); + } + final beta = + (state.mCY * state.mSS - state.mSY * state.mCS) / determinant; + final gamma = + (state.mSY * state.mCC - state.mCY * state.mCS) / determinant; + final mesor = state.meanY - beta * state.meanC - gamma * state.meanS; + final amplitude = math.sqrt(beta * beta + gamma * gamma); + final acro = math.atan2(-gamma, beta); + final w = 2 * math.pi / periodHours; + var phaseHours = -acro / w; + phaseHours = phaseHours % periodHours; + if (phaseHours < 0) phaseHours += periodHours; + + final ssTot = state.mYY; + final ssRes = state.mYY - + 2 * beta * state.mCY - + 2 * gamma * state.mSY + + beta * beta * state.mCC + + gamma * gamma * state.mSS + + 2 * beta * gamma * state.mCS; + final double r2 = ssTot == 0 ? 0.0 : (1 - ssRes / ssTot).clamp(0, 1); + final double r2Adj = (1 - (1 - r2) * (n - 1) / (n - 3)).clamp(0, 1); + return Metric( + value: CosinorFit( + mesor: mesor, + amplitude: amplitude, + acrophaseRad: acro, + acrophaseHours: phaseHours, + periodHours: periodHours, + r2: r2, + r2Adj: r2Adj, + ), + confidence: r2Adj.clamp(0.1, 0.95), + tier: Tier.high, + inputs_used: inputs, + note: 'single-component cosinor; anchor circadian phase on the nocturnal ' + 'trough, not the daytime peak', + ); +} diff --git a/lib/src/onehz/clinical/hrv_freq.dart b/lib/src/onehz/clinical/hrv_freq.dart index d759248..e0a8b7a 100644 --- a/lib/src/onehz/clinical/hrv_freq.dart +++ b/lib/src/onehz/clinical/hrv_freq.dart @@ -19,6 +19,9 @@ import '../types.dart'; import '../util.dart'; +import '../../aggregate/json_support.dart'; + +part 'hrv_freq_stream.dart'; class HrvFreq { final double? ulf; @@ -84,6 +87,15 @@ Metric hrvFreq( // Times in seconds for Hz output; tachogram value = NN (ms). final tSec = [for (final t in nnTimesMs) t / 1000.0]; + // A non-finite beat time has no place on the clock, and an infinite last + // time made the segment walk endless (it never reaches infinity). + if (tSec.any((t) => !t.isFinite)) { + return const Metric.absent( + tier: Tier.high, + inputs_used: inputs, + note: 'non-finite beat time', + ); + } final spanSec = tSec.last - tSec.first; if (spanSec <= 0) { return const Metric.absent( @@ -102,6 +114,19 @@ Metric hrvFreq( final vlf = _welchBandPower(tSec, nnMs, 0.003, 0.04, oversample: oversample); final lf = _welchBandPower(tSec, nnMs, 0.04, 0.15, oversample: oversample); final hfRaw = _welchBandPower(tSec, nnMs, 0.15, 0.40, oversample: oversample); + return _hrvFreqMetric(ulf, vlf, lf, hfRaw, + artifactFraction: artifactFraction, hfArtifactGate: hfArtifactGate); +} + +Metric _hrvFreqMetric( + double? ulf, + double? vlf, + double? lf, + double? hfRaw, { + required double artifactFraction, + required double hfArtifactGate, +}) { + const inputs = ['rr_cleaned', 'beat_times']; if (lf == null && hfRaw == null) { return const Metric.absent( tier: Tier.high, @@ -222,35 +247,46 @@ double? _welchBandPower( var k = 0; final step = segSec / 2; // 50 % overlap, the Welch default for (var start = tSec.first; start + segSec <= tSec.last; start += step) { - final end = start + segSec; - final ts = []; - final ys = []; - for (var i = 0; i < tSec.length; i++) { - if (tSec[i] < start) continue; - if (tSec[i] >= end) break; - ts.add(tSec[i]); - ys.add(y[i]); - } - // A segment has to be BOTH beat-dense and time-complete: a dropout in - // the middle leaves few beats spanning the full window, and its - // periodogram is a window function, not a spectrum. Endpoint span alone - // misses an internal dropout that still leaves beats near both edges, so - // also reject on the single largest gap between consecutive beats. - if (ts.length < minPointsPerSegment || ts.last - ts.first < segSec * 0.8) { - continue; - } - var maxGap = 0.0; - for (var i = 1; i < ts.length; i++) { - final g = ts[i] - ts[i - 1]; - if (g > maxGap) maxGap = g; - } - if (maxGap > segSec * 0.2) continue; - final ls = lombScargle(ts, ys, grid); - if (ls == null) continue; - final p = ls.bandPower(loHz, hiHz); + final p = _welchSegmentPower(tSec, y, start, segSec, loHz, hiHz, + grid: grid, minPointsPerSegment: minPointsPerSegment); + if (p == null) continue; if (!p.isFinite) continue; sum += p; k++; } return k == 0 ? null : sum / k; } + +/// One Welch segment. Streams call this exact function when a segment closes. +double? _welchSegmentPower( + List tSec, + List y, + double start, + double segSec, + double loHz, + double hiHz, { + required List grid, + required int minPointsPerSegment, +}) { + final end = start + segSec; + final ts = []; + final ys = []; + for (var i = 0; i < tSec.length; i++) { + if (tSec[i] < start) continue; + if (tSec[i] >= end) break; + ts.add(tSec[i]); + ys.add(y[i]); + } + if (ts.length < minPointsPerSegment || ts.last - ts.first < segSec * 0.8) { + return null; + } + var maxGap = 0.0; + for (var i = 1; i < ts.length; i++) { + final g = ts[i] - ts[i - 1]; + if (g > maxGap) maxGap = g; + } + if (maxGap > segSec * 0.2) return null; + final ls = lombScargle(ts, ys, grid); + if (ls == null) return null; + return ls.bandPower(loHz, hiHz); +} diff --git a/lib/src/onehz/clinical/hrv_freq_stream.dart b/lib/src/onehz/clinical/hrv_freq_stream.dart new file mode 100644 index 0000000..bdbb222 --- /dev/null +++ b/lib/src/onehz/clinical/hrv_freq_stream.dart @@ -0,0 +1,231 @@ +part of 'hrv_freq.dart'; + +class _HrvBandState { + final double loHz, hiHz, segSec; + double nextStart; + double sum; + int count; + + _HrvBandState(this.loHz, this.hiHz, this.nextStart, + {this.sum = 0.0, this.count = 0}) + : segSec = 10.0 / loHz; + + double? value(List times, List nn, double last, + double oversample) { + if (last - times.first < segSec) return null; + var total = sum; + var n = count; + final grid = _welchGrid(loHz, hiHz, segSec, oversample); + for (var start = nextStart; start + segSec <= last; start += segSec / 2) { + final p = _welchSegmentPower(times, nn, start, segSec, loHz, hiHz, + grid: grid, minPointsPerSegment: 16); + if (p != null && p.isFinite) { + total += p; + n++; + } + } + return n == 0 ? null : total / n; + } + + void settle(List times, List nn, double last, + double oversample) { + final grid = _welchGrid(loHz, hiHz, segSec, oversample); + while (nextStart + segSec <= last) { + final p = _welchSegmentPower(times, nn, nextStart, segSec, loHz, hiHz, + grid: grid, minPointsPerSegment: 16); + if (p != null && p.isFinite) { + sum += p; + count++; + } + nextStart += segSec / 2; + } + } + + Map toJson() => { + 'nextStart': encodeDouble(nextStart), + 'sum': encodeDouble(sum), + 'count': count, + }; +} + +List _welchGrid(double lo, double hi, double seg, double oversample) { + final df = 1.0 / seg / oversample; + return freqGrid(lo, hi, ((hi - lo) / df).ceil() + 1); +} + +// Laguna, Moody & Mark 1998 / Bigger 1992 HRV with Welch 1967 segmentation, as +// in hrvFreq(). Closed segments use the exact shared segment function; open data +// is evaluated provisionally with the same origin, grid and segment order. +/// Exact, append-only Welch state for [hrvFreq]. +class HrvFreqStream { + final double oversample; + final List _nn = [], _times = []; + late final List<_HrvBandState> _bands; + int _n = 0; + double? _first, _last; + + HrvFreqStream({this.oversample = 4.0}) { + _bands = [ + _HrvBandState(0.0003, 0.003, 0), + _HrvBandState(0.003, 0.04, 0), + _HrvBandState(0.04, 0.15, 0), + _HrvBandState(0.15, 0.40, 0), + ]; + } + + factory HrvFreqStream.fromJson(Map json) { + try { + final j = decodeMap(json); + if (field(j, 'type') != 'HrvFreqStream') { + throw const FormatException('not an HrvFreqStream checkpoint'); + } + if (field(j, 'version') != 1) { + throw const FormatException('unsupported HrvFreqStream version'); + } + final oversample = decodeDouble(field(j, 'oversample')); + final n = decodeCount(field(j, 'n')); + final first = decodeDoubleOrNull(field(j, 'first')); + final last = decodeDoubleOrNull(field(j, 'last')); + final nn = decodeDoubles(field(j, 'nn')); + final times = decodeDoubles(field(j, 'times')); + final encodedBands = decodeList(field(j, 'bands')); + if (encodedBands.length != 4 || nn.length != times.length || + !oversample.isFinite || oversample <= 0 || + (n == 0 && (first != null || last != null)) || + (n > 0 && (first == null || last == null))) { + throw const FormatException('inconsistent HrvFreqStream checkpoint'); + } + final s = HrvFreqStream(oversample: oversample) + .._n = n + .._first = first + .._last = last + .._nn.addAll(nn) + .._times.addAll(times); + if (!_nondecreasing(s._times)) { + throw const FormatException('decreasing checkpoint times'); + } + final configs = const [ + [0.0003, 0.003], + [0.003, 0.04], + [0.04, 0.15], + [0.15, 0.40], + ]; + s._bands.clear(); + for (var i = 0; i < configs.length; i++) { + final b = decodeMap(encodedBands[i]); + final next = decodeDouble(field(b, 'nextStart')); + final sum = decodeDouble(field(b, 'sum')); + final count = decodeCount(field(b, 'count')); + s._bands.add(_HrvBandState(configs[i][0], configs[i][1], next, + sum: sum, count: count)); + } + return s; + } on FormatException { + rethrow; + } catch (e) { + throw FormatException('malformed HrvFreqStream checkpoint: $e'); + } + } + + Map toJson() => { + 'type': 'HrvFreqStream', + 'version': 1, + 'oversample': encodeDouble(oversample), + 'n': _n, + 'first': _first == null ? null : encodeDouble(_first!), + 'last': _last == null ? null : encodeDouble(_last!), + 'nn': [for (final x in _nn) encodeDouble(x)], + 'times': [for (final x in _times) encodeDouble(x)], + 'bands': [for (final b in _bands) b.toJson()], + }; + + void fold(List nnMs, List nnTimesMs) { + if (nnMs.length != nnTimesMs.length) { + throw ArgumentError('nnMs and nnTimesMs must have the same length'); + } + if (nnMs.isEmpty) return; + final incoming = [for (final t in nnTimesMs) t / 1000.0]; + // Checked before any state changes: an infinite time would make the band + // settle endless, and a fold cannot be undone. + if (incoming.any((t) => !t.isFinite)) { + throw ArgumentError('beat times must be finite'); + } + if (!_nondecreasing(incoming) || + (_last != null && incoming.first < _last!)) { + throw StateError('beat times must be non-decreasing'); + } + if (_first == null) { + _first = incoming.first; + for (final b in _bands) { + b.nextStart = _first!; + } + } + _nn.addAll(nnMs); + _times.addAll(incoming); + _n += nnMs.length; + _last = incoming.last; + for (final b in _bands) { + b.settle(_times, _nn, _last!, oversample); + } + _compact(); + } + + Metric snapshot({ + List tailNnMs = const [], + List tailNnTimesMs = const [], + required double artifactFraction, + double hfArtifactGate = 0.15, + }) { + if (tailNnMs.length != tailNnTimesMs.length) { + throw ArgumentError('tailNnMs and tailNnTimesMs must have the same length'); + } + // A non-finite tail time: the batch's own answer (it abstains). + if (_n + tailNnMs.length < 16 || + _first == null || + tailNnTimesMs.any((t) => !(t / 1000.0).isFinite)) { + return hrvFreq([..._nn, ...tailNnMs], + [..._times.map((x) => x * 1000.0), ...tailNnTimesMs], + artifactFraction: artifactFraction, + hfArtifactGate: hfArtifactGate, + oversample: oversample); + } + final tail = [for (final t in tailNnTimesMs) t / 1000.0]; + if (!_nondecreasing(tail) || (tail.isNotEmpty && tail.first < _last!)) { + throw StateError('beat times must be non-decreasing'); + } + final times = [..._times, ...tail]; + final nn = [..._nn, ...tailNnMs]; + final last = times.last; + if (last - _first! <= 0) { + return hrvFreq(nn, [for (final t in times) t * 1000.0], + artifactFraction: artifactFraction, + hfArtifactGate: hfArtifactGate, + oversample: oversample); + } + final powers = [ + for (final b in _bands) b.value(times, nn, last, oversample), + ]; + return _hrvFreqMetric(powers[0], powers[1], powers[2], powers[3], + artifactFraction: artifactFraction, hfArtifactGate: hfArtifactGate); + } + + void _compact() { + if (_times.isEmpty) return; + var keepFrom = _bands.map((b) => b.nextStart).reduce((a, b) => a < b ? a : b); + var cut = 0; + while (cut < _times.length && _times[cut] < keepFrom) { + cut++; + } + if (cut > 0) { + _times.removeRange(0, cut); + _nn.removeRange(0, cut); + } + } +} + +bool _nondecreasing(List values) { + for (var i = 1; i < values.length; i++) { + if (values[i] < values[i - 1]) return false; + } + return true; +} diff --git a/lib/src/onehz/clinical/hrv_jitter_evidence.dart b/lib/src/onehz/clinical/hrv_jitter_evidence.dart new file mode 100644 index 0000000..aa6168c --- /dev/null +++ b/lib/src/onehz/clinical/hrv_jitter_evidence.dart @@ -0,0 +1,876 @@ +// Rejection and breathing-line exemption thresholds are an internal rule. +// Welch 1967 spectral estimation; Hirsch & Bishop 1981 provides the +// breathing/RSA basis. This is hrvTime()'s existing internal jitter policy, +// ESTIMATE with low confidence, not a published classifier. Shared formulas +// preserve the calculation; reordered sums can change decisions within rounding +// of a threshold. +// CLINICAL — mergeable evidence for the RMSSD jitter gate, for +// `hrvTimeAggregator`. +// +// Under [kNnDiffAcf1Floor] `hrvTime` decides on the difference RUNS themselves +// (rmssd_gate.dart): the white-noise share of the pooled Welch spectrum +// (64-beat segments every 32 beats from each run's start) and, for a refused +// timed night, the breathing-line test over the 5-min windows (128-beat +// segments every 64 beats from each in-window run's start). This aggregator +// keeps what those tests read as sums, so a night is never kept beat by beat: +// +// * NOISE SHARE. A run that starts and ends inside a state is reduced to its +// segment sums on the spot (as the batch does). A run cut by a chunk edge +// is a PIECE: its first and last 63 differences raw (the only ones whose +// coverage weight or segment depends on what lies past the edge), the +// segment sums of every segment fully inside it, and the squared +// differences of the rest by position mod 32 (their Hann coverage weight +// depends on nothing else). A piece cut on its left does not know where in +// its run it starts, so it keeps the segment sums for all 32 alignments; +// the join that tells it (or the series start, at finish) picks one. A +// piece is at most ~1.2k numbers, whatever its length. +// * BREATHING. An in-window run lies inside one 5-min window, so it is short; +// one cut by a chunk edge stays raw. Each finished run is reduced to its +// 128-beat segments' autocorrelation at lags 0..127, from which the +// window's spectrum at ANY frequency follows (the test reads it at +// frequencies that depend on the whole night's median beat): one record of +// ~130 numbers per window, O(windows). +// * The coarse-lattice test needs every distinct |difference| (it asks +// whether all are multiples of the smallest): kept as a sorted set. Its +// size is the number of distinct difference magnitudes, not of beats (on a +// 1 ms or 0.5 ms grid at most a few thousand). +// +// Sums are formed in another order than the batch's (and a cut piece's levels +// are integrated from its own start), so numbers agree to float rounding, not +// bit for bit; a verdict can differ only within rounding of a threshold. +// +// Not exported. + +import 'dart:collection'; +import 'dart:math' as math; +import 'dart:typed_data'; + +import '../../aggregate/aggregator.dart'; +import '../../aggregate/json_support.dart'; +import 'hrv_time.dart'; +import 'hrv_time_aggregator.dart' show Beats; +import 'rmssd_gate.dart'; + +const int _edge = 63; // raw differences kept at each end of a piece +const int _bins = kNoiseHiBin - kNoiseLoBin + 1; +const int _lags = 128; +const double _windowMs = 300000.0; + +double _t(double v) => v.isNaN ? 0.0 : v; + +/// Hann coverage of a point at offset a (mod 32) from the segment grid, when +/// two segments cover it. +final List _cw = [ + for (var a = 0; a < 32; a++) + noiseHann[a] * noiseHann[a] + noiseHann[a + 32] * noiseHann[a + 32] +]; + +class _Phase { + final Float64List psd = Float64List(kNoiseHiBin + 1); + double p = 0, p2 = 0; + int segs = 0; + + void addAll(_Phase o) { + for (var k = kNoiseLoBin; k <= kNoiseHiBin; k++) { + psd[k] += o.psd[k]; + } + p += o.p; + p2 += o.p2; + segs += o.segs; + } + + _Phase copy() => _Phase()..addAll(this); + + Object toJson() => [ + for (var k = kNoiseLoBin; k <= kNoiseHiBin; k++) encodeDouble(psd[k]), + encodeDouble(p), + encodeDouble(p2), + segs, + ]; + + static _Phase fromJson(Object? j) { + final l = decodeList(j); + if (l.length != _bins + 3) throw const FormatException('invalid phase'); + final ph = _Phase(); + for (var k = 0; k < _bins; k++) { + ph.psd[k + kNoiseLoBin] = decodeDouble(l[k]); + } + ph.p = decodeDouble(l[_bins]); + ph.p2 = decodeDouble(l[_bins + 1]); + ph.segs = decodeCount(l[_bins + 2]); + return ph; + } +} + +/// A contiguous run of differences cut by a chunk edge (see the header). +class _Piece { + _Piece(this.anchored) + : phases = [for (var i = 0; i < (anchored ? 1 : 32); i++) _Phase()]; + + /// Starts at its run's start (else its alignment in the run is unknown). + bool anchored; + + /// Ends at its run's end. + bool rightClosed = false; + int len = 0; // differences + final List front = []; + ListQueue back = ListQueue(); + final Float64List s2 = Float64List(32); + final List cnt = List.filled(32, 0); + + /// Levels of the last <= 64 points (integrated from the piece's start, or + /// from wherever a join left them; only differences within a segment + /// matter). + ListQueue xs = ListQueue()..add(0.0); + double x = 0.0; + List<_Phase> phases; + + /// Segments starting at or after this point are not formed (a join's + /// replay: they belong to the right piece's sums). + int stop = 1 << 62; + + final Float64List _seg = Float64List(kNoiseSegment); + + void append(double d) { + len++; + x += d; + xs.add(x); + if (xs.length > kNoiseSegment) xs.removeFirst(); + if (front.length < _edge) front.add(d); + back.add(d); + if (back.length > _edge) { + final popped = back.removeFirst(); + final idx = len - _edge - 1; + if (idx >= _edge) { + s2[idx % 32] += popped * popped; + cnt[idx % 32]++; + } + } + final start = len - (kNoiseSegment - 1); + if (start < 0 || start >= stop) return; + final phase = start % 32; + if (anchored && phase != 0) return; + var i = 0; + for (final v in xs) { + _seg[i++] = v; + } + final ph = phases[anchored ? 0 : phase]; + final p = addNoiseSegment(_seg, 0, ph.psd); + ph.p += p; + ph.p2 += p * p; + ph.segs++; + } + + /// Every difference, for a piece of at most 2 * 63 (front and back cover it). + List allDiffs() { + assert(len <= 2 * _edge); + final overlap = front.length - (len - back.length); + return [...front, ...back.skip(math.max(0, overlap))]; + } + + void anchor() { + if (anchored) return; + anchored = true; + phases = [phases[0]]; + } + + /// This piece, then [d], then [right] (open on its left, never anchored). + void join(double d, _Piece right) { + append(d); + if (right.len <= 2 * _edge) { + for (final v in right.allDiffs()) { + append(v); + } + rightClosed = right.rightClosed; + return; + } + final boundary = len; // right's point 0 + stop = boundary; + for (final v in right.front) { + append(v); + } + stop = 1 << 62; + // The replayed front sits in the interior of the joined piece. + var k = 0; + for (final v in back) { + final idx = boundary + k++; + if (idx >= _edge) { + s2[idx % 32] += v * v; + cnt[idx % 32]++; + } + } + back = ListQueue.of(right.back); + for (var r = 0; r < 32; r++) { + s2[(r + boundary) % 32] += right.s2[r]; + cnt[(r + boundary) % 32] += right.cnt[r]; + } + if (anchored) { + phases[0].addAll(right.phases[(32 - boundary % 32) % 32]); + } else { + for (var f = 0; f < 32; f++) { + phases[f].addAll(right.phases[((f - boundary) % 32 + 32) % 32]); + } + } + len = boundary + right.len; + xs = ListQueue.of(right.xs); + x = right.x; + rightClosed = right.rightClosed; + } + + /// Adds this piece, anchored and closed (a whole run), to [sums]' spectral + /// and coverage terms (not its squared-difference count: the state keeps + /// that). + void finalizeInto(NoiseShareSums sums) { + assert(anchored); + if (len <= 2 * _edge) { + final scratch = NoiseShareSums()..addRun(allDiffs()); + for (var k = kNoiseLoBin; k <= kNoiseHiBin; k++) { + sums.psd[k] += scratch.psd[k]; + } + sums.pSum += scratch.pSum; + sums.p2Sum += scratch.p2Sum; + sums.segs += scratch.segs; + sums.wsd += scratch.wsd; + sums.wsum += scratch.wsum; + return; + } + final ph = phases[0]; + for (var k = kNoiseLoBin; k <= kNoiseHiBin; k++) { + sums.psd[k] += ph.psd[k]; + } + sums.pSum += ph.p; + sums.p2Sum += ph.p2; + sums.segs += ph.segs; + for (var r = 0; r < 32; r++) { + final c = (_cw[r] + _cw[(r + 1) % 32]) / 2; + sums.wsd += c * s2[r]; + sums.wsum += c * cnt[r]; + } + // The edges: coverage from the segments at 0, 32, ... that fit the run. + final last = + len + 1 >= kNoiseSegment ? ((len + 1 - kNoiseSegment) ~/ 32) * 32 : -1; + double cov(int j) { + var c = 0.0; + for (var s = (j - kNoiseSegment + 1).clamp(0, 1 << 62); + s <= j && s <= last; + s++) { + if (s % 32 == 0) c += noiseHann[j - s] * noiseHann[j - s]; + } + return c; + } + + void edge(int i, double d) { + final c = (cov(i) + cov(i + 1)) / 2; + sums.wsd += c * d * d; + sums.wsum += c; + } + + for (var i = 0; i < front.length; i++) { + edge(i, front[i]); + } + var i = len - back.length; + for (final d in back) { + edge(i++, d); + } + } + + _Piece copy() { + final c = _Piece(anchored) + ..rightClosed = rightClosed + ..len = len + ..back = ListQueue.of(back) + ..xs = ListQueue.of(xs) + ..x = x + ..stop = stop + ..phases = [for (final p in phases) p.copy()]; + c.front.addAll(front); + c.s2.setAll(0, s2); + c.cnt.setAll(0, cnt); + return c; + } + + Object toJson() => { + 'anchored': anchored, + 'rightClosed': rightClosed, + 'len': len, + 'front': [for (final v in front) encodeDouble(v)], + 'back': [for (final v in back) encodeDouble(v)], + 's2': encodeDoubles(s2), + 'cnt': cnt, + 'xs': [for (final v in xs) encodeDouble(v)], + 'x': encodeDouble(x), + 'phases': [for (final p in phases) p.toJson()], + }; + + static _Piece fromJson(Object? j) { + final m = decodeMap(j); + final anchored = field(m, 'anchored'); + final rightClosed = field(m, 'rightClosed'); + if (anchored is! bool || rightClosed is! bool) { + throw const FormatException('invalid piece flags'); + } + final p = _Piece(anchored) + ..rightClosed = rightClosed + ..len = decodeCount(field(m, 'len')) + ..x = decodeDouble(field(m, 'x')); + p.front.addAll(decodeDoubles(field(m, 'front'))); + p.back = ListQueue.of(decodeDoubles(field(m, 'back'))); + p.xs = ListQueue.of(decodeDoubles(field(m, 'xs'))); + final s2 = decodeDoubles(field(m, 's2')); + final cnt = decodeList(field(m, 'cnt')); + final phases = decodeList(field(m, 'phases')); + final edge = math.min(p.len, _edge); + if (p.front.length != edge || + p.back.length != edge || + p.xs.length != math.min(p.len + 1, kNoiseSegment) || + s2.length != 32 || + cnt.length != 32 || + phases.length != (anchored ? 1 : 32)) { + throw const FormatException('inconsistent piece'); + } + p.s2.setAll(0, s2); + var interior = 0; + for (var r = 0; r < 32; r++) { + p.cnt[r] = decodeCount(cnt[r]); + interior += p.cnt[r]; + } + if (interior != math.max(0, p.len - 2 * _edge)) { + throw const FormatException('piece interior count'); + } + p.phases = [for (final ph in phases) _Phase.fromJson(ph)]; + return p; + } +} + +/// One 5-min window's breathing-test record: its beats' RR, its in-window +/// differences, and its 128-beat segments' lag sums. +class _Win { + double rrSum = 0; + int rrN = 0; + double sq = 0; + int n = 0; + int segs = 0; + final Float64List r = Float64List(_lags); + + void addAll(_Win o) { + rrSum += o.rrSum; + rrN += o.rrN; + sq += o.sq; + n += o.n; + segs += o.segs; + for (var m = 0; m < _lags; m++) { + r[m] += o.r[m]; + } + } + + /// The segments of one finished in-window run (from its start, as the + /// batch walks them). + void addRun(List diffs) { + const n = 128; + final x = List.filled(diffs.length + 1, 0.0); + for (var i = 0; i < diffs.length; i++) { + x[i + 1] = x[i] + diffs[i]; + } + final v = Float64List(n); + for (var s = 0; s + n <= x.length; s += n ~/ 2) { + var mean = 0.0; + for (var i = 0; i < n; i++) { + mean += x[s + i]; + } + mean /= n; + for (var i = 0; i < n; i++) { + v[i] = (x[s + i] - mean) * hann128[i]; + } + for (var m = 0; m < n; m++) { + var acc = 0.0; + for (var i = 0; i + m < n; i++) { + acc += v[i] * v[i + m]; + } + r[m] += acc; + } + segs++; + } + } + + /// [windowPsd] of this window at cycles-per-beat frequencies [f]: + /// Σ_segments |Σ v e^{iθk}|² = R0 + 2 Σ_m R_m cos(mθ), over the segments. + List? psd(List f) { + if (segs == 0) return null; + return [ + for (final fr in f) _cosSeries(2 * math.pi * fr) / segs, + ]; + } + + double _cosSeries(double theta) { + final c1 = math.cos(theta); + var prev = 1.0, cur = c1; // cos(0), cos(θ) + var sum = r[0]; + for (var m = 1; m < _lags; m++) { + sum += 2 * r[m] * cur; + final next = 2 * c1 * cur - prev; + prev = cur; + cur = next; + } + return sum; + } + + _Win copy() => _Win()..addAll(this); + + Object toJson() => [ + encodeDouble(rrSum), + rrN, + encodeDouble(sq), + n, + segs, + encodeDoubles(r), + ]; + + static _Win fromJson(Object? j) { + final l = decodeList(j); + if (l.length != 6) throw const FormatException('invalid window'); + final w = _Win() + ..rrSum = decodeDouble(l[0]) + ..rrN = decodeCount(l[1]) + ..sq = decodeDouble(l[2]) + ..n = decodeCount(l[3]) + ..segs = decodeCount(l[4]); + final r = decodeDoubles(l[5]); + if (r.length != _lags) throw const FormatException('invalid window lags'); + w.r.setAll(0, r); + return w; + } +} + +/// An in-window run cut by a chunk edge: raw (a window bounds it). +class _WinRun { + _WinRun(this.w, {required this.anchored}); + final int w; + bool anchored; + bool closed = false; + List diffs = []; + + _WinRun copy() => _WinRun(w, anchored: anchored) + ..closed = closed + ..diffs = List.of(diffs); + + Object toJson() => [ + w, + anchored, + closed, + [for (final d in diffs) encodeDouble(d)] + ]; + + static _WinRun? fromJson(Object? j) { + if (j == null) return null; + final l = decodeList(j); + if (l.length != 4 || l[0] is! int || l[1] is! bool || l[2] is! bool) { + throw const FormatException('invalid window run'); + } + return _WinRun(l[0] as int, anchored: l[1] as bool) + ..closed = l[2] as bool + ..diffs = decodeDoubles(l[3]).toList(); + } +} + +class JitterState { + int n = 0; + double firstNn = 0, firstT = 0, lastNn = 0, lastT = 0; + int lastW = 0; + + /// The run cut by the state's left edge (alignment unknown). + _Piece? head; + + /// The run cut by its right edge, when a seam lies inside the state. + _Piece? tail; + NoiseShareSums closed = NoiseShareSums(); + double ssd = 0; + int nd = 0; + // Only coarseLattice reads magnitudes numerically; it ignores their order. + // Checkpoints sort these sets on encode to retain their original ordering. + Set mags = {}; + + final SplayTreeMap wins = SplayTreeMap(); + _WinRun? headRun, tailRun; + double winSsd = 0; + int winNd = 0; + Set winMags = {}; + + JitterState copy() { + final c = JitterState() + ..n = n + ..firstNn = firstNn + ..firstT = firstT + ..lastNn = lastNn + ..lastT = lastT + ..lastW = lastW + ..head = head?.copy() + ..tail = tail?.copy() + ..ssd = ssd + ..nd = nd + ..mags = Set.of(mags) + ..headRun = headRun?.copy() + ..tailRun = tailRun?.copy() + ..winSsd = winSsd + ..winNd = winNd + ..winMags = Set.of(winMags); + _addSums(c.closed, closed); + for (final e in wins.entries) { + c.wins[e.key] = e.value.copy(); + } + return c; + } +} + +void _addSums(NoiseShareSums into, NoiseShareSums from) { + for (var k = kNoiseLoBin; k <= kNoiseHiBin; k++) { + into.psd[k] += from.psd[k]; + } + into.pSum += from.pSum; + into.p2Sum += from.p2Sum; + into.segs += from.segs; + into.wsd += from.wsd; + into.wsum += from.wsum; +} + +/// What the jitter gate reads, for a finished state. +class JitterEvidence { + JitterEvidence(this.sums, this.mags, this.windows, this.winSsd, this.winNd, + this.winMags); + final NoiseShareSums sums; + final Iterable mags; + final List<_Win> windows; + final double winSsd; + final int winNd; + final Iterable winMags; + + /// `_jitterRefused` from these sums. + bool refused() { + final share = noiseShareFrom(sums, + lattice: () => coarseLattice(mags, sums.ssd / sums.nd)); + return share == null || share >= kNnDiffNoiseShareCeiling; + } + + /// `_breathingRmssd` from the window records. + double? breathingRmssd() { + final meanRr = [for (final w in windows) w.rrSum / w.rrN]; + final idx = steadyBreathingFrom( + windows: windows.length, + meanRrMs: meanRr, + psdAt: (w, f) => windows[w].psd(f), + diffs: (w) => windows[w].n, + addSquares: (w, acc) => acc + windows[w].sq, + allSsd: winSsd, + allNd: winNd, + lattice: (msd) => coarseLattice(winMags, msd), + ); + if (idx == null) return null; + var ss = 0.0, n = 0; + for (final i in idx) { + ss += windows[i].sq; + n += windows[i].n; + } + return n == 0 ? null : math.sqrt(ss / n); + } +} + +/// The jitter-gate evidence over a beat batch (see the header). +class JitterEvidenceAggregator + extends Aggregator { + const JitterEvidenceAggregator({this.originMs, required this.gapAware}); + final double? originMs; + final bool gapAware; + + int _window(double t) { + final origin = originMs; + if (origin == null) throw StateError('gap-aware HRV needs originMs'); + return ((_t(t) - origin) / _windowMs).floor(); + } + + bool _contiguous(double prevT, double t, double nn) => + !gapAware || _t(t) - _t(prevT) <= nn + 0.5; + + @override + JitterState empty() => JitterState(); + + _Win _win(JitterState s, int w) => s.wins.putIfAbsent(w, _Win.new); + + void _inRun(JitterState s, double d) { + s.ssd += d * d; + s.nd++; + s.mags.add(d.abs()); + } + + void _inWindow(JitterState s, int w, double d) { + final win = _win(s, w); + win.sq += d * d; + win.n++; + s.winSsd += d * d; + s.winNd++; + s.winMags.add(d.abs()); + } + + void _closePiece(JitterState s, _Piece p) => p.finalizeInto(s.closed); + + void _closeRun(JitterState s, _WinRun r) => _win(s, r.w).addRun(r.diffs); + + @override + JitterState addSlice(JitterState s, Beats batch, int start, int end) { + for (var i = start; i < end; i++) { + final v = batch.nnMs[i], t = batch.timeMs[i]; + final w = gapAware ? _window(t) : 0; + if (s.n == 0) { + s.firstNn = v; + s.firstT = t; + s.head = _Piece(false); + if (gapAware) s.headRun = _WinRun(w, anchored: false); + } else { + final contig = _contiguous(s.lastT, t, v); + final d = v - s.lastNn; + if (contig) { + _inRun(s, d); + (s.tail ?? s.head!).append(d); + } else if (s.tail == null) { + s.head!.rightClosed = true; + s.tail = _Piece(true); + } else { + _closePiece(s, s.tail!); + s.tail = _Piece(true); + } + if (gapAware) { + if (contig && w == s.lastW) { + _inWindow(s, w, d); + (s.tailRun ?? s.headRun!).diffs.add(d); + } else { + if (s.tailRun == null) { + s.headRun!.closed = true; + } else { + _closeRun(s, s.tailRun!); + } + s.tailRun = _WinRun(w, anchored: true); + } + } + } + if (gapAware) { + final win = _win(s, w); + win.rrSum += v; + win.rrN++; + } + s.n++; + s.lastNn = v; + s.lastT = t; + s.lastW = w; + } + return s; + } + + @override + JitterState combine(JitterState l, JitterState r) { + if (r.n == 0) return l; + if (l.n == 0) return r; + final contig = _contiguous(l.lastT, r.firstT, r.firstNn); + final d = r.firstNn - l.lastNn; + // Noise-share pieces. + final lOpen = l.tail ?? l.head!; + final rHead = r.head!; + if (contig) { + _inRun(l, d); + lOpen.join(d, rHead); + if (l.tail == null) { + l.tail = r.tail; // the joined head is closed iff r had a seam + } else if (r.tail != null) { + _closePiece(l, lOpen); + l.tail = r.tail; + } + } else { + if (l.tail == null) { + l.head!.rightClosed = true; + } else { + _closePiece(l, l.tail!); + } + rHead.anchor(); + if (r.tail != null) { + _closePiece(l, rHead); + l.tail = r.tail; + } else { + l.tail = rHead; + } + } + _addSums(l.closed, r.closed); + l.ssd += r.ssd; + l.nd += r.nd; + l.mags.addAll(r.mags); + // Breathing windows. + if (gapAware) { + for (final e in r.wins.entries) { + _win(l, e.key).addAll(e.value); + } + final lRun = l.tailRun ?? l.headRun!; + final rRun = r.headRun!; + if (contig && l.lastW == _window(r.firstT)) { + _inWindow(l, l.lastW, d); + lRun.diffs + ..add(d) + ..addAll(rRun.diffs); + if (l.tailRun == null) { + lRun.closed = rRun.closed; + l.tailRun = r.tailRun; + } else if (r.tailRun != null) { + _closeRun(l, lRun); + l.tailRun = r.tailRun; + } + } else { + if (l.tailRun == null) { + l.headRun!.closed = true; + } else { + _closeRun(l, l.tailRun!); + } + rRun.anchored = true; + if (r.tailRun != null) { + _closeRun(l, rRun); + l.tailRun = r.tailRun; + } else { + l.tailRun = rRun; + } + } + l.winSsd += r.winSsd; + l.winNd += r.winNd; + l.winMags.addAll(r.winMags); + } + l.n += r.n; + l.lastNn = r.lastNn; + l.lastT = r.lastT; + l.lastW = r.lastW; + return l; + } + + @override + JitterEvidence finish(JitterState state) { + final s = state.copy(); + // The left edge is the series start: a run starts there. + if (s.head != null) { + s.head!.anchor(); + _closePiece(s, s.head!); + } + if (s.tail != null) _closePiece(s, s.tail!); + if (s.headRun != null) _closeRun(s, s.headRun!); + if (s.tailRun != null) _closeRun(s, s.tailRun!); + s.closed + ..ssd = s.ssd + ..nd = s.nd; + return JitterEvidence( + s.closed, s.mags, s.wins.values.toList(), s.winSsd, s.winNd, s.winMags); + } + + @override + bool get persistable => true; + + @override + Object? encodeState(JitterState s) => { + 'type': 'HrvJitterEvidence', + 'version': 1, + 'n': s.n, + 'first': [encodeDouble(s.firstNn), encodeDouble(s.firstT)], + 'last': [encodeDouble(s.lastNn), encodeDouble(s.lastT), s.lastW], + 'head': s.head?.toJson(), + 'tail': s.tail?.toJson(), + 'closed': [ + for (var k = kNoiseLoBin; k <= kNoiseHiBin; k++) + encodeDouble(s.closed.psd[k]), + encodeDouble(s.closed.pSum), + encodeDouble(s.closed.p2Sum), + s.closed.segs, + encodeDouble(s.closed.wsd), + encodeDouble(s.closed.wsum), + ], + 'ssd': encodeDouble(s.ssd), + 'nd': s.nd, + 'mags': [for (final m in (s.mags.toList()..sort())) encodeDouble(m)], + 'wins': [ + for (final e in s.wins.entries) [e.key, e.value.toJson()] + ], + 'headRun': s.headRun?.toJson(), + 'tailRun': s.tailRun?.toJson(), + 'winSsd': encodeDouble(s.winSsd), + 'winNd': s.winNd, + 'winMags': [for (final m in (s.winMags.toList()..sort())) encodeDouble(m)], + }; + + @override + JitterState decodeState(Object? json) { + final m = decodeMap(json); + if (field(m, 'type') != 'HrvJitterEvidence' || field(m, 'version') != 1) { + throw const FormatException('unsupported HrvJitterEvidence'); + } + final s = JitterState()..n = decodeCount(field(m, 'n')); + final first = decodeList(field(m, 'first')); + final last = decodeList(field(m, 'last')); + if (first.length != 2 || last.length != 3 || last[2] is! int) { + throw const FormatException('invalid edges'); + } + s + ..firstNn = decodeDouble(first[0]) + ..firstT = decodeDouble(first[1]) + ..lastNn = decodeDouble(last[0]) + ..lastT = decodeDouble(last[1]) + ..lastW = last[2] as int; + final head = field(m, 'head'), tail = field(m, 'tail'); + s.head = head == null ? null : _Piece.fromJson(head); + s.tail = tail == null ? null : _Piece.fromJson(tail); + final closed = decodeList(field(m, 'closed')); + if (closed.length != _bins + 5) throw const FormatException('invalid sums'); + for (var k = 0; k < _bins; k++) { + s.closed.psd[k + kNoiseLoBin] = decodeDouble(closed[k]); + } + s.closed + ..pSum = decodeDouble(closed[_bins]) + ..p2Sum = decodeDouble(closed[_bins + 1]) + ..segs = decodeCount(closed[_bins + 2]) + ..wsd = decodeDouble(closed[_bins + 3]) + ..wsum = decodeDouble(closed[_bins + 4]); + s + ..ssd = decodeDouble(field(m, 'ssd')) + ..nd = decodeCount(field(m, 'nd')) + ..mags = Set.of(decodeDoubles(field(m, 'mags'))) + ..headRun = _WinRun.fromJson(field(m, 'headRun')) + ..tailRun = _WinRun.fromJson(field(m, 'tailRun')) + ..winSsd = decodeDouble(field(m, 'winSsd')) + ..winNd = decodeCount(field(m, 'winNd')) + ..winMags = Set.of(decodeDoubles(field(m, 'winMags'))); + for (final e in decodeList(field(m, 'wins'))) { + final pair = decodeList(e); + if (pair.length != 2 || pair[0] is! int) { + throw const FormatException('invalid window entry'); + } + if (s.wins.containsKey(pair[0])) { + throw const FormatException('duplicate window'); + } + s.wins[pair[0] as int] = _Win.fromJson(pair[1]); + } + // The parts must describe one state. + final pieces = + (s.head == null ? 0 : s.head!.len) + (s.tail == null ? 0 : s.tail!.len); + final ok = (s.n == 0) == (s.head == null) && + (s.tail == null || + (s.head != null && + s.head!.rightClosed && + s.tail!.anchored && + !s.tail!.rightClosed)) && + (s.head == null || !s.head!.anchored) && + (s.tail != null || s.head == null || !s.head!.rightClosed) && + s.nd >= pieces && + s.nd < math.max(1, s.n) && + s.winNd <= s.nd && + s.mags.length <= s.nd && + s.winMags.length <= s.winNd && + (gapAware + ? ((s.n == 0) == (s.headRun == null) && + (s.tailRun == null || + (s.headRun!.closed && s.tailRun!.anchored)) && + (s.headRun == null || !s.headRun!.anchored) && + s.wins.values.fold(0, (a, w) => a + w.rrN) == s.n && + s.wins.values.fold(0, (a, w) => a + w.n) == s.winNd) + : (s.headRun == null && s.tailRun == null && s.wins.isEmpty)); + if (!ok) throw const FormatException('inconsistent HrvJitterEvidence'); + return s; + } +} diff --git a/lib/src/onehz/clinical/hrv_time.dart b/lib/src/onehz/clinical/hrv_time.dart index 70cd001..202c7e6 100644 --- a/lib/src/onehz/clinical/hrv_time.dart +++ b/lib/src/onehz/clinical/hrv_time.dart @@ -15,8 +15,11 @@ // makes it behaviour — see that constant for the measurement and the threshold. import 'dart:math' as math; +import 'dart:typed_data'; import '../types.dart'; import '../util.dart'; +import 'hrv_time_aggregator.dart'; +import 'rmssd_gate.dart'; /// Lag-1 ACF floor for the NN successive-difference series, below which RMSSD /// and pNN50 are REFUSED (SDNN / SDANN survive it and are the honest lead). @@ -121,82 +124,13 @@ const double kNnDiffNoiseShareCeiling = 0.7; /// clear of the floor, or diffs on a beat-time grid coarse against their size /// (see [_onCoarseLattice]). double? nnDiffNoiseShare(List> diffRuns, {int minSegments = 20}) { - const n = 64, kLo = 10, kHi = n ~/ 2; // kLo/n ≈ 0.15 cycles/beat - final w = [ - for (var i = 0; i < n; i++) 0.5 - 0.5 * math.cos(2 * math.pi * i / (n - 1)) - ]; - var w2 = 0.0; - for (final x in w) { - w2 += x * x; - } - final cosT = [ - for (var k = kLo; k <= kHi; k++) - [for (var i = 0; i < n; i++) math.cos(2 * math.pi * k * i / n)] - ]; - final sinT = [ - for (var k = kLo; k <= kHi; k++) - [for (var i = 0; i < n; i++) math.sin(2 * math.pi * k * i / n)] - ]; - final psd = List.filled(kHi + 1, 0.0); - var segs = 0, nd = 0; - var ssd = 0.0, wsd = 0.0, wsum = 0.0, pSum = 0.0, p2Sum = 0.0; + final sums = NoiseShareSums(); for (final r in diffRuns) { - for (final d in r) { - ssd += d * d; - nd++; - } - // Integrate the run back to RR levels (up to a constant the mean removes). - final x = List.filled(r.length + 1, 0.0); - for (var i = 0; i < r.length; i++) { - x[i + 1] = x[i] + r[i]; - } - final cov = List.filled(x.length, 0.0); // Σ w² per sample - for (var s = 0; s + n <= x.length; s += n ~/ 2) { - var m = 0.0; - for (var i = 0; i < n; i++) { - m += x[s + i]; - cov[s + i] += w[i] * w[i]; - } - m /= n; - final v = [for (var i = 0; i < n; i++) (x[s + i] - m) * w[i]]; - var p = 0.0; - for (var k = kLo; k <= kHi; k++) { - final c = cosT[k - kLo], sn = sinT[k - kLo]; - var re = 0.0, im = 0.0; - for (var i = 0; i < n; i++) { - re += v[i] * c[i]; - im -= v[i] * sn[i]; - } - psd[k] += re * re + im * im; - p += re * re + im * im; - } - pSum += p; - p2Sum += p * p; - segs++; - } - for (var i = 0; i < r.length; i++) { - final c = (cov[i] + cov[i + 1]) / 2; - wsd += c * r[i] * r[i]; - wsum += c; - } + sums.addRun(r); } - if (segs < minSegments || ssd == 0 || wsd == 0) return null; - if (_onCoarseLattice(diffRuns, ssd / nd)) return null; - // One loud stretch dominates the average: count segments by power, not number. - if (pSum * pSum / p2Sum < 0.75 * minSegments) return null; - final band = psd.sublist(kLo); - var peak = 0; - for (var k = 1; k < band.length; k++) { - if (band[k] > band[peak]) peak = k; - } - if (peak + kLo >= kHi - 1) return null; // ~0.477–0.5 band, see above - if (band.last > 0.2 * band[peak]) return null; // hump on Nyquist, see above - final med = median(band)!; - if (band[peak] < 4.5 * med) return null; // no line, no exemption - final floor = med / segs / w2; - final vetted = wsd / wsum, all = ssd / nd; - // 1 − (structured share of RMSSD²); equals 2σ²/mean(d²) when stationary. - return 1 - (vetted - 2 * floor) / math.max(vetted, all); + return noiseShareFrom(sums, + lattice: () => _onCoarseLattice(diffRuns, sums.ssd / sums.nd), + minSegments: minSegments); } /// True when every successive difference is a whole multiple of one step q and @@ -208,22 +142,8 @@ double? nnDiffNoiseShare(List> diffRuns, {int minSegments = 20}) { /// sets RMSSD. // ponytail: exact lattice only; a grid re-rounded to whole ms (7.8 ms → 7/8) // breaks the common step and is not caught. -bool _onCoarseLattice(List> diffRuns, double msd) { - var q = double.infinity; - for (final r in diffRuns) { - for (final d in r) { - if (d.abs() > 1e-6 && d.abs() < q) q = d.abs(); - } - } - if (q.isInfinite || msd >= 2 * q * q) return false; - for (final r in diffRuns) { - for (final d in r) { - final k = d.abs() / q; - if ((k - k.roundToDouble()).abs() > 1e-3) return false; - } - } - return true; -} +bool _onCoarseLattice(List> diffRuns, double msd) => + coarseLattice([for (final r in diffRuns) for (final d in r) d.abs()], msd); /// The one RMSSD jitter verdict every path shares: ACF1 below the floor AND /// the spectrum does not show a respiratory line on a low white floor. @@ -256,48 +176,6 @@ List _clearedWindows(double? nightAcf1, List rmssds, ]; } -final _hann128 = [ - for (var i = 0; i < 128; i++) 0.5 - 0.5 * math.cos(2 * math.pi * i / 127) -]; -final _hann128Sq = _hann128.fold(0.0, (a, w) => a + w * w); - -/// Welch power (128-beat Hann, 50 % overlap, averaged over segments) of one -/// window's RR, rebuilt from its difference runs, at cycles-per-beat -/// frequencies [f]. Null when no run holds a full segment. -List? _windowPsd(List> diffRuns, List f) { - const n = 128; - final p = List.filled(f.length, 0.0); - var segs = 0; - for (final r in diffRuns) { - final x = List.filled(r.length + 1, 0.0); - for (var i = 0; i < r.length; i++) { - x[i + 1] = x[i] + r[i]; - } - for (var s = 0; s + n <= x.length; s += n ~/ 2) { - var m = 0.0; - for (var i = 0; i < n; i++) { - m += x[s + i]; - } - m /= n; - for (var j = 0; j < f.length; j++) { - final c = math.cos(2 * math.pi * f[j]), sn = math.sin(2 * math.pi * f[j]); - var cr = 1.0, ci = 0.0, re = 0.0, im = 0.0; - for (var i = 0; i < n; i++) { - final v = (x[s + i] - m) * _hann128[i]; - re += v * cr; - im += v * ci; - final t = cr * c - ci * sn; - ci = cr * sn + ci * c; - cr = t; - } - p[j] += re * re + im * im; - } - segs++; - } - } - return segs == 0 ? null : [for (final v in p) v / segs]; -} - /// Last resort for a night the jitter gate refused: the windows whose /// breathing line holds still in Hz while the heart rate moves. /// @@ -324,115 +202,29 @@ List? _steadyBreathingWindows( nd++; } } - if (nd == 0 || _onCoarseLattice(all, ssd / nd)) return null; - final ref = median([for (final m in meanRrMs) if (m > 0) m]); - if (ref == null) return null; - // 0.15–0.5 cycles/beat at the night's median beat. - final fc = [for (var j = 38; j <= 128; j++) j / 256]; - final byBeat = List.filled(fc.length, 0.0); - final byHz = List.filled(fc.length, 0.0); - final cover = List.filled(fc.length, 0); - final peakHz = {}; // window -> its own peak, cycles per ms - final psd = >{}; // window -> its cycles-per-beat PSD - for (var w = 0; w < winRuns.length; w++) { - if (meanRrMs[w] <= 0) continue; - final pb = _windowPsd(winRuns[w], fc); - if (pb == null) continue; - final norm = median(pb)!; - if (norm <= 0) continue; - final k = meanRrMs[w] / ref; - final fh = [for (final f in fc) f * k]; - final ph = _windowPsd(winRuns[w], fh)!; - var best = -1; - for (var j = 0; j < fc.length; j++) { - byBeat[j] += pb[j] / norm; - if (fh[j] > 0.5) continue; - byHz[j] += ph[j] / norm; - cover[j]++; - if (best < 0 || ph[j] > ph[best]) best = j; - } - if (best >= 0) peakHz[w] = fc[best] / ref; - psd[w] = pb; - } - final nw = peakHz.length; - if (nw < 12) return null; - final bins = [for (var j = 0; j < fc.length; j++) if (cover[j] == nw) j]; - if (bins.length < 20) return null; - final hz = [for (final j in bins) byHz[j]]; - final bt = [for (final j in bins) byBeat[j]]; - var pk = 0; - for (var i = 1; i < hz.length; i++) { - if (hz[i] > hz[pk]) pk = i; - } - if (pk == 0 || pk == hz.length - 1) return null; - final f0 = fc[bins[pk]] / ref; - final brpm = f0 * 60000; - if (brpm < 8 || brpm > 30) return null; - // Same bins, same normalisation: the line must stand taller aligned in Hz - // than the beat pooling stands anywhere within ±12 bins of it. - var near = 0.0; - for (var i = math.max(0, pk - 12); i <= math.min(bt.length - 1, pk + 12); i++) { - near = math.max(near, bt[i]); - } - // The pooled noise ripple shrinks as 1/√windows, so the bar does too. - final promHz = hz[pk] / median(hz)!; - if (promHz < math.max(2.5, 1.5 + 7 / math.sqrt(nw))) return null; - if (promHz < 1.1 * near / median(bt)!) return null; - // Alternation lives at Nyquist on the beat axis, outside these bins. - if (byBeat.last / median(bt)! >= 0.5 * promHz) return null; - final keep = [ - for (final e in peakHz.entries) - if (math.log(e.value / f0).abs() <= 0.15) e.key - ]; - if (keep.length < 6) return null; - // The line proves breathing is there, not that it carries RMSSD: beat-time - // jitter loud enough to fail the gate would still be most of the number. - // Same ceiling as [nnDiffNoiseShare], over the kept windows. Jitter σ² has - // an RR spectrum σ²·(2 − 2cos ω), flat once divided by that shape, and its - // differences carry 6σ²; physiology only adds power on top. Breathing - // wanders across the night and its line spreads with it, so each window - // drops the bins of every rate the kept windows peaked at (plus a Hann - // main lobe), mapped onto its own beats. σ² is the floor of what is left, - // pooled over windows and smoothed over four resolution cells. - var lo = double.infinity, hi = 0.0; - for (final w in keep) { - lo = math.min(lo, peakHz[w]!); - hi = math.max(hi, peakHz[w]!); - } - var sq = 0.0, nAll = 0; - final acc = List.filled(fc.length, 0.0); - final wt = List.filled(fc.length, 0.0); - for (final w in keep) { - var n = 0; - for (final r in winRuns[w]) { - for (final d in r) { - sq += d * d; - n++; + return steadyBreathingFrom( + windows: winRuns.length, + meanRrMs: meanRrMs, + psdAt: (w, f) => windowPsd(winRuns[w], f), + diffs: (w) { + var n = 0; + for (final r in winRuns[w]) { + n += r.length; } - } - nAll += n; - final a = lo * meanRrMs[w] - 3 / 128, b = hi * meanRrMs[w] + 3 / 128; - for (var j = 0; j < fc.length; j++) { - if (fc[j] > a && fc[j] < b) continue; - acc[j] += n * psd[w]![j] / (2 - 2 * math.cos(2 * math.pi * fc[j])); - wt[j] += n; - } - } - var floor = double.infinity; - for (var j = 8; j < fc.length - 8; j++) { - var s = 0.0, m = 0; - for (var i = j - 8; i <= j + 8; i++) { - if (wt[i] < nAll / 2) break; - s += acc[i] / wt[i]; - m++; - } - if (m == 17) floor = math.min(floor, s / 17); - } - if (floor == double.infinity) return null; - // The lowest stretch of a noisy curve reads below its mean; 1.1 puts the - // floor back at these pooled sizes, so jitter alone is not undercounted. - final noise = 1.1 * 6 * floor / _hann128Sq * nAll; - return noise < kNnDiffNoiseShareCeiling * sq ? keep : null; + return n; + }, + addSquares: (w, acc) { + for (final r in winRuns[w]) { + for (final d in r) { + acc += d * d; + } + } + return acc; + }, + allSsd: ssd, + allNd: nd, + lattice: (msd) => _onCoarseLattice(all, msd), + ); } /// The window RMSSDs a windowed headline publishes, or null for none: the @@ -632,72 +424,51 @@ Metric hrvTime( ); } + // A non-finite beat time has no place on the clock: the 5-min segmenting + // threw on it. Same refusal as hrvFreq / rsaRespRate. + if (nnTimesMs != null && + nnTimesMs.length == nnMs.length && + nnTimesMs.any((t) => !t.isFinite)) { + return const Metric.absent( + tier: Tier.high, + inputs_used: inputs, + note: 'non-finite beat time', + ); + } + + // The values come from the same numeric core as [hrvTimeAggregator], run + // in one slice; this function adds only the envelope (confidence, note). + // // RMSSD / pNN50: root mean square of SUCCESSIVE differences — successive in // TIME, not merely adjacent in the compacted list. correctRr drops multi-beat // artifact runs while advancing its clock across them, so nn[i-1] and nn[i] // can sit either side of a seconds-long hole; differencing straight down the // list manufactured one large difference per dropped run. Same `keep`-mask // treatment irregular_rhythm.dart already applies. A pair is contiguous iff - // the elapsed time between the two beat times is the interval itself. + // the elapsed time between the two beat times is the interval itself. The + // differences are kept as contiguous RUNS (a seam ends a run) so the same + // pass feeds the lag-1 ACF without ever forming a pair across a hole. // - // The differences are kept as contiguous RUNS (a seam ends a run) so the same - // pass feeds [nnDiffAcf1] without ever forming a lag-1 pair across a hole. - final gapAware = nnTimesMs != null && nnTimesMs.length == nnMs.length; - final runs = >[]; - var run = []; - for (var i = 1; i < nnMs.length; i++) { - if (gapAware && nnTimesMs[i] - nnTimesMs[i - 1] > nnMs[i] + 0.5) { - if (run.isNotEmpty) { - runs.add(run); - run = []; - } - continue; - } - run.add(nnMs[i] - nnMs[i - 1]); - } - if (run.isNotEmpty) runs.add(run); - - var ssd = 0.0; - var nn50 = 0; - var pairs = 0; - for (final r in runs) { - for (final d in r) { - ssd += d * d; - if (d.abs() > 50) nn50++; - pairs++; - } - } // RMSSD and pNN50 are the two outputs made of successive differences, so they // are the two the jitter floor refuses. SDNN/SDANN are made of the levels and // are far less contaminated (jitter share 1–27 % against RMSSD's 11–100 % on // the audit corpus) — they keep publishing, which is what the header has - // always advised. - final acf1 = nnDiffAcf1(runs); - final jittery = _jitterRefused(acf1, runs); + // always advised. Refused: a long record may still show a breathing line + // steady in Hz; then RMSSD alone comes from those 5-min windows. + final gapAware = nnTimesMs != null && nnTimesMs.length == nnMs.length; + final nn = Float64List.fromList(nnMs); + final times = gapAware ? Float64List.fromList(nnTimesMs) : Float64List(nn.length); + final v = hrvTimeOneSlice(Beats(nn, times), + originMs: gapAware ? times[0] : null, gapAware: gapAware); + final acf1 = v.diffAcf1; + // pNN50 is null with a real ACF1 (>= 30 pairs) only when the floor refused. + final jittery = acf1 != null && v.pnn50 == null; // More beat-time than elapsed: the stream itself is wrong, so neither the // differences nor any breathing line in them describe one heart. final overCounted = coverage?.overCounted == true; - final usable = pairs > 0 && !jittery && !overCounted; - var rmssd = usable ? math.sqrt(ssd / pairs) : null; - final pnn50 = usable ? 100.0 * nn50 / pairs : null; - // Refused: a long record may still show a breathing line steady in Hz - // ([_steadyBreathingWindows]); then RMSSD alone comes from those 5-min windows. - if (jittery && gapAware && !overCounted) { - rmssd = _breathingRmssd(nnMs, nnTimesMs); - } + final rmssd = overCounted ? null : v.rmssd; + final pnn50 = overCounted ? null : v.pnn50; final byBreathing = jittery && rmssd != null; - final sdnn = stddev(nnMs); - - double? sdann, sdnnIndex; - if (gapAware) { - final seg = _fiveMinSegments(nnMs, nnTimesMs); - if (seg.length >= 2) { - final means = [for (final s in seg) mean(s)!]; - sdann = stddev(means); - final sds = [for (final s in seg) stddev(s)].whereType().toList(); - sdnnIndex = sds.isEmpty ? null : mean(sds); - } - } // Confidence scales with beat count (ultra-short reads are less reliable), // with the artifact fraction we were handed, and with the measured jitter @@ -716,9 +487,9 @@ Metric hrvTime( return Metric( value: HrvTime( rmssd: rmssd, - sdnn: sdnn, - sdann: sdann, - sdnnIndex: sdnnIndex, + sdnn: v.sdnn, + sdann: v.sdann, + sdnnIndex: v.sdnnIndex, pnn50: pnn50, nBeats: nnMs.length, diffAcf1: acf1, @@ -730,7 +501,7 @@ Metric hrvTime( ? '${_overcountNote(coverage!)}. SDNN/SDANN survive it and are the ' 'lead here. PRV not ECG-HRV.' : jittery - ? '${_jitterNote(acf1!)}. SDNN/SDANN survive it and are the lead here. ' + ? '${_jitterNote(acf1)}. SDNN/SDANN survive it and are the lead here. ' '${byBreathing ? 'RMSSD only from 5-min windows on a breathing ' 'line steady in Hz, at floor confidence. ' : ''}PRV not ECG-HRV.' : 'PRV not ECG-HRV; RMSSD/pNN50 are quantization-sensitive at 1 Hz ' @@ -738,40 +509,6 @@ Metric hrvTime( ); } -/// RMSSD pooled over the 5-min windows of [nn] that sit on a breathing line -/// steady in Hz, or null. Same seam rule as [hrvTime]. -double? _breathingRmssd(List nn, List times) { - final wins = >>{}; - final rrSum = {}, rrN = {}; - var prevWin = -1; - for (var i = 0; i < nn.length; i++) { - final w = ((times[i] - times.first) / 300000.0).floor(); - rrSum[w] = (rrSum[w] ?? 0) + nn[i]; - rrN[w] = (rrN[w] ?? 0) + 1; - final runs = wins[w] ??= >[]; - if (i > 0 && w == prevWin && times[i] - times[i - 1] <= nn[i] + 0.5) { - runs.last.add(nn[i] - nn[i - 1]); - } else { - runs.add([]); - } - prevWin = w; - } - final keys = wins.keys.toList()..sort(); - final idx = _steadyBreathingWindows( - [for (final k in keys) wins[k]!], [for (final k in keys) rrSum[k]! / rrN[k]!]); - if (idx == null) return null; - var ss = 0.0, n = 0; - for (final i in idx) { - for (final r in wins[keys[i]]!) { - for (final d in r) { - ss += d * d; - n++; - } - } - } - return n == 0 ? null : math.sqrt(ss / n); -} - /// Robust NOCTURNAL RMSSD (ms). /// /// A single whole-night RMSSD is dominated by the few high-Δ segments produced @@ -817,6 +554,13 @@ Metric nocturnalRmssd( note: 'too few NN intervals for windowed nocturnal RMSSD', ); } + if (nnTimesMs.any((t) => !t.isFinite)) { + return const Metric.absent( + tier: Tier.high, + inputs_used: inputs, + note: 'non-finite beat time', + ); + } final t0 = nnTimesMs.first; // Bucket beat INDICES by window index, so each window keeps its beat times and // the successive differences below can skip the ones that straddle a dropped @@ -1037,6 +781,14 @@ Metric sleepSessionRmssdDetail( ); } + if (rrTsMs.any((t) => !t.isFinite)) { + return const Metric.absent( + tier: Tier.high, + inputs_used: inputs, + note: 'non-finite beat time', + ); + } + final buckets = >{}; final bucketsTs = >{}; final inRr = []; @@ -1182,28 +934,6 @@ Metric sleepSessionRmssdDetail( ); } -/// Group NN intervals into consecutive 5-minute (300 000 ms) segments by beat -/// time. Segments with <2 beats are dropped. -List> _fiveMinSegments(List nn, List times) { - const segMs = 300000.0; - final out = >[]; - if (nn.isEmpty) return out; - final t0 = times.first; - var curIdx = 0; - var cur = []; - for (var i = 0; i < nn.length; i++) { - final idx = ((times[i] - t0) / segMs).floor(); - if (idx != curIdx) { - if (cur.length >= 2) out.add(cur); - cur = []; - curIdx = idx; - } - cur.add(nn[i]); - } - if (cur.length >= 2) out.add(cur); - return out; -} - /// Range-filter [300, 2000] ms + Malik-style ectopic rejection against a local /// median, returned as CONTIGUOUS RUNS of kept intervals. /// diff --git a/lib/src/onehz/clinical/hrv_time_aggregator.dart b/lib/src/onehz/clinical/hrv_time_aggregator.dart new file mode 100644 index 0000000..d202171 --- /dev/null +++ b/lib/src/onehz/clinical/hrv_time_aggregator.dart @@ -0,0 +1,507 @@ +import 'dart:math' as math; +import 'dart:typed_data'; + +import '../../aggregate/aggregator.dart'; +import '../../aggregate/json_support.dart'; +import '../../aggregate/primitives.dart'; +import '../incremental_core.dart'; +import 'hrv_jitter_evidence.dart'; +import 'hrv_time.dart'; + +const double _fiveMinutesMs = 300000.0; + +/// NN intervals and their beat times. A missing time is represented by NaN. +class Beats extends Batch { + Beats(this.nnMs, this.timeMs) : assert(nnMs.length == timeMs.length); + + factory Beats.fromJson(Object? json) { + final map = decodeMap(json); + final nn = decodeDoubles(field(map, 'nnMs')); + final times = decodeDoubles(field(map, 'timeMs')); + if (nn.length != times.length) { + throw const FormatException('Beats columns differ in length'); + } + return Beats(nn, times); + } + + final Float64List nnMs; + final Float64List timeMs; + + @override + Object? toJson() => + {'nnMs': encodeDoubles(nnMs), 'timeMs': encodeDoubles(timeMs)}; + + @override + int get length => nnMs.length; + + @override + Beats copyRange(int start, int end) => Beats( + Float64List(end - start)..setRange(0, end - start, nnMs, start), + Float64List(end - start)..setRange(0, end - start, timeMs, start)); + + @override + Beats join(Beats other) { + final nn = Float64List(length + other.length); + final times = Float64List(length + other.length); + nn + ..setRange(0, length, nnMs) + ..setRange(length, nn.length, other.nnMs); + times + ..setRange(0, length, timeMs) + ..setRange(length, times.length, other.timeMs); + return Beats(nn, times); + } +} + +/// Builds the composed time-domain HRV summary for a columnar beat batch. +/// +/// With [gapAware] false, times and five-minute bins are deliberately ignored. +// The jitter rejection/exemption thresholds are an internal rule, ESTIMATE +// with low confidence; Welch 1967 and Hirsch & Bishop 1981 support spectral +// and respiratory components only. See rmssd_gate.dart. +// Task Force 1996 time-domain HRV, now also the hrvTime() one-slice path. +// Moments, two-beat differences, three-beat lag products and five-minute bins +// retain the same sufficient statistics. gapAware:false omits timestamp- +// dependent bins and gap tests. Floating-point order can change rounding and +// jitter-threshold decisions. Checked field tolerances are 1e-9 absolute plus +// 1e-9 relative; near-constant differences and jitter gates have the +// qualifications in test/aggregate/hrv_time_aggregator_test.dart. +Aggregator hrvTimeAggregator({ + double? originMs, + bool gapAware = true, +}) { + final core = _numericCore(originMs: originMs, gapAware: gapAware); + // Streaming finish uses the evidence already collected and merged by fold. + final evidence = JitterEvidenceAggregator(originMs: originMs, gapAware: gapAware); + final combined = validateCheckpoint(both(core, evidence), + (json) => _validateVersioned(json, gapAware)); + final checked = gapAware ? _FiniteTimes(combined) : combined; + return checked.map((result) => _finish( + result.$1.$1, result.$1.$2, result.$1.$3, result.$1.$4, + () => result.$2, gapAware: gapAware)); +} + +/// Runs the same numeric core over one slice, building jitter evidence only +/// when the ACF floor requires it. This helper is not exported by the public libraries. +HrvTime hrvTimeOneSlice(Beats beats, { + double? originMs, + bool gapAware = true, +}) { + final core = _numericCore(originMs: originMs, gapAware: gapAware); + final result = (gapAware ? _FiniteTimes(core) : core).run(beats); + JitterEvidence? cached; + JitterEvidence evidence() => cached ??= JitterEvidenceAggregator( + originMs: originMs, gapAware: gapAware).run(beats); + return _finish(result.$1, result.$2, result.$3, result.$4, evidence, + gapAware: gapAware); +} + +typedef _CoreSummary = ( + _DiffSummary, _LagSummary, RunningMoments, Map +); + +// Mapping only the finished summaries leaves both the accumulation order and +// checkpoint state shape unchanged, including the gap-unaware branch. +Aggregator _numericCore({ + double? originMs, + required bool gapAware, +}) { + final diffs = both( + Lookback( + 2, _DiffStats(gapAware: gapAware), + batchFromJson: Beats.fromJson), + Lookback( + 3, _LagStats(gapAware: gapAware), + batchFromJson: Beats.fromJson), + ); + final levels = Moments((batch) => batch.nnMs); + if (!gapAware) { + return both(diffs, levels).map((result) => ( + result.$1.$1, result.$1.$2, result.$2, + const {})); + } + + final bins = GroupBy( + (batch, row) { + final origin = originMs; + if (origin == null) throw StateError('gap-aware HRV needs originMs'); + return ((_timeAt(batch, row) - origin) / _fiveMinutesMs).floor(); + }, + Moments((batch) => batch.nnMs), + ); + return both(diffs, both(levels, bins)).map((result) => ( + result.$1.$1, result.$1.$2, result.$2.$1, result.$2.$2)); +} + +/// Refuses a slice holding a non-finite beat time before any state changes +/// (a fold cannot be undone), as the other night streams do. Gap-aware only: +/// without gap awareness times are never read. +class _FiniteTimes extends Aggregator { + const _FiniteTimes(this.inner); + final Aggregator inner; + + @override + S empty() => inner.empty(); + + @override + S addSlice(S state, Beats batch, int start, int end) { + for (var i = start; i < end; i++) { + if (!batch.timeMs[i].isFinite) { + throw ArgumentError('beat times must be finite'); + } + } + return inner.addSlice(state, batch, start, end); + } + + @override + S combine(S left, S right) => inner.combine(left, right); + + @override + R finish(S state) => inner.finish(state); + + @override + bool get persistable => inner.persistable; + + @override + Object? encodeState(S state) => inner.encodeState(state); + + @override + S decodeState(Object? json) => inner.decodeState(json); +} + +/// Version 2: `[core, evidence]`, the evidence counting what the core counts. +void _validateVersioned(Object? json, bool gapAware) { + final state = decodeList(json); + if (state.length != 2) throw const FormatException('Expected core and evidence'); + _validateCheckpoint(state[0], gapAware); + final core = decodeList(state[0]); + final evidence = decodeMap(state[1]); + final diffs = decodeMap(field(decodeMap(decodeList(core[0])[0]), 'inner')); + final rawLevels = gapAware ? decodeList(core[1])[0] : core[1]; + if (decodeCount(field(evidence, 'n')) != + decodeCount(field(decodeMap(rawLevels), 'count')) || + decodeCount(field(evidence, 'nd')) != decodeCount(field(diffs, 'pairs'))) { + throw const FormatException('Evidence disagrees with the core'); + } +} + +void _validateCheckpoint(Object? json, bool gapAware) { + final state = decodeList(json); + if (state.length != 2) throw const FormatException('Expected two states'); + final lookbacks = decodeList(state[0]); + if (lookbacks.length != 2) throw const FormatException('Expected two lookbacks'); + final diffs = decodeMap(lookbacks[0]); + final lags = decodeMap(lookbacks[1]); + final beats = decodeCount(field(diffs, 'length')); + if (decodeCount(field(lags, 'length')) != beats) { + throw const FormatException('Lookback lengths disagree'); + } + final diffPairs = decodeCount(field(decodeMap(field(diffs, 'inner')), 'pairs')); + final lagPairs = decodeCount(field(decodeMap(field(lags, 'inner')), 'pairs')); + final diffLimit = beats > 0 ? beats - 1 : 0; + final lagLimit = beats > 1 ? beats - 2 : 0; + if (diffPairs > diffLimit || lagPairs > lagLimit) { + throw const FormatException('Pair count exceeds beat windows'); + } + + final Object? rawLevels; + Map? bins; + if (gapAware) { + final levelAndBins = decodeList(state[1]); + if (levelAndBins.length != 2) { + throw const FormatException('Expected levels and bins'); + } + rawLevels = levelAndBins[0]; + bins = decodeMap(levelAndBins[1]); + } else { + rawLevels = state[1]; + } + final levels = decodeCount(field(decodeMap(rawLevels), 'count')); + if (levels > beats) { + throw const FormatException('Level count exceeds beat count'); + } + if (bins == null) return; + var binned = 0; + for (final bin in bins.values) { + final count = decodeCount(field(decodeMap(bin), 'count')); + if (binned > levels - count) { + throw const FormatException('Bin counts exceed level count'); + } + binned += count; + } + if (binned != levels) { + throw const FormatException('Bin counts disagree with level count'); + } +} + +double _timeAt(Beats batch, int row) { + final time = batch.timeMs[row]; + return time.isNaN ? 0.0 : time; +} + +bool _contiguous(Beats batch, int previous, int current, bool gapAware) => + !gapAware || + _timeAt(batch, current) - _timeAt(batch, previous) <= + batch.nnMs[current] + 0.5; + +class _DiffSummary { + const _DiffSummary( + this.pairs, this.over50, this.mean, this.m2, this.squares); + final int pairs; + final int over50; + final double mean; + final double m2; + final double squares; +} + +class _DiffState { + int pairs = 0; + int over50 = 0; + double mean = 0.0; + double m2 = 0.0; + double squares = 0.0; + + @pragma('vm:prefer-inline') + void add(double value) { + final nextPairs = pairs + 1; + final delta = value - mean; + mean += delta / nextPairs; + m2 += delta * (value - mean); + pairs = nextPairs; + } + + @pragma('vm:prefer-inline') + void merge(_DiffState other) { + if (other.pairs == 0) return; + if (pairs == 0) { + pairs = other.pairs; + over50 = other.over50; + mean = other.mean; + m2 = other.m2; + squares = other.squares; + return; + } + final leftPairs = pairs; + final combinedPairs = leftPairs + other.pairs; + final delta = other.mean - mean; + m2 += other.m2 + delta * delta * leftPairs * other.pairs / combinedPairs; + mean += delta * other.pairs / combinedPairs; + pairs = combinedPairs; + over50 += other.over50; + squares += other.squares; + } +} + +/// Processes every two-row window supplied by [Lookback]. +class _DiffStats extends Aggregator<_DiffState, Beats, _DiffSummary> { + const _DiffStats({required this.gapAware}); + final bool gapAware; + + @override + _DiffState empty() => _DiffState(); + + @override + _DiffState addSlice(_DiffState state, Beats batch, int start, int end) { + for (var i = start + 1; i < end; i++) { + if (!_contiguous(batch, i - 1, i, gapAware)) continue; + final diff = batch.nnMs[i] - batch.nnMs[i - 1]; + state.add(diff); + state.squares += diff * diff; + if (diff.abs() > 50) state.over50++; + } + return state; + } + + @override + _DiffState combine(_DiffState left, _DiffState right) { + left.merge(right); + return left; + } + + @override + bool get persistable => true; + + @override + Object? encodeState(_DiffState state) => { + 'pairs': state.pairs, + 'over50': state.over50, + 'mean': encodeDouble(state.mean), + 'm2': encodeDouble(state.m2), + 'squares': encodeDouble(state.squares), + }; + + @override + _DiffState decodeState(Object? json) { + final map = decodeMap(json); + final pairs = decodeCount(field(map, 'pairs')); + final over50 = decodeCount(field(map, 'over50')); + if (over50 > pairs) throw const FormatException('over50 exceeds pairs'); + return _DiffState() + ..pairs = pairs + ..over50 = over50 + ..mean = decodeDouble(field(map, 'mean')) + ..m2 = decodeDouble(field(map, 'm2')) + ..squares = decodeDouble(field(map, 'squares')); + } + + @override + _DiffSummary finish(_DiffState state) => + _DiffSummary( + state.pairs, state.over50, state.mean, state.m2, state.squares); +} + +class _LagSummary { + const _LagSummary(this.pairs, this.meanA, this.meanB, this.coMoment); + final int pairs; + final double meanA; + final double meanB; + final double coMoment; +} + +class _LagState { + int pairs = 0; + double meanA = 0.0; + double meanB = 0.0; + double coMoment = 0.0; + + @pragma('vm:prefer-inline') + void add(double a, double b) { + final nextPairs = pairs + 1; + final deltaA = a - meanA; + final deltaB = b - meanB; + meanA += deltaA / nextPairs; + meanB += deltaB / nextPairs; + coMoment += deltaA * (b - meanB); + pairs = nextPairs; + } + + @pragma('vm:prefer-inline') + void merge(_LagState other) { + if (other.pairs == 0) return; + if (pairs == 0) { + pairs = other.pairs; + meanA = other.meanA; + meanB = other.meanB; + coMoment = other.coMoment; + return; + } + final leftPairs = pairs; + final combinedPairs = leftPairs + other.pairs; + final deltaA = other.meanA - meanA; + final deltaB = other.meanB - meanB; + coMoment += other.coMoment + + deltaA * deltaB * leftPairs * other.pairs / combinedPairs; + meanA += deltaA * other.pairs / combinedPairs; + meanB += deltaB * other.pairs / combinedPairs; + pairs = combinedPairs; + } +} + +/// Processes every three-row window supplied by [Lookback]. +class _LagStats extends Aggregator<_LagState, Beats, _LagSummary> { + const _LagStats({required this.gapAware}); + final bool gapAware; + + @override + _LagState empty() => _LagState(); + + @override + _LagState addSlice(_LagState state, Beats batch, int start, int end) { + for (var i = start + 2; i < end; i++) { + if (!_contiguous(batch, i - 2, i - 1, gapAware) || + !_contiguous(batch, i - 1, i, gapAware)) { + continue; + } + final first = batch.nnMs[i - 1] - batch.nnMs[i - 2]; + final second = batch.nnMs[i] - batch.nnMs[i - 1]; + state.add(first, second); + } + return state; + } + + @override + _LagState combine(_LagState left, _LagState right) { + left.merge(right); + return left; + } + + @override + bool get persistable => true; + + @override + Object? encodeState(_LagState state) => { + 'pairs': state.pairs, + 'meanA': encodeDouble(state.meanA), + 'meanB': encodeDouble(state.meanB), + 'coMoment': encodeDouble(state.coMoment), + }; + + @override + _LagState decodeState(Object? json) { + final map = decodeMap(json); + return _LagState() + ..pairs = decodeCount(field(map, 'pairs')) + ..meanA = decodeDouble(field(map, 'meanA')) + ..meanB = decodeDouble(field(map, 'meanB')) + ..coMoment = decodeDouble(field(map, 'coMoment')); + } + + @override + _LagSummary finish(_LagState state) => + _LagSummary(state.pairs, state.meanA, state.meanB, state.coMoment); +} + +HrvTime _finish(_DiffSummary diffs, _LagSummary lags, RunningMoments levels, + Map bins, JitterEvidence Function() evidence, + {required bool gapAware}) { + if (levels.count < 2) return HrvTime(nBeats: levels.count); + + double? acf1; + if (diffs.pairs >= 30) { + if (diffs.m2 > 0) { + acf1 = (lags.coMoment + + lags.pairs * + (lags.meanA - diffs.mean) * + (lags.meanB - diffs.mean)) / + diffs.m2; + } + } + // Under the floor hrvTime refuses only when the runs show no respiratory + // line on a low floor, and then a timed night may still publish RMSSD from + // the windows on a breathing line steady in Hz: both read from [evidence]. + final jittery = + acf1 != null && acf1 < kNnDiffAcf1Floor && evidence().refused(); + final breathing = jittery && gapAware ? evidence().breathingRmssd() : null; + + double? sdann; + double? sdnnIndex; + if (gapAware) { + final binMeans = RunningMoments(); + var binSdSum = 0.0; + var binCount = 0; + for (final bin in bins.values) { + if (bin.count < 2) continue; + binMeans.add(bin.mean!); + binSdSum += bin.sampleSd!; + binCount++; + } + if (binCount >= 2) { + sdann = binMeans.sampleSd; + sdnnIndex = binSdSum / binCount; + } + } + + return HrvTime( + nBeats: levels.count, + rmssd: diffs.pairs > 0 && !jittery + ? math.sqrt(diffs.squares / diffs.pairs) + : breathing, + pnn50: diffs.pairs > 0 && !jittery + ? 100.0 * diffs.over50 / diffs.pairs + : null, + sdnn: levels.sampleSd, + sdann: sdann, + sdnnIndex: sdnnIndex, + diffAcf1: acf1, + ); +} diff --git a/lib/src/onehz/clinical/irregular_diagnostics.dart b/lib/src/onehz/clinical/irregular_diagnostics.dart new file mode 100644 index 0000000..c2b91d2 --- /dev/null +++ b/lib/src/onehz/clinical/irregular_diagnostics.dart @@ -0,0 +1,258 @@ +// Internal rule for SD1/SD2, pNNx thresholds and sustained-window voting. +// Task Force 1996 supports successive-difference HRV components, not this +// classifier or its SD1/SD2 thresholds; see hrv_time.dart. +// Internal irregular-rhythm screen, ESTIMATE with low confidence. Batch and +// stream share irregularWindowVerdict(); centered moments reorder floating-point +// sums, while exact counters preserve window evidence. Shared adjacency excludes +// timestamp-gap pairs. Diagnostics account for evidence without adding a +// clinical estimator. +// CLINICAL — the evidence behind an irregular-rhythm screen verdict. +// +// What it is for: `irregularBeatScreen` computes the beat counts, the cleaning +// counts and the per-5-minute-window counts on its way to a flag, then throws +// them away. A reader who sees "flagged" or "not screened" cannot tell why. +// [IrregularDiagnostics] keeps them, for a screen that RAN and for one that +// ABSTAINED (an abstention carries the counts that caused it). +// +// Absent stays absent: an input the caller did not give is null, never 0. The +// batch call and the streaming `IrregularScreenState` must produce the same +// diagnostics for the same beats, however chunked or restored. + +import '../types.dart'; +import 'irregular_rhythm.dart'; + +/// What the RR corrector did before the screen saw the beats. Only the caller +/// has it (the screen sees the corrected NN), so it is handed in; batch passes +/// the fields of `RrCorrectionResult`, streaming those of `RrSnapshot`. +class RrCleaningCounts { + /// Beats fed to the corrector. + final int raw; + + /// Isolated artifacts replaced by the spline. + final int corrected; + + /// Beats dropped (part of a multi-beat run, or with no anchors to correct from). + final int dropped; + const RrCleaningCounts({ + required this.raw, + required this.corrected, + required this.dropped, + }); +} + +/// The artifact fraction to record as evidence: [fraction], or null when +/// [cleaning] says the corrector saw no beats (no denominator). +double? diagnosticArtifactFraction(double fraction, RrCleaningCounts? cleaning) => + cleaning != null && cleaning.raw == 0 ? null : fraction; + +/// Why a screen produced no value. Wire names (toJson): `too_few_beats`, +/// `artifact`, `no_successive_pairs`, `no_long_term_variability`. +enum IrregularAbstain { + tooFewBeats, + artifact, + noSuccessivePairs, + noLongTermVariability, +} + +/// The final window of the series. A series always ends mid-window, so the last +/// window is OPEN; it is counted, not silently dropped. Wire names: `none` +/// (no beats at all), `thin` (fewer than minWindowBeats, excluded from the +/// valid count), `unflagged` (valid, not flagged), `flagged` (valid, flagged). +enum IrregularOpenWindow { none, thin, unflagged, flagged } + +/// The thresholds the verdict was made against. +class IrregularThresholds { + final int minBeats; + final double maxArtifact; + final double sd1sd2Flag; + final double pnnThresholdMs; + final double pnnFlagPct; + final double windowMinutes; + final int minWindowBeats; + final double sustainedFraction; + const IrregularThresholds({ + required this.minBeats, + required this.maxArtifact, + required this.sd1sd2Flag, + required this.pnnThresholdMs, + required this.pnnFlagPct, + required this.windowMinutes, + required this.minWindowBeats, + required this.sustainedFraction, + }); +} + +/// Per-window counts. Totals INCLUDE the open window; [open] says what it was. +class IrregularWindowCounts { + /// Windows with at least one clean beat (a window opens on a beat). + final int total; + + /// Windows with at least minWindowBeats beats (the ones that vote). + final int valid; + + /// Valid windows where both SD1/SD2 and pNNx cleared their flag lines. + final int flagged; + + /// The final window: its beat count and what became of it. + final int openBeats; + final IrregularOpenWindow open; + const IrregularWindowCounts({ + required this.total, + required this.valid, + required this.flagged, + required this.openBeats, + required this.open, + }); + + /// flagged / valid, null (never 0 or NaN) when no window is valid. + double? get sustainedObserved => valid == 0 ? null : flagged / valid; +} + +class IrregularDiagnostics { + /// Null when the screen ran and produced a value. + final IrregularAbstain? abstain; + + /// Beats fed to the corrector / corrected / dropped. Null when the caller did + /// not hand [RrCleaningCounts] in (unknown, not zero). + final int? rrRaw, corrected, dropped; + + /// NN entries handed to the screen (NaN and out-of-range included) and the + /// ones inside [300, 2000] ms that it used (== `IrregularRhythm.nBeats`). + final int nnIn, nnKept; + + /// The artifact fraction handed in (0 when none was). Null when the corrector + /// was handed no beats ([RrCleaningCounts.raw] == 0): a share of nothing has + /// no value, and the `1 - cleanFraction` of an empty series (1.0) would read + /// as "every beat is an artifact". The verdict gate still reads the fraction + /// the caller passed; only this evidence field is absent. + final double? artifactFraction; + + /// Null when windows were not evaluated: no beat times (or a length that does + /// not match the beats), or a window config that fails closed. + final IrregularWindowCounts? windows; + final IrregularThresholds thresholds; + + const IrregularDiagnostics({ + required this.abstain, + required this.rrRaw, + required this.corrected, + required this.dropped, + required this.nnIn, + required this.nnKept, + required this.artifactFraction, + required this.windows, + required this.thresholds, + }); + + static const _abstainWire = { + IrregularAbstain.tooFewBeats: 'too_few_beats', + IrregularAbstain.artifact: 'artifact', + IrregularAbstain.noSuccessivePairs: 'no_successive_pairs', + IrregularAbstain.noLongTermVariability: 'no_long_term_variability', + }; + + /// Wire shape (version 1); see test/onehz/irregular_diagnostics_test.dart. + /// Doubles are written as they are held, so a JSON text round trip is exact. + Map toJson() => { + 'version': 1, + 'abstain': abstain == null ? null : _abstainWire[abstain], + 'beats': { + 'rr_raw': rrRaw, + 'nn_in': nnIn, + 'nn_kept': nnKept, + 'corrected': corrected, + 'dropped': dropped, + 'artifact_fraction': artifactFraction, + }, + 'windows': windows == null + ? null + : { + 'total': windows!.total, + 'valid': windows!.valid, + 'flagged': windows!.flagged, + 'sustained_observed': windows!.sustainedObserved, + 'open_beats': windows!.openBeats, + 'open': windows!.open.name, + }, + 'thresholds': { + 'min_beats': thresholds.minBeats, + 'max_artifact': thresholds.maxArtifact, + 'sd1sd2_flag': thresholds.sd1sd2Flag, + 'pnn_threshold_ms': thresholds.pnnThresholdMs, + 'pnn_flag_pct': thresholds.pnnFlagPct, + 'window_minutes': thresholds.windowMinutes, + 'min_window_beats': thresholds.minWindowBeats, + 'sustained_fraction': thresholds.sustainedFraction, + }, + }; + + /// Throws [FormatException] on a map of another version or a malformed one. + factory IrregularDiagnostics.fromJson(Map json) { + try { + return _restore(json); + } on FormatException { + rethrow; + } catch (e) { + throw FormatException('malformed IrregularDiagnostics: $e'); + } + } + + static IrregularDiagnostics _restore(Map j) { + if (j['version'] != 1) { + throw FormatException('unsupported IrregularDiagnostics version ${j['version']}'); + } + final b = j['beats'] as Map, th = j['thresholds'] as Map; + final w = j['windows'] as Map?; + final a = j['abstain'] as String?; + IrregularAbstain? abstain; + if (a != null) { + final hit = _abstainWire.entries.where((e) => e.value == a); + if (hit.isEmpty) throw FormatException('unknown abstain reason $a'); + abstain = hit.first.key; + } + double d(Map m, String k) => (m[k] as num).toDouble(); + return IrregularDiagnostics( + abstain: abstain, + rrRaw: b['rr_raw'] as int?, + corrected: b['corrected'] as int?, + dropped: b['dropped'] as int?, + nnIn: b['nn_in'] as int, + nnKept: b['nn_kept'] as int, + artifactFraction: + b['artifact_fraction'] == null ? null : d(b, 'artifact_fraction'), + windows: w == null + ? null + : IrregularWindowCounts( + total: w['total'] as int, + valid: w['valid'] as int, + flagged: w['flagged'] as int, + openBeats: w['open_beats'] as int, + open: IrregularOpenWindow.values.byName(w['open'] as String), + ), + thresholds: IrregularThresholds( + minBeats: th['min_beats'] as int, + maxArtifact: d(th, 'max_artifact'), + sd1sd2Flag: d(th, 'sd1sd2_flag'), + pnnThresholdMs: d(th, 'pnn_threshold_ms'), + pnnFlagPct: d(th, 'pnn_flag_pct'), + windowMinutes: d(th, 'window_minutes'), + minWindowBeats: th['min_window_beats'] as int, + sustainedFraction: d(th, 'sustained_fraction'), + ), + ); + } +} + +/// A screen verdict and the evidence behind it. +class IrregularScreenResult { + final Metric metric; + final IrregularDiagnostics diagnostics; + const IrregularScreenResult(this.metric, this.diagnostics); + + /// `metric.toJson((v) => v.toJson())` plus a `diagnostics` key: the envelope + /// edge persists. The existing keys are unchanged. + Map toJson() => { + ...metric.toJson((v) => v.toJson()), + 'diagnostics': diagnostics.toJson(), + }; +} diff --git a/lib/src/onehz/clinical/irregular_rhythm.dart b/lib/src/onehz/clinical/irregular_rhythm.dart index 2b0faf8..b4c9762 100644 --- a/lib/src/onehz/clinical/irregular_rhythm.dart +++ b/lib/src/onehz/clinical/irregular_rhythm.dart @@ -1,3 +1,11 @@ +// Internal rule for SD1/SD2, pNNx thresholds and sustained-window voting. +// Task Force 1996 supports successive-difference HRV components, not this +// classifier or its SD1/SD2 thresholds; see hrv_time.dart. +// Internal irregular-rhythm screen, ESTIMATE with low confidence. Batch and +// stream share irregularWindowVerdict(); centered moments reorder floating-point +// sums, while exact counters preserve window evidence. Shared adjacency excludes +// timestamp-gap pairs. Diagnostics account for evidence without adding a +// clinical estimator. // CLINICAL — 24/7 irregular-rhythm SCREEN (NOT a diagnosis). // // A pulse-derived screen for sustained beat-to-beat irregularity over a long RR @@ -17,6 +25,8 @@ import 'dart:math' as math; import '../types.dart'; import '../util.dart'; +import 'irregular_diagnostics.dart'; +import 'irregular_window.dart'; class IrregularRhythm { final double sd1; // ms — short-term (beat-to-beat) scatter @@ -75,14 +85,59 @@ Metric irregularBeatScreen( double windowMinutes = 5, int minWindowBeats = 40, double sustainedFraction = 0.5, +}) => + irregularBeatScreenDetailed( + rrMs, + nnTimesMs: nnTimesMs, + artifactFraction: artifactFraction, + minBeats: minBeats, + sd1sd2Flag: sd1sd2Flag, + pnnThresholdMs: pnnThresholdMs, + pnnFlagPct: pnnFlagPct, + maxArtifact: maxArtifact, + windowMinutes: windowMinutes, + minWindowBeats: minWindowBeats, + sustainedFraction: sustainedFraction, + ).metric; + +/// [irregularBeatScreen] and the evidence behind its verdict: the beat counts, +/// the per-window counts (the final OPEN window included), the thresholds, and, +/// when the screen abstained, which gate stopped it. The Metric is exactly the +/// one [irregularBeatScreen] returns. [cleaning] is what the RR corrector did +/// upstream, when the caller knows (otherwise those counts are null, not 0). +/// Windows are counted whenever beat times are given, whether or not the +/// aggregate looked irregular and whether or not the screen abstained. +IrregularScreenResult irregularBeatScreenDetailed( + List rrMs, { + List? nnTimesMs, + double artifactFraction = 0.0, + int minBeats = irregularScreenMinBeats, + double sd1sd2Flag = 0.70, + double pnnThresholdMs = 70, + double pnnFlagPct = 30, + double maxArtifact = 0.30, + double windowMinutes = 5, + int minWindowBeats = 40, + double sustainedFraction = 0.5, + RrCleaningCounts? cleaning, }) { const inputs = ['rr_cleaned']; + final thresholds = IrregularThresholds( + minBeats: minBeats, + maxArtifact: maxArtifact, + sd1sd2Flag: sd1sd2Flag, + pnnThresholdMs: pnnThresholdMs, + pnnFlagPct: pnnFlagPct, + windowMinutes: windowMinutes, + minWindowBeats: minWindowBeats, + sustainedFraction: sustainedFraction, + ); // The defensive [300, 2000] filter COMPACTS the series. Keep the mask too, so // successive differences below are taken only between beats that were both // kept AND adjacent in the input — otherwise every filtered beat manufactured // one spurious difference spanning the gap, which counted toward pNNx and // inflated sdsd/sd1, pushing both flag conditions toward a false "sustained - // irregularity" screen positive. + // irregularity" screen positive. (NaN fails both comparisons: not a beat.) final keep = [for (final v in rrMs) v >= 300 && v <= 2000]; final nn = [ for (var i = 0; i < rrMs.length; i++) @@ -117,33 +172,69 @@ Metric irregularBeatScreen( for (var i = 1; i < rrMs.length; i++) if (successive[i]) rrMs[i] - rrMs[i - 1] ]; + // Windows are built from the SAME clean beats as the aggregate (the [keep] + // mask), not the raw input — an artifact beat the aggregate correctly + // excludes must not be allowed back in to inflate one window's own ratio/pNN + // into a spurious per-window flag. Counted up front so an abstention can + // still report them; the verdict reads them only where it did before. + final windowsOk = irregularWindowConfigOk( + windowMinutes: windowMinutes, + minWindowBeats: minWindowBeats, + sustainedFraction: sustainedFraction); + IrregularWindowCounts? windows; + if (hasTimes && windowsOk) { + final nnTimes = [ + for (var i = 0; i < rrMs.length; i++) + if (keep[i]) nnTimesMs[i] + ]; + windows = _countWindows( + nn, + nnTimes, + nnAdjacent, + sd1sd2Flag: sd1sd2Flag, + pnnThresholdMs: pnnThresholdMs, + pnnFlagPct: pnnFlagPct, + windowMinutes: windowMinutes, + minWindowBeats: minWindowBeats, + ); + } + + IrregularScreenResult abstain(IrregularAbstain why, String note) => + IrregularScreenResult( + Metric.absent( + tier: Tier.estimate, inputs_used: inputs, note: note), + IrregularDiagnostics( + abstain: why, + rrRaw: cleaning?.raw, + corrected: cleaning?.corrected, + dropped: cleaning?.dropped, + nnIn: rrMs.length, + nnKept: nn.length, + artifactFraction: diagnosticArtifactFraction(artifactFraction, cleaning), + windows: windows, + thresholds: thresholds, + ), + ); + // Gate on the contiguous pairs actually averaged, not the beat count: a // series fragmented by holes can hold minBeats beats yet only a handful of // successive pairs, and SD1/pNNx would then come from those few diffs. if (diffs.length < minBeats - 1) { - return const Metric.absent( - tier: Tier.estimate, - inputs_used: inputs, - note: 'too few clean beats for an irregular-rhythm screen', - ); + return abstain(IrregularAbstain.tooFewBeats, + 'too few clean beats for an irregular-rhythm screen'); } if (artifactFraction > maxArtifact) { - return Metric.absent( - tier: Tier.estimate, - inputs_used: inputs, - note: 'artifact fraction ${(artifactFraction * 100).round()}% > ' - '${(maxArtifact * 100).round()}% — screen suppressed on noisy RR', - ); + return abstain( + IrregularAbstain.artifact, + 'artifact fraction ${(artifactFraction * 100).round()}% > ' + '${(maxArtifact * 100).round()}% — screen suppressed on noisy RR'); } final sdsd = stddev(diffs); final sdnn = stddev(nn); if (sdsd == null || sdnn == null) { - return const Metric.absent( - tier: Tier.estimate, - inputs_used: inputs, - note: 'no successive clean beats to build a Poincare plot from', - ); + return abstain(IrregularAbstain.noSuccessivePairs, + 'no successive clean beats to build a Poincare plot from'); } final sd1 = sdsd / math.sqrt2; final v = 2 * sdnn * sdnn - sd1 * sd1; @@ -152,12 +243,10 @@ Metric irregularBeatScreen( // SD1/SD2 is undefined without long-term variability to divide by; emitting // ratio 0.0 with sd1 = sd2 = 0 published "perfectly regular" as a // measurement of a degenerate series. - return const Metric.absent( - tier: Tier.estimate, - inputs_used: inputs, - note: 'no long-term variability (SD2 = 0) — the SD1/SD2 ratio is ' - 'undefined, not "perfectly regular"', - ); + return abstain( + IrregularAbstain.noLongTermVariability, + 'no long-term variability (SD2 = 0) — the SD1/SD2 ratio is ' + 'undefined, not "perfectly regular"'); } final ratio = sd1 / sd2; @@ -180,57 +269,55 @@ Metric irregularBeatScreen( // that is what "sustained" is supposed to mean. Falls back to the old // whole-span verdict only when times weren't supplied (short/sleep-only // callers where the whole span already IS roughly one physiological state). - // Windows are built from the SAME clean beats as the aggregate above (the - // [keep] mask), not the raw input — an artifact beat the aggregate - // correctly excludes must not be allowed back in here to inflate one - // window's own ratio/pNN into a spurious per-window flag. - final nnTimes = hasTimes - ? [ - for (var i = 0; i < rrMs.length; i++) - if (keep[i]) nnTimesMs[i] - ] - : const []; + // A bad window config fails CLOSED (never sustained): a misconfigured caller + // must never manufacture a medical false positive. final flag = aggregateHigh && (!hasTimes || - _sustainedAcrossWindows( - nn, - nnTimes, - nnAdjacent, - sd1sd2Flag: sd1sd2Flag, - pnnThresholdMs: pnnThresholdMs, - pnnFlagPct: pnnFlagPct, - windowMinutes: windowMinutes, - minWindowBeats: minWindowBeats, - sustainedFraction: sustainedFraction, - )); + (windows != null && + windows.valid > 0 && + windows.flagged / windows.valid >= sustainedFraction)); // Confidence scales with beat count (~5000 beats ≈ a full strong night) AND // with the artifact fraction we were handed — it used to ignore it entirely, // so a barely-passing 29 %-artifact night published at the same confidence as // a clean one. final conf = (nn.length / 5000.0 * (1 - artifactFraction)).clamp(0.2, 0.9); - return Metric( - value: IrregularRhythm( - sd1: sd1, - sd2: sd2, - sd1sd2: ratio, - pnnPct: pnnPct, - nBeats: nn.length, - flag: flag, + return IrregularScreenResult( + Metric( + value: IrregularRhythm( + sd1: sd1, + sd2: sd2, + sd1sd2: ratio, + pnnPct: pnnPct, + nBeats: nn.length, + flag: flag, + ), + confidence: conf, + tier: Tier.estimate, + inputs_used: inputs, + note: 'irregular-rhythm SCREEN (not a diagnosis): Poincaré SD1/SD2 + pNN' + '${pnnThresholdMs.round()}. PRV not ECG — wrist pulse misses P-waves. ' + 'Discuss with a clinician only if you have symptoms.', + ), + IrregularDiagnostics( + abstain: null, + rrRaw: cleaning?.raw, + corrected: cleaning?.corrected, + dropped: cleaning?.dropped, + nnIn: rrMs.length, + nnKept: nn.length, + artifactFraction: diagnosticArtifactFraction(artifactFraction, cleaning), + windows: windows, + thresholds: thresholds, ), - confidence: conf, - tier: Tier.estimate, - inputs_used: inputs, - note: 'irregular-rhythm SCREEN (not a diagnosis): Poincaré SD1/SD2 + pNN' - '${pnnThresholdMs.round()}. PRV not ECG — wrist pulse misses P-waves. ' - 'Discuss with a clinician only if you have symptoms.', ); } -/// True when the SD1/SD2 + pNNx criteria independently hold in a real -/// fraction of short, roughly-stationary windows across [rrMs] — not just in -/// the one number the whole span blends into. [timesMs] must be the same -/// length as [rrMs] and index-aligned (elapsed ms per beat). -bool _sustainedAcrossWindows( +/// Counts the short windows across [rrMs] (clean beats only, with their +/// [timesMs] and [adjacent] flags, all the same length): how many exist, how +/// many are thick enough to vote, how many of those flag, and what the final +/// OPEN window was. The window rule and per-window verdict are the ones +/// `IrregularScreenState` uses (irregular_window.dart). +IrregularWindowCounts _countWindows( List rrMs, List timesMs, // Aligned to [rrMs]: whether each beat was truly adjacent (no dropped @@ -242,53 +329,29 @@ bool _sustainedAcrossWindows( required double pnnFlagPct, required double windowMinutes, required int minWindowBeats, - required double sustainedFraction, }) { - if (timesMs.length != rrMs.length || - adjacent.length != rrMs.length || - rrMs.length < 2) { - return false; - } - // Fail CLOSED (never sustained) on a bad config — a misconfigured caller - // must never manufacture a medical false positive. - if (!windowMinutes.isFinite || - windowMinutes <= 0 || - minWindowBeats < 2 || - !sustainedFraction.isFinite || - sustainedFraction < 0 || - sustainedFraction > 1) { - return false; - } final windowMs = windowMinutes * 60000; - var windowStart = timesMs.first; - var validWindows = 0; - var flaggedWindows = 0; + var total = 0, valid = 0, flagged = 0; + var openBeats = 0; + var open = IrregularOpenWindow.none; var bucket = []; var bucketAdjacent = []; + var windowStart = timesMs.isEmpty ? 0.0 : timesMs.first; void flush() { - // Mirror the aggregate's `keep[i] && keep[i-1]` guard: never diff - // across a beat that was dropped as an artifact in the original series, - // even though it's now a consecutive pair in this compacted bucket. - final diffs = [ - for (var i = 1; i < bucket.length; i++) - if (bucketAdjacent[i]) bucket[i] - bucket[i - 1] - ]; - // Valid on contiguous pairs, not beats (same reason as the aggregate). - if (diffs.length >= minWindowBeats - 1) { - validWindows++; - final sdsd = stddev(diffs); - final sdnn = stddev(bucket); - if (sdsd != null && sdnn != null) { - final sd1 = sdsd / math.sqrt2; - final v = 2 * sdnn * sdnn - sd1 * sd1; - final sd2 = v > 0 ? math.sqrt(v) : 0.0; - if (sd2 > 0) { - final ratio = sd1 / sd2; - final over = diffs.where((d) => d.abs() > pnnThresholdMs).length; - final pnn = 100.0 * over / diffs.length; - if (ratio >= sd1sd2Flag && pnn >= pnnFlagPct) flaggedWindows++; - } - } + if (bucket.isEmpty) return; + final verdict = irregularWindowVerdict(bucket, bucketAdjacent, + sd1sd2Flag: sd1sd2Flag, + pnnThresholdMs: pnnThresholdMs, + pnnFlagPct: pnnFlagPct, + minWindowBeats: minWindowBeats); + total++; + openBeats = bucket.length; + if (verdict == null) { + open = IrregularOpenWindow.thin; + } else { + valid++; + if (verdict) flagged++; + open = verdict ? IrregularOpenWindow.flagged : IrregularOpenWindow.unflagged; } bucket = []; bucketAdjacent = []; @@ -303,6 +366,11 @@ bool _sustainedAcrossWindows( bucketAdjacent.add(adjacent[i]); } flush(); - if (validWindows == 0) return false; - return flaggedWindows / validWindows >= sustainedFraction; + return IrregularWindowCounts( + total: total, + valid: valid, + flagged: flagged, + openBeats: openBeats, + open: open, + ); } diff --git a/lib/src/onehz/clinical/irregular_rhythm_state.dart b/lib/src/onehz/clinical/irregular_rhythm_state.dart new file mode 100644 index 0000000..27b3c76 --- /dev/null +++ b/lib/src/onehz/clinical/irregular_rhythm_state.dart @@ -0,0 +1,427 @@ +// Internal rule for SD1/SD2, pNNx thresholds and sustained-window voting. +// Task Force 1996 supports successive-difference HRV components, not this +// classifier or its SD1/SD2 thresholds; see hrv_time.dart. +// Internal irregular-rhythm screen, ESTIMATE with low confidence. Batch and +// stream share irregularWindowVerdict(); centered moments reorder floating-point +// sums, while exact counters preserve window evidence. Shared adjacency excludes +// timestamp-gap pairs. Diagnostics account for evidence without adding a +// clinical estimator. +// CLINICAL — incremental irregular-rhythm screen. +// +// [IrregularScreenState] is `irregularBeatScreen` as a fold over the corrected +// NN series that `RrCorrector` settles. Feed it each fold's settled NN with +// [IrregularScreenState.fold], and ask [IrregularScreenState.evaluate] with the +// corrector's provisional tail: the Metric equals +// `irregularBeatScreen(allNn, nnTimesMs: allTimes, ...)` over settled ++ tail. +// +// The aggregates (SD1/SD2 from the spread of successive differences and of the +// levels, pNNx) are running sums (Welford), equal to the batch's two-pass +// figures to ~1e-13. The "sustained across 5-minute windows" test is a COUNT of +// flagged / valid closed windows plus the one open window, which is evaluated +// on a copy at the end of the data, so it is exact. The per-window verdict is +// the batch's own (`irregularWindowVerdict`). +// +// [IrregularScreenState.evaluateDetailed] adds the evidence behind the verdict +// (`IrregularDiagnostics`): the same beat and window counts the batch call +// reports, including the final OPEN window, for a screen that ran and one that +// abstained. The checkpoint (version 2) carries two more counters for it, the +// NN entries seen and the windows closed; a version 1 checkpoint lacks them and +// is REFUSED rather than restored with counts it never kept. +// +// Version 3 adds the previous kept beat's time: two kept beats are successive +// only when no time hole lies between them (the batch's seam test), so the +// fold needs it. Older checkpoints are refused for the same reason. + +import 'dart:math' as math; + +import '../types.dart'; +import 'irregular_diagnostics.dart'; +import 'irregular_rhythm.dart'; +import 'irregular_window.dart'; + +class IrregularScreenState { + final double sd1sd2Flag, pnnThresholdMs, pnnFlagPct, windowMinutes; + final int minWindowBeats; + final double sustainedFraction; + + // ---- running state over the corrected NN list ---- + bool _prevKept = false; // the previous INPUT beat was inside [300, 2000] + double _prevV = 0; + // time of the previous kept beat; NaN when unknown (a NaN time is never + // within a seam, exactly as in the batch comparison) + double _prevT = double.nan; + int _nIn = 0; // NN entries folded, usable or not + int _nKept = 0; + // successive differences between adjacent kept beats + int _dN = 0; + double _dMean = 0, _dM2 = 0; + int _over = 0; + // levels of kept beats + int _lN = 0; + double _lMean = 0, _lM2 = 0; + // sustained-window bookkeeping + double? _winStart; + List _bk = []; + List _bkAdj = []; + int _total = 0, _valid = 0, _flagged = 0; + + IrregularScreenState({ + this.sd1sd2Flag = 0.70, + this.pnnThresholdMs = 70, + this.pnnFlagPct = 30, + this.windowMinutes = 5, + this.minWindowBeats = 40, + this.sustainedFraction = 0.5, + }); + + /// A broken window config fails CLOSED in the batch (never sustained). Here it + /// also means no windows are tracked, so a bad config cannot grow the state. + bool get _windowsOk => irregularWindowConfigOk( + windowMinutes: windowMinutes, + minWindowBeats: minWindowBeats, + sustainedFraction: sustainedFraction); + + /// Restores a state from [toJson], parameters included. Throws + /// [FormatException] on a checkpoint of another type or version, or one that + /// is malformed. + factory IrregularScreenState.fromJson(Map json) { + try { + return _restore(json); + } on FormatException { + rethrow; + } catch (e) { + throw FormatException('malformed IrregularScreenState checkpoint: $e'); + } + } + + static IrregularScreenState _restore(Map j) { + if (j['type'] != 'IrregularScreenState') { + throw FormatException( + 'not an IrregularScreenState checkpoint: ${j['type']}'); + } + if (j['version'] != 3) { + throw FormatException( + 'unsupported IrregularScreenState version ${j['version']}'); + } + double d(String k) => (j[k] as num).toDouble(); + final s = IrregularScreenState( + sd1sd2Flag: d('sd1sd2Flag'), + pnnThresholdMs: d('pnnThresholdMs'), + pnnFlagPct: d('pnnFlagPct'), + windowMinutes: d('windowMinutes'), + minWindowBeats: j['minWindowBeats'] as int, + sustainedFraction: d('sustainedFraction'), + ); + s._prevKept = j['prevKept'] as bool; + s._prevV = d('prevV'); + // Present in every version 3 checkpoint; null only when unknown (NaN). + if (!j.containsKey('prevT')) throw const FormatException('missing prevT'); + s._prevT = (j['prevT'] as num?)?.toDouble() ?? double.nan; + s._nIn = j['nIn'] as int; + s._nKept = j['nKept'] as int; + s._dN = j['dN'] as int; + s._dMean = d('dMean'); + s._dM2 = d('dM2'); + s._over = j['over'] as int; + s._lN = j['lN'] as int; + s._lMean = d('lMean'); + s._lM2 = d('lM2'); + s._winStart = (j['winStart'] as num?)?.toDouble(); + s._bk = [for (final x in j['bk'] as List) (x as num).toDouble()]; + final bkAdj = j['bkAdj'] as List; + if (bkAdj.any((x) => x is! int || (x != 0 && x != 1))) { + throw const FormatException('invalid bkAdj'); + } + s._bkAdj = [for (final x in bkAdj) x == 1]; + s._total = j['total'] as int; + s._valid = j['valid'] as int; + s._flagged = j['flagged'] as int; + if (s._lN != s._nKept) throw const FormatException('invalid lN'); + if (s._dN > math.max(0, s._nKept - 1)) { + throw const FormatException('invalid dN'); + } + if (s._total > s._nKept - s._bk.length) { + throw const FormatException('invalid total'); + } + if (s._bk.isNotEmpty && s._winStart == null) { + throw const FormatException('invalid winStart'); + } + final ok = s._bk.length == s._bkAdj.length && + s._nKept >= 0 && + s._nIn >= s._nKept && + s._total >= s._valid && + s._dN >= 0 && + s._lN >= 0 && + s._over >= 0 && + s._over <= s._dN && + s._flagged >= 0 && + s._flagged <= s._valid; + if (!ok) { + throw const FormatException('inconsistent IrregularScreenState checkpoint'); + } + return s; + } + + /// Checkpoint: `{'version': 3, 'type': 'IrregularScreenState', ...}`, plain + /// JSON, bounded (running sums plus at most one open 5-minute window). + Map toJson() => { + 'version': 3, + 'type': 'IrregularScreenState', + 'sd1sd2Flag': sd1sd2Flag, + 'pnnThresholdMs': pnnThresholdMs, + 'pnnFlagPct': pnnFlagPct, + 'windowMinutes': windowMinutes, + 'minWindowBeats': minWindowBeats, + 'sustainedFraction': sustainedFraction, + 'prevKept': _prevKept, + 'prevV': _prevV, + 'prevT': _prevT.isFinite ? _prevT : null, + 'nIn': _nIn, + 'nKept': _nKept, + 'dN': _dN, + 'dMean': _dMean, + 'dM2': _dM2, + 'over': _over, + 'lN': _lN, + 'lMean': _lMean, + 'lM2': _lM2, + 'winStart': _winStart, + 'bk': _bk, + 'bkAdj': [for (final a in _bkAdj) a ? 1 : 0], + 'total': _total, + 'valid': _valid, + 'flagged': _flagged, + }; + + IrregularScreenState _copy() => IrregularScreenState( + sd1sd2Flag: sd1sd2Flag, + pnnThresholdMs: pnnThresholdMs, + pnnFlagPct: pnnFlagPct, + windowMinutes: windowMinutes, + minWindowBeats: minWindowBeats, + sustainedFraction: sustainedFraction, + ) + .._prevKept = _prevKept + .._prevV = _prevV + .._prevT = _prevT + .._nIn = _nIn + .._nKept = _nKept + .._dN = _dN + .._dMean = _dMean + .._dM2 = _dM2 + .._over = _over + .._lN = _lN + .._lMean = _lMean + .._lM2 = _lM2 + .._winStart = _winStart + .._bk = List.of(_bk) + .._bkAdj = List.of(_bkAdj) + .._total = _total + .._valid = _valid + .._flagged = _flagged; + + /// Closes the open window into the valid / flagged counts. + void _closeWindow() { + final verdict = irregularWindowVerdict(_bk, _bkAdj, + sd1sd2Flag: sd1sd2Flag, + pnnThresholdMs: pnnThresholdMs, + pnnFlagPct: pnnFlagPct, + minWindowBeats: minWindowBeats); + if (_bk.isNotEmpty) _total++; + if (verdict != null) { + _valid++; + if (verdict) _flagged++; + } + _bk = []; + _bkAdj = []; + } + + void _add(double v, double tm) { + _nIn++; + final kept = v >= 300 && v <= 2000; + if (kept) { + // Adjacent = the previous INPUT beat was kept too and no time hole lies + // between them: no difference is ever taken across a skipped beat or a + // dropout (the batch's `successive`). + final adjacent = _prevKept && tm - _prevT <= v + 0.5; + if (adjacent) { + final d = v - _prevV; + _dN++; + final dl = d - _dMean; + _dMean += dl / _dN; + _dM2 += dl * (d - _dMean); + if (d.abs() > pnnThresholdMs) _over++; + } + _lN++; + final dl = v - _lMean; + _lMean += dl / _lN; + _lM2 += dl * (v - _lMean); + _nKept++; + if (_windowsOk) { + final start = _winStart ??= tm; + if (tm - start >= windowMinutes * 60000) { + _closeWindow(); + _winStart = tm; + } + _bk.add(v); + _bkAdj.add(adjacent); + } + } + _prevKept = kept; + // Only read when the next beat is adjacent to a KEPT one; a NaN stored here + // would make the checkpoint unencodable. + if (kept) { + _prevV = v; + _prevT = tm; + } + } + + /// Appends settled NN ([nn], ms) and their beat times ([nnTimesMs], same + /// length). Beats outside [300, 2000] ms are skipped exactly as + /// `irregularBeatScreen` skips them, including its no-diff-across-a-gap rule. + void fold(List nn, List nnTimesMs) { + if (nn.length != nnTimesMs.length) { + throw ArgumentError('nn has ${nn.length} entries, nnTimesMs ' + '${nnTimesMs.length}'); + } + for (var i = 0; i < nn.length; i++) { + _add(nn[i], nnTimesMs[i]); + } + } + + /// The screen over everything folded plus the provisional ([tailNn], + /// [tailNnTimesMs]). Does not change the state. Absent (same note) exactly + /// where `irregularBeatScreen` is absent. + Metric evaluate( + List tailNn, + List tailNnTimesMs, { + double artifactFraction = 0.0, + int minBeats = irregularScreenMinBeats, + double maxArtifact = 0.30, + }) => + evaluateDetailed(tailNn, tailNnTimesMs, + artifactFraction: artifactFraction, + minBeats: minBeats, + maxArtifact: maxArtifact) + .metric; + + /// [evaluate] plus the evidence behind the verdict, equal to + /// `irregularBeatScreenDetailed(allNn, nnTimesMs: allTimes, ...)` over + /// settled ++ tail. [cleaning] is what the corrector did upstream (from its + /// snapshot), when known; otherwise those counts are null. Windows are + /// counted, the open one included, whether or not the screen abstained; they + /// are null when this state's window config fails closed. + IrregularScreenResult evaluateDetailed( + List tailNn, + List tailNnTimesMs, { + double artifactFraction = 0.0, + int minBeats = irregularScreenMinBeats, + double maxArtifact = 0.30, + RrCleaningCounts? cleaning, + }) { + const inputs = ['rr_cleaned']; + final c = _copy()..fold(tailNn, tailNnTimesMs); + final thresholds = IrregularThresholds( + minBeats: minBeats, + maxArtifact: maxArtifact, + sd1sd2Flag: sd1sd2Flag, + pnnThresholdMs: pnnThresholdMs, + pnnFlagPct: pnnFlagPct, + windowMinutes: windowMinutes, + minWindowBeats: minWindowBeats, + sustainedFraction: sustainedFraction, + ); + // The open window closes at the end of the data, on the copy. + IrregularWindowCounts? windows; + if (c._windowsOk) { + final openBeats = c._bk.length; + final before = c._valid, beforeFlagged = c._flagged; + c._closeWindow(); + final open = openBeats == 0 + ? IrregularOpenWindow.none + : c._valid == before + ? IrregularOpenWindow.thin + : c._flagged > beforeFlagged + ? IrregularOpenWindow.flagged + : IrregularOpenWindow.unflagged; + windows = IrregularWindowCounts( + total: c._total, + valid: c._valid, + flagged: c._flagged, + openBeats: openBeats, + open: open, + ); + } + IrregularDiagnostics diag(IrregularAbstain? why) => IrregularDiagnostics( + abstain: why, + rrRaw: cleaning?.raw, + corrected: cleaning?.corrected, + dropped: cleaning?.dropped, + nnIn: c._nIn, + nnKept: c._nKept, + artifactFraction: diagnosticArtifactFraction(artifactFraction, cleaning), + windows: windows, + thresholds: thresholds, + ); + IrregularScreenResult abstain(IrregularAbstain why, String note) => + IrregularScreenResult( + Metric.absent( + tier: Tier.estimate, inputs_used: inputs, note: note), + diag(why), + ); + + // Gate on contiguous pairs, as the batch does. + if (c._dN < minBeats - 1) { + return abstain(IrregularAbstain.tooFewBeats, + 'too few clean beats for an irregular-rhythm screen'); + } + if (artifactFraction > maxArtifact) { + return abstain( + IrregularAbstain.artifact, + 'artifact fraction ${(artifactFraction * 100).round()}% > ' + '${(maxArtifact * 100).round()}% — screen suppressed on noisy RR'); + } + final sdsd = c._dN < 2 ? null : math.sqrt(c._dM2 / (c._dN - 1)); + final sdnn = c._lN < 2 ? null : math.sqrt(c._lM2 / (c._lN - 1)); + if (sdsd == null || sdnn == null) { + return abstain(IrregularAbstain.noSuccessivePairs, + 'no successive clean beats to build a Poincare plot from'); + } + final sd1 = sdsd / math.sqrt2; + final v = 2 * sdnn * sdnn - sd1 * sd1; + final sd2 = v > 0 ? math.sqrt(v) : 0.0; + if (sd2 <= 0) { + return abstain( + IrregularAbstain.noLongTermVariability, + 'no long-term variability (SD2 = 0) — the SD1/SD2 ratio is ' + 'undefined, not "perfectly regular"'); + } + final ratio = sd1 / sd2; + final pnnPct = c._dN == 0 ? 0.0 : 100.0 * c._over / c._dN; + final aggregateHigh = ratio >= sd1sd2Flag && pnnPct >= pnnFlagPct; + final flag = aggregateHigh && + windows != null && + c._valid != 0 && + c._flagged / c._valid >= sustainedFraction; + final conf = (c._nKept / 5000.0 * (1 - artifactFraction)).clamp(0.2, 0.9); + return IrregularScreenResult( + Metric( + value: IrregularRhythm( + sd1: sd1, + sd2: sd2, + sd1sd2: ratio, + pnnPct: pnnPct, + nBeats: c._nKept, + flag: flag, + ), + confidence: conf, + tier: Tier.estimate, + inputs_used: inputs, + note: 'irregular-rhythm SCREEN (not a diagnosis): Poincaré SD1/SD2 + pNN' + '${pnnThresholdMs.round()}. PRV not ECG — wrist pulse misses P-waves. ' + 'Discuss with a clinician only if you have symptoms.', + ), + diag(null), + ); + } +} diff --git a/lib/src/onehz/clinical/irregular_window.dart b/lib/src/onehz/clinical/irregular_window.dart new file mode 100644 index 0000000..407fe00 --- /dev/null +++ b/lib/src/onehz/clinical/irregular_window.dart @@ -0,0 +1,65 @@ +// Internal rule for SD1/SD2, pNNx thresholds and sustained-window voting. +// Task Force 1996 supports successive-difference HRV components, not this +// classifier or its SD1/SD2 thresholds; see hrv_time.dart. +// Internal irregular-rhythm screen, ESTIMATE with low confidence. Batch and +// stream share irregularWindowVerdict(); centered moments reorder floating-point +// sums, while exact counters preserve window evidence. Shared adjacency excludes +// timestamp-gap pairs. Diagnostics account for evidence without adding a +// clinical estimator. +// CLINICAL — the per-window pieces of the irregular-rhythm screen, shared by +// `irregularBeatScreen` (whole series) and `IrregularScreenState` (streaming) +// so the two cannot disagree about what a flagged window is. +// +// Not exported from `onehz.dart`. + +import 'dart:math' as math; +import '../util.dart'; + +/// Fail-CLOSED check on the sustained-window config: a misconfigured caller +/// must never manufacture a medical false positive. +bool irregularWindowConfigOk( + {required double windowMinutes, + required int minWindowBeats, + required double sustainedFraction}) { + return windowMinutes.isFinite && + windowMinutes > 0 && + minWindowBeats >= 2 && + sustainedFraction.isFinite && + sustainedFraction >= 0 && + sustainedFraction <= 1; +} + +/// One closed window of clean beats: null when it is too thin to count as a +/// window, otherwise whether SD1/SD2 and pNNx both clear their flag lines. +/// [adjacent] is aligned to [bucket]: whether each beat directly followed the +/// previous one in the ORIGINAL series, with no time hole between them (never +/// diff across a dropped beat or a dropout). +bool? irregularWindowVerdict( + List bucket, + List adjacent, { + required double sd1sd2Flag, + required double pnnThresholdMs, + required double pnnFlagPct, + required int minWindowBeats, +}) { + // Mirror the aggregate's `keep[i] && keep[i-1]` guard: never diff across a + // beat that was dropped as an artifact in the original series, even though + // it's now a consecutive pair in this compacted bucket. + final diffs = [ + for (var i = 1; i < bucket.length; i++) + if (adjacent[i]) bucket[i] - bucket[i - 1] + ]; + // Valid on contiguous pairs, not beats (same reason as the aggregate). + if (diffs.length < minWindowBeats - 1) return null; + final sdsd = stddev(diffs); + final sdnn = stddev(bucket); + if (sdsd == null || sdnn == null) return false; + final sd1 = sdsd / math.sqrt2; + final v = 2 * sdnn * sdnn - sd1 * sd1; + final sd2 = v > 0 ? math.sqrt(v) : 0.0; + if (sd2 <= 0) return false; + final ratio = sd1 / sd2; + final over = diffs.where((d) => d.abs() > pnnThresholdMs).length; + final pnn = 100.0 * over / diffs.length; + return ratio >= sd1sd2Flag && pnn >= pnnFlagPct; +} diff --git a/lib/src/onehz/clinical/load_trimp.dart b/lib/src/onehz/clinical/load_trimp.dart index e8c7b3d..a62a9c6 100644 --- a/lib/src/onehz/clinical/load_trimp.dart +++ b/lib/src/onehz/clinical/load_trimp.dart @@ -23,8 +23,11 @@ // prediction. import 'dart:math' as math; +import 'dart:typed_data'; +import '../../aggregate/aggregator.dart'; import '../types.dart'; import '../util.dart'; +import 'load_trimp_aggregator.dart'; /// Banister TRIMP over a series of per-minute mean HRs. /// @@ -45,41 +48,11 @@ Metric banisterTrimp( required double? maxHr, required Sex sex, }) { - const inputs = ['hr_per_min', 'resting_hr', 'max_hr']; - // `maxHr <= restingHr` is FALSE when either is NaN, so the finiteness checks - // are load-bearing: a NaN anchor otherwise makes a NaN reserve, and every - // clamp below (`hrr < 0`, `hrr > 1`) is false for NaN too, so the sum comes - // out NaN inside a PRESENT metric. - if (restingHr == null || - maxHr == null || - !restingHr.isFinite || - !maxHr.isFinite || - maxHr <= restingHr || - hrPerMin.isEmpty) { - return const Metric.absent( - tier: Tier.estimate, - inputs_used: inputs, - note: 'Banister TRIMP needs finite measured RHR and HRmax (HRmax>RHR)', - ); - } - final reserve = maxHr - restingHr; - var trimp = 0.0; - for (final hr in hrPerMin) { - if (!hr.isFinite || hr <= 0) continue; // off-skin + non-finite guard - var hrr = (hr - restingHr) / reserve; - if (hrr < 0) hrr = 0; - if (hrr > 1) hrr = 1; - // ONE Banister implementation for the whole package: the sex-specific - // weighting factor y lives in [StrainScorer.banisterY]. 1 minute each. - trimp += 1.0 * hrr * StrainScorer.banisterY(hrr, female: sex == Sex.female); - } - return Metric( - value: trimp, - confidence: 0.6, - tier: Tier.estimate, - inputs_used: inputs, - note: 'Banister exponential TRIMP (wrist HR estimate)', - ); + // Runs [banisterTrimpAggregator] over the whole column in one slice, so the + // batch and chunked paths share one implementation. The aggregator carries + // the finiteness guards on the anchors and the off-skin / non-finite filter. + return banisterTrimpAggregator(restingHr: restingHr, maxHr: maxHr, sex: sex) + .run(DoubleColumn(Float64List.fromList(hrPerMin))); } /// Fraction of heart-rate reserve that simply BEING AWAKE costs — THE REFERENCE @@ -964,32 +937,13 @@ Metric ctlAtlTsb(List dailyTrimp, double atlDays = 7, int minDays = ctlAtlMinDays, int primeDays = 7}) { - const inputs = ['daily_trimp']; - if (dailyTrimp.length < minDays) { - return Metric.absent( - tier: Tier.estimate, - inputs_used: inputs, - note: needBaselineNote(have: dailyTrimp.length, need: minDays), - ); - } - final lc = 1 - math.exp(-1 / ctlDays); - final la = 1 - math.exp(-1 / atlDays); - final prime = math.min(math.max(primeDays, 1), dailyTrimp.length); - final seed = mean(dailyTrimp.sublist(0, prime))!; - var ctl = seed; - var atl = seed; - for (var i = prime; i < dailyTrimp.length; i++) { - ctl = ctl + lc * (dailyTrimp[i] - ctl); - atl = atl + la * (dailyTrimp[i] - atl); - } - final conf = (dailyTrimp.length / 42.0).clamp(0.3, 0.85); - return Metric( - value: LoadState(ctl, atl, ctl - atl), - confidence: conf, - tier: Tier.estimate, - inputs_used: inputs, - note: - 'Banister CTL(42d)/ATL(7d)/TSB, primed from the first $prime observed ' - 'days; descriptive load, not injury risk', - ); + // Runs [ctlAtlTsbAggregator] over the whole series in one slice. The + // aggregator holds the abstain rule (below [minDays]), the prime mean of the + // first [primeDays] days and the EWMA over the remainder. + return ctlAtlTsbAggregator( + ctlDays: ctlDays, + atlDays: atlDays, + minDays: minDays, + primeDays: primeDays) + .run(DoubleColumn(Float64List.fromList(dailyTrimp))); } diff --git a/lib/src/onehz/clinical/load_trimp_aggregator.dart b/lib/src/onehz/clinical/load_trimp_aggregator.dart new file mode 100644 index 0000000..45def59 --- /dev/null +++ b/lib/src/onehz/clinical/load_trimp_aggregator.dart @@ -0,0 +1,146 @@ +import 'dart:math' as math; + +import '../../aggregate/aggregator.dart'; +import '../../aggregate/json_support.dart'; +import '../../aggregate/primitives.dart'; +import '../types.dart'; +import 'load_trimp.dart'; + +const _trimpInputs = ['hr_per_min', 'resting_hr', 'max_hr']; +const _trimpNote = + 'Banister TRIMP needs finite measured RHR and HRmax (HRmax>RHR)'; + +// Morton, R. H., Fitz-Clarke, J. R., Banister, E. W. (1990). Modeling human +// performance in running. Journal of Applied Physiology 69(3):1171-1177. +// Supports HR-reserve exponential TRIMP weighting and the impulse-response +// model; the sex-specific coefficients are retained from Banister 1991. +// banisterTrimp() runs this per-minute HR- +// reserve sum in one slice. Merge order changes rounding; checked relative +// tolerance is 1e-12. Retains ESTIMATE tier and confidence 0.6. +/// Builds a mergeable Banister TRIMP summary over per-minute mean HR values. +/// +/// The resulting metric has the same missing-input and empty-input behaviour as +/// [banisterTrimp]. +Aggregator> banisterTrimpAggregator({ + required double? restingHr, + required double? maxHr, + required Sex sex, +}) { + // `maxHr <= restingHr` is FALSE when either is NaN, so the finiteness checks + // are load-bearing: a NaN anchor otherwise makes a NaN reserve, and every + // clamp below (`hrr < 0`, `hrr > 1`) is false for NaN too, so the sum comes + // out NaN inside a PRESENT metric. + final anchorsValid = restingHr != null && + maxHr != null && + restingHr.isFinite && + maxHr.isFinite && + maxHr > restingHr; + final reserve = anchorsValid ? maxHr - restingHr : 0.0; + final core = both( + Count(), + SumOf((batch) => batch.values, (hr) { + if (!anchorsValid || !hr.isFinite || hr <= 0) return double.nan; + // ONE Banister implementation for the whole package: the sex-specific + // weighting factor y lives in [StrainScorer.banisterY]. 1 minute each. + var hrr = (hr - restingHr) / reserve; + if (hrr < 0) hrr = 0.0; + if (hrr > 1) hrr = 1.0; + return hrr * StrainScorer.banisterY(hrr, female: sex == Sex.female); + }), + ); + return core.map((result) { + if (!anchorsValid || result.$1 == 0) { + return const Metric.absent( + tier: Tier.estimate, + inputs_used: _trimpInputs, + note: _trimpNote, + ); + } + return Metric( + value: result.$2, + confidence: 0.6, + tier: Tier.estimate, + inputs_used: _trimpInputs, + note: 'Banister exponential TRIMP (wrist HR estimate)', + ); + }); +} + +// Banister, E. W., Calvert, T. W., Savage, M. V., Bach, T. (1975). A systems +// model of training for athletic performance. Australian Journal of Sports +// Medicine 7:57-61. See also Morton, Fitz-Clarke & Banister 1990 above. +// CTL/ATL are a discrete EWMA fitness/fatigue adaptation, TSB = CTL - ATL; +// 42/7-day defaults and bounded-prefix priming are implementation choices. +// With a=1-lambda, f_y(f_x(c))=a*a*c+a*lambda*x+lambda*y, algebraically +// equal to two EWMA recurrence steps. +// Banister impulse-response model, now also the ctlAtlTsb() one-slice path. The +// identical bounded-prefix prime seeds affine compositions of the same EWMA +// recurrence. Composition is algebraically equal; checked absolute and relative +// tolerances are 1e-9. +/// Builds a mergeable CTL/ATL/TSB summary over a daily-TRIMP column (oldest +/// first), matching [ctlAtlTsb]. +/// +/// Both accumulators are primed with the mean of the first [primeDays] rows +/// (a bounded prefix), then each later row is one EWMA step, kept as an affine +/// map so chunks merge. With fewer rows than [primeDays] the prime is the row +/// count, as in the reference. +Aggregator> ctlAtlTsbAggregator({ + double ctlDays = 42, + double atlDays = 7, + int minDays = ctlAtlMinDays, + int primeDays = 7, +}) { + const inputs = ['daily_trimp']; + final lc = 1 - math.exp(-1 / ctlDays); + final la = 1 - math.exp(-1 / atlDays); + final core = both( + const Count(), + Prefix( + math.max(primeDays, 1), + both( + const Count(), + Sum((batch) => batch.values), + ), + both( + Ewma((batch) => batch.values, lc), + Ewma((batch) => batch.values, la), + ), + batchFromJson: DoubleColumn.fromJson, + ), + ); + final prime = math.max(primeDays, 1); + return validateCheckpoint(core, (json) { + final state = decodeList(json); + if (state.length != 2) throw const FormatException('Expected two states'); + final days = decodeCount(state[0]); + final rows = field(decodeMap(state[1]), 'rows'); + if (days == 0) { + if (rows != null) throw const FormatException('Prefix rows without days'); + return; + } + if (rows == null || decodeList(rows).length != (days < prime ? days : prime)) { + throw const FormatException('Prefix rows disagree with days'); + } + }).map((result) { + final days = result.$1; + if (days < minDays) { + return Metric.absent( + tier: Tier.estimate, + inputs_used: inputs, + note: needBaselineNote(have: days, need: minDays), + ); + } + final ((primed, primeSum), (ctlMap, atlMap)) = result.$2; + final seed = primeSum / primed; + final ctl = ctlMap.apply(seed); + final atl = atlMap.apply(seed); + return Metric( + value: LoadState(ctl, atl, ctl - atl), + confidence: (days / 42.0).clamp(0.3, 0.85), + tier: Tier.estimate, + inputs_used: inputs, + note: 'Banister CTL(42d)/ATL(7d)/TSB, primed from the first $primed ' + 'observed days; descriptive load, not injury risk', + ); + }); +} diff --git a/lib/src/onehz/clinical/nocturnal.dart b/lib/src/onehz/clinical/nocturnal.dart index d34cd23..804aa6f 100644 --- a/lib/src/onehz/clinical/nocturnal.dart +++ b/lib/src/onehz/clinical/nocturnal.dart @@ -167,18 +167,36 @@ const int hrDipMinSamples = 300; /// handful of samples is fabrication, not measurement. Metric hrDip(List dayHr, List nightHr, {int minSamples = hrDipMinSamples}) { - const inputs = ['hr_1hz_day', 'hr_1hz_night']; final dv = dayHr.where((h) => h > 0).toList(); + return _hrDip(dv.length, mean(dv), nightHr, minSamples); +} + +/// [hrDip] from the day side's valid-sample count and sum, accumulated in +/// sample order (so `daySum / dayCount` is the batch day mean exactly). For a +/// caller that keeps the waking day as running totals instead of a list. +// HR-dip bands and minimum sample count are an internal rule, ESTIMATE with +// low confidence. The >=10%, 0-10%, <0% bands borrow the BP-dip convention. +// Avram 2019 / Dial 2025 above concern nocturnal RHR, not validation of these +// HR-dip bands. The night mean here is the mean of all valid night samples. +// Computes hrDip() with identical day mean and shared scoring when daySum is +// accumulated in sample order over the same valid samples. +Metric hrDipFromDayTotals(int dayCount, double daySum, + List nightHr, {int minSamples = hrDipMinSamples}) => + _hrDip(dayCount, dayCount == 0 ? null : daySum / dayCount, nightHr, + minSamples); + +Metric _hrDip( + int dayCount, double? dm, List nightHr, int minSamples) { + const inputs = ['hr_1hz_day', 'hr_1hz_night']; final nv = nightHr.where((h) => h > 0).toList(); - if (dv.length < minSamples || nv.length < minSamples) { + if (dayCount < minSamples || nv.length < minSamples) { return Metric.absent( tier: Tier.high, inputs_used: inputs, note: 'HR dip needs ≥$minSamples valid samples on each of day and night ' - '(have day=${dv.length}, night=${nv.length})', + '(have day=$dayCount, night=${nv.length})', ); } - final dm = mean(dv); final nm = mean(nv); if (dm == null || nm == null || dm <= 0) { return const Metric.absent( @@ -189,7 +207,7 @@ Metric hrDip(List dayHr, List nightHr, } final dip = (dm - nm) / dm * 100; final band = dip >= 10 ? 'dipper' : (dip >= 0 ? 'non_dipper' : 'riser'); - final conf = ((dv.length + nv.length) / 14400.0).clamp(0.4, 0.9); + final conf = ((dayCount + nv.length) / 14400.0).clamp(0.4, 0.9); return Metric( value: HrDip(dip, dm, nm, band), confidence: conf, diff --git a/lib/src/onehz/clinical/prsa_aggregator.dart b/lib/src/onehz/clinical/prsa_aggregator.dart new file mode 100644 index 0000000..fdb0a7e --- /dev/null +++ b/lib/src/onehz/clinical/prsa_aggregator.dart @@ -0,0 +1,155 @@ +import 'dart:typed_data'; + +import '../../aggregate/aggregator.dart'; +import '../../aggregate/json_support.dart'; +import '../../aggregate/primitives.dart'; +import '../types.dart'; +import 'prsa.dart'; + +// Bauer et al. 2006 PRSA, as in decelerationCapacity()/accelerationCapacity(). +// Lookback(2*l+1) reproduces the same anchor bounds and [i-l,i+l) profile. +// Merged profile sums and anchor counts give the same mathematical result. +// Checked capacity/profile absolute and relative tolerances are 1e-9; confidence +// absolute tolerance is 1e-12. +/// Running PRSA sums: the per-offset profile total and the anchor count. +class _PrsaState { + _PrsaState(int l) + : profile = Float64List(2 * l), + anchors = 0; + _PrsaState._(this.profile, this.anchors); + final Float64List profile; + int anchors; +} + +/// Folds every window of `2l + 1` rows. The window starting at `s` belongs to +/// the anchor candidate `i = s + l`; it tests (x[i-1], x[i]) and adds rows +/// `s .. s + 2l - 1` (the reference's `[i - l, i + l)`). The extra last row +/// keeps the reference's `i < n - l` bound. Run under `Lookback(2l + 1)`. +class _PrsaWindows extends Aggregator<_PrsaState, DoubleColumn, _PrsaState> { + const _PrsaWindows(this.l, this.deceleration, this.cap); + final int l; + final bool deceleration; + final double cap; + + @override + _PrsaState empty() => _PrsaState(l); + @override + _PrsaState addSlice(_PrsaState state, DoubleColumn batch, int start, int end) { + final x = batch.values; + for (var s = start; s + 2 * l + 1 <= end; s++) { + final prev = x[s + l - 1]; + if (prev <= 0) continue; + final cur = x[s + l]; + if ((cur - prev > 0) != deceleration) continue; + if ((cur / prev - 1).abs() > cap) continue; + state.anchors++; + for (var k = 0; k < 2 * l; k++) { + state.profile[k] += x[s + k]; + } + } + return state; + } + + @override + _PrsaState combine(_PrsaState left, _PrsaState right) { + for (var k = 0; k < 2 * l; k++) { + left.profile[k] += right.profile[k]; + } + left.anchors += right.anchors; + return left; + } + + @override + _PrsaState finish(_PrsaState state) => state; + @override + bool get persistable => true; + @override + Object? encodeState(_PrsaState state) => + {'profile': encodeDoubles(state.profile), 'anchors': state.anchors}; + @override + _PrsaState decodeState(Object? json) { + final map = decodeMap(json); + final profile = decodeDoubles(field(map, 'profile')); + final anchors = decodeCount(field(map, 'anchors')); + if (profile.length != 2 * l) { + throw FormatException('PRSA profile needs $l * 2 entries'); + } + if (anchors < 0) throw const FormatException('Negative PRSA anchors'); + return _PrsaState._(profile, anchors); + } +} + +/// Mergeable PRSA deceleration or acceleration capacity, matching +/// [decelerationCapacity] and [accelerationCapacity]. +Aggregator> prsaAggregator({ + required bool deceleration, + int l = 2, + double anchorRatioCap = 0.05, +}) { + final kind = deceleration ? 'DC' : 'AC'; + const inputs = ['rr_cleaned']; + if (l < 2) { + // Same absent Metric as the reference; Lookback needs a valid window. + final absent = Metric.absent( + tier: Tier.high, + inputs_used: inputs, + note: 'PRSA ($kind) needs l≥2 for the s=2 Haar contrast', + ); + return Count().map((_) => absent); + } + final core = both( + Count(), + Lookback( + 2 * l + 1, + _PrsaWindows(l, deceleration, anchorRatioCap), + batchFromJson: DoubleColumn.fromJson, + ), + ); + return validateCheckpoint(core, (json) { + final state = decodeList(json); + if (state.length != 2) throw const FormatException('Expected two states'); + final count = decodeCount(state[0]); + final lookback = decodeMap(state[1]); + if (decodeCount(field(lookback, 'length')) != count) { + throw const FormatException('Lookback length disagrees with beat count'); + } + final anchors = decodeCount(field(decodeMap(field(lookback, 'inner')), 'anchors')); + final windows = count > 2 * l ? count - 2 * l : 0; + if (anchors > windows) { + throw const FormatException('PRSA anchors exceed windows'); + } + }).map((result) { + final (n, sums) = result; + if (n < 2 * l + 4) { + return Metric.absent( + tier: Tier.high, + inputs_used: inputs, + note: 'too few beats for PRSA ($kind)', + ); + } + final anchors = sums.anchors; + if (anchors == 0) { + return Metric.absent( + tier: Tier.high, + inputs_used: inputs, + note: 'no valid PRSA anchors ($kind)', + ); + } + final profile = [for (final v in sums.profile) v / anchors]; + final capacity = + (profile[l] + profile[l + 1] - profile[l - 1] - profile[l - 2]) / 4; + return Metric( + value: PrsaResult( + capacity: capacity, + profile: profile, + anchors: anchors, + kind: kind, + ), + confidence: (anchors / 1000.0).clamp(0.3, 0.95), + tier: Tier.high, + inputs_used: inputs, + note: 'Bauer 2006 PRSA $kind; needs a long overnight RR record for ' + 'mortality-grade stability', + ); + }); +} diff --git a/lib/src/onehz/clinical/readiness_lnrmssd.dart b/lib/src/onehz/clinical/readiness_lnrmssd.dart index 296dcf5..6dca13e 100644 --- a/lib/src/onehz/clinical/readiness_lnrmssd.dart +++ b/lib/src/onehz/clinical/readiness_lnrmssd.dart @@ -62,37 +62,84 @@ Metric readinessLnRmssd( int windowDays = 7, int minNights = readinessLnRmssdMinNights, }) { - const inputs = ['ln_rmssd_history']; if (historyLnRmssd.length < minNights) { - return Metric.absent( - tier: Tier.high, - inputs_used: inputs, - note: needBaselineNote(have: historyLnRmssd.length, need: minNights), - ); + return _tooShort(historyLnRmssd.length, minNights); } - final today = historyLnRmssd.last; final n = historyLnRmssd.length; final start = n - 1 - windowDays < 0 ? 0 : n - 1 - windowDays; - // this used to be historyLnRmssd.sublist(start), which runs to the END of - // the list - including tonight's own value in its own baseline. that - // pulls the mean/sd toward tonight, understating how far off a genuinely - // suppressed/elevated night actually is, worst right when the window is - // smallest (minNights). the baseline has to be strictly prior nights. - final List priorWindow; // all-or-nothing: calendarDays falls back to the row index for a label it // can't parse, and an index compared against epoch days is meaningless. if (dates != null && dates.length == n && dates.every((d) => DateTime.tryParse(d) != null)) { final day = calendarDays(dates); - priorWindow = [ - for (var i = 0; i < n - 1; i++) - if (day[i] < day[n - 1] && day[n - 1] - day[i] <= windowDays) - historyLnRmssd[i] - ]; - } else { - priorWindow = historyLnRmssd.sublist(start, n - 1); + return readinessLnRmssdFromSuffix( + n, + [ + for (var i = 0; i < n - 1; i++) + if (day[i] < day[n - 1] && day[n - 1] - day[i] <= windowDays) + historyLnRmssd[i], + historyLnRmssd.last, + ], + meanNnTodayMs: meanNnTodayMs, + windowDays: windowDays, + minNights: minNights, + ); } + if (windowDays >= 0) { + return readinessLnRmssdFromSuffix( + n, + historyLnRmssd.sublist(start, n), + meanNnTodayMs: meanNnTodayMs, + windowDays: windowDays, + minNights: minNights, + ); + } + // A negative window cannot be built as a suffix block, so it keeps the old + // path: otherwise sublist(start, n - 1) with start > n - 1 throws RangeError, + // as it always did. + final priorWindow = historyLnRmssd.sublist(start, n - 1); + return readinessLnRmssdFromSuffix( + n, + [...priorWindow, historyLnRmssd.last], + meanNnTodayMs: meanNnTodayMs, + windowDays: windowDays, + minNights: minNights, + ); +} + +const _inputs = ['ln_rmssd_history']; + +/// The absent result for a history shorter than [minNights]. +Metric _tooShort(int n, int minNights) => + Metric.absent( + tier: Tier.high, + inputs_used: _inputs, + note: needBaselineNote(have: n, need: minNights), + ); + +/// Shared by [readinessLnRmssd] and the aggregator (not public API): the +/// arithmetic after the length check, given the history length [n] and its +/// last min(windowDays + 1, n) values [suffix] (tonight last). +// Plews et al. 2013, as implemented by readinessLnRmssd(). Retains the history +// count and relevant prior-night suffix, then calls identical scoring +// arithmetic. The aggregator uses positional history; calendar-date filtering +// remains in the batch wrapper. +Metric readinessLnRmssdFromSuffix( + int n, + List suffix, { + double? meanNnTodayMs, + required int windowDays, + required int minNights, +}) { + const inputs = _inputs; + final today = suffix.last; + // this used to be historyLnRmssd.sublist(start), which runs to the END of + // the list - including tonight's own value in its own baseline. that + // pulls the mean/sd toward tonight, understating how far off a genuinely + // suppressed/elevated night actually is, worst right when the window is + // smallest (minNights). the baseline has to be strictly prior nights. + final priorWindow = suffix.sublist(0, suffix.length - 1); if (priorWindow.isEmpty || priorWindow.length < minNights - 1) { // empty only if minNights got set to 1 somewhere; short when a wear gap // left too few nights inside the calendar window. either way there's no diff --git a/lib/src/onehz/clinical/readiness_lnrmssd_aggregator.dart b/lib/src/onehz/clinical/readiness_lnrmssd_aggregator.dart new file mode 100644 index 0000000..5ad666f --- /dev/null +++ b/lib/src/onehz/clinical/readiness_lnrmssd_aggregator.dart @@ -0,0 +1,59 @@ +import '../../aggregate/aggregator.dart'; +import '../../aggregate/json_support.dart'; +import '../../aggregate/primitives.dart'; +import '../types.dart'; +import 'readiness_lnrmssd.dart'; + +// Plews et al. 2013, as implemented by readinessLnRmssd(). Retains the history +// count and relevant prior-night suffix, then calls identical scoring +// arithmetic. The aggregator uses positional history; calendar-date filtering +// remains in the batch wrapper. +/// Builds a mergeable lnRMSSD readiness summary over a nightly history column +/// (oldest first, tonight last), matching [readinessLnRmssd] exactly. +/// +/// Only the history length and the last [windowDays] + 1 nights matter, so the +/// state is a count and a bounded suffix. [windowDays] must be at least 0. +Aggregator> + readinessLnRmssdAggregator({ + double? meanNnTodayMs, + int windowDays = 7, + int minNights = readinessLnRmssdMinNights, +}) { + assert(windowDays >= 0); + const maxLastRows = 1 << 62; + final keptRows = windowDays >= maxLastRows ? maxLastRows : windowDays + 1; + final core = both( + const Count(), + LastRows(keptRows, + batchFromJson: DoubleColumn.fromJson), + ); + return validateCheckpoint(core, (json) { + final state = decodeList(json); + if (state.length != 2) throw const FormatException('Expected two states'); + final count = decodeCount(state[0]); + final rows = field(decodeMap(state[1]), 'rows'); + if (count == 0) { + if (rows != null) throw const FormatException('Rows without history'); + return; + } + if (rows == null || decodeList(rows).length != (count < keptRows ? count : keptRows)) { + throw const FormatException('History rows disagree with count'); + } + }).map((result) { + final n = result.$1; + if (n < minNights) { + return Metric.absent( + tier: Tier.high, + inputs_used: const ['ln_rmssd_history'], + note: needBaselineNote(have: n, need: minNights), + ); + } + return readinessLnRmssdFromSuffix( + n, + result.$2!.values.toList(), + meanNnTodayMs: meanNnTodayMs, + windowDays: windowDays, + minNights: minNights, + ); + }); +} diff --git a/lib/src/onehz/clinical/rmssd_gate.dart b/lib/src/onehz/clinical/rmssd_gate.dart new file mode 100644 index 0000000..6c8972d --- /dev/null +++ b/lib/src/onehz/clinical/rmssd_gate.dart @@ -0,0 +1,319 @@ +// Rejection and breathing-line exemption thresholds are an internal rule. +// Welch 1967 spectral estimation; Hirsch & Bishop 1981 provides the +// breathing/RSA basis. This is hrvTime()'s existing internal jitter policy, +// ESTIMATE with low confidence, not a published classifier. Shared formulas +// preserve the calculation; reordered sums can change decisions within rounding +// of a threshold. +// CLINICAL — the spectral half of the RMSSD jitter gate, as pieces. +// +// `hrvTime`, `nocturnalRmssd` and `sleepSessionWindowedRmssd` (hrv_time.dart) +// read the successive-difference runs themselves when the lag-1 ACF is under +// [kNnDiffAcf1Floor]: a respiratory line on a low white floor exempts the night +// ([nnDiffNoiseShare]), and a refused night may still publish from the 5-min +// windows on a breathing line steady in Hz. Both reduce to sums over Welch +// segments, so this file holds them as SUMS plus the decisions over those +// sums, written once: the batch fills the sums from the runs in hand, and +// `hrvTimeAggregator` fills the same sums from mergeable per-chunk state. +// +// Not exported from `onehz.dart`. + +import 'dart:math' as math; + +import '../util.dart'; +import 'hrv_time.dart' show kNnDiffNoiseShareCeiling; + +/// Welch segment length (beats) of the noise-share spectrum, the first and last +/// bins of its 0.15–0.5 cycles/beat band. +const int kNoiseSegment = 64, kNoiseLoBin = 10, kNoiseHiBin = kNoiseSegment ~/ 2; + +/// The segment's Hann window (symmetric, n − 1 in the denominator). +final List noiseHann = [ + for (var i = 0; i < kNoiseSegment; i++) + 0.5 - 0.5 * math.cos(2 * math.pi * i / (kNoiseSegment - 1)) +]; + +final double _noiseHannSq = () { + var w2 = 0.0; + for (final x in noiseHann) { + w2 += x * x; + } + return w2; +}(); + +final List> _cosT = [ + for (var k = kNoiseLoBin; k <= kNoiseHiBin; k++) + [for (var i = 0; i < kNoiseSegment; i++) math.cos(2 * math.pi * k * i / kNoiseSegment)] +]; +final List> _sinT = [ + for (var k = kNoiseLoBin; k <= kNoiseHiBin; k++) + [for (var i = 0; i < kNoiseSegment; i++) math.sin(2 * math.pi * k * i / kNoiseSegment)] +]; + +/// Adds the Welch segment `x[s .. s + 63]` (RR levels integrated from the +/// differences) to [psd] (bins [kNoiseLoBin]..[kNoiseHiBin]) and returns its +/// band power. +double addNoiseSegment(List x, int s, List psd) { + const n = kNoiseSegment; + var m = 0.0; + for (var i = 0; i < n; i++) { + m += x[s + i]; + } + m /= n; + final v = [for (var i = 0; i < n; i++) (x[s + i] - m) * noiseHann[i]]; + var p = 0.0; + for (var k = kNoiseLoBin; k <= kNoiseHiBin; k++) { + final c = _cosT[k - kNoiseLoBin], sn = _sinT[k - kNoiseLoBin]; + var re = 0.0, im = 0.0; + for (var i = 0; i < n; i++) { + re += v[i] * c[i]; + im -= v[i] * sn[i]; + } + psd[k] += re * re + im * im; + p += re * re + im * im; + } + return p; +} + +/// The sums [nnDiffNoiseShare] decides on. +class NoiseShareSums { + final List psd = List.filled(kNoiseHiBin + 1, 0.0); + int segs = 0, nd = 0; + double ssd = 0.0, wsd = 0.0, wsum = 0.0, pSum = 0.0, p2Sum = 0.0; + + /// One contiguous run of differences, in the batch's own order. + void addRun(List r) { + const n = kNoiseSegment; + for (final d in r) { + ssd += d * d; + nd++; + } + // Integrate the run back to RR levels (up to a constant the mean removes). + final x = List.filled(r.length + 1, 0.0); + for (var i = 0; i < r.length; i++) { + x[i + 1] = x[i] + r[i]; + } + final cov = List.filled(x.length, 0.0); // Σ w² per sample + for (var s = 0; s + n <= x.length; s += n ~/ 2) { + for (var i = 0; i < n; i++) { + cov[s + i] += noiseHann[i] * noiseHann[i]; + } + final p = addNoiseSegment(x, s, psd); + pSum += p; + p2Sum += p * p; + segs++; + } + for (var i = 0; i < r.length; i++) { + final c = (cov[i] + cov[i + 1]) / 2; + wsd += c * r[i] * r[i]; + wsum += c; + } + } +} + +/// The white-noise share from [s] (see [nnDiffNoiseShare] in hrv_time.dart for +/// what it measures and every refusal). [lattice] says whether the differences +/// sit on a coarse beat-time grid ([coarseLattice]); it is only asked when +/// the counts allow a verdict. +double? noiseShareFrom(NoiseShareSums s, + {required bool Function() lattice, int minSegments = 20}) { + final segs = s.segs; + if (segs < minSegments || s.ssd == 0 || s.wsd == 0) return null; + if (lattice()) return null; + // One loud stretch dominates the average: count segments by power, not number. + if (s.pSum * s.pSum / s.p2Sum < 0.75 * minSegments) return null; + final band = s.psd.sublist(kNoiseLoBin); + var peak = 0; + for (var k = 1; k < band.length; k++) { + if (band[k] > band[peak]) peak = k; + } + if (peak + kNoiseLoBin >= kNoiseHiBin - 1) return null; // ~0.477–0.5 band + if (band.last > 0.2 * band[peak]) return null; // hump on Nyquist + final med = median(band)!; + if (band[peak] < 4.5 * med) return null; // no line, no exemption + final floor = med / segs / _noiseHannSq; + final vetted = s.wsd / s.wsum, all = s.ssd / s.nd; + // 1 − (structured share of RMSSD²); equals 2σ²/mean(d²) when stationary. + return 1 - (vetted - 2 * floor) / math.max(vetted, all); +} + +/// True when every successive difference is a whole multiple of one step q +/// (the smallest |d| over 1e-6) and the mean squared difference [msd] is under +/// 2q². [magnitudes] are the |d| values (repeats change nothing), in any +/// order. See hrv_time.dart `_onCoarseLattice` for why. +bool coarseLattice(Iterable magnitudes, double msd) { + var q = double.infinity; + for (final a in magnitudes) { + if (a > 1e-6 && a < q) q = a; + } + if (q.isInfinite || msd >= 2 * q * q) return false; + for (final a in magnitudes) { + final k = a / q; + if ((k - k.roundToDouble()).abs() > 1e-3) return false; + } + return true; +} + +/// The breathing-line test's Hann window (128 beats). +final List hann128 = [ + for (var i = 0; i < 128; i++) 0.5 - 0.5 * math.cos(2 * math.pi * i / 127) +]; +final double _hann128Sq = hann128.fold(0.0, (a, w) => a + w * w); + +/// The cycles-per-beat axis the breathing-line test pools on: 0.15–0.5 +/// cycles/beat at the night's median beat. +final List breathingAxis = [for (var j = 38; j <= 128; j++) j / 256]; + +/// Welch power (128-beat Hann, 50 % overlap, averaged over segments) of one +/// window's RR, rebuilt from its difference runs, at cycles-per-beat +/// frequencies [f]. Null when no run holds a full segment. +List? windowPsd(List> diffRuns, List f) { + const n = 128; + final p = List.filled(f.length, 0.0); + var segs = 0; + for (final r in diffRuns) { + final x = List.filled(r.length + 1, 0.0); + for (var i = 0; i < r.length; i++) { + x[i + 1] = x[i] + r[i]; + } + for (var s = 0; s + n <= x.length; s += n ~/ 2) { + var m = 0.0; + for (var i = 0; i < n; i++) { + m += x[s + i]; + } + m /= n; + for (var j = 0; j < f.length; j++) { + final c = math.cos(2 * math.pi * f[j]), sn = math.sin(2 * math.pi * f[j]); + var cr = 1.0, ci = 0.0, re = 0.0, im = 0.0; + for (var i = 0; i < n; i++) { + final v = (x[s + i] - m) * hann128[i]; + re += v * cr; + im += v * ci; + final t = cr * c - ci * sn; + ci = cr * sn + ci * c; + cr = t; + } + p[j] += re * re + im * im; + } + segs++; + } + } + return segs == 0 ? null : [for (final v in p) v / segs]; +} + +/// The breathing-line test over [windows] 5-min windows, each read through +/// callbacks so a caller can hold its windows as runs or as sums: +/// [psdAt] is the window's [windowPsd] at the given frequencies (null without +/// a full segment), [diffs] its difference count, [addSquares] adds its +/// squared differences to a running total (in the caller's order), [meanRrMs] +/// its mean RR. [allSsd] / [allNd] are the squared differences and their count +/// over every window, and [lattice] is [coarseLattice] over every window's +/// differences at the mean square it is given. Returns the indices of the +/// windows whose own peak sits on the line, or null. See hrv_time.dart +/// `_steadyBreathingWindows` for the method. +List? steadyBreathingFrom({ + required int windows, + required List meanRrMs, + required List? Function(int w, List f) psdAt, + required int Function(int w) diffs, + required double Function(int w, double acc) addSquares, + required double allSsd, + required int allNd, + required bool Function(double msd) lattice, +}) { + if (allNd == 0 || lattice(allSsd / allNd)) return null; + final ref = median([for (final m in meanRrMs) if (m > 0) m]); + if (ref == null) return null; + final fc = breathingAxis; + final byBeat = List.filled(fc.length, 0.0); + final byHz = List.filled(fc.length, 0.0); + final cover = List.filled(fc.length, 0); + final peakHz = {}; // window -> its own peak, cycles per ms + final psd = >{}; // window -> its cycles-per-beat PSD + for (var w = 0; w < windows; w++) { + if (meanRrMs[w] <= 0) continue; + final pb = psdAt(w, fc); + if (pb == null) continue; + final norm = median(pb)!; + if (norm <= 0) continue; + final k = meanRrMs[w] / ref; + final fh = [for (final f in fc) f * k]; + final ph = psdAt(w, fh)!; + var best = -1; + for (var j = 0; j < fc.length; j++) { + byBeat[j] += pb[j] / norm; + if (fh[j] > 0.5) continue; + byHz[j] += ph[j] / norm; + cover[j]++; + if (best < 0 || ph[j] > ph[best]) best = j; + } + if (best >= 0) peakHz[w] = fc[best] / ref; + psd[w] = pb; + } + final nw = peakHz.length; + if (nw < 12) return null; + final bins = [for (var j = 0; j < fc.length; j++) if (cover[j] == nw) j]; + if (bins.length < 20) return null; + final hz = [for (final j in bins) byHz[j]]; + final bt = [for (final j in bins) byBeat[j]]; + var pk = 0; + for (var i = 1; i < hz.length; i++) { + if (hz[i] > hz[pk]) pk = i; + } + if (pk == 0 || pk == hz.length - 1) return null; + final f0 = fc[bins[pk]] / ref; + final brpm = f0 * 60000; + if (brpm < 8 || brpm > 30) return null; + // Same bins, same normalisation: the line must stand taller aligned in Hz + // than the beat pooling stands anywhere within ±12 bins of it. + var near = 0.0; + for (var i = math.max(0, pk - 12); i <= math.min(bt.length - 1, pk + 12); i++) { + near = math.max(near, bt[i]); + } + // The pooled noise ripple shrinks as 1/√windows, so the bar does too. + final promHz = hz[pk] / median(hz)!; + if (promHz < math.max(2.5, 1.5 + 7 / math.sqrt(nw))) return null; + if (promHz < 1.1 * near / median(bt)!) return null; + // Alternation lives at Nyquist on the beat axis, outside these bins. + if (byBeat.last / median(bt)! >= 0.5 * promHz) return null; + final keep = [ + for (final e in peakHz.entries) + if (math.log(e.value / f0).abs() <= 0.15) e.key + ]; + if (keep.length < 6) return null; + // Jitter that fails the gate would still be most of the number: same + // ceiling as the noise share, over the kept windows (see hrv_time.dart). + var lo = double.infinity, hi = 0.0; + for (final w in keep) { + lo = math.min(lo, peakHz[w]!); + hi = math.max(hi, peakHz[w]!); + } + var sq = 0.0, nAll = 0; + final acc = List.filled(fc.length, 0.0); + final wt = List.filled(fc.length, 0.0); + for (final w in keep) { + sq = addSquares(w, sq); + final n = diffs(w); + nAll += n; + final a = lo * meanRrMs[w] - 3 / 128, b = hi * meanRrMs[w] + 3 / 128; + for (var j = 0; j < fc.length; j++) { + if (fc[j] > a && fc[j] < b) continue; + acc[j] += n * psd[w]![j] / (2 - 2 * math.cos(2 * math.pi * fc[j])); + wt[j] += n; + } + } + var floor = double.infinity; + for (var j = 8; j < fc.length - 8; j++) { + var s = 0.0, m = 0; + for (var i = j - 8; i <= j + 8; i++) { + if (wt[i] < nAll / 2) break; + s += acc[i] / wt[i]; + m++; + } + if (m == 17) floor = math.min(floor, s / 17); + } + if (floor == double.infinity) return null; + // The lowest stretch of a noisy curve reads below its mean; 1.1 puts the + // floor back at these pooled sizes, so jitter alone is not undercounted. + final noise = 1.1 * 6 * floor / _hann128Sq * nAll; + return noise < kNnDiffNoiseShareCeiling * sq ? keep : null; +} diff --git a/lib/src/onehz/foundations/rr_correction.dart b/lib/src/onehz/foundations/rr_correction.dart index 8984013..41e6916 100644 --- a/lib/src/onehz/foundations/rr_correction.dart +++ b/lib/src/onehz/foundations/rr_correction.dart @@ -1,3 +1,11 @@ +// Lipponen & Tarvainen 2019 classification with Peltola 2012 correction policy. +// The unpublished artifact-run cap is an internal rule, ESTIMATE with low +// confidence: +// over-cap runs become gaps and each side is re-corrected. Without a cap split +// the result equals uncapped correction. Shared kernel arithmetic and raw seam +// context preserve capped batch NN/classes; settled stream plus provisional tail +// equals capped batch. RrSandwich preserves source-beat absolute timestamps; +// uncapped arbitrarily long artifact runs have no exactness guarantee. // FOUNDATION — RR artifact correction. // // Lipponen & Tarvainen 2019 ("A robust algorithm for heart rate variability @@ -13,8 +21,14 @@ // This is PRV. Output = cleaned NN series + per-beat artifact mask + clean // fraction. Multi-beat gaps are never silently bridged. -import 'dart:math' as math; -import '../util.dart'; +import 'rr_correction_kernel.dart'; + +/// Longest artifact run (beats) the corrector will carry. A longer run is not a +/// correction problem but an absence: it breaks the series into independent +/// segments and its beats are left out (a gap). 32 beats leaves no run on the +/// owner's 18 days of data over the cap (longest seen: 16) and bounds the +/// streaming state; see combined-bump/brief.md. +const int kMaxArtifactRun = 32; /// Artifact label for a single beat. enum BeatClass { normal, ectopic, longShort, missed, extra } @@ -34,6 +48,13 @@ class RrCorrectionResult { /// spliced out; pass the timestamps. final List nnTimesMs; + /// For each NN, the index of the input beat it came from: the beat itself, + /// or, for a spline-corrected beat, that beat's own index. Strictly + /// increasing; the input beats not listed here were dropped. Lets a caller + /// pair each NN with that beat's own (absolute) timestamp. Empty on a + /// hand-built result that did not track it. + final List nnSourceIndex; + /// Per-input-beat classification (same length as the input rr). final List classes; @@ -53,6 +74,7 @@ class RrCorrectionResult { required this.cleanFraction, required this.droppedCount, required this.correctedCount, + this.nnSourceIndex = const [], }); } @@ -74,6 +96,10 @@ class RrCorrectionResult { /// `rec_ts*1000` and therefore whole-second — and re-anchors to the real /// timestamp whenever the wall step exceeds the interval by more than /// [reanchorGapMs], i.e. at a dropout. +/// +/// [maxArtifactRun] caps a run of artifact beats: a longer run splits the series +/// (output = correctRr(before) ++ correctRr(after), times absolute, the run's +/// beats absent). null = uncapped, the pre-cap behaviour. RrCorrectionResult correctRr( List rrMs, { List? rrTsMs, @@ -81,7 +107,55 @@ RrCorrectionResult correctRr( int windowBeats = 91, double minThresholdMs = 100, double reanchorGapMs = 1000, + int? maxArtifactRun = kMaxArtifactRun, }) { + if (maxArtifactRun != null && maxArtifactRun < 1) { + throw ArgumentError.value(maxArtifactRun, 'maxArtifactRun'); + } + if (windowBeats < 1) throw ArgumentError.value(windowBeats, 'windowBeats'); + if (maxArtifactRun != null) { + // The overwhelmingly common path has no run over the cap. Reuse the + // original vectorized correction verbatim, including its clock, and only + // evaluate split windows when a run actually requires a cut. + var hardRun = 0, hardGap = false; + for (final rr in rrMs) { + hardRun = rr < 300 || rr > 2000 ? hardRun + 1 : 0; + if (hardRun > maxArtifactRun) { hardGap = true; break; } + } + if (!hardGap) { + final ordinary = correctRr(rrMs, rrTsMs: rrTsMs, alpha: alpha, + windowBeats: windowBeats, minThresholdMs: minThresholdMs, + reanchorGapMs: reanchorGapMs, maxArtifactRun: null); + var run = 0, gap = false; + for (final cls in ordinary.classes) { + run = cls == BeatClass.normal ? 0 : run + 1; + if (run > maxArtifactRun) { gap = true; break; } + } + if (!gap) return ordinary; + } + final w = RrCapWindow.evaluate(rrMs, + ts: rrTsMs, cap: maxArtifactRun, alpha: alpha, + windowBeats: windowBeats, minThresholdMs: minThresholdMs, + reanchorGapMs: reanchorGapMs); + final nn = [], times = [], source = []; + var normal = 0, dropped = 0, corrected = 0; + for (var i = 0; i < rrMs.length; i++) { + if (w.classes[i] == BeatClass.normal) normal++; + final v = w.values[i]; + if (v == null) { + dropped++; + } else { + nn.add(v); + times.add(w.times[i]); + source.add(i); + if (w.classes[i] != BeatClass.normal) corrected++; + } + } + return RrCorrectionResult(nn: nn, nnTimesMs: times, classes: w.classes, + cleanFraction: rrMs.isEmpty ? 0 : normal / rrMs.length, + droppedCount: dropped, correctedCount: corrected, + nnSourceIndex: source); + } final n = rrMs.length; // Beat-end time for EVERY input beat, kept or not. One array, built once, so // every branch below reads a clock instead of advancing its own. @@ -105,10 +179,12 @@ RrCorrectionResult correctRr( ]; final nn = []; final times = []; + final source = []; for (var i = 0; i < n; i++) { if (classes[i] == BeatClass.normal) { nn.add(rrMs[i]); times.add(tAll[i]); + source.add(i); } } return RrCorrectionResult( @@ -118,6 +194,7 @@ RrCorrectionResult correctRr( cleanFraction: nn.length / n, droppedCount: n - nn.length, correctedCount: 0, + nnSourceIndex: source, ); } @@ -133,32 +210,13 @@ RrCorrectionResult correctRr( // medRR: rr minus local median (for missed/extra long-range tests). final med = _slidingMedian(rrMs, windowBeats); - final mRR = List.generate(n, (i) { - final d = rrMs[i] - med[i]; - return d < 0 ? d * 2 : d; // paper asymmetry weight - }); + final mRR = List.generate(n, (i) => medianDeviation(rrMs[i], med[i])); final th2 = _slidingThreshold(mRR, windowBeats, alpha, minThresholdMs); - final classes = List.filled(n, BeatClass.normal); - for (var i = 0; i < n; i++) { - final hardLong = rrMs[i] > 2000; - final hardShort = rrMs[i] < 300; - final bigJump = dRR[i].abs() > th1[i]; - final bigDev = mRR[i].abs() > th2[i]; - if (hardLong || (bigDev && mRR[i] > 0)) { - // Long interval: likely a MISSED beat (interval ~ multiple of normal). - classes[i] = (med[i] > 0 && rrMs[i] > 1.5 * med[i]) - ? BeatClass.missed - : BeatClass.longShort; - } else if (hardShort || (bigDev && mRR[i] < 0)) { - // Short interval: likely an EXTRA (spurious) beat. - classes[i] = (med[i] > 0 && rrMs[i] < 0.6 * med[i]) - ? BeatClass.extra - : BeatClass.longShort; - } else if (bigJump) { - classes[i] = BeatClass.ectopic; - } - } + final classes = [ + for (var i = 0; i < n; i++) + classifyBeat(rrMs[i], dRR[i], th1[i], med[i], mRR[i], th2[i]) + ]; // Compensatory-pair reconciliation. A single ectopic/extra beat shows up as // TWO successive dRR spikes of opposite sign (the bad beat, then the @@ -169,17 +227,13 @@ RrCorrectionResult correctRr( // single isolated artifact rather than a spurious 2-beat run. for (var k = 1; k < n; k++) { if (classes[k] != BeatClass.ectopic) continue; - final prevBad = classes[k - 1] == BeatClass.extra || - classes[k - 1] == BeatClass.missed || - classes[k - 1] == BeatClass.ectopic || - classes[k - 1] == BeatClass.longShort; - if (!prevBad) continue; - final oppositeSign = dRR[k] * dRR[k - 1] < 0; - final valueNormal = med[k] > 0 && - rrMs[k] >= 300 && - rrMs[k] <= 2000 && - (rrMs[k] - med[k]).abs() <= 0.2 * med[k]; - if (oppositeSign && valueNormal) { + final prevBad = classes[k - 1] != BeatClass.normal; + if (isRecoveryBeat( + prevIsArtifact: prevBad, + rr: rrMs[k], + dRR: dRR[k], + dRRPrev: dRR[k - 1], + med: med[k])) { classes[k] = BeatClass.normal; } } @@ -189,6 +243,7 @@ RrCorrectionResult correctRr( final isArtifact = [for (final c in classes) c != BeatClass.normal]; final nn = []; final times = []; + final source = []; var dropped = 0; var corrected = 0; var i = 0; @@ -196,6 +251,7 @@ RrCorrectionResult correctRr( if (!isArtifact[i]) { nn.add(rrMs[i]); times.add(tAll[i]); + source.add(i); i++; continue; } @@ -216,6 +272,7 @@ RrCorrectionResult correctRr( // — 1.6 % of a night at one missed beat in sixty. nn.add(corr); times.add(tAll[i]); + source.add(i); corrected++; } else { dropped++; // no anchors -> honest drop; the clock already elapsed @@ -241,6 +298,7 @@ RrCorrectionResult correctRr( cleanFraction: normalCount / n, droppedCount: dropped, correctedCount: corrected, + nnSourceIndex: source, ); } @@ -306,42 +364,63 @@ List _slidingThreshold( final n = x.length; final out = List.filled(n, 0); final half = win ~/ 2; + final w = _SortedWindow(x, half); for (var i = 0; i < n; i++) { - final lo = math.max(0, i - half); - final hi = math.min(n - 1, i + half); - final seg = []; - for (var k = lo; k <= hi; k++) { - seg.add(x[k]); - } - final q1 = percentile(seg, 25) ?? 0; - final q3 = percentile(seg, 75) ?? 0; - final qd = (q3 - q1) / 2; - // Floor keeps a gross outlier detectable on (near-)quantized clean data - // where the QD genuinely collapses to 0 (constant RR). On any series with - // real beat-to-beat variability α·QD dominates the floor, so the floor - // never governs a physiological signal. - out[i] = math.max(alpha * qd, floor); + w.advanceTo(i); + out[i] = thresholdOfSorted(w.a, alpha, floor); } return out; } +/// Median of the window around each beat EXCLUDING the beat itself (falls back +/// to the beat when the window holds nothing else). Same value as sorting the +/// window-minus-self per beat, but the window is kept sorted as it slides and +/// the self element is skipped by index arithmetic instead of copied out. List _slidingMedian(List x, int win) { final n = x.length; final out = List.filled(n, 0); final half = win ~/ 2; + final w = _SortedWindow(x, half); for (var i = 0; i < n; i++) { - final lo = math.max(0, i - half); - final hi = math.min(n - 1, i + half); - final seg = []; - for (var k = lo; k <= hi; k++) { - if (k == i) continue; - seg.add(x[k]); - } - out[i] = median(seg) ?? x[i]; + w.advanceTo(i); + out[i] = medianExcluding(w.a, x[i]) ?? x[i]; } return out; } +/// Ascending-sorted copy of `x[max(0,i-half) .. min(n-1,i+half)]`, maintained +/// incrementally as [advanceTo] steps i forward by one (O(log w) search plus a +/// w-element memmove per beat instead of an O(w log w) sort). +class _SortedWindow { + final List _x; + final int _half; + final _set = SortedMultiset(); + List get a => _set.a; + int _lo = 0; + int _hi = -1; // inclusive; -1 = empty + + _SortedWindow(this._x, this._half); + + void advanceTo(int i) { + final n = _x.length; + final newLo = i - _half < 0 ? 0 : i - _half; + final newHi = i + _half > n - 1 ? n - 1 : i + _half; + if (_hi < _lo) { + // First call: build once. + _set.resetFrom(_x.getRange(newLo, newHi + 1)); + } else { + while (_lo < newLo) { + _set.remove(_x[_lo++]); + } + while (_hi < newHi) { + _set.add(_x[++_hi]); + } + } + _lo = newLo; + _hi = newHi; + } +} + /// Catmull-Rom cubic interpolation at the artifact index using the nearest two /// NORMAL beats on each side. Returns null if anchors are unavailable. double? _splineCorrect(List rr, List isArtifact, int idx) { @@ -353,19 +432,5 @@ double? _splineCorrect(List rr, List isArtifact, int idx) { for (var k = idx + 1; k < rr.length && right.length < 2; k++) { if (!isArtifact[k]) right.add(rr[k]); } - if (left.isEmpty || right.isEmpty) return null; - final p1 = left.last; - final p2 = right.first; - final p0 = left.length >= 2 ? left.first : p1; - final p3 = right.length >= 2 ? right.last : p2; - // Catmull-Rom at t=0.5 between p1 and p2. - const t = 0.5; - final t2 = t * t; - final t3 = t2 * t; - final v = 0.5 * - ((2 * p1) + - (-p0 + p2) * t + - (2 * p0 - 5 * p1 + 4 * p2 - p3) * t2 + - (-p0 + 3 * p1 - 3 * p2 + p3) * t3); - return v; + return splineMid(left, right); } diff --git a/lib/src/onehz/foundations/rr_correction_kernel.dart b/lib/src/onehz/foundations/rr_correction_kernel.dart new file mode 100644 index 0000000..8acb72b --- /dev/null +++ b/lib/src/onehz/foundations/rr_correction_kernel.dart @@ -0,0 +1,373 @@ +// Lipponen & Tarvainen 2019 classification with Peltola 2012 correction policy. +// The unpublished artifact-run cap is an internal rule, ESTIMATE with low +// confidence: +// over-cap runs become gaps and each side is re-corrected. Without a cap split +// the result equals uncapped correction. Shared kernel arithmetic and raw seam +// context preserve capped batch NN/classes; settled stream plus provisional tail +// equals capped batch. RrSandwich preserves source-beat absolute timestamps; +// uncapped arbitrarily long artifact runs have no exactness guarantee. +// FOUNDATION — the per-beat pieces of Lipponen–Tarvainen correction that both +// `correctRr` (whole series) and `RrCorrector` (streaming) run. +// +// One copy of the sorted window, the threshold, the leave-one-out median, the +// class decision, the compensatory-pair rule and the spline, so the batch and +// the stream cannot drift apart: the stream's contract is bit-identity with +// `correctRr`, and sharing the arithmetic is what keeps that cheap to hold. +// +// Not exported from `onehz.dart` — internal to foundations/. + +import 'dart:math' as math; +import '../util.dart'; +import 'rr_correction.dart' show BeatClass; + +/// First index whose element is not less than [v] under `compareTo` — the same +/// total order `List.sort()` uses (-0.0 < 0.0, NaN last), so a window +/// kept in this order matches the one a per-beat sort produced. +int lowerBound(List a, double v) { + var lo = 0; + var hi = a.length; + while (lo < hi) { + final m = (lo + hi) >> 1; + if (a[m].compareTo(v) < 0) { + lo = m + 1; + } else { + hi = m; + } + } + return lo; +} + +/// An ascending multiset of doubles: O(log w) search plus a w-element memmove +/// per add / remove, instead of an O(w log w) sort per beat. +class SortedMultiset { + /// The values, ascending. Read-only to callers. + final List a = []; + + void add(double v) => a.insert(lowerBound(a, v), v); + + /// Removes one occurrence of [v], which must be present. + void remove(double v) => a.removeAt(lowerBound(a, v)); + + void resetFrom(Iterable xs) { + a + ..clear() + ..addAll(xs) + ..sort(); + } +} + +/// `alpha × QD` of an ascending window, floored: QD = (Q3 − Q1)/2. +/// +/// Floor keeps a gross outlier detectable on (near-)quantized clean data where +/// the QD genuinely collapses to 0 (constant RR). On any series with real +/// beat-to-beat variability α·QD dominates the floor. +double thresholdOfSorted(List sorted, double alpha, double floor) { + final q1 = percentileSorted(sorted, 25) ?? 0; + final q3 = percentileSorted(sorted, 75) ?? 0; + final qd = (q3 - q1) / 2; + return math.max(alpha * qd, floor); +} + +/// `median(sorted minus one occurrence of v)` — [percentile]'s linear +/// interpolation at p = 50 over the sorted list with v's slot skipped, in the +/// same operation order so the result is bit-identical. Null if nothing is left. +double? medianExcluding(List sorted, double v) { + final len = sorted.length - 1; + if (len <= 0) return null; + final skip = lowerBound(sorted, v); + double at(int j) => j < skip ? sorted[j] : sorted[j + 1]; + if (len == 1) return at(0); + final rank = (50.0 / 100) * (len - 1); + final lo = rank.floor(); + final hi = rank.ceil(); + if (lo == hi) return at(lo); + final frac = rank - lo; + return at(lo) + (at(hi) - at(lo)) * frac; +} + +/// The paper's asymmetric deviation from the local median: below-median +/// deviations count double. +double medianDeviation(double rr, double med) { + final d = rr - med; + return d < 0 ? d * 2 : d; +} + +/// First-pass class of one beat from its dRR, mRR and the two thresholds. +BeatClass classifyBeat( + double rr, double dRR, double th1, double med, double mRR, double th2) { + final hardLong = rr > 2000; + final hardShort = rr < 300; + final bigJump = dRR.abs() > th1; + final bigDev = mRR.abs() > th2; + if (hardLong || (bigDev && mRR > 0)) { + // Long interval: likely a MISSED beat (interval ~ multiple of normal). + return (med > 0 && rr > 1.5 * med) ? BeatClass.missed : BeatClass.longShort; + } else if (hardShort || (bigDev && mRR < 0)) { + // Short interval: likely an EXTRA (spurious) beat. + return (med > 0 && rr < 0.6 * med) ? BeatClass.extra : BeatClass.longShort; + } else if (bigJump) { + return BeatClass.ectopic; + } + return BeatClass.normal; +} + +/// Compensatory-pair rule: beat k was flagged ONLY by its dRR jump, the beat +/// before it was an artifact, the two dRR spikes have opposite sign and k's own +/// value is normal and close to the local median — k is just the recovery from +/// the previous beat's event, so it is demoted to normal. +bool isRecoveryBeat({ + required bool prevIsArtifact, + required double rr, + required double dRR, + required double dRRPrev, + required double med, +}) { + if (!prevIsArtifact) return false; + final oppositeSign = dRR * dRRPrev < 0; + final valueNormal = + med > 0 && rr >= 300 && rr <= 2000 && (rr - med).abs() <= 0.2 * med; + return oppositeSign && valueNormal; +} + +/// Catmull-Rom at t = 0.5 between the nearest NORMAL beats either side of an +/// isolated artifact. [left] / [right] hold up to two values each, nearest to +/// the artifact LAST in [left] and FIRST in [right]. Null without an anchor on +/// both sides. +double? splineMid(List left, List right) { + if (left.isEmpty || right.isEmpty) return null; + final p1 = left.last; + final p2 = right.first; + final p0 = left.length >= 2 ? left.first : p1; + final p3 = right.length >= 2 ? right.last : p2; + const t = 0.5; + final t2 = t * t; + final t3 = t2 * t; + return 0.5 * + ((2 * p1) + + (-p0 + p2) * t + + (2 * p0 - 5 * p1 + 4 * p2 - p3) * t2 + + (-p0 + 3 * p1 - 3 * p2 + p3) * t3); +} + +/// Internal bounded-window evaluator shared by the capped batch and stream. +/// [from] has its complete left window in [rr], or is the segment's first beat. +/// Only classes at and after [from] are evaluated; the preceding class and +/// normal anchors are carried explicitly, so truncating old context cannot +/// change recovery or spline decisions. +class RrCapWindow { + final List classes; + final List values; + final List times; + final List gaps; + final List hardGaps; + RrCapWindow._(int n) + : classes = List.filled(n, BeatClass.normal), + values = List.filled(n, null), + times = List.filled(n, 0), + gaps = List.filled(n, false), + hardGaps = List.filled(n, false); + + static RrCapWindow evaluate( + List rr, { + List? ts, + required int cap, + double alpha = 5.2, + int windowBeats = 91, + double minThresholdMs = 100, + double reanchorGapMs = 1000, + int from = 0, + BeatClass previous = BeatClass.normal, + List leftNormals = const [], + bool leadingGap = false, + bool leadingHardGap = false, + int segmentStart = 0, + double clock = 0, + double? origin, + }) { + final out = RrCapWindow._(rr.length); + if (rr.isEmpty) return out; + final wall = ts != null && ts.length == rr.length; + final base = origin ?? (wall ? ts[0] - rr[0] : 0.0); + final h = windowBeats ~/ 2; + + List classify(int a, int b, int start, BeatClass prev) { + final n = b - a; + if (n < 3) { + return [ + for (var i = start; i < b; i++) + rr[i] >= 300 && rr[i] <= 2000 + ? BeatClass.normal + : BeatClass.longShort + ]; + } + final med = List.filled(n, 0); + final mrr = List.filled(n, 0); + final th1 = List.filled(n, 0); + final wd = SortedMultiset(), wr = SortedMultiset(); + var lo = a, hi = a; + double d(int i) => i == a ? 0 : rr[i] - rr[i - 1]; + for (var i = a; i < b; i++) { + final nextLo = math.max(a, i - h), nextHi = math.min(b, i + h + 1); + while (hi < nextHi) { + wd.add(d(hi)); + wr.add(rr[hi++]); + } + while (lo < nextLo) { + wd.remove(d(lo)); + wr.remove(rr[lo++]); + } + med[i - a] = medianExcluding(wr.a, rr[i]) ?? rr[i]; + mrr[i - a] = medianDeviation(rr[i], med[i - a]); + th1[i - a] = thresholdOfSorted(wd.a, alpha, minThresholdMs); + } + final wm = SortedMultiset(); + lo = a; + hi = a; + final cl = []; + for (var i = start; i < b; i++) { + final nextLo = math.max(a, i - h), nextHi = math.min(b, i + h + 1); + if (i == start) lo = nextLo; + if (i == start) hi = nextLo; + while (hi < nextHi) wm.add(mrr[hi++ - a]); + while (lo < nextLo) wm.remove(mrr[lo++ - a]); + var c = classifyBeat(rr[i], d(i), th1[i - a], med[i - a], mrr[i - a], + thresholdOfSorted(wm.a, alpha, minThresholdMs)); + if (i > a && + c == BeatClass.ectopic && + isRecoveryBeat( + prevIsArtifact: prev != BeatClass.normal, + rr: rr[i], + dRR: d(i), + dRRPrev: d(i - 1), + med: med[i - a])) { + c = BeatClass.normal; + } + cl.add(c); + prev = c; + } + return cl; + } + + // A stack avoids recursion depth proportional to the number of gaps. + final pending = <(int, int, int, BeatClass, List, bool, bool)>[ + ( + segmentStart, + rr.length, + from, + previous, + leftNormals, + leadingGap, + leadingHardGap + ) + ]; + while (pending.isNotEmpty) { + final (a, b, start, prev, anchors, inGap, hardLeading) = + pending.removeLast(); + if (start >= b) continue; + if (inGap && hardLeading && rr[start] >= 300 && rr[start] <= 2000) { + pending + .add((start, b, start, BeatClass.normal, const [], false, false)); + continue; + } + var hardCut = false; + var cutA = -1, cutB = -1; + // Physiologically impossible intervals are artifacts independently of + // every dispersion window. Cut these first: otherwise their medians can + // absorb valid beats on either side into the detected run itself. + for (var i = start; i < b;) { + if (rr[i] >= 300 && rr[i] <= 2000) { + i++; + continue; + } + var j = i + 1; + while (j < b && (rr[j] < 300 || rr[j] > 2000)) { + j++; + } + if (j - i > cap || (inGap && i == start)) { + cutA = i; + cutB = j; + hardCut = true; + break; + } + i = j; + } + final cl = cutA >= 0 ? [] : classify(a, b, start, prev); + if (cutA < 0 && inGap && cl.first == BeatClass.normal) { + pending + .add((start, b, start, BeatClass.normal, const [], false, false)); + continue; + } + if (cutA < 0) { + for (var i = start; i < b;) { + if (cl[i - start] == BeatClass.normal) { + i++; + continue; + } + var j = i + 1; + while (j < b && cl[j - start] != BeatClass.normal) { + j++; + } + if (j - i > cap || (inGap && i == start)) { + cutA = i; + cutB = j; + break; + } + i = j; + } + } + if (cutA >= 0) { + for (var i = cutA; i < cutB; i++) { + out.classes[i] = BeatClass.longShort; + out.gaps[i] = true; + out.hardGaps[i] = hardCut || (inGap && hardLeading && cutA == start); + } + // Cut BEFORE classifying either side, including their dispersion + // windows. The gap's labels are canonical, never provisional medians. + pending.add((cutB, b, cutB, BeatClass.normal, const [], false, false)); + pending.add((a, cutA, start, prev, anchors, inGap, hardLeading)); + continue; + } + for (var i = start; i < b; i++) { + out.classes[i] = cl[i - start]; + } + final normals = List.of(anchors); + for (var i = start; i < b;) { + if (out.classes[i] == BeatClass.normal) { + out.values[i] = rr[i]; + normals.add(rr[i]); + if (normals.length > 2) normals.removeAt(0); + i++; + continue; + } + var j = i + 1; + while (j < b && out.classes[j] != BeatClass.normal) { + j++; + } + if (j == i + 1) { + final right = []; + for (var k = j; k < b && right.length < 2; k++) { + if (out.classes[k] == BeatClass.normal) right.add(rr[k]); + } + out.values[i] = splineMid(normals, right); + } + i = j; + } + } + var cur = clock; + var gap = leadingGap; + for (var i = from; i < rr.length; i++) { + final reset = gap && !out.gaps[i]; + final dropout = + wall && i > 0 && ts[i] - ts[i - 1] - rr[i] > reanchorGapMs; + final anchored = wall ? ts[i] - base : 0.0; + cur = wall && reset + ? anchored + : dropout && anchored > cur + ? anchored + : cur + rr[i]; + out.times[i] = cur; + gap = out.gaps[i]; + } + return out; + } +} diff --git a/lib/src/onehz/foundations/rr_correction_stream.dart b/lib/src/onehz/foundations/rr_correction_stream.dart new file mode 100644 index 0000000..1e034eb --- /dev/null +++ b/lib/src/onehz/foundations/rr_correction_stream.dart @@ -0,0 +1,822 @@ +// Lipponen & Tarvainen 2019 classification with Peltola 2012 correction policy. +// The unpublished artifact-run cap is an internal rule, ESTIMATE with low +// confidence: +// over-cap runs become gaps and each side is re-corrected. Without a cap split +// the result equals uncapped correction. Shared kernel arithmetic and raw seam +// context preserve capped batch NN/classes; settled stream plus provisional tail +// equals capped batch. RrSandwich preserves source-beat absolute timestamps; +// uncapped arbitrarily long artifact runs have no exactness guarantee. +// FOUNDATION — streaming RR artifact correction. +// +// [RrCorrector] is `correctRr` (rr_correction.dart) turned into a fold: feed +// beats in chunks, get back the output that became FINAL with each chunk, and +// ask for the provisional tail on demand. At every prefix of the input, +// +// (all settled output so far) ++ (snapshot().tail*) == correctRr(prefix) +// +// bit for bit: nn, nnTimesMs, per-beat classes, counts, cleanFraction. Nothing +// is imputed here either: a beat whose neighbours are not there yet is not +// guessed at, it simply is not settled. +// +// Why a beat can be final long before the series ends: its class depends only +// on a centred window of `windowBeats ~/ 2` beats each side (twice that for +// the second threshold), one carried class, and — for the spline — the two +// nearest normal beats either side. Beats closer to the end of the data than +// that are PROVISIONAL, because `correctRr` clips its windows at the end. +// +// Derivation, with H = windowBeats ~/ 2 (45 at the default 91): +// +// beat clock t[i] prefix recurrence (current time, previous timestamp) +// dRR[i] rr[i] - rr[i-1] local +// th1, med QD of dRR / median of rr, over [i-H, i+H] needs rr up to i+H +// mRR from rr[i] and med[i] needs rr up to i+H +// th2 QD of mRR over [i-H, i+H] needs rr up to i+2H +// class first pass from the above, then the compensatory-pair rule, +// which reads only the previous FINAL class: one carried class. +// output an isolated artifact is spline-corrected from the two nearest +// normal beats each side (left pair carried, right pair needs +// classes up to the second normal after it); a run is dropped. +// +// So output settles 2H beats + (artifact run length) + 2 normals behind the +// newest beat. Beats nearer the end than that are provisional because +// `correctRr` clips its windows at the end of the data; [RrCorrector.snapshot] +// re-evaluates just that tail with the same clipping. +// +// With a cap, a future split also changes the windows to its left. The capped +// path keeps 4H + 2K + 4 beats provisional (248 at the defaults), then up to K +// more to avoid committing part of a short artifact run. Uncapped mode retains +// the original stage-by-stage latency described above. +// +// The arithmetic (window, threshold, class, spline) is shared with `correctRr` +// through rr_correction_kernel.dart. + +import 'dart:math' as math; + +import 'rr_correction.dart'; +import 'rr_correction_kernel.dart'; + +/// What [RrCorrector.fold] hands back: the output that became FINAL with that +/// fold, in order, ready to append to what the caller already holds. +class RrSettled { + /// Newly final cleaned NN intervals (ms), as `correctRr(...).nn` would hold. + final List nn; + + /// Beat times (ms) for [nn], as `correctRr(...).nnTimesMs` would hold. + final List nnTimes; + + /// Classes of the beats whose class became final with this fold, i.e. input + /// beats `[classifiedBeats before the fold, classifiedBeats after)`. A class + /// is final earlier than its output: a lone artifact waits for two normal + /// beats on its right before it can be spline-corrected. + final List classes; + + const RrSettled({ + required this.nn, + required this.nnTimes, + required this.classes, + }); +} + +/// The part of `correctRr(all beats so far)` that can still change, plus the +/// running totals. Recomputed on every [RrCorrector.snapshot]; never stored. +class RrSnapshot { + /// Beats folded so far. + final int n; + + /// Provisional NN / times that follow everything [RrCorrector.fold] has + /// settled so far. + final List tailNn; + final List tailNnTimes; + + /// Provisional classes of the beats `[classifiedBeats, n)`. + final List tailClasses; + + /// Totals as `correctRr(all beats so far)` reports them. + final int normalCount, droppedCount, correctedCount; + + /// Beats whose output is final / whose class is final. + final int settledBeats, classifiedBeats; + + const RrSnapshot({ + required this.n, + required this.tailNn, + required this.tailNnTimes, + required this.tailClasses, + required this.normalCount, + required this.droppedCount, + required this.correctedCount, + required this.settledBeats, + required this.classifiedBeats, + }); + + /// `correctRr`'s `cleanFraction`: 0 on an empty series, never NaN. + double get cleanFraction => n == 0 ? 0 : normalCount / n; +} + +/// Streaming Lipponen–Tarvainen correction. Same parameters as [correctRr]. +/// +/// Append-only: it cannot take beats back. Anything that changes data behind +/// the settled edge means starting a new corrector. +class RrCorrector { + final double alpha; + final int windowBeats; + final double minThresholdMs; + final double reanchorGapMs; + + /// Longest artifact run carried; a longer one is a gap. null = uncapped. + /// Same meaning as [correctRr]'s `maxArtifactRun`. + final int? maxArtifactRun; + final int _h; + final _CappedState? _capState; + + // ---- persisted state ---- + int _n = 0; // beats folded + bool? _wall; // timestamps given for this stream (null until the first beat) + double _origin = 0, _cur = 0, _prevTs = 0, _prevRr = 0; + int _off = 0; // global index of _rr[0] + final List _rr = [], _t = [], _d = []; + int _c1 = 0; // beats whose med / mRR / th1 are final + int _c2 = 0; // beats whose class is final + int _ce = 0; // beats whose output is final (handed out) + final List _fcls = []; // final classes for [_ce, _c2) + int _lastFinal = 0; // class index of beat _c2 - 1 + int _normalFinal = 0; + int _dropped = 0, _corrected = 0; + final List _lastNormals = []; // <= 2, oldest first + + // ---- derived, rebuilt on restore ---- + int _so = 0; // global index of _med[0] + final List _med = [], _mrr = [], _th1 = []; + final _wd = SortedMultiset(), _wr = SortedMultiset(), _wm = SortedMultiset(); + int _wPos = -1, _w2Pos = -1; + + RrCorrector({ + this.alpha = 5.2, + this.windowBeats = 91, + this.minThresholdMs = 100, + this.reanchorGapMs = 1000, + this.maxArtifactRun = kMaxArtifactRun, + }) : _h = windowBeats ~/ 2, + _capState = maxArtifactRun == null ? null : _CappedState() { + if (windowBeats < 1) throw ArgumentError.value(windowBeats, 'windowBeats'); + if (maxArtifactRun != null && maxArtifactRun! < 1) { + throw ArgumentError.value(maxArtifactRun, 'maxArtifactRun'); + } + } + + /// Restores a corrector from [toJson]. Throws [FormatException] on a + /// checkpoint of another type or version, or one that is malformed. + factory RrCorrector.fromJson(Map json) { + try { + return _restore(json); + } on FormatException { + rethrow; + } catch (e) { + throw FormatException('malformed RrCorrector checkpoint: $e'); + } + } + + static RrCorrector _restore(Map j) { + if (j['type'] != 'RrCorrector') { + throw FormatException('not an RrCorrector checkpoint: ${j['type']}'); + } + if (j['version'] != 2) { + throw FormatException('unsupported RrCorrector version ${j['version']}'); + } + List doubles(String k) => + [for (final x in j[k] as List) (x as num).toDouble()]; + final c = RrCorrector( + maxArtifactRun: j['maxArtifactRun'] as int?, + alpha: (j['alpha'] as num).toDouble(), + windowBeats: j['windowBeats'] as int, + minThresholdMs: (j['minThresholdMs'] as num).toDouble(), + reanchorGapMs: (j['reanchorGapMs'] as num).toDouble(), + ); + if (!j.containsKey('maxArtifactRun')) { + throw const FormatException('missing artifact cap'); + } + if (c._capState != null) { + c._capState.restore(j['capped'] as Map, c._h); + return c; + } + c._n = j['n'] as int; + c._wall = j['wall'] as bool?; + c._origin = (j['origin'] as num).toDouble(); + c._cur = (j['cur'] as num).toDouble(); + c._prevTs = (j['prevTs'] as num).toDouble(); + c._prevRr = (j['prevRr'] as num).toDouble(); + c._off = j['off'] as int; + c._rr.addAll(doubles('rr')); + c._t.addAll(doubles('t')); + c._d.addAll(doubles('d')); + c._c1 = j['c1'] as int; + c._c2 = j['c2'] as int; + c._ce = j['ce'] as int; + c._fcls.addAll((j['fcls'] as List).cast()); + c._lastFinal = j['lastFinal'] as int; + c._normalFinal = j['normalFinal'] as int; + c._dropped = j['dropped'] as int; + c._corrected = j['corrected'] as int; + c._lastNormals.addAll(doubles('lastNormals')); + if (c._n > 0 && c._wall == null) { + throw const FormatException('invalid wall'); + } + if (c._dropped < 0 || c._dropped > c._ce) { + throw const FormatException('invalid dropped'); + } + if (c._corrected < 0 || c._corrected > c._ce - c._dropped) { + throw const FormatException('invalid corrected'); + } + if (c._normalFinal < 0 || c._normalFinal > c._c2) { + throw const FormatException('invalid normalFinal'); + } + final buffered = c._n - c._off; + final ok = c._off >= 0 && + buffered >= 0 && + c._rr.length == buffered && + c._t.length == buffered && + c._d.length == buffered && + c._ce >= c._off && + c._ce <= c._c2 && + c._c2 <= c._c1 && + c._c1 <= c._n && + c._fcls.length == c._c2 - c._ce && + c._fcls.every((k) => k >= 0 && k < BeatClass.values.length) && + c._lastFinal >= 0 && + c._lastFinal < BeatClass.values.length && + c._lastNormals.length <= 2 && + c._off <= math.max(0, c._c2 - 2 * c._h - 1); + if (!ok) throw const FormatException('inconsistent RrCorrector checkpoint'); + c._rebuildStageArrays(); + return c; + } + + /// Checkpoint: `{'version': 2, 'type': 'RrCorrector', ...}`, plain JSON + /// (survives `jsonEncode`/`jsonDecode` bit-exactly), and bounded — it holds + /// the unsettled window, not the series. + Map toJson() { + if (_capState != null) { + return { + 'version': 2, 'type': 'RrCorrector', + 'maxArtifactRun': maxArtifactRun, 'alpha': alpha, + 'windowBeats': windowBeats, 'minThresholdMs': minThresholdMs, + 'reanchorGapMs': reanchorGapMs, 'capped': _capState.toJson(), + }; + } + return { + 'version': 2, + 'maxArtifactRun': maxArtifactRun, + 'type': 'RrCorrector', + 'alpha': alpha, + 'windowBeats': windowBeats, + 'minThresholdMs': minThresholdMs, + 'reanchorGapMs': reanchorGapMs, + 'n': _n, + 'wall': _wall, + 'origin': _origin, + 'cur': _cur, + 'prevTs': _prevTs, + 'prevRr': _prevRr, + 'off': _off, + 'rr': _rr, + 't': _t, + 'd': _d, + 'c1': _c1, + 'c2': _c2, + 'ce': _ce, + 'fcls': _fcls, + 'lastFinal': _lastFinal, + 'normalFinal': _normalFinal, + 'dropped': _dropped, + 'corrected': _corrected, + 'lastNormals': _lastNormals, + }; + } + + /// Beats whose output is final. + int get settledBeats => _capState?.edge ?? _ce; + + /// Appends beats. [tsMs] (same length as [rrMs]) is either given on EVERY + /// fold or on none; mixing throws [StateError]. A non-finite RR throws + /// [ArgumentError] and leaves the corrector unchanged. An empty chunk is a + /// no-op. + RrSettled fold(List rrMs, {List? tsMs}) { + if (_capState != null) return _capState.fold(this, rrMs, tsMs); + if (rrMs.isEmpty) { + return const RrSettled(nn: [], nnTimes: [], classes: []); + } + // Everything that can throw happens before the first mutation. + final wall = tsMs != null; + if (wall && tsMs.length != rrMs.length) { + throw ArgumentError('tsMs has ${tsMs.length} entries for ' + '${rrMs.length} beats'); + } + if (_wall != null && _wall != wall) { + throw StateError('timestamps must be given on every fold or on none'); + } + for (var k = 0; k < rrMs.length; k++) { + if (!rrMs[k].isFinite) throw ArgumentError('non-finite RR at index $k'); + if (wall && !tsMs[k].isFinite) { + throw ArgumentError('non-finite timestamp at index $k'); + } + } + _wall = wall; + for (var k = 0; k < rrMs.length; k++) { + final r = rrMs[k]; + final ts = wall ? tsMs[k] : 0.0; + if (_n == 0) { + _origin = wall ? ts - r : 0.0; + _cur = 0.0; + } + // Same clock as `correctRr`'s beat times: cumsum inside a contiguous run, + // re-anchored to the real timestamp at a dropout, never backwards. + final dropout = wall && _n > 0 && (ts - _prevTs) - r > reanchorGapMs; + final anchored = wall ? ts - _origin : 0.0; + _cur = (dropout && anchored > _cur) ? anchored : _cur + r; + if (wall) _prevTs = ts; + _rr.add(r); + _t.add(_cur); + _d.add(_n == 0 ? 0.0 : r - _prevRr); + _prevRr = r; + _n++; + } + final nn = [], times = []; + final classes = []; + _advance(classes); + _emitCommitted(nn, times); + _compact(); + return RrSettled(nn: nn, nnTimes: times, classes: classes); + } + + double _rrAt(int g) => _rr[g - _off]; + double _dAt(int g) => _d[g - _off]; + double _tAt(int g) => _t[g - _off]; + + void _advance(List classes) { + // Under 3 beats `correctRr` takes its short branch; nothing settles yet. + if (_n < 3) return; + while (_c1 + _h <= _n - 1) { + _stage1(_c1); + _c1++; + } + while (_c2 + 2 * _h <= _n - 1) { + classes.add(_stage2(_c2)); + _c2++; + } + } + + // th1 / med / mRR for beat g, whose window [max(0, g-h), g+h] is fully inside + // the data. + void _stage1(int g) { + final h = _h; + if (_wPos == g - 1 && _wPos >= 0) { + _wd.add(_dAt(g + h)); + _wr.add(_rrAt(g + h)); + if (g - 1 - h >= 0) { + _wd.remove(_dAt(g - 1 - h)); + _wr.remove(_rrAt(g - 1 - h)); + } + } else { + final lo = math.max(0, g - h); + _wd.resetFrom([for (var k = lo; k <= g + h; k++) _dAt(k)]); + _wr.resetFrom([for (var k = lo; k <= g + h; k++) _rrAt(k)]); + } + _wPos = g; + final m = medianExcluding(_wr.a, _rrAt(g)) ?? _rrAt(g); + _th1.add(thresholdOfSorted(_wd.a, alpha, minThresholdMs)); + _med.add(m); + _mrr.add(medianDeviation(_rrAt(g), m)); + } + + BeatClass _stage2(int g) { + final h = _h; + if (_w2Pos == g - 1 && _w2Pos >= 0) { + _wm.add(_mrr[g + h - _so]); + if (g - 1 - h >= 0) _wm.remove(_mrr[g - 1 - h - _so]); + } else { + final lo = math.max(0, g - h); + _wm.resetFrom([for (var k = lo; k <= g + h; k++) _mrr[k - _so]]); + } + _w2Pos = g; + final th2 = thresholdOfSorted(_wm.a, alpha, minThresholdMs); + final cls = _finalClass(g, _th1[g - _so], _med[g - _so], _mrr[g - _so], + th2, _lastFinal); + _fcls.add(cls.index); + _lastFinal = cls.index; + if (cls == BeatClass.normal) _normalFinal++; + return cls; + } + + /// Class of beat g given the class of g-1: first pass, then the + /// compensatory-pair rule. + BeatClass _finalClass(int g, double th1, double med, double mrr, double th2, + int prevClass) { + final rr = _rrAt(g), d = _dAt(g); + final cls = classifyBeat(rr, d, th1, med, mrr, th2); + if (g < 1 || cls != BeatClass.ectopic) return cls; + return isRecoveryBeat( + prevIsArtifact: prevClass != BeatClass.normal.index, + rr: rr, + dRR: d, + dRRPrev: _dAt(g - 1), + med: med) + ? BeatClass.normal + : cls; + } + + // Output over classes [ce, limit). Shared by the committed pass (limit = c2, + // isEnd false: an artifact whose fate is not decidable yet waits) and the + // provisional pass (limit = n, isEnd true: the end of the data resolves it + // exactly the way `correctRr` does). Returns the new ce. + int _emit({ + required int ce, + required int limit, + required bool isEnd, + required int Function(int g) clsAt, + required List lastNormals, + required void Function(double nn, double t) put, + required void Function(int runLen) drop, + required void Function() corrected, + }) { + const normal = 0; + while (ce < limit) { + if (clsAt(ce) == normal) { + put(_rrAt(ce), _tAt(ce)); + lastNormals.add(_rrAt(ce)); + if (lastNormals.length > 2) lastNormals.removeAt(0); + ce++; + continue; + } + var j = ce; + while (j < limit && clsAt(j) != normal) { + j++; + } + if (j == limit && !isEnd) break; + final runLen = j - ce; + if (runLen == 1) { + final right = []; + var k = j; + while (k < limit && right.length < 2) { + if (clsAt(k) == normal) right.add(_rrAt(k)); + k++; + } + if (right.length < 2 && k == limit && !isEnd) break; + final corr = splineMid(lastNormals, right); + if (corr != null) { + put(corr, _tAt(ce)); + corrected(); + } else { + drop(1); // no anchors: an honest drop; the clock already elapsed + } + } else { + drop(runLen); + } + ce = j; + } + return ce; + } + + void _emitCommitted(List nn, List times) { + final fo = _ce; + _ce = _emit( + ce: _ce, + limit: _c2, + isEnd: false, + clsAt: (g) => _fcls[g - fo], + lastNormals: _lastNormals, + put: (v, t) { + nn.add(v); + times.add(t); + }, + drop: (r) => _dropped += r, + corrected: () => _corrected++, + ); + final used = _ce - fo; + if (used > 0) _fcls.removeRange(0, used); + } + + /// Provisional tail and totals; does not change the corrector. + RrSnapshot snapshot() { + if (_capState != null) return _capState.snapshot(this); + final n = _n; + if (n == 0) { + return const RrSnapshot( + n: 0, + tailNn: [], + tailNnTimes: [], + tailClasses: [], + normalCount: 0, + droppedCount: 0, + correctedCount: 0, + settledBeats: 0, + classifiedBeats: 0); + } + if (n < 3) { + // `correctRr`'s short branch: plausible beats pass, the rest are + // long/short. Nothing is settled below 3 beats. + final nn = [], times = []; + final cl = []; + for (var g = 0; g < n; g++) { + final ok = _rrAt(g) >= 300 && _rrAt(g) <= 2000; + cl.add(ok ? BeatClass.normal : BeatClass.longShort); + if (ok) { + nn.add(_rrAt(g)); + times.add(_tAt(g)); + } + } + return RrSnapshot( + n: n, + tailNn: nn, + tailNnTimes: times, + tailClasses: cl, + normalCount: nn.length, + droppedCount: n - nn.length, + correctedCount: 0, + settledBeats: 0, + classifiedBeats: 0); + } + final h = _h; + final end = n - 1; + // Provisional stage 1 for beats [c1, n): windows clipped at the end. + final tMed = [], tMrr = [], tTh1 = []; + double mrrAt(int k) => k < _c1 ? _mrr[k - _so] : tMrr[k - _c1]; + for (var k = _c1; k < n; k++) { + final lo = math.max(0, k - h), hi = math.min(end, k + h); + final dd = [for (var q = lo; q <= hi; q++) _dAt(q)]..sort(); + final rs = [for (var q = lo; q <= hi; q++) _rrAt(q)]..sort(); + tTh1.add(thresholdOfSorted(dd, alpha, minThresholdMs)); + final m = medianExcluding(rs, _rrAt(k)) ?? _rrAt(k); + tMed.add(m); + tMrr.add(medianDeviation(_rrAt(k), m)); + } + // Provisional classes for [c2, n). + final tc = []; + var prev = _lastFinal; + var normalTent = 0; + for (var g = _c2; g < n; g++) { + final lo = math.max(0, g - h), hi = math.min(end, g + h); + final ms = [for (var q = lo; q <= hi; q++) mrrAt(q)]..sort(); + final th2 = thresholdOfSorted(ms, alpha, minThresholdMs); + final th1 = g < _c1 ? _th1[g - _so] : tTh1[g - _c1]; + final med = g < _c1 ? _med[g - _so] : tMed[g - _c1]; + final cls = _finalClass(g, th1, med, mrrAt(g), th2, prev); + tc.add(cls.index); + prev = cls.index; + if (cls == BeatClass.normal) normalTent++; + } + // Provisional output over [ce, n). + final fo = _ce; + final ln = List.of(_lastNormals); + final nn = [], times = []; + var dropped = _dropped, corrected = _corrected; + _emit( + ce: _ce, + limit: n, + isEnd: true, + clsAt: (g) => g < _c2 ? _fcls[g - fo] : tc[g - _c2], + lastNormals: ln, + put: (v, t) { + nn.add(v); + times.add(t); + }, + drop: (r) => dropped += r, + corrected: () => corrected++, + ); + return RrSnapshot( + n: n, + tailNn: nn, + tailNnTimes: times, + tailClasses: [for (final c in tc) BeatClass.values[c]], + normalCount: _normalFinal + normalTent, + droppedCount: dropped, + correctedCount: corrected, + settledBeats: _ce, + classifiedBeats: _c2, + ); + } + + /// Drops what no later beat can read: raw beats behind both the settled edge + /// and the oldest window a pending stage-2 beat still needs, and stage + /// arrays behind the oldest stage-2 window. + void _compact() { + final h = _h; + final newOff = math.max(0, math.min(_ce, _c2 - 2 * h - 1)); + if (newOff > _off) { + final drop = newOff - _off; + _rr.removeRange(0, drop); + _t.removeRange(0, drop); + _d.removeRange(0, drop); + _off = newOff; + } + final newSo = math.max(0, _c2 - h - 1); + if (newSo > _so && _med.isNotEmpty) { + final drop = math.min(newSo - _so, _med.length); + _med.removeRange(0, drop); + _mrr.removeRange(0, drop); + _th1.removeRange(0, drop); + _so += drop; + } + } + + /// The stage arrays are pure functions of the buffered raw beats; rebuild + /// them rather than persist them. + void _rebuildStageArrays() { + final h = _h; + _so = math.max(0, _c2 - h - 1); + _med.clear(); + _mrr.clear(); + _th1.clear(); + for (var g = _so; g < _c1; g++) { + final lo = math.max(0, g - h), hi = g + h; + final dd = [for (var q = lo; q <= hi; q++) _dAt(q)]..sort(); + final rs = [for (var q = lo; q <= hi; q++) _rrAt(q)]..sort(); + _th1.add(thresholdOfSorted(dd, alpha, minThresholdMs)); + final m = medianExcluding(rs, _rrAt(g)) ?? _rrAt(g); + _med.add(m); + _mrr.add(medianDeviation(_rrAt(g), m)); + } + _wPos = -1; + _w2Pos = -1; + } +} + +// The cap can cut a window that an uncapped detector has already read. Keep +// that decision provisional until both the run and its left influence have +// settled. No stage-1/2 arrays from across a cut are ever reused. +class _CappedState { + int n = 0, off = 0, edge = 0, segmentStart = 0; + int normal = 0, dropped = 0, corrected = 0; + bool? wall; + bool gap = false, hardGap = false; + double origin = 0, clock = 0; + BeatClass previous = BeatClass.normal; + final rr = [], ts = [], normals = []; + + RrCapWindow evaluate(RrCorrector c) => RrCapWindow.evaluate(rr, + ts: wall == true ? ts : null, + cap: c.maxArtifactRun!, + alpha: c.alpha, + windowBeats: c.windowBeats, + minThresholdMs: c.minThresholdMs, + reanchorGapMs: c.reanchorGapMs, + from: edge - off, + previous: previous, + leftNormals: normals, + leadingGap: gap, + leadingHardGap: hardGap, + segmentStart: math.max(0, segmentStart - off), + clock: clock, + origin: origin); + + RrSettled fold(RrCorrector c, List values, List? times) { + if (values.isEmpty) + return const RrSettled(nn: [], nnTimes: [], classes: []); + if (times != null && times.length != values.length) { + throw ArgumentError('RR and timestamp lengths differ'); + } + if (wall != null && wall != (times != null)) { + throw StateError('timestamps must be given on every fold or on none'); + } + for (var i = 0; i < values.length; i++) { + if (!values[i].isFinite || (times != null && !times[i].isFinite)) { + throw ArgumentError('non-finite beat at index $i'); + } + } + wall = times != null; + if (n == 0) origin = times == null ? 0 : times.first - values.first; + rr.addAll(values); + if (times != null) ts.addAll(times); + n += values.length; + final w = evaluate(c); + final nn = [], nt = [], classes = []; + // Two centred threshold windows, the cap decision and its backwards + // influence. Extra normal anchors settle isolated spline corrections. + var limit = math.max(edge, n - (4 * c._h + 2 * c.maxArtifactRun! + 4)); + // Do not split a short artifact run between folds: its final beat must + // never be mistaken for an isolated artifact at the next pending edge. + while (limit > edge && + w.classes[limit - off - 1] != BeatClass.normal && + !w.gaps[limit - off - 1]) { + limit--; + } + for (var i = edge - off; i < limit - off; i++) { + final cl = w.classes[i]; + classes.add(cl); + previous = cl; + if (cl == BeatClass.normal) { + normal++; + normals.add(rr[i]); + if (normals.length > 2) normals.removeAt(0); + } + if (w.gaps[i]) normals.clear(); + final v = w.values[i]; + if (v == null) { + dropped++; + } else { + nn.add(v); + nt.add(w.times[i]); + if (cl != BeatClass.normal) corrected++; + } + clock = w.times[i]; + if (gap && !w.gaps[i]) segmentStart = off + i; + gap = w.gaps[i]; + hardGap = w.hardGaps[i]; + } + edge = limit; + final nextOff = math.max(0, edge - 2 * c._h - 2); + final trim = nextOff - off; + if (trim > 0) { + rr.removeRange(0, trim); + if (wall == true) ts.removeRange(0, trim); + off = nextOff; + } + return RrSettled(nn: nn, nnTimes: nt, classes: classes); + } + + RrSnapshot snapshot(RrCorrector c) { + final w = evaluate(c); + final nn = [], nt = [], classes = []; + var nc = normal, dc = dropped, cc = corrected; + for (var i = edge - off; i < rr.length; i++) { + final cl = w.classes[i]; + classes.add(cl); + if (cl == BeatClass.normal) nc++; + final v = w.values[i]; + if (v == null) { + dc++; + } else { + nn.add(v); + nt.add(w.times[i]); + if (cl != BeatClass.normal) cc++; + } + } + return RrSnapshot( + n: n, + tailNn: nn, + tailNnTimes: nt, + tailClasses: classes, + normalCount: nc, + droppedCount: dc, + correctedCount: cc, + settledBeats: edge, + classifiedBeats: edge); + } + + Map toJson() => { + 'n': n, + 'off': off, + 'edge': edge, + 'segmentStart': segmentStart, + 'hardGap': hardGap, + 'normal': normal, + 'dropped': dropped, + 'corrected': corrected, + 'wall': wall, + 'gap': gap, + 'origin': origin, + 'clock': clock, + 'previous': previous.index, + 'rr': rr, + 'ts': ts, + 'normals': normals, + }; + + void restore(Map j, int half) { + n = j['n'] as int; + off = j['off'] as int; + edge = j['edge'] as int; + segmentStart = j['segmentStart'] as int; + hardGap = j['hardGap'] as bool; + normal = j['normal'] as int; + dropped = j['dropped'] as int; + corrected = j['corrected'] as int; + wall = j['wall'] as bool?; + gap = j['gap'] as bool; + origin = (j['origin'] as num).toDouble(); + clock = (j['clock'] as num).toDouble(); + previous = BeatClass.values[j['previous'] as int]; + List read(String key) => + [for (final v in j[key] as List) (v as num).toDouble()]; + rr.addAll(read('rr')); + ts.addAll(read('ts')); + normals.addAll(read('normals')); + if (segmentStart < 0 || + segmentStart > edge || + (hardGap && !gap) || + off != math.max(0, edge - 2 * half - 2) || + edge < off || + n < edge || + rr.length != n - off || + (gap && (previous == BeatClass.normal || normals.isNotEmpty)) || + (wall == true ? ts.length != rr.length : ts.isNotEmpty) || + (n > 0 && wall == null) || + normals.length > 2 || + normal < 0 || + dropped < 0 || + corrected < 0 || + normal + dropped + corrected != edge || + ![...rr, ...ts, ...normals, origin, clock].every((v) => v.isFinite)) { + throw const FormatException('inconsistent capped RR checkpoint'); + } + } +} diff --git a/lib/src/onehz/foundations/rr_sandwich.dart b/lib/src/onehz/foundations/rr_sandwich.dart new file mode 100644 index 0000000..ebf76dd --- /dev/null +++ b/lib/src/onehz/foundations/rr_sandwich.dart @@ -0,0 +1,284 @@ +// Lipponen & Tarvainen 2019 classification with Peltola 2012 correction policy. +// The unpublished artifact-run cap is an internal ESTIMATE with low confidence: +// over-cap runs become gaps and each side is re-corrected. Without a cap split +// the result equals uncapped correction. Shared kernel arithmetic and raw seam +// context preserve capped batch NN/classes; settled stream plus provisional tail +// equals capped batch. RrSandwich preserves source-beat absolute timestamps; +// uncapped arbitrarily long artifact runs have no exactness guarantee. +// FOUNDATION — the RR "sandwich": a mergeable summary over RAW RR that feeds +// capped, corrected NN into a downstream aggregator (docs/AGGREGATOR_DESIGN.md, +// "Proposed next step"). +// +// Contract (test/aggregate/rr_sandwich_test.dart pins it): +// * any chunking of a beat series (empty chunks and one-beat chunks +// included), combined left to right or right to left, finishes to what one +// pass gives, and that is `correctRr(all beats, maxArtifactRun: K)`: the +// inner aggregator sees exactly the capped NN, each with the ABSOLUTE raw +// timestamp of the input beat it came from (`nnSourceIndex`), never +// correctRr's first-beat-relative clock; +// * state is bounded (raw head, settled middle folded into `inner`, raw +// tail), never the series; +// * persistable when `inner` is. + +import 'dart:typed_data'; + +import '../../aggregate/aggregator.dart'; +import '../../aggregate/json_support.dart'; +import '../clinical/hrv_time_aggregator.dart' show Beats; +import 'rr_correction.dart'; + +/// Raw RR beats: interval (ms) and the beat's timestamp (epoch ms), the pair +/// `correctRr(rrMs, rrTsMs: tsMs)` takes. Timestamps non-decreasing. +class RawRr extends Batch { + RawRr(this.rrMs, this.tsMs) : assert(rrMs.length == tsMs.length); + + factory RawRr.of(List rrMs, List tsMs) => + RawRr(Float64List.fromList(rrMs), Float64List.fromList(tsMs)); + + factory RawRr.fromJson(Object? json) { + final map = decodeMap(json); + final rr = decodeDoubles(field(map, 'rrMs')); + final ts = decodeDoubles(field(map, 'tsMs')); + if (rr.length != ts.length) { + throw const FormatException('RawRr columns differ in length'); + } + return RawRr(rr, ts); + } + + final Float64List rrMs; + final Float64List tsMs; + + @override + int get length => rrMs.length; + + @override + RawRr copyRange(int start, int end) => RawRr( + Float64List(end - start)..setRange(0, end - start, rrMs, start), + Float64List(end - start)..setRange(0, end - start, tsMs, start)); + + @override + RawRr join(RawRr other) => RawRr( + Float64List(length + other.length) + ..setRange(0, length, rrMs) + ..setRange(length, length + other.length, other.rrMs), + Float64List(length + other.length) + ..setRange(0, length, tsMs) + ..setRange(length, length + other.length, other.tsMs)); + + @override + Object? toJson() => + {'rrMs': encodeDoubles(rrMs), 'tsMs': encodeDoubles(tsMs)}; +} + +/// Opaque [RrSandwich] state: the raw head (first `min(m, W)` beats), the +/// raw tail (last `W` beats, only once `m > W`), and the inner aggregator's +/// state over the NN of the settled middle. `m` is the number of beats seen. +class RrSandwichState { + RrSandwichState(this.m, this.head, this.tail, this.mid); + int m; + RawRr head, tail; + S mid; +} + +/// Raw RR -> capped, corrected NN -> [inner]. Parameters are `correctRr`'s. +/// +/// Each NN reaches [inner] with the ABSOLUTE timestamp (`rrTsMs`) of the input +/// beat it came from (`RrCorrectionResult.nnSourceIndex`), so a seam needs no +/// clock translation: there is no per-chunk clock at all. +/// +/// Locality. The class and the value of a beat depend only on the raw beats at +/// most [reach] either side of it: two centred threshold windows (4H, H = +/// windowBeats ~/ 2), a cap decision whose run is at most K long (the streaming +/// corrector's settle delay is 4H + 2K + 4; one more K covers a run that +/// straddles the edge), and the spline anchors. So a chunk of `m` beats has a +/// settled region `[reach, m - reach)` that is final whatever precedes or +/// follows it, and everything else is kept raw: W = 2 * reach beats at each +/// end. `combine` re-corrects only `left.tail ++ right.head` and settles the +/// beats between the two settled regions. State is bounded: head, tail, and +/// whatever [inner] keeps for the settled NN. +/// +/// With `maxArtifactRun: null` (uncapped) the reach is sized as for the +/// default cap; an artifact run longer than that is not bounded-state +/// exact, since the uncapped corrector lets one run reach arbitrarily far. +class RrSandwich extends Aggregator, RawRr, R> { + const RrSandwich( + this.inner, { + this.maxArtifactRun = kMaxArtifactRun, + this.alpha = 5.2, + this.windowBeats = 91, + this.minThresholdMs = 100, + this.reanchorGapMs = 1000, + }); + + /// Receives the settled NN (`nnMs`) with the absolute timestamp of each + /// one's source beat (`timeMs`). + final Aggregator inner; + final int? maxArtifactRun; + final double alpha; + final int windowBeats; + final double minThresholdMs; + final double reanchorGapMs; + + /// Beats of context either side that can change a beat's output. + int get reach => 4 * (windowBeats ~/ 2) + 3 * (maxArtifactRun ?? kMaxArtifactRun) + 4; + int get _w => 2 * reach; + + static final _none = RawRr(Float64List(0), Float64List(0)); + + /// NN of `correctRr(seg)` whose source beat lies in `[lo, hi)`, as a + /// [Beats] carrying each source beat's absolute timestamp. + Beats _nn(RawRr seg, int lo, int hi) { + final c = correctRr(seg.rrMs, + alpha: alpha, + windowBeats: windowBeats, + minThresholdMs: minThresholdMs, + reanchorGapMs: reanchorGapMs, + maxArtifactRun: maxArtifactRun); + final nn = [], t = []; + for (var k = 0; k < c.nn.length; k++) { + final i = c.nnSourceIndex[k]; + if (i >= lo && i < hi) { + nn.add(c.nn[k]); + t.add(seg.tsMs[i]); + } + } + return Beats(Float64List.fromList(nn), Float64List.fromList(t)); + } + + S _state(Beats b) => inner.addSlice(inner.empty(), b, 0, b.length); + + @override + RrSandwichState empty() => + RrSandwichState(0, _none, _none, inner.empty()); + + @override + RrSandwichState addSlice( + RrSandwichState state, RawRr batch, int start, int end) { + if (end <= start) return state; + return combine(state, _chunk(batch.copyRange(start, end))); + } + + RrSandwichState _chunk(RawRr raw) { + final m = raw.length, w = _w, r = reach; + if (m <= w) return RrSandwichState(m, raw, _none, inner.empty()); + return RrSandwichState(m, raw.copyRange(0, w), raw.copyRange(m - w, m), + _state(_nn(raw, r, m - r))); + } + + /// Beats `[lo, hi)` of the series [s] and [t] hold, joined; [t] follows [s]. + RawRr _range(RrSandwichState l, RrSandwichState r, int lo, int hi) { + final w = _w, ml = l.m; + RawRr part(RrSandwichState s, int a, int b) { + if (b <= a) return _none; + if (s.m <= w) return s.head.copyRange(a, b); + final base = s.m - w; // tail[0] is beat `base` + assert(a >= base, 'range reaches before the tail'); + return s.tail.copyRange(a - base, b - base); + } + + final left = part(l, lo, hi < ml ? hi : ml); + final a = lo > ml ? lo - ml : 0, b = hi - ml; + if (b <= a) return left; + assert(b <= r.head.length, 'range reaches past the head'); + final right = r.head.copyRange(a, b); + return left.length == 0 ? right : left.join(right); + } + + @override + RrSandwichState combine(RrSandwichState left, RrSandwichState right) { + if (left.m == 0) return right; + if (right.m == 0) return left; + final w = _w, rc = reach, ml = left.m, mr = right.m, m = ml + mr; + // Settled regions: left [rc, ml - rc), right (merged coordinates) + // [ml + rc, m - rc). The beats between them are settled here, from the + // raw windows around the seam. + final aEnd = ml > w ? ml - rc : rc; + final cStart = mr > w ? ml + rc : m - rc; + var mid = left.mid; + if (cStart > aEnd) { + final lo = aEnd - rc, hi = cStart + rc; + final seg = _range(left, right, lo, hi); + mid = inner.combine(mid, _state(_nn(seg, aEnd - lo, cStart - lo))); + } + if (mr > w) mid = inner.combine(mid, right.mid); + final head = ml >= w + ? left.head + : left.head.join(right.head.copyRange(0, (m < w ? m : w) - ml)); + var tail = _none; + if (m > w) { + tail = mr >= w + ? (mr == w ? right.head : right.tail) + : _range(left, right, m - w, ml).join(right.head); + } + return RrSandwichState(m, head, tail, mid); + } + + @override + R finish(RrSandwichState state) { + final m = state.m, w = _w, rc = reach; + if (m <= w) return inner.finish(_state(_nn(state.head, 0, m))); + // The settled middle is combined into fresh states; combine may consume + // its inputs, so work on a copy of the stored one. + final mid = inner.persistable + ? inner.decodeState(inner.encodeState(state.mid)) + : state.mid; + var out = _state(_nn(state.head, 0, rc)); + out = inner.combine(out, mid); + out = inner.combine(out, _state(_nn(state.tail, rc, w))); + return inner.finish(out); + } + + @override + bool get persistable => inner.persistable; + + @override + Object? encodeState(RrSandwichState state) => { + 'm': state.m, + 'head': state.head.toJson(), + 'tail': state.tail.toJson(), + 'mid': inner.encodeState(state.mid), + }; + + @override + RrSandwichState decodeState(Object? json) { + final map = decodeMap(json); + final m = field(map, 'm'); + if (m is! int || m < 0) throw const FormatException('bad RrSandwich m'); + final head = RawRr.fromJson(field(map, 'head')); + final tail = RawRr.fromJson(field(map, 'tail')); + final w = _w; + if (head.length != (m < w ? m : w) || tail.length != (m > w ? w : 0)) { + throw const FormatException('RrSandwich window lengths disagree with m'); + } + return RrSandwichState(m, head, tail, inner.decodeState(field(map, 'mid'))); + } +} + +/// Adapts a column aggregator over NN (e.g. `prsaAggregator`) to the [Beats] +/// batch [RrSandwich] feeds: it sees `Beats.nnMs` as a [DoubleColumn]. +class NnColumn extends Aggregator { + const NnColumn(this.inner); + final Aggregator inner; + + @override + S empty() => inner.empty(); + + @override + S addSlice(S state, Beats batch, int start, int end) => + inner.addSlice(state, DoubleColumn(batch.nnMs), start, end); + + @override + S combine(S left, S right) => inner.combine(left, right); + + @override + R finish(S state) => inner.finish(state); + + @override + bool get persistable => inner.persistable; + + @override + Object? encodeState(S state) => inner.encodeState(state); + + @override + S decodeState(Object? json) => inner.decodeState(json); +} diff --git a/lib/src/onehz/incremental.dart b/lib/src/onehz/incremental.dart new file mode 100644 index 0000000..a4a7c4a --- /dev/null +++ b/lib/src/onehz/incremental.dart @@ -0,0 +1,27 @@ +/// Stateful counterparts of the independent batch calculations. +library; + +import 'dart:math' as math; +import 'types.dart'; +import 'clinical/hrv_time.dart'; +import 'clinical/load_trimp.dart'; +import 'motion/enmo.dart'; +import 'workout/calories.dart'; +import 'incremental_core.dart'; + +export 'motion/enmo.dart' show defaultGravityWindowS; +export 'incremental_core.dart'; + +part 'incremental_hrv.dart'; +part 'incremental_enmo.dart'; +part 'incremental_minutes.dart'; +part 'incremental_state.dart'; + +enum CalculationMode { + periodicAwake, + sleep, + heavy, + forced; + + bool get canReuse => this == periodicAwake; +} diff --git a/lib/src/onehz/incremental_core.dart b/lib/src/onehz/incremental_core.dart new file mode 100644 index 0000000..08e78d2 --- /dev/null +++ b/lib/src/onehz/incremental_core.dart @@ -0,0 +1,636 @@ +import 'dart:math' as math; + +import '../aggregate/json_support.dart' show decodeDouble, encodeDouble; +import 'util.dart'; + +/// Removable, mergeable centered moments, with an origin to retain precision +/// when a series has a large offset and a small spread. +// Welford, B. P. (1962). Note on a method for calculating corrected sums of +// squares and products. Technometrics 4(3):419-420. +// add uses Welford's recurrence; remove algebraically reverses that update, +// with a nonnegative M2 clamp. Partition merging follows +// Chan, T. F., Golub, G. H., LeVeque, R. J. (1983). Algorithms for computing +// the sample variance: analysis and recommendations. The American Statistician +// 37(3):242-247. +// Removable/mergeable centered-moment recurrence. Algebraically equivalent to +// batch moments; floating-point order differs. The long-run drift fixture checks +// 1e-9, not bit identity. +class RunningMoments { + int _count = 0; + double _origin = 0, _meanOffset = 0, _m2 = 0; + + RunningMoments(); + + factory RunningMoments.fromJson(Map json) { + _checkVersion(json, 'RunningMoments'); + final result = RunningMoments(); + result._count = _readCount(json['count']); + result._origin = _readFinite(json['origin']); + result._meanOffset = _readFinite(json['meanOffset']); + result._m2 = _readM2(json['m2']); + if (result._m2 < 0 || + (result._count == 0 && + (result._origin != 0 || + result._meanOffset != 0 || + result._m2 != 0)) || + (result._count == 1 && result._m2 != 0)) { + throw const FormatException('Invalid centered moments'); + } + return result; + } + + int get count => _count; + double? get mean => _count == 0 ? null : _origin + _meanOffset; + double? get sampleSd => _count < 2 ? null : math.sqrt(_m2 / (_count - 1)); + double? get populationSd => _count == 0 ? null : math.sqrt(_m2 / _count); + + void add(double x) { + if (!x.isFinite) throw ArgumentError.value(x, 'x', 'Must be finite'); + if (_count == 0) { + _origin = x; + _count = 1; + return; + } + final centered = x - _origin; + final delta = centered - _meanOffset; + _count++; + _meanOffset += delta / _count; + _m2 += delta * (centered - _meanOffset); + } + + /// The caller must remove a value represented by this summary. + void remove(double x) { + if (!x.isFinite) throw ArgumentError.value(x, 'x', 'Must be finite'); + if (_count == 0) throw StateError('Cannot remove from empty moments'); + if (_count == 1) { + _count = 0; + _origin = _meanOffset = _m2 = 0; + return; + } + final centered = x - _origin; + final oldMean = _meanOffset; + _count--; + _meanOffset += (_meanOffset - centered) / _count; + _m2 = math.max(0, _m2 - (centered - oldMean) * (centered - _meanOffset)); + if (_count == 1) _m2 = 0; + } + + void merge(RunningMoments other) { + if (other._count == 0) return; + if (_count == 0) { + _count = other._count; + _origin = other._origin; + _meanOffset = other._meanOffset; + _m2 = other._m2; + return; + } + final otherCount = other._count; + final combinedCount = _count + otherCount; + final delta = (other._origin - _origin) + other._meanOffset - _meanOffset; + _m2 += other._m2 + delta * delta * _count * otherCount / combinedCount; + _meanOffset += delta * otherCount / combinedCount; + _count = combinedCount; + } + + Map toJson() => { + 'version': 1, + 'type': 'RunningMoments', + 'count': _count, + 'origin': encodeDouble(_origin), + 'meanOffset': encodeDouble(_meanOffset), + 'm2': encodeDouble(_m2), + }; +} + +/// Exact order statistics of integer-valued data, kept as a sorted +/// `value -> count` list. Heart rate is whole bpm with a few dozen distinct +/// values a night, so the list stays tiny however many samples it holds. Add, +/// remove and merge are exact; [percentile] returns the same double +/// `percentileSorted` returns for the expanded sorted list, not an estimate. +/// +/// Only for data that really is integral. Values that are nearly all distinct +/// (accelerometer magnitudes) gain nothing: the list is as long as the data. +// Hyndman, R. J., Fan, Y. (1996). Sample quantiles in statistical packages. +// The American Statistician 50(4):361-365. +// percentileSorted implements type 7: zero-based rank = (n - 1) * p / 100, +// linearly interpolating adjacent ranks; this histogram preserves that rule. +// Exact histogram reformulation of percentileSorted(): counts identify identical +// expanded-list ranks with the same interpolation. No new estimator. +class IntHistogram { + static const _maxInt64 = 0x7fffffffffffffff; + + List _values = []; + List _counts = []; + int _count = 0; + + IntHistogram(); + + factory IntHistogram.fromJson(Map json) { + _checkVersion(json, 'IntHistogram'); + final values = json['values'], counts = json['counts']; + if (values is! List || counts is! List || values.length != counts.length) { + throw const FormatException('Invalid histogram bins'); + } + final result = IntHistogram(); + for (var i = 0; i < values.length; i++) { + final v = values[i], c = counts[i]; + if (v is! int || c is! int || c < 1 || (i > 0 && v <= values[i - 1])) { + throw const FormatException('Invalid histogram bins'); + } + if (result._count > _maxInt64 - c) { + throw const FormatException('Invalid histogram counts'); + } + result._values.add(v); + result._counts.add(c); + result._count += c; + } + return result; + } + + /// Number of values held, counting repeats. + int get count => _count; + + /// Number of different values held. + int get distinct => _values.length; + + /// Linear-interpolated median, or null when empty. + double? get median => percentile(50); + + /// Index of [value], or the insertion point as `-(point + 1)`. + int _find(int value) { + var lo = 0, hi = _values.length; + while (lo < hi) { + final mid = (lo + hi) >> 1; + if (_values[mid] < value) { + lo = mid + 1; + } else { + hi = mid; + } + } + return lo < _values.length && _values[lo] == value ? lo : -(lo + 1); + } + + /// Adds [times] copies of [value], which must be a finite whole number. + /// Throws [StateError] if the total count would overflow int64. + void add(num value, [int times = 1]) { + if (!value.isFinite || value != value.truncate()) { + throw ArgumentError.value(value, 'value', 'Must be a whole number'); + } + if (times < 1) throw ArgumentError.value(times, 'times', 'Must be positive'); + if (_count > _maxInt64 - times) { + throw StateError('Histogram count would overflow int64'); + } + final v = value.toInt(); + final at = _find(v); + if (at >= 0) { + _counts[at] += times; + } else { + _values.insert(-at - 1, v); + _counts.insert(-at - 1, times); + } + _count += times; + } + + /// Removes one copy of [value]. The caller must remove a value it added. + void remove(num value) { + final at = value.isFinite && value == value.truncate() + ? _find(value.toInt()) + : -1; + if (at < 0) throw StateError('Cannot remove a value that is not present'); + if (--_counts[at] == 0) { + _values.removeAt(at); + _counts.removeAt(at); + } + _count--; + } + + /// Merges [other] into this histogram. + /// Throws [StateError] if the total count would overflow int64. + void merge(IntHistogram other) { + if (_count > _maxInt64 - other._count) { + throw StateError('Histogram count would overflow int64'); + } + final values = List.of(other._values), counts = List.of(other._counts); + for (var i = 0; i < values.length; i++) { + add(values[i], counts[i]); + } + } + + /// The [p]th percentile (0 to 100), interpolated between the two + /// neighbouring order statistics exactly as `percentileSorted` does. Null + /// when empty, never 0. + double? percentile(double p) { + if (_count == 0) return null; + if (!(p >= 0 && p <= 100)) throw ArgumentError.value(p, 'p', '0 to 100'); + if (_count == 1) return _values[0].toDouble(); + final rank = (p / 100) * (_count - 1); + final lo = rank.floor(); + final hi = rank.ceil(); + final low = _orderStatistic(lo); + if (lo == hi) return low; + final frac = rank - lo; + return low + (_orderStatistic(hi) - low) * frac; + } + + /// The value at zero-based sorted position [k]. + double _orderStatistic(int k) { + var seen = 0; + for (var i = 0; i < _values.length; i++) { + seen += _counts[i]; + if (k < seen) return _values[i].toDouble(); + } + throw StateError('Order statistic out of range'); + } + + Map toJson() => { + 'version': 1, + 'type': 'IntHistogram', + 'values': List.of(_values), + 'counts': List.of(_counts), + }; +} + +/// Fixed-grid spectral sums. Raw snapshots establish that each sync is a +/// genuine prefix; callers may revise or remove any earlier sample. +// Press & Rybicki 1989 lombScargle() on the identical fixed grid using retained +// trigonometric sums and centered moments. Algebraically equal; long-run drift +// fixtures check 1e-9, not bit identity. +class IncrementalLombScargle { + final List _frequencies; + List _t = [], _y = []; + List> _sums; + RunningMoments _moments = RunningMoments(); + int _processedPoints = 0; + bool _incremental = true; + double _origin = 0, _yOrigin = 0, _tMin = 0, _tMax = 0; + + IncrementalLombScargle(List frequencies) + : _frequencies = List.of(frequencies), + _sums = [for (final _ in frequencies) List.filled(7, 0.0)]; + + factory IncrementalLombScargle.fromJson(Map json) { + _checkVersion(json, 'IncrementalLombScargle'); + final result = IncrementalLombScargle(_readDoubles(json['frequencies'])); + result._t = _readDoubles(json['t']); + result._y = _readDoubles(json['y']); + result._processedPoints = _readCount(json['processedPoints']); + final incremental = json['incremental']; + if (incremental is! bool || result._t.length != result._y.length) { + throw const FormatException('Invalid spectral input state'); + } + result._incremental = incremental; + result._origin = _readFinite(json['origin']); + result._yOrigin = _readFinite(json['yOrigin']); + result._tMin = _readFinite(json['tMin']); + result._tMax = _readFinite(json['tMax']); + final moments = json['moments']; + if (moments is! Map) { + throw const FormatException('Missing spectral moments'); + } + result._moments = RunningMoments.fromJson(moments); + final sums = json['sums']; + if (sums is! List || sums.length != result._frequencies.length) { + throw const FormatException('Invalid spectral sum dimensions'); + } + result._sums = [ + for (final row in sums) _readDoubles(row, finite: true), + ]; + if (result._sums.any((row) => row.length != 7) || + (incremental && + (!result._allFinite(result._t, result._y) || + result._moments.count != result._t.length || + result._processedPoints < result._t.length)) || + (!incremental && result._moments.count != 0)) { + throw const FormatException('Invalid spectral accumulator'); + } + if (result._t.isEmpty || !incremental) { + if (result._origin != 0 || + result._yOrigin != 0 || + result._tMin != 0 || + result._tMax != 0 || + result._sums.any((row) => row.any((value) => value != 0))) { + throw const FormatException('Invalid empty spectral state'); + } + } + if (incremental && result._t.isNotEmpty) { + if (result._origin != result._t.first || + result._yOrigin != result._y.first || + result._tMin != result._t.reduce(math.min) || + result._tMax != result._t.reduce(math.max)) { + throw const FormatException('Invalid spectral origin or extrema'); + } + for (final row in result._sums) { + if (row[2] < 0 || + row[3] < 0 || + ((row[2] + row[3]) - result._t.length).abs() > + 1e-8 * result._t.length) { + throw const FormatException('Invalid trigonometric sums'); + } + } + } + return result; + } + + int get processedPoints => _processedPoints; + + bool _allFinite(List t, List y) => + _frequencies.every((x) => x.isFinite) && + t.every((x) => x.isFinite) && + y.every((x) => x.isFinite); + + LombScargle? sync(List t, List y, {bool force = false}) { + if (t.length != y.length) { + _reset(); + return lombScargle(t, y, _frequencies); + } + if (!_allFinite(t, y)) { + // The batch function defines the NaN/infinity behavior. Never fold these + // values into summaries that could contaminate later finite prefixes. + final result = lombScargle(t, y, _frequencies); + _reset(); + _incremental = false; + _t = List.of(t); + _y = List.of(y); + if (t.length >= 4 && _frequencies.isNotEmpty) + _processedPoints += t.length; + return result; + } + var prefix = !force && _incremental && t.length >= _t.length; + if (prefix) { + for (var i = 0; i < _t.length; i++) { + if (_t[i] != t[i] || _y[i] != y[i]) { + prefix = false; + break; + } + } + } + if (!prefix) _reset(); + final start = _t.length; + if (start == 0 && t.isNotEmpty) { + _origin = _tMin = _tMax = t.first; + _yOrigin = y.first; + } + for (var i = start; i < t.length; i++) { + _moments.add(y[i]); + _tMin = math.min(_tMin, t[i]); + _tMax = math.max(_tMax, t[i]); + for (var j = 0; j < _frequencies.length; j++) { + final w = 2 * math.pi * _frequencies[j]; + final c = math.cos(w * (t[i] - _origin)); + final s = math.sin(w * (t[i] - _origin)); + final centeredY = y[i] - _yOrigin; + final sums = _sums[j]; + sums[0] += c; + sums[1] += s; + sums[2] += c * c; + sums[3] += s * s; + sums[4] += c * s; + sums[5] += centeredY * c; + sums[6] += centeredY * s; + } + _processedPoints++; + } + _t = List.of(t); + _y = List.of(y); + return _render(); + } + + void _reset() { + _t = []; + _y = []; + _sums = [for (final _ in _frequencies) List.filled(7, 0.0)]; + _moments = RunningMoments(); + _origin = _yOrigin = _tMin = _tMax = 0; + _incremental = true; + } + + LombScargle? _render() { + final n = _moments.count; + final span = _tMax - _tMin; + if (n < 4 || _frequencies.isEmpty || _moments._m2 <= 0 || span <= 0) { + return null; + } + final centeredMean = (_moments._origin - _yOrigin) + _moments._meanOffset; + final spectrum = []; + for (var j = 0; j < _frequencies.length; j++) { + final frequency = _frequencies[j]; + if (frequency == 0) { + spectrum.add(const LsPoint(0, 0)); + continue; + } + final sums = _sums[j]; + final phi = 0.5 * math.atan2(2 * sums[4], sums[2] - sums[3]); + final a = math.cos(phi), b = math.sin(phi); + final uc = sums[5] - centeredMean * sums[0]; + final us = sums[6] - centeredMean * sums[1]; + final cNum = a * uc + b * us, sNum = a * us - b * uc; + final cDen = a * a * sums[2] + b * b * sums[3] + 2 * a * b * sums[4]; + final sDen = a * a * sums[3] + b * b * sums[2] - 2 * a * b * sums[4]; + final term1 = cDen == 0 ? 0.0 : cNum * cNum / cDen; + final term2 = sDen == 0 ? 0.0 : sNum * sNum / sDen; + spectrum.add(LsPoint(frequency, span / (n - 1) * (term1 + term2))); + } + return LombScargle(spectrum); + } + + Map toJson() => { + 'version': 1, + 'type': 'IncrementalLombScargle', + 'frequencies': _writeDoubles(_frequencies), + 't': _writeDoubles(_t), + 'y': _writeDoubles(_y), + 'processedPoints': _processedPoints, + 'incremental': _incremental, + 'origin': _origin, + 'yOrigin': _yOrigin, + 'tMin': _tMin, + 'tMax': _tMax, + 'moments': _moments.toJson(), + 'sums': [for (final row in _sums) List.of(row)], + }; +} + +void _checkVersion(Map json, String type) { + if (json['version'] is! int || json['version'] != 1 || json['type'] != type) { + throw FormatException('Unsupported $type checkpoint'); + } +} + +int _readCount(Object? value) { + if (value is! int || value < 0) throw const FormatException('Invalid count'); + return value; +} + +double _readFinite(Object? value) { + if (value is! num || !value.isFinite) + throw const FormatException('Invalid number'); + return value.toDouble(); +} + +double _readM2(Object? value) { + final result = decodeDouble(value); + if ((!result.isFinite && value is! String) || + result.isNaN || + result.isNegative) { + throw const FormatException('Invalid number'); + } + return result; +} + +List _readDoubles(Object? value, {bool finite = false}) { + if (value is! List) throw const FormatException('Expected a numeric list'); + return [for (final item in value) _readDouble(item, finite: finite)]; +} + +double _readDouble(Object? value, {bool finite = false}) { + if (value is num && (!finite || value.isFinite)) return value.toDouble(); + if (!finite) { + if (value == 'NaN') return double.nan; + if (value == 'Infinity') return double.infinity; + if (value == '-Infinity') return double.negativeInfinity; + } + throw const FormatException('Invalid number'); +} + +List _writeDoubles(List values) => [ + for (final value in values) + if (value.isFinite) + value + else if (value.isNaN) + 'NaN' + else if (value > 0) + 'Infinity' + else + '-Infinity', + ]; + +class _CalculationEntry { + final Object? dependencies; + final Object? value; + const _CalculationEntry(this.dependencies, this.value); +} + +/// Bounded least-recently-used results with owned dependency snapshots. +class CalculationCache { + final int maxEntries; + final Map _entries = {}; + int _computations = 0, _hits = 0; + + CalculationCache({this.maxEntries = 128}) { + if (maxEntries < 1) throw ArgumentError.value(maxEntries, 'maxEntries'); + } + + int get computations => _computations; + int get hits => _hits; + + T evaluate(String key, Object? dependencies, T Function() calculate, + {bool full = false}) { + final old = _entries[key]; + if (!full && old != null && _deepEqual(old.dependencies, dependencies)) { + final result = _copyValue(old.value) as T; + _entries.remove(key); + _entries[key] = old; + _hits++; + return result; + } + // Snapshot before calling user code, which may itself mutate dependencies. + // Prepare both copies before committing, so exceptions preserve the old entry. + final snapshot = _snapshotDependency(dependencies); + final value = _copyValue(calculate()); + final result = _copyValue(value) as T; + _entries.remove(key); + _entries[key] = _CalculationEntry(snapshot, value); + while (_entries.length > maxEntries) { + _entries.remove(_entries.keys.first); + } + _computations++; + return result; + } + + void clear() => _entries.clear(); +} + +Object? _snapshotDependency(Object? value) { + if (value is List) + return [for (final item in value) _snapshotDependency(item)]; + if (value is Map) { + return { + for (final entry in value.entries) + entry.key: _snapshotDependency(entry.value) + }; + } + return value; +} + +bool _deepEqual(Object? a, Object? b) { + if (identical(a, b)) return true; + if (a is num && b is num && a.isNaN && b.isNaN) return true; + if (a is List && b is List) { + if (a.length != b.length) return false; + for (var i = 0; i < a.length; i++) { + if (!_deepEqual(a[i], b[i])) return false; + } + return true; + } + if (a is Map && b is Map) { + if (a.length != b.length) return false; + for (final key in a.keys) { + if (!b.containsKey(key) || !_deepEqual(a[key], b[key])) return false; + } + return true; + } + return a == b; +} + +Object? _copyValue(Object? value) { + if (value is List) { + // toList dispatches on the original List and keeps its reified E. + final copy = value.toList(); + for (var i = 0; i < copy.length; i++) { + copy[i] = _copyValue(value[i]); + } + return copy; + } + if (value is Map) { + // Preserve the usual JSON map types, including typed scalar/list values. + // Unlike List.toList, Map.map creates new key/value types from its callback. + if (value is Map) return _copyMap(value); + if (value is Map) return _copyMap(value); + if (value is Map) return _copyMap(value); + if (value is Map) return _copyMap(value); + if (value is Map) return _copyMap(value); + if (value is Map) return _copyMap(value); + if (value is Map) return _copyMap(value); + if (value is Map) return _copyMap(value); + if (value is Map) return _copyMap(value); + if (value is Map) return _copyMap(value); + if (value is Map>) return _copyMap(value); + if (value is Map>) return _copyMap(value); + if (value is Map>) return _copyMap(value); + if (value is Map>) return _copyMap(value); + if (value is Map>) return _copyMap(value); + if (value is Map>) return _copyMap(value); + if (value is Map>) return _copyMap(value); + if (value is Map>) return _copyMap(value); + if (value is Map>) return _copyMap(value); + if (value is Map>) return _copyMap(value); + if (value is Map>>) + return _copyMap(value); + if (value is Map>) return _copyMap(value); + if (value is Map) return _copyMap(value); + if (value is Map) return _copyMap(value); + if (value is Map) return _copyMap(value); + if (value is Map) return _copyMap(value); + if (value is Map>) return _copyMap(value); + if (value is Map>) return _copyMap(value); + if (value is Map) return _copyMap(value); + return _copyMap(value); + } + // Metric and other immutable domain values retain their identity and type. + return value; +} + +Map _copyMap(Map value) => { + for (final entry in value.entries) + entry.key: _copyValue(entry.value) as V, + }; diff --git a/lib/src/onehz/incremental_enmo.dart b/lib/src/onehz/incremental_enmo.dart new file mode 100644 index 0000000..af19a49 --- /dev/null +++ b/lib/src/onehz/incremental_enmo.dart @@ -0,0 +1,198 @@ +part of 'incremental.dart'; + +class _MotionBin { + final List mags = []; + double magSum = 0, enmoSum = 0, dynSum = 0; + MotionMinute? row; + void render(int key) { + final mean = magSum / mags.length; + var deviation = 0.0; + for (final mag in mags) { + deviation += (mag - mean).abs(); + } + row = MotionMinute(key * 60000.0, mags.length, enmoSum / mags.length, + deviation / mags.length, mean, dynSum / mags.length); + } +} + +/// Explicit-reference motion updates retain the causal trailing axis queue. +/// Automatic calibration is retrospective and uses the complete batch path. +// van Hees 2013 ENMO and Vähä-Ypyä 2015 MAD, as in enmoSeries(); dynAmp remains +// the internal vector high-pass adaptation. Explicit gravity reference uses the +// same trailing queue; automatic calibration recomputes batch. Checked numeric +// field tolerance is max(1e-9 absolute, 1e-8 relative); sample counts and minute +// timestamps match exactly. +class IncrementalEnmoSeries { + List _valid = []; + final Map _bins = {}; + double? _gRef; + double _windowS = defaultGravityWindowS; + double _sx = 0, _sy = 0, _sz = 0; + int _lo = 0, _processedPoints = 0; + IncrementalEnmoSeries(); + + factory IncrementalEnmoSeries.fromJson(Map json) { + _version(json, 'IncrementalEnmoSeries'); + final state = IncrementalEnmoSeries(); + state._processedPoints = _count(json['processedPoints']); + state._gRef = + json['gRef'] == null ? null : _number(json['gRef'], finite: true); + state._windowS = _number(json['windowS'], finite: true); + state._sx = _number(json['sx'], finite: true); + state._sy = _number(json['sy'], finite: true); + state._sz = _number(json['sz'], finite: true); + state._lo = _count(json['lo']); + if (json['valid'] is! List || json['bins'] is! List) + throw const FormatException('Invalid motion state'); + state._valid = [ + for (final raw in json['valid'] as List) + (() { + final row = _numbers(raw, finite: true); + if (row.length != 4) + throw const FormatException('Invalid acceleration row'); + return AccelSample(row[0], row[1], row[2], row[3]); + })() + ]; + for (final raw in json['bins'] as List) { + final data = _map(raw); + if (data['key'] is! int) + throw const FormatException('Invalid motion minute'); + final key = data['key'] as int; + if (state._bins.containsKey(key)) + throw const FormatException('Duplicate motion minute'); + final bin = _MotionBin(); + bin.mags.addAll(_numbers(data['mags'], finite: true)); + bin.magSum = _number(data['magSum'], finite: true); + bin.enmoSum = _number(data['enmoSum'], finite: true); + bin.dynSum = _number(data['dynSum'], finite: true); + if (bin.mags.isEmpty || bin.enmoSum < 0 || bin.dynSum < 0) + throw const FormatException('Invalid motion sums'); + bin.render(key); + state._bins[key] = bin; + } + if (state._processedPoints < state._valid.length || + state._lo > math.max(0, state._valid.length - 1) || + state._bins.values.fold(0, (n, b) => n + b.mags.length) != + state._valid.length || + (state._gRef == null && state._valid.isNotEmpty)) + throw const FormatException('Inconsistent motion checkpoint'); + for (var i = 1; i < state._valid.length; i++) { + if (state._valid[i].tsMs < state._valid[i - 1].tsMs) + throw const FormatException('Unsorted motion state'); + } + return state; + } + int get processedPoints => _processedPoints; + EnmoResult sync(List samples, + {double? gRef, + int minSamplesPerMinute = 30, + double gravityWindowS = defaultGravityWindowS, + int? expectedMinutes, + bool force = false}) { + final valid = samples.where((s) => s.valid).toList() + ..sort((a, b) => a.tsMs.compareTo(b.tsMs)); + if (gRef == null || + !gRef.isFinite || + !gravityWindowS.isFinite || + valid.any((s) => + !s.tsMs.isFinite || + !s.x.isFinite || + !s.y.isFinite || + !s.z.isFinite)) { + final result = enmoSeries(samples, + gRef: gRef, + minSamplesPerMinute: minSamplesPerMinute, + gravityWindowS: gravityWindowS, + expectedMinutes: expectedMinutes); + _reset(); + _processedPoints += valid.length; + return result; + } + var append = !force && + gRef == _gRef && + gravityWindowS == _windowS && + valid.length >= _valid.length; + if (append) { + for (var i = 0; i < _valid.length; i++) { + final a = _valid[i], b = valid[i]; + if (a.tsMs != b.tsMs || a.x != b.x || a.y != b.y || a.z != b.z) { + append = false; + break; + } + } + } + if (!append) _reset(); + _gRef = gRef; + _windowS = gravityWindowS; + final changed = {}; + for (var i = _valid.length; i < valid.length; i++) { + final s = valid[i]; + _sx += s.x; + _sy += s.y; + _sz += s.z; + while (_lo < i && s.tsMs - valid[_lo].tsMs >= gravityWindowS * 1000) { + _sx -= valid[_lo].x; + _sy -= valid[_lo].y; + _sz -= valid[_lo].z; + _lo++; + } + final n = i - _lo + 1; + final dx = s.x - _sx / n, dy = s.y - _sy / n, dz = s.z - _sz / n; + final dyn = math.sqrt(dx * dx + dy * dy + dz * dz); + final mag = math.sqrt(s.x * s.x + s.y * s.y + s.z * s.z); + final key = (s.tsMs / 60000).floor(); + final bin = _bins[key] ??= _MotionBin(); + bin.mags.add(mag); + bin.magSum += mag; + bin.enmoSum += mag > gRef ? mag - gRef : 0; + bin.dynSum += dyn; + changed.add(key); + _processedPoints++; + } + _valid = valid; + for (final key in changed) { + _bins[key]!.render(key); + } + final keys = _bins.keys.toList()..sort(); + final covered = + _bins.values.where((b) => b.mags.length >= minSamplesPerMinute).length; + final span = keys.isEmpty ? 0 : keys.last - keys.first + 1; + final denominator = expectedMinutes ?? span; + return EnmoResult(gRef, [for (final key in keys) _bins[key]!.row!], + denominator <= 0 ? 0 : (covered / denominator).clamp(0.0, 1.0)); + } + + void _reset() { + _valid = []; + _bins.clear(); + _gRef = null; + _windowS = defaultGravityWindowS; + _sx = _sy = _sz = 0; + _lo = 0; + } + + Map toJson() => { + 'version': 1, + 'type': 'IncrementalEnmoSeries', + 'processedPoints': _processedPoints, + 'gRef': _gRef, + 'windowS': _windowS, + 'sx': _sx, + 'sy': _sy, + 'sz': _sz, + 'lo': _lo, + 'valid': [ + for (final s in _valid) [s.tsMs, s.x, s.y, s.z] + ], + 'bins': [ + for (final entry in _bins.entries) + { + 'key': entry.key, + 'mags': List.of(entry.value.mags), + 'magSum': entry.value.magSum, + 'enmoSum': entry.value.enmoSum, + 'dynSum': entry.value.dynSum, + } + ], + }; +} diff --git a/lib/src/onehz/incremental_hrv.dart b/lib/src/onehz/incremental_hrv.dart new file mode 100644 index 0000000..3c576a3 --- /dev/null +++ b/lib/src/onehz/incremental_hrv.dart @@ -0,0 +1,266 @@ +part of 'incremental.dart'; + +/// Appends NN moments, gap-aware differences and mutable five-minute bins. +// Task Force 1996 hrvTime() statistics with retained moments and gap-aware +// differences. Applicable jitter exemptions/fallbacks invoke batch hrvTime(); +// otherwise numerical accumulation differs. The jitter policy is an internal +// rule, ESTIMATE with low confidence. Welch 1967 and Hirsch & Bishop 1981 +// support spectral/respiratory components only; see rmssd_gate.dart. Checked numeric field tolerance is max(1e-9 +// absolute, 1e-8 relative); presence, tier, note, inputs and beat counts match +// exactly. +class IncrementalHrvTime { + List _nn = []; + List? _times; + RunningMoments _levels = RunningMoments(); + final List _bins = []; + RunningMoments _binMeans = RunningMoments(); + double _binSdSum = 0; + int _binIdx = 0; + int _pairs = 0, _over50 = 0, _lagPairs = 0, _processedPoints = 0; + double _diffSum = 0, _diffSquares = 0, _products = 0, _endpoints = 0; + double? _lastDiff; + bool _finite = true; + // Evidence for the jitter gate below the ACF floor, derived from the series + // (not checkpointed; rebuilt from `nn` / `times` on restore): the Welch + // segments `nnDiffNoiseShare` would see (64 points, stride 32, per + // contiguous run). See [sync] for why under 20 the batch can only refuse. + int _closedSegments = 0, _runDiffs = 0; + + IncrementalHrvTime(); + factory IncrementalHrvTime.fromJson(Map json) { + _version(json, 'IncrementalHrvTime'); + final state = IncrementalHrvTime(); + state._nn = _numbers(json['nn']); + state._times = json['times'] == null ? null : _numbers(json['times']); + state._processedPoints = _count(json['processedPoints']); + if (json['finite'] is! bool) + throw const FormatException('Invalid HRV mode'); + state._finite = json['finite'] as bool; + state._levels = RunningMoments.fromJson(_map(json['levels'])); + state._pairs = _count(json['pairs']); + state._over50 = _count(json['over50']); + state._lagPairs = _count(json['lagPairs']); + state._diffSum = _number(json['diffSum'], finite: true); + state._diffSquares = _number(json['diffSquares'], finite: true); + state._products = _number(json['products'], finite: true); + state._endpoints = _number(json['endpoints'], finite: true); + state._lastDiff = json['lastDiff'] == null + ? null + : _number(json['lastDiff'], finite: true); + if (json['binIdx'] is! int || json['bins'] is! List) + throw const FormatException('Invalid HRV bins'); + state._binIdx = json['binIdx'] as int; + for (final bin in json['bins'] as List) { + final moments = RunningMoments.fromJson(_map(bin)); + if (moments.count == 0) throw const FormatException('Empty HRV bin'); + state._bins.add(moments); + if (moments.count >= 2) { + state._binMeans.add(moments.mean!); + state._binSdSum += moments.sampleSd!; + } + } + final timed = + state._times != null && state._times!.length == state._nn.length; + if (state._processedPoints < state._nn.length || + state._over50 > state._pairs || + state._lagPairs > state._pairs || + state._pairs > math.max(0, state._nn.length - 1) || + state._diffSquares < 0 || + (state._finite && + (state._levels.count != state._nn.length || + state._nn.any((v) => !v.isFinite) || + (timed && state._times!.any((v) => !v.isFinite)))) || + (!state._finite && state._levels.count != 0) || + (state._finite && + timed && + state._bins.fold(0, (n, b) => n + b.count) != + state._nn.length) || + ((!timed || !state._finite) && state._bins.isNotEmpty)) { + throw const FormatException('Inconsistent HRV checkpoint'); + } + if (state._finite) { + for (var i = 0; i < state._nn.length; i++) { + state._evidence(i, state._nn, timed ? state._times : null); + } + } + return state; + } + + static int _segmentsOf(int diffs) => diffs + 1 < 64 ? 0 : (diffs + 1 - 64) ~/ 32 + 1; + + /// Updates the jitter-gate evidence for point [i] of [nn] ([times] when the + /// series is timed), splitting runs exactly as `hrvTime` does. + void _evidence(int i, List nn, List? times) { + if (i == 0) return; + if (times != null && times[i] - times[i - 1] > nn[i] + .5) { + _closedSegments += _segmentsOf(_runDiffs); + _runDiffs = 0; + } else { + _runDiffs++; + } + } + int get processedPoints => _processedPoints; + Metric sync(List nnMs, + {List? nnTimesMs, + double artifactFraction = 0, + bool force = false}) { + final timed = nnTimesMs != null && nnTimesMs.length == nnMs.length; + final oldTimed = _times != null && _times!.length == _nn.length; + final finite = nnMs.every((v) => v.isFinite) && + (!timed || nnTimesMs.every((v) => v.isFinite)); + if (!finite) { + final result = hrvTime(nnMs, + nnTimesMs: nnTimesMs, artifactFraction: artifactFraction); + _reset(); + _finite = false; + _nn = List.of(nnMs); + _times = nnTimesMs == null ? null : List.of(nnTimesMs); + _processedPoints += nnMs.length; + return result; + } + final append = !force && + _finite && + timed == oldTimed && + _prefix(_nn, nnMs) && + (!timed || _prefix(_times!, nnTimesMs)); + if (!append) _reset(); + for (var i = _nn.length; i < nnMs.length; i++) { + final value = nnMs[i]; + _evidence(i, nnMs, timed ? nnTimesMs : null); + _levels.add(value); + if (i > 0) { + if (timed && nnTimesMs[i] - nnTimesMs[i - 1] > value + .5) { + _lastDiff = null; + } else { + final d = value - nnMs[i - 1]; + _pairs++; + _diffSum += d; + _diffSquares += d * d; + if (d.abs() > 50) _over50++; + if (_lastDiff != null) { + _lagPairs++; + _products += _lastDiff! * d; + _endpoints += _lastDiff! + d; + } + _lastDiff = d; + } + } + if (timed) { + final idx = ((nnTimesMs[i] - nnTimesMs.first) / 300000).floor(); + if (_bins.isEmpty || idx != _binIdx) { + _bins.add(RunningMoments()); + _binIdx = idx; + } + final bin = _bins.last; + if (bin.count >= 2) { + _binMeans.remove(bin.mean!); + _binSdSum -= bin.sampleSd!; + } + bin.add(value); + if (bin.count >= 2) { + _binMeans.add(bin.mean!); + _binSdSum += bin.sampleSd!; + } + } + _processedPoints++; + } + _nn = List.of(nnMs); + _times = nnTimesMs == null ? null : List.of(nnTimesMs); + if (nnMs.length < 2) + return hrvTime(nnMs, + nnTimesMs: nnTimesMs, artifactFraction: artifactFraction); + double? acf; + if (_pairs >= 30) { + final m = _diffSum / _pairs; + final variance = _diffSquares - _diffSum * m; + if (variance > 0) + acf = (_products - m * _endpoints + _lagPairs * m * m) / variance; + // Reevaluate ill-conditioned sums and decisions at the refusal boundary + // using the batch function's direct centered differences. + if ((_diffSquares > 0 && variance.abs() < 1e-12 * _diffSquares) || + (acf != null && (acf - kNnDiffAcf1Floor).abs() < 1e-10)) { + _processedPoints += nnMs.length; + return hrvTime(nnMs, + nnTimesMs: nnTimesMs, artifactFraction: artifactFraction); + } + } + final jittery = acf != null && acf < kNnDiffAcf1Floor; + // Under the floor the batch refuses only when the spectrum shows no + // respiratory line on a low floor, and a refused timed night may still + // publish from windows on a breathing line steady in Hz (hrv_time.dart + // `_jitterRefused`, `_breathingRmssd`). Both read the difference runs, so + // when either could apply the batch decides, on the series handed in. + // With fewer than 20 segments neither can: the noise share is null (the + // night is refused), and the breathing test needs 12 windows each holding + // a contiguous run of 127 differences, which alone makes at least 3 + // segments per window (36). So a short or fragmented refused night stays + // on the sums. + if (jittery && _closedSegments + _segmentsOf(_runDiffs) >= 20) { + _processedPoints += nnMs.length; + return hrvTime(nnMs, + nnTimesMs: nnTimesMs, artifactFraction: artifactFraction); + } + final quality = + acf == null ? 1.0 : (1 - acf / kNnDiffAcf1Floor).clamp(0.0, 1.0); + return Metric( + value: HrvTime( + nBeats: nnMs.length, + sdnn: _levels.sampleSd, + rmssd: _pairs > 0 && !jittery + ? math.sqrt(_diffSquares / _pairs) + : null, + pnn50: _pairs > 0 && !jittery ? 100.0 * _over50 / _pairs : null, + sdann: _binMeans.count >= 2 ? _binMeans.sampleSd : null, + sdnnIndex: + _binMeans.count >= 2 ? _binSdSum / _binMeans.count : null, + diffAcf1: acf), + confidence: ((nnMs.length / 250.0).clamp(0.0, 1.0) * + quality * + (1 - artifactFraction)) + .clamp(.3, .95), + tier: Tier.high, + inputs_used: const ['rr_cleaned'], + note: jittery + ? 'rmssd_refused:acf1=${acf.toStringAsFixed(3)} — the NN successive ' + 'differences are essentially differenced white noise (−0.5 = pure, floor ' + '$kNnDiffAcf1Floor), so RMSSD/pNN50 would measure beat-timing jitter, not ' + 'vagal tone. SDNN/SDANN survive it and are the lead here. PRV not ECG-HRV.' + : 'PRV not ECG-HRV; RMSSD/pNN50 are quantization-sensitive at 1 Hz ' + '— lead with SDNN/SDANN'); + } + + void _reset() { + _nn = []; + _times = null; + _levels = RunningMoments(); + _bins.clear(); + _binMeans = RunningMoments(); + _binSdSum = 0; + _binIdx = 0; + _pairs = _over50 = _lagPairs = 0; + _diffSum = _diffSquares = _products = _endpoints = 0; + _lastDiff = null; + _finite = true; + _closedSegments = _runDiffs = 0; + } + + Map toJson() => { + 'version': 1, + 'type': 'IncrementalHrvTime', + 'nn': _nn.map(_encode).toList(), + 'times': _times?.map(_encode).toList(), + 'processedPoints': _processedPoints, + 'finite': _finite, + 'levels': _levels.toJson(), + 'bins': [for (final b in _bins) b.toJson()], + 'binIdx': _binIdx, + 'pairs': _pairs, + 'over50': _over50, + 'lagPairs': _lagPairs, + 'diffSum': _diffSum, + 'diffSquares': _diffSquares, + 'products': _products, + 'endpoints': _endpoints, + 'lastDiff': _lastDiff, + }; +} diff --git a/lib/src/onehz/incremental_minutes.dart b/lib/src/onehz/incremental_minutes.dart new file mode 100644 index 0000000..8affd79 --- /dev/null +++ b/lib/src/onehz/incremental_minutes.dart @@ -0,0 +1,376 @@ +part of 'incremental.dart'; + +class MinuteMetrics { + final Metric trimp; + final Metric strain; + final ({double total, double active, double basal, double walking})? energy; + final MinuteEnergySeries? minutes; + const MinuteMetrics( + {required this.trimp, required this.strain, this.energy, this.minutes}); +} + +class _MinuteBill { + final double hr; + final double? cadence; + final double trimp; + final MinuteEnergySource? source; + final double active, walking; + const _MinuteBill(this.hr, this.cadence, this.trimp, this.source, this.active, + this.walking); +} + +/// Retains nonlinear TRIMP and energy contributions by caller-supplied key. +/// Keys must be unique. Day duration must be a finite whole number of minutes, +/// matching the batch dailyEnergy API's integer duration. +// Banister 1991 / Morton 1990 TRIMP; Keytel 2005 / Mifflin-St Jeor 1990 / Tudor- +// Locke 2019 energy. Retains identical keyed minute contributions; edits replace +// them and dependency changes rebuild. Checked numeric field tolerance is +// max(1e-9 absolute, 1e-8 relative); presence, tier, inputs, notes, minute keys, +// counts, sources and abstention reasons match exactly. +class IncrementalMinuteMetrics { + final Map _bills = {}; + final List _keys = []; + final List<_MinuteBill> _orderedBills = []; + List? _parameters; + double _trimpTotal = 0; + double _hrActiveTotal = 0, _walkingTotal = 0; + double? _basalPerMinute; + MinuteEnergyPricer? _energyPricer; + bool _pricerReady = false; + int _processedMinutes = 0; + IncrementalMinuteMetrics(); + + factory IncrementalMinuteMetrics.fromJson(Map json) { + _version(json, 'IncrementalMinuteMetrics'); + final state = IncrementalMinuteMetrics(); + state._processedMinutes = _count(json['processedMinutes']); + state._trimpTotal = _number(json['trimpTotal'], finite: true); + state._basalPerMinute = json['basalPerMinute'] == null + ? null + : _number(json['basalPerMinute'], finite: true); + if (json['parameters'] != null) { + final parameters = json['parameters']; + if (parameters is! List || + parameters.length != 4 || + parameters[2] is! String || + !Sex.values.any((s) => s.name == parameters[2])) { + throw const FormatException('Invalid minute parameters'); + } + for (final i in [0, 1]) { + if (parameters[i] != null) _number(parameters[i]); + } + final profile = parameters[3]; + if (profile != null) { + if (profile is! List || profile.length != 4 || profile[3] is! String) + throw const FormatException('Invalid minute profile'); + for (var i = 0; i < 3; i++) { + _number(profile[i]); + } + } + state._parameters = [ + parameters[0], + parameters[1], + parameters[2], + profile == null ? null : List.of(profile as List), + ]; + } + if (json['bills'] is! List) + throw const FormatException('Invalid minute bills'); + var summedTrimp = 0.0; + for (final raw in json['bills'] as List) { + final data = _map(raw); + if (data['key'] is! int) + throw const FormatException('Invalid minute key'); + final key = data['key'] as int; + if (state._bills.containsKey(key)) + throw const FormatException('Duplicate minute key'); + final hr = _number(data['hr']); + final cadence = data['cadence'] == null ? null : _number(data['cadence']); + final trimp = _number(data['trimp'], finite: true); + final active = _number(data['active'], finite: true); + final walking = _number(data['walking'], finite: true); + final sourceName = data['source']; + MinuteEnergySource? source; + if (sourceName != null) { + if (sourceName is! String || + !MinuteEnergySource.values.any((s) => s.name == sourceName)) + throw const FormatException('Invalid energy source'); + source = MinuteEnergySource.values.byName(sourceName); + } + if (trimp < 0 || + active < 0 || + walking < 0 || + (source != MinuteEnergySource.cadence && walking != 0) || + (source != MinuteEnergySource.hr && active != 0) || + (state._basalPerMinute == null && source != null)) + throw const FormatException('Invalid minute contribution'); + state._bills[key] = + _MinuteBill(hr, cadence, trimp, source, active, walking); + state._keys.add(key); + state._orderedBills.add(state._bills[key]!); + summedTrimp += trimp; + state._hrActiveTotal += active; + state._walkingTotal += walking; + } + if (state._processedMinutes < state._bills.length || + (state._parameters == null && state._bills.isNotEmpty) || + (summedTrimp - state._trimpTotal).abs() > + 1e-9 * math.max(1, summedTrimp)) + throw const FormatException('Inconsistent minute checkpoint'); + // Older version-one checkpoints have all minute contributions and can + // reconstruct these totals. New checkpoints retain the unrounded sums. + for (final field in ['hrActiveTotal', 'walkingTotal']) { + if (!json.containsKey(field)) continue; + final saved = _number(json[field], finite: true); + final rebuilt = + field == 'hrActiveTotal' ? state._hrActiveTotal : state._walkingTotal; + if (saved < 0 || (saved - rebuilt).abs() > 1e-9 * math.max(1, rebuilt)) { + throw const FormatException('Inconsistent energy total'); + } + if (field == 'hrActiveTotal') { + state._hrActiveTotal = saved; + } else { + state._walkingTotal = saved; + } + } + return state; + } + int get processedMinutes => _processedMinutes; + + MinuteMetrics sync( + List minuteKeys, + List hr, { + List? cadenceSpm, + double? restingHr, + double? maxHr, + Sex sex = Sex.male, + WorkoutUserProfile? profile, + int dayMinutes = 1440, + double? quietHrr, + // As in [strainScoreMetric]. Defaults to calibrating: leaving it out can + // only understate confidence, never claim a settled level. + bool quietSettled = false, + bool force = false, + bool includeMinuteSeries = true, + }) { + if (minuteKeys.length != hr.length || + (cadenceSpm != null && cadenceSpm.length != hr.length)) + throw ArgumentError('Minute keys, HR and cadence must align'); + // A proven ordered prefix already has unique retained keys. Validate only + // new keys in that case; arbitrary edits use one set for validation and + // removal. Keep our own keys so caller mutations cannot poison this proof. + var keyPrefix = minuteKeys.length >= _keys.length; + final changed = []; + if (keyPrefix) { + for (var i = 0; i < _keys.length; i++) { + if (minuteKeys[i] != _keys[i]) { + keyPrefix = false; + break; + } + final old = _orderedBills[i]; + if (!_same(old.hr, hr[i]) || !_same(old.cadence, cadenceSpm?[i])) { + changed.add(i); + } + } + } + Set? retained; + if (keyPrefix) { + final added = minuteKeys.length - _keys.length > 1 ? {} : null; + for (var i = _keys.length; i < minuteKeys.length; i++) { + if (_bills.containsKey(minuteKeys[i]) || + (added != null && !added.add(minuteKeys[i]))) { + throw ArgumentError('Minute keys must be unique'); + } + changed.add(i); + } + } else { + retained = minuteKeys.toSet(); + if (retained.length != minuteKeys.length) { + throw ArgumentError('Minute keys must be unique'); + } + } + final parameters = [ + restingHr == null ? null : _encode(restingHr), + maxHr == null ? null : _encode(maxHr), + sex.name, + profile == null + ? null + : [ + _encode(profile.weightKg), + _encode(profile.heightCm), + _encode(profile.age), + profile.sex + ], + ]; + final reprice = force || !_equalState(_parameters, parameters); + final trimpValid = restingHr != null && + maxHr != null && + restingHr.isFinite && + maxHr.isFinite && + maxHr > restingHr; + if (reprice || !_pricerReady) { + _energyPricer = profile == null || restingHr == null || maxHr == null + ? null + : Calories.minuteEnergyPricer( + profile: profile, hrmax: maxHr, restingHr: restingHr); + _basalPerMinute = _energyPricer?.basalKcalPerMin; + _pricerReady = true; + } + if (reprice) { + _bills.clear(); + _trimpTotal = 0; + _hrActiveTotal = _walkingTotal = 0; + } + if (!keyPrefix && !reprice) { + for (final key in _bills.keys.toList()) { + if (!retained!.contains(key)) { + final old = _bills.remove(key)!; + _trimpTotal -= old.trimp; + _hrActiveTotal -= old.active; + _walkingTotal -= old.walking; + } + } + } + final workIndices = + reprice || !keyPrefix ? Iterable.generate(hr.length) : changed; + for (final i in workIndices) { + final key = minuteKeys[i], value = hr[i]; + final cadence = cadenceSpm?[i]; + final old = _bills[key]; + if (old != null && _same(old.hr, value) && _same(old.cadence, cadence)) + continue; + var trimp = 0.0; + if (trimpValid && value.isFinite && value > 0) { + final hrr = ((value - restingHr) / (maxHr - restingHr)).clamp(0.0, 1.0); + trimp = hrr * StrainScorer.banisterY(hrr, female: sex == Sex.female); + } + MinuteEnergySource? source; + var active = 0.0, walking = 0.0; + if (_energyPricer != null) { + final bill = _energyPricer!.price(value, cadence); + source = bill.source; + active = bill.active; + walking = bill.walking; + } + _trimpTotal += trimp - (old?.trimp ?? 0); + _hrActiveTotal += active - (old?.active ?? 0); + _walkingTotal += walking - (old?.walking ?? 0); + final bill = _MinuteBill(value, cadence, trimp, source, active, walking); + _bills[key] = bill; + if (keyPrefix && !reprice) { + if (i < _orderedBills.length) { + _orderedBills[i] = bill; + } else { + _orderedBills.add(bill); + } + } + _processedMinutes++; + } + if (keyPrefix) { + for (var i = _keys.length; i < minuteKeys.length; i++) { + _keys.add(minuteKeys[i]); + } + } else { + _keys + ..clear() + ..addAll(minuteKeys); + } + if (!keyPrefix || reprice) { + _orderedBills + ..clear() + ..addAll([for (final key in minuteKeys) _bills[key]!]); + } + _parameters = parameters; + // Rounding after removals can leave tiny negative residue in a zero day. + if (_trimpTotal < 0 && _trimpTotal > -1e-9) _trimpTotal = 0; + if (_hrActiveTotal < 0 && _hrActiveTotal > -1e-9) _hrActiveTotal = 0; + if (_walkingTotal < 0 && _walkingTotal > -1e-9) _walkingTotal = 0; + if (_bills.isEmpty) { + _trimpTotal = _hrActiveTotal = _walkingTotal = 0; + } + final trimp = !trimpValid || hr.isEmpty + ? banisterTrimp(const [], restingHr: restingHr, maxHr: maxHr, sex: sex) + : Metric( + value: _trimpTotal, + confidence: .6, + tier: Tier.estimate, + inputs_used: const ['hr_per_min', 'resting_hr', 'max_hr'], + note: 'Banister exponential TRIMP (wrist HR estimate)'); + final strain = strainScoreMetric(trimp.value, + wakeMinutes: hr.length.toDouble(), + quietHrr: quietHrr, + quietSettled: quietSettled, + female: sex == Sex.female); + final minutes = !includeMinuteSeries || _basalPerMinute == null + ? null + : Calories.assembleMinuteEnergySeries([ + for (final key in minuteKeys) + ( + minute: key, + source: _bills[key]!.source, + active: _bills[key]!.active, + walking: _bills[key]!.walking + ), + ], basalKcalPerMin: _basalPerMinute!); + final basal = (_basalPerMinute ?? 0) * dayMinutes; + final active = minutes?.active ?? (_hrActiveTotal + _walkingTotal); + final walking = minutes?.walking ?? _walkingTotal; + return MinuteMetrics( + trimp: trimp, + strain: strain, + minutes: minutes, + energy: _basalPerMinute == null + ? null + : ( + total: basal + active, + active: active, + basal: basal, + walking: walking + )); + } + + Map toJson() => { + 'version': 1, + 'type': 'IncrementalMinuteMetrics', + 'processedMinutes': _processedMinutes, + 'parameters': _parameters == null + ? null + : [ + _parameters![0], + _parameters![1], + _parameters![2], + _parameters![3] == null + ? null + : List.of(_parameters![3] as List), + ], + 'trimpTotal': _trimpTotal, + 'hrActiveTotal': _hrActiveTotal, + 'walkingTotal': _walkingTotal, + 'basalPerMinute': _basalPerMinute, + 'bills': [ + for (final entry in _bills.entries) + { + 'key': entry.key, + 'hr': _encode(entry.value.hr), + 'cadence': entry.value.cadence == null + ? null + : _encode(entry.value.cadence!), + 'trimp': entry.value.trimp, + 'source': entry.value.source?.name, + 'active': entry.value.active, + 'walking': entry.value.walking, + } + ], + }; +} + +bool _equalState(Object? a, Object? b) { + if (a is List && b is List) { + if (a.length != b.length) return false; + for (var i = 0; i < a.length; i++) { + if (!_equalState(a[i], b[i])) return false; + } + return true; + } + return a == b; +} diff --git a/lib/src/onehz/incremental_state.dart b/lib/src/onehz/incremental_state.dart new file mode 100644 index 0000000..45453fb --- /dev/null +++ b/lib/src/onehz/incremental_state.dart @@ -0,0 +1,50 @@ +part of 'incremental.dart'; + +void _version(Map json, String type) { + if (json['version'] is! int || json['version'] != 1 || json['type'] != type) { + throw FormatException('Unsupported $type checkpoint'); + } +} + +int _count(Object? value) { + if (value is! int || value < 0) throw const FormatException('Invalid count'); + return value; +} + +double _number(Object? value, {bool finite = false}) { + if (value is num && (!finite || value.isFinite)) return value.toDouble(); + if (!finite) { + if (value == 'NaN') return double.nan; + if (value == 'Infinity') return double.infinity; + if (value == '-Infinity') return double.negativeInfinity; + } + throw const FormatException('Invalid number'); +} + +Object _encode(double value) => value.isFinite + ? value + : value.isNaN + ? 'NaN' + : value > 0 + ? 'Infinity' + : '-Infinity'; +List _numbers(Object? value, {bool finite = false}) { + if (value is! List) throw const FormatException('Expected numeric list'); + return [for (final item in value) _number(item, finite: finite)]; +} + +Map _map(Object? value) { + if (value is! Map) + throw const FormatException('Expected map'); + return value; +} + +bool _same(double? a, double? b) => + a == b || (a != null && b != null && a.isNaN && b.isNaN); +bool _prefix(List previous, List next) { + if (next.length < previous.length) return false; + for (var i = 0; i < previous.length; i++) { + if (!_same(previous[i], next[i])) return false; + } + return true; +} diff --git a/lib/src/onehz/registry/group_registry.dart b/lib/src/onehz/registry/group_registry.dart new file mode 100644 index 0000000..8082a02 --- /dev/null +++ b/lib/src/onehz/registry/group_registry.dart @@ -0,0 +1,404 @@ +// group_registry.dart — the calculation-group registry (design 06 phase 1b). +// +// Types moved verbatim from edge's lib/compute/groups/group_types.dart (same +// API). Pure Dart, const-constructible. test/registry_closure_test.dart pins +// the declared groups and their source closures. + +/// What a group is. Foundations carry no cost rule; edge transforms are the +/// gating/assembly steps edge runs around analytics results. +enum GroupKind { algorithm, foundation, edgeTransform } + +/// How far in time a declared input reaches. +enum InputScope { day, dayAndNext, range, global } + +class InputDecl { + /// An id from edge's input catalogue. + final String id; + final InputScope scope; + const InputDecl(this.id, this.scope); +} + +class GroupDecl { + /// Stable; never reused after a split or merge. + final String id; + final GroupKind kind; + + /// Plain integer, bumped when the group's algorithm changes, including + /// output rounding or abstention changes. Dependency-only changes propagate + /// through [deps]; they do not require another bump of this counter. + final int counter; + + /// Exported function names, at most 3. + final List members; + + /// Exported functions whose output depends on this group, beyond the + /// cost-bearing [members]. Uncapped (design 06 section 13); the cost rule + /// and the member cap look at [members] only. Each function is owned by + /// exactly one group, as a member or an api entry. + final List api; + + /// Group ids this group reads, foundations included. + final List deps; + final List inputs; + + /// Repo-relative paths (lib/src/...) this group's output depends on. + final List files; + + const GroupDecl({ + required this.id, + required this.kind, + required this.counter, + required this.members, + this.api = const [], + this.deps = const [], + this.inputs = const [], + this.files = const [], + }); +} + +class GroupRegistry { + final int registryVersion; + final List groups; + const GroupRegistry(this.registryVersion, this.groups); +} + +/// The groups analytics declares (registry schema version 1). +/// +/// `members` are the group's cost-bearing entry functions (at most 3); `api` +/// lists the other exported functions whose output depends on the group, so +/// every function edge calls has exactly one owner. `files` is the import +/// closure of members and api, minus the files of the foundations the group +/// depends on. Names only in lib/aggregate.dart (`prsaAggregator`) count as +/// public. test/registry_closure_test.dart re-derives all of this from source. +const GroupRegistry kAnalyticsGroupRegistry = GroupRegistry(1, [ + GroupDecl( + id: 'found.core', + kind: GroupKind.foundation, + counter: 2, + members: ['median'], + api: ['needBaselineNote'], + files: ['lib/src/onehz/types.dart', 'lib/src/onehz/util.dart'], + ), + GroupDecl( + id: 'found.rr_correction', + kind: GroupKind.foundation, + counter: 2, + members: ['correctRr'], + deps: ['found.core'], + files: [ + 'lib/src/onehz/foundations/rr_correction.dart', + 'lib/src/onehz/foundations/rr_correction_kernel.dart', + ], + ), + GroupDecl( + id: 'found.baseline_ewma', + kind: GroupKind.foundation, + counter: 1, + members: ['Baselines.foldHistory', 'Baselines.deviation'], + api: ['mdc'], + deps: ['found.core'], + files: [ + 'lib/src/onehz/foundations/baseline.dart', + 'lib/src/onehz/foundations/ewma_baselines.dart', + ], + ), + GroupDecl( + id: 'found.calibration', + kind: GroupKind.foundation, + counter: 1, + members: ['calibrationFor'], + api: ['unknownFamilyNote'], + files: ['lib/src/onehz/device.dart'], + ), + GroupDecl( + id: 'found.energy', + kind: GroupKind.foundation, + counter: 1, + members: ['Calories.dailyEnergy'], + api: ['Calories.minuteEnergy', 'Calories.estimateBoutCalories'], + files: ['lib/src/onehz/workout/calories.dart'], + ), + GroupDecl( + id: 'found.staging_helpers', + kind: GroupKind.foundation, + counter: 1, + members: ['consolidateSleepStages'], + deps: ['found.core'], + files: [ + 'lib/src/onehz/sleep/accounting.dart', + 'lib/src/onehz/sleep/stager.dart', + ], + ), + GroupDecl( + id: 'sleep.stage', + kind: GroupKind.algorithm, + counter: 2, + members: ['segmentSleep', 'stageMinuteRanges', 'cardioStager'], + api: [ + 'resetCardioObservations', + 'takeCardioObservations', + 'detectSleepCycles', + 'sleepCyclesMetric', + 'hrLedSleepWindow', + 'zAngle', + 'habitualMidsleepSecFromHistory', + 'phillipsSri', + 'sleepDebt', + 'sleepNeed', + 'sleepPerformance', + 'recommendedBedtime', + 'recommendedWake', + ], + deps: [ + 'found.core', + 'found.rr_correction', + 'found.staging_helpers', + 'found.calibration', + ], + files: [ + 'lib/src/aggregate/aggregator.dart', + 'lib/src/aggregate/json_support.dart', + 'lib/src/aggregate/primitives.dart', + 'lib/src/onehz/clinical/hrv_jitter_evidence.dart', + 'lib/src/onehz/clinical/hrv_time.dart', + 'lib/src/onehz/clinical/hrv_time_aggregator.dart', + 'lib/src/onehz/clinical/rmssd_gate.dart', + 'lib/src/onehz/human/coaching.dart', + 'lib/src/onehz/human/sleep_regularity.dart', + 'lib/src/onehz/incremental_core.dart', + 'lib/src/onehz/sleep/advanced_stager.dart', + 'lib/src/onehz/sleep/band_offset.dart', + 'lib/src/onehz/sleep/cardio_row_cache.dart', + 'lib/src/onehz/sleep/cardio_stager.dart', + 'lib/src/onehz/sleep/cpc.dart', + 'lib/src/onehz/sleep/cycles.dart', + 'lib/src/onehz/sleep/epoch_rows.dart', + 'lib/src/onehz/sleep/hr_fallback.dart', + 'lib/src/onehz/sleep/hypnogram.dart', + 'lib/src/onehz/sleep/hypnogram_aggregator.dart', + 'lib/src/onehz/sleep/segment.dart', + 'lib/src/onehz/sleep/sri.dart', + 'lib/src/onehz/sleep/sri_aggregator.dart', + 'lib/src/onehz/sleep/stage_ranges.dart', + 'lib/src/onehz/sleep/van_hees.dart', + ], + ), + GroupDecl( + id: 'hrv.freq', + kind: GroupKind.algorithm, + counter: 2, + members: ['hrvFreq'], + api: [ + 'nightHrvShape', + 'hrvTime', + 'nocturnalRmssd', + 'rrCoverage', + 'sleepSessionRmssdDetail', + ], + deps: ['found.core', 'found.rr_correction'], + files: [ + 'lib/src/aggregate/aggregator.dart', + 'lib/src/aggregate/json_support.dart', + 'lib/src/aggregate/primitives.dart', + 'lib/src/onehz/clinical/hrv_freq.dart', + 'lib/src/onehz/clinical/hrv_freq_stream.dart', + 'lib/src/onehz/clinical/hrv_jitter_evidence.dart', + 'lib/src/onehz/clinical/hrv_time.dart', + 'lib/src/onehz/clinical/hrv_time_aggregator.dart', + 'lib/src/onehz/clinical/rmssd_gate.dart', + 'lib/src/onehz/incremental_core.dart', + 'lib/src/onehz/sleep/night_hrv_shape.dart', + ], + ), + GroupDecl( + id: 'night.rhythm', + kind: GroupKind.algorithm, + counter: 2, + members: ['rsaRespRate', 'cvhrApneaScreen', 'decelerationCapacity'], + api: [ + 'accelerationCapacity', + 'breathingRateVariability', + 'irregularBeatScreenDetailed', + 'prsaAggregator', + 'cosinor', + 'circadianNonparametric', + 'chronotype', + 'socialJetlag', + ], + deps: ['found.core', 'found.rr_correction'], + files: [ + 'lib/src/aggregate/aggregator.dart', + 'lib/src/aggregate/json_support.dart', + 'lib/src/aggregate/primitives.dart', + 'lib/src/onehz/clinical/cardiac_coherence.dart', + 'lib/src/onehz/clinical/cosinor.dart', + 'lib/src/onehz/clinical/cosinor_aggregator.dart', + 'lib/src/onehz/clinical/irregular_diagnostics.dart', + 'lib/src/onehz/clinical/irregular_rhythm.dart', + 'lib/src/onehz/clinical/irregular_window.dart', + 'lib/src/onehz/clinical/prsa.dart', + 'lib/src/onehz/clinical/prsa_aggregator.dart', + 'lib/src/onehz/foundations/baseline.dart', + 'lib/src/onehz/foundations/fusion.dart', + 'lib/src/onehz/human/circadian_lifestyle.dart', + 'lib/src/onehz/human/percentile_of_you.dart', + 'lib/src/onehz/incremental_core.dart', + 'lib/src/onehz/respiration/brv_trend.dart', + 'lib/src/onehz/respiration/cvhr_apnea.dart', + 'lib/src/onehz/respiration/relative_odi.dart', + 'lib/src/onehz/respiration/resp_rate.dart', + 'lib/src/onehz/sleep/circadian_np.dart', + 'lib/src/onehz/sleep/circadian_np_aggregator.dart', + 'lib/src/onehz/sleep/circadian_window.dart', + ], + ), + GroupDecl( + id: 'day.blocks', + kind: GroupKind.algorithm, + counter: 1, + members: ['detectNaps', 'autoDetectWorkouts', 'positionSeries'], + api: ['hrRecovery', 'AutoWorkoutDetector.motionPoints'], + deps: ['found.core', 'sleep.stage', 'found.calibration'], + files: [ + 'lib/src/onehz/motion/orientation.dart', + 'lib/src/onehz/sleep/nap.dart', + 'lib/src/onehz/sleep/van_hees.dart', + 'lib/src/onehz/workout/auto_detect.dart', + 'lib/src/onehz/workout/hr_recovery.dart', + 'lib/src/onehz/workout/sport.dart', + ], + ), + GroupDecl( + id: 'readiness.stress_illness', + kind: GroupKind.algorithm, + counter: 1, + members: ['readinessComposite', 'baevskyStressIndex', 'illnessCusum'], + api: [ + 'readinessLnRmssd', + 'glassBoxReadiness', + 'hrvInput', + 'respInput', + 'rhrInput', + 'tempInput', + 'nocturnalRhr', + 'cusumChangePoints', + 'multivariateAnomaly', + 'overreachingConjunction', + 'tempIllnessFlag', + ], + deps: [ + 'found.core', + 'night.rhythm', + 'hrv.freq', + 'found.rr_correction', + 'found.baseline_ewma', + ], + files: [ + 'lib/src/aggregate/aggregator.dart', + 'lib/src/aggregate/json_support.dart', + 'lib/src/aggregate/primitives.dart', + 'lib/src/onehz/clinical/cosinor.dart', + 'lib/src/onehz/clinical/cosinor_aggregator.dart', + 'lib/src/onehz/clinical/illness_cusum.dart', + 'lib/src/onehz/clinical/load_trimp.dart', + 'lib/src/onehz/clinical/load_trimp_aggregator.dart', + 'lib/src/onehz/clinical/nocturnal.dart', + 'lib/src/onehz/clinical/readiness_lnrmssd.dart', + 'lib/src/onehz/clinical/stress_si.dart', + 'lib/src/onehz/device.dart', + 'lib/src/onehz/human/overreaching_conjunction.dart', + 'lib/src/onehz/human/readiness_glassbox.dart', + 'lib/src/onehz/incremental_core.dart', + 'lib/src/onehz/wellness/anomaly.dart', + 'lib/src/onehz/wellness/changepoint.dart', + 'lib/src/onehz/wellness/readiness_composite.dart', + 'lib/src/onehz/wellness/temp_circadian.dart', + 'lib/src/onehz/wellness/temp_health.dart', + ], + ), + GroupDecl( + id: 'activity.load', + kind: GroupKind.algorithm, + counter: 2, + members: ['banisterTrimp', 'dailyActiveMinutes', 'strainScore'], + api: [ + 'strainFromNetTrimp', + 'strainScoreFromSeries', + 'strainCurveFromSeries', + 'strainTarget', + 'ctlAtlTsb', + 'netTrimpAboveQuiet', + 'dailyQuietWakingHrr', + 'personalQuietWakingHrr', + 'dailyDynSummary', + 'personalDynFloorFromDailySummaries', + 'shouldRefreezeFloor', + 'calibrateGRef', + 'enmoSeries', + 'sessionHrCeiling', + 'HeartRateZones.timeInZone', + 'HeartRateZones.zonesFromMaxHr', + 'HeartRateZones.reserveZones', + ], + deps: [ + 'found.core', + 'found.energy', + 'found.calibration', + 'found.baseline_ewma', + ], + files: [ + 'lib/src/aggregate/aggregator.dart', + 'lib/src/aggregate/json_support.dart', + 'lib/src/aggregate/primitives.dart', + 'lib/src/onehz/clinical/load_trimp.dart', + 'lib/src/onehz/clinical/load_trimp_aggregator.dart', + 'lib/src/onehz/clinical/vo2max.dart', + 'lib/src/onehz/human/coaching.dart', + 'lib/src/onehz/incremental_core.dart', + 'lib/src/onehz/motion/energy_fusion.dart', + 'lib/src/onehz/motion/enmo.dart', + 'lib/src/onehz/motion/steps.dart', + 'lib/src/onehz/workout/hr_zones.dart', + 'lib/src/onehz/workout/observed_max_hr.dart', + 'lib/src/onehz/workout/time_in_zone_aggregator.dart', + ], + ), + GroupDecl( + id: 'body.spo2_temp_coverage', + kind: GroupKind.algorithm, + counter: 1, + members: ['hrDip', 'nightlySkinTemp'], + api: ['hrDipFromDayTotals'], + deps: ['found.core', 'found.baseline_ewma', 'found.calibration'], + files: [ + 'lib/src/aggregate/aggregator.dart', + 'lib/src/aggregate/json_support.dart', + 'lib/src/onehz/clinical/cosinor.dart', + 'lib/src/onehz/clinical/cosinor_aggregator.dart', + 'lib/src/onehz/clinical/nocturnal.dart', + 'lib/src/onehz/device.dart', + 'lib/src/onehz/wellness/temp_circadian.dart', + ], + ), + GroupDecl( + id: 'human.insights', + kind: GroupKind.algorithm, + counter: 1, + members: ['alertnessForecast', 'scanAssociations', 'alcoholNightFlag'], + api: [ + 'roughNight', + 'sessionMorningEffects', + 'weekdayEffect', + 'cycleLengthSeries', + ], + deps: ['found.core', 'found.baseline_ewma'], + files: [ + 'lib/src/onehz/human/alertness_forecast.dart', + 'lib/src/onehz/human/associations.dart', + 'lib/src/onehz/human/event_detection.dart', + 'lib/src/onehz/human/session_cost.dart', + 'lib/src/onehz/human/weekday_effect.dart', + 'lib/src/onehz/wellness/cycle_lengths.dart', + ], + ), +]); diff --git a/lib/src/onehz/respiration/cvhr_apnea.dart b/lib/src/onehz/respiration/cvhr_apnea.dart index 5eef9b2..5ae6711 100644 --- a/lib/src/onehz/respiration/cvhr_apnea.dart +++ b/lib/src/onehz/respiration/cvhr_apnea.dart @@ -35,6 +35,7 @@ import 'dart:math' as math; import '../types.dart'; import '../util.dart'; +import '../../aggregate/json_support.dart'; class CvhrResult { final int cycleCount; // detected CVHR cycles @@ -127,6 +128,16 @@ Metric cvhrApneaScreen( } final tSec = [for (final t in nnTimesMs) t / 1000.0]; + // A non-finite beat time has no place on the 1 Hz grid: NaN slipped past + // every gap test and then failed the resample, and an infinite one would be + // a stretch of its own. Same answer as rsaRespRate / hrvFreq. + if (tSec.any((t) => !t.isFinite)) { + return const Metric.absent( + tier: Tier.high, + inputs_used: inputs, + note: 'non-finite beat time', + ); + } if (tSec.last - tSec.first <= 0) { return const Metric.absent( tier: Tier.high, @@ -135,45 +146,68 @@ Metric cvhrApneaScreen( ); } - // SEGMENT AT GAPS. Beats more than [maxGapSec] apart are two recordings, not - // one: interpolating a straight line across the hole invents a flat stretch - // with no cycles in it, and charging the hole to `analyzedHours` divides a - // real cycle count by time we never observed. Both push the index DOWN, so a - // charging break used to read as an improvement. - final segStart = [0]; - for (var i = 1; i < tSec.length; i++) { - if (tSec[i] - tSec[i - 1] > maxGapSec) segStart.add(i); - } - - var cycleCount = 0; - var analyzedHours = 0.0; - final depths = []; // ms - final widths = []; // s - for (var s = 0; s < segStart.length; s++) { - final lo = segStart[s]; - final hi = (s + 1 < segStart.length ? segStart[s + 1] : tSec.length) - 1; - final segT = tSec.sublist(lo, hi + 1); - final segNn = nnMs.sublist(lo, hi + 1); - final spanSec = segT.last - segT.first; - // Too short to hold even one cycle plus its baseline window — it also can't - // be scored, so it contributes neither cycles nor hours. - if (segT.length < 60 || spanSec < 60) continue; - analyzedHours += spanSec / 3600.0; - _scoreSegment( - segT, - segNn, + return _cvhrMetric(_segmentContributions(tSec, nnMs, minWidthSec: minWidthSec, maxWidthSec: maxWidthSec, depthWidthRatioMin: depthWidthRatioMin, envWindowSec: envWindowSec, - onCycle: (depth, width) { - cycleCount++; + maxGapSec: maxGapSec), artifactFraction); +} + +class _CvhrContribution { + final double hours; + final List depths, widths; + const _CvhrContribution(this.hours, this.depths, this.widths); + + Map toJson() => { + 'hours': encodeDouble(hours), + 'depths': [for (final x in depths) encodeDouble(x)], + 'widths': [for (final x in widths) encodeDouble(x)], + }; +} + +List<_CvhrContribution> _segmentContributions(List t, List nn, + {required double minWidthSec, + required double maxWidthSec, + required double depthWidthRatioMin, + required double envWindowSec, + required double maxGapSec}) { + final out = <_CvhrContribution>[]; + var lo = 0; + for (var hi = 1; hi <= t.length; hi++) { + if (hi != t.length && t[hi] - t[hi - 1] <= maxGapSec) continue; + final segT = t.sublist(lo, hi); + final spanSec = segT.last - segT.first; + if (segT.length >= 60 && spanSec >= 60) { + final depths = []; + final widths = []; + _scoreSegment(segT, nn.sublist(lo, hi), + minWidthSec: minWidthSec, + maxWidthSec: maxWidthSec, + depthWidthRatioMin: depthWidthRatioMin, + envWindowSec: envWindowSec, + onCycle: (depth, width) { depths.add(depth); widths.add(width); - }, - ); + }); + out.add(_CvhrContribution(spanSec / 3600.0, depths, widths)); + } + lo = hi; } + return out; +} +Metric _cvhrMetric( + List<_CvhrContribution> contributions, double artifactFraction) { + const inputs = ['rr_cleaned', 'beat_times']; + var analyzedHours = 0.0; + final depths = []; + final widths = []; + for (final c in contributions) { + analyzedHours += c.hours; + depths.addAll(c.depths); + widths.addAll(c.widths); + } if (analyzedHours <= 0) { return const Metric.absent( tier: Tier.high, @@ -181,7 +215,7 @@ Metric cvhrApneaScreen( note: 'no gap-free stretch long enough for a CVHR screen (<60 s)', ); } - + final cycleCount = depths.length; final perHour = cycleCount / analyzedHours; final conf = ((1 - artifactFraction) * 0.85).clamp(0.2, 0.85); return Metric( @@ -203,6 +237,343 @@ Metric cvhrApneaScreen( ); } +/// Exact, append-only state for [cvhrApneaScreen]. Closed gap-free stretches +/// are reduced to their scored contributions; only the current stretch's beats +/// are retained for the provisional end-window calculation. +// Hayano 2011 CVHR screening, as in cvhrApneaScreen(). Closed gap-free stretches +// retain scored contributions; the current stretch is rescored provisionally +// with shared batch stages. +class CvhrStream { + final double minWidthSec, maxWidthSec, depthWidthRatioMin, envWindowSec, + maxGapSec; + final List<_CvhrContribution> _closed = []; + final List _rawNn = [], _rawTimes = []; + final List _grid = [], _smooth = [], _base = [], _exc = []; + int _n = 0; + int _openCount = 0; + double? _first, _last, _openStart; + + CvhrStream({ + this.minWidthSec = 10, + this.maxWidthSec = 120, + this.depthWidthRatioMin = 0.7, + this.envWindowSec = 130, + this.maxGapSec = 30, + }); + + factory CvhrStream.fromJson(Map json) { + try { + final j = decodeMap(json); + if (field(j, 'type') != 'CvhrStream') { + throw const FormatException('not a CvhrStream checkpoint'); + } + if (field(j, 'version') != 1) { + throw const FormatException('unsupported CvhrStream version'); + } + final minWidthSec = decodeDouble(field(j, 'minWidthSec')); + final maxWidthSec = decodeDouble(field(j, 'maxWidthSec')); + final depthWidthRatioMin = decodeDouble(field(j, 'depthWidthRatioMin')); + final envWindowSec = decodeDouble(field(j, 'envWindowSec')); + final maxGapSec = decodeDouble(field(j, 'maxGapSec')); + final n = decodeCount(field(j, 'n')); + final first = decodeDoubleOrNull(field(j, 'first')); + final last = decodeDoubleOrNull(field(j, 'last')); + final openStart = decodeDoubleOrNull(field(j, 'openStart')); + final openCount = decodeCount(field(j, 'openCount')); + final rawNn = decodeDoubles(field(j, 'rawNn')); + final rawTimes = decodeDoubles(field(j, 'rawTimes')); + final grid = decodeDoubles(field(j, 'grid')); + final smooth = decodeDoubles(field(j, 'smooth')); + final base = decodeDoubles(field(j, 'base')); + final exc = decodeDoubles(field(j, 'exc')); + final encodedClosed = decodeList(field(j, 'closed')); + if (!minWidthSec.isFinite || !maxWidthSec.isFinite || + !depthWidthRatioMin.isFinite || !envWindowSec.isFinite || + !maxGapSec.isFinite || rawNn.length != rawTimes.length || + smooth.length > grid.length || base.length > smooth.length || + exc.length != base.length || n < openCount || + (n == 0 && (first != null || last != null || openStart != null || + openCount != 0 || rawNn.isNotEmpty || grid.isNotEmpty)) || + (n > 0 && (first == null || last == null || openStart == null || + openCount == 0 || rawNn.isEmpty))) { + throw const FormatException('inconsistent CvhrStream checkpoint'); + } + final s = CvhrStream( + minWidthSec: minWidthSec, + maxWidthSec: maxWidthSec, + depthWidthRatioMin: depthWidthRatioMin, + envWindowSec: envWindowSec, + maxGapSec: maxGapSec, + ) + .._n = n + .._first = first + .._last = last + .._openStart = openStart + .._openCount = openCount + .._rawNn.addAll(rawNn) + .._rawTimes.addAll(rawTimes) + .._grid.addAll(grid) + .._smooth.addAll(smooth) + .._base.addAll(base) + .._exc.addAll(exc); + if (n > 0 && + (s._rawTimes.isEmpty || !_cvhrNondecreasing(s._rawTimes))) { + throw const FormatException('decreasing checkpoint times'); + } + for (final value in encodedClosed) { + final c = decodeMap(value); + final hours = decodeDouble(field(c, 'hours')); + final depths = decodeDoubles(field(c, 'depths')); + final widths = decodeDoubles(field(c, 'widths')); + if (!hours.isFinite || hours < 0 || depths.length != widths.length) { + throw const FormatException('invalid CVHR segment'); + } + s._closed.add(_CvhrContribution(hours, depths.toList(), widths.toList())); + } + return s; + } on FormatException { + rethrow; + } catch (e) { + throw FormatException('malformed CvhrStream checkpoint: $e'); + } + } + + Map toJson() => { + 'type': 'CvhrStream', + 'version': 1, + 'minWidthSec': encodeDouble(minWidthSec), + 'maxWidthSec': encodeDouble(maxWidthSec), + 'depthWidthRatioMin': encodeDouble(depthWidthRatioMin), + 'envWindowSec': encodeDouble(envWindowSec), + 'maxGapSec': encodeDouble(maxGapSec), + 'n': _n, + 'first': _first == null ? null : encodeDouble(_first!), + 'last': _last == null ? null : encodeDouble(_last!), + 'openStart': _openStart == null ? null : encodeDouble(_openStart!), + 'openCount': _openCount, + 'rawNn': [for (final x in _rawNn) encodeDouble(x)], + 'rawTimes': [for (final x in _rawTimes) encodeDouble(x)], + 'grid': [for (final x in _grid) encodeDouble(x)], + 'smooth': [for (final x in _smooth) encodeDouble(x)], + 'base': [for (final x in _base) encodeDouble(x)], + 'exc': [for (final x in _exc) encodeDouble(x)], + 'closed': [for (final x in _closed) x.toJson()], + }; + + void fold(List nnMs, List nnTimesMs) { + if (nnMs.length != nnTimesMs.length) { + throw ArgumentError('nnMs and nnTimesMs must have the same length'); + } + if (nnMs.isEmpty) return; + final times = [for (final t in nnTimesMs) t / 1000.0]; + // Checked before any state changes (a fold cannot be undone): the batch + // abstains on such a time, so no snapshot after it could be the batch's. + if (times.any((t) => !t.isFinite)) { + throw ArgumentError('beat times must be finite'); + } + if (!_cvhrNondecreasing(times) || + (_last != null && times.first < _last!)) { + throw StateError('beat times must be non-decreasing'); + } + _first ??= times.first; + for (var i = 0; i < times.length; i++) { + if (_openCount > 0 && times[i] - _last! > maxGapSec) { + _closeOpen(); + } + _openStart ??= times[i]; + _openCount++; + _rawNn.add(nnMs[i]); + _rawTimes.add(times[i]); + _last = times[i]; + _settleOpen(); + } + _n += nnMs.length; + } + + Metric snapshot({ + List tailNnMs = const [], + List tailNnTimesMs = const [], + required double artifactFraction, + double maxArtifact = 0.30, + }) { + if (tailNnMs.length != tailNnTimesMs.length) { + throw ArgumentError('tailNnMs and tailNnTimesMs must have the same length'); + } + final tailTimes = [for (final t in tailNnTimesMs) t / 1000.0]; + if (!_cvhrNondecreasing(tailTimes) || + (tailTimes.isNotEmpty && _last != null && tailTimes.first < _last!)) { + throw StateError('beat times must be non-decreasing'); + } + const inputs = ['rr_cleaned', 'beat_times']; + final n = _n + tailNnMs.length; + if (n < 60) { + return const Metric.absent( + tier: Tier.high, + inputs_used: inputs, + note: 'too few beats for a CVHR screen (need ≥60)', + ); + } + if (artifactFraction > maxArtifact) { + return Metric.absent( + tier: Tier.high, + inputs_used: inputs, + note: 'artifact fraction ${round6(artifactFraction)} > gate ' + '— CVHR dip detection unreliable', + ); + } + // The folded times are finite (fold refuses others); a non-finite tail + // time gets the batch's answer. + if (tailTimes.any((t) => !t.isFinite)) { + return const Metric.absent( + tier: Tier.high, + inputs_used: inputs, + note: 'non-finite beat time', + ); + } + // Nothing folded yet: the tail's first beat is the origin, as in the batch. + final first = _first ?? (tailTimes.isEmpty ? null : tailTimes.first); + final last = tailTimes.isEmpty ? _last : tailTimes.last; + if (first == null || last == null || last - first <= 0) { + return const Metric.absent( + tier: Tier.high, + inputs_used: inputs, + note: 'degenerate beat times', + ); + } + return _cvhrMetric(_snapshotContributions(tailNnMs, tailTimes), + artifactFraction); + } + + void _closeOpen() { + final c = _openContribution(); + if (c != null) _closed.add(c); + _rawNn.clear(); + _rawTimes.clear(); + _grid.clear(); + _smooth.clear(); + _base.clear(); + _exc.clear(); + _openCount = 0; + _openStart = null; + } + + void _settleOpen() { + final start = _openStart!; + while (start + _grid.length < _rawTimes.last) { + final g = start + _grid.length; + _grid.add(_resampleLinear(_rawTimes, _rawNn, [g])[0]); + } + while (_smooth.length + 7 < _grid.length) { + _smooth.add(_savgolAt(_grid, _smooth.length, 7)); + } + final half = (envWindowSec / 2).round(); + while (_base.length + half < _smooth.length) { + final i = _base.length; + final b = _medianAt(_smooth, i, half); + _base.add(b); + _exc.add(math.max(0.0, _smooth[i] - b)); + } + var keep = 0; + final lastGrid = start + _grid.length - 1; + while (keep + 1 < _rawTimes.length && _rawTimes[keep + 1] <= lastGrid) { + keep++; + } + if (keep > 0) { + _rawTimes.removeRange(0, keep); + _rawNn.removeRange(0, keep); + } + } + + List<_CvhrContribution> _snapshotContributions( + List tailNn, List tailTimes) { + final out = <_CvhrContribution>[..._closed]; + if (_openCount == 0) { + if (tailTimes.isNotEmpty) { + out.addAll(_segmentContributions(tailTimes, tailNn, + minWidthSec: minWidthSec, + maxWidthSec: maxWidthSec, + depthWidthRatioMin: depthWidthRatioMin, + envWindowSec: envWindowSec, + maxGapSec: maxGapSec)); + } + return out; + } + var split = 0; + var previous = _last!; + while (split < tailTimes.length && tailTimes[split] - previous <= maxGapSec) { + previous = tailTimes[split++]; + } + final open = _openContribution(tailNn.sublist(0, split), + tailTimes.sublist(0, split)); + if (open != null) out.add(open); + if (split < tailTimes.length) { + out.addAll(_segmentContributions(tailTimes.sublist(split), tailNn.sublist(split), + minWidthSec: minWidthSec, + maxWidthSec: maxWidthSec, + depthWidthRatioMin: depthWidthRatioMin, + envWindowSec: envWindowSec, + maxGapSec: maxGapSec)); + } + return out; + } + + _CvhrContribution? _openContribution( + [List tailNn = const [], List tailTimes = const []]) { + final start = _openStart!; + final count = _openCount + tailNn.length; + final last = tailTimes.isEmpty ? _last! : tailTimes.last; + if (count < 60 || last - start < 60) return null; + final exc = _openExcursion(tailNn, tailTimes); + final depths = []; + final widths = []; + _cvhrCycles(exc, + minWidthSec: minWidthSec, + maxWidthSec: maxWidthSec, + depthWidthRatioMin: depthWidthRatioMin, + onCycle: (depth, width) { + depths.add(depth); + widths.add(width); + }); + return _CvhrContribution((last - start) / 3600.0, depths, widths); + } + + List _openExcursion(List tailNn, List tailTimes) { + final rawNn = [..._rawNn, ...tailNn]; + final rawTimes = [..._rawTimes, ...tailTimes]; + final grid = [..._grid]; + final start = _openStart!; + final last = rawTimes.last; + while (start + grid.length < last) { + grid.add(_resampleLinear(rawTimes, rawNn, [start + grid.length])[0]); + } + final nGrid = (last - start).floor() + 1; + while (grid.length < nGrid) { + grid.add(_resampleLinear(rawTimes, rawNn, [start + grid.length])[0]); + } + final smooth = [..._smooth]; + while (smooth.length < grid.length) { + smooth.add(_savgolAt(grid, smooth.length, 7)); + } + final half = (envWindowSec / 2).round(); + final base = [..._base]; + while (base.length < smooth.length) { + base.add(_medianAt(smooth, base.length, half)); + } + return [ + ..._exc, + for (var i = _exc.length; i < smooth.length; i++) + math.max(0.0, smooth[i] - base[i]), + ]; + } +} + +bool _cvhrNondecreasing(List values) { + for (var i = 1; i < values.length; i++) { + if (values[i] < values[i - 1]) return false; + } + return true; +} + // ───────────────────────────────────────────────────────────────────────────── // RESP-01 — the 30-night PERSONAL DISTRIBUTION. // @@ -414,6 +785,16 @@ void _scoreSegment( required double envWindowSec, required void Function(double depthMs, double widthSec) onCycle, }) { + final exc = _cvhrExcursion(t, nn, envWindowSec); + _cvhrCycles(exc, + minWidthSec: minWidthSec, + maxWidthSec: maxWidthSec, + depthWidthRatioMin: depthWidthRatioMin, + onCycle: onCycle); +} + +List _cvhrExcursion( + List t, List nn, double envWindowSec) { // Resample NN onto a uniform 1 Hz grid by piecewise-linear interpolation of // the tachogram (NN vs beat time). This gives evenly-spaced points for the // sliding-window baseline and the polynomial smoother. Only ever called on a @@ -433,18 +814,22 @@ void _scoreSegment( // The CVHR bradycardia shows up as a local NN MAXIMUM (NN ↑ ⇔ HR ↓) standing // proud of this baseline; the following tachycardia is the recovery valley. final halfEnv = (envWindowSec / 2).round(); - final base = List.filled(nGrid, 0); - for (var j = 0; j < nGrid; j++) { - final lo = math.max(0, j - halfEnv); - final hi = math.min(nGrid - 1, j + halfEnv); - base[j] = median(smooth.sublist(lo, hi + 1)) ?? smooth[j]; - } + final base = slidingMedians(smooth, halfEnv); // Baseline-subtracted bradycardic excursion (clip negatives — we only score // the NN-up bradycardia, not the tachycardia trough). final exc = [ for (var j = 0; j < nGrid; j++) math.max(0.0, smooth[j] - base[j]) ]; + return exc; +} + +void _cvhrCycles(List exc, + {required double minWidthSec, + required double maxWidthSec, + required double depthWidthRatioMin, + required void Function(double depthMs, double widthSec) onCycle}) { + final nGrid = exc.length; // Prominence threshold: a dip must clear a robust noise floor AND a fraction // of the typical excursion amplitude. We anchor it BELOW the median positive // excursion (Hayano's adaptive amplitude criterion is permissive enough to @@ -463,13 +848,15 @@ void _scoreSegment( // the run width and the peak depth; apply the width + depth/width gates. var i = 0; while (i < nGrid) { - if (exc[i] <= prom) { + // `!(x > prom)` rather than `x <= prom`: a NaN excursion is neither, and + // would stall this loop. A non-finite excursion never starts a dip. + if (!_above(exc[i], prom)) { i++; continue; } final start = i; var peakDepth = 0.0; - while (i < nGrid && exc[i] > prom) { + while (i < nGrid && _above(exc[i], prom)) { if (exc[i] > peakDepth) peakDepth = exc[i]; i++; } @@ -482,6 +869,10 @@ void _scoreSegment( } } +/// True when [e] is a finite excursion above [prom]. NaN and +-Infinity never +/// are, so they can neither start nor extend a dip. +bool _above(double e, double prom) => e.isFinite && e > prom; + /// Piecewise-linear resample of (t,y) onto a sorted [grid] of times (same unit). /// Out-of-range grid points clamp to the nearest endpoint value. List _resampleLinear( @@ -515,38 +906,43 @@ List _savgolQuadratic(List x, {int halfWin = 7}) { final n = x.length; final out = List.filled(n, 0); for (var i = 0; i < n; i++) { - final lo = math.max(0, i - halfWin); - final hi = math.min(n - 1, i + halfWin); - final m = hi - lo + 1; - if (m < 3) { - out[i] = x[i]; - continue; - } - // Fit y = a + b·u + c·u² with u centred at i; return a (value at u=0). - var s0 = 0.0, s1 = 0.0, s2 = 0.0, s3 = 0.0, s4 = 0.0; - var sy = 0.0, suy = 0.0, su2y = 0.0; - for (var k = lo; k <= hi; k++) { - final u = (k - i).toDouble(); - final u2 = u * u; - s0 += 1; - s1 += u; - s2 += u2; - s3 += u2 * u; - s4 += u2 * u2; - sy += x[k]; - suy += u * x[k]; - su2y += u2 * x[k]; - } - // Solve the 3x3 normal equations for [a,b,c]; we only need a. - final a = _solve3( - [s0, s1, s2, s1, s2, s3, s2, s3, s4], - [sy, suy, su2y], - ); - out[i] = a == null ? x[i] : a[0]; + out[i] = _savgolAt(x, i, halfWin); } return out; } +double _savgolAt(List x, int i, int halfWin) { + final lo = math.max(0, i - halfWin); + final hi = math.min(x.length - 1, i + halfWin); + final m = hi - lo + 1; + if (m < 3) return x[i]; + var s0 = 0.0, s1 = 0.0, s2 = 0.0, s3 = 0.0, s4 = 0.0; + var sy = 0.0, suy = 0.0, su2y = 0.0; + for (var k = lo; k <= hi; k++) { + final u = (k - i).toDouble(); + final u2 = u * u; + s0 += 1; + s1 += u; + s2 += u2; + s3 += u2 * u; + s4 += u2 * u2; + sy += x[k]; + suy += u * x[k]; + su2y += u2 * x[k]; + } + final a = _solve3( + [s0, s1, s2, s1, s2, s3, s2, s3, s4], + [sy, suy, su2y], + ); + return a == null ? x[i] : a[0]; +} + +double _medianAt(List x, int i, int halfWin) { + final lo = math.max(0, i - halfWin); + final hi = math.min(x.length - 1, i + halfWin); + return slidingMedians(x.sublist(lo, hi + 1), hi - lo + 1)[0]; +} + /// Solve a 3x3 linear system (row-major A, rhs b) via Cramer's rule. Null if /// singular. List? _solve3(List a, List b) { diff --git a/lib/src/onehz/respiration/cvhr_stream.dart b/lib/src/onehz/respiration/cvhr_stream.dart new file mode 100644 index 0000000..1421df2 --- /dev/null +++ b/lib/src/onehz/respiration/cvhr_stream.dart @@ -0,0 +1,3 @@ +/// Streaming CVHR is implemented beside its batch stages so both paths share +/// the exact scorer. +export 'cvhr_apnea.dart' show CvhrStream; diff --git a/lib/src/onehz/respiration/resp_rate.dart b/lib/src/onehz/respiration/resp_rate.dart index 7b5b7dc..4020de6 100644 --- a/lib/src/onehz/respiration/resp_rate.dart +++ b/lib/src/onehz/respiration/resp_rate.dart @@ -38,6 +38,7 @@ import 'dart:math' as math; import '../types.dart'; import '../util.dart'; import '../foundations/fusion.dart'; +import '../../aggregate/json_support.dart'; /// Hard physiological + Nyquist band for adult respiration on a 1 Hz signal. /// Lower 0.1 Hz = 6 br/min; upper 0.5 Hz = 30 br/min (the 1 Hz Nyquist limit). @@ -187,6 +188,15 @@ Metric rsaRespRate( ); } final tSec = [for (final t in nnTimesMs) t / 1000.0]; + // A non-finite beat time has no place on the clock, and an infinite last + // time made the sub-window walk below endless (it never reaches infinity). + if (tSec.any((t) => !t.isFinite)) { + return const Metric.absent( + tier: Tier.high, + inputs_used: inputs, + note: 'non-finite beat time', + ); + } final spanSec = tSec.last - tSec.first; if (spanSec <= 0) { return const Metric.absent( @@ -242,71 +252,104 @@ Metric rsaRespRate( 'respiratory peak against itself over time', ); } - final peaks = []; // br/min - final peakHz = []; // the same peaks in Hz, index-aligned - final peakPwr = []; // their spectral power, index-aligned - var atCeiling = 0; + final segments = <_RsaSegmentResult>[]; + for (var i = 0;; i++) { + final s = tSec.first + i * (segSec / 2); + if (s + segSec > tSec.last + 1e-9) break; + segments.add(_rsaSegment(tSec, nnMs, s, segSec)); + } + return _rsaMetricFromSegments(segments, + artifactFraction: artifactFraction, + tolBrpm: tolBrpm, + minConsensus: minConsensus, + segSec: segSec, + hiHz: hiHz); +} + +class _RsaSegmentResult { + // 0 = usable peak, 1 = ceiling, 2 = below HF band, 3 = thin, 4 = gappy + // (kept beats cover < 80 % of the sub-window). + final int kind; + final double? brpm, peakHz, power; + // kind 1 only: the sub-window's own ceiling, br/min. + final double? ceilingBrpm; + const _RsaSegmentResult(this.kind, + [this.brpm, this.peakHz, this.power, this.ceilingBrpm]); + + Map toJson() => { + 'kind': kind, + 'brpm': brpm == null ? null : encodeDouble(brpm!), + 'peakHz': peakHz == null ? null : encodeDouble(peakHz!), + 'power': power == null ? null : encodeDouble(power!), + 'ceiling': ceilingBrpm == null ? null : encodeDouble(ceilingBrpm!), + }; +} + +_RsaSegmentResult _rsaSegment( + List tSec, List nnMs, double start, double segSec) { + final lo = _lowerBound(tSec, start); + final hi = _lowerBound(tSec, start + segSec); + final k = hi - lo; + // A sub-window has to be BOTH time-complete and beat-dense: a dropout in + // the middle leaves few beats spanning the full 5 minutes, and its + // periodogram is a window function, not a spectrum. + final segT = tSec.sublist(lo, hi); + final segNn = nnMs.sublist(lo, hi); + // TIME-COMPLETE = the kept beats COVER ≥ 80 % of the sub-window. Σ NN, not + // first-to-last span: two beats near the edges with a 3-minute hole between + // them passed the span test and handed Lomb–Scargle a window function. + // Judged first, so a sub-window a sensor gap emptied is named a gap. + final coveredSec = segNn.fold(0, (a, b) => a + b) / 1000.0; + if (coveredSec < segSec * 0.8) return const _RsaSegmentResult(4); + if (k < 30) return const _RsaSegmentResult(3); + final segSpan = segT.last - segT.first; // still sets the grid resolution + // Per sub-window Nyquist from its median beat interval, gap-free, like the + // whole-input check. + final segHi = rsaCeilingHz(median(segNn)! / 1000.0); + if (segHi < rsaHiHz) return const _RsaSegmentResult(2); + final grid = math.max(64, ((segHi - rsaLoHz) * 4 * segSpan).ceil()); + final ls = lombScargle(segT, segNn, freqGrid(rsaLoHz, segHi, grid)); + if (ls == null) return const _RsaSegmentResult(3); + final pk = ls.peakFreq(rsaLoHz, segHi); + if (pk == null) return const _RsaSegmentResult(3); + if (pk >= segHi - (segHi - rsaLoHz) / (grid - 1)) { + return _RsaSegmentResult(1, null, null, null, segHi * 60); + } + return _RsaSegmentResult(0, pk * 60.0, pk, _powerAt(ls, pk)); +} + +Metric _rsaMetricFromSegments( + List<_RsaSegmentResult> segments, { + required double artifactFraction, + required double tolBrpm, + required double minConsensus, + required double segSec, + required double hiHz, +}) { + const inputs = ['rr_cleaned', 'beat_times']; + final peaks = []; + final peakHz = []; + final peakPwr = []; + var atCeiling = 0, belowBand = 0, thin = 0, gappy = 0; // The ceilings those sub-windows peaked at, br/min — their own, which // move with heart rate; not the whole input's. var ceilingLo = double.infinity, ceilingHi = 0.0; - var belowBand = 0; - var thin = 0; - var gappy = 0; - for (var s = tSec.first; s + segSec <= tSec.last + 1e-9; s += segSec / 2) { - final lo = _lowerBound(tSec, s); - final hi = _lowerBound(tSec, s + segSec); - final k = hi - lo; - // A sub-window has to be BOTH time-complete and beat-dense: a dropout in - // the middle leaves few beats spanning the full 5 minutes, and its - // periodogram is a window function, not a spectrum. - final segT = tSec.sublist(lo, hi); - final segNn = nnMs.sublist(lo, hi); - // TIME-COMPLETE = the kept beats COVER ≥ 80 % of the sub-window. Σ NN, not - // first-to-last span: two beats near the edges with a 3-minute hole between - // them passed the span test and handed Lomb–Scargle a window function. - // Judged first, so a sub-window a sensor gap emptied is named a gap. - final coveredSec = segNn.fold(0, (a, b) => a + b) / 1000.0; - if (coveredSec < segSec * 0.8) { - gappy++; - continue; - } - if (k < 30) { - thin++; - continue; - } - final segSpan = segT.last - segT.first; // still sets the grid resolution - // Per sub-window Nyquist: heart rate moves through the night, so the - // resolvable ceiling does too. A sub-window whose beat rate cannot cover - // the HF band is dropped for the SAME alias reason as the whole window — - // from its median beat interval, gap-free, like the whole-input check. - final segHi = rsaCeilingHz(median(segNn)! / 1000.0); - if (segHi < rsaHiHz) { + for (final segment in segments) { + if (segment.kind == 0) { + peaks.add(segment.brpm!); + peakHz.add(segment.peakHz!); + peakPwr.add(segment.power!); + } else if (segment.kind == 1) { + atCeiling++; + ceilingLo = math.min(ceilingLo, segment.ceilingBrpm!); + ceilingHi = math.max(ceilingHi, segment.ceilingBrpm!); + } else if (segment.kind == 2) { belowBand++; - continue; - } - final grid = math.max(64, ((segHi - rsaLoHz) * 4 * segSpan).ceil()); - final ls = lombScargle(segT, segNn, freqGrid(rsaLoHz, segHi, grid)); - if (ls == null) { - thin++; - continue; - } - final pk = ls.peakFreq(rsaLoHz, segHi); - if (pk == null) { + } else if (segment.kind == 4) { + gappy++; + } else { thin++; - continue; } - // A peak pinned to the top of the searchable band means the true rate is - // at or above what this sub-window can resolve. That is an ABSENCE, not a - // rate: reporting the edge would publish the ceiling as a measurement. - if (pk >= segHi - (segHi - rsaLoHz) / (grid - 1)) { - atCeiling++; - ceilingLo = math.min(ceilingLo, segHi * 60); - ceilingHi = math.max(ceilingHi, segHi * 60); - continue; - } - peaks.add(pk * 60.0); - peakHz.add(pk); - peakPwr.add(_powerAt(ls, pk)); } final dropped = atCeiling + belowBand + thin + gappy; final total = peaks.length + dropped; @@ -404,6 +447,265 @@ Metric rsaRespRate( ); } +/// Exact, append-only Welch state for [rsaRespRate]. +/// +/// The whole-input ceiling comes from the MEDIAN beat interval, so every folded +/// NN is kept once more in [_sorted] ([double.compareTo] order, as `median` +/// sorts); a snapshot merges the tail in. Checkpoint version 2 carries it; a +/// version 1 checkpoint (span-based ceiling, no gappy sub-windows) is refused. +// Welch 1967; Lomb 1976; Scargle 1982; Press & Rybicki 1989; DeBoer, Karemaker & +// Strackee 1984, as in rsaRespRate(). Shared segment scoring, retained exact NN +// median and provisional windows reproduce batch. +class RsaRespStream { + final List _nn = [], _times = []; + List _sorted = []; + final List<_RsaSegmentResult> _closed = []; + int _n = 0; + double? _first, _last, _nextStart; + + RsaRespStream(); + + factory RsaRespStream.fromJson(Map json) { + try { + final j = decodeMap(json); + if (field(j, 'type') != 'RsaRespStream') { + throw const FormatException('not an RsaRespStream checkpoint'); + } + if (field(j, 'version') != 2) { + throw const FormatException('unsupported RsaRespStream version'); + } + final n = decodeCount(field(j, 'n')); + final first = decodeDoubleOrNull(field(j, 'first')); + final last = decodeDoubleOrNull(field(j, 'last')); + final next = decodeDoubleOrNull(field(j, 'nextStart')); + final nn = decodeDoubles(field(j, 'nn')); + final times = decodeDoubles(field(j, 'times')); + final encoded = decodeList(field(j, 'closed')); + final sorted = decodeDoubles(field(j, 'sorted')); + if (nn.length != times.length || + sorted.length != n || + !_nondecreasingBy(sorted) || + (n == 0 && (first != null || last != null || next != null)) || + (n > 0 && (first == null || last == null || next == null))) { + throw const FormatException('inconsistent RsaRespStream checkpoint'); + } + final s = RsaRespStream() + .._n = n + .._first = first + .._last = last + .._nextStart = next + .._nn.addAll(nn) + .._times.addAll(times) + .._sorted = sorted; + if (!_nondecreasing(s._times)) { + throw const FormatException('decreasing checkpoint times'); + } + for (final value in encoded) { + final r = decodeMap(value); + final kind = field(r, 'kind'); + if (kind is! int || kind < 0 || kind > 4) { + throw const FormatException('invalid RSA segment'); + } + final brpm = decodeDoubleOrNull(field(r, 'brpm')); + final peakHz = decodeDoubleOrNull(field(r, 'peakHz')); + final power = decodeDoubleOrNull(field(r, 'power')); + final ceiling = decodeDoubleOrNull(field(r, 'ceiling')); + if ((kind == 0) != (brpm != null && peakHz != null && power != null) || + (kind == 1) != (ceiling != null)) { + throw const FormatException('invalid RSA segment'); + } + s._closed.add(_RsaSegmentResult(kind, brpm, peakHz, power, ceiling)); + } + return s; + } on FormatException { + rethrow; + } catch (e) { + throw FormatException('malformed RsaRespStream checkpoint: $e'); + } + } + + Map toJson() => { + 'type': 'RsaRespStream', + 'version': 2, + 'n': _n, + 'first': _first == null ? null : encodeDouble(_first!), + 'last': _last == null ? null : encodeDouble(_last!), + 'nextStart': _nextStart == null ? null : encodeDouble(_nextStart!), + 'nn': [for (final x in _nn) encodeDouble(x)], + 'times': [for (final x in _times) encodeDouble(x)], + 'closed': [for (final x in _closed) x.toJson()], + 'sorted': [for (final x in _sorted) encodeDouble(x)], + }; + + void fold(List nnMs, List nnTimesMs) { + if (nnMs.length != nnTimesMs.length) { + throw ArgumentError('nnMs and nnTimesMs must have the same length'); + } + if (nnMs.isEmpty) return; + final times = [for (final t in nnTimesMs) t / 1000.0]; + // Checked before any state changes: an infinite time would make [_settle] + // endless, and a fold cannot be undone. + if (times.any((t) => !t.isFinite)) { + throw ArgumentError('beat times must be finite'); + } + if (!_nondecreasing(times) || + (_last != null && times.first < _last!)) { + throw StateError('beat times must be non-decreasing'); + } + _first ??= times.first; + _nextStart ??= _first; + _nn.addAll(nnMs); + _times.addAll(times); + _sorted = _mergeSorted(_sorted, [...nnMs]..sort()); + _n += nnMs.length; + _last = times.last; + final span = _last! - _first!; + if (span >= 2 * rsaSegmentSec) { + _settle(); + _compact(); + } + } + + Metric snapshot({ + List tailNnMs = const [], + List tailNnTimesMs = const [], + required double artifactFraction, + double tolBrpm = 2.0, + double minConsensus = 0.5, + double maxArtifact = 0.30, + }) { + if (tailNnMs.length != tailNnTimesMs.length) { + throw ArgumentError('tailNnMs and tailNnTimesMs must have the same length'); + } + final tail = [for (final t in tailNnTimesMs) t / 1000.0]; + if (!_nondecreasing(tail) || + (tail.isNotEmpty && _last != null && tail.first < _last!)) { + throw StateError('beat times must be non-decreasing'); + } + final n = _n + tailNnMs.length; + // A non-finite tail time: the batch's own answer (it abstains). + if (n < 20 || _first == null || tail.any((t) => !t.isFinite)) { + return rsaRespRate([..._nn, ...tailNnMs], + [..._times.map((x) => x * 1000.0), ...tailNnTimesMs], + artifactFraction: artifactFraction, + tolBrpm: tolBrpm, + minConsensus: minConsensus, + maxArtifact: maxArtifact); + } + if (artifactFraction > maxArtifact) { + return Metric.absent( + tier: Tier.high, + inputs_used: const ['rr_cleaned', 'beat_times'], + note: 'artifact fraction ${round6(artifactFraction)} > gate ' + '— RSA peak unreliable'); + } + final times = [..._times, ...tail]; + final nn = [..._nn, ...tailNnMs]; + final last = times.last; + final span = last - _first!; + if (span <= 0) { + return rsaRespRate(nn, [for (final t in times) t * 1000.0], + artifactFraction: artifactFraction, + tolBrpm: tolBrpm, + minConsensus: minConsensus, + maxArtifact: maxArtifact); + } + // Same median the batch takes over settled ++ tail. + final beatSec = + percentileSorted(_mergeSorted(_sorted, [...tailNnMs]..sort()), 50)! / + 1000.0; + final hiHz = rsaCeilingHz(beatSec); + if (hiHz < rsaHiHz) { + return Metric.absent( + tier: Tier.high, + inputs_used: const ['rr_cleaned', 'beat_times'], + note: 'sleeping heart rate ${round6(60 / beatSec)} bpm (median beat ' + 'interval ${round6(beatSec)} s) can resolve breathing from beat ' + 'timing only up to ${round6(hiHz * 60)} br/min, below the top of ' + 'the normal 9–24 br/min band, so any peak could be an alias; rate ' + 'withheld'); + } + var segSec = rsaSegmentSec; + if (span < 2 * segSec) segSec = span / 2; + if (segSec < 60) { + return rsaRespRate(nn, [for (final t in times) t * 1000.0], + artifactFraction: artifactFraction, + tolBrpm: tolBrpm, + minConsensus: minConsensus, + maxArtifact: maxArtifact); + } + if (span < 2 * rsaSegmentSec) { + return rsaRespRate(nn, [for (final t in times) t * 1000.0], + artifactFraction: artifactFraction, + tolBrpm: tolBrpm, + minConsensus: minConsensus, + maxArtifact: maxArtifact); + } + final segments = <_RsaSegmentResult>[..._closed]; + for (var i = _closed.length;; i++) { + final start = _first! + i * (rsaSegmentSec / 2); + if (start + rsaSegmentSec > last + 1e-9) break; + segments.add(_rsaSegment(times, nn, start, rsaSegmentSec)); + } + return _rsaMetricFromSegments(segments, + artifactFraction: artifactFraction, + tolBrpm: tolBrpm, + minConsensus: minConsensus, + segSec: rsaSegmentSec, + hiHz: hiHz); + } + + void _settle() { + while (_nextStart! + rsaSegmentSec < _last! - 1e-9) { + _closed.add(_rsaSegment(_times, _nn, _nextStart!, rsaSegmentSec)); + _nextStart = _first! + _closed.length * (rsaSegmentSec / 2); + } + } + + void _compact() { + var cut = 0; + while (cut < _times.length && _times[cut] < _nextStart!) { + cut++; + } + if (cut > 0) { + _times.removeRange(0, cut); + _nn.removeRange(0, cut); + } + } +} + +/// [a] and [b] sorted by [double.compareTo]; the merge is sorted the same way. +List _mergeSorted(List a, List b) { + if (b.isEmpty) return a; + if (a.isEmpty) return b; + final out = List.filled(a.length + b.length, 0); + var i = 0, j = 0, k = 0; + while (i < a.length && j < b.length) { + out[k++] = a[i].compareTo(b[j]) <= 0 ? a[i++] : b[j++]; + } + while (i < a.length) { + out[k++] = a[i++]; + } + while (j < b.length) { + out[k++] = b[j++]; + } + return out; +} + +bool _nondecreasingBy(List values) { + for (var i = 1; i < values.length; i++) { + if (values[i].compareTo(values[i - 1]) < 0) return false; + } + return true; +} + +bool _nondecreasing(List values) { + for (var i = 1; i < values.length; i++) { + if (values[i] < values[i - 1]) return false; + } + return true; +} + /// RIIV respiratory rate from the 1 Hz green PPG ADC. /// /// Respiratory-Induced Intensity Variation: a 0.1–0.5 Hz band-pass on the green diff --git a/lib/src/onehz/respiration/respiration.dart b/lib/src/onehz/respiration/respiration.dart index 0b70bbd..31eb061 100644 --- a/lib/src/onehz/respiration/respiration.dart +++ b/lib/src/onehz/respiration/respiration.dart @@ -22,5 +22,6 @@ library onehz_respiration; export 'resp_rate.dart'; export 'cvhr_apnea.dart'; +export 'cvhr_stream.dart'; export 'relative_odi.dart'; export 'brv_trend.dart'; diff --git a/lib/src/onehz/sleep/advanced_stager.dart b/lib/src/onehz/sleep/advanced_stager.dart index 184d28a..87f3eed 100644 --- a/lib/src/onehz/sleep/advanced_stager.dart +++ b/lib/src/onehz/sleep/advanced_stager.dart @@ -36,12 +36,16 @@ // Pure Dart, dart:math only. import 'dart:math' as math; +import '../../aggregate/aggregator.dart'; import '../types.dart'; import '../util.dart'; import '../clinical/hrv_time.dart'; -import 'cardio_stager.dart' show cardioStager; -import 'accounting.dart' show SleepStage; +import 'epoch_rows.dart'; +import 'cardio_row_cache.dart'; import 'band_offset.dart'; +import 'hypnogram.dart'; +export 'hypnogram.dart'; +import 'hypnogram_aggregator.dart'; /// Which per-session staging engine [AdvancedSleepStager] runs. See the file /// header for the 2026-07 comparison behind this choice. Do not switch the @@ -99,86 +103,6 @@ class RespTs { const RespTs(this.ts, this.raw); } -/// A scored stage span. [stage] ∈ {'wake','light','deep','rem'}. -class StageSegment { - final int start; - final int end; - final String stage; - const StageSegment(this.start, this.end, this.stage); - Map toJson() => {'start': start, 'end': end, 'stage': stage}; -} - -/// One detected sleep session + its hypnogram. -class SleepSession { - final int start; - final int end; - final double efficiency; // [0,1] - final List stages; - final int? restingHr; // lowest 5-min rolling-mean HR (bpm) - final double? avgHrv; // mean RMSSD over 5-min windows (ms) - const SleepSession({ - required this.start, - required this.end, - required this.efficiency, - required this.stages, - required this.restingHr, - required this.avgHrv, - }); -} - -/// AASM-style hypnogram metrics over a session. -class HypnogramMetrics { - final int tibS; - final int tstS; - final int sptS; - /// Sleep-onset latency (s). NULL when NOTHING staged as sleep — reporting - /// the whole time in bed as "time to fall asleep" alongside TST 0 is a - /// fabricated measurement, not an abstention. - final int? solS; - final double remLatencyS; // NaN if no REM - final int wasoS; - final double efficiency; // [0,1] - final int disturbances; - final double deepMin; - final double remMin; - final double lightMin; - final double deepPct; - final double remPct; - final double lightPct; - const HypnogramMetrics({ - required this.tibS, - required this.tstS, - required this.sptS, - required this.solS, - required this.remLatencyS, - required this.wasoS, - required this.efficiency, - required this.disturbances, - required this.deepMin, - required this.remMin, - required this.lightMin, - required this.deepPct, - required this.remPct, - required this.lightPct, - }); - Map toJson() => { - 'tib_s': tibS, - 'tst_s': tstS, - 'spt_s': sptS, - 'sol_s': solS, - 'rem_latency_s': remLatencyS.isNaN ? null : remLatencyS, - 'waso_s': wasoS, - 'efficiency': efficiency, - 'disturbances': disturbances, - 'deep_min': deepMin, - 'rem_min': remMin, - 'light_min': lightMin, - 'deep_pct': deepPct, - 'rem_pct': remPct, - 'light_pct': lightPct, - }; -} - /// Advanced sleep stager — constants + the full V1/V2 algorithm. class AdvancedSleepStager { // ── Stage-0 constants ─────────────────────────────────────────────────────── @@ -315,6 +239,7 @@ class AdvancedSleepStager { int tzOffsetSec = 0, int Function(int tsSec)? tzOffsetResolver, StagingMethod method = StagingMethod.cardio, + CardioRowCache? rowCache, List> wristOff = const [], // each [start,end] List> bandSleepState = const [], // each [ts,state] }) { @@ -387,7 +312,8 @@ class AdvancedSleepStager { } final stages = switch (method) { - StagingMethod.cardio => _stageSessionCardio(p.start, p.end, grav, hrS, rrS), + StagingMethod.cardio => + _stageSessionCardio(p.start, p.end, grav, hrS, rrS, rowCache), StagingMethod.v2 => _stageSessionV2(p.start, p.end, grav, hrS, rrS), StagingMethod.v1 => _stageSession(p.start, p.end, grav, hrS, rrS, respS), }; @@ -438,9 +364,11 @@ class AdvancedSleepStager { List rr = const [], List resp = const [], StagingMethod method = StagingMethod.cardio, + CardioRowCache? rowCache, }) { final stages = switch (method) { - StagingMethod.cardio => _stageSessionCardio(startSec, endSec, gravity, hr, rr), + StagingMethod.cardio => + _stageSessionCardio(startSec, endSec, gravity, hr, rr, rowCache), StagingMethod.v2 => _stageSessionV2(startSec, endSec, gravity, hr, rr), StagingMethod.v1 => _stageSession(startSec, endSec, gravity, hr, rr, resp), }; @@ -1424,125 +1352,47 @@ class AdvancedSleepStager { /// as unobserved. static const int maxAccelCarryForwardSec = 60; - /// DEFAULT staging path — delegates to `cardioStager` (cardio_stager.dart). - /// See the file header for why this is the default. `cardioStager` expects - /// per-SECOND-indexed [AccelSample]/HR arrays (index i == second i from - /// [start]), not the sparse timestamped [GravTs]/[HrTs] lists this file - /// otherwise uses — so this builds that dense array explicitly: accel gaps - /// carry the last-known vector forward for at most - /// [maxAccelCarryForwardSec]; HR gaps fill with 0, which `cardioStager` - /// already documents as its own "off-skin" contract — no fabrication either - /// way. - /// - /// HONESTY: seconds with no usable accelerometer are NOT staged. The window - /// is split at those gaps and each contiguous usable RUN is staged on its own - /// (so a dropout cannot pollute the neighbouring run's night baselines - /// either); the gap seconds, runs too short to stage, and a window where - /// `cardioStager` itself abstains all come back as 'unobserved' — the "stay - /// unstaged" contract [stageWindow] documents. They must NEVER come back as - /// 'light', which is what a zero-data window used to report for its entire - /// length, and not as 'wake' either: an abstained run can still carry HR, so - /// `segmentSleep` would see it as observed and publish it as measured WASO. - static List _stageSessionCardio(int start, int end, - List grav, List hr, List rr) { - final span = end - start; - if (span <= 0) return const []; - final epSec = epochS.round(); - final minStageableSec = 3 * epSec; - if (span < minStageableSec) return [StageSegment(start, end, 'unobserved')]; - - // Invalid samples are NOT samples: they must not seed `usable`, and they - // must not become the source of a carry-forward. A second with only an - // undecoded vector falls through to the same unstaged→unobserved path as a - // second with no row at all (cardioStager itself never reads - // [AccelSample.valid], so it can only be protected here). - final gByTs = { - for (final g in grav) - if (g.valid && g.ts >= start && g.ts < end) g.ts: g - }; - final hByTs = {for (final h in hr) if (h.ts >= start && h.ts < end) h.ts: h}; - final accel = List.filled( - span, AccelSample(start * 1000.0, 0, 0, 1.0)); - final hr1hz = List.filled(span, 0.0); - // usable[i] — second i has a real accel sample or a BOUNDED carry-forward. - final usable = List.filled(span, false); - var lastRealIdx = -1; - for (var i = 0; i < span; i++) { - final ts = start + i; - final g = gByTs[ts]; - if (g != null) { - accel[i] = AccelSample(ts * 1000.0, g.x, g.y, g.z); - lastRealIdx = i; - usable[i] = true; - } else if (lastRealIdx >= 0 && (i - lastRealIdx) <= maxAccelCarryForwardSec) { - // Bounded carry-forward — see [maxAccelCarryForwardSec]. The TIMESTAMP - // is this second's, not the stale sample's: cardioStager centres its RR - // windows on `accel[mid].tsMs`, so copying the old sample wholesale - // mis-centred every RMSSD/LF-HF window inside a gap. - final p = accel[i - 1]; - accel[i] = AccelSample(ts * 1000.0, p.x, p.y, p.z); - usable[i] = true; - } - hr1hz[i] = hByTs[ts]?.bpm ?? 0.0; // 0 = off-skin, cardioStager's own contract. - } - - final rSeg = [for (final r in rr) if (r.ts >= start && r.ts < end) r]; - final rrMs = [for (final r in rSeg) r.rrMs]; - final rrTsMs = [for (final r in rSeg) r.ts * 1000.0]; - - // Everything not staged below stays 'unobserved' — no stage, no wake. - final perSec = List.filled(span, 'unobserved'); - var i = 0; - while (i < span) { - if (!usable[i]) { - i++; - continue; - } - var j = i; - while (j < span && usable[j]) { - j++; - } - if ((j - i) >= minStageableSec) { - final result = cardioStager( - hr1hz.sublist(i, j), - accel.sublist(i, j), - rrMs: rrMs, - rrTsMs: rrTsMs, - ); - final nEpoch = result.base.stages.length; - for (var e = 0; e < nEpoch; e++) { - final label = switch (result.base.stages[e]) { - SleepStage.wake => 'wake', - SleepStage.rem => 'rem', - SleepStage.nrem => (e < result.deepFlag.length && result.deepFlag[e]) - ? 'deep' - : 'light', - }; - final lo = i + e * epSec; - // The last epoch absorbs the run's sub-epoch remainder, exactly as - // [_buildSegments] extends the final segment to the window end. - final hi = e == nEpoch - 1 ? j : math.min(j, lo + epSec); - for (var k = lo; k < hi; k++) { - perSec[k] = label; - } - } - } - i = j; - } + // Dart's List.sort is not stable; index breaks ties so equal timestamps keep + // list order. + static List _stableByTs(List xs, int Function(T) ts) { + final idx = List.generate(xs.length, (i) => i) + ..sort((a, b) { + final c = ts(xs[a]).compareTo(ts(xs[b])); + return c != 0 ? c : a.compareTo(b); + }); + return [for (final i in idx) xs[i]]; + } - // Coalesce the per-second labels into contiguous [StageSegment]s tiling - // [start, end) — same segment semantics [_buildSegments] produces. - final segments = []; - var k = 0; - while (k < span) { - var m = k; - while (m < span && perSec[m] == perSec[k]) { - m++; - } - segments.add(StageSegment(start + k, start + m, perSec[k])); - k = m; - } - return segments; + /// Default staging path: 30 s epochs anchored at the session onset (the first + /// valid accelerometer sample at or after [start], phase 0), as before the + /// combined bump. Samples outside [start, end) are not read, so features are + /// clipped at the session edges. A usable run that follows a dropout the 60 s + /// carry-forward cannot bridge re-anchors at its own first sample. Other gaps + /// and abstained runs remain unobserved. + static List _stageSessionCardio(int start, int end, + List grav, List hr, List rr, + [CardioRowCache? rowCache]) { + if (end <= start) return const []; + final g = [for (final g in grav) if (g.ts >= start && g.ts < end) g]; + int? onset; + for (final x in g) { + if (x.valid && (onset == null || x.ts < onset)) onset = x.ts; + } + final rows = EpochRows( + rowCache: rowCache, anchorMs: onset == null ? null : onset * 1000); + final beats = _stableByTs( + [for (final r in rr) if (r.ts >= start && r.ts < end) r], (r) => r.ts); + rows.fold( + // Stable by time: equal timestamps keep list order (the oracle's last-wins). + hr: [for (final h in _stableByTs([ + for (final h in hr) if (h.ts >= start && h.ts < end) h + ], (h) => h.ts)) HrSample(h.ts * 1000.0, h.bpm)], + accel: [for (final x in _stableByTs(g, (x) => x.ts)) AccelSample(x.ts * 1000.0, x.x, x.y, x.z, valid: x.valid)], + rrMs: [for (final r in beats) r.rrMs], + rrTsMs: [for (final r in beats) r.ts * 1000.0], + throughMs: end * 1000 - 1, // the session end closes the data, as a clip + ); + return rows.stageWindow(start, end); } static List _stageSession(int start, int end, List grav, @@ -1955,59 +1805,7 @@ class AdvancedSleepStager { static HypnogramMetrics hypnogramMetrics(SleepSession session) { final segs = [...session.stages]..sort((a, b) => a.start.compareTo(b.start)); - final tib = math.max(0, session.end - session.start); - final sleepSegs = segs - .where((s) => s.stage == 'light' || s.stage == 'deep' || s.stage == 'rem') - .toList(); - var tst = 0, deepS = 0, remS = 0, lightS = 0; - for (final s in sleepSegs) { - final d = s.end - s.start; - tst += d; - if (s.stage == 'deep') deepS += d; - if (s.stage == 'rem') remS += d; - if (s.stage == 'light') lightS += d; - } - int onset, sptEnd; - int? sol; - if (sleepSegs.isNotEmpty) { - onset = sleepSegs.first.start; - sptEnd = sleepSegs.last.end; - sol = math.max(0, onset - session.start); - } else { - onset = session.end; - sptEnd = session.end; - sol = null; // never observed sleep — no latency to report - } - final remSegs = sleepSegs.where((s) => s.stage == 'rem').toList(); - final remLatency = - remSegs.isNotEmpty ? (remSegs.first.start - onset).toDouble() : double.nan; - var waso = 0, disturbances = 0; - for (final s in segs.where((s) => s.stage == 'wake')) { - final w0 = math.max(s.start, onset); - final w1 = math.min(s.end, sptEnd); - if (w1 > w0) { - waso += w1 - w0; - disturbances++; - } - } - final se = tib > 0 ? tst / tib : 0.0; - double pct(int x) => tst > 0 ? x / tst * 100 : 0; - return HypnogramMetrics( - tibS: tib, - tstS: tst, - sptS: math.max(0, sptEnd - onset), - solS: sol, - remLatencyS: remLatency, - wasoS: waso, - efficiency: math.min(1.0, se), - disturbances: disturbances, - deepMin: deepS / 60, - remMin: remS / 60, - lightMin: lightS / 60, - deepPct: pct(deepS), - remPct: pct(remS), - lightPct: pct(lightS), - ); + return HypnogramAggregator(session.start, session.end).runList(segs); } // ── shared math (population std, percentile) ────────────────────────────── diff --git a/lib/src/onehz/sleep/cardio_row_cache.dart b/lib/src/onehz/sleep/cardio_row_cache.dart new file mode 100644 index 0000000..4bc1cc3 --- /dev/null +++ b/lib/src/onehz/sleep/cardio_row_cache.dart @@ -0,0 +1,170 @@ +// Persistent, exact feature rows for the cardio sleep stager. + +import '../../aggregate/json_support.dart'; + +// Reformulates cardioStager() feature extraction and classification: van Hees +// 2013 motion and Long 2016 / Fonseca 2015/2018 cardiorespiratory staging. +// Internal fitted weights and rules remain ESTIMATE with low confidence. +// Session-onset anchoring and post-gap re-anchoring preserve the pre-bump grid; +// shared arithmetic preserves labels except for separately documented +// RR/spectral changes. Cached rows require matching dependencies; missing phases +// are recomputed. +/// A cache identity for one cardio feature epoch. +/// +/// [fingerprint] is an unsigned 64-bit FNV-1a digest assembled by the stager +/// from every input that can affect the row. It deliberately is not a Dart +/// [Object.hashCode]: cache files have to work across processes and releases. +class CardioRowKey { + final int startSec; + final int epochSec; + final int fingerprint; + + const CardioRowKey(this.startSec, this.epochSec, this.fingerprint); + + @override + bool operator ==(Object other) => + other is CardioRowKey && + startSec == other.startSec && + epochSec == other.epochSec && + fingerprint == other.fingerprint; + + @override + int get hashCode => Object.hash(startSec, epochSec, fingerprint); +} + +/// The seven raw feature values consumed by one cardio epoch classifier row. +class CardioEpochRow { + final double motion; + final double hr; + final double hrSd; + final double rmssd; + final double lfhf; + final double rk; + final double sdnn; + + const CardioEpochRow( + this.motion, this.hr, this.hrSd, this.rmssd, this.lfhf, this.rk, this.sdnn); +} + +/// Reusable, mergeable cache of exact cardio feature rows. +class CardioRowCache { + // 2: window RMSSD and R(k) skip seam pairs (upstream merge); older rows + // hold values computed across seams. + static const int _version = 2; + final Map _rows = {}; + + CardioRowCache(); + + /// Number of successful lookups since this cache was created or decoded. + int hits = 0; + + /// Number of unsuccessful lookups since this cache was created or decoded. + int misses = 0; + + int get length => _rows.length; + + CardioEpochRow? lookup(CardioRowKey key) { + final row = _rows[key]; + if (row == null) { + misses++; + } else { + hits++; + } + return row; + } + + void insert(CardioRowKey key, CardioEpochRow row) { + _rows[key] = row; + } + + void retainWhere(bool Function(CardioRowKey key) keep) { + _rows.removeWhere((key, _) => !keep(key)); + } + + /// Union [other] into this cache. Rows already held here win identically-keyed + /// ties; an identical key can only describe an identical input row. + void merge(CardioRowCache other) { + for (final entry in other._rows.entries) { + _rows.putIfAbsent(entry.key, () => entry.value); + } + } + + Map toJson() { + final entries = _rows.entries.toList() + ..sort((a, b) { + final k = a.key, q = b.key; + var c = k.startSec.compareTo(q.startSec); + if (c != 0) return c; + c = k.epochSec.compareTo(q.epochSec); + if (c != 0) return c; + return k.fingerprint.compareTo(q.fingerprint); + }); + return { + 'version': _version, + 'rows': [ + for (final entry in entries) + { + 'startSec': entry.key.startSec, + 'epochSec': entry.key.epochSec, + // Decimal strings keep a 64-bit word lossless in JSON + // implementations whose numeric type is IEEE-754. + 'fingerprint': entry.key.fingerprint.toSigned(64).toString(), + 'motion': encodeDouble(entry.value.motion), + 'hr': encodeDouble(entry.value.hr), + 'hrSd': encodeDouble(entry.value.hrSd), + 'rmssd': encodeDouble(entry.value.rmssd), + 'lfhf': encodeDouble(entry.value.lfhf), + 'rk': encodeDouble(entry.value.rk), + 'sdnn': encodeDouble(entry.value.sdnn), + } + ], + }; + } + + factory CardioRowCache.fromJson(Map json) { + try { + if (json['version'] != _version) { + throw const FormatException('Unsupported CardioRowCache version'); + } + final raw = json['rows']; + if (raw is! List) throw const FormatException('Invalid CardioRowCache rows'); + final cache = CardioRowCache(); + for (final item in raw) { + if (item is! Map) throw const FormatException('Invalid CardioRowCache row'); + final row = item.cast(); + int readInt(String name) { + final value = row[name]; + if (value is! int) throw FormatException('Invalid $name'); + return value; + } + final fingerprint = row['fingerprint']; + if (fingerprint is! String) { + throw const FormatException('Invalid fingerprint'); + } + final fp = int.tryParse(fingerprint); + if (fp == null) { + throw const FormatException('Invalid fingerprint'); + } + final key = CardioRowKey(readInt('startSec'), readInt('epochSec'), fp); + if (key.epochSec <= 0) throw const FormatException('Invalid epochSec'); + if (cache._rows.containsKey(key)) { + throw const FormatException('Duplicate CardioRowCache row'); + } + cache._rows[key] = CardioEpochRow( + decodeDouble(row['motion']), + decodeDouble(row['hr']), + decodeDouble(row['hrSd']), + decodeDouble(row['rmssd']), + decodeDouble(row['lfhf']), + decodeDouble(row['rk']), + decodeDouble(row['sdnn']), + ); + } + return cache; + } on FormatException { + rethrow; + } catch (_) { + throw const FormatException('Invalid CardioRowCache'); + } + } +} diff --git a/lib/src/onehz/sleep/cardio_stager.dart b/lib/src/onehz/sleep/cardio_stager.dart index 6738c3a..a8e73b2 100644 --- a/lib/src/onehz/sleep/cardio_stager.dart +++ b/lib/src/onehz/sleep/cardio_stager.dart @@ -77,7 +77,9 @@ // tier ESTIMATE; confidence reflects RR coverage (RMSSD drives REM/deep). import 'dart:math' as math; +import 'dart:typed_data'; import '../types.dart'; +import 'cardio_row_cache.dart'; import '../util.dart'; import 'accounting.dart' show SleepStage; import 'stager.dart' show StagerResult, consolidateSleepStages; @@ -88,7 +90,26 @@ class CardioStagerResult { final StagerResult base; final List deepFlag; final double confidence; - const CardioStagerResult(this.base, this.deepFlag, this.confidence); + + /// Per-epoch REM / deep scores as the classify loop computed them, BEFORE + /// the mode filter, Webster rescore and consolidation. Null where the epoch + /// was decided wake first (no score taken) or no axis was measurable. Same + /// length as `base.stages`, or empty when the stager abstained or the result + /// was built without them. Read through `epochEvidence` (stage_ranges.dart). + final List remScores; + final List deepScores; + + /// Per-epoch: the epoch carried at least one valid HR sample. + final List hrCovered; + + const CardioStagerResult( + this.base, + this.deepFlag, + this.confidence, { + this.remScores = const [], + this.deepScores = const [], + this.hrCovered = const [], + }); } const int _epochSec = 30; @@ -116,6 +137,13 @@ const double _wRemSdnn = 0.32; const double _wRemHrSd = 0.27; const double _wRemLfhf = 0.12; +/// The REM axis weights, exposed (read-only) for [CausalStager], which scores +/// REM from the SAME measured effect sizes instead of re-declaring them. +const double kRemWeightRk = _wRemRk; +const double kRemWeightSdnn = _wRemSdnn; +const double kRemWeightHrSd = _wRemHrSd; +const double kRemWeightLfhf = _wRemLfhf; + const double _wDeepRk = 0.53; const double _wDeepHrSd = 0.43; const double _wDeepSdnn = 0.41; @@ -382,10 +410,56 @@ CardioStagerResult cardioStager( SleepUserProfile? userProfile, double remScoreCut = _remScoreCut, double deepScoreCut = _deepScoreCut, + CardioRowCache? rowCache, +}) { + return classifyCardioEpochs( + cardioEpochFeatures( + hr1hz, + accel, + rrMs: rrMs, + rrTsMs: rrTsMs, + epochSec: epochSec, + rowCache: rowCache, + ), + epochSec: epochSec, + userProfile: userProfile ?? cardioUserProfile, + remScoreCut: remScoreCut, + deepScoreCut: deepScoreCut, + ); +} + +/// Extracts the raw per-epoch inputs consumed by [classifyCardioEpochs]. +/// +/// The returned rows cover `[startEpoch, endEpoch)`, clamped to the session's +/// complete epoch grid. A row for epoch `e` depends on: its own `perEpoch` +/// samples of [hr1hz] and [accel]; accel magnitudes from `300 s / 2` on either +/// side of that epoch's start/end for its local 1 g median (clipped at the +/// session bounds); [rrMs]/[rrTsMs] in the inclusive ±150 s window centred on +/// the epoch for RMSSD/SDNN; and those inputs in the inclusive ±90 s window +/// for LF/HF and R(k). It also depends on the full input's measured accel +/// cadence, its clipped length, [epochSec], and the absolute accel timestamps +/// that define the RR windows. No row depends on neighbouring feature rows or +/// any user profile/classification setting. +/// +/// [rrTsMs] must be non-decreasing when it is paired with [rrMs]. Invalid grid +/// inputs return zero rows, as does a session with fewer than three epochs. +// Reformulates cardioStager() feature extraction and classification: van Hees +// 2013 motion and Long 2016 / Fonseca 2015/2018 cardiorespiratory staging. +// Internal fitted weights and rules remain ESTIMATE with low confidence. +// Session-onset anchoring and post-gap re-anchoring preserve the pre-bump grid; +// shared arithmetic preserves labels except for separately documented +// RR/spectral changes. Cached rows require matching dependencies; missing phases +// are recomputed. +CardioEpochFeatures cardioEpochFeatures( + List hr1hz, + List accel, { + List rrMs = const [], + List rrTsMs = const [], + int epochSec = _epochSec, + int startEpoch = 0, + int? endEpoch, + CardioRowCache? rowCache, }) { - // Explicit arg wins (unit tests); else the ambient profile the edge set for - // this staging pass; else null ⇒ pure per-night-local (cold-start behavior). - final profile = userProfile ?? cardioUserProfile; // `_cleanBeatsInWindow` binary-searches this, so it has to be non-decreasing. // The producer guarantees it — beats inherit their record's second and the // records are sorted, and the one path that folds in loose live beats @@ -415,7 +489,7 @@ CardioStagerResult cardioStager( // the Webster/Cole-Kripke continuity rules below are specified at this epoch // length. Stretching `epochSec` to fit the device would make a number appear // where the evidence for it does not; abstain instead. - if (cadenceSec == null || cadenceSec > epochSec) return _abstain(epochSec); + if (cadenceSec == null || cadenceSec > epochSec) return _emptyCardioFeatures; final perEpoch = math.max(1, (epochSec / cadenceSec).round()); // `perEpoch` samples at this cadence must actually SPAN `epochSec` — a // cadence that does not divide it evenly (4 s against a 30 s epoch rounds @@ -427,9 +501,34 @@ CardioStagerResult cardioStager( // Abstain rather than publish a grid whose real spacing does not match its // own label. if ((perEpoch * cadenceSec - epochSec).abs() > 1e-6) - return _abstain(epochSec); + return _emptyCardioFeatures; final nEpoch = n ~/ perEpoch; - if (nEpoch < 3) return _abstain(epochSec); + if (nEpoch < 3) return _emptyCardioFeatures; + + final firstEpoch = startEpoch.clamp(0, nEpoch); + final lastEpoch = (endEpoch ?? nEpoch).clamp(firstEpoch, nEpoch); + final rows = lastEpoch - firstEpoch; + + // The cache is deliberately inserted before the acceleration magnitude and + // median work. A hit avoids the Lomb–Scargle calculation AND all of the + // otherwise per-pass feature extraction. A miss range delegates back to this + // exact routine with the same full arrays, so its local median's clipped + // bounds and every floating-point operation remain byte-for-byte unchanged. + if (rowCache != null) { + return _cachedCardioEpochFeatures( + hr1hz, + accel, + rrMs, + rrTsMs, + epochSec, + n, + cadenceSec, + perEpoch, + firstEpoch, + lastEpoch, + rowCache, + ); + } // ── per-second ENMO (motion) against a LOCALLY-ADAPTIVE 1 g reference ────── // A single whole-night gravity-magnitude reference (the old approach) is @@ -458,72 +557,227 @@ CardioStagerResult cardioStager( const int _gRefWinSec = 300; // 5 min window, centered per epoch // …in SAMPLES at the measured cadence. 300 at 1 Hz, unchanged. final gRefHalf = math.max(1, (_gRefWinSec / cadenceSec).round()) ~/ 2; - final gRefByEpoch = List.filled(nEpoch, 1.0); - for (var e = 0; e < nEpoch; e++) { - final es = e * perEpoch; - final lo = math.max(0, es - gRefHalf); - final hi = math.min(n, es + perEpoch + gRefHalf); - gRefByEpoch[e] = median(mag.sublist(lo, hi)) ?? 1.0; - } + final gRefByEpoch = _epochGRefMedians( + mag, perEpoch, gRefHalf, firstEpoch, lastEpoch); // ── per-epoch features ───────────────────────────────────────────────────── - final motion = List.filled(nEpoch, 0); - final hr = List.filled(nEpoch, double.nan); - final hrSd = List.filled(nEpoch, 0); - final rmssd = List.filled(nEpoch, double.nan); - final sdnn = List.filled(nEpoch, double.nan); + final motion = List.filled(rows, 0); + final hr = List.filled(rows, double.nan); + final hrSd = List.filled(rows, 0); + final rmssd = List.filled(rows, double.nan); // REM autonomic features (P1): LF/HF from the RR spectrum + R(k) = rolling // mean |ΔIHR|. REM's signature at 1 Hz beat-timing is a SYMPATHETIC shift — // LF/HF rises, RR variability drops (low RMSSD), and instantaneous HR gets // "jittery" (elevated |ΔIHR|). The old REM rule leaned only on the RMSSD drop // and under-called REM; these two independent axes recover it (OR-combined // below), gated by atonia + an HR floor so a plain arousal is not miscalled. - final lfhf = List.filled(nEpoch, double.nan); - final rk = List.filled(nEpoch, double.nan); + final sdnn = List.filled(rows, double.nan); + final lfhf = List.filled(rows, double.nan); + final rk = List.filled(rows, double.nan); - for (var e = 0; e < nEpoch; e++) { + for (var e = firstEpoch; e < lastEpoch; e++) { + final row = e - firstEpoch; final s = e * perEpoch, t = math.min(s + perEpoch, n); // motion = mean ENMO over the epoch, against THIS epoch's local reference. - final gRefE = gRefByEpoch[e]; + final gRefE = gRefByEpoch[row]; var ms = 0.0; for (var i = s; i < t; i++) { final d = mag[i] - gRefE; ms += d > 0 ? d : 0.0; } - motion[e] = ms / (t - s); + motion[row] = ms / (t - s); // hr mean/sd over valid (>0) seconds final hv = [ for (var i = s; i < t; i++) if (hr1hz[i] > 0) hr1hz[i] ]; - if (hv.isNotEmpty) hr[e] = mean(hv)!; - if (hv.length >= 2) hrSd[e] = stddev(hv) ?? 0; + if (hv.isNotEmpty) hr[row] = mean(hv)!; + if (hv.length >= 2) hrSd[row] = stddev(hv) ?? 0; // rmssd over RR beats within a ±2.5-min window centred on the epoch - rmssd[e] = _windowRmssd(rrMs, rrTsMs, accel, s, t, epochSec); - sdnn[e] = _windowSdnn(rrMs, rrTsMs, accel, s, t, epochSec); + rmssd[row] = _windowRmssd(rrMs, rrTsMs, accel, s, t, epochSec); + sdnn[row] = _windowSdnn(rrMs, rrTsMs, accel, s, t, epochSec); // LF/HF + R(k) over a ±90-s RR window centred on the epoch. final rem = _windowRemFeatures(rrMs, rrTsMs, accel, s, t, epochSec); - if (rem.lfhf != null) lfhf[e] = rem.lfhf!; - if (rem.rk != null) rk[e] = rem.rk!; + if (rem.lfhf != null) lfhf[row] = rem.lfhf!; + if (rem.rk != null) rk[row] = rem.rk!; } - return classifyCardioEpochs( - CardioEpochFeatures( - motion: motion, - hr: hr, - hrSd: hrSd, - rmssd: rmssd, - lfhf: lfhf, - rk: rk, - sdnn: sdnn, - ), - epochSec: epochSec, - userProfile: profile, - remScoreCut: remScoreCut, - deepScoreCut: deepScoreCut, + return CardioEpochFeatures( + motion: motion, + hr: hr, + hrSd: hrSd, + rmssd: rmssd, + lfhf: lfhf, + rk: rk, + sdnn: sdnn, ); } +/// Cache-aware extraction. The key's context is intentionally the contiguous +/// usable run supplied to this stager, not the caller's larger record: samples +/// outside that run cannot influence [cardioEpochFeatures]. +CardioEpochFeatures _cachedCardioEpochFeatures( + List hr1hz, + List accel, + List rrMs, + List rrTsMs, + int epochSec, + int n, + double cadenceSec, + int perEpoch, + int firstEpoch, + int lastEpoch, + CardioRowCache cache, +) { + final count = lastEpoch - firstEpoch; + final keys = List.filled( + count, const CardioRowKey(0, 0, 0)); + final values = List.filled(count, null); + final missing = List.filled(count, false); + final gRefHalf = math.max(1, (300 / cadenceSec).round()) ~/ 2; + for (var e = firstEpoch; e < lastEpoch; e++) { + final row = e - firstEpoch; + final key = _cardioRowKey( + hr1hz, accel, rrMs, rrTsMs, epochSec, n, cadenceSec, perEpoch, e, gRefHalf); + keys[row] = key; + final value = cache.lookup(key); + values[row] = value; + missing[row] = value == null; + } + + // One recursive feature extraction per contiguous cold span. It keeps the + // old vectorized median path intact and avoids turning a changed handful of + // rows into a handful of Lomb–Scargle setup passes. + var row = 0; + while (row < count) { + if (!missing[row]) { + row++; + continue; + } + final first = row; + while (row < count && missing[row]) { + row++; + } + final features = cardioEpochFeatures( + hr1hz, + accel, + rrMs: rrMs, + rrTsMs: rrTsMs, + epochSec: epochSec, + startEpoch: firstEpoch + first, + endEpoch: firstEpoch + row, + ); + for (var i = first; i < row; i++) { + final f = i - first; + final value = CardioEpochRow( + features.motion[f], + features.hr[f], + features.hrSd[f], + features.rmssd[f], + features.lfhf[f], + features.rk[f], + features.sdnn[f], + ); + values[i] = value; + cache.insert(keys[i], value); + } + } + return CardioEpochFeatures( + motion: [for (final value in values) value!.motion], + hr: [for (final value in values) value!.hr], + hrSd: [for (final value in values) value!.hrSd], + rmssd: [for (final value in values) value!.rmssd], + lfhf: [for (final value in values) value!.lfhf], + rk: [for (final value in values) value!.rk], + sdnn: [for (final value in values) value!.sdnn], + ); +} + +const int _fnvOffset64 = 0xcbf29ce484222325; +const int _fnvPrime64 = 0x100000001b3; +const int _u64Mask = (1 << 64) - 1; + +/// Exact-bit fingerprint of the data one row can observe. The clipped run +/// length never enters a row as a global scalar: it only changes an epoch whose +/// 300-s gravity reference window touches a run edge. The explicit clip +/// distances below capture precisely that dependency while rows away from both +/// edges survive a moved session boundary. +CardioRowKey _cardioRowKey( + List hr, + List accel, + List rr, + List rrTs, + int epochSec, + int n, + double cadenceSec, + int perEpoch, + int epoch, + int gRefHalf, +) { + var hash = _fnvOffset64; + void word(int value) { + hash ^= value & _u64Mask; + hash = (hash * _fnvPrime64) & _u64Mask; + } + void number(double value) { + final bytes = ByteData(8)..setFloat64(0, value, Endian.little); + word(bytes.getUint64(0, Endian.little)); + } + + final s = epoch * perEpoch; + final t = math.min(s + perEpoch, n); + final lo = math.max(0, s - gRefHalf); + final hi = math.min(n, t + gRefHalf); + word(0x43415244494f31); // domain separator: CARDIO1 + word(epochSec); + word(perEpoch); + number(cadenceSec); + // Only an actually clipped edge is length-sensitive. A stable marker for an + // unclipped side means shifting a far-away session edge preserves the row. + word(lo == 0 ? gRefHalf - s : -1); + word(hi == n ? t + gRefHalf - n : -1); + word(hi - lo); + for (var i = lo; i < hi; i++) { + final a = accel[i]; + number(a.tsMs); + number(a.x); + number(a.y); + number(a.z); + } + word(t - s); + for (var i = s; i < t; i++) { + number(hr[i]); + } + // RR has no influence at all for an unpaired input, matching the feature + // extractor's early exit. Otherwise include every raw beat in the widest + // (±150 s), inclusive window; the ±90 s REM window is a subset. + if (rr.length != rrTs.length) { + word(0); + } else { + word(1); + final mid = (s + t) ~/ 2; + final centre = accel[mid].tsMs; + final low = centre - 150000.0, high = centre + 150000.0; + var a = 0, b = rrTs.length; + while (a < b) { + final m = (a + b) >> 1; + if (rrTs[m] < low) { + a = m + 1; + } else { + b = m; + } + } + var count = 0; + for (var i = a; i < rrTs.length && rrTs[i] <= high; i++) { + number(rrTs[i]); + number(rr[i]); + count++; + } + word(count); + } + return CardioRowKey((accel[s].tsMs / 1000.0).round(), epochSec, hash); +} + /// Per-epoch feature vectors — the ONLY thing the staging DECISION layer sees. /// /// Splitting these out from [cardioStager] draws the line between "turn 1 Hz @@ -581,6 +835,86 @@ class CardioEpochFeatures { ].reduce(math.min); } +const _emptyCardioFeatures = CardioEpochFeatures( + motion: [], + hr: [], + hrSd: [], + rmssd: [], + lfhf: [], + rk: [], + sdnn: [], +); + +/// The 1 g reference windows advance by an epoch, rather than one sample, and +/// have an even length at the normal 1 Hz / 30 s grid. That is not the centred +/// odd-width shape accepted by [slidingMedians], but it uses the same one +/// sorted-window binary-insert/remove algorithm and percentile arithmetic. +List _epochGRefMedians( + List mag, + int perEpoch, + int halfWin, + int firstEpoch, + int lastEpoch, +) { + final rows = lastEpoch - firstEpoch; + if (rows == 0) return []; + + final n = mag.length; + var lo = math.max(0, firstEpoch * perEpoch - halfWin); + var hi = math.min(n, firstEpoch * perEpoch + perEpoch + halfWin); + final sorted = []; + for (var i = lo; i < hi; i++) { + _insertEpochSorted(sorted, mag[i]); + } + + final out = List.filled(rows, 1.0); + for (var row = 0; row < rows; row++) { + out[row] = percentileSorted(sorted, 50) ?? 1.0; + if (row + 1 == rows) break; + + final epoch = firstEpoch + row + 1; + final nextLo = math.max(0, epoch * perEpoch - halfWin); + final nextHi = math.min(n, epoch * perEpoch + perEpoch + halfWin); + while (lo < nextLo) { + _removeEpochSorted(sorted, mag[lo]); + lo++; + } + while (hi < nextHi) { + _insertEpochSorted(sorted, mag[hi]); + hi++; + } + } + return out; +} + +void _insertEpochSorted(List sorted, double value) { + var lo = 0; + var hi = sorted.length; + while (lo < hi) { + final mid = lo + (hi - lo) ~/ 2; + if (sorted[mid].compareTo(value) < 0) { + lo = mid + 1; + } else { + hi = mid; + } + } + sorted.insert(lo, value); +} + +void _removeEpochSorted(List sorted, double value) { + var lo = 0; + var hi = sorted.length; + while (lo < hi) { + final mid = lo + (hi - lo) ~/ 2; + if (sorted[mid].compareTo(value) < 0) { + lo = mid + 1; + } else { + hi = mid; + } + } + sorted.removeAt(lo); +} + /// The staging DECISION layer: per-epoch features → stages + deep flags. /// /// Extracted from [cardioStager] verbatim (no behavioural change) so the rules @@ -590,6 +924,13 @@ class CardioEpochFeatures { /// [remScoreCut] / [deepScoreCut] default to the shipped constants and exist so /// a calibration harness can sweep them on a DEVELOPMENT split and report the /// held-out result. Production callers should not pass them. +// Reformulates cardioStager() feature extraction and classification: van Hees +// 2013 motion and Long 2016 / Fonseca 2015/2018 cardiorespiratory staging. +// Internal fitted weights and rules remain ESTIMATE with low confidence. +// Session-onset anchoring and post-gap re-anchoring preserve the pre-bump grid; +// shared arithmetic preserves labels except for separately documented +// RR/spectral changes. Cached rows require matching dependencies; missing phases +// are recomputed. CardioStagerResult classifyCardioEpochs( CardioEpochFeatures f, { int epochSec = _epochSec, @@ -794,6 +1135,10 @@ CardioStagerResult classifyCardioEpochs( // ── classify ─────────────────────────────────────────────────────────────── final stages = List.filled(nEpoch, SleepStage.wake); final deepFlag = List.filled(nEpoch, false); + // Evidence only: recorded for stageMinuteRanges, never read back here, so + // the labels below are unchanged by keeping them. + final remScores = List.filled(nEpoch, null); + final deepScores = List.filled(nEpoch, null); for (var e = 0; e < nEpoch; e++) { final hrMed = hrMedLocal[e]; final hrArousal = hrArousalLocal[e]; @@ -867,18 +1212,7 @@ CardioStagerResult classifyCardioEpochs( final hrSdZ = (sleepHrSd.length >= 4 && hrSd[e] > 0) ? hrSdScale?.z(hrSd[e]) : null; - // Weighted mean over the axes that are actually MEASURABLE this epoch, so - // a night with no usable RR falls back to the hrSd axis alone instead of - // silently scoring every epoch as if the missing axes voted "no". - double? score(List<(double?, double)> terms) { - var num = 0.0, den = 0.0; - for (final (z, w) in terms) { - if (z == null || z.isNaN) continue; - num += z * w; - den += w.abs(); - } - return den == 0 ? null : num / den; - } + final score = weightedAxisScore; final remScore = score([ (rkZ, _wRemRk), @@ -892,6 +1226,8 @@ CardioStagerResult classifyCardioEpochs( (sdnnZ, -_wDeepSdnn), (lfhfZ, -_wDeepLfhf), ]); + remScores[e] = remScore; + deepScores[e] = deepScore; // Atonia and the HR floor are RETAINED as gates, not folded into the score: // they are not weak evidence for REM, they are preconditions. A large @@ -985,9 +1321,26 @@ CardioStagerResult classifyCardioEpochs( ), deepFlag, conf, + remScores: remScores, + deepScores: deepScores, + hrCovered: [for (var e = 0; e < nEpoch; e++) !hr[e].isNaN], ); } +/// Weighted mean of robust-z axis terms over the axes that are actually +/// MEASURABLE this epoch, so a night with no usable RR falls back to the hrSd +/// axis alone instead of silently scoring every epoch as if the missing axes +/// voted "no". Null when no axis is measurable. Shared with [CausalStager]. +double? weightedAxisScore(List<(double?, double)> terms) { + var num = 0.0, den = 0.0; + for (final (z, w) in terms) { + if (z == null || z.isNaN) continue; + num += z * w; + den += w.abs(); + } + return den == 0 ? null : num / den; +} + /// Honest "cannot stage this window" result: NO epochs, no deep flags, zero /// confidence. Callers must treat an empty [StagerResult.stages] as UNSTAGED /// (see `advanced_stager._stageSessionCardio`), never as sleep. @@ -1071,7 +1424,26 @@ double _windowSdnn(List rrMs, List rrTsMs, final mid = (s + t) ~/ 2; if (mid >= accel.length) return empty; final centreMs = accel[mid].tsMs; - final lo = centreMs - halfWinMs, hi = centreMs + halfWinMs; + return cleanRrBeatsBetween( + rrMs, rrTsMs, centreMs - halfWinMs, centreMs + halfWinMs); +} + +/// The physiologic-gated RR beats whose time falls in `[loMs, hiMs]` +/// (INCLUSIVE at both ends), with each beat's time rebased to [loMs] in seconds. +/// +/// The body of [_cleanBeatsInWindow], lifted out so [CausalStager] gathers its +/// trailing windows through the SAME gate (300-2000 ms, successive jump +/// <= 200 ms) instead of a second copy. [rrTsMs] must be non-decreasing and the +/// same length as [rrMs]; an empty or mismatched pair yields no beats. +({List beats, List tsSec, List seam}) cleanRrBeatsBetween( + List rrMs, + List rrTsMs, + double loMs, + double hiMs, +) { + const empty = (beats: [], tsSec: [], seam: []); + if (rrMs.isEmpty || rrTsMs.length != rrMs.length) return empty; + final lo = loMs, hi = hiMs; // [rrTsMs] is ascending, so lower-bound into it and stop at the far edge // instead of walking every beat of the night once per epoch per feature. // @@ -1266,8 +1638,15 @@ void _websterRescore(List sm, int epochSec) { // one; see [_cleanBeatsInWindow]. final win = _cleanBeatsInWindow(rrMs, rrTsMs, accel, s, t, halfWinMs: 90 * 1000); - final beats = win.beats; // clean RR (ms) - final beatTsSec = win.tsSec; // matching beat times (s), rebased to window + return remFeaturesFromBeats(win.beats, win.tsSec, win.seam); +} + +/// LF/HF and R(k) from already-gated beats ([beats] in ms, [beatTsSec] their +/// times in seconds, any common origin, [seam] as [cleanRrBeatsBetween] marks +/// it). The numeric half of [_windowRemFeatures], shared with [CausalStager]; +/// nulls when fewer than 16 beats (the spectral stability gate). +({double? lfhf, double? rk}) remFeaturesFromBeats( + List beats, List beatTsSec, List seam) { if (beats.length < 16) return (lfhf: null, rk: null); // spectral stability gate // R(k): mean absolute successive difference of instantaneous HR (bpm). @@ -1276,7 +1655,7 @@ void _websterRescore(List sm, int epochSec) { double? prevIhr; for (var k = 0; k < beats.length; k++) { final ihr = 60000.0 / beats[k]; - if (prevIhr != null && !win.seam[k]) { + if (prevIhr != null && !seam[k]) { rkSum += (ihr - prevIhr).abs(); rkCnt++; } diff --git a/lib/src/onehz/sleep/causal_stager.dart b/lib/src/onehz/sleep/causal_stager.dart new file mode 100644 index 0000000..6d457f1 --- /dev/null +++ b/lib/src/onehz/sleep/causal_stager.dart @@ -0,0 +1,1082 @@ +// SLEEP — CAUSAL (online) 3-class stager: wake | nrem | rem | absent. +// +// WHY THIS EXISTS. [cardioStager] is a RETROSPECTIVE night scorer: every +// feature window is centred on its epoch (so it reads 90-150 s of the future), +// the motion / HR / z-score baselines are whole-night, and three post-passes +// (mode filter, Webster rescore, bout consolidation) all look BOTH ways. None +// of that exists at 06:40 when something has to decide, right now, whether the +// sleeper is in REM. Re-running the retrospective stager every tick is not a +// causal stager either: its newest epochs would be scored on truncated windows +// and a baseline that is still moving. This file is the online counterpart, +// built from the SAME features, weights and cutoffs wherever those can be +// computed from the past alone: +// +// same motion (ENMO vs a local 1 g reference), HR mean / SD, SDNN, +// R(k), LF/HF; the physiologic RR gate; the REM axis weights +// (kRemWeight*, the measured DREAMT effect sizes); the shipped +// REM cutoff (kDefaultRemScoreCut); the wake / atonia / HR-floor +// rules; minHrCoverage; the 0.6 confidence ceiling. +// made causal every centred window becomes a TRAILING window ending at the +// epoch's end (300 s SDNN; 180 s R(k) and LF/HF — their full +// lengths; 120 s for the 1 g reference, see [_gRefWinSec]; 90 min +// for the local HR gates); +// whole-night baselines become expanding baselines over what has +// been seen so far (capped at [CausalStagerConfig.historyEpochs]). +// dropped Webster rescore and bout consolidation: both need the FUTURE +// side of a bout. Nothing replaces them; a causal wake is +// therefore not bridged the way the retrospective one is, and +// [CausalStageObservation.runSec] says how long a stage has +// persisted so the caller can demand persistence itself. The +// 3-epoch mode filter is replaced by a trailing 2-of-3 vote. +// not carried the personal profile blend ([SleepUserProfile]) and the deep +// overlay. This is wake / nrem / rem, per-night-local only. +// +// WHAT IT IS NOT. A wrist autonomic ESTIMATE (tier ESTIMATE), never +// polysomnography, never an EEG-defined stage, never a diagnosis. The offline +// stager scores kappa 0.13 against PSG on the held-out DREAMT split; this is +// a causal cousin of that stager with fewer smoothing passes, and agreement +// with the offline hypnogram is a consistency check, not an accuracy figure. +// `confidence` is EVIDENCE QUALITY (HR coverage, RR coverage), capped at +// [kMaxSleepConfidence] — it is not a probability that the stage is right. +// +// THE CONTRACT +// +// CausalStager.observe(window, priorState) -> CausalStageObservation +// +// * CAUSAL. Only samples stamped strictly before window.nowMs are read (a +// sample stamped t describes [t, t+1 s)); the rest are counted and ignored. An epoch is staged only after it has fully closed, +// and its features look backwards only. +// * REPLAYABLE. Pure and deterministic: no clock, no randomness, no I/O, no +// hash-order dependence. The caller supplies `nowMs`. The same sequence of +// windows gives byte-identical observations and states. +// * REPLACEMENT-TOLERANT. A window is authoritative for the time span it +// covers, per channel: stored samples inside that span are replaced by the +// window's. Re-sending a window, overlapping windows, and corrected +// re-sends converge on the state a clean feed would have produced. Samples +// older than 600 s behind the newest stored sample are IGNORED (counted in +// trace['ignoredStale']): their features are frozen. +// * SERIALISABLE. [CausalStagerState] is plain data (toJson/fromJson) and +// bounded: ~15 min of raw samples plus [CausalStagerConfig.historyEpochs] +// feature rows. It crosses an isolate boundary as-is. +// * HONEST. It abstains, with a [CausalAbstention] reason, instead of +// guessing: no / stale / regressing clock, off-skin, missing HR or accel, +// thin coverage, or warm-up evidence too thin. An absent stage always has +// confidence 0 and runSec 0, and a previous stage is never carried across +// an abstention. +// +// INPUT CONTRACT. 1 Hz HR and accel (one sample per second bucket; whole-second +// windows) plus RR beats whose timestamps are the record second (`rr_ts_ms`). +// Start from `prior == null` at in-bed time: the API stages whatever it is +// given and has no sleep-onset detector, so feeding it the waking afternoon +// would build its baselines from waking physiology. +// +// LATENCY, stated plainly: a stage describes the newest CLOSED 30 s epoch, and +// its RR features are trailing 3-5 min windows whose centres sit 1.5-2.5 min +// behind it — the price of not looking ahead. See docs/CAUSAL_STAGER.md for the +// measured agreement and the false- / late-trigger behaviour. + +import 'dart:math' as math; +import '../types.dart'; +import '../util.dart'; +import 'cardio_stager.dart' + show + cleanRrBeatsBetween, + kDefaultRemScoreCut, + kRemWeightHrSd, + kRemWeightLfhf, + kRemWeightRk, + kRemWeightSdnn, + minHrCoverage, + remFeaturesFromBeats, + weightedAxisScore; +import 'segment.dart' show kMaxSleepConfidence; + +// Published staging basis inherited from cardioStager(): Long 2016; Fonseca +// 2015/2018. This unpublished past-only adaptation is an internal rule, +// ESTIMATE with low +// confidence; trailing windows and baselines intentionally differ from +// retrospective staging. +/// Epoch length (s). Fixed: every cutoff inherited from [cardioStager] is +/// specified at 30 s. +const int kCausalEpochSec = 30; + +// Trailing window lengths (s). The RR windows keep the FULL length of the +// centred windows in cardio_stager.dart (RMSSD/SDNN ±150 s, R(k)/LF-HF ±90 s). +// +// The 1 g reference is the one place the full length does NOT carry over. The +// retrospective reference is a 330 s window CENTRED on the epoch, so after a +// static posture change it adapts as soon as the epoch reaches the change (it +// reads the future half). A trailing window of the same length has to wait +// until MORE THAN HALF of it is the new posture, i.e. 150 s of every posture +// change reads as "motion" — and a few consecutive big-move epochs are what the +// wake gate calls awake. That is the same posture artifact cardio_stager.dart's +// header describes, re-created by causality. A shorter trailing reference +// adapts in half its length while still being longer than the movement bouts +// it must not absorb. 120 s was picked from {300, 180, 120, 90} s by replaying +// the one in-tree night (test/onehz/fixtures; `tool/causal_stager_validate.dart`): +// wake over-call vs the offline stager fell 104 -> 83 -> 62 -> 52 min and kappa +// rose .305 -> .335 -> .369 -> .372. It is the one number in this file chosen +// on data, on a single night: treat the validation as optimistic by that much. +const int _gRefWinSec = 120; +const int _sdnnWinSec = 300; +const int _remWinSec = 180; + +/// Local HR gate window, epochs (90 min — one ultradian cycle; the retrospective +/// stager uses ±180 epochs, the causal one only the past half). +const int _hrWinEpochs = 180; + +/// Raw samples retained in the state (s). Must cover the longest feature +/// lookback ([_sdnnWinSec]) plus [_replaceHorizonSec]. +const int _rawRetainSec = 900; + +/// Samples this far behind the newest stored sample or older are ignored: a +/// row older than this has lost part of its lookback and is frozen. +const int _replaceHorizonSec = _rawRetainSec - _sdnnWinSec; + +/// Persisted-state schema version. Bump on any change to the state JSON. +const int kCausalStateVersion = 1; + +/// What the causal stager can say. [absent] always comes with an +/// [CausalAbstention] and zero confidence. +enum CausalStage { wake, nrem, rem, absent } + +/// Why the stager declined to name a stage. Checked roughly in declaration +/// order; [lowCoverage] is also raised for thin recent history once the +/// warm-up count is met. +enum CausalAbstention { + /// `nowMs` is earlier than a time already observed, or not finite. State is + /// returned untouched: acting on it would mean reading the future. + clockRegressed, + + /// No sample of any kind has ever been received. + noEvidence, + + /// Nothing at all arrived within [CausalStagerConfig.maxEvidenceAgeSec] of + /// `nowMs` (disconnect, strap out of range, delivery stalled). + staleEvidence, + + /// The newest closed epoch is mostly `hr == 0` (off-skin). Never read as + /// bradycardia. + offWrist, + + /// No valid HR in the newest closed epoch (absent, NaN or negative). + missingHr, + + /// No valid accel in the newest closed epoch. HR is checked first, so an + /// epoch with neither reports [missingHr]. + missingAccel, + + /// HR or accel covers under [minHrCoverage] of the epoch's seconds, or the + /// recent history is mostly unusable. + lowCoverage, + + /// Fewer than [CausalStagerConfig.warmupEpochs] usable epochs seen so far; + /// the baselines would be built on nothing. `note` reads + /// `warmup:have=H,need=N`. + warmup, +} + +/// Tunables. Carried inside [CausalStagerState] so a persisted state keeps the +/// settings it was built with. +class CausalStagerConfig { + /// Usable epochs required before any stage is named. 40 = 20 min. + /// An engineering floor (enough epochs for the robust-z baselines and the + /// local HR gates to exist), not a physiological claim. + final int warmupEpochs; + + /// Nothing received for longer than this ⇒ [CausalAbstention.staleEvidence]. + final double maxEvidenceAgeSec; + + /// Feature rows kept, which is also the span of the expanding baselines + /// (360 epochs = 3 h). + final int historyEpochs; + + /// REM decision cutoff on the weighted robust-z REM score. Defaults to the + /// shipped [kDefaultRemScoreCut]; exposed for calibration tooling only. + final double remScoreCut; + + const CausalStagerConfig({ + this.warmupEpochs = 40, + this.maxEvidenceAgeSec = 120, + this.historyEpochs = 360, + this.remScoreCut = kDefaultRemScoreCut, + }) : assert(warmupEpochs > 0), + assert(historyEpochs >= warmupEpochs), + assert(maxEvidenceAgeSec > 0); + + static const CausalStagerConfig defaults = CausalStagerConfig(); + + Map toJson() => { + 'warmup_epochs': warmupEpochs, + 'max_evidence_age_sec': maxEvidenceAgeSec, + 'history_epochs': historyEpochs, + 'rem_score_cut': remScoreCut, + }; + + static CausalStagerConfig? fromJson(Object? j) { + if (j is! Map) return null; + final w = j['warmup_epochs'], a = j['max_evidence_age_sec']; + final h = j['history_epochs'], c = j['rem_score_cut']; + if (w is! num || a is! num || h is! num || c is! num) return null; + if (w <= 0 || h < w || a <= 0 || !c.isFinite) return null; + return CausalStagerConfig( + warmupEpochs: w.toInt(), + maxEvidenceAgeSec: a.toDouble(), + historyEpochs: h.toInt(), + remScoreCut: c.toDouble(), + ); + } +} + +/// One call's worth of new (or re-sent) samples plus the decision time. +/// +/// Whole-second windows. Samples may overlap earlier windows (a re-send): see +/// the replacement rule in the file header. Timestamps are absolute epoch ms, +/// the same clock as [nowMs]. +class CausalSampleWindow { + /// Decision time (absolute epoch ms), supplied by the caller — this package + /// has no clock. Samples stamped at or after it are ignored (a sample + /// stamped t describes the second [t, t+1 s), which is not over yet). + final double nowMs; + final List hr; // hr == 0 is off-skin + final List accel; + final RrSeries rr; // tsMs = record second in ms; rrMs = interval + + CausalSampleWindow({ + required this.nowMs, + this.hr = const [], + this.accel = const [], + RrSeries? rr, + }) : rr = rr ?? RrSeries(const [], const []); +} + +/// One closed 30 s epoch's features. Plain data. +/// +/// `null` means "not measurable", never zero. [label] is the raw (unsmoothed) +/// stage decided when the epoch closed, from baselines built on epochs up to +/// and including this one — null when the epoch was unusable or the stager was +/// still warming up. +class CausalEpochRow { + /// Absolute index: floor(startMs / 30000). + final int epoch; + final int hrN; // seconds with valid on-skin HR (> 0) + final int hrOffN; // seconds with hr == 0 + final int accelN; // seconds with valid accel + final double? motion; // mean ENMO vs trailing 1 g reference (g) + final double? hr; // mean valid HR (bpm) + final double? hrSd; // SD of valid HR (bpm); null under 2 samples + final double? sdnn; // ms, trailing 300 s; null under 5 clean beats + final double? rk; // mean |ΔIHR| (bpm), trailing 180 s; null under 16 beats + final double? lfhf; // trailing 180 s; null under 16 beats + final CausalStage? label; + + const CausalEpochRow({ + required this.epoch, + required this.hrN, + required this.hrOffN, + required this.accelN, + this.motion, + this.hr, + this.hrSd, + this.sdnn, + this.rk, + this.lfhf, + this.label, + }); + + /// Seconds of valid HR / accel needed for the epoch to count. + static final int minSeconds = (minHrCoverage * kCausalEpochSec).ceil(); + + bool get offWrist => hrOffN > 0 && hrOffN >= hrN; + bool get usable => + !offWrist && hrN >= minSeconds && accelN >= minSeconds && hr != null; + + CausalEpochRow withLabel(CausalStage? l) => CausalEpochRow( + epoch: epoch, + hrN: hrN, + hrOffN: hrOffN, + accelN: accelN, + motion: motion, + hr: hr, + hrSd: hrSd, + sdnn: sdnn, + rk: rk, + lfhf: lfhf, + label: l, + ); + + List toJson() => + [epoch, hrN, hrOffN, accelN, motion, hr, hrSd, sdnn, rk, lfhf, label?.index]; + + static CausalEpochRow? fromJson(Object? j) { + if (j is! List || j.length != 11) return null; + for (var i = 0; i < 4; i++) { + if (j[i] is! num) return null; + } + double? d(Object? v) => v == null ? null : (v is num ? v.toDouble() : double.nan); + final vals = [for (var i = 4; i < 10; i++) d(j[i])]; + if (vals.any((v) => v != null && !v.isFinite)) return null; + final li = j[10]; + if (li != null && (li is! int || li < 0 || li > 2)) return null; + return CausalEpochRow( + epoch: (j[0] as num).toInt(), + hrN: (j[1] as num).toInt(), + hrOffN: (j[2] as num).toInt(), + accelN: (j[3] as num).toInt(), + motion: vals[0], + hr: vals[1], + hrSd: vals[2], + sdnn: vals[3], + rk: vals[4], + lfhf: vals[5], + label: li == null ? null : CausalStage.values[li as int], + ); + } +} + +/// The carried state. Plain data: JSON-serialisable and isolate-sendable. +/// +/// Treat every list as immutable. Raw tails are columns: +/// hr [second, bpm] accel [second, x, y, z] (valid only) +/// rr [ts_ms, rr_ms] +class CausalStagerState { + final CausalStagerConfig config; + + /// Latest `nowMs` observed; a smaller `nowMs` is a clock regression. + final double? lastNowMs; + + /// Latest second carrying valid on-skin HR AND valid accel, if ever. + final int? lastUsableSec; + final List> hrTail; + final List> accelTail; + final List> rrTail; + final List rows; // ascending by epoch, closed epochs only + + const CausalStagerState({ + required this.config, + this.lastNowMs, + this.lastUsableSec, + this.hrTail = const [], + this.accelTail = const [], + this.rrTail = const [], + this.rows = const [], + }); + + /// A fresh state — what `observe(window, null)` starts from. + factory CausalStagerState.initial( + [CausalStagerConfig config = CausalStagerConfig.defaults]) => + CausalStagerState(config: config); + + /// Newest stored second across all channels, if any. + int? get newestSec { + int? m; + void up(num v) { + final s = v.floor(); + if (m == null || s > m!) m = s; + } + + if (hrTail.isNotEmpty) up(hrTail.last[0]); + if (accelTail.isNotEmpty) up(accelTail.last[0]); + if (rrTail.isNotEmpty) up(rrTail.last[0] / 1000.0); + return m; + } + + Map toJson() => { + 'v': kCausalStateVersion, + 'cfg': config.toJson(), + 'last_now_ms': lastNowMs, + 'last_usable_sec': lastUsableSec, + 'hr': hrTail, + 'accel': accelTail, + 'rr': rrTail, + 'rows': [for (final r in rows) r.toJson()], + }; + + /// Null for anything that is not a current-version state this class wrote — + /// the caller starts fresh (which only costs a warm-up), it never gets a + /// guessed-at state. + static CausalStagerState? fromJson(Object? j) { + if (j is! Map || j['v'] != kCausalStateVersion) return null; + final cfg = CausalStagerConfig.fromJson(j['cfg']); + if (cfg == null) return null; + final now = j['last_now_ms'], usable = j['last_usable_sec']; + if (now != null && (now is! num || !now.isFinite)) return null; + if (usable != null && usable is! num) return null; + List>? cols(Object? v, int width) { + if (v is! List) return null; + final out = >[]; + for (final e in v) { + if (e is! List || e.length != width) return null; + final r = []; + for (final x in e) { + if (x is! num || !x.isFinite) return null; + r.add(x.toDouble()); + } + out.add(r); + } + return out; + } + + final hr = cols(j['hr'], 2), ac = cols(j['accel'], 4), rr = cols(j['rr'], 2); + final rowsJ = j['rows']; + if (hr == null || ac == null || rr == null || rowsJ is! List) return null; + final rows = []; + for (final r in rowsJ) { + final row = CausalEpochRow.fromJson(r); + if (row == null) return null; + rows.add(row); + } + return CausalStagerState( + config: cfg, + lastNowMs: (now as num?)?.toDouble(), + lastUsableSec: (usable as num?)?.toInt(), + hrTail: hr, + accelTail: ac, + rrTail: rr, + rows: rows, + ); + } +} + +/// One answer. [nextState] must be passed to the next call. +class CausalStageObservation { + /// The stage of the newest CLOSED 30 s epoch, or [CausalStage.absent]. + final CausalStage stage; + + /// Evidence quality in [0.15, kMaxSleepConfidence] when staged, exactly 0 + /// when absent. Not a probability of being right. + final double confidence; + + /// ms between `nowMs` and the end of the newest second that carried valid + /// on-skin HR and valid accel; null when there never was one. Present even + /// when absent — it is the staleness the caller should log. + final double? evidenceAgeMs; + + final CausalStagerState nextState; + + /// Non-null exactly when [stage] is [CausalStage.absent]. + final CausalAbstention? abstentionReason; + + /// Start (absolute ms) of the epoch [stage] describes; null if none. + final double? epochStartMs; + + /// How long [stage] has held, in seconds (whole epochs, trailing 2-of-3 + /// vote). 0 when absent. A caller wanting a "stable REM candidate" should + /// threshold this; `remEpisodeMinMin` (5 min) is the retrospective stager's + /// shortest credible REM bout. + final double runSec; + + /// `warmup:have=H,need=N` while warming up; null otherwise. + final String? note; + + /// Plain-data decision trace (features, thresholds, counters) for logging. + final Map trace; + + const CausalStageObservation({ + required this.stage, + required this.confidence, + required this.evidenceAgeMs, + required this.nextState, + required this.abstentionReason, + required this.epochStartMs, + required this.runSec, + required this.note, + required this.trace, + }); + + /// The observation without the state (persist [nextState] separately). + Map toJson() => { + 'stage': stage.name, + 'confidence': round6(confidence) ?? 0.0, + 'evidence_age_ms': evidenceAgeMs, + 'abstention_reason': abstentionReason?.name, + 'epoch_start_ms': epochStartMs, + 'run_sec': runSec, + if (note != null) 'note': note, + 'trace': trace, + }; +} + +/// The causal stager. See the file header for the contract. +abstract final class CausalStager { + /// Fold [window] into [priorState] (null = start of night) and name the stage + /// of the newest closed epoch, or abstain. + static CausalStageObservation observe( + CausalSampleWindow window, CausalStagerState? priorState) { + final st = priorState ?? CausalStagerState.initial(); + final cfg = st.config; + final now = window.nowMs; + + // 0. Clock. Going backwards would mean answering about a time the state has + // already moved past; an unusable clock cannot age anything. + if (!now.isFinite || (st.lastNowMs != null && now < st.lastNowMs!)) { + return _absent(st, CausalAbstention.clockRegressed); + } + + // 1. Merge (causal filter, span replacement) ----------------------------- + final m = _merge(st, window); + + // 2. Feature rows --------------------------------------------------------- + final newest = m.newestSec; + final kLatest = newest == null ? -1 : ((newest + 1) ~/ kCausalEpochSec) - 1; + final rows = _rebuildRows(st, cfg, m, kLatest); + + // 3. Bookkeeping that goes into the next state ---------------------------- + final usableSec = m.latestUsableSec ?? st.lastUsableSec; + final keepFromSec = (newest ?? 0) - _rawRetainSec; + final next = CausalStagerState( + config: cfg, + lastNowMs: now, + lastUsableSec: usableSec, + hrTail: [ + for (final e in m.hr) if (e[0] >= keepFromSec) e + ], + accelTail: [ + for (final e in m.accel) if (e[0] >= keepFromSec) e + ], + rrTail: [ + for (final e in m.rr) if (e[0] >= keepFromSec * 1000.0) e + ], + rows: rows, + ); + + final ageMs = usableSec == null + ? null + : math.max(0.0, now - (usableSec + 1) * 1000.0); + final trace = { + 'ignoredFuture': m.ignoredFuture, + 'ignoredStale': m.ignoredStale, + 'rows': rows.length, + }; + + CausalStageObservation no(CausalAbstention why, + {String? note, CausalEpochRow? row}) => + _mk(next, why, ageMs, trace, note: note, row: row); + + // 4. Reasons to abstain, most fundamental first --------------------------- + if (newest == null) return no(CausalAbstention.noEvidence); + if (now - (newest + 1) * 1000.0 > cfg.maxEvidenceAgeSec * 1000.0) { + return no(CausalAbstention.staleEvidence); + } + final byEpoch = {for (final r in rows) r.epoch: r}; + final usableCount = rows.where((r) => r.usable).length; + if (kLatest < 0) { + return no(CausalAbstention.warmup, + note: _warmNote(0, cfg.warmupEpochs)); + } + final row = byEpoch[kLatest]; + trace['epoch'] = kLatest; + if (row == null) return no(CausalAbstention.missingHr); + trace['hrN'] = row.hrN; + trace['hrOffN'] = row.hrOffN; + trace['accelN'] = row.accelN; + if (row.offWrist) return no(CausalAbstention.offWrist, row: row); + if (row.hrN == 0) return no(CausalAbstention.missingHr, row: row); + if (row.accelN == 0) return no(CausalAbstention.missingAccel, row: row); + if (!row.usable) return no(CausalAbstention.lowCoverage, row: row); + trace['usableRows'] = usableCount; + if (usableCount < cfg.warmupEpochs) { + return no(CausalAbstention.warmup, + note: _warmNote(usableCount, cfg.warmupEpochs), row: row); + } + final firstEpoch = rows.first.epoch; + final from = math.max(firstEpoch, kLatest - cfg.warmupEpochs + 1); + final span = kLatest - from + 1; + var recentUsable = 0, recentRr = 0; + for (var e = from; e <= kLatest; e++) { + final r = byEpoch[e]; + if (r != null && r.usable) { + recentUsable++; + if (r.sdnn != null) recentRr++; + } + } + if (recentUsable / span < minHrCoverage) { + return no(CausalAbstention.lowCoverage, row: row); + } + + // 5. Stage: trailing 2-of-3 vote over the labels decided at close time ---- + CausalStage? voted(int e) { + final here = byEpoch[e]?.label; + if (here == null) return null; + final counts = {here: 1}; + for (final d in const [1, 2]) { + final l = byEpoch[e - d]?.label; + if (l != null) counts[l] = (counts[l] ?? 0) + 1; + } + var best = here; + for (final en in counts.entries) { + if (en.value > counts[best]!) best = en.key; + } + return best; // a tie keeps the newest label + } + + final stage = voted(kLatest); + if (stage == null) { + // Warm and usable, but the label was never decided (e.g. the epoch + // closed before warm-up completed and nothing has rebuilt it). + return no(CausalAbstention.warmup, + note: _warmNote(usableCount, cfg.warmupEpochs), row: row); + } + var run = 1; + while (voted(kLatest - run) == stage) { + run++; + } + + final hrCov = recentUsable / span; + final rrCov = recentRr / span; + final conf = + ((0.35 + 0.25 * rrCov) * hrCov).clamp(0.15, kMaxSleepConfidence); + trace['raw'] = row.label?.name; + trace['motion'] = row.motion; + trace['hr'] = row.hr; + trace['hrSd'] = row.hrSd; + trace['sdnn'] = row.sdnn; + trace['rk'] = row.rk; + trace['lfhf'] = row.lfhf; + trace['hrCoverage'] = hrCov; + trace['rrCoverage'] = rrCov; + final why = _explain(rows, kLatest, cfg); + if (why != null) trace.addAll(why); + + return CausalStageObservation( + stage: stage, + confidence: conf.toDouble(), + evidenceAgeMs: ageMs, + nextState: next, + abstentionReason: null, + epochStartMs: kLatest * kCausalEpochSec * 1000.0, + runSec: run * kCausalEpochSec.toDouble(), + note: null, + trace: trace, + ); + } + + static String _warmNote(int have, int need) => 'warmup:have=$have,need=$need'; + + static CausalStageObservation _mk(CausalStagerState next, + CausalAbstention why, double? ageMs, Map trace, + {String? note, CausalEpochRow? row}) { + return CausalStageObservation( + stage: CausalStage.absent, + confidence: 0, + evidenceAgeMs: ageMs, + nextState: next, + abstentionReason: why, + epochStartMs: + row == null ? null : row.epoch * kCausalEpochSec * 1000.0, + runSec: 0, + note: note, + trace: trace, + ); + } + + static CausalStageObservation _absent( + CausalStagerState st, CausalAbstention why) { + return CausalStageObservation( + stage: CausalStage.absent, + confidence: 0, + evidenceAgeMs: null, + nextState: st, + abstentionReason: why, + epochStartMs: null, + runSec: 0, + note: null, + trace: { + 'ignoredFuture': 0, + 'ignoredStale': 0, + 'rows': st.rows.length, + }, + ); + } +} + +// ── Merge ──────────────────────────────────────────────────────────────────── + +class _Merged { + final List> hr; // [sec, bpm], ascending by sec + final List> accel; // [sec, x, y, z] + final List> rr; // [ts, rr], ascending by ts + final int ignoredFuture; + final int ignoredStale; + final int? dirtyFromSec; // earliest second touched by the window + final int? newestSec; + final int? latestUsableSec; + const _Merged(this.hr, this.accel, this.rr, this.ignoredFuture, + this.ignoredStale, this.dirtyFromSec, this.newestSec, + this.latestUsableSec); +} + +int _secOf(double tsMs) => (tsMs / 1000.0).floor(); + +_Merged _merge(CausalStagerState st, CausalSampleWindow w) { + final now = w.nowMs; + final prevNewest = st.newestSec; + // Anything this far behind (or more) the newest stored sample has frozen + // features: ignore it rather than half-apply it. + final staleBeforeMs = prevNewest == null + ? double.negativeInfinity + : (prevNewest - _replaceHorizonSec) * 1000.0; + var future = 0, stale = 0; + + // Time-acceptance, shared by all channels. + bool accept(double ts) { + if (!ts.isFinite) return false; + if (ts >= now) { + future++; + return false; + } + if (ts < staleBeforeMs) { + stale++; + return false; + } + return true; + } + + int? dirty; + void touch(int sec) { + if (dirty == null || sec < dirty!) dirty = sec; + } + + // HR: window span replaces stored span; last duplicate second wins. + final hrIn = {}; + int? hrLo, hrHi; + for (final s in w.hr) { + if (!accept(s.tsMs)) continue; + final sec = _secOf(s.tsMs); + hrLo = hrLo == null ? sec : math.min(hrLo, sec); + hrHi = hrHi == null ? sec : math.max(hrHi, sec); + if (s.hr.isFinite && s.hr >= 0) { + hrIn[sec] = s.hr; + } else { + hrIn.remove(sec); // an unusable sample is a missing one + } + } + final hr = _spliceSec(st.hrTail, hrLo, hrHi, { + for (final e in hrIn.entries) e.key: [e.key.toDouble(), e.value] + }); + if (hrLo != null) touch(hrLo); + + // Accel. + final acIn = >{}; + int? acLo, acHi; + for (final a in w.accel) { + if (!accept(a.tsMs)) continue; + final sec = _secOf(a.tsMs); + acLo = acLo == null ? sec : math.min(acLo, sec); + acHi = acHi == null ? sec : math.max(acHi, sec); + if (a.valid && a.x.isFinite && a.y.isFinite && a.z.isFinite) { + acIn[sec] = [sec.toDouble(), a.x, a.y, a.z]; + } else { + acIn.remove(sec); + } + } + final accel = _spliceSec(st.accelTail, acLo, acHi, acIn); + if (acLo != null) touch(acLo); + + // RR: ties on a second are normal, so no per-key upsert — the window's span + // replaces the stored span wholesale. + final n = math.min(w.rr.tsMs.length, w.rr.rrMs.length); + final rrIn = >[]; + double? rrLo, rrHi; + for (var i = 0; i < n; i++) { + final ts = w.rr.tsMs[i]; + if (!accept(ts)) continue; + rrLo = rrLo == null ? ts : math.min(rrLo, ts); + rrHi = rrHi == null ? ts : math.max(rrHi, ts); + final v = w.rr.rrMs[i]; + if (v.isFinite && v > 0) rrIn.add([ts, v]); + } + // Stable by timestamp; ties keep window order. + final order = List.generate(rrIn.length, (i) => i) + ..sort((a, b) { + final c = rrIn[a][0].compareTo(rrIn[b][0]); + return c != 0 ? c : a.compareTo(b); + }); + final rr = >[]; + if (rrLo == null) { + rr.addAll(st.rrTail); + } else { + for (final e in st.rrTail) { + if (e[0] < rrLo) rr.add(e); + } + for (final i in order) { + rr.add(rrIn[i]); + } + for (final e in st.rrTail) { + if (e[0] > rrHi!) rr.add(e); + } + touch(_secOf(rrLo)); + } + + int? newest; + void up(int s) { + if (newest == null || s > newest!) newest = s; + } + + if (hr.isNotEmpty) up(hr.last[0].toInt()); + if (accel.isNotEmpty) up(accel.last[0].toInt()); + if (rr.isNotEmpty) up(_secOf(rr.last[0])); + + // Newest second with valid on-skin HR AND valid accel. + int? usable; + { + final hrBy = {for (final e in hr) e[0].toInt(): e[1]}; + for (var i = accel.length - 1; i >= 0; i--) { + final sec = accel[i][0].toInt(); + final bpm = hrBy[sec]; + if (bpm != null && bpm > 0) { + usable = sec; + break; + } + } + } + return _Merged(hr, accel, rr, future, stale, dirty, newest, usable); +} + +/// Replace the stored entries with `lo <= sec <= hi` by [incoming]; keep the +/// rest; result ascending by second. +List> _spliceSec( + List> stored, int? lo, int? hi, Map> incoming) { + if (lo == null) return stored; + final out = >[ + for (final e in stored) if (e[0] < lo) e + ]; + final keys = incoming.keys.toList()..sort(); + for (final k in keys) { + out.add(incoming[k]!); + } + for (final e in stored) { + if (e[0] > hi!) out.add(e); + } + return out; +} + +// ── Feature rows ───────────────────────────────────────────────────────────── + +List _rebuildRows( + CausalStagerState st, CausalStagerConfig cfg, _Merged m, int kLatest) { + final oldestKept = kLatest - cfg.historyEpochs + 1; + final byEpoch = { + for (final r in st.rows) + if (r.epoch >= oldestKept && r.epoch <= kLatest) r.epoch: r + }; + final dirty = m.dirtyFromSec; + if (dirty != null && kLatest >= 0) { + final kLo = math.max((dirty / kCausalEpochSec).floor(), math.max(0, oldestKept)); + if (kLo <= kLatest) { + final hrBy = {for (final e in m.hr) e[0].toInt(): e[1]}; + final acBy = {for (final e in m.accel) e[0].toInt(): e}; + final rrTs = [for (final e in m.rr) e[0]]; + final rrV = [for (final e in m.rr) e[1]]; + final gRefMag = { + for (final e in acBy.entries) + e.key: math.sqrt(e.value[1] * e.value[1] + + e.value[2] * e.value[2] + + e.value[3] * e.value[3]) + }; + for (var k = kLo; k <= kLatest; k++) { + final row = _buildRow(k, hrBy, gRefMag, rrTs, rrV); + if (row == null) { + byEpoch.remove(k); + } else { + byEpoch[k] = row; + } + } + // Decide labels in epoch order, each on baselines built from rows up to + // and including its own epoch — never later ones. + for (var k = kLo; k <= kLatest; k++) { + final row = byEpoch[k]; + if (row == null) continue; + final d = _decide(byEpoch, k, cfg); + byEpoch[k] = row.withLabel(d?.stage); + } + } + } + final keys = byEpoch.keys.toList()..sort(); + return [for (final k in keys) byEpoch[k]!]; +} + +CausalEpochRow? _buildRow(int k, Map hrBy, + Map mag, List rrTs, List rrV) { + final s0 = k * kCausalEpochSec; + final hrVals = []; + var off = 0; + var anyHr = false; + for (var s = s0; s < s0 + kCausalEpochSec; s++) { + final b = hrBy[s]; + if (b == null) continue; + anyHr = true; + if (b > 0) { + hrVals.add(b); + } else { + off++; + } + } + final epochMags = [ + for (var s = s0; s < s0 + kCausalEpochSec; s++) + if (mag[s] != null) mag[s]! + ]; + if (!anyHr && epochMags.isEmpty) return null; + + // Motion: ENMO against the trailing 300 s median of |a| (the causal form of + // the retrospective stager's centred local 1 g reference). + double? motion; + if (epochMags.isNotEmpty) { + final ref = [ + for (var s = s0 + kCausalEpochSec - _gRefWinSec; + s < s0 + kCausalEpochSec; + s++) + if (mag[s] != null) mag[s]! + ]; + final g = median(ref) ?? 1.0; + var sum = 0.0; + for (final v in epochMags) { + final d = v - g; + sum += d > 0 ? d : 0.0; + } + motion = sum / epochMags.length; + } + + // RR: trailing windows ending at the epoch's last millisecond (a beat stamped + // at the next second boundary belongs to the next epoch). + final endMs = (s0 + kCausalEpochSec) * 1000.0 - 1e-3; + double? sdnn, rk, lfhf; + if (rrTs.isNotEmpty) { + final w300 = cleanRrBeatsBetween(rrV, rrTs, endMs - _sdnnWinSec * 1000.0, endMs); + if (w300.beats.length >= 5) sdnn = stddev(w300.beats); + final w180 = cleanRrBeatsBetween(rrV, rrTs, endMs - _remWinSec * 1000.0, endMs); + final rem = remFeaturesFromBeats(w180.beats, w180.tsSec, w180.seam); + rk = rem.rk; + lfhf = rem.lfhf; + } + + return CausalEpochRow( + epoch: k, + hrN: hrVals.length, + hrOffN: off, + accelN: epochMags.length, + motion: motion, + hr: hrVals.isEmpty ? null : mean(hrVals), + hrSd: hrVals.length >= 2 ? stddev(hrVals) : null, + sdnn: sdnn, + rk: rk, + lfhf: lfhf, + ); +} + +// ── Decision (the retrospective stager's rules, on expanding baselines) ────── + +class _Decision { + final CausalStage stage; + final double? remScore; + final double hrArousal, hrP25, hrMed; + final bool bigMove; + const _Decision(this.stage, this.remScore, this.hrArousal, this.hrP25, + this.hrMed, this.bigMove); +} + +/// Decide epoch [k] using only usable rows with epoch in (k - historyEpochs, k]. +/// Null when [k] is unusable or fewer than `warmupEpochs` usable rows exist. +_Decision? _decide( + Map byEpoch, int k, CausalStagerConfig cfg) { + final row = byEpoch[k]; + if (row == null || !row.usable) return null; + final base = [ + for (var e = k - cfg.historyEpochs + 1; e <= k; e++) + if (byEpoch[e] != null && byEpoch[e]!.usable) byEpoch[e]! + ]; + if (base.length < cfg.warmupEpochs) return null; + + // Motion thresholds: whole-window scalars (as offline: only the absolute + // magnitude references needed to be local). MAD 0 ⇒ the motion axis abstains. + final motSample = [for (final r in base) r.motion!]; + final motMed = median(motSample) ?? 0; + final motMadRaw = mad(motSample) ?? 0; + final motUsable = motMadRaw > 0 && motMadRaw.isFinite; + final motMad = motUsable ? motMadRaw : 0.0; + final stillCut = motMed + 1.5 * motMad; + final bigMoveCut = motMed + 5.0 * motMad; + bool still(CausalEpochRow r) => !motUsable || r.motion! <= stillCut; + bool bigMove(CausalEpochRow r) => motUsable && r.motion! > bigMoveCut; + + final sleepHr = [ + for (final r in base) + if (still(r)) r.hr! + ]; + final hrAll = [for (final r in base) r.hr!]; + final hrMedGlobal = median(sleepHr) ?? mean(hrAll)!; + + // Local trailing HR gates (retrospective: ±180 epochs; causal: past 180). + final win = [ + for (final r in base) + if (r.epoch >= k - _hrWinEpochs && still(r)) r.hr! + ]; + final sd = stddev(win); // before the sort: float sums are order-dependent + win.sort(); + final m = percentileSorted(win, 50); + double hrMed = hrMedGlobal, hrArousal = hrMedGlobal + 6.0, hrP25 = hrMedGlobal; + if (m != null) { + hrMed = m; + hrArousal = m + math.max(6.0, (sd ?? 6)); + hrP25 = percentileSorted(win, 25) ?? m; + } + + final hr = row.hr!; + final big = bigMove(row); + final prev = byEpoch[k - 1]; + final bigPrev = prev != null && prev.usable && bigMove(prev); + final hrUp = hr >= hrArousal; + final bigMoveWake = big && (hr >= hrMed || bigPrev); + if (hrUp || bigMoveWake) { + return _Decision(CausalStage.wake, null, hrArousal, hrP25, hrMed, big); + } + + // REM: weighted robust-z score over the axes measurable this epoch. + final sleepRk = [for (final r in base) if (still(r) && r.rk != null) r.rk!]; + final sleepSdnn = [for (final r in base) if (still(r) && r.sdnn != null) r.sdnn!]; + final sleepLfhf = [for (final r in base) if (still(r) && r.lfhf != null) r.lfhf!]; + final sleepHrSd = [ + for (final r in base) + if (still(r) && r.hrSd != null && r.hrSd! > 0) r.hrSd! + ]; + final rkZ = (sleepRk.length >= 4 && row.rk != null) + ? RobustScale.of(sleepRk)?.z(row.rk!) + : null; + final sdnnZ = (sleepSdnn.length >= 4 && row.sdnn != null) + ? RobustScale.of(sleepSdnn)?.z(row.sdnn!) + : null; + final lfhfZ = (sleepLfhf.length >= 4 && row.lfhf != null) + ? RobustScale.of(sleepLfhf)?.z(row.lfhf!) + : null; + final hrSdZ = (sleepHrSd.length >= 4 && row.hrSd != null && row.hrSd! > 0) + ? RobustScale.of(sleepHrSd)?.z(row.hrSd!) + : null; + final remScore = weightedAxisScore([ + (rkZ, kRemWeightRk), + (sdnnZ, kRemWeightSdnn), + (hrSdZ, kRemWeightHrSd), + (lfhfZ, kRemWeightLfhf), + ]); + final atonia = !big; + final hrTowardWake = hr >= hrP25; + final isRem = + remScore != null && remScore > cfg.remScoreCut && atonia && hrTowardWake; + return _Decision(isRem ? CausalStage.rem : CausalStage.nrem, remScore, + hrArousal, hrP25, hrMed, big); +} + +/// Trace fields for the newest epoch's decision, recomputed (not stored). +Map? _explain( + List rows, int k, CausalStagerConfig cfg) { + final byEpoch = {for (final r in rows) r.epoch: r}; + final d = _decide(byEpoch, k, cfg); + if (d == null) return null; + return { + 'remScore': d.remScore, + 'remCut': cfg.remScoreCut, + 'hrArousal': d.hrArousal, + 'hrP25': d.hrP25, + 'hrMedLocal': d.hrMed, + 'bigMove': d.bigMove, + }; +} diff --git a/lib/src/onehz/sleep/circadian_np.dart b/lib/src/onehz/sleep/circadian_np.dart index c80c478..ce0bfb1 100644 --- a/lib/src/onehz/sleep/circadian_np.dart +++ b/lib/src/onehz/sleep/circadian_np.dart @@ -17,9 +17,11 @@ // HONESTY: anchor circadian phase on the nocturnal trough (L5), never the // daytime peak — L5 is the trough-anchored marker the catalog requires. -import 'dart:math' as math; +import 'dart:typed_data'; +import '../../aggregate/aggregator.dart'; import '../types.dart'; import '../util.dart'; +import 'circadian_np_aggregator.dart'; class CircadianNp { final double interdailyStability; // IS @@ -60,112 +62,11 @@ Metric circadianNonparametric( int? m10Epochs, int? l5Epochs, }) { - const inputs = ['activity_or_hr_epochs']; - final n = x.length; - if (epochsPerDay <= 0 || n < epochsPerDay) { - return const Metric.absent( - tier: Tier.high, - inputs_used: inputs, - note: 'need ≥1 full day of epochs for circadian metrics', - ); - } - // M10 = 10 h, L5 = 5 h scaled to the epoch resolution. - final mEp = m10Epochs ?? (epochsPerDay * 10 ~/ 24); - final lEp = l5Epochs ?? (epochsPerDay * 5 ~/ 24); - - final grand = mean(x)!; - var ssTot = 0.0; - for (final v in x) { - final d = v - grand; - ssTot += d * d; - } - if (ssTot == 0) { - return const Metric.absent( - tier: Tier.high, - inputs_used: inputs, - note: 'no variance in activity series', - ); - } - - // IS: between-epoch-of-day variance vs total variance. - final p = epochsPerDay; - final hourMeans = List.filled(p, 0); - final hourCounts = List.filled(p, 0); - for (var i = 0; i < n; i++) { - final h = i % p; - hourMeans[h] += x[i]; - hourCounts[h]++; - } - for (var h = 0; h < p; h++) { - if (hourCounts[h] > 0) hourMeans[h] /= hourCounts[h]; - } - var ssHour = 0.0; - for (var h = 0; h < p; h++) { - final d = hourMeans[h] - grand; - ssHour += d * d; - } - final is_ = (n * ssHour) / (p * ssTot); - - // IV: mean-squared successive difference vs variance. - var ssDiff = 0.0; - for (var i = 1; i < n; i++) { - final d = x[i] - x[i - 1]; - ssDiff += d * d; - } - final iv = (n * ssDiff) / ((n - 1) * ssTot); - - // M10 / L5: best contiguous window over the AVERAGE day profile (hourMeans), - // circularly, so a window straddling midnight is allowed. - final m10 = _bestWindow(hourMeans, mEp, maximize: true); - final l5 = _bestWindow(hourMeans, lEp, maximize: false); - final ra = (m10.value + l5.value) > 0 - ? (m10.value - l5.value) / (m10.value + l5.value) - : 0.0; - - final days = n / epochsPerDay; - final conf = (days / 7.0).clamp(0.3, 0.95); - return Metric( - value: CircadianNp( - interdailyStability: is_.clamp(0, 1), - intradailyVariability: math.max(0, iv), - m10: m10.value, - l5: l5.value, - relativeAmplitude: ra.clamp(0, 1), - m10StartEpoch: m10.start, - l5StartEpoch: l5.start, - ), - confidence: conf, - tier: Tier.high, - inputs_used: inputs, - note: 'IS/IV/RA/L5/M10 (van Someren 1999); circadian phase anchored on the ' - 'L5 nocturnal trough, not the daytime peak', - ); -} - -class _Window { - final double value; - final int start; - const _Window(this.value, this.start); -} - -/// Best (max or min) mean over a contiguous circular window of length [w]. -_Window _bestWindow(List profile, int w, {required bool maximize}) { - final p = profile.length; - if (w >= p) { - return _Window(mean(profile) ?? 0, 0); - } - double? best; - var bestStart = 0; - for (var s = 0; s < p; s++) { - var sum = 0.0; - for (var k = 0; k < w; k++) { - sum += profile[(s + k) % p]; - } - final m = sum / w; - if (best == null || (maximize ? m > best : m < best)) { - best = m; - bestStart = s; - } - } - return _Window(best ?? 0, bestStart); + // Runs [circadianNpAggregator] over the whole series in one slice. IS is the + // between-epoch-of-day variance over the total variance, IV the mean-squared + // successive difference over the variance, and M10 / L5 the best circular + // window over the average day profile (so a window may straddle midnight). + return circadianNpAggregator(epochsPerDay, + m10Epochs: m10Epochs, l5Epochs: l5Epochs) + .run(EpochSeries(Float64List.fromList(x))); } diff --git a/lib/src/onehz/sleep/circadian_np_aggregator.dart b/lib/src/onehz/sleep/circadian_np_aggregator.dart new file mode 100644 index 0000000..a987de1 --- /dev/null +++ b/lib/src/onehz/sleep/circadian_np_aggregator.dart @@ -0,0 +1,378 @@ +import 'dart:typed_data'; + +import '../../aggregate/aggregator.dart'; +import '../../aggregate/json_support.dart'; +import '../../aggregate/primitives.dart'; +import '../types.dart'; +import 'circadian_np.dart'; +import 'circadian_window.dart'; + +/// Epoch-binned activity (or HR) values. +/// +/// [firstIndex] is the absolute index of row zero, so the epoch of day of a +/// row survives slicing and joining. +class EpochSeries extends Batch { + EpochSeries(this.x, {this.firstIndex = 0}); + + factory EpochSeries.fromJson(Object? json) { + final map = decodeMap(json); + return EpochSeries(decodeDoubles(field(map, 'x')), + firstIndex: decodeCount(field(map, 'firstIndex'))); + } + + final Float64List x; + final int firstIndex; + + @override + Object? toJson() => {'x': encodeDoubles(x), 'firstIndex': firstIndex}; + + @override + int get length => x.length; + + @override + EpochSeries copyRange(int start, int end) => EpochSeries( + Float64List(end - start)..setRange(0, end - start, x, start), + firstIndex: firstIndex + start); + + @override + EpochSeries join(EpochSeries other) { + assert(other.firstIndex == firstIndex + length); + return EpochSeries( + Float64List(length + other.length) + ..setRange(0, length, x) + ..setRange(length, length + other.length, other.x), + firstIndex: firstIndex); + } +} + +/// Sum of squared successive differences. Run under `Lookback(2)`, which +/// feeds the pair across each chunk seam. +class _SquaredDiffs extends Aggregator { + const _SquaredDiffs(); + @override + double empty() => 0.0; + @override + double addSlice(double state, EpochSeries batch, int start, int end) { + for (var i = start + 1; i < end; i++) { + final d = batch.x[i] - batch.x[i - 1]; + state += d * d; + } + return state; + } + + @override + double combine(double left, double right) => left + right; + @override + double finish(double state) => state; + @override + bool get persistable => true; + @override + Object? encodeState(double state) => encodeDouble(state); + @override + double decodeState(Object? json) => decodeDouble(json); +} + +/// A direct, non-skipping sum. Circadian's reference calculation deliberately +/// lets NaN and infinities propagate through every sum they touch. +class _RawSum extends Aggregator { + const _RawSum(this.column); + + final Float64List Function(EpochSeries batch) column; + + @override + double empty() => 0.0; + + @override + double addSlice(double state, EpochSeries batch, int start, int end) { + final values = column(batch); + for (var i = start; i < end; i++) { + state += values[i]; + } + return state; + } + + @override + double combine(double left, double right) => left + right; + + @override + double finish(double state) => state; + + @override + bool get persistable => true; + + // This retains the Sum checkpoint shape, so pre-existing finite checkpoints + // remain readable. + @override + Object? encodeState(double state) => + {'sum': encodeDouble(state), 'correction': 0.0}; + + @override + double decodeState(Object? json) { + final map = decodeMap(json); + return decodeDouble(field(map, 'sum')) + + decodeDouble(field(map, 'correction')); + } +} + +class _CircadianStats { + int count = 0; + double sum = 0.0; + double mean = 0.0; + double m2 = 0.0; +} + +/// Raw total plus mergeable centred squares for circadian variance. +/// +/// The raw total reproduces the reference grand mean. Centred squares retain +/// the finite-input stability of the former Moments block, while an individual +/// non-finite row immediately poisons them just as the reference's second +/// pass does. +class _CircadianStatsAggregator + extends Aggregator<_CircadianStats, EpochSeries, _CircadianStats> { + const _CircadianStatsAggregator(this.column); + + final Float64List Function(EpochSeries batch) column; + + @override + _CircadianStats empty() => _CircadianStats(); + + @override + _CircadianStats addSlice( + _CircadianStats state, EpochSeries batch, int start, int end) { + final values = column(batch); + for (var i = start; i < end; i++) { + final value = values[i]; + state.sum += value; + state.count++; + if (!value.isFinite) { + // The reference later evaluates (value - grand) squared for this row. + // Whatever grand is, this makes ssTot non-finite (and usually NaN). + state.m2 = double.nan; + state.mean = state.sum / state.count; + continue; + } + final delta = value - state.mean; + state.mean += delta / state.count; + state.m2 += delta * (value - state.mean); + } + return state; + } + + @override + _CircadianStats combine(_CircadianStats left, _CircadianStats right) { + if (right.count == 0) return left; + if (left.count == 0) { + left + ..count = right.count + ..sum = right.sum + ..mean = right.mean + ..m2 = right.m2; + return left; + } + final leftCount = left.count; + final count = leftCount + right.count; + final delta = right.mean - left.mean; + left + ..sum += right.sum + ..mean += delta * right.count / count + ..m2 += right.m2 + delta * delta * leftCount * right.count / count + ..count = count; + return left; + } + + @override + _CircadianStats finish(_CircadianStats state) => state; + + @override + bool get persistable => true; + + @override + Object? encodeState(_CircadianStats state) => { + 'version': 1, + 'type': 'CircadianStats', + 'count': state.count, + 'sum': encodeDouble(state.sum), + 'mean': encodeDouble(state.mean), + 'm2': encodeDouble(state.m2), + }; + + @override + _CircadianStats decodeState(Object? json) { + final map = decodeMap(json); + final type = field(map, 'type'); + if (field(map, 'version') != 1 || type is! String) { + throw const FormatException('Unsupported circadian statistics checkpoint'); + } + final result = _CircadianStats()..count = decodeCount(field(map, 'count')); + if (type == 'CircadianStats') { + result + ..sum = decodeDouble(field(map, 'sum')) + ..mean = decodeDouble(field(map, 'mean')) + ..m2 = decodeDouble(field(map, 'm2')); + if ((result.m2.isFinite && result.m2 < 0) || + (result.count == 0 && + (result.sum != 0 || result.mean != 0 || result.m2 != 0))) { + throw const FormatException('Invalid circadian statistics'); + } + return result; + } + if (type != 'RunningMoments') { + throw const FormatException('Unsupported circadian statistics checkpoint'); + } + // Older checkpoints used RunningMoments. Their finite mean is sufficient + // to resume, even though it did not preserve the reference's raw sum. + final origin = decodeDouble(field(map, 'origin')); + final meanOffset = decodeDouble(field(map, 'meanOffset')); + result + ..mean = origin + meanOffset + ..sum = result.mean * result.count + ..m2 = decodeDouble(field(map, 'm2')); + if (!origin.isFinite || + !meanOffset.isFinite || + (result.m2.isFinite && result.m2 < 0) || + (result.count == 0 && + (origin != 0 || meanOffset != 0 || result.m2 != 0))) { + throw const FormatException('Invalid circadian statistics'); + } + return result; + } +} + +/// Mergeable nonparametric circadian metrics, matching +/// [circadianNonparametric]. +// Witting 1990 / van Someren 1999 nonparametric circadian indices, now also the +// circadianNonparametric() one-slice path. Phase sums/counts, centered variance +// and successive squared differences reproduce the formulas. Checked relative +// tolerances are 1e-12 for integer fixtures and 1e-9 for real fixtures; real- +// valued near-ties can select different L5/M10 start epochs. +Aggregator> circadianNpAggregator( + int epochsPerDay, { + int? m10Epochs, + int? l5Epochs, +}) { + const inputs = ['activity_or_hr_epochs']; + const tooShort = Metric.absent( + tier: Tier.high, + inputs_used: inputs, + note: 'need ≥1 full day of epochs for circadian metrics', + ); + if (epochsPerDay <= 0) { + return Count().map((_) => tooShort); + } + final p = epochsPerDay; + final mEp = m10Epochs ?? (p * 10 ~/ 24); + final lEp = l5Epochs ?? (p * 5 ~/ 24); + Float64List column(EpochSeries b) => b.x; + final core = both( + both(Count(), _CircadianStatsAggregator(column)), + both( + GroupBy( + (EpochSeries batch, int row) => (batch.firstIndex + row) % p, + both(Count(), _RawSum(column)), + ), + Lookback(2, const _SquaredDiffs(), + batchFromJson: EpochSeries.fromJson), + ), + ); + return validateCheckpoint(core, (json) { + final state = decodeList(json); + if (state.length != 2) throw const FormatException('Expected two states'); + final totals = decodeList(state[0]); + final grouped = decodeList(state[1]); + if (totals.length != 2 || grouped.length != 2) { + throw const FormatException('Unexpected circadian checkpoint state'); + } + final count = decodeCount(totals[0]); + final statistics = decodeMap(totals[1]); + if (decodeCount(field(statistics, 'count')) != count) { + throw const FormatException('Circadian statistics disagree with total'); + } + final groups = decodeMap(grouped[0]); + final lookback = decodeMap(grouped[1]); + if (decodeCount(field(lookback, 'length')) != count) { + throw const FormatException('Lookback length disagrees with epoch count'); + } + final first = field(lookback, 'first'); + if (count == 0) { + if (first != null || groups.isNotEmpty) { + throw const FormatException('Circadian groups without epochs'); + } + return; + } + if (first == null) throw const FormatException('Missing first circadian epoch'); + final firstEpoch = EpochSeries.fromJson(first); + if (firstEpoch.length != 1) { + throw const FormatException('Expected one first circadian epoch'); + } + final start = firstEpoch.firstIndex % p; + final expectedGroups = count < p ? count : p; + if (groups.length != expectedGroups) { + throw const FormatException('Unexpected epoch-of-day groups'); + } + final fullDays = count ~/ p; + final remainder = count % p; + var groupTotal = 0; + for (final entry in groups.entries) { + final bin = int.tryParse(entry.key); + if (bin == null || bin < 0 || bin >= p) { + throw const FormatException('Epoch-of-day group is outside the record'); + } + final offset = (bin - start + p) % p; + final expected = fullDays + (offset < remainder ? 1 : 0); + final groupState = decodeList(entry.value); + if (groupState.length != 2 || + decodeCount(groupState[0]) != expected || + expected == 0 || + groupTotal > count - expected) { + throw const FormatException('Epoch-of-day group count disagrees'); + } + groupTotal += expected; + } + if (groupTotal != count) { + throw const FormatException('Epoch-of-day groups disagree with total'); + } + }).map((result) { + final ((n, stats), (byEpoch, ssDiff)) = result; + if (n < p) return tooShort; + final grand = stats.sum / n; + final ssTot = stats.m2; + if (ssTot == 0) { + return const Metric.absent( + tier: Tier.high, + inputs_used: inputs, + note: 'no variance in activity series', + ); + } + final profile = List.filled(p, 0); + var ssHour = 0.0; + for (var h = 0; h < p; h++) { + final (count, sum) = byEpoch[h]!; + profile[h] = sum / count; + final d = profile[h] - grand; + ssHour += d * d; + } + final is_ = (n * ssHour) / (p * ssTot); + final iv = (n * ssDiff) / ((n - 1) * ssTot); + final m10 = bestCircularWindow(profile, mEp, maximize: true); + final l5 = bestCircularWindow(profile, lEp, maximize: false); + final ra = (m10.value + l5.value) > 0 + ? (m10.value - l5.value) / (m10.value + l5.value) + : 0.0; + return Metric( + value: CircadianNp( + interdailyStability: is_.clamp(0, 1), + intradailyVariability: iv < 0 ? 0 : iv, + m10: m10.value, + l5: l5.value, + relativeAmplitude: ra.clamp(0, 1), + m10StartEpoch: m10.start, + l5StartEpoch: l5.start, + ), + confidence: (n / p / 7.0).clamp(0.3, 0.95), + tier: Tier.high, + inputs_used: inputs, + note: 'IS/IV/RA/L5/M10 (van Someren 1999); circadian phase anchored on ' + 'the L5 nocturnal trough, not the daytime peak', + ); + }); +} diff --git a/lib/src/onehz/sleep/circadian_window.dart b/lib/src/onehz/sleep/circadian_window.dart new file mode 100644 index 0000000..221d59d --- /dev/null +++ b/lib/src/onehz/sleep/circadian_window.dart @@ -0,0 +1,34 @@ +import '../util.dart'; + +/// Mean and start epoch of a circular profile window. +class CircularWindow { + final double value; + final int start; + const CircularWindow(this.value, this.start); +} + +/// Best (max or min) mean over a contiguous circular window of length [w]. +// Unchanged extraction of circadianNonparametric() circular L5/M10 search, +// Witting 1990 / van Someren 1999. Traversal, arithmetic and first-winner tie +// handling are unchanged. +CircularWindow bestCircularWindow(List profile, int w, + {required bool maximize}) { + final p = profile.length; + if (w >= p) { + return CircularWindow(mean(profile) ?? 0, 0); + } + double? best; + var bestStart = 0; + for (var s = 0; s < p; s++) { + var sum = 0.0; + for (var k = 0; k < w; k++) { + sum += profile[(s + k) % p]; + } + final m = sum / w; + if (best == null || (maximize ? m > best : m < best)) { + best = m; + bestStart = s; + } + } + return CircularWindow(best ?? 0, bestStart); +} diff --git a/lib/src/onehz/sleep/epoch_rows.dart b/lib/src/onehz/sleep/epoch_rows.dart new file mode 100644 index 0000000..cdd485a --- /dev/null +++ b/lib/src/onehz/sleep/epoch_rows.dart @@ -0,0 +1,826 @@ +// Anchored 30 s epoch feature rows, accumulated. +// test/onehz/epoch_rows_test.dart and epoch_rows_anchor_test.dart pin the +// contract. +// +// * Epochs are [kEpochMs] steps from an anchor. The first gap-free run (the +// run structure of valid accelerometer coverage, with the stager's 60 s +// carry-forward as the gap threshold) is anchored at `anchorMs` when given +// (the staging onset: grid option 2), else at its first sample. A run that +// follows a gap longer than the carry re-anchors at its own first sample, +// as the pre-bump stager did. A run's anchor is final as soon as it is +// known. Rows start at the anchor, or at the epoch holding the first +// sample when the anchor lies before it; earlier samples only serve as +// feature context. +// * A row's features are `cardioEpochFeatures`' for that epoch: motion, hr, +// hrSd from the epoch's own seconds (motion against the local 1 g median +// of the +-150 s of accel magnitudes around it), rmssd / sdnn from the +// +-150 s RR window centred on the epoch, lfhf / rk from the +-90 s one. +// Lookback therefore reaches 165 s (accel, HR) and 150 s (RR) past an +// epoch's edges. +// * A row is FINAL once every sample it can read has arrived (newest sample +// is at least ~3 min past the epoch); the last few rows are PROVISIONAL and +// are recomputed with the data clipped at the newest sample. +// * Chunk invariance is exact: any chunking, with a save/restore through JSON +// text between chunks, gives the same rows as one fold. + +import 'dart:math' as math; +import 'dart:typed_data'; + +import '../../aggregate/json_support.dart'; +import 'advanced_stager.dart' show AdvancedSleepStager, StageSegment; +import 'accounting.dart' show SleepStage; +import 'cardio_stager.dart'; +import 'cardio_row_cache.dart'; +import '../types.dart'; +import '../foundations/rr_correction_kernel.dart' show SortedMultiset; +import '../util.dart'; + +// Reformulates cardioStager() feature extraction and classification: van Hees +// 2013 motion and Long 2016 / Fonseca 2015/2018 cardiorespiratory staging. +// Internal fitted weights and rules remain ESTIMATE with low confidence. +// Session-onset anchoring and post-gap re-anchoring preserve the pre-bump grid; +// shared arithmetic preserves labels except for separately documented +// RR/spectral changes. Cached rows require matching dependencies; missing phases +// are recomputed. +/// Epoch length, ms. +const int kEpochMs = 30000; + +/// Seconds a valid accelerometer sample covers: the stager's bounded +/// carry-forward plus the sample's own second. A valid sample within this many +/// seconds of the previous one continues the run; a later one starts a new run. +const int _kCoverSec = AdvancedSleepStager.maxAccelCarryForwardSec + 1; + +/// One gap-free run of valid accelerometer coverage and its epoch grid. +class _Run { + _Run(this.startSec, this.endSec, this.anchorMs, this.cursorMs); + int startSec, endSec; // coverage [startSec, endSec) in seconds + final int anchorMs; // first sample; epochs are anchorMs + k * kEpochMs + int cursorMs; // next epoch start to generate +} + +/// Raw features of one epoch; NaN means not measurable. +class EpochFeatureRow { + final int startMs; + final double motion, hr, hrSd, rmssd, lfhf, rk, sdnn; + final int _hrSeconds; + const EpochFeatureRow(this.startMs, this.motion, this.hr, this.hrSd, + this.rmssd, this.lfhf, this.rk, this.sdnn, + {int hrSeconds = 0}) + : _hrSeconds = hrSeconds; + + List _encode() => [ + startMs, + for (final v in [motion, hr, hrSd, rmssd, lfhf, rk, sdnn]) + encodeDouble(v), + _hrSeconds + ]; + + static EpochFeatureRow _decode(Object? value) { + final l = decodeList(value); + if (l.length != 9 || + l[0] is! int || + l[8] is! int || + (l[8] as int) < 0) { + throw const FormatException('Invalid epoch row'); + } + return EpochFeatureRow( + l[0] as int, + decodeDouble(l[1]), + decodeDouble(l[2]), + decodeDouble(l[3]), + decodeDouble(l[4]), + decodeDouble(l[5]), + decodeDouble(l[6]), + decodeDouble(l[7]), + hrSeconds: l[8] as int); + } +} + +/// Final feature rows and the raw time lookback needed by pending rows. +/// Each fold advances all input channels through the same watermark. A channel +/// omitted from a fold has no samples there, rather than arriving late later. +/// +/// [continuingAnchorMs]: the first valid accelerometer sample folded joins a +/// gap-free run whose first sample (at that time) is not being re-folded, so the +/// grid steps from there. Use it to start folding mid-run without computing the +/// run's earlier rows. +/// +/// [anchorMs]: the first run's grid passes through this absolute time (rows are +/// `anchorMs + k * kEpochMs`). Without either option the first sample anchors. +class EpochRows { + EpochRows({CardioRowCache? rowCache, int? continuingAnchorMs, int? anchorMs}) + : _cache = rowCache, + _seedMs = continuingAnchorMs, + _anchorReq = continuingAnchorMs == null ? anchorMs : null; + final CardioRowCache? _cache; + int? _seedMs; + int? _anchorReq; // requested first-run anchor, until the first run opens + final _hr = []; + final _accel = []; + final _rr = [], _rt = []; + final _final = []; + final _pending = {}; + // Accel coverage with the same bounded 60-second carry as the stager, and + // each run's grid. These cost one entry per actual outage, not one per second. + final _runs = <_Run>[]; + int _genRun = 0; // runs before this one have generated all their epochs + int? _newest; + + int? get newestMs => _newest; + + /// Anchor of the first run's grid, mod [kEpochMs]; null while unknown. + int? get phaseMs { + final a = _runs.isNotEmpty ? _runs.first.anchorMs : (_seedMs ?? _anchorReq); + return a == null ? null : a % kEpochMs; + } + + /// First sample (ms) of every gap-free run seen so far: the anchors of the + /// epoch grids. Final as soon as the run's first sample has been folded. + List get runAnchorsMs => [for (final r in _runs) r.anchorMs]; + + /// Start of the epoch (of the run whose coverage includes [tsMs]) that holds + /// [tsMs], or null when no run covers it. + int? epochStartMsFor(int tsMs) { + for (var i = _runs.length - 1; i >= 0; i--) { + final r = _runs[i]; + if (tsMs >= r.anchorMs) { + return tsMs < r.endSec * 1000 ? _epochIn(r, tsMs) : null; + } + } + return null; + } + + // The run whose grid holds the epoch starting at [startMs]. + _Run? _runAt(int startMs) { + for (var i = _runs.length - 1; i >= 0; i--) { + if (startMs >= _runs[i].anchorMs) return _runs[i]; + } + return null; + } + + static int _epochIn(_Run r, int tsMs) => + r.anchorMs + (tsMs - r.anchorMs) ~/ kEpochMs * kEpochMs; + + // Extends the last run or opens a new one for a valid accel sample at tsMs. + void _cover(double tsMs) { + final ms = tsMs.floor(); + final second = (tsMs / 1000).floor(); + if (_runs.isNotEmpty && second <= _runs.last.endSec) { + final run = _runs.last; + run.endSec = math.max(run.endSec, second + _kCoverSec); + return; + } + final seed = _runs.isEmpty ? _seedMs : null; + if (seed != null && seed > ms) { + throw ArgumentError('continuing anchor is after the first sample'); + } + final want = _runs.isEmpty ? _anchorReq : null; + if (seed == null && want != null) { + // Rows start at the anchor, or at the epoch holding the first sample + // when the anchor lies before it. + _runs.add(_Run(second, second + _kCoverSec, want, + want >= ms ? want : want + (ms - want) ~/ kEpochMs * kEpochMs)); + } else { + final anchor = seed ?? ms; + _runs.add(_Run((anchor / 1000).floor(), second + _kCoverSec, anchor, + anchor + (ms - anchor) ~/ kEpochMs * kEpochMs)); + } + _seedMs = null; + _anchorReq = null; + } + + // Opens the epochs that exist so far: starts on a run's grid that lie inside + // its coverage and are not after the newest sample. + void _generate() { + for (var i = _genRun; i < _runs.length; i++) { + final r = _runs[i]; + final limit = math.min(r.endSec * 1000, _newest! + 1); + for (; r.cursorMs < limit; r.cursorMs += kEpochMs) { + _pending.add(r.cursorMs); + } + // only the newest run can still grow + if (i + 1 < _runs.length) _genRun = i + 1; + } + } + + // Rows of the trailing carry (epochs that start after a run's last real + // sample) exist for staging only: the pre-bump stager classified them, with + // the carried vector as their accelerometer. The row API hides them. + List _visible(List rows) { + final out = []; + var i = 0; + for (final row in rows) { + while (i + 1 < _runs.length && _runs[i + 1].anchorMs <= row.startMs) { + i++; + } + if (row.startMs < (_runs[i].endSec - _kCoverSec + 1) * 1000) { + out.add(row); + } + } + return out; + } + + List get finalRows => List.unmodifiable(_visible(_final)); + + void fold( + {List hr = const [], + List accel = const [], + List rrMs = const [], + List rrTsMs = const [], + int? throughMs}) { + if (rrMs.length != rrTsMs.length) { + throw ArgumentError('RR columns differ in length'); + } + void check(Iterable times, double? last) { + for (final t in times) { + if (!t.isFinite || (last != null && t < last)) { + throw ArgumentError('samples must have finite, ascending times'); + } + if (_newest != null && t.floor() < _newest! - 240000 + kEpochMs) { + throw ArgumentError('sample precedes the settled watermark'); + } + last = t; + } + } + + check(hr.map((v) => v.tsMs), _hr.isEmpty ? null : _hr.last.tsMs); + check(accel.map((v) => v.tsMs), _accel.isEmpty ? null : _accel.last.tsMs); + check(rrTsMs, _rt.isEmpty ? null : _rt.last); + void observe(double t) { + final ms = t.floor(); + _newest = _newest == null ? ms : math.max(_newest!, ms); + } + + for (final h in hr) { + _hr.add(h); + observe(h.tsMs); + } + for (final a in accel) { + _accel.add(a); + observe(a.tsMs); + if (a.valid) _cover(a.tsMs); + } + _rr.addAll(rrMs); + _rt.addAll(rrTsMs); + for (final t in rrTsMs) { + observe(t); + } + if (throughMs != null) { + _newest = _newest == null ? throughMs : math.max(_newest!, throughMs); + } + if (_newest == null) return; + _generate(); + final starts = _pending.where((s) => s <= _newest! - 240000).toList() + ..sort(); + _final.addAll(_rows(starts)); + _pending.removeAll(starts); + // The oldest still-possible row starts after the final watermark; retain + // its widest left window, even when there are no pending rows in a gap. + final oldest = _newest! - 240000 + 1 - 165000; + _hr.removeWhere((h) => h.tsMs < oldest); + // 61 s more: carry from a dropped sample can still reach a live window. + _accel.removeWhere((a) => a.tsMs < oldest - 61000); + final drop = _lower(_rt, oldest.toDouble()); + if (drop > 0) { + _rr.removeRange(0, drop); + _rt.removeRange(0, drop); + } + } + + List _provisional() { + final starts = _pending.toList()..sort(); + return _rows(starts); + } + + List provisionalRows() => _visible(_provisional()); + + List allRows() => _visible([..._final, ..._provisional()]); + + // The accelerometer as the stager sees it: valid samples plus the bounded + // carry-forward (the last vector repeated each second, at most 60 s, until + // the next real sample), so features match a dense per-second array. + List _effAccel() { + final out = []; + AccelSample? prev; + void carry(int toSecIncl) { + final p = prev!; + final from = (p.tsMs / 1000).floor(); + final to = math.min(toSecIncl, from + AdvancedSleepStager.maxAccelCarryForwardSec); + for (var sec = from + 1; sec <= to; sec++) { + out.add(AccelSample(sec * 1000.0, p.x, p.y, p.z)); + } + } + + for (final a in _accel) { + if (!a.valid) continue; + if (prev != null) carry((a.tsMs / 1000).floor() - 1); + // Two valid samples at one time: the later one stands. + if (out.isNotEmpty && out.last.tsMs == a.tsMs) out.removeLast(); + out.add(a); + prev = a; + } + if (prev != null) carry(_newest! ~/ 1000); + return out; + } + + static int _lower(List times, double value) { + var a = 0, b = times.length; + while (a < b) { + final m = (a + b) >> 1; + if (times[m] < value) { + a = m + 1; + } else { + b = m; + } + } + return a; + } + + static int _sampleLower( + List samples, double value, double Function(T) time) { + var a = 0, b = samples.length; + while (a < b) { + final m = (a + b) >> 1; + if (time(samples[m]) < value) { + a = m + 1; + } else { + b = m; + } + } + return a; + } + + List _rows(List starts) { + if (starts.isEmpty) return []; + final eff = _effAccel(); + if (_cache != null) return [for (final s in starts) _row(s, eff)]; + // Slide the same sorted 330-second gravity window used by the batch + // extractor, computing each sample's magnitude once per fold. + final mag = [ + for (final a in eff) math.sqrt(a.x * a.x + a.y * a.y + a.z * a.z) + ]; + final sorted = SortedMultiset(); + var lo = 0, hi = 0; + final out = []; + for (final start in starts) { + // The gravity reference reads only the epoch's own run, as the pre-bump + // stager classified each run on its own. + final run = _runAt(start); + final nextLo = _sampleLower( + eff, + math.max(start - 150000.0, + run == null ? -double.infinity : run.startSec * 1000.0), + (a) => a.tsMs); + final nextHi = _sampleLower( + eff, + math.min(start + 180000.0, + run == null ? double.infinity : run.endSec * 1000.0), + (a) => a.tsMs); + while (lo < nextLo && lo < hi) { + sorted.remove(mag[lo]); + lo++; + } + lo = nextLo; + hi = math.max(hi, nextLo); + while (hi < nextHi) { + sorted.add(mag[hi]); + hi++; + } + out.add(_row(start, eff, reference: percentileSorted(sorted.a, 50) ?? 1.0)); + } + return out; + } + + EpochFeatureRow _row(int start, List eff, {double? reference}) { + final end = start + kEpochMs; + final run = _runAt(start); + final ab = _sampleLower( + eff, + math.max(start - (reference == null ? 150000.0 : 0.0), + run == null ? -double.infinity : run.startSec * 1000.0), + (a) => a.tsMs); + final ae = _sampleLower( + eff, + math.min(end + (reference == null ? 150000.0 : 0.0), + run == null ? double.infinity : run.endSec * 1000.0), + (a) => a.tsMs); + final hb = _sampleLower(_hr, start.toDouble(), (h) => h.tsMs); + final he = _sampleLower(_hr, end.toDouble(), (h) => h.tsMs); + final av = [ + for (var i = ab; i < ae; i++) + if (eff[i].valid) eff[i] + ]; + // One value per timestamp, the last one: a later record replaces an + // earlier one at the same time (a later 0 off-skin erases it). + final hv = [ + for (var i = hb; i < he; i++) + if (_hr[i].valid && (i + 1 == he || _hr[i + 1].tsMs != _hr[i].tsMs)) + _hr[i].hr + ]; + final low = start - 135000.0, high = start + 165000.0; + final rb = _lower(_rt, low), re = _lower(_rt, high + 0.001); + CardioRowKey? key; + if (_cache != null) { + var hash = 0xcbf29ce484222325; + final bits = ByteData(8); + void number(double v) { + bits.setFloat64(0, v, Endian.little); + hash = (hash ^ bits.getUint64(0, Endian.little)) * 0x100000001b3; + } + + number(0x45504f434833.toDouble()); // EPOCH3: 81b9b4b changed row semantics. + for (final a in av) { + number(a.tsMs); + number(a.x); + number(a.y); + number(a.z); + } + number(-1); + for (var i = hb; i < he; i++) { + number(_hr[i].tsMs); + number(_hr[i].hr); + } + number(-2); + for (var i = rb; i < re; i++) { + number(_rt[i]); + number(_rr[i]); + } + key = CardioRowKey(start ~/ 1000, 30, hash); + final hit = _cache.lookup(key); + if (hit != null) + return EpochFeatureRow(start, hit.motion, hit.hr, hit.hrSd, hit.rmssd, + hit.lfhf, hit.rk, hit.sdnn, + hrSeconds: hv.length); + } + final mag = [ + for (final a in av) math.sqrt(a.x * a.x + a.y * a.y + a.z * a.z) + ]; + final ref = reference ?? median(mag) ?? 1.0; + var motion = 0.0, count = 0; + for (var i = 0; i < av.length; i++) { + if (av[i].tsMs >= start && av[i].tsMs < end) { + final d = mag[i] - ref; + motion += d > 0 ? d : 0.0; + count++; + } + } + final beats = cleanRrBeatsBetween(_rr, _rt, low, high); + var ss = 0.0, pairs = 0; + for (var i = 1; i < beats.beats.length; i++) { + if (beats.seam[i]) continue; + final d = beats.beats[i] - beats.beats[i - 1]; + ss += d * d; + pairs++; + } + final remBeats = + cleanRrBeatsBetween(_rr, _rt, start - 75000.0, start + 105000.0); + final rem = + remFeaturesFromBeats(remBeats.beats, remBeats.tsSec, remBeats.seam); + final row = EpochFeatureRow( + start, + count == 0 ? double.nan : motion / count, + mean(hv) ?? double.nan, + hv.length >= 2 ? stddev(hv) ?? 0 : 0, + beats.beats.length >= 5 && pairs >= 4 + ? math.sqrt(ss / pairs) + : double.nan, + rem.lfhf ?? double.nan, + rem.rk ?? double.nan, + beats.beats.length >= 5 + ? stddev(beats.beats) ?? double.nan + : double.nan, + hrSeconds: hv.length); + if (key != null) + _cache!.insert( + key, + CardioEpochRow(row.motion, row.hr, row.hrSd, row.rmssd, row.lfhf, + row.rk, row.sdnn)); + return row; + } + + /// Classifies cached features with session-wide baselines. Features retain + /// their observed context when a session's start or end moves. + List stageWindow(int startSec, int endSec) { + if (endSec <= startSec) return const []; + final rows = [..._final, ..._provisional()]; + final pieces = []; + var cursor = startSec; + final byStart = {for (final r in rows) r.startMs: r}; + for (final run in _runs) { + final lo = math.max(math.max(startSec, run.startSec), run.anchorMs ~/ 1000), + hi = math.min(endSec, run.endSec); + if (hi <= lo) continue; + if (lo > cursor) pieces.add(StageSegment(cursor, lo, 'unobserved')); + // A carry-only epoch still consumes time in the classifier. + // Keep its absent HR in the coverage denominator; omitting it can turn + // an otherwise abstained run into apparently sufficient cardiac data. + final first = lo * 1000 <= run.anchorMs + ? run.anchorMs + : _epochIn(run, lo * 1000); + final chosen = [ + // Whole epochs only: a sub-epoch remainder is absorbed by the last one. + for (var s = first; s + kEpochMs <= hi * 1000; s += kEpochMs) + byStart[s] ?? + EpochFeatureRow(s, 0, double.nan, 0, double.nan, double.nan, + double.nan, double.nan) + ]; + final features = CardioEpochFeatures( + motion: [for (final r in chosen) r.motion], + hr: [for (final r in chosen) r.hr], + hrSd: [for (final r in chosen) r.hrSd], + rmssd: [for (final r in chosen) r.rmssd], + lfhf: [for (final r in chosen) r.lfhf], + rk: [for (final r in chosen) r.rk], + sdnn: [for (final r in chosen) r.sdnn]); + final result = classifyCardioEpochs(features); + final labels = result.base.stages; + // The classifier's own per-epoch HR coverage gate abstains (no labels), + // exactly as the pre-bump stager did; there is no second, per-second one. + if (hi - lo < 90 || labels.isEmpty) { + pieces.add(StageSegment(lo, hi, 'unobserved')); + } else { + var at = lo; + for (var i = 0; i < labels.length; i++) { + final label = switch (labels[i]) { + SleepStage.wake => 'wake', + SleepStage.rem => 'rem', + SleepStage.nrem => result.deepFlag[i] ? 'deep' : 'light', + }; + final until = i + 1 == labels.length + ? hi + : math.min(hi, chosen[i + 1].startMs ~/ 1000); + if (until > at) pieces.add(StageSegment(at, until, label)); + at = until; + } + } + cursor = hi; + } + if (cursor < endSec) pieces.add(StageSegment(cursor, endSec, 'unobserved')); + final out = []; + for (final p in pieces) { + if (out.isNotEmpty && + out.last.end == p.start && + out.last.stage == p.stage) { + final prev = out.removeLast(); + out.add(StageSegment(prev.start, p.end, p.stage)); + } else { + out.add(p); + } + } + return out; + } + + Map toJson() => { + 'type': 'EpochRows', + 'version': 4, + 'phase': phaseMs, + 'newest': _newest, + 'seed': _seedMs, + 'anchor': _anchorReq, + 'genRun': _genRun, + 'rows': [for (final r in _final) r._encode()], + 'pending': _pending.toList()..sort(), + 'hr': [ + for (final h in _hr) [h.tsMs, encodeDouble(h.hr)] + ], + 'accel': [ + for (final a in _accel) + [ + a.tsMs, + encodeDouble(a.x), + encodeDouble(a.y), + encodeDouble(a.z), + a.valid + ] + ], + 'rr': [for (final v in _rr) encodeDouble(v)], + 'rt': _rt, + 'runs': [ + for (final r in _runs) [r.startSec, r.endSec, r.anchorMs, r.cursorMs] + ], + }; + + factory EpochRows.fromJson(Map json) { + try { + if (json['type'] != 'EpochRows' || json['version'] != 4) { + throw const FormatException('Unsupported EpochRows checkpoint'); + } + final er = EpochRows( + continuingAnchorMs: field(json, 'seed') as int?, + anchorMs: field(json, 'anchor') as int?); + er._genRun = field(json, 'genRun') as int; + er._newest = field(json, 'newest') as int?; + er._final.addAll([ + for (final v in decodeList(field(json, 'rows'))) + EpochFeatureRow._decode(v) + ]); + final pending = decodeList(field(json, 'pending')).cast(); + er._pending.addAll(pending); + if (er._pending.length != pending.length) { + throw const FormatException('Duplicate pending epoch'); + } + for (final v in decodeList(field(json, 'hr'))) { + final l = decodeList(v); + if (l.length != 2) throw const FormatException('Invalid HR row'); + er._hr.add(HrSample(decodeDouble(l[0]), decodeDouble(l[1]))); + } + for (final v in decodeList(field(json, 'accel'))) { + final l = decodeList(v); + if (l.length != 5 || l[4] is! bool) + throw const FormatException('Invalid accel row'); + er._accel.add(AccelSample(decodeDouble(l[0]), decodeDouble(l[1]), + decodeDouble(l[2]), decodeDouble(l[3]), + valid: l[4] as bool)); + } + er._rr.addAll(decodeDoubles(field(json, 'rr'))); + er._rt.addAll(decodeDoubles(field(json, 'rt'))); + for (final v in decodeList(field(json, 'runs'))) { + final l = decodeList(v); + if (l.length != 4 || + l.any((x) => x is! int) || + (l[0] as int) >= (l[1] as int) || + (l[3] as int) < (l[2] as int) || + ((l[3] as int) - (l[2] as int)) % kEpochMs != 0 || + (er._runs.isNotEmpty && (l[0] as int) <= er._runs.last.endSec)) { + throw const FormatException('Invalid coverage run'); + } + er._runs.add(_Run(l[0] as int, l[1] as int, l[2] as int, l[3] as int)); + } + // Every epoch start belongs to the grid of the run that covers it. + bool onGrid(int s) { + for (var i = er._runs.length - 1; i >= 0; i--) { + final r = er._runs[i]; + if (s >= r.anchorMs) { + return s < r.endSec * 1000 && (s - r.anchorMs) % kEpochMs == 0; + } + } + return false; + } + bool ascending(Iterable values) { + double? prev; + for (final v in values) { + if (!v.isFinite || (prev != null && v < prev)) return false; + prev = v; + } + return true; + } + + if (er._rr.length != er._rt.length || + !ascending(er._hr.map((h) => h.tsMs)) || + !ascending(er._accel.map((a) => a.tsMs)) || + !ascending(er._rt) || + !ascending(er._final.map((r) => r.startMs.toDouble())) || + er._final.map((r) => r.startMs).toSet().length != er._final.length || + er._final.any((r) => !onGrid(r.startMs)) || + er._pending.any((s) => + !onGrid(s) || + (er._final.isNotEmpty && s <= er._final.last.startMs)) || + er._genRun > er._runs.length || + (er._runs.isEmpty && er._genRun != 0) || + (er._runs.isNotEmpty && er._genRun >= er._runs.length) || + (er._seedMs != null && er._runs.isNotEmpty) || + (er._anchorReq != null && er._runs.isNotEmpty) || + field(json, 'phase') != er.phaseMs || + (er._newest != null && + (er._final.any((r) => r.startMs > er._newest! - 240000) || + er._pending.any((s) => + s <= er._newest! - 240000 || s > er._newest!) || + er._pending.length > 20 || + er._hr.any((h) => h.tsMs >= er._newest! + 1) || + er._accel.any((a) => a.tsMs >= er._newest! + 1) || + er._rt.any((t) => t >= er._newest! + 1) || + er._runs.any((r) => r.endSec > er._newest! ~/ 1000 + _kCoverSec))) || + (er._newest == null && + (er._final.isNotEmpty || + er._pending.isNotEmpty || + er._hr.isNotEmpty || + er._accel.isNotEmpty || + er._rt.isNotEmpty))) { + throw const FormatException('Inconsistent EpochRows checkpoint'); + } + return er; + } on FormatException { + rethrow; + } catch (e) { + throw FormatException('Malformed EpochRows checkpoint: $e'); + } + } +} + +class _PhaseEntry { + _PhaseEntry(this.rows, this.anchorMs, this.hr, this.accel, this.rr); + final EpochRows rows; + final int anchorMs; // earliest anchor these rows were built for + int hr, accel, rr; // cumulative samples already folded, per channel +} + +/// Epoch rows keyed by grid phase (`anchorMs % kEpochMs`). The caller hands the +/// cumulative input (append-only between calls). A held phase folds only the +/// samples it has not consumed; an unheld phase, or a request for an earlier +/// anchor, is built from scratch (an exact recompute). At most [maxPhases] +/// phases are kept, least recently used dropped. +class EpochRowsMemo { + EpochRowsMemo({this.maxPhases = 2, CardioRowCache? rowCache}) + : _cache = rowCache; + final int maxPhases; + final CardioRowCache? _cache; + final _held = {}; // least recently used first + + /// Phases built from scratch, rebuilds included. + int phaseBuilds = 0; + + /// Samples handed to [EpochRows.fold] (hr + accel + rr). + int samplesFolded = 0; + + List get phases => List.unmodifiable(_held.keys); + + EpochRows rowsFor(int anchorMs, + {List hr = const [], + List accel = const [], + List rrMs = const [], + List rrTsMs = const []}) { + final phase = anchorMs % kEpochMs; + var e = _held.remove(phase); + if (e != null && + (anchorMs < e.anchorMs || + hr.length < e.hr || + accel.length < e.accel || + rrMs.length < e.rr)) { + e = null; + } + if (e == null) { + phaseBuilds++; + e = _PhaseEntry(EpochRows(anchorMs: anchorMs, rowCache: _cache), anchorMs, + 0, 0, 0); + } + if (hr.length > e.hr || accel.length > e.accel || rrMs.length > e.rr) { + e.rows.fold( + hr: hr.sublist(e.hr), + accel: accel.sublist(e.accel), + rrMs: rrMs.sublist(e.rr), + rrTsMs: rrTsMs.sublist(e.rr)); + samplesFolded += + hr.length - e.hr + accel.length - e.accel + rrMs.length - e.rr; + e.hr = hr.length; + e.accel = accel.length; + e.rr = rrMs.length; + } + _held[phase] = e; + while (_held.length > maxPhases) { + _held.remove(_held.keys.first); + } + return e.rows; + } + + Map toJson() => { + 'type': 'EpochRowsMemo', + 'version': 1, + 'maxPhases': maxPhases, + 'phases': [ + for (final en in _held.entries) + { + 'phase': en.key, + 'anchor': en.value.anchorMs, + 'consumed': [en.value.hr, en.value.accel, en.value.rr], + 'rows': en.value.rows.toJson(), + } + ], + }; + + factory EpochRowsMemo.fromJson(Map json) { + try { + if (json['type'] != 'EpochRowsMemo' || json['version'] != 1) { + throw const FormatException('Unsupported EpochRowsMemo checkpoint'); + } + final max = field(json, 'maxPhases'); + if (max is! int || max < 1) throw const FormatException('Invalid maxPhases'); + final m = EpochRowsMemo(maxPhases: max); + for (final v in decodeList(field(json, 'phases'))) { + final p = (v as Map).cast(); + final phase = field(p, 'phase'), anchor = field(p, 'anchor'); + final c = decodeList(field(p, 'consumed')); + if (phase is! int || + anchor is! int || + anchor % kEpochMs != phase || + c.length != 3 || + c.any((x) => x is! int || x < 0) || + m._held.containsKey(phase)) { + throw const FormatException('Invalid memo phase'); + } + final rows = EpochRows.fromJson( + (field(p, 'rows') as Map).cast()); + if (rows.phaseMs != phase) { + throw const FormatException('Memo phase contradicts its rows'); + } + m._held[phase] = + _PhaseEntry(rows, anchor, c[0] as int, c[1] as int, c[2] as int); + } + if (m._held.length > max) throw const FormatException('Too many phases'); + return m; + } on FormatException { + rethrow; + } catch (e) { + throw FormatException('Malformed EpochRowsMemo checkpoint: $e'); + } + } +} diff --git a/lib/src/onehz/sleep/hypnogram.dart b/lib/src/onehz/sleep/hypnogram.dart new file mode 100644 index 0000000..7889bdc --- /dev/null +++ b/lib/src/onehz/sleep/hypnogram.dart @@ -0,0 +1,79 @@ +/// A scored stage span. [stage] is normally 'wake', 'light', 'deep', or 'rem'. +class StageSegment { + final int start; + final int end; + final String stage; + const StageSegment(this.start, this.end, this.stage); + Map toJson() => {'start': start, 'end': end, 'stage': stage}; +} + +/// One detected sleep session and its hypnogram. +class SleepSession { + final int start; + final int end; + final double efficiency; // [0,1] + final List stages; + final int? restingHr; // lowest 5-min rolling-mean HR (bpm) + final double? avgHrv; // mean RMSSD over 5-min windows (ms) + const SleepSession({ + required this.start, + required this.end, + required this.efficiency, + required this.stages, + required this.restingHr, + required this.avgHrv, + }); +} + +/// AASM-style hypnogram metrics over a session. +class HypnogramMetrics { + final int tibS; + final int tstS; + final int sptS; + /// Sleep-onset latency (s). NULL when NOTHING staged as sleep — reporting + /// the whole time in bed as "time to fall asleep" alongside TST 0 is a + /// fabricated measurement, not an abstention. + final int? solS; + final double remLatencyS; // NaN if no REM + final int wasoS; + final double efficiency; // [0,1] + final int disturbances; + final double deepMin; + final double remMin; + final double lightMin; + final double deepPct; + final double remPct; + final double lightPct; + const HypnogramMetrics({ + required this.tibS, + required this.tstS, + required this.sptS, + required this.solS, + required this.remLatencyS, + required this.wasoS, + required this.efficiency, + required this.disturbances, + required this.deepMin, + required this.remMin, + required this.lightMin, + required this.deepPct, + required this.remPct, + required this.lightPct, + }); + Map toJson() => { + 'tib_s': tibS, + 'tst_s': tstS, + 'spt_s': sptS, + 'sol_s': solS, + 'rem_latency_s': remLatencyS.isNaN ? null : remLatencyS, + 'waso_s': wasoS, + 'efficiency': efficiency, + 'disturbances': disturbances, + 'deep_min': deepMin, + 'rem_min': remMin, + 'light_min': lightMin, + 'deep_pct': deepPct, + 'rem_pct': remPct, + 'light_pct': lightPct, + }; +} diff --git a/lib/src/onehz/sleep/hypnogram_aggregator.dart b/lib/src/onehz/sleep/hypnogram_aggregator.dart new file mode 100644 index 0000000..5710cd5 --- /dev/null +++ b/lib/src/onehz/sleep/hypnogram_aggregator.dart @@ -0,0 +1,190 @@ +import 'dart:math' as math; + +import '../../aggregate/aggregator.dart'; +import '../../aggregate/json_support.dart'; +import 'hypnogram.dart'; + +// Definition basis: Berry, R. B., et al. The AASM Manual for the Scoring of +// Sleep and Associated Events: Rules, Terminology and Technical Specifications. +// American Academy of Sleep Medicine. TST sums light/deep/REM durations; SOL +// runs from session start to first sleep; efficiency is TST / session duration. +// Application conventions: session bounds stand in for lights-off/on, stages +// are inferred rather than PSG-scored, and WASO counts explicit wake only +// between first sleep and the end of final sleep, excluding terminal wake. +// Gaps are not counted as wake. Sleep segments are summed without clipping +// to session bounds; efficiency is capped at 1. No sleep gives null SOL. +// Reformulation of AdvancedSleepStager.hypnogramMetrics(): sums integer stage +// durations and retains onset, final wake, first REM and wake-transition +// boundaries, reproducing its existing hypnogram accounting. +/// Mergeable AASM-style hypnogram metrics over start-sorted segments. +class HypnogramAggregator + extends Aggregator<_HypnogramState, ListBatch, HypnogramMetrics> { + HypnogramAggregator(this.sessionStart, this.sessionEnd); + + final int sessionStart; + final int sessionEnd; + + @override + _HypnogramState empty() => _HypnogramState(); + + @override + _HypnogramState addSlice( + _HypnogramState state, ListBatch batch, int start, int end) { + for (var i = start; i < end; i++) { + final segment = batch.rows[i]; + switch (segment.stage) { + case 'light': + case 'deep': + case 'rem': + final duration = segment.end - segment.start; + state.tst += duration; + if (segment.stage == 'light') state.light += duration; + if (segment.stage == 'deep') state.deep += duration; + if (segment.stage == 'rem') { + state.rem += duration; + if (!state.hasRem) { + state + ..hasRem = true + ..firstRemStart = segment.start; + } + } + if (!state.hasSleep) { + state + ..hasSleep = true + ..onset = segment.start; + } + state.sptEnd = segment.end; + case 'wake': + state.wakes.add((segment.start, segment.end)); + } + } + return state; + } + + @override + _HypnogramState combine(_HypnogramState left, _HypnogramState right) { + left + ..tst += right.tst + ..deep += right.deep + ..rem += right.rem + ..light += right.light + ..wakes.addAll(right.wakes); + if (!left.hasSleep && right.hasSleep) { + left + ..hasSleep = true + ..onset = right.onset; + } + if (right.hasSleep) left.sptEnd = right.sptEnd; + if (!left.hasRem && right.hasRem) { + left + ..hasRem = true + ..firstRemStart = right.firstRemStart; + } + return left; + } + + @override + bool get persistable => true; + + @override + Object? encodeState(_HypnogramState state) => { + 'tst': state.tst, + 'deep': state.deep, + 'rem': state.rem, + 'light': state.light, + 'hasSleep': state.hasSleep, + 'onset': state.onset, + 'sptEnd': state.sptEnd, + 'hasRem': state.hasRem, + 'firstRemStart': state.firstRemStart, + 'wakes': [ + for (final wake in state.wakes) [wake.$1, wake.$2] + ], + }; + + @override + _HypnogramState decodeState(Object? json) { + final map = decodeMap(json); + final state = _HypnogramState() + ..tst = _int(field(map, 'tst')) + ..deep = _int(field(map, 'deep')) + ..rem = _int(field(map, 'rem')) + ..light = _int(field(map, 'light')) + ..hasSleep = _bool(field(map, 'hasSleep')) + ..onset = _int(field(map, 'onset')) + ..sptEnd = _int(field(map, 'sptEnd')) + ..hasRem = _bool(field(map, 'hasRem')) + ..firstRemStart = _int(field(map, 'firstRemStart')); + for (final item in decodeList(field(map, 'wakes'))) { + final pair = decodeList(item); + if (pair.length != 2) throw const FormatException('Expected a wake pair'); + state.wakes.add((_int(pair[0]), _int(pair[1]))); + } + if (state.deep + state.rem + state.light != state.tst) { + throw const FormatException('Stage totals disagree with tst'); + } + if (state.hasRem && !state.hasSleep) { + throw const FormatException('REM without sleep'); + } + return state; + } + + @override + HypnogramMetrics finish(_HypnogramState state) { + final tib = math.max(0, sessionEnd - sessionStart); + final onset = state.hasSleep ? state.onset : sessionEnd; + final sptEnd = state.hasSleep ? state.sptEnd : sessionEnd; + var waso = 0; + var disturbances = 0; + for (final wake in state.wakes) { + final w0 = math.max(wake.$1, onset); + final w1 = math.min(wake.$2, sptEnd); + if (w1 > w0) { + waso += w1 - w0; + disturbances++; + } + } + final efficiency = tib > 0 ? state.tst / tib : 0.0; + double pct(int value) => state.tst > 0 ? value / state.tst * 100 : 0; + return HypnogramMetrics( + tibS: tib, + tstS: state.tst, + sptS: math.max(0, sptEnd - onset), + solS: state.hasSleep ? math.max(0, onset - sessionStart) : null, + remLatencyS: + state.hasRem ? (state.firstRemStart - onset).toDouble() : double.nan, + wasoS: waso, + efficiency: math.min(1.0, efficiency), + disturbances: disturbances, + deepMin: state.deep / 60, + remMin: state.rem / 60, + lightMin: state.light / 60, + deepPct: pct(state.deep), + remPct: pct(state.rem), + lightPct: pct(state.light), + ); + } +} + +class _HypnogramState { + var tst = 0; + var deep = 0; + var rem = 0; + var light = 0; + var hasSleep = false; + var onset = 0; + var sptEnd = 0; + var hasRem = false; + var firstRemStart = 0; + final wakes = <(int, int)>[]; +} + +int _int(Object? value) { + if (value is! int) throw const FormatException('Expected an int'); + return value; +} + +bool _bool(Object? value) { + if (value is! bool) throw const FormatException('Expected a bool'); + return value; +} diff --git a/lib/src/onehz/sleep/segment.dart b/lib/src/onehz/sleep/segment.dart index f72cf46..6d1fc13 100644 --- a/lib/src/onehz/sleep/segment.dart +++ b/lib/src/onehz/sleep/segment.dart @@ -45,6 +45,7 @@ import 'van_hees.dart'; import 'accounting.dart' show SleepStage; import 'advanced_stager.dart'; import 'band_offset.dart'; +import 'cardio_row_cache.dart'; /// Minimum in-bed duration to qualify as the main sleep (ARCHITECTURE_V2: ~3 h). const int _minQualifyingSleepSec = 3 * 3600; @@ -431,6 +432,7 @@ SleepSegmentation segmentSleep( /// creates, extends or stages a night. (AdvancedSleepStager.detectSleep takes /// the same signal as [ts,state] pairs; not forwarded here.) List? bandSleepState, + CardioRowCache? rowCache, }) { final n = math.min(accel.length, hr1hz.length); // A forced window (manual entry / user confirmation, Approach 1) is asserted @@ -498,7 +500,8 @@ SleepSegmentation segmentSleep( if (offsetSec <= onsetSec) return SleepSegmentation.absent; // Stage the user-asserted window directly — no detection, no gates. final session = - AdvancedSleepStager.stageWindow(onsetSec, offsetSec, grav, hr, rr: rr); + AdvancedSleepStager.stageWindow(onsetSec, offsetSec, grav, hr, + rr: rr, rowCache: rowCache); chosen = _SleepGroup( sessions: [session], start: onsetSec, @@ -511,6 +514,7 @@ SleepSegmentation segmentSleep( hr, rr: rr, tzOffsetResolver: tzAt, + rowCache: rowCache, ); if (sessions.isEmpty) return SleepSegmentation.absent; diff --git a/lib/src/onehz/sleep/sleep.dart b/lib/src/onehz/sleep/sleep.dart index 2e2bd2f..0724905 100644 --- a/lib/src/onehz/sleep/sleep.dart +++ b/lib/src/onehz/sleep/sleep.dart @@ -15,6 +15,9 @@ // honesty-bounded; stager.dart holds only the shared post-processing // (StagerResult, Webster rescore, consolidation) it reuses. The // autonomic-HR-only stager that used to live in stager.dart is DELETED. +// - CAUSAL (online) wake/NREM/REM stager (causal_stager.dart) — +// the same features/weights/cutoffs on past-only windows, abstains with a +// reason; for live decisions, NOT a replacement for cardioStager. // - Cardiopulmonary Coupling (CPC) (cpc.dart) // - Nonparametric circadian IS/IV/RA/L5/M10 (circadian_np.dart) // @@ -28,6 +31,7 @@ export 'segment.dart'; export 'hr_fallback.dart'; export 'advanced_stager.dart'; export 'band_offset.dart'; +export 'hypnogram.dart'; export 'sri.dart'; export 'accounting.dart'; // stager.dart provides StagerResult + the shared Webster/consolidation @@ -35,6 +39,10 @@ export 'accounting.dart'; // `autonomicStager` is DELETED — `cardioStager` is the stager. export 'stager.dart'; export 'cardio_stager.dart'; +export 'stage_ranges.dart'; +export 'cardio_row_cache.dart'; +export 'epoch_rows.dart'; +export 'causal_stager.dart'; export 'cpc.dart'; export 'circadian_np.dart'; export 'cycles.dart'; diff --git a/lib/src/onehz/sleep/sri.dart b/lib/src/onehz/sleep/sri.dart index 88c8083..f78fa3a 100644 --- a/lib/src/onehz/sleep/sri.dart +++ b/lib/src/onehz/sleep/sri.dart @@ -16,8 +16,12 @@ // fixed number of epochs per day (e.g. 1440 one-minute epochs, or 86400 one- // second epochs). The vector spans ≥2 days. +import 'dart:typed_data'; + +import '../../aggregate/aggregator.dart'; import '../types.dart'; import '../util.dart'; +import 'sri_aggregator.dart'; /// One adjacent-day comparison, broken out of the total. /// @@ -104,6 +108,57 @@ Metric phillipsSri( ); } + final completeDaySpan = days * epochsPerDay; + // The former direct loop reads a short mask only while its entries are true. + // Preserve that historical short-circuit behaviour, including its RangeError. + if (valid != null && valid.length < completeDaySpan) { + return _legacyPhillipsSri( + sleepWake, + epochsPerDay, + valid: valid, + minPairCases: minPairCases, + ); + } + + return sriAggregator(epochsPerDay, minPairCases: minPairCases).run(Epochs( + Uint8List.fromList([ + for (var i = 0; i < completeDaySpan; i++) + sleepWake[i] ? 1 : 0, + ]), + valid: valid == null + ? null + : Uint8List.fromList([ + for (var i = 0; i < completeDaySpan; i++) + valid[i] ? 1 : 0, + ]), + )); +} + +/// The direct-loop implementation retained for malformed, short validity masks. +Metric _legacyPhillipsSri( + List sleepWake, + int epochsPerDay, { + List? valid, + int? minPairCases, +}) { + const inputs = ['sleep_wake_epochs']; + final n = sleepWake.length; + if (epochsPerDay <= 0 || n < 2 * epochsPerDay) { + return const Metric.absent( + tier: Tier.high, + inputs_used: inputs, + note: 'need ≥2 full days of clock-aligned sleep/wake epochs for SRI', + ); + } + final days = n ~/ epochsPerDay; + if (days < 2) { + return const Metric.absent( + tier: Tier.high, + inputs_used: inputs, + note: 'need ≥2 full days for SRI', + ); + } + final pairFloor = minPairCases ?? (epochsPerDay ~/ 2); var agreement = 0; var cases = 0; @@ -121,8 +176,6 @@ Metric phillipsSri( } agreement += pairAgreement; cases += pairCases; - // `> 0` is not redundant with the floor: a caller may pass - // `minPairCases: 0`, and a zero-case pair would make SriPair.sri NaN. if (pairCases > 0 && pairCases >= pairFloor) { pairs.add(SriPair(d, pairAgreement, pairCases)); } @@ -136,8 +189,6 @@ Metric phillipsSri( } final sri = 200.0 * agreement / cases - 100.0; - // Confidence: scales with the number of day-comparisons (more days, more - // stable). Saturates around a typical 7-day record. final conf = ((days - 1) / 7.0).clamp(0.3, 0.95); return Metric( value: SriResult(sri, days, cases, pairs), diff --git a/lib/src/onehz/sleep/sri_aggregator.dart b/lib/src/onehz/sleep/sri_aggregator.dart new file mode 100644 index 0000000..a901656 --- /dev/null +++ b/lib/src/onehz/sleep/sri_aggregator.dart @@ -0,0 +1,314 @@ +import 'dart:typed_data'; + +import '../../aggregate/aggregator.dart'; +import '../../aggregate/json_support.dart'; +import '../../aggregate/primitives.dart'; +import '../types.dart'; +import 'sri.dart'; + +/// Clock-aligned sleep/wake epochs, optionally accompanied by a validity mask. +/// +/// [firstIndex] is the absolute index of row zero. It lets [Lookback] preserve +/// the day a comparison belongs to when it joins copied chunks at a seam. +class Epochs extends Batch { + Epochs(this.asleep, {this.valid, this.firstIndex = 0}); + + factory Epochs.fromJson(Object? json) { + final map = decodeMap(json); + final asleep = _readBytes(field(map, 'asleep')); + final rawValid = field(map, 'valid'); + final valid = rawValid == null ? null : _readBytes(rawValid); + if (valid != null && valid.length != asleep.length) { + throw const FormatException('Epochs columns differ in length'); + } + return Epochs(asleep, + valid: valid, firstIndex: decodeCount(field(map, 'firstIndex'))); + } + + final Uint8List asleep; + final Uint8List? valid; + final int firstIndex; + + @override + Object? toJson() => { + 'asleep': asleep.toList(), + 'valid': valid?.toList(), + 'firstIndex': firstIndex, + }; + + @override + int get length => asleep.length; + + @override + Epochs copyRange(int start, int end) { + final copiedLength = end - start; + final copiedAsleep = Uint8List(copiedLength) + ..setRange(0, copiedLength, asleep, start); + final sourceValid = valid; + final copiedValid = sourceValid == null + ? null + : (Uint8List(copiedLength) + ..setRange(0, copiedLength, sourceValid, start)); + return Epochs(copiedAsleep, + valid: copiedValid, firstIndex: firstIndex + start); + } + + @override + Epochs join(Epochs other) { + assert(other.firstIndex == firstIndex + length); + final joinedAsleep = Uint8List(length + other.length) + ..setRange(0, length, asleep) + ..setRange(length, length + other.length, other.asleep); + Uint8List? joinedValid; + if (valid != null || other.valid != null) { + joinedValid = Uint8List(length + other.length); + for (var i = 0; i < length; i++) { + joinedValid[i] = valid == null ? 1 : valid![i]; + } + for (var i = 0; i < other.length; i++) { + joinedValid[length + i] = other.valid == null ? 1 : other.valid![i]; + } + } + return Epochs(joinedAsleep, valid: joinedValid, firstIndex: firstIndex); + } +} + +Uint8List _readBytes(Object? json) { + final list = decodeList(json); + final bytes = Uint8List(list.length); + for (var i = 0; i < list.length; i++) { + final value = list[i]; + if (value is! int || value < 0 || value > 255) { + throw const FormatException('Expected a byte'); + } + bytes[i] = value; + } + return bytes; +} + +class _PairCounts { + var agreement = 0; + var cases = 0; +} + +class _PairCountsAggregator + extends Aggregator<_PairCounts, Epochs, _PairCounts> { + const _PairCountsAggregator(this.epochsPerDay); + + final int epochsPerDay; + + @override + _PairCounts empty() => _PairCounts(); + + @override + _PairCounts addSlice(_PairCounts state, Epochs batch, int start, int end) { + final valid = batch.valid; + for (var i = start; i < end; i++) { + final previous = i - epochsPerDay; + if (valid == null || (valid[previous] != 0 && valid[i] != 0)) { + state.cases++; + if (batch.asleep[previous] == batch.asleep[i]) state.agreement++; + } + } + return state; + } + + @override + _PairCounts combine(_PairCounts left, _PairCounts right) { + left + ..agreement += right.agreement + ..cases += right.cases; + return left; + } + + @override + bool get persistable => true; + + @override + Object? encodeState(_PairCounts state) => + {'agreement': state.agreement, 'cases': state.cases}; + + @override + _PairCounts decodeState(Object? json) { + final map = decodeMap(json); + final agreement = decodeCount(field(map, 'agreement')); + final cases = decodeCount(field(map, 'cases')); + if (agreement > cases) { + throw const FormatException('agreement exceeds cases'); + } + return _PairCounts() + ..agreement = agreement + ..cases = cases; + } + + @override + _PairCounts finish(_PairCounts state) => state; +} + +/// Applies [GroupBy] only to windows that [Lookback] says are complete. +/// +/// A Lookback inner sees a length-k window ending at `start + k - 1` through +/// `end - 1`. GroupBy would otherwise reset that origin for each day run, so +/// this small adapter advances to the first complete window before grouping. +class _GroupedPairWindows + extends Aggregator, Epochs, Map> { + _GroupedPairWindows(this.epochsPerDay) + : _groups = GroupBy( + (batch, row) => (batch.firstIndex + row) ~/ epochsPerDay, + _PairCountsAggregator(epochsPerDay), + ); + + final int epochsPerDay; + final GroupBy _groups; + + @override + Map empty() => _groups.empty(); + + @override + Map addSlice( + Map state, Epochs batch, int start, int end) { + final firstWindowEnd = start + epochsPerDay; + if (firstWindowEnd < end) { + return _groups.addSlice(state, batch, firstWindowEnd, end); + } + return state; + } + + @override + Map combine( + Map left, Map right) => + _groups.combine(left, right); + + @override + Map finish(Map state) => + _groups.finish(state); + + @override + bool get persistable => _groups.persistable; + + @override + Object? encodeState(Map state) => + _groups.encodeState(state); + + @override + Map decodeState(Object? json) => + _groups.decodeState(json); +} + +/// Mergeable Phillips SRI over [Epochs]. +/// +/// The Lookback window has one row from each of two adjacent clock days. Its +/// last row determines the later-day key used by GroupBy. +// Phillips et al. 2017 SRI, as in phillipsSri(). Lookback pairs epochs exactly +// one day apart; integer agreement/case counts preserve the formula, validity +// mask, pair eligibility and complete-day truncation. +Aggregator> sriAggregator( + int epochsPerDay, { + int? minPairCases, +}) { + if (epochsPerDay <= 0) { + return Count().map((_) => const Metric.absent( + tier: Tier.high, + inputs_used: ['sleep_wake_epochs'], + note: 'need ≥2 full days of clock-aligned sleep/wake epochs for SRI', + )); + } + final pairs = Lookback, Map>( + epochsPerDay + 1, + _GroupedPairWindows(epochsPerDay), + batchFromJson: Epochs.fromJson, + ); + final core = both(Count(), pairs); + return validateCheckpoint(core, (json) { + final state = decodeList(json); + if (state.length != 2) throw const FormatException('Expected two states'); + final count = decodeCount(state[0]); + final lookback = decodeMap(state[1]); + if (decodeCount(field(lookback, 'length')) != count) { + throw const FormatException('Lookback length disagrees with epoch count'); + } + final groups = decodeMap(field(lookback, 'inner')); + if (count <= epochsPerDay) { + if (groups.isNotEmpty) { + throw const FormatException('Pair days without complete comparisons'); + } + return; + } + final first = decodeMap(field(lookback, 'first')); + final firstIndex = decodeCount(field(first, 'firstIndex')); + final firstDay = (firstIndex + epochsPerDay) ~/ epochsPerDay; + final lastDay = (firstIndex + count - 1) ~/ epochsPerDay; + if (groups.length != lastDay - firstDay + 1) { + throw const FormatException('Unexpected number of pair days'); + } + var totalCases = 0; + final pairLimit = count - epochsPerDay; + for (final entry in groups.entries) { + final day = int.tryParse(entry.key); + if (day == null || day < firstDay || day > lastDay) { + throw const FormatException('Pair day is outside the record'); + } + final pair = decodeMap(entry.value); + final cases = decodeCount(field(pair, 'cases')); + final dayStart = day * epochsPerDay - firstIndex; + final dayEnd = dayStart + epochsPerDay; + final lower = dayStart > epochsPerDay ? dayStart : epochsPerDay; + final upper = dayEnd < count ? dayEnd : count; + final slots = upper > lower ? upper - lower : 0; + if (cases > slots || totalCases > pairLimit - cases) { + throw const FormatException('Pair cases exceed available epochs'); + } + totalCases += cases; + } + }).map((result) { + const inputs = ['sleep_wake_epochs']; + final n = result.$1; + if (n < 2 * epochsPerDay) { + return const Metric.absent( + tier: Tier.high, + inputs_used: inputs, + note: 'need ≥2 full days of clock-aligned sleep/wake epochs for SRI', + ); + } + final days = n ~/ epochsPerDay; + if (days < 2) { + return const Metric.absent( + tier: Tier.high, + inputs_used: inputs, + note: 'need ≥2 full days for SRI', + ); + } + + final pairFloor = minPairCases ?? (epochsPerDay ~/ 2); + var agreement = 0; + var cases = 0; + final sriPairs = []; + for (var day = 1; day < days; day++) { + final pair = result.$2[day]; + if (pair == null) continue; + agreement += pair.agreement; + cases += pair.cases; + if (pair.cases > 0 && pair.cases >= pairFloor) { + sriPairs.add(SriPair(day, pair.agreement, pair.cases)); + } + } + if (cases == 0) { + return const Metric.absent( + tier: Tier.high, + inputs_used: inputs, + note: 'no valid epoch pairs for SRI', + ); + } + + final sri = 200.0 * agreement / cases - 100.0; + final confidence = ((days - 1) / 7.0).clamp(0.3, 0.95); + return Metric( + value: SriResult(sri, days, cases, sriPairs), + confidence: confidence, + tier: Tier.high, + inputs_used: inputs, + note: 'epoch-by-epoch 24-h concordance (Phillips 2017), ' + 'SRI=200·agreement/cases−100; NOT SD-of-midsleep', + ); + }); +} diff --git a/lib/src/onehz/sleep/stage_ranges.dart b/lib/src/onehz/sleep/stage_ranges.dart new file mode 100644 index 0000000..4e073d5 --- /dev/null +++ b/lib/src/onehz/sleep/stage_ranges.dart @@ -0,0 +1,177 @@ +// SLEEP — per-stage minute RANGES. +// +// The cardio stager gives each epoch one label. This file turns those labels +// into a [lo, hi] range of minutes per stage, so a caller can show how much of +// each stage figure the evidence actually pins down. +// +// WHAT THE RANGE IS. An honest BOUND, not a statistical confidence interval. +// The stager has no per-epoch posteriors: it compares two weighted robust-z +// scores against fixed cutoffs ([kDefaultRemScoreCut], [kDefaultDeepScoreCut]) +// and reports a night confidence from coverage. So the only per-epoch evidence +// is "which side of the cutoff, and how far". The range counts: +// lo[s] = epochs that can only be s: HR-covered, labelled s, and every score +// that decided the label at least `margin` from its cutoff; +// hi[s] = lo[s] + epochs that could still be s: uncovered or unlabelled +// epochs (any stage), and asleep epochs with a score inside the +// margin (REM score: REM and its NREM substage; deep score: light +// and deep). +// Owner decision (2026-10-19): a fully covered night has lo == hi only where +// every score is clear of its cutoff. Nothing here moves the stager's label. + +import 'accounting.dart' show SleepStage; +import 'cardio_stager.dart' + show CardioStagerResult, kDefaultDeepScoreCut, kDefaultRemScoreCut; + +// Internal rule, ESTIMATE with low confidence, derived from cardioStager() score +// cutoffs. Long 2016 / Fonseca 2015/2018 supports the underlying staging family, +// not the unpublished margin or bounds. This is not a confidence interval. +/// How far (in stage-score units) a score must sit from its cutoff before the +/// label it decided counts as clear. +/// +/// The scores are weighted means of robust z, and the shipped cutoffs were +/// chosen on an 11-night sweep with a 0.1 grid (see the table above +/// [kDefaultRemScoreCut]). One grid step moves median %REM by about 5 points +/// and %Deep by about 3, and the sweep cannot rank cutoffs finer than that. +/// So a score within 0.1 of its cutoff would flip under a neighbouring cutoff +/// the calibration could not tell apart from the shipped one. That is the only +/// precision we can justify; it is NOT a measured per-epoch error rate. +const double kStageScoreMargin = 0.1; + +/// One epoch's staging evidence, as [stageMinuteRanges] reads it. +class StageEpochEvidence { + /// The final label; null when the epoch has no label at all. + final SleepStage? stage; + + /// Deep flag (meaningful only when [stage] is NREM). + final bool deep; + + /// The epoch carried at least one valid HR sample. + final bool covered; + + /// The REM and deep scores that decided the label; null when not scored. + final double? remScore; + final double? deepScore; + + const StageEpochEvidence({ + this.stage, + this.deep = false, + this.covered = true, + this.remScore, + this.deepScore, + }); +} + +/// A [lo, hi] range of minutes. +class MinuteRange { + final double lo, hi; + const MinuteRange(this.lo, this.hi); + + Map toJson() => {'lo': lo, 'hi': hi}; +} + +/// Per-stage minute ranges over a staged window of [windowMin] minutes. +/// [nrem] is light ∪ deep. +class StageMinuteRanges { + final MinuteRange wake, light, deep, rem, nrem; + final double windowMin; + + const StageMinuteRanges({ + required this.wake, + required this.light, + required this.deep, + required this.rem, + required this.nrem, + required this.windowMin, + }); + + /// The four display stages as `{lo, hi}`. + Map> toJson() => { + 'wake': wake.toJson(), + 'light': light.toJson(), + 'deep': deep.toJson(), + 'rem': rem.toJson(), + }; +} + +extension CardioStagerEvidence on CardioStagerResult { + /// One entry per final epoch: the final label and deep flag, whether the + /// epoch had valid HR, and the scores the classify loop computed for it. + /// Empty when the stager abstained. + List get epochEvidence { + final stages = base.stages; + return [ + for (var e = 0; e < stages.length; e++) + StageEpochEvidence( + stage: stages[e], + deep: e < deepFlag.length && deepFlag[e], + // A result built without coverage cannot vouch for any epoch. + covered: e < hrCovered.length && hrCovered[e], + remScore: e < remScores.length ? remScores[e] : null, + deepScore: e < deepScores.length ? deepScores[e] : null, + ), + ]; + } +} + +const int _wake = 1, _light = 2, _deep = 4, _rem = 8; +const int _any = _wake | _light | _deep | _rem; +const int _nremSet = _light | _deep; + +/// Per-stage minute ranges for [epochs] ([epochSec] seconds each). Null when +/// there are no epochs. See the file header for the rule. +StageMinuteRanges? stageMinuteRanges( + List? epochs, { + int epochSec = 30, + double remScoreCut = kDefaultRemScoreCut, + double deepScoreCut = kDefaultDeepScoreCut, + double margin = kStageScoreMargin, +}) { + if (epochs == null || epochs.isEmpty) return null; + // Written as an open interval on `cut ± margin` so a score of exactly + // `cut + margin` compares equal to the bound and counts as clear. + bool near(double? s, double cut) => + s != null && s > cut - margin && s < cut + margin; + + final lo = {_wake: 0, _light: 0, _deep: 0, _rem: 0, _nremSet: 0}; + final hi = {_wake: 0, _light: 0, _deep: 0, _rem: 0, _nremSet: 0}; + for (final ep in epochs) { + final int can; // bitset of stages this epoch could be + if (!ep.covered || ep.stage == null) { + can = _any; + } else if (ep.stage == SleepStage.wake) { + can = _wake; + } else { + final deepNear = near(ep.deepScore, deepScoreCut); + final int sub; + if (deepNear) { + sub = _nremSet; + } else if (ep.stage == SleepStage.nrem) { + sub = ep.deep ? _deep : _light; + } else { + // A REM epoch's NREM neighbour is the substage its deep score points to. + final ds = ep.deepScore; + sub = (ds != null && ds > deepScoreCut) ? _deep : _light; + } + final remNear = near(ep.remScore, remScoreCut); + can = remNear + ? (_rem | sub) + : (ep.stage == SleepStage.rem ? _rem : sub); + } + for (final s in const [_wake, _light, _deep, _rem]) { + if (can & s != 0) hi[s] = hi[s]! + 1; + if (can == s) lo[s] = lo[s]! + 1; + } + if (can & _nremSet != 0) hi[_nremSet] = hi[_nremSet]! + 1; + if (can & ~_nremSet == 0) lo[_nremSet] = lo[_nremSet]! + 1; + } + final m = epochSec / 60.0; + MinuteRange r(int s) => MinuteRange(lo[s]! * m, hi[s]! * m); + return StageMinuteRanges( + wake: r(_wake), + light: r(_light), + deep: r(_deep), + rem: r(_rem), + nrem: r(_nremSet), + windowMin: epochs.length * m, + ); +} diff --git a/lib/src/onehz/util.dart b/lib/src/onehz/util.dart index ac69c03..f52e213 100644 --- a/lib/src/onehz/util.dart +++ b/lib/src/onehz/util.dart @@ -64,6 +64,85 @@ double? percentile(List values, double p) => /// Median (linear-interpolated), or null if empty. double? median(List xs) => percentile(xs, 50); +/// Linear-interpolated median of every clipped, centred [halfWin] window. +/// +/// `out[j]` is exactly +/// `median(xs.sublist(max(0, j - halfWin), min(xs.length - 1, j + halfWin) + 1))`. +/// In particular, ordering is [double.compareTo] ordering (including NaN last), +/// and an even-sized window uses [percentileSorted]'s interpolation. The input +/// is not changed. +/// +/// One sorted window is grown, then updated by binary insertion/removal; it +/// therefore avoids a fresh window allocation and sort at every position. +// Hyndman, R. J., Fan, Y. (1996). Sample quantiles in statistical packages. +// The American Statistician 50(4):361-365. +// percentileSorted implements type 7: zero-based rank = (n - 1) * p / 100. +// These clipped-window medians use that same adjacent-rank interpolation. +// Exact reformulation of clipped-window medians: insertion/removal preserves +// List.sort total order; percentileSorted retains the same ranks and +// interpolation. +List slidingMedians(List xs, int halfWin) { + if (xs.isEmpty) return []; + if (halfWin < 0) { + throw ArgumentError.value(halfWin, 'halfWin', 'must be non-negative'); + } + + final n = xs.length; + var lo = 0; + var hi = math.min(n, halfWin + 1); // exclusive + final sorted = []; + for (var i = lo; i < hi; i++) { + _insertSorted(sorted, xs[i]); + } + final out = List.filled(n, 0); + for (var j = 0; j < n; j++) { + out[j] = percentileSorted(sorted, 50)!; + if (j + 1 == n) break; + + final nextLo = math.max(0, j + 1 - halfWin); + final nextHi = math.min(n, j + 1 + halfWin + 1); // exclusive + while (lo < nextLo) { + _removeSorted(sorted, xs[lo]); + lo++; + } + while (hi < nextHi) { + _insertSorted(sorted, xs[hi]); + hi++; + } + } + return out; +} + +void _insertSorted(List sorted, double value) { + var lo = 0; + var hi = sorted.length; + while (lo < hi) { + final mid = lo + (hi - lo) ~/ 2; + if (sorted[mid].compareTo(value) < 0) { + lo = mid + 1; + } else { + hi = mid; + } + } + sorted.insert(lo, value); +} + +void _removeSorted(List sorted, double value) { + var lo = 0; + var hi = sorted.length; + while (lo < hi) { + final mid = lo + (hi - lo) ~/ 2; + if (sorted[mid].compareTo(value) < 0) { + lo = mid + 1; + } else { + hi = mid; + } + } + // [value] is known to be in the active window. A compare-equal duplicate is + // interchangeable for percentile arithmetic, including NaN and signed zero. + sorted.removeAt(lo); +} + /// Median absolute deviation, scaled to be a consistent estimator of σ for /// normal data (× 1.4826). Returns null if empty. NOTE: on quantized data the /// raw MAD can be 0 — callers must guard division (see [robustZ]). @@ -326,6 +405,12 @@ class LombScargle { /// /// This operates on NATIVE sample times — no resampling — which is exactly why /// it's the correct PSD for unevenly-sampled beat-time RR. +/// +/// PRECISION: every trig argument is built from `t - t.first`, never the raw +/// [t]. On epoch seconds (~1.8e9) the argument `2π·f·t` reaches ~1e10 rad, where +/// a double is only good to ~1e-6 rad, which cost up to ~5e-6 relative error in +/// the PSD. The spectrum is time-shift invariant, so the shift changes nothing +/// but that error (≈1e-13 after it). LombScargle? lombScargle(List t, List y, List freqsHz) { final n = t.length; if (n < 4 || y.length != n || freqsHz.isEmpty) return null; @@ -345,6 +430,7 @@ LombScargle? lombScargle(List t, List y, List freqsHz) { } final span = tMax - tMin; if (span <= 0) return null; + final t0 = t[0]; // One-sided periodogram → density: 2·Δt, with Δt the MEAN sample interval. final psdScale = 2.0 * span / (n - 1); @@ -358,14 +444,14 @@ LombScargle? lombScargle(List t, List y, List freqsHz) { // τ: phase reference for time-shift invariance. var sin2 = 0.0, cos2 = 0.0; for (final ti in t) { - sin2 += math.sin(2 * w * ti); - cos2 += math.cos(2 * w * ti); + sin2 += math.sin(2 * w * (ti - t0)); + cos2 += math.cos(2 * w * (ti - t0)); } final tau = math.atan2(sin2, cos2) / (2 * w); var cNum = 0.0, cDen = 0.0, sNum = 0.0, sDen = 0.0; for (var i = 0; i < n; i++) { - final arg = w * (t[i] - tau); + final arg = w * ((t[i] - t0) - tau); final c = math.cos(arg); final s = math.sin(arg); cNum += yc[i] * c; diff --git a/lib/src/onehz/workout/calories.dart b/lib/src/onehz/workout/calories.dart index 2edf051..a4926d0 100644 --- a/lib/src/onehz/workout/calories.dart +++ b/lib/src/onehz/workout/calories.dart @@ -326,41 +326,26 @@ class Calories { 'cadenceSpmPerMin (${cadenceSpmPerMin.length}) must align with ' 'hrPerMin (${hrPerMin.length}): one entry per wake minute'); } - // (weight/height/age still can't be told apart from their defaults here: - // WorkoutUserProfile's own constructor bakes 70/170/30 in at construction, - // so by the time a profile object arrives there is no way left to tell "the - // caller passed 70kg" from "nobody set a weight". Making those three - // nullable at the source is the real fix and a bigger change.) - final weightKg = profile.weightKg > 0 ? profile.weightKg : 70.0; - final heightCm = profile.heightCm > 0 ? profile.heightCm : 170.0; - final age = profile.age > 0 ? profile.age : 30.0; - final coeffs = resolveCoeffs(profile.sex); - final flexHr = activeGateHr(hrmax, restingHr); - if (flexHr == null) return null; - - final bmrDay = mifflinBmrKcalDay(weightKg, heightCm, age, profile.sex); - final basalPerMin = bmrDay / 1440.0; + // The prep and the per-minute billing are SHARED with [minuteEnergy] + // ([_energyCtx], [_bill]) — one definition of the computation, so the + // per-minute series and this total cannot drift. (weight/height/age still + // can't be told apart from their defaults there: WorkoutUserProfile's own + // constructor bakes 70/170/30 in at construction, so by the time a profile + // object arrives there is no way left to tell "the caller passed 70kg" from + // "nobody set a weight". Making those three nullable at the source is the + // real fix and a bigger change.) + final ctx = _energyCtx(profile, hrmax, restingHr); + if (ctx == null) return null; + final basalPerMin = ctx.basalPerMin; var active = 0.0; var walking = 0.0; for (var i = 0; i < hrPerMin.length; i++) { - final hr = hrPerMin[i]; - // `hr < flexHr` is false for NaN, so an unfiltered non-finite minute - // would bill active and carry its NaN into the day total. - if (hr.isFinite && hr >= flexHr) { - final activePerMin = - activeKcalPerS(coeffs, hr, hrmax, weightKg, age) * 60.0; - final surplus = activePerMin - basalPerMin; - if (surplus > 0) active += surplus; - continue; // HR billed the minute — cadence never doubles it. - } - // Below flex (or no HR at all — measured gait stands on its own): - // a measured walking cadence prices the minute the HR gate refused. - final cad = cadenceSpmPerMin?[i]; - if (cad == null) continue; - final met = metFromCadenceSpm(cad); - if (met == null) continue; - walking += (met - 1.0) * basalPerMin; + final b = _bill(ctx, hrPerMin[i], cadenceSpmPerMin?[i]); + // Adding the +0.0 of a minute that billed nothing is exact, so this is + // the same sum, in the same order, as skipping those minutes. + active += b.active; + walking += b.walking; } active += walking; final basal = basalPerMin * dayMinutes; @@ -372,6 +357,271 @@ class Calories { ); } + // --------------------------------------------------------------------------- + // Shared by [dailyEnergy] and [minuteEnergy]: the ONE definition of how a + // minute is billed. Nothing here is new arithmetic — it is the body that used + // to sit inside dailyEnergy's loop, moved so the two entry points cannot + // diverge. dailyEnergy's outputs are bit-for-bit what they were (golden + // tests in test/onehz/minute_energy_test.dart). + // --------------------------------------------------------------------------- + + static ({ + CalorieCoeffs coeffs, + double weightKg, + double age, + double hrmax, + double flexHr, + double basalPerMin, + })? _energyCtx(WorkoutUserProfile profile, double hrmax, double restingHr) { + final weightKg = profile.weightKg > 0 ? profile.weightKg : 70.0; + final heightCm = profile.heightCm > 0 ? profile.heightCm : 170.0; + final age = profile.age > 0 ? profile.age : 30.0; + final flexHr = activeGateHr(hrmax, restingHr); + if (flexHr == null) return null; + return ( + coeffs: resolveCoeffs(profile.sex), + weightKg: weightKg, + age: age, + hrmax: hrmax, + flexHr: flexHr, + basalPerMin: mifflinBmrKcalDay(weightKg, heightCm, age, profile.sex) / + 1440.0, + ); + } + + /// One minute's bill. `source` is null when the minute has nothing measured + /// (no usable HR and no billable cadence). `active` is the HR surplus over + /// basal (≥ 0) and `walking` the cadence surplus; a minute is only ever one + /// of the two. + static ({MinuteEnergySource? source, double active, double walking}) _bill( + ({ + CalorieCoeffs coeffs, + double weightKg, + double age, + double hrmax, + double flexHr, + double basalPerMin, + }) ctx, + double hr, + double? cadenceSpm, + ) { + // `hr < flexHr` is false for NaN, so an unfiltered non-finite minute + // would bill active and carry its NaN into the day total. + if (hr.isFinite && hr >= ctx.flexHr) { + final activePerMin = activeKcalPerS( + ctx.coeffs, hr, ctx.hrmax, ctx.weightKg, ctx.age) * + 60.0; + final surplus = activePerMin - ctx.basalPerMin; + // HR billed the minute — cadence never doubles it. + return ( + source: MinuteEnergySource.hr, + active: surplus > 0 ? surplus : 0.0, + walking: 0.0, + ); + } + // Below flex (or no HR at all — measured gait stands on its own): + // a measured walking cadence prices the minute the HR gate refused. + final met = cadenceSpm == null ? null : metFromCadenceSpm(cadenceSpm); + if (met != null) { + return ( + source: MinuteEnergySource.cadence, + active: 0.0, + walking: (met - 1.0) * ctx.basalPerMin, + ); + } + return ( + source: (hr.isFinite && hr > 0) ? MinuteEnergySource.rest : null, + active: 0.0, + walking: 0.0, + ); + } + + /// INTRADAY ENERGY: the computation [dailyEnergy] does, one record per + /// minute. Same inputs, same gate ([activeGateHr]), same billing — so the + /// series folds back to the daily figures EXACTLY: + /// + /// `series.active` == `dailyEnergy(...).active` (`==`, not "close") + /// `series.walking` == `dailyEnergy(...).walking` + /// `series.basalKcalPerMin * dayMinutes` == `dailyEnergy(...).basal` + /// + /// [hrPerMin] is index-aligned per-minute mean HR (bpm). Unlike [dailyEnergy] + /// the caller should pass the series WITH its gaps in place (0 / NaN for a + /// minute with no HR) rather than compacted, so each record keeps its place + /// on the clock. [cadenceSpmPerMin] is aligned the same way (see + /// [dailyEnergy]). [epochMinutes], when given, is each entry's epoch minute + /// (`tsSec ~/ 60`) and becomes [MinuteEnergy.minute]; otherwise `minute` is + /// the index. Both must match [hrPerMin]'s length or this throws. + /// + /// ABSTENTION, per minute, never filled or interpolated: + /// * HR usable (finite, > 0) → the minute is measured. At/above the gate + /// it bills the Keytel surplus ([MinuteEnergySource.hr]); below it, it + /// is measured rest — basal only, `active` 0 ([MinuteEnergySource.rest]). + /// * HR unusable but a billable cadence ([metFromCadenceSpm]) → walking + /// surplus ([MinuteEnergySource.cadence]), exactly as [dailyEnergy] does. + /// * Otherwise the record abstains: `abstained` is + /// [MinuteEnergyAbstain.noHr] and basal/active/walking/total are all null. + /// A minute's record depends on that minute's inputs alone. + /// + /// Returns null when the anchors cannot define a gate, like [dailyEnergy]. + /// + /// The series covers exactly the minutes it is given. [dailyEnergy]'s basal + /// floor is pro-rated over `dayMinutes` (the whole covered day, sleep + /// included) and is not a per-minute sum; use + /// [MinuteEnergySeries.basalKcalPerMin] for that rate. + static MinuteEnergySeries? minuteEnergy( + List hrPerMin, { + required WorkoutUserProfile profile, + required double hrmax, + required double restingHr, + List? cadenceSpmPerMin, + List? epochMinutes, + }) { + if (cadenceSpmPerMin != null && + cadenceSpmPerMin.length != hrPerMin.length) { + throw ArgumentError( + 'cadenceSpmPerMin (${cadenceSpmPerMin.length}) must align with ' + 'hrPerMin (${hrPerMin.length}): one entry per minute'); + } + if (epochMinutes != null && epochMinutes.length != hrPerMin.length) { + throw ArgumentError( + 'epochMinutes (${epochMinutes.length}) must align with ' + 'hrPerMin (${hrPerMin.length}): one entry per minute'); + } + final ctx = _energyCtx(profile, hrmax, restingHr); + if (ctx == null) return null; + final basal = ctx.basalPerMin; + + final minutes = []; + // The same two accumulators, in the same order, as dailyEnergy. + var hrActive = 0.0; + var walking = 0.0; + var covered = 0; + for (var i = 0; i < hrPerMin.length; i++) { + final b = _bill(ctx, hrPerMin[i], cadenceSpmPerMin?[i]); + final minute = epochMinutes?[i] ?? i; + final src = b.source; + if (src == null) { + minutes.add(MinuteEnergy._abstained(minute)); + continue; + } + covered++; + hrActive += b.active; + walking += b.walking; + final cad = src == MinuteEnergySource.cadence; + final active = cad ? b.walking : b.active; + minutes.add(MinuteEnergy._( + minute: minute, + source: src, + basal: basal, + active: active, + walking: cad ? b.walking : null, + total: basal + active, + )); + } + return MinuteEnergySeries._( + minutes: minutes, + basalKcalPerMin: basal, + active: hrActive + walking, + walking: walking, + coveredMinutes: covered, + ); + } + + /// Prepares fixed profile and anchor math once for independent minute bills. + /// Returns null under the same anchor gate as [dailyEnergy]. + // Keytel et al. 2005 active energy, Mifflin-St Jeor 1990 basal energy and Tudor- + // Locke et al. 2019 cadence METs, as in dailyEnergy(). Shared minute bills + // preserve bill order and separate HR/walking accumulation; hourly output sums + // covered records without extrapolation. + static MinuteEnergyPricer? minuteEnergyPricer({ + required WorkoutUserProfile profile, + required double hrmax, + required double restingHr, + }) { + final ctx = _energyCtx(profile, hrmax, restingHr); + if (ctx == null) return null; + return MinuteEnergyPricer._( + ctx.basalPerMin, (hr, cadence) => _bill(ctx, hr, cadence)); + } + + /// Assembles previously priced minute bills without evaluating HR or cadence. + /// Intended for keyed incremental callers that retain the exact bills and + /// replace only changed minutes. Walking is already included in active. + // Keytel et al. 2005 active energy, Mifflin-St Jeor 1990 basal energy and Tudor- + // Locke et al. 2019 cadence METs, as in dailyEnergy(). Shared minute bills + // preserve bill order and separate HR/walking accumulation; hourly output sums + // covered records without extrapolation. + static MinuteEnergySeries assembleMinuteEnergySeries( + List< + ({ + int minute, + MinuteEnergySource? source, + double active, + double walking + })> + bills, { + required double basalKcalPerMin, + }) { + final minutes = []; + var hrActive = 0.0, walking = 0.0; + var covered = 0; + for (final bill in bills) { + final source = bill.source; + if (source == null) { + minutes.add(MinuteEnergy._abstained(bill.minute)); + continue; + } + covered++; + final cadence = source == MinuteEnergySource.cadence; + if (cadence) { + walking += bill.walking; + } else { + hrActive += bill.active; + } + final active = cadence ? bill.walking : bill.active; + minutes.add(MinuteEnergy._( + minute: bill.minute, + source: source, + basal: basalKcalPerMin, + active: active, + walking: cadence ? bill.walking : null, + total: basalKcalPerMin + active, + )); + } + return MinuteEnergySeries._( + minutes: minutes, + basalKcalPerMin: basalKcalPerMin, + active: hrActive + walking, + walking: walking, + coveredMinutes: covered, + ); + } + + /// Sums [minutes] into hourly buckets, ascending, for hours that have at + /// least one record. Only COVERED minutes are summed and nothing is scaled up + /// for the rest: an hour with 30 covered minutes reports 30 minutes of + /// energy and `coverage` 0.5, and an hour with none reports null sums, not + /// zero. The bucket is `floor((minute + minuteOffset) / 60)`: with index + /// minutes that is "hours since the series start"; with epoch minutes pass + /// the local UTC offset in minutes (e.g. 330) to bucket by local hour. + // Keytel et al. 2005 active energy, Mifflin-St Jeor 1990 basal energy and Tudor- + // Locke et al. 2019 cadence METs, as in dailyEnergy(). Shared minute bills + // preserve bill order and separate HR/walking accumulation; hourly output sums + // covered records without extrapolation. + static List hourlyRollup( + List minutes, { + int minuteOffset = 0, + }) { + final byHour = >{}; + for (final m in minutes) { + (byHour[(m.minute + minuteOffset) ~/ 60] ??= []).add(m); + } + final hours = byHour.keys.toList()..sort(); + return [ + for (final h in hours) HourlyEnergy._from(h, byHour[h]!), + ]; + } + /// Estimate (kcal, kJ) for a workout bout. Each sample is weighted by the /// ELAPSED time to the next sample (capped at [mergeGapCapS], which defaults /// to [defaultMergeGapCapS] = 150 s), so a sparse stream is counted over real @@ -445,3 +695,159 @@ class Calories { ); } } + +/// Reusable fixed-profile calorie pricing, with no minute-record allocation. +class MinuteEnergyPricer { + final double basalKcalPerMin; + final ({MinuteEnergySource? source, double active, double walking}) Function( + double, double?) _price; + MinuteEnergyPricer._(this.basalKcalPerMin, this._price); + + ({MinuteEnergySource? source, double active, double walking}) price( + double hr, double? cadenceSpm) => + _price(hr, cadenceSpm); +} + +/// Why a minute carries no energy. +enum MinuteEnergyAbstain { + /// No usable HR (0, negative, NaN, infinite) and no billable cadence. + noHr, +} + +/// What a non-abstained minute was billed from. +enum MinuteEnergySource { + /// HR at/above the gate: Keytel surplus over basal. + hr, + + /// HR unusable or below the gate, but a measured walking cadence at/above + /// the CADENCE-Adults floor: walking surplus. + cadence, + + /// Measured HR below the gate: resting, basal only. + rest, +} + +/// One minute of [Calories.minuteEnergy]. kcal for THAT minute. Either +/// [abstained] is set and every number is null, or [source] is set and +/// basal/active/total are non-null. +class MinuteEnergy { + /// Index into the input, or the epoch minute when the caller supplied them. + final int minute; + final MinuteEnergySource? source; + final MinuteEnergyAbstain? abstained; + + /// Mifflin–St Jeor BMR for one minute. + final double? basal; + + /// Surplus over basal: the Keytel HR surplus, or the walking surplus for a + /// [MinuteEnergySource.cadence] minute. 0 for [MinuteEnergySource.rest]. + final double? active; + + /// The cadence surplus, already inside [active]. Null unless + /// [source] is [MinuteEnergySource.cadence]. + final double? walking; + + /// `basal + active`. + final double? total; + + const MinuteEnergy._({ + required this.minute, + required MinuteEnergySource this.source, + required double this.basal, + required double this.active, + required this.walking, + required double this.total, + }) : abstained = null; + + const MinuteEnergy._abstained(this.minute) + : source = null, + abstained = MinuteEnergyAbstain.noHr, + basal = null, + active = null, + walking = null, + total = null; +} + +/// Result of [Calories.minuteEnergy]. +class MinuteEnergySeries { + /// One record per input minute, in input order. + final List minutes; + + /// Basal kcal per minute (BMR ÷ 1440). `dailyEnergy(...).basal` is this × + /// `dayMinutes`. + final double basalKcalPerMin; + + /// Σ active over the series, folded exactly as [Calories.dailyEnergy] folds + /// it: `==` its `active`, walking surplus included. + final double active; + + /// Σ walking surplus; `==` `dailyEnergy(...).walking`. + final double walking; + + /// Minutes that did not abstain. + final int coveredMinutes; + + const MinuteEnergySeries._({ + required this.minutes, + required this.basalKcalPerMin, + required this.active, + required this.walking, + required this.coveredMinutes, + }); + + int get abstainedMinutes => minutes.length - coveredMinutes; +} + +/// One hour of [Calories.hourlyRollup]: sums over the hour's covered minutes. +class HourlyEnergy { + /// `floor((minute + minuteOffset) / 60)`. + final int hour; + + /// Covered minutes in this hour (0–60). + final int coveredMinutes; + + /// `coveredMinutes / 60`. + double get coverage => coveredMinutes / 60.0; + + /// Null when the hour has no covered minute. + final double? basal; + final double? active; + final double? total; + + /// Null when no minute of the hour was billed from cadence. + final double? walking; + + const HourlyEnergy._({ + required this.hour, + required this.coveredMinutes, + required this.basal, + required this.active, + required this.walking, + required this.total, + }); + + factory HourlyEnergy._from(int hour, List ms) { + var covered = 0; + var basal = 0.0, active = 0.0, total = 0.0, walking = 0.0; + var anyWalking = false; + for (final m in ms) { + if (m.abstained != null) continue; + covered++; + basal += m.basal!; + active += m.active!; + total += m.total!; + if (m.walking != null) { + walking += m.walking!; + anyWalking = true; + } + } + return HourlyEnergy._( + hour: hour, + coveredMinutes: covered, + basal: covered == 0 ? null : basal, + active: covered == 0 ? null : active, + total: covered == 0 ? null : total, + walking: anyWalking ? walking : null, + ); + } +} diff --git a/lib/src/onehz/workout/hr_zones.dart b/lib/src/onehz/workout/hr_zones.dart index 16c66e3..f1f2e5d 100644 --- a/lib/src/onehz/workout/hr_zones.dart +++ b/lib/src/onehz/workout/hr_zones.dart @@ -1,7 +1,9 @@ -import 'dart:math' as math; +import 'dart:typed_data'; +import '../../aggregate/aggregator.dart'; import '../types.dart'; import '../util.dart'; +import 'time_in_zone_aggregator.dart'; /// One display heart-rate zone defined by a bpm interval. class HeartRateZone { @@ -166,6 +168,9 @@ class HeartRateZones { return HeartRateZoneSet(zones: built, maxHr: maxHr, source: source); } + // Cadence validation, median-capped interval credit and tail credit are an + // internal rule, ESTIMATE with low confidence. Karvonen 1957 supports the + // HR-reserve zone option, not this duration-credit policy. /// Time-in-zone from a time-ordered HR stream. /// /// Each sample is credited with the duration until the next sample. The tail @@ -181,34 +186,12 @@ class HeartRateZones { List hr, HeartRateZoneSet zoneSet, ) { + // Runs [timeInZoneAggregator] over the sorted stream in one slice: the + // aggregator holds the cadence rule, the credit bounding and the tail. final sorted = [...hr]..sort((a, b) => a.tsMs.compareTo(b.tsMs)); - final zoneSeconds = List.filled(5, 0); - var below = 0.0; - - final tailSeconds = - sampleCadenceSeconds([for (final s in sorted) s.tsMs / 1000.0]); - if (tailSeconds == null) return null; - for (var i = 0; i < sorted.length; i++) { - final sample = sorted[i]; - if (!sample.valid) continue; - final durSeconds = i < sorted.length - 1 - ? _boundedGapSeconds(sorted[i + 1].tsMs - sample.tsMs, tailSeconds) - : tailSeconds; - final zone = zoneSet.zoneNumber(sample.hr); - if (zone >= 1) { - zoneSeconds[zone - 1] += durSeconds; - } else { - below += durSeconds; - } - } - return TimeInHeartRateZone(seconds: zoneSeconds, belowZone1: below); + return timeInZoneAggregator(zoneSet).run(HrStream( + Float64List.fromList([for (final s in sorted) s.tsMs]), + Float64List.fromList([for (final s in sorted) s.hr]), + )); } - - static double _boundedGapSeconds(double gapMs, double fallbackSeconds) { - final gapSeconds = gapMs / 1000.0; - return gapSeconds > 0 - ? math.min(gapSeconds, fallbackSeconds) - : fallbackSeconds; - } - } diff --git a/lib/src/onehz/workout/time_in_zone_aggregator.dart b/lib/src/onehz/workout/time_in_zone_aggregator.dart new file mode 100644 index 0000000..6cbfc04 --- /dev/null +++ b/lib/src/onehz/workout/time_in_zone_aggregator.dart @@ -0,0 +1,377 @@ +import 'dart:math' as math; +import 'dart:typed_data'; + +import '../../aggregate/aggregator.dart'; +import '../../aggregate/json_support.dart'; +import '../../aggregate/primitives.dart'; +import '../util.dart'; +import 'hr_zones.dart'; + +/// Time-ordered heart-rate samples held in timestamp and heart-rate columns. +class HrStream extends Batch { + HrStream(this.tsMs, this.hr) : assert(tsMs.length == hr.length); + + factory HrStream.fromJson(Object? json) { + final map = decodeMap(json); + final timestamps = decodeDoubles(field(map, 'tsMs')); + final heartRates = decodeDoubles(field(map, 'hr')); + if (timestamps.length != heartRates.length) { + throw const FormatException('HrStream columns differ in length'); + } + return HrStream(timestamps, heartRates); + } + + final Float64List tsMs; + final Float64List hr; + + @override + Object? toJson() => + {'tsMs': encodeDoubles(tsMs), 'hr': encodeDoubles(hr)}; + + @override + int get length => tsMs.length; + + @override + HrStream copyRange(int start, int end) => HrStream( + Float64List(end - start)..setRange(0, end - start, tsMs, start), + Float64List(end - start)..setRange(0, end - start, hr, start), + ); + + @override + HrStream join(HrStream other) { + final timestamps = Float64List(length + other.length); + final heartRates = Float64List(length + other.length); + timestamps + ..setRange(0, length, tsMs) + ..setRange(length, timestamps.length, other.tsMs); + heartRates + ..setRange(0, length, hr) + ..setRange(length, heartRates.length, other.hr); + return HrStream(timestamps, heartRates); + } +} + +/// Builds a mergeable time-in-heart-rate-zone summary over [HrStream]. +/// +/// It has the same cadence validation and duration crediting as +/// [HeartRateZones.timeInZone]. The exact gap histograms let it defer the +/// global cadence median until [Aggregator.finish]. +// Cadence validation, median-capped interval credit and tail credit are an +// internal rule, ESTIMATE with low confidence. Karvonen 1957 supports the +// HR-reserve zone option, not this duration-credit policy. +// Histogram reformulation of HeartRateZones.timeInZone(), now its one-slice +// path. Exact cadence-gap counts and zone-credit gaps retain the same global +// median, cadence validation and duration-credit rule. Grouped sums change +// rounding; checked tolerances are 1e-12 relative and 1e-9 absolute. +Aggregator timeInZoneAggregator( + HeartRateZoneSet zoneSet, +) => + _TimeInZoneAggregator(zoneSet); + +/// Number of distinct histogram values retained by a time-in-zone state. +/// +/// This accepts [Object] so callers can inspect an empty state without knowing +/// the private state type used by the aggregator. +int timeInZoneStateEntries(Object? state) { + if (state is! _TimeInZoneState) return 0; + return state.histograms().entryCount; +} + +class _TimeInZoneAggregator + extends Aggregator<_TimeInZoneState, HrStream, TimeInHeartRateZone?> { + _TimeInZoneAggregator(this.zoneSet) + : _windows = Lookback( + 2, _PairHistograms(zoneSet), + batchFromJson: HrStream.fromJson); + + final HeartRateZoneSet zoneSet; + final Lookback _windows; + + @override + _TimeInZoneState empty() => + _TimeInZoneState(_windows.empty(), (windows) => _windows.finish(windows)); + + @override + _TimeInZoneState addSlice( + _TimeInZoneState state, HrStream batch, int start, int end) { + state.windows = _windows.addSlice(state.windows, batch, start, end); + if (start < end) { + state.hasTail = true; + state.tailTimestampMs = batch.tsMs[end - 1]; + state.tailHr = batch.hr[end - 1]; + } + return state; + } + + @override + _TimeInZoneState combine( + _TimeInZoneState left, _TimeInZoneState right) { + left.windows = _windows.combine(left.windows, right.windows); + if (right.hasTail) { + left.hasTail = true; + left.tailTimestampMs = right.tailTimestampMs; + left.tailHr = right.tailHr; + } + return left; + } + + @override + bool get persistable => _windows.persistable; + + @override + Object? encodeState(_TimeInZoneState state) => { + 'windows': _windows.encodeState(state.windows), + 'hasTail': state.hasTail, + 'tailTimestampMs': encodeDouble(state.tailTimestampMs), + 'tailHr': encodeDouble(state.tailHr), + }; + + @override + _TimeInZoneState decodeState(Object? json) { + final map = decodeMap(json); + final windows = decodeMap(field(map, 'windows')); + final length = decodeCount(field(windows, 'length')); + final hasTail = field(map, 'hasTail'); + if (hasTail is! bool) throw const FormatException('Expected a bool'); + final result = empty() + ..windows = _windows.decodeState(windows) + ..hasTail = hasTail + ..tailTimestampMs = decodeDouble(field(map, 'tailTimestampMs')) + ..tailHr = decodeDouble(field(map, 'tailHr')); + if (hasTail != (length > 0)) { + throw const FormatException('Invalid tail'); + } + _validateHistogramCounts(result.histograms(), length); + return result; + } + + @override + TimeInHeartRateZone? finish(_TimeInZoneState state) { + final histograms = state.histograms(); + final cadence = _cadence(histograms.cadence); + if (cadence == null) return null; + + final seconds = List.filled(5, 0.0); + var belowZone1 = 0.0; + for (var zone = 0; zone < histograms.credits.length; zone++) { + var total = 0.0; + for (final entry in histograms.credits[zone].entries) { + final gap = entry.key; + final credited = gap > 0 ? math.min(gap, cadence) : cadence; + total += entry.value * credited; + } + if (zone == 0) { + belowZone1 += total; + } else { + seconds[zone - 1] += total; + } + } + if (state.hasTail && state.tailHr > 0) { + final zone = zoneSet.zoneNumber(state.tailHr); + if (zone == 0) { + belowZone1 += cadence; + } else { + seconds[zone - 1] += cadence; + } + } + return TimeInHeartRateZone(seconds: seconds, belowZone1: belowZone1); + } +} + +class _TimeInZoneState { + _TimeInZoneState(this.windows, this._finishWindows); + + dynamic windows; + final _ZoneHistograms Function(dynamic) _finishWindows; + bool hasTail = false; + double tailTimestampMs = 0.0; + double tailHr = 0.0; + + _ZoneHistograms histograms() => _finishWindows(windows); +} + +/// Processes every complete two-row window supplied by [Lookback]. +class _PairHistograms + extends Aggregator<_ZoneHistograms, HrStream, _ZoneHistograms> { + const _PairHistograms(this.zoneSet); + + final HeartRateZoneSet zoneSet; + + @override + _ZoneHistograms empty() => _ZoneHistograms(); + + @override + _ZoneHistograms addSlice( + _ZoneHistograms state, HrStream batch, int start, int end) { + for (var i = start + 1; i < end; i++) { + final previous = i - 1; + final cadenceGap = batch.tsMs[i] / 1000.0 - + batch.tsMs[previous] / 1000.0; + if (cadenceGap > 0) state.addCadence(cadenceGap); + + final previousHr = batch.hr[previous]; + if (previousHr > 0) { + final creditGap = + (batch.tsMs[i] - batch.tsMs[previous]) / 1000.0; + state.addCredit(zoneSet.zoneNumber(previousHr), creditGap); + } + } + return state; + } + + @override + _ZoneHistograms combine(_ZoneHistograms left, _ZoneHistograms right) { + left.merge(right); + return left; + } + + @override + _ZoneHistograms finish(_ZoneHistograms state) => state; + + @override + bool get persistable => true; + + @override + Object? encodeState(_ZoneHistograms state) => { + 'cadence': _encodeHistogram(state.cadence), + 'credits': [ + for (final histogram in state.credits) _encodeHistogram(histogram) + ], + }; + + @override + _ZoneHistograms decodeState(Object? json) { + final map = decodeMap(json); + final state = _ZoneHistograms(); + _decodeHistogram(field(map, 'cadence'), state.cadence, + requirePositive: true); + final credits = decodeList(field(map, 'credits')); + if (credits.length != state.credits.length) { + throw const FormatException('Expected six credit histograms'); + } + for (var zone = 0; zone < credits.length; zone++) { + _decodeHistogram(credits[zone], state.credits[zone]); + } + return state; + } +} + +class _ZoneHistograms { + final Map cadence = {}; + final List> credits = + List>.generate(6, (_) => {}); + + int get entryCount => cadence.length + + credits.fold(0, (count, histogram) => count + histogram.length); + + void addCadence(double gap) => _increment(cadence, gap); + + void addCredit(int zone, double gap) => _increment(credits[zone], gap); + + void merge(_ZoneHistograms other) { + _merge(cadence, other.cadence); + for (var zone = 0; zone < credits.length; zone++) { + _merge(credits[zone], other.credits[zone]); + } + } +} + +/// A double-keyed histogram as `[[gap, count], ...]`; keys stay lossless. +List _encodeHistogram(Map histogram) => [ + for (final entry in histogram.entries) + [encodeDouble(entry.key), entry.value] + ]; + +void _decodeHistogram(Object? json, Map into, + {bool requirePositive = false}) { + for (final item in decodeList(json)) { + final pair = decodeList(item); + if (pair.length != 2) throw const FormatException('Expected [gap, count]'); + final gap = decodeDouble(pair[0]); + if (!gap.isFinite || (requirePositive && gap <= 0)) { + throw const FormatException('Invalid histogram gap'); + } + final count = decodeCount(pair[1]); + if (count == 0 || into.containsKey(gap)) { + throw const FormatException('Invalid histogram entry'); + } + into[gap] = count; + } +} + +void _validateHistogramCounts(_ZoneHistograms histograms, int length) { + final pairCount = length == 0 ? 0 : length - 1; + _validateHistogramCount(histograms.cadence.values, pairCount); + var credits = 0; + for (final histogram in histograms.credits) { + for (final count in histogram.values) { + if (count > pairCount - credits) { + throw const FormatException('Too many credit histogram entries'); + } + credits += count; + } + } +} + +void _validateHistogramCount(Iterable counts, int maximum) { + var total = 0; + for (final count in counts) { + if (count > maximum - total) { + throw const FormatException('Too many cadence histogram entries'); + } + total += count; + } +} + +void _increment(Map histogram, double value) { + histogram[value] = (histogram[value] ?? 0) + 1; +} + +void _merge(Map left, Map right) { + for (final entry in right.entries) { + left[entry.key] = (left[entry.key] ?? 0) + entry.value; + } +} + +double? _cadence(Map histogram) { + var gapCount = 0; + for (final count in histogram.values) { + gapCount += count; + } + final median = _median(histogram, gapCount); + if (median == null || median > maxSupportedCadenceSec) return null; + + var inMode = 0; + for (final entry in histogram.entries) { + final gap = entry.key; + if (gap <= median * 2.0 && gap * 2.0 >= median) { + inMode += entry.value; + } + } + return inMode < gapCount * 0.5 ? null : median; +} + +/// Equivalent to `percentileSorted(expandedHistogram, 50)` without expansion. +double? _median(Map histogram, int count) { + if (count == 0) return null; + final sorted = histogram.keys.toList()..sort(); + if (count == 1) return sorted.single; + final rank = (50 / 100) * (count - 1); + final lo = rank.floor(); + final hi = rank.ceil(); + final loValue = _valueAt(sorted, histogram, lo); + if (lo == hi) return loValue; + final hiValue = _valueAt(sorted, histogram, hi); + final fraction = rank - lo; + return loValue + (hiValue - loValue) * fraction; +} + +double _valueAt(List sorted, Map histogram, int index) { + var remaining = index; + for (final value in sorted) { + final count = histogram[value]!; + if (remaining < count) return value; + remaining -= count; + } + throw StateError('histogram index out of range'); +} diff --git a/test/aggregate/adopted_aggregators_test.dart b/test/aggregate/adopted_aggregators_test.dart new file mode 100644 index 0000000..803a4e4 --- /dev/null +++ b/test/aggregate/adopted_aggregators_test.dart @@ -0,0 +1,228 @@ +// The public batch function of each toleranced calculation IS its aggregator +// run over the whole input in one slice: empty -> addSlice(all) -> finish. +// So the batch result must be BIT-IDENTICAL to that one-slice fold (not merely +// within the chunked-fold tolerance, which the per-calculation *_aggregator +// tests and the laws harness already check). +// +// Hypnogram metrics and SRI are already wired this way. This file covers the +// rest: HRV time domain, Banister TRIMP, Cosinor, time in zone, CTL/ATL/TSB, +// PRSA DC/AC, circadian IS/IV/M10/L5/RA, readiness lnRMSSD. +// +// Readiness already shares readinessLnRmssdFromSuffix with its aggregator, so +// its test is expected to pass today (the no-dates path). + +import 'dart:math' as math; +import 'dart:typed_data'; + +import 'package:openstrap_analytics/aggregate.dart'; +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +import '../support/hrv_beats.dart'; +import '../support/property.dart' show Rng; + +final _bd = ByteData(8); +int _bits(double d) { + _bd.setFloat64(0, d); + return _bd.getInt64(0); +} + +/// Equal as bit patterns (NaN == NaN, 0.0 != -0.0); other values by ==. +bool _same(Object? a, Object? b) { + if (a is double && b is double) return _bits(a) == _bits(b); + if (a is List && b is List) { + if (a.length != b.length) return false; + for (var i = 0; i < a.length; i++) { + if (!_same(a[i], b[i])) return false; + } + return true; + } + return a == b; +} + +List _meta(Metric m) => + [m.present, m.confidence, m.tier, m.inputs_used.join(','), m.note]; + +void _bitIdentical(List got, List want, String tag) { + expect(_same(got, want), isTrue, + reason: '$tag: batch is not bit-identical to the one-slice aggregator ' + 'run\n batch $got\n aggregator $want'); +} + +DoubleColumn _col(List x) => DoubleColumn(Float64List.fromList(x)); + +void main() { + test('HRV time domain: hrvTime == hrvTimeAggregator one slice', () { + for (final (mode, seed) in [(0, 11), (1, 12), (2, 13), (3, 14)]) { + final beats = hrvSeries(Rng(seed), 4000, mode); + final nn = beats.nnMs.toList(); + final times = beats.timeMs.toList(); + final batch = hrvTime(nn, nnTimesMs: times); + final agg = hrvTimeAggregator(originMs: beats.timeMs[0]).run(beats); + final b = batch.value ?? HrvTime(nBeats: beats.length); + _bitIdentical( + [...hrvFields(b), b.nBeats], [...hrvFields(agg), agg.nBeats], + 'hrv mode $mode'); + } + }); + + test('Banister TRIMP: banisterTrimp == banisterTrimpAggregator one slice', + () { + final r = Rng(21); + final hr = [ + for (var i = 0; i < 1500; i++) + r.nextBool(.05) ? double.nan : 45 + r.nextDouble() * 150 + ]; + for (final sex in [Sex.male, Sex.female]) { + final batch = + banisterTrimp(hr, restingHr: 52, maxHr: 192, sex: sex); + final agg = banisterTrimpAggregator(restingHr: 52, maxHr: 192, sex: sex) + .run(_col(hr)); + _bitIdentical([..._meta(batch), batch.value], + [..._meta(agg), agg.value], 'trimp $sex'); + } + }); + + test('Cosinor: cosinor == cosinorAggregator one slice', () { + final r = Rng(31); + final n = 72 * 4; + final t = [for (var i = 0; i < n; i++) i * 0.25]; + final y = [ + for (final h in t) + 60 + + 12 * math.cos(2 * math.pi * (h - 15) / 24) + + (r.nextDouble() - .5) * 6 + ]; + final batch = cosinor(t, y); + final agg = cosinorAggregator() + .run(TimedValues(Float64List.fromList(t), Float64List.fromList(y))); + List f(Metric m) => [ + ..._meta(m), + m.value?.mesor, + m.value?.amplitude, + m.value?.acrophaseRad, + m.value?.acrophaseHours, + m.value?.periodHours, + m.value?.r2, + m.value?.r2Adj, + ]; + _bitIdentical(f(batch), f(agg), 'cosinor'); + }); + + test('Time in zone: HeartRateZones.timeInZone == timeInZoneAggregator ' + 'one slice', () { + final r = Rng(41); + final zones = HeartRateZones.zonesFromMaxHr(190); + final ts = []; + final hr = []; + var t = 1.7e12; + var h = 120.0; + for (var i = 0; i < 6000; i++) { + // Jittered ~1 Hz clock, so the credited seconds are not whole numbers + // and the float sums depend on summation order. + t += r.nextBool(.02) + ? 1000.0 * r.intIn(5, 90) + : 1000.0 + r.intIn(-37, 37) / 8; + h = (h + (r.nextDouble() - .5) * 14).clamp(40.0, 195.0); + ts.add(t); + hr.add(r.nextBool(.03) ? 0 : h); + } + final batch = HeartRateZones.timeInZone( + [for (var i = 0; i < ts.length; i++) HrSample(ts[i], hr[i])], zones); + final agg = timeInZoneAggregator(zones).run( + HrStream(Float64List.fromList(ts), Float64List.fromList(hr))); + List f(TimeInHeartRateZone? z) => + z == null ? [null] : [...z.seconds, z.belowZone1]; + expect(batch, isNotNull); + _bitIdentical(f(batch), f(agg), 'time in zone'); + }); + + test('CTL/ATL/TSB: ctlAtlTsb == ctlAtlTsbAggregator one slice', () { + List f(Metric m) => + [..._meta(m), m.value?.ctl, m.value?.atl, m.value?.tsb]; + for (final (seed, days) in [(51, 120), (52, 45), (53, 200), (54, 365)]) { + final r = Rng(seed); + final trimp = [ + for (var i = 0; i < days; i++) + i % 7 == 6 ? 0.0 : 40 + r.nextDouble() * 160 + ]; + final batch = ctlAtlTsb(trimp); + final agg = ctlAtlTsbAggregator().run(_col(trimp)); + expect(batch.present, isTrue); + _bitIdentical(f(batch), f(agg), 'ctl/atl/tsb seed $seed, $days days'); + } + }); + + test('PRSA DC/AC: decelerationCapacity/accelerationCapacity == ' + 'prsaAggregator one slice', () { + final r = Rng(61); + var v = 900.0; + final nn = [ + for (var i = 0; i < 3000; i++) + v = (v + (r.nextDouble() - .5) * 90).clamp(700.0, 1100.0) + ]; + List f(Metric m) => [ + ..._meta(m), + m.value?.capacity, + m.value?.profile, + m.value?.anchors, + m.value?.kind, + ]; + for (final dec in [true, false]) { + final batch = + dec ? decelerationCapacity(nn) : accelerationCapacity(nn); + final agg = prsaAggregator(deceleration: dec).run(_col(nn)); + expect(batch.present, isTrue); + _bitIdentical(f(batch), f(agg), dec ? 'DC' : 'AC'); + } + }); + + test('Circadian IS/IV/M10/L5/RA: circadianNonparametric == ' + 'circadianNpAggregator one slice', () { + final r = Rng(71); + const perDay = 96; + final x = [ + for (var i = 0; i < perDay * 6; i++) + 50 + + 40 * math.sin(2 * math.pi * (i % perDay - 30) / perDay) + + r.nextDouble() * 25 + ]; + final batch = circadianNonparametric(x, perDay); + final agg = circadianNpAggregator(perDay) + .run(EpochSeries(Float64List.fromList(x))); + List f(Metric m) => [ + ..._meta(m), + m.value?.interdailyStability, + m.value?.intradailyVariability, + m.value?.m10, + m.value?.l5, + m.value?.relativeAmplitude, + m.value?.m10StartEpoch, + m.value?.l5StartEpoch, + ]; + expect(batch.present, isTrue); + _bitIdentical(f(batch), f(agg), 'circadian'); + }); + + test('Readiness lnRMSSD: readinessLnRmssd == readinessLnRmssdAggregator ' + 'one slice (shares readinessLnRmssdFromSuffix; may already pass)', () { + final r = Rng(81); + final hist = [for (var i = 0; i < 40; i++) 3.6 + (r.nextDouble() - .5) * .6]; + List f(Metric m) => [ + ..._meta(m), + m.value?.lnRmssdToday, + m.value?.rolling7Mean, + m.value?.rolling7Median, + m.value?.cvPct, + m.value?.z, + m.value?.swc, + m.value?.band, + m.value?.saturationFlag, + ]; + final batch = readinessLnRmssd(hist, meanNnTodayMs: 880); + final agg = + readinessLnRmssdAggregator(meanNnTodayMs: 880).run(_col(hist)); + expect(batch.present, isTrue); + _bitIdentical(f(batch), f(agg), 'readiness'); + }); +} diff --git a/test/aggregate/aggregator_core_test.dart b/test/aggregate/aggregator_core_test.dart new file mode 100644 index 0000000..215f772 --- /dev/null +++ b/test/aggregate/aggregator_core_test.dart @@ -0,0 +1,242 @@ +// Aggregator core: the Batch implementations, fold (with ranges) / run / +// runList, map and both. + +import 'dart:typed_data'; + +import 'package:openstrap_analytics/aggregate.dart'; +import 'package:test/test.dart'; + +import '../support/aggregator_laws.dart'; +import '../support/property.dart' show Rng; + +/// Sum of the int rows. State is a one-slot list so combine/addSlice can mutate. +class _SumRows extends Aggregator, ListBatch, int> { + const _SumRows(); + @override + List empty() => [0]; + @override + List addSlice(List s, ListBatch b, int start, int end) { + for (var i = start; i < end; i++) { + s[0] += b.rows[i]; + } + return s; + } + + @override + List combine(List left, List right) { + left[0] += right[0]; + return left; + } + + @override + int finish(List s) => s[0]; +} + +/// Rows seen, in order, as a list (state mutated in place). +class _Rows extends Aggregator, ListBatch, List> { + const _Rows(); + @override + List empty() => []; + @override + List addSlice(List s, ListBatch b, int start, int end) => + s..addAll(b.rows.sublist(start, end)); + @override + List combine(List left, List right) => left..addAll(right); + @override + List finish(List s) => List.of(s); +} + +bool _same(T a, T b) => a.toString() == b.toString(); + +void main() { + group('ListBatch', () { + test('length, rows and copyRange', () { + final b = ListBatch(['a', 'b', 'c', 'd']); + expect(b.length, 4); + expect(b.rows, ['a', 'b', 'c', 'd']); + expect(b.copyRange(1, 3).rows, ['b', 'c']); + expect(b.copyRange(0, 4).rows, ['a', 'b', 'c', 'd']); + expect(b.copyRange(2, 2).length, 0); + expect(b.copyRange(4, 4).length, 0); + expect(b.copyRange(1, 3).length, 2); + }); + + test('copyRange is a copy: later changes to the source do not show', () { + final rows = [1, 2, 3, 4]; + final b = ListBatch(rows); + final c = b.copyRange(1, 4); + rows[1] = 99; + rows[3] = 77; + expect(c.rows, [2, 3, 4]); + }); + + test('join puts this before other and leaves both unchanged', () { + final a = ListBatch([1, 2]), b = ListBatch([3, 4, 5]); + final j = a.join(b); + expect(j.rows, [1, 2, 3, 4, 5]); + expect(j.length, 5); + expect(a.rows, [1, 2]); + expect(b.rows, [3, 4, 5]); + expect(a.join(ListBatch([])).rows, [1, 2]); + expect(ListBatch([]).join(b).rows, [3, 4, 5]); + expect(ListBatch([]).join(ListBatch([])).length, 0); + }); + }); + + group('DoubleColumn', () { + test('length, values and copyRange', () { + final c = col([1, 2, 3, 4]); + expect(c.length, 4); + expect(c.copyRange(1, 3).values, [2.0, 3.0]); + expect(c.copyRange(0, 4).values, [1.0, 2.0, 3.0, 4.0]); + expect(c.copyRange(3, 3).length, 0); + }); + + test('copyRange is a copy: later changes to the source do not show', () { + final v = Float64List.fromList([1, 2, 3, 4]); + final c = DoubleColumn(v).copyRange(1, 3); + v[1] = 99; + v[2] = 77; + expect(c.values, [2.0, 3.0]); + }); + + test('join, with NaN kept as a missing value', () { + final a = col([1, double.nan]), b = col([3]); + final j = a.join(b); + expect(j.length, 3); + expect(j.values[0], 1.0); + expect(j.values[1].isNaN, isTrue); + expect(j.values[2], 3.0); + expect(a.length, 2); + expect(b.values, [3.0]); + expect(a.join(col([])).length, 2); + expect(col([]).join(b).values, [3.0]); + }); + }); + + group('fold, run, runList', () { + const sum = _SumRows(); + final batch = ListBatch([for (var i = 1; i <= 5; i++) i]); + + test('run of nothing is the identity result', () { + expect(sum.run(ListBatch([])), 0); + expect(sum.runList([]), 0); + }); + + test('run equals finish(fold) and folds in order', () { + expect(sum.run(batch), 15); + expect(sum.finish(sum.fold(batch)), 15); + expect(const _Rows().run(batch), [1, 2, 3, 4, 5]); + }); + + test('runList is run over a ListBatch', () { + expect(sum.runList([4, 5, 6]), 15); + expect(const _Rows().runList([9, 8, 7]), [9, 8, 7]); + }); + + test('fold takes a half-open range; end defaults to the length', () { + expect(sum.finish(sum.fold(batch, 1, 4)), 2 + 3 + 4); + expect(sum.finish(sum.fold(batch, 0, 5)), 15); + expect(sum.finish(sum.fold(batch, 2)), 3 + 4 + 5); + expect(sum.finish(sum.fold(batch, 5)), 0); + expect(sum.finish(sum.fold(batch, 2, 2)), 0); + expect(sum.finish(sum.fold(batch, 5, 5)), 0); + expect(const _Rows().finish(const _Rows().fold(batch, 1, 3)), [2, 3]); + }); + + test('fold ranges combine back to the whole at every split', () { + final xs = ListBatch([for (var i = 1; i <= 20; i++) i]); + for (var k = 0; k <= 20; k++) { + final s = sum.combine(sum.fold(xs, 0, k), sum.fold(xs, k, 20)); + expect(sum.finish(s), 210, reason: 'split at $k'); + } + }); + + test('addSlice continues a state: several slices equal one', () { + var s = sum.empty(); + s = sum.addSlice(s, batch, 0, 2); + s = sum.addSlice(s, batch, 2, 2); + s = sum.addSlice(s, batch, 2, 5); + expect(sum.finish(s), 15); + }); + + test('empty() returns independent states', () { + final a = sum.empty(), b = sum.empty(); + sum.addSlice(a, batch, 0, 5); + expect(sum.finish(b), 0); + }); + + test('chunked and sliced runs agree with the whole run', () { + const agg = _Rows(); + final r = Rng(99); + for (final b in randomIntBatches(1)) { + for (var t = 0; t < 6; t++) { + final bounds = randomBounds(r, b.length); + expect(runChunked(agg, b, bounds), b.rows, reason: '$bounds'); + expect(runChunked(agg, b, bounds, copy: true), b.rows, + reason: 'copy $bounds'); + expect(runChunkedRight(agg, b, bounds), b.rows, reason: '$bounds'); + expect(runSliced(agg, b, bounds), b.rows, reason: '$bounds'); + } + } + }); + }); + + group('map', () { + test('transforms the result only', () { + final a = const _SumRows().map((n) => 'n=$n'); + expect(a.runList([1, 2, 3]), 'n=6'); + expect(a.runList([]), 'n=0'); + expectLawful(a, randomIntBatches(2)); + }); + + test('composes', () { + final a = Count>().map((n) => n * 2).map((n) => n + 1); + expect(a.runList([7, 7, 7]), 7); + }); + + test('keeps ranges and seams working', () { + final a = const _SumRows().map((n) => n * 10); + final b = ListBatch([1, 2, 3, 4]); + expect(a.finish(a.combine(a.fold(b, 0, 2), a.fold(b, 2, 4))), 100); + expect(a.finish(a.fold(b, 1, 3)), 50); + }); + }); + + group('both', () { + final a = both(const _SumRows(), Count>()); + + test('runs two aggregators over the same rows', () { + expect(a.runList([3, 4, 5]), (12, 3)); + expect(a.runList([]), (0, 0)); + }); + + test('nests and stays lawful', () { + final n = both(a, const _Rows()); + final r = n.runList([3, 9, 5]); // records holding lists compare by identity + expect(r.$1, (17, 3)); + expect(r.$2, [3, 9, 5]); + expectLawful(n, randomIntBatches(3), equals: _same); + }); + + test('each side has its own fresh state', () { + final p = a.addSlice(a.empty(), ListBatch([5]), 0, 1); + final q = a.empty(); + expect(a.finish(q), (0, 0)); + expect(a.finish(p), (5, 1)); + }); + + test('combines across a seam and honours fold ranges', () { + final b = ListBatch([1, 2, 3, 4]); + expect(a.finish(a.combine(a.fold(b, 0, 2), a.fold(b, 2, 4))), (10, 4)); + expect(a.finish(a.fold(b, 1, 3)), (5, 2)); + }); + + test('Pair is a mutable holder', () { + final p = Pair(1, 'a'); + p.first = 2; + p.second = 'b'; + expect((p.first, p.second), (2, 'b')); + }); + }); +} diff --git a/test/aggregate/circadian_np_aggregator_test.dart b/test/aggregate/circadian_np_aggregator_test.dart new file mode 100644 index 0000000..d93f397 --- /dev/null +++ b/test/aggregate/circadian_np_aggregator_test.dart @@ -0,0 +1,317 @@ +// EpochSeries + circadianNpAggregator must reproduce circadianNonparametric +// (the reference, unchanged) under every slicing / chunking, and obey the laws. +// +// Contract: EpochSeries(Float64List x, {firstIndex = 0}); copyRange copies and +// shifts firstIndex; join asserts other.firstIndex == firstIndex + length. +// circadianNpAggregator(epochsPerDay, {m10Epochs, l5Epochs}) never throws; +// epochsPerDay <= 0 is the reference's absent Metric. +// +// Parity: integer data -> IS/IV/M10/L5/RA within relTol 1e-12 and start epochs +// EXACT. Real data -> values within relTol 1e-9, start epochs NOT compared (the +// window margin is not available to the equality, so only integer cases check +// starts). + +import 'dart:typed_data'; + +import 'package:openstrap_analytics/aggregate.dart'; +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +import '../support/checkpoint_support.dart'; +import '../support/aggregator_laws.dart'; +import '../support/property.dart' show Rng; + +List _meta(Metric m) => + [m.present, m.confidence, m.tier, m.inputs_used.join(','), m.note]; + +List _vals(Metric m) { + final v = m.value; + return [ + ..._meta(m), + v?.interdailyStability, v?.intradailyVariability, v?.m10, v?.l5, + v?.relativeAmplitude, + ]; +} + +List _withStarts(Metric m) => + [..._vals(m), m.value?.m10StartEpoch, m.value?.l5StartEpoch]; + +final _exact = fieldsEqual>(_withStarts, relTol: 1e-12); +final _real = fieldsEqual>(_vals, relTol: 1e-9); + +EpochSeries _es(List x, {int first = 0}) => EpochSeries( + Float64List.fromList([for (final v in x) v.toDouble()]), + firstIndex: first); + +/// Seeded series: [count] lengths in 0..10 days plus a partial day. +List _cases(int seed, int p, double Function(Rng) draw, + {int count = 10}) { + final r = Rng(seed); + return [ + _es([]), + _es([draw(r)]), + for (var i = 0; i < count; i++) + _es([ + for (var k = r.intIn(0, 10 * p + p - 1); k > 0; k--) draw(r) + ]), + _es([for (var k = 0; k < 3 * p; k++) draw(r)]), + ]; +} + +void _parity(String name, int p, {int? m10, int? l5, bool ints = true}) { + test(name, () { + final draw = ints + ? (Rng r) => r.intIn(0, 500).toDouble() + : (Rng r) => r.nextDouble() * 300 - 20; + expectConversion>( + (_) => circadianNpAggregator(p, m10Epochs: m10, l5Epochs: l5), + _cases(p * 31 + (ints ? 1 : 0), p, draw), + (b) => circadianNonparametric(b.x, p, m10Epochs: m10, l5Epochs: l5), + equals: ints ? _exact : _real, + seed: p, + ); + }); +} + +void main() { + group('EpochSeries batch', () { + final e = _es([for (var i = 0; i < 10; i++) i * 1.5]); + + test('copyRange shifts firstIndex and copies', () { + final c = e.copyRange(3, 8); + expect(c.length, 5); + expect(c.firstIndex, 3); + expect(c.x, e.x.sublist(3, 8)); + expect(c.copyRange(2, 4).firstIndex, 5); + expect(_es([1, 2, 3], first: 100).copyRange(1, 3).firstIndex, 101); + expect(e.copyRange(4, 4).length, 0); + e.x[4] = -1; + expect(c.x[1], 6.0); + e.x[4] = 6.0; + }); + + test('join of adjacent batches', () { + final j = e.copyRange(2, 5).join(e.copyRange(5, 5)).join(e.copyRange(5, 9)); + expect(j.firstIndex, 2); + expect(j.x, e.x.sublist(2, 9)); + final l = e.copyRange(0, 3); + final k = l.join(e.copyRange(3, 10)); + l.x[0] = 99; + expect(k.x[0], 0.0); + }); + + test('non-adjacent join asserts', () { + expect(() => e.copyRange(0, 3).join(e.copyRange(4, 10)), + throwsA(isA())); + expect(() => e.copyRange(3, 6).join(e.copyRange(0, 3)), + throwsA(isA())); + }); + }); + + group('parity with circadianNonparametric', () { + for (final p in [6, 7, 12, 24]) { + _parity('integer data, p=$p', p); + _parity('real data, p=$p', p, ints: false); + } + _parity('custom windows, integers', 12, m10: 5, l5: 3); + _parity('custom windows, real', 12, m10: 5, l5: 3, ints: false); + _parity('windows w >= p', 8, m10: 8, l5: 20); + _parity('window w == 1', 8, m10: 1, l5: 1); + + test('3 days of 1440 epochs, integers', () { + final r = Rng(1440); + final x = _es([for (var i = 0; i < 3 * 1440; i++) r.intIn(0, 900)]); + expectConversion>( + (_) => circadianNpAggregator(1440), [x], + (b) => circadianNonparametric(b.x, 1440), + // Laws are covered by the small-p cases above; here each finish runs + // the O(p·w) window search, so the law sweep alone took ~9 s. + equals: _exact, chunkings: 4, laws: false); + }); + }); + + test('persistable: a mid-stream checkpoint resumes to the one-pass result', () { + final x = _cases(0xC0DE, 12, (r) => r.intIn(0, 500).toDouble()).last; + expectCheckpointResumes(circadianNpAggregator(12), x, equals: _exact); + }); + + group('absent cases and extremes', () { + void same(String name, EpochSeries s, int p, {int? m10, int? l5}) => + test(name, () { + final want = circadianNonparametric(s.x, p, m10Epochs: m10, l5Epochs: l5); + final got = circadianNpAggregator(p, m10Epochs: m10, l5Epochs: l5) + .run(s); + expect(_exact(got, want), isTrue, reason: '$got vs $want'); + expect(got.present, want.present); + }); + + same('empty', _es([]), 4); + same('shorter than a day', _es([1, 2, 3]), 4); + same('constant series (no variance)', _es(List.filled(12, 7)), 4); + same('constant zero series', _es(List.filled(8, 0)), 4); + same('exactly one day', _es([1, 5, 2, 8]), 4); + same('partial trailing day uses n', _es([1, 5, 2, 8, 3, 9, 1]), 4); + for (final p in [0, -1, -24]) { + same('epochsPerDay $p', _es([1, 5, 2, 8, 3, 9, 1, 4]), p); + same('epochsPerDay $p, empty', _es([]), p); + } + + test('absent reasons differ', () { + expect(circadianNpAggregator(4).run(_es([1, 2])).note, + contains('≥1 full day')); + expect(circadianNpAggregator(4).run(_es(List.filled(8, 3))).note, + 'no variance in activity series'); + }); + + test('epochsPerDay <= 0 never throws on any path', () { + final s = _es([for (var i = 0; i < 9; i++) i * 1.0]); + final a = circadianNpAggregator(0); + expect(a.finish(a.empty()).present, isFalse); + expect(runChunked(a, s, [0, 4, 9]).present, isFalse); + expect(runSliced(a, s, [0, 0, 5, 9]).present, isFalse); + }); + }); + + group('missing and non-finite values follow the reference policy', () { + const nan = double.nan; + const inf = double.infinity; + + void same(String name, List x, int p, {int? m10, int? l5}) => + test(name, () { + final s = _es(x); + final want = circadianNonparametric(s.x, p, m10Epochs: m10, l5Epochs: l5); + final agg = circadianNpAggregator(p, m10Epochs: m10, l5Epochs: l5); + final got = agg.run(s); + expect(_exact(got, want), isTrue, reason: 'whole: $got vs $want'); + // Every path, not only the whole fold. + final n = x.length; + for (final bounds in [ + [0, n ~/ 2, n], + [0, 1, n], + [0, n - 1, n], + [for (var i = 0; i <= n; i++) i], + ]) { + expect(_exact(runChunked(agg, s, bounds), want), isTrue, + reason: 'chunked $bounds'); + expect(_exact(runSliced(agg, s, bounds), want), isTrue, + reason: 'sliced $bounds'); + } + }); + + same('review: six NaNs, epochsPerDay 6 (used to throw)', + [nan, nan, nan, nan, nan, nan], 6); + same('review: [1,2,3,4,5,NaN], epochsPerDay 6 (L5 and RA were wrong)', + [1, 2, 3, 4, 5, nan], 6); + same('NaN in the second day', [1, 2, 3, 4, 5, 6, 7, nan, 9, 1, 2, 3], 6); + same('a NaN in every epoch-of-day bin', [nan, 1, 2, 3, 4, 5, 6, nan, 8], 4); + same('a whole bin missing', [1, nan, 3, 4, 5, nan, 7, 8, 9, nan, 2, 3], 4); + same('NaN with custom windows', [3, 1, nan, 9, 2, 8, 7, nan, 5, 4], 5, + m10: 2, l5: 3); + same('one +Infinity', [1, 2, inf, 4, 5, 6, 7, 8], 4); + same('one -Infinity', [1, 2, -inf, 4, 5, 6, 7, 8], 4); + same('+Infinity and -Infinity', [1, inf, 3, -inf, 5, 6, 7, 8], 4); + same('NaN and Infinity together', [nan, inf, 3, 4, 5, 6, 7, 8], 4); + + test('series with sprinkled NaN agree with the reference', () { + for (final p in [4, 6, 12]) { + final r = Rng(0xA11 + p); + expectConversion>( + (_) => circadianNpAggregator(p), + [ + for (var c = 0; c < 14; c++) + _es([ + for (var k = r.intIn(p, 6 * p); k > 0; k--) + r.nextBool(c.isEven ? .08 : .3) + ? nan + : r.intIn(0, 500).toDouble() + ]), + ], + (b) => circadianNonparametric(b.x, p), + equals: _exact, + seed: p, + ); + } + }); + }); + + group('checkpoint relationships (counts and bins agree across children)', () { + // p = 6, 9 epochs: bins 0..5 all present, bins 0..2 hold two epochs. + final agg = circadianNpAggregator(6); + final x = _es([4, 8, 1, 9, 2, 7, 5, 3, 6]); + + // State: [[n, moments], [groups, lookback]]; a group is [count, sum]. + Map groups(Object? s) => ((s as List)[1] as List)[0] as Map; + Map lookback(Object? s) => ((s as List)[1] as List)[1] as Map; + + test('review case: group "0" removed is refused', () { + // Used to decode, then finish threw on the missing bin. + expectDamagedCheckpointRefused(agg, x, (s) { + groups(s).remove('0'); + return s; + }); + }); + test('another bin removed, or an extra bin added, is refused', () { + expectDamagedCheckpointRefused(agg, x, (s) { + groups(s).remove('5'); + return s; + }); + expectDamagedCheckpointRefused(agg, x, (s) { + groups(s)['6'] = groups(s)['0']; + return s; + }); + expectDamagedCheckpointRefused(agg, x, (s) { + groups(s)['-1'] = groups(s)['0']; + return s; + }); + }); + test('group counts that do not sum to the total are refused', () { + expectDamagedCheckpointRefused(agg, x, (s) { + final g = groups(s)['3'] as List; + g[0] = (g[0] as int) + 1; + return s; + }); + expectDamagedCheckpointRefused(agg, x, (s) { + ((s as List)[0] as List)[0] = 10; + return s; + }); + }); + test('a count that puts the epochs in the wrong bins is refused', () { + // Same total, but bins 0 and 3 swap their counts: 0 should hold two. + expectDamagedCheckpointRefused(agg, x, (s) { + final g0 = groups(s)['0'] as List, g5 = groups(s)['5'] as List; + g0[0] = 1; + g5[0] = 2; + return s; + }); + }); + test('a lookback row count that differs from the total is refused', () { + expectDamagedCheckpointRefused(agg, x, (s) { + lookback(s)['length'] = 12; + return s; + }); + }); + test('the bins required follow firstIndex, not just the count', () { + // Rows 4..8 only: epochs of day 4,5,0,1,2, so bin 3 must be absent. + final tail = EpochSeries(Float64List.fromList([1, 2, 3, 4, 5]), + firstIndex: 4); + final a = circadianNpAggregator(6); + final good = checkpointOf(a, tail); + expect(((good['state'] as List)[1] as List)[0] as Map, + hasLength(5)); // sanity: bins 0,1,2,4,5 + expect(decodeCheckpoint(a, good, kind: 'test', version: 1), + isNotNull); + expectDamagedCheckpointRefused(a, tail, (s) { + groups(s)['3'] = groups(s)['0']; + return s; + }); + expectDamagedCheckpointRefused(a, tail, (s) { + final g = groups(s); + g['3'] = g.remove('4'); // right number of bins, wrong ones + return s; + }); + }); + test('a consistent checkpoint resumes after the checks', () { + expectCheckpointResumes(agg, x, equals: _exact); + }); + }); +} diff --git a/test/aggregate/conversion_check_test.dart b/test/aggregate/conversion_check_test.dart new file mode 100644 index 0000000..4e7f3c5 --- /dev/null +++ b/test/aggregate/conversion_check_test.dart @@ -0,0 +1,260 @@ +// checkConversion: correct aggregators pass; each deliberately broken one is +// caught on the path that names its defect (and not on the paths it gets right). + +import 'package:openstrap_analytics/aggregate.dart'; +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +import '../support/aggregator_laws.dart'; +import '../support/property.dart' show Rng; + +typedef _B = ListBatch; + +int _sliceSum(_B b, int start, int end) { + var t = 0; + for (var i = start; i < end; i++) { + t += b.rows[i]; + } + return t; +} + +int _refSum(_B b) { + var t = 0; + for (final x in b.rows) { + t += x; + } + return t; +} + +/// A correct sum over int rows. +class _Sum extends Aggregator { + const _Sum(); + @override + int empty() => 0; + @override + int addSlice(int s, _B b, int start, int end) => s + _sliceSum(b, start, end); + @override + int combine(int left, int right) => left + right; + @override + int finish(int s) => s; +} + +/// combine drops the right side: wrong only when chunks were combined. +class _DropRight extends _Sum { + const _DropRight(); + @override + int combine(int left, int right) => left; +} + +/// addSlice restarts from the slice, so a second addSlice onto a state that +/// already holds data loses it. Only the 'sliced' path adds twice to one state. +class _IgnoresState extends _Sum { + const _IgnoresState(); + @override + int addSlice(int s, _B b, int start, int end) => _sliceSum(b, start, end); +} + +/// State: [sum, chunks]. combine is right only when the right side is a single +/// chunk (it drops a multi-chunk right side), so left-to-right combining works +/// and right-nested combining does not. +class _LeftOnly extends Aggregator, _B, int> { + const _LeftOnly(); + @override + List empty() => [0, 0]; + @override + List addSlice(List s, _B b, int start, int end) => + [s[0] + _sliceSum(b, start, end), s[1] + (s[1] == 0 ? 1 : 0)]; + @override + List combine(List l, List r) => + r[1] > 1 ? [l[0], l[1] + r[1]] : [l[0] + r[0], l[1] + r[1]]; + @override + int finish(List s) => s[0]; +} + +/// Keeps the FIRST batch it was shown plus index ranges into it, and reads the +/// rows only in finish (clamped, so a foreign batch cannot throw). Over one +/// batch this is a correct sum. Chunks cut out with copyRange are separate +/// batches with their own indexing, so the ranges of later chunks are read from +/// the first chunk's storage: wrong values. Only the 'copies' path differs. +class _Aliasing extends Aggregator<_AliasState, _B, int> { + const _Aliasing(); + @override + _AliasState empty() => _AliasState(null, []); + @override + _AliasState addSlice(_AliasState s, _B b, int start, int end) { + s.ranges.add([start, end]); + return _AliasState(s.source ?? b, s.ranges); + } + + @override + _AliasState combine(_AliasState l, _AliasState r) => + _AliasState(l.source ?? r.source, [...l.ranges, ...r.ranges]); + + @override + int finish(_AliasState s) { + final src = s.source; + if (src == null) return 0; + var t = 0; + for (final r in s.ranges) { + for (var i = r[0]; i < r[1] && i < src.length; i++) { + t += src.rows[i]; + } + } + return t; + } +} + +class _AliasState { + _AliasState(this.source, this.ranges); + final _B? source; + final List> ranges; +} + +/// Sum of (row - origin); origin comes from the case via build. +class _SumFrom extends Aggregator { + const _SumFrom(this.origin); + final int origin; + @override + int empty() => 0; + @override + int addSlice(int s, _B b, int start, int end) => + s + _sliceSum(b, start, end) - origin * (end - start); + @override + int combine(int left, int right) => left + right; + @override + int finish(int s) => s; +} + +int _originOf(_B b) => b.rows.isEmpty ? 0 : b.rows.first; + +Set _paths(List fs) => {for (final f in fs) f.path}; + +void main() { + final cases = randomIntBatches(8, count: 24, maxLen: 30); + + List check(Aggregator agg, + {int Function(_B)? reference, bool laws = true, int seed = 0}) => + checkConversion((_) => agg, cases, reference ?? _refSum, + laws: laws, seed: seed); + + group('correct aggregators pass', () { + test('Sum against a reference loop', () { + expectConversion((_) => const _Sum(), cases, _refSum); + expect(check(const _Sum()), isEmpty); + }); + + test('hypnogram aggregator against hypnogramMetrics', () { + final r = Rng(5); + const stages = ['wake', 'light', 'deep', 'rem']; + final batches = >[]; + for (var c = 0; c < 6; c++) { + final n = c == 0 ? 0 : r.intIn(1, 14); + var t = r.intIn(0, 60); + final segs = []; + for (var i = 0; i < n; i++) { + final d = r.intIn(1, 120); + segs.add(StageSegment(t, t + d, stages[r.intIn(0, 3)])); + t += d + r.intIn(0, 20); + } + batches.add(ListBatch(segs)); + } + expectConversion, HypnogramMetrics>( + (_) => HypnogramAggregator(0, 3000), + batches, + (b) => AdvancedSleepStager.hypnogramMetrics(SleepSession( + start: 0, + end: 3000, + efficiency: 0, + stages: b.rows, + restingHr: null, + avgHrv: null)), + equals: fieldsEqual((m) => [ + m.tibS, m.tstS, m.sptS, m.solS, m.remLatencyS, m.wasoS, + m.efficiency, m.disturbances, m.deepMin, m.remMin, m.lightMin, + m.deepPct, m.remPct, m.lightPct, + ]), + ); + }); + }); + + group('broken aggregators are caught on the right path', () { + test('combine drops right: chunked and law:split, not whole', () { + final p = _paths(check(const _DropRight())); + expect(p, containsAll(['chunked', 'law:split'])); + expect(p, isNot(contains('whole'))); + }); + + test('state aliasing the input: copies fails, whole passes', () { + final p = _paths(check(const _Aliasing())); + expect(p, contains('copies')); + expect(p, isNot(contains('whole'))); + expect(p, isNot(contains('chunked'))); + expect(p, isNot(contains('sliced'))); + }); + + test('addSlice ignoring its state: sliced and law:slicing', () { + final p = _paths(check(const _IgnoresState())); + expect(p, containsAll(['sliced', 'law:slicing'])); + expect(p, isNot(contains('whole'))); + expect(p, isNot(contains('chunked'))); + }); + + test('right-nested combine broken: chunkedRight, not chunked', () { + final p = _paths(check(const _LeftOnly(), laws: false)); + expect(p, contains('chunkedRight')); + expect(p, isNot(contains('whole'))); + expect(p, isNot(contains('chunked'))); + }); + + test('wrong reference everywhere: whole fails on every case', () { + final fs = check(const _Sum(), reference: (b) => _refSum(b) + 1); + final whole = fs.where((f) => f.path == 'whole'); + expect({for (final f in whole) f.caseIndex}, + {for (var i = 0; i < cases.length; i++) i}); + }); + + test('failures carry a detail string', () { + final fs = check(const _DropRight()); + expect(fs, isNotEmpty); + expect(fs.every((f) => f.detail.isNotEmpty), isTrue); + }); + }); + + group('build is context-indexed', () { + test('per-case parameter comes from build(input)', () { + final seen = <_B>[]; + expectConversion((input) { + seen.add(input); + return _SumFrom(_originOf(input)); + }, cases, (b) => _refSum(b) - _originOf(b) * b.length); + expect(seen, isNotEmpty); + expect(seen.toSet(), containsAll(cases)); + }); + + test('a build that ignores the case is caught', () { + final fs = checkConversion((_) => const _SumFrom(0), cases, + (b) => _refSum(b) - _originOf(b) * b.length); + expect(_paths(fs), contains('whole')); + }); + }); + + group('determinism and options', () { + test('same seed gives identical failures', () { + String show(List fs) => + fs.map((f) => '${f.path}|${f.caseIndex}|${f.detail}').join('\n'); + expect(show(check(const _Aliasing(), seed: 3)), + show(check(const _Aliasing(), seed: 3))); + expect(show(check(const _DropRight(), seed: 3)), + show(check(const _DropRight(), seed: 3))); + }); + + test('laws: false skips law paths', () { + expect(_paths(check(const _DropRight())).any((p) => p.startsWith('law:')), + isTrue); + expect( + _paths(check(const _DropRight(), laws: false)) + .any((p) => p.startsWith('law:')), + isFalse); + }); + }); +} diff --git a/test/aggregate/cosinor_aggregator_test.dart b/test/aggregate/cosinor_aggregator_test.dart new file mode 100644 index 0000000..e4f5044 --- /dev/null +++ b/test/aggregate/cosinor_aggregator_test.dart @@ -0,0 +1,587 @@ +// cosinorAggregator (sufficient statistics) must agree with cosinor() for any +// series and any chunking or slicing, and obey the aggregator laws. +// +// Contract used here: +// * TimedValues(tHours, y): equal-length Float64Lists (assert). Row r is the +// sample (tHours[r], y[r]). copyRange returns copies; join concatenates. +// TimedValues and cosinorAggregator are exported from +// package:openstrap_analytics/aggregate.dart. No NaN handling is promised +// or tested (cosinor() has none); a tHours/y length mismatch cannot occur. +// * cosinorAggregator({periodHours = 24, minPoints = cosinorMinPoints}) is an +// Aggregator<_, TimedValues, Metric>. Constructing it never +// throws, also for periodHours <= 0 (then every run is absent). +// * Absent exactly when cosinor() is absent: n < max(4, minPoints), +// periodHours <= 0 ('too few points for cosinor'), or a singular design +// (relative centered determinant <= 1e-12 of the diagonal product; +// 'singular cosinor design (insufficient phase coverage)'). +// Absent metrics carry the same tier, inputs_used and note. +// * Present: mesor, amplitude, acrophaseRad within relTol 1e-9 (acrophaseRad +// compared modulo 2 pi and with an absolute floor of 1e-9, because a phase +// near 0 or +-pi has no meaningful relative error; for an amplitude that is +// only rounding noise, below 1e-9 * max(1, |mesor|), the amplitude gets an +// absolute floor of 1e-12 * max(1, |mesor|) and the phases are not compared); +// acrophaseHours within +// relTol 1e-9 of the period, modulo periodHours; periodHours equal; r2, +// r2Adj, confidence absTol 1e-7; tier, inputs_used, note equal. +// * Batch now runs the same aggregator. The near-singular fixture below +// pins its relative gate; ordinary parity fixtures avoid that threshold. + +import 'dart:math' as math; +import 'dart:typed_data'; + +import 'package:openstrap_analytics/aggregate.dart'; +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +import '../support/checkpoint_support.dart'; +import '../support/aggregator_laws.dart'; +import '../support/property.dart' show Rng; + +const _twoPi = 2 * math.pi; + +typedef _Agg = Aggregator>; + +TimedValues _tv(List t, List y) => + TimedValues(Float64List.fromList(t), Float64List.fromList(y)); + +/// Distance of [a] and [b] on a circle of circumference [period]. +double _circ(double a, double b, double period) { + var d = (a - b) % period; + if (d < 0) d += period; + return math.min(d, period - d); +} + +/// True when a finite/non-finite mismatch (or two different non-finite values) +/// makes a tolerance comparison meaningless: NaN fails every `>` check, so +/// without this guard a NaN field would slip through as "equal". +bool _nonFiniteMismatch(double a, double b) { + if (a.isFinite && b.isFinite) return false; + if (a.isNaN && b.isNaN) return false; + return a != b; +} + +/// Null when [got] matches [want] per the spec tolerances, else why not. +String? _diff(Metric got, Metric want) { + if (got.present != want.present) { + return 'present ${got.present} vs ${want.present}'; + } + if (got.tier != want.tier || + got.inputs_used.join(',') != want.inputs_used.join(',') || + got.note != want.note) { + return 'metadata ${got.tier}/${got.inputs_used}/${got.note} vs ' + '${want.tier}/${want.inputs_used}/${want.note}'; + } + if (_nonFiniteMismatch(got.confidence, want.confidence) || + (got.confidence - want.confidence).abs() > 1e-7) { + return 'confidence ${got.confidence} vs ${want.confidence}'; + } + final g = got.value, w = want.value; + if (g == null || w == null) return null; + for (final (name, a, b) in [ + ('mesor', g.mesor, w.mesor), + ('amplitude', g.amplitude, w.amplitude), + ('acrophaseRad', g.acrophaseRad, w.acrophaseRad), + ('acrophaseHours', g.acrophaseHours, w.acrophaseHours), + ('r2', g.r2, w.r2), + ('r2Adj', g.r2Adj, w.r2Adj), + ]) { + if (_nonFiniteMismatch(a, b)) return '$name non-finite mismatch $a vs $b'; + } + bool rel(double a, double b, [double floor = 0]) => + (a - b).abs() <= math.max(floor, 1e-9 * math.max(a.abs(), b.abs())); + // A mesor near zero (data centred on 0) has no relative accuracy either: + // its rounding scales with the amplitude, so it gets the same data-scale + // floor. + final scale = math.max(1.0, math.max(w.mesor.abs(), w.amplitude)); + if (!rel(g.mesor, w.mesor, 1e-12 * scale)) { + return 'mesor ${g.mesor} vs ${w.mesor}'; + } + // An amplitude that is pure rounding noise (constant data) has no relative + // accuracy and no phase: compare it against a floor and skip the phases. + final floor = 1e-12 * math.max(1, w.mesor.abs()); + if (!rel(g.amplitude, w.amplitude, floor)) { + return 'amplitude ${g.amplitude} vs ${w.amplitude}'; + } + final noPhase = w.amplitude < 1000 * floor && g.amplitude < 1000 * floor; + if (!noPhase && + _circ(g.acrophaseRad, w.acrophaseRad, _twoPi) > + 1e-9 * math.max(1, w.acrophaseRad.abs())) { + return 'acrophaseRad ${g.acrophaseRad} vs ${w.acrophaseRad}'; + } + if (g.periodHours != w.periodHours) return 'periodHours'; + if (!noPhase && + _circ(g.acrophaseHours, w.acrophaseHours, w.periodHours) > + 1e-9 * w.periodHours) { + return 'acrophaseHours ${g.acrophaseHours} vs ${w.acrophaseHours}'; + } + if ((g.r2 - w.r2).abs() > 1e-7) return 'r2 ${g.r2} vs ${w.r2}'; + if ((g.r2Adj - w.r2Adj).abs() > 1e-7) { + return 'r2Adj ${g.r2Adj} vs ${w.r2Adj}'; + } + return null; +} + +bool _eq(Metric a, Metric b) => _diff(a, b) == null; + +void _same(Metric got, Metric want, String tag) { + expect(_diff(got, want), isNull, reason: tag); +} + +Metric _want(TimedValues d, double period, int minPoints) => + cosinor(d.tHours, d.y, periodHours: period, minPoints: minPoints); + +void _allPaths(_Agg a, TimedValues d, List b, Metric want, + String tag) { + _same(runChunked(a, d, b), want, '$tag chunks $b'); + _same(runChunked(a, d, b, copy: true), want, '$tag copies $b'); + _same(runChunkedRight(a, d, b), want, '$tag right $b'); + _same(runSliced(a, d, b), want, '$tag sliced $b'); +} + +/// [n] samples: y = mesor + amp cos(w (t - peak)) + noise. Times are a regular +/// grid of [step] hours from [t0] with jitter [jitter] (hours). +TimedValues _series(Rng r, int n, + {double period = 24, + double mesor = 33, + double amp = .3, + double noise = 0, + double peak = 15.5, + double t0 = 0, + double step = .25, + double jitter = 0}) { + final w = _twoPi / period; + final t = [], y = []; + for (var i = 0; i < n; i++) { + final ti = t0 + i * step + (jitter == 0 ? 0 : (r.nextDouble() - .5) * jitter); + t.add(ti); + y.add(mesor + + amp * math.cos(w * (ti - peak)) + + (noise == 0 ? 0 : (r.nextDouble() - .5) * 2 * noise)); + } + return _tv(t, y); +} + +void _parity(String tag, TimedValues d, + {double period = 24, int minPoints = cosinorMinPoints, int seed = 1}) { + final want = _want(d, period, minPoints); + final a = cosinorAggregator(periodHours: period, minPoints: minPoints); + _same(a.run(d), want, '$tag whole'); + final r = Rng(seed); + for (var k = 0; k < 3; k++) { + _allPaths(a, d, randomBounds(r, d.length), want, tag); + } + _allPaths(a, d, singletonBounds(d.length), want, '$tag singletons'); +} + +TimedValues _lawSeries(Rng r, int i) { + final n = r.intIn(0, 40); + return _series(r, n, + amp: [.3, 2.0][i % 2], + noise: [.02, .3, 1e-6][i % 3], + // Cover about one to two periods so the design is well conditioned. + step: 24 * (1 + r.nextDouble()) / math.max(n, 1), + jitter: .3, + peak: r.nextDouble() * 24); +} + +void main() { + test('tiny phase coverage returns an absent metric, not a fabricated fit', () { + // Distinct finite times, clustered around phase pi/4. The two centered + // design columns become nearly collinear despite enough observations. + final t = [for (var i = 0; i < 16; i++) 3.0 + i * 1e-6]; + final y = [for (var i = 0; i < t.length; i++) 60.0 + i * 0.01]; + final batch = _tv(t, y); + final agg = cosinorAggregator(); + final left = agg.addSlice(agg.empty(), batch, 0, 8); + final right = agg.addSlice(agg.empty(), batch, 8, batch.length); + for (final result in [agg.run(batch), + agg.finish(agg.combine(left, right)), cosinor(t, y)]) { + expect(result.value, isNull); + expect(result.present, isFalse); + expect(result.confidence, 0); + expect(result.tier, Tier.high); + expect(result.inputs_used, ['signal_timeseries']); + expect(result.note, + 'singular cosinor design (insufficient phase coverage)'); + } + }); + + group('TimedValues batch', () { + final d = _tv([0, 1, 2, 3, 4], [10, 11, 12, 13, 14]); + test('length, copyRange, join', () { + expect(d.length, 5); + final c = d.copyRange(1, 4); + expect(c.tHours, [1.0, 2.0, 3.0]); + expect(c.y, [11.0, 12.0, 13.0]); + final j = d.copyRange(0, 2).join(d.copyRange(2, 5)); + expect(j.tHours, d.tHours); + expect(j.y, d.y); + expect(d.copyRange(2, 2).length, 0); + }); + test('copies are independent of the source and of the join inputs', () { + final c = d.copyRange(0, 3); + d.y[0] = -1; + d.tHours[1] = -1; + expect(c.y[0], 10.0); + expect(c.tHours[1], 1.0); + final left = _tv([0, 1], [1, 2]), right = _tv([2], [3]); + final j = left.join(right); + left.y[0] = 99; + right.tHours[0] = 99; + expect(j.y, [1.0, 2.0, 3.0]); + expect(j.tHours, [0.0, 1.0, 2.0]); + }); + }); + + group('absent cases', () { + final good = _series(Rng(1), 48); + + test('too few points, default minPoints 8', () { + for (var n = 0; n <= 10; n++) { + final d = _series(Rng(n), n, noise: .1, step: .7); + final want = _want(d, 24, cosinorMinPoints); + expect(want.present, n >= 8, reason: 'n=$n'); + _parity('n=$n', d, seed: n); + } + final m = cosinorAggregator().run(_series(Rng(1), 7, noise: .1)); + expect(m.present, isFalse); + expect(m.note, 'too few points for cosinor'); + expect(m.confidence, 0); + }); + test('minPoints is honoured: larger, and smaller than 4 is floored to 4', + () { + for (final mp in [2, 4, 5, 12, 40]) { + for (var n = 0; n <= 14; n++) { + final d = _series(Rng(n), n, noise: .1, step: .7); + _parity('mp=$mp n=$n', d, minPoints: mp, seed: n + mp); + } + } + expect(cosinorAggregator(minPoints: 12) + .run(_series(Rng(1), 11, noise: .1, step: .7)).present, isFalse); + expect(cosinorAggregator(minPoints: 2) + .run(_series(Rng(1), 3, noise: .1, step: 5)).present, isFalse); + expect(cosinorAggregator(minPoints: 2) + .run(_series(Rng(1), 4, noise: .1, step: 5)).present, isTrue); + }); + test('periodHours <= 0 is absent, and building the aggregator is fine', () { + for (final p in [0.0, -24.0, -1e-9]) { + final want = _want(good, p, cosinorMinPoints); + expect(want.present, isFalse); + final a = cosinorAggregator(periodHours: p); + _same(a.run(good), want, 'period $p'); + _same(a.run(good.copyRange(0, 3)), _want(good.copyRange(0, 3), p, 8), + 'period $p short'); + _allPaths(a, good, [0, 10, 10, 30, 48], want, 'period $p'); + } + }); + test('empty batch and empty state are absent', () { + final a = cosinorAggregator(); + final want = _want(_tv([], []), 24, cosinorMinPoints); + _same(a.run(_tv([], [])), want, 'empty'); + _same(a.finish(a.empty()), want, 'empty state'); + }); + test('singular design: every sample at the same phase', () { + // Whole periods apart, so cos/sin are the same at every row. + final d = _tv([for (var i = 0; i < 20; i++) 24.0 * i], + [for (var i = 0; i < 20; i++) 33 + .1 * (i % 3)]); + final want = _want(d, 24, 8); + expect(want.present, isFalse); + expect(want.note, contains('singular')); + _parity('same phase (multiples of the period)', d); + final same = _tv(List.filled(12, 7.5), [for (var i = 0; i < 12; i++) 30.0 + i]); + expect(_want(same, 24, 8).present, isFalse); + _parity('all at one time', same); + }); + test('singular note and metadata match', () { + final d = _tv(List.filled(10, 3.0), [for (var i = 0; i < 10; i++) i * 1.0]); + final m = cosinorAggregator().run(d); + expect(m.present, isFalse); + expect(m.note, 'singular cosinor design (insufficient phase coverage)'); + expect(m.confidence, 0); + }); + }); + + group('values', () { + test('noise-free 33 +- 0.3 recovers the parameters', () { + final d = _series(Rng(1), 96, peak: 15.5); + final m = cosinorAggregator().run(d); + final v = m.value!; + expect(v.mesor, closeTo(33, 1e-9)); + expect(v.amplitude, closeTo(.3, 1e-9)); + expect(v.acrophaseHours, closeTo(15.5, 1e-7)); + expect(v.r2, closeTo(1, 1e-7)); + expect(v.r2Adj, closeTo(1, 1e-7)); + expect(v.periodHours, 24.0); + expect(m.confidence, closeTo(.95, 1e-7)); + }); + test('hand-worked: y = 10 + 2 cos(2 pi t / 24) on a 3-hour grid', () { + final t = [for (var i = 0; i < 16; i++) 3.0 * i]; + final y = [for (final x in t) 10 + 2 * math.cos(_twoPi * x / 24)]; + final v = cosinorAggregator().run(_tv(t, y)).value!; + expect(v.mesor, closeTo(10, 1e-9)); + expect(v.amplitude, closeTo(2, 1e-9)); + expect(v.acrophaseRad, closeTo(0, 1e-9)); + expect(_circ(v.acrophaseHours, 0, 24), lessThan(1e-7)); + }); + test('period 12 and a custom period are carried through', () { + final d = _series(Rng(2), 80, period: 12, peak: 4, step: .3); + final v = cosinorAggregator(periodHours: 12).run(d).value!; + expect(v.periodHours, 12.0); + expect(v.acrophaseHours, closeTo(4, 1e-6)); + expect(v.amplitude, closeTo(.3, 1e-9)); + }); + }); + + group('parity on named series', () { + final named = { + 'mesor 33 amp 0.3 exact, 24 h': (_series(Rng(1), 96), 24), + 'mesor 33 amp 0.3 exact, 12 h': (_series(Rng(2), 96, period: 12), 12), + 'mesor 33 amp 0.3, noise 0.01': ( + _series(Rng(3), 200, noise: .01, jitter: .1), + 24 + ), + 'mesor 33 amp 0.3, noise 0.3 (noisy)': ( + _series(Rng(4), 200, noise: .3, jitter: .1), + 24 + ), + 'mesor 33 amp 0.3, noise 2 (almost nothing)': ( + _series(Rng(5), 120, noise: 2), + 24 + ), + 'mesor 200 amp 0.01 exact': ( + _series(Rng(6), 150, mesor: 200, amp: .01), + 24 + ), + 'mesor 1000 amp 0.5 noisy': ( + _series(Rng(7), 150, mesor: 1000, amp: .5, noise: .2, jitter: .2), + 24 + ), + 'zero amplitude, tiny noise': ( + _series(Rng(8), 100, amp: 0, noise: .05), + 24 + ), + 'near-perfect, amp 20': (_series(Rng(9), 100, mesor: 60, amp: 20), 24), + 'near-perfect with 1e-6 noise': ( + _series(Rng(10), 100, noise: 1e-6, jitter: .05), + 24 + ), + 'hours since epoch (large t)': ( + _series(Rng(11), 100, t0: 480000, noise: .02), + 24 + ), + 'negative t': (_series(Rng(12), 100, t0: -500, noise: .02), 24), + 'exactly 8 points': (_series(Rng(13), 8, step: 3, noise: .1), 24), + 'peak at 0 h': (_series(Rng(14), 96, peak: 0, noise: .02), 24), + 'peak at 23.9 h': (_series(Rng(15), 96, peak: 23.9, noise: .02), 24), + 'period 12, noisy': ( + _series(Rng(16), 120, period: 12, noise: .2, amp: .6), + 12 + ), + 'period 24 data fitted at 12': ( + _series(Rng(17), 120, noise: .05, amp: 1), + 12 + ), + 'period 8': (_series(Rng(18), 90, period: 8, noise: .05, step: .5), 8), + 'irregular times over 3 days': ( + _series(Rng(19), 60, step: 1.2, jitter: 1, noise: .05, amp: 1), + 24 + ), + }; + var seed = 0; + named.forEach((name, c) { + final s = ++seed; + test(name, () { + final want = _want(c.$1, c.$2, cosinorMinPoints); + expect(want.present, isTrue, reason: 'fixture should be fittable'); + _parity(name, c.$1, period: c.$2, seed: s); + }); + }); + + test('the clamps agree: ssRes rounding must not leave r2 outside 0..1', () { + final d = _series(Rng(30), 200, mesor: 500, amp: .05); + _parity('r2 clamp', d); + final v = cosinorAggregator().run(d).value!; + expect(v.r2, inInclusiveRange(0, 1)); + expect(v.r2Adj, inInclusiveRange(0, 1)); + }); + test('constant y: ssTot is 0, r2 is 0 as in cosinor()', () { + final d = _series(Rng(31), 40, amp: 0, noise: 0); + final want = _want(d, 24, 8); + expect(want.present, isTrue); + expect(want.value!.r2, 0.0); + _parity('constant', d); + }); + }); + + group('random parity (seeded)', () { + test('random shape, period, size; whole, chunked, sliced', () { + final r = Rng(0xC051); + var checked = 0; + for (var t = 0; t < 60; t++) { + final period = [24.0, 12.0, 24.0, 8.0, 36.0][r.intIn(0, 4)]; + final n = r.nextBool(.15) ? r.intIn(0, 9) : r.intIn(10, 220); + final mesor = [0.0, 33.0, 33.0, 120.0, -15.0][r.intIn(0, 4)]; + final amp = r.nextBool(.15) ? 0.0 : [.3, 1.0, 12.0][r.intIn(0, 2)]; + // Noise-free flat data and 1e-6 noise on a flat series have no + // meaningful phase; keep them for series that have an amplitude. + final noise = amp == 0 + ? [.02, .3, 1.5][r.intIn(0, 2)] + : [0.0, 1e-6, .02, .3, 1.5][r.intIn(0, 4)]; + final d = _series(r, n, + period: period, + mesor: mesor, + amp: amp, + noise: noise, + peak: r.nextDouble() * period, + t0: r.nextBool(.2) ? 1e5 * r.nextDouble() : 0, + // 0.6 to 3 periods of coverage, so the design is well conditioned. + step: period * (.6 + r.nextDouble() * 2.4) / math.max(n, 1), + jitter: r.nextBool() ? 0 : .5); + final mp = r.nextBool(.7) ? cosinorMinPoints : r.intIn(2, 30); + _parity('t$t n=$n P=$period mesor=$mesor amp=$amp noise=$noise', d, + period: period, minPoints: mp, seed: t); + checked++; + } + expect(checked, 60); + }); + }); + + test('persistable: a mid-stream checkpoint resumes to the one-pass result', () { + final b = _lawSeries(Rng(0xC0DE), 3); + expectCheckpointResumes(cosinorAggregator(), b, equals: _eq); + }); + + group('lawful', () { + final r = Rng(0xC052); + final cases = [ + _tv([], []), + _tv([1], [2]), + _tv([1, 2], [2, 3]), + for (var i = 0; i < 12; i++) + _lawSeries(r, i), + ]; + test('default period', () { + expectLawful(cosinorAggregator(), cases, equals: _eq); + }); + test('period 12, minPoints 5', () { + expectLawful(cosinorAggregator(periodHours: 12, minPoints: 5), cases, + equals: _eq); + }); + test('periodHours 0 (always absent)', () { + expectLawful(cosinorAggregator(periodHours: 0), cases, equals: _eq); + }); + }); + + group('_diff comparator', () { + final base = cosinor(_series(Rng(1), 48, noise: .05).tHours, + _series(Rng(1), 48, noise: .05).y); + Metric with_( + {double? mesor, + double? amplitude, + double? rad, + double? hours, + double? r2, + double? r2Adj, + double? confidence}) => + Metric( + value: CosinorFit( + mesor: mesor ?? base.value!.mesor, + amplitude: amplitude ?? base.value!.amplitude, + acrophaseRad: rad ?? base.value!.acrophaseRad, + acrophaseHours: hours ?? base.value!.acrophaseHours, + periodHours: base.value!.periodHours, + r2: r2 ?? base.value!.r2, + r2Adj: r2Adj ?? base.value!.r2Adj, + ), + confidence: confidence ?? base.confidence, + tier: base.tier, + inputs_used: base.inputs_used, + note: base.note, + ); + + test('identical metrics are equal', () { + expect(_diff(with_(), base), isNull); + }); + test('r2 replaced by NaN is reported', () { + expect(_diff(with_(r2: double.nan), base), isNotNull); + expect(_diff(base, with_(r2: double.nan)), isNotNull); + }); + test('every field turned NaN or infinite is reported', () { + for (final bad in [double.nan, double.infinity]) { + expect(_diff(with_(r2Adj: bad), base), isNotNull, reason: 'r2Adj $bad'); + expect(_diff(with_(confidence: bad), base), isNotNull, + reason: 'confidence $bad'); + expect(_diff(with_(rad: bad), base), isNotNull, reason: 'rad $bad'); + expect(_diff(with_(hours: bad), base), isNotNull, reason: 'hours $bad'); + expect(_diff(with_(mesor: bad), base), isNotNull, reason: 'mesor $bad'); + expect(_diff(with_(amplitude: bad), base), isNotNull, + reason: 'amplitude $bad'); + } + }); + }); + + group('narrow phase coverage', () { + // 16 samples in 0.15 h of a 24 h period: the (c, s) design is almost + // collinear. cosinor()'s own 3x3 solve is not trustworthy here, so these + // tests make NO parity claim against cosinor(); they only demand that the + // aggregator agrees with itself across chunking. + TimedValues narrow(int n, {double step = .01}) { + final w = _twoPi / 24; + final t = [for (var i = 0; i < n; i++) step * i]; + return _tv(t, [ + for (final ti in t) 33 + .3 * math.cos(w * ti) + .1 * math.sin(w * ti) + ]); + } + + bool relClose(double a, double b, double tol) => + (a - b).abs() <= tol * math.max(a.abs(), b.abs()); + + test('whole vs split at 8 agree to 1e-9 relative (review case)', () { + final d = narrow(16); + final a = cosinorAggregator(); + final whole = a.run(d); + for (final split in [ + runChunked(a, d, [0, 8, 16]), + runChunked(a, d, [0, 8, 16], copy: true), + runChunkedRight(a, d, [0, 8, 16]), + ]) { + expect(split.present, whole.present, + reason: 'present ${split.present} vs ${whole.present}'); + if (!whole.present) continue; + final g = split.value!, w = whole.value!; + expect(relClose(g.amplitude, w.amplitude, 1e-9), isTrue, + reason: 'amplitude ${g.amplitude} vs ${w.amplitude}'); + expect(relClose(g.mesor, w.mesor, 1e-9), isTrue, + reason: 'mesor ${g.mesor} vs ${w.mesor}'); + } + }); + + test('every single split point of the review case agrees', () { + final d = narrow(16); + final a = cosinorAggregator(); + final whole = a.run(d); + for (var k = 1; k < 16; k++) { + final split = runChunked(a, d, [0, k, 16]); + expect(split.present, whole.present, reason: 'split $k presence'); + if (!whole.present) continue; + expect( + relClose(split.value!.amplitude, whole.value!.amplitude, 1e-9), + isTrue, + reason: 'split $k amplitude ' + '${split.value!.amplitude} vs ${whole.value!.amplitude}'); + expect(relClose(split.value!.mesor, whole.value!.mesor, 1e-9), isTrue, + reason: 'split $k mesor'); + } + }); + + test('narrow-coverage cases are lawful', () { + expectLawful(cosinorAggregator(), [ + _tv([], []), + narrow(16), + narrow(24), + narrow(16, step: .02), + narrow(20, step: .005), + ], equals: _eq); + }); + }); +} diff --git a/test/aggregate/ctl_atl_aggregator_test.dart b/test/aggregate/ctl_atl_aggregator_test.dart new file mode 100644 index 0000000..3626e7a --- /dev/null +++ b/test/aggregate/ctl_atl_aggregator_test.dart @@ -0,0 +1,289 @@ +// ctlAtlTsbAggregator must agree with ctlAtlTsb for the same daily series under +// any slicing or chunking, and obey the aggregator laws. +// +// Contract used here: +// * ctlAtlTsbAggregator({ctlDays = 42, atlDays = 7, minDays = ctlAtlMinDays, +// primeDays = 7}) is an Aggregator<_, DoubleColumn, Metric>; +// the column is daily TRIMP, oldest first. Exported from +// package:openstrap_analytics/aggregate.dart. +// * Fewer rows than minDays: absent, with the reference's note (the +// needBaselineNote), tier and inputs_used. +// * Otherwise ctl / atl / tsb within relTol 1e-9 (absTol 1e-9); confidence, +// tier, inputs_used and note (which interpolates the prime count +// min(max(primeDays, 1), rows)) are exactly equal. +// * Rows are all real days (a rest day is 0, not NaN); NaN input is out of +// scope. minDays < 1 is out of scope too (the reference throws on an empty +// series with minDays 0). +// * primeDays 0 behaves as 1 (reference clamps), primeDays > rows as rows. +// * Constructing the aggregator never throws for the parameters below. + +import 'dart:typed_data'; + +import 'package:openstrap_analytics/aggregate.dart'; +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +import '../support/checkpoint_support.dart'; +import '../support/aggregator_laws.dart'; +import '../support/property.dart' show Rng; + +typedef _Agg = Aggregator>; + +List _fields(Metric m) => [ + m.present, + m.value?.ctl, + m.value?.atl, + m.value?.tsb, + m.tier, + m.inputs_used.join(','), + m.note, + ]; + +final _eq = fieldsEqual>(_fields, relTol: 1e-9, absTol: 1e-9); + +/// _same, as a predicate: the fields within tolerance, confidence exact. +bool _eqExact(Metric got, Metric want) => + _eq(got, want) && got.confidence == want.confidence; + +void _same(Metric got, Metric want, String tag) { + expect(_eq(got, want), isTrue, + reason: '$tag\n got ${_fields(got)}\nwant ${_fields(want)}'); + expect(got.confidence, want.confidence, reason: '$tag confidence'); + expect(got.note, want.note, reason: '$tag note'); +} + +class _P { + const _P(this.name, {this.ctl = 42, this.atl = 7, this.min = ctlAtlMinDays, this.prime = 7}); + final String name; + final double ctl, atl; + final int min, prime; + _Agg get agg => ctlAtlTsbAggregator( + ctlDays: ctl, atlDays: atl, minDays: min, primeDays: prime); + Metric want(DoubleColumn c) => ctlAtlTsb(c.values.toList(), + ctlDays: ctl, atlDays: atl, minDays: min, primeDays: prime); +} + +const _params = [ + _P('default'), + _P('custom time constants', ctl: 30, atl: 5, min: 10, prime: 5), + _P('minDays < primeDays', min: 3, prime: 10), + _P('minDays 1, primeDays 7', min: 1), + _P('primeDays 1', prime: 1), + _P('primeDays 0 acts as 1', prime: 0), + _P('primeDays 14 = minDays', prime: 14), + _P('primeDays > minDays', min: 5, prime: 30), +]; + +void _allPaths(_Agg a, DoubleColumn c, List b, Metric want, + String tag) { + _same(runChunked(a, c, b), want, '$tag chunks $b'); + _same(runChunked(a, c, b, copy: true), want, '$tag copies $b'); + _same(runChunkedRight(a, c, b), want, '$tag right $b'); + _same(runSliced(a, c, b), want, '$tag sliced $b'); +} + +/// A daily-TRIMP-like series: rest days (0), spikes, ordinary workouts. +DoubleColumn _series(Rng r, int n) => DoubleColumn(Float64List.fromList([ + for (var i = 0; i < n; i++) + r.nextBool(.25) + ? 0.0 + : r.nextBool(.05) + ? 300 + r.nextDouble() * 9000 + : r.nextDouble() * 250 + ])); + +void main() { + group('absent below minDays', () { + test('identical to the reference for every short length', () { + for (final p in _params) { + final cols = [ + for (var n = 0; n < p.min && n < 16; n++) + col([for (var i = 0; i < n; i++) 10.0 + i]) + ]; + for (final c in cols) { + expect(p.want(c).present, isFalse, reason: '${p.name} n=${c.length}'); + } + expectConversion>( + (_) => p.agg, cols, p.want, + equals: _eqExact); + } + }); + + test('empty state is absent for minDays >= 1', () { + final a = ctlAtlTsbAggregator(); + _same(a.finish(a.empty()), ctlAtlTsb([]), 'empty state'); + _same(a.run(col([])), ctlAtlTsb([]), 'empty column'); + }); + + test('length minDays - 1 absent, minDays present (default)', () { + final a = ctlAtlTsbAggregator(); + expect(a.run(_iotaCol(13)).present, isFalse); + expect(a.run(_iotaCol(14)).present, isTrue); + expect(a.run(_iotaCol(14)).note, + contains('primed from the first 7 observed days')); + }); + }); + + group('values', () { + test('constant series: ctl = atl = the constant, tsb 0', () { + final c = col(List.filled(60, 120.0)); + final got = ctlAtlTsbAggregator().run(c); + expect(got.value!.ctl, closeTo(120, 1e-9)); + expect(got.value!.atl, closeTo(120, 1e-9)); + expect(got.value!.tsb, closeTo(0, 1e-9)); + _same(got, ctlAtlTsb(c.values.toList()), 'constant'); + }); + + test('exactly primeDays == rows: the value is the mean', () { + final p = const _P('p', min: 3, prime: 4); + final c = col([10, 20, 30, 40]); + final got = p.agg.run(c); + expect(got.value!.ctl, closeTo(25, 1e-12)); + expect(got.value!.atl, closeTo(25, 1e-12)); + _same(got, p.want(c), 'mean'); + }); + + test('fewer rows than primeDays: prime is the row count', () { + final p = const _P('p', min: 3, prime: 10); + final c = col([4, 8, 12, 16, 20]); + final got = p.agg.run(c); + expect(got.note, contains('first 5 observed days')); + _same(got, p.want(c), 'short prime'); + }); + + test('primeDays 0 and 1 seed from the first day', () { + for (final prime in [0, 1]) { + final p = _P('p', min: 2, prime: prime); + final c = col([100, 0, 0, 0, 50, 0, 0, 0, 0, 0]); + _same(p.agg.run(c), p.want(c), 'prime $prime'); + expect(p.agg.run(c).note, contains('first 1 observed days')); + } + }); + + test('rest days decay the load: all-zero tail', () { + final c = col([200, 200, 200, 200, 200, 200, 200, ...List.filled(40, 0.0)]); + final got = ctlAtlTsbAggregator().run(c); + _same(got, ctlAtlTsb(c.values.toList()), 'decay'); + expect(got.value!.atl, lessThan(got.value!.ctl)); + expect(got.value!.tsb, greaterThan(0)); + }); + + test('a single spike on the last day', () { + final xs = [...List.filled(30, 50.0), 5000.0]; + final c = col(xs); + _same(ctlAtlTsbAggregator().run(c), ctlAtlTsb(xs), 'spike'); + }); + + test('confidence is exact: rows/42 clamped to 0.3..0.85', () { + final a = ctlAtlTsbAggregator(); + expect(a.run(_iotaCol(14)).confidence, ctlAtlTsb(_iotaCol(14).values.toList()).confidence); + expect(a.run(_iotaCol(30)).confidence, 30 / 42.0); + expect(a.run(_iotaCol(300)).confidence, 0.85); + }); + }); + + group('random parity (seeded)', () { + for (final p in _params) { + test(p.name, () { + final r = Rng(0xC71 + p.name.length * 31 + p.min * 7 + p.prime); + final cols = []; + for (var t = 0; t < 30; t++) { + final n = r.nextBool(.2) ? r.intIn(0, 20) : r.intIn(0, 200); + cols.add(_series(r, n)); + } + expectConversion>( + (_) => p.agg, cols, p.want, + equals: _eqExact); + }); + } + + test('constant series of random lengths and values', () { + final r = Rng(0xC72); + final cols = []; + for (var t = 0; t < 30; t++) { + final n = r.intIn(0, 120); + final v = r.nextBool(.3) ? 0.0 : r.nextDouble() * 400; + cols.add(col(List.filled(n, v))); + } + expectConversion>( + (_) => ctlAtlTsbAggregator(), cols, (c) => ctlAtlTsb(c.values.toList()), + equals: _eqExact); + }); + + test('chunks straddle the prime boundary (primeDays 7)', () { + final r = Rng(0xC73); + final a = ctlAtlTsbAggregator(); + final c = _series(r, 40); + final want = ctlAtlTsb(c.values.toList()); + for (final cut in [1, 2, 6, 7, 8, 13, 14, 15]) { + _allPaths(a, c, [0, cut, 40], want, 'cut $cut'); + _allPaths(a, c, [0, 3, cut > 3 ? cut : 3, 40], want, 'cuts 3,$cut'); + } + }); + }); + + test('persistable: a mid-stream checkpoint resumes to the one-pass result', () { + final c = _series(Rng(0xC0DE), 60); + for (final p in [_params[0], _params[2]]) { + expectCheckpointResumes(p.agg, c, equals: _eq); + } + }); + + group('lawful', () { + final cols = [ + ...randomColumns(0xC74, nanP: 0, maxLen: 60, value: (r) => r.nextDouble() * 300), + for (final n in [13, 14, 15, 21, 22]) _iotaCol(n), + ]; + for (final p in [_params[0], _params[2], _params[5]]) { + test(p.name, () => expectLawful(p.agg, cols, equals: _eq)); + } + }); + + group('composition', () { + test('both(...) and map still work around it', () { + final a = both(ctlAtlTsbAggregator(), Count()) + .map((r) => (r.$1.present, r.$2)); + expect(a.run(_iotaCol(20)), (true, 20)); + expect(a.run(_iotaCol(5)), (false, 5)); + }); + }); + + group('checkpoint relationships (day count against the primed prefix)', () { + final agg = ctlAtlTsbAggregator(); // primeDays 7 + // State: [days, {rows: , tail: [..]}]; rows hold min(days, 7). + Object? set(Object? s, int days) { + (s as List)[0] = days; + return s; + } + + test('a day count below the prefix rows held is refused', () { + expectDamagedCheckpointRefused(agg, _iotaCol(10), (s) => set(s, 5)); + expectDamagedCheckpointRefused(agg, _iotaCol(10), (s) => set(s, 0)); + }); + test('a day count above what the prefix rows allow is refused', () { + expectDamagedCheckpointRefused(agg, _iotaCol(3), (s) => set(s, 9)); + expectDamagedCheckpointRefused(agg, _iotaCol(3), (s) => set(s, 4)); + }); + test('days with no prefix rows are refused', () { + expectDamagedCheckpointRefused(agg, _iotaCol(10), (s) { + (((s as List)[1]) as Map)['rows'] = null; + return s; + }); + }); + test('prefix rows with no days are refused', () { + expectDamagedCheckpointRefused(agg, _iotaCol(3), (s) => set(s, 0)); + }); + test('consistent checkpoints still restore', () { + for (final n in [0, 1, 6, 7, 8, 20]) { + final b = _iotaCol(n); + expect( + decodeCheckpoint(agg, checkpointOf(agg, b), + kind: 'test', version: 1), + isNotNull, + reason: 'n=$n'); + } + }); + }); +} + +DoubleColumn _iotaCol(int n) => col([for (var i = 0; i < n; i++) 5.0 + i * 3]); diff --git a/test/aggregate/ewma_prefix_test.dart b/test/aggregate/ewma_prefix_test.dart new file mode 100644 index 0000000..9fac2a7 --- /dev/null +++ b/test/aggregate/ewma_prefix_test.dart @@ -0,0 +1,286 @@ +// Affine, Ewma and Prefix (spec: research/r7-ewma.md). +// +// Decisions the implementation follows: +// * `const Affine(double scale, double offset)`, positional; identity is +// `Affine(1, 0)`; `apply(x) = scale * x + offset`; +// `a.then(b)` = b after a = (b.scale*a.scale, b.scale*a.offset + b.offset). +// * `Ewma(col, lambda)` is an Aggregator<_, B, Affine>, positional args. +// Its Affine maps the running value c to the value after all rows: +// one row x is c -> (1 - lambda) * c + lambda * x. NaN rows are skipped +// (identity). No rows (or only NaN rows) finish as Affine(1, 0). +// * `Prefix(k, head, tail)` (type args inferred) is an +// Aggregator<_, B, (R1, R2)>. Head sees rows [0, k) of the whole input via +// head.run on stored rows (so head.combine is never called), tail sees rows +// [k, n) in order. k = 0 gives head an empty batch. The stored prefix is a +// copy: mutating the source column after folding changes nothing. + +import 'package:openstrap_analytics/aggregate.dart'; +import 'package:test/test.dart'; + +import '../support/aggregator_laws.dart'; +import '../support/property.dart' show Rng; + +const nan = double.nan; + +List _aff(Affine a) => [a.scale, a.offset]; +final _affEq = fieldsEqual(_aff, relTol: 1e-12, absTol: 1e-15); + + +Ewma _ewma(double lambda) => + Ewma((b) => b.values, lambda); + +/// Reference: apply x_i one by one to the running value, skipping NaN. +double _loop(double seed, List xs, double lambda) { + var c = seed; + for (final x in xs) { + if (x.isNaN) continue; + c = c + lambda * (x - c); + } + return c; +} + +void _expectAffine(Affine got, Affine want, String tag) => expect( + _affEq(got, want), isTrue, + reason: '$tag got ${_aff(got)} want ${_aff(want)}'); + +typedef _Rows = List; + +/// Records every row it is shown, in order. Head variant never combines. +class _Rec extends Aggregator<_Rows, DoubleColumn, _Rows> { + const _Rec({this.combinable = true}); + final bool combinable; + @override + _Rows empty() => []; + @override + _Rows addSlice(_Rows s, DoubleColumn b, int start, int end) => + s..addAll([for (var i = start; i < end; i++) b.values[i]]); + @override + _Rows combine(_Rows l, _Rows r) { + if (!combinable) throw StateError('head.combine must not be called'); + return l..addAll(r); + } + + @override + _Rows finish(_Rows s) => List.of(s); +} + +DoubleColumn _iota(int n) => col([for (var i = 0; i < n; i++) i * 1.0]); + +void _is(Object? got, Object? want, {String? reason}) => + expect(got.toString(), want.toString(), reason: reason); + +bool _str(Object? a, Object? b) => a.toString() == b.toString(); + +void main() { + group('Affine', () { + test('apply, identity and then', () { + const a = Affine(2, 3); + const b = Affine(5, -1); + expect(a.apply(4), 11); + expect(const Affine(1, 0).apply(7.5), 7.5); + final ab = a.then(b); // b after a + expect(ab.scale, 10); + expect(ab.offset, 5 * 3 + -1); + for (final x in [-3.0, 0.0, 1.5, 100.0]) { + expect(ab.apply(x), b.apply(a.apply(x))); + } + _expectAffine(const Affine(1, 0).then(a), a, 'id.then(a)'); + _expectAffine(a.then(const Affine(1, 0)), a, 'a.then(id)'); + }); + + test('then is associative (seeded)', () { + final r = Rng(0xAF1); + Affine rnd() => Affine((r.nextDouble() - .5) * 4, (r.nextDouble() - .5) * 100); + for (var t = 0; t < 100; t++) { + final a = rnd(), b = rnd(), c = rnd(); + _expectAffine(a.then(b).then(c), a.then(b.then(c)), 't$t'); + } + }); + }); + + group('Ewma', () { + test('one row: (1 - lambda) c + lambda x', () { + final f = _ewma(0.25).run(col([8])); + expect(f.scale, closeTo(0.75, 1e-15)); + expect(f.offset, closeTo(2, 1e-15)); + expect(f.apply(4), closeTo(0.75 * 4 + 2, 1e-15)); + }); + + test('no rows, and only NaN rows, are the identity', () { + final e = _ewma(0.1); + _expectAffine(e.run(col([])), const Affine(1, 0), 'empty'); + _expectAffine(e.finish(e.empty()), const Affine(1, 0), 'empty state'); + _expectAffine(e.run(col([nan, nan, nan])), const Affine(1, 0), 'nans'); + }); + + test('values equal a direct loop (seeded, several lambdas)', () { + final r = Rng(0xE111); + for (final lambda in [0.5, 1 - 0.9, 0.02, 0.1331, 1.0, 0.0]) { + final e = _ewma(lambda); + for (var t = 0; t < 40; t++) { + final n = r.intIn(0, 30); + final xs = [ + for (var i = 0; i < n; i++) + r.nextBool(.2) ? nan : (r.nextDouble() - .3) * 500 + ]; + final seed = (r.nextDouble() - .5) * 300; + final got = e.run(col(xs)).apply(seed); + final want = _loop(seed, xs, lambda); + expect(got, closeTo(want, 1e-12 * (1 + want.abs())), + reason: 'lambda=$lambda t=$t'); + } + } + }); + + test('NaN rows are skipped, not decayed', () { + final e = _ewma(0.3); + final a = e.run(col([10, nan, 20, nan, nan, 5])); + final b = e.run(col([10, 20, 5])); + _expectAffine(a, b, 'nan vs removed'); + }); + + test('combine is left.then(right): chunking does not matter', () { + final r = Rng(0xE112); + final e = _ewma(0.07); + for (var t = 0; t < 60; t++) { + final n = r.intIn(0, 60); + final c = col([ + for (var i = 0; i < n; i++) r.nextBool(.1) ? nan : r.nextDouble() * 400 + ]); + final want = e.run(c); + final b = randomBounds(r, n); + _expectAffine(runChunked(e, c, b), want, 'chunked $b'); + _expectAffine(runChunked(e, c, b, copy: true), want, 'copy $b'); + _expectAffine(runChunkedRight(e, c, b), want, 'right $b'); + _expectAffine(runSliced(e, c, b), want, 'sliced $b'); + _expectAffine(runChunked(e, c, singletonBounds(n)), want, 'single'); + } + }); + + test('is lawful', () { + expectLawful(_ewma(0.05), randomColumns(0xE113, nanP: .2, maxLen: 40), + equals: _affEq); + }); + }); + + group('Prefix', () { + for (final k in [0, 1, 3, 7]) { + group('k = $k', () { + final p = Prefix(k, const _Rec(combinable: false), const _Rec()); + + (List, List) want(int n) => ( + [for (var i = 0; i < n && i < k; i++) i * 1.0], + [for (var i = k; i < n; i++) i * 1.0], + ); + + test('head sees exactly rows [0, k), tail rows [k, n), inputs of ' + 'every length', () { + for (var n = 0; n <= 20; n++) { + _is(p.run(_iota(n)), want(n), reason: 'n=$n'); + } + }); + + test('every single and double cut, all four evaluation paths', () { + for (final n in [0, 1, k, k + 1, 12]) { + final c = _iota(n); + for (var a = 0; a <= n; a++) { + for (var b = a; b <= n; b++) { + final bd = [0, a, b, n]; + _is(runChunked(p, c, bd), want(n), reason: 'chunk $bd'); + _is(runChunked(p, c, bd, copy: true), want(n), + reason: 'copy $bd'); + _is(runChunkedRight(p, c, bd), want(n), reason: 'right $bd'); + _is(runSliced(p, c, bd), want(n), reason: 'sliced $bd'); + } + } + } + }); + + test('random chunkings and slicings, empty chunks included', () { + final r = Rng(0x9F00 + k); + for (var t = 0; t < 80; t++) { + final n = r.intIn(0, 40); + final c = _iota(n); + final b = randomBounds(r, n); + _is(runChunked(p, c, b), want(n), reason: 'chunk $b n=$n'); + _is(runChunked(p, c, b, copy: true), want(n), reason: 'copy'); + _is(runChunkedRight(p, c, b), want(n), reason: 'right $b n=$n'); + _is(runSliced(p, c, b), want(n), reason: 'sliced $b n=$n'); + final s = singletonBounds(n); + _is(runChunked(p, c, s), want(n), reason: 'singletons'); + _is(runChunkedRight(p, c, s), want(n), reason: 'right singles'); + _is(runSliced(p, c, s), want(n), reason: 'sliced singles'); + } + }); + + test('is lawful with a recording head and tail', () { + expectLawful(p, [for (var n = 0; n <= 14; n++) _iota(n), _iota(40)], + equals: _str); + }); + + test('is lawful with Count and Sum', () { + final eq = fieldsEqual<(int, double)>((x) => [x.$1, x.$2], + relTol: 1e-12, absTol: 1e-9); + expectLawful( + Prefix(k, Count(), Sum((b) => b.values)), + randomColumns(0x9F10 + k, nanP: .2), + equals: eq); + final eq2 = fieldsEqual<(double, int)>((x) => [x.$1, x.$2], + relTol: 1e-12, absTol: 1e-9); + expectLawful( + Prefix(k, Sum((b) => b.values), Count()), + randomColumns(0x9F20 + k, nanP: .2), + equals: eq2); + }); + }); + } + + test('k = 3: seams of one-row chunks fill the prefix then the tail', () { + final p = Prefix(3, const _Rec(combinable: false), const _Rec()); + final c = _iota(6); + var s = p.fold(c, 0, 1); + s = p.combine(s, p.fold(c, 1, 2)); + _is(p.finish(s), ([0, 1], [])); + s = p.combine(s, p.fold(c, 2, 3)); + s = p.combine(s, p.fold(c, 3, 4)); + _is(p.finish(s), ([0, 1, 2], [3])); + // A right chunk bigger than the room left in the left prefix. + final r = p.combine(p.fold(c, 0, 2), p.fold(c, 2, 6)); + _is(p.finish(r), ([0, 1, 2], [3, 4, 5])); + // Empty left, empty right. + _is(p.finish(p.combine(p.empty(), p.fold(c, 0, 6))), ([0, 1, 2], [3, 4, 5])); + _is(p.finish(p.combine(p.fold(c, 0, 6), p.empty())), ([0, 1, 2], [3, 4, 5])); + }); + + test('the stored prefix is a copy of the rows', () { + final p = Prefix(3, const _Rec(combinable: false), const _Rec()); + final c = _iota(6); + final s = p.fold(c, 0, 6); + final s2 = p.fold(c, 0, 2); + for (var i = 0; i < 6; i++) { + c.values[i] = -99; + } + _is(p.finish(s), ([0, 1, 2], [3, 4, 5])); + _is(p.finish(s2), ([0, 1], [])); + }); + + test('composes with Ewma as the tail: mean seed, then decay', () { + final p = Prefix( + 4, Sum((b) => b.values), _ewma(0.2)); + final r = Rng(0x9F30); + for (var t = 0; t < 40; t++) { + final n = r.intIn(0, 30); + final xs = [for (var i = 0; i < n; i++) r.nextDouble() * 300]; + final c = col(xs); + final b = randomBounds(r, n); + final (sum, f) = runChunked(p, c, b); + final head = n < 4 ? n : 4; + expect(sum, closeTo(xs.take(head).fold(0.0, (a, x) => a + x), 1e-9)); + final seed = 50.0; + final got = f.apply(seed); + final want = _loop(seed, xs.skip(4).toList(), 0.2); + expect(got, closeTo(want, 1e-9 * (1 + want.abs())), reason: 't$t $b'); + } + }); + }); +} diff --git a/test/aggregate/group_by_test.dart b/test/aggregate/group_by_test.dart new file mode 100644 index 0000000..c6e064a --- /dev/null +++ b/test/aggregate/group_by_test.dart @@ -0,0 +1,259 @@ +// GroupBy: consecutive rows with equal key go to the inner aggregator as one +// slice; a key seen again later is combined into its existing state; finish +// maps every key through inner.finish. + +import 'dart:typed_data'; + +import 'package:openstrap_analytics/aggregate.dart'; +import 'package:openstrap_analytics/onehz.dart' show RunningMoments; +import 'package:test/test.dart'; + +import '../support/aggregator_laws.dart'; +import '../support/property.dart' show Rng; + +Float64List _v(DoubleColumn c) => c.values; +int _mod3(DoubleColumn c, int i) => c.values[i].toInt() % 3; +int _mod2(DoubleColumn c, int i) => c.values[i].toInt() % 2; +int _tens(DoubleColumn c, int i) => c.values[i] ~/ 10; + +/// Exact deep equality for (nested) maps; Map == is identity in Dart. +bool _mapsEqual(Object? a, Object? b) { + if (a is Map && b is Map) { + if (a.length != b.length) return false; + for (final k in a.keys) { + if (!b.containsKey(k) || !_mapsEqual(a[k], b[k])) return false; + } + return true; + } + return a == b; +} + +List _momentFields(Map m) { + final keys = m.keys.toList()..sort(); + return [ + keys.length, + for (final k in keys) ...[k, m[k]!.count, m[k]!.mean, m[k]!.sampleSd], + ]; +} + +/// Records every slice the inner is handed, as a list of its rows. +class _Slices extends Aggregator>, DoubleColumn, List>> { + const _Slices(); + @override + List> empty() => >[]; + @override + List> addSlice( + List> s, DoubleColumn b, int start, int end) { + if (end > start) s.add(b.values.sublist(start, end)); + return s; + } + + @override + List> combine(List> l, List> r) => + l..addAll(r); + @override + List> finish(List> s) => s; +} + +DoubleColumn _ints(List xs) => col([for (final x in xs) x * 1.0]); + +void main() { + final count3 = GroupBy(_mod3, Count()); + final sum3 = GroupBy(_mod3, Sum(_v)); + + group('values', () { + test('counts per key', () { + expect(count3.run(_ints([1, 2, 3, 4, 5, 6, 7])), {0: 2, 1: 3, 2: 2}); + }); + + test('sums per key', () { + final a = GroupBy(_mod2, Sum(_v)); + expect(a.run(_ints([1, 2, 3, 4, 5])), {1: 9.0, 0: 6.0}); + }); + + test('empty input gives an empty map', () { + expect(count3.run(col([])), isEmpty); + expect(count3.finish(count3.empty()), {}); + }); + + test('a key that never occurs is absent', () { + expect(count3.run(_ints([3, 6])).keys, [0]); + }); + + test('the key function sees the batch and a row index', () { + final rows = []; + final a = GroupBy( + (b, i) { + rows.add(i); + return 0; + }, + Count()); + a.run(_ints([5, 6, 7])); + expect(rows.toSet(), {0, 1, 2}); + }); + + test('works over a ListBatch', () { + final a = GroupBy, dynamic, int>( + (b, i) => b.rows[i].length, Count>()); + expect(a.runList(['a', 'bb', 'c', 'dd', 'e']), {1: 3, 2: 2}); + }); + }); + + group('runs of equal keys', () { + const slices = _Slices(); + final g = GroupBy>>(_tens, slices); + + test('consecutive equal keys reach the inner as one slice', () { + final r = g.run(col([10, 11, 12, 20, 21])); + expect(r, { + 1: [ + [10.0, 11.0, 12.0] + ], + 2: [ + [20.0, 21.0] + ], + }); + }); + + test('a key that reappears later is combined into its existing state, ' + 'in row order', () { + final r = g.run(col([10, 11, 20, 21, 22, 12, 30, 13, 14])); + expect(r.keys.toSet(), {1, 2, 3}); + expect(r[1]!.expand((s) => s).toList(), [10.0, 11.0, 12.0, 13.0, 14.0]); + expect(r[2], [ + [20.0, 21.0, 22.0] + ]); + expect(r[3], [ + [30.0] + ]); + // each run stayed together; the two runs of key 1 stay separate slices + expect(r[1], [ + [10.0, 11.0], + [12.0], + [13.0, 14.0] + ]); + }); + + test('alternating keys give one-row slices', () { + final r = g.run(col([10, 20, 11, 21, 12])); + expect(r[1], [ + [10.0], + [11.0], + [12.0] + ]); + expect(r[2], [ + [20.0], + [21.0] + ]); + }); + + test('a fold range groups only the rows inside it', () { + final c = col([10, 11, 20, 21, 12, 13]); + final r = g.finish(g.fold(c, 1, 5)); + expect(r[1], [ + [11.0], + [12.0] + ]); + expect(r[2], [ + [20.0, 21.0] + ]); + }); + + test('a run cut by a slice boundary arrives as two slices', () { + final c = col([10, 11, 12, 13]); + var s = g.empty(); + s = g.addSlice(s, c, 0, 2); + s = g.addSlice(s, c, 2, 4); + final r = g.finish(s); + expect(r[1]!.expand((x) => x).toList(), [10.0, 11.0, 12.0, 13.0]); + }); + }); + + group('combine', () { + test('disjoint keys are adopted', () { + final s = sum3.combine(sum3.fold(_ints([3, 6])), sum3.fold(_ints([1, 4]))); + expect(sum3.finish(s), {0: 9.0, 1: 5.0}); + }); + + test('overlapping keys merge with the inner combine', () { + final s = sum3.combine( + sum3.fold(_ints([1, 2, 3])), sum3.fold(_ints([4, 5, 6]))); + expect(sum3.finish(s), {0: 9.0, 1: 5.0, 2: 7.0}); + }); + + test('mixed shared and one-sided keys', () { + final s = sum3.combine(sum3.fold(_ints([1, 3])), sum3.fold(_ints([4, 5]))); + expect(sum3.finish(s), {0: 3.0, 1: 5.0, 2: 5.0}); + }); + + test('combining with empty on either side changes nothing', () { + final xs = _ints([1, 2, 3, 4]); + expect(sum3.finish(sum3.combine(sum3.empty(), sum3.fold(xs))), + sum3.run(xs)); + expect(sum3.finish(sum3.combine(sum3.fold(xs), sum3.empty())), + sum3.run(xs)); + }); + + test('order within a key is preserved across a seam', () { + final g = GroupBy>>( + _tens, const _Slices()); + final c = col([10, 20, 11, 12, 21, 13]); + final r = g.finish(g.combine(g.fold(c, 0, 3), g.fold(c, 3, 6))); + expect(r[1]!.expand((s) => s).toList(), [10.0, 11.0, 12.0, 13.0]); + expect(r[2]!.expand((s) => s).toList(), [20.0, 21.0]); + }); + }); + + group('laws', () { + final cols = randomColumns(0x6B01, nanP: 0, value: (r) => r.intIn(-20, 20) * 1.0); + + test('GroupBy(key, Count) is lawful with exact equality', () { + expectLawful(count3, cols, equals: _mapsEqual); + }); + + test('GroupBy(key, Sum) is lawful (integer-valued sums are exact)', () { + expectLawful(sum3, cols, equals: _mapsEqual); + }); + + test('nested GroupBy', () { + final nested = GroupBy>( + _mod2, GroupBy(_mod3, Sum(_v))); + expect(nested.run(_ints([1, 2, 3, 4, 5, 6])), { + 1: {1: 1.0, 0: 3.0, 2: 5.0}, + 0: {2: 2.0, 1: 4.0, 0: 6.0}, + }); + expect(nested.run(col([])), isEmpty); + expectLawful(nested, cols, equals: _mapsEqual); + }); + + test('tolerance-compared inner (Moments per key)', () { + final g = GroupBy( + _mod3, Moments(_v)); + final r = g.run(_ints([1, 2, 3, 4, 5, 10])); + expect(r.keys.toSet(), {0, 1, 2}); + expect(r[1]!.count, 3); + expect(r[1]!.mean, closeTo(5.0, 1e-12)); // (1+4+10)/3 + expect(r[1]!.sampleSd, closeTo(4.58257569495584, 1e-9)); + expectLawful(g, cols, + equals: fieldsEqual>(_momentFields, + relTol: 1e-9, absTol: 1e-9)); + }); + + test('random chunkings and slicings agree with the whole fold', () { + final r = Rng(0x6B02); + for (var t = 0; t < 40; t++) { + final c = _ints([for (var i = r.intIn(0, 60); i > 0; i--) r.intIn(0, 9)]); + final b = randomBounds(r, c.length); + final want = sum3.run(c); + expect(_mapsEqual(runChunked(sum3, c, b), want), isTrue, + reason: 'chunked $b'); + expect(_mapsEqual(runChunked(sum3, c, b, copy: true), want), isTrue, + reason: 'copies $b'); + expect(_mapsEqual(runChunkedRight(sum3, c, b), want), isTrue, + reason: 'right $b'); + expect(_mapsEqual(runSliced(sum3, c, b), want), isTrue, + reason: 'sliced $b'); + } + }); + }); +} diff --git a/test/aggregate/hrv_time_aggregator_test.dart b/test/aggregate/hrv_time_aggregator_test.dart new file mode 100644 index 0000000..a2e2cfc --- /dev/null +++ b/test/aggregate/hrv_time_aggregator_test.dart @@ -0,0 +1,553 @@ +// hrvTimeAggregator (assembled from lawful blocks) must agree with hrvTime() +// for any beat series and any chunking or slicing, and obey the laws. +// +// Contract used here: +// * gapAware (default): every beat carries its time; originMs is the first +// beat's time. run(beats) == hrvTime(nn, nnTimesMs: times).value. +// * gapAware false: times are ignored (they may even be NaN); run(beats) == +// hrvTime(nn).value, so sdann / sdnnIndex are null and seams are not +// skipped. +// * hrvTime returns an absent Metric below 2 beats. The aggregator then +// reports nBeats = the beat count and null for every other field. +// * originMs is needed as soon as a beat is binned; without it that is a +// StateError. Empty input with a null origin is fine. + +import 'package:openstrap_analytics/aggregate.dart'; +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +import '../support/checkpoint_support.dart'; +import '../support/aggregator_laws.dart'; +import '../support/hrv_beats.dart'; +import '../support/property.dart' show Rng; + +Aggregator _agg(Beats beats, {bool gapAware = true}) => + hrvTimeAggregator( + originMs: gapAware ? hrvOrigin(beats) : null, gapAware: gapAware); + +/// Every way of evaluating [bounds] against [want]: left-nested chunks (as +/// ranges and as copies), right-nested chunks, and one state sliced. +void _expectAllPaths(Aggregator agg, Beats beats, + List bounds, HrvTime want, String tag) { + expectHrvParity(runChunked(agg, beats, bounds), want, '$tag chunks $bounds'); + expectHrvParity(runChunked(agg, beats, bounds, copy: true), want, + '$tag copies $bounds'); + expectHrvParity( + runChunkedRight(agg, beats, bounds), want, '$tag right $bounds'); + expectHrvParity(runSliced(agg, beats, bounds), want, '$tag sliced $bounds'); +} + +/// [segments] runs of 12 beats alternating 860/800 ms, each starting at 860, +/// with a 5 s hole between runs. Inside a run the diffs alternate -60/+60 (diff +/// acf strongly negative). Run k ends with a -60 diff and run k+1 opens with a +/// -60 diff, so a lag pair formed across the hole would be (-60, -60), a +/// positive product that drags acf1 up. +Beats _alternatingRuns(int segments) { + final out = []; + var t = hrvT0; + for (var s = 0; s < segments; s++) { + for (var i = 0; i < 12; i++) { + final nn = i.isEven ? 860.0 : 800.0; + t += nn + (s > 0 && i == 0 ? 5000 : 0); + out.add((nnMs: nn, timeMs: t)); + } + } + return hrvBeats(out); +} + +void main() { + group('named series', () { + final named = { + 'no beats': hrvBeats([]), + 'one beat': hrvBeats([(nnMs: 800.0, timeMs: hrvT0 + 800)]), + 'two beats': hrvSeries(Rng(1), 2, 0), + 'three beats': hrvSeries(Rng(2), 3, 0), + '30 beats (29 diffs, acf1 absent)': hrvSeries(Rng(3), 30, 0, gapP: 0), + '31 beats (30 diffs, acf1 present)': hrvSeries(Rng(4), 31, 0, gapP: 0), + '60 smooth beats, no gaps': hrvSeries(Rng(5), 60, 0, gapP: 0), + '300 white-noise beats (jitter floor nulls rmssd and pnn50)': + hrvSeries(Rng(6), 300, 1, gapP: 0), + 'constant 1000 ms, 400 beats (zero variance, bin edge at beat 300)': + hrvConstant(400, 1000), + 'constant 2000 ms, 400 beats (3 bins)': hrvConstant(400, 2000), + 'slow beats spanning many bins': hrvSeries(Rng(7), 380, 2, gapP: .02), + 'many holes': hrvSeries(Rng(8), 200, 0, gapP: .3), + 'alternating runs': _alternatingRuns(4), + 'RSA at a slow heart, exempt from the jitter floor': + hrvSeries(Rng(61), 900, 3, gapP: 0), + 'RSA at a slow heart with holes': hrvSeries(Rng(62), 1500, 3, gapP: .004), + 'refused, published from a breathing line steady in Hz': + hrvSeries(Rng(63), 4000, 4, gapP: 0), + 'a hole of several bins': hrvBeats([ + for (var i = 0; i < 40; i++) + (nnMs: 800.0, timeMs: hrvT0 + (i + 1) * 800.0), + (nnMs: 900.0, timeMs: hrvT0 + 40 * 800 + 900000), + (nnMs: 910.0, timeMs: hrvT0 + 40 * 800 + 900910), + (nnMs: 905.0, timeMs: hrvT0 + 40 * 800 + 901815), + ]), + }; + + named.forEach((name, beats) { + test('$name: gap aware matches hrvTime', () { + final want = hrvExpected(beats, gapAware: true); + final a = _agg(beats); + expectHrvParity(a.run(beats), want, name); + final r = Rng(name.hashCode & 0xffff); + for (var t = 0; t < 3; t++) { + _expectAllPaths(a, beats, randomBounds(r, beats.length), want, name); + } + }); + test('$name: gapAware false matches hrvTime without times', () { + final want = hrvExpected(beats, gapAware: false); + final a = _agg(beats, gapAware: false); + expectHrvParity(a.run(beats), want, name); + final r = Rng(name.hashCode & 0xfffe); + _expectAllPaths(a, beats, randomBounds(r, beats.length), want, name); + }); + }); + + test('fewer than two beats: nBeats set, everything else null', () { + for (final n in [0, 1]) { + final beats = hrvSeries(Rng(n), n, 0); + for (final ga in [true, false]) { + final v = _agg(beats, gapAware: ga).run(beats); + expect(v.nBeats, n); + expect(hrvFields(v), everyElement(isNull)); + } + } + }); + + test('hand-worked: nn 800, 810, 790 contiguous', () { + final beats = hrvBeats([ + (nnMs: 800.0, timeMs: 1000.0), + (nnMs: 810.0, timeMs: 1810.0), + (nnMs: 790.0, timeMs: 2600.0), + ]); + final v = hrvTimeAggregator(originMs: 1000.0).run(beats); + expect(v.nBeats, 3); + // diffs 10, -20 -> rmssd sqrt((100+400)/2) + expect(v.rmssd, closeTo(15.811388300841896, 1e-9)); + expect(v.pnn50, 0.0); + expect(v.sdnn, closeTo(10.0, 1e-9)); + expect(v.diffAcf1, isNull); // fewer than 30 diffs + expect(v.sdann, isNull); // a single 5-minute segment + expect(v.sdnnIndex, isNull); + }); + }); + + group('below the jitter floor (the batch reads the difference runs)', () { + test('the series reach the paths they name', () { + final exempt = hrvTime( + hrvSeries(Rng(61), 900, 3, gapP: 0).nnMs.toList(), + nnTimesMs: hrvSeries(Rng(61), 900, 3, gapP: 0).timeMs.toList()); + expect(exempt.value!.diffAcf1, lessThan(kNnDiffAcf1Floor)); + expect(exempt.value!.rmssd, isNotNull); + expect(exempt.note, startsWith('PRV not ECG-HRV')); + final b = hrvSeries(Rng(63), 4000, 4, gapP: 0); + final fallback = hrvTime(b.nnMs.toList(), nnTimesMs: b.timeMs.toList()); + expect(fallback.value!.diffAcf1, lessThan(kNnDiffAcf1Floor)); + expect(fallback.value!.rmssd, isNotNull); + expect(fallback.note, contains('breathing line steady in Hz')); + // Without times the same series is refused (no windows to fall back on). + expect(hrvTime(b.nnMs.toList()).value!.rmssd, isNull); + }); + + test('random slow-heart and breathing-line series: whole, chunked, sliced', + () { + final r = Rng(0x5C14); + for (var t = 0; t < 16; t++) { + final mode = 3 + t % 2; + final n = mode == 3 ? r.intIn(600, 1600) : r.intIn(3800, 4400); + final beats = hrvSeries(r, n, mode, gapP: r.nextBool() ? 0 : .003); + for (final ga in [true, false]) { + final want = hrvExpected(beats, gapAware: ga); + final a = _agg(beats, gapAware: ga); + expectHrvParity(a.run(beats), want, 'case $t n=$n ga=$ga whole'); + _expectAllPaths( + a, beats, randomBounds(r, n), want, 'case $t n=$n ga=$ga'); + } + } + }); + }); + + group('random parity (seeded)', () { + test('gap aware: whole series, chunked, sliced', () { + final r = Rng(0x5C11); + for (var t = 0; t < 40; t++) { + final n = r.nextBool(.1) ? r.intIn(0, 3) : r.intIn(0, 400); + final beats = + hrvSeries(r, n, r.intIn(0, 2), gapP: r.nextBool() ? .02 : .15); + final want = hrvExpected(beats, gapAware: true); + final a = _agg(beats); + expectHrvParity(a.run(beats), want, 'case $t n=$n whole'); + for (var c = 0; c < 2; c++) { + _expectAllPaths(a, beats, randomBounds(r, n), want, 'case $t n=$n'); + } + } + }); + + test('gapAware false (times present but ignored)', () { + final r = Rng(0x5C12); + for (var t = 0; t < 30; t++) { + final n = r.intIn(0, 300); + final beats = hrvSeries(r, n, r.intIn(0, 1)); + final want = hrvExpected(beats, gapAware: false); + final a = _agg(beats, gapAware: false); + expectHrvParity(a.run(beats), want, 'case $t whole'); + _expectAllPaths(a, beats, randomBounds(r, n), want, 'case $t'); + } + }); + + test('gapAware false with no times at all (NaN times)', () { + final r = Rng(0x5C13); + for (var t = 0; t < 20; t++) { + final n = r.intIn(0, 250); + final beats = hrvSeries(r, n, r.intIn(0, 1), withTimes: false); + final want = hrvExpected(beats, gapAware: false); + final a = hrvTimeAggregator(gapAware: false); + _expectAllPaths(a, beats, randomBounds(r, n), want, 'case $t'); + } + }); + + test('every single split point (combine and slicing) of a series with ' + 'holes and several bins', () { + final beats = hrvSeries(Rng(31), 120, 2, gapP: .1); + final want = hrvExpected(beats, gapAware: true); + final a = _agg(beats); + for (var k = 0; k <= beats.length; k++) { + final s = a.combine( + a.fold(beats, 0, k), a.fold(beats, k, beats.length)); + expectHrvParity(a.finish(s), want, 'split at $k'); + expectHrvParity(runSliced(a, beats, [0, k, beats.length]), want, + 'sliced at $k'); + } + }); + + test('splits on and around a 5-minute bin edge', () { + final beats = hrvConstant(400, 1000); + final want = hrvExpected(beats, gapAware: true); + final a = _agg(beats); + for (final k in [1, 2, 298, 299, 300, 301, 399]) { + final s = a.combine(a.fold(beats, 0, k), a.fold(beats, k, 400)); + expectHrvParity(a.finish(s), want, 'split at $k'); + expectHrvParity(runSliced(a, beats, [0, k, 400]), want, 'sliced $k'); + } + }); + + test('one beat per chunk and per slice', () { + final beats = hrvSeries(Rng(41), 80, 0, gapP: .1); + final want = hrvExpected(beats, gapAware: true); + final a = _agg(beats); + final b = singletonBounds(beats.length); + expectHrvParity(runChunked(a, beats, b), want, 'singleton chunks'); + expectHrvParity(runChunked(a, beats, b, copy: true), want, 'copies'); + expectHrvParity(runSliced(a, beats, b), want, 'singleton slices'); + }); + }); + + group('gaps and lags', () { + test('lag pairs are not formed across a gap', () { + final beats = _alternatingRuns(4); // 44 diffs, seams dropped + final want = hrvExpected(beats, gapAware: true); + expect(want.diffAcf1, isNotNull); + // The scenario discriminates: bridging the gap (ignoring times) moves acf1. + final bridged = hrvExpected(beats, gapAware: false); + expect((bridged.diffAcf1! - want.diffAcf1!).abs(), greaterThan(.02)); + final a = _agg(beats); + expect(a.run(beats).diffAcf1, closeTo(want.diffAcf1!, 1e-9)); + // And in chunks and slices cut on and around the seams. + for (final k in [11, 12, 13, 23, 24, 25, 35, 36, 37]) { + final bounds = [0, k, beats.length]; + final s = a.combine(a.fold(beats, 0, k), a.fold(beats, k, beats.length)); + expect(a.finish(s).diffAcf1, closeTo(want.diffAcf1!, 1e-9), + reason: 'split at $k'); + expect(runSliced(a, beats, bounds).diffAcf1, closeTo(want.diffAcf1!, 1e-9), + reason: 'sliced at $k'); + expect(runChunked(a, beats, bounds, copy: true).diffAcf1, + closeTo(want.diffAcf1!, 1e-9), + reason: 'copies at $k'); + } + // Three-way cuts that put whole runs apart. + for (final b in [ + [0, 12, 24, 36, 48], + [0, 11, 13, 25, 48], + [0, 12, 12, 13, 36, 47, 48], + ]) { + _expectAllPaths(a, beats, b, want, 'runs'); + } + }); + + test('the hole rule: nn + 0.5 ms is contiguous, nn + 1.0 ms is a hole', () { + // diffs +100 then -30: contiguous keeps both (pNN50 50 %); a hole drops + // the -30 (pNN50 100 %). + final edge = hrvBeats([ + (nnMs: 800.0, timeMs: 1000.0), + (nnMs: 900.0, timeMs: 1900.5), // dt 900.5 = nn + 0.5 + (nnMs: 870.0, timeMs: 2771.0), // dt 870.5 = nn + 0.5 + ]); + final hole = hrvBeats([ + (nnMs: 800.0, timeMs: 1000.0), + (nnMs: 900.0, timeMs: 1900.5), + (nnMs: 870.0, timeMs: 2772.0), // dt 871.5 > nn + 0.5 + ]); + final a = hrvTimeAggregator(originMs: 1000.0); + expect(a.run(edge).pnn50, 50.0); + expect(a.run(hole).pnn50, 100.0); + expect(a.run(edge).nBeats, 3); + expect(a.run(hole).nBeats, 3); + // The same across every cut, because the gap check reads times at the seam. + for (final bounds in [ + [0, 1, 3], + [0, 2, 3], + [0, 1, 2, 3], + [0, 0, 1, 1, 3, 3] + ]) { + expect(runChunked(a, edge, bounds).pnn50, 50.0, reason: 'edge $bounds'); + expect(runSliced(a, edge, bounds).pnn50, 50.0, reason: 'edge $bounds'); + expect(runChunked(a, hole, bounds, copy: true).pnn50, 100.0, + reason: 'hole $bounds'); + expect(runSliced(a, hole, bounds).pnn50, 100.0, reason: 'hole $bounds'); + } + }); + + test('a contiguous seam with a hole in the gapAware false case is bridged', + () { + final hole = hrvBeats([ + (nnMs: 800.0, timeMs: 1000.0), + (nnMs: 900.0, timeMs: 1900.5), + (nnMs: 870.0, timeMs: 2772.0), + ]); + final a = hrvTimeAggregator(gapAware: false); + expect(a.run(hole).pnn50, 50.0); // times ignored: both diffs count + expect(a.run(hole).sdann, isNull); + expect(a.run(hole).sdnnIndex, isNull); + }); + }); + + group('origin', () { + test('a binned beat without an origin is a StateError', () { + final beats = hrvSeries(Rng(51), 20, 0, gapP: 0); + final a = hrvTimeAggregator(); // gapAware, no origin + expect(() => a.run(beats), throwsStateError); + expect(() => runChunked(a, beats, [0, 7, 20]), throwsStateError); + expect(() => runSliced(a, beats, [0, 7, 20]), throwsStateError); + expect(() => a.run(hrvBeats([(nnMs: 800.0, timeMs: hrvT0 + 800)])), + throwsStateError); + }); + + test('empty input with a null origin is fine', () { + final a = hrvTimeAggregator(); + final v = a.run(hrvBeats([])); + expect(v.nBeats, 0); + expect(hrvFields(v), everyElement(isNull)); + expect(a.finish(a.empty()).nBeats, 0); + expect(a.finish(a.combine(a.empty(), a.empty())).nBeats, 0); + expect(a.finish(a.fold(hrvSeries(Rng(52), 5, 0), 2, 2)).nBeats, 0); + }); + + test('gapAware false never needs an origin', () { + final beats = hrvSeries(Rng(53), 50, 0); + final v = hrvTimeAggregator(gapAware: false).run(beats); + expect(v.nBeats, 50); + }); + }); + + test('persistable: a mid-stream checkpoint resumes to the one-pass result', () { + final b = hrvSeries(Rng(0xC0DE), 90, 1, gapP: .08); + expectCheckpointResumes(hrvTimeAggregator(originMs: hrvOrigin(b)), b, + equals: hrvEqAll); + }); + + group('laws', () { + List cases(int seed, {int count = 8, int maxN = 120}) { + final r = Rng(seed); + return [ + hrvBeats([]), + hrvSeries(Rng(seed), 1, 0), + for (var i = 0; i < count; i++) + hrvSeries(r, r.intIn(0, maxN), r.intIn(0, 2), gapP: .08), + ]; + } + + test('below the jitter floor, gap aware and not, is lawful', () { + final r = Rng(0x5D05); + final group = [ + hrvSeries(Rng(61), 900, 3, gapP: 0), + hrvSeries(r, 1200, 3, gapP: .004), + hrvSeries(Rng(63), 4000, 4, gapP: 0), + ]; + // These series are thousands of beats long (the exemption needs 20 Welch + // segments in a run, the fallback 12 five-minute windows), so the split + // and associativity laws are checked at 10 split points, not 64. + for (final c in group) { + for (final agg in [ + hrvTimeAggregator(originMs: hrvOrigin(c)), + hrvTimeAggregator(gapAware: false), + ]) { + final v = checkLaws(agg, [c], equals: hrvEqAll, maxSplits: 10); + expect(v, isEmpty, + reason: v.map((x) => '${x.law}: ${x.detail}').join('\n')); + } + } + }); + + test('gap aware is lawful (cases grouped by origin)', () { + final byOrigin = >{}; + for (final c in cases(0x5D01)) { + (byOrigin[hrvOrigin(c)] ??= []).add(c); + } + byOrigin.forEach((origin, group) { + expectLawful(hrvTimeAggregator(originMs: origin), group, + equals: hrvEqAll); + }); + }); + + test('gap aware with a shared origin is lawful', () { + final shifted = [for (final c in cases(0x5D02)) hrvShifted(c, 5000.0)]; + expectLawful(hrvTimeAggregator(originMs: 5000.0), shifted, + equals: hrvEqAll); + }); + + test('gapAware false is lawful', () { + expectLawful(hrvTimeAggregator(gapAware: false), cases(0x5D03), + equals: hrvEqAll); + }); + + test('alternating runs are lawful at every split', () { + final beats = _alternatingRuns(4); + expectLawful(hrvTimeAggregator(originMs: hrvOrigin(beats)), [beats], + equals: hrvEqAll); + }); + }); + + group('numerical stability', () { + // The review's series: a steady 5 ms/beat ramp with a tiny odd-beat bump. + // Diffs are ~5 +- 1e-8, so raw sums of d and d^2 cancel catastrophically + // for the lag-1 autocorrelation; centred statistics do not. + Beats ramp(double bump) => hrvBeats([ + for (var i = 0; i <= 100; i++) + (nnMs: 800.0 + 5 * i + (i.isOdd ? bump : 0.0), timeMs: null), + ]); + + test('1e-8 perturbation: acf1 within 1e-6 of hrvTime, same RMSSD nullness', + () { + final beats = ramp(1e-8); + final want = hrvTime(beats.nnMs.toList()).value!; + final got = hrvTimeAggregator(gapAware: false).run(beats); + expect(want.diffAcf1, isNotNull); + expect(got.diffAcf1, isNotNull, reason: 'aggregator acf1 is null'); + expect((got.diffAcf1! - want.diffAcf1!).abs(), lessThan(1e-6), + reason: 'acf1 ${got.diffAcf1} vs ${want.diffAcf1}'); + expect(got.rmssd == null, want.rmssd == null, + reason: 'rmssd ${got.rmssd} vs ${want.rmssd}'); + }); + + test('1e-4 perturbation: whole vs split at 50 agree to 1e-9', () { + final beats = ramp(1e-4); + final agg = hrvTimeAggregator(gapAware: false); + final whole = agg.run(beats); + expect(whole.diffAcf1, isNotNull); + for (final copy in [false, true]) { + final split = runChunked(agg, beats, [0, 50, beats.length], copy: copy); + expect(split.diffAcf1, isNotNull); + expect((split.diffAcf1! - whole.diffAcf1!).abs(), lessThan(1e-9), + reason: 'copy=$copy split ${split.diffAcf1} whole ${whole.diffAcf1}'); + expect(split.rmssd == null, whole.rmssd == null); + } + }); + }); + + group('checkpoint relationships (beat counts across the children)', () { + final beats = hrvSeries(Rng(0x4A7), 60, 1, gapP: .08); + final agg = hrvTimeAggregator(originMs: hrvOrigin(beats)); + // State (version 2): [core, evidence]; core is [[diffs, lags], [levels, + // bins]]; diffs and lags are Lookback states ({length, first, last, + // inner}); levels is a Moments map and bins is {fiveMinuteBin: Moments}; + // evidence is the jitter-gate evidence map. + List core(Object? s) => (s as List)[0] as List; + Map diffs(Object? s) => (core(s)[0] as List)[0] as Map; + Map lags(Object? s) => (core(s)[0] as List)[1] as Map; + Map levels(Object? s) => (core(s)[1] as List)[0] as Map; + Map bins(Object? s) => (core(s)[1] as List)[1] as Map; + Map evidence(Object? s) => (s as List)[1] as Map; + + test('both lookbacks must have seen the same number of beats', () { + expectDamagedCheckpointRefused(agg, beats, (s) { + lags(s)['length'] = 61; + return s; + }); + }); + test('more level samples than beats is refused', () { + expectDamagedCheckpointRefused(agg, beats, (s) { + levels(s)['count'] = 70; + return s; + }); + }); + test('five-minute bin counts must add up to the level count', () { + expectDamagedCheckpointRefused(agg, beats, (s) { + final first = bins(s).values.first as Map; + first['count'] = (first['count'] as int) + 1; + return s; + }); + }); + test('more successive differences than beat pairs is refused', () { + expectDamagedCheckpointRefused(agg, beats, (s) { + final inner = diffs(s)['inner'] as Map; + inner['pairs'] = 60; + inner['over50'] = 0; + return s; + }); + }); + test('more lag pairs than beat triples is refused', () { + expectDamagedCheckpointRefused(agg, beats, (s) { + (lags(s)['inner'] as Map)['pairs'] = 59; + return s; + }); + }); + test('a version 1 checkpoint (no jitter evidence) is refused whole', () { + expectDamagedCheckpointRefused(agg, beats, (s) => core(s)); + }); + test('evidence of another type or version is refused', () { + expectDamagedCheckpointRefused(agg, beats, (s) { + evidence(s)['version'] = 2; + return s; + }); + expectDamagedCheckpointRefused(agg, beats, (s) { + evidence(s)['type'] = 'Other'; + return s; + }); + }); + test('evidence counting other beats or differences than the core', () { + expectDamagedCheckpointRefused(agg, beats, (s) { + evidence(s)['n'] = (evidence(s)['n'] as int) - 1; + return s; + }); + expectDamagedCheckpointRefused(agg, beats, (s) { + evidence(s)['nd'] = (evidence(s)['nd'] as int) - 1; + return s; + }); + }); + test('evidence whose parts disagree is refused', () { + for (final damage in [ + (e) => (e['head'] as Map)['len'] = ((e['head'] as Map)['len'] as int) + 1, + (e) => (e['head'] as Map)['anchored'] = true, + (e) => (e['head'] as Map)['phases'] = [], + (e) => ((e['head'] as Map)['cnt'] as List)[0] = 9999, + (e) => (e['wins'] as List).removeLast(), + (e) => e['headRun'] = null, + (e) => e['winNd'] = (e['nd'] as int) + 1, + ]) { + expectDamagedCheckpointRefused(agg, beats, (s) { + damage(evidence(s)); + return s; + }); + } + }); + test('a consistent checkpoint restores', () { + expect( + decodeCheckpoint(agg, checkpointOf(agg, beats), + kind: 'test', version: 1), + isNotNull); + }); + }); +} diff --git a/test/aggregate/hypnogram_aggregator_test.dart b/test/aggregate/hypnogram_aggregator_test.dart new file mode 100644 index 0000000..ac99cfb --- /dev/null +++ b/test/aggregate/hypnogram_aggregator_test.dart @@ -0,0 +1,461 @@ +// HypnogramAggregator parity: folding the sorted segments through the +// aggregator must reproduce AdvancedSleepStager.hypnogramMetrics exactly, for +// any input and any chunking. +// +// _referenceHypnogram is a FROZEN copy of the pre-aggregator implementation. +// Do not edit it when hypnogramMetrics is rewired to use the aggregator: the +// public function is checked against it so the rewire cannot drift. + +import 'dart:math' as math; + +import 'package:openstrap_analytics/aggregate.dart'; +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +import '../support/checkpoint_support.dart'; +import '../support/aggregator_laws.dart'; +import '../support/property.dart' show Rng; + +// ── frozen reference ──────────────────────────────────────────────────────── + +HypnogramMetrics _referenceHypnogram(SleepSession session) { + final segs = [...session.stages]..sort((a, b) => a.start.compareTo(b.start)); + final tib = math.max(0, session.end - session.start); + final sleepSegs = segs + .where((s) => s.stage == 'light' || s.stage == 'deep' || s.stage == 'rem') + .toList(); + var tst = 0, deepS = 0, remS = 0, lightS = 0; + for (final s in sleepSegs) { + final d = s.end - s.start; + tst += d; + if (s.stage == 'deep') deepS += d; + if (s.stage == 'rem') remS += d; + if (s.stage == 'light') lightS += d; + } + int onset, sptEnd; + int? sol; + if (sleepSegs.isNotEmpty) { + onset = sleepSegs.first.start; + sptEnd = sleepSegs.last.end; + sol = math.max(0, onset - session.start); + } else { + onset = session.end; + sptEnd = session.end; + sol = null; + } + final remSegs = sleepSegs.where((s) => s.stage == 'rem').toList(); + final remLatency = + remSegs.isNotEmpty ? (remSegs.first.start - onset).toDouble() : double.nan; + var waso = 0, disturbances = 0; + for (final s in segs.where((s) => s.stage == 'wake')) { + final w0 = math.max(s.start, onset); + final w1 = math.min(s.end, sptEnd); + if (w1 > w0) { + waso += w1 - w0; + disturbances++; + } + } + final se = tib > 0 ? tst / tib : 0.0; + double pct(int x) => tst > 0 ? x / tst * 100 : 0; + return HypnogramMetrics( + tibS: tib, + tstS: tst, + sptS: math.max(0, sptEnd - onset), + solS: sol, + remLatencyS: remLatency, + wasoS: waso, + efficiency: math.min(1.0, se), + disturbances: disturbances, + deepMin: deepS / 60, + remMin: remS / 60, + lightMin: lightS / 60, + deepPct: pct(deepS), + remPct: pct(remS), + lightPct: pct(lightS), + ); +} + +// ── helpers ───────────────────────────────────────────────────────────────── + +SleepSession _session(int start, int end, List stages) => + SleepSession( + start: start, + end: end, + efficiency: 0, + stages: stages, + restingHr: null, + avgHrv: null); + +/// Stable sort by start. With at most 30 segments, the sort inside the +/// reference/public function (insertion sort below 32 elements) is stable too, +/// so re-sorting this list leaves it unchanged and ties stay in input order. +List _sorted(List xs) { + final idx = [for (var i = 0; i < xs.length; i++) i]; + idx.sort((a, b) { + final c = xs[a].start.compareTo(xs[b].start); + return c != 0 ? c : a.compareTo(b); + }); + return [for (final i in idx) xs[i]]; +} + +String _show(HypnogramMetrics m) => m.toJson().toString() + + ' remLatencyS=${m.remLatencyS}'; + +/// Field by field, exactly: ints ==, doubles bit-identical (NaN == NaN). +void _expectSame(HypnogramMetrics got, HypnogramMetrics want, String tag) { + void i(String n, int? a, int? b) => expect(a, b, reason: '$tag: $n'); + void d(String n, double a, double b) => + expect(a.toString(), b.toString(), reason: '$tag: $n ($a vs $b)'); + i('tibS', got.tibS, want.tibS); + i('tstS', got.tstS, want.tstS); + i('sptS', got.sptS, want.sptS); + i('solS', got.solS, want.solS); + d('remLatencyS', got.remLatencyS, want.remLatencyS); + i('wasoS', got.wasoS, want.wasoS); + d('efficiency', got.efficiency, want.efficiency); + i('disturbances', got.disturbances, want.disturbances); + d('deepMin', got.deepMin, want.deepMin); + d('remMin', got.remMin, want.remMin); + d('lightMin', got.lightMin, want.lightMin); + d('deepPct', got.deepPct, want.deepPct); + d('remPct', got.remPct, want.remPct); + d('lightPct', got.lightPct, want.lightPct); +} + +List _fields(HypnogramMetrics m) => [ + m.tibS, m.tstS, m.sptS, m.solS, m.remLatencyS, m.wasoS, m.efficiency, + m.disturbances, m.deepMin, m.remMin, m.lightMin, m.deepPct, m.remPct, + m.lightPct, + ]; + +typedef _Agg = Aggregator, HypnogramMetrics>; + +ListBatch _batch(List xs) => ListBatch(xs); + +/// Whole run, then every chunked / sliced evaluation of [bounds], against [want]. +void _expectAllPaths(_Agg agg, List sorted, List bounds, + HypnogramMetrics want, String tag) { + final b = _batch(sorted); + _expectSame(runChunked(agg, b, bounds), want, '$tag chunks $bounds'); + _expectSame(runChunked(agg, b, bounds, copy: true), want, + '$tag copied chunks $bounds'); + _expectSame(runChunkedRight(agg, b, bounds), want, '$tag right $bounds'); + _expectSame(runSliced(agg, b, bounds), want, '$tag sliced $bounds'); +} + +const _stages = [ + 'wake', 'light', 'deep', 'rem', 'wake', 'light', 'deep', 'rem', // + 'unknown', 'REM', '', 'Wake', +]; + +/// A messy session: random bounds (sometimes end < start), up to [maxSegs] +/// segments placed anywhere (before, inside, after the session), random +/// durations (sometimes zero or negative), overlaps, unknown stage strings. +({int start, int end, List segs}) _messy(Rng r, int maxSegs) { + final start = r.intIn(-500, 1500); + final len = r.nextBool(.05) ? r.intIn(-50, 0) : r.intIn(0, 3000); + final end = start + len; + final n = r.intIn(0, maxSegs); + final segs = []; + for (var k = 0; k < n; k++) { + final s = r.intIn(start - 300, end + 300); + final d = r.nextBool(.05) ? r.intIn(-30, 0) : r.intIn(0, 600); + segs.add(StageSegment(s, s + d, _stages[r.nextInt(_stages.length)])); + } + return (start: start, end: end, segs: segs); +} + +/// A clean contiguous night starting at [start]: segments back to back. +({int start, int end, List segs}) _tidy(Rng r) { + final start = r.intIn(0, 1000); + var t = start + r.intIn(0, 120); + final segs = []; + final n = r.intIn(1, 28); + const names = ['wake', 'light', 'deep', 'rem', 'light', 'rem', 'wake']; + for (var k = 0; k < n; k++) { + final d = r.intIn(30, 900); + segs.add(StageSegment(t, t + d, names[r.nextInt(names.length)])); + t += d; + } + return (start: start, end: t + r.intIn(0, 200), segs: segs); +} + +void main() { + // Named scenarios, each with the sorted segments. + final scenarios = segs})>{ + 'empty stages': (start: 0, end: 1000, segs: []), + 'empty stages, zero-length session': (start: 50, end: 50, segs: []), + 'inverted session bounds': ( + start: 100, + end: 40, + segs: [StageSegment(100, 200, 'light')] + ), + 'only wake': ( + start: 0, + end: 600, + segs: [StageSegment(0, 300, 'wake'), StageSegment(300, 600, 'wake')] + ), + 'no sleep, only unknown stages': ( + start: 0, + end: 600, + segs: [StageSegment(0, 300, 'unknown'), StageSegment(300, 600, 'asleep')] + ), + 'wake before onset': ( + start: 0, + end: 1000, + segs: [ + StageSegment(0, 200, 'wake'), + StageSegment(200, 500, 'light'), + StageSegment(500, 900, 'rem'), + ] + ), + 'wake after last sleep': ( + start: 0, + end: 1000, + segs: [ + StageSegment(0, 400, 'light'), + StageSegment(400, 600, 'deep'), + StageSegment(600, 1000, 'wake'), + ] + ), + 'wake straddling onset and spt end': ( + start: 0, + end: 1000, + segs: [ + StageSegment(0, 300, 'wake'), + StageSegment(100, 250, 'light'), + StageSegment(250, 500, 'wake'), + StageSegment(500, 700, 'rem'), + StageSegment(650, 900, 'wake'), + ] + ), + 'wake in the middle of the night': ( + start: 0, + end: 3000, + segs: [ + StageSegment(0, 600, 'light'), + StageSegment(600, 700, 'wake'), + StageSegment(700, 1500, 'deep'), + StageSegment(1500, 1560, 'wake'), + StageSegment(1560, 2400, 'rem'), + StageSegment(2400, 3000, 'wake'), + ] + ), + 'rem only': ( + start: 10, + end: 700, + segs: [StageSegment(100, 400, 'rem')] + ), + 'single light segment': ( + start: 0, + end: 3600, + segs: [StageSegment(0, 3600, 'light')] + ), + 'tst exceeds time in bed (overlap)': ( + start: 0, + end: 100, + segs: [StageSegment(0, 100, 'light'), StageSegment(0, 100, 'deep')] + ), + 'overlapping sleep segments': ( + start: 0, + end: 1000, + segs: [ + StageSegment(0, 500, 'light'), + StageSegment(200, 800, 'rem'), + StageSegment(300, 400, 'wake'), + ] + ), + 'zero and negative length segments': ( + start: 0, + end: 500, + segs: [ + StageSegment(10, 10, 'light'), + StageSegment(20, 5, 'rem'), + StageSegment(30, 30, 'wake'), + StageSegment(40, 400, 'deep'), + ] + ), + 'segments entirely outside the session': ( + start: 1000, + end: 2000, + segs: [StageSegment(0, 100, 'light'), StageSegment(3000, 3100, 'rem')] + ), + 'tied starts keep input order': ( + start: 0, + end: 500, + segs: [ + StageSegment(10, 100, 'wake'), + StageSegment(10, 200, 'light'), + StageSegment(10, 50, 'rem'), + StageSegment(10, 300, 'wake'), + ] + ), + 'case-sensitive stage names': ( + start: 0, + end: 500, + segs: [StageSegment(0, 100, 'Light'), StageSegment(100, 200, 'REM')] + ), + }; + + List> allBounds(int n) => [ + [0, n], + for (var k = 0; k <= n; k++) [0, k, n], + [0, 0, 0, n, n], + if (n >= 3) [0, 1, 2, n], + if (n >= 3) [0, 1, 1, n - 1, n], + ]; + + group('named scenarios', () { + scenarios.forEach((name, sc) { + test('$name: aggregator == frozen reference == public function', () { + final sorted = _sorted(sc.segs); + final session = _session(sc.start, sc.end, sc.segs); + final want = _referenceHypnogram(session); + final _Agg agg = HypnogramAggregator(sc.start, sc.end); + _expectSame(agg.runList(sorted), want, 'run vs reference'); + _expectSame(AdvancedSleepStager.hypnogramMetrics(session), want, + 'public vs reference'); + _expectSame(agg.runList(sorted), + AdvancedSleepStager.hypnogramMetrics(session), 'run vs public'); + for (final b in allBounds(sorted.length)) { + _expectAllPaths(agg, sorted, b, want, name); + } + }); + }); + + test('no sleep: solS is null and remLatency is NaN', () { + final m = HypnogramAggregator(0, 600) + .runList([const StageSegment(0, 600, 'wake')]); + expect(m.solS, isNull); + expect(m.remLatencyS.isNaN, isTrue); + expect(m.tstS, 0); + expect(m.sptS, 0); + expect(m.wasoS, 0); + expect(m.efficiency, 0.0); + }); + + test('a known night, worked by hand', () { + // 0-100 wake, 100-400 light, 400-500 wake, 500-700 rem, 700-800 wake. + final m = HypnogramAggregator(0, 800).runList(const [ + StageSegment(0, 100, 'wake'), + StageSegment(100, 400, 'light'), + StageSegment(400, 500, 'wake'), + StageSegment(500, 700, 'rem'), + StageSegment(700, 800, 'wake'), + ]); + expect(m.tibS, 800); + expect(m.tstS, 500); + expect(m.sptS, 600); + expect(m.solS, 100); + expect(m.remLatencyS, 400.0); + expect(m.wasoS, 100); + expect(m.disturbances, 1); + expect(m.efficiency, 500 / 800); + expect(m.lightMin, 300 / 60); + expect(m.remMin, 200 / 60); + expect(m.deepMin, 0.0); + expect(m.remPct, 200 / 500 * 100); + }); + }); + + group('random parity (seeded)', () { + test('messy input: unchunked, both references', () { + final r = Rng(0xA11CE); + for (var t = 0; t < 120; t++) { + final g = _messy(r, 30); + final session = _session(g.start, g.end, g.segs); + final want = _referenceHypnogram(session); + final got = HypnogramAggregator(g.start, g.end).runList(_sorted(g.segs)); + _expectSame(got, want, 'case $t ${_show(want)}'); + _expectSame(AdvancedSleepStager.hypnogramMetrics(session), want, + 'public, case $t'); + } + }); + + test('tidy nights: unchunked, both references', () { + final r = Rng(0xB0B); + for (var t = 0; t < 60; t++) { + final g = _tidy(r); + final session = _session(g.start, g.end, g.segs); + final want = _referenceHypnogram(session); + _expectSame(HypnogramAggregator(g.start, g.end).runList(_sorted(g.segs)), + want, 'case $t'); + _expectSame(AdvancedSleepStager.hypnogramMetrics(session), want, + 'public, case $t'); + } + }); + + test('random chunkings and slicings, every evaluation path', () { + final r = Rng(0xC0DE); + for (var t = 0; t < 150; t++) { + final g = t.isEven ? _messy(r, 30) : _tidy(r); + final sorted = _sorted(g.segs); + final want = _referenceHypnogram(_session(g.start, g.end, g.segs)); + final _Agg agg = HypnogramAggregator(g.start, g.end); + for (var c = 0; c < 4; c++) { + _expectAllPaths(agg, sorted, randomBounds(r, sorted.length), want, + 'case $t'); + } + } + }); + + test('combine order: right-nested equals left-nested', () { + final r = Rng(0xD00D); + for (var t = 0; t < 40; t++) { + final g = _messy(r, 30); + final sorted = _sorted(g.segs); + final want = _referenceHypnogram(_session(g.start, g.end, g.segs)); + final _Agg agg = HypnogramAggregator(g.start, g.end); + final b = randomBounds(r, sorted.length); + _expectSame(runChunkedRight(agg, _batch(sorted), b), want, + 'case $t right-nested $b'); + _expectSame(runChunked(agg, _batch(sorted), b), want, + 'case $t left-nested $b'); + } + }); + }); + + test('persistable: a mid-stream checkpoint resumes to the one-pass result', () { + final g = _tidy(Rng(0xC0DE)); + expectCheckpointResumes( + HypnogramAggregator(g.start, g.end), _batch(_sorted(g.segs)), + equals: fieldsEqual(_fields)); + }); + + group('laws', () { + test('lawful on messy and tidy sorted inputs (exact equality)', () { + final r = Rng(0xFACE); + // The session bounds are an instance property, so one instance per + // session; the laws hold per instance. + for (var t = 0; t < 12; t++) { + final g = t.isEven ? _messy(r, 30) : _tidy(r); + final sorted = _sorted(g.segs); + final cases = >[ + _batch([]), + if (sorted.isNotEmpty) _batch([sorted.first]), + _batch(sorted), + _batch(sorted.sublist(0, sorted.length ~/ 2)), + _batch(sorted.sublist(sorted.length ~/ 2)), + ]; + expectLawful(HypnogramAggregator(g.start, g.end), cases, + equals: fieldsEqual(_fields)); + } + }); + + test('finish does not mutate: finishing twice and then adding more', () { + final _Agg agg = HypnogramAggregator(0, 1000); + const segs = [ + StageSegment(0, 100, 'wake'), + StageSegment(100, 300, 'light'), + StageSegment(300, 400, 'wake'), + StageSegment(400, 600, 'rem'), + ]; + final batch = _batch(segs); + var s = agg.fold(batch, 0, 3); + final a = agg.finish(s), b = agg.finish(s); + _expectSame(a, b, 'twice'); + s = agg.addSlice(s, batch, 3, 4); + _expectSame(agg.finish(s), agg.runList(segs), 'continued'); + }); + }); +} diff --git a/test/aggregate/last_rows_test.dart b/test/aggregate/last_rows_test.dart new file mode 100644 index 0000000..da0dc2a --- /dev/null +++ b/test/aggregate/last_rows_test.dart @@ -0,0 +1,75 @@ +// LastRows(k): a verbatim copy of the last min(k, n) rows of the whole input, +// null for an empty input. Persistable when batchFromJson is given. + +import 'package:openstrap_analytics/aggregate.dart'; +import 'package:test/test.dart'; + +import '../support/aggregator_laws.dart'; +import '../support/checkpoint_support.dart'; + +const nan = double.nan; + +List _vals(DoubleColumn? c) => [c == null, ...?c?.values]; +final _same = fieldsEqual(_vals); // NaN-aware, exact + +LastRows _last(int k) => + LastRows(k, batchFromJson: DoubleColumn.fromJson); + +void main() { + group('values', () { + final x = col([1, 2, 3, 4, 5]); + test('k = 1, 3, 8 on a longer and a shorter input', () { + expect(_last(1).run(x)!.values.toList(), [5]); + expect(_last(3).run(x)!.values.toList(), [3, 4, 5]); + expect(_last(8).run(x)!.values.toList(), [1, 2, 3, 4, 5]); // n < k + expect(_last(3).run(col([7, nan]))!.values.toList(), [7, nan]); + }); + test('empty input is null, also through finish(empty())', () { + for (final k in [1, 3, 8]) { + expect(_last(k).run(col([])), isNull); + expect(_last(k).finish(_last(k).empty()), isNull); + } + }); + test('honours the range and returns a copy', () { + final a = _last(3); + expect(a.finish(a.fold(x, 0, 4))!.values.toList(), [2, 3, 4]); + expect(a.finish(a.fold(x, 2, 2)), isNull); + final got = a.run(x)!; + got.values[0] = 99; + expect(x.values.toList(), [1, 2, 3, 4, 5]); + }); + test('combine keeps the last k of left ++ right', () { + final a = _last(3); + List at(int i, int j) => a + .finish(a.combine(a.fold(x, 0, i), a.fold(x, i, j)))! + .values + .toList(); + expect(at(4, 5), [3, 4, 5]); // right has 1 row: 2 from left + expect(at(2, 5), [3, 4, 5]); // right alone has k rows + expect(at(3, 4), [2, 3, 4]); + }); + }); + + group('laws', () { + for (final k in [1, 3, 8]) { + test('k=$k is lawful (slicing, finishPure, round trip)', () { + expectLawful(_last(k), randomColumns(40 + k), equals: _same); + }); + } + }); + + group('persistence', () { + test('persistable only with batchFromJson', () { + expect(LastRows(3).persistable, isFalse); + expect(_last(3).persistable, isTrue); + }); + test('a mid-stream checkpoint resumes to the one-pass result', () { + final c = randomColumns(77, count: 1, maxLen: 30).last; + for (final k in [1, 3, 8]) { + expectCheckpointResumes(_last(k), col([for (var i = 0; i < 20; i++) i * 1.5]), + equals: _same); + expectCheckpointResumes(_last(k), c, equals: _same); + } + }); + }); +} diff --git a/test/aggregate/laws_harness_test.dart b/test/aggregate/laws_harness_test.dart new file mode 100644 index 0000000..9cd219c --- /dev/null +++ b/test/aggregate/laws_harness_test.dart @@ -0,0 +1,707 @@ +// The law harness itself: lawful aggregators pass, deliberately broken ones are +// caught under the right law name (including 'slicing'), and fieldsEqual +// compares as documented. + +import 'dart:typed_data'; + +import 'package:openstrap_analytics/aggregate.dart'; +import 'package:test/test.dart'; + +import '../support/aggregator_laws.dart'; + +typedef _B = ListBatch; + +int _sliceSum(_B b, int start, int end) { + var t = 0; + for (var i = start; i < end; i++) { + t += b.rows[i]; + } + return t; +} + +/// A correct sum over int rows. +class _Sum extends Aggregator { + const _Sum(); + @override + int empty() => 0; + @override + int addSlice(int s, _B b, int start, int end) => s + _sliceSum(b, start, end); + @override + int combine(int left, int right) => left + right; + @override + int finish(int s) => s; +} + +/// Sum whose identity is 1: breaks the identity laws. +class _SumFromOne extends _Sum { + const _SumFromOne(); + @override + int empty() => 1; +} + +/// Combine drops the right side. +class _DropRight extends _Sum { + const _DropRight(); + @override + int combine(int left, int right) => left; +} + +/// addSlice ignores the state it is given (restarts from the slice), so a +/// second addSlice onto a state that already holds data loses it. fold (one +/// addSlice onto empty) and combine are both correct, so only 'slicing' sees it. +class _IgnoresState extends _Sum { + const _IgnoresState(); + @override + int addSlice(int s, _B b, int start, int end) => _sliceSum(b, start, end); +} + +/// addSlice always sums the whole batch, whatever range it is asked for. +class _IgnoresRange extends _Sum { + const _IgnoresRange(); + @override + int addSlice(int s, _B b, int start, int end) => + s + _sliceSum(b, 0, b.length); +} + +/// Mean kept as (mean, count); combine averages the two means unweighted. +class _UnweightedMean extends Aggregator, _B, double> { + const _UnweightedMean(); + @override + List empty() => [0, 0]; + @override + List addSlice(List s, _B b, int start, int end) { + for (var i = start; i < end; i++) { + final n = s[1] + 1; + s = [s[0] + (b.rows[i] - s[0]) / n, n]; + } + return s; + } + + @override + List combine(List l, List r) { + if (r[1] == 0) return l; + if (l[1] == 0) return r; + return [(l[0] + r[0]) / 2, l[1] + r[1]]; + } + + @override + double finish(List s) => s[0]; +} + +/// addSlice counts 1 per row, but joining two NON-empty chunks adds a bonus, so +/// the add path and the combine path disagree. Identities still hold. +class _JoinBonus extends Aggregator { + const _JoinBonus(); + @override + int empty() => 0; + @override + int addSlice(int s, _B b, int start, int end) => s + (end - start); + @override + int combine(int left, int right) => + left > 0 && right > 0 ? left + right + 1 : left + right; + @override + int finish(int s) => s; +} + +/// Wrong only when exactly 3 rows meet exactly 2 (a split of a 5-row batch). +class _WrongAtThreeTwo extends Aggregator, _B, int> { + const _WrongAtThreeTwo(); + @override + List empty() => [0, 0]; // sum, count + @override + List addSlice(List s, _B b, int start, int end) => + [s[0] + _sliceSum(b, start, end), s[1] + (end - start)]; + @override + List combine(List l, List r) => + [l[0] + r[0] + (l[1] == 3 && r[1] == 2 ? 1000 : 0), l[1] + r[1]]; + @override + int finish(List s) => s[0]; +} + +/// A correct sum that mutates and consumes both arguments in combine. +class _MutatingSum extends Aggregator, _B, int> { + const _MutatingSum(); + @override + List empty() => [0]; + @override + List addSlice(List s, _B b, int start, int end) { + s[0] += _sliceSum(b, start, end); + return s; + } + + @override + List combine(List left, List right) { + left[0] += right[0]; + right[0] = -999999; // consumed + return left; + } + + @override + int finish(List s) => s[0]; +} + +/// Same, but wipes `left` too and returns a brand new state. +class _ConsumingSum extends _MutatingSum { + const _ConsumingSum(); + @override + List combine(List left, List right) { + final out = [left[0] + right[0]]; + left[0] = -1; + right[0] = -1; + return out; + } +} + +/// A sum whose finish reads the state and then clears it. A single finish per +/// fresh state looks right, so only a law that finishes the SAME state twice +/// (or keeps using it afterwards) can see the damage. +class _DestructiveSum extends Aggregator, _B, int> { + const _DestructiveSum(); + @override + List empty() => [0]; + @override + List addSlice(List s, _B b, int start, int end) { + s[0] += _sliceSum(b, start, end); + return s; + } + + @override + List combine(List left, List right) { + left[0] += right[0]; + return left; + } + + @override + int finish(List s) { + final out = s[0]; + s[0] = 0; // destructive + return out; + } +} + +/// finish returns the right number the first time, but marks the state as +/// closed so later rows are ignored. Two finishes in a row agree; continuing to +/// add afterwards does not. +class _FinishFreezes extends Aggregator, _B, int> { + const _FinishFreezes(); + @override + List empty() => [0, 0]; // sum, frozen + @override + List addSlice(List s, _B b, int start, int end) { + if (s[1] == 0) s[0] += _sliceSum(b, start, end); + return s; + } + + @override + List combine(List left, List right) { + if (left[1] == 0) left[0] += right[0]; + return left; + } + + @override + int finish(List s) { + s[1] = 1; + return s[0]; + } +} + +/// A correct sum whose codec only breaks when the restored state is the RIGHT +/// operand of combine: a state read back from JSON carries a flag, and combine +/// drops the right side's total when that flag is set. Restoring only the left +/// operand (the old roundTrip law) never sees it. +class _RightRestoreDrops extends Aggregator, _B, int> { + const _RightRestoreDrops({this.faithful = false}); + + /// When true the same codec is lawful: combine keeps a restored right side. + final bool faithful; + @override + List empty() => [0, 0]; // total, restored flag + @override + List addSlice(List s, _B b, int start, int end) { + s[0] += _sliceSum(b, start, end); + return s; + } + + @override + List combine(List left, List right) => + [left[0] + (right[1] == 1 && !faithful ? 0 : right[0]), left[1]]; + @override + int finish(List s) => s[0]; + @override + bool get persistable => true; + @override + Object? encodeState(List s) => s[0]; + @override + List decodeState(Object? json) => [json as int, 1]; +} + +Set _laws(List v) => {for (final x in v) x.law}; + +void main() { + final cases = randomIntBatches(11, lo: 1, hi: 50); + + group('lawful aggregators', () { + test('a correct sum has no violations', () { + expect(checkLaws(const _Sum(), cases), isEmpty); + }); + test('a mutating combine is judged lawful when otherwise lawful', () { + expect(checkLaws(const _MutatingSum(), cases), isEmpty); + expect(checkLaws(const _ConsumingSum(), cases), isEmpty); + }); + test('no cases is vacuously lawful', () { + expect(checkLaws(const _Sum(), <_B>[]), isEmpty); + }); + test('empty and one-row cases alone are fine', () { + expect(checkLaws(const _Sum(), [_B([]), _B([5])]), isEmpty); + }); + test('long cases are bounded by maxSplits and still pass', () { + final long = _B([for (var i = 0; i < 300; i++) i]); + expect(checkLaws(const _Sum(), [long], maxSplits: 8), isEmpty); + }); + test('the harness leaves the case batches unchanged', () { + final b = _B([5, 6, 7, 8]); + checkLaws(const _Sum(), [b]); + expect(b.rows, [5, 6, 7, 8]); + }); + test('expectLawful passes for a lawful aggregator and fails for a broken ' + 'one', () { + expectLawful(const _Sum(), cases); + expect(() => expectLawful(const _DropRight(), cases), + throwsA(isA())); + }); + }); + + group('broken aggregators are caught', () { + test('(a) non-identity empty fires the identity laws', () { + final laws = _laws(checkLaws(const _SumFromOne(), cases)); + expect(laws, containsAll(['leftIdentity', 'rightIdentity'])); + }); + + test('(b) combine that drops the right side fires split', () { + final v = checkLaws(const _DropRight(), cases); + expect(_laws(v), contains('split')); + expect(v.every((x) => x.detail.isNotEmpty), isTrue); + }); + + test('(c) unweighted mean combine fires split or associativity', () { + bool close(double a, double b) => (a - b).abs() < 1e-9; + final laws = + _laws(checkLaws(const _UnweightedMean(), cases, equals: close)); + expect(laws.intersection({'split', 'associativity'}), isNotEmpty); + }); + + test('(d) addSlice inconsistent with combine fires singletons (identities ' + 'hold)', () { + final laws = _laws(checkLaws(const _JoinBonus(), cases)); + expect(laws, contains('singletons')); + expect(laws, contains('split')); + expect(laws, isNot(contains('leftIdentity'))); + expect(laws, isNot(contains('rightIdentity'))); + }); + + test('(e) addSlice that ignores the state on a second call fires only ' + '"slicing"', () { + final laws = _laws(checkLaws(const _IgnoresState(), cases)); + expect(laws, {'slicing'}); + }); + + test('(f) addSlice that ignores its range fires split and slicing', () { + final laws = _laws(checkLaws(const _IgnoresRange(), cases)); + expect(laws, containsAll(['split', 'slicing'])); + }); + + test('every split point of a short case is tried', () { + // Only the 3|2 split of a five-row batch is wrong. + final laws = _laws(checkLaws(const _WrongAtThreeTwo(), [ + _B([1, 2, 3, 4, 5]) + ])); + expect(laws, contains('split')); + }); + + test('a lawful case alongside a broken one still reports the broken one', + () { + final v = checkLaws(const _DropRight(), [ + _B([]), + _B([5]), + _B([1, 2, 3]) + ]); + expect(_laws(v), contains('split')); + }); + + test('results are deterministic for a given seed', () { + String render(List v) => + v.map((x) => '${x.law}|${x.detail}').join('\n'); + final a = checkLaws(const _IgnoresState(), cases, seed: 5); + final b = checkLaws(const _IgnoresState(), cases, seed: 5); + expect(render(a), render(b)); + expect(a, isNotEmpty); + }); + + test('a custom equals is honoured', () { + // Everything is "equal", so even a broken aggregator passes. + expect(checkLaws(const _DropRight(), cases, equals: (a, b) => true), + isEmpty); + expect(checkLaws(const _IgnoresState(), cases, equals: (a, b) => true), + isEmpty); + }); + }); + + group('roundTrip restores the right operand too', () { + final small = [ + _B([]), + _B([1, 2, 3]), + _B([4, 5]), + ]; + + test('a codec that breaks only as the right operand is caught as exactly ' + '{roundTrip}', () { + final v = checkLaws(const _RightRestoreDrops(), small); + expect(_laws(v), {'roundTrip'}); + expect(v.every((x) => x.detail.isNotEmpty), isTrue); + }); + + test('the review case [1,2,3] is caught', () { + expect( + _laws(checkLaws(const _RightRestoreDrops(), [ + _B([1, 2, 3]) + ])), + {'roundTrip'}); + }); + + test('checkConversion reports it as law:roundTrip', () { + final f = checkConversion, _B, int>( + (_) => const _RightRestoreDrops(), + small, + (b) => b.rows.fold(0, (s, x) => s + x)); + expect({for (final x in f) x.path}, {'law:roundTrip'}); + }); + + test('expectLawful fails for it', () { + expect(() => expectLawful(const _RightRestoreDrops(), small), + throwsA(isA())); + }); + + test('the same codec with a combine that keeps a restored right side is ' + 'clean', () { + expect(checkLaws(const _RightRestoreDrops(faithful: true), small), + isEmpty); + }); + }); + + group('finishPure', () { + test('a sum whose finish clears the state is caught with exactly ' + '{finishPure}', () { + expect(_laws(checkLaws(const _DestructiveSum(), cases)), {'finishPure'}); + }); + + test('a finish that closes the state to later rows is caught with exactly ' + '{finishPure}', () { + expect(_laws(checkLaws(const _FinishFreezes(), cases)), {'finishPure'}); + }); + + test('the violation names the case', () { + final v = checkLaws(const _DestructiveSum(), [ + _B([1, 2, 3]) + ]); + expect(v, isNotEmpty); + expect(v.every((x) => x.law == 'finishPure' && x.detail.isNotEmpty), + isTrue); + }); + + test('lawful aggregators report no finishPure violation', () { + for (final agg in >[ + const _Sum(), + const _MutatingSum(), + const _ConsumingSum(), + ]) { + expect(checkLaws(agg, cases), isEmpty, reason: '$agg'); + } + }); + + test('expectLawful fails for a destructive finish', () { + expect(() => expectLawful(const _DestructiveSum(), cases), + throwsA(isA())); + }); + }); + + group('fieldsEqual', () { + List same(List x) => x; + + test('two ints compare exactly, with no double conversion', () { + final eq = fieldsEqual>(same, relTol: 1e-9, absTol: 1e-9); + expect(eq([9007199254740992], [9007199254740993]), isFalse); + expect(eq([9007199254740993], [9007199254740993]), isTrue); + final loose = fieldsEqual>(same, relTol: 0.5, absTol: 10); + expect(loose([100], [101]), isFalse); + }); + + test('tolerance is absTol + relTol * max(|a|, |b|)', () { + final eq = fieldsEqual>(same, absTol: 0.1, relTol: 0.05); + // 0.15 <= 0.1 + 0.05 * 1.15; the max(absTol, rel) rule would say no. + expect(eq([1.0], [1.15]), isTrue); + expect(eq([1.15], [1.0]), isTrue); + expect(eq([1.0], [1.2]), isFalse); + }); + + test('a finite value never equals a non-finite one', () { + final eq = fieldsEqual>(same, absTol: 1e300, relTol: 1); + expect(eq([1.0], [double.infinity]), isFalse); + expect(eq([double.negativeInfinity], [1.0]), isFalse); + expect(eq([double.maxFinite], [double.infinity]), isFalse); + expect(eq([1.0], [double.nan]), isFalse); + expect(eq([double.nan], [double.infinity]), isFalse); + }); + + test('NaN equals NaN and each infinity only itself, tolerances or not', () { + final eq = fieldsEqual>(same, absTol: 1, relTol: 1); + expect(eq([double.nan], [double.nan]), isTrue); + expect(eq([double.infinity], [double.infinity]), isTrue); + expect(eq([double.negativeInfinity], [double.negativeInfinity]), isTrue); + expect(eq([double.infinity], [double.negativeInfinity]), isFalse); + }); + + test('an int and a double compare with tolerance', () { + final eq = fieldsEqual>(same, absTol: 0.1); + expect(eq([1], [1.05]), isTrue); + expect(eq([1], [1.5]), isFalse); + }); + + test('identical lists are equal; length mismatch is not', () { + final eq = fieldsEqual>(same); + expect(eq([1, 'a', null], [1, 'a', null]), isTrue); + expect(eq([1, 2], [1, 2, 3]), isFalse); + }); + + test('NaN equals NaN, but not a number', () { + final eq = fieldsEqual>(same); + expect(eq([double.nan], [double.nan]), isTrue); + expect(eq([double.nan], [1.0]), isFalse); + expect(eq([1.0], [double.nan]), isFalse); + }); + + test('null equals only null', () { + final eq = fieldsEqual>(same, relTol: 1, absTol: 1); + expect(eq([null], [null]), isTrue); + expect(eq([null], [0.0]), isFalse); + expect(eq([0.0], [null]), isFalse); + }); + + test('zero tolerance is exact', () { + final eq = fieldsEqual>(same); + expect(eq([1.0], [1.0]), isTrue); + expect(eq([1.0], [1.0000000000000002]), isFalse); + expect(eq([1], [1.0]), isTrue); + }); + + test('relTol scales with magnitude', () { + final eq = fieldsEqual>(same, relTol: 1e-9); + expect(eq([1000.0], [1000.0000001]), isTrue); // 1e-7 abs, 1e-10 rel + expect(eq([1000.0], [1000.00001]), isFalse); // 1e-5 abs, 1e-8 rel + expect(eq([0.0], [1e-12]), isFalse); // relTol alone gives nothing at 0 + }); + + test('absTol covers values near zero', () { + final eq = fieldsEqual>(same, absTol: 1e-6); + expect(eq([0.0], [1e-7]), isTrue); + expect(eq([0.0], [1e-5]), isFalse); + }); + + test('infinities match only themselves', () { + final eq = fieldsEqual>(same, relTol: 1e-9, absTol: 1e-9); + expect(eq([double.infinity], [double.infinity]), isTrue); + expect(eq([double.infinity], [double.negativeInfinity]), isFalse); + }); + + test('non-numeric fields compare with ==', () { + final eq = fieldsEqual>(same, relTol: 1, absTol: 1); + expect(eq(['a'], ['a']), isTrue); + expect(eq(['a'], ['b']), isFalse); + expect(eq([true], [true]), isTrue); + expect(eq([1], ['1']), isFalse); + }); + + test('the projection is applied to both sides', () { + final eq = + fieldsEqual<({int a, double b})>((r) => [r.a, r.b], absTol: 1e-6); + expect(eq((a: 1, b: 2.0), (a: 1, b: 2.0000001)), isTrue); + expect(eq((a: 1, b: 2.0), (a: 2, b: 2.0)), isFalse); + }); + + group('list fields', () { + test('equal-valued distinct List fields are equal', () { + final eq = fieldsEqual>(same); + expect( + eq([ + [1.0, 2.5] + ], [ + [1.0, 2.5] + ]), + isTrue); + }); + + test('Float64List equals a List with the same values', () { + final eq = fieldsEqual>(same); + final typed = Float64List.fromList([1.0, 2.5]); + expect( + eq([typed], [ + [1.0, 2.5] + ]), + isTrue); + expect( + eq([ + [1.0, 2.5] + ], [typed]), + isTrue); + }); + + test('elements use the numeric tolerance', () { + final eq = fieldsEqual>(same, relTol: 1e-3); + expect( + eq([ + [1000.0, 1.0] + ], [ + [1000.5, 1.0] + ]), + isTrue); + expect( + eq([ + [1000.0, 1.0] + ], [ + [1002.0, 1.0] + ]), + isFalse); + }); + + test('different lengths are unequal (guard)', () { + final eq = fieldsEqual>(same, absTol: 1); + expect( + eq([ + [1.0, 2.0] + ], [ + [1.0] + ]), + isFalse); + }); + + test('NaN elements equal each other', () { + final eq = fieldsEqual>(same); + expect( + eq([ + [double.nan, 1.0] + ], [ + [double.nan, 1.0] + ]), + isTrue); + expect( + eq([ + [double.nan] + ], [ + [1.0] + ]), + isFalse); + }); + + test('finite never equals non-finite inside a list (guard)', () { + final eq = fieldsEqual>(same, absTol: 1e300, relTol: 1); + expect( + eq([ + [1.0] + ], [ + [double.infinity] + ]), + isFalse); + }); + + test('int elements compare exactly', () { + final eq = fieldsEqual>(same, relTol: 1e-9, absTol: 1e-9); + expect( + eq([ + [9007199254740992] + ], [ + [9007199254740993] + ]), + isFalse); + expect( + eq([ + [9007199254740993] + ], [ + [9007199254740993] + ]), + isTrue); + }); + + test('null elements equal only null', () { + final eq = fieldsEqual>(same, absTol: 1); + expect( + eq([ + [null, 1.0] + ], [ + [null, 1.0] + ]), + isTrue); + expect( + eq([ + [null] + ], [ + [0.0] + ]), + isFalse); + }); + + test('nested lists recurse', () { + final eq = fieldsEqual>(same, absTol: 0.01); + expect( + eq([ + [ + [1.0, 2.0], + [3.0] + ] + ], [ + [ + [1.001, 2.0], + [3.0] + ] + ]), + isTrue); + expect( + eq([ + [ + [1.0, 2.0], + [3.0] + ] + ], [ + [ + [1.0, 2.0], + [3.5] + ] + ]), + isFalse); + }); + + test('a list never equals a non-list (guard)', () { + final eq = fieldsEqual>(same, absTol: 10, relTol: 1); + expect( + eq([ + [1.0] + ], [1.0]), + isFalse); + expect( + eq([1.0], [ + [1.0] + ]), + isFalse); + expect( + eq([ + [] + ], [null]), + isFalse); + }); + + test('empty lists are equal', () { + final eq = fieldsEqual>(same); + expect(eq([[]], [[]]), isTrue); + expect(eq([Float64List(0)], [[]]), isTrue); + }); + }); + }); +} diff --git a/test/aggregate/lookback_test.dart b/test/aggregate/lookback_test.dart new file mode 100644 index 0000000..4d9c45a --- /dev/null +++ b/test/aggregate/lookback_test.dart @@ -0,0 +1,255 @@ +// Lookback(k, inner): the inner aggregator sees every length-k window of the +// whole batch exactly once, in order, however the batch is sliced or chunked. +// +// The probe inner records each window it is shown. Per the contract, inner +// .addSlice(s, b, start, end) must itself handle the windows ending at rows +// start+k-1 .. end-1 (reading rows i-k+1..i of b), so the recorder does that +// and nothing else. Row values are their own index, so a window identifies +// itself. + +import 'package:openstrap_analytics/aggregate.dart'; +import 'package:test/test.dart'; + +import '../support/aggregator_laws.dart'; +import '../support/fold_law_support.dart' show foldBounds; +import '../support/property.dart' show Rng; + +typedef _Windows = List>; + +class _Record extends Aggregator<_Windows, DoubleColumn, _Windows> { + const _Record(this.k); + final int k; + @override + _Windows empty() => >[]; + @override + _Windows addSlice(_Windows s, DoubleColumn b, int start, int end) { + for (var i = start + k - 1; i < end; i++) { + s.add([for (var j = i - k + 1; j <= i; j++) b.values[j]]); + } + return s; + } + + @override + _Windows combine(_Windows left, _Windows right) => left..addAll(right); + @override + _Windows finish(_Windows s) => [for (final w in s) List.of(w)]; +} + +Lookback _lookback(int k) => + Lookback(k, _Record(k)); + +/// n rows whose values are 0, 1, 2, ... (distinct, so windows identify). +DoubleColumn _iota(int n) => col([for (var i = 0; i < n; i++) i * 1.0]); + +_Windows _expected(int n, int k) => [ + for (var i = k - 1; i < n; i++) + [for (var j = i - k + 1; j <= i; j++) j * 1.0] + ]; + +bool _same(Object? a, Object? b) => a.toString() == b.toString(); + +void main() { + for (final k in [1, 2, 3, 4]) { + group('k = $k', () { + final lb = _lookback(k); + + test('a single slice sees every window of the batch', () { + for (final n in [0, 1, k - 1, k, k + 1, 2 * k, 17]) { + if (n < 0) continue; + expect(lb.run(_iota(n)), _expected(n, k), reason: 'n=$n'); + } + }); + + test('a fold range sees only windows inside the range', () { + final c = _iota(12); + final got = lb.finish(lb.fold(c, 3, 9)); + expect(got, [ + for (var i = 3 + k - 1; i < 9; i++) + [for (var j = i - k + 1; j <= i; j++) j * 1.0] + ]); + }); + + test('every single split point (combine)', () { + final c = _iota(14); + for (var s = 0; s <= 14; s++) { + final r = lb.finish(lb.combine(lb.fold(c, 0, s), lb.fold(c, s, 14))); + expect(r, _expected(14, k), reason: 'split at $s'); + } + }); + + test('every two-slice cut on the slicing path (addSlice after data)', () { + final c = _iota(14); + for (var s = 0; s <= 14; s++) { + var st = lb.empty(); + st = lb.addSlice(st, c, 0, s); + st = lb.addSlice(st, c, s, 14); + expect(lb.finish(st), _expected(14, k), reason: 'cut at $s'); + } + }); + + test('every three-slice cut on both paths', () { + final c = _iota(10); + for (var a = 0; a <= 10; a++) { + for (var b = a; b <= 10; b++) { + final bounds = [0, a, b, 10]; + expect(runChunked(lb, c, bounds), _expected(10, k), + reason: 'chunked $bounds'); + expect(runChunkedRight(lb, c, bounds), _expected(10, k), + reason: 'right $bounds'); + expect(runSliced(lb, c, bounds), _expected(10, k), + reason: 'sliced $bounds'); + expect(runChunked(lb, c, bounds, copy: true), _expected(10, k), + reason: 'copies $bounds'); + } + } + }); + + test('random slicings and chunkings, empty chunks included', () { + final r = Rng(0x10B0 + k); + for (var t = 0; t < 80; t++) { + final n = r.intIn(0, 40); + final c = _iota(n); + final b = foldBounds(n, r.intIn(0, 9), r.nextInt(1 << 20), + special: [0, 1, 2, 3, 4]); + final want = _expected(n, k); + expect(runChunked(lb, c, b), want, reason: 'chunked $b n=$n'); + expect(runChunked(lb, c, b, copy: true), want, reason: 'copy $b n=$n'); + expect(runChunkedRight(lb, c, b), want, reason: 'right $b n=$n'); + expect(runSliced(lb, c, b), want, reason: 'sliced $b n=$n'); + } + }); + + test('one row per chunk and per slice', () { + for (final n in [0, 1, 2, 3, 9]) { + final c = _iota(n); + final b = singletonBounds(n); + expect(runChunked(lb, c, b), _expected(n, k), reason: 'chunks n=$n'); + expect(runChunked(lb, c, b, copy: true), _expected(n, k), + reason: 'copies n=$n'); + expect(runChunkedRight(lb, c, b), _expected(n, k), + reason: 'right n=$n'); + expect(runSliced(lb, c, b), _expected(n, k), reason: 'slices n=$n'); + } + }); + + test('empty chunks around and between data change nothing', () { + final c = _iota(8); + final want = _expected(8, k); + expect(runChunked(lb, c, [0, 0, 0, 3, 3, 3, 8, 8]), want); + expect(runSliced(lb, c, [0, 0, 3, 3, 8, 8]), want); + expect( + lb.finish(lb.combine(lb.combine(lb.empty(), lb.fold(c)), lb.empty())), + want); + expect( + lb.finish(lb.combine(lb.fold(c, 0, 4), + lb.combine(lb.empty(), lb.fold(c, 4, 8)))), + want); + }); + + test('is lawful', () { + expectLawful(lb, [for (var n = 0; n <= 12; n++) _iota(n), _iota(40)], + equals: _same); + }); + }); + } + + group('chunks shorter than k-1 rows', () { + test('k = 4: all chunks of 1 or 2 rows, mixed with empties', () { + final lb = _lookback(4); + final c = _iota(11); + final want = _expected(11, 4); + for (final b in [ + [0, 1, 2, 4, 5, 6, 8, 9, 11], + [0, 2, 3, 3, 4, 6, 7, 9, 10, 11], + [0, 1, 1, 2, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11], + ]) { + expect(runChunked(lb, c, b), want, reason: '$b'); + expect(runChunked(lb, c, b, copy: true), want, reason: 'copy $b'); + expect(runChunkedRight(lb, c, b), want, reason: 'right $b'); + expect(runSliced(lb, c, b), want, reason: 'sliced $b'); + } + }); + + test('k = 3: a seam of two one-row chunks yields no window by itself', () { + final lb = _lookback(3); + final c = _iota(3); + expect(lb.finish(lb.combine(lb.fold(c, 0, 1), lb.fold(c, 1, 2))), isEmpty); + final s = lb.combine( + lb.combine(lb.fold(c, 0, 1), lb.fold(c, 1, 2)), lb.fold(c, 2, 3)); + expect(lb.finish(s), [ + [0.0, 1.0, 2.0] + ]); + }); + + test('k = 4: no window when the whole batch is shorter than k', () { + final lb = _lookback(4); + expect(lb.run(_iota(3)), isEmpty); + expect(runChunked(lb, _iota(3), [0, 1, 2, 3]), isEmpty); + expect(runSliced(lb, _iota(3), [0, 1, 2, 3]), isEmpty); + }); + }); + + group('seam batches are tiny', () { + test('combining two 50-row chunks feeds the inner one seam of 2*(k-1) rows', + () { + // The inner records the length of every slice it is handed. The two + // folds hand it 50 rows each; the combine hands it exactly one more + // slice: the whole seam batch, last(k-1) + first(k-1) = 4 rows. + final seen = []; + final probe = _Spy(3, seen); + final lb = Lookback(3, probe); + final c = _iota(100); + lb.combine(lb.fold(c, 0, 50), lb.fold(c, 50, 100)); + expect(seen, [50, 50, 4]); + }); + }); + + group('results of the inner pass through', () { + test('Lookback(2, absolute successive differences)', () { + final lb = Lookback, double>( + 2, _AbsDiffSum()); + final c = col([1, 5, 2, 2, 10]); + expect(lb.run(c), 4 + 3 + 0 + 8); + expect(lb.run(col([])), 0.0); + expect(lb.run(col([7])), 0.0); + expectLawful(lb, randomColumns(0x10B9, nanP: 0), + equals: fieldsEqual((x) => [x], relTol: 1e-12, absTol: 1e-9)); + }); + + test('composes with both and map', () { + final a = both( + Lookback(2, _Record(2)), + Count()).map((r) => (r.$1.length, r.$2)); + expect(a.run(_iota(5)), (4, 5)); + expect(a.finish(a.combine(a.fold(_iota(5), 0, 2), a.fold(_iota(5), 2, 5))), + (4, 5)); + }); + }); +} + +class _Spy extends _Record { + _Spy(super.k, this.sizes); + final List sizes; + @override + _Windows addSlice(_Windows s, DoubleColumn b, int start, int end) { + sizes.add(end - start); + return super.addSlice(s, b, start, end); + } +} + +class _AbsDiffSum extends Aggregator, DoubleColumn, double> { + @override + List empty() => [0.0]; + @override + List addSlice(List s, DoubleColumn b, int start, int end) { + for (var i = start + 1; i < end; i++) { + s[0] += (b.values[i] - b.values[i - 1]).abs(); + } + return s; + } + + @override + List combine(List l, List r) => l..[0] += r[0]; + @override + double finish(List s) => s[0]; +} diff --git a/test/aggregate/persistence_test.dart b/test/aggregate/persistence_test.dart new file mode 100644 index 0000000..8d2ca5f --- /dev/null +++ b/test/aggregate/persistence_test.dart @@ -0,0 +1,986 @@ +// Round 10: persistent aggregator states and versioned checkpoints. +// +// SHAPE DECISIONS (the spec left these open; the implementation follows them): +// +// * Non-finite doubles are encoded as the strings 'NaN', 'Infinity' and +// '-Infinity' (the same strings incremental_core.dart's _readDouble already +// accepts). Finite doubles stay JSON numbers; -0.0 stays -0.0. Decoders +// accept an int where a double is expected. A decoder rejects any other +// string, bool, null, list or map in a double slot. +// * Every map-shaped encoding writes every key it has, with null for an +// absent value, and the decoder REQUIRES every key (a missing key is a +// FormatException, not a default). Decoders throw FormatException (never +// TypeError / RangeError / StateError) for any malformed input. +// * ListBatch element codec: `RowCodec(Object? Function(A) encode, +// A Function(Object?) decode)` plus the named constructor +// `ListBatch.coded(List rows, RowCodec codec)`. A coded batch +// keeps its codec through copyRange/join; `toJson()` is the JSON list of +// encoded rows. `RowCodec.batchFromJson(Object?)` returns a coded +// ListBatch (FormatException if the input is not a list; the row decode +// function itself must throw FormatException on a bad row). A plain +// `ListBatch(rows).toJson()` throws UnsupportedError. +// * Every persistable batch type X (DoubleColumn, Beats, TimedValues, +// HrStream, Epochs, EpochSeries) has `Object? toJson()` and a +// `factory X.fromJson(Object? json)` (FormatException on malformed input). +// The JSON must survive jsonEncode (so NaN/inf use the string encoding). +// * First / Last / Lookback / Prefix take an optional NAMED parameter +// `batchFromJson` (`B Function(Object?)`): First({batchFromJson}), +// Last({batchFromJson}), Lookback(k, inner, {batchFromJson}), +// Prefix(k, head, tail, {batchFromJson}). Persistable only when given +// (and, for Lookback/Prefix, when the stored inner/tail is persistable; +// Prefix keeps no head state, so the head does not matter). +// Without batchFromJson, encodeState/decodeState throw UnsupportedError. +// * GroupBy encodes a JSON map with the int keys as decimal strings; a key +// that is not a decimal int is a FormatException. +// * `both(a, b)` encodes its state as the two-element list [first, second]; +// any other length or type is a FormatException. +// * Moments.encodeState is exactly RunningMoments.toJson (a map), and +// Moments.decodeState is RunningMoments.fromJson behind a FormatException +// guard. +// * Aggregator defaults: persistable false; encodeState/decodeState throw +// UnsupportedError; Batch.toJson throws UnsupportedError. +// * The checkpoint envelope lives in lib/src/aggregate/checkpoint.dart and is +// exported from package:openstrap_analytics/aggregate.dart. +// encodeCheckpoint on a non-persistable aggregator throws UnsupportedError. + +import 'dart:convert'; +import 'dart:typed_data'; + +import 'package:openstrap_analytics/aggregate.dart'; +import 'package:openstrap_analytics/onehz.dart' show RunningMoments; +import 'package:test/test.dart'; + +import '../support/aggregator_laws.dart'; +import '../support/property.dart' show Rng; + +const nan = double.nan; +const inf = double.infinity; + +Object? wire(Object? x) => jsonDecode(jsonEncode(x)); + +bool close(double a, double b) => + a == b || + (a.isNaN && b.isNaN) || + (a - b).abs() <= 1e-9 * [1.0, a.abs(), b.abs()].reduce((x, y) => x > y ? x : y); + +bool closeOpt(double? a, double? b) => + a == null || b == null ? a == b : close(a, b); + +int bits(double x) => (ByteData(8)..setFloat64(0, x)).getInt64(0); + +bool sameCol(DoubleColumn? a, DoubleColumn? b) { + if (a == null || b == null) return a == b; + if (a.length != b.length) return false; + for (var i = 0; i < a.length; i++) { + if (!close(a.values[i], b.values[i])) return false; + } + return true; +} + +/// Mixed columns: specials, NaN, infinities, signed zero, large and tiny. +List allCases({bool finiteOnly = false, bool moderate = false}) { + final r = Rng(0x10A); + final specials = finiteOnly + ? [-0.0, 0.0, 5e-324, 0.1 + 0.2, -123456.789] + : moderate + ? [-0.0, 0.0, 5e-324, 0.1 + 0.2, -123456.789] + : [-0.0, 0.0, 5e-324, 0.1 + 0.2, 1.7976931348623157e308, -inf, inf]; + double draw() => r.nextBool(.2) + ? specials[r.intIn(0, specials.length - 1)] + : (r.nextDouble() - .5) * 2000; + return [ + col([]), + col([1]), + col([-0.0]), + if (!finiteOnly) col([nan, nan]), + for (var i = 0; i < 24; i++) + col([ + for (var n = r.intIn(0, 30); n > 0; n--) + !finiteOnly && r.nextBool(.12) ? nan : draw() + ]), + ]; +} + +/// For every split: decode(wire(encode(fold [0,k)))) continued by combine and +/// by addSlice must agree with the one-pass result, as must the full state. +void expectResumes, R>( + Aggregator agg, List cases, bool Function(R, R) eq) { + expect(agg.persistable, isTrue); + S rt(S s) => agg.decodeState(wire(agg.encodeState(s))); + for (var c = 0; c < cases.length; c++) { + final b = cases[c]; + final n = b.length; + final want = agg.run(b); + expect(eq(agg.finish(rt(agg.fold(b))), want), isTrue, + reason: 'full state, case $c'); + expect(eq(agg.finish(rt(agg.empty())), agg.finish(agg.empty())), isTrue, + reason: 'empty state'); + for (var k = 0; k <= n; k++) { + final viaCombine = + agg.finish(agg.combine(rt(agg.fold(b, 0, k)), agg.fold(b, k, n))); + expect(eq(viaCombine, want), isTrue, reason: 'combine, case $c split $k'); + final viaAdd = agg.finish(agg.addSlice(rt(agg.fold(b, 0, k)), b, k, n)); + expect(eq(viaAdd, want), isTrue, reason: 'addSlice, case $c split $k'); + final viaLeft = + agg.finish(agg.combine(agg.fold(b, 0, k), rt(agg.fold(b, k, n)))); + expect(eq(viaLeft, want), isTrue, reason: 'right restored, case $c split $k'); + } + } +} + +/// Variants of a JSON value with one thing broken. Lists keep their length +/// (shorter lists are legitimate for variable-length encodings). +Iterable mutants(Object? node, {bool removeKeys = true}) sync* { + const repls = ['zzz', true, {'bogus': 1}]; + if (node is Map) { + for (final key in node.keys) { + if (removeKeys) yield {...node}..remove(key); + for (final r in repls) { + yield {...node, key: r}; + } + for (final v in mutants(node[key], removeKeys: removeKeys)) { + yield {...node, key: v}; + } + } + } else if (node is List) { + for (var i = 0; i < node.length; i++) { + for (final r in repls) { + yield [...node]..[i] = r; + } + for (final v in mutants(node[i], removeKeys: removeKeys)) { + yield [...node]..[i] = v; + } + } + } +} + +void expectMalformed, R>( + Aggregator agg, Iterable payloads) { + var n = 0; + for (final p in payloads) { + n++; + expect(() => agg.decodeState(p), throwsFormatException, + reason: 'payload ${jsonEncode(p)}'); + } + expect(n, greaterThan(0)); +} + +/// Mutants of a real mid-stream encoding plus generic junk. +void expectRejectsDamage, R>( + Aggregator agg, B batch, + {bool removeKeys = true, List junk = const []}) { + final valid = wire(agg.encodeState(agg.fold(batch, 0, batch.length ~/ 2))); + expectMalformed( + agg, [...mutants(valid, removeKeys: removeKeys), null, 'zzz', true, ...junk]); +} + +Float64List _values(DoubleColumn b) => b.values; +Float64List _rowsAsDoubles(ListBatch b) => + Float64List.fromList([for (final r in b.rows) r.toDouble()]); + +/// Test-defined persistable aggregator: for each length-k window, adds the +/// first element. State is a plain finite double. +class _WindowFirst extends Aggregator { + const _WindowFirst(this.k); + final int k; + @override + double empty() => 0; + @override + double addSlice(double s, DoubleColumn b, int start, int end) { + for (var i = start + k - 1; i < end; i++) { + s += b.values[i - k + 1]; + } + return s; + } + + @override + double combine(double l, double r) => l + r; + @override + double finish(double s) => s; + @override + bool get persistable => true; + @override + Object? encodeState(double s) => s; + @override + double decodeState(Object? json) { + if (json is! num) throw const FormatException('number expected'); + return json.toDouble(); + } +} + +/// No persistence at all. +class _Plain extends Aggregator { + const _Plain(); + @override + int empty() => 0; + @override + int addSlice(int s, DoubleColumn b, int start, int end) => s + end - start; + @override + int combine(int l, int r) => l + r; + @override + int finish(int s) => s; +} + +typedef _IB = ListBatch; + +final _intCodec = RowCodec((v) => v, (o) { + if (o is! int) throw const FormatException('int expected'); + return o; +}); + +_IB _ib(List rows) => ListBatch.coded(rows, _intCodec); + +/// Sums rows and counts them; finish mixes both, so losing either shows. +/// With [dropCount] the codec silently forgets the count. +class _SumCount extends Aggregator, _IB, int> { + const _SumCount({this.dropCount = false, this.persists = true}); + final bool dropCount; + final bool persists; + @override + List empty() => [0, 0]; + @override + List addSlice(List s, _IB b, int start, int end) { + for (var i = start; i < end; i++) { + s[0] += b.rows[i]; + } + s[1] += end - start; + return s; + } + + @override + List combine(List l, List r) => + [l[0] + r[0], l[1] + r[1]]; + @override + int finish(List s) => s[0] * 1000 + s[1]; + @override + bool get persistable => persists; + @override + Object? encodeState(List s) => dropCount ? [s[0]] : [s[0], s[1]]; + @override + List decodeState(Object? json) { + if (json is! List || json.isEmpty || json.first is! int) { + throw const FormatException('bad'); + } + return [json.first as int, dropCount ? 0 : json[1] as int]; + } +} + +void main() { + group('non-persistable defaults', () { + test('an aggregator that overrides nothing is not persistable', () { + const a = _Plain(); + expect(a.persistable, isFalse); + expect(() => a.encodeState(0), throwsUnsupportedError); + expect(() => a.decodeState(0), throwsUnsupportedError); + }); + test('plain ListBatch and Batch.toJson throw UnsupportedError', () { + expect(() => ListBatch([1, 2]).toJson(), throwsUnsupportedError); + }); + }); + + group('persistable flags', () { + final sum = Sum(_values); + test('every primitive block is persistable', () { + expect(const Count().persistable, isTrue); + expect(CountFinite(_values).persistable, isTrue); + expect(CountWhere(_values, (v) => v > 0).persistable, isTrue); + expect(sum.persistable, isTrue); + expect(SumOf(_values, (v) => v * 2).persistable, isTrue); + expect(SumSq(_values).persistable, isTrue); + expect(Min(_values).persistable, isTrue); + expect(Max(_values).persistable, isTrue); + expect(Moments(_values).persistable, isTrue); + expect(Ewma(_values, .3).persistable, isTrue); + }); + test('First/Last/Lookback/Prefix are persistable only with batchFromJson', + () { + expect(First().persistable, isFalse); + expect(Last().persistable, isFalse); + expect( + First(batchFromJson: DoubleColumn.fromJson).persistable, + isTrue); + expect( + Last(batchFromJson: DoubleColumn.fromJson).persistable, + isTrue); + expect(Lookback(3, const _WindowFirst(3)) + .persistable, isFalse); + expect( + Lookback(3, const _WindowFirst(3), + batchFromJson: DoubleColumn.fromJson) + .persistable, + isTrue); + expect( + Prefix( + 2, const Count(), const _WindowFirst(1)) + .persistable, + isFalse); + expect( + Prefix( + 2, const Count(), _WindowFirst(1), + batchFromJson: DoubleColumn.fromJson) + .persistable, + isTrue); + expect(() => First().encodeState(First().empty()), + throwsUnsupportedError); + }); + test('a composite with a non-persistable child is not persistable', () { + const plain = _Plain(); + expect(both(sum, plain).persistable, isFalse); + expect(both(plain, sum).persistable, isFalse); + expect(both(sum, Count()).persistable, isTrue); + expect(plain.map((r) => r).persistable, isFalse); + expect(sum.map((r) => r).persistable, isTrue); + expect(GroupBy((b, i) => 0, plain).persistable, + isFalse); + expect( + GroupBy((b, i) => 0, const Count()).persistable, + isTrue); + expect( + Lookback(2, plain, + batchFromJson: DoubleColumn.fromJson) + .persistable, + isFalse); + expect( + Prefix(2, plain, plain, + batchFromJson: DoubleColumn.fromJson) + .persistable, + isFalse); + expect(() => both(sum, plain).encodeState(both(sum, plain).empty()), + throwsUnsupportedError); + }); + }); + + group('double encoding', () { + final agg = SumSq(_values); + test('non-finite doubles are the documented strings', () { + expect(agg.encodeState(nan), 'NaN'); + expect(agg.encodeState(inf), 'Infinity'); + expect(agg.encodeState(-inf), '-Infinity'); + expect(agg.decodeState('NaN'), isNaN); + expect(agg.decodeState('Infinity'), inf); + expect(agg.decodeState('-Infinity'), -inf); + }); + test('finite doubles round-trip bit-exactly through JSON text', () { + final r = Rng(7); + final xs = [ + 0.0, -0.0, 5e-324, -5e-324, 2.2250738585072014e-308, + 1.7976931348623157e308, -1.7976931348623157e308, 0.1 + 0.2, 1 / 3, + 9007199254740993.0, 123456789.123456789, + for (var i = 0; i < 200; i++) + (r.nextDouble() - .5) * (i.isEven ? 1e-300 : 1e300), + ]; + for (final x in xs) { + final back = agg.decodeState(wire(agg.encodeState(x))); + expect(bits(back), bits(x), reason: '$x'); + } + expect(agg.decodeState(wire(agg.encodeState(-0.0))).isNegative, isTrue); + }); + test('NaN and infinities survive jsonEncode (no JsonUnsupportedObjectError)', + () { + for (final x in [nan, inf, -inf]) { + final text = jsonEncode(agg.encodeState(x)); + final back = agg.decodeState(jsonDecode(text)); + expect(bits(back) == bits(x) || (x.isNaN && back.isNaN), isTrue); + } + }); + test('an int is accepted where a double is expected', () { + final v = agg.decodeState(3); + expect(v, 3.0); + expect(v, isA()); + expect(agg.decodeState(0), 0.0); + }); + test('malformed doubles', () { + expectMalformed(agg, [ + null, 'zzz', 'nan', 'inf', 'Infinit', '', '1.5', true, false, [], [1], + {}, {'v': 1} + ]); + }); + test('Min / Max state is a nullable double', () { + final mn = Min(_values); + final mx = Max(_values); + expect(mn.encodeState(null), isNull); + expect(mn.decodeState(null), isNull); + expect(mx.decodeState(null), isNull); + expect(mn.encodeState(-inf), '-Infinity'); + expect(mx.encodeState(inf), 'Infinity'); + expect(mn.decodeState(wire(mn.encodeState(-0.0)))!.isNegative, isTrue); + expectMalformed(mn, ['zzz', true, [], {}, 'nan']); + expectMalformed(mx, ['zzz', true, [], {}, 'nan']); + }); + }); + + group('counts', () { + final cs = >[ + const Count(), + CountFinite(_values), + CountWhere(_values, (v) => v > 0), + ]; + for (final (i, a) in cs.indexed) { + test('block $i round-trips on every split', () { + expectResumes(a, allCases(), (x, y) => x == y); + }); + test('block $i large ints are exact', () { + for (final big in [9007199254740993, 1 << 62, 9223372036854775807, 0]) { + expect(a.decodeState(wire(a.encodeState(big))), big); + } + }); + test('block $i malformed', () { + expectMalformed(a, [ + null, 'x', '5', 1.5, 2.0, -1, true, [], [1], {}, {'count': 1}, nan.toString() + ]); + }); + } + }); + + group('sums', () { + final aggs = >{ + 'Sum': Sum(_values), + 'SumOf': SumOf(_values, (v) => v * 2 - 1), + 'SumSq': SumSq(_values), + }; + aggs.forEach((name, a) { + test('$name round-trips on every split (moderate finite values)', () { + expectResumes(a, allCases(moderate: true), close); + }); + test('$name keeps +/-inf, NaN-from-cancellation and signed zero', () { + expectResumes(a, [ + col([1, inf, 2]), + col([-inf, 4]), + col([inf, -inf]), + col([-0.0]), + col([nan, 3, nan]), + ], close); + }); + test('$name empty state', () { + expect(a.finish(a.decodeState(wire(a.encodeState(a.empty())))), 0.0); + }); + test('$name malformed', () { + expectMalformed(a, [null, 'zzz', true, [], {}, 'nan']); + }); + }); + test('compensation survives the round trip', () { + final a = Sum(_values); + final b = col([1e16, 1.0, -1e16, 1.0, 1e-3, 1e16, 1.0, -1e16]); + final whole = a.run(b); + final half = a.fold(b, 0, 4); + final back = a.decodeState(wire(a.encodeState(half))); + expect(a.finish(a.addSlice(back, b, 4, 8)), whole); + }); + test('Sum and SumOf damaged mid-stream encodings are rejected', () { + final b = col([1.5, 2.5, 1e-9, 4, 5]); + expectRejectsDamage(Sum(_values), b); + expectRejectsDamage(SumOf(_values, (v) => v), b); + }); + }); + + group('min / max', () { + test('round trip on every split', () { + expectResumes(Min(_values), allCases(), closeOpt); + expectResumes(Max(_values), allCases(), closeOpt); + }); + test('infinite extrema and empty', () { + expectResumes(Min(_values), [col([3, -inf, 1]), col([])], closeOpt); + expectResumes(Max(_values), [col([3, inf, 1]), col([nan])], closeOpt); + }); + }); + + group('moments', () { + final a = Moments(_values); + bool eq(RunningMoments x, RunningMoments y) => + x.count == y.count && + closeOpt(x.mean, y.mean) && + closeOpt(x.sampleSd, y.sampleSd) && + closeOpt(x.populationSd, y.populationSd); + test('round trip on every split', () { + expectResumes(a, allCases(finiteOnly: true), eq); + }); + test('the encoding is RunningMoments.toJson', () { + final s = a.fold(col([1, 2, 4, 8])); + expect(a.encodeState(s), s.toJson()); + expect(a.encodeState(a.empty()), RunningMoments().toJson()); + }); + group('reachable non-finite moments (finite input overflowing m2)', () { + final over = col([1e155, -1e155]); // squared deviation overflows to inf + test('the state reaches Infinity from finite input', () { + expect(a.fold(over).populationSd, double.infinity); + }); + test('review case: encodeState survives jsonEncode and decodes back', () { + final state = a.fold(over); + late final String text; + expect(() => text = jsonEncode(a.encodeState(state)), returnsNormally); + final back = a.decodeState(jsonDecode(text)); + expect(back.count, 2); + expect(back.mean, state.mean); + expect(back.sampleSd, double.infinity); + expect(eq(a.finish(back), a.finish(state)), isTrue); + }); + test('the non-finite m2 uses the shared string encoding', () { + final json = wire(a.encodeState(a.fold(over))) as Map; + expect(json['m2'], 'Infinity'); + expect(json['count'], 2); + }); + // Split invariance once m2 has overflowed is out of domain (Welford's + // merge of an infinite m2 is not defined to match the one-pass fold); + // what must hold is that the reachable state round-trips, tested above. + test('a non-finite m2 is accepted only in the encoded string form', () { + final good = Map.of( + wire(a.encodeState(a.fold(over))) as Map); + expect(() => a.decodeState({...good, 'm2': 'Infinity'}), + returnsNormally); + for (final bad in [ + 'NaN', '-Infinity', 'inf', 'infinity', null, true, [], {}, + ]) { + expect(() => a.decodeState({...good, 'm2': bad}), + throwsFormatException, + reason: 'm2 ${jsonEncode(bad)}'); + } + }); + test('count and invariants are still validated with an infinite m2', () { + final good = Map.of( + wire(a.encodeState(a.fold(over))) as Map); + expectMalformed(a, [ + {...good, 'count': 1}, // one row cannot have spread + {...good, 'count': 0}, // nor none + {...good, 'count': -2}, + {...good, 'count': 1.5}, + {...good, 'origin': 'NaN'}, + {...good, 'origin': 'Infinity'}, + {...good, 'meanOffset': 'NaN'}, + {...good, 'version': 2}, + ]); + }); + }); + + test('malformed and inconsistent', () { + final good = Map.of( + wire(a.encodeState(a.fold(col([1, 2, 4])))) as Map); + expectMalformed(a, [ + null, 'zzz', 3, true, [], {}, + ...mutants(good), + {...good, 'count': -1}, + {...good, 'count': 1.5}, + {...good, 'version': 2}, + {...good, 'type': 'IntHistogram'}, + {...good, 'm2': -1.0}, + {...good, 'count': 0}, // origin/m2 non-zero with count 0 + {...good, 'count': 1}, // one row but m2 non-zero + {...good, 'origin': 'NaN'}, + ]); + }); + }); + + group('first / last', () { + final r = Rng(21); + final cases = [ + col([]), + col([nan]), + col([-0.0, 1, 2]), + col([inf, nan, -inf, 4]), + for (var i = 0; i < 6; i++) + col([for (var n = r.intIn(0, 9); n > 0; n--) r.nextDouble()]), + ]; + test('DoubleColumn batches (state holds a one-row copy)', () { + expectResumes(First(batchFromJson: DoubleColumn.fromJson), + cases, sameCol); + expectResumes(Last(batchFromJson: DoubleColumn.fromJson), + cases, sameCol); + }); + test('coded ListBatch batches', () { + bool eq(_IB? a, _IB? b) => + a == null || b == null ? a == b : a.rows.join(',') == b.rows.join(','); + final cs = [_ib([]), _ib([7]), _ib([1, 2, 3, 4]), _ib([-5, 9])]; + expectResumes(First<_IB>(batchFromJson: _intCodec.batchFromJson), cs, eq); + expectResumes(Last<_IB>(batchFromJson: _intCodec.batchFromJson), cs, eq); + }); + test('malformed', () { + final f = First(batchFromJson: DoubleColumn.fromJson); + final l = Last(batchFromJson: DoubleColumn.fromJson); + for (final a in [f, l]) { + expectRejectsDamage(a, col([1, 2, 3, 4])); + expectMalformed(a, [3, [], [1, 2], 'NaN']); + } + }); + }); + + group('group by', () { + _IB ints(List r) => _ib(r); + int key(_IB b, int i) => b.rows[i] ~/ 10; + final counts = GroupBy<_IB, int, int>(key, const Count<_IB>()); + final sums = GroupBy<_IB, dynamic, double>( + key, Sum<_IB>(_rowsAsDoubles)); + final cases = [ + ints([]), + ints([5]), + ints([1, 2, 11, 12, 13, 1, 2, -15, -16, 25]), + ints([for (var i = 0; i < 60; i++) (i * 7919) % 97 - 30]), + ints([1 << 40, 1 << 40, -(1 << 40), 3]), + ]; + test('round trip on every split (Count and Sum inner)', () { + expectResumes(counts, cases, (a, b) => a.toString() == b.toString()); + expectResumes( + sums, + cases, + (a, b) => + a.length == b.length && + a.keys.every((k) => b.containsKey(k) && close(a[k]!, b[k]!))); + }); + test('int keys are encoded as strings, huge and negative keys survive', () { + final big = GroupBy<_IB, int, int>( + (b, i) => b.rows[i], const Count<_IB>()); + final s = big.fold(ints([1 << 62, -7, -(1 << 62), 0, 9223372036854775807])); + final enc = big.encodeState(s) as Map; + expect(enc.keys.every((k) => k is String), isTrue); + final back = big.decodeState(wire(enc)); + expect(back.keys.toSet(), + {1 << 62, -7, -(1 << 62), 0, 9223372036854775807}); + }); + test('empty state is an empty map', () { + expect(counts.encodeState(counts.empty()), {}); + expect(counts.decodeState({}), isEmpty); + }); + test('malformed', () { + expectMalformed(counts, [ + null, 'zzz', 3, true, [], [1], + {'abc': 1}, {'1.5': 1}, {'': 1}, {'1': 'x'}, {'1': 1.5}, {'1': -1}, + {'1': null}, {'1': {}}, + ]); + final valid = wire(counts.encodeState(counts.fold(cases[2]))) as Map; + expectMalformed(counts, mutants(valid, removeKeys: false)); + }); + }); + + group('lookback', () { + DoubleColumn iota(int n) => col([for (var i = 0; i < n; i++) i * 1.5 + .25]); + for (final k in [1, 2, 3, 5]) { + final lb = Lookback(k, _WindowFirst(k), + batchFromJson: DoubleColumn.fromJson); + test('k=$k resumes mid-stream (edges and inner state restored)', () { + expectResumes(lb, [for (final n in [0, 1, 2, 4, 5, 6, 13]) iota(n)], + close); + }); + test('k=$k damaged encodings are rejected', () { + expectRejectsDamage(lb, iota(9)); + expectRejectsDamage(lb, iota(1)); + }); + } + test('a resumed seam equals the unbroken run for random cuts', () { + final r = Rng(5); + final lb = Lookback(4, const _WindowFirst(4), + batchFromJson: DoubleColumn.fromJson); + final b = col([for (var i = 0; i < 80; i++) r.nextDouble()]); + var s = lb.empty(); + var at = 0; + while (at < 80) { + final to = (at + r.intIn(0, 9)).clamp(0, 80); + s = lb.decodeState(wire(lb.encodeState(lb.addSlice(s, b, at, to)))); + at = to; + } + expect(close(lb.finish(s), lb.run(b)), isTrue); + }); + test('without batchFromJson encode/decode throw UnsupportedError', () { + final lb = Lookback(3, const _WindowFirst(3)); + expect(() => lb.encodeState(lb.empty()), throwsUnsupportedError); + expect(() => lb.decodeState(null), throwsUnsupportedError); + }); + }); + + group('prefix', () { + for (final k in [0, 1, 2, 5]) { + final p = Prefix( + k, const Count(), const _WindowFirst(1), + batchFromJson: DoubleColumn.fromJson); + bool eq((int, double) a, (int, double) b) => a.$1 == b.$1 && close(a.$2, b.$2); + test('k=$k resumes mid-stream', () { + expectResumes(p, [for (final n in [0, 1, 2, 3, 4, 7, 12]) col([for (var i = 0; i < n; i++) i * 2.5 - 3])], + eq); + }); + test('k=$k damaged encodings are rejected', () { + expectRejectsDamage(p, col([for (var i = 0; i < 12; i++) i * 1.5])); + expectRejectsDamage(p, col([1, 2])); + }); + } + }); + + group('ewma', () { + final a = Ewma(_values, .3); + bool eq(Affine x, Affine y) => close(x.scale, y.scale) && close(x.offset, y.offset); + test('round trip on every split', () { + expectResumes(a, allCases(moderate: true), eq); + }); + test('identity and non-finite maps survive', () { + expectResumes(a, [col([]), col([nan, nan]), col([inf, 1]), col([-0.0])], eq); + final id = a.decodeState(wire(a.encodeState(a.empty()))); + expect(a.finish(id).scale, 1.0); + expect(a.finish(id).offset, 0.0); + }); + test('malformed', () { + expectRejectsDamage(a, col([1, 2, 3, 4])); + expectMalformed(a, [3, [], {}]); + }); + }); + + group('both and map', () { + final sum = Sum(_values); + final pair = both(const Count(), sum); + test('both round trips on every split', () { + expectResumes(pair, allCases(moderate: true), + ((int, double) a, (int, double) b) => a.$1 == b.$1 && close(a.$2, b.$2)); + }); + test('both state is a two-element list', () { + final enc = pair.encodeState(pair.fold(col([1, 2, 3]))); + expect(enc, isA()); + expect((enc as List).length, 2); + expect(enc.first, 3); + }); + test('both malformed', () { + final valid = wire(pair.encodeState(pair.fold(col([1, 2, 3])))) as List; + expectMalformed(pair, [ + null, 'zzz', 3, true, {}, [], [1], [1, 2, 3], [valid[0]], + ['x', valid[1]], [valid[0], 'x'], [-1, valid[1]], + ...mutants(valid, removeKeys: false), + ]); + }); + test('map delegates to the inner codec', () { + final m = const Count().map((c) => c.toString()); + expect(m.persistable, isTrue); + expect(m.encodeState(42), const Count().encodeState(42)); + expect(m.decodeState(wire(m.encodeState(42))), 42); + expectResumes(m, allCases(), (a, b) => a == b); + expectMalformed(m, [null, 'x', -1, 1.5]); + }); + }); + + group('batch codecs', () { + test('DoubleColumn round-trips non-finite and signed-zero values', () { + final b = col([nan, inf, -inf, -0.0, 0.0, 5e-324, 0.1 + 0.2, 1e308]); + final back = DoubleColumn.fromJson(wire(b.toJson())); + expect(back.length, b.length); + for (var i = 0; i < b.length; i++) { + expect(bits(back.values[i]) == bits(b.values[i]) || + (b.values[i].isNaN && back.values[i].isNaN), isTrue, reason: '$i'); + } + expect(DoubleColumn.fromJson(wire(col([]).toJson())).length, 0); + }); + test('DoubleColumn accepts ints and rejects junk', () { + expect(DoubleColumn.fromJson([1, 2.5, 'NaN']).values[0], 1.0); + for (final bad in [ + null, 'zzz', 3, true, {}, {'v': []}, [true], ['zzz'], [null], [[]], ['nan'] + ]) { + expect(() => DoubleColumn.fromJson(bad), throwsFormatException, + reason: jsonEncode(bad)); + } + }); + test('coded ListBatch keeps its codec through copyRange and join', () { + final b = _ib([1, 2, 3, 4]); + expect(b.toJson(), [1, 2, 3, 4]); + expect(b.copyRange(1, 3).toJson(), [2, 3]); + expect(b.copyRange(0, 1).join(b.copyRange(2, 4)).toJson(), [1, 3, 4]); + final back = _intCodec.batchFromJson(wire(b.toJson())); + expect(back.rows, [1, 2, 3, 4]); + expect(back.toJson(), [1, 2, 3, 4]); + expect(_intCodec.batchFromJson([]).rows, isEmpty); + }); + test('coded ListBatch rejects malformed input', () { + for (final bad in [null, 'zzz', 3, {}, true, [1, 'x'], [null], [1.5]]) { + expect(() => _intCodec.batchFromJson(bad), throwsFormatException, + reason: jsonEncode(bad)); + } + }); + + final specials = Float64List.fromList([nan, inf, -inf, -0.0, 0.0, 1e-310, 2.5]); + Float64List sp(int n) => Float64List.fromList( + [for (var i = 0; i < n; i++) specials[i % specials.length]]); + bool same(Float64List a, Float64List b) { + if (a.length != b.length) return false; + for (var i = 0; i < a.length; i++) { + if (!(bits(a[i]) == bits(b[i]) || (a[i].isNaN && b[i].isNaN))) return false; + } + return true; + } + + test('Beats', () { + for (final n in [0, 1, 7]) { + final b = Beats(sp(n), sp(n + 3).sublist(3)); + final back = Beats.fromJson(wire(b.toJson())); + expect(same(back.nnMs, b.nnMs) && same(back.timeMs, b.timeMs), isTrue); + } + }); + test('TimedValues', () { + final b = TimedValues(sp(7), sp(7).sublist(0, 7)); + final back = TimedValues.fromJson(wire(b.toJson())); + expect(same(back.tHours, b.tHours) && same(back.y, b.y), isTrue); + }); + test('HrStream', () { + final b = HrStream(sp(7), sp(7)); + final back = HrStream.fromJson(wire(b.toJson())); + expect(same(back.tsMs, b.tsMs) && same(back.hr, b.hr), isTrue); + }); + test('EpochSeries keeps firstIndex', () { + final b = EpochSeries(sp(7), firstIndex: 1440); + final back = EpochSeries.fromJson(wire(b.toJson())); + expect(same(back.x, b.x), isTrue); + expect(back.firstIndex, 1440); + expect(EpochSeries.fromJson(wire(EpochSeries(sp(0)).toJson())).length, 0); + }); + test('Epochs keeps the mask (or its absence) and firstIndex', () { + final plain = Epochs(Uint8List.fromList([1, 0, 1, 1]), firstIndex: 96); + final p = Epochs.fromJson(wire(plain.toJson())); + expect(p.asleep, [1, 0, 1, 1]); + expect(p.valid, isNull); + expect(p.firstIndex, 96); + final masked = Epochs(Uint8List.fromList([1, 0, 1]), + valid: Uint8List.fromList([1, 1, 0])); + final m = Epochs.fromJson(wire(masked.toJson())); + expect(m.asleep, [1, 0, 1]); + expect(m.valid, [1, 1, 0]); + expect(m.firstIndex, 0); + expect(Epochs.fromJson(wire(Epochs(Uint8List(0)).toJson())).length, 0); + }); + + test('malformed batch payloads and mismatched columns', () { + final beats = Beats(sp(4), sp(4)); + final tv = TimedValues(sp(4), sp(4)); + final hr = HrStream(sp(4), sp(4)); + final es = EpochSeries(sp(4), firstIndex: 3); + final ep = Epochs(Uint8List.fromList([1, 0, 1]), + valid: Uint8List.fromList([1, 1, 0]), firstIndex: 5); + final epPlain = Epochs(Uint8List.fromList([1, 0, 1]), firstIndex: 5); + final cases = <(String, Object? Function(), Object? Function(Object?))>[ + ('Beats', beats.toJson, Beats.fromJson), + ('TimedValues', tv.toJson, TimedValues.fromJson), + ('HrStream', hr.toJson, HrStream.fromJson), + ('EpochSeries', es.toJson, EpochSeries.fromJson), + ('Epochs', ep.toJson, Epochs.fromJson), + ('Epochs (no mask)', epPlain.toJson, Epochs.fromJson), + ]; + for (final (name, toJson, from) in cases) { + final valid = wire(toJson()); + for (final bad in [null, 'zzz', 3, true, [], {}, ...mutants(valid)]) { + expect(() => from(bad), throwsFormatException, + reason: '$name ${jsonEncode(bad)}'); + } + } + // Two-column batches: shortening one column is an inconsistency. + for (final (name, toJson, from) in cases.where( + (c) => c.$1 != 'EpochSeries' && c.$1 != 'Epochs (no mask)')) { + final valid = wire(toJson()) as Map; + for (final key in valid.keys.where((k) => valid[k] is List)) { + final cut = { + ...valid, + key: (valid[key] as List).sublist(0, (valid[key] as List).length - 1) + }; + expect(() => from(cut), throwsFormatException, reason: '$name $key'); + } + } + }); + }); + + group('roundTrip law (harness self-test)', () { + final cases = [ + _ib([]), + _ib([4]), + _ib([1, 2, 3, 4, 5]), + _ib([9, 8, 7]), + ]; + Set laws(List v) => {for (final x in v) x.law}; + + test('a faithful codec has no violations', () { + expect(checkLaws(const _SumCount(), cases), isEmpty); + }); + test('a codec that drops a field is caught as exactly {roundTrip}', () { + final v = checkLaws(const _SumCount(dropCount: true), cases); + expect(laws(v), {'roundTrip'}); + expect(v.every((x) => x.detail.isNotEmpty), isTrue); + }); + test('a non-persistable aggregator is not asked to round trip', () { + expect(checkLaws(const _SumCount(dropCount: true, persists: false), cases), + isEmpty); + }); + test('checkConversion reports it as law:roundTrip', () { + final f = checkConversion, _IB, int>( + (_) => const _SumCount(dropCount: true), + cases, + (b) => b.rows.fold(0, (s, x) => s + x) * 1000 + b.length); + expect({for (final x in f) x.path}, {'law:roundTrip'}); + }); + test('expectLawful fails for the lossy codec and passes for the good one', + () { + expectLawful(const _SumCount(), cases); + expect(() => expectLawful(const _SumCount(dropCount: true), cases), + throwsA(isA())); + }); + test('the built-in blocks pass the new law', () { + expect(checkLaws(Sum(_values), allCases(moderate: true), equals: close), isEmpty); + expect(checkLaws(const Count(), allCases()), isEmpty); + }); + }); + + group('checkpoint envelope', () { + final sum = Sum(_values); + final b = col([1.5, nan, 2.25, 1e-3, 4]); + test('encodes kind, version and state', () { + final env = encodeCheckpoint(const Count(), 5, + kind: 'k', version: 2); + expect(env, {'kind': 'k', 'version': 2, 'state': 5}); + }); + test('resume through JSON text continues to the one-pass result', () { + final env = encodeCheckpoint(sum, sum.fold(b, 0, 2), + kind: 'sum', version: 1); + final state = decodeCheckpoint(sum, wire(env), + kind: 'sum', version: 1); + expect(state, isNotNull); + expect(close(sum.finish(sum.addSlice(state, b, 2, 5)), sum.run(b)), isTrue); + }); + test('a non-finite state still encodes to JSON text', () { + final a = SumSq(_values); + final env = encodeCheckpoint(a, inf, kind: 'x', version: 1); + expect(decodeCheckpoint(a, wire(env), kind: 'x', version: 1), inf); + }); + test('kind or version mismatch refuses with null', () { + final env = wire(encodeCheckpoint(sum, sum.fold(b), kind: 'a', version: 3)); + expect(decodeCheckpoint(sum, env, kind: 'b', version: 3), isNull); + expect(decodeCheckpoint(sum, env, kind: 'a', version: 2), isNull); + expect(decodeCheckpoint(sum, env, kind: 'a', version: 4), isNull); + expect(decodeCheckpoint(sum, env, kind: 'a', version: 3), isNotNull); + }); + test('malformed state or envelope refuses with null, never throws', () { + const c = Count(); + Map env(Object? state) => + {'kind': 'k', 'version': 1, 'state': state}; + for (final bad in [ + env('zzz'), env(-1), env(1.5), env(null), env([]), + {'kind': 'k', 'version': 1}, + {'kind': 'k', 'state': 1}, + {'version': 1, 'state': 1}, + {'kind': 7, 'version': 1, 'state': 1}, + {'kind': 'k', 'version': '1', 'state': 1}, + {'kind': 'k', 'version': 1.0, 'state': 1}, + {'kind': 'k', 'version': null, 'state': 1}, + null, 'zzz', 3, true, [], [1, 2, 3], {}, + ]) { + expect(decodeCheckpoint(c, bad, kind: 'k', version: 1), isNull, + reason: jsonEncode(bad)); + } + expect(decodeCheckpoint(c, env(4), kind: 'k', version: 1), 4); + }); + test('a damaged nested state refuses rather than guessing', () { + final lb = Lookback(3, const _WindowFirst(3), + batchFromJson: DoubleColumn.fromJson); + final data = col([for (var i = 0; i < 9; i++) i * 1.0]); + final good = wire(encodeCheckpoint(lb, lb.fold(data, 0, 5), + kind: 'lb', version: 1)) as Map; + expect(decodeCheckpoint(lb, good, kind: 'lb', version: 1), isNotNull); + for (final m in mutants(good['state'])) { + expect( + decodeCheckpoint(lb, {...good, 'state': m}, + kind: 'lb', version: 1), + isNull, + reason: jsonEncode(m)); + } + }); + test('a non-persistable aggregator cannot be checkpointed', () { + expect(() => encodeCheckpoint(const _Plain(), 0, kind: 'k', version: 1), + throwsUnsupportedError); + }); + }); +} diff --git a/test/aggregate/primitives_test.dart b/test/aggregate/primitives_test.dart new file mode 100644 index 0000000..2cbc651 --- /dev/null +++ b/test/aggregate/primitives_test.dart @@ -0,0 +1,271 @@ +// Primitives: known values (NaN skipping, empty gives the identity result), +// then the laws on seeded random columns. + +import 'dart:math' as math; +import 'dart:typed_data'; + +import 'package:openstrap_analytics/aggregate.dart'; +import 'package:openstrap_analytics/onehz.dart' show RunningMoments; +import 'package:test/test.dart'; + +import '../support/aggregator_laws.dart'; +import '../support/property.dart' show Rng; + +Float64List _v(DoubleColumn c) => c.values; + +const nan = double.nan; + +List _momentFields(RunningMoments m) => + [m.count, m.mean, m.sampleSd, m.populationSd]; + +// Aggregators over DoubleColumn. +final _count = Count(); +final _finite = CountFinite(_v); +final _positive = CountWhere(_v, (x) => x > 0); +final _sum = Sum(_v); +final _sumSq = SumSq(_v); +final _min = Min(_v); +final _max = Max(_v); +final _moments = Moments(_v); + +/// Every split point of [c] recombines to [want] (compared by [eq]). +void _splitsAgree(Aggregator a, DoubleColumn c, + bool Function(R, R) eq) { + final want = a.run(c); + for (var k = 0; k <= c.length; k++) { + final got = a.finish(a.combine(a.fold(c, 0, k), a.fold(c, k, c.length))); + expect(eq(got, want), isTrue, reason: 'split at $k: $got vs $want'); + } +} + +void main() { + final cols = randomColumns(11); + final pos = randomColumns(12, nanP: .1, value: (r) => 1 + r.nextDouble() * 100); + final ints = randomIntBatches(13); + + group('Count', () { + test('counts rows, NaN included', () { + expect(_count.run(col([1, nan, 3])), 3); + expect(_count.run(col([])), 0); + expect(Count>().runList(['a', 'b']), 2); + }); + test('honours the range', () { + final c = col([1, 2, 3, 4, 5]); + expect(_count.finish(_count.fold(c, 1, 4)), 3); + expect(_count.finish(_count.fold(c, 2, 2)), 0); + }); + }); + + group('CountFinite and CountWhere', () { + test('CountFinite skips NaN', () { + expect(_finite.run(col([1, nan, 3, nan])), 2); + expect(_finite.run(col([nan, nan])), 0); + expect(_finite.run(col([])), 0); + }); + test('CountWhere counts non-NaN rows passing the test', () { + expect(_positive.run(col([-1, 2, nan, 3, 0])), 2); + expect(_positive.run(col([])), 0); + // NaN is skipped before the test, even for a test NaN would pass. + final notPos = CountWhere(_v, (x) => !(x > 0)); + expect(notPos.run(col([-1, nan, 5, 0])), 2); + }); + test('both use the column getter', () { + final second = CountFinite>( + (b) => Float64List.fromList([for (final r in b.rows) r.$2])); + expect(second.runList([(1, nan), (2, 3), (3, 4)]), 2); + }); + }); + + group('Sum', () { + test('plain sums, empty is exactly 0.0', () { + expect(_sum.run(col([1.5, 2.5, -1.0])), 3.0); + final e = _sum.run(col([])); + expect(e, 0.0); + expect(e, isA()); + }); + test('skips NaN; all-NaN is 0.0', () { + expect(_sum.run(col([1, nan, 2, nan])), 3.0); + expect(_sum.run(col([nan, nan])), 0.0); + }); + test('is compensated: ten times 0.1 is exactly 1.0', () { + expect(_sum.run(col(List.filled(10, 0.1))), 1.0); + }); + test('is Neumaier compensated (big, small, minus big)', () { + expect(_sum.run(col([1e16, 1.0, -1e16])), 1.0); + expect(_sum.run(col([1.0, 1e16, -1e16])), 1.0); + }); + test('compensation survives chunking and repeated addSlice', () { + final c = col([1e16, 1.0, -1e16, 1.0]); + for (var k = 0; k <= 4; k++) { + final s = _sum.combine(_sum.fold(c, 0, k), _sum.fold(c, k, 4)); + expect(_sum.finish(s), 2.0, reason: 'split at $k'); + var one = _sum.addSlice(_sum.empty(), c, 0, k); + one = _sum.addSlice(one, c, k, 4); + expect(_sum.finish(one), 2.0, reason: 'sliced at $k'); + } + }); + test('finish does not mutate: finish twice, keep adding', () { + final c = col([1, 2, 3, 4]); + var s = _sum.fold(c, 0, 3); + expect(_sum.finish(s), 6.0); + expect(_sum.finish(s), 6.0); + s = _sum.addSlice(s, c, 3, 4); + expect(_sum.finish(s), 10.0); + }); + }); + + group('SumSq', () { + test('sums squares, skips NaN, empty is 0.0', () { + expect(_sumSq.run(col([1, -2, 3])), 14.0); + expect(_sumSq.run(col([2, nan, 3])), 13.0); + expect(_sumSq.run(col([])), 0.0); + expect(_sumSq.run(col([nan])), 0.0); + }); + }); + + group('Min and Max', () { + test('values', () { + final c = col([3, -1, 2]); + expect(_min.run(c), -1.0); + expect(_max.run(c), 3.0); + expect(_min.run(col([7])), 7.0); + expect(_max.run(col([-4, -9])), -4.0); + }); + test('null when no finite row', () { + expect(_min.run(col([])), isNull); + expect(_max.run(col([])), isNull); + expect(_min.run(col([nan, nan])), isNull); + expect(_max.run(col([nan])), isNull); + }); + test('NaN rows are skipped', () { + final c = col([nan, 5, nan, 2, 9, nan]); + expect(_min.run(c), 2.0); + expect(_max.run(c), 9.0); + }); + test('empty and all-NaN sides of a seam change nothing', () { + final full = col([4, 2]); + final nans = col([nan, nan]); + expect(_max.finish(_max.combine(_max.empty(), _max.fold(full))), 4.0); + expect(_max.finish(_max.combine(_max.fold(full), _max.empty())), 4.0); + expect(_max.finish(_max.combine(_max.fold(nans), _max.fold(full))), 4.0); + expect(_min.finish(_min.combine(_min.fold(full), _min.fold(nans))), 2.0); + expect(_max.finish(_max.combine(_max.empty(), _max.empty())), isNull); + expect(_min.finish(_min.combine(_min.fold(nans), _min.fold(nans))), isNull); + }); + }); + + group('Moments', () { + test('textbook sample', () { + final m = _moments.run(col([2, 4, 4, 4, 5, 5, 7, 9])); + expect(m.count, 8); + expect(m.mean, closeTo(5.0, 1e-12)); + expect(m.populationSd, closeTo(2.0, 1e-12)); + expect(m.sampleSd, closeTo(math.sqrt(32 / 7), 1e-12)); + }); + test('empty, one value, all NaN', () { + final e = _moments.run(col([])); + expect((e.count, e.mean, e.sampleSd), (0, null, null)); + final one = _moments.run(col([4])); + expect((one.count, one.mean, one.sampleSd), (1, 4.0, null)); + final n = _moments.run(col([nan, nan])); + expect((n.count, n.mean), (0, null)); + }); + test('skips NaN rows', () { + final m = _moments.run(col([nan, 2, 4, nan, 6])); + expect(m.count, 3); + expect(m.mean, closeTo(4.0, 1e-12)); + expect(m.sampleSd, closeTo(2.0, 1e-12)); + }); + test('merged across a seam matches the whole', () { + final c = col([for (var i = 0; i < 30; i++) math.sin(i) * 100 + 500]); + final whole = _moments.run(c); + for (final k in [0, 1, 15, 29, 30]) { + final r = _moments + .finish(_moments.combine(_moments.fold(c, 0, k), _moments.fold(c, k, 30))); + expect(r.count, whole.count); + expect(r.mean, closeTo(whole.mean!, 1e-9)); + expect(r.sampleSd, closeTo(whole.sampleSd!, 1e-9)); + } + }); + }); + + group('First and Last', () { + final first = First>(), last = Last>(); + + test('a one-row copy, null when empty', () { + final f = first.runList(['x', 'y', 'z'])!; + final l = last.runList(['x', 'y', 'z'])!; + expect(f.rows, ['x']); + expect(l.rows, ['z']); + expect(first.runList([]), isNull); + expect(last.runList([]), isNull); + }); + test('work on a column and on a range', () { + final c = col([10, 20, 30, 40]); + expect(First().run(c)!.values, [10.0]); + expect(Last().run(c)!.values, [40.0]); + final f = First(), l = Last(); + expect(f.finish(f.fold(c, 1, 3))!.values, [20.0]); + expect(l.finish(l.fold(c, 1, 3))!.values, [30.0]); + expect(f.finish(f.fold(c, 2, 2)), isNull); + }); + test('the copy does not alias the source', () { + final v = Float64List.fromList([1, 2, 3]); + final l = Last().run(DoubleColumn(v))!; + v[2] = 99; + expect(l.values, [3.0]); + }); + test('keep a seam in order, skipping empty chunks', () { + final f = First(), l = Last(); + final c = col([10, 20, 30]); + final fs = f.combine(f.combine(f.fold(c, 0, 0), f.fold(c, 0, 2)), f.fold(c, 2, 3)); + expect(f.finish(fs)!.values, [10.0]); + final ls = l.combine(l.combine(l.fold(c, 0, 2), l.fold(c, 2, 3)), l.fold(c, 3, 3)); + expect(l.finish(ls)!.values, [30.0]); + }); + }); + + group('every primitive is lawful on seeded random cases', () { + final tol = fieldsEqual((x) => [x], relTol: 1e-12, absTol: 1e-6); + List batchFields(DoubleColumn? b) => + b == null ? [null] : [b.length, ...b.values]; + + test('Count', () => expectLawful(_count, cols)); + test('CountFinite', () => expectLawful(_finite, cols)); + test('CountWhere', () => expectLawful(_positive, cols)); + test('Sum', () => expectLawful(_sum, cols, equals: tol)); + test('SumSq', () => expectLawful(_sumSq, cols, equals: tol)); + test('Min', () => expectLawful(_min, cols)); + test('Max', () => expectLawful(_max, cols)); + test('Moments (tolerance)', () { + expectLawful(_moments, pos, + equals: fieldsEqual(_momentFields, + relTol: 1e-9, absTol: 1e-9)); + }); + test('First', () { + expectLawful(First(), cols, + equals: fieldsEqual(batchFields)); + expectLawful(First>(), ints, + equals: (a, b) => a?.rows.toString() == b?.rows.toString()); + }); + test('Last', () { + expectLawful(Last(), cols, + equals: fieldsEqual(batchFields)); + expectLawful(Last>(), ints, + equals: (a, b) => a?.rows.toString() == b?.rows.toString()); + }); + test('both of primitives with mixed result types', () { + expectLawful( + both(_min, _count).map((r) => [r.$1, r.$2]), cols, + equals: (a, b) => a.toString() == b.toString()); + }); + test('splits agree on a column with NaN gaps', () { + final r = Rng(5); + final c = col([ + for (var i = 0; i < 25; i++) r.nextBool(.3) ? nan : r.intIn(-9, 9) * 1.0 + ]); + _splitsAgree(_finite, c, (a, b) => a == b); + _splitsAgree(_max, c, (a, b) => a == b); + }); + }); +} diff --git a/test/aggregate/prsa_aggregator_test.dart b/test/aggregate/prsa_aggregator_test.dart new file mode 100644 index 0000000..e6668f7 --- /dev/null +++ b/test/aggregate/prsa_aggregator_test.dart @@ -0,0 +1,144 @@ +// prsaAggregator must reproduce decelerationCapacity / accelerationCapacity +// (the references, unchanged) under every slicing / chunking, and obey the laws. +// +// Contract: prsaAggregator({required deceleration, l = 2, anchorRatioCap = 0.05}) +// folds a DoubleColumn of NN intervals (ms); l < 2 is the reference's absent +// Metric for every input and never throws. Persistable. +// +// Equality: anchors, kind, note, tier, inputs exact; capacity and every profile +// entry within 1e-9 (rel and abs); confidence within 1e-12. + +import 'dart:typed_data'; + +import 'package:openstrap_analytics/aggregate.dart'; +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +import '../support/aggregator_laws.dart'; +import '../support/checkpoint_support.dart'; +import '../support/property.dart' show Rng; + +List _exact(Metric m) => [ + m.present, m.tier, m.inputs_used.join(','), m.note, + m.value?.anchors, m.value?.kind, m.value?.profile.length, + ]; +List _near(Metric m) => + [m.value?.capacity, ...?m.value?.profile]; + +final _same = (Metric a, Metric b) => + fieldsEqual>(_exact)(a, b) && + fieldsEqual>(_near, relTol: 1e-9, absTol: 1e-9)(a, b) && + (a.confidence - b.confidence).abs() <= 1e-12; + +DoubleColumn _c(Iterable x) => + DoubleColumn(Float64List.fromList([for (final v in x) v.toDouble()])); + +/// Seeded NN series of [n] beats, 700-1100 ms, small steps; [draw] overrides. +List _walk(Rng r, int n) { + var v = 900.0; + return [ + for (var i = 0; i < n; i++) + v = (v + (r.nextDouble() - .5) * 90).clamp(700.0, 1100.0) + ]; +} + +List _cases(int seed) { + final r = Rng(seed); + return [ + _c([]), + _c([800]), + _c(_walk(r, 7)), // short for every l + _c(_walk(r, 14)), + for (var i = 0; i < 4; i++) _c(_walk(r, r.intIn(15, 120))), + _c(_walk(r, 400)), + _c([for (var i = 0; i < 40; i++) 700 + i * 2.0]), // monotone up + _c([for (var i = 0; i < 40; i++) 1100 - i * 2.0]), // monotone down + _c([ + for (final v in _walk(r, 60)) r.nextBool(.15) ? (r.nextBool() ? 0.0 : -v) : v + ]), // zero / negative values + ]; +} + +void main() { + group('parity with the reference', () { + for (final dec in [true, false]) { + for (final l in [2, 3, 5]) { + for (final cap in [0.05, 0.2]) { + test('${dec ? 'DC' : 'AC'}, l=$l, cap=$cap', () { + expectConversion>( + (_) => prsaAggregator( + deceleration: dec, l: l, anchorRatioCap: cap), + _cases(l * 100 + (dec ? 1 : 0)), + (b) { + final x = b.values.toList(); + return dec + ? decelerationCapacity(x, l: l, anchorRatioCap: cap) + : accelerationCapacity(x, l: l, anchorRatioCap: cap); + }, + equals: _same, + seed: l, + ); + }); + } + } + } + }); + + group('absent cases', () { + for (final l in [1, 0, -3]) { + test('l=$l is absent for every input and never throws', () { + final a = prsaAggregator(deceleration: true, l: l); + final b = _c(_walk(Rng(l + 9), 50)); + final want = decelerationCapacity(b.values.toList(), l: l); + expect(want.present, isFalse); + expect(_same(a.run(b), want), isTrue); + expect(_same(a.finish(a.empty()), want), isTrue); + expect(_same(runChunked(a, b, [0, 20, 50]), want), isTrue); + expect(_same(runSliced(a, b, [0, 0, 30, 50]), want), isTrue); + }); + } + }); + + test('persistable: a mid-stream checkpoint resumes to the one-pass result', () { + final agg = prsaAggregator(deceleration: false, l: 3); + expect(agg.persistable, isTrue); + expectCheckpointResumes(agg, _c(_walk(Rng(0xBA0E), 300)), equals: _same); + }); + + group('checkpoint relationships (beat count, window length, anchors)', () { + final agg = prsaAggregator(deceleration: true, l: 2); + final b = _c(_walk(Rng(0xBA0F), 40)); + // State: [n, lookback]; lookback inner is {profile, anchors}. + Map lookback(Object? s) => (s as List)[1] as Map; + Map inner(Object? s) => lookback(s)['inner'] as Map; + + test('the beat count must equal the lookback row count', () { + for (final n in [10, 39, 41]) { + expectDamagedCheckpointRefused(agg, b, (s) { + (s as List)[0] = n; + return s; + }, reason: 'n=$n'); + } + }); + test('more anchors than windows is refused', () { + // 40 beats hold 40 - 2l = 36 windows with l = 2. + expectDamagedCheckpointRefused(agg, b, (s) { + inner(s)['anchors'] = 37; + return s; + }); + }); + test('anchors with fewer beats than one window are refused', () { + final short = _c(_walk(Rng(1), 3)); + expectDamagedCheckpointRefused(agg, short, (s) { + inner(s)['anchors'] = 1; + return s; + }); + }); + test('a consistent checkpoint restores', () { + expect( + decodeCheckpoint(agg, checkpointOf(agg, b), + kind: 'test', version: 1), + isNotNull); + }); + }); +} diff --git a/test/aggregate/readiness_lnrmssd_aggregator_test.dart b/test/aggregate/readiness_lnrmssd_aggregator_test.dart new file mode 100644 index 0000000..443cd0a --- /dev/null +++ b/test/aggregate/readiness_lnrmssd_aggregator_test.dart @@ -0,0 +1,331 @@ +// readinessLnRmssdAggregator (Count + LastRows(windowDays + 1)) must reproduce +// the reference EXACTLY (every field, bit for bit), and the public +// readinessLnRmssd, which delegates to it, must too. `_reference` is a frozen +// copy of upstream's readinessLnRmssd at 77fe373 (median/MAD centre, the +// minNights - 1 prior-night gate), rows only (no dates). +// +// windowDays <= 0 decision. The reference with windowDays 0 keeps an empty +// prior window, so it returns the absent metric (need minNights + 1) and +// LastRows(1) can express that: windowDays 0 is checked on the aggregator +// like any other. With windowDays -1 the reference computes start = n, then +// sublist(n, n - 1) throws RangeError (once n >= minNights; shorter +// histories are absent first). LastRows(0) cannot be built, so the aggregator +// is NOT required to exist for windowDays < 0: the public function keeps the +// reference path there and must behave identically, throw included. + + +import 'package:openstrap_analytics/aggregate.dart'; +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +import '../support/aggregator_laws.dart'; +import '../support/checkpoint_support.dart'; +import '../support/property.dart' show Rng; + +Metric _reference( + List historyLnRmssd, { + double? meanNnTodayMs, + int windowDays = 7, + int minNights = readinessLnRmssdMinNights, +}) { + const inputs = ['ln_rmssd_history']; + if (historyLnRmssd.length < minNights) { + return Metric.absent( + tier: Tier.high, + inputs_used: inputs, + note: needBaselineNote(have: historyLnRmssd.length, need: minNights), + ); + } + final today = historyLnRmssd.last; + final n = historyLnRmssd.length; + final start = n - 1 - windowDays < 0 ? 0 : n - 1 - windowDays; + // this used to be historyLnRmssd.sublist(start), which runs to the END of + // the list - including tonight's own value in its own baseline. that + // pulls the mean/sd toward tonight, understating how far off a genuinely + // suppressed/elevated night actually is, worst right when the window is + // smallest (minNights). the baseline has to be strictly prior nights. + final List priorWindow; + priorWindow = historyLnRmssd.sublist(start, n - 1); + if (priorWindow.isEmpty || priorWindow.length < minNights - 1) { + // empty only if minNights got set to 1 somewhere; short when a wear gap + // left too few nights inside the calendar window. either way there's no + // baseline yet, same as not enough history. + return Metric.absent( + tier: Tier.high, + inputs_used: inputs, + note: needBaselineNote( + have: priorWindow.length + 1, need: minNights < 2 ? 2 : minNights), + ); + } + final m = mean(priorWindow)!; + final sd = stddev(priorWindow); + if (sd == null || m == 0) { + // The published outputs of this stack (CV, SWC, band) are all defined + // RELATIVE to the baseline's dispersion. With a single prior night the SD + // is UNDEFINED — and the metric used to fill in cvPct 0.0 and band + // 'normal', asserting "tonight is typical" on the strength of nothing at + // all. Abstain instead (a zero-but-DEFINED SD is a different case: CV is + // genuinely 0 and SWC is genuinely 0, so that still computes below). + return Metric.absent( + tier: Tier.high, + inputs_used: inputs, + note: 'lnRMSSD baseline dispersion undefined (needs ≥2 prior nights) — ' + 'CV/z/SWC/band are undefined', + ); + } + final cv = (sd / m).abs() * 100; + // ROBUST CENTRE AND SPREAD for the decision band. One illness night in a + // seven-night window both drags the mean down and inflates the SD, so a + // genuinely low night reads "normal": on a real window holding a 35.5 ms + // night, mean 4.465 / SD 0.405 put a 4.478 night (88 ms) inside the band, + // while the other six nights sat at 4.40–4.79. Median + MAD (×1.4826) are + // what the window looks like without that one night. MAD collapses only when + // more than half the window is identical; the SD stands in then. The CV + // above stays the classic SD/mean (Plews' published CV). + final centre = median(priorWindow)!; + final robustSd = mad(priorWindow)!; + final spread = robustSd > 0 ? robustSd : sd; + final z = spread > 0 ? (today - centre) / spread : null; + // Plews SWC ≈ 0.5 × within-window SD (a small worthwhile change in lnRMSSD). + final swc = 0.5 * spread; + + final String band; + if (today < centre - swc) { + band = 'suppressed'; + } else if (today > centre + swc) { + band = 'elevated'; + } else { + band = 'normal'; + } + + // LnRMSSD:RR saturation guard: when mean NN is long (low HR, high vagal tone) + // and lnRMSSD is at the top of the personal range, the metric saturates. + final saturation = + meanNnTodayMs != null && meanNnTodayMs > 1100 && z != null && z > 1.0; + + final conf = (priorWindow.length / windowDays.toDouble()).clamp(0.3, 0.9); + return Metric( + value: ReadinessLnRmssd( + lnRmssdToday: today, + rolling7Mean: m, + rolling7Median: centre, + cvPct: cv, + z: z, + swc: swc, + band: band, + saturationFlag: saturation, + ), + confidence: conf, + tier: Tier.high, + inputs_used: inputs, + note: saturation + ? 'lnRMSSD:RR saturation — high-HRV interpretation caution' + : 'Plews/Kiviniemi whole-night lnRMSSD readiness; PRV', + ); +} + +// Exact, NaN-aware (fieldsEqual with zero tolerance). +List _fields(Metric m) { + final v = m.value; + return [ + m.present, m.confidence, m.tier, m.inputs_used.join(','), m.note, + m.drivers, v?.lnRmssdToday, v?.rolling7Mean, v?.rolling7Median, v?.cvPct, v?.z, v?.swc, + v?.band, v?.saturationFlag, + ]; +} + +final _same = fieldsEqual>(_fields); + +DoubleColumn _hist(Rng r, int n, int style) => col([ + for (var i = 0; i < n; i++) + switch (style) { + 0 => 3.0 + r.nextDouble() * 2.0, + 1 => 4.2, // constant: sd == 0 + 2 => 0.0, // zeros: m == 0 + _ => r.nextBool(.3) ? 0.0 : 3.0 + r.nextDouble() * 2.0, + } + ]); + +List _cases(int seed) { + final r = Rng(seed); + return [ + for (var n = 0; n <= 40; n++) _hist(r, n, n % 4), + for (var n = 1; n <= 12; n++) _hist(r, n, 0), + for (var n = 3; n <= 20; n += 4) _hist(r, n, 1), + for (var n = 3; n <= 20; n += 4) _hist(r, n, 2), + ]; +} + +void main() { + group('parity with the reference', () { + for (final w in [0, 1, 3, 7, 14]) { + for (final nn in [null, 900.0, 1200.0]) { + test('windowDays=$w meanNn=$nn, minNights 1/2/4/6', () { + for (final mn in [1, 2, 4, 6]) { + expectConversion>( + (_) => readinessLnRmssdAggregator( + meanNnTodayMs: nn, windowDays: w, minNights: mn), + _cases(w * 31 + mn), + (b) => _reference(b.values.toList(), + meanNnTodayMs: nn, windowDays: w, minNights: mn), + equals: _same, + chunkings: 3, + seed: w + mn, + ); + } + }); + } + } + }); + + group('public readinessLnRmssd equals the reference', () { + for (final w in [0, 1, 3, 7, 14]) { + test('windowDays=$w', () { + for (final nn in [null, 900.0, 1200.0]) { + for (final mn in [1, 2, 4, 6]) { + for (final b in _cases(w + 500)) { + final h = b.values.toList(); + final got = readinessLnRmssd(h, + meanNnTodayMs: nn, windowDays: w, minNights: mn); + final want = _reference(h, + meanNnTodayMs: nn, windowDays: w, minNights: mn); + expect(_same(got, want), isTrue, + reason: 'n=${h.length} mn=$mn: $got vs $want'); + } + } + } + }); + } + test('default arguments', () { + for (final b in _cases(9)) { + final h = b.values.toList(); + expect(_same(readinessLnRmssd(h), _reference(h)), isTrue); + } + }); + test('windowDays=-1: absent when short, RangeError otherwise, as before', + () { + for (final b in _cases(3)) { + final h = b.values.toList(); + Object? run(Metric Function() f) { + try { + return f(); + } on RangeError { + return 'RangeError'; + } + } + + final got = run(() => readinessLnRmssd(h, windowDays: -1)); + final want = run(() => _reference(h, windowDays: -1)); + if (want is String) { + expect(got, want, reason: 'n=${h.length}'); + expect(h.length >= readinessLnRmssdMinNights, isTrue); + } else { + expect(got, isA>()); + expect( + _same(got as Metric, + want as Metric), + isTrue); + } + } + }); + }); + + test('persistable: a mid-stream checkpoint resumes to the one-pass result', + () { + final agg = readinessLnRmssdAggregator(meanNnTodayMs: 1200); + expect(agg.persistable, isTrue); + expectCheckpointResumes(agg, _hist(Rng(0x12), 30, 0), equals: _same); + }); + + group('checkpoint relationships (count against the kept rows)', () { + final agg = readinessLnRmssdAggregator(windowDays: 7); // keeps 8 rows + Metric? restore(Object? state) { + final got = decodeCheckpoint( + agg, {'kind': 'k', 'version': 1, 'state': state}, + kind: 'k', version: 1); + return got == null ? null : agg.finish(got); + } + + test('review case: count 4 with no rows is refused', () { + // Used to decode, then finish threw on the missing rows. + expect( + decodeCheckpoint( + agg, + { + 'kind': 'k', + 'version': 1, + 'state': [4, {'rows': null}] + }, + kind: 'k', + version: 1), + isNull); + }); + test('count 0 with rows present is refused', () { + final rows = ((checkpointOf(agg, _hist(Rng(1), 3, 0))['state'] as List)[1] + as Map)['rows']; + expect(restore([0, {'rows': rows}]), isNull); + }); + test('count that disagrees with the number of kept rows is refused', () { + final long = _hist(Rng(2), 30, 0); + // 30 nights keep 8 rows; claiming 5 (or 3 nights holding 8) is damage. + expectDamagedCheckpointRefused(agg, long, (s) { + (s as List)[0] = 5; + return s; + }); + expectDamagedCheckpointRefused(agg, long, (s) { + (s as List)[0] = 0; + return s; + }); + final short = _hist(Rng(3), 5, 0); // keeps all 5 + expectDamagedCheckpointRefused(agg, short, (s) { + (s as List)[0] = 9; + return s; + }); + expectDamagedCheckpointRefused(agg, short, (s) { + (s as List)[0] = 4; + return s; + }); + }); + test('a consistent checkpoint still restores', () { + for (final n in [0, 1, 5, 8, 9, 30]) { + final b = _hist(Rng(n), n, 0); + final got = decodeCheckpoint(agg, checkpointOf(agg, b), + kind: 'test', version: 1); + expect(got, isNotNull, reason: 'n=$n'); + expect(_same(agg.finish(got), agg.run(b)), isTrue); + } + }); + }); + + group('reads only the suffix of the history', () { + test('an unused prefix element is never read', () { + // Element 0 is not a double; only the last windowDays + 1 are read. + final h = ['unused', 3.0, 4.0, 5.0, 6.0].cast(); + final got = readinessLnRmssd(h, windowDays: 3); + final want = _reference(h, windowDays: 3); + expect(want.present, isTrue); + expect(_same(got, want), isTrue, reason: '$got vs $want'); + }); + test('a too-short history is refused without reading any element', () { + final h = ['a', 'b'].cast(); + final got = readinessLnRmssd(h); + expect(got.present, isFalse); + expect(got.note, needBaselineNote(have: 2, need: readinessLnRmssdMinNights)); + }); + }); + + group('windowDays at the int limit', () { + const huge = 0x7fffffffffffffff; + final h = [3.0, 4.0, 5.0, 6.0]; + test('the public function matches the reference', () { + final got = readinessLnRmssd(h, windowDays: huge); + expect(_same(got, _reference(h, windowDays: huge)), isTrue); + expect(got.present, isTrue); + }); + test('the aggregator matches the reference and resumes', () { + final agg = readinessLnRmssdAggregator(windowDays: huge); + expect(_same(agg.run(col(h)), _reference(h, windowDays: huge)), isTrue); + expectCheckpointResumes(agg, col(h), equals: _same); + }); + }); +} diff --git a/test/aggregate/rr_sandwich_test.dart b/test/aggregate/rr_sandwich_test.dart new file mode 100644 index 0000000..e0c72e8 --- /dev/null +++ b/test/aggregate/rr_sandwich_test.dart @@ -0,0 +1,509 @@ +// RED: the RR "sandwich" (combined bump, chunk A2 + A3). +// +// Contract (lib/src/onehz/foundations/rr_sandwich.dart): +// RrSandwich(inner, {maxArtifactRun = kMaxArtifactRun, ...}) is an +// Aggregator over RawRr (raw RR ms + beat timestamps). It corrects with the +// capped Lipponen-Tarvainen rule and feeds the settled NN (nnMs) into +// `inner`, each with the ABSOLUTE raw timestamp (`rrTsMs`) of the input beat +// it came from (`nnSourceIndex`): the beat itself, or for a spline-corrected +// beat that beat's own timestamp. Not correctRr's relative clock, so a seam +// needs no clock translation. +// +// * ANY chunking (random cuts, one-beat chunks, empty chunks, chunks as +// ranges or as copies, combined left- or right-nested, or one state sliced) +// finishes to exactly what one pass gives, and that is the capped +// `correctRr` over all the beats: nn bit for bit, paired with the absolute +// timestamps of the source beats. +// * It obeys the aggregator laws (identity, split, associativity, singletons, +// slicing, finishPure, roundTrip) and checkpoints/resumes mid-stream. +// * Its state is bounded: it keeps a raw head and tail, not the series, so an +// endless artifact run does not grow it. +// * A3: the night HRV time-domain and PRSA aggregators fed through it agree +// with the batch functions run on capped correctRr's NN, within their +// documented tolerances (1e-9). +// +// Data are integer RR with whole-second timestamps (synthRr: dropouts, ectopy, +// missed and extra beats, noise runs) and cleanly built long-run series +// (support/rr_runs.dart), so every clock value is an integer and "exactly" is +// attainable across chunk seams. +import 'dart:convert'; +import 'dart:typed_data'; + +import 'package:openstrap_analytics/aggregate.dart'; +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +import '../onehz/support/rr_compare.dart'; +import '../onehz/support/rr_runs.dart'; +import '../onehz/support/rr_synth.dart'; +import '../support/aggregator_laws.dart'; +import '../support/checkpoint_support.dart'; +import '../support/hrv_beats.dart'; +import '../support/property.dart'; + +/// Test inner: keeps every NN and its time (unbounded on purpose; the point is +/// to see exactly what the sandwich hands down). +class _Acc { + final nn = [], t = []; +} + +class _Collect extends Aggregator<_Acc, Beats, Beats> { + @override + _Acc empty() => _Acc(); + @override + _Acc addSlice(_Acc s, Beats b, int start, int end) { + for (var i = start; i < end; i++) { + s.nn.add(b.nnMs[i]); + s.t.add(b.timeMs[i]); + } + return s; + } + + @override + _Acc combine(_Acc l, _Acc r) { + l.nn.addAll(r.nn); + l.t.addAll(r.t); + return l; + } + + @override + Beats finish(_Acc s) => Beats(Float64List.fromList(s.nn), Float64List.fromList(s.t)); + + @override + bool get persistable => true; + @override + Object? encodeState(_Acc s) => {'nn': s.nn, 't': s.t}; + @override + _Acc decodeState(Object? json) { + final m = json as Map; + return _Acc() + ..nn.addAll([for (final v in m['nn'] as List) (v as num).toDouble()]) + ..t.addAll([for (final v in m['t'] as List) (v as num).toDouble()]); + } +} + +RrSandwich<_Acc, Beats> _sandwich({int? maxArtifactRun = kMaxArtifactRun}) => + RrSandwich<_Acc, Beats>(_Collect(), maxArtifactRun: maxArtifactRun); + +bool _sameBeats(Beats a, Beats b) { + if (a.length != b.length) return false; + for (var i = 0; i < a.length; i++) { + if (a.nnMs[i].compareTo(b.nnMs[i]) != 0 || + a.timeMs[i].compareTo(b.timeMs[i]) != 0) { + return false; + } + } + return true; +} + +typedef RrSeriesLike = ({List rr, List ts}); + +/// The oracle: capped (or [cap]ped) correctRr's NN, each paired with the +/// ABSOLUTE timestamp of the input beat it came from. +({List nn, List t}) _oracle(RrSeriesLike s, + {int? cap = kMaxArtifactRun}) { + final c = correctRr(s.rr, rrTsMs: s.ts, maxArtifactRun: cap); + return (nn: c.nn, t: [for (final i in c.nnSourceIndex) s.ts[i]]); +} + +void _expectIsCorrectRr(Beats got, RrSeriesLike s, String why, + {int? cap = kMaxArtifactRun}) { + final want = _oracle(s, cap: cap); + expectBitIdentical(got.nnMs, want.nn, '$why nn'); + expectBitIdentical(got.timeMs, want.t, '$why absolute times'); +} + +RawRr _raw(RrSeriesLike s) => RawRr.of(s.rr, s.ts); + +RrSeriesLike _synth(int seed, double hours, + {double noise = 0.5, double gaps = 12}) { + final d = synthRr(SynthConfig( + seed: seed, + hours: hours, + ectopicPerMin: 2, + missedPerMin: 1, + extraPerMin: 1, + noiseRunPerMin: noise, + gapPerHour: gaps)); + return (rr: d.rr, ts: d.ts); +} + +RrSeriesLike _runs(int seed, List segs, List runs, {bool near = true}) { + final (s, _) = withRuns(seed, segs, runs, nearRun: near); + return (rr: s.rr, ts: s.ts); +} + +void main() { + final cases = { + 'synthetic, dirty': _synth(23, 0.1), + 'synthetic, calm, with dropouts': _synth(5, 0.1, noise: 0.05, gaps: 30), + 'one 40-beat run': _runs(7, [200, 200], [40]), + 'two long runs': _runs(8, [120, 120, 120], [33, 45]), + }; + + group('single pass = capped correctRr', () { + for (final e in cases.entries) { + test(e.key, () { + final agg = _sandwich(); + _expectIsCorrectRr(agg.run(_raw(e.value)), e.value, e.key); + }); + } + + test('empty and tiny inputs', () { + for (final n in [0, 1, 2, 3, 5]) { + final s = _synth(3, 0.05); + final part = (rr: s.rr.sublist(0, n), ts: s.ts.sublist(0, n)); + _expectIsCorrectRr(_sandwich().run(_raw(part)), part, 'n=$n'); + } + }); + + test('maxArtifactRun: null is the uncapped corrector', () { + final s = _runs(9, [150, 150], [40]); + final got = _sandwich(maxArtifactRun: null).run(_raw(s)); + _expectIsCorrectRr(got, s, 'uncapped', cap: null); + }); + }); + + group('any chunking = one pass', () { + forAll('random cuts (empty and one-beat chunks included), every path', + G.intIn(1, 100000), (seed) { + final s = _synth(seed, 0.07); + final batch = _raw(s); + final want = _oracle(s); + final agg = _sandwich(); + final r = Rng(seed); + for (var k = 0; k < 3; k++) { + final bounds = randomBounds(r, batch.length); + for (final (name, got) in [ + ('chunked', runChunked(agg, batch, bounds)), + ('copies', runChunked(agg, batch, bounds, copy: true)), + ('right', runChunkedRight(agg, batch, bounds)), + ('sliced', runSliced(agg, batch, bounds)), + ]) { + expectBitIdentical(got.nnMs, want.nn, '$name $bounds nn'); + expectBitIdentical(got.timeMs, want.t, '$name $bounds t'); + } + } + }, cases: 25, budget: const Duration(seconds: 90)); + + test('one-beat chunks, combined left and right', () { + final s = _synth(31, 0.05); + final batch = _raw(s); + final agg = _sandwich(); + _expectIsCorrectRr( + runChunked(agg, batch, singletonBounds(batch.length)), s, 'left'); + _expectIsCorrectRr( + runChunkedRight(agg, batch, singletonBounds(batch.length)), s, 'right'); + }); + + test('cuts inside, at the edges of and across a long artifact run', () { + final (s, runs) = withRuns(41, [150, 150], [50], nearRun: true); + final batch = RawRr.of(s.rr, s.ts); + final (a, b) = runs.single; + final agg = _sandwich(); + for (final cuts in [ + [a - 1, a, a + 1], + [a + 10, a + 20, a + 33, a + 34], + [b - 1, b, b + 1], + [a - 60, a + 25, b + 60], + [a + 1, a + 2, a + 3, a + 4, b - 2, b - 1], + ]) { + final bounds = [0, ...cuts, batch.length]; + final want = _oracle((rr: s.rr, ts: s.ts)); + for (final got in [ + runChunked(agg, batch, bounds), + runChunkedRight(agg, batch, bounds), + runSliced(agg, batch, bounds), + ]) { + expectBitIdentical(got.nnMs, want.nn, 'cuts $cuts nn'); + expectBitIdentical(got.timeMs, want.t, 'cuts $cuts t'); + } + } + }); + + test('a cut at every beat of a dropout seam (the clock re-anchors)', () { + final original = _synth(51, 0.06, gaps: 60); + // A dropout rate is probabilistic; this seed currently draws none. + // Insert a definite two-minute hole so this seam test always runs. + final at = original.rr.length ~/ 2; + final s = (rr: original.rr, ts: [ + for (var i = 0; i < original.ts.length; i++) + original.ts[i] + (i >= at ? 120000 : 0), + ]); + final batch = _raw(s); + final agg = _sandwich(); + final want = _oracle(s); + var seams = 0; + for (var i = 1; i < s.rr.length; i++) { + if ((s.ts[i] - s.ts[i - 1]) - s.rr[i] > 1000) { + seams++; + final got = runChunked(agg, batch, [0, i - 1, i, i + 1, batch.length]); + expectBitIdentical(got.timeMs, want.t, 'seam at $i t'); + expectBitIdentical(got.nnMs, want.nn, 'seam at $i nn'); + } + } + expect(seams, greaterThan(0), reason: 'the series must have dropouts'); + }); + }); + + // Series much longer than the sandwich's head + tail windows, so most beats + // are settled into `inner` and a seam is re-corrected from the windows alone. + group('long series (settled middle in play)', () { + final long = { + 'synthetic, dirty, long': _synth(11, 0.7), + 'synthetic, calm, long, dropouts': _synth(12, 2.0, noise: 0.05, gaps: 30), + 'runs over the cap, long': _runs(13, [700, 700, 700], [40, 50]), + }; + + for (final e in long.entries) { + test('${e.key}: random cuts, every path', () { + final batch = _raw(e.value); + expect(batch.length, greaterThan(1200), reason: 'longer than the windows'); + final want = _oracle(e.value); + final agg = _sandwich(); + _expectIsCorrectRr(agg.run(batch), e.value, 'one pass'); + final r = Rng(e.key.length); + for (var k = 0; k < 4; k++) { + final bounds = randomBounds(r, batch.length); + for (final (name, got) in [ + ('chunked', runChunked(agg, batch, bounds)), + ('copies', runChunked(agg, batch, bounds, copy: true)), + ('right', runChunkedRight(agg, batch, bounds)), + ('sliced', runSliced(agg, batch, bounds)), + ]) { + expectBitIdentical(got.nnMs, want.nn, '$name $bounds nn'); + expectBitIdentical(got.timeMs, want.t, '$name $bounds t'); + } + } + }); + } + + forAll('dense artifacts (ectopy, runs, dropouts), random cuts', + G.intIn(1, 100000), (seed) { + final s = _synth(seed, 0.5, noise: 1.5, gaps: 20); + final batch = _raw(s); + final want = _oracle(s); + final agg = _sandwich(); + final bounds = randomBounds(Rng(seed), batch.length); + final got = runChunked(agg, batch, bounds); + expectBitIdentical(got.nnMs, want.nn, 'seed $seed $bounds nn'); + expectBitIdentical(got.timeMs, want.t, 'seed $seed $bounds t'); + }, cases: 12, budget: const Duration(seconds: 90)); + + test('even chunks of many sizes, left and right', () { + final s = long['synthetic, dirty, long']!; + final batch = _raw(s); + final agg = _sandwich(); + for (final size in [7, 100, 283, 284, 285, 560, 561, 1000]) { + final bounds = [ + for (var i = 0; i < batch.length; i += size) i, + batch.length, + ]; + _expectIsCorrectRr(runChunked(agg, batch, bounds), s, 'size $size left'); + _expectIsCorrectRr( + runChunkedRight(agg, batch, bounds), s, 'size $size right'); + } + }); + + test('one-beat chunks over a long series', () { + final s = long['runs over the cap, long']!; + final batch = _raw(s); + final agg = _sandwich(); + _expectIsCorrectRr( + runChunked(agg, batch, singletonBounds(batch.length)), s, 'left'); + _expectIsCorrectRr(runSliced(agg, batch, singletonBounds(batch.length)), + s, 'sliced'); + }); + + test('state size does not grow with the series', () { + // a bounded inner (Count), so only the sandwich's own windows are sized + final agg = RrSandwich(const Count()); + int chars(RrSeriesLike s) => + jsonEncode(agg.encodeState(agg.fold(_raw(s)))).length; + final small = _synth(14, 1.4), big = _synth(14, 2.8); + expect(big.rr.length, greaterThan(small.rr.length * 1.5)); + expect(chars(big), lessThan(chars(small) * 1.1 + 200)); + }); + }); + + group('laws and persistence', () { + final lawCases = [ + RawRr.of([], []), + RawRr.of([800], [1.7e12]), + _raw((rr: _synth(3, 0.05).rr.sublist(0, 3), ts: _synth(3, 0.05).ts.sublist(0, 3))), + _raw(_synth(4, 0.04)), + _raw(_runs(5, [100, 100], [36])), + _raw(_runs(6, [300, 300], [36])), + ]; + + test('identity, split, associativity, singletons, slicing, finishPure, ' + 'roundTrip', () { + expectLawful, RawRr, Beats>(_sandwich(), lawCases, + equals: _sameBeats); + }); + + test('is persistable', () { + expect(_sandwich().persistable, isTrue); + }); + + test('checkpoints and resumes mid-stream (through JSON text)', () { + final s = _runs(61, [160, 160], [45]); + final batch = _raw(s); + expectCheckpointResumes, RawRr, Beats>( + _sandwich(), batch, + equals: _sameBeats); + final agg = _sandwich(); + final a = 160; + for (final k in [0, 1, 50, a - 1, a + 1, a + 20, a + 33, a + 45, batch.length - 1, batch.length]) { + final env = jsonDecode(jsonEncode(encodeCheckpoint( + agg, agg.fold(batch, 0, k), + kind: 'rr', version: 1))); + final back = decodeCheckpoint>(agg, env, + kind: 'rr', version: 1); + expect(back, isNotNull, reason: 'restores at $k'); + final got = agg.finish(agg.addSlice(back!, batch, k, batch.length)); + _expectIsCorrectRr(got, s, 'resumed at $k'); + } + }); + + test('a checkpoint of the wrong kind or version is refused (null)', () { + final agg = _sandwich(); + final batch = _raw(_synth(3, 0.05)); + final env = jsonDecode(jsonEncode(encodeCheckpoint( + agg, agg.fold(batch, 0, 40), + kind: 'rr', version: 1))); + expect(decodeCheckpoint>(agg, env, kind: 'rr', version: 2), + isNull); + expect(decodeCheckpoint>(agg, {...env as Map, 'state': 'junk'}, + kind: 'rr', version: 1), isNull); + }); + + test('state is bounded: 10,000 artifact beats do not grow it', () { + final r = Rng(71); + final clean = cleanRr(r, 300); + final rr = [...clean, ...List.filled(10000, 3000.0), ...cleanRr(r, 300)]; + final s = rrSeries(rr); + final batch = RawRr.of(s.rr, s.ts); + // Count: a bounded inner, so only the sandwich's own state is measured. + final agg = RrSandwich(const Count()); + final mid = agg.fold(batch, 0, 300 + 10000); + final text = jsonEncode(agg.encodeState(mid)); + expect(text.length, lessThan(80000), reason: 'state chars inside the run'); + final end = agg.finish(agg.addSlice(mid, batch, 10300, batch.length)); + final want = correctRr(s.rr, rrTsMs: s.ts); + expect(end, want.nn.length); + // combining two halves of the run keeps it bounded too + final left = agg.fold(batch, 0, 5300), right = agg.fold(batch, 5300, batch.length); + final merged = agg.combine(left, right); + expect(jsonEncode(agg.encodeState(merged)).length, lessThan(80000)); + expect(agg.finish(merged), want.nn.length); + }); + }); + + group('A3: night aggregators fed through the sandwich', () { + // The reference: capped correctRr's NN into the batch function. + HrvTime hrvRef(RrSeriesLike s) { + final o = _oracle(s); + return hrvExpected(Beats(Float64List.fromList(o.nn), Float64List.fromList(o.t)), + gapAware: true); + } + + double? originOf(RrSeriesLike s) { + final o = _oracle(s); + return o.t.isEmpty ? null : o.t.first; + } + + for (final e in cases.entries) { + test('hrvTimeAggregator, ${e.key}', () { + final batch = _raw(e.value); + final agg = RrSandwich( + hrvTimeAggregator(originMs: originOf(e.value))); + final want = hrvRef(e.value); + expectHrvParity(agg.run(batch), want, 'one pass'); + final r = Rng(e.key.length); + for (var k = 0; k < 4; k++) { + final bounds = randomBounds(r, batch.length); + expectHrvParity(runChunked(agg, batch, bounds), want, 'chunks $bounds'); + expectHrvParity(runChunkedRight(agg, batch, bounds), want, 'right $bounds'); + expectHrvParity(runSliced(agg, batch, bounds), want, 'sliced $bounds'); + } + }); + } + + List exact(Metric m) => [ + m.present, m.tier, m.inputs_used.join(','), m.note, + m.value?.anchors, m.value?.kind, m.value?.profile.length, + ]; + List near(Metric m) => + [m.value?.capacity, ...?m.value?.profile]; + bool same(Metric a, Metric b) => + fieldsEqual>(exact)(a, b) && + fieldsEqual>(near, relTol: 1e-9, absTol: 1e-9)(a, b) && + (a.confidence - b.confidence).abs() <= 1e-12; + + for (final dec in [true, false]) { + for (final e in cases.entries) { + test('prsaAggregator ${dec ? 'DC' : 'AC'}, ${e.key}', () { + final batch = _raw(e.value); + final agg = RrSandwich>( + NnColumn>( + prsaAggregator(deceleration: dec))); + final nn = correctRr(e.value.rr, rrTsMs: e.value.ts).nn; + final want = + dec ? decelerationCapacity(nn) : accelerationCapacity(nn); + expect(same(agg.run(batch), want), isTrue, reason: 'one pass'); + final r = Rng(e.key.length + 1); + for (var k = 0; k < 4; k++) { + final bounds = randomBounds(r, batch.length); + expect(same(runChunked(agg, batch, bounds), want), isTrue, + reason: 'chunks $bounds'); + expect(same(runChunkedRight(agg, batch, bounds), want), isTrue, + reason: 'right $bounds'); + } + }); + } + } + + // Series longer than the windows: the settled middle is in play. + for (final e in { + 'synthetic, dirty, long': _synth(11, 0.7), + 'runs over the cap, long': _runs(13, [700, 700, 700], [40, 50]), + }.entries) { + test('hrvTimeAggregator and prsaAggregator, ${e.key}', () { + final batch = _raw(e.value); + expect(batch.length, greaterThan(1200)); + final hrv = RrSandwich( + hrvTimeAggregator(originMs: originOf(e.value))); + final prsa = RrSandwich>( + NnColumn>( + prsaAggregator(deceleration: true))); + final wantPrsa = decelerationCapacity(_oracle(e.value).nn); + final r = Rng(e.key.length + 2); + final wantHrv = hrvRef(e.value); + for (var k = 0; k < 3; k++) { + final bounds = randomBounds(r, batch.length); + expectHrvParity(runChunked(hrv, batch, bounds), wantHrv, 'hrv $bounds'); + expectHrvParity( + runChunkedRight(hrv, batch, bounds), wantHrv, 'hrv right $bounds'); + expect(same(runChunked(prsa, batch, bounds), wantPrsa), isTrue, + reason: 'prsa $bounds'); + expect(same(runChunkedRight(prsa, batch, bounds), wantPrsa), isTrue, + reason: 'prsa right $bounds'); + } + }); + } + + test('the sandwiched aggregators checkpoint and resume', () { + final s = cases['one 40-beat run']!; + final batch = _raw(s); + final hrv = RrSandwich( + hrvTimeAggregator(originMs: originOf(s))); + expect(hrv.persistable, isTrue); + expectCheckpointResumes, RawRr, HrvTime>(hrv, batch, + equals: (a, b) => hrvEqAll(a, b)); + final prsa = RrSandwich>( + NnColumn>(prsaAggregator(deceleration: true))); + expect(prsa.persistable, isTrue); + }); + }); +} diff --git a/test/aggregate/sri_aggregator_test.dart b/test/aggregate/sri_aggregator_test.dart new file mode 100644 index 0000000..d860446 --- /dev/null +++ b/test/aggregate/sri_aggregator_test.dart @@ -0,0 +1,643 @@ +// Epochs + sriAggregator, and the public phillipsSri rewired onto them, must +// reproduce the ORIGINAL phillipsSri exactly. `_referenceSri` below is a frozen +// copy of that original body. Do not edit it when phillipsSri is rewired. +// +// Contract used here: +// * Epochs(asleep, {valid, firstIndex = 0}): asleep is 0/1 per epoch, valid +// (optional, 0/1) the mask. The absolute epoch index of row r is +// firstIndex + r. valid == null means every epoch is valid. Exported from +// package:openstrap_analytics/aggregate.dart, with public fields asleep, +// valid (Uint8List?) and firstIndex. +// * copyRange(a, b) returns COPIES (the source can change afterwards) with +// firstIndex shifted by a; a null valid stays null. +// * join(other) requires other.firstIndex == firstIndex + length (assert; an +// AssertionError in debug mode). When exactly one side has a mask, the +// missing side counts as all-valid; the joined batch has a mask then, and +// none when neither side has one. The joined batch is a new, independent +// copy and keeps this.firstIndex. +// * sriAggregator(epochsPerDay, {minPairCases}) is built for epochsPerDay >= 1 +// and run on batches with firstIndex 0 (the whole record, or its chunks cut +// with copyRange / ranges of that one batch). Its result equals +// phillipsSri on the same record (sri bit-identical, days, cases, every +// pair, confidence, tier, inputs_used, note). +// * phillipsSri(sleepWake, epochsPerDay, valid:, minPairCases:) keeps its +// signature and behaviour, including epochsPerDay <= 0 (absent) and a +// trailing partial day (ignored). + +import 'dart:typed_data'; + +import 'package:openstrap_analytics/aggregate.dart'; +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +import '../support/checkpoint_support.dart'; +import '../support/aggregator_laws.dart'; +import '../support/property.dart' show Rng; + +// --------------------------------------------------------------------------- +// FROZEN COPY of phillipsSri as it was before the rewire. +// --------------------------------------------------------------------------- +Metric _referenceSri( + List sleepWake, + int epochsPerDay, { + List? valid, + int? minPairCases, +}) { + const inputs = ['sleep_wake_epochs']; + final n = sleepWake.length; + if (epochsPerDay <= 0 || n < 2 * epochsPerDay) { + return const Metric.absent( + tier: Tier.high, + inputs_used: inputs, + note: 'need ≥2 full days of clock-aligned sleep/wake epochs for SRI', + ); + } + final days = n ~/ epochsPerDay; + if (days < 2) { + return const Metric.absent( + tier: Tier.high, + inputs_used: inputs, + note: 'need ≥2 full days for SRI', + ); + } + + final pairFloor = minPairCases ?? (epochsPerDay ~/ 2); + var agreement = 0; + var cases = 0; + final pairs = []; + for (var d = 1; d < days; d++) { + var pairAgreement = 0; + var pairCases = 0; + for (var e = 0; e < epochsPerDay; e++) { + final iPrev = (d - 1) * epochsPerDay + e; + final iCur = d * epochsPerDay + e; + if (iCur >= n) break; + if (valid != null && (!valid[iPrev] || !valid[iCur])) continue; + pairCases++; + if (sleepWake[iPrev] == sleepWake[iCur]) pairAgreement++; + } + agreement += pairAgreement; + cases += pairCases; + if (pairCases > 0 && pairCases >= pairFloor) { + pairs.add(SriPair(d, pairAgreement, pairCases)); + } + } + if (cases == 0) { + return const Metric.absent( + tier: Tier.high, + inputs_used: inputs, + note: 'no valid epoch pairs for SRI', + ); + } + + final sri = 200.0 * agreement / cases - 100.0; + final conf = ((days - 1) / 7.0).clamp(0.3, 0.95); + return Metric( + value: SriResult(sri, days, cases, pairs), + confidence: conf, + tier: Tier.high, + inputs_used: inputs, + note: 'epoch-by-epoch 24-h concordance (Phillips 2017), ' + 'SRI=200·agreement/cases−100; NOT SD-of-midsleep', + ); +} +// --------------------------------------------------------------------------- + +typedef _Agg = Aggregator>; + +/// Everything observable, flattened. Compared EXACTLY (no tolerance). +List _fields(Metric m) { + final v = m.value; + return [ + m.present, + m.confidence, + m.tier, + m.inputs_used.join(','), + m.note, + v?.sri, + v?.days, + v?.cases, + v?.pairs.length, + if (v != null) + for (final p in v.pairs) ...[p.dayIndex, p.agreement, p.cases], + ]; +} + +final _eq = fieldsEqual>(_fields); + +void _same(Metric got, Metric want, String tag) { + // Bit-identical sri, not just ==. + if (want.value != null && got.value != null) { + expect(got.value!.sri.toString(), want.value!.sri.toString(), + reason: '$tag sri'); + } + expect(_fields(got), _fields(want), reason: tag); +} + +Epochs _epochs(List sw, [List? valid]) => Epochs( + Uint8List.fromList([for (final b in sw) b ? 1 : 0]), + valid: valid == null + ? null + : Uint8List.fromList([for (final b in valid) b ? 1 : 0]), + ); + +class _Rec { + _Rec(this.sw, this.valid); + final List sw; + final List? valid; + Epochs get epochs => _epochs(sw, valid); + int get n => sw.length; + @override + String toString() => + 'n=$n sw=${sw.map((b) => b ? 1 : 0).join()} ' + 'valid=${valid?.map((b) => b ? 1 : 0).join()}'; +} + +/// A record of [n] epochs: day-periodic sleep with [flipP] noise, optionally a +/// validity mask with holes (whole stretches and single epochs). +_Rec _record(Rng r, int n, int epd, {bool masked = false, double flipP = .1}) { + final base = [for (var e = 0; e < epd; e++) r.nextBool()]; + final sw = [ + for (var i = 0; i < n; i++) + r.nextBool(flipP) ? !base[i % epd] : base[i % epd] + ]; + List? valid; + if (masked) { + valid = [for (var i = 0; i < n; i++) !r.nextBool(.15)]; + if (n > 0 && r.nextBool(.4)) { + // A stretch of invalid epochs (a thin day). + final a = r.intIn(0, n - 1), len = r.intIn(1, epd + 2); + for (var i = a; i < a + len && i < n; i++) { + valid[i] = false; + } + } + } + return _Rec(sw, valid); +} + +void _allPaths( + _Agg a, Epochs e, List b, Metric want, String tag) { + _same(runChunked(a, e, b), want, '$tag chunks $b'); + _same(runChunked(a, e, b, copy: true), want, '$tag copies $b'); + _same(runChunkedRight(a, e, b), want, '$tag right $b'); + _same(runSliced(a, e, b), want, '$tag sliced $b'); +} + +void _checkRecord(Rng r, _Rec rec, int epd, int? floor, String tag, + {int slicings = 2}) { + final want = _referenceSri(rec.sw, epd, valid: rec.valid, minPairCases: floor); + // The public function (rewired) must still be the original. + _same(phillipsSri(rec.sw, epd, valid: rec.valid, minPairCases: floor), want, + '$tag phillipsSri'); + final a = sriAggregator(epd, minPairCases: floor); + final e = rec.epochs; + _same(a.run(e), want, '$tag run'); + for (var k = 0; k < slicings; k++) { + _allPaths(a, e, randomBounds(r, rec.n), want, tag); + } + // Day-aligned and one-off-day-aligned cuts exercise the day boundary. + final days = rec.n ~/ epd; + if (days >= 1) { + final aligned = [ + 0, + for (var d = 1; d <= days; d++) d * epd, + if (rec.n % epd != 0) rec.n, + ]; + _allPaths(a, e, aligned, want, '$tag aligned'); + final off = [ + 0, + for (var d = 1; d <= days; d++) + if (d * epd - 1 > 0) d * epd - 1, + rec.n, + ]; + _allPaths(a, e, off, want, '$tag off-by-one'); + } +} + +void main() { + group('Epochs batch', () { + Epochs mk(int n, {bool mask = false, int first = 0}) => Epochs( + Uint8List.fromList([for (var i = 0; i < n; i++) i % 3 == 0 ? 1 : 0]), + valid: mask + ? Uint8List.fromList([for (var i = 0; i < n; i++) i % 4 == 0 ? 0 : 1]) + : null, + firstIndex: first, + ); + + test('defaults and length', () { + final e = mk(5); + expect(e.length, 5); + expect(e.firstIndex, 0); + expect(e.valid, isNull); + expect(mk(0).length, 0); + }); + + test('copyRange cuts rows and shifts firstIndex', () { + final e = mk(10, mask: true); + final c = e.copyRange(3, 8); + expect(c.length, 5); + expect(c.firstIndex, 3); + expect(c.asleep, e.asleep.sublist(3, 8)); + expect(c.valid, e.valid!.sublist(3, 8)); + final cc = c.copyRange(2, 4); + expect(cc.firstIndex, 5); + expect(cc.asleep, e.asleep.sublist(5, 7)); + final shifted = mk(10, first: 100).copyRange(4, 10); + expect(shifted.firstIndex, 104); + }); + + test('copyRange of nothing, and of a batch without a mask', () { + final e = mk(6); + final empty = e.copyRange(4, 4); + expect(empty.length, 0); + expect(empty.firstIndex, 4); + expect(e.copyRange(0, 6).valid, isNull); + expect(e.copyRange(0, 6).length, 6); + }); + + test('copies are independent of the source, both ways', () { + final e = mk(8, mask: true); + final snapshot = Uint8List.fromList(e.asleep); + final snapshotValid = Uint8List.fromList(e.valid!); + final c = e.copyRange(2, 6); + e.asleep[3] ^= 1; + e.valid![4] ^= 1; + expect(c.asleep, snapshot.sublist(2, 6)); + expect(c.valid, snapshotValid.sublist(2, 6)); + final before = Uint8List.fromList(e.asleep); + c.asleep[0] ^= 1; + expect(e.asleep, before); + }); + + test('join of adjacent batches', () { + final e = mk(10, mask: true); + final j = e.copyRange(0, 4).join(e.copyRange(4, 10)); + expect(j.length, 10); + expect(j.firstIndex, 0); + expect(j.asleep, e.asleep); + expect(j.valid, e.valid); + final j2 = e.copyRange(2, 5).join(e.copyRange(5, 5)).join(e.copyRange(5, 9)); + expect(j2.firstIndex, 2); + expect(j2.asleep, e.asleep.sublist(2, 9)); + }); + + test('join result is independent of both inputs', () { + final e = mk(6, mask: true); + final left = e.copyRange(0, 3), right = e.copyRange(3, 6); + final j = left.join(right); + final want = Uint8List.fromList(j.asleep); + left.asleep[0] ^= 1; + right.asleep[0] ^= 1; + expect(j.asleep, want); + }); + + test('join with one side unmasked treats it as all valid', () { + final withMask = Epochs(Uint8List.fromList([1, 0]), + valid: Uint8List.fromList([0, 1])); + final plain = Epochs(Uint8List.fromList([1, 1]), firstIndex: 2); + expect(withMask.join(plain).valid, [0, 1, 1, 1]); + final plain0 = Epochs(Uint8List.fromList([1, 1])); + final maskedAfter = Epochs(Uint8List.fromList([0, 0]), + valid: Uint8List.fromList([1, 0]), firstIndex: 2); + expect(plain0.join(maskedAfter).valid, [1, 1, 1, 0]); + expect(plain0.join(Epochs(Uint8List(1), firstIndex: 2)).valid, isNull); + }); + + test('join of a non-adjacent batch asserts', () { + final e = mk(10); + expect(() => e.copyRange(0, 3).join(e.copyRange(4, 10)), + throwsA(isA())); + expect(() => e.copyRange(3, 6).join(e.copyRange(0, 3)), + throwsA(isA())); + }); + }); + + group('known results', () { + test('perfectly regular 3 days: sri 100, 2 pairs', () { + final sw = [for (var i = 0; i < 12; i++) (i % 4) < 2]; + final m = sriAggregator(4).run(_epochs(sw)); + _same(m, _referenceSri(sw, 4), 'regular'); + expect(m.value!.sri, 100.0); + expect(m.value!.days, 3); + expect(m.value!.cases, 8); + expect([for (final p in m.value!.pairs) p.dayIndex], [1, 2]); + }); + test('anti-phase 2 days: sri -100', () { + final sw = [true, true, false, false, false, false, true, true]; + final m = sriAggregator(4).run(_epochs(sw)); + expect(m.value!.sri, -100.0); + expect(m.value!.days, 2); + expect(m.confidence, 0.3); // (days-1)/7 clamped to 0.3..0.95 + }); + test('a trailing partial day is dropped at finish', () { + // 2 full days + 3 stray epochs that would disagree if they were used. + final sw = [true, false, true, false, true, false, true, false, + false, false, false]; + final m = sriAggregator(4).run(_epochs(sw)); + _same(m, _referenceSri(sw, 4), 'partial'); + expect(m.value!.days, 2); + expect(m.value!.cases, 4); + expect(m.value!.sri, 100.0); + }); + }); + + group('absent cases (notes included)', () { + final records = , List?, int, int?)>{ + 'empty': ([], null, 4, null), + 'one epoch': ([true], null, 4, null), + 'less than 2 days': ([for (var i = 0; i < 7; i++) i.isEven], null, 4, null), + 'exactly one day': ([true, false, true, false], null, 4, null), + '2 days minus one': ([for (var i = 0; i < 7; i++) true], null, 4, null), + 'no valid pair': ( + [for (var i = 0; i < 8; i++) true], + [true, true, false, false, false, false, true, true], + 4, + null + ), + 'mask all false': ( + [for (var i = 0; i < 12; i++) i.isEven], + List.filled(12, false), + 4, + 0 + ), + }; + records.forEach((name, rec) { + test(name, () { + final want = _referenceSri(rec.$1, + rec.$3, valid: rec.$2, minPairCases: rec.$4); + expect(want.present, isFalse); + final a = sriAggregator(rec.$3, minPairCases: rec.$4); + final e = _epochs(rec.$1, rec.$2); + _same(a.run(e), want, name); + _same(phillipsSri(rec.$1, rec.$3, valid: rec.$2, minPairCases: rec.$4), + want, name); + final r = Rng(name.hashCode & 0xffff); + for (var k = 0; k < 3; k++) { + _allPaths(a, e, randomBounds(r, e.length), want, name); + } + }); + }); + + test('the two notes are the two different absent reasons', () { + final short = sriAggregator(4).run(_epochs([true, false, true])); + expect(short.note, contains('≥2 full days of clock-aligned')); + final empty = sriAggregator(4).run(_epochs(List.filled(8, true), + List.filled(8, false))); + expect(empty.note, 'no valid epoch pairs for SRI'); + expect(short.confidence, 0); + expect(empty.confidence, 0); + }); + + test('phillipsSri with epochsPerDay <= 0 stays absent', () { + for (final epd in [0, -1, -1440]) { + final want = _referenceSri([true, false, true, false], epd); + _same(phillipsSri([true, false, true, false], epd), want, 'epd $epd'); + expect(want.present, isFalse); + } + }); + + test('empty batch finishes absent, also from an empty state', () { + final a = sriAggregator(4); + final want = _referenceSri([], 4); + _same(a.finish(a.empty()), want, 'empty state'); + _same(a.run(_epochs([])), want, 'empty batch'); + }); + }); + + group('minPairCases', () { + final r = Rng(0x5A1); + final rec = _record(r, 5 * 6 + 2, 6, masked: true, flipP: .3); + for (final floor in [null, 0, 1, 2, 3, 4, 6, 7, 100]) { + test('floor $floor', () { + _checkRecord(Rng(floor ?? -1), rec, 6, floor, 'floor $floor'); + }); + } + test('floor hides thin pairs but not their cases', () { + // Day 1 has 1 valid pair-case, day 2 has 4. + final sw = [for (var i = 0; i < 12; i++) i.isEven]; + final valid = [ + true, false, false, false, // day 0 + true, true, true, true, // day 1: only epoch 0 pairs with day 0 + true, true, true, true, // day 2 + ]; + final a = sriAggregator(4, minPairCases: 2).run(_epochs(sw, valid)); + _same(a, _referenceSri(sw, 4, valid: valid, minPairCases: 2), 'thin'); + expect(a.value!.cases, 5); + expect([for (final p in a.value!.pairs) p.dayIndex], [2]); + final z = sriAggregator(4, minPairCases: 0).run(_epochs(sw, valid)); + expect([for (final p in z.value!.pairs) p.dayIndex], [1, 2]); + }); + test('default floor is half the day (integer division)', () { + // epd 5 -> floor 2; a pair with 2 cases is kept, with 1 is not. + final sw = List.filled(15, true); + final valid = [ + true, true, true, true, true, + true, true, false, false, false, // 3 cases + true, false, false, false, false, // 0 cases with prev day mask... prev has 3 + ]; + final want = _referenceSri(sw, 5, valid: valid); + _same(sriAggregator(5).run(_epochs(sw, valid)), want, 'floor 2'); + }); + }); + + group('random parity (seeded)', () { + test('small epochsPerDay 4..12, lengths 0..7 days, masks, floors', () { + final r = Rng(0x5121); + for (var t = 0; t < 120; t++) { + final epd = r.intIn(4, 12); + final n = switch (r.intIn(0, 5)) { + 0 => r.intIn(0, 2 * epd + 1), // around the 2-day threshold + 1 => 2 * epd + r.intIn(0, epd - 1), // 2 days + partial + 2 => r.intIn(1, 7) * epd, // whole days + _ => r.intIn(0, 7 * epd + epd - 1), // anything, partial trailing day + }; + final rec = _record(r, n, epd, + masked: r.nextBool(.5), flipP: r.nextBool() ? .05 : .45); + final floor = switch (r.intIn(0, 3)) { + 0 => null, + 1 => 0, + 2 => r.intIn(1, epd), + _ => r.intIn(1, epd + 2), + }; + _checkRecord(r, rec, epd, floor, 't$t epd=$epd floor=$floor $rec'); + } + }); + + test('one 1440-epoch record of 3 days plus a partial fourth', () { + final r = Rng(0x1440); + for (final (n, masked) in [(3 * 1440, true), (3 * 1440 + 700, false)]) { + final rec = _record(r, n, 1440, masked: masked, flipP: .08); + _checkRecord(r, rec, 1440, null, 'n=$n', slicings: 2); + _checkRecord(r, rec, 1440, 100, 'n=$n floor 100', slicings: 1); + } + }); + + test('singleton chunks (state per epoch)', () { + final r = Rng(0x5106); + final rec = _record(r, 3 * 5 + 2, 5, masked: true); + final want = _referenceSri(rec.sw, 5, valid: rec.valid); + final a = sriAggregator(5); + _allPaths(a, rec.epochs, singletonBounds(rec.n), want, 'singletons'); + }); + }); + + test('persistable: a mid-stream checkpoint resumes to the one-pass result', () { + final b = _record(Rng(0xC0DE), 53, 4, masked: true, flipP: .3).epochs; + expectCheckpointResumes(sriAggregator(4), b, equals: _eq); + }); + + group('lawful (exact equality)', () { + final r = Rng(0x1A25); + List cases(int epd, {int count = 14}) => [ + _epochs([]), + _epochs([true]), + _epochs([true, false]), + for (var i = 0; i < count; i++) + _record(r, r.intIn(0, 4 * epd + epd ~/ 2), epd, + masked: r.nextBool(), flipP: .3) + .epochs, + ]; + for (final floor in [null, 0, 2]) { + test('epochsPerDay 4, floor $floor', + () => expectLawful(sriAggregator(4, minPairCases: floor), cases(4), + equals: _eq)); + } + test('epochsPerDay 7', + () => expectLawful(sriAggregator(7), cases(7, count: 10), equals: _eq)); + test('epochsPerDay 1', () { + // Degenerate day length: every epoch is its own day. + expectLawful(sriAggregator(1), cases(1, count: 10), equals: _eq); + }); + }); + + group('malformed input', () { + // Runs the frozen reference and the rewired phillipsSri on the same input: + // equal results, or the same exception type when the reference throws. + void sameAsReference( + String name, List sw, int epd, List? valid) { + test(name, () { + Metric? want; + Object? wantError; + try { + want = _referenceSri(sw, epd, valid: valid); + } catch (e) { + wantError = e; + } + if (wantError != null) { + expect(() => phillipsSri(sw, epd, valid: valid), + throwsA(predicate((e) => e.runtimeType == wantError.runtimeType)), + reason: 'reference threw $wantError'); + } else { + _same(phillipsSri(sw, epd, valid: valid), want!, name); + } + }); + } + + List pattern(int n) => [for (var i = 0; i < n; i++) i % 3 == 0]; + + sameAsReference('trailing partial day, mask one short of n (spec case)', + List.filled(9, true), 4, List.filled(8, true)); + sameAsReference('mask longer than n', pattern(8), 4, List.filled(12, true)); + sameAsReference('mask longer than n, with falses', pattern(9), 4, + [for (var i = 0; i < 20; i++) i % 5 != 0]); + sameAsReference('mask = complete span, partial day present', pattern(10), 4, + [for (var i = 0; i < 8; i++) i != 2]); + sameAsReference('mask shorter than n but covering complete days', + pattern(11), 4, [for (var i = 0; i < 9; i++) i != 5]); + sameAsReference('mask shorter than complete days, absent case', + pattern(8), 4, List.filled(4, false)); + sameAsReference('mask shorter than complete days, throwing case', + pattern(8), 4, List.filled(4, true)); + sameAsReference('mask shorter than complete days, throws later in the day', + pattern(12), 4, [for (var i = 0; i < 6; i++) i < 2 ? true : i != 4]); + sameAsReference('empty mask', pattern(8), 4, []); + + test('the shorter-mask cases really split into absent and throwing', () { + expect(_referenceSri(List.filled(8, true), 4, valid: List.filled(4, false)).present, + isFalse); + expect(() => _referenceSri(List.filled(8, true), 4, valid: List.filled(4, true)), + throwsRangeError); + }); + }); + + group('sriAggregator with epochsPerDay <= 0', () { + for (final epd in [0, -1, -5]) { + test('epochsPerDay $epd finishes as the absent phillipsSri metric', () { + final inputs = >[ + [], + [true], + List.filled(10, true), + [for (var i = 0; i < 40; i++) i % 3 == 0], + ]; + for (final sw in inputs) { + final want = _referenceSri(sw, epd); + expect(want.present, isFalse); + final got = sriAggregator(epd).run(_epochs(sw)); + _same(got, want, 'epd $epd n ${sw.length}'); + } + }); + test('epochsPerDay $epd never throws, masked or with minPairCases', () { + final sw = [for (var i = 0; i < 12; i++) i.isEven]; + final got = sriAggregator(epd, minPairCases: 2) + .run(_epochs(sw, List.filled(12, true))); + _same(got, _referenceSri(sw, epd, minPairCases: 2), 'masked'); + }); + } + }); + + group('checkpoint relationships (epoch count, window length, day pairs)', () { + // 14 epochs of 4 per day: days 1..3 hold pairs (day 3 is a partial day). + final agg = sriAggregator(4); + final b = _record(Rng(0x5A1), 14, 4, masked: false, flipP: .3).epochs; + // State: [n, lookback]; lookback inner is {dayKey: {agreement, cases}}. + Map lookback(Object? s) => (s as List)[1] as Map; + Map days(Object? s) => lookback(s)['inner'] as Map; + + test('the epoch count must equal the lookback row count', () { + expectDamagedCheckpointRefused(agg, b, (s) { + (s as List)[0] = 9; + return s; + }); + expectDamagedCheckpointRefused(agg, b, (s) { + (s as List)[0] = 20; + return s; + }); + }); + test('a day with more pair cases than epochs per day is refused', () { + expectDamagedCheckpointRefused(agg, b, (s) { + final d = days(s).values.first as Map; + d['cases'] = 5; + return s; + }); + }); + test('a day outside the later days of the record is refused', () { + // 14 epochs reach day 3; day 0 has no earlier day to compare with and + // day 9 is past the end. + for (final key in ['0', '9', '-1']) { + expectDamagedCheckpointRefused(agg, b, (s) { + final d = days(s); + d[key] = d.values.first; + return s; + }, reason: 'day $key'); + } + }); + test('pair cases beyond the epochs that can be compared are refused', () { + // 14 epochs allow at most 14 - 4 = 10 comparisons in total. + expectDamagedCheckpointRefused(agg, b, (s) { + final d = days(s); + for (final k in ['1', '2', '3']) { + (d[k] as Map) + ..['cases'] = 4 + ..['agreement'] = 4; + } + return s; + }); + }); + test('a consistent checkpoint restores', () { + expect( + decodeCheckpoint(agg, checkpointOf(agg, b), + kind: 'test', version: 1), + isNotNull); + }); + }); +} diff --git a/test/aggregate/sum_of_test.dart b/test/aggregate/sum_of_test.dart new file mode 100644 index 0000000..0b9e0b8 --- /dev/null +++ b/test/aggregate/sum_of_test.dart @@ -0,0 +1,149 @@ +// SumOf(col, f): Neumaier-compensated sum of f(x) over the rows where x is +// not NaN and f(x) is not NaN. +// +// Decisions the implementation follows: +// * f is never called with NaN (a NaN input row is skipped before f). An +// infinite input IS passed to f. +// * f returning NaN skips that row (it does not poison the sum). +// * The result is a double; nothing counted (or an empty batch) is exactly 0.0. +// * Compensation is the same as Sum: combine adds the other state's sum and +// correction, so the compensation survives chunking and addSlice. +// * Exported from package:openstrap_analytics/aggregate.dart. + +import 'dart:math' as math; +import 'dart:typed_data'; + +import 'package:openstrap_analytics/aggregate.dart'; +import 'package:test/test.dart'; + +import '../support/aggregator_laws.dart'; +import '../support/property.dart' show Rng; + +Float64List _v(DoubleColumn c) => c.values; + +const nan = double.nan; + +final _tol = fieldsEqual((x) => [x], relTol: 1e-12, absTol: 1e-6); + +SumOf _sumOf(double Function(double) f) => + SumOf(_v, f); + +void main() { + final id = _sumOf((x) => x); + final square = _sumOf((x) => x * x); + // Square root of positives; NaN (skip) for everything else. + final rootPos = _sumOf((x) => x > 0 ? math.sqrt(x) : nan); + + group('values', () { + test('plain sums of f(x)', () { + expect(id.run(col([1.5, 2.5, -1.0])), 3.0); + expect(square.run(col([1, 2, 3])), 14.0); + expect(_sumOf((x) => 2 * x + 1).run(col([1, 2, 3])), 15.0); + }); + test('empty is exactly 0.0, as a double', () { + final e = square.run(col([])); + expect(e, 0.0); + expect(e, isA()); + expect(square.finish(square.empty()), 0.0); + }); + test('NaN inputs are skipped, all-NaN is 0.0', () { + expect(square.run(col([1, nan, 2, nan])), 5.0); + expect(square.run(col([nan, nan])), 0.0); + }); + test('f is never called with NaN', () { + final seen = []; + final spy = _sumOf((x) { + seen.add(x); + return x; + }); + expect(spy.run(col([1, nan, 2, nan, nan, 3])), 6.0); + expect(seen, [1.0, 2.0, 3.0]); + }); + test('f returning NaN skips the row', () { + expect(rootPos.run(col([4, -1, 9, 0, nan, 16])), 9.0); + expect(rootPos.run(col([-1, -2, 0])), 0.0); + expect(_sumOf((_) => nan).run(col([1, 2, 3])), 0.0); + }); + test('an infinite input reaches f', () { + final finiteOnly = _sumOf((x) => x.isFinite ? x : nan); + expect(finiteOnly.run(col([1, double.infinity, 2, -double.infinity])), + 3.0); + expect(id.run(col([1, double.infinity])), double.infinity); + }); + test('honours the range', () { + final c = col([1, 2, 3, 4, 5]); + expect(square.finish(square.fold(c, 1, 4)), 4.0 + 9 + 16); + expect(square.finish(square.fold(c, 2, 2)), 0.0); + }); + test('reads the column through the getter, on any batch', () { + final second = SumOf>( + (b) => Float64List.fromList([for (final r in b.rows) r.$2]), + (x) => x * 10); + expect(second.runList([(1, nan), (2, 3), (3, 4)]), 70.0); + }); + }); + + group('compensation', () { + test('ten times 0.1 is exactly 1.0', () { + expect(id.run(col(List.filled(10, 0.1))), 1.0); + }); + test('Neumaier: big, small, minus big', () { + expect(id.run(col([1e16, 1.0, -1e16])), 1.0); + expect(id.run(col([1.0, 1e16, -1e16])), 1.0); + }); + test('compensation applies to f(x), not x', () { + // f(1e8) = 1e16, f(-1e8) = -1e16; the 1.0 would vanish in a naive sum. + final signedSquare = _sumOf((x) => x >= 0 ? x * x : -(x * x)); + expect(signedSquare.run(col([1e8, 1, -1e8])), 1.0); + }); + test('survives every split and repeated addSlice', () { + final c = col([1e16, 1.0, -1e16, 1.0]); + for (var k = 0; k <= 4; k++) { + final s = id.combine(id.fold(c, 0, k), id.fold(c, k, 4)); + expect(id.finish(s), 2.0, reason: 'split at $k'); + var one = id.addSlice(id.empty(), c, 0, k); + one = id.addSlice(one, c, k, 4); + expect(id.finish(one), 2.0, reason: 'sliced at $k'); + } + }); + test('finish does not mutate: finish twice, keep adding', () { + final c = col([1, 2, 3, 4]); + var s = id.fold(c, 0, 3); + expect(id.finish(s), 6.0); + expect(id.finish(s), 6.0); + s = id.addSlice(s, c, 3, 4); + expect(id.finish(s), 10.0); + }); + }); + + group('random parity (seeded)', () { + test('matches Sum of the mapped column under random chunkings', () { + final r = Rng(0x50F); + final sum = Sum(_v); + for (final c in randomColumns(21, count: 25)) { + final mapped = DoubleColumn(Float64List.fromList([ + for (final x in c.values) x.isNaN || x <= 0 ? nan : math.sqrt(x) + ])); + final want = sum.run(mapped); + for (var t = 0; t < 3; t++) { + final b = randomBounds(r, c.length); + for (final got in [ + runChunked(rootPos, c, b), + runChunked(rootPos, c, b, copy: true), + runChunkedRight(rootPos, c, b), + runSliced(rootPos, c, b), + ]) { + expect(_tol(got, want), isTrue, reason: '$got vs $want, $b'); + } + } + } + }); + }); + + group('lawful', () { + final cols = randomColumns(22); + test('identity f', () => expectLawful(id, cols, equals: _tol)); + test('square', () => expectLawful(square, cols, equals: _tol)); + test('f that skips rows', () => expectLawful(rootPos, cols, equals: _tol)); + }); +} diff --git a/test/aggregate/time_in_zone_aggregator_test.dart b/test/aggregate/time_in_zone_aggregator_test.dart new file mode 100644 index 0000000..9b1aa52 --- /dev/null +++ b/test/aggregate/time_in_zone_aggregator_test.dart @@ -0,0 +1,504 @@ +// timeInZoneAggregator must agree with HeartRateZones.timeInZone for the same +// (time-sorted) rows under any slicing or chunking, and obey the aggregator +// laws. +// +// Contract used here (docs: r6-time-in-zone.md): +// * HrStream(Float64List tsMs, Float64List hr) is a Batch; rows are +// sorted by tsMs; a row is valid when hr > 0 (as HrSample.valid). +// * timeInZoneAggregator(zoneSet) is an +// Aggregator<_, HrStream, TimeInHeartRateZone?>, all exported from +// package:openstrap_analytics/aggregate.dart. +// * The result is null exactly when HeartRateZones.timeInZone is null (the +// sampleCadenceSeconds rule, including fewer than 2 rows). Otherwise every +// zone's seconds and belowZone1 are within relTol 1e-12 / absTol 1e-9. +// * The median m is bit-identical to sampleCadenceSeconds'. Pinned here by +// streams whose only valid row is the LAST one: the only credit is the tail, +// so the zone total is m, compared with == (a single add to 0.0 is exact). +// * HOOK the implementer must add, exported from the same library: +// int timeInZoneStateEntries(Object? state) +// = total number of histogram entries (all histograms, all zones) held by a +// state returned by agg.fold(...). Must not throw on agg.empty() (returns 0 +// or a small number). +// * Duplicate-timestamp rows in the LARGE random streams carry the same hr as +// the row they duplicate: the reference re-sorts with an unstable sort, so +// tie order (and so which hr gets the fallback credit) is only defined for +// streams of <= 32 rows. The hand-written duplicate case uses 5 rows with +// differing hr. + +import 'dart:typed_data'; + +import 'package:openstrap_analytics/aggregate.dart'; +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +import '../support/checkpoint_support.dart'; +import '../support/aggregator_laws.dart'; +import '../support/property.dart' show Rng; + +typedef _Agg = Aggregator; + +final HeartRateZoneSet _zones = HeartRateZones.zonesFromMaxHr(200); +final _Agg _agg = timeInZoneAggregator(_zones); + +List _fields(TimeInHeartRateZone? t) => + t == null ? [null] : [...t.seconds, t.belowZone1]; + +final _eq = fieldsEqual(_fields, + relTol: 1e-12, absTol: 1e-9); + +/// A raw stream: timestamps in ms and hr. +class _S { + _S(this.ts, this.hr); + final List ts; + final List hr; + int get n => ts.length; + HrStream get stream => + HrStream(Float64List.fromList(ts), Float64List.fromList(hr)); + List get samples => + [for (var i = 0; i < n; i++) HrSample(ts[i], hr[i])]; + double? get cadence => + sampleCadenceSeconds([for (final t in ts) t / 1000.0]); +} + +TimeInHeartRateZone? _want(_S s) => + HeartRateZones.timeInZone(s.samples, _zones); + +void _same(TimeInHeartRateZone? got, TimeInHeartRateZone? want, String tag) { + expect(_eq(got, want), isTrue, + reason: '$tag\n got ${_fields(got)}\nwant ${_fields(want)}'); + expect(got == null, want == null, reason: '$tag null-ness'); +} + +/// Every way of evaluating the aggregator against the reference. +void _allPaths(_S s, List b, String tag) { + final want = _want(s); + final st = s.stream; + _same(_agg.run(st), want, '$tag whole'); + _same(runChunked(_agg, st, b), want, '$tag chunks $b'); + _same(runChunked(_agg, st, b, copy: true), want, '$tag copies $b'); + _same(runChunkedRight(_agg, st, b), want, '$tag right $b'); + _same(runSliced(_agg, st, b), want, '$tag sliced $b'); +} + +void _parity(_S s, String tag, Rng r) { + for (var k = 0; k < 3; k++) { + _allPaths(s, randomBounds(r, s.n), tag); + } + _allPaths(s, singletonBounds(s.n), '$tag singletons'); +} + +// ---- generators ------------------------------------------------------------ + +/// A bpm that hits every zone, the boundaries exactly, off-skin and above max. +double _bpm(Rng r) { + final k = r.nextDouble(); + if (k < .08) return 0; + if (k < .30) { + final z = _zones.zones[r.intIn(0, 4)]; + return r.nextBool() ? z.lower : z.upper; + } + if (k < .38) return 205 + r.nextDouble() * 30; // above zone 5 + if (k < .46) return 30 + r.nextDouble() * 60; // below zone 1 + return 90 + r.nextDouble() * 110; +} + +/// [n] rows at nominal [stepMs] with [jitterMs] jitter, a [dropP] chance of a +/// long [dropMs] gap, a [dupP] chance of a repeated timestamp (same hr, see +/// the header), an [offP] chance of an off-skin row. +_S _gen(Rng r, int n, + {double stepMs = 1000, + int jitterMs = 20, + double dropP = 0, + double dropMs = 600000, + double dupP = 0, + double offP = 0, + bool fractional = false}) { + final ts = []; + final hr = []; + var t = 1.7e12 + r.intIn(0, 999999); + for (var i = 0; i < n; i++) { + if (i > 0 && r.nextBool(dupP)) { + ts.add(ts.last); + hr.add(hr.last); + continue; + } + if (i > 0) { + t += r.nextBool(dropP) + ? dropMs + : stepMs + r.intIn(-jitterMs, jitterMs) + + (fractional ? r.nextDouble() : 0); + } + ts.add(t); + hr.add(r.nextBool(offP) ? 0 : _bpm(r)); + } + return _S(ts, hr); +} + +/// The input families of the spec, by name. +Map _families() => { + '1 Hz jitter': (r, n) => _gen(r, n), + '1 Hz fractional ms': (r, n) => _gen(r, n, fractional: true), + 'dropouts': (r, n) => _gen(r, n, dropP: .05), + 'duplicate timestamps': (r, n) => _gen(r, n, dupP: .15), + 'off-skin rows': (r, n) => _gen(r, n, offP: .3), + 'slow 5 s device': (r, n) => _gen(r, n, stepMs: 5000, jitterMs: 100), + 'dropouts + dups + off-skin': (r, n) => + _gen(r, n, dropP: .05, dupP: .1, offP: .15, fractional: true), + 'cadence > 300 s': (r, n) => _gen(r, n, stepMs: 400000, jitterMs: 1000), + 'fails the mode rule': (r, n) { + // Gaps cycle 1 s, 4 s, 10 s: median 4 s, only a third within 2x. + if (n == 0) return _S([], []); + final ts = [1.7e12]; + for (var i = 1; i < n; i++) { + ts.add(ts.last + const [1000.0, 4000.0, 10000.0][i % 3]); + } + return _S(ts, [for (var i = 0; i < n; i++) _bpm(r)]); + }, + }; + +void main() { + group('hand-worked', () { + test('one row, empty: null', () { + expect(_agg.run(HrStream(Float64List(0), Float64List(0))), isNull); + expect(_agg.run(_S([5000], [120]).stream), isNull); + expect(_want(_S([5000], [120])), isNull); + expect(_agg.finish(_agg.empty()), isNull); + }); + test('two rows: the tail gets m', () { + final s = _S([0, 1000], [110, 130]); + final want = _want(s)!; + final got = _agg.run(s.stream)!; + _same(got, want, 'two rows'); + expect(got.secondsInZone(1), closeTo(1, 1e-12)); + expect(got.secondsInZone(2), closeTo(1, 1e-12)); + }); + test('two rows, equal timestamps: no positive gap, null', () { + final s = _S([1000, 1000], [110, 130]); + expect(_want(s), isNull); + expect(_agg.run(s.stream), isNull); + }); + test('five regular rows, one per zone', () { + final s = _S([0, 60000, 120000, 180000, 240000], + [110, 130, 150, 170, 190]); + final got = _agg.run(s.stream)!; + for (var z = 1; z <= 5; z++) { + expect(got.secondsInZone(z), closeTo(60, 1e-9)); + } + expect(got.belowZone1, 0); + }); + test('long gap is capped at m; duplicate (gap 0) gets m', () { + final s = _S([0, 1000, 2000, 2000, 700000], [130, 150, 190, 130, 190]); + _same(_agg.run(s.stream), _want(s), 'capped'); + for (final b in [ + [0, 5], + [0, 2, 3, 5], + [0, 1, 1, 4, 5], + [0, 3, 4, 4, 5] + ]) { + _same(runChunked(_agg, s.stream, b), _want(s), 'chunks $b'); + _same(runSliced(_agg, s.stream, b), _want(s), 'sliced $b'); + } + }); + test('zone boundaries: exactly lower / upper, above zone 5, below zone 1', + () { + final hr = [ + for (final z in _zones.zones) ...[z.lower, z.upper], + _zones.maxHr + 40, + _zones.zones.first.lower - 0.5, + 0, + ]; + final s = _S([for (var i = 0; i < hr.length; i++) i * 1000.0], hr); + final want = _want(s)!; + _same(_agg.run(s.stream), want, 'boundaries'); + expect(want.belowZone1, greaterThan(0)); + expect(want.seconds[4], greaterThan(0)); + for (final b in [ + [0, hr.length], + [0, 1, 2, 5, hr.length], + [0, 4, 4, hr.length] + ]) { + _same(runChunked(_agg, s.stream, b), want, 'chunks $b'); + _same(runSliced(_agg, s.stream, b), want, 'sliced $b'); + } + }); + }); + + group('null exactly when the reference is null', () { + test('> 300 s cadence', () { + final s = _gen(Rng(1), 12, stepMs: 400000, jitterMs: 1000); + expect(_want(s), isNull); + _parity(s, '>300', Rng(2)); + expect(_agg.run(s.stream), isNull); + }); + test('stream failing the mode rule', () { + final s = _families()['fails the mode rule']!(Rng(3), 30); + expect(_want(s), isNull); + _parity(s, 'mode', Rng(4)); + expect(_agg.run(s.stream), isNull); + }); + test('exactly 300 s cadence is accepted, 301 s is not', () { + final ok = _S([for (var i = 0; i < 6; i++) i * 300000.0], + [100, 120, 140, 160, 180, 100]); + final no = _S([for (var i = 0; i < 6; i++) i * 301000.0], + [100, 120, 140, 160, 180, 100]); + expect(_want(ok), isNotNull); + expect(_want(no), isNull); + _parity(ok, '300', Rng(5)); + _parity(no, '301', Rng(6)); + }); + test('all timestamps identical: null', () { + final s = _S([5, 5, 5, 5], [100, 100, 100, 100]); + expect(_want(s), isNull); + _parity(s, 'dups', Rng(7)); + }); + }); + + group('random parity (seeded)', () { + var fam = 0; + _families().forEach((name, make) { + final seed = 0x7a11 + fam++; + test(name, () { + final r = Rng(seed); + var nonNull = 0; + for (var t = 0; t < 14; t++) { + final n = r.nextBool(.2) ? r.intIn(0, 4) : r.intIn(5, 160); + final s = make(r, n); + if (_want(s) != null) nonNull++; + _parity(s, '$name t$t n=$n', r); + } + if (!name.contains('>') && !name.contains('mode')) { + expect(nonNull, greaterThan(5), reason: 'generator too degenerate'); + } + }); + }); + + test('one-row and two-row chunks of a long stream', () { + final r = Rng(0x51); + final s = _gen(r, 90, dropP: .05, offP: .1, fractional: true); + for (final b in [ + [0, 1, 2, 90], + [0, 88, 89, 90], + [0, 45, 45, 45, 90], + [0, 0, 90, 90], + ]) { + _allPaths(s, b, 'edge $b'); + } + }); + }); + + group('cadence median is bit-identical to sampleCadenceSeconds', () { + // Only the last row is valid, so the sole credit is the tail: the zone + // total is exactly m (0.0 + m), compared with == on the double. + double total(TimeInHeartRateZone t) => + t.seconds.fold(t.belowZone1, (a, b) => a + b); + + _S tailOnly(Rng r, int n, {bool fractional = true, double step = 1000}) { + final s = _gen(r, n, + stepMs: step, jitterMs: 37, dropP: .05, fractional: fractional); + for (var i = 0; i < n - 1; i++) { + s.hr[i] = 0; + } + s.hr[n - 1] = 140; + return s; + } + + test('tail seconds == sampleCadenceSeconds, random streams and cuts', () { + final r = Rng(0xb17); + var checked = 0; + for (var t = 0; t < 40; t++) { + // Both even and odd gap counts (interpolated and exact medians). + final n = 2 + r.intIn(0, 120); + final s = tailOnly(r, n, step: r.nextBool() ? 1000 : 5000); + final m = s.cadence; + if (m == null) continue; + checked++; + final st = s.stream; + final b = randomBounds(r, n); + expect(total(_agg.run(st)!) == m, isTrue, + reason: 't$t whole ${total(_agg.run(st)!)} vs $m'); + expect(total(runChunked(_agg, st, b)!) == m, isTrue, + reason: 't$t chunks $b'); + expect(total(runChunked(_agg, st, b, copy: true)!) == m, isTrue, + reason: 't$t copies $b'); + expect(total(runChunkedRight(_agg, st, b)!) == m, isTrue, + reason: 't$t right $b'); + expect(total(runSliced(_agg, st, b)!) == m, isTrue, + reason: 't$t sliced $b'); + } + expect(checked, greaterThan(25)); + }); + + test('even gap count: the interpolated median, odd: the middle gap', () { + // gaps 1,2 -> 1.5 ; gaps 1,2,3 -> 2 (all inside the mode window). + final even = _S([0, 1000, 3000], [0, 0, 150]); + expect(total(_agg.run(even.stream)!) == 1.5, isTrue); + final odd = _S([0, 1000, 3000, 6000], [0, 0, 0, 150]); + expect(total(_agg.run(odd.stream)!) == 2.0, isTrue); + expect(even.cadence, 1.5); + expect(odd.cadence, 2.0); + }); + }); + + group('state size', () { + test('10,000 rows at 1 Hz with +-20 ms jitter: < 400 histogram entries', + () { + final r = Rng(0x5eed); + final s = _gen(r, 10000); + final entries = timeInZoneStateEntries(_agg.fold(s.stream)); + expect(entries, lessThan(400)); + expect(entries, greaterThan(0)); + _same(_agg.run(s.stream), _want(s), 'big'); + }); + test('does not grow with rows: 40,000 rows, 7 chunks combined', () { + final r = Rng(0x5eee); + final s = _gen(r, 40000); + final st = s.stream; + final b = [0, 5000, 5001, 12000, 20000, 20000, 33333, 40000]; + var acc = _agg.fold(st, b[0], b[1]); + for (var k = 1; k + 1 < b.length; k++) { + acc = _agg.combine(acc, _agg.fold(st, b[k], b[k + 1])); + } + expect(timeInZoneStateEntries(acc), lessThan(400)); + _same(_agg.finish(acc), _want(s), 'combined'); + }); + test('empty state is tiny', () { + expect(timeInZoneStateEntries(_agg.empty()), lessThan(10)); + }); + }); + + test('persistable: a mid-stream checkpoint resumes to the one-pass result', () { + expectCheckpointResumes(_agg, _gen(Rng(0xC0DE), 45, dropP: .1, offP: .2).stream, + equals: _eq); + }); + + group('lawful', () { + final r = Rng(0x1a3); + final cases = [ + HrStream(Float64List(0), Float64List(0)), + _S([1000], [120]).stream, + _S([0, 1000], [120, 0]).stream, + _S([0, 1000], [0, 150]).stream, + _S([0, 0], [100, 100]).stream, + _gen(r, 30).stream, + _gen(r, 45, dropP: .1, offP: .2).stream, + _gen(r, 45, dupP: .2, fractional: true).stream, + _gen(r, 25, stepMs: 5000, jitterMs: 100).stream, + _families()['fails the mode rule']!(r, 25).stream, + for (var i = 0; i < 15; i++) _gen(r, r.intIn(0, 40), offP: .1).stream, + ]; + test('every law holds', () => expectLawful(_agg, cases, equals: _eq)); + }); + + group('checkpoint histograms are consistent with the stored rows', () { + // Three rows one second apart at 150 bpm: two pairs, one tail. + final three = _S([0, 1000, 2000], [150, 150, 150]).stream; + // One row: no pairs, only a tail. + final one = _S([5000], [150]).stream; + // Two rows: one pair. + final two = _S([0, 1000], [150, 150]).stream; + + // State: {windows: {length, first, last, inner: {cadence, credits}}, + // hasTail, tailTimestampMs, tailHr}. Histograms are [[gap, n], ..]. + Map inner(Object? s) => ((s as Map)['windows'] as Map)['inner'] as Map; + List cadence(Object? s) => inner(s)['cadence'] as List; + List credits(Object? s) => inner(s)['credits'] as List; + // The (single) non-empty credit histogram. + List credit(Object? s) => + credits(s).cast().firstWhere((h) => h.isNotEmpty); + + void refuses(HrStream batch, void Function(Object? s) damage, + [String? why]) { + expectDamagedCheckpointRefused(_agg, batch, (s) { + damage(s); + return s; + }, reason: why); + } + + test('a good checkpoint has the shape these tests damage', () { + final s = checkpointOf(_agg, three)['state']; + expect(cadence(s), [ + [1.0, 2] + ]); + expect(credit(s), [ + [1.0, 2] + ]); + }); + + test('review case: histogram multiplicities inflated past the pairs', () { + refuses(two, (s) => cadence(s)[0][1] = 300); + refuses(two, (s) => credit(s)[0][1] = 300); + refuses(three, (s) => cadence(s)[0][1] = 3); + refuses(three, (s) => credit(s)[0][1] = 3); + }); + test('credits spread over several zones still sum to at most the pairs', + () { + refuses(three, (s) { + // Two pairs exist; 1 + 2 credits across two zones is three. + final h = credit(s); + final other = credits(s).cast().firstWhere((x) => x.isEmpty); + h[0][1] = 2; + other.add([1.0, 1]); + }); + }); + test('totals at exactly the pairs are accepted', () { + for (final batch in [two, three]) { + final good = checkpointOf(_agg, batch); + expect(decodeCheckpoint(_agg, good, kind: 'test', version: 1), + isNotNull); + } + }); + test('counts whose sum overflows an int are refused', () { + refuses(three, (s) { + cadence(s) + ..[0][1] = 0x7fffffffffffffff + ..add([2.0, 5]); + }); + refuses(three, (s) { + credit(s) + ..[0][1] = 0x7fffffffffffffff + ..add([2.0, 5]); + }); + }); + test('a histogram with one row behind it must be empty (no pairs)', () { + refuses(one, (s) => cadence(s).add([1.0, 1])); + refuses(one, (s) => credits(s)[0].add([1.0, 1])); + final empty = HrStream(Float64List(0), Float64List(0)); + expect(decodeCheckpoint( + _agg, checkpointOf(_agg, empty), kind: 'test', version: 1), + isNotNull); + }); + + test('review case: cadence [["NaN", 2]] is refused', () { + refuses(three, (s) => cadence(s) + ..clear() + ..add(['NaN', 2])); + }); + test('cadence keys must be finite and positive', () { + for (final key in ['NaN', 'Infinity', '-Infinity', 0.0, -1.0, -0.5]) { + refuses(three, (s) => cadence(s)[0][0] = key, 'cadence key $key'); + } + }); + test('credit keys must be finite; zero and negative are fine', () { + for (final key in ['NaN', 'Infinity', '-Infinity']) { + refuses(three, (s) => credit(s)[0][0] = key, 'credit key $key'); + } + // A zero or negative gap credits one cadence (the fallback), so these + // are reachable from non-monotonic timestamps and must still restore. + final back = _S([2000, 1000], [150, 150]).stream; // gap -1 s + final dup = _S([1000, 1000], [150, 150]).stream; // gap 0 + for (final batch in [back, dup]) { + final good = checkpointOf(_agg, batch); + expect(decodeCheckpoint(_agg, good, kind: 'test', version: 1), + isNotNull); + } + }); + + test('the tail exists exactly when there are rows', () { + refuses(three, (s) => (s as Map)['hasTail'] = false); + refuses(two, (s) => (s as Map)['hasTail'] = false); + refuses(one, (s) => (s as Map)['hasTail'] = false); + final empty = HrStream(Float64List(0), Float64List(0)); + refuses(empty, (s) => (s as Map)['hasTail'] = true); + }); + }); +} diff --git a/test/aggregate/trimp_aggregator_test.dart b/test/aggregate/trimp_aggregator_test.dart new file mode 100644 index 0000000..3065bd6 --- /dev/null +++ b/test/aggregate/trimp_aggregator_test.dart @@ -0,0 +1,231 @@ +// banisterTrimpAggregator must agree with banisterTrimp for the same rows under +// any slicing or chunking, and obey the aggregator laws. +// +// Contract used here: +// * banisterTrimpAggregator({restingHr, maxHr, sex}) is an +// Aggregator<_, DoubleColumn, Metric>; the column is the per-minute +// HR (NaN = missing). Exported from package:openstrap_analytics/aggregate.dart. +// * Anchors null / NaN / infinite, or maxHr <= restingHr: absent, for any rows +// (the note, tier and inputs_used equal banisterTrimp's). +// * Valid anchors and ZERO rows: absent (banisterTrimp's hrPerMin.isEmpty). +// * Valid anchors and at least one row, all of them non-finite or <= 0: +// PRESENT with value 0.0 and confidence 0.6. "Zero rows" means length 0 of +// the whole batch, not "no finite row". +// * value within relTol 1e-12; present, confidence, tier, inputs_used and +// note equal. +// * Constructing the aggregator never throws, whatever the anchors are. + +import 'dart:math' as math; +import 'dart:typed_data'; + +import 'package:openstrap_analytics/aggregate.dart'; +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +import '../support/checkpoint_support.dart'; +import '../support/aggregator_laws.dart'; +import '../support/property.dart' show Rng; + +const nan = double.nan; +const inf = double.infinity; + +typedef _Agg = Aggregator>; + +List _fields(Metric m) => [ + m.present, + m.value, + m.confidence, + m.tier, + m.inputs_used.join(','), + m.note, + ]; + +final _eq = fieldsEqual>(_fields, relTol: 1e-12); + +void _same(Metric got, Metric want, String tag) { + expect(_eq(got, want), isTrue, + reason: '$tag\n got ${_fields(got)}\nwant ${_fields(want)}'); +} + +_Agg _agg(double? rest, double? max, Sex sex) => + banisterTrimpAggregator(restingHr: rest, maxHr: max, sex: sex); + +Metric _want(DoubleColumn c, double? rest, double? max, Sex sex) => + banisterTrimp(c.values, restingHr: rest, maxHr: max, sex: sex); + +double _hr(Rng r) { + final k = r.nextDouble(); + if (k < .04) return 0; + if (k < .08) return -r.nextDouble() * 50; + if (k < .10) return inf; + if (k < .12) return -inf; + if (k < .30) return 30 + r.nextDouble() * 30; // around / below resting + if (k < .35) return 190 + r.nextDouble() * 40; // above max + return 50 + r.nextDouble() * 140; +} + +DoubleColumn _hrColumn(Rng r, int n, {double nanP = .1}) => + DoubleColumn(Float64List.fromList( + [for (var i = 0; i < n; i++) r.nextBool(nanP) ? nan : _hr(r)])); + +void main() { + group('absent cases', () { + final rows = col([60, 100, 150, 120]); + final badAnchors = { + 'null resting': (null, 190), + 'null max': (55, null), + 'both null': (null, null), + 'NaN resting': (nan, 190), + 'NaN max': (55, nan), + 'infinite resting': (-inf, 190), + 'infinite max': (55, inf), + 'max == resting': (60, 60), + 'max < resting': (80, 70), + }; + badAnchors.forEach((name, a) { + for (final sex in Sex.values) { + test('$name (${sex.name}): absent, same note as banisterTrimp', () { + final want = _want(rows, a.$1, a.$2, sex); + expect(want.present, isFalse); + final agg = _agg(a.$1, a.$2, sex); + // Empty and non-finite-only input are absent too, anchors rule. + expectConversion>( + (_) => agg, + [rows, col([]), col([nan, 0, -5])], + (c) => _want(c, a.$1, a.$2, sex), + equals: _eq); + }); + } + }); + + test('valid anchors, zero rows: absent', () { + for (final sex in Sex.values) { + final agg = _agg(55, 190, sex); + final want = _want(col([]), 55, 190, sex); + expect(want.present, isFalse); + _same(agg.run(col([])), want, 'empty'); + _same(agg.finish(agg.empty()), want, 'empty state'); + _same( + agg.finish(agg.combine( + agg.fold(col([]), 0, 0), agg.fold(col([]), 0, 0))), + want, + 'empty combined'); + } + }); + + test('rows that all vanish in a chunk still count as rows', () { + // 3 rows, none usable: PRESENT 0.0 (not absent) however it is cut. + final c = col([nan, 0, -3]); + for (final sex in Sex.values) { + final agg = _agg(55, 190, sex); + final want = _want(c, 55, 190, sex); + expect(want.present, isTrue); + expect(want.value, 0.0); + expectConversion>( + (_) => agg, [c], (c) => _want(c, 55, 190, sex), + equals: _eq, chunkings: 24); + } + }); + }); + + group('present with 0.0 when no row counts', () { + final cases = { + 'one NaN': col([nan]), + 'one zero': col([0]), + 'one negative': col([-40]), + 'infinities': col([inf, -inf]), + 'mixed junk': col([nan, 0, -1, inf, -inf, nan]), + }; + cases.forEach((name, c) { + for (final sex in Sex.values) { + test('$name (${sex.name})', () { + final want = _want(c, 55, 190, sex); + expect(want.present, isTrue); + expect(want.value, 0.0); + expect(want.confidence, 0.6); + final got = _agg(55, 190, sex).run(c); + _same(got, want, name); + expect(got.value, 0.0); + expect(got.confidence, 0.6); + }); + } + }); + }); + + group('values', () { + test('hand-worked: resting is 0, max is 0.64 e^1.92 (male)', () { + final a = _agg(50, 190, Sex.male); + expect(a.run(col([50])).value, 0.0); + expect(a.run(col([190])).value, closeTo(0.64 * math.exp(1.92), 1e-12)); + // Clamped: below resting counts 0 (but is still a present row), above + // max counts as max. + expect(a.run(col([40, 250])).value, + closeTo(0.64 * math.exp(1.92), 1e-12)); + }); + test('hand-worked: female coefficients 0.86 and 1.67', () { + final a = _agg(50, 190, Sex.female); + expect(a.run(col([190])).value, closeTo(0.86 * math.exp(1.67), 1e-12)); + final x = (120 - 50) / 140; + expect(a.run(col([120])).value, + closeTo(x * 0.86 * math.exp(1.67 * x), 1e-12)); + }); + test('metadata equals the reference on a plain day', () { + final c = col([60, 70, 90, 120, 150, 100]); + for (final sex in Sex.values) { + final got = _agg(55, 190, sex).run(c); + final want = _want(c, 55, 190, sex); + expect(got.confidence, 0.6); + expect(got.tier, want.tier); + expect(got.inputs_used, want.inputs_used); + expect(got.note, want.note); + _same(got, want, 'plain'); + } + }); + test('sexes differ', () { + final c = col([150, 160]); + expect(_agg(55, 190, Sex.male).run(c).value, + isNot(_agg(55, 190, Sex.female).run(c).value)); + }); + }); + + group('random parity (seeded)', () { + test('random anchors, sex, rows; every evaluation path', () { + final r = Rng(0x7217); + final params = Map.identity(); + for (var t = 0; t < 120; t++) { + final n = r.nextBool(.15) ? r.intIn(0, 3) : r.intIn(0, 300); + final c = _hrColumn(r, n, nanP: r.nextBool() ? .05 : .4); + // Mostly valid anchors; sometimes degenerate. + double? rest = 40 + r.nextDouble() * 40; + double? max = rest + (r.nextBool(.1) ? -5 : 60) + r.nextDouble() * 80; + if (r.nextBool(.05)) rest = null; + if (r.nextBool(.05)) max = nan; + params[c] = (rest, max, r.nextBool() ? Sex.male : Sex.female); + } + expectConversion>( + (c) => _agg(params[c]!.$1, params[c]!.$2, params[c]!.$3), + params.keys.toList(), + (c) => _want(c, params[c]!.$1, params[c]!.$2, params[c]!.$3), + equals: _eq); + }); + }); + + test('persistable: a mid-stream checkpoint resumes to the one-pass result', () { + final a = _agg(55, 190, Sex.male); + expectCheckpointResumes(a, _hrColumn(Rng(0xC0DE), 120), equals: _eq); + }); + + group('lawful', () { + final cols = [ + ...randomColumns(31, value: _hr, nanP: .1), + col([nan, 0, -2]), + col([inf, 120]), + ]; + for (final sex in Sex.values) { + test('valid anchors (${sex.name})', + () => expectLawful(_agg(55, 190, sex), cols, equals: _eq)); + } + test('invalid anchors', + () => expectLawful(_agg(null, 190, Sex.male), cols, equals: _eq)); + }); +} diff --git a/test/algo_version_test.dart b/test/algo_version_test.dart new file mode 100644 index 0000000..b51b688 --- /dev/null +++ b/test/algo_version_test.dart @@ -0,0 +1,20 @@ +// Counters move for direct algorithm/output changes, including rounding and +// abstention. Dependency-only changes propagate through deps. +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +void main() { + test('PR output changes move their owning groups to counter 2', () { + const changed = { + 'found.core', // Lomb-Scargle first-sample origin. + 'found.rr_correction', // Artifact-run cap, already bumped. + 'sleep.stage', // Already bumped; also owns adopted HRV staging paths. + 'hrv.freq', // HRV aggregator rounding and invalid-time abstention. + 'night.rhythm', // Cosinor/circadian and gap-aware irregular screening. + 'activity.load', // TRIMP, CTL/ATL and time-in-zone aggregation. + }; + expect({for (final g in kAnalyticsGroupRegistry.groups) + if (g.counter != 1) g.id: g.counter}, + {for (final id in changed) id: 2}); + }); +} diff --git a/test/incremental/day_curves_test.dart b/test/incremental/day_curves_test.dart new file mode 100644 index 0000000..014b19a --- /dev/null +++ b/test/incremental/day_curves_test.dart @@ -0,0 +1,192 @@ +// Oracle tests for the three DAY-side curves that live in edge: +// dayHrvCurve, dayRespCurve, _daytimeHrv (verbatim ports in edge_oracles.dart) +// Chunked by wall time like derive passes: RR with ts < T and accel rows with +// tsSec < T arrive together, watermark = T. JSON restart between chunks. +import 'dart:math' as math; + +import 'package:test/test.dart'; + +import '../../tool/incremental/edge_oracles.dart'; +import '../../tool/incremental/hrv_incr.dart'; +import '../../tool/incremental/oracle_util.dart'; +import '../../tool/incremental/resp_incr.dart'; +import '../../tool/incremental/synth.dart'; + +class _Day { + final RrData rr; + final AccelSeries acc; + _Day(this.rr, this.acc); + DaySub prefix(int rrN, int accN) => DaySub( + rr.rr.sublist(0, rrN), + rr.ts.sublist(0, rrN), + acc.tsSec.sublist(0, accN), + acc.ax.sublist(0, accN), + acc.ay.sublist(0, accN), + acc.az.sublist(0, accN)); +} + +_Day _makeDay({bool backwards = false}) { + final rr = backwards + ? synthRr(const SynthConfig(seed: 81, hours: 6, backwardsPerHour: 10)) + : realShapedDay(); + final from = (rr.ts.first / 1000).floor() - 2; + final to = (rr.ts.last / 1000).ceil() + 3; + return _Day(rr, synthAccel(5, from, to)); +} + +/// wall-clock cut points (seconds) like derive passes: mostly 15 min, some odd. +List _cutSecs(_Day d, math.Random r) { + final first = (d.rr.ts.first / 1000).floor(), last = (d.rr.ts.last / 1000).ceil() + 3; + final cuts = []; + var t = first; + while (t < last) { + t += [60, 300, 900, 900, 900, 1800, 3600][r.nextInt(7)] + r.nextInt(7); + cuts.add(math.min(t, last + 1)); + } + return cuts; +} + +void main() { + final day = _makeDay(); + final sleepOnset = 1700000000, sleepOffset = 1700000000 + 32040; + + test('dayHrvCurve: 24 h in random wall-time chunks, restart each chunk', () { + final r = math.Random(1); + var st = DayHrvCurveState(); + var rrAt = 0; + var maxRing = 0; + final cuts = _cutSecs(day, r); + for (var k = 0; k < cuts.length; k++) { + final T = cuts[k]; + var rrTo = rrAt; + while (rrTo < day.rr.length && day.rr.ts[rrTo] < T * 1000.0) { + rrTo++; + } + st.fold(day.rr.rr.sublist(rrAt, rrTo), day.rr.ts.sublist(rrAt, rrTo)); + rrAt = rrTo; + st = DayHrvCurveState.fromJson(jsonRoundTrip(st.toJson())); + maxRing = math.max(maxRing, st.ringSize); + if (k % 7 == 0 || k == cuts.length - 1) { + final want = oracleDayHrvCurve(day.prefix(rrTo, 0)); + expect(st.out.map((m) => '${m['t']}:${m['v']}').toList(), + want.map((m) => '${m['t']}:${m['v']}').toList(), + reason: 'chunk $k'); + } + } + // ignore: avoid_print + print('dayHrvCurve: points=${st.out.length}, max ring beats=$maxRing'); + }); + + test('dayHrvCurve with backwards timestamps', () { + final d = _makeDay(backwards: true); + final st = DayHrvCurveState(); + final r = math.Random(2); + var at = 0; + for (final cut in randomCuts(r, d.rr.length, sizes: [1, 10, 500, 3000])) { + st.fold(d.rr.rr.sublist(at, cut), d.rr.ts.sublist(at, cut)); + at = cut; + } + final want = oracleDayHrvCurve(d.prefix(d.rr.length, 0)); + expect(st.out.map((m) => '${m['t']}:${m['v']}').toList(), + want.map((m) => '${m['t']}:${m['v']}').toList()); + }); + + for (final lag in [0, 400]) { + test('_daytimeHrv: 24 h, night masked, accel lag=${lag}s behind RR, restart', () { + final r = math.Random(3); + var st = DaytimeHrvState(sleepOnset, sleepOffset); + var rrAt = 0, accAt = 0; + final cuts = _cutSecs(day, r); + var maxPending = 0; + for (var k = 0; k < cuts.length; k++) { + final T = cuts[k]; + var rrTo = rrAt, accTo = accAt; + while (rrTo < day.rr.length && day.rr.ts[rrTo] < T * 1000.0) { + rrTo++; + } + while (accTo < day.acc.length && day.acc.tsSec[accTo] < T - lag) { + accTo++; + } + st.fold( + day.rr.rr.sublist(rrAt, rrTo), + day.rr.ts.sublist(rrAt, rrTo), + day.acc.tsSec.sublist(accAt, accTo), + day.acc.ax.sublist(accAt, accTo), + day.acc.ay.sublist(accAt, accTo), + day.acc.az.sublist(accAt, accTo), + T - lag); + rrAt = rrTo; + accAt = accTo; + st = DaytimeHrvState.fromJson(jsonRoundTrip(st.toJson()), sleepOnset, sleepOffset); + maxPending = math.max(maxPending, st.pending); + if (k % 5 == 0 || k == cuts.length - 1) { + // the oracle only sees what has been delivered AND processed: beats + // pending (second not yet reported) are excluded from the prefix. + final processedRr = rrTo - st.pending; + final want = oracleDaytimeHrv(day.prefix(processedRr, accTo), sleepOnset, sleepOffset); + final got = st.result(); + expect(got['timeline'].toString(), want['timeline'].toString(), reason: 'chunk $k'); + expect(got['mean_rmssd'], want['mean_rmssd']); + expect(got['n_buckets'], want['n_buckets']); + } + } + // ignore: avoid_print + print('daytimeHrv: max pending beats at a chunk edge=$maxPending ' + 'buckets=${st.result()['n_buckets']}'); + if (lag > 0) expect(maxPending, greaterThan(0)); + }); + } + + test('_daytimeHrv with no sleep window (onset=offset=0)', () { + final st = DaytimeHrvState(0, 0); + st.fold(day.rr.rr, day.rr.ts, day.acc.tsSec, day.acc.ax, day.acc.ay, day.acc.az, + (day.rr.ts.last / 1000).ceil() + 10); + final want = oracleDaytimeHrv(day.prefix(day.rr.length, day.acc.length), 0, 0); + expect(st.result().toString(), want.toString()); + }); + + for (final lag in [0, 400]) { + test('dayRespCurve: 24 h, accel lag=${lag}s behind RR, restart each chunk', () { + final r = math.Random(4); + var st = DayRespCurveState(); + var rrAt = 0, accAt = 0; + final cuts = _cutSecs(day, r); + var maxPending = 0, maxRing = 0; + for (var k = 0; k < cuts.length; k++) { + final T = cuts[k]; + var rrTo = rrAt, accTo = accAt; + while (rrTo < day.rr.length && day.rr.ts[rrTo] < T * 1000.0) { + rrTo++; + } + while (accTo < day.acc.length && day.acc.tsSec[accTo] < T - lag) { + accTo++; + } + st.fold( + day.rr.rr.sublist(rrAt, rrTo), + day.rr.ts.sublist(rrAt, rrTo), + day.acc.tsSec.sublist(accAt, accTo), + day.acc.ax.sublist(accAt, accTo), + day.acc.ay.sublist(accAt, accTo), + day.acc.az.sublist(accAt, accTo), + T - lag); + rrAt = rrTo; + accAt = accTo; + st = DayRespCurveState.fromJson(jsonRoundTrip(st.toJson())); + maxPending = math.max(maxPending, st.pending); + maxRing = math.max(maxRing, st.ringBeats); + if (k % 6 == 0 || k == cuts.length - 1) { + final processedRr = rrTo - st.pending; + final want = oracleDayRespCurve(day.prefix(processedRr, accTo)); + expect(st.curve.map((m) => '${m['t']}:${m['v']}').toList(), + want.map((m) => '${m['t']}:${m['v']}').toList(), + reason: 'chunk $k'); + } + } + // ignore: avoid_print + print('dayRespCurve lag=$lag: points=${st.curve.length} ' + 'max pending=$maxPending max ring beats=$maxRing'); + expect(st.curve, isNotEmpty); + if (lag > 0) expect(maxPending, greaterThan(0)); + }); + } +} diff --git a/test/incremental/hrv_boundary_test.dart b/test/incremental/hrv_boundary_test.dart new file mode 100644 index 0000000..2b3702f --- /dev/null +++ b/test/incremental/hrv_boundary_test.dart @@ -0,0 +1,200 @@ +// Boundary / refusal-path oracle tests that RR-correction-driven data rarely +// reaches: beats landing EXACTLY on bin edges, the jitter gate (RMSSD refused), +// thin input => absent exactly where the oracle is. +import 'dart:math' as math; + +import 'package:openstrap_analytics/onehz.dart' + hide RrCorrector, RrSettled, RrSnapshot, IrregularScreenState; +import 'package:test/test.dart'; + +import '../../tool/incremental/hrv_incr.dart'; +import '../../tool/incremental/oracle_util.dart'; +import '../../tool/incremental/resp_incr.dart'; +import 'support.dart'; + +({List nn, List t}) _nn(int n, + {double step = 1000, double noise = 0, int seed = 1, double start = 0}) { + final r = math.Random(seed); + final nn = [], t = []; + var clock = start; + for (var i = 0; i < n; i++) { + final v = (step + 60 * math.sin(i * 0.21) + noise * (r.nextDouble() - .5)) + .roundToDouble(); + clock += v; + nn.add(v); + t.add(clock); + } + return (nn: nn, t: t); +} + +/// Equal null-ness and within 1e-9 relative (the aggregator's stated +/// tolerance; hrvTime now runs hrvTimeAggregator). +void _near9(double? got, double? want, String reason) { + expect(got == null, want == null, reason: reason); + if (got != null && want != null) { + expect(got, closeTo(want, 1e-9 * want.abs()), reason: reason); + } +} + +void main() { + test('night shape: beats exactly on 30-min edges, incl. last beat on an edge', () { + // 1000 ms steps from 0 => beat times are multiples of 1000, bins at 1.8e6 + for (final lastOnEdge in [false, true]) { + final n = 1800 * 4 + (lastOnEdge ? 0 : 777); + final nn = [], t = []; + for (var i = 1; i <= n; i++) { + nn.add(1000.0 + (i % 17) * 3); + t.add(i * 1000.0); // beat i ends at i*1000 ms + } + // make t0 = t[0] = 1000 => edges at 1000 + k*1.8e6 + for (final chunk in [1, 37, 900, 5000]) { + final st = NightShapeState(); + for (var a = 0; a < n; a += chunk) { + final b = math.min(n, a + chunk); + st.fold(nn.sublist(a, b), t.sublist(a, b)); + final want = nightHrvShape(nn.sublist(0, b), t.sublist(0, b)); + final got = st.evaluate(const [], const []); + sameEnvelope(got, want, why: 'edge n=$b chunk=$chunk'); + if (want.present) { + expect(got.value!.toJson(), want.value!.toJson(), reason: 'n=$b'); + } + } + } + } + }); + + test('night shape: dropout makes empty bins (holes) and a gap straddles a bin', () { + final a = _nn(2500, seed: 3); + final b = _nn(2500, seed: 4, start: a.t.last + 3 * 3600 * 1000.0); // 3 h gap + final nn = [...a.nn, ...b.nn], t = [...a.t, ...b.t]; + final st = NightShapeState(); + final r = math.Random(7); + var at = 0; + for (final cut in randomCuts(r, nn.length, sizes: [50, 700, 3000])) { + st.fold(nn.sublist(at, cut), t.sublist(at, cut)); + at = cut; + final want = nightHrvShape(nn.sublist(0, cut), t.sublist(0, cut)); + final got = st.evaluate(const [], const []); + sameEnvelope(got, want, why: 'cut=$cut'); + if (want.present) expect(got.value!.toJson(), want.value!.toJson()); + } + }); + + test('hrvTime: jitter gate refuses RMSSD/pNN50 exactly like the oracle', () { + final x = _nn(3000, noise: 260, seed: 5); // near-white successive diffs + final want0 = hrvTime(x.nn, nnTimesMs: x.t); + expect(want0.value!.rmssd, isNull, reason: 'fixture must trip the gate'); + for (final chunk in [1, 29, 30, 31, 600]) { + final acc = HrvTimeAcc(); + for (var a = 0; a < x.nn.length; a += chunk) { + final b = math.min(x.nn.length, a + chunk); + acc.fold(x.nn.sublist(a, b), x.t.sublist(a, b)); + final want = hrvTime(x.nn.sublist(0, b), nnTimesMs: x.t.sublist(0, b)); + final got = acc.evaluate(const [], const []); + expect(got.present, want.present); + if (!want.present) continue; + expect(got.value!.rmssd, want.value!.rmssd, reason: 'b=$b'); + expect(got.value!.pnn50, want.value!.pnn50, reason: 'b=$b'); + // 1e-9 relative: hrvTime now runs hrvTimeAggregator (was exact). + _near9(got.value!.sdann, want.value!.sdann, 'sdann b=$b'); + _near9(got.value!.sdnnIndex, want.value!.sdnnIndex, 'sdnnIndex b=$b'); + expect(got.value!.nBeats, want.value!.nBeats); + expect(got.note, want.note, reason: 'b=$b'); + } + } + }); + + test('hrvTime: gaps (seams) between runs, thin series', () { + final a = _nn(400, seed: 8), b = _nn(400, seed: 9, start: 5000000); + final nn = [...a.nn, ...b.nn], t = [...a.t, ...b.t]; + final acc = HrvTimeAcc(); + for (var n = 0; n <= nn.length; n += (n < 40 ? 1 : 97)) { + final acc2 = HrvTimeAcc()..fold(nn.sublist(0, n), t.sublist(0, n)); + final want = hrvTime(nn.sublist(0, n), nnTimesMs: t.sublist(0, n)); + final got = acc2.evaluate(const [], const []); + expect(got.present, want.present, reason: 'n=$n'); + if (want.present) { + expect(got.value!.rmssd, want.value!.rmssd, reason: 'n=$n'); + expect(got.value!.pnn50, want.value!.pnn50); + expect(got.value!.diffAcf1 == null, want.value!.diffAcf1 == null); + } + } + expect(acc.n, 0); + }); + + test('rsa: absent when oracle abstains (slow HR => ceiling below HF band)', () { + // 1500 ms beats: Nyquist 0.333 Hz < 0.40 => oracle refuses; so must we. + final x = _nn(900, step: 1500, noise: 10, seed: 2); + final st = RsaWelchState(); + var at = 0; + for (final cut in [200, 205, 500, 900]) { + st.fold(x.nn.sublist(at, cut), x.t.sublist(at, cut)); + at = cut; + final want = rsaRespRate(x.nn.sublist(0, cut), x.t.sublist(0, cut), + artifactFraction: 0.05); + final got = st.evaluate(const [], const [], artifactFraction: 0.05); + sameEnvelope(got, want, why: 'cut=$cut'); + expect(got.value?.brpm, want.value?.brpm); + } + }); + + test('rsa: artifact gate, thin (<20 beats), spans just below/at 600 s', () { + final x = _nn(1500, step: 800, noise: 30, seed: 6); + for (final af in [0.0, 0.31]) { + final st = RsaWelchState(); + var at = 0; + for (final cut in [10, 19, 20, 400, 749, 750, 751, 752, 900, 1500]) { + st.fold(x.nn.sublist(at, cut), x.t.sublist(at, cut)); + at = cut; + final want = rsaRespRate(x.nn.sublist(0, cut), x.t.sublist(0, cut), + artifactFraction: af); + final got = st.evaluate(const [], const [], artifactFraction: af); + sameEnvelope(got, want, why: 'cut=$cut af=$af'); + expect(got.value?.brpm, want.value?.brpm, reason: 'cut=$cut'); + } + } + }); + + // Under the ACF floor hrvTime reads the difference runs: a respiratory line + // exempts the night, a refused one may publish from a breathing line + // steady in Hz. HrvTimeAcc carries the library's mergeable evidence for both. + for (final (name, mode, n) in [ + ('exempt (RSA at a slow heart)', 3, 1200), + ('breathing line steady in Hz', 4, 4200), + ]) { + test('hrvTime below the floor: $name, chunks + restarts', () { + final r = math.Random(mode); + final nn = [], t = []; + var clock = 1e6; + for (var i = 0; i < n; i++) { + final s = (clock - 1e6) / 1000; + final v = mode == 3 + ? 1350 + 80 * math.sin(2 * math.pi * s / 3600) + + 40 * math.sin(2 * math.pi * .3 * s) + 3 * (r.nextDouble() - .5) + : 900 + 400 * s / 9000 + 10 * math.sin(2 * math.pi * .25 * s) + + 40 * (r.nextDouble() - .5); + clock += v; + nn.add(v); + t.add(clock); + } + final whole = hrvTime(nn, nnTimesMs: t); + expect(whole.value!.diffAcf1, lessThan(kNnDiffAcf1Floor)); + expect(whole.value!.rmssd, isNotNull, reason: 'the batch publishes'); + var acc = HrvTimeAcc(); + var at = 0; + for (final cut in randomCuts(math.Random(9), n, sizes: [37, 400, 1300])) { + acc.fold(nn.sublist(at, cut), t.sublist(at, cut)); + acc = HrvTimeAcc.fromJson(jsonRoundTrip(acc.toJson())); + at = cut; + final want = hrvTime(nn.sublist(0, cut), nnTimesMs: t.sublist(0, cut)); + final got = acc.evaluate(const [], const []); + expect(got.note, want.note, reason: 'cut=$cut'); + expect(got.value!.rmssd == null, want.value!.rmssd == null, reason: 'cut=$cut'); + if (want.value!.rmssd != null) { + expect((got.value!.rmssd! - want.value!.rmssd!).abs(), + lessThan(1e-9 * want.value!.rmssd!), reason: 'cut=$cut'); + } + } + }); + } +} diff --git a/test/incremental/hrv_resp_nn_test.dart b/test/incremental/hrv_resp_nn_test.dart new file mode 100644 index 0000000..d8f4635 --- /dev/null +++ b/test/incremental/hrv_resp_nn_test.dart @@ -0,0 +1,274 @@ +// Oracle tests for every estimator that consumes the corrected NN series: +// hrvTime, edge _hrvTimeline, nightHrvShape, rsaRespRate, _respPerWindow +// (+ breathingRateVariability on top), hrvFreq. +// The incremental states are fed the SETTLED NN from RrCorrector.fold and +// evaluated with the PROVISIONAL tail; the oracle is the batch function on +// correctRr(all beats so far). +import 'dart:convert'; +import 'dart:math' as math; + +import 'package:openstrap_analytics/onehz.dart' + hide RrCorrector, RrSettled, RrSnapshot, IrregularScreenState; +import 'package:test/test.dart'; + +import '../../tool/incremental/driver.dart'; +import '../../tool/incremental/edge_oracles.dart'; +import '../../tool/incremental/hrv_incr.dart'; +import '../../tool/incremental/oracle_util.dart'; +import '../../tool/incremental/resp_incr.dart'; +import '../../tool/incremental/synth.dart'; +import 'support.dart'; + +RrData _realFirstHours(double hours) { + final r = realNightRr()!; + final endMs = r.ts.first + hours * 3.6e6; + var n = r.ts.indexWhere((t) => t > endMs); + if (n < 0) n = r.length; + return RrData(r.rr.sublist(0, n), r.ts.sublist(0, n)); +} + +/// Equal null-ness and within 1e-9 relative (the aggregator's stated +/// tolerance; hrvTime now runs hrvTimeAggregator). +void _near9(double? got, double? want, String reason) { + expect(got == null, want == null, reason: reason); + if (got != null && want != null) { + expect(got, closeTo(want, 1e-9 * want.abs()), reason: reason); + } +} + +void main() { + final errSdnn = Err(), errAcf = Err(), errConf = Err(); + var hrvChecks = 0; + + group('hrvTime accumulators', () { + for (final restart in [false, true]) { + for (final name in ['real 3h', 'synthetic dirty 2h', 'synthetic clean 1h']) { + test('$name restart=$restart', () { + final s = name == 'real 3h' + ? _realFirstHours(3) + : name.contains('dirty') + ? synthRr(const SynthConfig( + seed: 61, hours: 2, ectopicPerMin: 1.5, noiseRunPerMin: 0.3)) + : synthRr(const SynthConfig( + seed: 62, + hours: 1, + ectopicPerMin: 0, + missedPerMin: 0, + extraPerMin: 0, + noiseRunPerMin: 0, + gapPerHour: 0)); + final r = math.Random(restart ? 5 : 6); + final cuts = randomCuts(r, s.length, sizes: [90, 181, 400, 1200, 3000]); + var acc = HrvTimeAcc(); + driveRr(s, cuts, (f) { + acc.fold(f.settled.nn, f.settled.nnTimes); + if (restart) acc = HrvTimeAcc.fromJson(jsonRoundTrip(acc.toJson())); + if (f.fold % 3 != 0 && f.fold != cuts.length - 1) return; + final o = correctRr(s.rr.sublist(0, f.n), rrTsMs: s.ts.sublist(0, f.n)); + final af = (1.0 - o.cleanFraction).clamp(0.0, 1.0).toDouble(); + expect(f.artifactFraction, af); + final want = hrvTime(o.nn, nnTimesMs: o.nnTimesMs, artifactFraction: af); + final got = acc.evaluate(f.snap.tailNn, f.snap.tailNnTimes, + artifactFraction: af); + hrvChecks++; + expect(got.present, want.present); + if (!want.present) return; + final a = got.value!, b = want.value!; + expect(a.nBeats, b.nBeats); + // rmssd / pnn50 bit-identical; sdann / sdnnIndex 1e-9 relative + // (hrvTime now runs hrvTimeAggregator; was exact). + expect(a.rmssd, b.rmssd, reason: 'rmssd'); + expect(a.pnn50, b.pnn50, reason: 'pnn50'); + _near9(a.sdann, b.sdann, 'sdann'); + _near9(a.sdnnIndex, b.sdnnIndex, 'sdnnIndex'); + // bounded error + errSdnn.add(a.sdnn, b.sdnn); + errAcf.add(a.diffAcf1, b.diffAcf1); + errConf.add(got.confidence, want.confidence); + expect(got.note, want.note); + }); + }); + } + } + test('error summary', () { + // ignore: avoid_print + print('hrvTime checks=$hrvChecks\n sdnn: $errSdnn\n acf1: $errAcf\n conf: $errConf'); + expect(errSdnn.maxRel, lessThan(1e-11)); + expect(errAcf.maxAbs, lessThan(1e-11)); + }); + }); + + group('hrv timeline + night shape', () { + for (final restart in [false, true]) { + test('real 8.9 h night, 15-min folds restart=$restart', () { + final s = realNightRr()!; + final origin = s.ts.first - s.rr.first; + var tl = HrvTimelineState(origin); + var shape = NightShapeState(); + final cuts = timeCuts(s.ts, 900); + final checks = {5, 20, 35, cuts.length - 1}; + driveRr(s, cuts, (f) { + tl.fold(f.settled.nn, f.settled.nnTimes); + shape.fold(f.settled.nn, f.settled.nnTimes); + if (!checks.contains(f.fold)) return; + final o = correctRr(s.rr.sublist(0, f.n), rrTsMs: s.ts.sublist(0, f.n)); + // timeline + expect( + tl.curve(f.snap.tailNn, f.snap.tailNnTimes) + .map((m) => '${m['t']}:${m['v']}') + .toList(), + oracleHrvTimeline(o.nn, o.nnTimesMs, origin) + .map((m) => '${m['t']}:${m['v']}') + .toList(), + reason: 'timeline fold ${f.fold}'); + // night shape + final want = nightHrvShape(o.nn, o.nnTimesMs); + final got = shape.evaluate(f.snap.tailNn, f.snap.tailNnTimes); + sameEnvelope(got, want, why: 'shape ${f.fold}'); + if (want.present) { + expect(got.value!.toJson(), want.value!.toJson()); + final gb = got.value!.bins, wb = want.value!.bins; + for (var i = 0; i < wb.length; i++) { + expect(gb[i].rmssdMs, wb[i].rmssdMs); + expect(gb[i].loMs, wb[i].loMs); + expect(gb[i].hiMs, wb[i].hiMs); + expect(gb[i].nBeats, wb[i].nBeats); + } + expect(got.value!.lastOverFirst, want.value!.lastOverFirst); + } + if (restart) { + // json round trip of the HRV timeline ring is covered by day curve; + // here we only re-create from scratch to prove state is the whole story + } + }, restart: restart); + }); + } + }, timeout: const Timeout(Duration(minutes: 10))); + + group('rsaRespRate / respPerWindow / BRV', () { + test('real night 15-min folds (state JSON-restarted each fold)', () { + final s = realNightRr()!; + var rsa = RsaWelchState(); + var wins = RespWindowsState(); + final cuts = timeCuts(s.ts, 900); + final checks = {0, 1, 2, 3, 12, 20, 30, 33, cuts.length - 1}; + var maxBuf = 0; + driveRr(s, cuts, (f) { + rsa.fold(f.settled.nn, f.settled.nnTimes); + rsa = RsaWelchState.fromJson(jsonRoundTrip(rsa.toJson())); + wins.fold(f.settled.nn, f.settled.nnTimes); + maxBuf = math.max(maxBuf, rsa.bufferedBeats); + if (!checks.contains(f.fold)) return; + final o = correctRr(s.rr.sublist(0, f.n), rrTsMs: s.ts.sublist(0, f.n)); + final af = (1.0 - o.cleanFraction).clamp(0.0, 1.0).toDouble(); + final want = rsaRespRate(o.nn, o.nnTimesMs, artifactFraction: af); + final got = rsa.evaluate(f.snap.tailNn, f.snap.tailNnTimes, + artifactFraction: af); + sameEnvelope(got, want, why: 'rsa fold ${f.fold}'); + expect(got.value?.brpm, want.value?.brpm); + expect(got.value?.peakHz, want.value?.peakHz); + expect(got.value?.power, want.value?.power); + // resp windows + BRV + final wantW = oracleRespPerWindow(o.nn, o.nnTimesMs); + final gotW = wins.evaluate(f.snap.tailNn, f.snap.tailNnTimes); + expect(gotW, orderedEquals(wantW), reason: 'windows fold ${f.fold}'); + if (wantW.length >= 3) { + final a = breathingRateVariability(gotW); + final b = breathingRateVariability(wantW); + expect(a.value!.toJson(), b.value!.toJson()); + } + }); + // ignore: avoid_print + print('rsa: max buffered settled beats=$maxBuf (state carries ' + '${maxBuf * 2} doubles + per-segment peaks)'); + }); + + test('synthetic dirty 2 h, random chunks + restart', () { + final s = synthRr(const SynthConfig(seed: 71, hours: 2.2, ectopicPerMin: 1)); + final r = math.Random(3); + final cuts = randomCuts(r, s.length, sizes: [90, 300, 1000, 2500]); + var rsa = RsaWelchState(); + driveRr(s, cuts, (f) { + rsa.fold(f.settled.nn, f.settled.nnTimes); + if (f.fold % 2 == 0) { + rsa = RsaWelchState.fromJson(jsonRoundTrip(rsa.toJson())); + } + final o = correctRr(s.rr.sublist(0, f.n), rrTsMs: s.ts.sublist(0, f.n)); + final af = (1.0 - o.cleanFraction).clamp(0.0, 1.0).toDouble(); + final want = rsaRespRate(o.nn, o.nnTimesMs, artifactFraction: af); + final got = rsa.evaluate(f.snap.tailNn, f.snap.tailNnTimes, + artifactFraction: af); + sameEnvelope(got, want, why: 'rsa fold ${f.fold} n=${f.n}'); + expect(got.value?.brpm, want.value?.brpm); + }); + }); + }, timeout: const Timeout(Duration(minutes: 15))); + + group('hrvFreq (LF/HF)', () { + test('real night 15-min folds, state JSON-restarted each fold', () { + final s = realNightRr()!; + var hf = HrvFreqState(); + final cuts = timeCuts(s.ts, 900); + final checks = {2, 10, 24, cuts.length - 1}; + var maxBuf = 0; + driveRr(s, cuts, (f) { + hf.fold(f.settled.nn, f.settled.nnTimes); + hf = HrvFreqState.fromJson(jsonRoundTrip(hf.toJson())); + maxBuf = math.max(maxBuf, hf.bufferedBeats); + if (!checks.contains(f.fold)) return; + final o = correctRr(s.rr.sublist(0, f.n), rrTsMs: s.ts.sublist(0, f.n)); + final af = (1.0 - o.cleanFraction).clamp(0.0, 1.0).toDouble(); + final want = hrvFreq(o.nn, o.nnTimesMs, artifactFraction: af); + final got = hf.evaluate(f.snap.tailNn, f.snap.tailNnTimes, + artifactFraction: af); + sameEnvelope(got, want, why: 'hrvFreq fold ${f.fold}'); + if (want.present) { + final a = got.value!, b = want.value!; + expect(a.ulf, b.ulf); + expect(a.vlf, b.vlf); + expect(a.lf, b.lf); + expect(a.hf, b.hf); + expect(a.total, b.total); + expect(a.lfhf, b.lfhf); + expect(a.nuLf, b.nuLf); + expect(a.totalBands, b.totalBands); + } + }); + // ignore: avoid_print + print('hrvFreq: max buffered beats=$maxBuf (ULF/VLF keep the record)'); + }); + test('includeUlf:false keeps the buffer small and is exact below 33 333 s', () { + final s = realNightRr()!; // 32 040 s span < ULF segment + var hf = HrvFreqState(includeUlf: false); + final cuts = timeCuts(s.ts, 900); + final checks = {4, 18, cuts.length - 1}; + var maxBuf = 0, maxBytes = 0; + driveRr(s, cuts, (f) { + hf.fold(f.settled.nn, f.settled.nnTimes); + final txt = jsonEncode(hf.toJson()); + maxBytes = math.max(maxBytes, txt.length); + hf = HrvFreqState.fromJson(jsonDecode(txt) as Map, + includeUlf: false); + maxBuf = math.max(maxBuf, hf.bufferedBeats); + if (!checks.contains(f.fold)) return; + final o = correctRr(s.rr.sublist(0, f.n), rrTsMs: s.ts.sublist(0, f.n)); + final af = (1.0 - o.cleanFraction).clamp(0.0, 1.0).toDouble(); + final want = hrvFreq(o.nn, o.nnTimesMs, artifactFraction: af); + final got = hf.evaluate(f.snap.tailNn, f.snap.tailNnTimes, + artifactFraction: af); + sameEnvelope(got, want, why: 'hrvFreq noULF fold ${f.fold}'); + if (want.present) { + expect(got.value!.vlf, want.value!.vlf); + expect(got.value!.lf, want.value!.lf); + expect(got.value!.hf, want.value!.hf); + expect(got.value!.ulf, want.value!.ulf); + expect(got.value!.total, want.value!.total); + } + }); + // ignore: avoid_print + print('hrvFreq(includeUlf:false): max buffered beats=$maxBuf, ' + 'max state JSON bytes=$maxBytes'); + expect(maxBuf, lessThan(8000)); + }); + }, timeout: const Timeout(Duration(minutes: 20))); +} diff --git a/test/incremental/hrv_windows_test.dart b/test/incremental/hrv_windows_test.dart new file mode 100644 index 0000000..1159591 --- /dev/null +++ b/test/incremental/hrv_windows_test.dart @@ -0,0 +1,205 @@ +// Oracle tests: nocturnalRmssd (with/without stage mask), sleepSessionWindowedRmssd +// (raw RR, bounds), irregularBeatScreen (day-long corrected NN). +import 'dart:math' as math; + +import 'package:openstrap_analytics/onehz.dart' + hide RrCorrector, RrSettled, RrSnapshot, IrregularScreenState; +import 'package:test/test.dart'; + +import '../../tool/incremental/driver.dart'; +import '../../tool/incremental/hrv_incr.dart'; +import '../../tool/incremental/oracle_util.dart'; +import '../../tool/incremental/synth.dart'; +import 'support.dart'; + +void _sameD(Metric a, Metric b, Err e, {String? why}) { + expect(a.present, b.present, reason: 'present $why'); + expect(a.note, b.note, reason: 'note $why'); + expect(a.tier, b.tier); + if (b.present) { + expect(a.value, b.value, reason: 'value $why'); // bit-identical + e.add(a.confidence, b.confidence); + } +} + +void main() { + final confErr = Err(); + + test('nocturnalRmssd: real night, 15-min folds, with and without a stage mask', () { + final s = realNightRr()!; + final spanSec = ((s.ts.last - s.ts.first) / 1000).ceil() + 10; + // blocky mask: asleep (true) ~70% of the time in 5-40 min blocks + final r = math.Random(1); + final mask = List.filled(spanSec, false); + for (var i = 0; i < spanSec;) { + final len = 300 + r.nextInt(2100); + final v = r.nextDouble() < 0.7; + for (var k = 0; k < len && i < spanSec; k++, i++) { + mask[i] = v; + } + } + final st = NocturnalRmssdState(); + final cuts = timeCuts(s.ts, 900); + final checks = {1, 3, 8, 15, 25, cuts.length - 1}; + driveRr(s, cuts, (f) { + st.fold(f.settled.nn, f.settled.nnTimes); + if (!checks.contains(f.fold)) return; + final o = correctRr(s.rr.sublist(0, f.n), rrTsMs: s.ts.sublist(0, f.n)); + for (final m in [null, mask]) { + final want = nocturnalRmssd(o.nn, o.nnTimesMs, stageMaskPerSec: m); + final got = st.evaluate(f.snap.tailNn, f.snap.tailNnTimes, stageMaskPerSec: m); + _sameD(got, want, confErr, why: 'fold ${f.fold} mask=${m != null}'); + } + }); + }); + + test('nocturnalRmssd: jitter gate (white-noise NN) refuses like the oracle', () { + final r = math.Random(4); + final nn = [], t = []; + var clock = 0.0; + for (var i = 0; i < 6000; i++) { + final v = (800 + 300 * (r.nextDouble() - .5)).roundToDouble(); + clock += v; + nn.add(v); + t.add(clock); + } + final st = NocturnalRmssdState(); + var at = 0; + for (final cut in randomCuts(math.Random(2), nn.length, sizes: [37, 400, 2000])) { + st.fold(nn.sublist(at, cut), t.sublist(at, cut)); + at = cut; + final want = nocturnalRmssd(nn.sublist(0, cut), t.sublist(0, cut)); + final got = st.evaluate(const [], const []); + _sameD(got, want, confErr, why: 'cut=$cut'); + } + expect(nocturnalRmssd(nn, t).present, isFalse, reason: 'fixture trips the gate'); + }); + + test('sleepSessionWindowedRmssd: raw RR, 15-min folds, session bounds inside the night', () { + final s = realNightRr()!; + final start = (s.ts.first / 1000).floor() + 1234; + final end = (s.ts.last / 1000).floor() - 987; + final cuts = timeCuts(s.ts, 900); + final st = SessionRmssdState(start, end); + var from = 0; + for (var k = 0; k < cuts.length; k++) { + st.fold(s.rr.sublist(from, cuts[k]), s.ts.sublist(from, cuts[k])); + from = cuts[k]; + if (k % 4 == 0 || k == cuts.length - 1) { + // oracle with the session end clipped to the data seen so far + final seenEnd = math.min(end, (s.ts[cuts[k] - 1] / 1000).round() + 1); + final want = sleepSessionWindowedRmssd( + s.rr.sublist(0, cuts[k]), s.ts.sublist(0, cuts[k]), + startSec: start, endSec: end); + final got = st.evaluate(); + _sameD(got, want, confErr, why: 'k=$k seenEnd=$seenEnd'); + } + } + }); + + test('sleepSessionWindowedRmssd: dirty synthetic night + seams', () { + final s = synthRr(const SynthConfig( + seed: 91, hours: 4, ectopicPerMin: 2, noiseRunPerMin: 0.6, gapPerHour: 10)); + final start = (s.ts.first / 1000).floor() + 100; + final end = (s.ts.last / 1000).floor() - 100; + final st = SessionRmssdState(start, end); + var at = 0; + for (final cut in randomCuts(math.Random(8), s.length, sizes: [3, 200, 1500])) { + st.fold(s.rr.sublist(at, cut), s.ts.sublist(at, cut)); + at = cut; + final want = sleepSessionWindowedRmssd( + s.rr.sublist(0, cut), s.ts.sublist(0, cut), + startSec: start, endSec: end); + _sameD(st.evaluate(), want, confErr, why: 'cut=$cut'); + } + }); + + group('irregularBeatScreen', () { + // AF-like bursts: irregularly irregular RR in 5-30 min blocks. + ({List nn, List t}) fixture(int seed, double afShare) { + final r = math.Random(seed); + final nn = [], t = []; + var clock = 0.0; + var af = false; + var left = 0; + while (clock < 8 * 3600 * 1000.0) { + if (--left <= 0) { + af = r.nextDouble() < afShare; + left = 300 + r.nextInt(2000); + } + final v = af + ? (420 + r.nextInt(700)).toDouble() + : (850 + 40 * math.sin(clock / 9000) + 25 * (r.nextDouble() - .5)).roundToDouble(); + clock += v; + nn.add(v); + t.add(clock); + } + return (nn: nn, t: t); + } + + for (final share in [0.0, 0.35, 0.75, 1.0]) { + for (final restart in [false, true]) { + test('af share=$share restart=$restart', () { + final x = fixture((share * 100).round() + 3, share); + var st = IrregularScreenState(); + var at = 0; + var flags = 0, checks = 0; + for (final cut in randomCuts(math.Random(5), x.nn.length, sizes: [90, 500, 4000])) { + st.fold(x.nn.sublist(at, cut), x.t.sublist(at, cut)); + at = cut; + if (restart) st = IrregularScreenState.fromJson(jsonRoundTrip(st.toJson())); + for (final af in [0.0, 0.31]) { + final want = irregularBeatScreen(x.nn.sublist(0, cut), + nnTimesMs: x.t.sublist(0, cut), artifactFraction: af); + final got = st.evaluate(const [], const [], artifactFraction: af); + expect(got.present, want.present, reason: 'cut=$cut af=$af'); + expect(got.note, want.note); + if (want.present) { + expect(got.value!.flag, want.value!.flag, reason: 'flag cut=$cut'); + expect(got.value!.nBeats, want.value!.nBeats); + expect(got.value!.pnnPct, want.value!.pnnPct); + expect((got.value!.sd1 - want.value!.sd1).abs(), lessThan(1e-9)); + expect((got.value!.sd2 - want.value!.sd2).abs(), lessThan(1e-9)); + confErr.add(got.confidence, want.confidence); + if (want.value!.flag) flags++; + checks++; + } + } + } + // ignore: avoid_print + print('irregular share=$share: checks=$checks flagged=$flags'); + }); + } + } + + test('whole day through RrCorrector -> screen (3 checkpoints)', () { + final day = realShapedDay(); + final st = IrregularScreenState(); + final cuts = timeCuts(day.ts, 900); + final checks = {cuts.length ~/ 3, 2 * cuts.length ~/ 3, cuts.length - 1}; + driveRr(day, cuts, (f) { + st.fold(f.settled.nn, f.settled.nnTimes); + if (!checks.contains(f.fold)) return; + final o = correctRr(day.rr.sublist(0, f.n), rrTsMs: day.ts.sublist(0, f.n)); + final af = (1.0 - o.cleanFraction).clamp(0.0, 1.0).toDouble(); + final want = irregularBeatScreen(o.nn, + nnTimesMs: o.nnTimesMs, artifactFraction: af); + final got = st.evaluate(f.snap.tailNn, f.snap.tailNnTimes, artifactFraction: af); + expect(got.present, want.present); + expect(got.note, want.note); + if (want.present) { + expect(got.value!.flag, want.value!.flag); + expect(got.value!.nBeats, want.value!.nBeats); + expect(got.value!.pnnPct, want.value!.pnnPct); + expect((got.value!.sd1 - want.value!.sd1).abs(), lessThan(1e-9)); + expect((got.value!.sd2 - want.value!.sd2).abs(), lessThan(1e-9)); + } + }); + }, timeout: const Timeout(Duration(minutes: 10))); + }); + + test('confidence error summary', () { + // ignore: avoid_print + print('confidence (derived from acf / counts): $confErr'); + }); +} diff --git a/test/incremental/rr_stream_test.dart b/test/incremental/rr_stream_test.dart new file mode 100644 index 0000000..8cfd538 --- /dev/null +++ b/test/incremental/rr_stream_test.dart @@ -0,0 +1,282 @@ +// Oracle test: streaming RrCorrector == correctRr over the series so far. +// BIT-identical (nn, nnTimes, per-beat classes, counts, cleanFraction), for +// random chunkings, restart-from-JSON between chunks, and awkward inputs. +import 'dart:math' as math; + +import 'package:openstrap_analytics/onehz.dart' + hide RrCorrector, RrSettled, RrSnapshot, IrregularScreenState; +import 'package:test/test.dart'; + +import '../../tool/incremental/oracle_util.dart'; +import '../../tool/incremental/rr_stream.dart'; +import '../../tool/incremental/synth.dart'; + +class _Run { + RrCorrector c; + final settledNn = [], settledT = []; + final classes = {}; + _Run(this.c) { + _hook(); + } + void _hook() => c.debugClassSink = (g, cls) => classes[g] = cls; + void restart() { + c = RrCorrector.fromJson(jsonRoundTrip(c.toJson())); + _hook(); + } + + void fold(List rr, List? ts) { + final s = c.fold(rr, tsMs: ts); + settledNn.addAll(s.nn); + settledT.addAll(s.nnTimes); + } +} + +void _expectSame(_Run run, List rr, List? ts, int n, + {double alpha = 5.2, int win = 91, String? why}) { + final oracle = correctRr(rr.sublist(0, n), + rrTsMs: ts?.sublist(0, n), alpha: alpha, windowBeats: win); + final snap = run.c.snapshot(withClasses: true); + expect(snap.n, n, reason: why); + final nn = [...run.settledNn, ...snap.tailNn]; + final tt = [...run.settledT, ...snap.tailNnTimes]; + expect(nn, orderedEquals(oracle.nn), reason: 'nn $why'); + expect(tt, orderedEquals(oracle.nnTimesMs), reason: 'nnTimes $why'); + expect(snap.cleanFraction, oracle.cleanFraction, reason: 'clean $why'); + expect(snap.droppedCount, oracle.droppedCount, reason: 'dropped $why'); + expect(snap.correctedCount, oracle.correctedCount, reason: 'corrected $why'); + // classes: sink (final) ++ tail (provisional) + final cl = [ + for (var g = 0; g < snap.classifiedBeats; g++) + BeatClass.values[run.classes[g]!], + ...snap.tailClasses ?? const [], + ]; + if (n >= 3) { + expect(cl, orderedEquals(oracle.classes), reason: 'classes $why'); + } +} + +void main() { + test('every prefix, 1 beat at a time (n = 0..260), with artefacts', () { + for (final seed in [3, 4]) { + final s = synthRr(SynthConfig( + seed: seed, + hours: 0.12, + ectopicPerMin: 2, + missedPerMin: 1, + extraPerMin: 1, + noiseRunPerMin: 0.8, + gapPerHour: 20)); + final n = math.min(260, s.length); + final run = _Run(RrCorrector()); + for (var i = 0; i < n; i++) { + run.fold([s.rr[i]], [s.ts[i]]); + _expectSame(run, s.rr, s.ts, i + 1, why: 'seed=$seed i=$i'); + } + } + }); + + test('every prefix with a JSON restart between EVERY beat', () { + final s = synthRr(SynthConfig( + seed: 11, + hours: 0.1, + ectopicPerMin: 2, + noiseRunPerMin: 0.6, + gapPerHour: 10)); + final n = math.min(230, s.length); + final run = _Run(RrCorrector()); + for (var i = 0; i < n; i++) { + run.fold([s.rr[i]], [s.ts[i]]); + run.restart(); + _expectSame(run, s.rr, s.ts, i + 1, why: 'i=$i'); + } + }); + + test('1-beat folds over 4000 dirty beats: settled output == oracle', () { + // Hits every alignment between a fold boundary and an isolated artefact's + // right-anchor search (the case a coarse chunking rarely lands on). + final s = synthRr(const SynthConfig( + seed: 51, + hours: 1.2, + ectopicPerMin: 3, + missedPerMin: 2, + extraPerMin: 2, + noiseRunPerMin: 1, + gapPerHour: 8)); + final n = math.min(4000, s.length); + final run = _Run(RrCorrector()); + for (var i = 0; i < n; i++) { + run.fold([s.rr[i]], [s.ts[i]]); + if (i % 997 == 0) run.restart(); + } + _expectSame(run, s.rr, s.ts, n, why: 'final'); + }); + + group('random chunking', () { + final cfgs = { + 'clean': const SynthConfig( + seed: 21, + hours: 1.5, + ectopicPerMin: 0, + missedPerMin: 0, + extraPerMin: 0, + noiseRunPerMin: 0, + gapPerHour: 0), + 'default': const SynthConfig(seed: 22, hours: 2), + 'dirty': const SynthConfig( + seed: 23, + hours: 1.5, + ectopicPerMin: 2, + missedPerMin: 1, + extraPerMin: 1, + noiseRunPerMin: 0.5, + gapPerHour: 12), + 'backwards-ts': const SynthConfig( + seed: 24, hours: 1.5, backwardsPerHour: 30, gapPerHour: 6), + }; + for (final e in cfgs.entries) { + for (final restart in [false, true]) { + test('${e.key} restart=$restart', () { + final s = synthRr(e.value); + final r = math.Random(e.value.seed * 7 + (restart ? 1 : 0)); + final cuts = randomCuts(r, s.length); + final run = _Run(RrCorrector()); + var from = 0; + var maxLag = 0; + for (var k = 0; k < cuts.length; k++) { + final to = cuts[k]; + run.fold(s.rr.sublist(from, to), s.ts.sublist(from, to)); + from = to; + maxLag = math.max(maxLag, to - run.c.settledBeats); + if (restart && r.nextBool()) run.restart(); + if (k % 11 == 0 || k == cuts.length - 1) { + _expectSame(run, s.rr, s.ts, to, why: '${e.key} cut=$to'); + } + } + // buffer is bounded: not the whole series + expect(run.c.bufferedBeats, lessThan(2000)); + // ignore: avoid_print + print('${e.key}: beats=${s.length} folds=${cuts.length} ' + 'max unsettled lag=$maxLag beats'); + }); + } + } + }); + + test('sub-second beat timestamps (non-integer ms) stream identically', () { + final s = synthRr(const SynthConfig(seed: 33, hours: 1, gapPerHour: 6)); + final r = math.Random(12); + final ts = [for (final t in s.ts) t + r.nextInt(1000) + r.nextDouble()]; + final run = _Run(RrCorrector()); + var from = 0; + for (final to in randomCuts(r, s.length)) { + run.fold(s.rr.sublist(from, to), ts.sublist(from, to)); + from = to; + if (to % 4 == 0 || to == s.length) _expectSame(run, s.rr, ts, to, why: 'to=$to'); + } + _expectSame(run, s.rr, ts, s.length); + }); + + test('no timestamps (null ts) streams identically', () { + final s = synthRr(const SynthConfig(seed: 31, hours: 1)); + final run = _Run(RrCorrector()); + final r = math.Random(5); + var from = 0; + for (final to in randomCuts(r, s.length)) { + run.fold(s.rr.sublist(from, to), null); + from = to; + if (to % 3 == 0 || to == s.length) { + _expectSame(run, s.rr, null, to, why: 'to=$to'); + } + } + _expectSame(run, s.rr, null, s.length); + }); + + for (final p in [ + (alpha: 5.2, win: 31), + (alpha: 4.0, win: 61), + (alpha: 5.2, win: 121), + (alpha: 6.5, win: 91), + ]) { + test('params alpha=${p.alpha} win=${p.win}', () { + final s = synthRr(const SynthConfig(seed: 41, hours: 1, ectopicPerMin: 1)); + final run = + _Run(RrCorrector(alpha: p.alpha, windowBeats: p.win)); + final r = math.Random(9); + var from = 0; + for (final to in randomCuts(r, s.length)) { + run.fold(s.rr.sublist(from, to), s.ts.sublist(from, to)); + from = to; + if (to % 5 == 0 || to == s.length) { + _expectSame(run, s.rr, s.ts, to, + alpha: p.alpha, win: p.win, why: 'to=$to'); + } + } + _expectSame(run, s.rr, s.ts, s.length, alpha: p.alpha, win: p.win); + }); + } + + group('degenerate inputs', () { + void drive(List rr, List ts, String why) { + final run = _Run(RrCorrector()); + final r = math.Random(3); + var from = 0; + for (final to in randomCuts(r, rr.length, sizes: [1, 2, 3, 50, 200])) { + run.fold(rr.sublist(from, to), ts.sublist(from, to)); + from = to; + _expectSame(run, rr, ts, to, why: '$why to=$to'); + } + } + + test('constant RR (QD = 0, floor governs, massive ties)', () { + final rr = List.filled(500, 800); + drive(rr, [for (var i = 0; i < 500; i++) 1.7e12 + i * 800.0], 'const'); + }); + test('constant with one outlier at start / middle / end', () { + for (final at in [0, 1, 250, 498, 499]) { + final rr = List.filled(500, 800)..[at] = 1700; + drive(rr, [for (var i = 0; i < 500; i++) i * 800.0], 'outlier@$at'); + } + }); + test('long artefact runs (>90 beats) and alternating artefacts', () { + final r = math.Random(5); + final rr = [ + for (var i = 0; i < 700; i++) + (i >= 150 && i < 290) + ? 250 + r.nextInt(2000).toDouble() + : (i >= 400 && i < 500 && i.isEven) + ? 2200 + : 800 + r.nextInt(40).toDouble() + ]; + var t = 0.0; + final ts = [ + for (final v in rr) (t += v) + ]; + drive(rr, ts, 'longrun'); + }); + test('artefact beat 0 and 1 (no left anchors), tail artefact (no right)', () { + final rr = [2500, 800, 810, 790, 805, for (var i = 0; i < 300; i++) 800 + (i % 7) * 3.0, 2500]; + drive(rr, [for (var i = 0; i < rr.length; i++) i * 800.0], 'edges'); + }); + test('n = 1, 2, 3 transitions', () { + final rr = [800, 810, 790, 805, 3000, 800]; + drive(rr, [for (var i = 0; i < rr.length; i++) i * 800.0], 'tiny'); + }); + }); + + test('REAL 8.9 h night: 15-min folds, restart each, 6 checkpoints', () { + final s = realNightRr()!; + final cuts = timeCuts(s.ts, 900); + final run = _Run(RrCorrector()); + var from = 0; + final checks = {0, cuts.length ~/ 5, cuts.length ~/ 2, cuts.length - 2, + cuts.length - 1}; + for (var k = 0; k < cuts.length; k++) { + run.fold(s.rr.sublist(from, cuts[k]), s.ts.sublist(from, cuts[k])); + from = cuts[k]; + run.restart(); + if (checks.contains(k)) { + _expectSame(run, s.rr, s.ts, cuts[k], why: 'real k=$k'); + } + } + }, timeout: const Timeout(Duration(minutes: 5))); +} diff --git a/test/incremental/support.dart b/test/incremental/support.dart new file mode 100644 index 0000000..64dff6a --- /dev/null +++ b/test/incremental/support.dart @@ -0,0 +1,31 @@ +import 'package:openstrap_analytics/onehz.dart' + hide RrCorrector, RrSettled, RrSnapshot, IrregularScreenState; +import 'package:test/test.dart'; + +/// Max-error recorder for approximate fields. +class Err { + double maxAbs = 0, maxRel = 0; + int n = 0; + void add(double? a, double? b) { + if (a == null || b == null) { + expect(a, b); + return; + } + n++; + final d = (a - b).abs(); + if (d > maxAbs) maxAbs = d; + final rel = b == 0 ? d : d / b.abs(); + if (rel > maxRel) maxRel = rel; + } + + @override + String toString() => 'n=$n maxAbs=${maxAbs.toStringAsExponential(2)} maxRel=${maxRel.toStringAsExponential(2)}'; +} + +void sameEnvelope(Metric a, Metric b, {String? why, bool exactConf = true}) { + expect(a.present, b.present, reason: 'present $why'); + expect(a.tier, b.tier, reason: 'tier $why'); + expect(a.inputs_used, b.inputs_used, reason: 'inputs $why'); + expect(a.note, b.note, reason: 'note $why'); + if (exactConf) expect(a.confidence, b.confidence, reason: 'conf $why'); +} diff --git a/test/incremental/thin_input_test.dart b/test/incremental/thin_input_test.dart new file mode 100644 index 0000000..2448448 --- /dev/null +++ b/test/incremental/thin_input_test.dart @@ -0,0 +1,117 @@ +// "Never fabricate": at EVERY short prefix (1-beat folds, n = 0..~700) every +// incremental estimator is absent exactly where its oracle is absent, present +// exactly where it is present, with the same note. +import 'package:openstrap_analytics/onehz.dart' + hide RrCorrector, RrSettled, RrSnapshot, IrregularScreenState; +import 'package:test/test.dart'; + +import '../../tool/incremental/driver.dart'; +import '../../tool/incremental/edge_oracles.dart'; +import '../../tool/incremental/hrv_incr.dart'; +import '../../tool/incremental/resp_incr.dart'; +import '../../tool/incremental/synth.dart'; +import 'support.dart'; + +void main() { + for (final name in ['clean', 'dirty']) { + test('thin-input sweep, 1-beat folds ($name)', () { + final s = name == 'clean' + ? synthRr(const SynthConfig( + seed: 101, + hours: 0.4, + ectopicPerMin: 0, + missedPerMin: 0, + extraPerMin: 0, + noiseRunPerMin: 0, + gapPerHour: 0)) + : synthRr(const SynthConfig( + seed: 102, hours: 0.4, ectopicPerMin: 3, noiseRunPerMin: 1, gapPerHour: 20)); + final n = s.length < 760 ? s.length : 760; + final sub = RrData(s.rr.sublist(0, n), s.ts.sublist(0, n)); + final cuts = [for (var i = 1; i <= n; i++) i]; + final acc = HrvTimeAcc(); + final noc = NocturnalRmssdState(); + final shape = NightShapeState(binMin: 2, minBeatsPerBin: 20); + final rsa = RsaWelchState(); + final hf = HrvFreqState(); + final irr = IrregularScreenState(); + final origin = sub.ts.first - sub.rr.first; + final tl = HrvTimelineState(origin); + final wins = RespWindowsState(windowMs: 120000, minBeats: 20); + var presentCount = {}; + void tally(String k, bool p) => presentCount[k] = (presentCount[k] ?? 0) + (p ? 1 : 0); + driveRr(sub, cuts, (f) { + acc.fold(f.settled.nn, f.settled.nnTimes); + noc.fold(f.settled.nn, f.settled.nnTimes); + shape.fold(f.settled.nn, f.settled.nnTimes); + rsa.fold(f.settled.nn, f.settled.nnTimes); + hf.fold(f.settled.nn, f.settled.nnTimes); + irr.fold(f.settled.nn, f.settled.nnTimes); + tl.fold(f.settled.nn, f.settled.nnTimes); + wins.fold(f.settled.nn, f.settled.nnTimes); + if (!(f.n <= 130 || f.n % 9 == 0)) return; + final o = correctRr(sub.rr.sublist(0, f.n), rrTsMs: sub.ts.sublist(0, f.n)); + final af = (1.0 - o.cleanFraction).clamp(0.0, 1.0).toDouble(); + final why = '$name n=${f.n}'; + // hrvTime + var w = hrvTime(o.nn, nnTimesMs: o.nnTimesMs, artifactFraction: af); + var g = acc.evaluate(f.snap.tailNn, f.snap.tailNnTimes, artifactFraction: af); + expect(g.present, w.present, reason: 'hrvTime $why'); + expect(g.note, w.note); + tally('hrvTime', w.present); + if (w.present) { + expect(g.value!.rmssd == null, w.value!.rmssd == null, reason: why); + expect(g.value!.sdann == null, w.value!.sdann == null, reason: why); + expect(g.value!.sdnnIndex == null, w.value!.sdnnIndex == null, reason: why); + expect(g.value!.diffAcf1 == null, w.value!.diffAcf1 == null, reason: why); + } + // nocturnal + final wn = nocturnalRmssd(o.nn, o.nnTimesMs); + final gn = noc.evaluate(f.snap.tailNn, f.snap.tailNnTimes); + expect(gn.present, wn.present, reason: 'nocturnal $why'); + expect(gn.note, wn.note); + tally('nocturnalRmssd', wn.present); + // shape (2-min bins so the 3-bin gate is reachable at this size) + final ws = nightHrvShape(o.nn, o.nnTimesMs, minBeatsPerBin: 20, binMin: 2); + final gs = shape.evaluate(f.snap.tailNn, f.snap.tailNnTimes); + sameEnvelope(gs, ws, why: 'shape $why'); + if (ws.present) expect(gs.value!.toJson(), ws.value!.toJson(), reason: why); + tally('nightShape', ws.present); + // rsa + final wr = rsaRespRate(o.nn, o.nnTimesMs, artifactFraction: af); + final gr = rsa.evaluate(f.snap.tailNn, f.snap.tailNnTimes, artifactFraction: af); + expect(gr.present, wr.present, reason: 'rsa $why'); + expect(gr.note, wr.note); + expect(gr.value?.brpm, wr.value?.brpm); + tally('rsa', wr.present); + // hrvFreq + final wf = hrvFreq(o.nn, o.nnTimesMs, artifactFraction: af); + final gf = hf.evaluate(f.snap.tailNn, f.snap.tailNnTimes, artifactFraction: af); + expect(gf.present, wf.present, reason: 'hrvFreq $why'); + expect(gf.note, wf.note); + tally('hrvFreq', wf.present); + if (wf.present) { + expect(gf.value!.lf, wf.value!.lf, reason: why); + expect(gf.value!.hf, wf.value!.hf, reason: why); + expect(gf.value!.vlf, wf.value!.vlf, reason: why); + } + // irregular (500-beat floor: absent until then, exactly) + final wi = irregularBeatScreen(o.nn, nnTimesMs: o.nnTimesMs, artifactFraction: af); + final gi = irr.evaluate(f.snap.tailNn, f.snap.tailNnTimes, artifactFraction: af); + expect(gi.present, wi.present, reason: 'irregular $why'); + expect(gi.note, wi.note); + tally('irregular', wi.present); + // timeline + final wt = oracleHrvTimeline(o.nn, o.nnTimesMs, origin); + expect(tl.curve(f.snap.tailNn, f.snap.tailNnTimes).length, wt.length, + reason: 'timeline $why'); + // resp windows + final ww = oracleRespPerWindow(o.nn, o.nnTimesMs, windowMs: 120000, minBeats: 20); + expect(wins.evaluate(f.snap.tailNn, f.snap.tailNnTimes), orderedEquals(ww), + reason: 'respWindows $why'); + }); + // ignore: avoid_print + print('$name: present-at-checked-prefix counts $presentCount'); + }); + } +} diff --git a/test/onehz/advanced_stager_test.dart b/test/onehz/advanced_stager_test.dart index 29424f5..7a8ffc2 100644 --- a/test/onehz/advanced_stager_test.dart +++ b/test/onehz/advanced_stager_test.dart @@ -9,6 +9,8 @@ import 'dart:math' as math; import 'package:test/test.dart'; import 'package:openstrap_analytics/onehz.dart'; +import 'support/pre_bump_stager.dart'; + void main() { // Build a synthetic capture at 1 Hz: [dayH] active day, [nightH] still night // with low sleeping HR + physiological RR, then [tailH] active. @@ -315,6 +317,193 @@ void main() { }); }); + group('grid option 2: staging anchors its epochs at the session start', () { + // Owner decision 2026-10-23: the gap-anchored grid leaves the bump. The + // default stager's labels equal the pre-bump stager's (frozen as an oracle + // in support/pre_bump_stager.dart, logic of 83fc308), whatever run the + // session starts in. + test('detected sessions: labels equal the pre-bump stager exactly', () { + final d = build(2, 7, 1); + final sessions = AdvancedSleepStager.detectSleep(d.grav, d.hr, rr: d.rr); + expect(sessions, isNotEmpty); + for (final s in sessions) { + final want = preBumpStageSessionCardio(s.start, s.end, d.grav, d.hr, d.rr); + expect(showSegments(s.stages), showSegments(want), + reason: 'session ${s.start}..${s.end}'); + } + }); + + test('a session that starts mid-run: every boundary is on ITS grid ' + '(start + k * 30 s), not the run\'s', () { + final d = build(2, 7, 1); + for (final (from, to) in [(777, 9000), (1234, 20000), (4321, 25000)]) { + final start = d.nightStart + from, end = d.nightStart + to; + final got = AdvancedSleepStager.stageWindow(start, end, d.grav, d.hr, + rr: d.rr); + var checked = 0; + for (var i = 1; i < got.stages.length; i++) { + final g = got.stages[i]; + if (g.stage == 'unobserved' || got.stages[i - 1].stage == 'unobserved') { + continue; + } + expect((g.start - start) % 30, 0, + reason: 'window +$from..+$to, boundary ${g.start}'); + checked++; + } + expect(checked, greaterThan(5), reason: 'window +$from..+$to'); + } + }); + + test('a session that starts mid-run: labels equal the pre-bump stager ' + 'exactly (epochs from the start, features clipped at the session)', () { + // Offsets of 0 and 30 s share the run's phase but not its context: the + // pre-bump stager reads nothing outside [start, end). + final d = build(2, 7, 1); + for (final (from, to) in [(30, 9000), (777, 9000), (1234, 20000), (4321, 25000)]) { + final start = d.nightStart + from, end = d.nightStart + to; + final got = AdvancedSleepStager.stageWindow(start, end, d.grav, d.hr, + rr: d.rr); + final want = preBumpStageSessionCardio(start, end, d.grav, d.hr, d.rr); + expect(showSegments(got.stages), showSegments(want), + reason: 'window +$from..+$to'); + } + }); + + test('after a dropout longer than the carry the next sample re-anchors ' + '(pre-bump); before it the grid is the session start\'s', () { + // Two still blocks, 110 s apart (> the 61 s usable-run gap; the second + // starts at 8110, 8110 % 30 = 10). + final g = stillBlocks([(5000, 3000), (8110, 3000)]); + final rr = [ + for (var t = 5000; t < 11110; t++) + if (t < 8000 || t >= 8110) RrTs(t, 1150.0 + (t.isEven ? 20 : -20)) + ]; + for (final start in [5000, 5017, 5400]) { + final got = AdvancedSleepStager.stageWindow(start, 11110, g.grav, g.hr, + rr: rr); + var before = 0, after = 0; + for (var i = 1; i < got.stages.length; i++) { + final a = got.stages[i - 1], b = got.stages[i]; + if (a.stage == 'unobserved' || b.stage == 'unobserved') continue; + if (b.start < 8000) { + expect((b.start - start) % 30, 0, reason: 'start $start, run 1: ${b.start}'); + before++; + } else { + expect((b.start - 8110) % 30, 0, reason: 'start $start, run 2: ${b.start}'); + after++; + } + } + expect(before, greaterThan(0), reason: 'start $start'); + expect(after, greaterThan(0), reason: 'start $start'); + } + }); + }); + + group('pre-bump parity edge cases', () { + void parity(int start, int end, List grav, List hr, + [List rr = const []]) { + final got = AdvancedSleepStager.stageWindow(start, end, grav, hr, rr: rr); + final want = preBumpStageSessionCardio(start, end, grav, hr, rr); + expect(showSegments(got.stages), showSegments(want)); + } + + List flatGrav(int from, int to) => + [for (var t = from; t < to; t++) GravTs(t, 0, 0, 1)]; + + test('sparse HR (0, 30, 60 s) over [0,90): light, as before', () { + parity(0, 90, flatGrav(0, 90), [for (final t in [0, 30, 60]) HrTs(t, 60)]); + }); + + test('sparse HR (seconds 0..44 only) over [0,119)', () { + parity(0, 119, flatGrav(0, 119), [for (var t = 0; t < 45; t++) HrTs(t, 60)]); + }); + + test('gravity context across an accel gap [500,580)', () { + const bump = { + 19: 1.03, 20: 1.02, 22: 1.02, 23: 1.02, 27: 1.03, 29: 1.02, + 30: 1.02, 31: 1.01, 32: 1.02, 36: 1.02, 38: 1.03, 39: 1.03, + }; + final grav = [ + for (var t = 0; t < 1200; t++) + if (t < 500 || t >= 580) GravTs(t, 0, 0, bump[t ~/ 30] ?? 1.0) + ]; + parity(0, 1200, grav, [for (var t = 0; t < 1200; t++) HrTs(t, 60)]); + }); + + for (final (name, bad) in [('NaN', double.nan), ('infinite', double.infinity)]) { + test('non-finite RR ($name) does not throw; labels equal the oracle', () { + parity(0, 90, flatGrav(0, 90), [for (var t = 0; t < 90; t++) HrTs(t, 60)], + [RrTs(10, bad)]); + }); + } + + test('duplicate HR timestamps (60 then 0): last value wins as before', () { + parity(0, 90, flatGrav(0, 90), [ + for (var t = 0; t < 90; t++) ...[HrTs(t, 60), HrTs(t, 0)] + ]); + }); + + test('a pre-81b9b4b (EPOCH2) cache entry is never served', () { + // Keys and values as the old extractor stored them for gravity (0,0,1) + // each second and HR 60 then 0 at each second over [0,90): HR 60 where + // the oracle's last-wins reading has none (0-90:unobserved). + final stale = CardioRowCache.fromJson({ + 'version': 2, + 'rows': [ + for (final (s, fp) in [ + (0, '-518569928497453641'), + (30, '6424718457559164343'), + (60, '6000305869726000567'), + ]) + { + 'startSec': s, + 'epochSec': 30, + 'fingerprint': fp, + 'motion': 0.0, + 'hr': 60.0, + 'hrSd': 0.0, + 'rmssd': 'NaN', + 'lfhf': 'NaN', + 'rk': 'NaN', + 'sdnn': 'NaN', + } + ], + }); + final grav = flatGrav(0, 90); + final hr = [for (var t = 0; t < 90; t++) ...[HrTs(t, 60), HrTs(t, 0)]]; + final got = AdvancedSleepStager.stageWindow(0, 90, grav, hr, + rowCache: stale); + expect(stale.hits, 0, reason: 'old-identity entries must not hit'); + expect(showSegments(got.stages), + showSegments(preBumpStageSessionCardio(0, 90, grav, hr, const []))); + }); + + test('gravity in reverse time order: labels equal the oracle', () { + parity(0, 90, flatGrav(0, 90).reversed.toList(), + [for (var t = 0; t < 90; t++) HrTs(t, 60)]); + }); + + test('HR in reverse time order: labels equal the oracle', () { + parity(0, 90, flatGrav(0, 90), + [for (var t = 89; t >= 0; t--) HrTs(t, 60)]); + }); + + test('unsorted HR with duplicate seconds keeps list-order last-wins', () { + // Seconds descending, but within a second the records stay (0, 60): the + // oracle keeps 60 (the later record), so the night is stageable. + parity(0, 90, flatGrav(0, 90), [ + for (var t = 89; t >= 0; t--) ...[HrTs(t, 0), HrTs(t, 60)] + ]); + }); + + test('RR in reverse time order does not throw (Sol r3)', () { + // [0,30): the oracle abstains even with assertions on (longer windows + // assert sorted RR there). + parity(0, 30, flatGrav(0, 30), [for (var t = 0; t < 30; t++) HrTs(t, 60)], + [RrTs(1, 900), RrTs(0, 900)]); + }); + }); + group('AASM hypnogram metrics (hand-checked)', () { test('TIB/TST/SOL/WASO/REM-latency from a crafted hypnogram', () { // 1000-s session: 0-100 wake (latency), 100-400 light, 400-600 deep, diff --git a/test/onehz/cardio_epoch_features_test.dart b/test/onehz/cardio_epoch_features_test.dart new file mode 100644 index 0000000..36514c5 --- /dev/null +++ b/test/onehz/cardio_epoch_features_test.dart @@ -0,0 +1,260 @@ +// cardioEpochFeatures (round 15, phase 0) - RED until the implementation lands. +// +// ASSUMED API (derived from reading cardioStager; the implementer may change +// the CALL SHAPE below, never the assertions): +// +// CardioEpochFeatures cardioEpochFeatures( +// List hr1hz, +// List accel, { +// List rrMs = const [], +// List rrTsMs = const [], +// int epochSec = 30, +// int startEpoch = 0, // first epoch row to compute (inclusive) +// int? endEpoch, // one past the last row; null = through nEpoch +// }); +// +// Same inputs as `cardioStager` (so the stager is literally +// `classifyCardioEpochs(cardioEpochFeatures(...), ...)`). The whole-night inputs +// stay whole: the epoch grid (cadence, perEpoch, nEpoch = n ~/ perEpoch) is +// derived from them exactly as today, and [startEpoch, endEpoch) only selects +// which rows are computed, clamped to [0, nEpoch]. Row k of a sub-range call is +// epoch startEpoch + k of the full call. When the whole input cannot be staged +// (cadence abstain, nEpoch < 3) the result has zero-length lists for any range. +// +// Pinned by this file: +// 1. cardioStager(...) == classifyCardioEpochs(cardioEpochFeatures(...)) +// on every result field, bit for bit. +// 2. rows for random sub-ranges [a, b) == rows a..b-1 of the full computation, +// all seven feature lists, bit for bit (NaN-ness included). + +import 'dart:math' as math; + +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +bool sameD(double a, double b) { + if (a.isNaN || b.isNaN) return a.isNaN && b.isNaN; + return a == b && a.isNegative == b.isNegative; +} + +Map> cols(CardioEpochFeatures f) => { + 'motion': f.motion, + 'hr': f.hr, + 'hrSd': f.hrSd, + 'rmssd': f.rmssd, + 'lfhf': f.lfhf, + 'rk': f.rk, + 'sdnn': f.sdnn, + }; + +void expectRowsEqual( + CardioEpochFeatures sub, CardioEpochFeatures full, int a, int b, String ctx) { + final s = cols(sub), f = cols(full); + for (final k in s.keys) { + expect(s[k]!.length, b - a, reason: '$ctx $k length for [$a,$b)'); + for (var i = 0; i < b - a; i++) { + if (!sameD(s[k]![i], f[k]![a + i])) { + fail('$ctx $k row ${a + i} of [$a,$b): ${s[k]![i]} vs ${f[k]![a + i]}'); + } + } + } +} + +void expectSameStager(CardioStagerResult a, CardioStagerResult b) { + expect(a.base.stages, b.base.stages); + expect(a.base.epochSec, b.base.epochSec); + expect(sameD(a.base.wakePct, b.base.wakePct), isTrue); + expect(sameD(a.base.nremPct, b.base.nremPct), isTrue); + expect(sameD(a.base.remPct, b.base.remPct), isTrue); + expect(a.deepFlag, b.deepFlag); + expect(sameD(a.confidence, b.confidence), isTrue); +} + +({List hr, List accel, List rr, List rrTs}) + genNight( + int seed, + int seconds, { + int cadenceSec = 1, + bool nanAccel = false, + bool hrDropouts = false, + bool rrOutliers = false, + bool withRr = true, +}) { + final r = math.Random(seed); + final n = seconds ~/ cadenceSec; + final hr = []; + final accel = []; + final rr = []; + final rrTs = []; + var posture = 1.0; + var moveLeft = 0; + var hrBase = 62.0; + for (var i = 0; i < n; i++) { + final ts = i * cadenceSec * 1000.0; + if (r.nextInt(900) == 0) posture = 0.95 + r.nextDouble() * 0.1; + if (moveLeft == 0 && r.nextInt(500) == 0) moveLeft = 5 + r.nextInt(40); + var m = posture + (r.nextDouble() - 0.5) * 0.002; + if (moveLeft > 0) { + m += r.nextDouble() * 0.3; + moveLeft--; + } + if (r.nextInt(2000) == 0) hrBase = 52 + r.nextDouble() * 25; + final cyc = 4 * math.sin(2 * math.pi * i * cadenceSec / 5400.0); + var h = hrBase + cyc + (r.nextDouble() - 0.5) * 4; + if (moveLeft > 0) h += 12; + if (hrDropouts && (i ~/ 200) % 7 == 3) h = 0; + if (hrDropouts && r.nextInt(300) == 0) h = double.nan; + hr.add(h); + var gx = m * 0.6; + final gy = m * 0.64, gz = m * 0.48; + if (nanAccel && r.nextInt(400) == 0) gx = double.nan; + accel.add(AccelSample(ts, gx, gy, gz)); + if (withRr) { + final nb = h.isNaN || h <= 0 ? 0 : (h * cadenceSec / 60).round(); + for (var b = 0; b < nb; b++) { + var v = 60000.0 / h + + (r.nextDouble() - 0.5) * 60 + + 40 * math.sin(2 * math.pi * i * cadenceSec / 25.0); + if (rrOutliers && r.nextInt(60) == 0) { + v = r.nextBool() ? 150 : 2600; + } else if (rrOutliers && r.nextInt(80) == 0) { + v += 350; + } + rr.add(v); + rrTs.add(ts); + } + } + } + return (hr: hr, accel: accel, rr: rr, rrTs: rrTs); +} + +void main() { + group('stager entry point == classify(features)', () { + void check(String name, int seed, int seconds, + {int cadenceSec = 1, + bool nanAccel = false, + bool hrDropouts = false, + bool rrOutliers = false, + bool withRr = true, + int epochSec = 30}) { + test(name, () { + final g = genNight(seed, seconds, + cadenceSec: cadenceSec, + nanAccel: nanAccel, + hrDropouts: hrDropouts, + rrOutliers: rrOutliers, + withRr: withRr); + final viaStager = cardioStager(g.hr, g.accel, + rrMs: g.rr, rrTsMs: g.rrTs, epochSec: epochSec); + final feats = cardioEpochFeatures(g.hr, g.accel, + rrMs: g.rr, rrTsMs: g.rrTs, epochSec: epochSec); + final viaParts = classifyCardioEpochs(feats, epochSec: epochSec); + expectSameStager(viaStager, viaParts); + }); + } + + check('40 min with RR', 61, 2400); + check('50 min, outliers + dropouts', 62, 3000, + rrOutliers: true, hrDropouts: true); + check('40 min, NaN accel', 63, 2400, nanAccel: true); + check('40 min without RR', 64, 2400, withRr: false); + check('10 s cadence', 65, 5400, cadenceSec: 10); + check('epochSec 60', 66, 2400, epochSec: 60); + check('5 s cadence abstains', 67, 3600, cadenceSec: 5); + check('2 epochs abstains', 68, 70); + }); + + group('feature rows', () { + test('full call has one row per epoch in all seven lists', () { + final g = genNight(71, 2400); + final f = cardioEpochFeatures(g.hr, g.accel, rrMs: g.rr, rrTsMs: g.rrTs); + expect(f.motion.length, 2400 ~/ 30); + for (final c in cols(f).values) { + expect(c.length, 80); + } + }); + + test('abstaining inputs give zero-length rows for any range', () { + final g = genNight(72, 70); + final f = cardioEpochFeatures(g.hr, g.accel, rrMs: g.rr, rrTsMs: g.rrTs); + expect(f.length, 0); + final f2 = cardioEpochFeatures(g.hr, g.accel, + rrMs: g.rr, rrTsMs: g.rrTs, startEpoch: 0, endEpoch: 2); + expect(f2.length, 0); + }); + + void subRanges(String name, int seed, int seconds, + {int cadenceSec = 1, + bool nanAccel = false, + bool hrDropouts = false, + bool rrOutliers = false}) { + test(name, () { + final g = genNight(seed, seconds, + cadenceSec: cadenceSec, + nanAccel: nanAccel, + hrDropouts: hrDropouts, + rrOutliers: rrOutliers); + final full = + cardioEpochFeatures(g.hr, g.accel, rrMs: g.rr, rrTsMs: g.rrTs); + final n = full.length; + expect(n, greaterThan(10)); + final r = math.Random(seed); + final ranges = >[ + [0, n], + [0, 1], + [n - 1, n], + [n, n], + [3, 3], + [0, 0], + [0, math.min(8, n)], + [math.max(0, n - 8), n], + for (var i = 0; i < 12; i++) + () { + final a = r.nextInt(n); + return [a, a + r.nextInt(n - a + 1)]; + }(), + ]; + for (final ab in ranges) { + final sub = cardioEpochFeatures(g.hr, g.accel, + rrMs: g.rr, + rrTsMs: g.rrTs, + startEpoch: ab[0], + endEpoch: ab[1]); + expectRowsEqual(sub, full, ab[0], ab[1], name); + } + }); + } + + subRanges('plain night', 81, 2700); + subRanges('RR outliers + HR dropouts', 82, 2700, + rrOutliers: true, hrDropouts: true); + subRanges('NaN accel magnitudes', 83, 2700, nanAccel: true); + subRanges('10 s cadence', 84, 5400, cadenceSec: 10); + + test('open-ended range and out-of-range bounds clamp', () { + final g = genNight(85, 2400); + final full = + cardioEpochFeatures(g.hr, g.accel, rrMs: g.rr, rrTsMs: g.rrTs); + final n = full.length; + final tail = cardioEpochFeatures(g.hr, g.accel, + rrMs: g.rr, rrTsMs: g.rrTs, startEpoch: 10); + expectRowsEqual(tail, full, 10, n, 'open-ended'); + final over = cardioEpochFeatures(g.hr, g.accel, + rrMs: g.rr, rrTsMs: g.rrTs, startEpoch: n - 5, endEpoch: n + 100); + expectRowsEqual(over, full, n - 5, n, 'clamped end'); + final past = cardioEpochFeatures(g.hr, g.accel, + rrMs: g.rr, rrTsMs: g.rrTs, startEpoch: n + 3, endEpoch: n + 9); + expect(past.length, 0); + }); + + test('no RR at all: rows are NaN for rmssd/sdnn/lfhf/rk, in any range', () { + final g = genNight(86, 2400, withRr: false); + final sub = cardioEpochFeatures(g.hr, g.accel, + startEpoch: 20, endEpoch: 30); + expect(sub.length, 10); + for (final c in [sub.rmssd, sub.sdnn, sub.lfhf, sub.rk]) { + expect(c.every((v) => v.isNaN), isTrue); + } + }); + }); +} diff --git a/test/onehz/cardio_row_cache_test.dart b/test/onehz/cardio_row_cache_test.dart new file mode 100644 index 0000000..a0c0caf --- /dev/null +++ b/test/onehz/cardio_row_cache_test.dart @@ -0,0 +1,550 @@ +// Exact cardio epoch-row cache (round 16) - RED until the implementation lands. +// +// Spec: openstrap-analytics.research/r16-row-cache.md. The contract is that a +// `CardioRowCache` passed as `rowCache:` to segmentSleep changes NOTHING about +// the output (bit-identical to no cache, every field), only how many epoch rows +// are recomputed. +// +// DECISIONS (pinned here; the implementer may not change the assertions): +// * `CardioRowCache.fromJson` THROWS FormatException on malformed input and on +// a missing / non-matching `version` field. It never returns a cache holding +// guessed rows. (Callers that want "empty cache on failure" catch it.) +// * `hits` / `misses` may be cumulative over the cache's life; the tests only +// ever use deltas around one call. +// * `CardioRowKey` exposes `startSec` (absolute epoch start, the session +// clock) and `epochSec`. These two names are the only key members used here +// (retainWhere predicates). Everything else about the key is opaque. +// * Change detection is checked as: the changed sample forces at least one +// miss, and every epoch whose full context (own epoch +- 150 s, plus one +// epoch of slack for window edges) cannot contain it still hits. +// * "Most epochs hit when only the end moved" = hits >= 80 % of the epochs the +// first run staged that still lie inside the second session. +// +// Synthetic data: a few hours at 1 Hz for forced-window cases; ONE full night +// for the auto-detect case. + +import 'dart:convert'; +import 'dart:math' as math; + +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +const int t0 = 1700000000; // absolute seconds, record start + +class Night { + final List accel; + final List hr; + final List rr; + final List rrTs; + Night(this.accel, this.hr, this.rr, this.rrTs); + + /// Samples with absolute second in [a, b). + Night slice(int a, int b) { + final ai = []; + final hi = []; + for (var i = 0; i < accel.length; i++) { + final s = accel[i].tsMs ~/ 1000; + if (s >= a && s < b) { + ai.add(accel[i]); + hi.add(hr[i]); + } + } + final r = [], rt = []; + for (var i = 0; i < rr.length; i++) { + final s = rrTs[i] ~/ 1000; + if (s >= a && s < b) { + r.add(rr[i]); + rt.add(rrTs[i]); + } + } + return Night(ai, hi, r, rt); + } + + /// Drop accel+hr positions and/or RR beats whose absolute second is in a + /// range (accel gaps are holes in the positional arrays, like real syncs). + Night withGaps({ + List<(int, int)> accelGaps = const [], + List<(int, int)> rrGaps = const [], + }) { + bool inAny(List<(int, int)> g, int s) => + g.any((r) => s >= r.$1 && s < r.$2); + final ai = []; + final hi = []; + for (var i = 0; i < accel.length; i++) { + if (inAny(accelGaps, accel[i].tsMs ~/ 1000)) continue; + ai.add(accel[i]); + hi.add(hr[i]); + } + final r = [], rt = []; + for (var i = 0; i < rr.length; i++) { + if (inAny(rrGaps, rrTs[i] ~/ 1000)) continue; + r.add(rr[i]); + rt.add(rrTs[i]); + } + return Night(ai, hi, r, rt); + } + + Night copy() => Night(List.of(accel), List.of(hr), List.of(rr), List.of(rrTs)); +} + +/// head minutes active, still minutes asleep, tail minutes active. +Night genNight(int seed, + {int headMin = 40, int stillMin = 150, int tailMin = 30, int start = t0}) { + final rnd = math.Random(seed); + final headS = headMin * 60, stillS = stillMin * 60, tailS = tailMin * 60; + final n = headS + stillS + tailS; + final accel = []; + final hr = []; + final rr = []; + final rrTs = []; + var hrBase = 50.0; + var moveLeft = 0; + var zBase = 1.0; + for (var i = 0; i < n; i++) { + final ts = (start + i) * 1000.0; + final active = i < headS || i >= headS + stillS; + double x, y = 0, z; + double h; + if (active) { + x = i.isEven ? 0.0 : 0.25; + z = 0.97; + h = 76 + (rnd.nextDouble() - 0.5) * 6; + } else { + if (rnd.nextInt(900) == 0) zBase = 0.97 + rnd.nextDouble() * 0.06; + if (moveLeft == 0 && rnd.nextInt(600) == 0) moveLeft = 5 + rnd.nextInt(30); + x = (rnd.nextDouble() - 0.5) * 0.002; + z = zBase + (rnd.nextDouble() - 0.5) * 0.002; + if (moveLeft > 0) { + x += rnd.nextDouble() * 0.3; + y += rnd.nextDouble() * 0.2; + moveLeft--; + } + if (rnd.nextInt(1500) == 0) hrBase = 47 + rnd.nextDouble() * 10; + h = hrBase + + 3 * math.sin(2 * math.pi * i / 5400.0) + + (rnd.nextDouble() - 0.5) * 3 + + (moveLeft > 0 ? 10 : 0); + } + accel.add(AccelSample(ts, x, y, z)); + hr.add(h); + final nb = (h / 60).round(); + for (var b = 0; b < nb; b++) { + rr.add(60000.0 / h + + (rnd.nextDouble() - 0.5) * 50 + + 35 * math.sin(2 * math.pi * i / 25.0)); + rrTs.add(ts); + } + } + return Night(accel, hr, rr, rrTs); +} + +SleepSegmentation stage(Night d, + {CardioRowCache? cache, int? onset, int? offset}) { + return segmentSleep( + d.accel, + d.hr, + rrMs: d.rr, + rrTsMs: d.rrTs, + tzOffsetSec: 0, + forcedWindow: onset == null || offset == null + ? null + : (onsetSec: onset, offsetSec: offset), + rowCache: cache, + ); +} + +String fingerprint(SleepSegmentation s) => jsonEncode({ + 'json': s.toJson(), + 'stages': s.stages.map((e) => e.toString()).toList(), + 'stages4': s.stages4, + 'conf': s.confidence.toString(), + 'eff': s.efficiencyPct.toString(), + }); + +/// Runs [stage] with [cache], returns (output, hitsDelta, missesDelta). +({SleepSegmentation out, int hits, int misses}) staged(Night d, + CardioRowCache cache, {int? onset, int? offset}) { + final h0 = cache.hits, m0 = cache.misses; + final out = stage(d, cache: cache, onset: onset, offset: offset); + return (out: out, hits: cache.hits - h0, misses: cache.misses - m0); +} + +CardioRowCache copyOf(CardioRowCache c) => + CardioRowCache.fromJson( + jsonDecode(jsonEncode(c.toJson())) as Map); + +void expectIdentical(SleepSegmentation a, SleepSegmentation b, String ctx) { + expect(fingerprint(a), fingerprint(b), reason: ctx); +} + +void main() { + // Shared 3.7 h record for the forced-window cases: 40 min head, 150 min + // still, 30 min tail. Window = the still part plus margins. + final base = genNight(7, headMin: 5, stillMin: 60, tailMin: 10); + final total = base.accel.length; + final winA0 = t0 + 600, winA1 = t0 + total - 600; + + test('sanity: the synthetic record stages without a cache', () { + final s = stage(base, onset: winA0, offset: winA1); + expect(s.present, isTrue); + expect(s.stages4.length, winA1 - winA0); + }); + + group('with cache == without cache', () { + test('forced window, cold cache, then warm cache', () { + final ref = stage(base, onset: winA0, offset: winA1); + final cache = CardioRowCache(); + final cold = staged(base, cache, onset: winA0, offset: winA1); + expectIdentical(cold.out, ref, 'cold'); + expect(cold.hits, 0); + expect(cold.misses, greaterThan(100)); + expect(cache.length, cold.misses); + final warm = staged(base, cache, onset: winA0, offset: winA1); + expectIdentical(warm.out, ref, 'warm'); + expect(warm.misses, 0); + expect(warm.hits, cold.misses); + }); + + test('RR gaps and accel gaps', () { + final d = base.withGaps( + rrGaps: [(t0 + 1500, t0 + 1900), (t0 + 2000, t0 + 2100)], + accelGaps: [(t0 + 2800, t0 + 2890), (t0 + 3300, t0 + 3310)], + ); + final ref = stage(d, onset: winA0, offset: winA1); + final cache = CardioRowCache(); + expectIdentical( + staged(d, cache, onset: winA0, offset: winA1).out, ref, 'cold'); + final warm = staged(d, cache, onset: winA0, offset: winA1); + expectIdentical(warm.out, ref, 'warm'); + expect(warm.misses, 0); + }); + + test('no RR at all', () { + final d = Night(base.accel, base.hr, const [], const []); + final ref = stage(d, onset: winA0, offset: winA1); + final cache = CardioRowCache(); + expectIdentical( + staged(d, cache, onset: winA0, offset: winA1).out, ref, 'cold'); + expectIdentical( + staged(d, cache, onset: winA0, offset: winA1).out, ref, 'warm'); + }); + + test('short sessions (under 3 epochs, a few epochs, ragged length)', () { + for (final len in [60, 100, 301, 905]) { + final a = t0 + 1800, b = a + len; + final ref = stage(base, onset: a, offset: b); + final cache = CardioRowCache(); + expectIdentical( + staged(base, cache, onset: a, offset: b).out, ref, 'cold $len'); + expectIdentical( + staged(base, cache, onset: a, offset: b).out, ref, 'warm $len'); + } + }); + + test('a cache full of rows from a DIFFERENT night never leaks in', () { + final other = genNight(99); + final cache = CardioRowCache(); + staged(other, cache, onset: winA0, offset: winA1); + expect(cache.length, greaterThan(100)); + final ref = stage(base, onset: winA0, offset: winA1); + final r = staged(base, cache, onset: winA0, offset: winA1); + expectIdentical(r.out, ref, 'same clock, other data'); + // Same timestamps, different values: essentially nothing may hit. + expect(r.hits, lessThan(r.misses ~/ 10 + 1)); + }); + + test('auto-detect path on one full night, with and without appended data', + () { + final night = genNight(11, headMin: 40, stillMin: 200, tailMin: 30); + final nightEnd = t0 + night.accel.length; + final ref = stage(night); + expect(ref.present, isTrue, reason: 'synthetic night must qualify'); + final cache = CardioRowCache(); + final cold = staged(night, cache); + expectIdentical(cold.out, ref, 'cold'); + expect(cold.misses, greaterThan(300)); + final warm = staged(night, cache); + expectIdentical(warm.out, ref, 'warm'); + expect(warm.misses, 0); + + // Same night plus 15 min of new data appended after the session. + // Build "appended" by concatenation (15 min of new data after the end). + final extra = genNight(12, headMin: 15, stillMin: 0, tailMin: 0, + start: nightEnd); + final appended = Night( + [...night.accel, ...extra.accel], + [...night.hr, ...extra.hr], + [...night.rr, ...extra.rr], + [...night.rrTs, ...extra.rrTs], + ); + final ref2 = stage(appended); + final app = staged(appended, cache); + expectIdentical(app.out, ref2, 'appended'); + expect(app.hits, greaterThan(0)); + expect(app.hits, greaterThanOrEqualTo((cold.misses * 0.8).floor())); + }); + }); + + group('reuse across windows', () { + test('same data in a window whose arrays start 1 h earlier', () { + // Edge stages D then D+1 windows over overlapping record ranges. + final cache = CardioRowCache(); + final early = base.slice(t0 + 20 * 60, t0 + total); // array starts later + final first = staged(early, cache, onset: winA0 + 600, offset: winA1); + expectIdentical(first.out, + stage(early, onset: winA0 + 600, offset: winA1), 'first'); + final second = staged(base, cache, onset: winA0 + 600, offset: winA1); + expectIdentical(second.out, + stage(base, onset: winA0 + 600, offset: winA1), 'second'); + expect(second.hits, greaterThan(0)); + }); + + test('session start moved by k*30 s', () { + for (final k in [1, 5, 20]) { + final cache = CardioRowCache(); + final first = staged(base, cache, onset: winA0, offset: winA1); + final staged1 = first.misses; + final newOnset = winA0 + k * 30; + final alive = copyOf(cache) + ..retainWhere((key) => key.startSec >= newOnset); + final second = staged(base, cache, onset: newOnset, offset: winA1); + expectIdentical(second.out, + stage(base, onset: newOnset, offset: winA1), 'start +$k'); + expect(second.hits, greaterThan(0)); + expect(second.hits, greaterThanOrEqualTo((alive.length * 0.8).floor()), + reason: 'k=$k of $staged1 first-run rows'); + } + }); + + test('session start moved by a non-multiple of the epoch (grid shifts)', + () { + final cache = CardioRowCache(); + staged(base, cache, onset: winA0, offset: winA1); + final r = staged(base, cache, onset: winA0 + 7, offset: winA1); + expectIdentical(r.out, stage(base, onset: winA0 + 7, offset: winA1), + 'start +7 s'); + }); + + test('only the end moved (earlier and later): most epochs hit', () { + for (final delta in [-1200, -90, 600]) { + final cache = CardioRowCache(); + staged(base, cache, onset: winA0, offset: winA1); + final newEnd = winA1 + delta; + final alive = copyOf(cache) + ..retainWhere((key) => key.startSec + key.epochSec <= newEnd); + final second = staged(base, cache, onset: winA0, offset: newEnd); + expectIdentical(second.out, + stage(base, onset: winA0, offset: newEnd), 'end $delta'); + expect(second.hits, greaterThanOrEqualTo((alive.length * 0.8).floor()), + reason: 'end moved by $delta s'); + expect(second.hits, greaterThan(0)); + } + }); + + test('extra data appended after the session (forced window unchanged)', () { + final extra = genNight(5, headMin: 20, stillMin: 0, tailMin: 0, + start: t0 + total); + final more = Night( + [...base.accel, ...extra.accel], + [...base.hr, ...extra.hr], + [...base.rr, ...extra.rr], + [...base.rrTs, ...extra.rrTs], + ); + final cache = CardioRowCache(); + final first = staged(base, cache, onset: winA0, offset: winA1); + final second = staged(more, cache, onset: winA0, offset: winA1); + expectIdentical(second.out, + stage(more, onset: winA0, offset: winA1), 'appended'); + expect(second.misses, 0); + expect(second.hits, first.misses); + }); + }); + + group('change detection', () { + // Context of an epoch: own 30 s, +-150 s (accel median, RR). One more epoch + // of slack either side for window edges, so at most 13 rows may be dirty. + const maxDirty = 13; + + Future mutate(String name, Night Function(Night) f, int atSec) async { + test(name, () { + final cache = CardioRowCache(); + final first = staged(base, cache, onset: winA0, offset: winA1); + final d = f(base.copy()); + final second = staged(d, cache, onset: winA0, offset: winA1); + expectIdentical(second.out, + stage(d, onset: winA0, offset: winA1), name); + expect(second.misses, greaterThanOrEqualTo(1)); + expect(second.misses, lessThanOrEqualTo(maxDirty), reason: name); + expect(second.hits, first.misses - second.misses); + }); + } + + const mid = 2400; // seconds into the record, deep inside the window + + mutate('one HR sample', (d) { + d.hr[mid] += 11; + return d; + }, mid); + + mutate('one accel sample', (d) { + final a = d.accel[mid]; + d.accel[mid] = AccelSample(a.tsMs, a.x + 0.4, a.y, a.z); + return d; + }, mid); + + mutate('one RR beat value', (d) { + final i = d.rrTs.indexWhere((t) => t >= (t0 + mid) * 1000.0); + d.rr[i] += 23; + return d; + }, mid); + + mutate('one RR beat removed', (d) { + final i = d.rrTs.indexWhere((t) => t >= (t0 + mid) * 1000.0); + d.rr.removeAt(i); + d.rrTs.removeAt(i); + return d; + }, mid); + + test('sample at the very start of the window only dirties near rows', () { + final cache = CardioRowCache(); + final first = staged(base, cache, onset: winA0, offset: winA1); + final d = base.copy(); + final i = winA0 - t0 + 3; + d.hr[i] += 9; + final second = staged(d, cache, onset: winA0, offset: winA1); + expectIdentical( + second.out, stage(d, onset: winA0, offset: winA1), 'edge'); + expect(second.misses, inInclusiveRange(1, maxDirty)); + expect(second.hits, first.misses - second.misses); + }); + }); + + group('json round trip', () { + test('populated cache -> json text -> cache: identical staging, all hits', + () { + final cache = CardioRowCache(); + final cold = staged(base, cache, onset: winA0, offset: winA1); + final text = jsonEncode(cache.toJson()); + final back = CardioRowCache.fromJson( + jsonDecode(text) as Map); + expect(back.length, cache.length); + expect(jsonEncode(back.toJson()), text, reason: 'lossless, stable'); + final warm = staged(base, back, onset: winA0, offset: winA1); + expectIdentical(warm.out, cold.out, 'revived'); + expect(warm.misses, 0); + expect(warm.hits, cold.misses); + }); + + test('toJson is deterministic across independent runs', () { + final a = CardioRowCache(), b = CardioRowCache(); + staged(base, a, onset: winA0, offset: winA1); + staged(base.copy(), b, onset: winA0, offset: winA1); + expect(jsonEncode(a.toJson()), jsonEncode(b.toJson())); + }); + + test('empty cache round trips', () { + final e = CardioRowCache.fromJson( + jsonDecode(jsonEncode(CardioRowCache().toJson())) + as Map); + expect(e.length, 0); + }); + + test('wrong or missing version throws FormatException', () { + final cache = CardioRowCache(); + staged(base, cache, onset: winA0, offset: winA0 + 1800); + final good = jsonDecode(jsonEncode(cache.toJson())) as Map; + expect(good.containsKey('version'), isTrue); + for (final v in [null, 'x', -1, 0, 9999, good['version'] is int ? (good['version'] as int) + 1 : 2]) { + final bad = Map.of(good)..['version'] = v; + expect(() => CardioRowCache.fromJson(bad), + throwsA(isA()), + reason: 'version=$v'); + } + final noVersion = Map.of(good)..remove('version'); + expect(() => CardioRowCache.fromJson(noVersion), + throwsA(isA())); + }); + + test('malformed payloads throw FormatException, never wrong rows', () { + final cache = CardioRowCache(); + staged(base, cache, onset: winA0, offset: winA0 + 1800); + final good = jsonDecode(jsonEncode(cache.toJson())) as Map; + expect(() => CardioRowCache.fromJson({}), + throwsA(isA())); + for (final k in good.keys.where((k) => k != 'version')) { + final bad = Map.of(good)..[k] = 'garbage'; + expect(() => CardioRowCache.fromJson(bad), + throwsA(isA()), + reason: 'key $k replaced by a string'); + final v = good[k]; + if (v is List && v.isNotEmpty) { + final badFirst = Map.of(good) + ..[k] = ['garbage', ...v.skip(1)]; + expect(() => CardioRowCache.fromJson(badFirst), + throwsA(isA()), + reason: 'key $k first element garbage'); + final truncated = Map.of(good) + ..[k] = v.sublist(0, v.length - 1); + // A shorter list is either rejected or yields strictly fewer rows + // (a parallel-array layout MUST reject); it must never add rows. + try { + final c = CardioRowCache.fromJson(truncated); + expect(c.length, lessThanOrEqualTo(cache.length)); + } on FormatException { + // acceptable + } + } + } + }); + }); + + group('retainWhere and merge', () { + test('retainWhere prunes, kept rows still hit, dropped rows miss', () { + final cache = CardioRowCache(); + final first = staged(base, cache, onset: winA0, offset: winA1); + final n = cache.length; + final cut = winA0 + 1800; + cache.retainWhere((k) => k.startSec < cut); + expect(cache.length, lessThan(n)); + expect(cache.length, greaterThan(0)); + final second = staged(base, cache, onset: winA0, offset: winA1); + expectIdentical(second.out, first.out, 'pruned'); + expect(second.hits, greaterThan(0)); + expect(second.misses, greaterThan(0)); + expect(second.hits + second.misses, first.misses); + cache.retainWhere((_) => false); + expect(cache.length, 0); + }); + + test('merge unions two caches; merged cache hits both windows', () { + final a = CardioRowCache(), b = CardioRowCache(); + final midS = winA0 + 1800; + final ra = staged(base, a, onset: winA0, offset: midS); + final rb = staged(base, b, onset: midS, offset: winA1); + // Wall-clock rows whose context is shared by both windows have the + // same key, including a row at the seam. Union must deduplicate them. + final unionKeys = { + for (final cache in [a, b]) + for (final row in cache.toJson()['rows'] as List) + '${row['startSec']}:${row['epochSec']}:${row['fingerprint']}', + }; + a.merge(b); + expect(a.length, unionKeys.length); + expect(a.length, lessThanOrEqualTo(ra.misses + rb.misses)); + final w1 = staged(base, a, onset: winA0, offset: midS); + final w2 = staged(base, a, onset: midS, offset: winA1); + expect(w1.misses, 0); + expect(w2.misses, 0); + expectIdentical(w1.out, stage(base, onset: winA0, offset: midS), 'w1'); + expectIdentical(w2.out, stage(base, onset: midS, offset: winA1), 'w2'); + }); + + test('merging a copy of itself changes nothing', () { + final a = CardioRowCache(); + staged(base, a, onset: winA0, offset: winA0 + 1800); + final n = a.length; + a.merge(copyOf(a)); + expect(a.length, n); + }); + }); +} diff --git a/test/onehz/causal_stager_real_night_test.dart b/test/onehz/causal_stager_real_night_test.dart new file mode 100644 index 0000000..0ea55fc --- /dev/null +++ b/test/onehz/causal_stager_real_night_test.dart @@ -0,0 +1,245 @@ +// REGRESSION — the causal stager on the REAL WHOOP-4 overnight capture. +// +// Same fixture as real_night_cardio_stager_test.dart (8 h 54 min, anonymised, +// 1 Hz HR + accel + RR). There are no epoch labels for it — only the Apple +// Watch's whole-night minutes — so what is pinned here is AGREEMENT WITH THE +// RETROSPECTIVE STAGER, not accuracy: the online rules must keep tracking the +// offline ones, abstain only when they should, and keep the REM trigger from +// firing away from REM. `tool/causal_stager_validate.dart` prints the full +// report these bands were set from (kappa 0.369, agreement 70.3 %, causal wake +// 62.5 min vs offline 26.0, REM 78 vs 150 min). Bands are wide on purpose. + +import 'dart:io'; + +import 'package:test/test.dart'; +import 'package:openstrap_analytics/onehz.dart'; + +const double _t0Ms = 1700000010000; // 30 s-aligned + +class _Night { + final List hr = []; + final List accel = []; + final List rr = [], rrTs = []; +} + +_Night? _load() { + final a = File('test/onehz/fixtures/real_night_2026_07_onehz.csv'); + final b = File('test/onehz/fixtures/real_night_2026_07_rr.csv'); + if (!a.existsSync() || !b.existsSync()) return null; + final n = _Night(); + final l1 = a.readAsLinesSync(); + for (var i = 1; i < l1.length; i++) { + if (l1[i].trim().isEmpty) continue; + final p = l1[i].split(','); + n.hr.add(double.parse(p[1])); + n.accel.add(AccelSample(double.parse(p[0]) * 1000.0, double.parse(p[2]), + double.parse(p[3]), double.parse(p[4]))); + } + final l2 = b.readAsLinesSync(); + for (var i = 1; i < l2.length; i++) { + if (l2[i].trim().isEmpty) continue; + final p = l2[i].split(','); + n.rrTs.add(double.parse(p[0])); + n.rr.add(double.parse(p[1])); + } + return n; +} + +List _replay(_Night n, int toSec, + {double Function(int sec, double bpm)? hrOf}) { + final out = []; + CausalStagerState? st; + var rrI = 0; + for (var t = 30; t <= toSec; t += 30) { + final lo = t - 30; + final ts = [], v = []; + while (rrI < n.rrTs.length && n.rrTs[rrI] < t * 1000.0) { + ts.add(_t0Ms + n.rrTs[rrI]); + v.add(n.rr[rrI]); + rrI++; + } + final o = CausalStager.observe( + CausalSampleWindow( + nowMs: _t0Ms + t * 1000.0, + hr: [ + for (var i = lo; i < t; i++) + HrSample(_t0Ms + i * 1000.0, hrOf?.call(i, n.hr[i]) ?? n.hr[i]) + ], + accel: [ + for (var i = lo; i < t; i++) + AccelSample(_t0Ms + i * 1000.0, n.accel[i].x, n.accel[i].y, + n.accel[i].z) + ], + rr: RrSeries(ts, v), + ), + st); + out.add(o); + st = o.nextState; + } + return out; +} + +double _kappa(List a, List b) { + final n = a.length; + final m = List.generate(3, (_) => List.filled(3, 0)); + for (var i = 0; i < n; i++) { + m[a[i]][b[i]]++; + } + var po = 0.0, pe = 0.0; + for (var i = 0; i < 3; i++) { + po += m[i][i]; + var c = 0; + for (var j = 0; j < 3; j++) { + c += m[j][i]; + } + pe += m[i].reduce((x, y) => x + y) * c / n; + } + return (po / n - pe / n) / (1 - pe / n); +} + +void main() { + final night = _load(); + if (night == null) { + test('real night fixtures not found', () { + markTestSkipped('real night fixtures not found'); + }); + return; + } + final cfg = CausalStagerConfig.defaults; + final off = cardioStager(night.hr, night.accel, + rrMs: night.rr, rrTsMs: night.rrTs) + .base + .stages; + const idx = {CausalStage.wake: 0, CausalStage.nrem: 1, CausalStage.rem: 2}; + const base = 1700000010000 ~/ 30000; + late List obs; + setUpAll(() => obs = _replay(night, night.hr.length)); + + int offK(CausalStageObservation o) => + (o.epochStartMs! / 30000).round() - base; + + test('abstains only while warming up, then stages every epoch', () { + final absent = obs.where((o) => o.stage == CausalStage.absent).toList(); + expect(absent.length, cfg.warmupEpochs - 1); + expect(absent.every((o) => o.abstentionReason == CausalAbstention.warmup), + isTrue); + expect(obs.last.stage, isNot(CausalStage.absent)); + }); + + test('tracks the retrospective stager (kappa and agreement bands)', () { + final a = [], b = []; + for (final o in obs.where((o) => o.stage != CausalStage.absent)) { + final k = offK(o); + if (k < 0 || k >= off.length) continue; + a.add(idx[o.stage]!); + b.add(off[k].index); + } + var agree = 0; + for (var i = 0; i < a.length; i++) { + if (a[i] == b[i]) agree++; + } + expect(agree / a.length, greaterThan(0.62)); // measured 0.703 + expect(_kappa(a, b), greaterThan(0.28)); // measured 0.369 + }); + + test('stage minutes stay sane: no wake blow-up, REM not collapsed', () { + var wake = 0, rem = 0, nrem = 0; + for (final o in obs.where((o) => o.stage != CausalStage.absent)) { + switch (o.stage) { + case CausalStage.wake: + wake++; + case CausalStage.rem: + rem++; + case CausalStage.nrem: + nrem++; + case CausalStage.absent: + break; + } + } + // Causal wake is NOT Webster-bridged, so it runs above the offline 26 min + // (measured 62.5). The guard is against the posture-artifact blow-up the + // 1 g reference window exists to prevent (a 300 s reference gave 104 min). + expect(wake * 0.5, lessThan(90)); + expect(rem * 0.5, greaterThan(40)); // measured 78 + expect(nrem * 0.5, greaterThan(250)); // measured 374 + }); + + test('REM trigger: stable REM lands near offline REM, never far from it', + () { + bool offRem(int k) => k >= 0 && k < off.length && off[k] == SleepStage.rem; + bool near(int k) { + for (var d = -10; d <= 10; d++) { + if (offRem(k + d)) return true; + } + return false; + } + + for (final x in [120, 300]) { + final trig = obs + .where((o) => + o.stage == CausalStage.rem && o.runSec >= x) + .map(offK) + .toList(); + expect(trig, isNotEmpty, reason: 'X=$x never fired'); + final far = trig.where((k) => !near(k)).length; + // measured 0 at X=120 and 300 + expect(far / trig.length, lessThan(0.05), reason: 'X=$x false triggers'); + } + // X=120 s catches most offline REM bouts inside the bout (measured 11/16). + final bouts = <(int, int)>[]; + for (var i = 0; i < off.length;) { + if (!offRem(i)) { + i++; + continue; + } + var j = i; + while (j < off.length && offRem(j)) { + j++; + } + bouts.add((i, j)); + i = j; + } + final trig = { + for (final o in obs) + if (o.stage == CausalStage.rem && o.runSec >= 120) offK(o) + }; + var caught = 0; + for (final (s, e) in bouts) { + if (List.generate(e - s, (i) => s + i).any(trig.contains)) caught++; + } + expect(caught / bouts.length, greaterThan(0.5)); + }); + + test('the whole replay is deterministic (first 3 h, twice)', () { + final a = _replay(night, 3 * 3600); + final b = _replay(night, 3 * 3600); + for (var i = 0; i < a.length; i++) { + expect(b[i].toJson().toString(), a[i].toJson().toString()); + } + // ...and equal to the corresponding prefix of the full replay (causality). + for (var i = 0; i < a.length; i++) { + expect(obs[i].toJson().toString(), a[i].toJson().toString()); + } + }); + + test('a strap-off gap mid-night abstains with offWrist, then recovers', () { + // 4h00-4h10 off-skin (hr 0), everything else untouched. + const lo = 4 * 3600, hi = 4 * 3600 + 600; + final r = _replay(night, 5 * 3600, + hrOf: (s, bpm) => (s >= lo && s < hi) ? 0.0 : bpm); + var inside = 0; + for (final o in r) { + if (o.epochStartMs == null) continue; + final t = (o.epochStartMs! - _t0Ms) / 1000 + 30; // epoch end, s + if (t >= lo + 30 && t <= hi) { + inside++; + expect(o.stage, CausalStage.absent, reason: 'at ${t}s'); + expect(o.abstentionReason, CausalAbstention.offWrist); + } + } + expect(inside, 20, reason: '10 min = 20 epochs checked'); + final after = r.where((o) => + o.epochStartMs != null && (o.epochStartMs! - _t0Ms) / 1000 > hi + 600); + expect(after.every((o) => o.stage != CausalStage.absent), isTrue); + }); +} diff --git a/test/onehz/causal_stager_test.dart b/test/onehz/causal_stager_test.dart new file mode 100644 index 0000000..ec8662d --- /dev/null +++ b/test/onehz/causal_stager_test.dart @@ -0,0 +1,661 @@ +// CAUSAL (online) sleep stager — contract tests. +// +// What is pinned here, in the order the tests appear: +// 1. causality — nothing after `nowMs` can influence an output +// 2. determinism — same inputs, same bytes; survives JSON + an isolate hop +// 3. replacement — re-sent / overlapping / corrected windows converge on +// the state a clean feed would have produced +// 4. abstention — every reason fires, carries its reason, and never +// leaves a stage, a confidence or a run behind +// 5. warm-up — exact have/need accounting, then it stages +// 6. behaviour — on separable synthetic regimes it stages what is there +// 7. state — plain data, bounded, rejects garbage +// +// The synthetic night is in support/causal_night.dart. + +import 'dart:convert'; +import 'dart:isolate'; + +import 'package:test/test.dart'; +import 'package:openstrap_analytics/onehz.dart'; + +import 'support/causal_night.dart'; + +/// Everything an observation decides, excluding bookkeeping counters in the +/// trace (e.g. how many future samples were ignored) — those legitimately +/// differ when a caller sends more than it should. +String core(CausalStageObservation o) => jsonEncode([ + o.stage.name, + o.confidence, + o.evidenceAgeMs, + o.abstentionReason?.name, + o.runSec, + o.epochStartMs, + o.note, + o.nextState.toJson(), + ]); + +/// Full signature, trace included. +String full(CausalStageObservation o) => + jsonEncode([o.toJson(), o.nextState.toJson()]); + +/// Stream a night in `stepSec` increments from `fromSec` to `toSec`, optionally +/// re-sending `overlapSec` of history in every window. Returns every +/// observation. +List stream( + SynthNight n, + int toSec, { + int fromSec = 0, + int stepSec = 30, + int overlapSec = 0, + CausalStagerState? start, +}) { + final out = []; + var state = start; + for (var t = fromSec + stepSec; t <= toSec; t += stepSec) { + final o = CausalStager.observe( + windowOf(n, t - stepSec - overlapSec, t), state); + out.add(o); + state = o.nextState; + } + return out; +} + +void expectAbsent(CausalStageObservation o, CausalAbstention why) { + expect(o.stage, CausalStage.absent); + expect(o.abstentionReason, why); + expect(o.confidence, 0, reason: 'absent ⇒ confidence 0'); + expect(o.runSec, 0); +} + +void main() { + final night = synthNight(standardNight()); + final warmSec = CausalStagerConfig.defaults.warmupEpochs * 30; + + group('exports', () { + test('the public barrel exposes the whole surface', () { + // Compiles only if every name is exported from onehz.dart. + expect(CausalStage.values.map((s) => s.name), + containsAll(['wake', 'nrem', 'rem', 'absent'])); + expect(CausalAbstention.values, isNotEmpty); + expect(CausalStagerConfig.defaults.warmupEpochs, greaterThan(0)); + }); + }); + + group('causality', () { + test('samples after nowMs never change any output or state', () { + CausalStagerState? sPast, sFut; + var sawIgnored = false; + for (var t = 30; t <= 100 * 60; t += 30) { + final past = windowOf(night, t - 30, t); + // same call, but the window also carries 10 min of FUTURE samples + final fut = windowOf(night, t - 30, t + 600, nowSec: t); + final a = CausalStager.observe(past, sPast); + final b = CausalStager.observe(fut, sFut); + expect(core(b), core(a), reason: 'diverged at t=${t}s'); + if (((b.trace['ignoredFuture'] as num?) ?? 0) > 0) sawIgnored = true; + sPast = a.nextState; + sFut = b.nextState; + } + expect(sawIgnored, isTrue, + reason: 'the guard must actually have been exercised'); + }); + + test('rewriting the future cannot move the present', () { + final base = stream(night, 90 * 60); + // Same history, then a window whose samples after `now` are garbage. + final t = 90 * 60; + final state = base[base.length - 2].nextState; + final clean = windowOf(night, t - 30, t); + final dirty = CausalSampleWindow( + nowMs: clean.nowMs, + hr: [ + ...clean.hr, + for (var i = 1; i <= 120; i++) + HrSample(clean.nowMs + i * 1000.0, 190), // a future "wake" + ], + accel: [ + ...clean.accel, + for (var i = 1; i <= 120; i++) + AccelSample(clean.nowMs + i * 1000.0, 3, 3, 3), + ], + rr: clean.rr, + ); + expect(core(CausalStager.observe(dirty, state)), + core(CausalStager.observe(clean, state))); + }); + + test('an output at time t is identical whether or not the stream continues', + () { + final short = stream(night, 80 * 60); + final long = stream(night, 120 * 60); + for (var i = 0; i < short.length; i++) { + expect(full(long[i]), full(short[i])); + } + }); + }); + + group('determinism', () { + test('the same inputs give byte-identical outputs and state', () { + final a = stream(night, 70 * 60); + final b = stream(night, 70 * 60); + for (var i = 0; i < a.length; i++) { + expect(full(b[i]), full(a[i])); + } + }); + + test('state survives JSON (restart / persistence) without changing results', + () { + CausalStagerState? plain, viaJson; + for (var t = 30; t <= 80 * 60; t += 30) { + final w = windowOf(night, t - 30, t); + final a = CausalStager.observe(w, plain); + final b = CausalStager.observe(w, viaJson); + expect(full(b), full(a), reason: 'diverged at t=${t}s'); + plain = a.nextState; + final j = jsonDecode(jsonEncode(b.nextState.toJson())); + viaJson = CausalStagerState.fromJson(j as Map); + expect(viaJson, isNotNull); + } + }); + + test('the result does not depend on how the stream is chunked', () { + final fine = stream(night, 3600, stepSec: 30).last; + final mid = stream(night, 3600, stepSec: 300).last; + final coarse = stream(night, 3600, stepSec: 3600).last; + expect(core(mid), core(fine)); + expect(core(coarse), core(fine)); + }); + + test('state and window cross an isolate boundary and give the same answer', + () async { + final upTo = stream(night, 60 * 60); + final state = upTo[upTo.length - 2].nextState; + final w = windowOf(night, 60 * 60 - 30, 60 * 60); + final local = CausalStager.observe(w, state); + final remote = await Isolate.run(() => CausalStager.observe(w, state)); + expect(full(remote), full(local)); + }); + }); + + group('incremental replacement', () { + test('re-sending the same window changes nothing', () { + final upTo = stream(night, 50 * 60); + final prior = upTo[upTo.length - 2].nextState; + final w = windowOf(night, 50 * 60 - 30, 50 * 60); + final first = CausalStager.observe(w, prior); + final again = CausalStager.observe(w, first.nextState); + expect(core(again), core(first)); + final thrice = CausalStager.observe(w, again.nextState); + expect(core(thrice), core(first)); + }); + + test('overlapping windows converge on the same state as a clean feed', () { + final clean = stream(night, 75 * 60); + final overlap = stream(night, 75 * 60, overlapSec: 120); + for (var i = 0; i < clean.length; i++) { + expect(core(overlap[i]), core(clean[i]), reason: 'step $i'); + } + }); + + test('a corrected re-send replaces earlier samples', () { + final t = 70 * 60; + final upTo = stream(night, t - 30); + final prior = upTo.last.nextState; + // First delivery of the last 3 min is corrupt (HR 40 bpm off-spec)... + final lo = t - 180; + final good = windowOf(night, lo, t); + // ...and every beat doubled, which only a span REPLACEMENT (not a + // per-key upsert) can undo. + final bad = CausalSampleWindow( + nowMs: good.nowMs, + hr: [for (final h in good.hr) HrSample(h.tsMs, 40)], + accel: good.accel, + rr: RrSeries([ + for (var i = 0; i < good.rr.length; i++) ...[ + good.rr.tsMs[i], + good.rr.tsMs[i] + ] + ], [ + for (var i = 0; i < good.rr.length; i++) ...[ + good.rr.rrMs[i], + good.rr.rrMs[i] + ] + ]), + ); + final afterBad = CausalStager.observe(bad, prior); + // ...then the producer re-sends the correct window. + final fixed = CausalStager.observe(good, afterBad.nextState); + final clean = CausalStager.observe(good, prior); + expect(core(fixed), core(clean)); + expect(jsonEncode(fixed.nextState.toJson()), + jsonEncode(clean.nextState.toJson())); + }); + + test('samples older than the replacement horizon are ignored, not applied', + () { + final t = 80 * 60; + final upTo = stream(night, t); + final state = upTo.last.nextState; + // Re-send an old minute with different content, with a fresh `now`. + final old = windowOf(night, 20 * 60, 21 * 60, nowSec: t); + final poisoned = CausalSampleWindow( + nowMs: old.nowMs, + hr: [for (final h in old.hr) HrSample(h.tsMs, 150)], + accel: old.accel, + rr: old.rr, + ); + final o = CausalStager.observe(poisoned, state); + expect(o.trace['ignoredStale'], greaterThan(0)); + expect(jsonEncode(o.nextState.toJson()), + jsonEncode(state.toJson())); + }); + + test('duplicate seconds inside one window: the later entry wins', () { + final t = 50 * 60; + final prior = stream(night, t - 30).last.nextState; + final w = windowOf(night, t - 30, t); + final withDup = CausalSampleWindow( + nowMs: w.nowMs, + hr: [for (final h in w.hr) HrSample(h.tsMs, 150), ...w.hr], + accel: w.accel, + rr: w.rr, + ); + expect(core(CausalStager.observe(withDup, prior)), + core(CausalStager.observe(w, prior))); + }); + }); + + group('abstention', () { + // Stream clean to `t`, then hand back state + the last observation. + CausalStagerState warm(int t) => stream(night, t).last.nextState; + final t = warmSec + 20 * 60; + + test('absent window, no prior: noEvidence, with a usable state', () { + final o = CausalStager.observe( + CausalSampleWindow(nowMs: kT0Ms, hr: const [], accel: const []), + null); + expectAbsent(o, CausalAbstention.noEvidence); + expect(o.evidenceAgeMs, isNull); + expect(o.epochStartMs, isNull); + expect(jsonEncode(o.nextState.toJson()), isNotEmpty); + }); + + test('data stopped (disconnect): staleEvidence, never a carried stage', () { + final s = warm(t); + final ok = CausalStager.observe(windowOf(night, t, t + 30), s); + expect(ok.stage, isNot(CausalStage.absent), reason: 'precondition'); + final o = CausalStager.observe( + CausalSampleWindow( + nowMs: night.t0Ms + (t + 30 + 300) * 1000.0, + hr: const [], + accel: const []), + ok.nextState); + expectAbsent(o, CausalAbstention.staleEvidence); + expect(o.evidenceAgeMs, greaterThanOrEqualTo(300 * 1000)); + }); + + test('evidence just inside the freshness limit still stages', () { + final s = warm(t); + final maxAge = CausalStagerConfig.defaults.maxEvidenceAgeSec; + final o = CausalStager.observe( + CausalSampleWindow( + nowMs: night.t0Ms + (t + maxAge - 5) * 1000.0, + hr: const [], + accel: const []), + s); + expect(o.stage, isNot(CausalStage.absent)); + expect(o.evidenceAgeMs, lessThanOrEqualTo(maxAge * 1000)); + }); + + test('clock going backwards: clockRegressed, state untouched', () { + final s = warm(t); + final o = CausalStager.observe( + windowOf(night, t - 120, t - 60, nowSec: t - 300), s); + expectAbsent(o, CausalAbstention.clockRegressed); + expect(jsonEncode(o.nextState.toJson()), jsonEncode(s.toJson())); + }); + + test('hr == 0 (off-skin) for the latest epoch: offWrist', () { + final s = warm(t); + final w = windowOf(night, t, t + 60); + final off = CausalSampleWindow( + nowMs: w.nowMs, + hr: [for (final h in w.hr) HrSample(h.tsMs, 0)], + accel: w.accel, + rr: w.rr, + ); + expectAbsent(CausalStager.observe(off, s), CausalAbstention.offWrist); + }); + + test('no HR at all in the latest epoch: missingHr', () { + final s = warm(t); + final w = windowOf(night, t, t + 60, hrOn: false); + expectAbsent(CausalStager.observe(w, s), CausalAbstention.missingHr); + }); + + test('non-finite HR is missing, not a value', () { + final s = warm(t); + final w = windowOf(night, t, t + 60); + final nan = CausalSampleWindow( + nowMs: w.nowMs, + hr: [for (final h in w.hr) HrSample(h.tsMs, double.nan)], + accel: w.accel, + rr: w.rr, + ); + expectAbsent(CausalStager.observe(nan, s), CausalAbstention.missingHr); + }); + + test('no accel in the latest epoch: missingAccel', () { + final s = warm(t); + final w = windowOf(night, t, t + 60, accelOn: false); + expectAbsent(CausalStager.observe(w, s), CausalAbstention.missingAccel); + }); + + test('accel flagged invalid: missingAccel', () { + final s = warm(t); + final w = windowOf(night, t, t + 60); + final inv = CausalSampleWindow( + nowMs: w.nowMs, + hr: w.hr, + accel: [ + for (final a in w.accel) + AccelSample(a.tsMs, a.x, a.y, a.z, valid: false) + ], + rr: w.rr, + ); + expectAbsent(CausalStager.observe(inv, s), CausalAbstention.missingAccel); + }); + + test('too few HR seconds in the epoch: lowCoverage', () { + final s = warm(t); + final w = windowOf(night, t, t + 60); + final sparse = CausalSampleWindow( + nowMs: w.nowMs, + hr: [ + for (var i = 0; i < w.hr.length; i++) + if (i % 30 < 4) w.hr[i] // 4 of every 30 seconds + ], + accel: w.accel, + rr: w.rr, + ); + expectAbsent( + CausalStager.observe(sparse, s), CausalAbstention.lowCoverage); + }); + + test('too few accel seconds in the epoch: lowCoverage', () { + final s = warm(t); + final w = windowOf(night, t, t + 60); + final sparse = CausalSampleWindow( + nowMs: w.nowMs, + hr: w.hr, + accel: [ + for (var i = 0; i < w.accel.length; i++) + if (i % 30 < 4) w.accel[i] + ], + rr: w.rr, + ); + expectAbsent( + CausalStager.observe(sparse, s), CausalAbstention.lowCoverage); + }); + + test('a staged stream recovers after a gap and does not carry old labels', + () { + final s = warm(t); + // 3 minutes of nothing (strap off the charger, say), then data again + final gapEnd = t + 180; + final resumed = CausalStager.observe( + windowOf(night, gapEnd, gapEnd + 60), s); + // the first closed epoch after the gap is fine; the point is that during + // the gap itself the answer was absent + final during = CausalStager.observe( + CausalSampleWindow( + nowMs: night.t0Ms + (t + 150) * 1000.0, + hr: const [], + accel: const []), + s); + expectAbsent(during, CausalAbstention.staleEvidence); + expect(resumed.stage, isNot(CausalStage.absent)); + }); + + test('a stream with no usable HR never produces a stage', () { + CausalStagerState? state; + for (var s = 30; s <= 60 * 60; s += 30) { + final w = windowOf(night, s - 30, s); + final blind = CausalSampleWindow( + nowMs: w.nowMs, + hr: [for (final h in w.hr) HrSample(h.tsMs, double.nan)], + accel: w.accel, + rr: w.rr, + ); + final o = CausalStager.observe(blind, state); + expect(o.stage, CausalStage.absent, reason: 'second $s'); + expect(o.abstentionReason, isNotNull); + state = o.nextState; + } + }); + + test('invariants: absent <=> reason; present ⇒ confidence in (0, 0.6]', () { + for (final o in stream(night, 100 * 60)) { + if (o.stage == CausalStage.absent) { + expect(o.abstentionReason, isNotNull); + expect(o.confidence, 0); + expect(o.runSec, 0); + } else { + expect(o.abstentionReason, isNull); + expect(o.confidence, inExclusiveRange(0, 0.6 + 1e-12)); + expect(o.runSec, greaterThan(0)); + expect(o.epochStartMs, isNotNull); + expect(o.evidenceAgeMs, isNotNull); + } + } + }); + }); + + group('warm-up', () { + test('abstains with an exact have/need count, then stages', () { + final need = CausalStagerConfig.defaults.warmupEpochs; + final obs = stream(night, (need + 6) * 30); + // After k calls, k epochs are closed (the stream starts epoch-aligned). + for (var k = 1; k <= need + 6; k++) { + final o = obs[k - 1]; + if (k < need) { + expectAbsent(o, CausalAbstention.warmup); + expect(o.note, 'warmup:have=$k,need=$need'); + } else { + expect(o.stage, isNot(CausalStage.absent), reason: 'k=$k'); + expect(o.note, isNull); + } + } + }); + + test('a thin start does not count epochs that were never usable', () { + // 10 min of off-skin first: none of it may count toward warm-up. + final need = CausalStagerConfig.defaults.warmupEpochs; + CausalStagerState? s; + for (var t = 30; t <= 600; t += 30) { + final w = windowOf(night, t - 30, t); + final off = CausalSampleWindow( + nowMs: w.nowMs, + hr: [for (final h in w.hr) HrSample(h.tsMs, 0)], + accel: w.accel, + rr: w.rr, + ); + s = CausalStager.observe(off, s).nextState; + } + final o = CausalStager.observe(windowOf(night, 600, 630), s); + expectAbsent(o, CausalAbstention.warmup); + expect(o.note, 'warmup:have=1,need=$need'); + }); + }); + + group('behaviour on separable regimes', () { + late List obs; + setUpAll(() => obs = stream(night, night.seconds)); + + // Ground truth for the epoch an observation describes. + String truthOf(CausalStageObservation o) { + final sec = ((o.epochStartMs! - night.t0Ms) / 1000).round(); + // majority regime across the 30 s epoch + final c = {}; + for (var i = sec; i < sec + 30; i++) { + c[night.truth[i]] = (c[night.truth[i]] ?? 0) + 1; + } + return c.entries.reduce((a, b) => a.value >= b.value ? a : b).key; + } + + test('REM is called inside REM blocks, mostly after the first minutes', () { + final inRem = obs.where((o) => + o.stage != CausalStage.absent && truthOf(o) == 'rem').toList(); + expect(inRem.length, greaterThan(20)); + final hit = inRem.where((o) => o.stage == CausalStage.rem).length; + expect(hit / inRem.length, greaterThan(0.6)); + }); + + test('NREM time is not called wake', () { + final inN = obs.where((o) => + o.stage != CausalStage.absent && truthOf(o) == 'nrem').toList(); + final wake = inN.where((o) => o.stage == CausalStage.wake).length; + expect(wake / inN.length, lessThan(0.05)); + }); + + // The REM "wake now" rule a caller would build: REM that has persisted for + // at least the shortest credible REM bout (remEpisodeMinMin = 5 min). + bool stableRem(CausalStageObservation o) => + o.stage == CausalStage.rem && o.runSec >= remEpisodeMinMin * 60; + + // [lo, hi) minute spans of the REM blocks in standardNight(). + const remBlocks = [(35, 47), (123, 138)]; + double minOf(CausalStageObservation o) => + (o.epochStartMs! - night.t0Ms) / 60000.0; + + test('every REM block raises a stable-REM candidate inside the block, ' + 'within 8 min of onset', () { + for (final (lo, hi) in remBlocks) { + final first = obs + .where((o) => stableRem(o) && minOf(o) >= lo && minOf(o) < hi) + .map(minOf) + .toList(); + expect(first, isNotEmpty, reason: 'REM block $lo-$hi min missed'); + expect(first.first - lo, lessThanOrEqualTo(8)); + } + }); + + test('stable-REM candidates far from any REM block are rare', () { + // "far" = more than 6 min after a block ended (trailing windows keep the + // tail of REM variability in view for a few minutes — the lag the doc + // states) and not inside one. + bool far(double m) => remBlocks.every((b) => m < b.$1 || m >= b.$2 + 6); + final farEpochs = obs + .where((o) => o.stage != CausalStage.absent && far(minOf(o))) + .toList(); + final falseTrig = farEpochs.where(stableRem).length; + expect(falseTrig / farEpochs.length, lessThan(0.06)); + }); + + test('a motion/HR wake block is called wake', () { + final inW = obs.where((o) => + o.stage != CausalStage.absent && truthOf(o) == 'wake').toList(); + expect(inW, isNotEmpty); + final hit = inW.where((o) => o.stage == CausalStage.wake).length; + expect(hit / inW.length, greaterThan(0.6)); + }); + + test('runSec counts how long the stage has persisted, in whole epochs', () { + var prev = CausalStage.absent; + var prevRun = 0.0; + for (final o in obs) { + if (o.stage == CausalStage.absent) { + prev = CausalStage.absent; + prevRun = 0; + continue; + } + expect(o.runSec % 30, 0); + if (o.stage == prev) { + expect(o.runSec, prevRun + 30); + } else { + expect(o.runSec, 30); + } + prev = o.stage; + prevRun = o.runSec; + } + }); + + test('evidence age is small for a live feed', () { + for (final o in obs.where((o) => o.stage != CausalStage.absent)) { + expect(o.evidenceAgeMs, lessThanOrEqualTo(2000)); + } + }); + }); + + group('shared building blocks (extracted from cardioStager, unchanged)', () { + test('weightedAxisScore: weighted mean over measurable axes only', () { + expect(weightedAxisScore([(1.0, 0.5), (null, 0.3), (double.nan, 0.2)]), + 1.0); + expect(weightedAxisScore([(2.0, 0.5), (-1.0, 0.5)]), 0.5); + expect(weightedAxisScore([(null, 0.5)]), isNull); + expect(weightedAxisScore([(1.0, -0.5), (3.0, 0.5)]), 1.0, + reason: 'a negative weight flips the axis; the denominator is |w|'); + }); + + test('cleanRrBeatsBetween is the same gate as the centred-window gather', + () { + final accel = [ + for (var i = 0; i < 30; i++) AccelSample(i * 1000.0, 0, 0, 1) + ]; + final ts = [9000, 10000, 10000, 15000, 20000, 20000, 21000]; + final rr = [900, 910, 5000, 920, 930, 240, 940]; // 5000/240 gated + final centred = + cleanBeatsInWindowForTest(rr, ts, accel, 0, 30, halfWinMs: 5000); + final direct = cleanRrBeatsBetween(rr, ts, 10000, 20000); + expect(direct.beats, centred); + expect(direct.tsSec.first, 0, reason: 'rebased to the window start'); + }); + + test('REM weights are the shipped effect sizes', () { + expect( + [kRemWeightRk, kRemWeightSdnn, kRemWeightHrSd, kRemWeightLfhf], + [0.43, 0.32, 0.27, 0.12]); + }); + }); + + group('state', () { + test('is plain JSON data', () { + final s = stream(night, 60 * 60).last.nextState; + final j = s.toJson(); + expect(jsonDecode(jsonEncode(j)), j); + }); + + test('is bounded no matter how long the stream runs', () { + final long = synthNight([('nrem', 7 * 3600)], seed: 5); + final s = stream(long, long.seconds, stepSec: 600).last.nextState; + final j = s.toJson(); + final cfg = CausalStagerConfig.defaults; + expect((j['rows'] as List).length, lessThanOrEqualTo(cfg.historyEpochs)); + expect((j['hr'] as List).length, lessThanOrEqualTo(1000)); + expect((j['accel'] as List).length, lessThanOrEqualTo(1000)); + }); + + test('a round-trip is the identity', () { + final s = stream(night, 45 * 60).last.nextState; + final back = CausalStagerState.fromJson( + jsonDecode(jsonEncode(s.toJson())) as Map); + expect(jsonEncode(back!.toJson()), jsonEncode(s.toJson())); + }); + + test('garbage and unknown versions are rejected (null), not guessed at', () { + final j = stream(night, 45 * 60).last.nextState.toJson(); + expect(CausalStagerState.fromJson({}), isNull); + expect(CausalStagerState.fromJson({...j, 'v': 999}), isNull); + expect(CausalStagerState.fromJson({...j, 'rows': 'nope'}), isNull); + expect(CausalStagerState.fromJson({...j, 'hr': [1, 2, 3]}), isNull); + }); + + test('an old state with a different config keeps its own config', () { + const cfg = CausalStagerConfig(warmupEpochs: 10); + final s0 = CausalStagerState.initial(cfg); + final o = CausalStager.observe(windowOf(night, 0, 30), s0); + expect(o.note, 'warmup:have=1,need=10'); + }); + }); +} diff --git a/test/onehz/cvhr_nan_test.dart b/test/onehz/cvhr_nan_test.dart new file mode 100644 index 0000000..f120b1a --- /dev/null +++ b/test/onehz/cvhr_nan_test.dart @@ -0,0 +1,98 @@ +// cvhrApneaScreen used to loop forever on a non-finite NN value: a NaN +// excursion made both `exc[i] <= prom` and `exc[i] > prom` false in +// _scoreSegment, so the scan index never advanced. +// +// A busy loop on the main isolate cannot be interrupted by the test timeout, +// so each call runs in a spawned isolate that is killed after 5 s. A hang +// therefore shows up as a clean test failure, not a stuck test run. +// +// Assertions, and why they hold: +// * The baseline series is constant NN (800 ms), so it has no dip at all. +// Every finite excursion is exactly 0, so no finite stretch can start a +// cycle. Non-finite samples must not start one either (the intended +// behaviour: a non-finite excursion is never above the prominence +// threshold). So the cycle count must be 0, which is also the count for +// the same series with the bad beats removed. +// * The series are 400 beats (~320 s), long enough to pass the >=60 beat and +// >=60 s gates and to be one gap-free segment, so the result is present. +import 'dart:isolate'; + +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +const _n = 400; +const _nn = 800.0; + +/// Entry point for the spawned isolate: msg = [SendPort, nn, tMs]. +void _entry(List msg) { + final port = msg[0] as SendPort; + final nn = (msg[1] as List).cast(); + final t = (msg[2] as List).cast(); + final m = cvhrApneaScreen(nn, t, artifactFraction: 0.0); + port.send(m.value?.cycleCount ?? -1); +} + +/// Cycle count, -1 when the metric is absent. Fails the test on a hang. +Future _count(List nn) async { + final t = [for (var i = 0; i < nn.length; i++) i * _nn]; + final rp = ReceivePort(); + final iso = await Isolate.spawn(_entry, [rp.sendPort, nn, t]); + try { + final r = await rp.first.timeout(const Duration(seconds: 5)); + return r as int; + } finally { + iso.kill(priority: Isolate.immediate); + rp.close(); + } +} + +List _flat() => List.filled(_n, _nn); + +List _with(Map bad) { + final x = _flat(); + bad.forEach((i, v) => x[i] = v); + return x; +} + +void main() { + const to = Timeout(Duration(seconds: 15)); + const nan = double.nan; + const inf = double.infinity; + + test('control: flat series has no cycles', () async { + expect(await _count(_flat()), 0); + }, timeout: to); + + test('single NaN mid-series returns, no cycles', () async { + expect(await _count(_with({200: nan})), 0); + }, timeout: to); + + test('NaN run mid-series returns, no cycles', () async { + expect( + await _count(_with({for (var i = 150; i < 190; i++) i: nan})), 0); + }, timeout: to); + + test('NaN at the start returns, no cycles', () async { + expect(await _count(_with({0: nan, 1: nan, 2: nan})), 0); + }, timeout: to); + + test('NaN at the end returns, no cycles', () async { + expect(await _count(_with({_n - 1: nan, _n - 2: nan})), 0); + }, timeout: to); + + test('all-NaN segment returns, no cycles', () async { + expect(await _count(List.filled(_n, nan)), 0); + }, timeout: to); + + test('+Infinity returns, no cycles', () async { + expect(await _count(_with({200: inf})), 0); + }, timeout: to); + + test('-Infinity returns, no cycles', () async { + expect(await _count(_with({200: -inf})), 0); + }, timeout: to); + + test('mixed NaN and +-Infinity returns, no cycles', () async { + expect(await _count(_with({50: nan, 120: inf, 260: -inf, 330: nan})), 0); + }, timeout: to); +} diff --git a/test/onehz/cvhr_stream_test.dart b/test/onehz/cvhr_stream_test.dart new file mode 100644 index 0000000..039f810 --- /dev/null +++ b/test/onehz/cvhr_stream_test.dart @@ -0,0 +1,599 @@ +// RED (round 22): exact streaming CVHR. +// +// CvhrStream.snapshot(...) must equal cvhrApneaScreen(folded ++ tail, ...) BIT +// FOR BIT at every fold: == on doubles, same NaN-ness and sign of zero, same +// note, confidence, quartiles. No tolerance. The batch function is the +// reference and does not change in this round. +// +// Locality facts the tests lean on (checked against cvhr_apnea.dart): +// * 1 Hz resample at grid g needs the beat bracketing g. A beat AT g that is +// the last beat clamps to y.last, but once a later beat exists the same +// point is y[j] + (y[j+1]-y[j]) * 1.0, which can differ in the last bit, +// so the on-the-grid tests below feed beats exactly on whole seconds. +// * Savitzky-Golay reads +-7 grid points, the median +-65 s (clipped at the +// segment ends); P75, prominence and run detection read the whole segment. +import 'dart:convert'; +import 'dart:isolate'; +import 'dart:math' as math; + +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +// ───────────────────────── synthetic nights ───────────────────────────────── + +class Night { + final List nn = [], times = []; +} + +/// A seeded night of [hours]: ~1000 ms NN with breathing and noise. With +/// [apnea], 30-60 s cyclic bradycardia (+80 ms half-sine bumps, 20 s wide, +/// every 45 s; the period wobbles per stretch) over alternating stretches of +/// the night. [jumps] are (atSec, lengthSec) holes in the recording clock. +/// [fast] = (fromSec, toSec) runs 500 ms NN there (>= 60 beats in < 60 s). +Night synth(int seed, double hours, + {bool apnea = true, + List<(double, double)> jumps = const [], + (double, double)? fast}) { + final rnd = math.Random(seed); + final n = Night(); + final endS = hours * 3600; + var s = 0.0; // recording clock, seconds + var t = 1.7e12; // beat clock, ms + final phase = rnd.nextDouble() * 6.28; + final jl = [...jumps]..sort((a, b) => a.$1.compareTo(b.$1)); + var ji = 0; + while (s < endS) { + while (ji < jl.length && s >= jl[ji].$1) { + t += jl[ji].$2 * 1000; + s += jl[ji].$2; + ji++; + } + var v = 1000.0 + + 25 * math.sin(2 * math.pi * 0.24 * s + phase) + + 30 * math.sin(s / 900 + seed) + + 6 * (rnd.nextDouble() - .5) * 2; + if (apnea) { + // Stretches of 12 min with events, 8 min without. + final stretch = (s % 1200); + if (stretch < 720) { + final period = 45.0 + 5 * (s ~/ 1200 % 3); + final u = s % period; + if (u < 20) v += 80 * math.sin(math.pi * u / 20); + } + } + if (fast != null && s >= fast.$1 && s < fast.$2) v = 500; + v = v.roundToDouble(); + t += v; + s += v / 1000; + n.nn.add(v); + n.times.add(t); + } + return n; +} + +// ───────────────────────── bit-for-bit comparison ─────────────────────────── + +bool sameD(double? a, double? b) { + if (a == null || b == null) return a == null && b == null; + if (a.isNaN || b.isNaN) return a.isNaN && b.isNaN; + return a == b && a.isNegative == b.isNegative; +} + +/// Empty string when equal, else the first difference. (A String so it can +/// cross an isolate boundary.) +String diffD(double? a, double? b, String what) => + sameD(a, b) ? '' : '$what: $a vs $b; '; + +String diffList(List? a, List? b, String what) { + if (a == null || b == null) return a == null && b == null ? '' : '$what null; '; + if (a.length != b.length) return '$what length; '; + final sb = StringBuffer(); + for (var i = 0; i < a.length; i++) { + sb.write(diffD(a[i], b[i], '$what[$i]')); + } + return sb.toString(); +} + +String diffCvhr(Metric a, Metric b) { + final sb = StringBuffer(); + if ((a.value == null) != (b.value == null)) return 'value presence; '; + sb.write(diffD(a.confidence, b.confidence, 'confidence')); + if (a.tier != b.tier) sb.write('tier; '); + if ('${a.inputs_used}' != '${b.inputs_used}') sb.write('inputs; '); + if (a.note != b.note) sb.write('note "${a.note}" vs "${b.note}"; '); + if ((a.drivers == null) != (b.drivers == null)) sb.write('drivers; '); + final x = a.value, y = b.value; + if (x == null || y == null) return sb.toString(); + if (x.cycleCount != y.cycleCount) { + sb.write('cycleCount ${x.cycleCount} vs ${y.cycleCount}; '); + } + sb.write(diffD(x.cvhrPerHour, y.cvhrPerHour, 'cvhrPerHour')); + sb.write(diffD(x.analyzedHours, y.analyzedHours, 'analyzedHours')); + sb.write(diffD(x.meanDepthMs, y.meanDepthMs, 'meanDepthMs')); + sb.write(diffD(x.meanWidthSec, y.meanWidthSec, 'meanWidthSec')); + sb.write(diffList(x.depthQuartilesMs, y.depthQuartilesMs, 'depthQ')); + sb.write(diffList(x.widthQuartilesSec, y.widthQuartilesSec, 'widthQ')); + return sb.toString(); +} + +void expectSame(Metric a, Metric b, String at) => + expect(diffCvhr(a, b), '', reason: at); + +// ───────────────────────── drivers ────────────────────────────────────────── + +/// Chunk boundaries (beat counts) with random sizes in [lo, hi], starting small +/// so the first folds sit below the 60-beat gate. +List chunkEnds(int n, math.Random rnd, int lo, int hi, {int first = 20}) { + final ends = []; + var at = first; + while (at < n) { + ends.add(at); + at += lo + rnd.nextInt(hi - lo + 1); + } + ends.add(n); + return ends; +} + +Metric batch(List nn, List t, int end, double af, + {double minW = 10, + double maxW = 120, + double ratio = 0.7, + double env = 130, + double maxA = 0.30, + double gap = 30}) => + cvhrApneaScreen(nn.sublist(0, end), t.sublist(0, end), + artifactFraction: af, + minWidthSec: minW, + maxWidthSec: maxW, + depthWidthRatioMin: ratio, + envWindowSec: env, + maxArtifact: maxA, + maxGapSec: gap); + +/// Fold [nn]/[t] by [ends]; after each fold compare an empty-tail and a +/// [tailBeats]-tail snapshot against the batch. Returns the first mismatch +/// ('' when none) so it can run inside an isolate. +String parity(List nn, List t, List ends, double af, + {int tailBeats = 100, + double minW = 10, + double maxW = 120, + double ratio = 0.7, + double env = 130, + double maxA = 0.30, + double gap = 30}) { + final s = CvhrStream( + minWidthSec: minW, + maxWidthSec: maxW, + depthWidthRatioMin: ratio, + envWindowSec: env, + maxGapSec: gap); + var from = 0; + for (final end in ends) { + s.fold(nn.sublist(from, end), t.sublist(from, end)); + from = end; + var d = diffCvhr( + s.snapshot(artifactFraction: af, maxArtifact: maxA), + batch(nn, t, end, af, + minW: minW, + maxW: maxW, + ratio: ratio, + env: env, + maxA: maxA, + gap: gap)); + if (d.isNotEmpty) return 'prefix $end: $d'; + final tEnd = math.min(nn.length, end + tailBeats); + d = diffCvhr( + s.snapshot( + tailNnMs: nn.sublist(end, tEnd), + tailNnTimesMs: t.sublist(end, tEnd), + artifactFraction: af, + maxArtifact: maxA), + batch(nn, t, tEnd, af, + minW: minW, + maxW: maxW, + ratio: ratio, + env: env, + maxA: maxA, + gap: gap)); + if (d.isNotEmpty) return 'prefix $end + tail ${tEnd - end}: $d'; + } + return ''; +} + +void expectParity(Night n, List ends, double af, + {int tailBeats = 100, + double minW = 10, + double maxW = 120, + double ratio = 0.7, + double env = 130, + double maxA = 0.30, + double gap = 30}) => + expect( + parity(n.nn, n.times, ends, af, + tailBeats: tailBeats, + minW: minW, + maxW: maxW, + ratio: ratio, + env: env, + maxA: maxA, + gap: gap), + ''); + +/// Isolate entry: msg = [SendPort, nn, t, ends, af]. A hang cannot be +/// interrupted by the test timeout, so the run is killed after 20 s. +void _entry(List msg) { + final port = msg[0] as SendPort; + final nn = (msg[1] as List).cast(); + final t = (msg[2] as List).cast(); + final ends = (msg[3] as List).cast(); + port.send(parity(nn, t, ends, msg[4] as double, tailBeats: 60)); +} + +Future parityIsolated( + List nn, List t, List ends, double af) async { + final rp = ReceivePort(); + final iso = + await Isolate.spawn(_entry, [rp.sendPort, nn, t, ends, af]); + try { + return await rp.first.timeout(const Duration(seconds: 20)) as String; + } finally { + iso.kill(priority: Isolate.immediate); + rp.close(); + } +} + +Object? wire(Map j) => jsonDecode(jsonEncode(j)); + +void expectMalformedRejected( + Map good, Object? Function(Map) revive) { + for (final k in good.keys) { + final missing = Map.from(good)..remove(k); + expect(() => revive(missing), throwsFormatException, reason: 'missing $k'); + final wrong = Map.from(good)..[k] = {'bogus': true}; + expect(() => revive(wrong), throwsFormatException, reason: 'wrong $k'); + } + expect(() => revive({}), throwsFormatException); + expect(() => revive({...good, 'type': 'NotAStream'}), throwsFormatException); + expect(() => revive({...good, 'version': 9999}), throwsFormatException); + expect(() => revive({...good, 'version': 'one'}), throwsFormatException); +} + +// Holes: two > 30 s gaps, a 45 s island (< 60 beats), a 40 s island of 500 ms +// beats (>= 60 beats, span < 60 s), then a long tail segment. +const _gappy = <(double, double)>[ + (2400, 90), + (3000, 60), + (3045, 60), + (5000, 45), + (5040, 70), + (7000, 31), +]; + +void main() { + group('guards: the generator really produces CVHR cycles', () { + test('batch finds cycles on the apnea night and few on the plain one', () { + final a = synth(1, 3); + final r = batch(a.nn, a.times, a.nn.length, 0.02).value!; + expect(r.cycleCount, greaterThan(20)); + expect(r.depthQuartilesMs, isNotNull); + final p = synth(1, 3, apnea: false); + final q = batch(p.nn, p.times, p.nn.length, 0.02).value!; + expect(q.cycleCount, lessThan(r.cycleCount ~/ 4)); + }); + + test('gappy night: several segments, cycles found, short islands skipped', + () { + final g = synth(2, 3, jumps: _gappy, fast: (5000, 5040)); + final r = batch(g.nn, g.times, g.nn.length, 0.02).value!; + expect(r.cycleCount, greaterThan(10)); + final whole = synth(2, 3, fast: (5000, 5040)); + expect(r.analyzedHours, + lessThan(batch(whole.nn, whole.times, whole.nn.length, 0.02) + .value! + .analyzedHours)); + }); + }); + + group('CvhrStream with nothing (or almost nothing) folded', () { + Metric viaTail(List nn, List t, int folded, + int end, double af) { + final s = CvhrStream(); + if (folded > 0) s.fold(nn.sublist(0, folded), t.sublist(0, folded)); + return s.snapshot( + tailNnMs: nn.sublist(folded, end), + tailNnTimesMs: t.sublist(folded, end), + artifactFraction: af); + } + + void expectEmptyPrefix(Night n, List ks, {int folded = 0}) { + for (final k in ks) { + final end = math.min(k, n.nn.length); + if (end <= folded) continue; + expectSame(viaTail(n.nn, n.times, folded, end, 0.02), + batch(n.nn, n.times, end, 0.02), 'folded $folded tail ${end - folded}'); + } + } + + test('fresh stream, tail across the 60-beat gate', () { + final n = synth(81, 1); + expectEmptyPrefix(n, [1, 2, 3, 30, 58, 59, 60, 61, 62, 100, 400]); + }); + + test('fresh stream, tail is a full 2 h night', () { + final n = synth(82, 2); + expectEmptyPrefix(n, [n.nn.length]); + }); + + test('fresh stream, tail with gaps > maxGapSec and short islands', () { + final n = synth(83, 3, jumps: _gappy, fast: (5000, 5040)); + expectEmptyPrefix(n, [200, 1500, 3000, 6000, n.nn.length]); + // gaps right at the tail edge + final cuts = []; + for (var i = 1; i < n.times.length; i++) { + if (n.times[i] - n.times[i - 1] > 30000) cuts.addAll([i - 1, i, i + 1]); + } + expectEmptyPrefix(n, cuts.where((c) => c > 0).toList()); + }); + + test('a few folded beats (below any segment) plus a long tail', () { + final n = synth(84, 2, jumps: const [(2400, 90)]); + for (final f in [1, 5, 30, 59, 60]) { + expectEmptyPrefix(n, [f + 1, f + 30, f + 61, n.nn.length], folded: f); + } + }); + }); + + group('CvhrStream parity', () { + test('no gaps, apnea cycles present (3 h), random chunks', () { + final n = synth(3, 3); + expectParity(n, chunkEnds(n.nn.length, math.Random(31), 300, 1400), 0.02); + }); + + test('no gaps, cycles absent (2 h)', () { + final n = synth(4, 2, apnea: false); + expectParity(n, chunkEnds(n.nn.length, math.Random(41), 300, 1400), 0.02); + }); + + test('several >30 s gaps and short islands (3 h)', () { + final n = synth(5, 3, jumps: _gappy, fast: (5000, 5040)); + final ends = chunkEnds(n.nn.length, math.Random(51), 150, 900); + expectParity(n, ends, 0.02); + }); + + test('folds that end exactly at, just before and just after each gap', () { + final n = synth(6, 1.5, jumps: const [(1500, 40), (3000, 35)]); + final ends = {n.nn.length}; + for (var i = 1; i < n.times.length; i++) { + if (n.times[i] - n.times[i - 1] > 30000) { + for (final d in [-2, -1, 0, 1, 2]) { + ends.add(i + d); + } + } + } + final sorted = ends.where((e) => e > 20).toList()..sort(); + expect(sorted.length, greaterThan(6)); + expectParity(n, sorted, 0.02, tailBeats: 30); + }); + + test('gap of exactly maxGapSec stays one segment, +1 ms splits it', () { + for (final extra in [0.0, 1.0]) { + final nn = [], t = []; + var clock = 0.0; + final rnd = math.Random(61); + for (var i = 0; i < 500; i++) { + nn.add(1000 + 40 * math.sin(i / 7) + 10 * rnd.nextDouble()); + clock += 1000; + if (i == 250) clock += 29000 + extra; // 30 000 (+1) ms between beats + t.add(clock); + } + final ends = [30, 200, 250, 251, 252, 400, 500]; + expect(parity(nn, t, ends, 0.02), '', reason: 'extra=$extra'); + expect(parity(nn, t, ends, 0.02, gap: 10), '', reason: 'gap 10'); + } + }); + + test('beats exactly on whole seconds, beat by beat (resample edge)', () { + // nn is independent of the times, so every grid point lands exactly on a + // beat; duplicate times (non-decreasing) are mixed in. + final rnd = math.Random(71); + final nn = [], t = []; + var clock = 5000.0; + for (var i = 0; i < 420; i++) { + nn.add(700 + 300 * rnd.nextDouble() + 90 * math.sin(i / 9)); + if (i % 37 != 36) clock += 1000; + t.add(clock); + } + expect(parity(nn, t, [for (var e = 61; e <= 420; e++) e], 0.02, + tailBeats: 3), + ''); + }); + + test('single-beat folds across 70 min of a segment (median window edges)', + () { + final n = synth(8, 1.2); + final ends = [for (var e = 40; e <= 400; e += 1) e] + + [for (var e = 400; e <= n.nn.length; e += 97) e]; + expectParity(n, ends, 0.02, tailBeats: 0); + }); + + test('artifact gate on / off and near the threshold', () { + final n = synth(9, 1.5); + final ends = chunkEnds(n.nn.length, math.Random(91), 400, 1200); + for (final af in [0.0, 0.1, 0.30, 0.3000001, 0.5]) { + expectParity(n, ends, af); + } + expectParity(n, ends, 0.2, maxA: 0.1); + expectParity(n, ends, 0.2, maxA: 0.25); + }); + + test('non-default options', () { + final n = synth(10, 1.5); + final ends = chunkEnds(n.nn.length, math.Random(101), 300, 1000); + expectParity(n, ends, 0.02, minW: 15, maxW: 90); + expectParity(n, ends, 0.02, ratio: 0.4); + expectParity(n, ends, 0.02, env: 90); // half = 45 + expectParity(n, ends, 0.02, env: 131); // half = round(65.5) = 66 + expectParity(n, ends, 0.02, gap: 5); + }); + + test('absent paths: fresh stream, <60 beats, degenerate times', () { + final s = CvhrStream(); + expectSame(s.snapshot(artifactFraction: 0), + cvhrApneaScreen(const [], const [], artifactFraction: 0), 'fresh'); + final n = synth(11, 0.3); + s.fold(n.nn.sublist(0, 40), n.times.sublist(0, 40)); + expectSame(s.snapshot(artifactFraction: 0), + batch(n.nn, n.times, 40, 0), '40 beats'); + expectSame( + s.snapshot( + tailNnMs: n.nn.sublist(40, 59), + tailNnTimesMs: n.times.sublist(40, 59), + artifactFraction: 0), + batch(n.nn, n.times, 59, 0), + '59 beats'); + final flat = CvhrStream() + ..fold(List.filled(80, 800.0), List.filled(80, 5000.0)); + expectSame( + flat.snapshot(artifactFraction: 0), + cvhrApneaScreen(List.filled(80, 800.0), List.filled(80, 5000.0), + artifactFraction: 0), + 'degenerate times'); + // 60 beats but a 59 s span, then 60 beats over 80 s. + for (final step in [1000.0, 1300.0]) { + final nn = List.filled(60, 900.0); + final t = [for (var i = 0; i < 60; i++) 1000 + i * step]; + final f = CvhrStream()..fold(nn, t); + expectSame(f.snapshot(artifactFraction: 0), + cvhrApneaScreen(nn, t, artifactFraction: 0), 'span step $step'); + } + }); + + test('tail that starts a new segment, or only extends the open one', () { + final n = synth(12, 1, jumps: const [(1800, 60)]); + var cut = 0; + for (var i = 1; i < n.times.length; i++) { + if (n.times[i] - n.times[i - 1] > 30000) cut = i; + } + final s = CvhrStream()..fold(n.nn.sublist(0, cut), n.times.sublist(0, cut)); + for (final k in [0, 1, 5, 59, 60, 61, 300]) { + final e = math.min(n.nn.length, cut + k); + expectSame( + s.snapshot( + tailNnMs: n.nn.sublist(cut, e), + tailNnTimesMs: n.times.sublist(cut, e), + artifactFraction: 0.02), + batch(n.nn, n.times, e, 0.02), + 'tail $k after the gap'); + } + }); + + test('NaN / +-inf NN: no hang, matches batch (with cycles in the night)', + () async { + final n = synth(13, 1.5); + final nn = [...n.nn]; + nn[300] = double.nan; + for (var i = 900; i < 940; i++) { + nn[i] = double.nan; + } + nn[1500] = double.infinity; + nn[2000] = double.negativeInfinity; + nn[0] = double.nan; + nn[nn.length - 1] = double.nan; + final ends = chunkEnds(nn.length, math.Random(131), 150, 700); + expect(await parityIsolated(nn, n.times, ends, 0.02), ''); + final all = List.filled(500, double.nan); + final t = [for (var i = 0; i < 500; i++) i * 800.0]; + expect( + await parityIsolated(all, t, [100, 300, 500], 0.0), ''); + }, timeout: const Timeout(Duration(seconds: 60))); + + test('8 h night with gaps: parity at checkpoints', () { + final n = synth(14, 8, jumps: _gappy + const [(15000, 45), (22000, 100)], + fast: (5000, 5040)); + expect(n.nn.length, greaterThan(25000)); + final ends = chunkEnds(n.nn.length, math.Random(141), 2500, 4000); + expectParity(n, ends, 0.02); + }, timeout: const Timeout(Duration(seconds: 60))); + }); + + group('CvhrStream state', () { + test('snapshot never mutates: different tails, then folding, still exact', + () { + final n = synth(15, 1.5, jumps: const [(2000, 50)]); + final len = n.nn.length; + final s = CvhrStream()..fold(n.nn.sublist(0, 2000), n.times.sublist(0, 2000)); + final json1 = jsonEncode(s.toJson()); + final base = s.snapshot(artifactFraction: 0.02); + s.snapshot( + tailNnMs: n.nn.sublist(2000, 2400), + tailNnTimesMs: n.times.sublist(2000, 2400), + artifactFraction: 0.02); + s.snapshot( + tailNnMs: [800, 800], + tailNnTimesMs: [n.times[1999] + 90000, n.times[1999] + 90800], + artifactFraction: 0.5); + expect(jsonEncode(s.toJson()), json1, reason: 'toJson unchanged'); + expectSame(s.snapshot(artifactFraction: 0.02), base, 'repeat snapshot'); + expectSame(base, batch(n.nn, n.times, 2000, 0.02), 'base'); + s.fold(n.nn.sublist(2000, len), n.times.sublist(2000, len)); + expectSame(s.snapshot(artifactFraction: 0.02), + batch(n.nn, n.times, len, 0.02), 'after tail snapshots'); + }); + + test('JSON round trip mid-night resumes bit-identically', () { + final n = synth(16, 2, jumps: _gappy, fast: (5000, 5040)); + final len = n.nn.length; + // cut inside the open segment, right at a gap, and in a closed island. + var gapAt = 0; + for (var i = 1; i < n.times.length; i++) { + if (n.times[i] - n.times[i - 1] > 30000) { + gapAt = i; + break; + } + } + final cuts = ({30, gapAt, gapAt + 1, len ~/ 3, (len * 2) ~/ 3, len}.toList() + ..sort()); + var s = CvhrStream(); + var from = 0; + for (final end in cuts) { + s.fold(n.nn.sublist(from, end), n.times.sublist(from, end)); + from = end; + if (end != len) { + final text = jsonEncode(s.toJson()); + s = CvhrStream.fromJson(jsonDecode(text) as Map); + expect(jsonEncode(s.toJson()), text, reason: 'lossless at $end'); + } + expectSame(s.snapshot(artifactFraction: 0.02), + batch(n.nn, n.times, end, 0.02), 'resumed prefix $end'); + } + // Options survive a round trip. + final f = CvhrStream(envWindowSec: 90, maxGapSec: 12, minWidthSec: 14); + final f2 = CvhrStream.fromJson(wire(f.toJson()) as Map); + f2.fold(n.nn.sublist(0, 1500), n.times.sublist(0, 1500)); + expectSame( + f2.snapshot(artifactFraction: 0.02), + batch(n.nn, n.times, 1500, 0.02, env: 90, gap: 12, minW: 14), + 'options survive round trip'); + }); + + test('malformed JSON is a FormatException', () { + final n = synth(17, 1, jumps: const [(1200, 40)]); + final s = CvhrStream()..fold(n.nn, n.times); + final good = wire(s.toJson()) as Map; + expect(good['type'], 'CvhrStream'); + expect(good['version'], 1); + expectMalformedRejected(good, (j) => CvhrStream.fromJson(j)); + // A fresh stream's checkpoint is rejected the same way. + final fresh = wire(CvhrStream().toJson()) as Map; + expectMalformedRejected(fresh, (j) => CvhrStream.fromJson(j)); + }); + + test('non-monotone fold is a StateError', () { + final s = CvhrStream()..fold([800, 810, 820], [1000, 1810, 2630]); + expect(() => s.fold([800], [2000]), throwsStateError); + expect(() => CvhrStream().fold([800, 800], [5000, 4000]), throwsStateError); + final eq = CvhrStream()..fold([800], [1000]); + expect(() => eq.fold([800], [1000]), returnsNormally); + }); + }); +} diff --git a/test/onehz/epoch_rows_anchor_test.dart b/test/onehz/epoch_rows_anchor_test.dart new file mode 100644 index 0000000..ef5c0fd --- /dev/null +++ b/test/onehz/epoch_rows_anchor_test.dart @@ -0,0 +1,352 @@ +// Combined bump, grid option 2 (owner decision 2026-10-23): onset-anchored +// staging plus a phase-keyed epoch-row memo. RED until the implementation lands. +// Contract: edge.research/combined-bump/option2-brief.md. +// +// New API under test (everything else already exists): +// +// EpochRows({CardioRowCache? rowCache, int? continuingAnchorMs, +// int? anchorMs}) +// anchorMs: the first run's grid passes through this absolute time, rows are +// anchorMs + k * kEpochMs. No row starts before anchorMs when it lies after +// the first sample (earlier samples only serve as feature context); when it +// lies before the first sample the first row is the epoch holding it. A run +// that follows a data gap longer than the usable-run gap still re-anchors at +// its first sample (this is what the pre-bump stager did; see +// test/onehz/support/pre_bump_stager.dart). +// int? EpochRows.phaseMs anchor mod kEpochMs of the first run's grid +// (null while no anchor/sample is known). +// toJson()['phase'] the same value, carried in the checkpoint; +// fromJson refuses a phase its rows contradict. +// +// EpochRowsMemo({int maxPhases = 2, CardioRowCache? rowCache}) +// EpochRows rowsFor(int anchorMs, {hr, accel, rrMs, rrTsMs}) +// Rows for the phase anchorMs % kEpochMs. The lists are the CUMULATIVE +// input from the start of the data (append-only between calls). A held +// phase folds only the samples it has not consumed; an unheld phase is +// built from scratch from the lists (exact recompute) and the least +// recently used phase beyond maxPhases is dropped. +// List phases held phases, least recently used first +// int phaseBuilds phases built from scratch (incl. rebuilds) +// int samplesFolded samples handed to EpochRows.fold, hr + accel + rr +// toJson() / EpochRowsMemo.fromJson(map) checkpoint; carries each held +// phase's rows and how many cumulative samples it consumed. A restored +// memo reuses a phase only when it is asked for the same phase. +import 'dart:convert'; + +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +import '../support/property.dart'; +import 'support/epoch_input.dart'; + +EpochRows _batch(EpochInput s, int anchorMs) { + final er = EpochRows(anchorMs: anchorMs); + s.foldInto(er, 0, s.seconds); + return er; +} + +EpochRows _chunked(EpochInput s, int anchorMs, List cuts, + {bool restart = false}) { + var er = EpochRows(anchorMs: anchorMs); + var at = 0; + for (final c in [...cuts, s.seconds]) { + s.foldInto(er, at, c); + at = c; + if (restart) { + er = EpochRows.fromJson( + jsonDecode(jsonEncode(er.toJson())) as Map); + } + } + return er; +} + +EpochRows _ask(EpochRowsMemo m, int anchorMs, EpochInput upTo) => + m.rowsFor(anchorMs, + hr: upTo.hr, accel: upTo.accel, rrMs: upTo.rrMs, rrTsMs: upTo.rrTs); + +void main() { + final t0 = kAlignedT0Ms + 13000; // phase 13000 + final night = EpochInput.make(nightSegs(), t0Ms: t0, seed: 41); + + group('EpochRows(anchorMs:) puts rows on anchorMs + k * 30 s', () { + for (final (name, a) in [ + ('before the first sample', t0 - 40000), + ('at the first sample', t0), + ('a few seconds in', t0 + 7000), + ('777 s in (a detected onset)', t0 + 777000), + ]) { + test('anchor $name', () { + final er = _batch(night, a); + final rows = er.allRows(); + expect(rows, isNotEmpty); + expect(er.phaseMs, a % kEpochMs); + for (final r in rows) { + expect((r.startMs - a) % kEpochMs, 0, reason: 'start ${r.startMs}'); + } + for (var i = 1; i < rows.length; i++) { + expect(rows[i].startMs - rows[i - 1].startMs, kEpochMs); + } + if (a >= t0) { + expect(rows.first.startMs, a, reason: 'no row before the anchor'); + } else { + expect(rows.first.startMs, lessThanOrEqualTo(t0)); + expect(rows.first.startMs + kEpochMs, greaterThan(t0), + reason: 'the epoch holding the first sample opens the grid'); + } + final lastEnd = t0 + night.seconds * 1000; + expect(rows.last.startMs + kEpochMs, + inInclusiveRange(lastEnd, lastEnd + kEpochMs)); + }); + } + + test('a different phase gives different row starts over the same samples', + () { + final a = _batch(night, t0 + 7000).allRows(); + final b = _batch(night, t0 + 19000).allRows(); + expect((a[3].startMs - b[3].startMs) % kEpochMs, isNot(0)); + }); + + forAll('any chunking, with JSON restarts (even before the first sample), ' + 'equals one fold on the same anchor', G.intIn(1, 100000), (seed) { + final r = Rng(seed); + final a = t0 + [0, 7000, 19000, 777000, -40000][r.nextInt(5)]; + final want = _batch(night, a).allRows(); + final cuts = randomCuts(r, night.seconds); + final got = _chunked(night, a, cuts, restart: r.nextBool()); + expectSameRows(got.allRows(), want, 'anchor $a cuts $cuts'); + }, cases: 10, budget: const Duration(seconds: 120)); + + test('an anchored checkpoint taken before any sample keeps the anchor', () { + final a = t0 + 777000; + final empty = EpochRows(anchorMs: a); + final back = EpochRows.fromJson( + jsonDecode(jsonEncode(empty.toJson())) as Map); + night.foldInto(back, 0, night.seconds); + expectSameRows(back.allRows(), _batch(night, a).allRows(), + 'resumed from an empty anchored checkpoint'); + }); + + test('resumed mid-night from a checkpoint = batch on that anchor', () { + final a = t0 + 7000; + final er = EpochRows(anchorMs: a); + night.foldInto(er, 0, 1700); + final back = EpochRows.fromJson( + jsonDecode(jsonEncode(er.toJson())) as Map); + night.foldInto(back, 1700, night.seconds); + expectSameRows(back.allRows(), _batch(night, a).allRows(), 'resumed'); + }); + + test('after a gap longer than the usable-run gap the next sample re-anchors ' + '(pre-bump behaviour): the onset grid is NOT kept across the gap', () { + // 100 s hole at +1300..+1400 s. Anchor +7 s; the second run starts at + // +1400 s, 1393 s = 13 s (mod 30) past the onset grid. + final s = EpochInput.make(nightSegs(), + t0Ms: t0, gaps: [(1300, 1400)], seed: 43); + final a = t0 + 7000; + final er = _batch(s, a); + final starts = er.allRows().map((r) => r.startMs).toList(); + for (final st in starts) { + final rel = st - (t0 + 1400000); + if (st < t0 + 1300000) { + expect((st - a) % kEpochMs, 0, reason: 'first run on the onset grid'); + } else { + expect(rel % kEpochMs, 0, reason: 'second run on its own grid: $st'); + } + } + expect(starts.contains(t0 + 1400000), isTrue); + expect(starts.contains(a + 1400000 ~/ 30000 * 30000 + 30000), isFalse, + reason: 'the old phase is gone after the gap'); + }); + + test('a hole the carry bridges keeps the onset grid', () { + final s = EpochInput.make(nightSegs(), + t0Ms: t0, gaps: [(1000, 1040)], seed: 45); + final a = t0 + 7000; + for (final r in _batch(s, a).allRows()) { + expect((r.startMs - a) % kEpochMs, 0, reason: 'start ${r.startMs}'); + } + }); + + test('the checkpoint carries the phase, and refuses one its rows contradict', + () { + final a = t0 + 7000; + final er = EpochRows(anchorMs: a); + night.foldInto(er, 0, 1200); + expect(er.phaseMs, a % kEpochMs); + final good = jsonDecode(jsonEncode(er.toJson())) as Map; + expect(good['phase'], a % kEpochMs); + expect(EpochRows.fromJson(good).phaseMs, a % kEpochMs); + final bad = {...good, 'phase': (a % kEpochMs + 1000) % kEpochMs}; + expect(() => EpochRows.fromJson(bad), throwsA(isA())); + }); + }); + + group('EpochRowsMemo: rows keyed by phase', () { + final a1 = t0 + 7000; // phase 20000 + final a2 = t0 + 19000; // phase 2000 (32000 mod 30000) + final a3 = t0 + 23000; // phase 6000 + + test('same phase: rows are reused and extended, never recomputed', () { + final m = EpochRowsMemo(); + final early = night.between(0, 1000); + final late = night.between(0, 1500); + final r1 = _ask(m, a1, early); + expect(m.phaseBuilds, 1); + expect(m.samplesFolded, early.sampleCount); + expectSameRows(r1.allRows(), _batch(early, a1).allRows(), 'first ask'); + final r2 = _ask(m, a1, late); + expect(m.phaseBuilds, 1, reason: 'same phase: no rebuild'); + expect(m.samplesFolded, late.sampleCount, + reason: 'every sample folded exactly once'); + expectSameRows(r2.allRows(), _batch(late, a1).allRows(), 'extended'); + // nothing new: nothing folded + _ask(m, a1, late); + expect(m.samplesFolded, late.sampleCount); + expect(m.phaseBuilds, 1); + expect(m.phases, [a1 % kEpochMs]); + }); + + test('an onset that moves by whole epochs keeps the phase and the rows', () { + final m = EpochRowsMemo(); + final all = night.between(0, 2000); + _ask(m, a1, all); + final moved = a1 + 20 * kEpochMs; + final r = _ask(m, moved, all); + expect(m.phaseBuilds, 1); + expect(m.samplesFolded, all.sampleCount); + final want = _batch(all, moved).allRows(); + expectSameRows( + [for (final x in r.allRows()) if (x.startMs >= moved) x], want, + 'rows from the new onset on'); + }); + + test('a different phase yields rows exactly equal to a fresh batch on that ' + 'phase, and the memo keeps the new phase', () { + final m = EpochRowsMemo(); + final all = night.between(0, 2400); + _ask(m, a1, all); + final r = _ask(m, a2, all); + expect(m.phaseBuilds, 2); + expect(m.phases.last, a2 % kEpochMs); + expectSameRows(r.allRows(), _batch(all, a2).allRows(), 'phase changed'); + for (final x in r.allRows()) { + expect((x.startMs - a2) % kEpochMs, 0); + } + // and the old phase is still exact when asked again with more data + final more = night.between(0, night.seconds); + final back = _ask(m, a1, more); + expectSameRows(back.allRows(), _batch(more, a1).allRows(), 'back to a1'); + }); + + test('the memo is bounded: at most 2 phases are kept, always exact', () { + final m = EpochRowsMemo(); + final all = night.between(0, 1800); + for (final a in [a1, a2, a3, a1, a3, a2]) { + final r = _ask(m, a, all); + expect(m.phases.length, lessThanOrEqualTo(2), reason: 'after $a'); + expect(m.phases.last, a % kEpochMs, reason: 'most recent is held'); + expectSameRows(r.allRows(), _batch(all, a).allRows(), 'anchor $a'); + } + expect(m.phaseBuilds, greaterThanOrEqualTo(4), + reason: 'a third phase evicts one; asking for it again rebuilds'); + }); + + test('checkpoint: same phase resumes without a rebuild', () { + final m = EpochRowsMemo(); + _ask(m, a1, night.between(0, 1000)); + final back = EpochRowsMemo.fromJson( + jsonDecode(jsonEncode(m.toJson())) as Map); + expect(back.phases, [a1 % kEpochMs]); + final more = night.between(0, 2200); + final r = _ask(back, a1, more); + expect(back.phaseBuilds, 0, reason: 'nothing rebuilt after the restore'); + expect(back.samplesFolded, more.sampleCount - night.between(0, 1000).sampleCount, + reason: 'only the samples the checkpoint had not consumed'); + expectSameRows(r.allRows(), _batch(more, a1).allRows(), 'resumed'); + }); + + test('checkpoint: a different phase rebuilds, never reusing the other ' + 'phase\'s rows, and still equals batch', () { + final m = EpochRowsMemo(); + _ask(m, a1, night.between(0, 1000)); + final back = EpochRowsMemo.fromJson( + jsonDecode(jsonEncode(m.toJson())) as Map); + final more = night.between(0, 2200); + final r = _ask(back, a2, more); + expect(back.phaseBuilds, 1); + expect(back.samplesFolded, more.sampleCount, + reason: 'built from all the samples'); + for (final x in r.allRows()) { + expect((x.startMs - a2) % kEpochMs, 0, reason: 'start ${x.startMs}'); + } + expectSameRows(r.allRows(), _batch(more, a2).allRows(), 'other phase'); + }); + + test('a memo embedding a version-3 row checkpoint is refused', () { + final m = EpochRowsMemo(); + _ask(m, a1, night.between(0, 1000)); + final good = jsonDecode(jsonEncode(m.toJson())) as Map; + for (final v in [1, 2, 3]) { + final bad = jsonDecode(jsonEncode(good)) as Map; + for (final p in bad['phases'] as List) { + (p['rows'] as Map)['version'] = v; + } + expect(() => EpochRowsMemo.fromJson(bad), + throwsA(isA()), reason: 'rows version $v'); + } + }); + + test('a checkpoint is refused when it is of another type or shape', () { + final m = EpochRowsMemo(); + _ask(m, a1, night.between(0, 600)); + final good = jsonDecode(jsonEncode(m.toJson())) as Map; + for (final bad in >[ + {...good, 'type': 'EpochRows'}, + {...good}..remove('type'), + {}, + ]) { + expect(() => EpochRowsMemo.fromJson(bad), + throwsA(isA()), reason: jsonEncode(bad)); + } + }); + }); + + group('staging from memo rows = batch staging from the onset', () { + for (final (name, onsetSec, endSec, firstAsk) in [ + ('onset 777 s in, fresh memo', 777, 2400, null), + ('onset 1234 s in, memo first asked on another phase', 1234, 2900, 3), + ('onset on a 30 s multiple of the data start', 600, 2100, null), + ]) { + test(name, () { + final base = t0 ~/ 1000; + final onset = base + onsetSec, end = base + endSec; + final want = AdvancedSleepStager.stageWindow( + onset, end, night.grav, night.hrTs, rr: night.rrTsSec) + .stages; + final m = EpochRowsMemo(); + final window = night.between(onsetSec, endSec); + if (firstAsk != null) { + _ask(m, t0 + 5000 + firstAsk * 1000, night.between(onsetSec, onsetSec + 900)); + } + final rows = _ask(m, onset * 1000, window); + expect(showStages(rows.stageWindow(onset, end)), showStages(want)); + }); + } + + test('staged in growing prefixes through one memo = batch at the end', () { + final base = t0 ~/ 1000; + final onset = base + 777, end = base + 2400; + final m = EpochRowsMemo(); + EpochRows? rows; + for (var hi = 1000; hi <= 2400; hi += 350) { + rows = _ask(m, onset * 1000, night.between(777, hi < 2400 ? hi : 2400)); + } + rows = _ask(m, onset * 1000, night.between(777, 2400)); + expect(m.phaseBuilds, 1); + final want = AdvancedSleepStager.stageWindow( + onset, end, night.grav, night.hrTs, rr: night.rrTsSec) + .stages; + expect(showStages(rows.stageWindow(onset, end)), showStages(want)); + }); + }); +} diff --git a/test/onehz/epoch_rows_test.dart b/test/onehz/epoch_rows_test.dart new file mode 100644 index 0000000..dca1122 --- /dev/null +++ b/test/onehz/epoch_rows_test.dart @@ -0,0 +1,549 @@ +// The gap-anchored 30 s epoch grid and its `EpochRows` accumulator (combined +// bump, chunk B1-B3; owner decision 2026-10-22: anchor at the first sample after +// each data gap, replacing the wall-clock grid). +// +// B1 Epochs are 30,000 ms steps from the FIRST sample of each gap-free run +// (valid accelerometer coverage, with the stager's usable-run gap: the +// 60 s carry-forward). A run starts at the first sample and again at the +// first sample after any gap longer than that. The anchor depends on the +// data only, never on session detection, and is final as soon as the +// run's first sample arrives. An epoch is clipped only by a data gap: an +// epoch the data covers is a row (computed on the seconds that exist); +// an epoch with no data is not a row. +// B2 `EpochRows` (lookback +-165 s accel / HR, +-150 s RR) emits per-epoch +// feature rows. Chunk invariance is EXACT (any chunking, with JSON +// save/restore between chunks); rows ~3 min behind the newest sample are +// FINAL and never change; the last few are provisional; checkpointable. +// On an aligned contiguous stream the interior rows are today's +// `cardioEpochFeatures` rows (LF/HF excepted, see below). +// B3 Staging from the accumulated rows (`EpochRows.stageWindow`) equals +// `AdvancedSleepStager.stageWindow` in one batch, exactly, and that +// hypnogram's boundaries sit on the run's 30 s grid, whatever the session +// bounds. +// +// LF/HF is compared for presence only against today's rows: the span-dependent +// frequency grid (cardio_stager.dart `remFeaturesFromBeats`) may be fixed in +// the same bump, which moves the value but not whether it exists. +import 'dart:convert'; + +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +import '../support/property.dart'; +import 'support/causal_night.dart'; +import 'support/pre_bump_stager.dart'; + +const _alignedT0 = 1700000010000; // = kT0Ms, a multiple of 30,000 + +/// A night as raw input, optionally with holes (both streams lose the same +/// seconds, and the beats in them). +class _Input { + final int t0Ms, seconds; + final List hr; + final List accel; + final List rrMs, rrTs; + _Input(this.t0Ms, this.seconds, this.hr, this.accel, this.rrMs, this.rrTs); + + factory _Input.make(List<(String, int)> segs, + {int seed = 11, int t0Ms = _alignedT0, List<(int, int)> gaps = const []}) { + final n = synthNight(segs, seed: seed, t0Ms: t0Ms.toDouble()); + bool inGap(double ts) => gaps.any((g) => + ts >= t0Ms + g.$1 * 1000.0 && ts < t0Ms + g.$2 * 1000.0); + final rr = [], rt = []; + for (var i = 0; i < n.rrMs.length; i++) { + if (!inGap(n.rrTsMs[i])) { + rr.add(n.rrMs[i]); + rt.add(n.rrTsMs[i]); + } + } + return _Input(t0Ms, n.seconds, [for (final h in n.hr) if (!inGap(h.tsMs)) h], + [for (final a in n.accel) if (!inGap(a.tsMs)) a], rr, rt); + } + + /// Samples with `loSec <= second < hiSec` (seconds since t0). + void foldInto(EpochRows er, int loSec, int hiSec) { + final lo = t0Ms + loSec * 1000.0, hi = t0Ms + hiSec * 1000.0; + bool inside(double t) => t >= lo && t < hi; + final ri = [ + for (var i = 0; i < rrTs.length; i++) + if (inside(rrTs[i])) i + ]; + er.fold( + hr: [for (final h in hr) if (inside(h.tsMs)) h], + accel: [for (final a in accel) if (inside(a.tsMs)) a], + rrMs: [for (final i in ri) rrMs[i]], + rrTsMs: [for (final i in ri) rrTs[i]], + ); + } + + List get grav => [ + for (final a in accel) GravTs(a.tsMs ~/ 1000, a.x, a.y, a.z) + ]; + List get hrTs => [for (final h in hr) HrTs(h.tsMs ~/ 1000, h.hr)]; + List get rrTsSec => + [for (var i = 0; i < rrMs.length; i++) RrTs(rrTs[i] ~/ 1000, rrMs[i])]; +} + +EpochRows _oneFold(_Input s) { + final er = EpochRows(); + s.foldInto(er, 0, s.seconds); + return er; +} + +/// Folds [s] in chunks cut at [cuts] (seconds, ascending; empty chunks allowed), +/// saving to JSON TEXT and restoring between chunks when [restart]. +EpochRows _chunked(_Input s, List cuts, {bool restart = false}) { + var er = EpochRows(); + var at = 0; + for (final c in [...cuts, s.seconds]) { + s.foldInto(er, at, c); + at = c; + if (restart) { + er = EpochRows.fromJson(jsonDecode(jsonEncode(er.toJson())) as Map); + } + } + return er; +} + +List _cuts(Rng r, int seconds) { + const sizes = [1, 2, 7, 30, 31, 59, 100, 171, 500]; + final out = []; + var at = 0; + while (true) { + at += r.nextBool(.15) ? 0 : 1 + r.nextInt(sizes[r.nextInt(sizes.length)]); + if (at >= seconds) return out; + out.add(at); + } +} + +bool _sameRow(EpochFeatureRow a, EpochFeatureRow b) => + a.startMs == b.startMs && + [ + [a.motion, b.motion], [a.hr, b.hr], [a.hrSd, b.hrSd], + [a.rmssd, b.rmssd], [a.lfhf, b.lfhf], [a.rk, b.rk], [a.sdnn, b.sdnn], + ].every((p) => p[0].compareTo(p[1]) == 0); + +void _expectSameRows(List got, List want, String why) { + expect(got.length, want.length, reason: '$why: row count'); + for (var i = 0; i < got.length; i++) { + if (!_sameRow(got[i], want[i])) { + fail('$why: row $i (start ${want[i].startMs}) differs: ' + 'got ${_show(got[i])} want ${_show(want[i])}'); + } + } +} + +String _show(EpochFeatureRow r) => + '[${r.startMs} motion ${r.motion} hr ${r.hr} hrSd ${r.hrSd} rmssd ${r.rmssd} ' + 'lfhf ${r.lfhf} rk ${r.rk} sdnn ${r.sdnn}]'; + +List<(String, int)> _night() => [ + ('nrem', 900), ('rem', 600), ('nrem', 900), ('wake', 300), ('nrem', 300), + ]; + +void main() { + group('B1: the gap-anchored grid', () { + test('kEpochMs is 30,000 ms', () => expect(kEpochMs, 30000)); + + test('no data: no runs, no epoch', () { + final er = EpochRows(); + expect(er.runAnchorsMs, isEmpty); + expect(er.epochStartMsFor(_alignedT0), isNull); + }); + + for (final offsetMs in [0, 13000, 29000, 13250]) { + test('data starting ${offsetMs} ms past a Unix boundary: epochs are 30 s ' + 'steps from the first sample, every covered epoch is a row', () { + final s = _Input.make([('nrem', 700), ('wake', 140)], t0Ms: _alignedT0 + offsetMs); + final er = _oneFold(s); + expect(er.runAnchorsMs, [s.t0Ms]); + final rows = er.allRows(); + expect(rows, isNotEmpty); + expect(rows.first.startMs, s.t0Ms, reason: 'the first sample opens the grid'); + for (final r in rows) { + expect((r.startMs - s.t0Ms) % kEpochMs, 0, reason: 'start ${r.startMs}'); + } + for (var i = 1; i < rows.length; i++) { + expect(rows[i].startMs - rows[i - 1].startMs, kEpochMs, + reason: 'contiguous data: no epoch skipped or repeated'); + } + final lastEnd = s.t0Ms + s.seconds * 1000; + expect(rows.last.startMs + kEpochMs, + inInclusiveRange(lastEnd, lastEnd + kEpochMs), + reason: 'the epoch holding the last sample is the last row'); + // the per-timestamp lookup agrees with the rows + for (final r in rows) { + expect(er.epochStartMsFor(r.startMs), r.startMs); + expect(er.epochStartMsFor(r.startMs + kEpochMs - 1), r.startMs); + } + expect(er.epochStartMsFor(s.t0Ms - 1), isNull, reason: 'before the data'); + }); + } + + test('the anchor is final as soon as the first sample arrives', () { + final s = _Input.make([('nrem', 900)], t0Ms: _alignedT0 + 13250); + final er = EpochRows(); + s.foldInto(er, 0, 5); + expect(er.runAnchorsMs, [s.t0Ms]); + s.foldInto(er, 5, 400); + s.foldInto(er, 400, s.seconds); + expect(er.runAnchorsMs, [s.t0Ms], reason: 'later data never moves it'); + // and it does not depend on a session: nothing has been asked of the rows + expect(er.allRows().first.startMs, s.t0Ms); + }); + + test('a gap longer than the usable-run gap re-anchors at the first sample ' + 'after it; a shorter one does not', () { + // next valid accel at +61 s after the last: the carry bridges it (one run); + // at +62 s it does not (second run). Same threshold as the stager's + // usable-run split. + for (final (holeSec, runs) in [(60, 1), (61, 2), (100, 2), (600, 2)]) { + // a hole of `holeSec` missing seconds after +599 (both streams) + final s = _Input.make([('nrem', 1500)], + gaps: [(600, 600 + holeSec)], seed: 17, t0Ms: _alignedT0 + 13000); + final er = _oneFold(s); + expect(er.runAnchorsMs.length, runs, reason: 'hole $holeSec s'); + if (runs == 2) { + expect(er.runAnchorsMs.last, s.t0Ms + (600 + holeSec) * 1000); + // the new grid steps from the new anchor, not the old one + final starts = er.allRows().map((r) => r.startMs).toSet(); + expect(starts.contains(s.t0Ms + (600 + holeSec) * 1000), isTrue); + if ((holeSec) % 30 != 0) { + expect(starts.contains(s.t0Ms + (630 + (holeSec ~/ 30) * 30) * 1000), + isFalse, reason: 'the old phase is gone after the gap'); + } + } else { + for (final r in er.allRows()) { + expect((r.startMs - s.t0Ms) % kEpochMs, 0); + } + } + } + }); + + test('gap-only clipping: an epoch inside a gap is not a row, an epoch the ' + 'gap only nibbles still is (on the seconds that exist)', () { + // gap A: 117 s (a new run, anchored at +417); gap B: 10 s in the middle + // of one epoch (bridged). Offsets are seconds since t0. + final s = _Input.make([('nrem', 1500)], + gaps: [(300, 417), (810, 820)], seed: 17); + final er = _oneFold(s); + expect(er.runAnchorsMs, [s.t0Ms, s.t0Ms + 417000]); + final byStart = {for (final r in er.allRows()) r.startMs: r}; + // the first run's grid ends with its coverage (last sample + the carry) + for (var e = 360; e < 417; e += 1) { + expect(byStart.containsKey(s.t0Ms + e * 1000), isFalse, + reason: 'second +${e}s lies in the gap, no epoch starts there'); + } + for (final e in [270, 417, 447, 777, 837]) { + expect(byStart.containsKey(s.t0Ms + e * 1000), isTrue, + reason: 'epoch at +${e}s has data'); + } + final nibbled = byStart[s.t0Ms + 807 * 1000]!; // 417 + 13 * 30 = 807 + final present = [ + for (final h in s.hr) + if (h.tsMs >= s.t0Ms + 807000 && h.tsMs < s.t0Ms + 837000 && h.valid) + h.hr + ]; + expect(present.length, 20); + expect(nibbled.hr, closeTo(present.reduce((a, b) => a + b) / present.length, 1e-9), + reason: 'mean HR of the 20 seconds that exist'); + }); + + test('nothing is trimmed to a "session": the first and last wake epochs of a ' + 'night are rows', () { + final s = _Input.make([('wake', 300), ('nrem', 600), ('wake', 300)]); + final rows = _oneFold(s).allRows(); + expect(rows.first.startMs, _alignedT0); + expect(rows.last.startMs, _alignedT0 + 1200 * 1000 - kEpochMs); + }); + + test('a run that continues data folded earlier (seeded anchor) keeps the ' + 'earlier grid', () { + final s = _Input.make([('nrem', 2400)], t0Ms: _alignedT0 + 13250); + final full = _oneFold(s); + // fold only the part from +400 s on, telling the accumulator that the run + // it joins was anchored at the true first sample + final part = EpochRows(continuingAnchorMs: s.t0Ms); + s.foldInto(part, 400, s.seconds); + expect(part.runAnchorsMs, [s.t0Ms]); + final starts = part.allRows().map((r) => r.startMs).toList(); + expect(starts, isNotEmpty); + for (final st in starts) { + expect((st - s.t0Ms) % kEpochMs, 0); + } + // rows well inside the fed part are the full-night rows + final want = {for (final r in full.allRows()) r.startMs: r}; + var same = 0; + for (final r in part.allRows()) { + if (r.startMs < s.t0Ms + 400000 + 200000) continue; + expect(_sameRow(r, want[r.startMs]!), isTrue, reason: 'row ${r.startMs}'); + same++; + } + expect(same, greaterThan(50)); + }); + }); + + group('B2: EpochRows', () { + final night = _Input.make(_night(), t0Ms: _alignedT0 + 13000); + late final whole = _oneFold(night).allRows(); + + test('rows exist and are not all provisional', () { + expect(whole.length, greaterThanOrEqualTo(100), reason: '3000 s, 30 s epochs'); + final er = _oneFold(night); + expect(er.finalRows, isNotEmpty); + expect(er.provisionalRows(), isNotEmpty); + expect(er.finalRows.length + er.provisionalRows().length, whole.length); + }); + + forAll('chunk invariance: any chunking (empty chunks, JSON restarts) = one fold', + G.intIn(1, 100000), (seed) { + final r = Rng(seed); + final cuts = _cuts(r, night.seconds); + final got = _chunked(night, cuts, restart: r.nextBool()); + _expectSameRows(got.allRows(), whole, 'cuts $cuts'); + }, cases: 12, budget: const Duration(seconds: 120)); + + test('one-second chunks give the same rows', () { + final s = _Input.make([('nrem', 600), ('wake', 120)], t0Ms: _alignedT0 + 7000); + final want = _oneFold(s).allRows(); + final er = EpochRows(); + for (var t = 0; t < s.seconds; t++) { + s.foldInto(er, t, t + 1); + } + _expectSameRows(er.allRows(), want, '1 s chunks'); + }); + + test('rows ~3 min behind the newest sample are final and never change; the ' + 'last few are provisional', () { + final er = EpochRows(); + var prevFinal = 0; + var at = 0; + for (final c in [..._cuts(Rng(5), night.seconds), night.seconds]) { + night.foldInto(er, at, c); + at = c; + final fin = er.finalRows; + // never retracted, never rewritten: a prefix of the full-night rows + expect(fin.length, greaterThanOrEqualTo(prevFinal)); + _expectSameRows(fin, whole.sublist(0, fin.length), 'final rows at +${c}s'); + prevFinal = fin.length; + // liveness: everything 4+ min behind the newest sample is final + final newest = er.newestMs!; + final settledBy = whole.where((r) => r.startMs <= newest - 240000).length; + expect(fin.length, greaterThanOrEqualTo(settledBy), + reason: 'rows 4 min behind the newest sample must be final (at +${c}s)'); + // the provisional rows continue the final ones, and are few + final prov = er.provisionalRows(); + expect(prov.length, lessThanOrEqualTo(14)); + if (fin.isNotEmpty && prov.isNotEmpty) { + expect(prov.first.startMs, fin.last.startMs + kEpochMs); + } + expect(er.allRows().length, fin.length + prov.length); + } + expect(er.newestMs, night.t0Ms + (night.seconds - 1) * 1000); + }); + + test('provisional rows match a fold of the data so far, clipped at its end', () { + final cut = 1500; + final er = EpochRows(); + night.foldInto(er, 0, cut); + final prefixOnly = _Input.make(_night(), t0Ms: _alignedT0 + 13000); + final fresh = EpochRows(); + prefixOnly.foldInto(fresh, 0, cut); + _expectSameRows(er.allRows(), fresh.allRows(), 'same prefix, same rows'); + // asking does not change anything + final before = jsonEncode(er.toJson()); + er.provisionalRows(); + er.allRows(); + expect(jsonEncode(er.toJson()), before); + }); + + test('checkpoint: toJson -> JSON text -> fromJson is lossless and resumes', () { + final er = EpochRows(); + night.foldInto(er, 0, 1700); + final text = jsonEncode(er.toJson()); + final back = EpochRows.fromJson(jsonDecode(text) as Map); + expect(jsonEncode(back.toJson()), text); + _expectSameRows(back.allRows(), er.allRows(), 'restored'); + night.foldInto(back, 1700, night.seconds); + _expectSameRows(back.allRows(), whole, 'resumed'); + }); + + test('an empty accumulator checkpoints and resumes', () { + final back = EpochRows.fromJson( + jsonDecode(jsonEncode(EpochRows().toJson())) as Map); + expect(back.allRows(), isEmpty); + expect(back.newestMs, isNull); + night.foldInto(back, 0, night.seconds); + _expectSameRows(back.allRows(), whole, 'from an empty checkpoint'); + }); + + test('a checkpoint of another type, version or shape is refused', () { + final er = EpochRows(); + night.foldInto(er, 0, 400); + final good = jsonDecode(jsonEncode(er.toJson())) as Map; + for (final bad in >[ + {...good, 'type': 'RrCorrector'}, + {...good, 'version': 999}, + {...good}..remove('type'), + {'type': good['type'], 'version': good['version']}, + {}, + ]) { + expect(() => EpochRows.fromJson(bad), throwsA(isA()), + reason: jsonEncode(bad)); + } + }); + + test('checkpoints written before 81b9b4b (versions 1-3) are refused', () { + final er = EpochRows(); + night.foldInto(er, 0, 1000); + final good = jsonDecode(jsonEncode(er.toJson())) as Map; + expect(good['version'], 4, reason: 'row semantics changed: new version'); + for (final v in [1, 2, 3]) { + // Finalized rows carry the old (wrong) values; the shape is otherwise + // what version 3 wrote. + final bad = {...good, 'version': v}; + expect(() => EpochRows.fromJson(bad), throwsA(isA()), + reason: 'version $v'); + } + }); + + test('on an aligned contiguous stream the interior rows are today\'s ' + 'cardioEpochFeatures rows', () { + final s = _Input.make(_night()); // aligned t0 + final rows = {for (final r in _oneFold(s).allRows()) r.startMs: r}; + final f = cardioEpochFeatures( + [for (final h in s.hr) h.hr], + s.accel, + rrMs: s.rrMs, + rrTsMs: s.rrTs, + ); + var compared = 0; + for (var e = 0; e < f.length; e++) { + final startSec = e * 30; + if (startSec < 165 || startSec + 30 + 165 > s.seconds) continue; + final r = rows[_alignedT0 + startSec * 1000]; + expect(r, isNotNull, reason: 'row for epoch $e'); + bool same(double a, double b) => a.compareTo(b) == 0; + expect(same(r!.motion, f.motion[e]), isTrue, reason: 'motion e=$e ${r.motion} vs ${f.motion[e]}'); + expect(same(r.hr, f.hr[e]), isTrue, reason: 'hr e=$e'); + expect(same(r.hrSd, f.hrSd[e]), isTrue, reason: 'hrSd e=$e'); + expect(same(r.rmssd, f.rmssd[e]), isTrue, reason: 'rmssd e=$e'); + expect(same(r.sdnn, f.sdnn[e]), isTrue, reason: 'sdnn e=$e'); + expect(same(r.rk, f.rk[e]), isTrue, reason: 'rk e=$e'); + expect(r.lfhf.isNaN, f.lfhf[e].isNaN, reason: 'lfhf presence e=$e'); + compared++; + } + expect(compared, greaterThan(80)); + }); + }); + + group('B3: staging on the gap-anchored grid', () { + SleepSession batch(_Input s, [int fromSec = 0, int? toSec]) => + AdvancedSleepStager.stageWindow(s.t0Ms ~/ 1000 + fromSec, + s.t0Ms ~/ 1000 + (toSec ?? s.seconds), s.grav, s.hrTs, rr: s.rrTsSec); + + String show(List l) => jsonEncode([for (final g in l) g.toJson()]); + + for (final (name, offsetMs, gaps) in [ + ('unaligned start, no gaps', 13000, <(int, int)>[]), + ('aligned start', 0, <(int, int)>[]), + ('unaligned start, a 100 s hole in accel and HR', 13000, [(1300, 1400)]), + ]) { + test('staging from accumulated rows = one batch, exactly ($name)', () { + final s = _Input.make(_night(), t0Ms: _alignedT0 + offsetMs, gaps: gaps, seed: 29); + final want = batch(s).stages; + final r = Rng(offsetMs + gaps.length); + final er = _chunked(s, _cuts(r, s.seconds), restart: true); + final got = er.stageWindow(s.t0Ms ~/ 1000, s.t0Ms ~/ 1000 + s.seconds); + expect(show(got), show(want)); + }); + } + + test('a session that begins inside a gap-free run is staged on ITS OWN ' + 'start (grid option 2: phase 0 at the session start, as pre-bump)', () { + // Replaces the gap-anchored expectation (owner decision 2026-10-23): the + // run\'s grid no longer decides where a session\'s epochs sit. + final s = _Input.make(_night(), t0Ms: _alignedT0 + 13000, seed: 37); + for (final (from, to) in [(1000, 2400), (777, 2111), (1234, 2999)]) { + final t0 = s.t0Ms ~/ 1000; + final got = batch(s, from, to).stages; + var checked = 0; + for (var i = 1; i < got.length; i++) { + if (got[i].stage == 'unobserved' || got[i - 1].stage == 'unobserved') { + continue; + } + expect((got[i].start - (t0 + from)) % 30, 0, + reason: 'session +$from..+${to}s: boundary ${got[i].start}'); + checked++; + } + // +777..+2111 has two boundaries in the pre-bump labels too. + expect(checked, greaterThan(1), reason: 'session +$from..+${to}s'); + expect(show(got), + show(preBumpStageSessionCardio(t0 + from, t0 + to, s.grav, s.hrTs, s.rrTsSec)), + reason: 'session +$from..+${to}s equals the pre-bump labels'); + } + }); + + test('the hypnogram\'s boundaries sit on the 30 s grid of the first sample', () { + for (final offsetMs in [13000, 21000]) { + final s = _Input.make(_night(), t0Ms: _alignedT0 + offsetMs, seed: 31); + final st = batch(s); + final stages = st.stages; + expect(stages.first.start, st.start); + expect(stages.last.end, st.end); + for (var i = 1; i < stages.length; i++) { + expect(stages[i].start, stages[i - 1].end, reason: 'tiles without holes'); + } + var checked = 0; + for (var i = 1; i < stages.length; i++) { + final edge = stages[i].start; + if (stages[i].stage == 'unobserved' || stages[i - 1].stage == 'unobserved') { + continue; + } + expect((edge - s.t0Ms ~/ 1000) % 30, 0, + reason: 'boundary at $edge (${stages[i - 1].stage} -> ' + '${stages[i].stage}), data starts at +${offsetMs ~/ 1000}s'); + checked++; + } + expect(checked, greaterThan(2), reason: 'the night must change stage'); + } + }); + + test('a hole shorter than the carry-forward does not move the grid', () { + final s = _Input.make(_night(), + t0Ms: _alignedT0 + 13000, gaps: [(1000, 1040)], seed: 33); + final stages = batch(s).stages; + for (var i = 1; i < stages.length; i++) { + if (stages[i].stage == 'unobserved' || stages[i - 1].stage == 'unobserved') { + continue; + } + expect((stages[i].start - s.t0Ms ~/ 1000) % 30, 0, + reason: 'boundary ${stages[i].start}'); + } + }); + + test('a hole longer than the carry-forward re-anchors the grid after it', () { + // 1100 % 30 = 20: the second run's grid is 20 s out of phase with the first + final s = _Input.make(_night(), + t0Ms: _alignedT0 + 13000, gaps: [(1000, 1100)], seed: 35); + final t0 = s.t0Ms ~/ 1000; + final stages = batch(s).stages; + var before = 0, after = 0; + for (var i = 1; i < stages.length; i++) { + if (stages[i].stage == 'unobserved' || stages[i - 1].stage == 'unobserved') { + continue; + } + final edge = stages[i].start - t0; + if (edge < 1000) { + expect(edge % 30, 0, reason: 'first run, boundary at +$edge'); + before++; + } else { + expect((edge - 1100) % 30, 0, reason: 'second run, boundary at +$edge'); + after++; + } + } + expect(before, greaterThan(0)); + expect(after, greaterThan(0)); + }); + }); +} diff --git a/test/onehz/fixtures/real_night_2026_07_hypnogram_v1.csv b/test/onehz/fixtures/real_night_2026_07_hypnogram_v1.csv new file mode 100644 index 0000000..6253fe1 --- /dev/null +++ b/test/onehz/fixtures/real_night_2026_07_hypnogram_v1.csv @@ -0,0 +1,57 @@ +rel_start_s,rel_end_s,stage +0,1980,light +1980,2670,rem +2670,5760,light +5760,6210,deep +6210,6240,light +6240,6540,deep +6540,7050,light +7050,7260,deep +7260,8130,light +8130,8370,deep +8370,8880,light +8880,9630,rem +9630,10140,light +10140,10350,deep +10350,13260,light +13260,14070,deep +14070,14160,light +14160,14850,deep +14850,15060,light +15060,15360,rem +15360,15420,wake +15420,15510,rem +15510,15630,wake +15630,15690,rem +15690,15780,wake +15780,16680,rem +16680,17220,light +17220,17520,rem +17520,17820,light +17820,18600,rem +18600,18900,light +18900,19500,rem +19500,21480,light +21480,22110,deep +22110,22680,light +22680,23880,rem +23880,23940,wake +23940,24060,light +24060,24180,wake +24180,24360,rem +24360,24510,wake +24510,24960,rem +24960,25050,wake +25050,25170,rem +25170,25230,wake +25230,25320,rem +25320,25470,wake +25470,26820,light +26820,27150,wake +27150,28260,light +28260,28290,wake +28290,29910,light +29910,31050,rem +31050,31290,wake +31290,31980,rem +31980,32041,wake diff --git a/test/onehz/incremental/calculation_cache_test.dart b/test/onehz/incremental/calculation_cache_test.dart new file mode 100644 index 0000000..cb55275 --- /dev/null +++ b/test/onehz/incremental/calculation_cache_test.dart @@ -0,0 +1,190 @@ +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +void main() { + test('cache retains null values and distinguishes changed list order', () { + final cache = CalculationCache(); + var calls = 0; + Object? calculate() { + calls++; + return null; + } + + expect(cache.evaluate('nullable', [1, 2, null], calculate), isNull); + expect(cache.evaluate('nullable', [1, 2, null], calculate), isNull); + expect(calls, 1); + expect(cache.evaluate('nullable', [2, 1, null], calculate), isNull); + expect(calls, 2); + expect(cache.computations, 2); + expect(cache.hits, 1); + }); + test('cache protects typed lists while retaining their return type', () { + final cache = CalculationCache(); + final original = [1.25, 3.5]; + final first = cache.evaluate>('list', 1, () => original); + first.add(9); + original[0] = 8; + final next = cache.evaluate>('list', 1, () => [100.0]); + expect(next, [1.25, 3.5]); + next.clear(); + expect(cache.evaluate>('list', 1, () => [100.0]), [1.25, 3.5]); + }); + test( + 'cache snapshots nested dependencies and deep compares maps, null and NaN', + () { + final cache = CalculationCache(); + final dependency = { + 'hr': [ + 70.0, + null, + double.nan, + {'day': 1} + ], + 'profile': {'sex': 'female', 'weight': 61.0}, + }; + var calls = 0; + int calculate() => ++calls; + expect(cache.evaluate('load', dependency, calculate), 1); + expect( + cache.evaluate( + 'load', + { + 'profile': {'weight': 61.0, 'sex': 'female'}, + 'hr': [ + 70.0, + null, + double.nan, + {'day': 1} + ], + }, + calculate), + 1); + expect(calls, 1); + expect(cache.computations, 1); + expect(cache.hits, 1); + (dependency['hr'] as List)[0] = 71.0; + expect(cache.evaluate('load', dependency, calculate), 2); + ((dependency['hr'] as List)[3] as Map)['day'] = 2; + expect(cache.evaluate('load', dependency, calculate), 3); + (dependency['profile'] as Map)['sex'] = 'male'; + expect(cache.evaluate('load', dependency, calculate), 4); + expect(cache.evaluate('load', dependency, calculate), 4); + expect(cache.computations, 4); + expect(cache.hits, 2); + }); + test('cache protects mutable values on first return and subsequent hits', () { + final cache = CalculationCache(); + final produced = { + 'nested': [ + {'value': 7} + ], + 'values': [1, 2] + }; + Map calculate() => produced; + final first = cache.evaluate('nested', [1], calculate); + (first['nested'][0] as Map)['value'] = 999; + (produced['values'] as List).add(3); + final second = cache.evaluate('nested', [1], calculate); + expect(second, { + 'nested': [ + {'value': 7} + ], + 'values': [1, 2] + }); + (second['values'] as List).clear(); + expect(cache.evaluate('nested', [1], calculate), { + 'nested': [ + {'value': 7} + ], + 'values': [1, 2] + }); + expect(cache.computations, 1); + expect(cache.hits, 2); + }); + test( + 'cache exceptions preserve prior dependencies and do not count as computations', + () { + final cache = CalculationCache(); + expect(cache.evaluate('load', [1], () => 7), 7); + expect( + () => cache.evaluate('load', [2], () => throw StateError('bad')), + throwsStateError); + expect(cache.computations, 1); + expect(cache.hits, 0); + expect(cache.evaluate('load', [1], () => 99), 7); + expect(cache.evaluate('load', [2], () => 8), 8); + expect(cache.computations, 2); + expect(cache.hits, 1); + expect( + () => cache.evaluate('load', [2], () => throw StateError('bad'), + full: true), + throwsStateError); + expect(cache.evaluate('load', [2], () => 99), 8); + expect(cache.computations, 2); + }); + for (final mode in CalculationMode.values) { + test('cache mode $mode controls full bypass', () { + expect(mode.canReuse, mode == CalculationMode.periodicAwake); + final cache = CalculationCache(); + var calls = 0; + int calculate() => ++calls; + expect(cache.evaluate('load', null, calculate, full: !mode.canReuse), 1); + expect(cache.evaluate('load', null, calculate, full: !mode.canReuse), + mode.canReuse ? 1 : 2); + expect(cache.computations, mode.canReuse ? 1 : 2); + expect(cache.hits, mode.canReuse ? 1 : 0); + }); + } + test('cache keys are isolated, full compute refreshes, clear discards values', + () { + final cache = CalculationCache(); + var calls = 0; + int calculate() => ++calls; + expect(cache.evaluate('a', null, calculate), 1); + expect(cache.evaluate('b', null, calculate), 2); + expect(cache.evaluate('a', null, calculate, full: true), 3); + expect(cache.evaluate('a', null, calculate), 3); + cache.clear(); + expect(cache.evaluate('a', null, calculate), 4); + expect(calls, 4); + }); + test('cache defaults to bounded 128 entries', () { + final cache = CalculationCache(); + var calls = 0; + for (var i = 0; i < 129; i++) { + cache.evaluate('metric-$i', null, () => ++calls); + } + for (var i = 0; i < 129; i++) { + cache.evaluate('metric-$i', null, () => ++calls); + } + expect(calls, greaterThan(129), + reason: 'at least one old key must be evicted'); + expect(cache.computations, calls); + }); + for (final capacity in [1, 3, 7]) { + test('cache bounded capacity=$capacity does not retain every entry', () { + final cache = CalculationCache(maxEntries: capacity); + var calls = 0; + for (var i = 0; i < capacity + 1; i++) { + cache.evaluate('metric-$i', [i], () => ++calls); + } + for (var i = 0; i < capacity + 1; i++) { + cache.evaluate('metric-$i', [i], () => ++calls); + } + expect(calls, greaterThan(capacity + 1)); + }); + } + test('cache handles typed immutable results without encoding them as JSON', + () { + final cache = CalculationCache(); + const result = Metric( + value: 7, confidence: .7, tier: Tier.estimate, inputs_used: ['hr']); + expect(cache.evaluate('metric', [1], () => result), same(result)); + expect( + cache.evaluate>( + 'metric', [1], () => throw StateError('hit')), + same(result)); + expect(cache.computations, 1); + expect(cache.hits, 1); + }); +} diff --git a/test/onehz/incremental/checkpoint_ownership_test.dart b/test/onehz/incremental/checkpoint_ownership_test.dart new file mode 100644 index 0000000..b9603ca --- /dev/null +++ b/test/onehz/incremental/checkpoint_ownership_test.dart @@ -0,0 +1,51 @@ +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +import '../support/incremental_compare.dart'; +import '../support/incremental_fixtures.dart'; + +void main() { + const original = + WorkoutUserProfile(weightKg: 80, heightCm: 180, age: 40, sex: 'male'); + for (final replacement in [ + const WorkoutUserProfile(weightKg: 95, heightCm: 180, age: 40, sex: 'male'), + const WorkoutUserProfile(weightKg: 80, heightCm: 160, age: 40, sex: 'male'), + const WorkoutUserProfile(weightKg: 80, heightCm: 180, age: 60, sex: 'male'), + const WorkoutUserProfile( + weightKg: 80, heightCm: 180, age: 40, sex: 'female'), + ]) { + test( + 'restored minute profile owns checkpoint ${replacement.weightKg}/' + '${replacement.heightCm}/${replacement.age}/${replacement.sex}', () { + final f = incrementalFixture(minutes: 35); + final originalState = IncrementalMinuteMetrics(); + originalState.sync(f.minuteKeys, f.hr, + cadenceSpm: f.cadence, + restingHr: 54, + maxHr: 186, + quietHrr: .12, + profile: original); + final json = originalState.toJson(); + final restored = IncrementalMinuteMetrics.fromJson(json); + final parameters = json['parameters'] as List; + final profile = parameters[3] as List; + profile.setAll(0, [ + replacement.weightKg, + replacement.heightCm, + replacement.age, + replacement.sex + ]); + final work = restored.processedMinutes; + final actual = restored.sync(f.minuteKeys, f.hr, + cadenceSpm: f.cadence, + restingHr: 54, + maxHr: 186, + quietHrr: .12, + profile: replacement); + minuteClose(actual, f.minuteKeys, f.hr, + cadence: f.cadence, profile: replacement); + expect(restored.processedMinutes - work, f.hr.length, + reason: 'changing profile reprices retained contributions'); + }); + } +} diff --git a/test/onehz/incremental/drift_test.dart b/test/onehz/incremental/drift_test.dart new file mode 100644 index 0000000..aaaae4e --- /dev/null +++ b/test/onehz/incremental/drift_test.dart @@ -0,0 +1,123 @@ +// Long-run drift: thousands of incremental updates, compared with the batch +// oracle along the way. The shorter parity tests cannot show error that only +// builds up over a full night of appends or a long sliding window. +import 'dart:io'; +import 'dart:math' as math; + +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; +import '../support/incremental_compare.dart'; + +double _rel(double? a, double? e) { + if (a == null || e == null) return a == e ? 0 : double.infinity; + final scale = e.abs(); + return scale < 1e-12 ? (a - e).abs() : (a - e).abs() / scale; +} + +/// Logs the worst relative error seen, so the margin under the bound is +/// visible in the test output and not only pass/fail. +void _report(String label, double worst) => + stdout.writeln('[drift] $label worst relative error ${worst.toStringAsExponential(2)}'); + +/// A night of beats on absolute epoch seconds, the magnitude production +/// timestamps carry, with a slow drift and respiratory modulation. +({List t, List y}) _night(int beats, {int seed = 7}) { + final rnd = math.Random(seed); + final t = [], y = []; + var clock = 1791028730.0; + for (var i = 0; i < beats; i++) { + final rr = 900 + + 120 * math.sin(i / 4000) + + 40 * math.sin(i * 2 * math.pi * .25 * .9) + + rnd.nextDouble() * 30; + clock += rr / 1000; + if (i % 2311 == 2310) clock += 45; // an occasional dropout + t.add(clock); + y.add(rr); + } + return (t: t, y: y); +} + +void main() { + final frequencies = [for (var k = 0; k <= 40; k++) .0033 + k * .01]; + + test('Lomb–Scargle over a full night of appends stays on the batch result', + () { + final night = _night(30000); + var state = IncrementalLombScargle(frequencies); + var worst = 0.0, worstAbsolute = 0.0; + var n = 0; + for (final checkpointAt in [500, 2000, 8000, 16000, 24000, 30000]) { + while (n < checkpointAt) { + n = math.min(checkpointAt, n + 13); + state.sync(night.t.sublist(0, n), night.y.sublist(0, n)); + } + final ts = night.t.sublist(0, n), ys = night.y.sublist(0, n); + final actual = state.sync(ts, ys)!; + // The oracle runs on times shifted by the first beat. The periodogram is + // shift-invariant, and the shift is exact here (Sterbenz), but on raw + // epoch seconds the batch's own trig arguments reach ~1e10 rad and lose + // ~1e-6 relative; checked against an 80-bit reference, that batch error + // is 4.6e-6 while shifted double precision is 3.8e-13. + final shifted = [for (final x in ts) x - ts.first]; + final expected = lombScargle(shifted, ys, frequencies)!; + final absolute = lombScargle(ts, ys, frequencies)!; + for (var i = 0; i < expected.spectrum.length; i++) { + final a = actual.spectrum[i].psd, e = expected.spectrum[i].psd; + worst = math.max(worst, _rel(a, e)); + expect(_rel(a, e), lessThan(1e-9), reason: 'f=${frequencies[i]}'); + worstAbsolute = + math.max(worstAbsolute, _rel(a, absolute.spectrum[i].psd)); + expect(_rel(a, absolute.spectrum[i].psd), lessThan(1e-5)); + } + // A restored checkpoint must continue the same way. + state = IncrementalLombScargle.fromJson(checkpoint(state.toJson())); + } + _report('lomb 30000 beats vs shifted batch', worst); + _report('lomb 30000 beats vs epoch-time batch', worstAbsolute); + }); + + test('moments over a long sliding window with removals stay on two-pass', + () { + // Large offset, small spread: the case where naive sum-of-squares loses + // every digit. 100 000 adds and 99 700 removals through a 300-value window. + final rnd = math.Random(11); + final window = []; + final state = RunningMoments(); + var worstMean = 0.0, worstSd = 0.0; + for (var i = 0; i < 100000; i++) { + final x = 1e6 + 50 * math.sin(i / 97) + rnd.nextDouble(); + window.add(x); + state.add(x); + if (window.length > 300) state.remove(window.removeAt(0)); + if (i % 997 == 0 || i == 99999) { + worstMean = math.max(worstMean, _rel(state.mean, mean(window))); + worstSd = math.max(worstSd, _rel(state.sampleSd, stddev(window))); + } + } + _report('moments window mean', worstMean); + _report('moments window sd', worstSd); + expect(worstMean, lessThan(1e-9)); + expect(worstSd, lessThan(1e-9)); + }); + + test('moments removed down to two values from a large set', () { + final rnd = math.Random(3); + final values = [ + for (var i = 0; i < 50000; i++) 800 + 100 * rnd.nextDouble() + ]; + final state = RunningMoments(); + for (final x in values) { + state.add(x); + } + var worstSd = 0.0; + while (values.length > 2) { + state.remove(values.removeLast()); + if (values.length % 1000 == 0 || values.length < 10) { + worstSd = math.max(worstSd, _rel(state.sampleSd, stddev(values))); + } + } + _report('moments shrink sd', worstSd); + expect(worstSd, lessThan(1e-9)); + }); +} diff --git a/test/onehz/incremental/enmo_series_test.dart b/test/onehz/incremental/enmo_series_test.dart new file mode 100644 index 0000000..2ac15ae --- /dev/null +++ b/test/onehz/incremental/enmo_series_test.dart @@ -0,0 +1,123 @@ +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; +import '../support/incremental_compare.dart'; +import '../support/incremental_fixtures.dart'; + +void main() { + for (final seed in [1, 42, 914]) { + for (final hz in [.5, 1.0, 2.0]) { + for (final chunk in [1, 31, 97]) { + test('ENMO explicit reference seed=$seed hz=$hz chunk=$chunk', () { + final f = incrementalFixture( + seed: seed, cadenceHz: hz, originMs: 1700000010000); + var state = IncrementalEnmoSeries(); + for (final n in prefixSizes(f.accel.length, chunk)) { + final a = f.accel.sublist(0, n); + enmoClose(state.sync(a, gRef: 1.013456789, expectedMinutes: 12), + enmoSeries(a, gRef: 1.013456789, expectedMinutes: 12)); + final work = state.processedPoints; + state.sync(a, gRef: 1.013456789, expectedMinutes: 12); + expect(state.processedPoints, work); + if (n == 31 || n == f.accel.length) { + state = + IncrementalEnmoSeries.fromJson(checkpoint(state.toJson())); + } + } + expect(state.processedPoints, lessThanOrEqualTo(f.accel.length)); + }); + } + } + } + for (final window in [0.0, 1.0, 15.0, 61.5]) { + for (final minimum in [1, 30, 60]) { + test('ENMO window=$window coverage minimum=$minimum', () { + final samples = [ + for (final t in [ + 0.0, + 999, + 1000, + 14999, + 15000, + 59999, + 60000, + 60001, + 119999, + 120000, + 199999 + ]) + AccelSample(t, t % 17 / 30, .1, 1), + ]; + final state = IncrementalEnmoSeries(); + for (var n = 0; n <= samples.length; n++) { + final a = samples.sublist(0, n); + enmoClose( + state.sync(a, + gRef: 1, + gravityWindowS: window, + minSamplesPerMinute: minimum), + enmoSeries(a, + gRef: 1, + gravityWindowS: window, + minSamplesPerMinute: minimum)); + } + }); + } + } + test('ENMO auto-calibration recomputes each changing prefix', () { + final f = incrementalFixture(); + var state = IncrementalEnmoSeries(); + for (final n in prefixSizes(f.accel.length, 29)) { + final a = f.accel.sublist(0, n); + enmoClose(state.sync(a), enmoSeries(a)); + state = IncrementalEnmoSeries.fromJson(checkpoint(state.toJson())); + } + }); + test('ENMO parameter changes, replace, remove, reorder and force', () { + final f = incrementalFixture(); + var a = [...f.accel]; + final state = IncrementalEnmoSeries(); + state.sync(a, gRef: 1); + enmoClose( + state.sync(a, + gRef: 1.05, + gravityWindowS: 31, + minSamplesPerMinute: 60, + expectedMinutes: 100), + enmoSeries(a, + gRef: 1.05, + gravityWindowS: 31, + minSamplesPerMinute: 60, + expectedMinutes: 100)); + final old = a[7]; + a[7] = AccelSample(old.tsMs, old.x + .07, old.y, old.z); + enmoClose(state.sync(a, gRef: 1), enmoSeries(a, gRef: 1)); + a = a.sublist(9, 140).reversed.toList(); + enmoClose(state.sync(a, gRef: 1), enmoSeries(a, gRef: 1)); + final work = state.processedPoints; + enmoClose(state.sync(a, gRef: 1, force: true), enmoSeries(a, gRef: 1)); + expect(state.processedPoints, greaterThan(work)); + enmoClose(state.sync([], gRef: 1), enmoSeries([], gRef: 1)); + }); + test('ENMO duplicate timestamps, invalid samples and constant vector', () { + final a = [ + for (var i = 0; i < 80; i++) + AccelSample((i ~/ 2) * 1000.0, .3, -.4, 1, valid: i % 3 != 0), + const AccelSample(500, 100, 100, 100, valid: false), + ]; + final state = IncrementalEnmoSeries(); + enmoClose(state.sync(a, gRef: 1), enmoSeries(a, gRef: 1)); + final invalid = + a.map((s) => AccelSample(s.tsMs, s.x, s.y, s.z, valid: false)).toList(); + enmoClose(state.sync(invalid), enmoSeries(invalid)); + }); + test('ENMO real capture streamed with batch-calibrated reference', () { + final f = realIncrementalFixture(); + expect(f.accel.length, 900); + final g = enmoSeries(f.accel).gRef; + final state = IncrementalEnmoSeries(); + for (final n in prefixSizes(f.accel.length, 113)) { + final a = f.accel.sublist(0, n); + enmoClose(state.sync(a, gRef: g), enmoSeries(a, gRef: g)); + } + }); +} diff --git a/test/onehz/incremental/hrv_time_test.dart b/test/onehz/incremental/hrv_time_test.dart new file mode 100644 index 0000000..9570da3 --- /dev/null +++ b/test/onehz/incremental/hrv_time_test.dart @@ -0,0 +1,158 @@ +import 'dart:math' as math; + +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; +import '../support/incremental_compare.dart'; +import '../support/incremental_fixtures.dart'; + +void main() { + test('HRV jitter gate refuses only successive-difference outputs', () { + final jitter = incrementalFixture(seed: 42, jitter: true, gaps: false); + final oracle = hrvTime(jitter.nn, nnTimesMs: jitter.times); + expect(oracle.value!.diffAcf1, lessThan(kNnDiffAcf1Floor)); + expect(oracle.value!.rmssd, isNull); + expect(oracle.value!.pnn50, isNull); + expect(oracle.value!.sdnn, isNotNull); + expect(oracle.value!.sdann, isNotNull); + hrvClose( + IncrementalHrvTime().sync(jitter.nn, nnTimesMs: jitter.times), oracle); + }); + for (final seed in [1, 42, 914]) { + for (final jitter in [false, true]) { + for (final chunk in [1, 37, 251]) { + test( + 'HRV seams and evolving bins seed=$seed jitter=$jitter chunk=$chunk', + () { + final f = incrementalFixture(seed: seed, jitter: jitter); + var state = IncrementalHrvTime(); + for (final n in prefixSizes(f.nn.length, chunk)) { + final nn = f.nn.sublist(0, n), t = f.times.sublist(0, n); + hrvClose(state.sync(nn, nnTimesMs: t, artifactFraction: .17), + hrvTime(nn, nnTimesMs: t, artifactFraction: .17)); + final work = state.processedPoints; + state.sync(nn, nnTimesMs: t, artifactFraction: .17); + expect(state.processedPoints, work); + if (n % 37 == 0 || n == f.nn.length) { + state = IncrementalHrvTime.fromJson(checkpoint(state.toJson())); + } + } + expect(state.processedPoints, lessThanOrEqualTo(f.nn.length + 2)); + }); + } + } + } + for (final origin in [0.0, 299999.5, 1700000010123.125]) { + test('HRV five-minute exact boundaries and shifted origin=$origin', () { + final nn = List.generate(610, (i) => 900 + (i % 11) * 7.0); + final t = List.generate(610, (i) => origin + i * 1000); + final state = IncrementalHrvTime(); + for (final n in [2, 299, 300, 301, 302, 599, 600, 601, 602, 610]) { + final y = nn.sublist(0, n), ts = t.sublist(0, n); + hrvClose(state.sync(y, nnTimesMs: ts), hrvTime(y, nnTimesMs: ts)); + } + }); + } + for (final artifacts in [0.0, .15, .8, 1.0]) { + test('HRV artifact confidence reprice=$artifacts', () { + final f = incrementalFixture(); + final state = IncrementalHrvTime(); + state.sync(f.nn, nnTimesMs: f.times); + hrvClose( + state.sync(f.nn, nnTimesMs: f.times, artifactFraction: artifacts), + hrvTime(f.nn, nnTimesMs: f.times, artifactFraction: artifacts)); + }); + } + test('HRV replacements, dropped prefix and force rebuild', () { + final f = incrementalFixture(); + var nn = [...f.nn], times = [...f.times]; + final state = IncrementalHrvTime(); + state.sync(nn, nnTimesMs: times); + nn[119] += 60; + hrvClose(state.sync(nn, nnTimesMs: times), hrvTime(nn, nnTimesMs: times)); + times[113] += 18000; + hrvClose(state.sync(nn, nnTimesMs: times), hrvTime(nn, nnTimesMs: times)); + nn = nn.sublist(27, 700); + times = times.sublist(27, 700); + hrvClose(state.sync(nn, nnTimesMs: times), hrvTime(nn, nnTimesMs: times)); + final work = state.processedPoints; + hrvClose(state.sync(nn, nnTimesMs: times, force: true), + hrvTime(nn, nnTimesMs: times)); + expect(state.processedPoints - work, nn.length); + hrvClose(state.sync(nn), hrvTime(nn)); + hrvClose(state.sync(nn, nnTimesMs: [1]), hrvTime(nn, nnTimesMs: [1])); + }); + test('HRV constant series, singleton, empty and all separated seams', () { + final state = IncrementalHrvTime(); + for (final n in [0, 1, 2, 29, 30, 31, 32, 80]) { + final nn = List.filled(n, 800); + final times = List.generate(n, (i) => i * 5000.0); + hrvClose(state.sync(nn), hrvTime(nn)); + hrvClose(state.sync(nn, nnTimesMs: times), hrvTime(nn, nnTimesMs: times)); + } + }); + // Below the ACF floor upstream's hrvTime reads the difference runs: a + // respiratory line on a low floor exempts the night, and a refused night may + // publish from the windows on a breathing line steady in Hz. Both are batch + // decisions the stream must reproduce at every prefix. + ({List nn, List t}) breathing( + {required double hours, + required double baseMs, + required double driftMs, + required double breathMs, + required double breathHz, + required double jitterMs}) { + final r = math.Random(7); + final nn = [], t = []; + var clock = 0.0; + while (clock < hours * 3.6e6) { + final v = baseMs + + driftMs * clock / (hours * 3.6e6) + + breathMs * math.sin(2 * math.pi * breathHz * clock / 1000) + + jitterMs * (r.nextDouble() - .5); + clock += v; + nn.add(v); + t.add(clock); + } + return (nn: nn, t: t); + } + + for (final (name, x, note) in [ + ( + 'a respiratory line exempts the night', + breathing( + hours: 1, baseMs: 1350, driftMs: 80, breathMs: 40, breathHz: .3, + jitterMs: 3), + 'PRV not ECG-HRV; RMSSD/pNN50', + ), + ( + 'a refused night publishes from a breathing line steady in Hz', + breathing( + hours: 2.5, baseMs: 900, driftMs: 400, breathMs: 10, breathHz: .25, + jitterMs: 40), + 'breathing line steady in Hz', + ), + ]) { + test('HRV below the ACF floor: $name', () { + final whole = hrvTime(x.nn, nnTimesMs: x.t); + expect(whole.value!.diffAcf1, lessThan(kNnDiffAcf1Floor), + reason: 'the fixture is under the floor'); + expect(whole.value!.rmssd, isNotNull, reason: 'and the batch publishes'); + expect(whole.note, contains(note), reason: 'by the path named'); + var state = IncrementalHrvTime(); + for (final n in prefixSizes(x.nn.length, 701)) { + final nn = x.nn.sublist(0, n), t = x.t.sublist(0, n); + hrvClose(state.sync(nn, nnTimesMs: t), hrvTime(nn, nnTimesMs: t)); + state = IncrementalHrvTime.fromJson(checkpoint(state.toJson())); + } + }); + } + test('HRV real capture chunk parity includes cleaned beat times', () { + final f = realIncrementalFixture(); + expect(f.nn.length, greaterThan(300)); + final state = IncrementalHrvTime(); + for (final n in prefixSizes(f.nn.length, 149)) { + final nn = f.nn.sublist(0, n), t = f.times.sublist(0, n); + hrvClose(state.sync(nn, nnTimesMs: t), hrvTime(nn, nnTimesMs: t)); + } + }); +} diff --git a/test/onehz/incremental/int_histogram_test.dart b/test/onehz/incremental/int_histogram_test.dart new file mode 100644 index 0000000..b9f2f8c --- /dev/null +++ b/test/onehz/incremental/int_histogram_test.dart @@ -0,0 +1,210 @@ +import 'dart:io'; +import 'dart:math' as math; + +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; +import '../support/incremental_compare.dart'; + +/// Exact equality, not a tolerance: the histogram must return the very double +/// `percentileSorted` returns for the expanded sorted list. +void _same(IntHistogram h, List values, {String? reason}) { + final sorted = [...values]..sort(); + expect(h.count, values.length, reason: reason); + for (final p in [0, 1, 5, 25, 30, 33.3, 50, 66.6, 75, 90, 95, 99, 100]) { + final expected = percentileSorted(sorted, p.toDouble()); + final actual = h.percentile(p.toDouble()); + expect(actual, expected, reason: '$reason p=$p'); + } + expect(h.median, percentileSorted(sorted, 50), reason: reason); + expect(h.median, median(values), reason: reason); +} + +List _realNightHr() => [ + for (final line in File('test/onehz/fixtures/real_night_2026_07_onehz.csv') + .readAsLinesSync() + .skip(1)) + if (line.trim().isNotEmpty) double.parse(line.split(',')[1]) + ]; + +void main() { + test('real overnight HR: exact parity at every prefix and few distinct values', + () { + final hr = _realNightHr(); + expect(hr.length, greaterThan(20000)); + final h = IntHistogram(); + for (var i = 0; i < hr.length; i++) { + h.add(hr[i].toInt()); + if (i % 997 == 0 || i == hr.length - 1) { + _same(h, hr.sublist(0, i + 1), reason: 'prefix ${i + 1}'); + } + } + expect(h.distinct, lessThan(80), reason: 'compaction is the point'); + }); + + test('real overnight HR: sliding window add/remove stays exact', () { + final hr = _realNightHr(); + const window = 1800; + final h = IntHistogram(); + for (var i = 0; i < 6000; i++) { + h.add(hr[i].toInt()); + if (i >= window) h.remove(hr[i - window].toInt()); + if (i % 251 == 0) { + _same(h, hr.sublist(math.max(0, i + 1 - window), i + 1), + reason: 'window ending $i'); + } + } + }); + + for (final seed in [1, 7, 42, 914]) { + test('randomized integer data, add/remove sequence seed=$seed', () { + final rnd = math.Random(seed); + final h = IntHistogram(); + final values = []; + for (var step = 0; step < 1500; step++) { + if (values.isNotEmpty && rnd.nextInt(10) < 4) { + final v = values.removeAt(rnd.nextInt(values.length)); + h.remove(v.toInt()); + } else { + // Includes negatives and a wide spread, not just heart rates. + final v = rnd.nextInt(400) - 100; + values.add(v.toDouble()); + h.add(v); + } + if (step % 7 == 0 || values.length < 6) { + _same(h, values, reason: 'step $step n=${values.length}'); + } + } + while (values.isNotEmpty) { + h.remove(values.removeLast().toInt()); + } + _same(h, values); + }); + } + + test('odd and even counts, single value, all-equal values', () { + for (final values in [ + [5.0], + [5.0, 5.0], + [4.0, 9.0], + [3.0, 1.0, 2.0], + [3.0, 1.0, 2.0, 10.0], + List.filled(7, 61.0), + [60.0, 60.0, 61.0], + ]) { + final h = IntHistogram(); + for (final v in values) { + h.add(v.toInt()); + } + _same(h, values, reason: '$values'); + } + }); + + test('p=0 and p=1 hit the extremes; p=100 is not a special case', () { + final h = IntHistogram(); + final values = [72.0, 55.0, 90.0, 61.0, 61.0]; + for (final v in values) { + h.add(v.toInt()); + } + expect(h.percentile(0), 55); + expect(h.percentile(100), 90); + expect(h.percentile(1), percentileSorted([...values]..sort(), 1)); + expect(h.percentile(0.5), percentileSorted([...values]..sort(), 0.5)); + }); + + test('empty histogram abstains with null, as the batch functions do', () { + final h = IntHistogram(); + expect(h.count, 0); + expect(h.distinct, 0); + expect(h.median, isNull); + expect(h.median, median(const [])); + expect(h.percentile(0), isNull); + expect(h.percentile(100), isNull); + expect(h.percentile(50), percentile(const [], 50)); + h.add(70); + h.remove(70); + expect(h.median, isNull, reason: 'emptied again'); + expect(h.distinct, 0, reason: 'no zero-count bins are retained'); + }); + + test('add with a repeat count, and removing one of several', () { + final h = IntHistogram() + ..add(60, 3) + ..add(70, 2); + _same(h, [60, 60, 60, 70, 70]); + h.remove(60); + _same(h, [60, 60, 70, 70]); + expect(h.distinct, 2); + }); + + test('removing a value that is not present throws and changes nothing', () { + final h = IntHistogram()..add(60); + expect(() => h.remove(61), throwsStateError); + expect(() => IntHistogram().remove(1), throwsStateError); + _same(h, [60]); + }); + + test('rejects values that are not whole numbers and bad arguments', () { + final h = IntHistogram(); + expect(() => h.add(double.nan), throwsArgumentError); + expect(() => h.add(double.infinity), throwsArgumentError); + expect(() => h.add(70.5), throwsArgumentError); + expect(() => h.add(70, 0), throwsArgumentError); + expect(() => h.add(70, -1), throwsArgumentError); + h.add(70.0); // a whole-number double is a whole number + expect(h.count, 1); + expect(() => h.percentile(-1), throwsArgumentError); + expect(() => h.percentile(100.5), throwsArgumentError); + expect(() => h.percentile(double.nan), throwsArgumentError); + }); + + for (final seed in [3, 11]) { + test('merge equals one histogram over the concatenation seed=$seed', () { + final rnd = math.Random(seed); + final a = [for (var i = 0; i < 300; i++) rnd.nextInt(60) + 40.0]; + final b = [for (var i = 0; i < 211; i++) rnd.nextInt(90) + 30.0]; + final ha = IntHistogram(), hb = IntHistogram(); + for (final v in a) { + ha.add(v.toInt()); + } + for (final v in b) { + hb.add(v.toInt()); + } + ha.merge(hb); + _same(ha, [...a, ...b]); + _same(hb, b, reason: 'the argument is left unchanged'); + ha.merge(IntHistogram()); + _same(ha, [...a, ...b], reason: 'merging empty is a no-op'); + final empty = IntHistogram()..merge(hb); + _same(empty, b, reason: 'merging into empty'); + empty.merge(empty); + _same(empty, [...b, ...b], reason: 'merging with itself doubles counts'); + }); + } + + test('checkpoint round-trips and continues; malformed input throws', () { + final h = IntHistogram(); + final values = []; + for (final v in [70, 70, 64, 91, 64, 64, 120]) { + h.add(v); + values.add(v.toDouble()); + } + final back = IntHistogram.fromJson(checkpoint(h.toJson())); + _same(back, values); + back.add(55); + values.add(55); + _same(back, values); + for (final bad in >[ + {}, + {'version': 2, 'type': 'IntHistogram', 'values': [], 'counts': []}, + {'version': 1, 'type': 'RunningMoments', 'values': [], 'counts': []}, + {'version': 1, 'type': 'IntHistogram', 'values': [1, 2], 'counts': [1]}, + {'version': 1, 'type': 'IntHistogram', 'values': [2, 1], 'counts': [1, 1]}, + {'version': 1, 'type': 'IntHistogram', 'values': [1, 1], 'counts': [1, 1]}, + {'version': 1, 'type': 'IntHistogram', 'values': [1], 'counts': [0]}, + {'version': 1, 'type': 'IntHistogram', 'values': [1.5], 'counts': [1]}, + ]) { + expect(() => IntHistogram.fromJson(bad), throwsFormatException, + reason: '$bad'); + } + }); +} diff --git a/test/onehz/incremental/irregular_diagnostics_stream_test.dart b/test/onehz/incremental/irregular_diagnostics_stream_test.dart new file mode 100644 index 0000000..7839b2d --- /dev/null +++ b/test/onehz/incremental/irregular_diagnostics_stream_test.dart @@ -0,0 +1,398 @@ +// Contract: the screen's DIAGNOSTICS are the same whether the beats arrive in +// one batch call or in chunks through `RrCorrector` + `IrregularScreenState`, +// across save/restore (design 04 "PRV diagnostics", item b). +// +// Oracle = `irregularBeatScreenDetailed` over `correctRr(prefix)`, with the +// corrector's own counts. At every cut, the streaming side is +// corrector.snapshot() -> state.evaluateDetailed(tail, cleaning: counts) +// and its diagnostics must equal the batch's JSON text, byte for byte: beat +// counts, corrected / dropped, artifact fraction, window counts, the final OPEN +// window, thresholds. Cuts are placed to split a 5-minute window, an isolated +// artifact (a spline correction that cannot settle until two normals follow it) +// and a multi-beat noise run (a drop). +// +// RED until implemented: `evaluateDetailed` / `irregularBeatScreenDetailed` +// are stubs that throw UnimplementedError. +import 'dart:convert'; +import 'dart:math' as math; + +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +import '../support/rr_chunking.dart'; + +String _json(Object? o) => jsonEncode(o); + +/// A day of beats with known artifacts: calm sinus with AF bursts, isolated +/// early beats (+ compensatory pause) at [isolated], a 6-beat noise run at each +/// of [runs], and one 40 s dropout. Stamps are end-of-beat epoch ms. +({List rr, List ts, List isolated, List runs}) _day( + int seed, + {int beats = 9000}) { + final r = math.Random(seed); + final rr = [], ts = []; + final isolated = [], runs = []; + var clock = 1.76e12; // epoch ms + var af = false; + var left = 0; + var droppedOut = false; + void emit(double v) { + clock += v; + rr.add(v); + ts.add(clock.floorToDouble()); + } + + while (rr.length < beats) { + if (--left <= 0) { + af = r.nextDouble() < 0.4; + left = 300 + r.nextInt(900); + } + final i = rr.length; + if (!droppedOut && i >= 4000) { + droppedOut = true; + clock += 40000; // a sensor dropout + } + if (i > 100 && i % 523 == 7) { + isolated.add(i); + emit(380); // early beat + emit(1450); // compensatory pause + continue; + } + if (i > 100 && i % 811 == 400) { + runs.add(i); + for (var k = 0; k < 6; k++) { + emit(250 + r.nextInt(1800).toDouble()); + } + continue; + } + emit(af + ? (420 + r.nextInt(700)).toDouble() + : (850 + 40 * math.sin(i / 70) + 12 * (r.nextDouble() - .5)) + .roundToDouble()); + } + return (rr: rr, ts: ts, isolated: isolated, runs: runs); +} + +IrregularScreenResult _batch(List rr, List ts, int n) { + final c = correctRr(rr.sublist(0, n), rrTsMs: ts.sublist(0, n)); + return irregularBeatScreenDetailed( + c.nn, + nnTimesMs: c.nnTimesMs, + artifactFraction: (1.0 - c.cleanFraction).clamp(0.0, 1.0), + cleaning: RrCleaningCounts( + raw: n, corrected: c.correctedCount, dropped: c.droppedCount), + ); +} + +IrregularScreenResult _stream(RrCorrector c, IrregularScreenState st) { + final s = c.snapshot(); + return st.evaluateDetailed( + s.tailNn, + s.tailNnTimes, + artifactFraction: (1.0 - s.cleanFraction).clamp(0.0, 1.0), + cleaning: RrCleaningCounts( + raw: s.n, corrected: s.correctedCount, dropped: s.droppedCount), + ); +} + +void _same(IrregularScreenResult got, IrregularScreenResult want, String why) { + expect(_json(got.diagnostics.toJson()), _json(want.diagnostics.toJson()), + reason: 'diagnostics $why'); + expect(got.metric.present, want.metric.present, reason: 'present $why'); + expect(got.metric.note, want.metric.note, reason: 'note $why'); + if (want.metric.present) { + expect(got.metric.value!.flag, want.metric.value!.flag, reason: 'flag $why'); + expect(got.metric.value!.nBeats, want.metric.value!.nBeats, + reason: 'nBeats $why'); + } +} + +/// Cut points: random chunking plus neighbourhoods of every artifact and of +/// every 5-minute beat-clock edge. +List _cuts( + ({List rr, List ts, List isolated, List runs}) d, + int seed) { + final n = d.rr.length; + final cuts = {...randomCuts(math.Random(seed), n, sizes: [90, 500, 1500])}; + for (final k in d.isolated) { + for (var o = -1; o <= 4; o++) { + cuts.add(k + o); + } + } + for (final m in d.runs) { + for (var o = 0; o <= 9; o++) { + cuts.add(m + o); + } + } + var next = d.ts.first + 300000; + for (var i = 0; i < n; i++) { + if (d.ts[i] >= next) { + cuts..add(i - 1)..add(i)..add(i + 1); + next += 300000; + } + } + return [ + for (final c in cuts) + if (c > 0 && c <= n) c + ]..sort(); +} + +void main() { + group('corrector + screen state == batch, at every cut', () { + for (final restart in [false, true]) { + test('save/restore between chunks: $restart', () { + final d = _day(restart ? 2 : 1); + final c0 = correctRr(d.rr, rrTsMs: d.ts); + expect(c0.correctedCount, greaterThan(5), reason: 'fixture corrects'); + expect(c0.droppedCount, greaterThan(10), reason: 'fixture drops'); + + var corr = RrCorrector(); + var st = IrregularScreenState(); + var at = 0; + final opens = {}; + var flaggedSeen = 0, presentSeen = 0, abstainedSeen = 0; + for (final cut in _cuts(d, 4)) { + final settled = + corr.fold(d.rr.sublist(at, cut), tsMs: d.ts.sublist(at, cut)); + st.fold(settled.nn, settled.nnTimes); + at = cut; + if (restart) { + corr = RrCorrector.fromJson(jsonDecode(jsonEncode(corr.toJson())) + as Map); + st = IrregularScreenState.fromJson( + jsonDecode(jsonEncode(st.toJson())) as Map); + } + final got = _stream(corr, st); + _same(got, _batch(d.rr, d.ts, cut), 'cut=$cut restart=$restart'); + final w = got.diagnostics.windows; + if (w != null) opens.add(w.open); + got.metric.present ? presentSeen++ : abstainedSeen++; + if (got.diagnostics.windows != null && + got.diagnostics.windows!.flagged > 0) { + flaggedSeen++; + } + } + expect(presentSeen, greaterThan(20), reason: 'the screen really ran'); + expect(abstainedSeen, greaterThan(0), reason: 'and abstained early on'); + expect(flaggedSeen, greaterThan(5), reason: 'windows really flagged'); + expect(opens, containsAll([IrregularOpenWindow.thin]), + reason: 'cuts landed in a window that had not yet filled'); + expect( + opens.contains(IrregularOpenWindow.unflagged) || + opens.contains(IrregularOpenWindow.flagged), + isTrue, + reason: 'and in valid open windows'); + final last = _stream(corr, st); + expect(last.diagnostics.rrRaw, d.rr.length); + expect(last.diagnostics.corrected, c0.correctedCount); + expect(last.diagnostics.dropped, c0.droppedCount); + }); + } + + test('evaluateDetailed does not change the state', () { + final d = _day(3, beats: 3000); + final corr = RrCorrector(); + final st = IrregularScreenState(); + final settled = corr.fold(d.rr, tsMs: d.ts); + st.fold(settled.nn, settled.nnTimes); + final before = jsonEncode(st.toJson()); + final a = _stream(corr, st); + final b = _stream(corr, st); + expect(jsonEncode(st.toJson()), before); + expect(_json(a.diagnostics.toJson()), _json(b.diagnostics.toJson())); + }); + + test('one fold of the whole day == a hundred small ones', () { + final d = _day(5, beats: 4000); + final one = RrCorrector(), many = RrCorrector(); + final s1 = IrregularScreenState(), s2 = IrregularScreenState(); + final r1 = one.fold(d.rr, tsMs: d.ts); + s1.fold(r1.nn, r1.nnTimes); + for (var i = 0; i < d.rr.length; i += 40) { + final to = math.min(d.rr.length, i + 40); + final r = many.fold(d.rr.sublist(i, to), tsMs: d.ts.sublist(i, to)); + s2.fold(r.nn, r.nnTimes); + } + _same(_stream(many, s2), _stream(one, s1), 'chunking'); + _same(_stream(one, s1), _batch(d.rr, d.ts, d.rr.length), 'vs batch'); + }); + }); + + group('folding a corrected NN series directly (no corrector)', () { + ({List nn, List t}) fixture(int seed, {double salt = 0.03}) { + final r = math.Random(seed); + final nn = [], t = []; + var clock = 0.0; + var af = false; + var left = 0; + while (clock < 4 * 3600 * 1000.0) { + if (--left <= 0) { + af = r.nextDouble() < 0.5; + left = 200 + r.nextInt(1500); + } + var v = af + ? (420 + r.nextInt(700)).toDouble() + : (850 + 40 * math.sin(clock / 9000)).roundToDouble(); + final u = r.nextDouble(); + if (u < salt / 3) { + v = 250; + } else if (u < 2 * salt / 3) { + v = 2500; + } else if (u < salt) { + v = double.nan; + } + clock += v.isFinite ? v : 800; + nn.add(v); + t.add(clock); + } + return (nn: nn, t: t); + } + + IrregularScreenResult batch(List nn, List t, int n, + {double af = 0, + int minBeats = irregularScreenMinBeats, + double maxArtifact = 0.30}) => + irregularBeatScreenDetailed(nn.sublist(0, n), + nnTimesMs: t.sublist(0, n), + artifactFraction: af, + minBeats: minBeats, + maxArtifact: maxArtifact); + + for (final restart in [false, true]) { + test('out-of-range and NaN beats are counted in nnIn, not nnKept; ' + 'restore: $restart', () { + final x = fixture(8); + var st = IrregularScreenState(); + var at = 0; + var sawGap = false; + for (final cut + in randomCuts(math.Random(9), x.nn.length, sizes: [7, 90, 700, 3000])) { + st.fold(x.nn.sublist(at, cut), x.t.sublist(at, cut)); + at = cut; + if (restart) { + st = IrregularScreenState.fromJson( + jsonDecode(jsonEncode(st.toJson())) as Map); + } + for (final af in [0.0, 0.31]) { + final got = st.evaluateDetailed(const [], const [], artifactFraction: af); + _same(got, batch(x.nn, x.t, cut, af: af), 'cut=$cut af=$af'); + if (got.diagnostics.nnIn > got.diagnostics.nnKept) sawGap = true; + } + } + expect(sawGap, isTrue, reason: 'fixture has beats the filter removes'); + }); + } + + test('every prefix 0..700, one beat at a time: abstention reasons and ' + 'counts match the batch the whole way', () { + final x = fixture(12, salt: 0.0); + final st = IrregularScreenState(); + final reasons = {}; + for (var i = 0; i < 700; i++) { + st.fold([x.nn[i]], [x.t[i]]); + final got = st.evaluateDetailed(const [], const []); + _same(got, batch(x.nn, x.t, i + 1), 'n=${i + 1}'); + reasons.add(got.diagnostics.abstain); + } + expect(reasons, containsAll([IrregularAbstain.tooFewBeats, null]), + reason: 'thin first, then the screen runs'); + }); + + test('a smaller beat floor is a threshold the diagnostics report', () { + final x = fixture(13, salt: 0.0); + final st = IrregularScreenState() + ..fold(x.nn.sublist(0, 300), x.t.sublist(0, 300)); + final got = + st.evaluateDetailed(const [], const [], minBeats: 100, maxArtifact: 0.2); + expect(got.diagnostics.thresholds.minBeats, 100); + expect(got.diagnostics.thresholds.maxArtifact, 0.2); + _same(got, batch(x.nn, x.t, 300, minBeats: 100, maxArtifact: 0.2), + 'minBeats 100'); + }); + + test('tail: evaluateDetailed(tail) == batch(settled ++ tail), state ' + 'untouched', () { + final x = fixture(31); + final st = IrregularScreenState(); + const settled = 2500; + st.fold(x.nn.sublist(0, settled), x.t.sublist(0, settled)); + final before = jsonEncode(st.toJson()); + for (final tail in [0, 1, 50, 399]) { + final to = settled + tail; + _same( + st.evaluateDetailed(x.nn.sublist(settled, to), x.t.sublist(settled, to)), + batch(x.nn, x.t, to), + 'tail=$tail'); + expect(jsonEncode(st.toJson()), before); + } + }); + }); + + group('old checkpoints', () { + test('a version 1 checkpoint (no input-beat or window-total counters) is ' + 'REFUSED, never restored with counts it did not keep', () { + final st = IrregularScreenState() + ..fold([for (var i = 0; i < 80; i++) 800.0 + i % 9], [ + for (var i = 0; i < 80; i++) 800.0 * (i + 1) + ]); + final v1 = jsonDecode(jsonEncode(st.toJson())) as Map + ..['version'] = 1 + ..remove('nIn') + ..remove('total'); + expect(() => IrregularScreenState.fromJson(v1), throwsFormatException); + // The current format still restores. + final now = IrregularScreenState.fromJson( + jsonDecode(jsonEncode(st.toJson())) as Map); + expect(now.evaluateDetailed(const [], const []).diagnostics.nnIn, 80); + expect(st.toJson()['version'], 3); + }); + + test('a version 2 checkpoint missing a counter is refused too', () { + final j = jsonDecode(jsonEncode(IrregularScreenState().toJson())) + as Map + ..remove('nIn'); + expect(() => IrregularScreenState.fromJson(j), throwsFormatException); + }); + }); + + group('absent input stays absent', () { + test('a fresh state: absent, zero counts, nothing invented', () { + final got = IrregularScreenState().evaluateDetailed(const [], const []); + _same(got, irregularBeatScreenDetailed(const [], nnTimesMs: const []), + 'empty'); + expect(got.metric.present, isFalse); + expect(got.diagnostics.nnIn, 0); + expect(got.diagnostics.windows!.total, 0); + expect(got.diagnostics.windows!.sustainedObserved, isNull); + expect(got.diagnostics.rrRaw, isNull, + reason: 'no cleaning counts handed in: unknown, not 0'); + }); + + test('only unusable beats folded: counted as input, none kept, no window', + () { + final st = IrregularScreenState() + ..fold([double.nan, 100, 5000, double.nan], [1, 2, 3, 4]); + final got = st.evaluateDetailed(const [], const []); + expect(got.diagnostics.nnIn, 4); + expect(got.diagnostics.nnKept, 0); + expect(got.diagnostics.windows!.total, 0); + expect(got.diagnostics.windows!.open, IrregularOpenWindow.none); + expect(got.metric.present, isFalse); + }); + + test('a window config that fails closed: windows null, same as the batch', + () { + final r = math.Random(4); + final nn = [for (var i = 0; i < 1200; i++) 400.0 + 100 * r.nextInt(13)]; + var clock = 0.0; + final t = [for (final v in nn) clock += v]; + final st = IrregularScreenState(windowMinutes: 0)..fold(nn, t); + final got = st.evaluateDetailed(const [], const []); + expect(got.diagnostics.windows, isNull); + _same( + got, + irregularBeatScreenDetailed(nn, nnTimesMs: t, windowMinutes: 0), + 'windowMinutes 0'); + }); + }); +} diff --git a/test/onehz/incremental/irregular_screen_state_test.dart b/test/onehz/incremental/irregular_screen_state_test.dart new file mode 100644 index 0000000..6b118f4 --- /dev/null +++ b/test/onehz/incremental/irregular_screen_state_test.dart @@ -0,0 +1,495 @@ +// Contract for `IrregularScreenState` (clinical/irregular_rhythm_state). +// +// Oracle = `irregularBeatScreen` on the same NN series. The state folds the NN +// that `RrCorrector` settles and is evaluated with the corrector's provisional +// tail; the Metric must match the batch call on settled ++ tail at every prefix +// and under any chunking / save-restore: +// * present, note, tier, inputs, flag, nBeats, pNN%: identical; +// * SD1 / SD2 / ratio / confidence: equal to 1e-9 relative (the state uses a +// running variance, the batch a two-pass one); +// * ABSENT exactly where the batch is absent, with the same note — a thin, +// noisy, flat or gappy series never gets a made-up verdict. +// +// RED until the state is implemented: every test fails with UnimplementedError +// from the stub. +import 'dart:convert'; +import 'dart:math' as math; + +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +import '../support/correct_rr_reference.dart'; +import '../support/rr_chunking.dart'; +import '../support/rr_synth.dart'; + +void _close(double got, double want, String why, {double rel = 1e-9}) { + final tol = rel * math.max(1.0, want.abs()); + expect((got - want).abs(), lessThanOrEqualTo(tol), + reason: '$why got=$got want=$want'); +} + +void _expectSameScreen(Metric got, + Metric want, String why) { + expect(got.present, want.present, reason: 'present $why'); + expect(got.note, want.note, reason: 'note $why'); + expect(got.tier, want.tier, reason: 'tier $why'); + expect(got.inputs_used, want.inputs_used, reason: 'inputs $why'); + if (!want.present) { + expect(got.value, isNull, reason: 'absent => no value $why'); + expect(got.confidence, 0, reason: 'absent => confidence 0 $why'); + return; + } + final g = got.value!, w = want.value!; + expect(g.flag, w.flag, reason: 'flag $why'); + expect(g.nBeats, w.nBeats, reason: 'nBeats $why'); + expect(g.pnnPct, w.pnnPct, reason: 'pnn $why'); + _close(g.sd1, w.sd1, 'sd1 $why'); + _close(g.sd2, w.sd2, 'sd2 $why'); + _close(g.sd1sd2, w.sd1sd2, 'sd1sd2 $why'); + _close(got.confidence, want.confidence, 'confidence $why', rel: 1e-12); +} + +/// AF-like bursts (irregularly irregular RR) in [afShare] of 5..30 min blocks, +/// otherwise sinus with slow drift. Optionally salted with out-of-range beats. +({List nn, List t}) _fixture(int seed, double afShare, + {double hours = 8, double outOfRange = 0}) { + final r = math.Random(seed); + final nn = [], t = []; + var clock = 0.0; + var af = false; + var left = 0; + while (clock < hours * 3600 * 1000.0) { + if (--left <= 0) { + af = r.nextDouble() < afShare; + left = 300 + r.nextInt(2000); + } + var v = af + ? (420 + r.nextInt(700)).toDouble() + : (850 + 40 * math.sin(clock / 9000) + 25 * (r.nextDouble() - .5)) + .roundToDouble(); + if (r.nextDouble() < outOfRange) v = r.nextBool() ? 2500 : 250; + clock += v; + nn.add(v); + t.add(clock); + } + return (nn: nn, t: t); +} + +Metric _batch(List nn, List t, int n, + {double af = 0.0, + int minBeats = irregularScreenMinBeats, + double maxArtifact = 0.30, + double sd1sd2Flag = 0.70, + double pnnThresholdMs = 70, + double pnnFlagPct = 30, + double windowMinutes = 5, + int minWindowBeats = 40, + double sustainedFraction = 0.5}) => + irregularBeatScreen(nn.sublist(0, n), + nnTimesMs: t.sublist(0, n), + artifactFraction: af, + minBeats: minBeats, + maxArtifact: maxArtifact, + sd1sd2Flag: sd1sd2Flag, + pnnThresholdMs: pnnThresholdMs, + pnnFlagPct: pnnFlagPct, + windowMinutes: windowMinutes, + minWindowBeats: minWindowBeats, + sustainedFraction: sustainedFraction); + +void main() { + group('folding a corrected NN series directly', () { + for (final share in [0.0, 0.35, 0.75, 1.0]) { + for (final restart in [false, true]) { + test('af share=$share, save/restore between chunks: $restart', () { + final x = _fixture((share * 100).round() + 3, share); + var st = IrregularScreenState(); + var at = 0; + var flags = 0, checks = 0; + final cuts = randomCuts(math.Random(5), x.nn.length, + sizes: [90, 500, 4000]); + for (final cut in cuts) { + st.fold(x.nn.sublist(at, cut), x.t.sublist(at, cut)); + at = cut; + if (restart) { + st = IrregularScreenState.fromJson(jsonRoundTrip(st.toJson())); + } + // 0.31 is over the 0.30 suppression line: must be absent. + for (final af in [0.0, 0.31]) { + final want = _batch(x.nn, x.t, cut, af: af); + final got = + st.evaluate(const [], const [], artifactFraction: af); + _expectSameScreen(got, want, 'cut=$cut af=$af'); + if (want.present) { + checks++; + if (want.value!.flag) flags++; + } + } + } + expect(checks, greaterThan(5), reason: 'screen was really evaluated'); + // ignore: avoid_print + print('irregular share=$share: checks=$checks flagged=$flags'); + if (share >= 0.75) { + expect(flags, greaterThan(0), reason: 'a flagged day is flagged'); + } + if (share == 0.0) expect(flags, 0); + }); + } + } + + test('out-of-range beats are skipped without bridging the gap', () { + // 2% of beats are 250 / 2500 ms. The batch never takes a difference + // ACROSS a skipped beat; the state must not either. + final x = _fixture(77, 0.5, hours: 4, outOfRange: 0.02); + var st = IrregularScreenState(); + var at = 0; + for (final cut in randomCuts(math.Random(6), x.nn.length, + sizes: [1, 7, 90, 700])) { + st.fold(x.nn.sublist(at, cut), x.t.sublist(at, cut)); + at = cut; + if (cut % 3 == 0) { + st = IrregularScreenState.fromJson(jsonRoundTrip(st.toJson())); + } + _expectSameScreen(st.evaluate(const [], const []), + _batch(x.nn, x.t, cut), 'cut=$cut'); + } + }); + + test('a window edge hit EXACTLY decides the verdict (>= closes the window)', + () { + // Window 1: 300 s of AF-like beats whose last interval lands the next + // beat on windowStart + 300000 EXACTLY. Window 2: calm, same trick. + // Window 3: calm, exactly minWindowBeats (40) beats, so it only counts + // as a window when the edge beat OPENS it. Closing on > instead of >= + // moves that beat into window 2, window 3 falls to 39 beats and stops + // counting, and the sustained fraction jumps from 1/3 to 1/2 (flag). + final r = math.Random(3); + final nn = [], t = []; + var clock = 0.0; + void add(double v) { + clock += v; + nn.add(v); + t.add(clock); + } + + void landOn(double target, double Function() next) { + while (target - clock > 2200) { + add(next()); + } + var rest = target - clock; + if (rest < 300) { + clock -= nn.removeLast(); + t.removeLast(); + rest = target - clock; + } + if (rest > 2000) { + add(rest / 2); + add(rest / 2); + } else { + add(rest); + } + } + + add(1000); // the first beat opens window 1 at t = 1000 + landOn(301000, () => 400.0 + 100 * r.nextInt(13)); // AF-like + double calm() => 800.0 + 10 * (r.nextInt(5) - 2); + landOn(601000, calm); + for (var i = 0; i < 39; i++) { + add(calm()); + } + expect(t.where((x) => x == 301000 || x == 601000).length, 2); + final want = _batch(nn, t, nn.length, minBeats: 500); + expect(want.present, isTrue, reason: 'sanity: crafted day is evaluable'); + expect(want.value!.flag, isFalse, reason: 'sanity: 1 of 3 windows flags'); + final st = IrregularScreenState(); + st.fold(nn, t); + _expectSameScreen(st.evaluate(const [], const []), want, 'edge day'); + // and split anywhere, with a restore, including right at the edge beats + for (final cut in [ + nn.length ~/ 2, + t.indexOf(301000), + t.indexOf(601000), + t.indexOf(601000) + 1 + ]) { + var a = IrregularScreenState() + ..fold(nn.sublist(0, cut), t.sublist(0, cut)); + a = IrregularScreenState.fromJson(jsonRoundTrip(a.toJson())); + a.fold(nn.sublist(cut), t.sublist(cut)); + _expectSameScreen(a.evaluate(const [], const []), want, 'cut=$cut'); + } + }); + + test('provisional tail: evaluate(tail) == batch(settled ++ tail), and ' + 'leaves the state untouched', () { + final x = _fixture(31, 0.75, hours: 3); + final st = IrregularScreenState(); + final settled = x.nn.length - 400; + st.fold(x.nn.sublist(0, settled), x.t.sublist(0, settled)); + final before = jsonEncode(st.toJson()); + for (final tailLen in [0, 1, 50, 399, 400]) { + final to = settled + tailLen; + final got = st.evaluate(x.nn.sublist(settled, to), x.t.sublist(settled, to)); + _expectSameScreen(got, _batch(x.nn, x.t, to), 'tail=$tailLen'); + expect(jsonEncode(st.toJson()), before, + reason: 'evaluate must not fold the tail into the state'); + } + // Fold the tail for real afterwards: still the batch answer. + st.fold(x.nn.sublist(settled), x.t.sublist(settled)); + _expectSameScreen(st.evaluate(const [], const []), + _batch(x.nn, x.t, x.nn.length), 'tail folded'); + }); + }); + + group('absent where the batch is absent (never fabricated)', () { + test('every prefix 0..700, one beat at a time: thin, then present', () { + final x = _fixture(12, 0.6, hours: 1); + final st = IrregularScreenState(); + var present = 0, absent = 0; + for (var i = 0; i < 700; i++) { + st.fold([x.nn[i]], [x.t[i]]); + final want = _batch(x.nn, x.t, i + 1); + final got = st.evaluate(const [], const []); + _expectSameScreen(got, want, 'n=${i + 1}'); + want.present ? present++ : absent++; + } + expect(absent, greaterThan(400), reason: 'thin prefixes were exercised'); + expect(present, greaterThan(100), reason: 'and then it speaks'); + }); + + test('a smaller beat floor and every prefix 0..120', () { + final x = _fixture(13, 1.0, hours: 1); + final st = IrregularScreenState(); + for (var i = 0; i < 120; i++) { + st.fold([x.nn[i]], [x.t[i]]); + _expectSameScreen( + st.evaluate(const [], const [], minBeats: 20), + _batch(x.nn, x.t, i + 1, minBeats: 20), + 'minBeats=20 n=${i + 1}'); + } + }); + + test('artifact fraction at / over the line, absent with the same note', () { + final x = _fixture(14, 0.5, hours: 2); + final st = IrregularScreenState()..fold(x.nn, x.t); + for (final af in [0.0, 0.29, 0.30, 0.3000001, 0.31, 0.5, 1.0]) { + _expectSameScreen(st.evaluate(const [], const [], artifactFraction: af), + _batch(x.nn, x.t, x.nn.length, af: af), 'af=$af'); + } + for (final cap in [0.1, 0.5]) { + _expectSameScreen( + st.evaluate(const [], const [], + artifactFraction: 0.2, maxArtifact: cap), + _batch(x.nn, x.t, x.nn.length, af: 0.2, maxArtifact: cap), + 'maxArtifact=$cap'); + } + }); + + test('flat series: SD2 = 0 is "undefined", not "perfectly regular"', () { + final nn = List.filled(900, 800); + final t = [for (var i = 1; i <= 900; i++) i * 800.0]; + final got = (IrregularScreenState()..fold(nn, t)) + .evaluate(const [], const []); + final want = _batch(nn, t, 900); + expect(want.present, isFalse, reason: 'the oracle abstains'); + _expectSameScreen(got, want, 'flat'); + }); + + test('no two adjacent clean beats: nothing to build a Poincare plot from', + () { + // every other beat out of range: 300 clean beats, zero adjacent pairs. + final nn = [for (var i = 0; i < 600; i++) i.isEven ? 800.0 : 250.0]; + final t = [for (var i = 1; i <= 600; i++) i * 800.0]; + final got = (IrregularScreenState()..fold(nn, t)) + .evaluate(const [], const [], minBeats: 5); + final want = _batch(nn, t, 600, minBeats: 5); + expect(want.present, isFalse); + _expectSameScreen(got, want, 'no adjacent pairs'); + }); + + test('empty state', () { + _expectSameScreen(IrregularScreenState().evaluate(const [], const []), + _batch(const [], const [], 0), 'empty'); + }); + }); + + group('parameters', () { + test('non-default thresholds and windows, with restore', () { + final x = _fixture(21, 0.4, hours: 3); + final params = ( + sd1sd2Flag: 0.5, + pnnThresholdMs: 50.0, + pnnFlagPct: 20.0, + windowMinutes: 2.0, + minWindowBeats: 20, + sustainedFraction: 0.3 + ); + var st = IrregularScreenState( + sd1sd2Flag: params.sd1sd2Flag, + pnnThresholdMs: params.pnnThresholdMs, + pnnFlagPct: params.pnnFlagPct, + windowMinutes: params.windowMinutes, + minWindowBeats: params.minWindowBeats, + sustainedFraction: params.sustainedFraction); + var at = 0; + for (final cut in randomCuts(math.Random(8), x.nn.length, + sizes: [100, 900, 3000])) { + st.fold(x.nn.sublist(at, cut), x.t.sublist(at, cut)); + at = cut; + st = IrregularScreenState.fromJson(jsonRoundTrip(st.toJson())); + _expectSameScreen( + st.evaluate(const [], const []), + _batch(x.nn, x.t, cut, + sd1sd2Flag: params.sd1sd2Flag, + pnnThresholdMs: params.pnnThresholdMs, + pnnFlagPct: params.pnnFlagPct, + windowMinutes: params.windowMinutes, + minWindowBeats: params.minWindowBeats, + sustainedFraction: params.sustainedFraction), + 'cut=$cut'); + } + // The parameters travel in the checkpoint. + expect(st.sd1sd2Flag, params.sd1sd2Flag); + expect(st.pnnThresholdMs, params.pnnThresholdMs); + expect(st.pnnFlagPct, params.pnnFlagPct); + expect(st.windowMinutes, params.windowMinutes); + expect(st.minWindowBeats, params.minWindowBeats); + expect(st.sustainedFraction, params.sustainedFraction); + }); + + test('a bad window config fails CLOSED: never a flag', () { + // AF all day, so the aggregate verdict is high; only the sustained-window + // check can hold the flag back, and a broken config must hold it back. + final x = _fixture(22, 1.0, hours: 2); + final base = _batch(x.nn, x.t, x.nn.length); + expect(base.present && base.value!.flag, isTrue, + reason: 'sanity: the default config flags this day'); + for (final bad in [ + IrregularScreenState(windowMinutes: 0), + IrregularScreenState(windowMinutes: -5), + IrregularScreenState(windowMinutes: double.nan), + IrregularScreenState(minWindowBeats: 1), + IrregularScreenState(sustainedFraction: 1.5), + IrregularScreenState(sustainedFraction: -0.1), + IrregularScreenState(sustainedFraction: double.nan), + ]) { + bad.fold(x.nn, x.t); + final want = _batch(x.nn, x.t, x.nn.length, + windowMinutes: bad.windowMinutes, + minWindowBeats: bad.minWindowBeats, + sustainedFraction: bad.sustainedFraction); + final got = bad.evaluate(const [], const []); + expect(want.present && !want.value!.flag, isTrue, + reason: 'oracle fails closed'); + _expectSameScreen(got, want, + 'window=${bad.windowMinutes} minBeats=${bad.minWindowBeats} ' + 'sustained=${bad.sustainedFraction}'); + } + }); + }); + + group('behind RrCorrector (the way the pipeline will use it)', () { + test('6 h dirty day, 15-minute passes, both states restored every pass', + () { + final s = synthRr(const SynthConfig( + seed: 5, + hours: 6, + ectopicPerMin: 0.8, + missedPerMin: 0.3, + extraPerMin: 0.3, + noiseRunPerMin: 0.1, + gapPerHour: 4)); + final cuts = timeCuts(s.ts, 900); + var corrector = RrCorrector(); + var st = IrregularScreenState(); + var from = 0, present = 0, maxBytes = 0; + for (var k = 0; k < cuts.length; k++) { + final settled = corrector.fold(s.rr.sublist(from, cuts[k]), + tsMs: s.ts.sublist(from, cuts[k])); + from = cuts[k]; + st.fold(settled.nn, settled.nnTimes); + corrector = RrCorrector.fromJson(jsonRoundTrip(corrector.toJson())); + st = IrregularScreenState.fromJson(jsonRoundTrip(st.toJson())); + maxBytes = math.max(maxBytes, jsonEncode(st.toJson()).length); + if (k % 4 != 0 && k != cuts.length - 1) continue; + final snap = corrector.snapshot(); + final ref = correctRrReference(s.rr.sublist(0, cuts[k]), + rrTsMs: s.ts.sublist(0, cuts[k])); + final af = (1.0 - ref.cleanFraction).clamp(0.0, 1.0).toDouble(); + final want = irregularBeatScreen(ref.nn, + nnTimesMs: ref.nnTimesMs, artifactFraction: af); + final got = st.evaluate(snap.tailNn, snap.tailNnTimes, + artifactFraction: 1.0 - snap.cleanFraction); + _expectSameScreen(got, want, 'pass $k n=${cuts[k]}'); + if (want.present) present++; + } + expect(present, greaterThan(3), reason: 'the screen was really run'); + expect(maxBytes, lessThan(16 * 1024), reason: 'checkpoint chars'); + }, timeout: const Timeout(Duration(minutes: 5))); + + test('23 h day: final screen equals the batch screen on the whole day', () { + final s = realShapedDay(); + final cuts = timeCuts(s.ts, 900); + final corrector = RrCorrector(); + final st = IrregularScreenState(); + var from = 0; + for (final to in cuts) { + final settled = + corrector.fold(s.rr.sublist(from, to), tsMs: s.ts.sublist(from, to)); + from = to; + st.fold(settled.nn, settled.nnTimes); + } + final snap = corrector.snapshot(); + final ref = correctRrReference(s.rr, rrTsMs: s.ts); + final af = (1.0 - ref.cleanFraction).clamp(0.0, 1.0).toDouble(); + final want = irregularBeatScreen(ref.nn, + nnTimesMs: ref.nnTimesMs, artifactFraction: af); + expect(want.present, isTrue, reason: 'sanity: a day is enough data'); + _expectSameScreen( + st.evaluate(snap.tailNn, snap.tailNnTimes, + artifactFraction: 1.0 - snap.cleanFraction), + want, + 'day'); + }, timeout: const Timeout(Duration(minutes: 10))); + }); + + group('contract', () { + test('checkpoint is versioned and typed like the other states', () { + final json = IrregularScreenState().toJson(); + expect(json['version'], 3); + expect(json['type'], 'IrregularScreenState'); + expect(() => IrregularScreenState.fromJson({...json, 'version': 2}), + throwsFormatException); + expect(() => IrregularScreenState.fromJson({...json, 'type': 'Other'}), + throwsFormatException); + expect(() => IrregularScreenState.fromJson({}), throwsFormatException); + }); + + test('checkpoint is bounded: running sums plus one open window', () { + final x = _fixture(41, 0.5, hours: 8); + expect(x.nn.length, greaterThan(20000)); + final st = IrregularScreenState(); + var at = 0, maxBytes = 0; + for (final cut in randomCuts(math.Random(2), x.nn.length, + sizes: [50, 500, 2000])) { + st.fold(x.nn.sublist(at, cut), x.t.sublist(at, cut)); + at = cut; + maxBytes = math.max(maxBytes, jsonEncode(st.toJson()).length); + } + // ~1.4 KB measured in the prototype; the series would be ~250 KB. + expect(maxBytes, lessThan(16 * 1024), reason: 'max checkpoint chars'); + }); + + test('save/restore is invisible: restored state folds on identically', () { + final x = _fixture(42, 0.5, hours: 3); + final cut = x.nn.length ~/ 2; + final a = IrregularScreenState()..fold(x.nn.sublist(0, cut), x.t.sublist(0, cut)); + final b = IrregularScreenState.fromJson(jsonRoundTrip(a.toJson())); + expect(jsonEncode(b.toJson()), jsonEncode(a.toJson())); + a.fold(x.nn.sublist(cut), x.t.sublist(cut)); + b.fold(x.nn.sublist(cut), x.t.sublist(cut)); + expect(jsonEncode(b.toJson()), jsonEncode(a.toJson())); + _expectSameScreen(b.evaluate(const [], const []), + a.evaluate(const [], const []), 'restored vs original'); + }); + }); +} diff --git a/test/onehz/incremental/lomb_scargle_test.dart b/test/onehz/incremental/lomb_scargle_test.dart new file mode 100644 index 0000000..4e9f6d4 --- /dev/null +++ b/test/onehz/incremental/lomb_scargle_test.dart @@ -0,0 +1,99 @@ +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; +import '../support/incremental_compare.dart'; +import '../support/incremental_fixtures.dart'; + +void main() { + final frequencies = [0.0, .0033, .01, .04, .07, .1, .15, .25, .4]; + for (final seed in [1, 42, 914]) { + for (final chunk in [1, 17, 128]) { + test('Lomb prefixes and checkpoint seed=$seed chunk=$chunk', () { + final f = incrementalFixture(seed: seed, beats: 160); + final t = f.times.map((x) => x / 1000).toList(); + var state = IncrementalLombScargle(frequencies); + for (final n in prefixSizes(t.length, chunk)) { + final ts = t.sublist(0, n), y = f.nn.sublist(0, n); + spectrumClose(state.sync(ts, y), lombScargle(ts, y, frequencies)); + final work = state.processedPoints; + spectrumClose(state.sync(ts, y), lombScargle(ts, y, frequencies)); + expect(state.processedPoints, work, + reason: 'identical prefix does no math'); + state = IncrementalLombScargle.fromJson(checkpoint(state.toJson())); + } + expect(state.processedPoints, lessThanOrEqualTo(t.length + 4), + reason: 'append work is linear in newly accepted samples'); + }); + } + test('Lomb replacement, shrink, shifted origin and force seed=$seed', () { + final f = incrementalFixture(seed: seed, beats: 160); + final state = IncrementalLombScargle(frequencies); + var t = f.times.map((x) => x / 1000).toList(), y = [...f.nn]; + spectrumClose(state.sync(t, y), lombScargle(t, y, frequencies)); + y[31] += 79; + spectrumClose(state.sync(t, y), lombScargle(t, y, frequencies)); + t[44] += .02; + spectrumClose(state.sync(t, y), lombScargle(t, y, frequencies)); + t = t.sublist(13, 141); + y = y.sublist(13, 141); + spectrumClose(state.sync(t, y), lombScargle(t, y, frequencies)); + t = t.map((x) => x + 12345.125).toList(); + spectrumClose(state.sync(t, y), lombScargle(t, y, frequencies)); + final work = state.processedPoints; + spectrumClose( + state.sync(t, y, force: true), lombScargle(t, y, frequencies)); + expect(state.processedPoints - work, t.length); + spectrumClose(state.sync([], []), lombScargle([], [], frequencies)); + }); + } + for (final grid in [ + [], + [0.0], + [.04, .04, .13], + [.1, .07, .4] + ]) { + test('Lomb preserves frequency grid $grid', () { + final f = incrementalFixture(beats: 31); + spectrumClose(IncrementalLombScargle(grid).sync(f.times, f.nn), + lombScargle(f.times, f.nn, grid)); + }); + } + test('Lomb constant values, collapsed time, and malformed lengths abstain', + () { + final state = IncrementalLombScargle(frequencies); + for (final input in <(List, List)>[ + ([0.0, 1, 2, 3], [7.0, 7, 7, 7]), + ([1.0, 1, 1, 1], [7.0, 8, 9, 10]), + ([1.0, 2, 3, 4], [7.0]) + ]) { + spectrumClose(state.sync(input.$1, input.$2), + lombScargle(input.$1, input.$2, frequencies)); + } + }); + for (final bad in [double.nan, double.infinity, double.negativeInfinity]) { + test('Lomb nonfinite fallback $bad', () { + final f = incrementalFixture(beats: 31); + final t = f.times.map((x) => x / 1000).toList(), y = [...f.nn]; + final state = IncrementalLombScargle(frequencies); + state.sync(t, y); + y[13] = bad; + spectrumClose(state.sync(t, y), lombScargle(t, y, frequencies)); + y[13] = f.nn[13]; + t[9] = bad; + spectrumClose(state.sync(t, y), lombScargle(t, y, frequencies)); + }); + } + test('Lomb malformed checkpoint throws an explicit input error', () { + expect(() => IncrementalLombScargle.fromJson({'frequencies': 'bad'}), + throwsA(anyOf(isA(), isA()))); + }); + test('Lomb real capture irregular cleaned beat times match batch', () { + final f = realIncrementalFixture(); + expect(f.nn.length, greaterThan(300)); + final t = f.times.map((x) => x / 1000).toList(); + final state = IncrementalLombScargle(frequencies); + for (final n in prefixSizes(f.nn.length, 173)) { + final ts = t.sublist(0, n), nn = f.nn.sublist(0, n); + spectrumClose(state.sync(ts, nn), lombScargle(ts, nn, frequencies)); + } + }); +} diff --git a/test/onehz/incremental/minute_metrics_test.dart b/test/onehz/incremental/minute_metrics_test.dart new file mode 100644 index 0000000..bad8f0d --- /dev/null +++ b/test/onehz/incremental/minute_metrics_test.dart @@ -0,0 +1,251 @@ +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; +import '../support/energy_day.dart'; +import '../support/incremental_compare.dart'; +import '../support/incremental_fixtures.dart'; + +void main() { + for (final entry in energyProfiles.entries) { + for (final sex in Sex.values) { + for (final chunk in [1, 19, 128]) { + test('minutes profile=${entry.key} sex=$sex chunk=$chunk', () { + final f = incrementalFixture(minutes: 87); + var state = IncrementalMinuteMetrics(); + for (final n in prefixSizes(f.hr.length, chunk)) { + final keys = f.minuteKeys.sublist(0, n), hr = f.hr.sublist(0, n); + final cadence = f.cadence.sublist(0, n); + final value = state.sync(keys, hr, + cadenceSpm: cadence, + restingHr: 54, + maxHr: 186, + sex: sex, + profile: entry.value, + quietHrr: .12, + dayMinutes: 900); + minuteClose(value, keys, hr, + cadence: cadence, + sex: sex, + profile: entry.value, + dayMinutes: 900); + final work = state.processedMinutes; + state.sync(keys, hr, + cadenceSpm: cadence, + restingHr: 54, + maxHr: 186, + sex: sex, + profile: entry.value, + quietHrr: .12, + dayMinutes: 900); + expect(state.processedMinutes, work); + if (n == 31 || n == f.hr.length) { + state = + IncrementalMinuteMetrics.fromJson(checkpoint(state.toJson())); + } + } + expect(state.processedMinutes, f.hr.length, + reason: 'each new minute is billed once'); + }); + } + } + } + for (final anchors in <(double?, double?)>[ + (null, null), + (54, null), + (null, 186), + (54, 54), + (70, 60), + (double.nan, 186), + (54, double.infinity), + (54, 186), + ]) { + test('minute measured HR anchor gates $anchors', () { + final f = incrementalFixture(minutes: 31); + final profile = energyProfiles['male']!; + final state = IncrementalMinuteMetrics(); + final value = state.sync(f.minuteKeys, f.hr, + restingHr: anchors.$1, + maxHr: anchors.$2, + profile: profile, + quietHrr: .12); + minuteClose(value, f.minuteKeys, f.hr, + rhr: anchors.$1, maxHr: anchors.$2, profile: profile); + }); + } + for (final quiet in [null, 0, -.1, .12, double.nan]) { + test('minute quiet waking HRR gate $quiet', () { + final f = incrementalFixture(minutes: 31); + minuteClose( + IncrementalMinuteMetrics().sync(f.minuteKeys, f.hr, + restingHr: 54, maxHr: 186, quietHrr: quiet), + f.minuteKeys, + f.hr, + quietHrr: quiet); + }); + } + test('minute sparse arbitrary keyed edits bill only affected minutes', () { + final f = incrementalFixture(minutes: 50); + final profile = energyProfiles['female']!; + var keys = [...f.minuteKeys], hr = [...f.hr]; + var cadence = [...f.cadence]; + final state = IncrementalMinuteMetrics(); + MinuteMetrics sync() => state.sync(keys, hr, + cadenceSpm: cadence, + restingHr: 54, + maxHr: 186, + profile: profile, + quietHrr: .12); + void check() => + minuteClose(sync(), keys, hr, cadence: cadence, profile: profile); + check(); + expect(state.processedMinutes, 50); + hr[11] += 29; + check(); + expect(state.processedMinutes, 51); + cadence[22] = 123; + check(); + expect(state.processedMinutes, 52); + keys.insert(5, 27999999); + hr.insert(5, 95); + cadence.insert(5, 120); + check(); + expect(state.processedMinutes, 53); + keys.removeAt(8); + hr.removeAt(8); + cadence.removeAt(8); + check(); + expect(state.processedMinutes, 53, + reason: 'deletion removes existing bill'); + keys = keys.reversed.toList(); + hr = hr.reversed.toList(); + cadence = cadence.reversed.toList(); + check(); + expect(state.processedMinutes, 53, + reason: 'same keyed minutes may be reordered'); + keys[0] += 10000; + check(); + expect(state.processedMinutes, 54); + final work = state.processedMinutes; + minuteClose( + state.sync(keys, hr, + cadenceSpm: cadence, + restingHr: 54, + maxHr: 186, + profile: profile, + quietHrr: .12, + force: true), + keys, + hr, + cadence: cadence, + profile: profile); + expect(state.processedMinutes - work, keys.length); + }); + test( + 'minute parameter changes recompute sex, profiles, anchors and day totals', + () { + final f = incrementalFixture(minutes: 47); + final state = IncrementalMinuteMetrics(); + for (final entry in energyProfiles.entries) { + for (final sex in Sex.values) { + for (final rhr in [54.0, 65.0]) { + final value = state.sync(f.minuteKeys, f.hr, + cadenceSpm: f.cadence, + restingHr: rhr, + maxHr: 190, + profile: entry.value, + sex: sex, + quietHrr: .18, + dayMinutes: 700); + minuteClose(value, f.minuteKeys, f.hr, + cadence: f.cadence, + rhr: rhr, + maxHr: 190, + profile: entry.value, + sex: sex, + quietHrr: .18, + dayMinutes: 700); + } + } + } + final p = energyProfiles['male']!; + minuteClose( + state.sync(f.minuteKeys, f.hr, + restingHr: 54, + maxHr: 186, + profile: p, + dayMinutes: 0, + quietHrr: .12), + f.minuteKeys, + f.hr, + profile: p, + dayMinutes: 0); + minuteClose( + state.sync(f.minuteKeys, f.hr, + restingHr: 54, + maxHr: 186, + profile: p, + dayMinutes: 900, + quietHrr: .12), + f.minuteKeys, + f.hr, + profile: p, + dayMinutes: 900); + }); + test('minute nonfinite HR/cadence and absent HR use exact abstentions', () { + final day = energyDay(n: 100); + final keys = List.generate(day.hr.length, (i) => i * 3); + final p = energyProfiles['nonbinary']!; + final state = IncrementalMinuteMetrics(); + for (final n in prefixSizes(day.hr.length, 13)) { + final k = keys.sublist(0, n), + h = day.hr.sublist(0, n), + c = day.cadence.sublist(0, n); + minuteClose( + state.sync(k, h, + cadenceSpm: c, + restingHr: 54, + maxHr: 186, + profile: p, + quietHrr: .12), + k, + h, + cadence: c, + profile: p); + } + }); + test( + 'minute JSON checkpoint continues exact precision with keyed replacement', + () { + final f = incrementalFixture(minutes: 61); + final hr = f.hr.map((x) => x + .0123456789123).toList(); + final p = energyProfiles['male']!; + final first = IncrementalMinuteMetrics(); + first.sync(f.minuteKeys.sublist(0, 31), hr.sublist(0, 31), + restingHr: 54, maxHr: 186, profile: p, quietHrr: .12); + final state = IncrementalMinuteMetrics.fromJson(checkpoint(first.toJson())); + hr[17] += 5.123456789123; + minuteClose( + state.sync(f.minuteKeys, hr, + restingHr: 54, maxHr: 186, profile: p, quietHrr: .12), + f.minuteKeys, + hr, + profile: p); + expect(state.processedMinutes, 62); + }); + test( + 'minute real capture means and sparse keys match batch load and calories', + () { + final f = realIncrementalFixture(); + expect(f.hr.length, 15); + final p = energyProfiles['male']!; + final state = IncrementalMinuteMetrics(); + for (var n = 0; n <= f.hr.length; n++) { + final k = f.minuteKeys.sublist(0, n), h = f.hr.sublist(0, n); + minuteClose( + state.sync(k, h, + restingHr: 54, maxHr: 186, profile: p, quietHrr: .12), + k, + h, + profile: p); + } + }); +} diff --git a/test/onehz/incremental/minute_summary_test.dart b/test/onehz/incremental/minute_summary_test.dart new file mode 100644 index 0000000..504b0ac --- /dev/null +++ b/test/onehz/incremental/minute_summary_test.dart @@ -0,0 +1,281 @@ +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; +import '../support/energy_day.dart'; +import '../support/incremental_compare.dart'; +import '../support/incremental_fixtures.dart'; + +void _summaryClose( + MinuteMetrics actual, + List hr, { + List? cadence, + double? rhr = 54, + double? maxHr = 186, + Sex sex = Sex.male, + WorkoutUserProfile? profile, + int dayMinutes = 1440, + double? quietHrr = .12, +}) { + expect(actual.minutes, isNull, + reason: 'summary requests omit minute artifacts'); + final trimp = banisterTrimp(hr, restingHr: rhr, maxHr: maxHr, sex: sex); + final strain = strainScoreMetric(trimp.value, + wakeMinutes: hr.length.toDouble(), + quietHrr: quietHrr, + quietSettled: false, + female: sex == Sex.female); + metricEnvelope(actual.trimp, trimp); + numberClose(actual.trimp.value, trimp.value); + metricEnvelope(actual.strain, strain); + numberClose(actual.strain.value, strain.value); + final energy = profile == null || rhr == null || maxHr == null + ? null + : Calories.dailyEnergy(hr, + profile: profile, + hrmax: maxHr, + restingHr: rhr, + dayMinutes: dayMinutes, + cadenceSpmPerMin: cadence); + if (energy == null) { + expect(actual.energy, isNull); + } else { + expect(actual.energy, isNotNull); + numberClose(actual.energy!.total, energy.total); + numberClose(actual.energy!.active, energy.active); + numberClose(actual.energy!.basal, energy.basal); + numberClose(actual.energy!.walking, energy.walking); + } +} + +void main() { + for (final seed in [1, 42, 914]) { + for (final entry in energyProfiles.entries) { + for (final chunk in [1, 19]) { + test( + 'summary append/restore seed=$seed profile=${entry.key} chunk=$chunk', + () { + final f = incrementalFixture(seed: seed, minutes: 61); + final sex = entry.key == 'female' ? Sex.female : Sex.male; + var state = IncrementalMinuteMetrics(); + for (final n in prefixSizes(f.hr.length, chunk)) { + final keys = f.minuteKeys.sublist(0, n), hr = f.hr.sublist(0, n); + final cad = f.cadence.sublist(0, n); + _summaryClose( + state.sync(keys, hr, + cadenceSpm: cad, + restingHr: 54, + maxHr: 186, + sex: sex, + profile: entry.value, + quietHrr: .12, + dayMinutes: 900, + includeMinuteSeries: false), + hr, + cadence: cad, + sex: sex, + profile: entry.value, + dayMinutes: 900); + expect(state.processedMinutes, n); + if (n == 31) { + state = + IncrementalMinuteMetrics.fromJson(checkpoint(state.toJson())); + } + } + final work = state.processedMinutes; + minuteClose( + state.sync(f.minuteKeys, f.hr, + cadenceSpm: f.cadence, + restingHr: 54, + maxHr: 186, + sex: sex, + profile: entry.value, + quietHrr: .12, + dayMinutes: 900), + f.minuteKeys, + f.hr, + cadence: f.cadence, + sex: sex, + profile: entry.value, + dayMinutes: 900); + expect(state.processedMinutes, work, + reason: + 'emitting retained minute artifacts requires no repricing'); + }); + } + } + } + test( + 'summary edits/removals/reorder and artifact toggles only reprice changed keys', + () { + final f = incrementalFixture(minutes: 50); + final profile = energyProfiles['male']!; + final state = IncrementalMinuteMetrics(); + var keys = [...f.minuteKeys], hr = [...f.hr]; + var cadence = [...f.cadence]; + MinuteMetrics sync({bool series = false, bool force = false}) => + state.sync(keys, hr, + cadenceSpm: cadence, + restingHr: 54, + maxHr: 186, + profile: profile, + quietHrr: .12, + force: force, + includeMinuteSeries: series); + void check() => + _summaryClose(sync(), hr, cadence: cadence, profile: profile); + // The existing default behavior remains the full minute series. + minuteClose(sync(series: true), keys, hr, + cadence: cadence, profile: profile); + expect(state.processedMinutes, 50); + check(); + expect(state.processedMinutes, 50); + hr[11] += 29; + check(); + expect(state.processedMinutes, 51); + cadence[22] = 123; + check(); + expect(state.processedMinutes, 52); + keys.insert(5, 27999999); + hr.insert(5, 95); + cadence.insert(5, 120); + check(); + expect(state.processedMinutes, 53); + keys.removeAt(8); + hr.removeAt(8); + cadence.removeAt(8); + check(); + expect(state.processedMinutes, 53); + keys = keys.reversed.toList(); + hr = hr.reversed.toList(); + cadence = cadence.reversed.toList(); + check(); + expect(state.processedMinutes, 53); + minuteClose(sync(series: true), keys, hr, + cadence: cadence, profile: profile); + expect(state.processedMinutes, 53); + _summaryClose(sync(force: true), hr, cadence: cadence, profile: profile); + expect(state.processedMinutes, 53 + keys.length); + keys.clear(); + hr.clear(); + cadence.clear(); + check(); + final work = state.processedMinutes; + minuteClose(sync(series: true), keys, hr, + cadence: cadence, profile: profile); + expect(state.processedMinutes, work); + }); + test( + 'summary anchor/profile/sex changes and day duration follow batch exactly', + () { + final f = incrementalFixture(minutes: 31); + var state = IncrementalMinuteMetrics(); + for (final p in energyProfiles.values) { + for (final anchors in <(double?, double?)>[ + (54, 186), + (65, 197), + (null, 186), + (54, null), + (70, 60), + (double.nan, 186) + ]) { + for (final sex in Sex.values) { + for (final duration in [0, 900, 1440]) { + _summaryClose( + state.sync(f.minuteKeys, f.hr, + cadenceSpm: f.cadence, + restingHr: anchors.$1, + maxHr: anchors.$2, + sex: sex, + profile: p, + quietHrr: .18, + dayMinutes: duration, + includeMinuteSeries: false), + f.hr, + cadence: f.cadence, + rhr: anchors.$1, + maxHr: anchors.$2, + sex: sex, + profile: p, + quietHrr: .18, + dayMinutes: duration); + state = + IncrementalMinuteMetrics.fromJson(checkpoint(state.toJson())); + } + } + } + } + }); + test('summary nonfinite HR/cadence preserves abstentions and finite energy', + () { + final f = energyDay(n: 91); + final keys = List.generate(f.hr.length, (i) => 28000000 + i * 2); + final p = energyProfiles['nonbinary']!; + final state = IncrementalMinuteMetrics(); + for (final n in prefixSizes(f.hr.length, 17)) { + final k = keys.sublist(0, n), + h = f.hr.sublist(0, n), + c = f.cadence.sublist(0, n); + _summaryClose( + state.sync(k, h, + cadenceSpm: c, + restingHr: 54, + maxHr: 186, + profile: p, + quietHrr: .12, + includeMinuteSeries: false), + h, + cadence: c, + profile: p); + } + }); + test('summary without profile still calculates clinical metrics', () { + final f = incrementalFixture(minutes: 31); + final state = IncrementalMinuteMetrics(); + _summaryClose( + state.sync(f.minuteKeys, f.hr, + restingHr: 54, + maxHr: 186, + quietHrr: .12, + includeMinuteSeries: false), + f.hr); + final work = state.processedMinutes; + minuteClose( + state.sync(f.minuteKeys, f.hr, + restingHr: 54, maxHr: 186, quietHrr: .12), + f.minuteKeys, + f.hr); + expect(state.processedMinutes, work); + final p = energyProfiles['male']!; + _summaryClose( + state.sync(f.minuteKeys, f.hr, + restingHr: 54, + maxHr: 186, + profile: p, + quietHrr: .12, + includeMinuteSeries: false), + f.hr, + profile: p); + }); + test('day duration reprices only basal, across short, DST and long days', () { + final f = incrementalFixture(seed: 5, minutes: 47); + final p = energyProfiles.values.first; + final state = IncrementalMinuteMetrics(); + MinuteMetrics sync(int dayMinutes) => state.sync(f.minuteKeys, f.hr, + cadenceSpm: f.cadence, + restingHr: 54, + maxHr: 186, + profile: p, + quietHrr: .12, + dayMinutes: dayMinutes, + includeMinuteSeries: false); + sync(1440); + final work = state.processedMinutes; + // Empty, one minute, an hour, both DST days, a whole and a double day, and + // a negative count (batch multiplies it through; parity, not validation). + for (final minutes in [0, 1, 60, 1380, 1439, 1440, 1441, 1500, 2880, -1]) { + _summaryClose(sync(minutes), f.hr, + cadence: f.cadence, profile: p, dayMinutes: minutes); + expect(state.processedMinutes, work, + reason: 'basal is a scalar over the day; no minute is repriced'); + } + }); +} diff --git a/test/onehz/incremental/rr_corrector_test.dart b/test/onehz/incremental/rr_corrector_test.dart new file mode 100644 index 0000000..ac4a1f6 --- /dev/null +++ b/test/onehz/incremental/rr_corrector_test.dart @@ -0,0 +1,569 @@ +// Contract for the streaming `RrCorrector` (foundations/rr_correction_stream). +// +// Oracle = the frozen copy of today's `correctRr` (support/correct_rr_reference +// .dart). At every prefix of the input, for any chunking and with the corrector +// saved to JSON and restored between chunks: +// +// settled output so far ++ snapshot().tail == correctRr(prefix) +// +// bit for bit (nn, nnTimes, per-beat classes, counts, cleanFraction). Settled +// output is final (a prefix of what the oracle says about the whole series), +// the settle horizon is bounded, and the checkpoint stays small. +// +// RED until RrCorrector is implemented: every test below fails with +// UnimplementedError from the stub. +import 'dart:convert'; +import 'dart:math' as math; + +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +import '../support/correct_rr_reference.dart'; +import '../support/rr_compare.dart'; +import '../support/rr_stream_driver.dart'; +import '../support/rr_synth.dart'; + +/// Series used by several groups. Cached: generating is cheap, the oracle runs +/// are not. +final _dirty = SynthConfig( + seed: 23, + hours: 0.4, + ectopicPerMin: 2, + missedPerMin: 1, + extraPerMin: 1, + noiseRunPerMin: 0.5, + gapPerHour: 12); + +void main() { + group('every prefix', () { + for (final seed in [3, 4]) { + test('1 beat at a time, n = 0..260, seed $seed', () { + final s = synthRr(SynthConfig( + seed: seed, + hours: 0.12, + ectopicPerMin: 2, + missedPerMin: 1, + extraPerMin: 1, + noiseRunPerMin: 0.8, + gapPerHour: 20)); + final n = math.min(260, s.length); + final run = RrStreamRun(); + run.expectMatchesOracle(s.rr, s.ts, 0, why: 'n=0'); + for (var i = 0; i < n; i++) { + run.fold([s.rr[i]], [s.ts[i]]); + run.expectMatchesOracle(s.rr, s.ts, i + 1, why: 'seed=$seed n=${i + 1}'); + } + }); + } + + test('with a JSON save/restore between EVERY beat (n = 0..230)', () { + final s = synthRr(const SynthConfig( + seed: 11, + hours: 0.1, + ectopicPerMin: 2, + noiseRunPerMin: 0.6, + gapPerHour: 10)); + final n = math.min(230, s.length); + final run = RrStreamRun(); + run.restart(); // a corrector restored from an EMPTY checkpoint + run.expectMatchesOracle(s.rr, s.ts, 0, why: 'empty restore'); + for (var i = 0; i < n; i++) { + run.fold([s.rr[i]], [s.ts[i]]); + run.restart(); + run.expectMatchesOracle(s.rr, s.ts, i + 1, why: 'n=${i + 1}'); + } + }); + + test('1-beat folds over 4000 dirty beats: every alignment of a fold edge ' + 'and an artefact\'s right-anchor search', () { + final s = synthRr(const SynthConfig( + seed: 51, + hours: 1.2, + ectopicPerMin: 3, + missedPerMin: 2, + extraPerMin: 2, + noiseRunPerMin: 1, + gapPerHour: 8)); + final n = math.min(4000, s.length); + final full = correctRrReference(s.rr.sublist(0, n), + rrTsMs: s.ts.sublist(0, n)); + final run = RrStreamRun(); + for (var i = 0; i < n; i++) { + run.fold([s.rr[i]], [s.ts[i]]); + if (i % 997 == 0) run.restart(); + if (i % 50 == 0) run.expectSettledIsPrefixOf(full, 'i=$i'); + } + run.expectMatchesOracle(s.rr, s.ts, n, why: 'final'); + }); + }); + + group('random chunking', () { + final cfgs = { + 'clean': const SynthConfig( + seed: 21, + hours: 0.4, + ectopicPerMin: 0, + missedPerMin: 0, + extraPerMin: 0, + noiseRunPerMin: 0, + gapPerHour: 0), + 'default': const SynthConfig(seed: 22, hours: 0.5), + 'dirty': _dirty, + 'backwards-ts': const SynthConfig( + seed: 24, hours: 0.4, backwardsPerHour: 30, gapPerHour: 6), + }; + for (final e in cfgs.entries) { + for (final restart in [false, true]) { + test('${e.key}, save/restore between chunks: $restart', () { + final s = synthRr(e.value); + final full = correctRrReference(s.rr, rrTsMs: s.ts); + final r = math.Random(e.value.seed * 7 + (restart ? 1 : 0)); + final cuts = randomCuts(r, s.length); + final run = RrStreamRun(); + var from = 0; + for (var k = 0; k < cuts.length; k++) { + final to = cuts[k]; + run.fold(s.rr.sublist(from, to), s.ts.sublist(from, to)); + from = to; + if (restart && r.nextBool()) run.restart(); + run.expectSettledIsPrefixOf(full, '${e.key} cut=$to'); + if (k % 13 == 0 || k == cuts.length - 1) { + run.expectMatchesOracle(s.rr, s.ts, to, + why: '${e.key} cut=$to'); + } + } + }); + } + } + + test('chunking does not matter: 1 chunk, 7 chunks and 1-beat chunks end ' + 'identical', () { + final s = synthRr(const SynthConfig(seed: 61, hours: 0.3)); + final out = []; + for (final size in [s.length, 400, 1]) { + final run = RrStreamRun(); + for (var at = 0; at < s.length; at += size) { + final to = math.min(s.length, at + size); + run.fold(s.rr.sublist(at, to), s.ts.sublist(at, to)); + } + final snap = run.c.snapshot(); + out.add(jsonEncode([ + [...run.nn, ...snap.tailNn], + [...run.nnTimes, ...snap.tailNnTimes], + snap.normalCount, + snap.droppedCount, + snap.correctedCount, + ])); + } + expect(out[1], out[0]); + expect(out[2], out[0]); + }); + }); + + group('timestamps', () { + test('sub-second timestamps', () { + final s = synthRr(const SynthConfig(seed: 33, hours: 0.4, gapPerHour: 6)); + final r = math.Random(12); + final ts = [for (final t in s.ts) t + r.nextInt(1000) + r.nextDouble()]; + final run = RrStreamRun(); + var from = 0; + final cuts = randomCuts(r, s.length); + for (var k = 0; k < cuts.length; k++) { + run.fold(s.rr.sublist(from, cuts[k]), ts.sublist(from, cuts[k])); + from = cuts[k]; + if (k % 9 == 0) run.restart(); + if (k % 9 == 0 || k == cuts.length - 1) { + run.expectMatchesOracle(s.rr, ts, cuts[k], why: 'cut=${cuts[k]}'); + } + } + }); + + test('no timestamps at all (the clock is the RR cumsum)', () { + final s = synthRr(const SynthConfig(seed: 31, hours: 0.4)); + final run = RrStreamRun(); + final r = math.Random(5); + var from = 0; + final cuts = randomCuts(r, s.length); + for (var k = 0; k < cuts.length; k++) { + run.fold(s.rr.sublist(from, cuts[k]), null); + from = cuts[k]; + if (k % 4 == 0) run.restart(); + if (k % 7 == 0 || k == cuts.length - 1) { + run.expectMatchesOracle(s.rr, null, cuts[k], why: 'cut=${cuts[k]}'); + } + } + }); + + test('dropouts: the clock re-anchors across a hole, also across a fold ' + 'boundary and a restore', () { + // 300 beats, a 40 s hole before beat 150, a fold edge exactly there. + final rr = [for (var i = 0; i < 300; i++) 800.0 + (i % 5) * 3]; + var t = 1.7e12; + final ts = [ + for (var i = 0; i < 300; i++) (t += rr[i] + (i == 150 ? 40000 : 0)) + ]; + for (final edge in [149, 150, 151]) { + final run = RrStreamRun(); + run.fold(rr.sublist(0, edge), ts.sublist(0, edge)); + run.restart(); + run.fold(rr.sublist(edge), ts.sublist(edge)); + run.expectMatchesOracle(rr, ts, 300, why: 'edge=$edge'); + } + }); + }); + + group('parameters', () { + for (final p in [ + (alpha: 5.2, win: 31, floor: 100.0, reanchor: 1000.0), + (alpha: 4.0, win: 61, floor: 100.0, reanchor: 1000.0), + (alpha: 5.2, win: 121, floor: 100.0, reanchor: 1000.0), + (alpha: 6.5, win: 91, floor: 30.0, reanchor: 400.0), + (alpha: 5.2, win: 4, floor: 0.0, reanchor: 1000.0), + (alpha: 5.2, win: 90, floor: 100.0, reanchor: 5000.0), + ]) { + test('alpha=${p.alpha} win=${p.win} floor=${p.floor} ' + 'reanchor=${p.reanchor}, with restore', () { + final s = synthRr(const SynthConfig( + seed: 41, hours: 0.4, ectopicPerMin: 1, gapPerHour: 20)); + final run = RrStreamRun.params( + alpha: p.alpha, + win: p.win, + floor: p.floor, + reanchor: p.reanchor); + final r = math.Random(9); + var from = 0; + final cuts = randomCuts(r, s.length); + for (var k = 0; k < cuts.length; k++) { + run.fold(s.rr.sublist(from, cuts[k]), s.ts.sublist(from, cuts[k])); + from = cuts[k]; + if (r.nextBool()) run.restart(); + if (k % 9 == 0 || k == cuts.length - 1) { + run.expectMatchesOracle(s.rr, s.ts, cuts[k], + alpha: p.alpha, + win: p.win, + floor: p.floor, + reanchor: p.reanchor, + why: 'cut=${cuts[k]}'); + } + } + // The parameters travel in the checkpoint. + expect(run.c.alpha, p.alpha); + expect(run.c.windowBeats, p.win); + expect(run.c.minThresholdMs, p.floor); + expect(run.c.reanchorGapMs, p.reanchor); + }); + } + }); + + group('degenerate inputs', () { + void drive(List rr, List ts, String why) { + final run = RrStreamRun(); + final r = math.Random(3); + var from = 0; + for (final to in randomCuts(r, rr.length, sizes: [1, 2, 3, 50, 200])) { + run.fold(rr.sublist(from, to), ts.sublist(from, to)); + from = to; + if (r.nextInt(3) == 0) run.restart(); + run.expectMatchesOracle(rr, ts, to, why: '$why to=$to'); + } + } + + test('constant RR (QD = 0, the floor governs, massive ties)', () { + drive(List.filled(500, 800), + [for (var i = 0; i < 500; i++) 1.7e12 + i * 800.0], 'const'); + }); + + test('constant with one outlier at start / middle / end', () { + for (final at in [0, 1, 250, 498, 499]) { + final rr = List.filled(500, 800)..[at] = 1700; + drive(rr, [for (var i = 0; i < 500; i++) i * 800.0], 'outlier@$at'); + } + }); + + test('artefact runs longer than the window, and alternating artefacts', () { + final r = math.Random(5); + final rr = [ + for (var i = 0; i < 700; i++) + (i >= 150 && i < 290) + ? 250 + r.nextInt(2000).toDouble() + : (i >= 400 && i < 500 && i.isEven) + ? 2200 + : 800 + r.nextInt(40).toDouble() + ]; + var t = 0.0; + drive(rr, [for (final v in rr) (t += v)], 'longrun'); + }); + + test('artefact at beat 0 (no left anchors) and at the tail (no right)', () { + final rr = [ + 2500, + 800, + 810, + 790, + 805, + for (var i = 0; i < 300; i++) 800 + (i % 7) * 3.0, + 2500 + ]; + drive(rr, [for (var i = 0; i < rr.length; i++) i * 800.0], 'edges'); + }); + + test('all artefacts: nothing is ever invented', () { + final rr = List.filled(400, 2500); + drive(rr, [for (var i = 0; i < 400; i++) 1.7e12 + i * 2500.0], 'all'); + }); + + test('n = 1, 2, 3 and just past (the < 3 beat branch hands over)', () { + final rr = [800, 810, 790, 805, 3000, 800]; + final ts = [for (var i = 0; i < 6; i++) 1.7e12 + i * 800.0]; + final run = RrStreamRun(); + for (var i = 0; i < 6; i++) { + run.fold([rr[i]], [ts[i]]); + run.expectMatchesOracle(rr, ts, i + 1, why: 'n=${i + 1}'); + } + final out = RrStreamRun() + ..fold([250, 2500], [1.7e12, 1.7e12 + 2500]); + out.expectMatchesOracle([250, 2500], [1.7e12, 1.7e12 + 2500], 2, + why: 'two implausible beats'); + }); + }); + + group('settle horizon and state size', () { + test('capped settled edge includes the cap decision and its left influence', + () { + // Dirty series: only check the bound where the oracle's FINAL classes of + // the 5 beats just before the 90-beat look-ahead are all normal. Then no + // artefact run is in flight at the classified edge and no lone artefact + // is still waiting for its two right-hand anchors. + final s = synthRr(const SynthConfig( + seed: 23, + hours: 1.0, + ectopicPerMin: 0.5, + missedPerMin: 0.2, + extraPerMin: 0.2, + noiseRunPerMin: 0.1, + gapPerHour: 6)); + final full = correctRrReference(s.rr, rrTsMs: s.ts); + expect(full.classes.where((c) => c != BeatClass.normal).length, + greaterThan(20), + reason: 'sanity: the series really has artefacts'); + final run = RrStreamRun(); + final r = math.Random(17); + var checked = 0, maxLag = 0, from = 0; + final cuts = randomCuts(r, s.length, sizes: [1, 3, 25, 90, 400]); + var lastSettled = 0; + for (final to in cuts) { + run.fold(s.rr.sublist(from, to), s.ts.sublist(from, to)); + from = to; + final settled = run.c.settledBeats; + expect(settled, greaterThanOrEqualTo(lastSettled), + reason: 'settledBeats never goes backwards'); + expect(settled, lessThanOrEqualTo(to)); + lastSettled = settled; + if (to < 130) continue; + final quiet = [ + for (var k = to - 95; k < to - 90; k++) full.classes[k] + ].every((c) => c == BeatClass.normal); + if (quiet) { + checked++; + maxLag = math.max(maxLag, to - settled); + expect(to - settled, lessThanOrEqualTo(280), + reason: 'to=$to settled=$settled'); + } + } + expect(checked, greaterThan(20), reason: 'the bound was really checked'); + // ignore: avoid_print + print('settle lag: checked=$checked max=$maxLag beats'); + }); + + test('a clean series settles right up to the look-ahead', () { + final r = math.Random(71); + var t = 1.7e12; + final rr = [ + for (var i = 0; i < 1500; i++) + (800 + 40 * math.sin(i / 9) + r.nextInt(21) - 10).roundToDouble() + ]; + final ts = [for (final v in rr) (t += v)]; + final full = correctRrReference(rr, rrTsMs: ts); + // The comparison below only holds if the series really is all normal. + expect(full.classes.every((c) => c == BeatClass.normal), isTrue); + final run = RrStreamRun(); + for (var at = 0; at < rr.length; at += 37) { + final to = math.min(rr.length, at + 37); + run.fold(rr.sublist(at, to), ts.sublist(at, to)); + if (to >= 91) { + expect(to - run.c.settledBeats, lessThanOrEqualTo(248), + reason: 'to=$to'); + } + } + }); + + test('checkpoint is bounded: it holds the window, not the series', () { + final s = synthRr(const SynthConfig(seed: 81, hours: 6)); + expect(s.length, greaterThan(20000)); + final run = RrStreamRun(); + var maxBytes = 0; + var from = 0; + for (final to in timeCuts(s.ts, 900)) { + run.fold(s.rr.sublist(from, to), s.ts.sublist(from, to)); + from = to; + maxBytes = math.max(maxBytes, jsonEncode(run.c.toJson()).length); + } + // ~4.6 KB measured in the prototype; the series would be ~300 KB. + expect(maxBytes, lessThan(32 * 1024), reason: 'max checkpoint chars'); + // ignore: avoid_print + print('checkpoint max: $maxBytes chars over ${s.length} beats'); + }); + }); + + group('contract', () { + test('snapshot() changes nothing: twice is equal, folding on is unaffected', + () { + final s = synthRr(const SynthConfig(seed: 91, hours: 0.2)); + final a = RrStreamRun(), b = RrStreamRun(); + final cut = s.length ~/ 2; + a.fold(s.rr.sublist(0, cut), s.ts.sublist(0, cut)); + b.fold(s.rr.sublist(0, cut), s.ts.sublist(0, cut)); + final before = jsonEncode(a.c.toJson()); + final one = a.c.snapshot(), two = a.c.snapshot(); + expect(jsonEncode(a.c.toJson()), before, reason: 'checkpoint unchanged'); + expectBitIdentical(one.tailNn, two.tailNn, 'tail twice'); + // b never snapshots; both then fold the rest and must agree. + a.fold(s.rr.sublist(cut), s.ts.sublist(cut)); + b.fold(s.rr.sublist(cut), s.ts.sublist(cut)); + expectBitIdentical(a.nn, b.nn, 'settled nn with/without snapshot'); + expect(jsonEncode(a.c.toJson()), jsonEncode(b.c.toJson())); + }); + + test('toJson survives jsonEncode/jsonDecode: restored == original', () { + final s = synthRr(_dirty); + final cut = s.length ~/ 2; + final a = RrStreamRun(); + a.fold(s.rr.sublist(0, cut), s.ts.sublist(0, cut)); + final restored = RrCorrector.fromJson(jsonRoundTrip(a.c.toJson())); + expect(jsonEncode(restored.toJson()), jsonEncode(a.c.toJson())); + expect(restored.settledBeats, a.c.settledBeats); + final x = a.c.fold(s.rr.sublist(cut), tsMs: s.ts.sublist(cut)); + final y = restored.fold(s.rr.sublist(cut), tsMs: s.ts.sublist(cut)); + expectBitIdentical(y.nn, x.nn, 'nn after restore'); + expectBitIdentical(y.nnTimes, x.nnTimes, 'times after restore'); + expectSameClasses(y.classes, x.classes, 'classes after restore'); + }); + + test('checkpoint is versioned and typed like the other states', () { + final json = RrCorrector().toJson(); + expect(json['version'], 2); + expect(json['type'], 'RrCorrector'); + expect(() => RrCorrector.fromJson({...json, 'version': 1}), + throwsFormatException); + expect(() => RrCorrector.fromJson({...json, 'type': 'Other'}), + throwsFormatException); + expect(() => RrCorrector.fromJson({}), throwsFormatException); + expect(() => RrCorrector.fromJson({'version': 1, 'type': 'RrCorrector'}), + throwsFormatException); + }); + + test('an empty chunk is a no-op', () { + final s = synthRr(const SynthConfig(seed: 92, hours: 0.1)); + final run = RrStreamRun(); + run.fold(s.rr.sublist(0, 300), s.ts.sublist(0, 300)); + final before = jsonEncode(run.c.toJson()); + final out = run.c.fold(const [], tsMs: const []); + expect(out.nn, isEmpty); + expect(out.nnTimes, isEmpty); + expect(out.classes, isEmpty); + expect(jsonEncode(run.c.toJson()), before); + }); + + test('a fresh corrector reports an empty series, never NaN', () { + final snap = RrCorrector().snapshot(); + expect(snap.n, 0); + expect(snap.tailNn, isEmpty); + expect(snap.tailNnTimes, isEmpty); + expect(snap.tailClasses, isEmpty); + expect(snap.cleanFraction, 0); + expect(snap.droppedCount, 0); + expect(snap.correctedCount, 0); + expect(RrCorrector().settledBeats, 0); + }); + + test('a non-finite RR is refused and leaves the corrector unchanged', () { + for (final bad in [double.nan, double.infinity, double.negativeInfinity]) { + final run = RrStreamRun(); + run.fold([800, 810, 790], [1.7e12, 1.7e12 + 810, 1.7e12 + 1600]); + final before = jsonEncode(run.c.toJson()); + // Good beats BEFORE the bad one in the same chunk must not stick. + expect( + () => run.c.fold([805, 800, bad, 810], + tsMs: [1.7e12 + 2400, 1.7e12 + 3200, 1.7e12 + 4000, 1.7e12 + 4800]), + throwsArgumentError, + reason: '$bad'); + expect(jsonEncode(run.c.toJson()), before, reason: 'unchanged $bad'); + } + }); + + test('timestamps are given on every fold or on none', () { + final a = RrCorrector()..fold([800, 810], tsMs: [1.7e12, 1.7e12 + 810]); + final before = jsonEncode(a.toJson()); + expect(() => a.fold([790, 805]), throwsStateError); + expect(jsonEncode(a.toJson()), before, reason: 'unchanged by the throw'); + final b = RrCorrector()..fold([800, 810]); + expect(() => b.fold([790, 805], tsMs: [1.0, 2.0]), throwsStateError); + }); + }); + + group('whole nights and days', () { + test('real 8.9 h WHOOP 4 night, 15-minute passes, restore every pass', () { + final s = realNightRr()!; + final full = correctRrReference(s.rr, rrTsMs: s.ts); + final cuts = timeCuts(s.ts, 900); + final run = RrStreamRun(); + var from = 0, maxLag = 0; + for (var k = 0; k < cuts.length; k++) { + run.fold(s.rr.sublist(from, cuts[k]), s.ts.sublist(from, cuts[k])); + from = cuts[k]; + run.restart(); + run.expectSettledIsPrefixOf(full, 'pass $k'); + maxLag = math.max(maxLag, cuts[k] - run.c.settledBeats); + if (k == 7 || k == 20) { + run.expectMatchesOracle(s.rr, s.ts, cuts[k], why: 'pass $k'); + } + } + run.expectMatchesOracle(s.rr, s.ts, s.length, why: 'final'); + // Final cross-check against the whole-series oracle, field by field. + final snap = run.c.snapshot(); + expectBitIdentical([...run.nn, ...snap.tailNn], full.nn, 'night nn'); + expect(sameBits(snap.cleanFraction, full.cleanFraction), isTrue); + expect(run.restarts, cuts.length); + // Real nights have artefact runs; the horizon stays within a few hundred. + expect(maxLag, lessThan(400), reason: 'max unsettled beats'); + // ignore: avoid_print + print('real night: ${s.length} beats, ${cuts.length} passes, ' + 'max unsettled=$maxLag'); + }, timeout: const Timeout(Duration(minutes: 5))); + + test('23 h day (~96k beats), 15-minute passes, restore every pass', () { + final s = realShapedDay(); + final full = correctRrReference(s.rr, rrTsMs: s.ts); + final cuts = timeCuts(s.ts, 900); + final run = RrStreamRun(); + var from = 0, maxBytes = 0, maxLag = 0; + for (var k = 0; k < cuts.length; k++) { + run.fold(s.rr.sublist(from, cuts[k]), s.ts.sublist(from, cuts[k])); + from = cuts[k]; + run.restart(); + run.expectSettledIsPrefixOf(full, 'pass $k'); + maxBytes = math.max(maxBytes, jsonEncode(run.c.toJson()).length); + maxLag = math.max(maxLag, cuts[k] - run.c.settledBeats); + if (k == 3 || k == 20) { + run.expectMatchesOracle(s.rr, s.ts, cuts[k], why: 'pass $k'); + } + } + run.expectMatchesOracle(s.rr, s.ts, s.length, why: 'final'); + expect(maxBytes, lessThan(32 * 1024), reason: 'checkpoint chars'); + expect(maxLag, lessThan(400), reason: 'max unsettled beats'); + // ignore: avoid_print + print('day: ${s.length} beats, ${cuts.length} passes, ' + 'max checkpoint=$maxBytes chars, max unsettled=$maxLag'); + }, timeout: const Timeout(Duration(minutes: 10))); + }); +} diff --git a/test/onehz/incremental/running_moments_test.dart b/test/onehz/incremental/running_moments_test.dart new file mode 100644 index 0000000..1c2f7cc --- /dev/null +++ b/test/onehz/incremental/running_moments_test.dart @@ -0,0 +1,71 @@ +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; +import '../support/incremental_compare.dart'; +import '../support/incremental_fixtures.dart'; + +void _check(RunningMoments state, List values) { + expect(state.count, values.length); + numberClose(state.mean, mean(values)); + numberClose(state.sampleSd, stddev(values)); + numberClose(state.populationSd, stddevPop(values)); +} + +void main() { + for (final seed in [1, 42, 914]) { + test('moments add/remove and JSON continuation seed=$seed', () { + var state = RunningMoments(); + final values = []; + _check(state, values); + final nn = incrementalFixture(seed: seed, beats: 100).nn; + for (final x in nn) { + values.add(x); + state.add(x); + _check(state, values); + if (values.length == 53) { + state = RunningMoments.fromJson(checkpoint(state.toJson())); + _check(state, values); + } + } + while (values.isNotEmpty) { + state.remove(values.removeAt(values.length ~/ 2)); + _check(state, values); + } + state.add(17); + _check(state, [17]); + }); + test('moments merge disjoint partitions seed=$seed', () { + final values = incrementalFixture(seed: seed, beats: 101).nn; + final left = RunningMoments(), right = RunningMoments(); + for (final x in values.take(37)) { + left.add(x); + } + for (final x in values.skip(37)) { + right.add(x); + } + left.merge(right); + _check(left, values); + _check(right, values.sublist(37)); + left.merge(RunningMoments()); + _check(left, values); + final copy = RunningMoments()..merge(left); + _check(copy, values); + }); + } + for (final values in [ + [], + [42.0], + List.filled(80, 42), + List.generate(80, (i) => 1e9 + (i % 7) * .25) + ]) { + test( + 'moments degenerate/stable data n=${values.length} first=${values.firstOrNull}', + () { + final state = RunningMoments(); + for (final x in values) { + state.add(x); + } + _check(state, values); + _check(RunningMoments.fromJson(checkpoint(state.toJson())), values); + }); + } +} diff --git a/test/onehz/irregular_diagnostics_empty_test.dart b/test/onehz/irregular_diagnostics_empty_test.dart new file mode 100644 index 0000000..6034bef --- /dev/null +++ b/test/onehz/irregular_diagnostics_empty_test.dart @@ -0,0 +1,54 @@ +// The screen's artifact fraction is a share of the beats the corrector saw. When +// it saw none there is no denominator: the diagnostics carry null (absent), never +// the 1.0 a caller computes from `1 - cleanFraction` of an empty series, which +// would read as "100% of beats are artifacts" beside zero beats. Verdicts are +// unchanged: the fraction handed in still drives the artifact gate. +import 'dart:convert'; + +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +const _none = RrCleaningCounts(raw: 0, corrected: 0, dropped: 0); + +void main() { + test('batch: zero beats in -> no artifact fraction, same absent verdict', () { + final r = irregularBeatScreenDetailed(const [], + artifactFraction: 1.0, cleaning: _none); + expect(r.diagnostics.artifactFraction, isNull); + expect((r.diagnostics.toJson()['beats'] as Map)['artifact_fraction'], isNull); + expect(r.diagnostics.rrRaw, 0, reason: 'the raw count stays a real zero'); + expect(r.metric.present, isFalse); + expect(jsonEncode(r.metric.toJson((v) => v.toJson())), + jsonEncode(irregularBeatScreen(const [], artifactFraction: 1.0) + .toJson((v) => v.toJson()))); + }); + + test('batch: the fraction is kept whenever the corrector saw beats, or the ' + 'caller gave no counts (unchanged)', () { + const seen = RrCleaningCounts(raw: 100, corrected: 5, dropped: 2); + expect( + irregularBeatScreenDetailed(const [], artifactFraction: 0.07, cleaning: seen) + .diagnostics + .artifactFraction, + 0.07); + expect(irregularBeatScreenDetailed(const []).diagnostics.artifactFraction, 0.0); + }); + + test('streaming: zero beats in -> no artifact fraction', () { + final r = IrregularScreenState().evaluateDetailed(const [], const [], + artifactFraction: 1.0, cleaning: _none); + expect(r.diagnostics.artifactFraction, isNull); + expect(r.metric.present, isFalse); + }); + + test('the wire round-trips an absent fraction', () { + final d = irregularBeatScreenDetailed(const [], + artifactFraction: 1.0, cleaning: _none) + .diagnostics; + final text = jsonEncode(d.toJson()); + final back = IrregularDiagnostics.fromJson( + (jsonDecode(text) as Map).cast()); + expect(back.artifactFraction, isNull); + expect(jsonEncode(back.toJson()), text); + }); +} diff --git a/test/onehz/irregular_diagnostics_test.dart b/test/onehz/irregular_diagnostics_test.dart new file mode 100644 index 0000000..138ae25 --- /dev/null +++ b/test/onehz/irregular_diagnostics_test.dart @@ -0,0 +1,574 @@ +// Contract for the irregular-rhythm screen's DIAGNOSTICS (batch side). +// +// Today `irregularBeatScreen` returns SD1/SD2/ratio/pNN/n_beats/flag and throws +// away the evidence: how many beats went in, how many survived the [300, 2000] +// filter, how the corrector treated them, how many 5-minute windows existed, +// how many voted, how many flagged, what the final OPEN window was, and what +// fraction of windows flagged against what the rule needs. An abstention also +// carried only a note string. The contract: +// +// a. `irregularBeatScreenDetailed` returns the SAME Metric as +// `irregularBeatScreen` plus an `IrregularDiagnostics`, present when the +// screen ran AND when it abstained (the abstention names its cause + counts); +// c. absent input stays absent: no beats => absent metric, zero counts, no +// fabricated windows; an unknown count is null, never 0; NaN is not a beat; +// d. toJson round-trips, existing JSON keys and verdicts are unchanged. +// +// (b, batch == resumed, is test/onehz/incremental/irregular_diagnostics_stream_test.dart.) +// +// RED until implemented: the stub throws UnimplementedError. +import 'dart:convert'; +import 'dart:math' as math; + +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +// ---- fixtures ------------------------------------------------------------- + +/// Calm, slowly drifting sinus: valid windows, none flagged. +List _calm(int n) => + [for (var i = 0; i < n; i++) (1000 + 15 * math.sin(i / 8)).roundToDouble()]; + +/// Irregularly irregular: every window of it flags. +List _af(int n, int seed) { + final r = math.Random(seed); + return [ + for (var i = 0; i < n; i++) + (800.0 + (r.nextBool() ? 250 : -50) + r.nextInt(120)).roundToDouble() + ]; +} + +List _cumsum(List rr) { + var t = 0.0; + return [for (final v in rr) t += v]; +} + +/// Three-window day crafted so every window edge is hand-known (the same trick +/// as the screen-state test): window 1 opens at the first beat (t = 1000) and +/// is AF-like; the beat that lands EXACTLY 300 s after a window's first beat +/// opens the next one. Window 2 is calm. The final window holds [lastBeats] +/// beats in total (the edge beat that opens it included) of [lastMode] +/// ('calm' | 'af') and is OPEN when the data ends. +({List nn, List t}) _threeWindowDay( + int lastBeats, String lastMode) { + final r = math.Random(3); + final nn = [], t = []; + var clock = 0.0; + void add(double v) { + clock += v; + nn.add(v); + t.add(clock); + } + + void landOn(double target, double Function() next) { + while (target - clock > 2200) { + add(next()); + } + var rest = target - clock; + if (rest < 300) { + clock -= nn.removeLast(); + t.removeLast(); + rest = target - clock; + } + if (rest > 2000) { + add(rest / 2); + add(rest / 2); + } else { + add(rest); + } + } + + double afLike() => 400.0 + 100 * r.nextInt(13); + double calm() => 800.0 + 10 * (r.nextInt(5) - 2); + add(1000); + landOn(301000, afLike); + landOn(601000, calm); + for (var i = 1; i < lastBeats; i++) { + add(lastMode == 'af' ? afLike() : calm()); + } + expect(t.where((x) => x == 301000 || x == 601000).length, 2, + reason: 'fixture: both window edges landed exactly'); + return (nn: nn, t: t); +} + +/// Independent window oracle: the same bucketing rule written out, each window +/// judged by calling the PUBLIC screen on its beats (whole-span verdict, no +/// times), so it shares no code with the window helpers it checks. Only for +/// series with no out-of-range beats (no gaps inside a bucket). +({int total, int valid, int flagged, int openBeats, IrregularOpenWindow open}) + _refWindows(List nn, List t, + {double windowMin = 5, int minWindowBeats = 40}) { + var total = 0, valid = 0, flagged = 0; + var bucket = []; + IrregularOpenWindow last = IrregularOpenWindow.none; + void close() { + if (bucket.isEmpty) return; + total++; + if (bucket.length < minWindowBeats) { + last = IrregularOpenWindow.thin; + } else { + valid++; + final m = irregularBeatScreen(bucket, minBeats: 2); + final f = m.present && m.value!.flag; + if (f) flagged++; + last = f ? IrregularOpenWindow.flagged : IrregularOpenWindow.unflagged; + } + } + + var start = t.isEmpty ? 0.0 : t.first; + var openBeats = 0; + for (var i = 0; i < nn.length; i++) { + if (t[i] - start >= windowMin * 60000) { + close(); + bucket = []; + start = t[i]; + } + bucket.add(nn[i]); + } + openBeats = bucket.length; + close(); + return ( + total: total, + valid: valid, + flagged: flagged, + openBeats: openBeats, + open: last, + ); +} + +String _json(Object? o) => jsonEncode(o); + +void main() { + group('present screen: the evidence behind a verdict', () { + test('beat counts: raw in, kept after the [300,2000] filter, NaN is not a ' + 'beat', () { + final rr = _af(1200, 1); + rr[10] = 250; // short + rr[11] = 2500; // long + rr[12] = double.nan; + rr[500] = 299.999; + final res = irregularBeatScreenDetailed(rr, + nnTimesMs: _cumsum(rr.map((v) => v.isFinite ? v : 800.0).toList())); + final d = res.diagnostics; + expect(d.nnIn, 1200); + expect(d.nnKept, 1200 - 4, reason: '250, 2500, NaN and 299.999 are out'); + expect(res.metric.present, isTrue); + expect(d.nnKept, res.metric.value!.nBeats, + reason: 'diagnostics and value agree on the beats analysed'); + expect(d.abstain, isNull); + expect(d.artifactFraction, 0.0); + }); + + test('the verdict is exactly irregularBeatScreen\'s, to the last digit', () { + for (final rr in [_calm(1200), _af(1200, 7), _af(900, 8)]) { + final t = _cumsum(rr); + final plain = irregularBeatScreen(rr, nnTimesMs: t); + final res = irregularBeatScreenDetailed(rr, nnTimesMs: t); + expect(_json(res.metric.toJson((v) => v.toJson())), + _json(plain.toJson((v) => v.toJson()))); + } + }); + + test('hand-known windows: AF-like, calm, then an OPEN window of exactly ' + 'minWindowBeats (40) calm beats is VALID and not flagged', () { + final x = _threeWindowDay(40, 'calm'); + final res = irregularBeatScreenDetailed(x.nn, nnTimesMs: x.t); + expect(res.metric.present, isTrue, reason: 'sanity: evaluable day'); + final w = res.diagnostics.windows!; + expect(w.total, 3); + expect(w.valid, 3); + expect(w.flagged, 1); + expect(w.openBeats, 40); + expect(w.open, IrregularOpenWindow.unflagged); + expect(w.sustainedObserved, closeTo(1 / 3, 1e-12)); + expect(res.diagnostics.thresholds.sustainedFraction, 0.5, + reason: 'required vs observed: 0.5 needed, 1/3 seen'); + expect(res.metric.value!.flag, isFalse); + }); + + test('hand-known windows: an open window of 39 beats is EXCLUDED (thin) but ' + 'still counted in total', () { + final x = _threeWindowDay(39, 'calm'); + final w = + irregularBeatScreenDetailed(x.nn, nnTimesMs: x.t).diagnostics.windows!; + expect(w.total, 3); + expect(w.valid, 2, reason: 'the 39-beat tail does not vote'); + expect(w.flagged, 1); + expect(w.openBeats, 39); + expect(w.open, IrregularOpenWindow.thin); + expect(w.sustainedObserved, 0.5); + }); + + test('hand-known windows: an open window of AF-like beats is VALID and ' + 'FLAGGED', () { + final x = _threeWindowDay(60, 'af'); + final res = irregularBeatScreenDetailed(x.nn, nnTimesMs: x.t); + final w = res.diagnostics.windows!; + expect(w.total, 3); + expect(w.valid, 3); + expect(w.flagged, 2); + expect(w.openBeats, 60); + expect(w.open, IrregularOpenWindow.flagged); + expect(w.sustainedObserved, closeTo(2 / 3, 1e-12)); + }); + + test('window counts equal an independent oracle across seeds', () { + for (final seed in [1, 2, 3, 4, 5, 6]) { + final r = math.Random(seed); + // Blocks of calm and AF of random length, so windows straddle modes. + final rr = []; + while (rr.length < 3000) { + rr.addAll(r.nextBool() ? _calm(100 + r.nextInt(500)) : _af(100 + r.nextInt(500), seed)); + } + final t = _cumsum(rr); + final ref = _refWindows(rr, t); + final res = irregularBeatScreenDetailed(rr, nnTimesMs: t); + final w = res.diagnostics.windows!; + expect(w.total, ref.total, reason: 'seed $seed'); + expect(w.valid, ref.valid, reason: 'seed $seed'); + expect(w.flagged, ref.flagged, reason: 'seed $seed'); + expect(w.openBeats, ref.openBeats, reason: 'seed $seed'); + expect(w.open, ref.open, reason: 'seed $seed'); + expect(w.valid, lessThanOrEqualTo(w.total)); + expect(w.flagged, lessThanOrEqualTo(w.valid)); + if (res.metric.present && res.metric.value!.flag) { + expect(w.sustainedObserved, greaterThanOrEqualTo(0.5), + reason: 'a flag implies the sustained rule was met'); + } + } + }); + + test('windows are counted even when the aggregate is not high (they explain ' + 'a NOT flagged too)', () { + final rr = _calm(1500); + final res = irregularBeatScreenDetailed(rr, nnTimesMs: _cumsum(rr)); + final w = res.diagnostics.windows!; + expect(res.metric.value!.flag, isFalse); + expect(w.total, greaterThan(1)); + expect(w.valid, greaterThan(1)); + expect(w.flagged, 0); + expect(w.sustainedObserved, 0.0, reason: 'valid windows, none flagged: 0, a fact'); + }); + + test('cleaning counts pass through; without them they are null, not 0', () { + final rr = _af(1200, 3); + final t = _cumsum(rr); + final withCounts = irregularBeatScreenDetailed(rr, + nnTimesMs: t, + artifactFraction: 0.07, + cleaning: const RrCleaningCounts(raw: 1290, corrected: 41, dropped: 49)); + expect(withCounts.diagnostics.rrRaw, 1290); + expect(withCounts.diagnostics.corrected, 41); + expect(withCounts.diagnostics.dropped, 49); + expect(withCounts.diagnostics.artifactFraction, 0.07); + final without = irregularBeatScreenDetailed(rr, nnTimesMs: t); + expect(without.diagnostics.rrRaw, isNull); + expect(without.diagnostics.corrected, isNull); + expect(without.diagnostics.dropped, isNull); + }); + + test('real corrector output: counts equal correctRr\'s own', () { + final r = math.Random(11); + final raw = [ + for (var i = 0; i < 2500; i++) + (i % 97 == 50 + ? 420 // isolated early beat + : (i % 211 >= 100 && i % 211 < 106) + ? 250 + r.nextInt(1800) // a noise run + : 900 + 30 * math.sin(i / 7) + r.nextInt(20)) + .toDouble() + ]; + final c = correctRr(raw); + expect(c.correctedCount, greaterThan(0), reason: 'fixture corrects'); + expect(c.droppedCount, greaterThan(0), reason: 'fixture drops'); + final res = irregularBeatScreenDetailed(c.nn, + nnTimesMs: c.nnTimesMs, + artifactFraction: (1 - c.cleanFraction).clamp(0.0, 1.0), + cleaning: RrCleaningCounts( + raw: raw.length, + corrected: c.correctedCount, + dropped: c.droppedCount)); + expect(res.diagnostics.rrRaw, raw.length); + expect(res.diagnostics.corrected, c.correctedCount); + expect(res.diagnostics.dropped, c.droppedCount); + expect(res.diagnostics.nnIn, c.nn.length); + }); + }); + + group('abstention carries the counts that caused it', () { + test('too few beats', () { + final rr = _calm(212); + final res = irregularBeatScreenDetailed(rr, nnTimesMs: _cumsum(rr)); + expect(res.metric.present, isFalse); + final d = res.diagnostics; + expect(d.abstain, IrregularAbstain.tooFewBeats); + expect(d.nnIn, 212); + expect(d.nnKept, 212); + expect(d.thresholds.minBeats, irregularScreenMinBeats); + expect(res.metric.note, contains('too few clean beats'), + reason: 'the existing note is unchanged'); + }); + + test('too few beats because the filter removed them (kept < minBeats <= in)', + () { + final rr = [ + for (var i = 0; i < 700; i++) i % 2 == 0 ? 1000.0 + i % 40 : 250.0 + ]; // 350 kept + final d = irregularBeatScreenDetailed(rr).diagnostics; + expect(d.abstain, IrregularAbstain.tooFewBeats); + expect(d.nnIn, 700); + expect(d.nnKept, 350); + }); + + test('artifact fraction over the line', () { + final rr = _calm(1200); + final res = irregularBeatScreenDetailed(rr, + nnTimesMs: _cumsum(rr), + artifactFraction: 0.45, + cleaning: const RrCleaningCounts(raw: 2200, corrected: 100, dropped: 900)); + expect(res.metric.present, isFalse); + final d = res.diagnostics; + expect(d.abstain, IrregularAbstain.artifact); + expect(d.artifactFraction, 0.45); + expect(d.thresholds.maxArtifact, 0.30); + expect(d.nnKept, 1200); + expect(d.dropped, 900); + expect(d.corrected, 100); + }); + + // The gate counts successive pairs (upstream, merged 2026-10-18), so with + // any minBeats >= 2 an all-isolated series is "too few"; minBeats 1 is the + // one setting that still reaches the no-pairs abstention. + test('no successive clean pairs (every kept beat is isolated)', () { + final rr = [ + for (var i = 0; i < 1000; i++) i % 2 == 0 ? 900.0 + i % 50 : 250.0 + ]; + expect(irregularBeatScreenDetailed(rr, minBeats: 100).diagnostics.abstain, + IrregularAbstain.tooFewBeats); + final res = irregularBeatScreenDetailed(rr, minBeats: 1); + expect(res.metric.present, isFalse); + expect(res.diagnostics.abstain, IrregularAbstain.noSuccessivePairs); + expect(res.diagnostics.nnIn, 1000); + expect(res.diagnostics.nnKept, 500); + }); + + test('no long-term variability (SD2 = 0), not "perfectly regular"', () { + final rr = [for (var i = 0; i < 1200; i++) 1000.0]; + final res = irregularBeatScreenDetailed(rr, nnTimesMs: _cumsum(rr)); + expect(res.metric.present, isFalse); + expect(res.diagnostics.abstain, IrregularAbstain.noLongTermVariability); + expect(res.diagnostics.nnKept, 1200); + }); + + test('an abstained screen still reports its windows (they were there)', () { + final x = _threeWindowDay(60, 'af'); + final res = irregularBeatScreenDetailed(x.nn, + nnTimesMs: x.t, artifactFraction: 0.5); + expect(res.metric.present, isFalse); + expect(res.diagnostics.abstain, IrregularAbstain.artifact); + expect(res.diagnostics.windows!.total, 3); + expect(res.diagnostics.windows!.flagged, 2); + }); + }); + + group('absent input stays absent', () { + test('no beats: absent metric, zero counts, no fabricated windows', () { + final res = irregularBeatScreenDetailed(const [], nnTimesMs: const []); + expect(res.metric.present, isFalse); + expect(res.metric.confidence, 0); + final d = res.diagnostics; + expect(d.abstain, IrregularAbstain.tooFewBeats); + expect(d.nnIn, 0); + expect(d.nnKept, 0); + expect(d.rrRaw, isNull, reason: 'no cleaning info given: unknown, not 0'); + final w = d.windows!; + expect((w.total, w.valid, w.flagged, w.openBeats), (0, 0, 0, 0)); + expect(w.open, IrregularOpenWindow.none); + expect(w.sustainedObserved, isNull, reason: 'no valid window: null, not 0 or NaN'); + }); + + test('no beat times: windows are NOT evaluated (null), not zero windows', () { + final res = irregularBeatScreenDetailed(_af(1200, 2)); + expect(res.diagnostics.windows, isNull); + expect(res.metric.present, isTrue); + }); + + test('times of the wrong length: windows null, whole-span verdict as before', + () { + final rr = _af(1200, 2); + final res = irregularBeatScreenDetailed(rr, nnTimesMs: _cumsum(rr).sublist(1)); + expect(res.diagnostics.windows, isNull); + expect(_json(res.metric.toJson((v) => v.toJson())), + _json(irregularBeatScreen(rr, nnTimesMs: _cumsum(rr).sublist(1)).toJson((v) => v.toJson()))); + }); + + test('a window config that fails closed: windows null, never flagged', () { + final rr = _af(1200, 2); + final res = irregularBeatScreenDetailed(rr, + nnTimesMs: _cumsum(rr), windowMinutes: 0); + expect(res.diagnostics.windows, isNull); + expect(res.metric.value!.flag, isFalse); + }); + }); + + group('JSON', () { + List variety() { + final x = _threeWindowDay(40, 'calm'); + final af = _af(1200, 5); + return [ + irregularBeatScreenDetailed(x.nn, + nnTimesMs: x.t, + cleaning: const RrCleaningCounts(raw: 900, corrected: 3, dropped: 4)), + irregularBeatScreenDetailed(af, nnTimesMs: _cumsum(af)), + irregularBeatScreenDetailed(af), // windows null, cleaning null + irregularBeatScreenDetailed(_calm(100)), // tooFewBeats + irregularBeatScreenDetailed(_calm(1200), artifactFraction: 0.5), + irregularBeatScreenDetailed([for (var i = 0; i < 1200; i++) 1000.0]), + irregularBeatScreenDetailed(const []), + ]; + } + + test('every shape survives real JSON text and comes back identical', () { + for (final r in variety()) { + final text = jsonEncode(r.diagnostics.toJson()); + final back = + IrregularDiagnostics.fromJson(jsonDecode(text) as Map); + expect(jsonEncode(back.toJson()), text); + expect(back.abstain, r.diagnostics.abstain); + expect(back.nnIn, r.diagnostics.nnIn); + expect(back.nnKept, r.diagnostics.nnKept); + expect(back.rrRaw, r.diagnostics.rrRaw); + expect(back.windows?.total, r.diagnostics.windows?.total); + expect(back.windows?.open, r.diagnostics.windows?.open); + } + }); + + test('the wire shape (what edge persists and Nerd stats reads)', () { + final x = _threeWindowDay(40, 'calm'); + final j = irregularBeatScreenDetailed(x.nn, + nnTimesMs: x.t, + artifactFraction: 0.1, + cleaning: const RrCleaningCounts(raw: 900, corrected: 3, dropped: 4)) + .diagnostics + .toJson(); + expect(j['version'], 1); + expect(j['abstain'], isNull); + final b = j['beats'] as Map; + expect(b.keys.toSet(), { + 'rr_raw', 'nn_in', 'nn_kept', 'corrected', 'dropped', 'artifact_fraction' + }); + expect(b['rr_raw'], 900); + expect(b['corrected'], 3); + expect(b['dropped'], 4); + expect(b['nn_in'], x.nn.length); + expect(b['artifact_fraction'], 0.1); + final w = j['windows'] as Map; + expect(w.keys.toSet(), { + 'total', 'valid', 'flagged', 'sustained_observed', 'open_beats', 'open' + }); + expect(w['total'], 3); + expect(w['open'], 'unflagged'); + expect(w['open_beats'], 40); + final th = j['thresholds'] as Map; + expect(th.keys.toSet(), { + 'min_beats', 'max_artifact', 'sd1sd2_flag', 'pnn_threshold_ms', + 'pnn_flag_pct', 'window_minutes', 'min_window_beats', 'sustained_fraction' + }); + expect(th['min_beats'], 500); + expect(th['sustained_fraction'], 0.5); + }); + + test('abstain wire names; unknown counts and windows are JSON null', () { + expect( + irregularBeatScreenDetailed(_calm(100)).diagnostics.toJson()['abstain'], + 'too_few_beats'); + expect( + irregularBeatScreenDetailed(_calm(1200), artifactFraction: 0.5) + .diagnostics.toJson()['abstain'], + 'artifact'); + final j = irregularBeatScreenDetailed(_calm(100)).diagnostics.toJson(); + expect(j['windows'], isNull); + expect((j['beats'] as Map)['rr_raw'], isNull); + expect((j['beats'] as Map)['corrected'], isNull); + expect((j['beats'] as Map)['dropped'], isNull); + expect( + irregularBeatScreenDetailed([for (var i = 0; i < 1200; i++) 1000.0]) + .diagnostics.toJson()['abstain'], + 'no_long_term_variability'); + expect( + irregularBeatScreenDetailed([ + for (var i = 0; i < 1000; i++) i % 2 == 0 ? 900.0 + i % 50 : 250.0 + ], minBeats: 1).diagnostics.toJson()['abstain'], + 'no_successive_pairs'); + }); + + test('fromJson refuses another version and malformed maps', () { + final j = irregularBeatScreenDetailed(_calm(1200)).diagnostics.toJson(); + expect(() => IrregularDiagnostics.fromJson({...j, 'version': 2}), + throwsFormatException); + expect(() => IrregularDiagnostics.fromJson({...j, 'beats': 'x'}), + throwsFormatException); + expect(() => IrregularDiagnostics.fromJson({...j, 'abstain': 'because'}), + throwsFormatException); + expect(() => IrregularDiagnostics.fromJson(const {}), throwsFormatException); + }); + + test('IrregularScreenResult.toJson = the old envelope + a diagnostics key', + () { + final rr = _af(1200, 9); + final t = _cumsum(rr); + final res = irregularBeatScreenDetailed(rr, nnTimesMs: t); + final env = res.toJson(); + final old = irregularBeatScreen(rr, nnTimesMs: t).toJson((v) => v.toJson()); + expect(env.keys.toSet(), {...old.keys, 'diagnostics'}); + expect(_json({...env}..remove('diagnostics')), _json(old), + reason: 'every pre-existing key and value is untouched'); + expect(_json(env['diagnostics']), _json(res.diagnostics.toJson())); + // Absent envelope too. + final ab = irregularBeatScreenDetailed(_calm(50)).toJson(); + expect(ab['value'], '—'); + expect(ab['confidence'], 0); + expect(ab['diagnostics'], isA()); + }); + }); + + group('regression: nothing the screen already says changes', () { + test('IrregularRhythm.toJson keeps exactly its six keys (green today)', () { + final j = irregularBeatScreen(_af(1200, 1)).value!.toJson(); + expect(j.keys.toSet(), + {'sd1_ms', 'sd2_ms', 'sd1_sd2', 'pnn_pct', 'n_beats', 'flag'}); + }); + + test('same flag / sd1 / sd2 / pnn / nBeats / confidence / note on the ' + 'existing test inputs, and on salted ones', () { + final r = math.Random(7); + final salted = _af(2000, 4); + salted[5] = 250; + salted[900] = 2200; + final cases = <({List rr, List? t, double af})>[ + (rr: [for (var i = 0; i < 1200; i++) 1000 + 15 * math.sin(i / 8)], t: null, af: 0.0), + (rr: [for (var i = 0; i < 1200; i++) 800.0 + (r.nextBool() ? 250 : -50) + r.nextInt(120)], t: null, af: 0.0), + (rr: [for (var i = 0; i < 100; i++) 1000.0], t: null, af: 0.0), + (rr: [for (var i = 0; i < 1200; i++) 1000.0], t: null, af: 0.4), + (rr: salted, t: null, af: 0.1), + ]; + for (final c in cases) { + final t = c.t ?? _cumsum(c.rr); + for (final times in [null, t]) { + final plain = irregularBeatScreen(c.rr, nnTimesMs: times, artifactFraction: c.af); + final det = irregularBeatScreenDetailed(c.rr, nnTimesMs: times, artifactFraction: c.af); + expect(det.metric.present, plain.present); + expect(det.metric.note, plain.note); + expect(det.metric.confidence, plain.confidence); + expect(det.metric.tier, plain.tier); + expect(det.metric.inputs_used, plain.inputs_used); + if (plain.present) { + final a = det.metric.value!, b = plain.value!; + expect((a.flag, a.nBeats, a.pnnPct, a.sd1, a.sd2, a.sd1sd2), + (b.flag, b.nBeats, b.pnnPct, b.sd1, b.sd2, b.sd1sd2)); + } + } + } + }); + }); +} diff --git a/test/onehz/lomb_scargle_precision_test.dart b/test/onehz/lomb_scargle_precision_test.dart new file mode 100644 index 0000000..0bc9875 --- /dev/null +++ b/test/onehz/lomb_scargle_precision_test.dart @@ -0,0 +1,119 @@ +// Batch Lomb–Scargle on absolute epoch seconds. Trig arguments `w * t` reach +// ~1e10 rad when `t` is ~1.8e9 s, where a double's spacing is ~1e-6 rad, so the +// periodogram used to carry ~1e-6 relative error that depends on the clock, not +// the data. `lombScargle` now measures time from the first sample. +import 'dart:math' as math; + +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +const _epoch = 1791028730.0; + +/// A night of beats. Times are multiples of 1/1024 s, so adding [_epoch] (a +/// whole number of seconds) is EXACT in a double: the epoch series and the +/// rebased series have identical differences, bit for bit. +({List t, List y}) _beats(int n, {int seed = 5}) { + final rnd = math.Random(seed); + final t = [], y = []; + var k = 0; + for (var i = 0; i < n; i++) { + final rr = 900 + + 90 * math.sin(i / 300) + + 45 * math.sin(i * 2 * math.pi * .25 * .9) + + rnd.nextDouble() * 30; + k += (rr / 1000 * 1024).round(); + t.add(k / 1024); + y.add(rr); + } + return (t: t, y: y); +} + +double _worstRel(LombScargle a, LombScargle b) { + expect(a.spectrum.length, b.spectrum.length); + var worst = 0.0; + for (var i = 0; i < a.spectrum.length; i++) { + final e = b.spectrum[i].psd; + final d = (a.spectrum[i].psd - e).abs(); + worst = math.max(worst, e == 0 ? d : d / e.abs()); + } + return worst; +} + +/// The same algorithm with every sum Kahan-compensated and time measured from +/// the first sample. It shares no code with `lombScargle`, so it checks the +/// shift and the summation, not just self-consistency. +LombScargle _kahanReference(List t, List y, List f) { + double ksum(Iterable xs) { + var s = 0.0, c = 0.0; + for (final x in xs) { + final v = x - c; + final u = s + v; + c = (u - s) - v; + s = u; + } + return s; + } + + final n = t.length; + final ts = [for (final v in t) v - t.first]; + final my = ksum(y) / n; + final yc = [for (final v in y) v - my]; + final span = ts.reduce(math.max) - ts.reduce(math.min); + final scale = 2.0 * span / (n - 1); + final out = []; + for (final hz in f) { + final w = 2 * math.pi * hz; + final tau = math.atan2(ksum([for (final v in ts) math.sin(2 * w * v)]), + ksum([for (final v in ts) math.cos(2 * w * v)])) / + (2 * w); + final c = [for (final v in ts) math.cos(w * (v - tau))]; + final s = [for (final v in ts) math.sin(w * (v - tau))]; + final cNum = ksum([for (var i = 0; i < n; i++) yc[i] * c[i]]); + final sNum = ksum([for (var i = 0; i < n; i++) yc[i] * s[i]]); + final cDen = ksum([for (final v in c) v * v]); + final sDen = ksum([for (final v in s) v * v]); + out.add(LsPoint(hz, 0.5 * (cNum * cNum / cDen + sNum * sNum / sDen) * scale)); + } + return LombScargle(out); +} + +void main() { + final grid = freqGrid(0.0033, 0.4, 120); + + test('epoch seconds and rebased seconds give the same spectrum', () { + final b = _beats(1200); + final epoch = [for (final v in b.t) v + _epoch]; + expect(epoch[17] - epoch[3], b.t[17] - b.t[3], reason: 'offset is exact'); + final a = lombScargle(epoch, b.y, grid)!; + final r = lombScargle(b.t, b.y, grid)!; + expect(_worstRel(a, r), lessThan(1e-11)); + }); + + test('epoch-second spectrum matches a compensated, shifted reference', () { + final b = _beats(1200, seed: 9); + final epoch = [for (final v in b.t) v + _epoch]; + final a = lombScargle(epoch, b.y, grid)!; + expect(_worstRel(a, _kahanReference(epoch, b.y, grid)), lessThan(1e-11)); + }); + + test('the shift is the first sample, so any time origin gives the same', () { + final b = _beats(400, seed: 2); + final base = lombScargle(b.t, b.y, grid)!; + for (final offset in [-1000.0, 3600.0, 1e6, 1.5e9]) { + final moved = lombScargle([for (final v in b.t) v + offset], b.y, grid)!; + expect(_worstRel(moved, base), lessThan(1e-11), reason: 'offset $offset'); + } + }); + + test('incremental state agrees with batch on epoch seconds, far inside 1e-8', + () { + final b = _beats(900, seed: 4); + final epoch = [for (final v in b.t) v + _epoch]; + final state = IncrementalLombScargle(grid); + for (final n in [10, 100, 450, 900]) { + final inc = state.sync(epoch.sublist(0, n), b.y.sublist(0, n))!; + final batch = lombScargle(epoch.sublist(0, n), b.y.sublist(0, n), grid)!; + expect(_worstRel(inc, batch), lessThan(1e-9), reason: 'n=$n'); + } + }); +} diff --git a/test/onehz/minute_energy_test.dart b/test/onehz/minute_energy_test.dart new file mode 100644 index 0000000..b8e1e50 --- /dev/null +++ b/test/onehz/minute_energy_test.dart @@ -0,0 +1,360 @@ +// Per-minute energy: the SAME computation `Calories.dailyEnergy` does, exposed +// minute by minute. Three things are pinned here: +// +// 1. dailyEnergy's outputs did not move (GOLDEN literals captured from the +// implementation BEFORE the refactor; `==`, not closeTo). The consuming app +// versions every derived day on these numbers. +// 2. the minute series folds back to dailyEnergy BIT-FOR-BIT. +// 3. a minute with nothing measured abstains — null and a reason, never a +// filled or interpolated value. + +import 'package:test/test.dart'; +import 'package:openstrap_analytics/src/onehz/workout/calories.dart'; +import 'support/energy_day.dart'; + +const _hrmax = 186.0; +const _rhr = 54.0; // gate = 54 + 0.40 * (186 - 54) = 106.8 bpm + +// Captured from dailyEnergy at 7fe67a7 (before minuteEnergy existed), over +// `energyDay()` with hrmax 186 / restingHr 54 / dayMinutes 1440. +// (total, active, basal, walking) +const _golden = { + 'male false': (8251.412972070117, 6460.162972070118, 1791.25, 0.0), + 'male true': ( + 8854.354873246793, + 7063.104873246794, + 1791.25, + 602.9419011766767 + ), + 'female false': (5546.635132551552, 4258.885132551552, 1287.75, 0.0), + 'female true': ( + 5980.096923613808, + 4692.346923613808, + 1287.75, + 433.4617910622563 + ), + 'nonbinary false': ( + 6563.416347695368, + 5008.291347695368, + 1555.1249999999998, + 0.0 + ), + 'nonbinary true': ( + 7086.877631194254, + 5531.752631194254, + 1555.1249999999998, + 523.4612834988859 + ), +}; +// First 700 minutes, dayMinutes 900, male, with cadence. +const _goldenPartial = + (4497.553602919252, 3378.0223529192513, 1119.53125, 311.26779559338775); + +void main() { + final day = energyDay(); + + group('dailyEnergy is unchanged (golden)', () { + for (final k in energyProfiles.keys) { + for (final withCad in [false, true]) { + test('$k cadence=$withCad', () { + final e = Calories.dailyEnergy(day.hr, + profile: energyProfiles[k]!, + hrmax: _hrmax, + restingHr: _rhr, + cadenceSpmPerMin: withCad ? day.cadence : null)!; + final g = _golden['$k $withCad']!; + expect(e.total, g.$1); + expect(e.active, g.$2); + expect(e.basal, g.$3); + expect(e.walking, g.$4); + }); + } + } + test('partial day, dayMinutes pro-rates basal', () { + final e = Calories.dailyEnergy(day.hr.sublist(0, 700), + profile: energyProfiles['male']!, + hrmax: _hrmax, + restingHr: _rhr, + dayMinutes: 900, + cadenceSpmPerMin: day.cadence.sublist(0, 700))!; + expect(e.total, _goldenPartial.$1); + expect(e.active, _goldenPartial.$2); + expect(e.basal, _goldenPartial.$3); + expect(e.walking, _goldenPartial.$4); + }); + }); + + group('minuteEnergy folds back to dailyEnergy exactly', () { + for (final k in energyProfiles.keys) { + for (final withCad in [false, true]) { + test('$k cadence=$withCad', () { + final cad = withCad ? day.cadence : null; + final d = Calories.dailyEnergy(day.hr, + profile: energyProfiles[k]!, + hrmax: _hrmax, + restingHr: _rhr, + cadenceSpmPerMin: cad)!; + final s = Calories.minuteEnergy(day.hr, + profile: energyProfiles[k]!, + hrmax: _hrmax, + restingHr: _rhr, + cadenceSpmPerMin: cad)!; + expect(s.minutes.length, day.hr.length); + expect(s.active, d.active); + expect(s.walking, d.walking); + // basal is a rate × minutes, not a sum of 1440 additions. + expect(s.basalKcalPerMin * 1440, d.basal); + expect(s.basalKcalPerMin * 1440 + s.active, d.total); + }); + } + } + + test('the app path: zero-HR minutes dropped (with their cadence) before ' + 'dailyEnergy equals the aligned series with that cadence masked', () { + // lib/compute/derivation_engine.dart wakeDayEnergy compacts the series. + final hr = []; + final cad = []; + for (var i = 0; i < day.hr.length; i++) { + if (day.hr[i] <= 0) continue; + hr.add(day.hr[i]); + cad.add(day.cadence[i]); + } + final compact = Calories.dailyEnergy(hr, + profile: energyProfiles['female']!, + hrmax: _hrmax, + restingHr: _rhr, + cadenceSpmPerMin: cad)!; + final s = Calories.minuteEnergy(day.hr, + profile: energyProfiles['female']!, + hrmax: _hrmax, + restingHr: _rhr, + cadenceSpmPerMin: [ + for (var i = 0; i < day.hr.length; i++) + day.hr[i] <= 0 ? null : day.cadence[i] + ])!; + expect(s.active, compact.active); + expect(s.walking, compact.walking); + }); + + test('partial series', () { + final s = Calories.minuteEnergy(day.hr.sublist(0, 700), + profile: energyProfiles['male']!, + hrmax: _hrmax, + restingHr: _rhr, + cadenceSpmPerMin: day.cadence.sublist(0, 700))!; + expect(s.active, _goldenPartial.$2); + expect(s.walking, _goldenPartial.$4); + expect(s.basalKcalPerMin * 900, _goldenPartial.$3); + }); + }); + + group('per-minute rules', () { + const p = WorkoutUserProfile( + weightKg: 80, heightCm: 180, age: 30, sex: 'male'); + final basal = Calories.mifflinBmrKcalDay(80, 180, 30, 'male') / 1440.0; + + MinuteEnergySeries run(List hr, {List? cad, List? at}) => + Calories.minuteEnergy(hr, + profile: p, + hrmax: _hrmax, + restingHr: _rhr, + cadenceSpmPerMin: cad, + epochMinutes: at)!; + + test('resting HR: measured rest, basal only', () { + final m = run([70.0]).minutes.single; + expect(m.source, MinuteEnergySource.rest); + expect(m.abstained, isNull); + expect(m.basal, basal); + expect(m.active, 0.0); + expect(m.walking, isNull); + expect(m.total, basal); + }); + + test('HR over the gate: Keytel surplus over basal', () { + final coeffs = Calories.resolveCoeffs('male'); + final surplus = Calories.activeKcalPerS(coeffs, 150.0, _hrmax, 80, 30) * + 60.0 - + basal; + final m = run([150.0]).minutes.single; + expect(m.source, MinuteEnergySource.hr); + expect(m.active, surplus); + expect(m.walking, isNull); + expect(m.total, basal + surplus); + }); + + test('no usable HR and no cadence abstains: null everywhere + reason', () { + for (final bad in [0.0, -3.0, double.nan, double.infinity]) { + final m = run([bad]).minutes.single; + expect(m.abstained, MinuteEnergyAbstain.noHr, reason: '$bad'); + expect(m.source, isNull); + expect(m.basal, isNull); + expect(m.active, isNull); + expect(m.walking, isNull); + expect(m.total, isNull); + } + }); + + test('a minute is computed from its own inputs only (no fill, no smoothing)', + () { + final a = run([70.0, 0.0, 150.0]).minutes; + final b = run([70.0, 0.0, 120.0]).minutes; + expect(a[1].total, isNull); + expect(b[1].total, isNull); + expect(a[0].total, b[0].total); + expect(a[2].total, isNot(b[2].total)); + }); + + test('cadence on a sub-flex minute bills walking surplus', () { + final m = run([80.0], cad: [120.0]).minutes.single; // 5 METs + expect(m.source, MinuteEnergySource.cadence); + expect(m.walking, (5.0 - 1.0) * basal); + expect(m.active, m.walking); + expect(m.total, basal + m.walking!); + }); + + test('cadence on an HR-billed minute is not billed twice', () { + final m = run([150.0], cad: [120.0]).minutes.single; + expect(m.source, MinuteEnergySource.hr); + expect(m.walking, isNull); + }); + + test('cadence below the CADENCE-Adults floor bills nothing', () { + expect(run([80.0], cad: [95.0]).minutes.single.source, + MinuteEnergySource.rest); + expect(run([0.0], cad: [95.0]).minutes.single.abstained, + MinuteEnergyAbstain.noHr); + }); + + test('cadence with no HR is still measured gait (same as dailyEnergy)', () { + final m = run([0.0], cad: [110.0]).minutes.single; + expect(m.source, MinuteEnergySource.cadence); + expect(m.walking, (4.0 - 1.0) * basal); // 110 spm = 4 METs + }); + + test('minute carries its epoch minute when given, else its index', () { + expect(run([70.0, 70.0]).minutes.map((m) => m.minute), [0, 1]); + expect(run([70.0, 70.0], at: [29000000, 29000007]).minutes + .map((m) => m.minute), [29000000, 29000007]); + }); + + test('coverage counters', () { + final s = run([70.0, 0.0, 150.0, double.nan]); + expect(s.coveredMinutes, 2); + expect(s.abstainedMinutes, 2); + }); + }); + + group('input contract', () { + const p = WorkoutUserProfile(); + test('unusable anchors → null, like dailyEnergy', () { + for (final (mx, rest) in [ + (186.0, 0.0), + (186.0, 190.0), + (double.nan, 54.0), + (186.0, double.nan), + ]) { + expect( + Calories.minuteEnergy([100.0], + profile: p, hrmax: mx, restingHr: rest), + isNull); + } + }); + test('misaligned cadence / epochMinutes throw', () { + expect( + () => Calories.minuteEnergy([100.0, 100.0], + profile: p, + hrmax: _hrmax, + restingHr: _rhr, + cadenceSpmPerMin: [null]), + throwsArgumentError); + expect( + () => Calories.minuteEnergy([100.0, 100.0], + profile: p, + hrmax: _hrmax, + restingHr: _rhr, + epochMinutes: [1]), + throwsArgumentError); + }); + test('empty input → empty series, zero everything', () { + final s = Calories.minuteEnergy(const [], + profile: p, hrmax: _hrmax, restingHr: _rhr)!; + expect(s.minutes, isEmpty); + expect(s.active, 0.0); + }); + }); + + group('hourlyRollup', () { + final s = Calories.minuteEnergy(day.hr, + profile: energyProfiles['male']!, + hrmax: _hrmax, + restingHr: _rhr, + cadenceSpmPerMin: day.cadence)!; + + test('24 hourly buckets over a full index-aligned day', () { + final h = Calories.hourlyRollup(s.minutes); + expect(h.length, 24); + expect([for (final b in h) b.hour], List.generate(24, (i) => i)); + }); + + test('hour sums are the sums of its covered minutes, with coverage', () { + final h = Calories.hourlyRollup(s.minutes); + for (final b in h) { + var active = 0.0, total = 0.0, cov = 0; + for (var m = b.hour * 60; m < b.hour * 60 + 60; m++) { + final r = s.minutes[m]; + if (r.abstained != null) continue; + active += r.active!; + total += r.total!; + cov++; + } + expect(b.coveredMinutes, cov); + expect(b.coverage, cov / 60.0); + expect(b.active, active); + expect(b.total, total); + } + expect(h.fold(0, (a, b) => a + b.active!), + closeTo(s.active, 1e-9)); + }); + + test('an hour with no covered minute is null, not zero', () { + final hr = [for (var i = 0; i < 120; i++) i < 60 ? 0.0 : 70.0]; + final h = Calories.hourlyRollup(Calories.minuteEnergy(hr, + profile: energyProfiles['male']!, + hrmax: _hrmax, + restingHr: _rhr)! + .minutes); + expect(h[0].coveredMinutes, 0); + expect(h[0].coverage, 0.0); + expect(h[0].total, isNull); + expect(h[0].active, isNull); + expect(h[0].basal, isNull); + expect(h[1].coveredMinutes, 60); + expect(h[1].total, isNotNull); + }); + + test('partial hour is a partial sum, flagged by coverage, not scaled', () { + final r = Calories.minuteEnergy([for (var i = 0; i < 30; i++) 70.0], + profile: energyProfiles['male']!, hrmax: _hrmax, restingHr: _rhr)!; + final h = Calories.hourlyRollup(r.minutes); + expect(h.single.coveredMinutes, 30); + expect(h.single.coverage, 0.5); + expect(h.single.basal, closeTo(30 * r.basalKcalPerMin, 1e-9)); // not 60 + }); + + test('epoch minutes bucket by local hour via minuteOffset', () { + // epoch minute 600 = 10:00 UTC. +330 (UTC+5:30) → 15:30. + final at = [600, 601, 629, 630]; + final r = Calories.minuteEnergy([70.0, 70.0, 70.0, 70.0], + profile: energyProfiles['male']!, + hrmax: _hrmax, + restingHr: _rhr, + epochMinutes: at)!; + final utc = Calories.hourlyRollup(r.minutes); + expect([for (final b in utc) b.hour], [10]); + final ist = Calories.hourlyRollup(r.minutes, minuteOffset: 330); + expect([for (final b in ist) b.hour], [15, 16]); + expect([for (final b in ist) b.coveredMinutes], [3, 1]); + }); + }); +} diff --git a/test/onehz/nonfinite_beat_times_test.dart b/test/onehz/nonfinite_beat_times_test.dart new file mode 100644 index 0000000..fa09042 --- /dev/null +++ b/test/onehz/nonfinite_beat_times_test.dart @@ -0,0 +1,135 @@ +// A non-finite beat time used to make rsaRespRate and hrvFreq walk their +// sub-windows forever (an infinite last time is never reached), which ran a +// test isolate out of memory on the dev box. They abstain now, and the +// streams refuse such a fold and answer a snapshot with the batch. +// cvhrApneaScreen / CvhrStream follow the same rule: CvhrStream took a NaN +// time and its snapshot then threw where the batch answered. +// hrvTime (and nocturnalRmssd, sleepSessionRmssdDetail, hrvTimeAggregator, +// IncrementalHrvTime) threw on one: "Infinity or NaN toInt". +import 'dart:typed_data'; + +import 'package:openstrap_analytics/aggregate.dart'; +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +void main() { + final nn = [for (var i = 0; i < 1500; i++) 900.0 + 40 * (i % 7)]; + final ts = []; + var t = 0.0; + for (final v in nn) { + t += v; + ts.add(t); + } + List withLast(double bad) => [...ts]..[ts.length - 1] = bad; + + for (final bad in [double.infinity, double.nan, double.negativeInfinity]) { + test('batch abstains on a last beat time of $bad', () { + final r = rsaRespRate(nn, withLast(bad), artifactFraction: 0); + expect(r.present, isFalse); + expect(r.note, 'non-finite beat time'); + final h = hrvFreq(nn, withLast(bad), artifactFraction: 0); + expect(h.present, isFalse); + expect(h.note, 'non-finite beat time'); + final c = cvhrApneaScreen(nn, withLast(bad), artifactFraction: 0); + expect(c.present, isFalse); + expect(c.note, 'non-finite beat time'); + }); + + test('streams refuse the fold, unchanged, on $bad', () { + final r = RsaRespStream()..fold(nn.sublist(0, 700), ts.sublist(0, 700)); + final h = HrvFreqStream()..fold(nn.sublist(0, 700), ts.sublist(0, 700)); + final c = CvhrStream()..fold(nn.sublist(0, 700), ts.sublist(0, 700)); + final rBefore = r.toJson(), hBefore = h.toJson(), cBefore = c.toJson(); + final badTs = [...ts.sublist(700, 800)]..[50] = bad; + expect(() => r.fold(nn.sublist(700, 800), badTs), throwsArgumentError); + expect(() => h.fold(nn.sublist(700, 800), badTs), throwsArgumentError); + expect(() => c.fold(nn.sublist(700, 800), badTs), throwsArgumentError); + expect(r.toJson(), rBefore); + expect(h.toJson(), hBefore); + expect(c.toJson(), cBefore); + }); + } + + test('a snapshot with an infinite tail time is the batch answer', () { + final r = RsaRespStream()..fold(nn.sublist(0, 1400), ts.sublist(0, 1400)); + final h = HrvFreqStream()..fold(nn.sublist(0, 1400), ts.sublist(0, 1400)); + final tailTs = withLast(double.infinity).sublist(1400); + final tailNn = nn.sublist(1400); + expect( + r.snapshot(tailNnMs: tailNn, tailNnTimesMs: tailTs, artifactFraction: 0) + .note, + 'non-finite beat time'); + expect( + h.snapshot(tailNnMs: tailNn, tailNnTimesMs: tailTs, artifactFraction: 0) + .note, + 'non-finite beat time'); + final c = CvhrStream()..fold(nn.sublist(0, 1400), ts.sublist(0, 1400)); + for (final bad in [double.infinity, double.nan]) { + final tail = withLast(bad).sublist(1400); + final got = + c.snapshot(tailNnMs: tailNn, tailNnTimesMs: tail, artifactFraction: 0); + expect(got.note, 'non-finite beat time'); + expect(got.note, + cvhrApneaScreen(nn, withLast(bad), artifactFraction: 0).note); + } + }); + + group('hrvTime family', () { + for (final bad in [double.infinity, double.nan, double.negativeInfinity]) { + test('hrvTime abstains on a beat time of $bad', () { + for (final at in [ts.length - 1, 0, 700]) { + final t = [...ts]..[at] = bad; + final m = hrvTime(nn, nnTimesMs: t); + expect(m.present, isFalse); + expect(m.note, 'non-finite beat time'); + } + // Without times nothing about them is read. + expect(hrvTime(nn).present, isTrue); + }); + + test('nocturnalRmssd abstains on a beat time of $bad', () { + final m = nocturnalRmssd(nn, withLast(bad)); + expect(m.present, isFalse); + expect(m.note, 'non-finite beat time'); + }); + + test('sleepSessionRmssdDetail abstains on a beat time of $bad', () { + final start = 1700000000; + final abs = [for (final x in ts) start * 1000.0 + x]; + abs[abs.length - 1] = bad; + final m = sleepSessionRmssdDetail(nn, abs, + startSec: start, endSec: start + 3 * 3600); + expect(m.present, isFalse); + expect(m.note, 'non-finite beat time'); + }); + + test('hrvTimeAggregator refuses the slice, state unchanged, on $bad', + () { + final agg = hrvTimeAggregator(originMs: ts.first); + Beats beats(List t) => Beats( + Float64List.fromList(nn.sublist(0, t.length)), + Float64List.fromList(t)); + final state = agg.fold(beats(ts.sublist(0, 700))); + final before = agg.encodeState(state); + final badTs = [...ts.sublist(0, 100)]..[50] = bad; + expect(() => agg.addSlice(state, beats(badTs), 0, badTs.length), + throwsArgumentError); + expect(agg.encodeState(state), before); + expect(() => agg.fold(beats(badTs)), throwsArgumentError); + // Times are ignored when not gap-aware, so NaN stays fine there. + final plain = hrvTimeAggregator(gapAware: false); + expect(() => plain.fold(beats(badTs)), returnsNormally); + }); + + test('IncrementalHrvTime snapshot with a tail time of $bad is the batch', + () { + final inc = IncrementalHrvTime(); + inc.sync(nn.sublist(0, 1400), nnTimesMs: ts.sublist(0, 1400)); + final got = inc.sync(nn, nnTimesMs: withLast(bad)); + expect(got.present, isFalse); + expect(got.note, 'non-finite beat time'); + expect(got.note, hrvTime(nn, nnTimesMs: withLast(bad)).note); + }); + } + }); +} diff --git a/test/onehz/phase0_parity_test.dart b/test/onehz/phase0_parity_test.dart new file mode 100644 index 0000000..c75508d --- /dev/null +++ b/test/onehz/phase0_parity_test.dart @@ -0,0 +1,1044 @@ +// PHASE 0 PARITY (round 15) - characterization tests. +// +// The `_ref*` functions below are FROZEN verbatim copies of the code as it +// stood before round 15 (CVHR baseline loop, 1 g reference loop, Welch segment +// scan, and the callers around them). The production code is being refactored +// for speed with a bit-identical contract, so every comparison here is exact: +// doubles with `identical`-style equality (== or both NaN), every result field. +// +// Do NOT "fix" or tidy the reference copies. If production output changes, this +// test is supposed to fail. +// +// Only deviation from a straight copy: the cardio stager reference calls the +// PUBLIC `cleanRrBeatsBetween` / `remFeaturesFromBeats` / `classifyCardioEpochs` +// where the original called private wrappers around them (same bodies). + +import 'dart:math' as math; + +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +// ───────────────────────── frozen CVHR reference ───────────────────────────── + +/// [p25, p50, p75] of [xs], or null when empty. Same linear-interpolated +/// percentile the rest of the package uses. +List? _refQuartiles(List xs) => xs.isEmpty + ? null + : [percentile(xs, 25)!, percentile(xs, 50)!, percentile(xs, 75)!]; + +Metric refCvhrApneaScreen( + List nnMs, + List nnTimesMs, { + required double artifactFraction, + double minWidthSec = 10, + double maxWidthSec = 120, + double depthWidthRatioMin = 0.7, // ms per second + double envWindowSec = 130, + double maxArtifact = 0.30, + double maxGapSec = 30, +}) { + const inputs = ['rr_cleaned', 'beat_times']; + if (nnMs.length < 60 || nnTimesMs.length != nnMs.length) { + return const Metric.absent( + tier: Tier.high, + inputs_used: inputs, + note: 'too few beats for a CVHR screen (need ≥60)', + ); + } + if (artifactFraction > maxArtifact) { + return Metric.absent( + tier: Tier.high, + inputs_used: inputs, + note: 'artifact fraction ${round6(artifactFraction)} > gate ' + '— CVHR dip detection unreliable', + ); + } + + final tSec = [for (final t in nnTimesMs) t / 1000.0]; + if (tSec.last - tSec.first <= 0) { + return const Metric.absent( + tier: Tier.high, + inputs_used: inputs, + note: 'degenerate beat times', + ); + } + + // SEGMENT AT GAPS. Beats more than [maxGapSec] apart are two recordings, not + // one: interpolating a straight line across the hole invents a flat stretch + // with no cycles in it, and charging the hole to `analyzedHours` divides a + // real cycle count by time we never observed. Both push the index DOWN, so a + // charging break used to read as an improvement. + final segStart = [0]; + for (var i = 1; i < tSec.length; i++) { + if (tSec[i] - tSec[i - 1] > maxGapSec) segStart.add(i); + } + + var cycleCount = 0; + var analyzedHours = 0.0; + final depths = []; // ms + final widths = []; // s + for (var s = 0; s < segStart.length; s++) { + final lo = segStart[s]; + final hi = (s + 1 < segStart.length ? segStart[s + 1] : tSec.length) - 1; + final segT = tSec.sublist(lo, hi + 1); + final segNn = nnMs.sublist(lo, hi + 1); + final spanSec = segT.last - segT.first; + // Too short to hold even one cycle plus its baseline window — it also can't + // be scored, so it contributes neither cycles nor hours. + if (segT.length < 60 || spanSec < 60) continue; + analyzedHours += spanSec / 3600.0; + _refScoreSegment( + segT, + segNn, + minWidthSec: minWidthSec, + maxWidthSec: maxWidthSec, + depthWidthRatioMin: depthWidthRatioMin, + envWindowSec: envWindowSec, + onCycle: (depth, width) { + cycleCount++; + depths.add(depth); + widths.add(width); + }, + ); + } + + if (analyzedHours <= 0) { + return const Metric.absent( + tier: Tier.high, + inputs_used: inputs, + note: 'no gap-free stretch long enough for a CVHR screen (<60 s)', + ); + } + + final perHour = cycleCount / analyzedHours; + final conf = ((1 - artifactFraction) * 0.85).clamp(0.2, 0.85); + return Metric( + value: CvhrResult( + cycleCount: cycleCount, + cvhrPerHour: perHour, + analyzedHours: analyzedHours, + meanDepthMs: depths.isEmpty ? null : mean(depths), + meanWidthSec: widths.isEmpty ? null : mean(widths), + depthQuartilesMs: _refQuartiles(depths), + widthQuartilesSec: _refQuartiles(widths), + ), + confidence: conf, + tier: Tier.high, + inputs_used: inputs, + note: 'CVHR/ACAT (Hayano) apnea SCREEN — NOT a diagnosis, NOT an AHI; ' + 'single-night CVHR has substantial night-to-night variability, ' + 'interpret as a trend over multiple nights', + ); +} + +/// Score ONE gap-free stretch: resample → smooth → sliding-median baseline → +/// positive-excursion runs → width + steepness gates. Calls [onCycle] with +/// (depthMs, widthSec) per surviving cycle. [t] in seconds, [nn] in ms. +void _refScoreSegment( + List t, + List nn, { + required double minWidthSec, + required double maxWidthSec, + required double depthWidthRatioMin, + required double envWindowSec, + required void Function(double depthMs, double widthSec) onCycle, +}) { + // Resample NN onto a uniform 1 Hz grid by piecewise-linear interpolation of + // the tachogram (NN vs beat time). This gives evenly-spaced points for the + // sliding-window baseline and the polynomial smoother. Only ever called on a + // stretch with no gap larger than the caller's tolerance, so no interpolation + // here spans a hole. + final t0 = t.first; + final nGrid = (t.last - t0).floor() + 1; + final grid = List.generate(nGrid, (i) => t0 + i); + final nnGrid = _refResampleLinear(t, nn, grid); + + // 2nd-order local-polynomial smoothing (Savitzky-Golay quadratic, ~7 s half + // window) to suppress beat jitter while keeping the cyclic shape. + final smooth = _refSavgolQuadratic(nnGrid, halfWin: 7); + + // Adaptive baseline over a ~130 s sliding median (spans ≥2 apnea cycles, so + // it tracks the eupneic level without being pulled into individual dips). + // The CVHR bradycardia shows up as a local NN MAXIMUM (NN ↑ ⇔ HR ↓) standing + // proud of this baseline; the following tachycardia is the recovery valley. + final halfEnv = (envWindowSec / 2).round(); + final base = List.filled(nGrid, 0); + for (var j = 0; j < nGrid; j++) { + final lo = math.max(0, j - halfEnv); + final hi = math.min(nGrid - 1, j + halfEnv); + base[j] = median(smooth.sublist(lo, hi + 1)) ?? smooth[j]; + } + // Baseline-subtracted bradycardic excursion (clip negatives — we only score + // the NN-up bradycardia, not the tachycardia trough). + final exc = [ + for (var j = 0; j < nGrid; j++) math.max(0.0, smooth[j] - base[j]) + ]; + + // Prominence threshold: a dip must clear a robust noise floor AND a fraction + // of the typical excursion amplitude. We anchor it BELOW the median positive + // excursion (Hayano's adaptive amplitude criterion is permissive enough to + // capture the whole bradycardia run, not just its tip) — a threshold at the + // peak would clip the run width to a few seconds and miss the cycle. + final posExc = [ + for (final e in exc) + if (e > 0) e + ]; + final excP75 = (posExc.isNotEmpty ? percentile(posExc, 75) : null) ?? 0; + // half of the upper-quartile excursion: well inside each genuine dip's run. + final prom = math.max(0.5 * excP75, 5.0); // ms + + // Find peaks of `exc`: local maxima exceeding `prom`, each isolated to one + // contiguous super-threshold run (so one bradycardia = one cycle). Measure + // the run width and the peak depth; apply the width + depth/width gates. + var i = 0; + while (i < nGrid) { + if (exc[i] <= prom) { + i++; + continue; + } + final start = i; + var peakDepth = 0.0; + while (i < nGrid && exc[i] > prom) { + if (exc[i] > peakDepth) peakDepth = exc[i]; + i++; + } + final widthSec = (i - start).toDouble(); // 1 Hz grid => samples == seconds + if (widthSec < minWidthSec || widthSec > maxWidthSec) continue; + // Depth-to-width steepness gate (ms per second): the bradycardia-tachycardia + // swing is steep; slow baseline drift is shallow per second. + if (peakDepth / widthSec < depthWidthRatioMin) continue; + onCycle(peakDepth, widthSec); + } +} + +/// Piecewise-linear resample of (t,y) onto a sorted [grid] of times (same unit). +/// Out-of-range grid points clamp to the nearest endpoint value. +List _refResampleLinear( + List t, List y, List grid) { + final out = List.filled(grid.length, 0); + var j = 0; + for (var k = 0; k < grid.length; k++) { + final g = grid[k]; + if (g <= t.first) { + out[k] = y.first; + continue; + } + if (g >= t.last) { + out[k] = y.last; + continue; + } + while (j < t.length - 1 && t[j + 1] < g) { + j++; + } + final t0 = t[j], t1 = t[j + 1]; + final span = t1 - t0; + final frac = span == 0 ? 0.0 : (g - t0) / span; + out[k] = y[j] + (y[j + 1] - y[j]) * frac; + } + return out; +} + +/// 2nd-order Savitzky-Golay smoothing via a local quadratic least-squares fit +/// over a symmetric window of half-width [halfWin]. Evaluated at the centre. +List _refSavgolQuadratic(List x, {int halfWin = 7}) { + final n = x.length; + final out = List.filled(n, 0); + for (var i = 0; i < n; i++) { + final lo = math.max(0, i - halfWin); + final hi = math.min(n - 1, i + halfWin); + final m = hi - lo + 1; + if (m < 3) { + out[i] = x[i]; + continue; + } + // Fit y = a + b·u + c·u² with u centred at i; return a (value at u=0). + var s0 = 0.0, s1 = 0.0, s2 = 0.0, s3 = 0.0, s4 = 0.0; + var sy = 0.0, suy = 0.0, su2y = 0.0; + for (var k = lo; k <= hi; k++) { + final u = (k - i).toDouble(); + final u2 = u * u; + s0 += 1; + s1 += u; + s2 += u2; + s3 += u2 * u; + s4 += u2 * u2; + sy += x[k]; + suy += u * x[k]; + su2y += u2 * x[k]; + } + // Solve the 3x3 normal equations for [a,b,c]; we only need a. + final a = _refSolve3( + [s0, s1, s2, s1, s2, s3, s2, s3, s4], + [sy, suy, su2y], + ); + out[i] = a == null ? x[i] : a[0]; + } + return out; +} + +/// Solve a 3x3 linear system (row-major A, rhs b) via Cramer's rule. Null if +/// singular. +List? _refSolve3(List a, List b) { + double det3(double a0, double a1, double a2, double a3, double a4, double a5, + double a6, double a7, double a8) => + a0 * (a4 * a8 - a5 * a7) - + a1 * (a3 * a8 - a5 * a6) + + a2 * (a3 * a7 - a4 * a6); + final d = det3(a[0], a[1], a[2], a[3], a[4], a[5], a[6], a[7], a[8]); + if (d == 0) return null; + final dx = det3(b[0], a[1], a[2], b[1], a[4], a[5], b[2], a[7], a[8]); + final dy = det3(a[0], b[0], a[2], a[3], b[1], a[5], a[6], b[2], a[8]); + final dz = det3(a[0], a[1], b[0], a[3], a[4], b[1], a[6], a[7], b[2]); + return [dx / d, dy / d, dz / d]; +} + +// ───────────────────────── frozen hrvFreq / Welch reference ────────────────── + +Metric refHrvFreq( + List nnMs, + List nnTimesMs, { + required double artifactFraction, + double hfArtifactGate = 0.15, + double oversample = 4.0, +}) { + const inputs = ['rr_cleaned', 'beat_times']; + if (nnMs.length < 16 || nnTimesMs.length != nnMs.length) { + return const Metric.absent( + tier: Tier.high, + inputs_used: inputs, + note: 'too few beats for a spectral estimate', + ); + } + + // Times in seconds for Hz output; tachogram value = NN (ms). + final tSec = [for (final t in nnTimesMs) t / 1000.0]; + final spanSec = tSec.last - tSec.first; + if (spanSec <= 0) { + return const Metric.absent( + tier: Tier.high, + inputs_used: inputs, + note: 'degenerate beat times', + ); + } + + // Each band gets its own segment length and its own resolution-matched grid. + // A band whose lowest frequency the record is too short to resolve returns + // null — absent, not a leakage-filled 0.0 (ULF used to be emitted as exactly + // 0.0 for any session over 333 s, which is not a 24-h record by any reading). + final ulf = + _refWelchBandPower(tSec, nnMs, 0.0003, 0.003, oversample: oversample); + final vlf = _refWelchBandPower(tSec, nnMs, 0.003, 0.04, oversample: oversample); + final lf = _refWelchBandPower(tSec, nnMs, 0.04, 0.15, oversample: oversample); + final hfRaw = _refWelchBandPower(tSec, nnMs, 0.15, 0.40, oversample: oversample); + if (lf == null && hfRaw == null) { + return const Metric.absent( + tier: Tier.high, + inputs_used: inputs, + note: 'record too short to resolve any HRV band', + ); + } + + final hfGated = artifactFraction > hfArtifactGate; + final hf = hfGated ? null : hfRaw; + + double? lfhf, nuLf, nuHf; + if (lf != null && hf != null && (lf + hf) > 0) { + lfhf = hf == 0 ? null : lf / hf; + nuLf = 100.0 * lf / (lf + hf); + nuHf = 100.0 * hf / (lf + hf); + } + // TOTAL POWER is by definition the sum over ALL bands (Task Force 1996). + // When HF is suppressed we do not have a trustworthy HF term, so a "total" + // is not computable: silently summing the withheld hfRaw back in republished + // exactly the quantity the gate withheld (the gated and ungated totals came + // out bit-identical), and dropping HF from the sum would republish a + // different quantity under the same name. Either way it would be dishonest — + // so total is WITHHELD alongside HF. + // + // A band the record could not RESOLVE is a different case, and it used to be + // handled with `?? 0` — which is exactly the absence-as-zero this package + // forbids, and it is not hypothetical: ULF needs 33 333 s of span (10 cycles + // at 0.0003 Hz) and a night is ~28 700 s, so ULF resolved on precisely no + // nights and the published total was silently the ULF-less sum. Withholding + // total instead would delete a rendered number on every night forever. So it + // is published with its composition NAMED in [totalBands]: the sum is over + // those bands and no others. + final bands = { + 'ulf': ulf, + 'vlf': vlf, + 'lf': lf, + 'hf': hfRaw + }; + final resolved = [ + for (final e in bands.entries) + if (e.value != null) e.key + ]; + final total = (hfGated || lf == null || hfRaw == null) + ? null + : [for (final b in resolved) bands[b]!].reduce((a, b) => a + b); + + // Confidence: penalize artifacts heavily; low-band-only reads still HIGH-ish. + final conf = ((1 - artifactFraction) * (hfGated ? 0.6 : 0.9)).clamp(0.2, 0.9); + return Metric( + value: HrvFreq( + ulf: ulf, + vlf: vlf, + lf: lf, + hf: hf, + total: total, + totalBands: total == null ? null : resolved, + lfhf: lfhf, + nuLf: nuLf, + nuHf: nuHf, + hfGated: hfGated, + ), + confidence: conf, + tier: Tier.high, + inputs_used: inputs, + note: hfGated + ? 'HF suppressed: artifact fraction ${round6(artifactFraction)} ' + '> gate — LF/VLF reported, HF/LF-HF/nu/total withheld' + : 'PRV spectrum; HF band quantization-limited at 1 Hz', + ); +} + +double? _refWelchBandPower( + List tSec, + List y, + double loHz, + double hiHz, { + double cyclesPerSegment = 10.0, + double oversample = 4.0, + int minPointsPerSegment = 16, +}) { + if (loHz <= 0 || hiHz <= loHz || tSec.length < minPointsPerSegment) + return null; + final span = tSec.last - tSec.first; + final segSec = cyclesPerSegment / loHz; + if (span < segSec) return null; // band not resolvable in this record + + final df = 1.0 / segSec / oversample; + final nGrid = ((hiHz - loHz) / df).ceil() + 1; + final grid = freqGrid(loHz, hiHz, nGrid); + + var sum = 0.0; + var k = 0; + final step = segSec / 2; // 50 % overlap, the Welch default + for (var start = tSec.first; start + segSec <= tSec.last; start += step) { + final end = start + segSec; + final ts = []; + final ys = []; + for (var i = 0; i < tSec.length; i++) { + if (tSec[i] < start) continue; + if (tSec[i] >= end) break; + ts.add(tSec[i]); + ys.add(y[i]); + } + // A segment has to be BOTH beat-dense and time-complete: a dropout in + // the middle leaves few beats spanning the full window, and its + // periodogram is a window function, not a spectrum. Endpoint span alone + // misses an internal dropout that still leaves beats near both edges, so + // also reject on the single largest gap between consecutive beats. + if (ts.length < minPointsPerSegment || ts.last - ts.first < segSec * 0.8) { + continue; + } + var maxGap = 0.0; + for (var i = 1; i < ts.length; i++) { + final g = ts[i] - ts[i - 1]; + if (g > maxGap) maxGap = g; + } + if (maxGap > segSec * 0.2) continue; + final ls = lombScargle(ts, ys, grid); + if (ls == null) continue; + final p = ls.bandPower(loHz, hiHz); + if (!p.isFinite) continue; + sum += p; + k++; + } + return k == 0 ? null : sum / k; +} + +// ───────────────────────── frozen cardio stager reference ──────────────────── + +CardioStagerResult refCardioStager( + List hr1hz, + List accel, { + List rrMs = const [], + List rrTsMs = const [], + int epochSec = 30, + SleepUserProfile? userProfile, +}) { + // Explicit arg wins (unit tests); else the ambient profile the edge set for + // this staging pass; else null ⇒ pure per-night-local (cold-start behavior). + final profile = userProfile ?? cardioUserProfile; + // `_cleanBeatsInWindow` binary-searches this, so it has to be non-decreasing. + // The producer guarantees it — beats inherit their record's second and the + // records are sorted, and the one path that folds in loose live beats + // re-sorts the pair explicitly. An `assert` rather than a linear fallback: + // a fallback would pay the scan back on every window to insure against a + // state the caller cannot reach, whereas this is free in release and loud in + // every test the moment somebody builds an unsorted series. + assert(() { + for (var i = 1; i < rrTsMs.length; i++) { + if (rrTsMs[i] < rrTsMs[i - 1]) return false; + } + return true; + }(), 'rrTsMs must be non-decreasing — the beat window is binary-searched'); + final n = math.min(hr1hz.length, accel.length); + // ── REAL-TIME epoch grid ────────────────────────────────────────────────── + // `n ~/ epochSec` conflated ARRAY POSITIONS with SECONDS. At 1 Hz they are + // the same number and everything below is unchanged; off 1 Hz they are not, + // and the failure was not an accuracy one — a 300 s band gave + // `96 ~/ 30 = 3` "epochs" of 2.5 REAL HOURS each, which the 30-s-tuned rules + // labelled NREM end to end and published as `wakePct = 0.0`: a night this + // stager could not read, reported as zero wake. No new parameter is needed + // for the fix — `accel[i].tsMs` is already the clock `_cleanBeatsInWindow` + // binary-searches, so the cadence is measurable from the input we have. + final cadenceSec = sampleCadenceSeconds( + [for (var i = 0; i < n; i++) accel[i].tsMs / 1000.0]); + // Coarser than one sample per epoch ⇒ there is no 30-s epoch to score, and + // the Webster/Cole-Kripke continuity rules below are specified at this epoch + // length. Stretching `epochSec` to fit the device would make a number appear + // where the evidence for it does not; abstain instead. + if (cadenceSec == null || cadenceSec > epochSec) return _refAbstain(epochSec); + final perEpoch = math.max(1, (epochSec / cadenceSec).round()); + // `perEpoch` samples at this cadence must actually SPAN `epochSec` — a + // cadence that does not divide it evenly (4 s against a 30 s epoch rounds + // `perEpoch` to 8, an 8 * 4 = 32 s epoch) drifts the real grid against the + // reported one. `epochSec` is not just a label: it is what the result + // reports (`CardioStagerResult.epochSec`) and what `consolidateSleepStages` + // / `_websterRescore` / `_mergeShortDeep` derive the Webster and bout + // thresholds from below, and the drift accumulates over the whole night. + // Abstain rather than publish a grid whose real spacing does not match its + // own label. + if ((perEpoch * cadenceSec - epochSec).abs() > 1e-6) + return _refAbstain(epochSec); + final nEpoch = n ~/ perEpoch; + if (nEpoch < 3) return _refAbstain(epochSec); + + // ── per-second ENMO (motion) against a LOCALLY-ADAPTIVE 1 g reference ────── + // A single whole-night gravity-magnitude reference (the old approach) is + // wrong on real WHOOP-4 units: the decoded gravity vector's magnitude is NOT + // perfectly orientation-invariant (per-axis gain/calibration isn't exact), + // so different STATIC sleep postures can read meaningfully apart in |accel| + // even though nothing is moving. Verified on a real overnight capture: a + // person lying rock-still for 30+ min (within-epoch stddev of |accel| < + // 0.0003 g) read |accel| = 1.0512 g against that whole-night's calibrated + // reference of 1.0348 g — a 0.0167 g "motion" score, ~3x the bigMove + // threshold, purely from holding a different (but equally static) posture + // than whichever one the night happened to calibrate against. Across that + // night, 389 of 421 "big move" epochs (92%) were this exact artifact (tiny + // within-epoch variance, large offset from the single global reference), + // not real movement — and the resulting misclassified WAKE blocks could not + // be fully bridged back by Webster rescore below. Recomputing the reference + // locally (a window wide enough to not react to real short movement bouts, + // narrow enough to track a genuine posture change within a few minutes) + // absorbs each posture as its own baseline instead of misreading it as + // sustained motion. + final mag = List.filled(n, 0); + for (var i = 0; i < n; i++) { + final a = accel[i]; + mag[i] = math.sqrt(a.x * a.x + a.y * a.y + a.z * a.z); + } + const int _gRefWinSec = 300; // 5 min window, centered per epoch + // …in SAMPLES at the measured cadence. 300 at 1 Hz, unchanged. + final gRefHalf = math.max(1, (_gRefWinSec / cadenceSec).round()) ~/ 2; + final gRefByEpoch = List.filled(nEpoch, 1.0); + for (var e = 0; e < nEpoch; e++) { + final es = e * perEpoch; + final lo = math.max(0, es - gRefHalf); + final hi = math.min(n, es + perEpoch + gRefHalf); + gRefByEpoch[e] = median(mag.sublist(lo, hi)) ?? 1.0; + } + + // ── per-epoch features ───────────────────────────────────────────────────── + final motion = List.filled(nEpoch, 0); + final hr = List.filled(nEpoch, double.nan); + final hrSd = List.filled(nEpoch, 0); + final rmssd = List.filled(nEpoch, double.nan); + final sdnn = List.filled(nEpoch, double.nan); + // REM autonomic features (P1): LF/HF from the RR spectrum + R(k) = rolling + // mean |ΔIHR|. REM's signature at 1 Hz beat-timing is a SYMPATHETIC shift — + // LF/HF rises, RR variability drops (low RMSSD), and instantaneous HR gets + // "jittery" (elevated |ΔIHR|). The old REM rule leaned only on the RMSSD drop + // and under-called REM; these two independent axes recover it (OR-combined + // below), gated by atonia + an HR floor so a plain arousal is not miscalled. + final lfhf = List.filled(nEpoch, double.nan); + final rk = List.filled(nEpoch, double.nan); + + for (var e = 0; e < nEpoch; e++) { + final s = e * perEpoch, t = math.min(s + perEpoch, n); + // motion = mean ENMO over the epoch, against THIS epoch's local reference. + final gRefE = gRefByEpoch[e]; + var ms = 0.0; + for (var i = s; i < t; i++) { + final d = mag[i] - gRefE; + ms += d > 0 ? d : 0.0; + } + motion[e] = ms / (t - s); + // hr mean/sd over valid (>0) seconds + final hv = [ + for (var i = s; i < t; i++) + if (hr1hz[i] > 0) hr1hz[i] + ]; + if (hv.isNotEmpty) hr[e] = mean(hv)!; + if (hv.length >= 2) hrSd[e] = stddev(hv) ?? 0; + // rmssd over RR beats within a ±2.5-min window centred on the epoch + rmssd[e] = _refWindowRmssd(rrMs, rrTsMs, accel, s, t, epochSec); + sdnn[e] = _refWindowSdnn(rrMs, rrTsMs, accel, s, t, epochSec); + // LF/HF + R(k) over a ±90-s RR window centred on the epoch. + final rem = _refWindowRemFeatures(rrMs, rrTsMs, accel, s, t, epochSec); + if (rem.lfhf != null) lfhf[e] = rem.lfhf!; + if (rem.rk != null) rk[e] = rem.rk!; + } + + return classifyCardioEpochs( + CardioEpochFeatures( + motion: motion, + hr: hr, + hrSd: hrSd, + rmssd: rmssd, + lfhf: lfhf, + rk: rk, + sdnn: sdnn, + ), + epochSec: epochSec, + userProfile: profile, + ); +} + +CardioStagerResult _refAbstain(int epochSec) => CardioStagerResult( + StagerResult( + stages: const [], + epochSec: epochSec, + wakePct: 0, + nremPct: 0, + remPct: 0, + ), + const [], + 0, + ); + +({List beats, List tsSec, List seam}) + _refCleanBeatsInWindow( + List rrMs, + List rrTsMs, + List accel, + int s, + int t, { + int halfWinMs = 150 * 1000, +}) { + const empty = (beats: [], tsSec: [], seam: []); + if (rrMs.isEmpty || rrTsMs.length != rrMs.length) return empty; + final mid = (s + t) ~/ 2; + if (mid >= accel.length) return empty; + final centreMs = accel[mid].tsMs; + return cleanRrBeatsBetween( + rrMs, rrTsMs, centreMs - halfWinMs, centreMs + halfWinMs); +} + +double _refWindowRmssd(List rrMs, List rrTsMs, + List accel, int s, int t, int epochSec) { + // Upstream's seam rule (merged 2026-10-18): skip seam pairs, gate on pairs. + final win = _refCleanBeatsInWindow(rrMs, rrTsMs, accel, s, t); + final beats = win.beats; + if (beats.length < 5) return double.nan; + var ss = 0.0; + var pairs = 0; + for (var i = 1; i < beats.length; i++) { + if (win.seam[i]) continue; + final d = beats[i] - beats[i - 1]; + ss += d * d; + pairs++; + } + return pairs >= 4 ? math.sqrt(ss / pairs) : double.nan; +} + +double _refWindowSdnn(List rrMs, List rrTsMs, + List accel, int s, int t, int epochSec) { + final beats = _refCleanBeatsInWindow(rrMs, rrTsMs, accel, s, t).beats; + if (beats.length < 5) return double.nan; + return stddev(beats) ?? double.nan; +} + +({double? lfhf, double? rk}) _refWindowRemFeatures(List rrMs, + List rrTsMs, List accel, int s, int t, int epochSec) { + final win = + _refCleanBeatsInWindow(rrMs, rrTsMs, accel, s, t, halfWinMs: 90 * 1000); + return remFeaturesFromBeats(win.beats, win.tsSec, win.seam); +} + +// ───────────────────────────── helpers ─────────────────────────────────────── + +bool sameD(double? a, double? b) { + if (a == null || b == null) return a == null && b == null; + if (a.isNaN || b.isNaN) return a.isNaN && b.isNaN; + return a == b && a.isNegative == b.isNegative; +} + +void expectSameD(double? a, double? b, String what) { + expect(sameD(a, b), isTrue, reason: '$what: $a vs $b'); +} + +void expectSameList(List? a, List? b, String what) { + if (a == null || b == null) { + expect(a == null && b == null, isTrue, reason: '$what nullness'); + return; + } + expect(a.length, b.length, reason: '$what length'); + for (var i = 0; i < a.length; i++) { + if (!sameD(a[i], b[i])) fail('$what[$i]: ${a[i]} vs ${b[i]}'); + } +} + +void expectSameMetric(Metric a, Metric b) { + expect(a.value == null, b.value == null, reason: 'value presence'); + expectSameD(a.confidence, b.confidence, 'confidence'); + expect(a.tier, b.tier); + expect(a.inputs_used, b.inputs_used); + expect(a.note, b.note); + expect(a.drivers == null, b.drivers == null); +} + +void expectSameCvhr(Metric a, Metric b) { + expectSameMetric(a, b); + final x = a.value, y = b.value; + if (x == null || y == null) return; + expect(x.cycleCount, y.cycleCount); + expectSameD(x.cvhrPerHour, y.cvhrPerHour, 'cvhrPerHour'); + expectSameD(x.analyzedHours, y.analyzedHours, 'analyzedHours'); + expectSameD(x.meanDepthMs, y.meanDepthMs, 'meanDepthMs'); + expectSameD(x.meanWidthSec, y.meanWidthSec, 'meanWidthSec'); + expectSameList(x.depthQuartilesMs, y.depthQuartilesMs, 'depthQuartiles'); + expectSameList(x.widthQuartilesSec, y.widthQuartilesSec, 'widthQuartiles'); +} + +void expectSameHrv(Metric a, Metric b) { + expectSameMetric(a, b); + final x = a.value, y = b.value; + if (x == null || y == null) return; + expectSameD(x.ulf, y.ulf, 'ulf'); + expectSameD(x.vlf, y.vlf, 'vlf'); + expectSameD(x.lf, y.lf, 'lf'); + expectSameD(x.hf, y.hf, 'hf'); + expectSameD(x.total, y.total, 'total'); + expect(x.totalBands, y.totalBands); + expectSameD(x.lfhf, y.lfhf, 'lfhf'); + expectSameD(x.nuLf, y.nuLf, 'nuLf'); + expectSameD(x.nuHf, y.nuHf, 'nuHf'); + expect(x.hfGated, y.hfGated); +} + +void expectSameStager(CardioStagerResult a, CardioStagerResult b) { + expect(a.base.stages, b.base.stages); + expect(a.base.epochSec, b.base.epochSec); + expectSameD(a.base.wakePct, b.base.wakePct, 'wakePct'); + expectSameD(a.base.nremPct, b.base.nremPct, 'nremPct'); + expectSameD(a.base.remPct, b.base.remPct, 'remPct'); + expect(a.deepFlag, b.deepFlag); + expectSameD(a.confidence, b.confidence, 'confidence'); +} + +// ───────────────────────────── generators ──────────────────────────────────── + +/// Beat series: NN with a ~[period]-s cyclic dip pattern plus noise. +/// gapEvery: mean beats between >30 s holes (0 = none) +/// nanFrac / dupFrac: fraction of NaN NN values / repeated timestamps +({List nn, List tMs}) genBeats( + int seed, + int n, { + double period = 55, + double amp = 60, + int gapEvery = 0, + double nanFrac = 0, + double dupFrac = 0, + double meanNn = 900, +}) { + final r = math.Random(seed); + final nn = []; + final t = []; + var tMs = 1000.0 * r.nextInt(1000); + for (var i = 0; i < n; i++) { + final ph = 2 * math.pi * (tMs / 1000.0) / period; + var v = meanNn + + amp * math.max(0.0, math.sin(ph)) - + 0.3 * amp * math.min(0.0, math.sin(ph)) + + (r.nextDouble() - 0.5) * 40; + if (nanFrac > 0 && r.nextDouble() < nanFrac) v = double.nan; + nn.add(v); + final isDup = dupFrac > 0 && i > 0 && r.nextDouble() < dupFrac; + if (!isDup) { + var step = v.isNaN ? meanNn : v; + if (gapEvery > 0 && r.nextInt(gapEvery) == 0) { + step += 31000 + r.nextInt(90000); + } + tMs += step; + } + t.add(tMs); + } + return (nn: nn, tMs: t); +} + +({List hr, List accel, List rr, List rrTs}) + genNight( + int seed, + int seconds, { + int cadenceSec = 1, + bool nanAccel = false, + bool hrDropouts = false, + bool rrOutliers = false, + bool withRr = true, +}) { + final r = math.Random(seed); + final n = seconds ~/ cadenceSec; + final hr = []; + final accel = []; + final rr = []; + final rrTs = []; + var posture = 1.0; + var moveLeft = 0; + var hrBase = 62.0; + for (var i = 0; i < n; i++) { + final ts = i * cadenceSec * 1000.0; + if (r.nextInt(900) == 0) posture = 0.95 + r.nextDouble() * 0.1; + if (moveLeft == 0 && r.nextInt(500) == 0) moveLeft = 5 + r.nextInt(40); + var m = posture + (r.nextDouble() - 0.5) * 0.002; + if (moveLeft > 0) { + m += r.nextDouble() * 0.3; + moveLeft--; + } + if (r.nextInt(2000) == 0) hrBase = 52 + r.nextDouble() * 25; + final cyc = 4 * math.sin(2 * math.pi * i * cadenceSec / 5400.0); + var h = hrBase + cyc + (r.nextDouble() - 0.5) * 4; + if (moveLeft > 0) h += 12; + if (hrDropouts && (i ~/ 200) % 7 == 3) h = 0; + if (hrDropouts && r.nextInt(300) == 0) h = double.nan; + hr.add(h); + var gx = m * 0.6; + final gy = m * 0.64, gz = m * 0.48; + if (nanAccel && r.nextInt(400) == 0) gx = double.nan; + accel.add(AccelSample(ts, gx, gy, gz)); + if (withRr) { + // beats inherit their record second => tied timestamps, ascending + final nb = h.isNaN || h <= 0 ? 0 : (h * cadenceSec / 60).round(); + for (var b = 0; b < nb; b++) { + var v = 60000.0 / h + + (r.nextDouble() - 0.5) * 60 + + 40 * math.sin(2 * math.pi * i * cadenceSec / 25.0); + if (rrOutliers && r.nextInt(60) == 0) { + v = r.nextBool() ? 150 : 2600; + } else if (rrOutliers && r.nextInt(80) == 0) { + v += 350; + } + rr.add(v); + rrTs.add(ts); + } + } + } + return (hr: hr, accel: accel, rr: rr, rrTs: rrTs); +} + +// ───────────────────────────── tests ───────────────────────────────────────── + +void main() { + group('CVHR baseline parity (frozen cvhrApneaScreen)', () { + void check(String name, int seed, int n, + {int gapEvery = 0, + double nanFrac = 0, + double dupFrac = 0, + double artifact = 0.05, + double period = 55}) { + test(name, () { + final b = genBeats(seed, n, + gapEvery: gapEvery, + nanFrac: nanFrac, + dupFrac: dupFrac, + period: period); + expectSameCvhr( + cvhrApneaScreen(b.nn, b.tMs, artifactFraction: artifact), + refCvhrApneaScreen(b.nn, b.tMs, artifactFraction: artifact), + ); + }); + } + + check('clean 1 h', 1, 3800); + check('clean 2 h, other period', 2, 7600, period: 83); + check('gaps', 3, 4000, gapEvery: 700); + check('many gaps (short segments dropped)', 4, 2500, gapEvery: 90); + // NaN NN values are deliberately NOT exercised here: a NaN reaching the run + // detection (`exc[i] <= prom` is false for NaN, and the inner + // `while (exc[i] > prom)` is false too) never advances `i` - an infinite loop + // in today's code, so no frozen output exists to compare against. + check('duplicate timestamps', 6, 3600, dupFrac: 0.05); + check('gaps + duplicates', 7, 4200, gapEvery: 600, dupFrac: 0.03); + check('exactly 60 beats', 8, 60); + check('59 beats (absent)', 9, 59); + check('short, 75 beats', 10, 75); + check('artifact gate (absent)', 11, 3600, artifact: 0.5); + + test('mismatched lengths and degenerate times', () { + final b = genBeats(12, 100); + expectSameCvhr( + cvhrApneaScreen(b.nn, b.tMs.sublist(1), artifactFraction: 0.0), + refCvhrApneaScreen(b.nn, b.tMs.sublist(1), artifactFraction: 0.0), + ); + final flat = List.filled(100, 5000.0); + expectSameCvhr( + cvhrApneaScreen(b.nn, flat, artifactFraction: 0.0), + refCvhrApneaScreen(b.nn, flat, artifactFraction: 0.0), + ); + }); + + test('non-default tunables', () { + final b = genBeats(13, 3600); + expectSameCvhr( + cvhrApneaScreen(b.nn, b.tMs, + artifactFraction: 0.1, + envWindowSec: 91, + minWidthSec: 8, + maxWidthSec: 100, + depthWidthRatioMin: 0.5, + maxGapSec: 20), + refCvhrApneaScreen(b.nn, b.tMs, + artifactFraction: 0.1, + envWindowSec: 91, + minWidthSec: 8, + maxWidthSec: 100, + depthWidthRatioMin: 0.5, + maxGapSec: 20), + ); + }); + + test('full 8 h night with gaps', () { + final b = genBeats(14, 30000, gapEvery: 6000, amp: 45); + final a = cvhrApneaScreen(b.nn, b.tMs, artifactFraction: 0.04); + expectSameCvhr( + a, refCvhrApneaScreen(b.nn, b.tMs, artifactFraction: 0.04)); + // Not vacuous: the synthetic dips must actually be scored. + expect(a.value!.cycleCount, greaterThan(0)); + }); + }); + + group('hrvFreq / Welch parity (frozen _welchBandPower)', () { + void check(String name, int seed, int n, + {int gapEvery = 0, + double nanFrac = 0, + double dupFrac = 0, + double artifact = 0.05, + double meanNn = 900}) { + test(name, () { + final b = genBeats(seed, n, + gapEvery: gapEvery, + nanFrac: nanFrac, + dupFrac: dupFrac, + meanNn: meanNn); + expectSameHrv( + hrvFreq(b.nn, b.tMs, artifactFraction: artifact), + refHrvFreq(b.nn, b.tMs, artifactFraction: artifact), + ); + }); + } + + check('clean 1.2 h', 21, 4500); + check('clean 1.5 h, faster heart', 22, 6000, meanNn: 700); + check('beat gaps reject segments', 23, 5000, gapEvery: 400); + check('NaN NN values', 24, 4000, nanFrac: 0.002); + check('duplicate timestamps', 25, 4000, dupFrac: 0.05); + check('gaps + NaN + duplicates', 26, 5000, + gapEvery: 900, nanFrac: 0.001, dupFrac: 0.02); + check('HF gated by artifact fraction', 27, 4000, artifact: 0.3); + check('short record, VLF unresolved', 28, 400); + check('16 beats', 29, 16); + check('15 beats (absent)', 30, 15); + check('very short span (nothing resolves)', 31, 50, meanNn: 400); + // 10 h of sparse beats so the ULF band (33 333 s segment) resolves too. + check('ULF resolves (10 h sparse)', 32, 6000, meanNn: 6000); + check('ULF with gaps', 33, 6000, meanNn: 6000, gapEvery: 1500); + + test('non-default oversample', () { + final b = genBeats(34, 3500); + expectSameHrv( + hrvFreq(b.nn, b.tMs, artifactFraction: 0.0, oversample: 2.5), + refHrvFreq(b.nn, b.tMs, artifactFraction: 0.0, oversample: 2.5), + ); + }); + + test('degenerate times and mismatched lengths', () { + final b = genBeats(35, 40); + expectSameHrv( + hrvFreq(b.nn, List.filled(40, 7000.0), artifactFraction: 0.0), + refHrvFreq(b.nn, List.filled(40, 7000.0), artifactFraction: 0.0), + ); + expectSameHrv( + hrvFreq(b.nn, b.tMs.sublist(2), artifactFraction: 0.0), + refHrvFreq(b.nn, b.tMs.sublist(2), artifactFraction: 0.0), + ); + }); + }); + + group('cardio stager parity (frozen 1 g reference loop + entry point)', () { + void check(String name, int seed, int seconds, + {int cadenceSec = 1, + bool nanAccel = false, + bool hrDropouts = false, + bool rrOutliers = false, + bool withRr = true}) { + test(name, () { + final g = genNight(seed, seconds, + cadenceSec: cadenceSec, + nanAccel: nanAccel, + hrDropouts: hrDropouts, + rrOutliers: rrOutliers, + withRr: withRr); + final a = cardioStager(g.hr, g.accel, rrMs: g.rr, rrTsMs: g.rrTs); + final b = refCardioStager(g.hr, g.accel, rrMs: g.rr, rrTsMs: g.rrTs); + expectSameStager(a, b); + // Not vacuous: normal inputs must actually stage something. + if (name.startsWith('staged')) { + expect(a.base.stages, isNotEmpty); + } + }); + } + + check('staged 40 min with RR', 41, 2400); + check('staged 50 min, RR outliers + HR dropouts', 42, 3000, + rrOutliers: true, hrDropouts: true); + check('staged 40 min, NaN accel magnitudes', 43, 2400, nanAccel: true); + check('staged 1 h without RR', 44, 3600, withRr: false); + check('staged 10 s cadence', 45, 7200, cadenceSec: 10); + check('5 s cadence (abstains: 30/5 grid)', 46, 7200, cadenceSec: 5); + check('60 s cadence (abstains)', 47, 14400, cadenceSec: 60); + check('short: 2 epochs (abstains)', 48, 70); + check('short: 3 epochs', 49, 95); + check('short: 100 s partial last epoch', 50, 100); + + test('mismatched hr/accel lengths and empty input', () { + final g = genNight(51, 1800); + expectSameStager( + cardioStager(g.hr.sublist(0, 1500), g.accel, + rrMs: g.rr, rrTsMs: g.rrTs), + refCardioStager(g.hr.sublist(0, 1500), g.accel, + rrMs: g.rr, rrTsMs: g.rrTs), + ); + expectSameStager(cardioStager(const [], const []), + refCardioStager(const [], const [])); + }); + + test('custom epochSec', () { + final g = genNight(52, 2400); + expectSameStager( + cardioStager(g.hr, g.accel, rrMs: g.rr, rrTsMs: g.rrTs, epochSec: 60), + refCardioStager(g.hr, g.accel, + rrMs: g.rr, rrTsMs: g.rrTs, epochSec: 60), + ); + }); + + test('long night (3 h)', () { + final g = genNight(53, 3 * 3600, hrDropouts: true, rrOutliers: true); + expectSameStager( + cardioStager(g.hr, g.accel, rrMs: g.rr, rrTsMs: g.rrTs), + refCardioStager(g.hr, g.accel, rrMs: g.rr, rrTsMs: g.rrTs), + ); + }); + }); +} diff --git a/test/onehz/rr_artifact_cap_test.dart b/test/onehz/rr_artifact_cap_test.dart new file mode 100644 index 0000000..773473c --- /dev/null +++ b/test/onehz/rr_artifact_cap_test.dart @@ -0,0 +1,336 @@ +// RED: RR artifact-run cap (combined bump, chunk A1). +// +// Contract: +// * `kMaxArtifactRun == 32` is the default of `correctRr(maxArtifactRun:)` and +// `RrCorrector(maxArtifactRun:)`; null means uncapped (today's behaviour). +// * Every artifact run <= K => output bit-identical to today's (the frozen +// reference in support/correct_rr_reference.dart): nn, nnTimesMs, classes, +// counts, cleanFraction. +// * A run LONGER than K splits the series: the output is +// correctRr(before) ++ correctRr(after), times kept absolute (the clock +// keeps running through the run), and the run's beats are ABSENT (a gap, +// not a correction). Each side is corrected as a series of its own, so its +// windows do not see the run. +// * The capped streaming `RrCorrector` equals the capped batch at every prefix +// (settled ++ snapshot tail), for any chunking and with JSON save/restore. +// * Its state is bounded: a checkpoint does not grow with an unbounded run. +// +// The long-run series are built so the split is unambiguous: clean integer RR, +// runs of 3000 ms beats (> 2000 is a hard artifact whatever the windows say), +// timestamps = exact cumulative sums, so every clock value is an integer and the +// comparison can be bit-exact. +import 'dart:convert'; +import 'dart:math' as math; + +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +import '../support/property.dart'; +import 'support/correct_rr_reference.dart'; +import 'support/rr_chunking.dart'; +import 'support/rr_compare.dart'; +import 'support/rr_runs.dart'; +import 'support/rr_synth.dart'; + +const _t0 = rrRunsT0; + +int _longestRun(List classes) { + var best = 0, cur = 0; + for (final c in classes) { + cur = c == BeatClass.normal ? 0 : cur + 1; + best = math.max(best, cur); + } + return best; +} + +/// What the cap promises for [runs] (each longer than K): the pieces between +/// the runs corrected on their own, their times moved onto the one clock. +void _expectSplit(RrCorrectionResult got, RunSeries s, List<(int, int)> runs, String why) { + final n = s.rr.length; + final bounds = <(int, int)>[]; + var from = 0; + for (final (a, b) in runs) { + bounds.add((from, a)); + from = b; + } + bounds.add((from, n)); + final nn = [], times = []; + for (final (a, b) in bounds) { + final piece = correctRr(s.rr.sublist(a, b), + rrTsMs: s.ts.sublist(a, b), maxArtifactRun: null); + final shift = (s.ts[a] - s.rr[a]) - (s.ts[0] - s.rr[0]); + nn.addAll(piece.nn); + times.addAll([for (final t in piece.nnTimesMs) t + shift]); + expectSameClasses(got.classes.sublist(a, b), piece.classes, + '$why: classes of piece [$a, $b)'); + } + for (final (a, b) in runs) { + // the run's own beats sit between the pieces: artifacts, absent from nn + expect(got.classes.sublist(a, b).every((c) => c != BeatClass.normal), isTrue, + reason: '$why: run [$a, $b) is all artifact'); + } + expect(got.classes.length, n, reason: '$why: one class per input beat'); + expectBitIdentical(got.nn, nn, '$why nn'); + expectBitIdentical(got.nnTimesMs, times, '$why nnTimesMs'); +} + +void main() { + test('kMaxArtifactRun is 32 (2x the longest run on 18 days of real data)', () { + expect(kMaxArtifactRun, 32); + }); + + group('runs <= K: bit-identical to today', () { + forAll('synthetic series whose longest artifact run fits', G.intIn(1, 100000), + (seed) { + final s = synthRr(SynthConfig( + seed: seed, + hours: 0.12, + ectopicPerMin: 1.5, + missedPerMin: 0.8, + extraPerMin: 0.8, + noiseRunPerMin: 0.6, + gapPerHour: 10)); + final want = correctRrReference(s.rr, rrTsMs: s.ts); + final longest = _longestRun(want.classes); + if (longest > kMaxArtifactRun) return; // another property's business + expectSameCorrection(correctRr(s.rr, rrTsMs: s.ts), want, 'default cap'); + expectSameCorrection( + correctRr(s.rr, rrTsMs: s.ts, maxArtifactRun: kMaxArtifactRun), + want, + 'K = 32'); + expectSameCorrection(correctRr(s.rr, rrTsMs: s.ts, maxArtifactRun: null), + want, 'uncapped'); + // The cap is a ceiling: K equal to the longest run changes nothing. + expectSameCorrection( + correctRr(s.rr, rrTsMs: s.ts, maxArtifactRun: math.max(1, longest)), + want, + 'K = longest run ($longest)'); + }, cases: 30, budget: const Duration(seconds: 60)); + + test('the checked-in real night and a real-shaped day', () { + final night = realNightRr()!; + final day = realShapedDay(); + for (final (name, d) in [('night', night), ('day', day)]) { + final want = correctRrReference(d.rr, rrTsMs: d.ts); + expect(_longestRun(want.classes), lessThanOrEqualTo(kMaxArtifactRun), + reason: '$name: precondition, no run over the cap'); + expectSameCorrection(correctRr(d.rr, rrTsMs: d.ts), want, name); + } + }); + + test('a run of exactly K is still corrected the old way (dropped, no split)', () { + final (s, runs) = withRuns(5, [300, 300], [kMaxArtifactRun], nearRun: true); + expect(runs.single.$2 - runs.single.$1, kMaxArtifactRun); + expectSameCorrection(correctRr(s.rr, rrTsMs: s.ts), + correctRrReference(s.rr, rrTsMs: s.ts), 'run == K'); + }); + }); + + group('a run longer than K is a gap', () { + test('K + 1 splits the series', () { + final (s, runs) = withRuns(6, [300, 300], [kMaxArtifactRun + 1]); + _expectSplit(correctRr(s.rr, rrTsMs: s.ts), s, runs, 'K+1'); + }); + + test('one 40-beat run, clean sides', () { + final (s, runs) = withRuns(7, [360, 360], [40]); + _expectSplit(correctRr(s.rr, rrTsMs: s.ts), s, runs, '40-run'); + }); + + test('isolated artifacts 20 beats either side of the run are judged without ' + 'the run in their window', () { + final (s, runs) = withRuns(8, [360, 360], [40], nearRun: true); + _expectSplit(correctRr(s.rr, rrTsMs: s.ts), s, runs, 'near-run artifacts'); + }); + + for (final seed in [11, 12, 13, 14, 15, 16]) { + test('two long runs, random lengths and sides (seed $seed)', () { + final r = Rng(seed); + final (s, runs) = withRuns( + seed, + [r.intIn(150, 400), r.intIn(150, 400), r.intIn(150, 400)], + [r.intIn(33, 90), r.intIn(33, 90)], + nearRun: r.nextBool()); + _expectSplit(correctRr(s.rr, rrTsMs: s.ts), s, runs, 'seed $seed'); + }); + } + + test('an explicit smaller K splits at that K', () { + final (s, runs) = withRuns(9, [300, 300], [6]); + _expectSplit(correctRr(s.rr, rrTsMs: s.ts, maxArtifactRun: 5), s, runs, 'K=5'); + // ...and the same run is under the default cap: not a split. + expectSameCorrection(correctRr(s.rr, rrTsMs: s.ts), + correctRrReference(s.rr, rrTsMs: s.ts), 'K=32 keeps the 6-run'); + }); + + test('uncapped (null) still drops the run and keeps one series', () { + final (s, _) = withRuns(10, [300, 300], [40]); + expectSameCorrection(correctRr(s.rr, rrTsMs: s.ts, maxArtifactRun: null), + correctRrReference(s.rr, rrTsMs: s.ts), 'null'); + }); + }); + + group('streaming RrCorrector with the cap', () { + void expectPrefix(RrCorrector c, List nn, List times, + List classes, RunSeries s, int n, int k, String why) { + final snap = c.snapshot(); + final want = correctRr(s.rr.sublist(0, n), + rrTsMs: s.ts.sublist(0, n), maxArtifactRun: k); + expectBitIdentical([...nn, ...snap.tailNn], want.nn, '$why nn'); + expectBitIdentical([...times, ...snap.tailNnTimes], want.nnTimesMs, '$why t'); + expectSameClasses([...classes, ...snap.tailClasses], want.classes, why); + expect(snap.droppedCount, want.droppedCount, reason: '$why dropped'); + expect(snap.correctedCount, want.correctedCount, reason: '$why corrected'); + expect(sameBits(snap.cleanFraction, want.cleanFraction), isTrue, + reason: '$why cleanFraction'); + } + + for (final k in [32, 5]) { + test('one beat at a time, every prefix, K = $k, a 40-beat run in the middle', () { + final (s, _) = withRuns(21, [110, 110], [40], nearRun: true); + final c = RrCorrector(maxArtifactRun: k); + final nn = [], times = []; + final classes = []; + for (var i = 0; i < s.rr.length; i++) { + final out = c.fold([s.rr[i]], tsMs: [s.ts[i]]); + nn.addAll(out.nn); + times.addAll(out.nnTimes); + classes.addAll(out.classes); + expectPrefix(c, nn, times, classes, s, i + 1, k, 'K=$k n=${i + 1}'); + } + }); + } + + for (final seed in [31, 32, 33]) { + test('random chunks with a JSON save/restore between them (seed $seed)', () { + final r = Rng(seed); + final (s, _) = withRuns( + seed, [r.intIn(120, 250), r.intIn(120, 250)], [r.intIn(33, 70)], + nearRun: true); + var c = RrCorrector(); + final nn = [], times = []; + final classes = []; + var at = 0; + for (final cut in randomCuts(math.Random(seed), s.rr.length)) { + final out = c.fold(s.rr.sublist(at, cut), tsMs: s.ts.sublist(at, cut)); + nn.addAll(out.nn); + times.addAll(out.nnTimes); + classes.addAll(out.classes); + at = cut; + c = RrCorrector.fromJson(jsonRoundTrip(c.toJson())); + expect(c.maxArtifactRun, kMaxArtifactRun, reason: 'cap survives restore'); + expectPrefix(c, nn, times, classes, s, at, kMaxArtifactRun, 'seed $seed n=$at'); + } + }); + } + + test('settling through a gap retains the new segment boundary', () { + final (s, _) = withRuns(51, [300, 700], [100], nearRun: true); + var c = RrCorrector(); + final nn = [], times = []; + final classes = []; + for (var i = 0; i < s.rr.length; i++) { + final out = c.fold([s.rr[i]], tsMs: [s.ts[i]]); + nn.addAll(out.nn); + times.addAll(out.nnTimes); + classes.addAll(out.classes); + if (i % 19 == 0 || i == s.rr.length - 1) { + c = RrCorrector.fromJson(jsonRoundTrip(c.toJson())); + expectPrefix(c, nn, times, classes, s, i + 1, kMaxArtifactRun, + 'across a settled gap at $i'); + } + } + }); + + test('maxArtifactRun survives the checkpoint (a value, the default, null)', () { + for (final k in [5, kMaxArtifactRun, null]) { + final c = RrCorrector(maxArtifactRun: k) + ..fold([for (var i = 0; i < 50; i++) 800.0 + i % 7], + tsMs: [for (var i = 0; i < 50; i++) _t0 + 800.0 * i]); + final back = RrCorrector.fromJson(jsonRoundTrip(c.toJson())); + expect(back.maxArtifactRun, k, reason: 'K = $k'); + } + expect(RrCorrector().maxArtifactRun, kMaxArtifactRun); + }); + + test('the checkpoint is bounded: 10,000 artifact beats do not grow it', () { + final r = Rng(41); + final clean = cleanRr(r, 300); + final c = RrCorrector(); + var t = _t0; + c.fold(clean, tsMs: [for (final v in clean) t += v]); + final before = jsonEncode(c.toJson()).length; + var worst = before; + for (var fed = 0; fed < 10000; fed += 100) { + final chunk = List.filled(100, 3000.0); + c.fold(chunk, tsMs: [for (final v in chunk) t += v]); + worst = math.max(worst, jsonEncode(c.toJson()).length); + } + expect(worst, lessThan(60000), reason: 'checkpoint chars during the run'); + expect(worst, lessThan(before * 3 + 2000), + reason: 'does not scale with the run (started at $before)'); + // and it resumes: after the run the series is a new segment, and the + // stream still equals the capped batch over everything it was fed. + final tail = cleanRr(r, 300); + final out = c.fold(tail, tsMs: [for (final v in tail) t += v]); + final rrAll = [...clean, ...List.filled(10000, 3000.0), ...tail]; + var u = _t0; + final all = RunSeries(rrAll, [for (final v in rrAll) u += v]); + final want = correctRr(rrAll, rrTsMs: all.ts); + final snap = c.snapshot(); + expect(snap.n, rrAll.length); + expect(out.nn.length, greaterThan(0), reason: 'the tail settles'); + expect(snap.droppedCount, want.droppedCount); + expect(snap.correctedCount, want.correctedCount); + expect(sameBits(snap.cleanFraction, want.cleanFraction), isTrue); + }); + }); + group('nnSourceIndex: which input beat each NN came from', () { + void expectSources(List rr, List ts, String why, + {int? cap = kMaxArtifactRun}) { + final got = correctRr(rr, rrTsMs: ts, maxArtifactRun: cap); + final src = got.nnSourceIndex; + expect(src.length, got.nn.length, reason: '$why length'); + for (var k = 0; k < src.length; k++) { + expect(src[k], inInclusiveRange(0, rr.length - 1), reason: '$why range'); + if (k > 0) expect(src[k], greaterThan(src[k - 1]), reason: '$why order'); + if (got.classes[src[k]] == BeatClass.normal) { + expect(sameBits(got.nn[k], rr[src[k]]), isTrue, reason: '$why value'); + } + } + // every beat that is not a source was dropped + expect(rr.length - src.length, got.droppedCount, reason: '$why dropped'); + } + + test('synthetic nights (corrected, dropped, gap beats)', () { + for (final seed in [3, 23, 77]) { + final d = synthRr(SynthConfig( + seed: seed, hours: 0.2, ectopicPerMin: 2, missedPerMin: 1, + extraPerMin: 1, noiseRunPerMin: 0.5, gapPerHour: 12)); + expectSources(d.rr, d.ts, 'seed $seed'); + expectSources(d.rr, d.ts, 'seed $seed uncapped', cap: null); + } + }); + + test('over the cap, tiny inputs, and empty', () { + final (s, runs) = withRuns(8, [120, 120, 120], [33, 45]); + expectSources(s.rr, s.ts, 'runs'); + final src = correctRr(s.rr, rrTsMs: s.ts).nnSourceIndex; + for (final (a, b) in runs) { + expect(src.where((i) => i >= a && i < b), isEmpty); + } + for (final n in [0, 1, 2, 3, 5]) { + expectSources(s.rr.sublist(0, n), s.ts.sublist(0, n), 'n=$n'); + } + }); + + test('a spline-corrected beat is its own source', () { + final rr = cleanRr(Rng(5), 200); + rr[100] = rr[100] * 0.5; + final ts = rrSeries(rr).ts; + final got = correctRr(rr, rrTsMs: ts); + expect(got.correctedCount, 1); + expect(got.nnSourceIndex, contains(100)); + }); + }); +} diff --git a/test/onehz/rr_correction_oracle_test.dart b/test/onehz/rr_correction_oracle_test.dart new file mode 100644 index 0000000..6fd1360 --- /dev/null +++ b/test/onehz/rr_correction_oracle_test.dart @@ -0,0 +1,322 @@ +// Guard for the `correctRr` speed rewrite: the production function must stay +// BIT-IDENTICAL to a frozen copy of the implementation it replaces +// (support/correct_rr_reference.dart) — nn, nnTimesMs, per-beat classes, counts +// and cleanFraction — on synthetic, real-shaped and hostile inputs. +// +// This passes against today's implementation by construction (it IS the frozen +// copy). It is the net under the rewrite, not a failing test. +import 'dart:math' as math; + +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +import 'support/correct_rr_reference.dart'; +import 'support/rr_compare.dart'; +import 'support/rr_synth.dart'; + +void _check(List rr, List? ts, String why, + {double alpha = 5.2, + int win = 91, + double floor = 100, + double reanchor = 1000}) { + final want = correctRrReference(rr, + rrTsMs: ts, + alpha: alpha, + windowBeats: win, + minThresholdMs: floor, + reanchorGapMs: reanchor); + final got = correctRr(rr, + rrTsMs: ts, + alpha: alpha, + windowBeats: win, + minThresholdMs: floor, + reanchorGapMs: reanchor); + expectSameCorrection(got, want, why); + // The input must not be modified. + expect(got.classes.length, rr.length, reason: '$why classes per input beat'); +} + +/// Timestamps for [rr] in one of several shapes. `null` = none supplied. +List? _timestamps(math.Random r, List rr, int mode) { + final n = rr.length; + var t = 1.7e12; + switch (mode) { + case 0: + return null; + case 1: // contiguous, whole-second like rec_ts*1000 + return [ + for (final v in rr) ((t += v) / 1000).floorToDouble() * 1000.0 + ]; + case 2: // dropouts of 3..600 s + return [ + for (final v in rr) + ((t += v + (r.nextDouble() < 0.01 ? 3000 + r.nextInt(600000) : 0)) / + 1000) + .floorToDouble() * + 1000.0 + ]; + case 3: // sub-second stamps + return [for (final v in rr) (t += v) + r.nextDouble() * 40]; + case 4: // counter-reset style: some stamps step backwards + return [ + for (final v in rr) + (t += v) - (r.nextDouble() < 0.02 ? 1000.0 * (1 + r.nextInt(3)) : 0) + ]; + case 5: // length mismatch: the oracle ignores the stamps entirely + return List.filled(math.max(0, n - 1), 1.7e12); + default: // every beat on the same stamp + return List.filled(n, 1.7e12); + } +} + +/// A hostile-ish RR series: drifting base, then bursts of every artefact shape. +List _randomRr(math.Random r, int n) { + final base = 500 + r.nextInt(700).toDouble(); + final rate = r.nextDouble() * 0.12; + final rr = []; + var drift = 0.0; + while (rr.length < n) { + drift += (r.nextDouble() - .5) * 8; + drift = drift.clamp(-200.0, 200.0); + final v = base + drift + (r.nextDouble() - .5) * 60; + if (r.nextDouble() < rate) { + switch (r.nextInt(7)) { + case 0: // ectopic + compensatory pause + rr..add((v * .6).roundToDouble())..add((v * 1.4).roundToDouble()); + case 1: // missed beat + rr.add((v * 2).roundToDouble()); + case 2: // extra beat + rr..add((v * .45).roundToDouble())..add((v * .55).roundToDouble()); + case 3: // multi-beat garbage run + for (var k = 3 + r.nextInt(12); k > 0; k--) { + rr.add(200 + r.nextInt(2500).toDouble()); + } + case 4: // saturated / zero / negative / absurd values + rr.add([0.0, -50.0, 2400.0, 5000.0, 1e6][r.nextInt(5)]); + case 5: // non-integer ms + rr.add(v + r.nextDouble()); + default: + rr.add(v.roundToDouble()); + } + } else { + rr.add(v.roundToDouble()); + } + } + return rr.sublist(0, n); +} + +void main() { + group('edge cases', () { + test('empty', () { + _check(const [], null, 'empty no ts'); + _check(const [], const [], 'empty with ts'); + }); + + test('fewer than the 3 beats the dispersion estimate needs', () { + final cases = >[ + [800], + [250], + [2500], + [300], + [2000], + [2001], + [800, 810], + [800, 250], + [2500, 2500], + [299, 300], + ]; + for (final rr in cases) { + _check(rr, null, 'short $rr no ts'); + _check(rr, [for (var i = 0; i < rr.length; i++) 1.7e12 + i * 800.0], + 'short $rr ts'); + } + }); + + test('every length around the window sizes, clean and dirty', () { + final r = math.Random(5); + final lengths = { + for (var n = 3; n <= 16; n++) n, + for (final c in [31, 45, 46, 47, 90, 91, 92, 93, 135, 136, 181, 182, 183]) + for (var d = -1; d <= 1; d++) c + d, + }; + for (final n in lengths) { + final rr = _randomRr(r, n); + _check(rr, null, 'n=$n no ts'); + _check(rr, _timestamps(r, rr, 2), 'n=$n ts'); + } + }); + + test('all artefacts', () { + final r = math.Random(6); + for (final n in [3, 4, 10, 100, 400]) { + _check(List.filled(n, 2500.0), null, 'all long n=$n'); + _check(List.filled(n, 250.0), null, 'all short n=$n'); + final rr = [for (var i = 0; i < n; i++) 250 + r.nextInt(2250).toDouble()]; + _check(rr, null, 'random garbage n=$n'); + _check(rr, _timestamps(r, rr, 1), 'random garbage ts n=$n'); + // alternating normal / artefact: no run is ever longer than 1 + final alt = [for (var i = 0; i < n; i++) i.isEven ? 800.0 : 2500.0]; + _check(alt, null, 'alternating n=$n'); + } + }); + + test('constant RR (QD = 0: the floor governs, massive ties)', () { + final rr = List.filled(500, 800); + _check(rr, null, 'constant'); + for (final at in [0, 1, 2, 250, 497, 498, 499]) { + final o = List.of(rr)..[at] = 1700; + _check(o, null, 'constant + outlier@$at'); + _check(o, [for (var i = 0; i < 500; i++) 1.7e12 + i * 800.0], + 'constant + outlier@$at ts'); + } + // two-valued series: dRR is mostly 0 with exact ties everywhere + final two = [for (var i = 0; i < 400; i++) i % 3 == 0 ? 810.0 : 800.0]; + _check(two, null, 'two-valued'); + }); + + test('artefact runs longer than the window, and at the edges', () { + final r = math.Random(7); + final rr = [ + for (var i = 0; i < 700; i++) + (i >= 150 && i < 290) + ? 250 + r.nextInt(2000).toDouble() + : (i >= 400 && i < 500 && i.isEven) + ? 2200 + : 800 + r.nextInt(40).toDouble() + ]; + _check(rr, null, 'long runs'); + _check(rr, _timestamps(r, rr, 1), 'long runs ts'); + final edges = [ + 2500, + 800, + 810, + 790, + 805, + for (var i = 0; i < 300; i++) 800 + (i % 7) * 3.0, + 2500 + ]; + _check(edges, null, 'artefacts at both ends'); + }); + + test('gaps, re-anchoring and timestamp shapes', () { + final r = math.Random(8); + final rr = _randomRr(r, 900); + for (var mode = 0; mode <= 6; mode++) { + _check(rr, _timestamps(r, rr, mode), 'ts mode $mode'); + } + // a dropout exactly at, just under and just over the re-anchor slack + for (final extra in [999.0, 1000.0, 1001.0]) { + var t = 1.7e12; + final ts = [ + for (var i = 0; i < 300; i++) + t += 800 + (i == 150 ? extra : 0) + ]; + _check(List.filled(300, 800.0), ts, 'slack extra=$extra'); + } + }); + + test('rrTsMs present vs absent changes only the clock', () { + final r = math.Random(9); + final rr = _randomRr(r, 600); + final ts = _timestamps(r, rr, 2)!; + final a = correctRr(rr, rrTsMs: ts); + final b = correctRr(rr); + expectSameClasses(a.classes, b.classes, 'classes ignore the clock'); + expectBitIdentical(a.nn, b.nn, 'nn ignores the clock'); + }); + + test('parameter grid', () { + final r = math.Random(10); + final rr = _randomRr(r, 450); + final ts = _timestamps(r, rr, 2); + for (final alpha in [5.2, 2.0, 8.0]) { + for (final win in [1, 2, 3, 4, 31, 61, 90, 91, 92, 121, 1000]) { + for (final floor in [100.0, 0.0, 250.0]) { + _check(rr, ts, 'a=$alpha w=$win f=$floor', + alpha: alpha, win: win, floor: floor); + } + } + } + for (final reanchor in [0.0, 400.0, 5000.0]) { + _check(rr, ts, 'reanchor=$reanchor', reanchor: reanchor); + } + }); + }); + + group('randomized', () { + // 90 independent series: random length, artefact mix, timestamp shape and + // parameters. Seeds are fixed so a failure names a reproducible case. + for (var seed = 0; seed < 90; seed++) { + test('seed $seed', () { + final r = math.Random(1000 + seed); + final n = [ + r.nextInt(12), + r.nextInt(200), + r.nextInt(900), + r.nextInt(2500) + ][r.nextInt(4)]; + final rr = _randomRr(r, n); + final ts = _timestamps(r, rr, r.nextInt(7)); + final custom = r.nextInt(4) == 0; + _check(rr, ts, 'seed=$seed n=$n', + alpha: custom ? [3.0, 4.5, 6.5][r.nextInt(3)] : 5.2, + win: custom ? [5, 31, 61, 121][r.nextInt(4)] : 91, + floor: custom ? [0.0, 50.0, 150.0][r.nextInt(3)] : 100, + reanchor: custom ? [0.0, 500.0, 3000.0][r.nextInt(3)] : 1000); + }); + } + }); + + group('real-shaped', () { + test('generator configs (clean, default, dirty, backwards-ts, noisy)', () { + final cfgs = [ + const SynthConfig( + seed: 21, + hours: 0.5, + ectopicPerMin: 0, + missedPerMin: 0, + extraPerMin: 0, + noiseRunPerMin: 0, + gapPerHour: 0), + const SynthConfig(seed: 22, hours: 0.6), + const SynthConfig( + seed: 23, + hours: 0.5, + ectopicPerMin: 2, + missedPerMin: 1, + extraPerMin: 1, + noiseRunPerMin: 0.5, + gapPerHour: 12), + const SynthConfig( + seed: 24, hours: 0.5, backwardsPerHour: 30, gapPerHour: 6), + const SynthConfig( + seed: 25, hours: 0.4, noiseRunPerMin: 2, gapPerHour: 30), + ]; + for (final c in cfgs) { + final s = synthRr(c); + _check(s.rr, s.ts, 'synth seed=${c.seed} ts'); + _check(s.rr, null, 'synth seed=${c.seed} no ts'); + } + }); + + test('real 8.9 h WHOOP 4 night: full, with and without timestamps', () { + final s = realNightRr(); + expect(s, isNotNull, reason: 'real night fixture missing'); + _check(s!.rr, s.ts, 'real night ts'); + _check(s.rr, null, 'real night no ts'); + }, timeout: const Timeout(Duration(minutes: 5))); + + test('real night prefixes (every window alignment)', () { + final s = realNightRr()!; + for (final n in [1, 2, 3, 4, 45, 46, 90, 91, 92, 181, 1000, 4999]) { + _check(s.rr.sublist(0, n), s.ts.sublist(0, n), 'real prefix n=$n'); + } + }, timeout: const Timeout(Duration(minutes: 5))); + + test('a whole 23 h day (~96k beats)', () { + final s = realShapedDay(); + expect(s.length, inInclusiveRange(90000, 100000)); + _check(s.rr, s.ts, 'day'); + }, timeout: const Timeout(Duration(minutes: 10))); + }); +} diff --git a/test/onehz/rr_correction_perf_test.dart b/test/onehz/rr_correction_perf_test.dart new file mode 100644 index 0000000..2aed018 --- /dev/null +++ b/test/onehz/rr_correction_perf_test.dart @@ -0,0 +1,48 @@ +// Performance guard for `correctRr`. +// +// Today every beat sorts five 91-element windows, so one call over a 23 h day +// (~96k beats) takes ~18 s on a loaded desktop and the app pays it on every +// derive pass. Sliding sorted windows give the same output in ~0.56 s (see +// test/onehz/rr_correction_oracle_test.dart for the bit-identical guard). +// +// The budgets are ~5x the rewrite's measured time, so a loaded CI host does not +// flake, and far below today's implementation, so this FAILS until the rewrite +// lands. Timing is wall clock around ONE call, JIT warm-up included. +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +import 'support/rr_synth.dart'; + +const _dayBudget = Duration(seconds: 3); // rewrite: 0.56 s; today: ~18 s +const _nightBudget = Duration(seconds: 1); // rewrite: 0.19 s; today: ~7 s + +void main() { + test('23 h day (~96k beats) corrects within ${_dayBudget.inSeconds} s', () { + // 5 h synthetic + the real 8.9 h night + 9 h synthetic = 96,712 beats. + final s = realShapedDay(); + expect(s.length, inInclusiveRange(90000, 100000)); + final sw = Stopwatch()..start(); + final out = correctRr(s.rr, rrTsMs: s.ts); + sw.stop(); + // ignore: avoid_print + print('correctRr: ${s.length} beats in ${sw.elapsedMilliseconds} ms'); + // Not a stub result: the call really processed the series. + expect(out.classes.length, s.length); + expect(sw.elapsed, lessThan(_dayBudget), + reason: '${sw.elapsedMilliseconds} ms for ${s.length} beats'); + }, timeout: const Timeout(Duration(minutes: 5))); + + test('real 8.9 h night (~32k beats) corrects within ' + '${_nightBudget.inSeconds} s', () { + final s = realNightRr(); + expect(s, isNotNull, reason: 'real night fixture missing'); + final sw = Stopwatch()..start(); + final out = correctRr(s!.rr, rrTsMs: s.ts); + sw.stop(); + // ignore: avoid_print + print('correctRr: ${s.length} beats in ${sw.elapsedMilliseconds} ms'); + expect(out.classes.length, s.length); + expect(sw.elapsed, lessThan(_nightBudget), + reason: '${sw.elapsedMilliseconds} ms for ${s.length} beats'); + }, timeout: const Timeout(Duration(minutes: 5))); +} diff --git a/test/onehz/sliding_median_test.dart b/test/onehz/sliding_median_test.dart new file mode 100644 index 0000000..947bc2b --- /dev/null +++ b/test/onehz/sliding_median_test.dart @@ -0,0 +1,178 @@ +// slidingMedians (round 15, phase 0) - RED until the implementation lands. +// +// Contract (r15-phase0.md section 1): for every j, +// out[j] == median(xs.sublist(max(0, j - halfWin), min(n - 1, j + halfWin) + 1)) +// bit for bit, where `median` is util.dart's (copy + List.sort default order, +// NaN last, linear-interpolated percentile at 50). +// +// Assumed signature: List slidingMedians(List xs, int halfWin) +// exported from package:openstrap_analytics/onehz.dart (via util.dart). + +import 'dart:math' as math; + +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +/// The definition, written independently of the implementation. +List naiveSlidingMedians(List xs, int halfWin) { + final n = xs.length; + return [ + for (var j = 0; j < n; j++) + median(xs.sublist( + math.max(0, j - halfWin), math.min(n - 1, j + halfWin) + 1))! + ]; +} + +bool sameD(double a, double b) { + if (a.isNaN || b.isNaN) return a.isNaN && b.isNaN; + return a == b && a.isNegative == b.isNegative; +} + +void expectBitIdentical(List xs, int halfWin) { + final got = slidingMedians(xs, halfWin); + final want = naiveSlidingMedians(xs, halfWin); + expect(got.length, want.length, reason: 'length xs=$xs h=$halfWin'); + for (var j = 0; j < want.length; j++) { + if (!sameD(got[j], want[j])) { + fail('j=$j h=$halfWin xs=$xs: got ${got[j]}, want ${want[j]}'); + } + } +} + +void main() { + const specials = [ + double.nan, + double.infinity, + double.negativeInfinity, + 0.0, + -0.0, + ]; + + test('empty input and zero window', () { + expect(slidingMedians(const [], 0), isEmpty); + expect(slidingMedians(const [], 5), isEmpty); + expectBitIdentical([3.5], 0); + expectBitIdentical([3.5], 7); + final xs = [4.0, 1.0, 9.0, 2.0]; + expect(slidingMedians(xs, 0), xs); // halfWin 0: each window is one value + }); + + test('does not mutate or alias its input', () { + final xs = [5.0, 1.0, 4.0, 2.0, 3.0]; + final copy = List.of(xs); + final out = slidingMedians(xs, 2); + expect(xs, copy); + expect(identical(out, xs), isFalse); + }); + + test('exhaustive small: n 0..20, halfWin 0..12, distinct values', () { + final r = math.Random(1); + for (var n = 0; n <= 20; n++) { + for (var h = 0; h <= 12; h++) { + expectBitIdentical([for (var i = 0; i < n; i++) r.nextDouble() * 100 - 50], h); + } + } + }); + + test('exhaustive small: heavy ties', () { + final r = math.Random(2); + for (var n = 0; n <= 20; n++) { + for (var h = 0; h <= 12; h++) { + for (var rep = 0; rep < 3; rep++) { + expectBitIdentical([for (var i = 0; i < n; i++) r.nextInt(3).toDouble()], h); + } + } + } + }); + + test('exhaustive small: NaN, +-inf, +-0.0 mixed with ties', () { + final r = math.Random(3); + for (var n = 0; n <= 20; n++) { + for (var h = 0; h <= 12; h++) { + for (var rep = 0; rep < 4; rep++) { + final xs = [ + for (var i = 0; i < n; i++) + r.nextInt(3) == 0 + ? specials[r.nextInt(specials.length)] + : r.nextInt(4).toDouble() - 1, + ]; + expectBitIdentical(xs, h); + } + } + } + }); + + test('all-special and all-equal series', () { + for (final v in specials) { + for (var n = 1; n <= 9; n++) { + for (var h = 0; h <= 5; h++) { + expectBitIdentical(List.filled(n, v), h); + } + } + } + for (final pair in [ + [double.nan, double.infinity], + [double.negativeInfinity, double.infinity], + [double.negativeInfinity, double.nan], + [-0.0, 0.0], + ]) { + for (var n = 2; n <= 12; n++) { + for (var h = 0; h <= 6; h++) { + expectBitIdentical([for (var i = 0; i < n; i++) pair[i % 2]], h); + } + } + } + }); + + test('infinities make even-window interpolation NaN exactly as the definition', + () { + // inf - inf and (-inf) + inf in the interpolation: the result must match + // the naive percentile arithmetic, not a "smarter" average. + expectBitIdentical([double.infinity, double.infinity], 1); + expectBitIdentical([double.negativeInfinity, double.infinity], 1); + expectBitIdentical([double.negativeInfinity, double.negativeInfinity, 1.0], 1); + expectBitIdentical([1.0, double.nan, 2.0, double.nan], 1); + }); + + test('halfWin >= n and very large halfWin', () { + final r = math.Random(4); + for (final n in [1, 2, 5, 17, 40]) { + final xs = [for (var i = 0; i < n; i++) r.nextDouble()]; + for (final h in [n - 1, n, n + 1, 3 * n, 1 << 30]) { + expectBitIdentical(xs, h); + } + } + }); + + test('monotone and reversed runs (window slides over sorted data)', () { + for (final n in [10, 33]) { + for (final h in [1, 3, 8]) { + expectBitIdentical([for (var i = 0; i < n; i++) i.toDouble()], h); + expectBitIdentical([for (var i = 0; i < n; i++) (n - i).toDouble()], h); + } + } + }); + + test('production-sized: CVHR window (halfWin 65) over 3000 s', () { + final r = math.Random(5); + final xs = [ + for (var i = 0; i < 3000; i++) + 900 + 60 * math.sin(i / 9.0) + (r.nextDouble() - 0.5) * 30 + ]; + expectBitIdentical(xs, 65); + // quantised values => many ties + expectBitIdentical([for (final x in xs) x.roundToDouble()], 65); + }); + + test('production-sized with sprinkled NaN and inf', () { + final r = math.Random(6); + final xs = [ + for (var i = 0; i < 1500; i++) + r.nextInt(40) == 0 + ? specials[r.nextInt(specials.length)] + : (r.nextDouble() * 20).roundToDouble() + ]; + expectBitIdentical(xs, 65); + expectBitIdentical(xs, 165); + }); +} diff --git a/test/onehz/stage_minute_ranges_test.dart b/test/onehz/stage_minute_ranges_test.dart new file mode 100644 index 0000000..2769534 --- /dev/null +++ b/test/onehz/stage_minute_ranges_test.dart @@ -0,0 +1,245 @@ +// SLEEP — per-stage minute RANGES (D27 revised, 2026-10-19). RED: new API. +// +// The cardio stager has no per-epoch probabilities, only robust-z scores +// against two fixed cutoffs plus a night confidence from RR coverage. So a +// stage's minutes get a BOUND, not a statistical interval: +// +// lo[s] = epochs that can only be s: covered, labelled s, and every score +// that decided the label clear of its cutoff by >= margin; +// hi[s] = lo[s] + epochs that could still be s: +// • uncovered epochs (no usable HR in the epoch, or no stage label at +// all) could be ANY stage; +// • an asleep epoch whose score is within `margin` of a cutoff counts +// toward both neighbouring stages (remScore ↔ REM / its NREM +// substage; deepScore ↔ light / deep). +// nrem = light ∪ deep (an epoch counts toward nrem.lo when every stage it +// could be is light or deep). +// +// API chosen here (lib/src/onehz/sleep/stage_ranges.dart, exported through +// `package:openstrap_analytics/onehz.dart`): +// +// class StageEpochEvidence { +// const StageEpochEvidence({SleepStage? stage, bool deep = false, +// bool covered = true, double? remScore, double? deepScore}); +// } +// class MinuteRange { final double lo, hi; } +// class StageMinuteRanges { +// final MinuteRange wake, light, deep, rem, nrem; +// final double windowMin; +// Map> toJson(); // wake/light/deep/rem → {lo,hi} +// } +// const double kStageScoreMargin; // calibrated, > 0 +// StageMinuteRanges? stageMinuteRanges(List? epochs, +// {int epochSec = 30, double remScoreCut = kDefaultRemScoreCut, +// double deepScoreCut = kDefaultDeepScoreCut, +// double margin = kStageScoreMargin}); +// extension on CardioStagerResult: List get epochEvidence +// — one entry per base.stages epoch, stage/deep equal to the FINAL labels, +// covered == the epoch carried at least one valid HR sample. +// +// "Point" minutes below are the stager's own labelled minutes (what the edge +// already shows); the range never moves the point, it brackets it. + +import 'dart:math' as math; + +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +const int _t0 = 1700000000; + +StageEpochEvidence _w() => const StageEpochEvidence(stage: SleepStage.wake); +StageEpochEvidence _l() => const StageEpochEvidence( + stage: SleepStage.nrem, remScore: -1.5, deepScore: -1.5); +StageEpochEvidence _d() => const StageEpochEvidence( + stage: SleepStage.nrem, deep: true, remScore: -1.5, deepScore: 1.5); +StageEpochEvidence _r() => const StageEpochEvidence( + stage: SleepStage.rem, remScore: 2.0, deepScore: -1.5); +StageEpochEvidence _u() => + const StageEpochEvidence(stage: SleepStage.nrem, covered: false); + +void _expectRange(MinuteRange r, double lo, double hi, String what) { + expect(r.lo, closeTo(lo, 1e-9), reason: '$what.lo'); + expect(r.hi, closeTo(hi, 1e-9), reason: '$what.hi'); +} + +void main() { + test('the margin is a real, positive number', () { + expect(kStageScoreMargin, greaterThan(0)); + expect(kStageScoreMargin.isFinite, isTrue); + }); + + group('absent input abstains', () { + test('null ⇒ null', () => expect(stageMinuteRanges(null), isNull)); + test('empty ⇒ null', () => expect(stageMinuteRanges(const []), isNull)); + }); + + test('every epoch covered and clear ⇒ lo == hi == point for every stage', () { + // 20 wake, 60 light, 30 deep, 40 rem epochs = 10/30/15/20 min. + final ev = [ + for (var i = 0; i < 20; i++) _w(), + for (var i = 0; i < 60; i++) _l(), + for (var i = 0; i < 30; i++) _d(), + for (var i = 0; i < 40; i++) _r(), + ]; + final r = stageMinuteRanges(ev, margin: 0.2)!; + _expectRange(r.wake, 10, 10, 'wake'); + _expectRange(r.light, 30, 30, 'light'); + _expectRange(r.deep, 15, 15, 'deep'); + _expectRange(r.rem, 20, 20, 'rem'); + _expectRange(r.nrem, 45, 45, 'nrem'); + expect(r.windowMin, closeTo(75, 1e-9)); + }); + + test('nothing covered ⇒ every stage is [0, window]', () { + final ev = [for (var i = 0; i < 120; i++) _u()]; + final r = stageMinuteRanges(ev)!; + for (final (name, range) in [ + ('wake', r.wake), + ('light', r.light), + ('deep', r.deep), + ('rem', r.rem), + ('nrem', r.nrem), + ]) { + _expectRange(range, 0, 60, name); + } + }); + + test('an epoch with no stage label at all is uncovered, not dropped', () { + final ev = [ + for (var i = 0; i < 10; i++) _l(), + for (var i = 0; i < 10; i++) const StageEpochEvidence(), + ]; + final r = stageMinuteRanges(ev, margin: 0.2)!; + expect(r.windowMin, closeTo(10, 1e-9)); + _expectRange(r.light, 5, 10, 'light'); + _expectRange(r.wake, 0, 5, 'wake'); + _expectRange(r.rem, 0, 5, 'rem'); + }); + + test('uncovered epochs widen every stage by their minutes', () { + // 40 covered light + 20 uncovered (stager fell through to NREM there). + final ev = [ + for (var i = 0; i < 40; i++) _l(), + for (var i = 0; i < 20; i++) _u(), + ]; + final r = stageMinuteRanges(ev, margin: 0.2)!; + _expectRange(r.light, 20, 30, 'light'); + _expectRange(r.wake, 0, 10, 'wake'); + _expectRange(r.deep, 0, 10, 'deep'); + _expectRange(r.rem, 0, 10, 'rem'); + _expectRange(r.nrem, 20, 30, 'nrem'); + }); + + test('a REM score inside the margin counts toward REM and its NREM neighbour', + () { + const cut = kDefaultRemScoreCut; + final ev = [ + for (var i = 0; i < 10; i++) _l(), + // labelled REM, but only 0.05 above the cutoff (margin 0.2) + for (var i = 0; i < 10; i++) + const StageEpochEvidence(stage: SleepStage.rem, remScore: cut + 0.05), + // labelled light, 0.05 below the cutoff + for (var i = 0; i < 10; i++) + const StageEpochEvidence(stage: SleepStage.nrem, remScore: cut - 0.05), + ]; + final r = stageMinuteRanges(ev, margin: 0.2)!; + _expectRange(r.rem, 0, 10, 'rem'); + _expectRange(r.light, 5, 15, 'light'); + _expectRange(r.deep, 0, 0, 'deep'); + _expectRange(r.wake, 0, 0, 'wake'); + }); + + test('a deep score inside the margin counts toward light and deep', () { + const cut = kDefaultDeepScoreCut; + final ev = [ + for (var i = 0; i < 10; i++) + const StageEpochEvidence( + stage: SleepStage.nrem, + deep: true, + remScore: -1.5, + deepScore: cut + 0.05), + for (var i = 0; i < 10; i++) _d(), + ]; + final r = stageMinuteRanges(ev, margin: 0.2)!; + _expectRange(r.deep, 5, 10, 'deep'); + _expectRange(r.light, 0, 5, 'light'); + // An epoch that is light-or-deep is certainly NREM. + _expectRange(r.nrem, 10, 10, 'nrem'); + }); + + test('scores exactly at the margin are clear', () { + const cut = kDefaultRemScoreCut; + final ev = [ + for (var i = 0; i < 10; i++) + const StageEpochEvidence(stage: SleepStage.rem, remScore: cut + 0.25), + ]; + final r = stageMinuteRanges(ev, margin: 0.25)!; + _expectRange(r.rem, 5, 5, 'rem'); + }); + + test('epochSec scales minutes', () { + final ev = [for (var i = 0; i < 12; i++) _w()]; + final r = stageMinuteRanges(ev, epochSec: 60)!; + _expectRange(r.wake, 12, 12, 'wake'); + expect(r.windowMin, closeTo(12, 1e-9)); + }); + + test('toJson carries the four display stages as {lo, hi}', () { + final r = stageMinuteRanges([_w(), _l(), _d(), _r(), _u()], margin: 0.2)!; + final j = r.toJson(); + expect(j.keys.toSet(), {'wake', 'light', 'deep', 'rem'}); + for (final v in j.values) { + expect(v.keys.toSet(), {'lo', 'hi'}); + expect(v['lo']!, lessThanOrEqualTo(v['hi']!)); + } + expect(j['wake'], {'lo': 0.5, 'hi': 1.0}); + }); + + group('CardioStagerResult.epochEvidence', () { + // 3 h still night at a steady 55 bpm with a 30-min off-skin gap in the + // middle (HR 0), RR absent: the gap's epochs must come out uncovered. + final n = 3 * 3600; + final accel = [ + for (var i = 0; i < n; i++) + AccelSample((_t0 + i) * 1000.0, 0.0, 0.0, 1.0) + ]; + final rng = math.Random(7); + final hr = [ + for (var i = 0; i < n; i++) + (i >= 3600 && i < 5400) ? 0.0 : 55 + rng.nextDouble() * 4 + ]; + final res = cardioStager(hr, accel); + + test('one entry per staged epoch, labels equal the final stages', () { + expect(res.base.stages, isNotEmpty); + final ev = res.epochEvidence; + expect(ev.length, res.base.stages.length); + for (var e = 0; e < ev.length; e++) { + expect(ev[e].stage, res.base.stages[e], reason: 'epoch $e stage'); + expect(ev[e].deep, res.deepFlag[e], reason: 'epoch $e deep'); + } + }); + + test('epochs with no valid HR are uncovered, the rest covered', () { + final ev = res.epochEvidence; + for (var e = 0; e < ev.length; e++) { + final inGap = e * 30 >= 3600 && (e + 1) * 30 <= 5400; + expect(ev[e].covered, !inGap, reason: 'epoch $e'); + } + }); + + test('the gap widens every stage range by its 30 minutes', () { + final r = stageMinuteRanges(res.epochEvidence)!; + expect(r.windowMin, closeTo(res.base.stages.length * 0.5, 1e-9)); + for (final range in [r.wake, r.light, r.deep, r.rem]) { + expect(range.hi - range.lo, greaterThanOrEqualTo(30 - 1e-9)); + } + }); + + test('an abstaining stager yields no evidence and therefore no range', () { + final off = cardioStager(List.filled(n, 0.0), accel); + expect(off.epochEvidence, isEmpty); + expect(stageMinuteRanges(off.epochEvidence), isNull); + }); + }); +} diff --git a/test/onehz/support/causal_night.dart b/test/onehz/support/causal_night.dart new file mode 100644 index 0000000..2c6486d --- /dev/null +++ b/test/onehz/support/causal_night.dart @@ -0,0 +1,132 @@ +// Deterministic synthetic 1 Hz nights for the causal-stager tests. +// +// Not a physiological simulator — just three separable regimes so the tests can +// check that the causal stager sees what is there: +// nrem : low, steady HR; quiet RR; still wrist +// rem : slightly higher HR, jittery instantaneous HR, wide RR spread; still +// wake : high HR; bursts of wrist motion +// RR beats carry the SAME second-quantised timestamps the device produces +// (`rr_ts_ms = rec_ts * 1000`, so several beats tie on one second). + +import 'dart:math' as math; +import 'package:openstrap_analytics/onehz.dart'; + +/// 30-s-aligned absolute start (1_700_000_010_000 / 30_000 is an integer). +const double kT0Ms = 1700000010000; + +class SynthNight { + final double t0Ms; + final List hr; + final List accel; + final List rrTsMs; + final List rrMs; + + /// Ground-truth regime per second ('nrem' | 'rem' | 'wake'). + final List truth; + const SynthNight( + this.t0Ms, this.hr, this.accel, this.rrTsMs, this.rrMs, this.truth); + int get seconds => truth.length; +} + +class _Rng { + int _s; + _Rng(int seed) : _s = seed == 0 ? 0x9E3779B9 : seed; + double next() { + _s ^= (_s << 13) & 0xFFFFFFFF; + _s ^= _s >> 17; + _s ^= (_s << 5) & 0xFFFFFFFF; + _s &= 0xFFFFFFFF; + return _s / 4294967296.0; + } + + double gauss() { + var a = 0.0; + for (var i = 0; i < 12; i++) { + a += next(); + } + return a - 6.0; + } +} + +SynthNight synthNight(List<(String, int)> segments, + {int seed = 11, double t0Ms = kT0Ms}) { + final rng = _Rng(seed); + final hr = []; + final accel = []; + final rrTs = []; + final rr = []; + final truth = []; + var beatClockMs = 0.0; // ms since t0, the next beat's time + var sec = 0; + for (final (kind, len) in segments) { + for (var k = 0; k < len; k++, sec++) { + final base = switch (kind) { 'rem' => 59.0, 'wake' => 80.0, _ => 54.0 }; + final hrSd = switch (kind) { 'rem' => 3.2, 'wake' => 2.0, _ => 0.9 }; + final rrSd = switch (kind) { 'rem' => 55.0, 'wake' => 25.0, _ => 14.0 }; + final bpm = (base + hrSd * rng.gauss()).roundToDouble(); + final ts = t0Ms + sec * 1000.0; + hr.add(HrSample(ts, bpm)); + // posture drifts every 20 min (a static change, not motion) + final posture = (sec ~/ 1200) % 3; + final g = switch (posture) { + 0 => const [0.30, 0.80, 0.50], + 1 => const [0.70, 0.20, 0.68], + _ => const [-0.20, 0.55, 0.81], + }; + final gn = math.sqrt(g[0] * g[0] + g[1] * g[1] + g[2] * g[2]); + var burst = 0.0; + if (kind == 'wake' && rng.next() < 0.35) burst = 0.25 + 0.2 * rng.next(); + accel.add(AccelSample( + ts, + g[0] / gn + 0.0015 * rng.gauss() + burst, + g[1] / gn + 0.0015 * rng.gauss(), + g[2] / gn + 0.0015 * rng.gauss(), + )); + // beats whose start falls inside this second + while (beatClockMs < (sec + 1) * 1000.0) { + final inst = 60000.0 / math.max(35.0, bpm); + final v = (inst + rrSd * rng.gauss()).clamp(380.0, 1900.0).toDouble(); + if (beatClockMs >= sec * 1000.0) { + rrTs.add(ts); + rr.add(v.roundToDouble()); + } + beatClockMs += v; + } + truth.add(kind); + } + } + return SynthNight(t0Ms, hr, accel, rrTs, rr, truth); +} + +/// A window of the night: samples with `lo <= second < hi` (seconds since t0), +/// decision time `nowSec`. `hi` may exceed `nowSec` to carry FUTURE samples. +CausalSampleWindow windowOf(SynthNight n, int loSec, int hiSec, + {int? nowSec, bool hrOn = true, bool accelOn = true, bool rrOn = true}) { + final hiC = math.min(hiSec, n.seconds); + final loC = math.max(0, loSec); + final loMs = n.t0Ms + loC * 1000.0, hiMs = n.t0Ms + hiC * 1000.0; + final rrIdx = [ + for (var i = 0; i < n.rrTsMs.length; i++) + if (n.rrTsMs[i] >= loMs && n.rrTsMs[i] < hiMs) i + ]; + return CausalSampleWindow( + nowMs: n.t0Ms + (nowSec ?? hiSec) * 1000.0, + hr: hrOn ? n.hr.sublist(loC, hiC) : const [], + accel: accelOn ? n.accel.sublist(loC, hiC) : const [], + rr: rrOn + ? RrSeries([for (final i in rrIdx) n.rrTsMs[i]], + [for (final i in rrIdx) n.rrMs[i]]) + : RrSeries(const [], const []), + ); +} + +/// A reasonably realistic ~3 h night used by most tests. +List<(String, int)> standardNight() => [ + ('nrem', 35 * 60), + ('rem', 12 * 60), + ('nrem', 40 * 60), + ('wake', 6 * 60), + ('nrem', 30 * 60), + ('rem', 15 * 60), + ('nrem', 25 * 60), + ]; diff --git a/test/onehz/support/correct_rr_reference.dart b/test/onehz/support/correct_rr_reference.dart new file mode 100644 index 0000000..999f68d --- /dev/null +++ b/test/onehz/support/correct_rr_reference.dart @@ -0,0 +1,344 @@ +// FROZEN REFERENCE: a verbatim copy of `correctRr` as it stood at analytics +// a19f439 (before the sliding-window speed rewrite), with its helpers and the +// two `util.dart` functions it calls (`percentile`, `median`) copied in too, so +// nothing the rewrite touches can change what this file computes. +// +// It is the oracle for the production `correctRr` and for the streaming +// `RrCorrector`: same input => bit-identical nn, nnTimesMs, classes, counts and +// cleanFraction. DO NOT "fix" or speed this file up; if the production +// behaviour is meant to change, that is a kAlgoVersion-level decision and this +// copy is replaced deliberately, in its own commit. +import 'dart:math' as math; + +import 'package:openstrap_analytics/onehz.dart' + show BeatClass, RrCorrectionResult; + +double? _percentileSortedRef(List sorted, double p) { + if (sorted.isEmpty) return null; + if (sorted.length == 1) return sorted[0]; + final rank = (p / 100) * (sorted.length - 1); + final lo = rank.floor(); + final hi = rank.ceil(); + if (lo == hi) return sorted[lo]; + final frac = rank - lo; + return sorted[lo] + (sorted[hi] - sorted[lo]) * frac; +} + +double? _percentileRef(List values, double p) => + _percentileSortedRef([...values]..sort(), p); + +double? _medianRef(List xs) => _percentileRef(xs, 50); + +/// Lipponen–Tarvainen RR artifact correction. +/// +/// [rrMs] the raw RR series (ms). [alpha] scales the time-varying threshold +/// (paper default 5.2 on the QD estimate). [windowBeats] sliding window for the +/// local dispersion estimate. +/// +/// [rrTsMs] the beat timestamps that came with [rrMs] (same length, sorted, +/// epoch ms). PASS THEM. Without them the output clock is Σrr, which splices +/// every sensor dropout out of the record: the intervals the band never +/// reported are not in [rrMs] at all, so no amount of book-keeping inside this +/// function can recover them. Measured on 13 real nights, the cumsum span was +/// 0.13–0.87 of the true rec_ts span, and the Lomb-Scargle spectrum built on it +/// moved LF/HF across the sympatho-vagal line (0.65 → 1.53 on one MG night). +/// With them, the clock cumsums RR *inside* a contiguous run — the RR values +/// are the only sub-second information that exists, since `rr_ts_ms` is +/// `rec_ts*1000` and therefore whole-second — and re-anchors to the real +/// timestamp whenever the wall step exceeds the interval by more than +/// [reanchorGapMs], i.e. at a dropout. +RrCorrectionResult correctRrReference( + List rrMs, { + List? rrTsMs, + double alpha = 5.2, + int windowBeats = 91, + double minThresholdMs = 100, + double reanchorGapMs = 1000, +}) { + final n = rrMs.length; + // Beat-end time for EVERY input beat, kept or not. One array, built once, so + // every branch below reads a clock instead of advancing its own. + final tAll = _beatTimesRef(rrMs, rrTsMs, reanchorGapMs); + if (n == 0) { + return const RrCorrectionResult( + nn: [], + nnTimesMs: [], + classes: [], + cleanFraction: 0, + droppedCount: 0, + correctedCount: 0, + ); + } + if (n < 3) { + // Too short to estimate dispersion: pass through physiologically-plausible + // beats only, classify the rest as long/short. No fabrication. + final classes = [ + for (final rr in rrMs) + (rr >= 300 && rr <= 2000) ? BeatClass.normal : BeatClass.longShort + ]; + final nn = []; + final times = []; + for (var i = 0; i < n; i++) { + if (classes[i] == BeatClass.normal) { + nn.add(rrMs[i]); + times.add(tAll[i]); + } + } + return RrCorrectionResult( + nn: nn, + nnTimesMs: times, + classes: classes, + cleanFraction: nn.length / n, + droppedCount: n - nn.length, + correctedCount: 0, + ); + } + + // dRR[i] = rr[i] - rr[i-1]. + final dRR = List.filled(n, 0); + for (var i = 1; i < n; i++) { + dRR[i] = rrMs[i] - rrMs[i - 1]; + } + + // Time-varying threshold from a sliding quartile-deviation (QD) of dRR. + // th1 ~ dispersion of dRR (short artifacts), th2 ~ dispersion of medianed RR. + final th1 = _slidingThresholdRef(dRR, windowBeats, alpha, minThresholdMs); + + // medRR: rr minus local median (for missed/extra long-range tests). + final med = _slidingMedianRef(rrMs, windowBeats); + final mRR = List.generate(n, (i) { + final d = rrMs[i] - med[i]; + return d < 0 ? d * 2 : d; // paper asymmetry weight + }); + final th2 = _slidingThresholdRef(mRR, windowBeats, alpha, minThresholdMs); + + final classes = List.filled(n, BeatClass.normal); + for (var i = 0; i < n; i++) { + final hardLong = rrMs[i] > 2000; + final hardShort = rrMs[i] < 300; + final bigJump = dRR[i].abs() > th1[i]; + final bigDev = mRR[i].abs() > th2[i]; + if (hardLong || (bigDev && mRR[i] > 0)) { + // Long interval: likely a MISSED beat (interval ~ multiple of normal). + classes[i] = (med[i] > 0 && rrMs[i] > 1.5 * med[i]) + ? BeatClass.missed + : BeatClass.longShort; + } else if (hardShort || (bigDev && mRR[i] < 0)) { + // Short interval: likely an EXTRA (spurious) beat. + classes[i] = (med[i] > 0 && rrMs[i] < 0.6 * med[i]) + ? BeatClass.extra + : BeatClass.longShort; + } else if (bigJump) { + classes[i] = BeatClass.ectopic; + } + } + + // Compensatory-pair reconciliation. A single ectopic/extra beat shows up as + // TWO successive dRR spikes of opposite sign (the bad beat, then the + // recovery). If beat i was flagged ONLY by the dRR jump (ectopic) but its + // value is physiologically normal and close to the local median, while its + // predecessor was a short/extra or long/missed event of opposite-sign dRR, + // then i is just the recovery — demote it to normal so the event stays a + // single isolated artifact rather than a spurious 2-beat run. + for (var k = 1; k < n; k++) { + if (classes[k] != BeatClass.ectopic) continue; + final prevBad = classes[k - 1] == BeatClass.extra || + classes[k - 1] == BeatClass.missed || + classes[k - 1] == BeatClass.ectopic || + classes[k - 1] == BeatClass.longShort; + if (!prevBad) continue; + final oppositeSign = dRR[k] * dRR[k - 1] < 0; + final valueNormal = med[k] > 0 && + rrMs[k] >= 300 && + rrMs[k] <= 2000 && + (rrMs[k] - med[k]).abs() <= 0.2 * med[k]; + if (oppositeSign && valueNormal) { + classes[k] = BeatClass.normal; + } + } + + // Build cleaned NN. Isolated single artifact -> cubic-spline (Catmull-Rom on + // the 4 surrounding NORMAL beats). A run of ≥2 consecutive artifacts -> drop. + final isArtifact = [for (final c in classes) c != BeatClass.normal]; + final nn = []; + final times = []; + var dropped = 0; + var corrected = 0; + var i = 0; + while (i < n) { + if (!isArtifact[i]) { + nn.add(rrMs[i]); + times.add(tAll[i]); + i++; + continue; + } + // Measure run length. + var j = i; + while (j < n && isArtifact[j]) { + j++; + } + final runLen = j - i; + if (runLen == 1) { + // Isolated -> spline-correct from surrounding normals. + final corr = _splineCorrectRef(rrMs, isArtifact, i); + if (corr != null) { + // The beat keeps its REAL time, not the interpolated one. `corr` is our + // best guess at what the NN *should* have been (a missed beat splits one + // ~2000 ms interval into two ~1000 ms ones), but the 2000 ms still + // elapsed. Advancing by `corr` deleted ~1 s of record per corrected beat + // — 1.6 % of a night at one missed beat in sixty. + nn.add(corr); + times.add(tAll[i]); + corrected++; + } else { + dropped++; // no anchors -> honest drop; the clock already elapsed + } + } else { + // Multi-beat run: NEVER interpolate. The clock is not touched here at all + // — `tAll` already carries the run's elapsed time, so the two sides of a + // dropped run are not spliced together. Not doing this compacted + // `nnTimesMs` (299.0 s of real record emitted as a 294.0 s span) and + // inflated everything keyed off it: cvhr_per_hour's analyzedHours, the + // Lomb-Scargle axis, spanSec — on exactly the noisy nights that needed + // the correction most. + dropped += runLen; + } + i = j; + } + + final normalCount = classes.where((c) => c == BeatClass.normal).length; + return RrCorrectionResult( + nn: nn, + nnTimesMs: times, + classes: classes, + cleanFraction: normalCount / n, + droppedCount: dropped, + correctedCount: corrected, + ); +} + +/// Beat-end time (ms, relative to the start of beat 0) for every input beat. +/// +/// RR intervals tile the timeline, so the clock advances for EVERY interval, +/// kept or dropped — but only for intervals the band actually REPORTED. A +/// sensor dropout is not an interval at all: it is absent from [rrMs], and +/// summing across it splices the two sides of the hole together. [rrTsMs] is +/// the only witness that the hole existed, so when the wall step between two +/// consecutive reported beats exceeds that beat's own interval by more than +/// [reanchorGapMs], the clock is RE-ANCHORED to the real timestamp instead of +/// accumulated. Inside a contiguous run the RR cumsum is kept, because +/// `rr_ts_ms = rec_ts*1000` is whole-second and the RR values are the only +/// sub-second information in the record. +/// +/// The slack has to be ≥ the max plausible RR (2,400 ms saturation) minus a +/// normal beat, or ordinary quantisation would re-anchor constantly; 1,000 ms +/// is the audit's figure and is ~1 whole quantisation step. +List _beatTimesRef( + List rrMs, List? rrTsMs, double reanchorGapMs) { + final n = rrMs.length; + final t = List.filled(n, 0); + if (n == 0) return t; + final wall = rrTsMs != null && rrTsMs.length == n; + final origin = wall ? rrTsMs[0] - rrMs[0] : 0.0; + var cur = 0.0; + for (var i = 0; i < n; i++) { + // The test is LOCAL — this beat's wall step against this beat's own + // interval — so a run's accumulated cumsum-vs-wall drift never triggers it. + // + // That drift is real and NOT resolved here. Inside contiguous runs of ≥200 + // beats, Σrr / rec_ts-span measures 0.9631 on gen4 against 0.9988 (W5) and + // 1.0013 (MG). Which clock is wrong is not decidable from the exports: the + // band's `hr` field is computed from the same beat detector as `rr_ms`, so + // it is not independent evidence, and a low rate of beats the band never + // reports produces the same deficit with a perfectly correct clock. So + // there is deliberately no `rrClockScale` — applying 0.963 to gen4's RR + // values would be actively wrong under that second explanation. The + // consequence to carry: **gen4 and gen5/MG nights are not comparable on any + // frequency-domain or per-hour quantity** until it is resolved. + final dropout = + wall && i > 0 && (rrTsMs[i] - rrTsMs[i - 1]) - rrMs[i] > reanchorGapMs; + final anchored = wall ? rrTsMs[i] - origin : 0.0; + // `> cur` keeps the clock monotonic even if timestamps arrive out of order + // or the cumsum has run ahead of the wall — a backwards jump is never taken. + cur = (dropout && anchored > cur) ? anchored : cur + rrMs[i]; + t[i] = cur; + } + return t; +} + +/// Time-varying threshold: alpha × QD of the SIGNED series x in a sliding +/// window, where QD = (Q3 − Q1)/2 (Lipponen & Tarvainen 2019, eq. for th1/th2: +/// "th = α · quartile deviation of dRR over the 91-beat window", α = 5.2). +/// +/// The quartile deviation MUST be taken on the SIGNED series. Taking it on |x| +/// folds the symmetric ±dRR distribution onto one side, collapsing QD by ~an +/// order of magnitude; the threshold then sinks to [floor] and the detector +/// degenerates into a fixed 100 ms cut-off that flags ordinary RSA as ectopy. +List _slidingThresholdRef( + List x, int win, double alpha, double floor) { + final n = x.length; + final out = List.filled(n, 0); + final half = win ~/ 2; + for (var i = 0; i < n; i++) { + final lo = math.max(0, i - half); + final hi = math.min(n - 1, i + half); + final seg = []; + for (var k = lo; k <= hi; k++) { + seg.add(x[k]); + } + final q1 = _percentileRef(seg, 25) ?? 0; + final q3 = _percentileRef(seg, 75) ?? 0; + final qd = (q3 - q1) / 2; + // Floor keeps a gross outlier detectable on (near-)quantized clean data + // where the QD genuinely collapses to 0 (constant RR). On any series with + // real beat-to-beat variability α·QD dominates the floor, so the floor + // never governs a physiological signal. + out[i] = math.max(alpha * qd, floor); + } + return out; +} + +List _slidingMedianRef(List x, int win) { + final n = x.length; + final out = List.filled(n, 0); + final half = win ~/ 2; + for (var i = 0; i < n; i++) { + final lo = math.max(0, i - half); + final hi = math.min(n - 1, i + half); + final seg = []; + for (var k = lo; k <= hi; k++) { + if (k == i) continue; + seg.add(x[k]); + } + out[i] = _medianRef(seg) ?? x[i]; + } + return out; +} + +/// Catmull-Rom cubic interpolation at the artifact index using the nearest two +/// NORMAL beats on each side. Returns null if anchors are unavailable. +double? _splineCorrectRef(List rr, List isArtifact, int idx) { + final left = []; + for (var k = idx - 1; k >= 0 && left.length < 2; k--) { + if (!isArtifact[k]) left.insert(0, rr[k]); + } + final right = []; + for (var k = idx + 1; k < rr.length && right.length < 2; k++) { + if (!isArtifact[k]) right.add(rr[k]); + } + if (left.isEmpty || right.isEmpty) return null; + final p1 = left.last; + final p2 = right.first; + final p0 = left.length >= 2 ? left.first : p1; + final p3 = right.length >= 2 ? right.last : p2; + // Catmull-Rom at t=0.5 between p1 and p2. + const t = 0.5; + final t2 = t * t; + final t3 = t2 * t; + final v = 0.5 * + ((2 * p1) + + (-p0 + p2) * t + + (2 * p0 - 5 * p1 + 4 * p2 - p3) * t2 + + (-p0 + 3 * p1 - 3 * p2 + p3) * t3); + return v; +} diff --git a/test/onehz/support/energy_day.dart b/test/onehz/support/energy_day.dart new file mode 100644 index 0000000..5b6dfd2 --- /dev/null +++ b/test/onehz/support/energy_day.dart @@ -0,0 +1,56 @@ +// Deterministic synthetic day for the per-minute energy tests: no randomness, +// no clock. Mixes every case dailyEnergy distinguishes — resting HR, HR over +// the flex gate, dropped (0) and non-finite HR, cadence on sub-flex minutes, +// cadence on HR-billed minutes, cadence on no-HR minutes. + +import 'package:openstrap_analytics/src/onehz/workout/calories.dart'; + +class EnergyDay { + final List hr; + final List cadence; + const EnergyDay(this.hr, this.cadence); +} + +/// [n] minutes. Same LCG as the rest of the suite's synthetic data. +EnergyDay energyDay({int n = 1440, int seed = 4242}) { + var s = seed; + double rnd() { + s = (s * 1103515245 + 12345) & 0x7fffffff; + return s / 0x7fffffff; + } + + final hr = []; + final cad = []; + for (var i = 0; i < n; i++) { + final r = rnd(); + double h; + if (r < 0.05) { + h = 0.0; // off-skin sentinel + } else if (r < 0.07) { + h = double.nan; + } else if (r < 0.60) { + h = 52.0 + rnd() * 40.0; // resting / light, mostly under the gate + } else { + h = 100.0 + rnd() * 75.0; // straddles the gate, tops out near hrmax + } + hr.add(h); + final c = rnd(); + if (c < 0.15) { + cad.add(90.0 + rnd() * 60.0); // below, inside and past the 100-130 range + } else if (c < 0.17) { + cad.add(double.nan); + } else { + cad.add(null); + } + } + return EnergyDay(hr, cad); +} + +const energyProfiles = { + 'male': WorkoutUserProfile( + weightKg: 82.5, heightCm: 181.0, age: 34, sex: 'male'), + 'female': WorkoutUserProfile( + weightKg: 61.0, heightCm: 167.0, age: 41, sex: 'female'), + 'nonbinary': WorkoutUserProfile( + weightKg: 70.0, heightCm: 172.5, age: 29, sex: 'nonbinary'), +}; diff --git a/test/onehz/support/epoch_input.dart b/test/onehz/support/epoch_input.dart new file mode 100644 index 0000000..f97e36a --- /dev/null +++ b/test/onehz/support/epoch_input.dart @@ -0,0 +1,105 @@ +// Shared fixtures for the option-2 (onset-anchored, phase-keyed) EpochRows +// tests: a synthetic night as raw samples, optionally with holes. +import 'dart:convert'; + +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +import '../../support/property.dart'; +import 'causal_night.dart'; + +const int kAlignedT0Ms = 1700000010000; // a multiple of 30,000 + +class EpochInput { + final int t0Ms, seconds; + final List hr; + final List accel; + final List rrMs, rrTs; + EpochInput(this.t0Ms, this.seconds, this.hr, this.accel, this.rrMs, this.rrTs); + + /// Both streams lose the same seconds [gaps] (seconds since t0), and the + /// beats in them. + factory EpochInput.make(List<(String, int)> segs, + {int seed = 11, int t0Ms = kAlignedT0Ms, List<(int, int)> gaps = const []}) { + final n = synthNight(segs, seed: seed, t0Ms: t0Ms.toDouble()); + bool inGap(double ts) => gaps + .any((g) => ts >= t0Ms + g.$1 * 1000.0 && ts < t0Ms + g.$2 * 1000.0); + final rr = [], rt = []; + for (var i = 0; i < n.rrMs.length; i++) { + if (!inGap(n.rrTsMs[i])) { + rr.add(n.rrMs[i]); + rt.add(n.rrTsMs[i]); + } + } + return EpochInput(t0Ms, n.seconds, [for (final h in n.hr) if (!inGap(h.tsMs)) h], + [for (final a in n.accel) if (!inGap(a.tsMs)) a], rr, rt); + } + + /// Samples with `loSec <= second < hiSec` (seconds since t0), as a new input + /// (same t0; `seconds` becomes hiSec - loSec only for bookkeeping). + EpochInput between(int loSec, int hiSec) { + final lo = t0Ms + loSec * 1000.0, hi = t0Ms + hiSec * 1000.0; + bool inside(double t) => t >= lo && t < hi; + final ri = [ + for (var i = 0; i < rrTs.length; i++) + if (inside(rrTs[i])) i + ]; + return EpochInput( + t0Ms, + seconds, + [for (final h in hr) if (inside(h.tsMs)) h], + [for (final a in accel) if (inside(a.tsMs)) a], + [for (final i in ri) rrMs[i]], + [for (final i in ri) rrTs[i]]); + } + + void foldInto(EpochRows er, int loSec, int hiSec) { + final p = between(loSec, hiSec); + er.fold(hr: p.hr, accel: p.accel, rrMs: p.rrMs, rrTsMs: p.rrTs); + } + + int get sampleCount => hr.length + accel.length + rrMs.length; + + List get grav => + [for (final a in accel) GravTs(a.tsMs ~/ 1000, a.x, a.y, a.z)]; + List get hrTs => [for (final h in hr) HrTs(h.tsMs ~/ 1000, h.hr)]; + List get rrTsSec => + [for (var i = 0; i < rrMs.length; i++) RrTs(rrTs[i] ~/ 1000, rrMs[i])]; +} + +List<(String, int)> nightSegs() => [ + ('nrem', 900), ('rem', 600), ('nrem', 900), ('wake', 300), ('nrem', 300), + ]; + +List randomCuts(Rng r, int seconds) { + const sizes = [1, 2, 7, 30, 31, 59, 100, 171, 500]; + final out = []; + var at = 0; + while (true) { + at += r.nextBool(.15) ? 0 : 1 + r.nextInt(sizes[r.nextInt(sizes.length)]); + if (at >= seconds) return out; + out.add(at); + } +} + +bool sameRow(EpochFeatureRow a, EpochFeatureRow b) => + a.startMs == b.startMs && + [ + [a.motion, b.motion], [a.hr, b.hr], [a.hrSd, b.hrSd], + [a.rmssd, b.rmssd], [a.lfhf, b.lfhf], [a.rk, b.rk], [a.sdnn, b.sdnn], + ].every((p) => p[0].compareTo(p[1]) == 0); + +void expectSameRows( + List got, List want, String why) { + expect(got.length, want.length, reason: '$why: row count'); + for (var i = 0; i < got.length; i++) { + if (!sameRow(got[i], want[i])) { + fail('$why: row $i differs: got start ${got[i].startMs} hr ${got[i].hr} ' + 'motion ${got[i].motion}; want start ${want[i].startMs} ' + 'hr ${want[i].hr} motion ${want[i].motion}'); + } + } +} + +String showStages(List l) => + jsonEncode([for (final g in l) g.toJson()]); diff --git a/test/onehz/support/incremental_compare.dart b/test/onehz/support/incremental_compare.dart new file mode 100644 index 0000000..7d59b5f --- /dev/null +++ b/test/onehz/support/incremental_compare.dart @@ -0,0 +1,146 @@ +import 'dart:convert'; +import 'dart:math' as math; + +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +void numberClose(double? actual, double? expected, {String? reason}) { + if (expected == null) { + expect(actual, isNull, reason: reason); + } else if (expected.isNaN) { + expect(actual?.isNaN, isTrue, reason: reason); + } else if (!expected.isFinite) { + expect(actual, expected, reason: reason); + } else { + expect(actual, isNotNull, reason: reason); + expect(actual, closeTo(expected, math.max(1e-9, expected.abs() * 1e-8)), + reason: reason); + } +} + +void metricEnvelope(Metric actual, Metric expected) { + expect(actual.present, expected.present); + numberClose(actual.confidence, expected.confidence); + expect(actual.tier, expected.tier); + expect(actual.inputs_used, expected.inputs_used); + expect(actual.note, expected.note); + expect(actual.drivers?.map((d) => d.toJson()).toList(), + expected.drivers?.map((d) => d.toJson()).toList()); +} + +void hrvClose(Metric actual, Metric expected) { + metricEnvelope(actual, expected); + if (expected.value == null) return; + final a = actual.value!, e = expected.value!; + expect(a.nBeats, e.nBeats); + numberClose(a.rmssd, e.rmssd); + numberClose(a.sdnn, e.sdnn); + numberClose(a.sdann, e.sdann); + numberClose(a.sdnnIndex, e.sdnnIndex); + numberClose(a.pnn50, e.pnn50); + numberClose(a.diffAcf1, e.diffAcf1); +} + +void spectrumClose(LombScargle? actual, LombScargle? expected) { + if (expected == null) { + expect(actual, isNull); + return; + } + expect(actual, isNotNull); + expect(actual!.spectrum.length, expected.spectrum.length); + for (var i = 0; i < expected.spectrum.length; i++) { + expect(actual.spectrum[i].freqHz, expected.spectrum[i].freqHz); + numberClose(actual.spectrum[i].psd, expected.spectrum[i].psd, + reason: 'frequency ${expected.spectrum[i].freqHz}'); + } + for (final band in [(0.0, .04), (.04, .15), (.15, .4)]) { + numberClose(actual.bandPower(band.$1, band.$2), + expected.bandPower(band.$1, band.$2)); + } +} + +void enmoClose(EnmoResult actual, EnmoResult expected) { + numberClose(actual.gRef, expected.gRef); + numberClose(actual.coverage, expected.coverage); + expect(actual.minutes.length, expected.minutes.length); + for (var i = 0; i < expected.minutes.length; i++) { + final a = actual.minutes[i], e = expected.minutes[i]; + expect(a.tsMinStartMs, e.tsMinStartMs); + expect(a.nSamples, e.nSamples); + numberClose(a.enmo, e.enmo); + numberClose(a.mad, e.mad); + numberClose(a.meanMag, e.meanMag); + numberClose(a.dynAmp, e.dynAmp); + } +} + +void minuteClose(MinuteMetrics actual, List keys, List hr, + {List? cadence, + double? rhr = 54, + double? maxHr = 186, + Sex sex = Sex.male, + WorkoutUserProfile? profile, + int dayMinutes = 1440, + double? quietHrr = .12, + bool quietSettled = false}) { + final trimp = banisterTrimp(hr, restingHr: rhr, maxHr: maxHr, sex: sex); + final strain = strainScoreMetric(trimp.value, + wakeMinutes: hr.length.toDouble(), + quietHrr: quietHrr, + quietSettled: quietSettled, + female: sex == Sex.female); + metricEnvelope(actual.trimp, trimp); + numberClose(actual.trimp.value, trimp.value); + metricEnvelope(actual.strain, strain); + numberClose(actual.strain.value, strain.value); + if (profile == null || rhr == null || maxHr == null) { + expect(actual.energy, isNull); + expect(actual.minutes, isNull); + return; + } + final e = Calories.dailyEnergy(hr, + profile: profile, + hrmax: maxHr, + restingHr: rhr, + dayMinutes: dayMinutes, + cadenceSpmPerMin: cadence); + final series = Calories.minuteEnergy(hr, + profile: profile, + hrmax: maxHr, + restingHr: rhr, + epochMinutes: keys, + cadenceSpmPerMin: cadence); + if (e == null) { + expect(actual.energy, isNull); + } else { + expect(actual.energy, isNotNull); + numberClose(actual.energy!.total, e.total); + numberClose(actual.energy!.active, e.active); + numberClose(actual.energy!.basal, e.basal); + numberClose(actual.energy!.walking, e.walking); + } + if (series == null) { + expect(actual.minutes, isNull); + return; + } + final a = actual.minutes!; + expect(a.coveredMinutes, series.coveredMinutes); + expect(a.abstainedMinutes, series.abstainedMinutes); + numberClose(a.basalKcalPerMin, series.basalKcalPerMin); + numberClose(a.active, series.active); + numberClose(a.walking, series.walking); + expect(a.minutes.length, series.minutes.length); + for (var i = 0; i < series.minutes.length; i++) { + final am = a.minutes[i], em = series.minutes[i]; + expect(am.minute, em.minute); + expect(am.source, em.source); + expect(am.abstained, em.abstained); + numberClose(am.basal, em.basal); + numberClose(am.active, em.active); + numberClose(am.total, em.total); + numberClose(am.walking, em.walking); + } +} + +Map checkpoint(Map json) => + jsonDecode(jsonEncode(json)) as Map; diff --git a/test/onehz/support/incremental_fixtures.dart b/test/onehz/support/incremental_fixtures.dart new file mode 100644 index 0000000..b9277cc --- /dev/null +++ b/test/onehz/support/incremental_fixtures.dart @@ -0,0 +1,113 @@ +import 'dart:io'; +import 'dart:math' as math; + +import 'package:openstrap_analytics/onehz.dart'; + +/// Reproducible inputs; no wall clock, device, network, or unseeded randomness. +class IncrementalFixture { + final List times, nn; + final List accel; + final List minuteKeys; + final List hr; + final List cadence; + const IncrementalFixture( + this.times, this.nn, this.accel, this.minuteKeys, this.hr, this.cadence); +} + +IncrementalFixture incrementalFixture( + {int seed = 42, + int beats = 850, + int seconds = 241, + int minutes = 121, + double originMs = 0, + double cadenceHz = 1, + bool gaps = true, + bool jitter = false}) { + final random = math.Random(seed); + final nn = [], times = []; + var time = originMs; + for (var i = 0; i < beats; i++) { + final interval = jitter + ? 800 + 160 * (random.nextDouble() - .5) + : 800 + + 75 * math.sin(i * .13 + seed * .003) + + 21 * math.sin(i * .031 + seed * .007); + time += interval; + if (gaps && i % 113 == 112) time += 21000; + nn.add(interval); + times.add(time); + } + final accel = []; + for (var i = 0; i < seconds; i++) { + if (gaps && i >= 70 && i < 139) continue; + accel.add(AccelSample( + originMs + i * 1000 / cadenceHz, + .22 * math.sin(i * .29) + random.nextDouble() * .007, + .11 * math.cos(i * .07), + 1.02 + .065 * math.sin(i * .23), + valid: i % 47 != 46)); + } + const hrBranches = [54.0, 95.0, 106.799999, 106.8, 130.0, 190.0, 0.0]; + const cadBranches = [null, 99.999, 100, 110, 120, 130, 160, 0]; + return IncrementalFixture( + times, + nn, + accel, + List.generate(minutes, (i) => 28000000 + i + i ~/ 17), + List.generate(minutes, (i) => hrBranches[(i + seed) % hrBranches.length]), + List.generate( + minutes, (i) => cadBranches[(i + seed) % cadBranches.length])); +} + +List prefixSizes(int length, int chunk) => { + 0, + 1, + 2, + 3, + 4, + 29, + 30, + 31, + for (var i = chunk; i < length; i += chunk) i, + length, + }.where((n) => n <= length).toList() + ..sort(); + +/// Checked-in anonymised WHOOP-4 overnight capture, July 2026. Provenance and +/// documentation: test/onehz/real_night_cardio_stager_test.dart. Timestamps were +/// rebased to zero and only HR, axes, and RR retained. This subset is +/// read directly from the repository; missing fixtures fail, never skip. +IncrementalFixture realIncrementalFixture({int seconds = 900}) { + final accel = [], hrSeconds = []; + for (final line in File('test/onehz/fixtures/real_night_2026_07_onehz.csv') + .readAsLinesSync() + .skip(1)) { + if (line.trim().isEmpty) continue; + final p = line.split(',').map(double.parse).toList(); + if (p[0] >= seconds) break; + hrSeconds.add(p[1]); + accel.add(AccelSample(p[0] * 1000, p[2], p[3], p[4])); + } + final raw = [], times = []; + for (final line in File('test/onehz/fixtures/real_night_2026_07_rr.csv') + .readAsLinesSync() + .skip(1)) { + if (line.trim().isEmpty) continue; + final p = line.split(',').map(double.parse).toList(); + if (p[0] >= seconds * 1000) break; + times.add(p[0]); + raw.add(p[1]); + } + final clean = correctRr(raw, rrTsMs: times); + final hr = []; + for (var i = 0; i < hrSeconds.length; i += 60) { + hr.add(mean(hrSeconds.sublist(i, math.min(i + 60, hrSeconds.length)))!); + } + return IncrementalFixture( + clean.nnTimesMs, + clean.nn, + accel, + List.generate(hr.length, (i) => i), + hr, + List.filled(hr.length, null)); +} diff --git a/test/onehz/support/pre_bump_stager.dart b/test/onehz/support/pre_bump_stager.dart new file mode 100644 index 0000000..b989d31 --- /dev/null +++ b/test/onehz/support/pre_bump_stager.dart @@ -0,0 +1,110 @@ +// The pre-bump default staging path, frozen as a test oracle. +// +// Verbatim logic of `AdvancedSleepStager._stageSessionCardio` at analytics +// 83fc308 (the base of the combined bump), with only the class-private names +// qualified. It anchors every 30 s epoch at the START OF EACH USABLE RUN inside +// the session window: the first run at the first usable second at or after +// `start` (the session start when the data reaches it), and every later run at +// the first usable second after a dropout the 60 s carry-forward cannot bridge. +// Samples outside [start, end) are not read at all (features are clipped at the +// session edges). +// +// Grid option 2 (owner decision 2026-10-23) restores exactly this behaviour: +// `AdvancedSleepStager.stageWindow(...).stages` must equal this function. +import 'dart:math' as math; + +import 'package:openstrap_analytics/onehz.dart'; + +List preBumpStageSessionCardio(int start, int end, + List grav, List hr, List rr) { + final span = end - start; + if (span <= 0) return const []; + const epSec = 30; + const minStageableSec = 3 * epSec; + if (span < minStageableSec) return [StageSegment(start, end, 'unobserved')]; + + final gByTs = { + for (final g in grav) + if (g.valid && g.ts >= start && g.ts < end) g.ts: g + }; + final hByTs = { + for (final h in hr) + if (h.ts >= start && h.ts < end) h.ts: h + }; + final accel = List.filled( + span, AccelSample(start * 1000.0, 0, 0, 1.0)); + final hr1hz = List.filled(span, 0.0); + final usable = List.filled(span, false); + var lastRealIdx = -1; + for (var i = 0; i < span; i++) { + final ts = start + i; + final g = gByTs[ts]; + if (g != null) { + accel[i] = AccelSample(ts * 1000.0, g.x, g.y, g.z); + lastRealIdx = i; + usable[i] = true; + } else if (lastRealIdx >= 0 && + (i - lastRealIdx) <= AdvancedSleepStager.maxAccelCarryForwardSec) { + final p = accel[i - 1]; + accel[i] = AccelSample(ts * 1000.0, p.x, p.y, p.z); + usable[i] = true; + } + hr1hz[i] = hByTs[ts]?.bpm ?? 0.0; + } + + final rSeg = [for (final r in rr) if (r.ts >= start && r.ts < end) r]; + final rrMs = [for (final r in rSeg) r.rrMs]; + final rrTsMs = [for (final r in rSeg) r.ts * 1000.0]; + + final perSec = List.filled(span, 'unobserved'); + var i = 0; + while (i < span) { + if (!usable[i]) { + i++; + continue; + } + var j = i; + while (j < span && usable[j]) { + j++; + } + if ((j - i) >= minStageableSec) { + final result = cardioStager( + hr1hz.sublist(i, j), + accel.sublist(i, j), + rrMs: rrMs, + rrTsMs: rrTsMs, + ); + final nEpoch = result.base.stages.length; + for (var e = 0; e < nEpoch; e++) { + final label = switch (result.base.stages[e]) { + SleepStage.wake => 'wake', + SleepStage.rem => 'rem', + SleepStage.nrem => (e < result.deepFlag.length && result.deepFlag[e]) + ? 'deep' + : 'light', + }; + final lo = i + e * epSec; + final hi = e == nEpoch - 1 ? j : math.min(j, lo + epSec); + for (var k = lo; k < hi; k++) { + perSec[k] = label; + } + } + } + i = j; + } + + final segments = []; + var k = 0; + while (k < span) { + var m = k; + while (m < span && perSec[m] == perSec[k]) { + m++; + } + segments.add(StageSegment(start + k, start + m, perSec[k])); + k = m; + } + return segments; +} + +String showSegments(List l) => + [for (final g in l) '${g.start}-${g.end}:${g.stage}'].join(' '); diff --git a/test/onehz/support/rr_chunking.dart b/test/onehz/support/rr_chunking.dart new file mode 100644 index 0000000..83c2eba --- /dev/null +++ b/test/onehz/support/rr_chunking.dart @@ -0,0 +1,40 @@ +// Shared helpers for the RR correction / streaming oracle tests. +import 'dart:convert'; +import 'dart:math' as math; + +/// JSON round trip through real text (what a SQLite blob would hold). +Map jsonRoundTrip(Map m) => + jsonDecode(jsonEncode(m)) as Map; + +/// Random chunk boundaries over [0, n): sizes drawn from [sizes] (a mix of 1, +/// tiny, one-window, and huge) so every alignment against the 91/181-beat +/// windows is hit. +List randomCuts(math.Random r, int n, + {List sizes = const [1, 2, 3, 7, 45, 46, 90, 91, 92, 180, 181, 500, 1500]}) { + final cuts = []; + var at = 0; + while (at < n) { + final s = sizes[r.nextInt(sizes.length)]; + at = math.min(n, at + 1 + r.nextInt(s)); + cuts.add(at); + } + return cuts; +} + +/// Cuts that fall every [sec] seconds of the beat clock: what a 15-minute +/// derive pass sees. [tsMs] sorted-ish epoch ms. +List timeCuts(List tsMs, double sec) { + final cuts = []; + if (tsMs.isEmpty) return cuts; + var next = tsMs.first + sec * 1000; + for (var i = 0; i < tsMs.length; i++) { + if (tsMs[i] >= next) { + cuts.add(i); + while (next <= tsMs[i]) { + next += sec * 1000; + } + } + } + cuts.add(tsMs.length); + return cuts; +} diff --git a/test/onehz/support/rr_compare.dart b/test/onehz/support/rr_compare.dart new file mode 100644 index 0000000..a2f772e --- /dev/null +++ b/test/onehz/support/rr_compare.dart @@ -0,0 +1,44 @@ +// Bit-for-bit comparison helpers for the RR correction oracle tests. +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +/// `compareTo` is 0 only for the same double bit pattern in every case that +/// matters here (it tells -0.0 from 0.0 and treats NaN as equal to NaN), which +/// `==` does not. +bool sameBits(double a, double b) => a.compareTo(b) == 0; + +/// Fails with the FIRST differing index instead of dumping two 96k-element +/// lists. +void expectBitIdentical(List got, List want, String why) { + if (got.length != want.length) { + fail('$why: length ${got.length} != ${want.length}'); + } + for (var i = 0; i < got.length; i++) { + if (!sameBits(got[i], want[i])) { + fail('$why: index $i got ${got[i]} want ${want[i]}'); + } + } +} + +void expectSameClasses(List got, List want, String why) { + if (got.length != want.length) { + fail('$why: classes length ${got.length} != ${want.length}'); + } + for (var i = 0; i < got.length; i++) { + if (got[i] != want[i]) { + fail('$why: class[$i] got ${got[i]} want ${want[i]}'); + } + } +} + +/// Everything a [RrCorrectionResult] carries. +void expectSameCorrection( + RrCorrectionResult got, RrCorrectionResult want, String why) { + expectBitIdentical(got.nn, want.nn, '$why nn'); + expectBitIdentical(got.nnTimesMs, want.nnTimesMs, '$why nnTimesMs'); + expectSameClasses(got.classes, want.classes, why); + expect(sameBits(got.cleanFraction, want.cleanFraction), isTrue, + reason: '$why cleanFraction ${got.cleanFraction} vs ${want.cleanFraction}'); + expect(got.droppedCount, want.droppedCount, reason: '$why droppedCount'); + expect(got.correctedCount, want.correctedCount, reason: '$why correctedCount'); +} diff --git a/test/onehz/support/rr_runs.dart b/test/onehz/support/rr_runs.dart new file mode 100644 index 0000000..86370dc --- /dev/null +++ b/test/onehz/support/rr_runs.dart @@ -0,0 +1,54 @@ +// Test support: clean RR series with long artifact runs, for the cap tests. +// Integer RR, timestamps = exact running sums (every clock value is an integer, +// so comparisons can be bit-exact). Seeded (property.dart's Rng). +import 'dart:math' as math; + +import '../../support/property.dart' show Rng; + +const rrRunsT0 = 1.7e12; + +class RunSeries { + final List rr, ts; + RunSeries(this.rr, this.ts); +} + +/// Clean integer RR (RSA-like wobble), timestamps the exact running sum. +List cleanRr(Rng r, int n) => [ + for (var i = 0; i < n; i++) + (850 + 40 * math.sin(i * 0.7) + 25 * math.sin(i * 0.13) + r.intIn(-15, 15)) + .roundToDouble() + ]; + +RunSeries rrSeries(List rr) { + var t = rrRunsT0; + return RunSeries(rr, [for (final v in rr) t += v]); +} + +/// Segments of [segLens] clean beats separated by runs of 3000 ms beats +/// ([runLens]); returns the series and each run's [start, end). With [nearRun], +/// one short (extra-like) beat sits 20 beats before and 20 after every run. +(RunSeries, List<(int, int)>) withRuns(int seed, List segLens, List runLens, + {bool nearRun = false}) { + final r = Rng(seed); + final rr = []; + final runs = <(int, int)>[]; + for (var k = 0; k < segLens.length; k++) { + final seg = cleanRr(r, segLens[k]); + if (nearRun) { + if (k > 0) seg[20] = (seg[20] * 0.55).roundToDouble(); + if (k < runLens.length) seg[seg.length - 20] = (seg[seg.length - 20] * 0.55).roundToDouble(); + } + rr.addAll(seg); + if (k < runLens.length) { + final a = rr.length; + // The last two values rise (2500, 3000), so the dRR spike on the normal + // beat that follows has the opposite sign and the compensatory-pair rule + // takes that beat back as normal: the run is exactly the junk beats. + rr.addAll(List.filled(runLens[k], 3000.0)); + rr[rr.length - 2] = 2500.0; + runs.add((a, rr.length)); + } + } + return (rrSeries(rr), runs); +} + diff --git a/test/onehz/support/rr_stream_driver.dart b/test/onehz/support/rr_stream_driver.dart new file mode 100644 index 0000000..6c8d07b --- /dev/null +++ b/test/onehz/support/rr_stream_driver.dart @@ -0,0 +1,100 @@ +// Driver + oracle comparison for the streaming RrCorrector tests. +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +import 'correct_rr_reference.dart'; +import 'rr_chunking.dart'; +import 'rr_compare.dart'; + +export 'rr_chunking.dart'; + +/// Accumulates everything a corrector has SETTLED, the way the app will hold it +/// between derive passes, and can save/restore the corrector at any boundary. +class RrStreamRun { + RrCorrector c; + final nn = []; + final nnTimes = []; + final classes = []; + int folded = 0; + int restarts = 0; + + RrStreamRun([RrCorrector? corrector]) : c = corrector ?? RrCorrector(); + + RrStreamRun.params( + {double alpha = 5.2, + int win = 91, + double floor = 100, + double reanchor = 1000}) + : c = RrCorrector( + alpha: alpha, + windowBeats: win, + minThresholdMs: floor, + reanchorGapMs: reanchor); + + RrSettled fold(List rr, List? ts) { + final s = c.fold(rr, tsMs: ts); + nn.addAll(s.nn); + nnTimes.addAll(s.nnTimes); + classes.addAll(s.classes); + folded += rr.length; + return s; + } + + /// Save to text and restore into a NEW corrector. + void restart() { + c = RrCorrector.fromJson(jsonRoundTrip(c.toJson())); + restarts++; + } + + /// `settled ++ tail` over the first [n] beats equals the frozen `correctRr` + /// on exactly those beats: nn, times, per-beat classes, counts, cleanFraction. + void expectMatchesOracle(List rr, List? ts, int n, + {double alpha = 5.2, + int win = 91, + double floor = 100, + double reanchor = 1000, + String why = ''}) { + expect(folded, n, reason: 'driver folded $why'); + final want = correctRrReference(rr.sublist(0, n), + rrTsMs: ts?.sublist(0, n), + alpha: alpha, + windowBeats: win, + minThresholdMs: floor, + reanchorGapMs: reanchor); + final snap = c.snapshot(); + expect(snap.n, n, reason: 'snapshot.n $why'); + expect(snap.classifiedBeats, classes.length, + reason: 'classifiedBeats == classes handed out $why'); + expect(snap.settledBeats, c.settledBeats, reason: 'settledBeats $why'); + expect(snap.settledBeats, lessThanOrEqualTo(snap.classifiedBeats), + reason: 'output cannot settle before its class $why'); + expect(snap.tailClasses.length, n - snap.classifiedBeats, + reason: 'tailClasses covers [classifiedBeats, n) $why'); + expectBitIdentical([...nn, ...snap.tailNn], want.nn, '$why nn'); + expectBitIdentical( + [...nnTimes, ...snap.tailNnTimes], want.nnTimesMs, '$why nnTimes'); + expectSameClasses( + [...classes, ...snap.tailClasses], want.classes, '$why classes'); + expect(snap.normalCount, + want.classes.where((k) => k == BeatClass.normal).length, + reason: 'normalCount $why'); + expect(snap.droppedCount, want.droppedCount, reason: 'dropped $why'); + expect(snap.correctedCount, want.correctedCount, reason: 'corrected $why'); + expect(sameBits(snap.cleanFraction, want.cleanFraction), isTrue, + reason: 'cleanFraction ${snap.cleanFraction} vs ${want.cleanFraction} ' + '$why'); + } + + /// Settled output is FINAL: it is a prefix of what the oracle says about the + /// WHOLE series, not just about the prefix seen so far. [full] is the oracle + /// run once over everything. + void expectSettledIsPrefixOf(RrCorrectionResult full, String why) { + expect(nn.length, lessThanOrEqualTo(full.nn.length), reason: '$why nn len'); + expectBitIdentical(nn, full.nn.sublist(0, nn.length), '$why settled nn'); + expectBitIdentical( + nnTimes, full.nnTimesMs.sublist(0, nnTimes.length), '$why settled t'); + expect(classes.length, lessThanOrEqualTo(full.classes.length)); + expectSameClasses( + classes, full.classes.sublist(0, classes.length), '$why settled cls'); + } +} diff --git a/test/onehz/support/rr_synth.dart b/test/onehz/support/rr_synth.dart new file mode 100644 index 0000000..ed0598d --- /dev/null +++ b/test/onehz/support/rr_synth.dart @@ -0,0 +1,151 @@ +// Test support: real-shaped RR generators and the real-night fixture loader. +// Deterministic (seeded), no wall clock. (Copied from the research prototypes in +// tool/incremental/synth.dart; the accelerometer generator stayed there because +// the day-curve states it feeds live in edge, not here.) +// +// "Real-shaped" means what the app really feeds correctRr: +// * integer-millisecond RR, end-of-beat timestamps quantised to WHOLE seconds +// (rr_ts_ms = rec_ts * 1000, several beats can share one stamp), +// * ectopic pairs, missed beats (2x interval), extra beats (half interval), +// multi-beat noise runs, sensor dropouts (no beats for seconds..minutes), +// * optional counter-reset style timestamp steps backwards. +import 'dart:io'; +import 'dart:math' as math; + +class RrData { + final List rr; + final List ts; // epoch-ish ms, whole seconds + RrData(this.rr, this.ts); + int get length => rr.length; +} + +class SynthConfig { + final int seed; + final double hours; + final double startSec; + final double ectopicPerMin; // premature beat + compensatory pause + final double missedPerMin; + final double extraPerMin; + final double noiseRunPerMin; // run of 3..12 garbage beats + final double gapPerHour; // dropouts of 5..900 s + final double backwardsPerHour; // ts steps back 1..3 s (counter reset-ish) + final double nightFraction; // fraction of the span that is "asleep" + const SynthConfig({ + this.seed = 1, + this.hours = 24, + this.startSec = 1.7e9, + this.ectopicPerMin = 0.3, + this.missedPerMin = 0.1, + this.extraPerMin = 0.1, + this.noiseRunPerMin = 0.05, + this.gapPerHour = 1.5, + this.backwardsPerHour = 0.0, + this.nightFraction = 0.33, + }); +} + +/// Physiologically-flavoured RR: slow HR drift + RSA (0.25 Hz) + LF (0.1 Hz) +/// + white jitter, integer ms, with injected artefacts. +RrData synthRr(SynthConfig c) { + final rnd = math.Random(c.seed); + final rr = [], ts = []; + final endT = c.hours * 3600.0; + var t = 0.0; // seconds since start (beat clock) + var phase = rnd.nextDouble() * 6.28; + final nightStart = endT * 0.05; + final nightEnd = nightStart + endT * c.nightFraction; + void emit(double interval) { + final v = interval.roundToDouble(); + t += v / 1000.0; + var stamp = (c.startSec + t).floorToDouble() * 1000.0; + rr.add(v); + ts.add(stamp); + } + + while (t < endT) { + final asleep = t >= nightStart && t < nightEnd; + // Base HR: sleep ~52 bpm, day 65..120 with activity bouts. + final bout = (math.sin(t / 1900.0 + c.seed) + 1) / 2; // 0..1 + final bpm = asleep ? 52 + 3 * math.sin(t / 5400) : 66 + 55 * bout * bout; + var base = 60000.0 / bpm; + final rsaAmp = asleep ? 45.0 : 22.0 * (1 - 0.7 * bout); + final rsaHz = asleep ? 0.24 : 0.27; + final v = base + + rsaAmp * math.sin(2 * math.pi * rsaHz * t + phase) + + 28 * math.sin(2 * math.pi * 0.1 * t) + + 8 * (rnd.nextDouble() - .5) * 2; + final beatMin = v / 60000.0; // minutes per beat → per-beat probabilities + final u = rnd.nextDouble(); + var p = c.noiseRunPerMin * beatMin; + if (u < p) { + final len = 3 + rnd.nextInt(10); + for (var k = 0; k < len; k++) { + emit(250 + rnd.nextInt(2100).toDouble()); + } + continue; + } + p += c.ectopicPerMin * beatMin; + if (u < p) { + emit(v * (0.55 + 0.15 * rnd.nextDouble())); + emit(v * (1.3 + 0.2 * rnd.nextDouble())); + continue; + } + p += c.missedPerMin * beatMin; + if (u < p) { + emit(v * (1.9 + 0.2 * rnd.nextDouble())); + continue; + } + p += c.extraPerMin * beatMin; + if (u < p) { + final f = 0.35 + 0.3 * rnd.nextDouble(); + emit(v * f); + emit(v * (1 - f)); + continue; + } + p += c.gapPerHour * beatMin / 60.0; + if (u < p) { + final gap = 5 + rnd.nextInt(896); + t += gap.toDouble(); + continue; + } + p += c.backwardsPerHour * beatMin / 60.0; + if (u < p && ts.isNotEmpty) { + emit(v); + ts[ts.length - 1] = ts.last - 1000.0 * (1 + rnd.nextInt(3)); + continue; + } + emit(v); + } + return RrData(rr, ts); +} + +/// The checked-in anonymised 8.9 h WHOOP-4 night (rebased to 0 s). Returns +/// null if the fixture is missing (callers decide whether to fail). +RrData? realNightRr({double startSec = 1.7e9}) { + final f = File('test/onehz/fixtures/real_night_2026_07_rr.csv'); + if (!f.existsSync()) return null; + final rr = [], ts = []; + for (final line in f.readAsLinesSync().skip(1)) { + if (line.trim().isEmpty) continue; + final p = line.split(','); + ts.add(startSec * 1000 + double.parse(p[0])); + rr.add(double.parse(p[1])); + } + return RrData(rr, ts); +} + +/// A whole 24 h-ish day: synthetic morning + REAL night + synthetic evening. +/// The synthetic parts carry the artefact load; the night is as recorded. +RrData realShapedDay({int seed = 7, double startSec = 1.7e9}) { + final real = realNightRr(startSec: startSec)!; + final nightSpanMs = real.ts.last - real.ts.first; + final morning = synthRr(SynthConfig( + seed: seed, hours: 5, startSec: startSec - 5 * 3600, nightFraction: 0)); + final evening = synthRr(SynthConfig( + seed: seed + 1, + hours: 9, + startSec: startSec + nightSpanMs / 1000 + 120, + nightFraction: 0)); + return RrData([...morning.rr, ...real.rr, ...evening.rr], + [...morning.ts, ...real.ts, ...evening.ts]); +} diff --git a/test/onehz/wallclock_agreement_test.dart b/test/onehz/wallclock_agreement_test.dart new file mode 100644 index 0000000..53d120c --- /dev/null +++ b/test/onehz/wallclock_agreement_test.dart @@ -0,0 +1,227 @@ +// GUARD: the gap-anchored grid against today's labels on the checked-in real +// night (combined bump, chunk B3 + B4; owner decision 2026-10-22). +// +// * `fixtures/real_night_2026_07_hypnogram_v1.csv` is today's hypnogram for +// that night (AdvancedSleepStager.stageWindow at analytics 83fc308, the +// run-start-anchored grid), computed for a window starting at the +// UNALIGNED second 1700000013, which is also the night's first sample. It is +// frozen: regenerate it only as a deliberate decision. +// * AGREEMENT GUARD (B4): per 30 s epoch of that grid (steps of 30 s from the +// first sample), the new label equals the frozen one at least +// [kMinAgreement] = 0.97 of the time (both staged; the epoch's midpoint +// second is looked up). Anchoring at the first sample of each gap-free run +// is the old grid for a window that starts at the data, so this is a near +// identity; the wall-clock grid it replaced scored 0.905 here. +// * The harness (tool/stager_harness.dart) cannot stand in for this: it +// scores the decision layer on an external PSG corpus (DREAMT feature +// export) that is not in the repo, and it has no grid-comparison mode. The +// skipped test at the end says what that needs. +import 'dart:convert'; +import 'dart:io'; + +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +import 'support/pre_bump_stager.dart'; + +const kMinAgreement = 0.97; +const _base = 1700000013; // unaligned: 1700000013 % 30 == 3 + +class _Night { + final List hr = []; + final List accel = []; + final List rrMs = [], rrTs = []; + int seconds = 0; + List get grav => + [for (final a in accel) GravTs(a.tsMs ~/ 1000, a.x, a.y, a.z)]; + List get hrTs => [for (final h in hr) HrTs(h.tsMs ~/ 1000, h.hr)]; + List get rrSec => + [for (var i = 0; i < rrMs.length; i++) RrTs(rrTs[i] ~/ 1000, rrMs[i])]; +} + +_Night? _load() { + final a = File('test/onehz/fixtures/real_night_2026_07_onehz.csv'); + final b = File('test/onehz/fixtures/real_night_2026_07_rr.csv'); + if (!a.existsSync() || !b.existsSync()) return null; + final n = _Night(); + for (final l in a.readAsLinesSync().skip(1)) { + if (l.trim().isEmpty) continue; + final p = l.split(','); + final s = int.parse(p[0]); + final ts = (_base + s) * 1000.0; + n.seconds = s + 1; + n.hr.add(HrSample(ts, double.parse(p[1]))); + n.accel.add(AccelSample( + ts, double.parse(p[2]), double.parse(p[3]), double.parse(p[4]))); + } + for (final l in b.readAsLinesSync().skip(1)) { + if (l.trim().isEmpty) continue; + final p = l.split(','); + n.rrTs.add(_base * 1000.0 + double.parse(p[0])); + n.rrMs.add(double.parse(p[1])); + } + return n; +} + +List<(int, int, String)> _golden() => [ + for (final l in File('test/onehz/fixtures/real_night_2026_07_hypnogram_v1.csv') + .readAsLinesSync() + .skip(1)) + if (l.trim().isNotEmpty) + ( + int.parse(l.split(',')[0]), + int.parse(l.split(',')[1]), + l.split(',')[2] + ) + ]; + +String? _at(List<(int, int, String)> segs, int rel) { + for (final s in segs) { + if (rel >= s.$1 && rel < s.$2) return s.$3; + } + return null; +} + +void main() { + final night = _load(); + + test('fixtures are present', () { + expect(night, isNotNull); + expect(night!.seconds, greaterThan(32000)); + expect(_golden().length, greaterThan(10)); + expect(_base % 30, isNot(0)); + }); + + final nightOrNull = night; + if (nightOrNull == null) return; + + SleepSession stageBatch() => AdvancedSleepStager.stageWindow( + _base, _base + nightOrNull.seconds, nightOrNull.grav, nightOrNull.hrTs, + rr: nightOrNull.rrSec); + + late final SleepSession now = stageBatch(); + + test('per-epoch agreement with today\'s labels is at least $kMinAgreement', () { + final old = _golden(); + final stages = [ + for (final g in now.stages) (g.start - _base, g.end - _base, g.stage) + ]; + var epochs = 0, compared = 0, agree = 0; + for (var start = _base; start + 30 <= _base + nightOrNull.seconds; start += 30) { + epochs++; + final mid = start + 15 - _base; + final a = _at(old, mid), b = _at(stages, mid); + if (a == null || b == null || a == 'unobserved' || b == 'unobserved') continue; + compared++; + if (a == b) agree++; + } + expect(compared, greaterThan(epochs * 0.9), + reason: 'most of the night must be staged under both grids'); + final agreement = agree / compared; + expect(agreement, greaterThanOrEqualTo(kMinAgreement), + reason: 'agreement $agree/$compared = ${agreement.toStringAsFixed(4)}'); + }); + + // Grid option 2 (owner decision 2026-10-23): the window start is phase 0 of + // the epoch grid, as before the bump. On this night the start is the first + // sample, so agreement with the frozen hypnogram is exactly 1.0 (it is 1.0 + // under the gap-anchored grid too; this keeps it there). + test('onset grid: agreement with the frozen hypnogram is exactly 1.0', () { + final old = _golden(); + final stages = [ + for (final g in now.stages) (g.start - _base, g.end - _base, g.stage) + ]; + var compared = 0, agree = 0; + for (var start = _base; start + 30 <= _base + nightOrNull.seconds; start += 30) { + final mid = start + 15 - _base; + final a = _at(old, mid), b = _at(stages, mid); + // Complete label equality: 'unobserved' (and missing) epochs count too. + compared++; + if (a == b) agree++; + } + expect(compared, greaterThan(1000)); + expect(agree, compared, reason: 'agreement $agree/$compared'); + }); + + test('a window that starts mid-night (unaligned to the first sample) is ' + 'staged on the onset grid: labels equal the pre-bump stager', () { + // 5 h in, 17 s off the 30 s steps of the first sample, 3 h long. + final start = _base + 18000 + 17, end = start + 3 * 3600; + final got = AdvancedSleepStager.stageWindow( + start, end, nightOrNull.grav, nightOrNull.hrTs, rr: nightOrNull.rrSec) + .stages; + final want = preBumpStageSessionCardio( + start, end, nightOrNull.grav, nightOrNull.hrTs, nightOrNull.rrSec); + expect(showSegments(got), showSegments(want)); + }); + + test('the real night\'s hypnogram boundaries sit on the grid of their ' + 'gap-free run (30 s steps from the run\'s first sample)', () { + // The run anchors, found independently of the code under test: a new run + // starts at the first valid sample more than 61 s after the previous one + // (the stager's 60 s carry-forward bridges anything shorter). + final anchors = []; + int? prev; + for (final g in nightOrNull.grav) { + if (prev == null || g.ts > prev + 61) anchors.add(g.ts); + prev = g.ts; + } + expect(anchors.first, _base); + final stages = now.stages; + var checked = 0; + for (var i = 1; i < stages.length; i++) { + if (stages[i].stage == 'unobserved' || stages[i - 1].stage == 'unobserved') { + continue; + } + final edge = stages[i].start; + final anchor = anchors.lastWhere((a) => a <= edge); + expect((edge - anchor) % 30, 0, + reason: 'boundary at $edge (${stages[i - 1].stage} -> ' + '${stages[i].stage}), run anchored at $anchor'); + checked++; + } + expect(checked, greaterThan(20)); + }); + + test('EpochRows over the whole night in 15-minute chunks with checkpoints ' + 'stages exactly like one batch, and its checkpoint stays small', () { + var er = EpochRows(); + var worst = 0; + for (var lo = 0; lo < nightOrNull.seconds; lo += 900) { + final hi = lo + 900 < nightOrNull.seconds ? lo + 900 : nightOrNull.seconds; + final t0 = (_base + lo) * 1000.0, t1 = (_base + hi) * 1000.0; + bool inside(double t) => t >= t0 && t < t1; + final ri = [ + for (var i = 0; i < nightOrNull.rrTs.length; i++) + if (inside(nightOrNull.rrTs[i])) i + ]; + er.fold( + hr: nightOrNull.hr.sublist(lo, hi), + accel: nightOrNull.accel.sublist(lo, hi), + rrMs: [for (final i in ri) nightOrNull.rrMs[i]], + rrTsMs: [for (final i in ri) nightOrNull.rrTs[i]], + ); + final text = jsonEncode(er.toJson()); + if (text.length > worst) worst = text.length; + er = EpochRows.fromJson(jsonDecode(text) as Map); + } + // ~1,070 rows of 8 numbers plus an unsettled tail of a few minutes of + // samples; the raw night (32,000 s x 5 channels) would be several times this. + expect(worst, lessThan(800000), reason: 'checkpoint chars'); + final got = er.stageWindow(_base, _base + nightOrNull.seconds); + expect(jsonEncode([for (final g in got) g.toJson()]), + jsonEncode([for (final g in now.stages) g.toJson()])); + }); + + test('DREAMT harness: per-epoch agreement of the gap-anchored grid with the ' + 'run-start grid on the PSG corpus', () { + // Needs (1) the DREAMT feature export (stager_harness.dart's fixture JSON), + // which is not in this repo, and (2) a grid-comparison mode in + // tool/stager_harness.dart: classify the same subject once on epochs cut at + // the subject's start and once on epochs cut 0..29 s later, report per-epoch + // agreement and kappa for both against the PSG labels. Threshold: 0.97. + fail('unreachable: skipped until the corpus and the harness mode exist'); + }, skip: 'DREAMT fixture is not in the repo and tool/stager_harness.dart has no ' + 'grid-comparison mode; run it on the machine that holds the corpus ' + '(B4 of combined-bump/brief.md).'); +} diff --git a/test/onehz/welch_stream_test.dart b/test/onehz/welch_stream_test.dart new file mode 100644 index 0000000..cec8e66 --- /dev/null +++ b/test/onehz/welch_stream_test.dart @@ -0,0 +1,633 @@ +// RED (round 18): exact streaming hrvFreq and RSA respiration. +// +// HrvFreqStream / RsaRespStream must equal the batch hrvFreq / rsaRespRate on +// (folded ++ tail) BIT FOR BIT at every fold: == on doubles, identical +// NaN-ness and sign of zero, same notes, same totalBands. No tolerance. +// +// The batch functions are the reference (they do not change in this round). +import 'dart:convert'; +import 'dart:math' as math; + +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +// ───────────────────────── realistic NN night ─────────────────────────────── + +class Night { + final List nn = [], times = []; +} + +/// A seeded sleep-like night of [hours] with ectopic/missed/extra beats, short +/// garbage runs and one 2-min dropout; cleaned by the production [correctRr] +/// (with timestamps), so the NN carries the same shapes as production input. +/// Breathing at 0.24 Hz, slow HR (~57 bpm), 0.1 Hz and VLF modulation. +Night synthNight(int seed, double hours, + {double bpmBase = 57, bool dropout = true}) { + final rnd = math.Random(seed); + const startMs = 1.7e12; + final endT = hours * 3600e3; + final rr = [], ts = []; + var t = 0.0; + final phase = rnd.nextDouble() * 6.28; + final rate = 0.005 + 0.01 * rnd.nextDouble(); + final dropAt = endT * (0.3 + 0.3 * rnd.nextDouble()); + var dropped = !dropout; + void emit(double v) { + v = v.roundToDouble(); + t += v; + rr.add(v); + ts.add(((startMs + t) / 1000).floorToDouble() * 1000); + } + + while (t < endT) { + final s = t / 1000; + final bpm = bpmBase + 7 * math.sin(s / 5400 + seed); + final v = 60000 / bpm + + 28 * math.sin(2 * math.pi * 0.24 * s + phase) + + 18 * math.sin(2 * math.pi * 0.1 * s) + + 22 * math.sin(2 * math.pi * 0.01 * s) + + 6 * (rnd.nextDouble() - .5) * 2; + if (!dropped && t >= dropAt) { + dropped = true; + t += 120e3; + continue; + } + if (rnd.nextDouble() < rate) { + final u = rnd.nextDouble(); + if (u < 0.4) { + emit(v * (0.55 + 0.15 * rnd.nextDouble())); + emit(v * (1.3 + 0.2 * rnd.nextDouble())); + } else if (u < 0.6) { + emit(v * (1.9 + 0.2 * rnd.nextDouble())); + } else if (u < 0.85) { + final f = 0.35 + 0.3 * rnd.nextDouble(); + emit(v * f); + emit(v * (1 - f)); + } else { + emit(250 + rnd.nextInt(2100).toDouble()); + } + continue; + } + emit(v); + } + final c = correctRr(rr, rrTsMs: ts); + return Night() + ..nn.addAll(c.nn) + ..times.addAll(c.nnTimesMs); +} + +// ───────────────────────── bit-for-bit comparison ─────────────────────────── + +bool sameD(double? a, double? b) { + if (a == null || b == null) return a == null && b == null; + if (a.isNaN || b.isNaN) return a.isNaN && b.isNaN; + return a == b && a.isNegative == b.isNegative; +} + +void expectSameD(double? a, double? b, String what) => + expect(sameD(a, b), isTrue, reason: '$what: $a vs $b'); + +void expectSameMetric(Metric a, Metric b, String at) { + expect(a.value == null, b.value == null, reason: '$at value presence'); + expectSameD(a.confidence, b.confidence, '$at confidence'); + expect(a.tier, b.tier, reason: at); + expect(a.inputs_used, b.inputs_used, reason: at); + expect(a.note, b.note, reason: '$at note'); + expect(a.drivers == null, b.drivers == null, reason: at); +} + +void expectSameHrv(Metric a, Metric b, String at) { + expectSameMetric(a, b, at); + final x = a.value, y = b.value; + if (x == null || y == null) return; + expectSameD(x.ulf, y.ulf, '$at ulf'); + expectSameD(x.vlf, y.vlf, '$at vlf'); + expectSameD(x.lf, y.lf, '$at lf'); + expectSameD(x.hf, y.hf, '$at hf'); + expectSameD(x.total, y.total, '$at total'); + expect(x.totalBands, y.totalBands, reason: '$at totalBands'); + expectSameD(x.lfhf, y.lfhf, '$at lfhf'); + expectSameD(x.nuLf, y.nuLf, '$at nuLf'); + expectSameD(x.nuHf, y.nuHf, '$at nuHf'); + expect(x.hfGated, y.hfGated, reason: at); +} + +void expectSameResp( + Metric a, Metric b, String at) { + expectSameMetric(a, b, at); + final x = a.value, y = b.value; + if (x == null || y == null) return; + expectSameD(x.brpm, y.brpm, '$at brpm'); + expectSameD(x.peakHz, y.peakHz, '$at peakHz'); + expectSameD(x.power, y.power, '$at power'); + expect(x.source, y.source, reason: at); +} + +// ───────────────────────── drivers ────────────────────────────────────────── + +/// Chunk boundaries (beat counts) with random sizes in [lo, hi], starting small +/// so the first folds sit below every minimum-beats / minimum-span gate. +List chunkEnds(int n, math.Random rnd, int lo, int hi, {int first = 8}) { + final ends = []; + var at = first; + while (at < n) { + ends.add(at); + at += lo + rnd.nextInt(hi - lo + 1); + } + ends.add(n); + return ends; +} + +Metric batchHrv(Night n, int end, double af, + {double gate = 0.15, double os = 4.0}) => + hrvFreq(n.nn.sublist(0, end), n.times.sublist(0, end), + artifactFraction: af, hfArtifactGate: gate, oversample: os); + +Metric batchRsa(Night n, int end, double af, + {double tol = 2.0, double cons = 0.5, double maxA = 0.30}) => + rsaRespRate(n.nn.sublist(0, end), n.times.sublist(0, end), + artifactFraction: af, + tolBrpm: tol, + minConsensus: cons, + maxArtifact: maxA); + +void hrvParity(Night n, List ends, double af, {int tailBeats = 100}) { + final s = HrvFreqStream(); + var from = 0; + for (final end in ends) { + s.fold(n.nn.sublist(from, end), n.times.sublist(from, end)); + from = end; + expectSameHrv(s.snapshot(artifactFraction: af), batchHrv(n, end, af), + 'hrv prefix $end'); + final tEnd = math.min(n.nn.length, end + tailBeats); + expectSameHrv( + s.snapshot( + tailNnMs: n.nn.sublist(end, tEnd), + tailNnTimesMs: n.times.sublist(end, tEnd), + artifactFraction: af), + batchHrv(n, tEnd, af), + 'hrv prefix $end + tail ${tEnd - end}'); + } +} + +void rsaParity(Night n, List ends, double af, {int tailBeats = 100}) { + final s = RsaRespStream(); + var from = 0; + for (final end in ends) { + s.fold(n.nn.sublist(from, end), n.times.sublist(from, end)); + from = end; + expectSameResp(s.snapshot(artifactFraction: af), batchRsa(n, end, af), + 'rsa prefix $end'); + final tEnd = math.min(n.nn.length, end + tailBeats); + expectSameResp( + s.snapshot( + tailNnMs: n.nn.sublist(end, tEnd), + tailNnTimesMs: n.times.sublist(end, tEnd), + artifactFraction: af), + batchRsa(n, tEnd, af), + 'rsa prefix $end + tail ${tEnd - end}'); + } +} + +Object? wire(Map j) => jsonDecode(jsonEncode(j)); + +/// Every top-level key removed, and every top-level value swapped for a wrong +/// type, must be a FormatException (never a TypeError / RangeError / success). +void expectMalformedRejected( + Map good, Object? Function(Map) revive) { + for (final k in good.keys) { + final missing = Map.from(good)..remove(k); + expect(() => revive(missing), throwsFormatException, reason: 'missing $k'); + final wrong = Map.from(good)..[k] = {'bogus': true}; + expect(() => revive(wrong), throwsFormatException, reason: 'wrong $k'); + } + expect(() => revive({}), throwsFormatException); + expect(() => revive({...good, 'type': 'NotAStream'}), throwsFormatException); + expect(() => revive({...good, 'version': 9999}), throwsFormatException); + expect(() => revive({...good, 'version': 'one'}), throwsFormatException); +} + +void main() { + group('fresh and short-prefix streams equal the batch on the tail', () { + final ks = [1, 2, 8, 15, 16, 17, 19, 20, 21, 29, 30, 31, 60, 200, 1000]; + + void check(Night n, List tails, {int folded = 0}) { + for (final k in tails) { + final end = math.min(folded + k, n.nn.length); + if (end <= folded) continue; + final h = HrvFreqStream(); + final r = RsaRespStream(); + if (folded > 0) { + h.fold(n.nn.sublist(0, folded), n.times.sublist(0, folded)); + r.fold(n.nn.sublist(0, folded), n.times.sublist(0, folded)); + } + final tn = n.nn.sublist(folded, end), tt = n.times.sublist(folded, end); + expectSameHrv( + h.snapshot(tailNnMs: tn, tailNnTimesMs: tt, artifactFraction: 0.02), + batchHrv(n, end, 0.02), + 'hrv folded $folded tail $k'); + expectSameResp( + r.snapshot(tailNnMs: tn, tailNnTimesMs: tt, artifactFraction: 0.02), + batchRsa(n, end, 0.02), + 'rsa folded $folded tail $k'); + } + } + + test('fresh stream, tails across the 16/20/30 gates', () { + check(synthNight(91, 1), ks); + }); + + test('fresh stream, tail is a full 2 h night (with a dropout)', () { + final n = synthNight(92, 2); + check(n, [n.nn.length]); + }); + + test('a few folded beats plus a long tail', () { + final n = synthNight(93, 2); + for (final f in [1, 5, 8, 15, 16, 19, 20]) { + check(n, [1, 5, 30, 200, n.nn.length], folded: f); + } + }); + }); + + group('HrvFreqStream', () { + test('prefix parity at every fold, empty and non-empty tail (2 h night)', + () { + final n = synthNight(1, 2); + final ends = chunkEnds(n.nn.length, math.Random(11), 700, 1500); + hrvParity(n, ends, 0.02); + }); + + test('HF gate path and non-default gate / oversample are bit-identical', + () { + final n = synthNight(2, 1.0); + final s = HrvFreqStream(oversample: 2.0); + final ends = chunkEnds(n.nn.length, math.Random(5), 700, 1300); + var from = 0; + for (final end in ends) { + s.fold(n.nn.sublist(from, end), n.times.sublist(from, end)); + from = end; + for (final af in [0.02, 0.2]) { + expectSameHrv(s.snapshot(artifactFraction: af), + batchHrv(n, end, af, os: 2.0), 'af $af prefix $end'); + } + expectSameHrv( + s.snapshot(artifactFraction: 0.1, hfArtifactGate: 0.05), + batchHrv(n, end, 0.1, gate: 0.05, os: 2.0), + 'gate 0.05 prefix $end'); + } + }); + + test('absent paths: <16 beats, degenerate times, fresh stream', () { + final fresh = HrvFreqStream(); + expectSameHrv( + fresh.snapshot(artifactFraction: 0), + hrvFreq(const [], const [], artifactFraction: 0), + 'fresh stream, no beats'); + // 15 beats: below the 16-beat floor. + final s = HrvFreqStream(); + final nn = List.generate(15, (i) => 800.0 + i); + final ts = List.generate(15, (i) => 1000.0 * i); + s.fold(nn, ts); + expectSameHrv(s.snapshot(artifactFraction: 0), + hrvFreq(nn, ts, artifactFraction: 0), '15 beats'); + // The 16th arrives as a tail only. + expectSameHrv( + s.snapshot( + tailNnMs: [900], + tailNnTimesMs: [15000], + artifactFraction: 0), + hrvFreq([...nn, 900], [...ts, 15000], artifactFraction: 0), + '15 + tail 1'); + // 20 beats at one instant: span 0 -> 'degenerate beat times'. + final d = HrvFreqStream(); + final dn = List.filled(20, 800); + final dt = List.filled(20, 5000); + d.fold(dn, dt); + expectSameHrv(d.snapshot(artifactFraction: 0), + hrvFreq(dn, dt, artifactFraction: 0), 'degenerate'); + // A short record: 20 min, no band but HF/LF resolvable? whatever the + // batch says (including 'record too short to resolve any HRV band'). + final short = synthNight(3, 0.05); + final sh = HrvFreqStream()..fold(short.nn, short.times); + expectSameHrv(sh.snapshot(artifactFraction: 0.01), + hrvFreq(short.nn, short.times, artifactFraction: 0.01), 'short'); + }); + + test('segments close exactly at start+seg: beat on / just before the edge', + () { + // 1 s beats: HF segment is 66.67 s, 50 % overlap. Probe every prefix + // length around several segment ends with single-beat folds. + final nn = List.filled(400, 1000); + final ts = List.generate(400, (i) => 1000.0 * i); + final n = Night() + ..nn.addAll(nn) + ..times.addAll(ts); + final s = HrvFreqStream(); + for (var i = 0; i < n.nn.length; i++) { + s.fold([n.nn[i]], [n.times[i]]); + if (i >= 60 && i <= 140 || i >= 190 && i <= 210) { + expectSameHrv(s.snapshot(artifactFraction: 0.01), + batchHrv(n, i + 1, 0.01), 'single-beat fold $i'); + } + } + }); + + test('8 h night: VLF segments close and stay exact', () { + final n = synthNight(4, 8); + final len = n.nn.length; + final ends = [len ~/ 3, (len * 2) ~/ 3, len]; + final s = HrvFreqStream(); + var from = 0; + for (final end in ends) { + s.fold(n.nn.sublist(from, end), n.times.sublist(from, end)); + from = end; + final got = s.snapshot(artifactFraction: 0.02); + expectSameHrv(got, batchHrv(n, end, 0.02), '8h prefix $end'); + if (end == len) { + expect(got.value?.vlf, isNotNull, reason: 'VLF resolved at 8 h'); + } + } + }); + + test('snapshot never mutates: different tails, then folding, still exact', + () { + final n = synthNight(5, 1.5); + final len = n.nn.length; + final mid = len ~/ 2; + final s = HrvFreqStream()..fold(n.nn.sublist(0, mid), n.times.sublist(0, mid)); + final before = s.snapshot(artifactFraction: 0.03); + // A long tail that would close many segments, then a different short one. + s.snapshot( + tailNnMs: n.nn.sublist(mid, len), + tailNnTimesMs: n.times.sublist(mid, len), + artifactFraction: 0.03); + s.snapshot( + tailNnMs: [700, 710, 720], + tailNnTimesMs: [ + n.times[mid - 1] + 700, + n.times[mid - 1] + 1410, + n.times[mid - 1] + 2130 + ], + artifactFraction: 0.5); + expectSameHrv(s.snapshot(artifactFraction: 0.03), before, 'repeat'); + expectSameHrv(before, batchHrv(n, mid, 0.03), 'before'); + // The real continuation overlaps what the tails pretended. + s.fold(n.nn.sublist(mid, len), n.times.sublist(mid, len)); + expectSameHrv(s.snapshot(artifactFraction: 0.03), batchHrv(n, len, 0.03), + 'after folding'); + // A tail may also be folded afterwards with the very same beats. + final json1 = jsonEncode(s.toJson()); + s.snapshot( + tailNnMs: [800], tailNnTimesMs: [n.times.last + 800], artifactFraction: 0); + expect(jsonEncode(s.toJson()), json1, reason: 'toJson unchanged'); + }); + + test('JSON round trip mid-night resumes bit-identically', () { + final n = synthNight(6, 1.5); + final len = n.nn.length; + final cuts = [len ~/ 5, (len * 2) ~/ 5, (len * 3) ~/ 5, len]; + var s = HrvFreqStream(); + var from = 0; + for (final end in cuts) { + s.fold(n.nn.sublist(from, end), n.times.sublist(from, end)); + from = end; + if (end != len) { + final text = jsonEncode(s.toJson()); + s = HrvFreqStream.fromJson( + jsonDecode(text) as Map); + // lossless: re-serializing the revived stream is byte-identical. + expect(jsonEncode(s.toJson()), text); + } + expectSameHrv(s.snapshot(artifactFraction: 0.02), + batchHrv(n, end, 0.02), 'resumed prefix $end'); + } + // A fresh stream also round-trips and keeps the oversample setting. + final f = HrvFreqStream(oversample: 3.0); + final f2 = HrvFreqStream.fromJson(wire(f.toJson()) as Map); + f2.fold(n.nn.sublist(0, 800), n.times.sublist(0, 800)); + expectSameHrv( + f2.snapshot(artifactFraction: 0.02), + batchHrv(n, 800, 0.02, os: 3.0), + 'oversample survives round trip'); + }); + + test('malformed JSON is a FormatException', () { + final n = synthNight(7, 1.2); + final s = HrvFreqStream()..fold(n.nn, n.times); + final good = wire(s.toJson()) as Map; + expectMalformedRejected(good, + (j) => HrvFreqStream.fromJson(j)); + }); + + test('non-monotone fold is a StateError', () { + final s = HrvFreqStream() + ..fold([800, 810, 820], [1000, 1810, 2630]); + // Earlier than the previous fold's last beat. + expect(() => s.fold([800], [2000]), throwsStateError); + // Decreasing inside one call. + expect(() => HrvFreqStream().fold([800, 800], [5000, 4000]), + throwsStateError); + // Equal times are allowed (non-decreasing) and do not throw. + final eq = HrvFreqStream()..fold([800], [1000]); + expect(() => eq.fold([800], [1000]), returnsNormally); + }); + }); + + group('RsaRespStream', () { + test('prefix parity at every fold, empty and non-empty tail (2 h night)', + () { + final n = synthNight(11, 2); + // First folds are tiny: < 20 beats, then span < 600 s where the batch + // shortens its segment to span/2. + final ends = chunkEnds(n.nn.length, math.Random(21), 400, 1300, first: 12); + rsaParity(n, ends, 0.02); + }); + + test('short-span prefixes (segment = span/2 regime) fold beat by beat', + () { + final n = synthNight(12, 0.4); + final s = RsaRespStream(); + for (var i = 0; i < n.nn.length && i < 900; i++) { + s.fold([n.nn[i]], [n.times[i]]); + if (i > 15 && i % 7 == 0) { + expectSameResp(s.snapshot(artifactFraction: 0.02), + batchRsa(n, i + 1, 0.02), 'single-beat fold $i'); + } + } + }); + + test('gates and options: artifact gate, tolerance, consensus', () { + final n = synthNight(13, 0.8); + final s = RsaRespStream(); + final ends = chunkEnds(n.nn.length, math.Random(2), 500, 900); + var from = 0; + for (final end in ends) { + s.fold(n.nn.sublist(from, end), n.times.sublist(from, end)); + from = end; + expectSameResp(s.snapshot(artifactFraction: 0.4), + batchRsa(n, end, 0.4), 'artifact 0.4 prefix $end'); + expectSameResp( + s.snapshot(artifactFraction: 0.4, maxArtifact: 0.5), + batchRsa(n, end, 0.4, maxA: 0.5), + 'maxArtifact 0.5 prefix $end'); + expectSameResp( + s.snapshot(artifactFraction: 0.05, tolBrpm: 0.1, minConsensus: 0.9), + batchRsa(n, end, 0.05, tol: 0.1, cons: 0.9), + 'strict consensus prefix $end'); + } + }); + + test('slow heart (NN 1400 ms): window ceiling below HF band -> withheld', + () { + final nn = List.generate(500, (i) => 1400.0 + 20 * math.sin(i / 3)); + final ts = []; + var t = 0.0; + for (final v in nn) { + t += v; + ts.add(t); + } + final n = Night() + ..nn.addAll(nn) + ..times.addAll(ts); + final s = RsaRespStream(); + for (var from = 0; from < 500; from += 100) { + s.fold(nn.sublist(from, from + 100), ts.sublist(from, from + 100)); + expectSameResp(s.snapshot(artifactFraction: 0.01), + batchRsa(n, from + 100, 0.01), 'slow HR prefix ${from + 100}'); + } + }); + + test('absent paths: fresh stream, <20 beats, degenerate times', () { + expectSameResp( + RsaRespStream().snapshot(artifactFraction: 0), + rsaRespRate(const [], const [], artifactFraction: 0), + 'fresh'); + final nn = List.generate(19, (i) => 800.0 + i); + final ts = List.generate(19, (i) => 1000.0 * i); + final s = RsaRespStream()..fold(nn, ts); + expectSameResp(s.snapshot(artifactFraction: 0), + rsaRespRate(nn, ts, artifactFraction: 0), '19 beats'); + expectSameResp( + s.snapshot( + tailNnMs: [900], tailNnTimesMs: [19000], artifactFraction: 0), + rsaRespRate([...nn, 900], [...ts, 19000], artifactFraction: 0), + '19 + tail 1'); + final dn = List.filled(25, 800); + final dt = List.filled(25, 5000); + final d = RsaRespStream()..fold(dn, dt); + expectSameResp(d.snapshot(artifactFraction: 0), + rsaRespRate(dn, dt, artifactFraction: 0), 'degenerate'); + }); + + test('segments close exactly at start+300 s: single-beat folds at the edge', + () { + // 1 s beats, 300 s segments, 150 s step: probe prefixes around ends. + final nn = List.filled(1000, 1000); + for (var i = 0; i < nn.length; i++) { + nn[i] += 40 * math.sin(2 * math.pi * 0.24 * i); + } + final ts = []; + for (var i = 0; i < nn.length; i++) { + ts.add(1000.0 * i); + } + final n = Night() + ..nn.addAll(nn) + ..times.addAll(ts); + final s = RsaRespStream(); + for (var i = 0; i < n.nn.length; i++) { + s.fold([n.nn[i]], [n.times[i]]); + final near = [299, 449, 599, 749, 899] + .any((e) => i >= e - 3 && i <= e + 3); + if (near || i == 119 || i == 120 || i == 121) { + expectSameResp(s.snapshot(artifactFraction: 0.01), + batchRsa(n, i + 1, 0.01), 'single-beat fold $i'); + } + } + }); + + test('night with a dropout: thin segments counted the same', () { + final n = synthNight(14, 1.5, dropout: true); + rsaParity(n, chunkEnds(n.nn.length, math.Random(3), 500, 1200), 0.01); + }); + + test('8 h night: parity at checkpoints', () { + final n = synthNight(15, 8); + final len = n.nn.length; + final s = RsaRespStream(); + var from = 0; + for (final end in [len ~/ 3, (len * 2) ~/ 3, len]) { + s.fold(n.nn.sublist(from, end), n.times.sublist(from, end)); + from = end; + expectSameResp(s.snapshot(artifactFraction: 0.02), + batchRsa(n, end, 0.02), '8h prefix $end'); + } + }); + + test('snapshot never mutates: different tails, then folding, still exact', + () { + final n = synthNight(16, 1.5); + final len = n.nn.length; + final mid = len ~/ 2; + final s = RsaRespStream()..fold(n.nn.sublist(0, mid), n.times.sublist(0, mid)); + final before = s.snapshot(artifactFraction: 0.03); + s.snapshot( + tailNnMs: n.nn.sublist(mid, len), + tailNnTimesMs: n.times.sublist(mid, len), + artifactFraction: 0.03); + s.snapshot( + tailNnMs: [700, 710, 720], + tailNnTimesMs: [ + n.times[mid - 1] + 700, + n.times[mid - 1] + 1410, + n.times[mid - 1] + 2130 + ], + artifactFraction: 0.5); + expectSameResp(s.snapshot(artifactFraction: 0.03), before, 'repeat'); + expectSameResp(before, batchRsa(n, mid, 0.03), 'before'); + s.fold(n.nn.sublist(mid, len), n.times.sublist(mid, len)); + expectSameResp(s.snapshot(artifactFraction: 0.03), + batchRsa(n, len, 0.03), 'after folding'); + final json1 = jsonEncode(s.toJson()); + s.snapshot( + tailNnMs: [800], tailNnTimesMs: [n.times.last + 800], artifactFraction: 0); + expect(jsonEncode(s.toJson()), json1, reason: 'toJson unchanged'); + }); + + test('JSON round trip mid-night resumes bit-identically', () { + final n = synthNight(17, 1.5); + final len = n.nn.length; + // Cuts include one inside the span<600 s regime. + final cuts = [60, len ~/ 5, (len * 2) ~/ 5, (len * 3) ~/ 5, len]; + var s = RsaRespStream(); + var from = 0; + for (final end in cuts) { + s.fold(n.nn.sublist(from, end), n.times.sublist(from, end)); + from = end; + if (end != len) { + final text = jsonEncode(s.toJson()); + s = RsaRespStream.fromJson(jsonDecode(text) as Map); + expect(jsonEncode(s.toJson()), text); + } + expectSameResp(s.snapshot(artifactFraction: 0.02), + batchRsa(n, end, 0.02), 'resumed prefix $end'); + } + }); + + test('malformed JSON is a FormatException', () { + final n = synthNight(18, 1.2); + final s = RsaRespStream()..fold(n.nn, n.times); + final good = wire(s.toJson()) as Map; + expectMalformedRejected(good, (j) => RsaRespStream.fromJson(j)); + }); + + test('non-monotone fold is a StateError', () { + final s = RsaRespStream()..fold([800, 810, 820], [1000, 1810, 2630]); + expect(() => s.fold([800], [2000]), throwsStateError); + expect(() => RsaRespStream().fold([800, 800], [5000, 4000]), + throwsStateError); + final eq = RsaRespStream()..fold([800], [1000]); + expect(() => eq.fold([800], [1000]), returnsNormally); + }); + }); +} diff --git a/test/properties/incremental_laws_test.dart b/test/properties/incremental_laws_test.dart new file mode 100644 index 0000000..b9e2f9b --- /dev/null +++ b/test/properties/incremental_laws_test.dart @@ -0,0 +1,3172 @@ +// LAWS of the incremental calculations (design 05, pilot cluster C2A, analytics +// side): the state objects behind `lib/src/onehz/incremental*.dart` -- +// `IncrementalHrvTime`, `IncrementalEnmoSeries`, `IncrementalLombScargle`, +// `IncrementalMinuteMetrics` (all four are fed the WHOLE series so far on every +// `sync` and keep sums for the prefix they have already seen), and the two +// mergeable summaries under them, `RunningMoments` and `IntHistogram`, plus the +// `CalculationCache` that guards the results. +// +// L1 chunk invariance, stated for a sync: syncing growing prefixes of a +// series in ANY chunking (empty prefixes and a save/restore through real +// JSON text between syncs included) gives the checkpoint of one sync of +// the whole series -- and, because a state that cannot continue starts +// over, the same holds after any DETOUR (an edited series, a dropped +// prefix, a forced rebuild, another artifact fraction / reference) as +// long as the last sync is of the same series. The checkpoint is +// compared as text without the work counters (`processedPoints`, +// `processedMinutes`: performance contracts, kept as examples). The +// checkpoint of the minute metrics lists its bills in the order the keys +// were first seen, so after a detour that REORDERS keys it is compared +// by key; chunking alone gives the same text. +// L1b streamed vs batch PARITY: after every checked sync the output is what +// the production batch function gives for the same series (`hrvTime`, +// `enmoSeries`, `lombScargle`, `banisterTrimp` / `strainScoreMetric` / +// `Calories`), counts and abstentions exact, numbers within +// max(1e-9, 1e-8 relative) (running sums against a two-pass batch). +// These batch functions are NOT independent oracles: they share helpers +// and constants with the incremental state (`hrvTime` is what the state +// delegates to at the refusal boundary, `kNnDiffAcf1Floor`, the Banister +// weight, the energy pricer, ...), so a fault in a shared helper passes +// this law. What it pins is that the streamed path agrees with the batch +// path; the shared helpers have their own tests. Only `RunningMoments` +// and `IntHistogram` are checked against references written here. +// L2 checkpoint round trip: `toJson` -> real JSON text -> `fromJson` -> +// `toJson` is the same text, and the restored state goes on exactly as +// the live one does (same outputs, same text). +// L3 a checkpoint of another type or version, a missing or mistyped part, +// parts that disagree (counts against the data they count), hostile +// sizes: refused whole with a FormatException, never another error type. +// A change the reader cannot tell from a legal checkpoint (one sum +// altered) may be accepted, but then the state holds exactly what was +// written; an unknown key is ignored, nothing else moves. +// L4 identity: syncing the series it already holds changes nothing (outputs +// and checkpoint), an empty first sync leaves a state that takes the +// series as a fresh one would, and the other way round the parameters +// belong to the CALL: handing other parameters rebuilds the state as a +// fresh sync of them would (there is nothing to refuse), a refused call +// (misaligned arguments, duplicate keys) leaves the state unchanged. +// L5 conservation: every beat / sample / minute is counted exactly once in +// the sums a state keeps (pairs, bins, trigonometric sums, bills), +// checked against counts made here from the INPUT, never from the code. +// M merge (`RunningMoments`, `IntHistogram`): merging the summaries of two +// parts is the summary of the concatenation (exact for the histogram, +// within float tolerance for the moments), associative, commutative, +// with the empty summary as identity -- the nearest thing to a monoid +// law the analytics side has. +// +// The laws describe the code at analytics 0fc57682. A failing law is first +// checked against the module's contract; only a violation of the intended +// contract is a bug, and a wrong expectation is fixed here, never in lib/. +// +// Replay a failure with the command in its report, e.g. +// PROPERTY_SEED= PROPERTY_CASE= TZ=UTC dart test \ +// test/properties/incremental_laws_test.dart --plain-name '' + +import 'dart:convert'; +import 'dart:math' as math; +import 'dart:typed_data'; + +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +import '../onehz/support/energy_day.dart'; +import '../onehz/support/incremental_compare.dart'; +import '../support/fold_law_support.dart'; +import '../support/law_registry.dart'; +import '../support/property.dart'; + +final _laws = LawSet(); + +typedef Json = Map; + +// ── the shape every sync module shares ────────────────────────────────────── + +/// One module under the sync laws. [In] is a whole series PLUS the parameters +/// of the call; everything else is how to drive it and how to look at it. +abstract class Ops { + String get name; + + /// The series of a recipe (a handful of ints; cached by the caller). + In expand(List recipe); + int length(In input); + + /// The first [k] items with the same parameters. + In prefix(In input, int k); + + /// A different series with the same parameters (a replaced item, a dropped + /// prefix, a shrunk tail, ...), chosen by [kind]. + In edit(In input, int kind, int salt); + + /// The same series under other parameters ([alt] picks), or null when the + /// module has none (the Lomb grid belongs to the constructor). + In? reconfigure(In input, int alt); + + /// A new state for (the parameters of) [input]. + dynamic fresh(In input); + dynamic fromJson(Json j); + Json toJson(dynamic state); + Object? sync(dynamic state, In input, {bool force = false}); + + /// The output as exact text. + String show(Object? out); + + /// Fails unless [out] is what the batch gives for [input]. + void close(Object? out, In input); + + /// Fails unless two outputs of the same series are the same. Exact by + /// default; a module that rebuilds a running sum on restore (the HRV bins) + /// says "within the batch tolerance" instead. + void sameOutput(Object? a, Object? b, String why) => + expect(show(a), show(b), reason: why); + + /// Keys of the checkpoint that count work, not state. + List get counters; + + /// The checkpoint as text, without the counters. + String norm(Json j) { + final c = Map.of(j); + for (final k in counters) { + c.remove(k); + } + return jsonEncode(c); + } + + /// Fails unless [got], the checkpoint after a detour (series replaced, items + /// dropped, a forced rebuild, other parameters) and the same series again, is + /// the checkpoint [want] of one sync. Exact text by default; a module whose + /// checkpoint is equal in MEANING but not in text after a detour (it is + /// keyed, and the order its entries were added in is not state) projects + /// first. + void sameState(Json got, Json want, String why) => + expect(norm(got), norm(want), reason: why); + + /// Hand-written invariants of a reachable state that, broken, must be + /// refused. + List get refuseMutations; + + /// Changes the reader cannot tell from a legal checkpoint. + List get faithfulMutations; + + /// Calls that must be refused (ArgumentError) and leave the state unchanged; + /// `applies` says whether the input has something to get wrong. + List<({String name, bool Function(In) applies, void Function(dynamic, In) call})> + get refusals => const []; + + /// Counts made from the INPUT that the checkpoint must agree with. + void conserved(Json checkpoint, In input, String why); + + /// Positions where the series changes character (window and bin edges). + List get special; + + Gen> recipeGen(int maxN); + List> get forced; + void observe(List recipe, void Function(String) bump); + Map get shares; +} + +String _text(Json j) => jsonEncode(j); + +dynamic _restore(Ops ops, dynamic st) => + ops.fromJson((jsonDecode(jsonEncode(ops.toJson(st))) as Map).cast()); + +/// (splits, cut seed, restart mask, detour). +typedef Plan = (int, int, int, int); + +final Gen _planGen = G.quad(G.intIn(0, 6), G.intIn(0, 1 << 20), + G.intIn(0, 127), G.elements(const [0, 0, 1, 2, 3, 4, 5])); + +typedef Case = (List, Plan); + +Gen _caseGen(Ops ops, int maxN) => G.pair(ops.recipeGen(maxN), _planGen); + +String _tagOf(Ops ops, Case c) => '${ops.name} ${c.$1} plan=${c.$2}'; + +final Map _expandCache = {}; + +In _expand(Ops ops, List recipe) { + final key = '${ops.name}|$recipe'; + if (_expandCache.length > 500) _expandCache.clear(); + return _expandCache.putIfAbsent(key, () => ops.expand(recipe)) as In; +} + +void _observeCase(Ops ops, Case c, void Function(String) bump) { + ops.observe(c.$1, bump); + final n = ops.length(_expand(ops, c.$1)); + final (splits, cutSeed, restart, detour) = c.$2; + final b = foldBounds(n, splits, cutSeed, special: ops.special); + if (b.length >= 4) bump('chunks: three or more'); + if (hasEmptyChunk(b)) bump('an empty prefix'); + if (restart != 0 && b.length > 2) bump('a restore between syncs'); + if (detour != 0) bump('detour: $detour'); +} + +Map _lawShares(Ops ops, {Set drop = const {}}) => { + ...ops.shares, + 'chunks: three or more': .4, + 'an empty prefix': .04, + 'a restore between syncs': .3, + for (var k = 1; k <= 5; k++) 'detour: $k': .02, + }..removeWhere((k, _) => drop.contains(k)); + +// ── the laws, written once for every module ───────────────────────────────── + +void _l1(Ops ops, Case c) { + final x = _expand(ops, c.$1); + final n = ops.length(x); + final (splits, cutSeed, restart, detour) = c.$2; + final tag = _tagOf(ops, c); + final single = ops.fresh(x); + final out1 = ops.sync(single, x); + final want = ops.toJson(single); + // Growing prefixes (the last one is the whole series), a save/restore after + // some of them, and a detour before the whole series. + var st = ops.fresh(x); + final bounds = foldBounds(n, splits, cutSeed, special: ops.special); + for (var k = 1; k + 1 < bounds.length; k++) { + ops.sync(st, ops.prefix(x, bounds[k])); + if ((restart >> (k % 7)) & 1 == 1) st = _restore(ops, st); + } + if (detour != 0) { + switch (detour) { + case 1: + ops.sync(st, ops.edit(x, 0, cutSeed)); // one item replaced + case 2: + ops.sync(st, ops.edit(x, 1, cutSeed)); // the first items dropped + case 3: + ops.sync(st, x, force: true); // a forced rebuild + case 4: + final other = ops.reconfigure(x, cutSeed); + if (other != null) ops.sync(st, other); // other parameters + default: + ops.sync(st, ops.edit(x, 2, cutSeed)); // the tail dropped + } + st = _restore(ops, st); + } + final out2 = ops.sync(st, x); + ops.sameState(ops.toJson(st), want, '$tag bounds=$bounds: the checkpoint is the same'); + ops.sameOutput(out2, out1, '$tag: the output is the same'); +} + +void _l1b(Ops ops, Case c) { + final x = _expand(ops, c.$1); + final n = ops.length(x); + final (splits, cutSeed, restart, detour) = c.$2; + var st = ops.fresh(x); + final bounds = foldBounds(n, splits, cutSeed, special: ops.special); + final picks = pickSeams(bounds.length - 1); + for (var k = 1; k < bounds.length; k++) { + final part = ops.prefix(x, bounds[k]); + final out = ops.sync(st, part); + if (picks.contains(k - 1) || k == bounds.length - 1) { + ops.close(out, part); + } + if ((restart >> (k % 7)) & 1 == 1) st = _restore(ops, st); + if (k == bounds.length - 2 && detour != 0) { + // Whatever the history, the output is the batch's for the series now. + final other = detour == 4 + ? ops.reconfigure(x, cutSeed) ?? ops.edit(x, 0, cutSeed) + : ops.edit(x, detour % 3, cutSeed); + ops.close(ops.sync(st, other), other); + } + } +} + +void _l2(Ops ops, Case c) { + final x = _expand(ops, c.$1); + final n = ops.length(x); + final (splits, cutSeed, _, _) = c.$2; + final tag = _tagOf(ops, c); + final bounds = foldBounds(n, splits, cutSeed, special: ops.special); + final at = bounds.length > 2 ? bounds[1] : n ~/ 2; + final live = ops.fresh(x); + ops.sync(live, ops.prefix(x, at)); + final text = _text(ops.toJson(live)); + final map = (jsonDecode(text) as Map).cast(); + final back = ops.fromJson(map); + expect(_text(ops.toJson(back)), text, reason: '$tag: write(read(b)) == b'); + // Not aliased to the map it came from. + for (final k in map.keys.toList()) { + final v = map[k]; + if (v is List) v.clear(); + } + expect(_text(ops.toJson(back)), text, reason: '$tag: not aliased to the map'); + final restored = ops.fromJson((jsonDecode(text) as Map).cast()); + // And it goes on identically: the rest of the series, then a re-sync. + final a = ops.sync(live, x), b = ops.sync(restored, x); + ops.sameOutput(b, a, '$tag: same output after the rest'); + expect(_text(ops.toJson(restored)), _text(ops.toJson(live)), + reason: '$tag: same checkpoint after the rest (counters included)'); + ops.sameOutput(ops.sync(restored, x), a, '$tag: twice'); +} + +/// The mutated checkpoint, what it may do, and (when that is not simply "the +/// text of the mutated map") the texts a faithful reading may give. +(Json, Must, List?) _mutate(Ops ops, Json src, Mut m) { + final j = (deepCopy(src) as Map).cast(); + final (kind, a, b) = m; + final keys = j.keys.toList(); + Must refuseUnlessNoop(Json bad) => + jsonEncode(bad) == jsonEncode(src) ? Must.faithful : Must.refuse; + switch (kind % 7) { + case 0: + const versions = [0, 2, 3, -1, 99, '1', null, true, 1 << 40]; + j['version'] = versions[a % versions.length]; + return (j, Must.refuse, null); + case 1: + const types = ['RunningMoments', 'Other', null, '', 1, 'incrementalhrvtime']; + j['type'] = types[a % types.length]; + return (j, Must.refuse, null); + case 2: + // A key gone: refused, or (the readers take an absent key for null where + // a part is nullable, and rebuild a total that older checkpoints did not + // carry) read back as null, or as it was. + final k = keys[a % keys.length]; + j.remove(k); + final asNull = (deepCopy(src) as Map).cast()..[k] = null; + return ( + j, + k == 'version' || k == 'type' ? Must.refuse : Must.faithful, + [_text(asNull), _text(src)] + ); + case 3: + final k = keys[a % keys.length]; + const junk = ['x', {}, [[]]]; + j[k] = junk[b % junk.length]; + return (j, Must.refuse, null); + case 4: + if (ops.refuseMutations.isEmpty) return (j, Must.faithful, null); + ops.refuseMutations[a % ops.refuseMutations.length](j); + return (j, refuseUnlessNoop(j), null); + case 5: + if (ops.faithfulMutations.isEmpty) return (j, Must.faithful, null); + ops.faithfulMutations[a % ops.faithfulMutations.length](j); + return (j, Must.faithful, null); + default: + j['futureField${a % 3}'] = const [1, 'x', [1]][b % 3]; + return (j, Must.ignored, null); + } +} + +String _outcome(Ops ops, Json j) { + try { + final s = ops.fromJson((deepCopy(j) as Map).cast()); + return 'accepted:${_text(ops.toJson(s))}'; + } on FormatException { + return 'refused'; + } catch (e) { + return 'WRONG ERROR TYPE: ${e.runtimeType}: $e'; + } +} + +void _l3(Ops ops, (Case, Mut) arg) { + final (c, m) = arg; + final tag = '${_tagOf(ops, c)} mutation=${m.$1 % 7}(${m.$2},${m.$3})'; + final x = _expand(ops, c.$1); + final st = ops.fresh(x); + ops.sync(st, x); + final src = (jsonDecode(_text(ops.toJson(st))) as Map).cast(); + final (bad, must, expected) = _mutate(ops, src, m); + final outcome = _outcome(ops, bad); + if (expected == null) { + expectMutationOutcome(outcome, must, _text(bad), _text(src), tag); + } else { + expect(outcome, isNot(startsWith('WRONG')), reason: '$tag: only FormatException'); + if (must == Must.refuse) { + expect(outcome, 'refused', reason: '$tag: refused whole'); + } else if (outcome != 'refused') { + expect(expected.map((t) => 'accepted:$t'), contains(outcome), + reason: '$tag: accepted means a faithful reading of what was written'); + } + } +} + +void _l4(Ops ops, Case c) { + final x = _expand(ops, c.$1); + final tag = _tagOf(ops, c); + final fresh = ops.fresh(x); + final out = ops.sync(fresh, x); + final text = ops.norm(ops.toJson(fresh)); + // Syncing what it already holds: nothing moves. + expect(ops.show(ops.sync(fresh, x)), ops.show(out), reason: '$tag: same output'); + expect(ops.norm(ops.toJson(fresh)), text, reason: '$tag: sync of the same is the identity'); + // An empty first sync, then the series: as if it were a fresh state. + final late = ops.fresh(x); + ops.sync(late, ops.prefix(x, 0)); + ops.sync(late, x); + expect(ops.norm(ops.toJson(late)), text, + reason: '$tag: an empty first sync leaves a state that takes the series afresh'); + // The converse: the parameters belong to the call. Other parameters rebuild + // the state as a fresh sync of them would (nothing is refused, even for a + // state restored from a checkpoint written under the first), and handing the + // first ones back rebuilds the first. + final other = ops.reconfigure(x, c.$2.$2); + if (other != null) { + var s = ops.fresh(x); + ops.sync(s, x); + s = _restore(ops, s); + ops.sync(s, other); + final f = ops.fresh(other); + ops.sync(f, other); + expect(ops.norm(ops.toJson(s)), ops.norm(ops.toJson(f)), + reason: '$tag: other parameters rebuild, as a fresh sync of them'); + ops.sync(s, x); + expect(ops.norm(ops.toJson(s)), text, reason: '$tag: and back'); + } + // A refused call leaves the checkpoint as it was. + final before = _text(ops.toJson(fresh)); + for (final r in ops.refusals) { + if (!r.applies(x)) continue; + expect(() => r.call(fresh, x), throwsArgumentError, reason: '$tag: ${r.name}'); + expect(_text(ops.toJson(fresh)), before, + reason: '$tag: ${r.name} leaves the state unchanged'); + } +} + +void _l5(Ops ops, Case c) { + final x = _expand(ops, c.$1); + final n = ops.length(x); + final (splits, cutSeed, restart, _) = c.$2; + final tag = _tagOf(ops, c); + var st = ops.fresh(x); + final bounds = foldBounds(n, splits, cutSeed, special: ops.special); + for (var k = 1; k < bounds.length; k++) { + final part = ops.prefix(x, bounds[k]); + ops.sync(st, part); + final json = (jsonDecode(_text(ops.toJson(st))) as Map).cast(); + ops.conserved(json, part, '$tag at ${bounds[k]}'); + // The reader takes back what the writer wrote. + st = (restart >> (k % 7)) & 1 == 1 ? _restore(ops, st) : st; + } +} + +/// Registers the five laws of [ops]. +void _registerSyncLaws(Ops ops, + {int maxN = 400, + int casesL1 = 120, + int casesL1b = 60, + Set dropL1b = const {}}) { + group(ops.name, () { + final reachFor = (Set drop) => Reach( + _lawShares(ops, drop: drop), (c, bump) => _observeCase(ops, c, bump)); + _laws.law( + 'L1 ${ops.name}: any chunking of the syncs, empty prefixes, restores and ' + 'detours included, gives the checkpoint text and output of one sync', + _caseGen(ops, maxN), + (c) => _l1(ops, c), + examples: [for (final r in ops.forced) (r, (3, 5 + r.length, 85, r.length % 6))], + cases: casesL1, + reach: reachFor(const {}), + ); + _laws.law( + 'L1b ${ops.name}: after every checked sync the output is the batch ' + "function's, history notwithstanding", + _caseGen(ops, maxN), + (c) => _l1b(ops, c), + examples: [for (final r in ops.forced) (r, (4, 9 + r.length, 127, (r.length + 1) % 6))], + cases: casesL1b, + reach: reachFor(dropL1b), + ); + _laws.law( + 'L2 ${ops.name}: write(read(b)) == b through JSON text, and the restored ' + 'state goes on identically', + _caseGen(ops, maxN), + (c) => _l2(ops, c), + examples: [for (final r in ops.forced) (r, (2, 3 + r.length, 0, 0))], + cases: 100, + reach: reachFor(const {'detour: 1', 'detour: 2', 'detour: 3', 'detour: 4', 'detour: 5'}), + ); + _laws.law<(Case, Mut)>( + 'L3 ${ops.name}: a mutated checkpoint is refused whole or holds exactly ' + 'what was written', + G.pair(_caseGen(ops, 200), mutGen(7)), + (arg) => _l3(ops, arg), + examples: [ + for (var i = 0; i < ops.forced.length; i++) + for (var k = 0; k < 7; k++) + ((ops.forced[i], (3, 5, 0, 0)), (k, 3 * i + k, 5 * i + 2 * k)), + for (var a = 0; a < ops.refuseMutations.length; a++) + for (var i = 0; i < ops.forced.length; i++) + ((ops.forced[i], (3, 5, 0, 0)), (4, a, i)), + for (var a = 0; a < ops.faithfulMutations.length; a++) + for (var i = 0; i < ops.forced.length; i++) + ((ops.forced[i], (3, 5, 0, 0)), (5, a, i)), + for (var a = 0; a < 9; a++) ((ops.forced.first, (3, 5, 0, 0)), (0, a, 0)), + ], + cases: 100, + reach: Reach<(Case, Mut)>({ + for (var k = 0; k < 7; k++) 'kind: $k': .03, + }, (arg, bump) => bump('kind: ${arg.$2.$1 % 7}')), + ); + _laws.law( + 'L4 ${ops.name}: syncing what it holds is the identity, an empty first ' + 'sync records nothing that matters, the parameters belong to the call, a ' + 'refused call changes nothing', + _caseGen(ops, maxN), + (c) => _l4(ops, c), + examples: [for (final r in ops.forced) (r, (3, 7 + r.length, 0, 0))], + cases: 100, + reach: reachFor(const {'detour: 1', 'detour: 2', 'detour: 3', 'detour: 4', 'detour: 5'}), + ); + _laws.law( + 'L5 ${ops.name}: every item is counted exactly once in the sums the state ' + 'keeps (counts made from the input)', + _caseGen(ops, maxN), + (c) => _l5(ops, c), + examples: [for (final r in ops.forced) (r, (4, 11 + r.length, 127, 0))], + cases: 100, + reach: reachFor(const {'detour: 1', 'detour: 2', 'detour: 3', 'detour: 4', 'detour: 5'}), + ); + }); +} + +// ═════════════════════════════════════════════════════════════════════════════ +// HRV +// ═════════════════════════════════════════════════════════════════════════════ + +const _hrvFlavours = [ + 'smooth', // two slow sines: a high lag-1 ACF of the differences is not expected + 'white jitter', // differenced white noise: the jitter gate refuses RMSSD / pNN50 + 'af', // irregularly irregular + 'constant', // every difference zero: variance zero + 'gaps', // a 21 s dropout every 113 beats + 'big steps', // every 7th beat 90 ms off: pNN50 moves + 'two valued', // alternating 800 / 1000: ACF of the differences is -1 + 'seconds grid', // 1 s beats on a whole-second grid: the 5-minute bin edges are hit exactly + 'gap edge', // the time step exceeds the beat by exactly, under and over the half millisecond + 'fifty', // successive differences of exactly, under and over 50 ms + // ~44 bpm, breathing at 18 br/min (0.3 Hz), a drifting heart rate: the lag-1 + // ACF of the differences is far under the jitter floor, yet the spectrum + // shows a respiratory line on a low floor, so the batch publishes RMSSD + // (upstream's noise-share exemption). + 'rsa slow heart', +]; +const _hrvOrigins = [0.0, 299999.5, 1700000010123.125, 1700000000000.0]; +const _hrvAf = [0.0, .17, .8, 1.0]; + +class HrvIn { + HrvIn(this.nn, this.t, this.af); + final List nn; + final List? t; + final double af; +} + +class HrvOps extends Ops { + @override + String get name => 'IncrementalHrvTime'; + + @override + HrvIn expand(List r) { + final (flavour, n, seed, timed, originIdx, afIdx) = (r[0], r[1], r[2], r[3], r[4], r[5]); + final g = Rng(seed * 7919 + flavour + 5); + final nn = [], t = []; + var clock = _hrvOrigins[originIdx]; + var fiftyBase = 800.0; + for (var i = 0; i < n; i++) { + final v = switch (flavour) { + 1 => 800 + 160 * (g.nextDouble() - .5), + 2 => 420 + g.nextInt(700).toDouble(), + 3 => 800.0, + 6 => i.isEven ? 800.0 : 1000.0, + 7 => 900.0 + (i % 11) * 7, + 9 => fiftyBase += const [50.0, -50.0, 49.0, -49.0, 51.0, -51.0][i % 6], + 10 => 1350 + + 80 * math.sin(2 * math.pi * (clock - _hrvOrigins[originIdx]) / 3.6e6) + + 40 * math.sin(2 * math.pi * .3 * (clock - _hrvOrigins[originIdx]) / 1000) + + 3 * (g.nextDouble() - .5), + 5 => 800 + 40 * math.sin(i * .13) + (i % 7 == 6 ? 90 : 0), + _ => 800 + 75 * math.sin(i * .13 + seed * .003) + 21 * math.sin(i * .031 + seed * .007), + }; + if (flavour == 4 && i % 113 == 112) clock += 21000; + clock += flavour == 7 ? 1000 : v; + if (flavour == 8 && i % 9 == 8) clock += const [.5, .25, .75, .5, .4999, .5001][(i ~/ 9) % 6]; + nn.add(v); + t.add(clock); + } + return HrvIn(nn, timed == 0 ? t : null, _hrvAf[afIdx]); + } + + @override + int length(HrvIn i) => i.nn.length; + @override + HrvIn prefix(HrvIn i, int k) => + HrvIn(i.nn.sublist(0, k), i.t?.sublist(0, k), i.af); + @override + HrvIn edit(HrvIn i, int kind, int salt) { + final n = i.nn.length; + if (n < 2) return i; + switch (kind % 3) { + case 0: + final nn = [...i.nn]; + nn[salt % n] += 60; + return HrvIn(nn, i.t, i.af); + case 1: + final d = 1 + salt % (n - 1); + return HrvIn(i.nn.sublist(d), i.t?.sublist(d), i.af); + default: + final d = 1 + salt % (n - 1); + return HrvIn(i.nn.sublist(0, n - d), i.t?.sublist(0, n - d), i.af); + } + } + + @override + HrvIn? reconfigure(HrvIn i, int alt) => HrvIn(i.nn, i.t, _hrvAf[(alt + 1) % 4] == i.af ? .5 : _hrvAf[(alt + 1) % 4]); + + @override + dynamic fresh(HrvIn input) => IncrementalHrvTime(); + @override + dynamic fromJson(Json j) => IncrementalHrvTime.fromJson(j); + @override + Json toJson(dynamic s) => (s as IncrementalHrvTime).toJson(); + @override + Object? sync(dynamic s, HrvIn i, {bool force = false}) => (s as IncrementalHrvTime) + .sync(i.nn, nnTimesMs: i.t, artifactFraction: i.af, force: force); + + @override + String show(Object? out) { + final m = out as Metric; + final v = m.value; + return [ + m.present, + m.confidence, + m.tier, + m.note, + m.inputs_used, + if (v != null) [v.nBeats, v.rmssd, v.sdnn, v.sdann, v.sdnnIndex, v.pnn50, v.diffAcf1], + ].join('|'); + } + + @override + void sameOutput(Object? a, Object? b, String why) { + // Restoring rebuilds the running sums of the bin means and SDs from the + // stored bins (they are not in the checkpoint), so a restored state's + // SDANN / SDNN index agree with one that never stopped to ~1e-14 relative, + // not bit for bit. + final x = a as Metric, y = b as Metric; + try { + hrvClose(x, y); + } on TestFailure catch (e) { + fail('$why: ${e.message}'); + } + } + + @override + void close(Object? out, HrvIn i) => + hrvClose(out as Metric, hrvTime(i.nn, nnTimesMs: i.t, artifactFraction: i.af)); + + @override + List get counters => const ['processedPoints']; + + @override + List get special => const [0, 1, 2, 29, 30, 31, 299, 300, 301, 302, 599, 600, 601]; + + @override + List get refuseMutations => [ + (j) { + final n = (j['nn'] as List).length; + if (n > 0) j['processedPoints'] = n - 1; + }, + (j) => j['over50'] = (j['pairs'] as int) + 1, + (j) => j['lagPairs'] = (j['pairs'] as int) + 1, + (j) => j['pairs'] = math.max(0, (j['nn'] as List).length - 1) + 1, + (j) => (j['levels'] as Map)['count'] = ((j['levels'] as Map)['count'] as int) + 1, + (j) { + final bins = j['bins'] as List; + if (bins.isNotEmpty) (bins.first as Map)['count'] = ((bins.first as Map)['count'] as int) + 1; + }, + (j) => (j['nn'] as List).add(800.0), + (j) { + final t = j['times'] as List?; + if (t != null && t.isNotEmpty) t.removeLast(); + }, + (j) { + if (((j['levels'] as Map)['count'] as int) > 0) j['finite'] = false; + }, + (j) { + final nn = j['nn'] as List; + if (nn.isNotEmpty && j['finite'] == true) nn[0] = 'NaN'; + }, + (j) => (j['bins'] as List).add({ + 'version': 1, + 'type': 'RunningMoments', + 'count': 0, + 'origin': 0.0, + 'meanOffset': 0.0, + 'm2': 0.0 + }), + (j) => j['diffSquares'] = -1.0, + ]; + + @override + List get faithfulMutations => [ + (j) { + final nn = j['nn'] as List; + if (nn.isNotEmpty) nn[0] = (nn[0] as num) + 1.0; + }, + (j) => j['diffSum'] = (j['diffSum'] as num) + 1.0, + (j) => j['products'] = (j['products'] as num) + 1.0, + (j) => j['endpoints'] = (j['endpoints'] as num) + 1.0, + (j) => j['lastDiff'] = ((j['lastDiff'] as num?) ?? 0) + 1.0, + (j) => j['binIdx'] = (j['binIdx'] as int) + 1, + (j) => j['processedPoints'] = (j['processedPoints'] as int) + 5, + (j) { + final t = j['times'] as List?; + if (t != null && t.isNotEmpty) t[0] = (t[0] as num) + 1.0; + }, + ]; + + @override + void conserved(Json cp, HrvIn i, String why) { + final n = i.nn.length; + expect((cp['nn'] as List).length, n, reason: 'the series is kept $why'); + expect((cp['levels'] as Map)['count'], n, reason: 'every beat is a level, once $why'); + final timed = i.t != null; + // Pairs made here: neighbours, except across a dropout (a time step longer + // than the beat itself, plus half a millisecond). + var pairs = 0, over = 0, lag = 0; + double? last; + for (var k = 1; k < n; k++) { + if (timed && i.t![k] - i.t![k - 1] > i.nn[k] + .5) { + last = null; + continue; + } + final d = i.nn[k] - i.nn[k - 1]; + pairs++; + if (d.abs() > 50) over++; + if (last != null) lag++; + last = d; + } + expect(cp['pairs'], pairs, reason: 'successive differences $why'); + expect(cp['over50'], over, reason: 'differences over 50 ms $why'); + expect(cp['lagPairs'], lag, reason: 'lag-1 pairs $why'); + final bins = cp['bins'] as List; + if (!timed) { + expect(bins, isEmpty, reason: 'no times, no bins $why'); + } else { + // Every beat is in exactly one five-minute bin; bins are runs of beats + // with the same index since the first beat. + var total = 0; + for (final b in bins) { + total += (b as Map)['count'] as int; + } + expect(total, n, reason: 'every beat is in one bin $why'); + var runs = 0; + int? prev; + for (var k = 0; k < n; k++) { + final idx = ((i.t![k] - i.t!.first) / 300000).floor(); + if (prev == null || idx != prev) runs++; + prev = idx; + } + expect(bins.length, runs, reason: 'bins are runs of the same five minutes $why'); + } + } + + @override + Gen> recipeGen(int maxN) => IntsGen([ + G.intIn(0, _hrvFlavours.length - 1), + SizeGen(maxN, pool: const [0, 1, 2, 3, 29, 30, 31, 32, 299, 300, 301, 302, 599, 600, 601, 610]), + G.intIn(0, 1 << 12), + G.elements(const [0, 0, 0, 1]), + G.intIn(0, _hrvOrigins.length - 1), + G.intIn(0, _hrvAf.length - 1), + ]); + + @override + List> get forced => [ + [0, 0, 1, 0, 0, 0], // empty + [0, 1, 1, 0, 0, 0], // one beat + [0, 2, 1, 0, 0, 0], // two + [3, 31, 1, 0, 0, 0], // constant, 30 pairs + [3, 29, 1, 1, 0, 0], // constant, no times + [1, 300, 2, 0, 0, 1], // jitter: RMSSD refused + [6, 120, 1, 0, 0, 0], // ACF of the differences exactly -1 + [0, 31, 1, 0, 0, 0], // exactly 30 pairs: the first with an ACF + [0, 32, 2, 1, 0, 0], + [7, 450, 1, 0, 0, 0], // exactly two 5-minute bins + [8, 120, 1, 0, 0, 0], // dropout edges + [9, 120, 1, 0, 0, 0], // differences of exactly 50 ms + [9, 120, 2, 1, 3, 1], + [7, 610, 1, 0, 0, 0], // whole-second grid across 5-minute edges + [7, 610, 1, 0, 1, 2], // shifted origin + [7, 610, 1, 0, 2, 3], + [4, 400, 3, 0, 0, 0], // dropouts + [5, 200, 1, 0, 3, 0], // big steps + [2, 250, 5, 1, 0, 1], // AF, no times + [0, 500, 7, 0, 3, 2], + [10, 900, 1, 0, 0, 0], // RSA at a slow heart: exempt from the jitter gate + [10, 900, 2, 1, 0, 1], // the same without times + [10, 900, 3, 0, 2, 0], + ]; + + @override + void observe(List r, void Function(String) bump) { + bump('flavour: ${_hrvFlavours[r[0]]}'); + final n = r[1]; + if (n == 0) bump('series: empty'); + if (n >= 1 && n < 30) bump('series: under 30 pairs'); + if (n >= 300) bump('series: past one 5-minute bin'); + if (r[3] == 1) bump('no times'); + if (r[5] != 0) bump('artifact fraction set'); + } + + @override + Map get shares => { + for (var f = 0; f < _hrvFlavours.length; f++) 'flavour: ${_hrvFlavours[f]}': .03, + 'series: empty': .01, + 'series: under 30 pairs': .05, + 'series: past one 5-minute bin': .15, + 'no times': .08, + 'artifact fraction set': .3, + }; +} + +// ═════════════════════════════════════════════════════════════════════════════ +// ENMO +// ═════════════════════════════════════════════════════════════════════════════ + +const _enmoFlavours = [ + 'regular', // a sample every 1/hz s, a still vector with noise + 'gaps + invalid', // a hole of 69 samples, every 47th sample invalid + 'duplicate times', // two samples per timestamp + 'unsorted', // the stream arrives newest first + 'constant vector', +]; +const _hz = [.5, 1.0, 2.0]; +const _enmoOrigins = [0.0, 1700000010000.0, 59000.0, 1700000040000.5]; + +/// (gRef, gravityWindowS, minSamplesPerMinute, expectedMinutes). A null gRef is +/// the auto-calibrated path (the batch every time); a NaN gRef the fallback. +const List<(double?, double, int, int?)> _enmoCfgs = [ + (1.013456789, defaultGravityWindowS, 30, 12), + (1.0, defaultGravityWindowS, 30, null), + (null, defaultGravityWindowS, 30, null), + (1.05, 31, 60, 100), + (1.0, 0.0, 1, null), + (1.0, 1.0, 60, null), + (1.0, 61.5, 30, 12), + (double.nan, defaultGravityWindowS, 30, null), + (double.infinity, defaultGravityWindowS, 30, null), +]; + +class EnmoIn { + EnmoIn(this.samples, this.cfg); + final List samples; + final int cfg; + (double?, double, int, int?) get c => _enmoCfgs[cfg]; +} + +class EnmoOps extends Ops { + @override + String get name => 'IncrementalEnmoSeries'; + + @override + EnmoIn expand(List r) { + final (n, hzIdx, seed, flavour, cfg, originIdx) = (r[0], r[1], r[2], r[3], r[4], r[5]); + final g = Rng(seed * 104729 + flavour + 7); + final hz = _hz[hzIdx]; + final out = []; + for (var i = 0; i < n; i++) { + if (flavour == 1 && i >= 70 && i < 139) continue; + final idx = flavour == 2 ? i ~/ 2 : i; + final ts = _enmoOrigins[originIdx] + idx * 1000 / hz; + if (flavour == 4) { + out.add(AccelSample(ts, .3, -.4, 1.0)); + } else { + out.add(AccelSample( + ts, + .22 * math.sin(i * .29) + g.nextDouble() * .007, + .11 * math.cos(i * .07), + 1.02 + .065 * math.sin(i * .23), + valid: !(flavour == 1 && i % 47 == 46))); + } + } + return EnmoIn(flavour == 3 ? out.reversed.toList() : out, cfg); + } + + @override + int length(EnmoIn i) => i.samples.length; + @override + EnmoIn prefix(EnmoIn i, int k) => EnmoIn(i.samples.sublist(0, k), i.cfg); + @override + EnmoIn edit(EnmoIn i, int kind, int salt) { + final n = i.samples.length; + if (n < 2) return i; + switch (kind % 3) { + case 0: + final a = [...i.samples]; + final o = a[salt % n]; + a[salt % n] = AccelSample(o.tsMs, o.x + .07, o.y, o.z, valid: o.valid); + return EnmoIn(a, i.cfg); + case 1: + return EnmoIn(i.samples.sublist(1 + salt % (n - 1)), i.cfg); + default: + return EnmoIn(i.samples.sublist(0, n - 1 - salt % (n - 1)), i.cfg); + } + } + + @override + EnmoIn? reconfigure(EnmoIn i, int alt) => + EnmoIn(i.samples, (i.cfg + 1 + alt % (_enmoCfgs.length - 1)) % _enmoCfgs.length); + + @override + dynamic fresh(EnmoIn input) => IncrementalEnmoSeries(); + @override + dynamic fromJson(Json j) => IncrementalEnmoSeries.fromJson(j); + @override + Json toJson(dynamic s) => (s as IncrementalEnmoSeries).toJson(); + @override + Object? sync(dynamic s, EnmoIn i, {bool force = false}) => + (s as IncrementalEnmoSeries).sync(i.samples, + gRef: i.c.$1, + gravityWindowS: i.c.$2, + minSamplesPerMinute: i.c.$3, + expectedMinutes: i.c.$4, + force: force); + + @override + String show(Object? out) { + final r = out as EnmoResult; + return [ + r.gRef, + r.coverage, + for (final m in r.minutes) + [m.tsMinStartMs, m.nSamples, m.enmo, m.mad, m.meanMag, m.dynAmp], + ].join('|'); + } + + @override + void close(Object? out, EnmoIn i) => enmoClose( + out as EnmoResult, + enmoSeries(i.samples, + gRef: i.c.$1, + gravityWindowS: i.c.$2, + minSamplesPerMinute: i.c.$3, + expectedMinutes: i.c.$4)); + + @override + List get counters => const ['processedPoints']; + + @override + List get special => const [0, 1, 2, 29, 30, 31, 59, 60, 61, 69, 70, 120, 121]; + + @override + List get refuseMutations => [ + (j) { + final n = (j['valid'] as List).length; + if (n > 0) j['processedPoints'] = n - 1; + }, + (j) => j['lo'] = (j['valid'] as List).length + 1, + (j) { + final bins = j['bins'] as List; + if (bins.isNotEmpty && ((bins.first as Map)['mags'] as List).isNotEmpty) { + ((bins.first as Map)['mags'] as List).removeLast(); + } + }, + (j) => (j['valid'] as List).add([1.0, 0.0, 0.0, 1.0]), + (j) { + if ((j['valid'] as List).isNotEmpty) j['gRef'] = null; + }, + (j) { + final v = j['valid'] as List; + if (v.length >= 2) (v[0] as List)[0] = ((v[1] as List)[0] as num) + 1.0; + }, + (j) { + final v = j['valid'] as List; + if (v.isNotEmpty) (v[0] as List).removeLast(); + }, + (j) => (j['bins'] as List).add({ + 'key': -1, + 'mags': [], + 'magSum': 0.0, + 'enmoSum': 0.0, + 'dynSum': 0.0 + }), + (j) { + final bins = j['bins'] as List; + if (bins.isNotEmpty) bins.add(deepCopy(bins.first)); + }, + (j) { + final bins = j['bins'] as List; + if (bins.isNotEmpty) (bins.first as Map)['enmoSum'] = -1.0; + }, + (j) => j['windowS'] = 'x', + ]; + + @override + List get faithfulMutations => [ + (j) => j['sx'] = (j['sx'] as num) + 1.0, + (j) => j['sy'] = (j['sy'] as num) + 1.0, + (j) => j['sz'] = (j['sz'] as num) + 1.0, + (j) => j['windowS'] = (j['windowS'] as num) + 1.0, + (j) => j['processedPoints'] = (j['processedPoints'] as int) + 5, + (j) { + final bins = j['bins'] as List; + if (bins.isNotEmpty) (bins.first as Map)['magSum'] = ((bins.first as Map)['magSum'] as num) + 1.0; + }, + (j) { + final bins = j['bins'] as List; + if (bins.isNotEmpty) (bins.first as Map)['dynSum'] = ((bins.first as Map)['dynSum'] as num) + 1.0; + }, + (j) { + final v = j['valid'] as List; + if (v.isNotEmpty) (v[0] as List)[1] = ((v[0] as List)[1] as num) + .01; + }, + ]; + + @override + void conserved(Json cp, EnmoIn i, String why) { + // Every valid sample is in exactly one minute bin (the minute its time falls + // in); a state with a non-finite reference or parameter keeps nothing. + final c = i.c; + final usable = c.$1 != null && c.$1!.isFinite && c.$2.isFinite; + final valid = i.samples.where((s) => s.valid).toList(); + if (!usable) { + expect((cp['valid'] as List), isEmpty, reason: 'nothing is kept $why'); + expect((cp['bins'] as List), isEmpty, reason: 'no bins $why'); + return; + } + expect((cp['valid'] as List).length, valid.length, reason: 'valid samples kept $why'); + final want = {}; + for (final s in valid) { + want.update((s.tsMs / 60000).floor(), (v) => v + 1, ifAbsent: () => 1); + } + final got = { + for (final b in cp['bins'] as List) + (b as Map)['key'] as int: (b['mags'] as List).length + }; + expect(got, want, reason: 'every valid sample is in the bin of its minute, once $why'); + } + + @override + Gen> recipeGen(int maxN) => IntsGen([ + SizeGen(maxN, pool: const [0, 1, 2, 3, 29, 30, 31, 59, 60, 61, 69, 70, 120, 121]), + G.intIn(0, _hz.length - 1), + G.intIn(0, 1 << 12), + G.intIn(0, _enmoFlavours.length - 1), + G.intIn(0, _enmoCfgs.length - 1), + G.intIn(0, _enmoOrigins.length - 1), + ]); + + @override + List> get forced => [ + [0, 1, 1, 0, 0, 0], // empty + [1, 1, 1, 0, 0, 0], // one sample + [31, 1, 1, 0, 1, 0], + [200, 1, 1, 1, 0, 1], // gaps + invalid + [300, 2, 1, 2, 3, 1], // duplicate timestamps + [150, 1, 1, 3, 1, 0], // newest first + [130, 1, 1, 4, 1, 1], // constant vector + [200, 0, 2, 0, 2, 1], // auto-calibrated + [200, 1, 2, 0, 7, 0], // NaN reference: the fallback + [200, 1, 2, 0, 8, 2], // infinite reference + [200, 1, 3, 0, 4, 2], // window of zero seconds + [240, 2, 4, 0, 6, 3], // wide window + [180, 1, 5, 0, 5, 2], + ]; + + @override + void observe(List r, void Function(String) bump) { + bump('flavour: ${_enmoFlavours[r[3]]}'); + final n = r[0]; + if (n == 0) bump('series: empty'); + if (n >= 1 && n < 30) bump('series: under a minute of samples'); + if (n >= 120) bump('series: two minutes or more'); + final c = _enmoCfgs[r[4]]; + if (c.$1 == null) bump('reference: auto'); + if (c.$1 != null && !c.$1!.isFinite) bump('reference: not finite'); + if (c.$1 != null && c.$1!.isFinite) bump('reference: given'); + if (c.$2 != defaultGravityWindowS) bump('window: not the default'); + } + + @override + Map get shares => { + for (final f in _enmoFlavours) 'flavour: $f': .03, + 'series: empty': .01, + 'series: under a minute of samples': .05, + 'series: two minutes or more': .2, + 'reference: auto': .05, + 'reference: not finite': .05, + 'reference: given': .4, + 'window: not the default': .2, + }; +} + +// ═════════════════════════════════════════════════════════════════════════════ +// Lomb-Scargle +// ═════════════════════════════════════════════════════════════════════════════ + +const _lombFlavours = [ + 'regular', // beats at ~0.8 s with slow modulation + 'irregular', // uneven gaps, dropouts + 'constant y', // nothing to analyse: abstains + 'collapsed t', // every time the same: abstains + 'duplicate t', // pairs of equal times + 'NaN y', // a non-finite value halfway: the batch decides + 'infinite t', // a non-finite time halfway + 'length mismatch', // one value fewer than times +]; +final _lombGrids = >[ + [0.0, .0033, .01, .04, .07, .1, .15, .25, .4], + [], + [0.0], + [.04, .04, .13], + [.1, .07, .4], + [for (var k = 0; k <= 40; k++) .0033 + k * .01], +]; +const _lombOrigins = [0.0, 12345.125, 1791028730.0]; + +class LombIn { + LombIn(this.t, this.y, this.grid); + final List t, y; + final int grid; + List get freqs => _lombGrids[grid]; +} + +class LombOps extends Ops { + @override + String get name => 'IncrementalLombScargle'; + + @override + LombIn expand(List r) { + final (n, seed, flavour, grid, originIdx) = (r[0], r[1], r[2], r[3], r[4]); + final g = Rng(seed * 6007 + flavour + 13); + var clock = _lombOrigins[originIdx]; + final t = [], y = []; + for (var i = 0; i < n; i++) { + final rr = 800 + 75 * math.sin(i * .13 + seed * .003) + 21 * math.sin(i * .031) + (flavour == 1 ? g.nextDouble() * 400 : 0); + if (flavour == 4) { + if (i.isEven) clock += rr / 1000; + } else if (flavour == 3) { + // time does not move + } else { + clock += rr / 1000; + } + if (flavour == 1 && i % 61 == 60) clock += 21; + t.add(clock); + y.add(flavour == 2 ? 800.0 : rr); + } + if (flavour == 5 && n > 4) y[n ~/ 2] = double.nan; + if (flavour == 6 && n > 4) t[n ~/ 2] = double.infinity; + if (flavour == 7 && n > 0) y.removeLast(); + return LombIn(t, y, grid); + } + + @override + int length(LombIn i) => i.t.length; + @override + LombIn prefix(LombIn i, int k) => + LombIn(i.t.sublist(0, k), i.y.sublist(0, math.min(k, i.y.length)), i.grid); + @override + LombIn edit(LombIn i, int kind, int salt) { + final n = i.t.length; + if (n < 2 || i.y.length != n) return i; + switch (kind % 3) { + case 0: + final y = [...i.y]; + y[salt % n] += 79; + return LombIn(i.t, y, i.grid); + case 1: + final d = 1 + salt % (n - 1); + return LombIn(i.t.sublist(d), i.y.sublist(d), i.grid); + default: + final d = 1 + salt % (n - 1); + return LombIn(i.t.sublist(0, n - d), i.y.sublist(0, n - d), i.grid); + } + } + + // The grid belongs to the constructor (and travels in the checkpoint). + @override + LombIn? reconfigure(LombIn i, int alt) => null; + + @override + dynamic fresh(LombIn input) => IncrementalLombScargle(_lombGrids[input.grid]); + @override + dynamic fromJson(Json j) => IncrementalLombScargle.fromJson(j); + @override + Json toJson(dynamic s) => (s as IncrementalLombScargle).toJson(); + @override + Object? sync(dynamic s, LombIn i, {bool force = false}) => + (s as IncrementalLombScargle).sync(i.t, i.y, force: force); + + @override + String show(Object? out) { + final r = out as LombScargle?; + if (r == null) return 'null'; + return [for (final p in r.spectrum) '${p.freqHz}:${p.psd}'].join('|'); + } + + /// The production batch function on times shifted by the first one (a + /// parity check, not an independent oracle): the periodogram is + /// shift-invariant, the shift is exact, and on raw epoch seconds the batch's + /// own trigonometric arguments lose ~1e-6 relative (drift_test.dart). + LombScargle? _batchOf(LombIn i) { + final ok = i.t.length == i.y.length && + i.t.isNotEmpty && + i.t.every((x) => x.isFinite) && + i.y.every((x) => x.isFinite); + if (!ok) return lombScargle(i.t, i.y, i.freqs); + final first = i.t.first; + return lombScargle([for (final x in i.t) x - first], i.y, i.freqs); + } + + @override + void close(Object? out, LombIn i) => spectrumClose(out as LombScargle?, _batchOf(i)); + + @override + void sameOutput(Object? a, Object? b, String why) { + try { + spectrumClose(a as LombScargle?, b as LombScargle?); + } on TestFailure catch (e) { + fail('$why: ${e.message}'); + } + } + + @override + List get counters => const ['processedPoints']; + + @override + List get special => const [0, 1, 2, 3, 4, 5, 30, 31, 60, 61, 62]; + + @override + List get refuseMutations => [ + (j) => (j['t'] as List).add(1.0), + (j) { + final y = j['y'] as List; + if (y.isNotEmpty) y.removeLast(); + }, + (j) { + final sums = j['sums'] as List; + if (sums.isNotEmpty) (sums.first as List).removeLast(); + }, + (j) { + final sums = j['sums'] as List; + if (sums.isNotEmpty) sums.removeLast(); + }, + (j) => (j['moments'] as Map)['count'] = ((j['moments'] as Map)['count'] as int) + 1, + (j) { + // (Only an incremental state ties its work counter to its data.) + final n = (j['t'] as List).length; + if (n > 0 && j['incremental'] == true) j['processedPoints'] = n - 1; + }, + (j) { + if (j['incremental'] == true && (j['t'] as List).isNotEmpty) { + j['origin'] = (j['origin'] as num) + 1.0; + } + }, + (j) { + if (j['incremental'] == true && (j['t'] as List).isNotEmpty) { + j['tMax'] = (j['tMax'] as num) + 1.0; + } + }, + (j) { + if (j['incremental'] == true && (j['t'] as List).isNotEmpty) { + final sums = j['sums'] as List; + if (sums.isNotEmpty) (sums.first as List)[2] = ((sums.first as List)[2] as num) + 5.0; + } + }, + (j) { + if (((j['moments'] as Map)['count'] as int) > 0) j['incremental'] = false; + }, + (j) => (j['frequencies'] as List).add(.5), + (j) { + if (j['incremental'] == true && (j['y'] as List).isNotEmpty) { + j['yOrigin'] = (j['yOrigin'] as num) + 1.0; + } + }, + ]; + + @override + List get faithfulMutations => [ + (j) { + final sums = j['sums'] as List; + if (sums.isNotEmpty) (sums.first as List)[0] = ((sums.first as List)[0] as num) + 1.0; + }, + (j) { + final sums = j['sums'] as List; + if (sums.isNotEmpty) (sums.first as List)[4] = ((sums.first as List)[4] as num) + 1.0; + }, + (j) { + final sums = j['sums'] as List; + if (sums.isNotEmpty) (sums.last as List)[5] = ((sums.last as List)[5] as num) + 1.0; + }, + (j) => j['processedPoints'] = (j['processedPoints'] as int) + 3, + (j) { + final m = j['moments'] as Map; + m['m2'] = ((m['m2'] as num) + 1.0); + }, + ]; + + @override + void conserved(Json cp, LombIn i, String why) { + final finite = i.t.length == i.y.length && + i.t.every((x) => x.isFinite) && + i.y.every((x) => x.isFinite); + if (!finite) { + // The batch decides; nothing is folded into the sums. + expect((cp['moments'] as Map)['count'], 0, reason: 'no moments $why'); + return; + } + final n = i.t.length; + expect((cp['moments'] as Map)['count'], n, reason: 'every value is a moment, once $why'); + expect((cp['t'] as List).length, n, reason: why); + // cos^2 + sin^2 = 1 at every point: the two squares sum to n per frequency. + if (n > 0) { + for (final row in cp['sums'] as List) { + final r = (row as List).cast(); + expect((r[2] + r[3] - n).abs() <= 1e-8 * n, isTrue, + reason: 'sum of cos^2 and sin^2 is the count ${r[2] + r[3]} vs $n $why'); + } + } + } + + @override + Gen> recipeGen(int maxN) => IntsGen([ + SizeGen(maxN, pool: const [0, 1, 2, 3, 4, 5, 30, 31, 60, 61, 62]), + G.intIn(0, 1 << 12), + G.intIn(0, _lombFlavours.length - 1), + G.intIn(0, _lombGrids.length - 1), + G.intIn(0, _lombOrigins.length - 1), + ]); + + @override + List> get forced => [ + [0, 1, 0, 0, 0], + [3, 1, 0, 0, 0], // under 4 points: abstains + [4, 1, 0, 0, 1], // exactly 4 + [60, 1, 0, 0, 0], + [60, 2, 0, 0, 2], // epoch-sized times + [100, 3, 1, 5, 0], // irregular, dense grid + [40, 1, 2, 0, 0], // constant y + [40, 1, 3, 0, 0], // collapsed time + [40, 1, 4, 0, 1], // duplicate times + [40, 1, 5, 0, 0], // NaN value + [40, 1, 6, 0, 0], // infinite time + [40, 1, 7, 0, 0], // lengths differ + [50, 1, 0, 1, 0], // empty grid + [50, 1, 0, 2, 0], // grid of zero only + [50, 1, 0, 3, 1], // repeated frequency + [50, 1, 0, 4, 0], // unsorted grid + ]; + + @override + void observe(List r, void Function(String) bump) { + bump('flavour: ${_lombFlavours[r[2]]}'); + final n = r[0]; + if (n == 0) bump('series: empty'); + if (n >= 1 && n < 4) bump('series: under 4 points'); + if (n >= 60) bump('series: 60 or more'); + bump('grid: ${r[3]}'); + if (r[4] == 2) bump('epoch-sized times'); + } + + @override + Map get shares => { + for (final f in _lombFlavours) 'flavour: $f': .02, + 'series: under 4 points': .03, + 'series: 60 or more': .3, + for (var g = 0; g < _lombGrids.length; g++) 'grid: $g': .03, + 'epoch-sized times': .15, + }; +} + +// ═════════════════════════════════════════════════════════════════════════════ +// Minute metrics +// ═════════════════════════════════════════════════════════════════════════════ + +const _hrBranches = [54.0, 95.0, 106.799999, 106.8, 130.0, 190.0, 0.0, double.nan, 40.0]; +const _cadBranches = [null, 99.999, 100, 110, 120, 130, 160, 0, double.nan]; + +/// (resting HR, max HR, sex, profile name, day minutes, quiet HRR). Anchors +/// that gate the TRIMP off (null, equal, inverted, NaN), a negative resting HR, +/// no profile (no energy), a short, a DST and a long day, and quiet-HRR gates. +final List<(double?, double?, Sex, String?, int, double?)> _minCfgs = [ + (54, 186, Sex.male, 'male', 1440, .12), + (65, 197, Sex.female, 'female', 900, .12), + (null, 186, Sex.male, 'male', 1440, .12), + (54, null, Sex.male, 'nonbinary', 1440, null), + (70, 60, Sex.male, 'male', 1440, .18), + (double.nan, 186, Sex.male, 'male', 0, .12), + (54, 186, Sex.female, null, 1440, .12), + (54, 186, Sex.male, 'nonbinary', 1380, 0.0), + (54, 186, Sex.male, 'female', 1441, -0.1), + (54, 186, Sex.male, 'male', 1500, double.nan), + (54, 54, Sex.male, 'male', 1440, .12), + // A resting HR below zero is accepted (finite, below the max): an HR of 0 + // (off-skin) is still billed nothing, where HR 0 would otherwise read as half + // the reserve. No profile, so the energy pricer does not take the anchors. + (-1, 1, Sex.male, null, 1440, .12), +]; + +class MinIn { + MinIn(this.keys, this.hr, this.cad, this.cfg, this.series); + final List keys; + final List hr; + final List? cad; + final int cfg; + final bool series; + WorkoutUserProfile? get profile { + final name = _minCfgs[cfg].$4; + return name == null ? null : energyProfiles[name]; + } +} + +String _metric(Metric m) => + '${m.present}/${m.value}/${m.confidence}/${m.tier}/${m.note}/${m.inputs_used}'; + +const _minTotals = ['trimpTotal', 'hrActiveTotal', 'walkingTotal']; + +class MinOps extends Ops { + @override + String get name => 'IncrementalMinuteMetrics'; + + @override + MinIn expand(List r) { + final (n, seed, cfg, cadMode, series) = (r[0], r[1], r[2], r[3], r[4]); + return MinIn( + List.generate(n, (i) => 28000000 + i + i ~/ 17), + List.generate(n, (i) => _hrBranches[(i + seed) % _hrBranches.length]), + cadMode == 0 ? null : List.generate(n, (i) => _cadBranches[(i + seed) % _cadBranches.length]), + cfg, + series == 1, + ); + } + + @override + int length(MinIn i) => i.keys.length; + @override + MinIn prefix(MinIn i, int k) => MinIn(i.keys.sublist(0, k), i.hr.sublist(0, k), + i.cad?.sublist(0, k), i.cfg, i.series); + @override + MinIn edit(MinIn i, int kind, int salt) { + final n = i.keys.length; + if (n < 2) return i; + switch (kind % 3) { + case 0: + // One minute's HR replaced, or two minutes swapped as whole items, or + // every minute in the opposite order: the keys are then not ascending + // and every key keeps its value, so the bills are reused in a new order. + switch (salt ~/ 3 % 3) { + case 0: + final hr = [...i.hr]; + hr[salt % n] += 29; + return MinIn(i.keys, hr, i.cad, i.cfg, i.series); + case 1: + final a = salt % n, b = (salt * 7 + 1) % n == a ? (a + 1) % n : (salt * 7 + 1) % n; + return _reorder(i, [ + for (var k = 0; k < n; k++) k == a ? b : (k == b ? a : k) + ]); + default: + return _reorder(i, [for (var k = n - 1; k >= 0; k--) k]); + } + case 1: + final d = 1 + salt % (n - 1); + return MinIn(i.keys.sublist(d), i.hr.sublist(d), i.cad?.sublist(d), i.cfg, i.series); + default: + final d = 1 + salt % (n - 1); + return MinIn(i.keys.sublist(0, n - d), i.hr.sublist(0, n - d), + i.cad?.sublist(0, n - d), i.cfg, i.series); + } + } + + /// The same minutes, each with its own HR and cadence, in the order [from]. + static MinIn _reorder(MinIn i, List from) => MinIn( + [for (final k in from) i.keys[k]], + [for (final k in from) i.hr[k]], + i.cad == null ? null : [for (final k in from) i.cad![k]], + i.cfg, + i.series); + + @override + MinIn? reconfigure(MinIn i, int alt) => + MinIn(i.keys, i.hr, i.cad, (i.cfg + 1 + alt % (_minCfgs.length - 1)) % _minCfgs.length, i.series); + + @override + dynamic fresh(MinIn input) => IncrementalMinuteMetrics(); + @override + dynamic fromJson(Json j) => IncrementalMinuteMetrics.fromJson(j); + @override + Json toJson(dynamic s) => (s as IncrementalMinuteMetrics).toJson(); + @override + Object? sync(dynamic s, MinIn i, {bool force = false}) { + final c = _minCfgs[i.cfg]; + return (s as IncrementalMinuteMetrics).sync(i.keys, i.hr, + cadenceSpm: i.cad, + restingHr: c.$1, + maxHr: c.$2, + sex: c.$3, + profile: i.profile, + dayMinutes: c.$5, + quietHrr: c.$6, + force: force, + includeMinuteSeries: i.series); + } + + @override + String show(Object? out) { + final m = out as MinuteMetrics; + final e = m.energy, s = m.minutes; + return [ + _metric(m.trimp), + _metric(m.strain), + if (e != null) [e.total, e.active, e.basal, e.walking], + if (s != null) ...[ + s.coveredMinutes, + s.abstainedMinutes, + s.basalKcalPerMin, + s.active, + s.walking, + for (final x in s.minutes) + [x.minute, x.source, x.abstained, x.basal, x.active, x.total, x.walking], + ], + ].join('|'); + } + + @override + void close(Object? out, MinIn i) { + final m = out as MinuteMetrics; + final c = _minCfgs[i.cfg]; + if (i.series) { + minuteClose(m, i.keys, i.hr, + cadence: i.cad, + rhr: c.$1, + maxHr: c.$2, + sex: c.$3, + profile: i.profile, + dayMinutes: c.$5, + quietHrr: c.$6); + return; + } + // A summary request omits the minute artifacts and equals the batch's + // trimp, strain and daily energy. + expect(m.minutes, isNull, reason: 'summary requests omit minute artifacts'); + final trimp = banisterTrimp(i.hr, restingHr: c.$1, maxHr: c.$2, sex: c.$3); + final strain = strainScoreMetric(trimp.value, + wakeMinutes: i.hr.length.toDouble(), + quietHrr: c.$6, + quietSettled: false, + female: c.$3 == Sex.female); + metricEnvelope(m.trimp, trimp); + numberClose(m.trimp.value, trimp.value); + metricEnvelope(m.strain, strain); + numberClose(m.strain.value, strain.value); + final p = i.profile; + final energy = p == null || c.$1 == null || c.$2 == null + ? null + : Calories.dailyEnergy(i.hr, + profile: p, + hrmax: c.$2!, + restingHr: c.$1!, + dayMinutes: c.$5, + cadenceSpmPerMin: i.cad); + if (energy == null) { + expect(m.energy, isNull); + } else { + expect(m.energy, isNotNull); + numberClose(m.energy!.total, energy.total); + numberClose(m.energy!.active, energy.active); + numberClose(m.energy!.basal, energy.basal); + numberClose(m.energy!.walking, energy.walking); + } + } + + @override + List get counters => const ['processedMinutes']; + + // After a detour the bills may have been added in another order and the totals + // carry add/remove float residue: the checkpoint is equal in meaning, not in + // text. The projection is BY KEY: every field but the bills and the three + // totals as text (the counters aside), the bills as a map from minute key to + // bill (each bill exact: a bill is computed from one minute alone), the totals + // within the reader's own tolerance, 1e-9 relative. + @override + void sameState(Json got, Json want, String why) { + Map rest(Json j) => { + for (final e in j.entries) + if (!counters.contains(e.key) && + e.key != 'bills' && + !_minTotals.contains(e.key)) + e.key: e.value + }; + expect(jsonEncode(rest(got)), jsonEncode(rest(want)), reason: '$why: parameters'); + Map byKey(Json j) { + final out = {}; + for (final b in j['bills'] as List) { + final m = (b as Map).cast(); + expect(out.containsKey(m['key']), isFalse, reason: '$why: a minute billed twice'); + out[m['key']] = jsonEncode(m); + } + return out; + } + + expect(byKey(got), byKey(want), reason: '$why: the bills, by minute key'); + for (final t in _minTotals) { + final g = (got[t] as num).toDouble(), w = (want[t] as num).toDouble(); + expect((g - w).abs() <= 1e-9 * math.max(1, w.abs()), isTrue, + reason: '$why: $t $g vs $w'); + } + } + + // The same series after a detour: its sums were added to and taken from, so + // the numbers are the batch's within the batch tolerance, not bit for bit; + // the minute series (a bill per minute, in key order) is exact. + @override + void sameOutput(Object? a, Object? b, String why) { + final x = a as MinuteMetrics, y = b as MinuteMetrics; + metricEnvelope(x.trimp, y.trimp); + numberClose(x.trimp.value, y.trimp.value, reason: '$why: trimp'); + metricEnvelope(x.strain, y.strain); + numberClose(x.strain.value, y.strain.value, reason: '$why: strain'); + expect(x.energy == null, y.energy == null, reason: '$why: energy present'); + if (x.energy != null) { + numberClose(x.energy!.total, y.energy!.total, reason: '$why: total'); + numberClose(x.energy!.active, y.energy!.active, reason: '$why: active'); + numberClose(x.energy!.basal, y.energy!.basal, reason: '$why: basal'); + numberClose(x.energy!.walking, y.energy!.walking, reason: '$why: walking'); + } + expect(x.minutes == null, y.minutes == null, reason: '$why: series present'); + if (x.minutes != null) { + final p = x.minutes!, q = y.minutes!; + expect([for (final m in p.minutes) [m.minute, m.source, m.abstained, m.basal, m.active, m.total, m.walking]], + [for (final m in q.minutes) [m.minute, m.source, m.abstained, m.basal, m.active, m.total, m.walking]], + reason: '$why: the minute series'); + expect((p.coveredMinutes, p.abstainedMinutes, p.basalKcalPerMin), + (q.coveredMinutes, q.abstainedMinutes, q.basalKcalPerMin), + reason: '$why: series counts'); + numberClose(p.active, q.active, reason: '$why: series active'); + numberClose(p.walking, q.walking, reason: '$why: series walking'); + } + } + + @override + List get special => const [0, 1, 2, 3, 29, 30, 31, 60, 61]; + + @override + List<({String name, bool Function(MinIn) applies, void Function(dynamic, MinIn) call})> + get refusals => [ + ( + name: 'one more HR than keys', + applies: (i) => true, + call: (s, i) => (s as IncrementalMinuteMetrics) + .sync(i.keys, [...i.hr, 70], cadenceSpm: i.cad) + ), + ( + name: 'one more cadence than keys', + applies: (i) => true, + call: (s, i) => (s as IncrementalMinuteMetrics).sync(i.keys, i.hr, + cadenceSpm: [...(i.cad ?? List.filled(i.hr.length, null)), 100]) + ), + ( + name: 'a key appended twice', + applies: (i) => i.keys.isNotEmpty, + call: (s, i) => (s as IncrementalMinuteMetrics) + .sync([...i.keys, i.keys.first], [...i.hr, 70]) + ), + ( + name: 'a duplicate inside a reordering', + applies: (i) => i.keys.length >= 2, + call: (s, i) => (s as IncrementalMinuteMetrics).sync( + [i.keys[1], i.keys[0], i.keys[1]], [70, 71, 72]) + ), + ]; + + Map _bill(Json j, int i) => ((j['bills'] as List)[i] as Map); + + @override + List get refuseMutations => [ + (j) { + final n = (j['bills'] as List).length; + if (n > 0) j['processedMinutes'] = n - 1; + }, + (j) => j['trimpTotal'] = (j['trimpTotal'] as num) + 5.0, + (j) { + final b = j['bills'] as List; + if (b.isNotEmpty) b.add(deepCopy(b.first)); + }, + (j) { + if ((j['bills'] as List).isNotEmpty) _bill(j, 0)['trimp'] = -1.0; + }, + (j) { + if ((j['bills'] as List).isNotEmpty) { + _bill(j, 0)['source'] = 'hr'; + _bill(j, 0)['walking'] = 5.0; + } + }, + (j) { + if ((j['bills'] as List).isNotEmpty) j['parameters'] = null; + }, + (j) { + if ((j['bills'] as List).any((b) => (b as Map)['source'] != null)) { + j['basalPerMinute'] = null; + } + }, + (j) => j['hrActiveTotal'] = (j['hrActiveTotal'] as num) + 5.0, + (j) { + final p = j['parameters'] as List?; + if (p != null) p[2] = 'bogus'; + }, + (j) { + final p = j['parameters'] as List?; + if (p != null) p.removeLast(); + }, + (j) { + if ((j['bills'] as List).isNotEmpty) _bill(j, 0)['source'] = 'nonsense'; + }, + (j) { + if ((j['bills'] as List).isNotEmpty) _bill(j, 0)['key'] = 'x'; + }, + (j) => j['bills'] = 'x', + (j) => j['walkingTotal'] = (j['walkingTotal'] as num) + 5.0, + ]; + + @override + List get faithfulMutations => [ + (j) { + if ((j['bills'] as List).isNotEmpty) _bill(j, 0)['hr'] = 111.0; + }, + (j) { + if ((j['bills'] as List).isNotEmpty) _bill(j, 0)['cadence'] = 111.0; + }, + (j) { + final p = j['parameters'] as List?; + if (p != null && p[0] != null && p[0] is num) p[0] = (p[0] as num) + 1.0; + }, + (j) => j['basalPerMinute'] = ((j['basalPerMinute'] as num?) ?? 0) + 1.0, + (j) => j['processedMinutes'] = (j['processedMinutes'] as int) + 5, + ]; + + @override + void conserved(Json cp, MinIn i, String why) { + final bills = cp['bills'] as List; + expect(bills.length, i.keys.length, reason: 'a bill for every minute, once $why'); + expect({for (final b in bills) (b as Map)['key']}, i.keys.toSet(), + reason: 'the bills are the minutes handed in $why'); + var trimp = 0.0, active = 0.0, walking = 0.0; + for (final b in bills) { + final m = b as Map; + trimp += m['trimp'] as num; + active += m['active'] as num; + walking += m['walking'] as num; + } + double tol(double v) => 1e-9 * math.max(1, v.abs()); + expect((cp['trimpTotal'] as num) - trimp, inInclusiveRange(-tol(trimp), tol(trimp)), + reason: 'the total is the sum of the bills (trimp) $why'); + expect((cp['hrActiveTotal'] as num) - active, inInclusiveRange(-tol(active), tol(active)), + reason: 'the total is the sum of the bills (active) $why'); + expect((cp['walkingTotal'] as num) - walking, inInclusiveRange(-tol(walking), tol(walking)), + reason: 'the total is the sum of the bills (walking) $why'); + // Each bill echoes the minute it prices. + final byKey = {for (final b in bills) (b as Map)['key']: b}; + for (var k = 0; k < i.keys.length; k++) { + final b = byKey[i.keys[k]]!; + final h = i.hr[k]; + expect(b['hr'], h.isNaN ? 'NaN' : h, reason: 'hr of ${i.keys[k]} $why'); + final c = i.cad?[k]; + expect(b['cadence'], c == null ? isNull : (c.isNaN ? 'NaN' : c), + reason: 'cadence of ${i.keys[k]} $why'); + } + } + + @override + Gen> recipeGen(int maxN) => IntsGen([ + SizeGen(maxN, pool: const [0, 1, 2, 3, 29, 30, 31, 60, 61]), + G.intIn(0, 1 << 12), + G.intIn(0, _minCfgs.length - 1), + G.elements(const [0, 1, 1]), + G.elements(const [0, 1]), + ]); + + @override + List> get forced => [ + [0, 1, 0, 0, 1], // no minutes + [0, 1, 6, 1, 0], + [1, 1, 0, 1, 1], // one minute + [31, 1, 0, 1, 1], + [61, 3, 1, 1, 0], + [47, 5, 2, 1, 1], // resting HR unknown + [47, 5, 3, 0, 0], // max HR unknown, quiet gate absent + [47, 5, 4, 1, 1], // anchors inverted + [47, 5, 5, 1, 0], // NaN anchor, day of zero minutes + [47, 5, 6, 1, 1], // no profile + [47, 5, 7, 1, 0], // DST-short day, quiet gate 0 + [47, 5, 8, 1, 1], // long day, negative quiet gate + [47, 5, 9, 1, 0], // NaN quiet gate + [47, 5, 10, 1, 1], // anchors equal + [47, 5, 11, 1, 1], // resting HR below zero: HR 0 is off-skin, billed nothing + [47, 5, 11, 0, 0], + [90, 914, 0, 1, 0], + ]; + + @override + void observe(List r, void Function(String) bump) { + final n = r[0]; + if (n == 0) bump('series: empty'); + if (n >= 1 && n < 30) bump('series: under 30 minutes'); + if (n >= 60) bump('series: an hour or more'); + final c = _minCfgs[r[2]]; + if (c.$1 == null || c.$2 == null || c.$1!.isNaN) bump('anchors: missing'); + if (c.$1 != null && c.$2 != null && c.$1!.isFinite && c.$1! >= c.$2!) bump('anchors: not ordered'); + if (c.$4 == null) bump('no profile'); + if (c.$1 != null && c.$1! < 0) bump('anchors: negative resting HR'); + if (c.$3 == Sex.female) bump('sex: female'); + if (r[3] == 1) bump('cadence given'); + bump(r[4] == 1 ? 'minute series requested' : 'summary only'); + } + + @override + Map get shares => { + 'series: empty': .01, + 'series: under 30 minutes': .05, + 'series: an hour or more': .1, + 'anchors: missing': .1, + 'anchors: not ordered': .03, + 'anchors: negative resting HR': .03, + 'no profile': .05, + 'sex: female': .05, + 'cadence given': .4, + 'minute series requested': .3, + 'summary only': .3, + }; +} + +// ═════════════════════════════════════════════════════════════════════════════ +// RunningMoments +// ═════════════════════════════════════════════════════════════════════════════ + +const _momFlavours = [ + 'heart rate', // whole bpm, a spread of 40 + 'large offset', // 1e9 plus a spread under 1: where sum-of-squares loses every digit + 'constant', // all equal: variance exactly zero + 'wide', // magnitudes from 1e-3 to 1e6, both signs + 'two valued', + 'small spread near zero', +]; + +/// The values of a recipe (flavour, n, seed). +List _momValues(int flavour, int n, int seed) { + final g = Rng(seed * 8191 + flavour + 3); + return [ + for (var i = 0; i < n; i++) + switch (flavour) { + 0 => 40.0 + g.nextInt(120), + 1 => 1e9 + (g.nextDouble() * .5 + (i % 7) * .25), + 2 => 42.0, + 3 => (g.nextBool() ? 1 : -1) * math.pow(10, g.nextDouble() * 9 - 3).toDouble(), + 4 => i.isEven ? 800.0 : 1000.0, + _ => (g.nextDouble() - .5) * 1e-3, + } + ]; +} + +/// Mean and sample / population SD of [x] the stable way: about the first value, +/// two passes. A reference written here (unlike the batch functions the sync +/// modules are compared with): independent of the Welford state under test, and +/// accurate where a plain sum is not (a 1e9 offset). +({double? mean, double? sd, double? pop}) _twoPass(List x) { + final n = x.length; + if (n == 0) return (mean: null, sd: null, pop: null); + final x0 = x.first; + var s = 0.0; + for (final v in x) { + s += v - x0; + } + final md = s / n; + var ss = 0.0; + for (final v in x) { + final d = v - x0 - md; + ss += d * d; + } + return ( + mean: x0 + md, + sd: n < 2 ? null : math.sqrt(ss / (n - 1)), + pop: math.sqrt(ss / n), + ); +} + +const double _eps = 2.220446049250313e-16; + +/// The moments hold what two passes say: the count exact, the mean to a few +/// ulps of its own size, the SDs to 1e-9 relative (plus the same ulps). +void _momentsMatch(RunningMoments m, List x, String why) { + final o = _twoPass(x); + expect(m.count, x.length, reason: 'count $why'); + if (x.isEmpty) { + expect(m.mean, isNull, reason: 'no values, no mean $why'); + expect(m.sampleSd, isNull, reason: why); + expect(m.populationSd, isNull, reason: why); + return; + } + final scale = x.fold(1, (a, v) => math.max(a, v.abs())); + final tol = 64 * _eps * scale; + expect((m.mean! - o.mean!).abs() <= tol + 1e-12 * (o.sd ?? 0), isTrue, + reason: 'mean ${m.mean} vs ${o.mean} (tol $tol) $why'); + void sdClose(double? got, double? want, String what) { + if (want == null) { + expect(got, isNull, reason: '$what is absent below two values $why'); + return; + } + expect(got, isNotNull, reason: '$what $why'); + final t = 1e-9 * want + tol; + expect((got! - want).abs() <= t, isTrue, + reason: '$what $got vs $want (tol $t) $why'); + } + + sdClose(m.sampleSd, o.sd, 'sample sd'); + sdClose(m.populationSd, o.pop, 'population sd'); +} + +RunningMoments _momOf(List x) { + final m = RunningMoments(); + for (final v in x) { + m.add(v); + } + return m; +} + +String _momText(RunningMoments m) => jsonEncode(m.toJson()); + +/// (flavour, n, seed, splits, cut seed). +typedef MomCase = List; + +final Gen _momGen = IntsGen([ + G.intIn(0, _momFlavours.length - 1), + SizeGen(400, pool: const [0, 1, 2, 3, 4, 5, 10]), + G.intIn(0, 1 << 12), + G.intIn(0, 6), + G.intIn(0, 1 << 20), +]); + +List> _parts(MomCase c, List x) { + final b = foldBounds(x.length, c[3], c[4]); + return [for (var k = 0; k + 1 < b.length; k++) x.sublist(b[k], b[k + 1])]; +} + +void _observeMom(MomCase c, void Function(String) bump) { + bump('flavour: ${_momFlavours[c[0]]}'); + if (c[1] == 0) bump('empty'); + if (c[1] == 1) bump('one value'); + if (c[1] >= 100) bump('100 values or more'); + final b = foldBounds(c[1], c[3], c[4]); + if (b.length >= 4) bump('three parts or more'); + if (hasEmptyChunk(b)) bump('an empty part'); +} + +final Map _momShares = { + for (final f in _momFlavours) 'flavour: $f': .03, + 'empty': .01, + 'one value': .01, + '100 values or more': .2, + 'three parts or more': .4, + 'an empty part': .05, +}; + +final List _momForced = [ + [0, 0, 1, 2, 3], // nothing + [0, 1, 1, 2, 3], // one value + [0, 2, 1, 2, 3], + [1, 300, 1, 3, 5], // large offset + [2, 80, 1, 3, 7], // constant + [3, 200, 2, 4, 11], // wide + [4, 51, 1, 6, 13], // two valued, odd + [5, 120, 4, 5, 17], + [0, 100, 3, 6, 19], // heart rate +]; + +void _momMerge(MomCase c) { + final x = _momValues(c[0], c[1], c[2]); + final parts = _parts(c, x); + final tag = 'moments $c parts=${[for (final p in parts) p.length]}'; + final ms = [for (final p in parts) _momOf(p)]; + final before = [for (final m in ms) _momText(m)]; + // Left to right: merging the summaries of the parts is the summary of the + // concatenation. + final left = RunningMoments(); + for (final m in ms) { + left.merge(m); + } + _momentsMatch(left, x, '$tag: left fold'); + for (var k = 0; k < ms.length; k++) { + expect(_momText(ms[k]), before[k], reason: '$tag: the argument is left unchanged'); + } + // Associativity: (a+b)+c and a+(b+c) read the same as the concatenation; the + // state differs only in rounding. + if (parts.length >= 3) { + final ab = _momOf(parts[0])..merge(_momOf(parts[1])); + final abc = ab..merge(_momOf([for (final p in parts.skip(2)) ...p])); + final bc = _momOf(parts[1])..merge(_momOf([for (final p in parts.skip(2)) ...p])); + final a_bc = _momOf(parts[0])..merge(bc); + _momentsMatch(abc, x, '$tag: (a+b)+c'); + _momentsMatch(a_bc, x, '$tag: a+(b+c)'); + } + // Commutativity: the order of two parts does not matter to what is read. + if (parts.length >= 2) { + final ab = _momOf(parts[0])..merge(_momOf(parts[1])); + final ba = _momOf(parts[1])..merge(_momOf(parts[0])); + final both = [...parts[0], ...parts[1]]; + _momentsMatch(ab, both, '$tag: a+b'); + _momentsMatch(ba, both, '$tag: b+a'); + } + // Identity, exactly: merging the empty summary changes nothing, and merging + // into the empty summary copies. + final whole = _momOf(x); + final text = _momText(whole); + whole.merge(RunningMoments()); + expect(_momText(whole), text, reason: '$tag: x + empty == x'); + expect(_momText(RunningMoments()..merge(whole)), text, reason: '$tag: empty + x == x'); + // A summary merged with itself is the multiset twice. + final twice = _momOf(x)..merge(whole); + _momentsMatch(twice, [...x, ...x], '$tag: x + x'); + final selfMerged = _momOf(x); + selfMerged.merge(selfMerged); + _momentsMatch(selfMerged, [...x, ...x], '$tag: merge with itself'); +} + +void _momAddRemove(MomCase c) { + final x = _momValues(c[0], c[1], c[2]); + final tag = 'moments $c'; + final g = Rng(c[4] + 5); + final m = RunningMoments(); + final held = []; + // A walk of adds and removes (of values that are held), checked against two + // passes along the way. + for (var i = 0; i < x.length; i++) { + m.add(x[i]); + held.add(x[i]); + if (held.length > 2 && g.nextBool(.4)) { + final v = held.removeAt(g.nextInt(held.length)); + m.remove(v); + } + if (i % 7 == 0 || i == x.length - 1) _momentsMatch(m, held, '$tag step $i'); + } + // add then remove the same value: back where it was. (A value inside the + // data's range: removing a downdate of a wild outlier is a numerical limit of + // every running variance, not a contract.) + final probe = _momOf(held); + final text = _momText(probe); + final extra = held.isEmpty ? 123.5 : held.last; + probe.add(extra); + probe.remove(extra); + _momentsMatch(probe, held, '$tag: add then remove'); + if (held.isEmpty) expect(_momText(probe), text, reason: '$tag: back to the empty state exactly'); + // Emptied again: exactly the fresh state. + while (held.isNotEmpty) { + m.remove(held.removeLast()); + } + expect(_momText(m), _momText(RunningMoments()), reason: '$tag: emptied is fresh'); +} + +void _momJson(MomCase c) { + final x = _momValues(c[0], c[1], c[2]); + final tag = 'moments $c'; + final m = _momOf(x); + final text = _momText(m); + final back = RunningMoments.fromJson((jsonDecode(text) as Map).cast()); + expect(_momText(back), text, reason: '$tag: write(read(b)) == b'); + _momentsMatch(back, x, '$tag: restored reads the same'); + // And goes on identically. + m.add(77); + back.add(77); + expect(_momText(back), _momText(m), reason: '$tag: same after one more'); + _momentsMatch(back, [...x, 77], tag); + // Identity and refusals leave the state unchanged. + final before = _momText(m); + expect(() => m.add(double.nan), throwsArgumentError); + expect(() => m.add(double.infinity), throwsArgumentError); + expect(() => m.remove(double.negativeInfinity), throwsArgumentError); + expect(() => RunningMoments().remove(1), throwsStateError); + expect(_momText(m), before, reason: '$tag: refused calls change nothing'); + expect(m.count, x.length + 1); + // Conservation: (n - 1) sample variance and n population variance are the + // same sum of squares. + if (m.count >= 2) { + final a = m.sampleSd! * m.sampleSd! * (m.count - 1); + final b = m.populationSd! * m.populationSd! * m.count; + expect((a - b).abs() <= 1e-9 * math.max(1, a.abs()), isTrue, + reason: '$tag: one sum of squares: $a vs $b'); + } +} + +(Json, Must) _momMutate(Json src, Mut m) { + final j = (deepCopy(src) as Map).cast(); + final (kind, a, b) = m; + switch (kind % 8) { + case 0: + const versions = [0, 2, -1, 99, '1', null, true, 1 << 40]; + j['version'] = versions[a % versions.length]; + return (j, Must.refuse); + case 1: + const types = ['IntHistogram', 'Other', null, '', 1]; + j['type'] = types[a % types.length]; + return (j, Must.refuse); + case 2: + j.remove(const ['count', 'origin', 'meanOffset', 'm2', 'version', 'type'][a % 6]); + return (j, Must.refuse); + case 3: + const junk = ['x', {}, [], null, 1.5]; + final k = const ['count', 'origin', 'meanOffset', 'm2'][a % 4]; + j[k] = k == 'count' ? junk[b % junk.length] : junk[b % 4]; + return (j, Must.refuse); + case 4: + // Contradictions a reachable state never shows. + switch (a % 5) { + case 0: + j['count'] = -1 - b % 5; + case 1: + j['m2'] = -1.0 - b % 5; + case 2: + j + ..['count'] = 0 + ..['origin'] = 1.0 + b % 7; // nothing counted, yet an origin + case 3: + j + ..['count'] = 1 + ..['m2'] = 1.0 + b % 7; // one value with a spread + default: + j + ..['count'] = 0 + ..['m2'] = 1.0; + } + return (j, jsonEncode(j) == jsonEncode(src) ? Must.faithful : Must.refuse); + case 5: + // Changes the reader cannot tell from a legal state. + switch (a % 4) { + case 0: + if ((j['count'] as int) >= 2) j['m2'] = (j['m2'] as num) + 1.0; + case 1: + if ((j['count'] as int) >= 1) j['meanOffset'] = (j['meanOffset'] as num) + 1.0; + case 2: + if ((j['count'] as int) >= 1) j['origin'] = (j['origin'] as num) + 1.0; + default: + if ((j['count'] as int) >= 2) j['count'] = (j['count'] as int) + 1 + b % 3; + } + return (j, Must.faithful); + case 6: + // Hostile size: the count claims far more than the sums can hold. + if ((j['count'] as int) >= 2) j['count'] = const [1 << 40, 1 << 62, 0x7fffffffffffffff][b % 3]; + return (j, Must.faithful); + default: + j['futureField${a % 3}'] = const [1, 'x', [1]][b % 3]; + return (j, Must.ignored); + } +} + +void _momL3((MomCase, Mut) arg) { + final (c, m) = arg; + final tag = 'moments $c mutation=${m.$1 % 8}(${m.$2},${m.$3})'; + final src = (jsonDecode(_momText(_momOf(_momValues(c[0], c[1], c[2])))) as Map).cast(); + final (bad, must) = _momMutate(src, m); + String outcome; + try { + outcome = 'accepted:${_momText(RunningMoments.fromJson((deepCopy(bad) as Map).cast()))}'; + } on FormatException { + outcome = 'refused'; + } catch (e) { + outcome = 'WRONG ERROR TYPE: ${e.runtimeType}: $e'; + } + expectMutationOutcome(outcome, must, _text(bad), _text(src), tag); +} + +// ═════════════════════════════════════════════════════════════════════════════ +// IntHistogram +// ═════════════════════════════════════════════════════════════════════════════ + +const _histFlavours = [ + 'heart rate', // 40..180 bpm + 'few values', // 3 distinct + 'one value', + 'negatives', // -100..299 + 'wide', // up to +-1e9 +]; + +List _histValues(int flavour, int n, int seed) { + final g = Rng(seed * 4099 + flavour + 1); + return [ + for (var i = 0; i < n; i++) + switch (flavour) { + 0 => 40 + g.nextInt(141), + 1 => 60 + g.nextInt(3), + 2 => 61, + 3 => g.nextInt(400) - 100, + _ => g.nextInt(2000000000) - 1000000000, + } + ]; +} + +IntHistogram _histOf(List x) { + final h = IntHistogram(); + for (final v in x) { + h.add(v); + } + return h; +} + +String _histText(IntHistogram h) => jsonEncode(h.toJson()); + +const _ps = [0, 1, 5, 25, 30, 33.3, 50, 66.6, 75, 90, 95, 99, 100]; + +/// The histogram reads exactly what the sorted expanded list reads: the very +/// double `percentileSorted` gives, count and distinct count exact, and a +/// canonical checkpoint (sorted bins, no zero counts). +void _histMatches(IntHistogram h, List x, String why) { + final sorted = [for (final v in x) v.toDouble()]..sort(); + expect(h.count, x.length, reason: 'count $why'); + expect(h.distinct, x.toSet().length, reason: 'distinct $why'); + if (x.isEmpty) { + expect(h.median, isNull, reason: 'no values, no median $why'); + for (final p in _ps) { + expect(h.percentile(p), isNull, reason: 'p$p $why'); + } + } else { + for (final p in _ps) { + expect(h.percentile(p), percentileSorted(sorted, p), reason: 'p$p $why'); + } + expect(h.median, percentileSorted(sorted, 50), reason: 'median $why'); + } + final j = h.toJson(); + expect(j['values'], ([...x.toSet()]..sort()), reason: 'sorted distinct bins $why'); + final counts = {}; + for (final v in x) { + counts.update(v, (c) => c + 1, ifAbsent: () => 1); + } + expect(j['counts'], [for (final v in j['values'] as List) counts[v]], reason: 'counts $why'); +} + +void _observeHist(MomCase c, void Function(String) bump) { + bump('flavour: ${_histFlavours[c[0]]}'); + if (c[1] == 0) bump('empty'); + if (c[1] == 1) bump('one value'); + if (c[1] >= 100) bump('100 values or more'); + final b = foldBounds(c[1], c[3], c[4]); + if (b.length >= 4) bump('three parts or more'); + if (hasEmptyChunk(b)) bump('an empty part'); +} + +final Map _histShares = { + for (final f in _histFlavours) 'flavour: $f': .03, + 'empty': .01, + 'one value': .01, + '100 values or more': .2, + 'three parts or more': .4, + 'an empty part': .05, +}; + +final Gen _histGen = IntsGen([ + G.intIn(0, _histFlavours.length - 1), + SizeGen(400, pool: const [0, 1, 2, 3, 4, 5, 10]), + G.intIn(0, 1 << 12), + G.intIn(0, 6), + G.intIn(0, 1 << 20), +]); + +final List _histForced = [ + [0, 0, 1, 2, 3], + [0, 1, 1, 2, 3], + [2, 2, 1, 2, 3], // two equal values + [1, 7, 1, 3, 5], + [0, 300, 1, 4, 7], + [3, 150, 2, 5, 9], + [4, 90, 3, 6, 11], + [0, 5, 4, 6, 13], +]; + +void _histMerge(MomCase c) { + final x = _histValues(c[0], c[1], c[2]); + final b = foldBounds(x.length, c[3], c[4]); + final parts = [for (var k = 0; k + 1 < b.length; k++) x.sublist(b[k], b[k + 1])]; + final tag = 'histogram $c parts=${[for (final p in parts) p.length]}'; + final hs = [for (final p in parts) _histOf(p)]; + final before = [for (final h in hs) _histText(h)]; + // The summary of the concatenation, exactly (bins and checkpoint included). + final left = IntHistogram(); + for (final h in hs) { + left.merge(h); + } + _histMatches(left, x, '$tag: left fold'); + expect(_histText(left), _histText(_histOf(x)), reason: '$tag: same checkpoint as one fold'); + for (var k = 0; k < hs.length; k++) { + expect(_histText(hs[k]), before[k], reason: '$tag: the argument is left unchanged'); + } + // Associative and commutative, exactly: the checkpoint is canonical. + if (parts.length >= 3) { + final rest = [for (final p in parts.skip(2)) ...p]; + final ab_c = _histOf(parts[0])..merge(_histOf(parts[1]))..merge(_histOf(rest)); + final a_bc = _histOf(parts[0])..merge(_histOf(parts[1])..merge(_histOf(rest))); + expect(_histText(ab_c), _histText(a_bc), reason: '$tag: associative'); + expect(_histText(ab_c), _histText(_histOf(x)), reason: '$tag: and the concatenation'); + } + if (parts.length >= 2) { + final ab = _histOf(parts[0])..merge(_histOf(parts[1])); + final ba = _histOf(parts[1])..merge(_histOf(parts[0])); + expect(_histText(ab), _histText(ba), reason: '$tag: commutative'); + } + // Identity. + final whole = _histOf(x); + final text = _histText(whole); + whole.merge(IntHistogram()); + expect(_histText(whole), text, reason: '$tag: x + empty == x'); + expect(_histText(IntHistogram()..merge(whole)), text, reason: '$tag: empty + x == x'); + // Itself, doubled. + final self = _histOf(x); + self.merge(self); + _histMatches(self, [...x, ...x], '$tag: merge with itself'); +} + +void _histAddRemove(MomCase c) { + final x = _histValues(c[0], c[1], c[2]); + final tag = 'histogram $c'; + final g = Rng(c[4] + 9); + final h = IntHistogram(); + final held = []; + for (var i = 0; i < x.length; i++) { + if (held.isNotEmpty && g.nextBool(.4)) { + final v = held.removeAt(g.nextInt(held.length)); + h.remove(v); + } else { + held.add(x[i]); + h.add(x[i]); + } + if (i % 5 == 0 || i == x.length - 1) _histMatches(h, held, '$tag step $i'); + } + // Refusals leave the histogram exactly as it was. + final before = _histText(h); + final absent = held.fold(0, (a, v) => math.max(a, v.abs())) + 1; + expect(() => h.remove(absent), throwsStateError, reason: tag); + expect(() => h.remove(double.nan), throwsStateError, reason: tag); + expect(() => h.remove(1.5), throwsStateError, reason: tag); + expect(() => h.add(double.nan), throwsArgumentError, reason: tag); + expect(() => h.add(double.infinity), throwsArgumentError, reason: tag); + expect(() => h.add(70.5), throwsArgumentError, reason: tag); + expect(() => h.add(70, 0), throwsArgumentError, reason: tag); + expect(() => h.add(70, -2), throwsArgumentError, reason: tag); + if (h.count > 0) { + // (An empty histogram answers null for any p; the range is checked once + // there is something to read.) + expect(() => h.percentile(-1), throwsArgumentError, reason: tag); + expect(() => h.percentile(100.5), throwsArgumentError, reason: tag); + expect(() => h.percentile(double.nan), throwsArgumentError, reason: tag); + } + expect(_histText(h), before, reason: '$tag: refused calls change nothing'); + // Repeat counts: add(v, k) is k adds of v; removing is one at a time. + final r = IntHistogram()..add(60, 3)..add(70, 2); + _histMatches(r, [60, 60, 60, 70, 70], '$tag: repeat count'); + r.remove(60); + _histMatches(r, [60, 60, 70, 70], '$tag: removing one of several'); + expect(_histText(IntHistogram()..add(5, 4)), _histText(_histOf([5, 5, 5, 5])), reason: tag); + // A whole-number double is a whole number. + final w = IntHistogram()..add(70.0); + _histMatches(w, [70], tag); + // Add then remove: where it was, no zero-count bin left behind. + final t = _histOf(held); + final text = _histText(t); + t.add(1234567); + t.remove(1234567); + expect(_histText(t), text, reason: '$tag: add then remove'); +} + +void _histJson(MomCase c) { + final x = _histValues(c[0], c[1], c[2]); + final tag = 'histogram $c'; + final h = _histOf(x); + final text = _histText(h); + final back = IntHistogram.fromJson((jsonDecode(text) as Map).cast()); + expect(_histText(back), text, reason: '$tag: write(read(b)) == b'); + _histMatches(back, x, '$tag: restored'); + back.add(55); + h.add(55); + expect(_histText(back), _histText(h), reason: '$tag: same after one more'); + _histMatches(back, [...x, 55], tag); + // Conservation: the counts add up to the count, every bin is held once. + final j = back.toJson(); + expect((j['counts'] as List).fold(0, (a, v) => a + (v as int)), back.count, reason: tag); +} + +(Json, Must) _histMutate(Json src, Mut m) { + final j = (deepCopy(src) as Map).cast(); + final (kind, a, b) = m; + List l(String k) => j[k] as List; + switch (kind % 8) { + case 0: + const versions = [0, 2, -1, 99, '1', null, true, 1 << 40]; + j['version'] = versions[a % versions.length]; + return (j, Must.refuse); + case 1: + const types = ['RunningMoments', 'Other', null, '', 1]; + j['type'] = types[a % types.length]; + return (j, Must.refuse); + case 2: + j.remove(const ['values', 'counts', 'version', 'type'][a % 4]); + return (j, Must.refuse); + case 3: + const junk = ['x', {}, null, 1.5, 3]; + j[const ['values', 'counts'][a % 2]] = junk[b % junk.length]; + return (j, Must.refuse); + case 4: + // The bins contradict themselves. + switch (a % 8) { + case 0: + l('values').add(1 << 40); + case 1: + if (l('counts').isNotEmpty) l('counts').removeLast(); + case 2: + if (l('counts').isNotEmpty) l('counts')[b % l('counts').length] = 0; + case 3: + if (l('counts').isNotEmpty) l('counts')[b % l('counts').length] = -1 - b % 4; + case 4: + if (l('values').length >= 2) { + final t = l('values')[0]; + l('values')[0] = l('values')[1]; + l('values')[1] = t; // out of order + } + case 5: + if (l('values').isNotEmpty) { + l('values').insert(0, l('values')[0]); // a repeated value + l('counts').insert(0, 1); + } + case 6: + if (l('values').isNotEmpty) l('values')[0] = 1.5; + default: + if (l('values').isNotEmpty) l('counts')[0] = 1.5; + } + return (j, jsonEncode(j) == jsonEncode(src) ? Must.faithful : Must.refuse); + case 5: + // A bin count changed: legal. + if (l('counts').isNotEmpty) l('counts')[b % l('counts').length] = (l('counts')[b % l('counts').length] as int) + 1 + a % 5; + return (j, Must.faithful); + case 6: + // Hostile counts: each bin plausible, the total past what an int can + // hold (the skipped FINDING below). + if (l('counts').isNotEmpty) l('counts')[0] = const [1 << 40, 1 << 61, 0x7fffffffffffffff][b % 3]; + return (j, Must.faithful); + default: + j['futureField${a % 3}'] = const [1, 'x', [1]][b % 3]; + return (j, Must.ignored); + } +} + +void _histL3((MomCase, Mut) arg) { + final (c, m) = arg; + final tag = 'histogram $c mutation=${m.$1 % 8}(${m.$2},${m.$3})'; + final src = (jsonDecode(_histText(_histOf(_histValues(c[0], c[1], c[2])))) as Map).cast(); + final (bad, must) = _histMutate(src, m); + String outcome; + try { + outcome = 'accepted:${_histText(IntHistogram.fromJson((deepCopy(bad) as Map).cast()))}'; + } on FormatException { + outcome = 'refused'; + } catch (e) { + outcome = 'WRONG ERROR TYPE: ${e.runtimeType}: $e'; + } + expectMutationOutcome(outcome, must, _text(bad), _text(src), tag); +} + +// ═════════════════════════════════════════════════════════════════════════════ +// CalculationCache +// ═════════════════════════════════════════════════════════════════════════════ + +/// The cache against a model written from its documented behaviour: bounded +/// least-recently-used results, each with an owned snapshot of the dependencies +/// it was computed from; a hit needs the same key and DEEP-equal dependencies +/// (NaN equals NaN) and no `full` request; a hit and a computation are counted; +/// a calculation that throws changes nothing; results are copies; the +/// dependencies are snapshotted before the calculation runs, so a calculation +/// that mutates them does not change what the entry was computed from. +class _CacheModel { + _CacheModel(this.cap); + final int cap; + final List<(String, Object?, Object?)> entries = []; // oldest first + int computations = 0, hits = 0; + + static bool eq(Object? a, Object? b) { + if (a is num && b is num) return a == b || (a.isNaN && b.isNaN); + if (a is List && b is List) { + return a.length == b.length && [for (var i = 0; i < a.length; i++) eq(a[i], b[i])].every((x) => x); + } + if (a is Map && b is Map) { + return a.length == b.length && a.keys.every((k) => b.containsKey(k) && eq(a[k], b[k])); + } + return a == b; + } + + static Object? copy(Object? v) => v is List + ? [for (final e in v) copy(e)] + : v is Map + ? {for (final e in v.entries) e.key: copy(e.value)} + : v; + + Object? evaluate(String key, Object? deps, Object? Function() calc, bool full) { + final at = entries.indexWhere((e) => e.$1 == key); + if (!full && at >= 0 && eq(entries[at].$2, deps)) { + final e = entries.removeAt(at); + entries.add(e); + hits++; + return copy(e.$3); + } + // The dependencies are snapshotted BEFORE the callback runs (the callback + // is user code and may mutate them): the entry holds what the caller passed + // in, and a later lookup with that value hits (CalculationCache.evaluate). + final snapshot = copy(deps); + final v = calc(); // may throw: nothing changes + if (at >= 0) entries.removeAt(at); + entries.add((key, snapshot, copy(v))); + while (entries.length > cap) { + entries.removeAt(0); + } + computations++; + return copy(v); + } +} + +/// A dependency: a small structure with ties, NaN, nesting and null. +Object? _dep(Rng g, int depth) { + switch (g.nextInt(depth > 0 ? 7 : 5)) { + case 0: + return g.nextInt(3); + case 1: + return null; + case 2: + return g.nextBool() ? double.nan : 1.5; + case 3: + return 'k${g.nextInt(2)}'; + case 4: + return g.nextBool(); + case 5: + return [for (var i = g.nextInt(3); i > 0; i--) _dep(g, depth - 1)]; + default: + return {for (var i = g.nextInt(3); i > 0; i--) 'a${g.nextInt(3)}': _dep(g, depth - 1)}; + } +} + +/// Dependencies a mutating calculation works on: nested lists and maps. +Object? _mutableDeps(Object? pooled) => { + 'p': _CacheModel.copy(pooled), + 'q': [1, [2, 3], 'x'], + 'r': {'s': [4, 5], 't': {'u': 6}}, + }; + +/// What a user calculation might do to the dependencies it was handed: change +/// them at depth, grow and shrink them. +void _mutateDeps(Object? d) { + final m = d as Map; + ((m['q'] as List)[1] as List)[0] = 99; + (m['q'] as List).add('added'); + ((m['r'] as Map)['s'] as List).removeLast(); + (((m['r'] as Map)['t']) as Map)['u'] = 7; + m['v'] = 1; +} + +void _cacheModelLaw(List c) { + // (steps, capacity, seed, how often the calculation mutates its dependencies) + final steps = c[0], cap = c[1] + 1, seed = c[2], mutMode = c[3]; + final g = Rng(seed * 31 + 7); + final cache = CalculationCache(maxEntries: cap); + final model = _CacheModel(cap); + final tag = 'cache steps=$steps cap=$cap seed=$seed mut=$mutMode'; + var made = 0; + // The calculations that actually ran, on each side: the cache runs the + // callback exactly when the model says it misses (or is told to recompute), + // failures included. + var cacheRuns = 0, modelRuns = 0, failures = 0; + // A small pool of dependency values, so equal dependencies recur. + final pool = [for (var i = 0; i < 4; i++) _dep(Rng(seed + i * 17), 2)]; + + /// One evaluation on both sides, each with its own copy of the dependencies + /// and its own callback (so a mutation lands on its own copy). + void both(String why, String key, Object? deps, bool full, bool throws, + bool mutates, List value) { + final depsC = _CacheModel.copy(deps), depsM = _CacheModel.copy(deps); + Object? calcC() { + cacheRuns++; + if (mutates) _mutateDeps(depsC); + if (throws) throw StateError('calculation failed'); + return [...value]; + } + + Object? calcM() { + modelRuns++; + if (mutates) _mutateDeps(depsM); + if (throws) throw StateError('calculation failed'); + return [...value]; + } + + Object? got, want; + Object? gotErr, wantErr; + try { + got = cache.evaluate(key, depsC, calcC, full: full); + } catch (e) { + gotErr = e; + failures++; + } + try { + want = model.evaluate(key, depsM, calcM, full); + } catch (e) { + wantErr = e; + } + expect(gotErr.runtimeType, wantErr.runtimeType, reason: '$tag $why: failures propagate'); + expect(got, want, reason: '$tag $why: result'); + expect(cache.computations, model.computations, reason: '$tag $why: computations'); + expect(cache.hits, model.hits, reason: '$tag $why: hits'); + expect(cacheRuns, modelRuns, reason: '$tag $why: calculations that ran'); + expect(cacheRuns, cache.computations + failures, + reason: '$tag $why: one run per computation or failure, none on a hit'); + // The result is the caller's: changing it changes no later hit. + if (got is List && got.isNotEmpty) got[0] = -99; + // The dependencies handed in are snapshotted: changing them afterwards + // does not turn a later hit into a miss (or the reverse). + if (depsC is List && depsC.isNotEmpty) depsC.add('mutated'); + if (depsM is List && depsM.isNotEmpty) depsM.add('mutated'); + } + + for (var i = 0; i < steps; i++) { + final key = 'key${g.nextInt(cap + 2)}'; + final mutates = mutMode > 0 && g.nextBool(mutMode * .2); + final picked = pool[g.nextInt(pool.length)]; + final deps = mutates ? _mutableDeps(picked) : picked; + final full = g.nextBool(.15); + final throws = g.nextBool(.1); + final value = [made++, made % 3]; + both('step $i', key, deps, full, throws, mutates, value); + if (mutates) { + // The contract: the entry was snapshotted before the callback, so a + // lookup with the value passed in (not the mutated one) is the hit, and + // the mutated value is a different dependency. + both('step $i, original value again', key, deps, false, false, false, [-1, -1]); + final after = _mutableDeps(picked); + _mutateDeps(after); + both('step $i, mutated value', key, after, false, false, false, [-2, -2]); + } + } + // Size: never more than the capacity, each key held once. + var held = 0; + for (var k = 0; k < cap + 2; k++) { + var probeCalls = 0; + final before = cache.computations; + cache.evaluate('key$k', _CacheModel.copy(null), () { + probeCalls++; + return [0]; + }); + if (cache.computations == before) held++; + expect(probeCalls <= 1, isTrue); + } + expect(held <= cap, isTrue, reason: '$tag: at most $cap entries held, held $held'); + cache.clear(); + var recomputed = false; + cache.evaluate('key0', null, () => recomputed = true); + expect(recomputed, isTrue, reason: '$tag: clear discards'); +} + +void main() { + _registerSyncLaws(HrvOps(), maxN: 800); + _registerSyncLaws(EnmoOps(), maxN: 900); + _registerSyncLaws(LombOps(), maxN: 300); + _registerSyncLaws(MinOps(), maxN: 300); + group('RunningMoments', () { + _laws.law( + 'M RunningMoments.merge: the summary of the parts merged is the summary of ' + 'the concatenation; associative, commutative, the empty summary is the ' + 'identity, a summary merged with itself is the multiset twice', + _momGen, + _momMerge, + examples: _momForced, + cases: 200, + reach: Reach(_momShares, _observeMom), + ); + _laws.law( + 'L1b RunningMoments: a walk of adds and removes reads what two passes read', + _momGen, + _momAddRemove, + examples: _momForced, + cases: 120, + reach: Reach(_momShares, _observeMom), + ); + _laws.law( + 'L2 RunningMoments: write(read(b)) == b, goes on identically, refused calls ' + 'change nothing, one sum of squares', + _momGen, + _momJson, + examples: _momForced, + cases: 120, + reach: Reach(_momShares, _observeMom), + ); + _laws.law<(MomCase, Mut)>( + 'L3 RunningMoments: a mutated checkpoint is refused whole or holds exactly ' + 'what was written', + G.pair(_momGen, mutGen(8)), + _momL3, + examples: [ + for (var k = 0; k < 8; k++) + for (var i = 0; i < 4; i++) (_momForced[3 + i], (k, 2 * k + i, 3 * k + i)), + for (var a = 0; a < 5; a++) (_momForced[3], (4, a, 1)), + for (var a = 0; a < 5; a++) (_momForced[1], (4, a, 1)), + ], + cases: 200, + reach: Reach<(MomCase, Mut)>({ + for (var k = 0; k < 8; k++) 'kind: $k': .03, + }, (arg, bump) => bump('kind: ${arg.$2.$1 % 8}')), + ); + }); + + group('IntHistogram', () { + _laws.law( + 'M IntHistogram.merge: the histogram of the parts merged is exactly the ' + 'histogram of the concatenation; associative, commutative, the empty ' + 'histogram is the identity', + _histGen, + _histMerge, + examples: _histForced, + cases: 200, + reach: Reach(_histShares, _observeHist), + ); + _laws.law( + 'L1b IntHistogram: a walk of adds and removes reads exactly what the ' + 'sorted list reads; refused calls change nothing', + _histGen, + _histAddRemove, + examples: _histForced, + cases: 150, + reach: Reach(_histShares, _observeHist), + ); + _laws.law( + 'L2 IntHistogram: write(read(b)) == b, goes on identically, the counts add ' + 'up', + _histGen, + _histJson, + examples: _histForced, + cases: 150, + reach: Reach(_histShares, _observeHist), + ); + _laws.law<(MomCase, Mut)>( + 'L3 IntHistogram: a mutated checkpoint is refused whole or holds exactly ' + 'what was written', + G.pair(_histGen, mutGen(8)), + _histL3, + examples: [ + for (var k = 0; k < 8; k++) + for (var i = 0; i < 4; i++) (_histForced[3 + i], (k, 2 * k + i, 3 * k + i)), + for (var a = 0; a < 8; a++) (_histForced[4], (4, a, 1)), + ], + cases: 200, + reach: Reach<(MomCase, Mut)>({ + for (var k = 0; k < 8; k++) 'kind: $k': .03, + }, (arg, bump) => bump('kind: ${arg.$2.$1 % 8}')), + ); + }); + + group('CalculationCache', () { + _laws.law>( + 'CalculationCache equals a model of bounded LRU results with deep-compared ' + 'dependency snapshots: results, hits, computations, failures, capacity', + IntsGen([G.intIn(0, 80), G.intIn(0, 6), G.intIn(0, 1 << 12), G.intIn(0, 3)]), + _cacheModelLaw, + examples: [ + [0, 0, 1, 0], + [1, 0, 2, 0], + [40, 0, 3, 0], // capacity 1 + [60, 2, 4, 0], // capacity 3 + [80, 6, 5, 0], // capacity 7 + [80, 1, 6, 0], + // Calculations that mutate their dependencies: always (mode 5 is a + // certainty), often, now and then. + [1, 0, 7, 5], // one step, always: the smallest case + [30, 0, 8, 5], // capacity 1, always + [60, 3, 9, 3], + [80, 6, 10, 2], + ], + cases: 200, + reach: Reach>({ + 'capacity 1': .05, + 'capacity 3 or more': .3, + 'many steps': .4, + 'calculations that mutate their dependencies': .5, + }, (c, bump) { + if (c[1] == 0) bump('capacity 1'); + if (c[1] >= 2) bump('capacity 3 or more'); + if (c[0] >= 40) bump('many steps'); + if (c[3] > 0 && c[0] > 0) bump('calculations that mutate their dependencies'); + }), + ); + }); + + test('CalculationCache snapshots the dependencies before the calculation runs, ' + 'so one that mutates them leaves the entry keyed on the original value', () { + // Written out, not from the model: the contract in evaluate() is "snapshot + // before calling user code, which may itself mutate dependencies". + Map original() => { + 'q': [1, [2, 3]], + 'r': {'s': [4, 5]}, + }; + final cache = CalculationCache(); + final deps = original(); + var runs = 0; + expect(cache.evaluate>('k', deps, () { + runs++; + ((deps['q'] as List)[1] as List)[0] = 99; // nested, in place + ((deps['r'] as Map)['s'] as List).add(6); + (deps['q'] as List).add('more'); + return [runs]; + }), [1]); + expect(deps['q'], [1, [99, 3], 'more'], reason: 'the callback did mutate them'); + expect((runs, cache.computations, cache.hits), (1, 1, 0)); + // The ORIGINAL value hits the entry computed from it: no new run. + expect(cache.evaluate>('k', original(), () { + runs++; + return [-1]; + }), [1], reason: 'the original value hits'); + expect((runs, cache.computations, cache.hits), (1, 1, 1)); + // The mutated value is a different dependency: a miss, the callback runs. + expect(cache.evaluate>('k', deps, () { + runs++; + return [2]; + }), [2], reason: 'the mutated value is a miss'); + expect((runs, cache.computations, cache.hits), (2, 2, 1)); + }); + + group('reader boundaries, written out at the edge (not drawn)', () { + double nextUp(double v) { + final bits = ByteData(8)..setFloat64(0, v); + final i = bits.getInt64(0); + bits.setInt64(0, v >= 0 ? i + 1 : i - 1); + return bits.getFloat64(0); + } + + double nextDown(double v) => -nextUp(-v); + + IncrementalMinuteMetrics readMin(Json j) => + IncrementalMinuteMetrics.fromJson( + (jsonDecode(jsonEncode(j)) as Map).cast()); + + test('IncrementalMinuteMetrics: the trimp total may differ from its bills ' + 'by 1e-9 relative -- the edge itself and just inside are read, just ' + 'outside is refused', () { + // No bills: the sum is 0 and the tolerance is exactly 1e-9 * max(1, 0). + const tol = 1e-9; + Json empty(double total) => IncrementalMinuteMetrics().toJson()..['trimpTotal'] = total; + for (final sign in [1.0, -1.0]) { + for (final total in [sign * tol, sign * nextDown(tol), sign * 0.5 * tol]) { + expect(readMin(empty(total)).toJson()['trimpTotal'], total, + reason: 'a total $total from no bills is within the tolerance'); + } + for (final total in [sign * nextUp(tol), sign * 2 * tol]) { + expect(() => readMin(empty(total)), throwsFormatException, + reason: 'a total $total from no bills is outside the tolerance'); + } + } + // With bills the tolerance scales with the sum of their trimp. + final ops = MinOps(); + final x = ops.expand([61, 3, 0, 1, 1]); + final live = ops.fresh(x) as IncrementalMinuteMetrics; + ops.sync(live, x); + final j = live.toJson(); + var sum = 0.0; + for (final b in j['bills'] as List) { + sum += (b as Map)['trimp'] as num; + } + expect(sum, greaterThan(1), reason: 'the bills carry some load'); + final scale = 1e-9 * math.max(1, sum); + expect(readMin(deepCopy(j) as Json..['trimpTotal'] = sum + 0.5 * scale) + .toJson()['trimpTotal'], sum + 0.5 * scale); + expect(readMin(deepCopy(j) as Json..['trimpTotal'] = sum - 0.5 * scale) + .toJson()['trimpTotal'], sum - 0.5 * scale); + expect(() => readMin(deepCopy(j) as Json..['trimpTotal'] = sum + 2 * scale), + throwsFormatException); + expect(() => readMin(deepCopy(j) as Json..['trimpTotal'] = sum - 2 * scale), + throwsFormatException); + }); + + test('IncrementalMinuteMetrics: an HR of 0 (off-skin) is billed nothing ' + 'whatever the anchors, a negative resting HR included', () { + final ops = MinOps(); + // Config 11: resting HR -1, max HR 1; an HR of 0 would be half the reserve. + final x = MinIn([28000000, 28000001, 28000002], [0.0, 40.0, 0.0], null, 11, true); + final out = ops.sync(ops.fresh(x), x) as MinuteMetrics; + final batch = banisterTrimp(x.hr, restingHr: -1, maxHr: 1, sex: Sex.male); + expect(batch.present, isTrue); + numberClose(out.trimp.value, batch.value); + final only = MinIn([28000000], [0.0], null, 11, true); + final none = ops.sync(ops.fresh(only), only) as MinuteMetrics; + expect(none.trimp.value, 0, reason: 'only an off-skin minute: no load'); + }); + + test('IncrementalLombScargle: a cos^2 or sin^2 sum of exactly zero is read, ' + 'a negative one is refused', () { + // The sums of squares are never negative; 0 is a legal value (the sums + // only have to add up to the number of points). A reader that refuses 0 + // refuses a checkpoint the sums permit. + final ops = LombOps(); + final x = ops.expand([40, 7, 0, 0, 0]); + final live = ops.fresh(x) as IncrementalLombScargle; + ops.sync(live, x); + final j = live.toJson(); + final rows = j['sums'] as List; + final total = ((rows[2] as List)[2] as num) + ((rows[2] as List)[3] as num); + IncrementalLombScargle read(Json m) => + IncrementalLombScargle.fromJson((jsonDecode(jsonEncode(m)) as Map).cast()); + for (final which in [2, 3]) { + final ok = deepCopy(j) as Json; + final row = (ok['sums'] as List)[2] as List; + row[which] = 0.0; + row[5 - which] = total; + expect(read(ok).toJson()['sums'], ok['sums'], + reason: 'a zero in column $which is read as written'); + // Negative, with the other column making up the total: only the sign is wrong. + final bad = deepCopy(j) as Json; + final badRow = (bad['sums'] as List)[2] as List; + badRow[which] = -1e-12; + badRow[5 - which] = total + 1e-12; + expect(() => read(bad), throwsFormatException, + reason: 'a negative sum in column $which is refused'); + } + }); + + // Accepted-checkpoint continuations. The reader accepts some checkpoints a + // writer never produces (a sum changed, still self-consistent); the state + // then goes on from exactly what was written. A sync that adds nothing must + // not quietly rebuild it from the series, so these hold the altered value + // through an identical sync. + Json viaText(Json j) => (jsonDecode(jsonEncode(j)) as Map).cast(); + + // If reader hardening ever refuses this altered state, this becomes a refusal test. + test('IncrementalHrvTime: an accepted two-beat checkpoint keeps its level ' + 'moments through an identical sync', () { + final live = IncrementalHrvTime()..sync([800.0, 900.0]); + final j = viaText(live.toJson()); + expect(((j['levels'] as Map)['m2'] as num).toDouble(), 5000.0); + (j['levels'] as Map)['m2'] = 20000.0; // positive: the reader cannot tell + final back = IncrementalHrvTime.fromJson(viaText(j)); + final out = back.sync([800.0, 900.0]); + expect(out.value!.sdnn, closeTo(math.sqrt(20000.0), 1e-9), + reason: 'the SDNN is the one the state holds, not a rebuilt 70.7'); + expect(((back.toJson()['levels'] as Map)['m2'] as num).toDouble(), 20000.0); + }); + + // If reader hardening ever refuses this altered state, this becomes a refusal test. + test('IncrementalEnmoSeries: accepted altered minute sums survive an ' + 'identical sync', () { + final a = [for (var i = 0; i < 90; i++) AccelSample(i * 1000.0, .3, -.4, 1)]; + final live = IncrementalEnmoSeries()..sync(a, gRef: 1); + final j = viaText(live.toJson()); + final bin = (j['bins'] as List)[0] as Map; + final n = (bin['mags'] as List).length; + bin['enmoSum'] = 3.0; + final back = IncrementalEnmoSeries.fromJson(viaText(j)); + final out = back.sync(a, gRef: 1); + expect(out.minutes.first.enmo, closeTo(3.0 / n, 1e-12), + reason: 'the minute reads the sum the checkpoint holds'); + expect(((back.toJson()['bins'] as List)[0] as Map)['enmoSum'], 3.0); + }); + + // If reader hardening ever refuses this altered state, this becomes a refusal test. + test('IncrementalLombScargle: accepted altered spectral sums survive an ' + 'identical sync', () { + final t = [for (var i = 0; i < 40; i++) i * .8 + (i % 3) * .05]; + final y = [for (var i = 0; i < 40; i++) 800.0 + 50 * (i % 5)]; + final live = IncrementalLombScargle([.04, .1, .25]); + final batch = live.sync(t, y)!; + final j = viaText(live.toJson()); + ((j['sums'] as List)[1] as List)[5] = 1234.5; // centred y * cos: unchecked + final back = IncrementalLombScargle.fromJson(viaText(j)); + final out = back.sync(t, y)!; + expect(out.spectrum[0].psd, batch.spectrum[0].psd, reason: 'the other rows are untouched'); + expect(out.spectrum[1].psd, isNot(closeTo(batch.spectrum[1].psd, 1e-6)), + reason: 'the altered row reads the sums the checkpoint holds'); + expect(((back.toJson()['sums'] as List)[1] as List)[5], 1234.5); + }); + + test('IncrementalMinuteMetrics: a detour through reordered minutes gives ' + 'the checkpoint (by key) and the output of one sync', () { + final ops = MinOps(); + for (final r in [ + [61, 3, 0, 1, 1], + [47, 5, 1, 0, 0], + [31, 1, 0, 1, 1], + [9, 4, 6, 1, 1], + ]) { + final x = ops.expand(r); + final single = ops.fresh(x) as IncrementalMinuteMetrics; + final want = ops.sync(single, x); + for (var salt = 3; salt < 9; salt++) { + // 3..5 swap two whole minutes, 6..8 put them all in reverse. + final other = ops.edit(x, 0, salt); + expect(other.keys.toSet(), x.keys.toSet(), reason: 'the same minutes $r $salt'); + expect(other.keys, isNot(x.keys), reason: 'in another order $r $salt'); + final st = ops.fresh(x) as IncrementalMinuteMetrics; + ops.sync(st, x); + ops.close(ops.sync(st, other), other); + final back = _restore(ops, st); + final got = ops.sync(back, x); + ops.sameState(ops.toJson(back), ops.toJson(single), 'reorder $r $salt'); + ops.sameOutput(got, want, 'reorder $r $salt'); + } + } + }); + }); + + test('CalculationCache refuses a capacity under one entry', () { + for (final bad in [0, -1, -100]) { + expect(() => CalculationCache(maxEntries: bad), throwsArgumentError, + reason: 'maxEntries: $bad'); + } + expect(CalculationCache(maxEntries: 1).maxEntries, 1); + expect(CalculationCache().maxEntries, 128); + }); + + group('FINDINGS (skipped: suspected gaps against the documented contract)', () { + test('IntHistogram public overflow checks preserve state', () { + const maxInt64 = 0x7fffffffffffffff; + Map snapshot(IntHistogram h) { + final json = h.toJson(); + return { + 'count': h.count, + 'values': json['values'], + 'counts': json['counts'], + 'toJson': json, + }; + } + + final h = IntHistogram()..add(7, maxInt64); + expect(h.count, maxInt64); + expect(h.toJson()['counts'], [maxInt64]); + final before = snapshot(h); + expect(() => h.add(7), throwsStateError); + expect(snapshot(h), before); + + final other = IntHistogram()..add(8); + expect(() => h.merge(other), throwsStateError); + expect(snapshot(h), before); + + expect(() => h.merge(h), throwsStateError); + expect(snapshot(h), before); + }); + + test('IntHistogram.fromJson refuses bin counts whose total does not fit an ' + 'int', () { + final bad = >[ + {'version': 1, 'type': 'IntHistogram', 'values': [1, 2, 3], 'counts': [1 << 62, 1 << 62, 1 << 62]}, + {'version': 1, 'type': 'IntHistogram', 'values': [1, 2], 'counts': [0x7fffffffffffffff, 1]}, + ]; + for (final j in bad) { + // Today: accepted, `count` wraps negative, and a later `percentile` + // throws StateError('Order statistic out of range'), not the + // FormatException a malformed checkpoint is documented to raise. + expect(() => IntHistogram.fromJson(j), throwsFormatException, reason: '$j'); + } + }); + }); + + group('the forced scenarios say what they claim', () { + test('HRV: empty, jitter gate, constant, dropouts, several 5-minute bins, no ' + 'times, an artifact fraction', () { + final ops = HrvOps(); + var absent = false, gated = false, multiBin = false, noTimes = false; + var constant = false, gaps = false, af = false; + for (final r in ops.forced) { + final x = ops.expand(r); + final st = ops.fresh(x); + final out = ops.sync(st, x) as Metric; + final j = ops.toJson(st); + if (!out.present) absent = true; + if (out.present && out.value!.rmssd == null && out.note!.startsWith('rmssd_refused')) { + gated = true; + } + if ((j['bins'] as List).length >= 2) multiBin = true; + if (x.t == null && x.nn.length >= 2) noTimes = true; + if (r[0] == 3 && x.nn.length >= 31) constant = true; + if (x.t != null) { + for (var i = 1; i < x.nn.length; i++) { + if (x.t![i] - x.t![i - 1] > x.nn[i] + .5) gaps = true; + } + } + if (x.af > 0) af = true; + } + expect([absent, gated, multiBin, noTimes, constant, gaps, af], + everyElement(isTrue)); + }); + + test('ENMO: empty, gaps, invalid samples, duplicate and unsorted times, a ' + 'given, an auto and a non-finite reference', () { + final ops = EnmoOps(); + var empty = false, invalid = false, dup = false, unsorted = false; + var given = false, auto = false, nonFinite = false, multiMinute = false; + for (final r in ops.forced) { + final x = ops.expand(r); + final out = ops.sync(ops.fresh(x), x) as EnmoResult; + if (x.samples.isEmpty) empty = true; + if (x.samples.any((s) => !s.valid)) invalid = true; + for (var i = 1; i < x.samples.length; i++) { + if (x.samples[i].tsMs == x.samples[i - 1].tsMs) dup = true; + if (x.samples[i].tsMs < x.samples[i - 1].tsMs) unsorted = true; + } + final g = x.c.$1; + if (g == null) auto = true; + if (g != null && !g.isFinite) nonFinite = true; + if (g != null && g.isFinite) given = true; + if (out.minutes.length >= 2) multiMinute = true; + } + expect([empty, invalid, dup, unsorted, given, auto, nonFinite, multiMinute], + everyElement(isTrue)); + }); + + test('Lomb: abstentions, a spectrum, a non-finite value, a mismatch, ' + 'epoch-sized times, an empty and a repeated grid', () { + final ops = LombOps(); + var abstains = false, spectrum = false, nonFinite = false; + var mismatch = false, epoch = false, emptyGrid = false, repeated = false; + for (final r in ops.forced) { + final x = ops.expand(r); + final out = ops.sync(ops.fresh(x), x) as LombScargle?; + if (out == null) abstains = true; + if (out != null && out.spectrum.isNotEmpty) spectrum = true; + if (x.t.any((v) => !v.isFinite) || x.y.any((v) => !v.isFinite)) nonFinite = true; + if (x.t.length != x.y.length) mismatch = true; + if (x.t.isNotEmpty && x.t.first > 1e9) epoch = true; + if (x.freqs.isEmpty) emptyGrid = true; + if (x.freqs.toSet().length < x.freqs.length) repeated = true; + } + expect([abstains, spectrum, nonFinite, mismatch, epoch, emptyGrid, repeated], + everyElement(isTrue)); + }); + + test('minute metrics: no minutes, missing / inverted / NaN anchors, no ' + 'profile, a short, a DST and a long day, both artifact modes', () { + final ops = MinOps(); + var empty = false, missing = false, inverted = false, nan = false; + var noProfile = false, shortDay = false, dst = false, longDay = false; + var series = false, summary = false, energy = false, negative = false; + for (final r in ops.forced) { + final x = ops.expand(r); + final out = ops.sync(ops.fresh(x), x) as MinuteMetrics; + final c = _minCfgs[x.cfg]; + if (x.keys.isEmpty) empty = true; + if (c.$1 == null || c.$2 == null) missing = true; + if (c.$1 != null && c.$2 != null && c.$1! >= c.$2!) inverted = true; + if (c.$1 != null && c.$1!.isNaN) nan = true; + if (c.$4 == null) noProfile = true; + if (c.$5 < 1440) shortDay = true; + if (c.$5 == 1380) dst = true; + if (c.$5 > 1440) longDay = true; + if (x.series) series = true; + if (!x.series) summary = true; + if (out.energy != null) energy = true; + if (c.$1 != null && c.$1! < 0 && out.trimp.present) negative = true; + } + expect([empty, missing, inverted, nan, noProfile, shortDay, dst, longDay, series, summary, energy, negative], + everyElement(isTrue)); + }); + + test('merge: empty, one value, two, constant, large offset, wide range, ' + 'odd and even counts', () { + var empty = false, one = false, two = false, constant = false; + var offset = false, wide = false, odd = false, even = false; + for (final c in _momForced) { + if (c[1] == 0) empty = true; + if (c[1] == 1) one = true; + if (c[1] == 2) two = true; + if (c[0] == 2 && c[1] > 2) constant = true; + if (c[0] == 1) offset = true; + if (c[0] == 3) wide = true; + if (c[1] > 2 && c[1].isOdd) odd = true; + if (c[1] > 2 && c[1].isEven) even = true; + } + expect([empty, one, two, constant, offset, wide, odd, even], everyElement(isTrue)); + }); + }); + + _laws.registerReachTest(); +} diff --git a/test/properties/rr_irregular_laws_test.dart b/test/properties/rr_irregular_laws_test.dart new file mode 100644 index 0000000..240ac7d --- /dev/null +++ b/test/properties/rr_irregular_laws_test.dart @@ -0,0 +1,2693 @@ +// LAWS of the streaming RR cleaning and the irregular-rhythm screen (design 05, +// pilot cluster C2A, analytics side): `RrCorrector` (foundations/ +// rr_correction_stream.dart), `IrregularScreenState` (clinical/ +// irregular_rhythm_state.dart) and the diagnostics they report +// (`IrregularDiagnostics`, clinical/irregular_diagnostics.dart), alone and +// integrated the way the day pass uses them (corrector settles NN, the screen +// folds it, the corrector's provisional tail and counts are handed to +// `evaluateDetailed`). +// +// L1 chunk invariance: any VALID chunking of the input (empty chunks and a +// save/restore through real JSON text between chunks included) gives the +// same checkpoint TEXT, the same settled output and the same snapshot as +// one fold of the whole input. This is NOT a monoid claim (neither module +// has a `combine`); it is a law of the append transition. The documented +// refusals (mixing timestamped and timestamp-free folds, a non-finite RR +// or timestamp, a length mismatch) are asserted as refusals that leave +// the state unchanged. +// L1b streamed vs batch: at every checked seam `settled ++ snapshot` is +// `correctRr` of the prefix bit for bit (the FROZEN reference copy for +// short series, the production batch beyond the reference's budget), and +// the screen is the batch screen: counts, flags, abstentions and the +// diagnostics text exact; SD1 / SD2 / ratio within 1e-9 relative and +// confidence within 1e-12 (running sums against a two-pass batch). The +// window counts are checked against a window partition written HERE, +// independently of `irregularWindowVerdict` (which both sides share). +// L2 checkpoint round trip: `toJson` -> real JSON text -> `fromJson` -> +// `toJson` is the same text, and the restored object reads what the live +// one reads on an explicit projection and goes on identically. +// L3 a checkpoint of another type or version, a truncated, padded or +// inconsistent one, hostile counts, parts that disagree: refused whole +// with a FormatException (never another error type, never a half-built +// object). A mutation the reader cannot tell from a legal checkpoint +// (one number changed) may be accepted, but then the object holds exactly +// what was written: nothing is silently normalised. +// L4 identity: an empty chunk leaves a checkpoint unchanged (the corrector +// records NOTHING on a first empty fold, not even whether timestamps are +// used), `evaluate*` never changes the state it reads, and the other +// way round a refused fold leaves the state unchanged. +// L5 conservation, integrated, provisional NN included: nn_in == rr_raw - +// dropped; normal + corrected + dropped == beats; the windows cover the +// kept NN (independent partition); flagged <= valid <= total <= kept; an +// invalid window config gives NO window evidence and cannot flag; the +// artifact fraction is a share of the beats the corrector saw, or ABSENT +// (null, never 1.0) when it saw none (0fc5768). +// W the diagnostics wire (what edge persists): round trip through JSON +// text, refusal of another version / unknown reason / malformed map, and +// the "absent, not zero" rules for the cleaning counts. +// +// The laws describe the code at analytics 0fc57682, with upstream's screen +// rules since (f026a4e merge): a pair across a time hole is not successive, the +// beat gate and window validity count successive pairs (minBeats - 1), and the +// screen checkpoint is version 3 (it carries the previous beat's time). A failing law is first +// checked against the module's contract; only a violation of the intended +// contract is a bug, and a wrong oracle is fixed here, never in lib/. +// +// A series is a RECIPE (flavour, beats, seed, ...) expanded deterministically, +// so a failing input shrinks to a few readable integers, not to thousands of +// doubles. Beats are capped at 3,000 per series and the chunking at 6 split +// points. No clock is read. +// +// Replay a failure with the command in its report, e.g. +// PROPERTY_SEED= PROPERTY_CASE= TZ=UTC dart test \ +// test/properties/rr_irregular_laws_test.dart --plain-name '' + +import 'dart:convert'; +import 'dart:math' as math; + +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +import '../onehz/support/correct_rr_reference.dart'; +import '../onehz/support/rr_compare.dart'; +import '../support/fold_law_support.dart'; +import '../support/law_registry.dart'; +import '../support/property.dart'; + +final _laws = LawSet(); + +// ── the recipe of a beat series ───────────────────────────────────────────── + +const _flavours = [ + 'clean', // regular beats, a little jitter + 'mixed', // ectopic pairs, missed / extra beats, noise runs, dropouts + 'flagged', // irregularly irregular (AF-like) throughout + 'gappy', // a dropout every 90 beats + 'backwards', // a beat time that steps back now and then + 'artefact heavy', // many artefacts + one run of 120 (longer than the window) + 'flat', // constant RR: QD = 0, massive ties + 'outliers', // flat, with an outlier at the start, middle and end + 'edge gaps', // dropouts of exactly, one under and one over each re-anchor gap + 'hard limits', // 300 and 2000 exactly, then a too-long and a too-short beat +]; +const int _nFlavours = 10; + +/// 2025-10-09 08:53:20 UTC in ms: a whole second, as production stamps are. +const double _t0 = 1760000000000; + +/// Corrector parameters: (alpha, windowBeats, minThresholdMs, reanchorGapMs). +/// Index 0 is the default. They travel in the checkpoint. +const List<(double, int, double, double)> _corrCfgs = [ + (5.2, 91, 100, 1000), + (5.2, 31, 100, 1000), + (4.0, 61, 100, 1000), + (6.5, 91, 30, 400), + (5.2, 4, 0, 1000), + (5.2, 90, 100, 5000), + (5.2, 1, 100, 1000), // a window of one beat: no look-ahead at all + (5.2, 2, 100, 1000), + (5.2, 3, 30, 400), +]; + +RrCorrector _newCorrector(int cfg) { + final c = _corrCfgs[cfg]; + return RrCorrector( + maxArtifactRun: null, // This harness pins the uncapped pipeline layout. + alpha: c.$1, windowBeats: c.$2, minThresholdMs: c.$3, reanchorGapMs: c.$4); +} + +class _Series { + _Series(this.rr, this.ts); + final List rr, ts; + int get n => rr.length; +} + +final Map<(int, int, int), _Series> _seriesCache = {}; + +/// The beats of a recipe (cached: expansion is not the cost under test). +_Series _series(int flavour, int n, int seed) => + _seriesCache.putIfAbsent((flavour, n, seed), () { + if (_seriesCache.length > 400) _seriesCache.clear(); + return _expandSeries(flavour, n, seed); + }); + +_Series _expandSeries(int flavour, int n, int seed) { + final r = Rng(seed * 7919 + flavour * 131 + 11); + final rr = [], ts = []; + var clock = _t0; + void emit(double v) { + final e = v.roundToDouble(); + clock += e; + rr.add(e); + // End-of-beat time quantised to whole seconds (rec_ts * 1000). + ts.add(flavour == 8 ? clock : (clock / 1000).floorToDouble() * 1000); + } + + var longRun = false; + while (rr.length < n) { + final i = rr.length; + final v = 800 + 40 * math.sin(i / 70) + 12 * (r.nextDouble() - .5); + switch (flavour) { + case 1: + case 5: + final u = r.nextDouble(); + final heavy = flavour == 5; + if (heavy && !longRun && i >= n ~/ 3 && n >= 240) { + longRun = true; + for (var k = 0; k < 120; k++) { + emit(250 + r.nextInt(2100).toDouble()); + } + } else if (u < (heavy ? .08 : .02) && i > 3) { + emit(v * .6); + emit(v * 1.4); + } else if (u < (heavy ? .10 : .03)) { + emit(v * 1.95); + } else if (u < (heavy ? .12 : .04)) { + emit(v * .5); + emit(v * .5); + } else if (u < (heavy ? .14 : .046)) { + for (var k = 3 + r.nextInt(10); k > 0; k--) { + emit(250 + r.nextInt(2100).toDouble()); + } + } else if (!heavy && u < .051) { + clock += 5000 + r.nextInt(300000); + emit(v); + } else { + emit(v); + } + case 2: + emit(420 + r.nextInt(700).toDouble()); + case 3: + if (i % 90 == 89) clock += 5000 + r.nextInt(115000); + emit(v); + case 4: + emit(v); + if (i % 97 == 96) ts[ts.length - 1] -= 1000.0 * (1 + r.nextInt(3)); + case 6: + emit(800); + case 7: + emit(i == 0 || i == n ~/ 2 || i == n - 1 ? 1300 : 800); + case 9: + // The edges of the plausible range, exactly, then one beat past each: + // 300 and 2000 are plausible, 2500 and 250 are not. + emit(i < 4 ? const [300.0, 2000.0, 2500.0, 250.0][i] : v); + case 8: + // The wall-clock step exceeds the beat by exactly the re-anchor gap of + // one of the configs (400, 1000, 5000 ms), one ms under it, one over. + emit(v); + if (i % 12 == 11) { + const gaps = [400.0, 399.0, 401.0, 1000.0, 999.0, 1001.0, 5000.0, 4999.0, 5001.0]; + clock += gaps[(i ~/ 12) % gaps.length]; + ts[ts.length - 1] = clock; // (exact milliseconds, not whole seconds) + } + default: + emit(v); + } + } + if (rr.length > n) { + rr.removeRange(n, rr.length); + ts.removeRange(n, ts.length); + } + return _Series(rr, ts); +} + +// ── the recipe of a chunking ──────────────────────────────────────────────── + +/// (flavour, beats, seed, mode: 0 timestamps given, 1 none). +typedef _Rec = (int, int, int, int); + +/// (splits, cut seed, restart mask, corrector config index). +typedef _Plan = (int, int, int, int); +typedef _Case = (_Rec, _Plan); + +class _FlavourGen extends Gen { + @override + int generate(Rng r, int size) => r.nextInt(_nFlavours); + @override + Iterable shrink(int v) => G.intIn(0, _nFlavours - 1).shrink(v); +} + +/// Beat counts: the shapes that matter (empty, under 3, the window edges 45 / +/// 91 and their neighbours), then growing with the case index up to [maxN]. +class _NGen extends Gen { + _NGen(this.maxN); + final int maxN; + static const _pool = [ + 0, 1, 2, 3, 4, 5, 10, 44, 45, 46, 47, 89, 90, 91, 92, 93, 94, 135, 136, // + 137, 181, 182, 183, 272, 273, + ]; + @override + int generate(Rng r, int size) { + final p = r.nextDouble(); + if (p < .04) return 0; + if (p < .29) { + final ok = [for (final x in _pool) if (x <= maxN) x]; + return ok[r.nextInt(ok.length)]; + } + if (p < .40) return r.intIn(math.min(500, maxN), maxN); + final cap = math.max(120, maxN * size ~/ 100); + if (p < .62) return r.intIn(0, math.min(cap, 400)); + return r.intIn(0, math.min(cap, maxN)); + } + + @override + Iterable shrink(int v) => G.intIn(0, maxN).shrink(v); +} + +class _CutSeedGen extends Gen { + @override + int generate(Rng r, int size) => r.nextInt(1 << 20); + @override + Iterable shrink(int v) => G.intIn(0, 1 << 20).shrink(v); +} + +Gen<_Case> _caseGen(int maxN) => G.pair( + G.quad(_FlavourGen(), _NGen(maxN), G.intIn(0, 1 << 16), G.elements(const [0, 0, 0, 1])), + G.quad(G.intIn(0, 6), _CutSeedGen(), G.intIn(0, 127), + G.elements(const [0, 0, 0, 1, 2, 3, 4, 5, 6, 7, 8])), + ); + +const _windowEdges = [0, 1, 2, 3, 44, 45, 46, 90, 91, 92, 93, 136, 137, 182, 183]; + +/// Chunk boundaries, half of the cuts on the edges of the corrector's windows +/// (the beats where classes and output settle). +List _bounds(int n, int splits, int cutSeed) => + foldBounds(n, splits, cutSeed, special: _windowEdges); + +// ── folding ───────────────────────────────────────────────────────────────── + +String _text(RrCorrector c) => jsonEncode(c.toJson()); + +RrCorrector _restore(RrCorrector c) => RrCorrector.fromJson( + (jsonDecode(jsonEncode(c.toJson())) as Map).cast()); + +List? _tsSlice(_Series s, bool wall, int a, int b) => + wall ? s.ts.sublist(a, b) : null; + +/// What a driven corrector handed out. +class _Run { + RrCorrector c; + final nn = [], times = []; + final classes = []; + _Run(this.c); +} + +void _foldInto(_Run run, _Series s, bool wall, int a, int b) { + final out = run.c.fold(s.rr.sublist(a, b), tsMs: _tsSlice(s, wall, a, b)); + run.nn.addAll(out.nn); + run.times.addAll(out.nnTimes); + run.classes.addAll(out.classes); +} + +/// Folds [s] in the chunks [bounds]; after chunk k the corrector is restored +/// through JSON text when bit k of [restart] is set. [seam] sees every seam +/// after the chunk (and the restore) with the end of the chunk. +_Run _drive(_Series s, bool wall, int cfg, List bounds, int restart, + {void Function(_Run run, int k, int at)? seam}) { + final run = _Run(_newCorrector(cfg)); + for (var k = 0; k + 1 < bounds.length; k++) { + _foldInto(run, s, wall, bounds[k], bounds[k + 1]); + if ((restart >> (k % 7)) & 1 == 1) run.c = _restore(run.c); + seam?.call(run, k, bounds[k + 1]); + } + return run; +} + +void _sameSnapshot(RrSnapshot a, RrSnapshot b, String why) { + expect(a.n, b.n, reason: 'n $why'); + expectBitIdentical(a.tailNn, b.tailNn, '$why tailNn'); + expectBitIdentical(a.tailNnTimes, b.tailNnTimes, '$why tailNnTimes'); + expectSameClasses(a.tailClasses, b.tailClasses, '$why tailClasses'); + expect(a.normalCount, b.normalCount, reason: 'normal $why'); + expect(a.droppedCount, b.droppedCount, reason: 'dropped $why'); + expect(a.correctedCount, b.correctedCount, reason: 'corrected $why'); + expect(a.settledBeats, b.settledBeats, reason: 'settled $why'); + expect(a.classifiedBeats, b.classifiedBeats, reason: 'classified $why'); + expect(sameBits(a.cleanFraction, b.cleanFraction), isTrue, + reason: 'cleanFraction $why'); +} + +/// Case-level facts from the recipe alone (no fold), for the reach observers. +void _observeRr(_Case c, void Function(String) bump) { + final (flavour, n, _, mode) = c.$1; + final (splits, cutSeed, restart, cfg) = c.$2; + bump('flavour: ${_flavours[flavour]}'); + if (n == 0) bump('beats: none'); + if (n > 0 && n < 3) bump('beats: under 3 (the short branch)'); + if (n >= 3 && n < 91) bump('beats: inside the window (3 to 90)'); + if (n >= 91) bump('beats: past the window (91 or more)'); + if (n >= 1000) bump('beats: 1000 or more'); + if (mode == 1) bump('no timestamps'); + if (cfg != 0) bump('non-default config'); + final b = _bounds(n, splits, cutSeed); + if (b.length >= 4) bump('chunks: three or more'); + if (b.length >= 6) bump('chunks: five or more'); + if (hasEmptyChunk(b)) bump('an empty chunk'); + if (restart != 0 && b.length > 2) bump('a restore between chunks'); + if (b.any((x) => x == 91 || x == 92 || x == 45 || x == 46)) { + bump('a cut on a window edge (45 / 46 / 91 / 92)'); + } +} + +const Map _rrShares = { + 'flavour: clean': .03, + 'flavour: mixed': .03, + 'flavour: flagged': .03, + 'flavour: gappy': .03, + 'flavour: backwards': .03, + 'flavour: artefact heavy': .03, + 'flavour: flat': .03, + 'flavour: outliers': .03, + 'flavour: edge gaps': .02, + 'flavour: hard limits': .02, + 'beats: none': .01, + 'beats: under 3 (the short branch)': .015, + 'beats: inside the window (3 to 90)': .1, + 'beats: past the window (91 or more)': .4, + 'no timestamps': .08, + 'non-default config': .2, + 'chunks: three or more': .4, + 'chunks: five or more': .15, + 'an empty chunk': .05, + 'a restore between chunks': .3, + 'a cut on a window edge (45 / 46 / 91 / 92)': .1, +}; + +// ── forced scenarios ──────────────────────────────────────────────────────── + +_Case _c(int flavour, int n, + {int seed = 5, + int mode = 0, + int splits = 3, + int cutSeed = 1, + int restart = 0, + int cfg = 0}) => + ((flavour, n, seed, mode), (splits, cutSeed, restart, cfg)); + +/// Thin, flat, artefact runs, boundaries (the window edges, exactly 3 beats and +/// exactly one window), timestamp-free, every config. Every law that takes a +/// recipe meets these first. +final List<_Case> _forced = [ + _c(0, 0), // no beats at all + _c(0, 1), // one beat + _c(0, 2, restart: 127), // two: the < 3 branch + _c(0, 3, restart: 127), // exactly three + _c(9, 1), // exactly 300 ms: plausible + _c(9, 2, restart: 127), // 300 and 2000 exactly + _c(9, 3, restart: 127), // and a too-long beat: the first main-path verdict + _c(9, 4, restart: 127), // and a too-short one + _c(9, 120, splits: 4, cutSeed: 28, restart: 127), + _c(1, 5, splits: 6, restart: 127), + _c(0, 44, splits: 4, cutSeed: 3), // under the half window + _c(0, 46, splits: 5, cutSeed: 4), // the first beat whose window is complete + _c(0, 90, splits: 3, cutSeed: 5), // one short of the first settled class + _c(0, 91, splits: 3, cutSeed: 6), // exactly the first settled class + _c(0, 92, splits: 3, cutSeed: 7), + _c(6, 200, splits: 4, cutSeed: 8, restart: 127), // flat: QD = 0 + _c(7, 150, splits: 4, cutSeed: 9), // outlier at start / middle / end + _c(7, 3), // the outlier is the first and the last beat + _c(5, 400, splits: 6, cutSeed: 10, restart: 85), // artefacts + a 120 run + _c(4, 300, splits: 5, cutSeed: 11, restart: 127), // a beat time steps back + _c(3, 300, splits: 5, cutSeed: 12), // dropouts + _c(1, 350, splits: 6, cutSeed: 13, mode: 1), // no timestamps + _c(2, 400, splits: 6, cutSeed: 14, restart: 127), // irregularly irregular + _c(1, 300, splits: 6, cutSeed: 15, cfg: 1), + _c(1, 300, splits: 6, cutSeed: 16, cfg: 2, restart: 127), + _c(1, 300, splits: 6, cutSeed: 17, cfg: 3, mode: 1), + _c(1, 300, splits: 6, cutSeed: 18, cfg: 4, restart: 127), // window of 4 beats + _c(5, 300, splits: 6, cutSeed: 19, cfg: 5), // an even window of 90 + _c(8, 300, splits: 5, cutSeed: 22, restart: 127), // dropouts exactly at the gap + _c(8, 300, splits: 5, cutSeed: 23, cfg: 3, restart: 85), // gap 400 + _c(8, 300, splits: 5, cutSeed: 24, cfg: 5, restart: 127), // gap 5000 + _c(1, 200, splits: 6, cutSeed: 25, cfg: 6, restart: 127), // a window of one beat + _c(5, 200, splits: 6, cutSeed: 26, cfg: 7), // two beats + _c(1, 200, splits: 6, cutSeed: 27, cfg: 8, restart: 127), // three beats + _c(1, 1200, splits: 6, cutSeed: 20, restart: 127), // past 1000 beats + _c(1, 3000, splits: 6, cutSeed: 21, restart: 127), // the cap +]; + +/// A day with settled beats, artefacts and several windows: the base the +/// mutated-checkpoint law cuts its mutations from. +final _Case _l3Base = _forced.firstWhere( + (c) => c.$1.$1 == 1 && c.$1.$2 >= 300 && c.$1.$4 == 0 && c.$2.$4 == 0); + +// ── RC: the corrector ─────────────────────────────────────────────────────── + +String _tag(_Case c) { + final ((f, n, seed, mode), (splits, cutSeed, restart, cfg)) = c; + return '${_flavours[f]} n=$n seed=$seed ts=${mode == 0} splits=$splits ' + 'cut=$cutSeed restart=$restart cfg=$cfg'; +} + +void _l1Corrector(_Case c) { + final (rec, plan) = c; + final (flavour, n, seed, mode) = rec; + final (splits, cutSeed, restart, cfg) = plan; + final s = _series(flavour, n, seed); + final wall = mode == 0; + final tag = _tag(c); + final whole = _drive(s, wall, cfg, [0, n], 0); + final bounds = _bounds(n, splits, cutSeed); + final got = _drive(s, wall, cfg, bounds, restart); + expect(_text(got.c), _text(whole.c), + reason: '$tag bounds=$bounds: the checkpoint text is the same'); + expectBitIdentical(got.nn, whole.nn, '$tag settled nn'); + expectBitIdentical(got.times, whole.times, '$tag settled times'); + expectSameClasses(got.classes, whole.classes, '$tag settled classes'); + _sameSnapshot(got.c.snapshot(), whole.c.snapshot(), tag); +} + + +/// Largest series the frozen reference is run on in a property (it is the +/// expensive side: about 0.14 ms a beat). Beyond it the production batch is +/// the oracle: weaker (it shares the kernel with the stream), but pinned to the +/// reference by rr_correction_oracle_test.dart. +const int _refCap = 300; + +RrCorrectionResult _oracle(_Series s, bool wall, int cfg, int at) { + final p = _corrCfgs[cfg]; + final rr = s.rr.sublist(0, at), ts = wall ? s.ts.sublist(0, at) : null; + return at <= _refCap + ? correctRrReference(rr, + rrTsMs: ts, + alpha: p.$1, + windowBeats: p.$2, + minThresholdMs: p.$3, + reanchorGapMs: p.$4) + : correctRr(rr, + rrTsMs: ts, + alpha: p.$1, + windowBeats: p.$2, + minThresholdMs: p.$3, + reanchorGapMs: p.$4); +} + +void _expectOracleAt(_Run run, _Series s, bool wall, int cfg, int at, String why) { + final want = _oracle(s, wall, cfg, at); + if (at <= _refCap) { + // The production batch is the oracle beyond the reference's budget; hold it + // to the reference wherever both run, so that swap stays justified. + final p = _corrCfgs[cfg]; + expectSameCorrection( + correctRr(s.rr.sublist(0, at), + rrTsMs: wall ? s.ts.sublist(0, at) : null, + alpha: p.$1, + windowBeats: p.$2, + minThresholdMs: p.$3, + reanchorGapMs: p.$4), + want, + '$why: production batch vs frozen reference'); + } + final snap = run.c.snapshot(); + expect(snap.n, at, reason: 'snapshot.n $why'); + expect(snap.classifiedBeats, run.classes.length, + reason: 'classifiedBeats == classes handed out $why'); + expect(snap.settledBeats, run.c.settledBeats, reason: 'settledBeats $why'); + expect(snap.settledBeats, lessThanOrEqualTo(snap.classifiedBeats), + reason: 'output cannot settle before its class $why'); + expectBitIdentical([...run.nn, ...snap.tailNn], want.nn, '$why nn'); + expectBitIdentical( + [...run.times, ...snap.tailNnTimes], want.nnTimesMs, '$why times'); + expectSameClasses([...run.classes, ...snap.tailClasses], want.classes, why); + expect(snap.normalCount, + want.classes.where((k) => k == BeatClass.normal).length, + reason: 'normalCount $why'); + expect(snap.droppedCount, want.droppedCount, reason: 'dropped $why'); + expect(snap.correctedCount, want.correctedCount, reason: 'corrected $why'); + expect(sameBits(snap.cleanFraction, want.cleanFraction), isTrue, + reason: 'cleanFraction ${snap.cleanFraction} vs ${want.cleanFraction} $why'); +} + +void _l1bCorrector(_Case c) { + final (rec, plan) = c; + final (flavour, n, seed, mode) = rec; + final (splits, cutSeed, restart, cfg) = plan; + final s = _series(flavour, n, seed); + final wall = mode == 0; + final tag = _tag(c); + final bounds = _bounds(n, splits, cutSeed); + final picks = pickSeams(bounds.length - 1); + _drive(s, wall, cfg, bounds, restart, seam: (run, k, at) { + if (picks.contains(k)) { + _expectOracleAt(run, s, wall, cfg, at, '$tag seam $k at $at'); + } + }); +} + +/// What a reader can see of a corrector, as text: its parameters, its settled +/// edge and its snapshot (every field). +String _projectCorrector(RrCorrector c) { + final s = c.snapshot(); + return jsonEncode([ + c.alpha, + c.windowBeats, + c.minThresholdMs, + c.reanchorGapMs, + c.settledBeats, + s.n, + s.tailNn, + s.tailNnTimes, + [for (final k in s.tailClasses) k.index], + s.normalCount, + s.droppedCount, + s.correctedCount, + s.settledBeats, + s.classifiedBeats, + s.cleanFraction, + ]); +} + +void _l2Corrector(_Case c) { + final (rec, plan) = c; + final (flavour, n, seed, mode) = rec; + final (splits, cutSeed, _, cfg) = plan; + final s = _series(flavour, n, seed); + final wall = mode == 0; + final tag = _tag(c); + final bounds = _bounds(n, splits, cutSeed); + // Fold the first part, copy the state through JSON text, fold the rest into + // both: the copy reads the same and goes on identically. + final at = bounds.length > 2 ? bounds[1] : n ~/ 2; + final live = _Run(_newCorrector(cfg)); + _foldInto(live, s, wall, 0, at); + final text = _text(live.c); + final map = (jsonDecode(text) as Map).cast(); + final back = _Run(RrCorrector.fromJson(map)); + expect(_text(back.c), text, reason: '$tag: write(read(b)) == b'); + expect(_projectCorrector(back.c), _projectCorrector(live.c), + reason: '$tag: the restored one reads what the live one reads'); + // The restored object owns its state: the map it was read from is not shared. + for (final k in ['rr', 't', 'd', 'fcls', 'lastNormals']) { + (map[k] as List).clear(); + } + expect(_text(back.c), text, reason: '$tag: not aliased to the map'); + final ownSettled = live.nn.length; + _foldInto(live, s, wall, at, n); + _foldInto(back, s, wall, at, n); + expectBitIdentical(back.nn, live.nn.sublist(ownSettled), '$tag nn after restore'); + expectBitIdentical(back.times, live.times.sublist(ownSettled), '$tag times after restore'); + expectSameClasses(back.classes, live.classes.sublist(live.classes.length - back.classes.length), tag); + expect(_text(back.c), _text(live.c), reason: '$tag: same checkpoint after the rest'); + expect(_projectCorrector(back.c), _projectCorrector(live.c), reason: tag); +} + +// ── L3: mutated checkpoints ───────────────────────────────────────────────── + +/// Applies mutation [m] to a copy of [src] (a decoded checkpoint). Returns the +/// mutated map and what it may do. The "must refuse" cases are the ones that +/// make the checkpoint contradict itself or its type; they are chosen from what +/// a reachable state can never look like, not from the reader's code. +(Map, Must) _mutateCorrector(Map src, Mut m) { + final j = (deepCopy(src) as Map).cast(); + final (kind, a, b) = m; + final keys = j.keys.toList(); + List l(String k) => j[k] as List; + switch (kind % 11) { + case 0: + const versions = [0, 1, 3, -1, 99, '2', null, true, 1 << 40, 'v2']; + j['version'] = versions[a % versions.length]; + return (j, Must.refuse); + case 1: + const types = ['IrregularScreenState', 'Other', null, '', 'rrcorrector', 1]; + j['type'] = types[a % types.length]; + return (j, Must.refuse); + case 2: + // A key that every real checkpoint has, gone ('wall' may legally be null). + final k = keys.where((k) => k != 'wall').toList()[a % (keys.length - 1)]; + j.remove(k); + return (j, Must.refuse); + case 3: + // The wrong kind of value under a key. + final k = keys.where((k) => k != 'wall').toList()[a % (keys.length - 1)]; + const junk = ['x', {}, [[]]]; + j[k] = junk[b % junk.length]; + return (j, Must.refuse); + case 4: + // The buffered arrays disagree with each other or with n - off. + final k = const ['rr', 't', 'd'][a % 3]; + final arr = l(k); + switch (b % 3) { + case 0: + if (arr.isEmpty) { + arr.add(1.0); + } else { + arr.removeLast(); + } + case 1: + arr.add(arr.isEmpty ? 1.0 : arr.last); + default: + arr.clear(); + if (j['n'] == j['off']) arr.add(1.0); + } + return (j, Must.refuse); + case 5: + // The counters contradict their own order off <= ce <= c2 <= c1 <= n. + switch (a % 6) { + case 0: + j['ce'] = (j['c2'] as int) + 1; + case 1: + j['c2'] = (j['c1'] as int) + 1; + case 2: + j['c1'] = (j['n'] as int) + 1; + case 3: + j['off'] = -1; + case 4: + j['n'] = (j['n'] as int) + 1 + b % 3; + default: + j['n'] = (j['n'] as int) - 1 - b % 3; + } + return (j, Must.refuse); + case 6: + // The class buffer: wrong length, or a class that is not one. + final f = l('fcls'); + switch (a % 3) { + case 0: + f.add(0); + case 1: + if (f.isEmpty) { + f.add(5); + } else { + f[b % f.length] = const [5, -1, 99, 1 << 40][b % 4]; + } + default: + if (f.isEmpty) { + f.add(0); + } else { + f.removeLast(); + } + } + return (j, Must.refuse); + case 7: + // The carried state: more than two normals, a class that is not one. + if (a.isEven) { + l('lastNormals') + ..add(800.0) + ..add(801.0) + ..add(802.0); + } else { + j['lastFinal'] = const [5, -1, 99, 1 << 40][b % 4]; + } + return (j, Must.refuse); + case 8: + // Hostile sizes: the checkpoint says it holds far more than it does. + const huge = [1 << 40, 1 << 62, 0x7fffffffffffffff, -1]; + j[const ['n', 'off', 'c1', 'c2', 'ce'][a % 5]] = huge[b % huge.length]; + return (j, Must.refuse); + case 9: + // One number changed: the reader cannot tell it from a legal checkpoint. + switch (a % 9) { + case 0: + if (l('rr').isNotEmpty) l('rr')[b % l('rr').length] = (l('rr')[b % l('rr').length] as num) + 1.0; + case 1: + if (l('t').isNotEmpty) l('t')[b % l('t').length] = (l('t')[b % l('t').length] as num) + 1.0; + case 2: + if (l('d').isNotEmpty) l('d')[b % l('d').length] = (l('d')[b % l('d').length] as num) + 1.0; + case 3: + j['alpha'] = (j['alpha'] as num) + 1.0; + case 4: + j['minThresholdMs'] = (j['minThresholdMs'] as num) + 1.0; + case 5: + j['reanchorGapMs'] = (j['reanchorGapMs'] as num) + 1.0; + case 6: + j['dropped'] = (j['dropped'] as int) + 1; + case 7: + j['normalFinal'] = (j['normalFinal'] as int) + 1; + default: + j['wall'] = !(j['wall'] as bool? ?? false); + } + return (j, Must.faithful); + default: + // A key this version does not know. + j['futureField${a % 3}'] = const [1, 'x', [1]][b % 3]; + return (j, Must.ignored); + } +} + +String _checkpointOutcome(Map j) { + try { + final c = RrCorrector.fromJson((deepCopy(j) as Map).cast()); + return 'accepted:${jsonEncode(c.toJson())}'; + } on FormatException { + return 'refused'; + } catch (e) { + return 'WRONG ERROR TYPE: ${e.runtimeType}: $e'; + } +} + +final Gen _mutGen = mutGen(11); + +void _l3Corrector((_Case, Mut) arg) { + final (c, m) = arg; + final (rec, plan) = c; + final (flavour, n, seed, mode) = rec; + final cfg = plan.$4; + final s = _series(flavour, n, seed); + final tag = '${_tag(c)} mutation=${m.$1 % 11}(${m.$2},${m.$3})'; + final run = _drive(s, mode == 0, cfg, [0, n], 0); + final src = (jsonDecode(_text(run.c)) as Map).cast(); + final (bad, must) = _mutateCorrector(src, m); + expectMutationOutcome(_checkpointOutcome(bad), must, jsonEncode(bad), + jsonEncode(src), tag); +} + +// ── L4 / refusals ─────────────────────────────────────────────────────────── + +void _l4Corrector(_Case c) { + final (rec, plan) = c; + final (flavour, n, seed, mode) = rec; + final (_, _, _, cfg) = plan; + final s = _series(flavour, n, seed); + final wall = mode == 0; + final tag = _tag(c); + // Identity: an empty chunk, with or without timestamps, at the start (before + // anything records the mode), in the middle and at the end. + final fresh = _text(_newCorrector(cfg)); + for (final withTs in [true, false]) { + final f = _newCorrector(cfg); + final out = f.fold(const [], tsMs: withTs ? const [] : null); + expect(out.nn, isEmpty, reason: tag); + expect(out.nnTimes, isEmpty, reason: tag); + expect(out.classes, isEmpty, reason: tag); + expect(_text(f), fresh, + reason: '$tag: an empty first fold records nothing (not even the mode)'); + } + final cut = n ~/ 2; + final run = _Run(_newCorrector(cfg)); + // An empty fold before the first beat must not decide whether timestamps are + // used: the series still folds in either mode and reads as if it never ran. + run.c.fold(const [], tsMs: wall ? null : const []); + _foldInto(run, s, wall, 0, cut); + final before = _text(run.c); + for (final withTs in [true, false]) { + final out = run.c.fold(const [], tsMs: withTs ? const [] : null); + expect(out.nn.length + out.nnTimes.length + out.classes.length, 0, reason: tag); + expect(_text(run.c), before, reason: '$tag: an empty chunk is the identity'); + } + // snapshot() reads and changes nothing. + final one = _projectCorrector(run.c); + expect(_text(run.c), before, reason: '$tag: snapshot() is read-only'); + expect(_projectCorrector(run.c), one, reason: '$tag: twice is equal'); + // Refusals leave the state exactly as it was. + if (cut > 0) { + final good = s.rr.sublist(cut, math.min(n, cut + 3)); + final goodTs = s.ts.sublist(cut, math.min(n, cut + 3)); + if (good.isNotEmpty) { + // The other mode. + expect(() => run.c.fold(good, tsMs: wall ? null : goodTs), throwsStateError, + reason: '$tag: timestamps on every fold or on none'); + expect(_text(run.c), before, reason: '$tag: unchanged by the mixed fold'); + // A length mismatch (timestamps on). + if (wall) { + expect(() => run.c.fold(good, tsMs: goodTs.sublist(0, goodTs.length - 1)), + throwsArgumentError, reason: tag); + expect(_text(run.c), before, reason: '$tag: unchanged by the mismatch'); + } + // A non-finite value AFTER good beats in the same chunk must not stick. + for (final bad in [double.nan, double.infinity, double.negativeInfinity]) { + final rr = [...good, bad], ts = [...goodTs, goodTs.last + 1000]; + expect(() => run.c.fold(rr, tsMs: wall ? ts : null), throwsArgumentError, + reason: '$tag: non-finite RR $bad'); + expect(_text(run.c), before, reason: '$tag: unchanged by $bad'); + if (wall) { + final badTs = [...goodTs, bad]; + expect(() => run.c.fold([...good, 800], tsMs: badTs), throwsArgumentError, + reason: '$tag: non-finite timestamp $bad'); + expect(_text(run.c), before, reason: '$tag: unchanged by timestamp $bad'); + } + } + } + } +} + +// ── L5: conservation ──────────────────────────────────────────────────────── + +void _checkConserved(RrSnapshot snap, _Run run, int cfg, String why) { + final n = snap.n; + // The class of a beat is final once the window two thresholds deep (2h beats + // each side) is in the data (header of rr_correction_stream.dart); under 3 + // beats nothing is. + final h = _corrCfgs[cfg].$2 ~/ 2; + expect(snap.classifiedBeats, n < 3 ? 0 : math.max(0, n - 2 * h), + reason: 'classes settle exactly 2h beats behind the newest $why'); + final nnTotal = run.nn.length + snap.tailNn.length; + expect(nnTotal, n - snap.droppedCount, + reason: 'nn_in == rr_raw - dropped $why'); + expect(snap.normalCount + snap.correctedCount + snap.droppedCount, n, + reason: 'every beat is kept, corrected or dropped, once $why'); + expect(snap.droppedCount >= 0 && snap.correctedCount >= 0, isTrue, reason: why); + expect(run.classes.length + snap.tailClasses.length, n, + reason: 'a class for every beat $why'); + final normals = [...run.classes, ...snap.tailClasses] + .where((k) => k == BeatClass.normal) + .length; + expect(snap.normalCount, normals, reason: 'normal count is the class count $why'); + expect(snap.settledBeats <= snap.classifiedBeats && snap.classifiedBeats <= n, + isTrue, + reason: 'settled <= classified <= folded $why'); + expect(snap.cleanFraction, n == 0 ? 0.0 : snap.normalCount / n, + reason: 'cleanFraction is a share of the beats, 0 on none, never NaN $why'); + expect(snap.cleanFraction >= 0 && snap.cleanFraction <= 1, isTrue, reason: why); + expect(run.nn.length, run.times.length, reason: why); + expect(snap.tailNn.length, snap.tailNnTimes.length, reason: why); + final all = [...run.times, ...snap.tailNnTimes]; + for (var i = 1; i < all.length; i++) { + if (all[i] < all[i - 1]) { + fail('beat times never run backwards, whatever the stamps do: ' + '${all[i - 1]} then ${all[i]} at $i $why'); + } + } +} + +void _l5Corrector(_Case c) { + final (rec, plan) = c; + final (flavour, n, seed, mode) = rec; + final (splits, cutSeed, restart, cfg) = plan; + final s = _series(flavour, n, seed); + final tag = _tag(c); + final bounds = _bounds(n, splits, cutSeed); + _drive(s, mode == 0, cfg, bounds, restart, seam: (run, k, at) { + _checkConserved(run.c.snapshot(), run, cfg, '$tag seam $k at $at'); + }); +} + +// ── SC: the screen state, folding a corrected NN series directly ──────────── + +const _nnFlavours = [ + 'sinus', // regular, slow drift + 'af', // irregularly irregular throughout + 'blocks', // blocks of AF and sinus, 5 to 30 minutes + 'flat', // constant + 'two point', // alternating 800 / 1000 + 'sinus + gaps', // sinus with dropouts (time jumps) + 'af heavy', // AF in most blocks + 'range edges', // 300 and 2000 exactly (kept), 299 and 2001 (not), then calm beats +]; +const int _nNnFlavours = 8; + +/// (out-of-range share, NaN share) of the entries. +const List<(double, double)> _salts = [(0, 0), (.02, 0), (.2, .02), (0, .02)]; + +class _Nn { + _Nn(this.nn, this.t); + final List nn, t; + int get n => nn.length; +} + +final Map<(int, int, int, int), _Nn> _nnCache = {}; + +_Nn _nnSeries(int flavour, int n, int seed, int salt) => + _nnCache.putIfAbsent((flavour, n, seed, salt), () { + if (_nnCache.length > 400) _nnCache.clear(); + return _expandNn(flavour, n, seed, salt); + }); + +_Nn _expandNn(int flavour, int n, int seed, int salt) { + final r = Rng(seed * 104729 + flavour * 31 + salt + 3); + final (outShare, nanShare) = _salts[salt]; + final nn = [], t = []; + var clock = 0.0; + var af = false; + var left = 0; + for (var i = 0; i < n; i++) { + if (--left <= 0) { + af = switch (flavour) { + 1 => true, + 2 => r.nextBool(.4), + 6 => r.nextBool(.8), + _ => false, + }; + left = 300 + r.nextInt(2000); + } + var v = switch (flavour) { + 3 => 800.0, + 4 => i.isEven ? 800.0 : 1000.0, + 7 => const [300.0, 2000.0, 299.0, 2001.0, 800.0, 1000.0, 850.0][i % 7], + _ => af + ? (420 + r.nextInt(700)).toDouble() + : (850 + 40 * math.sin(i / 90) + 25 * (r.nextDouble() - .5)).roundToDouble(), + }; + var step = v; + final u = r.nextDouble(); + if (u < nanShare) { + v = double.nan; + step = 800; + } else if (u < nanShare + outShare) { + v = r.nextBool() ? 2500 : 250; + step = v; + } + if (flavour == 5 && i % 211 == 210) clock += 40000 + r.nextInt(300000); + clock += step; + nn.add(v); + t.add(clock); + } + return _Nn(nn, t); +} + +/// (flavour, beats, seed, salt). +typedef _NnRec = (int, int, int, int); + +/// Screen parameters: (sd1sd2Flag, pnnThresholdMs, pnnFlagPct, windowMinutes, +/// minWindowBeats, sustainedFraction). The last five are invalid window +/// configs: the batch fails CLOSED on them and the state keeps no windows. +const List<(double, double, double, double, int, double)> _scCfgs = [ + (.70, 70, 30, 5, 40, .5), + (.70, 70, 30, 1, 10, .5), + (.50, 50, 20, 2, 2, 0.0), + (.70, 70, 30, 5, 40, 1.0), + (.70, 70, 30, 0, 40, .5), // invalid: no window length + (.70, 70, 30, 5, 1, .5), // invalid: windows of fewer than 2 beats + (.70, 70, 30, 5, 40, 1.5), // invalid: a fraction above 1 + (.70, 70, 30, -5, 40, -.1), // invalid: two at once + (.90, 100, 60, 10, 100, .25), +]; +bool _scCfgValid(int i) { + final c = _scCfgs[i]; + return c.$4 > 0 && c.$5 >= 2 && c.$6 >= 0 && c.$6 <= 1; +} + +IrregularScreenState _newState(int cfg) { + final c = _scCfgs[cfg]; + return IrregularScreenState( + sd1sd2Flag: c.$1, + pnnThresholdMs: c.$2, + pnnFlagPct: c.$3, + windowMinutes: c.$4, + minWindowBeats: c.$5, + sustainedFraction: c.$6); +} + +/// What `evaluateDetailed` is asked: (minBeats, maxArtifact, artifactFraction, +/// cleaning: 0 none, 1 raw=beats+3, 2 raw=0). +const List<(int, double, double, int)> _evals = [ + (500, .30, 0.0, 1), + (10, .30, 0.0, 1), + (40, .30, .07, 0), + (100, .05, .31, 1), + (2, 1.0, 1.0, 1), + (20, .30, .29, 0), + (1, .30, 0.0, 2), + (30, .30, .30, 1), // the artifact gate is strict: exactly the maximum passes +]; + +RrCleaningCounts? _cleaningFor(int kind, int beats) => switch (kind) { + 0 => null, + 1 => RrCleaningCounts(raw: beats + 3, corrected: 2, dropped: 3), + _ => const RrCleaningCounts(raw: 0, corrected: 0, dropped: 0), + }; + +IrregularScreenResult _evaluate(IrregularScreenState st, int ev, + {List tailNn = const [], List tailT = const [], int beats = 0}) { + final e = _evals[ev]; + return st.evaluateDetailed(tailNn, tailT, + artifactFraction: e.$3, + minBeats: e.$1, + maxArtifact: e.$2, + cleaning: _cleaningFor(e.$4, beats)); +} + +IrregularScreenResult _batchScreenOf(List nn, List t, int cfg, int ev) { + final c = _scCfgs[cfg]; + final e = _evals[ev]; + return irregularBeatScreenDetailed(nn, + nnTimesMs: t, + artifactFraction: e.$3, + minBeats: e.$1, + maxArtifact: e.$2, + sd1sd2Flag: c.$1, + pnnThresholdMs: c.$2, + pnnFlagPct: c.$3, + windowMinutes: c.$4, + minWindowBeats: c.$5, + sustainedFraction: c.$6, + cleaning: _cleaningFor(e.$4, nn.length)); +} + +String _stext(IrregularScreenState s) => jsonEncode(s.toJson()); + +IrregularScreenState _srestore(IrregularScreenState s) => + IrregularScreenState.fromJson( + (jsonDecode(jsonEncode(s.toJson())) as Map).cast()); + +void _close(double got, double want, String why, {double rel = 1e-9}) { + final tol = rel * math.max(1.0, want.abs()); + if (!((got - want).abs() <= tol)) { + fail('$why: got $got want $want (tolerance $tol)'); + } +} + +/// Whether input beat [i] and the one before it form a successive pair, from +/// the documented rule alone: both kept (300..2000 ms), neighbours in the input +/// and, when beat times are given, no time hole between them (the elapsed time +/// is at most the interval, +0.5 ms of rounding; a NaN time is never within). +bool _successive(List nn, List? t, int i) { + bool kept(double v) => v >= 300 && v <= 2000; + return i > 0 && + kept(nn[i]) && + kept(nn[i - 1]) && + (t == null || t[i] - t[i - 1] <= nn[i] + 0.5); +} + +/// The kept beats of [nn] (300..2000 ms) and the successive differences taken +/// only between two successive beats (see [_successive]), from the documented +/// rule alone. +({List kept, List diffs}) _keptAndDiffs( + List nn, List? t) { + bool kept(double v) => v >= 300 && v <= 2000; + return ( + kept: [for (final v in nn) if (kept(v)) v], + diffs: [ + for (var i = 1; i < nn.length; i++) + if (_successive(nn, t, i)) nn[i] - nn[i - 1] + ], + ); +} + +/// The successive pairs the screen gates on (its minimum is minBeats - 1). +int _pairsOf(List nn, List? t) => _keptAndDiffs(nn, t).diffs.length; + +/// Whether SD2 is numerically zero for [nn]: 2*SDNN^2 - SD1^2 is within 1e-9 of +/// its own scale of nothing (an exactly two-valued alternating series is the +/// plain case). There the sign of float noise decides "no long-term +/// variability" versus a tiny SD2 and a ratio of 1e8, and running sums and a +/// two-pass batch legitimately disagree. Computed independently, from the +/// documented rule. +bool _sd2Degenerate(List nn, List? t) { + final k = _keptAndDiffs(nn, t); + final sdnn = _sd(k.kept), sdsd = _sd(k.diffs); + if (sdnn == null || sdsd == null || sdnn == 0) return false; + final v = 2 * sdnn * sdnn - sdsd * sdsd / 2; + return v.abs() <= 1e-9 * 2 * sdnn * sdnn; +} + +/// A reported SD2 is "numerically zero" when sd2^2 is within the same 1e-9 of +/// the scale 2*SDNN^2 (= sd1^2 + sd2^2) that [_sd2Degenerate] uses. The bound +/// is the float noise of the cancellation 2*SDNN^2 - SD1^2: two sums of squares +/// of the same magnitude differ by a few ulps (about 1e-15 relative, times a +/// factor that grows with the beat count, well under 1e-12 at the 3,000 beats +/// the generators cap at), so 1e-9 leaves a margin of more than 1e3 and +/// still rejects any SD2 that is a real fraction of SD1. +void _expectNumericallyZeroSd2(IrregularRhythm r, String why) { + expect(r.sd2.isFinite && r.sd2 >= 0, isTrue, reason: 'sd2 finite, >= 0 $why'); + expect(r.sd2 * r.sd2 <= 1e-9 * (r.sd1 * r.sd1 + r.sd2 * r.sd2), isTrue, + reason: 'a reported SD2 where the other side abstains is numerically ' + 'zero (sd1 ${r.sd1}, sd2 ${r.sd2}) $why'); +} + +/// Counts, flags, abstentions, notes and the diagnostics text exact; SD1 / SD2 / +/// ratio within 1e-9 relative, confidence within 1e-12 (running sums against a +/// two-pass batch). +/// +/// Pass [nn] (the beats the screen saw) to allow the one ambiguity of a +/// numerically zero SD2 (see [_sd2Degenerate]): the two sides may disagree on +/// whether SD2 is zero. That exempts ONLY the SD2 presence and what hangs on +/// it: SD2 and the ratio (both sides, when both report: each only has to be +/// numerically zero), and, when exactly one side reports, the note and the +/// flag, because the abstaining side has nothing to compare them with. +/// Everything else is still held: tier, inputs, every diagnostics field but the +/// abstention reason, SD1 (against an independent value when only one side +/// reports), the beat count, pNN and confidence (each against an independent +/// value when only one side reports, so [pnnThresholdMs] and the screen's +/// [artifactFraction] are passed along with [nn]). When +/// exactly one side reports, the other must abstain with +/// `noLongTermVariability` and the reported SD2 must be numerically zero. +void _sameScreen(IrregularScreenResult got, IrregularScreenResult want, String why, + {List? nn, + List? t, + double? pnnThresholdMs, + double? artifactFraction}) { + assert(nn == null || (pnnThresholdMs != null && artifactFraction != null), + 'the independent values need the pNN threshold and the artifact fraction'); + final g = got.metric, w = want.metric; + final degenerate = nn != null && _sd2Degenerate(nn, t); + final ambiguous = degenerate && g.present != w.present; + String diagText(IrregularScreenResult r, {bool noAbstain = false}) { + final j = r.diagnostics.toJson(); + if (noAbstain) j['abstain'] = null; + return jsonEncode(j); + } + + expect(g.tier, w.tier, reason: 'tier $why'); + expect(g.inputs_used, w.inputs_used, reason: 'inputs $why'); + expect(diagText(got, noAbstain: ambiguous), diagText(want, noAbstain: ambiguous), + reason: 'diagnostics$why${ambiguous ? " (but the abstention reason)" : ""}'); + if (ambiguous) { + final rep = g.present ? got : want, abs = g.present ? want : got; + expect(abs.diagnostics.abstain, IrregularAbstain.noLongTermVariability, + reason: 'the only disagreement allowed at a numerically zero SD2 $why'); + expect(rep.diagnostics.abstain, isNull, reason: 'a reported screen did not abstain $why'); + expect(abs.metric.value, isNull, reason: 'absent => no value $why'); + expect(abs.metric.confidence, 0, reason: 'absent => confidence 0 $why'); + final v = rep.metric.value!; + _expectNumericallyZeroSd2(v, why); + final k = _keptAndDiffs(nn, t); + expect(v.nBeats, k.kept.length, reason: 'nBeats $why'); + expect(v.nBeats, rep.diagnostics.nnKept, reason: 'nBeats is the kept count $why'); + _close(v.sd1, _sd(k.diffs)! / math.sqrt2, 'sd1 against the independent value $why'); + // pNN and confidence do not depend on SD2: held to their documented values. + final over = k.diffs.where((d) => d.abs() > pnnThresholdMs!).length; + expect(v.pnnPct, k.diffs.isEmpty ? 0.0 : 100.0 * over / k.diffs.length, + reason: 'pnn against the independent value $why'); + _close(rep.metric.confidence, + (k.kept.length / 5000.0 * (1 - artifactFraction!)).clamp(0.2, 0.9), + 'confidence against the independent value $why', + rel: 1e-12); + return; + } + expect(g.present, w.present, reason: 'present $why'); + expect(g.note, w.note, reason: 'note $why'); + if (!w.present) { + expect(g.value, isNull, reason: 'absent => no value $why'); + expect(g.confidence, 0, reason: 'absent => confidence 0 $why'); + return; + } + final a = g.value!, b = w.value!; + expect(a.flag, b.flag, reason: 'flag $why'); + expect(a.nBeats, b.nBeats, reason: 'nBeats $why'); + expect(a.pnnPct, b.pnnPct, reason: 'pnn $why'); + _close(a.sd1, b.sd1, 'sd1 $why'); + if (degenerate) { + // Both report: SD2 and the ratio are float noise around zero on both sides. + _expectNumericallyZeroSd2(a, why); + _expectNumericallyZeroSd2(b, why); + } else { + _close(a.sd2, b.sd2, 'sd2 $why'); + _close(a.sd1sd2, b.sd1sd2, 'sd1sd2 $why'); + } + _close(g.confidence, w.confidence, 'confidence $why', rel: 1e-12); +} + +// The independent window oracle: written from the documented rule, not from +// `irregularWindowVerdict` (which the batch and the stream share). + +double? _sd(List x) { + if (x.length < 2) return null; + final m = x.reduce((a, b) => a + b) / x.length; + var ss = 0.0; + for (final v in x) { + ss += (v - m) * (v - m); + } + return math.sqrt(ss / (x.length - 1)); +} + +/// Verdict of one closed window of kept beats: null when it holds fewer than +/// [minBeats] - 1 successive differences (validity counts pairs, not beats); +/// else whether SD1/SD2 and pNNx both clear their flag lines. A difference is +/// taken only between two successive beats (never across a skipped beat or a +/// time hole). +bool? _refVerdict(List v, List adj, double flag, double pnnMs, + double pnnPct, int minBeats) { + final d = [ + for (var i = 1; i < v.length; i++) + if (adj[i]) v[i] - v[i - 1] + ]; + if (d.length < minBeats - 1) return null; + final sdsd = _sd(d), sdnn = _sd(v); + if (sdsd == null || sdnn == null) return false; + final sd1 = sdsd / math.sqrt2; + final q = 2 * sdnn * sdnn - sd1 * sd1; + final sd2 = q > 0 ? math.sqrt(q) : 0.0; + if (sd2 <= 0) return false; + final over = d.where((x) => x.abs() > pnnMs).length; + return sd1 / sd2 >= flag && 100.0 * over / d.length >= pnnPct; +} + +typedef _RefWindows = ({ + int total, + int valid, + int flagged, + int openBeats, + IrregularOpenWindow open, + int kept, + int nIn +}); + +/// The windows of [nn] / [t] (the whole input, kept or not) under screen +/// config [cfg]: a window opens on its first kept beat and closes when a kept +/// beat is a full window later; the sizes cover the kept beats exactly once. +_RefWindows _refWindows(List nn, List t, int cfg) { + final c = _scCfgs[cfg]; + final windowMs = c.$4 * 60000; + final sizes = []; + final verdicts = []; + var vals = [], adj = []; + double? start; + var kept = 0; + void close() { + if (vals.isEmpty) return; + sizes.add(vals.length); + verdicts.add(_refVerdict(vals, adj, c.$1, c.$2, c.$3, c.$5)); + vals = []; + adj = []; + } + + for (var i = 0; i < nn.length; i++) { + final isKept = nn[i] >= 300 && nn[i] <= 2000; + if (isKept) { + kept++; + start ??= t[i]; + if (t[i] - start >= windowMs) { + close(); + start = t[i]; + } + vals.add(nn[i]); + adj.add(_successive(nn, t, i)); + } + } + close(); + expect(sizes.fold(0, (a, b) => a + b), kept, + reason: 'the windows cover the kept NN exactly once'); + final valid = verdicts.where((v) => v != null).length; + return ( + total: sizes.length, + valid: valid, + flagged: verdicts.where((v) => v == true).length, + openBeats: sizes.isEmpty ? 0 : sizes.last, + open: sizes.isEmpty + ? IrregularOpenWindow.none + : verdicts.last == null + ? IrregularOpenWindow.thin + : verdicts.last! + ? IrregularOpenWindow.flagged + : IrregularOpenWindow.unflagged, + kept: kept, + nIn: nn.length, + ); +} + +/// The diagnostics' beat and window evidence against the independent oracle. +void _expectEvidence(IrregularScreenResult res, List nn, List t, + int cfg, String why) { + final dg = res.diagnostics; + final ref = _refWindows(nn, t, cfg); + expect(dg.nnIn, ref.nIn, reason: 'nn_in is every entry handed in $why'); + expect(dg.nnKept, ref.kept, reason: 'nn_kept is the in-range ones $why'); + final w = dg.windows; + if (!_scCfgValid(cfg)) { + expect(w, isNull, reason: 'an invalid config gives no window evidence $why'); + if (res.metric.present) { + expect(res.metric.value!.flag, isFalse, reason: 'and cannot flag $why'); + } + return; + } + expect(w, isNotNull, reason: 'a valid config counts windows $why'); + expect(w!.total, ref.total, reason: 'windows total $why'); + expect(w.valid, ref.valid, reason: 'windows valid $why'); + expect(w.flagged, ref.flagged, reason: 'windows flagged $why'); + expect(w.openBeats, ref.openBeats, reason: 'open window beats $why'); + expect(w.open, ref.open, reason: 'open window label $why'); + expect(w.flagged >= 0 && w.flagged <= w.valid && w.valid <= w.total, isTrue, + reason: 'flagged <= valid <= total $why'); + expect(w.total <= dg.nnKept, isTrue, reason: 'total <= kept $why'); + expect(w.total == 0, dg.nnKept == 0, reason: 'no windows <=> no kept NN $why'); + expect(w.valid * _scCfgs[cfg].$5 <= dg.nnKept, isTrue, + reason: 'valid windows are full $why'); + expect(w.sustainedObserved, w.valid == 0 ? isNull : w.flagged / w.valid, + reason: 'flagged / valid, absent (not 0) when no window is valid $why'); + final m = res.metric; + if (m.present && m.value!.flag) { + expect(w.valid > 0 && w.flagged >= _scCfgs[cfg].$6 * w.valid, isTrue, + reason: 'a flag is backed by enough flagged windows $why'); + } + expect(m.present, dg.abstain == null, reason: 'present <=> not abstained $why'); +} + +typedef _NnCase = ((int, int, int, int), (int, int, int, int)); + +/// ((flavour, beats, seed, salt), (cfg, splits + cut seed, restart mask, +/// evaluation)). +Gen<_NnCase> _nnCaseGen(int maxN) => G.pair( + G.quad(G.intIn(0, _nNnFlavours - 1), _NGen(maxN), G.intIn(0, 1 << 16), + G.elements(const [0, 0, 1, 2, 3])), + G.quad(G.intIn(0, _scCfgs.length - 1), G.intIn(0, 6 * (1 << 20) + (1 << 20)), + G.intIn(0, 127), G.intIn(0, _evals.length - 1)), + ); + +String _nnTag(_NnCase c) { + final ((f, n, seed, salt), (cfg, sc, restart, ev)) = c; + return '${_nnFlavours[f]} n=$n seed=$seed salt=$salt cfg=$cfg ' + 'splits/cut=${sc ~/ (1 << 20)}/${sc % (1 << 20)} restart=$restart eval=$ev'; +} + +List _nnBounds(_NnCase c) { + final n = c.$1.$2; + final sc = c.$2.$2; + return _bounds(n, sc ~/ (1 << 20), sc % (1 << 20)); +} + +void _observeNn(_NnCase c, void Function(String) bump) { + final ((f, n, _, salt), (cfg, _, restart, ev)) = c; + bump('flavour: ${_nnFlavours[f]}'); + if (n == 0) bump('beats: none'); + if (n >= 1 && n < 40) bump('beats: under a thin window'); + if (n >= 500) bump('beats: 500 or more'); + if (n >= 1500) bump('beats: 1500 or more'); + if (salt >= 1) bump('out-of-range entries'); + if (salt >= 2) bump('NaN entries'); + bump(_scCfgValid(cfg) ? 'config: valid' : 'config: invalid'); + if (cfg != 0) bump('config: not the default'); + final b = _nnBounds(c); + if (b.length >= 4) bump('chunks: three or more'); + if (hasEmptyChunk(b)) bump('an empty chunk'); + if (restart != 0 && b.length > 2) bump('a restore between chunks'); + bump('evaluation: $ev'); +} + +const Map _nnShares = { + 'flavour: sinus': .03, + 'flavour: af': .03, + 'flavour: blocks': .03, + 'flavour: flat': .03, + 'flavour: two point': .03, + 'flavour: sinus + gaps': .03, + 'flavour: af heavy': .03, + 'flavour: range edges': .03, + 'beats: under a thin window': .05, + 'beats: 500 or more': .08, + 'out-of-range entries': .3, + 'NaN entries': .2, + 'config: valid': .4, + 'config: invalid': .15, + 'config: not the default': .4, + 'chunks: three or more': .4, + 'an empty chunk': .05, + 'a restore between chunks': .3, + 'evaluation: 0': .03, + 'evaluation: 1': .03, + 'evaluation: 2': .03, + 'evaluation: 3': .03, + 'evaluation: 4': .03, + 'evaluation: 5': .03, + 'evaluation: 6': .03, + 'evaluation: 7': .03, +}; + +_NnCase _n(int flavour, int n, + {int seed = 5, + int salt = 0, + int cfg = 0, + int splits = 3, + int cut = 1, + int restart = 0, + int ev = 1}) => + ((flavour, n, seed, salt), (cfg, splits * (1 << 20) + cut, restart, ev)); + +final List<_NnCase> _nnForced = [ + _n(0, 0, ev: 0), // nothing folded + _n(0, 0, ev: 6), // nothing folded, zero beats seen by the corrector + _n(0, 1), + _n(0, 2), + _n(0, 39, splits: 4, cfg: 1, ev: 1), // one beat short of a thin window of 40 + _n(0, 40, splits: 4, cfg: 0, ev: 1), // exactly the minimum window + _n(0, 41, splits: 4, cfg: 0, ev: 1), + _n(0, 2, cfg: 2, ev: 4), // minimum window of 2 beats, minimum beats 2 + _n(0, 3, cfg: 2, ev: 4), // exactly two successive differences + _n(1, 3, cfg: 2, ev: 4), + _n(1, 4, cfg: 2, ev: 4), + _n(7, 200, cfg: 1, ev: 1, splits: 5, restart: 127), // the edges of the kept range + _n(7, 7, cfg: 2, ev: 4, splits: 3), // exactly one period + _n(3, 300, cfg: 0, ev: 1, splits: 5, restart: 127), // flat: SD2 = 0 + _n(4, 300, cfg: 2, ev: 1, splits: 5, restart: 127), // two point + _n(1, 900, cfg: 0, ev: 1, splits: 6, cut: 3, restart: 127), // AF: flags + _n(6, 1500, cfg: 1, ev: 1, splits: 6, cut: 4, restart: 127), // AF heavy, short windows + _n(2, 1200, cfg: 0, ev: 0, splits: 6, cut: 5), // blocks, default minimum 500 + _n(0, 800, salt: 2, cfg: 3, ev: 5, splits: 6, cut: 6, restart: 85), // out-of-range + NaN + _n(1, 400, salt: 3, cfg: 2, ev: 7, splits: 4, cut: 7, restart: 127), // NaN, artifact exactly at the maximum + _n(5, 900, cfg: 1, ev: 3, splits: 6, cut: 8), // gaps; artifact fraction over the maximum + _n(1, 600, cfg: 4, ev: 1, splits: 3, cut: 9), // invalid: no window length + _n(1, 600, cfg: 5, ev: 1, splits: 3, cut: 10), // invalid: windows of 1 beat + _n(1, 600, cfg: 6, ev: 1, splits: 3, cut: 11), // invalid: fraction 1.5 + _n(1, 600, cfg: 7, ev: 1, splits: 3, cut: 12), // invalid: two at once + _n(2, 700, cfg: 8, ev: 2, splits: 5, cut: 13), // wide thresholds + _n(0, 3000, cfg: 0, ev: 0, splits: 6, cut: 14, restart: 127), // the cap, default minimum + _n(1, 3000, cfg: 1, ev: 1, splits: 6, cut: 15, restart: 127), +]; + +/// Days with an open window, completed windows and counts worth contradicting: +/// the base the mutated-checkpoint law cuts its mutations from. +final List<_NnCase> _nnL3 = _nnForced + .where((c) => c.$1.$2 >= 600 && _scCfgValid(c.$2.$1)) + .take(3) + .toList(); + +/// Window edge hit EXACTLY: a kept beat a full window after the window's first +/// closes it (>=). Hand-built, so it does not depend on a generator reaching it. +({List nn, List t}) _edgeSeries() { + final nn = [], t = []; + // 60 beats at 5000 ms: the 61st lands exactly 300000 ms after the first. + for (var i = 0; i < 62; i++) { + nn.add(800.0 + (i % 7) * 11); + t.add(10000.0 + i * 5000); + } + return (nn: nn, t: t); +} + +void _foldChunks(IrregularScreenState Function() make, _Nn s, List bounds, + int restart, void Function(IrregularScreenState st, int k, int at)? seam, + {IrregularScreenState? into}) { + var st = into ?? make(); + for (var k = 0; k + 1 < bounds.length; k++) { + st.fold(s.nn.sublist(bounds[k], bounds[k + 1]), s.t.sublist(bounds[k], bounds[k + 1])); + if ((restart >> (k % 7)) & 1 == 1) st = _srestore(st); + seam?.call(st, k, bounds[k + 1]); + } +} + +void _scL1(_NnCase c) { + final ((f, n, seed, salt), (cfg, _, restart, ev)) = c; + final s = _nnSeries(f, n, seed, salt); + final tag = _nnTag(c); + final bounds = _nnBounds(c); + final whole = _newState(cfg)..fold(s.nn, s.t); + late IrregularScreenState got; + var st = _newState(cfg); + for (var k = 0; k + 1 < bounds.length; k++) { + st.fold(s.nn.sublist(bounds[k], bounds[k + 1]), s.t.sublist(bounds[k], bounds[k + 1])); + if ((restart >> (k % 7)) & 1 == 1) st = _srestore(st); + } + got = st; + expect(_stext(got), _stext(whole), reason: '$tag bounds=$bounds: same checkpoint text'); + for (var e = 0; e < _evals.length; e++) { + expect(jsonEncode(_evaluate(got, e, beats: n).toJson()), + jsonEncode(_evaluate(whole, e, beats: n).toJson()), + reason: '$tag: evaluation $e reads the same'); + } +} + +/// An evaluation with a provisional tail equals the state after folding it. +void _scL1b(_NnCase c) { + final ((f, n, seed, salt), (cfg, _, restart, ev)) = c; + final s = _nnSeries(f, n, seed, salt); + final tag = _nnTag(c); + final bounds = _nnBounds(c); + final picks = pickSeams(bounds.length - 1); + _foldChunks(() => _newState(cfg), s, bounds, restart, (st, k, at) { + if (!picks.contains(k)) return; + // The tail is what a corrector would still hold provisional: some of the + // beats after the seam. + final tailEnd = math.min(n, at + (at * 7 + k * 13) % 97); + final tailNn = s.nn.sublist(at, tailEnd), tailT = s.t.sublist(at, tailEnd); + final before = _stext(st); + final got = _evaluate(st, ev, tailNn: tailNn, tailT: tailT, beats: tailEnd); + expect(_stext(st), before, reason: '$tag: evaluate reads, never writes'); + final want = _batchScreenOf(s.nn.sublist(0, tailEnd), s.t.sublist(0, tailEnd), cfg, ev); + final why = '$tag seam $k at $at tail to $tailEnd'; + _sameScreen(got, want, why, + nn: s.nn.sublist(0, tailEnd), + t: s.t.sublist(0, tailEnd), + pnnThresholdMs: _scCfgs[cfg].$2, + artifactFraction: _evals[ev].$3); + _expectEvidence(got, s.nn.sublist(0, tailEnd), s.t.sublist(0, tailEnd), cfg, why); + }); +} + +String _projectScreen(IrregularScreenState st, int beats) => jsonEncode([ + for (var e = 0; e < _evals.length; e++) _evaluate(st, e, beats: beats).toJson(), + _evaluate(st, 0, tailNn: const [800, 810, 790], tailT: const [1e12, 1e12 + 810, 1e12 + 1600], beats: beats).toJson(), + ]); + +void _scL2(_NnCase c) { + final ((f, n, seed, salt), (cfg, _, restart, _)) = c; + final s = _nnSeries(f, n, seed, salt); + final tag = _nnTag(c); + final bounds = _nnBounds(c); + final at = bounds.length > 2 ? bounds[1] : n ~/ 2; + final live = _newState(cfg)..fold(s.nn.sublist(0, at), s.t.sublist(0, at)); + final text = _stext(live); + final map = (jsonDecode(text) as Map).cast(); + final back = IrregularScreenState.fromJson(map); + expect(_stext(back), text, reason: '$tag: write(read(b)) == b'); + expect(_projectScreen(back, at), _projectScreen(live, at), + reason: '$tag: the restored one reads what the live one reads'); + // Its parameters travel with it: a restored state is not re-owned by whoever + // reads it (the thresholds it reports are the ones it was built with). + final c0 = _scCfgs[cfg]; + expect(back.windowMinutes, c0.$4, reason: tag); + expect(back.minWindowBeats, c0.$5, reason: tag); + expect(back.sustainedFraction, c0.$6, reason: tag); + expect(back.sd1sd2Flag, c0.$1, reason: tag); + // Not aliased to the map it came from. + (map['bk'] as List).clear(); + (map['bkAdj'] as List).clear(); + expect(_stext(back), text, reason: '$tag: not aliased to the map'); + live.fold(s.nn.sublist(at), s.t.sublist(at)); + back.fold(s.nn.sublist(at), s.t.sublist(at)); + expect(_stext(back), _stext(live), reason: '$tag: same checkpoint after the rest'); + expect(_projectScreen(back, n), _projectScreen(live, n), reason: tag); +} + +(Map, Must) _mutateScreen(Map src, Mut m) { + final j = (deepCopy(src) as Map).cast(); + final (kind, a, b) = m; + final keys = j.keys.toList(); + List l(String k) => j[k] as List; + switch (kind % 11) { + case 0: + // Version 1 (before the diagnostics counters) and version 2 (before the + // previous beat's time) are REFUSED, not restored with state they never + // kept; so is everything else but 3. + const versions = [0, 1, 2, -1, 99, '3', null, true, 1 << 40]; + j['version'] = versions[a % versions.length]; + return (j, Must.refuse); + case 1: + const types = ['RrCorrector', 'Other', null, '', 'irregularscreenstate', 2]; + j['type'] = types[a % types.length]; + return (j, Must.refuse); + case 2: + final k = keys.where((k) => k != 'winStart').toList()[a % (keys.length - 1)]; + j.remove(k); + return (j, Must.refuse); + case 3: + final k = keys.where((k) => k != 'winStart').toList()[a % (keys.length - 1)]; + const junk = ['x', {}, [[]]]; + j[k] = junk[b % junk.length]; + return (j, Must.refuse); + case 4: + // The open window's beats and their neighbour flags disagree. + if (b.isEven) { + l('bkAdj').add(1); + } else if (l('bk').isEmpty) { + l('bk').add(800.0); + } else { + l('bk').removeLast(); + } + return (j, Must.refuse); + case 5: + // Counts that contradict each other: more kept than seen, more valid + // windows than windows, more flagged than valid, more over than diffs. + switch (a % 5) { + case 0: + j['nKept'] = (j['nIn'] as int) + 1; + case 1: + j['total'] = (j['valid'] as int) - 1; + case 2: + j['flagged'] = (j['valid'] as int) + 1; + case 3: + j['over'] = (j['dN'] as int) + 1; + default: + j['nIn'] = (j['nKept'] as int) - 1; + } + return (j, Must.refuse); + case 6: + // Negative counts. + j[const ['nIn', 'nKept', 'dN', 'lN', 'over', 'flagged'][a % 6]] = -1 - b % 5; + return (j, Must.refuse); + case 7: + // The neighbour flags are not flags. (Integers other than 0 / 1 are the + // reader's one leniency, see the skipped finding below.) + const notFlags = ['x', null, 1.5, true, []]; + if (l('bkAdj').isEmpty) { + l('bkAdj').add(notFlags[a % notFlags.length]); + l('bk').add(800.0); + } else { + l('bkAdj')[b % l('bkAdj').length] = notFlags[a % notFlags.length]; + } + return (j, Must.refuse); + case 8: + // Hostile sizes, and parts that disagree in ways the reader does not + // cross-check (the skipped FINDING below): refused, or exactly what was + // written. + const huge = [1 << 40, 1 << 62, 0x7fffffffffffffff]; + switch (a % 8) { + case 0: + j['nIn'] = huge[b % huge.length]; + case 1: + j['nKept'] = (j['nIn'] as int); + j['lN'] = (j['lN'] as int) + 1; // levels seen != kept beats + case 2: + j['dN'] = huge[b % huge.length]; // more differences than beats + case 3: + j['lN'] = huge[b % huge.length]; + case 4: + j['total'] = (j['nKept'] as int) + 1 + b % 5; // more windows than beats + case 5: + j['valid'] = huge[b % huge.length]; + j['total'] = huge[b % huge.length]; + case 6: + j['over'] = (j['dN'] as int); // every difference over the threshold + default: + l('bk').addAll(List.filled(1 + b % 3, 800.0)); + l('bkAdj').addAll(List.filled(1 + b % 3, 1)); + } + return (j, Must.faithful); + case 9: + // One number changed: the reader cannot tell it from a legal checkpoint. + switch (a % 9) { + case 0: + j['dMean'] = (j['dMean'] as num) + 1.0; + case 1: + j['lM2'] = (j['lM2'] as num) + 1.0; + case 2: + j['prevV'] = (j['prevV'] as num) + 1.0; + case 3: + j['prevKept'] = !(j['prevKept'] as bool); + case 4: + j['pnnFlagPct'] = (j['pnnFlagPct'] as num) + 1.0; + case 5: + j['sd1sd2Flag'] = (j['sd1sd2Flag'] as num) + 0.1; + case 6: + j['winStart'] = ((j['winStart'] as num?) ?? 0) + 1.0; + case 7: + j['prevT'] = ((j['prevT'] as num?) ?? 0) + 1.0; + default: + if (l('bk').isNotEmpty) l('bk')[b % l('bk').length] = (l('bk')[b % l('bk').length] as num) + 1.0; + } + return (j, Must.faithful); + default: + j['futureField${a % 3}'] = const [1, 'x', [1]][b % 3]; + return (j, Must.ignored); + } +} + +String _screenCheckpointOutcome(Map j) { + try { + final s = IrregularScreenState.fromJson((deepCopy(j) as Map).cast()); + return 'accepted:${jsonEncode(s.toJson())}'; + } on FormatException { + return 'refused'; + } catch (e) { + return 'WRONG ERROR TYPE: ${e.runtimeType}: $e'; + } +} + +void _scL3((_NnCase, Mut) arg) { + final (c, m) = arg; + final ((f, n, seed, salt), (cfg, _, _, _)) = c; + final tag = '${_nnTag(c)} mutation=${m.$1 % 11}(${m.$2},${m.$3})'; + final s = _nnSeries(f, n, seed, salt); + final st = _newState(cfg)..fold(s.nn, s.t); + final src = (jsonDecode(_stext(st)) as Map).cast(); + final (bad, must) = _mutateScreen(src, m); + expectMutationOutcome(_screenCheckpointOutcome(bad), must, jsonEncode(bad), + jsonEncode(src), tag); +} + +void _scL4(_NnCase c) { + final ((f, n, seed, salt), (cfg, _, _, ev)) = c; + final s = _nnSeries(f, n, seed, salt); + final tag = _nnTag(c); + final fresh = _stext(_newState(cfg)); + // An empty fold on a fresh state records nothing. + final empty = _newState(cfg)..fold(const [], const []); + expect(_stext(empty), fresh, reason: '$tag: an empty first fold records nothing'); + final cut = n ~/ 2; + final st = _newState(cfg)..fold(s.nn.sublist(0, cut), s.t.sublist(0, cut)); + final before = _stext(st); + st.fold(const [], const []); + expect(_stext(st), before, reason: '$tag: an empty fold is the identity'); + // evaluate reads and never writes, with or without a tail, twice equal. + final one = _evaluate(st, ev, beats: cut); + expect(_stext(st), before, reason: '$tag: evaluate is read-only'); + final tail = s.nn.sublist(cut, math.min(n, cut + 20)); + final tailT = s.t.sublist(cut, math.min(n, cut + 20)); + _evaluate(st, ev, tailNn: tail, tailT: tailT, beats: cut + tail.length); + expect(_stext(st), before, reason: '$tag: and with a tail'); + expect(jsonEncode(_evaluate(st, ev, beats: cut).toJson()), jsonEncode(one.toJson()), + reason: '$tag: twice is equal'); + // Refusal: lengths that disagree change nothing. + expect(() => st.fold([800, 810], const [1.0]), throwsArgumentError, reason: tag); + expect(() => st.fold(const [], [1.0]), throwsArgumentError, reason: tag); + expect(_stext(st), before, reason: '$tag: a refused fold leaves the state unchanged'); +} + +void _scL5(_NnCase c) { + final ((f, n, seed, salt), (cfg, _, restart, ev)) = c; + final s = _nnSeries(f, n, seed, salt); + final tag = _nnTag(c); + final bounds = _nnBounds(c); + _foldChunks(() => _newState(cfg), s, bounds, restart, (st, k, at) { + final res = _evaluate(st, ev, beats: at); + final why = '$tag seam $k at $at'; + final dg = res.diagnostics; + final inRange = [for (final v in s.nn.sublist(0, at)) if (v >= 300 && v <= 2000) v]; + expect(dg.nnIn, at, reason: 'nn_in is every entry folded $why'); + expect(dg.nnKept, inRange.length, reason: 'nn_kept is the in-range ones $why'); + expect(dg.nnKept <= dg.nnIn, isTrue, reason: why); + _expectEvidence(res, s.nn.sublist(0, at), s.t.sublist(0, at), cfg, why); + final m = res.metric; + // The beat gate counts successive pairs (minBeats - 1 of them), not beats: + // too few pairs abstains, and nothing past that gate does. + final pairs = _pairsOf(s.nn.sublist(0, at), s.t.sublist(0, at)); + final enoughPairs = pairs >= dg.thresholds.minBeats - 1; + expect(dg.abstain == IrregularAbstain.tooFewBeats, !enoughPairs, + reason: 'too few beats <=> fewer than minBeats - 1 pairs ($pairs) $why'); + if (m.present) { + expect(m.value!.nBeats, dg.nnKept, reason: 'nBeats == nn_kept $why'); + expect(dg.nnKept >= dg.thresholds.minBeats, isTrue, reason: 'min beats $why'); + } + if (dg.abstain == IrregularAbstain.artifact) { + expect(enoughPairs, isTrue, reason: why); + expect(dg.nnKept >= dg.thresholds.minBeats, isTrue, reason: why); + expect(_evals[ev].$3 > dg.thresholds.maxArtifact, isTrue, reason: why); + } + }); +} + +// ── IP: the corrector and the screen together, as the day pass runs them ───── + +/// (a beat series and its chunking, screen config index, evaluation index). +typedef _IpCase = (_Case, int, int); + +Gen<_IpCase> _ipGen(int maxN) => G.triple(_caseGen(maxN), + G.intIn(0, _scCfgs.length - 1), G.intIn(0, _evals.length - 1)); + +String _ipTag(_IpCase c) => '${_tag(c.$1)} screen=${c.$2} eval=${c.$3}'; + +void _observeIp(_IpCase c, void Function(String) bump) { + _observeRr(c.$1, bump); + final (flavour, n, _, _) = c.$1.$1; + bump(_scCfgValid(c.$2) ? 'config: valid' : 'config: invalid'); + if (c.$2 != 0) bump('config: not the default'); + bump('evaluation: ${c.$3}'); + if (flavour == 2 && n >= 600) bump('a long irregularly irregular day'); +} + +const Map _ipShares = { + ..._rrShares, + 'config: valid': .4, + 'config: invalid': .1, + 'config: not the default': .4, + 'evaluation: 0': .02, + 'evaluation: 1': .02, + 'evaluation: 2': .02, + 'evaluation: 3': .02, + 'evaluation: 4': .02, + 'evaluation: 5': .02, + 'evaluation: 6': .02, + 'evaluation: 7': .02, + 'a long irregularly irregular day': .02, +}; + +/// What the day pass computes at a seam: the corrector's provisional tail and +/// counts handed to the screen state. +IrregularScreenResult _ipStream(RrCorrector corr, IrregularScreenState st, int ev) { + final snap = corr.snapshot(); + final e = _evals[ev]; + return st.evaluateDetailed(snap.tailNn, snap.tailNnTimes, + artifactFraction: (1.0 - snap.cleanFraction).clamp(0.0, 1.0), + minBeats: e.$1, + maxArtifact: e.$2, + cleaning: RrCleaningCounts( + raw: snap.n, corrected: snap.correctedCount, dropped: snap.droppedCount)); +} + +void _ipBody(_IpCase arg) { + final (c, scr, ev) = arg; + final ((flavour, n, seed, mode), (splits, cutSeed, restart, cfg)) = c; + final s = _series(flavour, n, seed); + final wall = mode == 0; + final tag = _ipTag(arg); + final bounds = _bounds(n, splits, cutSeed); + final picks = pickSeams(bounds.length - 1); + var corr = _newCorrector(cfg); + var st = _newState(scr); + for (var k = 0; k + 1 < bounds.length; k++) { + final settled = corr.fold(s.rr.sublist(bounds[k], bounds[k + 1]), + tsMs: _tsSlice(s, wall, bounds[k], bounds[k + 1])); + st.fold(settled.nn, settled.nnTimes); + if ((restart >> (k % 7)) & 1 == 1) { + corr = _restore(corr); + st = _srestore(st); + } + if (!picks.contains(k)) continue; + final at = bounds[k + 1]; + final why = '$tag seam $k at $at'; + final got = _ipStream(corr, st, ev); + final want = _oracle(s, wall, cfg, at); + final e = _evals[ev]; + final cs = _scCfgs[scr]; + final batch = irregularBeatScreenDetailed(want.nn, + nnTimesMs: want.nnTimesMs, + artifactFraction: (1.0 - want.cleanFraction).clamp(0.0, 1.0), + minBeats: e.$1, + maxArtifact: e.$2, + sd1sd2Flag: cs.$1, + pnnThresholdMs: cs.$2, + pnnFlagPct: cs.$3, + windowMinutes: cs.$4, + minWindowBeats: cs.$5, + sustainedFraction: cs.$6, + cleaning: RrCleaningCounts( + raw: at, corrected: want.correctedCount, dropped: want.droppedCount)); + _sameScreen(got, batch, why, + nn: want.nn, + t: want.nnTimesMs, + pnnThresholdMs: cs.$2, + artifactFraction: (1.0 - want.cleanFraction).clamp(0.0, 1.0)); + // L5, integrated: the corrector's counts are the screen's evidence. + final dg = got.diagnostics; + expect(dg.rrRaw, at, reason: 'rr_raw is every beat the corrector saw $why'); + expect(dg.dropped, want.droppedCount, reason: 'dropped $why'); + expect(dg.corrected, want.correctedCount, reason: 'corrected $why'); + expect(dg.nnIn, at - dg.dropped!, reason: 'nn_in == rr_raw - dropped $why'); + expect(dg.corrected! + dg.dropped! <= at, isTrue, reason: why); + expect(dg.nnKept <= dg.nnIn, isTrue, reason: 'kept <= in $why'); + if (at == 0) { + expect(dg.artifactFraction, isNull, + reason: 'a share of no beats is absent, not 1.0 $why'); + } else { + expect(dg.artifactFraction, (1.0 - want.cleanFraction).clamp(0.0, 1.0), + reason: 'artifact fraction $why'); + } + if (dg.abstain == IrregularAbstain.artifact) { + // The gate reads the fraction handed in. With nothing seen (and a pair + // gate of zero) that is 1 - cleanFraction of no beats, while the + // diagnostics report the share as absent, not 1.0 (0fc5768). + final handed = (1.0 - want.cleanFraction).clamp(0.0, 1.0); + expect(handed > dg.thresholds.maxArtifact, isTrue, reason: why); + if (at != 0) { + expect(dg.artifactFraction! > dg.thresholds.maxArtifact, isTrue, reason: why); + } + } + _expectEvidence(got, want.nn, want.nnTimesMs, scr, why); + } +} + +// ── W: the diagnostics wire ───────────────────────────────────────────────── + +const List _fractions = [0.0, .07, .30, .31, 1.0, .5]; + +/// (flavour, beats, seed) of a small NN series, screen config, evaluation, and +/// the artifact fraction index handed in. +typedef _WireCase = (_NnRec, int, int, int); + +Gen<_WireCase> _wireGen() => G.quad( + G.quad(G.intIn(0, _nNnFlavours - 1), _NGen(300), G.intIn(0, 1 << 12), + G.elements(const [0, 1, 2, 3])), + G.intIn(0, _scCfgs.length - 1), + G.intIn(0, _evals.length - 1), + G.intIn(0, _fractions.length - 1)); + +IrregularScreenResult _wireResult(_WireCase c) { + final ((f, n, seed, salt), cfg, ev, fr) = c; + final s = _nnSeries(f, n, seed, salt); + final e = _evals[ev]; + final cs = _scCfgs[cfg]; + return irregularBeatScreenDetailed(s.nn, + nnTimesMs: s.t, + artifactFraction: _fractions[fr], + minBeats: math.min(e.$1, 30), + maxArtifact: e.$2, + sd1sd2Flag: cs.$1, + pnnThresholdMs: cs.$2, + pnnFlagPct: cs.$3, + windowMinutes: cs.$4, + minWindowBeats: cs.$5, + sustainedFraction: cs.$6, + cleaning: _cleaningFor(e.$4, n)); +} + +/// Every field a reader of the diagnostics can see. +String _projectDiagnostics(IrregularDiagnostics d) { + final w = d.windows, t = d.thresholds; + return jsonEncode([ + d.abstain?.index, + d.rrRaw, + d.corrected, + d.dropped, + d.nnIn, + d.nnKept, + d.artifactFraction, + w == null + ? null + : [w.total, w.valid, w.flagged, w.openBeats, w.open.index, w.sustainedObserved], + [ + t.minBeats, + t.maxArtifact, + t.sd1sd2Flag, + t.pnnThresholdMs, + t.pnnFlagPct, + t.windowMinutes, + t.minWindowBeats, + t.sustainedFraction + ], + ]); +} + +void _wL2(_WireCase c) { + final res = _wireResult(c); + final text = jsonEncode(res.diagnostics.toJson()); + final back = IrregularDiagnostics.fromJson( + (jsonDecode(text) as Map).cast()); + expect(jsonEncode(back.toJson()), text, reason: 'write(read(b)) == b'); + expect(_projectDiagnostics(back), _projectDiagnostics(res.diagnostics), + reason: 'the restored evidence reads what the live one reads'); + // The envelope edge persists: the old metric keys, unchanged, and one more. + final env = res.toJson(); + final old = res.metric.toJson((v) => v.toJson()); + for (final k in old.keys) { + expect(jsonEncode(env[k]), jsonEncode(old[k]), reason: 'envelope key $k'); + } + expect(env.keys.toSet(), {...old.keys, 'diagnostics'}); + expect(jsonEncode(env['diagnostics']), text); + // Absent stays absent on the wire: an unknown count or window is JSON null. + final beats = (jsonDecode(text) as Map)['beats'] as Map; + expect(beats['rr_raw'], res.diagnostics.rrRaw); + expect(beats['artifact_fraction'], res.diagnostics.artifactFraction); + expect((jsonDecode(text) as Map)['windows'] == null, res.diagnostics.windows == null); +} + +(Map, Must) _mutateWire(Map src, Mut m) { + final j = (deepCopy(src) as Map).cast(); + final (kind, a, b) = m; + Map sec(String k) => j[k] as Map; + switch (kind % 8) { + case 0: + const versions = [0, 2, 3, -1, 99, '1', null, true, 1 << 40]; + j['version'] = versions[a % versions.length]; + return (j, Must.refuse); + case 1: + const names = ['unknown', '', 'tooFewBeats', 'TOO_FEW_BEATS', 3, []]; + j['abstain'] = names[a % names.length]; + return (j, Must.refuse); + case 2: + j.remove(const ['beats', 'thresholds', 'version'][a % 3]); + return (j, Must.refuse); + case 3: + final k = const ['beats', 'thresholds'][a % 2]; + const junk = ['x', [], 1, null]; + j[k] = junk[b % junk.length]; + return (j, Must.refuse); + case 4: + // A field of the wrong kind inside a section. + final keys = [...sec('beats').keys.where((k) => k == 'nn_in' || k == 'nn_kept')]; + final tk = sec('thresholds').keys.toList(); + if (a.isEven) { + sec('beats')[keys[b % keys.length]] = const ['x', null, 1.5, []][a % 4]; + } else { + sec('thresholds')[tk[b % tk.length]] = const ['x', null, []][a % 3]; + } + return (j, Must.refuse); + case 5: + // The open window named something that is not one. + if (j['windows'] == null) { + j['windows'] = { + 'total': 1, + 'valid': 1, + 'flagged': 0, + 'sustained_observed': 0.0, + 'open_beats': 1, + 'open': 'bogus' + }; + } else { + sec('windows')['open'] = const ['bogus', '', 'FLAGGED', 3, null][a % 5]; + } + return (j, Must.refuse); + case 6: + // Numbers changed, or parts that disagree about the evidence: the wire + // is evidence, not a state, and carries no cross-check (the skipped + // FINDING below): refused, or exactly what was written. + switch (a % 6) { + case 0: + sec('beats')['nn_kept'] = (sec('beats')['nn_kept'] as int) + 1 + b % 5; + case 1: + sec('beats')['nn_in'] = (sec('beats')['nn_in'] as int) + 1; + case 2: + sec('thresholds')['max_artifact'] = (sec('thresholds')['max_artifact'] as num) + .1; + case 3: + sec('beats')['artifact_fraction'] = 1.0; + case 4: + if (j['windows'] != null) { + sec('windows')['flagged'] = (sec('windows')['valid'] as int) + 1 + b % 3; + } + default: + if (j['windows'] != null) sec('windows')['total'] = 0; + } + // `sustained_observed` is derived (flagged / valid) and written afresh; + // keep the mutated text consistent with that. + final w = j['windows'] as Map?; + if (w != null) { + w['sustained_observed'] = (w['valid'] as int) == 0 + ? null + : (w['flagged'] as int) / (w['valid'] as int); + } + return (j, Must.faithful); + default: + if (a % 4 == 3 && j['windows'] != null) { + // The derived figure is not read: whatever it says, the reader writes + // flagged / valid back. + sec('windows')['sustained_observed'] = 0.123456; + } else { + j['futureField${a % 3}'] = const [1, 'x', [1]][b % 3]; + } + return (j, Must.ignored); + } +} + +String _wireOutcome(Map j) { + try { + final d = IrregularDiagnostics.fromJson( + (deepCopy(j) as Map).cast()); + return 'accepted:${jsonEncode(d.toJson())}'; + } on FormatException { + return 'refused'; + } catch (e) { + return 'WRONG ERROR TYPE: ${e.runtimeType}: $e'; + } +} + +void _wL3((_WireCase, Mut) arg) { + final (c, m) = arg; + final tag = 'wire ${c.$1} cfg=${c.$2} eval=${c.$3} frac=${c.$4} ' + 'mutation=${m.$1 % 8}(${m.$2},${m.$3})'; + final src = (jsonDecode(jsonEncode(_wireResult(c).diagnostics.toJson())) as Map) + .cast(); + final (bad, must) = _mutateWire(src, m); + expectMutationOutcome(_wireOutcome(bad), must, jsonEncode(bad), jsonEncode(src), tag); +} + +/// The "absent, not zero" rules (0fc5768): cleaning counts are null when the +/// caller gave none; the artifact fraction is null when the corrector saw no +/// beats (a share of nothing; the 1.0 of `1 - cleanFraction` of an empty series +/// would read as every beat being an artifact), the fraction handed in +/// otherwise. The verdict gate still reads the fraction handed in. The batch +/// and the stream agree. +void _wAbsent(_WireCase c) { + final ((f, n, seed, salt), cfg, ev, fr) = c; + final s = _nnSeries(f, n, seed, salt); + final e = _evals[ev]; + final fraction = _fractions[fr]; + final cleaning = _cleaningFor(e.$4, n); + final tag = 'n=$n flavour=${_nnFlavours[f]} cleaning=${e.$4} fraction=$fraction'; + final batch = _batchScreenOf(s.nn, s.t, cfg, ev); + final st = _newState(cfg)..fold(s.nn, s.t); + final stream = st.evaluateDetailed(const [], const [], + artifactFraction: e.$3, minBeats: e.$1, maxArtifact: e.$2, cleaning: cleaning); + for (final r in [batch, stream]) { + final d = r.diagnostics; + expect(d.rrRaw, cleaning?.raw, reason: '$tag: rr_raw is absent without counts'); + expect(d.corrected, cleaning?.corrected, reason: tag); + expect(d.dropped, cleaning?.dropped, reason: tag); + final noBeats = cleaning != null && cleaning.raw == 0; + expect(d.artifactFraction, noBeats ? isNull : e.$3, + reason: '$tag: the artifact fraction of no beats is absent, not ${e.$3}'); + final wire = d.toJson()['beats'] as Map; + expect(wire['artifact_fraction'], d.artifactFraction, reason: tag); + // The gate still reads the fraction handed in. + // Past the pair gate (minBeats - 1 successive pairs), the artifact gate + // comes next. + if (_pairsOf(s.nn, s.t) >= e.$1 - 1 && e.$3 > e.$2) { + expect(d.abstain, IrregularAbstain.artifact, reason: '$tag: gate'); + } + if (d.abstain == IrregularAbstain.artifact) { + expect(e.$3 > e.$2, isTrue, reason: '$tag: gate'); + } + } + // And a screen that ran with no beats counted is still not 1.0. + // (fraction is also drawn directly: see the wire cases below.) + expect(fraction, _fractions[fr]); + final aj = jsonEncode(batch.diagnostics.toJson()), bj = jsonEncode(stream.diagnostics.toJson()); + if (n > 0 || cleaning == null) { + // Same beats in: the same evidence from both sides, abstention included + // (but at a numerically zero SD2, see _sd2Degenerate, the reason may differ). + String noReason(String t) => + jsonEncode((jsonDecode(t) as Map)..['abstain'] = null); + if (_sd2Degenerate(s.nn, s.t)) { + // Only the abstention reason may differ, and only between "reported" and + // "no long-term variability". + expect(noReason(bj), noReason(aj), reason: '$tag: batch and stream agree'); + final ba = batch.diagnostics.abstain, sa = stream.diagnostics.abstain; + if (ba != sa) { + expect({ba, sa}, {null, IrregularAbstain.noLongTermVariability}, + reason: '$tag: the only reason change allowed at a zero SD2'); + } + } else { + expect(bj, aj, reason: '$tag: batch and stream agree'); + } + } +} + +void main() { + group('RC the corrector', () { + _laws.law<_Case>( + 'L1 RrCorrector: any chunking, empty chunks and restores included, ' + 'gives the checkpoint text, settled output and snapshot of one fold', + _caseGen(3000), + _l1Corrector, + examples: _forced, + cases: 60, + reach: Reach<_Case>(_rrShares, _observeRr), + ); + _laws.law<_Case>( + 'L1b RrCorrector: at the seams, settled ++ snapshot is correctRr of the ' + 'prefix, bit for bit (frozen reference up to $_refCap beats)', + _caseGen(1500), + _l1bCorrector, + examples: [for (final c in _forced) if (c.$1.$2 <= 450) c], + cases: 8, + reach: Reach<_Case>({ + for (final e in _rrShares.entries) + if (!e.key.startsWith('flavour: ') && + !const { + 'beats: 1000 or more', + 'chunks: five or more', + 'beats: none', + 'beats: under 3 (the short branch)', + 'flavour: flagged', + 'beats: past the window (91 or more)', + }.contains(e.key)) + e.key: e.value + }, _observeRr), + ); + _laws.law<_Case>( + 'L2 RrCorrector: write(read(b)) == b, the restored one reads the same ' + 'and goes on identically', + _caseGen(1500), + _l2Corrector, + examples: _forced, + cases: 50, + reach: Reach<_Case>(_rrShares, _observeRr), + ); + _laws.law<(_Case, Mut)>( + 'L3 RrCorrector: a mutated checkpoint is refused whole or holds exactly ' + 'what was written', + G.pair(_caseGen(600), _mutGen), + _l3Corrector, + examples: [ + for (var i = 0; i < 11 * 4; i++) + (_forced[const [3, 8, 13, 16, 20, 17, 11][i % 7] + (i % 3 == 0 ? 0 : 0)], (i % 11, i ~/ 11, 7 + i)), + for (var k = 0; k < 11; k++) (_l3Base, (k, k, 0)), + for (var a = 0; a < 10; a++) (_l3Base, (0, a, 0)), + for (var a = 0; a < 5; a++) for (var b = 0; b < 4; b++) (_l3Base, (8, a, b)), + ], + cases: 200, + reach: Reach<(_Case, Mut)>({ + for (var k = 0; k < 11; k++) 'kind: $k': .02, + 'a state with settled beats': .3, + 'an empty state': .008, + }, (arg, bump) { + bump('kind: ${arg.$2.$1 % 11}'); + final n = arg.$1.$1.$2; + if (n >= 91) bump('a state with settled beats'); + if (n == 0) bump('an empty state'); + }), + ); + _laws.law<_Case>( + 'L4 RrCorrector: an empty chunk is the identity and records nothing; a ' + 'refused fold leaves the state unchanged; snapshot() reads only', + _caseGen(800), + _l4Corrector, + examples: _forced, + cases: 150, + reach: Reach<_Case>(_rrShares, _observeRr), + ); + _laws.law<_Case>( + 'L5 RrCorrector: beats are conserved (nn_in == rr_raw - dropped, normal + ' + 'corrected + dropped == beats), classes cover every beat, times never run ' + 'backwards', + _caseGen(3000), + _l5Corrector, + examples: _forced, + cases: 40, + reach: Reach<_Case>(_rrShares, _observeRr), + ); + }); + + group('SC the screen state', () { + final shares = _nnShares; + _laws.law<_NnCase>( + 'L1 IrregularScreenState: any chunking, empty chunks and restores ' + 'included, gives the checkpoint text and every evaluation of one fold', + _nnCaseGen(3000), + _scL1, + examples: _nnForced, + cases: 200, + reach: Reach<_NnCase>(shares, _observeNn), + ); + _laws.law<_NnCase>( + 'L1b IrregularScreenState: evaluate(tail) is the batch screen of settled ' + '++ tail; the window counts are an independent partition\'s', + _nnCaseGen(1500), + _scL1b, + examples: [for (final c in _nnForced) if (c.$1.$2 <= 1500) c], + cases: 150, + reach: Reach<_NnCase>({ + for (final e in shares.entries) + if (!const {'beats: 1500 or more', 'beats: 500 or more'}.contains(e.key)) + e.key: e.value + }, _observeNn), + ); + _laws.law<_NnCase>( + 'L2 IrregularScreenState: write(read(b)) == b, the restored one reads the ' + 'same, keeps its own parameters and goes on identically', + _nnCaseGen(1500), + _scL2, + examples: _nnForced, + cases: 200, + reach: Reach<_NnCase>(shares, _observeNn), + ); + _laws.law<(_NnCase, Mut)>( + 'L3 IrregularScreenState: a version 1 or otherwise mutated checkpoint is ' + 'refused whole or holds exactly what was written', + G.pair(_nnCaseGen(600), _mutGen), + _scL3, + examples: [ + for (var k = 0; k < 11; k++) + for (var a = 0; a < 3; a++) (_nnL3[a], (k, a + k, 3 * k + a)), + for (var a = 0; a < 9; a++) (_nnL3[0], (0, a, 0)), + // Every key but winStart removed in turn (prevT included: a version 3 + // checkpoint without it is refused, not read as an unknown time). + for (var a = 0; a < 25; a++) (_nnL3[0], (2, a, 0)), + (_nnL3[0], (9, 7, 0)), // prevT changed: held exactly + ], + cases: 200, + reach: Reach<(_NnCase, Mut)>({ + for (var k = 0; k < 11; k++) 'kind: $k': .02, + 'a state with an open window': .3, + }, (arg, bump) { + bump('kind: ${arg.$2.$1 % 11}'); + if (arg.$1.$1.$2 >= 40) bump('a state with an open window'); + }), + ); + _laws.law<_NnCase>( + 'L4 IrregularScreenState: an empty fold is the identity, evaluate never ' + 'writes, a refused fold leaves the state unchanged', + _nnCaseGen(800), + _scL4, + examples: _nnForced, + cases: 200, + reach: Reach<_NnCase>(shares, _observeNn), + ); + _laws.law<_NnCase>( + 'L5 IrregularScreenState: nn_in/nn_kept count the input, the windows ' + 'cover the kept NN, flagged <= valid <= total, an invalid config has no ' + 'windows and cannot flag', + _nnCaseGen(3000), + _scL5, + examples: _nnForced, + cases: 150, + reach: Reach<_NnCase>(shares, _observeNn), + ); + test('a kept beat a full window after the first closes the window (>=)', () { + final e = _edgeSeries(); + // 62 beats: the 61st sits exactly 300000 ms after the first. + final st = IrregularScreenState(minWindowBeats: 2)..fold(e.nn, e.t); + final res = st.evaluateDetailed(const [], const [], minBeats: 1); + expect(res.diagnostics.windows!.total, 2, reason: 'closed AT the edge'); + expect(res.diagnostics.windows!.openBeats, 2); + final ref = _refWindows(e.nn, e.t, 0); + expect(ref.total, 2); + // One millisecond short of the edge: still one window. + final t2 = [...e.t]..[60] -= 1; + final st2 = IrregularScreenState(minWindowBeats: 2)..fold(e.nn, t2); + expect(st2.evaluateDetailed(const [], const [], minBeats: 1).diagnostics.windows!.total, 2, + reason: 'beat 62 is past the edge anyway'); + final t3 = [...e.t.sublist(0, 61)]..[60] -= 1; + final st3 = IrregularScreenState(minWindowBeats: 2)..fold(e.nn.sublist(0, 61), t3); + expect(st3.evaluateDetailed(const [], const [], minBeats: 1).diagnostics.windows!.total, 1, + reason: 'one millisecond short of a full window: still open'); + }); + }); + + group('IP the corrector and the screen together', () { + _laws.law<_IpCase>( + 'L1b+L5 corrector -> screen: at the seams the diagnostics are the batch ' + 'diagnostics over the corrected prefix, and the beats are conserved', + _ipGen(1200), + _ipBody, + examples: [ + for (var i = 0; i < _forced.length; i++) + if (_forced[i].$1.$2 <= 450 && i % 2 == 0) (_forced[i], const [0, 1, 2, 3, 4, 8, 5, 6, 7][i % 9], const [0, 1, 2, 3, 4, 5, 6, 7][i % 8]), + // The flagged day under the default screen, and the same under each + // window config, at 500 beats and over. + (_c(2, 450, splits: 4, cutSeed: 31, restart: 127), 0, 1), + (_c(2, 450, splits: 4, cutSeed: 32, restart: 127), 1, 1), + (_c(2, 450, splits: 4, cutSeed: 33), 2, 1), + (_c(2, 450, splits: 4, cutSeed: 34), 3, 1), + (_c(2, 450, splits: 4, cutSeed: 35), 4, 1), + (_c(2, 450, splits: 4, cutSeed: 36), 5, 1), + (_c(2, 450, splits: 4, cutSeed: 37), 6, 1), + (_c(2, 450, splits: 4, cutSeed: 38), 7, 1), + (_c(2, 450, splits: 4, cutSeed: 39), 8, 1), + (_c(2, 600, splits: 3, cutSeed: 40), 0, 0), // default minimum of 500 + (_c(0, 0), 0, 6), // no beats, nothing seen by the corrector + (_c(0, 0, mode: 1), 1, 1), + ], + cases: 8, + reach: Reach<_IpCase>({ + for (final e in _ipShares.entries) + if (!e.key.startsWith('flavour: ') && + !const { + 'beats: 1000 or more', + 'chunks: five or more', + 'beats: none', + 'beats: under 3 (the short branch)', + 'flavour: flagged', + 'a long irregularly irregular day', + 'no timestamps', + 'config: invalid', + 'evaluation: 0', + 'evaluation: 1', + 'evaluation: 2', + 'evaluation: 3', + 'evaluation: 4', + 'evaluation: 5', + 'evaluation: 6', + 'evaluation: 7', + }.contains(e.key)) + e.key: e.value + }, _observeIp), + ); + }); + + group('W the diagnostics wire', () { + final reach = Reach<_WireCase>({ + 'abstained': .1, + 'ran': .05, + 'no cleaning counts': .1, + 'zero beats seen': .02, + 'windows present': .3, + 'windows absent (invalid config)': .05, + }, (c, bump) { + final ((f, n, _, _), cfg, ev, _) = c; + bump(n >= math.min(_evals[ev].$1, 30) && n >= 1 ? 'ran' : 'abstained'); + if (_evals[ev].$4 == 0) bump('no cleaning counts'); + if (_evals[ev].$4 == 2) bump('zero beats seen'); + bump(_scCfgValid(cfg) ? 'windows present' : 'windows absent (invalid config)'); + }); + final forcedWire = <_WireCase>[ + ((0, 0, 1, 0), 0, 6, 4), // zero beats, corrector saw none, fraction 1.0 + ((0, 0, 1, 0), 0, 0, 4), // zero beats, counts say 3 + ((0, 0, 1, 0), 0, 2, 4), // zero beats, no counts at all + ((1, 200, 1, 0), 0, 1, 1), + ((1, 300, 2, 0), 2, 4, 0), + ((2, 300, 3, 0), 1, 7, 2), // artifact fraction exactly at the maximum + ((2, 300, 3, 0), 1, 3, 3), // and just over it + ((3, 100, 1, 0), 0, 1, 0), // flat + ((4, 100, 1, 0), 2, 1, 0), // two point + ((0, 50, 1, 2), 4, 1, 0), // invalid window config + ((0, 50, 1, 0), 0, 1, 0), // exactly the minimum windows + ((5, 280, 1, 3), 3, 5, 1), + ]; + _laws.law<_WireCase>( + 'W-L2 IrregularDiagnostics: write(read(b)) == b through JSON text, the ' + 'restored evidence reads the same, the envelope keeps the old keys', + _wireGen(), + _wL2, + examples: forcedWire, + cases: 200, + reach: reach, + ); + _laws.law<(_WireCase, Mut)>( + 'W-L3 IrregularDiagnostics: another version, an unknown reason, a missing ' + 'or malformed section is refused whole; changed numbers are held exactly', + G.pair(_wireGen(), _mutGen), + _wL3, + examples: [ + for (var k = 0; k < 8; k++) + for (var i = 0; i < 3; i++) (forcedWire[3 + i], (k, 5 * k + i, 2 * k + i)), + for (var a = 0; a < 9; a++) (forcedWire[3], (0, a, 0)), + for (var a = 0; a < 6; a++) (forcedWire[3], (1, a, 0)), + for (var a = 0; a < 6; a++) (forcedWire[3], (6, a, 0)), + for (var a = 0; a < 5; a++) (forcedWire[9], (5, a, 0)), + ], + cases: 200, + reach: Reach<(_WireCase, Mut)>({ + for (var k = 0; k < 8; k++) 'kind: $k': .03, + }, (arg, bump) => bump('kind: ${arg.$2.$1 % 8}')), + ); + _laws.law<_WireCase>( + 'W-L5 diagnostics: cleaning counts are absent unless given, the artifact ' + 'fraction is absent when the corrector saw no beats and kept otherwise; ' + 'batch and stream agree (0fc5768)', + _wireGen(), + _wAbsent, + examples: forcedWire, + cases: 200, + reach: reach, + ); + }); + + group('RC accepted-checkpoint continuation', () { + test('a checkpoint the reader accepts with a nonzero first difference goes ' + 'on from it: the second beat can be the recovery of the first', () { + // A writer stores d[0] = 0, but the reader takes any finite difference. + // Resumed from d[0] = -200, beat 1 (+200 against beat 0) is a recovery + // from beat 0's jump, and is demoted to normal; with d[0] = 0 it is not. + // (If the reader is ever hardened to refuse a nonzero d[0], this becomes + // the refusal test.) + List classesAfter(double d0) { + final c = RrCorrector(windowBeats: 1, maxArtifactRun: null)..fold([600]); + final j = (jsonDecode(jsonEncode(c.toJson())) as Map).cast(); + (j['d'] as List)[0] = d0; + return RrCorrector.fromJson(j).fold([800, 800]).classes; + } + + expect(classesAfter(-200.0), + [BeatClass.ectopic, BeatClass.normal, BeatClass.normal]); + expect(classesAfter(0.0), isNot(classesAfter(-200.0)), + reason: 'the stored difference is what decides it'); + }); + }); + + group('the forced scenarios say what they claim', () { + test('every abstention reason, every open-window label, a flag and the ' + 'absent artifact fraction are met by hand-built series, and the stream ' + 'agrees with the batch on each', () { + // (nn, minBeats, artifact fraction, maxArtifact) -> the reason, known by hand. + List sinus(int n) => [for (var i = 0; i < n; i++) 800.0 + (i % 9) * 13]; + List af(int n) => + [for (var i = 0; i < n; i++) (i % 2 == 0 ? 450.0 : 1100.0) + (i % 5) * 7]; + // A [holeEvery] > 0 puts a 10 s dropout before every holeEvery-th beat. + List times(List nn, int holeEvery) { + var t = 0.0; + var i = 0; + return [ + for (final v in nn) + t += (v >= 300 && v <= 2000 ? v : 800) + + (holeEvery > 0 && i++ % holeEvery == holeEvery - 1 ? 10000 : 0) + ]; + } + + final cases = <(String, List, int, double, IrregularAbstain?, int)>[ + ('too few beats', sinus(30), 40, 0.0, IrregularAbstain.tooFewBeats, 0), + // 60 beats, but a dropout before every second one leaves 29 pairs: + // the gate counts successive pairs (39 needed), not beats. + ('enough beats, too few successive pairs', sinus(60), 40, 0.0, + IrregularAbstain.tooFewBeats, 2), + // Zero pairs clear a gate of minBeats - 1 = 0, and then there is no + // difference to build the Poincare plot from. + ('nothing at all', const [], 1, 0.0, IrregularAbstain.noSuccessivePairs, 0), + ('too noisy', sinus(60), 40, .31, IrregularAbstain.artifact, 0), + ('exactly at the artifact maximum', sinus(60), 40, .30, null, 0), + ('exactly the minimum beats', sinus(40), 40, 0.0, null, 0), + // Two kept beats that are not neighbours: no difference exists. + ('no successive pair', const [800, 5000, 810], 1, 0.0, + IrregularAbstain.noSuccessivePairs, 0), + // Two kept neighbours with a dropout between them: not successive. + ('no successive pair across a time hole', const [800, 810], 1, 0.0, + IrregularAbstain.noSuccessivePairs, 1), + ('no long-term variability', List.filled(60, 800), 40, 0.0, + IrregularAbstain.noLongTermVariability, 0), + ('irregularly irregular', af(200), 40, 0.0, null, 0), + ]; + final opens = {}; + var flagged = false; + for (final (name, nn, minBeats, fraction, reason, holeEvery) in cases) { + final t = times(nn, holeEvery); + final batch = irregularBeatScreenDetailed(nn, + nnTimesMs: t, minBeats: minBeats, artifactFraction: fraction); + final st = IrregularScreenState()..fold(nn, t); + final stream = st.evaluateDetailed(const [], const [], + minBeats: minBeats, artifactFraction: fraction); + expect(batch.diagnostics.abstain, reason, reason: '$name: batch'); + _sameScreen(stream, batch, name); + final w = batch.diagnostics.windows; + if (w != null) opens.add(w.open); + if (batch.metric.present && batch.metric.value!.flag) flagged = true; + } + expect(flagged, isTrue, reason: 'a flagged day'); + expect(opens, { + IrregularOpenWindow.none, + IrregularOpenWindow.thin, + IrregularOpenWindow.unflagged, + IrregularOpenWindow.flagged, + }, reason: 'every open-window label'); + }); + + test('the forced corrector days are thin, flat, artefact-heavy, boundary, ' + 'timestamp-free and past one window', () { + bool any(bool Function(_Case c, _Series s) f) => _forced.any((c) => + f(c, _series(c.$1.$1, c.$1.$2, c.$1.$3))); + for (final n in [0, 1, 2, 3, 44, 46, 90, 91, 92]) { + expect(any((c, s) => s.n == n), isTrue, reason: '$n beats'); + } + expect(any((c, s) => c.$1.$4 == 1), isTrue, reason: 'no timestamps'); + for (var cfg = 0; cfg < _corrCfgs.length; cfg++) { + expect(any((c, s) => c.$2.$4 == cfg), isTrue, reason: 'config $cfg'); + } + // Flat: QD = 0, everything ties. Outliers at the first and last beat. + expect(any((c, s) => _flavours[c.$1.$1] == 'flat' && s.rr.toSet().length == 1), + isTrue); + expect( + any((c, s) => + _flavours[c.$1.$1] == 'outliers' && + s.n >= 3 && + s.rr.first == 1300 && + s.rr.last == 1300), + isTrue); + // A run of artefacts longer than the window: dropped, never corrected. + expect( + any((c, s) => + _flavours[c.$1.$1] == 'artefact heavy' && + correctRr(s.rr, rrTsMs: s.ts).droppedCount >= 20), + isTrue, + reason: 'artefacts that are dropped, not corrected (a run of 120 garbage beats is longer than the corrector window)'); + // A beat time that steps back, a dropout. + expect( + any((c, s) { + if (_flavours[c.$1.$1] != 'backwards') return false; + for (var i = 1; i < s.n; i++) { + if (s.ts[i] < s.ts[i - 1]) return true; + } + return false; + }), + isTrue, + reason: 'beat times that step backwards'); + expect( + any((c, s) { + if (_flavours[c.$1.$1] != 'gappy') return false; + for (var i = 1; i < s.n; i++) { + if (s.ts[i] - s.ts[i - 1] > 5000) return true; + } + return false; + }), + isTrue, + reason: 'a dropout'); + }); + + test('the forced integrated days reach a flag, a thin open window, a valid ' + 'one and the abstentions', () { + final reasons = {}; + final opens = {}; + var flagged = false, noBeatsAbsent = false; + for (final c in _nnForced) { + final ((f, n, seed, salt), (cfg, _, _, ev)) = c; + final s = _nnSeries(f, n, seed, salt); + final res = _evaluate(_newState(cfg)..fold(s.nn, s.t), ev, beats: n); + reasons.add(res.diagnostics.abstain); + final w = res.diagnostics.windows; + if (w != null) opens.add(w.open); + if (res.metric.present && res.metric.value!.flag) flagged = true; + if (n == 0 && _evals[ev].$4 == 2 && res.diagnostics.artifactFraction == null) { + noBeatsAbsent = true; + } + } + expect(reasons, containsAll([ + null, + IrregularAbstain.tooFewBeats, + IrregularAbstain.artifact, + IrregularAbstain.noLongTermVariability, + ])); + expect(opens, containsAll([ + IrregularOpenWindow.none, + IrregularOpenWindow.thin, + IrregularOpenWindow.unflagged, + ])); + expect(flagged, isTrue, reason: 'a flagged screen among the forced days'); + expect(noBeatsAbsent, isTrue, reason: 'zero beats: the fraction is absent'); + }); + }); + + group('FINDINGS (skipped: suspected gaps against the documented contract)', () { + test('IrregularScreenState.fromJson bounds differences and closed windows', () { + final nn = [for (var i = 0; i < 300; i++) 800.0 + (i % 7) * 11]; + final t = [for (var i = 0; i < 300; i++) 10000.0 + i * 1000]; + final st = IrregularScreenState()..fold(nn, t); + Map fresh() => + (jsonDecode(jsonEncode(st.toJson())) as Map).cast(); + final good = fresh(); + expect((good['bk'] as List).length, 300); + expect(IrregularScreenState.fromJson(good).toJson(), good); + + for (final (field, value) in [('dN', 300), ('total', 1)]) { + final bad = fresh()..[field] = value; + expect(() => IrregularScreenState.fromJson(bad), throwsFormatException, + reason: '$field: $value'); + } + }); + + test('IrregularScreenState.fromJson refuses a checkpoint whose parts ' + 'contradict each other', () { + final nn = [for (var i = 0; i < 300; i++) 800.0 + (i % 7) * 11]; + final t = [for (var i = 0; i < 300; i++) 10000.0 + i * 1000]; + final st = IrregularScreenState()..fold(nn, t); + Map fresh() => + (jsonDecode(jsonEncode(st.toJson())) as Map).cast(); + final bad = )>{ + 'levels seen != kept beats': (j) => j['lN'] = 5, + 'more differences than beats': (j) => j['dN'] = 1 << 40, + 'more windows than beats': (j) => j['total'] = 1 << 40, + 'a neighbour flag that is not 0 / 1': (j) => (j['bkAdj'] as List)[0] = 2, + 'a negative neighbour flag': (j) => (j['bkAdj'] as List)[0] = -1, + 'an open window without a start': (j) => j['winStart'] = null, + }; + for (final e in bad.entries) { + final j = fresh(); + e.value(j); + expect(() => IrregularScreenState.fromJson(j), throwsFormatException, + reason: e.key); + } + }); + + test('RrCorrector.fromJson bounds counters by their pipeline stages', () { + final short = RrCorrector(maxArtifactRun: null)..fold([800.0, 800.0, 800.0]); + Map shortJson() => + (jsonDecode(jsonEncode(short.toJson())) as Map).cast(); + final shortCheckpoint = shortJson(); + expect(shortCheckpoint['n'], 3); + expect(shortCheckpoint['c2'], 0); + expect(shortCheckpoint['ce'], 0); + expect(RrCorrector.fromJson(shortCheckpoint).toJson(), shortCheckpoint); + for (final (field, value) in [ + ('normalFinal', 1), + ('dropped', 1), + ('corrected', 1), + ]) { + final bad = shortJson()..[field] = value; + expect(() => RrCorrector.fromJson(bad), throwsFormatException, + reason: '$field: $value'); + } + + final long = RrCorrector(maxArtifactRun: null)..fold([for (var i = 0; i < 300; i++) 800.0]); + final longCheckpoint = + (jsonDecode(jsonEncode(long.toJson())) as Map).cast(); + final c2 = longCheckpoint['c2'] as int; + final ce = longCheckpoint['ce'] as int; + expect(c2, greaterThan(0)); + expect(ce, greaterThan(0)); + expect(longCheckpoint['off'], greaterThan(0)); + expect(RrCorrector.fromJson(longCheckpoint).toJson(), longCheckpoint); + final tooManyOutcomes = { + ...longCheckpoint, + 'dropped': ce, + 'corrected': 1, + }; + expect(() => RrCorrector.fromJson(tooManyOutcomes), throwsFormatException, + reason: 'dropped + corrected exceeds ce'); + }); + + test('RrCorrector.fromJson refuses a checkpoint whose counters are ' + 'impossible', () { + final c = RrCorrector(maxArtifactRun: null) + ..fold([for (var i = 0; i < 300; i++) 800.0 + (i % 7) * 11], + tsMs: [for (var i = 0; i < 300; i++) 1.76e12 + i * 1000]); + Map fresh() => + (jsonDecode(jsonEncode(c.toJson())) as Map).cast(); + final bad = )>{ + 'beats folded but timestamp mode unknown': (j) => j['wall'] = null, + 'negative dropped count': (j) => j['dropped'] = -5, + 'more dropped than beats': (j) => j['dropped'] = 1 << 40, + 'negative corrected count': (j) => j['corrected'] = -1, + 'more normal beats than classified': (j) => j['normalFinal'] = 1 << 40, + }; + for (final e in bad.entries) { + final j = fresh(); + e.value(j); + expect(() => RrCorrector.fromJson(j), throwsFormatException, reason: e.key); + } + expect(() => RrCorrector.fromJson({...fresh(), 'alpha': -1.0}), returnsNormally); + }); + }); + + _laws.registerReachTest(); +} diff --git a/test/properties/stage_minute_ranges_laws_test.dart b/test/properties/stage_minute_ranges_laws_test.dart new file mode 100644 index 0000000..340a765 --- /dev/null +++ b/test/properties/stage_minute_ranges_laws_test.dart @@ -0,0 +1,223 @@ +// LAWS — per-stage minute ranges (D27 revised). RED: new API. +// +// See test/onehz/stage_minute_ranges_test.dart for the contract and the API. +// These laws hold for any night the stager could produce: +// L1 bounds: 0 ≤ lo ≤ point ≤ hi ≤ window per stage; Σlo ≤ window ≤ Σhi; +// deep and light sit inside nrem (lo and hi both). +// L2 coverage: covering more epochs never widens a range (lo never falls, +// hi never rises), whatever the newly covered epochs say. +// L3 clear: every epoch covered and every score clear of the margin ⇒ +// lo == hi == point for every stage. +// +// Run: dart test test/properties/stage_minute_ranges_laws_test.dart + +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +import '../support/law_registry.dart'; +import '../support/property.dart'; + +const double _margin = 0.2; +const double _remCut = kDefaultRemScoreCut; +const double _deepCut = kDefaultDeepScoreCut; + +/// One generated epoch: label index (0 none, 1 wake, 2 light, 3 deep, 4 rem), +/// (remScore, deepScore), and a coverage level: 0 uncovered in both nights of +/// L2, 1 uncovered in A but covered in B, 2 covered in both. +typedef _Ep = (int, (double?, double?), int); + +final Gen _score = G.doubleIn(-3, 3, boundaries: [ + _remCut, + _remCut - _margin, + _remCut + _margin, + _deepCut, + _deepCut - _margin, + _deepCut + _margin, +]); + +final Gen<_Ep> _ep = G.triple( + G.intIn(0, 4, origin: 2), + G.pair(G.nullable(_score, nullProbability: .15), + G.nullable(_score, nullProbability: .15)), + G.intIn(0, 2, origin: 2), +); + +final Gen> _night = + _SometimesAllCovered(G.listOf(_ep, minLen: 1, maxLen: 240)); + +/// Per-epoch coverage is drawn independently, so a night of any length is +/// almost never covered end to end. A fifth of the nights are forced fully +/// covered so L1 meets that case (it is the shape a good night has). +class _SometimesAllCovered extends Gen> { + _SometimesAllCovered(this.inner); + final Gen> inner; + + @override + List<_Ep> generate(Rng r, int size) { + final n = inner.generate(r, size); + if (!r.nextBool(.2)) return n; + return [for (final (l, s, _) in n) (l, s, 2)]; + } + + @override + Iterable> shrink(List<_Ep> v) => inner.shrink(v); + + @override + String show(List<_Ep> v) => inner.show(v); +} + +StageEpochEvidence _evidence(_Ep e, {required bool covered}) { + final (label, (rem, deep), _) = e; + final SleepStage? stage = switch (label) { + 1 => SleepStage.wake, + 2 || 3 => SleepStage.nrem, + 4 => SleepStage.rem, + _ => null, + }; + return StageEpochEvidence( + stage: stage, + deep: label == 3, + covered: covered, + remScore: stage == SleepStage.wake ? null : rem, + deepScore: stage == SleepStage.wake ? null : deep, + ); +} + +List _nightA(List<_Ep> n) => + [for (final e in n) _evidence(e, covered: e.$3 == 2)]; +List _nightB(List<_Ep> n) => + [for (final e in n) _evidence(e, covered: e.$3 >= 1)]; + +/// The stager's own labelled minutes per display stage. +Map _points(List ev) { + final p = {'wake': 0.0, 'light': 0.0, 'deep': 0.0, 'rem': 0.0, 'nrem': 0.0}; + for (final e in ev) { + switch (e.stage) { + case SleepStage.wake: + p['wake'] = p['wake']! + .5; + case SleepStage.nrem: + p[e.deep ? 'deep' : 'light'] = p[e.deep ? 'deep' : 'light']! + .5; + p['nrem'] = p['nrem']! + .5; + case SleepStage.rem: + p['rem'] = p['rem']! + .5; + case null: + break; + } + } + return p; +} + +Map _byName(StageMinuteRanges r) => { + 'wake': r.wake, + 'light': r.light, + 'deep': r.deep, + 'rem': r.rem, + 'nrem': r.nrem, + }; + +bool _near(double? s, double cut) => s != null && (s - cut).abs() < _margin; + +void _observe(List<_Ep> n, void Function(String) bump) { + if (n.any((e) => e.$3 == 0)) bump('uncovered epoch'); + if (n.any((e) => e.$3 == 1)) bump('newly covered epoch'); + if (n.every((e) => e.$3 == 2)) bump('all covered'); + if (n.any((e) => e.$1 == 0)) bump('unlabelled epoch'); + if (n.any((e) => e.$1 == 1)) bump('wake'); + if (n.any((e) => e.$1 == 3)) bump('deep'); + if (n.any((e) => e.$1 == 4)) bump('rem'); + if (n.any((e) => e.$1 >= 2 && _near(e.$2.$1, _remCut))) bump('near rem cut'); + if (n.any((e) => e.$1 >= 2 && _near(e.$2.$2, _deepCut))) { + bump('near deep cut'); + } +} + +/// L3's input: the same night with every epoch covered, labelled, and both +/// scores pushed clear of their cutoffs. +List _clear(List<_Ep> n) => [ + for (final (label, (rem, deep), _) in n) + _evidence(( + label == 0 ? 2 : label, + (_clearOf(rem, _remCut), _clearOf(deep, _deepCut)), + 2, + ), covered: true) + ]; + +double? _clearOf(double? s, double cut) { + if (s == null || !_near(s, cut)) return s; + return s >= cut ? cut + _margin : cut - _margin; +} + +void main() { + final laws = LawSet(); + final reach = Reach>({ + 'uncovered epoch': .5, + 'newly covered epoch': .5, + 'all covered': .02, + 'unlabelled epoch': .3, + 'wake': .5, + 'deep': .5, + 'rem': .5, + 'near rem cut': .3, + 'near deep cut': .3, + }, _observe); + + laws.law>('L1 ranges bracket the point and the window', _night, + (n) { + for (final ev in [_nightA(n), _nightB(n)]) { + final r = stageMinuteRanges(ev, margin: _margin)!; + final window = ev.length * .5; + expect(r.windowMin, closeTo(window, 1e-9)); + final pts = _points(ev); + var sumLo = 0.0, sumHi = 0.0; + _byName(r).forEach((name, range) { + expect(range.lo, greaterThanOrEqualTo(0), reason: '$name lo ≥ 0'); + expect(range.hi, lessThanOrEqualTo(window + 1e-9), + reason: '$name hi ≤ window'); + expect(range.lo, lessThanOrEqualTo(pts[name]! + 1e-9), + reason: '$name lo ≤ point'); + expect(range.hi, greaterThanOrEqualTo(pts[name]! - 1e-9), + reason: '$name point ≤ hi'); + if (name != 'nrem') { + sumLo += range.lo; + sumHi += range.hi; + } + }); + expect(sumLo, lessThanOrEqualTo(window + 1e-9), reason: 'Σlo ≤ window'); + expect(sumHi, greaterThanOrEqualTo(window - 1e-9), + reason: 'window ≤ Σhi'); + for (final sub in [r.deep, r.light]) { + expect(sub.lo, lessThanOrEqualTo(r.nrem.lo + 1e-9)); + expect(sub.hi, lessThanOrEqualTo(r.nrem.hi + 1e-9)); + } + } + }, reach: reach, examples: [ + [(0, (null, null), 0)], + [(4, (_remCut + .01, null), 2), (3, (null, _deepCut - .01), 2)], + ]); + + laws.law>('L2 more coverage never widens a range', _night, (n) { + final a = _byName(stageMinuteRanges(_nightA(n), margin: _margin)!); + final b = _byName(stageMinuteRanges(_nightB(n), margin: _margin)!); + for (final k in a.keys) { + expect(b[k]!.lo, greaterThanOrEqualTo(a[k]!.lo - 1e-9), + reason: '$k lo fell with more coverage'); + expect(b[k]!.hi, lessThanOrEqualTo(a[k]!.hi + 1e-9), + reason: '$k hi rose with more coverage'); + } + }, reach: reach, examples: [ + [(1, (null, null), 1), (2, (-1, -1), 0)], + ]); + + laws.law>('L3 covered and clear ⇒ lo == hi == point', _night, + (n) { + final ev = _clear(n); + final r = stageMinuteRanges(ev, margin: _margin)!; + final pts = _points(ev); + _byName(r).forEach((name, range) { + expect(range.lo, closeTo(pts[name]!, 1e-9), reason: '$name lo'); + expect(range.hi, closeTo(pts[name]!, 1e-9), reason: '$name hi'); + }); + }, reach: reach); + + laws.registerReachTest(); +} diff --git a/test/properties/welch_stream_laws_test.dart b/test/properties/welch_stream_laws_test.dart new file mode 100644 index 0000000..9b1d2c5 --- /dev/null +++ b/test/properties/welch_stream_laws_test.dart @@ -0,0 +1,1119 @@ +// LAWS of the append-only beat streams behind the night's spectral and CVHR +// estimators: `RsaRespStream` (respiration/resp_rate.dart, checkpoint v2), +// `HrvFreqStream` (clinical/hrv_freq_stream.dart, v1) and `CvhrStream` +// (respiration/cvhr_apnea.dart, v1). Each folds settled NN with their beat +// times and answers `snapshot` with a provisional tail; the batch functions +// (`rsaRespRate`, `hrvFreq`, `cvhrApneaScreen`) are the reference. +// +// L1 chunk invariance: any chunking of the folds (empty chunks and a +// save/restore through real JSON text between chunks included) gives +// the checkpoint TEXT of one fold of the whole input and the same +// snapshot under every evaluation. A law of the append transition, not +// a monoid claim (no stream has a `combine`). +// L1b streamed vs batch: at every checked seam `snapshot(tail)` is the +// batch of settled ++ tail BIT FOR BIT (doubles compared by their +// shortest round-trip text: NaN-ness and the sign of zero included), +// notes, confidence and every value field; and the snapshot did not +// write. +// L2 checkpoint round trip: `toJson` -> real JSON text -> `fromJson` -> +// `toJson` is the same text, the restored stream reads what the live +// one reads, is not aliased to the map it came from, and goes on +// identically. +// L3 a checkpoint of another type or version (RsaRespStream version 1, the +// span-based ceiling without gappy sub-windows, included), a missing or +// mistyped part, parts that disagree in a way the reader checks: +// refused whole with a FormatException, never another error type. A +// number changed where the reader cannot tell: refused, or held exactly +// as written. An unknown key: ignored. +// L4 identity: an empty fold changes nothing (on a fresh stream it records +// nothing), `snapshot` never writes, and a refused fold (lengths that +// differ, a non-finite beat time, a time that steps back) leaves the +// stream unchanged. +// L5 conservation, from counts made HERE from the input: the stream counts +// every folded beat once; RsaRespStream keeps exactly the sorted +// multiset of the folded NN and has closed exactly the sub-windows the +// documented walk has finished, and a reported rate rests on as many +// sub-windows as that walk visits; HrvFreqStream's bands have used no +// more segments than they have stepped over; CvhrStream's open stretch +// holds exactly the beats since the last gap over maxGapSec. +// +// A failing law is first checked against the module's contract; a law that +// fails because the library breaks it is fixed in lib/ with a rationale. +// +// A series is a RECIPE (flavour, beats, seed) expanded deterministically. +// Replay a failure with the command in its report, e.g. +// PROPERTY_SEED= PROPERTY_CASE= TZ=UTC dart test \ +// test/properties/welch_stream_laws_test.dart --plain-name '' + +import 'dart:convert'; +import 'dart:math' as math; + +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +import '../support/fold_law_support.dart'; +import '../support/law_registry.dart'; +import '../support/property.dart'; + +final _laws = LawSet(); + +typedef Json = Map; + +// ── the beats ─────────────────────────────────────────────────────────────── + +const _flavours = [ + 'sleep', // ~57 bpm, breathing at 0.24 Hz, LF and VLF waves, a little noise + 'slow heart', // NN ~1420 ms: the RSA ceiling sits under the HF band + 'gappy', // a 40 s dropout every 400 beats, a 150 s one every 700 + 'apnea', // +80 ms half-sine bumps, 20 s wide, every 45 s (CVHR cycles) + 'whole seconds', // beat times floored to whole seconds, ties included + 'outliers', // out-of-range NN (250 / 2600 ms) and an occasional NaN NN +]; +const int _nFlavours = 6; + +/// 2023-11-14 22:13:20 UTC in ms. +const double _t0 = 1.7e12; + +class _Night { + _Night(this.nn, this.t); + final List nn, t; + int get n => nn.length; +} + +final Map<(int, int, int), _Night> _cache = {}; + +_Night _night(int flavour, int n, int seed) => + _cache.putIfAbsent((flavour, n, seed), () { + if (_cache.length > 300) _cache.clear(); + return _expand(flavour, n, seed); + }); + +_Night _expand(int flavour, int n, int seed) { + final r = Rng(seed * 7919 + flavour * 131 + 7); + final phase = r.nextDouble() * 6.28; + final nn = [], t = []; + var clock = _t0; + for (var i = 0; i < n; i++) { + final s = (clock - _t0) / 1000; + final noise = 6 * (r.nextDouble() - .5) * 2; + final sleep = 1050 + + 60 * math.sin(s / 1500 + seed) + + 28 * math.sin(2 * math.pi * .24 * s + phase) + + 18 * math.sin(2 * math.pi * .1 * s) + + 22 * math.sin(2 * math.pi * .01 * s) + + noise; + var v = switch (flavour) { + 1 => 1420 + 30 * math.sin(2 * math.pi * .2 * s + phase) + noise, + 3 => 1000 + + 25 * math.sin(2 * math.pi * .25 * s + phase) + + (s % 45 < 20 ? 80 * math.sin(math.pi * (s % 45) / 20) : 0) + + noise, + _ => sleep, + }; + if (flavour == 5) { + final u = r.nextDouble(); + if (u < .01) { + v = double.nan; + } else if (u < .03) { + v = r.nextBool() ? 250 : 2600; + } + } + v = v.isNaN ? v : v.roundToDouble(); + if (flavour == 2 && i > 0 && i % 400 == 0) clock += 40000; + if (flavour == 2 && i > 0 && i % 700 == 0) clock += 150000; + clock += v.isFinite ? v : 800; + nn.add(v); + t.add(flavour == 4 ? (clock / 1000).floorToDouble() * 1000 : clock); + } + return _Night(nn, t); +} + +// ── exact text of an output ───────────────────────────────────────────────── + +/// A double as its shortest round-trip text: equal text is equal bits up to +/// the NaN payload (NaN and -0.0 are spelled out). +String _d(double? x) => x == null ? 'null' : x.toString(); + +String _envelope(Metric m) => + '${m.present}|${_d(m.confidence)}|${m.tier}|${m.inputs_used}|${m.note}|' + '${m.drivers}'; + +// ── one stream under the laws ─────────────────────────────────────────────── + +abstract class _Ops { + String get name; + String get type; + + /// Constructor configurations (index 0 is the default) and evaluations. + int get nCfg; + int get nEv; + + S fresh(int cfg); + void fold(S s, List nn, List t); + Metric snapshot(S s, int ev, List tailNn, List tailT); + Metric batch(List nn, List t, int cfg, int ev); + String showValue(Object v); + Json toJson(S s); + S fromJson(Json j); + + /// Every version but the current one is refused. + List get badVersions; + + /// Breaks of a reachable checkpoint the reader checks: must be refused. + List get refuseMutations; + + /// Numbers the reader cannot tell from legal ones: refused or held exactly. + List get faithfulMutations; + + /// Whether a fold with a non-finite beat time is refused. + bool get refusesNonFiniteTimes => true; + + /// Counts made from the input ([nn] / [t], everything folded so far). + void conserved(Json cp, List nn, List t, int cfg, String why); + + /// The output as exact text. + String show(Metric m) => + '${_envelope(m)}|${m.value == null ? '-' : showValue(m.value!)}'; + + Map get shares; + void observe(_Case c, void Function(String) bump); + List<_Case> get forced; +} + +/// ((flavour, beats, seed, config), (splits, cut seed, restart mask, evaluation)). +typedef _Case = ((int, int, int, int), (int, int, int, int)); + +class _NGen extends Gen { + _NGen(this.maxN, this.pool); + final int maxN; + final List pool; + @override + int generate(Rng r, int size) { + final p = r.nextDouble(); + if (p < .07) return 0; + if (p < .35) { + final ok = [for (final x in pool) if (x <= maxN) x]; + return ok[r.nextInt(ok.length)]; + } + if (p < .55) return r.intIn(math.min(700, maxN), maxN); + return r.intIn(0, maxN); + } + + @override + Iterable shrink(int v) => G.intIn(0, maxN).shrink(v); +} + +Gen<_Case> _caseGen(_Ops ops, int maxN, List pool) => G.pair( + G.quad(G.intIn(0, _nFlavours - 1), _NGen(maxN, pool), + G.intIn(0, 1 << 16), G.intIn(0, ops.nCfg - 1)), + G.quad(G.intIn(0, 5), G.intIn(0, 1 << 20), G.intIn(0, 127), + G.intIn(0, ops.nEv - 1)), + ); + +String _tag(_Ops ops, _Case c) { + final ((f, n, seed, cfg), (splits, cut, restart, ev)) = c; + return '${ops.name} ${_flavours[f]} n=$n seed=$seed cfg=$cfg ' + 'splits/cut=$splits/$cut restart=$restart ev=$ev'; +} + +/// Beat counts where the streams change behaviour: the batch minimums (16, 20, +/// 60) and the first spans that settle anything. +const _edges = [0, 1, 2, 15, 16, 17, 19, 20, 21, 59, 60, 61]; + +List _boundsOf(_Case c) { + final ((_, n, _, _), (splits, cut, _, _)) = c; + return foldBounds(n, splits, cut, special: _edges); +} + +S _restore(_Ops ops, S s) => + ops.fromJson((jsonDecode(jsonEncode(ops.toJson(s))) as Map).cast()); + +String _text(_Ops ops, S s) => jsonEncode(ops.toJson(s)); + +/// Folds [bounds] chunk by chunk, restoring through JSON text where [restart] +/// says, calling [seam] after each chunk. +S _drive(_Ops ops, _Night x, int cfg, List bounds, int restart, + [void Function(S s, int k, int at)? seam]) { + var s = ops.fresh(cfg); + for (var k = 0; k + 1 < bounds.length; k++) { + ops.fold(s, x.nn.sublist(bounds[k], bounds[k + 1]), + x.t.sublist(bounds[k], bounds[k + 1])); + if ((restart >> (k % 7)) & 1 == 1) s = _restore(ops, s); + seam?.call(s, k, bounds[k + 1]); + } + return s; +} + +// ── the laws, written once ────────────────────────────────────────────────── + +void _l1(_Ops ops, _Case c) { + final ((f, n, seed, cfg), (_, _, restart, _)) = c; + final x = _night(f, n, seed); + final tag = _tag(ops, c); + final whole = ops.fresh(cfg); + ops.fold(whole, x.nn, x.t); + final bounds = _boundsOf(c); + final got = _drive(ops, x, cfg, bounds, restart); + expect(_text(ops, got), _text(ops, whole), + reason: '$tag bounds=$bounds: the checkpoint of one fold'); + for (var e = 0; e < ops.nEv; e++) { + expect(ops.show(ops.snapshot(got, e, const [], const [])), + ops.show(ops.snapshot(whole, e, const [], const [])), + reason: '$tag: evaluation $e reads the same'); + } +} + +void _l1b(_Ops ops, _Case c) { + final ((f, n, seed, cfg), (_, _, restart, ev)) = c; + final x = _night(f, n, seed); + final tag = _tag(ops, c); + final bounds = _boundsOf(c); + final picks = pickSeams(bounds.length - 1); + _drive(ops, x, cfg, bounds, restart, (s, k, at) { + if (!picks.contains(k)) return; + // The tail is what a corrector would still hold provisional. + final end = math.min(n, at + (at * 7 + k * 13) % 400); + final before = _text(ops, s); + final got = ops.snapshot(s, ev, x.nn.sublist(at, end), x.t.sublist(at, end)); + expect(_text(ops, s), before, reason: '$tag: snapshot reads, never writes'); + final want = ops.batch(x.nn.sublist(0, end), x.t.sublist(0, end), cfg, ev); + expect(ops.show(got), ops.show(want), + reason: '$tag seam $k at $at tail to $end: the batch, bit for bit'); + }); +} + +String _project(_Ops ops, S s) => [ + for (var e = 0; e < ops.nEv; e++) ops.show(ops.snapshot(s, e, const [], const [])), + ].join('\n'); + +void _l2(_Ops ops, _Case c) { + final ((f, n, seed, cfg), _) = c; + final x = _night(f, n, seed); + final tag = _tag(ops, c); + final bounds = _boundsOf(c); + final at = bounds.length > 2 ? bounds[1] : n ~/ 2; + final live = ops.fresh(cfg); + ops.fold(live, x.nn.sublist(0, at), x.t.sublist(0, at)); + final text = _text(ops, live); + final map = (jsonDecode(text) as Map).cast(); + final back = ops.fromJson(map); + expect(_text(ops, back), text, reason: '$tag: write(read(b)) == b'); + expect(_project(ops, back), _project(ops, live), + reason: '$tag: the restored stream reads what the live one reads'); + for (final v in map.values) { + if (v is List) v.clear(); + } + expect(_text(ops, back), text, reason: '$tag: not aliased to the map'); + ops.fold(live, x.nn.sublist(at), x.t.sublist(at)); + ops.fold(back, x.nn.sublist(at), x.t.sublist(at)); + expect(_text(ops, back), _text(ops, live), reason: '$tag: goes on identically'); + expect(_project(ops, back), _project(ops, live), reason: tag); +} + +(Json, Must) _mutate(_Ops ops, Json src, Mut m) { + final j = (deepCopy(src) as Map).cast(); + final (kind, a, b) = m; + final keys = j.keys.toList(); + switch (kind % 7) { + case 0: + j['version'] = ops.badVersions[a % ops.badVersions.length]; + return (j, Must.refuse); + case 1: + const types = ['Other', null, '', 1, 'IrregularScreenState']; + j['type'] = types[a % types.length]; + return (j, Must.refuse); + case 2: + j.remove(keys[a % keys.length]); + return (j, Must.refuse); + case 3: + const junk = ['x', {}, [[]], true]; + j[keys[a % keys.length]] = junk[b % junk.length]; + return (j, Must.refuse); + case 4: + final before = jsonEncode(j); + ops.refuseMutations[a % ops.refuseMutations.length](j); + // A break that happens to be a no-op on this checkpoint is no break. + return (j, jsonEncode(j) == before ? Must.faithful : Must.refuse); + case 5: + ops.faithfulMutations[a % ops.faithfulMutations.length](j); + return (j, Must.faithful); + default: + j['futureField${a % 3}'] = const [1, 'x', [1]][b % 3]; + return (j, Must.ignored); + } +} + +String _outcome(_Ops ops, Json j) { + try { + return 'accepted:${_text(ops, ops.fromJson((deepCopy(j) as Map).cast()))}'; + } on FormatException { + return 'refused'; + } catch (e) { + return 'WRONG ERROR TYPE: ${e.runtimeType}: $e'; + } +} + +/// The checkpoint of one fold of a recipe (folding is not the cost under test). +final Map _foldedText = {}; + +void _l3(_Ops ops, (_Case, Mut) arg) { + final (c, m) = arg; + final ((f, n, seed, cfg), _) = c; + final tag = '${_tag(ops, c)} mutation=${m.$1 % 7}(${m.$2},${m.$3})'; + if (_foldedText.length > 300) _foldedText.clear(); + final text = _foldedText.putIfAbsent('${ops.name}|${c.$1}', () { + final x = _night(f, n, seed); + final s = ops.fresh(cfg); + ops.fold(s, x.nn, x.t); + return _text(ops, s); + }); + final src = (jsonDecode(text) as Map).cast(); + final (bad, must) = _mutate(ops, src, m); + expectMutationOutcome( + _outcome(ops, bad), must, jsonEncode(bad), jsonEncode(src), tag); +} + +void _l4(_Ops ops, _Case c) { + final ((f, n, seed, cfg), (_, _, _, ev)) = c; + final x = _night(f, n, seed); + final tag = _tag(ops, c); + final freshText = _text(ops, ops.fresh(cfg)); + final empty = ops.fresh(cfg); + ops.fold(empty, const [], const []); + expect(_text(ops, empty), freshText, reason: '$tag: an empty first fold records nothing'); + final cut = n ~/ 2; + final s = ops.fresh(cfg); + ops.fold(s, x.nn.sublist(0, cut), x.t.sublist(0, cut)); + final before = _text(ops, s); + ops.fold(s, const [], const []); + expect(_text(ops, s), before, reason: '$tag: an empty fold is the identity'); + final one = ops.show(ops.snapshot(s, ev, const [], const [])); + final end = math.min(n, cut + 50); + ops.snapshot(s, ev, x.nn.sublist(cut, end), x.t.sublist(cut, end)); + expect(_text(ops, s), before, reason: '$tag: snapshot never writes'); + expect(ops.show(ops.snapshot(s, ev, const [], const [])), one, + reason: '$tag: twice is equal'); + // Refused folds change nothing. + void refused(String what, void Function() call, Matcher throws) { + expect(call, throws, reason: '$tag: $what is refused'); + expect(_text(ops, s), before, reason: '$tag: $what leaves the stream unchanged'); + } + + refused('lengths that differ', + () => ops.fold(s, const [800.0, 810.0], [_t0 + 9e9]), throwsArgumentError); + if (ops.refusesNonFiniteTimes) { + final last = cut == 0 ? _t0 : x.t[cut - 1]; + for (final bad in [double.nan, double.infinity]) { + refused('a beat time of $bad', + () => ops.fold(s, const [800.0, 810.0], [last + 800, bad]), + throwsArgumentError); + } + } + if (cut > 0) { + refused('a time that steps back', + () => ops.fold(s, const [800.0], [x.t[cut - 1] - 1000]), throwsStateError); + } +} + +void _l5(_Ops ops, _Case c) { + final ((f, n, seed, cfg), (_, _, restart, _)) = c; + final x = _night(f, n, seed); + final tag = _tag(ops, c); + _drive(ops, x, cfg, _boundsOf(c), restart, (s, k, at) { + final cp = (jsonDecode(_text(ops, s)) as Map).cast(); + expect(cp['n'], at, reason: '$tag at $at: every folded beat counted once'); + ops.conserved(cp, x.nn.sublist(0, at), x.t.sublist(0, at), cfg, '$tag at $at'); + }); +} + +Map _sharesOf(_Ops ops) => { + for (final f in _flavours) 'flavour: $f': .05, + 'beats: under 20': .02, + 'beats: past 700': .25, + 'config: not the default': ops.nCfg > 1 ? .2 : 0, + 'chunks: three or more': .3, + 'an empty chunk': .03, + 'a restore between chunks': .2, + for (var e = 0; e < ops.nEv; e++) 'evaluation: $e': .04, + }..removeWhere((_, v) => v == 0); + +void _observe(_Ops ops, _Case c, void Function(String) bump) { + final ((f, n, _, cfg), (_, _, restart, ev)) = c; + bump('flavour: ${_flavours[f]}'); + if (n > 0 && n < 20) bump('beats: under 20'); + if (n > 700) bump('beats: past 700'); + if (cfg != 0) bump('config: not the default'); + final b = _boundsOf(c); + if (b.length >= 4) bump('chunks: three or more'); + if (hasEmptyChunk(b)) bump('an empty chunk'); + if (restart != 0 && b.length > 2) bump('a restore between chunks'); + bump('evaluation: $ev'); + ops.observe(c, bump); +} + +void _register(_Ops ops, {int maxN = 1200, int cases = 60}) { + const pool = [0, 1, 15, 16, 19, 20, 59, 60, 61, 300, 700, 1200]; + final shares = {..._sharesOf(ops), ...ops.shares}; + final reach = Reach<_Case>(shares, (c, bump) => _observe(ops, c, bump)); + final noEval = Reach<_Case>({ + for (final e in shares.entries) + if (!e.key.startsWith('evaluation: ')) e.key: e.value + }, (c, bump) => _observe(ops, c, bump)); + group(ops.name, () { + _laws.law<_Case>( + 'L1 ${ops.name}: any chunking, empty chunks and restores included, gives ' + 'the checkpoint text and every snapshot of one fold', + _caseGen(ops, maxN, pool), + (c) => _l1(ops, c), + examples: ops.forced, + cases: cases, + reach: noEval, + ); + _laws.law<_Case>( + 'L1b ${ops.name}: snapshot(tail) at a seam is the batch of settled ++ ' + 'tail, bit for bit', + _caseGen(ops, maxN, pool), + (c) => _l1b(ops, c), + examples: ops.forced, + cases: cases, + reach: reach, + ); + _laws.law<_Case>( + 'L2 ${ops.name}: write(read(b)) == b, the restored stream reads the same ' + 'and goes on identically', + _caseGen(ops, maxN, pool), + (c) => _l2(ops, c), + examples: ops.forced, + cases: cases, + reach: noEval, + ); + _laws.law<(_Case, Mut)>( + 'L3 ${ops.name}: another version or type, a missing, mistyped or ' + 'inconsistent part is refused whole; a changed number is held exactly', + G.pair(_caseGen(ops, 1200, pool), mutGen(7)), + (arg) => _l3(ops, arg), + examples: [ + for (var i = 0; i < ops.forced.length; i++) + for (var k = 0; k < 7; k++) (ops.forced[i], (k, 3 * i + k, 5 * i + 2 * k)), + for (var a = 0; a < ops.badVersions.length; a++) (ops.forced.last, (0, a, 0)), + for (var a = 0; a < 40; a++) (ops.forced.last, (2, a, 0)), + for (var a = 0; a < ops.refuseMutations.length; a++) (ops.forced.last, (4, a, 0)), + for (var a = 0; a < ops.faithfulMutations.length; a++) (ops.forced.last, (5, a, 0)), + ], + cases: 100, + reach: Reach<(_Case, Mut)>({ + for (var k = 0; k < 7; k++) 'kind: $k': .05, + }, (arg, bump) => bump('kind: ${arg.$2.$1 % 7}')), + ); + _laws.law<_Case>( + 'L4 ${ops.name}: an empty fold is the identity, snapshot never writes, a ' + 'refused fold leaves the stream unchanged', + _caseGen(ops, maxN, pool), + (c) => _l4(ops, c), + examples: ops.forced, + cases: cases, + reach: reach, + ); + _laws.law<_Case>( + 'L5 ${ops.name}: every folded beat is counted once, and what the stream ' + 'kept and closed is what the input says', + _caseGen(ops, maxN, pool), + (c) => _l5(ops, c), + examples: ops.forced, + cases: cases, + reach: noEval, + ); + }); +} + +_Case _c(int flavour, int n, + {int seed = 3, int cfg = 0, int splits = 4, int cut = 1, int restart = 0, int ev = 0}) => + ((flavour, n, seed, cfg), (splits, cut, restart, ev)); + +// ═════════════════════════════════════════════════════════════════════════════ +// RsaRespStream +// ═════════════════════════════════════════════════════════════════════════════ + +/// (artifactFraction, tolBrpm, minConsensus, maxArtifact). +const List<(double, double, double, double)> _rsaEvals = [ + (.02, 2.0, .5, .30), + (0.0, 2.0, .5, .30), + (.31, 2.0, .5, .30), // over the artifact gate + (.30, 2.0, .5, .30), // exactly at it: passes + (.10, 1.0, .7, .30), // a tighter agreement test + (.05, 3.0, .3, .50), +]; + +/// The sub-window starts of the batch walk over [t] (seconds), and the +/// segment length it uses. +(int, double) _rsaWalk(List tSec) { + final span = tSec.last - tSec.first; + var segSec = rsaSegmentSec; + if (span < 2 * segSec) segSec = span / 2; + var k = 0; + for (var i = 0;; i++) { + final s = tSec.first + i * (segSec / 2); + if (s + segSec > tSec.last + 1e-9) break; + k++; + } + return (k, segSec); +} + +class _RsaOps extends _Ops { + @override + String get name => 'RsaRespStream'; + @override + String get type => 'RsaRespStream'; + @override + int get nCfg => 1; + @override + int get nEv => _rsaEvals.length; + @override + RsaRespStream fresh(int cfg) => RsaRespStream(); + @override + void fold(RsaRespStream s, List nn, List t) => s.fold(nn, t); + @override + Metric snapshot(RsaRespStream s, int ev, List tailNn, List tailT) { + final e = _rsaEvals[ev]; + return s.snapshot( + tailNnMs: tailNn, + tailNnTimesMs: tailT, + artifactFraction: e.$1, + tolBrpm: e.$2, + minConsensus: e.$3, + maxArtifact: e.$4); + } + + @override + Metric batch(List nn, List t, int cfg, int ev) { + final e = _rsaEvals[ev]; + return rsaRespRate(nn, t, + artifactFraction: e.$1, tolBrpm: e.$2, minConsensus: e.$3, maxArtifact: e.$4); + } + + @override + String showValue(Object v) { + final r = v as RespEstimate; + return '${_d(r.brpm)}|${_d(r.peakHz)}|${_d(r.power)}|${r.source}|' + '${r.usableSubwindows}|${r.subwindows}'; + } + + @override + Json toJson(RsaRespStream s) => s.toJson(); + @override + RsaRespStream fromJson(Json j) => RsaRespStream.fromJson(j); + + @override + List get badVersions => + const [1, 0, 3, -1, '2', null, true, 2.5, 1 << 40]; + + @override + List get refuseMutations => [ + (j) => (j['nn'] as List).add(800.0), // nn and times disagree + (j) => (j['sorted'] as List).add(800.0), // more sorted values than beats + (j) { + final s = j['sorted'] as List; + if (s.length >= 2 && s.first != s.last) { + j['sorted'] = s.reversed.toList(); // not sorted + } + }, + (j) { + if ((j['n'] as int) > 0) j['first'] = null; // beats without a start + }, + (j) { + if ((j['n'] as int) == 0) j['first'] = 0.0; // a start without beats + }, + (j) => (j['closed'] as List).add({ + 'kind': 5, // no such sub-window outcome + 'brpm': null, + 'peakHz': null, + 'power': null, + 'ceiling': null + }), + (j) => (j['closed'] as List).add({ + 'kind': 0, // a usable peak without its numbers + 'brpm': null, + 'peakHz': null, + 'power': null, + 'ceiling': null + }), + (j) => (j['closed'] as List).add({ + 'kind': 1, // at the ceiling, without the ceiling + 'brpm': null, + 'peakHz': null, + 'power': null, + 'ceiling': null + }), + (j) { + final t = j['times'] as List; + if (t.length >= 2 && t.first != t.last) { + j['times'] = t.reversed.toList(); // times that step back + } + }, + ]; + + @override + List get faithfulMutations => [ + (j) { + for (final c in j['closed'] as List) { + final m = c as Map; + if (m['kind'] == 0) { + m['brpm'] = (m['brpm'] as num) + 1.0; + return; + } + } + }, + (j) { + if (j['nextStart'] != null) j['nextStart'] = (j['nextStart'] as num) + 1.0; + }, + (j) { + final nn = j['nn'] as List; + if (nn.isNotEmpty && nn[0] is num) nn[0] = (nn[0] as num) + 1.0; + }, + (j) => (j['closed'] as List).add({ + 'kind': 3, // one more thin sub-window + 'brpm': null, + 'peakHz': null, + 'power': null, + 'ceiling': null + }), + ]; + + @override + void conserved(Json cp, List nn, List t, int cfg, String why) { + // The sorted multiset of every folded NN, in double.compareTo order. + final sorted = [...nn]..sort(); + expect(jsonEncode(cp['sorted']), + jsonEncode([for (final v in sorted) v.isFinite ? v : v.toString()]), + reason: 'the sorted NN are every folded NN once $why'); + // Closed sub-windows: the starts first + i*150 s whose window ends before + // the last beat (by more than 1e-9 s), once the span reaches 600 s. + final closed = (cp['closed'] as List).length; + var want = 0; + if (t.isNotEmpty) { + final first = t.first / 1000, last = t.last / 1000; + if (last - first >= 2 * rsaSegmentSec) { + while (first + want * (rsaSegmentSec / 2) + rsaSegmentSec < last - 1e-9) { + want++; + } + } + } + expect(closed, want, reason: 'closed sub-windows $why'); + // A reported rate rests on the sub-windows the batch walk visits. + final m = snapshot(fromJson(cp), 1, const [], const []); + if (m.present) { + final r = m.value as RespEstimate; + final (k, _) = _rsaWalk([for (final x in t) x / 1000]); + expect(r.subwindows, k, reason: 'sub-windows tried $why'); + expect(r.usableSubwindows! >= 3 && r.usableSubwindows! <= k, isTrue, + reason: 'usable sub-windows $why'); + } + } + + @override + Map get shares => { + 'a rate': .05, + 'withheld below the HF band': .02, + 'a gap that empties a sub-window': .01, + }; + + @override + void observe(_Case c, void Function(String) bump) { + final ((f, n, _, _), _) = c; + if (n > 700 && (f == 0 || f == 4)) bump('a rate'); + if (n > 30 && f == 1) bump('withheld below the HF band'); + if (n > 750 && f == 2) bump('a gap that empties a sub-window'); + } + + @override + List<_Case> get forced => [ + _c(0, 0), + _c(0, 19, ev: 1), + _c(0, 20, ev: 1), + _c(0, 700, splits: 5, cut: 2, restart: 85), + _c(0, 1600, splits: 6, cut: 3, restart: 127, ev: 1), + _c(1, 900, ev: 1), // withheld: the ceiling is under the band + _c(2, 1600, splits: 6, cut: 4, restart: 42, ev: 1), // gappy + _c(4, 1800, splits: 5, cut: 5, ev: 4), // whole seconds + _c(5, 1500, splits: 4, cut: 6, restart: 3, ev: 5), // outliers and NaN + _c(3, 1500, ev: 2), // over the artifact gate + _c(0, 1500, splits: 6, cut: 7, ev: 3), // exactly at it + ]; +} + +// ═════════════════════════════════════════════════════════════════════════════ +// HrvFreqStream +// ═════════════════════════════════════════════════════════════════════════════ + +const _oversamples = [4.0, 2.0, 8.0]; + +/// (artifactFraction, hfArtifactGate). +const List<(double, double)> _hrvEvals = [ + (.02, .15), + (.20, .15), // HF gated + (.15, .15), // exactly at the gate: not gated + (0.0, .05), +]; + +/// Band lower edges: the segment of a band is 10 / lo seconds. +const _bandLo = [0.0003, 0.003, 0.04, 0.15]; + +class _HrvOps extends _Ops { + @override + String get name => 'HrvFreqStream'; + @override + String get type => 'HrvFreqStream'; + @override + int get nCfg => _oversamples.length; + @override + int get nEv => _hrvEvals.length; + @override + HrvFreqStream fresh(int cfg) => HrvFreqStream(oversample: _oversamples[cfg]); + @override + void fold(HrvFreqStream s, List nn, List t) => s.fold(nn, t); + @override + Metric snapshot(HrvFreqStream s, int ev, List tailNn, List tailT) { + final e = _hrvEvals[ev]; + return s.snapshot( + tailNnMs: tailNn, + tailNnTimesMs: tailT, + artifactFraction: e.$1, + hfArtifactGate: e.$2); + } + + @override + Metric batch(List nn, List t, int cfg, int ev) { + final e = _hrvEvals[ev]; + return hrvFreq(nn, t, + artifactFraction: e.$1, hfArtifactGate: e.$2, oversample: _oversamples[cfg]); + } + + @override + String showValue(Object v) { + final h = v as HrvFreq; + return [ + for (final x in [h.ulf, h.vlf, h.lf, h.hf, h.total, h.lfhf, h.nuLf, h.nuHf]) _d(x), + h.totalBands, + h.hfGated, + ].join('|'); + } + + @override + Json toJson(HrvFreqStream s) => s.toJson(); + @override + HrvFreqStream fromJson(Json j) => HrvFreqStream.fromJson(j); + + @override + List get badVersions => const [0, 2, -1, '1', null, true, 1.5, 1 << 40]; + + @override + List get refuseMutations => [ + (j) => (j['nn'] as List).add(800.0), // nn and times disagree + (j) => (j['bands'] as List).removeLast(), // three bands + (j) => j['oversample'] = 0.0, + (j) => j['oversample'] = -4.0, + (j) { + if ((j['n'] as int) > 0) j['last'] = null; + }, + (j) { + if ((j['n'] as int) == 0) j['first'] = 0.0; + }, + (j) => ((j['bands'] as List).first as Map)['count'] = -1, + (j) { + final t = j['times'] as List; + if (t.length >= 2 && t.first != t.last) j['times'] = t.reversed.toList(); + }, + ]; + + @override + List get faithfulMutations => [ + (j) { + final b = (j['bands'] as List)[2] as Map; + b['sum'] = (b['sum'] as num) + 1.0; + }, + (j) { + final b = (j['bands'] as List)[3] as Map; + b['count'] = (b['count'] as int) + 1; + }, + (j) { + final b = (j['bands'] as List)[1] as Map; + b['nextStart'] = (b['nextStart'] as num) + 1.0; + }, + (j) => j['oversample'] = 3.0, + ]; + + @override + void conserved(Json cp, List nn, List t, int cfg, String why) { + // A band has used at most the segments it has stepped over, and has stepped + // only over segments that end by the last beat. + final bands = cp['bands'] as List; + for (var i = 0; i < 4; i++) { + final b = bands[i] as Map; + final count = b['count'] as int; + if (t.isEmpty) { + expect(count, 0, reason: 'no beats, no segments $why'); + continue; + } + final segSec = 10.0 / _bandLo[i]; + final first = t.first / 1000, last = t.last / 1000; + var stepped = 0; + for (var s = first; s + segSec <= last; s += segSec / 2) { + stepped++; + } + final next = (b['nextStart'] as num).toDouble(); + var walked = 0; + for (var s = first; s < next; s += segSec / 2) { + walked++; + } + expect(walked, stepped, reason: 'band $i stepped over every ended segment $why'); + expect(count <= stepped, isTrue, reason: 'band $i used segments $why'); + } + } + + @override + Map get shares => { + 'LF and HF resolved': .05, + 'gated HF': .02, + }; + + @override + void observe(_Case c, void Function(String) bump) { + final ((f, n, _, _), (_, _, _, ev)) = c; + if (n > 700 && f != 5) bump('LF and HF resolved'); + if (n > 700 && ev == 1) bump('gated HF'); + } + + @override + List<_Case> get forced => [ + _c(0, 0), + _c(0, 15, ev: 1), + _c(0, 16, ev: 1), + _c(0, 700, splits: 5, cut: 2, restart: 85), + _c(0, 1600, splits: 6, cut: 3, restart: 127, cfg: 1), + _c(2, 1600, splits: 6, cut: 4, restart: 42, cfg: 2, ev: 1), // gappy, gated + _c(4, 1800, splits: 5, cut: 5, ev: 2), // whole seconds, at the gate + _c(5, 1500, splits: 4, cut: 6, restart: 3, ev: 3), // outliers and NaN + _c(1, 1500, ev: 0), + _c(3, 1500, splits: 6, cut: 7, ev: 0), + ]; +} + +// ═════════════════════════════════════════════════════════════════════════════ +// CvhrStream +// ═════════════════════════════════════════════════════════════════════════════ + +/// (minWidthSec, maxWidthSec, depthWidthRatioMin, envWindowSec, maxGapSec). +const List<(double, double, double, double, double)> _cvhrCfgs = [ + (10, 120, .7, 130, 30), + (5, 60, .5, 90, 10), + (10, 120, 1.2, 200, 60), +]; + +/// (artifactFraction, maxArtifact). +const List<(double, double)> _cvhrEvals = [ + (.02, .30), + (.31, .30), // over the gate + (.30, .30), // exactly at it + (0.0, .10), +]; + +class _CvhrOps extends _Ops { + @override + String get name => 'CvhrStream'; + @override + String get type => 'CvhrStream'; + @override + int get nCfg => _cvhrCfgs.length; + @override + int get nEv => _cvhrEvals.length; + @override + CvhrStream fresh(int cfg) { + final c = _cvhrCfgs[cfg]; + return CvhrStream( + minWidthSec: c.$1, + maxWidthSec: c.$2, + depthWidthRatioMin: c.$3, + envWindowSec: c.$4, + maxGapSec: c.$5); + } + + @override + void fold(CvhrStream s, List nn, List t) => s.fold(nn, t); + @override + Metric snapshot(CvhrStream s, int ev, List tailNn, List tailT) { + final e = _cvhrEvals[ev]; + return s.snapshot( + tailNnMs: tailNn, tailNnTimesMs: tailT, artifactFraction: e.$1, maxArtifact: e.$2); + } + + @override + Metric batch(List nn, List t, int cfg, int ev) { + final c = _cvhrCfgs[cfg]; + final e = _cvhrEvals[ev]; + return cvhrApneaScreen(nn, t, + artifactFraction: e.$1, + maxArtifact: e.$2, + minWidthSec: c.$1, + maxWidthSec: c.$2, + depthWidthRatioMin: c.$3, + envWindowSec: c.$4, + maxGapSec: c.$5); + } + + @override + String showValue(Object v) { + final r = v as CvhrResult; + return '${r.cycleCount}|${_d(r.cvhrPerHour)}|${_d(r.analyzedHours)}|' + '${_d(r.meanDepthMs)}|${_d(r.meanWidthSec)}|' + '${r.depthQuartilesMs?.map(_d).toList()}|${r.widthQuartilesSec?.map(_d).toList()}'; + } + + @override + Json toJson(CvhrStream s) => s.toJson(); + @override + CvhrStream fromJson(Json j) => CvhrStream.fromJson(j); + + @override + List get badVersions => const [0, 2, -1, '1', null, true, 1.5, 1 << 40]; + + @override + List get refuseMutations => [ + (j) => (j['rawNn'] as List).add(800.0), // rawNn and rawTimes disagree + (j) => (j['exc'] as List).add(0.0), // exc and base disagree + (j) => j['openCount'] = (j['n'] as int) + 1, // more open beats than beats + (j) { + if ((j['n'] as int) > 0) j['openStart'] = null; + }, + (j) { + if ((j['n'] as int) == 0) j['first'] = 0.0; + }, + (j) => j['maxGapSec'] = 'Infinity', + (j) => (j['closed'] as List).add({ + 'hours': -1.0, // negative observed hours + 'depths': [], + 'widths': [] + }), + (j) => (j['closed'] as List).add({ + 'hours': 1.0, // depths and widths disagree + 'depths': [1.0], + 'widths': [] + }), + ]; + + @override + List get faithfulMutations => [ + (j) { + final g = j['grid'] as List; + if (g.isNotEmpty) g[0] = (g[0] as num) + 1.0; + }, + (j) => (j['closed'] as List).add({ + 'hours': 0.5, + 'depths': [12.0], + 'widths': [20.0] + }), + (j) => j['envWindowSec'] = (j['envWindowSec'] as num) + 2.0, + ]; + + @override + void conserved(Json cp, List nn, List t, int cfg, String why) { + // The open stretch holds exactly the beats since the last gap longer than + // maxGapSec (all of them when there is none). + final gap = _cvhrCfgs[cfg].$5; + var open = 0; + for (var i = 0; i < t.length; i++) { + if (i > 0 && t[i] / 1000 - t[i - 1] / 1000 > gap) open = 0; + open++; + } + expect(cp['openCount'], open, reason: 'beats of the open stretch $why'); + // Closed stretches observe no more hours than the input spans. + var hours = 0.0; + for (final c in cp['closed'] as List) { + hours += ((c as Map)['hours'] as num).toDouble(); + } + final span = t.isEmpty ? 0.0 : (t.last - t.first) / 3.6e6; + expect(hours <= span + 1e-9, isTrue, reason: 'closed hours within the span $why'); + } + + @override + Map get shares => { + 'cycles to find': .02, + 'several stretches': .03, + }; + + @override + void observe(_Case c, void Function(String) bump) { + final ((f, n, _, _), _) = c; + if (n > 300 && f == 3) bump('cycles to find'); + if (n > 400 && f == 2) bump('several stretches'); + } + + @override + List<_Case> get forced => [ + _c(0, 0), + _c(3, 59, ev: 0), + _c(3, 60, ev: 0), + _c(3, 700, splits: 5, cut: 2, restart: 85), + _c(3, 1600, splits: 6, cut: 3, restart: 127, cfg: 1), + _c(2, 1600, splits: 6, cut: 4, restart: 42, cfg: 2), // gappy + _c(4, 1800, splits: 5, cut: 5, ev: 2), // whole seconds, at the gate + _c(5, 1500, splits: 4, cut: 6, restart: 3, ev: 3), // outliers and NaN + _c(3, 1500, ev: 1), // over the gate + _c(0, 1500, splits: 6, cut: 7), + ]; +} + +void main() { + _register(_RsaOps()); + _register(_HrvOps()); + _register(_CvhrOps()); + + group('the forced cases say what they claim', () { + test('RSA: a rate, a withheld slow heart, gappy sub-windows', () { + final ops = _RsaOps(); + final notes = []; + var rate = false; + for (final c in ops.forced) { + final ((f, n, seed, cfg), (_, _, _, ev)) = c; + final x = _night(f, n, seed); + final m = ops.batch(x.nn, x.t, cfg, ev); + if (m.present) rate = true; + notes.add(m.note ?? ''); + } + expect(rate, isTrue, reason: 'a rate among the forced nights'); + expect(notes.any((s) => s.startsWith('sleeping heart rate')), isTrue, + reason: 'a night withheld below the HF band'); + expect(notes.any((s) => s.contains('artifact fraction')), isTrue); + }); + + test('HrvFreq: resolved bands, a gated HF', () { + final ops = _HrvOps(); + var resolved = false, gated = false; + for (final c in ops.forced) { + final ((f, n, seed, cfg), (_, _, _, ev)) = c; + final x = _night(f, n, seed); + final m = ops.batch(x.nn, x.t, cfg, ev); + final v = m.value as HrvFreq?; + if (v != null && v.lf != null && v.hf != null) resolved = true; + if (v != null && v.hfGated) gated = true; + } + expect(resolved, isTrue); + expect(gated, isTrue); + }); + + test('CVHR: cycles found, an artifact abstention', () { + final ops = _CvhrOps(); + var cycles = false, gate = false; + for (final c in ops.forced) { + final ((f, n, seed, cfg), (_, _, _, ev)) = c; + final x = _night(f, n, seed); + final m = ops.batch(x.nn, x.t, cfg, ev); + if (m.present && (m.value as CvhrResult).cycleCount > 0) cycles = true; + if (!m.present && (m.note ?? '').contains('artifact fraction')) gate = true; + } + expect(cycles, isTrue); + expect(gate, isTrue); + }); + }); + + _laws.registerReachTest(); +} diff --git a/test/registry_closure_test.dart b/test/registry_closure_test.dart new file mode 100644 index 0000000..857d4be --- /dev/null +++ b/test/registry_closure_test.dart @@ -0,0 +1,414 @@ +// Design 06 phase 1b: the group registry is complete and honest. +// +// RED until `kAnalyticsGroupRegistry` declares the real groups. The registry +// is a hand-written claim ("this group's output depends on these files"), so +// the test re-derives the truth from the source and compares: +// +// (a) the registry validates (ids, deps, cycles, <=3 members, counter >= 1); +// (b) every member is a real public top-level name of the package; +// (c) the import closure of each group's member files stays inside the +// group's own `files` plus those of the foundations it depends on; +// (d) every lib/src file reachable from any exported function belongs to +// some group. +// +// HOW (b) KNOWS THE PUBLIC API. The package has no runtime or analyzer +// dependency (pure Dart, zero deps, by design), so the test does not link the +// analyzer. It follows `export` directives from lib/onehz.dart (honouring +// `show`/`hide`, which this package uses) and reads top-level declarations at +// column 0 after stripping comments. That reads the same files the compiler +// does, needs no hand-maintained map to rot, and a rename or a removed export +// fails here. Its one weakness is an unusual top-level layout the regex does +// not see; the sanity test below pins known names so a scanner regression is +// loud rather than silently permissive. +// +// HOW (c)/(d) WALK. Directives `import`, `export`, `part`, `part of` that name +// a lib/src file (relative, or package:openstrap_analytics/src/...) are edges. +// A `part` file is the same compilation unit, so it is an edge both ways. +import 'dart:io'; + +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +const _registryFile = 'lib/src/onehz/registry/group_registry.dart'; +const _pkg = 'package:openstrap_analytics/'; + +final _root = Directory.current.path; + +String _read(String rel) => File('$_root/$rel').readAsStringSync(); + +String _stripComments(String s) => s + .replaceAll(RegExp(r'/\*[\s\S]*?\*/'), '') + .replaceAll(RegExp(r'^\s*//.*$', multiLine: true), ''); + +String _norm(String from, String target) { + if (target.startsWith(_pkg)) return 'lib/${target.substring(_pkg.length)}'; + final base = from.substring(0, from.lastIndexOf('/')); + final parts = [...base.split('/')]; + for (final seg in target.split('/')) { + if (seg == '..') { + parts.removeLast(); + } else if (seg != '.' && seg.isNotEmpty) { + parts.add(seg); + } + } + return parts.join('/'); +} + +/// One whole directive statement; its URIs are pulled out separately because a +/// conditional import/export names several (`'a.dart' if (cond) 'b.dart'`). +final _directive = RegExp( + r'^(import|export|part of|part)\s+([^;]*);', + multiLine: true); +final _quoted = RegExp(r'''(['"])(.*?)\1'''); + +typedef _Directive = ({ + String kind, + String path, + List show, + List hide +}); + +/// lib/src files named by directive source [src] (the text of [file]), with +/// their kind and filters. Every URI of a conditional directive is an edge +/// (the closure must cover each platform's alternative), and both quote +/// styles are read. `show`/`hide` follow the last URI. +List<_Directive> _parseDirectives(String file, String src) { + final out = <_Directive>[]; + for (final m in _directive.allMatches(_stripComments(src))) { + final body = m.group(2)!; + final uris = _quoted.allMatches(body).toList(); + if (uris.isEmpty) continue; // `part of name;` + final rest = body.substring(uris.last.end); + List names(String kw) { + final mm = RegExp('$kw\\s+([\\w\\s,]+)').firstMatch(rest); + if (mm == null) return const []; + return mm.group(1)!.split(',').map((e) => e.trim()).where((e) => e.isNotEmpty).toList(); + } + + for (final u in uris) { + final target = u.group(2)!; + if (target.startsWith('dart:')) continue; + if (target.startsWith('package:') && !target.startsWith(_pkg)) continue; + final path = _norm(file, target); + if (!path.startsWith('lib/src/')) continue; + out.add((kind: m.group(1)!, path: path, show: names('show'), hide: names('hide'))); + } + } + return out; +} + +List<_Directive> _directives(String file) => + _parseDirectives(file, _read(file)); + +final _typeDecl = RegExp( + r'^(?:(?:abstract|final|base|sealed|interface|mixin)\s+)*(?:class|mixin|enum|typedef|extension type)\s+(\w+)', + multiLine: true); +final _fnDecl = RegExp( + r'^(?!(?:import|export|part|library|class|enum|typedef|const|final|var|late|abstract|mixin|extension|static|return|if|for|while|switch|else|sealed|base|interface)\b)[A-Za-z_][\w<>,?\[\]. ]*?\s(\w+)\s*(?:<[^>(]*>)?\(', + multiLine: true); + +/// Public top-level names declared in [file] itself, split by kind. +({Set fns, Set types}) _declared(String file) { + final src = _stripComments(_read(file)); + final types = {for (final m in _typeDecl.allMatches(src)) m.group(1)!}; + final fns = {for (final m in _fnDecl.allMatches(src)) m.group(1)!} + ..removeWhere((n) => n.startsWith('_') || types.contains(n)); + types.removeWhere((n) => n.startsWith('_')); + return (fns: fns, types: types); +} + +/// name -> declaring file, for everything lib/onehz.dart exports (public API). +/// [fnsOnly] keeps top-level functions only (constructors/types excluded). +Map _publicApi( + {bool fnsOnly = false, List libs = const ['lib/onehz.dart']}) { + final api = {}; + final seen = {}; + // Returns the names [file] exposes to an importer. + Map expose(String file) { + final out = {}; + final d = _declared(file); + for (final n in d.fns) { + out[n] = file; + } + if (!fnsOnly) { + for (final n in d.types) { + out[n] = file; + } + } + for (final e in _directives(file).where((e) => e.kind == 'export')) { + final sub = expose(e.path); + for (final en in sub.entries) { + if (e.show.isNotEmpty && !e.show.contains(en.key)) continue; + if (e.hide.contains(en.key)) continue; + out[en.key] = en.value; + } + } + return out; + } + + // The libraries are not under lib/src, so read their exports directly. + for (final lib in libs) { + for (final e in _directives(lib).where((e) => e.kind == 'export')) { + if (!seen.add(e.path)) continue; + for (final en in expose(e.path).entries) { + if (e.show.isNotEmpty && !e.show.contains(en.key)) continue; + if (e.hide.contains(en.key)) continue; + api[en.key] = en.value; + } + } + } + return api; +} + +/// Every lib/src file reachable from [start] over imports/exports/parts +/// (parts count both ways: a `part of` file belongs to its library). +Set _closure(Iterable start) { + final seen = {}; + final queue = [...start]; + while (queue.isNotEmpty) { + final f = queue.removeLast(); + if (!seen.add(f)) continue; + for (final d in _directives(f)) { + queue.add(d.path); + } + } + return seen; +} + +/// Edge also calls `prsaAggregator`, exported by the second public library +/// (lib/aggregate.dart), so members and api entries may name either. +const _publicLibs = ['lib/onehz.dart', 'lib/aggregate.dart']; + +/// Declaring file of member [name] (`fn`, `Type`, or `Type.method`), or null. +String? _memberFile(Map api, String name) { + final dot = name.indexOf('.'); + if (dot < 0) return api[name]; + final file = api[name.substring(0, dot)]; + if (file == null) return null; + final method = name.substring(dot + 1); + return _stripComments(_read(file)).contains(RegExp('\\b$method\\s*[(<]')) + ? file + : null; +} + +/// `name -> group ids` for every name owned by more than one slot: a name in +/// two groups, or in one group's members and api (or twice in either). +Map> _duplicateOwners(GroupRegistry r) { + final owners = >{}; + for (final g in r.groups) { + for (final m in [...g.members, ...g.api]) { + (owners[m] ??= []).add(g.id); + } + } + return owners..removeWhere((_, v) => v.length < 2); +} + +Set _transitiveDeps(GroupRegistry r, GroupDecl g) { + final byId = {for (final x in r.groups) x.id: x}; + final out = {}; + final q = [...g.deps]; + while (q.isNotEmpty) { + final d = q.removeLast(); + if (d == g.id || !out.add(d)) continue; + q.addAll(byId[d]?.deps ?? const []); + } + return out; +} + +const _expectedGroups = [ + 'sleep.stage', + 'hrv.freq', + 'night.rhythm', + 'day.blocks', + 'readiness.stress_illness', + 'activity.load', + 'body.spo2_temp_coverage', +]; + +void main() { + const reg = kAnalyticsGroupRegistry; + + group('scanner sanity (guards the guard)', () { + test('finds known public functions and types', () { + final api = _publicApi(); + expect(api['hrvFreq'], isNotNull); + expect(api['dispersionBelowQuantum'], isNotNull); + expect(api['RunningMoments'] ?? api['RrCorrector'], isNotNull); + expect(_publicApi(fnsOnly: true).length, greaterThan(100)); + }); + test('closure walk reaches transitive imports', () { + final f = _publicApi()['hrvFreq']!; + final c = _closure([f]); + expect(c, contains(f)); + expect(c.length, greaterThan(1)); + expect(_closure(_publicApi(fnsOnly: true).values.toSet()).length, + greaterThan(60)); + }); + test('hide filters are honoured', () { + expect(_publicApi().containsKey('readinessLnRmssdFromSuffix'), isFalse); + }); + }); + + group('directive parser (guards the guard)', () { + const f = 'lib/src/onehz/a.dart'; + List paths(String src) => + _parseDirectives(f, src).map((d) => d.path).toList(); + test('reads double-quoted directives', () { + expect(paths('import "b.dart";\nexport "c/d.dart" show X;'), + ['lib/src/onehz/b.dart', 'lib/src/onehz/c/d.dart']); + }); + test('reads every URI of a conditional import', () { + expect( + paths("import 'stub.dart'\n if (dart.library.io) 'io.dart'\n" + ' if (dart.library.js_interop) "web.dart" as impl;'), + ['lib/src/onehz/stub.dart', 'lib/src/onehz/io.dart', 'lib/src/onehz/web.dart']); + }); + test('conditional export keeps show/hide for every alternative', () { + final d = _parseDirectives( + f, "export 'a1.dart' if (dart.library.io) 'a2.dart' show Foo, Bar;"); + expect(d.map((e) => e.path), + ['lib/src/onehz/a1.dart', 'lib/src/onehz/a2.dart']); + for (final e in d) { + expect(e.kind, 'export'); + expect(e.show, ['Foo', 'Bar']); + } + }); + test('part, part of by URI, package URIs; skips dart: and foreign packages', () { + expect( + paths("part 'p.dart';\npart of \"q.dart\";\n" + "import 'package:openstrap_analytics/src/onehz/z.dart';\n" + "import 'dart:io';\nimport 'package:other/x.dart';\npart of lib;"), + ['lib/src/onehz/p.dart', 'lib/src/onehz/q.dart', 'lib/src/onehz/z.dart']); + }); + test('commented-out directives are ignored', () { + expect(paths("// import 'gone.dart';\n/* export 'also.dart'; */"), isEmpty); + }); + }); + + group('registry declares the analytics groups', () { + test('declares every expected algorithm group and >= 1 foundation', () { + final ids = reg.groups.map((g) => g.id).toSet(); + expect(ids, containsAll(_expectedGroups)); + expect(ids.where((i) => i.startsWith('found.')), isNotEmpty); + }); + + test('(a) validates: unique ids, deps resolve, acyclic, <=3 members, counter>=1', () { + final problems = []; + final byId = {}; + for (final g in reg.groups) { + if (byId.containsKey(g.id)) problems.add('${g.id}: duplicate id'); + byId[g.id] = g; + if (g.members.isEmpty) problems.add('${g.id}: no members'); + if (g.members.length > 3) problems.add('${g.id}: >3 members'); + if (g.counter < 1) problems.add('${g.id}: counter < 1'); + } + for (final g in reg.groups) { + for (final d in g.deps) { + if (!byId.containsKey(d)) problems.add('${g.id}: dep $d undeclared'); + } + } + final state = {}; + void visit(String id) { + state[id] = 1; + for (final d in byId[id]!.deps) { + if (!byId.containsKey(d)) continue; + if (state[d] == 1) problems.add('$id: cycle through $d'); + if (state[d] == null) visit(d); + } + state[id] = 2; + } + + for (final id in byId.keys) { + if (state[id] == null) visit(id); + } + expect(reg.groups, isNotEmpty, reason: 'registry is empty'); + expect(problems, isEmpty, reason: problems.join('\n')); + }); + + test('every listed file exists under lib/src', () { + final bad = []; + for (final g in reg.groups) { + for (final f in g.files) { + if (!f.startsWith('lib/src/') || !File('$_root/$f').existsSync()) { + bad.add('${g.id}: $f'); + } + } + } + expect(reg.groups, isNotEmpty, reason: 'registry is empty'); + expect(bad, isEmpty, reason: bad.join('\n')); + }); + }); + + test('a function is a member or api entry of exactly one group', () { + final dup = [ + for (final e in _duplicateOwners(reg).entries) + '${e.key}: ${e.value.join(', ')}', + ]; + expect(reg.groups, isNotEmpty, reason: 'registry is empty'); + expect(dup, isEmpty, reason: dup.join('\n')); + }); + + group('duplicate-owner check (guards the guard)', () { + GroupDecl g(String id, List m, [List api = const []]) => + GroupDecl(id: id, kind: GroupKind.algorithm, counter: 1, members: m, api: api); + test('flags one name in two groups', () { + final r = GroupRegistry(1, [g('a', ['f']), g('b', [], ['f'])]); + expect(_duplicateOwners(r).keys, ['f']); + }); + test('flags a name in one group\'s members and api', () { + final r = GroupRegistry(1, [g('a', ['f'], ['f'])]); + expect(_duplicateOwners(r).keys, ['f']); + }); + test('accepts distinct names', () { + final r = GroupRegistry(1, [g('a', ['f']), g('b', ['h'], ['i'])]); + expect(_duplicateOwners(r), isEmpty); + }); + }); + + test('(b) every member is a real public top-level name', () { + final api = _publicApi(libs: _publicLibs); + final bad = [ + for (final g in reg.groups) + for (final m in [...g.members, ...g.api]) + if (_memberFile(api, m) == null) '${g.id}: $m', + ]; + expect(reg.groups, isNotEmpty, reason: 'registry is empty'); + expect(bad, isEmpty, reason: 'not exported: ${bad.join(', ')}'); + }); + + test('(c) a group\'s import closure stays inside its files + foundations', () { + final api = _publicApi(libs: _publicLibs); + final byId = {for (final g in reg.groups) g.id: g}; + final leaks = []; + for (final g in reg.groups) { + final allowed = {...g.files}; + for (final d in _transitiveDeps(reg, g)) { + final dg = byId[d]; + if (dg != null && dg.kind == GroupKind.foundation) { + allowed.addAll(dg.files); + } + } + final start = [ + for (final m in [...g.members, ...g.api]) + if (_memberFile(api, m) != null) _memberFile(api, m)!, + ]; + final missing = _closure(start).difference(allowed) + ..remove(_registryFile); + for (final f in missing.toList()..sort()) { + leaks.add('${g.id} reaches unlisted $f'); + } + } + expect(reg.groups, isNotEmpty, reason: 'registry is empty'); + expect(leaks, isEmpty, reason: leaks.join('\n')); + }); + + test('(d) every file reachable from an exported function is in some group', () { + final api = _publicApi(fnsOnly: true); + final owned = {for (final g in reg.groups) ...g.files}; + final reach = _closure(api.values.toSet())..remove(_registryFile); + final orphans = reach.difference(owned).toList()..sort(); + expect(reg.groups, isNotEmpty, reason: 'registry is empty'); + expect(orphans, isEmpty, + reason: '${orphans.length} unowned: ${orphans.take(20).join(', ')}'); + }); +} diff --git a/test/support/aggregator_laws.dart b/test/support/aggregator_laws.dart new file mode 100644 index 0000000..20a29c0 --- /dev/null +++ b/test/support/aggregator_laws.dart @@ -0,0 +1,128 @@ +// Shared helpers for the column-slice aggregator tests: the law assertion, +// seeded batch generation, and the three ways to evaluate an aggregator over a +// batch (whole, chunked + combined, one state sliced). + +import 'dart:typed_data'; + +import 'package:openstrap_analytics/aggregate.dart'; +import 'package:test/test.dart'; + +import 'fold_law_support.dart' show foldBounds; +import 'property.dart' show Rng; + +/// Asserts that [agg] satisfies every aggregator law on [cases]. The failure +/// message lists each violated law with its detail. +void expectLawful, R>( + Aggregator agg, + List cases, { + ResultEquals? equals, +}) { + final violations = checkLaws(agg, cases, equals: equals); + expect(violations, isEmpty, + reason: violations.isEmpty + ? null + : 'aggregator laws violated:\n' + '${violations.map((v) => ' ${v.law}: ${v.detail}').join('\n')}'); +} + +/// Seeded random double columns. Always starts with the empty column, a +/// singleton and a pair, then [count] columns of length `0..maxLen`. A value +/// is NaN with probability [nanP]; [value] draws the finite ones. +List randomColumns(int seed, + {int count = 30, + int maxLen = 40, + double nanP = .15, + double Function(Rng)? value}) { + final r = Rng(seed); + final draw = value ?? (Rng r) => (r.nextDouble() - .5) * 2000; + DoubleColumn col(int n) => DoubleColumn(Float64List.fromList( + [for (var i = 0; i < n; i++) r.nextBool(nanP) ? double.nan : draw(r)])); + return [ + col(0), + col(1), + col(2), + for (var i = 0; i < count; i++) col(r.intIn(0, maxLen)), + ]; +} + +/// Seeded random int batches, same shape as [randomColumns]. +List> randomIntBatches(int seed, + {int count = 30, int maxLen = 40, int lo = -1000, int hi = 1000}) { + final r = Rng(seed); + ListBatch b(int n) => + ListBatch([for (var i = 0; i < n; i++) r.intIn(lo, hi)]); + return [b(0), b(1), b(2), for (var i = 0; i < count; i++) b(r.intIn(0, maxLen))]; +} + +/// A column of the given values. +DoubleColumn col(List xs) => DoubleColumn(Float64List.fromList(xs)); + +/// Chunk bounds `[0, ...cuts, n]` for a batch of [n] rows. Cuts may repeat or +/// sit on 0 / 1 / 2 / 3 / n-1 / n, so there are empty, singleton and very short +/// chunks. +List randomBounds(Rng r, int n) => + foldBounds(n, r.intIn(0, 6), r.nextInt(1 << 30), special: [0, 1, 2, 3]); + +/// Folds each chunk `[bounds[k], bounds[k+1])` from a fresh state and combines +/// the chunk states left to right, then finishes. With [copy] each chunk is +/// first cut out with `copyRange` and folded as a batch of its own (so the +/// inner aggregator cannot peek at rows before the chunk); otherwise chunks are +/// ranges of the one batch. +R runChunked, R>( + Aggregator agg, B batch, List bounds, + {bool copy = false}) { + S chunk(int a, int b) => copy + ? agg.fold(batch.copyRange(a, b)) + : agg.fold(batch, a, b); + var acc = chunk(bounds[0], bounds[1]); + for (var k = 1; k + 1 < bounds.length; k++) { + acc = agg.combine(acc, chunk(bounds[k], bounds[k + 1])); + } + return agg.finish(acc); +} + +/// Same chunks, combined right to left: `c0 + (c1 + (c2 + ...))`. +R runChunkedRight, R>( + Aggregator agg, B batch, List bounds) { + var acc = agg.fold(batch, bounds[bounds.length - 2], bounds.last); + for (var k = bounds.length - 3; k >= 0; k--) { + acc = agg.combine(agg.fold(batch, bounds[k], bounds[k + 1]), acc); + } + return agg.finish(acc); +} + +/// The 'slicing' path: ONE state, repeated `addSlice` over the consecutive +/// slices `[bounds[k], bounds[k+1])` (empty slices included), then finish. +R runSliced, R>( + Aggregator agg, B batch, List bounds) { + var state = agg.empty(); + for (var k = 0; k + 1 < bounds.length; k++) { + state = agg.addSlice(state, batch, bounds[k], bounds[k + 1]); + } + return agg.finish(state); +} + +/// Plain bounds for the chunk-size-one case: `[0, 1, 2, ..., n]` (`[0, 0]` for +/// an empty batch). +List singletonBounds(int n) => n == 0 ? [0, 0] : [for (var i = 0; i <= n; i++) i]; + +/// Asserts that [build] over every case in [cases] reproduces [reference] on +/// every evaluation path of [checkConversion] (and, with [laws], the aggregator +/// laws). The failure reason lists the first few failures, one per line. +void expectConversion, R>( + Aggregator Function(B input) build, + List cases, + R Function(B input) reference, { + ResultEquals? equals, + int chunkings = 8, + int seed = 0, + bool laws = true, +}) { + final failures = checkConversion(build, cases, reference, + equals: equals, chunkings: chunkings, seed: seed, laws: laws); + expect(failures, isEmpty, + reason: failures.isEmpty + ? null + : 'conversion check failed (${failures.length}):\n' + '${failures.take(5).map((f) => ' ${f.path} (case ${f.caseIndex}): ${f.detail}').join('\n')}'); +} diff --git a/test/support/checkpoint_support.dart b/test/support/checkpoint_support.dart new file mode 100644 index 0000000..97005c3 --- /dev/null +++ b/test/support/checkpoint_support.dart @@ -0,0 +1,60 @@ +// Shared by the module aggregator tests: a mid-stream checkpoint must resume. + +import 'dart:convert'; + +import 'package:openstrap_analytics/aggregate.dart'; +import 'package:test/test.dart'; + +/// Asserts that [agg] is persistable and that a mid-stream checkpoint resumes: +/// fold the first half of [batch], checkpoint it through JSON text, restore, +/// keep folding the second half, and compare with the one-pass result. +void expectCheckpointResumes, R>( + Aggregator agg, + B batch, { + ResultEquals? equals, +}) { + expect(agg.persistable, isTrue); + final half = batch.length ~/ 2; + final env = jsonDecode(jsonEncode(encodeCheckpoint( + agg, agg.fold(batch, 0, half), + kind: 'test', version: 1))); + final restored = decodeCheckpoint(agg, env, kind: 'test', version: 1); + expect(restored, isNotNull); + final got = agg.finish(agg.addSlice(restored as S, batch, half, batch.length)); + final want = agg.run(batch); + final same = equals ?? (R a, R b) => a == b; + expect(same(got, want), isTrue, reason: 'resumed $got vs one-pass $want'); +} + +/// The JSON-text form of [agg]'s state after folding all of [batch], inside a +/// checkpoint envelope (kind 'test', version 1). +Map checkpointOf, R>( + Aggregator agg, B batch) => + (jsonDecode(jsonEncode( + encodeCheckpoint(agg, agg.fold(batch), kind: 'test', version: 1))) + as Map) + .cast(); + +/// Asserts that a checkpoint whose children disagree is refused with null. +/// +/// The undamaged checkpoint of [batch] must restore; then [damage] edits a +/// deep copy of its state JSON (in place, or returning a replacement) and the +/// result must restore as null, because the damage is a relationship between +/// children that each decode on their own. +void expectDamagedCheckpointRefused, R>( + Aggregator agg, + B batch, + Object? Function(Object? state) damage, { + String? reason, +}) { + final good = checkpointOf(agg, batch); + expect(decodeCheckpoint(agg, good, kind: 'test', version: 1), isNotNull, + reason: 'the undamaged checkpoint must restore'); + final state = jsonDecode(jsonEncode(good['state'])); + final damaged = damage(state) ?? state; + expect(jsonEncode(damaged), isNot(jsonEncode(good['state'])), + reason: 'the damage must change the state'); + final env = {...good, 'state': damaged}; + expect(decodeCheckpoint(agg, env, kind: 'test', version: 1), isNull, + reason: reason ?? jsonEncode(env)); +} diff --git a/test/support/fold_law_support.dart b/test/support/fold_law_support.dart new file mode 100644 index 0000000..0443fde --- /dev/null +++ b/test/support/fold_law_support.dart @@ -0,0 +1,133 @@ +// Pieces the fold-law files share (design 05, pilot cluster C2A): chunk +// boundaries, the mutation outcome vocabulary, and fixed-length int-vector +// generators (a recipe is a handful of ints, so a failing input shrinks to a +// few readable numbers instead of thousands of doubles). + +import 'dart:convert'; +import 'dart:math' as math; + +import 'package:test/test.dart'; + +import 'property.dart'; + +/// Chunk boundaries `[0, ...cuts, n]`. Cuts may repeat or sit on 0 / n, which +/// makes empty chunks; half of them are placed on [special] positions (the +/// edges of a module's windows), the rest uniformly. +List foldBounds(int n, int splits, int cutSeed, + {List special = const [0, 1, 2, 3]}) { + final r = Rng(cutSeed + 17); + final cuts = []; + for (var k = 0; k < splits; k++) { + final v = r.nextBool(.5) + ? (r.nextBool(.2) ? n - 1 + r.nextInt(2) : special[r.nextInt(special.length)]) + : r.intIn(0, n); + cuts.add(v.clamp(0, n)); + } + cuts.sort(); + return [0, ...cuts, n]; +} + +bool hasEmptyChunk(List b) { + for (var k = 0; k + 1 < b.length; k++) { + if (b[k + 1] == b[k]) return true; + } + return false; +} + +/// The seams a law checks against an expensive reference or batch parity check: first, middle, last. +Set pickSeams(int seams) => + {0, seams ~/ 2, seams - 1}..removeWhere((k) => k < 0); + +/// What a mutated checkpoint is allowed to do. +enum Must { + /// Refused with a FormatException, nothing else. + refuse, + + /// Refused, or accepted holding exactly what was written. + faithful, + + /// Refused, or accepted as the ORIGINAL (the change is ignored). + ignored, +} + +/// (kind, a, b): which mutation, and two parameters. +typedef Mut = (int, int, int); + +Object? deepCopy(Object? o) => jsonDecode(jsonEncode(o)); + +/// Checks one outcome of reading a mutated checkpoint: [outcome] is `refused`, +/// `accepted:` or `WRONG ERROR TYPE...`. +void expectMutationOutcome(String outcome, Must must, String faithfulText, + String originalText, String tag) { + expect(outcome, isNot(startsWith('WRONG')), reason: '$tag: only FormatException'); + switch (must) { + case Must.refuse: + expect(outcome, 'refused', reason: '$tag: refused whole'); + case Must.faithful: + if (outcome != 'refused') { + expect(outcome, 'accepted:$faithfulText', + reason: '$tag: accepted means exactly what was written'); + } + case Must.ignored: + if (outcome != 'refused') { + expect(outcome, 'accepted:$originalText', + reason: '$tag: an unknown key is ignored, nothing else moves'); + } + } +} + +class MutKindGen extends Gen { + MutKindGen(this.kinds); + final int kinds; + @override + int generate(Rng r, int size) => r.nextInt(kinds); + @override + Iterable shrink(int v) => G.intIn(0, kinds - 1).shrink(v); +} + +Gen mutGen(int kinds) => + G.triple(MutKindGen(kinds), G.intIn(0, 1 << 20), G.intIn(0, 1 << 20)); + +/// A size: the interesting small values first, then growing with the case +/// index up to [maxN] (cost: a property has two seconds). +class SizeGen extends Gen { + SizeGen(this.maxN, {this.pool = const [0, 1, 2, 3, 4, 5, 10]}); + final int maxN; + final List pool; + @override + int generate(Rng r, int size) { + final p = r.nextDouble(); + if (p < .04) return 0; + if (p < .29) { + final ok = [for (final x in pool) if (x <= maxN) x]; + return ok[r.nextInt(ok.length)]; + } + if (p < .40) return r.intIn(math.min(500, maxN), maxN); + final cap = math.max(120, maxN * size ~/ 100); + if (p < .62) return r.intIn(0, math.min(cap, 400)); + return r.intIn(0, math.min(cap, maxN)); + } + + @override + Iterable shrink(int v) => G.intIn(0, maxN).shrink(v); +} + +/// A fixed-length vector of ints, each slot with its own generator, each +/// shrunk on its own. +class IntsGen extends Gen> { + IntsGen(this.slots); + final List> slots; + @override + List generate(Rng r, int size) => [for (final g in slots) g.generate(r, size)]; + @override + Iterable> shrink(List v) sync* { + for (var i = 0; i < slots.length; i++) { + for (final c in slots[i].shrink(v[i])) { + yield [...v.sublist(0, i), c, ...v.sublist(i + 1)]; + } + } + } + + @override + String show(List v) => '$v'; +} diff --git a/test/support/hrv_beats.dart b/test/support/hrv_beats.dart new file mode 100644 index 0000000..ea9839d --- /dev/null +++ b/test/support/hrv_beats.dart @@ -0,0 +1,134 @@ +// Shared HRV test helpers: Beats builders, beat-series generators and the +// hrvTime parity check. + +import 'dart:math' as math; +import 'dart:typed_data'; + +import 'package:openstrap_analytics/aggregate.dart'; +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +import 'property.dart' show Rng; + +typedef Beat = ({double nnMs, double? timeMs}); + +const double hrvT0 = 1000000.0; +const double hrvTol = 1e-9; + +List hrvFields(HrvTime v) => + [v.rmssd, v.sdnn, v.sdann, v.sdnnIndex, v.pnn50, v.diffAcf1]; + +final hrvEqAll = fieldsEqual((v) => [...hrvFields(v), v.nBeats], + relTol: hrvTol, absTol: hrvTol); + +/// Beats from records. A null time becomes NaN (missing double). +Beats hrvBeats(List xs) => Beats( + Float64List.fromList([for (final b in xs) b.nnMs]), + Float64List.fromList([for (final b in xs) b.timeMs ?? double.nan]), + ); + +/// The beat time of the first row, or null for an empty batch. This is the +/// origin the tests give the aggregator. +double? hrvOrigin(Beats b) => b.length == 0 ? null : b.timeMs[0]; + +/// [b] with every time moved so the first beat sits at [origin]. +Beats hrvShifted(Beats b, double origin) { + if (b.length == 0) return b; + final d = origin - b.timeMs[0]; + return Beats(Float64List.fromList(b.nnMs), + Float64List.fromList([for (final t in b.timeMs) t + d])); +} + +/// What hrvTime says, mapped to the aggregator's expectation. +HrvTime hrvExpected(Beats beats, {required bool gapAware}) { + final nn = beats.nnMs.toList(); + final times = [for (final t in beats.timeMs) t.isNaN ? 0.0 : t]; + final m = hrvTime(nn, nnTimesMs: gapAware ? times : null); + return m.value ?? HrvTime(nBeats: beats.length); +} + +void expectHrvParity(HrvTime got, HrvTime want, String tag) { + expect(got.nBeats, want.nBeats, reason: '$tag: nBeats'); + for (final (name, a, b) in [ + ('rmssd', got.rmssd, want.rmssd), + ('sdnn', got.sdnn, want.sdnn), + ('sdann', got.sdann, want.sdann), + ('sdnnIndex', got.sdnnIndex, want.sdnnIndex), + ('pnn50', got.pnn50, want.pnn50), + ('diffAcf1', got.diffAcf1, want.diffAcf1), + ]) { + expect(a == null, b == null, reason: '$tag: $name null-ness ($a vs $b)'); + if (a != null && b != null) { + expect((a - b).abs() <= hrvTol + hrvTol * b.abs(), isTrue, + reason: '$tag: $name $a vs $b'); + } + } +} + +/// Builds a beat series with times. Values are multiples of 0.5 so the gap +/// rule's float arithmetic is exact in both implementations. +/// mode 0: smooth (random walk), mode 1: white noise (jittery, so the +/// RMSSD/pNN50 floor nulls them), mode 2: slow beats (spans several 5-minute +/// bins in few beats). +/// Below the jitter floor, where hrvTime reads the difference runs: +/// mode 3: RSA at a slow heart, ~44 bpm breathing at 0.3 Hz with a drifting +/// heart rate (ACF1 ~ -0.8, but a respiratory line on a low floor exempts +/// it once the runs hold 20 Welch segments); mode 4: a heart rate drifting +/// from ~67 to ~46 bpm under heavy beat-time jitter with a weak breathing +/// line at 0.25 Hz (refused, then published from the 5-min windows on the +/// line steady in Hz once there are 12 of them, ~1 h). +/// [gapP] is the chance a beat is preceded by a hole; [withTimes] false leaves +/// every time NaN. +Beats hrvSeries(Rng r, int n, int mode, + {double gapP = .05, bool withTimes = true}) { + final out = []; + var t = hrvT0; + var nn = mode == 2 ? 1700.0 : 800.0; + for (var i = 0; i < n; i++) { + final s = (t - hrvT0) / 1000; + switch (mode) { + case 3: + nn = (2 * + (1350 + + 80 * math.sin(2 * math.pi * s / 3600) + + 40 * math.sin(2 * math.pi * .3 * s) + + 3 * (r.nextDouble() - .5))) + .roundToDouble() / + 2; + case 4: + nn = (2 * + (900 + + 400 * s / 9000 + + 10 * math.sin(2 * math.pi * .25 * s) + + 40 * (r.nextDouble() - .5))) + .roundToDouble() / + 2; + case 1: + nn = 600 + r.intIn(0, 800) / 1.0; + case 2: + nn = (nn + r.intIn(-60, 60)).clamp(1200, 2000).toDouble(); + default: + nn = (nn + r.intIn(-40, 40)).clamp(450, 1300).toDouble(); + } + if (mode < 3 && r.nextBool(.04)) nn += r.intIn(40, 200); // an occasional big step + var extra = 0.0; + if (i > 0) { + final p = r.nextDouble(); + if (p < gapP) { + // a hole; sometimes a whole 5-minute bin or more + extra = r.nextBool(.15) + ? r.intIn(300000, 700000).toDouble() + : r.intIn(600, 30000).toDouble(); + } else if (mode < 3 && p < gapP + .04) { + extra = [0.5, 1.0, 0.0][r.nextInt(3)]; // right at the 0.5 ms tolerance + } + } + t += nn + extra; + out.add((nnMs: nn, timeMs: withTimes ? t : null)); + } + return hrvBeats(out); +} + +/// Contiguous series with exact constant nn (acf1 undefined: zero variance). +Beats hrvConstant(int n, double nn) => + hrvBeats([for (var i = 1; i <= n; i++) (nnMs: nn, timeMs: hrvT0 + i * nn)]); diff --git a/test/support/law_registry.dart b/test/support/law_registry.dart new file mode 100644 index 0000000..7f78f3a --- /dev/null +++ b/test/support/law_registry.dart @@ -0,0 +1,124 @@ +// LAW REGISTRY — every law states which situations its default cases must +// reach, and one test per file measures that (design 05, generator reach). +// +// A property that never meets the situation it claims to cover is a green test +// that checks nothing. So each law is registered with a [Reach]: named +// situations, each with a share of the law's default case count it must reach, +// and an observer that names the situations of one input WITHOUT running the +// code under test (the situations come from the generator's model). The reach +// test re-draws the law's default cases (the forced examples are not counted: +// they are the part that is hand-placed) and fails when a share is not met. +// +// PROPERTY_REACH_REPORT=1 prints what each law reached, to retune the shares +// after a generator change. +// +// final laws = LawSet(); +// laws.law('L1 ...', gen, body, reach: Reach({...}, observe)); +// laws.registerReachTest(); // once, last in main() + +import 'dart:io' show Platform; +import 'dart:math' as math; + +import 'package:test/test.dart'; + +import 'property.dart'; + +/// What a law's default cases must reach. +class Reach { + const Reach(this.shares, this.observe); + + /// Situation name -> share of the default case count it must reach + /// (need = max(1, floor(cases * share))). + final Map shares; + + /// Names the situations of one input by calling `bump('name')`. + final void Function(T arg, void Function(String) bump) observe; +} + +class _Registered { + _Registered(this.name, this.cases, this.shares, this.measure); + final String name; + final int cases; + final Map shares; + final Map Function() measure; +} + +class LawSet { + final List<_Registered> _registry = []; + + /// Number of registered laws. + int get length => _registry.length; + + /// Registers [name] as a property (forced [examples] first, then [cases] + /// generated ones) and records its reach. + void law( + String name, + Gen gen, + void Function(T) body, { + required Reach reach, + List examples = const [], + int cases = 30, + String genVersion = 'g1', + Duration? budget, + Object? skip, + }) { + _registry.add(_Registered(name, cases, reach.shares, () { + final got = {for (final k in reach.shares.keys) k: 0}; + final r = runProperty( + name: name, + gen: gen, + config: PropertyConfig( + cases: cases, + // A generous guard, but PROPERTY_BUDGET_MS (the serial acceptance + // run) is the hard limit here too: drawing a law's cases is cheap. + budget: Duration( + milliseconds: int.tryParse( + Platform.environment['PROPERTY_BUDGET_MS'] ?? '') ?? + 60000)), + body: (v) => reach.observe(v, (k) => got[k] = (got[k] ?? 0) + 1), + genVersion: genVersion, + ); + if (!r.passed) fail(r.failure!.report); + return got; + })); + forAll(name, gen, body, + examples: examples, + cases: cases, + genVersion: genVersion, + // The spec's cap is 2 s per property, measured serially on the dev + // desktop. The in-test guard that fails a runaway has 2.5x headroom on + // top, so a suite running 12 test files at once does not trip it. + // PROPERTY_BUDGET_MS replaces it as the HARD limit (see + // PropertyConfig.fromEnvironment): the acceptance run is + // PROPERTY_BUDGET_MS=2000 dart test -j 1 test/properties/ + // and any property over 2 s there fails the run. + budget: budget ?? const Duration(seconds: 5), + skip: skip); + } + + /// The test that every law's default cases reach what the law claims. + void registerReachTest() { + group('generator reach', () { + test('every law sees every situation in its default cases', () { + expect(_registry, isNotEmpty); + final weak = []; + for (final law in _registry) { + final got = law.measure(); + if (Platform.environment.containsKey('PROPERTY_REACH_REPORT')) { + // ignore: avoid_print + print('REACH ${law.name.split(' ').first}/${law.cases} cases: ' + '${got.entries.map((e) => "${e.key}=${e.value}").join("; ")}'); + } + for (final e in law.shares.entries) { + final need = math.max(1, (law.cases * e.value).floor()); + if (got[e.key]! < need) { + weak.add('"${law.name}": ${e.key} in ${got[e.key]} of ' + '${law.cases} cases, need $need'); + } + } + } + expect(weak, isEmpty, reason: weak.join('\n')); + }); + }); + } +} diff --git a/test/support/property.dart b/test/support/property.dart new file mode 100644 index 0000000..89456a3 --- /dev/null +++ b/test/support/property.dart @@ -0,0 +1,753 @@ +// PROPERTY HARNESS — seeded, shrinking, replayable property tests. +// +// Port of the edge harness (edge test/support/property.dart, design 05 pilot): +// same generators, seeds, replay and budget rules, on package:test only (this +// package is pure Dart). Keep the two in step; the RNG is pinned to reference +// outputs in property_test.dart so a replay command means the same thing in +// both repos. +// +// In-house on purpose (design 05, section 3): no dependency, fully +// deterministic, and every failure prints what is needed to replay it. +// +// forAll('no two icons overlap', G.listOf(item, maxLen: 60), (items) { +// ... expect(...) ... +// }, examples: [...forced scenarios...]); +// +// How a run is built +// • SEED. Each property has its own seed, derived from its NAME (so adding or +// reordering properties never changes another property's inputs). +// PROPERTY_SEED overrides it for every property in the run. +// • CASES. Case `i` draws from its own generator stream `caseSeed(seed, i)` and +// a size that depends on `i` only, so PROPERTY_CASE=i replays exactly that +// input whatever the iteration count. Default 200 cases; PROPERTY_ITERATIONS +// overrides it (and scales the wall budget with it). +// • FORCED SCENARIOS. `examples` always run first, in order, before any +// generated case. They are never shrunk (they are explicit). Replay one with +// PROPERTY_CASE=forced:. +// • FAILURE. The first failing input is shrunk (bounded by `shrinkLimit` body +// runs) and printed with seed, case, generator version and a one-line replay +// command. Shrinkers are domain-preserving: a candidate is always something +// the generator itself could have produced. +// • BUDGET. A property that takes longer than its wall budget (default 2 s) +// fails. Shrinking counts against the same budget: when it runs out the +// best failing input so far is reported, marked "over budget (shrinking)". The budget is the ONLY +// use of a real clock; it is injectable for the harness's own tests. +// +// GENERATOR VERSION. A seed only means something for the generator that drew +// it. When a property's generator changes shape, bump its `genVersion`; the +// report prints it so an old replay command is recognised as stale. +// +// Environment: PROPERTY_SEED, PROPERTY_CASE, PROPERTY_ITERATIONS, +// PROPERTY_BUDGET_MS (fixes the budget, no scaling). + +import 'dart:io' show Platform; +import 'dart:math' as math; + +import 'package:test/test.dart'; + +// ── randomness ────────────────────────────────────────────────────────────── + +/// True on dart2js, where `1` and `1.0` are the same value. +const bool _onJs = identical(1, 1.0); + +/// splitmix64 (Steele, Lea, Flood; the public-domain reference generator): +/// tiny, fast, and identical on every Dart VM version (unlike `dart:math` +/// Random, whose stream is not promised to be stable). The first outputs for +/// seeds 0 and 1234567 are pinned in property_test.dart against values computed +/// independently. +/// +/// ASSUMPTION: a Dart VM `int` is a 64-bit two's-complement integer whose `+` +/// and `*` wrap, and `>>>` is a logical shift. That is what the arithmetic +/// below relies on. This harness is VM-only and must not be compiled for the +/// web (ints are doubles there); the constructor refuses to run on JS. +class Rng { + Rng(int seed) : _s = seed { + if (_onJs) { + throw UnsupportedError('test/support/property.dart needs 64-bit VM ints'); + } + } + int _s; + + static const int _gamma = 0x9E3779B97F4A7C15; + + static int mix(int x) { + var z = x; + z = (z ^ (z >>> 30)) * 0xBF58476D1CE4E5B9; + z = (z ^ (z >>> 27)) * 0x94D049BB133111EB; + return z ^ (z >>> 31); + } + + int _next() { + _s += _gamma; + return mix(_s); + } + + /// The next raw 64-bit output (a signed Dart int; same bits as the + /// reference's unsigned value). + int next64() => _next(); + + /// A uniform int in `[0, bound)`. `bound` must be in `1 .. 2^53`. + int nextInt(int bound) { + assert(bound > 0 && bound <= (1 << 53), 'bound out of range: $bound'); + return (_next() >>> 1) % bound; + } + + /// A uniform int in `[min, max]`, both inclusive. + int intIn(int min, int max) { + assert(min <= max); + return min + nextInt(max - min + 1); + } + + /// A uniform double in `[0, 1)`. + double nextDouble() => (_next() >>> 11) / 9007199254740992.0; + + /// True with probability [p]. + bool nextBool([double p = .5]) => nextDouble() < p; +} + +/// The default seed of the property called [name] (32-bit FNV-1a). +int seedFor(String name) { + var h = 0x811C9DC5; + for (final u in name.codeUnits) { + h = ((h ^ u) * 0x01000193) & 0xFFFFFFFF; + } + return h; +} + +/// The generator stream of case [index] of a property seeded [seed]. Depends on +/// nothing else, so one case can be replayed alone. +int caseSeed(int seed, int index) => + Rng.mix(Rng.mix(seed) + index * 0x9E3779B97F4A7C15) & 0x7FFFFFFFFFFF; + +// ── generators ────────────────────────────────────────────────────────────── + +/// Draws values and shrinks them. [shrink] must only return values this +/// generator could itself have produced (domain-preserving), none equal to the +/// input, simplest first, and finitely many per call. +abstract class Gen { + const Gen(); + + /// A value; [size] grows 10..100 across the cases of a run. + T generate(Rng r, int size); + + /// Simpler candidates than [v], simplest first. May be lazy. + Iterable shrink(T v); + + /// A readable, copy-pasteable rendering for the failure report. + String show(T v) => '$v'; +} + +class _IntGen extends Gen { + _IntGen(this.min, this.max, int? origin) + : origin = (origin ?? 0).clamp(min, max).toInt() { + assert(min <= max && max - min < (1 << 53)); + } + final int min, max, origin; + + @override + int generate(Rng r, int size) { + final p = r.nextDouble(); + if (p < .06) return min; + if (p < .12) return max; + if (p < .16) return origin; + return r.intIn(min, max); + } + + @override + Iterable shrink(int v) sync* { + if (v == origin) return; + yield origin; + final d = v - origin; + var step = d ~/ 2; + final seen = {origin}; + while (step != 0) { + final c = v - step; + if (seen.add(c)) yield c; + step ~/= 2; + } + final one = v - d.sign; + if (seen.add(one) && one != v) yield one; + } +} + +class _DoubleGen extends Gen { + _DoubleGen(this.min, this.max, this.boundaries, this.specials, double? origin, + this.integerBias) + : origin = (origin ?? 0).clamp(min, max).toDouble() { + assert(min.isFinite && max.isFinite && min <= max); + } + final double min, max, origin, integerBias; + final List boundaries, specials; + + List get _pool => [ + min, + max, + if (origin != min && origin != max) origin, + for (final b in boundaries) + if (b >= min && b <= max) b, + ]; + + @override + double generate(Rng r, int size) { + final p = r.nextDouble(); + if (specials.isNotEmpty && p < .08) { + return specials[r.nextInt(specials.length)]; + } + if (p < .08 + .17) { + final pool = _pool; + return pool[r.nextInt(pool.length)]; + } + var v = min + r.nextDouble() * (max - min); + if (r.nextBool(integerBias)) v = v.roundToDouble(); + return v.clamp(min, max).toDouble(); + } + + @override + Iterable shrink(double v) sync* { + if (!v.isFinite) { + yield origin; // a special shrinks to the simplest finite value + return; + } + if (v == origin) return; + final d = v - origin; + if (d.abs() < 1e-6) { + yield origin; + return; + } + final seen = {v}; + Iterable cand() sync* { + yield origin; + for (final b in _pool) { + if ((b - origin).abs() < d.abs()) yield b; + } + yield v.truncateToDouble(); + yield v - d / 2; + yield v - d / 4; + yield v - d / 8; + } + + for (final c in cand()) { + if (c >= min && c <= max && seen.add(c)) yield c; + } + } +} + +class _BoolGen extends Gen { + @override + bool generate(Rng r, int size) => r.nextBool(); + @override + Iterable shrink(bool v) => v ? const [false] : const []; +} + +class _ElementsGen extends Gen { + _ElementsGen(this.values) { + assert(values.isNotEmpty); + } + final List values; + @override + T generate(Rng r, int size) => values[r.nextInt(values.length)]; + @override + Iterable shrink(T v) { + final i = values.indexOf(v); + return i <= 0 ? const [] : values.take(i); + } +} + +class _ListGen extends Gen> { + _ListGen(this.elem, this.minLen, this.maxLen) { + assert(minLen >= 0 && minLen <= maxLen); + } + final Gen elem; + final int minLen, maxLen; + + @override + List generate(Rng r, int size) { + final cap = minLen + ((maxLen - minLen) * (size + 1) / 101).ceil(); + final p = r.nextDouble(); + final n = p < .08 + ? minLen + : p < .16 + ? maxLen + : r.intIn(minLen, math.max(minLen, cap)); + return [for (var i = 0; i < n; i++) elem.generate(r, size)]; + } + + @override + Iterable> shrink(List v) sync* { + // Shorter first: drop chunks, big to small. + if (v.length > minLen) { + for (var chunk = math.max(1, v.length ~/ 2); chunk >= 1; chunk ~/= 2) { + if (v.length - chunk < minLen) continue; + for (var at = 0; at + chunk <= v.length; at += chunk) { + yield [...v.sublist(0, at), ...v.sublist(at + chunk)]; + } + } + } + // Then simpler elements, in place. + for (var i = 0; i < v.length; i++) { + for (final e in elem.shrink(v[i])) { + yield [...v.sublist(0, i), e, ...v.sublist(i + 1)]; + } + } + } + + @override + String show(List v) => '[${v.map(elem.show).join(', ')}]'; +} + +class _NullableGen extends Gen { + _NullableGen(this.inner, this.nullProbability); + final Gen inner; + final double nullProbability; + @override + T? generate(Rng r, int size) => + r.nextBool(nullProbability) ? null : inner.generate(r, size); + @override + Iterable shrink(T? v) sync* { + if (v == null) return; + yield null; + yield* inner.shrink(v); + } + + @override + String show(T? v) => v == null ? 'null' : inner.show(v); +} + +/// Candidate lists of the components of a tuple, shrunk SEPARATELY and then +/// TOGETHER (the k-th simpler candidate of every component at once), so a law +/// that only fails on correlated components (a == b) still shrinks. +Iterable> _shrinkTuple( + List> gens, List v) sync* { + final cands = [ + for (var i = 0; i < gens.length; i++) gens[i].shrink(v[i]).take(12).toList() + ]; + // Joint steps first: they make the biggest strides and are the only ones + // that work when the components are correlated. + final joint = cands.map((c) => c.length).reduce(math.min); + for (var k = 0; k < joint; k++) { + yield [for (final c in cands) c[k]]; + } + for (var i = 0; i < gens.length; i++) { + for (final c in gens[i].shrink(v[i])) { + yield [...v.sublist(0, i), c, ...v.sublist(i + 1)]; + } + } +} + +class _TupleGen extends Gen { + _TupleGen(this.gens, this.build, this.parts); + final List> gens; + final R Function(List) build; + final List Function(R) parts; + + @override + R generate(Rng r, int size) => + build([for (final g in gens) g.generate(r, size)]); + @override + Iterable shrink(R v) => _shrinkTuple(gens, parts(v)).map(build); + @override + String show(R v) { + final p = parts(v); + return '(${[for (var i = 0; i < gens.length; i++) gens[i].show(p[i])].join(', ')})'; + } +} + +/// The generator constructors. +abstract final class G { + /// An int in `[min, max]`; shrinks toward [origin] (default: zero, or the + /// bound nearest zero). + static Gen intIn(int min, int max, {int? origin}) => + _IntGen(min, max, origin); + + /// A double in `[min, max]` (finite). [boundaries] are drawn deliberately; + /// [specials] (NaN, infinities) are drawn occasionally and shrink to + /// [origin]. [integerBias] is the share of draws snapped to whole numbers + /// (ties and exact thresholds are where the interesting laws bend). + static Gen doubleIn(double min, double max, + {List boundaries = const [], + List specials = const [], + double? origin, + double integerBias = .25}) => + _DoubleGen(min, max, boundaries, specials, origin, integerBias); + + static Gen boolean() => _BoolGen(); + + /// One of [values]; shrinks toward the earlier ones. + static Gen elements(List values) => _ElementsGen(values); + + /// A list of `minLen .. maxLen` elements (the cap is a hard bound). + static Gen> listOf(Gen elem, + {int minLen = 0, required int maxLen}) => + _ListGen(elem, minLen, maxLen); + + static Gen nullable(Gen inner, {double nullProbability = .25}) => + _NullableGen(inner, nullProbability); + + static Gen<(A, B)> pair(Gen a, Gen b) => _TupleGen( + [a, b], (p) => (p[0] as A, p[1] as B), (r) => [r.$1, r.$2]); + + static Gen<(A, B, C)> triple(Gen a, Gen b, Gen c) => + _TupleGen([a, b, c], (p) => (p[0] as A, p[1] as B, p[2] as C), + (r) => [r.$1, r.$2, r.$3]); + + static Gen<(A, B, C, D)> quad( + Gen a, Gen b, Gen c, Gen d) => + _TupleGen( + [a, b, c, d], + (p) => (p[0] as A, p[1] as B, p[2] as C, p[3] as D), + (r) => [r.$1, r.$2, r.$3, r.$4]); +} + +// ── configuration and result ──────────────────────────────────────────────── + +class PropertyConfig { + const PropertyConfig({ + this.seed, + this.cases = 200, + this.caseOnly, + this.budget = const Duration(seconds: 2), + this.shrinkLimit = 300, + this.testFile, + }); + + /// Reads PROPERTY_SEED / PROPERTY_CASE / PROPERTY_ITERATIONS / + /// PROPERTY_BUDGET_MS from [env]. A malformed value throws: a typo must not + /// silently run the default. + factory PropertyConfig.fromEnvironment( + Map env, { + String? testFile, + int? seed, + int defaultCases = 200, + Duration defaultBudget = const Duration(seconds: 2), + int shrinkLimit = 300, + }) { + int? number(String key, {int min = 0}) { + final raw = env[key]; + if (raw == null || raw.isEmpty) return null; + final n = int.tryParse(raw); + if (n == null || n < min) { + throw FormatException('$key must be an integer >= $min', raw); + } + return n; + } + + final envSeed = number('PROPERTY_SEED', min: -(1 << 62)); + final iterations = number('PROPERTY_ITERATIONS', min: 1); + final budgetMs = number('PROPERTY_BUDGET_MS', min: 1); + final caseRaw = env['PROPERTY_CASE']; + if (caseRaw != null && + caseRaw.isNotEmpty && + !RegExp(r'^(forced:)?\d+$').hasMatch(caseRaw)) { + throw FormatException('PROPERTY_CASE must be N or forced:N', caseRaw); + } + final cases = iterations ?? defaultCases; + final scale = math.max(1.0, cases / defaultCases); + return PropertyConfig( + seed: envSeed ?? seed, + cases: cases, + caseOnly: caseRaw == null || caseRaw.isEmpty ? null : caseRaw, + budget: budgetMs != null + ? Duration(milliseconds: budgetMs) + : iterations != null + ? Duration( + milliseconds: (defaultBudget.inMilliseconds * scale).round()) + : defaultBudget, + shrinkLimit: shrinkLimit, + testFile: testFile, + ); + } + + /// null: derived from the property name. + final int? seed; + final int cases; + + /// Replay only this case: `N` or `forced:N`. null: the whole run. + final String? caseOnly; + final Duration budget; + + /// Most body runs spent shrinking one failure. + final int shrinkLimit; + + /// Repo-relative path printed in the replay command. + final String? testFile; +} + +enum FailureKind { counterexample, forcedExample, budget, generator } + +class PropertyFailure { + const PropertyFailure({ + required this.kind, + required this.seed, + required this.caseId, + required this.input, + required this.error, + required this.replay, + required this.shrinkSteps, + required this.report, + this.shrinkOverBudget = false, + }); + final FailureKind kind; + final int seed; + + /// `17` for a generated case, `forced:1` for an explicit example. + final String caseId; + + /// The (shrunk) failing input, as the generator shows it. + final String input; + final String error; + final String replay; + + /// Body runs spent shrinking. + final int shrinkSteps; + final String report; + + /// Shrinking stopped because the property's wall budget ran out; [input] is + /// the best (smallest) failing input found so far, not necessarily minimal. + final bool shrinkOverBudget; +} + +class PropertyResult { + const PropertyResult( + {required this.casesRun, required this.forcedRun, this.failure}); + final int casesRun; + final int forcedRun; + final PropertyFailure? failure; + bool get passed => failure == null; +} + +// ── running ───────────────────────────────────────────────────────────────── + +String _errorText(Object e, StackTrace st) { + final msg = e.toString(); + final text = msg.length > 900 ? '${msg.substring(0, 900)}...' : msg; + if (e is TestFailure) return text; + // A non-assertion error (RangeError, StateError, ...) needs its origin. + final frames = st.toString().split('\n').where((l) => l.isNotEmpty).take(6); + return '$text\n ${frames.join('\n ')}'; +} + +/// Runs the property and returns the outcome; never throws for a failing body. +/// [forAll] is the test-registering wrapper. +PropertyResult runProperty({ + required String name, + required Gen gen, + required void Function(T) body, + List examples = const [], + PropertyConfig config = const PropertyConfig(), + String genVersion = 'g1', + Duration Function()? elapsed, +}) { + final seed = config.seed ?? seedFor(name); + late final Duration Function() clock; + if (elapsed != null) { + clock = elapsed; + } else { + final sw = Stopwatch()..start(); + clock = () => sw.elapsed; + } + + String? attempt(T v) { + try { + body(v); + return null; + } catch (e, st) { + return _errorText(e, st); + } + } + + String replayFor(String? caseId) => + 'PROPERTY_SEED=$seed ${caseId == null ? '' : 'PROPERTY_CASE=$caseId '}' + 'dart test ' + "${config.testFile ?? ''} " + "--plain-name '${name.replaceAll("'", r"'\''")}'"; + + PropertyFailure fail(FailureKind kind, String caseId, String input, + String error, + {int shrinkSteps = 0, + int accepted = 0, + String? original, + bool overBudget = false}) { + final replay = replayFor(kind == FailureKind.budget ? null : caseId); + final b = StringBuffer() + ..writeln('Property "$name" FAILED (${kind.name})') + ..writeln(' seed: $seed case: $caseId generator: $genVersion' + ' shrink limit: ${config.shrinkLimit}'); + if (kind == FailureKind.budget) { + b.writeln(' $error'); + } else { + b.writeln(shrinkSteps > 0 + ? ' input (shrunk: $accepted accepted of $shrinkSteps tried): $input' + : ' input: $input'); + if (overBudget) { + b.writeln(' over budget (shrinking): stopped after ${config.budget.inMilliseconds} ' + 'ms; this is the best failing input so far, not necessarily minimal'); + } + if (original != null && original != input) { + b.writeln(' original input: $original'); + } + b.writeln(' error: $error'); + } + b.writeln(' replay: $replay'); + return PropertyFailure( + kind: kind, + seed: seed, + caseId: caseId, + input: input, + error: error, + replay: replay, + shrinkSteps: shrinkSteps, + report: b.toString(), + shrinkOverBudget: overBudget); + } + + PropertyFailure shrunk(String caseId, T original, String error) { + var cur = original; + var err = error; + var tried = 0, accepted = 0; + var overBudget = false; + try { + outer: + while (true) { + var progressed = false; + for (final c in gen.shrink(cur)) { + if (tried >= config.shrinkLimit) break outer; + // The wall budget covers shrinking too (a replay is exempt). + if (config.caseOnly == null && clock() > config.budget) { + overBudget = true; + break outer; + } + tried++; + final e = attempt(c); + if (e != null) { + cur = c; + err = e; + accepted++; + progressed = true; + break; + } + } + if (!progressed) break; + } + } catch (_) { + // A shrinker that throws ends shrinking; the best input so far stands. + } + return fail(FailureKind.counterexample, caseId, gen.show(cur), err, + shrinkSteps: tried, + accepted: accepted, + original: gen.show(original), + overBudget: overBudget); + } + + PropertyResult done(int cases, int forced, [PropertyFailure? f]) => + PropertyResult(casesRun: cases, forcedRun: forced, failure: f); + + // One generated case; returns a failure or null. + PropertyFailure? runCase(int i) { + final T input; + try { + input = gen.generate(Rng(caseSeed(seed, i)), math.min(100, 10 + i)); + } catch (e, st) { + return fail(FailureKind.generator, '$i', '', + _errorText(e, st)); + } + final e = attempt(input); + return e == null ? null : shrunk('$i', input, e); + } + + PropertyFailure? runForced(int i) { + final e = attempt(examples[i]); + return e == null + ? null + : fail(FailureKind.forcedExample, 'forced:$i', gen.show(examples[i]), e); + } + + final only = config.caseOnly; + if (only != null) { + if (only.startsWith('forced:')) { + final i = int.parse(only.substring(7)); + if (i >= examples.length) { + throw StateError('PROPERTY_CASE=$only: only ${examples.length} forced ' + 'examples in "$name"'); + } + return done(0, 1, runForced(i)); + } + return done(1, 0, runCase(int.parse(only))); + } + + PropertyFailure? overBudget(int cases, int forced) => clock() > config.budget + ? fail( + FailureKind.budget, + 'n/a', + '', + 'wall budget ${config.budget.inMilliseconds} ms exceeded after ' + '$cases cases + $forced forced examples ' + '(${clock().inMilliseconds} ms)') + : null; + + var forcedRun = 0, casesRun = 0; + for (var i = 0; i < examples.length; i++) { + final f = runForced(i); + forcedRun++; + if (f != null) return done(casesRun, forcedRun, f); + final b = overBudget(casesRun, forcedRun); + if (b != null) return done(casesRun, forcedRun, b); + } + for (var i = 0; i < config.cases; i++) { + final f = runCase(i); + casesRun++; + if (f != null) return done(casesRun, forcedRun, f); + final b = overBudget(casesRun, forcedRun); + if (b != null) return done(casesRun, forcedRun, b); + } + return done(casesRun, forcedRun); +} + +String? _callerTestFile() { + final m = RegExp(r'test/[\w/.\-]+\.dart'); + for (final line in StackTrace.current.toString().split('\n')) { + final hit = m.firstMatch(line)?.group(0); + if (hit != null && !RegExp(r'support/(property|law_registry)\.dart$').hasMatch(hit)) { + return hit; + } + } + return null; +} + +/// Registers one `test` that checks [body] on [examples] (forced, first) and on +/// generated cases. Replay with the command in the failure report. +/// +/// [cases], [budget] and [seed] are this property's defaults; the PROPERTY_* +/// environment wins. [skip] is for a law that currently fails because of a +/// suspected bug: say which in the reason. +void forAll( + String name, + Gen gen, + void Function(T) body, { + List examples = const [], + int? cases, + int? seed, + Duration? budget, + int? shrinkLimit, + String genVersion = 'g1', + Object? skip, +}) { + final file = _callerTestFile(); + test(name, () { + final config = PropertyConfig.fromEnvironment( + Platform.environment, + testFile: file, + seed: seed, + defaultCases: cases ?? 200, + defaultBudget: budget ?? const Duration(seconds: 2), + shrinkLimit: shrinkLimit ?? 300, + ); + final r = runProperty( + name: name, + gen: gen, + body: body, + examples: examples, + config: config, + genVersion: genVersion); + if (!r.passed) fail(r.failure!.report); + }, skip: skip); +} diff --git a/test/support/property_test.dart b/test/support/property_test.dart new file mode 100644 index 0000000..69938a3 --- /dev/null +++ b/test/support/property_test.dart @@ -0,0 +1,778 @@ +// Self-tests of the property harness (test/support/property.dart). +// +// The harness decides whether the property suites can be trusted, so its own +// guarantees are pinned here: deterministic and independent seeds, replay of a +// single case, forced examples first, domain-preserving bounded shrinking, the +// failure report (seed, case, input, generator version, replay command) and +// the per-property wall budget. + +import 'dart:math' as math; + +import 'package:test/test.dart'; + +import 'property.dart'; + +/// Runs [gen] through the harness with a body that records every input and +/// never fails. Returns the recorded inputs (forced examples first). +List _inputs(Gen gen, + {String name = 'rec', + int cases = 30, + List examples = const [], + int? seed, + String? caseOnly}) { + final seen = []; + final r = runProperty( + name: name, + gen: gen, + body: seen.add, + examples: examples, + config: PropertyConfig(seed: seed, cases: cases, caseOnly: caseOnly), + ); + expect(r.passed, isTrue, reason: r.failure?.report); + return seen; +} + +PropertyFailure _fails(Gen gen, bool Function(T) holds, + {List examples = const [], + int cases = 200, + int shrinkLimit = 300, + String name = 'p', + int seed = 7, + String? caseOnly, + String? testFile}) { + final r = runProperty( + name: name, + gen: gen, + body: (v) { + if (!holds(v)) throw StateError('does not hold: $v'); + }, + examples: examples, + config: PropertyConfig( + seed: seed, + cases: cases, + shrinkLimit: shrinkLimit, + caseOnly: caseOnly, + testFile: testFile), + ); + expect(r.passed, isFalse, reason: 'the property should have failed'); + return r.failure!; +} + +void main() { + group('rng', () { + test('the same seed gives the same sequence, another seed another', () { + List draw(int seed) { + final r = Rng(seed); + return [for (var i = 0; i < 20; i++) r.nextInt(1 << 20)]; + } + + expect(draw(5), draw(5)); + expect(draw(5), isNot(draw(6))); + }); + + test('intIn is inclusive, nextDouble is in [0, 1), nextBool honours p', () { + final r = Rng(42); + final ints = {for (var i = 0; i < 400; i++) r.intIn(-2, 2)}; + expect(ints, {-2, -1, 0, 1, 2}); + for (var i = 0; i < 400; i++) { + final d = r.nextDouble(); + expect(d >= 0 && d < 1, isTrue); + } + expect(r.intIn(3, 3), 3); + expect([for (var i = 0; i < 50; i++) r.nextBool(0)], everyElement(false)); + expect([for (var i = 0; i < 50; i++) r.nextBool(1)], everyElement(true)); + }); + + test('next64 is SplitMix64: the first outputs match the reference values', + () { + // Computed independently with python3, masking every step to 64 bits: + // s = (s + 0x9E3779B97F4A7C15) & M + // z = ((s ^ (s >> 30)) * 0xBF58476D1CE4E5B9) & M + // z = ((z ^ (z >> 27)) * 0x94D049BB133111EB) & M; out = z ^ (z >> 31) + // Seed 0 starts 0xE220A8397B1DCDAF, the published SplitMix64 test value. + // Dart's signed int holds the same 64 bits, so compare as hex. + String hex(int v) => + (v >>> 32).toRadixString(16).padLeft(8, '0') + + (v & 0xFFFFFFFF).toRadixString(16).padLeft(8, '0'); + final z = Rng(0); + expect([for (var i = 0; i < 5; i++) hex(z.next64())], [ + 'e220a8397b1dcdaf', + '6e789e6aa1b965f4', + '06c45d188009454f', + 'f88bb8a8724c81ec', + '1b39896a51a8749b', + ]); + final k = Rng(1234567); + expect([for (var i = 0; i < 5; i++) hex(k.next64())], [ + '599ed017fb08fc85', + '2c73f08458540fa5', + '883ebce5a3f27c77', + '3fbef740e9177b3f', + 'e3b8346708cb5ecd', + ]); + }); + + test('the stream is pinned: a replay command stays valid across SDKs', () { + final r = Rng(1); + expect([for (var i = 0; i < 5; i++) r.nextInt(1000)], + [232, 259, 295, 117, 380]); + expect(Rng(1).nextDouble(), 0.5665615751722809); + expect(seedFor('a law'), 1308619236); + expect(caseSeed(9, 3), 136612318154068); + }); + + test('draws are spread: a long run touches every bucket of 16', () { + final r = Rng(1); + final buckets = {for (var i = 0; i < 2000; i++) r.nextInt(16)}; + expect(buckets.length, 16); + }); + }); + + group('rng ranges', () { + forAll<(int, int)>( + 'rng draws stay in range: nextInt(bound) in [0, bound), nextDouble in [0, 1)', + G.pair( + G.intIn(-(1 << 40), 1 << 40), + G.intIn(0, 9)), + (arg) { + final (seed, pick) = arg; + const bounds = [1, 2, 3, 7, 1000, 1 << 20, (1 << 53) - 1, 1 << 53]; + final r = Rng(seed); + for (var i = 0; i < 40; i++) { + final b = bounds[(pick + i) % bounds.length]; + final n = r.nextInt(b); + expect(n >= 0 && n < b, isTrue, reason: 'nextInt($b) = $n'); + final d = r.nextDouble(); + expect(d >= 0 && d < 1, isTrue, reason: 'nextDouble() = $d'); + final k = r.intIn(0, b - 1); + expect(k >= 0 && k < b, isTrue, reason: 'intIn(0, ${b - 1}) = $k'); + } + expect(Rng(seed).nextInt(1), 0); + }, examples: const [(0, 0), (-1, 7), (1 << 40, 3)]); + }); + + group('seeds', () { + test('seedFor is stable per name and differs between names', () { + expect(seedFor('a law'), seedFor('a law')); + expect(seedFor('a law'), isNot(seedFor('another law'))); + }); + + test('case seeds are independent of each other and of the run length', () { + expect(caseSeed(9, 3), caseSeed(9, 3)); + expect(caseSeed(9, 3), isNot(caseSeed(9, 4))); + expect(caseSeed(9, 3), isNot(caseSeed(10, 3))); + }); + + test('two properties with different names draw different inputs', () { + final gen = G.listOf(G.intIn(0, 1 << 20), maxLen: 8); + expect(_inputs(gen, name: 'one', cases: 5), + isNot(_inputs(gen, name: 'two', cases: 5))); + }); + + test('the same property is reproduced exactly', () { + final gen = G.pair(G.intIn(-50, 50), G.doubleIn(0, 1)); + expect(_inputs(gen, cases: 40), _inputs(gen, cases: 40)); + }); + }); + + group('config from the environment', () { + test('defaults: 200 cases, 2 s, no seed, no single case', () { + final c = PropertyConfig.fromEnvironment(const {}); + expect(c.cases, 200); + expect(c.budget, const Duration(seconds: 2)); + expect(c.seed, isNull); + expect(c.caseOnly, isNull); + }); + + test('PROPERTY_SEED, PROPERTY_CASE and PROPERTY_ITERATIONS override', () { + final c = PropertyConfig.fromEnvironment(const { + 'PROPERTY_SEED': '12345', + 'PROPERTY_CASE': '17', + 'PROPERTY_ITERATIONS': '1000', + }, testFile: 'test/x_test.dart'); + expect(c.seed, 12345); + expect(c.caseOnly, '17'); + expect(c.cases, 1000); + expect(c.testFile, 'test/x_test.dart'); + }); + + test('raising the iterations scales the wall budget with them', () { + final c = + PropertyConfig.fromEnvironment(const {'PROPERTY_ITERATIONS': '1000'}); + expect(c.budget, const Duration(seconds: 10)); + final fixed = PropertyConfig.fromEnvironment(const { + 'PROPERTY_ITERATIONS': '1000', + 'PROPERTY_BUDGET_MS': '3000', + }); + expect(fixed.budget, const Duration(milliseconds: 3000)); + }); + + test('a malformed number is loud, not ignored', () { + expect(() => PropertyConfig.fromEnvironment(const {'PROPERTY_SEED': 'x'}), + throwsFormatException); + expect( + () => PropertyConfig.fromEnvironment( + const {'PROPERTY_ITERATIONS': '0'}), + throwsFormatException); + }); + }); + + group('running', () { + test('runs exactly the configured number of cases', () { + final r = runProperty( + name: 'n', + gen: G.intIn(0, 9), + body: (_) {}, + config: const PropertyConfig(cases: 37), + ); + expect(r.passed, isTrue); + expect(r.casesRun, 37); + expect(r.forcedRun, 0); + }); + + test('forced examples run first, in order, then the generated cases', () { + final seen = _inputs(G.intIn(100, 200), + examples: [5, 6, 7], cases: 10); + expect(seen.take(3), [5, 6, 7]); + expect(seen.length, 13); + final r = runProperty( + name: 'f', + gen: G.intIn(0, 9), + body: (_) {}, + examples: [1, 2], + config: const PropertyConfig(cases: 4)); + expect((r.forcedRun, r.casesRun), (2, 4)); + }); + + test('generated cases stay inside the generator, even the first ones', () { + final seen = _inputs(G.intIn(-3, 11), cases: 200); + expect(seen.every((v) => v >= -3 && v <= 11), isTrue); + }); + + test('a single case replays exactly the input it had in the full run', () { + final gen = G.pair(G.intIn(-1000, 1000), G.listOf(G.boolean(), maxLen: 9)); + final all = _inputs(gen, cases: 60, seed: 99); + for (final k in [0, 1, 17, 59]) { + expect( + _inputs(gen, cases: 60, seed: 99, caseOnly: '$k') + .map(gen.show) + .toList(), + [gen.show(all[k])], + reason: 'case $k'); + } + }); + + test('a case replays the same whatever the iteration count', () { + final gen = G.listOf(G.intIn(0, 1 << 20), maxLen: 30); + final short = _inputs(gen, cases: 25, seed: 3); + final long = _inputs(gen, cases: 80, seed: 3); + expect(long.take(25), short); + }); + + test('a forced example replays by its forced id, generated cases skip it', + () { + final seen = _inputs(G.intIn(100, 200), + examples: [5, 6, 7], cases: 10, caseOnly: 'forced:1'); + expect(seen, [6]); + final one = _inputs(G.intIn(100, 200), + examples: [5, 6, 7], cases: 10, seed: 1, caseOnly: '2'); + expect(one.length, 1); + expect(one.single, inInclusiveRange(100, 200)); + }); + + test('a replay of a case beyond the run length still runs that case', () { + final seen = _inputs(G.intIn(0, 9), cases: 5, caseOnly: '40'); + expect(seen.length, 1); + }); + + test('a body that completes normally is passed, whatever it returns', () { + final r = runProperty( + name: 'ret', gen: G.intIn(0, 3), body: (_) => 5 as dynamic); + expect(r.passed, isTrue); + }); + }); + + group('failure report', () { + test('names the seed, case, shrunk input, generator version, replay', () { + final f = _fails(G.intIn(0, 1000), (v) => v < 50, + name: 'ints stay small', seed: 4242, testFile: 'test/x_test.dart'); + expect(f.kind, FailureKind.counterexample); + expect(f.seed, 4242); + expect(f.input, '50'); + expect(f.caseId, matches(RegExp(r'^\d+$'))); + expect(f.error, contains('does not hold')); + expect( + f.replay, + 'PROPERTY_SEED=4242 PROPERTY_CASE=${f.caseId} ' + "dart test test/x_test.dart --plain-name 'ints stay small'"); + for (final s in [ + 'ints stay small', + 'seed: 4242', + 'case: ${f.caseId}', + 'generator: g1', + 'shrink limit: 300', + 'input', + '50', + f.replay, + ]) { + expect(f.report, contains(s)); + } + }); + + test('the generator version is whatever the property declares', () { + final r = runProperty( + name: 'v', + gen: G.intIn(0, 9), + body: (_) => throw StateError('x'), + genVersion: 'layout-gen/3', + config: const PropertyConfig(seed: 1)); + expect(r.failure!.report, contains('generator: layout-gen/3')); + }); + + test('a quote in the name is escaped in the replay command', () { + final f = _fails(G.intIn(0, 9), (_) => false, name: "it's a law"); + expect(f.replay, contains(r"--plain-name 'it'\''s a law'")); + }); + + test('an unknown test file is a visible placeholder, not omitted', () { + final f = _fails(G.intIn(0, 9), (_) => false); + expect(f.replay, contains('dart test ')); + }); + + test('the replay command reproduces the failure', () { + final gen = G.listOf(G.intIn(0, 99), maxLen: 25); + final f = _fails(gen, (l) => l.length < 20, seed: 11); + final again = + _fails(gen, (l) => l.length < 20, seed: 11, caseOnly: f.caseId); + expect(again.input, f.input); + expect(again.caseId, f.caseId); + }); + + test('is deterministic: the same run produces the same report', () { + final gen = G.pair(G.intIn(0, 500), G.doubleIn(0, 1)); + String report() => _fails(gen, (p) => p.$1 < 400).report; + expect(report(), report()); + }); + + test('a thrown TestFailure keeps its message', () { + final r = runProperty( + name: 'tf', + gen: G.intIn(0, 3), + body: (v) => expect(v, 99, reason: 'my reason'), + config: const PropertyConfig(seed: 1)); + expect(r.failure!.error, contains('my reason')); + }); + + test('a failing forced example is reported as forced, not shrunk', () { + final f = _fails(G.intIn(0, 1000), (v) => v != 777, examples: [3, 777]); + expect(f.kind, FailureKind.forcedExample); + expect(f.caseId, 'forced:1'); + expect(f.input, '777'); + expect(f.shrinkSteps, 0); + expect(f.replay, contains('PROPERTY_CASE=forced:1')); + }); + + test('a generator that throws is reported as a generator failure', () { + final r = runProperty( + name: 'g', + gen: _Boom(), + body: (_) {}, + config: const PropertyConfig(seed: 1, cases: 3)); + expect(r.failure!.kind, FailureKind.generator); + expect(r.failure!.report, contains('boom')); + }); + }); + + group('shrinking ints', () { + test('finds the smallest failing int above zero', () { + expect(_fails(G.intIn(0, 1000), (v) => v < 50).input, '50'); + }); + + test('shrinks negatives toward zero', () { + expect(_fails(G.intIn(-100, 100), (v) => v > -7).input, '-7'); + }); + + test('shrinks toward the bound nearest zero when zero is out of range', + () { + expect(_fails(G.intIn(10, 100), (_) => false).input, '10'); + expect(_fails(G.intIn(-100, -10), (_) => false).input, '-10'); + }); + + test('an explicit origin is the shrink target', () { + expect(_fails(G.intIn(0, 100, origin: 60), (_) => false).input, '60'); + }); + + test('never leaves the generator domain', () { + final tried = []; + runProperty( + name: 'dom', + gen: G.intIn(13, 91), + body: (v) { + tried.add(v); + if (v > 20) throw StateError('x'); + }, + config: const PropertyConfig(seed: 5)); + expect(tried.every((v) => v >= 13 && v <= 91), isTrue); + }); + + test('a boolean shrinks to false', () { + expect(_fails(G.boolean(), (_) => false).input, 'false'); + }); + + test('elements shrink toward the earlier ones', () { + expect( + _fails(G.elements(['a', 'b', 'c', 'd']), (v) => v == 'a').input, 'b'); + }); + }); + + group('shrinking doubles', () { + test('reaches zero or the nearest bound when everything fails', () { + expect(_fails(G.doubleIn(-5, 5), (_) => false).input, '0.0'); + expect(_fails(G.doubleIn(5, 10), (_) => false).input, '5.0'); + }); + + test('lands just above a threshold, not far from it', () { + final f = _fails(G.doubleIn(0, 100), (v) => v < 3.5); + final v = double.parse(f.input); + expect(v, inInclusiveRange(3.5, 5)); + }); + + test('generation reaches the declared boundaries exactly', () { + final seen = _inputs( + G.doubleIn(-10, 10, boundaries: const [-10, 10, 0, 1e-9]), + cases: 200); + expect(seen, containsAll([-10.0, 10.0, 0.0, 1e-9])); + expect(seen.every((v) => v >= -10 && v <= 10), isTrue); + }); + + test('specials (NaN, infinities) appear when declared, never otherwise', + () { + final withSpecials = _inputs( + G.doubleIn(0, 1, specials: const [double.nan, double.infinity]), + cases: 200); + expect(withSpecials.any((v) => v.isNaN), isTrue); + expect(withSpecials.any((v) => v == double.infinity), isTrue); + final plain = _inputs(G.doubleIn(0, 1), cases: 200); + expect(plain.every((v) => v.isFinite && v >= 0 && v <= 1), isTrue); + }); + + test('a failing special shrinks to a simple finite value if that fails too', + () { + expect( + _fails(G.doubleIn(0, 1, specials: const [double.nan]), (_) => false) + .input, + '0.0'); + }); + + test('shrink candidates stay in range, differ from the value, are bounded', + () { + final g = G.doubleIn(-3, 100, boundaries: const [50]); + for (final v in [99.5, -2.25, 0.0, 3.0, 50.0, -3.0, 1e-12]) { + final c = g.shrink(v).toList(); + expect(c.length, lessThanOrEqualTo(40), reason: '$v'); + for (final x in c) { + expect(x >= -3 && x <= 100, isTrue, reason: '$v -> $x'); + expect(x, isNot(v), reason: '$v -> $x'); + } + } + expect(g.shrink(0.0), isEmpty, reason: 'the origin is already simplest'); + }); + + test('following the closest candidate always terminates', () { + final g = G.doubleIn(-3, 100); + var v = 87.123456; + var steps = 0; + while (true) { + final c = g.shrink(v).toList(); + if (c.isEmpty) break; + v = c.last; + expect(++steps, lessThan(200), reason: 'stuck at $v'); + } + expect(v, 0); + }); + + test('int shrink candidates: origin first, in range, never the value', () { + final g = G.intIn(-20, 40); + for (final v in [-20, -1, 0, 1, 7, 40]) { + final c = g.shrink(v).toList(); + if (v == 0) { + expect(c, isEmpty); + continue; + } + expect(c.first, 0); + expect(c.every((x) => x >= -20 && x <= 40 && x != v), isTrue); + expect(c.length, lessThanOrEqualTo(12)); + } + }); + }); + + group('shrinking lists, tuples, nullables', () { + test('a list shrinks to the shortest failing length of zeros', () { + final f = _fails(G.listOf(G.intIn(0, 9), maxLen: 20), (l) => l.length < 3); + expect(f.input, '[0, 0, 0]'); + }); + + test('a list keeps the one element that matters', () { + final f = _fails( + G.listOf(G.intIn(0, 20), maxLen: 20), (l) => !l.any((e) => e >= 5)); + expect(f.input, '[5]'); + }); + + test('a list never goes under its minimum or over its cap', () { + final lens = []; + runProperty>( + name: 'len', + gen: G.listOf(G.intIn(0, 9), minLen: 2, maxLen: 7), + body: (l) { + lens.add(l.length); + throw StateError('always'); + }, + config: const PropertyConfig(seed: 3)); + expect(lens.every((n) => n >= 2 && n <= 7), isTrue); + expect(_fails(G.listOf(G.intIn(0, 9), minLen: 2, maxLen: 7), (_) => false) + .input, '[0, 0]'); + }); + + test('the size cap holds across all generated cases', () { + final seen = _inputs(G.listOf(G.boolean(), maxLen: 60), cases: 200); + expect(seen.map((l) => l.length).reduce(math.max), 60, + reason: 'the cap itself is reached'); + expect(seen.every((l) => l.length <= 60), isTrue); + }); + + test('tuple components shrink TOGETHER when only a joint step still fails', + () { + // Fails only while a and b are within 3 of each other and a >= 5: moving + // ONE component far breaks the closeness, so only a joint step makes + // progress. + final f = _fails(G.pair(G.intIn(0, 1000), G.intIn(0, 1000)), + (p) => !((p.$1 - p.$2).abs() <= 3 && p.$1 >= 5), + cases: 2000, seed: 5); + final nums = RegExp(r'\d+').allMatches(f.input).map((m) => int.parse(m[0]!)); + final (a, b) = (nums.first, nums.last); + expect((a - b).abs(), lessThanOrEqualTo(3)); + expect(a, lessThanOrEqualTo(12), reason: f.input); + }); + + test('a triple and a quad shrink every component', () { + expect( + _fails(G.triple(G.intIn(0, 9), G.intIn(-9, 9), G.boolean()), + (_) => false) + .input, + '(0, 0, false)'); + expect( + _fails( + G.quad(G.intIn(0, 9), G.doubleIn(0, 1), G.boolean(), + G.elements(const ['x', 'y'])), + (_) => false) + .input, + '(0, 0.0, false, x)'); + }); + + test('a nullable shrinks to null first', () { + expect(_fails(G.nullable(G.intIn(5, 9)), (_) => false).input, 'null'); + expect( + _fails(G.nullable(G.intIn(5, 9), nullProbability: 0), + (v) => v == null || v < 7) + .input, + '7'); + }); + + test('a list and an int inside a tuple each shrink to their own minimum', + () { + final f = _fails( + G.pair(G.listOf(G.intIn(0, 9), minLen: 1, maxLen: 12), G.intIn(0, 11)), + (p) => !(p.$1.length >= 6 && p.$2 >= 4)); + expect(f.input, '([0, 0, 0, 0, 0, 0], 4)'); + }); + }); + + group('shrink bound', () { + test('the shrink limit caps the extra body runs', () { + var calls = 0; + final r = runProperty>( + name: 'limit', + gen: G.listOf(G.intIn(0, 1000), minLen: 40, maxLen: 50), + body: (_) { + calls++; + throw StateError('always'); + }, + config: const PropertyConfig(seed: 3, shrinkLimit: 5)); + expect(r.passed, isFalse); + expect(calls, lessThanOrEqualTo(1 + 5)); + expect(r.failure!.shrinkSteps, lessThanOrEqualTo(5)); + expect(r.failure!.report, contains('shrink limit: 5')); + }); + + test('with a limit of zero the original input is reported', () { + final f = _fails(G.intIn(500, 1000), (_) => false, shrinkLimit: 0); + expect(f.shrinkSteps, 0); + expect(int.parse(f.input), inInclusiveRange(500, 1000)); + }); + + test('the report shows the original input when shrinking changed it', () { + final f = _fails(G.intIn(0, 1000), (v) => v < 50, seed: 8); + expect(f.report, contains('original input')); + final z = _fails(G.intIn(0, 1000), (_) => false, examples: [0]); + expect(z.report, isNot(contains('original input'))); + }); + }); + + group('wall budget', () { + test('PROPERTY_BUDGET_MS is the hard limit over a property\'s looser default', + () { + // A law registers a 5 s in-test guard (headroom for a loaded parallel + // suite); the serial acceptance run sets PROPERTY_BUDGET_MS=2000 and + // that must be what fails a property, not the 5 s default. + final c = PropertyConfig.fromEnvironment( + const {'PROPERTY_BUDGET_MS': '2000'}, + defaultBudget: const Duration(seconds: 5), + ); + expect(c.budget, const Duration(milliseconds: 2000)); + var ticks = 0; + final r = runProperty( + name: 'serial acceptance', + gen: G.intIn(0, 9), + body: (_) {}, + config: c, + // 1 s per reading: over 2 s on the third, well inside the 5 s default. + elapsed: () => Duration(seconds: ticks++), + ); + expect(r.failure!.kind, FailureKind.budget); + expect(r.failure!.report, contains('2000 ms')); + }); + + test('a property over its budget fails, naming budget and cases run', () { + var ticks = 0; + final r = runProperty( + name: 'slow', + gen: G.intIn(0, 9), + body: (_) {}, + config: const PropertyConfig(seed: 1, cases: 200), + elapsed: () => Duration(milliseconds: 500 * ticks++), + ); + expect(r.failure!.kind, FailureKind.budget); + expect(r.casesRun, lessThan(200)); + expect(r.failure!.report, contains('2000 ms')); + expect(r.failure!.report, contains('budget')); + }); + + test('shrinking counts against the budget and reports the best so far', () { + // The clock reads past the budget after the 6th reading: the failure at + // case 0 is found, shrinking gets a few attempts, then stops. + var reads = 0; + final r = runProperty>( + name: 'slow shrink', + gen: G.listOf(G.intIn(0, 1000), minLen: 40, maxLen: 50), + body: (_) => throw StateError('always'), + config: const PropertyConfig(seed: 1), + elapsed: () => Duration(milliseconds: 500 * reads++), + ); + final f = r.failure!; + expect(f.kind, FailureKind.counterexample); + expect(f.shrinkOverBudget, isTrue); + expect(f.report, contains('over budget (shrinking)')); + expect(f.shrinkSteps, lessThanOrEqualTo(6), + reason: 'unbudgeted, this shrink takes dozens of runs'); + // The best-so-far input is still a valid, failing input of the domain. + final len = RegExp(', ').allMatches(f.input).length + 1; + expect(len, inInclusiveRange(40, 50)); + }); + + test('shrinking inside the budget is not marked', () { + final r = runProperty( + name: 'quick shrink', + gen: G.intIn(500, 1000), + body: (_) => throw StateError('always'), + config: const PropertyConfig(seed: 1), + elapsed: () => const Duration(milliseconds: 1), + ); + expect(r.failure!.shrinkOverBudget, isFalse); + expect(r.failure!.report, isNot(contains('over budget'))); + expect(r.failure!.input, '500'); + }); + + test('a single-case replay keeps shrinking whatever the clock says', () { + final r = runProperty( + name: 'replay shrink', + gen: G.intIn(500, 1000), + body: (_) => throw StateError('always'), + config: const PropertyConfig(seed: 1, caseOnly: '0'), + elapsed: () => const Duration(hours: 1), + ); + expect(r.failure!.shrinkOverBudget, isFalse); + expect(r.failure!.input, '500'); + }); + + test('a property inside its budget passes', () { + final r = runProperty( + name: 'fast', + gen: G.intIn(0, 9), + body: (_) {}, + config: const PropertyConfig(seed: 1, cases: 20), + elapsed: () => const Duration(milliseconds: 5), + ); + expect(r.passed, isTrue); + }); + + test('a counterexample is reported before the budget', () { + final r = runProperty( + name: 'both', + gen: G.intIn(0, 9), + body: (_) => throw StateError('x'), + config: const PropertyConfig(seed: 1), + elapsed: () => const Duration(hours: 1), + ); + expect(r.failure!.kind, FailureKind.counterexample); + }); + + test('the default clock is a real stopwatch', () { + final r = runProperty( + name: 'spin', + gen: G.intIn(0, 9), + body: (_) { + final sw = Stopwatch()..start(); + while (sw.elapsedMilliseconds < 4) {} + }, + config: const PropertyConfig( + seed: 1, cases: 50, budget: Duration(milliseconds: 1)), + ); + expect(r.failure?.kind, FailureKind.budget); + }); + + test('a single-case replay is not held to the budget', () { + final r = runProperty( + name: 'replay', + gen: G.intIn(0, 9), + body: (_) {}, + config: const PropertyConfig(seed: 1, caseOnly: '3'), + elapsed: () => const Duration(hours: 1), + ); + expect(r.passed, isTrue); + }); + }); + + group('forAll', () { + var ran = 0; + forAll('self: forAll runs examples and generated cases', + G.intIn(0, 100), (v) { + ran++; + expect(v >= 0 && v <= 100, isTrue); + }, examples: const [0, 100], cases: 25); + + test('forAll ran its forced examples plus its cases', () { + expect(ran, anyOf(0, 27), + reason: '0 when this test alone is selected by name'); + }); + + forAll<(int, int)>('self: addition commutes', G.pair(G.intIn(-99, 99), G.intIn(-99, 99)), + (p) => expect(p.$1 + p.$2, p.$2 + p.$1)); + }); +} + +class _Boom extends Gen { + @override + int generate(Rng r, int size) => throw StateError('boom'); + @override + Iterable shrink(int v) => const []; +} diff --git a/tool/bench_aggregate.dart b/tool/bench_aggregate.dart new file mode 100644 index 0000000..d14e88d --- /dev/null +++ b/tool/bench_aggregate.dart @@ -0,0 +1,215 @@ +// Run with: dart compile exe tool/bench_aggregate.dart -o /tmp/bench_aggregate +// /tmp/bench_aggregate +import 'dart:math' as math; +import 'dart:typed_data'; + +import 'package:openstrap_analytics/aggregate.dart'; +import 'package:openstrap_analytics/onehz.dart'; + +Object? _sink; + +void main() { + final values = DoubleColumn(Float64List.fromList( + [for (var i = 0; i < 86400; i++) ((i * 7919) % 2003 - 1001).toDouble()])); + final summary = both( + both(Sum((batch) => batch.values), const Count()), + both(Min((batch) => batch.values), + Max((batch) => batch.values)), + ); + final beats = _beats(30000); + final origin = beats.timeMs.first; + final composed = hrvTimeAggregator(originMs: origin); + + final rows = <(String, String, int)>[ + ('86,400 doubles', 'hand sum/count/min/max', + _medianMicros(() => _doubleHandLoop(values))), + ('86,400 doubles', 'both blocks, one slice', + _medianMicros(() => summary.run(values))), + ('86,400 doubles', 'both blocks, 60-row slices', + _medianMicros(() => _sliced(summary, values, 60))), + ('30,000 beats', 'hrvTime() batch', _medianMicros( + () => hrvTime(beats.nnMs.toList(), nnTimesMs: beats.timeMs.toList()))), + ('30,000 beats', 'hand HRV loop', _medianMicros(() => _handHrv(beats))), + ('30,000 beats', 'composed, one slice', + _medianMicros(() => composed.run(beats))), + ('30,000 beats', 'composed, 24 slices + combine', + _medianMicros(() => _combined(composed, beats, 24))), + ('30,000 beats', 'composed, 1-row slices', + _medianMicros(() => _sliced(composed, beats, 1))), + ]; + + final reference = hrvTime(beats.nnMs.toList(), nnTimesMs: beats.timeMs.toList()) + .value!; + final result = composed.run(beats); + final parity = _hrvParity(result, reference); + final handUs = rows[4].$3; + final oneSliceUs = rows[5].$3; + final chunksUs = rows[6].$3; + + print('Median microseconds (7 timed runs after 3 warm-up runs)'); + print('${'workload'.padRight(16)} ${'operation'.padRight(36)} us'); + for (final row in rows) { + print('${row.$1.padRight(16)} ${row.$2.padRight(36)} ${row.$3}'); + } + print('parity composed vs hrvTime (1e-9): $parity'); + print('HRV ratios: one slice ${(oneSliceUs / handUs).toStringAsFixed(2)}x; ' + '24 slices ${(chunksUs / handUs).toStringAsFixed(2)}x (target <= 2.00x)'); + if (!parity || _sink == null) { + throw StateError('Benchmark parity or result observability failed'); + } +} + +(double, int, double, double) _doubleHandLoop(DoubleColumn batch) { + var sum = 0.0; + var count = 0; + var min = double.infinity; + var max = -double.infinity; + for (final value in batch.values) { + if (value.isNaN) continue; + sum += value; + count++; + if (value < min) min = value; + if (value > max) max = value; + } + return (sum, count, min, max); +} + +Beats _beats(int count) { + final random = math.Random(0xA661E6A7); + final nn = Float64List(count); + final times = Float64List(count); + var time = 1000000.0; + var interval = 900.0; + for (var i = 0; i < count; i++) { + interval = (interval + (random.nextDouble() - .5) * 24) + .clamp(800.0, 1100.0) + .toDouble(); + final gap = i > 0 && random.nextDouble() < .0007 + ? 1000 + random.nextDouble() * 30000 + : 0.0; + time += interval + gap; + nn[i] = interval; + times[i] = time; + } + return Beats(nn, times); +} + +R _combined(Aggregator aggregator, Beats batch, + int chunks) { + dynamic state = aggregator.empty(); + for (var chunk = 0; chunk < chunks; chunk++) { + final start = batch.length * chunk ~/ chunks; + final end = batch.length * (chunk + 1) ~/ chunks; + state = aggregator.combine(state, aggregator.fold(batch, start, end)); + } + return aggregator.finish(state); +} + +R _sliced, R>(Aggregator aggregator, B batch, + int sliceSize) { + var state = aggregator.empty(); + for (var start = 0; start < batch.length; start += sliceSize) { + final end = math.min(start + sliceSize, batch.length); + state = aggregator.addSlice(state, batch, start, end); + } + return aggregator.finish(state); +} + +int _medianMicros(Object? Function() action) { + for (var warm = 0; warm < 3; warm++) { + _sink = action(); + } + final samples = []; + for (var run = 0; run < 7; run++) { + final watch = Stopwatch()..start(); + _sink = action(); + watch.stop(); + samples.add(watch.elapsedMicroseconds); + } + samples.sort(); + return samples[samples.length ~/ 2]; +} + +bool _hrvParity(HrvTime a, HrvTime b) => + a.nBeats == b.nBeats && + _same(a.rmssd, b.rmssd) && + _same(a.sdnn, b.sdnn) && + _same(a.sdann, b.sdann) && + _same(a.sdnnIndex, b.sdnnIndex) && + _same(a.pnn50, b.pnn50) && + _same(a.diffAcf1, b.diffAcf1); + +bool _same(double? a, double? b) { + if (a == null || b == null) return a == b; + return (a - b).abs() <= 1e-9 + 1e-9 * b.abs(); +} + +class _Moments { + var count = 0; + var mean = 0.0; + var m2 = 0.0; + + void add(double value) { + count++; + final delta = value - mean; + mean += delta / count; + m2 += delta * (value - mean); + } + + double? get sampleSd => count < 2 ? null : math.sqrt(m2 / (count - 1)); +} + +/// Direct baseline for the same metrics, deliberately independent of the +/// aggregate graph. +HrvTime _handHrv(Beats beats) { + final levels = _Moments(); + final bins = {}; + var pairs = 0, over50 = 0, lagPairs = 0; + var sum = 0.0, squares = 0.0, products = 0.0, endpoints = 0.0; + for (var i = 0; i < beats.length; i++) { + final nn = beats.nnMs[i]; + levels.add(nn); + final key = ((beats.timeMs[i] - beats.timeMs.first) / 300000.0).floor(); + (bins[key] ??= _Moments()).add(nn); + if (i == 0 || beats.timeMs[i] - beats.timeMs[i - 1] > nn + 0.5) continue; + final diff = nn - beats.nnMs[i - 1]; + pairs++; + sum += diff; + squares += diff * diff; + if (diff.abs() > 50) over50++; + if (i >= 2 && beats.timeMs[i - 1] - beats.timeMs[i - 2] <= + beats.nnMs[i - 1] + 0.5) { + final previous = beats.nnMs[i - 1] - beats.nnMs[i - 2]; + lagPairs++; + products += previous * diff; + endpoints += previous + diff; + } + } + if (levels.count < 2) return HrvTime(nBeats: levels.count); + double? acf1; + if (pairs >= 30) { + final mean = sum / pairs; + final variance = squares - sum * mean; + if (variance > 0) { + acf1 = (products - mean * endpoints + lagPairs * mean * mean) / variance; + } + } + final jittery = acf1 != null && acf1 < kNnDiffAcf1Floor; + final means = _Moments(); + var sdSum = 0.0, binCount = 0; + for (final bin in bins.values) { + if (bin.count < 2) continue; + means.add(bin.mean); + sdSum += bin.sampleSd!; + binCount++; + } + return HrvTime( + nBeats: levels.count, + rmssd: pairs > 0 && !jittery ? math.sqrt(squares / pairs) : null, + pnn50: pairs > 0 && !jittery ? 100.0 * over50 / pairs : null, + sdnn: levels.sampleSd, + sdann: binCount >= 2 ? means.sampleSd : null, + sdnnIndex: binCount >= 2 ? sdSum / binCount : null, + diffAcf1: acf1, + ); +} diff --git a/tool/bench_night_bins.dart b/tool/bench_night_bins.dart new file mode 100644 index 0000000..1132f81 --- /dev/null +++ b/tool/bench_night_bins.dart @@ -0,0 +1,173 @@ +// Before/after for two night metrics edge re-runs on every pass: the 30-min +// RSA bins behind breathing-rate variability (edge `_respPerWindow`, copied +// here) and PRSA deceleration/acceleration capacity. 32 passes of 15 min; each +// pass folds the newly settled beats and asks with a ~100-beat provisional +// tail. "After" keeps closed bins' rates and folds PRSA with prsaAggregator. + +import 'dart:typed_data'; + +import 'package:openstrap_analytics/aggregate.dart'; +import 'package:openstrap_analytics/onehz.dart'; +import '../test/onehz/phase0_parity_test.dart' as frozen; + +const _windowMs = 1800000.0; + +/// Edge's `_respPerWindow`, verbatim apart from the name. +List batchBins(List nn, List nnTimes) { + if (nn.isEmpty || nn.length != nnTimes.length) return const []; + final t0 = nnTimes.first; + final binsNn = >{}; + final binsTs = >{}; + for (var i = 0; i < nn.length; i++) { + final idx = ((nnTimes[i] - t0) / _windowMs).floor(); + (binsNn[idx] ??= []).add(nn[i]); + (binsTs[idx] ??= []).add(nnTimes[i]); + } + final out = []; + final idxs = binsNn.keys.toList()..sort(); + for (final idx in idxs) { + final segNn = binsNn[idx]!; + if (segNn.length < 60) continue; + final r = rsaRespRate(segNn, binsTs[idx]!, artifactFraction: 0.0); + final b = r.present ? r.value!.brpm : null; + if (b != null) out.add(b); + } + return out; +} + +/// Closed bins (a settled beat exists in a later bin) are scored once. +class BinStream { + final closed = []; + final _openNn = [], _openTs = []; + double? _t0; + int _openIdx = 0; + + int _idx(double t) => ((t - _t0!) / _windowMs).floor(); + + void _close() { + if (_openNn.isNotEmpty) closed.addAll(batchBinsFrom(_openNn, _openTs)); + _openNn.clear(); + _openTs.clear(); + } + + // One bin's rate, through the batch on that bin alone. + List batchBinsFrom(List nn, List ts) { + if (nn.length < 60) return const []; + final r = rsaRespRate(nn, ts, artifactFraction: 0.0); + return r.present && r.value!.brpm != null ? [r.value!.brpm!] : const []; + } + + void fold(List nn, List ts) { + for (var i = 0; i < nn.length; i++) { + _t0 ??= ts[i]; + final k = _idx(ts[i]); + if (k != _openIdx) { + _close(); + _openIdx = k; + } + _openNn.add(nn[i]); + _openTs.add(ts[i]); + } + } + + List snapshot(List tailNn, List tailTs) { + if (_t0 == null) return batchBins(tailNn, tailTs); + final out = [...closed]; + var nn = [..._openNn], ts = [..._openTs]; + var k = _openIdx; + for (var i = 0; i < tailNn.length; i++) { + final j = _idx(tailTs[i]); + if (j != k) { + out.addAll(batchBinsFrom(nn, ts)); + nn = []; + ts = []; + k = j; + } + nn.add(tailNn[i]); + ts.add(tailTs[i]); + } + out.addAll(batchBinsFrom(nn, ts)); + return out; + } +} + +bool _sameList(List a, List b) { + if (a.length != b.length) return false; + for (var i = 0; i < a.length; i++) { + if (a[i] != b[i]) return false; + } + return true; +} + +bool _samePrsa(Metric a, Metric b) { + if (a.present != b.present || a.note != b.note) return false; + if (!a.present) return true; + final x = a.value!, y = b.value!; + return x.capacity == y.capacity && + x.anchors == y.anchors && + _sameList(x.profile, y.profile) && + a.confidence == b.confidence; +} + +void main() { + final night = frozen.genBeats(9031, 33000, period: 55, amp: 60); + final bins = BinStream(); + final dcAgg = prsaAggregator(deceleration: true); + final acAgg = prsaAggregator(deceleration: false); + Object? dc = dcAgg.empty(), ac = acAgg.empty(); + var from = 0, parity = true; + var binsAfter = 0, binsBefore = 0, prsaAfter = 0, prsaBefore = 0; + final start = night.tMs.first; + print('pass bins after/before us prsa after/before us parity'); + for (var pass = 1; pass <= 32; pass++) { + final until = start + pass * 15 * 60 * 1000.0; + var end = from; + while (end < night.nn.length && night.tMs[end] <= until) { + end++; + } + if (end == from) continue; + final tailEnd = (end + 100).clamp(0, night.nn.length); + final tailNn = night.nn.sublist(end, tailEnd); + final tailTs = night.tMs.sublist(end, tailEnd); + final allNn = night.nn.sublist(0, tailEnd); + final allTs = night.tMs.sublist(0, tailEnd); + + var w = Stopwatch()..start(); + bins.fold(night.nn.sublist(from, end), night.tMs.sublist(from, end)); + final b1 = bins.snapshot(tailNn, tailTs); + final bA = w.elapsedMicroseconds; + w = Stopwatch()..start(); + final b0 = batchBins(allNn, allTs); + final bB = w.elapsedMicroseconds; + + w = Stopwatch()..start(); + final settled = DoubleColumn(Float64List.fromList(night.nn.sublist(from, end))); + dc = dcAgg.addSlice(dc, settled, 0, settled.length); + ac = acAgg.addSlice(ac, settled, 0, settled.length); + final tail = DoubleColumn(Float64List.fromList(tailNn)); + Metric snap( + Aggregator> agg, Object? s) { + final copy = agg.decodeState(agg.encodeState(s)); + return agg.finish(agg.addSlice(copy, tail, 0, tail.length)); + } + + final dc1 = snap(dcAgg, dc), ac1 = snap(acAgg, ac); + final pA = w.elapsedMicroseconds; + w = Stopwatch()..start(); + final dc0 = decelerationCapacity(allNn), ac0 = accelerationCapacity(allNn); + final pB = w.elapsedMicroseconds; + from = end; + + final same = _sameList(b1, b0) && _samePrsa(dc1, dc0) && _samePrsa(ac1, ac0); + parity = parity && same; + binsAfter += bA; + binsBefore += bB; + prsaAfter += pA; + prsaBefore += pB; + print('${pass.toString().padLeft(4)} ${'$bA/$bB'.padLeft(18)} ' + '${'$pA/$pB'.padLeft(18)} $same'); + } + print('total bins after $binsAfter us, before $binsBefore us; ' + 'prsa after $prsaAfter us, before $prsaBefore us; parity $parity'); + if (!parity) throw StateError('parity failed'); +} diff --git a/tool/bench_night_nn.dart b/tool/bench_night_nn.dart new file mode 100644 index 0000000..5400960 --- /dev/null +++ b/tool/bench_night_nn.dart @@ -0,0 +1,350 @@ +// Measurement tool: three ways to get the night's NN on every overnight derive +// pass. Not a test; no lib/ changes. AOT-friendly, seeded, no I/O except print. +// +// today : correctRr(sleep-window prefix) every pass (O(n) per pass) +// (a) : per-night RrCorrector, fold only the new beats, snapshot each pass +// (claimed exact: settled ++ snapshot().tail == correctRr(prefix)) +// (b) : correct the WHOLE day once, slice the result to the sleep window +// (differs near the window edges: correctRr clips windows at slice ends) +// +// Day layout: 8 h wake, 8 h sleep, 8 h wake (both window edges are interior to +// the day). 32 passes, one per 15 min of the night; the prefix grows each pass. +// +// dart compile exe tool/bench_night_nn.dart -o /tmp/bench_night_nn +// /tmp/bench_night_nn [seed ...] (default seeds 1 2 3) +import 'dart:math' as math; + +import 'package:openstrap_analytics/onehz.dart'; + +const _startMs = 1.7e12; +const _hMs = 3600e3; +const _sleepStartMs = _startMs + 8 * _hMs; +const _sleepEndMs = _startMs + 16 * _hMs; +const _passMs = 15 * 60e3; +const _passes = 32; + +class Day { + final List rr = [], ts = []; +} + +Day synth(int seed) { + final rnd = math.Random(seed); + final d = Day(); + final wakeRate = 0.02 + 0.01 * rnd.nextDouble(); // 2..3 % of beats + final sleepRate = 0.005 + 0.01 * rnd.nextDouble(); // 0.5..1.5 % + var t = 0.0; // ms since start + final phase = rnd.nextDouble() * 6.28; + void emit(double v) { + v = v.roundToDouble(); + t += v; + d.rr.add(v); + d.ts.add(((_startMs + t) / 1000).floorToDouble() * 1000); + } + + // Multi-beat garbage runs (4..8 beats) and one 2-min dropout inside the sleep. + final runAt = [ + for (var i = 0; i < 3; i++) 8 * 3600e3 + (1 + 6 * rnd.nextDouble()) * _hMs + ]..sort(); + final dropAt = 8 * 3600e3 + (2 + 4 * rnd.nextDouble()) * _hMs; + var runIdx = 0; + var dropped = false; + const endT = 24 * 3600e3; + while (t < endT) { + final sleep = t >= 8 * 3600e3 && t < 16 * 3600e3; + final s = t / 1000; + final bpm = sleep + ? 57 + 7 * math.sin(s / 5400 + seed) + : 85 + 13 * math.sin(s / 1900 + seed); + final base = 60000 / bpm; + final v = base + + (sleep ? 28 : 14) * math.sin(2 * math.pi * (sleep ? .24 : .27) * s + phase) + + 18 * math.sin(2 * math.pi * 0.1 * s) + + (sleep ? 6 : 8) * (rnd.nextDouble() - .5) * 2; + if (sleep && !dropped && t >= dropAt) { + dropped = true; + t += 120e3; + continue; + } + if (sleep && runIdx < runAt.length && t >= runAt[runIdx]) { + runIdx++; + final len = 4 + rnd.nextInt(5); + for (var k = 0; k < len; k++) { + emit(250 + rnd.nextInt(2100).toDouble()); + } + continue; + } + if (rnd.nextDouble() < (sleep ? sleepRate : wakeRate)) { + final u = rnd.nextDouble(); + if (u < 0.4) { + emit(v * (0.55 + 0.15 * rnd.nextDouble())); // ectopic + pause + emit(v * (1.3 + 0.2 * rnd.nextDouble())); + } else if (u < 0.6) { + emit(v * (1.9 + 0.2 * rnd.nextDouble())); // missed beat + } else if (u < 0.85) { + final f = 0.35 + 0.3 * rnd.nextDouble(); // extra beat + emit(v * f); + emit(v * (1 - f)); + } else { + emit(250 + rnd.nextInt(2100).toDouble()); // lone garbage beat + } + continue; + } + emit(v); + } + return d; +} + +/// Replica of correctRr's private clock, so whole-day output can be mapped back +/// to input beat indices (the kept beats' times are tAll[i] bit for bit). +List beatTimes(List rr, List ts) { + final t = List.filled(rr.length, 0); + final origin = ts[0] - rr[0]; + var cur = 0.0; + for (var i = 0; i < rr.length; i++) { + final dropout = i > 0 && (ts[i] - ts[i - 1]) - rr[i] > 1000; + final anchored = ts[i] - origin; + cur = (dropout && anchored > cur) ? anchored : cur + rr[i]; + t[i] = cur; + } + return t; +} + +int lowerBound(List a, double x, [int lo = 0]) { + var hi = a.length; + while (lo < hi) { + final m = (lo + hi) >> 1; + if (a[m] < x) { + lo = m + 1; + } else { + hi = m; + } + } + return lo; +} + +/// beat index -> NN value for one correctRr result (kept beats only). +Map byBeat(RrCorrectionResult r, List tAll, int offset) { + final m = {}; + var j = 0; + for (var k = 0; k < r.nn.length; k++) { + j = lowerBound(tAll, r.nnTimesMs[k], j); + if (tAll[j] != r.nnTimesMs[k]) throw StateError('beat map broke'); + m[j + offset] = r.nn[k]; + } + return m; +} + +class Down { + final double? rmssd, sdnn, pnn50, dc, ac, lf, hf, total; + Down(this.rmssd, this.sdnn, this.pnn50, this.dc, this.ac, this.lf, this.hf, + this.total); + List get all => [rmssd, sdnn, pnn50, dc, ac, lf, hf, total]; + static const names = ['RMSSD', 'SDNN', 'pNN50', 'DC', 'AC', 'LF', 'HF', 'TotP']; +} + +Down downstream(List nn, List times, double clean) { + final art = 1 - clean; + final h = hrvTime(nn, nnTimesMs: times, artifactFraction: art).value; + final dc = decelerationCapacity(nn).value; + final ac = accelerationCapacity(nn).value; + final f = hrvFreq(nn, times, artifactFraction: art).value; + return Down(h?.rmssd, h?.sdnn, h?.pnn50, dc?.capacity, ac?.capacity, f?.lf, + f?.hf, f?.total); +} + +String ms(int us) => (us / 1000).toStringAsFixed(2); +String f(double? x, [int p = 3]) => x == null ? 'null' : x.toStringAsFixed(p); + +bool sameList(List a, List b) { + if (a.length != b.length) return false; + for (var i = 0; i < a.length; i++) { + if (a[i] != b[i]) return false; + } + return true; +} + +void run(int seed) { + final day = synth(seed); + final n = day.rr.length; + final i0 = lowerBound(day.ts, _sleepStartMs); + final i1All = lowerBound(day.ts, _sleepEndMs); + print('\n==== seed $seed: day $n beats, sleep window beats [$i0,$i1All) ' + '= ${i1All - i0} ===='); + + // Whole day, as edge already does: fold (timed) then derive the day result. + final swW = Stopwatch()..start(); + final dayCorr = RrCorrector(); + dayCorr.fold(day.rr, tsMs: day.ts); + dayCorr.snapshot(); + swW.stop(); + final swWd = Stopwatch()..start(); + final whole = correctRr(day.rr, rrTsMs: day.ts); + swWd.stop(); + final tWhole = beatTimes(day.rr, day.ts); + final wholeMap = byBeat(whole, tWhole, 0); + // normal-beat prefix sums for (b)'s cleanFraction + final normPre = List.filled(n + 1, 0); + for (var i = 0; i < n; i++) { + normPre[i + 1] = normPre[i] + (whole.classes[i] == BeatClass.normal ? 1 : 0); + } + + // (a) state + final corr = RrCorrector(); + final aNn = [], aT = []; + var folded = i0; + + final tToday = [], tA = [], tB = []; + final exact = []; + final diffCnt = [], diffMaxMs = [], dClean = []; + final nearStart = [], nearEnd = []; // diffs within 60 s of an edge + final farDiff = []; // diffs > 300 s from both edges + final worstEdgeS = []; // max over diffs of min(dist start, dist end) + final rel = List.generate(8, (_) => []); + final abs = List.generate(8, (_) => []); + Down? finalToday, finalB; + var finalLine = ''; + + for (var p = 1; p <= _passes; p++) { + final cut = _sleepStartMs + p * _passMs; + final i1 = lowerBound(day.ts, cut); + final rr = day.rr.sublist(i0, i1), ts = day.ts.sublist(i0, i1); + + final sw = Stopwatch()..start(); + final today = correctRr(rr, rrTsMs: ts); + sw.stop(); + tToday.add(sw.elapsedMicroseconds); + + sw + ..reset() + ..start(); + final settled = corr.fold(day.rr.sublist(folded, i1), + tsMs: day.ts.sublist(folded, i1)); + folded = i1; + aNn.addAll(settled.nn); + aT.addAll(settled.nnTimes); + final snap = corr.snapshot(); + final aOutNn = [...aNn, ...snap.tailNn]; + final aOutT = [...aT, ...snap.tailNnTimes]; + final aClean = snap.cleanFraction; + sw.stop(); + tA.add(sw.elapsedMicroseconds); + exact.add(sameList(aOutNn, today.nn) && + sameList(aOutT, today.nnTimesMs) && + aClean == today.cleanFraction); + + sw + ..reset() + ..start(); + final bNn = [], bT = []; + final base = tWhole[i0]; + final lo = lowerBound(whole.nnTimesMs, tWhole[i0]); + final hi = lowerBound(whole.nnTimesMs, tWhole[i1 - 1], lo); + final hiIncl = (hi < whole.nn.length && whole.nnTimesMs[hi] == tWhole[i1 - 1]) + ? hi + 1 + : hi; + for (var k = lo; k < hiIncl; k++) { + bNn.add(whole.nn[k]); + bT.add(whole.nnTimesMs[k] - base + rr[0]); // rebase to the slice's frame + } + final bClean = (normPre[i1] - normPre[i0]) / (i1 - i0); + sw.stop(); + tB.add(sw.elapsedMicroseconds); + + // Diff by input beat index. + final tAllToday = beatTimes(rr, ts); + final tm = byBeat(today, tAllToday, i0); + var cnt = 0, near0 = 0, near1 = 0, far = 0; + var mx = 0.0, worst = 0.0; + final keys = {...tm.keys, ...wholeMap.keys.where((k) => k >= i0 && k < i1)}; + for (final k in keys) { + final x = tm[k], y = wholeMap[k]; + if (x != null && y != null && x == y) continue; + cnt++; + if (x != null && y != null) mx = math.max(mx, (x - y).abs()); + final ds = (day.ts[k] - _sleepStartMs) / 1000; + final de = (cut - day.ts[k]) / 1000; + final dm = math.min(ds, de); + if (ds <= 60) near0++; + if (de <= 60) near1++; + if (dm > 300) far++; + worst = math.max(worst, dm); + } + diffCnt.add(cnt); + diffMaxMs.add(mx); + dClean.add(bClean - today.cleanFraction); + nearStart.add(near0); + nearEnd.add(near1); + farDiff.add(far); + worstEdgeS.add(worst); + + final dt = downstream(today.nn, today.nnTimesMs, today.cleanFraction); + final db = downstream(bNn, bT, bClean); + for (var m = 0; m < 8; m++) { + final x = dt.all[m], y = db.all[m]; + if (x == null || y == null) continue; + abs[m].add((y - x).abs()); + rel[m].add(x == 0 ? 0 : (y - x).abs() / x.abs() * 100); + } + if (p == _passes) { + finalToday = dt; + finalB = db; + finalLine = 'final pass: today nn=${today.nn.length} clean=' + '${today.cleanFraction.toStringAsFixed(5)} | (b) nn=${bNn.length} ' + 'clean=${bClean.toStringAsFixed(5)}'; + } + } + + double sum(Iterable x) => x.fold(0.0, (a, b) => a + b); + double mean(List x) => x.isEmpty ? 0 : sum(x) / x.length; + double lastHalf(List x) => mean(x.sublist(x.length ~/ 2)); + + print('-- 1. time (ms), AOT, single run per pass --'); + print('pass: ${[1, 8, 16, 24, 32].map((p) => 'p$p').join(' ')}'); + String row(String name, List t) => + '${name.padRight(8)}${[1, 8, 16, 24, 32].map((p) => ms(t[p - 1]).padLeft(8)).join()}' + ' total ${ms(t.fold(0, (a, b) => a + b)).padLeft(9)}' + ' mean/pass ${ms((sum(t) / t.length).round())}' + ' mean last16 ${(lastHalf(t) / 1000).toStringAsFixed(2)}'; + print(row('today', tToday)); + print(row('(a)', tA)); + print(row('(b)', tB)); + print('whole-day RrCorrector fold+snapshot (paid by edge): ${ms(swW.elapsedMicroseconds)} ms;' + ' whole-day correctRr: ${ms(swWd.elapsedMicroseconds)} ms'); + + print('-- 2. exactness of (a) vs today: all $_passes passes bit-identical = ' + '${exact.every((e) => e)} (${exact.where((e) => e).length}/$_passes) --'); + + print('-- 3. (b) vs today --'); + print(finalLine); + print('differing NN beats (changed value or kept in one only):'); + print(' final pass : ${diffCnt.last} of ' + '${i1All - i0} beats; within 60 s of window start ${nearStart.last}, ' + 'of window end ${nearEnd.last}, >300 s from both ${farDiff.last}; ' + 'farthest from an edge ${worstEdgeS.last.toStringAsFixed(1)} s; ' + 'max |dNN| ${diffMaxMs.last.toStringAsFixed(1)} ms; ' + 'dClean ${dClean.last.toStringAsExponential(2)}'); + print(' mean/pass : ${mean(diffCnt).toStringAsFixed(1)} beats; ' + 'within 60 s of start ${mean(nearStart).toStringAsFixed(1)}, of end ' + '${mean(nearEnd).toStringAsFixed(1)}, >300 s from both ' + '${mean(farDiff).toStringAsFixed(1)}; farthest max over passes ' + '${worstEdgeS.reduce(math.max).toStringAsFixed(1)} s; ' + 'max |dNN| over passes ${diffMaxMs.reduce(math.max).toStringAsFixed(1)} ms; ' + 'mean |dClean| ${mean([for (final x in dClean) x.abs()]).toStringAsExponential(2)}'); + print('downstream: metric | today(final) | (b)(final) | abs d final | rel% final' + ' | mean abs d | mean rel%'); + for (var m = 0; m < 8; m++) { + final x = finalToday!.all[m], y = finalB!.all[m]; + final ad = (x == null || y == null) ? null : (y - x).abs(); + final rd = (ad == null || x == 0) ? null : ad / x!.abs() * 100; + print(' ${Down.names[m].padRight(6)}| ${f(x).padLeft(11)} | ${f(y).padLeft(11)}' + ' | ${f(ad, 4).padLeft(9)} | ${f(rd, 3).padLeft(8)} | ' + '${f(mean(abs[m]), 4).padLeft(9)} | ${f(mean(rel[m]), 3).padLeft(8)}'); + } +} + +void main(List args) { + final seeds = args.isEmpty ? [1, 2, 3] : args.map(int.parse).toList(); + for (final s in seeds) { + run(s); + } +} diff --git a/tool/bench_phase0.dart b/tool/bench_phase0.dart new file mode 100644 index 0000000..c1edee8 --- /dev/null +++ b/tool/bench_phase0.dart @@ -0,0 +1,296 @@ +// Round 15, phase 0 benchmark. The frozen before implementations live in the +// parity characterization test and are deliberately imported rather than +// copied again: that file is the versioned source of truth for this comparison. + +import 'package:openstrap_analytics/onehz.dart'; +import '../test/onehz/phase0_parity_test.dart' as frozen; + +int _medianUs(void Function() f, {int runs = 3}) { + if (runs == 1) { + final sw = Stopwatch()..start(); + f(); + sw.stop(); + return sw.elapsedMicroseconds; + } + final samples = []; + f(); // warm up JIT and lazy static initialization. + for (var i = 0; i < runs; i++) { + final sw = Stopwatch()..start(); + f(); + sw.stop(); + samples.add(sw.elapsedMicroseconds); + } + samples.sort(); + return samples[samples.length ~/ 2]; +} + +int _medianPreparedUs(void Function() prepare, void Function() f, + {int runs = 3}) { + if (runs == 1) { + prepare(); + final sw = Stopwatch()..start(); + f(); + sw.stop(); + return sw.elapsedMicroseconds; + } + prepare(); + f(); // JIT warmup; each measured pass gets the same prepared cache state. + final samples = []; + for (var i = 0; i < runs; i++) { + prepare(); + final sw = Stopwatch()..start(); + f(); + sw.stop(); + samples.add(sw.elapsedMicroseconds); + } + samples.sort(); + return samples[samples.length ~/ 2]; +} + +bool _same(double? a, double? b) => + a == null || b == null ? a == b : (a.isNaN && b.isNaN) || a == b; + +bool _sameList(List? a, List? b) { + if (a == null || b == null) return a == null && b == null; + if (a.length != b.length) return false; + for (var i = 0; i < a.length; i++) { + if (!_same(a[i], b[i])) return false; + } + return true; +} + +bool _sameCvhr(Metric a, Metric b) { + final x = a.value, y = b.value; + return _same(a.confidence, b.confidence) && + ((x == null || y == null) + ? x == null && y == null + : x.cycleCount == y.cycleCount && + _same(x.cvhrPerHour, y.cvhrPerHour) && + _same(x.analyzedHours, y.analyzedHours) && + _same(x.meanDepthMs, y.meanDepthMs) && + _same(x.meanWidthSec, y.meanWidthSec) && + _sameList(x.depthQuartilesMs, y.depthQuartilesMs) && + _sameList(x.widthQuartilesSec, y.widthQuartilesSec)); +} + +void _cvhrStreamBench() { + // 32 × 15 min, about eight hours at 56 bpm. Each pass makes the newly + // settled prefix permanent and compares its snapshot with the batch scorer. + final night = frozen.genBeats(9019, 33000, period: 55, amp: 60); + final stream = CvhrStream(); + var from = 0, streamTotal = 0, batchTotal = 0; + var parity = true; + final start = night.tMs.first; + print('CVHR stream: pass stream us batch us parity'); + for (var pass = 1; pass <= 32; pass++) { + final until = start + pass * 15 * 60 * 1000.0; + var end = from; + while (end < night.nn.length && night.tMs[end] <= until) { + end++; + } + if (end == from) continue; + stream.fold(night.nn.sublist(from, end), night.tMs.sublist(from, end)); + from = end; + final streamWatch = Stopwatch()..start(); + final streamed = stream.snapshot(artifactFraction: 0.02); + streamWatch.stop(); + final batchWatch = Stopwatch()..start(); + final batch = cvhrApneaScreen(night.nn.sublist(0, end), + night.tMs.sublist(0, end), artifactFraction: 0.02); + batchWatch.stop(); + final streamUs = streamWatch.elapsedMicroseconds; + final batchUs = batchWatch.elapsedMicroseconds; + streamTotal += streamUs; + batchTotal += batchUs; + final same = _sameCvhr(streamed, batch); + parity = parity && same; + print('cvhr pass ${pass.toString().padLeft(2)} ' + '${streamUs.toString().padLeft(9)} ${batchUs.toString().padLeft(9)} ' + 'parity=$same'); + } + print('cvhr total ${streamTotal.toString().padLeft(9)} ' + '${batchTotal.toString().padLeft(9)}'); + print('cvhr stream parity: $parity'); + if (!parity) throw StateError('CVHR stream parity failed'); +} + +bool _sameHrv(Metric a, Metric b) { + final x = a.value, y = b.value; + return _same(a.confidence, b.confidence) && + ((x == null || y == null) + ? x == null && y == null + : _same(x.ulf, y.ulf) && + _same(x.vlf, y.vlf) && + _same(x.lf, y.lf) && + _same(x.hf, y.hf) && + _same(x.total, y.total) && + _same(x.lfhf, y.lfhf) && + _same(x.nuLf, y.nuLf) && + _same(x.nuHf, y.nuHf) && + x.hfGated == y.hfGated && + x.totalBands.toString() == y.totalBands.toString()); +} + +bool _sameStager(CardioStagerResult a, CardioStagerResult b) => + a.base.stages.toString() == b.base.stages.toString() && + a.base.epochSec == b.base.epochSec && + _same(a.base.wakePct, b.base.wakePct) && + _same(a.base.nremPct, b.base.nremPct) && + _same(a.base.remPct, b.base.remPct) && + a.deepFlag.toString() == b.deepFlag.toString() && + _same(a.confidence, b.confidence); + +bool _sameRsa(Metric a, Metric b) { + final x = a.value, y = b.value; + return _same(a.confidence, b.confidence) && + a.tier == b.tier && + a.inputs_used.toString() == b.inputs_used.toString() && + a.note == b.note && + ((x == null || y == null) + ? x == null && y == null + : _same(x.brpm, y.brpm) && + _same(x.peakHz, y.peakHz) && + _same(x.power, y.power) && + x.source == y.source); +} + +void _welchStreamBench() { + // 32 × 15 min, about eight hours at 56 bpm. The tail is provisional: it is + // included in each snapshot and batch call, then folded in a later pass. + final night = frozen.genBeats(9018, 33000, period: 53, amp: 50); + final hrv = HrvFreqStream(); + final rsa = RsaRespStream(); + var from = 0; + var hrvStreamTotal = 0, hrvBatchTotal = 0; + var rsaStreamTotal = 0, rsaBatchTotal = 0; + var parity = true; + final start = night.tMs.first; + print('Welch streams: pass stream hrv/rsa us batch hrv/rsa us parity'); + for (var pass = 1; pass <= 32; pass++) { + final until = start + pass * 15 * 60 * 1000.0; + var end = from; + while (end < night.nn.length && night.tMs[end] <= until) { + end++; + } + if (end == from) continue; + hrv.fold(night.nn.sublist(from, end), night.tMs.sublist(from, end)); + rsa.fold(night.nn.sublist(from, end), night.tMs.sublist(from, end)); + from = end; + final tailEnd = (end + 100).clamp(0, night.nn.length); + final tailNn = night.nn.sublist(end, tailEnd); + final tailTimes = night.tMs.sublist(end, tailEnd); + final hrvStreamWatch = Stopwatch()..start(); + final streamHrv = hrv.snapshot( + tailNnMs: tailNn, tailNnTimesMs: tailTimes, artifactFraction: 0.02); + hrvStreamWatch.stop(); + final rsaStreamWatch = Stopwatch()..start(); + final streamRsa = rsa.snapshot( + tailNnMs: tailNn, tailNnTimesMs: tailTimes, artifactFraction: 0.02); + rsaStreamWatch.stop(); + final hrvBatchWatch = Stopwatch()..start(); + final batchHrv = hrvFreq(night.nn.sublist(0, tailEnd), + night.tMs.sublist(0, tailEnd), artifactFraction: 0.02); + hrvBatchWatch.stop(); + final rsaBatchWatch = Stopwatch()..start(); + final batchRsa = rsaRespRate(night.nn.sublist(0, tailEnd), + night.tMs.sublist(0, tailEnd), artifactFraction: 0.02); + rsaBatchWatch.stop(); + final hrvStreamUs = hrvStreamWatch.elapsedMicroseconds; + final rsaStreamUs = rsaStreamWatch.elapsedMicroseconds; + final hrvBatchUs = hrvBatchWatch.elapsedMicroseconds; + final rsaBatchUs = rsaBatchWatch.elapsedMicroseconds; + hrvStreamTotal += hrvStreamUs; + rsaStreamTotal += rsaStreamUs; + hrvBatchTotal += hrvBatchUs; + rsaBatchTotal += rsaBatchUs; + final same = _sameHrv(streamHrv, batchHrv) && _sameRsa(streamRsa, batchRsa); + parity = parity && same; + print('welch pass ${pass.toString().padLeft(2)} ' + '${hrvStreamUs.toString().padLeft(8)}/${rsaStreamUs.toString().padLeft(8)} ' + '${hrvBatchUs.toString().padLeft(8)}/${rsaBatchUs.toString().padLeft(8)} ' + 'parity=$same'); + } + print('welch total ' + '${hrvStreamTotal.toString().padLeft(8)}/${rsaStreamTotal.toString().padLeft(8)} ' + '${hrvBatchTotal.toString().padLeft(8)}/${rsaBatchTotal.toString().padLeft(8)}'); + print('welch stream parity: $parity'); + if (!parity) throw StateError('Welch stream parity failed'); +} + +void main(List args) { + // 8 h / 30k beats / one 8 h night, respectively. Fixed seeds make both the + // timings and the parity gate reproducible. + final cvhr = frozen.genBeats(9001, 32000, period: 55, amp: 60); + final hrv = frozen.genBeats(9002, 30000, period: 53, amp: 50); + final night = frozen.genNight(9003, 8 * 60 * 60, rrOutliers: true); + + final cvhrBefore = () => frozen.refCvhrApneaScreen(cvhr.nn, cvhr.tMs, + artifactFraction: 0.02); + final cvhrAfter = () => + cvhrApneaScreen(cvhr.nn, cvhr.tMs, artifactFraction: 0.02); + final hrvBefore = () => + frozen.refHrvFreq(hrv.nn, hrv.tMs, artifactFraction: 0.02); + final hrvAfter = () => hrvFreq(hrv.nn, hrv.tMs, artifactFraction: 0.02); + final stagerBefore = () => frozen.refCardioStager(night.hr, night.accel, + rrMs: night.rr, rrTsMs: night.rrTs); + final stagerAfter = () => cardioStager(night.hr, night.accel, + rrMs: night.rr, rrTsMs: night.rrTs); + + final selected = args.isEmpty ? {'cvhr', 'hrv', 'cardio'} : args.toSet(); + print('case before us after us'); + var parity = true; + void row(String key, String name, void Function() before, void Function() after, + bool same) { + if (!selected.contains(key)) return; + parity = parity && same; + final beforeUs = _medianUs(before); + final afterUs = _medianUs(after); + print('${name.padRight(28)} ${beforeUs.toString().padLeft(9)} ' + '${afterUs.toString().padLeft(9)} parity=$same'); + } + row('cvhr', 'cvhrApneaScreen (8 h)', cvhrBefore, cvhrAfter, + _sameCvhr(cvhrBefore(), cvhrAfter())); + row('hrv', 'hrvFreq (30k beats)', hrvBefore, hrvAfter, + _sameHrv(hrvBefore(), hrvAfter())); + row('cardio', 'cardioStager (one night)', stagerBefore, stagerAfter, + _sameStager(stagerBefore(), stagerAfter())); + if (args.isEmpty || selected.contains('cvhr-stream')) _cvhrStreamBench(); + if (selected.contains('cardio') || selected.contains('cache')) { + final extra = frozen.genNight(9004, 15 * 60, rrOutliers: true); + const nightMs = 8 * 60 * 60 * 1000.0; + final appendedAccel = [ + ...night.accel, + for (final a in extra.accel) + AccelSample(a.tsMs + nightMs, a.x, a.y, a.z, valid: a.valid), + ]; + final appendedHr = [...night.hr, ...extra.hr]; + final appendedRr = [...night.rr, ...extra.rr]; + final appendedRrTs = [ + ...night.rrTs, + for (final ts in extra.rrTs) ts + nightMs, + ]; + CardioRowCache? cache; + void warmNight() { + cache = CardioRowCache(); + cardioStager(night.hr, night.accel, + rrMs: night.rr, rrTsMs: night.rrTs, rowCache: cache); + } + final coldUs = _medianUs(() => cardioStager(night.hr, night.accel, + rrMs: night.rr, rrTsMs: night.rrTs, rowCache: CardioRowCache()), + runs: 1); + final warmUs = _medianPreparedUs(warmNight, () => cardioStager( + night.hr, night.accel, + rrMs: night.rr, rrTsMs: night.rrTs, rowCache: cache), runs: 1); + final appendedUs = _medianPreparedUs(warmNight, () => cardioStager( + appendedHr, appendedAccel, + rrMs: appendedRr, rrTsMs: appendedRrTs, rowCache: cache), runs: 1); + print('cardio cache (8 h) cold us $coldUs'); + print('cardio cache (8 h) warm us $warmUs'); + print('cardio cache warm +15 min us $appendedUs'); + } + if (args.isEmpty || selected.contains('welch')) _welchStreamBench(); + print('parity: frozen references == production: $parity'); + if (!parity) { + throw StateError('frozen-reference parity failed'); + } +} diff --git a/tool/causal_stager_validate.dart b/tool/causal_stager_validate.dart new file mode 100644 index 0000000..1fc0386 --- /dev/null +++ b/tool/causal_stager_validate.dart @@ -0,0 +1,461 @@ +// VALIDATION HARNESS — replay a recorded night through the CAUSAL stager and +// score it against the retrospective one. +// +// WHAT THIS CAN AND CANNOT SAY. The only recorded night in the repo +// (test/onehz/fixtures/real_night_2026_07_*.csv) has no epoch labels at all — +// just the Apple Watch's whole-night stage minutes, quoted in +// test/onehz/real_night_cardio_stager_test.dart. There is no PSG-labelled set +// in-tree (the DREAMT corpus behind `tool/stager_harness.dart` lives outside +// the repo). So everything below is AGREEMENT WITH THE OFFLINE STAGER, which +// is itself a kappa-0.13 wrist estimate against PSG. It measures how closely +// the online rules track the retrospective ones and how the "REM, wake now" +// trigger behaves against that reference — NOT stage accuracy. One night, one +// subject: read the numbers as a smoke test with a ruler, not as a result. +// +// Usage: +// dart run tool/causal_stager_validate.dart [onehz.csv rr.csv] [--step S] +// [--warmup N] +// defaults: the in-tree real night, 30 s steps, the shipped warm-up. +// +// Fixture columns: onehz.csv `rel_sec,hr,ax,ay,az`; rr.csv `rel_ms,rr_ms`. + +import 'dart:convert'; +import 'dart:io'; + +import 'package:openstrap_analytics/onehz.dart'; + +const _fixtures = 'test/onehz/fixtures/'; + +/// Shift the fixture's 0-based clock to a realistic, 30 s-aligned absolute one. +const double _t0Ms = 1700000010000; + +void main(List args) { + final pos = [for (final a in args) if (!a.startsWith('--')) a]; + int intArg(String n, int d) { + final i = args.indexOf(n); + return (i >= 0 && i + 1 < args.length) ? int.parse(args[i + 1]) : d; + } + + final stepSec = intArg('--step', 30); + final warm = intArg('--warmup', CausalStagerConfig.defaults.warmupEpochs); + final onehz = pos.isNotEmpty ? pos[0] : '${_fixtures}real_night_2026_07_onehz.csv'; + final rrPath = pos.length > 1 ? pos[1] : '${_fixtures}real_night_2026_07_rr.csv'; + if (!File(onehz).existsSync() || !File(rrPath).existsSync()) { + stderr.writeln('fixture not found: $onehz / $rrPath'); + exitCode = 66; + return; + } + final night = _load(onehz, rrPath); + stdout.writeln('night: ${night.hr.length} s ' + '(${(night.hr.length / 3600).toStringAsFixed(2)} h), ' + '${night.rrTs.length} RR beats'); + + // ── offline reference ────────────────────────────────────────────────────── + final off = cardioStager(night.hr, night.accel, + rrMs: night.rr, rrTsMs: night.rrTs); + final offStages = off.base.stages; + final nEp = offStages.length; + stdout.writeln('offline epochs: $nEp confidence ${off.confidence}'); + _minutes('offline cardioStager', offStages.map(_name).toList()); + stdout.writeln('Apple Watch reference (summary only): wake 3 REM 162 ' + 'light 330 deep 38 min'); + stdout.writeln(''); + + // ── causal replay ────────────────────────────────────────────────────────── + final cfg = CausalStagerConfig(warmupEpochs: warm); + final run = _replay(night, stepSec, cfg); + final again = _replay(night, stepSec, cfg); + final deterministic = _digest(run) == _digest(again); + stdout.writeln('causal replay: step ${stepSec}s, warm-up $warm epochs ' + '(${warm * 30 / 60} min), deterministic on replay: $deterministic'); + + // Latest observation per epoch (a finer step can describe an epoch twice). + final perEpoch = {}; + for (final o in run) { + if (o.stage != CausalStage.absent) { + perEpoch[(o.epochStartMs! / 30000).round()] = o; + } + } + final reasons = {}; + for (final o in run) { + if (o.stage == CausalStage.absent) { + reasons[o.abstentionReason!.name] = + (reasons[o.abstentionReason!.name] ?? 0) + 1; + } + } + final base = (_t0Ms / 30000).round(); + stdout.writeln('observations ${run.length}: staged ${run.length - reasons.values.fold(0, (a, b) => a + b)}, ' + 'abstained $reasons'); + final causal = List.filled(nEp, null); + for (final e in perEpoch.entries) { + final i = e.key - base; + if (i >= 0 && i < nEp) causal[i] = e.value.stage.name; + } + _minutes('causal (staged epochs only)', + [for (final c in causal) if (c != null) c]); + stdout.writeln(''); + + // ── agreement, by lag ────────────────────────────────────────────────────── + // causal(k) is built from TRAILING windows, so it describes the physiology a + // couple of minutes earlier. Compare causal(k) with offline(k - d). + stdout.writeln('AGREEMENT with offline cardioStager (staged epochs only)'); + stdout.writeln(' lag d: causal(k) vs offline(k-d) n agree kappa'); + var bestD = 0; + var bestK = -2.0; + for (final d in [0, 1, 2, 3, 4, 6, 8]) { + final a = [], b = []; + for (var k = d; k < nEp; k++) { + final c = causal[k]; + if (c == null) continue; + a.add(_idx(c)); + b.add(_idx(_name(offStages[k - d]))); + } + final kap = _kappa(a, b); + if (kap > bestK) { + bestK = kap; + bestD = d; + } + stdout.writeln(' d=$d (${d * 30}s)${' ' * (21 - '$d (${d * 30}s)'.length)}' + '${a.length.toString().padLeft(6)} ${_pct(_agree(a, b))} ${_f(kap)}'); + } + stdout.writeln(' best lag: d=$bestD'); + { + final a = [], b = []; + for (var k = 0; k < nEp; k++) { + final c = causal[k]; + if (c == null) continue; + a.add(_idx(c)); + b.add(_idx(_name(offStages[k]))); + } + stdout.writeln(''); + stdout.writeln('CONFUSION at d=0 (rows = offline, cols = causal)'); + _confusion(b, a); + } + + // ── the REM "wake now" use case ──────────────────────────────────────────── + _remTrigger(perEpoch, base, offStages, nEp, causal); + + // ── warm-up sensitivity ──────────────────────────────────────────────────── + stdout.writeln(''); + stdout.writeln('WARM-UP sensitivity (d=0) warm-up staged% agree kappa'); + for (final w in [20, 40, 60, 120]) { + final r = _replay(night, 30, CausalStagerConfig(warmupEpochs: w)); + final a = [], b = []; + var staged = 0; + for (final o in r) { + if (o.stage == CausalStage.absent) continue; + staged++; + final i = (o.epochStartMs! / 30000).round() - base; + if (i < 0 || i >= nEp) continue; + a.add(_idx(o.stage.name)); + b.add(_idx(_name(offStages[i]))); + } + stdout.writeln(' ${(w * 0.5).toStringAsFixed(0).padLeft(4)} min ' + '${_pct(staged / nEp)} ${_pct(_agree(a, b))} ${_f(_kappa(a, b))}'); + } +} + +// ── REM trigger analysis ─────────────────────────────────────────────────────── + +/// "Wake now" = REM that has persisted for X seconds (`runSec >= X`), the rule +/// a caller would build from the observation. 300 s is [remEpisodeMinMin], the +/// retrospective stager's shortest credible REM bout; the table below shows +/// what shorter thresholds trade. +void _remTrigger(Map perEpoch, int base, + List offStages, int nEp, List causal) { + bool offRem(int k) => k >= 0 && k < nEp && offStages[k] == SleepStage.rem; + // Offline REM within +-5 min (10 epochs): trailing windows keep the edges of a + // bout in view for a few minutes, so "near" is the fair no-false-trigger test. + bool nearRem(int k) { + for (var d = -10; d <= 10; d++) { + if (offRem(k + d)) return true; + } + return false; + } + + bool trigAt(int k, int x) { + final o = perEpoch[k + base]; + return o != null && o.stage == CausalStage.rem && o.runSec >= x; + } + + final bouts = <(int, int)>[]; + for (var i = 0; i < nEp;) { + if (!offRem(i)) { + i++; + continue; + } + var j = i; + while (j < nEp && offRem(j)) { + j++; + } + bouts.add((i, j)); + i = j; + } + + var nStaged = 0, nOffRem = 0, nNear = 0; + for (var k = 0; k < nEp; k++) { + if (causal[k] == null) continue; + nStaged++; + if (offRem(k)) nOffRem++; + if (nearRem(k)) nNear++; + } + stdout.writeln(''); + stdout.writeln('REM "WAKE NOW" TRIGGER reference = ${bouts.length} offline REM ' + 'bouts as the retrospective stager emits them (several are under 5 min)'); + stdout.writeln(' chance level for a trigger landing on offline REM: ' + '${_pct(nOffRem / nStaged)}; within +-5 min of it: ${_pct(nNear / nStaged)}' + ' <- the "no false trigger" test is only as strict as this'); + stdout.writeln(' rule: causal stage = rem AND run >= X s'); + stdout.writeln(' X s trigger epochs @offlineREM @+-5min falseEpochs ' + 'bouts caught (of ${bouts.length}) lateness min (median) ' + 'late-tail epochs'); + for (final x in [60, 120, 180, 300, 420]) { + final eps = [for (var k = 0; k < nEp; k++) if (trigAt(k, x)) k]; + final onRem = eps.where(offRem).length; + final near = eps.where(nearRem).length; + var caught = 0; + final late = []; + for (final (s, e) in bouts) { + for (var k = s; k < e; k++) { + if (trigAt(k, x)) { + caught++; + late.add((k - s) * 0.5); + break; + } + } + } + // Trigger epochs that are not REM but follow a bout within 5 min: the + // late tail, i.e. "woke you just after REM ended". + final tail = eps.where((k) { + if (offRem(k)) return false; + for (var d = 1; d <= 10; d++) { + if (offRem(k - d)) return true; + } + return false; + }).length; + String p(int v) => eps.isEmpty ? ' n/a' : _pct(v / eps.length); + stdout.writeln(' ${x.toString().padLeft(5)} ${eps.length.toString().padLeft(13)}' + ' ${p(onRem)} ${p(near)} ${(eps.length - near).toString().padLeft(11)}' + ' ${caught.toString().padLeft(16)}' + ' ${late.isEmpty ? "n/a".padLeft(20) : _median(late).toStringAsFixed(1).padLeft(20)}' + ' ${tail.toString().padLeft(16)}'); + } + + stdout.writeln(''); + stdout.writeln(' per offline bout at X=300 / X=120 (first trigger INSIDE the bout):'); + for (final (s, e) in bouts) { + String f(int x) { + for (var k = s; k < e; k++) { + if (trigAt(k, x)) return '+${((k - s) * 0.5).toStringAsFixed(1)} min'; + } + return 'none'; + } + + stdout.writeln(' ${_hm(s)}-${_hm(e)} (${((e - s) * 0.5).toStringAsFixed(1).padLeft(4)} min)' + ' X=300: ${f(300).padRight(9)} X=120: ${f(120)}'); + } + + // Natural-Wake-shaped windows: every must-be-up time T on the 30 s grid whose + // whole window [T-N, T) was observable; fire once, at the first trigger epoch. + for (final x in [120, 300]) { + stdout.writeln(''); + stdout.writeln('WINDOW SIMULATION, X=$x s (fire once at the first trigger ' + 'epoch in [T-N, T))'); + stdout.writeln(' N min windows fired @offlineREM @+-5min ' + 'fired w/o REM near missed(REM in window) quiet(no REM)'); + for (final nMin in [15, 30, 60, 120]) { + final nW = nMin * 2; + var windows = 0, fired = 0, onRem = 0, near = 0, falseFire = 0; + var missed = 0, quiet = 0; + for (var t = nW; t <= nEp; t++) { + var ok = true; + for (var k = t - nW; k < t; k++) { + if (causal[k] == null) { + ok = false; + break; + } + } + if (!ok) continue; + windows++; + int? f; + var remInWin = false; + for (var k = t - nW; k < t; k++) { + if (f == null && trigAt(k, x)) f = k; + if (offRem(k)) remInWin = true; + } + if (f != null) { + fired++; + if (offRem(f)) onRem++; + if (nearRem(f)) { + near++; + } else { + falseFire++; + } + } else if (remInWin) { + missed++; + } else { + quiet++; + } + } + String c(int v, [int w = 9]) => v.toString().padLeft(w); + stdout.writeln(' ${nMin.toString().padLeft(4)} ${c(windows)}${c(fired)}' + '${c(onRem, 13)}${c(near, 9)}${c(falseFire, 19)}${c(missed, 21)}${c(quiet, 15)}'); + } + } +} + +// ── replay + metrics ────────────────────────────────────────────────────────── + +List _replay( + _Night n, int stepSec, CausalStagerConfig cfg) { + final out = []; + CausalStagerState? st = CausalStagerState.initial(cfg); + final secs = n.hr.length; + var rrI = 0; + for (var t = stepSec; t <= secs; t += stepSec) { + final lo = t - stepSec; + final loMs = _t0Ms + lo * 1000.0, hiMs = _t0Ms + t * 1000.0; + final rrTs = [], rrV = []; + while (rrI < n.rrTs.length && _t0Ms + n.rrTs[rrI] < hiMs) { + if (_t0Ms + n.rrTs[rrI] >= loMs) { + rrTs.add(_t0Ms + n.rrTs[rrI]); + rrV.add(n.rr[rrI]); + } + rrI++; + } + final o = CausalStager.observe( + CausalSampleWindow( + nowMs: hiMs, + hr: [ + for (var i = lo; i < t; i++) HrSample(_t0Ms + i * 1000.0, n.hr[i]) + ], + accel: [ + for (var i = lo; i < t; i++) + AccelSample(_t0Ms + i * 1000.0, n.accel[i].x, n.accel[i].y, + n.accel[i].z) + ], + rr: RrSeries(rrTs, rrV), + ), + st); + out.add(o); + st = o.nextState; + } + return out; +} + +String _digest(List r) { + var h = 0; + for (final o in r) { + for (final c in jsonEncode(o.toJson()).codeUnits) { + h = (h * 31 + c) & 0x7fffffff; + } + } + return '$h'; +} + +class _Night { + final List hr; + final List accel; + final List rr, rrTs; + _Night(this.hr, this.accel, this.rr, this.rrTs); +} + +_Night _load(String onehz, String rrPath) { + final hr = []; + final accel = []; + final l1 = File(onehz).readAsLinesSync(); + for (var i = 1; i < l1.length; i++) { + if (l1[i].trim().isEmpty) continue; + final p = l1[i].split(','); + hr.add(double.parse(p[1])); + accel.add(AccelSample(double.parse(p[0]) * 1000.0, double.parse(p[2]), + double.parse(p[3]), double.parse(p[4]))); + } + final rr = [], rrTs = []; + final l2 = File(rrPath).readAsLinesSync(); + for (var i = 1; i < l2.length; i++) { + if (l2[i].trim().isEmpty) continue; + final p = l2[i].split(','); + rrTs.add(double.parse(p[0])); + rr.add(double.parse(p[1])); + } + return _Night(hr, accel, rr, rrTs); +} + +String _name(SleepStage s) => s.name; +int _idx(String s) => const {'wake': 0, 'nrem': 1, 'rem': 2}[s]!; +String _pct(double v) => '${(100 * v).toStringAsFixed(1).padLeft(5)}%'; +String _f(double v) => v.isNaN ? ' n/a' : v.toStringAsFixed(3).padLeft(6); +String _hm(int epoch) { + final m = epoch ~/ 2; + return '${m ~/ 60}h${(m % 60).toString().padLeft(2, '0')}'; +} + +double _median(List v) { + final s = [...v]..sort(); + return s[s.length ~/ 2]; +} + +void _minutes(String label, List stages) { + final c = {'wake': 0, 'nrem': 0, 'rem': 0}; + for (final s in stages) { + c[s] = c[s]! + 1; + } + stdout.writeln('$label: wake ${c['wake']! / 2} min nrem ${c['nrem']! / 2} min ' + 'rem ${c['rem']! / 2} min (${stages.length / 2} min)'); +} + +double _agree(List a, List b) { + if (a.isEmpty) return double.nan; + var m = 0; + for (var i = 0; i < a.length; i++) { + if (a[i] == b[i]) m++; + } + return m / a.length; +} + +double _kappa(List a, List b) { + final n = a.length; + if (n == 0) return double.nan; + final m = List.generate(3, (_) => List.filled(3, 0)); + for (var i = 0; i < n; i++) { + m[a[i]][b[i]]++; + } + var po = 0.0, pe = 0.0; + for (var i = 0; i < 3; i++) { + po += m[i][i]; + final r = m[i].reduce((x, y) => x + y); + var c = 0; + for (var j = 0; j < 3; j++) { + c += m[j][i]; + } + pe += r * c / n; + } + po /= n; + pe /= n; + return pe >= 1 ? double.nan : (po - pe) / (1 - pe); +} + +/// rows = `ref`, cols = `got`. +void _confusion(List ref, List got) { + const names = ['wake', 'nrem', 'rem']; + final m = List.generate(3, (_) => List.filled(3, 0)); + for (var i = 0; i < ref.length; i++) { + m[ref[i]][got[i]]++; + } + stdout.writeln(' ${names.map((s) => s.padLeft(7)).join()} recall'); + for (var i = 0; i < 3; i++) { + final tot = m[i].reduce((a, b) => a + b); + stdout.writeln(' ${names[i].padRight(9)} ${m[i].map((v) => v.toString().padLeft(7)).join()}' + ' ${tot == 0 ? " n/a" : _pct(m[i][i] / tot)}'); + } + final prec = []; + for (var j = 0; j < 3; j++) { + var col = 0; + for (var i = 0; i < 3; i++) { + col += m[i][j]; + } + prec.add(col == 0 ? ' n/a' : _pct(m[j][j] / col).padLeft(7)); + } + stdout.writeln(' precision ${prec.join()}'); +} diff --git a/tool/combined_bump_probe.dart b/tool/combined_bump_probe.dart new file mode 100644 index 0000000..6a526e8 --- /dev/null +++ b/tool/combined_bump_probe.dart @@ -0,0 +1,189 @@ +import 'dart:io'; +import 'dart:convert'; +import 'package:openstrap_analytics/onehz.dart'; + +void main(List args) { + const base = 1700000013; + final accel = [], hr = []; + final rr = [], ts = []; + for (final l in File('test/onehz/fixtures/real_night_2026_07_onehz.csv') + .readAsLinesSync() + .skip(1)) { + final p = l.split(','); + if (p.length < 5) continue; + final t = (base + int.parse(p[0])) * 1000.0; + hr.add(HrSample(t, double.parse(p[1]))); + accel.add(AccelSample( + t, double.parse(p[2]), double.parse(p[3]), double.parse(p[4]))); + } + for (final l in File('test/onehz/fixtures/real_night_2026_07_rr.csv') + .readAsLinesSync() + .skip(1)) { + final p = l.split(','); + if (p.length < 2) continue; + rr.add(double.parse(p[1])); + ts.add(base * 1000.0 + double.parse(p[0])); + } + if (args.contains('rr-bench')) { + for (var i = 0; i < 8; i++) { + correctRr(rr, rrTsMs: ts); + correctRr(rr, rrTsMs: ts, maxArtifactRun: null); + } + print('correctRr capped: ${medianUs(() => correctRr(rr, rrTsMs: ts))} us'); + print( + 'correctRr uncapped: ${medianUs(() => correctRr(rr, rrTsMs: ts, maxArtifactRun: null))} us'); + return; + } + if (args.contains('bench')) { + benchmark(hr, accel, rr, ts); + return; + } + final old = <(int, int, String)>[]; + for (final l + in File('test/onehz/fixtures/real_night_2026_07_hypnogram_v1.csv') + .readAsLinesSync() + .skip(1)) { + final p = l.split(','); + if (p.length < 3) continue; + old.add((base + int.parse(p[0]), base + int.parse(p[1]), p[2])); + } + void compare(String name, List segments) { + var same = 0, count = 0; + final confusion = {}; + for (var t = (base + 29) ~/ 30 * 30 + 15; + t + 15 <= base + hr.length; + t += 30) { + final a = old.where((s) => t >= s.$1 && t < s.$2).first.$3; + final b = segments.where((s) => t >= s.start && t < s.end).first.stage; + if (a == 'unobserved' || b == 'unobserved') continue; + count++; + if (a == b) same++; + if (a != b) confusion.update('$a->$b', (v) => v + 1, ifAbsent: () => 1); + } + print('$name agreement=$same/$count=${same / count} $confusion'); + } + + final er = EpochRows(); + final sw = Stopwatch()..start(); + er.fold(hr: hr, accel: accel, rrMs: rr, rrTsMs: ts); + final rows = er.allRows(); + sw.stop(); + print('EpochRows full ${sw.elapsedMicroseconds} us, ${rows.length} rows'); + compare('EpochRows', er.stageWindow(base, base + hr.length)); + for (final lowHz in [1 / 180, 0.04]) { + final lfhf = []; + for (final row in rows) { + final b = cleanRrBeatsBetween( + rr, ts, row.startMs - 75000.0, row.startMs + 105000.0); + final ls = b.beats.length < 16 + ? null + : lombScargle(b.tsSec, b.beats, freqGrid(lowHz, 0.4, 240)); + final hf = ls?.bandPower(0.15, 0.4) ?? 0; + lfhf.add(hf > 0 ? ls!.bandPower(0.04, 0.15) / hf : double.nan); + } + final result = classifyCardioEpochs(CardioEpochFeatures( + motion: [for (final r in rows) r.motion], + hr: [for (final r in rows) r.hr], + hrSd: [for (final r in rows) r.hrSd], + rmssd: [for (final r in rows) r.rmssd], + lfhf: lfhf, + rk: [for (final r in rows) r.rk], + sdnn: [for (final r in rows) r.sdnn])); + compare('fixed grid $lowHz', [ + for (var i = 0; i < result.base.stages.length; i++) + StageSegment( + i == 0 ? base : rows[i].startMs ~/ 1000, + i == result.base.stages.length - 1 + ? base + hr.length + : rows[i + 1].startMs ~/ 1000, + result.base.stages[i] == SleepStage.wake + ? 'wake' + : result.base.stages[i] == SleepStage.rem + ? 'rem' + : result.deepFlag[i] + ? 'deep' + : 'light') + ]); + } + for (final skip in [0, 27]) { + final s = Stopwatch()..start(); + final result = cardioStager( + [for (final h in hr.skip(skip)) h.hr], accel.sublist(skip), + rrMs: rr, rrTsMs: ts); + s.stop(); + print('cardioStager skip=$skip ${s.elapsedMicroseconds} us'); + compare('direct skip=$skip', [ + for (var i = 0; i < result.base.stages.length; i++) + StageSegment( + base + skip + i * 30, + i == result.base.stages.length - 1 + ? base + hr.length + : base + skip + (i + 1) * 30, + result.base.stages[i] == SleepStage.wake + ? 'wake' + : result.base.stages[i] == SleepStage.rem + ? 'rem' + : result.deepFlag[i] + ? 'deep' + : 'light') + ]); + } +} + +int medianUs(void Function() run) { + run(); + final samples = []; + for (var i = 0; i < 3; i++) { + final sw = Stopwatch()..start(); + run(); + samples.add(sw.elapsedMicroseconds); + } + samples.sort(); + return samples[1]; +} + +void benchmark(List hr, List accel, List rr, + List ts) { + final values = [for (final h in hr) h.hr]; + print('Real night: ${hr.length} seconds, ${rr.length} beats'); + print('EpochRows fold + allRows: ${medianUs(() { + final er = EpochRows(); + er.fold(hr: hr, accel: accel, rrMs: rr, rrTsMs: ts); + er.allRows(); + })} us'); + print( + 'cardioEpochFeatures: ${medianUs(() => cardioEpochFeatures(values, accel, rrMs: rr, rrTsMs: ts))} us'); + print('correctRr capped: ${medianUs(() => correctRr(rr, rrTsMs: ts))} us'); + print( + 'correctRr uncapped: ${medianUs(() => correctRr(rr, rrTsMs: ts, maxArtifactRun: null))} us'); + final er = EpochRows(); + var rb = 0, streamed = 0, batch = 0, checkpointChars = 0; + final t0 = hr.first.tsMs; + for (var begin = 0; begin < hr.length; begin += 900) { + final end = (begin + 900).clamp(0, hr.length); + final rt = t0 + end * 1000; + var re = rb; + while (re < ts.length && ts[re] < rt) { + re++; + } + final watch = Stopwatch()..start(); + er.fold( + hr: hr.sublist(begin, end), + accel: accel.sublist(begin, end), + rrMs: rr.sublist(rb, re), + rrTsMs: ts.sublist(rb, re)); + er.allRows(); + streamed += watch.elapsedMicroseconds; + checkpointChars = jsonEncode(er.toJson()).length; + watch.reset(); + cardioEpochFeatures(values.sublist(0, end), accel.sublist(0, end), + rrMs: rr.sublist(0, re), rrTsMs: ts.sublist(0, re)); + batch += watch.elapsedMicroseconds; + rb = re; + } + print( + '15-minute passes total: EpochRows $streamed us, batch features $batch us'); + print('EpochRows final checkpoint: $checkpointChars chars'); + print( + 'RrSandwich benchmark unavailable: generic clock translation is unresolved'); +} diff --git a/tool/incremental/bench_all.dart b/tool/incremental/bench_all.dart new file mode 100644 index 0000000..9c26413 --- /dev/null +++ b/tool/incremental/bench_all.dart @@ -0,0 +1,283 @@ +// Timing: whole-series oracle vs incremental (fold + evaluate) per 15-min pass. +// dart run tool/incremental/bench_all.dart +import 'dart:convert'; +import 'package:openstrap_analytics/onehz.dart' + hide RrCorrector, RrSettled, RrSnapshot, IrregularScreenState; +import 'driver.dart'; +import 'edge_oracles.dart'; +import 'hrv_incr.dart'; +import 'oracle_util.dart'; +import 'resp_incr.dart'; +import 'synth.dart'; + +double ms(Stopwatch s) => s.elapsedMicroseconds / 1000.0; +double sum(List x) => x.fold(0.0, (a, b) => a + b); +double mean(List x) => x.isEmpty ? 0 : sum(x) / x.length; +double mx(List x) => x.fold(0.0, (a, b) => a > b ? a : b); + +class Row { + final String name; + final double oracleFullMs; + final List passMs; // fold+evaluate per pass + final int stateBytes; + Row(this.name, this.oracleFullMs, this.passMs, this.stateBytes); + String line() => + '| $name | ${oracleFullMs.toStringAsFixed(0)} | ${mean(passMs).toStringAsFixed(1)} | ${mx(passMs).toStringAsFixed(1)} | ${sum(passMs).toStringAsFixed(0)} | $stateBytes |'; +} + +T timed(Stopwatch sw, T Function() f, List sink) { + sw..reset()..start(); + final r = f(); + sink.add(ms(sw)); + return r; +} + +void main() { + final night = realNightRr()!; + final cuts = timeCuts(night.ts, 900); + final steps = []; + driveRr(night, cuts, steps.add); + final o = correctRr(night.rr, rrTsMs: night.ts); + final af = (1.0 - o.cleanFraction).clamp(0.0, 1.0).toDouble(); + print('night: ${night.length} beats, ${steps.length} folds, ' + 'NN=${o.nn.length}, artifactFraction=${af.toStringAsFixed(4)}'); + final sw = Stopwatch(); + final rows = []; + + Row bench(String name, double Function() oracle, void Function(FoldStep, List) pass, int Function() bytes) { + oracle(); // warm + sw..reset()..start(); + oracle(); + final fullMs = ms(sw); + final ps = []; + for (final f in steps) { + pass(f, ps); + } + final r = Row(name, fullMs, ps, bytes()); + rows.add(r); + return r; + } + + // hrvTime + { + var acc = HrvTimeAcc(); + bench('hrvTime scalars (night)', () { + sw..reset()..start(); + hrvTime(o.nn, nnTimesMs: o.nnTimesMs, artifactFraction: af); + return ms(sw); + }, (f, ps) { + timed(sw, () { + acc.fold(f.settled.nn, f.settled.nnTimes); + acc.evaluate(f.snap.tailNn, f.snap.tailNnTimes, artifactFraction: f.artifactFraction); + }, ps); + }, () => jsonEncode(acc.toJson()).length); + } + // nocturnal rmssd + { + final st = NocturnalRmssdState(); + bench('nocturnalRmssd', () { + sw..reset()..start(); + nocturnalRmssd(o.nn, o.nnTimesMs); + return ms(sw); + }, (f, ps) { + timed(sw, () { + st.fold(f.settled.nn, f.settled.nnTimes); + st.evaluate(f.snap.tailNn, f.snap.tailNnTimes); + }, ps); + }, () => st.recs.length * 60); + } + // session rmssd (raw) + { + final start = (night.ts.first / 1000).floor() + 1234; + final end = (night.ts.last / 1000).floor() - 987; + final st = SessionRmssdState(start, end); + var from = 0; + var k = 0; + bench('sleepSessionWindowedRmssd (raw RR)', () { + sw..reset()..start(); + sleepSessionWindowedRmssd(night.rr, night.ts, startSec: start, endSec: end); + return ms(sw); + }, (f, ps) { + timed(sw, () { + st.fold(night.rr.sublist(from, cuts[k]), night.ts.sublist(from, cuts[k])); + from = cuts[k]; + k++; + st.evaluate(); + }, ps); + }, () => st.recs.length * 60); + } + // night shape + { + final st = NightShapeState(); + bench('nightHrvShape', () { + sw..reset()..start(); + nightHrvShape(o.nn, o.nnTimesMs); + return ms(sw); + }, (f, ps) { + timed(sw, () { + st.fold(f.settled.nn, f.settled.nnTimes); + st.evaluate(f.snap.tailNn, f.snap.tailNnTimes); + }, ps); + }, () => st.closed.length * 40); + } + // hrv timeline + { + final origin = night.ts.first - night.rr.first; + final st = HrvTimelineState(origin); + bench('_hrvTimeline (night)', () { + sw..reset()..start(); + oracleHrvTimeline(o.nn, o.nnTimesMs, origin); + return ms(sw); + }, (f, ps) { + timed(sw, () { + st.fold(f.settled.nn, f.settled.nnTimes); + st.curve(f.snap.tailNn, f.snap.tailNnTimes); + }, ps); + }, () => st.out.length * 20 + 900 * 16); + } + // rsa + { + var st = RsaWelchState(); + bench('rsaRespRate (night)', () { + sw..reset()..start(); + rsaRespRate(o.nn, o.nnTimesMs, artifactFraction: af); + return ms(sw); + }, (f, ps) { + timed(sw, () { + st.fold(f.settled.nn, f.settled.nnTimes); + st.evaluate(f.snap.tailNn, f.snap.tailNnTimes, artifactFraction: f.artifactFraction); + }, ps); + }, () => jsonEncode(st.toJson()).length); + } + // resp windows + { + final st = RespWindowsState(); + bench('_respPerWindow (30-min bins)', () { + sw..reset()..start(); + oracleRespPerWindow(o.nn, o.nnTimesMs); + return ms(sw); + }, (f, ps) { + timed(sw, () { + st.fold(f.settled.nn, f.settled.nnTimes); + st.evaluate(f.snap.tailNn, f.snap.tailNnTimes); + }, ps); + }, () => st.closed.length * 8 + 12000); + } + // hrvFreq + { + var st = HrvFreqState(); + bench('hrvFreq LF/HF (night)', () { + sw..reset()..start(); + hrvFreq(o.nn, o.nnTimesMs, artifactFraction: af); + return ms(sw); + }, (f, ps) { + timed(sw, () { + st.fold(f.settled.nn, f.settled.nnTimes); + st.evaluate(f.snap.tailNn, f.snap.tailNnTimes, artifactFraction: f.artifactFraction); + }, ps); + }, () => jsonEncode(st.toJson()).length); + } + print('\nNIGHT estimators (8.9 h, 32k beats, 15-min passes)'); + print('| estimator | oracle whole-night ms | incr mean ms/pass | max | sum over passes | state bytes (json) |'); + print('|---|---|---|---|---|---|'); + rows.forEach((r) => print(r.line())); + + // ---------------- day-long ---------------- + final day = realShapedDay(); + final dcuts = timeCuts(day.ts, 900); + final dsteps = []; + driveRr(day, dcuts, dsteps.add); + final from = (day.ts.first / 1000).floor() - 2, to = (day.ts.last / 1000).ceil() + 3; + final acc = synthAccel(5, from, to); + final dsub = DaySub(day.rr, day.ts, acc.tsSec, acc.ax, acc.ay, acc.az); + final drows = []; + print('\nday: ${day.length} beats, ${dsteps.length} folds'); + // irregular + { + final o2 = correctRr(day.rr, rrTsMs: day.ts); + final af2 = (1.0 - o2.cleanFraction).clamp(0.0, 1.0).toDouble(); + irregularBeatScreen(o2.nn, nnTimesMs: o2.nnTimesMs, artifactFraction: af2); + sw..reset()..start(); + irregularBeatScreen(o2.nn, nnTimesMs: o2.nnTimesMs, artifactFraction: af2); + final full = ms(sw); + var st = IrregularScreenState(); + final ps = []; + for (final f in dsteps) { + timed(sw, () { + st.fold(f.settled.nn, f.settled.nnTimes); + st.evaluate(f.snap.tailNn, f.snap.tailNnTimes, artifactFraction: f.artifactFraction); + }, ps); + } + drows.add(Row('irregularBeatScreen (day NN, excl. correctRr)', full, ps, jsonEncode(st.toJson()).length)); + } + // day hrv curve + { + oracleDayHrvCurve(dsub); + sw..reset()..start(); + oracleDayHrvCurve(dsub); + final full = ms(sw); + final st = DayHrvCurveState(); + final ps = []; + var a = 0; + for (final cut in dcuts) { + timed(sw, () { + st.fold(day.rr.sublist(a, cut), day.ts.sublist(a, cut)); + }, ps); + a = cut; + } + drows.add(Row('dayHrvCurve', full, ps, jsonEncode(st.toJson()).length)); + } + // daytime hrv + { + oracleDaytimeHrv(dsub, 1700000000, 1700032040); + sw..reset()..start(); + oracleDaytimeHrv(dsub, 1700000000, 1700032040); + final full = ms(sw); + final st = DaytimeHrvState(1700000000, 1700032040); + final ps = []; + var ra = 0, aa = 0; + for (final cut in dcuts) { + final T = (day.ts[cut - 1] / 1000).floor() + 1; + var at = aa; + while (at < acc.length && acc.tsSec[at] < T) { + at++; + } + timed(sw, () { + st.fold(day.rr.sublist(ra, cut), day.ts.sublist(ra, cut), acc.tsSec.sublist(aa, at), + acc.ax.sublist(aa, at), acc.ay.sublist(aa, at), acc.az.sublist(aa, at), T); + st.result(); + }, ps); + ra = cut; + aa = at; + } + drows.add(Row('_daytimeHrv', full, ps, jsonEncode(st.toJson()).length)); + } + // day resp curve + { + oracleDayRespCurve(dsub); + sw..reset()..start(); + oracleDayRespCurve(dsub); + final full = ms(sw); + final st = DayRespCurveState(); + final ps = []; + var ra = 0, aa = 0; + for (final cut in dcuts) { + final T = (day.ts[cut - 1] / 1000).floor() + 1; + var at = aa; + while (at < acc.length && acc.tsSec[at] < T) { + at++; + } + timed(sw, () { + st.fold(day.rr.sublist(ra, cut), day.ts.sublist(ra, cut), acc.tsSec.sublist(aa, at), + acc.ax.sublist(aa, at), acc.ay.sublist(aa, at), acc.az.sublist(aa, at), T); + }, ps); + ra = cut; + aa = at; + } + drows.add(Row('dayRespCurve', full, ps, jsonEncode(st.toJson()).length)); + } + print('\nDAY estimators (23 h, 97k beats, 15-min passes)'); + print('| estimator | oracle whole-day ms | incr mean ms/pass | max | sum over passes | state bytes (json) |'); + print('|---|---|---|---|---|---|'); + drows.forEach((r) => print(r.line())); +} diff --git a/tool/incremental/bench_rr.dart b/tool/incremental/bench_rr.dart new file mode 100644 index 0000000..61ff942 --- /dev/null +++ b/tool/incremental/bench_rr.dart @@ -0,0 +1,83 @@ +// Timing: oracle correctRr vs streaming RrCorrector on a 23 h real-shaped day. +// dart run tool/incremental/bench_rr.dart +import 'dart:convert'; +import 'package:openstrap_analytics/onehz.dart' + hide RrCorrector, RrSettled, RrSnapshot, IrregularScreenState; +import 'oracle_util.dart'; +import 'rr_stream.dart'; +import 'synth.dart'; + +double ms(Stopwatch s) => s.elapsedMicroseconds / 1000.0; + +void main() { + final day = realShapedDay(); + final n = day.length; + print('day beats=$n span h=${(day.ts.last - day.ts.first) / 3.6e6}'); + // warm + correctRr(day.rr.sublist(0, 4000), rrTsMs: day.ts.sublist(0, 4000)); + + final sw = Stopwatch()..start(); + final oracle = correctRr(day.rr, rrTsMs: day.ts); + print('oracle one-shot: ${ms(sw).toStringAsFixed(0)} ms ' + '(${(ms(sw) * 1000 / n).toStringAsFixed(1)} us/beat)'); + + // streaming, one fold of the whole day + sw..reset()..start(); + final one = RrCorrector(); + final s1 = one.fold(day.rr, tsMs: day.ts); + final snap1 = one.snapshot(); + print('stream one-shot fold+snapshot: ${ms(sw).toStringAsFixed(0)} ms ' + '(${(ms(sw) * 1000 / n).toStringAsFixed(1)} us/beat)'); + final same = [...s1.nn, ...snap1.tailNn]; + print(' identical nn: ${same.length == oracle.nn.length && _eq(same, oracle.nn)}'); + + // streaming, 15-minute folds, JSON restart every fold (worst case) + final cuts = timeCuts(day.ts, 900); + var c = RrCorrector(); + final foldMs = [], snapMs = [], jsonMs = []; + var bytes = 0, maxBytes = 0; + var from = 0; + final all = []; + for (final to in cuts) { + sw..reset()..start(); + final s = c.fold(day.rr.sublist(from, to), tsMs: day.ts.sublist(from, to)); + foldMs.add(ms(sw)); + all.addAll(s.nn); + sw..reset()..start(); + final snap = c.snapshot(); + snapMs.add(ms(sw)); + if (to == cuts.last) all.addAll(snap.tailNn); + sw..reset()..start(); + final txt = jsonEncode(c.toJson()); + c = RrCorrector.fromJson(jsonRoundTrip(jsonDecode(txt) as Map)); + jsonMs.add(ms(sw)); + bytes = txt.length; + if (bytes > maxBytes) maxBytes = bytes; + from = to; + } + print('stream 15-min folds: ${cuts.length} folds'); + print(' fold mean=${_mean(foldMs).toStringAsFixed(1)} ms max=${_max(foldMs).toStringAsFixed(1)} sum=${_sum(foldMs).toStringAsFixed(0)}'); + print(' snapshot mean=${_mean(snapMs).toStringAsFixed(2)} ms max=${_max(snapMs).toStringAsFixed(2)}'); + print(' json enc+dec+restore mean=${_mean(jsonMs).toStringAsFixed(2)} ms max=${_max(jsonMs).toStringAsFixed(2)} state bytes max=$maxBytes'); + print(' identical nn over 96 folds: ${_eq(all, oracle.nn)}'); + + // what the app does today: whole-day correctRr at pass k + for (final frac in [0.25, 0.5, 0.75, 1.0]) { + final m = (n * frac).floor(); + sw..reset()..start(); + correctRr(day.rr.sublist(0, m), rrTsMs: day.ts.sublist(0, m)); + print('oracle at ${(frac * 100).round()}% of day (n=$m): ${ms(sw).toStringAsFixed(0)} ms'); + } +} + +bool _eq(List a, List b) { + if (a.length != b.length) return false; + for (var i = 0; i < a.length; i++) { + if (a[i] != b[i]) return false; + } + return true; +} + +double _mean(List x) => x.isEmpty ? 0 : _sum(x) / x.length; +double _sum(List x) => x.fold(0.0, (a, b) => a + b); +double _max(List x) => x.fold(0.0, (a, b) => a > b ? a : b); diff --git a/tool/incremental/driver.dart b/tool/incremental/driver.dart new file mode 100644 index 0000000..4d3b552 --- /dev/null +++ b/tool/incremental/driver.dart @@ -0,0 +1,32 @@ +// Test/bench driver: RR stream -> RrCorrector -> per-fold callback with the +// SETTLED NN delta and the PROVISIONAL tail, so downstream states can be +// folded exactly the way the app would (settled in, tail evaluated on a copy). +import 'oracle_util.dart'; +import 'rr_stream.dart'; +import 'synth.dart'; + +class FoldStep { + final int fold; + final int n; // raw beats folded so far + final RrSettled settled; // newly settled NN + final RrSnapshot snap; // provisional tail + totals + FoldStep(this.fold, this.n, this.settled, this.snap); + double get artifactFraction => + (1.0 - snap.cleanFraction).clamp(0.0, 1.0).toDouble(); +} + +void driveRr(RrData s, List cuts, void Function(FoldStep) onFold, + {bool restart = false, int restartEvery = 1}) { + var c = RrCorrector(); + var from = 0; + for (var k = 0; k < cuts.length; k++) { + final to = cuts[k]; + final st = c.fold(s.rr.sublist(from, to), tsMs: s.ts.sublist(from, to)); + final snap = c.snapshot(); + onFold(FoldStep(k, to, st, snap)); + if (restart && k % restartEvery == 0) { + c = RrCorrector.fromJson(jsonRoundTrip(c.toJson())); + } + from = to; + } +} diff --git a/tool/incremental/edge_oracles.dart b/tool/incremental/edge_oracles.dart new file mode 100644 index 0000000..e4f6348 --- /dev/null +++ b/tool/incremental/edge_oracles.dart @@ -0,0 +1,260 @@ +// VERBATIM PORTS of the app-side (edge) functions that are not in analytics: +// dayHrvCurve, dayRespCurve, _daytimeHrv, _hrvTimeline, _respPerWindow. +// Source: edge-bgmem @ 26762fb5, lib/compute/derivation_engine.dart (~8532, +// 8641, 8866) and lib/compute/onehz_pipeline.dart (_hrvTimeline ~1926, +// _respPerWindow ~1692). Only change: `Substrate` -> the plain [DaySub] below +// and the CalculationCache plumbing removed (state == null path). They are the +// ORACLES the incremental states are compared with. +import 'dart:math' as math; + +import 'package:openstrap_analytics/onehz.dart' + hide RrCorrector, RrSettled, RrSnapshot, IrregularScreenState; + +/// edge `accelPlausible` (substrate.dart:100). +bool accelPlausible(double ax, double ay, double az) { + final magSq = ax * ax + ay * ay + az * az; + return magSq > 0 && magSq <= 4.0 * 4.0; +} + +const double kQuietCutG = 0.02; // _quietEnmoCutG gen4 / gen5 + +class DaySub { + final List rrMs, rrTsMs; + final List tsSec; + final List ax, ay, az; + DaySub(this.rrMs, this.rrTsMs, this.tsSec, this.ax, this.ay, this.az); + int get length => tsSec.length; + bool accelPresentAt(int i) => accelPlausible(ax[i], ay[i], az[i]); +} + +List> oracleDayHrvCurve(DaySub s) { + final ts = [], rr = []; + for (var i = 0; i < s.rrMs.length; i++) { + final v = s.rrMs[i]; + if (v >= 300 && v <= 2000) { + ts.add(s.rrTsMs[i]); + rr.add(v); + } + } + if (rr.length < 10) return const []; + const winMs = 300000.0; + final out = >[]; + var lo = 0; + var lastEmit = -1e18; + for (var i = 0; i < rr.length; i++) { + while (ts[i] - ts[lo] > winMs) { + lo++; + } + if (i - lo >= 10 && ts[i] - lastEmit > 60000) { + double? calculate() { + var ssd = 0.0; + var nd = 0; + for (var k = lo + 1; k <= i; k++) { + final d = rr[k] - rr[k - 1]; + if (d.abs() > 0.20 * rr[k - 1] || d.abs() > 200) continue; + ssd += d * d; + nd++; + } + if (nd < 8) return null; + final rmssd = math.sqrt(ssd / nd); + return rmssd <= 220 ? double.parse(rmssd.toStringAsFixed(1)) : null; + } + + final value = calculate(); + lastEmit = ts[i]; + if (value != null) out.add({'t': (ts[i] / 1000).round(), 'v': value}); + } + } + return out; +} + +const double kRespQuietFraction = 0.9; + +List> oracleDayRespCurve(DaySub s, + {double? cut = kQuietCutG, + double? Function(List nn, List nnt)? estimator}) { + if (cut == null) return const []; + final quietPrefix = List.filled(s.length + 1, 0); + for (var i = 0; i < s.length; i++) { + var q = 0; + if (s.accelPresentAt(i)) { + final mag = math.sqrt( + s.ax[i] * s.ax[i] + s.ay[i] * s.ay[i] + s.az[i] * s.az[i], + ); + if ((mag - 1.0).abs() <= cut) q = 1; + } + quietPrefix[i + 1] = quietPrefix[i] + q; + } + final ts = [], rr = []; + for (var i = 0; i < s.rrMs.length; i++) { + final v = s.rrMs[i]; + if (v >= 300 && v <= 2000) { + ts.add(s.rrTsMs[i]); + rr.add(v); + } + } + if (rr.length < 60) return const []; + const winMs = 180000.0; + final out = >[]; + var lo = 0; + var lastEmit = -1e18; + var qLo = 0, qHi = 0; + for (var i = 0; i < rr.length; i++) { + while (ts[i] - ts[lo] > winMs) { + lo++; + } + if (i - lo >= 30 && ts[i] - lastEmit > 300000) { + final loSec = (ts[lo] / 1000).floor(); + final hiSec = (ts[i] / 1000).ceil(); + while (qLo < s.length && s.tsSec[qLo] < loSec) { + qLo++; + } + if (qHi < qLo) qHi = qLo; + while (qHi < s.length && s.tsSec[qHi] < hiSec) { + qHi++; + } + final spanSec = hiSec - loSec; + final stillSec = quietPrefix[qHi] - quietPrefix[qLo]; + final nn = rr.sublist(lo, i + 1); + final t0 = ts[lo]; + final nnt = [for (var k = lo; k <= i; k++) ts[k] - t0]; + double? calculate() { + if (spanSec <= 0 || stillSec < kRespQuietFraction * spanSec) { + return null; + } + if (estimator != null) return estimator(nn, nnt); + final est = rsaRespRate(nn, nnt, artifactFraction: 0.15); + return est.present ? est.value!.brpm : null; + } + + final brpm = calculate(); + lastEmit = ts[i]; + if (brpm != null) { + out.add({ + 't': (ts[i] / 1000).round(), + 'v': double.parse(brpm.toStringAsFixed(1)), + }); + } + } + } + return out; +} + +Map oracleDaytimeHrv(DaySub s, int onsetSec, int offsetSec, + {double? cut = kQuietCutG}) { + const binSec = 300; + if (cut == null) { + return { + 'timeline': const >[], + 'mean_rmssd': null, + 'n_buckets': 0, + }; + } + final quiet = {}; + for (var i = 0; i < s.length; i++) { + if (!s.accelPresentAt(i)) continue; + final mag = + math.sqrt(s.ax[i] * s.ax[i] + s.ay[i] * s.ay[i] + s.az[i] * s.az[i]); + if ((mag - 1.0).abs() <= cut) quiet.add(s.tsSec[i]); + } + final bins = >{}; + double? prev; + for (var k = 0; k < s.rrMs.length; k++) { + final tSec = s.rrTsMs[k] ~/ 1000; + if (offsetSec > onsetSec && tSec >= onsetSec && tSec < offsetSec) { + prev = null; + continue; + } + if (!quiet.contains(tSec)) { + prev = null; + continue; + } + final v = s.rrMs[k]; + if (v < 300 || v > 2000) { + prev = null; + continue; + } + if (prev != null) { + final d = v - prev; + if (d.abs() <= 200) (bins[tSec ~/ binSec] ??= []).add(d * d); + } + prev = v; + } + final timeline = >[]; + final means = []; + final keys = bins.keys.toList()..sort(); + for (final b in keys) { + final sq = bins[b]!; + if (sq.length < 5) continue; + final rmssd = math.sqrt(sq.reduce((a, c) => a + c) / sq.length); + timeline.add({ + 't': b * binSec, + 'rmssd': (rmssd * 10).round() / 10.0, + 'n': sq.length, + }); + means.add(rmssd); + } + final mean = means.isEmpty ? null : means.reduce((a, c) => a + c) / means.length; + return { + 'timeline': timeline, + 'mean_rmssd': mean == null ? null : (mean * 10).round() / 10.0, + 'n_buckets': timeline.length, + }; +} + +double _round(double v, int dp) { + final p = math.pow(10, dp); + return (v * p).round() / p; +} + +List> oracleHrvTimeline( + List nn, List nnTimes, double? originMs) { + if (nn.length < 10 || nnTimes.length != nn.length || originMs == null) { + return const []; + } + const winMs = 300000.0; + final out = >[]; + var lo = 0; + for (var i = 0; i < nn.length; i++) { + while (nnTimes[i] - nnTimes[lo] > winMs) { + lo++; + } + if (nnTimes[i] - nnTimes[0] < winMs) continue; + if (i - lo >= 10) { + var ssd = 0.0; + for (var k = lo + 1; k <= i; k++) { + final diff = nn[k] - nn[k - 1]; + ssd += diff * diff; + } + final rmssd = math.sqrt(ssd / (i - lo)); + final tSec = ((originMs + nnTimes[i]) / 1000).round(); + if (out.isEmpty || tSec - out.last['t']! > 60) { + out.add({'t': tSec, 'v': _round(rmssd, 1)}); + } + } + } + return out; +} + +List oracleRespPerWindow(List nn, List nnTimes, + {double windowMs = 1800000.0, int minBeats = 60}) { + if (nn.isEmpty || nn.length != nnTimes.length) return const []; + final t0 = nnTimes.first; + final binsNn = >{}; + final binsTs = >{}; + for (var i = 0; i < nn.length; i++) { + final idx = ((nnTimes[i] - t0) / windowMs).floor(); + (binsNn[idx] ??= []).add(nn[i]); + (binsTs[idx] ??= []).add(nnTimes[i]); + } + final out = []; + final idxs = binsNn.keys.toList()..sort(); + for (final idx in idxs) { + final segNn = binsNn[idx]!; + if (segNn.length < minBeats) continue; + final r = rsaRespRate(segNn, binsTs[idx]!, artifactFraction: 0.0); + final b = r.present ? r.value!.brpm : null; + if (b != null) out.add(b); + } + return out; +} diff --git a/tool/incremental/err_probe.dart b/tool/incremental/err_probe.dart new file mode 100644 index 0000000..e2efacd --- /dev/null +++ b/tool/incremental/err_probe.dart @@ -0,0 +1,39 @@ +// Max abs/rel error of the Welford-based fields (irregular SD1/SD2, hrvTime SDNN) +// over many random prefixes. dart run tool/incremental/err_probe.dart +import 'dart:math' as math; +import 'package:openstrap_analytics/onehz.dart' + hide RrCorrector, RrSettled, RrSnapshot, IrregularScreenState; +import 'hrv_incr.dart'; +import 'synth.dart'; + +void main() { + final s = realNightRr()!; + final o = correctRr(s.rr, rrTsMs: s.ts); + var maxRelSdnn = 0.0, maxRelSd1 = 0.0, maxRelSd2 = 0.0, maxAcf = 0.0; + var checks = 0; + final r = math.Random(1); + final acc = HrvTimeAcc(); + final irr = IrregularScreenState(); + var at = 0; + while (at < o.nn.length) { + final to = math.min(o.nn.length, at + 1 + r.nextInt(2500)); + acc.fold(o.nn.sublist(at, to), o.nnTimesMs.sublist(at, to)); + irr.fold(o.nn.sublist(at, to), o.nnTimesMs.sublist(at, to)); + at = to; + if (to < 600) continue; + final nn = o.nn.sublist(0, to), t = o.nnTimesMs.sublist(0, to); + final w = hrvTime(nn, nnTimesMs: t); + final g = acc.evaluate(const [], const []); + maxRelSdnn = math.max(maxRelSdnn, (g.value!.sdnn! - w.value!.sdnn!).abs() / w.value!.sdnn!); + maxAcf = math.max(maxAcf, (g.value!.diffAcf1! - w.value!.diffAcf1!).abs()); + final wi = irregularBeatScreen(nn, nnTimesMs: t); + final gi = irr.evaluate(const [], const []); + if (wi.present) { + maxRelSd1 = math.max(maxRelSd1, (gi.value!.sd1 - wi.value!.sd1).abs() / wi.value!.sd1); + maxRelSd2 = math.max(maxRelSd2, (gi.value!.sd2 - wi.value!.sd2).abs() / wi.value!.sd2); + } + checks++; + } + print('checks=$checks hrvTime sdnn maxRel=$maxRelSdnn acf1 maxAbs=$maxAcf ' + 'irregular sd1 maxRel=$maxRelSd1 sd2 maxRel=$maxRelSd2'); +} diff --git a/tool/incremental/hrv_incr.dart b/tool/incremental/hrv_incr.dart new file mode 100644 index 0000000..872d8d3 --- /dev/null +++ b/tool/incremental/hrv_incr.dart @@ -0,0 +1,1598 @@ +// RESEARCH PROTOTYPES — incremental HRV scalars and curves. +// +// Common shape: `fold(settled input)` mutates the carried state; `evaluate` +// / `curve` answers for "settled + provisional tail" WITHOUT mutating it (the +// tail is re-evaluated on a copy). The settled input is what RrCorrector.fold +// returns; the provisional tail is RrSnapshot.tailNn/tailNnTimes. +import 'dart:math' as math; + +import 'dart:typed_data'; + +import 'package:openstrap_analytics/aggregate.dart' show Beats; +import 'package:openstrap_analytics/onehz.dart' + hide RrCorrector, RrSettled, RrSnapshot, IrregularScreenState; +import 'package:openstrap_analytics/src/onehz/clinical/hrv_jitter_evidence.dart'; + +// =========================================================================== +// 1. hrvTime scalars: RMSSD, pNN50, SDNN, SDANN, SDNN-index, diffAcf1 +// =========================================================================== +// +// RMSSD/pNN50/pairs: sums of d*d, count of |d|>50 taken in beat order are the +// SAME float sequence as the batch loop, so they are BIT-identical. +// SDNN: Welford instead of the batch's two-pass (mean, then sum (x-m)^2) — +// agrees to ~1e-13 relative, NOT bit-identical (a two-pass over the whole +// series cannot be folded). +// ACF1: algebraic form from (n, S1, S2, P, E, L); ~1e-14 absolute. The jitter +// gate (acf1 < -0.35) can only differ when |acf1 + 0.35| < ~1e-12. +// SDANN/SDNN-index: the 5-min segments close for good once a later beat +// arrives; their (mean, sd) are computed ONCE with the batch's own +// mean()/stddev() => bit-identical, and the final stddev(means)/mean(sds) +// run over <= 288 numbers with the batch's own functions. +class HrvTimeAcc { + int n = 0; + double prevNn = 0, prevT = 0, t0 = 0; + double wMean = 0, wM2 = 0; + int pairs = 0, over50 = 0, lagPairs = 0; + double ssd = 0, s1 = 0, s2 = 0, prod = 0, endp = 0; + double? lastDiff; + int segIdx = 0; + List cur = []; + List segMeans = [], segSds = []; + // Under the ACF floor hrvTime reads the difference runs (noise-share + // exemption, breathing-line fallback): the library's mergeable evidence for + // them, fed beat by beat (origin = the first beat). + JitterState ev = JitterState(); + JitterEvidenceAggregator? _evAgg; + + HrvTimeAcc(); + + HrvTimeAcc copy() { + final c = HrvTimeAcc() + ..n = n + ..prevNn = prevNn + ..prevT = prevT + ..t0 = t0 + ..wMean = wMean + ..wM2 = wM2 + ..pairs = pairs + ..over50 = over50 + ..lagPairs = lagPairs + ..ssd = ssd + ..s1 = s1 + ..s2 = s2 + ..prod = prod + ..endp = endp + ..lastDiff = lastDiff + ..segIdx = segIdx + ..cur = List.of(cur) + ..segMeans = List.of(segMeans) + ..segSds = List.of(segSds) + ..ev = ev.copy() + .._evAgg = _evAgg; + return c; + } + + void add(double v, double t) { + _evAgg ??= JitterEvidenceAggregator(originMs: t, gapAware: true); + ev = _evAgg!.addSlice( + ev, Beats(Float64List.fromList([v]), Float64List.fromList([t])), 0, 1); + if (n == 0) { + t0 = t; + } else if (t - prevT > v + .5) { + lastDiff = null; // seam: ends a diff run + } else { + final d = v - prevNn; + pairs++; + ssd += d * d; + s1 += d; + s2 += d * d; + if (d.abs() > 50) over50++; + if (lastDiff != null) { + lagPairs++; + prod += lastDiff! * d; + endp += lastDiff! + d; + } + lastDiff = d; + } + n++; + final dl = v - wMean; + wMean += dl / n; + wM2 += dl * (v - wMean); + final idx = ((t - t0) / 300000.0).floor(); + if (idx != segIdx) { + if (cur.length >= 2) { + segMeans.add(mean(cur)!); + segSds.add(stddev(cur)!); + } + cur = []; + segIdx = idx; + } + cur.add(v); + prevNn = v; + prevT = t; + } + + void fold(List nn, List t) { + for (var i = 0; i < nn.length; i++) { + add(nn[i], t[i]); + } + } + + /// `hrvTime(settled ++ tail, nnTimesMs: ..., artifactFraction)`. + Metric evaluate(List tailNn, List tailT, + {double artifactFraction = 0.0}) { + final a = copy()..fold(tailNn, tailT); + return a._finish(artifactFraction); + } + + Metric _finish(double artifactFraction) { + const inputs = ['rr_cleaned']; + if (n < 2) { + return const Metric.absent( + tier: Tier.high, + inputs_used: inputs, + note: 'too few NN intervals', + ); + } + double? acf1; + if (pairs >= 30) { + final m = s1 / pairs; + final varSum = s2 - m * s1; + final cov = prod - m * endp + lagPairs * m * m; + acf1 = varSum > 0 ? cov / varSum : null; + } + final evidence = acf1 != null && acf1 < kNnDiffAcf1Floor + ? _evAgg!.finish(ev) + : null; + final jittery = evidence != null && evidence.refused(); + var rmssd = (pairs > 0 && !jittery) ? math.sqrt(ssd / pairs) : null; + final pnn50 = (pairs > 0 && !jittery) ? 100.0 * over50 / pairs : null; + if (jittery) rmssd = evidence.breathingRmssd(); + final byBreathing = jittery && rmssd != null; + final sdnn = n >= 2 ? math.sqrt(wM2 / (n - 1)) : null; + double? sdann, sdnnIndex; + final means = [...segMeans], sds = [...segSds]; + if (cur.length >= 2) { + means.add(mean(cur)!); + sds.add(stddev(cur)!); + } + if (means.length >= 2) { + sdann = stddev(means); + sdnnIndex = sds.isEmpty ? null : mean(sds); + } + final q = acf1 == null ? 1.0 : (1 - acf1 / kNnDiffAcf1Floor).clamp(0.0, 1.0); + final conf = byBreathing + ? 0.3 + : ((n / 250.0).clamp(0.0, 1.0) * q * (1 - artifactFraction)) + .clamp(0.3, 0.95); + return Metric( + value: HrvTime( + rmssd: rmssd, + sdnn: sdnn, + sdann: sdann, + sdnnIndex: sdnnIndex, + pnn50: pnn50, + nBeats: n, + diffAcf1: acf1, + ), + confidence: conf, + tier: Tier.high, + inputs_used: inputs, + note: jittery + ? 'rmssd_refused:acf1=${acf1!.toStringAsFixed(3)} — the NN successive ' + 'differences are essentially differenced white noise (−0.5 = pure, floor ' + '$kNnDiffAcf1Floor), so RMSSD/pNN50 would measure beat-timing jitter, not ' + 'vagal tone. SDNN/SDANN survive it and are the lead here. ' + '${byBreathing ? 'RMSSD only from 5-min windows on a breathing ' + 'line steady in Hz, at floor confidence. ' : ''}PRV not ECG-HRV.' + : 'PRV not ECG-HRV; RMSSD/pNN50 are quantization-sensitive at 1 Hz ' + '— lead with SDNN/SDANN', + ); + } + + Map toJson() => { + 'n': n, + 'prevNn': prevNn, + 'prevT': prevT, + 't0': t0, + 'wMean': wMean, + 'wM2': wM2, + 'pairs': pairs, + 'over50': over50, + 'lagPairs': lagPairs, + 'ssd': ssd, + 's1': s1, + 's2': s2, + 'prod': prod, + 'endp': endp, + 'lastDiff': lastDiff, + 'segIdx': segIdx, + 'cur': cur, + 'segMeans': segMeans, + 'segSds': segSds, + 'ev': _evAgg == null ? null : _evAgg!.encodeState(ev), + }; + + factory HrvTimeAcc.fromJson(Map j) { + List dl(Object? o) => [for (final x in o as List) (x as num).toDouble()]; + return HrvTimeAcc() + ..n = j['n'] as int + ..prevNn = (j['prevNn'] as num).toDouble() + ..prevT = (j['prevT'] as num).toDouble() + ..t0 = (j['t0'] as num).toDouble() + ..wMean = (j['wMean'] as num).toDouble() + ..wM2 = (j['wM2'] as num).toDouble() + ..pairs = j['pairs'] as int + ..over50 = j['over50'] as int + ..lagPairs = j['lagPairs'] as int + ..ssd = (j['ssd'] as num).toDouble() + ..s1 = (j['s1'] as num).toDouble() + ..s2 = (j['s2'] as num).toDouble() + ..prod = (j['prod'] as num).toDouble() + ..endp = (j['endp'] as num).toDouble() + ..lastDiff = (j['lastDiff'] as num?)?.toDouble() + ..segIdx = j['segIdx'] as int + ..cur = dl(j['cur']) + ..segMeans = dl(j['segMeans']) + ..segSds = dl(j['segSds']) + .._restoreEvidence(j['ev']); + } + + void _restoreEvidence(Object? e) { + if (e == null || n == 0) return; + _evAgg = JitterEvidenceAggregator(originMs: t0, gapAware: true); + ev = _evAgg!.decodeState(e); + } +} + +// =========================================================================== +// 2. edge `_hrvTimeline`: trailing 5-min RMSSD of the corrected NN, >60 s apart +// =========================================================================== +// Right-context 0 over the NN list, so a point is final the moment its NN beat +// is settled. State = ring of the last 5 min of NN, first time, last emitted t. +class HrvTimelineState { + final double originMs; + List _nn = [], _t = []; + int _head = 0; + double? _t0; + int? _lastT; + final List> out = []; + + HrvTimelineState(this.originMs); + + static double _round(double v, int dp) { + final p = math.pow(10, dp); + return (v * p).round() / p; + } + + void _step(double v, double tm, List> sink) { + _t0 ??= tm; + _nn.add(v); + _t.add(tm); + while (tm - _t[_head] > 300000.0) { + _head++; + } + if (tm - _t0! < 300000.0) return; + final cnt = _nn.length - 1 - _head; + if (cnt >= 10) { + var ssd = 0.0; + for (var k = _head + 1; k < _nn.length; k++) { + final diff = _nn[k] - _nn[k - 1]; + ssd += diff * diff; + } + final rmssd = math.sqrt(ssd / cnt); + final tSec = ((originMs + tm) / 1000).round(); + if (_lastT == null || tSec - _lastT! > 60) { + sink.add({'t': tSec, 'v': _round(rmssd, 1)}); + _lastT = tSec; + } + } + } + + void _trim() { + if (_head > 512) { + _nn = _nn.sublist(_head); + _t = _t.sublist(_head); + _head = 0; + } + } + + void fold(List nn, List t) { + for (var i = 0; i < nn.length; i++) { + _step(nn[i], t[i], out); + } + _trim(); + } + + /// settled points ++ provisional points from [tailNn]. + List> curve(List tailNn, List tailT) { + final c = HrvTimelineState(originMs) + .._nn = List.of(_nn) + .._t = List.of(_t) + .._head = _head + .._t0 = _t0 + .._lastT = _lastT; + final extra = >[]; + for (var i = 0; i < tailNn.length; i++) { + c._step(tailNn[i], tailT[i], extra); + } + return [...out, ...extra]; + } +} + +// =========================================================================== +// 3. nightHrvShape: per-30-min-bin RMSSD (+ band) over the night +// =========================================================================== +class NightShapeState { + final double binMin; + final int minBeatsPerBin; + double? _t0; + int _bin = 0; + List _nn = [], _t = []; + int _count = 0; + double _firstT = 0, _lastT = 0; + // closed bins by index (holes included): [n, rmssd?, lo?, hi?] + final List> closed = []; + + NightShapeState({this.binMin = 30, this.minBeatsPerBin = kMinBeatsPerHrvBin}); + + double get _binMs => binMin * 60000.0; + double _hi(int b) => _t0! + (b + 1) * _binMs; + + static List _binValue( + List nn, List t, int minBeats) { + final n = nn.length; + if (n < minBeats) return [n, null, null, null]; + final m = hrvTime(nn, nnTimesMs: t); + final rmssd = m.value?.rmssd; + final se = rmssd == null ? null : rmssd / math.sqrt(2 * n); + return [ + n, + rmssd, + se == null ? null : math.max(0.0, rmssd! - 1.96 * se), + se == null ? null : rmssd! + 1.96 * se, + ]; + } + + void _step(double v, double tm, List> sink) { + if (_t0 == null) { + _t0 = tm; + _firstT = tm; + } + _lastT = tm; + _count++; + while (tm >= _hi(_bin)) { + sink.add(_binValue(_nn, _t, minBeatsPerBin)); + _nn = []; + _t = []; + _bin++; + } + _nn.add(v); + _t.add(tm); + } + + void fold(List nn, List t) { + for (var i = 0; i < nn.length; i++) { + _step(nn[i], t[i], closed); + } + } + + Metric evaluate(List tailNn, List tailT) { + const inputs = ['rr_cleaned', 'beat_times']; + final c = NightShapeState(binMin: binMin, minBeatsPerBin: minBeatsPerBin) + .._t0 = _t0 + .._bin = _bin + .._nn = List.of(_nn) + .._t = List.of(_t) + .._count = _count + .._firstT = _firstT + .._lastT = _lastT + ..closed.addAll(closed); + for (var i = 0; i < tailNn.length; i++) { + c._step(tailNn[i], tailT[i], c.closed); + } + final total = c._count; + if (total < minBeatsPerBin) { + return Metric.absent( + tier: Tier.high, + inputs_used: inputs, + note: 'too few beats for a nightly shape ' + '($total, need ≥$minBeatsPerBin)', + ); + } + final binMs = _binMs; + final span = c._lastT - c._firstT; + final nBins = math.max(1, (span / binMs).ceil()); + if (nBins < 3) { + return Metric.absent( + tier: Tier.high, + inputs_used: inputs, + note: + 'night spans ${(span / 3600000).toStringAsFixed(1)} h — under three ' + '${binMin.round()}-min bins there is no shape to describe', + ); + } + final bins = []; + for (var b = 0; b < nBins; b++) { + final List v = b < c.closed.length + ? c.closed[b] + : (b == c._bin + ? _binValue(c._nn, c._t, minBeatsPerBin) + : [0, null, null, null]); + bins.add(NightHrvBin( + startSec: (b * binMs / 1000).round(), + nBeats: v[0]!.toInt(), + rmssdMs: v[1]?.toDouble(), + loMs: v[2]?.toDouble(), + hiMs: v[3]?.toDouble(), + )); + } + final third = bins.length ~/ 3; + double? meanOf(Iterable xs) { + final v = [ + for (final b in xs) + if (b.present) b.rmssdMs! + ]; + return v.length < 2 ? null : mean(v); + } + + final first = meanOf(bins.take(third)); + final last = meanOf(bins.skip(bins.length - third)); + final ratio = + (first == null || last == null || first == 0) ? null : last / first; + return Metric( + value: NightHrvShape( + bins: bins, + firstThirdMs: first, + lastThirdMs: last, + lastOverFirst: ratio, + ), + confidence: (0.8 * bins.where((b) => b.present).length / bins.length) + .clamp(0.0, 0.8), + tier: Tier.high, + inputs_used: inputs, + note: 'per-bin RMSSD (${binMin.round()}-min bins, PRV not ECG-HRV) — a ' + 'DESCRIPTION of the night, never a cause. A low first third is equally ' + 'consistent with alcohol, a late meal, late training, a warm room, ' + 'illness onset, or nothing. Render each bin as a band, not a point.', + ); + } +} + +// =========================================================================== +// 4. edge `dayHrvCurve`: trailing 5-min RMSSD on gated RR, emitted every 60 s +// =========================================================================== +// Right-context 0. Exact append-only. State: ring of gated beats of the last +// 5 min, lastEmit, emitted list. (Works on raw substrate RR, not on NN.) +class DayHrvCurveState { + List _ts = [], _rr = []; + int _head = 0; + double _lastEmit = -1e18; + final List> out = []; + + DayHrvCurveState(); + + void fold(List rrMs, List rrTsMs) { + for (var q = 0; q < rrMs.length; q++) { + final v = rrMs[q]; + if (!(v >= 300 && v <= 2000)) continue; + _ts.add(rrTsMs[q]); + _rr.add(v); + final i = _ts.length - 1; + while (_ts[i] - _ts[_head] > 300000.0) { + _head++; + } + if (i - _head >= 10 && _ts[i] - _lastEmit > 60000) { + double? value; + var ssd = 0.0; + var nd = 0; + for (var k = _head + 1; k <= i; k++) { + final d = _rr[k] - _rr[k - 1]; + if (d.abs() > 0.20 * _rr[k - 1] || d.abs() > 200) continue; + ssd += d * d; + nd++; + } + if (nd >= 8) { + final rmssd = math.sqrt(ssd / nd); + value = rmssd <= 220 ? double.parse(rmssd.toStringAsFixed(1)) : null; + } + _lastEmit = _ts[i]; + if (value != null) out.add({'t': (_ts[i] / 1000).round(), 'v': value}); + } + } + if (_head > 512) { + _ts = _ts.sublist(_head); + _rr = _rr.sublist(_head); + _head = 0; + } + } + + int get ringSize => _ts.length - _head; + + Map toJson() => { + 'ts': _ts.sublist(_head), + 'rr': _rr.sublist(_head), + 'lastEmit': _lastEmit, + 'out': out, + }; + + factory DayHrvCurveState.fromJson(Map j) { + final s = DayHrvCurveState(); + s._ts = [for (final x in j['ts'] as List) (x as num).toDouble()]; + s._rr = [for (final x in j['rr'] as List) (x as num).toDouble()]; + s._lastEmit = (j['lastEmit'] as num).toDouble(); + for (final m in j['out'] as List) { + final mm = m as Map; + s.out.add({'t': mm['t'] as num, 'v': mm['v'] as num}); + } + return s; + } +} + +// =========================================================================== +// 5. edge `_daytimeHrv`: 5-min bins of squared successive differences over +// quiet seconds, outside the (known) sleep window +// =========================================================================== +// Each pair (prev, v) lands in bin tSec~/300 and only needs (a) whether the +// second of each beat is quiet and (b) the previous beat in the quiet chain. +// State: per-bin (sum of d^2, count) in beat order, the chain's `prev`, quiet +// seconds near the head, pending beats whose second the accelerometer has not +// reported yet (watermark). Sleep window fixed at construction; a changed +// window means refold (only bins straddling onset/offset differ — see report). +class DaytimeHrvState { + final int onsetSec, offsetSec; + final double cut; + final Map _sum = {}; + final Map _cnt = {}; + double? _prev; + final Set _quiet = {}; + final List _pendRr = [], _pendTs = []; + int _maxBeatSec = 0; + + DaytimeHrvState(this.onsetSec, this.offsetSec, {this.cut = 0.02}); + + static bool _isQuiet(double ax, double ay, double az, double cut) { + final magSq = ax * ax + ay * ay + az * az; + if (!(magSq > 0 && magSq <= 16.0)) return false; + final mag = math.sqrt(ax * ax + ay * ay + az * az); + return (mag - 1.0).abs() <= cut; + } + + /// [watermarkSec]: every accel row with tsSec < watermarkSec has been given. + void fold(List rrMs, List rrTsMs, List accTs, + List ax, List ay, List az, int watermarkSec) { + for (var i = 0; i < accTs.length; i++) { + if (_isQuiet(ax[i], ay[i], az[i], cut)) _quiet.add(accTs[i]); + } + _pendRr.addAll(rrMs); + _pendTs.addAll(rrTsMs); + var used = 0; + for (; used < _pendRr.length; used++) { + final tSec = _pendTs[used] ~/ 1000; + if (tSec >= watermarkSec) break; // second not fully known yet + _beat(_pendRr[used], tSec); + } + _pendRr.removeRange(0, used); + _pendTs.removeRange(0, used); + // forget quiet seconds far behind the head (RR is time-ordered) + if (_quiet.length > 4000) { + final keep = _maxBeatSec - 120; + _quiet.removeWhere((s) => s < keep); + } + } + + void _beat(double v, int tSec) { + if (tSec > _maxBeatSec) _maxBeatSec = tSec; + if (offsetSec > onsetSec && tSec >= onsetSec && tSec < offsetSec) { + _prev = null; + return; + } + if (!_quiet.contains(tSec)) { + _prev = null; + return; + } + if (v < 300 || v > 2000) { + _prev = null; + return; + } + if (_prev != null) { + final d = v - _prev!; + if (d.abs() <= 200) { + final b = tSec ~/ 300; + _sum[b] = (_sum[b] ?? 0.0) + d * d; + _cnt[b] = (_cnt[b] ?? 0) + 1; + } + } + _prev = v; + } + + Map result() { + final timeline = >[]; + final means = []; + final keys = _sum.keys.toList()..sort(); + for (final b in keys) { + final n = _cnt[b]!; + if (n < 5) continue; + final rmssd = math.sqrt(_sum[b]! / n); + timeline.add({'t': b * 300, 'rmssd': (rmssd * 10).round() / 10.0, 'n': n}); + means.add(rmssd); + } + final mean = means.isEmpty ? null : means.reduce((a, c) => a + c) / means.length; + return { + 'timeline': timeline, + 'mean_rmssd': mean == null ? null : (mean * 10).round() / 10.0, + 'n_buckets': timeline.length, + }; + } + + int get pending => _pendRr.length; + + Map toJson() => { + 'sum': {for (final e in _sum.entries) '${e.key}': e.value}, + 'cnt': {for (final e in _cnt.entries) '${e.key}': e.value}, + 'prev': _prev, + 'quiet': _quiet.toList(), + 'pendRr': _pendRr, + 'pendTs': _pendTs, + 'maxBeatSec': _maxBeatSec, + }; + + factory DaytimeHrvState.fromJson( + Map j, int onsetSec, int offsetSec, + {double cut = 0.02}) { + final s = DaytimeHrvState(onsetSec, offsetSec, cut: cut); + (j['sum'] as Map).forEach((k, v) => s._sum[int.parse(k as String)] = (v as num).toDouble()); + (j['cnt'] as Map).forEach((k, v) => s._cnt[int.parse(k as String)] = v as int); + s._prev = (j['prev'] as num?)?.toDouble(); + s._quiet.addAll((j['quiet'] as List).cast()); + s._pendRr.addAll([for (final x in j['pendRr'] as List) (x as num).toDouble()]); + s._pendTs.addAll([for (final x in j['pendTs'] as List) (x as num).toDouble()]); + s._maxBeatSec = j['maxBeatSec'] as int; + return s; + } +} + +// =========================================================================== +// 6. nocturnalRmssd (median of 5-min window RMSSDs) and +// sleepSessionWindowedRmssd (mean of cleaned 5-min window RMSSDs) +// =========================================================================== +// Both are "closed windows" estimators: a window's RMSSD is a pure function of +// the beats inside it, and the headline is an order-statistic / mean over the +// <= ~100 window values. State = per-window record. The stage mask (nocturnal) +// and the session bounds (session) are applied at READ time over those records, +// so a re-staged night does not invalidate them. +// +// Upstream's jitter gate is no longer a pooled ACF1 alone: a night below the +// ACF1 floor can still pass on a respiratory line in the pooled spectrum (then +// every window must clear the gate itself), or, as a last resort, on the +// windows whose breathing line holds still in Hz. Those need each window's +// difference RUNS, not additive sums, so a record keeps its runs (a window +// holds a few hundred differences) and the gate is the batch's own code over +// them, bit-identical. + +/// One closed window. [kind]: 0 usable, 1 over-counted (more beat-time than +/// the window has), 2 thin (fewer differences than the floor). Only a usable +/// window carries [rmssd], [runs] and [meanRr]. +class _WinRec { + final int idx; + final int kind; + final double rmssd; + final List> runs; + final double meanRr; + _WinRec(this.idx, this.kind, this.rmssd, this.runs, this.meanRr); + _WinRec.dropped(this.idx, this.kind) + : rmssd = double.nan, + runs = const [], + meanRr = double.nan; + Object toJson() => [idx, kind, rmssd.isFinite ? rmssd : null, runs, + meanRr.isFinite ? meanRr : null]; +} + +/// Σd² and the count of [runs], summed in the batch's order. +(double, int) _ssdOf(List> runs) { + var ssd = 0.0; + var nd = 0; + for (final r in runs) { + for (final d in r) { + ssd += d * d; + nd++; + } + } + return (ssd, nd); +} + +class NocturnalRmssdState { + final double windowMs; + final int minBeatsPerWindow; + double? _t0; + int _idx = 0; + List _nn = [], _t = []; + final List<_WinRec> recs = []; + + NocturnalRmssdState({this.windowMs = 300000.0, this.minBeatsPerWindow = 5}); + + _WinRec? _close(int idx, List nn, List t) { + if (nn.length < minBeatsPerWindow + 1) return null; + final winRuns = >[]; + var run = []; + for (var k = 1; k < nn.length; k++) { + if (t[k] - t[k - 1] > nn[k] + 0.5) { + if (run.isNotEmpty) { + winRuns.add(run); + run = []; + } + continue; + } + run.add(nn[k] - nn[k - 1]); + } + if (run.isNotEmpty) winRuns.add(run); + final (ssd, nd) = _ssdOf(winRuns); + if (nd < minBeatsPerWindow) return null; + return _WinRec(idx, 0, math.sqrt(ssd / nd), winRuns, mean(nn)!); + } + + void _step(double v, double tm, List<_WinRec> sink) { + _t0 ??= tm; + final idx = ((tm - _t0!) / windowMs).floor(); + if (idx != _idx && _nn.isNotEmpty) { + final r = _close(_idx, _nn, _t); + if (r != null) sink.add(r); + _nn = []; + _t = []; + } + _idx = idx; + _nn.add(v); + _t.add(tm); + } + + int _count = 0; + void fold(List nn, List t) { + for (var i = 0; i < nn.length; i++) { + _step(nn[i], t[i], recs); + } + _count += nn.length; + } + + Metric evaluate(List tailNn, List tailT, + {List? stageMaskPerSec, RrCoverage? coverage}) { + const inputs = ['rr_cleaned', 'beat_times']; + if (coverage != null && coverage.overCounted) { + return Metric.absent( + tier: Tier.high, + inputs_used: inputs, + note: rrOvercountNote(coverage), + ); + } + final c = NocturnalRmssdState( + windowMs: windowMs, minBeatsPerWindow: minBeatsPerWindow) + .._t0 = _t0 + .._idx = _idx + .._nn = List.of(_nn) + .._t = List.of(_t) + ..recs.addAll(recs); + for (var i = 0; i < tailNn.length; i++) { + c._step(tailNn[i], tailT[i], c.recs); + } + final total = _count + tailNn.length; + if (total < minBeatsPerWindow + 1) { + return const Metric.absent( + tier: Tier.high, + inputs_used: inputs, + note: 'too few NN intervals for windowed nocturnal RMSSD', + ); + } + final all = [...c.recs]; + final open = c._close(c._idx, c._nn, c._t); + if (open != null) all.add(open); + final rmssds = []; + final runs = >[]; + final perWindow = >>[]; + final meanRr = []; + for (final r in all) { + if (stageMaskPerSec != null) { + final midSec = ((r.idx + 0.5) * windowMs / 1000.0).floor(); + final keep = midSec >= 0 && + midSec < stageMaskPerSec.length && + stageMaskPerSec[midSec]; + if (!keep) continue; + } + rmssds.add(r.rmssd); + runs.addAll(r.runs); + perWindow.add(r.runs); + meanRr.add(r.meanRr); + } + final acf1 = nnDiffAcf1(runs); + final refused = _jitterRefused(acf1, runs); + if (rmssds.isEmpty && !refused) { + return const Metric.absent( + tier: Tier.high, + inputs_used: inputs, + note: 'no usable 5-min windows for nocturnal RMSSD', + ); + } + final kept = _keptWindows(acf1, refused, rmssds, perWindow, meanRr); + if (kept == null) { + return Metric.absent( + tier: Tier.high, + inputs_used: inputs, + note: refused + ? _jitterNote(acf1!) + : '${_jitterNote(acf1!)}; no 5-min window clears it on its own', + ); + } + final robust = median(kept.rmssds)!; + final conf = kept.byBreathing + ? 0.3 + : ((kept.rmssds.length / 12.0).clamp(0.0, 1.0) * _acf1Quality(acf1)) + .clamp(0.3, 0.95); + return Metric( + value: robust, + confidence: conf, + tier: Tier.high, + inputs_used: inputs, + note: 'robust nocturnal RMSSD = MEDIAN of ${kept.rmssds.length} consecutive ' + '5-min-window RMSSDs (REM/arousal-robust). PRV not ECG-HRV; ' + 'RMSSD is quantization-sensitive at 1 Hz.${kept.note}', + ); + } +} + +// ---- verbatim copies of hrv_time.dart's private RMSSD gate (private there) -- +/// True when every successive difference is a whole multiple of one step q and +/// the mean squared difference is under 2q². Beat times rounded to a grid q +/// make RR a two-level sequence on a near-constant heart: its differences are +/// 0 or ±q, a deterministic sawtooth at frac(RR/q) cycles/beat, so it reads as +/// one clean line and the white-floor test cannot see it. Rounding alone puts +/// mean(d²) at 2·min(f, 1−f)·q² ≤ q², so under 2q² the grid, not the heart, +/// sets RMSSD. +// ponytail: exact lattice only; a grid re-rounded to whole ms (7.8 ms → 7/8) +// breaks the common step and is not caught. +bool _onCoarseLattice(List> diffRuns, double msd) { + var q = double.infinity; + for (final r in diffRuns) { + for (final d in r) { + if (d.abs() > 1e-6 && d.abs() < q) q = d.abs(); + } + } + if (q.isInfinite || msd >= 2 * q * q) return false; + for (final r in diffRuns) { + for (final d in r) { + final k = d.abs() / q; + if ((k - k.roundToDouble()).abs() > 1e-3) return false; + } + } + return true; +} + +/// The one RMSSD jitter verdict every path shares: ACF1 below the floor AND +/// the spectrum does not show a respiratory line on a low white floor. +bool _jitterRefused(double? acf1, List> runs, + {int minSegments = 20}) { + if (acf1 == null || acf1 >= kNnDiffAcf1Floor) return false; + final share = nnDiffNoiseShare(runs, minSegments: minSegments); + return share == null || share >= kNnDiffNoiseShareCeiling; +} + +/// One 5-min window judged on its own: a measured ACF1 at or above the floor, +/// or its own spectral line (4 segments is what a window at HR ~40 holds). +bool _windowClears(List> runs) { + final a = nnDiffAcf1(runs); + return a != null && !_jitterRefused(a, runs, minSegments: 4); +} + +/// The window RMSSDs a headline may use. A night that cleared the floor +/// outright keeps them all. One that only passed through the spectral +/// exemption proved a line somewhere in the pooled power, not in every window: +/// quiet jitter-only windows can win the median or dilute the mean while a few +/// loud breathing windows carry the pooled share. There every window has to +/// clear the gate itself. +List _clearedWindows(double? nightAcf1, List rmssds, + List>> winRuns) { + if (nightAcf1 == null || nightAcf1 >= kNnDiffAcf1Floor) return rmssds; + return [ + for (var i = 0; i < rmssds.length; i++) + if (_windowClears(winRuns[i])) rmssds[i] + ]; +} + +final _hann128 = [ + for (var i = 0; i < 128; i++) 0.5 - 0.5 * math.cos(2 * math.pi * i / 127) +]; +final _hann128Sq = _hann128.fold(0.0, (a, w) => a + w * w); + +/// Welch power (128-beat Hann, 50 % overlap, averaged over segments) of one +/// window's RR, rebuilt from its difference runs, at cycles-per-beat +/// frequencies [f]. Null when no run holds a full segment. +List? _windowPsd(List> diffRuns, List f) { + const n = 128; + final p = List.filled(f.length, 0.0); + var segs = 0; + for (final r in diffRuns) { + final x = List.filled(r.length + 1, 0.0); + for (var i = 0; i < r.length; i++) { + x[i + 1] = x[i] + r[i]; + } + for (var s = 0; s + n <= x.length; s += n ~/ 2) { + var m = 0.0; + for (var i = 0; i < n; i++) { + m += x[s + i]; + } + m /= n; + for (var j = 0; j < f.length; j++) { + final c = math.cos(2 * math.pi * f[j]), sn = math.sin(2 * math.pi * f[j]); + var cr = 1.0, ci = 0.0, re = 0.0, im = 0.0; + for (var i = 0; i < n; i++) { + final v = (x[s + i] - m) * _hann128[i]; + re += v * cr; + im += v * ci; + final t = cr * c - ci * sn; + ci = cr * sn + ci * c; + cr = t; + } + p[j] += re * re + im * im; + } + segs++; + } + } + return segs == 0 ? null : [for (final v in p) v / segs]; +} + +/// Last resort for a night the jitter gate refused: the windows whose +/// breathing line holds still in Hz while the heart rate moves. +/// +/// RSA follows breathing (Hirsch & Bishop 1981), a rate in breaths per +/// minute, so its line sits at a fixed frequency in Hz; a heart rate that +/// drifts across the night slides it in cycles per beat. Beat-timing jitter, +/// alternation and grid artifacts are tied to the beat, not the clock. So each +/// 5-min window's spectrum is pooled twice, once on a cycles-per-beat axis and +/// once rescaled by its own mean RR onto a Hz axis, and the night passes only +/// when the Hz pooling shows a line in 8–30 br/min that is sharper than the +/// beat pooling. If the heart rate barely moves the two poolings coincide and +/// the night stays refused: stability in Hz says nothing there. +/// +/// Ceiling: needs >= 12 windows and a heart rate that wanders; breathing at +/// half the heart rate is still an alternation and still refused. +/// Returns the indices of the windows whose own peak sits on the line. +List? _steadyBreathingWindows( + List>> winRuns, List meanRrMs) { + var ssd = 0.0, nd = 0; + final all = [for (final w in winRuns) ...w]; + for (final r in all) { + for (final d in r) { + ssd += d * d; + nd++; + } + } + if (nd == 0 || _onCoarseLattice(all, ssd / nd)) return null; + final ref = median([for (final m in meanRrMs) if (m > 0) m]); + if (ref == null) return null; + // 0.15–0.5 cycles/beat at the night's median beat. + final fc = [for (var j = 38; j <= 128; j++) j / 256]; + final byBeat = List.filled(fc.length, 0.0); + final byHz = List.filled(fc.length, 0.0); + final cover = List.filled(fc.length, 0); + final peakHz = {}; // window -> its own peak, cycles per ms + final psd = >{}; // window -> its cycles-per-beat PSD + for (var w = 0; w < winRuns.length; w++) { + if (meanRrMs[w] <= 0) continue; + final pb = _windowPsd(winRuns[w], fc); + if (pb == null) continue; + final norm = median(pb)!; + if (norm <= 0) continue; + final k = meanRrMs[w] / ref; + final fh = [for (final f in fc) f * k]; + final ph = _windowPsd(winRuns[w], fh)!; + var best = -1; + for (var j = 0; j < fc.length; j++) { + byBeat[j] += pb[j] / norm; + if (fh[j] > 0.5) continue; + byHz[j] += ph[j] / norm; + cover[j]++; + if (best < 0 || ph[j] > ph[best]) best = j; + } + if (best >= 0) peakHz[w] = fc[best] / ref; + psd[w] = pb; + } + final nw = peakHz.length; + if (nw < 12) return null; + final bins = [for (var j = 0; j < fc.length; j++) if (cover[j] == nw) j]; + if (bins.length < 20) return null; + final hz = [for (final j in bins) byHz[j]]; + final bt = [for (final j in bins) byBeat[j]]; + var pk = 0; + for (var i = 1; i < hz.length; i++) { + if (hz[i] > hz[pk]) pk = i; + } + if (pk == 0 || pk == hz.length - 1) return null; + final f0 = fc[bins[pk]] / ref; + final brpm = f0 * 60000; + if (brpm < 8 || brpm > 30) return null; + // Same bins, same normalisation: the line must stand taller aligned in Hz + // than the beat pooling stands anywhere within ±12 bins of it. + var near = 0.0; + for (var i = math.max(0, pk - 12); i <= math.min(bt.length - 1, pk + 12); i++) { + near = math.max(near, bt[i]); + } + // The pooled noise ripple shrinks as 1/√windows, so the bar does too. + final promHz = hz[pk] / median(hz)!; + if (promHz < math.max(2.5, 1.5 + 7 / math.sqrt(nw))) return null; + if (promHz < 1.1 * near / median(bt)!) return null; + // Alternation lives at Nyquist on the beat axis, outside these bins. + if (byBeat.last / median(bt)! >= 0.5 * promHz) return null; + final keep = [ + for (final e in peakHz.entries) + if (math.log(e.value / f0).abs() <= 0.15) e.key + ]; + if (keep.length < 6) return null; + // The line proves breathing is there, not that it carries RMSSD: beat-time + // jitter loud enough to fail the gate would still be most of the number. + // Same ceiling as [nnDiffNoiseShare], over the kept windows. Jitter σ² has + // an RR spectrum σ²·(2 − 2cos ω), flat once divided by that shape, and its + // differences carry 6σ²; physiology only adds power on top. Breathing + // wanders across the night and its line spreads with it, so each window + // drops the bins of every rate the kept windows peaked at (plus a Hann + // main lobe), mapped onto its own beats. σ² is the floor of what is left, + // pooled over windows and smoothed over four resolution cells. + var lo = double.infinity, hi = 0.0; + for (final w in keep) { + lo = math.min(lo, peakHz[w]!); + hi = math.max(hi, peakHz[w]!); + } + var sq = 0.0, nAll = 0; + final acc = List.filled(fc.length, 0.0); + final wt = List.filled(fc.length, 0.0); + for (final w in keep) { + var n = 0; + for (final r in winRuns[w]) { + for (final d in r) { + sq += d * d; + n++; + } + } + nAll += n; + final a = lo * meanRrMs[w] - 3 / 128, b = hi * meanRrMs[w] + 3 / 128; + for (var j = 0; j < fc.length; j++) { + if (fc[j] > a && fc[j] < b) continue; + acc[j] += n * psd[w]![j] / (2 - 2 * math.cos(2 * math.pi * fc[j])); + wt[j] += n; + } + } + var floor = double.infinity; + for (var j = 8; j < fc.length - 8; j++) { + var s = 0.0, m = 0; + for (var i = j - 8; i <= j + 8; i++) { + if (wt[i] < nAll / 2) break; + s += acc[i] / wt[i]; + m++; + } + if (m == 17) floor = math.min(floor, s / 17); + } + if (floor == double.infinity) return null; + // The lowest stretch of a noisy curve reads below its mean; 1.1 puts the + // floor back at these pooled sizes, so jitter alone is not undercounted. + final noise = 1.1 * 6 * floor / _hann128Sq * nAll; + return noise < kNnDiffNoiseShareCeiling * sq ? keep : null; +} + +/// The window RMSSDs a windowed headline publishes, or null for none: the +/// usual gate first, and only when it leaves nothing, the windows on a +/// breathing line that holds still in Hz ([_steadyBreathingWindows]), which +/// publish at the confidence floor. +({List rmssds, bool byBreathing, String note})? _keptWindows( + double? acf1, + bool refused, + List rmssds, + List>> winRuns, + List meanRrMs) { + final kept = refused ? const [] : _clearedWindows(acf1, rmssds, winRuns); + if (kept.isNotEmpty) return (rmssds: kept, byBreathing: false, note: ''); + if (acf1 == null || acf1 >= kNnDiffAcf1Floor) return null; + final idx = _steadyBreathingWindows(winRuns, meanRrMs); + if (idx == null) return null; + return ( + rmssds: [for (final i in idx) rmssds[i]], + byBreathing: true, + note: ' Jitter gate failed; kept the ${idx.length} windows on a breathing ' + 'line steady in Hz while heart rate drifted, at floor confidence.', + ); +} + +/// Confidence multiplier for a measured [acf1]: 1.0 on a smooth tachogram, +/// falling linearly to 0 at [kNnDiffAcf1Floor] so confidence bottoms out +/// exactly where RMSSD is refused. 1.0 when ACF1 could not be measured. +double _acf1Quality(double? acf1) => + acf1 == null ? 1.0 : (1 - acf1 / kNnDiffAcf1Floor).clamp(0.0, 1.0); + +String _jitterNote(double acf1) => + 'rmssd_refused:acf1=${acf1.toStringAsFixed(3)} — the NN successive ' + 'differences are essentially differenced white noise (−0.5 = pure, floor ' + '$kNnDiffAcf1Floor), so RMSSD/pNN50 would measure beat-timing jitter, not ' + 'vagal tone'; + +/// Intervals [rrCoverage] counts as beat-time, ms. Wider than the cleaners' +/// 2000 ms on purpose: a 2000–2400 ms beat is a slow heart (25–30 bpm), still +/// real elapsed time; only a glitch (e.g. a 65 535 ms sentinel) must not be +/// summed, or it would fake an over-count on a short span. +const double _rrCoverageMinMs = 300; +const double _rrCoverageMaxMs = 2400; + +bool _rrCoverageCounts(double rrMs) => + rrMs >= _rrCoverageMinMs && rrMs <= _rrCoverageMaxMs; + +// ---- verbatim copy of hrv_time.dart `_cleanWindowRuns` (private there) ---- +List> _cleanWindowRuns(List rr, List ts) { + const radius = 2; + const threshold = 0.20; + // Range filter first, keeping each survivor's position in [rr] — BOTH filters + // break a run, so neither one's compaction can manufacture a difference. + final nn = []; + final at = []; + final nnTs = []; + for (var i = 0; i < rr.length; i++) { + if (rr[i] >= 300 && rr[i] <= 2000) { + nn.add(rr[i]); + at.add(i); + nnTs.add(ts[i]); + } + } + final runs = >[]; + var run = []; + var lastKept = -2; + var lastTs = 0.0; + for (var i = 0; i < nn.length; i++) { + var keep = true; + if (nn.length > radius) { + final lo = math.max(0, i - radius); + final hi = math.min(nn.length - 1, i + radius); + final neighbors = []; + for (var j = lo; j <= hi; j++) { + if (j != i) neighbors.add(nn[j]); + } + final med = neighbors.length < 2 ? null : median(neighbors); + if (med != null && med > 0) keep = (nn[i] - med).abs() / med <= threshold; + } + if (!keep) { + if (run.isNotEmpty) { + runs.add(run); + run = []; + } + continue; + } + if (run.isNotEmpty && + (at[i] != lastKept + 1 || nnTs[i] - lastTs > nn[i] + 1000.0)) { + runs.add(run); + run = []; + } + run.add(nn[i]); + lastKept = at[i]; + lastTs = nnTs[i]; + } + if (run.isNotEmpty) runs.add(run); + return runs; +} + +/// Raw-RR stream (NOT the corrected NN): a bucket is final the moment a beat of +/// a later bucket arrives. Session bounds fixed at construction. +class SessionRmssdState { + final int startSec, endSec, windowSec, minDiffsPerWindow; + int _idx = -1; + List _rr = [], _ts = []; + final List<_WinRec> recs = []; + int _seen = 0; // beats folded, in the session or not + // Session-wide coverage (rrCoverage over the in-session beats), from + // running figures: the earliest beat (first of equals) and its interval, + // the latest beat time, and the plausible RR summed in input order. + int _inN = 0; + double _loTs = 0, _loRr = 0, _hiTs = 0, _sumRr = 0; + int _implausible = 0; + + SessionRmssdState(this.startSec, this.endSec, + {this.windowSec = 300, this.minDiffsPerWindow = kMinDiffsPerRmssdWindow}); + + _WinRec _close(int idx, List winRr, List winTs) { + // Per-window over-count, as the batch judges it. + final winStart = startSec + idx * windowSec; + final winSec = math.min(winStart + windowSec, endSec) - winStart; + var bankedMs = 0.0; + var lo = 0; + for (var i = 0; i < winRr.length; i++) { + if (winTs[i] < winTs[lo]) lo = i; + if (_rrCoverageCounts(winRr[i])) bankedMs += winRr[i]; + } + final overhangMs = _rrCoverageCounts(winRr[lo]) ? winRr[lo] : 0.0; + if (bankedMs > kRrCoverageCeiling * (winSec * 1000 + overhangMs)) { + return _WinRec.dropped(idx, 1); + } + final rrRuns = [ + for (final r in _cleanWindowRuns(winRr, winTs)) + if (r.length >= 2) r + ]; + final diffRuns = [ + for (final r in rrRuns) [for (var i = 1; i < r.length; i++) r[i] - r[i - 1]] + ]; + final (ssd, nd) = _ssdOf(diffRuns); + if (nd < minDiffsPerWindow) return _WinRec.dropped(idx, 2); + return _WinRec(idx, 0, math.sqrt(ssd / nd), diffRuns, + mean([for (final r in rrRuns) ...r])!); + } + + void _step(double rr, double ts, List<_WinRec> sink) { + _seen++; + final tsSec = (ts / 1000.0).round(); + if (tsSec < startSec || tsSec >= endSec) return; + if (_inN == 0 || ts < _loTs) { + _loTs = ts; + _loRr = rr; + } + if (_inN == 0 || ts > _hiTs) _hiTs = ts; + if (_rrCoverageCounts(rr)) { + _sumRr += rr; + } else { + _implausible++; + } + _inN++; + final idx = (tsSec - startSec) ~/ windowSec; + if (idx != _idx && _rr.isNotEmpty) { + sink.add(_close(_idx, _rr, _ts)); + _rr = []; + _ts = []; + } + _idx = idx; + _rr.add(rr); + _ts.add(ts); + } + + void fold(List rr, List ts) { + for (var i = 0; i < rr.length; i++) { + _step(rr[i], ts[i], recs); + } + } + + /// `rrCoverage` of the in-session beats, or null where the batch has none. + /// The duplicate count is not kept (it needs every beat); the refusal reads + /// only the coverage. + RrCoverage? _coverage() { + if (_inN < 2) return null; + final spanSec = + (_hiTs - _loTs + (_rrCoverageCounts(_loRr) ? _loRr : 0)) / 1000.0; + if (!spanSec.isFinite || spanSec < kRrCoverageMinSpanSec) return null; + final sumSec = _sumRr / 1000.0; + return RrCoverage( + coverage: sumSec / spanSec, + sumRrSec: sumSec, + spanSec: spanSec, + beats: _inN, + implausibleBeats: _implausible, + duplicateBeats: 0, + ); + } + + /// The open bucket is closed on a copy; the state is not changed. + Metric evaluate() { + const inputs = ['rr_sleep_window']; + if (startSec <= 0 || endSec <= startSec || _seen == 0) { + return const Metric.absent( + tier: Tier.high, + inputs_used: inputs, + note: 'invalid or empty RR session window', + ); + } + if (_inN == 0) { + return const Metric.absent( + tier: Tier.high, + inputs_used: inputs, + note: 'no RR beats inside the session window', + ); + } + final cov = _coverage(); + if (cov != null && cov.overCounted) { + return Metric.absent( + tier: Tier.high, + inputs_used: inputs, + note: rrOvercountNote(cov), + ); + } + final all = [...recs]; + if (_rr.isNotEmpty) all.add(_close(_idx, _rr, _ts)); + final rmssds = []; + final runs = >[]; + final perWindow = >>[]; + final meanRr = []; + var overCountedWindows = 0, thinWindows = 0; + for (final r in all) { + if (r.kind == 1) { + overCountedWindows++; + continue; + } + if (r.kind == 2) { + thinWindows++; + continue; + } + rmssds.add(r.rmssd); + runs.addAll(r.runs); + perWindow.add(r.runs); + meanRr.add(r.meanRr); + } + final droppedWhy = [ + if (thinWindows > 0) + '$thinWindows window(s) had fewer than $minDiffsPerWindow clean ' + 'successive differences', + if (overCountedWindows > 0) + '$overCountedWindows window(s) held more beat-time than elapsed time', + ]; + final dropped = droppedWhy.isEmpty ? '' : ' (${droppedWhy.join('; ')})'; + final acf1 = nnDiffAcf1(runs); + final refused = _jitterRefused(acf1, runs); + if (rmssds.isEmpty && !refused) { + return Metric.absent( + tier: Tier.high, + inputs_used: inputs, + note: 'no valid 5-min windows for sleep-session RMSSD$dropped', + ); + } + final kept = _keptWindows(acf1, refused, rmssds, perWindow, meanRr); + if (kept == null) { + return Metric.absent( + tier: Tier.high, + inputs_used: inputs, + note: refused + ? '${_jitterNote(acf1!)}$dropped' + : '${_jitterNote(acf1!)}; no 5-min window clears it on its own' + '$dropped', + ); + } + final conf = kept.byBreathing + ? 0.3 + : ((kept.rmssds.length / 12.0).clamp(0.0, 1.0) * _acf1Quality(acf1)) + .clamp(0.3, 0.95); + return Metric( + value: mean(kept.rmssds)!, + confidence: conf, + tier: Tier.high, + inputs_used: inputs, + note: 'sleep-session HRV: mean RMSSD over cleaned 5-min windows' + '$dropped.${kept.note}', + ); + } +} + +// =========================================================================== +// 7. irregularBeatScreen over the (day-long) corrected NN +// =========================================================================== +// Aggregates (SD1/SD2 from stddev of diffs and of levels, pNN70) are sums => +// Welford, ~1e-13. The "sustained across 5-min windows" test is a COUNT of +// flagged/valid closed windows plus one open window evaluated on a copy => exact. +class IrregularScreenState { + // elementwise stream state over the corrected NN list + bool _prevKept = false; + double _prevV = 0; + // Time of the previous kept beat: a pair across a time hole is not + // successive (the batch's seam test). NaN = unknown, never within a seam. + double _prevT = double.nan; + int _nKept = 0; + // diff stats (adjacent kept pairs) + int _dN = 0; + double _dMean = 0, _dM2 = 0; + int _over = 0; + // level stats (kept beats) + int _lN = 0; + double _lMean = 0, _lM2 = 0; + // windows + double? _winStart; + List _bk = []; + List _bkAdj = []; + int _valid = 0, _flagged = 0; + + final double sd1sd2Flag, pnnThresholdMs, pnnFlagPct, windowMs, sustainedFraction; + final int minWindowBeats; + + IrregularScreenState({ + this.sd1sd2Flag = 0.70, + this.pnnThresholdMs = 70, + this.pnnFlagPct = 30, + double windowMinutes = 5, + this.minWindowBeats = 40, + this.sustainedFraction = 0.5, + }) : windowMs = windowMinutes * 60000; + + IrregularScreenState _copy() { + final c = IrregularScreenState( + sd1sd2Flag: sd1sd2Flag, + pnnThresholdMs: pnnThresholdMs, + pnnFlagPct: pnnFlagPct, + windowMinutes: windowMs / 60000, + minWindowBeats: minWindowBeats, + sustainedFraction: sustainedFraction) + .._prevKept = _prevKept + .._prevV = _prevV + .._prevT = _prevT + .._nKept = _nKept + .._dN = _dN + .._dMean = _dMean + .._dM2 = _dM2 + .._over = _over + .._lN = _lN + .._lMean = _lMean + .._lM2 = _lM2 + .._winStart = _winStart + .._bk = List.of(_bk) + .._bkAdj = List.of(_bkAdj) + .._valid = _valid + .._flagged = _flagged; + return c; + } + + void _flush() { + final diffs = [ + for (var i = 1; i < _bk.length; i++) + if (_bkAdj[i]) _bk[i] - _bk[i - 1] + ]; + // Valid on successive pairs, not beats (the batch's window rule). + if (diffs.length >= minWindowBeats - 1) { + _valid++; + final sdsd = stddev(diffs); + final sdnn = stddev(_bk); + if (sdsd != null && sdnn != null) { + final sd1 = sdsd / math.sqrt2; + final v = 2 * sdnn * sdnn - sd1 * sd1; + final sd2 = v > 0 ? math.sqrt(v) : 0.0; + if (sd2 > 0) { + final ratio = sd1 / sd2; + final over = diffs.where((d) => d.abs() > pnnThresholdMs).length; + final pnn = 100.0 * over / diffs.length; + if (ratio >= sd1sd2Flag && pnn >= pnnFlagPct) _flagged++; + } + } + } + _bk = []; + _bkAdj = []; + } + + void _add(double v, double tm) { + final kept = v >= 300 && v <= 2000; + if (kept) { + // previous INPUT element was kept and no time hole lies between them + final adjacent = _prevKept && tm - _prevT <= v + 0.5; + if (adjacent) { + final d = v - _prevV; + _dN++; + final dl = d - _dMean; + _dMean += dl / _dN; + _dM2 += dl * (d - _dMean); + if (d.abs() > pnnThresholdMs) _over++; + } + _lN++; + final dl = v - _lMean; + _lMean += dl / _lN; + _lM2 += dl * (v - _lMean); + _nKept++; + _winStart ??= tm; + if (tm - _winStart! >= windowMs) { + _flush(); + _winStart = tm; + } + _bk.add(v); + _bkAdj.add(adjacent); + } + _prevKept = kept; + if (kept) { + _prevV = v; + _prevT = tm; + } + } + + void fold(List nn, List t) { + for (var i = 0; i < nn.length; i++) { + _add(nn[i], t[i]); + } + } + + Metric evaluate(List tailNn, List tailT, + {double artifactFraction = 0.0, + int minBeats = irregularScreenMinBeats, + double maxArtifact = 0.30}) { + const inputs = ['rr_cleaned']; + final c = _copy()..fold(tailNn, tailT); + // The batch gates on the successive pairs it averages, not on beats. + if (c._dN < minBeats - 1) { + return const Metric.absent( + tier: Tier.estimate, + inputs_used: inputs, + note: 'too few clean beats for an irregular-rhythm screen', + ); + } + if (artifactFraction > maxArtifact) { + return Metric.absent( + tier: Tier.estimate, + inputs_used: inputs, + note: 'artifact fraction ${(artifactFraction * 100).round()}% > ' + '${(maxArtifact * 100).round()}% — screen suppressed on noisy RR', + ); + } + final sdsd = c._dN < 2 ? null : math.sqrt(c._dM2 / (c._dN - 1)); + final sdnn = c._lN < 2 ? null : math.sqrt(c._lM2 / (c._lN - 1)); + if (sdsd == null || sdnn == null) { + return const Metric.absent( + tier: Tier.estimate, + inputs_used: inputs, + note: 'no successive clean beats to build a Poincare plot from', + ); + } + final sd1 = sdsd / math.sqrt2; + final v = 2 * sdnn * sdnn - sd1 * sd1; + final sd2 = v > 0 ? math.sqrt(v) : 0.0; + if (sd2 <= 0) { + return const Metric.absent( + tier: Tier.estimate, + inputs_used: inputs, + note: 'no long-term variability (SD2 = 0) — the SD1/SD2 ratio is ' + 'undefined, not "perfectly regular"', + ); + } + final ratio = sd1 / sd2; + final pnnPct = c._dN == 0 ? 0.0 : 100.0 * c._over / c._dN; + final aggregateHigh = ratio >= sd1sd2Flag && pnnPct >= pnnFlagPct; + var flag = false; + if (aggregateHigh) { + c._flush(); // the open window closes at end of data + flag = c._valid != 0 && c._flagged / c._valid >= sustainedFraction; + } + final conf = (c._nKept / 5000.0 * (1 - artifactFraction)).clamp(0.2, 0.9); + return Metric( + value: IrregularRhythm( + sd1: sd1, + sd2: sd2, + sd1sd2: ratio, + pnnPct: pnnPct, + nBeats: c._nKept, + flag: flag, + ), + confidence: conf, + tier: Tier.estimate, + inputs_used: inputs, + note: 'irregular-rhythm SCREEN (not a diagnosis): Poincaré SD1/SD2 + pNN' + '${pnnThresholdMs.round()}. PRV not ECG — wrist pulse misses P-waves. ' + 'Discuss with a clinician only if you have symptoms.', + ); + } + + Map toJson() => { + 'prevKept': _prevKept, + 'prevV': _prevV, + 'prevT': _prevT.isFinite ? _prevT : null, + 'nKept': _nKept, + 'dN': _dN, + 'dMean': _dMean, + 'dM2': _dM2, + 'over': _over, + 'lN': _lN, + 'lMean': _lMean, + 'lM2': _lM2, + 'winStart': _winStart, + 'bk': _bk, + 'bkAdj': [for (final a in _bkAdj) a ? 1 : 0], + 'valid': _valid, + 'flagged': _flagged, + }; + + factory IrregularScreenState.fromJson(Map j) { + final s = IrregularScreenState(); + s._prevKept = j['prevKept'] as bool; + s._prevV = (j['prevV'] as num).toDouble(); + s._prevT = (j['prevT'] as num?)?.toDouble() ?? double.nan; + s._nKept = j['nKept'] as int; + s._dN = j['dN'] as int; + s._dMean = (j['dMean'] as num).toDouble(); + s._dM2 = (j['dM2'] as num).toDouble(); + s._over = j['over'] as int; + s._lN = j['lN'] as int; + s._lMean = (j['lMean'] as num).toDouble(); + s._lM2 = (j['lM2'] as num).toDouble(); + s._winStart = (j['winStart'] as num?)?.toDouble(); + s._bk = [for (final x in j['bk'] as List) (x as num).toDouble()]; + s._bkAdj = [for (final x in j['bkAdj'] as List) x == 1]; + s._valid = j['valid'] as int; + s._flagged = j['flagged'] as int; + return s; + } +} diff --git a/tool/incremental/lag_stats.dart b/tool/incremental/lag_stats.dart new file mode 100644 index 0000000..ebd0506 --- /dev/null +++ b/tool/incremental/lag_stats.dart @@ -0,0 +1,49 @@ +// Settle horizon in the wild: how many beats / seconds behind the newest beat +// is the last SETTLED output, distribution over 1-beat folds. +import 'package:openstrap_analytics/onehz.dart' + hide RrCorrector, RrSettled, RrSnapshot, IrregularScreenState; +import 'rr_stream.dart'; +import 'synth.dart'; + +void report(String name, RrData s) { + final c = RrCorrector(); + final lagBeats = [], lagSec = []; + var ce0 = 0; + for (var i = 0; i < s.length; i++) { + c.fold([s.rr[i]], tsMs: [s.ts[i]]); + final settled = c.settledBeats; + if (settled > ce0 || true) { + lagBeats.add(i + 1 - settled); + lagSec.add(settled == 0 ? 0 : (s.ts[i] - s.ts[settled - 1]) / 1000.0); + } + ce0 = settled; + } + lagBeats.sort(); + lagSec.sort(); + double q(List x, double p) => x[((x.length - 1) * p).round()].toDouble(); + print('$name: beats=${s.length} lag beats p50=${q(lagBeats, .5)} p95=${q(lagBeats, .95)} ' + 'p99=${q(lagBeats, .99)} max=${lagBeats.last} lag s p50=${q(lagSec, .5).toStringAsFixed(0)} ' + 'p99=${q(lagSec, .99).toStringAsFixed(0)} max=${lagSec.last.toStringAsFixed(0)} ' + 'state buffer max beats: ${c.bufferedBeats}'); +} + +void main() { + report('real night', realNightRr()!); + report('synthetic day (default artefacts)', synthRr(const SynthConfig(seed: 3, hours: 8))); + report('synthetic dirty', synthRr(const SynthConfig( + seed: 4, hours: 3, ectopicPerMin: 3, missedPerMin: 2, extraPerMin: 2, noiseRunPerMin: 1, gapPerHour: 8))); + // one-shot micro bench, 3 runs each, 32k beats + final n = realNightRr()!; + final sw = Stopwatch(); + final o = [], r = []; + for (var k = 0; k < 3; k++) { + sw..reset()..start(); + correctRr(n.rr, rrTsMs: n.ts); + o.add(sw.elapsedMilliseconds); + sw..reset()..start(); + final c = RrCorrector()..fold(n.rr, tsMs: n.ts); + c.snapshot(); + r.add(sw.elapsedMilliseconds); + } + print('32k-beat night one-shot ms: oracle $o streaming $r'); +} diff --git a/tool/incremental/oracle_util.dart b/tool/incremental/oracle_util.dart new file mode 100644 index 0000000..6f12de6 --- /dev/null +++ b/tool/incremental/oracle_util.dart @@ -0,0 +1,40 @@ +// Shared helpers for the incremental oracle tests. +import 'dart:convert'; +import 'dart:math' as math; + +/// JSON round trip through real text (what a SQLite blob would hold). +Map jsonRoundTrip(Map m) => + jsonDecode(jsonEncode(m)) as Map; + +/// Random chunk boundaries over [0, n): sizes drawn from [sizes] (a mix of 1, +/// tiny, one-window, and huge) so every alignment against the 91/181-beat +/// windows is hit. +List randomCuts(math.Random r, int n, + {List sizes = const [1, 2, 3, 7, 45, 46, 90, 91, 92, 180, 181, 500, 1500]}) { + final cuts = []; + var at = 0; + while (at < n) { + final s = sizes[r.nextInt(sizes.length)]; + at = math.min(n, at + 1 + r.nextInt(s)); + cuts.add(at); + } + return cuts; +} + +/// Cuts that fall every [sec] seconds of the beat clock: what a 15-minute +/// derive pass sees. [tsMs] sorted-ish epoch ms. +List timeCuts(List tsMs, double sec) { + final cuts = []; + if (tsMs.isEmpty) return cuts; + var next = tsMs.first + sec * 1000; + for (var i = 0; i < tsMs.length; i++) { + if (tsMs[i] >= next) { + cuts.add(i); + while (next <= tsMs[i]) { + next += sec * 1000; + } + } + } + cuts.add(tsMs.length); + return cuts; +} diff --git a/tool/incremental/resp_incr.dart b/tool/incremental/resp_incr.dart new file mode 100644 index 0000000..86e5a9e --- /dev/null +++ b/tool/incremental/resp_incr.dart @@ -0,0 +1,861 @@ +// RESEARCH PROTOTYPES — incremental breathing-rate and spectral estimators. +// +// All three Welch-style estimators (rsaRespRate, hrvFreq, and the per-30-min +// bins behind BRV) have the same structure: the answer is an aggregate over +// SUB-WINDOWS that are each a pure function of the beats inside them. A +// sub-window is final as soon as a beat past its end is settled, so the state +// is "list of finished sub-window results + the beats since the next +// unfinished sub-window". Aggregates (median, consensus, sum/k) are re-run on +// that list with the batch's own arithmetic => bit-identical. +import 'dart:math' as math; + +import 'package:openstrap_analytics/onehz.dart' + hide RrCorrector, RrSettled, RrSnapshot, IrregularScreenState; + +int _lowerBound(List a, double v) { + var lo = 0, hi = a.length; + while (lo < hi) { + final m = (lo + hi) >> 1; + if (a[m] < v) { + lo = m + 1; + } else { + hi = m; + } + } + return lo; +} + +double _powerAt(LombScargle ls, double fHz) { + double best = 0; + double bestDist = double.infinity; + for (final pt in ls.spectrum) { + final d = (pt.freqHz - fHz).abs(); + if (d < bestDist) { + bestDist = d; + best = pt.psd; + } + } + return best; +} + +// =========================================================================== +// rsaRespRate +// =========================================================================== +class _SegCounts { + final List peaks = [], peakHz = [], peakPwr = []; + int atCeiling = 0, belowBand = 0, thin = 0, gappy = 0; + // The at-ceiling sub-windows' own ceilings, br/min (min and max). + double ceilingLo = double.infinity, ceilingHi = 0.0; + _SegCounts copy() { + final c = _SegCounts(); + c.peaks.addAll(peaks); + c.peakHz.addAll(peakHz); + c.peakPwr.addAll(peakPwr); + c.atCeiling = atCeiling; + c.belowBand = belowBand; + c.thin = thin; + c.gappy = gappy; + c.ceilingLo = ceilingLo; + c.ceilingHi = ceilingHi; + return c; + } +} + +/// [a] and [b] sorted by [double.compareTo] (as `median` sorts), merged. +List _mergeSorted(List a, List b) { + if (b.isEmpty) return a; + if (a.isEmpty) return b; + final out = List.filled(a.length + b.length, 0); + var i = 0, j = 0, k = 0; + while (i < a.length && j < b.length) { + out[k++] = a[i].compareTo(b[j]) <= 0 ? a[i++] : b[j++]; + } + while (i < a.length) { + out[k++] = a[i++]; + } + while (j < b.length) { + out[k++] = b[j++]; + } + return out; +} + +/// Usable sub-windows at which the batch's confidence reaches full weight +/// (resp_rate.dart `_rsaFullWeightSubwindows`). +const double _rsaFullWeightSubwindows = 23; + +class RsaWelchState { + static const double segSec = rsaSegmentSec; // 300 + // buffer of settled beats (ms and s), from the next unfinished sub-window + List _tMs = [], _ts = [], _nn = []; + // Every NN folded, sorted: the whole-input ceiling is the MEDIAN beat + // interval, which needs all of them (upstream's gap-free ceiling). + List _sorted = []; + // A non-finite beat time was folded: the batch abstains on it. + bool _nonFinite = false; + double? _first; + int _count = 0; + double _lastT = 0; + bool _folding = false; + double _next = 0; + _SegCounts _sc = _SegCounts(); + + RsaWelchState(); + + int get bufferedBeats => _nn.length; + int get foldedSegments => + _sc.peaks.length + _sc.atCeiling + _sc.belowBand + _sc.thin + _sc.gappy; + + void fold(List nn, List tMs) { + _sorted = _mergeSorted(_sorted, [...nn]..sort()); + for (final t in tMs) { + if (!(t / 1000.0).isFinite) _nonFinite = true; + } + if (_nonFinite) { + // Nothing settles after a non-finite time (the walk below would never + // end); the batch abstains, and so does [evaluate]. + _nn.addAll(nn); + _tMs.addAll(tMs); + _ts.addAll([for (final t in tMs) t / 1000.0]); + _count += nn.length; + return; + } + for (var i = 0; i < nn.length; i++) { + _nn.add(nn[i]); + _tMs.add(tMs[i]); + final s = tMs[i] / 1000.0; + _ts.add(s); + _first ??= s; + _lastT = s; + _count++; + } + if (!_folding && _count > 0 && _lastT - _first! >= 2 * segSec) { + _folding = true; + _next = _first!; + } + if (_folding) { + while (_next + segSec <= _lastT) { + _evalSeg(_next, _ts, _nn, _sc); + _next += segSec / 2; + } + final cut = _lowerBound(_ts, _next); + if (cut > 0) { + _tMs = _tMs.sublist(cut); + _ts = _ts.sublist(cut); + _nn = _nn.sublist(cut); + } + } + } + + static void _evalSeg( + double s, List tSec, List nnMs, _SegCounts c) { + final lo = _lowerBound(tSec, s); + final hi = _lowerBound(tSec, s + segSec); + final k = hi - lo; + final segT = tSec.sublist(lo, hi); + final segNn = nnMs.sublist(lo, hi); + // Time-complete = the kept beats COVER >= 80 % of the sub-window (sum of + // NN, not first-to-last span), judged before the beat count. + final coveredSec = segNn.fold(0, (a, b) => a + b) / 1000.0; + if (coveredSec < segSec * 0.8) { + c.gappy++; + return; + } + if (k < 30) { + c.thin++; + return; + } + final segSpan = segT.last - segT.first; + // The sub-window's ceiling from its median beat interval, gap-free. + final segHi = rsaCeilingHz(median(segNn)! / 1000.0); + if (segHi < rsaHiHz) { + c.belowBand++; + return; + } + final grid = math.max(64, ((segHi - rsaLoHz) * 4 * segSpan).ceil()); + final ls = lombScargle(segT, segNn, freqGrid(rsaLoHz, segHi, grid)); + if (ls == null) { + c.thin++; + return; + } + final pk = ls.peakFreq(rsaLoHz, segHi); + if (pk == null) { + c.thin++; + return; + } + if (pk >= segHi - (segHi - rsaLoHz) / (grid - 1)) { + c.atCeiling++; + c.ceilingLo = math.min(c.ceilingLo, segHi * 60); + c.ceilingHi = math.max(c.ceilingHi, segHi * 60); + return; + } + c.peaks.add(pk * 60.0); + c.peakHz.add(pk); + c.peakPwr.add(_powerAt(ls, pk)); + } + + /// `rsaRespRate(settled ++ tail, artifactFraction: ...)`. + Metric evaluate(List tailNn, List tailTMs, + {required double artifactFraction, + double tolBrpm = 2.0, + double minConsensus = 0.5, + double maxArtifact = 0.30}) { + const inputs = ['rr_cleaned', 'beat_times']; + final n = _count + tailNn.length; + if (n < 20) { + return const Metric.absent( + tier: Tier.high, + inputs_used: inputs, + note: 'too few beats for an RSA spectral estimate (need ≥20)', + ); + } + final allNn = [..._nn, ...tailNn]; + final allTMs = [..._tMs, ...tailTMs]; + final allTs = [..._ts, for (final t in tailTMs) t / 1000.0]; + final first = _first ?? allTs.first; + final last = allTs.last; + final spanSec = last - first; + if (!_folding) { + // short record (< 2 sub-windows): the oracle's segSec = span/2 rule + // makes every window length depend on the total. Buffer holds all + // beats here (no folding yet, and a non-finite time never starts it), + // so just call the oracle. + return rsaRespRate(allNn, allTMs, artifactFraction: artifactFraction); + } + if (artifactFraction > maxArtifact) { + return Metric.absent( + tier: Tier.high, + inputs_used: inputs, + note: 'artifact fraction ${round6(artifactFraction)} > gate ' + '— RSA peak unreliable', + ); + } + if (_nonFinite || allTs.any((t) => !t.isFinite)) { + return const Metric.absent( + tier: Tier.high, + inputs_used: inputs, + note: 'non-finite beat time', + ); + } + if (spanSec <= 0) { + return const Metric.absent( + tier: Tier.high, + inputs_used: inputs, + note: 'degenerate beat times', + ); + } + // The beat interval is the median NN over settled ++ tail (gap-free). + final beatSec = + percentile(_mergeSorted(_sorted, [...tailNn]..sort()), 50)! / 1000.0; + final hiHz = rsaCeilingHz(beatSec); + if (hiHz < rsaHiHz) { + return Metric.absent( + tier: Tier.high, + inputs_used: inputs, + note: 'sleeping heart rate ${round6(60 / beatSec)} bpm (median beat ' + 'interval ${round6(beatSec)} s) can resolve breathing from beat ' + 'timing only up to ${round6(hiHz * 60)} br/min, below the top of the ' + 'normal 9–24 br/min band, so any peak could be an alias; rate ' + 'withheld', + ); + } + // provisional sub-windows on a copy: those that end after the settled edge + final sc = _sc.copy(); + var s = _next; + while (s + segSec <= last + 1e-9) { + _evalSeg(s, allTs, allNn, sc); + s += segSec / 2; + } + return _finish(sc, segSec, hiHz, artifactFraction, tolBrpm, minConsensus); + } + + static Metric _finish(_SegCounts c, double segSec, double hiHz, + double artifactFraction, double tolBrpm, double minConsensus) { + const inputs = ['rr_cleaned', 'beat_times']; + final peaks = c.peaks, peakHz = c.peakHz, peakPwr = c.peakPwr; + final dropped = c.atCeiling + c.belowBand + c.thin + c.gappy; + final total = peaks.length + dropped; + if (peaks.length < 3) { + // The largest counter is the reason; ties go to the earlier entry. + final reasons = <(int, String)>[ + ( + c.belowBand, + '${c.belowBand} of $total sub-windows had a beat rate too low to ' + 'cover the HF band (any peak could be an alias)' + ), + ( + c.atCeiling, + c.atCeiling == 0 + ? '' + : '${c.atCeiling} of $total sub-windows peaked at/above their ' + 'resolvable ceiling (${round6(c.ceilingLo)}' + '${c.ceilingHi > c.ceilingLo ? '–${round6(c.ceilingHi)}' : ''} ' + 'br/min)' + ), + ( + c.gappy, + '${c.gappy} of $total ${round6(segSec)}s sub-windows had less than ' + '80 % of their time covered by clean beats (sensor gaps or ' + 'rejected beats)' + ), + ( + c.thin, + '${c.thin} of $total sub-windows had fewer than 30 beats or no ' + 'spectral peak' + ), + ]; + var top = reasons.first; + for (final r in reasons.skip(1)) { + if (r.$1 > top.$1) top = r; + } + final why = + dropped == 0 ? 'only ${peaks.length} usable sub-windows' : top.$2; + return Metric.absent( + tier: Tier.high, + inputs_used: inputs, + note: 'no stable HF respiratory peak resolved — $why', + ); + } + final medBrpm0 = median(peaks)!; + var within = 0; + for (final p in peaks) { + if ((p - medBrpm0).abs() <= tolBrpm) within++; + } + final consensus = within / peaks.length; + if (consensus < minConsensus) { + return Metric.absent( + tier: Tier.high, + inputs_used: inputs, + note: 'HF peak unstable across the window\'s own sub-windows — only ' + '$within of ${peaks.length} ${round6(segSec)}s sub-windows fall ' + 'within ${round6(tolBrpm)} br/min of the median; withheld', + ); + } + var best = 0; + for (var i = 1; i < peaks.length; i++) { + if ((peaks[i] - medBrpm0).abs() < (peaks[best] - medBrpm0).abs()) best = i; + } + final brpm = peaks[best]; + final conf = ((1 - artifactFraction) * + consensus * + (peaks.length / _rsaFullWeightSubwindows).clamp(0.0, 1.0)) + .clamp(0.2, 0.9); + return Metric( + value: RespEstimate(brpm, peakHz[best], peakPwr[best], 'rsa', + usableSubwindows: peaks.length, subwindows: total), + confidence: conf, + tier: Tier.high, + inputs_used: inputs, + note: 'RSA HF-peak respiratory rate (Lomb-Scargle on native beat times, ' + 'median of ${peaks.length} ${round6(segSec)}s sub-windows, $within of ' + 'them within ${round6(tolBrpm)} br/min of it — brpm, peak_hz and power ' + 'all come from the medoid sub-window); PRV-derived; this window could ' + 'resolve up to ${round6(hiHz * 60)} br/min', + ); + } + + Map toJson() => { + 'tMs': _tMs, + 'nn': _nn, + 'sorted': _sorted, + 'nonFinite': _nonFinite, + 'first': _first, + 'count': _count, + 'lastT': _lastT, + 'folding': _folding, + 'next': _next, + 'peaks': _sc.peaks, + 'peakHz': _sc.peakHz, + 'peakPwr': _sc.peakPwr, + 'atCeiling': _sc.atCeiling, + 'belowBand': _sc.belowBand, + 'thin': _sc.thin, + 'gappy': _sc.gappy, + 'ceilingLo': _sc.ceilingLo.isFinite ? _sc.ceilingLo : null, + 'ceilingHi': _sc.ceilingHi, + }; + + factory RsaWelchState.fromJson(Map j) { + List dl(Object? o) => [for (final x in o as List) (x as num).toDouble()]; + final s = RsaWelchState(); + s._tMs = dl(j['tMs']); + s._ts = [for (final t in s._tMs) t / 1000.0]; + s._nn = dl(j['nn']); + s._sorted = dl(j['sorted']); + s._nonFinite = j['nonFinite'] as bool; + s._first = (j['first'] as num?)?.toDouble(); + s._count = j['count'] as int; + s._lastT = (j['lastT'] as num).toDouble(); + s._folding = j['folding'] as bool; + s._next = (j['next'] as num).toDouble(); + s._sc.peaks.addAll(dl(j['peaks'])); + s._sc.peakHz.addAll(dl(j['peakHz'])); + s._sc.peakPwr.addAll(dl(j['peakPwr'])); + s._sc.atCeiling = j['atCeiling'] as int; + s._sc.belowBand = j['belowBand'] as int; + s._sc.thin = j['thin'] as int; + s._sc.gappy = j['gappy'] as int; + s._sc.ceilingLo = (j['ceilingLo'] as num?)?.toDouble() ?? double.infinity; + s._sc.ceilingHi = (j['ceilingHi'] as num).toDouble(); + return s; + } +} + +// =========================================================================== +// _respPerWindow (feeds BRV): 30-min bins of NN from the first NN beat +// =========================================================================== +class RespWindowsState { + final double windowMs; + final int minBeats; + double? _t0; + int _bin = 0; + RsaWelchState _open = RsaWelchState(); + int _openCount = 0; + final List closed = []; + + RespWindowsState({this.windowMs = 1800000.0, this.minBeats = 60}); + + void _closeOpen(List sink, RsaWelchState st, int count) { + if (count < minBeats) return; + final r = st.evaluate(const [], const [], artifactFraction: 0.0); + final b = r.present ? r.value!.brpm : null; + if (b != null) sink.add(b); + } + + void fold(List nn, List tMs) { + var from = 0; + for (var i = 0; i < nn.length; i++) { + _t0 ??= tMs[i]; + final idx = ((tMs[i] - _t0!) / windowMs).floor(); + if (idx != _bin) { + _open.fold(nn.sublist(from, i), tMs.sublist(from, i)); + _openCount += i - from; + from = i; + _closeOpen(closed, _open, _openCount); + _open = RsaWelchState(); + _openCount = 0; + _bin = idx; + } + } + _open.fold(nn.sublist(from), tMs.sublist(from)); + _openCount += nn.length - from; + } + + /// `_respPerWindow(settled ++ tail)`. + List evaluate(List tailNn, List tailTMs) { + final out = [...closed]; + if (tailNn.isEmpty) { + _closeOpen(out, _open, _openCount); + return out; + } + // tail beats may open new bins: split by bin index + var curIdx = _bin; + var curSettledOpen = _open; + var curCount = _openCount; + var from = 0; + var useCopy = true; // the first bin continues the settled open bin + for (var i = 0; i < tailNn.length; i++) { + final idx = ((tailTMs[i] - (_t0 ?? tailTMs[0])) / windowMs).floor(); + if (idx != curIdx) { + _flushTail(out, curSettledOpen, curCount, useCopy, tailNn.sublist(from, i), + tailTMs.sublist(from, i)); + useCopy = false; + curSettledOpen = RsaWelchState(); + curCount = 0; + curIdx = idx; + from = i; + } + } + _flushTail(out, curSettledOpen, curCount, useCopy, tailNn.sublist(from), + tailTMs.sublist(from)); + return out; + } + + void _flushTail(List out, RsaWelchState st, int count, bool settled, + List nn, List t) { + final total = count + nn.length; + if (total < minBeats) return; + final r = st.evaluate(nn, t, artifactFraction: 0.0); + final b = r.present ? r.value!.brpm : null; + if (b != null) out.add(b); + } +} + +// =========================================================================== +// hrvFreq (LF/HF): Welch-averaged Lomb-Scargle band powers, 4 bands +// =========================================================================== +class _Band { + final double lo, hi, segSec, step; + final List grid; + double sum = 0; + int k = 0; + double next = 0; + bool started = false; + _Band(this.lo, this.hi, double oversample) + : segSec = 10.0 / lo, + step = (10.0 / lo) / 2, + grid = _gridFor(lo, hi, 10.0 / lo, oversample); + static List _gridFor(double lo, double hi, double seg, double os) { + final df = 1.0 / seg / os; + final nGrid = ((hi - lo) / df).ceil() + 1; + return freqGrid(lo, hi, nGrid); + } +} + +class HrvFreqState { + final double oversample; + + /// ULF needs a 33 333 s segment: it can only ever resolve on a record longer + /// than 9.26 h, and while unresolved it forces the WHOLE record to be kept. + /// With [includeUlf] false the state stays small (VLF buffer ~1.5 segments + /// = 83 min) but is only valid while the record spans < 33 333 s; beyond + /// that [evaluate] throws and the caller must use the batch function. + final bool includeUlf; + final List<_Band> _bands; + List _ts = [], _y = []; + double? _first; + int _count = 0; + double _lastT = 0; + + HrvFreqState({this.oversample = 4.0, this.includeUlf = true}) + : _bands = [ + _Band(0.0003, 0.003, oversample), + _Band(0.003, 0.04, oversample), + _Band(0.04, 0.15, oversample), + _Band(0.15, 0.40, oversample), + ]; + + int get bufferedBeats => _ts.length; + + /// One sub-window of one band; returns the band power or null (skipped). + static double? _seg(_Band b, double start, List tSec, List y) { + final end = start + b.segSec; + final lo = _lowerBound(tSec, start); + final hi = _lowerBound(tSec, end); + final ts = tSec.sublist(lo, hi); + final ys = y.sublist(lo, hi); + if (ts.length < 16 || ts.last - ts.first < b.segSec * 0.8) return null; + var maxGap = 0.0; + for (var i = 1; i < ts.length; i++) { + final g = ts[i] - ts[i - 1]; + if (g > maxGap) maxGap = g; + } + if (maxGap > b.segSec * 0.2) return null; + final ls = lombScargle(ts, ys, b.grid); + if (ls == null) return null; + final p = ls.bandPower(b.lo, b.hi); + if (!p.isFinite) return null; + return p; + } + + void fold(List nn, List tMs) { + for (var i = 0; i < nn.length; i++) { + _y.add(nn[i]); + final s = tMs[i] / 1000.0; + _ts.add(s); + _first ??= s; + _lastT = s; + _count++; + } + if (_count == 0) return; + for (final b in _bands) { + if (!includeUlf && identical(b, _bands[0])) continue; + if (!b.started) { + b.started = true; + b.next = _first!; + } + while (b.next + b.segSec <= _lastT) { + final p = _seg(b, b.next, _ts, _y); + if (p != null) { + b.sum += p; + b.k++; + } + b.next += b.step; + } + } + var minNext = double.infinity; + for (final b in _bands) { + if (!includeUlf && identical(b, _bands[0])) continue; + if (b.next < minNext) minNext = b.next; + } + final cut = _lowerBound(_ts, minNext); + if (cut > 0) { + _ts = _ts.sublist(cut); + _y = _y.sublist(cut); + } + } + + double? _band(_Band b, List tSec, List y, int count, + double first, double last) { + if (count < 16) return null; + final span = last - first; + if (span < b.segSec) return null; + var sum = b.sum; + var k = b.k; + var start = b.started ? b.next : first; + while (start + b.segSec <= last) { + final p = _seg(b, start, tSec, y); + if (p != null) { + sum += p; + k++; + } + start += b.step; + } + return k == 0 ? null : sum / k; + } + + Metric evaluate(List tailNn, List tailTMs, + {required double artifactFraction, double hfArtifactGate = 0.15}) { + const inputs = ['rr_cleaned', 'beat_times']; + final n = _count + tailNn.length; + if (n < 16) { + return const Metric.absent( + tier: Tier.high, + inputs_used: inputs, + note: 'too few beats for a spectral estimate', + ); + } + final ts = [..._ts, for (final t in tailTMs) t / 1000.0]; + final y = [..._y, ...tailNn]; + final first = _first ?? ts.first; + final last = ts.last; + final spanSec = last - first; + if (spanSec <= 0) { + return const Metric.absent( + tier: Tier.high, + inputs_used: inputs, + note: 'degenerate beat times', + ); + } + if (!includeUlf && spanSec >= _bands[0].segSec) { + throw StateError('ULF is resolvable (span >= ${_bands[0].segSec} s): ' + 'this state excludes it; use hrvFreq() or includeUlf: true'); + } + final ulf = includeUlf ? _band(_bands[0], ts, y, n, first, last) : null; + final vlf = _band(_bands[1], ts, y, n, first, last); + final lf = _band(_bands[2], ts, y, n, first, last); + final hfRaw = _band(_bands[3], ts, y, n, first, last); + if (lf == null && hfRaw == null) { + return const Metric.absent( + tier: Tier.high, + inputs_used: inputs, + note: 'record too short to resolve any HRV band', + ); + } + final hfGated = artifactFraction > hfArtifactGate; + final hf = hfGated ? null : hfRaw; + double? lfhf, nuLf, nuHf; + if (lf != null && hf != null && (lf + hf) > 0) { + lfhf = hf == 0 ? null : lf / hf; + nuLf = 100.0 * lf / (lf + hf); + nuHf = 100.0 * hf / (lf + hf); + } + final bands = {'ulf': ulf, 'vlf': vlf, 'lf': lf, 'hf': hfRaw}; + final resolved = [ + for (final e in bands.entries) + if (e.value != null) e.key + ]; + final total = (hfGated || lf == null || hfRaw == null) + ? null + : [for (final b in resolved) bands[b]!].reduce((a, b) => a + b); + final conf = ((1 - artifactFraction) * (hfGated ? 0.6 : 0.9)).clamp(0.2, 0.9); + return Metric( + value: HrvFreq( + ulf: ulf, + vlf: vlf, + lf: lf, + hf: hf, + total: total, + totalBands: total == null ? null : resolved, + lfhf: lfhf, + nuLf: nuLf, + nuHf: nuHf, + hfGated: hfGated, + ), + confidence: conf, + tier: Tier.high, + inputs_used: inputs, + note: hfGated + ? 'HF suppressed: artifact fraction ${round6(artifactFraction)} ' + '> gate — LF/VLF reported, HF/LF-HF/nu/total withheld' + : 'PRV spectrum; HF band quantization-limited at 1 Hz', + ); + } + + Map toJson() => { + 'ts': _ts, + 'y': _y, + 'first': _first, + 'count': _count, + 'lastT': _lastT, + 'bands': [ + for (final b in _bands) + {'sum': b.sum, 'k': b.k, 'next': b.next, 'started': b.started} + ], + }; + + factory HrvFreqState.fromJson(Map j, + {double oversample = 4.0, bool includeUlf = true}) { + final s = HrvFreqState(oversample: oversample, includeUlf: includeUlf); + s._ts = [for (final x in j['ts'] as List) (x as num).toDouble()]; + s._y = [for (final x in j['y'] as List) (x as num).toDouble()]; + s._first = (j['first'] as num?)?.toDouble(); + s._count = j['count'] as int; + s._lastT = (j['lastT'] as num).toDouble(); + final bs = j['bands'] as List; + for (var i = 0; i < s._bands.length; i++) { + final m = bs[i] as Map; + s._bands[i] + ..sum = (m['sum'] as num).toDouble() + ..k = m['k'] as int + ..next = (m['next'] as num).toDouble() + ..started = m['started'] as bool; + } + return s; + } +} + +// =========================================================================== +// edge `dayRespCurve`: 3-min RSA windows every 5 min, stillness-gated +// =========================================================================== +// Right-context 0 in RR, but the stillness gate reads accelerometer seconds up +// to ceil(ts[i]/1000), so an attempt WAITS for the accel watermark. Exact and +// append-only once an attempt has run. +class DayRespCurveState { + final double cut; + List _ts = [], _rr = []; + int _head = 0; + int _gated = 0; + double _lastEmit = -1e18; + final List> _out = []; + final List _rowSec = []; + final List _rowQuiet = []; + final List _pendRr = [], _pendTs = []; + int attempts = 0; + + DayRespCurveState({this.cut = 0.02}); + + List> get curve => _gated < 60 ? const [] : _out; + + static bool _present(double ax, double ay, double az) { + final magSq = ax * ax + ay * ay + az * az; + return magSq > 0 && magSq <= 16.0; + } + + void fold(List rrMs, List rrTsMs, List accTs, + List ax, List ay, List az, int watermarkSec) { + for (var i = 0; i < accTs.length; i++) { + var q = false; + if (_present(ax[i], ay[i], az[i])) { + final mag = + math.sqrt(ax[i] * ax[i] + ay[i] * ay[i] + az[i] * az[i]); + q = (mag - 1.0).abs() <= cut; + } + _rowSec.add(accTs[i]); + _rowQuiet.add(q); + } + _pendRr.addAll(rrMs); + _pendTs.addAll(rrTsMs); + var used = 0; + for (; used < _pendRr.length; used++) { + final v = _pendRr[used]; + if (!(v >= 300 && v <= 2000)) continue; + final t = _pendTs[used]; + // would this beat trigger an attempt? decide on the ring + this beat + final i = _ts.length; // index it will take + var head = _head; + // while (ts[i]-ts[lo] > win) lo++ — evaluated with the beat appended + final tsTmp = t; + while (head < i && tsTmp - _ts[head] > 180000.0) { + head++; + } + final attempt = (i - head >= 30) && (tsTmp - _lastEmit > 300000); + if (attempt) { + final hiSec = (tsTmp / 1000).ceil(); + if (watermarkSec < hiSec) break; // accel for this window not here yet + } + _ts.add(t); + _rr.add(v); + _gated++; + _head = head; + if (attempt) _attempt(i); + } + _pendRr.removeRange(0, used); + _pendTs.removeRange(0, used); + // trim the rings + if (_head > 256) { + _ts = _ts.sublist(_head); + _rr = _rr.sublist(_head); + _head = 0; + } + if (_ts.isNotEmpty) { + final keepSec = (_ts[_head] / 1000).floor() - 5; + var k = 0; + while (k < _rowSec.length && _rowSec[k] < keepSec) { + k++; + } + if (k > 0) { + _rowSec.removeRange(0, k); + _rowQuiet.removeRange(0, k); + } + } + } + + void _attempt(int i) { + final loSec = (_ts[_head] / 1000).floor(); + final hiSec = (_ts[i] / 1000).ceil(); + var still = 0; + for (var r = 0; r < _rowSec.length; r++) { + if (_rowSec[r] >= loSec && _rowSec[r] < hiSec && _rowQuiet[r]) still++; + } + final spanSec = hiSec - loSec; + double? brpm; + if (!(spanSec <= 0 || still < 0.9 * spanSec)) { + attempts++; + final nn = _rr.sublist(_head, i + 1); + final t0 = _ts[_head]; + final nnt = [for (var k = _head; k <= i; k++) _ts[k] - t0]; + final est = rsaRespRate(nn, nnt, artifactFraction: 0.15); + brpm = est.present ? est.value!.brpm : null; + } + _lastEmit = _ts[i]; + if (brpm != null) { + _out.add({ + 't': (_ts[i] / 1000).round(), + 'v': double.parse(brpm.toStringAsFixed(1)), + }); + } + } + + int get pending => _pendRr.length; + int get ringBeats => _ts.length - _head; + + Map toJson() => { + 'ts': _ts.sublist(_head), + 'rr': _rr.sublist(_head), + 'gated': _gated, + 'lastEmit': _lastEmit, + 'out': _out, + 'rowSec': _rowSec, + 'rowQuiet': [for (final q in _rowQuiet) q ? 1 : 0], + 'pendRr': _pendRr, + 'pendTs': _pendTs, + }; + + factory DayRespCurveState.fromJson(Map j, {double cut = 0.02}) { + List dl(Object? o) => [for (final x in o as List) (x as num).toDouble()]; + final s = DayRespCurveState(cut: cut); + s._ts = dl(j['ts']); + s._rr = dl(j['rr']); + s._gated = j['gated'] as int; + s._lastEmit = (j['lastEmit'] as num).toDouble(); + for (final m in j['out'] as List) { + final mm = m as Map; + s._out.add({'t': mm['t'] as num, 'v': mm['v'] as num}); + } + s._rowSec.addAll((j['rowSec'] as List).cast()); + s._rowQuiet.addAll([for (final q in j['rowQuiet'] as List) q == 1]); + s._pendRr.addAll(dl(j['pendRr'])); + s._pendTs.addAll(dl(j['pendTs'])); + return s; + } +} diff --git a/tool/incremental/rr_stream.dart b/tool/incremental/rr_stream.dart new file mode 100644 index 0000000..6499b3a --- /dev/null +++ b/tool/incremental/rr_stream.dart @@ -0,0 +1,579 @@ +// RESEARCH PROTOTYPE — streaming Lipponen-Tarvainen RR correction. +// +// Goal: fold NEW beats onto saved state and get, after every fold, EXACTLY +// what `correctRr` returns over the whole series so far (bit-identical nn, +// nnTimes, classes, counts, cleanFraction), while never re-reading the settled +// prefix. +// +// Why that is possible (derivation, with H = windowBeats ~/ 2 = 45): +// beat clock t[i] prefix recurrence (cur, prevTs, origin) +// dRR[i] rr[i]-rr[i-1] local +// th1[i] QD of dRR over [i-H, i+H] -> needs rr up to i+H +// med[i] median of rr[i-H..i+H] \ {i} -> needs rr up to i+H +// mRR[i] f(rr[i], med[i]) -> needs rr up to i+H +// th2[i] QD of mRR over [i-H, i+H] -> needs rr up to i+2H +// class0[i] f(rr,dRR,th1,th2,mRR,med) -> final once n-1 >= i+2H +// class[i] left-to-right compensatory-pair pass; reads only final +// class[i-1], so it folds with one carried class. +// output a run of artifacts is dropped, a single one is spline- +// interpolated from the 2 nearest NORMAL beats each side. The +// left anchors are carried (last two normals); the right anchors +// need classes up to the 2nd normal after the artifact. So the +// settle horizon is 2H beats + (artifact run length) + 2 normals. +// Windows are clipped at n-1, so the last 2H beats are PROVISIONAL: [snapshot] +// re-evaluates just that tail (directly, with the oracle's own clipping) and +// is exactly what correctRr would say at the current n. +import 'dart:math' as math; + +import 'package:openstrap_analytics/onehz.dart' + hide RrCorrector, RrSettled, RrSnapshot, IrregularScreenState; + +/// Provisional + counters view after a fold. `settled*` is NOT here: [fold] +/// returns the newly settled output; this is the part that can still change. +class RrSnapshot { + final int n; + final List tailNn; + final List tailNnTimes; + final int normalCount, droppedCount, correctedCount; + final int settledBeats, classifiedBeats; + final List? tailClasses; // classes of beats [classifiedBeats, n) + const RrSnapshot({ + required this.n, + required this.tailNn, + required this.tailNnTimes, + required this.normalCount, + required this.droppedCount, + required this.correctedCount, + required this.settledBeats, + required this.classifiedBeats, + required this.tailClasses, + }); + double get cleanFraction => n == 0 ? 0 : normalCount / n; +} + +class RrSettled { + final List nn = []; + final List nnTimes = []; +} + +/// Sorted multiset window with O(log w) search + O(w) memmove, w ~ 91. +class _Sorted { + final List a = []; + int _lb(double v) { + var lo = 0, hi = a.length; + while (lo < hi) { + final m = (lo + hi) >> 1; + if (a[m] < v) { + lo = m + 1; + } else { + hi = m; + } + } + return lo; + } + + void add(double v) => a.insert(_lb(v), v); + void remove(double v) => a.removeAt(_lb(v)); + void resetFrom(Iterable xs) { + a + ..clear() + ..addAll(xs) + ..sort(); + } +} + +class RrCorrector { + final double alpha; + final int windowBeats; + final double minThresholdMs; + final double reanchorGapMs; + final int _h; + + // ---- prefix state (the persisted tuple) ---- + int n = 0; // total beats folded + bool? _wall; // timestamps supplied for this stream + double _origin = 0, _cur = 0, _prevTs = 0, _prevRr = 0; + int _off = 0; // global index of _rr[0] + final List _rr = [], _t = [], _d = []; + int _so = 0; // global index of _med[0] (stage arrays) + final List _med = [], _mrr = [], _th1 = []; + int _c1 = 0; // beats with med/mRR/th1 final + int _c2 = 0; // beats with class final + int _ce = 0; // beats whose OUTPUT is final (emitted) + final List _fcls = []; // final classes for [_ce, _c2) + int _lastFinal = 0; // class index of beat _c2-1 + int _normalFinal = 0; + int _dropped = 0, _corrected = 0; + final List _lastNormals = []; // <= 2, oldest first + + // ---- derived caches (not persisted) ---- + final _Sorted _wd = _Sorted(), _wr = _Sorted(), _wm = _Sorted(); + int _wPos = -1, _w2Pos = -1; + + RrCorrector({ + this.alpha = 5.2, + this.windowBeats = 91, + this.minThresholdMs = 100, + this.reanchorGapMs = 1000, + }) : _h = windowBeats ~/ 2 { + if (windowBeats < 3) throw ArgumentError('windowBeats >= 3'); + } + + /// Test hook: called once per beat when its class becomes final. + void Function(int g, int cls)? debugClassSink; + + int get settledBeats => _ce; + int get classifiedBeats => _c2; + int get bufferedBeats => n - _off; + + // ------------------------------------------------------------------ fold + /// Append beats. Returns the output that became FINAL with this fold. + RrSettled fold(List rrMs, {List? tsMs}) { + final out = RrSettled(); + if (rrMs.isEmpty) return out; + final wall = tsMs != null && tsMs.length == rrMs.length; + _wall ??= wall; + if (_wall != wall) { + throw StateError('timestamps must be supplied for every fold or none'); + } + for (var k = 0; k < rrMs.length; k++) { + final r = rrMs[k]; + if (!r.isFinite) throw ArgumentError('non-finite rr at $k'); + final i = n; + double cur; + final ts = wall ? tsMs[k] : 0.0; + if (i == 0) { + _origin = wall ? ts - r : 0.0; + _cur = 0.0; + } + final dropout = wall && i > 0 && (ts - _prevTs) - r > reanchorGapMs; + final anchored = wall ? ts - _origin : 0.0; + cur = (dropout && anchored > _cur) ? anchored : _cur + r; + _cur = cur; + if (wall) _prevTs = ts; + _rr.add(r); + _t.add(cur); + _d.add(i == 0 ? 0.0 : r - _prevRr); + _prevRr = r; + n++; + } + _advance(out); + _compact(); + return out; + } + + double _rrAt(int g) => _rr[g - _off]; + double _dAt(int g) => _d[g - _off]; + double _tAt(int g) => _t[g - _off]; + + void _advance(RrSettled out) { + if (n < 3) return; + final h = _h; + while (_c1 + h <= n - 1) { + _stage1(_c1); + _c1++; + } + while (_c2 + 2 * h <= n - 1) { + _stage2(_c2); + _c2++; + } + _emitCommitted(out); + } + + // th1/med/mRR for beat g; windows [max(0,g-h), g+h] fully inside the data. + void _stage1(int g) { + final h = _h; + if (_wPos == g - 1 && _wPos >= 0) { + _wd.add(_dAt(g + h)); + _wr.add(_rrAt(g + h)); + if (g - 1 - h >= 0) { + _wd.remove(_dAt(g - 1 - h)); + _wr.remove(_rrAt(g - 1 - h)); + } + } else { + final lo = math.max(0, g - h); + _wd.resetFrom([for (var k = lo; k <= g + h; k++) _dAt(k)]); + _wr.resetFrom([for (var k = lo; k <= g + h; k++) _rrAt(k)]); + } + _wPos = g; + final th1 = _thOf(_wd.a); + final m = _medExcl(_wr.a, _rrAt(g)) ?? _rrAt(g); + final d = _rrAt(g) - m; + _med.add(m); + _mrr.add(d < 0 ? d * 2 : d); + _th1.add(th1); + } + + void _stage2(int g) { + final h = _h; + if (_w2Pos == g - 1 && _w2Pos >= 0) { + _wm.add(_mrr[g + h - _so]); + if (g - 1 - h >= 0) _wm.remove(_mrr[g - 1 - h - _so]); + } else { + final lo = math.max(0, g - h); + _wm.resetFrom([for (var k = lo; k <= g + h; k++) _mrr[k - _so]]); + } + _w2Pos = g; + final th2 = _thOf(_wm.a); + var cls = _class0(_rrAt(g), _dAt(g), _th1[g - _so], _med[g - _so], + _mrr[g - _so], th2); + cls = _reconcile(g, cls, _lastFinal, _rrAt(g), _dAt(g), + g > 0 ? _dAt(g - 1) : 0, _med[g - _so]); + _fcls.add(cls.index); + debugClassSink?.call(g, cls.index); + _lastFinal = cls.index; + if (cls == BeatClass.normal) _normalFinal++; + } + + double _thOf(List sorted) { + final q1 = percentileSorted(sorted, 25) ?? 0; + final q3 = percentileSorted(sorted, 75) ?? 0; + final qd = (q3 - q1) / 2; + return math.max(alpha * qd, minThresholdMs); + } + + /// median of [sorted] with ONE occurrence of [v] removed — what + /// `_slidingMedian` computes (it skips k == i), without re-sorting. + static double? _medExcl(List sorted, double v) { + final len = sorted.length - 1; + if (len <= 0) return null; + var lo = 0, hi = sorted.length; + while (lo < hi) { + final m = (lo + hi) >> 1; + if (sorted[m] < v) { + lo = m + 1; + } else { + hi = m; + } + } + final p = lo; + double at(int j) => j < p ? sorted[j] : sorted[j + 1]; + if (len == 1) return at(0); + final rank = (50.0 / 100) * (len - 1); + final l = rank.floor(), u = rank.ceil(); + if (l == u) return at(l); + final frac = rank - l; + return at(l) + (at(u) - at(l)) * frac; + } + + BeatClass _class0( + double rr, double d, double th1, double med, double mrr, double th2) { + final hardLong = rr > 2000; + final hardShort = rr < 300; + final bigJump = d.abs() > th1; + final bigDev = mrr.abs() > th2; + if (hardLong || (bigDev && mrr > 0)) { + return (med > 0 && rr > 1.5 * med) ? BeatClass.missed : BeatClass.longShort; + } else if (hardShort || (bigDev && mrr < 0)) { + return (med > 0 && rr < 0.6 * med) ? BeatClass.extra : BeatClass.longShort; + } else if (bigJump) { + return BeatClass.ectopic; + } + return BeatClass.normal; + } + + BeatClass _reconcile(int g, BeatClass cls, int prevCls, double rr, double d, + double dPrev, double med) { + if (g < 1 || cls != BeatClass.ectopic) return cls; + final prevBad = prevCls != BeatClass.normal.index; + if (!prevBad) return cls; + final opposite = d * dPrev < 0; + final valueNormal = + med > 0 && rr >= 300 && rr <= 2000 && (rr - med).abs() <= 0.2 * med; + return (opposite && valueNormal) ? BeatClass.normal : cls; + } + + static double? _spline(List left, List right) { + if (left.isEmpty || right.isEmpty) return null; + final p1 = left.last; + final p2 = right.first; + final p0 = left.length >= 2 ? left.first : p1; + final p3 = right.length >= 2 ? right.last : p2; + const t = 0.5; + final t2 = t * t; + final t3 = t2 * t; + return 0.5 * + ((2 * p1) + + (-p0 + p2) * t + + (2 * p0 - 5 * p1 + 4 * p2 - p3) * t2 + + (-p0 + 3 * p1 - 3 * p2 + p3) * t3); + } + + // Emission over classes [ce, limit). Shared by the committed pass (limit = + // c2, isEnd=false: undecidable artifacts wait) and the provisional pass + // (limit = n, isEnd=true: end of data resolves them exactly like the batch). + int _emit({ + required int ce, + required int limit, + required bool isEnd, + required int Function(int g) clsAt, + required List lastNormals, + required void Function(double nn, double t) put, + required void Function(int runLen) drop, + required void Function() corrected, + }) { + const normal = 0; + while (ce < limit) { + if (clsAt(ce) == normal) { + put(_rrAt(ce), _tAt(ce)); + lastNormals.add(_rrAt(ce)); + if (lastNormals.length > 2) lastNormals.removeAt(0); + ce++; + continue; + } + var j = ce; + while (j < limit && clsAt(j) != normal) { + j++; + } + if (j == limit && !isEnd) break; + final runLen = j - ce; + if (runLen == 1) { + final right = []; + var k = j; + while (k < limit && right.length < 2) { + if (clsAt(k) == normal) right.add(_rrAt(k)); + k++; + } + if (right.length < 2 && k == limit && !isEnd) break; + final corr = _spline(lastNormals, right); + if (corr != null) { + put(corr, _tAt(ce)); + corrected(); + } else { + drop(1); + } + } else { + drop(runLen); + } + ce = j; + } + return ce; + } + + void _emitCommitted(RrSettled out) { + final fo = _ce; + _ce = _emit( + ce: _ce, + limit: _c2, + isEnd: false, + clsAt: (g) => _fcls[g - fo], + lastNormals: _lastNormals, + put: (v, t) { + out.nn.add(v); + out.nnTimes.add(t); + }, + drop: (r) => _dropped += r, + corrected: () => _corrected++, + ); + // forget classes that are now emitted + final used = _ce - fo; + if (used > 0) _fcls.removeRange(0, used); + } + + // ------------------------------------------------------------- snapshot + /// The part of `correctRr(all beats so far)` that is not settled yet, plus + /// the totals. Settled output (from [fold]) ++ tail == the oracle, exactly. + RrSnapshot snapshot({bool withClasses = false}) { + if (n == 0) { + return const RrSnapshot( + n: 0, + tailNn: [], + tailNnTimes: [], + normalCount: 0, + droppedCount: 0, + correctedCount: 0, + settledBeats: 0, + classifiedBeats: 0, + tailClasses: []); + } + if (n < 3) { + // oracle's n<3 branch (nothing is ever settled below 3 beats) + final nn = [], times = []; + final cl = []; + for (var g = 0; g < n; g++) { + final ok = _rrAt(g) >= 300 && _rrAt(g) <= 2000; + cl.add(ok ? BeatClass.normal : BeatClass.longShort); + if (ok) { + nn.add(_rrAt(g)); + times.add(_tAt(g)); + } + } + return RrSnapshot( + n: n, + tailNn: nn, + tailNnTimes: times, + normalCount: nn.length, + droppedCount: n - nn.length, + correctedCount: 0, + settledBeats: 0, + classifiedBeats: 0, + tailClasses: withClasses ? cl : null); + } + final h = _h; + final end = n - 1; + // ---- provisional stage 1 for beats [c1, n): windows clipped at `end` + final tMed = [], tMrr = [], tTh1 = []; + double mrrAt(int k) => + k < _c1 ? _mrr[k - _so] : tMrr[k - _c1]; + for (var k = _c1; k < n; k++) { + final lo = math.max(0, k - h), hi = math.min(end, k + h); + final dd = [for (var q = lo; q <= hi; q++) _dAt(q)]..sort(); + final rs = [for (var q = lo; q <= hi; q++) _rrAt(q)]..sort(); + tTh1.add(_thOf(dd)); + final m = _medExcl(rs, _rrAt(k)) ?? _rrAt(k); + final d = _rrAt(k) - m; + tMed.add(m); + tMrr.add(d < 0 ? d * 2 : d); + } + // ---- provisional classes for [c2, n) + final tc = []; + var prev = _lastFinal; + var normalTent = 0; + for (var g = _c2; g < n; g++) { + final lo = math.max(0, g - h), hi = math.min(end, g + h); + final ms = [for (var q = lo; q <= hi; q++) mrrAt(q)]..sort(); + final th2 = _thOf(ms); + final th1 = g < _c1 ? _th1[g - _so] : tTh1[g - _c1]; + final med = g < _c1 ? _med[g - _so] : tMed[g - _c1]; + final mrr = mrrAt(g); + var cls = _class0(_rrAt(g), _dAt(g), th1, med, mrr, th2); + cls = _reconcile( + g, cls, prev, _rrAt(g), _dAt(g), g > 0 ? _dAt(g - 1) : 0, med); + tc.add(cls.index); + prev = cls.index; + if (cls == BeatClass.normal) normalTent++; + } + // ---- provisional emission over [ce, n) + final fo = _ce; + final ln = List.of(_lastNormals); + final nn = [], times = []; + var dropped = _dropped, corrected = _corrected; + _emit( + ce: _ce, + limit: n, + isEnd: true, + clsAt: (g) => g < _c2 ? _fcls[g - fo] : tc[g - _c2], + lastNormals: ln, + put: (v, t) { + nn.add(v); + times.add(t); + }, + drop: (r) => dropped += r, + corrected: () => corrected++, + ); + return RrSnapshot( + n: n, + tailNn: nn, + tailNnTimes: times, + normalCount: _normalFinal + normalTent, + droppedCount: dropped, + correctedCount: corrected, + settledBeats: _ce, + classifiedBeats: _c2, + tailClasses: withClasses + ? [for (final c in tc) BeatClass.values[c]] + : null, + ); + } + + // --------------------------------------------------------------- compact + void _compact() { + final h = _h; + final newOff = math.max(0, math.min(_ce, _c2 - 2 * h - 1)); + if (newOff > _off) { + final drop = newOff - _off; + _rr.removeRange(0, drop); + _t.removeRange(0, drop); + _d.removeRange(0, drop); + _off = newOff; + } + final newSo = math.max(0, _c2 - h - 1); + if (newSo > _so && _med.isNotEmpty) { + final drop = math.min(newSo - _so, _med.length); + _med.removeRange(0, drop); + _mrr.removeRange(0, drop); + _th1.removeRange(0, drop); + _so += drop; + } + } + + // ------------------------------------------------------------------ json + Map toJson() => { + 'v': 1, + 'alpha': alpha, + 'win': windowBeats, + 'floor': minThresholdMs, + 'reanchor': reanchorGapMs, + 'n': n, + 'wall': _wall, + 'origin': _origin, + 'cur': _cur, + 'prevTs': _prevTs, + 'prevRr': _prevRr, + 'off': _off, + 'rr': _rr, + 't': _t, + 'd': _d, + 'c1': _c1, + 'c2': _c2, + 'ce': _ce, + 'fcls': _fcls, + 'lastFinal': _lastFinal, + 'normalFinal': _normalFinal, + 'dropped': _dropped, + 'corrected': _corrected, + 'lastNormals': _lastNormals, + }; + + factory RrCorrector.fromJson(Map j) { + List dl(Object? o) => [for (final x in o as List) (x as num).toDouble()]; + final c = RrCorrector( + alpha: (j['alpha'] as num).toDouble(), + windowBeats: j['win'] as int, + minThresholdMs: (j['floor'] as num).toDouble(), + reanchorGapMs: (j['reanchor'] as num).toDouble(), + ); + c.n = j['n'] as int; + c._wall = j['wall'] as bool?; + c._origin = (j['origin'] as num).toDouble(); + c._cur = (j['cur'] as num).toDouble(); + c._prevTs = (j['prevTs'] as num).toDouble(); + c._prevRr = (j['prevRr'] as num).toDouble(); + c._off = j['off'] as int; + c._rr.addAll(dl(j['rr'])); + c._t.addAll(dl(j['t'])); + c._d.addAll(dl(j['d'])); + c._c1 = j['c1'] as int; + c._c2 = j['c2'] as int; + c._ce = j['ce'] as int; + c._fcls.addAll((j['fcls'] as List).cast()); + c._lastFinal = j['lastFinal'] as int; + c._normalFinal = j['normalFinal'] as int; + c._dropped = j['dropped'] as int; + c._corrected = j['corrected'] as int; + c._lastNormals.addAll(dl(j['lastNormals'])); + c._rebuildStageArrays(); + return c; + } + + /// Stage arrays are pure functions of the buffered raw beats; recompute. + void _rebuildStageArrays() { + final h = _h; + _so = math.max(0, _c2 - h - 1); + _med.clear(); + _mrr.clear(); + _th1.clear(); + for (var g = _so; g < _c1; g++) { + final lo = math.max(0, g - h), hi = g + h; + final dd = [for (var q = lo; q <= hi; q++) _dAt(q)]..sort(); + final rs = [for (var q = lo; q <= hi; q++) _rrAt(q)]..sort(); + _th1.add(_thOf(dd)); + final m = _medExcl(rs, _rrAt(g)) ?? _rrAt(g); + final d = _rrAt(g) - m; + _med.add(m); + _mrr.add(d < 0 ? d * 2 : d); + } + _wPos = -1; + _w2Pos = -1; + } +} diff --git a/tool/incremental/size_probe.dart b/tool/incremental/size_probe.dart new file mode 100644 index 0000000..bfb88ec --- /dev/null +++ b/tool/incremental/size_probe.dart @@ -0,0 +1,27 @@ +// JSON size of each carried state at the end of the real night (no timing). +import 'dart:convert'; +import 'driver.dart'; +import 'hrv_incr.dart'; +import 'oracle_util.dart'; +import 'resp_incr.dart'; +import 'synth.dart'; + +void main() { + final night = realNightRr()!; + final noc = NocturnalRmssdState(); + final shape = NightShapeState(); + final tl = HrvTimelineState(night.ts.first - night.rr.first); + final wins = RespWindowsState(); + final cuts = timeCuts(night.ts, 900); + driveRr(night, cuts, (f) { + noc.fold(f.settled.nn, f.settled.nnTimes); + shape.fold(f.settled.nn, f.settled.nnTimes); + tl.fold(f.settled.nn, f.settled.nnTimes); + wins.fold(f.settled.nn, f.settled.nnTimes); + }); + int sz(Object o) => jsonEncode(o).length; + print('nocturnal recs=${noc.recs.length} bytes=${sz([for (final r in noc.recs) (r as dynamic).toJson()])}'); + print('night shape closed bins=${shape.closed.length} bytes=${sz(shape.closed)}'); + print('timeline points=${tl.out.length} bytes=${sz(tl.out)} (+ ring <= 900 beats x 2 doubles ~ 30 KB)'); + print('resp windows closed=${wins.closed.length} bytes=${sz(wins.closed)}'); +} diff --git a/tool/incremental/synth.dart b/tool/incremental/synth.dart new file mode 100644 index 0000000..1c7bd90 --- /dev/null +++ b/tool/incremental/synth.dart @@ -0,0 +1,197 @@ +// Research scaffolding: real-shaped RR / accelerometer generators and the real +// night fixture loader. Deterministic (seeded), no wall clock. +// +// "Real-shaped" means what the app really feeds correctRr: +// * integer-millisecond RR, end-of-beat timestamps quantised to WHOLE seconds +// (rr_ts_ms = rec_ts * 1000, several beats can share one stamp), +// * ectopic pairs, missed beats (2x interval), extra beats (half interval), +// multi-beat noise runs, sensor dropouts (no beats for seconds..minutes), +// * optional counter-reset style timestamp steps backwards. +import 'dart:io'; +import 'dart:math' as math; + +class RrData { + final List rr; + final List ts; // epoch-ish ms, whole seconds + RrData(this.rr, this.ts); + int get length => rr.length; +} + +class SynthConfig { + final int seed; + final double hours; + final double startSec; + final double ectopicPerMin; // premature beat + compensatory pause + final double missedPerMin; + final double extraPerMin; + final double noiseRunPerMin; // run of 3..12 garbage beats + final double gapPerHour; // dropouts of 5..900 s + final double backwardsPerHour; // ts steps back 1..3 s (counter reset-ish) + final double nightFraction; // fraction of the span that is "asleep" + const SynthConfig({ + this.seed = 1, + this.hours = 24, + this.startSec = 1.7e9, + this.ectopicPerMin = 0.3, + this.missedPerMin = 0.1, + this.extraPerMin = 0.1, + this.noiseRunPerMin = 0.05, + this.gapPerHour = 1.5, + this.backwardsPerHour = 0.0, + this.nightFraction = 0.33, + }); +} + +/// Physiologically-flavoured RR: slow HR drift + RSA (0.25 Hz) + LF (0.1 Hz) +/// + white jitter, integer ms, with injected artefacts. +RrData synthRr(SynthConfig c) { + final rnd = math.Random(c.seed); + final rr = [], ts = []; + final endT = c.hours * 3600.0; + var t = 0.0; // seconds since start (beat clock) + var phase = rnd.nextDouble() * 6.28; + final nightStart = endT * 0.05; + final nightEnd = nightStart + endT * c.nightFraction; + void emit(double interval) { + final v = interval.roundToDouble(); + t += v / 1000.0; + var stamp = (c.startSec + t).floorToDouble() * 1000.0; + rr.add(v); + ts.add(stamp); + } + + while (t < endT) { + final asleep = t >= nightStart && t < nightEnd; + // Base HR: sleep ~52 bpm, day 65..120 with activity bouts. + final bout = (math.sin(t / 1900.0 + c.seed) + 1) / 2; // 0..1 + final bpm = asleep ? 52 + 3 * math.sin(t / 5400) : 66 + 55 * bout * bout; + var base = 60000.0 / bpm; + final rsaAmp = asleep ? 45.0 : 22.0 * (1 - 0.7 * bout); + final rsaHz = asleep ? 0.24 : 0.27; + final v = base + + rsaAmp * math.sin(2 * math.pi * rsaHz * t + phase) + + 28 * math.sin(2 * math.pi * 0.1 * t) + + 8 * (rnd.nextDouble() - .5) * 2; + final beatMin = v / 60000.0; // minutes per beat → per-beat probabilities + final u = rnd.nextDouble(); + var p = c.noiseRunPerMin * beatMin; + if (u < p) { + final len = 3 + rnd.nextInt(10); + for (var k = 0; k < len; k++) { + emit(250 + rnd.nextInt(2100).toDouble()); + } + continue; + } + p += c.ectopicPerMin * beatMin; + if (u < p) { + emit(v * (0.55 + 0.15 * rnd.nextDouble())); + emit(v * (1.3 + 0.2 * rnd.nextDouble())); + continue; + } + p += c.missedPerMin * beatMin; + if (u < p) { + emit(v * (1.9 + 0.2 * rnd.nextDouble())); + continue; + } + p += c.extraPerMin * beatMin; + if (u < p) { + final f = 0.35 + 0.3 * rnd.nextDouble(); + emit(v * f); + emit(v * (1 - f)); + continue; + } + p += c.gapPerHour * beatMin / 60.0; + if (u < p) { + final gap = 5 + rnd.nextInt(896); + t += gap.toDouble(); + continue; + } + p += c.backwardsPerHour * beatMin / 60.0; + if (u < p && ts.isNotEmpty) { + emit(v); + ts[ts.length - 1] = ts.last - 1000.0 * (1 + rnd.nextInt(3)); + continue; + } + emit(v); + } + return RrData(rr, ts); +} + +/// The checked-in anonymised 8.9 h WHOOP-4 night (rebased to 0 s). Returns +/// null if the fixture is missing (callers decide whether to fail). +RrData? realNightRr({double startSec = 1.7e9}) { + final f = File('test/onehz/fixtures/real_night_2026_07_rr.csv'); + if (!f.existsSync()) return null; + final rr = [], ts = []; + for (final line in f.readAsLinesSync().skip(1)) { + if (line.trim().isEmpty) continue; + final p = line.split(','); + ts.add(startSec * 1000 + double.parse(p[0])); + rr.add(double.parse(p[1])); + } + return RrData(rr, ts); +} + +/// A whole 24 h-ish day: synthetic morning + REAL night + synthetic evening. +/// The synthetic parts carry the artefact load; the night is as recorded. +RrData realShapedDay({int seed = 7, double startSec = 1.7e9}) { + final real = realNightRr(startSec: startSec)!; + final nightSpanMs = real.ts.last - real.ts.first; + final morning = synthRr(SynthConfig( + seed: seed, hours: 5, startSec: startSec - 5 * 3600, nightFraction: 0)); + final evening = synthRr(SynthConfig( + seed: seed + 1, + hours: 9, + startSec: startSec + nightSpanMs / 1000 + 120, + nightFraction: 0)); + return RrData([...morning.rr, ...real.rr, ...evening.rr], + [...morning.ts, ...real.ts, ...evening.ts]); +} + +/// Per-second 1 Hz accel for [fromSec, toSec): stillness / motion bouts, +/// absent (all-zero) seconds and gaps. Rows carry epoch seconds. +class AccelSeries { + final List tsSec; + final List ax, ay, az; + AccelSeries(this.tsSec, this.ax, this.ay, this.az); + int get length => tsSec.length; +} + +AccelSeries synthAccel(int seed, int fromSec, int toSec, + {double stillFraction = 0.55, double gapPerHour = 2.0}) { + final rnd = math.Random(seed); + final ts = [], ax = [], ay = [], az = []; + var still = true; + var left = 60 + rnd.nextInt(1200); + var s = fromSec; + while (s < toSec) { + if (rnd.nextDouble() < gapPerHour / 3600.0) { + s += 5 + rnd.nextInt(300); + continue; + } + if (--left <= 0) { + still = rnd.nextDouble() < stillFraction; + left = 30 + rnd.nextInt(1500); + } + double x, y, z; + if (rnd.nextDouble() < 0.0008) { + x = y = z = 0; // fill row: accel absent + } else if (still) { + // gravity only: |g| within 0.02 g of 1 (some seconds slightly off) + final jitter = (rnd.nextDouble() - .5) * (rnd.nextDouble() < 0.1 ? .08 : .01); + x = 0.3 + (rnd.nextDouble() - .5) * .002; + y = 0.8 + (rnd.nextDouble() - .5) * .002; + z = math.sqrt(math.max(0.0, 1 - x * x - y * y)) + jitter; + } else { + x = (rnd.nextDouble() - .5) * 2.4; + y = (rnd.nextDouble() - .5) * 2.4; + z = 1 + (rnd.nextDouble() - .5) * 2.0; + } + ts.add(s); + ax.add(x); + ay.add(y); + az.add(z); + s++; + } + return AccelSeries(ts, ax, ay, az); +} diff --git a/tool/incremental_benchmark.dart b/tool/incremental_benchmark.dart new file mode 100644 index 0000000..c814058 --- /dev/null +++ b/tool/incremental_benchmark.dart @@ -0,0 +1,142 @@ +// Run with: nix develop -c dart run tool/incremental_benchmark.dart +// Same deterministic growing inputs on both paths. Timings describe this +// process and machine; processed counts measure the incremental math work. +import 'dart:math' as math; + +import 'package:openstrap_analytics/onehz.dart'; + +void main() { + final nn = [], times = []; + var time = 0.0; + for (var i = 0; i < 7200; i++) { + final value = 800 + 60 * math.sin(i * .13) + 20 * math.sin(i * .031); + time += value; + if (i % 731 == 730) time += 17000; + nn.add(value); + times.add(time); + } + final frequencies = List.generate(32, (i) => .01 + i * .012); + final seconds = [for (final t in times) t / 1000]; + final accel = List.generate( + 7200, + (i) => AccelSample(i * 1000.0, .1 * math.sin(i * .19), + .05 * math.cos(i * .07), 1 + .04 * math.sin(i * .31))); + final hr = List.generate(600, (i) => 75.0 + 65 * (1 + math.sin(i * .07)) / 2); + final keys = List.generate(hr.length, (i) => 28000000 + i); + final cadence = + List.generate(hr.length, (i) => i % 5 == 0 ? 110 : null); + const profile = + WorkoutUserProfile(weightKg: 80, heightCm: 180, age: 35, sex: 'male'); + + final spectrum = IncrementalLombScargle(frequencies); + compare( + 'Lomb, 32 frequencies', + nn.length, + 60, + (n) => lombScargle(seconds.sublist(0, n), nn.sublist(0, n), frequencies), + (n) => spectrum.sync(seconds.sublist(0, n), nn.sublist(0, n)), + () => spectrum.processedPoints); + final hrv = IncrementalHrvTime(); + compare( + 'Time HRV', + nn.length, + 60, + (n) => hrvTime(nn.sublist(0, n), nnTimesMs: times.sublist(0, n)), + (n) => hrv.sync(nn.sublist(0, n), nnTimesMs: times.sublist(0, n)), + () => hrv.processedPoints); + final motion = IncrementalEnmoSeries(); + compare( + 'Motion, explicit gravity reference', + accel.length, + 60, + (n) => enmoSeries(accel.sublist(0, n), gRef: 1), + (n) => motion.sync(accel.sublist(0, n), gRef: 1), + () => motion.processedPoints); + final minutes = IncrementalMinuteMetrics(); + compare('Minute load and energy', hr.length, 1, (n) { + final input = hr.sublist(0, n), walking = cadence.sublist(0, n); + final load = banisterTrimp(input, restingHr: 55, maxHr: 185, sex: Sex.male); + final strain = strainScoreMetric(load.value, + wakeMinutes: n.toDouble(), quietHrr: .12, quietSettled: false, female: false); + final energy = Calories.dailyEnergy(input, + profile: profile, hrmax: 185, restingHr: 55, cadenceSpmPerMin: walking); + final series = Calories.minuteEnergy(input, + profile: profile, + hrmax: 185, + restingHr: 55, + cadenceSpmPerMin: walking, + epochMinutes: keys.sublist(0, n)); + return MinuteMetrics( + trimp: load, strain: strain, energy: energy, minutes: series); + }, + (n) => minutes.sync(keys.sublist(0, n), hr.sublist(0, n), + cadenceSpm: cadence.sublist(0, n), + restingHr: 55, + maxHr: 185, + profile: profile, + quietHrr: .12), + () => minutes.processedMinutes); + final summary = IncrementalMinuteMetrics(); + compare('Minute summaries (no minute series)', hr.length, 1, (n) { + final input = hr.sublist(0, n), walking = cadence.sublist(0, n); + final load = banisterTrimp(input, restingHr: 55, maxHr: 185, sex: Sex.male); + final strain = strainScoreMetric(load.value, + wakeMinutes: n.toDouble(), quietHrr: .12, quietSettled: false, female: false); + final energy = Calories.dailyEnergy(input, + profile: profile, hrmax: 185, restingHr: 55, cadenceSpmPerMin: walking); + return MinuteMetrics(trimp: load, strain: strain, energy: energy); + }, + (n) => summary.sync(keys.sublist(0, n), hr.sublist(0, n), + cadenceSpm: cadence.sublist(0, n), + restingHr: 55, + maxHr: 185, + profile: profile, + quietHrr: .12, + includeMinuteSeries: false), + () => summary.processedMinutes); +} + +Object? _sink; + +void compare(String name, int length, int chunk, Object? Function(int) batch, + Object? Function(int) incremental, int Function() work) { + // Warm the same growing-prefix workloads that will be measured, including + // the append checks and artifact rendering on the incremental path. + for (var warm = 0; warm < 3; warm++) { + for (var n = chunk; n <= length; n += chunk) { + _sink = batch(n); + } + incremental(0); + for (var n = chunk; n <= length; n += chunk) { + _sink = incremental(n); + } + } + final fullTimes = [], deltaTimes = []; + var batchPoints = 0, processed = 0; + for (var repeat = 0; repeat < 5; repeat++) { + batchPoints = 0; + final full = Stopwatch()..start(); + for (var n = chunk; n <= length; n += chunk) { + _sink = batch(n); + batchPoints += n; + } + full.stop(); + fullTimes.add(full.elapsedMicroseconds); + incremental(0); + final workBefore = work(); + final delta = Stopwatch()..start(); + for (var n = chunk; n <= length; n += chunk) { + _sink = incremental(n); + } + delta.stop(); + deltaTimes.add(delta.elapsedMicroseconds); + processed = work() - workBefore; + } + fullTimes.sort(); + deltaTimes.sort(); + print('$name: batch ${fullTimes[2] / 1000} ms, ' + 'incremental ${deltaTimes[2] / 1000} ms (median of 5 warmed runs); ' + '$batchPoints batch input visits, $processed incremental points'); + // Keep results observable so the benchmark represents published output. + if (_sink == null) throw StateError('Benchmark produced no output'); +}