Skip to content

Preserve normal distribution tails and scaled results across overflow and underflow - #489

Open
jmh530 wants to merge 4 commits into
libmir:masterfrom
jmh530:codex/normal-tail-precision
Open

jmh530 wants to merge 4 commits into
libmir:masterfrom
jmh530:codex/normal-tail-precision

Conversation

@jmh530

@jmh530 jmh530 commented Oct 13, 2026

Copy link
Copy Markdown
Contributor

Normal distribution functions can lose representable results when intermediate calculations overflow or underflow. This PR corrects those cases in mir.math.func.normal while preserving public signatures and ordinary calculation paths where possible.

Changes

  • Evaluate the normal CDF tail exponential directly as exp(-xx/2). Previously, sqrt(exp(-xx)) underflowed prematurely: normalCDF(-30.0) returned zero instead of approximately 4.906713927148187e-198.
  • Add a half-exponent specialization to the split-square helper, preserving its existing behavior for other callers. Use a float-specific split to limit rounding error.
  • Propagate NaN through normalCDF while retaining infinite endpoints.
  • Recover standardized observations when subtracting finite, opposite-signed values overflows. For example, normalCDF(1e308, -1e308, 1e308) should evaluate at a standardized observation of 2 rather than infinity.
  • Include a small standard deviation in the exponent when an intermediate normal density is subnormal or zero, allowing the scaled PDF to remain representable.
  • Recover finite scaled inverse CDF results when an overflowing product is offset by the mean.
  • Preserve extended precision in the central inverse CDF approximation instead of narrowing its final multiplier to double.
  • Correct the documented CDF normalization and explain how to evaluate small upper-tail probabilities without subtraction from one.

Validation

Added float, double, and real regressions using independent high-precision references, including subnormal tails, extreme locations, tiny scales, inverse endpoints, and extended-precision quartiles.

Full DMD and LDC suites passed during development. Final optimized focused tests passed with both compilers.

The recovery checks introduce measurable overhead on some scaled paths; ordinary formulas and rounding are retained outside the exceptional cases.

These fixes also underpin the local normal implementations in libmir/mir-stat#159. This PR independently benefits callers using mir.math.func.normal.

AI-Assisted-By: Codex (GPT-6 Astra Medium)

Evaluate exp(-x*x/2) directly with the split-square helper instead of taking sqrt(exp(-x*x)). The old intermediate underflowed early, losing accuracy near -27 and returning zero at -30 even though the normal CDF is representable.

Add an optional half-exponent specialization while retaining the existing expx2 behavior for other callers. Use a coarser exact-square split for float tails. Propagate NaN inputs while preserving infinite endpoints.

Cover float, double, and extended-real tails with independent high-precision references, including a subnormal double result. Document the platform exponential and subnormal accuracy limits.

AI-Assisted-By: Codex (GPT-6 Astra Medium)
Standardize finite opposite-signed observations by dividing before subtraction when x - mean overflows. Preserve the ordinary subtraction path for normal PDF and CDF calls.

Include small standard deviations in the exponent when the unscaled density is subnormal or zero, so scaling can recover a representable PDF. For inverse CDFs, halve the finite terms before combining an overflowing scaled quantile with its location.

Retain endpoint infinities and existing arithmetic outside the recovery paths. Add float, double, and real regressions for extreme locations, subnormal scales, and finite quantiles after cancellation.

AI-Assisted-By: Codex (GPT-6 Astra Medium)
Use the requested floating-point type for the final sqrt(2*pi) multiplier when it has more precision than double. The unconditional double cast discarded precision after evaluating the central polynomial in real.

Preserve the existing float and double arithmetic. Add independent first- and third-quartile references on platforms with extended real precision.

AI-Assisted-By: Codex (GPT-6 Astra Medium)
Correct the integral's normalization to 1/sqrt(2*pi), consistent with its exp(-t*t/2) integrand. Explain that small upper-tail probabilities should be evaluated as normalCDF(-x), rather than lost by subtracting a rounded CDF from one.

AI-Assisted-By: Codex (GPT-6 Astra Medium)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant