Skip to content

feat: add absorb for in-place merges of buffered digests - #6

Open
kei-nan wants to merge 1 commit into
MnO2:masterfrom
kei-nan:feat-absorb
Open

kei-nan wants to merge 1 commit into
MnO2:masterfrom
kei-nan:feat-absorb

Conversation

@kei-nan

@kei-nan kei-nan commented Oct 13, 2026

Copy link
Copy Markdown

Folding many small digests into one, for example per-batch digests built with push, currently means flushing each one and calling merge_digests. That sorts, compresses and allocates for every digest, even when it holds only a few dozen values.

This PR adds TDigest::absorb(&mut self, other: TDigest):

  • If other has never flushed, its buffered values are pushed into self. The result is exactly what pushing them into self directly would give.
  • Otherwise, both digests are flushed and combined with merge_digests.
  • The result keeps the larger max_size. As with push, call flush before queries.

This is a new public method, so it's a minor-version addition with no existing behavior changed.

Tests

  • Unit tests: absorbing buffered chunks equals pushing directly (full PartialEq); absorbing a flushed digest equals merge_digests; empty digests in both positions; the larger max_size is kept.
  • Proptest: absorbing buffered chunks of any size below the flush threshold equals pushing directly.
  • Forcing absorb onto the merge path makes the buffered-path tests fail.

Bench and docs

  • New absorb_small_digests bench: 10,000 values in 64-value digests, absorb vs. flush + merge_digests.
  • README.md and docs/architecture.md describe the method, and the cost table has a row for it.

Checked locally: cargo test (debug and release, all feature sets), clippy -D warnings, rustdoc -D warnings, no_std, and the MSRV 1.62 checks from CI.

This is independent of the integer-key sort PR. If both land, absorb pushes through whichever buffer representation is in place.

🤖 Generated with Claude Code

Folding many small digests into one means flushing each and calling
merge_digests, which sorts, compresses and allocates per digest even
when it holds only a few dozen values.

TDigest::absorb merges another digest in place. If the other digest
has never flushed, its buffered values are pushed into the receiver,
exactly as if they had been pushed there directly. Otherwise both are
flushed and combined with merge_digests. The result keeps the larger
max_size.

Adds unit and property tests for both paths and an
absorb_small_digests bench comparing absorb with flush plus
merge_digests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant