Skip to content

feat(kll): add weighted update - #293

Open
shanielh wants to merge 1 commit into
apache:mainfrom
shanielh:kll-weighted-update
Open

shanielh wants to merge 1 commit into
apache:mainfrom
shanielh:kll-weighted-update

Conversation

@shanielh

@shanielh shanielh commented Oct 7, 2026

Copy link
Copy Markdown

Adds KllSketch::update_with_weight(item, weight), equivalent to calling update(item) weight times. This mirrors update(item, weight) on the KLL sketches in datasketches-java. The name follows CountMinSketch::update_with_weight.

Behavior

  • If weight is smaller than the free capacity, the item is inserted weight times as plain updates. No compaction can happen on this path.
  • Otherwise, the update builds an exact sketch of the weighted item and merges it in. Since an item at level h counts as 2^h, the exact sketch holds one copy of the item at each level whose bit is set in weight. Bits above the top level fold into weight >> 60 copies at level 60, because level capacities are only defined up to MAX_NUM_LEVELS (61). This path costs one merge with a sketch of at most 75 items, however large the weight.
  • A weight of zero is a no-op, the same as FrequentItemsSketch::update_with_count.
  • If the stream weight would exceed u64::MAX, it panics without modifying the sketch, the same as update.
  • merge's body moves into a private merge_unchecked, so the weighted path can skip checks it has already done without discarding a Result. merge is otherwise unchanged.
  • Serialization is unchanged.

Tests

New module tests-integration/tests/kll_test/update.rs:

  • zero weight
  • equality with repeated updates in exact mode
  • a single item with a 2^40 + 12345 weight
  • u64::MAX - 1 weight, with a serialization round trip
  • large weights merged into a full sketch
  • 10,000 weighted items in estimation mode, ranks within the k=200 error bound, with a serialization round trip
  • strings
  • overflow leaves the sketch unchanged

cargo x test, cargo clippy --all-features --all-targets --workspace -- -D warnings and cargo +nightly fmt --all --check pass locally. I did not run taplo, typos or hawkeye locally, so CI will be their first run.

Companion PR for datasketches-cpp: apache/datasketches-cpp#542

🤖 Generated with Claude Code

KllSketch::update_with_weight(item, weight) is equivalent to calling
update(item) weight times. A weight that fits into the free capacity is
applied as plain updates; a larger one builds an exact sketch holding one
copy of the item at each level whose bit is set in the weight and merges
it, so the cost is logarithmic in the weight. This mirrors the weighted
update in datasketches-java.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

The randomized accuracy test can fail legitimately, and the complexity documentation overstates the implementation guarantee.

Review effort: Balanced
Findings: 1 Medium severity · 2 Low severity

Open (3)
What changed in this PR

Adds efficient weighted updates to KLL sketches.

Changes:

  • Adds update_with_weight and internal merge refactoring.
  • Adds weighted-update integration tests and changelog documentation.
File Description
datasketches/​src/​kll/​sketch.rs Implements weighted updates.
tests-integration/​tests/​kll_test/​update.rs Tests weighted-update behavior.
tests-integration/​tests/​kll_test/​main.rs Registers the new tests.
CHANGELOG.md Documents the API.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +114 to +115
(rank - true_rank).abs() <= RANK_EPS_FOR_K_200,
"item {item}: rank {rank}, true rank {true_rank}"
Comment thread CHANGELOG.md
### New features

* `KllSketch` is now available behind the `kll` feature, with rank, quantile, PMF, and CDF queries, merging, serialization, custom ordered item types, and a `KllFloat` adapter for non-NaN floating-point values.
* `KllSketch` is now available behind the `kll` feature, with rank, quantile, PMF, and CDF queries, merging, serialization, custom ordered item types, and a `KllFloat` adapter for non-NaN floating-point values. `KllSketch::update_with_weight` adds an item repeated a given number of times, at a cost logarithmic in the weight.
Comment on lines +148 to +150
/// The result is equivalent to calling [`update`](Self::update) `weight` times, at a cost
/// that grows with the logarithm of `weight` rather than with `weight` itself. A weight of
/// zero is a no-op.
@tisonkun
tisonkun requested review from PsiACE and notfilippo October 8, 2026 17:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants