Repository navigation
Compaction: find level-0 cross-source candidates by scanning cells instead of searching - #740
Merged
Merged
Conversation
One vector against a tile of queries, with a scalar default and a Panama implementation that loads each vector chunk once and feeds eight accumulators, so a streamed vector is scored against every query that wants it in one pass. The interface method carries a default, so the native support class is unchanged.
PreEncodedCodeCache: a chunked memory-mapped array of fixed-width entries indexed by output ordinal, holding either codes or the vectors themselves. A row can be warmed with small positional reads, which pull only the pages asked for where a fault on the mapping would drag in the kernel's readahead window. BandedReverseCandidateBuffer: reverse offers from a smaller source's candidates onto a larger source's nodes, kept in a bounded band of ordinals and spilled past it, so peak memory is independent of node count. RawVectorCode: a compressor that stores the vector itself, which lets the merge reuse the code-cache plumbing as its vector store. PQRetrainer gains a training entry point for a second, wide code, and ProductQuantization exposes read-only accessors for a subspace's width and codebook and the global centroid, which the wide-code decoder and the sidecar lookup table read.
…ty provider HubMap holds the merged upper layers resident, adjacency and vectors, so cells can be assigned and probed by a greedy descent and a beam over level 1 without touching a source file per query. WideDecoder turns a candidate-store entry into a vector, whether the entry is a wide product-quantisation code or the vector itself. ExactJoin holds one node range's cell-walk results as packed per-target min-heaps of (score, ordinal), sized by the range and the search width. CompactVamanaDiversityProvider is the pruning rule, with its threshold step; SelectedVecCache is the set of already-selected neighbours the rule compares each candidate against.
Level 0 of a merge no longer searches each target graph per node. The upper layers are merged first, with the greedy descent and beam search of every other source they always used, and their merged level 1 becomes the cell hierarchy: every source's level-1 nodes are cells, so the partition is as fine as the merged index's own. The compactor then assigns the output ordinals by region: every live node is placed in the cell of the merged level-1 node it descends to, so each source's members of a cell occupy one contiguous ordinal range, and a pre-encode pass copies every live vector into a scratch region of the output in that order. A node then picks its probe cells with a beam over level 1, and each target's probed cells are streamed once against every node of the range that probes them, a sequential read where the search was a pointer chase, with every candidate score exact. Survivors and retained edges pass through the diversity rule and are written; record writes follow ordinal order and are sequential too. Reverse offers carry candidates from smaller sources to larger ones so each pair is joined once. The merge is the same for every input and every memory regime. Whatever the sources carry, the scan runs on the vectors; the input type decides only whether the output carries codes, which the fused or sidecar strategy retrains and pre-encodes in the same pass. Only the pairwise diversity comparisons use a wide product-quantisation code trained at merge time, four dimensions per subspace, so a node's selection touches one row per candidate rather than one per candidate-neighbour pair and the code stays resident when the vectors do not; the pruning threshold is recomputed from the decoded candidate so both sides of the rule carry the same bias. The coarser code also folds near-duplicates onto one code, which the rule then prunes, so the degree is spent on distinct neighbours. Output ordinals are the compactor's: the constructors no longer take ordinal mappers, the mapping is read back through ordinalMappers, and deleted nodes receive none, so no record, code slot or sidecar entry is spent on them. Graph search remains as the per-node fallback for a node with no cell to scan; the merged upper layers are written after level 0. The strategies and writer follow: pre-encode caches are reserved past the projected end of the output from a watermark so several can coexist, wide entries are written in ordinal order, and the fused strategy's per-edge re-encode hooks are gone since every code comes from the cache. Tests cover the cell join on fused and full-precision sources for dot product, Euclidean and cosine, ordinal assignment and deletions through the compactor's mapping, and sidecar merges. The benchmarks no longer supply mappers; they score searches on the compacted graph's own vectors and translate results through the compactor's mapping before comparing them with the ground truth, which is expressed in dataset rows.
Output ordinals and how they are assigned, the upper layers merged first and their level 1 as the cells, the cell join, the merge store, and the wide code the diversity rule runs on.
The recall path needed the compacted graph's ordinals mapped back to dataset rows, and did it by materialising one int per output ordinal. That array is redundant: the mappers already answer the question, and both implementations return OMITTED for an ordinal they do not own, so a short scan over them is enough. Translating the ground truth into output ordinals instead removes the array (24-40 MB), the pass over every output ordinal that built it, and the need for a toRow overload in AccuracyMetrics, which returns to its upstream form. The translation also moves out of the timed region in CompactorBenchmark, where it had been inflating the reported compaction time.
dian-lun-lin
requested review from
MarkWolters,
ashkrisk,
jshook and
tlwillke
as code owners
October 2, 2026 18:20
Contributor
|
Before you submit for review:
If you did not complete any of these, then please explain below. |
Compaction reserves scratch sections past the projected end of the output file, and more than one strategy can hold mappings on them: the inline pre-encode cache and the raw-vector store map regions of the same file. Cleanup truncated as soon as the first strategy was finished, while the second still had its mappings open. Linux allows shrinking a file under a live mapping, so this passed locally; Windows keeps the file open for the mapping's lifetime and fails the call, which is where the CI failures came from. Unmapping and resizing are now separate phases. The strategies expose releaseMappings(), pendingTruncateOffset() and clearPendingTruncate(), and the compactor truncates once, after every strategy has released, to the smallest reserved offset. The sidecar entry point is included in that call because its strategy holds a mapping until writeSidecar has copied out of it. Truncation failure is now logged instead of thrown. The graph is complete and valid by that point and the only consequence is an unused tail left on disk, which is not worth failing a finished merge over. Also widened the catch around Unsafe.invokeCleaner. A JDK that refuses Unsafe memory access throws something other than IllegalArgumentException, which would have propagated out of a successful compaction's cleanup.
Covers the mechanism, the breaking constructor change and its migration, the fallback when the merged upper layers yield no cells, and the measured results across full-precision, fused-PQ and PQ-sidecar inputs. States the recall cost and the attribution caveat rather than leaving them to be discovered: most of the in-RAM gain comes from processing partitions smallest-first with bounded reverse offers, and the cell join itself earns the above-RAM numbers.
Contributor
Author
|
@jshook @MarkWolters Could you please have a look? Thanks |
The merged hub map held the vectors of every level-1 node on the heap, the one part of the merge whose resident size grows with the dataset (about 3 % of it). The compactor now writes the level-1 vectors once to a file beside the output, maps it read-only in chunks and unlinks it; the beam copies a row out of the mapping before scoring it. Levels 2 and above and all adjacency stay on the heap, and the mapping is released when the merge finishes. Cells, descents, beams and the output are unchanged.
* Add failing repro: failed level-0 pass returns while its tasks still run * Add failing repro: hierarchical merge of low-dimension vectors throws * Await in-flight batches before a failed or interrupted pass returns * Cap the wide code's subspace count at the vector dimension
Contributor
Author
|
Merged @jshook's fix PR with no performance impact. @MarkWolters @tlwillke, could you please help review? Thanks! |
jshook
approved these changes
Oct 15, 2026
jshook
left a comment
jshook
left a comment
Contributor
There was a problem hiding this comment.
This has passed extensive operational testing and is shown to be robust under strict memory controls.
MarkWolters
requested changes
Oct 16, 2026
…ow the file precomputeCodeCache reserves scratch past the projected end of the output and extends the file before it maps and encodes. The fused and sidecar strategies recorded the truncate offset only after it returned, so a mapping or encode failure (which falls back to per-write encoding and lets the merge succeed) left the scratch tail after the footer and the output could not be loaded. The cut point does not depend on the pre-encode; record it first.
MarkWolters
approved these changes
Oct 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Compaction: merge graphs with a cell join
Replaces the per-node graph search at level 0 with a scan over cells, so that cross-source candidates are found by streaming contiguous runs of vectors instead of walking each target graph once per node. The main goal is to address larger-than-memory compaction performance.
Scope: Compaction only, additive elsewhere
Nothing outside compaction changes behaviour, and nothing outside compaction has a line removed. Within compaction the merge is substantially rewritten, including a breaking constructor change, which the API section covers.
API change
The compactor now assigns its own output ordinals. Both constructors lose their
List<OrdinalMapper> remappersparameter, and the mapping is published after the fact:This is a breaking change for callers that supplied their own mapping. It is required rather than cosmetic: the cell join needs output ordinals it chose itself, because locality in the output ordinal space is what makes a cell's members contiguous on disk and therefore readable in one sequential pass. A caller-supplied mapping cannot provide that.
Migration for a caller that previously passed
OrdinalMapper.OffsetMapper: drop the argument, and readordinalMappers()afterwards to learn where each source node landed.oldToNewof a deleted node isOrdinalMapper.OMITTED; live ordinals are dense.compact(Path)andcompact(Path, Path)are unchanged.setRefineAfterCompactionis gone, along with the refinement pass it gated.Preconditions unchanged. Every source must still carry
INLINE_VECTORS. Fused-PQ and PQ-sidecar sources are both supported, through the same entry points as before.What changes inside the merge
Cells. The compactor's own upper-layer merge runs first, and its merged level-1 nodes become the cells. Every live node is assigned to the cell its vector descends to.
Ordinals in cell order. Output ordinals are handed out cell by cell, so one source's members of one cell occupy consecutive rows, and a cell is one sequential read.
A scan instead of a search. Each node names the cells it wants, ranked by a width-64 beam over the merged top layer and cut off at 4,096 vectors per larger source. Those requests are inverted so each cell is read once per group and scored against every node that asked for it.
Exactness. Candidate ordering is exact throughout. Approximation appears in exactly two places: which cells a node asks for, and the pairwise comparisons inside diversity selection, which run on vectors decoded from a wide PQ code. The wide code is measurably better there than exact comparisons, by 0.4 points, because coarse codes collide near-duplicates.
Reverse offers. Sources are processed smallest first; a node scans only larger sources and the reverse direction arrives as bounded offers, so the largest source scans nothing.
No system properties are introduced. Behaviour is fixed at the measured operating point: merged cells, probe beam 64, scan budget 4,096.
Results
Three datasets, four partitions each, degree 32, 64 threads, cold page cache per leg, no swap, one run at a time on a 32-physical-core host with 495 GB RAM and NVMe RAID 0. Memory is a cgroup v2
MemoryMaxset as a fraction of the on-disk input index S. Ratio is S divided by that limit, so ratio above 1 means the index does not fit.Speedup over
mainWe use the main branch as our baseline. The table shows baseline compaction time divided by Cell Join compaction time.
Cell Join is faster at every setting measured, including with the whole host available. Values marked
est.are legs where the baseline was stopped once it had run ten times the Cell Join wall time; their throughput is measured over the nodes completed and extrapolated, so they are lower bounds on the gap. 9.72x is the one above-RAM figure where both sides ran to completion.Throughput, nodes per second
mainmainmainCell Join varies by 1.24x to 1.39x across the whole memory range and tracks vector width rather than memory. The baseline falls by 20x to 41x.
Recall@10, and where it costs
Difference is Cell Join minus
main, in points. Blank cells are settings where the baseline was stopped, so it produced no graph to measure.Cell Join gives up 0.3 to 1.0 points of recall on full-precision inputs. The baseline's recall is identical to five decimals at every memory setting, so it is deterministic; Cell Join varies about 0.4 points run to run, from thread scheduling and a per-run wide-code retrain, which puts the smallest deficits inside its own noise.
Quantized inputs, ratio 1.33
On fused inputs the recall deficit disappears, and the speedup shrinks, because the baseline can navigate on codes that are already resident. PQ sidecar is the opposite case and the worst one for the baseline:
mainonly uses codes that are written inline, so a sidecar's codes are retrained and rewritten but never navigated on, and the write pass re-reads and re-encodes every source vector. It pays for them twice and uses them never.Testing
Gate green, 27 tests, zero failures, zero errors, including a
dotProductMultikernel test against the scalar reference, a sidecar-parity recall assertion, and a near-duplicate merge test.Requirements
When the cell join engages, cells come from the merged upper layers, so they need at least one source with a hierarchy. If no hierarchy, level 0 falls back to the existing per-node graph search. The merge is correct either way, only the level-0 strategy changes.