Prevent temporary SIMD allocations in Panama vector kernels

[tlwillke]

Make SIMD species explicit throughout Panama kernels, including intermediate arithmetic results, so the JIT can eliminate temporary vector objects and backing arrays without relying on caller inlining. Resolve heap arrays once outside arithmetic loops where applicable. This problem seems specific to JDKs prior to 24/25.

Covers PQ/NVQ encoding and scoring, dot product, L2, cosine, arithmetic, reductions and Hamming distance. Preserves arithmetic and reduction order, APIs, provider selection, the on-disk format and the VectorAccess callback.

Validation includes scalar-oracle and bitwise regression tests, offsets/tails, aliasing, special values, concurrency, and fresh Ada002, MPNet, Cohere and OpenAI 3072 1M builds and queries.

Across the four 1M datasets, cumulative GC-reclaimed heap during query passes fell 54–84% with one thread. With eight threads, it fell 61–83% for MPNet, Ada002 and OpenAI3072, while Cohere remained essentially unchanged.

Panama query throughput improves 22.7–38.2% with one thread and 8.7–103.7% with eight threads across the reported recall@10 settings. The fix keeps concrete SIMD types visible to HotSpot so temporary vector objects and backing arrays can be eliminated, including on the resident VectorAccess encoding path.

Matched benchmark configuration

Tested main 201dd576 versus allocation arm cb0f5fb5. Native disabled; Panama, JDK 23, preferred 512-bit species, -Xmx128g, resident inputs, DOT_PRODUCT, FusedPQ enabled, PQ subspace width 8/K=256/no centering/unweighted, NVQ two subvectors, V6, degree 32/efConstruction 100/overflow 1.2, hierarchy and refinement enabled, pruning disabled. Builds use 48 insertion, cleanup and writer workers; queries use CPU 0 or CPUs 0–7. All 10,000 queries are evaluated for MPNet/Ada002/Cohere and all 1,000 for OpenAI3072.

Reproduce: Compare current main with this branch using JDK 23 and the configuration above. In both arms, use --add-modules=jdk.incubator.vector, -Djvector.experimental.enable_native_vectorization=false, and -Djvector.parallel_writer=true. Measure cached queries with dedicated 1/8-worker passes; collect memory/GC separately from clean QPS passes.

Every table cell below is main → this PR. Each arm trains a fresh independent codebook and builds its own graph.

1-thread queries

Dataset	QPS, 10@1×	QPS, 10@2×	Recall@10, 1× (%)	Recall@10, 2× (%)	GC reclaimed (GiB)
MPNet 1M	6,302 → 8,711	5,574 → 7,244	62.501 → 62.950	79.711 → 79.907	29.18 → 4.64
Ada002 1M	3,455 → 4,691	3,097 → 4,041	53.291 → 54.004	73.591 → 74.551	24.23 → 7.72
Cohere 1M	5,247 → 6,889	4,618 → 5,738	58.912 → 58.718	78.200 → 78.077	14.87 → 5.83
OpenAI3072 1M	1,883 → 2,311	1,761 → 2,212	62.690 → 63.950	78.990 → 79.940	26.52 → 12.10

8-thread queries

Dataset	QPS, 10@1×	QPS, 10@2×	Recall@10, 1× (%)	Recall@10, 2× (%)	GC reclaimed (GiB)
MPNet 1M	54,918 → 59,700	43,993 → 53,205	62.501 → 62.950	79.711 → 79.907	25.84 → 4.48
Ada002 1M	28,870 → 35,733	24,621 → 30,324	53.291 → 54.004	73.591 → 74.551	27.53 → 6.49
Cohere 1M	37,011 → 49,910	31,300 → 42,844	58.912 → 58.718	78.200 → 78.077	4.87 → 4.88
OpenAI3072 1M	10,380 → 18,114	8,349 → 17,010	62.690 → 63.950	78.990 → 79.940	27.00 → 10.48

GC is cumulative reclaimed Java heap from separate fixed-workload memory passes covering all five standard search settings; it is not resident memory or allocated bytes. QPS comes from clean throughput passes.

Recall and visited counts match exactly between one and eight threads within every arm. Independent codebooks and concurrent graph construction can change recall between runs. Controlled checks found identical recall/visited on the same 1M indexes with either kernel implementation and byte-identical 100k graphs with a shared codebook, codes and serial insertion order. Scalar-oracle and bitwise kernel tests also passed.

[github-actions]

Before you submit for review:

• Does your PR follow guidelines from CONTRIBUTIONS.md?
• Did you summarize what this PR does clearly and concisely?
• Did you include performance data for changes which may be performance impacting?
• Did you include useful docs for any user-facing changes or features?
• Did you include useful javadocs for developer oriented changes, explaining new concepts or key changes?
• Did you rebase your branch onto the latest main for regression testing and PR submission?
• Did you trigger regression testing via Run Bench Main and review results?
• Did you adhere to the code formatting guidelines (TBD)
• Did you group your changes for easy review, providing meaningful descriptions for each commit?
• Did you ensure that all files contain the correct copyright header?
• Did you add documentation for this feature to the release notes directory?

If you did not complete any of these, then please explain below.

[ashkrisk]

I'm finding it difficult to reproduce the results on my System. Here are the results I'm seeing right now:

This compares QPS vs recall for cohere-english-v3-1M with JDK 21 and JDK 25. In both cases there isn't too much difference between the two branches.

Details

main commit: 598ef09e8
fix-alloc commit: 3270244b7
BenchYAML config: config.yaml (PQ + NVQ)
Java heap: -Xmx40G
Machine: Intel GNR with 24 cores / 48 threads
metric: DOT_PRODUCT
Java flags:

    "--add-modules=jdk.incubator.vector"
    "-Djvector.experimental.enable_native_vectorization=false"
    "-Djvector.parallel_writer=true"
    "-ea"

I saw similar results using mpnet-1m. Is there something I'm missing here?

More charts

[MarkWolters]

My only concern is specified in the comment I left. Once that is cleared up for me I will approve.

[r-devulap]

Same problem for me, I am unable to reproduce the numbers with the flags specified here. I dont see anything different in microbenchmarks either for dotProduct/l2 :

Dims	Main (Panama) ns/op	Fix PQ Vector Allocations (Panama) ns/op	Delta (ns)	Delta (%)
1024	44.566	45.916	+1.350	+3.03%
3072	124.102	129.562	+5.460	+4.40%

For ada1m, on a x8i.8xlarge Intel GNR with 16 cores/32 threads: QPS for topk=10:

Overquery	Main QPS	Fix PQ Vector Allocations QPS	Delta QPS	Delta %
1	83,076.6	84,696.0	+1,619.4	+1.95%
2	66,487.7	71,372.6	+4,884.9	+7.35%
5	47,034.4	45,503.3	-1,531.1	-3.26%
10	29,690.4	29,411.3	-279.1	-0.94%

Am i missing something here? I am running with JDK 25 and with these flags: --add-modules=jdk.incubator.vector, -Djvector.experimental.enable_native_vectorization=false, and -Djvector.parallel_writer=true.

[tlwillke]

@ashkrisk On Cohere 1M with standard BenchYAML, the PR reduced query-window GC reclaimed from 27.57 to 1.42 GiB on JDK 23. On JDK 25, main already showed no sampled SIMD boxing: GC reclaimed changed only from 1.44 to 1.35 GiB, with modest QPS gains. The benefit therefore depends on the JDK and workload. This is what you are seeing.

The audit also found that added type casts made some kernels too large to inline. I have validated a compact fix for PQ gathering and am addressing the remaining cases, including NVQ, while retaining the explicit SIMD types and allocation fixes. Validation continues on both JDKs.

I still think it is worth pursuing since we have users that are not on JDK 25.

PS - the impact on low overquery cases is more profound and of more practical importance.

[tlwillke]

@r-devulap please see my response to @ashkrisk .