native: Highway is the default vector provider where the wheel ships its library

docs/modules/ROOT/pages/how-to/vectors.adoc

@@ -292,6 +292,15 @@ print(notes.get_size())
 4
 ----
 
+[[native-provider]]
+== Distance kernels in the native package
+
+The two packages compute vector distances differently. `{jvm-pkg-name}` runs jvector's Panama kernels on the Java Vector API. `{native-pkg-name}` is built without the Vector API, so on Linux x86_64 and aarch64 it uses jvector's Highway kernels, a native library the wheel ships next to the engine, and on macOS and Windows jvector's scalar Java kernels.
+
+To choose, set `ARCADEDB_VECTOR_PROVIDER` before the first database is opened in the process: `native` for Highway (the default where the wheel ships it) or `default` for the scalar kernels. In our laptop measurement (medians, 100,000 vectors, `26.11.1.dev0`) Highway built indexes 2.2 to 2.5 times faster and answered queries 14 to 30 percent faster than the scalar kernels at 768 and 1536 dimensions; at 128 dimensions the build took the same time and queries were 6 percent slower. Both return the same results.
+
+For vector search the JVM package is faster still: see xref:which-package.adoc#vector-search[Which package?].
+
 == Next steps
 
 * xref:how-to/databases-and-queries.adoc[Databases and queries]

docs/modules/ROOT/pages/which-package.adoc

@@ -13,7 +13,7 @@ Two packages give you the ArcadeDB engine inside your Python process. Both use t
 
 | `{jvm-pkg-name}`
 | The JVM build. The Java engine runs in your process through https://jpype.readthedocs.io/[JPype], on a bundled Java 25 runtime.
-| Heavy, long-running loads, where the JIT compiler makes the engine faster over time. Also the Java-object APIs (`get_java_database` and similar), JVM options, `jvm_kwargs` and `start_jvm`.
+| Heavy, long-running loads, where the JIT compiler makes the engine faster over time, and vector search where index build and query speed matter more than memory (see <<vector-search>>). Also the Java-object APIs (`get_java_database` and similar), JVM options, `jvm_kwargs` and `start_jvm`.
 |===
 
 [source,shell,subs="attributes+"]
@@ -72,6 +72,10 @@ The API is the same. The runtime is not. These are the differences you can plan
 | Measure on your own workload.
 | Measure on your own workload.
 
+| [[vector-search]]Vector search (`LSM_VECTOR`)
+| Slower to build and to query, smaller in memory. Against the JVM build, index builds took 1.28 to 1.62 times as long and median queries 1.22 to 1.87 times as long, and peak memory was 0.51 to 0.67 of the JVM build's (medians, 100,000 vectors of 128, 768 and 1536 dimensions, `26.11.1.dev0`, one laptop, with the Highway kernels that are the default on Linux, see xref:how-to/vectors.adoc#native-provider[Distance kernels in the native package]).
+| Faster to build and to query, larger in memory.
+
 | Wheel size
 | Smaller. It carries no Java runtime.
 | Larger. It carries a Java 25 runtime for its platform.

src/arcadedb/_native/lib.py

@@ -90,6 +90,16 @@ def _log_isolate_args(user_args, log_dir) -> list:
 _HEAP_ARGS = ("-Xmx", "-XX:MaximumHeapSizePercent=", "-XX:MaxRAM=")
 
 
+def _default_vector_provider(libdir: Path, env) -> None:
+    """Pick jvector's Highway kernels (ARCADEDB_VECTOR_PROVIDER=native) where the wheel ships their library, libjvector-<arch>.so next
+    to the engine library (Linux x86-64 and aarch64). Measured in #153: faster index builds and queries at 768 and 1536 dimensions, the
+    same build and slightly slower queries at 128, the same results. Without the library the engine keeps jvector's scalar provider.
+    A value the user set wins; ARCADEDB_VECTOR_PROVIDER=default forces the scalar provider.
+    """
+    if not env.get("ARCADEDB_VECTOR_PROVIDER") and any(libdir.glob("libjvector-*.so")):
+        env["ARCADEDB_VECTOR_PROVIDER"] = "native"
+
+
 def _heap_isolate_args(user_args) -> list:
     """The isolate argument that limits the heap to ``HEAP_PERCENT`` of the memory; none when the user already sets the heap (theirs wins).
 
@@ -151,6 +161,7 @@ def _load():
         _check_cpu()
         # jvector's Highway library (libjvector-<arch>.so) is looked for next to the engine library
         os.environ.setdefault("ARCADEDB_NATIVE_LIBDIR", str(path.resolve().parent))
+        _default_vector_provider(path.resolve().parent, os.environ)
         lib = ctypes.CDLL(str(path))
         isolate = ctypes.c_void_p()
         thread = ctypes.c_void_p()

tests/conftest.py

@@ -54,6 +54,8 @@
     collect_ignore.append("test_native_async_callbacks.py")
     # the short call path of a point read (codec.encode_params, lib.exec_query, first()), observed on the native library
     collect_ignore.append("test_native_point_read.py")
+    # the vector provider the native glue picks by default (ARCADEDB_VECTOR_PROVIDER, #153), observed on the native library
+    collect_ignore.append("test_native_vector_provider.py")
 if not os.path.isfile(
     os.path.join(os.path.dirname(os.path.dirname(__file__)), "docs", "index.md")
 ):

tests/test_native_vector_provider.py

@@ -0,0 +1,72 @@
+"""Native build: jvector's Highway kernels are the default vector provider where the wheel ships their library (#153).
+
+The engine forces jvector's scalar provider in every native image unless ARCADEDB_VECTOR_PROVIDER says otherwise. The glue sets it to
+``native`` (Highway) when libjvector-<arch>.so sits next to the engine library, which the wheel ships on Linux x86-64 and aarch64. A
+value the user set wins, and ``default`` forces the scalar provider.
+"""
+
+import json
+import os
+import subprocess  # nosec B404 - test-controlled child process
+import sys
+
+import pytest
+
+from arcadedb._native import lib as native_lib
+
+CHILD = """
+import json
+from arcadedb._native import lib
+print(json.dumps(lib.diag()))
+"""
+
+
+def test_highway_is_picked_when_its_library_is_next_to_the_engine(tmp_path):
+    (tmp_path / "libjvector-x86_64.so").write_bytes(b"")
+    env = {}
+    native_lib._default_vector_provider(tmp_path, env)
+    assert env == {"ARCADEDB_VECTOR_PROVIDER": "native"}
+
+
+@pytest.mark.parametrize("value", ["default", "panama", "native"])
+def test_a_value_the_user_set_wins(tmp_path, value):
+    (tmp_path / "libjvector-x86_64.so").write_bytes(b"")
+    env = {"ARCADEDB_VECTOR_PROVIDER": value}
+    native_lib._default_vector_provider(tmp_path, env)
+    assert env == {"ARCADEDB_VECTOR_PROVIDER": value}
+
+
+def test_without_the_library_the_scalar_provider_stays(tmp_path):
+    env = {}
+    native_lib._default_vector_provider(tmp_path, env)
+    assert env == {}
+
+
+def _provider(tmp_path, value=None) -> str:
+    """The VectorizationProvider class an isolate of a fresh process runs, with ARCADEDB_VECTOR_PROVIDER unset or set to ``value``."""
+    env = {k: v for k, v in os.environ.items() if k != "ARCADEDB_VECTOR_PROVIDER"}
+    env["ARCADEDB_LOG_DIR"] = str(tmp_path / "log")
+    if value is not None:
+        env["ARCADEDB_VECTOR_PROVIDER"] = value
+    result = subprocess.run(  # nosec B603 - fixed argument list, no shell
+        [sys.executable, "-c", CHILD],
+        cwd=tmp_path,
+        env=env,
+        capture_output=True,
+        text=True,
+        timeout=300,
+    )
+    assert result.returncode == 0, result.stderr[-2000:]
+    return json.loads(result.stdout.strip().splitlines()[-1])["provider"]
+
+
+def test_the_default_provider_follows_the_shipped_library(tmp_path):
+    shipped = any(native_lib.library_path().resolve().parent.glob("libjvector-*.so"))
+    expected = (
+        "NativeVectorizationProvider" if shipped else "DefaultVectorizationProvider"
+    )
+    assert _provider(tmp_path).endswith(expected)
+
+
+def test_default_forces_the_scalar_provider(tmp_path):
+    assert _provider(tmp_path, "default").endswith("DefaultVectorizationProvider")