[tae898]

DRAFT. Prototype of backlog row N-R2: query results as columns, the read-side counterpart of the columnar insert (#514). Untimed work only: the design note, the tests, a JVM-side gate and one image build with an untimed before/after probe (queued, see Pending). A timed ab.py A/B is queued before any merge; do not merge before it.

STACKED ON #521 (which is stacked on #520): the reply code in ArcadeNative (replyOf, withFloatTail, out) is what those two PRs change, and the JSON fallback of this change is #521's arcadedb_n_next_bin reply, so this branch starts from native-next-float-tail. The first four commits are #520's and #521's; review the last one. When #520 and #521 merge this PR needs only a rebase.

Why (found by the study in .notes/bench/repros/native-result-transport-20261025, local notes repo): a warm query is not the engine alone. Point lookups: engine 27-41%, JSON encode 11-20%, Python JSON decode 18-24%, other (ctypes call, the arcadedb_free crossing, string_at, dict building) 27-30%. Wide SELECTs: Python decode about 50% (json.loads with the object_hook), encode 8-36%. The byte-loop copy is 0-2% (#520 is a cleanup). The target is the cost of building the Python objects from columns, the "columnar floor" of that study.

Change:

• New entry points arcadedb_n_exec_col and arcadedb_n_next_col. A batch is answered as one frame: a 32-byte header, a small JSON header (column names, kinds, offsets), one buffer per column in the layout of arcadedb_n_insert_colp (int64, float64, bool bytes, Arrow string offsets plus UTF-8 blob, float32 vectors, validity and presence bitmaps; dates as days or microseconds; record ids as text), the record meta as three more columns. ColResult.java; the frame is documented at its top and in the design note native-columnar-results-20261025/README.md.
• Heterogeneous records: one column per property (union of names), a validity bitmap (value or null) and a presence bitmap (property exists), and a per-row order table when a row lists its properties in another order than the columns. {"a": None} and {} stay different.
• JSON per batch when the columns cannot carry the rows: nested map, list or record, BigDecimal/BigInteger/Set/other arrays, a column that mixes kinds (int and double), a year outside 1-9999, an unpaired surrogate, more than 256 columns, a table less than a quarter full, or fewer than ARCADEDB_NATIVE_COLRES_MIN rows (default 16: a point lookup gains nothing from a frame). The JSON batch is Native: later result batches carry vectors in the float32 tail (arcadedb_n_next_bin) (draft, stacked on #520) #521's reply, vectors in the float32 tail.
• No per-call overhead that the columns cannot lose: one crossing, and Python hands the call a reusable per-thread reply buffer (512 KB) that the image writes into with one bulk copy, so a reply that fits needs no malloc, no arcadedb_free crossing and no strlen. A reply that does not fit, and every error, comes in a malloc'd block that Python copies out and frees (the returned pointer says which).
• Python: _native/colres.py (the decoder: memoryview.cast plus tolist(), numpy only for big vector matrices), lib.exec_col / lib.next_col, NativeDatabase._exec and NativeResultSet._fill use them for queries when the library has them. ARCADEDB_NATIVE_COLRES=0 forces the JSON text of Native: later result batches carry vectors in the float32 tail (arcadedb_n_next_bin) (draft, stacked on #520) #521 in the same image; ARCADEDB_NATIVE_COLRES_MIN=<n> sets the smallest columnar batch. Result objects, to_list, to_columns, to_dataframe and the cursor are unchanged. Old entry points are unchanged, so an older Python on this image keeps working.

Local checks, done:

• Tests first (bindings/python/tests/test_native_columnar_results.py, 8 tests): mixed types with nulls, unicode and control characters, ints at 2^31/2^53/2^62 and the int64 edges, doubles with -0.0/inf/nan/1e-300/MAX/MIN, float32 vectors of 0/3/8/32 elements and missing, record ids, DATE and DATETIME, record rows with meta, missing and null properties, results the columns cannot carry, point lookups, empty results, 5,000 rows in several batches, a cursor read row by row; each query runs in JSON mode, in columns for every batch and with the default, and the three must agree bit for bit (types and key order included) and match the inserted literals. Against the unmodified Native: later result batches carry vectors in the float32 tail (arcadedb_n_next_bin) (draft, stacked on #520) #521 image: 6 pass, the 2 asserts that the columnar entry points were used fail (red), as intended.
• JVM gate (jvm_gate.sh in the notes folder): the encoder on a JVM against the engine jars, 18 queries incl. dense and sparse records, projections with nulls, nested lists, mixed int/double, batches of 5-7 rows; the Python decoder must reproduce rows, meta and drained of the old NativeCodec.appendBatch text of the same batch: 0 mismatches; every fallback is one of the listed reasons.
• Maven compile gate of native-spike on the final tree: passes. black and isort via pre-commit: pass.

Found by the first probe run and fixed here (commit b5012e2): the record meta columns (@rid, @type) wrote a validity bitmap out of an 8-byte array, so a record-row batch of more than 64 rows failed (IndexOutOfBoundsException: Range [0, 0 + 32) out of bounds for length 8). The 20-row cases and the projection tests had hidden it. Tests first: test_record_rows_of_several_batches and test_the_probes_wide_table_in_every_mode failed with that error on the image built before the fix; the JVM gate (700 rows per table, batches of 256 and 2048) reproduced it. Then one rebuild (2m33s, XMX=8g, cores 8-15) of this tree.

Local checks on the fixed image: the new test file and the #520/#521 transport tests, 16 passed with ARCADEDB_NATIVE_COLRES=1 and 16 passed with =0; JVM gate 0 mismatches; Maven compile gate; black and isort. Digests of decoded rows (5,000 mixed rows incl. nulls, unicode, ints at 2^31/2^53, -0.0/inf/nan, vectors of every length, dates, sparse records, nested values, point lookup, aggregation): JVM wheel = native JSON = native columns = native default on all six queries (record ids compared by shape across wheels; the RID vs DatabaseRID type difference is backlog P-02 and untouched, exact text equal across the three native modes).

Measured (2026-10-25, untimed, laptop, 2 runs per arm; stage clocks, not an ab.py A/B; the timed A/B is queued before any merge)

Microseconds per warm query (mean of runs). wall = what a user pays; engine = engine-only pass (varies +-10% between arms); fetch = foreign calls plus copy and free; py decode = text or column decode; other = the rest.

query	arm	runs	wall A	engine C	fetch	py decode	other	reply KB	frames	json batches	A vs JSON
pt_sql_rec	JSON (COLRES=0)	2	38.5	10.8	23.0	6.5	10.1	0.3	0	0	100%
pt_sql_rec	columns, min 16 rows (default)	2	39.8	10.8	23.5	6.2	8.3	0.2	0	40,200	103%
pt_sql_rec	columns, min 1 row	2	45.5	10.3	22.6	12.6	9.2	0.5	40,200	0	118%
pt_sql_scalar	JSON (COLRES=0)	2	30.4	20.7	20.3	4.6	9.8	0.1	0	0	100%
pt_sql_scalar	columns, min 16 rows (default)	2	30.4	13.0	19.5	4.3	7.3	0.1	0	80,400	100%
pt_sql_scalar	columns, min 1 row	2	37.3	13.9	20.9	7.3	8.5	0.2	80,400	0	122%
pt_cypher	JSON (COLRES=0)	2	27.3	11.0	16.2	4.3	8.7	0.1	0	0	100%
pt_cypher	columns, min 16 rows (default)	2	25.9	10.2	16.1	4.1	6.8	0.1	0	100,600	95%
pt_cypher	columns, min 1 row	2	31.7	10.8	18.7	7.4	8.4	0.2	100,600	0	116%
hop2_cypher	JSON (COLRES=0)	2	409.6	350.1	323.2	56.0	21.1	6.1	0	0	100%
hop2_cypher	columns, min 16 rows (default)	2	338.1	323.4	282.0	39.1	18.3	3.1	3,050	100,600	83%
hop2_cypher	columns, min 1 row	2	367.6	343.4	306.2	41.3	19.8	3.1	103,650	0	90%
agg_sql	JSON (COLRES=0)	2	57,453.8	60,519.9	58,662.9	36.8	34.2	0.7	0	0	100%
agg_sql	columns, min 16 rows (default)	2	53,226.1	53,394.1	54,631.5	39.6	24.9	0.7	3,050	100,683	93%
agg_sql	columns, min 1 row	2	56,927.2	54,846.1	54,918.9	46.3	24.7	0.5	103,733	0	99%
wide10k_proj	JSON (COLRES=0)	2	87,400.6	64,877.0	44,543.5	36,128.5	3,897.2	2,464.7	0	0	100%
wide10k_proj	columns, min 16 rows (default)	2	55,089.6	58,810.5	32,370.9	18,475.4	3,619.6	1,842.4	3,308	100,683	63%
wide10k_proj	columns, min 1 row	2	57,078.2	53,558.2	33,021.6	18,596.0	3,630.2	1,842.4	103,991	0	65%
wide10k_star	JSON (COLRES=0)	2	90,555.6	11,188.6	43,232.6	39,720.3	4,564.7	2,648.7	0	0	100%
wide10k_star	columns, min 16 rows (default)	2	65,183.5	10,237.9	31,045.5	29,427.6	3,334.9	2,016.9	3,566	100,683	72%
wide10k_star	columns, min 1 row	2	64,230.7	10,454.3	30,227.6	28,437.9	3,190.0	2,016.9	104,249	0	71%
wide100k_proj	JSON (COLRES=0)	2	912,784.8	569,245.3	397,042.2	442,831.7	39,942.7	24,748.8	0	0	100%
wide100k_proj	columns, min 16 rows (default)	2	680,276.5	577,715.0	313,234.3	266,424.6	109,498.8	18,422.0	4,266	100,683	75%
wide100k_proj	columns, min 1 row	2	655,157.1	516,231.4	305,886.7	263,328.9	109,046.6	18,422.0	104,949	0	72%

Reading: wide SELECTs take 63% (wide10k_proj), 72% (wide10k_star) and 75% (wide100k_proj) of the JSON arm's wall (Python decode 36.1 -> 18.5 ms and 443 -> 266 ms for the projections). Point lookups with the default min of 16 (JSON text in the reused buffer) are 95-103% of JSON: no visible change untimed. One-row frames (min 1) are 16-22% slower, so the default minimum stays above 1. 2-hop Cypher (100 rows) 83%, aggregation 93% (engine-dominated, inside the pass-to-pass noise). Peak RSS 1.05-1.15 GB in every arm. Untimed and indicative: the timed ab.py A/B decides the defaults (rule in the design note native-columnar-results-20261025/README.md, written before any measurement).

🤖 Generated with Claude Code

[tae898]

Closing here: the bindings moved to ArcadeData/arcadedb-python. The branch stays for the queued A/B native-colres-ab-laptop-20261025; if it passes, the change is ported there.