Skip to content

Native: record ids go to the engine as a parameter, not in the statement text - #202

Merged
tae898 merged 2 commits into
mainfrom
perf/native-rid-param
Nov 3, 2026
Merged

tae898 merged 2 commits into
mainfrom
perf/native-rid-param

Conversation

@tae898

@tae898 tae898 commented Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator

Port of humemai/arcadedb-embedded-python#529 (branch perf/native-rid-param, closed there when the bindings moved, never merged) onto current main, narrowed after a first timed A/B. Refs #19. Opened as a draft: a timed A/B decides it before merge.

On the native build a record id was pasted into the SQL text, so each traversal step was a new statement text. The engine's parsed-statement cache is keyed by the statement text (arcadedb.sqlStatementCache, default 300) and its literal parameterizer keeps every literal in a FROM target, so SELECT FROM #1:0 and SELECT FROM #1:1 are two entries; a loop over more than 300 records evicts the statements an application re-runs. In this PR the id goes over as a parameter (the RID the codec already tags as R) for Vertex.get_out_edges/get_in_edges/get_both_edges, Vertex.new_edge, Edge.get_in/get_out, save() (UPDATE :rid MERGE :row) and delete() (DELETE FROM :rid). NativeDatabase.lookup_by_rid keeps main's literal text (see the A/B below). Python only, no image rebuild, and _native/database.py is identical to main. One paragraph on this is in the Record ids in rows section of how-to/databases-and-queries.adoc.

First timed A/B (tools/ab.py, laptop, cores 6 and 7, main's native wheel against the same wheel with this PR's first head 185ef1f26b swapped in, 12 clean pairs per figure, the A/A control clean, guards flat). Ratios are PR over main, below 1 is faster:

figure median 95% CI
lookup_by_rid 1.022 [1.019, 1.025]
get_out_edges('Knows') 0.792 [0.790, 0.794]
Edge.get_in() 0.792 [0.789, 0.794]
lookup_by_rid(...).modify(), set, save() 0.805 [0.801, 0.809]

The pre-written rule dropped the PR because the lookup_by_rid headline got slower. This head therefore narrows the change to the paths that got faster and puts lookup_by_rid back to main's text. new_edge and delete() have no timed figure of their own; they follow the same pattern as the paths that were timed and are kept for that reason. A second run on the narrowed head, with its rule written before it, decides whether this leaves draft.

Untimed counts (25 records, tests/test_native_statement_texts.py, native only and not collected on JPype): lookup_by_rid still one text per record, as on main; get_out_edges("Knows"), get_in_edges() and get_both_edges() over the 24 linked vertices 72 calls and now 3 texts; Edge.get_in/get_out over 24 edges 48 calls and now 2 texts; save() and delete() one text each; new_edge two texts (with and without CONTENT) per label instead of one per pair. The behaviour tests added to tests/test_graph_api.py (traversal over a 12-vertex chain, save and delete by id, new_edge with and without properties, the refusals of lookup_by_rid) run on both builds.

The statement-cache finding was read from the engine source at 26f0bf889b (26.10.x); main now links a newer engine, so the text-count test is also the check that the engine still keeps a record id in a FROM target in the cache key. Draft PRs skip the test-native.yml legs, so the PR's wheel and the parity run come from gh workflow run test-native.yml --ref perf/native-rid-param. Checks run locally on the changed files: compile, pyflakes (only the findings main already has). The tests have not been run on this port; the parity run and the JVM legs are the first run.

🤖 Generated with Claude Code

tae898 and others added 2 commits November 3, 2026 08:22
…ent text

Port of humemai/arcadedb-embedded-python#529 (branch perf/native-rid-param, never merged there) onto current main.

On the native build a record id was pasted into the SQL text, so each traversal step and each lookup_by_rid was a new statement text. The engine caches a parsed statement by its text (arcadedb.sqlStatementCache, 300 entries) and its literal parameterizer keeps a literal in a FROM target, so 25 records meant 25 cache entries. The id now goes over as a parameter (the RID the codec already tags as R): lookup_by_rid (SELECT FROM ?), Vertex.get_out_edges/get_in_edges/get_both_edges, Vertex.new_edge, Edge.get_in/get_out, save() (UPDATE :rid MERGE :row) and delete() (DELETE FROM :rid). Id validation and error messages are unchanged.

test_native_statement_texts.py counts the distinct statement texts (native only, not collected on JPype); the new behaviour tests in test_graph_api.py run on both builds. Untimed counts only: a timed A/B decides it before merge.

Refs #19

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…2 % slower there

First timed A/B (ab.py, laptop, native wheel of main vs this PR, 12 clean pairs): lookup_rid_us 1.022 [1.019, 1.025] (slower), out_edges_us 0.792 [0.790, 0.794], edge_ends_us 0.792 [0.789, 0.794], save_us 0.805 [0.801, 0.809] (faster). Guards flat, A/A clean. Narrow the PR to the paths that got faster; lookup_by_rid returns to main's text. database.py is now identical to main.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tae898

tae898 commented Nov 3, 2026

Copy link
Copy Markdown
Collaborator Author

Second timed A/B, on this narrowed head 3256b013e5 against main 317ae30 (laptop timed lane, ab.py --noise, ABBA pairs, an A/A control; the rule for this run was written in the study before it started, with lookup_rid_us moved from a headline to a guard since lookup no longer changes). Both arms use main's native wheel; arm B swaps in this PR's records.py only.

figure role clean pairs median PR / main 95% CI
lookup_rid_us guard 12 1.015 [1.010, 1.027]
out_edges_us headline 12 0.785 [0.780, 0.790]
edge_ends_us headline 12 0.790 [0.787, 0.795]
save_us headline 12 0.857 [0.853, 0.864]
ctl_first_us guard 12 1.002 [0.995, 1.006]
ctl_count_us guard 12 0.993 [0.983, 1.002]

Result: ok (all 6 B/A figures have >= 10 clean pairs and CI width <= 0.08, the A/A control is clean). Verdict by that rule: SHIP. The edge getters and edge ends are about a fifth faster and save about a seventh, because every record now shares one cached statement text instead of one per record.

One figure to keep in view: lookup_rid_us reads 1 to 3 % slower although lookup_by_rid and the record wrapper are unchanged in this PR (the diff touches only the edge, save, delete and new_edge statements). It is inside the guard band, so the rule passes it; my guess is an indirect effect of the benchmark's statement-cache state from the figures that run before it in the same process, not a cost of this change. If someone wants it pinned down, I can rerun the lookup figure alone on both arms.

@tae898
tae898 marked this pull request as ready for review November 3, 2026 01:20
@tae898
tae898 merged commit a3d151c into main Nov 3, 2026
98 checks passed
@tae898
tae898 deleted the perf/native-rid-param branch November 3, 2026 02:00
tae898 added a commit that referenced this pull request Nov 3, 2026
…both native-only test ignores)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tae898 added a commit that referenced this pull request Nov 3, 2026
Conflicts resolved:
- src/arcadedb/_native/records.py: keeps this branch's _save_update (lazy fingerprint, refuses a stale overwrite) and applies #202's rule inside it: the record id goes to the engine as the :rid parameter for both the stored-record read (_SELECT = "SELECT FROM :rid") and the UPDATE .. MERGE .. RETURN AFTER (_UPDATE now carries RETURN AFTER, which only _save_update uses). Without this, #202's test_native_statement_texts fails on the pasted ids in the save path.
- tests/conftest.py, tests/test_graph_api.py: both sides appended; kept both. The duplicate `import pytest` from the two sides is dropped and main's import order kept (local isort disagrees with main's CI order; left as main has it).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tae898 added a commit that referenced this pull request Nov 3, 2026
No textual conflicts: main's #202 (record ids as a :rid parameter, src/arcadedb/_native/records.py and database.py) and #201 (run_in_transaction witness) touch other hunks than this branch. The record-id rule of #202 is kept: this branch pastes no record id into SQL text (the lazy per-hit vector load, if present, goes through lookup_by_rid, which keeps its literal id by design, see tests/test_native_statement_texts.py).

Static checks after the merge: compile() and pyflakes of the changed files (the unused typing.Optional in records.py is on main too), pre-commit black and isort pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tae898 added a commit that referenced this pull request Nov 3, 2026
No textual conflicts: main's #202 (record ids as a :rid parameter, src/arcadedb/_native/records.py and database.py) and #201 (run_in_transaction witness) touch other hunks than this branch. The record-id rule of #202 is kept: this branch pastes no record id into SQL text (the lazy per-hit vector load, if present, goes through lookup_by_rid, which keeps its literal id by design, see tests/test_native_statement_texts.py).

Static checks after the merge: compile() and pyflakes of the changed files (the unused typing.Optional in records.py is on main too), pre-commit black and isort pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant