Repository navigation
JVM: keep NaN, inf and -inf in to_json_list and in nested values of the columnar readers - #181
Merged
Merged
Conversation
… the columnar readers
The engine's JSON writers turn non-finite doubles into null, so the JVM build
returned None from to_json_list and, inside a list or a map, from to_columns,
to_dataframe and to_arrow (native keeps them). RowBatcher and ColumnBatcher now
replace each one with the tag {"\u0001": ["f", "NaN"]} and results.py turns it back
into a float, only when the tag character is in the text, so batches without a
non-finite value parse as before. to_json_list returns Python floats, as native.
Java is not compiled here; CI build is the compile check.
Against the main 26.11.1.dev0 wheels the JVM wheel fails the to_json_list, iter_json_batches and nested list/map tests (71 failed) and the native wheel passes the whole file (102 passed). The Java change is compiled and tested by CI.
Closes #179
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#180 writes nested list/map values of the columnar readers with the TypedRows tagged encoding and decodes them with _typed_hook, the path to_list() uses, which keeps NaN and the infinities; its version of the json column (ColumnBatcher and results.py) is kept, and this branch's keepNonFinite/_loads_nonfinite now cover to_json_list and iter_json_batches (RowBatcher). tests/test_nonfinite_floats.py still checks every reader. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tae898
marked this pull request as ready for review
November 1, 2026 13:18
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #179
JVM build only (the native readers already return non-finite floats).
src/java/com/arcadedb/python/RowBatcher.java:keepNonFinitereplaces a non-finiteDouble/Float(recursing through collections, maps and float/double arrays, copying only when one is found) with the tag the typed rows use; called inappendRow(to_json_list,iter_json_batches) and inColumnBatcher's json branch.src/arcadedb/results.py:_loads_nonfinitedecodes the tag tofloat('nan')/float('inf')/float('-inf'), and only takes the hook path when a batch contains the tag, so plain batches keep the C decoder.to_json_listreturns Python floats, as on native; its docstring says the result is not strict JSON.tests/test_nonfinite_floats.py(NaN, inf, -inf; top level, in a list, in a map; every reader; rows around a non-finite one unchanged; a string holding the tag character is not turned into a float).Verification: against the 26.11.1.dev0 wheels of main the JVM wheel fails the to_json_list, iter_json_batches and nested tests (71) and the native wheel passes the whole file (102). Java is compiled and tested by this PR's CI. Updated after #180 merged: main was merged into this branch. For nested list/map values of the columnar readers #180's tagged encoding (TypedRows +
_typed_hook) is kept, which keeps non-finite floats the wayto_list()does; this PR'skeepNonFinite/_loads_nonfinitenow coverto_json_listanditer_json_batches(RowBatcher). The tests still check every reader.🤖 Generated with Claude Code