Repository navigation
Read zarr v2 NWB files with zarr-python 3 - #4875
Open
h-mayorquin wants to merge 2 commits into
Open
h-mayorquin wants to merge 2 commits into
h-mayorquin wants to merge 2 commits into
Conversation
Route zarr v2 NWB files, local or streamed with stream_mode="zarr", to pynwb, where hdmf-zarr's NWBZarrV2IO keeps the backward compatibility, so the in-house reader only needs to read zarr v3.
With zarr-python 2 the in-house reader reads zarr v2 files, and the pynwb streaming path passes an FSStore that hdmf-zarr<0.14 rejects. Raise a clear error when hdmf-zarr is missing, and stop skipping the zarr streaming test with zarr-python 3.
Member
|
Thansk @h-mayorquin, that sounds good! Is it ok to first merge #4260 and then remove the added logic to use this one? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR ensures that we can read NWB zarr files in the zarr v2 format, written by hdmf-zarr<0.14, as we move forward to zarr-python 3. The problem currently is that zarr-python 3 cannot parse some of the arrays in these old files (variable-length string columns with an integer fill value, for example), so opening them fails.
What this PR does is use the backward compatibility of the pynwb reader to handle the old files: hdmf-zarr reads them with
NWBZarrV2IO. That way, our in-house zarr reader remains simpler, since it only needs to support zarr v3. The format is detected from the root metadata file (zarr.jsonor.zgroup) without opening the folder, both for local files and for files streamed withstream_mode="zarr", and v2 files are routed to pynwb with a warning, since it can be slower.NWBZarrV2IOis called withallow_pickle=Truebecause hdmf-zarr<0.14 pickled object references, such as thegroupcolumn of the electrodes table, so most of these files cannot be read without it. Since hdmf-zarr 0.14 requires Python 3.12, reading v2 files with zarr-python 3 needs Python 3.12 or newer; on Python 3.11 the in-house reader still reads v3 files.This is a different approach from #4260, which adds zarr v3 support and still reads v2 files with our in-house reader, working around zarr-python 3: it lists the store directly and skips the children that cannot be parsed. With this PR, those workarounds are no longer needed.
I went with the simplest way to unblock reading these files. If users of the old datasets need faster reading, we can add backward support to the in-house reader later, backed by a benchmark that shows the gain is worth the extra code.