Repository navigation
Resolve an empty store name to the default store - #1575
dimitri-yatsenko wants to merge 4 commits into
Conversation
The bare-@ forms (<npy@>, <object@>, and the other store codecs) keep the store name as "" so a codec can tell "store requested" from "no store". Config.get_store_spec treated only None as the default, so inserting, fetching, or collecting such a column raised "Storage '' is requested but not configured in stores". An empty name now resolves to stores.default, or to stores.filepath_default for filepath references. The garbage collector compared each reference's recorded store name with the store it collects. Bare-@ references record "", so a collector bound to the default store by name counted their files as orphans and, with dry_run=False, deleted files that existing rows still reference. A reference with an empty or missing store name now matches the default store.
An empty store name means the default store only when stores.default is set. Without it, get_store_spec and an insert into a <npy@> column must raise "stores.default is not configured" rather than pick some store.
|
The default-store resolution and the named-collector fix look right to me, and the destructive GC test pins the second one nicely. One thing I noticed while reading, which I may be missing context on: before this change, GarbageCollector(schema, store="") failed in the constructor because get_store_spec("") raised. Now it builds (the backend resolves to the default store), but self.store stays "", so in _references neither A smaller question on semantics: bare-@ references record store "" rather than the store they were written to, so they follow whatever stores.default is at read or collect time. Is that intended if stores.default is later re-pointed? |
… collector GarbageCollector(schema, store="") built a collector whose store stayed "", so _references matched no reference and collect(dry_run=False) would have deleted every file in the default store. An empty name now builds the default collector, as None does. Codecs recorded the store name as declared, so a bare-@ reference stored "" and followed stores.default: re-pointing the default would break reads, and a collector bound to the old store by name would delete live files. Codecs now record the resolved name (new Config.resolve_store_name, which get_store_spec also uses), so each object stays with the store it was written to. Bare @ could not be written before this change, so no stored metadata has an empty name; metadata that records none still means the current default.
|
Thanks, both points were right. Fixed in 52eff8f. Empty store name in the collector. Bare- New tests check that the metadata records the resolved name, and that after re-pointing |
Fixes #1574.
Why
The bare-
@store forms (<npy@>,<object@>, and the other store codecs) are documented to use the default store, but every insert, fetch, or garbage collection on such a column raisedStorage '' is requested but not configured in stores. The declaration keeps the store name as""so a codec can tell "store requested" from "no store";Config.get_store_specresolved the default only forNone.Once such columns can be written, a second problem appears:
GarbageCollectorcompared each reference's recorded store name with the store it collects. Bare-@references record"", so a collector bound to the default store by name would have counted their files as orphans, andcollect(dry_run=False)would have deleted files that existing rows still reference.What changes
Config.get_store_spec: an empty store name resolves likeNone, tostores.default, or tostores.filepath_defaultfor filepath references.GarbageCollector._references: a reference whose recorded store name is empty or missing matches the default store.Tests
get_store_spec("")resolves to the default store and, for filepath references, to the filepath default; with nostores.defaultconfigured it raisesstores.default is not configured.TestBareAtUsesDefaultStoreintest_gc.py): an insert into a<npy@>column raises when no default store is configured; a<npy@>column round-trips through the default store; and a destructive collection of the default store by name keeps the row's file and reports it as referenced. Without thegc.pychange this test fails because the file is deleted; without either change the four tests of the default-store path fail.