Fix pickling of NewType instances defined in __main__ on Python 3.10+ - #604
Open
charan-rathore wants to merge 1 commit into
Open
charan-rathore wants to merge 1 commit into
charan-rathore wants to merge 1 commit into
Conversation
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #520.
On Python 3.10 and newer,
typing.NewTypeis a class and its instances define__reduce__returning their qualified name. cloudpickle therefore pickles themby reference. When the NewType is defined in
__main__(a script or anotebook), loading the pickle in a fresh process fails with
AttributeError: Can't get attribute 'RunDict' on <module '__main__' ...>.This PR registers a dedicated reducer for
typing.NewType, following the_typevar_reduceshape @ogrisel suggested in the issue:reference.
__main__, or in modules registered withregister_pickle_by_value) are reconstructed by value with_make_newtypeand recorded in the dynamic class tracker, so identity is preserved across
processes and within a single pickle stream.
typing.NewTypeis a function and the existingfunction pickling path already works, so the reducer is only registered when
typing.NewTypeis a class.Name, qualname, module and supertype are all preserved, and nested supertypes
(a NewType whose supertype is another NewType) roundtrip correctly.
Tests:
__main__NewType (theexact cloudpickle fails to pickle NewType in main module by value on Python 3.10 and newer #520 repro), identity across a subprocess echo, memoization within one
pickle stream, nested supertypes, importable NewType by reference, and
modules registered with
register_pickle_by_value.AttributeError) and pass with this change.
Caveats: the 3.8/3.9 guard (
isinstance(typing.NewType, type)) is untested locally because no such interpreter was available.black --checkflagstests/cloudpickle_ipykernel_test.pyon unpatched master too (existing formatting drift); this change does not touch that file.