Skip to content

Fix pickling of NewType instances defined in __main__ on Python 3.10+ - #604

Open
charan-rathore wants to merge 1 commit into
cloudpipe:masterfrom
charan-rathore:fix-newtype-main-pickling
Open

charan-rathore wants to merge 1 commit into
cloudpipe:masterfrom
charan-rathore:fix-newtype-main-pickling

Conversation

@charan-rathore

Copy link
Copy Markdown

Fixes #520.

On Python 3.10 and newer, typing.NewType is a class and its instances define
__reduce__ returning their qualified name. cloudpickle therefore pickles them
by reference. When the NewType is defined in __main__ (a script or a
notebook), loading the pickle in a fresh process fails with
AttributeError: Can't get attribute 'RunDict' on <module '__main__' ...>.

This PR registers a dedicated reducer for typing.NewType, following the
_typevar_reduce shape @ogrisel suggested in the issue:

  • NewType instances that are importable from their module are still pickled by
    reference.
  • Dynamic ones (defined in __main__, or in modules registered with
    register_pickle_by_value) are reconstructed by value with _make_newtype
    and recorded in the dynamic class tracker, so identity is preserved across
    processes and within a single pickle stream.
  • On Python older than 3.10, typing.NewType is a function and the existing
    function pickling path already works, so the reducer is only registered when
    typing.NewType is a class.

Name, qualname, module and supertype are all preserved, and nested supertypes
(a NewType whose supertype is another NewType) roundtrip correctly.

Tests:

  • New regression tests cover: fresh-process load of a __main__ NewType (the
    exact cloudpickle fails to pickle NewType in main module by value on Python 3.10 and newer #520 repro), identity across a subprocess echo, memoization within one
    pickle stream, nested supertypes, importable NewType by reference, and
    modules registered with register_pickle_by_value.
  • All new dynamic-NewType tests fail on current master (PicklingError /
    AttributeError) and pass with this change.
  • Full suite on Python 3.10.12: 285 passed, 3 skipped, 2 xfailed.
  • Focused typing tests pass on Python 3.12, 3.13 and 3.14.

Caveats: the 3.8/3.9 guard (isinstance(typing.NewType, type)) is untested locally because no such interpreter was available. black --check flags tests/cloudpickle_ipykernel_test.py on unpatched master too (existing formatting drift); this change does not touch that file.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

cloudpickle fails to pickle NewType in main module by value on Python 3.10 and newer

1 participant