Skip to content

Python layer 127+ logs "failed to enable ddtrace support for unittest" at ERROR on every init (ci_visibility stripped, unittest integration still enabled) #871

Closed

Description

@bubirose

Expected Behavior

Importing unittest (directly or through a dependency) in a function using the Datadog Python layer does not produce errors. Test-runner integrations should be inert in Lambda, since the layer deliberately removes CI Visibility.

Actual Behavior

Every cold start and provisioned-concurrency init logs this at ERROR:

[ERROR] failed to enable ddtrace support for unittest: No module named 'ddtrace.internal.ci_visibility'

Requests are unaffected, but the message trips error-log monitors. It started when we moved from layer 116 to 127. On the same function and code, ddtrace's startup line went from "Configured ddtrace instrumentation for 78 integration(s)" to 84.

Cause, from reading the layer Dockerfile on main and dd-trace-py v4.12.2 (the version layer 127 bundles for Python >= 3.10):

  1. ddtrace/_monkey.py has "unittest": True in PATCH_MODULES, and datadog_lambda/patch.py calls ddtrace.patch_all() without disabling it. So an import hook is registered for unittest.
  2. The Dockerfile's "CI Visibility paths/integrations" step removes ddtrace/internal/ci_visibility (added in chore: remove all ddtrace ci viz paths from layer #765). It also removes ddtrace/contrib/unittest/, but in ddtrace 4.x the integration lives at ddtrace/contrib/internal/unittest/. That directory survives, and its patch.py imports ddtrace.internal.ci_visibility on line 29.
  3. When unittest is imported, the hook (_monkey.py line 278) imports that patch module, which fails. The hook is registered with raise_errors=False, so the failure is logged at ERROR rather than raised.

Removing contrib/internal/unittest/ alone would not fix it: the hook would still fail to import the patch module and log the same ERROR. The integration needs to be disabled. For example, datadog_lambda could call patch_all_dd(unittest=False), or more generally turn off the test-runner integrations whose dependencies the layer strips. The same likely applies to pytest and pytest_bdd if they are in the default patch list, and their strip paths in the Dockerfile have the same pre-4.x layout. We have not checked those.

Layers 128 and 129 appear unchanged in this respect: nothing related is in their release notes, and the Dockerfile on main has the same strip list.

Steps to Reproduce the Problem

  1. Deploy a Python 3.11 function with Datadog-Python311-ARM:127 and Datadog-Extension-ARM:99, with DD_TRACE_ENABLED=true.
  2. Have the handler module run import unittest at module level (we see it through a dependency, not our own code).
  3. Invoke it, then check the function's logs for failed to enable ddtrace support for unittest.

Specifications

  • Datadog Lambda Layer version: Python 127 (dd-trace-py 4.12.2), extension 99
  • Python version: 3.11 (arm64)

Stacktrace

[ERROR] failed to enable ddtrace support for unittest: No module named 'ddtrace.internal.ci_visibility'
Traceback (most recent call last):
  File "./python/lib/python3.11/site-packages/ddtrace/_monkey.py", line 278, in on_import
  File "/var/lang/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "./python/lib/python3.11/site-packages/datadog_lambda/cold_start.py", line 130, in wrapped_method
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "./python/lib/python3.11/site-packages/ddtrace/contrib/internal/unittest/patch.py", line 29, in <module>
ModuleNotFoundError: No module named 'ddtrace.internal.ci_visibility'

Workaround: set DD_TRACE_UNITTEST_ENABLED=false on the function. _patch_all honours it; DD_PATCH_MODULES does not apply on the patch_all() path.

Activity

  1. rithikanarayan commented on Oct 21, 2026

    @rithikanarayan
    Contributor

    Hi @bubirose, thanks for reporting this issue! We are currently addressing it in #873 and will release a new version of the Lambda layer with the fix once it is ready. I'll provide another update here once the new version of datadog-lambda is released. Feel free to let us know if you see any other issues, thanks!

  2. rithikanarayan commented on Oct 21, 2026

    @rithikanarayan
    Contributor

    Hi @bubirose, v130 of the datadog-lambda library has been released with the fix! I tested it in a Python 3.11 ARM Lambda to ensure that there are no error logs being emitted, but please reach out if you have any more issues. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions