Skip to content

create OTel tracer provider once per process - #2041

Open
avanish-garg wants to merge 1 commit into
temporalio:mainfrom
avanish-garg:fix/lambda-otel-warm-invocations
Open

avanish-garg wants to merge 1 commit into
temporalio:mainfrom
avanish-garg:fix/lambda-otel-warm-invocations

Conversation

@avanish-garg

Copy link
Copy Markdown
Contributor

What was changed

The provider-building code in lambda_worker/otel.py moved into _ensure_tracer_provider, which apply_defaults now calls. It installs the global replay-safe TracerProvider once and returns early on later calls while that provider is still the global one. If set_tracer_provider did not take (another provider already owns the global slot), the unused provider is shut down immediately. The apply_defaults docstring now says the provider is created once per process. Adds a changelog/fixed fragment. apply_tracing is unchanged (it builds a fresh plugin per invocation and its flush hook looks up the global provider each time).

The diff looks bigger than the change: most of it is the existing provider-building block moved into the helper. Logic and warning text are unchanged; a cold start behaves exactly as before.

Why?

run_worker calls configure on every Lambda invocation, so apply_defaults built a new TracerProvider plus OTLP BatchSpanProcessor (daemon thread and gRPC channel) each time. OpenTelemetry only allows the global provider to be set once, so from the second warm invocation the new provider was never installed or shut down, and a thread and channel leaked per invocation.

Trade-offs: the first call wins, so endpoint and service name are fixed for the process lifetime (they come from env or OtelOptions, static per container). The module-level cache has no lock, as configure runs sequentially in the main thread. A foreign global provider still triggers OpenTelemetry's "Overriding" warning on each call; avoiding that would mean skipping provider creation whenever a non-proxy provider is already global, which is a behavior change I left out.

Checklist

  1. Closes [Bug] lambda_worker otel.apply_defaults leaks a span exporter thread and gRPC channel on every warm invocation #2040

  2. How was this tested:
    New tests in tests/contrib/aws/lambda_worker/test_otel.py (class TestApplyDefaultsWarmInvocations):

  • test_reuses_provider_across_invocations: the exporter is built once across several apply_defaults calls.
  • test_does_not_warn_about_overriding_provider
  • test_unused_provider_is_shut_down_when_global_already_set
  • test_flush_hook_exports_on_every_invocation: spans are still exported on warm invocations 2 and 3.

Without the change 3 of these fail and 67 other tests in tests/contrib/aws/lambda_worker pass; with it all 70 pass. The reproduction from the issue now shows 1 export thread after each of 5 invocations (it grew 1, 2, 3, 4, 5 before). Ruff, pyright, basedpyright and mypy were run on the changed files only. Not run: the full lint task, the rest of the suite, a real Lambda or a real OTLP collector (the tests use fakes and a recording exporter). The new tests reset OpenTelemetry's global provider through trace._TRACER_PROVIDER and _TRACER_PROVIDER_SET_ONCE via monkeypatch, as there is no public reset API.

  1. Any docs updates needed?
    No. Docstring updated. The README still mentions apply_metrics where the function is build_metrics_telemetry_config; I left that out of this PR.

apply_defaults is called from the configure callback, which run_worker
invokes on every Lambda invocation. Each call built a new
TracerProvider with an OTLP BatchSpanProcessor, but OpenTelemetry only
allows the global provider to be set once, so from the second warm
invocation on the new provider was never installed or shut down,
leaking an export thread and gRPC channel per invocation.

Install the provider once and reuse it on later calls. If another
provider already owns the global slot, shut the unused one down.
@avanish-garg
avanish-garg requested a review from a team as a code owner October 29, 2026 13:04

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] lambda_worker otel.apply_defaults leaks a span exporter thread and gRPC channel on every warm invocation

1 participant