Repository navigation
create OTel tracer provider once per process - #2041
Open
avanish-garg wants to merge 1 commit into
Open
avanish-garg wants to merge 1 commit into
avanish-garg wants to merge 1 commit into
Conversation
apply_defaults is called from the configure callback, which run_worker invokes on every Lambda invocation. Each call built a new TracerProvider with an OTLP BatchSpanProcessor, but OpenTelemetry only allows the global provider to be set once, so from the second warm invocation on the new provider was never installed or shut down, leaking an export thread and gRPC channel per invocation. Install the provider once and reuse it on later calls. If another provider already owns the global slot, shut the unused one down.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What was changed
The provider-building code in
lambda_worker/otel.pymoved into_ensure_tracer_provider, whichapply_defaultsnow calls. It installs the global replay-safeTracerProvideronce and returns early on later calls while that provider is still the global one. Ifset_tracer_providerdid not take (another provider already owns the global slot), the unused provider is shut down immediately. Theapply_defaultsdocstring now says the provider is created once per process. Adds achangelog/fixedfragment.apply_tracingis unchanged (it builds a fresh plugin per invocation and its flush hook looks up the global provider each time).The diff looks bigger than the change: most of it is the existing provider-building block moved into the helper. Logic and warning text are unchanged; a cold start behaves exactly as before.
Why?
run_workercallsconfigureon every Lambda invocation, soapply_defaultsbuilt a newTracerProviderplus OTLPBatchSpanProcessor(daemon thread and gRPC channel) each time. OpenTelemetry only allows the global provider to be set once, so from the second warm invocation the new provider was never installed or shut down, and a thread and channel leaked per invocation.Trade-offs: the first call wins, so endpoint and service name are fixed for the process lifetime (they come from env or
OtelOptions, static per container). The module-level cache has no lock, asconfigureruns sequentially in the main thread. A foreign global provider still triggers OpenTelemetry's "Overriding" warning on each call; avoiding that would mean skipping provider creation whenever a non-proxy provider is already global, which is a behavior change I left out.Checklist
Closes [Bug] lambda_worker otel.apply_defaults leaks a span exporter thread and gRPC channel on every warm invocation #2040
How was this tested:
New tests in
tests/contrib/aws/lambda_worker/test_otel.py(classTestApplyDefaultsWarmInvocations):test_reuses_provider_across_invocations: the exporter is built once across severalapply_defaultscalls.test_does_not_warn_about_overriding_providertest_unused_provider_is_shut_down_when_global_already_settest_flush_hook_exports_on_every_invocation: spans are still exported on warm invocations 2 and 3.Without the change 3 of these fail and 67 other tests in
tests/contrib/aws/lambda_workerpass; with it all 70 pass. The reproduction from the issue now shows 1 export thread after each of 5 invocations (it grew 1, 2, 3, 4, 5 before). Ruff, pyright, basedpyright and mypy were run on the changed files only. Not run: the full lint task, the rest of the suite, a real Lambda or a real OTLP collector (the tests use fakes and a recording exporter). The new tests reset OpenTelemetry's global provider throughtrace._TRACER_PROVIDERand_TRACER_PROVIDER_SET_ONCEviamonkeypatch, as there is no public reset API.No. Docstring updated. The README still mentions
apply_metricswhere the function isbuild_metrics_telemetry_config; I left that out of this PR.