Skip to content

Defer building the OpenTelemetry tracestate until it's injected or exported - #12982

Open
dougqh wants to merge 4 commits into
dougqh/ptags-decision-maker-tablefrom
dougqh/ptags-lazy-otel-state
Open

dougqh wants to merge 4 commits into
dougqh/ptags-decision-maker-tablefrom
dougqh/ptags-lazy-otel-state

Conversation

@dougqh

@dougqh dougqh commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

What Does This Do

#12397 made every root probability decision build an OtelTraceState, inside the sampling lock. Only W3C tracestate injection and OTLP export ever read it.

This PR defers building it for the common case: a fresh probability decision with no inherited ot tracestate member, not rejected by the rate limiter.

  • tryUpdateProbabilitySamplingDecision installs a new ProbabilitySamplingState. It keeps only the trace id low-order bits and the rate, and builds the ot member on the first getOtelTraceState().
  • Racing readers agree on a single instance through a CAS. Duplicates would still be correct, because the state is immutable. One instance keeps the identity-based "unchanged" checks from installing a new state or clearing the W3C header cache for nothing. It also means publication doesn't depend on OtelTraceState keeping its fields final. The CAS runs once per injected or exported trace, uncontended.
  • PropagationTags.SamplingState is no longer final. preservesInheritedOtelPosition now reads through the getter.
  • The decision-maker computation moves out of installSamplingState into nextDecisionMakerTagValue, so the new path can share it.

Every other path is unchanged:

  • states derived from an inherited ot member
  • rate-limiter rejection
  • priority-only decisions
  • forceKeep

Motivation

This is part of the follow-ups to #12397's per-trace sampling overhead (APMLP-2048).

Stacked on #12981. Without #12981, moving the decision-maker computation into its own method made the remaining "-" + mechanism concat escape. That added 24 B/op to priority-only decisions. With #12981 the concat is gone and that path is unchanged.

Benchmark

Decision and header paths: SamplingDecisionBenchmark

Before = #12981; after = this branch. -prof gc, B/op.

arm before after Δ
ptagsProbabilityKeep (decision only) 216 168 −48
ptagsProbabilityKeepDatadogHeader 440 392 −48
ptagsProbabilityKeepW3CHeader 752 768 +16
ptagsProbabilityKeepOtlpTracestate 960 1072 +112, see below
ptagsPriorityKeep 152 152 0
spanDefaultSamplerKeep 1048 1000 −48
spanRuleSamplerKeep 1048 1000 −48
spanManualKeep 888 888 0
  • Never read (Datadog-only propagation): the full 48 B saving holds end to end.
  • W3C inject: the OtelTraceState is still built, plus 16 B for the two inputs ProbabilitySamplingState carries.
  • OTLP tracestate: with -XX:-DoEscapeAnalysis the gap is +16 B (1080 → 1096) at equal throughput. The rest is a scalar-replacement win the old code got in this direct-call arm and this code doesn't. It only applies to OTLP export with rule sampling: under OTLP export the default sampler is ParentBasedAlwaysOnSampler, a priority-only decision that never creates a ProbabilitySamplingState. Real export goes through OtlpTraceJson.writeSpan, so whether the JIT effect carries over there isn't measured.

Scenarios: PropagationScenariosBenchmark

A scenario benchmark: each arm is a whole request driven through the tracer's public API, with a default-configured tracer (datadog,tracecontext,baggage):

  • headNoPropagation, headPropagation: a root that decides, with or without an outbound inject
  • leaf, intermediate: extract an upstream decision, with or without relaying it

It's here to put this PR in context of #12397's overall cost. It only uses API unchanged since v1.66.0, so it builds onto both commits.

#12397 alone (79a1710fcd^ vs 79a1710fcd, B/op):

  • head, no propagation: +32 to +104 B
  • head, propagating: +48 to +288 B
  • leaf: +160 to +260 B
  • intermediate: +256 to +495 B

This PR on that benchmark (default sampler, v1.66.0 → master → this branch):

scenario v1.66.0 master this branch
head, no propagation 976 1096 1000
head, propagating 3688 3656 3656
leaf 2109 2342 2339
intermediate 4642 5056 4976

B/op on the larger scenario arms varies by ±100 B between forks, so treat small deltas there as noise.

Additional Notes

  • New ProbabilitySamplingStateTest covers:
    • the deferred value matches the eager conversion (keep, drop, rate 1.0, rate 0.0)
    • repeated reads return a single instance
    • rate-limiter rejection records no otel state
    • a later forceKeep starts from the deferred state
  • The full dd-trace-core test suite passes, apart from:
    • 2 Docker-dependent TracerConnectionReliabilityTest cases (no Docker locally)
    • DDAgentWriterCombinedTest and PendingTraceBufferTest, which failed under full-suite load and pass when run on their own

🤖 Generated with Claude Code

…isions

Every root probability decision built an OtelTraceState inside the sampling
lock, though only W3C injection and OTLP export read it. Install a
ProbabilitySamplingState that keeps the trace id bits and rate, and builds
the ot member on first read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dougqh dougqh added tag: no release notes Changes to exclude from release notes type: refactoring comp: context propagation Trace context propagation tag: ai generated Largely based on code generated by an AI or LLM labels Oct 21, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@datadog-prod-us1-6

This comment has been minimized.

@dd-octo-sts

dd-octo-sts Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.00 s 13.95 s [-0.5%; +1.3%] (no difference)
startup:insecure-bank:tracing:Agent 12.87 s 12.99 s [-1.7%; -0.1%] (maybe better)
startup:petclinic:appsec:Agent 17.06 s 16.90 s [-0.0%; +1.9%] (no difference)
startup:petclinic:iast:Agent 16.33 s 16.95 s [-8.0%; +0.7%] (no difference)
startup:petclinic:profiling:Agent 16.47 s 16.10 s [-1.9%; +6.6%] (no difference)
startup:petclinic:sca:Agent 16.96 s 16.85 s [-0.4%; +1.6%] (no difference)
startup:petclinic:tracing:Agent 15.67 s 16.20 s [-7.3%; +0.7%] (no difference)

Commit: 1d4880bb · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

dougqh and others added 2 commits October 21, 2026 15:13
…hape cost

SamplingDecisionBenchmark gains arms that follow a probability decision with
the Datadog header, the W3C tracestate header and the OTLP tracestate, so the
deferred OtelTraceState is measured where it's built or skipped.

ServiceShapePropagationBenchmark measures head, leaf and intermediate
services with a default-configured tracer. It only uses API unchanged since
v1.66.0, so it can be built onto release tags to put a cost on #12397.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Describe it as a scenario benchmark: each arm is a whole request driven
through the tracer's public API, rather than one method in isolation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dougqh
dougqh marked this pull request as ready for review October 21, 2026 19:38
@dougqh
dougqh requested review from a team as code owners October 21, 2026 19:38
@dougqh
dougqh requested review from MilanGarnier, khanayan123, mcculls and ygree and removed request for a team and khanayan123 October 21, 2026 19:38
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-21T19:40:22.753290Z 1d4880b Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@datadog-prod-us1-6 datadog-prod-us1-6 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: PASS

More details

The static review found no actionable regressions in sampling overrides, concurrent publication, propagation, or header caching from deferring OpenTelemetry state construction.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Bits Code Review · Commit 1d4880b


SamplingState state = ptags.samplingState();

assertSame(state.getOtelTraceState(), state.getOtelTraceState());

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could consider adding a multi-threaded test, since this just checks serial access?

@mcculls mcculls left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: context propagation Trace context propagation tag: ai generated Largely based on code generated by an AI or LLM tag: no release notes Changes to exclude from release notes type: refactoring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants