Skip to content

Defer parsing the extracted OpenTelemetry tracestate member - #12987

Draft
dougqh wants to merge 1 commit into
dougqh/ptags-lazy-otel-statefrom
dougqh/ptags-lazy-inherited-otel
Draft

dougqh wants to merge 1 commit into
dougqh/ptags-lazy-otel-statefrom
dougqh/ptags-lazy-inherited-otel

Conversation

@dougqh

@dougqh dougqh commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

What Does This Do

Since #12397, extraction parses the inherited ot tracestate member into an OtelTraceState and checks it against the inherited decision, all on the request thread. Only W3C injection, OTLP export and a local change of decision ever read it, so a leaf service exporting to the Datadog agent never needs it.

This keeps the raw member and parses it on first read:

  • InheritedSamplingState, alongside Defer building the OpenTelemetry tracestate until it's injected or exported #12982's ProbabilitySamplingState, holds the raw member, its original size and the pending consistency checks. The first getOtelTraceState() parses the member, applies the checks, and publishes the result with a CAS. An invalid member resolves to null, as before.

  • The eager sites now defer:

    • extraction (W3CPTagsCodec.fromHeaderValue)
    • the compound merge (updateW3CTracestateFrom)
    • EXTERNAL_OVERRIDE / UNKNOWN reconciles

    The last one includes root-span construction from an extracted trace (DDSpanContext.java:453): the extracted priority isn't on the PTags yet there, so every continued trace was parsing the member at span construction.

  • Local decisions (forceKeep, manual or AppSec keep) still parse and rewrite the member, as before.

Output is unchanged. A check drops th when rv >= th doesn't match the decision, and dropping it is idempotent and leaves the state consistent with any decision. So any sequence of checks equals one check per decision, in any order, and two bits record them.

Motivation

#12981 and #12982 recover #12397's head decision-path cost but none of its extract-side cost, which leaf and intermediate services pay. This applies the deferral model from the TraceSegment design (APMLP-2049) to the ot member. The larger extract-side change is left to that work.

Benchmark

PropagationScenariosBenchmark (added in #12982). Before = #12982; after = this branch. Leaf and intermediate with -f 3, -prof gc.

scenario sampler upstream B/op before B/op after Δ ops/µs before ops/µs after
leaf default sameVersion 2341 ± 1 2325 ± 3 −16 2.33 ± 0.14 2.61 ± 0.10
leaf default withOtelMember 2551 ± 2 2535 ± 3 −16 2.36 ± 0.08 2.39 ± 0.16
leaf rule sameVersion 2340 ± 1 2360 ± 44 +20 2.35 ± 0.13 2.60 ± 0.22
leaf rule withOtelMember 2598 ± 73 2545 ± 16 −52 2.36 ± 0.12 2.49 ± 0.13
intermediate default sameVersion 5052 ± 77 5090 ± 53 +38 1.26 ± 0.08 1.27 ± 0.06
intermediate default withOtelMember 5268 ± 91 5340 ± 83 +72 1.38 ± 0.11 1.18 ± 0.10
intermediate rule sameVersion 4963 ± 82 5079 ± 42 +116 1.28 ± 0.08 1.13 ± 0.16
intermediate rule withOtelMember 5170 ± 16 5210 ± 30 +40 1.23 ± 0.10 1.26 ± 0.06
  • Leaf: about 10% more throughput in two of four arms, beyond the error bars, because the parse is off the request thread. Allocation: −16 B.
  • Intermediate: still parses at inject, so it's roughly flat to slightly more allocation (partly within noise), with mixed throughput.
  • Head scenarios (1 fork): unchanged without propagation; −48 / −96 B with propagation, likely variance between forks.

Why the allocation saving is small:

  • The deferred OtelTraceState is ~64 B.
  • InheritedSamplingState is ~16 B bigger than SamplingState.
  • An extracted trace builds three sampling-state snapshots: the W3C context's initial one, the compound merge, and the reconcile at root-span construction.
  • −64 + 3 × 16 = −16 B, which matches the measurement.

The snapshots, not the parsed object, dominate the extract-side cost. Shrinking InheritedSamplingState (one field for raw-or-resolved, packed size and checks) would get closer to −40 B. That's left to the TraceSegment work, which replaces the snapshots instead.

Additional Notes

  • InheritedSamplingStateTest: for 10 kinds of ot member × all four check combinations, lazy resolution equals parsing first then checking, and check order doesn't matter.
  • InheritedOtelExtractionTest: real compound and OTel-only extraction plus a finished leaf span leaves the member unparsed, and the header output is still correct, including dropping an inconsistent th.
  • Mutation-checked: disabling either deferred path fails these tests.
  • The full dd-trace-core suite passes, apart from:
    • 2 Docker-dependent TracerConnectionReliabilityTest cases
    • DDAgentWriterCombinedTest and PendingTraceBufferTest, which fail under full-suite load and pass on their own

🤖 Generated with Claude Code

Extraction parsed the inherited ot member and checked it against the
inherited decision on the request thread, at extraction, compound merge and
root span construction, though only W3C injection, OTLP export and a local
change of decision read it. Keep the raw member in an InheritedSamplingState
and parse it on first read. Consistency checks made before then are recorded
as pending and applied in order when it's parsed, so the output is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dougqh dougqh added tag: no release notes Changes to exclude from release notes type: refactoring comp: context propagation Trace context propagation tag: ai generated Largely based on code generated by an AI or LLM labels Oct 21, 2026
@datadog-prod-us1-6

This comment has been minimized.

@dd-octo-sts

dd-octo-sts Bot commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 13.99 s 13.96 s [-0.3%; +0.8%] (no difference)
startup:insecure-bank:tracing:Agent 12.89 s 12.94 s [-1.0%; +0.3%] (no difference)
startup:petclinic:appsec:Agent 17.17 s 17.08 s [-0.3%; +1.4%] (no difference)
startup:petclinic:iast:Agent 17.06 s 16.99 s [-0.5%; +1.2%] (no difference)
startup:petclinic:profiling:Agent 16.76 s 16.80 s [-1.3%; +0.7%] (no difference)
startup:petclinic:sca:Agent 17.05 s 16.81 s [+0.6%; +2.2%] (maybe worse)
startup:petclinic:tracing:Agent 16.11 s 15.68 s [-1.6%; +7.1%] (no difference)

Commit: 7ed5a2e8 · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: context propagation Trace context propagation tag: ai generated Largely based on code generated by an AI or LLM tag: no release notes Changes to exclude from release notes type: refactoring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant