[DarcyRaynerDD]

Overview

Fixes undercounted aws.lambda.enhanced.errors, timeouts and out_of_memory when a function times out, crashes or OOMs after the extension has switched to periodic or continuous flushing.

Cause. At shutdown, the Telemetry API listener pushed the Tombstone (the "all events processed" marker) straight onto the event bus. Telemetry events take a longer path: listener → logs agent → event bus. The main loop stops reading at the first Tombstone. So a platform.runtimeDone, platform.report or OOM log line for the failing invocation that was still in the logs agent got dropped. That lost the invocation's errors, timeouts, OOM, duration and memory metrics, its timeout span, and its last log lines. End mode was protected for the counts, because it waits for runtimeDone before calling /next. This is a regression: #886 drained the bus until it closed, and #947 replaced that with a wait that stops at the first Tombstone.

Changes:

• The logs agent sends the Tombstone after its telemetry channel disconnects, which happens once the listener has shut down and dropped its sender, or after a 250 ms drain timeout. Every telemetry event now reaches the bus ahead of the Tombstone.
• The listener no longer sends a Tombstone or holds an event bus sender.
• The busy-spin drain loop is gone. It's replaced with an awaited recv under the timeout.
• The listener returns 503 instead of panicking when the logs agent has already stopped receiving. That can now happen if the drain times out mid-request. A panic would have aborted the process in release builds (panic = "abort").
• A Managed Instance shutdown comment is corrected. Behavior there is unchanged.
• The failure-accounting integration suite from test(integration): add failure-accounting suite for timeout/crash enhanced metrics #1454 is registered in the CI matrix, so it becomes blocking together with this fix.

Implications:

• Counts: affected customers may see errors/timeouts/OOM rise to match CloudWatch.
• Shutdown time: usually unchanged. At worst it grows by 250 ms if the listener doesn't close.
• Telemetry sent after SHUTDOWN: the listener still closes right away, so Lambda's POST is refused. CI on fix(shutdown): handle runtimeDone without metrics; wait for the processor before the final flush #1457 showed that Lambda redelivers that telemetry to the restarted extension, which counts it there. It is delayed rather than lost. It would only be lost if no new environment ever starts, for example on scale-in.

Stacked on #1454, which adds the failure-accounting integration suite that reproduces the loss on main.

Testing

• New unit tests:
  • an event sent to the logs agent after shutdown starts reaches the event bus before the Tombstone
  • the Tombstone is still sent when the channel never closes
  • events still queued when the drain deadline passes are forwarded before the Tombstone
  • the listener returns 503 after the receiver is dropped
• Existing checks: cargo test --workspace, cargo clippy and cargo fmt pass.
• Integration (pending): the failure-accounting suite runs on this branch with 12 resets per run. The Default-strategy function should match exactly on every attempt. Please check all attempts, since retry: 2 can hide a failed one.

🤖 Generated with Claude Code

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-23T17:13:19.154797Z	—	New commits

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3390ac2c03

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Drain queued telemetry before honoring the deadline

When shutdown finds a large telemetry backlog or processing takes more than 250 ms, this fixed absolute deadline expires while events can still be queued in self.rx; the next timeout_at then returns Err, the agent sends the Tombstone, and the main loop proceeds to its final flush while the remaining logs and lifecycle events are dropped. Apply the timeout only while waiting for the listener to disconnect or for another event, while always draining already-queued events before emitting the Tombstone.

Useful? React with 👍 / 👎.

[DarcyRaynerDD]

I don't think this can happen with tokio 1.50:

• Timeout::poll polls the inner future before the timer (time/timeout.rs:199-206), and Receiver::recv returns Ready for every buffered message (sync/mpsc/chan.rs:302-306). An expired deadline therefore can't win while events are queued. The loop keeps draining past the deadline and only hits Err once the queue is empty.
• Budget exhaustion doesn't change that. recv only returns Pending for budget when the budget is already zero before the call (chan.rs:295). In that case poll_delay polls the Sleep under the same budget (sleep.rs:430), so the task yields and resumes rather than timing out.

beb8006 adds test_queued_events_forwarded_after_drain_timeout. It queues enough forwarded events to fill the event bus past the deadline, then 500 function logs (which don't yield), then a runtimeDone, and asserts that every forwarded event reaches the bus before the Tombstone. It passes without any change to the drain loop. I also checked multi-thread runtimes with concurrent senders and 200k-event backlogs, and nothing was dropped.

The remaining gap is a send that has claimed a slot but not finished writing it at the exact moment of the deadline. That's microseconds wide and equivalent to an event arriving just after the 250 ms cap, which is the intended tradeoff. Bounding the shutdown sequence as a whole against Lambda's deadline is a planned follow-up.

[datadog-datadog-prod-us1]

✨ Unblock PR with BitsAI

❌ Errors

Your PR has failed checks. Please review the issues below and take necessary action before merging.

🚦 1 Pipeline job failed

DataDog/datadog-lambda-extension | e2e-test-status (amd64) — 🔧 Needs a code fix, caused by this PR

View more details · View in GitLab

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: beb8006 | Docs | View more details | Give us feedback!}

[lucaspimentel]

@DataDog review

[datadog-datadog-prod-us1]

More details

The logs agent places the Tombstone after drained telemetry, allowing the main loop to process shutdown events before stopping. The completed static review found no actionable regression.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit beb8006}

[litianningdatadog]

LGTM