test(integration): add failure-accounting suite for timeout/crash enhanced metrics by DarcyRaynerDD · Pull Request #1454 · DataDog/datadog-lambda-extension

DarcyRaynerDD · 2026-10-23T16:38:21Z

Overview

Adds a failure-accounting integration suite that checks whether aws.lambda.enhanced.invocations, errors and timeouts match what Lambda reports when a function times out or crashes under steady load.

No existing suite covered timeouts or busy functions. The extension's Default flush strategy switches from flushing at the end of each invocation to periodic flushing after about 20 invocations in one environment. A timeout or crash resets the environment, so the suite spaces failures 30 invocations apart. That way each fresh environment has switched to periodic flushing before the next failure.

Two Node functions, same code: one on the Default flush strategy, and one pinned to end as a control. Both have reserved concurrency 1 so the invocations share one environment at a time.
The schedule: 370 invocations per function, called one after another. Every 30th invocation times out or calls process.exit(1), alternating, for 12 failures: 6 timeouts and 6 crashes.
Ground truth comes from the Invoke responses (FunctionError, and Sandbox.Timedout for timeouts).
Assertions: Datadog counts must match exactly for both functions.
Debug logs (DD_LOG_LEVEL=debug) are on for the Default function, so a lost count can be traced in the extension logs.
Throttle retries: invokes throttled right after a reset are retried. Throttled invokes never run, so this doesn't change the expected counts.

This suite fails intermittently on main today. The first CI run (6 failures per attempt, extension built from 522d3ff) showed:

Attempt	Function	Invocations	Errors	Timeouts
Expected (both)		310	6	3
1	Default strategy	310	5	2
1	`end` strategy (control)	310	6	3
2 (retry)	both	310	6	3

Only the error and timeout counts were lost; invocations were not. That points to the failing invocation's platform.runtimeDone being dropped at shutdown.
Only the function that had switched to periodic flushing was affected. The end control counted everything.
The lost count never arrived. It stayed at 5/2 for 21 polls over about 12 minutes.

The fix is in #1455, stacked on this PR. This PR doesn't add the suite to the CI matrix (.gitlab/datasources/test-suites.yaml), so it can merge without making main flaky. #1455 registers the suite alongside the fix. The integration jobs use retry: 2, which can hide a failed attempt, so on #1455 check every attempt of this suite, not just the final status.

Testing

npx tsc --noEmit -p . and a CDK synth of the stack pass.
I ran the suite in CI on this branch before removing it from the matrix. The results are in the table above (6 failures per attempt). The 12-failure version failed again on a2bb33a.

🤖 Generated with Claude Code

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

chatgpt-codex-connector · 2026-10-23T16:38:31Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-23T17:10:40.263366Z	`a086bc4`	New commits

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

datadog-prod-us1-4 · 2026-10-23T16:38:36Z

❌ Errors

Your PR has failed checks. Please review the issues below and take necessary action before merging.

🚦 1 Pipeline job failed

Bottlecap (Rust) | Audit

View more details · View in GitHub Actions

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: a086bc4 | Docs | View more details | Give us feedback!}

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a2bb33adb0

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-10-23T16:41:25Z

  - name: payload-size-e2e
  - name: dsm
  - name: durable-cold-start
+  - name: failure-accounting


Keep the known-flaky suite out of the blocking matrix

When the current default flush strategy loses a timeout/error metric—the race this test is explicitly intended to reproduce—the exact-count assertions fail, yet .gitlab/templates/pipeline.yaml.tpl makes every suite except payload-size-e2e blocking. Adding this entry therefore makes unrelated pipelines intermittently fail on the pre-fix implementation; retry: 2 only reduces or hides the failure rather than making the suite reliable. Enable the matrix entry only together with the extension fix, or otherwise keep it non-blocking until that fix lands.

Useful? React with 👍 / 👎.

Agreed. a086bc4 takes failure-accounting out of the matrix in this PR, and #1455 adds it back alongside the fix, so the suite only becomes blocking once the race is fixed.

…anced metrics Invokes a Node function 310 times sequentially with a timeout or crash every 50th invocation, under both the Default and End flush strategies, and asserts aws.lambda.enhanced.invocations/errors/timeouts match the Invoke responses. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ble debug logs The first CI run lost one timeout's errors/timeouts counts on the default-strategy function in one of two attempts, so the loss is a race. More resets per run make it more likely to show up, and debug logs show when runtimeDone reaches the extension relative to shutdown. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ix lands The suite reproduces a race that the current extension loses intermittently, so registering it before the fix would make unrelated pipelines flaky. The fix PR registers it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

lucaspimentel · 2026-10-26T14:17:26Z

@DataDog review

datadog-prod-us1-4

More details

The suite targets the known timeout/crash metric-loss race and remains outside the CI matrix until the fix in #1455, as confirmed in the human discussion. The completed static reviewer returned no reportable findings.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit a086bc4}

Copilot AI balanced review requested due to automatic review settings October 23, 2026 16:38

DarcyRaynerDD requested a review from a team as a code owner October 23, 2026 16:38

DarcyRaynerDD requested a review from shreyamalpani October 23, 2026 16:38

Copilot AI reviewed Oct 23, 2026

DarcyRaynerDD mentioned this pull request Oct 23, 2026

fix(shutdown): send Tombstone after telemetry is drained #1455

Open

chatgpt-codex-connector Bot reviewed Oct 23, 2026

View reviewed changes

DarcyRaynerDD and others added 3 commits October 23, 2026 12:45

DarcyRaynerDD force-pushed the darcy.rayner/failure-accounting-integration-test branch from a2bb33a to a086bc4 Compare October 23, 2026 17:08

DarcyRaynerDD added this pull request to stack #1456 October 23, 2026 17:54

DarcyRaynerDD mentioned this pull request Oct 23, 2026

fix(shutdown): handle runtimeDone without metrics; wait for the processor before the final flush #1457

Closed

datadog-prod-us1-4 Bot reviewed Oct 26, 2026

View reviewed changes

litianningdatadog self-requested a review October 26, 2026 17:18

DarcyRaynerDD closed this Oct 26, 2026

DarcyRaynerDD reopened this Oct 26, 2026

litianningdatadog approved these changes Oct 26, 2026

View reviewed changes

Navigation Menu

Repository navigation

test(integration): add failure-accounting suite for timeout/crash enhanced metrics - #1454

test(integration): add failure-accounting suite for timeout/crash enhanced metrics#1454
DarcyRaynerDD wants to merge 3 commits into
mainfrom
darcy.rayner/failure-accounting-integration-test

DarcyRaynerDD commented Oct 23, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

chatgpt-codex-connector Bot commented Oct 23, 2026 •

edited

Loading

Uh oh!

datadog-prod-us1-4 Bot commented Oct 23, 2026 •

edited by datadog-prod-us1-5 Bot

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Oct 23, 2026

Uh oh!

DarcyRaynerDD Oct 23, 2026

Uh oh!

lucaspimentel commented Oct 26, 2026

Uh oh!

datadog-prod-us1-4 Bot left a comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

DarcyRaynerDD commented Oct 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Testing

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot commented Oct 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

datadog-prod-us1-4 Bot commented Oct 23, 2026 • edited by datadog-prod-us1-5 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ Errors

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

DarcyRaynerDD Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

lucaspimentel commented Oct 26, 2026

Uh oh!

datadog-prod-us1-4 Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

DarcyRaynerDD commented Oct 23, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Oct 23, 2026 •

edited

Loading

datadog-prod-us1-4 Bot commented Oct 23, 2026 •

edited by datadog-prod-us1-5 Bot

Loading