[litianningdatadog]

Overview

Part of SVLS-8713: bottlecap is replacing its own logs aggregator and flusher with the shared datadog-logs-agent crate from serverless-components. Lambda users must not notice the swap, so the plan is: pin today's behavior with tests (this stack), fix two real bugs first so the swap stays a pure refactor (this stack), then swap the aggregator and the flusher (bc-1 and bc-2, which will be opened after the serverless-components PRs merge).

Pins what the logs pipeline does today so the coming swap to datadog-logs-agent can be shown to change nothing for users. Almost all of it is tests; the only production change is a small refactor.

• refactor: the code that builds the logs aggregator, agent and flusher moves from main.rs into build_logs_pipeline() in the library, so tests run exactly the production wiring. Runtime behavior is unchanged.
• fake logs intake: a test server (HTTP and HTTPS with a self-signed cert) that records every request, decompresses zstd bodies and answers with scripted status codes or hangs.
• contract tests (tests/logs_contract_test.rs) through the real pipeline: every log field by field (START/END/REPORT, plain and JSON logs, logs before START, Managed Instance, processing rules, OOM, durable execution); URL, headers, method and compression per destination (Datadog, OPW, two additional endpoints, custom intake URL with a path); retries, redrive and drops for 400/401/403/413/429/500, timeouts and refused connections; batching; oversized logs; and warning/error line counts.
• benchmark (benches/logs_pipeline.rs, criterion dev-dependency): process-and-batch and flush, used as the baseline for the swap.

Two contract tests record known bugs on purpose (OPW + additional endpoints; one log over 5 MiB blocking the queue); the next two PRs fix them and update those tests.

Testing

755 tests pass (lib + contract + integration). clippy (-D warnings) and fmt clean. Every commit compiles. The benchmark only adds a dev-dependency; the extension binary is unchanged.

Stack

Each PR is nested under the PR it builds on (and targets). Review and merge from the top down.

serverless-components (datadog-logs-agent options; merge first)

• main
  • [SVLS-8713] test(logs-agent): pin current default behavior with characterization tests serverless-components#189 characterization tests (pin today's defaults)
    • [SVLS-8713] build(logs-agent): let users choose the FIPS TLS backend serverless-components#190 default / fips features
      • [SVLS-8713] feat(logs-agent): accept pre-serialized logs and an oversize policy serverless-components#191 pre-serialized logs + oversize policy
        • [SVLS-8713] feat(logs-agent): optional API key resolver and Datadog intake URL override serverless-components#192 API key resolver + intake URL override
          • [SVLS-8713] feat(logs-agent): add a retry-only method and an optional flush policy serverless-components#193 retry() + FlushPolicy
            • [SVLS-8713] perf(logs-agent): compress each log batch only once serverless-components#194 compress each batch once
              • [SVLS-8713] perf(logs-agent): compress log batches before waiting for the API key serverless-components#195 compress before waiting for the API key

datadog-lambda-extension (bottlecap)

• main
  • 👉 [SVLS-8713] test(logs): contract tests, fake logs intake and benchmark for the logs pipeline #1448 contract tests, fake intake, benchmark ← this PR
    • [SVLS-8713] fix(logs): send additional endpoints the Datadog intake format when OPW is on #1449 fix: OPW applies only to the primary endpoint
      • [SVLS-8713] fix(logs): drop a log too large for any payload instead of blocking the queue #1450 fix: drop a log too large for any payload
        • [SVLS-8713] test(logs): pin that a reloaded API key is used on the next flush #1451 pin: reloaded API key is used
          • bc-1: use the shared aggregator (opens after the serverless-components PRs merge)
            • bc-2: use the shared flusher (opens after the serverless-components PRs merge)

🤖 Generated with Claude Code

[datadog-datadog-us1-prod]

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 31214a0 | Docs | View more details | Give us feedback!}