Skip to content

[SVLS-8713] perf(logs-agent): compress each log batch only once - #194

Draft
litianningdatadog wants to merge 1 commit into
tianning.li/logs-sc-3b-retry-and-policyfrom
tianning.li/logs-sc-4-compress-once
Draft

litianningdatadog wants to merge 1 commit into
tianning.li/logs-sc-3b-retry-and-policyfrom
tianning.li/logs-sc-4-compress-once

Conversation

@litianningdatadog

@litianningdatadog litianningdatadog commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do?

Each batch is compressed at most once (per-batch OnceLock) and the same bytes are reused for the primary and every additional endpoint. The bytes sent don't change.

Motivation

Part of SVLS-8713: bottlecap (the Datadog Lambda Extension) is replacing its own logs aggregator and flusher with this crate. Two rules for the whole stack:

  • Nothing changes for existing users of this crate (for example serverless-compat): every new behavior is opt-in, defaults and public API stay the same (additions only), no new environment variables, no new dependencies.
  • Nothing changes for Lambda users: the options let bottlecap reproduce its current behavior exactly.

The flusher compressed the same batch again for every endpoint; with additional endpoints that multiplies CPU work, which matters on small Lambda functions.

Additional Notes

None.

Describe how to test/QA your changes

111 tests pass (new: primary and extras receive identical bytes equal to compressing once; each batch in a multi-batch flush is still compressed on its own).

Stack

Each PR is nested under the PR it builds on (and targets). Review and merge from the top down.

serverless-components (datadog-logs-agent options; merge first)

datadog-lambda-extension (bottlecap)

🤖 Generated with Claude Code

Why: the flusher compressed the same batch again for every endpoint. With
additional endpoints configured, that multiplied CPU work, which matters for
bottlecap on small Lambda functions.

What: each batch is now compressed at most once, the first time an endpoint
needs it, and the same bytes are reused for every other endpoint. The bytes
sent do not change. New tests check that the primary and every additional
endpoint receive identical bytes, equal to compressing the batch once, and
that each batch in a flush is still compressed on its own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@litianningdatadog
litianningdatadog force-pushed the tianning.li/logs-sc-3b-retry-and-policy branch from 5ac08a9 to a562801 Compare October 21, 2026 19:52
@litianningdatadog
litianningdatadog force-pushed the tianning.li/logs-sc-4-compress-once branch from 89ab609 to 84cb56d Compare October 21, 2026 19:52

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant