Skip to content

ci: gate concurrent retention and logging throughput - #422

Merged
codeforester merged 103 commits into
mainfrom
ci/391-20261003-ci-add-concurrency-and-log-throughput-scenarios-to-the-compa
Oct 5, 2026
Merged

codeforester merged 103 commits into
mainfrom
ci/391-20261003-ci-add-concurrency-and-log-throughput-scenarios-to-the-compa

Conversation

@codeforester

@codeforester codeforester commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The comparative benchmark now gates concurrent retention and sustained logging. Twelve synchronized processes share a warmed cache, and persistent/ephemeral logging emits 3,000 records per sample. Reports use schema v2 with explicit latency, throughput, ratio, and budget units; the retained CI artifact remains compatible by filename.

Issue

Fixes #391

Branch maintenance

PR #421, the Project Intake parent car, has merged into main. GitHub retargeted this benchmark PR from the parent branch to main; the benchmark changes remain unchanged in the current head. This car is now the first open implementation car in the 0.5.0 train, followed by the release candidate in #423.

Current-head validation

At 0418cd45a59a9465934e85a6ed1edbb1f7759ffa: the branch includes the current main and the benchmark gate changes. The prior local validation covered uv lock freshness and baseline, runtime, strict typing, style, and contracts with all declared extras: 660 passed, 1 warning, and 294 subtests passed in 10.63s. Hosted checks are being rerun after the retarget; report their live result from the Checks tab before merge.

…07-us-record-about-20x-stdlib' into bug/378-20261003-bug-run-bundle-retention-is-inoperative-on-native-windows
…e-on-native-windows' into enhancement/386-20261003-perf-retention-serializes-concurrent-invocations-on-an-untim
…urrent-invocations-on-an-untim' into security/385-20261003-security-validate-trust-of-ancestor-discovered-project-confi
…or-discovered-project-confi' into bug/379-20261003-bug-json-single-envelope-stdout-contract-is-broken-by-any-wr
…ct-is-broken-by-any-wr' into ci/311-20261003-v1-0-make-project-intake-resilient-to-graphql-quota-exhausti
…ff-check-and-mypy-do-not-cover-the-compatibility-consum
…-compatibility-consum' into bug/387-20261003-bug-configure-logger-closes-consumer-owned-handlers-and-forc
…wned-handlers-and-forc' into enhancement/381-20261003-perf-lifecycle-logging-costs-107-us-record-about-20x-stdlib
…07-us-record-about-20x-stdlib' into bug/378-20261003-bug-run-bundle-retention-is-inoperative-on-native-windows
…e-on-native-windows' into enhancement/386-20261003-perf-retention-serializes-concurrent-invocations-on-an-untim
…urrent-invocations-on-an-untim' into security/385-20261003-security-validate-trust-of-ancestor-discovered-project-confi
…or-discovered-project-confi' into bug/379-20261003-bug-json-single-envelope-stdout-contract-is-broken-by-any-wr
…ct-is-broken-by-any-wr' into ci/311-20261003-v1-0-make-project-intake-resilient-to-graphql-quota-exhausti
…07-us-record-about-20x-stdlib' into bug/378-20261003-bug-run-bundle-retention-is-inoperative-on-native-windows
…ff-check-and-mypy-do-not-cover-the-compatibility-consum
…d-mypy-do-not-cover-the-compatibility-consum' into bug/387-20261003-bug-configure-logger-closes-consumer-owned-handlers-and-forc

# Conflicts:
#	docs/integrations.md
…ogger-closes-consumer-owned-handlers-and-forc' into enhancement/381-20261003-perf-lifecycle-logging-costs-107-us-record-about-20x-stdlib
…fecycle-logging-costs-107-us-record-about-20x-stdlib' into bug/378-20261003-bug-run-bundle-retention-is-inoperative-on-native-windows
…retention-is-inoperative-on-native-windows' into enhancement/386-20261003-perf-retention-serializes-concurrent-invocations-on-an-untim
…tention-serializes-concurrent-invocations-on-an-untim' into security/385-20261003-security-validate-trust-of-ancestor-discovered-project-confi
…alidate-trust-of-ancestor-discovered-project-confi' into bug/379-20261003-bug-json-single-envelope-stdout-contract-is-broken-by-any-wr
…-envelope-stdout-contract-is-broken-by-any-wr' into ci/311-20261003-v1-0-make-project-intake-resilient-to-graphql-quota-exhausti
…t-intake-resilient-to-graphql-quota-exhausti' into ci/391-20261003-ci-add-concurrency-and-log-throughput-scenarios-to-the-compa
@codeforester

Copy link
Copy Markdown
Contributor Author

Follow-up on current head 1cddd44: the new benchmark gate was calibrated against retained hosted artifacts. Functional/platform checks were green; the Unix persistent-logging p95 was 55.58 us/record and the macOS repeats reached 129.66–160.78 us/record due hosted filesystem variance. The documented caps are now Unix 80 and macOS 200 us/record, with the original ~107 us/record regression signal still below the caps. Benchmark unit tests and Ruff pass locally.

@codeforester

Copy link
Copy Markdown
Contributor Author

Addressed the benchmark-gate concern in the pushed head 258f90b0.

The stress run now measures plain stdlib FileHandler logging in the same process and requires a persistent-to-stdlib p95 ratio in addition to the existing absolute cap. The macOS absolute cap remains a filesystem sanity ceiling, while the ratio is the regression signal. The ratio cap is 12x across profiles; the full 31-sample local macOS run measured 63.44 us/record persistent, 6.36 us/record stdlib, and a 9.98x ratio.

Validation: the supported benchmark gate passes with 31 samples, and the contracts/style gates pass.

Base automatically changed from ci/311-20261003-v1-0-make-project-intake-resilient-to-graphql-quota-exhausti to main October 5, 2026 16:06
@codeforester

Copy link
Copy Markdown
Contributor Author

Resolved the merge conflicts by merging current main, including merged PR #421, into this branch in commit 0418cd4. Preserved the #391 benchmark report v2 and contention/logging gate documentation, the Project Intake recovery section, current main logging sidecar recovery, and the combined hot-path tests. Local validation passed: benchmark/CI-focused tests, full pytest, Ruff check and format, strict mypy, and documentation validation. GitHub reports the PR as conflict-free; hosted checks are running on the new head. The PR remains open and unmerged.

@codeforester

Copy link
Copy Markdown
Contributor Author

Investigated the failed Benchmark (unix/windows) checks. The failures were filesystem-tail outliers in the pre-existing persistence scenario: Unix reached 588 ms p95 with a 15.7 ms median, and Windows reached 1123 ms p95 with a 54.6 ms median; all #391 concurrency and logging-throughput gates passed. A Windows rerun passed, confirming runner variance. Added commit b486c93: Unix retains the 50 ms sustained median guard and now uses a 750 ms p95 filesystem-tail sanity ceiling, backed by the retained artifact and regression tests. Local full pytest, benchmark tests, Ruff, strict mypy, formatting, and docs validation pass. Hosted checks are running on the new head; the PR remains open and unmerged.

@codeforester

Copy link
Copy Markdown
Contributor Author

Hosted confirmation: the fresh benchmark run for b486c93 passed Benchmark (unix) and Benchmark (windows); macOS remains queued. The Unix persistence-tail calibration no longer blocks the gate, while the sustained median and #391 logging/concurrency ratio checks remain enforced.

@codeforester
codeforester merged commit 827e4ac into main Oct 5, 2026
129 of 130 checks passed
@codeforester
codeforester deleted the ci/391-20261003-ci-add-concurrency-and-log-throughput-scenarios-to-the-compa branch October 5, 2026 16:55
codeforester added a commit that referenced this pull request Oct 5, 2026
## Summary

Prepare the immutable 0.5.0 release candidate: align VERSION and package
metadata, cut the dated changelog section, document the 0.4.x migration
boundary, and provide the publication/evidence checklist. The changelog
validator now permits the newest VERSION-matching section to precede its
tag while retaining tag checks for published history.

This draft is the final car of the 0.5.0 train, after #422. The release
must also incorporate the remaining existing 0.5.0 PRs before final
review. Refresh the proposed date and final changelog against the
integrated release commit.

Publication is pending: independent approval, final hosted checks,
TestPyPI rehearsal, annotated tag on protected main, production
environment approval, PyPI/GitHub publication, and matching
checksums/SBOM/attestations. No tag or package was published by this PR
train. Keep the release issue open until that evidence is complete.

Refs #307.

## Branch maintenance

Refs #426. Targets the benchmark branch for #422 and remains the final
release car in the ordered stack. Retarget and refresh after #422 is
merged; preserve the order so the release candidate is reviewed against
the integrated benchmark changes.

## Current-head validation

At `845e5051ebff1c7bd72adf995a27d86499a4fe9c`: uv lock freshness and the
full aggregate (coverage, typing, style, contracts, benchmark, static
and dependency security) passed locally with all declared extras.
Runtime result: 661 passed, 1 warning, and 294 subtests passed in
12.86s.

Hosted checks currently report 111/111 successful checks with no
failures or pending checks. Publication evidence remains separate and is
intentionally not claimed by this PR.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci: add concurrency and log-throughput scenarios to the comparative benchmark

1 participant