Performance: benchmark suite v2 — the benchmark-game set plus web-request and real-app workloads, per-commit history in JSON, a public chart page, and bench-gate extended to all of it (benchmarks/, script/bench-gate.php, docs/pages/bench.html)

Why

benchmarks/ holds 7 programs from 2015 (fibo, Ack, mandelbrot, simple) and benchmarks/README.md's table is dated 2026-07-27 — it was not regenerated after wave 1 changed every number. script/bench-gate.sh (#36196) compares ratios for those 7 only. Nothing measures strings, arrays, objects, regex, JSON, templates, or a web request — the shapes the audit found 500x off. A compiler's performance claim is only as good as its benchmark suite.

Deliverable

benchmarks/v2/: nbody, spectral-norm, fannkuch-redux, binary-trees, fasta, k-nucleotide, regex-redux, json-roundtrip (encode/decode 10 MB), template-render (string building, 100k rows), assoc-heavy (100k string keys, lookups, array_*), object-graph (1M method calls + property access), sort-mixed, str-builder (.=, implode, sprintf), closure-heavy, exceptions (10k throw/catch), plus web-request (MiniWebApp / and /api/status via phpc fcgi, 2,000 requests, req/s and p99 vs php-fpm in the same image).
Each program deterministic, output-verified against Zend before timing; VM / JIT / AOT columns, ratio to Zend, best-of-N with a per-run cap (no hangs).
benchmarks/history/<sha>.json appended by the nightly; docs/pages/bench.html renders the ratio history per program (sparkline); benchmarks/README.md table generated from the latest JSON only.
bench-gate baseline = the JSON of the last release tag; regression threshold 20 % per program, 10 % on the geometric mean.

Done when

16 programs committed with verified outputs; PHP_8_2=$(command -v php) php script/bench.php --v2 regenerates the table in < 15 min
Nightly history + chart page live; bench-gate covers v2 and is red on a deliberate 2x slowdown
benchmarks/README.md no longer contains any hand-typed number

claim: implementer-worker-lane-b — starting this run (lane B tie-break #36385 % 3 == 1; lane overflow: #36143–#36148 closed; #36387 claimed by lane A)

Bench: add v2 suite + gate for strings/arrays/objects shapes (#36385) (…

Merged #36459 (Part of #36385)

Verification:

./script/bench-gate.sh --v2 → OK (5-case AOT subset, 20% ratio band)
Deliberate ~2× baseline poison → exit 1 (call-heavy.ratio +177%)
php script/bench.php --v2 → 16 programs, RESULTS.json + history JSON
./script/aot-smoke.sh → 9/9

Still open on #36385: web-request column, live chart, scrub legacy README prose numbers.

claim: implementer-worker-lane-b — continuing this run (lane B tie-break #36385 % 3 == 1). Next: web-request column + chart page + scrub hand-typed README numbers after #36459.

Bench: web-request column, live chart page, scrub hand-typed README r…

Merged #36462 (Part of #36385)

Landed:

script/bench-web-request.php — MiniWebApp home + api/status req/s (Zend builtin vs phpc serve; AOT/php-fpm n/a with reasons)
script/generate-bench-chart.php — live docs/pages/bench.html sparklines from benchmarks/history/*.json + RESULTS
Scrubbed hand-typed speedup tables from benchmarks/README.md

Verify:

BenchGateTest 8/8
bench-gate.sh --v2 OK
aot-smoke 9/9
web-request @200: zend ~391 req/s, phpc_serve ~22 req/s

Still open on #36385: nightly history publisher, php-fpm column, 2000-req soak / live chart vs php-fpm in docs, full --v2 table refresh in <15 min on a quiet box.

claim: implementer-worker-lane-b — starting this run (lane B tie-break #36385 % 3 == 1; prior claim >48h). Next: finish remaining #36385 deliverables (web-request column / chart / README scrub) or next v2 bench programs after #36462.

note: #36385 deliverables largely landed (#36459/#36462). Pivoted this run to Part of #36403 (VM Concern extract / size-budget ratchet) — lane B.

claim: implementer-worker-lane-b — starting this run (lane B tie-break #36385 % 3 == 1; lane overflow: #36143–#36148 / north-star children closed; #36387 has open PR #37237 claimed <48h)

Fix: loop-carried typed += must not fold as 0+i (#36385) (#37240)

Merged #37689 (Part of #36385) — implementer-worker-lane-b (lane B #36385 % 3 == 1; lane overflow: #36143–#36148 / #36142 children closed).

Slice

Done-when honesty for README timings + php-fpm column automation:

script/check-bench-readme-sync.php — v2 table ↔ RESULTS.json; ban hand-typed speed claims / timing decimals outside markers; wired into check-generated-docs.sh.
bench-web-request.php — disposable php-fpm pool + FastCGI client when binary is on PATH (pinned image still honest n/a).

Verify

./script/phpunit.sh --filter BenchGateTest → 12/12
php script/check-bench-readme-sync.php (+ --self-test) → OK
./script/aot-smoke.sh → 9/9

Remaining Done-when

php-fpm numbers still need an image that ships php-fpm (or a documented install path in the release image)
Issue checkboxes / full close still need maintainer judgment on history+chart completeness

claim: implementer-worker-lane-b — starting this run (lane B #36385 % 3 == 1; lane overflow: #36143–#36148 / #36142 children closed/green; #36391 claimed <48h by runtime-migration; #36386 claimed <48h for **107).

Next Done-when slice: legacy micro-suite README table must sync from committed RESULTS (same honesty as v2) and/or publish tip history + chart ratchet so nightly/chart Done-when is gate-enforced rather than manual.

claim: implementer-worker-lane-b — starting this run (lane B #36385 % 3 == 1; lane overflow: #36143–#36148 / #36142 children closed/green).

Next Done-when slice after #37689: legacy micro-suite README table must sync from committed RESULTS (same honesty as v2) and/or publish tip history + chart ratchet so nightly/chart Done-when is gate-enforced rather than manual.

Bench: legacy RESULTS.json sync + tip history/chart ratchet (#36385) (#…

Merged #37705 (Part of #36385) — implementer-worker-lane-b (lane B #36385 % 3 == 1; lane overflow: #36143–#36148 / #36142 children closed).

Slice

Done-when honesty for legacy README + nightly/chart ratchet:

Committed benchmarks/RESULTS.json; legacy README table must match (same as v2).
script/bench.php writes legacy RESULTS on regen; --render covers both suites.
Tip history benchmarks/history/7300b18e64.json; generate-bench-chart.php --check + check-bench-readme-sync enforce chart embeds RESULTS generated_at and matches regenerate.

Verify

php script/check-bench-readme-sync.php (+ --self-test) → OK
php script/generate-bench-chart.php --check → OK (4 history points)
vendor/bin/phpunit --filter BenchGateTest → 12/12
php script/check-size-budgets.php → OK
Docker image unavailable this host (GHCR denied / Launchpad PPA timeout on docker-build-22); no lowering touched so aot-smoke / north-star5-fast not re-run.

Remaining Done-when

php-fpm numbers still need an image that ships php-fpm
Issue checkboxes / full close still need maintainer judgment on history+chart completeness

claim: implementer-worker-lane-b — starting this run (lane B #36385 % 3 == 1; lane overflow: #36143–#36148 / #36142 children closed/green). Next Done-when slice after #37705: close remaining honesty gaps (web-request/php-fpm column, v2 program count/gate coverage, or another mergeable Done-when checkbox).

Bench: ship php-fpm in CI image and fill web-request column (#36385) (#…

Merged #37707 (Part of #36385) — implementer-worker-lane-b (lane B #36385 % 3 == 1; lane overflow: #36143–#36148 / #36142 children closed).

Slice

Done-when web-request / php-fpm column:

php-compiler:22.04-dev Dockerfile installs php${PHP_VERSION}-fpm.
bench-web-request.php finds versioned /usr/sbin binaries; committed WEB_REQUEST.json now has real php_fpm FastCGI numbers (200 paired requests) + chart refresh.
Gate: Dockerfile must list fpm; null php_fpm_s still requires an honesty note.

Verify

PHP_COMPILER_BENCH_WEB_REQUESTS=200 php script/bench-web-request.php (with php8.2-fpm installed) → php_fpm filled
generate-bench-chart.php --check → OK
vendor/bin/phpunit --filter BenchGateTest → 12/12
check-bench-readme-sync.php / check-size-budgets.php → OK

Remaining Done-when

Hosts must make docker-build-22 (or otherwise install fpm) before the tagged image has the binary without a one-shot apt.
Maintainer judgment on full issue checkbox close / history+chart completeness.

claim: implementer-worker-lane-b — starting this run (lane B #36385 % 3 == 1; lane overflow: #36143–#36148 / #36142 children closed/green). Next Done-when slice after #37707: v2 program count / gate coverage honesty (16 programs + bench-gate coverage), or remaining measurement honesty gaps.

Bench: v2 geomean +10% band catches diffuse AOT regressions (#36385) (#…

Merged #37710 (Part of #36385) — implementer-worker-lane-b (lane B #36385 % 3 == 1; lane overflow: #36143–#36148 / #36142 children closed).

Slice

Done-when suite geomean band (deliverable: +10% on geometric mean of AOT/Zend ratios):

bench-gate --v2 compares suite geomean vs BASELINE.json (geomean_tolerance_percent=10).
php script/bench-gate.php --self-test-v2-geomean — uniform +15% (under per-case +20%) must trip geomean.ratio.

Verify

./script/phpunit.sh --filter BenchGateTest → 13/13
--self-test-v2-2x + --self-test-v2-geomean → OK
check-bench-readme-sync + check-size-budgets → OK

claim: implementer-worker-lane-b — starting this run (lane B tie-break #36385 % 3 == 1; lane overflow: #36143–#36148 / #36142 children closed). Next: Done-when remaining — default bench.php --v2 wall under 15 min (VM timeouts on binary-trees/fannkuch).

claim: implementer-worker-lane-b — starting this run (lane B tie-break #36385 % 3 == 1; lane overflow: #36143–#36148 / #36142 children closed; #36391/#36386 claimed <48h by peer lanes). Next: Done-when remaining — default bench.php --v2 wall under 15 min (VM timeouts on binary-trees/fannkuch).

Bench: scale binary-trees/fannkuch so v2 VM finishes under 15 min wall (

Merged #37720 (Part of #36385) — implementer-worker-lane-b (lane B #36385 % 3 == 1; lane overflow: #36143–#36148 / #36142 children closed; #36391/#36386 claimed <48h).

Slice

Scale binary-trees maxDepth 10→7 and fannkuch-redux n 8→6 so VM/JIT finish under the 25s cap (no more full-cap timeout burns with n/a). @bench-skip-vm + wall projection in check-bench-readme-sync.php (projected 465.8s / 900s budget).

Verified

./script/phpunit.sh --filter BenchGateTest → 14/14
php script/check-bench-readme-sync.php / --check-wall → OK
php script/bench-gate.php --v2 → OK
./script/aot-smoke.sh → 9/9
make north-star5-verify-fast → OK

Navigation Menu

Repository navigation

Performance: benchmark suite v2 — the benchmark-game set plus web-request and real-app workloads, per-commit history in JSON, a public chart page, and `bench-gate` extended to all of it (benchmarks/, script/bench-gate.php, docs/pages/bench.html) #36385

Category

Why

Deliverable

Done when

51 remaining items

Slice

Verify

Remaining Done-when

Slice

Verify

Remaining Done-when

Slice

Verify

Remaining Done-when

Slice

Verify

Slice

Verified

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Performance: benchmark suite v2 — the benchmark-game set plus web-request and real-app workloads, per-commit history in JSON, a public chart page, and bench-gate extended to all of it (benchmarks/, script/bench-gate.php, docs/pages/bench.html) #36385

Description

Category

Why

Deliverable

Done when

Activity

PurHur commented on Sep 3, 2026

PurHur commented on Sep 3, 2026

PurHur commented on Sep 3, 2026

PurHur commented on Sep 3, 2026

PurHur commented on Sep 5, 2026

PurHur commented on Sep 5, 2026

PurHur commented on Sep 7, 2026

51 remaining items

PurHur commented on Sep 9, 2026

Slice

Verify

Remaining Done-when

PurHur commented on Sep 9, 2026

PurHur commented on Sep 14, 2026

PurHur commented on Sep 14, 2026

Slice

Verify

Remaining Done-when

PurHur commented on Sep 14, 2026

PurHur commented on Sep 14, 2026

Slice

Verify

Remaining Done-when

PurHur commented on Sep 14, 2026

PurHur commented on Sep 14, 2026

Slice

Verify

PurHur commented on Sep 15, 2026

PurHur commented on Sep 18, 2026

PurHur commented on Sep 18, 2026

Slice

Verified

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Performance: benchmark suite v2 — the benchmark-game set plus web-request and real-app workloads, per-commit history in JSON, a public chart page, and `bench-gate` extended to all of it (benchmarks/, script/bench-gate.php, docs/pages/bench.html) #36385