Split image tests into one file per feature area and test more features

.github/workflows/service_lint.yml

@@ -43,6 +43,7 @@ jobs:
             --color=always --severity=warning \
             scripts/*.sh \
             scripts/tests/*.sh \
+            scripts/tests/image/*.sh \
             src/common/usr/local/bin/* \
             src/s6/usr/local/bin/* \
             .oss-scanner/run-variation \

AGENTS.md

@@ -115,18 +115,19 @@ When modifying the version pipeline, the base config (`php-versions-base-config.
   - a new way for untrusted input to reach the image (a web server location, a healthcheck, a workflow trigger), a change to `SECURITY.md` scope, or a new intentional default the scanner should not report: `.oss-scanner/threat_model.md`
 
   After changing any of these, build it with `docker build -f .oss-scanner/Dockerfile -t oss-scanner .`, then confirm the affected variations start with no network: `docker run --rm -it --privileged --network none --user root oss-scanner bash`, then `.oss-scanner/run-variation <variation> &` and `curl localhost:8080/healthcheck`.
-- Every PR run starts with `service_lint.yml` (script tests, actionlint for workflows, hadolint for `src/variations/*/Dockerfile` and `.oss-scanner/Dockerfile` with project decisions recorded in `.hadolint.yaml`, ShellCheck at warning severity for `scripts/*.sh`, `src/common/usr/local/bin/*`, `src/s6/usr/local/bin/*`, `.oss-scanner/run-variation`, and `entrypoint.d/*.sh`) before any image builds. New linters go there. The local commands are in the contributing docs.
-- `scripts/test-image.sh` checks that an image starts, runs as an unprivileged user, has no world-writable files or directories (outside sticky ones like `/tmp`), reports the expected PHP version, loads the default extensions, reaches a healthy HEALTHCHECK, and (for images with a web server) serves a mounted `index.php` through NGINX, Apache, or Caddy. Run it locally against any image before opening a PR that touches startup behavior. Keep the checks generic across variations: it detects the web server from the image's `*_HTTP_PORT` environment variables rather than the image name.
+- Every PR run starts with `service_lint.yml` (script tests, actionlint for workflows, hadolint for `src/variations/*/Dockerfile` and `.oss-scanner/Dockerfile` with project decisions recorded in `.hadolint.yaml`, ShellCheck at warning severity for `scripts/*.sh`, `scripts/tests/image/*.sh`, `src/common/usr/local/bin/*`, `src/s6/usr/local/bin/*`, `.oss-scanner/run-variation`, and `entrypoint.d/*.sh`) before any image builds. New linters go there. The local commands are in the contributing docs.
+- `scripts/test-image.sh <image> [php-version]` runs the image tests in `scripts/tests/image/`: one file per feature area (`php.sh`, `fpm.sh`, `security.sh`, `entrypoint.sh`, `laravel-automations.sh`, `permissions.sh`, `lifecycle.sh`, `web-server.sh`, `ssl.sh`, `trusted-proxy.sh`, `frankenphp.sh`, `octane.sh`), each `test_*` function one test, shared helpers and image facts in `lib.sh`, mounted files in `fixtures/` (`laravel/artisan` records commands instead of running Laravel). Every test runs even after one fails. Add a test to the file for its area, or a new file for a new area; never add checks to `scripts/test-image.sh` itself. Keep tests generic across variations: a file or test declares what it needs with `requires` (`healthcheck`, `web_server`, `https`, `frankenphp`, `fpm`), detected from the image's environment variables rather than its name. Run it locally against any image before opening a PR that touches startup behavior; `TEST_ONLY=<file>` runs one area.
 - Every build job records its image details (tags, Depot Registry reference, build ID, digest) as an `image-details-*` artifact; the publish job adds a `.published.json` record with the compressed size per architecture. `service_report.yml` merges them with `scripts/build-summary.sh` (the published record wins) into one table on the run summary and in the PR comment.
 - `fail-fast` is off, so one failed image never cancels the others. `trigger_auto-retry-failed-builds.yml` re-runs the failed jobs of a production, beta, or PR run once when only a few jobs failed and no newer run exists. It also handles runs that end `cancelled` because a Depot runner never connected, but skips a run cancelled by hand (no failed jobs). It finds runs by workflow file, because `workflow_run.name` is the run's `run-name` title. If that also fails, or too many jobs failed to look transient, it opens a `ci-failure` issue assigned to the maintainer with the failed jobs, their first error annotation, the log tail, and a link to re-run. Only production and beta runs open issues; PR runs have the PR comment. GitHub's own failure emails do not cover this case reliably: scheduled runs notify whoever last edited the cron line, and re-run attempts are triggered by the Actions bot.
 
 ## Verification
 
-There is no automated test suite for image logic. To verify changes:
+To verify image changes:
 1. Build the affected variation locally with `scripts/dev.sh`
-2. Run the built image and confirm the change works: `docker run --rm -it serversideup/php:8.4-fpm-nginx-bookworm sh`
-3. For entrypoint script changes, test on both Debian and Alpine builds
-4. Run `shellcheck` on any modified shell scripts when available
+2. Run `bash scripts/test-image.sh <tag>` against it, and add a test in `scripts/tests/image/` for new behavior
+3. Run the built image and confirm the change works: `docker run --rm -it serversideup/php:8.4-fpm-nginx-bookworm sh`
+4. For entrypoint script changes, test on both Debian and Alpine builds
+5. Run `shellcheck` on any modified shell scripts when available
 
 ## Key Design Decisions
 

docs/content/docs/1.getting-started/99.contributing.md

@@ -98,7 +98,7 @@ Every pull request starts with a `lint` group that runs before any image is buil
 docker run --rm -v "$PWD:/repo" -w /repo rhysd/actionlint:latest -color
 
 docker run --rm -v "$PWD:/repo" -w /repo koalaman/shellcheck:stable --severity=warning \
-  scripts/*.sh scripts/tests/*.sh src/common/usr/local/bin/* src/s6/usr/local/bin/* .oss-scanner/run-variation $(find src -path '*entrypoint.d*' -name '*.sh')
+  scripts/*.sh scripts/tests/*.sh scripts/tests/image/*.sh src/common/usr/local/bin/* src/s6/usr/local/bin/* .oss-scanner/run-variation $(find src -path '*entrypoint.d*' -name '*.sh')
 
 docker run --rm -v "$PWD:/repo" -w /repo hadolint/hadolint:latest hadolint src/variations/*/Dockerfile .oss-scanner/Dockerfile
 
@@ -109,13 +109,41 @@ The script tests live in one file, `scripts/tests/run.sh`, as plain assertions o
 
 The Depot runner labels we use are listed in `.github/actionlint.yaml` so actionlint recognizes them. New linters belong in `.github/workflows/service_lint.yml`.
 
+### Image tests
+`scripts/test-image.sh` runs an image through the tests in `scripts/tests/image/`, one file per feature area (`php.sh`, `entrypoint.sh`, `laravel-automations.sh`, `web-server.sh`, `ssl.sh`, ...). CI runs it against every image before anything is published. Run it against any image you built or pulled:
+
+```bash [Terminal]
+bash scripts/test-image.sh serversideup/php:8.4-fpm-nginx
+
+# Run one feature area
+TEST_ONLY=web-server bash scripts/test-image.sh serversideup/php:8.4-fpm-nginx
+```
+
+Every test runs even after one fails, and the summary lists each failure with its output.
+
+To add a test, add a `test_` function to the file for its feature area, or create a new file in `scripts/tests/image/`. Each function is one test, named after the function. `requires` skips the file when the image doesn't have the feature (or just the test, when you call it inside one; `skip "reason"` skips one for any other reason), and `setup` starts one container that every test in the file shares:
+
+```bash [scripts/tests/image/web-server.sh]
+requires healthcheck web_server
+
+setup() {
+    start_container
+}
+
+test_serves_php() {
+    expect_body / "serversideup-php-ok:${php_version}"
+}
+```
+
+A test passes when it returns and fails when a command fails or it calls `fail "message"`. The helpers (`start_container`, `run_php`, `expect_body`, ...) and the image facts (`php_version`, `http_port`, `container`, ...) live in `scripts/tests/image/lib.sh`. Files the tests mount go in `scripts/tests/image/fixtures/`: `public/` is the web server's document root, `laravel/` is a stand-in Laravel app whose `artisan` records each command it receives, and `entrypoint.d/` holds sample startup scripts.
+
 ### What happens when you open a pull request
 Every pull request that touches `src/`, `scripts/`, or the workflows builds the full image matrix for both `amd64` and `arm64`. Jobs are grouped by variation, so you can see at a glance whether `cli`, `fpm`, `fpm-apache`, `fpm-nginx`, and `frankenphp` all build.
 
 - **Pull requests from forks** are built on isolated builders and never leave them. Each build job also loads its amd64 image and runs the image tests against it, so a green check means your change builds everywhere and passes the tests on amd64. Fork builds can't save the compiled PHP base images, so they start from the official PHP images on Docker Hub instead. In the hours after a new PHP release, before Docker Hub has it, fork builds fail. Changes to how PHP is compiled are only tested when a maintainer runs the build for your PR.
 - **Pull requests from this repository** go through the same three phases as a release: build, test, publish. Images are published to `serversideup/php-dev` with your PR number as the tag prefix (for example `serversideup/php-dev:699-8.4-fpm-nginx-alpine`) only after the image tests pass. A comment on the PR shows the status and the pull commands.
 - **Every run ends with a summary table** listing each image with its compressed size per architecture, so a size regression is visible before merge.
-- **Nothing is published until it passes the image tests.** Every build is saved to a private registry first. Every image is pulled from there on native amd64 and arm64 runners and checked: it starts, runs unprivileged, has no world-writable files or directories, reports the right PHP version, loads the default extensions, becomes healthy, and serves a PHP file through its web server. Only then is the exact same image promoted to Docker Hub, and the published digest is checked against the tested one, so what you pull is what was tested. If any image fails to build or fails its test, nothing from that run is published. You can run the same check locally with `bash scripts/test-image.sh serversideup/php-dev:<tag>`.
+- **Nothing is published until it passes the image tests.** Every build is saved to a private registry first. Every image is pulled from there on native amd64 and arm64 runners and checked: it starts, runs unprivileged, has no world-writable files or directories, reports the right PHP version, loads the default extensions, becomes healthy, stops cleanly, serves PHP through its web server in every SSL mode and trusted proxy setting, and runs startup scripts and Laravel automations the way the docs describe. Only then is the exact same image promoted to Docker Hub, and the published digest is checked against the tested one, so what you pull is what was tested. If any image fails to build or fails its test, nothing from that run is published. You can run the same check locally with `bash scripts/test-image.sh serversideup/php-dev:<tag>`.
 - **Transient failures heal themselves.** If a few jobs fail because a registry or mirror hiccupped, CI re-runs just those jobs once. If a release or weekly rebuild is still red after that, CI opens an issue for the maintainers with the failed jobs and their errors.
 - **Maintainers** can publish a fork's images for testing by running the "Docker Publish (PR Images)" workflow manually with the PR number.