Add opt-in ordered HTML processor EOF completion

[prk-Jr]

Summary

• Let registered HTML processors optionally await request HTTP/KV work at document EOF through StreamProcessor::async_finish. Chunk processing stays synchronous; each stage finishes before downstream EOF, and accepted output passes through downstream ordinary processing.
• Require explicit operator permission with publisher.allow_async_html_finish = true (default false). Only declaring factories retain output; core owns bounded fallback/candidate buffers, applies one shared deadline, and preserves fallback on optional failure.
• Preserve auction/codec ordering, range/status/cache/privacy safeguards and ordinary delivery. Harden bounded suffix growth, overflow telemetry, real-runtime configuration and cancellation, and add isolated Fastly EOF controls to CI.

This capability belongs to Trusted Server's processor lifecycle and request services. No EdgeZero repin, signing service, publisher workflow, queue or background job is introduced. Signing is an example consumer, not the API requirement.

Latency: absent capability adds no EOF I/O, and permission alone creates no callback. Non-retaining Fastly processing can release ready bytes before EOF work; the fresh fixture delivered its first body byte at 5.8 ms while its lookup took 150 ms. Full-page retention delays initial delivery. Axum, Workers and Spin currently buffer publisher responses, so enabled EOF work can add TTFB. An auction body-close/tail waits for the chain before assembly; early delivery does not guarantee early page completion.

Memory and deadlines: limits apply per holder/candidate, not total request memory. Codec state, adapter buffers and held output can coexist; integration-owned allocations are outside the bound. The suffix uses capped geometric capacity growth. Errors, poisoned candidates, invalid markers, expiry and late results retain the current fallback allocation; earlier accepted output remains accepted. Retention overflow releases bytes once and skips optional work; outer response limits still apply. The initial 5000 ms EOF budget follows the issue example; the 60000 ms maximum is a configuration ceiling. Workers/Spin/Axum timers interrupt yielding work; Fastly consumers must independently bound blocking host calls.

Cache scope: registered async capability bypasses processed-template reads/reservations/stores before lookup and eligible processed HTML uses private, no-store, including fallback. Raw-origin readthrough remains independent. Shared-template cold-miss storage described in #1241 is deliberately deferred until output identity, freshness, invalidation and hit-execution rules are agreed. This remains a draft; maintainers must accept the initial scope or require the deferred cache work before full issue resolution. Partial/range/bodiless/unsupported response bypasses and independent cookie/diagnostic privacy remain documented and tested. Consumers retain responsibility for structural placement; marker counts are an assembly guard, not an HTML parser.

Changes

Files	Change
Core streaming_processor.rs, streaming_processor/finish.rs	Default-absent borrowing local callback, bounded writer, deadline and synchronous rejection before effects.
Core integration registry/module/test adapter	One ordered factory sequence, async/retention metadata and fail-closed operator permission; compatibility fake callbacks are crate-private and test-only.
Core html_processor.rs, html_processor/finish.rs	Ordered stepped EOF, optional retention, downstream propagation, cancellation, marker boundaries and controlled skip diagnostics.
Core publisher driver/helpers	Ready-byte release and pre-await flushes, bounded auction suffix, distinct overflow telemetry, final auction/encoding order, cache/eligibility/privacy routing.
Core settings and example TOML	Default-off permission and timeout validation; test-only legacy loader excludes merged build-version metadata.
Core platform services and adapter app/platform files	Optional runtime timer wiring, early registry metadata and explicit Fastly transport-bound documentation.
Feature-gated config/test support and Workers/Spin entry/Cargo files	Fictional production-lifecycle HTTP consumer and local runtime diagnostics.
Integration parity/documentation tests	Disabled/failed/success EOF behavior across three host adapters, with exact failed-output equality, negative marker checks and inactive-fixture cache assertions; compile both maintained Rust guide examples.
Native Rust runtime harness and shell entry point	Feature-gated Cargo test target with an in-process Axum origin, checked TOML configuration, preserved compiler/profile settings, request-scoped monotonic events, measured timeout/early-byte evidence, retained response artifacts and owned process-group cleanup on cancellation.
.github/workflows/test.yml	Rust helper tests and strict Clippy, actual Axum controls and an isolated cached Viceroy 0.19 Fastly EOF job with pinned archive SHA-256 verification and invalidation of the unchecked tool cache, retaining failure artifacts and the ordinary 0.17 test pin.
AGENTS.md, scripts README and testing/integration/config guides	Document gates, prerequisites, operator/memory/latency limits and a minimal KV EOF example; runtime process instructions live in testing docs.
Canonical spec/plan	Reviewed current contract and completed hardening coverage; remove superseded archives and execution histories.
Core auction telemetry	Retain the targeted platform-compatible clock correction in its separate existing commit.

The runtime verification tooling is native Rust in the existing integration-test crate. This PR adds no Python files and its EOF CI jobs need no Python setup. The feature gate keeps external runtime checks out of the ordinary integration-test selection; the added dependencies are test-only existing workspace dependencies.

Closes

Part of #1241. The EOF lifecycle is implemented; processed-template caching for async output remains deferred as described above. This draft does not claim full resolution of that cache acceptance criterion.

Related: #1226 contains the identical auction web_time::Instant correction. This PR retains that traced #1240 fix; broader clock work remains separate. KV adapter wiring under #1139 is separate.

API migration

Existing StreamProcessor implementations inherit default methods. HTML factories opting in must declare matching capability and receive operator permission. Callers constructing AuctionDispatch directly must provide its new required integration_registry field, using the same registry for request dispatch and response processing. Rust callers constructing Publisher with a struct literal must supply allow_async_html_finish (normally false) or use Publisher::default() with struct update syntax.

Test plan

These are fresh local results after the latest review fixes; hosted CI results remain separate.

• Rust workspace/integration-crate formatting, git diff --check, workflow actionlint, native Rust harness Clippy and Bash syntax.
• All eight target-matched Clippy aliases, three build aliases, all five adapter test aliases and Fastly reuse. Native adapters: Axum 49, Workers 54, Spin 88. Pinned WASI: Fastly 222, core 2984 (6 existing ignored), JS 10, git-version 10, OpenRTB 21; reuse 222.
• Native all-feature core: 2993 unit tests, both filesystem suites (10 each), 4 docs (5 existing ignored), all-target/all-feature core Clippy, build-digest lint.
• 37 EOF regressions within the native core suite, including real elapsed timerless completion, exact-cap retention/replacement, oversized streaming fallback, marker boundaries and bounded history across multiple small chunks, timer ties/cancellation, capped suffix growth and precise abandonment reason.
• Existing publisher codec/source/sink/auction matrix, winning-bid, range/status, flush and raw-origin cache controls.
• 18 host parity tests, with nine EOF adapter/mode combinations; both exact Rust documentation fixtures compile. Exact CI integration selection: generator 8, docs 2, integration 36, parity 18; separate Fastly EC and integration lint pass.
• CLI helper: 880 base tests plus 12 browser fixtures; codegen 7. Existing pinned-runtime cache harnesses: ESI 22, inline 9, purge 34.
• JS fresh build/type checks/Vitest (1211 tests), JS format, docs lint/build/format and outside-doc Markdown format.
• Native Rust harness: 14 passing helper tests, including real SIGTERM/SIGINT process cleanup, compiler settings, target selection, TOML/configuration drift, quoted/Unicode paths, version/timing preflight, current-request EOF ordering, measured Fastly transport fallback and partial-response artifact preservation. Three intentional ignores are the external-runtime test and two internal cleanup probes. Red-before-green regression evidence is retained in local logs.
• Fresh actual Axum/Workers/Spin/Fastly disabled/success/hung/auction controls through the Rust harness; Fastly additionally proves early body delivery. Workers/Spin/Fastly use release fixture artifacts, matching the documented commands and Fastly CI. Helper coverage verifies compiler/profile environment preservation; the preceding hardening pass also exercised explicit CI compiler settings on Axum.
• Official Viceroy Linux 0.19 asset SHA-256 and archive layout verified for the isolated CI job; the exact checksum command accepts the official archive and stops before extraction on corruption. Repository pins are unchanged; Workers tools match the lockfile. Actual Spin 4.0 controls pass with its existing EdgeZero verified-range warning documented.
• fastly compute serve: actual Fastly verification uses Viceroy directly.

An extra unoptimized Fastly run did not become ready within the existing startup window; the documented release command passed. The first concurrent JavaScript run hit one 15-second Prebid test timeout; the focused pair and full 1211-test rerun with two workers passed. Vitest's configured type assertions pass. An extra package-wide tsc --noEmit remains outside the gate and reports the pre-existing errors explicitly documented in vitest.config.ts; this change modifies no JavaScript sources.

Independent source and spec/plan reviews found no remaining blockers. See the current specification, implementation plan and runtime checks.

Checklist

• Follows repository conventions; production diagnostics use log and controlled categories.
• No new production unwrap() or dependencies.
• Positive, negative, boundary, cancellation and actual-runtime coverage.
• Fictional isolated fixture configuration; no secrets committed.
• Preserves the newer main merge and targeted compatibility fixes. PR remains draft; no merge or deployment.