[Copilot]

Fixes #4205

Real-model memory evaluations start a private TraceDecay daemon before initializing each scenario and keep ownership of their commands, socket, profile, and fixture through cleanup. Setup, daemon, agent, cancellation, and cleanup failures remain visible in the evaluation report. Owned process groups are stopped and reaped before their inputs are removed; an unconfirmed shutdown retains ownership and reports the remaining paths and PIDs.

The runner now uses canonical retained fact-store responses throughout seeding and assertions. Missing or malformed fact content fails instead of passing a zero-count assertion as empty content. It unwraps evidence/effect payloads, sends source_label, distinguishes list projections from search hits, and reads typed trust and retrieval telemetry. Scenario source selectors retain their existing meaning. The selected shipping binary and existing model-cost opt-in are preserved.

Validation on 565eb581ad9126f50ea8067e2b26b7b87a0f8e51:

• All 45 tests in //evals/memory:real_model_runner_test passed through Bazel. Four focused contract regressions failed before the fix and passed afterward.
• An independent real-binary run used the maintained memory-supersede-without-dup scenario. Private daemon startup and initialization, canonical fact seeding, list/search reads, and the Hermes installer all succeeded under an isolated HOME. Updating the same FactId to pnpm passed all three scenario assertions.
• That run verified the daemon was reaped, its PID was absent, and the temporary profile and socket directory were removed.
• The CLI binary was built through Bazel at df24429c7a; only the Python runner and its tests changed afterward. The verified binary SHA-256 was 75a33bc296f9266de60cb79c2460e300f8cffec0fdef395175092ea6c879d7d8.
• The prior exact-head memory suite passed 12 tests and the broker-handoff regression passed once. The Rust sources are unchanged by these two Python commits.
• Independent source review accepted the canonical caller cutover. Hosted verification of the published head remains required before merge.

No model/provider turn was run. The runtime result covers the real deterministic setup, fact operations, installer, assertions, and cleanup.

Latest Linux correction at a2efdb7ae5d483b50af99ac49ff383e8a10a27b5:

• Fleet run 39881981798 passed 168 of 169 targets. The sole failure was a stale Kiro fixture expecting {} when the real daemon completed its advisory cycle and emitted a valid zero-finding lookup notice.
• Adopted the existing master fixture correction: preserve the baseline response, require the exact feedback prefix and canonical typed notice validation, and assert both finding counts remain zero. No eval or production code changed in this follow-up.
• The exact formerly failing host_event_fixture_test::native_host_event_fixtures_execute_provider_admission_paths test passed remotely: 1 passed, 0 failed, 135 filtered out (16.84s). Bazel invocation. Rustc and TestRunner were remote, local fallback disabled, priority 10, uncached test execution.
• Bazel formatting, commitlint, and whitespace checks passed. Fresh hosted CI on this pushed head remains required before merge.

[changeset-bot]

⚠️ No Changeset found

Latest commit: a2efdb7

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[ScriptedAlchemy]

Cross-referenced #4205 and pushed 1be4cbb.

The remaining setup hole was profile allocation: create_eval_environment() ran before the guarded scenario lifetime, so an OSError could escape before main wrote report.json. Acquisition now happens inside the existing error-reporting path, and cleanup tolerates an environment that was never acquired.

The new regression calls the actual main and environment-creation path, makes TemporaryDirectory raise OSError("No space left on device"), and verifies exit code 1, exactly one persisted structured error report, and zero daemon/Hermes/Cursor calls. It failed before the fix and passes afterward.

Validation: python3 evals/memory/test_run_real_model.py — 21 tests passed; git diff --check and conventional commit lint passed. The isolated fake-daemon tests needed Unix-socket permission. No paid model invocation or native build was used.

[ScriptedAlchemy]

Published the completed lifecycle repair in f934c5e, preserving the concurrent branch changes and their regression coverage.

Commands and daemon teardown share one owned-process helper with forced-stop/reap fallback and runnable-descendant verification. The scenario owns its fixture before daemon startup or seeding, keeps inputs until shutdown is confirmed, and records original operation failures separately from cleanup failures. Signal/report handling covers binary resolution, scenario gaps, teardown, and report publication.

Directory cleanup now reports failures while attempting independent cleanup, identifies the retained socket directory, and preserves reports when path inspection is denied. Removed the obsolete fixture cleanup wrapper and duplicate deletion.

Validation: all 38 focused Python tests passed on the exact final tree (25.011s), including real process groups and both SIGINT/SIGTERM. Python syntax, diff checks, and commitlint passed. The Bazel target remains //evals/memory:real_model_runner_test; current-head CI supplies native/Bazel verification. No model tokens were spent. This addresses #4205's lifecycle and failure-reporting scope.

[ScriptedAlchemy]

Pushed d28a2a8 on top of the concurrent fc71833 allocation fix, preserving its raw-path ownership guard and cancellation-identity test.

This follow-up removes silent cleanup-error suppression during failed allocation. If environment construction or cancellation fails and profile deletion also fails, the existing scenario report now preserves the original failure, the cleanup failure, and the retained profile path. Retained-path inspection accepts both the raw path from a failed Path conversion and a completed Path object, and reports inaccessible paths conservatively. Cancellation still exits 130 before a second scenario or process can start.

Validation: all 41 Python tests passed on this exact tree in 23.329 seconds. Coverage includes environment-copy, path-conversion, and owner-construction cancellation, deletion/stat denial, real subprocess/descendant cleanup, signals, and interrupted report writing. Syntax, diff checks, and commitlint pass. Native/Bazel validation remains in CI; no live model turn was performed.