feat(evals): case-level live board, per-case logs and a run summary

[miguelg719]

Stack

4 of 4 in the evals TUI overhaul. Base: #3089.
#3087 concurrency · #3088 config v2 + env · #3089 doctor · → #3090

Summary

A 6-wide HardBench run used to render as one row that kept flipping between running, passed and failed. Every suite names all its cases agent/<suite>, and the board keyed rows by task × model, so every in-flight case and every trial overwrote the same row. Concurrency looked like 1 whatever -c said.

Before (6 in flight):

    Task                                                     Model               Result
  ⣾ agent/hardbenchmark                                      openai/gpt-5.4-mini running

After:

Live board

• Row identity: progress events carry a row key (cell + case + trial) and a case label: short id, domain and question, read from id|task_id, web|website and ques|confirmed_task (framework/caseIdentity.ts).
• One row per case in flight: id domain · question · phase · elapsed.
  • Phases (session → agent → verify) are reported from inside the harness and target code through an AsyncLocalStorage row context (framework/rowContext.ts).
  • Elapsed time turns yellow once a row outlasts the p75 of finished rows.
• Status bar: progress, ✓ / ✗ / ⏱ counts, elapsed time and ETA, plus the queue line from feat(evals): per-provider concurrency under the global cap #3087 (running · queued · openai 6/6 · throttled). Queued rows are counted there and never take a row.
• Recent: the last few finishes with their outcome kind (max_turns, sdk_error, gated, ungraded) and session URL.
• Keys: esc stop, v logs off → all → one, ? help. They work in argv mode and the REPL, and the REPL no longer echoes keystrokes onto the board.
• Non-TTY (CI, pipes): one line per event, with no cursor escapes.

Logs

• Every case writes its own file: .trajectories/<run>/logs/<id>-<site>__<provider>-<model>.log. Default runs used to drop all output.
• Console output from inside a row goes to that row's file instead of over the board.
• -v streams the lines above the board, attributed by case id; --follow <id> streams one case.
• --json writes only the summary object to stdout; the header, board and logs go to stderr.

Summary

• Headline: suite · runs · duration · tokens, plus cost when the harness reports it.
• Per-cell table (harness × tool × model): pass rate, max_turns, sdk_error, gated (the judge passed it, a deterministic gate such as no_browser_use failed it), ungraded, retried.
• Failures: infra failures first, with reason and session URL; rubric fails listed by case id.
• Links: the experiment URL, the trajectory group and the log directory.

• Dead judge: when the judge graded nothing (a retired model, a bad key), every row fails closed, so the pass rate says nothing about the agent. The run now says so and exits 1:

Also

• evals run --help lists each harness's tool surfaces (default starred) and the suites, from a registry (framework/benchSuites.ts) that parse and the planner now share.
• The --preview header lists every constant column, so a single-cell run still names its tool surface.
• The README run section is rewritten around the new screenshots, replacing run.gif.

Notes

• Screenshots are rendered from the real ProgressRenderer and summary code with fixture events and a fixed clock (ANSI → HTML → Playwright), not mocked up.
• Apply the shared eval contract across existing harnesses #2903 (isolated CODEX_HOME) touches benchHarness.ts, runner.ts and run.test.ts. A trial merge conflicted in 6 hunks, all mechanical.

Test plan

• vitest packages/evals/tests: 954 passing. New: caseIdentity, rowLog, runSummary, progress (board layout, p75 colouring, non-TTY output, --json routing to stderr), resultsSummary, benchSuites, help. Also extended runnerProgressEvents (distinct row keys across cases × trials, phases landing on the right concurrent row, per-row log files) and run.test (v cycling, --follow, dead judge exit 1).
• tsc, fmt:check
• Live evals run b:hardbenchmark --harness codex --tool stagehand_facade -e browserbase: the board showed one row per case with phases and the queue line. (That run's facade calls were cancelled by a local ~/.codex/config.toml approvals_reviewer setting; Apply the shared eval contract across existing harnesses #2903's isolated CODEX_HOME fixes that and it's unrelated to this PR.)

---

Summary by cubic

The evals run experience gets a case-level live board, per-case log files, and an end-of-run summary. A suite run used to collapse every in-flight case into one row (all cases share the task name agent/<suite>), so concurrency looked like 1 regardless of -c; now each case in flight gets its own row.

• Board rows are keyed by case id, domain, question, and trial, show the current phase (session → agent → verify) and elapsed time, and list the latest finishes with their outcome kind and session URL; queued rows only appear in the status line.
• Every case writes its own log file under .trajectories/<run>/logs/; -v streams all case logs above the board, --follow <id> streams one, and the v/esc/? keys work in both the REPL and argv modes. Filenames keep the model provider and get ~2/~3 suffixes on collision; a filesystem error disables that row's file instead of failing the row.
• Runs end with a per-cell table (pass rate, max_turns, sdk_error, gated, ungraded, retried), infra failures with session URLs, and a dead-judge banner (exit 1) when the judge graded nothing.
• --json emits only the summary to stdout; non-TTY output is one line per event with no cursor escapes. The board draws on stderr and sizes to that stream, Ctrl/Meta chords stay off it so Ctrl+C still interrupts, and the preview header names an overridden model once.
• evals run --help lists each harness's tool surfaces and the suites from a shared registry that discovery also reads, so a listed suite always gets discovered. The README run section is rewritten around screenshots rendered from the real renderer, replacing run.gif.

^{Written for commit be4fc36. Summary will update on new commits.}

[changeset-bot]

⚠️ No Changeset found

Latest commit: be4fc36

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[cubic-dev-ai]

2 issues found across 40 files

Confidence score: 3/5

• caseIdentity.ts can give cases from different suites the same log file and summary name when dataset IDs overlap. Include the suite or task name in both identities.
• parse.test.ts duplicates the suite list from benchSuites.ts, so adding a benchmark can break this test or require a manual, order-sensitive update. Build the expected message from the shared suite list.

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/evals/framework/caseIdentity.ts">

<violation number="1" location="packages/evals/framework/caseIdentity.ts:66">
P2: This case label is not unique across suites, so overlapping dataset IDs cause separate rows to share a log file and summary name. Include the suite/task name in the derived log and display identity.</violation>
</file>

<file name="packages/evals/tests/tui/parse.test.ts">

<violation number="1" location="packages/evals/tests/tui/parse.test.ts:165">
P3: This test duplicates the suite list that benchSuites.ts is meant to own, so adding a benchmark to BENCH_SUITE_NAMES breaks this test and forces a manual, order-sensitive update here. Build the expected message from the registry instead: `new RegExp(`Supported: ${listBenchSuites().join(", ")}\.`)`, importing `listBenchSuites` from `../../framework/benchSuites.js`.</violation>
</file>

Architecture diagram

sequenceDiagram
    participant CLI as CLI / REPL
    participant Runner as Bench Runner
    participant RC as RowContext (AsyncLocalStorage)
    participant RowLog as RowLogWriter
    participant Harness as Harness / Agent
    participant BB as Browserbase
    participant Verifier as Verifier / Judge
    participant TUI as TUI Renderer / Board

    Note over CLI,Runner: Run lifecycle with per-case identity and logs
    
    CLI->>Runner: runEvals(options)
    Runner->>Runner: generate run token + logDir
    
    loop Each testcase
        Runner->>Runner: rowKey (cell + case + trial)
        Runner->>RC: runInRowContext(rowContext)
        RC->>RowLog: RowLogWriter (per-case file)
        
        alt Provider slot available
            Runner->>Harness: executeBenchTask(input)
            Harness->>RC: reportRowPhase("session")
            Harness->>BB: launch browser session
            BB-->>Harness: session URL
            Harness->>RC: reportRowPhase("agent", {sessionUrl})
            Harness->>Harness: run agent with tool surface
            Harness->>RC: reportRowPhase("verify")
            Harness->>Verifier: grade external trajectory
            Verifier->>RC: verify phase
        else Provider throttled/queued
            Runner->>TUI: onQueue (queued count)
            Note over Runner,TUI: Row stays queued, never takes board row
        end
        
        Note over Harness,RC: Log lines from EvalLogger or console
        Harness->>RC: logToRow(entry)
        RC->>RowLog: write entry to per-case log file
        RC->>TUI: onProgress log event (if verbose)
        
        alt Retry on 429 / backpressure
            Runner->>RowLog: attempt(n) marker
            Runner->>Harness: retry execution
        end
        
        RowLog->>Runner: close() + logPath
        Runner->>TUI: started/passed/failed with rowKey, case, outcome
    end
    
    Note over Runner,TUI: Live board rendering
    TUI->>TUI: onStart (one row per case in flight)
    TUI->>TUI: onPhase (session → agent → verify)
    TUI->>TUI: p75 elapsed threshold (yellow highlight)
    
    alt TTY animated
        TUI->>CLI: keys (esc/v/?)
        CLI->>TUI: onKey / setStopping
    else JSON mode
        Runner-->>CLI: --json summary
        CLI->>TUI: stderr only
    end
    
    Note over Runner,TUI: End-of-run summary
    Runner->>Runner: buildRunSummaryJson()
    Runner->>TUI: renderRunSummary
    
    alt Dead judge (graded 0)
        TUI->>CLI: red banner + exit 1
    else Normal completion
        TUI->>CLI: cell table + failures + links
    end

Loading

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>