fix(evals): follow-ups to the TUI overhaul

[miguelg719]

Follow-up to the evals TUI overhaul (#3087, #3088, #3089, #3090, all merged).

Summary

Four small fixes from a whole-stack review, each in code the stack adds or touches:

• Provider 429 retry backs off (feat(evals): per-provider concurrency under the global cap #3087's scheduler). A provider-throttled row released its slot and re-acquired it at once, calling the provider again milliseconds after the 429 and usually wasting its single retry. It now waits PROVIDER_RETRY_DELAY_MS (10 s, abort-aware) first, like the Browserbase create-429 path.
• harnesses.<h>.tool stays out of core-tier runs (feat(evals): config v2 with local overrides and one .env loader #3088). The per-harness tool default was folded into coreToolSurface, so it overrode core.tool for core runs too; an agent-mount-only default made every core run fail. It's now benchToolSurface, applied only when the run has bench tasks.
• A finished row's late log lines are dropped, and its log always closes (feat(evals): case-level live board, per-case logs and a run summary #3090). Timers or child output a row leaves behind keep its row context: their lines reopened the row's log file (leaking the handle), could print over the summary with -v, and late phases could redraw the row. The log now closes in a finally, so a row that throws still flushes its file.
• The cursor comes back if a run exits mid-board. Esc, Ctrl+C and SIGTERM in evals run mode call process.exit with the cursor hidden; the board now registers an exit hook while it hides it.

Test plan

• vitest packages/evals/tests: 1210 passing (the three Playwright-backed integration files need a local browser)
• tsc, oxlint (0 errors), fmt:check
• Every new test fails with its fix reverted: provider backoff, core vs bench tool surface (both tiers), late-write drop, late-line drop in the runner, close-on-throw, cursor exit hook

---

Summary by cubic

Follow-up fixes to the evals TUI overhaul (#3087–#3090), each driven by a whole-stack review and covered by a regression test that fails with the fix reverted.

Bug Fixes

• Provider-throttled rows wait 10 s (PROVIDER_RETRY_DELAY_MS) before retrying, instead of re-acquiring their slot and calling the provider again milliseconds after the 429.
• harnesses.<h>.tool no longer overrides core.tool for core-tier runs; the per-harness tool default applies only when the run has bench tasks.
• A finished row's late log lines are dropped, and its log file always closes, including when the row throws.
• The live board re-shows the cursor if the process exits mid-run (esc, Ctrl+C, SIGTERM).

^{Written for commit 0e11a0a. Summary will update on new commits.}

[changeset-bot]

⚠️ No Changeset found

Latest commit: 0e11a0a

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[cubic-dev-ai]

2 issues found across 12 files

Confidence score: 3/5

• In providerConcurrency.ts, a non-TTY run with no other live handles can exit during the unreferenced 10-second wait, skipping the retry. Keep the retry timer referenced until it fires.
• In run.ts, mixed core/bench runs can give core rows the harness default tool when building their context. Keep the bench default separate and apply it only to bench rows.

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/evals/framework/providerConcurrency.ts">

<violation number="1" location="packages/evals/framework/providerConcurrency.ts:604">
P2: The default `sleep` unrefs its timer, so a non-TTY run with no other live handles can exit during this new 10-second wait and skip the retry. Keep the retry timer referenced until it fires.</violation>
</file>

<file name="packages/evals/tui/commands/run.ts">

<violation number="1" location="packages/evals/tui/commands/run.ts:465">
P2: Mixed core/bench runs overwrite `core.tool` with the harness default, and core rows then pass that value to `buildCoreContext`. Keep the bench default separate and apply it only when planning bench rows.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic</sub>

[cubic-dev-ai]

P2: The default sleep unrefs its timer, so a non-TTY run with no other live handles can exit during this new 10-second wait and skip the retry. Keep the retry timer referenced until it fires.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/evals/framework/providerConcurrency.ts, line 604:

<comment>The default `sleep` unrefs its timer, so a non-TTY run with no other live handles can exit during this new 10-second wait and skip the retry. Keep the retry timer referenced until it fires.</comment>

<file context>
@@ -594,10 +598,16 @@ export async function runGatedRow(options: RunGatedRowOptions): Promise<TaskResu
+      // Back off before the retry. A provider row has just released its slot,
+      // so without a wait it re-acquires at once and calls the provider again
+      // milliseconds after the 429.
+      await sleep(
+        source === "browserbase"
+          ? (options.browserbaseRetryDelayMs ?? BROWSERBASE_RETRY_DELAY_MS)
</file context>

[cubic-dev-ai]

P2: Mixed core/bench runs overwrite core.tool with the harness default, and core rows then pass that value to buildCoreContext. Keep the bench default separate and apply it only when planning bench rows.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/evals/tui/commands/run.ts, line 465:

<comment>Mixed core/bench runs overwrite `core.tool` with the harness default, and core rows then pass that value to `buildCoreContext`. Keep the bench default separate and apply it only when planning bench rows.</comment>

<file context>
@@ -459,6 +459,12 @@ export async function runCommand(
+  // The harness's tool default applies to bench rows only; core-tier runs
+  // keep core.tool (or --tool).
+  if (options.benchToolSurface && tasks.some((task) => task.tier === "bench")) {
+    options = { ...options, coreToolSurface: options.benchToolSurface };
+  }
+
</file context>