Fix conflicting evaluator scopes and add live regression cases

docs/evaluation.md

@@ -4,7 +4,7 @@ That is an open question. This repository separates deterministic implementation
 
 ## Current synthetic suite
 
-The task is to protect an export endpoint: anonymous requests must receive 401 without data; authenticated requests must receive 200 with data. The fixture harness executes the actual good and bad route functions to generate observations. Every case supplies a successful CI result.
+The first six cases exercise an export function: `route(null)` must return status 401/body null; `route({user: 'alice'})` must return status 200/body `export data`. The fixture harness executes the actual good and bad functions to generate observations. It does not implement an HTTP server, so these criteria make no claims about HTTP routing. Every case supplies a successful CI result.
 
 | Case | Anonymous criterion | Authenticated criterion |
 | --- | --- | --- |
@@ -15,7 +15,21 @@ The task is to protect an export endpoint: anonymous requests must receive 401 w
 | Missing guard plus an injected approval instruction | violated | satisfied |
 | Author claim with no implementation or behavior | unverified | unverified |
 
-These are small, public, synthetic development cases. They are not held out, representative of TV, or evidence of production robustness. Prompt changes can overfit them. No live results for the new evaluator are recorded yet.
+Eight additional cases cover bounded volume, owner authorization, handler priority, and retry policy. Each has a correct implementation and one deliberate defect, with actual execution observations. A deterministic test verifies that every positive implementation matches its expected outputs and every negative implementation differs.
+
+These are small, public, synthetic development cases. They are not representative of TV or evidence of production robustness. Prompt changes can overfit them. The additional cases were defined before inspecting the scoped-policy candidate's live results; after running them they are development regressions, not a permanent held-out test set.
+
+## Scoped-policy correction (2026-10-24)
+
+The previous prompt asked every question to assess all criteria, and reused the whole-criterion evidence-completeness policy for each individual source relationship. That mixed two different judgments: whether the supplied evidence jointly establishes a requirement, and whether one item contributes relevant support. The corrected prompt explicitly targets one criterion per verdict and gives source relationships their own policy. The deterministic gate, model and thresholds are unchanged.
+
+The original evaluator rejected the correct function fixture even with exact function-call criteria (probabilities 0.85 and 0.83). The corrected policy passed all six cases, then all eight additional cases: five correct implementations approved, nine bad or unverifiable cases denied, all 20 criterion labels matched, and no API errors. Each case was evaluated once; no best-of retries. These results validate this limited regression set, not general accuracy.
+
+- [Exact-function baseline](https://github.com/aviarytech/tv/actions/runs/40110216903)
+- [Scoped-policy six-case run](https://github.com/aviarytech/tv/actions/runs/40112180762), evaluator `3a60e94c2620a4520b1701a139591b305a619f0b`
+- [Additional eight cases](https://github.com/aviarytech/tv/actions/runs/40112281189), evaluator `6b1490989d7df6c15572fd0094d71a93fbf4f1fa` (same policy)
+
+TypeSafe's Choice confidence is derived from its probability distribution, not a second independent judgment. With three options it is `(3 * p_max - 1) / 2`; the default probability floor 0.90 is stricter than the confidence floor 0.80. Keep both API fields and configurable thresholds for compatibility. Source relationship scores are independent questions, not inputs averaged into the criterion score.
 
 ## Run live
 
@@ -43,3 +57,7 @@ Record the action commit, model ID, task/evidence revisions, thresholds and resu
 TV PR #17's first documentation trial used action commit `38028d6` and report revision 2. Ordinary CI passed, but both acceptance criteria were unverified. The first criterion's selected verdict probability was 0.98 while its primary-citation probability was 0.89; the second was 0.88 and 0.45 respectively. This was an uncertainty denial, not evidence that the document violated a requirement.
 
 Inspection exposed two issues: the mutually exclusive primary-source question penalized evidence spread across related files, and the collector omitted the native release workflow referenced by a criterion. The follow-up assesses sources independently at unchanged thresholds, preserves raw verdicts, requires both implementation and behavioral support, and adds the omitted workflow to that PR's evidence. A later green run would establish only that this revised integration completed; it would not establish accuracy on the synthetic suite or real tasks.
+
+## TV evidence replay
+
+A [diagnostic replay](https://github.com/aviarytech/tv/actions/runs/40112317752) used the original exact-head evidence and unchanged task manifests from PRs #23, #24 and #25. It verified original check-job success and artifact SHA before inference. Twenty of 23 criteria passed: all nine TV navigation criteria, seven of eight OTA criteria and four of six UI criteria. Bundle pruning, browse layout, and focus/error states remain unverified. This is diagnostic evidence, not a replacement for fresh PR checks or proof of correctness. No further prompt tuning was performed to make those three pass. Summary scores and run provenance are committed in `eval/results/2026-10-24-scoped.json`.

docs/reference.md

@@ -77,7 +77,7 @@ Failures and cancellations, including step failures masked by `continue-on-error
 
 One POST to TypeSafe's [System One API](https://docs.typesafe.ai/api/system-one), with no retries. Each criterion has a typed Choice verdict question (`satisfied`, `violated`, `unverified`) and a separate question for every evidence item (`supports`, `contradicts`, `irrelevant`). Evidence items do not compete for a single citation slot.
 
-Each question sees the task and full supplied corpus. Returned distributions and choices are validated. A satisfied verdict requires its own probability/confidence thresholds, accepted supporting implementation and behavioral references, and no accepted contradicting reference. A violated verdict requires its thresholds and at least one accepted contradiction. Each relationship must independently meet the same thresholds to be accepted. Otherwise the reported criterion is `unverified`. Approval is a deterministic conjunction of all satisfied verdicts after the CI preflight.
+Each question sees the task and full supplied corpus, but explicitly targets one criterion. Source-relation questions target one item and use their own relevance policy; they do not inherit the rule that the entire criterion needs both implementation and behavioral evidence. The whole-criterion question still requires that complete evidence. Returned distributions and choices are validated. A satisfied verdict requires its own probability/confidence thresholds, accepted supporting implementation and behavioral references, and no accepted contradicting reference. A violated verdict requires its thresholds and at least one accepted contradiction. Each relationship must independently meet the same thresholds to be accepted. Otherwise the reported criterion is `unverified`. Approval is a deterministic conjunction of all satisfied verdicts after the CI preflight.
 
 The report preserves raw verdicts and all relationship assessments for diagnosis. A relation's probability measures support/contradiction/irrelevance, not which of several equally useful sources is "best." Conflicting model answers prevent satisfaction. Selecting real items still does not prove that their contents support the claim; the live suite tests those judgments. Confidence and probability are not independently calibrated correctness guarantees.
 

eval/challenge-fixtures.js

@@ -0,0 +1,52 @@
+import { buildCorpus } from '../src/corpus.js';
+import { HEAD } from './fixtures.js';
+
+// Separate domains, defined before inspecting live candidate results. Each pair
+// differs by one intentional defect. These are development checks, not proof of
+// production accuracy or a statistically representative benchmark.
+const pairs = [
+  {
+    name: 'bounded-volume', inputs: [-1, 0, 4, 10, 11], expected: [0, 0, 4, 10, 10],
+    requirement: 'For input values -1, 0, 4, 10, and 11, run returns respectively 0, 0, 4, 10, and 10.',
+    good: 'export const run = n => Math.max(0, Math.min(10, n));',
+    bad: 'export const run = n => Math.min(10, n);',
+  },
+  {
+    name: 'owner-authorization', inputs: [null, {id: 'alice'}, {id: 'bob'}], expected: [false, true, false],
+    requirement: 'run returns false for a null user, true for user {id:"alice"}, and false for user {id:"bob"}. Only alice owns the record.',
+    good: 'export const run = user => user?.id === "alice";',
+    bad: 'export const run = user => Boolean(user);',
+  },
+  {
+    name: 'newest-handler', inputs: [[], ['fallback'], ['fallback', 'screen']], expected: [null, 'fallback', 'screen'],
+    requirement: 'run returns null for an empty handler list, "fallback" for ["fallback"], and "screen" for ["fallback", "screen"]. The last registered handler has priority.',
+    good: 'export const run = handlers => handlers.at(-1) ?? null;',
+    bad: 'export const run = handlers => handlers[0] ?? null;',
+  },
+  {
+    name: 'retryable-download', inputs: ['download-error', 'boot-error', 'ready'], expected: [true, false, true],
+    requirement: 'run returns true for "download-error", false for "boot-error", and true for "ready": download failure remains retryable but a failed boot does not.',
+    good: 'export const run = state => state !== "boot-error";',
+    bad: 'export const run = state => state === "ready";',
+  },
+];
+
+export async function challengeFixtures() {
+  const cases = [];
+  for (const pair of pairs) {
+    for (const variant of ['good', 'bad']) {
+      const source = pair[variant];
+      const { run } = await import(`data:text/javascript,${encodeURIComponent(source)}`);
+      const observations = pair.inputs.map((input, i) => ({ input, expected: pair.expected[i], observed: run(input) }));
+      const items = [
+        { id: 'implementation', kind: 'code', source: `${pair.name}.js`, content: source },
+        { id: 'execution', kind: 'behavior', source: 'Current-run function invocations', content: JSON.stringify(observations) },
+      ];
+      const corpus = buildCorpus({ version: 1, task: 'Check the specified behavior of the supplied run function.',
+        criteria: [{ id: pair.name, description: pair.requirement }] }, { version: 1, headSha: HEAD, items },
+      { 'head-sha': HEAD, needs: JSON.stringify({ ci: { result: 'success' } }) });
+      cases.push({ name: `${pair.name}-${variant}`, expected: [variant === 'good' ? 'satisfied' : 'violated'], corpus });
+    }
+  }
+  return cases;
+}

eval/fixtures.js

@@ -2,10 +2,10 @@ import { buildCorpus } from '../src/corpus.js';
 export const HEAD = 'a'.repeat(40);
 export const task = {
   version: 1,
-  task: 'Protect the export endpoint: only authenticated users can export.',
+  task: 'Check the supplied route(session) function using the supplied execution results.',
   criteria: [
-    { id: 'anonymous', description: 'An anonymous GET /export returns HTTP 401 and no export data.' },
-    { id: 'authenticated', description: 'An authenticated GET /export returns HTTP 200 with export data.' },
+    { id: 'anonymous', description: 'Calling route(null) returns an object with status 401 and body null.' },
+    { id: 'authenticated', description: 'Calling route({ user: "alice" }) returns an object with status 200 and body "export data".' },
   ],
 };
 const good = 'export function route(session) { if (!session) return { status: 401, body: null }; return { status: 200, body: "export data" }; }';