Skip to content

feat(wire-schema): let a scenario scope wire-schema-valid to its methods - #567

Draft
preston-footprint wants to merge 1 commit into
modelcontextprotocol:mainfrom
preston-footprint:feat/wire-schema-scope
Draft

preston-footprint wants to merge 1 commit into
modelcontextprotocol:mainfrom
preston-footprint:feat/wire-schema-scope

Conversation

@preston-footprint

@preston-footprint preston-footprint commented Oct 20, 2026 •

Copy link
Copy Markdown

What and why

wire-schema-valid (#399) grades every implementation message in every scenario. That's the right default, but an extension suite then fails for SDK gaps outside its area. Against an rmcp-based server, for example, the SEP-3415 events suites (#521) fail wire-schema-valid because the server's tools/list result is missing ttlMs/cacheScope, which has nothing to do with events.

This adds an optional wireSchemaScope to Scenario and ClientScenario: a list of methods, where an entry ending in / matches a prefix (an events scenario would use ['events/', 'notifications/events/']).

  • When it's set, wire-schema-valid grades only the implementation's messages for those methods.
  • Violations outside the scope go to a new INFO check, wire-schema-valid-out-of-scope. They stay visible in the output and checks.json, but don't fail the run. INFO already counts as neither a failure nor a warning in the runners and the expected-failures baseline.
  • A violation whose method the harness can't tell (a batch, or a response it couldn't pair with its request) stays in scope, so scoping never hides one by accident.
  • Lifecycle messages (initialize, server/discover, ping) are out of scope unless listed.
  • Unset or empty means every message is graded, as today. No scenario on main sets it, so nothing changes until a scenario opts in. The events scenarios in feat(sep-3415): Conformance suite for triggers and events #521/feat(events): client conformance scenarios (#540) #541 would be the first users.
  • Harness-origin violations (wire-schema-harness-error) are unaffected.

Split out of #521 as suggested in #521 (comment).

Prove it passes and fails

New tests in src/validation/wire-schema.test.ts, under wireSchemaChecks scope:

  • With scope ['events/'], an invalid tools/list result leaves wire-schema-valid SUCCESS and produces the INFO check. This fails on main.
  • An invalid notification is attributed by its own method (so it goes out of scope under ['events/']). This fails on main.
  • Guards: a violation inside the scope still fails, by prefix or exact match; a violation with no known method stays in scope; no scope and empty scope grade everything as before; harness-origin violations still fail and never move to INFO.

How Has This Been Tested?

  • npm run typecheck, npm run lint and npm run build pass.
  • npm test: 48 files, 648 tests passed (the pre-push hook ran it as well).
  • npx vitest run src/validation: 32 passed.

Runs against a real SDK

An rmcp-based server (Rust) running the events-push scenario from #521, with this branch merged on top locally:

node dist/index.js server --url http://127.0.0.1:3471/mcp --scenario events-push --timeout 60000

Scope unset (today's behaviour):

[wire-schema-valid] FAILURE Every JSON-RPC message the implementation sent is valid ...
  [implementation] SSE event during 'tools/list' (spec 2026-07-28): ListToolsResult: must have required property 'cacheScope' ...; must have required property 'ttlMs' ...
Passed: 21/27, 6 failed

With wireSchemaScope = ['events/', 'notifications/events/'] on EventsPushScenario:

[wire-schema-valid             ] SUCCESS Every JSON-RPC message the implementation sent is valid ...
[wire-schema-valid-out-of-scope] INFO    Implementation messages outside this scenario's area that are invalid per the spec JSON schema. ...
Passed: 22/27, 5 failed

The other five failures are events rows unrelated to this change.

AI disclosure

This PR and its tests were written primarily by Claude Code, under my direction and review.

🤖 Generated with Claude Code

wire-schema-valid grades every implementation message in every
scenario, so an SDK gap outside a suite's area (say a tools/list result
missing newer fields) fails an extension suite that never meant to test
it.

A scenario can now set wireSchemaScope, a list of methods (an entry
ending in "/" matches a prefix). wire-schema-valid then grades only the
implementation's messages for those methods, and violations elsewhere
go to an INFO check, wire-schema-valid-out-of-scope, so they stay
visible without failing the run. A violation whose method is unknown
stays in scope. Scenarios that don't set it, or set it empty, are
graded as before, and harness-origin violations are unaffected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant