Model picker: probe reachability per credential, hide dead models, curated short list with price + latency #1979

Aaron (2026-10-28): the model picker shows models that don't work, and there are far too many to look through.

Ground truth

• The catalog comes from models.dev, which is every model each provider offers, not what our credentials can call.
• A live Converse probe on 2026-10-28 found that our Bedrock key reached about 10 of roughly 150 listed Bedrock ids. The rest return IAM AccessDenied, "not available for this account" (a missing Marketplace agreement), or ValidationException (bare ids that are profile-only). The same is true for every other provider.
• Every entry looks identical in the picker, so a dead choice only fails when you send a message.

Design: show what works, and only a short list by default

1. Reachability probe per credential. On connect, on a timer (daily), and on demand, the server runs a one-token probe for each catalog model of each connected provider and caches {provider, model, credential_fp, status: ok|denied|unavailable|invalid|rate-limited|error, latency_ms, checked_at}.

   • Probes are rate-limited and cost fractions of a cent.
   • The cache is shared across sessions and machines on the hub.
   • Statuses are named. A model that fails a real request gets demoted until its next probe.

2. The picker hides dead models by default. A "show unavailable (N)" toggle reveals them greyed, with the reason ("not in your Bedrock allowlist", "needs Marketplace agreement"). Unprobed models show as "unchecked", never as working.

3. A short list first. The default view is a curated set:

   • pinned: the user's favorites;
   • recommended: per deployment, set in config or the charter (e.g. Sonnet 5.5 default, Haiku 5.5 fast, Opus 5.5 strong);
   • recently used.

   Everything else is behind search. Group by role ("default / fast / strong / open-weight") rather than by provider.

4. Price and capability inline. Each row shows $in/$out/cache-read per Mtok, context length, and tool support from the catalog, plus measured latency. Sort by "cheapest that works" when searching.

5. Fail fast on a dead pick. If a session's chosen model is later found unreachable, the composer shows a named warning with a one-click switch to the recommended equivalent. It is never a silent failure mid-turn.

6. Config pins. A deployment can set model.allowlist / model.recommended so a lab only ever sees its approved models.

Acceptance

• With the hub's current Bedrock key, the default picker lists only probe-OK models, and the count matches a probe run (test with a fake provider: ok, denied, and unavailable models present → only ok shown; the toggle reveals the rest with reasons).
• A newly granted model (e.g. after an IAM change) appears within one probe cycle or a manual refresh.
• Picking a model that then fails demotes it, and the warning appears.
• The default view shows at most about 12 entries.