Skip to content

Fill OpenRouter-shaped catalog facts from the bound schema, with per-field provenance #53

Closed

Description

@tombeckenham

Follow-up to #50 (activity + schemaEndpointId). Every listed model on GET /v1/models should carry the same facts OpenRouter publishes on its Models API standard, filled from this service’s sources — the bound request schema first, then the provider’s own listing/docs — with per-field provenance so a consumer can see why a value is there.

#58 lands the listing/docs slice for chat/text hosts. It skipped the thing we uniquely have: the generation schema the model is already bound to.

OpenRouter is compare-only. Never copy an OpenRouter value into another provider’s catalog, labeled or not, and not even to fill a null. Use it only as a discrepancy signal. If no first-party source states a fact, the field stays null.

Target shape

Mirror OpenRouter’s model object, in our camelCase catalog fields:

OpenRouter Us Primary source
context_length contextWindow listing / docs (schema almost never has this)
top_provider.max_completion_tokens maxOutput listing / docs (max_tokens.maximum is a weak ceiling)
architecture.input_modalities / output_modalities modalities bound schema content-parts or listing
supported_parameters capabilities (string[] for chat/text) bound request schema properties, then listing flags
default_parameters not stored yet JSON Schema default on those properties
pricing (+ overrides) pricing listing / pricing page / extras. Never the schema
tokenizer / instruct_type not stored yet docs, if we care later
benchmarks out of scope third party

Chat/text capabilities use OpenRouter’s parameter names: tools, tool_choice, max_tokens, temperature, top_p, reasoning, include_reasoning, structured_outputs, response_format, stop, frequency_penalty, presence_penalty, seed (and top_k / reasoning_effort when the native schema has them).

Speech / image / video / world-model hosts keep host-native capability objects where those flags do not apply. Schema walk still fills modalities and the request properties that are on that model’s route.

Fill order (per field, never overwrite a stronger rung)

Strongest first. A weaker source only writes when the field (or flag) is still null.

Rung What it is Self-heals?
listing A field on the provider’s own /models row or extras JSON yes, every poll
upstream-spec / generated A property on the bound generation request/response schema (schemaEndpointId → current input schema). Inherits that schema version’s derivation yes, every spec sync
docs-derived Deterministic parse of a provider docs page (markdown table / bullet). Same fail-closed rule as #58 yes, every poll (memoised)
docs-extracted Extraction model over leftover prose, each value carrying the verbatim sentence no

generated specs that are OpenAI’s document with the host swapped (DeepSeek, Cerebras, DashScope, …) do not stamp OpenAI’s tools onto that host’s models. Only a spec the provider publishes (Groq Stainless, Anthropic, Gemini, xAI, BytePlus Ark, FAL per-model, …) counts as upstream-spec / honest generated.

Grain:

  • Model-grained (FAL, Reactor, …): the schema is the model. Walking it is the whole supported_parameters + modalities answer.
  • Provider-grained, one family per route (Anthropic v1/messages, Gemini :predict vs :generateContent): stamp the route’s properties onto every model bound to that route. This is how Anthropic tools / tool_choice / temperature get filled — they are on the messages schema, not the Models API row.
  • Provider-grained, mixed family on one route (OpenAI chat/completions): schema is an upper bound. Activity + schemaEndpointId already keep whisper off chat flags. Per-model listing/docs overlay where a family on the same route does not honor a field (o-series vs gpt).

We store presence, not absence. If the schema has temperature and the model docs never mention it, the flag is present from upstream-spec. A later docs-extracted “this model ignores temperature” is a separate, explicit deny — do not invent denies.

Fine-grained provenance

Not one source on the row. Each stored fact names its winning rung, and capabilities is per flag.

Reuse the schema provenance idea (sourceUrl, sourceHash, fetchedAt, derivation) with extra locators:

interface FactSource {
  derivation:
    | 'listing'
    | 'upstream-spec'
    | 'generated'
    | 'docs-derived'
    | 'docs-extracted'
  sourceUrl?: string
  sourceHash?: string
  fetchedAt?: number
  /** Bound generation route, when the winner is the schema. */
  endpointId?: string
  /** JSON pointer or field name in the source document. */
  path?: string
  /** Verbatim sentence; required for `docs-extracted`. */
  quote?: string
}

interface ModelFactSources {
  contextWindow?: FactSource
  maxOutput?: FactSource
  modalities?: FactSource
  pricing?: FactSource
  /** Per OpenRouter parameter name (or per host-native key). */
  capabilities?: Record<string, FactSource>
}

Store as JSON on models (new column, e.g. fact_sources). The poller / schema-sync writes it next to the values. A flag that moved from docs-derived to upstream-spec is a real model.updated.

Expose

  • GET /v1/models — values only (list stays small). ?provenance=1 includes factSources.
  • GET /v1/models/{provider}/{id} — always includes factSources.
  • Same provenance object shape as GET /v1/schemas/... so agents already know how to read a trust ladder.
  • _links.schema already points at the bound input schema; factSources.capabilities.tools.endpointId should match schemaEndpointId when the winner is the schema.

OpenRouter: compare only

Join native provider/rawId (and undated aliases) to OpenRouter’s id. The join produces one thing:

  1. Discrepancy. After first-party rungs, if OpenRouter has a different value (or has a flag we lack, or we have a flag they lack), record it. Do not change our value.
    • On the detail payload: discrepancies: [{ field, ours, openrouter, oursDerivation, openrouterId }].
    • On the changes feed: model.fact_mismatch (or a dedicated /v1/models/discrepancies) so it is watchable.
    • Typical: we have Anthropic tools from v1/messages and they agree; we lack Grok tools from extras and they have them — that mismatch is the cue to walk xAI’s chat schema.

No fill-nulls rung. A catalog that copied OpenRouter, even labeled, would cap accuracy at theirs and blur what the provider itself states. A catalog that shows the diff uses them as a checker. A null with an OpenRouter discrepancy is the honest answer when the provider publishes nothing.

What #58 already did

Listing/docs fill for openai, anthropic, gemini, grok, groq, jina, fireworks, mistral, moonshot, sambanova, hyperbolic, novita, cohere. Together already has context_length + pricing. OpenRouter is the reference catalog. Gemini schemaEndpointId is the bind the schema walk needs.

Still empty on purpose until this path: DeepSeek / Cerebras / DashScope / Perplexity (bare rows and generated OpenAI specs — schema walk is invalid until they have their own document). Grok capabilities: null is a #58 listing gap; the chat schema should fill tools / temperature / … for language models.

Speech/media (elevenlabs, fal, replicate, reactor, bfl, kling, …): schema walk is the right next fill for request properties and modalities; pricing stays listing/docs.

Implementation slices (each a PR)

  1. fact_sources column + detail API (?provenance=1 on the list). No new facts. Backfill listing-derived Fill contextWindow, maxOutput, modalities, and request-feature capabilities on grain=provider catalogs #58 values as derivation: listing (or docs-derived for OpenAI pages / Grok context table).
  2. Bound-schema walker — given schemaEndpointId + current input (and output) schema, emit supported_parameters + modalities. Skip generated OpenAI-borrowed specs. Merge under the ladder (listing wins on conflict of presence from a per-model field; schema fills nulls).
  3. OpenRouter compare — join, write discrepancies, changes event. No writes to fact columns.
  4. Pricing — listing/docs/overrides; same factSources.pricing.
  5. docs-extracted — leftover prose only (reasoning.mandatory, Gemini modalities, o-series temperature deny, bare-row host docs), quote required.

Fail closed stays: a docs parse of zero rows throws; a schema walk of a missing bind writes nothing (does not null listing facts).

Consumer

TanStack/ai#1308. Ids and activity already come from modelschemas. Facts still come from our OpenRouter catalog until first-party rungs cover the provider being synced. Where modelschemas has a null, the consumer decides whether to fall back. modelschemas never makes that choice for it.

Out of scope

Activity

  1. added a commit that references this issue on Sep 6, 2026
    4292884
  2. changed the title [-]Fill contextWindow, modalities, pricing, and request-feature capabilities on grain=provider catalogs[/-] [+]Fill contextWindow, modalities, pricing, and request-feature capabilities on every catalog[/+] on Sep 10, 2026
  3. changed the title [-]Fill contextWindow, modalities, pricing, and request-feature capabilities on every catalog[/-] [+]Fill OpenRouter-shaped catalog facts from the bound schema, with per-field provenance[/+] on Sep 10, 2026
  4. tombeckenham commented on Sep 14, 2026

    @tombeckenham
    CollaboratorAuthor

    Slice 5 (pricing as an OpenRouter-shaped blob) is superseded. A blob you cannot evaluate is the wrong target — see Openstory #1605 / #1606 rate cards.

    New issues (no parent; attach to a milestone when one exists):

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions