Model Runtime API

A provider-neutral execution data plane for routing, controlling, observing, and metering AI model calls.

简体中文 · Documentation · OpenAPI · v0.1.0

Status: v0.1.0 pre-alpha. The protocol and reference runtime are usable for evaluation; provider adapters are contract-tested with sanitized local fixtures, not certified against live provider accounts. The reference server is unauthenticated and must remain on loopback or behind a trusted gateway.

Why this exists

AI providers expose similar capabilities through different request shapes, stream events, tool-call formats, usage fields, and asynchronous task lifecycles. An application should not need to own every provider difference, but a portability layer should not pretend those differences disappear.

Model Runtime API standardizes the execution lifecycle while keeping provider behavior behind an explicit SPI:

flowchart LR
    caller[Application / Agent] --> gateway[Authenticated AI Gateway]
    gateway --> runtime[Model Runtime API]
    runtime --> route{Capability route}
    route --> p1[Typed SSE provider]
    route --> p2[Content-block provider]
    route --> p3[Async media queue]
    p1 --> events[Ordered runtime events]
    p2 --> events
    p3 --> events
    events --> gateway

Loading

The gateway owns tenant trust. The runtime owns provider execution.

Choose your path

I want to...	Start here
Understand boundaries and packages	Architecture
Integrate an application	Quick start and the TypeScript, Go, or Python client
Configure a public provider adapter	Provider adapters
Discover models or evaluate scenarios	Model discovery and Scenario evaluation
Put an authenticated gateway in front	Gateway integration
Implement another adapter	Adding a provider
Review protocol semantics	Runtime model, Protocol, and OpenAPI
Verify security and release evidence	Threat model and v0.1.0 verification
Prepare reproducible source artifacts	Source release and Delivery checklist

Runtime responsibilities

The runtime owns	The runtime deliberately does not own
Ability and capability-based provider selection	Caller authentication and tenant authorization
Provider-local concurrency, queue admission, retry budget, and fallback	Tenant RPM/TPM, commercial quotas, or budgets
Idempotent submit, status, result, cancellation, and resumable SSE	Wallets, prices, discounts, invoices, or payment
Provider error normalization and route-attempt evidence	Model-quality equivalence across providers
Immutable usage facts and metadata-only telemetry mapping	Prompt/output logging by default

Quick start

Requirements: Node.js 22.12 or later and pnpm 10.

pnpm install
pnpm check
pnpm dev

The default server binds to 127.0.0.1:4320 and loads only the deterministic Mock Provider.

curl -s http://127.0.0.1:4320/v1/runtime

curl -s -X POST http://127.0.0.1:4320/v1/executions \
  -H 'content-type: application/json' \
  -H 'idempotency-key: request-public' \
  -d '{
    "ability":"text-generation",
    "requirements":{"stream":true},
    "input":[{"type":"text","text":"hello"}]
  }'

The create call returns 202 Accepted and an execution ID:

curl -N http://127.0.0.1:4320/v1/executions/EXECUTION_ID/events
curl -s http://127.0.0.1:4320/v1/executions/EXECUTION_ID
curl -s http://127.0.0.1:4320/v1/executions/EXECUTION_ID/result
curl -s -X POST http://127.0.0.1:4320/v1/executions/EXECUTION_ID/cancel

One execution lifecycle

Concept image only. The state machine below is normative.

stateDiagram-v2
    [*] --> accepted
    accepted --> routing
    routing --> running: target selected
    routing --> routing: retryable attempt failed
    running --> routing: retryable failure before output
    running --> succeeded: one terminal completion
    accepted --> cancelled
    routing --> cancelled
    running --> cancelled
    routing --> failed
    running --> failed

Loading

Every event has an execution-scoped monotonic sequence. Consumers reconnect with after or Last-Event-ID; the EventStore materializes text, tool arguments, media results, artifacts, and usage from the append-only event stream.

Automatic fallback stops once text, tool calls, or a result has been exposed. A failed partial stream is terminal; callers must not combine it with another model's answer. fal tasks also suppress automatic retries after acceptance because remote execution may continue despite cancellation.

Provider adapters

Adapter	Public protocol shape	Normalized behavior
OpenAI Responses	Typed response SSE	Text, function calls, usage, terminal completion
Anthropic Messages	Content-block SSE	Text blocks, partial tool JSON, usage, message stop
fal Queue	Submit, status, result, cancel	Queue progress, result object, artifact references
Mock	Deterministic in-process stream	Development, conformance, and CI fixtures

Adapters are disabled unless selected by deployment configuration. Configuration references credential environment variable names, never credential values. Implementation evidence is recorded in the public source registry.

Usage is not billing

The runtime records provider-reported or runtime-derived facts such as input/output/cache/reasoning tokens, media counts or seconds, and tool requests:

usage fact -> price resolution -> customer ledger -> invoice
     ^
     Model Runtime API stops here

Price catalogs, currency conversion, discounts, adjustments, and customer billing remain separate. See ADR-0002.

Workspace packages

Package	Responsibility
@model-runtime/protocol	Types, events, state, generated request schema, and usage vocabulary
@model-runtime/transport	Shared standards-based SSE parser and bounded JSON response handling
@model-runtime/core	Provider registry, routing, flow control, EventStore, and execution runtime
@model-runtime/server	Loopback HTTP/SSE server and safe provider configuration
@model-runtime/conformance	Reusable provider lifecycle assertions
@model-runtime/provider-*	Mock, OpenAI Responses, Anthropic Messages, and fal Queue adapters
@model-runtime/sdk-typescript	TypeScript HTTP/SSE client
@model-runtime/otel	Metadata-only OpenTelemetry GenAI attribute mapping
@model-runtime/catalog	Validated capability snapshots, account-visible model discovery and visibility radar
@model-runtime/evaluation	Bounded scenario assertions, private evidence and exact-profile recommendations
sdk/go, sdk/python	Go and Python runtime clients

Workspace package names remain private until the public API receives enough implementation feedback.

The optional encrypted file store retains results, SSE cursors and idempotency across restart. Interrupted work is marked failed with an unknown remote outcome; recovery does not automatically repeat provider calls. The default memory store has explicit capacity/retention limits. Durable storage requires a private writable volume and a dedicated encryption key supplied by environment-variable name.

Deployment and security

docker build -t model-runtime-api:local .
docker run --rm -p 127.0.0.1:4320:4320 \
  -e MODEL_RUNTIME_HOST=0.0.0.0 model-runtime-api:local

The image runs as non-root and supports a read-only root filesystem. It includes Mock only unless a deployment supplies MODEL_RUNTIME_CONFIG. The Kubernetes example under deploy/ uses fictional image names and keeps the runtime as a sidecar.

All examples are public and fictional. Do not submit credentials, private endpoints, customer data, production logs, private prices, or proprietary routing policy. Run pnpm scan:public before every commit. See Security, Contributing, and the testing guide.

License

Apache License 2.0.