[Marketen]

Why

From 18 Oct 15:33 UTC our Qwen 3.8 27B server answered every request with a 502 in about 0.25 s, for about a day: 193 failed requests, 0 successful, 3 accounts. Every request still paid for a full provider attempt, and the gateway had no way to step around a provider it already knew was down. Models that do have a fallback provider (DeepSeek → Doubleword, MiniMax → Novita) only switched after each request had failed on the primary first.

This is the gateway part of the self-healing work for the Nexus Gateway error-rate goal.

What changes

Circuit breaker, per provider and upstream model (services/breaker.go):

• Opening: after 5 provider failures in a row, the circuit opens for 30 s.
• While open: requests skip that provider. They go straight to the model's fallback provider if it has one; otherwise they fail at once with 503 provider_unavailable, cause=circuit_open and a Retry-After header.
• Recovery: when the pause ends, one request probes the provider. Success closes the circuit; failure doubles the pause, up to 5 min.
• What counts as a failure: outages, timeouts, 5xx and provider_configuration_error (our key or account at the provider). Rejected requests, 4xx answers, rate limits and canceled clients never count.

One retry for quick failures:

• When: a model without a fallback gets a connection error, or a 502/503 the provider returned within 5 s, before any output.
• What happens: the request is retried once after 250 ms.
• Limit: a request now reaches at most two provider attempts: the provider and its fallback, or the provider and one retry.

Observability:

• nexus_gateway_circuit_open{provider,provider_model} (1 while open) and nexus_gateway_circuit_transitions_total{provider,provider_model,state}.
• Retries are counted in the existing nexus_gateway_provider_retries_total{reason="transient_retry"}.
• New log lines: provider circuit opened / provider circuit closed / retrying provider after a quick failure.

Configuration (apps/gateway/selfhealing.go; defaults in parentheses):

• PROVIDER_BREAKER_FAILURES (5; 0 turns the breaker off)
• PROVIDER_BREAKER_OPEN_SECONDS (30)
• PROVIDER_BREAKER_MAX_OPEN_SECONDS (300)
• PROVIDER_RETRY_FAST_FAILURES (true)

Behaviour changes to note

• During a provider outage without a fallback: after the circuit opens, callers get 503 with Retry-After instead of the provider's 502. Clients that honour Retry-After back off by themselves.
• Breaker state lives in memory, per gateway process. A restart closes every circuit; the TEE gateway keeps its own.

Not in this PR

• Showing model availability in /v1/models, so apps can switch models themselves.
• A fallback for Qwen. There is no second host for that model yet, which is an infrastructure and product decision.

Testing

• go vet ./apps/gateway/..., go test ./..., and go test -race on services and pkg.
• New tests:
  • the breaker opens, holds, lets exactly one probe through, reopens with a doubled and capped pause, and closes, all on a fake clock;
  • releasing a probe slot, and the breaker switched off;
  • a quick 502 is retried once and the caller gets the answer;
  • rejected requests, rate limits and canceled clients are neither retried nor counted;
  • a provider that keeps failing gets its circuit opened, after which requests fail fast with 503 and Retry-After without reaching it;
  • with a fallback, once the circuit is open the primary is skipped and every request goes to the fallback.
• TestSystemOne repeats provider failures on purpose to check status passthrough, so it turns the breaker off.

🤖 Generated with Claude Code