[Marketen]

Why

Last week's production errors were hard to explain from the gateway's own output:

• 252 internal_error responses logged no cause. They turned out to be Presidio PII analyzer timeouts (1.5 s, fails closed), but that took correlating latencies by hand, because the dependency and cause metadata was never logged. A later automated diagnosis blamed metering instead, for the same reason.
• Rejected requests (unsupported_model, invalid_field, …) logged no model or account, so "216 calls to a model ID that doesn't exist" could only be recovered from a Prometheus label.
• Nothing tied an error to a gateway release; the only version signal was the Docker image digest in Loki labels.
• nexus_gateway_generations_total{outcome="error"} mixed caller mistakes (bad key, no balance, client cancel) with provider and Nexus failures, so an error rate could not be computed from it.

This is the logging and metrics half of the gateway error classification work: every error should carry enough to point at its fix.

What changes

Classification. domain.FaultOf says who has to act on an error: caller, provider, nexus (our code, config or provider account) or unclear (provider_rejected_request: a bad request or a compatibility gap). It maps by error code; an error can name its fault explicitly where the code points elsewhere (an unregistered provider and a full confidential replay store answer provider_unavailable but are Nexus's). Unknown codes count as nexus.

When the client has already left, a Nexus-side failure that follows (the PII analyzer, auth, routing or credit-hold call being canceled) is reported as client_canceled, not as a Nexus fault.

Logs. request error / request failed, provider request failed, generation failed and provider failed; trying fallback carry fault, error_code, gateway_status and an allowlist of metadata. Values set by gateway code (dependency, cause, statuses) are logged as they are. Values that come from callers, providers or other services (upstream type and code, requested model, account, metering/router codes) are logged only when they look like identifiers, otherwise as [redacted]. Messages, reason and upstream_error are never logged.

Errors that end a request before any provider call carry the account and catalog model, or the requested model ID when it didn't resolve. Internal errors and dependency failures set a fixed cause:

Dependency	Causes
PII analyzer	pii_filter_timeout, pii_filter_failed
Metering	metering_unreachable, metering_error_status, metering_bad_response, credit_hold_lost
Router	router_unreachable, router_error_status, router_bad_response, router_not_configured, router_empty_model
Providers	provider_key_missing, provider_not_registered
Other	replay_store_full, untyped_error

The pii filter error warning now has the request ID. Typesafe's missing-key error now answers clients with the generic internal error, like the other adapters, instead of naming the missing key.

Version. New pkg/buildinfo (Version, Commit), set via -ldflags -X in apps/gateway/Dockerfile. The release workflow passes VERSION and the commit as build args. Every log line gets version, and nexus_gateway_build_info{version,commit} 1 is exported. Local builds report dev.

Metrics.

• New nexus_gateway_errors_total{endpoint,model,provider,phase,fault,error_code,upstream_status}:
  • phase is request (before any provider call, including what the handlers reject: missing key, unreadable or invalid body, confidential envelope checks), provider (provider failed before output) or stream (after output began).
• New nexus_gateway_fallbacks_total{endpoint,model,provider,fallback_provider,fault,error_code}: primary failures that a fallback covered, which otherwise never show up as errors.
• Fix: nexus_gateway_http_requests_total labelled every request route="unmatched". RequestID hands the mux a copy of the request, so r.Pattern never reached the outer metrics middleware. The route now comes from mux.Handler(r).
• Fix: a model name that never resolved (unknown model, failed auth) became the model label on generations_total, so any caller, even unauthenticated, could create unbounded series. It is now unresolved, and resolved requests are labelled with the catalog's model ID.

Not in this PR

• The Nitro enclave build (Dockerfile.enclave) does not set the version yet. That build is reproducible and attested, so it deserves its own change.
• Existing dashboards or queries will see new labels if they filtered:
  • generations_total by an unresolved model name;
  • route="unmatched".

Testing

• go vet ./..., go test ./..., and go test -race ./apps/gateway/... ./pkg/....
• New tests:
  • fault mapping and explicit faults;
  • the log allowlist (prompt-like text is never logged; non-identifier provider values are redacted);
  • PII failure causes (timeout vs. other);
  • a client leaving during PII masking is client_canceled;
  • an unknown model is a caller fault with no caller-supplied metric label;
  • an unregistered provider is a Nexus fault;
  • fallbacks are counted;
  • handler rejections are counted;
  • metering causes;
  • the route label for matched and unmatched requests.
• Built with -ldflags -X …Version=v9.9.9-test and ran it: every log line has "version":"v9.9.9-test" and /metrics exposes nexus_gateway_build_info{commit="abc123",version="v9.9.9-test"} 1.

🤖 Generated with Claude Code