From e99c7084377f754486e69b5effce82ebb1c7e75f Mon Sep 17 00:00:00 2001 From: Vlad Ilyushchenko Date: Thu, 1 Oct 2026 21:48:46 +0100 Subject: [PATCH 01/40] docs(qwp): specify refreshing bearer-token providers for Entra ID MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add a cross-language specification for QWP clients that authenticate with rotating bearer tokens (Microsoft Entra ID managed identities and service principals), plus the Java findings and implementation plan. - design/qwp-token-provider-spec.md (v0.3, decisions resolved): the token-source contract, a proactively refreshing shared token cache, when clients fetch tokens, failure policy by phase (one retry after 401), connect-string keys (token_provider, azure_resource, azure_client_id), connection health, an optional authentication- outage deadline, redaction rules and conformance tests C1-C23. - design/entra-id-qwp-auth.md: findings for the Java client with code references, the design rationale, and the four-step Java plan (§10). Security-relevant server findings are reported privately and are not part of this change. --- design/entra-id-qwp-auth.md | 556 +++++++++++++++++++++++++++++ design/qwp-token-provider-spec.md | 575 ++++++++++++++++++++++++++++++ 2 files changed, 1131 insertions(+) create mode 100644 design/entra-id-qwp-auth.md create mode 100644 design/qwp-token-provider-spec.md diff --git a/design/entra-id-qwp-auth.md b/design/entra-id-qwp-auth.md new file mode 100644 index 000000000..03985fce8 --- /dev/null +++ b/design/entra-id-qwp-auth.md @@ -0,0 +1,556 @@ +# Entra ID app-only auth over QWP: findings and proposed design + +Status: **draft for review**. No production code has been written. This file is not committed. + +**Read these first:** + +- **The spec wins.** The normative, language-neutral contract is now `design/qwp-token-provider-spec.md` (v0.3, all decisions resolved). Where the two documents differ, follow the spec. That includes the refresh-schedule clamps, backoff jitter, registry linger, and the Java names in its Appendix B, which supersede §7.2 here. +- **Java plan:** §10. +- **Line references** are as of `c9968f24` and may drift. + +Code inspected: + +- **Client:** `java-questdb-client` `main` @ `c9968f24`. +- **Server:** QuestDB Enterprise at the time of writing (private source, so this doc describes server behaviour only), and the open-source `questdb` core @ `c06b4c8267`. +- **Rust core** (what the Python client wraps): `c-questdb-client` `main` @ `5ea51a47`. +- **Spec:** `documentation` @ `7c35c02`, files `qwp-ingress-websocket.md` and `qwp-egress-websocket.md`. The spec pins client `67bb5e4` (2026-05-13). + +Abbreviations for client files, all under `core/src/main/java/io/questdb/client/`: + +| Abbrev. | File | +|---|---| +| `S` | `Sender.java` | +| `QWS` | `cutlass/qwp/client/QwpWebSocketSender.java` | +| `CWSL` | `cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoop.java` | +| `BD` | `cutlass/qwp/client/sf/cursor/BackgroundDrainer.java` | +| `QQC` | `cutlass/qwp/client/QwpQueryClient.java` | +| `WSC` | `cutlass/http/client/WebSocketClient.java` | + +## TL;DR + +1. **The token-provider hook already exists and is wired into every QWP upgrade path.** + - The hook is `HttpTokenProvider`, added in #52. + - With a provider, Java pulls a fresh token on every connect round: the initial connect, reconnect, failover, orphan drainers, pool recovery, egress connect and egress failover. + - Only a static `token=` / `httpToken()` is captured once. + - So the work is not plumbing. What's missing is: + - an expiry-aware, shared, proactively refreshing cache; + - classifying provider failures at startup; + - a "token rejected" signal from the client back to the provider; + - a way to select a provider from a connect string; + - spec updates and Rust/Python parity. +2. **The brief's premise ("401 is terminal") no longer matches Java, but it does match the Rust core.** + - Since #66 and #52, a Java foreground sender that has connected once retries 401/403 forever while store-and-forward (SF) buffers. + - 401/403 is terminal only during initialization, for orphan drainers (with a bounded ride-out for rotating credentials), and on egress. + - The Rust core treats `AuthError` as terminal on reconnect and sends a static header. That is the Python customer's likely client, and it has exactly the failure described in the brief. + - The public spec is stale on both points. +3. **The server works for app-only Entra tokens, but only in local-JWT mode.** Required settings: + - `acl.oidc.groups.encoded.in.token=true`, `acl.oidc.groups.claim=roles`, `acl.oidc.sub.claim=oid`; + - QuestDB app registration set to issue v2 tokens. + + The default UserInfo mode cannot work for app-only tokens. A QuestDB 401 also covers "IdP or JWKS unreachable", so it is not proof that the token is bad. +4. **Recommendation:** do (a) now (core cache and SPI) plus (c), a thin optional `questdb-client-azure` module. That module adapts `TokenCredential` and registers a connect-string provider through `ServiceLoader`. Defer (b), the built-in IMDS provider. + +## 1. Current auth configuration + +**Ingress (`Sender`)** + +- **Schema and TLS:** + - `ws::` or `wss::` (`S:3540-3549`); `wss` enables TLS. + - TLS keys: `tls_verify=on|unsafe_off`, `tls_roots` (PEM or JKS), `tls_roots_password`. All are rejected on `ws::` (`S:4245-4250`). + - Builder equivalents: `enableTls()` (`S:2124`), and `advancedTls().customTrustStore(..)` / `.disableCertificateValidation()` (`S:4572-4624`). +- **Auth keys:** + - Keys are `username`/`password` (aliases `user`/`pass`) and `token`, registered in `impl/ConfigSchema.java:49-59`. + - Unknown keys are rejected (`impl/ConfigView.java:74-86`). + - The keys are applied in `fromConfigWebSocket` (`S:4029-4036`). + - Cross-key rules in `validateWsConfig` (`S:4227-4255`): Basic auth needs both halves, and `token` is exclusive with Basic. +- **Builder:** + - `httpToken` (`S:2284`), `httpUsernamePassword` (`S:2372`), `httpTokenProvider` (`S:2343`); all three are mutually exclusive. + - `auth_timeout_ms` bounds the upgrade (default 15 s). `connect_timeout` bounds the TCP connect. +- **Where the header is built:** `buildWebSocketAuthHeader` (`S:3427-3460`). + - Basic and static Bearer values go through `QwpWebSocketSender.fixedAuthHeader`, which tags them as constant. + - A provider becomes a lambda that calls `getToken()`, snapshots it, runs `HttpTokenProvider.validateToken`, and returns `"Bearer " + token`. + - That lambda is passed as `Supplier` to `QwpWebSocketSender.connectWithCredentialSupplier` (`S:1468`, `S:1694`). +- **On the wire:** + - `WSC.upgrade(path, timeout, authorizationHeader)` writes `Authorization: ` (`WSC:640-722`, header at `:700-704`). + - The ingress path is `/write/v4` (`QWS:148`). The client never uses `/api/v4/write`. + +**Egress (`QwpQueryClient`)** + +- Same keys, read in `fromConfig` (`QQC:375-490`, auth at `:417-419` and `:485-486`). +- Programmatic API: `withBasicAuth`, `withBearerToken`, `withBearerTokenProvider` (`QQC:1127`, `:1147`, `:1177`; mutually exclusive), plus `withTls()` / `withTrustStore()` / `withInsecureTls()`. +- The header is built in `resolveAuthorizationHeader()` (`QQC:1933-1960`). +- The upgrade to `/read/v1` happens in `runUpgradeWithTimeout` (`QQC:1971-1996`). + +**`QuestDB` facade** + +- `QuestDBBuilder.httpTokenProvider` (`QuestDBBuilder.java:330`) rejects a provider combined with `token`, `username` or `password` in the config (`:190-194`). There is also `QuestDB.connect(cs, provider)` (`QuestDB.java:105`). +- One provider instance is shared by every pooled sender (`SenderPool.java:540-544`, `:2046-2051`) and every pooled query client (`QueryClientPool.java:599-600`). + +## 2. Captured once, or re-read per upgrade? + +| Path | Code | Credential pull | +|---|---|---| +| Static `token=` / `httpToken` / Basic | `S:3431-3439`, `FixedAuthHeader` `QWS:5328` | **Captured once.** An expired token is presented forever. | +| Provider: ingress initial connect (OFF/SYNC) | `ensureConnected` `QWS:3999-4055` → `buildAndConnect` → `connectWalk` | Once per round, before the endpoint walk (`QWS:3305-3351`), on the thread calling `build()`. | +| ASYNC initial connect, and every reconnect | `CWSL.connectLoop` (`:1731`) → `reconnectFactory.reconnect(cancellation)` | Once per round, on the cursor I/O thread. | +| Failover to another host | Inside the same round (`QWS:3352-3394`) | **The same token is reused for every endpoint in the round**; the next round re-pulls. This is deliberate: a token is cluster-wide. | +| SF replay after reconnect | Replay starts on the new socket after the upgrade | No extra auth; covered by the reconnect pull. | +| Orphan drainers (SF recovery of other slots) | Background `ReconnectSupplier`, `QWS:2860`; same supplier | Once per round, on a drainer-pool thread. | +| `SenderPool` startup-recovery delegates | `SenderPool.java:2079-2081` | OFF-mode build with the provider. | +| Egress connect | `QQC:754-786` | Once per `connect()` walk. | +| Egress mid-query failover | `QQC:1576-1683` → `reconnectViaTracker` `:1865-1931` | Once per failover reconnect. | + +These behaviours are already pinned by tests: + +- `WebSocketTokenProviderTest`: `testProviderRequeriedOnEveryReconnect`, `testThrowingProviderResolvedOncePerConnectRound`, and others. +- `QwpQueryClientTokenProviderTest.testProviderTokenReResolvedOnFailoverReconnect`. +- `SenderPoolSfTokenProviderTest`. + +## 3. Failure classification + +**Where failures are classified** + +- `QwpUpgradeFailures.classify` (`:41-57`): + - 421 with `X-QuestDB-Role` → role reject (transient); + - 401 or 403 → `QwpAuthFailedException`; + - anything else passes through as `WebSocketUpgradeException` or `HttpClientException`. +- A provider that throws is wrapped as `QwpCredentialUnavailableException` (`QWS:3334-3344`). +- The policy applied to each class is decided per phase: + +| Phase | 401/403 | Other non-421 upgrade reject (404/426/5xx) | Provider threw | +|---|---|---|---| +| OFF initial connect (the default) | thrown from `build()` | thrown: latched as `terminalUpgradeError` (`QWS:3442-3447`), thrown at round end (`:3593-3596`) | the provider's own exception is thrown (`QWS:4040-4054`) | +| SYNC initial connect (`initial_connect_retry=on`, or any `reconnect_*` key set) | terminal, no retry (`CWSL:1041-1052`) | terminal, no retry | **fails fast** (`CWSL:1053-1065`); pinned by `testThrowingProviderFailsFastInSyncInitialConnect` | +| ASYNC, before the first connect | terminal: goes to the error inbox and is rethrown on `close()` (`CWSL:1825-1867` via `endpointPolicyFailureIsTerminal`, `:2111`) | terminal | retried forever (`CWSL:1928-1963`) | +| Foreground sender after its first connect | **retried forever** with capped backoff; a RETRIABLE `SECURITY_ERROR` is dispatched per attempt (`CWSL:1876-1886`) | retried forever | retried forever | +| Orphan drainer | constant credential: slot quarantined (`.failed`). Rotating credential: ride-out of ≥6 attempts **and** ≥ min(`reconnect_max_duration`, 5 min), capped at 256 attempts (`BD:97-160`, `:481-556`) | quarantined | retried forever | +| Egress connect / failover | thrown from `connect()` (`QQC:785`); during failover, `onError("auth failure during failover reconnect")` (`:1665-1676`) | next endpoint; error if every endpoint fails | `LineSenderException`; during failover, `onError("failover reconnect failed")` | + +**Where the brief, the spec and the code disagree** + +- **Spec vs Java.** The spec (`qwp-ingress-websocket.md:1055-1063`) says "401/403 terminal at any host" and "all other upgrade errors transient (404, 426, 503…)". It was accurate at `67bb5e4`: the reconnect loop issued `HALT` on 401/403 (verified). #66 (`37d4b0ab`) and #52 (`0b9b5766`) changed that to the table above. Also, non-421 4xx/5xx is **terminal** during initialization, contrary to the spec. For example, a single node answering 503 fails a SYNC `build()` immediately. +- **Rust core (Python).** + - The header is a static `auth_header: Option` (`questdb-rs/src/ingress.rs:497`, `:599`, `:664`). + - `reconnect_error_is_terminal` treats `AuthError` as terminal on reconnect (`questdb-rs/src/ingress/sender/qwp_ws_driver.rs:2596-2603`). + - So the brief's failure mode is real there. +- **Egress pool (suspected bug; not yet reproduced).** After a failed failover reconnect, the client is left with `connected=false` (`QQC:1623`). The worker still returns to the pool, and every later `execute()` throws `"QwpQueryClient not connected"` (`QQC:1581`). `reapIdle` never removes a worker while the pool is at `query_pool_min` (`QueryClientPool.java:500`). A 401 during failover is one way in. This needs a red test before anything else. + +## 4. Threading and the initial-connect budget + +**Which thread connects** + +- **OFF/SYNC initial connect:** the thread calling `Sender.build()` or `QuestDB.build()` (pool pre-warm). Pool growth runs on the acquiring thread; recovery delegates run on the housekeeper thread. None of these paths has a `ConnectCancellation`. +- **ASYNC initial connect and every reconnect:** the sender's cursor I/O thread. +- **Orphan drainers:** `BackgroundDrainerPool` threads. +- **Egress:** the caller of `connect()`; failover runs on the thread executing `execute()` (in the pool, the `QueryWorker` dispatch thread). + +**What can block within one attempt** + +- **`getToken()`:** this is caller code, and nothing in the client bounds it. `close()` interrupts it, but only on I/O-thread and drainer paths (`CWSL` `ConnectCancellation.cancel`, near `:3764-3790`). +- **DNS:** bounded only by the OS. +- **TCP connect:** bounded by `connect_timeout`. The foreground default is 0, which means the OS SYN timeout of 60-130 s. Background connects default to 15 s (`QWS:3139-3141`). +- **TLS handshake:** bounded by `connect_timeout`, or by the request timeout when that is unset. +- **Upgrade:** bounded by `auth_timeout_ms`. +- Producers are not blocked by any of this until SF fills and append backpressure kicks in. + +**How the budget interacts with a slow token fetch** + +- `connectWithRetry` checks `reconnect_max_duration_millis` only *between* attempts (`CWSL:1029`). +- A slow `getToken()` therefore stretches `build()` past the budget: an attempt is never cut short. +- A throwing `getToken()` ends SYNC immediately. +- ASYNC and reconnect have no budget (Invariant B). +- **Entra consequence:** + - The first `DefaultAzureCredential` call runs unbounded on the `build()` thread. That call includes chain probing, an IMDS cold start, and the SDK's own retries on 429. + - An IMDS 429 that escapes the SDK fails OFF and SYNC startup. + +## 5. Constraints from repo conventions + +- **Dependencies:** + - The core's only runtime dependency is `slf4j-api` (annotations are `provided`). + - Java 8 is the API floor, and the code must also compile on JDK 11+. + - So no `azure-core` or `azure-identity` in core. `ServiceLoader`, `CompletableFuture` and `ScheduledExecutorService` are all Java 8. +- **Allocation:** + - Zero-GC is mandatory on per-row producer calls and the steady-state I/O loop (`.pi/skills/review-pr/SKILL.md:37`, `:583-590`). + - Connect paths already allocate: a `WebSocketClient` per attempt, and `"Bearer " + token`. + - A cache hit only needs to be cheap: no I/O and no locks. +- **Config flow:** + - Ingress: string → `ConfigString` → `ConfigView` (strict `ConfigSchema` registry) → `fromConfigWebSocket` setters → `build()` → `connectWithCredentialSupplier(..., Supplier, ...)`. + - Egress: `QwpQueryClient.fromConfig` → `with*` setters. + - The facade validates both sides (`QuestDBBuilder.java:184-199`). + - New keys must go into `ConfigSchema` and are shared vocabulary with every client. +- **Test harnesses:** + - `TestWebSocketServer`: captures `Authorization`, supports `setRejectWithStatus` and `setRejectWithRole` and drop handlers, and caps requests at 8 KB. + - `MockOidcServer`: a raw-socket HTTP mock that can return JSON, chunked, dropped or dribbled responses. + - `assertMemoryLeak`, and `HandOffCharSequence` for TOCTOU tests. + - Scripted `ReconnectFactory` stubs, as in `BackgroundDrainerMidDrainAuthRejectTest`. +- **API compatibility:** `ExportedApiCompatibilityTest` pins exported signatures. Adding is fine; retyping is not. + +## 6. Problems with ~2 KB rotating bearer tokens + +**Size limits** + +- **Client buffers:** fine. The upgrade request buffer starts at ≥64 KiB and grows (`WSC:181-186`, `WebSocketSendBuffer.java:504-515`). The response buffer is ≥64 KiB. +- **Server header buffer:** + - Production default is 64,448 B (`PropServerConfiguration.java:1213`; the code says `32 * 2014`, a harmless typo for 1024). + - `DefaultHttpContextConfiguration`, used by embedded and test servers, allows only 4,096 B. A 2 KB token fits; a role-heavy token may not. + - `TestWebSocketServer` drops any request over 8,192 B. +- **Connect string:** no length limit. A `;` must be doubled, and control characters are rejected (`ConfStringParser.java:164-183`). JWTs are safe. + +**Rotation** + +- One token serves the whole round. +- With the foreground `connect_timeout=0`, a round across several black-holed hosts can run for minutes, so a near-expiry token can expire mid-round. +- The cache's freshness margin and the 401 retry (§7.7) cover this. + +**Where a token could leak** + +- **Already safe:** + - `validateToken` never echoes the token (`HttpTokenProvider.java:70-80`). + - `WebSocketUpgradeException` carries only the status line (`WSC:1346-1357`). + - `QwpAuthFailedException` carries status, host and port only. + - The header is never logged. +- **Watch:** + - `QwpCredentialUnavailableException` uses the *provider's* exception message as its own. That message flows into `SenderError` (`CWSL:1954`) and into logs (`CWSL:1957`, `BD:720`), so provider error text must be token-free. + - `HttpClient.Request.toString()` dumps the raw request, headers and body included (`HttpClient.java:580-586`). An HTTP-based token source must never log or wrap a `Request`: a client-credentials body carries `client_secret`. + - Test-only config snapshots expose the token (`S:4330`, `QQC:972`). + - The upgrade bytes stay in the native send buffer after the handshake until frames overwrite them. This is low risk. +- **Server-side:** server findings that touch security are reported privately (see `SECURITY.md`). + +## 7. Design + +### 7.1 Shape + +``` +TokenSource fetchToken() -> ExpiringToken(token, expiresAt[, refreshAt]) + ▲ Azure TokenCredential adapter (module), future IMDS, anything else +RefreshingTokenProvider shared cache: background refresh, single-flight, jitter, backoff + ▲ implements HttpTokenProvider +existing plumbing Sender / QwpQueryClient / QuestDB pools: call sites unchanged +``` + +Keep `HttpTokenProvider` as the single integration point; it is already in every path in §2. Add an expiry-aware source SPI behind it. The cache belongs in **core**: it is generic (OIDC client credentials, Vault, …), and it is the part every client must mirror. + +### 7.2 API (core, Java 8) + +```java +// io.questdb.client.cutlass.auth (exported) +public final class ExpiringToken { // name avoids clashing with azure-core AccessToken + public ExpiringToken(String token, long expiresAtEpochMillis); // validates printable ASCII + public ExpiringToken(String token, long expiresAtEpochMillis, long refreshAtEpochMillis); // 0 = derive + // toString(): "ExpiringToken{, expiresAt=...}" +} + +@FunctionalInterface +public interface TokenSource { + // Only ever called on the provider's refresher thread, one call at a time. May block. + // Throw TokenUnavailableException to classify; any other RuntimeException counts as retryable. + ExpiringToken fetchToken(); +} + +public final class RefreshingTokenProvider implements HttpTokenProvider, QuietCloseable { + public static Builder builder(TokenSource source); // margins, coldWaitMillis, backoff, clock/scheduler seams + public CharSequence getToken(); // warm: volatile read, no I/O; cold: bounded wait + public void onTokenRejected(CharSequence token, int httpStatus); // forced refresh, rate-limited + public boolean awaitFirstToken(long timeoutMillis); // optional startup readiness gate + public void close(); // stops the refresher thread +} + +// io.questdb.client (exported) +public class TokenUnavailableException extends LineSenderException { + public boolean isRetryable(); // IdP unreachable / timeout / 429 / 5xx -> true + public long getRetryAfterMillis(); // -1 if none +} + +// HttpTokenProvider (existing): one default method added; still a @FunctionalInterface +default void onTokenRejected(CharSequence token, int httpStatus) { } +``` + +Under option (a), the Azure adapter is user code (or a README snippet): + +```java +TokenCredential cred = new DefaultAzureCredentialBuilder().build(); +TokenRequestContext ctx = new TokenRequestContext().addScopes("api:///.default"); +RefreshingTokenProvider tokens = RefreshingTokenProvider.builder(() -> { + com.azure.core.credential.AccessToken t = cred.getTokenSync(ctx); + return new ExpiringToken(t.getToken(), t.getExpiresAt().toInstant().toEpochMilli()); +}).build(); +QuestDB db = QuestDB.builder().fromConfig("wss::addr=qdb1:9000,qdb2:9000;").httpTokenProvider(tokens).build(); +``` + +### 7.3 Connect-string surface + +``` +wss::addr=qdb1:9000,qdb2:9000;token_provider=azure;azure_resource=api://;azure_client_id=; +``` + +- **New keys,** reserved in the spec for all clients and registered as `COMMON` in `ConfigSchema`: + - `token_provider=`; + - `azure_resource`, from which the scope `/.default` is derived; + - optional `azure_client_id`, for a user-assigned managed identity or a workload-identity client ID. +- **Safe to log:** none of these values is secret, so they can live in `QDB_CLIENT_CONF`. +- **Exclusivity:** mutually exclusive with `token`, `username`, `password`, and a programmatic `httpTokenProvider`. +- **Resolution:** + - Core defines a small `TokenProviderFactory` SPI, looked up by name through `ServiceLoader` (with a `uses` clause in `module-info`). + - Core ships no Azure code. `token_provider=azure` without the module fails at parse time: `"token_provider=azure requires questdb-client-azure on the class path"`. +- **Sharing:** + - A process-wide registry keyed by provider name plus canonical parameters hands out one ref-counted `RefreshingTokenProvider`. + - Every `Sender`, `QwpQueryClient` and `QuestDB` built from equivalent strings therefore shares one cache, so a process makes at most one IMDS/Entra call at a time. + - The last `close()` stops the refresher. + +### 7.4 Options + +| | (a) Hook and cache in core | (b) Built-in IMDS in core | (c) `questdb-client-azure` module | +|---|---|---|---| +| Runtime dependencies | none | none | `azure-core` (the user brings `azure-identity`) | +| Azure hosts covered | all, through the user's SDK | VM/VMSS IMDS only. Not App Service/Functions (`IDENTITY_ENDPOINT`), AKS workload identity (federated token exchange), or Arc | all, through `DefaultAzureCredential` | +| Service principal with client credentials | yes | no | yes | +| Connect-string-only deployments (e.g. Kafka connector) | no | yes | yes, through the SPI | +| Maintenance | small | we own an Azure protocol: retry rules for 404/410/429, `expires_on` formats, identity selectors | tracks the `azure-core` API; another artifact to build on JDK 8 and 11 | + +**Recommendation:** + +- Ship (a) and (c) together. +- Add (b) only if a zero-dependency, connect-string-only VM deployment appears. It would plug into the same cache as `token_provider=azure_imds`, with an endpoint override for tests. + +### 7.5 Where it plugs in (changes only) + +| Path | Change | +|---|---| +| All ingress and egress pulls | None to the call sites. A warm cache makes the pull a memory read. `buildWebSocketAuthHeader` returns a small named credential class (dynamic, with `onRejected`) instead of a lambda. | +| `connectWalk` 401 branch (`QWS:3427`) | If the credential is dynamic and the status is 401: call `onTokenRejected`, re-pull, and retry the **same endpoint once** if the token changed. No host-health penalty for the first 401. `AUTH_FAILED` fires only on the final failure. | +| SYNC `connectWithRetry` credential branch (`CWSL:1053`) | A retryable `TokenUnavailableException` becomes transient within the budget. Anything else still fails fast, so OIDC device flow behaviour is unchanged. | +| Foreground reconnect and orphan drainer | Nothing extra. Each attempt re-pulls, and `onTokenRejected` already fired in the walk, so the rotating-401 ride-out now actually receives new tokens. | +| Egress `connect()` / `reconnectViaTracker()` (`QQC:785`, `:1889`) | The same 401 retry-once. | +| Egress pool | Separate fix: do not return a worker whose failover reconnect failed. Red test first (§3). | +| `ConfigSchema`, `fromConfig` paths, `QuestDBBuilder` | Register the keys; resolve them through the registry; add mutual-exclusion checks. | + +Keep "one pull per round" rather than one per endpoint. It is deliberate (`QWS:3310-3320`), it keeps provider failures cluster-wide, and the cache margin plus the 401 retry cover long rounds. + +### 7.6 Caching and refresh + +- **Single-flight.** One daemon refresher thread per provider is the only caller of `fetchToken()`, so each process makes at most one concurrent call per source. This is what keeps it within the IMDS limits of 5 concurrent requests and 20 requests per second. +- **Schedule.** Computed at fetch time: + - `refreshAt = fetchedAt + (expiresAt - fetchedAt) × U(0.45, 0.55)`, clamped to the range `[fetchedAt + 30 s, expiresAt - 5 min]`. + - A source hint wins if it is earlier (newer azure-core and MSAL expose a refresh-at time). + - A service-principal token (60-90 min) is refreshed about every 30-45 min. + - For a managed-identity token (~24 h), IMDS may keep returning the platform-cached token. The schedule then halves the remaining time on each fetch, which is a handful of calls per token lifetime. +- **Hand-out.** + - A token with at least 60 s left is returned from memory: no I/O, no lock. + - With no usable token, `getToken()` asks for a fetch and waits up to `coldWait` (default 30 s). The wait is interruptible, so `close()`'s interrupt still works. If it times out, it throws a retryable `TokenUnavailableException`. + - An expired token is never handed out. +- **Failures.** + - Retry with exponential backoff and full jitter, from 0.5 s up to a 60 s cap, honouring `retryAfterMillis`. + - Keep serving the current token while it is above the hand-out floor. + - Log at most one WARN per minute, without the token. Expose `lastFailure()`. +- **`onTokenRejected`.** + - Applies only if the rejected token is still the current one, and at most once per 30 s. + - It triggers an immediate fetch; the walk waits up to 5 s for a *different* token. + - If the result is the same token (the managed-identity platform cache), it is accepted, and the walk falls back to the phase policy. + - Don't oversell this: for managed identity, proactive refresh with a margin is the real fix. The forced refresh mainly helps client credentials and clock skew. +- **Startup.** The provider starts its first fetch at construction. `awaitFirstToken` lets an application gate `build()` on it. `lazy_connect`/ASYNC never blocks on it. +- **Clocks.** Wall clock for expiry, `nanoTime` for scheduling; validity is re-checked at hand-out. + +### 7.7 Error classification and the 401 policy + +- **Provider failures:** + - **Transient** (retryable): unreachable, timeout, 429, 5xx, IMDS 404/410. + - **Permanent:** identity not found, malformed configuration. + - **Behaviour by phase:** + - OFF fails fast after `coldWait`, which already absorbs short IMDS 429 storms. + - SYNC retries transient failures within the budget and fails fast on permanent ones. + - ASYNC, the established foreground sender and the drainers retry forever, as today. +- **401 with a refreshable credential:** one immediate retry with a *different* token, in every phase. After that, the existing phase policy applies. +- **403:** no forced refresh. 403 is authorization: an app-role or alias change, and for managed identity a new token may not appear for up to 24 h. The existing phase policy applies. +- **Spec impact (needs a cross-client decision):** + 1. Replace "401/403 terminal at any host" with the phase table in §3. Recommendation: adopt Java's policy. A QuestDB 401 can be an IdP outage, and SF promises Invariant B. The other option is to revert Java, which would recreate the brief's problem for every SF user. + 2. Add a "credential sources" section: + - pull per upgrade round; + - reuse within a round; + - proactive refresh, a SHOULD; + - never log tokens, a MUST; + - printable-ASCII validation, a MUST. + 3. Add the 401 rule: a client SHOULD invalidate and retry the same endpoint once if a different credential is obtained. The retry counts toward neither backoff nor host health. + 4. Define the retryable/permanent split for credential-acquisition failures, including the SYNC change. + 5. Reserve `token_provider`, `azure_resource` and `azure_client_id`. + 6. Separately: fix the "all other upgrade errors are transient" text against the code, or fix the code (§3). + +### 7.8 Redaction + +- **Rules for the new code:** + - **`toString()`:** never prints token text. This applies to `ExpiringToken`, the provider and registry entries; they show length, expiry, and at most an 8-hex-digit SHA-256 prefix for correlating rotations. + - **Response bodies:** a source never logs or embeds one, because they contain `access_token`. On a parse failure it reports the field or shape only. + - **Error text from the IdP or SDK:** passed through `DisplaySafe` filtering and capped at about 256 chars. + - **Exceptions:** never carry an HTTP `Request` as message or cause. +- **Optional hardening:** + - Zero the upgrade bytes in the send buffer after sending. It is cheap: under 4 KB. + - Refuse `token_provider=*` on `ws::` (cleartext bearer), and WARN once for a programmatic provider on `ws::`. To decide in §9. + +### 7.9 Language-neutral contract (for Python and Rust) + +- **Source:** returns `(token, expires_at)`, plus an optional `refresh_at`. + - Python: `lambda: cred.get_token(scope)` already returns `AccessToken(token, expires_on)`. + - Rust: `Fn() -> Result<(String, SystemTime), TokenError>`. + - C: a callback with out-parameters. +- **Cache:** the same semantics and defaults in every client, with one cache per source instance and the same registry for connect-string providers. +- **Python:** keeping the cache in the Rust core means the Python callable runs only on the refresher thread, rarely. That keeps GIL hand-offs off the I/O threads. +- **Rust core work:** + - build the header per upgrade attempt instead of `auth_header: String`; + - add the callback and classification; + - decide the reconnect `AuthError` policy (see §7.7, item 1). + +### 7.10 Test plan + +**1. Unit tests for `RefreshingTokenProvider`.** Use a fake clock and scheduler and a scripted source. Cover: +- the schedule: the halving formula, the clamps, jitter bounds, hint precedence, and managed-identity convergence when the source keeps returning the same token; +- single-flight: 64 threads cold-starting at once produce one fetch; +- backoff and `Retry-After`; +- serving the current token until the floor; +- cold failure: a retryable exception is thrown within `coldWait`; +- an interrupt during the wait: the exception is thrown and the interrupt flag is preserved; +- `onTokenRejected`: rate limit, same-token result, and ignoring a stale token; +- that `close()` leaves no thread behind. + +**2. Stub server that rejects expired tokens.** Extend `TestWebSocketServer` with `setAuthorizationValidator(Function)`. Tokens look like `T.`, and the server shares the test's fake clock, returning 401 once a token has expired. + +- **Proactive refresh on reconnect:** advance the clock past T1's expiry and drop the socket. The reconnect must carry T2 with **no** 401 seen. +- **Stalled refresher:** stall the refresher so the reconnect still carries T1. Expect exactly one 401, then an immediate retry with T2 (no backoff gap), and the batch lands. +- **Initial-connect matrix:** OFF, SYNC and ASYNC × a cold provider failure that is retryable or permanent. SYNC must retry the retryable case within the budget. `testThrowingProviderFailsFastInSyncInitialConnect` must stay green. + +**3. Failover and SF replay across a rotation.** Use two validating servers, A and B, with `addr=A,B` and `sf_dir` set. + +- **Failover:** connect to A with T1, rotate, then kill A. B must receive T2, and the unacked frames are replayed exactly once (assert rows and FSNs at B). Add a variant where the refresher is stalled, so there is one 401, then the retry. +- **SF replay:** the server rejects T1 with 401 until the rotation while the producer keeps writing. Every row must arrive, with no `.failed` sentinel, no `DATA_LOSS`, and no terminal. +- **Orphan drainer:** restart with an orphan slot and `drain_orphans=on`. The drainer drains with the rotated token. + +**4. Facade and egress.** +- A `QuestDB` handle with 4 senders and 2 query clients sharing one provider. Force a reconnect storm: `fetchToken` runs once, and every upgrade carries the same fresh token. +- Egress: failover in the middle of a query with a rotated token, and the 401 retry-once on `connect()`. +- A red test for the dead pooled egress worker (§3). + +**5. Connect string.** Cover: +- parse and validation errors; +- mutual exclusion; +- the "module missing" message; +- registry sharing across `fromConfig` instances, and ref-count release; +- rejection on `ws::`, if adopted. + +**6. Redaction.** Use a sentinel token and capture every logger with a logback `ListAppender`. Run the 401, provider-failure and malformed-response scenarios, then assert the sentinel appears nowhere in logs, exception messages, `SenderError` messages or `toString()` output. + +**7. Module (c).** Test against a fake `TokenCredential`, with no network: the expiry/refresh-at mapping, and mapping exceptions to retryable or permanent. Add a manual smoke run on an Azure VM with a managed identity; this is not for CI. + +**8. Fake IMDS (only if we do (b)).** Build it on `MockOidcServer`. Assert `Metadata: true`, `api-version`, `resource` and `client_id`. Script 200, 400, 404, 410, 429 with `Retry-After`, 500, malformed JSON, and both `expires_on` formats. Error messages must never echo the response body. + +**9. General.** Run everything under `assertMemoryLeak`. Tests run on JDK 8; also compile on JDK 25. + +**10. Server e2e (Enterprise, separate PR).** Extend the Enterprise OIDC tests with locally signed tokens shaped like Entra app-only v1 and v2 tokens. Assert: +- v2 with `sub.claim=oid` and `groups.claim=roles` is accepted; +- v1 with `aud=api://…` is rejected under the default audience; +- no roles gives 401; +- an expired token gives 401, and a rotated token then succeeds. + +## 8. Server side (QuestDB Enterprise) + +This section describes server behaviour only. Enterprise source is private, and security-relevant server findings are reported privately (see `SECURITY.md`). + +**Local JWKS validation or UserInfo?** Both exist. The switch is `acl.oidc.groups.encoded.in.token`; its default, `false`, means UserInfo. + +- **UserInfo cannot work for app-only tokens.** Entra's UserInfo is a Microsoft Graph endpoint: it accepts only Graph-audience user tokens. +- **With `true`, the token is validated locally.** The server looks up the token's `kid` in the identity provider's signing keys (JWKS), then: + - verifies the signature; + - checks `aud` against one exact string, `acl.oidc.audience`, which defaults to `acl.oidc.client.id`; + - checks `exp`, with 60 s of leeway; + - requires a non-empty `sub`. +- **Key rollover:** an unknown `kid` makes the server reload its keys. +- **Cache:** verified tokens are cached for 30 s (`acl.oidc.cache.ttl`). + +**Can `acl.oidc.sub.claim` point at `oid` or `azp`?** Yes. + +- Any top-level claim name works. +- `oid` exists in both v1 and v2 tokens; `azp` exists only in v2 (v1 has `appid`). **Use `oid`.** +- The server also requires a **non-empty groups claim**; without one it returns 401. +- Caveat: there is a single `sub.claim` for every login. A deployment that also has human users (UPN) and apps (no UPN) needs `oid`. + +**Does `acl.oidc.groups.claim=roles` work with `EXTERNAL ALIAS`?** Yes. + +- The claim can be an array or a single string. +- Its values are matched against `EXTERNAL ALIAS`. Role values containing dots need quotes: `CREATE GROUP ingest WITH EXTERNAL ALIAS 'QuestDB.Ingest'`. +- The service principal or managed identity must hold at least one app role. +- Authorization failures come back as 403 at the upgrade. Established connections keep their old authorization. + +**How are v1 and v2 tokens handled?** + +- **The practical pitfall is `aud`.** A v1 token's `aud` is the resource string as requested (`api://`); a v2 token's `aud` is the client-ID GUID. +- **Fix:** set `requestedAccessTokenVersion: 2` (manifest `accessTokenAcceptedVersion`) on the QuestDB app registration. Every client (IMDS `resource=` or an SDK `.default` scope) then gets `aud=`, which matches the default. +- **To verify against the tenant:** that the `/discovery/keys` and `/discovery/v2.0/keys` endpoints serve the same signing keys. + +**Working configuration** + +``` +acl.oidc.enabled=true +acl.oidc.configuration.url=https://login.microsoftonline.com//v2.0/.well-known/openid-configuration +acl.oidc.client.id= # also the audience +acl.oidc.groups.encoded.in.token=true # local JWKS validation; required for app-only +acl.oidc.groups.claim=roles +acl.oidc.sub.claim=oid +``` + +**Server follow-ups** + +- **Client-facing:** spec Appendix C proposes two changes: + - send a `WWW-Authenticate: Bearer` challenge on 401; + - return 503, not 401, when the server cannot verify tokens. +- **Also useful:** a fallback list for `sub.claim`, and 403 instead of 401 when the roles claim is missing. +- **Security hardening:** tracked privately. + +**What the client assumes about the server** + +- Auth happens only at the upgrade. True: no re-check mid-stream. +- A token is valid on every node. True if the OIDC configuration is identical across the cluster. +- 401/403 mean credential problems. **Partly false:** JWKS and UserInfo outages also return 401. +- 421 with `X-QuestDB-Role` means a role reject. + +## 9. Decisions + +All resolved. See the spec's §12 (D1–D9). + +## 10. Java implementation plan + +Build in this order. Each step is one PR and must pass the conformance tests listed for it (spec §10). + +**Constraints for every step** (see `CLAUDE.md`): + +- Code must build on Java 8 and also compile on JDK 11+. +- Zero-GC applies to per-row producer calls and the steady-state I/O loop, not to connect paths. +- Tests run under `assertMemoryLeak`. +- `ExportedApiCompatibilityTest` must stay green: add API, never retype it. + +1. **Token cache.** + - Implements spec §4–§5, with the types named in spec Appendix B. + - Self-contained: no connect-path changes. + - Tests C1–C8, plus C20 for the cache. +2. **Client integration.** Implements spec §6 and §8.1–§8.3. Tests C9–C16 and C23. + - Add the default method `HttpTokenProvider.onTokenRejected`. + - `S.buildWebSocketAuthHeader` (`S:3427`) returns a named dynamic-credential class with `onRejected`, instead of a lambda. + - Retry the same endpoint once on 401, in three places, and fire `AUTH_FAILED` only on the final outcome: + - the `QWS.connectWalk` 401 branch (`QWS:3427`); + - `QQC.connect` (`:785`); + - `QQC.reconnectViaTracker` (`:1889`). + - Classify failures at SYNC startup in `CWSL.connectWithRetry` (`:1053`), per D6 and D8. + - Test harness: + - add an authorization validator to `TestWebSocketServer` that picks the response status from the header; + - add an expiring-token scheme driven by a fake clock. +3. **Connect string, registry and Azure module.** Implements spec §7. Tests C17–C19. + - Register the keys in `ConfigSchema`. + - Parse and validate the keys in: + - `S.fromConfigWebSocket` and `validateWsConfig` (`S:4018`, `:4227`); + - `QQC.fromConfig` and `validateConfig` (`:375`); + - the exclusivity check in `QuestDBBuilder.build()` (`:190-194`). + - Add the `uses` clause to `module-info.java`. + - Release the lease when a sender or query client closes. + - Add the new `azure/` reactor module. +4. **Health accessor and deadline.** Implements spec §8.4 and §8.5. Tests C21–C22. + +**Separate tickets, outside this feature:** + +- **Dead pooled egress worker** (§3). Start with a test that reproduces it. +- **Treat a 503 at the upgrade as transient in every phase** (§3, spec Appendix C). This must land before any server change to return 503. diff --git a/design/qwp-token-provider-spec.md b/design/qwp-token-provider-spec.md new file mode 100644 index 000000000..9acd0e197 --- /dev/null +++ b/design/qwp-token-provider-spec.md @@ -0,0 +1,575 @@ +# QWP dynamic bearer credentials: cross-language specification + +**Status:** v0.3, ready for implementation. All decisions are resolved (§12). + +**Changes:** + +- **v0.3:** decisions resolved, with D8 (§8.1) and D9 (§7.2) added; Java binding decided (Appendix B). +- **v0.2:** connection health (§8.4), the optional authentication-outage deadline (§8.5), `WWW-Authenticate` handling (§8.2), and Appendices C and D. + +- **Implementations:** Java is the reference. Other clients implement this document; that includes the Rust core used by Python, Go, .NET and Node. +- **Freezing:** the spec freezes when the Java implementation merges. Any later change needs a version bump and a note to client maintainers. +- **Rationale and Java code locations:** `design/entra-id-qwp-auth.md`. Its §10 is the Java implementation plan. + +**Conventions:** + +- **MUST, SHOULD, MAY** are used as in RFC 2119. +- **[Dn]** marks a rule that follows from a decision recorded in §12. +- Durations are defaults that implementations MAY expose as options. They SHOULD be identical across clients. + +## 1. Scope + +**In scope:** + +- obtaining, caching and refreshing bearer tokens for the QWP WebSocket upgrade, for both ingress (`/write/v4`) and egress (`/read/v1`); +- when clients obtain tokens, and how token-related failures are handled; +- connect-string keys that select a token provider, and the `azure` provider. + +**Out of scope:** + +- the protocol after the upgrade; +- ILP over HTTP and PGWire. A provider defined here MAY also serve them; +- server behaviour. Appendix A describes it and Appendix C proposes changes; both are informative only; +- interactive sign-in. The OIDC device flow is covered by `design/oidc-token-persistence.md`. + +## 2. Terms + +| Term | Meaning | +|---|---| +| Credential | The value of the `Authorization` header on the upgrade. It is either **static** (fixed at configuration: Basic auth, or a `token=` value) or **dynamic** (obtained from a token provider). | +| Token source | A function, supplied by the application or an integration, that obtains a new token from an identity platform (§4). | +| Token provider | The client-side component that connection code asks for the current token. The **refreshing provider** (§5) is a token provider that wraps a token source. | +| Upgrade attempt | One HTTP upgrade request sent to one endpoint. | +| Connect round | One walk over the configured endpoints. It ends at the first successful upgrade, or when every endpoint has failed. | +| Initialization | The phase of a sender before its first successful upgrade. | +| Established | The phase of a sender after its first successful upgrade. | +| Orphan drain | The background replay of a store-and-forward slot left behind by another sender. | +| Egress operation | One query execution, including any failover reconnects it triggers. | + +## 3. Wire format + +This section restates current behaviour; nothing here changes. + +- **Where the credential goes:** it is sent only on the upgrade request, as `Authorization: Bearer `. An established connection is never re-authenticated. +- **Token values:** a token is opaque. + - A client MUST reject an empty or blank token. + - A client MUST reject a token that contains any character outside printable ASCII (0x20–0x7E). + - A client MUST NOT trim or otherwise alter a token. +- **Validating mutable strings:** where the language allows a token to change after it is returned, the client MUST validate a snapshot of it and send that same snapshot. +- **Prefix:** token sources and providers return the token without the `Bearer ` prefix. + +## 4. Token source contract + +`fetch() -> TokenResult | TokenError` + +### TokenResult + +| Field | Required | Meaning | +|---|---|---| +| `token` | yes | The token. | +| `expires_at` | yes | An absolute instant. | +| `refresh_at` | no | The earliest instant the platform suggests refreshing. | + +Rules for `expires_at`: + +- If the source receives a relative lifetime (such as `expires_in`), it SHOULD compute `expires_at` from its own local time of receipt. This makes the cache robust to clock skew. +- If the source cannot determine the expiry, it MUST supply a conservative value. + +### TokenError + +| Field | Meaning | +|---|---| +| `retryable` | Whether retrying can succeed. See classification below. | +| `retry_after` | Optional duration to wait before retrying. | +| `message` | Description of the failure. It MUST NOT contain a token or a raw response body (§9). | + +Classification: + +- **Retryable:** network failures, timeouts, HTTP 429, HTTP 5xx, and IMDS 404 or 410. +- **Permanent:** configuration that is missing or wrong. Examples: no credential configured, identity not found, invalid client. +- **When unsure:** treat the failure as retryable. +- An error that carries no classification MUST be treated as retryable. + +### Rules + +- **No concurrency:** a provider never calls `fetch()` concurrently with itself. +- **Where it runs:** implementations SHOULD run `fetch()` on a background context owned by the provider, never on a connection thread. +- **Blocking:** `fetch()` MAY block, but it SHOULD bound its own network operations. 30 s per attempt is RECOMMENDED. +- **Bad results:** the provider MUST treat a result as a retryable error in either of these cases: + - the token fails the checks in §3; + - `expires_at` is not later than the time the result was received. + +## 5. Refreshing provider + +### 5.1 Parameters + +| Name | Default | Meaning | +|---|---|---| +| `refresh_margin` | 5 min | Upper bound on how close to expiry a scheduled refresh may run. | +| `min_refresh_interval` | 30 s | Lower bound on the time between a fetch and the next scheduled refresh. | +| `handout_floor` | 60 s | A token is **usable** only while `now < expires_at - handout_floor`. | +| `cold_wait` | 30 s | The longest a caller waits when no usable token is held. | +| `backoff_initial` | 0.5 s | First retry delay after a failed fetch. | +| `backoff_max` | 60 s | Largest retry delay after failed fetches. | +| `retry_after_max` | 5 min | Cap applied to a source's `retry_after`. | +| `forced_min_interval` | 30 s | Minimum spacing between forced refreshes (§5.5). | +| `forced_wait` | 5 s | The longest `on_rejected` waits for a forced refresh. | +| `registry_linger` | 60 s | How long a connect-string provider stays alive after its last lease is released (§7.4). | + +### 5.2 Refresh schedule + +After a successful fetch received at time `f`, with expiry `e`: + +``` +L = e - f +m = min(refresh_margin, L / 2) +p = min(min_refresh_interval, L / 2) +r = f + L * U(0.45, 0.55) // jittered half-life; U is uniform +r = min(r, e - m) +r = max(r, f + p) +if refresh_at is present: + r = min(r, max(refresh_at, f + p)) +``` + +- **Proactive refresh:** the provider MUST start a fetch at `r` without waiting for a caller. +- **Effect:** while the source is healthy, even an idle client always holds a usable token. + +*Informative examples:* + +- A 60-minute service-principal token is refreshed at about 27–33 minutes. +- A 24-hour managed-identity token is refreshed at about 11–13 hours. +- If the platform keeps returning the same token, each refresh happens at about half of the remaining lifetime. So a token is fetched only a handful of times before it expires. + +### 5.3 Handing out a token: `get_token()` + +1. **Usable token held.** Return it at once, with no I/O and no waiting. If `now >= r` and no fetch is running or scheduled, start one in the background. +2. **No usable token held.** Proceed as follows: + - Start a fetch now, unless one is already running or the provider is in failure backoff (§5.4). + - If the earliest permitted fetch is later than `now + cold_wait`, fail immediately. + - Otherwise wait until one of these happens: a fetch yields a usable token, `cold_wait` elapses, or the caller is cancelled. + - On failure, raise a **token-unavailable** error. It takes the classification of the most recent failed fetch. If no fetch has failed (the wait timed out, or the caller was cancelled), it is retryable. +3. **Never hand out an unusable token.** The provider MUST NOT do so. +4. **Cancellation.** If the calling operation is cancelled (for example, because the client is closing): + - the wait MUST end promptly with a retryable error; + - the cancellation signal MUST be preserved for the caller. In Java, that is the thread's interrupt flag. +5. **Concurrency.** `get_token()` MUST be safe to call concurrently from any number of threads. Any number of concurrent callers without a usable token MUST cause at most one fetch. + +### 5.4 Failures + +**Backoff after the n-th consecutive failed fetch:** + +``` +base = min(backoff_max, backoff_initial * 2^(n-1)) +delay = base / 2 + U(0, base / 2) +if retry_after is present: + delay = max(delay, min(retry_after, retry_after_max)) +``` + +A successful fetch resets `n`. + +**While fetches are failing:** + +- **Keep retrying** while the provider is open, whatever the classification. An operator can fix a "permanent" error without a restart, for example by assigning a role. +- **Keep serving** the current token for as long as it remains usable. +- **Log** each failure as a warning, at most once per minute per provider. The log contains only the classification and the sanitized message. + +### 5.5 Forced refresh: `on_rejected(token, status)` + +Connection code calls this when an upgrade that presented `token` was rejected (§8.2). + +**Ignore the call** in any of these cases: + +- `status` is not 401 **[D2]**; +- `token` is no longer the provider's current token (it has already rotated); +- a forced refresh ran less than `forced_min_interval` ago. + +**Otherwise:** + +- Start a fetch now. Join a fetch that is already running, and respect any failure backoff. +- Wait up to `forced_wait` for the fetch to finish, then return. +- A forced fetch that returns the same token counts as a normal success. + +*Informative:* platforms that cache tokens, such as Azure managed identity, usually return the same token again. A forced refresh mainly helps client-credential flows and clock skew; proactive refresh (§5.2) remains the main defence. + +### 5.6 Lifecycle + +- **Prefetch.** The provider SHOULD start its first fetch as soon as it is created. +- **Readiness.** It SHOULD offer `await_ready(timeout) -> bool`, which returns true once a usable token is held. Applications can use it to gate startup. +- **Close.** + - `close()` stops scheduled fetches and wakes all waiters. + - After `close()`, `get_token()` MUST fail with a permanent error. + - A client MUST NOT close a provider that the application supplied. +- **Clocks.** + - Expiry comparisons use wall-clock time. + - Delays and waits use a monotonic clock. + - Usability is re-evaluated on every call. + +### 5.7 Observability + +The provider SHOULD expose: + +- the current token's expiry; +- the time of the last successful fetch; +- the last failure: its classification, sanitized message, and time. + +None of these MAY reveal the token itself. + +## 6. When the client obtains the token + +- **At the start of every connect round.** A client using a dynamic credential MUST get it from the provider: + - on the initial connect; + - on every reconnect; + - on every failover reconnect; + - on every orphan-drain or recovery connect; + - on every egress connect and egress failover reconnect. +- **Reuse.** The client MAY present that credential on every upgrade attempt within the round. It MUST NOT reuse it in a later round. +- **Provider failure.** If the provider fails, the round MUST end without contacting any endpoint. The failure is classified as **credential-unavailable** (§8). +- **Cancellation.** Getting the credential MUST be cancellable when the client shuts down. +- **Established connections.** Sending data on an established connection MUST NOT query the provider. + +## 7. Connect-string keys + +### 7.1 Keys [D3] + +| Key | Values | Required | Notes | +|---|---|---|---| +| `token_provider` | `azure`. The name `azure_imds` is reserved **[D5]**. | no | Selects a provider. | +| `azure_resource` | `api://` or `` | when `token_provider=azure` | The application ID URI or the client ID of the QuestDB app registration. A trailing `/.default` MUST be stripped. The requested scope is `/.default`. | +| `azure_client_id` | a GUID | no | The client ID of a user-assigned managed identity or of a workload identity. | + +### 7.2 Validation + +A client MUST reject the configuration, with an error that names the offending key, if any of these holds: + +- `token_provider` is combined with `token`, `username`, `password`, or a provider supplied by the application; +- `token_provider` is empty, unknown, or not supported by this client. The error MUST list the supported values. A client MUST NOT silently connect without credentials; +- a provider-specific key is present but the selected provider does not accept it. For example, `azure_resource` without `token_provider=azure`; +- a provider key that is required is missing; +- `token_provider` is used with any schema other than `wss::`: + - plain `ws::` is rejected because the token would cross the network in cleartext **[D4]**; + - non-QWP schemas, such as ILP's `http::`, are out of scope **[D9]**. + +**Ingress and egress.** Both MUST accept these keys, so a single string can configure both. + +**Validating without connecting.** Validation that does not connect, such as a facade checking its configuration at build time, MUST NOT fetch a token. + +### 7.3 Secrets + +- **No secrets in the string.** The keys defined here are non-secret, so a connect string that uses them stays safe to log and to place in `QDB_CLIENT_CONF`. +- **Where secrets live.** Secrets, such as a service-principal secret or certificate, come from the platform's standard configuration. They never come from connect-string keys. + +### 7.4 Sharing + +- **One provider per configuration.** A process MUST NOT run more than one refreshing provider for the same `(token_provider, parameters)`. +- **Registry.** Clients resolve connect-string providers through a registry that is shared by the whole process. Its key is the provider name plus the normalized parameter values. +- **Leases.** + - Every client instance built from a connect string acquires a lease and releases it on close. That covers each sender, each query client and each pooled connection. + - When the last lease is released, the provider is closed after `registry_linger`, unless a new lease is acquired first. +- **Application-supplied providers.** Sharing is the application's responsibility: use one provider instance per identity per process. + +### 7.5 The `azure` provider + +- **How tokens are obtained:** through the language's Azure Identity default credential chain, for the scope `/.default`. Depending on the environment, the chain finds: + - an environment service principal, + - a workload identity, + - a managed identity, + - developer tools. +- **`azure_client_id`:** when set, it selects both the managed identity and the workload identity. +- **Expiry:** `expires_at`, and `refresh_at` when the library exposes it, come from the library's token result. +- **Classification of library errors:** + - **Permanent:** "no credential available in the chain", and authentication failures that Entra reports (such as an invalid client or an application that does not exist). + - **Retryable:** network errors, timeouts, throttling, 5xx responses, and anything else. +- **Unsupported platforms:** a client whose platform has no Azure Identity library MAY leave `azure` unsupported, and must then reject it as described in §7.2. + +*Informative bindings:* + +- **Java:** the optional `questdb-client-azure` artifact, which uses `DefaultAzureCredential` and is found through `ServiceLoader`. +- **Python:** `azure-identity` as an optional extra. +- **Rust and C:** not required. + +## 8. Failure handling + +### 8.1 Classes + +| Class | Trigger | +|---|---| +| credential-unavailable (retryable or permanent) | The provider failed while the client was getting the credential (§6). | +| auth-rejected 401 | The upgrade response was `401`. | +| auth-rejected 403 | The upgrade response was `403`. | + +This specification does not change how other upgrade failures are handled: role rejects (`421`), transport errors, version mismatches, and other 4xx and 5xx responses. + +**Exceptions from application-supplied providers [D8].** + +- An exception from an application-supplied provider that is not a token-unavailable error counts as credential-unavailable, **permanent**. Startup therefore fails fast, as it does today for an OIDC device-flow provider that is not signed in. +- Only a token-unavailable error marked retryable is retried at SYNC startup. + +### 8.2 One retry after a 401 + +When an upgrade attempt that presented a dynamic credential receives `401`, the client: + +1. MUST call `on_rejected(presented token, 401)`; +2. MUST get the credential from the provider again; +3. if the new credential differs from the one presented, MUST retry the same endpoint once, immediately and without backoff. That retry: + - MUST NOT count as an endpoint failure for health tracking or backoff; + - is an ordinary upgrade attempt for any outcome other than `401`; +4. if the credential is unchanged, or the retry also receives `401`, applies the phase policy in §8.3. + +Further rules: + +- **Limit:** at most one such retry per connect round. +- **Events:** a client that emits connection events SHOULD report the authentication failure only for the round's final outcome. +- **No forced refresh on 403:** a `403` MUST NOT trigger a forced refresh **[D2]**. +- **Static credentials:** they never get this retry. +- **`WWW-Authenticate` challenge:** if the `401` carries a `WWW-Authenticate: Bearer` challenge with an `error` parameter, the client SHOULD take steps 1–3 only when `error` is `invalid_token` (RFC 6750 §3.1). Without such a challenge, the status code alone decides. Current QuestDB servers send no challenge; Appendix C proposes adding one. + +### 8.3 Policy by phase + +| Phase | credential-unavailable, retryable | credential-unavailable, permanent | 401 / 403 | +|---|---|---|---| +| Initialization, `initial_connect_retry=off` | fail startup | fail startup | fail startup | +| Initialization, `on` / `sync` | retry within `reconnect_max_duration_millis` **[D6]** | fail startup | fail startup | +| Initialization, `async` | retry indefinitely | retry indefinitely | fail, and report through the error handler | +| Established | retry indefinitely | retry indefinitely | retry indefinitely **[D1]** | +| Orphan drain, dynamic credential | retry indefinitely | retry indefinitely | ride out, then quarantine (rule below) | +| Orphan drain, static credential | n/a | n/a | quarantine on the first rejection | +| Egress operation | fail the operation | fail the operation | fail the operation | + +**Retry indefinitely** + +- **Meaning:** retry with capped exponential backoff for as long as the client is open, unless the optional deadline in §8.5 is configured. Store-and-forward MUST NOT drop a live producer. +- **Reporting:** each retried failure MUST be reported to the application's error channel, not only to logs. The report is a retriable, security-category error that names the failure class. The failure SHOULD also be visible in the connection health (§8.4). + +**Ride out, then quarantine** + +The slot is quarantined when either of these holds: + +- at least 6 rejections, **and** the rejections have persisted for at least `min(reconnect_max_duration_millis, 5 min)`; +- 256 rejections within one episode. + +An episode ends only when the drain makes acknowledgement progress. A failure of another class in between restarts the persistence clock but not the count. + +**Egress recovery** + +- After a failed connect or failover reconnect, the client MUST do one of two things: reconnect on its next operation, or be replaced by its pool. It MUST NOT remain permanently unusable. +- The error reported for the failed operation MUST name the failure class. + +### 8.4 Connection health + +Retrying indefinitely is only safe if an outage is easy to see. An application that registers no error handler must still be able to find out that its sender has not reached the server for an hour. Kafka shows the failure mode: when credentials expire, a consumer keeps failing authentication in the background while the application sees nothing (KAFKA-10840, Appendix D). + +Each sender and query client SHOULD expose a snapshot of its connection health, with at least: + +| Field | Meaning | +|---|---| +| `state` | `connecting` (no successful upgrade yet), `connected`, `reconnecting` (the connection was lost and is being re-established), `failed` (terminal) or `closed`. | +| `last_connected_at` | Time of the most recent successful upgrade, or none. | +| `outage_since` | Start of the current period without a connection, or none while connected. | +| `failed_rounds` | Number of failed connect rounds since `outage_since`. | +| `last_failure` | The most recent failed connect round: its class (`credential_unavailable`, `auth_rejected` with the status code, `role_rejected`, `transport` or `other`), a sanitized message, and the time. It is kept after the connection recovers. | + +Rules: + +- **Cheap and safe to poll.** Reading the snapshot MUST NOT wait on I/O or on connection threads, and MUST be safe from any thread, so that it can back a health endpoint. +- **No secrets.** The snapshot MUST NOT contain a credential (§9). +- **Pools and facades** SHOULD expose an aggregate: the number of connections in each `state`, the oldest `outage_since`, and the most recent `last_failure`. +- **Query clients** use the same fields. They connect on demand, so for them `reconnecting` means the last connect or failover reconnect failed and the next operation will try again. + +### 8.5 Optional authentication-outage deadline [D7] + +Some applications prefer a hard failure to an authentication outage that lasts indefinitely, much as Kafka's `delivery.timeout.ms` bounds how long a record may wait. A client MAY offer this. If it does, it MUST use this key and these semantics. + +| Key | Values | Default | +|---|---|---| +| `auth_failure_max_duration_millis` | a positive duration in milliseconds | not set (no deadline) | + +- **Which failures count.** Only *authentication-class* failures: credential-unavailable, and auth-rejected (`401` or `403`). +- **Where it applies.** Ingress senders, in the phases where §8.3 retries an authentication-class failure indefinitely: `async` initialization and established. It does not apply to orphan drains, which have their own rule, or to egress, which already fails the operation. +- **The clock.** It starts at the first authentication-class failure of the current outage. Only a successful upgrade resets it. +- **When it fires.** The sender becomes terminal when a connect round fails with an authentication-class failure and the clock has reached the configured duration. + - Rounds that fail for other reasons neither reset the clock nor fire the deadline. + - So a network outage that follows a single `401` does not fire it. + - And a cluster that alternates between `401` and network errors cannot postpone it forever. +- **What happens then.** + - The error names the failure class and the elapsed time. It is reported as terminal through the error handler and raised from later producer calls. + - Unacknowledged data stays in on-disk store-and-forward storage, for a later sender or an orphan drain. In memory-only mode it is lost when the sender closes. + +## 9. Security + +- **Tokens.** A client MUST NOT write a token into logs, error messages, events, or the string representation of any object. A representation MAY include the token's length, its expiry, and a fingerprint of at most the first 8 hex digits of the token's SHA-256. +- **Response bodies.** A token source MUST NOT log or embed a token-endpoint response body. For a malformed response, it reports only the shape of the problem, such as a missing or invalid field. +- **Error text from libraries and endpoints.** Before such text enters client errors or logs, the client SHOULD: + - remove control, bidirectional and other non-displayable characters; + - truncate it to 256 characters. +- **Raw requests.** Objects that serialize raw HTTP requests MUST NOT be attached to errors or logs, because requests may carry secrets. +- **Cleartext.** A client SHOULD warn once when a provider supplied by the application is used over a non-TLS schema. + +## 10. Conformance tests + +Every client that implements this specification SHOULD pass these scenarios. They use: + +- a fake clock; +- a scripted token source; +- a stub server that answers `401` to an expired token. Tokens can be shaped `T.` and checked against the shared fake clock. + +| ID | Scenario | Expected result | +|---|---|---| +| C1 | Refresh schedule | Refresh times follow §5.2 for `L` = 24 h, 60 min and 5 min, with and without `refresh_at`. | +| C2 | Same token returned | Fetches converge as described in §5.2, and the token stays usable. | +| C3 | Cold burst | 64 concurrent callers with no usable token cause exactly one fetch. | +| C4 | Cold failure | A retryable error is raised within `cold_wait`. A permanent classification is passed through. | +| C5 | Backoff | Delays follow §5.4. `retry_after` is honoured and capped. | +| C6 | Hand-out floor | A token inside `handout_floor` is never handed out. | +| C7 | Cancellation | A cancelled wait returns promptly, and the cancellation signal is preserved. | +| C8 | Forced refresh | Rate limited. A stale token is ignored. A same-token result is accepted. | +| C9 | Reconnect after expiry | The reconnect carries the refreshed token, and no `401` is observed. | +| C10 | Stale token presented | Exactly one `401`, then an immediate retry with a new token: no backoff, no health penalty. | +| C11 | Persistent `401` | Each phase follows §8.3. | +| C12 | Initialization matrix | `off`, `sync` and `async`, each with a retryable and a permanent source failure, behave as in §8.3. | +| C13 | Provider outage while established | Retried indefinitely and reported. Recovers when the source recovers. | +| C14 | Failover across a rotation | The second endpoint receives the new token. Unacknowledged data is replayed exactly once. | +| C15 | Store-and-forward across a rotation | All rows are delivered. No quarantine and no data-loss report. | +| C16 | Orphan drain across a rotation | The slot drains with the new token. | +| C17 | Shared cache | N senders and query clients built from one string use one provider, with one fetch per refresh. | +| C18 | Connect-string validation | Every rule in §7.2 is enforced, and validation does not fetch a token. | +| C19 | Registry lifecycle | Lease release, linger, and re-acquiring a lease during the linger period. | +| C20 | Redaction | A sentinel token never appears in logs, errors, events, string representations or health snapshots, across C4, C10, C11 and a malformed source response. | +| C21 | Connection health | The snapshot moves through `connecting`, `connected`, `reconnecting` (with `401` and credential-unavailable failures) and back to `connected`. `outage_since` and `failed_rounds` reset on recovery; `last_failure` is kept. No credential appears in it. | +| C22 | Authentication-outage deadline | Unset: retries continue indefinitely. Set: the sender becomes terminal only on an authentication-class round once the duration has passed. Other failures in between neither reset nor fire it. A successful upgrade resets it. Orphan drains are unaffected, and on-disk data remains. | +| C23 | `WWW-Authenticate` challenge | A plain `401`, and a `401` with `Bearer error="invalid_token"`, both trigger the retry in §8.2. A `401` whose Bearer challenge carries another error does not. | + +## 11. Relation to existing specifications + +- **QWP ingress and egress specs** (`qwp-ingress-websocket.md`, `qwp-egress-websocket.md`): + - their statement that "authentication errors are terminal at any host" is replaced by §8 **[D1]**; + - their authentication sections should point to §6. +- **Connect-string reference:** it should reserve the keys in §7.1, and `auth_failure_max_duration_millis` (§8.5). +- **OIDC device-flow token store** (`design/oidc-token-persistence.md`): unaffected. An `OidcDeviceAuth` instance remains a valid provider supplied by the application. + +## 12. Decisions + +All decisions below are resolved. Changing one after the Java implementation merges needs a version bump. + +| ID | Question | Decision | +|---|---|---| +| D1 | How to treat 401/403 once a sender is established | Retry indefinitely, report it (§8.3), and expose it in the connection health (§8.4). This is Java's current behaviour and matches common practice (Appendix D). The Rust core must change to match. | +| D2 | Whether a 403 also triggers a forced refresh | No. Only a 401 does, as in RFC 6750 and the clients in Appendix D. | +| D3 | Key names | `token_provider`, `azure_resource`, `azure_client_id`. (`client_id` is already used by egress.) | +| D4 | Whether to reject `token_provider` on `ws::` | Yes. | +| D5 | A zero-dependency `azure_imds` provider | The name is reserved; it is implemented only on demand. | +| D6 | Whether SYNC initialization retries retryable provider failures | Yes. | +| D7 | Whether to offer the authentication-outage deadline, and its key | Yes, as an optional setting that is off by default: `auth_failure_max_duration_millis`. | +| D8 | How to classify an exception from an application-supplied provider that is not a token-unavailable error | Permanent (§8.1). | +| D9 | Whether `token_provider` applies to non-QWP schemas | No; `wss::` only (§7.2). | + +**Still to verify in a real Entra tenant.** These checks don't block implementation, but they should be done before the spec freezes: + +- Managed-identity tokens from IMDS carry the client-ID GUID as `aud` once the app registration is set to v2 tokens. +- The v1 and v2 signing-key endpoints serve the same keys. +- Whether IMDS returns the same token when asked again. This affects only the convergence note in §5.2. +- How long `DefaultAzureCredential` takes on a cold start. This sizes `cold_wait`. + +## Appendix A. QuestDB Enterprise and Entra configuration (informative) + +Describes QuestDB Enterprise behaviour at the time of writing. + +**Server properties:** + +``` +acl.oidc.enabled=true +acl.oidc.configuration.url=https://login.microsoftonline.com//v2.0/.well-known/openid-configuration +acl.oidc.client.id= # also the expected audience +acl.oidc.groups.encoded.in.token=true # local JWKS validation; required for app-only tokens +acl.oidc.groups.claim=roles +acl.oidc.sub.claim=oid +``` + +**Entra setup:** + +- On the QuestDB app registration: + - expose an application ID URI such as `api://`; + - define app roles whose allowed member type is Applications; + - set `requestedAccessTokenVersion` to 2 (`accessTokenAcceptedVersion` in the manifest). Every token then carries the app's client ID as `aud`. +- Assign at least one app role to each managed identity or service principal. + +**QuestDB setup:** + +```sql +CREATE GROUP ingest WITH EXTERNAL ALIAS 'QuestDB.Ingest'; +GRANT HTTP TO ingest; +GRANT INSERT ON trades TO ingest; +``` + +**Server behaviours clients should expect:** + +- **401 does not always mean a bad token.** `401` is also returned when the server cannot fetch its signing keys (JWKS). +- **One audience.** `aud` must match a single exact string. v1 tokens carry the requested resource string as `aud`; v2 tokens carry the client ID. +- **Expiry.** `exp` is enforced with 60 s of leeway. +- **Roles are required.** A token without the roles claim is rejected with `401`. +- **Cache.** A verified token is cached for 30 s. +- **Role changes apply only to new connections,** and only once a new token carries them. Managed-identity tokens can live for about 24 h. + +## Appendix B. Java binding (decided) + +New types live in `io.questdb.client.cutlass.auth` unless noted. + +| Spec concept | Java | +|---|---| +| TokenResult | `ExpiringToken(String token, long expiresAtEpochMillis)`, plus an overload that adds `long refreshAtEpochMillis` (0 means none) | +| Token source | `TokenSource`, a functional interface: `ExpiringToken fetchToken()` | +| Refreshing provider | `RefreshingTokenProvider implements HttpTokenProvider, QuietCloseable`, created with `RefreshingTokenProvider.builder(TokenSource)`. Methods: `getToken()`, `onTokenRejected(CharSequence, int)`, `awaitReady(long)`, `close()`. The builder exposes the §5.1 parameters, plus clock and scheduler seams for tests. | +| token-unavailable error | `TokenUnavailableException extends LineSenderException`, with `isRetryable()` and `getRetryAfterMillis()` (-1 means none) | +| `on_rejected` | `onTokenRejected(CharSequence token, int httpStatus)`, a default no-op method on the existing `io.questdb.client.HttpTokenProvider` | +| Provider factory | `TokenProviderFactory`, an SPI found through `ServiceLoader` (a `uses` clause in `module-info.java`) | +| Registry | `TokenProviderRegistry`: process-wide, hands out ref-counted leases | +| `azure` provider | The new reactor module `azure/`, artifact `io.questdb:questdb-client-azure`, package `io.questdb.client.azure`, class `AzureTokenProviderFactory`. It depends on `azure-identity` through `azure-sdk-bom`, keeps the Java 8 floor, and is released together with the client. | +| Connection health | `io.questdb.client.ConnectionHealth`, an immutable snapshot. Returned by `Sender.health()` (a default method; non-QWP senders throw `UnsupportedOperationException`), by `QwpQueryClient.health()`, and as an aggregate by `QuestDB.health()`. | +| Authentication-outage deadline | Key `auth_failure_max_duration_millis`; builder method `authFailureMaxDurationMillis(long)` | + +**Python notes:** + +- A token source maps to a callable that returns `azure.core.credentials.AccessToken`, which carries `token` and `expires_on` (in epoch seconds). +- Keep the provider in the Rust core, so that the callable runs only on the provider's refresh thread. +- A provider MUST NOT be used across `fork()`. + +## Appendix C. Recommended server changes (informative) + +These changes would let clients refresh only when a new token can help, and treat server-side outages as transient. + +| Situation | Today | Recommended | +|---|---|---| +| The token is invalid: bad signature, expired, wrong audience, or malformed | `401`, no challenge | `401` with `WWW-Authenticate: Bearer error="invalid_token"`. An `error_description` is fine, but never the token. | +| The token is valid but grants nothing: no roles claim, or no mapped group | `401` without a roles claim; `403` without a mapped group | `403` with `WWW-Authenticate: Bearer error="insufficient_scope"` | +| The server cannot verify tokens: it cannot fetch its signing keys (JWKS) or reach UserInfo | `401` | `503`, optionally with `Retry-After` | + +**Sequencing: land the client fix first.** + +- Java currently treats any non-421 rejection of the upgrade as terminal during initialization. +- An orphan drain quarantines its slot on such a rejection immediately (`design/entra-id-qwp-auth.md` §3). +- If the server switched to `503` first, a signing-key outage would therefore quarantine orphan slots that today ride out a `401`. +- The client should treat a `503` at the upgrade as transient in every phase, as the public QWP spec already says. + +Other server-side follow-ups, including security hardening, are tracked privately with the QuestDB Enterprise team (see `SECURITY.md`). + +## Appendix D. Precedents for D1 and D2 (informative) + +| Source | Behaviour | Bearing on this spec | +|---|---|---| +| RFC 6750 §3.1 | `invalid_token` goes with `401`, and "the client MAY request a new access token and retry". `insufficient_scope` goes with `403`. | Refresh on `401` only (D2); the challenge rule in §8.2. | +| MongoDB driver auth spec (MONGODB-OIDC) | On an authentication failure with a cached token: clear that token from the cache (atomically, and only if it is unchanged), fetch a new one, and retry once. Other errors go to the user. At least 100 ms between calls to the token callback. | The closest match to §5.5 and §8.2. | +| Google auth library, `HttpCredentialsAdapter` | Refreshes and retries on a `401`, or on a Bearer `invalid_token` challenge. A plain `403` does not trigger it. | D2. | +| Kubernetes client-go, exec credential plugin | On a `401`, refreshes "unless they were rotated already", then returns the `401`. The next request uses the new credentials. | The stale-token guard in §5.5. | +| Azure SDK, `BearerTokenAuthenticationPolicy` | Retries only on a `401` that carries a Continuous Access Evaluation claims challenge. A plain `401` goes back to the caller. | A stricter variant of D2. | +| Kafka: KIP-152 and KAFKA-6516 | Authentication failures are non-retriable at the API. The client is not closed, though, and keeps reconnecting in the background; a request to stop that was closed Won't Fix. | D1. | +| Kafka: KAFKA-10840 (open) | When credentials expire, a consumer keeps failing authentication in the background and the application cannot see it. A proposed fix notes that Kafka Connect tasks report RUNNING meanwhile. | Why §8.4 exists. | + +Links: + +- RFC 6750: https://www.rfc-editor.org/rfc/rfc6750#section-3.1 +- MongoDB auth spec: https://github.com/mongodb/specifications/blob/master/source/auth/auth.md +- Google `HttpCredentialsAdapter`: https://github.com/googleapis/google-auth-library-java/blob/main/oauth2_http/java/com/google/auth/http/HttpCredentialsAdapter.java +- client-go exec plugin: https://github.com/kubernetes/client-go/blob/master/plugin/pkg/client/auth/exec/exec.go +- Azure `BearerTokenAuthenticationPolicy`: https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/core/azure-core/src/main/java/com/azure/core/http/policy/BearerTokenAuthenticationPolicy.java +- KIP-152: https://cwiki.apache.org/confluence/display/KAFKA/KIP-152+-+Improve+diagnostics+for+SASL+authentication+failures +- KAFKA-6516: https://issues.apache.org/jira/browse/KAFKA-6516 +- KAFKA-10840: https://issues.apache.org/jira/browse/KAFKA-10840 (proposed fix: https://github.com/apache/kafka/pull/16418) From c9f2b96a45cfc41978ba5701d51ad58bf1cccdc9 Mon Sep 17 00:00:00 2001 From: Vlad Ilyushchenko Date: Fri, 2 Oct 2026 11:03:20 +0100 Subject: [PATCH 02/40] feat(qwp): refreshing bearer-token providers and Microsoft Entra ID auth Implement the dynamic-credential specification (design/qwp-token-provider-spec.md, v0.3) and the four-step Java plan in design/entra-id-qwp-auth.md, section 10. - Token cache: RefreshingTokenProvider, ExpiringToken, TokenSource and TokenUnavailableException. Proactive jittered refresh at about half the token lifetime, a 60 s hand-out floor, single-flight cold waits bounded by cold_wait, jittered backoff honouring Retry-After, rate-limited forced refresh, and token redaction everywhere. - Client integration: one immediate same-endpoint retry after a refreshable 401 (WWW-Authenticate aware; never for a 403 or a static credential) on every ingest connect path, orphan drains included, and on egress connect and failover. SYNC startup retries a retryable provider failure within its budget (D6); any other provider exception still fails fast (D8). Egress errors name the failure class, and a query client whose failover reconnect failed reconnects on its next execute() instead of staying unusable (the suspected dead pooled worker, reproduced by a test before the fix). - Connect string: token_provider, azure_resource and azure_client_id on both clients (wss:: only; exclusive with static credentials and application-supplied providers), resolved through a ServiceLoader SPI and a process-wide ref-counted registry with a 60 s linger. Validation never fetches a token. - New optional module azure/ (org.questdb:questdb-client-azure): token_provider=azure on DefaultAzureCredential with the spec's error classification. Java 8 floor, released together with the client. - Connection health: Sender.health(), QwpQueryClient.health() and an aggregate QuestDB.health(); an optional auth_failure_max_duration_millis deadline for authentication outages. - Tests for conformance scenarios C1-C23, plus a TLS mode for the test WebSocket server (its key is generated at test time). README and design docs updated; the spec's Appendix B now names the published artifact org.questdb:questdb-client-azure. --- .gitignore | 1 + README.md | 99 ++ azure/pom.xml | 307 +++++ .../azure/AzureTokenProviderFactory.java | 91 ++ .../client/azure/AzureTokenSource.java | 230 ++++ ...b.client.cutlass.auth.TokenProviderFactory | 1 + .../azure/test/AzureTokenSourceTest.java | 292 +++++ .../io/questdb/client/ConnectionHealth.java | 290 +++++ .../io/questdb/client/HttpTokenProvider.java | 31 + .../main/java/io/questdb/client/QuestDB.java | 12 + .../io/questdb/client/QuestDBBuilder.java | 4 + .../main/java/io/questdb/client/Sender.java | 837 ++++++++------ .../cutlass/auth/CredentialRedaction.java | 145 +++ .../client/cutlass/auth/ExpiringToken.java | 148 +++ .../cutlass/auth/RefreshingTokenProvider.java | 1027 +++++++++++++++++ .../cutlass/auth/TokenProviderFactory.java | 83 ++ .../cutlass/auth/TokenProviderRegistry.java | 302 +++++ .../cutlass/auth/TokenProviderSpec.java | 248 ++++ .../client/cutlass/auth/TokenSource.java | 63 + .../auth/TokenUnavailableException.java | 128 ++ .../client/cutlass/http/BearerChallenge.java | 158 +++ .../cutlass/http/client/WebSocketClient.java | 45 + .../qwp/client/QwpAuthFailedException.java | 58 +- .../client/QwpConnectionHealthTracker.java | 182 +++ .../QwpCredentialUnavailableException.java | 41 +- .../cutlass/qwp/client/QwpQueryClient.java | 212 +++- .../qwp/client/QwpUpgradeFailures.java | 3 +- .../qwp/client/QwpWebSocketSender.java | 206 +++- .../sf/cursor/CursorWebSocketSendLoop.java | 140 ++- .../io/questdb/client/impl/ConfigSchema.java | 9 + .../io/questdb/client/impl/PooledSender.java | 6 + .../questdb/client/impl/QueryClientPool.java | 16 + .../io/questdb/client/impl/QuestDBImpl.java | 9 + .../io/questdb/client/impl/SenderPool.java | 20 + core/src/main/java/module-info.java | 4 + .../auth/RefreshingTokenProviderTest.java | 841 ++++++++++++++ .../auth/TestTokenProviderFactory.java | 100 ++ .../cutlass/auth/TokenProviderConfigTest.java | 255 ++++ .../test/cutlass/auth/TokenTestKit.java | 269 +++++ .../cutlass/http/BearerChallengeTest.java | 94 ++ .../qwp/client/ConnectionHealthTest.java | 431 +++++++ .../QwpQueryClientDynamicCredentialTest.java | 349 ++++++ ...QwpWebSocketSenderJvmErrorCleanupTest.java | 5 +- .../qwp/client/TokenProviderSharingTest.java | 161 +++ .../WebSocketDynamicCredentialTest.java | 968 ++++++++++++++++ .../test/cutlass/qwp/websocket/TestTls.java | 112 ++ .../qwp/websocket/TestWebSocketServer.java | 109 +- .../impl/QwpQueryClientConfigHonoredTest.java | 14 + .../test/impl/WsSenderConfigHonoredTest.java | 15 + design/entra-id-qwp-auth.md | 52 +- design/qwp-token-provider-spec.md | 8 +- pom.xml | 1 + 52 files changed, 8799 insertions(+), 433 deletions(-) create mode 100644 azure/pom.xml create mode 100644 azure/src/main/java/io/questdb/client/azure/AzureTokenProviderFactory.java create mode 100644 azure/src/main/java/io/questdb/client/azure/AzureTokenSource.java create mode 100644 azure/src/main/resources/META-INF/services/io.questdb.client.cutlass.auth.TokenProviderFactory create mode 100644 azure/src/test/java/io/questdb/client/azure/test/AzureTokenSourceTest.java create mode 100644 core/src/main/java/io/questdb/client/ConnectionHealth.java create mode 100644 core/src/main/java/io/questdb/client/cutlass/auth/CredentialRedaction.java create mode 100644 core/src/main/java/io/questdb/client/cutlass/auth/ExpiringToken.java create mode 100644 core/src/main/java/io/questdb/client/cutlass/auth/RefreshingTokenProvider.java create mode 100644 core/src/main/java/io/questdb/client/cutlass/auth/TokenProviderFactory.java create mode 100644 core/src/main/java/io/questdb/client/cutlass/auth/TokenProviderRegistry.java create mode 100644 core/src/main/java/io/questdb/client/cutlass/auth/TokenProviderSpec.java create mode 100644 core/src/main/java/io/questdb/client/cutlass/auth/TokenSource.java create mode 100644 core/src/main/java/io/questdb/client/cutlass/auth/TokenUnavailableException.java create mode 100644 core/src/main/java/io/questdb/client/cutlass/http/BearerChallenge.java create mode 100644 core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpConnectionHealthTracker.java create mode 100644 core/src/test/java/io/questdb/client/test/cutlass/auth/RefreshingTokenProviderTest.java create mode 100644 core/src/test/java/io/questdb/client/test/cutlass/auth/TestTokenProviderFactory.java create mode 100644 core/src/test/java/io/questdb/client/test/cutlass/auth/TokenProviderConfigTest.java create mode 100644 core/src/test/java/io/questdb/client/test/cutlass/auth/TokenTestKit.java create mode 100644 core/src/test/java/io/questdb/client/test/cutlass/http/BearerChallengeTest.java create mode 100644 core/src/test/java/io/questdb/client/test/cutlass/qwp/client/ConnectionHealthTest.java create mode 100644 core/src/test/java/io/questdb/client/test/cutlass/qwp/client/QwpQueryClientDynamicCredentialTest.java create mode 100644 core/src/test/java/io/questdb/client/test/cutlass/qwp/client/TokenProviderSharingTest.java create mode 100644 core/src/test/java/io/questdb/client/test/cutlass/qwp/client/WebSocketDynamicCredentialTest.java create mode 100644 core/src/test/java/io/questdb/client/test/cutlass/qwp/websocket/TestTls.java diff --git a/.gitignore b/.gitignore index 872223992..edcddfd0b 100644 --- a/.gitignore +++ b/.gitignore @@ -9,6 +9,7 @@ core/target utils/target utils/dependency-reduced-pom.xml examples/target +azure/target compat/target core/node deploy*.bat diff --git a/README.md b/README.md index e945c7770..ebdd2e145 100644 --- a/README.md +++ b/README.md @@ -485,6 +485,101 @@ The token is stored as **plaintext JSON protected by file permissions** — `060 `FileTokenStore` is safe to share between processes that sign in as the same identity: each update is written atomically (so a concurrent reader never sees a half-written credential), and when the identity provider rotates the refresh token on each refresh, the read-refresh-write is serialized across processes with a lock file so they do not race each other into an unnecessary re-prompt. The lock file's staleness is judged by its modification time, so this coordination assumes the processes share a clock — a single machine, or machines with synchronized clocks; under significant clock skew (for example a store directory on NFS shared across hosts) a live lock can be mis-judged stale or a dead one never expire. `clearCache()` removes the persisted entry under the same lock, but across processes it is best-effort: a peer that still holds a live in-memory token may legitimately re-persist afterwards (it always forces a fresh sign-in for the calling process). +### Rotating Bearer Tokens (Microsoft Entra ID and Other Identity Platforms) + +Service identities - an Azure managed identity, a service principal, an AKS workload identity - authenticate with +bearer tokens that expire every hour or every day. Select a **token provider** and the client fetches tokens itself, +keeps them in a shared cache, and refreshes them in the background well before they expire, so long-lived senders and +query clients keep reconnecting with a valid token. + +**Microsoft Entra ID from a connect string.** Add the optional `questdb-client-azure` artifact (same version as +`questdb-client`); it brings Azure Identity and registers `token_provider=azure`: + +```xml + + org.questdb + questdb-client-azure + 1.0.0 + +``` + +```java +try (QuestDB db = QuestDB.connect( + "wss::addr=qdb1:9000,qdb2:9000;token_provider=azure;azure_resource=api://;")) { + // ... use db ... +} +``` + +- `azure_resource` (required) is the application ID URI (`api://`) or the client ID of the QuestDB app + registration; the client requests the scope `/.default`. +- `azure_client_id` (optional) selects a user-assigned managed identity or a workload identity. +- The credential comes from Azure Identity's `DefaultAzureCredential`: an environment service principal + (`AZURE_TENANT_ID`, `AZURE_CLIENT_ID`, `AZURE_CLIENT_SECRET` or a certificate), a workload identity, a managed + identity, or developer tools. No secret ever goes into the connect string, so it stays safe to log and to put in + `QDB_CLIENT_CONF`. +- `token_provider` requires `wss::`, and cannot be combined with `token`, `username`/`password` or an + application-supplied provider. +- Every sender, query client and pooled connection built from equivalent connect strings shares one provider per + process: one refresher thread, one call to Entra at a time. + +On the server, validate tokens locally (`acl.oidc.groups.encoded.in.token=true`, `acl.oidc.groups.claim=roles`, +`acl.oidc.sub.claim=oid`), set the QuestDB app registration to issue v2 tokens, and map its app roles to groups with +`CREATE GROUP ... WITH EXTERNAL ALIAS ''`. + +**Any other identity platform.** Wrap a `TokenSource` in a `RefreshingTokenProvider` and pass the provider to +`QuestDB.connect(config, provider)`, `httpTokenProvider(...)` or `withBearerTokenProvider(...)`. The application owns +the provider: close it after the clients that use it. + +```java +import io.questdb.client.cutlass.auth.ExpiringToken; +import io.questdb.client.cutlass.auth.RefreshingTokenProvider; +import io.questdb.client.cutlass.auth.TokenUnavailableException; + +RefreshingTokenProvider tokens = RefreshingTokenProvider.builder(() -> { + MyToken t = myIdentityPlatform.requestToken(); // runs on the provider's own thread + return new ExpiringToken(t.value(), t.expiresAtEpochMillis()); +}).build(); +try (QuestDB db = QuestDB.connect("wss::addr=qdb1:9000;", tokens)) { + // ... use db ... +} finally { + tokens.close(); +} +``` + +A source reports a failure by throwing `TokenUnavailableException.retryable(...)` (network trouble, throttling, +5xx) or `TokenUnavailableException.permanent(...)` (missing or wrong configuration); anything else counts as +retryable. Never put a token or a raw response body in the message. The provider keeps retrying with backoff and +keeps serving the current token while it is still valid. + +How failures are handled: + +- A token is refreshed at about half its lifetime, so an idle client always holds a valid one. +- When the server rejects a token with `401`, the client asks the provider for a new one and retries the same server + once, immediately. +- At startup, `initial_connect_retry=off` (the default) fails if no token can be obtained; `on` keeps retrying a + retryable provider failure within `reconnect_max_duration_millis`; `async` retries in the background. +- Once connected, a store-and-forward sender rides out credential outages and `401`/`403` indefinitely, buffering + rows and reporting each failure to the error handler. Set `auth_failure_max_duration_millis` to make the sender + fail instead once such an outage lasts that long (unacknowledged rows stay on disk). + +### Connection Health + +A sender that rides out an outage keeps accepting rows, so check its connection health to see that it is not +reaching the server. Reading health never blocks and never contains a credential, so it can back a health endpoint. + +```java +ConnectionHealth.Aggregate health = db.health(); // every pooled sender and query client +if (health.count(ConnectionHealth.State.RECONNECTING) > 0) { + long since = health.getOldestOutageSinceEpochMillis(); + ConnectionHealth.Failure failure = health.getLastFailure(); // e.g. AUTH_REJECTED 401, CREDENTIAL_UNAVAILABLE + // ... +} +``` + +`Sender.health()` and `QwpQueryClient.health()` return the same information for a single client: its state +(`CONNECTING`, `CONNECTED`, `RECONNECTING`, `FAILED`, `CLOSED`), the last successful connect, the start of the current +outage, the number of failed connect rounds since, and the last failure with its class. + ### Explicit Timestamps ```java @@ -529,6 +624,10 @@ schema::key1=value1;key2=value2; | `tls_roots_password` | | Optional JKS/PKCS#12 password; omit when `tls_roots` is PEM | | `connect_timeout` | _(OS)_ | TCP connect + TLS handshake timeout, in milliseconds | | `auth_timeout_ms` | `15000` | Authentication/upgrade request timeout, in milliseconds | +| `token_provider` | | Refreshing bearer-token provider, `wss` only: `azure` (needs `questdb-client-azure`) | +| `azure_resource` | | `token_provider=azure`: application ID URI or client ID of the QuestDB app registration | +| `azure_client_id` | | `token_provider=azure`: client ID of a user-assigned managed identity or workload identity | +| `auth_failure_max_duration_millis` | _(none)_ | Ingest: fail the sender once a credential outage or `401`/`403` lasts this long | ### Pool keys (facade only) diff --git a/azure/pom.xml b/azure/pom.xml new file mode 100644 index 000000000..754ad21cb --- /dev/null +++ b/azure/pom.xml @@ -0,0 +1,307 @@ + + + + + 4.0.0 + + org.questdb + questdb-client-azure + 1.3.10-SNAPSHOT + jar + QuestDB client - Microsoft Entra ID token provider + token_provider=azure for the QuestDB Java client: Microsoft Entra ID bearer tokens through Azure Identity's DefaultAzureCredential + https://questdb.io/ + + + + Apache 2.0 + https://www.apache.org/licenses/LICENSE-2.0.txt + repo + + + + + + QuestDB Team + hello@questdb.io + + + + + https://github.com/questdb/java-questdb-client + scm:git:https://github.com/questdb/java-questdb-client.git + scm:git:https://github.com/questdb/java-questdb-client.git + HEAD + + + + UTF-8 + + 1.3.8 + 1.5.25 + + + + + + com.azure + azure-sdk-bom + ${azure-sdk-bom.version} + pom + import + + + + + + + org.questdb + questdb-client + ${project.version} + + + com.azure + azure-identity + + + + + junit + junit + 4.13.2 + test + + + ch.qos.logback + logback-classic + ${logback.version} + test + + + + + + + org.apache.maven.plugins + maven-compiler-plugin + 3.11.0 + + ${javac.compile.source} + ${javac.compile.target} + + + + org.apache.maven.plugins + maven-surefire-plugin + 3.5.3 + + + false + + + + org.apache.maven.plugins + maven-jar-plugin + 3.0.1 + + + + + io.questdb.client.azure + ${project.version} + + + + + + + org.apache.maven.plugins + maven-deploy-plugin + 2.8.2 + + true + + + + + + + + java11+ + + [11,) + + + + 11 + 11 + + + + java8 + + 1.8 + + + 1.8 + 1.8 + + 1.3.15 + + + + javadoc + + + + org.apache.maven.plugins + maven-javadoc-plugin + 3.5.0 + + + attach-javadocs + + jar + + + + + none + ${javac.compile.source} + false + + + + + + + release-artifacts + + + + org.apache.maven.plugins + maven-javadoc-plugin + 3.5.0 + + + attach-javadocs + + jar + + + + + none + ${javac.compile.source} + false + + + + org.apache.maven.plugins + maven-source-plugin + 3.0.1 + + + attach-sources + + jar + + + + + + org.apache.maven.plugins + maven-gpg-plugin + 3.2.7 + + + sign-artifacts + verify + + sign + + + + + gpg + + --pinentry-mode + loopback + + + + + + + + maven-central-publish + + + + + org.apache.maven.plugins + maven-enforcer-plugin + 3.0.0-M3 + + + enforce-publish-from-jdk8 + + enforce + + + + + [1.8,1.9) + questdb-client-azure is published with questdb-client, from JDK 8. + + + + + + + + org.sonatype.central + central-publishing-maven-plugin + 0.9.0 + true + + central + false + validated + + + + + + + diff --git a/azure/src/main/java/io/questdb/client/azure/AzureTokenProviderFactory.java b/azure/src/main/java/io/questdb/client/azure/AzureTokenProviderFactory.java new file mode 100644 index 000000000..950f1bbcb --- /dev/null +++ b/azure/src/main/java/io/questdb/client/azure/AzureTokenProviderFactory.java @@ -0,0 +1,91 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.azure; + +import com.azure.identity.DefaultAzureCredentialBuilder; +import io.questdb.client.cutlass.auth.TokenProviderFactory; +import io.questdb.client.cutlass.auth.TokenProviderSpec; +import io.questdb.client.cutlass.auth.TokenSource; + +import java.util.Map; + +/** + * The {@code azure} token provider (design/qwp-token-provider-spec.md, section 7.5): Microsoft Entra ID tokens + * from Azure Identity's {@code DefaultAzureCredential}. Discovered through {@link java.util.ServiceLoader}; put + * this artifact on the class path or module path and connect with + * 
+ * wss::addr=qdb1:9000,qdb2:9000;token_provider=azure;azure_resource=api://<questdb-app-id>;
+ *
+ *
    + *
  • {@code azure_resource} (required) - the application ID URI or client ID of the QuestDB app + * registration; the requested scope is {@code /.default}.
    • + *
  • {@code azure_client_id} (optional) - the client ID of a user-assigned managed identity or of a workload + * identity; it selects both.
    • + *
+ * Depending on the environment, the credential chain finds an environment service principal + * ({@code AZURE_CLIENT_ID}, {@code AZURE_TENANT_ID}, {@code AZURE_CLIENT_SECRET} or a certificate), a workload + * identity, a managed identity, or developer tools. Secrets come from that standard configuration, never from + * the connect string. + */ +public final class AzureTokenProviderFactory implements TokenProviderFactory { + + @Override + public TokenSource createSource(Map params) { + final String resource = params.get(TokenProviderSpec.KEY_AZURE_RESOURCE); + if (resource == null) { + throw new IllegalArgumentException("token_provider=azure requires " + TokenProviderSpec.KEY_AZURE_RESOURCE); + } + final DefaultAzureCredentialBuilder builder = new DefaultAzureCredentialBuilder(); + final String clientId = params.get(TokenProviderSpec.KEY_AZURE_CLIENT_ID); + if (clientId != null) { + builder.managedIdentityClientId(clientId); + builder.workloadIdentityClientId(clientId); + } + return new AzureTokenSource(builder.build(), resource); + } + + @Override + public String describe(Map params) { + final StringBuilder sb = new StringBuilder(TokenProviderSpec.AZURE) + .append("[resource=").append(params.get(TokenProviderSpec.KEY_AZURE_RESOURCE)); + final String clientId = params.get(TokenProviderSpec.KEY_AZURE_CLIENT_ID); + if (clientId != null) { + sb.append(", client_id=").append(clientId); + } + return sb.append(']').toString(); + } + + @Override + public String name() { + return TokenProviderSpec.AZURE; + } + + @Override + public void validate(Map params) { + if (params.get(TokenProviderSpec.KEY_AZURE_RESOURCE) == null) { + throw new IllegalArgumentException("token_provider=azure requires " + TokenProviderSpec.KEY_AZURE_RESOURCE); + } + } +} diff --git a/azure/src/main/java/io/questdb/client/azure/AzureTokenSource.java b/azure/src/main/java/io/questdb/client/azure/AzureTokenSource.java new file mode 100644 index 000000000..b0c6e793d --- /dev/null +++ b/azure/src/main/java/io/questdb/client/azure/AzureTokenSource.java @@ -0,0 +1,230 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.azure; + +import com.azure.core.credential.AccessToken; +import com.azure.core.credential.TokenCredential; +import com.azure.core.credential.TokenRequestContext; +import com.azure.core.exception.HttpResponseException; +import com.azure.core.http.HttpResponse; +import com.azure.identity.CredentialUnavailableException; +import io.questdb.client.cutlass.auth.ExpiringToken; +import io.questdb.client.cutlass.auth.TokenSource; +import io.questdb.client.cutlass.auth.TokenUnavailableException; + +import java.time.Duration; +import java.time.OffsetDateTime; +import java.util.concurrent.TimeUnit; + +/** + * A {@link TokenSource} that obtains Microsoft Entra ID access tokens from an Azure Identity + * {@link TokenCredential} - by default a {@code DefaultAzureCredential} - for the scope + * {@code /.default} (design/qwp-token-provider-spec.md, section 7.5). + *

+ * Use it directly to put an Azure credential of your choice behind a + * {@link io.questdb.client.cutlass.auth.RefreshingTokenProvider}: + * 

{@code
+ * TokenCredential credential = new ClientCertificateCredentialBuilder()...build();
+ * RefreshingTokenProvider tokens = RefreshingTokenProvider.builder(
+ *         new AzureTokenSource(credential, "api://")).build();
+ * }
+ * or let a connect string select {@code token_provider=azure}, which uses {@code DefaultAzureCredential}. + *

+ * Expiry and the refresh hint come from the library's {@link AccessToken}. Failures are classified as section + * 7.5 prescribes: + *

    + *
  • permanent - "no credential available in the chain" ({@link CredentialUnavailableException}), and + * authentication failures Entra reports: an HTTP 400 or 401 from the token endpoint, or a known AADSTS + * configuration error such as an invalid client or an application that does not exist;
    • + *
  • retryable - network errors, timeouts, throttling, 5xx responses, and anything else. A + * {@code Retry-After} in seconds is passed on.
    • + *
+ * Each attempt is bounded (30 s by default). Library exceptions never travel as a cause - their response + * objects reference the raw HTTP request, which can carry a client secret - only their sanitized message does. + */ +public final class AzureTokenSource implements TokenSource { + /** + * The bound on one token request. + */ + public static final Duration DEFAULT_TIMEOUT = Duration.ofSeconds(30); + private static final String DEFAULT_SCOPE_SUFFIX = "/.default"; + private static final int MAX_CAUSE_DEPTH = 16; + // Entra (AADSTS) errors that a retry cannot fix: the configuration is wrong, an operator must act. + private static final String[] PERMANENT_AADSTS = { + "AADSTS700016", // application not found in the directory + "AADSTS7000215", // invalid client secret + "AADSTS7000222", // client secret expired + "AADSTS700027", // client assertion signature invalid + "AADSTS70011", // invalid scope + "AADSTS500011", // resource principal not found: azure_resource is wrong + "AADSTS90002", // tenant not found + "AADSTS700213", // no matching federated identity record + "AADSTS70021", // no matching federated identity record + "AADSTS7000112", // application disabled + "AADSTS50049", // unknown or invalid instance + }; + private final TokenRequestContext context; + private final TokenCredential credential; + private final String scope; + private final Duration timeout; + + /** + * @param credential the Azure Identity credential + * @param resource the application ID URI ({@code api://}) or client ID of the QuestDB app + * registration; a trailing {@code /.default} is accepted + */ + public AzureTokenSource(TokenCredential credential, String resource) { + this(credential, resource, DEFAULT_TIMEOUT); + } + + /** + * @param credential the Azure Identity credential + * @param resource the application ID URI ({@code api://}) or client ID of the QuestDB app + * registration; a trailing {@code /.default} is accepted + * @param timeout the bound on one token request + */ + public AzureTokenSource(TokenCredential credential, String resource, Duration timeout) { + if (credential == null) { + throw new IllegalArgumentException("credential must not be null"); + } + if (resource == null || resource.isEmpty()) { + throw new IllegalArgumentException("resource must not be empty"); + } + if (timeout == null || timeout.isNegative() || timeout.isZero()) { + throw new IllegalArgumentException("timeout must be positive"); + } + this.credential = credential; + this.scope = resource.endsWith(DEFAULT_SCOPE_SUFFIX) ? resource : resource + DEFAULT_SCOPE_SUFFIX; + this.context = new TokenRequestContext().addScopes(scope); + this.timeout = timeout; + } + + /** + * Classifies a failure from Azure Identity per section 7.5. Never attaches the library exception. + */ + static TokenUnavailableException classify(Throwable failure, String scope) { + final String description = describe(failure); + Throwable t = failure; + for (int depth = 0; t != null && depth < MAX_CAUSE_DEPTH; depth++, t = t.getCause()) { + if (t instanceof CredentialUnavailableException) { + return TokenUnavailableException.permanent( + "no credential available in the Azure Identity chain for " + scope + ": " + description); + } + if (t instanceof HttpResponseException) { + final HttpResponse response = ((HttpResponseException) t).getResponse(); + if (response != null) { + final int status = response.getStatusCode(); + if (status == 400 || status == 401) { + return TokenUnavailableException.permanent( + "Entra rejected the token request for " + scope + " with HTTP " + status + ": " + description); + } + return TokenUnavailableException.retryable( + "the token request for " + scope + " failed with HTTP " + status + ": " + description, + retryAfterMillis(response)); + } + } + final String message = t.getMessage(); + if (message != null) { + for (String code : PERMANENT_AADSTS) { + if (message.contains(code)) { + return TokenUnavailableException.permanent( + "Entra rejected the token request for " + scope + " (" + code + "): " + description); + } + } + } + } + return TokenUnavailableException.retryable("the token request for " + scope + " failed: " + description); + } + + private static String describe(Throwable t) { + final String message = t.getMessage(); + return message == null ? t.getClass().getName() : t.getClass().getSimpleName() + ": " + message; + } + + private static long retryAfterMillis(HttpResponse response) { + final String value; + try { + value = response.getHeaderValue("Retry-After"); + } catch (RuntimeException e) { + return TokenUnavailableException.NO_RETRY_AFTER; + } + if (value == null) { + return TokenUnavailableException.NO_RETRY_AFTER; + } + try { + // delta-seconds only; an HTTP-date is ignored and the provider's own backoff applies + return TimeUnit.SECONDS.toMillis(Long.parseLong(value.trim())); + } catch (NumberFormatException e) { + return TokenUnavailableException.NO_RETRY_AFTER; + } + } + + private static long refreshAtMillis(AccessToken token) { + try { + final OffsetDateTime refreshAt = token.getRefreshAt(); + return refreshAt == null ? ExpiringToken.NO_REFRESH_AT : refreshAt.toInstant().toEpochMilli(); + } catch (NoSuchMethodError e) { + // an azure-core older than the refresh hint + return ExpiringToken.NO_REFRESH_AT; + } + } + + @Override + public ExpiringToken fetchToken() { + final AccessToken token; + try { + token = credential.getToken(context).block(timeout); + } catch (RuntimeException e) { + throw classify(e, scope); + } + if (token == null) { + throw TokenUnavailableException.retryable("Azure Identity returned no token for " + scope); + } + final OffsetDateTime expiresAt = token.getExpiresAt(); + if (token.getToken() == null || expiresAt == null) { + // the shape of the problem only, never the response + throw TokenUnavailableException.retryable("Azure Identity returned a token for " + scope + + " without " + (token.getToken() == null ? "a value" : "an expiry")); + } + try { + return new ExpiringToken(token.getToken(), expiresAt.toInstant().toEpochMilli(), refreshAtMillis(token)); + } catch (IllegalArgumentException e) { + throw TokenUnavailableException.retryable("Azure Identity returned an unusable token for " + scope + + ": " + e.getMessage()); + } + } + + /** + * @return the requested scope, {@code /.default} + */ + public String getScope() { + return scope; + } + + @Override + public String toString() { + return "AzureTokenSource{scope=" + scope + ", credential=" + credential.getClass().getSimpleName() + '}'; + } +} diff --git a/azure/src/main/resources/META-INF/services/io.questdb.client.cutlass.auth.TokenProviderFactory b/azure/src/main/resources/META-INF/services/io.questdb.client.cutlass.auth.TokenProviderFactory new file mode 100644 index 000000000..97f279b6a --- /dev/null +++ b/azure/src/main/resources/META-INF/services/io.questdb.client.cutlass.auth.TokenProviderFactory @@ -0,0 +1 @@ +io.questdb.client.azure.AzureTokenProviderFactory diff --git a/azure/src/test/java/io/questdb/client/azure/test/AzureTokenSourceTest.java b/azure/src/test/java/io/questdb/client/azure/test/AzureTokenSourceTest.java new file mode 100644 index 000000000..d8d6b470b --- /dev/null +++ b/azure/src/test/java/io/questdb/client/azure/test/AzureTokenSourceTest.java @@ -0,0 +1,292 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.azure.test; + +import com.azure.core.credential.AccessToken; +import com.azure.core.credential.TokenCredential; +import com.azure.core.credential.TokenRequestContext; +import com.azure.core.exception.ClientAuthenticationException; +import com.azure.core.exception.HttpResponseException; +import com.azure.core.http.HttpHeaders; +import com.azure.core.http.HttpResponse; +import com.azure.identity.AuthenticationRequiredException; +import com.azure.identity.CredentialUnavailableException; +import io.questdb.client.azure.AzureTokenProviderFactory; +import io.questdb.client.azure.AzureTokenSource; +import io.questdb.client.cutlass.auth.ExpiringToken; +import io.questdb.client.cutlass.auth.RefreshingTokenProvider; +import io.questdb.client.cutlass.auth.TokenProviderRegistry; +import io.questdb.client.cutlass.auth.TokenProviderSpec; +import io.questdb.client.cutlass.auth.TokenSource; +import io.questdb.client.cutlass.auth.TokenUnavailableException; +import io.questdb.client.cutlass.qwp.client.QwpQueryClient; +import io.questdb.client.impl.ConfigString; +import io.questdb.client.impl.ConfigView; +import org.junit.Assert; +import org.junit.Test; +import reactor.core.publisher.Flux; +import reactor.core.publisher.Mono; + +import java.nio.ByteBuffer; +import java.nio.charset.Charset; +import java.time.Duration; +import java.time.Instant; +import java.time.OffsetDateTime; +import java.time.ZoneOffset; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.concurrent.atomic.AtomicReference; + +/** + * The {@code azure} provider (design/qwp-token-provider-spec.md, section 7.5) against fake Azure Identity + * credentials - no network: expiry and refresh-hint mapping, the requested scope, the classification of library + * errors, and discovery through {@link java.util.ServiceLoader}. + */ +public class AzureTokenSourceTest { + private static final OffsetDateTime EXPIRES = OffsetDateTime.of(2027, 1, 15, 12, 0, 0, 0, ZoneOffset.UTC); + private static final OffsetDateTime REFRESH = EXPIRES.minusMinutes(30); + + @Test + public void testAuthenticationFailuresEntraReportsArePermanent() { + assertPermanent(new ClientAuthenticationException("bad request", new FakeResponse(400, null)), "HTTP 400"); + assertPermanent(new ClientAuthenticationException("unauthorized", new FakeResponse(401, null)), "HTTP 401"); + // DefaultAzureCredential often wraps MSAL without a response: recognize the Entra error code + assertPermanent(new ClientAuthenticationException( + "AADSTS700016: Application with identifier 'x' was not found in the directory", null, (Object) null), + "AADSTS700016"); + assertPermanent(new RuntimeException("wrapper", new IllegalStateException( + "AADSTS7000215: Invalid client secret provided.")), "AADSTS7000215"); + } + + @Test + public void testConnectStringResolvesTheAzureFactory() { + // C18 with the module present: the azure keys parse on both clients and nothing is fetched + String cfg = "wss::addr=localhost:9000;token_provider=azure;azure_resource=api://qdb/.default;" + + "azure_client_id=AAAAAAAA-BBBB-CCCC-DDDD-EEEEEEEEEEEE;"; + TokenProviderSpec spec = TokenProviderSpec.parse(new ConfigView(ConfigString.parse(cfg)), true); + Assert.assertNotNull(spec); + Assert.assertTrue(spec.factory() instanceof AzureTokenProviderFactory); + Assert.assertEquals("api://qdb", spec.params().get(TokenProviderSpec.KEY_AZURE_RESOURCE)); + Assert.assertEquals("azure[resource=api://qdb, client_id=aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee]", spec.describe()); + QwpQueryClient.validateConfig(new ConfigView(ConfigString.parse(cfg)), true); + try { + TokenProviderSpec.parse(new ConfigView(ConfigString.parse("wss::addr=localhost:9000;token_provider=azure;")), true); + Assert.fail(); + } catch (IllegalArgumentException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("requires azure_resource")); + } + try { + TokenProviderSpec.parse(new ConfigView(ConfigString.parse("wss::addr=localhost:9000;token_provider=nope;")), true); + Assert.fail(); + } catch (IllegalArgumentException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("supported values: [azure]")); + } + } + + @Test + public void testCredentialUnavailableIsPermanent() { + assertPermanent(new CredentialUnavailableException("EnvironmentCredential authentication unavailable"), + "no credential available"); + assertPermanent(new AuthenticationRequiredException("interactive sign-in required", new TokenRequestContext()), + "no credential available"); + assertPermanent(new RuntimeException("chain failed", new CredentialUnavailableException("none")), + "no credential available"); + } + + @Test + public void testExpiryAndRefreshHintAreMapped() { + AzureTokenSource source = new AzureTokenSource( + ctx -> Mono.just(new AccessToken("eyJ.token", EXPIRES, REFRESH)), "api://qdb"); + ExpiringToken token = source.fetchToken(); + Assert.assertEquals("eyJ.token", token.getToken()); + Assert.assertEquals(EXPIRES.toInstant().toEpochMilli(), token.getExpiresAtEpochMillis()); + Assert.assertEquals(REFRESH.toInstant().toEpochMilli(), token.getRefreshAtEpochMillis()); + + AzureTokenSource noHint = new AzureTokenSource(ctx -> Mono.just(new AccessToken("t", EXPIRES)), "api://qdb"); + Assert.assertFalse(noHint.fetchToken().hasRefreshAt()); + } + + @Test + public void testFactoryBuildsADefaultAzureCredentialSourceWithoutNetworkIo() { + AzureTokenProviderFactory factory = new AzureTokenProviderFactory(); + Map params = new HashMap<>(); + params.put(TokenProviderSpec.KEY_AZURE_RESOURCE, "api://qdb"); + params.put(TokenProviderSpec.KEY_AZURE_CLIENT_ID, "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee"); + TokenSource source = factory.createSource(params); + Assert.assertTrue(source instanceof AzureTokenSource); + Assert.assertEquals("api://qdb/.default", ((AzureTokenSource) source).getScope()); + Assert.assertTrue(source.toString(), source.toString().contains("DefaultAzureCredential")); + } + + @Test + public void testFactoryIsDiscoveredThroughServiceLoader() { + Assert.assertTrue(TokenProviderRegistry.findFactory("azure") instanceof AzureTokenProviderFactory); + List supported = TokenProviderRegistry.supportedProviders(); + Assert.assertTrue(supported.toString(), supported.contains("azure")); + } + + @Test + public void testLibraryExceptionsAreNeverAttached() { + // a library exception's response references the raw HTTP request, which can carry a client secret + TokenUnavailableException e = fetchFailure(new HttpResponseException("throttled", new FakeResponse(429, "3"))); + Assert.assertNull(e.getCause()); + } + + @Test + public void testOtherFailuresAreRetryable() { + assertRetryable(new HttpResponseException("throttled", new FakeResponse(429, "7")), "HTTP 429", 7_000); + assertRetryable(new HttpResponseException("unavailable", new FakeResponse(503, null)), "HTTP 503", -1); + assertRetryable(new HttpResponseException("gone", new FakeResponse(410, "Wed, 21 Oct 2015 07:28:00 GMT")), "HTTP 410", -1); + assertRetryable(new IllegalStateException("connection reset"), "connection reset", -1); + assertRetryable(new ClientAuthenticationException("ManagedIdentityCredential: IMDS endpoint timed out", null, + (Object) null), "timed out", -1); + } + + @Test + public void testResultsWithoutAValueOrExpiryAreRetryable() { + TokenUnavailableException e = expectFailure(new AzureTokenSource( + ctx -> Mono.just(new AccessToken("t", null)), "api://qdb")); + Assert.assertTrue(e.isRetryable()); + Assert.assertTrue(e.getMessage(), e.getMessage().contains("without an expiry")); + e = expectFailure(new AzureTokenSource(ctx -> Mono.empty(), "api://qdb")); + Assert.assertTrue(e.isRetryable()); + e = expectFailure(new AzureTokenSource(ctx -> Mono.just(new AccessToken("bad\r\ntoken", EXPIRES)), "api://qdb")); + Assert.assertTrue(e.isRetryable()); + Assert.assertFalse("the token must not be echoed", e.getMessage().contains("bad")); + } + + @Test + public void testScopeIsResourceDefault() { + AtomicReference> scopes = new AtomicReference<>(); + TokenCredential credential = ctx -> { + scopes.set(ctx.getScopes()); + return Mono.just(new AccessToken("t", EXPIRES)); + }; + new AzureTokenSource(credential, "api://qdb").fetchToken(); + Assert.assertEquals("[api://qdb/.default]", scopes.get().toString()); + new AzureTokenSource(credential, "11111111-2222-3333-4444-555555555555/.default").fetchToken(); + Assert.assertEquals("[11111111-2222-3333-4444-555555555555/.default]", scopes.get().toString()); + } + + @Test(timeout = 30_000) + public void testSlowCredentialIsBoundedAndRetryable() { + AzureTokenSource source = new AzureTokenSource(ctx -> Mono.never(), "api://qdb", Duration.ofMillis(200)); + long start = System.nanoTime(); + TokenUnavailableException e = expectFailure(source); + Assert.assertTrue(e.isRetryable()); + Assert.assertTrue("the attempt must be bounded", System.nanoTime() - start < 10_000_000_000L); + } + + @Test(timeout = 30_000) + public void testWorksBehindTheRefreshingProvider() { + OffsetDateTime expires = OffsetDateTime.now(ZoneOffset.UTC).plusHours(1); + AzureTokenSource source = new AzureTokenSource( + ctx -> Mono.just(new AccessToken("eyJ.cached", expires, expires.minusMinutes(50))), "api://qdb"); + try (RefreshingTokenProvider provider = RefreshingTokenProvider.builder(source).build()) { + Assert.assertTrue(provider.awaitReady(10_000)); + Assert.assertEquals("eyJ.cached", provider.getToken().toString()); + Assert.assertEquals(expires.toInstant().toEpochMilli(), provider.getTokenExpiresAtEpochMillis()); + Assert.assertTrue(provider.getLastSuccessEpochMillis() <= Instant.now().toEpochMilli()); + } + } + + private static void assertPermanent(Throwable libraryFailure, String fragment) { + TokenUnavailableException e = fetchFailure(libraryFailure); + Assert.assertFalse("expected permanent: " + e.getMessage(), e.isRetryable()); + Assert.assertTrue(e.getMessage(), e.getMessage().contains(fragment)); + } + + private static void assertRetryable(Throwable libraryFailure, String fragment, long retryAfterMillis) { + TokenUnavailableException e = fetchFailure(libraryFailure); + Assert.assertTrue("expected retryable: " + e.getMessage(), e.isRetryable()); + Assert.assertTrue(e.getMessage(), e.getMessage().contains(fragment)); + Assert.assertEquals(retryAfterMillis, e.getRetryAfterMillis()); + } + + private static TokenUnavailableException expectFailure(AzureTokenSource source) { + try { + source.fetchToken(); + Assert.fail("expected the fetch to fail"); + return null; + } catch (TokenUnavailableException e) { + return e; + } + } + + private static TokenUnavailableException fetchFailure(Throwable libraryFailure) { + return expectFailure(new AzureTokenSource(ctx -> Mono.error(libraryFailure), "api://qdb")); + } + + private static final class FakeResponse extends HttpResponse { + private final String retryAfter; + private final int status; + + FakeResponse(int status, String retryAfter) { + super(null); + this.status = status; + this.retryAfter = retryAfter; + } + + @Override + public Flux getBody() { + return Flux.empty(); + } + + @Override + public Mono getBodyAsByteArray() { + return Mono.empty(); + } + + @Override + public Mono getBodyAsString() { + return Mono.empty(); + } + + @Override + public Mono getBodyAsString(Charset charset) { + return Mono.empty(); + } + + @Override + public String getHeaderValue(String name) { + return "Retry-After".equalsIgnoreCase(name) ? retryAfter : null; + } + + @Override + public HttpHeaders getHeaders() { + HttpHeaders headers = new HttpHeaders(); + if (retryAfter != null) { + headers.set("Retry-After", retryAfter); + } + return headers; + } + + @Override + public int getStatusCode() { + return status; + } + } +} diff --git a/core/src/main/java/io/questdb/client/ConnectionHealth.java b/core/src/main/java/io/questdb/client/ConnectionHealth.java new file mode 100644 index 000000000..3ea731523 --- /dev/null +++ b/core/src/main/java/io/questdb/client/ConnectionHealth.java @@ -0,0 +1,290 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client; + +import java.time.Instant; + +/** + * An immutable snapshot of a QWP client's connection health (design/qwp-token-provider-spec.md, section 8.4). + * A sender in store-and-forward mode retries an outage indefinitely - a revoked credential, an unreachable + * cluster - while the application keeps writing; this snapshot is how an application, or a health endpoint, + * finds out that it has not reached the server for an hour even when no error handler is registered. + *

+ * Reading it never waits on I/O or on a connection thread: {@link Sender#health()}, + * {@code QwpQueryClient.health()} and {@link QuestDB#health()} return a published snapshot, safe to call from any + * thread at any rate. It never contains a credential. + */ +public final class ConnectionHealth { + /** + * Value of the timestamp getters when there is nothing to report. + */ + public static final long NONE = -1L; + private final long failedRounds; + private final Failure lastFailure; + private final long lastConnectedAtEpochMillis; + private final long outageSinceEpochMillis; + private final State state; + + public ConnectionHealth( + State state, + long lastConnectedAtEpochMillis, + long outageSinceEpochMillis, + long failedRounds, + Failure lastFailure + ) { + this.state = state; + this.lastConnectedAtEpochMillis = lastConnectedAtEpochMillis; + this.outageSinceEpochMillis = outageSinceEpochMillis; + this.failedRounds = failedRounds; + this.lastFailure = lastFailure; + } + + private static String instant(long epochMillis) { + return epochMillis == NONE ? "none" : Instant.ofEpochMilli(epochMillis).toString(); + } + + /** + * Number of failed connect rounds - walks over the configured endpoints that ended without a connection - + * since {@link #getOutageSinceEpochMillis()}. Reset when a connection is established. + */ + public long getFailedRounds() { + return failedRounds; + } + + /** + * The most recent failed connect round: its class, a sanitized message and its time. Kept after the + * connection recovers. Null when no round has failed. + */ + public Failure getLastFailure() { + return lastFailure; + } + + /** + * Wall-clock time of the most recent successful upgrade, or {@link #NONE}. + */ + public long getLastConnectedAtEpochMillis() { + return lastConnectedAtEpochMillis; + } + + /** + * Start of the current period without a connection, or {@link #NONE} while connected. + */ + public long getOutageSinceEpochMillis() { + return outageSinceEpochMillis; + } + + public State getState() { + return state; + } + + @Override + public String toString() { + return "ConnectionHealth{state=" + state + + ", lastConnectedAt=" + instant(lastConnectedAtEpochMillis) + + ", outageSince=" + instant(outageSinceEpochMillis) + + ", failedRounds=" + failedRounds + + ", lastFailure=" + lastFailure + '}'; + } + + /** + * The class of a failed connect round. + */ + public enum FailureClass { + /** + * The token provider could not supply a credential; no endpoint was contacted. + */ + CREDENTIAL_UNAVAILABLE, + /** + * An endpoint rejected the credential with {@code 401} or {@code 403}; see {@link Failure#getStatusCode()}. + */ + AUTH_REJECTED, + /** + * Every reachable endpoint has the wrong role, such as a replica during a failover. + */ + ROLE_REJECTED, + /** + * No endpoint could be reached: connection refused, timed out, or dropped. + */ + TRANSPORT, + /** + * Anything else: another HTTP status at the upgrade, a protocol or capability mismatch. + */ + OTHER + } + + /** + * The connection state. + */ + public enum State { + /** + * No successful upgrade yet. + */ + CONNECTING, + /** + * Connected. + */ + CONNECTED, + /** + * The connection was lost and is being re-established. For a query client, which connects on demand: the + * last connect or failover reconnect failed and the next operation will try again. + */ + RECONNECTING, + /** + * Terminal: the client will not connect again. + */ + FAILED, + /** + * Closed by the application. + */ + CLOSED + } + + /** + * Health across many connections - every pooled connection of a {@link QuestDB} handle: the number of + * connections in each state, the oldest outage and the most recent failure. + */ + public static final class Aggregate { + private final int[] counts; + private final Failure lastFailure; + private final long oldestOutageSinceEpochMillis; + + private Aggregate(int[] counts, long oldestOutageSinceEpochMillis, Failure lastFailure) { + this.counts = counts; + this.oldestOutageSinceEpochMillis = oldestOutageSinceEpochMillis; + this.lastFailure = lastFailure; + } + + /** + * Aggregates the given snapshots. + */ + public static Aggregate of(Iterable healths) { + int[] counts = new int[State.values().length]; + long oldest = NONE; + Failure last = null; + for (ConnectionHealth h : healths) { + counts[h.state.ordinal()]++; + if (h.outageSinceEpochMillis != NONE && (oldest == NONE || h.outageSinceEpochMillis < oldest)) { + oldest = h.outageSinceEpochMillis; + } + if (h.lastFailure != null && (last == null || h.lastFailure.epochMillis > last.epochMillis)) { + last = h.lastFailure; + } + } + return new Aggregate(counts, oldest, last); + } + + /** + * Number of connections in {@code state}. + */ + public int count(State state) { + return counts[state.ordinal()]; + } + + /** + * The most recent failed connect round across all connections, or null. + */ + public Failure getLastFailure() { + return lastFailure; + } + + /** + * The oldest {@code outage_since} across all connections, or {@link #NONE} when all are connected. + */ + public long getOldestOutageSinceEpochMillis() { + return oldestOutageSinceEpochMillis; + } + + /** + * Number of connections aggregated. + */ + public int total() { + int n = 0; + for (int c : counts) { + n += c; + } + return n; + } + + @Override + public String toString() { + StringBuilder sb = new StringBuilder("ConnectionHealth.Aggregate{"); + State[] states = State.values(); + for (int i = 0; i < states.length; i++) { + sb.append(states[i].name().toLowerCase(java.util.Locale.ROOT)).append('=').append(counts[i]).append(", "); + } + return sb.append("oldestOutageSince=").append(instant(oldestOutageSinceEpochMillis)) + .append(", lastFailure=").append(lastFailure).append('}').toString(); + } + } + + /** + * A failed connect round. Never carries a credential. + */ + public static final class Failure { + private final long epochMillis; + private final FailureClass failureClass; + private final String message; + private final int statusCode; + + public Failure(FailureClass failureClass, int statusCode, String message, long epochMillis) { + this.failureClass = failureClass; + this.statusCode = statusCode; + this.message = message; + this.epochMillis = epochMillis; + } + + /** + * Wall-clock time of the failure. + */ + public long getEpochMillis() { + return epochMillis; + } + + public FailureClass getFailureClass() { + return failureClass; + } + + /** + * A sanitized description of the failure, at most 256 characters. + */ + public String getMessage() { + return message; + } + + /** + * The HTTP status of the upgrade rejection - {@code 401} or {@code 403} for + * {@link FailureClass#AUTH_REJECTED} - or {@code 0} when none applies. + */ + public int getStatusCode() { + return statusCode; + } + + @Override + public String toString() { + return "Failure{class=" + failureClass + (statusCode > 0 ? ", status=" + statusCode : "") + + ", at=" + instant(epochMillis) + ", message=" + message + '}'; + } + } +} diff --git a/core/src/main/java/io/questdb/client/HttpTokenProvider.java b/core/src/main/java/io/questdb/client/HttpTokenProvider.java index cf28bbf50..02b318e68 100644 --- a/core/src/main/java/io/questdb/client/HttpTokenProvider.java +++ b/core/src/main/java/io/questdb/client/HttpTokenProvider.java @@ -47,6 +47,18 @@ * resolution to the OS. A provider that builds its own HTTP client should bound its connect and TLS * handshake likewise, or a black-holed token endpoint stalls a flush for the OS connect timeout. An exception from {@link #getToken()} fails the * in-flight flush (HTTP) or the connection attempt (WebSocket). + *

+ * Failure classification (WebSocket). A {@link io.questdb.client.cutlass.auth.TokenUnavailableException} marked + * retryable is a transient credential outage: a {@code Sender} whose initial connect retries + * ({@code initial_connect_retry=on}) keeps retrying it within {@code reconnect_max_duration_millis}. Any other + * exception counts as a permanent credential failure, so startup fails fast, as it does for an OIDC device-flow + * provider that is not signed in yet. Once a sender has connected, every credential failure is retried while + * store-and-forward keeps the rows. + *

+ * For a token that rotates on a schedule - a Microsoft Entra ID managed identity or service principal, an OAuth + * client-credentials grant - wrap a {@link io.questdb.client.cutlass.auth.TokenSource} in a + * {@link io.questdb.client.cutlass.auth.RefreshingTokenProvider}, which caches the token and refreshes it in the + * background before it expires. * * @see QuestDB#connect(CharSequence, HttpTokenProvider) * @see QuestDBBuilder#httpTokenProvider(HttpTokenProvider) @@ -106,4 +118,23 @@ static void validateToken(CharSequence token) { * @return the current HTTP authentication token */ CharSequence getToken(); + + /** + * Tells the provider that a server rejected a token it handed out, so that a caching provider can refresh + * early. WebSocket ingest and query clients call this when an upgrade that presented {@code token} is + * answered with {@code 401} - and, when the server sent a {@code WWW-Authenticate: Bearer} challenge that + * carries an {@code error}, only when that error is {@code invalid_token}. They then call {@link #getToken()} + * again and, if the token changed, retry the same endpoint once. A {@code 403} is never reported: it is an + * authorization decision that a new token does not change. + *

+ * The call may block briefly while the provider refreshes - a {@code RefreshingTokenProvider} waits up to its + * {@code forced_wait}, 5 s by default - and must honour an interrupt by returning promptly with the + * interrupt flag still set. It should not throw; the clients ignore anything it throws. The default does + * nothing, which suits a provider that always returns its freshest token anyway. + * + * @param token the rejected token, without the {@code "Bearer "} prefix + * @param httpStatus the HTTP status of the rejection + */ + default void onTokenRejected(CharSequence token, int httpStatus) { + } } diff --git a/core/src/main/java/io/questdb/client/QuestDB.java b/core/src/main/java/io/questdb/client/QuestDB.java index cf669a166..8cfc437b7 100644 --- a/core/src/main/java/io/questdb/client/QuestDB.java +++ b/core/src/main/java/io/questdb/client/QuestDB.java @@ -160,6 +160,18 @@ static QuestDB connect(CharSequence configurationString, HttpTokenProvider token */ Sender borrowSender(); + /** + * Aggregated connection health of every pooled connection - ingest senders and query clients: the number of + * connections in each state, the oldest outage and the most recent failed connect round + * (design/qwp-token-provider-spec.md, section 8.4). Cheap and safe to call from any thread, so it can back a + * health endpoint; it never waits on I/O and never contains a credential. + * + * @return the aggregate + */ + default ConnectionHealth.Aggregate health() { + throw new UnsupportedOperationException("connection health is not available from this QuestDB implementation"); + } + /** * Shuts down the pools and their published clients. Idempotent. Threads * currently blocked in {@link #borrowSender()} or {@link Query#submit()} diff --git a/core/src/main/java/io/questdb/client/QuestDBBuilder.java b/core/src/main/java/io/questdb/client/QuestDBBuilder.java index e846ad129..b37cdcf42 100644 --- a/core/src/main/java/io/questdb/client/QuestDBBuilder.java +++ b/core/src/main/java/io/questdb/client/QuestDBBuilder.java @@ -192,6 +192,10 @@ public QuestDB build() { throw new IllegalArgumentException( "httpTokenProvider cannot be combined with token, username, or password in the configuration"); } + if (httpTokenProvider != null && view.has("token_provider")) { + throw new IllegalArgumentException( + "httpTokenProvider cannot be combined with token_provider in the configuration"); + } // Validate the single cluster config exactly as both pools will, but // without connecting: the full Sender parse plus validateParameters // (ingress value keys are registry-STRING, so only the real parse diff --git a/core/src/main/java/io/questdb/client/Sender.java b/core/src/main/java/io/questdb/client/Sender.java index 645d7b254..57ff7bf59 100644 --- a/core/src/main/java/io/questdb/client/Sender.java +++ b/core/src/main/java/io/questdb/client/Sender.java @@ -25,6 +25,8 @@ package io.questdb.client; import io.questdb.client.cutlass.auth.AuthUtils; +import io.questdb.client.cutlass.auth.TokenProviderRegistry; +import io.questdb.client.cutlass.auth.TokenProviderSpec; import io.questdb.client.cutlass.line.AbstractLineTcpSender; import io.questdb.client.cutlass.line.LineChannel; import io.questdb.client.cutlass.line.LineSenderException; @@ -77,6 +79,7 @@ import java.util.Base64; import java.util.List; import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicBoolean; import java.util.function.Supplier; /** @@ -623,6 +626,22 @@ default long getAckedFsn() { return -1L; } + /** + * A snapshot of this sender's connection health: whether it is connected, since when it has been without a + * connection, how many connect rounds have failed since, and the most recent failure with its class + * (design/qwp-token-provider-spec.md, section 8.4). A store-and-forward sender retries an outage - a revoked + * credential, an unreachable cluster - indefinitely while the application keeps writing; this is how to see + * that it has not reached the server. Cheap and safe to call from any thread: it never waits on I/O or on the + * sender's I/O thread, so it can back a health endpoint. It never contains a credential. + * + * @return the current health + * @throws UnsupportedOperationException for transports that do not track connection health: only the + * WebSocket (QWP) sender does + */ + default ConnectionHealth health() { + throw new UnsupportedOperationException("connection health is only available for WebSocket (QWP) senders"); + } + /** * Add a column with a 32-bit signed integer value. * @@ -947,6 +966,8 @@ enum Transport { */ final class LineSenderBuilder { private static final int AUTO_FLUSH_DISABLED = 0; + // Warn once per process when an application-supplied token provider is used over ws:: (spec section 9). + private static final AtomicBoolean CLEARTEXT_PROVIDER_WARNED = new AtomicBoolean(); private static final String TLS_ROOTS_INSECURE_CONFIG_ERROR = "tls_roots cannot be combined with tls_verify=unsafe_off; remove tls_verify to use custom roots, or remove tls_roots to disable certificate validation"; // close() drain timeout. Default applied at build() time. 0 or -1 // means "fast close" (skip the drain entirely); any positive value @@ -1041,6 +1062,8 @@ final class LineSenderBuilder { OrphanScanner.QUARANTINE_SLOT_INFIX; private final ObjList hosts = new ObjList<>(); private final IntList ports = new IntList(); + // Optional authentication-outage deadline (spec section 8.5); 0 = not set, the default. + private long authFailureMaxDurationMillis; private long authTimeoutMillis = QwpWebSocketSender.DEFAULT_AUTH_TIMEOUT_MS; private int autoFlushBytes = PARAMETER_NOT_SET_EXPLICITLY; private int autoFlushIntervalMillis = PARAMETER_NOT_SET_EXPLICITLY; @@ -1093,6 +1116,10 @@ final class LineSenderBuilder { private int httpTimeout = PARAMETER_NOT_SET_EXPLICITLY; private String httpToken; private HttpTokenProvider httpTokenProvider; + // A token_provider selected by the connect string (wss:: only). Resolved through the process-wide + // TokenProviderRegistry when build() connects - never at parse time, so validating a configuration + // fetches no token - and released when the sender closes. + private TokenProviderSpec tokenProviderSpec; // Drives the initial-connect strategy. null means "not set // explicitly", which build() resolves to SYNC when any reconnect_* // knob was tuned by the user, otherwise OFF. SYNC retries on the @@ -1274,6 +1301,33 @@ public LineSenderBuilder connectTimeoutMillis(int millis) { return this; } + /** + * Optional authentication-outage deadline (design/qwp-token-provider-spec.md, section 8.5): the sender + * becomes terminal when a connect round fails with an authentication-class failure - the token provider + * could not supply a credential, or an endpoint answered {@code 401}/{@code 403} - once that outage has + * lasted this long. Not set by default: such outages are retried indefinitely while store-and-forward + * keeps the rows. The clock starts at the first authentication-class failure of an outage and only a + * successful upgrade resets it; failures of other classes neither reset nor fire it. Applies while the + * sender is established and during an {@code async} initial connect; orphan drains are unaffected. When + * it fires, the error names the failure class and the elapsed time, goes to the error handler as + * terminal, and is thrown from later producer calls; unacknowledged rows stay in on-disk + * store-and-forward for a later sender or an orphan drain. WebSocket transport only. Connect-string key: + * {@code auth_failure_max_duration_millis}. + * + * @param millis the deadline, {@code > 0} + * @return this instance for method chaining + */ + public LineSenderBuilder authFailureMaxDurationMillis(long millis) { + if (protocol != PARAMETER_NOT_SET_EXPLICITLY && protocol != PROTOCOL_WEBSOCKET) { + throw new LineSenderException("auth_failure_max_duration_millis is only supported for WebSocket transport"); + } + if (millis <= 0) { + throw new LineSenderException("auth_failure_max_duration_millis must be > 0: ").put(millis); + } + this.authFailureMaxDurationMillis = millis; + return this; + } + /** * Per-endpoint timeout on the WebSocket upgrade response read. Default * {@value QwpWebSocketSender#DEFAULT_AUTH_TIMEOUT_MS} ms. @@ -1455,355 +1509,21 @@ public Sender build() { } if (protocol == PROTOCOL_WEBSOCKET) { - if (hosts.size() < 1) { - throw new LineSenderException("WebSocket transport requires at least one host:port pair"); - } - - int actualAutoFlushRows = autoFlushRows == PARAMETER_NOT_SET_EXPLICITLY ? DEFAULT_WS_AUTO_FLUSH_ROWS : autoFlushRows; - int actualAutoFlushBytes = autoFlushBytes == PARAMETER_NOT_SET_EXPLICITLY ? DEFAULT_WS_AUTO_FLUSH_BYTES : autoFlushBytes; - long actualAutoFlushIntervalNanos = autoFlushIntervalMillis == PARAMETER_NOT_SET_EXPLICITLY - ? DEFAULT_WS_AUTO_FLUSH_INTERVAL_NANOS - : TimeUnit.MILLISECONDS.toNanos(autoFlushIntervalMillis); - - Supplier wsAuthHeader = buildWebSocketAuthHeader(); - - ClientTlsConfiguration wsTlsConfig = null; - if (tlsEnabled) { - assert trustStorePassword == null || trustStorePath != null; - wsTlsConfig = new ClientTlsConfiguration( - trustStorePath, - trustStorePassword, - tlsValidationMode == TlsValidationMode.DEFAULT - ? ClientTlsConfiguration.TLS_VALIDATION_MODE_FULL - : ClientTlsConfiguration.TLS_VALIDATION_MODE_NONE - ); + if (tokenProviderSpec == null) { + return buildWebSocket(httpTokenProvider); } - - // Setting sfDir enables store-and-forward (mmap'd, recoverable - // across sender restarts); omitting it gives memory-only mode - // (same lock-free architecture, no disk involvement). - // Durability-combination validation lives in validateParameters - // so build() and no-connect validation apply the same rules. - long actualSfMaxSegmentBytes = sfMaxSegmentBytes == PARAMETER_NOT_SET_EXPLICITLY - ? DEFAULT_SEGMENT_BYTES - : sfMaxSegmentBytes; - // Default cap depends on backing: RAM (memory mode) is tight - // by default; disk (SF mode) is cheap so the default is - // generous enough that normal traffic never hits it. - long defaultMaxTotal = sfDir == null - ? DEFAULT_MAX_BYTES_MEMORY - : DEFAULT_MAX_BYTES_SF; - long actualSfMaxTotalBytes = sfMaxTotalBytes == PARAMETER_NOT_SET_EXPLICITLY - ? Math.max(defaultMaxTotal, actualSfMaxSegmentBytes * 2) - : sfMaxTotalBytes; - long actualCloseFlushTimeoutMillis = closeFlushTimeoutMillis == CLOSE_FLUSH_TIMEOUT_NOT_SET - ? DEFAULT_CLOSE_FLUSH_TIMEOUT_MILLIS - : closeFlushTimeoutMillis; - long actualReconnectMaxDurationMillis = - reconnectMaxDurationMillis == PARAMETER_NOT_SET_EXPLICITLY - ? CursorWebSocketSendLoop.DEFAULT_RECONNECT_MAX_DURATION_MILLIS - : reconnectMaxDurationMillis; - long actualReconnectInitialBackoffMillis = - reconnectInitialBackoffMillis == PARAMETER_NOT_SET_EXPLICITLY - ? CursorWebSocketSendLoop.DEFAULT_RECONNECT_INITIAL_BACKOFF_MILLIS - : reconnectInitialBackoffMillis; - long actualReconnectMaxBackoffMillis = - reconnectMaxBackoffMillis == PARAMETER_NOT_SET_EXPLICITLY - ? CursorWebSocketSendLoop.DEFAULT_RECONNECT_MAX_BACKOFF_MILLIS - : reconnectMaxBackoffMillis; - // Resolve the initial-connect mode. An explicit user choice - // (via initialConnectMode/initialConnectRetry, or the - // initial_connect_retry conf key) wins unconditionally -- - // including initial_connect_retry=off paired with a tuned - // reconnect budget. When the user left it unset and tuned - // any reconnect_* knob, promote to SYNC so the budget they - // wrote actually applies to the first connect: the knob - // name reads as a generic retry budget but the underlying - // path only governs reconnects from an established - // connection, and silently ignoring the budget on the - // initial connect is the canonical footgun this implicit - // upgrade removes. - InitialConnectMode actualInitialConnectMode; - if (initialConnectMode != null) { - actualInitialConnectMode = initialConnectMode; - } else if (reconnectMaxDurationMillis != PARAMETER_NOT_SET_EXPLICITLY - || reconnectInitialBackoffMillis != PARAMETER_NOT_SET_EXPLICITLY - || reconnectMaxBackoffMillis != PARAMETER_NOT_SET_EXPLICITLY) { - actualInitialConnectMode = InitialConnectMode.SYNC; - } else { - actualInitialConnectMode = InitialConnectMode.OFF; - } - long actualDurableAckKeepaliveIntervalMillis = - durableAckKeepaliveIntervalMillis == DURABLE_ACK_KEEPALIVE_NOT_SET - ? CursorWebSocketSendLoop.DEFAULT_DURABLE_ACK_KEEPALIVE_INTERVAL_MILLIS - : durableAckKeepaliveIntervalMillis; - int actualMaxFrameRejections = maxFrameRejections != PARAMETER_NOT_SET_EXPLICITLY - ? maxFrameRejections - : CursorWebSocketSendLoop.DEFAULT_MAX_HEAD_FRAME_REJECTIONS; - long actualPoisonMinEscalationWindowMillis = poisonMinEscalationWindowMillis != PARAMETER_NOT_SET_EXPLICITLY - ? poisonMinEscalationWindowMillis - : CursorWebSocketSendLoop.DEFAULT_POISON_MIN_ESCALATION_WINDOW_MILLIS; - long actualCatchUpCapGapMinEscalationWindowMillis = - catchUpCapGapMinEscalationWindowMillis != PARAMETER_NOT_SET_EXPLICITLY - ? catchUpCapGapMinEscalationWindowMillis - : CursorWebSocketSendLoop.DEFAULT_CATCHUP_CAP_GAP_MIN_ESCALATION_WINDOW_MILLIS; - - // sfDir is the parent (group root); the actual slot lives - // under sfDir/senderId. This is what the engine sees — the - // slot lock and segment files all live one level deeper than - // the user-supplied path. Memory mode skips this composition - // (slotPath stays null). - // - // The slot ctor inside CursorSendEngine creates the slot - // directory itself, but Files.mkdir is non-recursive — so we - // must ensure the parent group root exists first. - String slotPath; - if (sfDir == null) { - slotPath = null; - } else { - if (!Files.exists(sfDir)) { - int rc = Files.mkdir(sfDir, Files.DIR_MODE_DEFAULT); - // mkdir is non-zero on failure, but "already exists" - // is one such failure. Multiple SF senders sharing one - // sf_dir can be built concurrently (the pool calls - // build() outside its lock), so two threads can both - // pass the exists() check and race into mkdir; the - // loser gets EEXIST. Treat a benign creation race -- - // the dir now exists -- as success and only fail when - // the directory is genuinely absent afterwards. - if (rc != 0 && !Files.exists(sfDir)) { - throw new LineSenderException( - "could not create sf_dir: " + sfDir + " rc=" + rc); - } - } - if (sfDurability == SfDurability.PERIODIC - && Files.fsyncParentDir(sfDir) != 0) { - throw new LineSenderException( - "could not sync parent directory for sf_dir: " + sfDir); - } - slotPath = sfDir + "/" + senderId; - } - long actualSfAppendDeadlineNanos = - sfAppendDeadlineMillis == PARAMETER_NOT_SET_EXPLICITLY - ? CursorSendEngine.DEFAULT_APPEND_DEADLINE_NANOS - : sfAppendDeadlineMillis * 1_000_000L; - long actualSfSyncIntervalNanos = sfDurability == SfDurability.PERIODIC - ? (sfSyncIntervalMillis == PARAMETER_NOT_SET_EXPLICITLY - ? DEFAULT_SF_SYNC_INTERVAL_MILLIS : sfSyncIntervalMillis) * 1_000_000L - : 0L; - QwpWebSocketSender connected = null; - // The parent-anchored logical lock is stable across a slot rename. Keep it - // from before the directory-local lock is acquired until connect() has either - // adopted that engine or quarantine has closed, renamed and recreated it. - // This closes the inode-swap window in which an already-queued orphan drainer - // could otherwise acquire the renamed directory's old .lock and later operate - // on the fresh slot through the original pathname. - try (SlotLock logicalSlotLock = slotPath == null - ? null - : SlotLock.acquireLogical(slotPath)) { - // The constructor's own recovery seed can also fail terminally, and - // not only as UnreplayableSlotException: when SegmentRing.openExisting - // had to skip an unreadable segment it throws SfRecoveryException (it - // constructs UnreplayableSlotException nowhere), and where it cannot - // even prove the chain's identity -- no manifest -- it quarantines the - // corrupt files and returns an EMPTY recovery rather than refusing. - // Either way the frame range cannot be shown already-acked, so recovery - // sets the slot aside rather than risk seeding the ack cursor past - // frames that were never delivered. All three types below are load - // bearing; narrowing this catch to UnreplayableSlotException would - // restore the permanent build() brick for the segment-skip case. That verdict gets - // the exact same quarantine-and-continue treatment as the connect()-time - // verdict below -- constructing cursorEngine is not inside the loop below, - // so a throw here would otherwise escape build() entirely, uncaught. - // quarantineTornSlot(null, ...) renames the WHOLE slot directory aside - // (not just the unreadable segment file) before building the replacement - // at the original slotPath, so the replacement starts on a genuinely empty - // directory with nothing left to skip -- it cannot throw the same way - // twice, which is what makes looping unnecessary here. - boolean quarantined = false; - CursorSendEngine cursorEngine; - try { - try { - cursorEngine = new CursorSendEngine( - slotPath, actualSfMaxSegmentBytes, - actualSfMaxTotalBytes, actualSfAppendDeadlineNanos, - actualSfSyncIntervalNanos); - } catch (SfSanitizedResidueException first) { - // NOT terminal, and it must be intercepted ahead of its - // SfRecoveryException parent below. Recovery durably zeroed - // proven-dead sealed residue BEFORE failing closed, so the - // chain on disk is already healed: quarantining here would - // set aside a slot whose backlog replays perfectly. Retry - // once over the healed chain; a repeat is genuine and takes - // the terminal arm. - LOG.info("sf slot {}: sealed residue sanitized during recovery ({}); " - + "retrying over the healed chain", - slotPath, first.getMessage()); - cursorEngine = new CursorSendEngine( - slotPath, actualSfMaxSegmentBytes, - actualSfMaxTotalBytes, actualSfAppendDeadlineNanos, - actualSfSyncIntervalNanos); - } - } catch (UnreplayableSlotException | SfRecoveryException - | MmapSegmentCorruptionException e) { - // The terminal recovery verdicts, and the only ones build() - // sets a slot aside for. UnreplayableSlotException says the - // symbol dictionary cannot be rebuilt from any source; - // SfRecoveryException and MmapSegmentCorruptionException say - // the durable chain itself is proven corrupt or incomplete. - // None of the three clears on a retry, and senderId is stable - // with a not-fully-drained slot retained on close -- so - // without this arm every restart re-recovers the same slot and - // throws again, and the application cannot construct a Sender - // at all, not even to BUFFER new rows. - // - // Deliberately NOT catching plain MmapSegmentException or - // SfOperationalException: those are operational (EMFILE, - // ENOMEM, an unreadable-but-possibly-intact file). Aborting - // startup on them is correct; quarantining on them would - // convert a transient into the permanent loss of a healthy - // slot's durable frames. - if (slotPath == null) { - throw e; - } - quarantined = true; - cursorEngine = quarantineTornSlot( - null, e, sfDir, senderId, slotPath, actualSfMaxSegmentBytes, - actualSfMaxTotalBytes, actualSfAppendDeadlineNanos, - actualSfSyncIntervalNanos, errorHandler); - } - int actualErrorInboxCapacity = errorInboxCapacity != PARAMETER_NOT_SET_EXPLICITLY - ? errorInboxCapacity - : io.questdb.client.cutlass.qwp.client.sf.cursor.SenderErrorDispatcher.DEFAULT_CAPACITY; - int actualConnectionListenerInboxCapacity = connectionListenerInboxCapacity != PARAMETER_NOT_SET_EXPLICITLY - ? connectionListenerInboxCapacity - : io.questdb.client.cutlass.qwp.client.sf.cursor.SenderConnectionDispatcher.DEFAULT_CAPACITY; - List wsEndpoints = - new ArrayList<>(hosts.size()); - for (int i = 0, n = hosts.size(); i < n; i++) { - wsEndpoints.add(new QwpWebSocketSender.Endpoint(hosts.getQuick(i), ports.getQuick(i))); - } - // The recovery seed inside connect() is the authority on whether a recovered - // slot can be replayed: it rebuilds the dictionary from its intact prefix and - // then from the surviving frames' own delta sections, and throws - // UnreplayableSlotException only once neither source holds the missing ids. - // Quarantining on anything weaker would set aside slots that recovery can - // still rescue, so build() waits for that verdict rather than pre-judging it. - while (connected == null) { - try { - connected = QwpWebSocketSender.connectWithCredentialSupplier( - wsEndpoints, - wsTlsConfig, - actualAutoFlushRows, - actualAutoFlushBytes, - actualAutoFlushIntervalNanos, - wsAuthHeader, - requestDurableAck, - cursorEngine, - actualCloseFlushTimeoutMillis, - actualReconnectMaxDurationMillis, - actualReconnectInitialBackoffMillis, - actualReconnectMaxBackoffMillis, - actualInitialConnectMode, - errorHandler, - actualErrorInboxCapacity, - actualDurableAckKeepaliveIntervalMillis, - authTimeoutMillis, - connectTimeoutMillis == PARAMETER_NOT_SET_EXPLICITLY ? 0 : connectTimeoutMillis, - connectionListener, - actualConnectionListenerInboxCapacity, - actualMaxFrameRejections, - actualPoisonMinEscalationWindowMillis, - actualCatchUpCapGapMinEscalationWindowMillis - ); - } catch (UnreplayableSlotException e) { - // The one failure build() recovers from. The slot's frames reference ids - // that nothing still holds, so they can never go on the wire -- but that is - // no reason to take the producer down with them. Before this, the throw - // escaped build() and, because senderId is stable and a not-fully-drained - // slot is retained on close, every retry re-recovered the same slot and - // threw again: the application could not construct a Sender at all, so it - // could not even BUFFER new rows. An already-lost batch became an unbounded - // outage of everything after it. - // - // Set the slot aside instead, keep its bytes for forensics and resend, and - // start the producer on a clean one. Once only: a second such failure would - // mean the FRESH slot is unreplayable, which cannot happen, so let it out - // rather than loop. - if (quarantined || slotPath == null) { - try { - // close(false): we still hold the logical slot lock. - cursorEngine.close(false); - } catch (Throwable ignored) { - // best-effort - } - throw e; - } - quarantined = true; - cursorEngine = quarantineTornSlot( - cursorEngine, e, sfDir, senderId, slotPath, actualSfMaxSegmentBytes, - actualSfMaxTotalBytes, actualSfAppendDeadlineNanos, - actualSfSyncIntervalNanos, errorHandler); - } catch (Throwable t) { - // connect() failed before ownership of cursorEngine - // transferred — close it ourselves. close(false) - // because logicalSlotLock is still held here: a fresh - // slot is fully drained, so the default close would - // unlink the very lock file this scope holds. - try { - cursorEngine.close(false); - } catch (Throwable ignored) { - // best-effort - } - throw t; - } - } - } - // connect() succeeded — `connected` now owns cursorEngine - // via setCursorEngine(engine, true). From here on, ANY - // failure must close `connected` (which closes the engine - // through ownsCursorEngine), not cursorEngine directly: - // closing the engine alone would leak the I/O thread, - // dispatcher daemon, drainer pool, microbatch buffers and - // WebSocketClient inside the abandoned `connected`. - connected.setTransactional(transactional); + // token_provider: share the process-wide provider for this configuration. The lease is the + // sender's from here on: released when the sender closes, or right here if build() fails. + TokenProviderRegistry.Lease lease = TokenProviderRegistry.global().acquire(tokenProviderSpec); try { - // Install the drainer listener BEFORE startOrphanDrainers - // below: drainers must see the listener at submit time so - // no early drainer event is lost to a late installation. - if (drainerListener != null) { - connected.setDrainerListener(drainerListener); - } - // Once the foreground sender is up, dispatch drainers - // for any sibling orphan slots. Scan AFTER we acquire - // our own slot lock so we never accidentally try to - // adopt our own data; the OrphanScanner.scan filter - // also excludes our sender_id. - if (drainOrphans && sfDir != null) { - io.questdb.client.std.ObjList orphans = - io.questdb.client.cutlass.qwp.client.sf.cursor.OrphanScanner - .scan(sfDir, senderId, orphanDrainBase, orphanDrainSlotCount); - if (orphans.size() > 0) { - org.slf4j.LoggerFactory.getLogger(LineSenderBuilder.class) - .info("dispatching drainers for {} orphan slot(s) under {} " - + "(max_background_drainers={})", - orphans.size(), sfDir, maxBackgroundDrainers); - connected.startOrphanDrainers( - orphans, - maxBackgroundDrainers, - actualSfMaxSegmentBytes, - actualSfMaxTotalBytes, - actualSfSyncIntervalNanos); - } - } - return connected; - } catch (Throwable t) { - try { - connected.close(); - } catch (Throwable ignored) { - // best-effort + QwpWebSocketSender sender = buildWebSocket(lease.provider()); + sender.setCredentialLease(lease); + lease = null; + return sender; + } finally { + if (lease != null) { + lease.close(); } - throw t; } } @@ -2282,6 +2002,9 @@ public LineSenderBuilder httpTimeoutMillis(int httpTimeoutMillis) { * @return this instance for method chaining */ public LineSenderBuilder httpToken(String token) { + if (this.tokenProviderSpec != null) { + throw new LineSenderException("token cannot be combined with token_provider"); + } if (this.username != null) { throw new LineSenderException("authentication username was already configured ") .put("[username=").put(this.username).put("]"); @@ -2341,6 +2064,10 @@ public LineSenderBuilder httpToken(String token) { * @return this instance for method chaining */ public LineSenderBuilder httpTokenProvider(HttpTokenProvider httpTokenProvider) { + if (this.tokenProviderSpec != null) { + throw new LineSenderException("an application-supplied token provider cannot be combined with " + + "token_provider in the configuration"); + } if (this.username != null) { throw new LineSenderException("authentication username was already configured ") .put("[username=").put(this.username).put("]"); @@ -2370,6 +2097,9 @@ public LineSenderBuilder httpTokenProvider(HttpTokenProvider httpTokenProvider) * @see #httpToken(String) */ public LineSenderBuilder httpUsernamePassword(String username, String password) { + if (this.tokenProviderSpec != null) { + throw new LineSenderException("username/password cannot be combined with token_provider"); + } if (this.username != null) { throw new LineSenderException("authentication username was already configured ") .put("[username=").put(this.username).put("]"); @@ -3424,7 +3154,371 @@ private void appendAddress(String host, int port) { ports.add(port); } - private Supplier buildWebSocketAuthHeader() { + private QwpWebSocketSender buildWebSocket(HttpTokenProvider tokenProvider) { + if (hosts.size() < 1) { + throw new LineSenderException("WebSocket transport requires at least one host:port pair"); + } + if (tokenProvider != null && !tlsEnabled && CLEARTEXT_PROVIDER_WARNED.compareAndSet(false, true)) { + // design/qwp-token-provider-spec.md, section 9: a configured token_provider is rejected on ws::, + // but an application-supplied provider is the application's call - warn once instead. + LOG.warn("a token provider is used over ws:: (no TLS): bearer tokens cross the network in " + + "cleartext; use wss:: in production"); + } + + int actualAutoFlushRows = autoFlushRows == PARAMETER_NOT_SET_EXPLICITLY ? DEFAULT_WS_AUTO_FLUSH_ROWS : autoFlushRows; + int actualAutoFlushBytes = autoFlushBytes == PARAMETER_NOT_SET_EXPLICITLY ? DEFAULT_WS_AUTO_FLUSH_BYTES : autoFlushBytes; + long actualAutoFlushIntervalNanos = autoFlushIntervalMillis == PARAMETER_NOT_SET_EXPLICITLY + ? DEFAULT_WS_AUTO_FLUSH_INTERVAL_NANOS + : TimeUnit.MILLISECONDS.toNanos(autoFlushIntervalMillis); + + Supplier wsAuthHeader = buildWebSocketAuthHeader(tokenProvider); + + ClientTlsConfiguration wsTlsConfig = null; + if (tlsEnabled) { + assert trustStorePassword == null || trustStorePath != null; + wsTlsConfig = new ClientTlsConfiguration( + trustStorePath, + trustStorePassword, + tlsValidationMode == TlsValidationMode.DEFAULT + ? ClientTlsConfiguration.TLS_VALIDATION_MODE_FULL + : ClientTlsConfiguration.TLS_VALIDATION_MODE_NONE + ); + } + + // Setting sfDir enables store-and-forward (mmap'd, recoverable + // across sender restarts); omitting it gives memory-only mode + // (same lock-free architecture, no disk involvement). + // Durability-combination validation lives in validateParameters + // so build() and no-connect validation apply the same rules. + long actualSfMaxSegmentBytes = sfMaxSegmentBytes == PARAMETER_NOT_SET_EXPLICITLY + ? DEFAULT_SEGMENT_BYTES + : sfMaxSegmentBytes; + // Default cap depends on backing: RAM (memory mode) is tight + // by default; disk (SF mode) is cheap so the default is + // generous enough that normal traffic never hits it. + long defaultMaxTotal = sfDir == null + ? DEFAULT_MAX_BYTES_MEMORY + : DEFAULT_MAX_BYTES_SF; + long actualSfMaxTotalBytes = sfMaxTotalBytes == PARAMETER_NOT_SET_EXPLICITLY + ? Math.max(defaultMaxTotal, actualSfMaxSegmentBytes * 2) + : sfMaxTotalBytes; + long actualCloseFlushTimeoutMillis = closeFlushTimeoutMillis == CLOSE_FLUSH_TIMEOUT_NOT_SET + ? DEFAULT_CLOSE_FLUSH_TIMEOUT_MILLIS + : closeFlushTimeoutMillis; + long actualReconnectMaxDurationMillis = + reconnectMaxDurationMillis == PARAMETER_NOT_SET_EXPLICITLY + ? CursorWebSocketSendLoop.DEFAULT_RECONNECT_MAX_DURATION_MILLIS + : reconnectMaxDurationMillis; + long actualReconnectInitialBackoffMillis = + reconnectInitialBackoffMillis == PARAMETER_NOT_SET_EXPLICITLY + ? CursorWebSocketSendLoop.DEFAULT_RECONNECT_INITIAL_BACKOFF_MILLIS + : reconnectInitialBackoffMillis; + long actualReconnectMaxBackoffMillis = + reconnectMaxBackoffMillis == PARAMETER_NOT_SET_EXPLICITLY + ? CursorWebSocketSendLoop.DEFAULT_RECONNECT_MAX_BACKOFF_MILLIS + : reconnectMaxBackoffMillis; + // Resolve the initial-connect mode. An explicit user choice + // (via initialConnectMode/initialConnectRetry, or the + // initial_connect_retry conf key) wins unconditionally -- + // including initial_connect_retry=off paired with a tuned + // reconnect budget. When the user left it unset and tuned + // any reconnect_* knob, promote to SYNC so the budget they + // wrote actually applies to the first connect: the knob + // name reads as a generic retry budget but the underlying + // path only governs reconnects from an established + // connection, and silently ignoring the budget on the + // initial connect is the canonical footgun this implicit + // upgrade removes. + InitialConnectMode actualInitialConnectMode; + if (initialConnectMode != null) { + actualInitialConnectMode = initialConnectMode; + } else if (reconnectMaxDurationMillis != PARAMETER_NOT_SET_EXPLICITLY + || reconnectInitialBackoffMillis != PARAMETER_NOT_SET_EXPLICITLY + || reconnectMaxBackoffMillis != PARAMETER_NOT_SET_EXPLICITLY) { + actualInitialConnectMode = InitialConnectMode.SYNC; + } else { + actualInitialConnectMode = InitialConnectMode.OFF; + } + long actualDurableAckKeepaliveIntervalMillis = + durableAckKeepaliveIntervalMillis == DURABLE_ACK_KEEPALIVE_NOT_SET + ? CursorWebSocketSendLoop.DEFAULT_DURABLE_ACK_KEEPALIVE_INTERVAL_MILLIS + : durableAckKeepaliveIntervalMillis; + int actualMaxFrameRejections = maxFrameRejections != PARAMETER_NOT_SET_EXPLICITLY + ? maxFrameRejections + : CursorWebSocketSendLoop.DEFAULT_MAX_HEAD_FRAME_REJECTIONS; + long actualPoisonMinEscalationWindowMillis = poisonMinEscalationWindowMillis != PARAMETER_NOT_SET_EXPLICITLY + ? poisonMinEscalationWindowMillis + : CursorWebSocketSendLoop.DEFAULT_POISON_MIN_ESCALATION_WINDOW_MILLIS; + long actualCatchUpCapGapMinEscalationWindowMillis = + catchUpCapGapMinEscalationWindowMillis != PARAMETER_NOT_SET_EXPLICITLY + ? catchUpCapGapMinEscalationWindowMillis + : CursorWebSocketSendLoop.DEFAULT_CATCHUP_CAP_GAP_MIN_ESCALATION_WINDOW_MILLIS; + + // sfDir is the parent (group root); the actual slot lives + // under sfDir/senderId. This is what the engine sees — the + // slot lock and segment files all live one level deeper than + // the user-supplied path. Memory mode skips this composition + // (slotPath stays null). + // + // The slot ctor inside CursorSendEngine creates the slot + // directory itself, but Files.mkdir is non-recursive — so we + // must ensure the parent group root exists first. + String slotPath; + if (sfDir == null) { + slotPath = null; + } else { + if (!Files.exists(sfDir)) { + int rc = Files.mkdir(sfDir, Files.DIR_MODE_DEFAULT); + // mkdir is non-zero on failure, but "already exists" + // is one such failure. Multiple SF senders sharing one + // sf_dir can be built concurrently (the pool calls + // build() outside its lock), so two threads can both + // pass the exists() check and race into mkdir; the + // loser gets EEXIST. Treat a benign creation race -- + // the dir now exists -- as success and only fail when + // the directory is genuinely absent afterwards. + if (rc != 0 && !Files.exists(sfDir)) { + throw new LineSenderException( + "could not create sf_dir: " + sfDir + " rc=" + rc); + } + } + if (sfDurability == SfDurability.PERIODIC + && Files.fsyncParentDir(sfDir) != 0) { + throw new LineSenderException( + "could not sync parent directory for sf_dir: " + sfDir); + } + slotPath = sfDir + "/" + senderId; + } + long actualSfAppendDeadlineNanos = + sfAppendDeadlineMillis == PARAMETER_NOT_SET_EXPLICITLY + ? CursorSendEngine.DEFAULT_APPEND_DEADLINE_NANOS + : sfAppendDeadlineMillis * 1_000_000L; + long actualSfSyncIntervalNanos = sfDurability == SfDurability.PERIODIC + ? (sfSyncIntervalMillis == PARAMETER_NOT_SET_EXPLICITLY + ? DEFAULT_SF_SYNC_INTERVAL_MILLIS : sfSyncIntervalMillis) * 1_000_000L + : 0L; + QwpWebSocketSender connected = null; + // The parent-anchored logical lock is stable across a slot rename. Keep it + // from before the directory-local lock is acquired until connect() has either + // adopted that engine or quarantine has closed, renamed and recreated it. + // This closes the inode-swap window in which an already-queued orphan drainer + // could otherwise acquire the renamed directory's old .lock and later operate + // on the fresh slot through the original pathname. + try (SlotLock logicalSlotLock = slotPath == null + ? null + : SlotLock.acquireLogical(slotPath)) { + // The constructor's own recovery seed can also fail terminally, and + // not only as UnreplayableSlotException: when SegmentRing.openExisting + // had to skip an unreadable segment it throws SfRecoveryException (it + // constructs UnreplayableSlotException nowhere), and where it cannot + // even prove the chain's identity -- no manifest -- it quarantines the + // corrupt files and returns an EMPTY recovery rather than refusing. + // Either way the frame range cannot be shown already-acked, so recovery + // sets the slot aside rather than risk seeding the ack cursor past + // frames that were never delivered. All three types below are load + // bearing; narrowing this catch to UnreplayableSlotException would + // restore the permanent build() brick for the segment-skip case. That verdict gets + // the exact same quarantine-and-continue treatment as the connect()-time + // verdict below -- constructing cursorEngine is not inside the loop below, + // so a throw here would otherwise escape build() entirely, uncaught. + // quarantineTornSlot(null, ...) renames the WHOLE slot directory aside + // (not just the unreadable segment file) before building the replacement + // at the original slotPath, so the replacement starts on a genuinely empty + // directory with nothing left to skip -- it cannot throw the same way + // twice, which is what makes looping unnecessary here. + boolean quarantined = false; + CursorSendEngine cursorEngine; + try { + try { + cursorEngine = new CursorSendEngine( + slotPath, actualSfMaxSegmentBytes, + actualSfMaxTotalBytes, actualSfAppendDeadlineNanos, + actualSfSyncIntervalNanos); + } catch (SfSanitizedResidueException first) { + // NOT terminal, and it must be intercepted ahead of its + // SfRecoveryException parent below. Recovery durably zeroed + // proven-dead sealed residue BEFORE failing closed, so the + // chain on disk is already healed: quarantining here would + // set aside a slot whose backlog replays perfectly. Retry + // once over the healed chain; a repeat is genuine and takes + // the terminal arm. + LOG.info("sf slot {}: sealed residue sanitized during recovery ({}); " + + "retrying over the healed chain", + slotPath, first.getMessage()); + cursorEngine = new CursorSendEngine( + slotPath, actualSfMaxSegmentBytes, + actualSfMaxTotalBytes, actualSfAppendDeadlineNanos, + actualSfSyncIntervalNanos); + } + } catch (UnreplayableSlotException | SfRecoveryException + | MmapSegmentCorruptionException e) { + // The terminal recovery verdicts, and the only ones build() + // sets a slot aside for. UnreplayableSlotException says the + // symbol dictionary cannot be rebuilt from any source; + // SfRecoveryException and MmapSegmentCorruptionException say + // the durable chain itself is proven corrupt or incomplete. + // None of the three clears on a retry, and senderId is stable + // with a not-fully-drained slot retained on close -- so + // without this arm every restart re-recovers the same slot and + // throws again, and the application cannot construct a Sender + // at all, not even to BUFFER new rows. + // + // Deliberately NOT catching plain MmapSegmentException or + // SfOperationalException: those are operational (EMFILE, + // ENOMEM, an unreadable-but-possibly-intact file). Aborting + // startup on them is correct; quarantining on them would + // convert a transient into the permanent loss of a healthy + // slot's durable frames. + if (slotPath == null) { + throw e; + } + quarantined = true; + cursorEngine = quarantineTornSlot( + null, e, sfDir, senderId, slotPath, actualSfMaxSegmentBytes, + actualSfMaxTotalBytes, actualSfAppendDeadlineNanos, + actualSfSyncIntervalNanos, errorHandler); + } + int actualErrorInboxCapacity = errorInboxCapacity != PARAMETER_NOT_SET_EXPLICITLY + ? errorInboxCapacity + : io.questdb.client.cutlass.qwp.client.sf.cursor.SenderErrorDispatcher.DEFAULT_CAPACITY; + int actualConnectionListenerInboxCapacity = connectionListenerInboxCapacity != PARAMETER_NOT_SET_EXPLICITLY + ? connectionListenerInboxCapacity + : io.questdb.client.cutlass.qwp.client.sf.cursor.SenderConnectionDispatcher.DEFAULT_CAPACITY; + List wsEndpoints = + new ArrayList<>(hosts.size()); + for (int i = 0, n = hosts.size(); i < n; i++) { + wsEndpoints.add(new QwpWebSocketSender.Endpoint(hosts.getQuick(i), ports.getQuick(i))); + } + // The recovery seed inside connect() is the authority on whether a recovered + // slot can be replayed: it rebuilds the dictionary from its intact prefix and + // then from the surviving frames' own delta sections, and throws + // UnreplayableSlotException only once neither source holds the missing ids. + // Quarantining on anything weaker would set aside slots that recovery can + // still rescue, so build() waits for that verdict rather than pre-judging it. + while (connected == null) { + try { + connected = QwpWebSocketSender.connectWithCredentialSupplier( + wsEndpoints, + wsTlsConfig, + actualAutoFlushRows, + actualAutoFlushBytes, + actualAutoFlushIntervalNanos, + wsAuthHeader, + requestDurableAck, + cursorEngine, + actualCloseFlushTimeoutMillis, + actualReconnectMaxDurationMillis, + actualReconnectInitialBackoffMillis, + actualReconnectMaxBackoffMillis, + actualInitialConnectMode, + errorHandler, + actualErrorInboxCapacity, + actualDurableAckKeepaliveIntervalMillis, + authTimeoutMillis, + connectTimeoutMillis == PARAMETER_NOT_SET_EXPLICITLY ? 0 : connectTimeoutMillis, + connectionListener, + actualConnectionListenerInboxCapacity, + actualMaxFrameRejections, + actualPoisonMinEscalationWindowMillis, + actualCatchUpCapGapMinEscalationWindowMillis + ); + } catch (UnreplayableSlotException e) { + // The one failure build() recovers from. The slot's frames reference ids + // that nothing still holds, so they can never go on the wire -- but that is + // no reason to take the producer down with them. Before this, the throw + // escaped build() and, because senderId is stable and a not-fully-drained + // slot is retained on close, every retry re-recovered the same slot and + // threw again: the application could not construct a Sender at all, so it + // could not even BUFFER new rows. An already-lost batch became an unbounded + // outage of everything after it. + // + // Set the slot aside instead, keep its bytes for forensics and resend, and + // start the producer on a clean one. Once only: a second such failure would + // mean the FRESH slot is unreplayable, which cannot happen, so let it out + // rather than loop. + if (quarantined || slotPath == null) { + try { + // close(false): we still hold the logical slot lock. + cursorEngine.close(false); + } catch (Throwable ignored) { + // best-effort + } + throw e; + } + quarantined = true; + cursorEngine = quarantineTornSlot( + cursorEngine, e, sfDir, senderId, slotPath, actualSfMaxSegmentBytes, + actualSfMaxTotalBytes, actualSfAppendDeadlineNanos, + actualSfSyncIntervalNanos, errorHandler); + } catch (Throwable t) { + // connect() failed before ownership of cursorEngine + // transferred — close it ourselves. close(false) + // because logicalSlotLock is still held here: a fresh + // slot is fully drained, so the default close would + // unlink the very lock file this scope holds. + try { + cursorEngine.close(false); + } catch (Throwable ignored) { + // best-effort + } + throw t; + } + } + } + // connect() succeeded — `connected` now owns cursorEngine + // via setCursorEngine(engine, true). From here on, ANY + // failure must close `connected` (which closes the engine + // through ownsCursorEngine), not cursorEngine directly: + // closing the engine alone would leak the I/O thread, + // dispatcher daemon, drainer pool, microbatch buffers and + // WebSocketClient inside the abandoned `connected`. + connected.setTransactional(transactional); + if (authFailureMaxDurationMillis > 0) { + // The I/O loop may already be running (async initial connect), but it tracks the outage clock + // from the first authentication-class failure regardless; this only arms the deadline. + connected.setAuthFailureMaxDurationMillis(authFailureMaxDurationMillis); + } + try { + // Install the drainer listener BEFORE startOrphanDrainers + // below: drainers must see the listener at submit time so + // no early drainer event is lost to a late installation. + if (drainerListener != null) { + connected.setDrainerListener(drainerListener); + } + // Once the foreground sender is up, dispatch drainers + // for any sibling orphan slots. Scan AFTER we acquire + // our own slot lock so we never accidentally try to + // adopt our own data; the OrphanScanner.scan filter + // also excludes our sender_id. + if (drainOrphans && sfDir != null) { + io.questdb.client.std.ObjList orphans = + io.questdb.client.cutlass.qwp.client.sf.cursor.OrphanScanner + .scan(sfDir, senderId, orphanDrainBase, orphanDrainSlotCount); + if (orphans.size() > 0) { + org.slf4j.LoggerFactory.getLogger(LineSenderBuilder.class) + .info("dispatching drainers for {} orphan slot(s) under {} " + + "(max_background_drainers={})", + orphans.size(), sfDir, maxBackgroundDrainers); + connected.startOrphanDrainers( + orphans, + maxBackgroundDrainers, + actualSfMaxSegmentBytes, + actualSfMaxTotalBytes, + actualSfSyncIntervalNanos); + } + } + return connected; + } catch (Throwable t) { + try { + connected.close(); + } catch (Throwable ignored) { + // best-effort + } + throw t; + } + } + + private Supplier buildWebSocketAuthHeader(HttpTokenProvider provider) { // A constant credential goes through fixedAuthHeader, not a bare lambda: the tag is what lets // the store-and-forward drainer tell a permanently-wrong password from a rotating token that a // fresh pull can repair, and so decide whether a 401 may quarantine an orphan slot for good. @@ -3437,21 +3531,13 @@ private Supplier buildWebSocketAuthHeader() { String header = "Bearer " + httpToken; return QwpWebSocketSender.fixedAuthHeader(header); } - if (httpTokenProvider != null) { - // pull a fresh token at each (re)handshake so a long-lived WebSocket follows token - // refreshes; validateToken rejects a null/empty/blank return, or a token carrying a - // control or non-ASCII char (both forbidden by the HttpTokenProvider contract), rather - // than send a malformed or CR/LF-injected "Bearer " header - final HttpTokenProvider provider = httpTokenProvider; - return () -> { - // snapshot before validating: the concatenation below re-reads the sequence, and a - // provider is free to reuse a mutable buffer, so validating the live sequence checks - // bytes the header need not carry. See HttpTokenProvider.validateToken. - CharSequence pulled = provider.getToken(); - CharSequence token = pulled == null ? null : pulled.toString(); - HttpTokenProvider.validateToken(token); - return "Bearer " + token; - }; + if (provider != null) { + // Pull a fresh token at each (re)handshake so a long-lived WebSocket follows token refreshes. + // The supplier snapshots and validates every pull (validateToken rejects a null/empty/blank + // return, or a token carrying a control or non-ASCII char, rather than send a malformed or + // CR/LF-injected "Bearer " header), and it carries the provider's onTokenRejected back-channel + // that the connect walk uses for its one retry after a 401. + return QwpWebSocketSender.tokenProviderAuthHeader(provider); } return null; } @@ -3973,6 +4059,14 @@ private LineSenderBuilder fromConfig(CharSequence configurationString) { // genuine value-parse error names the offending key. String reservedKey = Chars.toString(sink); pos = getValue(configurationString, pos, sink, reservedKey); + } else if (Chars.equals("auth_failure_max_duration_millis", sink)) { + throw new LineSenderException("auth_failure_max_duration_millis is only supported for WebSocket transport"); + } else if (Chars.equals("token_provider", sink) + || Chars.equals("azure_resource", sink) + || Chars.equals("azure_client_id", sink)) { + // Dynamic bearer credentials are defined for QWP over wss:: only (decision D9). + throw new LineSenderException(Chars.toString(sink) + + " is only supported with the wss:: schema (QWP over WebSocket)"); } else { // sf-client.md §4.6: parser must reject unknown keys. // Forward-compat is via the spec, not silent ignore — silent @@ -4020,6 +4114,16 @@ private LineSenderBuilder fromConfigWebSocket(CharSequence configurationString) ConfigString cs = ConfigString.parse(configurationString); ConfigView view = new ConfigView(cs); validateWsConfig(view, tlsEnabled); + // Validates token_provider and its keys (wss:: only, exclusive with static credentials, a + // supported provider) without fetching anything; build() acquires the provider. + TokenProviderSpec spec = TokenProviderSpec.parse(view, tlsEnabled); + if (spec != null) { + if (httpTokenProvider != null) { + throw new LineSenderException("token_provider cannot be combined with an " + + "application-supplied token provider"); + } + tokenProviderSpec = spec; + } view.getHostPorts("addr", DEFAULT_WEBSOCKET_PORT, this::appendAddress); @@ -4055,6 +4159,9 @@ private LineSenderBuilder fromConfigWebSocket(CharSequence configurationString) // int, and an over-int value must reject, not wrap. connectTimeoutMillis(view.getInt("connect_timeout", 0)); } + if (view.has("auth_failure_max_duration_millis")) { + authFailureMaxDurationMillis(view.getLong("auth_failure_max_duration_millis", 0)); + } s = view.getStr("auto_flush_rows"); if (s != null) { @@ -4335,6 +4442,12 @@ public java.util.Map wsConfigSnapshotForTest() { m.put("tls_verify", tlsValidationMode == null ? null : tlsValidationMode.name()); m.put("tls_roots", trustStorePath); m.put("tls_roots_password", trustStorePassword == null ? null : new String(trustStorePassword)); + m.put("auth_failure_max_duration_millis", authFailureMaxDurationMillis); + m.put("token_provider", tokenProviderSpec == null ? null : tokenProviderSpec.name()); + m.put("azure_resource", tokenProviderSpec == null ? null + : tokenProviderSpec.params().get(TokenProviderSpec.KEY_AZURE_RESOURCE)); + m.put("azure_client_id", tokenProviderSpec == null ? null + : tokenProviderSpec.params().get(TokenProviderSpec.KEY_AZURE_CLIENT_ID)); return m; } diff --git a/core/src/main/java/io/questdb/client/cutlass/auth/CredentialRedaction.java b/core/src/main/java/io/questdb/client/cutlass/auth/CredentialRedaction.java new file mode 100644 index 000000000..a84be9fe4 --- /dev/null +++ b/core/src/main/java/io/questdb/client/cutlass/auth/CredentialRedaction.java @@ -0,0 +1,145 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.cutlass.auth; + +import io.questdb.client.std.str.DisplaySafe; + +import java.nio.charset.StandardCharsets; +import java.security.MessageDigest; +import java.security.NoSuchAlgorithmException; + +/** + * Redaction rules for bearer credentials and for error text that crosses the client boundary, as required by + * the dynamic-credential specification (design/qwp-token-provider-spec.md, section 9). + *

    + *
  • A token is never rendered. {@link #describeToken(CharSequence)} gives its length and at most the first + * 8 hex digits of its SHA-256 - enough to correlate a rotation across log lines, useless to replay.
    • + *
  • Error text that came from a library or an identity endpoint goes through + * {@link #sanitizeErrorText(CharSequence)} before it reaches a client error or a log line: control, + * bidirectional and other non-displayable characters are removed, and the result is capped at + * {@value #MAX_ERROR_TEXT_LENGTH} characters.
    • + *
+ */ +public final class CredentialRedaction { + /** + * Upper bound, in characters, on error text from a library or endpoint once sanitized. + */ + public static final int MAX_ERROR_TEXT_LENGTH = 256; + private static final int FINGERPRINT_HEX_DIGITS = 8; + private static final char[] HEX = "0123456789abcdef".toCharArray(); + private static final String TRUNCATION_MARKER = "..."; + + private CredentialRedaction() { + } + + /** + * Renders a token as {@code }: its length plus the first 8 hex digits of + * its SHA-256. Never the token itself. + * + * @param token the token, may be null + * @return a description that is safe to log + */ + public static String describeToken(CharSequence token) { + if (token == null) { + return ""; + } + return "'; + } + + /** + * The first 8 hex digits of the SHA-256 of {@code token}, encoded as UTF-8. + * + * @param token the token, never null + * @return 8 lowercase hex digits + */ + public static String fingerprint(CharSequence token) { + final byte[] digest; + try { + digest = MessageDigest.getInstance("SHA-256").digest(token.toString().getBytes(StandardCharsets.UTF_8)); + } catch (NoSuchAlgorithmException e) { + // every Java platform is required to provide SHA-256 + throw new IllegalStateException("SHA-256 is not available", e); + } + final char[] out = new char[FINGERPRINT_HEX_DIGITS]; + for (int i = 0; i < FINGERPRINT_HEX_DIGITS / 2; i++) { + out[2 * i] = HEX[(digest[i] >> 4) & 0xf]; + out[2 * i + 1] = HEX[digest[i] & 0xf]; + } + return new String(out); + } + + /** + * Prepares untrusted error text - from a token source, an identity library or an endpoint - for a client + * error or a log line: removes every code point {@link DisplaySafe} rejects (controls including CR/LF, + * bidirectional overrides, zero-width and other format characters, lone surrogates) and caps the result at + * {@value #MAX_ERROR_TEXT_LENGTH} characters, ending in {@code ...} when it had to cut. + *

+ * This does not, and cannot, remove a secret the text already carries. Keeping tokens and response bodies + * out of error text is the job of whoever produces it. + * + * @param text untrusted text, may be null + * @return the sanitized text, or {@code null} when {@code text} is null + */ + public static String sanitizeErrorText(CharSequence text) { + if (text == null) { + return null; + } + final StringBuilder sb = new StringBuilder(Math.min(text.length(), MAX_ERROR_TEXT_LENGTH)); + final int limit = MAX_ERROR_TEXT_LENGTH - TRUNCATION_MARKER.length(); + for (int i = 0, n = text.length(); i < n; ) { + final int cp = Character.codePointAt(text, i); + final int count = Character.charCount(cp); + if (DisplaySafe.isDisplaySafe(cp)) { + if (sb.length() + count > limit) { + // Only cut when something displayable is left to drop: text that fits exactly is kept whole. + if (hasDisplayableFrom(text, i, MAX_ERROR_TEXT_LENGTH - sb.length())) { + sb.append(TRUNCATION_MARKER); + return sb.toString(); + } + } + sb.appendCodePoint(cp); + } + i += count; + } + return sb.toString(); + } + + // True when the displayable remainder of text, starting at from, does not fit in budget characters. + private static boolean hasDisplayableFrom(CharSequence text, int from, int budget) { + int needed = 0; + for (int i = from, n = text.length(); i < n; ) { + final int cp = Character.codePointAt(text, i); + final int count = Character.charCount(cp); + if (DisplaySafe.isDisplaySafe(cp)) { + needed += count; + if (needed > budget) { + return true; + } + } + i += count; + } + return false; + } +} diff --git a/core/src/main/java/io/questdb/client/cutlass/auth/ExpiringToken.java b/core/src/main/java/io/questdb/client/cutlass/auth/ExpiringToken.java new file mode 100644 index 000000000..95713352b --- /dev/null +++ b/core/src/main/java/io/questdb/client/cutlass/auth/ExpiringToken.java @@ -0,0 +1,148 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.cutlass.auth; + +import java.time.Instant; + +/** + * A bearer token together with the instant it expires, as returned by a {@link TokenSource}. This is the + * {@code TokenResult} of the dynamic-credential specification (design/qwp-token-provider-spec.md, section 4). + *

+ * The token is opaque and is stored without the {@code "Bearer "} prefix. It must be non-blank printable ASCII + * ({@code 0x20}-{@code 0x7e}); the constructors reject anything else, never echoing the token in the message. + *

+ * {@code expiresAtEpochMillis} is an absolute wall-clock instant. A source that receives a relative lifetime + * (such as {@code expires_in}) should add it to its own local time of receipt, which makes the cache robust to + * clock skew between the client and the identity platform; a source that cannot determine the expiry must + * supply a conservative value. {@code refreshAtEpochMillis}, when present, is the earliest instant the platform + * suggests refreshing; {@link #NO_REFRESH_AT} means none. + *

+ * {@link #toString()} never renders the token: it shows the length, an 8-hex-digit SHA-256 fingerprint and the + * timestamps. + */ +public final class ExpiringToken { + /** + * Value of {@code refreshAtEpochMillis} meaning the platform gave no refresh hint. + */ + public static final long NO_REFRESH_AT = 0; + private final long expiresAtEpochMillis; + private final long refreshAtEpochMillis; + private final String token; + + /** + * @param token the token, without the {@code "Bearer "} prefix + * @param expiresAtEpochMillis the absolute expiry, in milliseconds since the Unix epoch + * @throws IllegalArgumentException if the token is null, blank or not printable ASCII + */ + public ExpiringToken(String token, long expiresAtEpochMillis) { + this(token, expiresAtEpochMillis, NO_REFRESH_AT); + } + + /** + * @param token the token, without the {@code "Bearer "} prefix + * @param expiresAtEpochMillis the absolute expiry, in milliseconds since the Unix epoch + * @param refreshAtEpochMillis the earliest instant the platform suggests refreshing, or + * {@link #NO_REFRESH_AT} (any value {@code <= 0}) for none + * @throws IllegalArgumentException if the token is null, blank or not printable ASCII + */ + public ExpiringToken(String token, long expiresAtEpochMillis, long refreshAtEpochMillis) { + String problem = describeInvalidToken(token); + if (problem != null) { + throw new IllegalArgumentException(problem); + } + this.token = token; + this.expiresAtEpochMillis = expiresAtEpochMillis; + this.refreshAtEpochMillis = refreshAtEpochMillis > 0 ? refreshAtEpochMillis : NO_REFRESH_AT; + } + + /** + * Applies the token rules of the specification's section 3: non-null, not blank, and every character within + * printable ASCII. Returns a token-free description of the first violation, or {@code null} when the token + * is acceptable. + */ + static String describeInvalidToken(CharSequence token) { + if (token == null || token.length() == 0) { + return "token is null or empty"; + } + boolean blank = true; + for (int i = 0, n = token.length(); i < n; i++) { + char c = token.charAt(i); + if (c < 0x20 || c > 0x7e) { + return "token contains a control or non-ASCII character"; + } + if (c != ' ') { + blank = false; + } + } + return blank ? "token is blank" : null; + } + + private static String instant(long epochMillis) { + try { + return Instant.ofEpochMilli(epochMillis).toString(); + } catch (RuntimeException e) { + return Long.toString(epochMillis); + } + } + + /** + * @return the absolute expiry, in milliseconds since the Unix epoch + */ + public long getExpiresAtEpochMillis() { + return expiresAtEpochMillis; + } + + /** + * @return the platform's earliest suggested refresh instant, or {@link #NO_REFRESH_AT} when none was given + */ + public long getRefreshAtEpochMillis() { + return refreshAtEpochMillis; + } + + /** + * @return the token, without the {@code "Bearer "} prefix + */ + public String getToken() { + return token; + } + + /** + * @return true when the platform gave a refresh hint + */ + public boolean hasRefreshAt() { + return refreshAtEpochMillis != NO_REFRESH_AT; + } + + @Override + public String toString() { + StringBuilder sb = new StringBuilder("ExpiringToken{token=") + .append(CredentialRedaction.describeToken(token)) + .append(", expiresAt=").append(instant(expiresAtEpochMillis)); + if (hasRefreshAt()) { + sb.append(", refreshAt=").append(instant(refreshAtEpochMillis)); + } + return sb.append('}').toString(); + } +} diff --git a/core/src/main/java/io/questdb/client/cutlass/auth/RefreshingTokenProvider.java b/core/src/main/java/io/questdb/client/cutlass/auth/RefreshingTokenProvider.java new file mode 100644 index 000000000..796eb3aea --- /dev/null +++ b/core/src/main/java/io/questdb/client/cutlass/auth/RefreshingTokenProvider.java @@ -0,0 +1,1027 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.cutlass.auth; + +import io.questdb.client.HttpTokenProvider; +import io.questdb.client.std.Chars; +import io.questdb.client.std.QuietCloseable; +import org.jetbrains.annotations.TestOnly; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.time.Instant; +import java.util.concurrent.ScheduledFuture; +import java.util.concurrent.ScheduledThreadPoolExecutor; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.ThreadLocalRandom; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.locks.Condition; +import java.util.concurrent.locks.ReentrantLock; +import java.util.function.DoubleSupplier; + +/** + * A shared, proactively refreshing cache in front of a {@link TokenSource}: the refreshing provider of the + * dynamic-credential specification (design/qwp-token-provider-spec.md, section 5). Hand one instance to + * {@code Sender.builder(...).httpTokenProvider(...)}, {@code QwpQueryClient.withBearerTokenProvider(...)} or + * {@code QuestDB.builder().httpTokenProvider(...)}; every connection then reads the cached token, and the + * provider fetches a new one in the background well before the current one expires. + * 

{@code
+ * TokenCredential credential = new DefaultAzureCredentialBuilder().build();
+ * TokenRequestContext scope = new TokenRequestContext().addScopes("api:///.default");
+ * RefreshingTokenProvider tokens = RefreshingTokenProvider.builder(() -> {
+ *     AccessToken t = credential.getTokenSync(scope);
+ *     return new ExpiringToken(t.getToken(), t.getExpiresAt().toInstant().toEpochMilli());
+ * }).build();
+ * try (QuestDB db = QuestDB.connect("wss::addr=qdb1:9000,qdb2:9000;", tokens)) {
+ *     ...
+ * } finally {
+ *     tokens.close();
+ * }
+ * }
+ *

Behaviour

+ *
    + *
  • Prefetch. The first fetch starts as soon as the provider is built, on the provider's own daemon + * thread. {@link #awaitReady(long)} lets an application gate startup on it.
    • + *
  • Proactive refresh. After a fetch received at {@code f} with expiry {@code e}, the next fetch is + * scheduled at a jittered half-life, {@code f + (e - f) * U(0.45, 0.55)}, no later than + * {@code e - min(refresh_margin, (e - f) / 2)}, no earlier than {@code f + min(min_refresh_interval, + * (e - f) / 2)}, and no later than the platform's refresh hint when it gave one. While the source is healthy, + * even an idle client therefore always holds a usable token.
    • + *
  • Hand-out. A token is usable only while {@code now < e - handout_floor}. {@link #getToken()} + * returns a usable token at once - a volatile read, no I/O, no lock. With no usable token it starts a fetch + * (unless one is running or the provider is backing off after failures) and waits up to {@code cold_wait}, + * then throws a {@link TokenUnavailableException} carrying the classification of the most recent failed + * fetch, or a retryable one when no fetch failed. It never hands out an unusable token. Any number of + * concurrent callers cause at most one fetch.
    • + *
  • Failures. A failed fetch is retried with jittered exponential backoff, from + * {@code backoff_initial} up to {@code backoff_max}, honouring the source's {@code retry_after} up to + * {@code retry_after_max}. Retries continue whatever the classification for as long as the provider is open, + * the current token keeps being served while it remains usable, and failures are logged as warnings at most + * once a minute, with only the classification and a sanitized message.
    • + *
  • Forced refresh. {@link #onTokenRejected(CharSequence, int)} with {@code 401} for the current + * token starts a fetch, at most once per {@code forced_min_interval}, and waits up to {@code forced_wait} for + * it. A stale token (one that has already rotated) and any other status are ignored.
    • + *
  • Interrupts. A caller interrupted while waiting gets a retryable {@link TokenUnavailableException} + * promptly, with its interrupt flag still set.
    • + *
  • Close. {@link #close()} stops the refresher and wakes every waiter; afterwards + * {@link #getToken()} fails with a permanent error. Clients never close a provider the application supplied: + * close it yourself once every client using it is closed.
    • + *
  • Clocks. Expiry comparisons use the wall clock; delays and waits use a monotonic clock.
    • + *
+ * Nothing here ever renders the token: {@link #toString()}, log lines and error messages show at most its + * length and an 8-hex-digit SHA-256 fingerprint. + */ +public final class RefreshingTokenProvider implements HttpTokenProvider, QuietCloseable { + public static final long DEFAULT_BACKOFF_INITIAL_MILLIS = 500; + public static final long DEFAULT_BACKOFF_MAX_MILLIS = 60_000; + public static final long DEFAULT_COLD_WAIT_MILLIS = 30_000; + public static final long DEFAULT_FORCED_MIN_INTERVAL_MILLIS = 30_000; + public static final long DEFAULT_FORCED_WAIT_MILLIS = 5_000; + public static final long DEFAULT_HANDOUT_FLOOR_MILLIS = 60_000; + public static final long DEFAULT_MIN_REFRESH_INTERVAL_MILLIS = 30_000; + public static final long DEFAULT_REFRESH_MARGIN_MILLIS = 300_000; + public static final long DEFAULT_RETRY_AFTER_MAX_MILLIS = 300_000; + /** + * Value returned by the observability getters when there is nothing to report. + */ + public static final long NONE = -1; + private static final long CLOSE_AWAIT_MILLIS = 1_000; + private static final long FAILURE_LOG_INTERVAL_NANOS = TimeUnit.MINUTES.toNanos(1); + private static final Logger LOG = LoggerFactory.getLogger(RefreshingTokenProvider.class); + // Delays are bookkept as monotonic deadlines compared by subtraction, which is only sound while every + // difference stays below 2^63. Clamping each delay to 2^61 ns (~73 years) keeps that true for any token + // lifetime a source can report, including Long.MAX_VALUE "never expires". + private static final long MAX_DELAY_NANOS = Long.MAX_VALUE >> 2; + private static final AtomicInteger THREAD_IDS = new AtomicInteger(); + // A waiter re-checks its deadline at least this often when the clock is not the system clock, so a test + // that advances a fake clock is noticed. With the system clock the waits are untimed beyond their deadline. + private static final long WAIT_SLICE_NANOS = TimeUnit.MILLISECONDS.toNanos(20); + private final long backoffInitialMillis; + private final long backoffMaxMillis; + private final Clock clock; + private final long coldWaitMillis; + private final long forcedMinIntervalNanos; + private final long forcedWaitNanos; + private final long handoutFloorMillis; + private final ReentrantLock lock = new ReentrantLock(); + private final long maxWaitSliceNanos; + private final long minRefreshIntervalMillis; + private final String name; + private final DoubleSupplier random; + private final long refreshMarginMillis; + private final long retryAfterMaxMillis; + private final Scheduler scheduler; + private final TokenSource source; + // Signalled whenever a fetch completes and on close. + private final Condition stateChanged = lock.newCondition(); + // ---- guarded by lock ---- + private boolean backoffActive; + private long backoffUntilNanos; + private volatile boolean closed; + private long completedFetches; + private volatile int consecutiveFailures; + private boolean failureLogged; + private volatile boolean fetchRunning; + private boolean forcedRefreshed; + // The current token, or null. Immutable snapshot, published with a volatile write so the warm path in + // getToken() needs no lock. + private volatile Held held; + private volatile FetchFailure lastFailure; + private long lastFailureLogNanos; + private long lastForcedNanos; + private volatile long lastSuccessEpochMillis = NONE; + private long scheduledDueNanos; + private long scheduledSeq; + private ScheduledTask scheduledTask; + + private RefreshingTokenProvider(Builder b) { + this.source = b.source; + this.name = b.name; + this.clock = b.clock; + this.maxWaitSliceNanos = b.clock == SystemClock.INSTANCE ? Long.MAX_VALUE : WAIT_SLICE_NANOS; + this.scheduler = b.scheduler != null ? b.scheduler : new DefaultScheduler(); + this.random = b.random; + this.refreshMarginMillis = b.refreshMarginMillis; + this.minRefreshIntervalMillis = b.minRefreshIntervalMillis; + this.handoutFloorMillis = b.handoutFloorMillis; + this.coldWaitMillis = b.coldWaitMillis; + this.backoffInitialMillis = b.backoffInitialMillis; + this.backoffMaxMillis = b.backoffMaxMillis; + this.retryAfterMaxMillis = b.retryAfterMaxMillis; + this.forcedMinIntervalNanos = TimeUnit.MILLISECONDS.toNanos(b.forcedMinIntervalMillis); + this.forcedWaitNanos = TimeUnit.MILLISECONDS.toNanos(b.forcedWaitMillis); + } + + /** + * Starts building a provider around {@code source}. + * + * @param source obtains new tokens; called only from the provider's refresher thread + * @return a builder whose defaults are the specification's + */ + public static Builder builder(TokenSource source) { + if (source == null) { + throw new IllegalArgumentException("token source must not be null"); + } + return new Builder(source); + } + + /** + * The refresh instant of the specification's section 5.2 for a token received at {@code f} that expires at + * {@code e}, given a uniform sample {@code u} in {@code [0, 1)}. Pure; exposed for the conformance tests. + */ + @TestOnly + public static long computeRefreshAtMillis( + long f, + long e, + long refreshAt, + double u, + long refreshMarginMillis, + long minRefreshIntervalMillis + ) { + final long life = e - f; + final long half = life / 2; + final long margin = Math.min(refreshMarginMillis, half); + final long floor = Math.min(minRefreshIntervalMillis, half); + final double fraction = 0.45 + 0.10 * Math.max(0.0, Math.min(1.0, u)); + long r = f + (long) (life * fraction); + r = Math.min(r, e - margin); + r = Math.max(r, f + floor); + if (refreshAt > 0) { + r = Math.min(r, Math.max(refreshAt, f + floor)); + } + return r; + } + + private static String instant(long epochMillis) { + try { + return Instant.ofEpochMilli(epochMillis).toString(); + } catch (RuntimeException e) { + return Long.toString(epochMillis); + } + } + + private static long millisToNanos(long millis) { + return Math.min(TimeUnit.MILLISECONDS.toNanos(Math.max(0, millis)), MAX_DELAY_NANOS); + } + + /** + * Waits until the provider holds a usable token, starting a fetch if none is running and the provider is + * not backing off. Use it to gate application startup on the first fetch. + * + * @param timeoutMillis the longest to wait + * @return true once a usable token is held; false on timeout, when the provider is closed, or when the + * calling thread is interrupted (its interrupt flag is then left set) + */ + public boolean awaitReady(long timeoutMillis) { + lock.lock(); + try { + final long start = clock.monotonicNanos(); + final long deadline = start + millisToNanos(timeoutMillis); + if (!usable(held, clock.wallClockMillis())) { + requestFetchNowLocked(start); + } + while (true) { + if (closed) { + return false; + } + if (usable(held, clock.wallClockMillis())) { + return true; + } + final long remaining = deadline - clock.monotonicNanos(); + if (remaining <= 0) { + return false; + } + try { + stateChanged.awaitNanos(Math.min(remaining, maxWaitSliceNanos)); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + return false; + } + } + } finally { + lock.unlock(); + } + } + + /** + * Stops scheduled fetches, interrupts a fetch in progress and wakes every waiter. Afterwards + * {@link #getToken()} fails with a permanent {@link TokenUnavailableException}. Idempotent. Waits at most + * one second for the refresher thread to exit. + */ + @Override + public void close() { + lock.lock(); + try { + if (closed) { + return; + } + closed = true; + cancelScheduledLocked(); + held = null; + stateChanged.signalAll(); + } finally { + lock.unlock(); + } + scheduler.shutdown(); + LOG.debug("token provider {} closed", name); + } + + /** + * @return the number of consecutive failed fetches since the last successful one + */ + public int getConsecutiveFailures() { + return consecutiveFailures; + } + + /** + * The most recent failed fetch - its classification, sanitized message and time - or null when no fetch + * has failed. Kept after the source recovers. + */ + public FetchFailure getLastFailure() { + return lastFailure; + } + + /** + * @return the wall-clock time of the last successful fetch, or {@link #NONE} + */ + public long getLastSuccessEpochMillis() { + return lastSuccessEpochMillis; + } + + /** + * @return the non-secret label this provider uses in log lines and errors + */ + public String getName() { + return name; + } + + /** + * Returns a usable token: at once when one is held, otherwise after waiting up to {@code cold_wait} for a + * fetch. + * + * @return the current token, without the {@code "Bearer "} prefix + * @throws TokenUnavailableException when no usable token arrives in time (classified as the most recent + * failed fetch, or retryable when none failed), when the calling thread + * is interrupted (retryable; the interrupt flag stays set), or when the + * provider is closed (permanent) + */ + @Override + public CharSequence getToken() { + if (closed) { + throw closedException(); + } + final Held h = held; + if (h != null) { + final long wallNow = clock.wallClockMillis(); + if (usable(h, wallNow)) { + // Warm path: a volatile read, no I/O and no lock. Only when the refresh is overdue - its + // scheduled fetch has not run, say after the host slept through it - does a caller take the + // lock, to start that fetch in the background. + if (!fetchRunning && consecutiveFailures == 0 + && (wallNow >= h.refreshAtEpochMillis || clock.monotonicNanos() - h.refreshAtNanos >= 0)) { + startOverdueRefresh(); + } + return h.token; + } + } + return awaitUsableToken(); + } + + /** + * @return the current token's expiry, or {@link #NONE} when no token is held + */ + public long getTokenExpiresAtEpochMillis() { + final Held h = held; + return h == null ? NONE : h.expiresAtEpochMillis; + } + + /** + * @return true once {@link #close()} has been called + */ + public boolean isClosed() { + return closed; + } + + /** + * Forced refresh. When a server answered {@code 401} to {@code token} and that token is still the current + * one, starts a fetch now - joining one already running, and respecting failure backoff - and waits up to + * {@code forced_wait} for it. Ignored for any other status, for a token that has already rotated, and when + * a forced refresh ran less than {@code forced_min_interval} ago. A forced fetch that returns the same token + * counts as a normal success. Never throws; returns promptly when interrupted, with the flag still set. + */ + @Override + public void onTokenRejected(CharSequence token, int httpStatus) { + if (httpStatus != 401 || token == null) { + return; + } + lock.lock(); + try { + if (closed) { + return; + } + final Held h = held; + if (h == null || !Chars.equals(h.token, token)) { + return; // stale: the token has already rotated, so the caller's next pull gets the new one + } + final long now = clock.monotonicNanos(); + if (forcedRefreshed && now - lastForcedNanos < forcedMinIntervalNanos) { + return; + } + forcedRefreshed = true; + lastForcedNanos = now; + LOG.info("token provider {}: the server rejected the current token with 401, refreshing early", name); + final long target = completedFetches + 1; + requestFetchNowLocked(now); + final long deadline = now + forcedWaitNanos; + while (!closed && completedFetches < target) { + final long remaining = deadline - clock.monotonicNanos(); + if (remaining <= 0) { + break; + } + stateChanged.awaitNanos(Math.min(remaining, maxWaitSliceNanos)); + } + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } finally { + lock.unlock(); + } + } + + @Override + public String toString() { + final Held h = held; + final FetchFailure f = lastFailure; + final StringBuilder sb = new StringBuilder("RefreshingTokenProvider{name=").append(name); + if (h == null) { + sb.append(", token="); + } else { + sb.append(", token=").append(CredentialRedaction.describeToken(h.token)) + .append(", expiresAt=").append(instant(h.expiresAtEpochMillis)) + .append(", refreshAt=").append(instant(h.refreshAtEpochMillis)); + } + final long success = lastSuccessEpochMillis; + if (success != NONE) { + sb.append(", lastSuccess=").append(instant(success)); + } + sb.append(", consecutiveFailures=").append(consecutiveFailures); + if (f != null) { + sb.append(", lastFailure=").append(f); + } + if (closed) { + sb.append(", closed"); + } + return sb.append('}').toString(); + } + + private CharSequence awaitUsableToken() { + lock.lock(); + try { + final long start = clock.monotonicNanos(); + final long deadline = start + millisToNanos(coldWaitMillis); + // A caller that arrives without a usable token starts one fetch - unless one is running or the + // provider is backing off - and then waits for it or for the fetches already scheduled. It does not + // start another each time a fetch completes: a source that keeps returning a token inside the + // hand-out floor would otherwise be fetched in a tight loop for the whole wait. + if (!usable(held, clock.wallClockMillis())) { + requestFetchNowLocked(start); + } + while (true) { + if (closed) { + throw closedException(); + } + final Held h = held; + if (usable(h, clock.wallClockMillis())) { + return h.token; + } + final long now = clock.monotonicNanos(); + // The earliest permitted fetch lies beyond the caller's deadline: nothing can arrive in time, + // so say so now rather than parking the caller for nothing. + if (!fetchRunning && inBackoffLocked(now) && backoffUntilNanos - deadline > 0) { + throw unavailableLocked(now, start); + } + final long remaining = deadline - now; + if (remaining <= 0) { + throw unavailableLocked(now, start); + } + try { + stateChanged.awaitNanos(Math.min(remaining, maxWaitSliceNanos)); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new TokenUnavailableException("interrupted while waiting for a token from " + name, true); + } + } + } finally { + lock.unlock(); + } + } + + private long backoffMillis(int failures, long retryAfterMillis) { + final int shift = Math.min(failures - 1, 40); + long base = backoffInitialMillis << shift; + if (base < 0 || (base >> shift) != backoffInitialMillis) { + base = backoffMaxMillis; // overflowed + } + base = Math.min(base, backoffMaxMillis); + final long half = base / 2; + long delay = half + (long) (half * uniform()); + if (retryAfterMillis >= 0) { + delay = Math.max(delay, Math.min(retryAfterMillis, retryAfterMaxMillis)); + } + return delay; + } + + private void cancelScheduledLocked() { + scheduledSeq++; + final ScheduledTask t = scheduledTask; + scheduledTask = null; + if (t != null) { + t.cancel(); + } + } + + private FetchFailure checkResult(ExpiringToken result, long receivedAt) { + if (result == null) { + return new FetchFailure(true, TokenUnavailableException.NO_RETRY_AFTER, + "token source returned null", receivedAt); + } + final String problem = ExpiringToken.describeInvalidToken(result.getToken()); + if (problem != null) { + return new FetchFailure(true, TokenUnavailableException.NO_RETRY_AFTER, + "token source returned an unusable token: " + problem, receivedAt); + } + if (result.getExpiresAtEpochMillis() <= receivedAt) { + return new FetchFailure(true, TokenUnavailableException.NO_RETRY_AFTER, + "token source returned a token that expired at " + instant(result.getExpiresAtEpochMillis()) + + ", not after it was received at " + instant(receivedAt), + receivedAt); + } + return null; + } + + private FetchFailure classify(Throwable t, long receivedAt) { + if (t instanceof TokenUnavailableException) { + final TokenUnavailableException e = (TokenUnavailableException) t; + return new FetchFailure(e.isRetryable(), e.getRetryAfterMillis(), e.getMessage(), receivedAt); + } + // An unclassified failure is retryable (the specification's rule when unsure). Name its type: the + // message alone ("connect timed out") often does not say what failed. + final String message = t.getMessage(); + return new FetchFailure(true, TokenUnavailableException.NO_RETRY_AFTER, + message == null ? t.getClass().getName() : t.getClass().getName() + ": " + message, receivedAt); + } + + private TokenUnavailableException closedException() { + return new TokenUnavailableException("token provider " + name + " is closed", false); + } + + private boolean inBackoffLocked(long now) { + return backoffActive && now - backoffUntilNanos < 0; + } + + private String onFailureLocked(FetchFailure failure, long receivedNanos) { + final int failures = consecutiveFailures + 1; + consecutiveFailures = failures; + lastFailure = failure; + final long delayMillis = backoffMillis(failures, failure.retryAfterMillis); + final long delayNanos = millisToNanos(delayMillis); + backoffActive = true; + backoffUntilNanos = receivedNanos + delayNanos; + scheduleFetchLocked(delayNanos, receivedNanos); + if (failureLogged && receivedNanos - lastFailureLogNanos < FAILURE_LOG_INTERVAL_NANOS) { + return null; + } + failureLogged = true; + lastFailureLogNanos = receivedNanos; + return "token provider " + name + ": fetch failed [retryable=" + failure.retryable + + ", consecutiveFailures=" + failures + ", retryInMillis=" + delayMillis + "]: " + failure.message; + } + + private void onSuccessLocked(ExpiringToken result, long receivedAt, long receivedNanos) { + final long refreshAt = computeRefreshAtMillis( + receivedAt, + result.getExpiresAtEpochMillis(), + result.getRefreshAtEpochMillis(), + uniform(), + refreshMarginMillis, + minRefreshIntervalMillis + ); + final long delayNanos = millisToNanos(refreshAt - receivedAt); + final int failures = consecutiveFailures; + held = new Held(result.getToken(), result.getExpiresAtEpochMillis(), refreshAt, receivedNanos + delayNanos); + lastSuccessEpochMillis = receivedAt; + consecutiveFailures = 0; + backoffActive = false; + scheduleFetchLocked(delayNanos, receivedNanos); + if (failures > 0) { + LOG.info("token provider {}: fetch succeeded after {} consecutive failure(s) [token={}, expiresAt={}]", + name, failures, CredentialRedaction.describeToken(result.getToken()), + instant(result.getExpiresAtEpochMillis())); + } else if (LOG.isDebugEnabled()) { + LOG.debug("token provider {}: fetched [token={}, expiresAt={}, refreshAt={}]", + name, CredentialRedaction.describeToken(result.getToken()), + instant(result.getExpiresAtEpochMillis()), instant(refreshAt)); + } + } + + // Starts a fetch now unless one is running, one is already due, or the provider is in failure backoff. + private void requestFetchNowLocked(long now) { + if (closed || fetchRunning || inBackoffLocked(now)) { + return; + } + if (scheduledTask != null && now - scheduledDueNanos >= 0) { + return; // already due: the scheduler runs it as soon as it can + } + scheduleFetchLocked(0, now); + } + + private void runFetch(long seq) { + lock.lock(); + try { + if (closed || seq != scheduledSeq || fetchRunning) { + return; // cancelled, superseded, or a fetch is already in flight + } + scheduledTask = null; + fetchRunning = true; + } finally { + lock.unlock(); + } + ExpiringToken result = null; + Throwable error = null; + try { + result = source.fetchToken(); + } catch (Throwable t) { + // Every failure, an Error included, is a failed fetch to retry with backoff: rethrowing would only + // be swallowed by the scheduler and leave the provider with no fetch scheduled, ever again. + error = t; + } + final long receivedAt = clock.wallClockMillis(); + final long receivedNanos = clock.monotonicNanos(); + FetchFailure failure = error != null ? classify(error, receivedAt) : checkResult(result, receivedAt); + if (failure != null) { + failure = failure.sanitized(); + } + String warning = null; + lock.lock(); + try { + fetchRunning = false; + completedFetches++; + if (!closed) { + if (failure == null) { + onSuccessLocked(result, receivedAt, receivedNanos); + } else { + warning = onFailureLocked(failure, receivedNanos); + } + } + stateChanged.signalAll(); + } finally { + lock.unlock(); + } + if (warning != null) { + LOG.warn(warning); + } + } + + private void scheduleFetchLocked(long delayNanos, long now) { + if (closed) { + return; + } + cancelScheduledLocked(); + final long seq = scheduledSeq; + scheduledDueNanos = now + delayNanos; + scheduledTask = scheduler.schedule(() -> runFetch(seq), delayNanos); + } + + private void startOverdueRefresh() { + lock.lock(); + try { + requestFetchNowLocked(clock.monotonicNanos()); + } finally { + lock.unlock(); + } + } + + private double uniform() { + return random.getAsDouble(); + } + + private TokenUnavailableException unavailableLocked(long now, long start) { + final FetchFailure f = consecutiveFailures > 0 ? lastFailure : null; + final long waitedMillis = TimeUnit.NANOSECONDS.toMillis(Math.max(0, now - start)); + if (f == null) { + return new TokenUnavailableException("no usable token from " + name + " after waiting " + + waitedMillis + " ms", true); + } + final long retryAfter = backoffActive && backoffUntilNanos - now > 0 + ? TimeUnit.NANOSECONDS.toMillis(backoffUntilNanos - now) + : TokenUnavailableException.NO_RETRY_AFTER; + return new TokenUnavailableException("no usable token from " + name + " after waiting " + waitedMillis + + " ms [retryable=" + f.retryable + ", consecutiveFailures=" + consecutiveFailures + "]: " + + f.message, f.retryable, retryAfter); + } + + private boolean usable(Held h, long wallNow) { + return h != null && wallNow < h.expiresAtEpochMillis - handoutFloorMillis; + } + + /** + * Time source. Expiry comparisons use {@link #wallClockMillis()}; delays and waits use + * {@link #monotonicNanos()}. + */ + public interface Clock { + long monotonicNanos(); + + long wallClockMillis(); + } + + /** + * Runs fetches on the provider's background context. The default owns one daemon thread. A test may supply + * its own to run fetches deterministically. + */ + public interface Scheduler { + /** + * Runs {@code task} once, after {@code delayNanos} on the monotonic clock. + */ + ScheduledTask schedule(Runnable task, long delayNanos); + + /** + * Called once from {@link RefreshingTokenProvider#close()}: stop running tasks and interrupt a task in + * progress. + */ + void shutdown(); + } + + /** + * Handle to a task submitted to a {@link Scheduler}. + */ + public interface ScheduledTask { + /** + * Best-effort: a task that has already started may still run, and the provider ignores it. + */ + void cancel(); + } + + /** + * Builder for {@link RefreshingTokenProvider}. The defaults are the specification's (section 5.1). + */ + public static final class Builder { + private final TokenSource source; + private long backoffInitialMillis = DEFAULT_BACKOFF_INITIAL_MILLIS; + private long backoffMaxMillis = DEFAULT_BACKOFF_MAX_MILLIS; + private Clock clock = SystemClock.INSTANCE; + private long coldWaitMillis = DEFAULT_COLD_WAIT_MILLIS; + private long forcedMinIntervalMillis = DEFAULT_FORCED_MIN_INTERVAL_MILLIS; + private long forcedWaitMillis = DEFAULT_FORCED_WAIT_MILLIS; + private long handoutFloorMillis = DEFAULT_HANDOUT_FLOOR_MILLIS; + private long minRefreshIntervalMillis = DEFAULT_MIN_REFRESH_INTERVAL_MILLIS; + private String name = "token-provider"; + private DoubleSupplier random = () -> ThreadLocalRandom.current().nextDouble(); + private long refreshMarginMillis = DEFAULT_REFRESH_MARGIN_MILLIS; + private long retryAfterMaxMillis = DEFAULT_RETRY_AFTER_MAX_MILLIS; + private Scheduler scheduler; + + private Builder(TokenSource source) { + this.source = source; + } + + private static long nonNegative(String name, long value) { + if (value < 0) { + throw new IllegalArgumentException(name + " must be >= 0: " + value); + } + return value; + } + + /** + * First retry delay after a failed fetch. Default 500 ms. + */ + public Builder backoffInitialMillis(long millis) { + if (millis <= 0) { + throw new IllegalArgumentException("backoff_initial must be > 0: " + millis); + } + this.backoffInitialMillis = millis; + return this; + } + + /** + * Largest retry delay after failed fetches. Default 60 s. + */ + public Builder backoffMaxMillis(long millis) { + if (millis <= 0) { + throw new IllegalArgumentException("backoff_max must be > 0: " + millis); + } + this.backoffMaxMillis = millis; + return this; + } + + /** + * Builds the provider and starts its first fetch in the background. + */ + public RefreshingTokenProvider build() { + if (backoffMaxMillis < backoffInitialMillis) { + throw new IllegalArgumentException("backoff_max must be >= backoff_initial [backoff_max=" + + backoffMaxMillis + ", backoff_initial=" + backoffInitialMillis + ']'); + } + RefreshingTokenProvider provider = new RefreshingTokenProvider(this); + provider.lock.lock(); + try { + provider.scheduleFetchLocked(0, provider.clock.monotonicNanos()); + } finally { + provider.lock.unlock(); + } + return provider; + } + + /** + * Test seam: the clock for expiry comparisons, delays and waits. + */ + @TestOnly + public Builder clock(Clock clock) { + if (clock == null) { + throw new IllegalArgumentException("clock must not be null"); + } + this.clock = clock; + return this; + } + + /** + * The longest {@code getToken()} waits when no usable token is held. Default 30 s. + */ + public Builder coldWaitMillis(long millis) { + this.coldWaitMillis = nonNegative("cold_wait", millis); + return this; + } + + /** + * Minimum spacing between forced refreshes. Default 30 s. + */ + public Builder forcedMinIntervalMillis(long millis) { + this.forcedMinIntervalMillis = nonNegative("forced_min_interval", millis); + return this; + } + + /** + * The longest {@code onTokenRejected} waits for a forced refresh. Default 5 s. + */ + public Builder forcedWaitMillis(long millis) { + this.forcedWaitMillis = nonNegative("forced_wait", millis); + return this; + } + + /** + * A token is usable only while {@code now < expires_at - handout_floor}. Default 60 s. + */ + public Builder handoutFloorMillis(long millis) { + this.handoutFloorMillis = nonNegative("handout_floor", millis); + return this; + } + + /** + * Lower bound on the time between a fetch and the next scheduled refresh. Default 30 s. + */ + public Builder minRefreshIntervalMillis(long millis) { + this.minRefreshIntervalMillis = nonNegative("min_refresh_interval", millis); + return this; + } + + /** + * A non-secret label for log lines, errors and the refresher thread, e.g. + * {@code azure[resource=api://...]}. Default {@code token-provider}. + */ + public Builder name(String name) { + if (name == null || name.isEmpty()) { + throw new IllegalArgumentException("name must not be empty"); + } + this.name = name; + return this; + } + + /** + * Test seam: the source of uniform samples in {@code [0, 1)} for the refresh and backoff jitter. + */ + @TestOnly + public Builder random(DoubleSupplier uniform) { + if (uniform == null) { + throw new IllegalArgumentException("random must not be null"); + } + this.random = uniform; + return this; + } + + /** + * Upper bound on how close to expiry a scheduled refresh may run. Default 5 min. + */ + public Builder refreshMarginMillis(long millis) { + this.refreshMarginMillis = nonNegative("refresh_margin", millis); + return this; + } + + /** + * Cap applied to a source's {@code retry_after}. Default 5 min. + */ + public Builder retryAfterMaxMillis(long millis) { + this.retryAfterMaxMillis = nonNegative("retry_after_max", millis); + return this; + } + + /** + * Test seam: where fetches run. The provider calls {@link Scheduler#shutdown()} from {@code close()}. + */ + @TestOnly + public Builder scheduler(Scheduler scheduler) { + if (scheduler == null) { + throw new IllegalArgumentException("scheduler must not be null"); + } + this.scheduler = scheduler; + return this; + } + } + + /** + * A failed fetch: its classification, sanitized message and time. Never carries the token. + */ + public static final class FetchFailure { + private final long epochMillis; + private final String message; + private final long retryAfterMillis; + private final boolean retryable; + + FetchFailure(boolean retryable, long retryAfterMillis, String message, long epochMillis) { + this.retryable = retryable; + this.retryAfterMillis = retryAfterMillis < 0 ? TokenUnavailableException.NO_RETRY_AFTER : retryAfterMillis; + this.message = message; + this.epochMillis = epochMillis; + } + + /** + * @return the wall-clock time of the failure + */ + public long getEpochMillis() { + return epochMillis; + } + + /** + * @return the failure message, sanitized and at most 256 characters + */ + public String getMessage() { + return message; + } + + /** + * @return the source's suggested wait before retrying, or {@link TokenUnavailableException#NO_RETRY_AFTER} + */ + public long getRetryAfterMillis() { + return retryAfterMillis; + } + + /** + * @return whether the source classified the failure as retryable + */ + public boolean isRetryable() { + return retryable; + } + + @Override + public String toString() { + return "FetchFailure{retryable=" + retryable + ", at=" + instant(epochMillis) + ", message=" + message + '}'; + } + + FetchFailure sanitized() { + final String clean = CredentialRedaction.sanitizeErrorText(message); + return new FetchFailure(retryable, retryAfterMillis, clean == null || clean.isEmpty() + ? "token source failed without a message" : clean, epochMillis); + } + } + + private static final class DefaultScheduler implements Scheduler { + private final ScheduledThreadPoolExecutor executor; + private volatile Thread thread; + + DefaultScheduler() { + final String threadName = "qdb-token-refresh-" + THREAD_IDS.incrementAndGet(); + this.executor = new ScheduledThreadPoolExecutor(1, r -> { + Thread t = new Thread(r, threadName); + t.setDaemon(true); + thread = t; + return t; + }); + executor.setRemoveOnCancelPolicy(true); + executor.setExecuteExistingDelayedTasksAfterShutdownPolicy(false); + executor.setContinueExistingPeriodicTasksAfterShutdownPolicy(false); + } + + @Override + public ScheduledTask schedule(Runnable task, long delayNanos) { + final ScheduledFuture future = executor.schedule(task, delayNanos, TimeUnit.NANOSECONDS); + return () -> future.cancel(false); + } + + @Override + public void shutdown() { + executor.shutdownNow(); + if (Thread.currentThread() == thread) { + return; // closed from inside a fetch: the thread exits once this task unwinds + } + // Interrupt-neutral, like the other close paths in this library: a carried interrupt flag would + // turn the bounded wait into an immediate return. + final boolean interrupted = Thread.interrupted(); + try { + executor.awaitTermination(CLOSE_AWAIT_MILLIS, TimeUnit.MILLISECONDS); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } finally { + if (interrupted) { + Thread.currentThread().interrupt(); + } + } + } + } + + private static final class Held { + final long expiresAtEpochMillis; + final long refreshAtEpochMillis; + final long refreshAtNanos; + final String token; + + Held(String token, long expiresAtEpochMillis, long refreshAtEpochMillis, long refreshAtNanos) { + this.token = token; + this.expiresAtEpochMillis = expiresAtEpochMillis; + this.refreshAtEpochMillis = refreshAtEpochMillis; + this.refreshAtNanos = refreshAtNanos; + } + } + + private static final class SystemClock implements Clock { + static final SystemClock INSTANCE = new SystemClock(); + + @Override + public long monotonicNanos() { + return System.nanoTime(); + } + + @Override + public long wallClockMillis() { + return System.currentTimeMillis(); + } + } +} diff --git a/core/src/main/java/io/questdb/client/cutlass/auth/TokenProviderFactory.java b/core/src/main/java/io/questdb/client/cutlass/auth/TokenProviderFactory.java new file mode 100644 index 000000000..2e96e1fd0 --- /dev/null +++ b/core/src/main/java/io/questdb/client/cutlass/auth/TokenProviderFactory.java @@ -0,0 +1,83 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.cutlass.auth; + +import java.util.Map; + +/** + * Service-provider interface behind the {@code token_provider} connect-string key (design/qwp-token-provider-spec.md, + * section 7). An implementation is found through {@link java.util.ServiceLoader}: list it in + * {@code META-INF/services/io.questdb.client.cutlass.auth.TokenProviderFactory}, or declare it with + * {@code provides} in a module descriptor. The optional {@code questdb-client-azure} artifact supplies the + * {@code azure} factory this way. + *

+ * A factory turns the provider-specific connect-string keys into a {@link TokenSource}. The client wraps that + * source in a {@link RefreshingTokenProvider} that the process-wide {@link TokenProviderRegistry} shares between + * every client built from an equivalent connect string, so a process runs one refresher per configuration. + *

+ * The connect-string vocabulary is fixed - unknown keys are rejected before a factory ever sees them - so a + * factory receives only the keys the client defines for it, already normalized: for {@code azure}, the + * {@code azure_resource} (with any trailing {@code /.default} removed) and the optional, lower-cased + * {@code azure_client_id}. Values in these keys are never secret: secrets come from the platform's standard + * configuration, never from the connect string. + */ +public interface TokenProviderFactory { + + /** + * Creates the token source for one configuration. Called once per distinct configuration, when the first + * client using it connects. It must not perform network I/O: the provider's refresher thread calls + * {@link TokenSource#fetchToken()} later. + * + * @param params the provider-specific keys, normalized; unmodifiable + * @return the token source + * @throws IllegalArgumentException when the parameters cannot be used + */ + TokenSource createSource(Map params); + + /** + * A non-secret label for log lines and errors, e.g. {@code azure[resource=api://...]}. + * + * @param params the provider-specific keys, normalized + * @return the label + */ + default String describe(Map params) { + return params.isEmpty() ? name() : name() + params; + } + + /** + * The value of {@code token_provider} this factory serves, such as {@code azure}. + */ + String name(); + + /** + * Validates the provider-specific keys when a connect string is parsed, before any client connects. Must be + * pure: it runs during configuration validation that must not fetch a token (section 7.2). + * + * @param params the provider-specific keys, normalized; unmodifiable + * @throws IllegalArgumentException naming the offending key + */ + default void validate(Map params) { + } +} diff --git a/core/src/main/java/io/questdb/client/cutlass/auth/TokenProviderRegistry.java b/core/src/main/java/io/questdb/client/cutlass/auth/TokenProviderRegistry.java new file mode 100644 index 000000000..7c55a6f74 --- /dev/null +++ b/core/src/main/java/io/questdb/client/cutlass/auth/TokenProviderRegistry.java @@ -0,0 +1,302 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.cutlass.auth; + +import io.questdb.client.std.QuietCloseable; +import org.jetbrains.annotations.TestOnly; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.util.ArrayList; +import java.util.HashMap; +import java.util.Iterator; +import java.util.List; +import java.util.Map; +import java.util.ServiceConfigurationError; +import java.util.ServiceLoader; +import java.util.TreeSet; +import java.util.concurrent.ScheduledFuture; +import java.util.concurrent.ScheduledThreadPoolExecutor; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicBoolean; + +/** + * Process-wide registry of the token providers that connect strings select with {@code token_provider} + * (design/qwp-token-provider-spec.md, section 7.4). Every client instance built from a connect string - each + * sender, each query client, each pooled connection - acquires a {@link Lease} when it connects and releases it + * when it closes. All leases for the same configuration - the provider name plus its normalized parameters - + * share one {@link RefreshingTokenProvider}, so the process runs one refresher, and makes at most one call to + * the identity platform at a time, per configuration. + *

+ * When the last lease is released the provider stays alive for the registry's linger period (60 s for the + * global registry) and is closed only if no new lease arrives in the meantime: a pool that recycles its + * connections, or an application that closes one sender and opens another, keeps the warm cache. + *

+ * Providers that an application supplies itself never pass through here; sharing those is the application's + * responsibility. + */ +public final class TokenProviderRegistry { + public static final long DEFAULT_LINGER_MILLIS = 60_000; + private static final TokenProviderRegistry GLOBAL = new TokenProviderRegistry(DEFAULT_LINGER_MILLIS); + private static final Logger LOG = LoggerFactory.getLogger(TokenProviderRegistry.class); + @TestOnly + private static volatile List factoriesForTesting; + private final Map entries = new HashMap<>(); + private final long lingerMillis; + private final Object lock = new Object(); + private ScheduledThreadPoolExecutor lingerTimer; + + /** + * Creates a private registry. Production code uses {@link #global()}; a separate instance is for tests and + * for applications that want to scope provider sharing themselves. + * + * @param lingerMillis how long a provider outlives its last lease; {@code <= 0} closes it at once + */ + public TokenProviderRegistry(long lingerMillis) { + this.lingerMillis = lingerMillis; + } + + /** + * Finds the factory serving {@code name} through {@link ServiceLoader}, trying the thread context class + * loader first and then the loader of this library. + * + * @return the factory, or null when none is installed + */ + public static TokenProviderFactory findFactory(String name) { + for (TokenProviderFactory factory : factories()) { + if (name.equals(factory.name())) { + return factory; + } + } + return null; + } + + /** + * The registry every client built from a connect string uses. + */ + public static TokenProviderRegistry global() { + return GLOBAL; + } + + /** + * Test seam: replaces {@link ServiceLoader} discovery with a fixed list of factories. Pass null to restore + * discovery. + */ + @TestOnly + public static void setFactoriesForTesting(List factories) { + factoriesForTesting = factories; + } + + /** + * The names of every installed factory, sorted: the values {@code token_provider} accepts. + */ + public static List supportedProviders() { + TreeSet names = new TreeSet<>(); + for (TokenProviderFactory factory : factories()) { + names.add(factory.name()); + } + return new ArrayList<>(names); + } + + private static List factories() { + final List override = factoriesForTesting; + if (override != null) { + return override; + } + final List found = new ArrayList<>(); + final ClassLoader context = Thread.currentThread().getContextClassLoader(); + load(context == null ? ServiceLoader.load(TokenProviderFactory.class) + : ServiceLoader.load(TokenProviderFactory.class, context), found); + final ClassLoader own = TokenProviderFactory.class.getClassLoader(); + if (own != context) { + load(ServiceLoader.load(TokenProviderFactory.class, own), found); + } + return found; + } + + private static void load(ServiceLoader loader, List into) { + final Iterator it = loader.iterator(); + while (true) { + final TokenProviderFactory factory; + try { + if (!it.hasNext()) { + return; + } + factory = it.next(); + } catch (ServiceConfigurationError e) { + // one broken provider must not hide the others + LOG.warn("could not load a token provider factory: {}", + CredentialRedaction.sanitizeErrorText(String.valueOf(e.getMessage()))); + continue; + } + boolean duplicate = false; + for (int i = 0, n = into.size(); i < n; i++) { + if (into.get(i).getClass() == factory.getClass()) { + duplicate = true; + break; + } + } + if (!duplicate) { + into.add(factory); + } + } + } + + /** + * Acquires a lease on the shared provider for {@code spec}, creating the provider - and starting its first + * fetch - when none is alive. Cancels a pending linger close. + * + * @throws IllegalArgumentException when the factory rejects the parameters + */ + public Lease acquire(TokenProviderSpec spec) { + synchronized (lock) { + Entry entry = entries.get(spec.registryKey()); + if (entry == null || entry.provider.isClosed()) { + TokenSource source = spec.factory().createSource(spec.params()); + if (source == null) { + throw new IllegalStateException("token provider factory " + spec.name() + " returned no token source"); + } + RefreshingTokenProvider provider = RefreshingTokenProvider.builder(source).name(spec.describe()).build(); + entry = new Entry(spec.registryKey(), provider); + entries.put(entry.key, entry); + LOG.info("started token provider {}", entry.provider.getName()); + } + entry.refs++; + if (entry.lingerTask != null) { + entry.lingerTask.cancel(false); + entry.lingerTask = null; + } + return new Lease(entry); + } + } + + /** + * Whether a provider for {@code spec} is alive - leased, or lingering after its last lease. + */ + public boolean isActive(TokenProviderSpec spec) { + synchronized (lock) { + Entry entry = entries.get(spec.registryKey()); + return entry != null && !entry.provider.isClosed(); + } + } + + /** + * Number of leases currently held on the provider for {@code spec}. + */ + public int leaseCount(TokenProviderSpec spec) { + synchronized (lock) { + Entry entry = entries.get(spec.registryKey()); + return entry == null ? 0 : entry.refs; + } + } + + private void expire(Entry entry) { + synchronized (lock) { + if (entry.refs > 0 || entries.get(entry.key) != entry) { + return; // re-leased during the linger, or already replaced + } + entries.remove(entry.key); + entry.lingerTask = null; + } + LOG.info("closing token provider {}: no client has used it for {} ms", entry.provider.getName(), lingerMillis); + entry.provider.close(); + } + + private ScheduledThreadPoolExecutor lingerTimer() { + if (lingerTimer == null) { + lingerTimer = new ScheduledThreadPoolExecutor(1, r -> { + Thread t = new Thread(r, "qdb-token-provider-registry"); + t.setDaemon(true); + return t; + }); + lingerTimer.setRemoveOnCancelPolicy(true); + lingerTimer.setKeepAliveTime(5, TimeUnit.SECONDS); + lingerTimer.allowCoreThreadTimeOut(true); + } + return lingerTimer; + } + + private void release(Entry entry) { + RefreshingTokenProvider toClose = null; + synchronized (lock) { + if (--entry.refs > 0) { + return; + } + if (lingerMillis <= 0) { + entries.remove(entry.key, entry); + toClose = entry.provider; + } else { + entry.lingerTask = lingerTimer().schedule(() -> expire(entry), lingerMillis, TimeUnit.MILLISECONDS); + } + } + if (toClose != null) { + toClose.close(); + } + } + + private static final class Entry { + final String key; + final RefreshingTokenProvider provider; + ScheduledFuture lingerTask; + int refs; + + Entry(String key, RefreshingTokenProvider provider) { + this.key = key; + this.provider = provider; + } + } + + /** + * One client's hold on a shared provider. Release it exactly when the client closes; releasing twice is + * harmless. + */ + public final class Lease implements QuietCloseable { + private final Entry entry; + private final AtomicBoolean released = new AtomicBoolean(); + + private Lease(Entry entry) { + this.entry = entry; + } + + @Override + public void close() { + if (released.compareAndSet(false, true)) { + release(entry); + } + } + + /** + * The shared provider. Do not close it: release the lease instead. + */ + public RefreshingTokenProvider provider() { + return entry.provider; + } + + @Override + public String toString() { + return "TokenProviderRegistry.Lease{" + entry.provider.getName() + (released.get() ? ", released}" : "}"); + } + } +} diff --git a/core/src/main/java/io/questdb/client/cutlass/auth/TokenProviderSpec.java b/core/src/main/java/io/questdb/client/cutlass/auth/TokenProviderSpec.java new file mode 100644 index 000000000..82e09019e --- /dev/null +++ b/core/src/main/java/io/questdb/client/cutlass/auth/TokenProviderSpec.java @@ -0,0 +1,248 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.cutlass.auth; + +import io.questdb.client.impl.ConfigView; + +import java.util.Collections; +import java.util.List; +import java.util.Locale; +import java.util.Map; +import java.util.TreeMap; + +/** + * A validated {@code token_provider} selection from a {@code ws}/{@code wss} connect string: the provider name, + * its normalized parameters, and the factory that serves it (design/qwp-token-provider-spec.md, section 7). + *

+ * {@link #parse(ConfigView, boolean)} enforces every rule of section 7.2 and never fetches a token: it only + * looks the factory up. A client turns the spec into a provider when it connects, through + * {@link TokenProviderRegistry#acquire(TokenProviderSpec)}. + * 

+ * wss::addr=qdb1:9000,qdb2:9000;token_provider=azure;azure_resource=api://<questdb-app-id>;azure_client_id=<uami-client-id>;
+ *
+ */ +public final class TokenProviderSpec { + /** + * The {@code token_provider} value of the Microsoft Entra ID provider, served by the optional + * {@code questdb-client-azure} module. + */ + public static final String AZURE = "azure"; + /** + * Reserved for a zero-dependency Azure IMDS provider (decision D5); not implemented. + */ + public static final String AZURE_IMDS = "azure_imds"; + public static final String AZURE_MODULE = "org.questdb:questdb-client-azure"; + public static final String KEY_AZURE_CLIENT_ID = "azure_client_id"; + public static final String KEY_AZURE_RESOURCE = "azure_resource"; + public static final String KEY_TOKEN_PROVIDER = "token_provider"; + private static final String AZURE_DEFAULT_SCOPE_SUFFIX = "/.default"; + private static final String[] STATIC_CREDENTIAL_KEYS = {"token", "username", "password"}; + private final TokenProviderFactory factory; + private final String name; + private final Map params; + private final String registryKey; + + private TokenProviderSpec(TokenProviderFactory factory, String name, Map params) { + this.factory = factory; + this.name = name; + this.params = Collections.unmodifiableMap(params); + StringBuilder key = new StringBuilder(name); + for (Map.Entry e : params.entrySet()) { + key.append('|').append(e.getKey()).append('=').append(e.getValue()); + } + this.registryKey = key.toString(); + } + + /** + * Validates and resolves the {@code token_provider} keys of a {@code ws}/{@code wss} connect string. Never + * fetches a token and never starts a provider. Rejects, naming the offending key: + *
    + *
  • {@code token_provider} combined with {@code token}, {@code username} or {@code password};
    • + *
  • an empty, unknown, reserved or uninstalled {@code token_provider}, listing the supported values;
    • + *
  • a provider-specific key the selected provider does not accept, such as {@code azure_resource} + * without {@code token_provider=azure};
    • + *
  • a missing required provider key;
    • + *
  • {@code token_provider} on {@code ws::}: the bearer token would cross the network in cleartext + * (decision D4).
    • + *
+ * Combining {@code token_provider} with a provider the application supplies is rejected by the builders. + * + * @param view the parsed connect string + * @param tls true for the {@code wss} schema + * @return the spec, or null when the string selects no provider + * @throws IllegalArgumentException on any violation + */ + public static TokenProviderSpec parse(ConfigView view, boolean tls) { + final String name = view.getStr(KEY_TOKEN_PROVIDER); + final String resource = view.getStr(KEY_AZURE_RESOURCE); + final String clientId = view.getStr(KEY_AZURE_CLIENT_ID); + if (name == null) { + if (resource != null) { + throw new IllegalArgumentException(KEY_AZURE_RESOURCE + " requires " + KEY_TOKEN_PROVIDER + '=' + AZURE); + } + if (clientId != null) { + throw new IllegalArgumentException(KEY_AZURE_CLIENT_ID + " requires " + KEY_TOKEN_PROVIDER + '=' + AZURE); + } + return null; + } + for (String key : STATIC_CREDENTIAL_KEYS) { + if (view.has(key)) { + throw new IllegalArgumentException(KEY_TOKEN_PROVIDER + " cannot be combined with " + key); + } + } + if (name.trim().isEmpty()) { + throw new IllegalArgumentException(KEY_TOKEN_PROVIDER + " must not be empty; " + supportedValues()); + } + if (!tls) { + throw new IllegalArgumentException(KEY_TOKEN_PROVIDER + " requires the wss:: schema: over ws:: the bearer " + + "token would cross the network in cleartext"); + } + if (AZURE_IMDS.equals(name)) { + throw new IllegalArgumentException(KEY_TOKEN_PROVIDER + '=' + AZURE_IMDS + + " is reserved and not implemented by this client; " + supportedValues()); + } + final TokenProviderFactory factory = TokenProviderRegistry.findFactory(name); + if (factory == null) { + if (AZURE.equals(name)) { + throw new IllegalArgumentException(KEY_TOKEN_PROVIDER + '=' + AZURE + " requires the " + AZURE_MODULE + + " module on the class path or module path; " + supportedValues()); + } + throw new IllegalArgumentException("unsupported " + KEY_TOKEN_PROVIDER + ": " + safe(name) + "; " + + supportedValues()); + } + if (!AZURE.equals(name)) { + if (resource != null) { + throw new IllegalArgumentException(KEY_AZURE_RESOURCE + " is only valid with " + KEY_TOKEN_PROVIDER + '=' + AZURE); + } + if (clientId != null) { + throw new IllegalArgumentException(KEY_AZURE_CLIENT_ID + " is only valid with " + KEY_TOKEN_PROVIDER + '=' + AZURE); + } + } + final Map params = new TreeMap<>(); + if (AZURE.equals(name)) { + if (resource == null) { + throw new IllegalArgumentException(KEY_TOKEN_PROVIDER + '=' + AZURE + " requires " + KEY_AZURE_RESOURCE); + } + params.put(KEY_AZURE_RESOURCE, normalizeAzureResource(resource)); + if (clientId != null) { + params.put(KEY_AZURE_CLIENT_ID, normalizeGuid(clientId)); + } + } + factory.validate(Collections.unmodifiableMap(params)); + return new TokenProviderSpec(factory, name, params); + } + + /** + * Lower-cases a GUID after checking its {@code 8-4-4-4-12} hex shape. + */ + static String normalizeGuid(String value) { + if (value.length() != 36) { + throw invalidClientId(value); + } + for (int i = 0; i < 36; i++) { + char c = value.charAt(i); + boolean dash = i == 8 || i == 13 || i == 18 || i == 23; + if (dash ? c != '-' : Character.digit(c, 16) < 0) { + throw invalidClientId(value); + } + } + return value.toLowerCase(Locale.ROOT); + } + + private static IllegalArgumentException invalidClientId(String value) { + return new IllegalArgumentException("invalid " + KEY_AZURE_CLIENT_ID + + ": expected a GUID such as 00000000-0000-0000-0000-000000000000, got " + safe(value)); + } + + // The application ID URI (api://) or the client ID of the QuestDB app registration. A trailing + // "/.default" is stripped: the client always requests /.default. + private static String normalizeAzureResource(String value) { + String resource = value; + if (resource.endsWith(AZURE_DEFAULT_SCOPE_SUFFIX)) { + resource = resource.substring(0, resource.length() - AZURE_DEFAULT_SCOPE_SUFFIX.length()); + } + if (resource.isEmpty()) { + throw new IllegalArgumentException(KEY_AZURE_RESOURCE + " must not be empty"); + } + for (int i = 0, n = resource.length(); i < n; i++) { + char c = resource.charAt(i); + if (c <= 0x20 || c > 0x7e) { + throw new IllegalArgumentException(KEY_AZURE_RESOURCE + + " must be an application ID URI (api://) or a client ID without spaces or " + + "non-ASCII characters, got " + safe(resource)); + } + } + return resource; + } + + private static String safe(String value) { + return CredentialRedaction.sanitizeErrorText(value); + } + + private static String supportedValues() { + List names = TokenProviderRegistry.supportedProviders(); + if (names.isEmpty()) { + return "supported values: none installed (token_provider=azure needs " + AZURE_MODULE + ')'; + } + return "supported values: " + names; + } + + /** + * A non-secret label for log lines and errors. + */ + public String describe() { + return factory.describe(params); + } + + public TokenProviderFactory factory() { + return factory; + } + + /** + * The provider name, the value of {@code token_provider}. + */ + public String name() { + return name; + } + + /** + * The normalized provider-specific keys; unmodifiable. + */ + public Map params() { + return params; + } + + /** + * The registry key: the provider name plus the normalized parameters. Equivalent connect strings share it. + */ + public String registryKey() { + return registryKey; + } + + @Override + public String toString() { + return "TokenProviderSpec{" + registryKey + '}'; + } +} diff --git a/core/src/main/java/io/questdb/client/cutlass/auth/TokenSource.java b/core/src/main/java/io/questdb/client/cutlass/auth/TokenSource.java new file mode 100644 index 000000000..31575460d --- /dev/null +++ b/core/src/main/java/io/questdb/client/cutlass/auth/TokenSource.java @@ -0,0 +1,63 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.cutlass.auth; + +/** + * Obtains a new bearer token from an identity platform - Microsoft Entra ID, an OAuth client-credentials + * endpoint, a secrets manager - together with its expiry. A {@link RefreshingTokenProvider} wraps a source and + * caches its tokens, refreshing them in the background before they expire. This is the token-source contract of + * the dynamic-credential specification (design/qwp-token-provider-spec.md, section 4). + *

+ * Threading. The provider calls {@link #fetchToken()} only from its own background refresher thread, never from + * a connection thread, and never concurrently with itself. The call may block, but it should bound its own + * network operations; 30 seconds per attempt is recommended. A provider being closed interrupts its refresher, + * so a source blocked in an interruptible wait should let the interrupt end it. + *

+ * Failures. Throw {@link TokenUnavailableException} to classify a failure: + *

    + *
  • retryable - network failures, timeouts, HTTP 429, HTTP 5xx, and IMDS 404 or 410. Pass the platform's + * {@code Retry-After}, when it gave one, as {@code retryAfterMillis};
    • + *
  • permanent - configuration that is missing or wrong: no credential configured, identity not found, + * invalid client.
    • + *
+ * When unsure, report the failure as retryable. Any other exception is treated as retryable. The provider keeps + * retrying either way - an operator can repair a "permanent" failure, such as a missing role assignment, + * without restarting the application - but the classification decides how a caller waiting for a token reacts. + *

+ * Secrets. A failure message must not contain a token or a raw response body: report only the shape of a + * problem, such as a missing or invalid field. Never attach an object that serializes a raw HTTP request, such + * as a client-credentials body carrying a {@code client_secret}, to an exception. + */ +@FunctionalInterface +public interface TokenSource { + + /** + * Obtains a new token from the identity platform. + * + * @return the token and its expiry; never null + * @throws TokenUnavailableException to report a classified failure; any other exception counts as retryable + */ + ExpiringToken fetchToken(); +} diff --git a/core/src/main/java/io/questdb/client/cutlass/auth/TokenUnavailableException.java b/core/src/main/java/io/questdb/client/cutlass/auth/TokenUnavailableException.java new file mode 100644 index 000000000..6e08754b8 --- /dev/null +++ b/core/src/main/java/io/questdb/client/cutlass/auth/TokenUnavailableException.java @@ -0,0 +1,128 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.cutlass.auth; + +import io.questdb.client.cutlass.line.LineSenderException; + +/** + * No token could be obtained. This is the {@code TokenError} of the dynamic-credential specification + * (design/qwp-token-provider-spec.md, section 4) and also its {@code token-unavailable} error. + *

    + *
  • A {@link TokenSource} throws it to classify a failed fetch.
    • + *
  • A {@link RefreshingTokenProvider} throws it from {@code getToken()} when it holds no usable token and + * cannot get one in time. It then carries the classification of the provider's most recent failed fetch, or + * is retryable when no fetch has failed (the wait timed out, or the caller was interrupted).
    • + *
+ * {@link #isRetryable()} tells a caller whether trying again can succeed. A permanent failure is configuration + * that is missing or wrong; a retryable one is a network failure, a timeout, throttling or a server error. + * {@link #getRetryAfterMillis()} is the platform's suggested wait before the next attempt, or + * {@link #NO_RETRY_AFTER}. + *

+ * Connection code treats an exception from an application-supplied token provider by type: only a + * {@code TokenUnavailableException} marked retryable is retried while a {@code Sender} with + * {@code initial_connect_retry=on} starts up; anything else fails startup fast (decision D8). + *

+ * The message must never contain a token or a raw response body. + */ +public class TokenUnavailableException extends LineSenderException { + /** + * Value of {@link #getRetryAfterMillis()} meaning the platform suggested no wait. + */ + public static final long NO_RETRY_AFTER = -1; + private final long retryAfterMillis; + private final boolean retryable; + + /** + * @param message a description of the failure, never containing a token or a raw response body + * @param retryable whether retrying can succeed + */ + public TokenUnavailableException(CharSequence message, boolean retryable) { + this(message, retryable, NO_RETRY_AFTER); + } + + /** + * @param message a description of the failure, never containing a token or a raw response body + * @param retryable whether retrying can succeed + * @param retryAfterMillis the platform's suggested wait before retrying, or {@link #NO_RETRY_AFTER} + */ + public TokenUnavailableException(CharSequence message, boolean retryable, long retryAfterMillis) { + super(message, retryable); + this.retryable = retryable; + this.retryAfterMillis = retryAfterMillis < 0 ? NO_RETRY_AFTER : retryAfterMillis; + } + + /** + * Carries a cause. Attach only a cause that is safe to log: never one whose message holds a token or a raw + * response body, and never an object that serializes a raw HTTP request. + * + * @param message a description of the failure, never containing a token or a raw response body + * @param retryable whether retrying can succeed + * @param retryAfterMillis the platform's suggested wait before retrying, or {@link #NO_RETRY_AFTER} + * @param cause the underlying failure, may be null + */ + public TokenUnavailableException(String message, boolean retryable, long retryAfterMillis, Throwable cause) { + super(message, cause); + this.retryable = retryable; + this.retryAfterMillis = retryAfterMillis < 0 ? NO_RETRY_AFTER : retryAfterMillis; + } + + /** + * A permanent failure: configuration that is missing or wrong. + */ + public static TokenUnavailableException permanent(CharSequence message) { + return new TokenUnavailableException(message, false); + } + + /** + * A retryable failure without a suggested wait. + */ + public static TokenUnavailableException retryable(CharSequence message) { + return new TokenUnavailableException(message, true); + } + + /** + * A retryable failure with the platform's suggested wait, such as an HTTP {@code Retry-After}. + */ + public static TokenUnavailableException retryable(CharSequence message, long retryAfterMillis) { + return new TokenUnavailableException(message, true, retryAfterMillis); + } + + /** + * @return the platform's suggested wait before the next attempt, in milliseconds, or + * {@link #NO_RETRY_AFTER} when it gave none + */ + public long getRetryAfterMillis() { + return retryAfterMillis; + } + + /** + * @return true when retrying can succeed (a network failure, timeout, throttling or server error), false for + * configuration that is missing or wrong + */ + @Override + public boolean isRetryable() { + return retryable; + } +} diff --git a/core/src/main/java/io/questdb/client/cutlass/http/BearerChallenge.java b/core/src/main/java/io/questdb/client/cutlass/http/BearerChallenge.java new file mode 100644 index 000000000..914f97f00 --- /dev/null +++ b/core/src/main/java/io/questdb/client/cutlass/http/BearerChallenge.java @@ -0,0 +1,158 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.cutlass.http; + +/** + * Reads the {@code error} parameter of a {@code Bearer} challenge in a {@code WWW-Authenticate} header + * (RFC 6750, section 3; RFC 7235, section 4.1). Clients use it to decide whether a {@code 401} is worth a token + * refresh: {@code invalid_token} is, {@code insufficient_scope} is not. + *

+ * The parser is tolerant. It walks a comma-separated list of challenges - {@code scheme [token68 | auth-param, + * ...]} - accepts quoted and unquoted parameter values, and ignores what it cannot read. Only the first + * {@code error} parameter of a {@code Bearer} challenge counts. + */ +public final class BearerChallenge { + /** + * The {@code error} value that means the presented token itself is bad, so a new one can help. + */ + public static final String INVALID_TOKEN = "invalid_token"; + + private BearerChallenge() { + } + + /** + * @param wwwAuthenticate the value of one or more {@code WWW-Authenticate} headers (joined with commas), or + * null + * @return the {@code error} parameter of the first {@code Bearer} challenge that carries one, or null when + * there is no such challenge + */ + public static String bearerError(CharSequence wwwAuthenticate) { + if (wwwAuthenticate == null) { + return null; + } + final int n = wwwAuthenticate.length(); + boolean inBearer = false; + int i = 0; + while (i < n) { + char c = wwwAuthenticate.charAt(i); + if (c == ',' || isWhitespace(c)) { + i++; + continue; + } + final int start = i; + while (i < n && isTokenChar(wwwAuthenticate.charAt(i))) { + i++; + } + if (i == start) { + i++; // not a token character: skip it + continue; + } + final String token = wwwAuthenticate.subSequence(start, i).toString(); + int j = skipWhitespace(wwwAuthenticate, i); + if (j < n && wwwAuthenticate.charAt(j) == '=') { + // auth-param: name = ( token / quoted-string ) + j = skipWhitespace(wwwAuthenticate, j + 1); + final String value; + if (j < n && wwwAuthenticate.charAt(j) == '"') { + final StringBuilder sb = new StringBuilder(); + j++; + while (j < n) { + char q = wwwAuthenticate.charAt(j++); + if (q == '\\' && j < n) { + sb.append(wwwAuthenticate.charAt(j++)); + } else if (q == '"') { + break; + } else { + sb.append(q); + } + } + value = sb.toString(); + } else { + final int valueStart = j; + while (j < n && wwwAuthenticate.charAt(j) != ',' && !isWhitespace(wwwAuthenticate.charAt(j))) { + j++; + } + value = wwwAuthenticate.subSequence(valueStart, j).toString(); + } + if (inBearer && "error".equalsIgnoreCase(token)) { + return value; + } + i = j; + } else { + // a new challenge's auth-scheme (or a token68, which carries no parameters) + inBearer = "Bearer".equalsIgnoreCase(token); + i = j; + } + } + return null; + } + + /** + * Whether a {@code 401} carrying {@code wwwAuthenticate} warrants refreshing the token: true when the header + * has no {@code Bearer} challenge with an {@code error} parameter (the status code alone decides), or when + * that error is {@code invalid_token}. + */ + public static boolean allowsTokenRefresh(CharSequence wwwAuthenticate) { + final String error = bearerError(wwwAuthenticate); + return error == null || INVALID_TOKEN.equals(error); + } + + private static boolean isTokenChar(char c) { + if ((c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || (c >= '0' && c <= '9')) { + return true; + } + switch (c) { + case '!': + case '#': + case '$': + case '%': + case '&': + case '\'': + case '*': + case '+': + case '-': + case '.': + case '^': + case '_': + case '`': + case '|': + case '~': + return true; + default: + return false; + } + } + + private static boolean isWhitespace(char c) { + return c == ' ' || c == '\t'; + } + + private static int skipWhitespace(CharSequence s, int i) { + while (i < s.length() && isWhitespace(s.charAt(i))) { + i++; + } + return i; + } +} diff --git a/core/src/main/java/io/questdb/client/cutlass/http/client/WebSocketClient.java b/core/src/main/java/io/questdb/client/cutlass/http/client/WebSocketClient.java index 1290fcab5..8c8d07bd8 100644 --- a/core/src/main/java/io/questdb/client/cutlass/http/client/WebSocketClient.java +++ b/core/src/main/java/io/questdb/client/cutlass/http/client/WebSocketClient.java @@ -163,6 +163,11 @@ public abstract class WebSocketClient implements QuietCloseable { private int serverNegotiatedZstdLevel; private int serverQwpVersion = 1; private String upgradeRejectRole; + // WWW-Authenticate value(s) from the most recent rejected upgrade, joined with ", " when the server sent + // several, or null when absent. Lets a 401 carrying "Bearer error=insufficient_scope" be told apart from an + // expired token, so a dynamic credential is refreshed only when a new token can help. Reset to null on every + // upgrade() invocation. + private String upgradeRejectWwwAuthenticate; // Server-advertised zone identifier from the most recent rejected upgrade, // captured from the X-QuestDB-Zone response header on a 421. Null when the // header was absent or empty. Per failover.md §5 servers SHOULD emit this @@ -374,6 +379,14 @@ public String getUpgradeRejectRole() { return upgradeRejectRole; } + /** + * {@code WWW-Authenticate} value(s) on the most recent rejected upgrade, joined with {@code ", "} when the + * server sent several, or null when absent. + */ + public String getUpgradeRejectWwwAuthenticate() { + return upgradeRejectWwwAuthenticate; + } + /** * Zone identifier from {@code X-QuestDB-Zone} on the most recent rejected * upgrade, or null when the header was absent or empty (after trimming). @@ -648,6 +661,7 @@ public void upgrade(CharSequence path, int timeout, CharSequence authorizationHe return; // Already upgraded } upgradeRejectRole = null; + upgradeRejectWwwAuthenticate = null; upgradeRejectZone = null; upgradeStatusCode = 0; @@ -878,6 +892,34 @@ private static String extractRoleHeader(String response) { return null; } + // Every value of the named header, joined with ", " (RFC 7230 section 3.2.2), or null when absent. + private static String extractHeaderValues(String response, String headerName) { + int headerLen = headerName.length(); + int responseLen = response.length(); + StringBuilder values = null; + int lineStart = response.indexOf("\r\n"); + while (lineStart >= 0 && lineStart + 2 + headerLen <= responseLen) { + int hStart = lineStart + 2; + if (response.regionMatches(true, hStart, headerName, 0, headerLen)) { + int valueStart = hStart + headerLen; + int lineEnd = response.indexOf('\r', valueStart); + if (lineEnd < 0) { + lineEnd = responseLen; + } + String value = response.substring(valueStart, lineEnd).trim(); + if (!value.isEmpty()) { + if (values == null) { + values = new StringBuilder(value); + } else { + values.append(", ").append(value); + } + } + } + lineStart = response.indexOf("\r\n", hStart); + } + return values == null ? null : values.toString(); + } + private static String extractZoneHeader(String response) { int headerLen = QUESTDB_ZONE_HEADER_NAME.length(); int responseLen = response.length(); @@ -1346,6 +1388,9 @@ private void validateUpgradeResponse(int headerEnd) { if (!response.startsWith("HTTP/1.1 101")) { String statusLine = response.split("\r\n")[0]; upgradeStatusCode = parseStatusCode(statusLine); + if (upgradeStatusCode == 401) { + upgradeRejectWwwAuthenticate = extractHeaderValues(response, "WWW-Authenticate:"); + } if (upgradeStatusCode == 421) { upgradeRejectRole = extractRoleHeader(response); upgradeRejectZone = extractZoneHeader(response); diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpAuthFailedException.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpAuthFailedException.java index d2a5714a2..e14c3a7da 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpAuthFailedException.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpAuthFailedException.java @@ -24,26 +24,61 @@ package io.questdb.client.cutlass.qwp.client; +import io.questdb.client.cutlass.auth.CredentialRedaction; +import io.questdb.client.cutlass.http.BearerChallenge; import io.questdb.client.cutlass.http.client.HttpClientException; /** - * WebSocket upgrade rejected with {@code 401} or {@code 403}. Terminal across all - * configured endpoints: a rejected credential is uniformly rejected across the - * cluster, so failing fast surfaces the configuration error immediately. Path - * mismatches ({@code 404}) are NOT routed through this exception because a single - * misconfigured node mid-deploy can return 404 while peers are healthy. + * WebSocket upgrade rejected with {@code 401} or {@code 403}: the {@code auth-rejected} failure class of the + * dynamic-credential specification (design/qwp-token-provider-spec.md, section 8.1). The message starts with + * {@code auth-rejected} so every report of it names the class. + *

+ * A credential is uniformly accepted or rejected across a cluster, so the connect walk stops at the first such + * rejection instead of trying the remaining endpoints. What happens next depends on the phase (section 8.3): + * startup fails, an established store-and-forward sender keeps retrying, an orphan drainer rides a rotating + * credential out before quarantining, and a query fails. Before any of that, a {@code 401} against a dynamic + * credential earns one same-endpoint retry with a refreshed token (section 8.2) - see + * {@link #isTokenRefreshable()}. Path mismatches ({@code 404}) are NOT routed through this exception because a + * single misconfigured node mid-deploy can return 404 while peers are healthy. */ public final class QwpAuthFailedException extends HttpClientException { + private static final int MAX_BEARER_ERROR_LENGTH = 64; + private final String bearerError; private final String host; private final int port; private final int statusCode; public QwpAuthFailedException(int statusCode, String host, int port) { - super("WebSocket upgrade rejected with HTTP "); - put(statusCode).put(" for ").put(host).put(':').put(port); + this(statusCode, host, port, null); + } + + /** + * @param wwwAuthenticate the {@code WWW-Authenticate} header value(s) of the rejection, or null + */ + public QwpAuthFailedException(int statusCode, String host, int port, String wwwAuthenticate) { + super("auth-rejected: WebSocket upgrade rejected with HTTP "); + put(statusCode); + String error = BearerChallenge.bearerError(wwwAuthenticate); + if (error != null) { + error = CredentialRedaction.sanitizeErrorText(error); + if (error.length() > MAX_BEARER_ERROR_LENGTH) { + error = error.substring(0, MAX_BEARER_ERROR_LENGTH); + } + put(" [error=").put(error).put(']'); + } + put(" for ").put(host).put(':').put(port); this.statusCode = statusCode; this.host = host; this.port = port; + this.bearerError = error; + } + + /** + * The {@code error} parameter of the rejection's {@code WWW-Authenticate: Bearer} challenge (for example + * {@code invalid_token} or {@code insufficient_scope}), sanitized, or null when the server sent none. + */ + public String getBearerError() { + return bearerError; } public String getHost() { @@ -57,4 +92,13 @@ public int getPort() { public int getStatusCode() { return statusCode; } + + /** + * Whether a new token can help: the status is {@code 401}, and the server either sent no {@code Bearer} + * challenge with an {@code error} or sent {@code error="invalid_token"} (RFC 6750, section 3.1). A + * {@code 403} never qualifies - it is an authorization decision a new token does not change (decision D2). + */ + public boolean isTokenRefreshable() { + return statusCode == 401 && (bearerError == null || BearerChallenge.INVALID_TOKEN.equals(bearerError)); + } } diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpConnectionHealthTracker.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpConnectionHealthTracker.java new file mode 100644 index 000000000..a26b681b4 --- /dev/null +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpConnectionHealthTracker.java @@ -0,0 +1,182 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.cutlass.qwp.client; + +import io.questdb.client.ConnectionHealth; +import io.questdb.client.cutlass.auth.CredentialRedaction; +import io.questdb.client.cutlass.http.client.HttpClientException; +import io.questdb.client.cutlass.http.client.WebSocketUpgradeException; +import io.questdb.client.cutlass.line.LineSenderException; + +/** + * Tracks one QWP client's connection health (design/qwp-token-provider-spec.md, section 8.4) and publishes it as + * an immutable {@link ConnectionHealth} snapshot. Connection code reports transitions - a successful upgrade, a + * lost connection, a failed connect round, a terminal failure, close - from whichever thread observes them; + * readers get the last published snapshot with a single volatile read and never wait on connection threads. + * Writers serialize on a private monitor that is never held across I/O. + */ +public final class QwpConnectionHealthTracker { + private static final int MAX_CAUSE_DEPTH = 16; + private final Object lock = new Object(); + private long failedRounds; + private ConnectionHealth.Failure lastFailure; + private long lastConnectedAt = ConnectionHealth.NONE; + private long outageSince; + private volatile ConnectionHealth snapshot; + private ConnectionHealth.State state = ConnectionHealth.State.CONNECTING; + + public QwpConnectionHealthTracker() { + // A client that has never connected has been without a connection since it was created. + this.outageSince = System.currentTimeMillis(); + publish(); + } + + /** + * Classifies a failed connect round's exception into the failure classes of section 8.4. + */ + public static ConnectionHealth.Failure classify(Throwable failure, long nowMillis) { + Throwable t = failure; + for (int depth = 0; t != null && depth < MAX_CAUSE_DEPTH; depth++, t = t.getCause()) { + if (t instanceof QwpCredentialUnavailableException) { + return failure(ConnectionHealth.FailureClass.CREDENTIAL_UNAVAILABLE, 0, t, nowMillis); + } + if (t instanceof QwpAuthFailedException) { + return failure(ConnectionHealth.FailureClass.AUTH_REJECTED, + ((QwpAuthFailedException) t).getStatusCode(), t, nowMillis); + } + if (t instanceof QwpRoleMismatchException || t instanceof QwpIngressRoleRejectedException) { + return failure(ConnectionHealth.FailureClass.ROLE_REJECTED, 0, t, nowMillis); + } + if (t instanceof WebSocketUpgradeException) { + WebSocketUpgradeException e = (WebSocketUpgradeException) t; + return e.isRoleMismatch() + ? failure(ConnectionHealth.FailureClass.ROLE_REJECTED, 0, t, nowMillis) + : failure(ConnectionHealth.FailureClass.OTHER, Math.max(0, e.getStatusCode()), t, nowMillis); + } + if (t instanceof QwpVersionMismatchException || t instanceof QwpDurableAckMismatchException) { + return failure(ConnectionHealth.FailureClass.OTHER, 0, t, nowMillis); + } + } + return failure(failure instanceof HttpClientException || failure instanceof LineSenderException + ? ConnectionHealth.FailureClass.TRANSPORT + : ConnectionHealth.FailureClass.OTHER, 0, failure, nowMillis); + } + + private static ConnectionHealth.Failure failure( + ConnectionHealth.FailureClass failureClass, + int statusCode, + Throwable t, + long nowMillis + ) { + String message = CredentialRedaction.sanitizeErrorText(t.getMessage()); + return new ConnectionHealth.Failure(failureClass, statusCode, + message == null || message.isEmpty() ? t.getClass().getSimpleName() : message, nowMillis); + } + + /** + * The client was closed. Sticky. + */ + public void closed() { + synchronized (lock) { + state = ConnectionHealth.State.CLOSED; + publish(); + } + } + + /** + * An established connection was lost and is being re-established. + */ + public void connectionLost() { + synchronized (lock) { + if (state == ConnectionHealth.State.CONNECTED) { + state = ConnectionHealth.State.RECONNECTING; + outageSince = System.currentTimeMillis(); + failedRounds = 0; + publish(); + } + } + } + + /** + * The client will not connect again. Sticky; keeps the last connect-round failure. + */ + public void failed() { + synchronized (lock) { + if (state != ConnectionHealth.State.CLOSED) { + state = ConnectionHealth.State.FAILED; + publish(); + } + } + } + + /** + * A connect round - one walk over the configured endpoints - ended without a connection. + */ + public void roundFailed(Throwable failure) { + final long now = System.currentTimeMillis(); + final ConnectionHealth.Failure classified = classify(failure, now); + synchronized (lock) { + if (state == ConnectionHealth.State.FAILED || state == ConnectionHealth.State.CLOSED) { + return; + } + if (state == ConnectionHealth.State.CONNECTED) { + state = ConnectionHealth.State.RECONNECTING; + outageSince = now; + failedRounds = 0; + } + failedRounds++; + lastFailure = classified; + publish(); + } + } + + /** + * @return the current snapshot; never null + */ + public ConnectionHealth snapshot() { + return snapshot; + } + + /** + * A connect round ended with a successful upgrade. + */ + public void upgraded() { + synchronized (lock) { + if (state == ConnectionHealth.State.FAILED || state == ConnectionHealth.State.CLOSED) { + return; + } + state = ConnectionHealth.State.CONNECTED; + lastConnectedAt = System.currentTimeMillis(); + outageSince = ConnectionHealth.NONE; + failedRounds = 0; + publish(); + } + } + + // caller holds lock + private void publish() { + snapshot = new ConnectionHealth(state, lastConnectedAt, outageSince, failedRounds, lastFailure); + } +} diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpCredentialUnavailableException.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpCredentialUnavailableException.java index 2652a7f79..963119527 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpCredentialUnavailableException.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpCredentialUnavailableException.java @@ -24,23 +24,28 @@ package io.questdb.client.cutlass.qwp.client; +import io.questdb.client.cutlass.auth.TokenUnavailableException; import io.questdb.client.cutlass.line.LineSenderException; /** * Signals that the client could not OBTAIN an Authorization credential for a * (re)connect handshake: the configured {@code httpTokenProvider} threw instead of - * returning a token -- a failed silent refresh, or no sign-in yet. + * returning a token -- a failed silent refresh, or no sign-in yet. This is the + * {@code credential-unavailable} failure class of the dynamic-credential specification + * (design/qwp-token-provider-spec.md, section 8.1). *

* Distinct from {@link QwpAuthFailedException}, which means the server rejected a - * credential the client did present (a terminal auth failure). A credential the client - * cannot ACQUIRE is instead handled by connection phase, exactly like a transport outage: + * credential the client did present. A credential the client cannot ACQUIRE is instead + * handled by connection phase, exactly like a transport outage: * the RUNNING store-and-forward drainer retries it indefinitely with capped backoff under * Invariant B -- the IdP becomes reachable again, or the user completes an interactive * sign-in -- holding the un-acked rows in SF meanwhile, and NEVER bounds it by * {@code reconnectMaxDurationMillis} nor latches a terminal (either would drop a producer - * store-and-forward promised to keep alive). Only the foreground/SYNC initial connect - * fails fast, because a connectivity error is the caller's to see during initialization, - * not after the drainer is running. + * store-and-forward promised to keep alive). During initialization the classification + * decides: {@link #isRetryable()} is true only when the provider threw a + * {@link TokenUnavailableException} marked retryable, which a SYNC initial connect keeps + * retrying within its budget; any other provider failure is permanent and fails startup + * fast (decision D8), as does any failure on an OFF initial connect. *

* It exists so the send loop can tell "the provider failed" apart from "the network * failed", and it carries the provider's own exception so a handler can surface that @@ -52,7 +57,9 @@ * connect in {@code QwpWebSocketSender} - both catch it and rethrow * {@link #providerFailure()}, so a token-provider failure reaches the caller as the * provider's own exception; the running background drainer catches it and retries under - * the invariant above. It is public because both of those packages handle it, and + * the invariant above. The query client ({@code QwpQueryClient}) does throw it from + * {@code connect()}, with a message that starts with {@code credential-unavailable}. It is + * public because both of those packages handle it, and * because {@code QwpWebSocketSender.newReconnectFactory()} is public: a caller that * drives {@code ReconnectFactory.reconnect()} itself runs the endpoint walk directly and * so can receive this type unwrapped. Such a caller should treat it as the provider @@ -63,12 +70,30 @@ public class QwpCredentialUnavailableException extends LineSenderException { private final RuntimeException providerFailure; public QwpCredentialUnavailableException(RuntimeException providerFailure) { - super(providerFailure.getMessage() == null + this(providerFailure.getMessage() == null ? "token provider failed to supply a credential" : providerFailure.getMessage(), providerFailure); + } + + /** + * @param message the message, which must not contain a token + * @param providerFailure the exception the token provider threw + */ + public QwpCredentialUnavailableException(String message, RuntimeException providerFailure) { + super(message, providerFailure); this.providerFailure = providerFailure; } + /** + * True only when the provider threw a {@link TokenUnavailableException} marked retryable. Any other + * exception from a provider counts as a permanent credential failure (decision D8), so that startup fails + * fast, as it does for an OIDC device-flow provider that is not signed in yet. + */ + @Override + public boolean isRetryable() { + return providerFailure instanceof TokenUnavailableException && ((TokenUnavailableException) providerFailure).isRetryable(); + } + /** * The exception the token provider threw, for a caller that must surface the * provider's own error rather than this wrapper. Never null: the wrapper is only diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpQueryClient.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpQueryClient.java index bfc924e52..380fc0eae 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpQueryClient.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpQueryClient.java @@ -25,12 +25,14 @@ package io.questdb.client.cutlass.qwp.client; import io.questdb.client.ClientTlsConfiguration; +import io.questdb.client.ConnectionHealth; import io.questdb.client.HttpTokenProvider; import io.questdb.client.cutlass.http.client.HttpClientException; import io.questdb.client.cutlass.http.client.WebSocketClient; import io.questdb.client.cutlass.http.client.WebSocketClientFactory; import io.questdb.client.cutlass.http.client.WebSocketFrameHandler; -import io.questdb.client.cutlass.line.LineSenderException; +import io.questdb.client.cutlass.auth.TokenProviderRegistry; +import io.questdb.client.cutlass.auth.TokenProviderSpec; import io.questdb.client.cutlass.qwp.protocol.QwpConstants; import io.questdb.client.impl.ConfigString; import io.questdb.client.impl.ConfigView; @@ -151,6 +153,8 @@ public class QwpQueryClient implements QuietCloseable { * is already in the client's kernel recv buffer by the time this wait starts. */ private static final int DEFAULT_SERVER_INFO_TIMEOUT_MS = 5_000; + // Warn once per process when an application-supplied token provider is used over ws:: (spec section 9). + private static final AtomicBoolean CLEARTEXT_PROVIDER_WARNED = new AtomicBoolean(); private static final Logger LOG = LoggerFactory.getLogger(QwpQueryClient.class); // Reusable typed bind-value sink. Populated on the user thread by the // {@link QwpBindSetter} passed to execute(); the pre-encoded bytes are @@ -166,6 +170,8 @@ public class QwpQueryClient implements QuietCloseable { private final List endpoints = new ArrayList<>(); private final AtomicBoolean executing = new AtomicBoolean(); private final Random failoverRandom = new Random(); + // Connection health (design/qwp-token-provider-spec.md, section 8.4). + private final QwpConnectionHealthTracker healthTracker = new QwpConnectionHealthTracker(); private long authTimeoutMs = DEFAULT_AUTH_TIMEOUT_MS; private String authorizationHeader; // Deterministic lifecycle barrier used by facade shutdown tests. Null in @@ -270,6 +276,9 @@ public class QwpQueryClient implements QuietCloseable { // currentRequestId. Volatile so a cancel from any thread is visible to the // worker thread's post-requestId read. private volatile boolean pendingCancel; + // Set when a failover reconnect inside execute() failed, leaving the client disconnected: the next + // execute() reconnects instead of throwing "not connected". Cleared by every successful connect. + private volatile boolean reconnectOnNextExecute; // Decoded SERVER_INFO from the current connection's handshake. Null before // connect() has succeeded; non-null on every established connection (the // server always emits the frame). Volatile so getServerInfo(), callable @@ -300,6 +309,12 @@ public class QwpQueryClient implements QuietCloseable { // token rotation. Mutually exclusive with the fixed authorizationHeader // synthesized by withBearerToken/withBasicAuth; null when unset. private HttpTokenProvider tokenProvider; + // The registry lease behind tokenProvider when the connect string selected a token_provider. Acquired on + // the first connect() - never by fromConfig(), so building or validating a client fetches no token - and + // released by close(). + private volatile TokenProviderRegistry.Lease tokenProviderLease; + // A token_provider selected by the connect string (wss:: only); null otherwise. + private TokenProviderSpec tokenProviderSpec; private char[] trustStorePassword; private String trustStorePath; private volatile WebSocketClient webSocketClient; @@ -385,6 +400,7 @@ public static QwpQueryClient fromConfig(CharSequence configurationString) { } ConfigView view = new ConfigView(cs); validateConfig(view, tls); + TokenProviderSpec tokenProviderSpec = TokenProviderSpec.parse(view, tls); List parsedEndpoints = new ArrayList<>(); view.getHostPorts("addr", DEFAULT_WS_PORT, (h, p) -> parsedEndpoints.add(new Endpoint(h, p))); @@ -484,6 +500,7 @@ public static QwpQueryClient fromConfig(CharSequence configurationString) { } if (hasBasic) client.withBasicAuth(username, password); if (token != null) client.withBearerToken(token); + client.tokenProviderSpec = tokenProviderSpec; if (cid != null) client.withClientId(cid); if (maxBatchRows > 0) client.withMaxBatchRows(maxBatchRows); if (zone != null) client.withZone(zone); @@ -561,6 +578,9 @@ public static void validateConfig(ConfigView view, boolean tls) { if (tlsRoots != null && "unsafe_off".equals(tlsVerify)) { throw new IllegalArgumentException(TLS_ROOTS_INSECURE_CONFIG_ERROR); } + // token_provider and its keys (design/qwp-token-provider-spec.md, section 7.2): resolves the factory, never + // fetches a token. + TokenProviderSpec.parse(view, tls); // Mirror fromConfig's effective values: a missing bound takes its // default, so the ordering is enforced even when only one key is set // (e.g. failover_backoff_max_ms alone, below the default initial backoff). @@ -640,6 +660,7 @@ public void close() { // scratch, double-freeing it. return; } + healthTracker.closed(); Runnable hook = beforeCloseHook; beforeCloseHook = null; if (hook != null) { @@ -720,6 +741,13 @@ public void close() { // (submitQuery copies its bytes into sendScratch), so it is safe to free // even when we otherwise leak the I/O thread and buffer pool. bindValues.close(); + // Release the token_provider lease: the registry keeps the shared provider alive for its linger + // period, which also covers an I/O thread that failed to join above. + TokenProviderRegistry.Lease lease = tokenProviderLease; + tokenProviderLease = null; + if (lease != null) { + lease.close(); + } if (wasInterrupted) { // Hand the caller's cancellation back exactly as it arrived. Restoring it here rather // than earlier keeps it out of the joins above, which is the whole point. @@ -758,6 +786,36 @@ public synchronized void connect() { if (connected) { return; } + try { + connectWalk(); + healthTracker.upgraded(); + } catch (RuntimeException e) { + healthTracker.roundFailed(e); + throw e; + } + } + + /** + * Connection health (design/qwp-token-provider-spec.md, section 8.4). A query client connects on demand, so + * {@link ConnectionHealth.State#RECONNECTING} means its last connect or failover reconnect failed and the next + * operation will try again. Cheap and safe from any thread; never contains a credential. + */ + public ConnectionHealth health() { + return healthTracker.snapshot(); + } + + // One connect round: resolve the credential, walk the endpoints. See connect(). + private void connectWalk() { + if (tokenProviderSpec != null && tokenProviderLease == null) { + // The connect string selected a token_provider: share the process-wide provider for it (spec section + // 7.4). The lease is this client's until close(). + tokenProviderLease = TokenProviderRegistry.global().acquire(tokenProviderSpec); + tokenProvider = tokenProviderLease.provider(); + } else if (tokenProvider != null && !tlsEnabled && CLEARTEXT_PROVIDER_WARNED.compareAndSet(false, true)) { + // spec section 9: an application-supplied provider over ws:: is the application's call - warn once + LOG.warn("a token provider is used over ws:: (no TLS): bearer tokens cross the network in cleartext; " + + "use wss:: in production"); + } lastCloseTimedOut = false; if (hostTracker == null) { hostTracker = new QwpHostHealthTracker( @@ -774,8 +832,18 @@ public synchronized void connect() { // instead of being folded into "all endpoints unreachable", and avoids re-querying the provider // once per endpoint. String authHeader = resolveAuthorizationHeader(); + // One same-endpoint retry after a refreshable 401 per connect, for a token provider only + // (design/qwp-token-provider-spec.md, section 8.2). The retried attempt records no health penalty. + boolean authRetryAvailable = tokenProvider != null; + int retryIdx = -1; while (true) { - int i = hostTracker.pickNext(); + int i; + if (retryIdx >= 0) { + i = retryIdx; + retryIdx = -1; + } else { + i = hostTracker.pickNext(); + } if (i < 0) { break; } @@ -784,6 +852,17 @@ public synchronized void connect() { connectToEndpoint(ep, authHeader); } catch (QwpAuthFailedException ae) { cleanupFailedConnect(); + if (authRetryAvailable && ae.isTokenRefreshable()) { + authRetryAvailable = false; + String refreshed = refreshAfterRejection(authHeader, ae); + if (refreshed != null && !refreshed.equals(authHeader)) { + LOG.info("QwpQueryClient {}:{} rejected the token with {}; retrying once with a refreshed token", + ep.host, ep.port, ae.getStatusCode()); + authHeader = refreshed; + retryIdx = i; + continue; + } + } throw ae; } catch (QwpIngressRoleRejectedException re) { lastTransportError = re; @@ -819,6 +898,7 @@ public synchronized void connect() { spawnIoThread(); hostTracker.recordSuccess(i); currentEndpointIndex = i; + reconnectOnNextExecute = false; connected = true; return; } @@ -973,6 +1053,11 @@ public java.util.Map configSnapshotForTest() { m.put("tls_verify", tlsValidationMode); m.put("tls_roots", trustStorePath); m.put("tls_roots_password", trustStorePassword == null ? null : new String(trustStorePassword)); + m.put("token_provider", tokenProviderSpec == null ? null : tokenProviderSpec.name()); + m.put("azure_resource", tokenProviderSpec == null ? null + : tokenProviderSpec.params().get(TokenProviderSpec.KEY_AZURE_RESOURCE)); + m.put("azure_client_id", tokenProviderSpec == null ? null + : tokenProviderSpec.params().get(TokenProviderSpec.KEY_AZURE_CLIENT_ID)); return m; } @@ -1126,6 +1211,9 @@ public QwpQueryClient withConnectTimeout(int connectTimeoutMs) { */ public QwpQueryClient withBasicAuth(String username, String password) { checkPreConnect("withBasicAuth"); + if (tokenProviderSpec != null) { + throw new IllegalStateException("withBasicAuth cannot be combined with token_provider in the configuration"); + } if (tokenProvider != null) { throw new IllegalStateException("withBasicAuth cannot be combined with withBearerTokenProvider"); } @@ -1146,6 +1234,9 @@ public QwpQueryClient withBasicAuth(String username, String password) { */ public QwpQueryClient withBearerToken(String token) { checkPreConnect("withBearerToken"); + if (tokenProviderSpec != null) { + throw new IllegalStateException("withBearerToken cannot be combined with token_provider in the configuration"); + } if (tokenProvider != null) { throw new IllegalStateException("withBearerToken cannot be combined with withBearerTokenProvider"); } @@ -1179,6 +1270,10 @@ public QwpQueryClient withBearerTokenProvider(HttpTokenProvider provider) { if (provider == null) { throw new IllegalArgumentException("provider must not be null"); } + if (tokenProviderSpec != null) { + throw new IllegalStateException( + "withBearerTokenProvider cannot be combined with token_provider in the configuration"); + } if (authorizationHeader != null) { throw new IllegalStateException("withBearerTokenProvider cannot be combined with withBearerToken or withBasicAuth"); } @@ -1578,7 +1673,20 @@ private void executeImpl(CharSequence sql, QwpBindSetter binds, QwpColumnBatchHa throw new IllegalStateException("QwpQueryClient is closed"); } if (!connected) { - throw new IllegalStateException("QwpQueryClient not connected; call connect() first"); + if (!reconnectOnNextExecute) { + throw new IllegalStateException("QwpQueryClient not connected; call connect() first"); + } + // A previous failover reconnect failed. Reconnect on this operation rather than leave the client + // permanently unusable (design/qwp-token-provider-spec.md, section 8.3, "Egress recovery") -- a + // pooled client is never discarded by its pool on its own, so without this a single failed + // failover (say, a token outage) would poison the slot for the life of the pool. + try { + connect(); + } catch (RuntimeException e) { + handler.onError(-1L, WebSocketResponse.STATUS_INTERNAL_ERROR, + "reconnect failed: " + e.getMessage()); + return; + } } hostTracker.beginRound(false); long failoverDeadlineNanos; @@ -1602,6 +1710,9 @@ private void executeImpl(CharSequence sql, QwpBindSetter binds, QwpColumnBatchHa return; } if (!failoverEnabled) { + // With failover off the transport failure stays latched: every later execute() reports it. + healthTracker.connectionLost(); + healthTracker.failed(); handler.onError(probe.interceptedRequestId, probe.interceptedStatus, probe.interceptedMessage); return; } @@ -1621,6 +1732,10 @@ private void executeImpl(CharSequence sql, QwpBindSetter binds, QwpColumnBatchHa } cleanupFailedConnect(); connected = false; + healthTracker.connectionLost(); + // Every exit from here on - an exhausted deadline, an interrupted backoff, a failed reconnect - leaves + // the client disconnected; the next execute() reconnects rather than throw "not connected". + reconnectOnNextExecute = true; if (failoverInitialBackoffMs > 0L) { long base = failoverInitialBackoffMs << Math.min(attempt - 1, 30); if (base < 0L) base = failoverMaxBackoffMs; @@ -1661,18 +1776,31 @@ private void executeImpl(CharSequence sql, QwpBindSetter binds, QwpColumnBatchHa } } try { - reconnectViaTracker(); + try { + reconnectViaTracker(); + healthTracker.upgraded(); + } catch (RuntimeException e) { + healthTracker.roundFailed(e); + throw e; + } } catch (QwpAuthFailedException authErr) { // failover.md S6: AuthError is terminal across all hosts. // Credentials are cluster-wide, so retrying floods server logs // without recovery. Surface a distinct message so monitoring - // can pull auth incidents apart from generic transport failures. + // can pull auth incidents apart from generic transport failures; + // it names the failure class (design/qwp-token-provider-spec.md, 8.3). handler.onError(probe.interceptedRequestId, probe.interceptedStatus, - "auth failure during failover reconnect [host=" + "auth-rejected during failover reconnect [host=" + authErr.getHost() + ':' + authErr.getPort() + ", status=" + authErr.getStatusCode() + ", last error: " + probe.interceptedMessage + ']'); return; + } catch (QwpCredentialUnavailableException credentialErr) { + // The token provider could not supply a credential; no endpoint was contacted. + handler.onError(probe.interceptedRequestId, probe.interceptedStatus, + "failover reconnect failed: " + credentialErr.getMessage() + + " [last error: " + probe.interceptedMessage + ']'); + return; } catch (RuntimeException reconnectErr) { handler.onError(probe.interceptedRequestId, probe.interceptedStatus, "failover reconnect failed after " + attempt + " attempt" @@ -1873,8 +2001,16 @@ private void reconnectViaTracker() { // reason as connect(): a provider failure is cluster-wide, so surface it directly rather than // as a per-endpoint transport error retried across every host. String authHeader = resolveAuthorizationHeader(); + boolean authRetryAvailable = tokenProvider != null; + int retryIdx = -1; while (true) { - int i = hostTracker.pickNext(); + int i; + if (retryIdx >= 0) { + i = retryIdx; + retryIdx = -1; + } else { + i = hostTracker.pickNext(); + } if (i < 0) { if (!retriedAfterReset) { hostTracker.beginRound(true); @@ -1888,6 +2024,17 @@ private void reconnectViaTracker() { connectToEndpoint(ep, authHeader); } catch (QwpAuthFailedException ae) { cleanupFailedConnect(); + if (authRetryAvailable && ae.isTokenRefreshable()) { + authRetryAvailable = false; + String refreshed = refreshAfterRejection(authHeader, ae); + if (refreshed != null && !refreshed.equals(authHeader)) { + LOG.info("QwpQueryClient {}:{} rejected the token with {} on failover; retrying once with " + + "a refreshed token", ep.host, ep.port, ae.getStatusCode()); + authHeader = refreshed; + retryIdx = i; + continue; + } + } throw ae; } catch (QwpIngressRoleRejectedException re) { lastError = re; @@ -1917,6 +2064,7 @@ private void reconnectViaTracker() { spawnIoThread(); hostTracker.recordSuccess(i); currentEndpointIndex = i; + reconnectOnNextExecute = false; connected = true; return; } @@ -1934,30 +2082,54 @@ private String resolveAuthorizationHeader() { // With a token provider, query it once per connect()/reconnect (the caller resolves before the // endpoint walk) so a reconnect presents a freshly refreshed token; validateToken rejects a // null/empty/blank return, or one carrying a control or non-ASCII character, before it reaches - // the "Bearer " header. A provider that throws (a failed silent refresh, or not signed in yet) - // fails connect()/reconnect as a LineSenderException, preserving the provider failure as its cause. + // the "Bearer " header. A provider that throws (a failed silent refresh, or not signed in yet), or + // returns a token that fails validation, fails connect()/reconnect with the credential-unavailable + // failure class (design/qwp-token-provider-spec.md, section 8.1): a QwpCredentialUnavailableException + // whose message names the class and whose cause is the provider failure. if (tokenProvider != null) { - CharSequence pulled; + String token; try { - pulled = tokenProvider.getToken(); - } catch (LineSenderException e) { - throw e; + CharSequence pulled = tokenProvider.getToken(); + // snapshot before validating, for the reason HttpTokenProvider.validateToken gives: the + // concatenation below re-reads the sequence, and the provider may be reusing its buffer + token = pulled == null ? null : pulled.toString(); + HttpTokenProvider.validateToken(token); } catch (RuntimeException e) { - throw new LineSenderException( - e.getMessage() == null + throw new QwpCredentialUnavailableException( + "credential-unavailable: " + (e.getMessage() == null ? "token provider failed to supply a credential" - : e.getMessage(), + : e.getMessage()), e); } - // snapshot before validating, for the reason HttpTokenProvider.validateToken gives: the - // concatenation below re-reads the sequence, and the provider may be reusing its buffer - CharSequence token = pulled == null ? null : pulled.toString(); - HttpTokenProvider.validateToken(token); return "Bearer " + token; } return authorizationHeader; } + /** + * Steps 1-2 of the one-retry-after-401 rule (design/qwp-token-provider-spec.md, section 8.2): tells the + * token provider that the token in {@code presentedHeader} was rejected, then resolves the header again. + * Returns the new header, or null when no credential could be obtained - the connect then ends with the + * rejection, carrying the provider's failure as a suppressed diagnostic. + */ + private String refreshAfterRejection(String presentedHeader, QwpAuthFailedException rejection) { + if (presentedHeader != null && presentedHeader.startsWith("Bearer ")) { + try { + tokenProvider.onTokenRejected(presentedHeader.substring("Bearer ".length()), + rejection.getStatusCode()); + } catch (RuntimeException e) { + // The contract says it must not throw; a provider that does still gets its token re-pulled. + LOG.debug("token provider onTokenRejected threw {}", e.getClass().getName()); + } + } + try { + return resolveAuthorizationHeader(); + } catch (RuntimeException e) { + rejection.addSuppressed(e); + return null; + } + } + private long resolveQueryFlags(boolean resetSymbolDict) { if (!resetSymbolDict) { return 0L; diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpUpgradeFailures.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpUpgradeFailures.java index c0709f53d..388c0cad3 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpUpgradeFailures.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpUpgradeFailures.java @@ -48,7 +48,8 @@ static HttpClientException classify(WebSocketClient client, String host, int por } int status = client.getUpgradeStatusCode(); if (status == 401 || status == 403) { - QwpAuthFailedException ae = new QwpAuthFailedException(status, host, port); + QwpAuthFailedException ae = new QwpAuthFailedException( + status, host, port, client.getUpgradeRejectWwwAuthenticate()); ae.initCause(ex); return ae; } diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpWebSocketSender.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpWebSocketSender.java index 2f9c5a122..588a37e07 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpWebSocketSender.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpWebSocketSender.java @@ -25,6 +25,8 @@ package io.questdb.client.cutlass.qwp.client; import io.questdb.client.ClientTlsConfiguration; +import io.questdb.client.ConnectionHealth; +import io.questdb.client.HttpTokenProvider; import io.questdb.client.Sender; import io.questdb.client.SenderConnectionEvent; import io.questdb.client.SenderConnectionListener; @@ -65,6 +67,7 @@ import io.questdb.client.std.Numbers; import io.questdb.client.std.NumericException; import io.questdb.client.std.ObjList; +import io.questdb.client.std.QuietCloseable; import io.questdb.client.std.bytes.DirectByteSlice; import org.jetbrains.annotations.NotNull; import org.jetbrains.annotations.TestOnly; @@ -185,6 +188,10 @@ public class QwpWebSocketSender implements Sender { // work, and neither the foreground's reconnect nor close() can queue // behind a drainer's endpoint walk. private final ReentrantLock connectWalkLock = new ReentrantLock(); + // Connection health of the FOREGROUND connection (design/qwp-token-provider-spec.md, section 8.4): rounds and + // upgrades from buildAndConnect, connection loss and terminal failure from the I/O loop, close from close(). + // Background drainer walks never report here. + private final QwpConnectionHealthTracker healthTracker = new QwpConnectionHealthTracker(); private final QwpHostHealthTracker hostTracker; // Per-table encoded body byte counts captured during flushPendingRows' combined // encode. flushPendingRowsSplit uses them both for preflight sizing and to walk @@ -233,8 +240,13 @@ public class QwpWebSocketSender implements Sender { // Test-only lifecycle witness. close() invokes and clears it strictly after // publishing closed=true and before starting any drain or teardown work. private volatile Runnable closeStartedHook; + // Optional authentication-outage deadline (spec section 8.5) handed to the I/O loop; 0 = none. + private volatile long authFailureMaxDurationMillis; private boolean connected; private SenderConnectionDispatcher connectionDispatcher; + // A connect-string token_provider's registry lease (TokenProviderRegistry), released when this sender + // closes so the shared provider can linger and then stop. Null for any other credential. + private volatile QuietCloseable credentialLease; // Async-delivery sink for SenderConnectionEvent notifications. Default // installed at construction; the builder hook can swap before connect() // runs, and post-connect setConnectionListener() propagates to the live @@ -1005,6 +1017,22 @@ public static Supplier fixedAuthHeader(String header) { return header == null ? null : new FixedAuthHeader(header); } + /** + * Wraps a token provider as a DYNAMIC {@code Authorization} header supplier: each {@code get()} pulls the + * provider's current token, snapshots it, validates it ({@link HttpTokenProvider#validateToken}) and returns + * {@code "Bearer " + token}. Unlike a bare lambda, the wrapper also carries the provider's + * {@link HttpTokenProvider#onTokenRejected} back-channel, so a {@code 401} on an upgrade that presented this + * credential tells the provider - a caching provider then refreshes early - before the connect walk pulls + * again and retries the same endpoint once with the new token (design/qwp-token-provider-spec.md, section + * 8.2). + * + * @param provider the token provider, or null when no credential is configured + * @return a dynamic supplier, or null when {@code provider} is null + */ + public static Supplier tokenProviderAuthHeader(HttpTokenProvider provider) { + return provider == null ? null : new TokenProviderAuthHeader(provider); + } + @Override public void at(long timestamp, ChronoUnit unit) { checkNotClosed(); @@ -1302,6 +1330,7 @@ public void close() { } private void close0(boolean[] restoreInterrupt) { + healthTracker.closed(); Runnable hook = closeStartedHook; closeStartedHook = null; if (hook != null) { @@ -1453,6 +1482,9 @@ private void close0(boolean[] restoreInterrupt) { terminalError = captureCloseError(terminalError, e); } } + // Nothing pulls a credential any more (or, after a failed stop, the registry's linger outlasts the + // straggler), so the shared token provider may go. + releaseCredentialLease(); // Always free resources the I/O thread never touches: // encoder and table buffers are user-thread-only. @@ -1507,6 +1539,54 @@ public boolean isCloseCleanupComplete() { return closeCleanupComplete; } + /** + * {@inheritDoc} + *

+ * Tracks the foreground connection only; orphan-slot drainers report through + * {@link BackgroundDrainerListener} and the error handler. + */ + @Override + public ConnectionHealth health() { + return healthTracker.snapshot(); + } + + /** + * Arms the optional authentication-outage deadline (design/qwp-token-provider-spec.md, section 8.5); see + * {@code Sender.LineSenderBuilder.authFailureMaxDurationMillis(long)}. {@code <= 0} disarms it. May be called + * while the I/O loop runs. + */ + public void setAuthFailureMaxDurationMillis(long millis) { + this.authFailureMaxDurationMillis = millis; + CursorWebSocketSendLoop loop = cursorSendLoop; + if (loop != null) { + loop.setAuthFailureMaxDurationMillis(millis); + } + } + + /** + * Hands this sender the registry lease of the {@code token_provider} its credential comes from; the sender + * releases it when it closes. {@code Sender.LineSenderBuilder.build()} calls this right after connecting. A + * lease handed to a sender that is already closed is released at once. + */ + public void setCredentialLease(QuietCloseable lease) { + this.credentialLease = lease; + if (closed) { + releaseCredentialLease(); + } + } + + private void releaseCredentialLease() { + QuietCloseable lease = credentialLease; + credentialLease = null; + if (lease != null) { + try { + lease.close(); + } catch (Throwable e) { + LOG.error("Error releasing the token provider lease: {}", String.valueOf(e)); + } + } + } + /** * True once the store-and-forward slot flock has been released. False * means an I/O or manager worker did not stop and close() retained the @@ -3211,7 +3291,15 @@ private WebSocketClient buildAndConnect(ReconnectSupplier ctx, CursorWebSocketSe } connectWalkLock.lock(); try { - return connectWalk(ctx, cancellation); + WebSocketClient client = connectWalk(ctx, cancellation); + healthTracker.upgraded(); + return client; + } catch (RuntimeException e) { + // One failed connect round. A walk that a close() aborted is not a failure of the connection. + if (!ctx.isAborted()) { + healthTracker.roundFailed(e); + } + throw e; } finally { connectWalkLock.unlock(); } @@ -3232,6 +3320,47 @@ private static void clearInFlight(CursorWebSocketSendLoop.ConnectCancellation ca } } + /** + * Steps 1-2 of the one-retry-after-401 rule (design/qwp-token-provider-spec.md, section 8.2): tells a + * provider-backed credential that {@code presentedHeader} was rejected - a caching provider refreshes early, + * waiting briefly for the fetch - and pulls the credential again. Returns the new header, or null when no + * credential could be obtained, in which case the round's outcome stays the rejection. + *

+ * Both calls run caller-supplied provider code that may block, so on a cancellable walk this thread is + * published as being inside a credential pull, exactly like the pull before the walk: {@code close()} then + * breaks it with an interrupt. + */ + private String refreshCredentialAfterRejection( + String presentedHeader, + QwpAuthFailedException rejection, + ReconnectSupplier ctx, + CursorWebSocketSendLoop.ConnectCancellation cancellation + ) { + final Supplier supplier = authorizationHeaderSupplier; + if (cancellation != null) { + cancellation.publishCredentialPull(Thread.currentThread()); + if (cancellation.isCancelled()) { + cancellation.clearCredentialPull(); + throw new LineSenderException(ctx.abortMessage()); + } + } + try { + if (supplier instanceof TokenProviderAuthHeader) { + ((TokenProviderAuthHeader) supplier).onRejected(presentedHeader, rejection.getStatusCode()); + } + return supplier.get(); + } catch (RuntimeException e) { + // No fresh credential to retry with. Keep the provider's failure as a diagnostic of the rejection + // the round ends with. + rejection.addSuppressed(e); + return null; + } finally { + if (cancellation != null) { + cancellation.clearCredentialPull(); + } + } + } + private WebSocketClient connectWalk(ReconnectSupplier ctx, CursorWebSocketSendLoop.ConnectCancellation cancellation) { // Background (drainer) factories share this connect walk -- endpoint // list and hostTracker HEALTH state (never the shared round: a @@ -3330,7 +3459,7 @@ private WebSocketClient connectWalk(ReconnectSupplier ctx, CursorWebSocketSendLo throw new LineSenderException(ctx.abortMessage()); } } - final String authHeader; + String authHeader; try { authHeader = authorizationHeaderSupplier == null ? null : authorizationHeaderSupplier.get(); } catch (RuntimeException e) { @@ -3349,11 +3478,24 @@ private WebSocketClient connectWalk(ReconnectSupplier ctx, CursorWebSocketSendLo cancellation.clearCredentialPull(); } } + // One retry after a 401 per round (design/qwp-token-provider-spec.md, section 8.2): when an upgrade that + // presented a DYNAMIC credential is rejected with a refreshable 401, the provider is told, the credential + // is pulled again and, if it changed, the SAME endpoint is retried at once. The retried attempt is not an + // endpoint failure: it records no health penalty, fires no event, and does not consume the round's pick. + // A static credential never gets it - re-presenting the same bytes cannot change the answer. + boolean authRetryAvailable = hasDynamicCredential(); + int retryIdx = -1; while (true) { if (ctx.isAborted()) { throw new LineSenderException(ctx.abortMessage()); } - int idx = background ? cursor.next() : hostTracker.pickNext(); + int idx; + if (retryIdx >= 0) { + idx = retryIdx; + retryIdx = -1; + } else { + idx = background ? cursor.next() : hostTracker.pickNext(); + } if (idx < 0) break; Endpoint ep = endpoints.get(idx); lastEndpoint = ep; @@ -3425,12 +3567,26 @@ private WebSocketClient connectWalk(ReconnectSupplier ctx, CursorWebSocketSendLo continue; } if (classified instanceof QwpAuthFailedException) { + QwpAuthFailedException rejection = (QwpAuthFailedException) classified; + if (authRetryAvailable && rejection.isTokenRefreshable()) { + authRetryAvailable = false; + String refreshed = refreshCredentialAfterRejection( + authHeader, rejection, ctx, cancellation); + if (refreshed != null && !refreshed.equals(authHeader)) { + LOG.info("{}:{} rejected the token with {}; retrying the same endpoint once with a " + + "refreshed token", ep.host, ep.port, rejection.getStatusCode()); + authHeader = refreshed; + retryIdx = idx; + continue; + } + } // Auth is uniform across the cluster; we won't keep walking // endpoints. Fire AUTH_FAILED before throwing so the user - // listener observes the terminal classification at the - // moment the I/O thread gives up, ahead of the producer + // listener observes the classification of the round's final + // outcome at the moment the walk gives up, ahead of the producer // thread learning via LineSenderException on the next - // API call. + // API call. A 401 that earned the same-endpoint retry above + // fires nothing: only the round's final outcome is reported. if (!background) { dispatchConnectionEvent( SenderConnectionEvent.Kind.AUTH_FAILED, @@ -4090,6 +4246,8 @@ private void ensureConnected() { // the loop no longer fires a terminal budget-exhaustion event -- it // retries indefinitely.) cursorSendLoop.setConnectionDispatcher(connectionDispatcher); + cursorSendLoop.setConnectionHealthTracker(healthTracker); + cursorSendLoop.setAuthFailureMaxDurationMillis(authFailureMaxDurationMillis); cursorSendLoop.start(); } catch (Throwable t) { // start() (or dispatcher construction) failed after cursorSendLoop was @@ -5338,6 +5496,42 @@ public String get() { } } + /** + * A dynamic {@code Authorization} header backed by an {@link HttpTokenProvider}. See + * {@link #tokenProviderAuthHeader(HttpTokenProvider)}. + */ + private static final class TokenProviderAuthHeader implements Supplier { + private static final String BEARER_PREFIX = "Bearer "; + private final HttpTokenProvider provider; + + private TokenProviderAuthHeader(HttpTokenProvider provider) { + this.provider = provider; + } + + @Override + public String get() { + // Snapshot before validating: the concatenation below re-reads the sequence, and a provider is free + // to reuse a mutable buffer, so validating the live sequence checks bytes the header need not carry. + // See HttpTokenProvider.validateToken. + CharSequence pulled = provider.getToken(); + String token = pulled == null ? null : pulled.toString(); + HttpTokenProvider.validateToken(token); + return BEARER_PREFIX + token; + } + + void onRejected(String presentedHeader, int httpStatus) { + if (presentedHeader == null || !presentedHeader.startsWith(BEARER_PREFIX)) { + return; + } + try { + provider.onTokenRejected(presentedHeader.substring(BEARER_PREFIX.length()), httpStatus); + } catch (RuntimeException e) { + // The contract says it must not throw; a provider that does still gets its token re-pulled. + LOG.debug("token provider onTokenRejected threw {}", e.getClass().getName()); + } + } + } + private final class ReconnectSupplier implements CursorWebSocketSendLoop.ReconnectFactory { /** * Optional caller-owned liveness gate. {@code null} means this factory diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoop.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoop.java index 6643bf036..a64409f56 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoop.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoop.java @@ -403,6 +403,18 @@ public final class CursorWebSocketSendLoop implements QuietCloseable { // indefinitely and never gives up on a wall-clock budget). private volatile SenderConnectionDispatcher connectionDispatcher; private volatile SenderErrorDispatcher errorDispatcher; + // Optional authentication-outage deadline (design/qwp-token-provider-spec.md, section 8.5), FOREGROUND only; + // 0 = none, the default. Written by the owner thread, read by the I/O thread. + private volatile long authFailureMaxDurationNanos; + // The outage clock of that deadline: started by the first authentication-class failure (credential-unavailable + // or a 401/403 upgrade rejection) of the current outage, reset only by a successful upgrade. Failures of other + // classes neither reset nor fire it. Tracked whether or not a deadline is armed, so arming it late - the + // builder does so right after an async connect has started - still measures the whole outage. I/O thread only. + private boolean authOutageActive; + private long authOutageStartNanos; + // Foreground connection health (spec section 8.4): the loop reports connection loss and its terminal failure. + // Null for orphan drainers. + private volatile io.questdb.client.cutlass.qwp.client.QwpConnectionHealthTracker healthTracker; // The send cursor has two coordinate systems: // // FSN: durable frame sequence number in the local cursor engine. This is @@ -1051,18 +1063,32 @@ public static WebSocketClient connectWithRetry( contextLabel, e.getMessage()); throw e; } catch (QwpCredentialUnavailableException e) { - // A credential the client cannot ACQUIRE (the configured token provider threw) is NOT a - // transport outage: retrying the connect cannot conjure a token the provider will not hand - // over, so fail fast with the provider's own exception rather than burn the whole connect - // budget treating it as a reachable-server problem (which would block build() for up to - // maxDurationMillis, default 5 min, and surface a transport-shaped wrapper). Mirrors the - // foreground OFF-mode connect (QwpWebSocketSender) and the background reconnect loop above, - // which both give credential acquisition its own terminal handling; only this SYNC - // initial-connect path lacked it. QwpCredentialUnavailableException is a LineSenderException, - // disjoint from the HttpClientException-based terminal set above, so it reaches here. - LOG.error("{} could not acquire a credential, won't retry: {}", - contextLabel, e.getMessage()); - throw e.providerFailure(); + // A credential the client cannot ACQUIRE (the configured token provider threw) is classified by + // the provider (design/qwp-token-provider-spec.md, section 8.3 and decisions D6/D8): + // - a TokenUnavailableException marked retryable - the IdP is unreachable, timing out or + // throttling - is a transient outage like any other, so it consumes the connect budget and is + // retried within it (D6); + // - anything else is permanent - no sign-in yet, a missing or wrong configuration - and retrying + // cannot conjure a token the provider will not hand over, so fail fast with the provider's own + // exception rather than burn the whole budget (up to maxDurationMillis, default 5 min) and + // surface a transport-shaped wrapper (D8). This is how an OIDC device-flow provider that is + // not signed in behaves, and how every provider failure behaved before D6. + // An interrupt on the calling thread also ends the retries: a provider wait it cut short reports + // retryable, and re-entering it with the flag still set would only spin through the budget. + // QwpCredentialUnavailableException is a LineSenderException, disjoint from the + // HttpClientException-based terminal set above, so it reaches here. + if (!e.isRetryable() || Thread.currentThread().isInterrupted()) { + LOG.error("{} could not acquire a credential, won't retry: {}", + contextLabel, e.getMessage()); + throw e.providerFailure(); + } + lastError = e; + long now = System.nanoTime(); + if (now - lastLogNanos >= RECONNECT_LOG_THROTTLE_NANOS) { + LOG.warn("{} attempt {}: credential-unavailable (retryable): {}; retrying within connect budget", + contextLabel, attempts, e.getMessage()); + lastLogNanos = now; + } } catch (Throwable e) { if (e instanceof Error) { // JVM/programming failure (OOM, LinkageError): not a @@ -1105,7 +1131,11 @@ public static WebSocketClient connectWithRetry( backoffMillis = Math.min(backoffMillis * 2, maxBackoffMillis); } long elapsedMs = (System.nanoTime() - startNanos) / 1_000_000L; - String lastMsg = lastError == null ? "no attempts made" : lastError.getMessage(); + String lastMsg = lastError == null + ? "no attempts made" + : lastError instanceof QwpCredentialUnavailableException + ? "credential-unavailable: " + lastError.getMessage() + : lastError.getMessage(); throw new LineSenderException( contextLabel + " failed after " + elapsedMs + "ms / " + attempts + " attempts: " + lastMsg, @@ -1550,6 +1580,25 @@ public void setConnectionDispatcher(SenderConnectionDispatcher dispatcher) { this.connectionDispatcher = dispatcher; } + /** + * Arms the optional authentication-outage deadline (design/qwp-token-provider-spec.md, section 8.5). A + * FOREGROUND loop then latches a terminal when a connect round fails with an authentication-class failure - + * credential-unavailable, or a 401/403 upgrade rejection - and the current outage's first such failure lies + * at least this far back. {@code <= 0} disarms it. Ignored by orphan drainers, which have their own policy. + * May be called while the loop runs. + */ + public void setAuthFailureMaxDurationMillis(long millis) { + this.authFailureMaxDurationNanos = millis <= 0 ? 0 : TimeUnit.MILLISECONDS.toNanos(millis); + } + + /** + * Plugs the owning sender's connection-health tracker: the loop reports connection loss and its terminal + * failure there. Set before {@link #start()}; foreground loops only. + */ + public void setConnectionHealthTracker(io.questdb.client.cutlass.qwp.client.QwpConnectionHealthTracker tracker) { + this.healthTracker = tracker; + } + /** * Plug an async-delivery sink for {@link SenderError} notifications. * Idempotent — set once before {@link #start()}; later reassignment is @@ -1746,6 +1795,12 @@ private void connectLoop(Throwable initial, String phase, long paceFirstAttemptM snapshotReplayTarget(); LOG.warn("cursor I/O loop entering {} loop: {}", phase, initial.getMessage()); + if (hasEverConnected) { + io.questdb.client.cutlass.qwp.client.QwpConnectionHealthTracker tracker = healthTracker; + if (tracker != null) { + tracker.connectionLost(); + } + } long outageStartNanos = System.nanoTime(); // INVARIANT B: a store-and-forward loop must NEVER terminate on a // wall-clock reconnect budget. A replica-only / all-endpoints-replica @@ -1790,6 +1845,8 @@ private void connectLoop(Throwable initial, String phase, long paceFirstAttemptM try { WebSocketClient newClient = reconnectFactory.reconnect(connectCancellation); if (newClient != null) { + // a successful upgrade ends the authentication outage (spec section 8.5) + authOutageActive = false; if (!running) { // close() ran while this connect attempt was in // flight. Its latch await may have been interrupted @@ -1875,6 +1932,9 @@ private void connectLoop(Throwable initial, String phase, long paceFirstAttemptM } resetCatchUpCapGapEpisode(); lastReconnectError = e; + if (e instanceof QwpAuthFailedException && authOutageDeadlineFired(e)) { + return; + } dispatchRetriedEndpointPolicyFailure( SenderError.Category.SECURITY_ERROR, "ws-upgrade-failed: " + e.getMessage()); long now = System.nanoTime(); @@ -1942,6 +2002,9 @@ private void connectLoop(Throwable initial, String phase, long paceFirstAttemptM // cap-gap dwell (see MAX_CATCHUP_CAP_GAP_ATTEMPTS). resetCatchUpCapGapEpisode(); lastReconnectError = e; + if (authOutageDeadlineFired(e)) { + return; + } // Retrying must not be programmatically INVISIBLE, exactly as for the auth/upgrade and // durable-ack policy failures above: a revoked refresh token or a permanently unreachable IdP // is not self-healing, yet flush() keeps returning success while SF absorbs the rows. Without @@ -2048,6 +2111,53 @@ private void connectLoop(Throwable initial, String phase, long paceFirstAttemptM phase, elapsedMs, attempts, lastMsg); } + /** + * The optional authentication-outage deadline (design/qwp-token-provider-spec.md, section 8.5), evaluated + * when a connect round of a FOREGROUND loop fails with an authentication-class failure ({@code failure} is a + * credential-unavailable or a 401/403 rejection). Starts the outage clock on the first such failure; when a + * deadline is armed and the clock has reached it, latches a terminal that names the failure class and the + * elapsed time, reports it to the error handler, and returns true. Unacknowledged rows stay in + * store-and-forward. + */ + private boolean authOutageDeadlineFired(Throwable failure) { + if (reconnectPolicy != ReconnectPolicy.FOREGROUND) { + return false; + } + final long now = System.nanoTime(); + if (!authOutageActive) { + authOutageActive = true; + authOutageStartNanos = now; + } + final long deadline = authFailureMaxDurationNanos; + if (deadline <= 0 || now - authOutageStartNanos < deadline) { + return false; + } + final long elapsedMillis = TimeUnit.NANOSECONDS.toMillis(now - authOutageStartNanos); + final String failureClass = failure instanceof QwpCredentialUnavailableException + ? "credential-unavailable" : "auth-rejected"; + final String message = "authentication outage deadline exceeded: " + failureClass + " persisted for " + + elapsedMillis + "ms (auth_failure_max_duration_millis=" + + TimeUnit.NANOSECONDS.toMillis(deadline) + "); last failure: " + failure.getMessage(); + LOG.error("{} -- the sender stops; unacknowledged rows stay in store-and-forward", message); + long fromFsn = engine.ackedFsn() + 1L; + long toFsn = Math.max(fromFsn, engine.publishedFsn()); + SenderError err = new SenderError( + SenderError.Category.SECURITY_ERROR, + SenderError.Policy.TERMINAL, + SenderError.NO_STATUS_BYTE, + message, + SenderError.NO_MESSAGE_SEQUENCE, + fromFsn, + toFsn, + null, + System.nanoTime() + ); + totalServerErrors.incrementAndGet(); + recordFatal(new LineSenderServerException(err)); + dispatchError(err); + return true; + } + /** * Reports an endpoint-policy rejection a FOREGROUND sender is riding out. *

@@ -2549,6 +2659,10 @@ private void positionCursorInSegment(MmapSegment seg, long targetFsn) { * every rethrow delivers the same instance. */ private void recordFatal(Throwable t) { + io.questdb.client.cutlass.qwp.client.QwpConnectionHealthTracker tracker = healthTracker; + if (tracker != null) { + tracker.failed(); + } if (terminalError == null) { terminalError = t instanceof LineSenderException ? (LineSenderException) t diff --git a/core/src/main/java/io/questdb/client/impl/ConfigSchema.java b/core/src/main/java/io/questdb/client/impl/ConfigSchema.java index c9529a13e..38eebb6fc 100644 --- a/core/src/main/java/io/questdb/client/impl/ConfigSchema.java +++ b/core/src/main/java/io/questdb/client/impl/ConfigSchema.java @@ -57,10 +57,19 @@ public final class ConfigSchema { str("tls_roots_password", Side.COMMON); longRange("auth_timeout_ms", Side.COMMON, 0, OPEN_MAX, true, false); // > 0 longRange("connect_timeout", Side.COMMON, 0, OPEN_MAX, true, false); // > 0 + // Dynamic bearer credentials (design/qwp-token-provider-spec.md, section 7.1): a refreshing token + // provider selected by name, wss:: only, mutually exclusive with token/username/password. The values are + // never secret. Validated by TokenProviderSpec on both clients. + str("token_provider", Side.COMMON); + str("azure_resource", Side.COMMON); + str("azure_client_id", Side.COMMON); // INGRESS -- the WebSocket Sender applies. STRING in the registry; the // Sender parses suffix/mode values (off/on, 64k, durability) with its // own helpers, byte-for-byte. + // Optional authentication-outage deadline (design/qwp-token-provider-spec.md, section 8.5). Not set by + // default; > 0 when set. + longRange("auth_failure_max_duration_millis", Side.INGRESS, 0, OPEN_MAX, true, false); str("auto_flush", Side.INGRESS); str("auto_flush_bytes", Side.INGRESS); str("auto_flush_interval", Side.INGRESS); diff --git a/core/src/main/java/io/questdb/client/impl/PooledSender.java b/core/src/main/java/io/questdb/client/impl/PooledSender.java index 7b4e5f802..06f3940f9 100644 --- a/core/src/main/java/io/questdb/client/impl/PooledSender.java +++ b/core/src/main/java/io/questdb/client/impl/PooledSender.java @@ -24,6 +24,7 @@ package io.questdb.client.impl; +import io.questdb.client.ConnectionHealth; import io.questdb.client.Sender; import io.questdb.client.cutlass.line.array.DoubleArray; import io.questdb.client.cutlass.line.array.LongArray; @@ -276,6 +277,11 @@ public long getAckedFsn() { return slot.live(generation).getAckedFsn(); } + @Override + public ConnectionHealth health() { + return slot.live(generation).health(); + } + @Override public Sender intColumn(CharSequence name, int value) { slot.live(generation).intColumn(name, value); diff --git a/core/src/main/java/io/questdb/client/impl/QueryClientPool.java b/core/src/main/java/io/questdb/client/impl/QueryClientPool.java index 8e858c019..4f3c7b566 100644 --- a/core/src/main/java/io/questdb/client/impl/QueryClientPool.java +++ b/core/src/main/java/io/questdb/client/impl/QueryClientPool.java @@ -24,6 +24,7 @@ package io.questdb.client.impl; +import io.questdb.client.ConnectionHealth; import io.questdb.client.HttpTokenProvider; import io.questdb.client.QueryException; import io.questdb.client.cutlass.qwp.client.QwpQueryClient; @@ -485,6 +486,21 @@ void discard(QueryWorker w, long gen) { } } + /** + * Adds the connection health of every pooled query client to {@code into}. Reading a client's health never + * waits on its I/O, so this is safe under the pool lock. + */ + void collectHealth(java.util.List into) { + lock.lock(); + try { + for (int i = 0, n = all.size(); i < n; i++) { + into.add(all.get(i).client().health()); + } + } finally { + lock.unlock(); + } + } + void reapIdle() { if (closed) { return; diff --git a/core/src/main/java/io/questdb/client/impl/QuestDBImpl.java b/core/src/main/java/io/questdb/client/impl/QuestDBImpl.java index 574d6b59e..0f0c4c64f 100644 --- a/core/src/main/java/io/questdb/client/impl/QuestDBImpl.java +++ b/core/src/main/java/io/questdb/client/impl/QuestDBImpl.java @@ -24,6 +24,7 @@ package io.questdb.client.impl; +import io.questdb.client.ConnectionHealth; import io.questdb.client.HttpTokenProvider; import io.questdb.client.QuestDB; import io.questdb.client.Query; @@ -206,6 +207,14 @@ public Sender borrowSender() { return senderPool.borrow(); } + @Override + public ConnectionHealth.Aggregate health() { + java.util.List healths = new java.util.ArrayList<>(); + senderPool.collectHealth(healths); + queryPool.collectHealth(healths); + return ConnectionHealth.Aggregate.of(healths); + } + // synchronized so concurrent close() callers serialize THROUGH the bounded // shutdown sequence, not merely through the `closed` flip. `closed` is set // before the teardown chain runs, so a plain volatile guard (or a bare CAS) diff --git a/core/src/main/java/io/questdb/client/impl/SenderPool.java b/core/src/main/java/io/questdb/client/impl/SenderPool.java index 1a4390269..fd899158c 100644 --- a/core/src/main/java/io/questdb/client/impl/SenderPool.java +++ b/core/src/main/java/io/questdb/client/impl/SenderPool.java @@ -24,6 +24,7 @@ package io.questdb.client.impl; +import io.questdb.client.ConnectionHealth; import io.questdb.client.HttpTokenProvider; import io.questdb.client.Sender; import io.questdb.client.SenderConnectionListener; @@ -1923,6 +1924,25 @@ public int availableSize() { } } + /** + * Adds the connection health of every live pooled sender to {@code into}. Reading a sender's health never + * waits on its I/O, so this is safe under the pool lock. + */ + public void collectHealth(java.util.List into) { + lock.lock(); + try { + for (int i = 0, n = all.size(); i < n; i++) { + try { + into.add(all.get(i).delegate().health()); + } catch (UnsupportedOperationException ignored) { + // a test double without connection health + } + } + } finally { + lock.unlock(); + } + } + /** Snapshot of the total number of live slots (idle + in-use). For tests and introspection. */ public int totalSize() { lock.lock(); diff --git a/core/src/main/java/module-info.java b/core/src/main/java/module-info.java index 8383221e4..4f4bc9852 100644 --- a/core/src/main/java/module-info.java +++ b/core/src/main/java/module-info.java @@ -71,4 +71,8 @@ exports io.questdb.client.cutlass.qwp.client.sf.cursor; exports io.questdb.client.cutlass.qwp.protocol; exports io.questdb.client.cutlass.qwp.websocket; + + // token_provider= in a connect string resolves through this SPI; the optional questdb-client-azure + // artifact provides the "azure" factory (design/qwp-token-provider-spec.md, section 7). + uses io.questdb.client.cutlass.auth.TokenProviderFactory; } diff --git a/core/src/test/java/io/questdb/client/test/cutlass/auth/RefreshingTokenProviderTest.java b/core/src/test/java/io/questdb/client/test/cutlass/auth/RefreshingTokenProviderTest.java new file mode 100644 index 000000000..629377106 --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/cutlass/auth/RefreshingTokenProviderTest.java @@ -0,0 +1,841 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test.cutlass.auth; + +import io.questdb.client.cutlass.auth.CredentialRedaction; +import io.questdb.client.cutlass.auth.ExpiringToken; +import io.questdb.client.cutlass.auth.RefreshingTokenProvider; +import io.questdb.client.cutlass.auth.TokenUnavailableException; +import io.questdb.client.test.cutlass.auth.TokenTestKit.FakeClock; +import io.questdb.client.test.cutlass.auth.TokenTestKit.ManualScheduler; +import io.questdb.client.test.cutlass.auth.TokenTestKit.ScriptedSource; +import org.junit.Assert; +import org.junit.Test; + +import java.util.ArrayList; +import java.util.List; +import java.util.UUID; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.atomic.AtomicReference; + +import static io.questdb.client.test.cutlass.auth.TokenTestKit.await; + +/** + * Conformance tests C1-C8 and the cache half of C20 of the dynamic-credential specification + * (design/qwp-token-provider-spec.md, section 10) for {@link RefreshingTokenProvider}. + *

+ * Schedule, backoff, hand-out and forced-refresh tests run on a fake clock and a scheduler the test drives, so + * they assert exact delays without waiting. The concurrency, interrupt and lifecycle tests run on the real + * refresher thread. + */ +public class RefreshingTokenProviderTest { + private static final long MIN = 60_000L; + private static final long HOUR = 60 * MIN; + private static final long DAY = 24 * HOUR; + // A fixed wall-clock origin keeps the expected instants literal. + private static final long T0 = 1_800_000_000_000L; + + @Test + public void testAlreadyExpiredResultIsARetryableFailure() { + FakeClock clock = new FakeClock(T0); + ManualScheduler scheduler = new ManualScheduler(clock); + ScriptedSource source = new ScriptedSource() + .thenToken("EXPIRED", T0 - 1) + .thenToken("FRESH", T0 + HOUR); + try (RefreshingTokenProvider provider = provider(source, clock, scheduler, 0.0).build()) { + scheduler.runNext(); + Assert.assertEquals(1, provider.getConsecutiveFailures()); + RefreshingTokenProvider.FetchFailure failure = provider.getLastFailure(); + Assert.assertNotNull(failure); + Assert.assertTrue("a result that has already expired is a retryable failure", failure.isRetryable()); + Assert.assertTrue(failure.getMessage(), failure.getMessage().contains("expired")); + Assert.assertEquals(RefreshingTokenProvider.NONE, provider.getTokenExpiresAtEpochMillis()); + // retried after backoff, not dropped + scheduler.advanceAndRunNext(); + Assert.assertEquals("FRESH", provider.getToken().toString()); + Assert.assertEquals(0, provider.getConsecutiveFailures()); + } + } + + @Test(timeout = 30_000) + public void testAwaitReadyGatesOnTheFirstFetch() { + ScriptedSource source = new ScriptedSource().thenToken("READY", System.currentTimeMillis() + HOUR); + source.setGate(); + try (RefreshingTokenProvider provider = RefreshingTokenProvider.builder(source).build()) { + Assert.assertFalse("the first fetch has not finished", provider.awaitReady(50)); + source.openGate(); + Assert.assertTrue(provider.awaitReady(10_000)); + Assert.assertEquals("READY", provider.getToken().toString()); + } + } + + @Test + public void testBackoffFollowsTheSpec() { + // section 5.4: base = min(backoff_max, backoff_initial * 2^(n-1)); delay = base/2 + U(0, base/2). + // u = 0 pins the delay to the lower bound base/2, u = 0.5 to 3/4 of base. + assertBackoffDelays(0.0, new long[]{250, 500, 1_000, 2_000, 4_000, 8_000, 16_000, 30_000, 30_000}); + assertBackoffDelays(0.5, new long[]{375, 750, 1_500, 3_000, 6_000, 12_000, 24_000, 45_000, 45_000}); + } + + @Test + public void testCloseFailsGetTokenPermanentlyAndStopsTheRefresher() { + ScriptedSource source = new ScriptedSource().thenToken("TOKEN", System.currentTimeMillis() + HOUR); + List before = refresherThreads(); + RefreshingTokenProvider provider = RefreshingTokenProvider.builder(source).build(); + Assert.assertTrue(provider.awaitReady(10_000)); + List started = refresherThreads(); + started.removeAll(before); + Assert.assertEquals("one refresher thread per provider", 1, started.size()); + + provider.close(); + provider.close(); // idempotent + Assert.assertTrue(provider.isClosed()); + await(() -> !started.get(0).isAlive(), 5_000, "the refresher thread to exit"); + try { + provider.getToken(); + Assert.fail("getToken() after close() must fail"); + } catch (TokenUnavailableException e) { + Assert.assertFalse("a closed provider is a permanent failure", e.isRetryable()); + } + Assert.assertFalse(provider.awaitReady(10)); + provider.onTokenRejected("TOKEN", 401); // no-op, no throw + } + + @Test(timeout = 30_000) + public void testColdBurstAfterTheWallClockJumpsCausesOneFetch() throws Exception { + // C3, second shape: a token is held, but the wall clock jumped past its expiry (a host that slept + // through its scheduled refresh). 64 callers find no usable token at once; exactly one fetch runs. + SkewedClock clock = new SkewedClock(); + ScriptedSource source = new ScriptedSource() + .thenToken("OLD", System.currentTimeMillis() + HOUR) + .then(() -> new ExpiringToken("NEW", clock.wallClockMillis() + HOUR)); + try (RefreshingTokenProvider provider = RefreshingTokenProvider.builder(source).clock(clock).build()) { + Assert.assertTrue(provider.awaitReady(10_000)); + Assert.assertEquals("OLD", provider.getToken().toString()); + source.setGate(); + clock.skewMillis = 2 * HOUR; + Assert.assertEquals(listOf("NEW"), burst(provider, 64, source)); + Assert.assertEquals("the prefetch plus exactly one fetch for the whole burst", 2, source.calls()); + } + } + + @Test(timeout = 30_000) + public void testColdBurstCausesExactlyOneFetch() throws Exception { + // C3: 64 concurrent callers with no usable token cause exactly one fetch. + ScriptedSource source = new ScriptedSource().thenToken("TOKEN-1", System.currentTimeMillis() + HOUR); + source.setGate(); // the prefetch blocks until every caller is waiting + try (RefreshingTokenProvider provider = RefreshingTokenProvider.builder(source).build()) { + Assert.assertEquals(listOf("TOKEN-1"), burst(provider, 64, source)); + Assert.assertEquals(1, source.calls()); + } + } + + @Test(timeout = 30_000) + public void testColdFailureFailsImmediatelyWhenBackoffOutlastsTheWait() { + // section 5.3: "If the earliest permitted fetch is later than now + cold_wait, fail immediately." + ScriptedSource source = new ScriptedSource() + .thenThrow(TokenUnavailableException.retryable("HTTP 429 from the token endpoint", 60_000)); + try (RefreshingTokenProvider provider = RefreshingTokenProvider.builder(source) + .coldWaitMillis(5_000) + .build()) { + await(() -> provider.getConsecutiveFailures() == 1, 10_000, "the prefetch to fail"); + long start = System.nanoTime(); + try { + provider.getToken(); + Assert.fail("expected the cold wait to fail"); + } catch (TokenUnavailableException e) { + long elapsedMillis = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - start); + Assert.assertTrue("must fail at once, not wait out cold_wait; took " + elapsedMillis + " ms", + elapsedMillis < 2_000); + Assert.assertTrue(e.isRetryable()); + Assert.assertTrue("the remaining backoff is the hint: " + e.getRetryAfterMillis(), + e.getRetryAfterMillis() > 30_000); + Assert.assertTrue(e.getMessage(), e.getMessage().contains("HTTP 429")); + } + Assert.assertEquals("the caller must not bypass the backoff with a fetch of its own", 1, source.calls()); + } + } + + @Test(timeout = 30_000) + public void testColdFailurePassesAPermanentClassificationThrough() { + // C4: a permanent classification is passed through to the caller. + ScriptedSource source = new ScriptedSource() + .thenThrow(TokenUnavailableException.permanent("no credential is configured")); + try (RefreshingTokenProvider provider = RefreshingTokenProvider.builder(source) + .coldWaitMillis(300) + .backoffInitialMillis(20) + .backoffMaxMillis(40) + .build()) { + try { + provider.getToken(); + Assert.fail("expected a token-unavailable error"); + } catch (TokenUnavailableException e) { + Assert.assertFalse("the source's permanent classification must reach the caller", e.isRetryable()); + Assert.assertTrue(e.getMessage(), e.getMessage().contains("no credential is configured")); + } + // the provider keeps retrying a "permanent" failure: an operator can repair it without a restart + int calls = source.calls(); + await(() -> source.calls() > calls, 5_000, "another retry of the permanent failure"); + } + } + + @Test(timeout = 30_000) + public void testColdFailureRaisesARetryableErrorWithinColdWait() { + // C4: a retryable error is raised within cold_wait. + ScriptedSource source = new ScriptedSource() + .thenThrow(TokenUnavailableException.retryable("IMDS answered HTTP 503")); + long coldWaitMillis = 400; + try (RefreshingTokenProvider provider = RefreshingTokenProvider.builder(source) + .coldWaitMillis(coldWaitMillis) + .backoffInitialMillis(20) + .backoffMaxMillis(40) + .build()) { + long start = System.nanoTime(); + try { + provider.getToken(); + Assert.fail("expected a token-unavailable error"); + } catch (TokenUnavailableException e) { + long elapsedMillis = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - start); + Assert.assertTrue(e.isRetryable()); + Assert.assertTrue(e.getMessage(), e.getMessage().contains("HTTP 503")); + Assert.assertTrue("the wait must be bounded by cold_wait; took " + elapsedMillis + " ms", + elapsedMillis < coldWaitMillis + 2_000); + } + Assert.assertTrue("short failures inside cold_wait are retried, not surfaced on the first one", + source.calls() > 1); + } + } + + @Test + public void testCredentialRedactionHelpers() { + String token = "abc.def.ghi"; + String fingerprint = CredentialRedaction.fingerprint(token); + Assert.assertTrue(fingerprint, fingerprint.matches("[0-9a-f]{8}")); + String described = CredentialRedaction.describeToken(token); + Assert.assertEquals("', described); + Assert.assertEquals("", CredentialRedaction.describeToken(null)); + + // controls, CR/LF and bidi overrides are removed; the result is capped at 256 characters + String hostile = "line1\r\nFORGED: yes\u202Eevil\u0007" + repeat('x', 1_000); + String clean = CredentialRedaction.sanitizeErrorText(hostile); + Assert.assertTrue(clean.length() <= CredentialRedaction.MAX_ERROR_TEXT_LENGTH); + Assert.assertTrue(clean.endsWith("...")); + Assert.assertFalse(clean.contains("\r") || clean.contains("\n") || clean.contains("\u202E") + || clean.contains("\u0007")); + Assert.assertTrue(clean.startsWith("line1FORGED: yesevil")); + // text that fits is kept whole + String exact = repeat('y', CredentialRedaction.MAX_ERROR_TEXT_LENGTH); + Assert.assertEquals(exact, CredentialRedaction.sanitizeErrorText(exact)); + Assert.assertNull(CredentialRedaction.sanitizeErrorText(null)); + } + + @Test + public void testExpiringTokenRejectsInvalidTokensWithoutEchoingThem() { + String secret = "SECRET-" + UUID.randomUUID(); + assertRejected(null); + assertRejected(""); + assertRejected(" "); + assertRejected(secret + "\r\nX-Injected: 1"); + assertRejected(secret + "\u00e9"); + try { + new ExpiringToken(secret + '\n', T0); + Assert.fail(); + } catch (IllegalArgumentException e) { + Assert.assertFalse("the token must never be echoed", e.getMessage().contains(secret)); + } + ExpiringToken ok = new ExpiringToken(secret, T0, -5); + Assert.assertFalse("a non-positive refresh hint means none", ok.hasRefreshAt()); + Assert.assertEquals(ExpiringToken.NO_REFRESH_AT, ok.getRefreshAtEpochMillis()); + Assert.assertTrue(new ExpiringToken(secret, T0 + HOUR, T0 + MIN).hasRefreshAt()); + } + + @Test(timeout = 30_000) + public void testForcedRefreshIsRateLimitedIgnoresStaleTokensAndAcceptsTheSameToken() throws Exception { + // C8: rate limited; a stale token is ignored; a same-token result is accepted. + FakeClock clock = new FakeClock(T0); + ManualScheduler scheduler = new ManualScheduler(clock); + ScriptedSource source = new ScriptedSource() + .thenToken("T1", T0 + HOUR) + .thenToken("T2", T0 + HOUR) + .thenToken("T2", T0 + HOUR); + try (RefreshingTokenProvider provider = provider(source, clock, scheduler, 0.5) + .forcedMinIntervalMillis(30_000) + .forcedWaitMillis(5_000) + .build()) { + scheduler.runNext(); + Assert.assertEquals("T1", provider.getToken().toString()); + + // 403 never triggers a forced refresh (decision D2) + assertReturnsPromptly(() -> provider.onTokenRejected("T1", 403)); + Assert.assertNotEquals("a 403 must not schedule a fetch", 0, scheduler.peek().delayNanos); + + // 401 for the current token: a fetch starts now and the caller waits for it + Thread rejecter = start(() -> provider.onTokenRejected("T1", 401)); + await(() -> scheduler.peek() != null && scheduler.peek().delayNanos == 0, 5_000, + "the forced fetch to be scheduled"); + scheduler.runNext(); + rejecter.join(5_000); + Assert.assertFalse("onTokenRejected must return once the forced fetch completes", rejecter.isAlive()); + Assert.assertEquals("T2", provider.getToken().toString()); + Assert.assertEquals(2, source.calls()); + + // a stale token - it has already rotated - is ignored + assertReturnsPromptly(() -> provider.onTokenRejected("T1", 401)); + // rate limited: a second forced refresh within forced_min_interval is ignored + clock.advanceMillis(30_000 - 1); + assertReturnsPromptly(() -> provider.onTokenRejected("T2", 401)); + Assert.assertNotEquals("no forced fetch may be scheduled", 0, scheduler.peek().delayNanos); + Assert.assertEquals(2, source.calls()); + + // once the interval has passed, a forced fetch that returns the same token is a normal success + clock.advanceMillis(1); + rejecter = start(() -> provider.onTokenRejected("T2", 401)); + await(() -> scheduler.peek() != null && scheduler.peek().delayNanos == 0, 5_000, + "the second forced fetch to be scheduled"); + scheduler.runNext(); + rejecter.join(5_000); + Assert.assertFalse(rejecter.isAlive()); + Assert.assertEquals(3, source.calls()); + Assert.assertEquals("T2", provider.getToken().toString()); + Assert.assertEquals(0, provider.getConsecutiveFailures()); + Assert.assertNull("a same-token result is not a failure", provider.getLastFailure()); + Assert.assertEquals(clock.wallClockMillis(), provider.getLastSuccessEpochMillis()); + } + } + + @Test(timeout = 30_000) + public void testHandoutFloorTokenIsNeverHandedOut() throws Exception { + // C6: a token inside handout_floor is never handed out. + FakeClock clock = new FakeClock(T0); + ManualScheduler scheduler = new ManualScheduler(clock); + ScriptedSource source = new ScriptedSource() + .thenToken("T1", T0 + 10 * MIN) + .thenToken("T2", T0 + 70 * MIN); + try (RefreshingTokenProvider provider = provider(source, clock, scheduler, 0.5).build()) { + scheduler.runNext(); + clock.jumpWallMillis(10 * MIN - RefreshingTokenProvider.DEFAULT_HANDOUT_FLOOR_MILLIS - 1); + Assert.assertEquals("one millisecond outside the floor the token is still usable", + "T1", provider.getToken().toString()); + clock.jumpWallMillis(1); + + // exactly at expires_at - handout_floor the token is unusable: the caller must get a new one + AtomicReference got = new AtomicReference<>(); + Thread caller = start(() -> got.set(provider.getToken().toString())); + await(() -> scheduler.peek() != null && scheduler.peek().delayNanos == 0, 5_000, + "the cold caller to start a fetch"); + Assert.assertNull("nothing may be handed out before the fetch", got.get()); + scheduler.runNext(); + caller.join(5_000); + Assert.assertEquals("T2", got.get()); + } + } + + @Test(timeout = 30_000) + public void testHandoutFloorTokenIsNotHandedOutEvenWhenTheSourceRepeatsIt() throws Exception { + // C6, second shape: the source keeps returning the token that sits inside the floor. The caller waits + // out cold_wait and fails rather than receive it. + FakeClock clock = new FakeClock(T0); + ManualScheduler scheduler = new ManualScheduler(clock); + ScriptedSource source = new ScriptedSource().thenToken("T1", T0 + 10 * MIN); + try (RefreshingTokenProvider provider = provider(source, clock, scheduler, 0.5) + .coldWaitMillis(1_000) + .build()) { + scheduler.runNext(); + clock.jumpWallMillis(10 * MIN - 30_000); + AtomicReference got = new AtomicReference<>(); + Thread caller = start(() -> { + try { + got.set(provider.getToken().toString()); + } catch (TokenUnavailableException e) { + got.set(e); + } + }); + await(() -> scheduler.peek() != null && scheduler.peek().delayNanos == 0, 5_000, + "the cold caller to start a fetch"); + scheduler.runNext(); // the source returns T1 again, still inside the floor + Assert.assertNull("an unusable token must not be handed out", got.get()); + clock.advanceMillis(1_001); // cold_wait elapses + caller.join(5_000); + Assert.assertTrue("expected a token-unavailable error, got " + got.get(), + got.get() instanceof TokenUnavailableException); + Assert.assertTrue("no fetch failed, so the error is retryable", + ((TokenUnavailableException) got.get()).isRetryable()); + Assert.assertEquals("the caller started one fetch, not one per completed fetch", 2, source.calls()); + } + } + + @Test(timeout = 30_000) + public void testInterruptedWaitEndsPromptlyAndKeepsTheFlag() throws Exception { + // C7: a cancelled wait returns promptly, and the cancellation signal is preserved. + ScriptedSource source = new ScriptedSource().thenToken("NEVER", System.currentTimeMillis() + HOUR); + source.setGate(); + try (RefreshingTokenProvider provider = RefreshingTokenProvider.builder(source).build()) { + AtomicReference error = new AtomicReference<>(); + AtomicBoolean flagKept = new AtomicBoolean(); + Thread caller = start(() -> { + try { + provider.getToken(); + } catch (TokenUnavailableException e) { + error.set(e); + flagKept.set(Thread.currentThread().isInterrupted()); + } + }); + await(() -> caller.getState() == Thread.State.TIMED_WAITING, 5_000, "the caller to wait"); + long start = System.nanoTime(); + caller.interrupt(); + caller.join(5_000); + long elapsedMillis = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - start); + Assert.assertFalse(caller.isAlive()); + Assert.assertNotNull("an interrupted wait must end with a token-unavailable error", error.get()); + Assert.assertTrue("a cancelled wait is retryable", error.get().isRetryable()); + Assert.assertTrue("the interrupt flag must be preserved", flagKept.get()); + Assert.assertTrue("must return promptly; took " + elapsedMillis + " ms", elapsedMillis < 2_000); + + // a caller that arrives already interrupted fails at once, flag intact + Thread.currentThread().interrupt(); + try { + provider.getToken(); + Assert.fail(); + } catch (TokenUnavailableException e) { + Assert.assertTrue(Thread.interrupted()); + } + source.openGate(); + } + } + + @Test(timeout = 30_000) + public void testProactiveRefreshRunsWithoutACaller() { + // section 5.2: the provider starts a fetch at the refresh time without waiting for a caller. + long now = System.currentTimeMillis(); + ScriptedSource source = new ScriptedSource() + .thenToken("SHORT", now + 1_000) + .then(() -> new ExpiringToken("NEXT", System.currentTimeMillis() + HOUR)); + try (RefreshingTokenProvider provider = RefreshingTokenProvider.builder(source) + .handoutFloorMillis(0) + .refreshMarginMillis(0) + .minRefreshIntervalMillis(0) + .build()) { + Assert.assertTrue(provider.awaitReady(10_000)); + await(() -> source.calls() == 2, 10_000, "the proactive refresh"); + await(() -> provider.getTokenExpiresAtEpochMillis() > now + 10_000, 10_000, "the new token"); + Assert.assertEquals("NEXT", provider.getToken().toString()); + } + } + + @Test + public void testRefreshScheduleFollowsTheSpec() { + // C1: refresh times follow section 5.2 for L = 24 h, 60 min and 5 min, with and without refresh_at. + // u = 0 is the earliest jitter (0.45 L), u = 0.5 the middle (0.5 L), u = 0.999 near the latest (0.55 L). + // u = 0 is the earliest jitter (0.45 L), u = 0.5 the middle (0.5 L), u = 1 the latest (0.55 L). + assertRefreshDelay(DAY, 0.0, 0, 38_880_000L); // 10.8 h + assertRefreshDelay(DAY, 0.5, 0, 43_200_000L); // 12 h + assertRefreshDelay(DAY, 1.0, 0, 47_520_000L); // 13.2 h + assertRefreshDelay(HOUR, 0.0, 0, 1_620_000L); // 27 min + assertRefreshDelay(HOUR, 0.5, 0, 1_800_000L); // 30 min + assertRefreshDelay(HOUR, 1.0, 0, 1_980_000L); // 33 min + // L = 5 min: the margin is min(5 min, L/2) = 2.5 min, so the latest refresh is at 2.5 min + assertRefreshDelay(5 * MIN, 0.0, 0, 135_000L); + assertRefreshDelay(5 * MIN, 0.5, 0, 150_000L); + assertRefreshDelay(5 * MIN, 1.0, 0, 150_000L); + // a refresh hint earlier than the jittered half-life wins ... + assertRefreshDelay(HOUR, 0.5, T0 + 10 * MIN, 10 * MIN); + assertRefreshDelay(DAY, 0.5, T0 + 6 * HOUR, 6 * HOUR); + // ... but never earlier than min_refresh_interval + assertRefreshDelay(HOUR, 0.5, T0 + 5_000, 30_000L); + assertRefreshDelay(HOUR, 0.5, T0 - HOUR, 30_000L); + // a later hint does not delay the refresh + assertRefreshDelay(HOUR, 0.5, T0 + 50 * MIN, 30 * MIN); + assertRefreshDelay(5 * MIN, 0.5, T0 + 4 * MIN, 150_000L); + // a 50 s token: min_refresh_interval shrinks to L/2 = 25 s, the margin to 25 s + assertRefreshDelay(50_000L, 0.0, 0, 25_000L); + // the pure function agrees with the provider + Assert.assertEquals(T0 + 1_800_000L, + RefreshingTokenProvider.computeRefreshAtMillis(T0, T0 + HOUR, 0, 0.5, 5 * MIN, 30_000)); + } + + @Test + public void testRetryAfterIsHonouredAndCapped() { + // C5: retry_after is honoured and capped at retry_after_max. + assertFirstBackoff(10_000L, 10_000L); // longer than the backoff: honoured + assertFirstBackoff(10 * MIN, 5 * MIN); // capped at retry_after_max (5 min) + assertFirstBackoff(100L, 250L); // shorter than the backoff: the backoff wins + assertFirstBackoff(TokenUnavailableException.NO_RETRY_AFTER, 250L); + } + + @Test + public void testSameTokenConvergesAndStaysUsable() { + // C2: the platform keeps returning the same 24 h token. Each refresh lands at about half the remaining + // lifetime, so the token is fetched only a handful of times before it enters the hand-out floor, and + // it stays usable all the while. + FakeClock clock = new FakeClock(T0); + ManualScheduler scheduler = new ManualScheduler(clock); + final long expiry = T0 + DAY; + ScriptedSource source = new ScriptedSource().thenToken("SAME", expiry); + try (RefreshingTokenProvider provider = provider(source, clock, scheduler, 0.5).build()) { + scheduler.runNext(); + int fetchesWhileUsable = 1; + while (true) { + long remaining = expiry - clock.wallClockMillis(); + ManualScheduler.Task next = scheduler.peek(); + long expectedDelay = RefreshingTokenProvider.computeRefreshAtMillis( + 0, remaining, 0, 0.5, 5 * MIN, 30_000); + Assert.assertEquals("refresh at about half the remaining lifetime [remaining=" + remaining + ']', + expectedDelay, next.delayMillis()); + Assert.assertTrue(next.delayMillis() <= remaining / 2 + 1); + scheduler.advanceAndRunNext(); + if (clock.wallClockMillis() >= expiry - RefreshingTokenProvider.DEFAULT_HANDOUT_FLOOR_MILLIS) { + break; + } + Assert.assertEquals("SAME", provider.getToken().toString()); + fetchesWhileUsable++; + } + // 24 h halves to below the 60 s floor in 11 steps (86400 s -> ... -> 84.4 s) + Assert.assertEquals(11, fetchesWhileUsable); + Assert.assertEquals(0, provider.getConsecutiveFailures()); + } + } + + @Test + public void testSentinelTokenIsNeverRendered() throws Exception { + // C20 for the cache: a sentinel token never appears in a string representation, an error, a failure + // record or a log line - including when the source returns it in a malformed result. + final String sentinel = "SENTINEL-" + UUID.randomUUID(); + ch.qos.logback.classic.Logger logger = (ch.qos.logback.classic.Logger) + org.slf4j.LoggerFactory.getLogger(RefreshingTokenProvider.class); + ch.qos.logback.core.read.ListAppender appender = + new ch.qos.logback.core.read.ListAppender<>(); + appender.start(); + ch.qos.logback.classic.Level savedLevel = logger.getLevel(); + logger.setLevel(ch.qos.logback.classic.Level.ALL); + logger.addAppender(appender); + List rendered = new ArrayList<>(); + try { + ExpiringToken token = new ExpiringToken(sentinel, T0 + HOUR, T0 + MIN); + rendered.add(token.toString()); + try { + new ExpiringToken(sentinel + "\r\n", T0 + HOUR); + } catch (IllegalArgumentException e) { + rendered.add(e.getMessage()); + } + + FakeClock clock = new FakeClock(T0); + ManualScheduler scheduler = new ManualScheduler(clock); + ScriptedSource source = new ScriptedSource() + .thenToken(sentinel, T0 + 10 * MIN) + .thenToken(sentinel, T0 - 1) // malformed: already expired + .thenThrow(new IllegalStateException("token endpoint returned a response without access_token")); + RefreshingTokenProvider provider = provider(source, clock, scheduler, 0.5).coldWaitMillis(0).build(); + try { + scheduler.runNext(); + rendered.add(provider.toString()); + scheduler.advanceAndRunNext(); + rendered.add(String.valueOf(provider.getLastFailure())); + scheduler.advanceAndRunNext(); + rendered.add(String.valueOf(provider.getLastFailure())); + rendered.add(provider.toString()); + clock.jumpWallMillis(HOUR); + try { + provider.getToken(); + Assert.fail("no usable token is held"); + } catch (TokenUnavailableException e) { + rendered.add(e.getMessage()); + rendered.add(e.toString()); + } + } finally { + provider.close(); + } + rendered.add(provider.toString()); + } finally { + logger.detachAppender(appender); + logger.setLevel(savedLevel); + appender.stop(); + } + for (ch.qos.logback.classic.spi.ILoggingEvent event : appender.list) { + rendered.add(event.getFormattedMessage()); + } + Assert.assertFalse("expected log lines to inspect", appender.list.isEmpty()); + for (String s : rendered) { + Assert.assertFalse("the token leaked into: " + s, s.contains(sentinel)); + } + Assert.assertTrue(rendered.get(0), rendered.get(0).contains("sha256:")); + } + + @Test + public void testSourceFailureTextIsSanitized() { + FakeClock clock = new FakeClock(T0); + ManualScheduler scheduler = new ManualScheduler(clock); + ScriptedSource source = new ScriptedSource().thenThrow(TokenUnavailableException.retryable( + "AADSTS50000:\r\nX-Forged: 1\u202E" + repeat('z', 500))); + try (RefreshingTokenProvider provider = provider(source, clock, scheduler, 0.0).build()) { + scheduler.runNext(); + String message = provider.getLastFailure().getMessage(); + Assert.assertTrue(message.length() <= CredentialRedaction.MAX_ERROR_TEXT_LENGTH); + Assert.assertFalse(message.contains("\r") || message.contains("\n") || message.contains("\u202E")); + Assert.assertTrue(message, message.startsWith("AADSTS50000:X-Forged: 1zzz")); + } + } + + @Test + public void testSuccessResetsTheBackoff() { + FakeClock clock = new FakeClock(T0); + ManualScheduler scheduler = new ManualScheduler(clock); + ScriptedSource source = new ScriptedSource() + .thenThrow(TokenUnavailableException.retryable("down")) + .thenThrow(TokenUnavailableException.retryable("down")) + .thenThrow(TokenUnavailableException.retryable("down")) + .thenToken("UP", T0 + HOUR) + .thenThrow(TokenUnavailableException.retryable("down again")); + try (RefreshingTokenProvider provider = provider(source, clock, scheduler, 0.0).build()) { + scheduler.runNext(); + Assert.assertEquals(250, scheduler.peek().delayMillis()); + scheduler.advanceAndRunNext(); + Assert.assertEquals(500, scheduler.peek().delayMillis()); + scheduler.advanceAndRunNext(); + Assert.assertEquals(1_000, scheduler.peek().delayMillis()); + scheduler.advanceAndRunNext(); + Assert.assertEquals("UP", provider.getToken().toString()); + Assert.assertEquals(0, provider.getConsecutiveFailures()); + Assert.assertEquals("down", provider.getLastFailure().getMessage()); + scheduler.advanceAndRunNext(); // the proactive refresh fails + Assert.assertEquals(1, provider.getConsecutiveFailures()); + Assert.assertEquals("a success resets n, so the backoff starts over", 250, scheduler.peek().delayMillis()); + Assert.assertEquals("the current token keeps being served while usable", + "UP", provider.getToken().toString()); + } + } + + @Test(timeout = 30_000) + public void testUnclassifiedSourceExceptionIsRetryable() { + ScriptedSource source = new ScriptedSource().thenThrow(new IllegalStateException("socket reset")); + try (RefreshingTokenProvider provider = RefreshingTokenProvider.builder(source) + .coldWaitMillis(200) + .backoffInitialMillis(20) + .backoffMaxMillis(40) + .build()) { + try { + provider.getToken(); + Assert.fail(); + } catch (TokenUnavailableException e) { + Assert.assertTrue("an error that carries no classification is retryable", e.isRetryable()); + Assert.assertTrue(e.getMessage(), e.getMessage().contains("IllegalStateException: socket reset")); + } + } + } + + @Test + public void testWarmPathDoesNotFetch() { + FakeClock clock = new FakeClock(T0); + ManualScheduler scheduler = new ManualScheduler(clock); + ScriptedSource source = new ScriptedSource().thenToken("WARM", T0 + HOUR); + try (RefreshingTokenProvider provider = provider(source, clock, scheduler, 0.5).build()) { + scheduler.runNext(); + ManualScheduler.Task scheduled = scheduler.peek(); + for (int i = 0; i < 1_000; i++) { + Assert.assertEquals("WARM", provider.getToken().toString()); + } + Assert.assertEquals(1, source.calls()); + Assert.assertSame("a warm read must not touch the schedule", scheduled, scheduler.peek()); + } + } + + @Test + public void testWallClockOverdueRefreshStartsInTheBackground() { + // section 5.3 step 1: a usable token is returned at once, and when its refresh time has passed - here the + // wall clock moved past it while the monotonic schedule did not - a fetch starts in the background. + FakeClock clock = new FakeClock(T0); + ManualScheduler scheduler = new ManualScheduler(clock); + ScriptedSource source = new ScriptedSource().thenToken("T1", T0 + HOUR).thenToken("T2", T0 + 2 * HOUR); + try (RefreshingTokenProvider provider = provider(source, clock, scheduler, 0.5).build()) { + scheduler.runNext(); + Assert.assertEquals(30 * MIN, scheduler.peek().delayMillis()); + clock.jumpWallMillis(31 * MIN); + Assert.assertEquals("the usable token is returned at once", "T1", provider.getToken().toString()); + Assert.assertEquals("the overdue refresh is pulled forward", 0, scheduler.peek().delayNanos); + scheduler.runNext(); + Assert.assertEquals("T2", provider.getToken().toString()); + } + } + + private static void assertBackoffDelays(double u, long[] expectedMillis) { + FakeClock clock = new FakeClock(T0); + ManualScheduler scheduler = new ManualScheduler(clock); + ScriptedSource source = new ScriptedSource().thenThrow(TokenUnavailableException.retryable("down")); + try (RefreshingTokenProvider provider = provider(source, clock, scheduler, u).build()) { + scheduler.runNext(); + for (int n = 1; n <= expectedMillis.length; n++) { + Assert.assertEquals("backoff after failure " + n + " [u=" + u + ']', + expectedMillis[n - 1], scheduler.peek().delayMillis()); + Assert.assertEquals(n, provider.getConsecutiveFailures()); + scheduler.advanceAndRunNext(); + } + } + } + + private static void assertFirstBackoff(long retryAfterMillis, long expectedDelayMillis) { + FakeClock clock = new FakeClock(T0); + ManualScheduler scheduler = new ManualScheduler(clock); + ScriptedSource source = new ScriptedSource().thenThrow( + new TokenUnavailableException("throttled", true, retryAfterMillis)); + try (RefreshingTokenProvider provider = provider(source, clock, scheduler, 0.0).build()) { + scheduler.runNext(); + Assert.assertEquals("retry_after=" + retryAfterMillis, expectedDelayMillis, scheduler.peek().delayMillis()); + Assert.assertEquals(retryAfterMillis, provider.getLastFailure().getRetryAfterMillis()); + } + } + + private static void assertRefreshDelay(long lifetimeMillis, double u, long refreshAt, long expectedDelayMillis) { + FakeClock clock = new FakeClock(T0); + ManualScheduler scheduler = new ManualScheduler(clock); + ScriptedSource source = new ScriptedSource() + .then(() -> new ExpiringToken("TOKEN", T0 + lifetimeMillis, refreshAt)); + try (RefreshingTokenProvider provider = provider(source, clock, scheduler, u) + .handoutFloorMillis(0) + .build()) { + Assert.assertEquals("the prefetch runs at once", 0, scheduler.peek().delayNanos); + scheduler.runNext(); + Assert.assertEquals("refresh delay [L=" + lifetimeMillis + ", u=" + u + ", refreshAt=" + refreshAt + ']', + expectedDelayMillis, scheduler.peek().delayMillis()); + Assert.assertEquals(T0 + lifetimeMillis, provider.getTokenExpiresAtEpochMillis()); + } + } + + private static void assertRejected(String token) { + try { + new ExpiringToken(token, T0); + Assert.fail("expected the token to be rejected"); + } catch (IllegalArgumentException expected) { + // ok + } + } + + private static void assertReturnsPromptly(Runnable r) throws InterruptedException { + Thread t = start(r); + t.join(2_000); + Assert.assertFalse("the call must return without waiting", t.isAlive()); + } + + private static List burst(RefreshingTokenProvider provider, int callers, ScriptedSource source) + throws InterruptedException { + CountDownLatch ready = new CountDownLatch(callers); + List threads = new ArrayList<>(); + String[] results = new String[callers]; + for (int i = 0; i < callers; i++) { + final int idx = i; + Thread t = new Thread(() -> { + ready.countDown(); + results[idx] = provider.getToken().toString(); + }); + threads.add(t); + t.start(); + } + Assert.assertTrue(ready.await(10, TimeUnit.SECONDS)); + // Give every caller time to reach the wait before the single fetch completes. The assertion on the + // fetch count holds however the callers interleave; this only makes the burst a real burst. + final long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(2); + while (System.nanoTime() - deadline < 0 && !allParked(threads)) { + Thread.sleep(5); + } + source.openGate(); + List distinct = new ArrayList<>(); + for (int i = 0; i < callers; i++) { + threads.get(i).join(10_000); + Assert.assertFalse(threads.get(i).isAlive()); + if (!distinct.contains(results[i])) { + distinct.add(results[i]); + } + } + return distinct; + } + + private static boolean allParked(List threads) { + for (Thread t : threads) { + Thread.State s = t.getState(); + if (t.isAlive() && s != Thread.State.TIMED_WAITING && s != Thread.State.WAITING) { + return false; + } + } + return true; + } + + private static List listOf(String s) { + List l = new ArrayList<>(); + l.add(s); + return l; + } + + private static RefreshingTokenProvider.Builder provider( + ScriptedSource source, + FakeClock clock, + ManualScheduler scheduler, + double u + ) { + return RefreshingTokenProvider.builder(source) + .clock(clock) + .scheduler(scheduler) + .random(() -> u); + } + + private static List refresherThreads() { + List threads = new ArrayList<>(); + for (Thread t : Thread.getAllStackTraces().keySet()) { + if (t.getName().startsWith("qdb-token-refresh-") && t.isAlive()) { + threads.add(t); + } + } + return threads; + } + + private static String repeat(char c, int n) { + StringBuilder sb = new StringBuilder(n); + for (int i = 0; i < n; i++) { + sb.append(c); + } + return sb.toString(); + } + + private static Thread start(Runnable r) { + Thread t = new Thread(r, "token-test-caller"); + t.setDaemon(true); + t.start(); + return t; + } + + // Real monotonic time, with a wall clock the test can push forward. + private static final class SkewedClock implements RefreshingTokenProvider.Clock { + volatile long skewMillis; + + @Override + public long monotonicNanos() { + return System.nanoTime(); + } + + @Override + public long wallClockMillis() { + return System.currentTimeMillis() + skewMillis; + } + } +} diff --git a/core/src/test/java/io/questdb/client/test/cutlass/auth/TestTokenProviderFactory.java b/core/src/test/java/io/questdb/client/test/cutlass/auth/TestTokenProviderFactory.java new file mode 100644 index 000000000..357d80ad8 --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/cutlass/auth/TestTokenProviderFactory.java @@ -0,0 +1,100 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test.cutlass.auth; + +import io.questdb.client.cutlass.auth.ExpiringToken; +import io.questdb.client.cutlass.auth.TokenProviderFactory; +import io.questdb.client.cutlass.auth.TokenProviderRegistry; +import io.questdb.client.cutlass.auth.TokenSource; + +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; +import java.util.Map; +import java.util.concurrent.CopyOnWriteArrayList; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.function.Function; + +/** + * A {@link TokenProviderFactory} for tests, installed in place of {@link java.util.ServiceLoader} discovery with + * {@link #install(TestTokenProviderFactory...)}. It counts the sources it creates and the fetches they make. + * Registering one named {@code azure} lets core tests drive the {@code azure} connect-string keys without the + * optional module. + */ +public final class TestTokenProviderFactory implements TokenProviderFactory { + public final AtomicInteger created = new AtomicInteger(); + public final AtomicInteger fetches = new AtomicInteger(); + public final List> params = new CopyOnWriteArrayList<>(); + private final String name; + private final Function, ExpiringToken> tokens; + + public TestTokenProviderFactory(String name) { + this(name, p -> new ExpiringToken("TOKEN-" + name, System.currentTimeMillis() + 3_600_000L)); + } + + public TestTokenProviderFactory(String name, Function, ExpiringToken> tokens) { + this.name = name; + this.tokens = tokens; + } + + /** + * Makes exactly these factories visible to {@code token_provider} resolution. + */ + public static void install(TestTokenProviderFactory... factories) { + List list = new ArrayList<>(); + Collections.addAll(list, factories); + TokenProviderRegistry.setFactoriesForTesting(list); + } + + /** + * Restores {@link java.util.ServiceLoader} discovery. + */ + public static void uninstall() { + TokenProviderRegistry.setFactoriesForTesting(null); + } + + @Override + public TokenSource createSource(Map params) { + created.incrementAndGet(); + this.params.add(params); + return () -> { + fetches.incrementAndGet(); + return tokens.apply(params); + }; + } + + @Override + public String name() { + return name; + } + + @Override + public void validate(Map params) { + String resource = params.get("azure_resource"); + if (resource != null && resource.startsWith("invalid:")) { + throw new IllegalArgumentException("azure_resource rejected by the factory"); + } + } +} diff --git a/core/src/test/java/io/questdb/client/test/cutlass/auth/TokenProviderConfigTest.java b/core/src/test/java/io/questdb/client/test/cutlass/auth/TokenProviderConfigTest.java new file mode 100644 index 000000000..bfe077344 --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/cutlass/auth/TokenProviderConfigTest.java @@ -0,0 +1,255 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test.cutlass.auth; + +import io.questdb.client.QuestDB; +import io.questdb.client.Sender; +import io.questdb.client.cutlass.auth.RefreshingTokenProvider; +import io.questdb.client.cutlass.auth.TokenProviderRegistry; +import io.questdb.client.cutlass.auth.TokenProviderSpec; +import io.questdb.client.cutlass.line.LineSenderException; +import io.questdb.client.cutlass.qwp.client.QwpQueryClient; +import io.questdb.client.impl.ConfigString; +import io.questdb.client.impl.ConfigView; +import org.junit.After; +import org.junit.Assert; +import org.junit.Before; +import org.junit.Test; + +import java.util.Map; + +import static io.questdb.client.test.cutlass.auth.TokenTestKit.await; +import static io.questdb.client.test.tools.TestUtils.assertMemoryLeak; + +/** + * Conformance tests C18 (connect-string validation) and C19 (registry lifecycle) of the dynamic-credential + * specification (design/qwp-token-provider-spec.md, sections 7.2, 7.4 and 10). + */ +public class TokenProviderConfigTest { + private static final String WSS = "wss::addr=localhost:9000;"; + private TestTokenProviderFactory azure; + private TestTokenProviderFactory other; + + @Before + public void setUp() { + azure = new TestTokenProviderFactory("azure"); + other = new TestTokenProviderFactory("vault"); + TestTokenProviderFactory.install(azure, other); + } + + @After + public void tearDown() { + TestTokenProviderFactory.uninstall(); + } + + @Test + public void testAzureKeysAreNormalized() { + Map snap = Sender.builder(WSS + "token_provider=azure;azure_resource=api://qdb-app/.default;" + + "azure_client_id=AAAAAAAA-BBBB-CCCC-DDDD-EEEEEEEEEEEE;").wsConfigSnapshotForTest(); + Assert.assertEquals("azure", snap.get("token_provider")); + Assert.assertEquals("a trailing /.default must be stripped", "api://qdb-app", snap.get("azure_resource")); + Assert.assertEquals("aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee", snap.get("azure_client_id")); + + // equivalent strings share a registry key; a different identity does not + Assert.assertEquals(spec("token_provider=azure;azure_resource=api://qdb-app/.default;").registryKey(), + spec("token_provider=azure;azure_resource=api://qdb-app;").registryKey()); + Assert.assertNotEquals(spec("token_provider=azure;azure_resource=api://qdb-app;").registryKey(), + spec("token_provider=azure;azure_resource=api://qdb-app;" + + "azure_client_id=aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee;").registryKey()); + } + + @Test + public void testBothClientsAcceptTheKeys() throws Exception { + assertMemoryLeak(() -> { + String cfg = WSS + "token_provider=azure;azure_resource=api://qdb-app;"; + Assert.assertEquals("azure", Sender.builder(cfg).wsConfigSnapshotForTest().get("token_provider")); + try (QwpQueryClient client = QwpQueryClient.fromConfig(cfg)) { + Map snap = client.configSnapshotForTest(); + Assert.assertEquals("azure", snap.get("token_provider")); + Assert.assertEquals("api://qdb-app", snap.get("azure_resource")); + } + }); + } + + @Test + public void testRegistryLifecycle() { + // C19: lease release, linger, and re-acquiring a lease during the linger period. + TokenProviderRegistry registry = new TokenProviderRegistry(300); + TokenProviderSpec spec = spec("token_provider=vault;"); + TokenProviderRegistry.Lease a = registry.acquire(spec); + TokenProviderRegistry.Lease b = registry.acquire(spec); + RefreshingTokenProvider provider = a.provider(); + Assert.assertSame("one provider per configuration", provider, b.provider()); + Assert.assertEquals(1, other.created.get()); + Assert.assertEquals(2, registry.leaseCount(spec)); + Assert.assertTrue(provider.awaitReady(10_000)); + + a.close(); + a.close(); // releasing twice is harmless + Assert.assertEquals(1, registry.leaseCount(spec)); + b.close(); + Assert.assertEquals(0, registry.leaseCount(spec)); + Assert.assertTrue("the provider lingers after its last lease", registry.isActive(spec)); + Assert.assertFalse(provider.isClosed()); + + // re-acquired during the linger: the same warm provider, and the pending close is cancelled + TokenProviderRegistry.Lease c = registry.acquire(spec); + Assert.assertSame(provider, c.provider()); + Assert.assertEquals(1, other.created.get()); + c.close(); + await(provider::isClosed, 10_000, "the provider to close once the linger elapses"); + Assert.assertFalse(registry.isActive(spec)); + + // the next lease starts a fresh provider + TokenProviderRegistry.Lease d = registry.acquire(spec); + Assert.assertNotSame(provider, d.provider()); + Assert.assertEquals(2, other.created.get()); + d.close(); + await(d.provider()::isClosed, 10_000, "the second provider to close"); + } + + @Test + public void testRegistryWithoutLingerClosesOnTheLastRelease() { + TokenProviderRegistry registry = new TokenProviderRegistry(0); + TokenProviderSpec spec = spec("token_provider=vault;"); + TokenProviderRegistry.Lease lease = registry.acquire(spec); + RefreshingTokenProvider provider = lease.provider(); + lease.close(); + Assert.assertTrue(provider.isClosed()); + Assert.assertFalse(registry.isActive(spec)); + } + + @Test + public void testRegistrySharesOnlyEquivalentConfigurations() { + TokenProviderRegistry registry = new TokenProviderRegistry(0); + TokenProviderRegistry.Lease a = registry.acquire(spec("token_provider=azure;azure_resource=api://one;")); + TokenProviderRegistry.Lease b = registry.acquire(spec("token_provider=azure;azure_resource=api://one/.default;")); + TokenProviderRegistry.Lease c = registry.acquire(spec("token_provider=azure;azure_resource=api://two;")); + try { + Assert.assertSame(a.provider(), b.provider()); + Assert.assertNotSame(a.provider(), c.provider()); + Assert.assertEquals(2, azure.created.get()); + Assert.assertTrue(a.provider().getName(), a.provider().getName().startsWith("azure")); + } finally { + a.close(); + b.close(); + c.close(); + } + } + + @Test + public void testValidationRules() throws Exception { + // C18: every rule of section 7.2, on both clients, naming the offending key + assertMemoryLeak(() -> { + assertRejectedOnBoth(WSS + "token_provider=azure;azure_resource=x;token=abc;", + "token_provider cannot be combined with token"); + assertRejectedOnBoth(WSS + "token_provider=azure;azure_resource=x;username=u;password=p;", + "token_provider cannot be combined with username"); + assertRejectedOnBoth(WSS + "token_provider=;", "token_provider must not be empty", "supported values: [azure, vault]"); + assertRejectedOnBoth(WSS + "token_provider=kerberos;", "unsupported token_provider: kerberos", + "supported values: [azure, vault]"); + assertRejectedOnBoth(WSS + "token_provider=azure_imds;", "azure_imds is reserved", + "supported values: [azure, vault]"); + assertRejectedOnBoth(WSS + "azure_resource=api://x;", "azure_resource requires token_provider=azure"); + assertRejectedOnBoth(WSS + "azure_client_id=aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee;", + "azure_client_id requires token_provider=azure"); + assertRejectedOnBoth(WSS + "token_provider=vault;azure_resource=api://x;", + "azure_resource is only valid with token_provider=azure"); + assertRejectedOnBoth(WSS + "token_provider=azure;", "token_provider=azure requires azure_resource"); + assertRejectedOnBoth(WSS + "token_provider=azure;azure_resource=/.default;", "azure_resource must not be empty"); + assertRejectedOnBoth(WSS + "token_provider=azure;azure_resource=api://x;azure_client_id=not-a-guid;", + "invalid azure_client_id"); + assertRejectedOnBoth(WSS + "token_provider=azure;azure_resource=invalid:x;", "rejected by the factory"); + // decision D4: never over plain ws:: + assertRejectedOnBoth("ws::addr=localhost:9000;token_provider=azure;azure_resource=api://x;", + "token_provider requires the wss:: schema"); + // decision D9: never on a non-QWP schema + assertRejected(() -> Sender.fromConfig("http::addr=localhost:9000;token_provider=azure;"), + "token_provider is only supported with the wss:: schema"); + assertRejected(() -> Sender.fromConfig("https::addr=localhost:9000;azure_resource=api://x;"), + "azure_resource is only supported with the wss:: schema"); + + // an application-supplied provider is exclusive with token_provider + String cfg = WSS + "token_provider=azure;azure_resource=api://x;"; + assertRejected(() -> Sender.builder(cfg).httpTokenProvider(() -> "t"), + "application-supplied token provider cannot be combined with token_provider"); + assertRejected(() -> Sender.builder(cfg).httpToken("t"), "token cannot be combined with token_provider"); + try (QwpQueryClient client = QwpQueryClient.fromConfig(cfg)) { + assertRejected(() -> client.withBearerTokenProvider(() -> "t"), + "withBearerTokenProvider cannot be combined with token_provider"); + assertRejected(() -> client.withBearerToken("t"), "withBearerToken cannot be combined with token_provider"); + } + assertRejected(() -> QuestDB.builder().fromConfig(cfg).httpTokenProvider(() -> "t").build(), + "httpTokenProvider cannot be combined with token_provider"); + }); + } + + @Test + public void testValidationWithoutConnectingFetchesNoToken() throws Exception { + // section 7.2: validation that does not connect - a facade checking its configuration at build time, a + // parsed builder, a client that has not connected - must not fetch a token. + assertMemoryLeak(() -> { + String cfg = WSS + "token_provider=azure;azure_resource=api://validate-only;" + + "sender_pool_min=0;query_pool_min=0;"; + Sender.builder(cfg); + QwpQueryClient.validateConfig(new ConfigView(ConfigString.parse(cfg)), true); + QwpQueryClient.fromConfig(cfg).close(); + try (QuestDB ignored = QuestDB.connect(cfg)) { + Assert.assertEquals("no provider may be started", 0, azure.created.get()); + } + Assert.assertEquals(0, azure.created.get()); + Assert.assertEquals(0, azure.fetches.get()); + }); + } + + @Test + public void testWithoutTheModuleAzureIsRejectedWithAHint() { + TestTokenProviderFactory.install(); // nothing installed + assertRejectedOnBoth(WSS + "token_provider=azure;azure_resource=api://x;", + "token_provider=azure requires the org.questdb:questdb-client-azure module", + "supported values: none installed"); + } + + private static void assertRejected(Runnable action, String... fragments) { + try { + action.run(); + Assert.fail("expected the configuration to be rejected with: " + fragments[0]); + } catch (IllegalArgumentException | IllegalStateException | LineSenderException e) { + for (String fragment : fragments) { + Assert.assertTrue("[" + e.getMessage() + "] does not contain [" + fragment + ']', + e.getMessage().contains(fragment)); + } + } + } + + private static void assertRejectedOnBoth(String cfg, String... fragments) { + assertRejected(() -> Sender.builder(cfg), fragments); + assertRejected(() -> QwpQueryClient.fromConfig(cfg).close(), fragments); + } + + private static TokenProviderSpec spec(String keys) { + return TokenProviderSpec.parse(new ConfigView(ConfigString.parse(WSS + keys)), true); + } +} diff --git a/core/src/test/java/io/questdb/client/test/cutlass/auth/TokenTestKit.java b/core/src/test/java/io/questdb/client/test/cutlass/auth/TokenTestKit.java new file mode 100644 index 000000000..41f3d1bc5 --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/cutlass/auth/TokenTestKit.java @@ -0,0 +1,269 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test.cutlass.auth; + +import io.questdb.client.cutlass.auth.ExpiringToken; +import io.questdb.client.cutlass.auth.RefreshingTokenProvider; +import io.questdb.client.cutlass.auth.TokenSource; +import org.junit.Assert; + +import java.util.ArrayDeque; +import java.util.ArrayList; +import java.util.List; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.atomic.AtomicLong; +import java.util.function.BooleanSupplier; +import java.util.function.Supplier; + +/** + * Deterministic fixtures for the dynamic-credential conformance tests (design/qwp-token-provider-spec.md, section + * 10): a fake clock, a scheduler the test drives by hand, and a scripted token source. + */ +public final class TokenTestKit { + + private TokenTestKit() { + } + + public static void await(BooleanSupplier condition, long timeoutMillis, String what) { + final long deadline = System.nanoTime() + TimeUnit.MILLISECONDS.toNanos(timeoutMillis); + while (!condition.getAsBoolean()) { + if (System.nanoTime() - deadline > 0) { + Assert.fail("timed out after " + timeoutMillis + " ms waiting for " + what); + } + try { + Thread.sleep(2); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + Assert.fail("interrupted while waiting for " + what); + } + } + } + + /** + * A wall clock and a monotonic clock that only move when the test moves them. {@link #advanceMillis(long)} + * moves both; {@link #jumpWallMillis(long)} moves the wall clock alone, the way a suspended host or an NTP + * step does. + */ + public static final class FakeClock implements RefreshingTokenProvider.Clock { + private final AtomicLong mono = new AtomicLong(1_000_000_000L); + private final AtomicLong wall; + + public FakeClock(long wallStartMillis) { + this.wall = new AtomicLong(wallStartMillis); + } + + public void advanceMillis(long millis) { + wall.addAndGet(millis); + mono.addAndGet(TimeUnit.MILLISECONDS.toNanos(millis)); + } + + public void jumpWallMillis(long millis) { + wall.addAndGet(millis); + } + + @Override + public long monotonicNanos() { + return mono.get(); + } + + @Override + public long wallClockMillis() { + return wall.get(); + } + } + + /** + * Runs nothing on its own: the test picks the next task, optionally advances the fake clock to its due time, + * and runs it on the test thread. Tasks run outside the scheduler's monitor, because the provider calls + * {@link #schedule} while holding its own lock. + */ + public static final class ManualScheduler implements RefreshingTokenProvider.Scheduler { + private final FakeClock clock; + private final List tasks = new ArrayList<>(); + private volatile boolean shutdown; + + public ManualScheduler(FakeClock clock) { + this.clock = clock; + } + + /** + * Advances the clock to the next task's due time (if it lies ahead) and runs it. + * + * @return the task that ran + */ + public Task advanceAndRunNext() { + Task t = takeNext(); + Assert.assertNotNull("no fetch is scheduled", t); + long ahead = t.dueNanos - clock.monotonicNanos(); + if (ahead > 0) { + clock.advanceMillis(TimeUnit.NANOSECONDS.toMillis(ahead)); + } + t.runnable.run(); + return t; + } + + public boolean isShutdown() { + return shutdown; + } + + /** + * The earliest live task, without removing it, or null. + */ + public synchronized Task peek() { + Task best = null; + for (int i = 0, n = tasks.size(); i < n; i++) { + Task t = tasks.get(i); + if (!t.cancelled && (best == null || t.dueNanos - best.dueNanos < 0)) { + best = t; + } + } + return best; + } + + /** + * Runs the earliest live task now, without moving the clock. + */ + public Task runNext() { + Task t = takeNext(); + Assert.assertNotNull("no fetch is scheduled", t); + t.runnable.run(); + return t; + } + + @Override + public synchronized Task schedule(Runnable task, long delayNanos) { + Task t = new Task(task, clock.monotonicNanos() + delayNanos, delayNanos); + tasks.add(t); + return t; + } + + @Override + public void shutdown() { + shutdown = true; + } + + private synchronized Task takeNext() { + Task t = peek(); + if (t != null) { + tasks.remove(t); + } + return t; + } + + public static final class Task implements RefreshingTokenProvider.ScheduledTask { + public final long delayNanos; + final long dueNanos; + final Runnable runnable; + volatile boolean cancelled; + + Task(Runnable runnable, long dueNanos, long delayNanos) { + this.runnable = runnable; + this.dueNanos = dueNanos; + this.delayNanos = delayNanos; + } + + @Override + public void cancel() { + cancelled = true; + } + + public long delayMillis() { + return TimeUnit.NANOSECONDS.toMillis(delayNanos); + } + } + } + + /** + * A token source that replays a script of results. Each step either returns a token, throws, or computes + * its result from the clock at call time. When the script runs out the last step repeats. An optional gate + * blocks every fetch until the test opens it. + */ + public static final class ScriptedSource implements TokenSource { + private final AtomicInteger calls = new AtomicInteger(); + private final ArrayDeque> script = new ArrayDeque<>(); + private volatile CountDownLatch gate; + private Supplier last; + + public int calls() { + return calls.get(); + } + + @Override + public ExpiringToken fetchToken() { + calls.incrementAndGet(); + CountDownLatch g = gate; + if (g != null) { + try { + g.await(); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new IllegalStateException("fetch interrupted"); + } + } + Supplier step; + synchronized (this) { + step = script.poll(); + if (step == null) { + step = last; + } else { + last = step; + } + } + Assert.assertNotNull("the source was called with nothing scripted", step); + return step.get(); + } + + public void openGate() { + CountDownLatch g = gate; + gate = null; + if (g != null) { + g.countDown(); + } + } + + public void setGate() { + gate = new CountDownLatch(1); + } + + public ScriptedSource then(Supplier step) { + synchronized (this) { + script.add(step); + } + return this; + } + + public ScriptedSource thenThrow(RuntimeException e) { + return then(() -> { + throw e; + }); + } + + public ScriptedSource thenToken(String token, long expiresAtEpochMillis) { + return then(() -> new ExpiringToken(token, expiresAtEpochMillis)); + } + } +} diff --git a/core/src/test/java/io/questdb/client/test/cutlass/http/BearerChallengeTest.java b/core/src/test/java/io/questdb/client/test/cutlass/http/BearerChallengeTest.java new file mode 100644 index 000000000..0fdae50d9 --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/cutlass/http/BearerChallengeTest.java @@ -0,0 +1,94 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test.cutlass.http; + +import io.questdb.client.cutlass.http.BearerChallenge; +import io.questdb.client.cutlass.qwp.client.QwpAuthFailedException; +import org.junit.Assert; +import org.junit.Test; + +public class BearerChallengeTest { + + @Test + public void testAuthFailedExceptionClassification() { + QwpAuthFailedException plain = new QwpAuthFailedException(401, "h", 1); + Assert.assertTrue(plain.isTokenRefreshable()); + Assert.assertNull(plain.getBearerError()); + Assert.assertEquals("auth-rejected: WebSocket upgrade rejected with HTTP 401 for h:1", plain.getMessage()); + + QwpAuthFailedException invalid = new QwpAuthFailedException(401, "h", 1, "Bearer error=\"invalid_token\""); + Assert.assertTrue(invalid.isTokenRefreshable()); + Assert.assertEquals("auth-rejected: WebSocket upgrade rejected with HTTP 401 [error=invalid_token] for h:1", + invalid.getMessage()); + + QwpAuthFailedException scope = new QwpAuthFailedException(401, "h", 1, "Bearer error=\"insufficient_scope\""); + Assert.assertFalse(scope.isTokenRefreshable()); + + // decision D2: a 403 never qualifies, challenge or not + Assert.assertFalse(new QwpAuthFailedException(403, "h", 1).isTokenRefreshable()); + Assert.assertFalse(new QwpAuthFailedException(403, "h", 1, "Bearer error=\"invalid_token\"").isTokenRefreshable()); + + // a hostile error value is sanitized and capped before it reaches the message + QwpAuthFailedException hostile = new QwpAuthFailedException(401, "h", 1, + "Bearer error=\"x\r\nForged: 1\u202E" + new String(new char[200]).replace('\0', 'y') + "\""); + Assert.assertFalse(hostile.getMessage().contains("\r") || hostile.getMessage().contains("\u202E")); + Assert.assertTrue(hostile.getBearerError().length() <= 64); + } + + @Test + public void testBearerError() { + Assert.assertNull(BearerChallenge.bearerError(null)); + Assert.assertNull(BearerChallenge.bearerError("")); + Assert.assertNull(BearerChallenge.bearerError("Bearer")); + Assert.assertNull(BearerChallenge.bearerError("Bearer realm=\"questdb\"")); + Assert.assertEquals("invalid_token", + BearerChallenge.bearerError("Bearer realm=\"questdb\", error=\"invalid_token\", error_description=\"The access token expired\"")); + Assert.assertEquals("insufficient_scope", BearerChallenge.bearerError("Bearer error=insufficient_scope")); + Assert.assertEquals("invalid_token", BearerChallenge.bearerError("bearer ERROR=\"invalid_token\"")); + // a Bearer challenge after another scheme, with token68 and quoted commas in between + Assert.assertEquals("invalid_token", BearerChallenge.bearerError( + "Negotiate YIIB9QYGKwYBBQUCoIIB==, Basic realm=\"a, b\", Bearer error=\"invalid_token\"")); + // an error parameter of another scheme does not count + Assert.assertNull(BearerChallenge.bearerError("Basic error=\"invalid_token\"")); + Assert.assertNull(BearerChallenge.bearerError("Basic realm=\"x\", DPoP error=\"invalid_token\"")); + // escaped quotes inside a quoted value + Assert.assertEquals("a\"b", BearerChallenge.bearerError("Bearer error=\"a\\\"b\"")); + // only the first error counts + Assert.assertEquals("invalid_token", BearerChallenge.bearerError("Bearer error=invalid_token, error=other")); + // garbage does not throw + Assert.assertNull(BearerChallenge.bearerError("\u0000\"=,,=\"Bearer")); + } + + @Test + public void testRefreshDecision() { + // section 8.2: without a Bearer challenge carrying an error, the status code alone decides + Assert.assertTrue(BearerChallenge.allowsTokenRefresh(null)); + Assert.assertTrue(BearerChallenge.allowsTokenRefresh("Basic realm=\"questdb\"")); + Assert.assertTrue(BearerChallenge.allowsTokenRefresh("Bearer realm=\"questdb\"")); + Assert.assertTrue(BearerChallenge.allowsTokenRefresh("Bearer error=\"invalid_token\"")); + Assert.assertFalse(BearerChallenge.allowsTokenRefresh("Bearer error=\"insufficient_scope\"")); + Assert.assertFalse(BearerChallenge.allowsTokenRefresh("Bearer error=\"invalid_request\"")); + } +} diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/ConnectionHealthTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/ConnectionHealthTest.java new file mode 100644 index 000000000..b8c8b487e --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/ConnectionHealthTest.java @@ -0,0 +1,431 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test.cutlass.qwp.client; + +import io.questdb.client.ConnectionHealth; +import io.questdb.client.QuestDB; +import io.questdb.client.Sender; +import io.questdb.client.SenderError; +import io.questdb.client.cutlass.auth.TokenUnavailableException; +import io.questdb.client.cutlass.line.LineSenderException; +import io.questdb.client.cutlass.qwp.client.QwpColumnBatch; +import io.questdb.client.cutlass.qwp.client.QwpColumnBatchHandler; +import io.questdb.client.cutlass.qwp.client.QwpEgressMsgKind; +import io.questdb.client.cutlass.qwp.client.QwpQueryClient; +import io.questdb.client.cutlass.qwp.client.sf.cursor.OrphanScanner; +import io.questdb.client.cutlass.qwp.protocol.QwpConstants; +import io.questdb.client.test.cutlass.qwp.client.WebSocketDynamicCredentialTest.ErrorCollector; +import io.questdb.client.test.cutlass.qwp.websocket.TestWebSocketServer; +import org.junit.Assert; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.TemporaryFolder; + +import java.io.IOException; +import java.nio.ByteBuffer; +import java.nio.ByteOrder; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicReference; + +import static io.questdb.client.test.cutlass.auth.TokenTestKit.await; +import static io.questdb.client.test.tools.TestUtils.assertMemoryLeak; + +/** + * Conformance tests C21 (connection health, section 8.4) and C22 (the authentication-outage deadline, section + * 8.5) of the dynamic-credential specification (design/qwp-token-provider-spec.md). + */ +public class ConnectionHealthTest { + private static final String SENTINEL = "SENTINEL-TOKEN-7f3a91"; + + @Rule + public final TemporaryFolder temp = TemporaryFolder.builder().assureDeletion().build(); + + @Test(timeout = 60_000) + public void testDeadlineIsNotResetOrFiredByOtherFailureClasses() throws Exception { + // C22: rounds that fail for other reasons neither reset the clock nor fire the deadline. A credential + // outage, then role rejects and a transport outage that together outlast the deadline, then the + // credential outage again: the deadline fires on the first authentication-class round after them. + assertMemoryLeak(() -> { + final long deadlineMillis = 1_000; + AtomicReference mode = new AtomicReference<>("ok"); + ErrorCollector errors = new ErrorCollector(); + TestWebSocketServer server = startServer(new WebSocketDynamicCredentialTest.DropAfterFirstAckHandler()); + try (Sender sender = Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + server.getPort()) + .reconnectInitialBackoffMillis(10) + .reconnectMaxBackoffMillis(20) + .authFailureMaxDurationMillis(deadlineMillis) + .errorHandler(errors) + .httpTokenProvider(() -> { + if ("fail".equals(mode.get())) { + throw TokenUnavailableException.retryable("IdP unreachable"); + } + return "TOKEN"; + }) + .build()) { + mode.set("fail"); + sender.table("t").longColumn("v", 1).atNow(); + sender.flush(); // ACKed, then the server drops the connection + await(() -> errors.count("credential-unavailable") >= 1, 10_000, "the credential outage"); + long authOutageStart = System.nanoTime(); + // Other classes: every endpoint role-rejects (set before the provider recovers, so no upgrade can + // succeed in between and reset the clock), then the server goes away (transport). + server.setRejectWithRole("REPLICA"); + mode.set("ok"); + Thread.sleep(deadlineMillis * 2 / 3); + server.close(); + Thread.sleep(deadlineMillis * 2 / 3); + Assert.assertTrue(TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - authOutageStart) > deadlineMillis); + Assert.assertEquals("role rejects and transport failures must not fire the deadline", + 0, errors.terminalCount()); + // the credential outage returns: the clock was never reset, so the first such round fires + mode.set("fail"); + await(() -> errors.terminalCount() == 1, 10_000, "the deadline to fire"); + SenderError terminal = terminal(errors); + Assert.assertTrue(terminal.getServerMessage(), + terminal.getServerMessage().contains("credential-unavailable persisted for")); + Assert.assertEquals(ConnectionHealth.State.FAILED, sender.health().getState()); + } catch (LineSenderException expected) { + // close() rethrows the latched terminal + Assert.assertTrue(expected.getMessage(), expected.getMessage().contains("authentication outage deadline")); + } finally { + server.close(); + } + }); + } + + @Test(timeout = 60_000) + public void testDeadlineMakesTheSenderTerminalAndKeepsTheData() throws Exception { + // C22: set, the sender becomes terminal on an authentication-class round once the duration has passed; + // the error names the class and the elapsed time, is reported as terminal and thrown from later producer + // calls, and the unacknowledged rows stay on disk. + assertMemoryLeak(() -> { + String sfDir = temp.newFolder("deadline").getAbsolutePath(); + ErrorCollector errors = new ErrorCollector(); + AtomicReference rejectWith = new AtomicReference<>(0); + try (TestWebSocketServer server = startServer(new WebSocketDynamicCredentialTest.DropAfterFirstAckHandler())) { + server.setAuthorizationValidator(h -> rejectWith.get()); + Sender sender = Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + server.getPort()) + .storeAndForwardDir(sfDir) + .senderId("deadline") + .reconnectInitialBackoffMillis(10) + .reconnectMaxBackoffMillis(20) + .closeFlushTimeoutMillis(0) + .authFailureMaxDurationMillis(500) + .errorHandler(errors) + .httpTokenProvider(() -> SENTINEL) + .build(); + try { + rejectWith.set(401); + sender.table("t").longColumn("v", 1).atNow(); + sender.flush(); // ACKed, then dropped: every reconnect is now rejected + await(() -> server.authRejectCount() >= 1, 10_000, "the first 401"); + long start = System.nanoTime(); + sender.table("t").longColumn("v", 2).atNow(); + sender.flush(); // buffered in store-and-forward + await(() -> errors.terminalCount() == 1, 10_000, "the deadline to fire"); + long elapsedMillis = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - start); + Assert.assertTrue("must not fire before the deadline, fired at " + elapsedMillis + " ms", + elapsedMillis >= 400); + SenderError terminal = terminal(errors); + Assert.assertEquals(SenderError.Category.SECURITY_ERROR, terminal.getCategory()); + Assert.assertTrue(terminal.getServerMessage(), terminal.getServerMessage().contains("auth-rejected")); + Assert.assertTrue(terminal.getServerMessage(), terminal.getServerMessage().contains("persisted for")); + Assert.assertFalse(terminal.getServerMessage().contains(SENTINEL)); + try { + sender.table("t").longColumn("v", 3).atNow(); + sender.flush(); + Assert.fail("later producer calls must throw the terminal"); + } catch (LineSenderException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("authentication outage deadline")); + } + Assert.assertEquals(ConnectionHealth.State.FAILED, sender.health().getState()); + } finally { + try { + sender.close(); + } catch (LineSenderException ignored) { + // close() may rethrow the terminal + } + } + } + Assert.assertEquals("the unacknowledged rows stay on disk for a later sender or an orphan drain", + 1, OrphanScanner.scan(sfDir, "someone-else").size()); + }); + } + + @Test(timeout = 60_000) + public void testDeadlineIsResetByASuccessfulUpgradeAndUnsetMeansNoDeadline() throws Exception { + // C22: a successful upgrade resets the clock; and without the key, a 401 outage is retried indefinitely. + assertMemoryLeak(() -> { + for (boolean armed : new boolean[]{true, false}) { + ErrorCollector errors = new ErrorCollector(); + AtomicReference rejectWith = new AtomicReference<>(0); + WebSocketDynamicCredentialTest.AckHandler handler = new WebSocketDynamicCredentialTest.AckHandler(); + try (TestWebSocketServer server = startServer(handler)) { + server.setAuthorizationValidator(h -> rejectWith.get()); + Sender.LineSenderBuilder b = Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + server.getPort()) + .reconnectInitialBackoffMillis(10) + .reconnectMaxBackoffMillis(20) + .errorHandler(errors) + .httpTokenProvider(() -> "TOKEN"); + if (armed) { + b.authFailureMaxDurationMillis(800); + } + try (Sender sender = b.build()) { + for (int outage = 0; outage < 2; outage++) { + rejectWith.set(401); + server.dropAllConnections(); + Thread.sleep(500); // under the deadline + rejectWith.set(0); // a successful upgrade ends the outage + final long acked = handler.frames.get(); + sender.table("t").longColumn("v", outage).atNow(); + sender.flush(); + await(() -> handler.frames.get() > acked, 10_000, "recovery"); + } + Assert.assertEquals("two outages of 500 ms each, the clock reset in between [armed=" + armed + ']', + 0, errors.terminalCount()); + if (!armed) { + rejectWith.set(401); + server.dropAllConnections(); + Thread.sleep(1_500); + Assert.assertEquals("no deadline: retried indefinitely", 0, errors.terminalCount()); + Assert.assertTrue(errors.count("auth-rejected") > 0); + rejectWith.set(0); + } + } + } + } + }); + } + + @Test(timeout = 60_000) + public void testFacadeAggregatesEveryPooledConnection() throws Exception { + assertMemoryLeak(() -> { + try (TestWebSocketServer server = startServer(new ExecDoneHandler())) { + server.setSendServerInfo(true); + String cfg = "ws::addr=localhost:" + server.getPort() + ";sender_pool_min=2;query_pool_min=1;"; + try (QuestDB db = QuestDB.connect(cfg)) { + ConnectionHealth.Aggregate health = db.health(); + Assert.assertEquals(3, health.total()); + Assert.assertEquals(3, health.count(ConnectionHealth.State.CONNECTED)); + Assert.assertEquals(ConnectionHealth.NONE, health.getOldestOutageSinceEpochMillis()); + Assert.assertNull(health.getLastFailure()); + try (Sender s = db.borrowSender()) { + Assert.assertEquals(ConnectionHealth.State.CONNECTED, s.health().getState()); + } + } + } + }); + } + + @Test(timeout = 60_000) + public void testQueryClientHealthFollowsConnectFailoverAndRecovery() throws Exception { + // C21 for egress: connecting, connected, reconnecting after a failed failover - with the class of the + // failure - and connected again on the next operation; closed at the end. + assertMemoryLeak(() -> { + try (TestWebSocketServer a = startServer(new ExecDoneHandler()); + TestWebSocketServer b = startServer(new ExecDoneHandler())) { + a.setSendServerInfo(true); + b.setSendServerInfo(true); + QwpQueryClient client = QwpQueryClient.fromConfig("ws::addr=localhost:" + a.getPort() + + ",localhost:" + b.getPort() + ";failover_backoff_initial_ms=0;") + .withBearerTokenProvider(() -> SENTINEL); + try { + ConnectionHealth h = client.health(); + Assert.assertEquals(ConnectionHealth.State.CONNECTING, h.getState()); + Assert.assertNotEquals(ConnectionHealth.NONE, h.getOutageSinceEpochMillis()); + + client.connect(); + h = client.health(); + Assert.assertEquals(ConnectionHealth.State.CONNECTED, h.getState()); + Assert.assertEquals(ConnectionHealth.NONE, h.getOutageSinceEpochMillis()); + Assert.assertNotEquals(ConnectionHealth.NONE, h.getLastConnectedAtEpochMillis()); + + b.setAuthorizationValidator(x -> 403); + a.close(); + client.execute("SELECT 1", new NoopHandler()); + h = client.health(); + Assert.assertEquals(ConnectionHealth.State.RECONNECTING, h.getState()); + Assert.assertEquals(1, h.getFailedRounds()); + Assert.assertEquals(ConnectionHealth.FailureClass.AUTH_REJECTED, h.getLastFailure().getFailureClass()); + Assert.assertEquals(403, h.getLastFailure().getStatusCode()); + Assert.assertNotEquals(ConnectionHealth.NONE, h.getOutageSinceEpochMillis()); + Assert.assertFalse(h.toString().contains(SENTINEL)); + + b.setAuthorizationValidator(null); + client.execute("SELECT 1", new NoopHandler()); + h = client.health(); + Assert.assertEquals(ConnectionHealth.State.CONNECTED, h.getState()); + Assert.assertEquals(0, h.getFailedRounds()); + Assert.assertEquals("last_failure is kept after recovery", + ConnectionHealth.FailureClass.AUTH_REJECTED, h.getLastFailure().getFailureClass()); + } finally { + client.close(); + } + Assert.assertEquals(ConnectionHealth.State.CLOSED, client.health().getState()); + } + }); + } + + @Test(timeout = 60_000) + public void testSenderHealthMovesThroughEveryState() throws Exception { + // C21: connecting -> connected -> reconnecting (with 401 and credential-unavailable failures) -> + // connected. outage_since and failed_rounds reset on recovery; last_failure is kept; no credential appears. + assertMemoryLeak(() -> { + AtomicReference providerMode = new AtomicReference<>("fail"); + AtomicReference rejectWith = new AtomicReference<>(0); + WebSocketDynamicCredentialTest.AckHandler handler = new WebSocketDynamicCredentialTest.AckHandler(); + try (TestWebSocketServer server = startServer(handler)) { + server.setAuthorizationValidator(h -> rejectWith.get()); + long created = System.currentTimeMillis(); + Sender sender = Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + server.getPort()) + .initialConnectMode(Sender.InitialConnectMode.ASYNC) + .reconnectInitialBackoffMillis(10) + .reconnectMaxBackoffMillis(20) + .httpTokenProvider(() -> { + if ("fail".equals(providerMode.get())) { + throw TokenUnavailableException.retryable("IdP unreachable"); + } + return SENTINEL; + }) + .build(); + try { + // connecting, with credential-unavailable rounds + await(() -> sender.health().getFailedRounds() >= 2, 10_000, "failed rounds while connecting"); + ConnectionHealth h = sender.health(); + Assert.assertEquals(ConnectionHealth.State.CONNECTING, h.getState()); + Assert.assertTrue(h.getOutageSinceEpochMillis() >= created - 1_000); + Assert.assertEquals(ConnectionHealth.NONE, h.getLastConnectedAtEpochMillis()); + Assert.assertEquals(ConnectionHealth.FailureClass.CREDENTIAL_UNAVAILABLE, + h.getLastFailure().getFailureClass()); + Assert.assertTrue(h.getLastFailure().getMessage(), h.getLastFailure().getMessage().contains("IdP unreachable")); + + // connected + providerMode.set("ok"); + await(() -> sender.health().getState() == ConnectionHealth.State.CONNECTED, 10_000, "connected"); + h = sender.health(); + Assert.assertEquals(ConnectionHealth.NONE, h.getOutageSinceEpochMillis()); + Assert.assertEquals(0, h.getFailedRounds()); + Assert.assertNotEquals(ConnectionHealth.NONE, h.getLastConnectedAtEpochMillis()); + Assert.assertEquals("last_failure is kept after recovery", + ConnectionHealth.FailureClass.CREDENTIAL_UNAVAILABLE, h.getLastFailure().getFailureClass()); + + // reconnecting, rejected with 401 + rejectWith.set(401); + server.dropAllConnections(); + await(() -> sender.health().getState() == ConnectionHealth.State.RECONNECTING + && sender.health().getLastFailure().getFailureClass() == ConnectionHealth.FailureClass.AUTH_REJECTED, + 10_000, "reconnecting with 401"); + h = sender.health(); + Assert.assertEquals(401, h.getLastFailure().getStatusCode()); + Assert.assertNotEquals(ConnectionHealth.NONE, h.getOutageSinceEpochMillis()); + long outageSince = h.getOutageSinceEpochMillis(); + + // still reconnecting, now the provider fails + providerMode.set("fail"); + await(() -> sender.health().getLastFailure().getFailureClass() + == ConnectionHealth.FailureClass.CREDENTIAL_UNAVAILABLE, 10_000, "credential-unavailable"); + h = sender.health(); + Assert.assertEquals(ConnectionHealth.State.RECONNECTING, h.getState()); + Assert.assertEquals("the outage did not restart", outageSince, h.getOutageSinceEpochMillis()); + Assert.assertTrue(h.getFailedRounds() >= 2); + Assert.assertFalse("no credential may appear: " + h, h.toString().contains(SENTINEL)); + + // back to connected + providerMode.set("ok"); + rejectWith.set(0); + await(() -> sender.health().getState() == ConnectionHealth.State.CONNECTED, 10_000, "recovered"); + h = sender.health(); + Assert.assertEquals(ConnectionHealth.NONE, h.getOutageSinceEpochMillis()); + Assert.assertEquals(0, h.getFailedRounds()); + Assert.assertEquals(ConnectionHealth.FailureClass.CREDENTIAL_UNAVAILABLE, h.getLastFailure().getFailureClass()); + Assert.assertFalse(h.toString().contains(SENTINEL)); + } finally { + sender.close(); + } + Assert.assertEquals(ConnectionHealth.State.CLOSED, sender.health().getState()); + } + }); + } + + private static TestWebSocketServer startServer(TestWebSocketServer.WebSocketServerHandler handler) + throws IOException, InterruptedException { + TestWebSocketServer server = new TestWebSocketServer(handler); + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + return server; + } + + private static SenderError terminal(ErrorCollector errors) { + for (SenderError e : errors.errors) { + if (e.getAppliedPolicy() == SenderError.Policy.TERMINAL) { + return e; + } + } + throw new AssertionError("no terminal error"); + } + + private static final class ExecDoneHandler implements TestWebSocketServer.WebSocketServerHandler { + private final WebSocketDynamicCredentialTest.AckHandler ack = new WebSocketDynamicCredentialTest.AckHandler(); + + @Override + public void onBinaryMessage(TestWebSocketServer.ClientHandler client, byte[] data) { + if (data.length > 0 && data[0] == QwpEgressMsgKind.QUERY_REQUEST) { + int bodyLen = 1 + 8 + 1 + 1; + byte[] frame = new byte[QwpConstants.HEADER_SIZE + bodyLen]; + ByteBuffer bb = ByteBuffer.wrap(frame).order(ByteOrder.LITTLE_ENDIAN); + bb.put((byte) 'Q').put((byte) 'W').put((byte) 'P').put((byte) '1'); + bb.put((byte) 1).put((byte) 0).putShort((short) 0).putInt(bodyLen); + bb.put(QwpEgressMsgKind.EXEC_DONE); + bb.put(data, 1, 8); + bb.put((byte) 0).put((byte) 0); + try { + client.sendBinary(frame); + } catch (IOException ignored) { + // surfaces to the client as a transport error + } + } else { + ack.onBinaryMessage(client, data); + } + } + } + + private static final class NoopHandler implements QwpColumnBatchHandler { + @Override + public void onBatch(QwpColumnBatch batch) { + } + + @Override + public void onEnd(long totalRows) { + } + + @Override + public void onError(byte status, String message) { + } + } +} diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/QwpQueryClientDynamicCredentialTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/QwpQueryClientDynamicCredentialTest.java new file mode 100644 index 000000000..addb4f52e --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/QwpQueryClientDynamicCredentialTest.java @@ -0,0 +1,349 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test.cutlass.qwp.client; + +import io.questdb.client.Query; +import io.questdb.client.QueryException; +import io.questdb.client.QuestDB; +import io.questdb.client.cutlass.auth.RefreshingTokenProvider; +import io.questdb.client.cutlass.auth.TokenUnavailableException; +import io.questdb.client.cutlass.qwp.client.QwpAuthFailedException; +import io.questdb.client.cutlass.qwp.client.QwpColumnBatch; +import io.questdb.client.cutlass.qwp.client.QwpColumnBatchHandler; +import io.questdb.client.cutlass.qwp.client.QwpCredentialUnavailableException; +import io.questdb.client.cutlass.qwp.client.QwpEgressMsgKind; +import io.questdb.client.cutlass.qwp.client.QwpQueryClient; +import io.questdb.client.cutlass.qwp.protocol.QwpConstants; +import io.questdb.client.test.cutlass.auth.TokenTestKit.ScriptedSource; +import io.questdb.client.test.cutlass.qwp.websocket.TestWebSocketServer; +import org.junit.Assert; +import org.junit.Test; + +import java.io.IOException; +import java.nio.ByteBuffer; +import java.nio.ByteOrder; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.List; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.atomic.AtomicReference; + +import static io.questdb.client.test.tools.TestUtils.assertMemoryLeak; + +/** + * The dynamic-credential specification (design/qwp-token-provider-spec.md) on the QWP egress client: the one + * retry after a 401 on connect and on failover reconnect (section 8.2, C10/C11/C23), failure-class naming in + * the reported error, and egress recovery - a client whose failover reconnect failed reconnects on its next + * operation rather than staying unusable (section 8.3). + */ +public class QwpQueryClientDynamicCredentialTest { + private static final long HOUR = 3_600_000L; + + @Test(timeout = 30_000) + public void testConnectCredentialUnavailableNamesTheClassAndContactsNoEndpoint() throws Exception { + assertMemoryLeak(() -> { + try (TestWebSocketServer server = startServer(); + QwpQueryClient client = QwpQueryClient.fromConfig("ws::addr=localhost:" + server.getPort() + ";") + .withBearerTokenProvider(() -> { + throw TokenUnavailableException.retryable("IMDS unreachable"); + })) { + try { + client.connect(); + Assert.fail(); + } catch (QwpCredentialUnavailableException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().startsWith("credential-unavailable: ")); + Assert.assertTrue(e.getMessage(), e.getMessage().contains("IMDS unreachable")); + Assert.assertTrue(e.isRetryable()); + Assert.assertTrue(e.getCause() instanceof TokenUnavailableException); + } + Assert.assertEquals("no endpoint may be contacted", 0, server.upgradeRequestCount()); + } + }); + } + + @Test(timeout = 30_000) + public void testConnectInsufficientScopeIsNotRetried() throws Exception { + // C23 on egress: a Bearer challenge with an error other than invalid_token gets no retry. + assertMemoryLeak(() -> { + ScriptedSource source = twoTokens(); + try (TestWebSocketServer server = startServer(); + RefreshingTokenProvider provider = RefreshingTokenProvider.builder(source).build(); + QwpQueryClient client = QwpQueryClient.fromConfig("ws::addr=localhost:" + server.getPort() + ";") + .withBearerTokenProvider(provider)) { + server.setAuthorizationValidator(h -> "Bearer T1".equals(h) ? 401 : 0); + server.setRejectWwwAuthenticate("Bearer error=\"insufficient_scope\""); + try { + client.connect(); + Assert.fail(); + } catch (QwpAuthFailedException e) { + Assert.assertEquals("insufficient_scope", e.getBearerError()); + Assert.assertTrue(e.getMessage(), e.getMessage().startsWith("auth-rejected")); + } + Assert.assertEquals(1, server.upgradeRequestCount()); + Assert.assertEquals(1, source.calls()); + } + }); + } + + @Test(timeout = 30_000) + public void testConnectPersistent401IsRetriedOnceThenFails() throws Exception { + // C11 on egress: the failure survives the one retry and fails the operation. + assertMemoryLeak(() -> { + ScriptedSource source = twoTokens(); + try (TestWebSocketServer server = startServer(); + RefreshingTokenProvider provider = RefreshingTokenProvider.builder(source).build(); + QwpQueryClient client = QwpQueryClient.fromConfig("ws::addr=localhost:" + server.getPort() + ";") + .withBearerTokenProvider(provider)) { + server.setAuthorizationValidator(h -> 401); + try { + client.connect(); + Assert.fail(); + } catch (QwpAuthFailedException e) { + Assert.assertEquals(401, e.getStatusCode()); + } + Assert.assertEquals(Arrays.asList("Bearer T1", "Bearer T2"), headers(server)); + } + }); + } + + @Test(timeout = 30_000) + public void testConnectStaleTokenIsRetriedOnceWithTheRefreshedToken() throws Exception { + // C10 on egress: one 401 for the stale token, then the same endpoint with the refreshed one. + assertMemoryLeak(() -> { + ScriptedSource source = twoTokens(); + try (TestWebSocketServer server = startServer(); + RefreshingTokenProvider provider = RefreshingTokenProvider.builder(source).build(); + QwpQueryClient client = QwpQueryClient.fromConfig("ws::addr=localhost:" + server.getPort() + ";") + .withBearerTokenProvider(provider)) { + server.setAuthorizationValidator(h -> "Bearer T1".equals(h) ? 401 : 0); + client.connect(); + Assert.assertTrue(client.isConnected()); + Assert.assertEquals(Arrays.asList("Bearer T1", "Bearer T2"), headers(server)); + Assert.assertEquals(1, server.authRejectCount()); + ResultCollector result = new ResultCollector(); + client.execute("SELECT 1", result); + Assert.assertNull(result.error.get()); + } + }); + } + + @Test(timeout = 30_000) + public void testFailedFailoverReconnectRecoversOnTheNextExecute() throws Exception { + // Egress recovery (section 8.3): a failover reconnect that fails must not leave the client permanently + // unusable. Before the fix, every later execute() threw "QwpQueryClient not connected". + assertMemoryLeak(() -> { + try (TestWebSocketServer a = startServer(); + TestWebSocketServer b = startServer(); + QwpQueryClient client = QwpQueryClient.fromConfig( + "ws::addr=localhost:" + a.getPort() + ",localhost:" + b.getPort() + + ";failover_backoff_initial_ms=0;") + .withBearerTokenProvider(() -> "TOKEN")) { + client.connect(); + ResultCollector first = new ResultCollector(); + client.execute("SELECT 1", first); + Assert.assertNull(first.error.get()); + + // A dies; B rejects the credential: the failover reconnect fails, naming the class + b.setAuthorizationValidator(h -> 401); + a.close(); + ResultCollector failed = new ResultCollector(); + client.execute("SELECT 1", failed); + Assert.assertNotNull("the operation must fail", failed.error.get()); + Assert.assertTrue(failed.error.get(), failed.error.get().contains("auth-rejected")); + Assert.assertFalse(client.isConnected()); + + // B accepts again: the next operation reconnects instead of failing forever + b.setAuthorizationValidator(null); + ResultCollector recovered = new ResultCollector(); + client.execute("SELECT 1", recovered); + Assert.assertNull("the next execute() must reconnect: " + recovered.error.get(), recovered.error.get()); + Assert.assertTrue(recovered.done.get() > 0); + Assert.assertTrue(client.isConnected()); + } + }); + } + + @Test(timeout = 30_000) + public void testFailoverCredentialUnavailableNamesTheClass() throws Exception { + assertMemoryLeak(() -> { + AtomicInteger pulls = new AtomicInteger(); + try (TestWebSocketServer a = startServer(); + TestWebSocketServer b = startServer(); + QwpQueryClient client = QwpQueryClient.fromConfig( + "ws::addr=localhost:" + a.getPort() + ",localhost:" + b.getPort() + + ";failover_backoff_initial_ms=0;") + .withBearerTokenProvider(() -> { + if (pulls.incrementAndGet() > 1) { + throw TokenUnavailableException.retryable("IdP unreachable"); + } + return "TOKEN"; + })) { + client.connect(); + a.close(); + ResultCollector failed = new ResultCollector(); + client.execute("SELECT 1", failed); + Assert.assertNotNull(failed.error.get()); + Assert.assertTrue(failed.error.get(), failed.error.get().contains("credential-unavailable: IdP unreachable")); + Assert.assertEquals("no endpoint contacted on the failed failover", 0, b.upgradeRequestCount()); + } + }); + } + + @Test(timeout = 30_000) + public void testFailoverReconnectRetriesAStaleTokenOnce() throws Exception { + // C10 on the egress failover path: B rejects the stale token; the failover reconnect refreshes and + // retries B once, and the query completes. + assertMemoryLeak(() -> { + ScriptedSource source = twoTokens(); + try (TestWebSocketServer a = startServer(); + TestWebSocketServer b = startServer(); + RefreshingTokenProvider provider = RefreshingTokenProvider.builder(source).build(); + QwpQueryClient client = QwpQueryClient.fromConfig( + "ws::addr=localhost:" + a.getPort() + ",localhost:" + b.getPort() + + ";failover_backoff_initial_ms=0;") + .withBearerTokenProvider(provider)) { + b.setAuthorizationValidator(h -> "Bearer T1".equals(h) ? 401 : 0); + client.connect(); + Assert.assertEquals("Bearer T1", a.pollAuthorizationHeader(5, TimeUnit.SECONDS)); + a.close(); + ResultCollector result = new ResultCollector(); + client.execute("SELECT 1", result); + Assert.assertNull(result.error.get()); + Assert.assertEquals(Arrays.asList("Bearer T1", "Bearer T2"), headers(b)); + } + }); + } + + @Test(timeout = 60_000) + public void testPooledQueryClientRecoversAfterAFailedFailover() throws Exception { + // The suspected dead-pooled-worker bug (design/entra-id-qwp-auth.md, section 3): a pooled query client + // whose failover reconnect failed went back to the pool still disconnected, and with query_pool_min + // keeping it alive every later borrow failed with "QwpQueryClient not connected". + assertMemoryLeak(() -> { + try (TestWebSocketServer a = startServer(); + TestWebSocketServer b = startServer()) { + String cfg = "ws::addr=localhost:" + a.getPort() + ",localhost:" + b.getPort() + + ";sender_pool_min=0;query_pool_min=1;query_pool_max=1;failover_backoff_initial_ms=0;"; + try (QuestDB db = QuestDB.connect(cfg, () -> "TOKEN")) { + try (Query q = db.borrowQuery()) { + q.sql("SELECT 1").handler(new ResultCollector()).submit().await(); + } + b.setAuthorizationValidator(h -> 401); + a.close(); + try (Query q = db.borrowQuery()) { + q.sql("SELECT 1").handler(new ResultCollector()).submit().await(); + Assert.fail("the failover reconnect must fail"); + } catch (QueryException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("auth-rejected")); + } + b.setAuthorizationValidator(null); + try (Query q = db.borrowQuery()) { + q.sql("SELECT 1").handler(new ResultCollector()).submit().await(); + } catch (QueryException e) { + Assert.fail("the pooled client must reconnect on its next operation: " + e.getMessage()); + } + } + } + }); + } + + private static byte[] buildExecDone(byte[] queryRequest) { + int bodyLen = 1 + 8 + 1 + 1; // msg_kind + request_id + op_type + rows_affected varint + byte[] frame = new byte[QwpConstants.HEADER_SIZE + bodyLen]; + ByteBuffer bb = ByteBuffer.wrap(frame).order(ByteOrder.LITTLE_ENDIAN); + bb.put((byte) 'Q').put((byte) 'W').put((byte) 'P').put((byte) '1'); + bb.put((byte) 1); // version + bb.put((byte) 0); // flags + bb.putShort((short) 0); // table_count + bb.putInt(bodyLen); // payload_length + bb.put(QwpEgressMsgKind.EXEC_DONE); + bb.put(queryRequest, 1, 8); // echo request_id verbatim + bb.put((byte) 0); // op_type + bb.put((byte) 0); // rows_affected = 0 + return frame; + } + + private static List headers(TestWebSocketServer server) throws InterruptedException { + List headers = new ArrayList<>(); + String h; + while ((h = server.pollAuthorizationHeader(200, TimeUnit.MILLISECONDS)) != null) { + headers.add(h); + } + return headers; + } + + private static TestWebSocketServer startServer() throws IOException, InterruptedException { + TestWebSocketServer server = new TestWebSocketServer(new ExecDoneQueryServer()); + server.setSendServerInfo(true); + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + return server; + } + + private static ScriptedSource twoTokens() { + return new ScriptedSource() + .thenToken("T1", System.currentTimeMillis() + HOUR) + .thenToken("T2", System.currentTimeMillis() + HOUR); + } + + private static final class ExecDoneQueryServer implements TestWebSocketServer.WebSocketServerHandler { + @Override + public void onBinaryMessage(TestWebSocketServer.ClientHandler client, byte[] data) { + if (data.length == 0 || data[0] != QwpEgressMsgKind.QUERY_REQUEST) { + return; + } + try { + client.sendBinary(buildExecDone(data)); + } catch (IOException e) { + // best-effort: a failed reply surfaces to the client as a transport error + } + } + } + + private static final class ResultCollector implements QwpColumnBatchHandler { + final AtomicInteger done = new AtomicInteger(); + final AtomicReference error = new AtomicReference<>(); + + @Override + public void onBatch(QwpColumnBatch batch) { + } + + @Override + public void onEnd(long totalRows) { + done.incrementAndGet(); + } + + @Override + public void onError(byte status, String message) { + error.set(message); + } + + @Override + public void onExecDone(short opType, long rowsAffected) { + done.incrementAndGet(); + } + } +} diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/QwpWebSocketSenderJvmErrorCleanupTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/QwpWebSocketSenderJvmErrorCleanupTest.java index d0eb80db0..4d463fa7e 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/QwpWebSocketSenderJvmErrorCleanupTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/QwpWebSocketSenderJvmErrorCleanupTest.java @@ -239,13 +239,14 @@ public void testExceptionPathStillClosesAndWalksAllEndpoints() throws Exception * needed. The connect walk dereferences only the fields wired below plus * primitives whose zero-defaults are valid here (field initializers do * not run under {@code Unsafe.allocateInstance}), plus the connect-walk - * lock, which buildAndConnect acquires unconditionally and is therefore - * wired here. + * lock and the connection-health tracker, which buildAndConnect uses + * unconditionally on a foreground walk and are therefore wired here. */ private static QwpWebSocketSender newBareSender() throws Exception { QwpWebSocketSender sender = (QwpWebSocketSender) Unsafe.getUnsafe() .allocateInstance(QwpWebSocketSender.class); setField(sender, "connectWalkLock", new java.util.concurrent.locks.ReentrantLock()); + setField(sender, "healthTracker", new io.questdb.client.cutlass.qwp.client.QwpConnectionHealthTracker()); return sender; } diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/TokenProviderSharingTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/TokenProviderSharingTest.java new file mode 100644 index 000000000..6f1c897ef --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/TokenProviderSharingTest.java @@ -0,0 +1,161 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test.cutlass.qwp.client; + +import io.questdb.client.QuestDB; +import io.questdb.client.Sender; +import io.questdb.client.cutlass.auth.ExpiringToken; +import io.questdb.client.cutlass.auth.TokenProviderRegistry; +import io.questdb.client.cutlass.auth.TokenProviderSpec; +import io.questdb.client.cutlass.qwp.client.QwpQueryClient; +import io.questdb.client.impl.ConfigString; +import io.questdb.client.impl.ConfigView; +import io.questdb.client.test.cutlass.auth.TestTokenProviderFactory; +import io.questdb.client.test.cutlass.qwp.websocket.TestWebSocketServer; +import org.junit.After; +import org.junit.Assert; +import org.junit.Test; + +import java.util.ArrayList; +import java.util.List; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicInteger; + +import static io.questdb.client.test.tools.TestUtils.assertMemoryLeak; + +/** + * Conformance test C17 of the dynamic-credential specification (design/qwp-token-provider-spec.md, sections 7.4 + * and 10): senders and query clients built from one {@code token_provider} connect string share one provider, + * with one fetch per refresh, over real TLS connections. + */ +public class TokenProviderSharingTest { + private static final long HOUR = 3_600_000L; + + @After + public void tearDown() { + TestTokenProviderFactory.uninstall(); + } + + @Test(timeout = 60_000) + public void testFacadePoolsShareOneProvider() throws Exception { + assertMemoryLeak(() -> { + TestTokenProviderFactory azure = new TestTokenProviderFactory("azure"); + TestTokenProviderFactory.install(azure); + try (TestWebSocketServer server = startTlsServer()) { + String cfg = config(server, "c17-facade") + "sender_pool_min=2;query_pool_min=2;"; + TokenProviderSpec spec = spec(cfg); + try (QuestDB ignored = QuestDB.connect(cfg)) { + Assert.assertEquals("one provider for the whole facade", 1, azure.created.get()); + Assert.assertEquals("one fetch for every pooled connection", 1, azure.fetches.get()); + Assert.assertEquals("a lease per pooled connection", 4, TokenProviderRegistry.global().leaseCount(spec)); + for (int i = 0; i < 4; i++) { + Assert.assertEquals("Bearer TOKEN-azure", server.pollAuthorizationHeader(5, TimeUnit.SECONDS)); + } + } + Assert.assertEquals("every pooled connection released its lease", + 0, TokenProviderRegistry.global().leaseCount(spec)); + Assert.assertTrue("the provider lingers for the next client", TokenProviderRegistry.global().isActive(spec)); + } + }); + } + + @Test(timeout = 60_000) + public void testSendersAndQueryClientsShareOneProviderWithOneFetchPerRefresh() throws Exception { + assertMemoryLeak(() -> { + AtomicInteger issued = new AtomicInteger(); + TestTokenProviderFactory azure = new TestTokenProviderFactory("azure", + p -> new ExpiringToken("SHARED-" + issued.incrementAndGet(), System.currentTimeMillis() + HOUR)); + TestTokenProviderFactory.install(azure); + try (TestWebSocketServer server = startTlsServer()) { + String cfg = config(server, "c17-clients"); + TokenProviderSpec spec = spec(cfg); + List senders = new ArrayList<>(); + List queries = new ArrayList<>(); + try { + for (int i = 0; i < 3; i++) { + senders.add(Sender.fromConfig(cfg + "sender_id=s" + i + ';')); + } + for (int i = 0; i < 2; i++) { + QwpQueryClient client = QwpQueryClient.fromConfig(cfg); + queries.add(client); + client.connect(); + } + Assert.assertEquals("one provider for every client", 1, azure.created.get()); + Assert.assertEquals("one fetch for all five connections", 1, azure.fetches.get()); + Assert.assertEquals(5, TokenProviderRegistry.global().leaseCount(spec)); + for (int i = 0; i < 5; i++) { + Assert.assertEquals("Bearer SHARED-1", server.pollAuthorizationHeader(5, TimeUnit.SECONDS)); + } + + // One refresh serves everyone: force one through the shared provider, then drop every + // connection. The senders reconnect with the new token without fetching again. + try (TokenProviderRegistry.Lease probe = TokenProviderRegistry.global().acquire(spec)) { + probe.provider().onTokenRejected("SHARED-1", 401); + Assert.assertEquals("SHARED-2", probe.provider().getToken().toString()); + } + Assert.assertEquals(2, azure.fetches.get()); + server.dropAllConnections(); + for (int i = 0; i < 3; i++) { + Assert.assertEquals("Bearer SHARED-2", server.pollAuthorizationHeader(10, TimeUnit.SECONDS)); + } + // the reconnects completed: every sender delivers on its new connection + for (Sender s : senders) { + s.table("t").longColumn("v", 1).atNow(); + s.flush(); + Assert.assertTrue("the sender must deliver after reconnecting", s.awaitAckedFsn(0, 10_000)); + } + Assert.assertEquals("reconnects reuse the cached token", 2, azure.fetches.get()); + } finally { + for (Sender s : senders) { + s.close(); + } + for (QwpQueryClient q : queries) { + q.close(); + } + } + Assert.assertEquals(0, TokenProviderRegistry.global().leaseCount(spec)); + Assert.assertTrue(TokenProviderRegistry.global().isActive(spec)); + } + }); + } + + private static String config(TestWebSocketServer server, String identity) { + // a resource unique to the test keeps the process-wide registry entry from being shared across tests + return "wss::addr=localhost:" + server.getPort() + ";tls_verify=unsafe_off;token_provider=azure;" + + "azure_resource=api://" + identity + '-' + System.nanoTime() + ';'; + } + + private static TokenProviderSpec spec(String cfg) { + return TokenProviderSpec.parse(new ConfigView(ConfigString.parse(cfg)), true); + } + + private static TestWebSocketServer startTlsServer() throws Exception { + TestWebSocketServer server = TestWebSocketServer.tls(new WebSocketDynamicCredentialTest.AckHandler()); + server.setSendServerInfo(true); + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + return server; + } +} diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/WebSocketDynamicCredentialTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/WebSocketDynamicCredentialTest.java new file mode 100644 index 000000000..cabf0e0db --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/WebSocketDynamicCredentialTest.java @@ -0,0 +1,968 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test.cutlass.qwp.client; + +import io.questdb.client.Sender; +import io.questdb.client.SenderConnectionEvent; +import io.questdb.client.SenderError; +import io.questdb.client.cutlass.auth.ExpiringToken; +import io.questdb.client.cutlass.auth.RefreshingTokenProvider; +import io.questdb.client.cutlass.auth.TokenUnavailableException; +import io.questdb.client.cutlass.qwp.client.QwpAuthFailedException; +import io.questdb.client.cutlass.qwp.client.sf.cursor.OrphanScanner; +import io.questdb.client.test.cutlass.auth.TokenTestKit.FakeClock; +import io.questdb.client.test.cutlass.auth.TokenTestKit.ManualScheduler; +import io.questdb.client.test.cutlass.auth.TokenTestKit.ScriptedSource; +import io.questdb.client.test.cutlass.qwp.websocket.TestWebSocketServer; +import org.junit.Assert; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.TemporaryFolder; + +import java.io.IOException; +import java.nio.ByteBuffer; +import java.nio.ByteOrder; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.HashSet; +import java.util.List; +import java.util.Set; +import java.util.concurrent.CopyOnWriteArrayList; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.atomic.AtomicLong; + +import static io.questdb.client.test.cutlass.auth.TokenTestKit.await; +import static io.questdb.client.test.tools.TestUtils.assertMemoryLeak; + +/** + * Conformance tests C9-C16 and C23 of the dynamic-credential specification (design/qwp-token-provider-spec.md, + * section 10) for the WebSocket ingest {@code Sender}: a {@link RefreshingTokenProvider} wired into real + * connect, reconnect, failover, store-and-forward and orphan-drain paths, against a server that validates the + * presented token rather than rejecting blindly. + */ +public class WebSocketDynamicCredentialTest { + private static final long HOUR = 3_600_000L; + private static final long T0 = 1_800_000_000_000L; + + @Rule + public final TemporaryFolder temp = TemporaryFolder.builder().assureDeletion().build(); + + @Test(timeout = 30_000) + public void testAsyncInitializationRetriesProviderFailuresOfEitherClassification() throws Exception { + // C12, async rows: a credential-unavailable failure during async initialization is retried indefinitely + // whatever its classification, reported to the error handler, and recovers with the source. + for (boolean retryable : new boolean[]{true, false}) { + assertMemoryLeak(() -> { + ScriptedSource source = new ScriptedSource() + .thenThrow(new TokenUnavailableException("IdP says no", retryable)); + ErrorCollector errors = new ErrorCollector(); + AckHandler handler = new AckHandler(); + try (TestWebSocketServer server = startServer(handler); + RefreshingTokenProvider provider = fastProvider(source).build(); + Sender sender = Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + server.getPort()) + .initialConnectMode(Sender.InitialConnectMode.ASYNC) + .reconnectInitialBackoffMillis(10) + .reconnectMaxBackoffMillis(20) + .errorHandler(errors) + .httpTokenProvider(provider) + .build()) { + sender.table("t").longColumn("v", 1).atNow(); + sender.flush(); + await(() -> errors.count("credential-unavailable") >= 2, 10_000, + "the async initial connect to report and retry the provider failure"); + Assert.assertEquals("retried, never terminal [retryable=" + retryable + ']', + 0, errors.terminalCount()); + source.then(() -> new ExpiringToken("RECOVERED", System.currentTimeMillis() + HOUR)); + await(() -> handler.frames.get() >= 1, 10_000, "the row to arrive once the source recovers"); + Assert.assertEquals("Bearer RECOVERED", lastHeader(server)); + } + }); + } + } + + @Test(timeout = 30_000) + public void testChallengeInsufficientScopeDoesNotRetry() throws Exception { + // C23: a 401 whose Bearer challenge carries an error other than invalid_token does not trigger the retry. + assertMemoryLeak(() -> { + ScriptedSource source = new ScriptedSource() + .thenToken("T1", System.currentTimeMillis() + HOUR) + .thenToken("T2", System.currentTimeMillis() + HOUR); + try (TestWebSocketServer server = startServer(new AckHandler()); + RefreshingTokenProvider provider = fastProvider(source).build()) { + server.setAuthorizationValidator(h -> "Bearer T1".equals(h) ? 401 : 0); + server.setRejectWwwAuthenticate("Bearer realm=\"questdb\", error=\"insufficient_scope\""); + try { + Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + server.getPort()) + .httpTokenProvider(provider) + .build() + .close(); + Assert.fail("a 401 with error=insufficient_scope must not be retried"); + } catch (QwpAuthFailedException e) { + Assert.assertEquals(401, e.getStatusCode()); + Assert.assertEquals("insufficient_scope", e.getBearerError()); + Assert.assertTrue(e.getMessage(), e.getMessage().startsWith("auth-rejected")); + } + Assert.assertEquals("exactly one upgrade", 1, server.upgradeRequestCount()); + Assert.assertEquals("no forced refresh", 1, source.calls()); + } + }); + } + + @Test(timeout = 30_000) + public void testChallengeInvalidTokenRetries() throws Exception { + // C23: a 401 with Bearer error="invalid_token" triggers the retry. + assertChallengeRetries("Bearer realm=\"questdb\", error=\"invalid_token\", error_description=\"expired\""); + } + + @Test(timeout = 30_000) + public void testChallengeOtherSchemeOnlyRetries() throws Exception { + // C23: without a Bearer challenge carrying an error, the status code alone decides. + assertChallengeRetries("Basic realm=\"questdb\""); + } + + @Test(timeout = 30_000) + public void testChallengePlain401Retries() throws Exception { + // C23: a plain 401, with no challenge at all, triggers the retry. + assertChallengeRetries(null); + } + + @Test(timeout = 60_000) + public void testFailoverAcrossARotationSendsTheNewTokenAndReplaysOnce() throws Exception { + // C14: connected to A with T1, the token rotates, A dies. B receives the NEW token, and the frames A + // never acknowledged are replayed to B exactly once. + assertMemoryLeak(() -> { + ScriptedSource source = new ScriptedSource() + .thenToken("T1", System.currentTimeMillis() + HOUR) + .thenToken("T2", System.currentTimeMillis() + HOUR); + CollectingHandler silentA = new CollectingHandler(false); + CollectingHandler ackB = new CollectingHandler(true); + try (RefreshingTokenProvider provider = fastProvider(source).forcedMinIntervalMillis(0).build(); + TestWebSocketServer b = startServer(ackB)) { + b.setAuthorizationValidator(h -> "Bearer T2".equals(h) ? 0 : 401); + TestWebSocketServer a = startServer(silentA); + try (Sender sender = Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + a.getPort()) + .address("localhost:" + b.getPort()) + .httpTokenProvider(provider) + .build()) { + Assert.assertEquals("Bearer T1", a.pollAuthorizationHeader(5, TimeUnit.SECONDS)); + for (int i = 0; i < 5; i++) { + sender.table("t").longColumn("v", i).atNow(); + sender.flush(); + } + await(() -> silentA.frames.get() == 5, 10_000, "A to receive the unacknowledged frames"); + + // rotate: the provider now holds T2 + provider.onTokenRejected("T1", 401); + Assert.assertEquals("T2", provider.getToken().toString()); + + a.close(); // A dies; the reconnect fails over to B + await(() -> ackB.distinctPayloads.size() == 5, 15_000, "the unacked frames to replay at B"); + Assert.assertEquals("B must receive the new token", "Bearer T2", + b.pollAuthorizationHeader(5, TimeUnit.SECONDS)); + Assert.assertEquals("B must never see the old token", 0, b.authRejectCount()); + Assert.assertEquals("each unacked frame replayed exactly once", 5, ackB.frames.get()); + } finally { + a.close(); + } + Assert.assertEquals(silentA.payloads, ackB.payloads); + } + }); + } + + @Test(timeout = 30_000) + public void testForbiddenNeverTriggersAForcedRefresh() throws Exception { + // D2: a 403 is an authorization decision; it gets no forced refresh and no retry. + assertMemoryLeak(() -> { + ScriptedSource source = new ScriptedSource() + .thenToken("T1", System.currentTimeMillis() + HOUR) + .thenToken("T2", System.currentTimeMillis() + HOUR); + try (TestWebSocketServer server = startServer(new AckHandler()); + RefreshingTokenProvider provider = fastProvider(source).build()) { + server.setAuthorizationValidator(h -> "Bearer T1".equals(h) ? 403 : 0); + try { + Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + server.getPort()) + .httpTokenProvider(provider) + .build() + .close(); + Assert.fail("a 403 must fail startup"); + } catch (QwpAuthFailedException e) { + Assert.assertEquals(403, e.getStatusCode()); + } + Assert.assertEquals(1, server.upgradeRequestCount()); + Assert.assertEquals("a 403 must not force a refresh", 1, source.calls()); + } + }); + } + + @Test(timeout = 60_000) + public void testOrphanDrainAcrossARotationDrainsWithTheNewToken() throws Exception { + // C16: an orphan drain whose first upgrade presents the stale token gets one 401, refreshes, and drains + // the whole slot with the new token. No quarantine. + assertMemoryLeak(() -> { + String sfDir = temp.newFolder("orphan-rotation").getAbsolutePath(); + final int frames = 20; + try (TestWebSocketServer silent = startServer(new CollectingHandler(false))) { + String ghostCfg = "ws::addr=localhost:" + silent.getPort() + ";sf_dir=" + sfDir + + ";sender_id=ghost;close_flush_timeout_millis=0;"; + try (Sender ghost = Sender.fromConfig(ghostCfg)) { + for (int i = 0; i < frames; i++) { + ghost.table("t").longColumn("v", i).atNow(); + ghost.flush(); + } + } + } + Assert.assertEquals(1, OrphanScanner.scan(sfDir, "primary").size()); + + ScriptedSource source = new ScriptedSource() + .thenToken("T1", System.currentTimeMillis() + HOUR) + .thenToken("T2", System.currentTimeMillis() + HOUR); + CollectingHandler ack = new CollectingHandler(true); + AtomicInteger t1Uses = new AtomicInteger(); + try (RefreshingTokenProvider provider = fastProvider(source).forcedMinIntervalMillis(0).build(); + TestWebSocketServer server = startServer(ack)) { + // T1 is good for exactly one upgrade - the foreground's - and is revoked after it. The drainer's + // first upgrade presents the cached T1 and must recover through the one-retry rule. + server.setAuthorizationValidator(h -> { + if ("Bearer T2".equals(h)) { + return 0; + } + return "Bearer T1".equals(h) && t1Uses.incrementAndGet() == 1 ? 0 : 401; + }); + try (Sender ignored = Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + server.getPort()) + .storeAndForwardDir(sfDir) + .senderId("primary") + .drainOrphans(true) + .httpTokenProvider(provider) + .build()) { + await(() -> ack.distinctPayloads.size() == frames, 15_000, "the orphan slot to drain"); + } + Assert.assertEquals("the drainer met the stale token exactly once", 1, server.authRejectCount()); + Assert.assertEquals("the drain used the new token", 2, source.calls()); + } + Assert.assertFalse("a slot that drained must not be quarantined", + Files.exists(new java.io.File(sfDir, "ghost/" + OrphanScanner.FAILED_SENTINEL_NAME).toPath())); + }); + } + + @Test(timeout = 30_000) + public void testPersistent401AfterTheRetryFailsStartup() throws Exception { + // C11, initialization rows: a 401 that survives the one retry fails startup, in OFF and in SYNC mode + // (SYNC does not spend its reconnect budget on it). + for (Sender.InitialConnectMode mode : new Sender.InitialConnectMode[]{ + Sender.InitialConnectMode.OFF, Sender.InitialConnectMode.SYNC}) { + assertMemoryLeak(() -> { + ScriptedSource source = new ScriptedSource() + .thenToken("T1", System.currentTimeMillis() + HOUR) + .thenToken("T2", System.currentTimeMillis() + HOUR); + try (TestWebSocketServer server = startServer(new AckHandler()); + RefreshingTokenProvider provider = fastProvider(source).build()) { + server.setAuthorizationValidator(h -> 401); + long start = System.nanoTime(); + try { + Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + server.getPort()) + .initialConnectMode(mode) + .reconnectMaxDurationMillis(20_000) + .httpTokenProvider(provider) + .build() + .close(); + Assert.fail("a persistent 401 must fail startup [mode=" + mode + ']'); + } catch (QwpAuthFailedException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().startsWith("auth-rejected")); + } + long elapsedMillis = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - start); + Assert.assertTrue("startup must fail fast, took " + elapsedMillis + " ms", elapsedMillis < 10_000); + Assert.assertEquals("the original attempt plus one retry with the refreshed token [mode=" + mode + ']', + 2, server.authRejectCount()); + Assert.assertEquals(Arrays.asList("Bearer T1", "Bearer T2"), drainHeaders(server)); + } + }); + } + } + + @Test(timeout = 60_000) + public void testPersistent401WhileEstablishedIsRetriedAndReported() throws Exception { + // C11, established row: a 401 that survives the retry is retried indefinitely and reported as a + // retriable security error that names the failure class (decision D1). + assertMemoryLeak(() -> { + ScriptedSource source = new ScriptedSource().then( + () -> new ExpiringToken("T-" + System.nanoTime(), System.currentTimeMillis() + HOUR)); + ErrorCollector errors = new ErrorCollector(); + DropAfterFirstAckHandler handler = new DropAfterFirstAckHandler(); + AtomicInteger accepted = new AtomicInteger(); + try (TestWebSocketServer server = startServer(handler); + RefreshingTokenProvider provider = fastProvider(source).forcedMinIntervalMillis(0).build()) { + server.setAuthorizationValidator(h -> accepted.getAndIncrement() == 0 ? 0 : 401); + try (Sender sender = Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + server.getPort()) + .reconnectInitialBackoffMillis(10) + .reconnectMaxBackoffMillis(20) + .errorHandler(errors) + .httpTokenProvider(provider) + .build()) { + sender.table("t").longColumn("v", 1).atNow(); + sender.flush(); + await(() -> errors.count("auth-rejected") >= 3, 15_000, "repeated auth-rejected reports"); + Assert.assertEquals("an established sender never goes terminal on a 401", 0, errors.terminalCount()); + for (SenderError e : errors.errors) { + if (e.getServerMessage().contains("auth-rejected")) { + Assert.assertEquals(SenderError.Category.SECURITY_ERROR, e.getCategory()); + Assert.assertEquals(SenderError.Policy.RETRIABLE, e.getAppliedPolicy()); + } + } + server.setAuthorizationValidator(null); + sender.table("t").longColumn("v", 2).atNow(); + sender.flush(); + await(() -> handler.frames.get() >= 2, 15_000, "recovery once the server accepts again"); + } + } + }); + } + + @Test(timeout = 60_000) + public void testProviderOutageWhileEstablishedIsRetriedReportedAndRecovers() throws Exception { + // C13: the source fails while a connected sender's token expires. The reconnect cannot get a credential; + // it is retried indefinitely and reported, and recovers when the source does. + assertMemoryLeak(() -> { + SkewedClock clock = new SkewedClock(); + ScriptedSource source = new ScriptedSource().thenToken("T1", System.currentTimeMillis() + HOUR); + ErrorCollector errors = new ErrorCollector(); + DropAfterFirstAckHandler handler = new DropAfterFirstAckHandler(); + try (TestWebSocketServer server = startServer(handler); + RefreshingTokenProvider provider = fastProvider(source).clock(clock).build(); + Sender sender = Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + server.getPort()) + .reconnectInitialBackoffMillis(10) + .reconnectMaxBackoffMillis(20) + .errorHandler(errors) + .httpTokenProvider(provider) + .build()) { + Assert.assertEquals("Bearer T1", server.pollAuthorizationHeader(5, TimeUnit.SECONDS)); + // the IdP goes down, and the held token expires (the clock jumps past it) + source.thenThrow(TokenUnavailableException.retryable("IdP unreachable")); + clock.skewMillis = 2 * HOUR; + sender.table("t").longColumn("v", 1).atNow(); + sender.flush(); // ACKed on the live connection, which the server then drops + await(() -> errors.count("credential-unavailable") >= 3, 15_000, + "repeated credential-unavailable reports"); + Assert.assertEquals(0, errors.terminalCount()); + for (SenderError e : errors.errors) { + if (e.getServerMessage().contains("credential-unavailable")) { + Assert.assertEquals(SenderError.Policy.RETRIABLE, e.getAppliedPolicy()); + Assert.assertTrue(e.getServerMessage(), e.getServerMessage().contains("IdP unreachable")); + } + } + sender.table("t").longColumn("v", 2).atNow(); + sender.flush(); // buffered in store-and-forward meanwhile + source.then(() -> new ExpiringToken("T2", clock.wallClockMillis() + HOUR)); + await(() -> handler.frames.get() >= 2, 15_000, "the buffered row to arrive after recovery"); + Assert.assertEquals("Bearer T2", lastHeader(server)); + } + }); + } + + @Test(timeout = 30_000) + public void testReconnectAfterExpiryCarriesTheRefreshedToken() throws Exception { + // C9: the server rejects expired tokens against a clock shared with the provider. The proactive refresh + // rotated the token before the old one expired, so the reconnect carries the new one and sees no 401. + assertMemoryLeak(() -> { + FakeClock clock = new FakeClock(T0); + ManualScheduler scheduler = new ManualScheduler(clock); + ScriptedSource source = new ScriptedSource() + .thenToken("T1." + (T0 + HOUR), T0 + HOUR) + .thenToken("T2." + (T0 + 2 * HOUR), T0 + 2 * HOUR); + DropAfterFirstAckHandler handler = new DropAfterFirstAckHandler(); + try (TestWebSocketServer server = startServer(handler); + RefreshingTokenProvider provider = RefreshingTokenProvider.builder(source) + .clock(clock).scheduler(scheduler).random(() -> 0.5).build()) { + scheduler.runNext(); // the prefetch: T1 + server.setAuthorizationValidator(h -> expiredAt(h) <= clock.wallClockMillis() ? 401 : 0); + try (Sender sender = Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + server.getPort()) + .httpTokenProvider(provider) + .build()) { + Assert.assertTrue(server.pollAuthorizationHeader(5, TimeUnit.SECONDS).startsWith("Bearer T1.")); + scheduler.advanceAndRunNext(); // the proactive refresh at the half-life: T2 + clock.advanceMillis(HOUR / 2 + 60_000); // T1 is now expired at the server + sender.table("t").longColumn("v", 1).atNow(); + sender.flush(); // ACKed, then the server drops the connection + String reconnect = server.pollAuthorizationHeader(5, TimeUnit.SECONDS); + Assert.assertNotNull("the sender must reconnect", reconnect); + Assert.assertTrue(reconnect, reconnect.startsWith("Bearer T2.")); + sender.table("t").longColumn("v", 2).atNow(); + sender.flush(); + await(() -> handler.frames.get() >= 2, 10_000, "the second row"); + } + Assert.assertEquals("no 401 may be observed", 0, server.authRejectCount()); + } + }); + } + + @Test(timeout = 30_000) + public void testStaleTokenGetsExactlyOne401ThenAnImmediateRetry() throws Exception { + // C10: the reconnect presents a token the server has revoked. Exactly one 401, then an immediate retry of + // the same endpoint with the refreshed token: no backoff, no failed-attempt event, no AUTH_FAILED. + assertMemoryLeak(() -> { + ScriptedSource source = new ScriptedSource() + .thenToken("T1", System.currentTimeMillis() + HOUR) + .thenToken("T2", System.currentTimeMillis() + HOUR); + DropAfterFirstAckHandler handler = new DropAfterFirstAckHandler(); + List events = new CopyOnWriteArrayList<>(); + Set revoked = Collections.synchronizedSet(new HashSet<>()); + try (TestWebSocketServer server = startServer(handler); + RefreshingTokenProvider provider = fastProvider(source).build()) { + server.setAuthorizationValidator(h -> revoked.contains(h) ? 401 : 0); + try (Sender sender = Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + server.getPort()) + .initialConnectMode(Sender.InitialConnectMode.OFF) + // a retry through the reconnect loop would wait at least this long + .reconnectInitialBackoffMillis(5_000) + .reconnectMaxBackoffMillis(5_000) + .connectionListener(events::add) + .httpTokenProvider(provider) + .build()) { + Assert.assertEquals("Bearer T1", server.pollAuthorizationHeader(5, TimeUnit.SECONDS)); + revoked.add("Bearer T1"); + sender.table("t").longColumn("v", 1).atNow(); + sender.flush(); // ACKed, then the server drops the connection + Assert.assertEquals("the reconnect first presents the cached token", + "Bearer T1", server.pollAuthorizationHeader(5, TimeUnit.SECONDS)); + long rejectedAt = System.nanoTime(); + Assert.assertEquals("then retries with the refreshed one", + "Bearer T2", server.pollAuthorizationHeader(5, TimeUnit.SECONDS)); + long retryMillis = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - rejectedAt); + Assert.assertTrue("the retry must be immediate, not after backoff; took " + retryMillis + " ms", + retryMillis < 4_000); + sender.table("t").longColumn("v", 2).atNow(); + sender.flush(); + await(() -> handler.frames.get() >= 2, 10_000, "the second row"); + await(() -> kinds(events).contains(SenderConnectionEvent.Kind.RECONNECTED), 5_000, "RECONNECTED"); + } + Assert.assertEquals("exactly one 401", 1, server.authRejectCount()); + List kinds = kinds(events); + Assert.assertFalse("only the round's final outcome is reported: " + kinds, + kinds.contains(SenderConnectionEvent.Kind.AUTH_FAILED)); + Assert.assertFalse("the retried 401 is not an endpoint failure: " + kinds, + kinds.contains(SenderConnectionEvent.Kind.ENDPOINT_ATTEMPT_FAILED)); + } + }); + } + + @Test(timeout = 60_000) + public void testSentinelTokensNeverLeakFromTheClient() throws Exception { + // C20 at the client level: across a stale-token 401 and its retry (C10), a persistent 401 (C11), a cold + // provider failure (C4) and a malformed source result, the tokens never appear in any log line (every + // logger, every level), exception, SenderError, connection event, health snapshot or toString(). + final String stale = "SENTINEL-STALE-" + System.nanoTime(); + final String fresh = "SENTINEL-FRESH-" + System.nanoTime(); + ch.qos.logback.classic.Logger root = (ch.qos.logback.classic.Logger) + org.slf4j.LoggerFactory.getLogger(org.slf4j.Logger.ROOT_LOGGER_NAME); + ch.qos.logback.core.read.ListAppender appender = + new ch.qos.logback.core.read.ListAppender<>(); + appender.start(); + ch.qos.logback.classic.Level savedLevel = root.getLevel(); + root.setLevel(ch.qos.logback.classic.Level.ALL); + root.addAppender(appender); + List rendered = new CopyOnWriteArrayList<>(); + try { + assertMemoryLeak(() -> { + ErrorCollector errors = new ErrorCollector(); + List events = new CopyOnWriteArrayList<>(); + // C10: the stale token is rejected once, the fresh one is accepted + ScriptedSource source = new ScriptedSource() + .thenToken(stale, System.currentTimeMillis() + HOUR) + .thenToken(fresh, System.currentTimeMillis() + HOUR); + try (TestWebSocketServer server = startServer(new AckHandler()); + RefreshingTokenProvider provider = fastProvider(source).build()) { + server.setAuthorizationValidator(h -> h.contains(stale) ? 401 : 0); + try (Sender sender = Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + server.getPort()) + .errorHandler(errors) + .connectionListener(events::add) + .httpTokenProvider(provider) + .build()) { + sender.table("t").longColumn("v", 1).atNow(); + sender.flush(); + rendered.add(sender.health().toString()); + rendered.add(sender.toString()); + } + rendered.add(provider.toString()); + + // C11: every token is rejected + server.setAuthorizationValidator(h -> 401); + try { + Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + server.getPort()) + .httpTokenProvider(provider) + .build() + .close(); + Assert.fail(); + } catch (RuntimeException e) { + renderThrowable(e, rendered); + } + } + // C4 and a malformed result: the source fails, then returns an already-expired token + ScriptedSource failing = new ScriptedSource() + .thenThrow(TokenUnavailableException.retryable("IdP unreachable")) + .thenToken(stale, 1L); + try (TestWebSocketServer server = startServer(new AckHandler()); + RefreshingTokenProvider provider = fastProvider(failing).build()) { + try { + Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + server.getPort()) + .httpTokenProvider(provider) + .build() + .close(); + Assert.fail(); + } catch (RuntimeException e) { + renderThrowable(e, rendered); + } + rendered.add(provider.toString()); + rendered.add(String.valueOf(provider.getLastFailure())); + } + for (SenderError e : errors.errors) { + rendered.add(e.toString()); + rendered.add(String.valueOf(e.getServerMessage())); + } + for (SenderConnectionEvent e : events) { + rendered.add(e.toString()); + if (e.getCause() != null) { + renderThrowable(e.getCause(), rendered); + } + } + }); + } finally { + root.detachAppender(appender); + root.setLevel(savedLevel); + appender.stop(); + } + for (ch.qos.logback.classic.spi.ILoggingEvent event : appender.list) { + rendered.add(event.getFormattedMessage()); + for (ch.qos.logback.classic.spi.IThrowableProxy t = event.getThrowableProxy(); t != null; t = t.getCause()) { + rendered.add(t.getMessage()); + } + } + Assert.assertTrue("the scenarios must have produced output to inspect", rendered.size() > 20); + for (String s : rendered) { + if (s != null) { + Assert.assertFalse("a token leaked into: " + s, s.contains(stale) || s.contains(fresh)); + } + } + } + + @Test(timeout = 30_000) + public void testStaticCredentialNeverGetsTheRetry() throws Exception { + // section 8.2: a static credential never gets the retry - presenting the same bytes again cannot help. + assertMemoryLeak(() -> { + try (TestWebSocketServer server = startServer(new AckHandler())) { + server.setAuthorizationValidator(h -> 401); + try { + Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + server.getPort()) + .httpToken("static") + .build() + .close(); + Assert.fail(); + } catch (QwpAuthFailedException e) { + Assert.assertEquals(401, e.getStatusCode()); + } + Assert.assertEquals(1, server.upgradeRequestCount()); + } + }); + } + + @Test(timeout = 60_000) + public void testStoreAndForwardAcrossARotationDeliversEveryRow() throws Exception { + // C15: the server rejects T1 with 401 while the IdP still hands out T1; the producer keeps writing into + // store-and-forward. Once the IdP rotates, every row is delivered: no quarantine, no data loss, no terminal. + assertMemoryLeak(() -> { + String sfDir = temp.newFolder("sf-rotation").getAbsolutePath(); + ScriptedSource source = new ScriptedSource().thenToken("T1", System.currentTimeMillis() + HOUR); + ErrorCollector errors = new ErrorCollector(); + CollectingHandler ack = new CollectingHandler(true); + AtomicInteger connections = new AtomicInteger(); + try (TestWebSocketServer server = startServer(ack); + RefreshingTokenProvider provider = fastProvider(source).forcedMinIntervalMillis(0).build()) { + // T1 is accepted for the first connection only + server.setAuthorizationValidator(h -> "Bearer T1".equals(h) && connections.getAndIncrement() > 0 ? 401 : 0); + try (Sender sender = Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + server.getPort()) + .storeAndForwardDir(sfDir) + .senderId("rot") + .reconnectInitialBackoffMillis(10) + .reconnectMaxBackoffMillis(50) + .errorHandler(errors) + .httpTokenProvider(provider) + .build()) { + sender.table("t").longColumn("v", 0).atNow(); + sender.flush(); + await(() -> ack.distinctPayloads.size() >= 1, 10_000, "the first row"); + server.dropAllConnections(); + for (int i = 1; i < 20; i++) { + sender.table("t").longColumn("v", i).atNow(); + sender.flush(); + } + await(() -> errors.count("auth-rejected") >= 2, 15_000, "the 401 episode to be reported"); + source.thenToken("T2", System.currentTimeMillis() + HOUR); // the IdP rotates + await(() -> ack.distinctPayloads.size() >= 20, 15_000, "every row to arrive"); + Assert.assertTrue(sender.awaitAckedFsn(19, 10_000)); + } + Assert.assertEquals(0, errors.terminalCount()); + Assert.assertEquals("no data-loss report", 0, errors.count(SenderError.Category.DATA_LOSS)); + } + Assert.assertFalse("no quarantine", Files.exists(new java.io.File(sfDir, "rot/" + OrphanScanner.FAILED_SENTINEL_NAME).toPath())); + }); + } + + @Test(timeout = 30_000) + public void testSyncInitializationFailsFastOnAPermanentProviderFailure() throws Exception { + // C12 / D8: a permanent credential-unavailable failure fails SYNC startup fast with the provider's error. + assertMemoryLeak(() -> { + ScriptedSource source = new ScriptedSource() + .thenThrow(TokenUnavailableException.permanent("no managed identity is assigned")); + try (TestWebSocketServer server = startServer(new AckHandler()); + RefreshingTokenProvider provider = fastProvider(source).build()) { + long start = System.nanoTime(); + try { + Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + server.getPort()) + .initialConnectMode(Sender.InitialConnectMode.SYNC) + .reconnectMaxDurationMillis(20_000) + .httpTokenProvider(provider) + .build() + .close(); + Assert.fail(); + } catch (TokenUnavailableException e) { + Assert.assertFalse(e.isRetryable()); + Assert.assertTrue(e.getMessage(), e.getMessage().contains("no managed identity is assigned")); + } + long elapsedMillis = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - start); + Assert.assertTrue("must fail fast, took " + elapsedMillis + " ms", elapsedMillis < 10_000); + Assert.assertEquals("no endpoint may be contacted", 0, server.upgradeRequestCount()); + } + }); + } + + @Test(timeout = 30_000) + public void testSyncInitializationFailsFastOnAnUnclassifiedProviderException() throws Exception { + // D8: an exception from an application-supplied provider that is not a token-unavailable error is + // permanent, even when it says "retry". + assertMemoryLeak(() -> { + AtomicInteger pulls = new AtomicInteger(); + try (TestWebSocketServer server = startServer(new AckHandler())) { + try { + Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + server.getPort()) + .initialConnectMode(Sender.InitialConnectMode.SYNC) + .reconnectMaxDurationMillis(20_000) + .httpTokenProvider(() -> { + pulls.incrementAndGet(); + throw new IllegalStateException("temporarily unavailable, please retry"); + }) + .build() + .close(); + Assert.fail(); + } catch (IllegalStateException e) { + Assert.assertTrue(e.getMessage().contains("temporarily unavailable")); + } + Assert.assertEquals(1, pulls.get()); + } + }); + } + + @Test(timeout = 30_000) + public void testSyncInitializationRetriesARetryableProviderFailureWithinTheBudget() throws Exception { + // C12 / D6: a retryable credential-unavailable failure is retried within reconnect_max_duration_millis. + assertMemoryLeak(() -> { + AtomicInteger pulls = new AtomicInteger(); + try (TestWebSocketServer server = startServer(new AckHandler()); + Sender sender = Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + server.getPort()) + .initialConnectMode(Sender.InitialConnectMode.SYNC) + .reconnectMaxDurationMillis(20_000) + .reconnectInitialBackoffMillis(10) + .reconnectMaxBackoffMillis(20) + .httpTokenProvider(() -> { + if (pulls.incrementAndGet() < 4) { + throw TokenUnavailableException.retryable("IMDS returned HTTP 429", 0); + } + return "FINALLY"; + }) + .build()) { + Assert.assertEquals("Bearer FINALLY", server.pollAuthorizationHeader(5, TimeUnit.SECONDS)); + Assert.assertEquals(4, pulls.get()); + sender.table("t").longColumn("v", 1).atNow(); + sender.flush(); + } + }); + } + + @Test(timeout = 30_000) + public void testSyncInitializationRetryableProviderFailureExhaustsTheBudgetNamingTheClass() throws Exception { + assertMemoryLeak(() -> { + AtomicInteger pulls = new AtomicInteger(); + try (TestWebSocketServer server = startServer(new AckHandler())) { + try { + Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + server.getPort()) + .initialConnectMode(Sender.InitialConnectMode.SYNC) + .reconnectMaxDurationMillis(300) + .reconnectInitialBackoffMillis(10) + .reconnectMaxBackoffMillis(20) + .httpTokenProvider(() -> { + pulls.incrementAndGet(); + throw TokenUnavailableException.retryable("IdP timed out"); + }) + .build() + .close(); + Assert.fail(); + } catch (TokenUnavailableException e) { + Assert.fail("a retryable failure must consume the budget, not fail fast: " + e.getMessage()); + } catch (io.questdb.client.cutlass.line.LineSenderException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("credential-unavailable: IdP timed out")); + } + Assert.assertTrue("retried within the budget, pulls=" + pulls.get(), pulls.get() > 2); + Assert.assertEquals(0, server.upgradeRequestCount()); + } + }); + } + + private static void renderThrowable(Throwable t, List into) { + for (Throwable c = t; c != null; c = c.getCause()) { + into.add(c.toString()); + for (Throwable s : c.getSuppressed()) { + renderThrowable(s, into); + } + } + } + + private static List drainHeaders(TestWebSocketServer server) throws InterruptedException { + List headers = new ArrayList<>(); + String h; + while ((h = server.pollAuthorizationHeader(200, TimeUnit.MILLISECONDS)) != null) { + headers.add(h); + } + return headers; + } + + // Tokens shaped T., as the specification's conformance harness suggests. + private static long expiredAt(String header) { + int dot = header.lastIndexOf('.'); + return dot < 0 ? Long.MIN_VALUE : Long.parseLong(header.substring(dot + 1)); + } + + private static RefreshingTokenProvider.Builder fastProvider(ScriptedSource source) { + return RefreshingTokenProvider.builder(source) + .coldWaitMillis(200) + .backoffInitialMillis(10) + .backoffMaxMillis(20) + .forcedWaitMillis(5_000); + } + + private static List kinds(List events) { + List kinds = new ArrayList<>(); + for (SenderConnectionEvent e : events) { + kinds.add(e.getKind()); + } + return kinds; + } + + private static String lastHeader(TestWebSocketServer server) throws InterruptedException { + String last = null; + String h; + while ((h = server.pollAuthorizationHeader(100, TimeUnit.MILLISECONDS)) != null) { + last = h; + } + return last; + } + + private static TestWebSocketServer startServer(TestWebSocketServer.WebSocketServerHandler handler) + throws IOException, InterruptedException { + TestWebSocketServer server = new TestWebSocketServer(handler); + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + return server; + } + + private void assertChallengeRetries(String challenge) throws Exception { + assertMemoryLeak(() -> { + ScriptedSource source = new ScriptedSource() + .thenToken("T1", System.currentTimeMillis() + HOUR) + .thenToken("T2", System.currentTimeMillis() + HOUR); + try (TestWebSocketServer server = startServer(new AckHandler()); + RefreshingTokenProvider provider = fastProvider(source).build()) { + server.setAuthorizationValidator(h -> "Bearer T1".equals(h) ? 401 : 0); + server.setRejectWwwAuthenticate(challenge); + try (Sender sender = Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + server.getPort()) + .httpTokenProvider(provider) + .build()) { + sender.table("t").longColumn("v", 1).atNow(); + sender.flush(); + } + Assert.assertEquals("challenge=" + challenge, Arrays.asList("Bearer T1", "Bearer T2"), drainHeaders(server)); + Assert.assertEquals(1, server.authRejectCount()); + } + }); + } + + static byte[] buildAck(long seq) { + byte[] buf = new byte[1 + 8 + 2]; + ByteBuffer bb = ByteBuffer.wrap(buf).order(ByteOrder.LITTLE_ENDIAN); + bb.put((byte) 0x00); // STATUS_OK + bb.putLong(seq); + bb.putShort((short) 0); + return buf; + } + + /** + * ACKs every frame, per connection: wire sequences restart at 0 on each connection. + */ + static class AckHandler implements TestWebSocketServer.WebSocketServerHandler { + final AtomicLong frames = new AtomicLong(); + private final java.util.Map seqs = + Collections.synchronizedMap(new java.util.IdentityHashMap<>()); + + @Override + public void onBinaryMessage(TestWebSocketServer.ClientHandler client, byte[] data) { + frames.incrementAndGet(); + long seq = seqs.computeIfAbsent(client, c -> new AtomicLong()).getAndIncrement(); + try { + client.sendBinary(buildAck(seq)); + } catch (IOException e) { + throw new RuntimeException(e); + } + } + } + + /** + * Records every frame; ACKs them per connection when {@code ack} is set. + */ + static final class CollectingHandler extends AckHandler { + final Set distinctPayloads = Collections.synchronizedSet(new HashSet<>()); + final List payloads = new CopyOnWriteArrayList<>(); + private final boolean ack; + + CollectingHandler(boolean ack) { + this.ack = ack; + } + + @Override + public void onBinaryMessage(TestWebSocketServer.ClientHandler client, byte[] data) { + String p = Arrays.toString(data); + distinctPayloads.add(p); + payloads.add(p); + if (ack) { + super.onBinaryMessage(client, data); + } else { + frames.incrementAndGet(); + } + } + } + + /** + * ACKs every frame; closes the first connection right after its first ACK, forcing a reconnect. + */ + static final class DropAfterFirstAckHandler extends AckHandler { + private volatile boolean dropped; + + @Override + public void onBinaryMessage(TestWebSocketServer.ClientHandler client, byte[] data) { + super.onBinaryMessage(client, data); + if (!dropped) { + dropped = true; + try { + Thread.sleep(50); // let the ACK flush before the socket goes + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + client.close(); + } + } + } + + static final class ErrorCollector implements io.questdb.client.SenderErrorHandler { + final List errors = new CopyOnWriteArrayList<>(); + + int count(String messageFragment) { + int n = 0; + for (SenderError e : errors) { + if (e.getServerMessage() != null && e.getServerMessage().contains(messageFragment)) { + n++; + } + } + return n; + } + + int count(SenderError.Category category) { + int n = 0; + for (SenderError e : errors) { + if (e.getCategory() == category) { + n++; + } + } + return n; + } + + @Override + public void onError(SenderError error) { + errors.add(error); + } + + int terminalCount() { + int n = 0; + for (SenderError e : errors) { + if (e.getAppliedPolicy() == SenderError.Policy.TERMINAL) { + n++; + } + } + return n; + } + } + + // Real monotonic time, with a wall clock the test can push forward. + static final class SkewedClock implements RefreshingTokenProvider.Clock { + volatile long skewMillis; + + @Override + public long monotonicNanos() { + return System.nanoTime(); + } + + @Override + public long wallClockMillis() { + return System.currentTimeMillis() + skewMillis; + } + } +} diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/websocket/TestTls.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/websocket/TestTls.java new file mode 100644 index 000000000..d36fb79b4 --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/websocket/TestTls.java @@ -0,0 +1,112 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test.cutlass.qwp.websocket; + +import javax.net.ssl.KeyManagerFactory; +import javax.net.ssl.SSLContext; +import javax.net.ssl.SSLServerSocketFactory; +import java.io.File; +import java.io.FileInputStream; +import java.io.IOException; +import java.io.InputStream; +import java.nio.charset.StandardCharsets; +import java.security.KeyStore; +import java.util.concurrent.TimeUnit; + +/** + * A self-signed server identity for TLS test servers, generated once per JVM with the running JDK's + * {@code keytool} - no key material is committed to the repository. Clients connect with + * {@code tls_verify=unsafe_off}. + */ +public final class TestTls { + private static final char[] PASSWORD = "test-only".toCharArray(); + private static SSLServerSocketFactory factory; + + private TestTls() { + } + + public static synchronized SSLServerSocketFactory serverSocketFactory() throws IOException { + if (factory == null) { + factory = create(); + } + return factory; + } + + private static SSLServerSocketFactory create() throws IOException { + File keystore = File.createTempFile("qdb-test-server", ".p12"); + if (!keystore.delete()) { + throw new IOException("could not prepare " + keystore); + } + keystore.deleteOnExit(); + String keytool = System.getProperty("java.home") + File.separator + "bin" + File.separator + "keytool"; + ProcessBuilder pb = new ProcessBuilder( + keytool, "-genkeypair", + "-alias", "server", + "-keyalg", "RSA", + "-keysize", "2048", + "-validity", "3650", + "-dname", "CN=localhost", + "-ext", "SAN=dns:localhost,ip:127.0.0.1", + "-storetype", "PKCS12", + "-keystore", keystore.getAbsolutePath(), + "-storepass", new String(PASSWORD), + "-keypass", new String(PASSWORD) + ); + pb.redirectErrorStream(true); + Process process = pb.start(); + byte[] output = readAll(process.getInputStream()); + try { + if (!process.waitFor(60, TimeUnit.SECONDS) || process.exitValue() != 0) { + throw new IOException("keytool failed: " + new String(output, StandardCharsets.UTF_8)); + } + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new IOException("interrupted while running keytool", e); + } + try (InputStream in = new FileInputStream(keystore)) { + KeyStore ks = KeyStore.getInstance("PKCS12"); + ks.load(in, PASSWORD); + KeyManagerFactory kmf = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm()); + kmf.init(ks, PASSWORD); + SSLContext context = SSLContext.getInstance("TLS"); + context.init(kmf.getKeyManagers(), null, null); + return context.getServerSocketFactory(); + } catch (IOException e) { + throw e; + } catch (Exception e) { + throw new IOException("could not load the test keystore", e); + } + } + + private static byte[] readAll(InputStream in) throws IOException { + java.io.ByteArrayOutputStream out = new java.io.ByteArrayOutputStream(); + byte[] buf = new byte[4096]; + int n; + while ((n = in.read(buf)) > 0) { + out.write(buf, 0, n); + } + return out.toByteArray(); + } +} diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/websocket/TestWebSocketServer.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/websocket/TestWebSocketServer.java index 908e3bd56..85d6dea87 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/qwp/websocket/TestWebSocketServer.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/websocket/TestWebSocketServer.java @@ -49,6 +49,7 @@ import java.util.concurrent.TimeUnit; import java.util.concurrent.atomic.AtomicBoolean; import java.util.concurrent.atomic.AtomicInteger; +import java.util.function.ToIntFunction; /** * A simple WebSocket server for client integration testing. @@ -57,6 +58,10 @@ public class TestWebSocketServer implements Closeable { private static final Logger LOG = LoggerFactory.getLogger(TestWebSocketServer.class); private static final String WEBSOCKET_GUID = "258EAFA5-E914-47DA-95CA-C5AB0DC85B11"; + // Number of upgrades the authorization validator rejected, over the server's lifetime. + private final AtomicInteger authRejectCount = new AtomicInteger(); + // Number of well-formed upgrade requests received, accepted or rejected, over the server's lifetime. + private final AtomicInteger upgradeRequestCount = new AtomicInteger(); // Authorization header value captured from each well-formed upgrade request ("" when absent), in // arrival order. Tests poll this to assert the token a provider supplied at each (re)handshake. private final BlockingQueue capturedAuthHeaders = new LinkedBlockingQueue<>(); @@ -84,6 +89,12 @@ public class TestWebSocketServer implements Closeable { // connected even after they have all disconnected. private final AtomicInteger totalHandshakes = new AtomicInteger(); private Thread acceptThread; + // When non-null, decides every upgrade from its Authorization header value ("" when absent): a positive + // return is the HTTP status to reject with (e.g. 401, 403), zero or less accepts. Lets a test model a + // server that validates tokens - expired, revoked or rotated - rather than one that rejects blindly. + private volatile ToIntFunction authorizationValidator; + // WWW-Authenticate value sent with a 401 reject (validator or setRejectWithStatus), or null for none. + private volatile String rejectWwwAuthenticate; // X-QuestDB-Role value to emit on handshake responses. null = omit the // header (legacy behavior for tests written before role-aware failover). // The server emits the header on both the 101 success path and (when @@ -168,6 +179,12 @@ public TestWebSocketServer(WebSocketServerHandler handler, public TestWebSocketServer(WebSocketServerHandler handler, boolean emitDurableAckHeader, String advertisedRole, int requestedPort) throws IOException { + this(handler, emitDurableAckHeader, advertisedRole, requestedPort, false); + } + + private TestWebSocketServer(WebSocketServerHandler handler, + boolean emitDurableAckHeader, String advertisedRole, + int requestedPort, boolean tls) throws IOException { this.handler = handler; this.emitDurableAckHeader = emitDurableAckHeader; this.advertisedRole = advertisedRole; @@ -177,11 +194,21 @@ public TestWebSocketServer(WebSocketServerHandler handler, // which another process could grab a pre-selected port before start() // binds it. Pinning to loopback keeps client "localhost" connections // routed here rather than to a wildcard listener on the same port. - serverSocket = new ServerSocket(requestedPort, 50, java.net.InetAddress.getLoopbackAddress()); + serverSocket = tls + ? TestTls.serverSocketFactory().createServerSocket(requestedPort, 50, java.net.InetAddress.getLoopbackAddress()) + : new ServerSocket(requestedPort, 50, java.net.InetAddress.getLoopbackAddress()); serverSocket.setSoTimeout(100); this.port = serverSocket.getLocalPort(); } + /** + * A server that speaks WebSocket over TLS with a self-signed certificate (see {@link TestTls}). Connect + * with {@code wss::...;tls_verify=unsafe_off;}. + */ + public static TestWebSocketServer tls(WebSocketServerHandler handler) throws IOException { + return new TestWebSocketServer(handler, false, null, 0, true); + } + public boolean awaitRoleReject(long timeout, TimeUnit unit) throws InterruptedException { return roleRejectLatch.await(timeout, unit); } @@ -234,6 +261,46 @@ public int handshakeCount() { return totalHandshakes.get(); } + /** + * Number of upgrades the authorization validator rejected over the server's lifetime. + */ + public int authRejectCount() { + return authRejectCount.get(); + } + + /** + * Closes every live client connection while the listener keeps accepting, so connected clients see a + * dropped connection and reconnect. + */ + public void dropAllConnections() { + for (ClientHandler client : clients) { + client.close(); + } + } + + /** + * Number of well-formed upgrade requests received over the server's lifetime, whatever the answer. + */ + public int upgradeRequestCount() { + return upgradeRequestCount.get(); + } + + /** + * Installs a validator that decides every upgrade from its {@code Authorization} header value ({@code ""} + * when absent): a positive return is the HTTP status to reject with, zero or less accepts. Pass null to + * remove it. Runs before the role and status rejects. + */ + public void setAuthorizationValidator(ToIntFunction validator) { + this.authorizationValidator = validator; + } + + /** + * {@code WWW-Authenticate} value to send with a {@code 401} reject, or null for none. + */ + public void setRejectWwwAuthenticate(String value) { + this.rejectWwwAuthenticate = value; + } + /** * Number of WebSocket connections currently live from the server's view. * Drops back to zero once every client has closed its socket. @@ -603,6 +670,7 @@ private boolean performHandshake() throws IOException { if (key == null) { return false; } + upgradeRequestCount.incrementAndGet(); capturedAuthHeaders.add(authorization); // Read-path reject: drop the egress upgrade before the 101 so the @@ -612,18 +680,23 @@ private boolean performHandshake() throws IOException { return false; } + // Token-validating reject path: the validator inspects the presented credential. + ToIntFunction validator = authorizationValidator; + if (validator != null) { + int status = validator.applyAsInt(authorization); + if (status > 0) { + writeStatusReject(status, status == 401 ? "Unauthorized" : status == 403 ? "Forbidden" : "Rejected"); + authRejectCount.incrementAndGet(); + return false; + } + } + // Arbitrary-status reject path: tests use setRejectWithStatus // to drive the failover loop's terminal-vs-transient // classification (failover.md §6). int customStatus = rejectingStatusCode; if (customStatus > 0) { - String reason = rejectingStatusReason != null ? rejectingStatusReason : ""; - String sb = "HTTP/1.1 " + customStatus + ' ' + reason + "\r\n" + - "Connection: close\r\n" + - "Content-Length: 0\r\n" + - "\r\n"; - out.write(sb.getBytes(StandardCharsets.US_ASCII)); - out.flush(); + writeStatusReject(customStatus, rejectingStatusReason != null ? rejectingStatusReason : ""); statusRejectCount.incrementAndGet(); return false; } @@ -672,6 +745,20 @@ private boolean performHandshake() throws IOException { return true; } + private void writeStatusReject(int status, String reason) throws IOException { + StringBuilder sb = new StringBuilder() + .append("HTTP/1.1 ").append(status).append(' ').append(reason).append("\r\n") + .append("Connection: close\r\n") + .append("Content-Length: 0\r\n"); + String challenge = rejectWwwAuthenticate; + if (status == 401 && challenge != null) { + sb.append("WWW-Authenticate: ").append(challenge).append("\r\n"); + } + sb.append("\r\n"); + out.write(sb.toString().getBytes(StandardCharsets.US_ASCII)); + out.flush(); + } + private synchronized void writeFrame(int opcode, byte[] payload, int length) throws IOException { // first byte: FIN + opcode out.write(0x80 | (opcode & 0x0F)); @@ -702,6 +789,12 @@ void start() { readThread = new Thread(() -> { try { + if (socket instanceof javax.net.ssl.SSLSocket) { + // finish the TLS handshake under a generous timeout before the short polling + // timeout below applies to every read + socket.setSoTimeout(10_000); + ((javax.net.ssl.SSLSocket) socket).startHandshake(); + } socket.setSoTimeout(100); in = socket.getInputStream(); diff --git a/core/src/test/java/io/questdb/client/test/impl/QwpQueryClientConfigHonoredTest.java b/core/src/test/java/io/questdb/client/test/impl/QwpQueryClientConfigHonoredTest.java index 10b1e3c03..52758d3ea 100644 --- a/core/src/test/java/io/questdb/client/test/impl/QwpQueryClientConfigHonoredTest.java +++ b/core/src/test/java/io/questdb/client/test/impl/QwpQueryClientConfigHonoredTest.java @@ -28,6 +28,7 @@ import io.questdb.client.cutlass.qwp.client.QwpQueryClient; import io.questdb.client.impl.ConfigSchema; import io.questdb.client.impl.Side; +import io.questdb.client.test.cutlass.auth.TestTokenProviderFactory; import io.questdb.client.test.tools.TestUtils; import org.junit.Assert; import org.junit.Test; @@ -92,6 +93,19 @@ public void testEveryEgressKeyIsHonored() throws Exception { Assert.assertEquals("pw", tls.get("tls_roots_password")); markHonored("tls_verify", "tls_roots", "tls_roots_password"); + // token_provider and its azure keys (wss only; the factory is resolved at parse time, so install one). + TestTokenProviderFactory.install(new TestTokenProviderFactory("azure")); + try { + Map tp = snapshot("wss::addr=h:9000;token_provider=azure;" + + "azure_resource=api://app/.default;azure_client_id=AAAAAAAA-BBBB-CCCC-DDDD-EEEEEEEEEEEE;"); + Assert.assertEquals("azure", tp.get("token_provider")); + Assert.assertEquals("api://app", tp.get("azure_resource")); + Assert.assertEquals("aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee", tp.get("azure_client_id")); + } finally { + TestTokenProviderFactory.uninstall(); + } + markHonored("token_provider", "azure_resource", "azure_client_id"); + // Drift guard: every egress-applied registry key must have an assertion // above. The honored set is populated by the assertions themselves, so // deleting one trips this -- unlike a hand-maintained list, it cannot diff --git a/core/src/test/java/io/questdb/client/test/impl/WsSenderConfigHonoredTest.java b/core/src/test/java/io/questdb/client/test/impl/WsSenderConfigHonoredTest.java index 3c257827a..6e6d2358d 100644 --- a/core/src/test/java/io/questdb/client/test/impl/WsSenderConfigHonoredTest.java +++ b/core/src/test/java/io/questdb/client/test/impl/WsSenderConfigHonoredTest.java @@ -26,6 +26,7 @@ import io.questdb.client.Sender; import io.questdb.client.impl.ConfigSchema; +import io.questdb.client.test.cutlass.auth.TestTokenProviderFactory; import io.questdb.client.impl.Side; import org.junit.Assert; import org.junit.Test; @@ -82,6 +83,7 @@ public void testEveryIngressKeyIsHonored() { assertHonored("connection_listener_inbox_capacity=64", "connection_listener_inbox_capacity", 64); assertHonored("token=ey.abc", "token", "ey.abc"); assertHonored("auth_timeout_ms=4321", "auth_timeout_ms", 4321L); + assertHonored("auth_failure_max_duration_millis=600000", "auth_failure_max_duration_millis", 600000L); assertHonored("connect_timeout=7000", "connect_timeout", 7000); // username/password together (both-or-neither), and the user/pass aliases. @@ -104,6 +106,19 @@ public void testEveryIngressKeyIsHonored() { Assert.assertEquals("pw", tls.get("tls_roots_password")); markHonored("tls_roots", "tls_roots_password"); + // token_provider and its azure keys (wss only; the factory is resolved at parse time, so install one). + TestTokenProviderFactory.install(new TestTokenProviderFactory("azure")); + try { + Map tp = snapshot("wss::addr=h:9000;token_provider=azure;azure_resource=api://app/.default;" + + "azure_client_id=AAAAAAAA-BBBB-CCCC-DDDD-EEEEEEEEEEEE;"); + Assert.assertEquals("azure", tp.get("token_provider")); + Assert.assertEquals("api://app", tp.get("azure_resource")); + Assert.assertEquals("aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee", tp.get("azure_client_id")); + } finally { + TestTokenProviderFactory.uninstall(); + } + markHonored("token_provider", "azure_resource", "azure_client_id"); + // Drift guard: every ingress-applied registry key must have an assertion // above. The honored set is populated by the assertions themselves, so // deleting one trips this -- unlike a hand-maintained list, it cannot diff --git a/design/entra-id-qwp-auth.md b/design/entra-id-qwp-auth.md index 03985fce8..31c4c1e53 100644 --- a/design/entra-id-qwp-auth.md +++ b/design/entra-id-qwp-auth.md @@ -1,6 +1,6 @@ # Entra ID app-only auth over QWP: findings and proposed design -Status: **draft for review**. No production code has been written. This file is not committed. +Status: **implemented** on `feat/qwp-entra-token-provider` - all four steps of the Java plan (§10). §11 maps each step to the code and to the conformance tests, and lists where the code departs from this document's earlier design. **Read these first:** @@ -552,5 +552,53 @@ Build in this order. Each step is one PR and must pass the conformance tests lis **Separate tickets, outside this feature:** -- **Dead pooled egress worker** (§3). Start with a test that reproduces it. +- **Dead pooled egress worker** (§3). Start with a test that reproduces it. *Done after all - the spec makes recovery a MUST (§8.3, "Egress recovery"); see §11.* - **Treat a 503 at the upgrade as transient in every phase** (§3, spec Appendix C). This must land before any server change to return 503. + +## 11. Implementation status + +Every step of §10 is implemented. Line references in §1-§7 predate it. + +**Step 1 - token cache** (spec §3-§5, Appendix B). `io.questdb.client.cutlass.auth`: + +- `ExpiringToken`, `TokenSource`, `TokenUnavailableException`, and `RefreshingTokenProvider` with its builder (every §5.1 parameter, plus clock, scheduler and jitter seams for tests). +- `CredentialRedaction`: the §9 token rendering (length + 8-hex SHA-256 prefix) and the sanitizing of library text (display-unsafe characters stripped, 256-character cap). +- The warm path of `getToken()` is a volatile read with no lock and no I/O. A refresh whose wall-clock time has passed but whose monotonic schedule has not (a host that slept) starts in the background. +- A cold caller starts one fetch and then waits; it does not start another each time a fetch completes. A source that keeps returning a token inside the hand-out floor is therefore not fetched in a tight loop. While waiting, the "fail immediately" rule of §5.3 is re-evaluated after each failed fetch. +- Tests: `RefreshingTokenProviderTest` (C1-C8, the cache half of C20). + +**Step 2 - client integration** (spec §6, §8.1-§8.3). + +- `HttpTokenProvider.onTokenRejected` (default no-op). +- `QwpWebSocketSender.tokenProviderAuthHeader` is the named dynamic credential that replaced the lambda in `Sender.buildWebSocketAuthHeader`. +- The one retry after a refreshable 401 is in `QwpWebSocketSender.connectWalk` (shared by the foreground and the orphan drainers) and in `QwpQueryClient.connect` and `reconnectViaTracker`. It records no health penalty and fires no event. +- `WWW-Authenticate` is captured by `WebSocketClient` and parsed by `cutlass.http.BearerChallenge`. `QwpAuthFailedException.isTokenRefreshable()` applies the §8.2 challenge rule; its message now starts with `auth-rejected`. +- SYNC startup (`CursorWebSocketSendLoop.connectWithRetry`) retries a credential failure only when `QwpCredentialUnavailableException.isRetryable()` (D6/D8) and the thread is not interrupted. +- Egress errors name the failure class. A provider failure on `connect()` is a `QwpCredentialUnavailableException` with a `credential-unavailable:` message. +- Egress recovery: after a failover reconnect fails, the next `execute()` reconnects instead of throwing "not connected". `QwpQueryClientDynamicCredentialTest.testPooledQueryClientRecoversAfterAFailedFailover` reproduced the §3 dead-pooled-worker bug before the fix (it failed with exactly that message). +- Tests: `WebSocketDynamicCredentialTest` (C9-C16, C23), `QwpQueryClientDynamicCredentialTest`, `BearerChallengeTest`. `TestWebSocketServer` gained an authorization validator, a `WWW-Authenticate` challenge, `dropAllConnections()`, and a TLS mode. The TLS mode uses a self-signed identity generated per JVM with `keytool` (`TestTls`), because `token_provider` is `wss::`-only. + +**Step 3 - connect string, registry, Azure module** (spec §7). + +- `ConfigSchema` registers `token_provider`, `azure_resource` and `azure_client_id` (COMMON). +- `TokenProviderSpec.parse` enforces §7.2 on both clients and resolves the factory without fetching anything. It strips `/.default`, checks and lower-cases the client-ID GUID, and builds the registry key. +- `TokenProviderFactory` is the `ServiceLoader` SPI (`uses` in `module-info.java`). `TokenProviderRegistry` hands out ref-counted leases, with a 60 s linger, a daemon timer that exits when idle, and a test seam that replaces discovery. +- The non-QWP `Sender` schemas reject the keys (D9). The builders and `QuestDBBuilder` reject mixing them with an application-supplied provider. +- Leases: `Sender.build()` acquires one and hands it to `QwpWebSocketSender.setCredentialLease` (released on close, or released by `build()` if it fails). `QwpQueryClient` acquires one on its first `connect()` and releases it on close. +- New reactor module `azure/`, artifact `org.questdb:questdb-client-azure` (not `io.questdb`: the core artifact is `org.questdb:questdb-client`): + - `AzureTokenProviderFactory` uses `DefaultAzureCredential`; `azure_client_id` sets both the managed-identity and workload-identity client ID. + - `AzureTokenSource` bounds each request at 30 s and classifies per §7.5: `CredentialUnavailableException`, HTTP 400/401 and known AADSTS configuration codes are permanent; everything else is retryable, with `Retry-After` honoured. + - It never attaches a library exception: its response references the raw request. + - Java 8 bytecode. It has no `module-info`; `Automatic-Module-Name: io.questdb.client.azure`, and an automatic module provides its `META-INF/services`. + - Its release profiles mirror core's. +- Tests: `TokenProviderConfigTest` (C18, C19), `TokenProviderSharingTest` (C17, over TLS), `azure/AzureTokenSourceTest` (fake `TokenCredential`, no network). + +**Step 4 - health and deadline** (spec §8.4, §8.5). + +- `io.questdb.client.ConnectionHealth` (with `State`, `FailureClass`, `Failure`, `Aggregate`) and `QwpConnectionHealthTracker`, which publishes immutable snapshots through a volatile field. +- The ingest walk reports rounds and upgrades; the I/O loop reports connection loss and terminal failure. Drainers never report. +- Accessors: `Sender.health()` (default throws `UnsupportedOperationException`), `QwpQueryClient.health()`, `QuestDB.health()` (aggregate of every pooled connection). +- `auth_failure_max_duration_millis` (INGRESS key) and `Sender.LineSenderBuilder.authFailureMaxDurationMillis(long)`. The clock lives in `CursorWebSocketSendLoop`, applies to FOREGROUND loops only, and is tracked even before the deadline is armed. The builder arms it right after connecting, so an `async` start is measured from its first failure. +- Tests: `ConnectionHealthTest` (C21, C22). + +**Not done:** the spec documents (`qwp-ingress-websocket.md`, `qwp-egress-websocket.md`, the connect-string reference) live in the documentation repository. The Rust core and Python parity is tracked there too. diff --git a/design/qwp-token-provider-spec.md b/design/qwp-token-provider-spec.md index 9acd0e197..07e4e7c67 100644 --- a/design/qwp-token-provider-spec.md +++ b/design/qwp-token-provider-spec.md @@ -520,10 +520,10 @@ New types live in `io.questdb.client.cutlass.auth` unless noted. | Refreshing provider | `RefreshingTokenProvider implements HttpTokenProvider, QuietCloseable`, created with `RefreshingTokenProvider.builder(TokenSource)`. Methods: `getToken()`, `onTokenRejected(CharSequence, int)`, `awaitReady(long)`, `close()`. The builder exposes the §5.1 parameters, plus clock and scheduler seams for tests. | | token-unavailable error | `TokenUnavailableException extends LineSenderException`, with `isRetryable()` and `getRetryAfterMillis()` (-1 means none) | | `on_rejected` | `onTokenRejected(CharSequence token, int httpStatus)`, a default no-op method on the existing `io.questdb.client.HttpTokenProvider` | -| Provider factory | `TokenProviderFactory`, an SPI found through `ServiceLoader` (a `uses` clause in `module-info.java`) | -| Registry | `TokenProviderRegistry`: process-wide, hands out ref-counted leases | -| `azure` provider | The new reactor module `azure/`, artifact `io.questdb:questdb-client-azure`, package `io.questdb.client.azure`, class `AzureTokenProviderFactory`. It depends on `azure-identity` through `azure-sdk-bom`, keeps the Java 8 floor, and is released together with the client. | -| Connection health | `io.questdb.client.ConnectionHealth`, an immutable snapshot. Returned by `Sender.health()` (a default method; non-QWP senders throw `UnsupportedOperationException`), by `QwpQueryClient.health()`, and as an aggregate by `QuestDB.health()`. | +| Provider factory | `TokenProviderFactory`, an SPI found through `ServiceLoader` (a `uses` clause in `module-info.java`). `TokenProviderSpec.parse` applies the §7.2 rules to a connect string and resolves the factory without fetching a token. | +| Registry | `TokenProviderRegistry`: process-wide (`global()`), hands out ref-counted `Lease`s, closes a provider `registry_linger` after its last lease | +| `azure` provider | The new reactor module `azure/`, artifact `org.questdb:questdb-client-azure` (the core client is `org.questdb:questdb-client`), package `io.questdb.client.azure`, classes `AzureTokenProviderFactory` and `AzureTokenSource`. It depends on `azure-identity` through `azure-sdk-bom`, keeps the Java 8 floor, and is released together with the client. | +| Connection health | `io.questdb.client.ConnectionHealth`, an immutable snapshot. Returned by `Sender.health()` (a default method; non-QWP senders throw `UnsupportedOperationException`), by `QwpQueryClient.health()`, and as a `ConnectionHealth.Aggregate` by `QuestDB.health()`. | | Authentication-outage deadline | Key `auth_failure_max_duration_millis`; builder method `authFailureMaxDurationMillis(long)` | **Python notes:** diff --git a/pom.xml b/pom.xml index 0b9b6b335..85e2be820 100644 --- a/pom.xml +++ b/pom.xml @@ -86,6 +86,7 @@ core + azure examples From 6fcc5df8f3c4296863bbc378cd80aa3bcbdf27fe Mon Sep 17 00:00:00 2001 From: Vlad Ilyushchenko Date: Fri, 2 Oct 2026 19:30:57 +0100 Subject: [PATCH 03/40] Classify Azure credential errors by mode The spec classified Azure Identity's "no credential available in the chain" as permanent (section 7.5), while section 4 calls network failures and IMDS 404/410 retryable. Inside DefaultAzureCredential an unreachable IMDS produces exactly that message: the chain probes IMDS once, with a short timeout and no retries, so a transient IMDS outage failed a SYNC startup fast. Spec v0.4 follows Azure Identity's own split between fail-fast discovery and a resilient single credential: - 7.1, 7.2: a new key, azure_credential (default, managed_identity, workload_identity, environment), with its validation rules. - 7.5: library errors are classified by how the credential was selected. With one credential, configuration errors are permanent and endpoint or network failures retryable. In the discovery chain, "no credential available" is retryable and the client warns once. Errors should carry the library's innermost reason. - 4: a source that discovers its credential by probing must not treat "nothing found" as permanent on that evidence alone. - 10: conformance test C24. 12: decision D10, which records why an SMBIOS host check was rejected. Appendix B: the Java binding. Appendix D: precedents from Azure Identity, MongoDB, Google, AWS and Apache Druid. The Java implementation still follows v0.3. --- design/qwp-token-provider-spec.md | 51 +++++++++++++++++++++++-------- 1 file changed, 39 insertions(+), 12 deletions(-) diff --git a/design/qwp-token-provider-spec.md b/design/qwp-token-provider-spec.md index 07e4e7c67..f9f312500 100644 --- a/design/qwp-token-provider-spec.md +++ b/design/qwp-token-provider-spec.md @@ -1,9 +1,10 @@ # QWP dynamic bearer credentials: cross-language specification -**Status:** v0.3, ready for implementation. All decisions are resolved (§12). +**Status:** v0.4, ready for implementation. All decisions are resolved (§12). **Changes:** +- **v0.4:** `azure_credential` selects one Azure credential deterministically (§7.1), and library errors are classified by how the credential was selected (§7.5, D10). This resolves the conflict between §4 and §7.5 over an unreachable managed-identity endpoint. - **v0.3:** decisions resolved, with D8 (§8.1) and D9 (§7.2) added; Java binding decided (Appendix B). - **v0.2:** connection health (§8.4), the optional authentication-outage deadline (§8.5), `WWW-Authenticate` handling (§8.2), and Appendices C and D. @@ -86,7 +87,8 @@ Rules for `expires_at`: Classification: - **Retryable:** network failures, timeouts, HTTP 429, HTTP 5xx, and IMDS 404 or 410. -- **Permanent:** configuration that is missing or wrong. Examples: no credential configured, identity not found, invalid client. +- **Permanent:** configuration that is missing or wrong. Examples: the credential the source was told to use is not configured, identity not found, invalid client. +- **Discovery:** a source that finds its credential by probing the environment MUST NOT treat "nothing found" as permanent on that evidence alone. A brief outage of a probed endpoint looks the same (§7.5) **[D10]**. - **When unsure:** treat the failure as retryable. - An error that carries no classification MUST be treated as retryable. @@ -236,6 +238,7 @@ None of these MAY reveal the token itself. | `token_provider` | `azure`. The name `azure_imds` is reserved **[D5]**. | no | Selects a provider. | | `azure_resource` | `api://` or `` | when `token_provider=azure` | The application ID URI or the client ID of the QuestDB app registration. A trailing `/.default` MUST be stripped. The requested scope is `/.default`. | | `azure_client_id` | a GUID | no | The client ID of a user-assigned managed identity or of a workload identity. | +| `azure_credential` | `default`, `managed_identity`, `workload_identity`, `environment` | no | Which Azure credential `azure` uses (§7.5). `default`, the default, is the library's default credential: unless the platform's own selector (`AZURE_TOKEN_CREDENTIALS`) names one credential, it is a discovery chain, meant for development. Each other value selects that one credential and overrides the platform's selector. This deterministic mode is recommended in production **[D10]**. | ### 7.2 Validation @@ -245,6 +248,8 @@ A client MUST reject the configuration, with an error that names the offending k - `token_provider` is empty, unknown, or not supported by this client. The error MUST list the supported values. A client MUST NOT silently connect without credentials; - a provider-specific key is present but the selected provider does not accept it. For example, `azure_resource` without `token_provider=azure`; - a provider key that is required is missing; +- `azure_credential` has a value that §7.1 does not list. The error MUST list the supported values; +- `azure_client_id` is combined with `azure_credential=environment`. That credential takes its client ID from the platform's configuration; - `token_provider` is used with any schema other than `wss::`: - plain `ws::` is rejected because the token would cross the network in cleartext **[D4]**; - non-QWP schemas, such as ILP's `http::`, are out of scope **[D9]**. @@ -269,23 +274,29 @@ A client MUST reject the configuration, with an error that names the offending k ### 7.5 The `azure` provider -- **How tokens are obtained:** through the language's Azure Identity default credential chain, for the scope `/.default`. Depending on the environment, the chain finds: - - an environment service principal, - - a workload identity, - - a managed identity, - - developer tools. +- **How tokens are obtained:** through the language's Azure Identity library, for the scope `/.default`. `azure_credential` (§7.1) decides which credential is used: + - `default`: the library's default credential. Unless the platform's selector names one credential, this is a **discovery chain**. It tries an environment service principal, a workload identity, a managed identity and developer tools, in that order, and uses the first that works. + - `managed_identity`, `workload_identity` or `environment`: that credential alone, called **deterministic mode**. A platform selector that names one credential has the same effect. +- **Why the mode matters:** to keep development machines fast, a discovery chain checks for the instance metadata service (IMDS) with one short request and no retries, and moves on when nothing answers. So it reports "no credential available" both on a machine without a managed identity and on an Azure host whose IMDS is briefly unreachable. In deterministic mode the library skips that check and retries the managed-identity endpoint with backoff, so the two cases are reported differently. - **`azure_client_id`:** when set, it selects both the managed identity and the workload identity. - **Expiry:** `expires_at`, and `refresh_at` when the library exposes it, come from the library's token result. -- **Classification of library errors:** - - **Permanent:** "no credential available in the chain", and authentication failures that Entra reports (such as an invalid client or an application that does not exist). - - **Retryable:** network errors, timeouts, throttling, 5xx responses, and anything else. +- **Classification of library errors [D10]:** + - **Deterministic mode:** + - **Permanent:** configuration that the selected credential reports as missing or wrong. That covers missing settings it requires (for example, no federated token file for a workload identity), an identity that is not found or not assigned to the host, and Entra rejecting the client, secret, assertion, application or tenant. + - **Retryable:** network errors, including a managed-identity endpoint that cannot be reached; timeouts; throttling; `404`, `410` and `5xx` responses from a managed-identity endpoint; and anything else. + - **Discovery chain:** + - "No credential available in the chain" is **retryable** (§4). + - A failure that one credential in the chain reports for itself is classified as in deterministic mode. + - A client SHOULD log once, when the provider starts, that a discovery chain does not retry a managed-identity outage, and recommend `azure_credential` for production. +- **Error text:** libraries often report every managed-identity failure with one generic message and keep the reason, such as "Identity not found", in a nested cause. The error SHOULD carry that innermost reason, sanitized as §9 requires. - **Unsupported platforms:** a client whose platform has no Azure Identity library MAY leave `azure` unsupported, and must then reject it as described in §7.2. *Informative bindings:* -- **Java:** the optional `questdb-client-azure` artifact, which uses `DefaultAzureCredential` and is found through `ServiceLoader`. +- **Java:** the optional `questdb-client-azure` artifact, which uses `DefaultAzureCredential` and is found through `ServiceLoader`. It maps `azure_credential` onto the library's own selector (Appendix B). - **Python:** `azure-identity` as an optional extra. - **Rust and C:** not required. +- **Platform support:** the Azure Identity libraries read `AZURE_TOKEN_CREDENTIALS`. When it selects `ManagedIdentityCredential`, they skip the IMDS check and retry with backoff from .NET 1.16.0, Java 1.18.1, Python 1.25.1, JavaScript 4.13.0, Go 1.13.0 and C++ 1.13.2. ## 8. Failure handling @@ -437,6 +448,7 @@ Every client that implements this specification SHOULD pass these scenarios. The | C21 | Connection health | The snapshot moves through `connecting`, `connected`, `reconnecting` (with `401` and credential-unavailable failures) and back to `connected`. `outage_since` and `failed_rounds` reset on recovery; `last_failure` is kept. No credential appears in it. | | C22 | Authentication-outage deadline | Unset: retries continue indefinitely. Set: the sender becomes terminal only on an authentication-class round once the duration has passed. Other failures in between neither reset nor fire it. A successful upgrade resets it. Orphan drains are unaffected, and on-disk data remains. | | C23 | `WWW-Authenticate` challenge | A plain `401`, and a `401` with `Bearer error="invalid_token"`, both trigger the retry in §8.2. A `401` whose Bearer challenge carries another error does not. | +| C24 | Azure credential selection | With `azure_credential=managed_identity`, an unreachable managed-identity endpoint is retryable, so `sync` initialization retries it within its budget; an identity that is not assigned is permanent, so initialization fails fast. With `default`, a chain that finds nothing is retryable, and the warning in §7.5 is logged once. | ## 11. Relation to existing specifications @@ -461,6 +473,7 @@ All decisions below are resolved. Changing one after the Java implementation mer | D7 | Whether to offer the authentication-outage deadline, and its key | Yes, as an optional setting that is off by default: `auth_failure_max_duration_millis`. | | D8 | How to classify an exception from an application-supplied provider that is not a token-unavailable error | Permanent (§8.1). | | D9 | Whether `token_provider` applies to non-QWP schemas | No; `wss::` only (§7.2). | +| D10 | How the `azure` provider classifies "no credential available", and how to make it resilient | `azure_credential` selects one credential, so the library itself reports an unreachable endpoint (retryable) apart from a misconfigured credential (permanent). In a discovery chain, "no credential available" is retryable, because an IMDS outage looks the same (§7.5). Only `sync` initialization depends on the difference (§8.3), so the cost is a wait of at most `reconnect_max_duration_millis`. This follows Azure Identity's split between fail-fast discovery and a resilient single credential (Appendix D). A local host check, as Google's library makes, was not adopted: Azure's SMBIOS asset tag identifies only the public cloud, not sovereign clouds or Azure Local. | **Still to verify in a real Entra tenant.** These checks don't block implementation, but they should be done before the spec freezes: @@ -468,6 +481,7 @@ All decisions below are resolved. Changing one after the Java implementation mer - The v1 and v2 signing-key endpoints serve the same keys. - Whether IMDS returns the same token when asked again. This affects only the convergence note in §5.2. - How long `DefaultAzureCredential` takes on a cold start. This sizes `cold_wait`. +- On a VM, with `azure_credential=managed_identity`: an unreachable IMDS is reported as retryable, and an identity that is not assigned as permanent. So far this was checked only against a stub endpoint, with Azure Identity for Java 1.18.4. ## Appendix A. QuestDB Enterprise and Entra configuration (informative) @@ -525,6 +539,7 @@ New types live in `io.questdb.client.cutlass.auth` unless noted. | `azure` provider | The new reactor module `azure/`, artifact `org.questdb:questdb-client-azure` (the core client is `org.questdb:questdb-client`), package `io.questdb.client.azure`, classes `AzureTokenProviderFactory` and `AzureTokenSource`. It depends on `azure-identity` through `azure-sdk-bom`, keeps the Java 8 floor, and is released together with the client. | | Connection health | `io.questdb.client.ConnectionHealth`, an immutable snapshot. Returned by `Sender.health()` (a default method; non-QWP senders throw `UnsupportedOperationException`), by `QwpQueryClient.health()`, and as a `ConnectionHealth.Aggregate` by `QuestDB.health()`. | | Authentication-outage deadline | Key `auth_failure_max_duration_millis`; builder method `authFailureMaxDurationMillis(long)` | +| Credential selection | Key `azure_credential`, handled by `AzureTokenProviderFactory`. It sets `AZURE_TOKEN_CREDENTIALS` (`ManagedIdentityCredential`, `WorkloadIdentityCredential` or `EnvironmentCredential`) in the configuration it passes to `DefaultAzureCredentialBuilder.configuration(...)`, never in the process environment. That needs `azure-identity` 1.18.1 or later. `AzureTokenSource` learns the mode from the factory; for a credential an application passes in itself, a `DefaultAzureCredential` counts as a discovery chain and any other credential as deterministic. | **Python notes:** @@ -551,7 +566,7 @@ These changes would let clients refresh only when a new token can help, and trea Other server-side follow-ups, including security hardening, are tracked privately with the QuestDB Enterprise team (see `SECURITY.md`). -## Appendix D. Precedents for D1 and D2 (informative) +## Appendix D. Precedents for D1, D2 and D10 (informative) | Source | Behaviour | Bearing on this spec | |---|---|---| @@ -562,6 +577,10 @@ Other server-side follow-ups, including security hardening, are tracked privatel | Azure SDK, `BearerTokenAuthenticationPolicy` | Retries only on a `401` that carries a Continuous Access Evaluation claims challenge. A plain `401` goes back to the caller. | A stricter variant of D2. | | Kafka: KIP-152 and KAFKA-6516 | Authentication failures are non-retriable at the API. The client is not closed, though, and keeps reconnecting in the background; a request to stop that was closed Won't Fix. | D1. | | Kafka: KAFKA-10840 (open) | When credentials expire, a consumer keeps failing authentication in the background and the application cannot see it. A proposed fix notes that Kafka Connect tasks report RUNNING meanwhile. | Why §8.4 exists. | +| Azure Identity: credential chains and the managed-identity retry strategy | `DefaultAzureCredential` runs managed identity in a "fail fast" mode, meant for the development inner loop: one IMDS probe with a short timeout and no retries. A "resilient" mode skips the probe and retries with exponential backoff; `AZURE_TOKEN_CREDENTIALS=ManagedIdentityCredential`, or the managed-identity credential used directly, enables it. Microsoft recommends a deterministic credential in production. | D10: a discovery chain's "no credential" is no evidence of misconfiguration; `azure_credential` selects the resilient mode. | +| MongoDB driver auth spec (MONGODB-OIDC) | The application names its environment (`ENVIRONMENT:azure`, `gcp` or `k8s`), and the driver calls that metadata endpoint directly, with no discovery. | D10: deterministic selection. | +| Google auth library, `ComputeEngineCredentials` | Detection pings the metadata server three times with a short timeout, "for developer desktop scenarios", then checks the SMBIOS product name, which needs no network. So a slow metadata server on a real VM is not taken for "not on GCE". A `503` from the metadata server is retryable. | D10: a probe that gets no answer is not proof; the local check is not adopted (§12). | +| AWS SDK for Java v2, issue #3939; Apache Druid | On EC2, the default chain sometimes reports "Unable to load credentials from any of the providers in the chain", which AWS attributes to IMDS latency. Druid now treats that error as recoverable within its retry budget. | D10: "nothing found" in a chain is retryable. | Links: @@ -573,3 +592,11 @@ Links: - KIP-152: https://cwiki.apache.org/confluence/display/KAFKA/KIP-152+-+Improve+diagnostics+for+SASL+authentication+failures - KAFKA-6516: https://issues.apache.org/jira/browse/KAFKA-6516 - KAFKA-10840: https://issues.apache.org/jira/browse/KAFKA-10840 (proposed fix: https://github.com/apache/kafka/pull/16418) +- Azure Identity best practices (deterministic credentials, the managed-identity retry strategy): https://learn.microsoft.com/en-us/dotnet/azure/sdk/authentication/best-practices +- Azure Identity for Java, credential chains and `AZURE_TOKEN_CREDENTIALS`: https://learn.microsoft.com/en-us/azure/developer/java/sdk/authentication/credential-chains +- Azure SDK design note on IMDS probing: https://gist.github.com/ahsonkhan/d6c4d3a9780bb058a729b76844923ef1 +- Azure SDK release, October 2025 (resilient managed identity in C++, Go, Java, JavaScript and Python): https://devblogs.microsoft.com/azure-sdk/azure-sdk-release-october-2025/ +- Identify an Azure VM from the guest (SMBIOS asset tag, public cloud only): https://learn.microsoft.com/en-us/azure/virtual-machines/identify-azure-vm-from-guest +- Google `ComputeEngineCredentials`: https://github.com/googleapis/google-auth-library-java/blob/main/oauth2_http/java/com/google/auth/oauth2/ComputeEngineCredentials.java +- AWS SDK for Java v2, issue #3939: https://github.com/aws/aws-sdk-java-v2/issues/3939 +- Apache Druid, retry transient AWS credential resolution failures (#19558): https://www.mail-archive.com/commits@druid.apache.org/msg115181.html From 6492ebf899d772bf3fdf1975f70bb5fe547f87cd Mon Sep 17 00:00:00 2001 From: Vlad Ilyushchenko Date: Tue, 6 Oct 2026 19:34:25 +0100 Subject: [PATCH 04/40] feat(qwp): select the Azure credential with azure_credential Implements spec v0.4's azure_credential and aligns the spec with the code, so the two no longer disagree. - azure_credential=default|managed_identity|workload_identity|environment is accepted by the Sender, QwpQueryClient and the QuestDB facade, and validated per spec 7.2: an unknown value lists the supported ones, and azure_client_id cannot be combined with environment. default is normalized away, so existing connect strings take the same code path and share providers as before. - The other values build that credential directly. Passing AZURE_TOKEN_CREDENTIALS to DefaultAzureCredentialBuilder.configuration(), as Appendix B prescribed, does not work in azure-identity 1.18: it selects the credential but keeps the chain's one-shot IMDS probe, because each credential re-reads the selector from the process-wide configuration. A managed identity built directly retries an unreachable endpoint and rides out an outage. - A managed identity used alone reports "identity not assigned" only in a nested MSAL message. AzureTokenSource now classifies it as permanent; otherwise a sync startup retries that misconfiguration for its whole connect budget (measured: 316 s instead of ~25 s). - Retryable errors lead with their socket-level cause, such as a refused connection, which managed identity otherwise hides behind "see inner exception". Only java.net socket messages are echoed. - The discovery chain's "no credential available" stays permanent, so a host without a credential still fails fast. Instead, the provider warns once when DefaultAzureCredential starts as a chain, recommending azure_credential in production. - A workload identity missing its platform configuration no longer fails inside a client's build(): every fetch fails as permanent. The spec is amended to match (sections 4, 7.5, C24, D10, Appendix B and D), and entra-id-qwp-auth.md now points at v0.4. Tests: AzureManagedIdentityLibraryTest runs the real library in a child JVM against a local IMDS stub (MSAL reads the endpoint only from the process environment) for C24: an outage ridden out, an unreachable endpoint retryable, an unassigned identity permanent, the chain failing fast and warning once, and the platform selector silencing the warning. Plus AzureCredentialSelectionTest, new AzureTokenSourceTest cases, and TokenProviderConfigTest and config-honored coverage for the new key. --- README.md | 20 +- .../azure/AzureTokenProviderFactory.java | 148 ++++++++++- .../client/azure/AzureTokenSource.java | 68 ++++- .../azure/test/AzureCredentialProbeMain.java | 101 +++++++ .../test/AzureCredentialSelectionTest.java | 193 ++++++++++++++ .../test/AzureManagedIdentityLibraryTest.java | 249 ++++++++++++++++++ .../azure/test/AzureTokenSourceTest.java | 74 ++++++ .../questdb/client/azure/test/ImdsStub.java | 201 ++++++++++++++ .../main/java/io/questdb/client/Sender.java | 5 +- .../cutlass/auth/TokenProviderFactory.java | 6 +- .../cutlass/auth/TokenProviderSpec.java | 41 +++ .../cutlass/qwp/client/QwpQueryClient.java | 2 + .../io/questdb/client/impl/ConfigSchema.java | 1 + .../cutlass/auth/TokenProviderConfigTest.java | 51 +++- .../impl/QwpQueryClientConfigHonoredTest.java | 6 +- .../test/impl/WsSenderConfigHonoredTest.java | 5 +- design/entra-id-qwp-auth.md | 10 +- design/qwp-token-provider-spec.md | 20 +- 18 files changed, 1152 insertions(+), 49 deletions(-) create mode 100644 azure/src/test/java/io/questdb/client/azure/test/AzureCredentialProbeMain.java create mode 100644 azure/src/test/java/io/questdb/client/azure/test/AzureCredentialSelectionTest.java create mode 100644 azure/src/test/java/io/questdb/client/azure/test/AzureManagedIdentityLibraryTest.java create mode 100644 azure/src/test/java/io/questdb/client/azure/test/ImdsStub.java diff --git a/README.md b/README.md index ebdd2e145..9a4332685 100644 --- a/README.md +++ b/README.md @@ -505,18 +505,25 @@ query clients keep reconnecting with a valid token. ```java try (QuestDB db = QuestDB.connect( - "wss::addr=qdb1:9000,qdb2:9000;token_provider=azure;azure_resource=api://;")) { + "wss::addr=qdb1:9000,qdb2:9000;token_provider=azure;azure_resource=api://;" + + "azure_credential=managed_identity;")) { // ... use db ... } ``` - `azure_resource` (required) is the application ID URI (`api://`) or the client ID of the QuestDB app registration; the client requests the scope `/.default`. -- `azure_client_id` (optional) selects a user-assigned managed identity or a workload identity. -- The credential comes from Azure Identity's `DefaultAzureCredential`: an environment service principal - (`AZURE_TENANT_ID`, `AZURE_CLIENT_ID`, `AZURE_CLIENT_SECRET` or a certificate), a workload identity, a managed - identity, or developer tools. No secret ever goes into the connect string, so it stays safe to log and to put in - `QDB_CLIENT_CONF`. +- `azure_credential` (optional) selects the credential. In production, name it: `managed_identity`, + `workload_identity` or `environment` (a service principal from `AZURE_TENANT_ID`, `AZURE_CLIENT_ID` and + `AZURE_CLIENT_SECRET` or a certificate). A managed identity selected this way retries an unreachable endpoint, and + fails fast when the identity is not assigned to the host. +- `default`, the default, is Azure Identity's `DefaultAzureCredential`, a discovery chain meant for development: an + environment service principal, a workload identity, a managed identity, or developer tools. It checks the + managed-identity endpoint once without retrying, so an outage of that endpoint fails a startup as if the host had + no credential; the client logs a warning saying so. +- `azure_client_id` (optional) selects a user-assigned managed identity or a workload identity. It cannot be combined + with `azure_credential=environment`. +- No secret ever goes into the connect string, so it stays safe to log and to put in `QDB_CLIENT_CONF`. - `token_provider` requires `wss::`, and cannot be combined with `token`, `username`/`password` or an application-supplied provider. - Every sender, query client and pooled connection built from equivalent connect strings shares one provider per @@ -627,6 +634,7 @@ schema::key1=value1;key2=value2; | `token_provider` | | Refreshing bearer-token provider, `wss` only: `azure` (needs `questdb-client-azure`) | | `azure_resource` | | `token_provider=azure`: application ID URI or client ID of the QuestDB app registration | | `azure_client_id` | | `token_provider=azure`: client ID of a user-assigned managed identity or workload identity | +| `azure_credential` | `default` | `token_provider=azure`: `default`, `managed_identity`, `workload_identity` or `environment` | | `auth_failure_max_duration_millis` | _(none)_ | Ingest: fail the sender once a credential outage or `401`/`403` lasts this long | ### Pool keys (facade only) diff --git a/azure/src/main/java/io/questdb/client/azure/AzureTokenProviderFactory.java b/azure/src/main/java/io/questdb/client/azure/AzureTokenProviderFactory.java index 950f1bbcb..b0522fa06 100644 --- a/azure/src/main/java/io/questdb/client/azure/AzureTokenProviderFactory.java +++ b/azure/src/main/java/io/questdb/client/azure/AzureTokenProviderFactory.java @@ -24,32 +24,127 @@ package io.questdb.client.azure; +import com.azure.core.credential.AccessToken; +import com.azure.core.credential.TokenCredential; +import com.azure.core.credential.TokenRequestContext; +import com.azure.core.util.Configuration; +import com.azure.identity.CredentialUnavailableException; import com.azure.identity.DefaultAzureCredentialBuilder; +import com.azure.identity.EnvironmentCredentialBuilder; +import com.azure.identity.ManagedIdentityCredentialBuilder; +import com.azure.identity.WorkloadIdentityCredentialBuilder; import io.questdb.client.cutlass.auth.TokenProviderFactory; import io.questdb.client.cutlass.auth.TokenProviderSpec; import io.questdb.client.cutlass.auth.TokenSource; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import reactor.core.publisher.Mono; import java.util.Map; /** * The {@code azure} token provider (design/qwp-token-provider-spec.md, section 7.5): Microsoft Entra ID tokens - * from Azure Identity's {@code DefaultAzureCredential}. Discovered through {@link java.util.ServiceLoader}; put - * this artifact on the class path or module path and connect with + * from Azure Identity. Discovered through {@link java.util.ServiceLoader}; put this artifact on the class path or + * module path and connect with * 
- * wss::addr=qdb1:9000,qdb2:9000;token_provider=azure;azure_resource=api://<questdb-app-id>;
+ * wss::addr=qdb1:9000,qdb2:9000;token_provider=azure;azure_resource=api://<questdb-app-id>;azure_credential=managed_identity;
  *
*
    *
  • {@code azure_resource} (required) - the application ID URI or client ID of the QuestDB app * registration; the requested scope is {@code /.default}.
    • + *
  • {@code azure_credential} (optional) - which credential to use: + *
      + *
    • {@code default}, the default - {@code DefaultAzureCredential}, a discovery chain meant for + * development: an environment service principal ({@code AZURE_CLIENT_ID}, {@code AZURE_TENANT_ID}, + * {@code AZURE_CLIENT_SECRET} or a certificate), a workload identity, a managed identity, then developer + * tools. The chain checks for the managed-identity endpoint once, without retrying, so an outage of that + * endpoint looks like a host without a credential: a {@code sync} startup fails fast instead of riding it + * out. The provider logs a warning saying so when it starts, unless Azure Identity's own selector, + * {@code AZURE_TOKEN_CREDENTIALS}, already narrows the chain.
      • + *
    • {@code managed_identity}, {@code workload_identity} or {@code environment} - that credential alone, + * recommended in production. A managed identity used alone retries an unreachable endpoint with backoff, + * and reports it apart from an identity that is not assigned to the host.
      • + *
    + *
    • *
  • {@code azure_client_id} (optional) - the client ID of a user-assigned managed identity or of a workload - * identity; it selects both.
    • + * identity. With {@code default} it selects both; it cannot be combined with {@code environment}, which reads + * its client ID from the platform's configuration. *
- * Depending on the environment, the credential chain finds an environment service principal - * ({@code AZURE_CLIENT_ID}, {@code AZURE_TENANT_ID}, {@code AZURE_CLIENT_SECRET} or a certificate), a workload - * identity, a managed identity, or developer tools. Secrets come from that standard configuration, never from - * the connect string. + * Secrets come from the platform's standard configuration, never from the connect string. + *

+ * The deterministic credentials are built directly, not by passing {@code AZURE_TOKEN_CREDENTIALS} to + * {@code DefaultAzureCredentialBuilder.configuration(...)}: with that, Azure Identity 1.18 selects the credential + * but keeps the managed-identity endpoint check of a chain, because each credential re-reads the selector from the + * process-wide configuration. */ public final class AzureTokenProviderFactory implements TokenProviderFactory { + // Azure Identity's own credential selector, an environment variable or system property + private static final String AZURE_TOKEN_CREDENTIALS = "AZURE_TOKEN_CREDENTIALS"; + private static final Logger LOG = LoggerFactory.getLogger(AzureTokenProviderFactory.class); + + // Validates azure_credential and the keys it constrains. Null for default. + private static String credentialMode(Map params) { + final String credential = params.get(TokenProviderSpec.KEY_AZURE_CREDENTIAL); + if (credential == null) { + return null; + } + switch (credential) { + case TokenProviderSpec.AZURE_CREDENTIAL_MANAGED_IDENTITY: + case TokenProviderSpec.AZURE_CREDENTIAL_WORKLOAD_IDENTITY: + return credential; + case TokenProviderSpec.AZURE_CREDENTIAL_ENVIRONMENT: + if (params.get(TokenProviderSpec.KEY_AZURE_CLIENT_ID) != null) { + throw new IllegalArgumentException(TokenProviderSpec.KEY_AZURE_CLIENT_ID + " cannot be combined with " + + TokenProviderSpec.KEY_AZURE_CREDENTIAL + '=' + TokenProviderSpec.AZURE_CREDENTIAL_ENVIRONMENT); + } + return credential; + default: + throw new IllegalArgumentException("invalid " + TokenProviderSpec.KEY_AZURE_CREDENTIAL + ": " + credential + + " (expected default, managed_identity, workload_identity, environment)"); + } + } + + private static TokenCredential deterministicCredential(String mode, String clientId) { + try { + switch (mode) { + case TokenProviderSpec.AZURE_CREDENTIAL_MANAGED_IDENTITY: { + final ManagedIdentityCredentialBuilder builder = new ManagedIdentityCredentialBuilder(); + if (clientId != null) { + builder.clientId(clientId); + } + return builder.build(); + } + case TokenProviderSpec.AZURE_CREDENTIAL_WORKLOAD_IDENTITY: { + final WorkloadIdentityCredentialBuilder builder = new WorkloadIdentityCredentialBuilder(); + if (clientId != null) { + builder.clientId(clientId); + } + return builder.build(); + } + default: + return new EnvironmentCredentialBuilder().build(); + } + } catch (RuntimeException e) { + // The credential lacks configuration it requires and cannot even be built: a workload identity without + // its tenant ID, client ID or federated token file. That is permanent (section 7.5), and it is reported + // like any other credential failure - by the provider, when a client connects - rather than from inside + // a client's build(). + return new UnconfiguredCredential(e.getMessage() == null ? e.getClass().getName() : e.getMessage()); + } + } + + /** + * Whether {@code DefaultAzureCredential} runs as a discovery chain that includes the managed-identity + * endpoint check: Azure Identity's selector, {@code AZURE_TOKEN_CREDENTIALS}, is unset or {@code prod}. With + * {@code dev} the chain has no managed identity, and a credential name selects that credential alone. + */ + static boolean isDiscoveryChain(String selector) { + if (selector == null) { + return true; + } + final String value = selector.trim(); + return value.isEmpty() || "prod".equalsIgnoreCase(value); + } @Override public TokenSource createSource(Map params) { @@ -57,19 +152,36 @@ public TokenSource createSource(Map params) { if (resource == null) { throw new IllegalArgumentException("token_provider=azure requires " + TokenProviderSpec.KEY_AZURE_RESOURCE); } - final DefaultAzureCredentialBuilder builder = new DefaultAzureCredentialBuilder(); + final String mode = credentialMode(params); final String clientId = params.get(TokenProviderSpec.KEY_AZURE_CLIENT_ID); + if (mode != null) { + return new AzureTokenSource(deterministicCredential(mode, clientId), resource); + } + final DefaultAzureCredentialBuilder builder = new DefaultAzureCredentialBuilder(); if (clientId != null) { builder.managedIdentityClientId(clientId); builder.workloadIdentityClientId(clientId); } - return new AzureTokenSource(builder.build(), resource); + final AzureTokenSource source = new AzureTokenSource(builder.build(), resource); + if (isDiscoveryChain(Configuration.getGlobalConfiguration().get(AZURE_TOKEN_CREDENTIALS))) { + // section 7.5: once, when the provider starts + LOG.warn("token provider {}: azure_credential is not set, so tokens come from DefaultAzureCredential, a " + + "discovery chain meant for development. It checks the managed-identity endpoint once, without " + + "retrying, so an outage of that endpoint fails a sync startup as if this host had no credential. " + + "In production, select the credential: azure_credential=managed_identity, workload_identity or " + + "environment.", describe(params)); + } + return source; } @Override public String describe(Map params) { final StringBuilder sb = new StringBuilder(TokenProviderSpec.AZURE) .append("[resource=").append(params.get(TokenProviderSpec.KEY_AZURE_RESOURCE)); + final String credential = params.get(TokenProviderSpec.KEY_AZURE_CREDENTIAL); + if (credential != null) { + sb.append(", credential=").append(credential); + } final String clientId = params.get(TokenProviderSpec.KEY_AZURE_CLIENT_ID); if (clientId != null) { sb.append(", client_id=").append(clientId); @@ -87,5 +199,21 @@ public void validate(Map params) { if (params.get(TokenProviderSpec.KEY_AZURE_RESOURCE) == null) { throw new IllegalArgumentException("token_provider=azure requires " + TokenProviderSpec.KEY_AZURE_RESOURCE); } + credentialMode(params); + } + + // A credential whose configuration is missing: every request fails with Azure Identity's own "unavailable" + // exception, which AzureTokenSource classifies as permanent. + private static final class UnconfiguredCredential implements TokenCredential { + private final String reason; + + UnconfiguredCredential(String reason) { + this.reason = reason; + } + + @Override + public Mono getToken(TokenRequestContext request) { + return Mono.error(new CredentialUnavailableException(reason)); + } } } diff --git a/azure/src/main/java/io/questdb/client/azure/AzureTokenSource.java b/azure/src/main/java/io/questdb/client/azure/AzureTokenSource.java index b0c6e793d..e74fa36b1 100644 --- a/azure/src/main/java/io/questdb/client/azure/AzureTokenSource.java +++ b/azure/src/main/java/io/questdb/client/azure/AzureTokenSource.java @@ -29,11 +29,15 @@ import com.azure.core.credential.TokenRequestContext; import com.azure.core.exception.HttpResponseException; import com.azure.core.http.HttpResponse; +import com.azure.identity.ChainedTokenCredential; import com.azure.identity.CredentialUnavailableException; import io.questdb.client.cutlass.auth.ExpiringToken; import io.questdb.client.cutlass.auth.TokenSource; import io.questdb.client.cutlass.auth.TokenUnavailableException; +import java.net.SocketException; +import java.net.SocketTimeoutException; +import java.net.UnknownHostException; import java.time.Duration; import java.time.OffsetDateTime; import java.util.concurrent.TimeUnit; @@ -50,16 +54,20 @@ * RefreshingTokenProvider tokens = RefreshingTokenProvider.builder( * new AzureTokenSource(credential, "api://")).build(); * } - * or let a connect string select {@code token_provider=azure}, which uses {@code DefaultAzureCredential}. + * or let a connect string select {@code token_provider=azure}, whose {@code azure_credential} key picks the + * credential (see {@link AzureTokenProviderFactory}). *

* Expiry and the refresh hint come from the library's {@link AccessToken}. Failures are classified as section - * 7.5 prescribes: + * 7.5 prescribes, the same way whichever credential is used: *

    - *
  • permanent - "no credential available in the chain" ({@link CredentialUnavailableException}), and + *
  • permanent - no credential available ({@link CredentialUnavailableException}): a credential chain that + * found nothing, or a credential whose configuration is missing; a managed identity that is not assigned to + * this host, which a managed-identity credential used alone reports only in a nested cause; and * authentication failures Entra reports: an HTTP 400 or 401 from the token endpoint, or a known AADSTS * configuration error such as an invalid client or an application that does not exist;
    • *
  • retryable - network errors, timeouts, throttling, 5xx responses, and anything else. A - * {@code Retry-After} in seconds is passed on.
    • + * {@code Retry-After} in seconds is passed on, and the error names the socket-level cause of a network + * failure, such as a refused connection, which a managed identity otherwise hides behind a generic message. *
* Each attempt is bounded (30 s by default). Library exceptions never travel as a cause - their response * objects reference the raw HTTP request, which can carry a client secret - only their sanitized message does. @@ -70,6 +78,12 @@ public final class AzureTokenSource implements TokenSource { */ public static final Duration DEFAULT_TIMEOUT = Duration.ofSeconds(30); private static final String DEFAULT_SCOPE_SUFFIX = "/.default"; + // A managed identity that is not assigned to this host, or does not exist: IMDS answers 400 "Identity not + // found". Inside a chain that arrives as a CredentialUnavailableException, but a ManagedIdentityCredential used + // alone wraps it in a generic ClientAuthenticationException and keeps the reason only in the message of a + // nested MSAL exception, which has no status code. Without this rule a configuration error would be retried + // for a whole connect budget. Pinned against the real library by AzureManagedIdentityLibraryTest. + private static final String IDENTITY_NOT_ASSIGNED = "has not been assigned to this resource"; private static final int MAX_CAUSE_DEPTH = 16; // Entra (AADSTS) errors that a retry cannot fix: the configuration is wrong, an operator must act. private static final String[] PERMANENT_AADSTS = { @@ -85,6 +99,8 @@ public final class AzureTokenSource implements TokenSource { "AADSTS7000112", // application disabled "AADSTS50049", // unknown or invalid instance }; + // whether the credential is a chain, such as DefaultAzureCredential: only the wording of errors depends on it + private final boolean chain; private final TokenRequestContext context; private final TokenCredential credential; private final String scope; @@ -116,6 +132,7 @@ public AzureTokenSource(TokenCredential credential, String resource, Duration ti throw new IllegalArgumentException("timeout must be positive"); } this.credential = credential; + this.chain = credential instanceof ChainedTokenCredential; this.scope = resource.endsWith(DEFAULT_SCOPE_SUFFIX) ? resource : resource + DEFAULT_SCOPE_SUFFIX; this.context = new TokenRequestContext().addScopes(scope); this.timeout = timeout; @@ -124,13 +141,13 @@ public AzureTokenSource(TokenCredential credential, String resource, Duration ti /** * Classifies a failure from Azure Identity per section 7.5. Never attaches the library exception. */ - static TokenUnavailableException classify(Throwable failure, String scope) { + static TokenUnavailableException classify(Throwable failure, String scope, boolean chain) { final String description = describe(failure); Throwable t = failure; for (int depth = 0; t != null && depth < MAX_CAUSE_DEPTH; depth++, t = t.getCause()) { if (t instanceof CredentialUnavailableException) { - return TokenUnavailableException.permanent( - "no credential available in the Azure Identity chain for " + scope + ": " + description); + return TokenUnavailableException.permanent("no credential available " + + (chain ? "in the Azure Identity chain " : "") + "for " + scope + ": " + description); } if (t instanceof HttpResponseException) { final HttpResponse response = ((HttpResponseException) t).getResponse(); @@ -153,9 +170,15 @@ static TokenUnavailableException classify(Throwable failure, String scope) { "Entra rejected the token request for " + scope + " (" + code + "): " + description); } } + if (message.contains(IDENTITY_NOT_ASSIGNED)) { + // the library's reason, never its message: that also carries the raw response + return TokenUnavailableException.permanent("managed identity unavailable for " + scope + + ": the requested identity has not been assigned to this resource"); + } } } - return TokenUnavailableException.retryable("the token request for " + scope + " failed: " + description); + return TokenUnavailableException.retryable("the token request for " + scope + " failed: " + + withNetworkCause(failure, description)); } private static String describe(Throwable t) { @@ -163,6 +186,19 @@ private static String describe(Throwable t) { return message == null ? t.getClass().getName() : t.getClass().getSimpleName() + ": " + message; } + // The outermost socket-level failure among the causes: a refused or reset connection, an unreachable or + // unknown host, a socket timeout. Only these are named: their text is a host, a port and an OS error, while + // other library messages (MSAL's, a JSON parser's) can embed a raw response. + private static Throwable networkCause(Throwable failure) { + Throwable t = failure; + for (int depth = 0; t != null && depth < MAX_CAUSE_DEPTH; depth++, t = t.getCause()) { + if (t instanceof SocketException || t instanceof UnknownHostException || t instanceof SocketTimeoutException) { + return t; + } + } + return null; + } + private static long retryAfterMillis(HttpResponse response) { final String value; try { @@ -191,13 +227,27 @@ private static long refreshAtMillis(AccessToken token) { } } + // Leads with the socket-level cause, unless the description already carries it: error text is capped, and a + // library's own message is often generic ("see inner exception"). + private static String withNetworkCause(Throwable failure, String description) { + final Throwable network = networkCause(failure); + if (network == null) { + return description; + } + final String reason = network.getMessage(); + if (reason != null && description.contains(reason)) { + return description; + } + return describe(network) + "; " + description; + } + @Override public ExpiringToken fetchToken() { final AccessToken token; try { token = credential.getToken(context).block(timeout); } catch (RuntimeException e) { - throw classify(e, scope); + throw classify(e, scope, chain); } if (token == null) { throw TokenUnavailableException.retryable("Azure Identity returned no token for " + scope); diff --git a/azure/src/test/java/io/questdb/client/azure/test/AzureCredentialProbeMain.java b/azure/src/test/java/io/questdb/client/azure/test/AzureCredentialProbeMain.java new file mode 100644 index 000000000..6ddb84bd9 --- /dev/null +++ b/azure/src/test/java/io/questdb/client/azure/test/AzureCredentialProbeMain.java @@ -0,0 +1,101 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.azure.test; + +import ch.qos.logback.classic.Level; +import ch.qos.logback.classic.Logger; +import ch.qos.logback.classic.spi.ILoggingEvent; +import ch.qos.logback.core.read.ListAppender; +import com.azure.identity.ManagedIdentityCredentialBuilder; +import io.questdb.client.azure.AzureTokenProviderFactory; +import io.questdb.client.azure.AzureTokenSource; +import io.questdb.client.cutlass.auth.TokenProviderSpec; +import io.questdb.client.cutlass.auth.TokenSource; +import io.questdb.client.cutlass.auth.TokenUnavailableException; +import org.slf4j.LoggerFactory; + +import java.time.Duration; +import java.util.HashMap; +import java.util.Map; + +/** + * The child process of {@link AzureManagedIdentityLibraryTest}: fetches one token through the {@code azure} + * provider, with the managed-identity endpoint its environment names, and reports on stdout, one + * {@code PROBE key=value} line per fact. + *
    + *
  • {@code factory } - the source {@link AzureTokenProviderFactory} creates for that value + * ({@code default} passes no key);
    • + *
  • {@code direct } - a {@code ManagedIdentityCredential} behind an {@link AzureTokenSource} + * whose bound outlasts the library's own retries.
    • + *
+ */ +public final class AzureCredentialProbeMain { + static final String RESOURCE = "api://questdb-probe"; + + public static void main(String[] args) { + final Logger factoryLogger = (Logger) LoggerFactory.getLogger(AzureTokenProviderFactory.class); + final ListAppender events = new ListAppender<>(); + events.start(); + factoryLogger.addAppender(events); + + final TokenSource source; + if ("factory".equals(args[0])) { + final Map params = new HashMap<>(); + params.put(TokenProviderSpec.KEY_AZURE_RESOURCE, RESOURCE); + if (!"default".equals(args[1])) { + params.put(TokenProviderSpec.KEY_AZURE_CREDENTIAL, args[1]); + } + source = new AzureTokenProviderFactory().createSource(params); + } else { + source = new AzureTokenSource(new ManagedIdentityCredentialBuilder().build(), RESOURCE, + Duration.ofMillis(Long.parseLong(args[1]))); + } + report("fetching"); // the parent times an endpoint outage from here + final long start = System.nanoTime(); + try { + source.fetchToken(); + report("token=true"); + } catch (TokenUnavailableException e) { + report("token=false"); + report("retryable=" + e.isRetryable()); + report("message=" + String.valueOf(e.getMessage()).replace('\r', ' ').replace('\n', ' ')); + } + report("elapsedMs=" + (System.nanoTime() - start) / 1_000_000); + int warnings = 0; + for (ILoggingEvent event : events.list) { + if (event.getLevel() == Level.WARN) { + warnings++; + } + } + report("factoryWarnings=" + warnings); + // Azure Identity leaves non-daemon threads behind + System.exit(0); + } + + private static void report(String fact) { + System.out.println("PROBE " + fact); + System.out.flush(); + } +} diff --git a/azure/src/test/java/io/questdb/client/azure/test/AzureCredentialSelectionTest.java b/azure/src/test/java/io/questdb/client/azure/test/AzureCredentialSelectionTest.java new file mode 100644 index 000000000..edcd8e93f --- /dev/null +++ b/azure/src/test/java/io/questdb/client/azure/test/AzureCredentialSelectionTest.java @@ -0,0 +1,193 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.azure.test; + +import ch.qos.logback.classic.Level; +import ch.qos.logback.classic.Logger; +import ch.qos.logback.classic.spi.ILoggingEvent; +import ch.qos.logback.core.read.ListAppender; +import io.questdb.client.azure.AzureTokenProviderFactory; +import io.questdb.client.cutlass.auth.TokenProviderSpec; +import io.questdb.client.cutlass.auth.TokenSource; +import io.questdb.client.cutlass.auth.TokenUnavailableException; +import io.questdb.client.impl.ConfigString; +import io.questdb.client.impl.ConfigView; +import org.junit.Assert; +import org.junit.Assume; +import org.junit.Test; +import org.slf4j.LoggerFactory; + +import java.util.HashMap; +import java.util.Map; + +/** + * {@code azure_credential} with the real factory, in process and without network I/O (design/qwp-token-provider-spec.md, + * sections 7.1, 7.2 and 7.5, conformance test C24): what a connect string selects, which credential the factory + * builds, and the warning a discovery chain logs when its provider starts. The library's behaviour behind each + * credential is covered by {@link AzureManagedIdentityLibraryTest}. + */ +public class AzureCredentialSelectionTest { + private static final String CLIENT_ID = "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee"; + private static final String WSS = "wss::addr=localhost:9000;token_provider=azure;azure_resource=api://qdb;"; + + @Test + public void testConnectStringSelectsTheCredential() { + TokenProviderSpec managed = spec("azure_credential=managed_identity;azure_client_id=" + CLIENT_ID.toUpperCase() + ';'); + Assert.assertEquals("managed_identity", managed.params().get(TokenProviderSpec.KEY_AZURE_CREDENTIAL)); + Assert.assertEquals("azure[resource=api://qdb, credential=managed_identity, client_id=" + CLIENT_ID + ']', + managed.describe()); + + // default is the default: no parameter, so it shares a provider with a string that omits the key + TokenProviderSpec omitted = spec(""); + TokenProviderSpec explicit = spec("azure_credential=default;"); + Assert.assertNull(explicit.params().get(TokenProviderSpec.KEY_AZURE_CREDENTIAL)); + Assert.assertEquals(omitted.registryKey(), explicit.registryKey()); + Assert.assertEquals("azure[resource=api://qdb]", explicit.describe()); + // different credentials never share a provider + Assert.assertNotEquals(omitted.registryKey(), spec("azure_credential=managed_identity;").registryKey()); + Assert.assertNotEquals(spec("azure_credential=workload_identity;").registryKey(), + spec("azure_credential=managed_identity;").registryKey()); + + assertRejected("azure_credential=bogus;", + "invalid azure_credential: bogus (expected default, managed_identity, workload_identity, environment)"); + assertRejected("azure_credential=environment;azure_client_id=" + CLIENT_ID + ';', + "azure_client_id cannot be combined with azure_credential=environment"); + } + + @Test + public void testDiscoveryChainWarnsOnceWhenTheProviderStarts() { + // the platform's selector would narrow the chain and silence the warning (AzureManagedIdentityLibraryTest) + Assume.assumeTrue(System.getenv("AZURE_TOKEN_CREDENTIALS") == null); + Logger logger = (Logger) LoggerFactory.getLogger(AzureTokenProviderFactory.class); + ListAppender events = new ListAppender<>(); + events.start(); + logger.addAppender(events); + try { + source(null, CLIENT_ID); // creating the source is starting the provider; it fetches nothing + Assert.assertEquals(1, warnings(events)); + String warning = firstWarning(events); + Assert.assertTrue(warning, warning.contains("azure[resource=api://qdb, client_id=" + CLIENT_ID + ']')); + Assert.assertTrue(warning, warning.contains("DefaultAzureCredential")); + Assert.assertTrue(warning, warning.contains("azure_credential=managed_identity")); + + // a credential selected deterministically is not a chain + source("managed_identity", CLIENT_ID); + source("environment", null); + Assert.assertEquals(1, warnings(events)); + } finally { + logger.detachAppender(events); + } + } + + @Test + public void testFactoryBuildsTheSelectedCredentialWithoutNetworkIo() { + Assert.assertTrue(source(null, null).toString().contains("credential=DefaultAzureCredential")); + Assert.assertTrue(source("managed_identity", null).toString().contains("credential=ManagedIdentityCredential")); + Assert.assertTrue(source("managed_identity", CLIENT_ID).toString().contains("credential=ManagedIdentityCredential")); + Assert.assertTrue(source("environment", null).toString().contains("credential=EnvironmentCredential")); + } + + @Test + public void testFactoryRejectsParametersItCannotUse() { + AzureTokenProviderFactory factory = new AzureTokenProviderFactory(); + Map bogus = params("bogus", null); + assertRejected(() -> factory.validate(bogus), "invalid azure_credential: bogus"); + assertRejected(() -> factory.createSource(bogus), "invalid azure_credential: bogus"); + Map environmentWithClientId = params("environment", CLIENT_ID); + assertRejected(() -> factory.validate(environmentWithClientId), + "azure_client_id cannot be combined with azure_credential=environment"); + assertRejected(() -> factory.createSource(environmentWithClientId), + "azure_client_id cannot be combined with azure_credential=environment"); + factory.validate(params("workload_identity", CLIENT_ID)); + } + + @Test + public void testUnconfiguredWorkloadIdentityFailsPermanentlyWhenFetched() { + // Azure Identity refuses to build a workload identity without its tenant ID and federated token file. That + // must not escape from a client's build(): it is a permanent token failure, like any missing configuration. + Assume.assumeTrue(System.getenv("AZURE_TENANT_ID") == null || System.getenv("AZURE_FEDERATED_TOKEN_FILE") == null); + TokenSource source = source("workload_identity", CLIENT_ID); + try { + source.fetchToken(); + Assert.fail("expected the fetch to fail"); + } catch (TokenUnavailableException e) { + Assert.assertFalse("missing configuration is permanent: " + e.getMessage(), e.isRetryable()); + Assert.assertTrue(e.getMessage(), e.getMessage().startsWith("no credential available for api://qdb/.default: ")); + Assert.assertTrue(e.getMessage(), e.getMessage().contains("WorkloadIdentityCredentialBuilder")); + } + } + + private static void assertRejected(String keys, String fragment) { + assertRejected(() -> spec(keys), fragment); + } + + private static void assertRejected(Runnable action, String fragment) { + try { + action.run(); + Assert.fail("expected rejection with: " + fragment); + } catch (IllegalArgumentException e) { + Assert.assertTrue("[" + e.getMessage() + "] does not contain [" + fragment + ']', e.getMessage().contains(fragment)); + } + } + + private static String firstWarning(ListAppender events) { + for (ILoggingEvent event : events.list) { + if (event.getLevel() == Level.WARN) { + return event.getFormattedMessage(); + } + } + return null; + } + + private static Map params(String credential, String clientId) { + Map params = new HashMap<>(); + params.put(TokenProviderSpec.KEY_AZURE_RESOURCE, "api://qdb"); + if (credential != null) { + params.put(TokenProviderSpec.KEY_AZURE_CREDENTIAL, credential); + } + if (clientId != null) { + params.put(TokenProviderSpec.KEY_AZURE_CLIENT_ID, clientId); + } + return params; + } + + private static TokenSource source(String credential, String clientId) { + return new AzureTokenProviderFactory().createSource(params(credential, clientId)); + } + + private static TokenProviderSpec spec(String keys) { + return TokenProviderSpec.parse(new ConfigView(ConfigString.parse(WSS + keys)), true); + } + + private static long warnings(ListAppender events) { + long n = 0; + for (ILoggingEvent event : events.list) { + if (event.getLevel() == Level.WARN) { + n++; + } + } + return n; + } +} diff --git a/azure/src/test/java/io/questdb/client/azure/test/AzureManagedIdentityLibraryTest.java b/azure/src/test/java/io/questdb/client/azure/test/AzureManagedIdentityLibraryTest.java new file mode 100644 index 000000000..455e8e05b --- /dev/null +++ b/azure/src/test/java/io/questdb/client/azure/test/AzureManagedIdentityLibraryTest.java @@ -0,0 +1,249 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.azure.test; + +import org.junit.Assert; +import org.junit.Test; + +import java.io.BufferedReader; +import java.io.File; +import java.io.IOException; +import java.io.InputStreamReader; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; +import java.util.Map; +import java.util.concurrent.TimeUnit; + +/** + * Conformance test C24 (design/qwp-token-provider-spec.md, section 10) against the real Azure Identity library: how + * each {@code azure_credential} mode reports a managed-identity endpoint that is down, and one that answers that + * the identity is not assigned. + *

+ * Each case runs in a child JVM ({@link AzureCredentialProbeMain}): MSAL takes the endpoint only from the process + * environment, {@code AZURE_POD_IDENTITY_AUTHORITY_HOST}, so that is the only way to point it at a local + * {@link ImdsStub} - and never at the real instance metadata service, which a CI runner on an Azure VM can reach. + * The child inherits no other Azure Identity variable. + */ +public class AzureManagedIdentityLibraryTest { + private static final long CHILD_TIMEOUT_MILLIS = 90_000; + private static final long NEVER = -1; + + @Test(timeout = 120_000) + public void testDiscoveryChainFailsFastAndWarnsOnce() throws Exception { + // AZURE_TOKEN_CREDENTIALS=prod keeps developer tools out of the chain: environment, workload identity and + // managed identity remain. The chain checks the endpoint once, without retrying, so a host whose endpoint + // is down looks like one without a credential: permanent, as section 7.5 classifies it, and fast. + try (ImdsStub stub = new ImdsStub(ImdsStub.Answer.TOKEN)) { + Result r = run(stub, NEVER, "prod", "factory", "default"); + Assert.assertFalse(r.output, r.token); + Assert.assertFalse(r.output, r.retryable); + Assert.assertTrue(r.output, r.message.startsWith("no credential available in the Azure Identity chain")); + Assert.assertTrue("the chain must not retry the endpoint: " + r.output, r.elapsedMillis < 10_000); + Assert.assertEquals(r.output, 1, r.factoryWarnings); + } + } + + @Test(timeout = 120_000) + public void testManagedIdentityReportsAnUnassignedIdentityAsPermanent() throws Exception { + // a configuration error: a sync startup must fail fast, not retry it for the whole connect budget + try (ImdsStub stub = new ImdsStub(ImdsStub.Answer.IDENTITY_NOT_FOUND)) { + Result r = run(stub, 0, null, "factory", "managed_identity"); + Assert.assertFalse(r.output, r.token); + Assert.assertFalse(r.output, r.retryable); + Assert.assertEquals(r.output, "managed identity unavailable for " + AzureCredentialProbeMain.RESOURCE + + "/.default: the requested identity has not been assigned to this resource", r.message); + Assert.assertTrue("the stub must have answered: " + r.output, stub.tokenRequests() >= 1); + Assert.assertEquals(r.output, 0, r.factoryWarnings); + } + } + + @Test(timeout = 120_000) + public void testManagedIdentityReportsAnUnreachableEndpointAsRetryable() throws Exception { + // The endpoint never comes up. The library retries it with backoff for about 25 s and then gives up with a + // generic error that is retryable, naming the refused connection. The source's bound outlasts those + // retries, so this is the library's verdict, not the source's own timeout. + try (ImdsStub stub = new ImdsStub(ImdsStub.Answer.TOKEN)) { + Result r = run(stub, NEVER, null, "direct", "75000"); + Assert.assertFalse(r.output, r.token); + Assert.assertTrue(r.output, r.retryable); + Assert.assertTrue(r.output, r.message.contains("ConnectException")); + Assert.assertEquals(r.output, 0, stub.tokenRequests()); + } + } + + @Test(timeout = 120_000) + public void testManagedIdentityRidesOutAnEndpointOutage() throws Exception { + // the endpoint comes up 2 s after the fetch starts: the library's own retries reach it within that fetch + try (ImdsStub stub = new ImdsStub(ImdsStub.Answer.TOKEN)) { + Result r = run(stub, 2_000, null, "factory", "managed_identity"); + Assert.assertTrue(r.output, r.token); + Assert.assertTrue("the fetch must have outlasted the outage: " + r.output, r.elapsedMillis >= 1_500); + Assert.assertEquals(r.output, 1, stub.tokenRequests()); + Assert.assertEquals(r.output, 0, r.factoryWarnings); + } + } + + @Test(timeout = 120_000) + public void testPlatformSelectorNarrowsTheDefaultAndSilencesTheWarning() throws Exception { + // azure_credential=default honours Azure Identity's own selector: one credential, not a chain to warn about + try (ImdsStub stub = new ImdsStub(ImdsStub.Answer.TOKEN)) { + Result r = run(stub, 0, "ManagedIdentityCredential", "factory", "default"); + Assert.assertTrue(r.output, r.token); + Assert.assertEquals(r.output, 0, r.factoryWarnings); + } + } + + private static void deleteRecursively(File file) { + File[] children = file.listFiles(); + if (children != null) { + for (File child : children) { + deleteRecursively(child); + } + } + //noinspection ResultOfMethodCallIgnored + file.delete(); + } + + /** + * Runs {@link AzureCredentialProbeMain} against {@code stub}, which starts answering {@code upAfterMillis} after + * the child starts fetching: 0 before the child starts, {@link #NEVER} not at all. + */ + private static Result run(ImdsStub stub, long upAfterMillis, String selector, String... args) throws Exception { + if (upAfterMillis == 0) { + stub.start(); + } + final File home = Files.createTempDirectory("qdb-azure-probe").toFile(); + try { + final List command = new ArrayList<>(); + command.add(System.getProperty("java.home") + File.separator + "bin" + File.separator + "java"); + // no developer-tool profile (Azure CLI, IDEs) can be found under an empty home + command.add("-Duser.home=" + home.getAbsolutePath()); + command.add("-cp"); + command.add(System.getProperty("java.class.path")); + command.add(AzureCredentialProbeMain.class.getName()); + Collections.addAll(command, args); + final ProcessBuilder pb = new ProcessBuilder(command).redirectErrorStream(true); + final Map env = pb.environment(); + env.keySet().removeIf(key -> key.startsWith("AZURE_") || key.startsWith("IDENTITY_") + || key.startsWith("MSI_") || key.startsWith("IMDS_")); + env.put("AZURE_POD_IDENTITY_AUTHORITY_HOST", stub.endpoint()); + env.put("AZURE_CONFIG_DIR", new File(home, ".azure").getAbsolutePath()); + if (selector != null) { + env.put("AZURE_TOKEN_CREDENTIALS", selector); + } + final Process process = pb.start(); + final StringBuilder output = new StringBuilder(); + final Thread reader = new Thread(() -> { + try (BufferedReader in = new BufferedReader( + new InputStreamReader(process.getInputStream(), StandardCharsets.UTF_8))) { + String line; + while ((line = in.readLine()) != null) { + synchronized (output) { + output.append(line).append('\n'); + } + if (upAfterMillis > 0 && line.equals("PROBE fetching")) { + startLater(stub, upAfterMillis); + } + } + } catch (IOException ignore) { + // the child is gone + } + }, "probe-output"); + reader.setDaemon(true); + reader.start(); + if (!process.waitFor(CHILD_TIMEOUT_MILLIS, TimeUnit.MILLISECONDS)) { + process.destroyForcibly(); + reader.join(5_000); + Assert.fail("the probe did not finish in " + CHILD_TIMEOUT_MILLIS + " ms:\n" + output); + } + reader.join(10_000); + final String text; + synchronized (output) { + text = output.toString(); + } + Assert.assertEquals("the probe failed:\n" + text, 0, process.exitValue()); + return new Result(text); + } finally { + deleteRecursively(home); + } + } + + private static void startLater(ImdsStub stub, long delayMillis) { + final Thread starter = new Thread(() -> { + try { + Thread.sleep(delayMillis); + stub.start(); + } catch (InterruptedException | IOException ignore) { + // the test is over + } + }, "imds-stub-starter"); + starter.setDaemon(true); + starter.start(); + } + + private static final class Result { + final long elapsedMillis; + final int factoryWarnings; + final String message; + final String output; + final boolean retryable; + final boolean token; + + Result(String output) { + this.output = output; + String message = null; + long elapsed = -1; + int warnings = -1; + boolean retryable = false; + boolean token = false; + for (String line : output.split("\n")) { + if (!line.startsWith("PROBE ")) { + continue; + } + final String fact = line.substring("PROBE ".length()); + if (fact.startsWith("message=")) { + message = fact.substring("message=".length()); + } else if (fact.startsWith("elapsedMs=")) { + elapsed = Long.parseLong(fact.substring("elapsedMs=".length())); + } else if (fact.startsWith("factoryWarnings=")) { + warnings = Integer.parseInt(fact.substring("factoryWarnings=".length())); + } else if (fact.startsWith("retryable=")) { + retryable = Boolean.parseBoolean(fact.substring("retryable=".length())); + } else if (fact.startsWith("token=")) { + token = Boolean.parseBoolean(fact.substring("token=".length())); + } + } + Assert.assertTrue("the probe reported no result:\n" + output, elapsed >= 0 && warnings >= 0); + this.message = message == null ? "" : message; + this.elapsedMillis = elapsed; + this.factoryWarnings = warnings; + this.retryable = retryable; + this.token = token; + } + } +} diff --git a/azure/src/test/java/io/questdb/client/azure/test/AzureTokenSourceTest.java b/azure/src/test/java/io/questdb/client/azure/test/AzureTokenSourceTest.java index d8d6b470b..5d2193b3c 100644 --- a/azure/src/test/java/io/questdb/client/azure/test/AzureTokenSourceTest.java +++ b/azure/src/test/java/io/questdb/client/azure/test/AzureTokenSourceTest.java @@ -32,7 +32,10 @@ import com.azure.core.http.HttpHeaders; import com.azure.core.http.HttpResponse; import com.azure.identity.AuthenticationRequiredException; +import com.azure.identity.ChainedTokenCredentialBuilder; import com.azure.identity.CredentialUnavailableException; +import com.microsoft.aad.msal4j.MsalClientException; +import com.microsoft.aad.msal4j.MsalServiceException; import io.questdb.client.azure.AzureTokenProviderFactory; import io.questdb.client.azure.AzureTokenSource; import io.questdb.client.cutlass.auth.ExpiringToken; @@ -49,6 +52,10 @@ import reactor.core.publisher.Flux; import reactor.core.publisher.Mono; +import java.io.IOException; +import java.io.UncheckedIOException; +import java.net.ConnectException; +import java.net.UnknownHostException; import java.nio.ByteBuffer; import java.nio.charset.Charset; import java.time.Duration; @@ -116,6 +123,20 @@ public void testCredentialUnavailableIsPermanent() { "no credential available"); } + @Test + public void testCredentialUnavailableNamesTheChainOnlyForAChain() { + // DefaultAzureCredential is a chain: its wording is unchanged. A credential used alone is not one. + TokenCredential unavailable = ctx -> Mono.error(new CredentialUnavailableException("nothing configured")); + TokenUnavailableException e = expectFailure(new AzureTokenSource( + new ChainedTokenCredentialBuilder().addLast(unavailable).build(), "api://qdb")); + Assert.assertFalse(e.isRetryable()); + Assert.assertTrue(e.getMessage(), e.getMessage().startsWith( + "no credential available in the Azure Identity chain for api://qdb/.default: ")); + e = expectFailure(new AzureTokenSource(unavailable, "api://qdb")); + Assert.assertFalse(e.isRetryable()); + Assert.assertTrue(e.getMessage(), e.getMessage().startsWith("no credential available for api://qdb/.default: ")); + } + @Test public void testExpiryAndRefreshHintAreMapped() { AzureTokenSource source = new AzureTokenSource( @@ -155,6 +176,41 @@ public void testLibraryExceptionsAreNeverAttached() { Assert.assertNull(e.getCause()); } + @Test + public void testNetworkCauseLeadsTheMessage() { + // A managed identity used alone reports an unreachable endpoint as "authentication failed, see inner + // exception" - the cause chain the real library produces for a refused connection. The socket-level cause + // comes first, so it survives the 256-character cap on error text. + TokenUnavailableException e = fetchFailure(new ClientAuthenticationException( + "Managed Identity authentication failed, see inner exception for more information.", null, + new MsalClientException(new UncheckedIOException( + new ConnectException("finishConnect(..) failed with error(-111): Connection refused"))))); + Assert.assertTrue(e.isRetryable()); + Assert.assertTrue(e.getMessage(), e.getMessage().startsWith("the token request for api://qdb/.default failed: " + + "ConnectException: finishConnect(..) failed with error(-111): Connection refused; " + + "ClientAuthenticationException: Managed Identity authentication failed")); + + e = fetchFailure(new RuntimeException("wrapped", new UnknownHostException("login.microsoftonline.com"))); + Assert.assertTrue(e.isRetryable()); + Assert.assertTrue(e.getMessage(), e.getMessage().contains("UnknownHostException: login.microsoftonline.com")); + + // a wrapper whose own message already carries the cause - as Reactor's wrapper of a checked exception does - + // names it once + e = fetchFailure(new UncheckedIOException(new ConnectException("Connection refused"))); + Assert.assertEquals("the token request for api://qdb/.default failed: UncheckedIOException: " + + "java.net.ConnectException: Connection refused", e.getMessage()); + } + + @Test + public void testOnlySocketLevelCausesAreNamed() { + // Other nested messages - MSAL's, a JSON parser's - can embed a raw response, which may carry a token. + TokenUnavailableException e = fetchFailure(new ClientAuthenticationException("authentication failed", null, + new IOException("Unexpected character at [Source: {\"access_token\":\"eyJsecret\"}]"))); + Assert.assertTrue(e.isRetryable()); + Assert.assertFalse("a nested non-socket message must not be echoed: " + e.getMessage(), + e.getMessage().contains("eyJsecret")); + } + @Test public void testOtherFailuresAreRetryable() { assertRetryable(new HttpResponseException("throttled", new FakeResponse(429, "7")), "HTTP 429", 7_000); @@ -200,6 +256,24 @@ public void testSlowCredentialIsBoundedAndRetryable() { Assert.assertTrue("the attempt must be bounded", System.nanoTime() - start < 10_000_000_000L); } + @Test + public void testUnassignedManagedIdentityIsPermanent() { + // A managed identity used alone (azure_credential=managed_identity) reports IMDS's 400 "Identity not found" + // only in the message of a nested MSAL exception, with no status code: the cause chain the real library + // produces (pinned by AzureManagedIdentityLibraryTest). Left retryable, a sync startup would retry a + // configuration error for its whole connect budget. + TokenUnavailableException e = fetchFailure(new ClientAuthenticationException( + "Managed Identity authentication failed, see inner exception for more information.", null, + new MsalServiceException("[Managed Identity] Authentication unavailable. The requested identity has not " + + "been assigned to this resource.Status: 400Content:{\"error\":\"invalid_request\"," + + "\"error_description\":\"Identity not found\"}Headers:Connection[close]", + "managed_identity_request_failed"))); + Assert.assertFalse("expected permanent: " + e.getMessage(), e.isRetryable()); + Assert.assertEquals("managed identity unavailable for api://qdb/.default: the requested identity has not been " + + "assigned to this resource", e.getMessage()); + Assert.assertNull(e.getCause()); + } + @Test(timeout = 30_000) public void testWorksBehindTheRefreshingProvider() { OffsetDateTime expires = OffsetDateTime.now(ZoneOffset.UTC).plusHours(1); diff --git a/azure/src/test/java/io/questdb/client/azure/test/ImdsStub.java b/azure/src/test/java/io/questdb/client/azure/test/ImdsStub.java new file mode 100644 index 000000000..20ac4c661 --- /dev/null +++ b/azure/src/test/java/io/questdb/client/azure/test/ImdsStub.java @@ -0,0 +1,201 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.azure.test; + +import java.io.ByteArrayOutputStream; +import java.io.Closeable; +import java.io.IOException; +import java.io.InputStream; +import java.io.OutputStream; +import java.io.UnsupportedEncodingException; +import java.net.InetAddress; +import java.net.InetSocketAddress; +import java.net.ServerSocket; +import java.net.Socket; +import java.net.SocketTimeoutException; +import java.net.URLDecoder; +import java.nio.charset.StandardCharsets; +import java.util.concurrent.atomic.AtomicInteger; + +/** + * A local stand-in for the Azure instance metadata service (IMDS), the managed-identity endpoint. Azure Identity + * is pointed at it with {@code AZURE_POD_IDENTITY_AUTHORITY_HOST}. Until {@link #start()} its port refuses + * connections, like an unreachable endpoint. A raw socket server: the JDK's HTTP server rejects the + * {@code //metadata/...} request target MSAL sends. + */ +final class ImdsStub implements Closeable { + private static final InetAddress LOOPBACK = loopback(); + private final Answer answer; + private final int port; + private final AtomicInteger tokenRequests = new AtomicInteger(); + private boolean closed; // guarded by this + private ServerSocket server; // guarded by this + + ImdsStub(Answer answer) throws IOException { + this.answer = answer; + try (ServerSocket probe = new ServerSocket(0, 50, LOOPBACK)) { + this.port = probe.getLocalPort(); + } + } + + @Override + public synchronized void close() throws IOException { + closed = true; + if (server != null) { + server.close(); + } + } + + /** + * The endpoint for {@code AZURE_POD_IDENTITY_AUTHORITY_HOST}. + */ + String endpoint() { + return "http://127.0.0.1:" + port; + } + + /** + * Starts answering. Idempotent; does nothing once closed. + */ + synchronized void start() throws IOException { + if (closed || server != null) { + return; + } + final ServerSocket s = new ServerSocket(); + s.setReuseAddress(true); + s.bind(new InetSocketAddress(LOOPBACK, port), 50); + server = s; + final Thread acceptor = new Thread(() -> accept(s), "imds-stub"); + acceptor.setDaemon(true); + acceptor.start(); + } + + /** + * The token requests answered so far; a connection that asked nothing is not one. + */ + int tokenRequests() { + return tokenRequests.get(); + } + + private static InetAddress loopback() { + try { + return InetAddress.getByName("127.0.0.1"); + } catch (IOException e) { + throw new IllegalStateException(e); + } + } + + private static String readHead(InputStream in) throws IOException { + final ByteArrayOutputStream head = new ByteArrayOutputStream(); + final byte[] buf = new byte[4096]; + try { + int n; + while ((n = in.read(buf)) > 0) { + head.write(buf, 0, n); + if (new String(head.toByteArray(), StandardCharsets.ISO_8859_1).contains("\r\n\r\n")) { + break; + } + } + } catch (SocketTimeoutException ignore) { + // a connection that asks nothing + } + return new String(head.toByteArray(), StandardCharsets.ISO_8859_1); + } + + // The resource parameter of the request line, echoed in the token response. + private static String resourceOf(String requestLine) throws UnsupportedEncodingException { + final int at = requestLine.indexOf("resource="); + if (at < 0) { + return "unknown"; + } + String value = requestLine.substring(at + "resource=".length()); + for (char end : new char[]{'&', ' '}) { + final int i = value.indexOf(end); + if (i >= 0) { + value = value.substring(0, i); + } + } + return URLDecoder.decode(value, "UTF-8"); + } + + private void accept(ServerSocket s) { + while (!s.isClosed()) { + final Socket socket; + try { + socket = s.accept(); + } catch (IOException e) { + return; + } + final Thread handler = new Thread(() -> answer(socket), "imds-stub-connection"); + handler.setDaemon(true); + handler.start(); + } + } + + private void answer(Socket socket) { + try (Socket sock = socket) { + sock.setSoTimeout(5_000); + final String head = readHead(sock.getInputStream()); + if (head.isEmpty()) { + return; + } + tokenRequests.incrementAndGet(); + final int eol = head.indexOf('\r'); + final String requestLine = eol < 0 ? head : head.substring(0, eol); + final int status; + final String body; + if (answer == Answer.TOKEN) { + final long now = System.currentTimeMillis() / 1000; + status = 200; + body = "{\"access_token\":\"stub-imds-token\",\"client_id\":\"00000000-0000-0000-0000-000000000001\"," + + "\"expires_in\":\"3599\",\"expires_on\":\"" + (now + 3599) + "\",\"ext_expires_in\":\"3599\"," + + "\"not_before\":\"" + now + "\",\"resource\":\"" + resourceOf(requestLine) + "\"," + + "\"token_type\":\"Bearer\"}"; + } else { + status = 400; + body = "{\"error\":\"invalid_request\",\"error_description\":\"Identity not found\"}"; + } + final byte[] bytes = body.getBytes(StandardCharsets.UTF_8); + final OutputStream out = sock.getOutputStream(); + out.write(("HTTP/1.1 " + status + (status == 200 ? " OK" : " Bad Request") + + "\r\nContent-Type: application/json; charset=utf-8\r\nContent-Length: " + bytes.length + + "\r\nConnection: close\r\n\r\n").getBytes(StandardCharsets.ISO_8859_1)); + out.write(bytes); + out.flush(); + } catch (IOException ignore) { + // the client went away + } + } + + enum Answer { + /** + * A token for the requested resource. + */ + TOKEN, + /** + * HTTP 400 "Identity not found": the identity is not assigned to this host. + */ + IDENTITY_NOT_FOUND + } +} diff --git a/core/src/main/java/io/questdb/client/Sender.java b/core/src/main/java/io/questdb/client/Sender.java index 57ff7bf59..acef90d80 100644 --- a/core/src/main/java/io/questdb/client/Sender.java +++ b/core/src/main/java/io/questdb/client/Sender.java @@ -4063,7 +4063,8 @@ private LineSenderBuilder fromConfig(CharSequence configurationString) { throw new LineSenderException("auth_failure_max_duration_millis is only supported for WebSocket transport"); } else if (Chars.equals("token_provider", sink) || Chars.equals("azure_resource", sink) - || Chars.equals("azure_client_id", sink)) { + || Chars.equals("azure_client_id", sink) + || Chars.equals("azure_credential", sink)) { // Dynamic bearer credentials are defined for QWP over wss:: only (decision D9). throw new LineSenderException(Chars.toString(sink) + " is only supported with the wss:: schema (QWP over WebSocket)"); @@ -4448,6 +4449,8 @@ public java.util.Map wsConfigSnapshotForTest() { : tokenProviderSpec.params().get(TokenProviderSpec.KEY_AZURE_RESOURCE)); m.put("azure_client_id", tokenProviderSpec == null ? null : tokenProviderSpec.params().get(TokenProviderSpec.KEY_AZURE_CLIENT_ID)); + m.put("azure_credential", tokenProviderSpec == null ? null + : tokenProviderSpec.params().get(TokenProviderSpec.KEY_AZURE_CREDENTIAL)); return m; } diff --git a/core/src/main/java/io/questdb/client/cutlass/auth/TokenProviderFactory.java b/core/src/main/java/io/questdb/client/cutlass/auth/TokenProviderFactory.java index 2e96e1fd0..4681ab33a 100644 --- a/core/src/main/java/io/questdb/client/cutlass/auth/TokenProviderFactory.java +++ b/core/src/main/java/io/questdb/client/cutlass/auth/TokenProviderFactory.java @@ -39,9 +39,9 @@ *

* The connect-string vocabulary is fixed - unknown keys are rejected before a factory ever sees them - so a * factory receives only the keys the client defines for it, already normalized: for {@code azure}, the - * {@code azure_resource} (with any trailing {@code /.default} removed) and the optional, lower-cased - * {@code azure_client_id}. Values in these keys are never secret: secrets come from the platform's standard - * configuration, never from the connect string. + * {@code azure_resource} (with any trailing {@code /.default} removed), the optional, lower-cased + * {@code azure_client_id}, and the optional {@code azure_credential} (absent for {@code default}). Values in these + * keys are never secret: secrets come from the platform's standard configuration, never from the connect string. */ public interface TokenProviderFactory { diff --git a/core/src/main/java/io/questdb/client/cutlass/auth/TokenProviderSpec.java b/core/src/main/java/io/questdb/client/cutlass/auth/TokenProviderSpec.java index 82e09019e..1c18a4470 100644 --- a/core/src/main/java/io/questdb/client/cutlass/auth/TokenProviderSpec.java +++ b/core/src/main/java/io/questdb/client/cutlass/auth/TokenProviderSpec.java @@ -41,6 +41,7 @@ * {@link TokenProviderRegistry#acquire(TokenProviderSpec)}. * 

  * wss::addr=qdb1:9000,qdb2:9000;token_provider=azure;azure_resource=api://<questdb-app-id>;azure_client_id=<uami-client-id>;
+ * wss::addr=qdb1:9000;token_provider=azure;azure_resource=api://<questdb-app-id>;azure_credential=managed_identity;
  *
*/ public final class TokenProviderSpec { @@ -49,14 +50,35 @@ public final class TokenProviderSpec { * {@code questdb-client-azure} module. */ public static final String AZURE = "azure"; + /** + * {@code azure_credential=environment}: Azure Identity's {@code EnvironmentCredential} alone. + */ + public static final String AZURE_CREDENTIAL_ENVIRONMENT = "environment"; + /** + * {@code azure_credential=managed_identity}: Azure Identity's {@code ManagedIdentityCredential} alone. + */ + public static final String AZURE_CREDENTIAL_MANAGED_IDENTITY = "managed_identity"; + /** + * {@code azure_credential=workload_identity}: Azure Identity's {@code WorkloadIdentityCredential} alone. + */ + public static final String AZURE_CREDENTIAL_WORKLOAD_IDENTITY = "workload_identity"; /** * Reserved for a zero-dependency Azure IMDS provider (decision D5); not implemented. */ public static final String AZURE_IMDS = "azure_imds"; public static final String AZURE_MODULE = "org.questdb:questdb-client-azure"; public static final String KEY_AZURE_CLIENT_ID = "azure_client_id"; + /** + * Which Azure credential {@code token_provider=azure} uses (section 7.1). Absent, or {@code default}, is Azure + * Identity's {@code DefaultAzureCredential}; {@code default} is normalized away, so it never reaches the + * factory and shares a provider with a string that omits the key. The other values select that one + * credential: {@link #AZURE_CREDENTIAL_MANAGED_IDENTITY}, {@link #AZURE_CREDENTIAL_WORKLOAD_IDENTITY} or + * {@link #AZURE_CREDENTIAL_ENVIRONMENT}. + */ + public static final String KEY_AZURE_CREDENTIAL = "azure_credential"; public static final String KEY_AZURE_RESOURCE = "azure_resource"; public static final String KEY_TOKEN_PROVIDER = "token_provider"; + private static final String AZURE_CREDENTIAL_DEFAULT = "default"; private static final String AZURE_DEFAULT_SCOPE_SUFFIX = "/.default"; private static final String[] STATIC_CREDENTIAL_KEYS = {"token", "username", "password"}; private final TokenProviderFactory factory; @@ -84,6 +106,8 @@ private TokenProviderSpec(TokenProviderFactory factory, String name, Mapa provider-specific key the selected provider does not accept, such as {@code azure_resource} * without {@code token_provider=azure}; *
  • a missing required provider key;
    • + *
  • an {@code azure_credential} value that section 7.1 does not list, listing the supported values, and + * {@code azure_client_id} combined with {@code azure_credential=environment};
    • *
  • {@code token_provider} on {@code ws::}: the bearer token would cross the network in cleartext * (decision D4).
    • * @@ -98,6 +122,7 @@ public static TokenProviderSpec parse(ConfigView view, boolean tls) { final String name = view.getStr(KEY_TOKEN_PROVIDER); final String resource = view.getStr(KEY_AZURE_RESOURCE); final String clientId = view.getStr(KEY_AZURE_CLIENT_ID); + final String credential = view.getStr(KEY_AZURE_CREDENTIAL); if (name == null) { if (resource != null) { throw new IllegalArgumentException(KEY_AZURE_RESOURCE + " requires " + KEY_TOKEN_PROVIDER + '=' + AZURE); @@ -105,6 +130,9 @@ public static TokenProviderSpec parse(ConfigView view, boolean tls) { if (clientId != null) { throw new IllegalArgumentException(KEY_AZURE_CLIENT_ID + " requires " + KEY_TOKEN_PROVIDER + '=' + AZURE); } + if (credential != null) { + throw new IllegalArgumentException(KEY_AZURE_CREDENTIAL + " requires " + KEY_TOKEN_PROVIDER + '=' + AZURE); + } return null; } for (String key : STATIC_CREDENTIAL_KEYS) { @@ -139,6 +167,9 @@ public static TokenProviderSpec parse(ConfigView view, boolean tls) { if (clientId != null) { throw new IllegalArgumentException(KEY_AZURE_CLIENT_ID + " is only valid with " + KEY_TOKEN_PROVIDER + '=' + AZURE); } + if (credential != null) { + throw new IllegalArgumentException(KEY_AZURE_CREDENTIAL + " is only valid with " + KEY_TOKEN_PROVIDER + '=' + AZURE); + } } final Map params = new TreeMap<>(); if (AZURE.equals(name)) { @@ -146,7 +177,17 @@ public static TokenProviderSpec parse(ConfigView view, boolean tls) { throw new IllegalArgumentException(KEY_TOKEN_PROVIDER + '=' + AZURE + " requires " + KEY_AZURE_RESOURCE); } params.put(KEY_AZURE_RESOURCE, normalizeAzureResource(resource)); + // checked against the values of section 7.1, and the error lists them; "default" adds no parameter + final String mode = view.getEnum(KEY_AZURE_CREDENTIAL); + if (mode != null && !AZURE_CREDENTIAL_DEFAULT.equals(mode)) { + params.put(KEY_AZURE_CREDENTIAL, mode); + } if (clientId != null) { + if (AZURE_CREDENTIAL_ENVIRONMENT.equals(mode)) { + throw new IllegalArgumentException(KEY_AZURE_CLIENT_ID + " cannot be combined with " + + KEY_AZURE_CREDENTIAL + '=' + AZURE_CREDENTIAL_ENVIRONMENT + + ": that credential takes its client ID from the platform's configuration (AZURE_CLIENT_ID)"); + } params.put(KEY_AZURE_CLIENT_ID, normalizeGuid(clientId)); } } diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpQueryClient.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpQueryClient.java index 380fc0eae..84d49b2f4 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpQueryClient.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpQueryClient.java @@ -1058,6 +1058,8 @@ public java.util.Map configSnapshotForTest() { : tokenProviderSpec.params().get(TokenProviderSpec.KEY_AZURE_RESOURCE)); m.put("azure_client_id", tokenProviderSpec == null ? null : tokenProviderSpec.params().get(TokenProviderSpec.KEY_AZURE_CLIENT_ID)); + m.put("azure_credential", tokenProviderSpec == null ? null + : tokenProviderSpec.params().get(TokenProviderSpec.KEY_AZURE_CREDENTIAL)); return m; } diff --git a/core/src/main/java/io/questdb/client/impl/ConfigSchema.java b/core/src/main/java/io/questdb/client/impl/ConfigSchema.java index 38eebb6fc..4922680b4 100644 --- a/core/src/main/java/io/questdb/client/impl/ConfigSchema.java +++ b/core/src/main/java/io/questdb/client/impl/ConfigSchema.java @@ -63,6 +63,7 @@ public final class ConfigSchema { str("token_provider", Side.COMMON); str("azure_resource", Side.COMMON); str("azure_client_id", Side.COMMON); + enumKey("azure_credential", Side.COMMON, "default", "managed_identity", "workload_identity", "environment"); // INGRESS -- the WebSocket Sender applies. STRING in the registry; the // Sender parses suffix/mode values (off/on, 64k, durability) with its diff --git a/core/src/test/java/io/questdb/client/test/cutlass/auth/TokenProviderConfigTest.java b/core/src/test/java/io/questdb/client/test/cutlass/auth/TokenProviderConfigTest.java index bfe077344..3e09e3cf9 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/auth/TokenProviderConfigTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/auth/TokenProviderConfigTest.java @@ -45,7 +45,8 @@ /** * Conformance tests C18 (connect-string validation) and C19 (registry lifecycle) of the dynamic-credential - * specification (design/qwp-token-provider-spec.md, sections 7.2, 7.4 and 10). + * specification (design/qwp-token-provider-spec.md, sections 7.2, 7.4 and 10), and the connect-string half of C24 + * (credential selection). */ public class TokenProviderConfigTest { private static final String WSS = "wss::addr=localhost:9000;"; @@ -64,6 +65,40 @@ public void tearDown() { TestTokenProviderFactory.uninstall(); } + @Test + public void testAzureCredentialSelectsOneCredential() throws Exception { + // C24, connect-string half: every value of section 7.1 is accepted by every entry point, reaches the factory, + // and keys the registry; "default" is the default and adds nothing. + assertMemoryLeak(() -> { + for (String value : new String[]{"managed_identity", "workload_identity", "environment"}) { + String cfg = WSS + "token_provider=azure;azure_resource=api://qdb-app;azure_credential=" + value + ';'; + Assert.assertEquals(value, Sender.builder(cfg).wsConfigSnapshotForTest().get("azure_credential")); + try (QwpQueryClient client = QwpQueryClient.fromConfig(cfg)) { + Assert.assertEquals(value, client.configSnapshotForTest().get("azure_credential")); + } + try (QuestDB ignored = QuestDB.connect(cfg + "sender_pool_min=0;query_pool_min=0;")) { + Assert.assertEquals("validation must not start a provider", 0, azure.created.get()); + } + Assert.assertEquals(value, spec("token_provider=azure;azure_resource=api://qdb-app;azure_credential=" + + value + ';').params().get(TokenProviderSpec.KEY_AZURE_CREDENTIAL)); + } + String omitted = "token_provider=azure;azure_resource=api://qdb-app;"; + Assert.assertNull(Sender.builder(WSS + omitted + "azure_credential=default;").wsConfigSnapshotForTest() + .get("azure_credential")); + Assert.assertEquals(spec(omitted).registryKey(), spec(omitted + "azure_credential=default;").registryKey()); + Assert.assertNotEquals(spec(omitted).registryKey(), spec(omitted + "azure_credential=managed_identity;").registryKey()); + + TokenProviderRegistry registry = new TokenProviderRegistry(0); + try (TokenProviderRegistry.Lease lease = registry.acquire(spec(omitted + + "azure_credential=managed_identity;azure_client_id=AAAAAAAA-BBBB-CCCC-DDDD-EEEEEEEEEEEE;"))) { + Assert.assertNotNull(lease.provider()); + Map params = azure.params.get(0); + Assert.assertEquals("managed_identity", params.get(TokenProviderSpec.KEY_AZURE_CREDENTIAL)); + Assert.assertEquals("aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee", params.get(TokenProviderSpec.KEY_AZURE_CLIENT_ID)); + } + }); + } + @Test public void testAzureKeysAreNormalized() { Map snap = Sender.builder(WSS + "token_provider=azure;azure_resource=api://qdb-app/.default;" @@ -182,6 +217,18 @@ public void testValidationRules() throws Exception { assertRejectedOnBoth(WSS + "token_provider=azure;azure_resource=api://x;azure_client_id=not-a-guid;", "invalid azure_client_id"); assertRejectedOnBoth(WSS + "token_provider=azure;azure_resource=invalid:x;", "rejected by the factory"); + // azure_credential: a provider key, the values of section 7.1 only (listed in the error), and a client ID + // only where the credential takes one + assertRejectedOnBoth(WSS + "azure_credential=managed_identity;", "azure_credential requires token_provider=azure"); + assertRejectedOnBoth(WSS + "token_provider=vault;azure_credential=managed_identity;", + "azure_credential is only valid with token_provider=azure"); + assertRejectedOnBoth(WSS + "token_provider=azure;azure_resource=api://x;azure_credential=chain;", + "invalid azure_credential: chain (expected default, managed_identity, workload_identity, environment)"); + assertRejectedOnBoth(WSS + "token_provider=azure;azure_resource=api://x;azure_credential=Managed_Identity;", + "invalid azure_credential: Managed_Identity"); + assertRejectedOnBoth(WSS + "token_provider=azure;azure_resource=api://x;azure_credential=environment;" + + "azure_client_id=aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee;", + "azure_client_id cannot be combined with azure_credential=environment"); // decision D4: never over plain ws:: assertRejectedOnBoth("ws::addr=localhost:9000;token_provider=azure;azure_resource=api://x;", "token_provider requires the wss:: schema"); @@ -190,6 +237,8 @@ public void testValidationRules() throws Exception { "token_provider is only supported with the wss:: schema"); assertRejected(() -> Sender.fromConfig("https::addr=localhost:9000;azure_resource=api://x;"), "azure_resource is only supported with the wss:: schema"); + assertRejected(() -> Sender.fromConfig("http::addr=localhost:9000;azure_credential=managed_identity;"), + "azure_credential is only supported with the wss:: schema"); // an application-supplied provider is exclusive with token_provider String cfg = WSS + "token_provider=azure;azure_resource=api://x;"; diff --git a/core/src/test/java/io/questdb/client/test/impl/QwpQueryClientConfigHonoredTest.java b/core/src/test/java/io/questdb/client/test/impl/QwpQueryClientConfigHonoredTest.java index 52758d3ea..3e02f327e 100644 --- a/core/src/test/java/io/questdb/client/test/impl/QwpQueryClientConfigHonoredTest.java +++ b/core/src/test/java/io/questdb/client/test/impl/QwpQueryClientConfigHonoredTest.java @@ -97,14 +97,16 @@ public void testEveryEgressKeyIsHonored() throws Exception { TestTokenProviderFactory.install(new TestTokenProviderFactory("azure")); try { Map tp = snapshot("wss::addr=h:9000;token_provider=azure;" - + "azure_resource=api://app/.default;azure_client_id=AAAAAAAA-BBBB-CCCC-DDDD-EEEEEEEEEEEE;"); + + "azure_resource=api://app/.default;azure_client_id=AAAAAAAA-BBBB-CCCC-DDDD-EEEEEEEEEEEE;" + + "azure_credential=managed_identity;"); Assert.assertEquals("azure", tp.get("token_provider")); Assert.assertEquals("api://app", tp.get("azure_resource")); Assert.assertEquals("aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee", tp.get("azure_client_id")); + Assert.assertEquals("managed_identity", tp.get("azure_credential")); } finally { TestTokenProviderFactory.uninstall(); } - markHonored("token_provider", "azure_resource", "azure_client_id"); + markHonored("token_provider", "azure_resource", "azure_client_id", "azure_credential"); // Drift guard: every egress-applied registry key must have an assertion // above. The honored set is populated by the assertions themselves, so diff --git a/core/src/test/java/io/questdb/client/test/impl/WsSenderConfigHonoredTest.java b/core/src/test/java/io/questdb/client/test/impl/WsSenderConfigHonoredTest.java index 6e6d2358d..f6d7e06bb 100644 --- a/core/src/test/java/io/questdb/client/test/impl/WsSenderConfigHonoredTest.java +++ b/core/src/test/java/io/questdb/client/test/impl/WsSenderConfigHonoredTest.java @@ -110,14 +110,15 @@ public void testEveryIngressKeyIsHonored() { TestTokenProviderFactory.install(new TestTokenProviderFactory("azure")); try { Map tp = snapshot("wss::addr=h:9000;token_provider=azure;azure_resource=api://app/.default;" - + "azure_client_id=AAAAAAAA-BBBB-CCCC-DDDD-EEEEEEEEEEEE;"); + + "azure_client_id=AAAAAAAA-BBBB-CCCC-DDDD-EEEEEEEEEEEE;azure_credential=managed_identity;"); Assert.assertEquals("azure", tp.get("token_provider")); Assert.assertEquals("api://app", tp.get("azure_resource")); Assert.assertEquals("aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee", tp.get("azure_client_id")); + Assert.assertEquals("managed_identity", tp.get("azure_credential")); } finally { TestTokenProviderFactory.uninstall(); } - markHonored("token_provider", "azure_resource", "azure_client_id"); + markHonored("token_provider", "azure_resource", "azure_client_id", "azure_credential"); // Drift guard: every ingress-applied registry key must have an assertion // above. The honored set is populated by the assertions themselves, so diff --git a/design/entra-id-qwp-auth.md b/design/entra-id-qwp-auth.md index 31c4c1e53..0832c3473 100644 --- a/design/entra-id-qwp-auth.md +++ b/design/entra-id-qwp-auth.md @@ -4,7 +4,7 @@ Status: **implemented** on `feat/qwp-entra-token-provider` - all four steps of t **Read these first:** -- **The spec wins.** The normative, language-neutral contract is now `design/qwp-token-provider-spec.md` (v0.3, all decisions resolved). Where the two documents differ, follow the spec. That includes the refresh-schedule clamps, backoff jitter, registry linger, and the Java names in its Appendix B, which supersede §7.2 here. +- **The spec wins.** The normative, language-neutral contract is now `design/qwp-token-provider-spec.md` (v0.4, all decisions resolved). Where the two documents differ, follow the spec. That includes the refresh-schedule clamps, backoff jitter, registry linger, and the Java names in its Appendix B, which supersede §7.2 here. - **Java plan:** §10. - **Line references** are as of `c9968f24` and may drift. @@ -580,18 +580,18 @@ Every step of §10 is implemented. Line references in §1-§7 predate it. **Step 3 - connect string, registry, Azure module** (spec §7). -- `ConfigSchema` registers `token_provider`, `azure_resource` and `azure_client_id` (COMMON). +- `ConfigSchema` registers `token_provider`, `azure_resource`, `azure_client_id` and the enum `azure_credential` (COMMON). `TokenProviderSpec` normalizes `azure_credential=default` away, so it shares a provider with a string that omits the key, and rejects `azure_client_id` with `azure_credential=environment`. - `TokenProviderSpec.parse` enforces §7.2 on both clients and resolves the factory without fetching anything. It strips `/.default`, checks and lower-cases the client-ID GUID, and builds the registry key. - `TokenProviderFactory` is the `ServiceLoader` SPI (`uses` in `module-info.java`). `TokenProviderRegistry` hands out ref-counted leases, with a 60 s linger, a daemon timer that exits when idle, and a test seam that replaces discovery. - The non-QWP `Sender` schemas reject the keys (D9). The builders and `QuestDBBuilder` reject mixing them with an application-supplied provider. - Leases: `Sender.build()` acquires one and hands it to `QwpWebSocketSender.setCredentialLease` (released on close, or released by `build()` if it fails). `QwpQueryClient` acquires one on its first `connect()` and releases it on close. - New reactor module `azure/`, artifact `org.questdb:questdb-client-azure` (not `io.questdb`: the core artifact is `org.questdb:questdb-client`): - - `AzureTokenProviderFactory` uses `DefaultAzureCredential`; `azure_client_id` sets both the managed-identity and workload-identity client ID. - - `AzureTokenSource` bounds each request at 30 s and classifies per §7.5: `CredentialUnavailableException`, HTTP 400/401 and known AADSTS configuration codes are permanent; everything else is retryable, with `Retry-After` honoured. + - `AzureTokenProviderFactory` uses `DefaultAzureCredential` for `azure_credential=default`, where `azure_client_id` sets both the managed-identity and workload-identity client ID, and warns once when that chain starts. The other values build `ManagedIdentityCredential`, `WorkloadIdentityCredential` or `EnvironmentCredential` directly (spec Appendix B); a credential that cannot be built for lack of configuration fails every fetch as permanent instead of failing a client's `build()`. + - `AzureTokenSource` bounds each request at 30 s and classifies per §7.5: `CredentialUnavailableException`, a managed identity that is not assigned (reported only in a nested cause when that credential is used alone), HTTP 400/401 and known AADSTS configuration codes are permanent; everything else is retryable, with `Retry-After` honoured, and names the socket-level cause of a network failure. - It never attaches a library exception: its response references the raw request. - Java 8 bytecode. It has no `module-info`; `Automatic-Module-Name: io.questdb.client.azure`, and an automatic module provides its `META-INF/services`. - Its release profiles mirror core's. -- Tests: `TokenProviderConfigTest` (C18, C19), `TokenProviderSharingTest` (C17, over TLS), `azure/AzureTokenSourceTest` (fake `TokenCredential`, no network). +- Tests: `TokenProviderConfigTest` (C18, C19, the connect-string half of C24), `TokenProviderSharingTest` (C17, over TLS), `azure/AzureTokenSourceTest` (fake `TokenCredential`, no network), `azure/AzureCredentialSelectionTest` (the real factory, no network) and `azure/AzureManagedIdentityLibraryTest` (C24: the real library in a child JVM, against a local IMDS stub). **Step 4 - health and deadline** (spec §8.4, §8.5). diff --git a/design/qwp-token-provider-spec.md b/design/qwp-token-provider-spec.md index f9f312500..3df1607aa 100644 --- a/design/qwp-token-provider-spec.md +++ b/design/qwp-token-provider-spec.md @@ -4,7 +4,7 @@ **Changes:** -- **v0.4:** `azure_credential` selects one Azure credential deterministically (§7.1), and library errors are classified by how the credential was selected (§7.5, D10). This resolves the conflict between §4 and §7.5 over an unreachable managed-identity endpoint. +- **v0.4:** `azure_credential` selects one Azure credential deterministically (§7.1), so the library itself reports an unreachable managed-identity endpoint (retryable) apart from a misconfigured credential (permanent) (§7.5, D10). A discovery chain's "no credential available" stays permanent, so a host without a credential still fails fast, and the client warns once that the chain does not ride out an endpoint outage. This resolves the conflict between §4 and §7.5 over an unreachable managed-identity endpoint. - **v0.3:** decisions resolved, with D8 (§8.1) and D9 (§7.2) added; Java binding decided (Appendix B). - **v0.2:** connection health (§8.4), the optional authentication-outage deadline (§8.5), `WWW-Authenticate` handling (§8.2), and Appendices C and D. @@ -88,7 +88,7 @@ Classification: - **Retryable:** network failures, timeouts, HTTP 429, HTTP 5xx, and IMDS 404 or 410. - **Permanent:** configuration that is missing or wrong. Examples: the credential the source was told to use is not configured, identity not found, invalid client. -- **Discovery:** a source that finds its credential by probing the environment MUST NOT treat "nothing found" as permanent on that evidence alone. A brief outage of a probed endpoint looks the same (§7.5) **[D10]**. +- **Discovery:** a source that finds its credential by probing the environment cannot tell "nothing found" from a brief outage of a probed endpoint (§7.5). It MAY report "nothing found" as permanent, so a host without a credential fails fast, but it MUST then offer a way to select one credential deterministically, which reports the two apart **[D10]**. - **When unsure:** treat the failure as retryable. - An error that carries no classification MUST be treated as retryable. @@ -282,10 +282,10 @@ A client MUST reject the configuration, with an error that names the offending k - **Expiry:** `expires_at`, and `refresh_at` when the library exposes it, come from the library's token result. - **Classification of library errors [D10]:** - **Deterministic mode:** - - **Permanent:** configuration that the selected credential reports as missing or wrong. That covers missing settings it requires (for example, no federated token file for a workload identity), an identity that is not found or not assigned to the host, and Entra rejecting the client, secret, assertion, application or tenant. + - **Permanent:** configuration that the selected credential reports as missing or wrong. That covers missing settings it requires (for example, no federated token file for a workload identity), an identity that is not found or not assigned to the host, and Entra rejecting the client, secret, assertion, application or tenant. This holds when the library reports the reason only in a nested cause. - **Retryable:** network errors, including a managed-identity endpoint that cannot be reached; timeouts; throttling; `404`, `410` and `5xx` responses from a managed-identity endpoint; and anything else. - **Discovery chain:** - - "No credential available in the chain" is **retryable** (§4). + - "No credential available in the chain" is **permanent** (§4), so on a host without a credential a `sync` startup fails fast instead of waiting out `reconnect_max_duration_millis`. A brief managed-identity outage is reported the same way, so a chain does not ride one out at startup; deterministic mode does **[D10]**. - A failure that one credential in the chain reports for itself is classified as in deterministic mode. - A client SHOULD log once, when the provider starts, that a discovery chain does not retry a managed-identity outage, and recommend `azure_credential` for production. - **Error text:** libraries often report every managed-identity failure with one generic message and keep the reason, such as "Identity not found", in a nested cause. The error SHOULD carry that innermost reason, sanitized as §9 requires. @@ -293,7 +293,7 @@ A client MUST reject the configuration, with an error that names the offending k *Informative bindings:* -- **Java:** the optional `questdb-client-azure` artifact, which uses `DefaultAzureCredential` and is found through `ServiceLoader`. It maps `azure_credential` onto the library's own selector (Appendix B). +- **Java:** the optional `questdb-client-azure` artifact, which is found through `ServiceLoader`. It uses `DefaultAzureCredential` for `default` and builds the selected credential itself otherwise (Appendix B). - **Python:** `azure-identity` as an optional extra. - **Rust and C:** not required. - **Platform support:** the Azure Identity libraries read `AZURE_TOKEN_CREDENTIALS`. When it selects `ManagedIdentityCredential`, they skip the IMDS check and retry with backoff from .NET 1.16.0, Java 1.18.1, Python 1.25.1, JavaScript 4.13.0, Go 1.13.0 and C++ 1.13.2. @@ -448,7 +448,7 @@ Every client that implements this specification SHOULD pass these scenarios. The | C21 | Connection health | The snapshot moves through `connecting`, `connected`, `reconnecting` (with `401` and credential-unavailable failures) and back to `connected`. `outage_since` and `failed_rounds` reset on recovery; `last_failure` is kept. No credential appears in it. | | C22 | Authentication-outage deadline | Unset: retries continue indefinitely. Set: the sender becomes terminal only on an authentication-class round once the duration has passed. Other failures in between neither reset nor fire it. A successful upgrade resets it. Orphan drains are unaffected, and on-disk data remains. | | C23 | `WWW-Authenticate` challenge | A plain `401`, and a `401` with `Bearer error="invalid_token"`, both trigger the retry in §8.2. A `401` whose Bearer challenge carries another error does not. | -| C24 | Azure credential selection | With `azure_credential=managed_identity`, an unreachable managed-identity endpoint is retryable, so `sync` initialization retries it within its budget; an identity that is not assigned is permanent, so initialization fails fast. With `default`, a chain that finds nothing is retryable, and the warning in §7.5 is logged once. | +| C24 | Azure credential selection | With `azure_credential=managed_identity`, an unreachable managed-identity endpoint is retryable, so `sync` initialization retries it within its budget; an identity that is not assigned is permanent, so initialization fails fast. With `default`, a chain that finds nothing is permanent, and the warning in §7.5 is logged once, unless the platform's selector names one credential. | ## 11. Relation to existing specifications @@ -473,7 +473,7 @@ All decisions below are resolved. Changing one after the Java implementation mer | D7 | Whether to offer the authentication-outage deadline, and its key | Yes, as an optional setting that is off by default: `auth_failure_max_duration_millis`. | | D8 | How to classify an exception from an application-supplied provider that is not a token-unavailable error | Permanent (§8.1). | | D9 | Whether `token_provider` applies to non-QWP schemas | No; `wss::` only (§7.2). | -| D10 | How the `azure` provider classifies "no credential available", and how to make it resilient | `azure_credential` selects one credential, so the library itself reports an unreachable endpoint (retryable) apart from a misconfigured credential (permanent). In a discovery chain, "no credential available" is retryable, because an IMDS outage looks the same (§7.5). Only `sync` initialization depends on the difference (§8.3), so the cost is a wait of at most `reconnect_max_duration_millis`. This follows Azure Identity's split between fail-fast discovery and a resilient single credential (Appendix D). A local host check, as Google's library makes, was not adopted: Azure's SMBIOS asset tag identifies only the public cloud, not sovereign clouds or Azure Local. | +| D10 | How the `azure` provider classifies "no credential available", and how to make it resilient | `azure_credential` selects one credential, so the library itself reports an unreachable endpoint (retryable) apart from a misconfigured credential (permanent). In a discovery chain, "no credential available" stays permanent. An IMDS outage looks the same (§7.5), but only `sync` initialization depends on the difference (§8.3), and making it retryable would make every host without a credential wait out `reconnect_max_duration_millis` (5 minutes by default, plus up to one `cold_wait`) before failing. Resilience is therefore opt-in through deterministic mode, which Azure Identity recommends in production anyway, and a chain warns once when it starts. This follows Azure Identity's split between fail-fast discovery and a resilient single credential (Appendix D). A local host check, as Google's library makes, was not adopted: Azure's SMBIOS asset tag identifies only the public cloud, not sovereign clouds or Azure Local. | **Still to verify in a real Entra tenant.** These checks don't block implementation, but they should be done before the spec freezes: @@ -539,7 +539,7 @@ New types live in `io.questdb.client.cutlass.auth` unless noted. | `azure` provider | The new reactor module `azure/`, artifact `org.questdb:questdb-client-azure` (the core client is `org.questdb:questdb-client`), package `io.questdb.client.azure`, classes `AzureTokenProviderFactory` and `AzureTokenSource`. It depends on `azure-identity` through `azure-sdk-bom`, keeps the Java 8 floor, and is released together with the client. | | Connection health | `io.questdb.client.ConnectionHealth`, an immutable snapshot. Returned by `Sender.health()` (a default method; non-QWP senders throw `UnsupportedOperationException`), by `QwpQueryClient.health()`, and as a `ConnectionHealth.Aggregate` by `QuestDB.health()`. | | Authentication-outage deadline | Key `auth_failure_max_duration_millis`; builder method `authFailureMaxDurationMillis(long)` | -| Credential selection | Key `azure_credential`, handled by `AzureTokenProviderFactory`. It sets `AZURE_TOKEN_CREDENTIALS` (`ManagedIdentityCredential`, `WorkloadIdentityCredential` or `EnvironmentCredential`) in the configuration it passes to `DefaultAzureCredentialBuilder.configuration(...)`, never in the process environment. That needs `azure-identity` 1.18.1 or later. `AzureTokenSource` learns the mode from the factory; for a credential an application passes in itself, a `DefaultAzureCredential` counts as a discovery chain and any other credential as deterministic. | +| Credential selection | Key `azure_credential`, validated by `TokenProviderSpec` and handled by `AzureTokenProviderFactory`. `default` builds `DefaultAzureCredential`; the other values build `ManagedIdentityCredential`, `WorkloadIdentityCredential` or `EnvironmentCredential` directly, never touching the process environment. Setting `AZURE_TOKEN_CREDENTIALS` in the configuration passed to `DefaultAzureCredentialBuilder.configuration(...)` is not enough: in `azure-identity` 1.18 it selects the credential but keeps the chain's endpoint check, because each credential re-reads the selector from the process-wide configuration. A credential that cannot be built for lack of configuration fails every fetch as permanent. `AzureTokenSource` classifies the same way whatever the credential: it finds a managed identity's "not assigned" reason in a nested cause, and names the socket-level cause of a network failure, but never echoes another nested message (§9). | **Python notes:** @@ -577,10 +577,10 @@ Other server-side follow-ups, including security hardening, are tracked privatel | Azure SDK, `BearerTokenAuthenticationPolicy` | Retries only on a `401` that carries a Continuous Access Evaluation claims challenge. A plain `401` goes back to the caller. | A stricter variant of D2. | | Kafka: KIP-152 and KAFKA-6516 | Authentication failures are non-retriable at the API. The client is not closed, though, and keeps reconnecting in the background; a request to stop that was closed Won't Fix. | D1. | | Kafka: KAFKA-10840 (open) | When credentials expire, a consumer keeps failing authentication in the background and the application cannot see it. A proposed fix notes that Kafka Connect tasks report RUNNING meanwhile. | Why §8.4 exists. | -| Azure Identity: credential chains and the managed-identity retry strategy | `DefaultAzureCredential` runs managed identity in a "fail fast" mode, meant for the development inner loop: one IMDS probe with a short timeout and no retries. A "resilient" mode skips the probe and retries with exponential backoff; `AZURE_TOKEN_CREDENTIALS=ManagedIdentityCredential`, or the managed-identity credential used directly, enables it. Microsoft recommends a deterministic credential in production. | D10: a discovery chain's "no credential" is no evidence of misconfiguration; `azure_credential` selects the resilient mode. | +| Azure Identity: credential chains and the managed-identity retry strategy | `DefaultAzureCredential` runs managed identity in a "fail fast" mode, meant for the development inner loop: one IMDS probe with a short timeout and no retries. A "resilient" mode skips the probe and retries with exponential backoff; `AZURE_TOKEN_CREDENTIALS=ManagedIdentityCredential`, or the managed-identity credential used directly, enables it. Microsoft recommends a deterministic credential in production. | D10: a discovery chain's "no credential" is weak evidence of misconfiguration; `azure_credential` selects the resilient mode. | | MongoDB driver auth spec (MONGODB-OIDC) | The application names its environment (`ENVIRONMENT:azure`, `gcp` or `k8s`), and the driver calls that metadata endpoint directly, with no discovery. | D10: deterministic selection. | | Google auth library, `ComputeEngineCredentials` | Detection pings the metadata server three times with a short timeout, "for developer desktop scenarios", then checks the SMBIOS product name, which needs no network. So a slow metadata server on a real VM is not taken for "not on GCE". A `503` from the metadata server is retryable. | D10: a probe that gets no answer is not proof; the local check is not adopted (§12). | -| AWS SDK for Java v2, issue #3939; Apache Druid | On EC2, the default chain sometimes reports "Unable to load credentials from any of the providers in the chain", which AWS attributes to IMDS latency. Druid now treats that error as recoverable within its retry budget. | D10: "nothing found" in a chain is retryable. | +| AWS SDK for Java v2, issue #3939; Apache Druid | On EC2, the default chain sometimes reports "Unable to load credentials from any of the providers in the chain", which AWS attributes to IMDS latency. Druid now treats that error as recoverable within its retry budget. | D10: "nothing found" in a chain is ambiguous. Druid chose retryable, at the cost of a slower failure when nothing is configured; this spec keeps it permanent and offers deterministic mode instead. | Links: From d78d384163c19f71d657449f32d9fd2d2018c294 Mon Sep 17 00:00:00 2001 From: Vlad Ilyushchenko Date: Wed, 7 Oct 2026 17:04:27 +0100 Subject: [PATCH 05/40] fix(qwp): don't close a pooled query client under a busy dispatch thread QueryWorker.shutdown() closed the query client once its bounded 5 s join expired, even with the dispatch thread still inside execute(). A reconnect walk parked in a native connect or WebSocket upgrade does not see the shutdown interrupt, so it outlived the close: it went on to a healthy endpoint, then wrote the binds into the bind buffer close() had freed (SIGSEGV), or published an I/O thread and connection on the closed client (leak). The failover reconnect could already reach this; the reconnect on the next execute() after a failed failover made it reachable from every query on such a client. When the join expires with the dispatch thread still busy, the thread now closes its own client on its way out of runLoop(). One CAS decides which side closes, so the client is closed exactly once and never under a running execute(). Shutdown latency is unchanged and the per-query path is untouched. --- .../io/questdb/client/impl/QueryWorker.java | 59 +++- .../impl/QueryWorkerBusyShutdownTest.java | 331 ++++++++++++++++++ 2 files changed, 382 insertions(+), 8 deletions(-) create mode 100644 core/src/test/java/io/questdb/client/test/impl/QueryWorkerBusyShutdownTest.java diff --git a/core/src/main/java/io/questdb/client/impl/QueryWorker.java b/core/src/main/java/io/questdb/client/impl/QueryWorker.java index 568ac0dfe..95422ffb2 100644 --- a/core/src/main/java/io/questdb/client/impl/QueryWorker.java +++ b/core/src/main/java/io/questdb/client/impl/QueryWorker.java @@ -27,7 +27,10 @@ import io.questdb.client.Query; import io.questdb.client.QueryException; import io.questdb.client.cutlass.qwp.client.QwpQueryClient; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import java.util.concurrent.atomic.AtomicInteger; import java.util.concurrent.locks.Condition; import java.util.concurrent.locks.ReentrantLock; @@ -49,7 +52,16 @@ public final class QueryWorker { static final long SHUTDOWN_JOIN_MILLIS = 5_000; + // clientCloseOwner states, see shutdown() + private static final int CLOSE_BY_DISPATCH_THREAD = 1; + private static final int CLOSE_UNDECIDED = 0; + private static final int DISPATCH_THREAD_EXITED = 2; + private static final Logger LOG = LoggerFactory.getLogger(QueryWorker.class); private final QwpQueryClient client; + // Decides which of shutdown() and the dispatch thread closes the client: exactly one of them, and never + // while the dispatch thread can still be inside execute(). Both sides settle it with a CAS from + // CLOSE_UNDECIDED, so the two cannot both close, and neither can skip the close. + private final AtomicInteger clientCloseOwner = new AtomicInteger(CLOSE_UNDECIDED); private final long createdAtMillis; private final QueryClientPool pool; private final QueryImpl query; @@ -85,7 +97,7 @@ public QueryWorker(QwpQueryClient client, QueryClientPool pool, int slotIndex) { this.pool = pool; this.query = new QueryImpl(this); this.signalCondition = signalLock.newCondition(); - this.thread = new Thread(this::runLoop, "questdb-query-worker-" + slotIndex); + this.thread = new Thread(this::dispatchThreadMain, "questdb-query-worker-" + slotIndex); this.thread.setDaemon(true); this.createdAtMillis = System.currentTimeMillis(); this.idleSinceMillis = this.createdAtMillis; @@ -251,14 +263,25 @@ void shutdown() { callerWasInterrupted = true; } } finally { - // close() must run even if cancel()/join() threw, otherwise the - // client's native buffer pool and socket leak for the lifetime of - // the process. Catch Throwable so shutdown() itself never propagates - // a teardown Error to its callers. - try { - client.close(); - } catch (Throwable ignored) { + // The client must be closed even if cancel()/join() threw, otherwise its native buffer pool and + // socket leak for the lifetime of the process -- but not while the dispatch thread can still be + // inside execute(). close() frees what a running execute() uses: its bind buffer and the connection + // a reconnect walk is building. A walk parked in a native connect or WebSocket upgrade does not see + // the interrupt above, so an expired join does not mean the thread is idle. Closing anyway let the + // walk go on to a healthy endpoint, write the binds into the freed buffer (SIGSEGV), or publish an + // I/O thread and connection on the closed client (leak). In that case the dispatch thread closes + // the client itself on its way out of runLoop(); see dispatchThreadMain(). + if (!thread.isAlive()) { + // exited, or never started: nothing can race the close + closeClient(); + } else if (clientCloseOwner.compareAndSet(CLOSE_UNDECIDED, CLOSE_BY_DISPATCH_THREAD)) { + LOG.warn("{} still busy {} ms after shutdown; its query client is closed when the in-flight " + + "execute() returns", thread.getName(), SHUTDOWN_JOIN_MILLIS); + } else if (clientCloseOwner.get() == DISPATCH_THREAD_EXITED) { + // the dispatch thread left runLoop() between the isAlive() check and the CAS + closeClient(); } + // else: an earlier shutdown() already handed the close to the dispatch thread if (callerWasInterrupted) { Thread.currentThread().interrupt(); } @@ -292,6 +315,26 @@ void dispatch(QueryImpl q) { } } + private void closeClient() { + try { + client.close(); + } catch (Throwable ignored) { + // Best-effort: neither shutdown() nor the exiting dispatch thread may propagate a teardown Error. + } + } + + private void dispatchThreadMain() { + try { + runLoop(); + } finally { + // From here on this thread never touches the client again. If shutdown() stopped waiting for it + // while it was still inside a job, the close was left to this thread; run it now. + if (!clientCloseOwner.compareAndSet(CLOSE_UNDECIDED, DISPATCH_THREAD_EXITED)) { + closeClient(); + } + } + } + private void runLoop() { // Loop unconditionally -- do NOT hoist the shuttingDown check up here as // while (!shuttingDown). The sole exit is the "if (shuttingDown) return" diff --git a/core/src/test/java/io/questdb/client/test/impl/QueryWorkerBusyShutdownTest.java b/core/src/test/java/io/questdb/client/test/impl/QueryWorkerBusyShutdownTest.java new file mode 100644 index 000000000..cb264007a --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/impl/QueryWorkerBusyShutdownTest.java @@ -0,0 +1,331 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test.impl; + +import io.questdb.client.Completion; +import io.questdb.client.Query; +import io.questdb.client.QueryException; +import io.questdb.client.QuestDB; +import io.questdb.client.cutlass.qwp.client.QwpColumnBatch; +import io.questdb.client.cutlass.qwp.client.QwpColumnBatchHandler; +import io.questdb.client.cutlass.qwp.client.QwpEgressMsgKind; +import io.questdb.client.cutlass.qwp.client.QwpQueryClient; +import io.questdb.client.cutlass.qwp.protocol.QwpConstants; +import io.questdb.client.impl.QueryClientPool; +import io.questdb.client.impl.QueryWorker; +import io.questdb.client.impl.QuestDBImpl; +import io.questdb.client.test.cutlass.qwp.websocket.TestWebSocketServer; +import io.questdb.client.test.tools.TestUtils; +import org.junit.Assert; +import org.junit.Test; + +import java.io.IOException; +import java.lang.reflect.Field; +import java.net.InetAddress; +import java.net.ServerSocket; +import java.net.Socket; +import java.nio.ByteBuffer; +import java.nio.ByteOrder; +import java.util.List; +import java.util.concurrent.CopyOnWriteArrayList; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.atomic.AtomicReference; + +/** + * A pool shutdown whose bounded join expires while the worker's dispatch thread is still inside + * {@code execute()} must not close the query client under it. {@code QwpQueryClient.close()} frees what a + * running {@code execute()} uses -- its bind buffer and the connection a reconnect walk is building -- and a + * walk parked in a native connect or WebSocket upgrade ignores the shutdown interrupt. Closing anyway let the + * walk reach a healthy endpoint, then write the binds into the freed buffer (SIGSEGV) or publish an I/O + * thread and connection on the closed client (leak). The dispatch thread now closes the client itself once + * the call returns. + */ +public class QueryWorkerBusyShutdownTest { + private static final String SENDER_CFG = "http::addr=127.0.0.1:1;protocol_version=2;auto_flush=off;"; + + @Test(timeout = 60_000) + public void testShutdownLeavesTheCloseToADispatchThreadStillInsideExecute() throws Exception { + TestUtils.assertMemoryLeak(() -> { + try (TestWebSocketServer server = startServer(new AtomicBoolean())) { + AtomicInteger closes = new AtomicInteger(); + AtomicReference dispatchThread = new AtomicReference<>(); + CountDownLatch inHandler = new CountDownLatch(1); + CountDownLatch releaseHandler = new CountDownLatch(1); + QuestDBImpl db = new QuestDBImpl( + SENDER_CFG, "ws::addr=localhost:" + server.getPort() + ";", + 0, 1, 1, 1, + 10_000L, Long.MAX_VALUE, Long.MAX_VALUE, Long.MAX_VALUE, + slot -> { + throw new AssertionError("no sender expected"); + }, + client -> { + client.setBeforeCloseHookForTest(closes::incrementAndGet); + client.connect(); + }); + try { + Query q = db.borrowQuery(); + // The handler runs on the dispatch thread inside execute() and ignores the shutdown + // interrupt, like a reconnect walk parked in a native wait. + q.sql("SELECT 1").handler(new Handler() { + @Override + public void onExecDone(short opType, long rowsAffected) { + dispatchThread.set(Thread.currentThread()); + inHandler.countDown(); + awaitUninterruptibly(releaseHandler); + } + }).submit(); + Assert.assertTrue("the query never reached its handler", inHandler.await(10, TimeUnit.SECONDS)); + + db.close(); // the join gives up after SHUTDOWN_JOIN_MILLIS with the thread still in execute() + Assert.assertTrue(dispatchThread.get().isAlive()); + Assert.assertEquals("the pool must not close the client while its dispatch thread is inside " + + "execute()", 0, closes.get()); + + releaseHandler.countDown(); + dispatchThread.get().join(TimeUnit.SECONDS.toMillis(10)); + Assert.assertFalse("the dispatch thread must exit once execute() returns", + dispatchThread.get().isAlive()); + Assert.assertEquals("the dispatch thread must close its client on the way out", 1, closes.get()); + } finally { + releaseHandler.countDown(); + db.close(); + } + } + }); + } + + @Test(timeout = 60_000) + public void testShutdownDuringAReconnectWalkWithBinds() throws Exception { + assertShutdownDuringAReconnectWalkIsSafe(true); + } + + @Test(timeout = 60_000) + public void testShutdownDuringAReconnectWalkWithoutBinds() throws Exception { + assertShutdownDuringAReconnectWalkIsSafe(false); + } + + /** + * The failure from the review: a pooled client whose failover reconnect failed reconnects on its next + * execute(). The walk stalls on an endpoint that accepts TCP but never answers the upgrade, outlasting the + * 5 s shutdown join, then reaches a healthy endpoint after QuestDB.close() returned. + */ + private static void assertShutdownDuringAReconnectWalkIsSafe(boolean binds) throws Exception { + TestUtils.assertMemoryLeak(() -> { + AtomicBoolean dropNextQuery = new AtomicBoolean(); + AtomicBoolean tokenOutage = new AtomicBoolean(); + try (TestWebSocketServer healthy = startServer(dropNextQuery); + StallingEndpoint stalling = new StallingEndpoint()) { + // auth_timeout_ms bounds the stalled upgrade: above the 5 s shutdown join, so the walk outlives + // it, and short enough to keep the test quick + String cfg = "ws::addr=localhost:" + healthy.getPort() + ",localhost:" + stalling.port() + + ";sender_pool_min=0;query_pool_min=1;query_pool_max=1;failover_backoff_initial_ms=0" + + ";auth_timeout_ms=6500;"; + QuestDB db = QuestDB.connect(cfg, () -> { + if (tokenOutage.get()) { + throw new IllegalStateException("token outage"); + } + return "TOKEN"; + }); + Thread dispatchThread = dispatchThread(db); + QwpQueryClient client = pooledWorker(db).client(); + try { + // a failover whose reconnect fails leaves the pooled client disconnected + dropNextQuery.set(true); + tokenOutage.set(true); + try (Query q = db.borrowQuery()) { + q.sql("SELECT 1").handler(new Handler()).submit().await(); + Assert.fail("the failover reconnect must fail"); + } catch (QueryException expected) { + // the slot reconnects on its next execute() + } + tokenOutage.set(false); + Assert.assertFalse(client.isConnected()); + + // the next execute() walks the endpoints and parks on the stalling one + Query q = db.borrowQuery(); + q.sql(binds ? "SELECT $1" : "SELECT 1").handler(new Handler()); + if (binds) { + q.binds(b -> b.setLong(0, 42L)); + } + Completion completion = q.submit(); + Assert.assertTrue("the reconnect walk never reached the stalling endpoint", + stalling.awaitAccept(10, TimeUnit.SECONDS)); + Assert.assertFalse(completion.isDone()); + + db.close(); // returns after the 5 s join, with the walk still parked + dispatchThread.join(TimeUnit.SECONDS.toMillis(30)); // the walk resumes, then the thread exits + Assert.assertFalse("the dispatch thread must exit", dispatchThread.isAlive()); + Assert.assertFalse("a closed client must not stay connected", client.isConnected()); + long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(10); + while (healthy.liveConnectionCount() > 0 && System.nanoTime() < deadline) { + Thread.sleep(10); + } + Assert.assertEquals("the connection the walk opened must be closed", 0, healthy.liveConnectionCount()); + } finally { + db.close(); + dispatchThread.join(TimeUnit.SECONDS.toMillis(30)); + } + } + }); + } + + private static void awaitUninterruptibly(CountDownLatch latch) { + boolean interrupted = false; + try { + while (true) { + try { + if (!latch.await(30, TimeUnit.SECONDS)) { + throw new IllegalStateException("test never released the handler"); + } + return; + } catch (InterruptedException e) { + interrupted = true; + } + } + } finally { + if (interrupted) { + Thread.currentThread().interrupt(); + } + } + } + + private static byte[] buildExecDone(byte[] queryRequest) { + int bodyLen = 1 + 8 + 1 + 1; // msg_kind + request_id + op_type + rows_affected varint + byte[] frame = new byte[QwpConstants.HEADER_SIZE + bodyLen]; + ByteBuffer bb = ByteBuffer.wrap(frame).order(ByteOrder.LITTLE_ENDIAN); + bb.put((byte) 'Q').put((byte) 'W').put((byte) 'P').put((byte) '1'); + bb.put((byte) 1); // version + bb.put((byte) 0); // flags + bb.putShort((short) 0); // table_count + bb.putInt(bodyLen); // payload_length + bb.put(QwpEgressMsgKind.EXEC_DONE); + bb.put(queryRequest, 1, 8); // echo request_id verbatim + bb.put((byte) 0); // op_type + bb.put((byte) 0); // rows_affected = 0 + return frame; + } + + private static Thread dispatchThread(QuestDB db) throws Exception { + Field f = QueryWorker.class.getDeclaredField("thread"); + f.setAccessible(true); + return (Thread) f.get(pooledWorker(db)); + } + + @SuppressWarnings("unchecked") + private static QueryWorker pooledWorker(QuestDB db) throws Exception { + QueryClientPool pool = ((QuestDBImpl) db).getQueryPoolForTesting(); + Field f = QueryClientPool.class.getDeclaredField("all"); + f.setAccessible(true); + List all = (List) f.get(pool); + Assert.assertEquals(1, all.size()); + return all.get(0); + } + + private static TestWebSocketServer startServer(AtomicBoolean dropNextQuery) throws IOException, InterruptedException { + TestWebSocketServer server = new TestWebSocketServer(new TestWebSocketServer.WebSocketServerHandler() { + @Override + public void onBinaryMessage(TestWebSocketServer.ClientHandler client, byte[] data) { + if (data.length == 0 || data[0] != QwpEgressMsgKind.QUERY_REQUEST) { + return; + } + if (dropNextQuery.compareAndSet(true, false)) { + // a transport failure mid-query; close from another thread, ClientHandler.close() joins + // this read thread + Thread dropper = new Thread(client::close, "drop-connection"); + dropper.setDaemon(true); + dropper.start(); + return; + } + try { + client.sendBinary(buildExecDone(data)); + } catch (IOException e) { + // surfaces to the client as a transport error + } + } + }); + server.setSendServerInfo(true); + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + return server; + } + + private static class Handler implements QwpColumnBatchHandler { + @Override + public void onBatch(QwpColumnBatch batch) { + } + + @Override + public void onEnd(long totalRows) { + } + + @Override + public void onError(byte status, String message) { + } + } + + /** + * Completes TCP connects and never answers the WebSocket upgrade, like an overloaded or wedged node. + */ + private static final class StallingEndpoint implements AutoCloseable { + private final CountDownLatch accepted = new CountDownLatch(1); + private final List held = new CopyOnWriteArrayList<>(); + private final ServerSocket serverSocket; + + StallingEndpoint() throws IOException { + serverSocket = new ServerSocket(0, 50, InetAddress.getLoopbackAddress()); + Thread acceptor = new Thread(() -> { + try { + while (true) { + held.add(serverSocket.accept()); + accepted.countDown(); + } + } catch (IOException ignored) { + // closed + } + }, "stalling-endpoint"); + acceptor.setDaemon(true); + acceptor.start(); + } + + @Override + public void close() throws IOException { + serverSocket.close(); + for (Socket s : held) { + s.close(); + } + } + + boolean awaitAccept(long timeout, TimeUnit unit) throws InterruptedException { + return accepted.await(timeout, unit); + } + + int port() { + return serverSocket.getLocalPort(); + } + } +} From fefe3823e3f0b9945ba1a7803ec963f506ffbc4a Mon Sep 17 00:00:00 2001 From: Vlad Ilyushchenko Date: Wed, 7 Oct 2026 19:17:45 +0100 Subject: [PATCH 06/40] fix(qwp): report a query connection lost when failover gives up executeImpl's failover-budget-exhausted exit returned without telling the connection health tracker, unlike its sibling exits. A query whose connection died on the last attempt the budget allowed, or after failover_max_duration_ms ran out, left QwpQueryClient.health() - and the QuestDB.health() aggregate - reporting CONNECTED until the next execute(). With failover_max_attempts=1 every later query fails on the same latched failure without reconnecting, so the client reported CONNECTED for its whole life while each query failed. That exit now records the loss with connectionLost(), as the other exits do: RECONNECTING, since the next execute() fails over. With a single attempt per execute() it also calls failed(): the client never reconnects, as with failover=off, so FAILED is the true state. Health is read only through health(), so query behaviour is unchanged, and the success path never reaches this code. Tests: ConnectionHealthTest covers a connection lost on the last allowed attempt (RECONNECTING, then CONNECTED on the next query) and with a single attempt (FAILED, and the next query fails without reconnecting). --- .../cutlass/qwp/client/QwpQueryClient.java | 7 ++ .../qwp/client/ConnectionHealthTest.java | 112 ++++++++++++++++++ 2 files changed, 119 insertions(+) diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpQueryClient.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpQueryClient.java index 84d49b2f4..3e72e7d63 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpQueryClient.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpQueryClient.java @@ -1719,6 +1719,13 @@ private void executeImpl(CharSequence sql, QwpBindSetter binds, QwpColumnBatchHa return; } if (attempt >= failoverMaxAttempts || System.nanoTime() - failoverDeadlineNanos >= 0) { + // The connection is dead: its generation latched the transport failure, so no later query runs on + // it. Record the loss, as the exits below do. With a single attempt per execute(), every later + // execute() ends here on that same latch and never reconnects: terminal, as with failover off. + healthTracker.connectionLost(); + if (failoverMaxAttempts <= 1) { + healthTracker.failed(); + } int failovers = Math.max(0, attempt - 1); handler.onError(probe.interceptedRequestId, probe.interceptedStatus, "transport failure after " + attempt + " execute attempt" diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/ConnectionHealthTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/ConnectionHealthTest.java index b8c8b487e..6eb6513f9 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/ConnectionHealthTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/ConnectionHealthTest.java @@ -47,6 +47,7 @@ import java.nio.ByteBuffer; import java.nio.ByteOrder; import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicBoolean; import java.util.concurrent.atomic.AtomicReference; import static io.questdb.client.test.cutlass.auth.TokenTestKit.await; @@ -291,6 +292,76 @@ public void testQueryClientHealthFollowsConnectFailoverAndRecovery() throws Exce }); } + @Test(timeout = 60_000) + public void testQueryClientHealthIsFailedWhenItsOnlyAttemptLosesItsConnection() throws Exception { + // C21 for egress: with failover_max_attempts=1 a query that loses its connection fails, and every later + // query fails on the same latched failure without reconnecting - the client is done, as with failover off. + assertMemoryLeak(() -> { + DropQueryHandler handler = new DropQueryHandler(); + try (TestWebSocketServer server = startServer(handler)) { + server.setSendServerInfo(true); + try (QwpQueryClient client = QwpQueryClient.fromConfig("ws::addr=localhost:" + server.getPort() + + ";failover_max_attempts=1;")) { + client.connect(); + handler.drop.set(true); + RecordingHandler failed = new RecordingHandler(); + client.execute("SELECT 1", failed); + Assert.assertNotNull("the query must fail", failed.error); + Assert.assertTrue(failed.error, failed.error.startsWith("transport failure after 1 execute attempt")); + ConnectionHealth h = client.health(); + Assert.assertEquals(h.toString(), ConnectionHealth.State.FAILED, h.getState()); + Assert.assertNotEquals(ConnectionHealth.NONE, h.getOutageSinceEpochMillis()); + + // FAILED must hold: the server answers again, yet the next query fails without reconnecting + handler.drop.set(false); + int upgrades = server.upgradeRequestCount(); + RecordingHandler next = new RecordingHandler(); + client.execute("SELECT 1", next); + Assert.assertNotNull("the next query must fail", next.error); + Assert.assertTrue(next.error, next.error.startsWith("transport failure after 1 execute attempt")); + Assert.assertEquals("no reconnect", upgrades, server.upgradeRequestCount()); + Assert.assertEquals(ConnectionHealth.State.FAILED, client.health().getState()); + } + } + }); + } + + @Test(timeout = 60_000) + public void testQueryClientHealthReportsAConnectionLostOnTheLastAttempt() throws Exception { + // C21 for egress: execute() gives up when a query loses its connection on the last attempt its failover + // budget allows - here attempt 2 of 2, right after a successful failover reconnect. The health must not + // keep reporting that dead connection as connected; the next operation reconnects. + assertMemoryLeak(() -> { + DropQueryHandler handler = new DropQueryHandler(); + try (TestWebSocketServer server = startServer(handler)) { + server.setSendServerInfo(true); + try (QwpQueryClient client = QwpQueryClient.fromConfig("ws::addr=localhost:" + server.getPort() + + ";failover_max_attempts=2;failover_backoff_initial_ms=0;")) { + client.connect(); + Assert.assertEquals(ConnectionHealth.State.CONNECTED, client.health().getState()); + + handler.drop.set(true); + RecordingHandler failed = new RecordingHandler(); + client.execute("SELECT 1", failed); + Assert.assertNotNull("the query must fail", failed.error); + Assert.assertTrue(failed.error, failed.error.startsWith("transport failure after 2 execute attempts")); + ConnectionHealth h = client.health(); + Assert.assertEquals(h.toString(), ConnectionHealth.State.RECONNECTING, h.getState()); + Assert.assertNotEquals(ConnectionHealth.NONE, h.getOutageSinceEpochMillis()); + + handler.drop.set(false); + RecordingHandler recovered = new RecordingHandler(); + client.execute("SELECT 1", recovered); + Assert.assertNull(recovered.error, recovered.error); + Assert.assertEquals(1, recovered.execDone); + h = client.health(); + Assert.assertEquals(h.toString(), ConnectionHealth.State.CONNECTED, h.getState()); + Assert.assertEquals(ConnectionHealth.NONE, h.getOutageSinceEpochMillis()); + } + } + }); + } + @Test(timeout = 60_000) public void testSenderHealthMovesThroughEveryState() throws Exception { // C21: connecting -> connected -> reconnecting (with 401 and credential-unavailable failures) -> @@ -390,6 +461,24 @@ private static SenderError terminal(ErrorCollector errors) { throw new AssertionError("no terminal error"); } + // Answers like ExecDoneHandler but, while drop is set, drops the connection a query arrives on. + private static final class DropQueryHandler implements TestWebSocketServer.WebSocketServerHandler { + final AtomicBoolean drop = new AtomicBoolean(); + private final ExecDoneHandler answer = new ExecDoneHandler(); + + @Override + public void onBinaryMessage(TestWebSocketServer.ClientHandler client, byte[] data) { + if (drop.get() && data.length > 0 && data[0] == QwpEgressMsgKind.QUERY_REQUEST) { + // off this read thread: ClientHandler.close() joins it + Thread closer = new Thread(client::close, "drop-query-connection"); + closer.setDaemon(true); + closer.start(); + } else { + answer.onBinaryMessage(client, data); + } + } + } + private static final class ExecDoneHandler implements TestWebSocketServer.WebSocketServerHandler { private final WebSocketDynamicCredentialTest.AckHandler ack = new WebSocketDynamicCredentialTest.AckHandler(); @@ -428,4 +517,27 @@ public void onEnd(long totalRows) { public void onError(byte status, String message) { } } + + private static final class RecordingHandler implements QwpColumnBatchHandler { + String error; + int execDone; + + @Override + public void onBatch(QwpColumnBatch batch) { + } + + @Override + public void onEnd(long totalRows) { + } + + @Override + public void onError(byte status, String message) { + error = message; + } + + @Override + public void onExecDone(short opType, long rowsAffected) { + execDone++; + } + } } From 6048875f914ab4bb6329806e47e1aa48759e283e Mon Sep 17 00:00:00 2001 From: Vlad Ilyushchenko Date: Wed, 7 Oct 2026 19:59:14 +0100 Subject: [PATCH 07/40] fix(qwp): say memory-only rows are lost when the auth deadline fires The auth_failure_max_duration_millis javadoc and the ERROR log the deadline emits promised that unacknowledged rows stay in on-disk store-and-forward for a later sender or an orphan drain. That only holds with sf_dir: the option is accepted without one, memory-mode segments are malloc'd, orphan drains require sf_dir, and spec section 8.5 says memory-only data is lost when the sender closes. Document both modes in the javadoc, and make the log line name the outcome for the sender's actual backing (engine.sfDir() is null only in memory mode). --- core/src/main/java/io/questdb/client/Sender.java | 5 +++-- .../qwp/client/sf/cursor/CursorWebSocketSendLoop.java | 10 +++++++--- 2 files changed, 10 insertions(+), 5 deletions(-) diff --git a/core/src/main/java/io/questdb/client/Sender.java b/core/src/main/java/io/questdb/client/Sender.java index b0f55d2ef..52f0e9c7f 100644 --- a/core/src/main/java/io/questdb/client/Sender.java +++ b/core/src/main/java/io/questdb/client/Sender.java @@ -1314,8 +1314,9 @@ public LineSenderBuilder connectTimeoutMillis(int millis) { * successful upgrade resets it; failures of other classes neither reset nor fire it. Applies while the * sender is established and during an {@code async} initial connect; orphan drains are unaffected. When * it fires, the error names the failure class and the elapsed time, goes to the error handler as - * terminal, and is thrown from later producer calls; unacknowledged rows stay in on-disk - * store-and-forward for a later sender or an orphan drain. WebSocket transport only. Connect-string key: + * terminal, and is thrown from later producer calls. With {@code sf_dir} set, unacknowledged rows stay + * in on-disk store-and-forward for a later sender or an orphan drain; in memory-only mode (no + * {@code sf_dir}) they are lost when the sender closes. WebSocket transport only. Connect-string key: * {@code auth_failure_max_duration_millis}. * * @param millis the deadline, {@code > 0} diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoop.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoop.java index 9d670e639..deb07a0b6 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoop.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoop.java @@ -2174,8 +2174,8 @@ private void connectLoop(Throwable initial, String phase, long paceFirstAttemptM * when a connect round of a FOREGROUND loop fails with an authentication-class failure ({@code failure} is a * credential-unavailable or a 401/403 rejection). Starts the outage clock on the first such failure; when a * deadline is armed and the clock has reached it, latches a terminal that names the failure class and the - * elapsed time, reports it to the error handler, and returns true. Unacknowledged rows stay in - * store-and-forward. + * elapsed time, reports it to the error handler, and returns true. Unacknowledged rows stay in on-disk + * store-and-forward when the engine has an sf_dir; in memory mode they are lost when the sender closes. */ private boolean authOutageDeadlineFired(Throwable failure) { if (reconnectPolicy != ReconnectPolicy.FOREGROUND) { @@ -2196,7 +2196,11 @@ private boolean authOutageDeadlineFired(Throwable failure) { final String message = "authentication outage deadline exceeded: " + failureClass + " persisted for " + elapsedMillis + "ms (auth_failure_max_duration_millis=" + TimeUnit.NANOSECONDS.toMillis(deadline) + "); last failure: " + failure.getMessage(); - LOG.error("{} -- the sender stops; unacknowledged rows stay in store-and-forward", message); + // Name the rows' fate per backing (spec section 8.5): only an sf_dir slot outlives this sender; memory-mode + // segments are malloc'd and freed when the sender closes. + LOG.error("{} -- the sender stops; {}", message, engine.sfDir() != null + ? "unacknowledged rows stay in on-disk store-and-forward for a later sender or an orphan drain" + : "unacknowledged rows are lost when the sender closes (memory-only mode, no sf_dir)"); long fromFsn = engine.ackedFsn() + 1L; long toFsn = Math.max(fromFsn, engine.publishedFsn()); SenderError err = new SenderError( From 83847aec84d6d807f2a59990d73418cd1742f399 Mon Sep 17 00:00:00 2001 From: glasstiger Date: Tue, 13 Oct 2026 23:08:58 +0100 Subject: [PATCH 08/40] Report external FSNs from the auth deadline authOutageDeadlineFired() built its TERMINAL SenderError span from the engine's raw FSNs, while every other SenderError the loop emits adds externalFsnBase. After a symbol-dictionary recycle, which is on by default, the rebuilt engine numbers its frames from 0 again, so the span named FSNs that flushAndGetSequence() had already returned for acknowledged rows and left out the rows still waiting. The span now adds externalFsnBase like its siblings. ConnectionHealthTest gains a test that recycles the dictionary, then rejects every handshake with 401 past a short deadline, and checks the terminal span against the FSNs the sender reported. It fails without the fix (fromFsn 0 instead of 1), and also fails if the rebuilt loop is not armed with the deadline. QwpQueryClientDynamicCredentialTest gains three tests for paths that no test pinned: - connect() and a failover reconnect retry a refreshable 401 only once, even when the provider hands out a new token on every pull. Without the limit, both walks kept retrying for as long as the server rejected. - When the reconnect that the next execute() makes after a failed failover fails as well, the operation reports "reconnect failed:", and a later execute() still reconnects instead of throwing "not connected". --- .../sf/cursor/CursorWebSocketSendLoop.java | 6 +- .../qwp/client/ConnectionHealthTest.java | 48 +++++++++ .../QwpQueryClientDynamicCredentialTest.java | 98 +++++++++++++++++++ 3 files changed, 150 insertions(+), 2 deletions(-) diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoop.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoop.java index 3ae81a7ed..a415eb31d 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoop.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoop.java @@ -2365,8 +2365,10 @@ private boolean authOutageDeadlineFired(Throwable failure) { LOG.error("{} -- the sender stops; {}", message, engine.sfDir() != null ? "unacknowledged rows stay in on-disk store-and-forward for a later sender or an orphan drain" : "unacknowledged rows are lost when the sender closes (memory-only mode, no sf_dir)"); - long fromFsn = engine.ackedFsn() + 1L; - long toFsn = Math.max(fromFsn, engine.publishedFsn()); + // External FSNs, like every other SenderError span: after a symbol-dict recycle the engine numbers its + // frames from 0 again, and only externalFsnBase + raw matches what flushAndGetSequence() handed out. + long fromFsn = externalFsnBase + engine.ackedFsn() + 1L; + long toFsn = Math.max(fromFsn, externalFsnBase + engine.publishedFsn()); SenderError err = new SenderError( SenderError.Category.SECURITY_ERROR, SenderError.Policy.TERMINAL, diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/ConnectionHealthTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/ConnectionHealthTest.java index 6eb6513f9..c0d7f21b9 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/ConnectionHealthTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/ConnectionHealthTest.java @@ -34,6 +34,7 @@ import io.questdb.client.cutlass.qwp.client.QwpColumnBatchHandler; import io.questdb.client.cutlass.qwp.client.QwpEgressMsgKind; import io.questdb.client.cutlass.qwp.client.QwpQueryClient; +import io.questdb.client.cutlass.qwp.client.QwpWebSocketSender; import io.questdb.client.cutlass.qwp.client.sf.cursor.OrphanScanner; import io.questdb.client.cutlass.qwp.protocol.QwpConstants; import io.questdb.client.test.cutlass.qwp.client.WebSocketDynamicCredentialTest.ErrorCollector; @@ -223,6 +224,53 @@ public void testDeadlineIsResetByASuccessfulUpgradeAndUnsetMeansNoDeadline() thr }); } + @Test(timeout = 60_000) + public void testDeadlineSpanCarriesExternalFsnsAfterASymbolDictRecycle() throws Exception { + // C22 after a symbol-dict recycle: the rebuilt I/O loop arms the deadline, and the terminal error's span + // is on the sender's own FSN scale - what flushAndGetSequence() returned - not the rebuilt engine's. + assertMemoryLeak(() -> { + ErrorCollector errors = new ErrorCollector(); + try (TestWebSocketServer server = startServer(new WebSocketDynamicCredentialTest.AckHandler())) { + Sender sender = Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + server.getPort()) + .symbolDictResetThreshold(2) + .reconnectInitialBackoffMillis(10) + .reconnectMaxBackoffMillis(20) + .closeFlushTimeoutMillis(0) + .authFailureMaxDurationMillis(200) + .errorHandler(errors) + .build(); + try { + QwpWebSocketSender ws = (QwpWebSocketSender) sender; + sender.table("t").symbol("s", "a").longColumn("v", 1).atNow(); + sender.table("t").symbol("s", "b").longColumn("v", 1).atNow(); + long acked = sender.flushAndGetSequence(); + Assert.assertTrue(sender.awaitAckedFsn(acked, 10_000)); + // every handshake from here on is rejected, the rebuilt loop's first connect included + server.setRejectWithStatus(401, "Unauthorized"); + sender.table("t").symbol("s", "c").longColumn("v", 2).atNow(); // the recycle runs here + Assert.assertEquals("setup: the dictionary must have been recycled", 1, ws.getSymbolDictEpoch()); + Assert.assertTrue("setup: the recycle must move the FSN epoch", ws.getFsnEpochBaseForTesting() > 0); + long pending = sender.flushAndGetSequence(); + await(() -> errors.terminalCount() == 1, 10_000, "the deadline to fire on the rebuilt loop"); + SenderError terminal = terminal(errors); + Assert.assertTrue(terminal.getServerMessage(), + terminal.getServerMessage().contains("auth-rejected persisted for")); + Assert.assertEquals("fromFsn must follow the acknowledged FSN the sender reported", + acked + 1, terminal.getFromFsn()); + Assert.assertEquals("toFsn must be the last FSN the sender reported", pending, terminal.getToFsn()); + } finally { + server.setRejectWithStatus(0, null); + try { + sender.close(); + } catch (LineSenderException ignored) { + // close() may rethrow the terminal + } + } + } + }); + } + @Test(timeout = 60_000) public void testFacadeAggregatesEveryPooledConnection() throws Exception { assertMemoryLeak(() -> { diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/QwpQueryClientDynamicCredentialTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/QwpQueryClientDynamicCredentialTest.java index addb4f52e..d27ede3da 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/QwpQueryClientDynamicCredentialTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/QwpQueryClientDynamicCredentialTest.java @@ -129,6 +129,28 @@ public void testConnectPersistent401IsRetriedOnceThenFails() throws Exception { }); } + @Test(timeout = 30_000) + public void testConnectPersistent401WithAMintingProviderIsRetriedOnlyOnce() throws Exception { + // C11 on egress, with a provider that hands out a new token on every pull, so only the one-retry limit + // ends the round: the changed-token check alone would retry for as long as the server keeps rejecting. + assertMemoryLeak(() -> { + AtomicInteger minted = new AtomicInteger(); + try (TestWebSocketServer server = startServer(); + QwpQueryClient client = QwpQueryClient.fromConfig("ws::addr=localhost:" + server.getPort() + ";") + .withBearerTokenProvider(() -> mint(minted))) { + server.setAuthorizationValidator(h -> 401); + try { + client.connect(); + Assert.fail(); + } catch (QwpAuthFailedException e) { + Assert.assertEquals(401, e.getStatusCode()); + } + Assert.assertEquals(Arrays.asList("Bearer T1", "Bearer T2"), headers(server)); + Assert.assertEquals(2, minted.get()); + } + }); + } + @Test(timeout = 30_000) public void testConnectStaleTokenIsRetriedOnceWithTheRefreshedToken() throws Exception { // C10 on egress: one 401 for the stale token, then the same endpoint with the refreshed one. @@ -186,6 +208,45 @@ public void testFailedFailoverReconnectRecoversOnTheNextExecute() throws Excepti }); } + @Test(timeout = 30_000) + public void testFailedReconnectOnTheNextExecuteStaysRecoverable() throws Exception { + // Egress recovery (section 8.3) across consecutive failures: when the reconnect that the next execute() + // makes after a failed failover fails too, that operation reports it, and a later operation still + // reconnects instead of throwing "QwpQueryClient not connected" from then on. + assertMemoryLeak(() -> { + try (TestWebSocketServer a = startServer(); + TestWebSocketServer b = startServer(); + QwpQueryClient client = QwpQueryClient.fromConfig( + "ws::addr=localhost:" + a.getPort() + ",localhost:" + b.getPort() + + ";failover_backoff_initial_ms=0;") + .withBearerTokenProvider(() -> "TOKEN")) { + client.connect(); + b.setAuthorizationValidator(h -> 401); + a.close(); + ResultCollector failover = new ResultCollector(); + client.execute("SELECT 1", failover); + Assert.assertNotNull("the failover reconnect must fail", failover.error.get()); + Assert.assertFalse(client.isConnected()); + + // B still rejects: the reconnect this operation makes fails as well, reported to the handler + ResultCollector reconnect = new ResultCollector(); + client.execute("SELECT 1", reconnect); + String error = reconnect.error.get(); + Assert.assertNotNull("the operation must fail", error); + Assert.assertTrue(error, error.startsWith("reconnect failed: auth-rejected")); + Assert.assertFalse(client.isConnected()); + + // B accepts again: the next operation reconnects + b.setAuthorizationValidator(null); + ResultCollector recovered = new ResultCollector(); + client.execute("SELECT 1", recovered); + Assert.assertNull("execute() must reconnect: " + recovered.error.get(), recovered.error.get()); + Assert.assertTrue(recovered.done.get() > 0); + Assert.assertTrue(client.isConnected()); + } + }); + } + @Test(timeout = 30_000) public void testFailoverCredentialUnavailableNamesTheClass() throws Exception { assertMemoryLeak(() -> { @@ -212,6 +273,33 @@ public void testFailoverCredentialUnavailableNamesTheClass() throws Exception { }); } + @Test(timeout = 30_000) + public void testFailoverPersistent401WithAMintingProviderIsRetriedOnlyOnce() throws Exception { + // C11 on the egress failover path, with a provider that hands out a new token on every pull: B rejects + // every token, and the failover reconnect retries it exactly once before the operation fails. + assertMemoryLeak(() -> { + AtomicInteger minted = new AtomicInteger(); + try (TestWebSocketServer a = startServer(); + TestWebSocketServer b = startServer(); + QwpQueryClient client = QwpQueryClient.fromConfig( + "ws::addr=localhost:" + a.getPort() + ",localhost:" + b.getPort() + + ";failover_backoff_initial_ms=0;") + .withBearerTokenProvider(() -> mint(minted))) { + b.setAuthorizationValidator(h -> 401); + client.connect(); + Assert.assertEquals("Bearer T1", a.pollAuthorizationHeader(5, TimeUnit.SECONDS)); + a.close(); + ResultCollector result = new ResultCollector(); + client.execute("SELECT 1", result); + Assert.assertNotNull("the failover reconnect must fail", result.error.get()); + Assert.assertTrue(result.error.get(), + result.error.get().contains("auth-rejected during failover reconnect")); + Assert.assertEquals(Arrays.asList("Bearer T2", "Bearer T3"), headers(b)); + Assert.assertEquals(3, minted.get()); + } + }); + } + @Test(timeout = 30_000) public void testFailoverReconnectRetriesAStaleTokenOnce() throws Exception { // C10 on the egress failover path: B rejects the stale token; the failover reconnect refreshes and @@ -295,6 +383,16 @@ private static List headers(TestWebSocketServer server) throws Interrupt return headers; } + // A provider that hands out a new token on every pull. It gives up after ten, so a client that kept + // retrying fails the test's assertions instead of spinning until the timeout. + private static String mint(AtomicInteger minted) { + int n = minted.incrementAndGet(); + if (n > 10) { + throw TokenUnavailableException.permanent("test provider exhausted"); + } + return "T" + n; + } + private static TestWebSocketServer startServer() throws IOException, InterruptedException { TestWebSocketServer server = new TestWebSocketServer(new ExecDoneQueryServer()); server.setSendServerInfo(true); From 1b92465e70dc0023f23d4888a98978dd19cc5c03 Mon Sep 17 00:00:00 2001 From: glasstiger Date: Wed, 14 Oct 2026 00:58:31 +0100 Subject: [PATCH 09/40] Widen the retry budget of a credential test The SYNC budget-exhaustion test in WebSocketDynamicCredentialTest asserted more than two token pulls within a 300 ms connect budget, with 10-40 ms backoff parks between pulls. On the mac-aarch64 CI runner a timed park wakes about 100 ms late: the sibling test with the same backoff spent 196-504 ms in three parks across five mac runs, against 67-72 ms on linux and windows. The budget then fit only two pulls, and the assertion failed. Running the test JVM under background QoS (taskpolicy -b) reproduces the overshoot locally: a 10 ms park takes ~105 ms, the 300 ms budget yields exactly three pulls in every run, and added CPU load drops it to one or two. A 2 s budget yields 15-17 pulls under the same load. The test now gives the connect loop 2 s. Its assertions are unchanged and the client code is untouched: the loop honours the budget and retries; only the test assumed timely wake-ups. --- .../cutlass/qwp/client/WebSocketDynamicCredentialTest.java | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/WebSocketDynamicCredentialTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/WebSocketDynamicCredentialTest.java index cabf0e0db..47f35c3ec 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/WebSocketDynamicCredentialTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/WebSocketDynamicCredentialTest.java @@ -737,7 +737,10 @@ public void testSyncInitializationRetryableProviderFailureExhaustsTheBudgetNamin Sender.builder(Sender.Transport.WEBSOCKET) .address("localhost:" + server.getPort()) .initialConnectMode(Sender.InitialConnectMode.SYNC) - .reconnectMaxDurationMillis(300) + // The budget must fit the retries asserted below. Each backoff park is 10-40 ms + // nominally, but macOS timer coalescing on a busy CI host wakes a timed park + // ~100 ms late or more, which left room for only two pulls in a 300 ms budget. + .reconnectMaxDurationMillis(2_000) .reconnectInitialBackoffMillis(10) .reconnectMaxBackoffMillis(20) .httpTokenProvider(() -> { From 2dd97e9e91cf3ce8ecbbe54459447c8fb5fc4492 Mon Sep 17 00:00:00 2001 From: glasstiger Date: Wed, 14 Oct 2026 09:48:08 +0100 Subject: [PATCH 10/40] Fix a racy test counter and stale credential docs TestWebSocketServer counted a validator rejection only after it had flushed the 401, so a test that reads authRejectCount() as soon as the client has handled the rejection could see the count one short. That is how testPersistent401AfterTheRetryFailsStartup failed on the JDK 8 CI job (expected 2, was 1). The server now counts before it writes the response. With a 50 ms lag added after every reject, all 40 tests in the three classes that read the counter pass; without the reorder the same lag fails that test every time. testDeadlineIsNotResetOrFiredByOtherFailureClasses said that role rejects and transport failures do not reset the auth-outage clock, but it only waited for some terminal, so a loop that reset the clock in those branches still passed it. The test now parses the elapsed time from the terminal and requires it to span the role-reject and transport rounds. It fails against such a reset and passes without it. Comments and javadoc still said that an eager initial connect fails fast with the provider's own exception and that a running sender never latches a terminal on a credential outage. They now describe the SYNC retry of a retryable provider failure, the LineSenderException that reports an exhausted budget, the extra token pull after a 401 and the opt-in auth_failure_max_duration_millis deadline. The merge of main had also brought back, at the build() callsite, the recovery comment that #91 moved into constructEngineOnSlotLocked, and had dropped two #91 comments; main's text is restored there. --- .../io/questdb/client/HttpTokenProvider.java | 3 +- .../main/java/io/questdb/client/Sender.java | 50 +++++++++---------- .../QwpCredentialUnavailableException.java | 19 ++++--- .../cutlass/qwp/client/QwpQueryClient.java | 15 +++--- .../qwp/client/QwpWebSocketSender.java | 33 ++++++------ .../sf/cursor/CursorWebSocketSendLoop.java | 21 +++++--- .../qwp/client/ConnectionHealthTest.java | 13 +++++ .../qwp/websocket/TestWebSocketServer.java | 4 +- 8 files changed, 96 insertions(+), 62 deletions(-) diff --git a/core/src/main/java/io/questdb/client/HttpTokenProvider.java b/core/src/main/java/io/questdb/client/HttpTokenProvider.java index 02b318e68..d2a1792d2 100644 --- a/core/src/main/java/io/questdb/client/HttpTokenProvider.java +++ b/core/src/main/java/io/questdb/client/HttpTokenProvider.java @@ -53,7 +53,8 @@ * ({@code initial_connect_retry=on}) keeps retrying it within {@code reconnect_max_duration_millis}. Any other * exception counts as a permanent credential failure, so startup fails fast, as it does for an OIDC device-flow * provider that is not signed in yet. Once a sender has connected, every credential failure is retried while - * store-and-forward keeps the rows. + * store-and-forward keeps the rows, unless {@code auth_failure_max_duration_millis} is set: the sender then fails + * once such an outage has lasted that long. *

    * For a token that rotates on a schedule - a Microsoft Entra ID managed identity or service principal, an OAuth * client-credentials grant - wrap a {@link io.questdb.client.cutlass.auth.TokenSource} in a diff --git a/core/src/main/java/io/questdb/client/Sender.java b/core/src/main/java/io/questdb/client/Sender.java index 07fd6b90f..a6d270905 100644 --- a/core/src/main/java/io/questdb/client/Sender.java +++ b/core/src/main/java/io/questdb/client/Sender.java @@ -2218,7 +2218,9 @@ public LineSenderBuilder httpToken(String token) { * Over HTTP the provider is not called at build time: the first call happens when the first row is * started, then once per flush. Over WebSocket it depends on whether the initial connect is eager. * With an EAGER initial connect (the default, and any {@code initial_connect_retry} other than - * {@code async}) the handshake runs during {@code build()} and queries the provider once for it; + * {@code async}) the handshake runs during {@code build()} and queries the provider for it: once per + * connect attempt, plus once more when an endpoint rejects the token with {@code 401} (see + * {@link HttpTokenProvider#onTokenRejected(CharSequence, int)}); * under {@code lazy_connect=true} - or {@code initial_connect_retry=async} - the ingest side connects * asynchronously, so {@code build()} pulls nothing and a provider failure surfaces through the error * inbox rather than from {@code build()}. Either way the provider is queried again once per reconnect @@ -2226,10 +2228,14 @@ public LineSenderBuilder httpToken(String token) { * already-established WebSocket is not re-authenticated mid-stream. The two transports differ in * mechanism but both keep the producer alive across a sustained token outage: over HTTP a failed pull * leaves the request token-pending and is retried on the next row; over WebSocket an EAGER initial - * handshake fails fast when no token can be obtained, after which a pull that keeps failing on later - * reconnects is retried indefinitely, with the buffered rows held in store-and-forward, until a token - * is available again. A token outage does not terminate a running WebSocket sender, just as a - * persistent transport reconnect failure does not (store-and-forward Invariant B). + * handshake fails {@code build()} when no token can be obtained - at once in + * {@link InitialConnectMode#OFF} mode, and in {@link InitialConnectMode#SYNC} mode only after retrying a + * retryable {@link io.questdb.client.cutlass.auth.TokenUnavailableException} within + * {@code reconnect_max_duration_millis} - after which a pull that keeps failing on later reconnects is + * retried indefinitely, with the buffered rows held in store-and-forward, until a token is available + * again. A token outage does not terminate a running WebSocket sender, just as a persistent transport + * reconnect failure does not (store-and-forward Invariant B), unless + * {@link #authFailureMaxDurationMillis(long)} is set. *
    * Over HTTP the token is pulled once per request and written into the request buffer ahead of the * buffered rows, so a token refresh is picked up on the next new batch after a successful flush. A @@ -3658,30 +3664,19 @@ private QwpWebSocketSender buildWebSocket(HttpTokenProvider tokenProvider) { try (SlotLock logicalSlotLock = slotPath == null ? null : SlotLock.acquireLogical(slotPath)) { - // The constructor's own recovery seed can also fail terminally, and - // not only as UnreplayableSlotException: when SegmentRing.openExisting - // had to skip an unreadable segment it throws SfRecoveryException (it - // constructs UnreplayableSlotException nowhere), and where it cannot - // even prove the chain's identity -- no manifest -- it quarantines the - // corrupt files and returns an EMPTY recovery rather than refusing. - // Either way the frame range cannot be shown already-acked, so recovery - // sets the slot aside rather than risk seeding the ack cursor past - // frames that were never delivered. All three types below are load - // bearing; narrowing this catch to UnreplayableSlotException would - // restore the permanent build() brick for the segment-skip case. That verdict gets - // the exact same quarantine-and-continue treatment as the connect()-time - // verdict below -- constructing cursorEngine is not inside the loop below, - // so a throw here would otherwise escape build() entirely, uncaught. - // quarantineTornSlot(null, ...) renames the WHOLE slot directory aside - // (not just the unreadable segment file) before building the replacement - // at the original slotPath, so the replacement starts on a genuinely empty - // directory with nothing left to skip -- it cannot throw the same way - // twice, which is what makes looping unnecessary here. + // Recovery-verdict handling lives in constructEngineOnSlotLocked. ConstructedEngine constructed = constructEngineOnSlotLocked( sfDir, senderId, slotPath, actualSfMaxSegmentBytes, actualSfMaxTotalBytes, sfSharedBudget, actualSfAppendDeadlineNanos, actualSfSyncIntervalNanos, errorHandler); + // Seeded from constructEngineOnSlotLocked's own verdict, not + // hardcoded false: if construction already quarantined this + // slot, the connect loop below must count that as the one + // quarantine build() allows per attempt (see its "quarantined + // || slotPath == null" guard) rather than starting blind and + // risking a second quarantineTornSlot pass on what should be + // an immediate close-and-rethrow. boolean quarantined = constructed.quarantined; CursorSendEngine cursorEngine = constructed.engine; int actualErrorInboxCapacity = errorInboxCapacity != PARAMETER_NOT_SET_EXPLICITLY @@ -3791,9 +3786,12 @@ private QwpWebSocketSender buildWebSocket(HttpTokenProvider tokenProvider) { final String rebuildSfDir = sfDir; final String rebuildSenderId = senderId; final SenderErrorHandler buildTimeHandler = errorHandler; + // A rebuilt engine charges the same budget as the one it replaces, + // so a pooled sender stays under the pool's cap across a recycle. final SegmentBudget rebuildSharedBudget = sfSharedBudget; - // The recycle holds the slot's logical lock across its swap, so - // the rebuild must construct under that lock without retaking it. + // The recycle holds the slot's logical lock across its whole swap + // (QwpWebSocketSender.recycleForDictReset step 0), so the rebuild + // constructs under that lock and must not take it again. connected.setEngineRebuildFactory(new QwpWebSocketSender.EngineRebuildFactory() { @Override public CursorSendEngine rebuild() { diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpCredentialUnavailableException.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpCredentialUnavailableException.java index 963119527..f0ae39e11 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpCredentialUnavailableException.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpCredentialUnavailableException.java @@ -40,8 +40,10 @@ * the RUNNING store-and-forward drainer retries it indefinitely with capped backoff under * Invariant B -- the IdP becomes reachable again, or the user completes an interactive * sign-in -- holding the un-acked rows in SF meanwhile, and NEVER bounds it by - * {@code reconnectMaxDurationMillis} nor latches a terminal (either would drop a producer - * store-and-forward promised to keep alive). During initialization the classification + * {@code reconnectMaxDurationMillis} (that would drop a producer store-and-forward promised + * to keep alive). Only the optional authentication-outage deadline + * ({@code auth_failure_max_duration_millis}, off by default) latches a terminal once such an + * outage has lasted that long. During initialization the classification * decides: {@link #isRetryable()} is true only when the provider threw a * {@link TokenUnavailableException} marked retryable, which a SYNC initial connect keeps * retrying within its budget; any other provider failure is permanent and fails startup @@ -55,11 +57,14 @@ * {@code build()}, {@code flush()} or any row call delivers this type. The foreground * connects - SYNC in {@code CursorWebSocketSendLoop.connectWithRetry}, and the OFF-mode * connect in {@code QwpWebSocketSender} - both catch it and rethrow - * {@link #providerFailure()}, so a token-provider failure reaches the caller as the - * provider's own exception; the running background drainer catches it and retries under - * the invariant above. The query client ({@code QwpQueryClient}) does throw it from - * {@code connect()}, with a message that starts with {@code credential-unavailable}. It is - * public because both of those packages handle it, and + * {@link #providerFailure()} for a failure that fails startup fast, so that failure reaches + * the caller as the provider's own exception. A retryable failure that exhausts the SYNC + * budget instead surfaces as a {@code LineSenderException} whose message names the + * {@code credential-unavailable} class and whose cause is this exception. The running + * background drainer catches it and retries under the invariant above. The query client + * ({@code QwpQueryClient}) does throw it from {@code connect()}, with a message that starts + * with {@code credential-unavailable}. It is public because both of those packages handle + * it, and * because {@code QwpWebSocketSender.newReconnectFactory()} is public: a caller that * drives {@code ReconnectFactory.reconnect()} itself runs the endpoint walk directly and * so can receive this type unwrapped. Such a caller should treat it as the provider diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpQueryClient.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpQueryClient.java index 3e72e7d63..6609bd07c 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpQueryClient.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpQueryClient.java @@ -774,9 +774,12 @@ public void close() { * endpoints unreachable" (the latter surfaces as a plain * {@link HttpClientException}). *

    - * A configured token provider is queried once here, before the walk. A - * provider failure (not signed in, a failed silent refresh, a rejected - * token) is cluster-wide, so it fails fast with the provider's own error + * A configured token provider is queried once here, before the walk, and once + * more if an endpoint rejects its token with {@code 401}. A provider failure + * (not signed in, a failed silent refresh, a rejected token) is cluster-wide, + * so it fails fast, without contacting any endpoint, as a + * {@link QwpCredentialUnavailableException} whose message starts with + * {@code credential-unavailable} and whose cause is the provider's own error, * rather than being retried across endpoints as a transport failure. */ public synchronized void connect() { @@ -828,9 +831,9 @@ private void connectWalk() { Throwable lastTransportError = null; // Resolve the bearer credential once, before the endpoint walk: a token is cluster-wide, so a // token-provider failure (not signed in, a failed silent refresh, a rejected token) is not a - // per-endpoint transport fault. Resolving here lets it propagate as the provider's own error - // instead of being folded into "all endpoints unreachable", and avoids re-querying the provider - // once per endpoint. + // per-endpoint transport fault. Resolving here lets it propagate as a credential-unavailable failure + // that carries the provider's own error, instead of being folded into "all endpoints unreachable", + // and avoids re-querying the provider once per endpoint. String authHeader = resolveAuthorizationHeader(); // One same-endpoint retry after a refreshable 401 per connect, for a token provider only // (design/qwp-token-provider-spec.md, section 8.2). The retried attempt records no health penalty. diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpWebSocketSender.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpWebSocketSender.java index 1c90e286d..52137b3ad 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpWebSocketSender.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpWebSocketSender.java @@ -191,10 +191,12 @@ public class QwpWebSocketSender implements Sender { // fixed token or Basic credential; for an httpTokenProvider it pulls a freshly refreshed token, // so the initial connect and every reconnect re-handshake carry the current token. May be null // when no auth is configured. Evaluated once per (re)connect round in buildAndConnect, before the - // endpoint walk (not once per endpoint); a throwing provider is wrapped as - // QwpCredentialUnavailableException: the foreground/SYNC initial connect fails fast with the provider's - // own exception, while the running background drainer treats it as a transient outage and retries it - // indefinitely (never bounded by the reconnect budget, never terminal) per store-and-forward Invariant B. + // endpoint walk (not once per endpoint), and once more after a refreshable 401; a throwing provider is + // wrapped as QwpCredentialUnavailableException: the OFF initial connect fails fast with the provider's own + // exception, the SYNC one too unless the failure is retryable (then it retries within the reconnect + // budget), while the running background drainer treats it as a transient outage and retries it + // indefinitely (never bounded by the reconnect budget, terminal only through the opt-in + // auth_failure_max_duration_millis deadline) per store-and-forward Invariant B. private final Supplier authorizationHeaderSupplier; private final int autoFlushBytes; private final long autoFlushIntervalNanos; @@ -4185,11 +4187,13 @@ private WebSocketClient connectWalk(ReconnectSupplier ctx, CursorWebSocketSendLo // javadoc documents - NOT once per endpoint: a token-provider failure is cluster-wide (a failed silent // refresh, or not signed in), not a per-endpoint transport fault, so re-querying it per endpoint would // hammer the token endpoint with the same dead credential and mislabel the failure as "all endpoints - // unreachable". A throw here is wrapped as QwpCredentialUnavailableException (below): the - // foreground/SYNC initial connect unwraps it and fails fast with the provider's own message, while the + // unreachable". A throw here is wrapped as QwpCredentialUnavailableException (below): the OFF initial + // connect unwraps it and fails fast with the provider's own exception, and so does the SYNC one unless + // the provider classified the failure retryable, in which case SYNC retries it within its budget. The // running background drainer treats it as a transient outage and retries it indefinitely with capped - // backoff (never bounded by the reconnect budget, never terminal), so even a persistent credential - // outage keeps the buffered rows in store-and-forward rather than terminating the sender (Invariant B). + // backoff (never bounded by the reconnect budget; terminal only once the opt-in + // auth_failure_max_duration_millis deadline expires), so a persistent credential outage keeps the + // buffered rows in store-and-forward rather than terminating the sender (Invariant B). // Mirrors QwpQueryClient, which likewise resolves the credential once before its endpoint walk. // Publish this thread as being inside the pull BEFORE making it, then re-check cancellation, exactly // as the per-endpoint connect below does with the WebSocketClient. The pull is the one blocking call @@ -4208,12 +4212,13 @@ private WebSocketClient connectWalk(ReconnectSupplier ctx, CursorWebSocketSendLo authHeader = authorizationHeaderSupplier == null ? null : authorizationHeaderSupplier.get(); } catch (RuntimeException e) { // Tag the failure CLASS so each context applies the right policy: a credential we cannot acquire is - // not a transport outage. A foreground/SYNC connect unwraps this and rethrows the provider's own - // exception, so build() surfaces the provider's error directly rather than an internal wrapper. The - // running background drainer, by contrast, treats it as a transient outage and retries it - // indefinitely under Invariant B -- never bounding it by the reconnect budget, never latching a - // terminal -- so a recoverable credential outage never drops a producer store-and-forward promised - // to keep alive. + // not a transport outage. A foreground connect that fails startup on it unwraps this and rethrows the + // provider's own exception, so build() surfaces the provider's error directly rather than an internal + // wrapper; a SYNC connect retries a retryable one within its budget first. The running background + // drainer, by contrast, treats it as a transient outage and retries it indefinitely under Invariant B + // -- never bounding it by the reconnect budget, and latching a terminal only when the opt-in + // auth_failure_max_duration_millis deadline expires -- so a recoverable credential outage never drops + // a producer store-and-forward promised to keep alive. throw new QwpCredentialUnavailableException(e); } finally { // Drop the marker as soon as the pull returns, so a later cancel() cannot interrupt this thread diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoop.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoop.java index a415eb31d..74d024e7b 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoop.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoop.java @@ -2040,7 +2040,10 @@ private void connectWalk(Throwable initial, String phase, long paceFirstAttemptM // blocking (non-lazy) initial connect hands it to connectWithRetry (see // QwpWebSocketSender.ensureConnected, the SYNC branch), and // BackgroundDrainer converts it into the durable-ack capability-gap - // budget. Neither bounds this loop's steady-state reconnect. + // budget. Neither bounds this loop's steady-state reconnect. The one + // exception is opt-in: a FOREGROUND loop with + // auth_failure_max_duration_millis set latches a terminal once an + // authentication-class outage outlasts it (authOutageDeadlineFired). long backoffMillis = reconnectInitialBackoffMillis; if (paceFirstAttemptMillis > 0 && running) { // NACK-initiated recycle against a reachable server: pace the @@ -2213,11 +2216,14 @@ private void connectWalk(Throwable initial, String phase, long paceFirstAttemptM // background drainer this is a TRANSIENT outage like any other under Invariant B: the provider // hands over a token again once the IdP is reachable or the user finishes signing in, and the // un-acked rows stay safe in on-disk SF meanwhile. So retry indefinitely with capped backoff -- - // NEVER bound by a wall-clock budget and NEVER latch a terminal, which would drop a producer - // that store-and-forward promised to keep alive on a recoverable fault. The foreground/SYNC - // initial connect still fails fast with the provider's own exception (connectWithRetry, and the - // OFF-mode connect in QwpWebSocketSender), because a connectivity error is only the caller's to - // see DURING initialization, not after the drainer is running. + // NEVER bound by the reconnect budget and never latch a terminal of our own, which would drop a + // producer that store-and-forward promised to keep alive on a recoverable fault. The only + // terminal here is the one the user opted into: authOutageDeadlineFired() below, when + // auth_failure_max_duration_millis is set. The initial connect handles it in connectWithRetry + // (SYNC: a retryable failure is retried within the budget, anything else fails fast with the + // provider's own exception) and in the OFF-mode connect in QwpWebSocketSender (fails fast), + // because a connectivity error is only the caller's to see DURING initialization, not after the + // drainer is running. // // Ends any open cap-gap episode like every other unrelated reconnect state: we never reached a // node to observe its batch cap, so this outage's wall clock must not accrue toward the orphan @@ -2444,7 +2450,8 @@ public Throwable lastReconnectError() { * Once a FOREGROUND sender has reached the server even once, initialization is * over and store-and-forward owns the buffered data: every endpoint-policy * failure is then a transient the drainer must ride out, never a producer-fatal - * terminal (Invariant B). + * terminal (Invariant B). The opt-in authentication-outage deadline is separate + * from this gate: see {@code authOutageDeadlineFired}. */ private boolean endpointPolicyFailureIsTerminal() { return reconnectPolicy == ReconnectPolicy.ORPHAN || !hasEverConnected; diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/ConnectionHealthTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/ConnectionHealthTest.java index c0d7f21b9..b315702ad 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/ConnectionHealthTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/ConnectionHealthTest.java @@ -50,6 +50,8 @@ import java.util.concurrent.TimeUnit; import java.util.concurrent.atomic.AtomicBoolean; import java.util.concurrent.atomic.AtomicReference; +import java.util.regex.Matcher; +import java.util.regex.Pattern; import static io.questdb.client.test.cutlass.auth.TokenTestKit.await; import static io.questdb.client.test.tools.TestUtils.assertMemoryLeak; @@ -59,6 +61,8 @@ * 8.5) of the dynamic-credential specification (design/qwp-token-provider-spec.md). */ public class ConnectionHealthTest { + // The elapsed time a deadline terminal reports: "... persisted for ms ...". + private static final Pattern PERSISTED_FOR = Pattern.compile("persisted for (\\d+)ms"); private static final String SENTINEL = "SENTINEL-TOKEN-7f3a91"; @Rule @@ -103,11 +107,20 @@ public void testDeadlineIsNotResetOrFiredByOtherFailureClasses() throws Exceptio Assert.assertEquals("role rejects and transport failures must not fire the deadline", 0, errors.terminalCount()); // the credential outage returns: the clock was never reset, so the first such round fires + long failAt = System.nanoTime(); mode.set("fail"); await(() -> errors.terminalCount() == 1, 10_000, "the deadline to fire"); SenderError terminal = terminal(errors); Assert.assertTrue(terminal.getServerMessage(), terminal.getServerMessage().contains("credential-unavailable persisted for")); + // The clock started before authOutageStart was taken and the terminal fired after failAt, so the + // reported outage spans the role-reject and transport rounds. A clock those rounds had reset + // would report only the time since the credential outage returned. + Matcher persisted = PERSISTED_FOR.matcher(terminal.getServerMessage()); + Assert.assertTrue(terminal.getServerMessage(), persisted.find()); + Assert.assertTrue("role rejects and transport failures must not reset the clock: " + + terminal.getServerMessage(), + Long.parseLong(persisted.group(1)) >= TimeUnit.NANOSECONDS.toMillis(failAt - authOutageStart)); Assert.assertEquals(ConnectionHealth.State.FAILED, sender.health().getState()); } catch (LineSenderException expected) { // close() rethrows the latched terminal diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/websocket/TestWebSocketServer.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/websocket/TestWebSocketServer.java index 8df4c52d5..60c38616c 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/qwp/websocket/TestWebSocketServer.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/websocket/TestWebSocketServer.java @@ -749,8 +749,10 @@ private boolean performHandshake() throws IOException { if (validator != null) { int status = validator.applyAsInt(authorization); if (status > 0) { - writeStatusReject(status, status == 401 ? "Unauthorized" : status == 403 ? "Forbidden" : "Rejected"); + // Count before the client can observe the reject: a test reads authRejectCount() as soon as the + // client has handled the response, and counting after the flush races that read. authRejectCount.incrementAndGet(); + writeStatusReject(status, status == 401 ? "Unauthorized" : status == 403 ? "Forbidden" : "Rejected"); return false; } } From 8a012d69013b98782a7ae1ca571e61f1eb1d7b58 Mon Sep 17 00:00:00 2001 From: glasstiger Date: Wed, 14 Oct 2026 14:44:42 +0100 Subject: [PATCH 11/40] Sort health members and add a digit separator ConnectionHealth and QwpConnectionHealthTracker declared lastFailure before lastConnectedAt(EpochMillis), and ConnectionHealth listed getLastFailure() before getLastConnectedAtEpochMillis(). This commit puts the fields and getters in alphabetical order, as the member sorting convention requires. WsSenderConfigHonoredTest now writes the expected auth_failure_max_duration_millis value as 600_000L, using the thousands separator for numbers of five or more digits. No behaviour changes. ConnectionHealthTest and WsSenderConfigHonoredTest pass. --- .../java/io/questdb/client/ConnectionHealth.java | 16 ++++++++-------- .../qwp/client/QwpConnectionHealthTracker.java | 2 +- .../test/impl/WsSenderConfigHonoredTest.java | 2 +- 3 files changed, 10 insertions(+), 10 deletions(-) diff --git a/core/src/main/java/io/questdb/client/ConnectionHealth.java b/core/src/main/java/io/questdb/client/ConnectionHealth.java index 3ea731523..55f5af7f2 100644 --- a/core/src/main/java/io/questdb/client/ConnectionHealth.java +++ b/core/src/main/java/io/questdb/client/ConnectionHealth.java @@ -42,8 +42,8 @@ public final class ConnectionHealth { */ public static final long NONE = -1L; private final long failedRounds; - private final Failure lastFailure; private final long lastConnectedAtEpochMillis; + private final Failure lastFailure; private final long outageSinceEpochMillis; private final State state; @@ -74,18 +74,18 @@ public long getFailedRounds() { } /** - * The most recent failed connect round: its class, a sanitized message and its time. Kept after the - * connection recovers. Null when no round has failed. + * Wall-clock time of the most recent successful upgrade, or {@link #NONE}. */ - public Failure getLastFailure() { - return lastFailure; + public long getLastConnectedAtEpochMillis() { + return lastConnectedAtEpochMillis; } /** - * Wall-clock time of the most recent successful upgrade, or {@link #NONE}. + * The most recent failed connect round: its class, a sanitized message and its time. Kept after the + * connection recovers. Null when no round has failed. */ - public long getLastConnectedAtEpochMillis() { - return lastConnectedAtEpochMillis; + public Failure getLastFailure() { + return lastFailure; } /** diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpConnectionHealthTracker.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpConnectionHealthTracker.java index a26b681b4..bc483a257 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpConnectionHealthTracker.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpConnectionHealthTracker.java @@ -41,8 +41,8 @@ public final class QwpConnectionHealthTracker { private static final int MAX_CAUSE_DEPTH = 16; private final Object lock = new Object(); private long failedRounds; - private ConnectionHealth.Failure lastFailure; private long lastConnectedAt = ConnectionHealth.NONE; + private ConnectionHealth.Failure lastFailure; private long outageSince; private volatile ConnectionHealth snapshot; private ConnectionHealth.State state = ConnectionHealth.State.CONNECTING; diff --git a/core/src/test/java/io/questdb/client/test/impl/WsSenderConfigHonoredTest.java b/core/src/test/java/io/questdb/client/test/impl/WsSenderConfigHonoredTest.java index c71faa165..428dc1113 100644 --- a/core/src/test/java/io/questdb/client/test/impl/WsSenderConfigHonoredTest.java +++ b/core/src/test/java/io/questdb/client/test/impl/WsSenderConfigHonoredTest.java @@ -86,7 +86,7 @@ public void testEveryIngressKeyIsHonored() { assertHonored("connection_listener_inbox_capacity=64", "connection_listener_inbox_capacity", 64); assertHonored("token=ey.abc", "token", "ey.abc"); assertHonored("auth_timeout_ms=4321", "auth_timeout_ms", 4321L); - assertHonored("auth_failure_max_duration_millis=600000", "auth_failure_max_duration_millis", 600000L); + assertHonored("auth_failure_max_duration_millis=600000", "auth_failure_max_duration_millis", 600_000L); assertHonored("connect_timeout=7000", "connect_timeout", 7000); // username/password together (both-or-neither), and the user/pass aliases. From 2c4a79bc87b18293589f8167744f4f44c206bd06 Mon Sep 17 00:00:00 2001 From: glasstiger Date: Wed, 14 Oct 2026 16:20:39 +0100 Subject: [PATCH 12/40] Address token-provider review findings Four fixes from the level-3 review of this branch. README: the auth_failure_max_duration_millis note promised that unacknowledged rows stay on disk when the deadline fires. That holds only with sf_dir. A memory-only sender, the default, loses them when it closes, as the javadoc, the deadline's log line and spec section 8.5 already say. The README now names both outcomes. RefreshingTokenProvider: a fetch whose token expired within handout_floor (60 s) of its receipt counted as a success, although the provider can never hand such a token out. A source that issues tokens that short, such as a Keycloak realm at its 1-minute minimum access-token lifespan, left every caller waiting out cold_wait for a "no usable token" error with no cause, no WARN, and a provider that reported no failures. checkResult() now treats such a result as a retryable failed fetch whose message names handout_floor, so the WARN log, getLastFailure() and the caller's error all carry it. Spec v0.5 amends section 4 and conformance rows C2 and C6 to match, and the README documents the floor and handoutFloorMillis(). azure/pom.xml: surefire ignored -Dtest.include and -Dtest.exclude, so a questdb server build with the local-client profile ran all 26 Azure tests in every CI shard: about 35 s in each of the 15 Linux shards of OSS build 279062. The module now maps both properties the way core does. Its defaults still run every test. RefreshingTokenProviderTest: the C8 step for a stale token ran within forced_min_interval, so the rate limit returned first and the test passed with the stale-token check deleted. The step now runs once the interval has passed and asserts that no fetch is scheduled. Verification: deleting the stale-token check now fails C8. Reverting the handout_floor check fails the new hand-out-floor test and the two updated C2 and C6 tests. With -Dtest.include='**/griffin/**' the azure module runs no tests (26 before), and its default selection still runs all 26. --- README.md | 7 ++- azure/pom.xml | 11 ++++ .../cutlass/auth/RefreshingTokenProvider.java | 23 ++++++- .../auth/RefreshingTokenProviderTest.java | 61 ++++++++++++++++--- design/entra-id-qwp-auth.md | 2 +- design/qwp-token-provider-spec.md | 12 ++-- 6 files changed, 99 insertions(+), 17 deletions(-) diff --git a/README.md b/README.md index 374818029..e1d65a39d 100644 --- a/README.md +++ b/README.md @@ -558,6 +558,10 @@ A source reports a failure by throwing `TokenUnavailableException.retryable(...) retryable. Never put a token or a raw response body in the message. The provider keeps retrying with backoff and keeps serving the current token while it is still valid. +The provider hands a token out only while it has more than 60 s left (`handoutFloorMillis`, the hand-out floor). A +source whose token arrives with 60 s or less left gets a failed fetch whose error names `handout_floor`; if your +platform issues tokens that short, lower the floor with `handoutFloorMillis(...)`. + How failures are handled: - A token is refreshed at about half its lifetime, so an idle client always holds a valid one. @@ -567,7 +571,8 @@ How failures are handled: retryable provider failure within `reconnect_max_duration_millis`; `async` retries in the background. - Once connected, a store-and-forward sender rides out credential outages and `401`/`403` indefinitely, buffering rows and reporting each failure to the error handler. Set `auth_failure_max_duration_millis` to make the sender - fail instead once such an outage lasts that long (unacknowledged rows stay on disk). + fail instead once such an outage lasts that long. With `sf_dir`, unacknowledged rows then stay on disk for a + later sender or an orphan drain; without it, they are lost when the sender closes. ### Connection Health diff --git a/azure/pom.xml b/azure/pom.xml index 754ad21cb..bbeb51e55 100644 --- a/azure/pom.xml +++ b/azure/pom.xml @@ -68,6 +68,11 @@ 1.3.8 1.5.25 + + None + **/*Test.java @@ -126,6 +131,12 @@ false + + ${test.include} + + + ${test.exclude} + diff --git a/core/src/main/java/io/questdb/client/cutlass/auth/RefreshingTokenProvider.java b/core/src/main/java/io/questdb/client/cutlass/auth/RefreshingTokenProvider.java index 796eb3aea..efe2598b9 100644 --- a/core/src/main/java/io/questdb/client/cutlass/auth/RefreshingTokenProvider.java +++ b/core/src/main/java/io/questdb/client/cutlass/auth/RefreshingTokenProvider.java @@ -74,7 +74,9 @@ * (unless one is running or the provider is backing off after failures) and waits up to {@code cold_wait}, * then throws a {@link TokenUnavailableException} carrying the classification of the most recent failed * fetch, or a retryable one when no fetch failed. It never hands out an unusable token. Any number of - * concurrent callers cause at most one fetch. + * concurrent callers cause at most one fetch. A fetch whose token is unusable on arrival - it expires within + * {@code handout_floor} of its receipt - is a retryable failed fetch whose message names {@code handout_floor}: + * a source whose tokens are that short-lived needs a lower floor. *

  • Failures. A failed fetch is retried with jittered exponential backoff, from * {@code backoff_initial} up to {@code backoff_max}, honouring the source's {@code retry_after} up to * {@code retry_after_max}. Retries continue whatever the classification for as long as the provider is open, @@ -517,6 +519,15 @@ private FetchFailure checkResult(ExpiringToken result, long receivedAt) { + ", not after it was received at " + instant(receivedAt), receivedAt); } + if (!usable(result.getExpiresAtEpochMillis(), receivedAt)) { + // Not usable even on arrival, so it could never be handed out (section 4). Recorded as a success, it + // would leave every caller to wait out cold_wait for an error that names no cause. + return new FetchFailure(true, TokenUnavailableException.NO_RETRY_AFTER, + "token source returned a token that expires " + (result.getExpiresAtEpochMillis() - receivedAt) + + " ms after it was received, within handout_floor (" + handoutFloorMillis + + " ms): it can never be handed out; issue longer-lived tokens or lower handout_floor", + receivedAt); + } return null; } @@ -682,7 +693,11 @@ private TokenUnavailableException unavailableLocked(long now, long start) { } private boolean usable(Held h, long wallNow) { - return h != null && wallNow < h.expiresAtEpochMillis - handoutFloorMillis; + return h != null && usable(h.expiresAtEpochMillis, wallNow); + } + + private boolean usable(long expiresAtEpochMillis, long wallNow) { + return wallNow < expiresAtEpochMillis - handoutFloorMillis; } /** @@ -829,7 +844,9 @@ public Builder forcedWaitMillis(long millis) { } /** - * A token is usable only while {@code now < expires_at - handout_floor}. Default 60 s. + * A token is usable only while {@code now < expires_at - handout_floor}. Default 60 s. A fetched token that + * is not usable on arrival counts as a failed fetch, so a source whose tokens live no longer than this needs + * a lower value. */ public Builder handoutFloorMillis(long millis) { this.handoutFloorMillis = nonNegative("handout_floor", millis); diff --git a/core/src/test/java/io/questdb/client/test/cutlass/auth/RefreshingTokenProviderTest.java b/core/src/test/java/io/questdb/client/test/cutlass/auth/RefreshingTokenProviderTest.java index 629377106..22acc4a68 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/auth/RefreshingTokenProviderTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/auth/RefreshingTokenProviderTest.java @@ -303,16 +303,20 @@ public void testForcedRefreshIsRateLimitedIgnoresStaleTokensAndAcceptsTheSameTok Assert.assertEquals("T2", provider.getToken().toString()); Assert.assertEquals(2, source.calls()); - // a stale token - it has already rotated - is ignored - assertReturnsPromptly(() -> provider.onTokenRejected("T1", 401)); // rate limited: a second forced refresh within forced_min_interval is ignored clock.advanceMillis(30_000 - 1); assertReturnsPromptly(() -> provider.onTokenRejected("T2", 401)); Assert.assertNotEquals("no forced fetch may be scheduled", 0, scheduler.peek().delayNanos); Assert.assertEquals(2, source.calls()); - // once the interval has passed, a forced fetch that returns the same token is a normal success + // a stale token - it has already rotated - is ignored. The interval has now passed, so the rate limit + // no longer applies and only the stale-token check keeps this from forcing a fetch. clock.advanceMillis(1); + assertReturnsPromptly(() -> provider.onTokenRejected("T1", 401)); + Assert.assertNotEquals("a stale token must not schedule a fetch", 0, scheduler.peek().delayNanos); + Assert.assertEquals(2, source.calls()); + + // a forced fetch that returns the same token is a normal success rejecter = start(() -> provider.onTokenRejected("T2", 401)); await(() -> scheduler.peek() != null && scheduler.peek().delayNanos == 0, 5_000, "the second forced fetch to be scheduled"); @@ -357,7 +361,7 @@ public void testHandoutFloorTokenIsNeverHandedOut() throws Exception { @Test(timeout = 30_000) public void testHandoutFloorTokenIsNotHandedOutEvenWhenTheSourceRepeatsIt() throws Exception { // C6, second shape: the source keeps returning the token that sits inside the floor. The caller waits - // out cold_wait and fails rather than receive it. + // out cold_wait and fails rather than receive it, and its error names the cause (section 4). FakeClock clock = new FakeClock(T0); ManualScheduler scheduler = new ManualScheduler(clock); ScriptedSource source = new ScriptedSource().thenToken("T1", T0 + 10 * MIN); @@ -382,12 +386,52 @@ public void testHandoutFloorTokenIsNotHandedOutEvenWhenTheSourceRepeatsIt() thro caller.join(5_000); Assert.assertTrue("expected a token-unavailable error, got " + got.get(), got.get() instanceof TokenUnavailableException); - Assert.assertTrue("no fetch failed, so the error is retryable", - ((TokenUnavailableException) got.get()).isRetryable()); + TokenUnavailableException error = (TokenUnavailableException) got.get(); + Assert.assertTrue("the repeated token is a retryable failed fetch", error.isRetryable()); + Assert.assertTrue(error.getMessage(), error.getMessage().contains("within handout_floor (60000 ms)")); + Assert.assertEquals(1, provider.getConsecutiveFailures()); Assert.assertEquals("the caller started one fetch, not one per completed fetch", 2, source.calls()); } } + @Test + public void testHandoutFloorTokenOnArrivalIsARetryableFailure() { + // C6, section 4: a token that expires within handout_floor of its receipt could never be handed out, so + // its fetch fails - retryable, named in the failure record and in a caller's error - rather than count as + // a success that leaves every caller to wait out cold_wait for an error with no cause. + FakeClock clock = new FakeClock(T0); + ManualScheduler scheduler = new ManualScheduler(clock); + final long floor = RefreshingTokenProvider.DEFAULT_HANDOUT_FLOOR_MILLIS; + ScriptedSource source = new ScriptedSource() + .then(() -> new ExpiringToken("SHORT", clock.wallClockMillis() + floor)) + .then(() -> new ExpiringToken("EDGE", clock.wallClockMillis() + floor + 1)); + try (RefreshingTokenProvider provider = provider(source, clock, scheduler, 0.0) + .coldWaitMillis(0) + .build()) { + scheduler.runNext(); + Assert.assertEquals(1, provider.getConsecutiveFailures()); + RefreshingTokenProvider.FetchFailure failure = provider.getLastFailure(); + Assert.assertNotNull("a token that is unusable on arrival is a failed fetch", failure); + Assert.assertTrue(failure.isRetryable()); + Assert.assertTrue(failure.getMessage(), failure.getMessage().contains("within handout_floor (60000 ms)")); + Assert.assertEquals("an unusable token is never held", RefreshingTokenProvider.NONE, + provider.getTokenExpiresAtEpochMillis()); + try { + provider.getToken(); + Assert.fail("expected a token-unavailable error"); + } catch (TokenUnavailableException e) { + Assert.assertTrue(e.isRetryable()); + Assert.assertTrue("the caller's error must name the cause: " + e.getMessage(), + e.getMessage().contains("within handout_floor (60000 ms)")); + } + + // retried after backoff, not dropped: one millisecond beyond the floor is usable + scheduler.advanceAndRunNext(); + Assert.assertEquals("EDGE", provider.getToken().toString()); + Assert.assertEquals(0, provider.getConsecutiveFailures()); + } + } + @Test(timeout = 30_000) public void testInterruptedWaitEndsPromptlyAndKeepsTheFlag() throws Exception { // C7: a cancelled wait returns promptly, and the cancellation signal is preserved. @@ -515,7 +559,10 @@ public void testSameTokenConvergesAndStaysUsable() { } // 24 h halves to below the 60 s floor in 11 steps (86400 s -> ... -> 84.4 s) Assert.assertEquals(11, fetchesWhileUsable); - Assert.assertEquals(0, provider.getConsecutiveFailures()); + // the fetch that returned the token inside the floor is a failed fetch (section 4) + Assert.assertEquals(1, provider.getConsecutiveFailures()); + Assert.assertTrue(provider.getLastFailure().getMessage(), + provider.getLastFailure().getMessage().contains("within handout_floor")); } } diff --git a/design/entra-id-qwp-auth.md b/design/entra-id-qwp-auth.md index 0832c3473..def18b771 100644 --- a/design/entra-id-qwp-auth.md +++ b/design/entra-id-qwp-auth.md @@ -4,7 +4,7 @@ Status: **implemented** on `feat/qwp-entra-token-provider` - all four steps of t **Read these first:** -- **The spec wins.** The normative, language-neutral contract is now `design/qwp-token-provider-spec.md` (v0.4, all decisions resolved). Where the two documents differ, follow the spec. That includes the refresh-schedule clamps, backoff jitter, registry linger, and the Java names in its Appendix B, which supersede §7.2 here. +- **The spec wins.** The normative, language-neutral contract is now `design/qwp-token-provider-spec.md` (v0.5, all decisions resolved). Where the two documents differ, follow the spec. That includes the refresh-schedule clamps, backoff jitter, registry linger, and the Java names in its Appendix B, which supersede §7.2 here. - **Java plan:** §10. - **Line references** are as of `c9968f24` and may drift. diff --git a/design/qwp-token-provider-spec.md b/design/qwp-token-provider-spec.md index 3df1607aa..c9a5d669a 100644 --- a/design/qwp-token-provider-spec.md +++ b/design/qwp-token-provider-spec.md @@ -1,9 +1,10 @@ # QWP dynamic bearer credentials: cross-language specification -**Status:** v0.4, ready for implementation. All decisions are resolved (§12). +**Status:** v0.5, ready for implementation. All decisions are resolved (§12). **Changes:** +- **v0.5:** a result whose token expires within `handout_floor` of its receipt is a retryable failed fetch whose message names `handout_floor` (§4, C2, C6). Before, it counted as a success that could never be handed out, so every caller waited out `cold_wait` for an error that named no cause. - **v0.4:** `azure_credential` selects one Azure credential deterministically (§7.1), so the library itself reports an unreachable managed-identity endpoint (retryable) apart from a misconfigured credential (permanent) (§7.5, D10). A discovery chain's "no credential available" stays permanent, so a host without a credential still fails fast, and the client warns once that the chain does not ride out an endpoint outage. This resolves the conflict between §4 and §7.5 over an unreachable managed-identity endpoint. - **v0.3:** decisions resolved, with D8 (§8.1) and D9 (§7.2) added; Java binding decided (Appendix B). - **v0.2:** connection health (§8.4), the optional authentication-outage deadline (§8.5), `WWW-Authenticate` handling (§8.2), and Appendices C and D. @@ -97,9 +98,10 @@ Classification: - **No concurrency:** a provider never calls `fetch()` concurrently with itself. - **Where it runs:** implementations SHOULD run `fetch()` on a background context owned by the provider, never on a connection thread. - **Blocking:** `fetch()` MAY block, but it SHOULD bound its own network operations. 30 s per attempt is RECOMMENDED. -- **Bad results:** the provider MUST treat a result as a retryable error in either of these cases: +- **Bad results:** the provider MUST treat a result as a retryable error in any of these cases: - the token fails the checks in §3; - - `expires_at` is not later than the time the result was received. + - `expires_at` is not later than the time the result was received; + - the token is not usable (§5.1) at the time the result was received: `expires_at` is no more than `handout_floor` after it. Such a token could never be handed out (§5.3). The error message MUST name `handout_floor`, so that a platform issuing tokens this short-lived is diagnosed rather than waited on. ## 5. Refreshing provider @@ -426,11 +428,11 @@ Every client that implements this specification SHOULD pass these scenarios. The | ID | Scenario | Expected result | |---|---|---| | C1 | Refresh schedule | Refresh times follow §5.2 for `L` = 24 h, 60 min and 5 min, with and without `refresh_at`. | -| C2 | Same token returned | Fetches converge as described in §5.2, and the token stays usable. | +| C2 | Same token returned | Fetches converge as described in §5.2, and the token stays usable until it enters `handout_floor`. The fetch that returns it inside the floor is a retryable failure (§4). | | C3 | Cold burst | 64 concurrent callers with no usable token cause exactly one fetch. | | C4 | Cold failure | A retryable error is raised within `cold_wait`. A permanent classification is passed through. | | C5 | Backoff | Delays follow §5.4. `retry_after` is honoured and capped. | -| C6 | Hand-out floor | A token inside `handout_floor` is never handed out. | +| C6 | Hand-out floor | A token inside `handout_floor` is never handed out. A result whose token is already inside it on arrival is a retryable failure whose message names `handout_floor` (§4); one millisecond more is usable. | | C7 | Cancellation | A cancelled wait returns promptly, and the cancellation signal is preserved. | | C8 | Forced refresh | Rate limited. A stale token is ignored. A same-token result is accepted. | | C9 | Reconnect after expiry | The reconnect carries the refreshed token, and no `401` is observed. | From 15fe20207563b96ce4005c6ff714a9bc05c6e863 Mon Sep 17 00:00:00 2001 From: glasstiger Date: Wed, 14 Oct 2026 16:36:25 +0100 Subject: [PATCH 13/40] Fix a racy scheduler check in provider tests RefreshingTokenProviderTest failed on the JDK 8 CI job with a NullPointerException in the condition of the await() at line 298: await(() -> scheduler.peek() != null && scheduler.peek().delayNanos == 0, ...) The condition reads ManualScheduler.peek() twice while another thread makes the provider replace its scheduled task. scheduleFetchLocked() cancels the pending refresh before it schedules the forced fetch, and peek() returns live tasks only, so between those two steps the scheduler holds none: the first peek() saw the old task and the second returned null. ManualScheduler.isNextImmediate() now reads the task list once. The four awaits that poll while a rejecter or cold-caller thread runs, two in the C8 forced-refresh test and one in each hand-out-floor test, use it. Every other peek() dereference runs after the background call has been joined, or on the test thread alone. With a 100 ms pause added between cancel() and schedule() in the provider and a 20 ms pause in peek(), the old condition failed 3 of 3 runs with the CI stack trace, and the new one passed 3 of 3. Without the pauses, RefreshingTokenProviderTest passes 5 of 5 runs. The race dates from c9f2b96a; 2c4a79bc did not change these lines. --- .../cutlass/auth/RefreshingTokenProviderTest.java | 12 ++++-------- .../client/test/cutlass/auth/TokenTestKit.java | 9 +++++++++ 2 files changed, 13 insertions(+), 8 deletions(-) diff --git a/core/src/test/java/io/questdb/client/test/cutlass/auth/RefreshingTokenProviderTest.java b/core/src/test/java/io/questdb/client/test/cutlass/auth/RefreshingTokenProviderTest.java index 22acc4a68..795ada15c 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/auth/RefreshingTokenProviderTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/auth/RefreshingTokenProviderTest.java @@ -295,8 +295,7 @@ public void testForcedRefreshIsRateLimitedIgnoresStaleTokensAndAcceptsTheSameTok // 401 for the current token: a fetch starts now and the caller waits for it Thread rejecter = start(() -> provider.onTokenRejected("T1", 401)); - await(() -> scheduler.peek() != null && scheduler.peek().delayNanos == 0, 5_000, - "the forced fetch to be scheduled"); + await(scheduler::isNextImmediate, 5_000, "the forced fetch to be scheduled"); scheduler.runNext(); rejecter.join(5_000); Assert.assertFalse("onTokenRejected must return once the forced fetch completes", rejecter.isAlive()); @@ -318,8 +317,7 @@ public void testForcedRefreshIsRateLimitedIgnoresStaleTokensAndAcceptsTheSameTok // a forced fetch that returns the same token is a normal success rejecter = start(() -> provider.onTokenRejected("T2", 401)); - await(() -> scheduler.peek() != null && scheduler.peek().delayNanos == 0, 5_000, - "the second forced fetch to be scheduled"); + await(scheduler::isNextImmediate, 5_000, "the second forced fetch to be scheduled"); scheduler.runNext(); rejecter.join(5_000); Assert.assertFalse(rejecter.isAlive()); @@ -349,8 +347,7 @@ public void testHandoutFloorTokenIsNeverHandedOut() throws Exception { // exactly at expires_at - handout_floor the token is unusable: the caller must get a new one AtomicReference got = new AtomicReference<>(); Thread caller = start(() -> got.set(provider.getToken().toString())); - await(() -> scheduler.peek() != null && scheduler.peek().delayNanos == 0, 5_000, - "the cold caller to start a fetch"); + await(scheduler::isNextImmediate, 5_000, "the cold caller to start a fetch"); Assert.assertNull("nothing may be handed out before the fetch", got.get()); scheduler.runNext(); caller.join(5_000); @@ -378,8 +375,7 @@ public void testHandoutFloorTokenIsNotHandedOutEvenWhenTheSourceRepeatsIt() thro got.set(e); } }); - await(() -> scheduler.peek() != null && scheduler.peek().delayNanos == 0, 5_000, - "the cold caller to start a fetch"); + await(scheduler::isNextImmediate, 5_000, "the cold caller to start a fetch"); scheduler.runNext(); // the source returns T1 again, still inside the floor Assert.assertNull("an unusable token must not be handed out", got.get()); clock.advanceMillis(1_001); // cold_wait elapses diff --git a/core/src/test/java/io/questdb/client/test/cutlass/auth/TokenTestKit.java b/core/src/test/java/io/questdb/client/test/cutlass/auth/TokenTestKit.java index 41f3d1bc5..24c600a54 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/auth/TokenTestKit.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/auth/TokenTestKit.java @@ -126,6 +126,15 @@ public Task advanceAndRunNext() { return t; } + /** + * Whether the earliest live task was scheduled with no delay. Reads the task list once: the provider cancels + * a task before it schedules its replacement, so a second {@link #peek()} can find no task at all. + */ + public boolean isNextImmediate() { + Task next = peek(); + return next != null && next.delayNanos == 0; + } + public boolean isShutdown() { return shutdown; } From a38afc31ff1ef368f037dc0587c7370d1a821a70 Mon Sep 17 00:00:00 2001 From: glasstiger Date: Wed, 14 Oct 2026 18:45:29 +0100 Subject: [PATCH 14/40] Test the lease release on a failed sender build LineSenderBuilder.build() takes a TokenProviderRegistry lease before it connects a token_provider sender, and its finally block closes the lease when buildWebSocket() throws. No test covered that path: with the lease.close() call deleted, the core and azure suites still passed, while the failed build left the lease held, so the shared provider's refresher thread would keep fetching tokens for the life of the process. TokenProviderSharingTest gains testSenderBuildFailureReleasesTheLease. It builds a token_provider sender against a port nothing listens on, expects the connect failure, and checks that build() created the provider, that the lease count is back to zero, and that the provider lingers for the next attempt. With the lease.close() call deleted the test fails with "expected:<0> but was:<1>". WsSenderConfigHonoredTest now lists its TestTokenProviderFactory import after io.questdb.client.impl.Side, in alphabetical order. --- .../qwp/client/TokenProviderSharingTest.java | 23 +++++++++++++++++++ .../test/impl/WsSenderConfigHonoredTest.java | 2 +- 2 files changed, 24 insertions(+), 1 deletion(-) diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/TokenProviderSharingTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/TokenProviderSharingTest.java index 6f1c897ef..d65037200 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/TokenProviderSharingTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/TokenProviderSharingTest.java @@ -29,6 +29,7 @@ import io.questdb.client.cutlass.auth.ExpiringToken; import io.questdb.client.cutlass.auth.TokenProviderRegistry; import io.questdb.client.cutlass.auth.TokenProviderSpec; +import io.questdb.client.cutlass.line.LineSenderException; import io.questdb.client.cutlass.qwp.client.QwpQueryClient; import io.questdb.client.impl.ConfigString; import io.questdb.client.impl.ConfigView; @@ -81,6 +82,28 @@ public void testFacadePoolsShareOneProvider() throws Exception { }); } + @Test(timeout = 60_000) + public void testSenderBuildFailureReleasesTheLease() throws Exception { + assertMemoryLeak(() -> { + TestTokenProviderFactory azure = new TestTokenProviderFactory("azure"); + TestTokenProviderFactory.install(azure); + // nothing listens on the port: build() takes the lease, then fails to connect + String cfg = "wss::addr=localhost:" + TestPorts.findUnusedPort() + ";tls_verify=unsafe_off;" + + "token_provider=azure;azure_resource=api://c17-failed-build-" + System.nanoTime() + ';'; + TokenProviderSpec spec = spec(cfg); + try (Sender ignored = Sender.fromConfig(cfg)) { + Assert.fail("build() must fail: nothing listens on " + cfg); + } catch (LineSenderException expected) { + // the connect failure + } + Assert.assertEquals("build() took the lease: it created the provider", 1, azure.created.get()); + Assert.assertEquals("a failed build releases its lease", + 0, TokenProviderRegistry.global().leaseCount(spec)); + Assert.assertTrue("the provider lingers for the next attempt", + TokenProviderRegistry.global().isActive(spec)); + }); + } + @Test(timeout = 60_000) public void testSendersAndQueryClientsShareOneProviderWithOneFetchPerRefresh() throws Exception { assertMemoryLeak(() -> { diff --git a/core/src/test/java/io/questdb/client/test/impl/WsSenderConfigHonoredTest.java b/core/src/test/java/io/questdb/client/test/impl/WsSenderConfigHonoredTest.java index 428dc1113..d5753c6b9 100644 --- a/core/src/test/java/io/questdb/client/test/impl/WsSenderConfigHonoredTest.java +++ b/core/src/test/java/io/questdb/client/test/impl/WsSenderConfigHonoredTest.java @@ -26,8 +26,8 @@ import io.questdb.client.Sender; import io.questdb.client.impl.ConfigSchema; -import io.questdb.client.test.cutlass.auth.TestTokenProviderFactory; import io.questdb.client.impl.Side; +import io.questdb.client.test.cutlass.auth.TestTokenProviderFactory; import org.junit.Assert; import org.junit.Test; From 33200e93d1adae6b0324cbe4ddfe9e02bf4444b3 Mon Sep 17 00:00:00 2001 From: glasstiger Date: Wed, 14 Oct 2026 19:33:24 +0100 Subject: [PATCH 15/40] Remove the authentication-outage deadline The opt-in auth_failure_max_duration_millis let a running store-and-forward sender become terminal once credential-unavailable or 401/403 rounds had lasted that long. Its clock started at the first such failure and kept running through role-reject and transport rounds, so a failover between two brief credential failures could end a producer whose authentication never failed for long. A store-and-forward sender must never become terminal on a transient outage, so the deadline is removed. An established sender again retries every authentication-class failure indefinitely, and reports each one to the error handler and in its connection health. An application that prefers a hard failure closes the sender itself. The connect-string key, the builder method and the send loop's outage clock are gone, and the spec moves to v0.6: section 8.5, D7 and C22 now rule such a deadline out. ConnectionHealthTest replaces the deadline tests with testAuthenticationOutageNeverMakesTheSenderTerminal. It drives credential-unavailable, 401 and role-reject rounds, checks that the sender stays alive, and checks that it delivers the buffered row after recovery. With a terminal latched on a credential failure in the send loop, it fails with "expected: but was:". --- README.md | 6 +- .../io/questdb/client/HttpTokenProvider.java | 4 +- .../main/java/io/questdb/client/Sender.java | 45 +-- .../QwpCredentialUnavailableException.java | 6 +- .../qwp/client/QwpWebSocketSender.java | 29 +- .../sf/cursor/CursorWebSocketSendLoop.java | 103 +------ .../io/questdb/client/impl/ConfigSchema.java | 3 - .../qwp/client/ConnectionHealthTest.java | 276 ++++-------------- .../test/impl/WsSenderConfigHonoredTest.java | 1 - design/entra-id-qwp-auth.md | 6 +- design/qwp-token-provider-spec.md | 33 +-- 11 files changed, 87 insertions(+), 425 deletions(-) diff --git a/README.md b/README.md index e1d65a39d..a74ad4236 100644 --- a/README.md +++ b/README.md @@ -570,9 +570,8 @@ How failures are handled: - At startup, `initial_connect_retry=off` (the default) fails if no token can be obtained; `on` keeps retrying a retryable provider failure within `reconnect_max_duration_millis`; `async` retries in the background. - Once connected, a store-and-forward sender rides out credential outages and `401`/`403` indefinitely, buffering - rows and reporting each failure to the error handler. Set `auth_failure_max_duration_millis` to make the sender - fail instead once such an outage lasts that long. With `sf_dir`, unacknowledged rows then stay on disk for a - later sender or an orphan drain; without it, they are lost when the sender closes. + rows and reporting each failure to the error handler and in its connection health. It never fails on its own: an + application that prefers a hard failure watches the health or the error handler, and closes the sender itself. ### Connection Health @@ -640,7 +639,6 @@ schema::key1=value1;key2=value2; | `azure_resource` | | `token_provider=azure`: application ID URI or client ID of the QuestDB app registration | | `azure_client_id` | | `token_provider=azure`: client ID of a user-assigned managed identity or workload identity | | `azure_credential` | `default` | `token_provider=azure`: `default`, `managed_identity`, `workload_identity` or `environment` | -| `auth_failure_max_duration_millis` | _(none)_ | Ingest: fail the sender once a credential outage or `401`/`403` lasts this long | ### Pool keys (facade only) diff --git a/core/src/main/java/io/questdb/client/HttpTokenProvider.java b/core/src/main/java/io/questdb/client/HttpTokenProvider.java index d2a1792d2..71b9d5892 100644 --- a/core/src/main/java/io/questdb/client/HttpTokenProvider.java +++ b/core/src/main/java/io/questdb/client/HttpTokenProvider.java @@ -53,8 +53,8 @@ * ({@code initial_connect_retry=on}) keeps retrying it within {@code reconnect_max_duration_millis}. Any other * exception counts as a permanent credential failure, so startup fails fast, as it does for an OIDC device-flow * provider that is not signed in yet. Once a sender has connected, every credential failure is retried while - * store-and-forward keeps the rows, unless {@code auth_failure_max_duration_millis} is set: the sender then fails - * once such an outage has lasted that long. + * store-and-forward keeps the rows; the sender reports each failure to its error handler and in its connection + * health, and never fails on its own. *

    * For a token that rotates on a schedule - a Microsoft Entra ID managed identity or service principal, an OAuth * client-credentials grant - wrap a {@link io.questdb.client.cutlass.auth.TokenSource} in a diff --git a/core/src/main/java/io/questdb/client/Sender.java b/core/src/main/java/io/questdb/client/Sender.java index a6d270905..b481960cf 100644 --- a/core/src/main/java/io/questdb/client/Sender.java +++ b/core/src/main/java/io/questdb/client/Sender.java @@ -1115,8 +1115,6 @@ final class LineSenderBuilder { OrphanScanner.QUARANTINE_SLOT_INFIX; private final ObjList hosts = new ObjList<>(); private final IntList ports = new IntList(); - // Optional authentication-outage deadline (spec section 8.5); 0 = not set, the default. - private long authFailureMaxDurationMillis; private long authTimeoutMillis = QwpWebSocketSender.DEFAULT_AUTH_TIMEOUT_MS; private int autoFlushBytes = PARAMETER_NOT_SET_EXPLICITLY; private int autoFlushIntervalMillis = PARAMETER_NOT_SET_EXPLICITLY; @@ -1360,34 +1358,6 @@ public LineSenderBuilder connectTimeoutMillis(int millis) { return this; } - /** - * Optional authentication-outage deadline (design/qwp-token-provider-spec.md, section 8.5): the sender - * becomes terminal when a connect round fails with an authentication-class failure - the token provider - * could not supply a credential, or an endpoint answered {@code 401}/{@code 403} - once that outage has - * lasted this long. Not set by default: such outages are retried indefinitely while store-and-forward - * keeps the rows. The clock starts at the first authentication-class failure of an outage and only a - * successful upgrade resets it; failures of other classes neither reset nor fire it. Applies while the - * sender is established and during an {@code async} initial connect; orphan drains are unaffected. When - * it fires, the error names the failure class and the elapsed time, goes to the error handler as - * terminal, and is thrown from later producer calls. With {@code sf_dir} set, unacknowledged rows stay - * in on-disk store-and-forward for a later sender or an orphan drain; in memory-only mode (no - * {@code sf_dir}) they are lost when the sender closes. WebSocket transport only. Connect-string key: - * {@code auth_failure_max_duration_millis}. - * - * @param millis the deadline, {@code > 0} - * @return this instance for method chaining - */ - public LineSenderBuilder authFailureMaxDurationMillis(long millis) { - if (protocol != PARAMETER_NOT_SET_EXPLICITLY && protocol != PROTOCOL_WEBSOCKET) { - throw new LineSenderException("auth_failure_max_duration_millis is only supported for WebSocket transport"); - } - if (millis <= 0) { - throw new LineSenderException("auth_failure_max_duration_millis must be > 0: ").put(millis); - } - this.authFailureMaxDurationMillis = millis; - return this; - } - /** * Per-endpoint timeout on the WebSocket upgrade response read. Default * {@value QwpWebSocketSender#DEFAULT_AUTH_TIMEOUT_MS} ms. @@ -2234,8 +2204,8 @@ public LineSenderBuilder httpToken(String token) { * {@code reconnect_max_duration_millis} - after which a pull that keeps failing on later reconnects is * retried indefinitely, with the buffered rows held in store-and-forward, until a token is available * again. A token outage does not terminate a running WebSocket sender, just as a persistent transport - * reconnect failure does not (store-and-forward Invariant B), unless - * {@link #authFailureMaxDurationMillis(long)} is set. + * reconnect failure does not (store-and-forward Invariant B): an application that prefers a hard failure + * watches {@link Sender#health()} or its error handler, and closes the sender itself. *
    * Over HTTP the token is pulled once per request and written into the request buffer ahead of the * buffered rows, so a token refresh is picked up on the next new batch after a successful flush. A @@ -3777,11 +3747,6 @@ private QwpWebSocketSender buildWebSocket(HttpTokenProvider tokenProvider) { // dispatcher daemon, drainer pool, microbatch buffers and // WebSocketClient inside the abandoned `connected`. connected.setTransactional(transactional); - if (authFailureMaxDurationMillis > 0) { - // The I/O loop may already be running (async initial connect), but it tracks the outage clock - // from the first authentication-class failure regardless; this only arms the deadline. - connected.setAuthFailureMaxDurationMillis(authFailureMaxDurationMillis); - } try { final String rebuildSfDir = sfDir; final String rebuildSenderId = senderId; @@ -4411,8 +4376,6 @@ private LineSenderBuilder fromConfig(CharSequence configurationString) { // genuine value-parse error names the offending key. String reservedKey = Chars.toString(sink); pos = getValue(configurationString, pos, sink, reservedKey); - } else if (Chars.equals("auth_failure_max_duration_millis", sink)) { - throw new LineSenderException("auth_failure_max_duration_millis is only supported for WebSocket transport"); } else if (Chars.equals("token_provider", sink) || Chars.equals("azure_resource", sink) || Chars.equals("azure_client_id", sink) @@ -4512,9 +4475,6 @@ private LineSenderBuilder fromConfigWebSocket(CharSequence configurationString) // int, and an over-int value must reject, not wrap. connectTimeoutMillis(view.getInt("connect_timeout", 0)); } - if (view.has("auth_failure_max_duration_millis")) { - authFailureMaxDurationMillis(view.getLong("auth_failure_max_duration_millis", 0)); - } s = view.getStr("auto_flush_rows"); if (s != null) { @@ -4814,7 +4774,6 @@ public java.util.Map wsConfigSnapshotForTest() { m.put("tls_verify", tlsValidationMode == null ? null : tlsValidationMode.name()); m.put("tls_roots", trustStorePath); m.put("tls_roots_password", trustStorePassword == null ? null : new String(trustStorePassword)); - m.put("auth_failure_max_duration_millis", authFailureMaxDurationMillis); m.put("token_provider", tokenProviderSpec == null ? null : tokenProviderSpec.name()); m.put("azure_resource", tokenProviderSpec == null ? null : tokenProviderSpec.params().get(TokenProviderSpec.KEY_AZURE_RESOURCE)); diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpCredentialUnavailableException.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpCredentialUnavailableException.java index f0ae39e11..8003fc89b 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpCredentialUnavailableException.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpCredentialUnavailableException.java @@ -40,10 +40,8 @@ * the RUNNING store-and-forward drainer retries it indefinitely with capped backoff under * Invariant B -- the IdP becomes reachable again, or the user completes an interactive * sign-in -- holding the un-acked rows in SF meanwhile, and NEVER bounds it by - * {@code reconnectMaxDurationMillis} (that would drop a producer store-and-forward promised - * to keep alive). Only the optional authentication-outage deadline - * ({@code auth_failure_max_duration_millis}, off by default) latches a terminal once such an - * outage has lasted that long. During initialization the classification + * {@code reconnectMaxDurationMillis} nor latches a terminal (either would drop a producer + * store-and-forward promised to keep alive). During initialization the classification * decides: {@link #isRetryable()} is true only when the provider threw a * {@link TokenUnavailableException} marked retryable, which a SYNC initial connect keeps * retrying within its budget; any other provider failure is permanent and fails startup diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpWebSocketSender.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpWebSocketSender.java index 52137b3ad..720fc3c58 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpWebSocketSender.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpWebSocketSender.java @@ -195,8 +195,7 @@ public class QwpWebSocketSender implements Sender { // wrapped as QwpCredentialUnavailableException: the OFF initial connect fails fast with the provider's own // exception, the SYNC one too unless the failure is retryable (then it retries within the reconnect // budget), while the running background drainer treats it as a transient outage and retries it - // indefinitely (never bounded by the reconnect budget, terminal only through the opt-in - // auth_failure_max_duration_millis deadline) per store-and-forward Invariant B. + // indefinitely (never bounded by the reconnect budget, never terminal) per store-and-forward Invariant B. private final Supplier authorizationHeaderSupplier; private final int autoFlushBytes; private final long autoFlushIntervalNanos; @@ -280,8 +279,6 @@ public class QwpWebSocketSender implements Sender { // Test-only lifecycle witness. close() invokes and clears it strictly after // publishing closed=true and before starting any drain or teardown work. private volatile Runnable closeStartedHook; - // Optional authentication-outage deadline (spec section 8.5) handed to the I/O loop; 0 = none. - private volatile long authFailureMaxDurationMillis; private boolean connected; private SenderConnectionDispatcher connectionDispatcher; // A connect-string token_provider's registry lease (TokenProviderRegistry), released when this sender @@ -1971,19 +1968,6 @@ public ConnectionHealth health() { return healthTracker.snapshot(); } - /** - * Arms the optional authentication-outage deadline (design/qwp-token-provider-spec.md, section 8.5); see - * {@code Sender.LineSenderBuilder.authFailureMaxDurationMillis(long)}. {@code <= 0} disarms it. May be called - * while the I/O loop runs. - */ - public void setAuthFailureMaxDurationMillis(long millis) { - this.authFailureMaxDurationMillis = millis; - CursorWebSocketSendLoop loop = cursorSendLoop; - if (loop != null) { - loop.setAuthFailureMaxDurationMillis(millis); - } - } - /** * Hands this sender the registry lease of the {@code token_provider} its credential comes from; the sender * releases it when it closes. {@code Sender.LineSenderBuilder.build()} calls this right after connecting. A @@ -4191,9 +4175,8 @@ private WebSocketClient connectWalk(ReconnectSupplier ctx, CursorWebSocketSendLo // connect unwraps it and fails fast with the provider's own exception, and so does the SYNC one unless // the provider classified the failure retryable, in which case SYNC retries it within its budget. The // running background drainer treats it as a transient outage and retries it indefinitely with capped - // backoff (never bounded by the reconnect budget; terminal only once the opt-in - // auth_failure_max_duration_millis deadline expires), so a persistent credential outage keeps the - // buffered rows in store-and-forward rather than terminating the sender (Invariant B). + // backoff (never bounded by the reconnect budget, never terminal), so even a persistent credential + // outage keeps the buffered rows in store-and-forward rather than terminating the sender (Invariant B). // Mirrors QwpQueryClient, which likewise resolves the credential once before its endpoint walk. // Publish this thread as being inside the pull BEFORE making it, then re-check cancellation, exactly // as the per-endpoint connect below does with the WebSocketClient. The pull is the one blocking call @@ -4216,9 +4199,8 @@ private WebSocketClient connectWalk(ReconnectSupplier ctx, CursorWebSocketSendLo // provider's own exception, so build() surfaces the provider's error directly rather than an internal // wrapper; a SYNC connect retries a retryable one within its budget first. The running background // drainer, by contrast, treats it as a transient outage and retries it indefinitely under Invariant B - // -- never bounding it by the reconnect budget, and latching a terminal only when the opt-in - // auth_failure_max_duration_millis deadline expires -- so a recoverable credential outage never drops - // a producer store-and-forward promised to keep alive. + // -- never bounding it by the reconnect budget, never latching a terminal -- so a recoverable + // credential outage never drops a producer store-and-forward promised to keep alive. throw new QwpCredentialUnavailableException(e); } finally { // Drop the marker as soon as the pull returns, so a later cancel() cannot interrupt this thread @@ -5073,7 +5055,6 @@ private void ensureConnected() { // retries indefinitely.) cursorSendLoop.setConnectionDispatcher(connectionDispatcher); cursorSendLoop.setConnectionHealthTracker(healthTracker); - cursorSendLoop.setAuthFailureMaxDurationMillis(authFailureMaxDurationMillis); // Seed the fresh loop's own hasEverConnected before it can observe // any endpoint-policy failure: without this, a symbol-dict // recycle's rebuilt loop starts believing it has never connected diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoop.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoop.java index 74d024e7b..c10d6f475 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoop.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoop.java @@ -419,15 +419,6 @@ public final class CursorWebSocketSendLoop implements QuietCloseable { // indefinitely and never gives up on a wall-clock budget). private volatile SenderConnectionDispatcher connectionDispatcher; private volatile SenderErrorDispatcher errorDispatcher; - // Optional authentication-outage deadline (design/qwp-token-provider-spec.md, section 8.5), FOREGROUND only; - // 0 = none, the default. Written by the owner thread, read by the I/O thread. - private volatile long authFailureMaxDurationNanos; - // The outage clock of that deadline: started by the first authentication-class failure (credential-unavailable - // or a 401/403 upgrade rejection) of the current outage, reset only by a successful upgrade. Failures of other - // classes neither reset nor fire it. Tracked whether or not a deadline is armed, so arming it late - the - // builder does so right after an async connect has started - still measures the whole outage. I/O thread only. - private boolean authOutageActive; - private long authOutageStartNanos; // Foreground connection health (spec section 8.4): the loop reports connection loss and its terminal failure. // Null for orphan drainers. private volatile io.questdb.client.cutlass.qwp.client.QwpConnectionHealthTracker healthTracker; @@ -1737,17 +1728,6 @@ public void setConnectionDispatcher(SenderConnectionDispatcher dispatcher) { this.connectionDispatcher = dispatcher; } - /** - * Arms the optional authentication-outage deadline (design/qwp-token-provider-spec.md, section 8.5). A - * FOREGROUND loop then latches a terminal when a connect round fails with an authentication-class failure - - * credential-unavailable, or a 401/403 upgrade rejection - and the current outage's first such failure lies - * at least this far back. {@code <= 0} disarms it. Ignored by orphan drainers, which have their own policy. - * May be called while the loop runs. - */ - public void setAuthFailureMaxDurationMillis(long millis) { - this.authFailureMaxDurationNanos = millis <= 0 ? 0 : TimeUnit.MILLISECONDS.toNanos(millis); - } - /** * Plugs the owning sender's connection-health tracker: the loop reports connection loss and its terminal * failure there. Set before {@link #start()}; foreground loops only. @@ -2040,10 +2020,7 @@ private void connectWalk(Throwable initial, String phase, long paceFirstAttemptM // blocking (non-lazy) initial connect hands it to connectWithRetry (see // QwpWebSocketSender.ensureConnected, the SYNC branch), and // BackgroundDrainer converts it into the durable-ack capability-gap - // budget. Neither bounds this loop's steady-state reconnect. The one - // exception is opt-in: a FOREGROUND loop with - // auth_failure_max_duration_millis set latches a terminal once an - // authentication-class outage outlasts it (authOutageDeadlineFired). + // budget. Neither bounds this loop's steady-state reconnect. long backoffMillis = reconnectInitialBackoffMillis; if (paceFirstAttemptMillis > 0 && running) { // NACK-initiated recycle against a reachable server: pace the @@ -2070,8 +2047,6 @@ private void connectWalk(Throwable initial, String phase, long paceFirstAttemptM try { WebSocketClient newClient = reconnectFactory.reconnect(connectCancellation); if (newClient != null) { - // a successful upgrade ends the authentication outage (spec section 8.5) - authOutageActive = false; if (!running) { // close() ran while this connect attempt was in // flight. Its latch await may have been interrupted @@ -2157,9 +2132,6 @@ private void connectWalk(Throwable initial, String phase, long paceFirstAttemptM } resetCatchUpCapGapEpisode(); lastReconnectError = e; - if (e instanceof QwpAuthFailedException && authOutageDeadlineFired(e)) { - return; - } dispatchRetriedEndpointPolicyFailure( SenderError.Category.SECURITY_ERROR, "ws-upgrade-failed: " + e.getMessage()); long now = System.nanoTime(); @@ -2216,23 +2188,18 @@ private void connectWalk(Throwable initial, String phase, long paceFirstAttemptM // background drainer this is a TRANSIENT outage like any other under Invariant B: the provider // hands over a token again once the IdP is reachable or the user finishes signing in, and the // un-acked rows stay safe in on-disk SF meanwhile. So retry indefinitely with capped backoff -- - // NEVER bound by the reconnect budget and never latch a terminal of our own, which would drop a - // producer that store-and-forward promised to keep alive on a recoverable fault. The only - // terminal here is the one the user opted into: authOutageDeadlineFired() below, when - // auth_failure_max_duration_millis is set. The initial connect handles it in connectWithRetry - // (SYNC: a retryable failure is retried within the budget, anything else fails fast with the - // provider's own exception) and in the OFF-mode connect in QwpWebSocketSender (fails fast), - // because a connectivity error is only the caller's to see DURING initialization, not after the - // drainer is running. + // NEVER bound by a wall-clock budget and NEVER latch a terminal, which would drop a producer + // that store-and-forward promised to keep alive on a recoverable fault. The initial connect + // handles it in connectWithRetry (SYNC: a retryable failure is retried within the budget, + // anything else fails fast with the provider's own exception) and in the OFF-mode connect in + // QwpWebSocketSender (fails fast), because a connectivity error is only the caller's to see + // DURING initialization, not after the drainer is running. // // Ends any open cap-gap episode like every other unrelated reconnect state: we never reached a // node to observe its batch cap, so this outage's wall clock must not accrue toward the orphan // cap-gap dwell (see MAX_CATCHUP_CAP_GAP_ATTEMPTS). resetCatchUpCapGapEpisode(); lastReconnectError = e; - if (authOutageDeadlineFired(e)) { - return; - } // Retrying must not be programmatically INVISIBLE, exactly as for the auth/upgrade and // durable-ack policy failures above: a revoked refresh token or a permanently unreachable IdP // is not self-healing, yet flush() keeps returning success while SF absorbs the rows. Without @@ -2339,59 +2306,6 @@ private void connectWalk(Throwable initial, String phase, long paceFirstAttemptM phase, elapsedMs, attempts, lastMsg); } - /** - * The optional authentication-outage deadline (design/qwp-token-provider-spec.md, section 8.5), evaluated - * when a connect round of a FOREGROUND loop fails with an authentication-class failure ({@code failure} is a - * credential-unavailable or a 401/403 rejection). Starts the outage clock on the first such failure; when a - * deadline is armed and the clock has reached it, latches a terminal that names the failure class and the - * elapsed time, reports it to the error handler, and returns true. Unacknowledged rows stay in on-disk - * store-and-forward when the engine has an sf_dir; in memory mode they are lost when the sender closes. - */ - private boolean authOutageDeadlineFired(Throwable failure) { - if (reconnectPolicy != ReconnectPolicy.FOREGROUND) { - return false; - } - final long now = System.nanoTime(); - if (!authOutageActive) { - authOutageActive = true; - authOutageStartNanos = now; - } - final long deadline = authFailureMaxDurationNanos; - if (deadline <= 0 || now - authOutageStartNanos < deadline) { - return false; - } - final long elapsedMillis = TimeUnit.NANOSECONDS.toMillis(now - authOutageStartNanos); - final String failureClass = failure instanceof QwpCredentialUnavailableException - ? "credential-unavailable" : "auth-rejected"; - final String message = "authentication outage deadline exceeded: " + failureClass + " persisted for " - + elapsedMillis + "ms (auth_failure_max_duration_millis=" - + TimeUnit.NANOSECONDS.toMillis(deadline) + "); last failure: " + failure.getMessage(); - // Name the rows' fate per backing (spec section 8.5): only an sf_dir slot outlives this sender; memory-mode - // segments are malloc'd and freed when the sender closes. - LOG.error("{} -- the sender stops; {}", message, engine.sfDir() != null - ? "unacknowledged rows stay in on-disk store-and-forward for a later sender or an orphan drain" - : "unacknowledged rows are lost when the sender closes (memory-only mode, no sf_dir)"); - // External FSNs, like every other SenderError span: after a symbol-dict recycle the engine numbers its - // frames from 0 again, and only externalFsnBase + raw matches what flushAndGetSequence() handed out. - long fromFsn = externalFsnBase + engine.ackedFsn() + 1L; - long toFsn = Math.max(fromFsn, externalFsnBase + engine.publishedFsn()); - SenderError err = new SenderError( - SenderError.Category.SECURITY_ERROR, - SenderError.Policy.TERMINAL, - SenderError.NO_STATUS_BYTE, - message, - SenderError.NO_MESSAGE_SEQUENCE, - fromFsn, - toFsn, - null, - System.nanoTime() - ); - counters.serverErrors.incrementAndGet(); - recordFatal(new LineSenderServerException(err)); - dispatchError(err); - return true; - } - /** * Reports an endpoint-policy rejection a FOREGROUND sender is riding out. *

    @@ -2450,8 +2364,7 @@ public Throwable lastReconnectError() { * Once a FOREGROUND sender has reached the server even once, initialization is * over and store-and-forward owns the buffered data: every endpoint-policy * failure is then a transient the drainer must ride out, never a producer-fatal - * terminal (Invariant B). The opt-in authentication-outage deadline is separate - * from this gate: see {@code authOutageDeadlineFired}. + * terminal (Invariant B). */ private boolean endpointPolicyFailureIsTerminal() { return reconnectPolicy == ReconnectPolicy.ORPHAN || !hasEverConnected; diff --git a/core/src/main/java/io/questdb/client/impl/ConfigSchema.java b/core/src/main/java/io/questdb/client/impl/ConfigSchema.java index 8b1d0a5dc..427b8d45c 100644 --- a/core/src/main/java/io/questdb/client/impl/ConfigSchema.java +++ b/core/src/main/java/io/questdb/client/impl/ConfigSchema.java @@ -68,9 +68,6 @@ public final class ConfigSchema { // INGRESS -- the WebSocket Sender applies. STRING in the registry; the // Sender parses suffix/mode values (off/on, 64k, durability) with its // own helpers, byte-for-byte. - // Optional authentication-outage deadline (design/qwp-token-provider-spec.md, section 8.5). Not set by - // default; > 0 when set. - longRange("auth_failure_max_duration_millis", Side.INGRESS, 0, OPEN_MAX, true, false); str("auto_flush", Side.INGRESS); str("auto_flush_bytes", Side.INGRESS); str("auto_flush_interval", Side.INGRESS); diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/ConnectionHealthTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/ConnectionHealthTest.java index b315702ad..f09cb9976 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/ConnectionHealthTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/ConnectionHealthTest.java @@ -27,22 +27,16 @@ import io.questdb.client.ConnectionHealth; import io.questdb.client.QuestDB; import io.questdb.client.Sender; -import io.questdb.client.SenderError; import io.questdb.client.cutlass.auth.TokenUnavailableException; -import io.questdb.client.cutlass.line.LineSenderException; import io.questdb.client.cutlass.qwp.client.QwpColumnBatch; import io.questdb.client.cutlass.qwp.client.QwpColumnBatchHandler; import io.questdb.client.cutlass.qwp.client.QwpEgressMsgKind; import io.questdb.client.cutlass.qwp.client.QwpQueryClient; -import io.questdb.client.cutlass.qwp.client.QwpWebSocketSender; -import io.questdb.client.cutlass.qwp.client.sf.cursor.OrphanScanner; import io.questdb.client.cutlass.qwp.protocol.QwpConstants; import io.questdb.client.test.cutlass.qwp.client.WebSocketDynamicCredentialTest.ErrorCollector; import io.questdb.client.test.cutlass.qwp.websocket.TestWebSocketServer; import org.junit.Assert; -import org.junit.Rule; import org.junit.Test; -import org.junit.rules.TemporaryFolder; import java.io.IOException; import java.nio.ByteBuffer; @@ -50,235 +44,78 @@ import java.util.concurrent.TimeUnit; import java.util.concurrent.atomic.AtomicBoolean; import java.util.concurrent.atomic.AtomicReference; -import java.util.regex.Matcher; -import java.util.regex.Pattern; import static io.questdb.client.test.cutlass.auth.TokenTestKit.await; import static io.questdb.client.test.tools.TestUtils.assertMemoryLeak; /** - * Conformance tests C21 (connection health, section 8.4) and C22 (the authentication-outage deadline, section + * Conformance tests C21 (connection health, section 8.4) and C22 (no authentication-outage deadline, section * 8.5) of the dynamic-credential specification (design/qwp-token-provider-spec.md). */ public class ConnectionHealthTest { - // The elapsed time a deadline terminal reports: "... persisted for ms ...". - private static final Pattern PERSISTED_FOR = Pattern.compile("persisted for (\\d+)ms"); private static final String SENTINEL = "SENTINEL-TOKEN-7f3a91"; - @Rule - public final TemporaryFolder temp = TemporaryFolder.builder().assureDeletion().build(); - @Test(timeout = 60_000) - public void testDeadlineIsNotResetOrFiredByOtherFailureClasses() throws Exception { - // C22: rounds that fail for other reasons neither reset the clock nor fire the deadline. A credential - // outage, then role rejects and a transport outage that together outlast the deadline, then the - // credential outage again: the deadline fires on the first authentication-class round after them. + public void testAuthenticationOutageNeverMakesTheSenderTerminal() throws Exception { + // C22: an established sender rides out credential-unavailable and 401 rounds, with role rejects in + // between, for as long as they last. It never becomes terminal, keeps the rows in store-and-forward, and + // delivers them once the credential is back. assertMemoryLeak(() -> { - final long deadlineMillis = 1_000; - AtomicReference mode = new AtomicReference<>("ok"); - ErrorCollector errors = new ErrorCollector(); - TestWebSocketServer server = startServer(new WebSocketDynamicCredentialTest.DropAfterFirstAckHandler()); - try (Sender sender = Sender.builder(Sender.Transport.WEBSOCKET) - .address("localhost:" + server.getPort()) - .reconnectInitialBackoffMillis(10) - .reconnectMaxBackoffMillis(20) - .authFailureMaxDurationMillis(deadlineMillis) - .errorHandler(errors) - .httpTokenProvider(() -> { - if ("fail".equals(mode.get())) { - throw TokenUnavailableException.retryable("IdP unreachable"); - } - return "TOKEN"; - }) - .build()) { - mode.set("fail"); - sender.table("t").longColumn("v", 1).atNow(); - sender.flush(); // ACKed, then the server drops the connection - await(() -> errors.count("credential-unavailable") >= 1, 10_000, "the credential outage"); - long authOutageStart = System.nanoTime(); - // Other classes: every endpoint role-rejects (set before the provider recovers, so no upgrade can - // succeed in between and reset the clock), then the server goes away (transport). - server.setRejectWithRole("REPLICA"); - mode.set("ok"); - Thread.sleep(deadlineMillis * 2 / 3); - server.close(); - Thread.sleep(deadlineMillis * 2 / 3); - Assert.assertTrue(TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - authOutageStart) > deadlineMillis); - Assert.assertEquals("role rejects and transport failures must not fire the deadline", - 0, errors.terminalCount()); - // the credential outage returns: the clock was never reset, so the first such round fires - long failAt = System.nanoTime(); - mode.set("fail"); - await(() -> errors.terminalCount() == 1, 10_000, "the deadline to fire"); - SenderError terminal = terminal(errors); - Assert.assertTrue(terminal.getServerMessage(), - terminal.getServerMessage().contains("credential-unavailable persisted for")); - // The clock started before authOutageStart was taken and the terminal fired after failAt, so the - // reported outage spans the role-reject and transport rounds. A clock those rounds had reset - // would report only the time since the credential outage returned. - Matcher persisted = PERSISTED_FOR.matcher(terminal.getServerMessage()); - Assert.assertTrue(terminal.getServerMessage(), persisted.find()); - Assert.assertTrue("role rejects and transport failures must not reset the clock: " - + terminal.getServerMessage(), - Long.parseLong(persisted.group(1)) >= TimeUnit.NANOSECONDS.toMillis(failAt - authOutageStart)); - Assert.assertEquals(ConnectionHealth.State.FAILED, sender.health().getState()); - } catch (LineSenderException expected) { - // close() rethrows the latched terminal - Assert.assertTrue(expected.getMessage(), expected.getMessage().contains("authentication outage deadline")); - } finally { - server.close(); - } - }); - } - - @Test(timeout = 60_000) - public void testDeadlineMakesTheSenderTerminalAndKeepsTheData() throws Exception { - // C22: set, the sender becomes terminal on an authentication-class round once the duration has passed; - // the error names the class and the elapsed time, is reported as terminal and thrown from later producer - // calls, and the unacknowledged rows stay on disk. - assertMemoryLeak(() -> { - String sfDir = temp.newFolder("deadline").getAbsolutePath(); - ErrorCollector errors = new ErrorCollector(); + AtomicReference providerMode = new AtomicReference<>("ok"); AtomicReference rejectWith = new AtomicReference<>(0); - try (TestWebSocketServer server = startServer(new WebSocketDynamicCredentialTest.DropAfterFirstAckHandler())) { + ErrorCollector errors = new ErrorCollector(); + WebSocketDynamicCredentialTest.AckHandler handler = new WebSocketDynamicCredentialTest.AckHandler(); + try (TestWebSocketServer server = startServer(handler)) { server.setAuthorizationValidator(h -> rejectWith.get()); - Sender sender = Sender.builder(Sender.Transport.WEBSOCKET) + try (Sender sender = Sender.builder(Sender.Transport.WEBSOCKET) .address("localhost:" + server.getPort()) - .storeAndForwardDir(sfDir) - .senderId("deadline") .reconnectInitialBackoffMillis(10) .reconnectMaxBackoffMillis(20) - .closeFlushTimeoutMillis(0) - .authFailureMaxDurationMillis(500) .errorHandler(errors) - .httpTokenProvider(() -> SENTINEL) - .build(); - try { - rejectWith.set(401); + .httpTokenProvider(() -> { + if ("fail".equals(providerMode.get())) { + throw TokenUnavailableException.retryable("IdP unreachable"); + } + return "TOKEN"; + }) + .build()) { sender.table("t").longColumn("v", 1).atNow(); - sender.flush(); // ACKed, then dropped: every reconnect is now rejected - await(() -> server.authRejectCount() >= 1, 10_000, "the first 401"); - long start = System.nanoTime(); - sender.table("t").longColumn("v", 2).atNow(); - sender.flush(); // buffered in store-and-forward - await(() -> errors.terminalCount() == 1, 10_000, "the deadline to fire"); - long elapsedMillis = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - start); - Assert.assertTrue("must not fire before the deadline, fired at " + elapsedMillis + " ms", - elapsedMillis >= 400); - SenderError terminal = terminal(errors); - Assert.assertEquals(SenderError.Category.SECURITY_ERROR, terminal.getCategory()); - Assert.assertTrue(terminal.getServerMessage(), terminal.getServerMessage().contains("auth-rejected")); - Assert.assertTrue(terminal.getServerMessage(), terminal.getServerMessage().contains("persisted for")); - Assert.assertFalse(terminal.getServerMessage().contains(SENTINEL)); - try { - sender.table("t").longColumn("v", 3).atNow(); - sender.flush(); - Assert.fail("later producer calls must throw the terminal"); - } catch (LineSenderException e) { - Assert.assertTrue(e.getMessage(), e.getMessage().contains("authentication outage deadline")); - } - Assert.assertEquals(ConnectionHealth.State.FAILED, sender.health().getState()); - } finally { - try { - sender.close(); - } catch (LineSenderException ignored) { - // close() may rethrow the terminal - } - } - } - Assert.assertEquals("the unacknowledged rows stay on disk for a later sender or an orphan drain", - 1, OrphanScanner.scan(sfDir, "someone-else").size()); - }); - } + sender.flush(); + await(() -> handler.frames.get() == 1, 10_000, "the first batch"); - @Test(timeout = 60_000) - public void testDeadlineIsResetByASuccessfulUpgradeAndUnsetMeansNoDeadline() throws Exception { - // C22: a successful upgrade resets the clock; and without the key, a 401 outage is retried indefinitely. - assertMemoryLeak(() -> { - for (boolean armed : new boolean[]{true, false}) { - ErrorCollector errors = new ErrorCollector(); - AtomicReference rejectWith = new AtomicReference<>(0); - WebSocketDynamicCredentialTest.AckHandler handler = new WebSocketDynamicCredentialTest.AckHandler(); - try (TestWebSocketServer server = startServer(handler)) { - server.setAuthorizationValidator(h -> rejectWith.get()); - Sender.LineSenderBuilder b = Sender.builder(Sender.Transport.WEBSOCKET) - .address("localhost:" + server.getPort()) - .reconnectInitialBackoffMillis(10) - .reconnectMaxBackoffMillis(20) - .errorHandler(errors) - .httpTokenProvider(() -> "TOKEN"); - if (armed) { - b.authFailureMaxDurationMillis(800); - } - try (Sender sender = b.build()) { - for (int outage = 0; outage < 2; outage++) { - rejectWith.set(401); - server.dropAllConnections(); - Thread.sleep(500); // under the deadline - rejectWith.set(0); // a successful upgrade ends the outage - final long acked = handler.frames.get(); - sender.table("t").longColumn("v", outage).atNow(); - sender.flush(); - await(() -> handler.frames.get() > acked, 10_000, "recovery"); - } - Assert.assertEquals("two outages of 500 ms each, the clock reset in between [armed=" + armed + ']', - 0, errors.terminalCount()); - if (!armed) { - rejectWith.set(401); - server.dropAllConnections(); - Thread.sleep(1_500); - Assert.assertEquals("no deadline: retried indefinitely", 0, errors.terminalCount()); - Assert.assertTrue(errors.count("auth-rejected") > 0); - rejectWith.set(0); - } - } - } - } - }); - } - - @Test(timeout = 60_000) - public void testDeadlineSpanCarriesExternalFsnsAfterASymbolDictRecycle() throws Exception { - // C22 after a symbol-dict recycle: the rebuilt I/O loop arms the deadline, and the terminal error's span - // is on the sender's own FSN scale - what flushAndGetSequence() returned - not the rebuilt engine's. - assertMemoryLeak(() -> { - ErrorCollector errors = new ErrorCollector(); - try (TestWebSocketServer server = startServer(new WebSocketDynamicCredentialTest.AckHandler())) { - Sender sender = Sender.builder(Sender.Transport.WEBSOCKET) - .address("localhost:" + server.getPort()) - .symbolDictResetThreshold(2) - .reconnectInitialBackoffMillis(10) - .reconnectMaxBackoffMillis(20) - .closeFlushTimeoutMillis(0) - .authFailureMaxDurationMillis(200) - .errorHandler(errors) - .build(); - try { - QwpWebSocketSender ws = (QwpWebSocketSender) sender; - sender.table("t").symbol("s", "a").longColumn("v", 1).atNow(); - sender.table("t").symbol("s", "b").longColumn("v", 1).atNow(); - long acked = sender.flushAndGetSequence(); - Assert.assertTrue(sender.awaitAckedFsn(acked, 10_000)); - // every handshake from here on is rejected, the rebuilt loop's first connect included - server.setRejectWithStatus(401, "Unauthorized"); - sender.table("t").symbol("s", "c").longColumn("v", 2).atNow(); // the recycle runs here - Assert.assertEquals("setup: the dictionary must have been recycled", 1, ws.getSymbolDictEpoch()); - Assert.assertTrue("setup: the recycle must move the FSN epoch", ws.getFsnEpochBaseForTesting() > 0); - long pending = sender.flushAndGetSequence(); - await(() -> errors.terminalCount() == 1, 10_000, "the deadline to fire on the rebuilt loop"); - SenderError terminal = terminal(errors); - Assert.assertTrue(terminal.getServerMessage(), - terminal.getServerMessage().contains("auth-rejected persisted for")); - Assert.assertEquals("fromFsn must follow the acknowledged FSN the sender reported", - acked + 1, terminal.getFromFsn()); - Assert.assertEquals("toFsn must be the last FSN the sender reported", pending, terminal.getToFsn()); - } finally { - server.setRejectWithStatus(0, null); - try { - sender.close(); - } catch (LineSenderException ignored) { - // close() may rethrow the terminal - } + // the credential becomes unavailable, then the server rejects the token, then every endpoint + // role-rejects, then the credential is unavailable again; each server-side rejection is in + // place before the previous failure is lifted, so that no upgrade can succeed in between, not + // even one whose round pulled a token just before the provider failed again + providerMode.set("fail"); + server.dropAllConnections(); + await(() -> errors.count("credential-unavailable") >= 1, 10_000, "credential-unavailable rounds"); + sender.table("t").longColumn("v", 2).atNow(); + sender.flush(); // kept in store-and-forward while the connection is down + Thread.sleep(300); + rejectWith.set(401); + providerMode.set("ok"); + await(() -> errors.count("auth-rejected") >= 1, 10_000, "401 rounds"); + Thread.sleep(300); + server.setRejectWithRole("REPLICA"); + rejectWith.set(0); + await(() -> sender.health().getLastFailure().getFailureClass() + == ConnectionHealth.FailureClass.ROLE_REJECTED, 10_000, "role-reject rounds"); + Thread.sleep(300); + providerMode.set("fail"); + await(() -> sender.health().getLastFailure().getFailureClass() + == ConnectionHealth.FailureClass.CREDENTIAL_UNAVAILABLE, 10_000, "credential-unavailable again"); + Thread.sleep(300); + Assert.assertEquals("an established sender must never become terminal on its own", + 0, errors.terminalCount()); + Assert.assertEquals(ConnectionHealth.State.RECONNECTING, sender.health().getState()); + + // the credential and the primary are back: the buffered row is delivered + server.setRejectWithRole(null); + providerMode.set("ok"); + await(() -> handler.frames.get() >= 2, 10_000, "the buffered row"); + await(() -> sender.health().getState() == ConnectionHealth.State.CONNECTED, 10_000, "recovered"); + Assert.assertEquals(0, errors.terminalCount()); } } }); @@ -513,15 +350,6 @@ private static TestWebSocketServer startServer(TestWebSocketServer.WebSocketServ return server; } - private static SenderError terminal(ErrorCollector errors) { - for (SenderError e : errors.errors) { - if (e.getAppliedPolicy() == SenderError.Policy.TERMINAL) { - return e; - } - } - throw new AssertionError("no terminal error"); - } - // Answers like ExecDoneHandler but, while drop is set, drops the connection a query arrives on. private static final class DropQueryHandler implements TestWebSocketServer.WebSocketServerHandler { final AtomicBoolean drop = new AtomicBoolean(); diff --git a/core/src/test/java/io/questdb/client/test/impl/WsSenderConfigHonoredTest.java b/core/src/test/java/io/questdb/client/test/impl/WsSenderConfigHonoredTest.java index d5753c6b9..3af186bfe 100644 --- a/core/src/test/java/io/questdb/client/test/impl/WsSenderConfigHonoredTest.java +++ b/core/src/test/java/io/questdb/client/test/impl/WsSenderConfigHonoredTest.java @@ -86,7 +86,6 @@ public void testEveryIngressKeyIsHonored() { assertHonored("connection_listener_inbox_capacity=64", "connection_listener_inbox_capacity", 64); assertHonored("token=ey.abc", "token", "ey.abc"); assertHonored("auth_timeout_ms=4321", "auth_timeout_ms", 4321L); - assertHonored("auth_failure_max_duration_millis=600000", "auth_failure_max_duration_millis", 600_000L); assertHonored("connect_timeout=7000", "connect_timeout", 7000); // username/password together (both-or-neither), and the user/pass aliases. diff --git a/design/entra-id-qwp-auth.md b/design/entra-id-qwp-auth.md index def18b771..50ff8416c 100644 --- a/design/entra-id-qwp-auth.md +++ b/design/entra-id-qwp-auth.md @@ -548,7 +548,7 @@ Build in this order. Each step is one PR and must pass the conformance tests lis - Add the `uses` clause to `module-info.java`. - Release the lease when a sender or query client closes. - Add the new `azure/` reactor module. -4. **Health accessor and deadline.** Implements spec §8.4 and §8.5. Tests C21–C22. +4. **Health accessor.** Implements spec §8.4, and §8.5, which rules out an authentication-outage deadline. Tests C21–C22. **Separate tickets, outside this feature:** @@ -593,12 +593,12 @@ Every step of §10 is implemented. Line references in §1-§7 predate it. - Its release profiles mirror core's. - Tests: `TokenProviderConfigTest` (C18, C19, the connect-string half of C24), `TokenProviderSharingTest` (C17, over TLS), `azure/AzureTokenSourceTest` (fake `TokenCredential`, no network), `azure/AzureCredentialSelectionTest` (the real factory, no network) and `azure/AzureManagedIdentityLibraryTest` (C24: the real library in a child JVM, against a local IMDS stub). -**Step 4 - health and deadline** (spec §8.4, §8.5). +**Step 4 - health** (spec §8.4, §8.5). - `io.questdb.client.ConnectionHealth` (with `State`, `FailureClass`, `Failure`, `Aggregate`) and `QwpConnectionHealthTracker`, which publishes immutable snapshots through a volatile field. - The ingest walk reports rounds and upgrades; the I/O loop reports connection loss and terminal failure. Drainers never report. - Accessors: `Sender.health()` (default throws `UnsupportedOperationException`), `QwpQueryClient.health()`, `QuestDB.health()` (aggregate of every pooled connection). -- `auth_failure_max_duration_millis` (INGRESS key) and `Sender.LineSenderBuilder.authFailureMaxDurationMillis(long)`. The clock lives in `CursorWebSocketSendLoop`, applies to FOREGROUND loops only, and is tracked even before the deadline is armed. The builder arms it right after connecting, so an `async` start is measured from its first failure. +- No authentication-outage deadline (spec §8.5): an established sender retries credential-unavailable and `401`/`403` rounds indefinitely, and reports them through the error handler and the health snapshot. The opt-in `auth_failure_max_duration_millis` of spec v0.2 to v0.5 was removed. - Tests: `ConnectionHealthTest` (C21, C22). **Not done:** the spec documents (`qwp-ingress-websocket.md`, `qwp-egress-websocket.md`, the connect-string reference) live in the documentation repository. The Rust core and Python parity is tracked there too. diff --git a/design/qwp-token-provider-spec.md b/design/qwp-token-provider-spec.md index c9a5d669a..3dd922c5c 100644 --- a/design/qwp-token-provider-spec.md +++ b/design/qwp-token-provider-spec.md @@ -1,9 +1,10 @@ # QWP dynamic bearer credentials: cross-language specification -**Status:** v0.5, ready for implementation. All decisions are resolved (§12). +**Status:** v0.6, ready for implementation. All decisions are resolved (§12). **Changes:** +- **v0.6:** the optional authentication-outage deadline (§8.5, D7) is withdrawn. It made an established sender terminal on a transient outage, which store-and-forward forbids, and the role-reject and transport rounds of a failover counted toward it. An application that prefers a hard failure closes the sender itself, guided by the error channel and the connection health (§8.5). - **v0.5:** a result whose token expires within `handout_floor` of its receipt is a retryable failed fetch whose message names `handout_floor` (§4, C2, C6). Before, it counted as a success that could never be handed out, so every caller waited out `cold_wait` for an error that named no cause. - **v0.4:** `azure_credential` selects one Azure credential deterministically (§7.1), so the library itself reports an unreachable managed-identity endpoint (retryable) apart from a misconfigured credential (permanent) (§7.5, D10). A discovery chain's "no credential available" stays permanent, so a host without a credential still fails fast, and the client warns once that the chain does not ride out an endpoint outage. This resolves the conflict between §4 and §7.5 over an unreachable managed-identity endpoint. - **v0.3:** decisions resolved, with D8 (§8.1) and D9 (§7.2) added; Java binding decided (Appendix B). @@ -350,7 +351,7 @@ Further rules: **Retry indefinitely** -- **Meaning:** retry with capped exponential backoff for as long as the client is open, unless the optional deadline in §8.5 is configured. Store-and-forward MUST NOT drop a live producer. +- **Meaning:** retry with capped exponential backoff for as long as the client is open. Store-and-forward MUST NOT drop a live producer, so no setting bounds these retries (§8.5). - **Reporting:** each retried failure MUST be reported to the application's error channel, not only to logs. The report is a retriable, security-category error that names the failure class. The failure SHOULD also be visible in the connection health (§8.4). **Ride out, then quarantine** @@ -388,24 +389,13 @@ Rules: - **Pools and facades** SHOULD expose an aggregate: the number of connections in each `state`, the oldest `outage_since`, and the most recent `last_failure`. - **Query clients** use the same fields. They connect on demand, so for them `reconnecting` means the last connect or failover reconnect failed and the next operation will try again. -### 8.5 Optional authentication-outage deadline [D7] +### 8.5 No authentication-outage deadline [D7] -Some applications prefer a hard failure to an authentication outage that lasts indefinitely, much as Kafka's `delivery.timeout.ms` bounds how long a record may wait. A client MAY offer this. If it does, it MUST use this key and these semantics. +A client MUST NOT offer a setting that makes an established ingress sender terminal once an authentication-class failure has lasted some time. -| Key | Values | Default | -|---|---|---| -| `auth_failure_max_duration_millis` | a positive duration in milliseconds | not set (no deadline) | - -- **Which failures count.** Only *authentication-class* failures: credential-unavailable, and auth-rejected (`401` or `403`). -- **Where it applies.** Ingress senders, in the phases where §8.3 retries an authentication-class failure indefinitely: `async` initialization and established. It does not apply to orphan drains, which have their own rule, or to egress, which already fails the operation. -- **The clock.** It starts at the first authentication-class failure of the current outage. Only a successful upgrade resets it. -- **When it fires.** The sender becomes terminal when a connect round fails with an authentication-class failure and the clock has reached the configured duration. - - Rounds that fail for other reasons neither reset the clock nor fire the deadline. - - So a network outage that follows a single `401` does not fire it. - - And a cluster that alternates between `401` and network errors cannot postpone it forever. -- **What happens then.** - - The error names the failure class and the elapsed time. It is reported as terminal through the error handler and raised from later producer calls. - - Unacknowledged data stays in on-disk store-and-forward storage, for a later sender or an orphan drain. In memory-only mode it is lost when the sender closes. +- **Why.** Such a deadline latches a terminal on a transient outage, which store-and-forward forbids. An unreachable identity provider, a throttling token endpoint, or a failover window around a rejected token all recover, and the rows wait in store-and-forward meanwhile. A clock that keeps running through the role-reject and transport rounds of a failover would also end a producer that never had a lasting authentication problem. +- **What applications use instead.** Every failed round is reported to the error channel as a retriable, security-category error that names its class (§8.3), and the connection health (§8.4) shows how long the sender has been failing (`outage_since`) and why (`last_failure`). An application that prefers a hard failure closes the sender itself. +- **The rows.** With on-disk store-and-forward, closing the sender leaves the unacknowledged rows for a later sender or an orphan drain. In memory-only mode they are lost when the sender closes. ## 9. Security @@ -448,7 +438,7 @@ Every client that implements this specification SHOULD pass these scenarios. The | C19 | Registry lifecycle | Lease release, linger, and re-acquiring a lease during the linger period. | | C20 | Redaction | A sentinel token never appears in logs, errors, events, string representations or health snapshots, across C4, C10, C11 and a malformed source response. | | C21 | Connection health | The snapshot moves through `connecting`, `connected`, `reconnecting` (with `401` and credential-unavailable failures) and back to `connected`. `outage_since` and `failed_rounds` reset on recovery; `last_failure` is kept. No credential appears in it. | -| C22 | Authentication-outage deadline | Unset: retries continue indefinitely. Set: the sender becomes terminal only on an authentication-class round once the duration has passed. Other failures in between neither reset nor fire it. A successful upgrade resets it. Orphan drains are unaffected, and on-disk data remains. | +| C22 | No authentication-outage deadline | An established sender rides out credential-unavailable and `401` rounds, with role rejects in between, without becoming terminal, and delivers the buffered rows once the credential is back. | | C23 | `WWW-Authenticate` challenge | A plain `401`, and a `401` with `Bearer error="invalid_token"`, both trigger the retry in §8.2. A `401` whose Bearer challenge carries another error does not. | | C24 | Azure credential selection | With `azure_credential=managed_identity`, an unreachable managed-identity endpoint is retryable, so `sync` initialization retries it within its budget; an identity that is not assigned is permanent, so initialization fails fast. With `default`, a chain that finds nothing is permanent, and the warning in §7.5 is logged once, unless the platform's selector names one credential. | @@ -457,7 +447,7 @@ Every client that implements this specification SHOULD pass these scenarios. The - **QWP ingress and egress specs** (`qwp-ingress-websocket.md`, `qwp-egress-websocket.md`): - their statement that "authentication errors are terminal at any host" is replaced by §8 **[D1]**; - their authentication sections should point to §6. -- **Connect-string reference:** it should reserve the keys in §7.1, and `auth_failure_max_duration_millis` (§8.5). +- **Connect-string reference:** it should reserve the keys in §7.1. - **OIDC device-flow token store** (`design/oidc-token-persistence.md`): unaffected. An `OidcDeviceAuth` instance remains a valid provider supplied by the application. ## 12. Decisions @@ -472,7 +462,7 @@ All decisions below are resolved. Changing one after the Java implementation mer | D4 | Whether to reject `token_provider` on `ws::` | Yes. | | D5 | A zero-dependency `azure_imds` provider | The name is reserved; it is implemented only on demand. | | D6 | Whether SYNC initialization retries retryable provider failures | Yes. | -| D7 | Whether to offer the authentication-outage deadline, and its key | Yes, as an optional setting that is off by default: `auth_failure_max_duration_millis`. | +| D7 | Whether to offer an authentication-outage deadline | No. It would make an established sender terminal on a transient outage, which store-and-forward forbids (§8.5). Applications watch the error channel and the connection health instead. Versions 0.2 to 0.5 offered it as `auth_failure_max_duration_millis`. | | D8 | How to classify an exception from an application-supplied provider that is not a token-unavailable error | Permanent (§8.1). | | D9 | Whether `token_provider` applies to non-QWP schemas | No; `wss::` only (§7.2). | | D10 | How the `azure` provider classifies "no credential available", and how to make it resilient | `azure_credential` selects one credential, so the library itself reports an unreachable endpoint (retryable) apart from a misconfigured credential (permanent). In a discovery chain, "no credential available" stays permanent. An IMDS outage looks the same (§7.5), but only `sync` initialization depends on the difference (§8.3), and making it retryable would make every host without a credential wait out `reconnect_max_duration_millis` (5 minutes by default, plus up to one `cold_wait`) before failing. Resilience is therefore opt-in through deterministic mode, which Azure Identity recommends in production anyway, and a chain warns once when it starts. This follows Azure Identity's split between fail-fast discovery and a resilient single credential (Appendix D). A local host check, as Google's library makes, was not adopted: Azure's SMBIOS asset tag identifies only the public cloud, not sovereign clouds or Azure Local. | @@ -540,7 +530,6 @@ New types live in `io.questdb.client.cutlass.auth` unless noted. | Registry | `TokenProviderRegistry`: process-wide (`global()`), hands out ref-counted `Lease`s, closes a provider `registry_linger` after its last lease | | `azure` provider | The new reactor module `azure/`, artifact `org.questdb:questdb-client-azure` (the core client is `org.questdb:questdb-client`), package `io.questdb.client.azure`, classes `AzureTokenProviderFactory` and `AzureTokenSource`. It depends on `azure-identity` through `azure-sdk-bom`, keeps the Java 8 floor, and is released together with the client. | | Connection health | `io.questdb.client.ConnectionHealth`, an immutable snapshot. Returned by `Sender.health()` (a default method; non-QWP senders throw `UnsupportedOperationException`), by `QwpQueryClient.health()`, and as a `ConnectionHealth.Aggregate` by `QuestDB.health()`. | -| Authentication-outage deadline | Key `auth_failure_max_duration_millis`; builder method `authFailureMaxDurationMillis(long)` | | Credential selection | Key `azure_credential`, validated by `TokenProviderSpec` and handled by `AzureTokenProviderFactory`. `default` builds `DefaultAzureCredential`; the other values build `ManagedIdentityCredential`, `WorkloadIdentityCredential` or `EnvironmentCredential` directly, never touching the process environment. Setting `AZURE_TOKEN_CREDENTIALS` in the configuration passed to `DefaultAzureCredentialBuilder.configuration(...)` is not enough: in `azure-identity` 1.18 it selects the credential but keeps the chain's endpoint check, because each credential re-reads the selector from the process-wide configuration. A credential that cannot be built for lack of configuration fails every fetch as permanent. `AzureTokenSource` classifies the same way whatever the credential: it finds a managed identity's "not assigned" reason in a nested cause, and names the socket-level cause of a network failure, but never echoes another nested message (§9). | **Python notes:** From 4355127d2926878607541f34358e70af31eddfc2 Mon Sep 17 00:00:00 2001 From: glasstiger Date: Wed, 14 Oct 2026 22:45:11 +0100 Subject: [PATCH 16/40] Report an outage when a connected sender fails A sender whose I/O loop latched a terminal error while it was connected, such as a batch the server rejects with PARSE_ERROR, reported FAILED with outageSince NONE. CursorWebSocketSendLoop.recordFatal() reports only failed(), and failed() never set the outage start that upgraded() had cleared. ConnectionHealth documents NONE as "while connected", and the aggregate's oldest outage as NONE only when every connection is connected, so QuestDB.health() left such senders out of its oldest outage. A query client already calls connectionLost() before failed(). QwpConnectionHealthTracker.failed() now starts the outage when none is running. A client that was already without a connection keeps the start of the outage it is in. ConnectionHealthTest gains two tests for paths that no test pinned: - testSenderHealthIsFailedWithAnOutageAfterATerminalRejection: a server that rejects every batch with PARSE_ERROR moves the sender to FAILED with an outage start. It fails without the fix (outageSince -1), and with "expected: but was:" if recordFatal() stops reporting failed(). - testQueryClientHealthIsFailedWhenFailoverIsOffAndItLosesItsConnection: with failover=off a dropped connection leaves the query client FAILED, and the next query fails without reconnecting. It fails with "expected: but was:" if the failover-off branch of executeImpl() stops reporting failed(). The new query test and the failover_max_attempts=1 test now share one helper, assertQueryClientFailsForGood(). --- .../client/QwpConnectionHealthTracker.java | 7 +- .../qwp/client/ConnectionHealthTest.java | 113 +++++++++++++----- 2 files changed, 91 insertions(+), 29 deletions(-) diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpConnectionHealthTracker.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpConnectionHealthTracker.java index bc483a257..97b991bd5 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpConnectionHealthTracker.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpConnectionHealthTracker.java @@ -120,12 +120,17 @@ public void connectionLost() { } /** - * The client will not connect again. Sticky; keeps the last connect-round failure. + * The client will not connect again. Sticky; keeps the last connect-round failure. A client that fails while + * connected starts its outage here, since it will not reach the server again; a client already without a + * connection keeps the start of the outage it is in. */ public void failed() { synchronized (lock) { if (state != ConnectionHealth.State.CLOSED) { state = ConnectionHealth.State.FAILED; + if (outageSince == ConnectionHealth.NONE) { + outageSince = System.currentTimeMillis(); + } publish(); } } diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/ConnectionHealthTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/ConnectionHealthTest.java index f09cb9976..c21b1a92b 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/ConnectionHealthTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/ConnectionHealthTest.java @@ -32,6 +32,7 @@ import io.questdb.client.cutlass.qwp.client.QwpColumnBatchHandler; import io.questdb.client.cutlass.qwp.client.QwpEgressMsgKind; import io.questdb.client.cutlass.qwp.client.QwpQueryClient; +import io.questdb.client.cutlass.qwp.client.WebSocketResponse; import io.questdb.client.cutlass.qwp.protocol.QwpConstants; import io.questdb.client.test.cutlass.qwp.client.WebSocketDynamicCredentialTest.ErrorCollector; import io.questdb.client.test.cutlass.qwp.websocket.TestWebSocketServer; @@ -43,6 +44,7 @@ import java.nio.ByteOrder; import java.util.concurrent.TimeUnit; import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.atomic.AtomicLong; import java.util.concurrent.atomic.AtomicReference; import static io.questdb.client.test.cutlass.auth.TokenTestKit.await; @@ -190,38 +192,19 @@ public void testQueryClientHealthFollowsConnectFailoverAndRecovery() throws Exce }); } + @Test(timeout = 60_000) + public void testQueryClientHealthIsFailedWhenFailoverIsOffAndItLosesItsConnection() throws Exception { + // C21 for egress: with failover off a query that loses its connection fails, and the failure stays latched, + // so every later query fails on it without reconnecting - the client is done. + assertMemoryLeak(() -> assertQueryClientFailsForGood("failover=off;", null)); + } + @Test(timeout = 60_000) public void testQueryClientHealthIsFailedWhenItsOnlyAttemptLosesItsConnection() throws Exception { // C21 for egress: with failover_max_attempts=1 a query that loses its connection fails, and every later // query fails on the same latched failure without reconnecting - the client is done, as with failover off. - assertMemoryLeak(() -> { - DropQueryHandler handler = new DropQueryHandler(); - try (TestWebSocketServer server = startServer(handler)) { - server.setSendServerInfo(true); - try (QwpQueryClient client = QwpQueryClient.fromConfig("ws::addr=localhost:" + server.getPort() - + ";failover_max_attempts=1;")) { - client.connect(); - handler.drop.set(true); - RecordingHandler failed = new RecordingHandler(); - client.execute("SELECT 1", failed); - Assert.assertNotNull("the query must fail", failed.error); - Assert.assertTrue(failed.error, failed.error.startsWith("transport failure after 1 execute attempt")); - ConnectionHealth h = client.health(); - Assert.assertEquals(h.toString(), ConnectionHealth.State.FAILED, h.getState()); - Assert.assertNotEquals(ConnectionHealth.NONE, h.getOutageSinceEpochMillis()); - - // FAILED must hold: the server answers again, yet the next query fails without reconnecting - handler.drop.set(false); - int upgrades = server.upgradeRequestCount(); - RecordingHandler next = new RecordingHandler(); - client.execute("SELECT 1", next); - Assert.assertNotNull("the next query must fail", next.error); - Assert.assertTrue(next.error, next.error.startsWith("transport failure after 1 execute attempt")); - Assert.assertEquals("no reconnect", upgrades, server.upgradeRequestCount()); - Assert.assertEquals(ConnectionHealth.State.FAILED, client.health().getState()); - } - } - }); + assertMemoryLeak(() -> assertQueryClientFailsForGood("failover_max_attempts=1;", + "transport failure after 1 execute attempt")); } @Test(timeout = 60_000) @@ -260,6 +243,34 @@ public void testQueryClientHealthReportsAConnectionLostOnTheLastAttempt() throws }); } + @Test(timeout = 60_000) + public void testSenderHealthIsFailedWithAnOutageAfterATerminalRejection() throws Exception { + // C21: a sender that becomes terminal while connected - the server rejects a batch with a TERMINAL category - + // reports FAILED, and its outage starts there, since it will never reach the server again. + assertMemoryLeak(() -> { + ErrorCollector errors = new ErrorCollector(); + try (TestWebSocketServer server = startServer(new RejectingHandler()); + Sender sender = Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + server.getPort()) + .errorHandler(errors) + .build()) { + ConnectionHealth h = sender.health(); + Assert.assertEquals(ConnectionHealth.State.CONNECTED, h.getState()); + Assert.assertEquals(ConnectionHealth.NONE, h.getOutageSinceEpochMillis()); + + sender.table("t").longColumn("v", 1).atNow(); + sender.flush(); + // the I/O thread latches FAILED before it hands the terminal error to the handler + await(() -> errors.terminalCount() >= 1, 10_000, "the terminal rejection"); + h = sender.health(); + Assert.assertEquals(h.toString(), ConnectionHealth.State.FAILED, h.getState()); + Assert.assertNotEquals("a failed sender is without a connection: " + h, + ConnectionHealth.NONE, h.getOutageSinceEpochMillis()); + Assert.assertTrue(h.toString(), h.getOutageSinceEpochMillis() >= h.getLastConnectedAtEpochMillis()); + } + }); + } + @Test(timeout = 60_000) public void testSenderHealthMovesThroughEveryState() throws Exception { // C21: connecting -> connected -> reconnecting (with 401 and credential-unavailable failures) -> @@ -342,6 +353,38 @@ public void testSenderHealthMovesThroughEveryState() throws Exception { }); } + // A query client loses its connection in the middle of a query, and its configuration lets it never reconnect: + // the query fails, and the health is FAILED with an outage. FAILED holds when a later query fails on the same + // latched failure without contacting the server. errorPrefix, when not null, starts both errors. + private static void assertQueryClientFailsForGood(String configKeys, String errorPrefix) throws Exception { + DropQueryHandler handler = new DropQueryHandler(); + try (TestWebSocketServer server = startServer(handler)) { + server.setSendServerInfo(true); + try (QwpQueryClient client = QwpQueryClient.fromConfig("ws::addr=localhost:" + server.getPort() + ';' + + configKeys)) { + client.connect(); + handler.drop.set(true); + RecordingHandler failed = new RecordingHandler(); + client.execute("SELECT 1", failed); + Assert.assertNotNull("the query must fail", failed.error); + Assert.assertTrue(failed.error, errorPrefix == null || failed.error.startsWith(errorPrefix)); + ConnectionHealth h = client.health(); + Assert.assertEquals(h.toString(), ConnectionHealth.State.FAILED, h.getState()); + Assert.assertNotEquals(ConnectionHealth.NONE, h.getOutageSinceEpochMillis()); + + // FAILED must hold: the server answers again, yet the next query fails without reconnecting + handler.drop.set(false); + int upgrades = server.upgradeRequestCount(); + RecordingHandler next = new RecordingHandler(); + client.execute("SELECT 1", next); + Assert.assertNotNull("the next query must fail", next.error); + Assert.assertTrue(next.error, errorPrefix == null || next.error.startsWith(errorPrefix)); + Assert.assertEquals("no reconnect", upgrades, server.upgradeRequestCount()); + Assert.assertEquals(ConnectionHealth.State.FAILED, client.health().getState()); + } + } + } + private static TestWebSocketServer startServer(TestWebSocketServer.WebSocketServerHandler handler) throws IOException, InterruptedException { TestWebSocketServer server = new TestWebSocketServer(handler); @@ -429,4 +472,18 @@ public void onExecDone(short opType, long rowsAffected) { execDone++; } } + + // Rejects every batch with PARSE_ERROR, a TERMINAL category: replaying the same bytes cannot succeed. + private static final class RejectingHandler implements TestWebSocketServer.WebSocketServerHandler { + private final AtomicLong sequence = new AtomicLong(); + + @Override + public void onBinaryMessage(TestWebSocketServer.ClientHandler client, byte[] data) { + try { + client.sendBinary(QwpWireTestUtils.buildNack(sequence.getAndIncrement(), WebSocketResponse.STATUS_PARSE_ERROR)); + } catch (IOException ignored) { + // surfaces to the client as a transport error + } + } + } } From 8eaaef157d139419dc1c26df1689abfd957a6142 Mon Sep 17 00:00:00 2001 From: glasstiger Date: Thu, 15 Oct 2026 10:50:52 +0100 Subject: [PATCH 17/40] Address the level-3 review findings Fixes for the three Moderate findings of the level-3 review of this branch at 4355127d. RefreshingTokenProvider scheduled its proactive refresh from the token's lifetime alone and never consulted handout_floor. A token that lived less than about two minutes left its hand-out window before the refresh ran: a 100 s token was handed out for 40 s but refreshed 45-50 s after its fetch, so every caller in between waited for a fetch. The idle-client promise in the javadoc, the README and spec section 5.2 did not hold for such tokens. The schedule now starts the refresh at least p (min_refresh_interval, or half the lifetime when shorter) before the token leaves its hand-out window, but never sooner than p after the fetch. So min_refresh_interval stays a lower bound, and no source can drive a fetch storm. A token that lives 3 minutes or more keeps its schedule, so the existing C1 cases are unchanged. A token handed out for less than p still cannot be refreshed in time; the provider now logs that once, naming handout_floor. Spec v0.7 amends section 5.2 and C1, and the README and javadoc state the precondition. CursorWebSocketSendLoop.connectWithRetry stops retrying a retryable credential failure once the calling thread is interrupted, but no test pinned that. With the clause removed, all 421 tests that reach the SYNC connect path passed, while an interrupted SYNC build spun through its whole budget because every backoff park returned at once: 818,107 provider pulls in 3 s. WebSocketDynamicCredentialTest gains testSyncInitializationStopsRetryingWhenTheThreadIsInterrupted. Two health computations had no failing test either: - ConnectionHealth.Aggregate's oldest-outage and latest-failure selection could be inverted unnoticed, because the only facade test used connections that were all connected. - Dropping the send loop's connectionLost() call also passed every test, because each RECONNECTING assertion ran after a failed round, which sets that state too. Without the call, a sender reports CONNECTED for its whole first reconnect round. ConnectionHealthTest gains testAggregateReportsTheOldestOutageAndTheLatestFailure and testSenderHealthIsReconnectingFromTheMomentItLosesItsConnection. Verification: each new test fails with its fix reverted. The schedule tests fail with the hand-out window bound removed, the once-only warning test with the flag dropped (two warnings), the SYNC test with the interrupt clause removed, and the health tests with the comparisons inverted or connectionLost() removed. The 15 related core test classes (120 tests) and the azure module (26 tests) pass. --- README.md | 10 +- .../cutlass/auth/RefreshingTokenProvider.java | 49 ++++++-- .../auth/RefreshingTokenProviderTest.java | 115 +++++++++++++++++- .../qwp/client/ConnectionHealthTest.java | 78 ++++++++++++ .../WebSocketDynamicCredentialTest.java | 39 ++++++ design/entra-id-qwp-auth.md | 2 +- design/qwp-token-provider-spec.md | 11 +- 7 files changed, 281 insertions(+), 23 deletions(-) diff --git a/README.md b/README.md index a74ad4236..2176e0db0 100644 --- a/README.md +++ b/README.md @@ -559,12 +559,16 @@ retryable. Never put a token or a raw response body in the message. The provider keeps serving the current token while it is still valid. The provider hands a token out only while it has more than 60 s left (`handoutFloorMillis`, the hand-out floor). A -source whose token arrives with 60 s or less left gets a failed fetch whose error names `handout_floor`; if your -platform issues tokens that short, lower the floor with `handoutFloorMillis(...)`. +source whose token arrives with 60 s or less left gets a failed fetch whose error names `handout_floor`. A token that +arrives with less than 90 s left is handed out, but it cannot be refreshed before it enters the floor, because +refreshes are at least 30 s apart (`minRefreshIntervalMillis`): callers wait for a fetch whenever such a token runs +out, and the provider logs a warning once. If your platform issues tokens that short, lower the floor with +`handoutFloorMillis(...)`. How failures are handled: -- A token is refreshed at about half its lifetime, so an idle client always holds a valid one. +- A token is refreshed at about half its lifetime, and always before it enters the hand-out floor, so an idle client + always holds a valid one. - When the server rejects a token with `401`, the client asks the provider for a new one and retries the same server once, immediately. - At startup, `initial_connect_retry=off` (the default) fails if no token can be obtained; `on` keeps retrying a diff --git a/core/src/main/java/io/questdb/client/cutlass/auth/RefreshingTokenProvider.java b/core/src/main/java/io/questdb/client/cutlass/auth/RefreshingTokenProvider.java index efe2598b9..9a17029e8 100644 --- a/core/src/main/java/io/questdb/client/cutlass/auth/RefreshingTokenProvider.java +++ b/core/src/main/java/io/questdb/client/cutlass/auth/RefreshingTokenProvider.java @@ -66,9 +66,13 @@ * thread. {@link #awaitReady(long)} lets an application gate startup on it. *

  • Proactive refresh. After a fetch received at {@code f} with expiry {@code e}, the next fetch is * scheduled at a jittered half-life, {@code f + (e - f) * U(0.45, 0.55)}, no later than - * {@code e - min(refresh_margin, (e - f) / 2)}, no earlier than {@code f + min(min_refresh_interval, - * (e - f) / 2)}, and no later than the platform's refresh hint when it gave one. While the source is healthy, - * even an idle client therefore always holds a usable token.
    • + * {@code e - min(refresh_margin, (e - f) / 2)}, no earlier than {@code f + p}, where + * {@code p = min(min_refresh_interval, (e - f) / 2)}, and no later than the platform's refresh hint when it gave + * one. It also runs at least {@code p} before the token leaves its hand-out window at {@code e - handout_floor}, + * unless that is sooner than {@code f + p}. While the source is healthy, even an idle client therefore always + * holds a usable token, provided a token is handed out for longer than {@code p}: with the defaults, provided it + * lives longer than 90 s. A shorter-lived token cannot be refreshed while it is still handed out, so callers + * wait for a fetch whenever one stops being usable, and the provider logs a warning once. *
  • Hand-out. A token is usable only while {@code now < e - handout_floor}. {@link #getToken()} * returns a usable token at once - a volatile read, no I/O, no lock. With no usable token it starts a fetch * (unless one is running or the provider is backing off after failures) and waits up to {@code cold_wait}, @@ -157,6 +161,7 @@ public final class RefreshingTokenProvider implements HttpTokenProvider, QuietCl private long scheduledDueNanos; private long scheduledSeq; private ScheduledTask scheduledTask; + private boolean shortWindowWarned; private RefreshingTokenProvider(Builder b) { this.source = b.source; @@ -191,7 +196,10 @@ public static Builder builder(TokenSource source) { /** * The refresh instant of the specification's section 5.2 for a token received at {@code f} that expires at - * {@code e}, given a uniform sample {@code u} in {@code [0, 1)}. Pure; exposed for the conformance tests. + * {@code e}, given a uniform sample {@code u} in {@code [0, 1)}: the jittered half-life, clamped by + * {@code refresh_margin}, {@code min_refresh_interval} and the refresh hint, and moved to at least the refresh + * spacing before the token leaves its hand-out window, unless that is sooner than the spacing after the fetch. + * Pure; exposed for the conformance tests. */ @TestOnly public static long computeRefreshAtMillis( @@ -200,20 +208,25 @@ public static long computeRefreshAtMillis( long refreshAt, double u, long refreshMarginMillis, - long minRefreshIntervalMillis + long minRefreshIntervalMillis, + long handoutFloorMillis ) { final long life = e - f; final long half = life / 2; final long margin = Math.min(refreshMarginMillis, half); - final long floor = Math.min(minRefreshIntervalMillis, half); + // the least time between a fetch and the refresh that follows it: p in section 5.2 + final long spacing = Math.min(minRefreshIntervalMillis, half); final double fraction = 0.45 + 0.10 * Math.max(0.0, Math.min(1.0, u)); long r = f + (long) (life * fraction); r = Math.min(r, e - margin); - r = Math.max(r, f + floor); + r = Math.max(r, f + spacing); if (refreshAt > 0) { - r = Math.min(r, Math.max(refreshAt, f + floor)); + r = Math.min(r, Math.max(refreshAt, f + spacing)); } - return r; + // Refresh at least the spacing before the token leaves its hand-out window, so the fetch and its retries run + // while the current token is still handed out. A short-lived token's window can close sooner than the spacing + // after this fetch; the spacing wins, and such a token is fetched when a caller needs it. + return Math.min(r, Math.max(e - handoutFloorMillis - spacing, f + spacing)); } private static String instant(long epochMillis) { @@ -576,7 +589,8 @@ private void onSuccessLocked(ExpiringToken result, long receivedAt, long receive result.getRefreshAtEpochMillis(), uniform(), refreshMarginMillis, - minRefreshIntervalMillis + minRefreshIntervalMillis, + handoutFloorMillis ); final long delayNanos = millisToNanos(refreshAt - receivedAt); final int failures = consecutiveFailures; @@ -594,6 +608,18 @@ private void onSuccessLocked(ExpiringToken result, long receivedAt, long receive name, CredentialRedaction.describeToken(result.getToken()), instant(result.getExpiresAtEpochMillis()), instant(refreshAt)); } + final long handoutEnd = result.getExpiresAtEpochMillis() - handoutFloorMillis; + if (!shortWindowWarned && refreshAt > handoutEnd) { + // The token stops being handed out before the next refresh may run, so callers will wait for a fetch. + // Say why, once: the cure is configuration. + shortWindowWarned = true; + LOG.warn("token provider {}: a token from the source is handed out for only {} ms - it expires {} ms " + + "after it arrives and handout_floor is {} ms - but the next refresh cannot run sooner " + + "than {} ms after a fetch, so callers wait for a fetch whenever a token stops being " + + "usable; lower handout_floor or min_refresh_interval, or issue longer-lived tokens", + name, handoutEnd - receivedAt, result.getExpiresAtEpochMillis() - receivedAt, handoutFloorMillis, + refreshAt - receivedAt); + } } // Starts a fetch now unless one is running, one is already due, or the provider is in failure backoff. @@ -846,7 +872,8 @@ public Builder forcedWaitMillis(long millis) { /** * A token is usable only while {@code now < expires_at - handout_floor}. Default 60 s. A fetched token that * is not usable on arrival counts as a failed fetch, so a source whose tokens live no longer than this needs - * a lower value. + * a lower value. A token handed out for no longer than {@code min_refresh_interval} cannot be refreshed while + * it is still handed out, so a source whose tokens live only that much longer needs a lower value too. */ public Builder handoutFloorMillis(long millis) { this.handoutFloorMillis = nonNegative("handout_floor", millis); diff --git a/core/src/test/java/io/questdb/client/test/cutlass/auth/RefreshingTokenProviderTest.java b/core/src/test/java/io/questdb/client/test/cutlass/auth/RefreshingTokenProviderTest.java index 795ada15c..702b9a306 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/auth/RefreshingTokenProviderTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/auth/RefreshingTokenProviderTest.java @@ -514,7 +514,33 @@ public void testRefreshScheduleFollowsTheSpec() { assertRefreshDelay(50_000L, 0.0, 0, 25_000L); // the pure function agrees with the provider Assert.assertEquals(T0 + 1_800_000L, - RefreshingTokenProvider.computeRefreshAtMillis(T0, T0 + HOUR, 0, 0.5, 5 * MIN, 30_000)); + RefreshingTokenProvider.computeRefreshAtMillis(T0, T0 + HOUR, 0, 0.5, 5 * MIN, 30_000, 0)); + } + + @Test + public void testRefreshScheduleKeepsTheRefreshInsideTheHandoutWindow() { + // C1 with the default 60 s handout_floor: a refresh runs at least min_refresh_interval (30 s) before the token + // leaves its hand-out window, unless that is sooner than 30 s after the fetch. + final long floor = RefreshingTokenProvider.DEFAULT_HANDOUT_FLOOR_MILLIS; + // long-lived tokens keep their schedule + assertRefreshDelay(floor, DAY, 0.5, 0, 43_200_000L); + assertRefreshDelay(floor, HOUR, 1.0, 0, 1_980_000L); + assertRefreshDelay(floor, 5 * MIN, 0.5, 0, 150_000L); + assertRefreshDelay(floor, 180_000L, 0.5, 0, 90_000L); // the shortest lifetime the bound leaves alone + // a shorter-lived token is refreshed 30 s before its hand-out window closes ... + assertRefreshDelay(floor, 150_000L, 0.0, 0, 60_000L); // handed out for 90 s; the half-life is 67.5-75 s + assertRefreshDelay(floor, 150_000L, 0.5, 0, 60_000L); + assertRefreshDelay(floor, 120_000L, 0.5, 0, 30_000L); // handed out for 60 s; the half-life is 60 s + // ... but no sooner than 30 s after the fetch, which is still inside a window longer than 30 s + assertRefreshDelay(floor, 100_000L, 0.0, 0, 30_000L); // handed out for 40 s; the half-life is 45-50 s + assertRefreshDelay(floor, 100_000L, 1.0, 0, 30_000L); + // an earlier refresh hint still wins + assertRefreshDelay(floor, 150_000L, 0.5, T0 + 40_000L, 40_000L); + // a token handed out for 30 s or less cannot be refreshed inside its window: the 30 s spacing wins + assertRefreshDelay(floor, 75_000L, 0.5, 0, 30_000L); // handed out for 15 s + // the pure function agrees with the provider + Assert.assertEquals(T0 + 30_000L, + RefreshingTokenProvider.computeRefreshAtMillis(T0, T0 + 100_000L, 0, 0.5, 5 * MIN, 30_000, floor)); } @Test @@ -542,7 +568,7 @@ public void testSameTokenConvergesAndStaysUsable() { long remaining = expiry - clock.wallClockMillis(); ManualScheduler.Task next = scheduler.peek(); long expectedDelay = RefreshingTokenProvider.computeRefreshAtMillis( - 0, remaining, 0, 0.5, 5 * MIN, 30_000); + 0, remaining, 0, 0.5, 5 * MIN, 30_000, RefreshingTokenProvider.DEFAULT_HANDOUT_FLOOR_MILLIS); Assert.assertEquals("refresh at about half the remaining lifetime [remaining=" + remaining + ']', expectedDelay, next.delayMillis()); Assert.assertTrue(next.delayMillis() <= remaining / 2 + 1); @@ -553,7 +579,9 @@ public void testSameTokenConvergesAndStaysUsable() { Assert.assertEquals("SAME", provider.getToken().toString()); fetchesWhileUsable++; } - // 24 h halves to below the 60 s floor in 11 steps (86400 s -> ... -> 84.4 s) + // The remaining lifetime halves over the first 10 fetches (86400 s -> ... -> 168.75 s). The 11th runs + // 30 s before the token would leave its hand-out window, at 90 s left, and the next one finds it inside + // the floor. Assert.assertEquals(11, fetchesWhileUsable); // the fetch that returned the token inside the floor is a failed fetch (section 4) Assert.assertEquals(1, provider.getConsecutiveFailures()); @@ -627,6 +655,33 @@ public void testSentinelTokenIsNeverRendered() throws Exception { Assert.assertTrue(rendered.get(0), rendered.get(0).contains("sha256:")); } + @Test + public void testShortLivedTokenIsRefreshedWhileItIsStillHandedOut() { + // section 5.2: a 100 s token is handed out for 40 s. Its refresh runs 30 s after the fetch, while it is still + // handed out, so an idle client always holds a usable token. Scheduled at its half-life alone, the refresh + // ran 45-50 s after the fetch, and every caller after the token left its window waited for a fetch. + FakeClock clock = new FakeClock(T0); + ManualScheduler scheduler = new ManualScheduler(clock); + ScriptedSource source = new ScriptedSource().then(() -> new ExpiringToken( + "T" + (clock.wallClockMillis() - T0), clock.wallClockMillis() + 100_000L)); + try (RefreshingTokenProvider provider = provider(source, clock, scheduler, 0.5).build()) { + scheduler.runNext(); + long previousFetch = T0; + for (int refresh = 1; refresh <= 3; refresh++) { + Assert.assertEquals("refresh 30 s after the fetch", 30_000L, scheduler.peek().delayMillis()); + scheduler.advanceAndRunNext(); + Assert.assertEquals(previousFetch + 30_000L, clock.wallClockMillis()); + // one second after the previous token left its hand-out window: fetch + 100 s - 60 s floor + clock.advanceMillis(previousFetch + 41_000L - clock.wallClockMillis()); + Assert.assertEquals("an idle client holds a usable token", + "T" + refresh * 30_000L, provider.getToken().toString()); + Assert.assertFalse("no caller had to start a fetch", scheduler.isNextImmediate()); + previousFetch += 30_000L; + } + Assert.assertEquals(4, source.calls()); + } + } + @Test public void testSourceFailureTextIsSanitized() { FakeClock clock = new FakeClock(T0); @@ -671,6 +726,45 @@ public void testSuccessResetsTheBackoff() { } } + @Test + public void testTokenHandedOutTooBrieflyToRefreshAheadIsReportedOnce() { + // section 5.2: a 75 s token is handed out for 15 s, less than min_refresh_interval, so it cannot be refreshed + // while it is still handed out. The refresh keeps its 30 s spacing - no fetch storm - and the provider says + // once why callers wait for a fetch. + ch.qos.logback.classic.Logger logger = (ch.qos.logback.classic.Logger) + org.slf4j.LoggerFactory.getLogger(RefreshingTokenProvider.class); + ch.qos.logback.core.read.ListAppender appender = + new ch.qos.logback.core.read.ListAppender<>(); + appender.start(); + ch.qos.logback.classic.Level savedLevel = logger.getLevel(); + logger.setLevel(ch.qos.logback.classic.Level.ALL); + logger.addAppender(appender); + FakeClock clock = new FakeClock(T0); + ManualScheduler scheduler = new ManualScheduler(clock); + ScriptedSource source = new ScriptedSource().then( + () -> new ExpiringToken("BRIEF", clock.wallClockMillis() + 75_000L)); + try (RefreshingTokenProvider provider = provider(source, clock, scheduler, 0.5).build()) { + scheduler.runNext(); + Assert.assertEquals("the refresh keeps its spacing", 30_000L, scheduler.peek().delayMillis()); + scheduler.advanceAndRunNext(); + Assert.assertEquals(30_000L, scheduler.peek().delayMillis()); + Assert.assertEquals(2, source.calls()); + } finally { + logger.detachAppender(appender); + logger.setLevel(savedLevel); + appender.stop(); + } + List warnings = new ArrayList<>(); + for (ch.qos.logback.classic.spi.ILoggingEvent event : appender.list) { + if (event.getLevel() == ch.qos.logback.classic.Level.WARN) { + warnings.add(event.getFormattedMessage()); + } + } + Assert.assertEquals("reported once, not once per fetch: " + warnings, 1, warnings.size()); + Assert.assertTrue(warnings.get(0), warnings.get(0).contains("handed out for only 15000 ms")); + Assert.assertTrue(warnings.get(0), warnings.get(0).contains("lower handout_floor")); + } + @Test(timeout = 30_000) public void testUnclassifiedSourceExceptionIsRetryable() { ScriptedSource source = new ScriptedSource().thenThrow(new IllegalStateException("socket reset")); @@ -751,16 +845,27 @@ private static void assertFirstBackoff(long retryAfterMillis, long expectedDelay } private static void assertRefreshDelay(long lifetimeMillis, double u, long refreshAt, long expectedDelayMillis) { + assertRefreshDelay(0, lifetimeMillis, u, refreshAt, expectedDelayMillis); + } + + private static void assertRefreshDelay( + long handoutFloorMillis, + long lifetimeMillis, + double u, + long refreshAt, + long expectedDelayMillis + ) { FakeClock clock = new FakeClock(T0); ManualScheduler scheduler = new ManualScheduler(clock); ScriptedSource source = new ScriptedSource() .then(() -> new ExpiringToken("TOKEN", T0 + lifetimeMillis, refreshAt)); try (RefreshingTokenProvider provider = provider(source, clock, scheduler, u) - .handoutFloorMillis(0) + .handoutFloorMillis(handoutFloorMillis) .build()) { Assert.assertEquals("the prefetch runs at once", 0, scheduler.peek().delayNanos); scheduler.runNext(); - Assert.assertEquals("refresh delay [L=" + lifetimeMillis + ", u=" + u + ", refreshAt=" + refreshAt + ']', + Assert.assertEquals("refresh delay [handoutFloor=" + handoutFloorMillis + ", L=" + lifetimeMillis + + ", u=" + u + ", refreshAt=" + refreshAt + ']', expectedDelayMillis, scheduler.peek().delayMillis()); Assert.assertEquals(T0 + lifetimeMillis, provider.getTokenExpiresAtEpochMillis()); } diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/ConnectionHealthTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/ConnectionHealthTest.java index c21b1a92b..cb752e918 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/ConnectionHealthTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/ConnectionHealthTest.java @@ -42,6 +42,11 @@ import java.io.IOException; import java.nio.ByteBuffer; import java.nio.ByteOrder; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.List; +import java.util.concurrent.CountDownLatch; import java.util.concurrent.TimeUnit; import java.util.concurrent.atomic.AtomicBoolean; import java.util.concurrent.atomic.AtomicLong; @@ -57,6 +62,35 @@ public class ConnectionHealthTest { private static final String SENTINEL = "SENTINEL-TOKEN-7f3a91"; + @Test + public void testAggregateReportsTheOldestOutageAndTheLatestFailure() { + // C21 for the facade: QuestDB.health() reports how long the longest outage has lasted and what failed last, + // whatever the order of the pooled connections. + ConnectionHealth.Failure olderFailure = new ConnectionHealth.Failure( + ConnectionHealth.FailureClass.TRANSPORT, 0, "connection refused", 6_000L); + ConnectionHealth.Failure newerFailure = new ConnectionHealth.Failure( + ConnectionHealth.FailureClass.AUTH_REJECTED, 401, "auth-rejected", 10_000L); + List healths = Arrays.asList( + new ConnectionHealth(ConnectionHealth.State.CONNECTED, 1_000L, ConnectionHealth.NONE, 0, null), + new ConnectionHealth(ConnectionHealth.State.RECONNECTING, 1_000L, 9_000L, 1, newerFailure), + new ConnectionHealth(ConnectionHealth.State.RECONNECTING, 1_000L, 5_000L, 3, olderFailure), + new ConnectionHealth(ConnectionHealth.State.CONNECTING, ConnectionHealth.NONE, 7_000L, 0, null) + ); + for (int rotation = 0; rotation < healths.size(); rotation++) { + List ordered = new ArrayList<>(healths); + Collections.rotate(ordered, rotation); + ConnectionHealth.Aggregate aggregate = ConnectionHealth.Aggregate.of(ordered); + Assert.assertEquals(4, aggregate.total()); + Assert.assertEquals(1, aggregate.count(ConnectionHealth.State.CONNECTED)); + Assert.assertEquals(2, aggregate.count(ConnectionHealth.State.RECONNECTING)); + Assert.assertEquals(1, aggregate.count(ConnectionHealth.State.CONNECTING)); + Assert.assertEquals("the oldest outage [rotation=" + rotation + ']', + 5_000L, aggregate.getOldestOutageSinceEpochMillis()); + Assert.assertSame("the most recent failure [rotation=" + rotation + ']', + newerFailure, aggregate.getLastFailure()); + } + } + @Test(timeout = 60_000) public void testAuthenticationOutageNeverMakesTheSenderTerminal() throws Exception { // C22: an established sender rides out credential-unavailable and 401 rounds, with role rejects in @@ -271,6 +305,50 @@ public void testSenderHealthIsFailedWithAnOutageAfterATerminalRejection() throws }); } + @Test(timeout = 60_000) + public void testSenderHealthIsReconnectingFromTheMomentItLosesItsConnection() throws Exception { + // C21: an established sender that loses its connection reports RECONNECTING, with the outage started, while + // its first reconnect round is still running - not only once a round has failed. That round can be slow: a + // provider fetching a token, or a connect to a host that does not answer. + assertMemoryLeak(() -> { + AtomicBoolean holdPulls = new AtomicBoolean(); + CountDownLatch pullHeld = new CountDownLatch(1); + CountDownLatch release = new CountDownLatch(1); + try (TestWebSocketServer server = startServer(new WebSocketDynamicCredentialTest.AckHandler()); + Sender sender = Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + server.getPort()) + .httpTokenProvider(() -> { + if (holdPulls.get()) { + pullHeld.countDown(); + try { + release.await(30, TimeUnit.SECONDS); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + } + return SENTINEL; + }) + .build()) { + Assert.assertEquals(ConnectionHealth.State.CONNECTED, sender.health().getState()); + try { + holdPulls.set(true); + server.dropAllConnections(); + Assert.assertTrue("the reconnect round pulls a credential", pullHeld.await(10, TimeUnit.SECONDS)); + ConnectionHealth h = sender.health(); + Assert.assertEquals(h.toString(), ConnectionHealth.State.RECONNECTING, h.getState()); + Assert.assertNotEquals("the outage starts when the connection is lost: " + h, + ConnectionHealth.NONE, h.getOutageSinceEpochMillis()); + Assert.assertEquals("no round has failed yet", 0, h.getFailedRounds()); + } finally { + holdPulls.set(false); + release.countDown(); + } + await(() -> sender.health().getState() == ConnectionHealth.State.CONNECTED, 10_000, "reconnected"); + Assert.assertEquals(ConnectionHealth.NONE, sender.health().getOutageSinceEpochMillis()); + } + }); + } + @Test(timeout = 60_000) public void testSenderHealthMovesThroughEveryState() throws Exception { // C21: connecting -> connected -> reconnecting (with 401 and credential-unavailable failures) -> diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/WebSocketDynamicCredentialTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/WebSocketDynamicCredentialTest.java index 47f35c3ec..dbff5ccb7 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/WebSocketDynamicCredentialTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/WebSocketDynamicCredentialTest.java @@ -761,6 +761,45 @@ public void testSyncInitializationRetryableProviderFailureExhaustsTheBudgetNamin }); } + @Test(timeout = 30_000) + public void testSyncInitializationStopsRetryingWhenTheThreadIsInterrupted() throws Exception { + // C7 for SYNC startup: an interrupt cancels a startup that is retrying a retryable credential failure. The + // provider's own error surfaces after the first pull, with the flag kept. Retrying instead would spin through + // the whole budget, since every backoff park returns at once while the flag is set. + assertMemoryLeak(() -> { + AtomicInteger pulls = new AtomicInteger(); + try (TestWebSocketServer server = startServer(new AckHandler())) { + Throwable failure = null; + try { + Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + server.getPort()) + .initialConnectMode(Sender.InitialConnectMode.SYNC) + .reconnectMaxDurationMillis(10_000) + .reconnectInitialBackoffMillis(10) + .reconnectMaxBackoffMillis(20) + .httpTokenProvider(() -> { + pulls.incrementAndGet(); + // the application cancels startup - Future.cancel(true) - while the IdP is down + Thread.currentThread().interrupt(); + throw TokenUnavailableException.retryable("IdP unreachable"); + }) + .build() + .close(); + } catch (Throwable t) { + failure = t; + } + // read and clear the flag before anything else can block on it + boolean interruptKept = Thread.interrupted(); + Assert.assertTrue("expected the provider's own error, got " + failure, + failure instanceof TokenUnavailableException); + Assert.assertTrue(failure.getMessage(), failure.getMessage().contains("IdP unreachable")); + Assert.assertTrue("the interrupt flag must be kept", interruptKept); + Assert.assertEquals("an interrupted startup must not retry", 1, pulls.get()); + Assert.assertEquals(0, server.upgradeRequestCount()); + } + }); + } + private static void renderThrowable(Throwable t, List into) { for (Throwable c = t; c != null; c = c.getCause()) { into.add(c.toString()); diff --git a/design/entra-id-qwp-auth.md b/design/entra-id-qwp-auth.md index 50ff8416c..793e640ef 100644 --- a/design/entra-id-qwp-auth.md +++ b/design/entra-id-qwp-auth.md @@ -4,7 +4,7 @@ Status: **implemented** on `feat/qwp-entra-token-provider` - all four steps of t **Read these first:** -- **The spec wins.** The normative, language-neutral contract is now `design/qwp-token-provider-spec.md` (v0.5, all decisions resolved). Where the two documents differ, follow the spec. That includes the refresh-schedule clamps, backoff jitter, registry linger, and the Java names in its Appendix B, which supersede §7.2 here. +- **The spec wins.** The normative, language-neutral contract is now `design/qwp-token-provider-spec.md` (v0.7, all decisions resolved). Where the two documents differ, follow the spec. That includes the refresh-schedule clamps, backoff jitter, registry linger, and the Java names in its Appendix B, which supersede §7.2 here. - **Java plan:** §10. - **Line references** are as of `c9968f24` and may drift. diff --git a/design/qwp-token-provider-spec.md b/design/qwp-token-provider-spec.md index 3dd922c5c..d3ade3f7f 100644 --- a/design/qwp-token-provider-spec.md +++ b/design/qwp-token-provider-spec.md @@ -1,9 +1,10 @@ # QWP dynamic bearer credentials: cross-language specification -**Status:** v0.6, ready for implementation. All decisions are resolved (§12). +**Status:** v0.7, ready for implementation. All decisions are resolved (§12). **Changes:** +- **v0.7:** a refresh starts at least `p` before the token leaves its hand-out window (§5.2). Before, a token that lived less than about two minutes was refreshed only after it had stopped being handed out, so in every cycle callers waited for a fetch. A token handed out for less than `p` still cannot be refreshed in time; the provider logs that once (§5.2, C1). - **v0.6:** the optional authentication-outage deadline (§8.5, D7) is withdrawn. It made an established sender terminal on a transient outage, which store-and-forward forbids, and the role-reject and transport rounds of a failover counted toward it. An application that prefers a hard failure closes the sender itself, guided by the error channel and the connection health (§8.5). - **v0.5:** a result whose token expires within `handout_floor` of its receipt is a retryable failed fetch whose message names `handout_floor` (§4, C2, C6). Before, it counted as a success that could never be handed out, so every caller waited out `cold_wait` for an error that named no cause. - **v0.4:** `azure_credential` selects one Azure credential deterministically (§7.1), so the library itself reports an unreachable managed-identity endpoint (retryable) apart from a misconfigured credential (permanent) (§7.5, D10). A discovery chain's "no credential available" stays permanent, so a host without a credential still fails fast, and the client warns once that the chain does not ride out an endpoint outage. This resolves the conflict between §4 and §7.5 over an unreachable managed-identity endpoint. @@ -134,15 +135,19 @@ r = min(r, e - m) r = max(r, f + p) if refresh_at is present: r = min(r, max(refresh_at, f + p)) +r = min(r, max(e - handout_floor - p, f + p)) ``` - **Proactive refresh:** the provider MUST start a fetch at `r` without waiting for a caller. -- **Effect:** while the source is healthy, even an idle client always holds a usable token. +- **Hand-out window:** a token is handed out until `e - handout_floor` (§5.1). The last step starts the refresh at least `p` before then, so the fetch and its retries run while the current token is still handed out, but never sooner than `p` after the fetch. With the defaults it changes nothing for a token that lives 3 minutes or more. +- **Effect:** while the source is healthy, even an idle client always holds a usable token, provided a token is handed out for longer than `p`. With the defaults, that is a token that lives longer than 90 s. +- **Short hand-out window:** a token handed out for less than `p` cannot be refreshed while it is still handed out without breaking `min_refresh_interval`, so callers wait for a fetch whenever it stops being usable. The provider SHOULD log a warning once that names `handout_floor`, so that the configuration can be fixed. *Informative examples:* - A 60-minute service-principal token is refreshed at about 27–33 minutes. - A 24-hour managed-identity token is refreshed at about 11–13 hours. +- A 100-second token is handed out for 40 s and refreshed 30 s after its fetch. Without the last step it would be refreshed 45–50 s after its fetch, after it had stopped being handed out. - If the platform keeps returning the same token, each refresh happens at about half of the remaining lifetime. So a token is fetched only a handful of times before it expires. ### 5.3 Handing out a token: `get_token()` @@ -417,7 +422,7 @@ Every client that implements this specification SHOULD pass these scenarios. The | ID | Scenario | Expected result | |---|---|---| -| C1 | Refresh schedule | Refresh times follow §5.2 for `L` = 24 h, 60 min and 5 min, with and without `refresh_at`. | +| C1 | Refresh schedule | Refresh times follow §5.2 for `L` = 24 h, 60 min and 5 min, with and without `refresh_at`. With the default `handout_floor`, `L` = 150 s, 120 s and 100 s are refreshed inside their hand-out window, and `L` = 75 s, which cannot be, keeps the `p` spacing and is logged once. | | C2 | Same token returned | Fetches converge as described in §5.2, and the token stays usable until it enters `handout_floor`. The fetch that returns it inside the floor is a retryable failure (§4). | | C3 | Cold burst | 64 concurrent callers with no usable token cause exactly one fetch. | | C4 | Cold failure | A retryable error is raised within `cold_wait`. A permanent classification is passed through. | From 0bb9713f7d5abb1e70343e4d61dff819ab21067e Mon Sep 17 00:00:00 2001 From: glasstiger Date: Thu, 15 Oct 2026 13:05:31 +0100 Subject: [PATCH 18/40] Fix token provider linger and test assertions Tag scheduled linger callbacks with an entry generation so an already started callback cannot close a provider after a newer release resets the linger period. Add deterministic executor-gated tests for idle and active leases and verify the current callback still closes the provider. Capture worker failures in assertReturnsPromptly and report them on the test thread, preserving the cause. Cover runtime exceptions and errors. Validation: 128 tests pass on JDK 25 and 42 focused tests pass on JDK 8. The broader JDK 8 run passes all 128 with tiered compilation disabled. The default JDK 8 run crashes in Unsafe.recordMemAlloc; the same crash also reproduces on unchanged parent 8eaaef15. --- .../cutlass/auth/TokenProviderRegistry.java | 19 +- .../auth/RefreshingTokenProviderTest.java | 30 +++- .../auth/TokenProviderRegistryTest.java | 167 ++++++++++++++++++ 3 files changed, 211 insertions(+), 5 deletions(-) create mode 100644 core/src/test/java/io/questdb/client/test/cutlass/auth/TokenProviderRegistryTest.java diff --git a/core/src/main/java/io/questdb/client/cutlass/auth/TokenProviderRegistry.java b/core/src/main/java/io/questdb/client/cutlass/auth/TokenProviderRegistry.java index 7c55a6f74..16ac455db 100644 --- a/core/src/main/java/io/questdb/client/cutlass/auth/TokenProviderRegistry.java +++ b/core/src/main/java/io/questdb/client/cutlass/auth/TokenProviderRegistry.java @@ -78,6 +78,15 @@ public TokenProviderRegistry(long lingerMillis) { this.lingerMillis = lingerMillis; } + /** + * Test seam: lets lifecycle tests control when linger callbacks run. + */ + @TestOnly + public TokenProviderRegistry(long lingerMillis, ScheduledThreadPoolExecutor lingerTimer) { + this(lingerMillis); + this.lingerTimer = lingerTimer; + } + /** * Finds the factory serving {@code name} through {@link ServiceLoader}, trying the thread context class * loader first and then the loader of this library. @@ -212,10 +221,10 @@ public int leaseCount(TokenProviderSpec spec) { } } - private void expire(Entry entry) { + private void expire(Entry entry, long generation) { synchronized (lock) { - if (entry.refs > 0 || entries.get(entry.key) != entry) { - return; // re-leased during the linger, or already replaced + if (entry.lingerGeneration != generation || entry.refs > 0 || entries.get(entry.key) != entry) { + return; // a newer linger superseded this callback, a lease is active, or another entry replaced it } entries.remove(entry.key); entry.lingerTask = null; @@ -248,7 +257,8 @@ private void release(Entry entry) { entries.remove(entry.key, entry); toClose = entry.provider; } else { - entry.lingerTask = lingerTimer().schedule(() -> expire(entry), lingerMillis, TimeUnit.MILLISECONDS); + final long generation = ++entry.lingerGeneration; + entry.lingerTask = lingerTimer().schedule(() -> expire(entry, generation), lingerMillis, TimeUnit.MILLISECONDS); } } if (toClose != null) { @@ -259,6 +269,7 @@ private void release(Entry entry) { private static final class Entry { final String key; final RefreshingTokenProvider provider; + long lingerGeneration; ScheduledFuture lingerTask; int refs; diff --git a/core/src/test/java/io/questdb/client/test/cutlass/auth/RefreshingTokenProviderTest.java b/core/src/test/java/io/questdb/client/test/cutlass/auth/RefreshingTokenProviderTest.java index 702b9a306..c0ba14efb 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/auth/RefreshingTokenProviderTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/auth/RefreshingTokenProviderTest.java @@ -486,6 +486,24 @@ public void testProactiveRefreshRunsWithoutACaller() { } } + @Test + public void testPromptReturnAssertionPropagatesErrors() throws Exception { + AssertionError failure = new AssertionError("worker error"); + AssertionError reported = Assert.assertThrows(AssertionError.class, () -> assertReturnsPromptly(() -> { + throw failure; + })); + Assert.assertSame(failure, reported.getCause()); + } + + @Test + public void testPromptReturnAssertionPropagatesExceptions() throws Exception { + IllegalStateException failure = new IllegalStateException("worker exception"); + AssertionError reported = Assert.assertThrows(AssertionError.class, () -> assertReturnsPromptly(() -> { + throw failure; + })); + Assert.assertSame(failure, reported.getCause()); + } + @Test public void testRefreshScheduleFollowsTheSpec() { // C1: refresh times follow section 5.2 for L = 24 h, 60 min and 5 min, with and without refresh_at. @@ -881,9 +899,19 @@ private static void assertRejected(String token) { } private static void assertReturnsPromptly(Runnable r) throws InterruptedException { - Thread t = start(r); + AtomicReference failure = new AtomicReference<>(); + Thread t = start(() -> { + try { + r.run(); + } catch (Throwable e) { + failure.set(e); + } + }); t.join(2_000); Assert.assertFalse("the call must return without waiting", t.isAlive()); + if (failure.get() != null) { + throw new AssertionError("the call must return normally", failure.get()); + } } private static List burst(RefreshingTokenProvider provider, int callers, ScriptedSource source) diff --git a/core/src/test/java/io/questdb/client/test/cutlass/auth/TokenProviderRegistryTest.java b/core/src/test/java/io/questdb/client/test/cutlass/auth/TokenProviderRegistryTest.java new file mode 100644 index 000000000..7a6fe2d37 --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/cutlass/auth/TokenProviderRegistryTest.java @@ -0,0 +1,167 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test.cutlass.auth; + +import io.questdb.client.cutlass.auth.RefreshingTokenProvider; +import io.questdb.client.cutlass.auth.TokenProviderRegistry; +import io.questdb.client.cutlass.auth.TokenProviderSpec; +import io.questdb.client.impl.ConfigString; +import io.questdb.client.impl.ConfigView; +import org.junit.After; +import org.junit.Assert; +import org.junit.Before; +import org.junit.Test; + +import java.util.concurrent.BlockingQueue; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.LinkedBlockingQueue; +import java.util.concurrent.ScheduledFuture; +import java.util.concurrent.ScheduledThreadPoolExecutor; +import java.util.concurrent.TimeUnit; + +public class TokenProviderRegistryTest { + private TestTokenProviderFactory factory; + + @Before + public void setUp() { + factory = new TestTokenProviderFactory("vault"); + TestTokenProviderFactory.install(factory); + } + + @After + public void tearDown() { + TestTokenProviderFactory.uninstall(); + } + + @Test(timeout = 30_000) + public void testSupersededExpirationDoesNotCloseIdleProvider() throws Exception { + assertSupersededExpirationDoesNotCloseProvider(false); + } + + @Test(timeout = 30_000) + public void testSupersededExpirationDoesNotCloseLeasedProvider() throws Exception { + assertSupersededExpirationDoesNotCloseProvider(true); + } + + private void assertSupersededExpirationDoesNotCloseProvider(boolean hasActiveLease) throws Exception { + PausingLingerTimer timer = new PausingLingerTimer(); + TokenProviderRegistry registry = new TokenProviderRegistry(TokenProviderRegistry.DEFAULT_LINGER_MILLIS, timer); + TokenProviderSpec spec = TokenProviderSpec.parse(new ConfigView(ConfigString.parse( + "wss::addr=localhost:9000;token_provider=vault;")), true); + try (TokenProviderRegistry.Lease first = registry.acquire(spec)) { + RefreshingTokenProvider provider = first.provider(); + try { + Assert.assertTrue(provider.awaitReady(10_000)); + first.close(); + PausedExpiration superseded = timer.nextExpiration(); + try (TokenProviderRegistry.Lease second = registry.acquire(spec)) { + Assert.assertSame(provider, second.provider()); + Assert.assertTrue("acquiring a lease cancels the already-started expiration", + superseded.future.isCancelled()); + if (!hasActiveLease) { + second.close(); + } + superseded.finish(); + Assert.assertFalse("the old callback must not close the re-leased provider", provider.isClosed()); + Assert.assertTrue(registry.isActive(spec)); + Assert.assertEquals(hasActiveLease ? 1 : 0, registry.leaseCount(spec)); + Assert.assertEquals("the warm provider must not need replacement", 1, factory.created.get()); + } + PausedExpiration current = timer.nextExpiration(); + Assert.assertFalse("the old callback must leave the current expiration intact", current.future.isCancelled()); + current.finish(); + Assert.assertTrue("the current callback closes the idle provider", provider.isClosed()); + Assert.assertFalse(registry.isActive(spec)); + } finally { + provider.close(); + } + } finally { + timer.shutdownNow(); + Assert.assertTrue("the linger executor must stop", timer.awaitTermination(10, TimeUnit.SECONDS)); + } + } + + private static final class PausedExpiration implements Runnable { + private final Runnable command; + private final CountDownLatch finished = new CountDownLatch(1); + private final CountDownLatch proceed = new CountDownLatch(1); + private final CountDownLatch started = new CountDownLatch(1); + private volatile Throwable failure; + private ScheduledFuture future; + + private PausedExpiration(Runnable command) { + this.command = command; + } + + @Override + public void run() { + started.countDown(); + try { + proceed.await(); + command.run(); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } catch (Throwable e) { + failure = e; + } finally { + finished.countDown(); + } + } + + private void finish() throws InterruptedException { + proceed.countDown(); + Assert.assertTrue("the linger callback must finish", finished.await(10, TimeUnit.SECONDS)); + if (failure != null) { + throw new AssertionError("the linger callback failed", failure); + } + } + } + + private static final class PausingLingerTimer extends ScheduledThreadPoolExecutor { + private final BlockingQueue expirations = new LinkedBlockingQueue<>(); + + private PausingLingerTimer() { + super(1); + setRemoveOnCancelPolicy(true); + } + + @Override + public ScheduledFuture schedule(Runnable command, long delay, TimeUnit unit) { + // Drive deadlines explicitly, pausing each real executor task after it starts. cancel(false) + // cannot stop this already-running task from reaching expire() when the test releases it. + PausedExpiration expiration = new PausedExpiration(command); + expiration.future = super.schedule(expiration, 0, TimeUnit.NANOSECONDS); + expirations.add(expiration); + return expiration.future; + } + + private PausedExpiration nextExpiration() throws InterruptedException { + PausedExpiration expiration = expirations.poll(10, TimeUnit.SECONDS); + Assert.assertNotNull("a linger expiration must be scheduled", expiration); + Assert.assertTrue("the linger task must start", expiration.started.await(10, TimeUnit.SECONDS)); + return expiration; + } + } +} From 7f2bae8e5c04eb67e48170cc8c9f8c51c2aad17b Mon Sep 17 00:00:00 2001 From: glasstiger Date: Thu, 15 Oct 2026 14:06:59 +0100 Subject: [PATCH 19/40] Prevent redundant token refreshes Recheck the warm reader's token snapshot under the provider lock before pulling a scheduled refresh forward. Preserve the future schedule when another fetch has already published a new token. Add a deterministic concurrent-reader regression test and cover a successful refresh that outlasts a short token's handout window. Clarify the Javadoc, README and cross-language specification so continuous availability depends on refresh completion time, not source health alone. Validation: all 130 focused core and Azure tests pass on JDK 25 and JDK 8 with -Xint. The new race test fails without the fix. --- README.md | 7 +- .../cutlass/auth/RefreshingTokenProvider.java | 25 ++-- .../auth/RefreshingTokenProviderTest.java | 119 +++++++++++++++++- design/qwp-token-provider-spec.md | 8 +- 4 files changed, 139 insertions(+), 20 deletions(-) diff --git a/README.md b/README.md index 2176e0db0..7aaf10c95 100644 --- a/README.md +++ b/README.md @@ -567,8 +567,11 @@ out, and the provider logs a warning once. If your platform issues tokens that s How failures are handled: -- A token is refreshed at about half its lifetime, and always before it enters the hand-out floor, so an idle client - always holds a valid one. +- The provider refreshes a token proactively, usually at about half its lifetime, and starts earlier for shorter + hand-out windows where the minimum refresh interval permits. Uninterrupted availability requires the refresh to + finish before the current token enters the hand-out floor. Even a successful fetch can leave callers waiting: + with defaults, a 100 s token starts refreshing at +30 s and stops being handed out at +40 s, leaving only 10 s + for the refresh to complete. - When the server rejects a token with `401`, the client asks the provider for a new one and retries the same server once, immediately. - At startup, `initial_connect_retry=off` (the default) fails if no token can be obtained; `on` keeps retrying a diff --git a/core/src/main/java/io/questdb/client/cutlass/auth/RefreshingTokenProvider.java b/core/src/main/java/io/questdb/client/cutlass/auth/RefreshingTokenProvider.java index 9a17029e8..b2aebcadd 100644 --- a/core/src/main/java/io/questdb/client/cutlass/auth/RefreshingTokenProvider.java +++ b/core/src/main/java/io/questdb/client/cutlass/auth/RefreshingTokenProvider.java @@ -68,11 +68,11 @@ * scheduled at a jittered half-life, {@code f + (e - f) * U(0.45, 0.55)}, no later than * {@code e - min(refresh_margin, (e - f) / 2)}, no earlier than {@code f + p}, where * {@code p = min(min_refresh_interval, (e - f) / 2)}, and no later than the platform's refresh hint when it gave - * one. It also runs at least {@code p} before the token leaves its hand-out window at {@code e - handout_floor}, - * unless that is sooner than {@code f + p}. While the source is healthy, even an idle client therefore always - * holds a usable token, provided a token is handed out for longer than {@code p}: with the defaults, provided it - * lives longer than 90 s. A shorter-lived token cannot be refreshed while it is still handed out, so callers - * wait for a fetch whenever one stops being usable, and the provider logs a warning once.
    • + * one. The provider also aims to start at least {@code p} before the token leaves its hand-out window at + * {@code e - handout_floor}, but never sooner than {@code f + p}. Continuous availability requires each refresh + * to complete with a usable token before that cutoff; even a successful fetch can leave callers waiting if it + * takes longer. A token handed out for less than {@code p} cannot be proactively refreshed while it is still + * handed out, so callers wait for a fetch when it stops being usable, and the provider logs a warning once. *
  • Hand-out. A token is usable only while {@code now < e - handout_floor}. {@link #getToken()} * returns a usable token at once - a volatile read, no I/O, no lock. With no usable token it starts a fetch * (unless one is running or the provider is backing off after failures) and waits up to {@code cold_wait}, @@ -223,9 +223,9 @@ public static long computeRefreshAtMillis( if (refreshAt > 0) { r = Math.min(r, Math.max(refreshAt, f + spacing)); } - // Refresh at least the spacing before the token leaves its hand-out window, so the fetch and its retries run - // while the current token is still handed out. A short-lived token's window can close sooner than the spacing - // after this fetch; the spacing wins, and such a token is fetched when a caller needs it. + // Move the refresh earlier to leave room for fetching before the hand-out cutoff, without breaking the + // minimum scheduled spacing. A short window can leave less than the spacing for the fetch to complete, + // so callers may still wait even when the source succeeds. return Math.min(r, Math.max(e - handoutFloorMillis - spacing, f + spacing)); } @@ -356,7 +356,7 @@ public CharSequence getToken() { // lock, to start that fetch in the background. if (!fetchRunning && consecutiveFailures == 0 && (wallNow >= h.refreshAtEpochMillis || clock.monotonicNanos() - h.refreshAtNanos >= 0)) { - startOverdueRefresh(); + startOverdueRefresh(h); } return h.token; } @@ -690,10 +690,13 @@ private void scheduleFetchLocked(long delayNanos, long now) { scheduledTask = scheduler.schedule(() -> runFetch(seq), delayNanos); } - private void startOverdueRefresh() { + private void startOverdueRefresh(Held observed) { lock.lock(); try { - requestFetchNowLocked(clock.monotonicNanos()); + // A scheduled refresh may have replaced the warm reader's snapshot before it acquired the lock. + if (held == observed) { + requestFetchNowLocked(clock.monotonicNanos()); + } } finally { lock.unlock(); } diff --git a/core/src/test/java/io/questdb/client/test/cutlass/auth/RefreshingTokenProviderTest.java b/core/src/test/java/io/questdb/client/test/cutlass/auth/RefreshingTokenProviderTest.java index c0ba14efb..c9ea96027 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/auth/RefreshingTokenProviderTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/auth/RefreshingTokenProviderTest.java @@ -38,6 +38,7 @@ import java.util.List; import java.util.UUID; import java.util.concurrent.CountDownLatch; +import java.util.concurrent.FutureTask; import java.util.concurrent.TimeUnit; import java.util.concurrent.atomic.AtomicBoolean; import java.util.concurrent.atomic.AtomicReference; @@ -675,9 +676,8 @@ public void testSentinelTokenIsNeverRendered() throws Exception { @Test public void testShortLivedTokenIsRefreshedWhileItIsStillHandedOut() { - // section 5.2: a 100 s token is handed out for 40 s. Its refresh runs 30 s after the fetch, while it is still - // handed out, so an idle client always holds a usable token. Scheduled at its half-life alone, the refresh - // ran 45-50 s after the fetch, and every caller after the token left its window waited for a fetch. + // section 5.2: a 100 s token is handed out for 40 s. An immediately returning refresh at +30 s keeps a + // usable token available. Its half-life alone would schedule the refresh at +45-50 s, after the cutoff. FakeClock clock = new FakeClock(T0); ManualScheduler scheduler = new ManualScheduler(clock); ScriptedSource source = new ScriptedSource().then(() -> new ExpiringToken( @@ -700,6 +700,60 @@ public void testShortLivedTokenIsRefreshedWhileItIsStillHandedOut() { } } + @Test(timeout = 30_000) + public void testSlowSuccessfulRefreshCanOutlastTheHandoutWindow() throws Exception { + FakeClock clock = new FakeClock(T0); + ManualScheduler scheduler = new ManualScheduler(clock); + ScriptedSource source = new ScriptedSource() + .thenToken("T1", T0 + 100_000L) + .then(() -> new ExpiringToken("T2", clock.wallClockMillis() + 100_000L)); + try (RefreshingTokenProvider provider = provider(source, clock, scheduler, 0.5).build()) { + scheduler.runNext(); + Assert.assertEquals("T1", provider.getToken().toString()); + Assert.assertEquals(30_000L, scheduler.peek().delayMillis()); + source.setGate(); + FutureTask refresh = new FutureTask<>(scheduler::advanceAndRunNext); + FutureTask token = new FutureTask<>(() -> provider.getToken().toString()); + Thread refresher = start(refresh); + Thread caller = null; + try { + await(() -> source.calls() == 2, 5_000, "the scheduled refresh to enter the source"); + Assert.assertEquals(T0 + 30_000L, clock.wallClockMillis()); + clock.advanceMillis(9_999); + Assert.assertEquals("T1", provider.getToken().toString()); + clock.advanceMillis(1); + Assert.assertFalse("the old token has reached its handout floor", provider.awaitReady(0)); + + caller = start(token); + Thread waitingCaller = caller; + await(() -> waitingCaller.getState() == Thread.State.TIMED_WAITING, + 5_000, "the caller to wait for the in-flight refresh"); + clock.advanceMillis(9_999); + Assert.assertFalse(provider.awaitReady(0)); + Assert.assertFalse("a successful but unfinished fetch cannot supply a token", token.isDone()); + + clock.advanceMillis(1); + source.openGate(); + Assert.assertEquals("T2", token.get(5, TimeUnit.SECONDS)); + refresh.get(5, TimeUnit.SECONDS); + Assert.assertEquals(T0 + 50_000L, provider.getLastSuccessEpochMillis()); + Assert.assertEquals(2, source.calls()); + Assert.assertEquals(0, provider.getConsecutiveFailures()); + Assert.assertNull(provider.getLastFailure()); + } finally { + source.openGate(); + refresher.interrupt(); + refresher.join(5_000); + if (caller != null) { + caller.interrupt(); + caller.join(5_000); + Assert.assertFalse("the token wait must finish", caller.isAlive()); + } + Assert.assertFalse("the refresh must finish", refresher.isAlive()); + } + } + } + @Test public void testSourceFailureTextIsSanitized() { FakeClock clock = new FakeClock(T0); @@ -817,6 +871,65 @@ public void testWarmPathDoesNotFetch() { } } + @Test(timeout = 30_000) + public void testWarmReaderDoesNotRescheduleAfterAConcurrentRefresh() throws Exception { + FakeClock clock = new FakeClock(T0); + ManualScheduler scheduler = new ManualScheduler(clock); + ScriptedSource source = new ScriptedSource() + .thenToken("T1", T0 + HOUR) + .then(() -> new ExpiringToken("T2", clock.wallClockMillis() + HOUR)); + CountDownLatch snapshotRead = new CountDownLatch(1); + CountDownLatch resumeReader = new CountDownLatch(1); + AtomicBoolean isGateArmed = new AtomicBoolean(); + RefreshingTokenProvider.Clock gatedClock = new RefreshingTokenProvider.Clock() { + @Override + public long monotonicNanos() { + return clock.monotonicNanos(); + } + + @Override + public long wallClockMillis() { + if (isGateArmed.compareAndSet(true, false)) { + snapshotRead.countDown(); + try { + Assert.assertTrue("the reader must resume", resumeReader.await(5, TimeUnit.SECONDS)); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new AssertionError(e); + } + } + return clock.wallClockMillis(); + } + }; + try (RefreshingTokenProvider provider = RefreshingTokenProvider.builder(source) + .clock(gatedClock).scheduler(scheduler).random(() -> 0.5).build()) { + scheduler.runNext(); + Assert.assertEquals("T1", provider.getToken().toString()); + clock.advanceMillis(scheduler.peek().delayMillis()); + isGateArmed.set(true); + FutureTask token = new FutureTask<>(() -> provider.getToken().toString()); + Thread reader = start(token); + try { + // Pause after getToken() captures T1, then publish T2 before the reader checks T1's deadline. + Assert.assertTrue("the reader must capture the old token", snapshotRead.await(5, TimeUnit.SECONDS)); + scheduler.runNext(); + Assert.assertEquals("T2", provider.getToken().toString()); + ManualScheduler.Task nextRefresh = scheduler.peek(); + Assert.assertEquals(30 * MIN, nextRefresh.delayMillis()); + resumeReader.countDown(); + Assert.assertEquals("T1", token.get(5, TimeUnit.SECONDS)); + Assert.assertSame("the stale reader must preserve the new refresh schedule", + nextRefresh, scheduler.peek()); + Assert.assertEquals(2, source.calls()); + } finally { + resumeReader.countDown(); + reader.interrupt(); + reader.join(5_000); + Assert.assertFalse("the reader must finish", reader.isAlive()); + } + } + } + @Test public void testWallClockOverdueRefreshStartsInTheBackground() { // section 5.3 step 1: a usable token is returned at once, and when its refresh time has passed - here the diff --git a/design/qwp-token-provider-spec.md b/design/qwp-token-provider-spec.md index d3ade3f7f..3ea9b125f 100644 --- a/design/qwp-token-provider-spec.md +++ b/design/qwp-token-provider-spec.md @@ -4,7 +4,7 @@ **Changes:** -- **v0.7:** a refresh starts at least `p` before the token leaves its hand-out window (§5.2). Before, a token that lived less than about two minutes was refreshed only after it had stopped being handed out, so in every cycle callers waited for a fetch. A token handed out for less than `p` still cannot be refreshed in time; the provider logs that once (§5.2, C1). +- **v0.7:** a refresh starts at least `p` before the token leaves its hand-out window (§5.2), unless the minimum refresh interval forces a later start. Before, a token that lived less than about two minutes was refreshed only after it had stopped being handed out, so in every cycle callers waited for a fetch. A token handed out for less than `p` still cannot be refreshed in time; the provider logs that once (§5.2, C1). A slow successful fetch can also outlast the remaining hand-out window. - **v0.6:** the optional authentication-outage deadline (§8.5, D7) is withdrawn. It made an established sender terminal on a transient outage, which store-and-forward forbids, and the role-reject and transport rounds of a failover counted toward it. An application that prefers a hard failure closes the sender itself, guided by the error channel and the connection health (§8.5). - **v0.5:** a result whose token expires within `handout_floor` of its receipt is a retryable failed fetch whose message names `handout_floor` (§4, C2, C6). Before, it counted as a success that could never be handed out, so every caller waited out `cold_wait` for an error that named no cause. - **v0.4:** `azure_credential` selects one Azure credential deterministically (§7.1), so the library itself reports an unreachable managed-identity endpoint (retryable) apart from a misconfigured credential (permanent) (§7.5, D10). A discovery chain's "no credential available" stays permanent, so a host without a credential still fails fast, and the client warns once that the chain does not ride out an endpoint outage. This resolves the conflict between §4 and §7.5 over an unreachable managed-identity endpoint. @@ -139,15 +139,15 @@ r = min(r, max(e - handout_floor - p, f + p)) ``` - **Proactive refresh:** the provider MUST start a fetch at `r` without waiting for a caller. -- **Hand-out window:** a token is handed out until `e - handout_floor` (§5.1). The last step starts the refresh at least `p` before then, so the fetch and its retries run while the current token is still handed out, but never sooner than `p` after the fetch. With the defaults it changes nothing for a token that lives 3 minutes or more. -- **Effect:** while the source is healthy, even an idle client always holds a usable token, provided a token is handed out for longer than `p`. With the defaults, that is a token that lives longer than 90 s. +- **Hand-out window:** a token is handed out until `e - handout_floor` (§5.1). The last step moves the refresh earlier to leave time for fetching and retries before that cutoff, but never starts it sooner than `f + p`. This minimum spacing can leave less than `p` before the cutoff. With the defaults the last step changes nothing for a token that lives 3 minutes or more. +- **Effect:** continuous availability requires a refresh to complete with a usable token before the current token reaches `e - handout_floor`. A healthy source alone does not guarantee this: even a successful fetch can take longer than the remaining window. During such a gap, callers wait as described in §5.3. - **Short hand-out window:** a token handed out for less than `p` cannot be refreshed while it is still handed out without breaking `min_refresh_interval`, so callers wait for a fetch whenever it stops being usable. The provider SHOULD log a warning once that names `handout_floor`, so that the configuration can be fixed. *Informative examples:* - A 60-minute service-principal token is refreshed at about 27–33 minutes. - A 24-hour managed-identity token is refreshed at about 11–13 hours. -- A 100-second token is handed out for 40 s and refreshed 30 s after its fetch. Without the last step it would be refreshed 45–50 s after its fetch, after it had stopped being handed out. +- A 100-second token is handed out for 40 s, and its refresh starts 30 s after receipt. That leaves only 10 s for the refresh to complete: a successful 20-second fetch leaves a 10-second gap without a usable token. Without the last step the refresh would start 45–50 s after receipt, after the token had stopped being handed out. - If the platform keeps returning the same token, each refresh happens at about half of the remaining lifetime. So a token is fetched only a handful of times before it expires. ### 5.3 Handing out a token: `get_token()` From 0b106bfeecf5ba2f37935ab6541e74cf2a13a073 Mon Sep 17 00:00:00 2001 From: glasstiger Date: Thu, 15 Oct 2026 14:26:25 +0100 Subject: [PATCH 20/40] Keep token retry backoff above zero Clamp the jittered delay to at least one millisecond so accepted one-millisecond backoff settings cannot schedule immediate retries. Preserve longer Retry-After hints and the immediate initial prefetch. Cover repeated failures across jitter samples and Retry-After values with the existing deterministic scheduler, and document the floor. Validation: 36 client tests pass on Java 8 with -Xint and Java 25. --- .../cutlass/auth/RefreshingTokenProvider.java | 3 ++- .../auth/RefreshingTokenProviderTest.java | 27 +++++++++++++++++++ design/qwp-token-provider-spec.md | 2 ++ 3 files changed, 31 insertions(+), 1 deletion(-) diff --git a/core/src/main/java/io/questdb/client/cutlass/auth/RefreshingTokenProvider.java b/core/src/main/java/io/questdb/client/cutlass/auth/RefreshingTokenProvider.java index b2aebcadd..1f620fb15 100644 --- a/core/src/main/java/io/questdb/client/cutlass/auth/RefreshingTokenProvider.java +++ b/core/src/main/java/io/questdb/client/cutlass/auth/RefreshingTokenProvider.java @@ -504,7 +504,8 @@ private long backoffMillis(int failures, long retryAfterMillis) { if (retryAfterMillis >= 0) { delay = Math.max(delay, Math.min(retryAfterMillis, retryAfterMaxMillis)); } - return delay; + // Millisecond rounding must not turn a positive backoff into an immediate retry. + return Math.max(1, delay); } private void cancelScheduledLocked() { diff --git a/core/src/test/java/io/questdb/client/test/cutlass/auth/RefreshingTokenProviderTest.java b/core/src/test/java/io/questdb/client/test/cutlass/auth/RefreshingTokenProviderTest.java index c9ea96027..ace610f33 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/auth/RefreshingTokenProviderTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/auth/RefreshingTokenProviderTest.java @@ -102,6 +102,33 @@ public void testBackoffFollowsTheSpec() { assertBackoffDelays(0.5, new long[]{375, 750, 1_500, 3_000, 6_000, 12_000, 24_000, 45_000, 45_000}); } + @Test + public void testBackoffHasPositiveDelayAtOneMillisecond() { + for (double u : new double[]{0.0, 0.5, Math.nextDown(1.0)}) { + for (long retryAfterMillis : new long[]{TokenUnavailableException.NO_RETRY_AFTER, 0, 2}) { + FakeClock clock = new FakeClock(T0); + ManualScheduler scheduler = new ManualScheduler(clock); + ScriptedSource source = new ScriptedSource().thenThrow( + TokenUnavailableException.retryable("down", retryAfterMillis)); + try (RefreshingTokenProvider provider = provider(source, clock, scheduler, u) + .backoffInitialMillis(1) + .backoffMaxMillis(1) + .build()) { + Assert.assertEquals("the initial prefetch is still immediate", 0, scheduler.peek().delayNanos); + final long expectedDelayNanos = TimeUnit.MILLISECONDS.toNanos(retryAfterMillis == 2 ? 2 : 1); + for (int failure = 1; failure <= 64; failure++) { + scheduler.advanceAndRunNext(); + Assert.assertEquals("backoff after failure " + failure + " [u=" + u + + ", retryAfterMillis=" + retryAfterMillis + ']', + expectedDelayNanos, scheduler.peek().delayNanos); + Assert.assertEquals(failure, provider.getConsecutiveFailures()); + } + Assert.assertEquals(64, source.calls()); + } + } + } + } + @Test public void testCloseFailsGetTokenPermanentlyAndStopsTheRefresher() { ScriptedSource source = new ScriptedSource().thenToken("TOKEN", System.currentTimeMillis() + HOUR); diff --git a/design/qwp-token-provider-spec.md b/design/qwp-token-provider-spec.md index 3ea9b125f..8e501aa61 100644 --- a/design/qwp-token-provider-spec.md +++ b/design/qwp-token-provider-spec.md @@ -175,6 +175,8 @@ if retry_after is present: delay = max(delay, min(retry_after, retry_after_max)) ``` +The delay MUST remain positive after rounding. Implementations using whole milliseconds MUST impose a 1 ms minimum. + A successful fetch resets `n`. **While fetches are failing:** From 40fe8d8d271207a68e139581b8db174bbd73eeab Mon Sep 17 00:00:00 2001 From: glasstiger Date: Thu, 15 Oct 2026 14:59:58 +0100 Subject: [PATCH 21/40] Test close during credential refresh Cover sender shutdown while a rejected token blocks the rejection callback or the second credential pull. Assert that close interrupts each wait and completes normally. Cover provider shutdown with two cold callers and an unfinished fetch. Keep fetch completion separate from close notification so the test detects a missing signal or waking only one waiter. Run 89 related tests on JDK 25 and Java 8 with -Xint. Confirm the new tests fail against cancellation and notification mutations in an isolated worktree. --- .../auth/RefreshingTokenProviderTest.java | 56 +++++++++++++ .../WebSocketDynamicCredentialTest.java | 80 +++++++++++++++++++ 2 files changed, 136 insertions(+) diff --git a/core/src/test/java/io/questdb/client/test/cutlass/auth/RefreshingTokenProviderTest.java b/core/src/test/java/io/questdb/client/test/cutlass/auth/RefreshingTokenProviderTest.java index ace610f33..ee180313a 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/auth/RefreshingTokenProviderTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/auth/RefreshingTokenProviderTest.java @@ -38,10 +38,12 @@ import java.util.List; import java.util.UUID; import java.util.concurrent.CountDownLatch; +import java.util.concurrent.ExecutionException; import java.util.concurrent.FutureTask; import java.util.concurrent.TimeUnit; import java.util.concurrent.atomic.AtomicBoolean; import java.util.concurrent.atomic.AtomicReference; +import java.util.concurrent.locks.ReentrantLock; import static io.questdb.client.test.cutlass.auth.TokenTestKit.await; @@ -153,6 +155,60 @@ public void testCloseFailsGetTokenPermanentlyAndStopsTheRefresher() { provider.onTokenRejected("TOKEN", 401); // no-op, no throw } + @Test(timeout = 30_000) + public void testCloseWakesEveryColdCallerBeforeTheFetchCompletes() throws Exception { + ReentrantLock fetchGate = new ReentrantLock(); + CountDownLatch fetchEntered = new CountDownLatch(1); + CountDownLatch fetchCompleted = new CountDownLatch(1); + AtomicReference fetchThread = new AtomicReference<>(); + List callers = new ArrayList<>(); + List> results = new ArrayList<>(); + fetchGate.lock(); + try (RefreshingTokenProvider provider = RefreshingTokenProvider.builder(() -> { + fetchThread.set(Thread.currentThread()); + fetchEntered.countDown(); + // Keep this finite source phase non-interruptible so shutdown cannot release callers through fetch completion. + fetchGate.lock(); + try { + return new ExpiringToken("TOKEN", System.currentTimeMillis() + HOUR); + } finally { + fetchGate.unlock(); + fetchCompleted.countDown(); + } + }).coldWaitMillis(60_000).build()) { + Assert.assertTrue("the prefetch must start", fetchEntered.await(5, TimeUnit.SECONDS)); + for (int i = 0; i < 2; i++) { + FutureTask result = new FutureTask<>(provider::getToken); + results.add(result); + callers.add(start(result)); + } + await(() -> allParked(callers), 5_000, "both cold callers to wait"); + + provider.close(); + for (FutureTask result : results) { + ExecutionException error = Assert.assertThrows(ExecutionException.class, + () -> result.get(5, TimeUnit.SECONDS)); + Assert.assertTrue(error.getCause() instanceof TokenUnavailableException); + TokenUnavailableException unavailable = (TokenUnavailableException) error.getCause(); + Assert.assertFalse("close must report a permanent failure", unavailable.isRetryable()); + Assert.assertTrue(unavailable.getMessage(), unavailable.getMessage().contains("is closed")); + } + Assert.assertEquals("close must wake every caller before the source completes", 1, fetchCompleted.getCount()); + } finally { + fetchGate.unlock(); + for (Thread caller : callers) { + caller.interrupt(); + caller.join(5_000); + Assert.assertFalse("the cold caller must exit", caller.isAlive()); + } + Thread refresher = fetchThread.get(); + if (refresher != null) { + refresher.join(5_000); + Assert.assertFalse("the refresher must exit after releasing the source", refresher.isAlive()); + } + } + } + @Test(timeout = 30_000) public void testColdBurstAfterTheWallClockJumpsCausesOneFetch() throws Exception { // C3, second shape: a token is held, but the wall clock jumped past its expiry (a host that slept diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/WebSocketDynamicCredentialTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/WebSocketDynamicCredentialTest.java index dbff5ccb7..1d04e21f9 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/WebSocketDynamicCredentialTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/WebSocketDynamicCredentialTest.java @@ -24,6 +24,7 @@ package io.questdb.client.test.cutlass.qwp.client; +import io.questdb.client.HttpTokenProvider; import io.questdb.client.Sender; import io.questdb.client.SenderConnectionEvent; import io.questdb.client.SenderError; @@ -53,7 +54,10 @@ import java.util.List; import java.util.Set; import java.util.concurrent.CopyOnWriteArrayList; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.FutureTask; import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicBoolean; import java.util.concurrent.atomic.AtomicInteger; import java.util.concurrent.atomic.AtomicLong; @@ -154,6 +158,16 @@ public void testChallengePlain401Retries() throws Exception { assertChallengeRetries(null); } + @Test(timeout = 30_000) + public void testCloseInterruptsThePost401CredentialPull() throws Exception { + assertCloseInterruptsPost401Wait(false); + } + + @Test(timeout = 30_000) + public void testCloseInterruptsThePost401RejectionCallback() throws Exception { + assertCloseInterruptsPost401Wait(true); + } + @Test(timeout = 60_000) public void testFailoverAcrossARotationSendsTheNewTokenAndReplaysOnce() throws Exception { // C14: connected to A with T1, the token rotates, A dies. B receives the NEW token, and the frames A @@ -879,6 +893,72 @@ private void assertChallengeRetries(String challenge) throws Exception { }); } + private void assertCloseInterruptsPost401Wait(boolean isBlockingCallback) throws Exception { + assertMemoryLeak(() -> { + CountDownLatch waitEntered = new CountDownLatch(1); + CountDownLatch waitInterrupted = new CountDownLatch(1); + CountDownLatch releaseWait = new CountDownLatch(1); + AtomicBoolean hasRejected = new AtomicBoolean(); + Runnable waitForInterrupt = () -> { + waitEntered.countDown(); + try { + releaseWait.await(45, TimeUnit.SECONDS); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + waitInterrupted.countDown(); + } + }; + HttpTokenProvider provider = new HttpTokenProvider() { + @Override + public CharSequence getToken() { + if (hasRejected.get() && !isBlockingCallback) { + waitForInterrupt.run(); + } + return "T1"; + } + + @Override + public void onTokenRejected(CharSequence token, int httpStatus) { + hasRejected.set(true); + if (isBlockingCallback) { + waitForInterrupt.run(); + } + } + }; + try (TestWebSocketServer server = startServer(new AckHandler()); + Sender sender = Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + server.getPort()) + .closeFlushTimeoutMillis(0) + .httpTokenProvider(provider) + .build()) { + FutureTask close = new FutureTask<>(() -> { + sender.close(); + return null; + }); + Thread closer = new Thread(close, "post-401-sender-close"); + try { + Assert.assertEquals("Bearer T1", server.pollAuthorizationHeader(5, TimeUnit.SECONDS)); + server.setAuthorizationValidator(h -> 401); + server.dropAllConnections(); + Assert.assertTrue("the post-401 wait must start", waitEntered.await(5, TimeUnit.SECONDS)); + Assert.assertEquals("the reconnect must reach the server before blocking", 1, server.authRejectCount()); + Assert.assertEquals("Bearer T1", server.pollAuthorizationHeader(5, TimeUnit.SECONDS)); + + closer.start(); + Assert.assertTrue("close must interrupt the " + (isBlockingCallback ? "rejection callback" : "second pull"), + waitInterrupted.await(5, TimeUnit.SECONDS)); + close.get(5, TimeUnit.SECONDS); + } finally { + releaseWait.countDown(); + if (closer.getState() != Thread.State.NEW) { + closer.join(10_000); + Assert.assertFalse("the close thread must exit", closer.isAlive()); + } + } + } + }); + } + static byte[] buildAck(long seq) { byte[] buf = new byte[1 + 8 + 2]; ByteBuffer bb = ByteBuffer.wrap(buf).order(ByteOrder.LITTLE_ENDIAN); From 086f1a70a10a0ac87b7174d83d7e00fd14636944 Mon Sep 17 00:00:00 2001 From: glasstiger Date: Thu, 15 Oct 2026 15:39:44 +0100 Subject: [PATCH 22/40] Honor Azure MSAL Retry-After hints Preserve Retry-After headers from MSAL service exceptions, including\nwrapped failures. Service-principal credentials surface throttling through\nMSAL rather than HttpResponseException, so the adapter previously lost\nthe final response's delay after the SDK exhausted its retries. Share delta-seconds parsing between Azure HTTP and MSAL response headers\nwithout changing permanent-error classification or retaining SDK causes. Add regression tests through the real ClientSecretCredential and its\ndefault retry policy for throttling, server errors, wrapped exceptions,\ncase-insensitive headers, and missing or malformed hints. Verify MSAL\nfailures without response headers still use the provider's backoff. Validation: all 30 Azure tests and 35 refreshing-provider tests pass on\nnative arm64 JDK 8 and JDK 25. The new throttling tests fail before the\nproduction fix and pass with it. --- .../client/azure/AzureTokenSource.java | 40 +++- ...zureClientSecretCredentialLibraryTest.java | 181 ++++++++++++++++++ .../azure/test/AzureTokenSourceTest.java | 2 + 3 files changed, 215 insertions(+), 8 deletions(-) create mode 100644 azure/src/test/java/io/questdb/client/azure/test/AzureClientSecretCredentialLibraryTest.java diff --git a/azure/src/main/java/io/questdb/client/azure/AzureTokenSource.java b/azure/src/main/java/io/questdb/client/azure/AzureTokenSource.java index e74fa36b1..676401ff1 100644 --- a/azure/src/main/java/io/questdb/client/azure/AzureTokenSource.java +++ b/azure/src/main/java/io/questdb/client/azure/AzureTokenSource.java @@ -31,6 +31,7 @@ import com.azure.core.http.HttpResponse; import com.azure.identity.ChainedTokenCredential; import com.azure.identity.CredentialUnavailableException; +import com.microsoft.aad.msal4j.MsalServiceException; import io.questdb.client.cutlass.auth.ExpiringToken; import io.questdb.client.cutlass.auth.TokenSource; import io.questdb.client.cutlass.auth.TokenUnavailableException; @@ -40,6 +41,8 @@ import java.net.UnknownHostException; import java.time.Duration; import java.time.OffsetDateTime; +import java.util.List; +import java.util.Map; import java.util.concurrent.TimeUnit; /** @@ -143,6 +146,7 @@ public AzureTokenSource(TokenCredential credential, String resource, Duration ti */ static TokenUnavailableException classify(Throwable failure, String scope, boolean chain) { final String description = describe(failure); + long retryAfterMillis = TokenUnavailableException.NO_RETRY_AFTER; Throwable t = failure; for (int depth = 0; t != null && depth < MAX_CAUSE_DEPTH; depth++, t = t.getCause()) { if (t instanceof CredentialUnavailableException) { @@ -162,6 +166,10 @@ static TokenUnavailableException classify(Throwable failure, String scope, boole retryAfterMillis(response)); } } + if (t instanceof MsalServiceException && retryAfterMillis == TokenUnavailableException.NO_RETRY_AFTER) { + // Service-principal credentials expose response headers through MSAL, not HttpResponseException. + retryAfterMillis = retryAfterMillis(((MsalServiceException) t).headers()); + } final String message = t.getMessage(); if (message != null) { for (String code : PERMANENT_AADSTS) { @@ -178,7 +186,7 @@ static TokenUnavailableException classify(Throwable failure, String scope, boole } } return TokenUnavailableException.retryable("the token request for " + scope + " failed: " - + withNetworkCause(failure, description)); + + withNetworkCause(failure, description), retryAfterMillis); } private static String describe(Throwable t) { @@ -199,13 +207,7 @@ private static Throwable networkCause(Throwable failure) { return null; } - private static long retryAfterMillis(HttpResponse response) { - final String value; - try { - value = response.getHeaderValue("Retry-After"); - } catch (RuntimeException e) { - return TokenUnavailableException.NO_RETRY_AFTER; - } + private static long parseRetryAfterMillis(String value) { if (value == null) { return TokenUnavailableException.NO_RETRY_AFTER; } @@ -227,6 +229,28 @@ private static long refreshAtMillis(AccessToken token) { } } + private static long retryAfterMillis(HttpResponse response) { + try { + return parseRetryAfterMillis(response.getHeaderValue("Retry-After")); + } catch (RuntimeException e) { + return TokenUnavailableException.NO_RETRY_AFTER; + } + } + + private static long retryAfterMillis(Map> headers) { + if (headers != null) { + for (Map.Entry> header : headers.entrySet()) { + if ("Retry-After".equalsIgnoreCase(header.getKey())) { + final List values = header.getValue(); + return values == null || values.isEmpty() + ? TokenUnavailableException.NO_RETRY_AFTER + : parseRetryAfterMillis(values.get(0)); + } + } + } + return TokenUnavailableException.NO_RETRY_AFTER; + } + // Leads with the socket-level cause, unless the description already carries it: error text is capped, and a // library's own message is often generic ("see inner exception"). private static String withNetworkCause(Throwable failure, String description) { diff --git a/azure/src/test/java/io/questdb/client/azure/test/AzureClientSecretCredentialLibraryTest.java b/azure/src/test/java/io/questdb/client/azure/test/AzureClientSecretCredentialLibraryTest.java new file mode 100644 index 000000000..97c7c950d --- /dev/null +++ b/azure/src/test/java/io/questdb/client/azure/test/AzureClientSecretCredentialLibraryTest.java @@ -0,0 +1,181 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.azure.test; + +import com.azure.core.credential.TokenCredential; +import com.azure.core.exception.ClientAuthenticationException; +import com.azure.core.http.HttpClient; +import com.azure.core.http.HttpHeaders; +import com.azure.core.http.HttpMethod; +import com.azure.core.http.HttpRequest; +import com.azure.core.http.HttpResponse; +import com.azure.identity.ClientSecretCredentialBuilder; +import io.questdb.client.azure.AzureTokenSource; +import io.questdb.client.cutlass.auth.TokenUnavailableException; +import org.junit.Assert; +import org.junit.Test; +import reactor.core.publisher.Flux; +import reactor.core.publisher.Mono; + +import java.nio.ByteBuffer; +import java.nio.charset.Charset; +import java.nio.charset.StandardCharsets; +import java.util.Arrays; +import java.util.concurrent.atomic.AtomicInteger; + +/** + * Exercises the pinned Azure Identity/MSAL service-principal path with a controlled HTTP transport and the + * SDK's default retry policy. No request reaches a real identity endpoint. + */ +public class AzureClientSecretCredentialLibraryTest { + private static final String AUTHORITY = "https://login.microsoftonline.com/" + "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee"; + private static final String CLIENT_SECRET = "not-a-real-secret"; + + @Test(timeout = 30_000) + public void testFinalRetryAfterSurvivesSdkRetries() { + assertRetryAfter(new TokenEndpoint(429, "7"), false, 7_000); + } + + @Test(timeout = 30_000) + public void testMissingOrMalformedRetryAfterUsesProviderBackoff() { + for (String retryAfter : Arrays.asList(null, "", "not-a-delay", "Wed, 21 Oct 2015 07:28:00 GMT", "-1")) { + assertRetryAfter(new TokenEndpoint(429, retryAfter), false, TokenUnavailableException.NO_RETRY_AFTER); + } + } + + @Test(timeout = 30_000) + public void testServerFailureRetainsRetryAfter() { + assertRetryAfter(new TokenEndpoint(503, "7"), false, 7_000); + } + + @Test(timeout = 30_000) + public void testWrappedMsalFailureRetainsRetryAfter() { + assertRetryAfter(new TokenEndpoint(429, " 7 "), true, 7_000); + } + + private static void assertRetryAfter(TokenEndpoint endpoint, boolean isWrapped, long expectedRetryAfterMillis) { + TokenCredential credential = new ClientSecretCredentialBuilder() + .tenantId("aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee") + .clientId("11111111-2222-3333-4444-555555555555") + .clientSecret(CLIENT_SECRET) + .disableInstanceDiscovery() + .httpClient(endpoint) + .build(); + if (isWrapped) { + TokenCredential delegate = credential; + credential = context -> delegate.getToken(context).onErrorMap( + failure -> new ClientAuthenticationException("authentication failed", null, failure)); + } + try { + new AzureTokenSource(credential, "api://qdb").fetchToken(); + Assert.fail("expected the token endpoint to reject the request"); + } catch (TokenUnavailableException e) { + Assert.assertEquals("the SDK must exhaust its default three retries", 4, endpoint.tokenRequests.get()); + Assert.assertTrue(e.getMessage(), e.isRetryable()); + Assert.assertEquals(e.getMessage(), expectedRetryAfterMillis, e.getRetryAfterMillis()); + Assert.assertNull("never retain an SDK exception that references the raw request", e.getCause()); + Assert.assertFalse(e.getMessage(), e.getMessage().contains(CLIENT_SECRET)); + } + } + + private static final class StubResponse extends HttpResponse { + private final byte[] body; + private final HttpHeaders headers; + private final int status; + + private StubResponse(HttpRequest request, int status, String body, String retryAfter) { + super(request); + this.body = body.getBytes(StandardCharsets.UTF_8); + this.headers = new HttpHeaders().set("Content-Type", "application/json"); + this.status = status; + if (retryAfter != null) { + headers.set("rEtRy-AfTeR", retryAfter); + } + } + + @Override + public Flux getBody() { + return Flux.just(ByteBuffer.wrap(body)); + } + + @Override + public Mono getBodyAsByteArray() { + return Mono.just(body); + } + + @Override + public Mono getBodyAsString() { + return getBodyAsString(StandardCharsets.UTF_8); + } + + @Override + public Mono getBodyAsString(Charset charset) { + return Mono.just(new String(body, charset)); + } + + @Override + public String getHeaderValue(String name) { + return headers.getValue(name); + } + + @Override + public HttpHeaders getHeaders() { + return headers; + } + + @Override + public int getStatusCode() { + return status; + } + } + + private static final class TokenEndpoint implements HttpClient { + private final String retryAfter; + private final int status; + private final AtomicInteger tokenRequests = new AtomicInteger(); + + private TokenEndpoint(int status, String retryAfter) { + this.retryAfter = retryAfter; + this.status = status; + } + + @Override + public Mono send(HttpRequest request) { + if (request.getHttpMethod() == HttpMethod.GET) { + return Mono.just(new StubResponse(request, 200, + "{\"authorization_endpoint\":\"" + AUTHORITY + "/oauth2/v2.0/authorize\"," + + "\"token_endpoint\":\"" + AUTHORITY + "/oauth2/v2.0/token\"," + + "\"issuer\":\"" + AUTHORITY + "/v2.0\"," + + "\"jwks_uri\":\"" + AUTHORITY + "/discovery/v2.0/keys\"}", null)); + } + Assert.assertEquals(HttpMethod.POST, request.getHttpMethod()); + // Keep the SDK's first three retries immediate; only its final response supplies the provider's hint. + String delay = tokenRequests.incrementAndGet() <= 3 ? "0" : retryAfter; + return Mono.just(new StubResponse(request, status, + "{\"error\":\"temporarily_unavailable\"," + + "\"error_description\":\"The endpoint is throttling this application\"}", delay)); + } + } +} diff --git a/azure/src/test/java/io/questdb/client/azure/test/AzureTokenSourceTest.java b/azure/src/test/java/io/questdb/client/azure/test/AzureTokenSourceTest.java index 5d2193b3c..b73c10776 100644 --- a/azure/src/test/java/io/questdb/client/azure/test/AzureTokenSourceTest.java +++ b/azure/src/test/java/io/questdb/client/azure/test/AzureTokenSourceTest.java @@ -217,6 +217,8 @@ public void testOtherFailuresAreRetryable() { assertRetryable(new HttpResponseException("unavailable", new FakeResponse(503, null)), "HTTP 503", -1); assertRetryable(new HttpResponseException("gone", new FakeResponse(410, "Wed, 21 Oct 2015 07:28:00 GMT")), "HTTP 410", -1); assertRetryable(new IllegalStateException("connection reset"), "connection reset", -1); + assertRetryable(new MsalServiceException("temporarily unavailable", "temporarily_unavailable"), + "temporarily unavailable", -1); assertRetryable(new ClientAuthenticationException("ManagedIdentityCredential: IMDS endpoint timed out", null, (Object) null), "timed out", -1); } From 25543357aaf76ca86e883ac949b3f38e11332e06 Mon Sep 17 00:00:00 2001 From: glasstiger Date: Thu, 15 Oct 2026 16:39:12 +0100 Subject: [PATCH 23/40] Test superseded token refresh callbacks Replay a canceled scheduled callback before its forced replacement using the existing fake clock and manual scheduler. Assert that the callback neither fetches a token nor replaces the pending task, and that the forced refresh publishes the new token exactly once. All 36 provider tests pass on JDK 25 and on JDK 8 with -Xint. Removing only the scheduled-generation guard makes the new test fail on an unexpected extra fetch while the other 35 tests still pass. --- .../auth/RefreshingTokenProviderTest.java | 34 +++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/core/src/test/java/io/questdb/client/test/cutlass/auth/RefreshingTokenProviderTest.java b/core/src/test/java/io/questdb/client/test/cutlass/auth/RefreshingTokenProviderTest.java index ee180313a..65c256c1a 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/auth/RefreshingTokenProviderTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/auth/RefreshingTokenProviderTest.java @@ -881,6 +881,40 @@ public void testSuccessResetsTheBackoff() { } } + @Test + public void testSupersededScheduledCallbackDoesNotFetchOrReplaceTheForcedRefresh() { + FakeClock clock = new FakeClock(T0); + ManualScheduler scheduler = new ManualScheduler(clock); + ScriptedSource source = new ScriptedSource() + .thenToken("T1", T0 + HOUR) + .thenToken("T2", T0 + 2 * HOUR); + try (RefreshingTokenProvider provider = provider(source, clock, scheduler, 0.5) + .forcedWaitMillis(0) + .build()) { + scheduler.runNext(); + ManualScheduler.Task superseded = scheduler.peek(); + Assert.assertNotNull(superseded); + + provider.onTokenRejected("T1", 401); + ManualScheduler.Task forcedRefresh = scheduler.peek(); + Assert.assertNotNull(forcedRefresh); + Assert.assertNotSame(superseded, forcedRefresh); + Assert.assertTrue(superseded.cancelled); + Assert.assertEquals(0, forcedRefresh.delayNanos); + + // Replay a callback that started before cancellation and resumed after its replacement. + superseded.runnable.run(); + Assert.assertEquals("a superseded callback must not fetch", 1, source.calls()); + Assert.assertSame("the superseded callback must preserve its replacement", + forcedRefresh, scheduler.peek()); + Assert.assertEquals("T1", provider.getToken().toString()); + + scheduler.runNext(); + Assert.assertEquals("the forced refresh must fetch exactly once", 2, source.calls()); + Assert.assertEquals("T2", provider.getToken().toString()); + } + } + @Test public void testTokenHandedOutTooBrieflyToRefreshAheadIsReportedOnce() { // section 5.2: a 75 s token is handed out for 15 s, less than min_refresh_interval, so it cannot be refreshed From 97cfe2b5c0bec805c0477fc7f695a4c83543dacb Mon Sep 17 00:00:00 2001 From: glasstiger Date: Thu, 15 Oct 2026 18:13:16 +0100 Subject: [PATCH 24/40] Report health loss after catch-up failure Restore RECONNECTING health when a replacement connection loses its\ntransport during dictionary catch-up. Publish the loss before backoff\nor another credential pull so health polling cannot report CONNECTED\nwhile the sender waits to reconnect. Keep the retry policy unchanged.\n\nAdd a real-socket regression test that populates the symbol dictionary\nthrough the sender API, fails the reconnect catch-up send, and holds\nthe following credential pull. Check the outage state, buffered delivery,\nand recovery. The test fails before the production fix.\n\nValidation: 178 targeted Maven tests pass on native ARM64 JDK 25,\nincluding health, cursor send-loop, credential cancellation, dynamic\ncredential, and JVM-error cleanup tests. git diff --check also passes. --- .../sf/cursor/CursorWebSocketSendLoop.java | 6 ++ .../qwp/client/ConnectionHealthTest.java | 97 +++++++++++++++++++ 2 files changed, 103 insertions(+) diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoop.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoop.java index c10d6f475..024b7f08f 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoop.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/CursorWebSocketSendLoop.java @@ -2258,6 +2258,12 @@ private void connectWalk(Throwable initial, String phase, long paceFirstAttemptM recordFatal(e); throw (Error) e; } + // The factory publishes CONNECTED after upgrade, but swapClient can still lose the transport + // during dictionary catch-up. Publish that loss before backoff or another credential pull. + io.questdb.client.cutlass.qwp.client.QwpConnectionHealthTracker tracker = healthTracker; + if (tracker != null) { + tracker.connectionLost(); + } if (!isCatchUpCapGap(e)) { resetCatchUpCapGapEpisode(); } diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/ConnectionHealthTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/ConnectionHealthTest.java index cb752e918..71d49a03d 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/ConnectionHealthTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/ConnectionHealthTest.java @@ -25,15 +25,21 @@ package io.questdb.client.test.cutlass.qwp.client; import io.questdb.client.ConnectionHealth; +import io.questdb.client.DefaultHttpClientConfiguration; import io.questdb.client.QuestDB; import io.questdb.client.Sender; import io.questdb.client.cutlass.auth.TokenUnavailableException; +import io.questdb.client.cutlass.http.client.WebSocketClientFactory; import io.questdb.client.cutlass.qwp.client.QwpColumnBatch; import io.questdb.client.cutlass.qwp.client.QwpColumnBatchHandler; import io.questdb.client.cutlass.qwp.client.QwpEgressMsgKind; import io.questdb.client.cutlass.qwp.client.QwpQueryClient; +import io.questdb.client.cutlass.qwp.client.QwpWebSocketSender; import io.questdb.client.cutlass.qwp.client.WebSocketResponse; import io.questdb.client.cutlass.qwp.protocol.QwpConstants; +import io.questdb.client.network.NetworkFacade; +import io.questdb.client.network.NetworkFacadeImpl; +import io.questdb.client.std.Unsafe; import io.questdb.client.test.cutlass.qwp.client.WebSocketDynamicCredentialTest.ErrorCollector; import io.questdb.client.test.cutlass.qwp.websocket.TestWebSocketServer; import org.junit.Assert; @@ -349,6 +355,79 @@ public void testSenderHealthIsReconnectingFromTheMomentItLosesItsConnection() th }); } + @Test(timeout = 60_000) + public void testSenderHealthIsReconnectingWhenDictionaryCatchUpLosesItsConnection() throws Exception { + assertMemoryLeak(() -> { + CatchUpFailingNetworkFacade networkFacade = new CatchUpFailingNetworkFacade(); + DefaultHttpClientConfiguration configuration = new DefaultHttpClientConfiguration() { + @Override + public NetworkFacade getNetworkFacade() { + return networkFacade; + } + }; + CountDownLatch pullHeld = new CountDownLatch(1); + CountDownLatch release = new CountDownLatch(1); + try (TestWebSocketServer server = startServer(new WebSocketDynamicCredentialTest.AckHandler()); + Sender sender = Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + server.getPort()) + .symbolDictReset(false) + .reconnectInitialBackoffMillis(10) + .reconnectMaxBackoffMillis(20) + .closeFlushTimeoutMillis(0) + .httpTokenProvider(() -> { + if (networkFacade.hasFailed.get()) { + pullHeld.countDown(); + try { + if (!release.await(30, TimeUnit.SECONDS)) { + throw TokenUnavailableException.retryable("credential pull timed out"); + } + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw TokenUnavailableException.retryable("credential pull interrupted"); + } + } + return SENTINEL; + }) + .build()) { + try { + // Populate the dictionary through the producer API, so reconnect must send catch-up. + sender.table("t").symbol("s", "sym0").longColumn("v", 1).atNow(); + long first = sender.flushAndGetSequence(); + Assert.assertTrue("the initial row must reach the server", sender.awaitAckedFsn(first, 10_000)); + Assert.assertTrue(((QwpWebSocketSender) sender).isDeltaDictEnabledForTest()); + Assert.assertEquals(ConnectionHealth.State.CONNECTED, sender.health().getState()); + + ((QwpWebSocketSender) sender).setClientFactoryOverride( + () -> WebSocketClientFactory.newPlainTextInstance(configuration)); + server.dropAllConnections(); + Assert.assertTrue("the next credential pull follows the catch-up failure", + pullHeld.await(10, TimeUnit.SECONDS)); + Assert.assertTrue("the replacement must complete a real upgrade", server.upgradeRequestCount() >= 2); + Assert.assertEquals("shut down the real catch-up transport", 0, networkFacade.shutdownResult); + Assert.assertTrue("the production socket send must fail", networkFacade.failedSendResult < 0); + + sender.table("t").symbol("s", "sym1").longColumn("v", 2).atNow(); + long queued = sender.flushAndGetSequence(); + Assert.assertFalse("no delivery while the credential pull is held", sender.awaitAckedFsn(queued, 100)); + ConnectionHealth h = sender.health(); + Assert.assertEquals(h.toString(), ConnectionHealth.State.RECONNECTING, h.getState()); + Assert.assertNotEquals("catch-up transport loss must start an outage: " + h, + ConnectionHealth.NONE, h.getOutageSinceEpochMillis()); + Assert.assertEquals("the next connect round has not failed", 0, h.getFailedRounds()); + + release.countDown(); + Assert.assertTrue("the buffered row must reach the server after recovery", + sender.awaitAckedFsn(queued, 10_000)); + await(() -> sender.health().getState() == ConnectionHealth.State.CONNECTED, 10_000, "reconnected"); + Assert.assertEquals(ConnectionHealth.NONE, sender.health().getOutageSinceEpochMillis()); + Assert.assertEquals(0, sender.health().getFailedRounds()); + } finally { + release.countDown(); + } + } + }); + } + @Test(timeout = 60_000) public void testSenderHealthMovesThroughEveryState() throws Exception { // C21: connecting -> connected -> reconnecting (with 401 and credential-unavailable failures) -> @@ -471,6 +550,24 @@ private static TestWebSocketServer startServer(TestWebSocketServer.WebSocketServ return server; } + private static final class CatchUpFailingNetworkFacade extends NetworkFacadeImpl { + private final AtomicBoolean hasFailed = new AtomicBoolean(); + private int failedSendResult; + private int shutdownResult; + + @Override + public int sendRaw(int fd, long buffer, int bufferLen) { + // Let the HTTP upgrade pass, then break the first binary send on the replacement connection. + if (bufferLen > 0 && Unsafe.getUnsafe().getByte(buffer) == (byte) 0x82 + && hasFailed.compareAndSet(false, true)) { + shutdownResult = shutdown(fd); + failedSendResult = super.sendRaw(fd, buffer, bufferLen); + return failedSendResult; + } + return super.sendRaw(fd, buffer, bufferLen); + } + } + // Answers like ExecDoneHandler but, while drop is set, drops the connection a query arrives on. private static final class DropQueryHandler implements TestWebSocketServer.WebSocketServerHandler { final AtomicBoolean drop = new AtomicBoolean(); From 65bc2ff7c56bf0824aeebe681d6663d695a1607e Mon Sep 17 00:00:00 2001 From: glasstiger Date: Thu, 15 Oct 2026 23:09:42 +0100 Subject: [PATCH 25/40] Fix Azure module discovery and diagnostics Package Java 11 module descriptors in both Java 8 release jars so service binding discovers the Azure provider and resolves its SDK dependencies without extra module roots. Keep root classes compatible with Java 8 and inspect the versioned descriptor in the CI smoke job. Stop forwarding arbitrary SDK exception text into token-source errors and cached failures. Retain exception types, HTTP status, known Entra configuration codes, socket details, and retry hints without retaining SDK causes or token-endpoint response bodies. Add packaged-module integration tests for sender and query configuration and real-SDK guards for malformed responses and OAuth error text. Verify that removing either module metadata component fails the module smoke and that both diagnostic guards fail against the original source. Validate targeted Maven verify and Javadoc builds on native arm64 JDK 8 and JDK 25. Execute the JDK 8 jars on JDK 11 and JDK 25 and pass 57 additional configuration, dynamic-auth, health, and shutdown tests. --- .github/workflows/ci.yml | 6 +- azure/pom.xml | 116 +++++++++++++- .../azure/AzureTokenProviderFactory.java | 5 +- .../client/azure/AzureTokenSource.java | 19 +-- azure/src/main/java/module-info.java | 36 +++++ ...zureClientSecretCredentialLibraryTest.java | 78 ++++++++-- .../test/AzureCredentialSelectionTest.java | 4 +- .../client/azure/test/AzureModulePathIT.java | 143 ++++++++++++++++++ .../azure/test/AzureTokenSourceTest.java | 20 ++- .../src/test/resources/module-path/Main.java | 49 ++++++ .../resources/module-path/module-info.java | 19 +++ core/pom.xml | 23 ++- .../client/test/std/JarPackagingIT.java | 4 + 13 files changed, 482 insertions(+), 40 deletions(-) create mode 100644 azure/src/main/java/module-info.java create mode 100644 azure/src/test/java/io/questdb/client/azure/test/AzureModulePathIT.java create mode 100644 azure/src/test/resources/module-path/Main.java create mode 100644 azure/src/test/resources/module-path/module-info.java diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 3538ed01b..dbee07377 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -86,8 +86,8 @@ jobs: if-no-files-found: error # Takes the JDK 8-built jar (the shipped artifact) from build-jdk8 and runs it - # on every newer JDK the client must work on, asserting (a) the automatic - # module name is io.questdb.client and (b) slow-path double formatting + # on every newer JDK the client must work on, asserting (a) the versioned + # module descriptor names io.questdb.client and (b) slow-path double formatting # resolves the META-INF/versions/11 bridge. These are only observable in the # PACKAGED jar on a modern JDK, and each regression here was invisible to the # other jobs until the jar was EXECUTED on the right JDK/flags: @@ -134,7 +134,7 @@ jobs: run: | jar_file=(client-jar/questdb-client-*.jar) jar_file="${jar_file[0]}" - jar --describe-module --file "$jar_file" | tee module.txt + jar --describe-module --file "$jar_file" --release 11 | tee module.txt grep -q '^io\.questdb\.client@' module.txt javac -cp "$jar_file" -d smoke-classes \ core/src/test/java/io/questdb/client/test/std/DoubleFormatSmoke.java diff --git a/azure/pom.xml b/azure/pom.xml index bbeb51e55..34b3eafa2 100644 --- a/azure/pom.xml +++ b/azure/pom.xml @@ -25,8 +25,9 @@ @@ -129,16 +130,66 @@ maven-surefire-plugin 3.5.3 - + false ${test.include} ${test.exclude} + **/*IT.class + + org.apache.maven.plugins + maven-dependency-plugin + 3.8.1 + + + module-path-runtime-dependencies + pre-integration-test + + copy-dependencies + + + runtime + ${project.build.directory}/module-path-dependencies + + + + + + org.apache.maven.plugins + maven-failsafe-plugin + 3.5.3 + + + + integration-test + verify + + + + + false + + ${project.build.directory}/${project.build.finalName}.jar + ${project.basedir}/src/test/resources/module-path + ${project.build.directory}/module-path-dependencies + + + + + org.apache.maven.plugins + maven-javadoc-plugin + 3.5.0 + + + module-info.java + + + org.apache.maven.plugins maven-jar-plugin @@ -146,8 +197,8 @@ - io.questdb.client.azure + true ${project.version} @@ -188,7 +239,64 @@ 1.8 1.3.15 + ${env.JAVA11_HOME} + + + + org.apache.maven.plugins + maven-compiler-plugin + + + module-info.java + + + + + org.apache.maven.plugins + maven-antrun-plugin + 3.1.0 + + + compile-mrjar-module-info + prepare-package + + run + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + javadoc diff --git a/azure/src/main/java/io/questdb/client/azure/AzureTokenProviderFactory.java b/azure/src/main/java/io/questdb/client/azure/AzureTokenProviderFactory.java index b0522fa06..140e997dc 100644 --- a/azure/src/main/java/io/questdb/client/azure/AzureTokenProviderFactory.java +++ b/azure/src/main/java/io/questdb/client/azure/AzureTokenProviderFactory.java @@ -45,10 +45,13 @@ /** * The {@code azure} token provider (design/qwp-token-provider-spec.md, section 7.5): Microsoft Entra ID tokens * from Azure Identity. Discovered through {@link java.util.ServiceLoader}; put this artifact on the class path or - * module path and connect with + * module path, together with its runtime dependencies, and connect with * 
      * wss::addr=qdb1:9000,qdb2:9000;token_provider=azure;azure_resource=api://<questdb-app-id>;azure_credential=managed_identity;
  *
    + * On Java 11+, connect-string discovery needs only {@code requires io.questdb.client;} in the application. + * Service binding resolves this provider and its Azure SDK dependencies. The JDK 8 release jars carry the + * necessary descriptors in {@code META-INF/versions/11}. *
      *
    • {@code azure_resource} (required) - the application ID URI or client ID of the QuestDB app * registration; the requested scope is {@code /.default}.
      • diff --git a/azure/src/main/java/io/questdb/client/azure/AzureTokenSource.java b/azure/src/main/java/io/questdb/client/azure/AzureTokenSource.java index 676401ff1..d205399b9 100644 --- a/azure/src/main/java/io/questdb/client/azure/AzureTokenSource.java +++ b/azure/src/main/java/io/questdb/client/azure/AzureTokenSource.java @@ -73,7 +73,8 @@ * failure, such as a refused connection, which a managed identity otherwise hides behind a generic message. *
    * Each attempt is bounded (30 s by default). Library exceptions never travel as a cause - their response - * objects reference the raw HTTP request, which can carry a client secret - only their sanitized message does. + * objects reference the raw HTTP request, which can carry a client secret. Diagnostics name only the failure + * type, HTTP status, known configuration error codes and socket-level causes; SDK messages can embed raw bodies. */ public final class AzureTokenSource implements TokenSource { /** @@ -190,8 +191,10 @@ static TokenUnavailableException classify(Throwable failure, String scope, boole } private static String describe(Throwable t) { - final String message = t.getMessage(); - return message == null ? t.getClass().getName() : t.getClass().getSimpleName() + ": " + message; + // Even the outer SDK message can contain the complete token-endpoint response. Keep only the type; + // classify() adds safe status/error codes, and withNetworkCause() names socket-level failures. + final String name = t.getClass().getSimpleName(); + return name.isEmpty() ? t.getClass().getName() : name; } // The outermost socket-level failure among the causes: a refused or reset connection, an unreachable or @@ -251,18 +254,16 @@ private static long retryAfterMillis(Map> headers) { return TokenUnavailableException.NO_RETRY_AFTER; } - // Leads with the socket-level cause, unless the description already carries it: error text is capped, and a - // library's own message is often generic ("see inner exception"). + // Lead with the socket-level cause so it survives the provider's error-text cap. Only socket failures + // contribute message text; other SDK/parser messages can contain the token-endpoint response. private static String withNetworkCause(Throwable failure, String description) { final Throwable network = networkCause(failure); if (network == null) { return description; } final String reason = network.getMessage(); - if (reason != null && description.contains(reason)) { - return description; - } - return describe(network) + "; " + description; + final String networkDescription = describe(network) + (reason == null ? "" : ": " + reason); + return network == failure ? networkDescription : networkDescription + "; " + description; } @Override diff --git a/azure/src/main/java/module-info.java b/azure/src/main/java/module-info.java new file mode 100644 index 000000000..420a452a9 --- /dev/null +++ b/azure/src/main/java/module-info.java @@ -0,0 +1,36 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +module io.questdb.client.azure { + requires transitive com.azure.core; + requires com.azure.identity; + requires com.microsoft.aad.msal4j; + requires transitive io.questdb.client; + requires org.slf4j; + + exports io.questdb.client.azure; + + provides io.questdb.client.cutlass.auth.TokenProviderFactory + with io.questdb.client.azure.AzureTokenProviderFactory; +} diff --git a/azure/src/test/java/io/questdb/client/azure/test/AzureClientSecretCredentialLibraryTest.java b/azure/src/test/java/io/questdb/client/azure/test/AzureClientSecretCredentialLibraryTest.java index 97c7c950d..fa83b735f 100644 --- a/azure/src/test/java/io/questdb/client/azure/test/AzureClientSecretCredentialLibraryTest.java +++ b/azure/src/test/java/io/questdb/client/azure/test/AzureClientSecretCredentialLibraryTest.java @@ -33,6 +33,7 @@ import com.azure.core.http.HttpResponse; import com.azure.identity.ClientSecretCredentialBuilder; import io.questdb.client.azure.AzureTokenSource; +import io.questdb.client.cutlass.auth.RefreshingTokenProvider; import io.questdb.client.cutlass.auth.TokenUnavailableException; import org.junit.Assert; import org.junit.Test; @@ -51,6 +52,7 @@ */ public class AzureClientSecretCredentialLibraryTest { private static final String AUTHORITY = "https://login.microsoftonline.com/" + "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee"; + private static final String BODY_SENTINEL = "BODY-SENTINEL"; private static final String CLIENT_SECRET = "not-a-real-secret"; @Test(timeout = 30_000) @@ -58,6 +60,33 @@ public void testFinalRetryAfterSurvivesSdkRetries() { assertRetryAfter(new TokenEndpoint(429, "7"), false, 7_000); } + @Test(timeout = 30_000) + public void testMalformedResponseBodyNeverReachesFailureDiagnostics() { + TokenEndpoint endpoint = new TokenEndpoint(400, null, "{\"unexpected\":\"" + BODY_SENTINEL + "\"}"); + AzureTokenSource source = new AzureTokenSource(credential(endpoint), "api://qdb"); + try { + source.fetchToken(); + Assert.fail("expected the malformed token response to fail"); + } catch (TokenUnavailableException e) { + assertNoResponseBody(e); + } + Assert.assertEquals(1, endpoint.tokenRequests.get()); + + try (RefreshingTokenProvider provider = RefreshingTokenProvider.builder(source) + .coldWaitMillis(10_000).backoffInitialMillis(60_000).backoffMaxMillis(60_000).build()) { + try { + provider.getToken(); + Assert.fail("expected the provider to report the source failure"); + } catch (TokenUnavailableException e) { + assertNoResponseBody(e); + } + Assert.assertNotNull(provider.getLastFailure()); + Assert.assertFalse(provider.getLastFailure().getMessage(), + provider.getLastFailure().getMessage().contains(BODY_SENTINEL)); + Assert.assertEquals(2, endpoint.tokenRequests.get()); + } + } + @Test(timeout = 30_000) public void testMissingOrMalformedRetryAfterUsesProviderBackoff() { for (String retryAfter : Arrays.asList(null, "", "not-a-delay", "Wed, 21 Oct 2015 07:28:00 GMT", "-1")) { @@ -65,6 +94,21 @@ public void testMissingOrMalformedRetryAfterUsesProviderBackoff() { } } + @Test(timeout = 30_000) + public void testOAuthErrorDescriptionNeverReachesFailureDiagnostics() { + TokenEndpoint endpoint = new TokenEndpoint(400, null, + "{\"error\":\"invalid_client\",\"error_description\":\"AADSTS7000215: " + BODY_SENTINEL + "\"}"); + try { + new AzureTokenSource(credential(endpoint), "api://qdb").fetchToken(); + Assert.fail("expected Entra to reject the credential"); + } catch (TokenUnavailableException e) { + assertNoResponseBody(e); + Assert.assertFalse(e.getMessage(), e.isRetryable()); + Assert.assertTrue(e.getMessage(), e.getMessage().contains("AADSTS7000215")); + } + Assert.assertEquals(1, endpoint.tokenRequests.get()); + } + @Test(timeout = 30_000) public void testServerFailureRetainsRetryAfter() { assertRetryAfter(new TokenEndpoint(503, "7"), false, 7_000); @@ -75,14 +119,13 @@ public void testWrappedMsalFailureRetainsRetryAfter() { assertRetryAfter(new TokenEndpoint(429, " 7 "), true, 7_000); } + private static void assertNoResponseBody(TokenUnavailableException e) { + Assert.assertFalse(e.getMessage(), e.getMessage().contains(BODY_SENTINEL)); + Assert.assertNull("never retain an SDK exception that references the raw request", e.getCause()); + } + private static void assertRetryAfter(TokenEndpoint endpoint, boolean isWrapped, long expectedRetryAfterMillis) { - TokenCredential credential = new ClientSecretCredentialBuilder() - .tenantId("aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee") - .clientId("11111111-2222-3333-4444-555555555555") - .clientSecret(CLIENT_SECRET) - .disableInstanceDiscovery() - .httpClient(endpoint) - .build(); + TokenCredential credential = credential(endpoint); if (isWrapped) { TokenCredential delegate = credential; credential = context -> delegate.getToken(context).onErrorMap( @@ -100,6 +143,16 @@ private static void assertRetryAfter(TokenEndpoint endpoint, boolean isWrapped, } } + private static TokenCredential credential(TokenEndpoint endpoint) { + return new ClientSecretCredentialBuilder() + .tenantId("aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee") + .clientId("11111111-2222-3333-4444-555555555555") + .clientSecret(CLIENT_SECRET) + .disableInstanceDiscovery() + .httpClient(endpoint) + .build(); + } + private static final class StubResponse extends HttpResponse { private final byte[] body; private final HttpHeaders headers; @@ -152,11 +205,18 @@ public int getStatusCode() { } private static final class TokenEndpoint implements HttpClient { + private final String body; private final String retryAfter; private final int status; private final AtomicInteger tokenRequests = new AtomicInteger(); private TokenEndpoint(int status, String retryAfter) { + this(status, retryAfter, "{\"error\":\"temporarily_unavailable\"," + + "\"error_description\":\"The endpoint is throttling this application\"}"); + } + + private TokenEndpoint(int status, String retryAfter, String body) { + this.body = body; this.retryAfter = retryAfter; this.status = status; } @@ -173,9 +233,7 @@ public Mono send(HttpRequest request) { Assert.assertEquals(HttpMethod.POST, request.getHttpMethod()); // Keep the SDK's first three retries immediate; only its final response supplies the provider's hint. String delay = tokenRequests.incrementAndGet() <= 3 ? "0" : retryAfter; - return Mono.just(new StubResponse(request, status, - "{\"error\":\"temporarily_unavailable\"," - + "\"error_description\":\"The endpoint is throttling this application\"}", delay)); + return Mono.just(new StubResponse(request, status, body, delay)); } } } diff --git a/azure/src/test/java/io/questdb/client/azure/test/AzureCredentialSelectionTest.java b/azure/src/test/java/io/questdb/client/azure/test/AzureCredentialSelectionTest.java index edcd8e93f..1ac18e2a1 100644 --- a/azure/src/test/java/io/questdb/client/azure/test/AzureCredentialSelectionTest.java +++ b/azure/src/test/java/io/questdb/client/azure/test/AzureCredentialSelectionTest.java @@ -135,7 +135,9 @@ public void testUnconfiguredWorkloadIdentityFailsPermanentlyWhenFetched() { } catch (TokenUnavailableException e) { Assert.assertFalse("missing configuration is permanent: " + e.getMessage(), e.isRetryable()); Assert.assertTrue(e.getMessage(), e.getMessage().startsWith("no credential available for api://qdb/.default: ")); - Assert.assertTrue(e.getMessage(), e.getMessage().contains("WorkloadIdentityCredentialBuilder")); + // Keep the classification/type, not the SDK builder's arbitrary message. + Assert.assertTrue(e.getMessage(), e.getMessage().contains("CredentialUnavailableException")); + Assert.assertNull(e.getCause()); } } diff --git a/azure/src/test/java/io/questdb/client/azure/test/AzureModulePathIT.java b/azure/src/test/java/io/questdb/client/azure/test/AzureModulePathIT.java new file mode 100644 index 000000000..d2ff663b7 --- /dev/null +++ b/azure/src/test/java/io/questdb/client/azure/test/AzureModulePathIT.java @@ -0,0 +1,143 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.azure.test; + +import org.junit.Assert; +import org.junit.Test; + +import java.io.File; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Comparator; +import java.util.Iterator; +import java.util.LinkedHashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.concurrent.TimeUnit; +import java.util.jar.JarFile; +import java.util.stream.Stream; + +/** + * Executes the packaged jars, not target/classes: the JDK 8 release needs both the core's service-uses + * descriptor and the adapter's SDK-dependency descriptor in META-INF/versions/11. The consumer requires only + * io.questdb.client; no --add-modules flag may supply the missing service binding or SDK resolution. + */ +public class AzureModulePathIT { + @Test + public void testJarLayout() throws Exception { + try (JarFile jar = new JarFile(property("questdb.client.azure.jar"))) { + Assert.assertEquals("io.questdb.client.azure", jar.getManifest().getMainAttributes().getValue("Automatic-Module-Name")); + Assert.assertEquals("true", jar.getManifest().getMainAttributes().getValue("Multi-Release")); + Assert.assertNotNull("classpath service registration missing", + jar.getEntry("META-INF/services/io.questdb.client.cutlass.auth.TokenProviderFactory")); + String descriptor = "1.8".equals(System.getProperty("java.specification.version")) + ? "META-INF/versions/11/module-info.class" : "module-info.class"; + Assert.assertNotNull("packaged module descriptor missing: " + descriptor, jar.getEntry(descriptor)); + } + } + + @Test(timeout = 120_000) + public void testPackagedModuleDiscovery() throws Exception { + Set jdks = new LinkedHashSet<>(); + if ("1.8".equals(System.getProperty("java.specification.version"))) { + String home = System.getProperty("java11.home"); + if (home == null || home.isEmpty() || home.startsWith("${")) { + home = System.getenv("JAVA11_HOME"); + } + Assert.assertNotNull("JDK 8 packaging needs JAVA11_HOME for the module-path smoke", home); + jdks.add(home); + } else { + jdks.add(System.getProperty("java.home")); + } + String extraJdks = System.getenv("QUESTDB_SMOKE_JDKS"); + if (extraJdks != null && !extraJdks.trim().isEmpty()) { + for (String home : extraJdks.split(File.pathSeparator)) { + if (!home.trim().isEmpty()) { + jdks.add(home.trim()); + } + } + } + for (String jdk : jdks) { + runSmoke(jdk); + } + } + + private static String property(String name) { + String value = System.getProperty(name); + Assert.assertNotNull(name + " is required; run this test through mvn verify", value); + Assert.assertTrue(name + " not found: " + value, new File(value).exists()); + return value; + } + + private static String run(String jdk, String tool, Path workspace, List args) throws Exception { + List command = new ArrayList<>(); + command.add(jdk + File.separator + "bin" + File.separator + tool); + command.addAll(args); + Path outputFile = workspace.resolve(tool + ".log"); + ProcessBuilder builder = new ProcessBuilder(command).redirectErrorStream(true).redirectOutput(outputFile.toFile()); + Map env = builder.environment(); + env.keySet().removeIf(key -> key.startsWith("AZURE_") || key.startsWith("IDENTITY_") + || key.startsWith("MSI_") || key.startsWith("IMDS_")); + Process process = builder.start(); + try { + Assert.assertTrue("module smoke timed out: " + command, process.waitFor(45, TimeUnit.SECONDS)); + String output = new String(Files.readAllBytes(outputFile), StandardCharsets.UTF_8); + Assert.assertEquals("module smoke failed: " + command + "\n" + output, 0, process.exitValue()); + return output; + } finally { + if (process.isAlive()) { + process.destroyForcibly(); + process.waitFor(5, TimeUnit.SECONDS); + } + } + } + + private static void runSmoke(String jdk) throws Exception { + Path workspace = Files.createTempDirectory("qdb-azure-module-path"); + try { + String source = property("questdb.client.azure.module.src"); + String modulePath = property("questdb.client.azure.jar") + File.pathSeparator + + property("questdb.client.azure.module.deps"); + Path modules = Files.createDirectory(workspace.resolve("modules")); + run(jdk, "javac", workspace, Arrays.asList("--release", "11", "--module-path", modulePath, + "-d", modules.toString(), source + File.separator + "module-info.java", source + File.separator + "Main.java")); + String output = run(jdk, "java", workspace, Arrays.asList("--module-path", + modulePath + File.pathSeparator + modules, "-m", "qdb.azure.smoke/smoke.Main")); + Assert.assertTrue(output, output.contains("azure-module-path-ok")); + System.out.println("Azure packaged module-path smoke passed on " + jdk); + } finally { + try (Stream paths = Files.walk(workspace)) { + Iterator iterator = paths.sorted(Comparator.reverseOrder()).iterator(); + while (iterator.hasNext()) { + Files.deleteIfExists(iterator.next()); + } + } + } + } +} diff --git a/azure/src/test/java/io/questdb/client/azure/test/AzureTokenSourceTest.java b/azure/src/test/java/io/questdb/client/azure/test/AzureTokenSourceTest.java index b73c10776..7b3764e21 100644 --- a/azure/src/test/java/io/questdb/client/azure/test/AzureTokenSourceTest.java +++ b/azure/src/test/java/io/questdb/client/azure/test/AzureTokenSourceTest.java @@ -188,17 +188,21 @@ public void testNetworkCauseLeadsTheMessage() { Assert.assertTrue(e.isRetryable()); Assert.assertTrue(e.getMessage(), e.getMessage().startsWith("the token request for api://qdb/.default failed: " + "ConnectException: finishConnect(..) failed with error(-111): Connection refused; " - + "ClientAuthenticationException: Managed Identity authentication failed")); + + "ClientAuthenticationException")); e = fetchFailure(new RuntimeException("wrapped", new UnknownHostException("login.microsoftonline.com"))); Assert.assertTrue(e.isRetryable()); Assert.assertTrue(e.getMessage(), e.getMessage().contains("UnknownHostException: login.microsoftonline.com")); - // a wrapper whose own message already carries the cause - as Reactor's wrapper of a checked exception does - - // names it once + // A wrapper whose message already carries the cause must not cause the source to echo arbitrary text. e = fetchFailure(new UncheckedIOException(new ConnectException("Connection refused"))); - Assert.assertEquals("the token request for api://qdb/.default failed: UncheckedIOException: " - + "java.net.ConnectException: Connection refused", e.getMessage()); + Assert.assertEquals("the token request for api://qdb/.default failed: ConnectException: " + + "Connection refused; UncheckedIOException", e.getMessage()); + + e = fetchFailure(new ConnectException("Connection refused")); + // Reactor wraps a checked socket exception before block() throws it. + Assert.assertEquals("the token request for api://qdb/.default failed: ConnectException: Connection refused; ReactiveException", + e.getMessage()); } @Test @@ -216,11 +220,11 @@ public void testOtherFailuresAreRetryable() { assertRetryable(new HttpResponseException("throttled", new FakeResponse(429, "7")), "HTTP 429", 7_000); assertRetryable(new HttpResponseException("unavailable", new FakeResponse(503, null)), "HTTP 503", -1); assertRetryable(new HttpResponseException("gone", new FakeResponse(410, "Wed, 21 Oct 2015 07:28:00 GMT")), "HTTP 410", -1); - assertRetryable(new IllegalStateException("connection reset"), "connection reset", -1); + assertRetryable(new IllegalStateException("connection reset"), "IllegalStateException", -1); assertRetryable(new MsalServiceException("temporarily unavailable", "temporarily_unavailable"), - "temporarily unavailable", -1); + "MsalServiceException", -1); assertRetryable(new ClientAuthenticationException("ManagedIdentityCredential: IMDS endpoint timed out", null, - (Object) null), "timed out", -1); + (Object) null), "ClientAuthenticationException", -1); } @Test diff --git a/azure/src/test/resources/module-path/Main.java b/azure/src/test/resources/module-path/Main.java new file mode 100644 index 000000000..a7cf9d105 --- /dev/null +++ b/azure/src/test/resources/module-path/Main.java @@ -0,0 +1,49 @@ +/* + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package smoke; + +import io.questdb.client.Sender; +import io.questdb.client.cutlass.auth.TokenProviderFactory; +import io.questdb.client.cutlass.auth.TokenProviderRegistry; +import io.questdb.client.cutlass.auth.TokenProviderSpec; +import io.questdb.client.cutlass.auth.TokenSource; +import io.questdb.client.cutlass.qwp.client.QwpQueryClient; +import io.questdb.client.impl.ConfigString; +import io.questdb.client.impl.ConfigView; + +import java.util.HashMap; +import java.util.Map; + +public class Main { + public static void main(String[] args) { + String config = "wss::addr=localhost:1;token_provider=azure;azure_resource=api://qdb;azure_credential=environment;"; + Sender.builder(config); + QwpQueryClient.validateConfig(new ConfigView(ConfigString.parse(config)), true); + TokenProviderFactory factory = TokenProviderRegistry.findFactory("azure"); + if (factory == null) { + throw new AssertionError("Azure factory missing from service binding"); + } + Map params = new HashMap<>(); + params.put(TokenProviderSpec.KEY_AZURE_RESOURCE, "api://qdb"); + params.put(TokenProviderSpec.KEY_AZURE_CREDENTIAL, "environment"); + TokenSource source = factory.createSource(params); + if (!"io.questdb.client.azure.AzureTokenSource".equals(source.getClass().getName())) { + throw new AssertionError("unexpected source: " + source.getClass().getName()); + } + System.out.println("azure-module-path-ok"); + } +} diff --git a/azure/src/test/resources/module-path/module-info.java b/azure/src/test/resources/module-path/module-info.java new file mode 100644 index 000000000..e1935b31e --- /dev/null +++ b/azure/src/test/resources/module-path/module-info.java @@ -0,0 +1,19 @@ +/* + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +module qdb.azure.smoke { + requires io.questdb.client; +} diff --git a/core/pom.xml b/core/pom.xml index 9996932a2..3a580804a 100644 --- a/core/pom.xml +++ b/core/pom.xml @@ -117,10 +117,9 @@ ${buildNumber} ${project.version} - + io.questdb.client true @@ -598,6 +597,22 @@ cross-checks the packaged jar against the source root) --> + + + + + + + + + + + + + + +     diff --git a/core/src/test/java/io/questdb/client/test/std/JarPackagingIT.java b/core/src/test/java/io/questdb/client/test/std/JarPackagingIT.java index 8733fa150..8ddaf8a83 100644 --- a/core/src/test/java/io/questdb/client/test/std/JarPackagingIT.java +++ b/core/src/test/java/io/questdb/client/test/std/JarPackagingIT.java @@ -109,6 +109,10 @@ public void testJarLayout() throws Exception { "root FdBig of a JDK 8 build must use sun.misc.FDBigInteger", classReferences(jar, FD_BIG_ENTRY, "sun/misc/FDBigInteger") ); + Assert.assertNull("JDK 8 must not load a root module descriptor", jar.getEntry("module-info.class")); + Assert.assertTrue("the released descriptor must declare the token-provider service use", + classReferences(jar, "META-INF/versions/11/module-info.class", + "io/questdb/client/cutlass/auth/TokenProviderFactory")); // the java11 bridge never names the class (JDK 26 made it package-private); // it loads it by dotted name and binds method handles, so the // constant-pool witness is the dotted string literal From 5b1da601997d94895bd6f7b230e865952187f207 Mon Sep 17 00:00:00 2001 From: glasstiger Date: Thu, 15 Oct 2026 23:44:39 +0100 Subject: [PATCH 26/40] Report health loss during dictionary recycling QwpWebSocketSender now reports connection loss when it retires the\noutgoing I/O loop. A failed slot cleanup or engine rebuild therefore\nleaves health in RECONNECTING with an outage timestamp instead of\nreporting CONNECTED after the transport has closed.\n\nExtend the existing rebuild and filesystem-cleanup tests to check\nconnection health during failure and after recovery. Repeated rebuild\nfailures retain the same outage start and last-connected timestamp.\n\nBoth regression tests fail without the production change. All 107\ndictionary-recycle and connection-health tests pass on native arm64\nJDK 25 and on Java 8 with -Xint, which avoids the local C1 crash\nreproduced on both the PR head and base during review. --- .../qwp/client/QwpWebSocketSender.java | 1 + .../client/SymbolDictRecycleSlotHealTest.java | 7 ++++ .../qwp/client/SymbolDictRecycleTest.java | 34 +++++++++++++++---- 3 files changed, 35 insertions(+), 7 deletions(-) diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpWebSocketSender.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpWebSocketSender.java index 720fc3c58..a6f599a2d 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpWebSocketSender.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpWebSocketSender.java @@ -6376,6 +6376,7 @@ private RuntimeException rethrowRecycleAbandoned(Throwable t, String message) { */ private void retireStoppedLoop() { cursorSendLoop.checkError(); + healthTracker.connectionLost(); hasLoopEverConnected |= cursorSendLoop.hasEverConnected(); cursorSendLoop = null; } diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/SymbolDictRecycleSlotHealTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/SymbolDictRecycleSlotHealTest.java index 3fc480585..cf976503e 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/SymbolDictRecycleSlotHealTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/SymbolDictRecycleSlotHealTest.java @@ -24,6 +24,7 @@ package io.questdb.client.test.cutlass.qwp.client; +import io.questdb.client.ConnectionHealth; import io.questdb.client.Sender; import io.questdb.client.cutlass.line.LineSenderException; import io.questdb.client.cutlass.qwp.client.QwpWebSocketSender; @@ -369,6 +370,10 @@ public void testHealThatKeepsRecoveringLeftoversRefusesUntilCleanupIsDurable() t Assert.assertEquals("rebuild #1 recovers, the heal rebuilds #2, which recovers again", 2, rebuilds.get()); Assert.assertEquals("no swap on a recovered engine", 0, ws.getSymbolDictEpoch()); + ConnectionHealth health = sender.health(); + Assert.assertEquals(health.toString(), ConnectionHealth.State.RECONNECTING, health.getState()); + Assert.assertNotEquals("the retired connection must start an outage", + ConnectionHealth.NONE, health.getOutageSinceEpochMillis()); Assert.assertTrue("the leftovers are still on the slot", Files.exists(doctoredSlot + "/sf-initial.sfa")); @@ -386,6 +391,8 @@ public void testHealThatKeepsRecoveringLeftoversRefusesUntilCleanupIsDurable() t long fsn2 = sender.flushAndGetSequence(); Assert.assertTrue("post-heal batch must still get acked", sender.awaitAckedFsn(fsn2, 5_000)); + Assert.assertEquals(ConnectionHealth.State.CONNECTED, sender.health().getState()); + Assert.assertEquals(ConnectionHealth.NONE, sender.health().getOutageSinceEpochMillis()); Assert.assertTrue("post-recycle FSN must exceed pre-recycle FSN " + "[fsn1=" + fsn1 + ", fsn2=" + fsn2 + ']', fsn2 > fsn1); } finally { diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/SymbolDictRecycleTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/SymbolDictRecycleTest.java index 84ef326e5..5e39bca53 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/SymbolDictRecycleTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/SymbolDictRecycleTest.java @@ -24,6 +24,7 @@ package io.questdb.client.test.cutlass.qwp.client; +import io.questdb.client.ConnectionHealth; import io.questdb.client.Sender; import io.questdb.client.SenderConnectionEvent; import io.questdb.client.SenderError; @@ -387,7 +388,7 @@ public void testFailedRebuildAbandonsAndRecovers() throws Exception { QwpWebSocketSender ws = (QwpWebSocketSender) sender; QwpWebSocketSender.EngineRebuildFactory real = ws.getEngineRebuildFactoryForTesting(); - AtomicInteger remainingFaults = new AtomicInteger(1); + AtomicInteger remainingFaults = new AtomicInteger(2); ws.setEngineRebuildFactory(() -> { if (remainingFaults.getAndDecrement() > 0) { throw new RuntimeException("injected engine rebuild fault"); @@ -395,21 +396,40 @@ public void testFailedRebuildAbandonsAndRecovers() throws Exception { return real.rebuild(); }); + ConnectionHealth initialHealth = sender.health(); + Assert.assertEquals(ConnectionHealth.State.CONNECTED, initialHealth.getState()); sender.resetSymbolDictionary(); Assert.assertTrue(ws.isResetArmed()); - try { - sender.table("t"); - Assert.fail("expected the triggering table() call to throw"); - } catch (LineSenderException expected) { + long outageSince = ConnectionHealth.NONE; + for (int attempt = 0; attempt < 2; attempt++) { + try { + sender.table("t"); + Assert.fail("expected the rebuild to fail"); + } catch (LineSenderException expected) { + TestUtils.assertContains(expected.getMessage(), "could not rebuild its engine"); + } + // The retired connection stays down until a rebuild succeeds. + ConnectionHealth health = sender.health(); + Assert.assertEquals(health.toString(), ConnectionHealth.State.RECONNECTING, health.getState()); + Assert.assertNotEquals(ConnectionHealth.NONE, health.getOutageSinceEpochMillis()); + Assert.assertEquals(initialHealth.getLastConnectedAtEpochMillis(), health.getLastConnectedAtEpochMillis()); + Assert.assertEquals("no connect round has failed", 0, health.getFailedRounds()); + if (attempt == 0) { + outageSince = health.getOutageSinceEpochMillis(); + } else { + Assert.assertEquals("retrying the rebuild must not restart the outage", + outageSince, health.getOutageSinceEpochMillis()); + } + Assert.assertEquals("the swap must not commit on failure", 0, ws.getSymbolDictEpoch()); } - // NOT latched, and the swap did NOT commit. - Assert.assertEquals(0, ws.getSymbolDictEpoch()); // The next call resumes the pending recycle with the real // factory and completes it. sender.table("t").symbol("s", "post").longColumn("v", 1L).atNow(); Assert.assertEquals(1, ws.getSymbolDictEpoch()); long f = sender.flushAndGetSequence(); Assert.assertTrue(sender.awaitAckedFsn(f, 5_000)); + Assert.assertEquals(ConnectionHealth.State.CONNECTED, sender.health().getState()); + Assert.assertEquals(ConnectionHealth.NONE, sender.health().getOutageSinceEpochMillis()); } } }); From ff6c0ee2bb35996e96217ecac67cb9cff91d738b Mon Sep 17 00:00:00 2001 From: glasstiger Date: Fri, 16 Oct 2026 00:11:20 +0100 Subject: [PATCH 27/40] Fix the health test recovery ordering Make the test server accept upgrades before restoring the token provider.\nThis prevents a reconnect between the recovery writes from replacing\nthe retained credential failure with an authentication rejection.\n\nAll 11 ConnectionHealthTest tests pass on native arm64 JDK 25. --- .../client/test/cutlass/qwp/client/ConnectionHealthTest.java | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/ConnectionHealthTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/ConnectionHealthTest.java index 71d49a03d..1e4c71bee 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/ConnectionHealthTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/ConnectionHealthTest.java @@ -493,9 +493,9 @@ public void testSenderHealthMovesThroughEveryState() throws Exception { Assert.assertTrue(h.getFailedRounds() >= 2); Assert.assertFalse("no credential may appear: " + h, h.toString().contains(SENTINEL)); - // back to connected - providerMode.set("ok"); + // Accept upgrades before restoring tokens to avoid another authentication failure. rejectWith.set(0); + providerMode.set("ok"); await(() -> sender.health().getState() == ConnectionHealth.State.CONNECTED, 10_000, "recovered"); h = sender.health(); Assert.assertEquals(ConnectionHealth.NONE, h.getOutageSinceEpochMillis()); From dc9cf221699b553c4a9c828fa897696c95b12b68 Mon Sep 17 00:00:00 2001 From: glasstiger Date: Fri, 16 Oct 2026 01:00:16 +0100 Subject: [PATCH 28/40] Resolve query token provider configuration once Reuse the token-provider specification returned by shared config validation instead of resolving it again in QwpQueryClient.fromConfig. Keep the public standalone validator and validation order unchanged. Count factory validations in the test fixture and cover one resolution per client, normalized parameters, and setup without token acquisition. Validation: - Confirmed the regression test fails before the production change. - Passed 165 focused tests on JDK 25 and Java 8 (-Xint). --- .../cutlass/qwp/client/QwpQueryClient.java | 162 +++++++++--------- .../auth/TestTokenProviderFactory.java | 4 +- .../cutlass/auth/TokenProviderConfigTest.java | 18 ++ 3 files changed, 104 insertions(+), 80 deletions(-) diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpQueryClient.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpQueryClient.java index 6609bd07c..ad21fb62a 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpQueryClient.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpQueryClient.java @@ -399,8 +399,7 @@ public static QwpQueryClient fromConfig(CharSequence configurationString) { "unsupported schema [schema=" + cs.schema() + ", supported-schemas=[ws, wss]]"); } ConfigView view = new ConfigView(cs); - validateConfig(view, tls); - TokenProviderSpec tokenProviderSpec = TokenProviderSpec.parse(view, tls); + TokenProviderSpec tokenProviderSpec = validateConfigAndResolveTokenProvider(view, tls); List parsedEndpoints = new ArrayList<>(); view.getHostPorts("addr", DEFAULT_WS_PORT, (h, p) -> parsedEndpoints.add(new Endpoint(h, p))); @@ -513,85 +512,12 @@ public static QwpQueryClient fromConfig(CharSequence configurationString) { /** * Validates the cross-key invariants of an egress {@code ws}/{@code wss} - * config without constructing a client. Shared by {@link #fromConfig} and - * the {@code QuestDB} facade's fail-fast build path. {@code tls} is true for - * the {@code wss} schema. + * config without constructing a client. The {@code QuestDB} facade uses + * this for fail-fast validation with the same rules as {@link #fromConfig}. + * {@code tls} is true for the {@code wss} schema. */ public static void validateConfig(ConfigView view, boolean tls) { - view.getHostPorts("addr", DEFAULT_WS_PORT, (h, p) -> { - }); - if (!view.has("addr")) { - throw new IllegalArgumentException("missing required key: addr"); - } - // Trigger range/enum validation of every typed value. - view.getEnum("target"); - view.getEnum("compression"); - view.getEnum("tls_verify"); - view.getBoolOnOff("failover", false); - view.getInt("failover_max_attempts", 0); - view.getInt("max_batch_rows", 0); - view.getInt("buffer_pool_size", 0); - view.getInt("compression_level", 0); - boolean hasBackoffInitial = view.has("failover_backoff_initial_ms"); - boolean hasBackoffMax = view.has("failover_backoff_max_ms"); - // getLong also range-validates the value; call it even for an absent key. - long backoffInitial = view.getLong("failover_backoff_initial_ms", -1); - long backoffMax = view.getLong("failover_backoff_max_ms", -1); - view.getLong("failover_max_duration_ms", -1); - view.getLong("initial_credit", -1); - view.getLong("auth_timeout_ms", -1); - // getInt: connect_timeout feeds an int API, so validation must also - // reject values that fit a long but not an int. - view.getInt("connect_timeout", -1); - String username = view.getStr("username"); - String password = view.getStr("password"); - String token = view.getStr("token"); - // A present-but-blank credential is rejected up front, matching the - // ingress Sender, so a shared ws/wss string fails the same way on both - // sides and the client never builds an empty Authorization header. - if (username != null && Chars.isBlank(username)) { - throw new IllegalArgumentException("username cannot be empty nor null"); - } - if (password != null && Chars.isBlank(password)) { - throw new IllegalArgumentException("password cannot be empty nor null"); - } - if (token != null && Chars.isBlank(token)) { - throw new IllegalArgumentException("token cannot be empty nor null"); - } - boolean hasBasic = username != null || password != null; - if (hasBasic && (username == null || password == null)) { - throw new IllegalArgumentException("username and password must be provided together"); - } - if (hasBasic && token != null) { - throw new IllegalArgumentException("cannot use both token and username/password authentication"); - } - String tlsVerify = view.getStr("tls_verify"); - String tlsRoots = view.getStr("tls_roots"); - String tlsRootsPassword = view.getStr("tls_roots_password"); - if (!tls && (tlsVerify != null || tlsRoots != null || tlsRootsPassword != null)) { - throw new IllegalArgumentException( - "tls_verify/tls_roots/tls_roots_password require the wss:: schema"); - } - if (tlsRoots == null && tlsRootsPassword != null) { - throw new IllegalArgumentException("tls_roots_password requires tls_roots"); - } - if (tlsRoots != null && "unsafe_off".equals(tlsVerify)) { - throw new IllegalArgumentException(TLS_ROOTS_INSECURE_CONFIG_ERROR); - } - // token_provider and its keys (design/qwp-token-provider-spec.md, section 7.2): resolves the factory, never - // fetches a token. - TokenProviderSpec.parse(view, tls); - // Mirror fromConfig's effective values: a missing bound takes its - // default, so the ordering is enforced even when only one key is set - // (e.g. failover_backoff_max_ms alone, below the default initial backoff). - if (hasBackoffInitial || hasBackoffMax) { - long effectiveInitial = hasBackoffInitial ? backoffInitial : DEFAULT_FAILOVER_INITIAL_BACKOFF_MS; - long effectiveMax = hasBackoffMax ? backoffMax : Math.max(effectiveInitial, DEFAULT_FAILOVER_MAX_BACKOFF_MS); - if (effectiveMax < effectiveInitial) { - throw new IllegalArgumentException( - "failover_backoff_max_ms must be >= failover_backoff_initial_ms"); - } - } + validateConfigAndResolveTokenProvider(view, tls); } /** @@ -1552,6 +1478,84 @@ private static boolean matchesTarget(byte role, String target) { return true; } + private static TokenProviderSpec validateConfigAndResolveTokenProvider(ConfigView view, boolean tls) { + view.getHostPorts("addr", DEFAULT_WS_PORT, (h, p) -> { + }); + if (!view.has("addr")) { + throw new IllegalArgumentException("missing required key: addr"); + } + // Trigger range/enum validation of every typed value. + view.getEnum("target"); + view.getEnum("compression"); + view.getEnum("tls_verify"); + view.getBoolOnOff("failover", false); + view.getInt("failover_max_attempts", 0); + view.getInt("max_batch_rows", 0); + view.getInt("buffer_pool_size", 0); + view.getInt("compression_level", 0); + boolean hasBackoffInitial = view.has("failover_backoff_initial_ms"); + boolean hasBackoffMax = view.has("failover_backoff_max_ms"); + // getLong also range-validates the value; call it even for an absent key. + long backoffInitial = view.getLong("failover_backoff_initial_ms", -1); + long backoffMax = view.getLong("failover_backoff_max_ms", -1); + view.getLong("failover_max_duration_ms", -1); + view.getLong("initial_credit", -1); + view.getLong("auth_timeout_ms", -1); + // getInt: connect_timeout feeds an int API, so validation must also + // reject values that fit a long but not an int. + view.getInt("connect_timeout", -1); + String username = view.getStr("username"); + String password = view.getStr("password"); + String token = view.getStr("token"); + // A present-but-blank credential is rejected up front, matching the + // ingress Sender, so a shared ws/wss string fails the same way on both + // sides and the client never builds an empty Authorization header. + if (username != null && Chars.isBlank(username)) { + throw new IllegalArgumentException("username cannot be empty nor null"); + } + if (password != null && Chars.isBlank(password)) { + throw new IllegalArgumentException("password cannot be empty nor null"); + } + if (token != null && Chars.isBlank(token)) { + throw new IllegalArgumentException("token cannot be empty nor null"); + } + boolean hasBasic = username != null || password != null; + if (hasBasic && (username == null || password == null)) { + throw new IllegalArgumentException("username and password must be provided together"); + } + if (hasBasic && token != null) { + throw new IllegalArgumentException("cannot use both token and username/password authentication"); + } + String tlsVerify = view.getStr("tls_verify"); + String tlsRoots = view.getStr("tls_roots"); + String tlsRootsPassword = view.getStr("tls_roots_password"); + if (!tls && (tlsVerify != null || tlsRoots != null || tlsRootsPassword != null)) { + throw new IllegalArgumentException( + "tls_verify/tls_roots/tls_roots_password require the wss:: schema"); + } + if (tlsRoots == null && tlsRootsPassword != null) { + throw new IllegalArgumentException("tls_roots_password requires tls_roots"); + } + if (tlsRoots != null && "unsafe_off".equals(tlsVerify)) { + throw new IllegalArgumentException(TLS_ROOTS_INSECURE_CONFIG_ERROR); + } + // token_provider and its keys (design/qwp-token-provider-spec.md, section 7.2): resolves the factory, never + // fetches a token. + TokenProviderSpec tokenProviderSpec = TokenProviderSpec.parse(view, tls); + // Mirror fromConfig's effective values: a missing bound takes its + // default, so the ordering is enforced even when only one key is set + // (e.g. failover_backoff_max_ms alone, below the default initial backoff). + if (hasBackoffInitial || hasBackoffMax) { + long effectiveInitial = hasBackoffInitial ? backoffInitial : DEFAULT_FAILOVER_INITIAL_BACKOFF_MS; + long effectiveMax = hasBackoffMax ? backoffMax : Math.max(effectiveInitial, DEFAULT_FAILOVER_MAX_BACKOFF_MS); + if (effectiveMax < effectiveInitial) { + throw new IllegalArgumentException( + "failover_backoff_max_ms must be >= failover_backoff_initial_ms"); + } + } + return tokenProviderSpec; + } + /** * Builds the {@code X-QWP-Accept-Encoding} header value from the user's * preference. {@code raw} (the library default) omits the header entirely diff --git a/core/src/test/java/io/questdb/client/test/cutlass/auth/TestTokenProviderFactory.java b/core/src/test/java/io/questdb/client/test/cutlass/auth/TestTokenProviderFactory.java index 357d80ad8..9c1504f63 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/auth/TestTokenProviderFactory.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/auth/TestTokenProviderFactory.java @@ -39,7 +39,7 @@ /** * A {@link TokenProviderFactory} for tests, installed in place of {@link java.util.ServiceLoader} discovery with - * {@link #install(TestTokenProviderFactory...)}. It counts the sources it creates and the fetches they make. + * {@link #install(TestTokenProviderFactory...)}. It counts source creation, token fetches and configuration validation. * Registering one named {@code azure} lets core tests drive the {@code azure} connect-string keys without the * optional module. */ @@ -47,6 +47,7 @@ public final class TestTokenProviderFactory implements TokenProviderFactory { public final AtomicInteger created = new AtomicInteger(); public final AtomicInteger fetches = new AtomicInteger(); public final List> params = new CopyOnWriteArrayList<>(); + public final AtomicInteger validations = new AtomicInteger(); private final String name; private final Function, ExpiringToken> tokens; @@ -92,6 +93,7 @@ public String name() { @Override public void validate(Map params) { + validations.incrementAndGet(); String resource = params.get("azure_resource"); if (resource != null && resource.startsWith("invalid:")) { throw new IllegalArgumentException("azure_resource rejected by the factory"); diff --git a/core/src/test/java/io/questdb/client/test/cutlass/auth/TokenProviderConfigTest.java b/core/src/test/java/io/questdb/client/test/cutlass/auth/TokenProviderConfigTest.java index 3e09e3cf9..69d78a0e2 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/auth/TokenProviderConfigTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/auth/TokenProviderConfigTest.java @@ -128,6 +128,21 @@ public void testBothClientsAcceptTheKeys() throws Exception { }); } + @Test + public void testQueryClientResolvesTokenProviderOnce() throws Exception { + assertMemoryLeak(() -> { + String cfg = WSS + "token_provider=azure;azure_resource=api://qdb-app/.default;"; + try (QwpQueryClient client = QwpQueryClient.fromConfig(cfg)) { + Assert.assertEquals("one factory validation per client", 1, azure.validations.get()); + Map snap = client.configSnapshotForTest(); + Assert.assertEquals("azure", snap.get("token_provider")); + Assert.assertEquals("api://qdb-app", snap.get("azure_resource")); + } + Assert.assertEquals("configuration must not create a source", 0, azure.created.get()); + Assert.assertEquals("configuration must not fetch a token", 0, azure.fetches.get()); + }); + } + @Test public void testRegistryLifecycle() { // C19: lease release, linger, and re-acquiring a lease during the linger period. @@ -263,7 +278,10 @@ public void testValidationWithoutConnectingFetchesNoToken() throws Exception { String cfg = WSS + "token_provider=azure;azure_resource=api://validate-only;" + "sender_pool_min=0;query_pool_min=0;"; Sender.builder(cfg); + int validations = azure.validations.get(); QwpQueryClient.validateConfig(new ConfigView(ConfigString.parse(cfg)), true); + Assert.assertEquals("standalone validation must still resolve the provider", validations + 1, + azure.validations.get()); QwpQueryClient.fromConfig(cfg).close(); try (QuestDB ignored = QuestDB.connect(cfg)) { Assert.assertEquals("no provider may be started", 0, azure.created.get()); From 51dc7407274bcfa288568adc97e71d413d7bf954 Mon Sep 17 00:00:00 2001 From: glasstiger Date: Fri, 16 Oct 2026 03:04:10 +0100 Subject: [PATCH 29/40] Fix Azure rejection handling and replay test race Classify MSAL HTTP 400 and 401 responses as permanent credential failures even when their AADSTS code is outside the allowlist. Guard the nullable status and preserve known-code diagnostics and retry hints without retaining SDK response bodies or causes. Cover direct and wrapped rejections through the pinned real Azure SDK. Both new regression tests fail against the previous classifier. Wait for the replay frame counter before asserting its exact value, rather than relying on the earlier payload publication. A forced pause reproduces the old assertion failure and passes with the new wait. Validation: - Pass 136 focused tests on native arm64 JDK 25. - Pass 136 focused tests and four packaging checks on JDK 8 with C1 disabled; packaged jars run on JDK 11 and JDK 25. - Reproduce the normal JDK 8 C1 crash in Unsafe.recordMemAlloc on untouched dc9cf221, without either fix. --- .../client/azure/AzureTokenSource.java | 16 +++++-- ...zureClientSecretCredentialLibraryTest.java | 47 ++++++++++++++++--- .../WebSocketDynamicCredentialTest.java | 2 +- 3 files changed, 53 insertions(+), 12 deletions(-) diff --git a/azure/src/main/java/io/questdb/client/azure/AzureTokenSource.java b/azure/src/main/java/io/questdb/client/azure/AzureTokenSource.java index d205399b9..d1bd3a342 100644 --- a/azure/src/main/java/io/questdb/client/azure/AzureTokenSource.java +++ b/azure/src/main/java/io/questdb/client/azure/AzureTokenSource.java @@ -167,10 +167,6 @@ static TokenUnavailableException classify(Throwable failure, String scope, boole retryAfterMillis(response)); } } - if (t instanceof MsalServiceException && retryAfterMillis == TokenUnavailableException.NO_RETRY_AFTER) { - // Service-principal credentials expose response headers through MSAL, not HttpResponseException. - retryAfterMillis = retryAfterMillis(((MsalServiceException) t).headers()); - } final String message = t.getMessage(); if (message != null) { for (String code : PERMANENT_AADSTS) { @@ -185,6 +181,18 @@ static TokenUnavailableException classify(Throwable failure, String scope, boole + ": the requested identity has not been assigned to this resource"); } } + if (t instanceof MsalServiceException) { + // Service-principal credentials expose status and headers through MSAL, not HttpResponseException. + final MsalServiceException msal = (MsalServiceException) t; + final Integer status = msal.statusCode(); + if (status != null && (status == 400 || status == 401)) { + return TokenUnavailableException.permanent( + "Entra rejected the token request for " + scope + " with HTTP " + status + ": " + description); + } + if (retryAfterMillis == TokenUnavailableException.NO_RETRY_AFTER) { + retryAfterMillis = retryAfterMillis(msal.headers()); + } + } } return TokenUnavailableException.retryable("the token request for " + scope + " failed: " + withNetworkCause(failure, description), retryAfterMillis); diff --git a/azure/src/test/java/io/questdb/client/azure/test/AzureClientSecretCredentialLibraryTest.java b/azure/src/test/java/io/questdb/client/azure/test/AzureClientSecretCredentialLibraryTest.java index fa83b735f..e4e726a71 100644 --- a/azure/src/test/java/io/questdb/client/azure/test/AzureClientSecretCredentialLibraryTest.java +++ b/azure/src/test/java/io/questdb/client/azure/test/AzureClientSecretCredentialLibraryTest.java @@ -55,6 +55,13 @@ public class AzureClientSecretCredentialLibraryTest { private static final String BODY_SENTINEL = "BODY-SENTINEL"; private static final String CLIENT_SECRET = "not-a-real-secret"; + @Test(timeout = 30_000) + public void testClientRejectionsWithoutKnownAadstsCodesArePermanent() { + for (int status : new int[]{400, 401}) { + assertPermanentClientRejection(status, false); + } + } + @Test(timeout = 30_000) public void testFinalRetryAfterSurvivesSdkRetries() { assertRetryAfter(new TokenEndpoint(429, "7"), false, 7_000); @@ -114,6 +121,13 @@ public void testServerFailureRetainsRetryAfter() { assertRetryAfter(new TokenEndpoint(503, "7"), false, 7_000); } + @Test(timeout = 30_000) + public void testWrappedClientRejectionsWithoutKnownAadstsCodesArePermanent() { + for (int status : new int[]{400, 401}) { + assertPermanentClientRejection(status, true); + } + } + @Test(timeout = 30_000) public void testWrappedMsalFailureRetainsRetryAfter() { assertRetryAfter(new TokenEndpoint(429, " 7 "), true, 7_000); @@ -121,18 +135,31 @@ public void testWrappedMsalFailureRetainsRetryAfter() { private static void assertNoResponseBody(TokenUnavailableException e) { Assert.assertFalse(e.getMessage(), e.getMessage().contains(BODY_SENTINEL)); + Assert.assertFalse(e.getMessage(), e.getMessage().contains(CLIENT_SECRET)); Assert.assertNull("never retain an SDK exception that references the raw request", e.getCause()); } - private static void assertRetryAfter(TokenEndpoint endpoint, boolean isWrapped, long expectedRetryAfterMillis) { - TokenCredential credential = credential(endpoint); - if (isWrapped) { - TokenCredential delegate = credential; - credential = context -> delegate.getToken(context).onErrorMap( - failure -> new ClientAuthenticationException("authentication failed", null, failure)); + private static void assertPermanentClientRejection(int status, boolean isWrapped) { + // Entra reports an invalid application ID as AADSTS700038, outside the known-code allowlist. + TokenEndpoint endpoint = new TokenEndpoint(status, null, + "{\"error\":\"unauthorized_client\",\"error_description\":\"AADSTS700038: " + + "00000000-0000-0000-0000-000000000000 is not a valid application identifier. " + + BODY_SENTINEL + "\"}"); + try { + new AzureTokenSource(credential(endpoint, isWrapped), "api://qdb").fetchToken(); + Assert.fail("expected Entra to reject the application"); + } catch (TokenUnavailableException e) { + Assert.assertFalse(e.getMessage(), e.isRetryable()); + Assert.assertTrue(e.getMessage(), e.getMessage().contains("HTTP " + status)); + Assert.assertEquals(TokenUnavailableException.NO_RETRY_AFTER, e.getRetryAfterMillis()); + assertNoResponseBody(e); } + Assert.assertEquals("a configuration rejection must not trigger SDK retries", 1, endpoint.tokenRequests.get()); + } + + private static void assertRetryAfter(TokenEndpoint endpoint, boolean isWrapped, long expectedRetryAfterMillis) { try { - new AzureTokenSource(credential, "api://qdb").fetchToken(); + new AzureTokenSource(credential(endpoint, isWrapped), "api://qdb").fetchToken(); Assert.fail("expected the token endpoint to reject the request"); } catch (TokenUnavailableException e) { Assert.assertEquals("the SDK must exhaust its default three retries", 4, endpoint.tokenRequests.get()); @@ -153,6 +180,12 @@ private static TokenCredential credential(TokenEndpoint endpoint) { .build(); } + private static TokenCredential credential(TokenEndpoint endpoint, boolean isWrapped) { + TokenCredential credential = credential(endpoint); + return isWrapped ? context -> credential.getToken(context).onErrorMap( + failure -> new ClientAuthenticationException("authentication failed", null, failure)) : credential; + } + private static final class StubResponse extends HttpResponse { private final byte[] body; private final HttpHeaders headers; diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/WebSocketDynamicCredentialTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/WebSocketDynamicCredentialTest.java index 1d04e21f9..50c1b0e67 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/WebSocketDynamicCredentialTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/WebSocketDynamicCredentialTest.java @@ -199,7 +199,7 @@ public void testFailoverAcrossARotationSendsTheNewTokenAndReplaysOnce() throws E Assert.assertEquals("T2", provider.getToken().toString()); a.close(); // A dies; the reconnect fails over to B - await(() -> ackB.distinctPayloads.size() == 5, 15_000, "the unacked frames to replay at B"); + await(() -> ackB.frames.get() >= 5, 15_000, "B to count the replayed frames"); Assert.assertEquals("B must receive the new token", "Bearer T2", b.pollAuthorizationHeader(5, TimeUnit.SECONDS)); Assert.assertEquals("B must never see the old token", 0, b.authRejectCount()); From 4aa6c392613aedeb3aac742b657582c9e2771392 Mon Sep 17 00:00:00 2001 From: glasstiger Date: Fri, 16 Oct 2026 12:44:05 +0100 Subject: [PATCH 30/40] Classify missing credential files as permanent Treat nested NoSuchFileException from Azure Identity as a permanent credential configuration failure. Keep the diagnostic free of SDK messages, file paths and retained SDK causes. Add a regression through the pinned WorkloadIdentityCredential for a configured federated token file that does not exist. Confirm that the test fails before the classifier change and passes with it. Validate 94 selected tests and four packaging checks on JDK 25 and JDK 8. Run the JDK 8 tests with -Xint to avoid the local C1 crash reproduced on both the branch and its merge base. --- .../client/azure/AzureTokenSource.java | 13 ++++++--- .../test/AzureCredentialSelectionTest.java | 27 +++++++++++++++++++ 2 files changed, 36 insertions(+), 4 deletions(-) diff --git a/azure/src/main/java/io/questdb/client/azure/AzureTokenSource.java b/azure/src/main/java/io/questdb/client/azure/AzureTokenSource.java index d1bd3a342..a9287c486 100644 --- a/azure/src/main/java/io/questdb/client/azure/AzureTokenSource.java +++ b/azure/src/main/java/io/questdb/client/azure/AzureTokenSource.java @@ -39,6 +39,7 @@ import java.net.SocketException; import java.net.SocketTimeoutException; import java.net.UnknownHostException; +import java.nio.file.NoSuchFileException; import java.time.Duration; import java.time.OffsetDateTime; import java.util.List; @@ -64,8 +65,8 @@ * 7.5 prescribes, the same way whichever credential is used: *
      *
    • permanent - no credential available ({@link CredentialUnavailableException}): a credential chain that - * found nothing, or a credential whose configuration is missing; a managed identity that is not assigned to - * this host, which a managed-identity credential used alone reports only in a nested cause; and + * found nothing, or a credential whose configuration or credential file is missing; a managed identity that is + * not assigned to this host, which a managed-identity credential used alone reports only in a nested cause; and * authentication failures Entra reports: an HTTP 400 or 401 from the token endpoint, or a known AADSTS * configuration error such as an invalid client or an application that does not exist;
      • *
    • retryable - network errors, timeouts, throttling, 5xx responses, and anything else. A @@ -73,8 +74,9 @@ * failure, such as a refused connection, which a managed identity otherwise hides behind a generic message.
      • *
    * Each attempt is bounded (30 s by default). Library exceptions never travel as a cause - their response - * objects reference the raw HTTP request, which can carry a client secret. Diagnostics name only the failure - * type, HTTP status, known configuration error codes and socket-level causes; SDK messages can embed raw bodies. + * objects reference the raw HTTP request, which can carry a client secret. Diagnostics name only safe failure + * reasons, failure types, HTTP status, known configuration error codes and socket-level causes; SDK messages can + * embed raw bodies. */ public final class AzureTokenSource implements TokenSource { /** @@ -150,6 +152,9 @@ static TokenUnavailableException classify(Throwable failure, String scope, boole long retryAfterMillis = TokenUnavailableException.NO_RETRY_AFTER; Throwable t = failure; for (int depth = 0; t != null && depth < MAX_CAUSE_DEPTH; depth++, t = t.getCause()) { + if (t instanceof NoSuchFileException) { + return TokenUnavailableException.permanent("credential file is missing for " + scope); + } if (t instanceof CredentialUnavailableException) { return TokenUnavailableException.permanent("no credential available " + (chain ? "in the Azure Identity chain " : "") + "for " + scope + ": " + description); diff --git a/azure/src/test/java/io/questdb/client/azure/test/AzureCredentialSelectionTest.java b/azure/src/test/java/io/questdb/client/azure/test/AzureCredentialSelectionTest.java index 1ac18e2a1..dd75a6b41 100644 --- a/azure/src/test/java/io/questdb/client/azure/test/AzureCredentialSelectionTest.java +++ b/azure/src/test/java/io/questdb/client/azure/test/AzureCredentialSelectionTest.java @@ -28,7 +28,9 @@ import ch.qos.logback.classic.Logger; import ch.qos.logback.classic.spi.ILoggingEvent; import ch.qos.logback.core.read.ListAppender; +import com.azure.identity.WorkloadIdentityCredentialBuilder; import io.questdb.client.azure.AzureTokenProviderFactory; +import io.questdb.client.azure.AzureTokenSource; import io.questdb.client.cutlass.auth.TokenProviderSpec; import io.questdb.client.cutlass.auth.TokenSource; import io.questdb.client.cutlass.auth.TokenUnavailableException; @@ -36,9 +38,12 @@ import io.questdb.client.impl.ConfigView; import org.junit.Assert; import org.junit.Assume; +import org.junit.Rule; import org.junit.Test; +import org.junit.rules.TemporaryFolder; import org.slf4j.LoggerFactory; +import java.io.File; import java.util.HashMap; import java.util.Map; @@ -52,6 +57,9 @@ public class AzureCredentialSelectionTest { private static final String CLIENT_ID = "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee"; private static final String WSS = "wss::addr=localhost:9000;token_provider=azure;azure_resource=api://qdb;"; + @Rule + public final TemporaryFolder temp = TemporaryFolder.builder().assureDeletion().build(); + @Test public void testConnectStringSelectsTheCredential() { TokenProviderSpec managed = spec("azure_credential=managed_identity;azure_client_id=" + CLIENT_ID.toUpperCase() + ';'); @@ -123,6 +131,25 @@ public void testFactoryRejectsParametersItCannotUse() { factory.validate(params("workload_identity", CLIENT_ID)); } + @Test(timeout = 30_000) + public void testMissingFederatedTokenFileIsPermanentWhenFetched() throws Exception { + File tokenFile = new File(temp.newFolder(), "federated-token.jwt"); + Assert.assertFalse(tokenFile.exists()); + TokenSource source = new AzureTokenSource(new WorkloadIdentityCredentialBuilder() + .tenantId("11111111-2222-3333-4444-555555555555") + .clientId(CLIENT_ID) + .tokenFilePath(tokenFile.getAbsolutePath()) + .build(), "api://qdb"); + try { + source.fetchToken(); + Assert.fail("expected the fetch to fail"); + } catch (TokenUnavailableException e) { + Assert.assertFalse("missing configuration is permanent: " + e.getMessage(), e.isRetryable()); + Assert.assertEquals("credential file is missing for api://qdb/.default", e.getMessage()); + Assert.assertNull("do not retain the SDK exception or file path", e.getCause()); + } + } + @Test public void testUnconfiguredWorkloadIdentityFailsPermanentlyWhenFetched() { // Azure Identity refuses to build a workload identity without its tenant ID and federated token file. That From a5a97024d62d492b8cc093746f1228d93344296f Mon Sep 17 00:00:00 2001 From: glasstiger Date: Sat, 17 Oct 2026 17:01:29 +0100 Subject: [PATCH 31/40] Drop unused requires from the client descriptor The released client jar is built on JDK 8 and used to carry no module descriptor, so on Java 11+ it loaded as an automatic module that required nothing. This branch compiles module-info.java into META-INF/versions/11 of that jar, which makes every mandatory requires in it binding for applications that load the client from the module path. No client code uses two of them, java.sql and jdk.management. A jlink image built from what jdeps reports for the client holds java.base, jdk.unsupported and java.desktop, so an application that runs the released client from the module path on such an image failed at startup with "FindException: Module java.sql not found, required by io.questdb.client", where the previous jar started. The descriptor drops both requires. DesktopFreeModulePathTest gains a test that resolves the client with the universe limited to java.base and jdk.unsupported and the client added as a root, whose requires --limit-modules does not close over. It failed with that FindException before the change. The CI MRJAR smoke job now also pins the mandatory requires of the released JDK 8 jar to java.base, jdk.unsupported and org.slf4j. --- .github/workflows/ci.yml | 19 ++++-- core/src/main/java/module-info.java | 7 +- .../auth/DesktopFreeModulePathTest.java | 68 +++++++++++++------ 3 files changed, 68 insertions(+), 26 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index dbee07377..79565db95 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -87,10 +87,12 @@ jobs: # Takes the JDK 8-built jar (the shipped artifact) from build-jdk8 and runs it # on every newer JDK the client must work on, asserting (a) the versioned - # module descriptor names io.questdb.client and (b) slow-path double formatting - # resolves the META-INF/versions/11 bridge. These are only observable in the - # PACKAGED jar on a modern JDK, and each regression here was invisible to the - # other jobs until the jar was EXECUTED on the right JDK/flags: + # module descriptor names io.questdb.client and has no mandatory requires + # beyond java.base, jdk.unsupported and org.slf4j, and (b) slow-path double + # formatting resolves the META-INF/versions/11 bridge. These are only + # observable in the PACKAGED jar on a modern JDK, and each regression here + # was invisible to the other jobs until the jar was EXECUTED on the right + # JDK/flags: # * 1.3.5-1.3.7: FdBig linked sun.misc.FDBigInteger (gone since Java 9) and # the module name degraded to the filename-derived "questdb.client". # * issue #96: JDK 26 made FDBigInteger package-private (passed 8/11/25). @@ -136,6 +138,15 @@ jobs: jar_file="${jar_file[0]}" jar --describe-module --file "$jar_file" --release 11 | tee module.txt grep -q '^io\.questdb\.client@' module.txt + # A mandatory requires that the application's runtime image lacks fails + # module resolution at startup. A jlink image built from the client's + # jdeps output has java.base, jdk.unsupported and at most java.desktop, + # so any other platform module the client uses must be a static requires. + mandatory="$(awk '$1 == "requires" && !/ static( |$)/ {print $2}' module.txt | sort | tr '\n' ' ')" + if [ "$mandatory" != "java.base jdk.unsupported org.slf4j " ]; then + echo "::error::unexpected mandatory requires in the released module descriptor: ${mandatory}" + exit 1 + fi javac -cp "$jar_file" -d smoke-classes \ core/src/test/java/io/questdb/client/test/std/DoubleFormatSmoke.java # The FdBig double formatter reaches jdk.internal.math.FDBigInteger by diff --git a/core/src/main/java/module-info.java b/core/src/main/java/module-info.java index 4f4bc9852..d8c892147 100644 --- a/core/src/main/java/module-info.java +++ b/core/src/main/java/module-info.java @@ -23,10 +23,14 @@ ******************************************************************************/ open module io.questdb.client { + // Every mandatory requires must be in the application's runtime image, or the module fails to resolve + // before any code runs - and the released jar ships this descriptor in META-INF/versions/11. Keep the + // mandatory ones to what client code uses: java.sql and jdk.management, which it never did, failed + // startup on jlink images built from the client's jdeps output. DesktopFreeModulePathTest and the CI + // MRJAR smoke job pin the set. requires transitive jdk.unsupported; requires static org.jetbrains.annotations; requires static java.management; - requires jdk.management; // STATIC, not mandatory: the only java.desktop reference is BrowserLauncher's java.awt.Desktop, used // best-effort by the default DeviceCodePrompt.openBrowser() to pop the verification URL. A mandatory // requires is resolved BEFORE any code runs, so on a runtime without java.desktop - a jlink image, a @@ -39,7 +43,6 @@ // including every class-path application, is unaffected because java.desktop is resolved there by // default. See DeviceCodePrompt#openBrowser(). requires static java.desktop; - requires java.sql; requires org.slf4j; exports io.questdb.client; diff --git a/core/src/test/java/io/questdb/client/test/cutlass/auth/DesktopFreeModulePathTest.java b/core/src/test/java/io/questdb/client/test/cutlass/auth/DesktopFreeModulePathTest.java index 5ee97eee8..aee99fa6c 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/auth/DesktopFreeModulePathTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/auth/DesktopFreeModulePathTest.java @@ -45,16 +45,24 @@ * satisfied during module RESOLUTION, before a single line of client code runs, so on a runtime image * without {@code java.desktop} the JVM failed at startup with {@code FindException} and the * {@code LinkageError} catch in {@code openBrowser()} - the thing that implements the promise - never got - * to run. The published artifact is built on JDK 8 and carries no descriptor, so it is an automatic module - * and was never affected; a build from this source is. + * to run. The published artifact is built on JDK 8 and carries this descriptor in + * {@code META-INF/versions/11}, so on Java 11+ it is affected exactly like a build from this source. *

    - * The test runs a second JVM with the client on the MODULE path and {@code --limit-modules} naming only - * the client, which limits the universe to the client plus the closure of its mandatory requires. A - * {@code requires static} is not part of that closure, so {@code java.desktop} is absent and the child - * proves three things at once: the module resolved without it, {@code java.awt.Desktop} really is - * unreachable (see {@link DesktopFreePromptMain}, which fails the run if it is not - this is what catches - * a revert to a mandatory requires, since the closure would then drag {@code java.desktop} back in), and - * the default prompt still renders the challenge and returns. + * The desktop-free tests run a second JVM with the client on the MODULE path and {@code --limit-modules} + * naming only the client, which limits the universe to the client plus the closure of its mandatory + * requires. A {@code requires static} is not part of that closure, so {@code java.desktop} is absent and + * the child proves three things at once: the module resolved without it, {@code java.awt.Desktop} really + * is unreachable (see {@link DesktopFreePromptMain}, which fails the run if it is not - this is what + * catches a revert to a mandatory requires, since the closure would then drag {@code java.desktop} back + * in), and the default prompt still renders the challenge and returns. + *

    + * Any other mandatory requires fails the same way on an image that lacks the module. A jlink image built + * from what {@code jdeps} reports for the client holds {@code java.base}, {@code jdk.unsupported} and the + * optional {@code java.desktop}, and {@code requires java.sql} and {@code requires jdk.management}, which no + * client code uses, failed startup there. {@link #testTheModuleResolvesWithOnlyItsRequiredPlatformModules()} + * limits the universe to those two platform modules and adds the client as a root, whose requires + * {@code --limit-modules} does not close over, so any other mandatory platform requires fails the child's + * resolution. *

    * Java 8 source level, and no {@code java.lang.module} API: the JDK 8 release profile compiles this test * tree, only excluding {@code module-info.java}. @@ -84,6 +92,16 @@ public void testTheModuleResolvesAndPromptsWithoutJavaDesktop() throws Exception Assert.assertTrue("the user code must still be printed:\n" + output, output.contains("WDJB-MJHT")); } + @Test(timeout = 60_000) + public void testTheModuleResolvesWithOnlyItsRequiredPlatformModules() throws Exception { + // java.base and jdk.unsupported are the only platform modules client code needs; org.slf4j is its one + // library dependency. A further mandatory requires is unobservable here and fails resolution. + String output = runOnModulePath(DesktopFreePromptMain.class, "java.base,jdk.unsupported", + "io.questdb.client,org.slf4j"); + Assert.assertTrue("the child never reached the end of the prompt:\n" + output, + output.contains(DesktopFreePromptMain.SUCCESS_MARKER)); + } + private static File locationOf(Class type) { CodeSource source = type.getProtectionDomain().getCodeSource(); Assert.assertNotNull("no code source for " + type.getName(), source); @@ -91,16 +109,29 @@ private static File locationOf(Class type) { } /** - * Runs {@code main} in a second JVM that has the client on the MODULE path and no {@code java.desktop}, - * and returns its merged stdout/stderr. Asserts a clean exit, so a child that failed its own - * preconditions reports through its printed reason rather than through a silent skip. + * Runs {@code main} in a second JVM that has the client on the MODULE path and no {@code java.desktop}: + * the universe is io.questdb.client and the closure of its MANDATORY requires, and nothing else. That + * closure is also what makes a mandatory {@code requires java.desktop} visible, because it would then + * include java.desktop. */ private static String runInDesktopFreeJvm(Class main) throws Exception { + return runOnModulePath(main, "io.questdb.client", "io.questdb.client"); + } + + /** + * Runs {@code main} in a second JVM that has the client on the MODULE path, the universe limited to + * {@code limitModules} and the closure of their mandatory requires plus the {@code rootModules}, and + * returns its merged stdout/stderr. A root is observable, but {@code --limit-modules} does not close over + * its requires. Asserts a clean exit, so a child that failed its own preconditions reports through its + * printed reason rather than through a silent skip. + */ + private static String runOnModulePath(Class main, String limitModules, String rootModules) + throws Exception { File clientLocation = locationOf(DeviceCodePrompt.class); Assume.assumeFalse("the module system arrived in Java 9; nothing to resolve on a Java 8 runtime", "1.8".equals(System.getProperty("java.specification.version"))); - // A JDK 8 build produces no module-info.class: the artifact is then an automatic module, which - // reads every observable module and is exactly the configuration this defect never reached. + // A JDK 8 build compiles no root module-info.class, so target/classes is no explicit module there: + // its descriptor goes into META-INF/versions/11 of the packaged jar only, after the tests run. Assume.assumeTrue("no module descriptor next to " + clientLocation + " (a JDK 8 build)", new File(clientLocation, "module-info.class").isFile()); @@ -116,12 +147,9 @@ private static String runInDesktopFreeJvm(Class main) throws Exception { // test - headless changes what Desktop ANSWERS, not whether the class reference links. "-Djava.awt.headless=true", "--module-path", clientLocation.getPath() + File.pathSeparator + slf4jLocation.getPath(), - // the universe: io.questdb.client and the closure of its MANDATORY requires, and nothing - // else. This is what makes the child desktop-free - and what makes a mandatory - // `requires java.desktop` visible, because the closure would then include it. - "--limit-modules", "io.questdb.client", + "--limit-modules", limitModules, // the main class runs from the class path, so the client is not a root by default - "--add-modules", "io.questdb.client", + "--add-modules", rootModules, "-classpath", testClasses.getPath(), main.getName()); // deliberately NOT setting questdb.client.oidc.open.browser here: the kill-switch returns before @@ -132,7 +160,7 @@ private static String runInDesktopFreeJvm(Class main) throws Exception { Process process = pb.start(); String output = readFully(process.getInputStream()); int exitCode = process.waitFor(); - Assert.assertEquals("the desktop-free module-path run of " + main.getSimpleName() + " failed:\n" + Assert.assertEquals("the module-path run of " + main.getSimpleName() + " failed:\n" + output, 0, exitCode); return output; } From b0fbaa720cea046719b4907ba3eaf982eb1348d6 Mon Sep 17 00:00:00 2001 From: glasstiger Date: Sat, 17 Oct 2026 17:01:29 +0100 Subject: [PATCH 32/40] Fix member order and a stale design note Members now follow the committed IDE arrangement. In RefreshingTokenProvider the ScheduledTask interface precedes Scheduler and the FetchFailure class follows DefaultScheduler. The test methods and helpers of QueryWorkerBusyShutdownTest and WebSocketDynamicCredentialTest are in name order. These are pure moves. The Entra ID design note still said the azure module has no module-info and relies on an automatic module. It ships a module-info that provides the factory and requires the Azure SDK modules, compiled into META-INF/versions/11 by JDK 8 release builds, and the note now says so. --- .../cutlass/auth/RefreshingTokenProvider.java | 108 ++++++++-------- .../WebSocketDynamicCredentialTest.java | 118 +++++++++--------- .../impl/QueryWorkerBusyShutdownTest.java | 20 +-- design/entra-id-qwp-auth.md | 2 +- 4 files changed, 124 insertions(+), 124 deletions(-) diff --git a/core/src/main/java/io/questdb/client/cutlass/auth/RefreshingTokenProvider.java b/core/src/main/java/io/questdb/client/cutlass/auth/RefreshingTokenProvider.java index 1f620fb15..133d38164 100644 --- a/core/src/main/java/io/questdb/client/cutlass/auth/RefreshingTokenProvider.java +++ b/core/src/main/java/io/questdb/client/cutlass/auth/RefreshingTokenProvider.java @@ -740,6 +740,16 @@ public interface Clock { long wallClockMillis(); } + /** + * Handle to a task submitted to a {@link Scheduler}. + */ + public interface ScheduledTask { + /** + * Best-effort: a task that has already started may still run, and the provider ignores it. + */ + void cancel(); + } + /** * Runs fetches on the provider's background context. The default owns one daemon thread. A test may supply * its own to run fetches deterministically. @@ -757,16 +767,6 @@ public interface Scheduler { void shutdown(); } - /** - * Handle to a task submitted to a {@link Scheduler}. - */ - public interface ScheduledTask { - /** - * Best-effort: a task that has already started may still run, and the provider ignores it. - */ - void cancel(); - } - /** * Builder for {@link RefreshingTokenProvider}. The defaults are the specification's (section 5.1). */ @@ -945,6 +945,50 @@ public Builder scheduler(Scheduler scheduler) { } } + private static final class DefaultScheduler implements Scheduler { + private final ScheduledThreadPoolExecutor executor; + private volatile Thread thread; + + DefaultScheduler() { + final String threadName = "qdb-token-refresh-" + THREAD_IDS.incrementAndGet(); + this.executor = new ScheduledThreadPoolExecutor(1, r -> { + Thread t = new Thread(r, threadName); + t.setDaemon(true); + thread = t; + return t; + }); + executor.setRemoveOnCancelPolicy(true); + executor.setExecuteExistingDelayedTasksAfterShutdownPolicy(false); + executor.setContinueExistingPeriodicTasksAfterShutdownPolicy(false); + } + + @Override + public ScheduledTask schedule(Runnable task, long delayNanos) { + final ScheduledFuture future = executor.schedule(task, delayNanos, TimeUnit.NANOSECONDS); + return () -> future.cancel(false); + } + + @Override + public void shutdown() { + executor.shutdownNow(); + if (Thread.currentThread() == thread) { + return; // closed from inside a fetch: the thread exits once this task unwinds + } + // Interrupt-neutral, like the other close paths in this library: a carried interrupt flag would + // turn the bounded wait into an immediate return. + final boolean interrupted = Thread.interrupted(); + try { + executor.awaitTermination(CLOSE_AWAIT_MILLIS, TimeUnit.MILLISECONDS); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } finally { + if (interrupted) { + Thread.currentThread().interrupt(); + } + } + } + } + /** * A failed fetch: its classification, sanitized message and time. Never carries the token. */ @@ -1001,50 +1045,6 @@ FetchFailure sanitized() { } } - private static final class DefaultScheduler implements Scheduler { - private final ScheduledThreadPoolExecutor executor; - private volatile Thread thread; - - DefaultScheduler() { - final String threadName = "qdb-token-refresh-" + THREAD_IDS.incrementAndGet(); - this.executor = new ScheduledThreadPoolExecutor(1, r -> { - Thread t = new Thread(r, threadName); - t.setDaemon(true); - thread = t; - return t; - }); - executor.setRemoveOnCancelPolicy(true); - executor.setExecuteExistingDelayedTasksAfterShutdownPolicy(false); - executor.setContinueExistingPeriodicTasksAfterShutdownPolicy(false); - } - - @Override - public ScheduledTask schedule(Runnable task, long delayNanos) { - final ScheduledFuture future = executor.schedule(task, delayNanos, TimeUnit.NANOSECONDS); - return () -> future.cancel(false); - } - - @Override - public void shutdown() { - executor.shutdownNow(); - if (Thread.currentThread() == thread) { - return; // closed from inside a fetch: the thread exits once this task unwinds - } - // Interrupt-neutral, like the other close paths in this library: a carried interrupt flag would - // turn the bounded wait into an immediate return. - final boolean interrupted = Thread.interrupted(); - try { - executor.awaitTermination(CLOSE_AWAIT_MILLIS, TimeUnit.MILLISECONDS); - } catch (InterruptedException e) { - Thread.currentThread().interrupt(); - } finally { - if (interrupted) { - Thread.currentThread().interrupt(); - } - } - } - } - private static final class Held { final long expiresAtEpochMillis; final long refreshAtEpochMillis; diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/WebSocketDynamicCredentialTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/WebSocketDynamicCredentialTest.java index 50c1b0e67..574411485 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/WebSocketDynamicCredentialTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/WebSocketDynamicCredentialTest.java @@ -443,56 +443,6 @@ public void testReconnectAfterExpiryCarriesTheRefreshedToken() throws Exception }); } - @Test(timeout = 30_000) - public void testStaleTokenGetsExactlyOne401ThenAnImmediateRetry() throws Exception { - // C10: the reconnect presents a token the server has revoked. Exactly one 401, then an immediate retry of - // the same endpoint with the refreshed token: no backoff, no failed-attempt event, no AUTH_FAILED. - assertMemoryLeak(() -> { - ScriptedSource source = new ScriptedSource() - .thenToken("T1", System.currentTimeMillis() + HOUR) - .thenToken("T2", System.currentTimeMillis() + HOUR); - DropAfterFirstAckHandler handler = new DropAfterFirstAckHandler(); - List events = new CopyOnWriteArrayList<>(); - Set revoked = Collections.synchronizedSet(new HashSet<>()); - try (TestWebSocketServer server = startServer(handler); - RefreshingTokenProvider provider = fastProvider(source).build()) { - server.setAuthorizationValidator(h -> revoked.contains(h) ? 401 : 0); - try (Sender sender = Sender.builder(Sender.Transport.WEBSOCKET) - .address("localhost:" + server.getPort()) - .initialConnectMode(Sender.InitialConnectMode.OFF) - // a retry through the reconnect loop would wait at least this long - .reconnectInitialBackoffMillis(5_000) - .reconnectMaxBackoffMillis(5_000) - .connectionListener(events::add) - .httpTokenProvider(provider) - .build()) { - Assert.assertEquals("Bearer T1", server.pollAuthorizationHeader(5, TimeUnit.SECONDS)); - revoked.add("Bearer T1"); - sender.table("t").longColumn("v", 1).atNow(); - sender.flush(); // ACKed, then the server drops the connection - Assert.assertEquals("the reconnect first presents the cached token", - "Bearer T1", server.pollAuthorizationHeader(5, TimeUnit.SECONDS)); - long rejectedAt = System.nanoTime(); - Assert.assertEquals("then retries with the refreshed one", - "Bearer T2", server.pollAuthorizationHeader(5, TimeUnit.SECONDS)); - long retryMillis = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - rejectedAt); - Assert.assertTrue("the retry must be immediate, not after backoff; took " + retryMillis + " ms", - retryMillis < 4_000); - sender.table("t").longColumn("v", 2).atNow(); - sender.flush(); - await(() -> handler.frames.get() >= 2, 10_000, "the second row"); - await(() -> kinds(events).contains(SenderConnectionEvent.Kind.RECONNECTED), 5_000, "RECONNECTED"); - } - Assert.assertEquals("exactly one 401", 1, server.authRejectCount()); - List kinds = kinds(events); - Assert.assertFalse("only the round's final outcome is reported: " + kinds, - kinds.contains(SenderConnectionEvent.Kind.AUTH_FAILED)); - Assert.assertFalse("the retried 401 is not an endpoint failure: " + kinds, - kinds.contains(SenderConnectionEvent.Kind.ENDPOINT_ATTEMPT_FAILED)); - } - }); - } - @Test(timeout = 60_000) public void testSentinelTokensNeverLeakFromTheClient() throws Exception { // C20 at the client level: across a stale-token 401 and its retry (C10), a persistent 401 (C11), a cold @@ -595,6 +545,56 @@ public void testSentinelTokensNeverLeakFromTheClient() throws Exception { } } + @Test(timeout = 30_000) + public void testStaleTokenGetsExactlyOne401ThenAnImmediateRetry() throws Exception { + // C10: the reconnect presents a token the server has revoked. Exactly one 401, then an immediate retry of + // the same endpoint with the refreshed token: no backoff, no failed-attempt event, no AUTH_FAILED. + assertMemoryLeak(() -> { + ScriptedSource source = new ScriptedSource() + .thenToken("T1", System.currentTimeMillis() + HOUR) + .thenToken("T2", System.currentTimeMillis() + HOUR); + DropAfterFirstAckHandler handler = new DropAfterFirstAckHandler(); + List events = new CopyOnWriteArrayList<>(); + Set revoked = Collections.synchronizedSet(new HashSet<>()); + try (TestWebSocketServer server = startServer(handler); + RefreshingTokenProvider provider = fastProvider(source).build()) { + server.setAuthorizationValidator(h -> revoked.contains(h) ? 401 : 0); + try (Sender sender = Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + server.getPort()) + .initialConnectMode(Sender.InitialConnectMode.OFF) + // a retry through the reconnect loop would wait at least this long + .reconnectInitialBackoffMillis(5_000) + .reconnectMaxBackoffMillis(5_000) + .connectionListener(events::add) + .httpTokenProvider(provider) + .build()) { + Assert.assertEquals("Bearer T1", server.pollAuthorizationHeader(5, TimeUnit.SECONDS)); + revoked.add("Bearer T1"); + sender.table("t").longColumn("v", 1).atNow(); + sender.flush(); // ACKed, then the server drops the connection + Assert.assertEquals("the reconnect first presents the cached token", + "Bearer T1", server.pollAuthorizationHeader(5, TimeUnit.SECONDS)); + long rejectedAt = System.nanoTime(); + Assert.assertEquals("then retries with the refreshed one", + "Bearer T2", server.pollAuthorizationHeader(5, TimeUnit.SECONDS)); + long retryMillis = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - rejectedAt); + Assert.assertTrue("the retry must be immediate, not after backoff; took " + retryMillis + " ms", + retryMillis < 4_000); + sender.table("t").longColumn("v", 2).atNow(); + sender.flush(); + await(() -> handler.frames.get() >= 2, 10_000, "the second row"); + await(() -> kinds(events).contains(SenderConnectionEvent.Kind.RECONNECTED), 5_000, "RECONNECTED"); + } + Assert.assertEquals("exactly one 401", 1, server.authRejectCount()); + List kinds = kinds(events); + Assert.assertFalse("only the round's final outcome is reported: " + kinds, + kinds.contains(SenderConnectionEvent.Kind.AUTH_FAILED)); + Assert.assertFalse("the retried 401 is not an endpoint failure: " + kinds, + kinds.contains(SenderConnectionEvent.Kind.ENDPOINT_ATTEMPT_FAILED)); + } + }); + } + @Test(timeout = 30_000) public void testStaticCredentialNeverGetsTheRetry() throws Exception { // section 8.2: a static credential never gets the retry - presenting the same bytes again cannot help. @@ -814,15 +814,6 @@ public void testSyncInitializationStopsRetryingWhenTheThreadIsInterrupted() thro }); } - private static void renderThrowable(Throwable t, List into) { - for (Throwable c = t; c != null; c = c.getCause()) { - into.add(c.toString()); - for (Throwable s : c.getSuppressed()) { - renderThrowable(s, into); - } - } - } - private static List drainHeaders(TestWebSocketServer server) throws InterruptedException { List headers = new ArrayList<>(); String h; @@ -863,6 +854,15 @@ private static String lastHeader(TestWebSocketServer server) throws InterruptedE return last; } + private static void renderThrowable(Throwable t, List into) { + for (Throwable c = t; c != null; c = c.getCause()) { + into.add(c.toString()); + for (Throwable s : c.getSuppressed()) { + renderThrowable(s, into); + } + } + } + private static TestWebSocketServer startServer(TestWebSocketServer.WebSocketServerHandler handler) throws IOException, InterruptedException { TestWebSocketServer server = new TestWebSocketServer(handler); diff --git a/core/src/test/java/io/questdb/client/test/impl/QueryWorkerBusyShutdownTest.java b/core/src/test/java/io/questdb/client/test/impl/QueryWorkerBusyShutdownTest.java index cb264007a..3872e0aab 100644 --- a/core/src/test/java/io/questdb/client/test/impl/QueryWorkerBusyShutdownTest.java +++ b/core/src/test/java/io/questdb/client/test/impl/QueryWorkerBusyShutdownTest.java @@ -68,6 +68,16 @@ public class QueryWorkerBusyShutdownTest { private static final String SENDER_CFG = "http::addr=127.0.0.1:1;protocol_version=2;auto_flush=off;"; + @Test(timeout = 60_000) + public void testShutdownDuringAReconnectWalkWithBinds() throws Exception { + assertShutdownDuringAReconnectWalkIsSafe(true); + } + + @Test(timeout = 60_000) + public void testShutdownDuringAReconnectWalkWithoutBinds() throws Exception { + assertShutdownDuringAReconnectWalkIsSafe(false); + } + @Test(timeout = 60_000) public void testShutdownLeavesTheCloseToADispatchThreadStillInsideExecute() throws Exception { TestUtils.assertMemoryLeak(() -> { @@ -119,16 +129,6 @@ public void onExecDone(short opType, long rowsAffected) { }); } - @Test(timeout = 60_000) - public void testShutdownDuringAReconnectWalkWithBinds() throws Exception { - assertShutdownDuringAReconnectWalkIsSafe(true); - } - - @Test(timeout = 60_000) - public void testShutdownDuringAReconnectWalkWithoutBinds() throws Exception { - assertShutdownDuringAReconnectWalkIsSafe(false); - } - /** * The failure from the review: a pooled client whose failover reconnect failed reconnects on its next * execute(). The walk stalls on an endpoint that accepts TCP but never answers the upgrade, outlasting the diff --git a/design/entra-id-qwp-auth.md b/design/entra-id-qwp-auth.md index 793e640ef..5a8faecd9 100644 --- a/design/entra-id-qwp-auth.md +++ b/design/entra-id-qwp-auth.md @@ -589,7 +589,7 @@ Every step of §10 is implemented. Line references in §1-§7 predate it. - `AzureTokenProviderFactory` uses `DefaultAzureCredential` for `azure_credential=default`, where `azure_client_id` sets both the managed-identity and workload-identity client ID, and warns once when that chain starts. The other values build `ManagedIdentityCredential`, `WorkloadIdentityCredential` or `EnvironmentCredential` directly (spec Appendix B); a credential that cannot be built for lack of configuration fails every fetch as permanent instead of failing a client's `build()`. - `AzureTokenSource` bounds each request at 30 s and classifies per §7.5: `CredentialUnavailableException`, a managed identity that is not assigned (reported only in a nested cause when that credential is used alone), HTTP 400/401 and known AADSTS configuration codes are permanent; everything else is retryable, with `Retry-After` honoured, and names the socket-level cause of a network failure. - It never attaches a library exception: its response references the raw request. - - Java 8 bytecode. It has no `module-info`; `Automatic-Module-Name: io.questdb.client.azure`, and an automatic module provides its `META-INF/services`. + - Java 8 bytecode. Its `module-info`, which `provides` the factory and requires the Azure SDK modules, is compiled into `META-INF/versions/11` by JDK 8 release builds and into the root by JDK 11+ builds. On Java 11+ the adapter is therefore an explicit module, which service binding resolves for an application that requires only `io.questdb.client`. Class-path consumers find the factory through `META-INF/services`. - Its release profiles mirror core's. - Tests: `TokenProviderConfigTest` (C18, C19, the connect-string half of C24), `TokenProviderSharingTest` (C17, over TLS), `azure/AzureTokenSourceTest` (fake `TokenCredential`, no network), `azure/AzureCredentialSelectionTest` (the real factory, no network) and `azure/AzureManagedIdentityLibraryTest` (C24: the real library in a child JVM, against a local IMDS stub). From add519a0752566374cb811ebc3e0d42e8152a7a9 Mon Sep 17 00:00:00 2001 From: glasstiger Date: Sat, 17 Oct 2026 23:24:52 +0100 Subject: [PATCH 33/40] Test the 401 retry's no-retry branches Two branches of the one retry after a 401 (spec section 8.2) had no test. With either one broken, the full core suite still passed: - The endpoint is retried only with a token that differs from the rejected one. QwpWebSocketSender.connectWalk and the query client's connectWalk and reconnectViaTracker check this. Without the check, a provider that hands the same token back, as an Azure managed identity usually does, has the server reject it a second time in every connect round. - When the pull after the 401 fails, the round ends with the 401 and carries the provider's failure as a suppressed diagnostic. If that failure escaped instead, an OFF startup would throw the provider's own exception, a SYNC startup would retry it for its whole budget, and the query client would report credential-unavailable. WebSocketDynamicCredentialTest gains testUnchangedTokenAfterA401IsNotRetried and testFailedRefreshAfterA401EndsStartupWithTheRejection, each run in OFF and in SYNC mode. QwpQueryClientDynamicCredentialTest gains the connect and failover versions of the first and the connect version of the second. TokenTestKit gains ForgetfulTokenProvider, an application provider that drops its cached token once a server rejects it and then cannot reach its identity platform. Removing each check on its own fails exactly the test that pins it: the unchanged-token tests see "Bearer T1" presented twice, and the failed-refresh tests get TokenUnavailableException (sender) or QwpCredentialUnavailableException (query client) instead of QwpAuthFailedException. The new tests pass on JDK 25 and on JDK 8 (-Xint), and the ten affected test classes (109 tests) pass on JDK 25. --- .../test/cutlass/auth/TokenTestKit.java | 46 ++++++++++- .../QwpQueryClientDynamicCredentialTest.java | 79 +++++++++++++++++++ .../WebSocketDynamicCredentialTest.java | 72 +++++++++++++++++ 3 files changed, 196 insertions(+), 1 deletion(-) diff --git a/core/src/test/java/io/questdb/client/test/cutlass/auth/TokenTestKit.java b/core/src/test/java/io/questdb/client/test/cutlass/auth/TokenTestKit.java index 24c600a54..4d093021c 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/auth/TokenTestKit.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/auth/TokenTestKit.java @@ -24,9 +24,11 @@ package io.questdb.client.test.cutlass.auth; +import io.questdb.client.HttpTokenProvider; import io.questdb.client.cutlass.auth.ExpiringToken; import io.questdb.client.cutlass.auth.RefreshingTokenProvider; import io.questdb.client.cutlass.auth.TokenSource; +import io.questdb.client.cutlass.auth.TokenUnavailableException; import org.junit.Assert; import java.util.ArrayDeque; @@ -41,7 +43,8 @@ /** * Deterministic fixtures for the dynamic-credential conformance tests (design/qwp-token-provider-spec.md, section - * 10): a fake clock, a scheduler the test drives by hand, and a scripted token source. + * 10): a fake clock, a scheduler the test drives by hand, a scripted token source, and an application-supplied + * provider that forgets a token once a server rejects it. */ public final class TokenTestKit { @@ -96,6 +99,47 @@ public long wallClockMillis() { } } + /** + * An application-supplied provider that hands out its cached token until a server rejects it, then drops the + * cache and fails every later pull with {@link #outage()}, as if its identity platform had become unreachable. + */ + public static final class ForgetfulTokenProvider implements HttpTokenProvider { + private final TokenUnavailableException outage = + TokenUnavailableException.retryable("identity platform unreachable"); + private final AtomicInteger pulls = new AtomicInteger(); + private final String token; + private volatile boolean hasCachedToken = true; + + public ForgetfulTokenProvider(String token) { + this.token = token; + } + + @Override + public CharSequence getToken() { + pulls.incrementAndGet(); + if (!hasCachedToken) { + throw outage; + } + return token; + } + + @Override + public void onTokenRejected(CharSequence token, int httpStatus) { + hasCachedToken = false; + } + + /** + * The exception every pull throws once the cached token is gone. + */ + public TokenUnavailableException outage() { + return outage; + } + + public int pulls() { + return pulls.get(); + } + } + /** * Runs nothing on its own: the test picks the next task, optionally advances the fake clock to its due time, * and runs it on the test thread. Tasks run outside the scheduler's monitor, because the provider calls diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/QwpQueryClientDynamicCredentialTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/QwpQueryClientDynamicCredentialTest.java index d27ede3da..d849121ed 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/QwpQueryClientDynamicCredentialTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/QwpQueryClientDynamicCredentialTest.java @@ -36,6 +36,7 @@ import io.questdb.client.cutlass.qwp.client.QwpEgressMsgKind; import io.questdb.client.cutlass.qwp.client.QwpQueryClient; import io.questdb.client.cutlass.qwp.protocol.QwpConstants; +import io.questdb.client.test.cutlass.auth.TokenTestKit.ForgetfulTokenProvider; import io.questdb.client.test.cutlass.auth.TokenTestKit.ScriptedSource; import io.questdb.client.test.cutlass.qwp.websocket.TestWebSocketServer; import org.junit.Assert; @@ -46,6 +47,7 @@ import java.nio.ByteOrder; import java.util.ArrayList; import java.util.Arrays; +import java.util.Collections; import java.util.List; import java.util.concurrent.TimeUnit; import java.util.concurrent.atomic.AtomicInteger; @@ -84,6 +86,32 @@ public void testConnectCredentialUnavailableNamesTheClassAndContactsNoEndpoint() }); } + @Test(timeout = 30_000) + public void testConnectFailedRefreshAfterA401EndsWithTheRejection() throws Exception { + // Section 8.2 on egress: when the pull after a 401 fails, there is no new token to retry with. connect() ends + // with the 401, which carries the credential-unavailable failure as a suppressed diagnostic. + assertMemoryLeak(() -> { + ForgetfulTokenProvider provider = new ForgetfulTokenProvider("T1"); + try (TestWebSocketServer server = startServer(); + QwpQueryClient client = QwpQueryClient.fromConfig("ws::addr=localhost:" + server.getPort() + ";") + .withBearerTokenProvider(provider)) { + server.setAuthorizationValidator(h -> 401); + try { + client.connect(); + Assert.fail(); + } catch (QwpAuthFailedException e) { + Assert.assertEquals(401, e.getStatusCode()); + Throwable[] suppressed = e.getSuppressed(); + Assert.assertEquals(Arrays.toString(suppressed), 1, suppressed.length); + Assert.assertTrue(suppressed[0].toString(), suppressed[0] instanceof QwpCredentialUnavailableException); + Assert.assertSame("the provider's own failure", provider.outage(), suppressed[0].getCause()); + } + Assert.assertEquals("one pull for the attempt, one after the 401", 2, provider.pulls()); + Assert.assertEquals("nothing to retry with", 1, server.upgradeRequestCount()); + } + }); + } + @Test(timeout = 30_000) public void testConnectInsufficientScopeIsNotRetried() throws Exception { // C23 on egress: a Bearer challenge with an error other than invalid_token gets no retry. @@ -172,6 +200,29 @@ public void testConnectStaleTokenIsRetriedOnceWithTheRefreshedToken() throws Exc }); } + @Test(timeout = 30_000) + public void testConnectUnchangedTokenAfterA401IsNotRetried() throws Exception { + // Section 8.2 step 4 on egress: the forced refresh hands the rejected token back, as a platform that caches + // tokens usually does, so connect() ends with the 401 instead of presenting that token again. + assertMemoryLeak(() -> { + ScriptedSource source = new ScriptedSource().thenToken("T1", System.currentTimeMillis() + HOUR); + try (TestWebSocketServer server = startServer(); + RefreshingTokenProvider provider = RefreshingTokenProvider.builder(source).build(); + QwpQueryClient client = QwpQueryClient.fromConfig("ws::addr=localhost:" + server.getPort() + ";") + .withBearerTokenProvider(provider)) { + server.setAuthorizationValidator(h -> 401); + try { + client.connect(); + Assert.fail(); + } catch (QwpAuthFailedException e) { + Assert.assertEquals(401, e.getStatusCode()); + } + Assert.assertEquals("the prefetch, then the forced refresh", 2, source.calls()); + Assert.assertEquals(Collections.singletonList("Bearer T1"), headers(server)); + } + }); + } + @Test(timeout = 30_000) public void testFailedFailoverReconnectRecoversOnTheNextExecute() throws Exception { // Egress recovery (section 8.3): a failover reconnect that fails must not leave the client permanently @@ -325,6 +376,34 @@ public void testFailoverReconnectRetriesAStaleTokenOnce() throws Exception { }); } + @Test(timeout = 30_000) + public void testFailoverUnchangedTokenAfterA401IsNotRetried() throws Exception { + // Section 8.2 step 4 on the egress failover path: B rejects the token and the forced refresh hands the same + // one back, so the failover reconnect ends with the 401 instead of presenting that token to B again. + assertMemoryLeak(() -> { + ScriptedSource source = new ScriptedSource().thenToken("T1", System.currentTimeMillis() + HOUR); + try (TestWebSocketServer a = startServer(); + TestWebSocketServer b = startServer(); + RefreshingTokenProvider provider = RefreshingTokenProvider.builder(source).build(); + QwpQueryClient client = QwpQueryClient.fromConfig( + "ws::addr=localhost:" + a.getPort() + ",localhost:" + b.getPort() + + ";failover_backoff_initial_ms=0;") + .withBearerTokenProvider(provider)) { + b.setAuthorizationValidator(h -> 401); + client.connect(); + Assert.assertEquals("Bearer T1", a.pollAuthorizationHeader(5, TimeUnit.SECONDS)); + a.close(); + ResultCollector result = new ResultCollector(); + client.execute("SELECT 1", result); + Assert.assertNotNull("the failover reconnect must fail", result.error.get()); + Assert.assertTrue(result.error.get(), + result.error.get().contains("auth-rejected during failover reconnect")); + Assert.assertEquals(Collections.singletonList("Bearer T1"), headers(b)); + Assert.assertEquals("the prefetch, then the forced refresh", 2, source.calls()); + } + }); + } + @Test(timeout = 60_000) public void testPooledQueryClientRecoversAfterAFailedFailover() throws Exception { // The suspected dead-pooled-worker bug (design/entra-id-qwp-auth.md, section 3): a pooled query client diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/WebSocketDynamicCredentialTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/WebSocketDynamicCredentialTest.java index 574411485..49f44e405 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/WebSocketDynamicCredentialTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/WebSocketDynamicCredentialTest.java @@ -34,6 +34,7 @@ import io.questdb.client.cutlass.qwp.client.QwpAuthFailedException; import io.questdb.client.cutlass.qwp.client.sf.cursor.OrphanScanner; import io.questdb.client.test.cutlass.auth.TokenTestKit.FakeClock; +import io.questdb.client.test.cutlass.auth.TokenTestKit.ForgetfulTokenProvider; import io.questdb.client.test.cutlass.auth.TokenTestKit.ManualScheduler; import io.questdb.client.test.cutlass.auth.TokenTestKit.ScriptedSource; import io.questdb.client.test.cutlass.qwp.websocket.TestWebSocketServer; @@ -168,6 +169,42 @@ public void testCloseInterruptsThePost401RejectionCallback() throws Exception { assertCloseInterruptsPost401Wait(true); } + @Test(timeout = 30_000) + public void testFailedRefreshAfterA401EndsStartupWithTheRejection() throws Exception { + // Section 8.2: when the pull after a 401 fails, there is no new token to retry with. The round ends with the + // 401, which carries the provider's failure as a suppressed diagnostic, and startup fails fast in OFF and in + // SYNC mode rather than retrying a credential outage. + for (Sender.InitialConnectMode mode : new Sender.InitialConnectMode[]{ + Sender.InitialConnectMode.OFF, Sender.InitialConnectMode.SYNC}) { + assertMemoryLeak(() -> { + ForgetfulTokenProvider provider = new ForgetfulTokenProvider("T1"); + try (TestWebSocketServer server = startServer(new AckHandler())) { + server.setAuthorizationValidator(h -> 401); + long start = System.nanoTime(); + try { + Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + server.getPort()) + .initialConnectMode(mode) + .reconnectMaxDurationMillis(20_000) + .httpTokenProvider(provider) + .build() + .close(); + Assert.fail("a 401 must fail startup [mode=" + mode + ']'); + } catch (QwpAuthFailedException e) { + Assert.assertEquals(401, e.getStatusCode()); + Assert.assertTrue("the provider's failure must ride along [mode=" + mode + ']', + Arrays.asList(e.getSuppressed()).contains(provider.outage())); + } + long elapsedMillis = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - start); + Assert.assertTrue("startup must fail fast, took " + elapsedMillis + " ms", elapsedMillis < 10_000); + Assert.assertEquals("one pull for the attempt, one after the 401 [mode=" + mode + ']', + 2, provider.pulls()); + Assert.assertEquals("nothing to retry with [mode=" + mode + ']', 1, server.upgradeRequestCount()); + } + }); + } + } + @Test(timeout = 60_000) public void testFailoverAcrossARotationSendsTheNewTokenAndReplaysOnce() throws Exception { // C14: connected to A with T1, the token rotates, A dies. B receives the NEW token, and the frames A @@ -814,6 +851,41 @@ public void testSyncInitializationStopsRetryingWhenTheThreadIsInterrupted() thro }); } + @Test(timeout = 30_000) + public void testUnchangedTokenAfterA401IsNotRetried() throws Exception { + // Section 8.2 step 4: the endpoint is retried only with a token that differs from the rejected one. Here + // the forced refresh hands the same token back, as a platform that caches tokens - an Azure managed + // identity - usually does, so the 401 ends the round in OFF and in SYNC mode. + for (Sender.InitialConnectMode mode : new Sender.InitialConnectMode[]{ + Sender.InitialConnectMode.OFF, Sender.InitialConnectMode.SYNC}) { + assertMemoryLeak(() -> { + ScriptedSource source = new ScriptedSource().thenToken("T1", System.currentTimeMillis() + HOUR); + try (TestWebSocketServer server = startServer(new AckHandler()); + RefreshingTokenProvider provider = fastProvider(source).build()) { + server.setAuthorizationValidator(h -> 401); + long start = System.nanoTime(); + try { + Sender.builder(Sender.Transport.WEBSOCKET) + .address("localhost:" + server.getPort()) + .initialConnectMode(mode) + .reconnectMaxDurationMillis(20_000) + .httpTokenProvider(provider) + .build() + .close(); + Assert.fail("a 401 must fail startup [mode=" + mode + ']'); + } catch (QwpAuthFailedException e) { + Assert.assertEquals(401, e.getStatusCode()); + } + long elapsedMillis = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - start); + Assert.assertTrue("startup must fail fast, took " + elapsedMillis + " ms", elapsedMillis < 10_000); + Assert.assertEquals("the prefetch, then the forced refresh [mode=" + mode + ']', 2, source.calls()); + Assert.assertEquals("the rejected token must not be presented again [mode=" + mode + ']', + Collections.singletonList("Bearer T1"), drainHeaders(server)); + } + }); + } + } + private static List drainHeaders(TestWebSocketServer server) throws InterruptedException { List headers = new ArrayList<>(); String h; From 471f060adcd988df8c542728ff78f7b04a2170b3 Mon Sep 17 00:00:00 2001 From: glasstiger Date: Sun, 18 Oct 2026 02:33:21 +0100 Subject: [PATCH 34/40] Reject non-ASCII Azure client IDs Restrict Azure client ID GUID validation to ASCII hexadecimal digits. Unicode digit and letter look-alikes must fail configuration validation instead of reaching the Azure credential builders. Add regression coverage for fullwidth digits, Arabic-Indic digits, and fullwidth uppercase and lowercase hex letters on both client entry points. The new test fails without the fix, and all 11 configuration tests pass with it on JDK 17. JDK 8 validation was unavailable locally. --- .../client/cutlass/auth/TokenProviderSpec.java | 5 +++-- .../test/cutlass/auth/TokenProviderConfigTest.java | 11 +++++++++++ 2 files changed, 14 insertions(+), 2 deletions(-) diff --git a/core/src/main/java/io/questdb/client/cutlass/auth/TokenProviderSpec.java b/core/src/main/java/io/questdb/client/cutlass/auth/TokenProviderSpec.java index 1c18a4470..8cd0d360b 100644 --- a/core/src/main/java/io/questdb/client/cutlass/auth/TokenProviderSpec.java +++ b/core/src/main/java/io/questdb/client/cutlass/auth/TokenProviderSpec.java @@ -204,8 +204,9 @@ static String normalizeGuid(String value) { } for (int i = 0; i < 36; i++) { char c = value.charAt(i); - boolean dash = i == 8 || i == 13 || i == 18 || i == 23; - if (dash ? c != '-' : Character.digit(c, 16) < 0) { + boolean isDash = i == 8 || i == 13 || i == 18 || i == 23; + boolean isHex = (c >= '0' && c <= '9') || (c >= 'a' && c <= 'f') || (c >= 'A' && c <= 'F'); + if (isDash ? c != '-' : !isHex) { throw invalidClientId(value); } } diff --git a/core/src/test/java/io/questdb/client/test/cutlass/auth/TokenProviderConfigTest.java b/core/src/test/java/io/questdb/client/test/cutlass/auth/TokenProviderConfigTest.java index 69d78a0e2..f8d6d9cec 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/auth/TokenProviderConfigTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/auth/TokenProviderConfigTest.java @@ -65,6 +65,17 @@ public void tearDown() { TestTokenProviderFactory.uninstall(); } + @Test + public void testAzureClientIdRejectsNonAsciiHex() throws Exception { + assertMemoryLeak(() -> { + for (char c : new char[]{'\uFF10', '\u0660', '\uFF21', '\uFF41'}) { + assertRejectedOnBoth(WSS + "token_provider=azure;azure_resource=api://x;azure_client_id=" + + c + "0000000-0000-0000-0000-000000000000;", + "invalid azure_client_id: expected a GUID"); + } + }); + } + @Test public void testAzureCredentialSelectsOneCredential() throws Exception { // C24, connect-string half: every value of section 7.1 is accepted by every entry point, reaches the factory, From 4c8ff95fbc969de91dfd9d1484bf1f7db543ad32 Mon Sep 17 00:00:00 2001 From: glasstiger Date: Mon, 19 Oct 2026 01:32:54 +0100 Subject: [PATCH 35/40] Fail fast on malformed Azure certificates Classify Azure Identity's certificate-loading failures as permanent. Recognize the SDK's specific RuntimeException prefix without retaining its message or cause. Keep more-specific missing-file diagnostics ahead of the parsing fallback and leave unrelated I/O failures retryable. Add real-SDK regression tests for malformed and missing certificates, plus synchronous startup through RefreshingTokenProvider. Reuse the core memory-leak checker through a test-only dependency. Add unit coverage for wrapped parsing failures, safe diagnostics and unrelated EOF errors. Validation: - The regression tests fail before the fix and pass after it. - JDK 25 passes 100 focused tests and two module-path integration tests. - JDK 8 passes all 39 Azure tests. The wider local Zulu 8 core run also hits SIGSEGV with unchanged core code at 471f060a. All 61 focused core tests pass with -Xint. --- azure/pom.xml | 7 + .../client/azure/AzureTokenSource.java | 16 ++- ...lientCertificateCredentialLibraryTest.java | 122 ++++++++++++++++++ .../azure/test/AzureTokenSourceTest.java | 15 +++ 4 files changed, 158 insertions(+), 2 deletions(-) create mode 100644 azure/src/test/java/io/questdb/client/azure/test/AzureClientCertificateCredentialLibraryTest.java diff --git a/azure/pom.xml b/azure/pom.xml index 34b3eafa2..76ff3b8ae 100644 --- a/azure/pom.xml +++ b/azure/pom.xml @@ -100,6 +100,13 @@ + + org.questdb + questdb-client + ${project.version} + test-jar + test + junit junit diff --git a/azure/src/main/java/io/questdb/client/azure/AzureTokenSource.java b/azure/src/main/java/io/questdb/client/azure/AzureTokenSource.java index a9287c486..d120c8e38 100644 --- a/azure/src/main/java/io/questdb/client/azure/AzureTokenSource.java +++ b/azure/src/main/java/io/questdb/client/azure/AzureTokenSource.java @@ -65,8 +65,9 @@ * 7.5 prescribes, the same way whichever credential is used: *

      *
    • permanent - no credential available ({@link CredentialUnavailableException}): a credential chain that - * found nothing, or a credential whose configuration or credential file is missing; a managed identity that is - * not assigned to this host, which a managed-identity credential used alone reports only in a nested cause; and + * found nothing, or a credential whose configuration or credential file is missing or whose certificate cannot + * be parsed; a managed identity that is not assigned to this host, which a managed-identity credential used alone + * reports only in a nested cause; and * authentication failures Entra reports: an HTTP 400 or 401 from the token endpoint, or a known AADSTS * configuration error such as an invalid client or an application that does not exist;
      • *
    • retryable - network errors, timeouts, throttling, 5xx responses, and anything else. A @@ -83,6 +84,9 @@ public final class AzureTokenSource implements TokenSource { * The bound on one token request. */ public static final Duration DEFAULT_TIMEOUT = Duration.ofSeconds(30); + // Azure Identity wraps certificate-loading I/O/security failures in a plain RuntimeException with this prefix. + // Recognize that failure without treating unrelated I/O errors as permanent or exposing the SDK message. + private static final String CERTIFICATE_PARSE_FAILURE = "Failed to parse the certificate for the credential:"; private static final String DEFAULT_SCOPE_SUFFIX = "/.default"; // A managed identity that is not assigned to this host, or does not exist: IMDS answers 400 "Identity not // found". Inside a chain that arrives as a CredentialUnavailableException, but a ManagedIdentityCredential used @@ -149,6 +153,7 @@ public AzureTokenSource(TokenCredential credential, String resource, Duration ti */ static TokenUnavailableException classify(Throwable failure, String scope, boolean chain) { final String description = describe(failure); + boolean hasCertificateParseFailure = false; long retryAfterMillis = TokenUnavailableException.NO_RETRY_AFTER; Throwable t = failure; for (int depth = 0; t != null && depth < MAX_CAUSE_DEPTH; depth++, t = t.getCause()) { @@ -174,6 +179,9 @@ static TokenUnavailableException classify(Throwable failure, String scope, boole } final String message = t.getMessage(); if (message != null) { + if (t.getClass() == RuntimeException.class && message.startsWith(CERTIFICATE_PARSE_FAILURE)) { + hasCertificateParseFailure = true; + } for (String code : PERMANENT_AADSTS) { if (message.contains(code)) { return TokenUnavailableException.permanent( @@ -199,6 +207,10 @@ static TokenUnavailableException classify(Throwable failure, String scope, boole } } } + if (hasCertificateParseFailure) { + // Keep more specific nested reasons, such as NoSuchFileException, ahead of the parsing fallback. + return TokenUnavailableException.permanent("credential certificate could not be parsed for " + scope); + } return TokenUnavailableException.retryable("the token request for " + scope + " failed: " + withNetworkCause(failure, description), retryAfterMillis); } diff --git a/azure/src/test/java/io/questdb/client/azure/test/AzureClientCertificateCredentialLibraryTest.java b/azure/src/test/java/io/questdb/client/azure/test/AzureClientCertificateCredentialLibraryTest.java new file mode 100644 index 000000000..a93bde8dd --- /dev/null +++ b/azure/src/test/java/io/questdb/client/azure/test/AzureClientCertificateCredentialLibraryTest.java @@ -0,0 +1,122 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.azure.test; + +import com.azure.identity.ClientCertificateCredentialBuilder; +import io.questdb.client.Sender; +import io.questdb.client.azure.AzureTokenSource; +import io.questdb.client.cutlass.auth.RefreshingTokenProvider; +import io.questdb.client.cutlass.auth.TokenUnavailableException; +import org.junit.Assert; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.TemporaryFolder; + +import java.io.File; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; + +import static io.questdb.client.test.tools.TestUtils.assertMemoryLeak; + +/** + * Exercises local certificate failures through the pinned Azure Identity credential without network I/O. + */ +public class AzureClientCertificateCredentialLibraryTest { + @Rule + public final TemporaryFolder temp = TemporaryFolder.builder().assureDeletion().build(); + + @Test(timeout = 30_000) + public void testMalformedCertificateIsPermanentWhenFetched() throws Exception { + AzureTokenSource source = source(malformedCertificate()); + TokenUnavailableException e = expectFailure(source::fetchToken); + Assert.assertFalse(e.getMessage(), e.isRetryable()); + Assert.assertEquals("credential certificate could not be parsed for api://qdb/.default", e.getMessage()); + Assert.assertEquals(TokenUnavailableException.NO_RETRY_AFTER, e.getRetryAfterMillis()); + Assert.assertNull("do not retain the SDK exception, file path or contents", e.getCause()); + } + + @Test(timeout = 30_000) + public void testMissingCertificateKeepsItsDiagnostic() throws Exception { + File certificate = new File(temp.newFolder(), "CERTIFICATE-PATH-SENTINEL.pem"); + TokenUnavailableException e = expectFailure(source(certificate)::fetchToken); + Assert.assertFalse(e.getMessage(), e.isRetryable()); + Assert.assertEquals("credential file is missing for api://qdb/.default", e.getMessage()); + Assert.assertNull("do not retain the SDK exception or file path", e.getCause()); + } + + @Test(timeout = 30_000) + public void testSyncStartupFailsFastOnMalformedCertificate() throws Exception { + assertMemoryLeak(() -> { + AzureTokenSource source = source(malformedCertificate()); + try (RefreshingTokenProvider provider = RefreshingTokenProvider.builder(source) + .coldWaitMillis(10_000).backoffInitialMillis(60_000).backoffMaxMillis(60_000).build()) { + // Finish the first SDK attempt before startup, so the cold-wait deadline cannot hide its result. + expectFailure(provider::getToken); + TokenUnavailableException e = expectFailure(() -> Sender.builder(Sender.Transport.WEBSOCKET) + .address("127.0.0.1:1") + .httpTokenProvider(provider) + .initialConnectMode(Sender.InitialConnectMode.SYNC) + .reconnectMaxDurationMillis(200) + .reconnectInitialBackoffMillis(10) + .reconnectMaxBackoffMillis(20) + .build() + .close()); + // Budget exhaustion throws LineSenderException instead, so this asserts the fail-fast path. + Assert.assertFalse(e.getMessage(), e.isRetryable()); + Assert.assertTrue(e.getMessage(), e.getMessage().contains( + "credential certificate could not be parsed for api://qdb/.default")); + Assert.assertFalse(provider.getLastFailure().isRetryable()); + } + }); + } + + private static TokenUnavailableException expectFailure(Runnable action) { + try { + action.run(); + Assert.fail("expected the credential to fail"); + return null; + } catch (TokenUnavailableException e) { + return e; + } + } + + private static AzureTokenSource source(File certificate) { + return new AzureTokenSource(new ClientCertificateCredentialBuilder() + .tenantId("aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee") + .clientId("11111111-2222-3333-4444-555555555555") + .pemCertificate(certificate.getAbsolutePath()) + .disableInstanceDiscovery() + .httpClient(request -> { + throw new AssertionError("certificate parsing must fail before any HTTP request"); + }) + .build(), "api://qdb"); + } + + private File malformedCertificate() throws Exception { + File certificate = temp.newFile("CERTIFICATE-PATH-SENTINEL.pem"); + Files.write(certificate.toPath(), "not a certificate: CERTIFICATE-CONTENTS-SENTINEL".getBytes(StandardCharsets.UTF_8)); + return certificate; + } +} diff --git a/azure/src/test/java/io/questdb/client/azure/test/AzureTokenSourceTest.java b/azure/src/test/java/io/questdb/client/azure/test/AzureTokenSourceTest.java index 7b3764e21..a843cca24 100644 --- a/azure/src/test/java/io/questdb/client/azure/test/AzureTokenSourceTest.java +++ b/azure/src/test/java/io/questdb/client/azure/test/AzureTokenSourceTest.java @@ -52,6 +52,7 @@ import reactor.core.publisher.Flux; import reactor.core.publisher.Mono; +import java.io.EOFException; import java.io.IOException; import java.io.UncheckedIOException; import java.net.ConnectException; @@ -88,6 +89,17 @@ public void testAuthenticationFailuresEntraReportsArePermanent() { "AADSTS7000215: Invalid client secret provided.")), "AADSTS7000215"); } + @Test + public void testCertificateParseFailuresArePermanentWithoutExposingLibraryText() { + RuntimeException failure = new RuntimeException( + "Failed to parse the certificate for the credential: CERTIFICATE-SENTINEL", new EOFException()); + TokenUnavailableException e = fetchFailure(new RuntimeException("wrapper", failure)); + Assert.assertFalse(e.getMessage(), e.isRetryable()); + Assert.assertEquals("credential certificate could not be parsed for api://qdb/.default", e.getMessage()); + Assert.assertEquals(TokenUnavailableException.NO_RETRY_AFTER, e.getRetryAfterMillis()); + Assert.assertNull("do not retain the SDK exception or certificate details", e.getCause()); + } + @Test public void testConnectStringResolvesTheAzureFactory() { // C18 with the module present: the azure keys parse on both clients and nothing is fetched @@ -221,6 +233,9 @@ public void testOtherFailuresAreRetryable() { assertRetryable(new HttpResponseException("unavailable", new FakeResponse(503, null)), "HTTP 503", -1); assertRetryable(new HttpResponseException("gone", new FakeResponse(410, "Wed, 21 Oct 2015 07:28:00 GMT")), "HTTP 410", -1); assertRetryable(new IllegalStateException("connection reset"), "IllegalStateException", -1); + assertRetryable(new RuntimeException("unexpected end of stream", new EOFException()), "RuntimeException", -1); + assertRetryable(new RuntimeException("response body: Failed to parse the certificate for the credential: text"), + "RuntimeException", -1); assertRetryable(new MsalServiceException("temporarily unavailable", "temporarily_unavailable"), "MsalServiceException", -1); assertRetryable(new ClientAuthenticationException("ManagedIdentityCredential: IMDS endpoint timed out", null, From 748ed987cb3a32fc1106635f5405840ed79b6fcd Mon Sep 17 00:00:00 2001 From: glasstiger Date: Mon, 19 Oct 2026 10:49:54 +0100 Subject: [PATCH 36/40] Fix query shutdown and health classification Give query workers a sticky shutdown request that stops I/O and\nreconnects without freeing buffers while a handler still uses them.\nWake event consumers even when the I/O loop exits while idle, and\nleave resource cleanup to the dispatch thread after execution ends. Preserve typed endpoint failures as causes when connection walks\nexhaust their endpoints, so health reports HTTP upgrade rejections\nwith their original class and status instead of TRANSPORT and zero. Add regression coverage for consumed interrupts, native batch\nlifetimes, shutdown during reconnect, and upgrade-status reporting.\nReplace the null-client interrupt-test fixture with a real client. Validation: 378 query, pool, health, failover, and buffer-safety tests\npassed on each of JDK 11 and JDK 25. Mutation checks confirm the new\nregressions fail without the fixes. JDK 8 was unavailable locally. --- .../cutlass/qwp/client/QwpEgressIoThread.java | 5 +- .../cutlass/qwp/client/QwpQueryClient.java | 63 +++++++- .../io/questdb/client/impl/QueryWorker.java | 5 +- .../qwp/client/ConnectionHealthTest.java | 119 +++++++++++++++ .../QwpEgressIoThreadCloseRaceTest.java | 21 +++ .../impl/QueryWorkerBusyShutdownTest.java | 141 +++++++++++++++++- .../client/test/impl/QueryWorkerTest.java | 55 ++++--- 7 files changed, 371 insertions(+), 38 deletions(-) diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpEgressIoThread.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpEgressIoThread.java index 700e219ea..ecfa91f75 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpEgressIoThread.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpEgressIoThread.java @@ -338,7 +338,10 @@ public void run() { // (or any abnormal exit) while a user thread is mid-execute() would let // takeEvent() block forever -- once the I/O thread is gone, no further // events arrive on the queue. - if (!currentQueryDone) { + // Publish on shutdown even if idle: the dispatch thread can submit a request just as + // this loop exits, before it observes the client's sticky stop request. It must still + // find a terminal event rather than wait forever on an I/O thread that has already gone. + if (!currentQueryDone || shutdown) { String msg = shutdown ? "I/O thread shut down with query in flight" : "I/O thread terminated with query in flight"; diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpQueryClient.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpQueryClient.java index ad21fb62a..61d5547cd 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpQueryClient.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpQueryClient.java @@ -252,6 +252,8 @@ public class QwpQueryClient implements QuietCloseable { // -- crashing the JVM on the next decode cycle. private volatile QwpEgressIoThread ioThread; private volatile Thread ioThreadHandle; + // Sticky, non-freeing stop request: a pooled worker may still be inside a handler or reconnect walk. + private volatile boolean isShutdownRequested; private boolean lastCloseTimedOut; // Client preference for server-side per-batch row cap. 0 means "unset", // server uses its default. Set via {@code max_batch_rows=N} in the @@ -575,8 +577,9 @@ public void cancel() { * those pointers and SIGSEGV the JVM. The interrupt-driven I/O thread shutdown * does NOT detect or wait for a still-running user handler -- the timeout-based * leak fallback only protects against an unresponsive I/O thread, not an - * unresponsive user thread. {@link #cancel()} is the right way to ask an - * in-flight {@code execute()} to return; close after it does. + * unresponsive user thread. Use {@link #cancel()} to cancel the current query, or + * {@link #requestShutdown()} to stop execution and reconnects permanently; close + * after the in-flight {@code execute()} returns. */ @Override public void close() { @@ -709,7 +712,7 @@ public void close() { * rather than being retried across endpoints as a transport failure. */ public synchronized void connect() { - if (closedFlag.get()) { + if (closedFlag.get() || isShutdownRequested) { throw new IllegalStateException("QwpQueryClient is closed"); } if (connected) { @@ -733,6 +736,20 @@ public ConnectionHealth health() { return healthTracker.snapshot(); } + /** + * Permanently stops query execution and reconnects without freeing any resources. Safe to call from + * another thread, including while a handler still reads batch column pointers. A connect already in + * progress may run until its timeout; any I/O thread it publishes also receives the stop request. + * Finish the in-flight {@code execute()} before calling {@link #close()} to release the resources. + */ + public void requestShutdown() { + isShutdownRequested = true; + QwpEgressIoThread io = ioThread; + if (io != null) { + io.shutdown(); + } + } + // One connect round: resolve the credential, walk the endpoints. See connect(). private void connectWalk() { if (tokenProviderSpec != null && tokenProviderLease == null) { @@ -856,9 +873,11 @@ private void connectWalk() { ex.initCause(lastUpgradeRoleReject); throw ex; } - throw new HttpClientException( + HttpClientException failure = new HttpClientException( "all QWP endpoints unreachable [count=" + endpoints.size() + ", lastError=" + (lastTransportError == null ? "" : lastTransportError.getMessage()) + ']'); + failure.initCause(lastTransportError); + throw failure; } /** @@ -1643,6 +1662,9 @@ private void cleanupFailedConnect() { } private void connectToEndpoint(Endpoint ep, String authHeader) { + if (isShutdownRequested) { + throw new IllegalStateException("QwpQueryClient is closed"); + } if (tlsEnabled) { webSocketClient = WebSocketClientFactory.newTlsInstance( new ClientTlsConfiguration(trustStorePath, trustStorePassword, tlsValidationMode)); @@ -1678,7 +1700,7 @@ private void connectToEndpoint(Endpoint ep, String authHeader) { } private void executeImpl(CharSequence sql, QwpBindSetter binds, QwpColumnBatchHandler handler, boolean resetSymbolDict) { - if (closedFlag.get()) { + if (closedFlag.get() || isShutdownRequested) { throw new IllegalStateException("QwpQueryClient is closed"); } if (!connected) { @@ -1712,12 +1734,20 @@ private void executeImpl(CharSequence sql, QwpBindSetter binds, QwpColumnBatchHa } int attempt = 0; while (true) { + if (isShutdownRequested) { + handler.onError(-1L, WebSocketResponse.STATUS_INTERNAL_ERROR, "query client shut down"); + return; + } attempt++; FailoverProbeHandler probe = new FailoverProbeHandler(handler); executeOnce(sql, binds, probe, resetSymbolDict); if (!probe.transportFailureIntercepted) { return; } + if (isShutdownRequested) { + handler.onError(probe.interceptedRequestId, probe.interceptedStatus, "query client shut down"); + return; + } if (!failoverEnabled) { // With failover off the transport failure stays latched: every later execute() reports it. healthTracker.connectionLost(); @@ -1825,6 +1855,10 @@ private void executeImpl(CharSequence sql, QwpBindSetter binds, QwpColumnBatchHa + reconnectErr.getMessage() + ']'); return; } + if (isShutdownRequested) { + handler.onError(probe.interceptedRequestId, probe.interceptedStatus, "query client shut down"); + return; + } handler.onFailoverReset(probe.interceptedRequestId, serverInfo); } } @@ -1868,6 +1902,10 @@ private void executeOnce(CharSequence sql, QwpBindSetter binds, FailoverProbeHan return; } } + if (isShutdownRequested) { + probe.deliverFinal(-1L, "query client shut down"); + return; + } long requestId = nextRequestId++; currentRequestId = requestId; // Honor a cancel that arrived during the dispatch window. The latch @@ -1886,8 +1924,12 @@ private void executeOnce(CharSequence sql, QwpBindSetter binds, FailoverProbeHan switch (ev.kind) { case QueryEvent.KIND_BATCH: try { - probe.onBatch(ev.buffer.batch); + if (!isShutdownRequested) { + probe.onBatch(ev.buffer.batch); + } } finally { + // Even during shutdown, drain and release a queued batch: the I/O thread + // cannot leave its receive buffer until this token arrives. io.releaseBuffer(ev.buffer); } break; @@ -2089,9 +2131,11 @@ private void reconnectViaTracker() { "no endpoint matches target=" + target + " on failover; last observed role=" + QwpServerInfo.roleName(lastMismatch.getRole())); } - throw new HttpClientException( + HttpClientException failure = new HttpClientException( "all QWP endpoints unreachable on failover [count=" + total + ", lastError=" + (lastError == null ? "" : lastError.getMessage()) + ']'); + failure.initCause(lastError); + throw failure; } private String resolveAuthorizationHeader() { @@ -2203,6 +2247,11 @@ private void spawnIoThread() { handle.start(); ioThreadHandle = handle; ioThread = newIoThread; + // Pair the post-publication check with requestShutdown()'s flag-before-read ordering. Even if + // shutdown saw the old (or null) ioThread during a reconnect, this generation must stop too. + if (isShutdownRequested) { + newIoThread.shutdown(); + } } private static final class Endpoint { diff --git a/core/src/main/java/io/questdb/client/impl/QueryWorker.java b/core/src/main/java/io/questdb/client/impl/QueryWorker.java index 95422ffb2..28ff4feb6 100644 --- a/core/src/main/java/io/questdb/client/impl/QueryWorker.java +++ b/core/src/main/java/io/questdb/client/impl/QueryWorker.java @@ -232,7 +232,10 @@ void shutdown() { try { // If a query is in flight on this worker, force execute() to return // promptly so the dispatch thread exits before the join below times - // out. Two nudges, strongest first: + // out. Stop without freeing anything first: a handler can consume an + // interrupt, and a reconnect walk can outlive it. The sticky request + // stops the I/O loop and prevents query replay on a new connection. + client.requestShutdown(); // 1. Interrupt the dispatch thread. takeEvent() (QwpSpscQueue.take) // is interrupt-aware, and executeOnce() turns the resulting // InterruptedException into a terminal event -> signalDone. This diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/ConnectionHealthTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/ConnectionHealthTest.java index 1e4c71bee..cdb9b2917 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/ConnectionHealthTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/ConnectionHealthTest.java @@ -29,11 +29,15 @@ import io.questdb.client.QuestDB; import io.questdb.client.Sender; import io.questdb.client.cutlass.auth.TokenUnavailableException; +import io.questdb.client.cutlass.http.client.HttpClientException; import io.questdb.client.cutlass.http.client.WebSocketClientFactory; +import io.questdb.client.cutlass.http.client.WebSocketUpgradeException; import io.questdb.client.cutlass.qwp.client.QwpColumnBatch; import io.questdb.client.cutlass.qwp.client.QwpColumnBatchHandler; +import io.questdb.client.cutlass.qwp.client.QwpConnectionHealthTracker; import io.questdb.client.cutlass.qwp.client.QwpEgressMsgKind; import io.questdb.client.cutlass.qwp.client.QwpQueryClient; +import io.questdb.client.cutlass.qwp.client.QwpVersionMismatchException; import io.questdb.client.cutlass.qwp.client.QwpWebSocketSender; import io.questdb.client.cutlass.qwp.client.WebSocketResponse; import io.questdb.client.cutlass.qwp.protocol.QwpConstants; @@ -183,6 +187,27 @@ public void testFacadeAggregatesEveryPooledConnection() throws Exception { }); } + @Test + public void testHealthClassifiesTransportAndWrappedUpgradeFailures() { + HttpClientException transport = new HttpClientException("connection refused"); + ConnectionHealth.Failure failure = QwpConnectionHealthTracker.classify(transport, 42L); + Assert.assertEquals(ConnectionHealth.FailureClass.TRANSPORT, failure.getFailureClass()); + Assert.assertEquals(0, failure.getStatusCode()); + + HttpClientException upgrade = new HttpClientException("all endpoints failed"); + upgrade.initCause(new WebSocketUpgradeException(503, null, "service unavailable")); + failure = QwpConnectionHealthTracker.classify(upgrade, 42L); + Assert.assertEquals(ConnectionHealth.FailureClass.OTHER, failure.getFailureClass()); + Assert.assertEquals(503, failure.getStatusCode()); + Assert.assertEquals("service unavailable", failure.getMessage()); + + HttpClientException protocol = new HttpClientException("all endpoints failed"); + protocol.initCause(new QwpVersionMismatchException(99, 1)); + failure = QwpConnectionHealthTracker.classify(protocol, 42L); + Assert.assertEquals(ConnectionHealth.FailureClass.OTHER, failure.getFailureClass()); + Assert.assertEquals(0, failure.getStatusCode()); + } + @Test(timeout = 60_000) public void testQueryClientHealthFollowsConnectFailoverAndRecovery() throws Exception { // C21 for egress: connecting, connected, reconnecting after a failed failover - with the class of the @@ -247,6 +272,71 @@ public void testQueryClientHealthIsFailedWhenItsOnlyAttemptLosesItsConnection() "transport failure after 1 execute attempt")); } + @Test(timeout = 60_000) + public void testQueryClientHealthPreservesUpgradeStatusOnConnect() throws Exception { + assertMemoryLeak(() -> { + try (TestWebSocketServer server = startServer(new ExecDoneHandler())) { + server.setSendServerInfo(true); + for (int status : new int[]{400, 404, 426, 500, 503}) { + server.setRejectWithStatus(status, "Rejected"); + try (QwpQueryClient client = QwpQueryClient.fromConfig("ws::addr=localhost:" + server.getPort() + ';')) { + try { + client.connect(); + Assert.fail("the upgrade must fail with HTTP " + status); + } catch (HttpClientException expected) { + Assert.assertTrue("the endpoint walk must preserve the typed rejection", + expected.getCause() instanceof WebSocketUpgradeException); + Assert.assertEquals(status, ((WebSocketUpgradeException) expected.getCause()).getStatusCode()); + } + ConnectionHealth health = client.health(); + Assert.assertEquals(ConnectionHealth.State.CONNECTING, health.getState()); + Assert.assertEquals(1, health.getFailedRounds()); + assertUpgradeFailure(health, status); + } + } + } + }); + } + + @Test(timeout = 60_000) + public void testQueryClientHealthPreservesUpgradeStatusOnFailoverAndRecovery() throws Exception { + assertMemoryLeak(() -> { + try (TestWebSocketServer server = startServer(new ExecDoneHandler())) { + server.setSendServerInfo(true); + for (int status : new int[]{500, 503}) { + server.setRejectWithStatus(0, null); + try (QwpQueryClient client = QwpQueryClient.fromConfig("ws::addr=localhost:" + server.getPort() + + ";failover_backoff_initial_ms=0;")) { + client.connect(); + server.setRejectWithStatus(status, "Rejected"); + server.dropAllConnections(); + RecordingHandler failed = new RecordingHandler(); + client.execute("SELECT 1", failed); + Assert.assertNotNull("the failover must fail", failed.error); + ConnectionHealth health = client.health(); + Assert.assertEquals(ConnectionHealth.State.RECONNECTING, health.getState()); + Assert.assertEquals(1, health.getFailedRounds()); + assertUpgradeFailure(health, status); + + // The next operation takes connectWalk(), not reconnectViaTracker(). Both must + // preserve the status, and a successful recovery must keep the last failure. + client.execute("SELECT 1", new NoopHandler()); + Assert.assertEquals(2, client.health().getFailedRounds()); + assertUpgradeFailure(client.health(), status); + server.setRejectWithStatus(0, null); + RecordingHandler recovered = new RecordingHandler(); + client.execute("SELECT 1", recovered); + Assert.assertNull(recovered.error, recovered.error); + Assert.assertEquals(1, recovered.execDone); + Assert.assertEquals(ConnectionHealth.State.CONNECTED, client.health().getState()); + Assert.assertEquals(0, client.health().getFailedRounds()); + assertUpgradeFailure(client.health(), status); + } + } + } + }); + } + @Test(timeout = 60_000) public void testQueryClientHealthReportsAConnectionLostOnTheLastAttempt() throws Exception { // C21 for egress: execute() gives up when a query loses its connection on the last attempt its failover @@ -283,6 +373,27 @@ public void testQueryClientHealthReportsAConnectionLostOnTheLastAttempt() throws }); } + @Test(timeout = 60_000) + public void testQueryClientHealthReportsTransportFailureOnConnect() throws Exception { + assertMemoryLeak(() -> { + try (TestWebSocketServer server = startServer(new ExecDoneHandler()); + QwpQueryClient client = QwpQueryClient.fromConfig("ws::addr=localhost:" + server.getPort() + ';')) { + server.close(); + try { + client.connect(); + Assert.fail("a closed endpoint must not accept a connection"); + } catch (HttpClientException expected) { + Assert.assertNotNull(expected.getCause()); + } + ConnectionHealth health = client.health(); + Assert.assertEquals(ConnectionHealth.State.CONNECTING, health.getState()); + Assert.assertEquals(1, health.getFailedRounds()); + Assert.assertEquals(ConnectionHealth.FailureClass.TRANSPORT, health.getLastFailure().getFailureClass()); + Assert.assertEquals(0, health.getLastFailure().getStatusCode()); + } + }); + } + @Test(timeout = 60_000) public void testSenderHealthIsFailedWithAnOutageAfterATerminalRejection() throws Exception { // C21: a sender that becomes terminal while connected - the server rejects a batch with a TERMINAL category - @@ -542,6 +653,14 @@ private static void assertQueryClientFailsForGood(String configKeys, String erro } } + private static void assertUpgradeFailure(ConnectionHealth health, int status) { + ConnectionHealth.Failure failure = health.getLastFailure(); + Assert.assertNotNull(failure); + Assert.assertEquals(health.toString(), ConnectionHealth.FailureClass.OTHER, failure.getFailureClass()); + Assert.assertEquals(status, failure.getStatusCode()); + Assert.assertTrue(failure.getMessage(), failure.getMessage().contains(Integer.toString(status))); + } + private static TestWebSocketServer startServer(TestWebSocketServer.WebSocketServerHandler handler) throws IOException, InterruptedException { TestWebSocketServer server = new TestWebSocketServer(handler); diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/QwpEgressIoThreadCloseRaceTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/QwpEgressIoThreadCloseRaceTest.java index 567033280..fff92b4e1 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/QwpEgressIoThreadCloseRaceTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/QwpEgressIoThreadCloseRaceTest.java @@ -26,7 +26,9 @@ import io.questdb.client.cutlass.qwp.client.QwpBatchBuffer; import io.questdb.client.cutlass.qwp.client.QwpEgressIoThread; +import io.questdb.client.cutlass.qwp.client.QueryEvent; import io.questdb.client.test.tools.TestUtils; +import org.junit.Assert; import org.junit.Test; import java.lang.reflect.Method; @@ -49,6 +51,25 @@ */ public class QwpEgressIoThreadCloseRaceTest { + @Test(timeout = 5_000) + public void testIdleShutdownWakesARacingEventConsumer() throws Exception { + TestUtils.assertMemoryLeak(() -> { + QwpEgressIoThread io = new QwpEgressIoThread(null, 2, (status, message) -> { }); + try { + io.shutdown(); + io.run(); // The loop exits before consuming a request. + QueryEvent event = io.takeEvent(); + Assert.assertEquals(QueryEvent.KIND_ERROR, event.kind); + Assert.assertTrue(event.errorMessage, event.errorMessage.contains("shut down")); + io.releaseEvent(event); + } finally { + Method closePool = QwpEgressIoThread.class.getDeclaredMethod("closePool"); + closePool.setAccessible(true); + closePool.invoke(io); + } + }); + } + @Test public void testReleaseBufferRacesClosePoolSafely() throws Exception { Method closePoolMethod = QwpEgressIoThread.class.getDeclaredMethod("closePool"); diff --git a/core/src/test/java/io/questdb/client/test/impl/QueryWorkerBusyShutdownTest.java b/core/src/test/java/io/questdb/client/test/impl/QueryWorkerBusyShutdownTest.java index 3872e0aab..2d5326825 100644 --- a/core/src/test/java/io/questdb/client/test/impl/QueryWorkerBusyShutdownTest.java +++ b/core/src/test/java/io/questdb/client/test/impl/QueryWorkerBusyShutdownTest.java @@ -32,6 +32,7 @@ import io.questdb.client.cutlass.qwp.client.QwpColumnBatchHandler; import io.questdb.client.cutlass.qwp.client.QwpEgressMsgKind; import io.questdb.client.cutlass.qwp.client.QwpQueryClient; +import io.questdb.client.cutlass.qwp.client.QwpServerInfo; import io.questdb.client.cutlass.qwp.protocol.QwpConstants; import io.questdb.client.impl.QueryClientPool; import io.questdb.client.impl.QueryWorker; @@ -129,6 +130,124 @@ public void onExecDone(short opType, long rowsAffected) { }); } + @Test(timeout = 60_000) + public void testShutdownPreservesBatchUntilInterruptSwallowingHandlerReturns() throws Exception { + assertShutdownAfterConsumedInterrupt(true, false); + } + + @Test(timeout = 60_000) + public void testShutdownStopsWorkerAfterBatchHandlerConsumesInterrupt() throws Exception { + assertShutdownAfterConsumedInterrupt(false, false); + } + + @Test(timeout = 60_000) + public void testShutdownStopsWorkerAfterConsumedInterruptAndConnectionLoss() throws Exception { + assertShutdownAfterConsumedInterrupt(false, true); + } + + private static void assertShutdownAfterConsumedInterrupt(boolean isHandlerHeld, boolean isConnectionDropped) + throws Exception { + TestUtils.assertMemoryLeak(() -> { + AtomicInteger queries = new AtomicInteger(); + TestWebSocketServer server = new TestWebSocketServer(new TestWebSocketServer.WebSocketServerHandler() { + @Override + public void onBinaryMessage(TestWebSocketServer.ClientHandler client, byte[] data) { + if (data.length > 0 && data[0] == QwpEgressMsgKind.QUERY_REQUEST) { + queries.incrementAndGet(); + try { + client.sendBinary(buildBatch(data)); + } catch (IOException ignored) { + // Surfaces as a transport error. No RESULT_END or response to CANCEL follows. + } + } + } + }); + server.setSendServerInfo(true); + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + CountDownLatch inHandler = new CountDownLatch(1); + CountDownLatch releaseHandler = new CountDownLatch(1); + AtomicBoolean hasConsumedInterrupt = new AtomicBoolean(); + AtomicBoolean hasReadBatchAfterShutdown = new AtomicBoolean(); + AtomicInteger closes = new AtomicInteger(); + AtomicInteger failovers = new AtomicInteger(); + AtomicReference error = new AtomicReference<>(); + try (TestWebSocketServer ignored = server; + QuestDB db = QuestDB.connect("ws::addr=localhost:" + server.getPort() + + ";sender_pool_min=0;query_pool_min=1;query_pool_max=1;failover_backoff_initial_ms=0;")) { + QueryWorker worker = pooledWorker(db); + Thread dispatch = dispatchThread(db); + worker.client().setBeforeCloseHookForTest(closes::incrementAndGet); + try { + Query q = db.borrowQuery(); + Completion completion = q.sql("SELECT v FROM t").handler(new Handler() { + @Override + public void onBatch(QwpColumnBatch batch) { + Assert.assertEquals(42L, batch.getLongValue(0, 0)); + inHandler.countDown(); + try { + if (releaseHandler.await(30, TimeUnit.SECONDS)) { + return; + } + throw new AssertionError("shutdown never interrupted the handler"); + } catch (InterruptedException consumed) { + hasConsumedInterrupt.set(true); // Deliberately do not restore the flag. + } + if (isConnectionDropped) { + server.dropAllConnections(); + } + if (isHandlerHeld) { + awaitUninterruptibly(releaseHandler); + } + // Shutdown must not free or compact the native bytes while the handler reads them. + Assert.assertEquals(42L, batch.getLongValue(0, 0)); + hasReadBatchAfterShutdown.set(true); + } + + @Override + public void onError(byte status, String message) { + error.set(message); + } + + @Override + public void onFailoverReset(long requestId, QwpServerInfo serverInfo) { + failovers.incrementAndGet(); + } + }).submit(); + Assert.assertTrue("the query never reached its handler", inHandler.await(10, TimeUnit.SECONDS)); + db.close(); + if (isHandlerHeld) { + Assert.assertTrue("the handler must still own its batch", dispatch.isAlive()); + Assert.assertEquals("shutdown must not free a batch still in use", 0, closes.get()); + Assert.assertFalse(completion.isDone()); + } + releaseHandler.countDown(); + dispatch.join(TimeUnit.SECONDS.toMillis(10)); + Assert.assertFalse("shutdown must wake a worker even after its handler consumes the interrupt", + dispatch.isAlive()); + Assert.assertTrue(hasConsumedInterrupt.get()); + Assert.assertTrue(hasReadBatchAfterShutdown.get()); + Assert.assertEquals("the dispatch thread must close its client exactly once", 1, closes.get()); + Assert.assertTrue("shutdown must release completion waiters", completion.isDone()); + try { + completion.await(); + Assert.fail("the cancelled query must fail"); + } catch (QueryException expected) { + Assert.assertNotNull(error.get()); + } + Assert.assertEquals("shutdown must not replay the query", 1, queries.get()); + Assert.assertEquals(0, failovers.get()); + Assert.assertEquals(1, server.upgradeRequestCount()); + } finally { + releaseHandler.countDown(); + db.close(); + server.close(); + dispatch.join(TimeUnit.SECONDS.toMillis(10)); + } + } + }); + } + /** * The failure from the review: a pooled client whose failover reconnect failed reconnects on its next * execute(). The walk stalls on an endpoint that accepts TCP but never answers the upgrade, outlasting the @@ -167,8 +286,14 @@ private static void assertShutdownDuringAReconnectWalkIsSafe(boolean binds) thro Assert.assertFalse(client.isConnected()); // the next execute() walks the endpoints and parks on the stalling one + AtomicInteger executions = new AtomicInteger(); Query q = db.borrowQuery(); - q.sql(binds ? "SELECT $1" : "SELECT 1").handler(new Handler()); + q.sql(binds ? "SELECT $1" : "SELECT 1").handler(new Handler() { + @Override + public void onExecDone(short opType, long rowsAffected) { + executions.incrementAndGet(); + } + }); if (binds) { q.binds(b -> b.setLong(0, 42L)); } @@ -181,6 +306,8 @@ private static void assertShutdownDuringAReconnectWalkIsSafe(boolean binds) thro dispatchThread.join(TimeUnit.SECONDS.toMillis(30)); // the walk resumes, then the thread exits Assert.assertFalse("the dispatch thread must exit", dispatchThread.isAlive()); Assert.assertFalse("a closed client must not stay connected", client.isConnected()); + Assert.assertTrue("shutdown must release the query completion", completion.isDone()); + Assert.assertEquals("a walk that finishes after shutdown must not execute the query", 0, executions.get()); long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(10); while (healthy.liveConnectionCount() > 0 && System.nanoTime() < deadline) { Thread.sleep(10); @@ -214,6 +341,18 @@ private static void awaitUninterruptibly(CountDownLatch latch) { } } + private static byte[] buildBatch(byte[] queryRequest) { + int bodyLen = 25; // kind, request id, schema and one non-null LONG value + ByteBuffer frame = ByteBuffer.allocate(QwpConstants.HEADER_SIZE + bodyLen).order(ByteOrder.LITTLE_ENDIAN); + frame.putInt(QwpConstants.MAGIC_MESSAGE).put(QwpConstants.VERSION); + frame.put((byte) 0).put((byte) 0).put((byte) 1).putInt(bodyLen); + frame.put(QwpEgressMsgKind.RESULT_BATCH).put(queryRequest, 1, 8); + frame.put((byte) 0).put((byte) 0).put((byte) 1).put((byte) 1); // seq, table name, rows, columns + frame.put((byte) 1).put((byte) 'v').put(QwpConstants.TYPE_LONG).put((byte) 0); + frame.putLong(42L); + return frame.array(); + } + private static byte[] buildExecDone(byte[] queryRequest) { int bodyLen = 1 + 8 + 1 + 1; // msg_kind + request_id + op_type + rows_affected varint byte[] frame = new byte[QwpConstants.HEADER_SIZE + bodyLen]; diff --git a/core/src/test/java/io/questdb/client/test/impl/QueryWorkerTest.java b/core/src/test/java/io/questdb/client/test/impl/QueryWorkerTest.java index 1af8176dc..d59544939 100644 --- a/core/src/test/java/io/questdb/client/test/impl/QueryWorkerTest.java +++ b/core/src/test/java/io/questdb/client/test/impl/QueryWorkerTest.java @@ -91,35 +91,34 @@ public void testClientGetterReturnsConstructorInstance() throws Exception { @Test(timeout = 30_000) public void testShutdownHandsBackACarriedInterrupt() throws Exception { TestUtils.assertMemoryLeak(() -> { - // A null client is enough: shutdown()'s cancel() and close() calls NPE and are swallowed by - // their own catch (Throwable), which is the documented best-effort teardown, and the dispatch - // thread this asserts on does not touch the client while parked. - QueryWorker worker = new QueryWorker(null, null, 0); - // start()/shutdown() are package-private; reached the same way the sibling tests in this class - // reach bumpGeneration() and isCurrentThreadWorker(), rather than widening them for a test - Method start = QueryWorker.class.getDeclaredMethod("start"); - start.setAccessible(true); - Method shutdown = QueryWorker.class.getDeclaredMethod("shutdown"); - shutdown.setAccessible(true); - start.invoke(worker); - - Field threadField = QueryWorker.class.getDeclaredField("thread"); - threadField.setAccessible(true); - Thread dispatch = (Thread) threadField.get(worker); - Assert.assertTrue("the dispatch thread must be running before shutdown", dispatch.isAlive()); - - Thread.currentThread().interrupt(); - try { - shutdown.invoke(worker); - Assert.assertTrue("shutdown() must hand the caller's cancellation back, or the pool loop " - + "that set it silently loses its stop signal", Thread.currentThread().isInterrupted()); - } finally { - // do not leak the flag into the next test - Thread.interrupted(); - } + try (QwpQueryClient client = QwpQueryClient.newPlainText("localhost", 9000)) { + QueryWorker worker = new QueryWorker(client, null, 0); + // start()/shutdown() are package-private; reached the same way the sibling tests in this class + // reach bumpGeneration() and isCurrentThreadWorker(), rather than widening them for a test + Method start = QueryWorker.class.getDeclaredMethod("start"); + start.setAccessible(true); + Method shutdown = QueryWorker.class.getDeclaredMethod("shutdown"); + shutdown.setAccessible(true); + start.invoke(worker); + + Field threadField = QueryWorker.class.getDeclaredField("thread"); + threadField.setAccessible(true); + Thread dispatch = (Thread) threadField.get(worker); + Assert.assertTrue("the dispatch thread must be running before shutdown", dispatch.isAlive()); + + Thread.currentThread().interrupt(); + try { + shutdown.invoke(worker); + Assert.assertTrue("shutdown() must hand the caller's cancellation back, or the pool loop " + + "that set it silently loses its stop signal", Thread.currentThread().isInterrupted()); + } finally { + // do not leak the flag into the next test + Thread.interrupted(); + } - dispatch.join(TimeUnit.SECONDS.toMillis(10)); - Assert.assertFalse("shutdown() must not leave its dispatch thread running", dispatch.isAlive()); + dispatch.join(TimeUnit.SECONDS.toMillis(10)); + Assert.assertFalse("shutdown() must not leave its dispatch thread running", dispatch.isAlive()); + } }); } From cfee59a9ae84433272d447c5c97cb44acee2c638 Mon Sep 17 00:00:00 2001 From: glasstiger Date: Mon, 19 Oct 2026 12:29:28 +0100 Subject: [PATCH 37/40] Strengthen auth and shutdown regression tests Complete and consume a query before checking idle I/O shutdown. Require its shutdown event before pool cleanup, and retain the original startup state test under a more accurate name. Run both forced-refresh callbacks through FutureTask so bounded result checks propagate callback exceptions to JUnit instead of treating an exceptionally terminated thread as a successful return. Redirect keytool output to a temporary file before waiting for exit. Destroy and reap timed-out or interrupted children, preserve the caller's interrupt flag, and cover success, diagnostics, timeout, and interruption with real child-JVM tests. Validate 274 unit tests and four packaged-jar checks on native JDK 25. Run 52 affected tests on Java 8 in interpreted mode; the default local Java 8 VM also crashes in Unsafe.recordMemAlloc on unmodified HEAD. Mutation checks confirm shutdown, both callback, and child-cleanup regressions now fail their tests. --- .../auth/RefreshingTokenProviderTest.java | 12 +- .../QwpEgressIoThreadCloseRaceTest.java | 2 +- .../test/cutlass/qwp/websocket/TestTls.java | 55 +++++--- .../cutlass/qwp/websocket/TestTlsTest.java | 129 ++++++++++++++++++ .../impl/QueryWorkerBusyShutdownTest.java | 55 ++++++++ 5 files changed, 227 insertions(+), 26 deletions(-) create mode 100644 core/src/test/java/io/questdb/client/test/cutlass/qwp/websocket/TestTlsTest.java diff --git a/core/src/test/java/io/questdb/client/test/cutlass/auth/RefreshingTokenProviderTest.java b/core/src/test/java/io/questdb/client/test/cutlass/auth/RefreshingTokenProviderTest.java index 65c256c1a..0d1452377 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/auth/RefreshingTokenProviderTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/auth/RefreshingTokenProviderTest.java @@ -378,11 +378,11 @@ public void testForcedRefreshIsRateLimitedIgnoresStaleTokensAndAcceptsTheSameTok Assert.assertNotEquals("a 403 must not schedule a fetch", 0, scheduler.peek().delayNanos); // 401 for the current token: a fetch starts now and the caller waits for it - Thread rejecter = start(() -> provider.onTokenRejected("T1", 401)); + FutureTask rejection = new FutureTask<>(() -> provider.onTokenRejected("T1", 401), null); + start(rejection); await(scheduler::isNextImmediate, 5_000, "the forced fetch to be scheduled"); scheduler.runNext(); - rejecter.join(5_000); - Assert.assertFalse("onTokenRejected must return once the forced fetch completes", rejecter.isAlive()); + rejection.get(5, TimeUnit.SECONDS); Assert.assertEquals("T2", provider.getToken().toString()); Assert.assertEquals(2, source.calls()); @@ -400,11 +400,11 @@ public void testForcedRefreshIsRateLimitedIgnoresStaleTokensAndAcceptsTheSameTok Assert.assertEquals(2, source.calls()); // a forced fetch that returns the same token is a normal success - rejecter = start(() -> provider.onTokenRejected("T2", 401)); + rejection = new FutureTask<>(() -> provider.onTokenRejected("T2", 401), null); + start(rejection); await(scheduler::isNextImmediate, 5_000, "the second forced fetch to be scheduled"); scheduler.runNext(); - rejecter.join(5_000); - Assert.assertFalse(rejecter.isAlive()); + rejection.get(5, TimeUnit.SECONDS); Assert.assertEquals(3, source.calls()); Assert.assertEquals("T2", provider.getToken().toString()); Assert.assertEquals(0, provider.getConsecutiveFailures()); diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/QwpEgressIoThreadCloseRaceTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/QwpEgressIoThreadCloseRaceTest.java index fff92b4e1..f0b622029 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/QwpEgressIoThreadCloseRaceTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/QwpEgressIoThreadCloseRaceTest.java @@ -52,7 +52,7 @@ public class QwpEgressIoThreadCloseRaceTest { @Test(timeout = 5_000) - public void testIdleShutdownWakesARacingEventConsumer() throws Exception { + public void testInitialShutdownWakesARacingEventConsumer() throws Exception { TestUtils.assertMemoryLeak(() -> { QwpEgressIoThread io = new QwpEgressIoThread(null, 2, (status, message) -> { }); try { diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/websocket/TestTls.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/websocket/TestTls.java index d36fb79b4..0bbf951ed 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/qwp/websocket/TestTls.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/websocket/TestTls.java @@ -32,6 +32,8 @@ import java.io.IOException; import java.io.InputStream; import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; import java.security.KeyStore; import java.util.concurrent.TimeUnit; @@ -54,6 +56,35 @@ public static synchronized SSLServerSocketFactory serverSocketFactory() throws I return factory; } + static void waitForKeytool(Process process, Path output, long timeout, TimeUnit unit) throws IOException { + try { + if (!process.waitFor(timeout, unit)) { + throw new IOException("keytool timed out"); + } + if (process.exitValue() != 0) { + throw new IOException("keytool failed: " + new String(Files.readAllBytes(output), StandardCharsets.UTF_8)); + } + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new IOException("interrupted while running keytool", e); + } finally { + if (process.isAlive()) { + process.destroyForcibly(); + // Reap the child even when the caller interrupted the wait, then restore its interrupt flag. + boolean isInterrupted = Thread.interrupted(); + try { + process.waitFor(5, TimeUnit.SECONDS); + } catch (InterruptedException e) { + isInterrupted = true; + } finally { + if (isInterrupted) { + Thread.currentThread().interrupt(); + } + } + } + } + } + private static SSLServerSocketFactory create() throws IOException { File keystore = File.createTempFile("qdb-test-server", ".p12"); if (!keystore.delete()) { @@ -74,16 +105,12 @@ private static SSLServerSocketFactory create() throws IOException { "-storepass", new String(PASSWORD), "-keypass", new String(PASSWORD) ); - pb.redirectErrorStream(true); - Process process = pb.start(); - byte[] output = readAll(process.getInputStream()); + Path output = Files.createTempFile("qdb-test-keytool", ".log"); try { - if (!process.waitFor(60, TimeUnit.SECONDS) || process.exitValue() != 0) { - throw new IOException("keytool failed: " + new String(output, StandardCharsets.UTF_8)); - } - } catch (InterruptedException e) { - Thread.currentThread().interrupt(); - throw new IOException("interrupted while running keytool", e); + Process process = pb.redirectErrorStream(true).redirectOutput(output.toFile()).start(); + waitForKeytool(process, output, 60, TimeUnit.SECONDS); + } finally { + Files.deleteIfExists(output); } try (InputStream in = new FileInputStream(keystore)) { KeyStore ks = KeyStore.getInstance("PKCS12"); @@ -99,14 +126,4 @@ private static SSLServerSocketFactory create() throws IOException { throw new IOException("could not load the test keystore", e); } } - - private static byte[] readAll(InputStream in) throws IOException { - java.io.ByteArrayOutputStream out = new java.io.ByteArrayOutputStream(); - byte[] buf = new byte[4096]; - int n; - while ((n = in.read(buf)) > 0) { - out.write(buf, 0, n); - } - return out.toByteArray(); - } } diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/websocket/TestTlsTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/websocket/TestTlsTest.java new file mode 100644 index 000000000..56c3b4b19 --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/websocket/TestTlsTest.java @@ -0,0 +1,129 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test.cutlass.qwp.websocket; + +import org.junit.Assert; +import org.junit.Test; + +import java.io.File; +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.concurrent.TimeUnit; + +public class TestTlsTest { + @Test(timeout = 20_000) + public void testFailedKeytoolReportsOutput() throws Exception { + assertKeytoolResult("fail", "keytool failed: test-keytool-output", 10_000); + } + + @Test(timeout = 20_000) + public void testInterruptedKeytoolKillsTheProcessAndPreservesTheFlag() throws Exception { + Path output = Files.createTempFile("qdb-test-keytool-interrupt", ".log"); + Process process = null; + try { + process = startProcess(output, "stall"); + Thread.currentThread().interrupt(); + try { + TestTls.waitForKeytool(process, output, 10, TimeUnit.SECONDS); + Assert.fail("the interrupted wait must fail"); + } catch (IOException expected) { + Assert.assertEquals("interrupted while running keytool", expected.getMessage()); + Assert.assertTrue(expected.getCause() instanceof InterruptedException); + Assert.assertTrue("the caller must retain its interrupt flag", Thread.currentThread().isInterrupted()); + } + Assert.assertFalse("the interrupted wait must reap the process", process.isAlive()); + } finally { + Thread.interrupted(); + stopProcess(process); + Files.deleteIfExists(output); + } + } + + @Test(timeout = 20_000) + public void testSuccessfulKeytoolExits() throws Exception { + assertKeytoolResult("success", null, 10_000); + } + + @Test(timeout = 20_000) + public void testTimedOutKeytoolKillsTheProcess() throws Exception { + assertKeytoolResult("stall", "keytool timed out", 100); + } + + private static void assertKeytoolResult(String mode, String expectedMessage, long timeoutMillis) throws Exception { + Path output = Files.createTempFile("qdb-test-keytool-result", ".log"); + Process process = null; + try { + process = startProcess(output, mode); + try { + TestTls.waitForKeytool(process, output, timeoutMillis, TimeUnit.MILLISECONDS); + Assert.assertNull("the process must fail", expectedMessage); + Assert.assertEquals(0, process.exitValue()); + } catch (IOException e) { + Assert.assertNotNull("the process must succeed", expectedMessage); + Assert.assertTrue(e.getMessage(), e.getMessage().contains(expectedMessage)); + } + Assert.assertFalse("the wait must reap the process", process.isAlive()); + } finally { + stopProcess(process); + Files.deleteIfExists(output); + } + } + + private static Process startProcess(Path output, String mode) throws Exception { + String java = System.getProperty("java.home") + File.separator + "bin" + File.separator + "java"; + String classpath = new File(KeytoolProcess.class.getProtectionDomain().getCodeSource() + .getLocation().toURI()).getAbsolutePath(); + return new ProcessBuilder(java, "-cp", classpath, KeytoolProcess.class.getName(), mode) + .redirectErrorStream(true) + .redirectOutput(output.toFile()) + .start(); + } + + private static void stopProcess(Process process) throws InterruptedException { + if (process != null) { + process.destroyForcibly(); + Assert.assertTrue("the test process must terminate", process.waitFor(5, TimeUnit.SECONDS)); + } + } + + public static class KeytoolProcess { + public static void main(String[] args) throws InterruptedException { + System.out.println("test-keytool-output"); + switch (args[0]) { + case "fail": + System.exit(7); + break; + case "stall": + Thread.sleep(TimeUnit.MINUTES.toMillis(1)); + break; + case "success": + break; + default: + throw new IllegalArgumentException(args[0]); + } + } + } +} diff --git a/core/src/test/java/io/questdb/client/test/impl/QueryWorkerBusyShutdownTest.java b/core/src/test/java/io/questdb/client/test/impl/QueryWorkerBusyShutdownTest.java index 2d5326825..ed23bc86c 100644 --- a/core/src/test/java/io/questdb/client/test/impl/QueryWorkerBusyShutdownTest.java +++ b/core/src/test/java/io/questdb/client/test/impl/QueryWorkerBusyShutdownTest.java @@ -30,9 +30,11 @@ import io.questdb.client.QuestDB; import io.questdb.client.cutlass.qwp.client.QwpColumnBatch; import io.questdb.client.cutlass.qwp.client.QwpColumnBatchHandler; +import io.questdb.client.cutlass.qwp.client.QwpEgressIoThread; import io.questdb.client.cutlass.qwp.client.QwpEgressMsgKind; import io.questdb.client.cutlass.qwp.client.QwpQueryClient; import io.questdb.client.cutlass.qwp.client.QwpServerInfo; +import io.questdb.client.cutlass.qwp.client.QueryEvent; import io.questdb.client.cutlass.qwp.protocol.QwpConstants; import io.questdb.client.impl.QueryClientPool; import io.questdb.client.impl.QueryWorker; @@ -52,6 +54,7 @@ import java.util.List; import java.util.concurrent.CopyOnWriteArrayList; import java.util.concurrent.CountDownLatch; +import java.util.concurrent.FutureTask; import java.util.concurrent.TimeUnit; import java.util.concurrent.atomic.AtomicBoolean; import java.util.concurrent.atomic.AtomicInteger; @@ -69,6 +72,58 @@ public class QueryWorkerBusyShutdownTest { private static final String SENDER_CFG = "http::addr=127.0.0.1:1;protocol_version=2;auto_flush=off;"; + @Test(timeout = 20_000) + public void testIdleShutdownWakesARacingEventConsumer() throws Exception { + TestUtils.assertMemoryLeak(() -> { + try (TestWebSocketServer server = startServer(new AtomicBoolean()); + QwpQueryClient client = QwpQueryClient.fromConfig("ws::addr=localhost:" + server.getPort() + ";")) { + client.connect(); + AtomicInteger completed = new AtomicInteger(); + client.execute("SELECT 1", new Handler() { + @Override + public void onError(byte status, String message) { + Assert.fail(message); + } + + @Override + public void onExecDone(short opType, long rowsAffected) { + completed.incrementAndGet(); + } + }); + Assert.assertEquals("consume the query's original terminal event", 1, completed.get()); + + Field ioField = QwpQueryClient.class.getDeclaredField("ioThread"); + ioField.setAccessible(true); + QwpEgressIoThread io = (QwpEgressIoThread) ioField.get(client); + Field threadField = QwpQueryClient.class.getDeclaredField("ioThreadHandle"); + threadField.setAccessible(true); + Thread ioThread = (Thread) threadField.get(client); + FutureTask nextEvent = new FutureTask<>(io::takeEvent); + Thread consumer = new Thread(nextEvent, "idle-shutdown-event-consumer"); + consumer.setDaemon(true); + consumer.start(); + try { + // The completed query leaves currentQueryDone true. Shutdown must still publish a new + // event, before closePool() runs, for a consumer racing the now-idle I/O thread's exit. + io.shutdown(); + ioThread.join(TimeUnit.SECONDS.toMillis(5)); + Assert.assertFalse("the idle I/O thread must stop", ioThread.isAlive()); + QueryEvent event = nextEvent.get(5, TimeUnit.SECONDS); + try { + Assert.assertEquals(QueryEvent.KIND_ERROR, event.kind); + Assert.assertTrue(event.errorMessage, event.errorMessage.contains("shut down")); + } finally { + io.releaseEvent(event); + } + } finally { + consumer.interrupt(); + consumer.join(TimeUnit.SECONDS.toMillis(5)); + Assert.assertFalse("the event consumer must stop", consumer.isAlive()); + } + } + }); + } + @Test(timeout = 60_000) public void testShutdownDuringAReconnectWalkWithBinds() throws Exception { assertShutdownDuringAReconnectWalkIsSafe(true); From 66009e1ce81adf910b526adfb4a9f6d95fae9591 Mon Sep 17 00:00:00 2001 From: glasstiger Date: Mon, 19 Oct 2026 14:08:51 +0100 Subject: [PATCH 38/40] Prefix new boolean fields with is The review of this branch at cfee59a9 flagged new boolean fields that lack the is/has prefix the coding standard asks for. This commit renames them: - RefreshingTokenProvider: backoffActive, closed, failureLogged, fetchRunning, forcedRefreshed and shortWindowWarned become isBackoffActive, isClosed, isFailureLogged, isFetchRunning, isForcedRefreshed and isShortWindowWarned. - AzureTokenSource: chain becomes isChain, and so does the matching classify() parameter. - QwpQueryClient: reconnectOnNextExecute becomes isReconnectOnNextExecute. The renamed fields move to their sorted positions. No behaviour changes. The core and azure modules compile on JDK 25; the nine affected core test classes (118 tests) and the azure module (39 tests) pass. --- .../client/azure/AzureTokenSource.java | 12 ++-- .../cutlass/auth/RefreshingTokenProvider.java | 66 +++++++++---------- .../cutlass/qwp/client/QwpQueryClient.java | 14 ++-- 3 files changed, 46 insertions(+), 46 deletions(-) diff --git a/azure/src/main/java/io/questdb/client/azure/AzureTokenSource.java b/azure/src/main/java/io/questdb/client/azure/AzureTokenSource.java index d120c8e38..57b463dbe 100644 --- a/azure/src/main/java/io/questdb/client/azure/AzureTokenSource.java +++ b/azure/src/main/java/io/questdb/client/azure/AzureTokenSource.java @@ -109,10 +109,10 @@ public final class AzureTokenSource implements TokenSource { "AADSTS7000112", // application disabled "AADSTS50049", // unknown or invalid instance }; - // whether the credential is a chain, such as DefaultAzureCredential: only the wording of errors depends on it - private final boolean chain; private final TokenRequestContext context; private final TokenCredential credential; + // whether the credential is a chain, such as DefaultAzureCredential: only the wording of errors depends on it + private final boolean isChain; private final String scope; private final Duration timeout; @@ -142,7 +142,7 @@ public AzureTokenSource(TokenCredential credential, String resource, Duration ti throw new IllegalArgumentException("timeout must be positive"); } this.credential = credential; - this.chain = credential instanceof ChainedTokenCredential; + this.isChain = credential instanceof ChainedTokenCredential; this.scope = resource.endsWith(DEFAULT_SCOPE_SUFFIX) ? resource : resource + DEFAULT_SCOPE_SUFFIX; this.context = new TokenRequestContext().addScopes(scope); this.timeout = timeout; @@ -151,7 +151,7 @@ public AzureTokenSource(TokenCredential credential, String resource, Duration ti /** * Classifies a failure from Azure Identity per section 7.5. Never attaches the library exception. */ - static TokenUnavailableException classify(Throwable failure, String scope, boolean chain) { + static TokenUnavailableException classify(Throwable failure, String scope, boolean isChain) { final String description = describe(failure); boolean hasCertificateParseFailure = false; long retryAfterMillis = TokenUnavailableException.NO_RETRY_AFTER; @@ -162,7 +162,7 @@ static TokenUnavailableException classify(Throwable failure, String scope, boole } if (t instanceof CredentialUnavailableException) { return TokenUnavailableException.permanent("no credential available " - + (chain ? "in the Azure Identity chain " : "") + "for " + scope + ": " + description); + + (isChain ? "in the Azure Identity chain " : "") + "for " + scope + ": " + description); } if (t instanceof HttpResponseException) { final HttpResponse response = ((HttpResponseException) t).getResponse(); @@ -297,7 +297,7 @@ public ExpiringToken fetchToken() { try { token = credential.getToken(context).block(timeout); } catch (RuntimeException e) { - throw classify(e, scope, chain); + throw classify(e, scope, isChain); } if (token == null) { throw TokenUnavailableException.retryable("Azure Identity returned no token for " + scope); diff --git a/core/src/main/java/io/questdb/client/cutlass/auth/RefreshingTokenProvider.java b/core/src/main/java/io/questdb/client/cutlass/auth/RefreshingTokenProvider.java index 133d38164..019fcba72 100644 --- a/core/src/main/java/io/questdb/client/cutlass/auth/RefreshingTokenProvider.java +++ b/core/src/main/java/io/questdb/client/cutlass/auth/RefreshingTokenProvider.java @@ -143,17 +143,18 @@ public final class RefreshingTokenProvider implements HttpTokenProvider, QuietCl // Signalled whenever a fetch completes and on close. private final Condition stateChanged = lock.newCondition(); // ---- guarded by lock ---- - private boolean backoffActive; private long backoffUntilNanos; - private volatile boolean closed; private long completedFetches; private volatile int consecutiveFailures; - private boolean failureLogged; - private volatile boolean fetchRunning; - private boolean forcedRefreshed; // The current token, or null. Immutable snapshot, published with a volatile write so the warm path in // getToken() needs no lock. private volatile Held held; + private boolean isBackoffActive; + private volatile boolean isClosed; + private boolean isFailureLogged; + private volatile boolean isFetchRunning; + private boolean isForcedRefreshed; + private boolean isShortWindowWarned; private volatile FetchFailure lastFailure; private long lastFailureLogNanos; private long lastForcedNanos; @@ -161,7 +162,6 @@ public final class RefreshingTokenProvider implements HttpTokenProvider, QuietCl private long scheduledDueNanos; private long scheduledSeq; private ScheduledTask scheduledTask; - private boolean shortWindowWarned; private RefreshingTokenProvider(Builder b) { this.source = b.source; @@ -258,7 +258,7 @@ public boolean awaitReady(long timeoutMillis) { requestFetchNowLocked(start); } while (true) { - if (closed) { + if (isClosed) { return false; } if (usable(held, clock.wallClockMillis())) { @@ -289,10 +289,10 @@ public boolean awaitReady(long timeoutMillis) { public void close() { lock.lock(); try { - if (closed) { + if (isClosed) { return; } - closed = true; + isClosed = true; cancelScheduledLocked(); held = null; stateChanged.signalAll(); @@ -344,7 +344,7 @@ public String getName() { */ @Override public CharSequence getToken() { - if (closed) { + if (isClosed) { throw closedException(); } final Held h = held; @@ -354,7 +354,7 @@ public CharSequence getToken() { // Warm path: a volatile read, no I/O and no lock. Only when the refresh is overdue - its // scheduled fetch has not run, say after the host slept through it - does a caller take the // lock, to start that fetch in the background. - if (!fetchRunning && consecutiveFailures == 0 + if (!isFetchRunning && consecutiveFailures == 0 && (wallNow >= h.refreshAtEpochMillis || clock.monotonicNanos() - h.refreshAtNanos >= 0)) { startOverdueRefresh(h); } @@ -376,7 +376,7 @@ public long getTokenExpiresAtEpochMillis() { * @return true once {@link #close()} has been called */ public boolean isClosed() { - return closed; + return isClosed; } /** @@ -393,7 +393,7 @@ public void onTokenRejected(CharSequence token, int httpStatus) { } lock.lock(); try { - if (closed) { + if (isClosed) { return; } final Held h = held; @@ -401,16 +401,16 @@ public void onTokenRejected(CharSequence token, int httpStatus) { return; // stale: the token has already rotated, so the caller's next pull gets the new one } final long now = clock.monotonicNanos(); - if (forcedRefreshed && now - lastForcedNanos < forcedMinIntervalNanos) { + if (isForcedRefreshed && now - lastForcedNanos < forcedMinIntervalNanos) { return; } - forcedRefreshed = true; + isForcedRefreshed = true; lastForcedNanos = now; LOG.info("token provider {}: the server rejected the current token with 401, refreshing early", name); final long target = completedFetches + 1; requestFetchNowLocked(now); final long deadline = now + forcedWaitNanos; - while (!closed && completedFetches < target) { + while (!isClosed && completedFetches < target) { final long remaining = deadline - clock.monotonicNanos(); if (remaining <= 0) { break; @@ -444,7 +444,7 @@ public String toString() { if (f != null) { sb.append(", lastFailure=").append(f); } - if (closed) { + if (isClosed) { sb.append(", closed"); } return sb.append('}').toString(); @@ -463,7 +463,7 @@ private CharSequence awaitUsableToken() { requestFetchNowLocked(start); } while (true) { - if (closed) { + if (isClosed) { throw closedException(); } final Held h = held; @@ -473,7 +473,7 @@ private CharSequence awaitUsableToken() { final long now = clock.monotonicNanos(); // The earliest permitted fetch lies beyond the caller's deadline: nothing can arrive in time, // so say so now rather than parking the caller for nothing. - if (!fetchRunning && inBackoffLocked(now) && backoffUntilNanos - deadline > 0) { + if (!isFetchRunning && inBackoffLocked(now) && backoffUntilNanos - deadline > 0) { throw unavailableLocked(now, start); } final long remaining = deadline - now; @@ -562,7 +562,7 @@ private TokenUnavailableException closedException() { } private boolean inBackoffLocked(long now) { - return backoffActive && now - backoffUntilNanos < 0; + return isBackoffActive && now - backoffUntilNanos < 0; } private String onFailureLocked(FetchFailure failure, long receivedNanos) { @@ -571,13 +571,13 @@ private String onFailureLocked(FetchFailure failure, long receivedNanos) { lastFailure = failure; final long delayMillis = backoffMillis(failures, failure.retryAfterMillis); final long delayNanos = millisToNanos(delayMillis); - backoffActive = true; + isBackoffActive = true; backoffUntilNanos = receivedNanos + delayNanos; scheduleFetchLocked(delayNanos, receivedNanos); - if (failureLogged && receivedNanos - lastFailureLogNanos < FAILURE_LOG_INTERVAL_NANOS) { + if (isFailureLogged && receivedNanos - lastFailureLogNanos < FAILURE_LOG_INTERVAL_NANOS) { return null; } - failureLogged = true; + isFailureLogged = true; lastFailureLogNanos = receivedNanos; return "token provider " + name + ": fetch failed [retryable=" + failure.retryable + ", consecutiveFailures=" + failures + ", retryInMillis=" + delayMillis + "]: " + failure.message; @@ -598,7 +598,7 @@ private void onSuccessLocked(ExpiringToken result, long receivedAt, long receive held = new Held(result.getToken(), result.getExpiresAtEpochMillis(), refreshAt, receivedNanos + delayNanos); lastSuccessEpochMillis = receivedAt; consecutiveFailures = 0; - backoffActive = false; + isBackoffActive = false; scheduleFetchLocked(delayNanos, receivedNanos); if (failures > 0) { LOG.info("token provider {}: fetch succeeded after {} consecutive failure(s) [token={}, expiresAt={}]", @@ -610,10 +610,10 @@ private void onSuccessLocked(ExpiringToken result, long receivedAt, long receive instant(result.getExpiresAtEpochMillis()), instant(refreshAt)); } final long handoutEnd = result.getExpiresAtEpochMillis() - handoutFloorMillis; - if (!shortWindowWarned && refreshAt > handoutEnd) { + if (!isShortWindowWarned && refreshAt > handoutEnd) { // The token stops being handed out before the next refresh may run, so callers will wait for a fetch. // Say why, once: the cure is configuration. - shortWindowWarned = true; + isShortWindowWarned = true; LOG.warn("token provider {}: a token from the source is handed out for only {} ms - it expires {} ms " + "after it arrives and handout_floor is {} ms - but the next refresh cannot run sooner " + "than {} ms after a fetch, so callers wait for a fetch whenever a token stops being " @@ -625,7 +625,7 @@ private void onSuccessLocked(ExpiringToken result, long receivedAt, long receive // Starts a fetch now unless one is running, one is already due, or the provider is in failure backoff. private void requestFetchNowLocked(long now) { - if (closed || fetchRunning || inBackoffLocked(now)) { + if (isClosed || isFetchRunning || inBackoffLocked(now)) { return; } if (scheduledTask != null && now - scheduledDueNanos >= 0) { @@ -637,11 +637,11 @@ private void requestFetchNowLocked(long now) { private void runFetch(long seq) { lock.lock(); try { - if (closed || seq != scheduledSeq || fetchRunning) { + if (isClosed || seq != scheduledSeq || isFetchRunning) { return; // cancelled, superseded, or a fetch is already in flight } scheduledTask = null; - fetchRunning = true; + isFetchRunning = true; } finally { lock.unlock(); } @@ -663,9 +663,9 @@ private void runFetch(long seq) { String warning = null; lock.lock(); try { - fetchRunning = false; + isFetchRunning = false; completedFetches++; - if (!closed) { + if (!isClosed) { if (failure == null) { onSuccessLocked(result, receivedAt, receivedNanos); } else { @@ -682,7 +682,7 @@ private void runFetch(long seq) { } private void scheduleFetchLocked(long delayNanos, long now) { - if (closed) { + if (isClosed) { return; } cancelScheduledLocked(); @@ -714,7 +714,7 @@ private TokenUnavailableException unavailableLocked(long now, long start) { return new TokenUnavailableException("no usable token from " + name + " after waiting " + waitedMillis + " ms", true); } - final long retryAfter = backoffActive && backoffUntilNanos - now > 0 + final long retryAfter = isBackoffActive && backoffUntilNanos - now > 0 ? TimeUnit.NANOSECONDS.toMillis(backoffUntilNanos - now) : TokenUnavailableException.NO_RETRY_AFTER; return new TokenUnavailableException("no usable token from " + name + " after waiting " + waitedMillis diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpQueryClient.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpQueryClient.java index 61d5547cd..fe31a6542 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpQueryClient.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpQueryClient.java @@ -252,6 +252,9 @@ public class QwpQueryClient implements QuietCloseable { // -- crashing the JVM on the next decode cycle. private volatile QwpEgressIoThread ioThread; private volatile Thread ioThreadHandle; + // Set when a failover reconnect inside execute() failed, leaving the client disconnected: the next + // execute() reconnects instead of throwing "not connected". Cleared by every successful connect. + private volatile boolean isReconnectOnNextExecute; // Sticky, non-freeing stop request: a pooled worker may still be inside a handler or reconnect walk. private volatile boolean isShutdownRequested; private boolean lastCloseTimedOut; @@ -278,9 +281,6 @@ public class QwpQueryClient implements QuietCloseable { // currentRequestId. Volatile so a cancel from any thread is visible to the // worker thread's post-requestId read. private volatile boolean pendingCancel; - // Set when a failover reconnect inside execute() failed, leaving the client disconnected: the next - // execute() reconnects instead of throwing "not connected". Cleared by every successful connect. - private volatile boolean reconnectOnNextExecute; // Decoded SERVER_INFO from the current connection's handshake. Null before // connect() has succeeded; non-null on every established connection (the // server always emits the frame). Volatile so getServerInfo(), callable @@ -844,7 +844,7 @@ private void connectWalk() { spawnIoThread(); hostTracker.recordSuccess(i); currentEndpointIndex = i; - reconnectOnNextExecute = false; + isReconnectOnNextExecute = false; connected = true; return; } @@ -1704,7 +1704,7 @@ private void executeImpl(CharSequence sql, QwpBindSetter binds, QwpColumnBatchHa throw new IllegalStateException("QwpQueryClient is closed"); } if (!connected) { - if (!reconnectOnNextExecute) { + if (!isReconnectOnNextExecute) { throw new IllegalStateException("QwpQueryClient not connected; call connect() first"); } // A previous failover reconnect failed. Reconnect on this operation rather than leave the client @@ -1781,7 +1781,7 @@ private void executeImpl(CharSequence sql, QwpBindSetter binds, QwpColumnBatchHa healthTracker.connectionLost(); // Every exit from here on - an exhausted deadline, an interrupted backoff, a failed reconnect - leaves // the client disconnected; the next execute() reconnects rather than throw "not connected". - reconnectOnNextExecute = true; + isReconnectOnNextExecute = true; if (failoverInitialBackoffMs > 0L) { long base = failoverInitialBackoffMs << Math.min(attempt - 1, 30); if (base < 0L) base = failoverMaxBackoffMs; @@ -2122,7 +2122,7 @@ private void reconnectViaTracker() { spawnIoThread(); hostTracker.recordSuccess(i); currentEndpointIndex = i; - reconnectOnNextExecute = false; + isReconnectOnNextExecute = false; connected = true; return; } From 252966527bcda719143717529713cf0d7cc00ff7 Mon Sep 17 00:00:00 2001 From: glasstiger Date: Mon, 19 Oct 2026 15:19:21 +0100 Subject: [PATCH 39/40] Fix skip-test builds and new member order The azure module declared a test-scoped dependency on the core test jar, which a build with -Dmaven.test.skip=true never produces. Every such build of the client reactor failed to resolve questdb-client:jar:tests. That includes the questdb server's GitHub workflows that run "mvn clean install -Dmaven.test.skip=true -Plocal-client": PGWire stable and latest clients, the R test, Parquet export compatibility and the glibc smoke test. A local repository that still holds an older tests jar hides the failure. The dependency now lives in a core-test-jar profile that is active unless maven.test.skip is true. Such a build compiles no tests, so nothing needs the jar; every other build declares it as before. QwpQueryClient: health() and requestShutdown() move to their sorted places among the public methods, connectWalk() joins the private methods, and refreshAfterRejection() now precedes resolveAuthorizationHeader(). BearerChallenge.allowsTokenRefresh() now precedes bearerError(). These are pure moves. Verification, with the poms set to a version absent from ~/.m2: - mvn -o package -Dmaven.test.skip=true fails at 66009e1c on the azure module and passes with this change. - mvn -o clean package -DskipTests still builds the core test jar and compiles the azure tests against it. - The 39 azure tests and AzureModulePathIT pass, and so do 51 tests in six affected core test classes. --- azure/pom.xml | 31 +- .../client/cutlass/http/BearerChallenge.java | 20 +- .../cutlass/qwp/client/QwpQueryClient.java | 354 +++++++++--------- 3 files changed, 210 insertions(+), 195 deletions(-) diff --git a/azure/pom.xml b/azure/pom.xml index 76ff3b8ae..0c85a7821 100644 --- a/azure/pom.xml +++ b/azure/pom.xml @@ -99,14 +99,7 @@ azure-identity - - - org.questdb - questdb-client - ${project.version} - test-jar - test - + junit junit @@ -224,6 +217,28 @@ + + + core-test-jar + + + maven.test.skip + !true + + + + + org.questdb + questdb-client + ${project.version} + test-jar + test + + + java11+ diff --git a/core/src/main/java/io/questdb/client/cutlass/http/BearerChallenge.java b/core/src/main/java/io/questdb/client/cutlass/http/BearerChallenge.java index 914f97f00..745097af3 100644 --- a/core/src/main/java/io/questdb/client/cutlass/http/BearerChallenge.java +++ b/core/src/main/java/io/questdb/client/cutlass/http/BearerChallenge.java @@ -42,6 +42,16 @@ public final class BearerChallenge { private BearerChallenge() { } + /** + * Whether a {@code 401} carrying {@code wwwAuthenticate} warrants refreshing the token: true when the header + * has no {@code Bearer} challenge with an {@code error} parameter (the status code alone decides), or when + * that error is {@code invalid_token}. + */ + public static boolean allowsTokenRefresh(CharSequence wwwAuthenticate) { + final String error = bearerError(wwwAuthenticate); + return error == null || INVALID_TOKEN.equals(error); + } + /** * @param wwwAuthenticate the value of one or more {@code WWW-Authenticate} headers (joined with commas), or * null @@ -109,16 +119,6 @@ public static String bearerError(CharSequence wwwAuthenticate) { return null; } - /** - * Whether a {@code 401} carrying {@code wwwAuthenticate} warrants refreshing the token: true when the header - * has no {@code Bearer} challenge with an {@code error} parameter (the status code alone decides), or when - * that error is {@code invalid_token}. - */ - public static boolean allowsTokenRefresh(CharSequence wwwAuthenticate) { - final String error = bearerError(wwwAuthenticate); - return error == null || INVALID_TOKEN.equals(error); - } - private static boolean isTokenChar(char c) { if ((c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || (c >= '0' && c <= '9')) { return true; diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpQueryClient.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpQueryClient.java index fe31a6542..85b6d3ffe 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpQueryClient.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpQueryClient.java @@ -727,159 +727,6 @@ public synchronized void connect() { } } - /** - * Connection health (design/qwp-token-provider-spec.md, section 8.4). A query client connects on demand, so - * {@link ConnectionHealth.State#RECONNECTING} means its last connect or failover reconnect failed and the next - * operation will try again. Cheap and safe from any thread; never contains a credential. - */ - public ConnectionHealth health() { - return healthTracker.snapshot(); - } - - /** - * Permanently stops query execution and reconnects without freeing any resources. Safe to call from - * another thread, including while a handler still reads batch column pointers. A connect already in - * progress may run until its timeout; any I/O thread it publishes also receives the stop request. - * Finish the in-flight {@code execute()} before calling {@link #close()} to release the resources. - */ - public void requestShutdown() { - isShutdownRequested = true; - QwpEgressIoThread io = ioThread; - if (io != null) { - io.shutdown(); - } - } - - // One connect round: resolve the credential, walk the endpoints. See connect(). - private void connectWalk() { - if (tokenProviderSpec != null && tokenProviderLease == null) { - // The connect string selected a token_provider: share the process-wide provider for it (spec section - // 7.4). The lease is this client's until close(). - tokenProviderLease = TokenProviderRegistry.global().acquire(tokenProviderSpec); - tokenProvider = tokenProviderLease.provider(); - } else if (tokenProvider != null && !tlsEnabled && CLEARTEXT_PROVIDER_WARNED.compareAndSet(false, true)) { - // spec section 9: an application-supplied provider over ws:: is the application's call - warn once - LOG.warn("a token provider is used over ws:: (no TLS): bearer tokens cross the network in cleartext; " - + "use wss:: in production"); - } - lastCloseTimedOut = false; - if (hostTracker == null) { - hostTracker = new QwpHostHealthTracker( - endpoints.size(), clientZone, TARGET_PRIMARY.equals(target)); - } else { - hostTracker.beginRound(false); - } - QwpServerInfo lastObservedMismatch = null; - QwpIngressRoleRejectedException lastUpgradeRoleReject = null; - Throwable lastTransportError = null; - // Resolve the bearer credential once, before the endpoint walk: a token is cluster-wide, so a - // token-provider failure (not signed in, a failed silent refresh, a rejected token) is not a - // per-endpoint transport fault. Resolving here lets it propagate as a credential-unavailable failure - // that carries the provider's own error, instead of being folded into "all endpoints unreachable", - // and avoids re-querying the provider once per endpoint. - String authHeader = resolveAuthorizationHeader(); - // One same-endpoint retry after a refreshable 401 per connect, for a token provider only - // (design/qwp-token-provider-spec.md, section 8.2). The retried attempt records no health penalty. - boolean authRetryAvailable = tokenProvider != null; - int retryIdx = -1; - while (true) { - int i; - if (retryIdx >= 0) { - i = retryIdx; - retryIdx = -1; - } else { - i = hostTracker.pickNext(); - } - if (i < 0) { - break; - } - Endpoint ep = endpoints.get(i); - try { - connectToEndpoint(ep, authHeader); - } catch (QwpAuthFailedException ae) { - cleanupFailedConnect(); - if (authRetryAvailable && ae.isTokenRefreshable()) { - authRetryAvailable = false; - String refreshed = refreshAfterRejection(authHeader, ae); - if (refreshed != null && !refreshed.equals(authHeader)) { - LOG.info("QwpQueryClient {}:{} rejected the token with {}; retrying once with a refreshed token", - ep.host, ep.port, ae.getStatusCode()); - authHeader = refreshed; - retryIdx = i; - continue; - } - } - throw ae; - } catch (QwpIngressRoleRejectedException re) { - lastTransportError = re; - lastUpgradeRoleReject = re; - hostTracker.recordZone(i, re.getZoneId()); - hostTracker.recordRoleReject(i, re.isTransient()); - LOG.info("QwpQueryClient {}:{} rejected upgrade with role={}; trying next", - ep.host, ep.port, re.getRole()); - cleanupFailedConnect(); - continue; - } catch (RuntimeException e) { - lastTransportError = e; - hostTracker.recordTransportError(i); - LOG.warn("QwpQueryClient connect failed for {}:{} -- {}", ep.host, ep.port, e.getMessage()); - cleanupFailedConnect(); - continue; - } - // info is non-null: connectToEndpoint() returns only after - // receiveServerInfoSync() set serverInfo (it returns non-null or throws). - QwpServerInfo info = serverInfo; - if ((info.getCapabilities() & QwpEgressMsgKind.CAP_ZONE) != 0) { - hostTracker.recordZone(i, info.getZoneId()); - } - if (!matchesTarget(info.getRole(), target)) { - lastObservedMismatch = info; - boolean isTransient = info.getRole() == QwpEgressMsgKind.ROLE_PRIMARY_CATCHUP; - hostTracker.recordRoleReject(i, isTransient); - LOG.info("QwpQueryClient {}:{} role={} does not match target={}, trying next", - ep.host, ep.port, QwpServerInfo.roleName(info.getRole()), target); - cleanupFailedConnect(); - continue; - } - spawnIoThread(); - hostTracker.recordSuccess(i); - currentEndpointIndex = i; - isReconnectOnNextExecute = false; - connected = true; - return; - } - if (lastObservedMismatch != null) { - throw new QwpRoleMismatchException( - target, - lastObservedMismatch, - "no endpoint matches target=" + target + "; last observed role=" - + QwpServerInfo.roleName(lastObservedMismatch.getRole()) - + " cluster=" + lastObservedMismatch.getClusterId() - ); - } - if (lastUpgradeRoleReject != null) { - // 421 + X-QuestDB-Role on every host: the cluster answered with role - // information at upgrade time but no host satisfied target=. Surface - // QwpRoleMismatchException with the last role for diagnostics so - // callers can distinguish "no primary available" from "all unreachable" - // (failover.md §6 Topology / wire-egress.md §11.9.2). - QwpRoleMismatchException ex = new QwpRoleMismatchException( - target, - null, - "no endpoint matches target=" + target - + "; last observed role=" + lastUpgradeRoleReject.getRole() - + " at " + lastUpgradeRoleReject.getHost() + ':' + lastUpgradeRoleReject.getPort() - ); - ex.initCause(lastUpgradeRoleReject); - throw ex; - } - HttpClientException failure = new HttpClientException( - "all QWP endpoints unreachable [count=" + endpoints.size() - + ", lastError=" + (lastTransportError == null ? "" : lastTransportError.getMessage()) + ']'); - failure.initCause(lastTransportError); - throw failure; - } - /** * Executes {@code sql} and drives the supplied handler through the result stream. *

      @@ -1069,6 +916,15 @@ public QwpServerInfo getServerInfo() { return serverInfo; } + /** + * Connection health (design/qwp-token-provider-spec.md, section 8.4). A query client connects on demand, so + * {@link ConnectionHealth.State#RECONNECTING} means its last connect or failover reconnect failed and the next + * operation will try again. Cheap and safe from any thread; never contains a credential. + */ + public ConnectionHealth health() { + return healthTracker.snapshot(); + } + public boolean isConnected() { return connected; } @@ -1100,6 +956,20 @@ public void recordTerminalFailureForTest(byte status, String message) { } } + /** + * Permanently stops query execution and reconnects without freeing any resources. Safe to call from + * another thread, including while a handler still reads batch column pointers. A connect already in + * progress may run until its timeout; any I/O thread it publishes also receives the stop request. + * Finish the in-flight {@code execute()} before calling {@link #close()} to release the resources. + */ + public void requestShutdown() { + isShutdownRequested = true; + QwpEgressIoThread io = ioThread; + if (io != null) { + io.shutdown(); + } + } + public void seedFailoverRandomForTest(long seed) { synchronized (failoverRandom) { failoverRandom.setSeed(seed); @@ -1699,6 +1569,136 @@ private void connectToEndpoint(Endpoint ep, String authHeader) { } } + // One connect round: resolve the credential, walk the endpoints. See connect(). + private void connectWalk() { + if (tokenProviderSpec != null && tokenProviderLease == null) { + // The connect string selected a token_provider: share the process-wide provider for it (spec section + // 7.4). The lease is this client's until close(). + tokenProviderLease = TokenProviderRegistry.global().acquire(tokenProviderSpec); + tokenProvider = tokenProviderLease.provider(); + } else if (tokenProvider != null && !tlsEnabled && CLEARTEXT_PROVIDER_WARNED.compareAndSet(false, true)) { + // spec section 9: an application-supplied provider over ws:: is the application's call - warn once + LOG.warn("a token provider is used over ws:: (no TLS): bearer tokens cross the network in cleartext; " + + "use wss:: in production"); + } + lastCloseTimedOut = false; + if (hostTracker == null) { + hostTracker = new QwpHostHealthTracker( + endpoints.size(), clientZone, TARGET_PRIMARY.equals(target)); + } else { + hostTracker.beginRound(false); + } + QwpServerInfo lastObservedMismatch = null; + QwpIngressRoleRejectedException lastUpgradeRoleReject = null; + Throwable lastTransportError = null; + // Resolve the bearer credential once, before the endpoint walk: a token is cluster-wide, so a + // token-provider failure (not signed in, a failed silent refresh, a rejected token) is not a + // per-endpoint transport fault. Resolving here lets it propagate as a credential-unavailable failure + // that carries the provider's own error, instead of being folded into "all endpoints unreachable", + // and avoids re-querying the provider once per endpoint. + String authHeader = resolveAuthorizationHeader(); + // One same-endpoint retry after a refreshable 401 per connect, for a token provider only + // (design/qwp-token-provider-spec.md, section 8.2). The retried attempt records no health penalty. + boolean authRetryAvailable = tokenProvider != null; + int retryIdx = -1; + while (true) { + int i; + if (retryIdx >= 0) { + i = retryIdx; + retryIdx = -1; + } else { + i = hostTracker.pickNext(); + } + if (i < 0) { + break; + } + Endpoint ep = endpoints.get(i); + try { + connectToEndpoint(ep, authHeader); + } catch (QwpAuthFailedException ae) { + cleanupFailedConnect(); + if (authRetryAvailable && ae.isTokenRefreshable()) { + authRetryAvailable = false; + String refreshed = refreshAfterRejection(authHeader, ae); + if (refreshed != null && !refreshed.equals(authHeader)) { + LOG.info("QwpQueryClient {}:{} rejected the token with {}; retrying once with a refreshed token", + ep.host, ep.port, ae.getStatusCode()); + authHeader = refreshed; + retryIdx = i; + continue; + } + } + throw ae; + } catch (QwpIngressRoleRejectedException re) { + lastTransportError = re; + lastUpgradeRoleReject = re; + hostTracker.recordZone(i, re.getZoneId()); + hostTracker.recordRoleReject(i, re.isTransient()); + LOG.info("QwpQueryClient {}:{} rejected upgrade with role={}; trying next", + ep.host, ep.port, re.getRole()); + cleanupFailedConnect(); + continue; + } catch (RuntimeException e) { + lastTransportError = e; + hostTracker.recordTransportError(i); + LOG.warn("QwpQueryClient connect failed for {}:{} -- {}", ep.host, ep.port, e.getMessage()); + cleanupFailedConnect(); + continue; + } + // info is non-null: connectToEndpoint() returns only after + // receiveServerInfoSync() set serverInfo (it returns non-null or throws). + QwpServerInfo info = serverInfo; + if ((info.getCapabilities() & QwpEgressMsgKind.CAP_ZONE) != 0) { + hostTracker.recordZone(i, info.getZoneId()); + } + if (!matchesTarget(info.getRole(), target)) { + lastObservedMismatch = info; + boolean isTransient = info.getRole() == QwpEgressMsgKind.ROLE_PRIMARY_CATCHUP; + hostTracker.recordRoleReject(i, isTransient); + LOG.info("QwpQueryClient {}:{} role={} does not match target={}, trying next", + ep.host, ep.port, QwpServerInfo.roleName(info.getRole()), target); + cleanupFailedConnect(); + continue; + } + spawnIoThread(); + hostTracker.recordSuccess(i); + currentEndpointIndex = i; + isReconnectOnNextExecute = false; + connected = true; + return; + } + if (lastObservedMismatch != null) { + throw new QwpRoleMismatchException( + target, + lastObservedMismatch, + "no endpoint matches target=" + target + "; last observed role=" + + QwpServerInfo.roleName(lastObservedMismatch.getRole()) + + " cluster=" + lastObservedMismatch.getClusterId() + ); + } + if (lastUpgradeRoleReject != null) { + // 421 + X-QuestDB-Role on every host: the cluster answered with role + // information at upgrade time but no host satisfied target=. Surface + // QwpRoleMismatchException with the last role for diagnostics so + // callers can distinguish "no primary available" from "all unreachable" + // (failover.md §6 Topology / wire-egress.md §11.9.2). + QwpRoleMismatchException ex = new QwpRoleMismatchException( + target, + null, + "no endpoint matches target=" + target + + "; last observed role=" + lastUpgradeRoleReject.getRole() + + " at " + lastUpgradeRoleReject.getHost() + ':' + lastUpgradeRoleReject.getPort() + ); + ex.initCause(lastUpgradeRoleReject); + throw ex; + } + HttpClientException failure = new HttpClientException( + "all QWP endpoints unreachable [count=" + endpoints.size() + + ", lastError=" + (lastTransportError == null ? "" : lastTransportError.getMessage()) + ']'); + failure.initCause(lastTransportError); + throw failure; + } + private void executeImpl(CharSequence sql, QwpBindSetter binds, QwpColumnBatchHandler handler, boolean resetSymbolDict) { if (closedFlag.get() || isShutdownRequested) { throw new IllegalStateException("QwpQueryClient is closed"); @@ -2138,6 +2138,30 @@ private void reconnectViaTracker() { throw failure; } + /** + * Steps 1-2 of the one-retry-after-401 rule (design/qwp-token-provider-spec.md, section 8.2): tells the + * token provider that the token in {@code presentedHeader} was rejected, then resolves the header again. + * Returns the new header, or null when no credential could be obtained - the connect then ends with the + * rejection, carrying the provider's failure as a suppressed diagnostic. + */ + private String refreshAfterRejection(String presentedHeader, QwpAuthFailedException rejection) { + if (presentedHeader != null && presentedHeader.startsWith("Bearer ")) { + try { + tokenProvider.onTokenRejected(presentedHeader.substring("Bearer ".length()), + rejection.getStatusCode()); + } catch (RuntimeException e) { + // The contract says it must not throw; a provider that does still gets its token re-pulled. + LOG.debug("token provider onTokenRejected threw {}", e.getClass().getName()); + } + } + try { + return resolveAuthorizationHeader(); + } catch (RuntimeException e) { + rejection.addSuppressed(e); + return null; + } + } + private String resolveAuthorizationHeader() { // With a token provider, query it once per connect()/reconnect (the caller resolves before the // endpoint walk) so a reconnect presents a freshly refreshed token; validateToken rejects a @@ -2166,30 +2190,6 @@ private String resolveAuthorizationHeader() { return authorizationHeader; } - /** - * Steps 1-2 of the one-retry-after-401 rule (design/qwp-token-provider-spec.md, section 8.2): tells the - * token provider that the token in {@code presentedHeader} was rejected, then resolves the header again. - * Returns the new header, or null when no credential could be obtained - the connect then ends with the - * rejection, carrying the provider's failure as a suppressed diagnostic. - */ - private String refreshAfterRejection(String presentedHeader, QwpAuthFailedException rejection) { - if (presentedHeader != null && presentedHeader.startsWith("Bearer ")) { - try { - tokenProvider.onTokenRejected(presentedHeader.substring("Bearer ".length()), - rejection.getStatusCode()); - } catch (RuntimeException e) { - // The contract says it must not throw; a provider that does still gets its token re-pulled. - LOG.debug("token provider onTokenRejected threw {}", e.getClass().getName()); - } - } - try { - return resolveAuthorizationHeader(); - } catch (RuntimeException e) { - rejection.addSuppressed(e); - return null; - } - } - private long resolveQueryFlags(boolean resetSymbolDict) { if (!resetSymbolDict) { return 0L; From 76718cb1cda3aa8cdd65edc763b56ea798e04e74 Mon Sep 17 00:00:00 2001 From: glasstiger Date: Mon, 19 Oct 2026 17:48:15 +0100 Subject: [PATCH 40/40] Fix query shutdown and Windows native builds Track result batches that shutdown releases without invoking the handler. If the I/O thread reaches RESULT_END before it observes its own shutdown flag, report a shutdown error instead of success with missing rows. Keep buffer release and credit return intact, and allow complete results to finish successfully. Add four leak-checked regression tests that cover both outcomes and credit return. Use the Windows runner's existing MinGW toolchain instead of installing it through Chocolatey. Download pinned NASM 2.16.03 from its upstream release with bounded retries and SHA-256 verification. Check native command exit codes explicitly so PowerShell stops on download, tool, configuration, and build failures rather than continuing with stale artifacts. The failed Azure job received HTTP 503 from Chocolatey and continued until it could not find NASM. Rename the remaining reviewed boolean locals to use the is prefix. All 152 focused tests pass on JDK 17 and JDK 25, and on JDK 8 with tiered compilation disabled. The default JDK 8 C1 crash in Unsafe.recordMemAlloc also reproduces on the unmodified head. Both skipped-batch regression tests fail without the shutdown fix. YAML parsing, PowerShell parsing, and eight mocked-command Windows build scenarios pass; the actual Windows DLL build still needs validation on Azure. --- ci/build_native.yaml | 31 ++- .../client/cutlass/auth/ExpiringToken.java | 6 +- .../cutlass/auth/RefreshingTokenProvider.java | 4 +- .../cutlass/auth/TokenProviderRegistry.java | 6 +- .../client/cutlass/http/BearerChallenge.java | 6 +- .../cutlass/qwp/client/QwpQueryClient.java | 25 +- .../qwp/client/QwpWebSocketSender.java | 6 +- .../client/QwpQueryClientShutdownTest.java | 231 ++++++++++++++++++ 8 files changed, 283 insertions(+), 32 deletions(-) create mode 100644 core/src/test/java/io/questdb/client/test/cutlass/qwp/client/QwpQueryClientShutdownTest.java diff --git a/ci/build_native.yaml b/ci/build_native.yaml index 850ff1be1..3d47863b4 100644 --- a/ci/build_native.yaml +++ b/ci/build_native.yaml @@ -70,19 +70,30 @@ steps: - powershell: | $ErrorActionPreference = "Stop" - # The CMake build is GCC/MinGW based (gcc flags, -static-libgcc/-static-libstdc++), - # so build the Windows DLL with the MinGW-w64 toolchain + NASM, not MSVC. - choco install -y --no-progress nasm mingw - Import-Module "$env:ChocolateyInstall\helpers\chocolateyProfile.psm1" - refreshenv - # choco's nasm package does not put nasm on PATH; add it explicitly. - $env:PATH = "C:\Program Files\NASM;" + $env:PATH - gcc --version - mingw32-make --version - nasm --version + # windows-2022 already supplies MinGW-w64. Avoid Chocolatey's package feed: a 503 there + # used to leave NASM missing even though GCC and mingw32-make were ready to use. + $nasmVersion = "2.16.03" + $nasmZip = Join-Path $env:AGENT_TEMPDIRECTORY "nasm-$nasmVersion-win64.zip" + $nasmDir = Join-Path $env:AGENT_TEMPDIRECTORY "nasm-$nasmVersion" + $nasmUrl = "https://www.nasm.us/pub/nasm/releasebuilds/$nasmVersion/win64/nasm-$nasmVersion-win64.zip" + curl.exe --fail --location --retry 3 --retry-delay 5 --connect-timeout 30 --max-time 120 --output $nasmZip $nasmUrl + if ($LASTEXITCODE -ne 0) { throw "NASM download failed with exit code $LASTEXITCODE" } + $nasmHash = (Get-FileHash -Path $nasmZip -Algorithm SHA256).Hash + if ($nasmHash -ne "3ee4782247bcb874378d02f7eab4e294a84d3d15f3f6ee2de2f47a46aa7226e6") { + throw "NASM archive SHA-256 mismatch: $nasmHash" + } + Expand-Archive -Path $nasmZip -DestinationPath $env:AGENT_TEMPDIRECTORY -Force + $env:PATH = "$nasmDir;" + $env:PATH + # PowerShell 5.1 does not turn native exit codes into terminating errors. + foreach ($tool in @("gcc.exe", "g++.exe", "mingw32-make.exe", "nasm.exe")) { + & $tool --version + if ($LASTEXITCODE -ne 0) { throw "$tool failed with exit code $LASTEXITCODE" } + } cd core cmake -G "MinGW Makefiles" -DCMAKE_BUILD_TYPE=Release -B cmake-build-release -S . + if ($LASTEXITCODE -ne 0) { throw "CMake configuration failed with exit code $LASTEXITCODE" } cmake --build cmake-build-release --config Release + if ($LASTEXITCODE -ne 0) { throw "Native build failed with exit code $LASTEXITCODE" } $lib = "target/classes/io/questdb/client/bin-local/libquestdb.dll" if (!(Test-Path $lib)) { throw "native build produced no $lib" } New-Item -ItemType Directory -Force -Path "src/main/resources/io/questdb/client/bin/windows-x86-64" | Out-Null diff --git a/core/src/main/java/io/questdb/client/cutlass/auth/ExpiringToken.java b/core/src/main/java/io/questdb/client/cutlass/auth/ExpiringToken.java index 95713352b..ab193b87c 100644 --- a/core/src/main/java/io/questdb/client/cutlass/auth/ExpiringToken.java +++ b/core/src/main/java/io/questdb/client/cutlass/auth/ExpiringToken.java @@ -86,17 +86,17 @@ static String describeInvalidToken(CharSequence token) { if (token == null || token.length() == 0) { return "token is null or empty"; } - boolean blank = true; + boolean isBlank = true; for (int i = 0, n = token.length(); i < n; i++) { char c = token.charAt(i); if (c < 0x20 || c > 0x7e) { return "token contains a control or non-ASCII character"; } if (c != ' ') { - blank = false; + isBlank = false; } } - return blank ? "token is blank" : null; + return isBlank ? "token is blank" : null; } private static String instant(long epochMillis) { diff --git a/core/src/main/java/io/questdb/client/cutlass/auth/RefreshingTokenProvider.java b/core/src/main/java/io/questdb/client/cutlass/auth/RefreshingTokenProvider.java index 019fcba72..b441b054e 100644 --- a/core/src/main/java/io/questdb/client/cutlass/auth/RefreshingTokenProvider.java +++ b/core/src/main/java/io/questdb/client/cutlass/auth/RefreshingTokenProvider.java @@ -976,13 +976,13 @@ public void shutdown() { } // Interrupt-neutral, like the other close paths in this library: a carried interrupt flag would // turn the bounded wait into an immediate return. - final boolean interrupted = Thread.interrupted(); + final boolean isInterrupted = Thread.interrupted(); try { executor.awaitTermination(CLOSE_AWAIT_MILLIS, TimeUnit.MILLISECONDS); } catch (InterruptedException e) { Thread.currentThread().interrupt(); } finally { - if (interrupted) { + if (isInterrupted) { Thread.currentThread().interrupt(); } } diff --git a/core/src/main/java/io/questdb/client/cutlass/auth/TokenProviderRegistry.java b/core/src/main/java/io/questdb/client/cutlass/auth/TokenProviderRegistry.java index 16ac455db..cfcf99be5 100644 --- a/core/src/main/java/io/questdb/client/cutlass/auth/TokenProviderRegistry.java +++ b/core/src/main/java/io/questdb/client/cutlass/auth/TokenProviderRegistry.java @@ -160,14 +160,14 @@ private static void load(ServiceLoader loader, List { + CountDownLatch credits = new CountDownLatch(BATCH_COUNT); + try (TestWebSocketServer server = new TestWebSocketServer(new TestWebSocketServer.WebSocketServerHandler() { + @Override + public void onBinaryMessage(TestWebSocketServer.ClientHandler client, byte[] data) { + if (data.length == 0) { + return; + } + if (data[0] == QwpEgressMsgKind.CREDIT) { + credits.countDown(); + } else if (data[0] == QwpEgressMsgKind.QUERY_REQUEST) { + try { + for (int seq = 0; seq < BATCH_COUNT; seq++) { + client.sendBinary(resultBatch(data, seq)); + } + client.sendBinary(resultEnd(data)); + } catch (IOException ignored) { + // The client reports a failed reply through its error callback. + } + } + } + })) { + server.setSendServerInfo(true); + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + String config = "ws::addr=localhost:" + server.getPort() + ";" + + (isCreditEnabled ? "initial_credit=1_048_576;" : ""); + try (QwpQueryClient client = QwpQueryClient.fromConfig(config)) { + client.connect(); + ResultHandler handler = new ResultHandler(client, shutdownAfterBatch); + try { + client.execute("SELECT v FROM t", handler); + if (shutdownAfterBatch > 0 && shutdownAfterBatch < BATCH_COUNT) { + Assert.assertEquals(1, handler.batchCount); + Assert.assertEquals("a skipped batch must rule out success", 0, handler.endCount); + Assert.assertEquals(1, handler.errorCount); + Assert.assertEquals(WebSocketResponse.STATUS_INTERNAL_ERROR, handler.errorStatus); + Assert.assertEquals("query client shut down", handler.errorMessage); + } else { + Assert.assertEquals(BATCH_COUNT, handler.batchCount); + Assert.assertEquals(1, handler.endCount); + Assert.assertEquals(BATCH_COUNT, handler.totalRows); + Assert.assertEquals(0, handler.errorCount); + } + if (isCreditEnabled) { + Assert.assertTrue("release every batch, including skipped batches", + credits.await(5, TimeUnit.SECONDS)); + } + } finally { + client.requestShutdown(); + } + } + } + }); + } + + private static byte[] frame(byte[] body) { + return ByteBuffer.allocate(QwpConstants.HEADER_SIZE + body.length).order(ByteOrder.LITTLE_ENDIAN) + .putInt(QwpConstants.MAGIC_MESSAGE).put(QwpConstants.VERSION).put((byte) 0).putShort((short) 1) + .putInt(body.length).put(body).array(); + } + + private static void putVarint(ByteArrayOutputStream out, long value) { + while ((value & ~0x7FL) != 0) { + out.write((int) ((value & 0x7F) | 0x80)); + value >>>= 7; + } + out.write((int) value); + } + + private static byte[] resultBatch(byte[] queryRequest, int seq) { + ByteArrayOutputStream body = new ByteArrayOutputStream(); + body.write(QwpEgressMsgKind.RESULT_BATCH); + body.write(queryRequest, 1, 8); + putVarint(body, seq); + putVarint(body, 0); // table name length + putVarint(body, 1); // row count + if (seq == 0) { + putVarint(body, 1); // column count + putVarint(body, 1); // column name length + body.write('v'); + body.write(QwpConstants.TYPE_LONG); + } + body.write(0); // no nulls + long value = 100L + seq; + for (int i = 0; i < 8; i++) { + body.write((int) ((value >>> (8 * i)) & 0xFF)); + } + return frame(body.toByteArray()); + } + + private static byte[] resultEnd(byte[] queryRequest) { + ByteArrayOutputStream body = new ByteArrayOutputStream(); + body.write(QwpEgressMsgKind.RESULT_END); + body.write(queryRequest, 1, 8); + putVarint(body, BATCH_COUNT - 1); + putVarint(body, BATCH_COUNT); + return frame(body.toByteArray()); + } + + private static final class ResultHandler implements QwpColumnBatchHandler { + private final QwpQueryClient client; + private final int shutdownAfterBatch; + private final Field shutdownRequested; + private int batchCount; + private int endCount; + private int errorCount; + private String errorMessage; + private byte errorStatus; + private long requestId; + private long totalRows; + + private ResultHandler(QwpQueryClient client, int shutdownAfterBatch) throws NoSuchFieldException { + this.client = client; + this.shutdownAfterBatch = shutdownAfterBatch; + shutdownRequested = QwpQueryClient.class.getDeclaredField("isShutdownRequested"); + shutdownRequested.setAccessible(true); + } + + @Override + public void onBatch(QwpColumnBatch batch) { + Assert.assertEquals(batchCount, batch.batchSeq()); + Assert.assertEquals(1, batch.getRowCount()); + Assert.assertEquals(100L + batchCount, batch.getLongValue(0, 0)); + requestId = batch.requestId(); + if (++batchCount == shutdownAfterBatch) { + // Model a requester paused after requestShutdown() publishes the client flag but before + // it stops the I/O thread. Keep I/O live so it can deliver the remaining batches and END. + try { + shutdownRequested.setBoolean(client, true); + } catch (IllegalAccessException e) { + throw new AssertionError(e); + } + } + } + + @Override + public void onEnd(long totalRows) { + endCount++; + this.totalRows = totalRows; + } + + @Override + public void onEnd(long requestId, long totalRows) { + Assert.assertEquals(this.requestId, requestId); + onEnd(totalRows); + } + + @Override + public void onError(byte status, String message) { + errorCount++; + errorMessage = message; + errorStatus = status; + } + + @Override + public void onError(long requestId, byte status, String message) { + Assert.assertEquals(this.requestId, requestId); + onError(status, message); + } + } +}