From 7c7978278e30c7d2048bd123ade0d52779c50555 Mon Sep 17 00:00:00 2001 From: Steven Sklar Date: Tue, 13 Oct 2026 16:13:51 -0400 Subject: [PATCH 1/2] docs(operator): singleton has no default PDB; recommend node auto-repair Co-Authored-By: Claude Opus 5.5 --- .../configuration.md | 11 ++++---- .../known-limitations.md | 16 +++++++---- .../operations/database.md | 28 +++++++++++-------- 3 files changed, 33 insertions(+), 22 deletions(-) diff --git a/documentation/enterprise-kubernetes-operator/configuration.md b/documentation/enterprise-kubernetes-operator/configuration.md index 4778f9cc2..0133c84ec 100644 --- a/documentation/enterprise-kubernetes-operator/configuration.md +++ b/documentation/enterprise-kubernetes-operator/configuration.md @@ -227,12 +227,13 @@ worker-node IAM can provide ambient ECR pull access; IRSA cannot. `spec.scheduling` supports node selectors, affinity, tolerations, topology spread, priority class, and an operator-managed PodDisruptionBudget (PDB). -Without an override, pods get soft hostname anti-affinity, soft zone spread, and -a PDB with `minAvailable: 1` for one instance or `instances-1` for HA. +Without an override, pods get soft hostname anti-affinity and soft zone spread. +A replicated cluster also gets a PDB with `minAvailable: instances-1`. -The single-instance default intentionally blocks voluntary eviction. Before a -node drain, scale out, lower `spec.scheduling.podDisruptionBudget.minAvailable`, -or disable the PDB after accepting the availability risk. Follow the +A one-instance cluster gets no PDB by default, because a budget over a single +pod would block every node drain. A drain therefore restarts the database. Set +`spec.scheduling.podDisruptionBudget.enabled: true` to block voluntary eviction +instead. Follow the [node maintenance runbook](/docs/enterprise-kubernetes-operator/operations/database/#node-maintenance-and-disruption-budgets). Custom affinity or topology spread replaces the corresponding default rather than merging with it. diff --git a/documentation/enterprise-kubernetes-operator/known-limitations.md b/documentation/enterprise-kubernetes-operator/known-limitations.md index a35fc35d8..3acfe5794 100644 --- a/documentation/enterprise-kubernetes-operator/known-limitations.md +++ b/documentation/enterprise-kubernetes-operator/known-limitations.md @@ -16,15 +16,19 @@ operational planning. A one-instance cluster has no replica to promote. On an unreachable node, the operator does not create a second pod while the old writer might still run. -Recovery waits for safe node/volume recovery or human action. Use store-backed -replication before an incident; see +Recovery waits for safe node/volume recovery or human action. Turn on your +platform's node auto-repair so that an unhealthy node is replaced without +waiting for a person; see +[Node maintenance](/docs/enterprise-kubernetes-operator/operations/database/#node-maintenance-and-disruption-budgets). +Use store-backed replication before an incident; see [Scale out](/docs/enterprise-kubernetes-operator/operations/database/#scale-out). -### Singleton drain is blocked by default +### Draining a singleton's node restarts the database -The default PodDisruptionBudget has `minAvailable: 1`, so voluntary drain of the -singleton waits indefinitely. Before maintenance, scale out, lower -`minAvailable`, or disable the PDB after accepting downtime. See +A one-instance cluster has no PodDisruptionBudget by default. A voluntary drain +evicts the only pod, and the operator recreates it on another node in the same +zone. Expect a short outage on every node drain or node upgrade. To avoid it, +scale out before maintenance. See [Node maintenance](/docs/enterprise-kubernetes-operator/operations/database/#node-maintenance-and-disruption-budgets). ### Failover is never automatic diff --git a/documentation/enterprise-kubernetes-operator/operations/database.md b/documentation/enterprise-kubernetes-operator/operations/database.md index fbc53c6fb..a65ad421c 100644 --- a/documentation/enterprise-kubernetes-operator/operations/database.md +++ b/documentation/enterprise-kubernetes-operator/operations/database.md @@ -405,27 +405,27 @@ into a new cluster. ## Node maintenance and disruption budgets -The default PodDisruptionBudget (PDB) uses `minAvailable: 1` for a singleton and -`instances-1` for a replicated cluster. A singleton therefore blocks voluntary -eviction and can make `kubectl drain` wait indefinitely. +A replicated cluster gets a default PodDisruptionBudget (PDB) with +`minAvailable: instances-1`, which permits one voluntary disruption at a time. -Before planned maintenance, choose one safe option: +A singleton gets no PDB by default. `kubectl drain` and managed node upgrades +evict its only pod, and the operator recreates it on another node in the same +zone as its volume. The database is down until the new pod is ready. To avoid +that outage, scale out and wait for a healthy, current replica before planned +maintenance. -1. scale out and wait for a healthy, current replica; -2. lower `spec.scheduling.podDisruptionBudget.minAvailable`; or -3. set `spec.scheduling.podDisruptionBudget.enabled: false` only after accepting - database downtime. +To make a singleton block voluntary eviction instead, enable the PDB. A drain of +its node then waits until you lower `minAvailable` or disable the PDB again: ```yaml spec: scheduling: podDisruptionBudget: enabled: true - minAvailable: 0 ``` -Restore the normal PDB after maintenance and verify the current-generation -writer contract: `Available=True/PrimaryReady`, `Progressing=False/Settled`, and +After maintenance, verify the current-generation writer contract: +`Available=True/PrimaryReady`, `Progressing=False/Settled`, and `WriteHealthy=True/Healthy`. A PDB protects only voluntary disruption. On node loss, the operator reports @@ -435,6 +435,12 @@ the node/volume or follow the explicit [promotion and failover](/docs/enterprise-kubernetes-operator/high-availability/#promotion-and-failover) procedure. +A node that is `NotReady` while its machine keeps running leaves the pod stuck +until the machine is repaired or replaced. Turn on node auto-repair so this +happens without a person: AKS repairs unhealthy nodes automatically, and on EKS +you enable node auto repair on the managed node group. Once the node is gone, +the operator recreates the pod on a healthy node. + ## Delete a database cluster Deleting a `QuestDBCluster` removes operator-owned pods, Services, ConfigMaps, From e429923c5132e65bc5f9e86e00e7a5f79f65d4d8 Mon Sep 17 00:00:00 2001 From: Steven Sklar Date: Fri, 16 Oct 2026 10:58:15 -0400 Subject: [PATCH 2/2] docs(operator): node repair may reboot rather than replace Co-Authored-By: Claude Opus 5.5 --- .../enterprise-kubernetes-operator/operations/database.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/documentation/enterprise-kubernetes-operator/operations/database.md b/documentation/enterprise-kubernetes-operator/operations/database.md index a65ad421c..04edd95cf 100644 --- a/documentation/enterprise-kubernetes-operator/operations/database.md +++ b/documentation/enterprise-kubernetes-operator/operations/database.md @@ -438,8 +438,8 @@ procedure. A node that is `NotReady` while its machine keeps running leaves the pod stuck until the machine is repaired or replaced. Turn on node auto-repair so this happens without a person: AKS repairs unhealthy nodes automatically, and on EKS -you enable node auto repair on the managed node group. Once the node is gone, -the operator recreates the pod on a healthy node. +you enable node auto repair on the managed node group. Once the node is +repaired or replaced, the pod runs again. ## Delete a database cluster