Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 6 additions & 5 deletions documentation/enterprise-kubernetes-operator/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -227,12 +227,13 @@ worker-node IAM can provide ambient ECR pull access; IRSA cannot.

`spec.scheduling` supports node selectors, affinity, tolerations, topology
spread, priority class, and an operator-managed PodDisruptionBudget (PDB).
Without an override, pods get soft hostname anti-affinity, soft zone spread, and
a PDB with `minAvailable: 1` for one instance or `instances-1` for HA.
Without an override, pods get soft hostname anti-affinity and soft zone spread.
A replicated cluster also gets a PDB with `minAvailable: instances-1`.

The single-instance default intentionally blocks voluntary eviction. Before a
node drain, scale out, lower `spec.scheduling.podDisruptionBudget.minAvailable`,
or disable the PDB after accepting the availability risk. Follow the
A one-instance cluster gets no PDB by default, because a budget over a single
pod would block every node drain. A drain therefore restarts the database. Set
`spec.scheduling.podDisruptionBudget.enabled: true` to block voluntary eviction
instead. Follow the
[node maintenance runbook](/docs/enterprise-kubernetes-operator/operations/database/#node-maintenance-and-disruption-budgets).
Custom affinity or topology spread replaces the corresponding default rather
than merging with it.
Expand Down
16 changes: 10 additions & 6 deletions documentation/enterprise-kubernetes-operator/known-limitations.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,15 +16,19 @@ operational planning.

A one-instance cluster has no replica to promote. On an unreachable node, the
operator does not create a second pod while the old writer might still run.
Recovery waits for safe node/volume recovery or human action. Use store-backed
replication before an incident; see
Recovery waits for safe node/volume recovery or human action. Turn on your
platform's node auto-repair so that an unhealthy node is replaced without
waiting for a person; see
[Node maintenance](/docs/enterprise-kubernetes-operator/operations/database/#node-maintenance-and-disruption-budgets).
Use store-backed replication before an incident; see
[Scale out](/docs/enterprise-kubernetes-operator/operations/database/#scale-out).

### Singleton drain is blocked by default
### Draining a singleton's node restarts the database

The default PodDisruptionBudget has `minAvailable: 1`, so voluntary drain of the
singleton waits indefinitely. Before maintenance, scale out, lower
`minAvailable`, or disable the PDB after accepting downtime. See
A one-instance cluster has no PodDisruptionBudget by default. A voluntary drain
evicts the only pod, and the operator recreates it on another node in the same
zone. Expect a short outage on every node drain or node upgrade. To avoid it,
scale out before maintenance. See
[Node maintenance](/docs/enterprise-kubernetes-operator/operations/database/#node-maintenance-and-disruption-budgets).

### Failover is never automatic
Expand Down
28 changes: 17 additions & 11 deletions documentation/enterprise-kubernetes-operator/operations/database.md
Original file line number Diff line number Diff line change
Expand Up @@ -405,27 +405,27 @@ into a new cluster.

## Node maintenance and disruption budgets

The default PodDisruptionBudget (PDB) uses `minAvailable: 1` for a singleton and
`instances-1` for a replicated cluster. A singleton therefore blocks voluntary
eviction and can make `kubectl drain` wait indefinitely.
A replicated cluster gets a default PodDisruptionBudget (PDB) with
`minAvailable: instances-1`, which permits one voluntary disruption at a time.

Before planned maintenance, choose one safe option:
A singleton gets no PDB by default. `kubectl drain` and managed node upgrades
evict its only pod, and the operator recreates it on another node in the same
zone as its volume. The database is down until the new pod is ready. To avoid
that outage, scale out and wait for a healthy, current replica before planned
maintenance.

1. scale out and wait for a healthy, current replica;
2. lower `spec.scheduling.podDisruptionBudget.minAvailable`; or
3. set `spec.scheduling.podDisruptionBudget.enabled: false` only after accepting
database downtime.
To make a singleton block voluntary eviction instead, enable the PDB. A drain of
its node then waits until you lower `minAvailable` or disable the PDB again:

```yaml
spec:
scheduling:
podDisruptionBudget:
enabled: true
minAvailable: 0
```

Restore the normal PDB after maintenance and verify the current-generation
writer contract: `Available=True/PrimaryReady`, `Progressing=False/Settled`, and
After maintenance, verify the current-generation writer contract:
`Available=True/PrimaryReady`, `Progressing=False/Settled`, and
`WriteHealthy=True/Healthy`.

A PDB protects only voluntary disruption. On node loss, the operator reports
Expand All @@ -435,6 +435,12 @@ the node/volume or follow the explicit
[promotion and failover](/docs/enterprise-kubernetes-operator/high-availability/#promotion-and-failover)
procedure.

A node that is `NotReady` while its machine keeps running leaves the pod stuck
until the machine is repaired or replaced. Turn on node auto-repair so this
happens without a person: AKS repairs unhealthy nodes automatically, and on EKS
you enable node auto repair on the managed node group. Once the node is
repaired or replaced, the pod runs again.

## Delete a database cluster

Deleting a `QuestDBCluster` removes operator-owned pods, Services, ConfigMaps,
Expand Down
Loading