Skip to content

Prepare kube-prometheus-stack 92.14.0 for DeepOps 26.12 - #1437

Open
dholt wants to merge 4 commits into
masterfrom
dholt/kps-92
Open

dholt wants to merge 4 commits into
masterfrom
dholt/kps-92

Conversation

@dholt

@dholt dholt commented Oct 27, 2026

Copy link
Copy Markdown
Contributor

Summary

Prepare the monitoring chart upgrade for DeepOps 26.12, not 26.11.

  • Bump kube-prometheus-stack to 92.14.0 (Prometheus Operator v0.94.2).
  • Use the pod's rotating projected token for authenticated control-plane scraping instead of creating a non-expiring service-account token Secret.
  • Document the 88.6.5 upgrade, exact bundled CRDs, reviewed saved-value merge, image migrations, Linux scheduling changes and rollback limitations.
  • Correct five component mirror overrides to the chart's effective image keys, preserving configured tags/digests.
  • Refuse unhealthy existing releases until operators follow the manual CRD-first recovery procedure. Missing releases use plain helm install; healthy releases continue to skip installation.
  • Cover all ten monitoring CRDs in the existing destructive reset path; that path remains explicitly forbidden for upgrades.

Validation

  • Full local validation suite: 148 tests pass, no skips, with the exact unpacked chart supplied for Helm render tests.
  • Required role lint: passed; shell syntax, YAML parsing and diff whitespace checks passed.
  • Independent review passed on the current head.
  • Disposable Kubernetes 1.31 test environments: fresh 92.14.0 and 88.6.5 → 92.14.0 upgrades proved all ten CRDs Established, operator reconciliation, positive/negative authorization checks, authenticated control-plane targets through projected-token rotation, preserved PVC/data and ingress hosts.
  • GPU-enabled fresh/upgrade tests proved live DCGM samples, eleven GPU dashboard panel queries, Grafana datasource queries and retained GPU history. All 29 chart dashboards plus the GPU dashboard loaded.

Evidence boundaries and limitations

Live chart/configuration tests used the preceding commit; the chart and both monitoring value files are unchanged in the final installer fix. Final mirror mapping and fail-closed installer behavior have behavioral and exact-chart render test coverage, not live installer coverage. Live tests used direct Helm and the documented CRD-first procedure.

The GPU test device could not expose NVLink or PCIe profiling counters; those panels had no data on either version. Existing non-GPU endpoint failures and node-exporter port conflicts were identical on the baseline and target, not new regressions. Multi-node placement, external scrapers and rollback were not live-tested. Run:ai's 88.6.5 numeric chart floor is satisfied; runtime compatibility is not certified.

Mirror mappings are render-verified for 92.14.0 only. Auxiliary images require separate site configuration. Existing site values must be reviewed and migrated; do not blindly reuse old chart defaults or delete CRDs for recovery.

dholt and others added 4 commits October 27, 2026 09:07
Use rotating projected credentials for control-plane scraping instead of a
non-expiring token Secret. Document pinned CRD migration from 88.6.5
and verify both site-value examples with offline Helm renders.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
Merge reviewed saved user values before target site settings, document
image-key migration, and cover token TLS wiring and all reset CRDs.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
Map installer mirror references to the effective chart image objects,
preserving registry ports, nested paths, tags and digests. Exercise the
installer arguments through the exact chart's offline render.

Refuse existing unhealthy releases and lookup errors before workload
changes. Use plain install for missing releases, leaving upgrades and
failed-release recovery to the reviewed exact-package CRD-first guide.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
Allow reviewed Helm-owned field conflicts with a dry-run-first retry.
Stop on failed recovery commands and require all ten CRD version
annotations to match the operator before upgrading workloads.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
@dholt
dholt requested a review from michael-balint October 27, 2026 22:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant