Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion gpu-operator/getting-started.rst
Original file line number Diff line number Diff line change
Expand Up @@ -421,8 +421,10 @@ To view all the options, run ``helm show values nvidia/gpu-operator``.
- ``{}``

* - ``dcgmExporter.enabled``
- By default, the Operator gathers GPU telemetry in Kubernetes using `DCGM Exporter <https://docs.nvidia.com/datacenter/cloud-native/gpu-telemetry/latest/dcgm-exporter.html>`_.
- By default, the Operator gathers GPU telemetry in Kubernetes using `DCGM Exporter <https://docs.nvidia.com/datacenter/cloud-native/gpu-telemetry/latest/dcgm-exporter.html>`_.
Set this value to ``false`` to disable it.
For all DCGM Exporter values, such as custom metrics files and the ``ServiceMonitor``, refer to
`DCGM Exporter Helm Values <https://docs.nvidia.com/datacenter/cloud-native/gpu-telemetry/latest/dcgm-exporter.html#dcgm-exporter-helm-values>`_.
Available values are ``true`` (default) or ``false``.
- ``true``

Expand Down
29 changes: 28 additions & 1 deletion gpu-operator/gpu-sharing.rst
Original file line number Diff line number Diff line change
Expand Up @@ -84,11 +84,38 @@ the mixed MIG strategy.
Limitations
===========

- DCGM-Exporter does not support associating metrics to containers when GPU time-slicing is enabled with the NVIDIA Kubernetes Device Plugin.
- The Operator does not monitor changes to a time-slicing config map.
Refer to :ref:`time-slicing-update-config-map`.


.. _time-slicing-gpu-metrics:

GPU Metrics with Time-Slicing
=============================

By default, DCGM Exporter does not associate GPU metrics with the pods that share a GPU through time-slicing or MPS.
To associate them, set the ``KUBERNETES_VIRTUAL_GPUS`` environment variable to ``true`` in ``dcgmExporter.env``.
For example, in the Helm values:

.. code-block:: yaml

dcgmExporter:
env:
- name: KUBERNETES_VIRTUAL_GPUS
value: "true"

Or patch the cluster policy.
The patch replaces the whole ``env`` list, so include any variables that you already set:

.. code-block:: console

$ kubectl patch clusterpolicies.nvidia.com/cluster-policy --type=merge \
-p '{"spec": {"dcgmExporter": {"env": [{"name": "KUBERNETES_VIRTUAL_GPUS", "value": "true"}]}}}'

For the other DCGM Exporter settings in the Operator, refer to
`DCGM Exporter with the NVIDIA GPU Operator <https://docs.nvidia.com/datacenter/cloud-native/gpu-telemetry/latest/dcgm-exporter.html>`__.


Changes to Node Labels
======================

Expand Down
4 changes: 4 additions & 0 deletions gpu-operator/life-cycle-policy.rst
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,10 @@ Refer to :ref:`Upgrading the NVIDIA GPU Operator` for more information.

**D** = Default driver, **R** = Recommended driver

.. TODO: When the next GPU Operator release is published, add a column for it.
Set DCGM Exporter to v4.7.0-4.9.0 and DCGM to 4.7.0-2 in the new column, or to the versions that the release ships.
DCGM Exporter 4.9.0 needs DCGM 4.7 or later, so the two rows must change together.

.. flat-table::
:header-rows: 2

Expand Down
72 changes: 72 additions & 0 deletions gpu-operator/release-notes.rst
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,78 @@ Refer to the :ref:`GPU Operator Component Matrix` for a list of software compone

----

.. DRAFT: Release notes for the next GPU Operator release, which ships DCGM Exporter 4.9.0.
Release notes are written when a release is published. Move this text into a new
version section above and replace the placeholders, then delete this comment.
This is a comment only because the GPU Operator version number and date are not known yet.

New Features
------------

* Updated software component versions:

- NVIDIA DCGM Exporter v4.7.0-4.9.0
- NVIDIA DCGM 4.7.0-2

* DCGM Exporter 4.9.0 renames its default metrics to the DCGM 4.7 field names and changes several units.
For example, ``DCGM_FI_DEV_GPU_UTIL`` in percent is now ``DCGM_FI_DEV_GPU_UTIL_RATIO``, a ratio from 0 to 1,
and the ``DCGM_FI_DRIVER_VERSION`` label is now ``DCGM_FI_SYSTEM_DRIVER_VERSION``.
Before you upgrade, update dashboards, alerts, recording rules, and autoscaler rules, such as HPA or KEDA, that use DCGM Exporter metrics,
and rescale thresholds for the unit changes, for example ``> 80`` becomes ``> 0.8`` for GPU utilization.
Re-import the Grafana dashboard from the DCGM Exporter repository.
For the full list of names and units, refer to the
`DCGM Exporter release notes <https://docs.nvidia.com/datacenter/dcgm/latest/release-notes/dcgm-exporter.html>`__.

Improvements
------------

* The Operator sets ``/etc/dcgm-exporter/default-counters.csv`` as the default DCGM Exporter metrics file.
(`GPU Operator PR #3110 <https://github.com/NVIDIA/gpu-operator/pull/3110>`__)

* On Red Hat OpenShift Container Platform, the ``accelerator_*`` recording rules read the new DCGM Exporter metric names and fall back to the old names,
so the OpenShift console dashboards keep working during the upgrade.
(`GPU Operator PR #3110 <https://github.com/NVIDIA/gpu-operator/pull/3110>`__)

* The DCGM Exporter daemon set has a startup probe that allows up to five minutes for the exporter to start before the liveness probe applies.
(`GPU Operator PR #3110 <https://github.com/NVIDIA/gpu-operator/pull/3110>`__)

* On Red Hat OpenShift Container Platform, ``accelerator_memory_used_bytes`` reports framebuffer memory used in bytes instead of memory-copy utilization,
and ``accelerator_memory_total_bytes`` is populated with the default metrics.
(`GPU Operator PR #3110 <https://github.com/NVIDIA/gpu-operator/pull/3110>`__)

* DCGM Exporter 4.9.0 stops at startup on more invalid settings.
An invalid value in ``dcgmExporter.env`` or ``dcgmExporter.args`` stops the pod with an error message in the pod log.

Known Issues
------------

* DCGM Exporter 4.9.0 needs a DCGM 4.7 or later host engine. With an older host engine, such as DCGM 4.6.1, the exporter exits at startup
with the error ``the DCGM hostengine refused these fields: ...``, which lists the fields the host engine does not support.
This affects clusters where the Operator runs the standalone DCGM host engine, that is ``dcgm.enabled=true``,
which is the default on OpenShift, or a ``GPUCluster`` with standalone DCGM:

- Upgrade the ``dcgm`` and ``dcgmExporter`` images together.
If you set ``dcgm.version`` or ``dcgmExporter.version``, change both.
- During the upgrade, the exporter pod on a node can restart a few times until the DCGM pod on that node runs 4.7.
- If you run DCGM on the host, upgrade it to 4.7 or later.
- If you must keep an older host engine, use a custom metrics file with the older field names.
The DCGM Exporter release notes list the old and new names.
This pairing is untested and unsupported.

* If you roll the Operator back to a release whose ``accelerator_*`` rules read only the old metric names,
also roll back DCGM Exporter, or the ``accelerator_*`` series are empty.
Roll back the ``dcgm`` and ``dcgmExporter`` images together, to the versions that Operator release ships.

Removals and Deprecations
-------------------------

* ``/etc/dcgm-exporter/dcp-metrics-included.csv`` is deprecated. It has the same metrics as ``default-counters.csv``.

* The older DCGM field names in custom metrics files still work but are deprecated. NVIDIA recommends the DCGM 4.7 names.
Refer to `Custom Metrics Files <https://docs.nvidia.com/datacenter/cloud-native/gpu-telemetry/latest/dcgm-exporter.html#dcgm-exporter-custom-metrics>`__.

* The ``DCGM_EXPORTER_CONFIGMAP_DATA`` environment variable is deprecated. Use ``dcgmExporter.config.name`` instead.

.. _v26.7.2:

26.7.2
Expand Down
48 changes: 18 additions & 30 deletions gpu-telemetry/about-telemetry.rst
Original file line number Diff line number Diff line change
Expand Up @@ -20,39 +20,27 @@
About GPU Telemetry
###################

Monitoring stacks usually consist of a collector, a time-series database to store metrics and a visualization layer.
A popular open-source stack is `Prometheus <https://prometheus.io/>`_ used along with `Grafana <https://grafana.com/>`_ as
the visualization tool to create rich dashboards. Prometheus also includes an `Alertmanager <https://github.com/prometheus/alertmanager>`_,
to create and manage alerts. Prometheus is deployed along with `kube-state-metrics <https://github.com/kubernetes/kube-state-metrics>`_ and
`node_exporter <https://github.com/prometheus/node_exporter>`_ to expose cluster-level metrics for Kubernetes API objects and node-level
metrics such as CPU utilization.
The NVIDIA GPU Operator deploys NVIDIA DCGM Exporter on every GPU node by default.
DCGM Exporter reads GPU metrics from `NVIDIA DCGM <https://developer.nvidia.com/dcgm>`__
and exposes them for `Prometheus <https://prometheus.io/>`__.
You can view the metrics in Grafana or, on Red Hat OpenShift, in the web console.

This documentation covers what the GPU Operator does for GPU telemetry:

To gather GPU telemetry in Kubernetes, its recommended to use DCGM Exporter. DCGM Exporter, based on `DCGM <https://developer.nvidia.com/dcgm>`_ exposes
GPU metrics for Prometheus and can be visualized using Grafana. DCGM Exporter is architected to take advantage of
``KubeletPodResources`` `API <https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/>`_ and exposes GPU metrics in a format that can be
scraped by Prometheus. A ``ServiceMonitor`` is also included to expose endpoints.
* :doc:`dcgm-exporter` describes the resources that the Operator creates,
the Operator values for DCGM Exporter, custom metrics files, Prometheus scraping, and a separate DCGM host engine.
* :external+ocp:doc:`enable-gpu-monitoring-dashboard` describes the GPU dashboard in the OpenShift web console.

.. note::
DCGM and DCGM Exporter are deployed by default with the `NVIDIA GPU Operator <https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/>`_.
For DCGM Exporter itself, such as standalone and Helm installs, the configuration file, command-line options,
and the list of metrics, refer to the
`DCGM Exporter documentation <https://docs.nvidia.com/datacenter/dcgm/latest/installation/install-dcgm-exporter.html>`__.


*************************
Benefits of GPU Telemetry
*************************

Understanding GPU usage provides important insights for IT administrators managing a data center.
Trends in GPU metrics correlate with workload behavior and make it possible to optimize resource allocation,
diagnose anomalies, and increase overall data center efficiency.
As GPUs become more mainstream, users would like to get access to GPU metrics to monitor GPU resources, just
like they do today for CPUs.

*************************
***********
Quick Links
*************************
***********

* `DCGM Exporter GitHub repository <https://github.com/NVIDIA/dcgm-exporter>`_
* `DCGM Documentation <https://docs.nvidia.com/datacenter/dcgm/latest/>`_
* `DCGM Exporter Documentation <https://docs.nvidia.com/datacenter/cloud-native/gpu-telemetry/dcgm-exporter.html>`_
* `Integrating GPU Telemetry into Kubernetes <https://docs.nvidia.com/datacenter/cloud-native/gpu-telemetry/integrating-telemetry-kubernetes.html>`_
* `NVIDIA GPU Operator Documentation <https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/>`_
* `NVIDIA GPU Operator documentation <https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/>`__
* `DCGM Exporter documentation <https://docs.nvidia.com/datacenter/dcgm/latest/installation/install-dcgm-exporter.html>`__
* `DCGM Exporter release notes <https://docs.nvidia.com/datacenter/dcgm/latest/release-notes/dcgm-exporter.html>`__
* `DCGM Exporter GitHub repository <https://github.com/NVIDIA/dcgm-exporter>`__
* `DCGM documentation <https://docs.nvidia.com/datacenter/dcgm/latest/>`__
Loading
Loading