diff --git a/gpu-operator/getting-started.rst b/gpu-operator/getting-started.rst index b15d04db4..53f60bd3a 100644 --- a/gpu-operator/getting-started.rst +++ b/gpu-operator/getting-started.rst @@ -421,8 +421,10 @@ To view all the options, run ``helm show values nvidia/gpu-operator``. - ``{}`` * - ``dcgmExporter.enabled`` - - By default, the Operator gathers GPU telemetry in Kubernetes using `DCGM Exporter `_. + - By default, the Operator gathers GPU telemetry in Kubernetes using `DCGM Exporter `_. Set this value to ``false`` to disable it. + For all DCGM Exporter values, such as custom metrics files and the ``ServiceMonitor``, refer to + `DCGM Exporter Helm Values `_. Available values are ``true`` (default) or ``false``. - ``true`` diff --git a/gpu-operator/gpu-sharing.rst b/gpu-operator/gpu-sharing.rst index 0c2ff5882..46eee6ff7 100644 --- a/gpu-operator/gpu-sharing.rst +++ b/gpu-operator/gpu-sharing.rst @@ -84,11 +84,38 @@ the mixed MIG strategy. Limitations =========== -- DCGM-Exporter does not support associating metrics to containers when GPU time-slicing is enabled with the NVIDIA Kubernetes Device Plugin. - The Operator does not monitor changes to a time-slicing config map. Refer to :ref:`time-slicing-update-config-map`. +.. _time-slicing-gpu-metrics: + +GPU Metrics with Time-Slicing +============================= + +By default, DCGM Exporter does not associate GPU metrics with the pods that share a GPU through time-slicing or MPS. +To associate them, set the ``KUBERNETES_VIRTUAL_GPUS`` environment variable to ``true`` in ``dcgmExporter.env``. +For example, in the Helm values: + +.. code-block:: yaml + + dcgmExporter: + env: + - name: KUBERNETES_VIRTUAL_GPUS + value: "true" + +Or patch the cluster policy. +The patch replaces the whole ``env`` list, so include any variables that you already set: + +.. code-block:: console + + $ kubectl patch clusterpolicies.nvidia.com/cluster-policy --type=merge \ + -p '{"spec": {"dcgmExporter": {"env": [{"name": "KUBERNETES_VIRTUAL_GPUS", "value": "true"}]}}}' + +For the other DCGM Exporter settings in the Operator, refer to +`DCGM Exporter with the NVIDIA GPU Operator `__. + + Changes to Node Labels ====================== diff --git a/gpu-operator/life-cycle-policy.rst b/gpu-operator/life-cycle-policy.rst index 4744f24df..36cbfe447 100644 --- a/gpu-operator/life-cycle-policy.rst +++ b/gpu-operator/life-cycle-policy.rst @@ -84,6 +84,10 @@ Refer to :ref:`Upgrading the NVIDIA GPU Operator` for more information. **D** = Default driver, **R** = Recommended driver +.. TODO: When the next GPU Operator release is published, add a column for it. + Set DCGM Exporter to v4.7.0-4.9.0 and DCGM to 4.7.0-2 in the new column, or to the versions that the release ships. + DCGM Exporter 4.9.0 needs DCGM 4.7 or later, so the two rows must change together. + .. flat-table:: :header-rows: 2 diff --git a/gpu-operator/release-notes.rst b/gpu-operator/release-notes.rst index b99a7f6b8..1aa252140 100644 --- a/gpu-operator/release-notes.rst +++ b/gpu-operator/release-notes.rst @@ -33,6 +33,78 @@ Refer to the :ref:`GPU Operator Component Matrix` for a list of software compone ---- +.. DRAFT: Release notes for the next GPU Operator release, which ships DCGM Exporter 4.9.0. + Release notes are written when a release is published. Move this text into a new + version section above and replace the placeholders, then delete this comment. + This is a comment only because the GPU Operator version number and date are not known yet. + + New Features + ------------ + + * Updated software component versions: + + - NVIDIA DCGM Exporter v4.7.0-4.9.0 + - NVIDIA DCGM 4.7.0-2 + + * DCGM Exporter 4.9.0 renames its default metrics to the DCGM 4.7 field names and changes several units. + For example, ``DCGM_FI_DEV_GPU_UTIL`` in percent is now ``DCGM_FI_DEV_GPU_UTIL_RATIO``, a ratio from 0 to 1, + and the ``DCGM_FI_DRIVER_VERSION`` label is now ``DCGM_FI_SYSTEM_DRIVER_VERSION``. + Before you upgrade, update dashboards, alerts, recording rules, and autoscaler rules, such as HPA or KEDA, that use DCGM Exporter metrics, + and rescale thresholds for the unit changes, for example ``> 80`` becomes ``> 0.8`` for GPU utilization. + Re-import the Grafana dashboard from the DCGM Exporter repository. + For the full list of names and units, refer to the + `DCGM Exporter release notes `__. + + Improvements + ------------ + + * The Operator sets ``/etc/dcgm-exporter/default-counters.csv`` as the default DCGM Exporter metrics file. + (`GPU Operator PR #3110 `__) + + * On Red Hat OpenShift Container Platform, the ``accelerator_*`` recording rules read the new DCGM Exporter metric names and fall back to the old names, + so the OpenShift console dashboards keep working during the upgrade. + (`GPU Operator PR #3110 `__) + + * The DCGM Exporter daemon set has a startup probe that allows up to five minutes for the exporter to start before the liveness probe applies. + (`GPU Operator PR #3110 `__) + + * On Red Hat OpenShift Container Platform, ``accelerator_memory_used_bytes`` reports framebuffer memory used in bytes instead of memory-copy utilization, + and ``accelerator_memory_total_bytes`` is populated with the default metrics. + (`GPU Operator PR #3110 `__) + + * DCGM Exporter 4.9.0 stops at startup on more invalid settings. + An invalid value in ``dcgmExporter.env`` or ``dcgmExporter.args`` stops the pod with an error message in the pod log. + + Known Issues + ------------ + + * DCGM Exporter 4.9.0 needs a DCGM 4.7 or later host engine. With an older host engine, such as DCGM 4.6.1, the exporter exits at startup + with the error ``the DCGM hostengine refused these fields: ...``, which lists the fields the host engine does not support. + This affects clusters where the Operator runs the standalone DCGM host engine, that is ``dcgm.enabled=true``, + which is the default on OpenShift, or a ``GPUCluster`` with standalone DCGM: + + - Upgrade the ``dcgm`` and ``dcgmExporter`` images together. + If you set ``dcgm.version`` or ``dcgmExporter.version``, change both. + - During the upgrade, the exporter pod on a node can restart a few times until the DCGM pod on that node runs 4.7. + - If you run DCGM on the host, upgrade it to 4.7 or later. + - If you must keep an older host engine, use a custom metrics file with the older field names. + The DCGM Exporter release notes list the old and new names. + This pairing is untested and unsupported. + + * If you roll the Operator back to a release whose ``accelerator_*`` rules read only the old metric names, + also roll back DCGM Exporter, or the ``accelerator_*`` series are empty. + Roll back the ``dcgm`` and ``dcgmExporter`` images together, to the versions that Operator release ships. + + Removals and Deprecations + ------------------------- + + * ``/etc/dcgm-exporter/dcp-metrics-included.csv`` is deprecated. It has the same metrics as ``default-counters.csv``. + + * The older DCGM field names in custom metrics files still work but are deprecated. NVIDIA recommends the DCGM 4.7 names. + Refer to `Custom Metrics Files `__. + + * The ``DCGM_EXPORTER_CONFIGMAP_DATA`` environment variable is deprecated. Use ``dcgmExporter.config.name`` instead. + .. _v26.7.2: 26.7.2 diff --git a/gpu-telemetry/about-telemetry.rst b/gpu-telemetry/about-telemetry.rst index eded05eba..df5ececab 100644 --- a/gpu-telemetry/about-telemetry.rst +++ b/gpu-telemetry/about-telemetry.rst @@ -20,39 +20,27 @@ About GPU Telemetry ################### -Monitoring stacks usually consist of a collector, a time-series database to store metrics and a visualization layer. -A popular open-source stack is `Prometheus `_ used along with `Grafana `_ as -the visualization tool to create rich dashboards. Prometheus also includes an `Alertmanager `_, -to create and manage alerts. Prometheus is deployed along with `kube-state-metrics `_ and -`node_exporter `_ to expose cluster-level metrics for Kubernetes API objects and node-level -metrics such as CPU utilization. +The NVIDIA GPU Operator deploys NVIDIA DCGM Exporter on every GPU node by default. +DCGM Exporter reads GPU metrics from `NVIDIA DCGM `__ +and exposes them for `Prometheus `__. +You can view the metrics in Grafana or, on Red Hat OpenShift, in the web console. +This documentation covers what the GPU Operator does for GPU telemetry: -To gather GPU telemetry in Kubernetes, its recommended to use DCGM Exporter. DCGM Exporter, based on `DCGM `_ exposes -GPU metrics for Prometheus and can be visualized using Grafana. DCGM Exporter is architected to take advantage of -``KubeletPodResources`` `API `_ and exposes GPU metrics in a format that can be -scraped by Prometheus. A ``ServiceMonitor`` is also included to expose endpoints. +* :doc:`dcgm-exporter` describes the resources that the Operator creates, + the Operator values for DCGM Exporter, custom metrics files, Prometheus scraping, and a separate DCGM host engine. +* :external+ocp:doc:`enable-gpu-monitoring-dashboard` describes the GPU dashboard in the OpenShift web console. -.. note:: - DCGM and DCGM Exporter are deployed by default with the `NVIDIA GPU Operator `_. +For DCGM Exporter itself, such as standalone and Helm installs, the configuration file, command-line options, +and the list of metrics, refer to the +`DCGM Exporter documentation `__. - -************************* -Benefits of GPU Telemetry -************************* - -Understanding GPU usage provides important insights for IT administrators managing a data center. -Trends in GPU metrics correlate with workload behavior and make it possible to optimize resource allocation, -diagnose anomalies, and increase overall data center efficiency. -As GPUs become more mainstream, users would like to get access to GPU metrics to monitor GPU resources, just -like they do today for CPUs. - -************************* +*********** Quick Links -************************* +*********** -* `DCGM Exporter GitHub repository `_ -* `DCGM Documentation `_ -* `DCGM Exporter Documentation `_ -* `Integrating GPU Telemetry into Kubernetes `_ -* `NVIDIA GPU Operator Documentation `_ +* `NVIDIA GPU Operator documentation `__ +* `DCGM Exporter documentation `__ +* `DCGM Exporter release notes `__ +* `DCGM Exporter GitHub repository `__ +* `DCGM documentation `__ diff --git a/gpu-telemetry/dcgm-exporter.rst b/gpu-telemetry/dcgm-exporter.rst index d6d936ef6..bbaa59fa8 100644 --- a/gpu-telemetry/dcgm-exporter.rst +++ b/gpu-telemetry/dcgm-exporter.rst @@ -16,255 +16,357 @@ .. headings (h1/h2/h3/h4/h5) are # * = - -.. Date: April 13 2021 -.. Author: pramarao +.. _dcgm-exporter-gpu-operator: -############# -DCGM Exporter -############# +########################################## +DCGM Exporter with the NVIDIA GPU Operator +########################################## -************ -Introduction -************ +`DCGM Exporter `__ reads GPU metrics from +`NVIDIA DCGM `__ and exposes them for Prometheus at ``/metrics``. +The NVIDIA GPU Operator deploys DCGM Exporter on every GPU node by default. -`DCGM-Exporter `__ is a tool based on the -Go APIs to `NVIDIA DCGM `__ that allows you to gather -GPU metrics and understand workload behavior or monitor GPUs in clusters. DCGM Exporter is -written in Go and exposes GPU metrics at an HTTP endpoint (``/metrics``) for monitoring solutions -such as Prometheus. +This page covers what the GPU Operator does for DCGM Exporter and how you configure it through the Operator. +For DCGM Exporter itself, refer to the DCGM Exporter documentation: -For information on the profiling metrics available from DCGM, refer to -:external+dcgm:ref:`profiling` in the DCGM documentation. +* `Install DCGM Exporter `__, + including standalone containers, packages, and the exporter's own Helm chart. +* `Command-line and YAML reference `__ + for options, environment variables, and the configuration file. +* `DCGM Exporter metrics `__ + for metric names, units, labels, and MIG behavior. +* `Configure Prometheus for DCGM Exporter `__. +* `DCGM Exporter release notes `__, + including the metric name changes in DCGM Exporter 4.9.0. -.. TODO: include a high-level diagram of the dcgm-exporter stack in Kubernetes -You can run DCGM Exporter as a standalone container or deployed as a -daemonset on GPU nodes in a Kubernetes cluster. +************************* +What the Operator Deploys +************************* -Because DCGM Exporter starts `nv-hostengine` as an embedded process (for collecting metrics), -appropriate configuration options should be used if DCGM Exporter is run on systems (such as -NVIDIA DGX) that have DCGM (or rather `nv-hostengine`) running. +The Operator creates the following resources in the Operator namespace, such as ``gpu-operator``: -********************* -Running DCGM Exporter -********************* +.. list-table:: + :widths: 30 70 + :header-rows: 1 -The DCGM Exporter container can be run using a container engine such as Docker. In this mode, DCGM Exporter -starts `nv-hostengine` as an embedded process and starts publishing metrics: + * - Resource + - Description -.. image:: graphics/dcgm-exporter_embedded.png - :width: 800 + * - DaemonSet ``nvidia-dcgm-exporter`` + - Runs one DCGM Exporter pod on each node that has the ``nvidia.com/gpu.deploy.dcgm-exporter=true`` label. + The pods have the ``app=nvidia-dcgm-exporter`` label. -.. code-block:: console + * - Service ``nvidia-dcgm-exporter`` + - Exposes port ``9400``, named ``gpu-metrics``. - $ DCGM_EXPORTER_VERSION=2.1.4-2.3.1 && \ - docker run -d --rm \ - --gpus all \ - --net host \ - --cap-add SYS_ADMIN \ - nvcr.io/nvidia/k8s/dcgm-exporter:${DCGM_EXPORTER_VERSION}-ubuntu20.04 \ - -f /etc/dcgm-exporter/dcp-metrics-included.csv + * - ServiceMonitor ``nvidia-dcgm-exporter`` + - Tells the Prometheus Operator to scrape ``/metrics`` on the ``gpu-metrics`` port. + The GPU Operator creates it only when ``dcgmExporter.serviceMonitor.enabled`` is ``true`` + and the ``ServiceMonitor`` custom resource definition exists in the cluster. -Retrieve the metrics: + * - PrometheusRule ``nvidia-dcgm-exporter`` + - OpenShift only. + Records the ``accelerator_*`` series that the OpenShift console uses. + Refer to :external+ocp:ref:`enable-gpu-monitoring-dashboard`. -.. code-block:: console +By default, DCGM Exporter runs an embedded DCGM host engine inside its container. + +The Operator sets the following environment variables on the exporter container: + +* ``DCGM_EXPORTER_LISTEN=:9400`` +* ``DCGM_EXPORTER_KUBERNETES=true``, which adds pod, namespace, and container labels to the metrics of GPUs that pods use. +* ``DCGM_EXPORTER_COLLECTORS=/etc/dcgm-exporter/default-counters.csv``, the default metrics file in the image. + When you supply a custom metrics file, the value is ``/etc/dcgm-exporter/dcgm-metrics.csv``. +* ``DCGM_REMOTE_HOSTENGINE_INFO=nvidia-dcgm:5555``, only when the standalone DCGM host engine is enabled. + For a ``GPUCluster`` with standalone DCGM, the value is ``nvidia-dcgm-dra:5555``. - $ curl localhost:9400/metrics +Variables in ``dcgmExporter.env`` are applied last and replace these values. +Environment variables also take precedence over the exporter's configuration file, ``/etc/dcgm-exporter/config.yaml``, in the image. + +DCGM Exporter 4.9.0 stops at startup on more invalid settings. +An invalid value in ``dcgmExporter.env`` or ``dcgmExporter.args`` stops the pod with an error message. +To see the message, view the pod logs: .. code-block:: console - # HELP DCGM_FI_DEV_SM_CLOCK SM clock frequency (in MHz). - # TYPE DCGM_FI_DEV_SM_CLOCK gauge - # HELP DCGM_FI_DEV_MEM_CLOCK Memory clock frequency (in MHz). - # TYPE DCGM_FI_DEV_MEM_CLOCK gauge - # HELP DCGM_FI_DEV_MEMORY_TEMP Memory temperature (in C). - # TYPE DCGM_FI_DEV_MEMORY_TEMP gauge - ... - DCGM_FI_DEV_SM_CLOCK{gpu="0", UUID="GPU-604ac76c-d9cf-fef3-62e9-d92044ab6e52"} 139 - DCGM_FI_DEV_MEM_CLOCK{gpu="0", UUID="GPU-604ac76c-d9cf-fef3-62e9-d92044ab6e52"} 405 - DCGM_FI_DEV_MEMORY_TEMP{gpu="0", UUID="GPU-604ac76c-d9cf-fef3-62e9-d92044ab6e52"} 9223372036854775794 - ... - -*************************** -DCGM-Exporter Customization -*************************** - -DCGM-Exporter has various options for adjusting its default behavior. Each option supports both a command-line flag and environment variable. - -+----------------------------------+----------------------------------------------------------------------+---------------------+----------------------------------+ -| Environment Variable | Description | Command-Line Flag | Value | -+==================================+======================================================================+=====================+==================================+ -| ``$DCGM_EXPORTER_COLLECTORS`` | Path to file containing DCGM fields to collect. Default: | ``-f`` | File Path | -| | "/etc/dcgm-exporter/default-counters.csv" | | | -+----------------------------------+----------------------------------------------------------------------+---------------------+----------------------------------+ -| ``$DCGM_EXPORTER_LISTEN`` | Address of listening http server. Default: ":9400" | ``-a`` | Address | -+----------------------------------+----------------------------------------------------------------------+---------------------+----------------------------------+ -| ``$DCGM_EXPORTER_INTERVAL`` | Interval of time at which point metrics are collected. Unit is | ``-c`` | Interval | -| | milliseconds. Default: 30000 | | | -+----------------------------------+----------------------------------------------------------------------+---------------------+----------------------------------+ -| ``$DCGM_EXPORTER_KUBERNETES`` | Enable kubernetes mapping metrics to kubernetes pods. Default: false | ``-k`` | Boolean | -+----------------------------------+----------------------------------------------------------------------+---------------------+----------------------------------+ -| ``$DCGM_EXPORTER_CONFIGMAP_DATA``| ConfigMap namespace and name containing DCGM fields to collect. | ``-m`` | Namespace:Name | -| | Default: "none" | | | -+----------------------------------+----------------------------------------------------------------------+---------------------+----------------------------------+ -| ``$DCGM_REMOTE_HOSTENGINE_INFO`` | Connect to remote hostengine at Host:Port. Default: NA | ``-r`` | Host:Port | -| | (dcgm-exporter will start in embedded mode) | | | -+----------------------------------+----------------------------------------------------------------------+---------------------+----------------------------------+ -| ``$DCGM_EXPORTER_DEVICES_STR`` | Specify which devices to monitor. Default: all GPU instances in | ``-d`` | Device String | -| | MIG mode, all GPUs if MIG disabled. | | (see following note) | -+----------------------------------+----------------------------------------------------------------------+---------------------+----------------------------------+ + $ kubectl logs -n gpu-operator -l app=nvidia-dcgm-exporter -.. note:: - Device String Syntax: ``[f] | [g[:id1[,-id2]]] | [i[:id1[,-id2]]]`` - If an id list is used, then devices with matching IDs must exist on the system. For example: +************************* +DCGM Exporter Helm Values +************************* - * ``f`` = Monitor all GPUs if MIG is disabled, or all GPU instances if MIG is enabled - * ``g`` = Monitor all GPUs - * ``i`` = Monitor all GPU instances - * ``g:0,1`` = monitor GPUs 0 and 1 - * ``i:0,2-4`` = monitor GPU instances 0, 2, 3, and 4. +Set these values when you install or upgrade the Operator with Helm. +The same fields are under ``spec.dcgmExporter`` in the ``ClusterPolicy`` resource, +except ``config.create`` and ``config.data``, which only the Helm chart uses. -.. warning:: - 1. ``-i`` cannot be specified unless MIG mode is enabled. +.. list-table:: + :widths: 35 45 20 + :header-rows: 1 - 2. Any time indices are specified, those indices must exist on the system. + * - Parameter + - Description + - Default - 3. In MIG mode, only ``f`` or ``i`` with a range can be specified. GPUs are not assigned to pods and therefore reporting must occur at the GPU instance level. (default: ``f``) + * - ``dcgmExporter.enabled`` + - When set to ``true``, the Operator deploys DCGM Exporter. + - ``true`` + + * - ``dcgmExporter.repository``, ``image``, ``version`` + - Image of DCGM Exporter. + The default version is the one that the Operator release ships. + - Refer to ``helm show values`` + * - ``dcgmExporter.env`` + - List of environment variables, as ``name`` and ``value`` pairs, to set in the DCGM Exporter container. + - ``[]`` + + * - ``dcgmExporter.args`` + - List of command-line arguments to pass to DCGM Exporter. + - ``none`` + * - ``dcgmExporter.resources`` + - Resource requests and limits for the DCGM Exporter pods. + - ``{}`` -Connecting to an existing DCGM agent -====================================== + * - ``dcgmExporter.annotations`` + - Map of custom annotations to add to the DCGM Exporter pods. + - ``{}`` -In this scenario, system images include DCGM and have `nv-hostengine` running already. Examples include -the DGX systems that bundles drivers, DCGM, etc. in the system image. To avoid any compatibility issues, -it is recommended to have DCGM Exporter connect to the existing `nv-hostengine` daemon to gather/publish -GPU telemetry data. + * - ``dcgmExporter.config.name`` + - Name of a ConfigMap that holds a custom metrics file. + Refer to :ref:`dcgm-exporter-custom-metrics`. + - ``none`` -.. warning:: + * - ``dcgmExporter.config.create`` + - Helm only. + When set to ``true`` and ``config.data`` is set, the chart creates the ConfigMap named by ``config.name``. + - ``none`` - The DCGM Exporter container image includes a DCGM client library (``libdcgm.so``) to communicate with - `nv-hostengine`. In this deployment scenario we have DCGM Exporter (or rather ``libdcgm.so``) connect - to an existing `nv-hostengine` running on the host. The DCGM client library uses an internal protocol to exchange - information with `nv-hostengine`. To avoid any potential incompatibilities between the container image's DCGM client library - and the host's `nv-hostengine`, it is strongly recommended to use a version of DCGM on which DCGM Exporter is based is - greater than or equal to (but not less than) the version of DCGM running on the host. This can be easily determined by - comparing the version tags of the DCGM Exporter image and by running ``nv-hostengine --version`` on the host. + * - ``dcgmExporter.config.data`` + - Helm only. + Contents of the metrics file for the ConfigMap that the chart creates. + - ``none`` -In this scenario, we use the ``-r`` option to connect to an existing `nv-hostengine` process: + * - ``dcgmExporter.serviceMonitor.enabled`` + - When set to ``true``, the Operator creates a ``ServiceMonitor`` for DCGM Exporter. + - ``true`` -.. code-block:: console + * - ``dcgmExporter.serviceMonitor.interval`` + - Interval at which Prometheus scrapes DCGM Exporter. + - ``15s`` - $ DCGM_EXPORTER_VERSION=2.1.4-2.3.1 && - docker run -d --rm \ - --gpus all \ - --net host \ - --cap-add SYS_ADMIN \ - nvcr.io/nvidia/k8s/dcgm-exporter:${DCGM_EXPORTER_VERSION}-ubuntu20.04 \ - -r localhost:5555 -f /etc/dcgm-exporter/dcp-metrics-included.csv + * - ``dcgmExporter.serviceMonitor.scrapeTimeout`` + - Timeout for one scrape. The value must not be greater than the interval. + - ``10s`` -Connecting to a DCGM standalone container -========================================= + * - ``dcgmExporter.serviceMonitor.honorLabels`` + - When set to ``true``, Prometheus keeps the metric labels when they collide with target labels. + - ``false`` -In this scenario the DCGM `nv-hostengine` runs in a separate container on the same host making its client port available to -DCGM-Exporter as well as dcgmi client commands. + * - ``dcgmExporter.serviceMonitor.additionalLabels`` + - Map of labels to add to the ``ServiceMonitor``. + - ``{}`` -.. image:: graphics/dcgm_and_dcgm-exporter.png - :width: 800 + * - ``dcgmExporter.serviceMonitor.relabelings`` + - List of Prometheus relabeling rules for the scrape target. + - ``[]`` -.. warning:: + * - ``dcgmExporter.serviceMonitor.metricRelabelings`` + - List of Prometheus relabeling rules for scraped samples, applied before ingestion. + - ``[]`` - Similar to the warning when connecting to an existing DCGM agent, the DCGM Exporter container image includes a - DCGM client library (``libdcgm.so``) to communicate with `nv-hostengine` running in a separate container. - The DCGM client library in use by DCGM-Exporter uses an internal protocol to exchange information with `nv-hostengine`. - To avoid any potential incompatibilities between the container image's DCGM client library - and the standalone DCGM container's `nv-hostengine`, it is strongly recommended to ensure the version of DCGM on which - DCGM Exporter is based is greater than or equal to (but not less than) the version of DCGM running in the standalone - container. This can be easily determined by comparing the version tags of the DCGM Exporter and `dcgm` standalone image. + * - ``dcgmExporter.service.type`` + - Type of the ``nvidia-dcgm-exporter`` service. + - ``ClusterIP`` -First, start the standalone DCGM container with the `nv-hostengine` port available to external applications: + * - ``dcgmExporter.service.internalTrafficPolicy`` + - `Internal traffic policy `__ + of the ``nvidia-dcgm-exporter`` service. + - ``Cluster`` -.. code-block:: console + * - ``dcgmExporter.hostNetwork`` + - When set to ``true``, the DCGM Exporter pods use the host's network namespace. + - ``false`` - $DCGM_VERSION=2.2.9 && - docker run -d --rm \ - --gpus all \ - --cap-add SYS_ADMIN \ - -p 5555:5555 \ - nvidia/dcgm:${DCGM_VERSION}-ubuntu20.04 + * - ``dcgmExporter.hostPID`` + - When set to ``true``, the DCGM Exporter pods use the host's process ID namespace. + - ``false`` -Second, start the dcgm-exporter container with ``r`` option to connect to an existing `nv-hostengine` port: + * - ``dcgmExporter.enablePodLabels`` + - When set to ``true``, pod labels are added to the GPU metrics. + The Operator grants the exporter read access to pods. + - ``false`` -.. code-block:: console + * - ``dcgmExporter.enablePodUID`` + - When set to ``true``, the pod UID is added to the GPU metrics. + The Operator grants the exporter read access to pods. + - ``false`` - $DCGM_EXPORTER_VERSION=2.2.9-2.5.0 && - docker run -d --rm \ - --gpus all \ - --net host \ - --cap-add SYS_ADMIN \ - nvcr.io/nvidia/k8s/dcgm-exporter:${DCGM_EXPORTER_VERSION}-ubuntu20.04 \ - -r localhost:5555 -f /etc/dcgm-exporter/dcp-metrics-included.csv + * - ``dcgmExporter.podLabelAllowlistRegex`` + - List of regular expressions that select which pod labels are added. + Use it to limit the number of Prometheus series. + - ``none`` -In this scenario `dcgmi` commands run on the host will also connect to the `nv-hostengine` running in the standalone -DCGM container: + * - ``dcgmExporter.hpcJobMapping.enabled`` + - When set to ``true``, GPU metrics get HPC job labels from files that a workload manager, such as Slurm, writes on the node. + The Operator mounts the job mapping directory from the host. + - ``none`` -.. code-block:: console + * - ``dcgmExporter.hpcJobMapping.directory`` + - Host directory that holds the HPC job mapping files. + - ``/var/lib/dcgm-exporter/job-mapping`` + + * - ``dcgm.enabled`` + - When set to ``true``, the Operator deploys the standalone DCGM host engine. + Refer to :ref:`dcgm-exporter-hostengine`. + - ``false`` + +For the other Operator options, refer to :external+gpuop:ref:`chart-customization-options`. + + +.. _dcgm-exporter-custom-metrics: + +******************** +Custom Metrics Files +******************** + +By default, DCGM Exporter collects the metrics in ``/etc/dcgm-exporter/default-counters.csv`` in the image. +To collect a different set, supply your own metrics file in a ConfigMap. +For the file format and the available metrics, refer to +`DCGM Exporter metrics `__. + +DCGM Exporter 4.9.0 uses the DCGM 4.7 field names, such as ``DCGM_FI_DEV_GPU_UTIL_RATIO``. +It still loads the older names, such as ``DCGM_FI_DEV_GPU_UTIL``, but they are deprecated. +A metric that uses an older name keeps that name and its older unit. + +#. Create a ConfigMap in the Operator namespace. + Store the file under the key ``dcgm-metrics.csv``: + + .. code-block:: console + + $ kubectl create configmap custom-dcgm-exporter-metrics -n gpu-operator \ + --from-file=dcgm-metrics.csv=./dcgm-metrics.csv + +#. Set ``dcgmExporter.config.name`` to the ConfigMap name. + With Helm: + + .. code-block:: console + + $ helm upgrade gpu-operator nvidia/gpu-operator -n gpu-operator \ + --reuse-values --set dcgmExporter.config.name=custom-dcgm-exporter-metrics + + Or patch the cluster policy: + + .. code-block:: console + + $ kubectl patch clusterpolicies.nvidia.com/cluster-policy --type=merge \ + -p '{"spec": {"dcgmExporter": {"config": {"name": "custom-dcgm-exporter-metrics"}}}}' - dcgmi discovery -l - 1 GPU found. - +--------+----------------------------------------------------------------------+ - | GPU ID | Device Information | - +--------+----------------------------------------------------------------------+ - | 0 | Name: Quadro RTX 6000 | - | | PCI Bus ID: 00000000:65:00.0 | - | | Device UUID: GPU-2f6576bf-3c29-1fbb-068d-e74c4a97f0c5 | - +--------+----------------------------------------------------------------------+ +The Operator mounts the file at ``/etc/dcgm-exporter/dcgm-metrics.csv`` +and sets ``DCGM_EXPORTER_COLLECTORS`` to that path. +The ConfigMap replaces only the metrics file. +The configuration file in the image still applies. + +Instead of the first step, you can have the Helm chart create the ConfigMap. +Set ``dcgmExporter.config.create=true`` and put the file contents in ``dcgmExporter.config.data``. + +.. note:: + + The Operator mounts the file with ``subPath``, so Kubernetes does not update the file in running pods when you edit the ConfigMap. + After you change the ConfigMap, restart the DCGM Exporter pods: + + .. code-block:: console + + $ kubectl delete pod -n gpu-operator -l app=nvidia-dcgm-exporter + +Do not use the ``DCGM_EXPORTER_CONFIGMAP_DATA`` environment variable to read metrics from a ConfigMap. +It is deprecated in DCGM Exporter 4.9.0 and logs a warning at each start. + + +.. _dcgm-exporter-prometheus-scrape: ******************************** -Multi-Instance GPU (MIG) Support +Collecting Metrics in Prometheus ******************************** -The new Multi-Instance GPU (MIG) feature allows the GPUs based on the NVIDIA Ampere architecture to be -securely partitioned into up to seven separate GPU Instances for CUDA applications, providing multiple users -with separate GPU resources for optimal GPU utilization. +If your cluster runs the `Prometheus Operator `__, +for example through the ``kube-prometheus-stack`` Helm chart, +install it, or at least its custom resource definitions, before the GPU Operator. +The GPU Operator then creates the ``nvidia-dcgm-exporter`` ``ServiceMonitor``. -For more information on MIG, refer to the MIG `User Guide `_. +Prometheus must select the ``ServiceMonitor``. +By default, ``kube-prometheus-stack`` selects only ``ServiceMonitor`` objects that have its ``release`` label. +Do one of the following: -.. note:: +* Add the label through the GPU Operator: - Support for MIG in DCGM Exporter was added starting with ``2.4.0-rc.2``. Replace the container image with this tag in the - command line examples above: ``2.1.8-2.4.0-rc.2-ubuntu20.04``. If you are connecting to an existing DCGM on the host system, - ensure that you upgrade to at least 2.1.8 on the host system. + .. code-block:: console -DCGM Exporter publishes metrics for both the entire GPU as well as individual MIG devices (or GPU instances) -as can be seen in the output below: + $ helm upgrade gpu-operator nvidia/gpu-operator -n gpu-operator \ + --reuse-values --set dcgmExporter.serviceMonitor.additionalLabels.release= -.. code-block:: console +* Configure Prometheus to select all ``ServiceMonitor`` objects. + In ``kube-prometheus-stack``, set ``prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false``. + +To check that Prometheus collects the metrics, query a default metric, such as ``DCGM_FI_DEV_GPU_UTIL_RATIO``. +DCGM Exporter 4.9.0 and later use this name, and the value is a ratio from 0 to 1. +Earlier versions use ``DCGM_FI_DEV_GPU_UTIL``, in percent. + +If you do not use the Prometheus Operator, scrape the ``nvidia-dcgm-exporter`` service on port ``9400``. +Refer to `Configure Prometheus for DCGM Exporter `__. + +For a Grafana dashboard, import ``dcgm-exporter-dashboard.json`` from the +`DCGM Exporter repository `__. +The dashboard on the ``main`` branch uses the metric names of DCGM Exporter 4.9.0 and later. +For an earlier version, use the file from the matching release tag. +On OpenShift, refer to :external+ocp:ref:`enable-gpu-monitoring-dashboard`. + + +******************************* +Metrics for Shared GPUs and MIG +******************************* + +With GPU time-slicing or MPS, set ``KUBERNETES_VIRTUAL_GPUS=true`` in ``dcgmExporter.env`` +so that the metrics are associated with the pods that share the GPU. +Refer to :external+gpuop:ref:`gpu-sharing`. + +With MIG, DCGM Exporter reports metrics for each GPU instance. +The MIG Manager stops the DCGM Exporter pods while it changes the MIG configuration of a node. +For the MIG labels and metrics, refer to +`DCGM Exporter metrics `__. + + +.. _dcgm-exporter-hostengine: + +********************************* +Using a Separate DCGM Host Engine +********************************* + +By default, each DCGM Exporter pod runs its own embedded DCGM host engine. +You can use a separate host engine instead: + +* **Standalone DCGM from the Operator.** + Set ``dcgm.enabled=true``. + The Operator deploys the ``nvidia-dcgm`` daemon set and points DCGM Exporter at the ``nvidia-dcgm`` service on port ``5555``. + The default ``ClusterPolicy`` on OpenShift sets ``dcgm.enabled`` to ``true``. + +* **DCGM on the host**, such as on NVIDIA DGX systems. + Keep ``dcgm.enabled=false`` and point DCGM Exporter at the host engine on the node: + + .. code-block:: yaml + + dcgmExporter: + hostNetwork: true + env: + - name: DCGM_REMOTE_HOSTENGINE_INFO + value: localhost:5555 - DCGM_FI_DEV_SM_CLOCK{gpu="0",UUID="GPU-34319582-d595-d1c7-d1d2-179bcfa61660",device="nvidia0",Hostname="ub20-a100-k8s"} 1215 - DCGM_FI_DEV_MEM_CLOCK{gpu="0",UUID="GPU-34319582-d595-d1c7-d1d2-179bcfa61660",device="nvidia0",Hostname="ub20-a100-k8s"} 1215 - DCGM_FI_DEV_MEMORY_TEMP{gpu="0",UUID="GPU-34319582-d595-d1c7-d1d2-179bcfa61660",device="nvidia0",Hostname="ub20-a100-k8s"} 69 - DCGM_FI_DEV_GPU_TEMP{gpu="0",UUID="GPU-34319582-d595-d1c7-d1d2-179bcfa61660",device="nvidia0",Hostname="ub20-a100-k8s"} 61 - DCGM_FI_DEV_POWER_USAGE{gpu="0",UUID="GPU-34319582-d595-d1c7-d1d2-179bcfa61660",device="nvidia0",Hostname="ub20-a100-k8s"} 409.692000 - DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION{gpu="0",UUID="GPU-34319582-d595-d1c7-d1d2-179bcfa61660",device="nvidia0",Hostname="ub20-a100-k8s"} 319159391 - DCGM_FI_DEV_PCIE_REPLAY_COUNTER{gpu="0",UUID="GPU-34319582-d595-d1c7-d1d2-179bcfa61660",device="nvidia0",Hostname="ub20-a100-k8s"} 0 - DCGM_FI_DEV_XID_ERRORS{gpu="0",UUID="GPU-34319582-d595-d1c7-d1d2-179bcfa61660",device="nvidia0",Hostname="ub20-a100-k8s"} 0 - DCGM_FI_DEV_FB_FREE{gpu="0",UUID="GPU-34319582-d595-d1c7-d1d2-179bcfa61660",device="nvidia0",Hostname="ub20-a100-k8s"} 35690 - DCGM_FI_DEV_FB_USED{gpu="0",UUID="GPU-34319582-d595-d1c7-d1d2-179bcfa61660",device="nvidia0",Hostname="ub20-a100-k8s"} 4845 - DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL{gpu="0",UUID="GPU-34319582-d595-d1c7-d1d2-179bcfa61660",device="nvidia0",Hostname="ub20-a100-k8s"} 0 - DCGM_FI_DEV_VGPU_LICENSE_STATUS{gpu="0",UUID="GPU-34319582-d595-d1c7-d1d2-179bcfa61660",device="nvidia0",Hostname="ub20-a100-k8s"} 0 - DCGM_FI_PROF_GR_ENGINE_ACTIVE{gpu="0",UUID="GPU-34319582-d595-d1c7-d1d2-179bcfa61660",device="nvidia0",Hostname="ub20-a100-k8s"} 0.995630 - DCGM_FI_PROF_PIPE_TENSOR_ACTIVE{gpu="0",UUID="GPU-34319582-d595-d1c7-d1d2-179bcfa61660",device="nvidia0",Hostname="ub20-a100-k8s"} 0.929260 - DCGM_FI_PROF_DRAM_ACTIVE{gpu="0",UUID="GPU-34319582-d595-d1c7-d1d2-179bcfa61660",device="nvidia0",Hostname="ub20-a100-k8s"} 0.690789 - DCGM_FI_PROF_PCIE_TX_BYTES{gpu="0",UUID="GPU-34319582-d595-d1c7-d1d2-179bcfa61660",device="nvidia0",Hostname="ub20-a100-k8s"} 33011804 - DCGM_FI_PROF_PCIE_RX_BYTES{gpu="0",UUID="GPU-34319582-d595-d1c7-d1d2-179bcfa61660",device="nvidia0",Hostname="ub20-a100-k8s"} 97863601 - - DCGM_FI_DEV_XID_ERRORS{gpu="0",UUID="GPU-34319582-d595-d1c7-d1d2-179bcfa61660",device="nvidia0",GPU_I_PROFILE="1g.5gb",GPU_I_ID="13",Hostname="ub20-a100-k8s"} 0 - DCGM_FI_PROF_GR_ENGINE_ACTIVE{gpu="0",UUID="GPU-34319582-d595-d1c7-d1d2-179bcfa61660",device="nvidia0",GPU_I_PROFILE="1g.5gb",GPU_I_ID="13",Hostname="ub20-a100-k8s"} 0.995687 - DCGM_FI_PROF_PIPE_TENSOR_ACTIVE{gpu="0",UUID="GPU-34319582-d595-d1c7-d1d2-179bcfa61660",device="nvidia0",GPU_I_PROFILE="1g.5gb",GPU_I_ID="13",Hostname="ub20-a100-k8s"} 0.930433 - DCGM_FI_PROF_DRAM_ACTIVE{gpu="0",UUID="GPU-34319582-d595-d1c7-d1d2-179bcfa61660",device="nvidia0",GPU_I_PROFILE="1g.5gb",GPU_I_ID="13",Hostname="ub20-a100-k8s"} 0.800339 - - -For more information on the profiling metrics and how to interpret the metrics, refer to :external+dcgm:ref:`profiling metrics` in the `NVIDIA DCGM User Guide`. +The host engine must run DCGM 4.7 or later for DCGM Exporter 4.9.0. +With an older host engine, DCGM Exporter 4.9.0 exits at startup, because its default metrics use DCGM 4.7 fields. +Use the DCGM and DCGM Exporter versions that the Operator release ships. +If you set ``dcgm.version``, upgrade it together with ``dcgmExporter.version``. +For more information, refer to +`Connect to a DCGM host engine `__. diff --git a/gpu-telemetry/graphics/dcgm-e2e/001-dcgm-e2e-prom-screenshot.png b/gpu-telemetry/graphics/dcgm-e2e/001-dcgm-e2e-prom-screenshot.png deleted file mode 100644 index 47d88a97a..000000000 Binary files a/gpu-telemetry/graphics/dcgm-e2e/001-dcgm-e2e-prom-screenshot.png and /dev/null differ diff --git a/gpu-telemetry/graphics/dcgm-e2e/002-dcgm-e2e-grafana-screenshot.png b/gpu-telemetry/graphics/dcgm-e2e/002-dcgm-e2e-grafana-screenshot.png deleted file mode 100644 index 0dc6172ed..000000000 Binary files a/gpu-telemetry/graphics/dcgm-e2e/002-dcgm-e2e-grafana-screenshot.png and /dev/null differ diff --git a/gpu-telemetry/graphics/dcgm-e2e/003-dcgm-e2e-grafana-home-screenshot.png b/gpu-telemetry/graphics/dcgm-e2e/003-dcgm-e2e-grafana-home-screenshot.png deleted file mode 100644 index 5f008ea2d..000000000 Binary files a/gpu-telemetry/graphics/dcgm-e2e/003-dcgm-e2e-grafana-home-screenshot.png and /dev/null differ diff --git a/gpu-telemetry/graphics/dcgm-e2e/004-dcgm-e2e-grafana-manage-screenshot.png b/gpu-telemetry/graphics/dcgm-e2e/004-dcgm-e2e-grafana-manage-screenshot.png deleted file mode 100644 index e7b16c32f..000000000 Binary files a/gpu-telemetry/graphics/dcgm-e2e/004-dcgm-e2e-grafana-manage-screenshot.png and /dev/null differ diff --git a/gpu-telemetry/graphics/dcgm-e2e/005-dcgm-e2e-grafana-import-screenshot.png b/gpu-telemetry/graphics/dcgm-e2e/005-dcgm-e2e-grafana-import-screenshot.png deleted file mode 100644 index 509a4693c..000000000 Binary files a/gpu-telemetry/graphics/dcgm-e2e/005-dcgm-e2e-grafana-import-screenshot.png and /dev/null differ diff --git a/gpu-telemetry/graphics/dcgm-e2e/006-dcgm-e2e-grafana-import-screenshot.png b/gpu-telemetry/graphics/dcgm-e2e/006-dcgm-e2e-grafana-import-screenshot.png deleted file mode 100644 index 4373860b7..000000000 Binary files a/gpu-telemetry/graphics/dcgm-e2e/006-dcgm-e2e-grafana-import-screenshot.png and /dev/null differ diff --git a/gpu-telemetry/graphics/dcgm-e2e/007-dcgm-e2e-grafana-import-screenshot.png b/gpu-telemetry/graphics/dcgm-e2e/007-dcgm-e2e-grafana-import-screenshot.png deleted file mode 100644 index 2b476c501..000000000 Binary files a/gpu-telemetry/graphics/dcgm-e2e/007-dcgm-e2e-grafana-import-screenshot.png and /dev/null differ diff --git a/gpu-telemetry/graphics/dcgm-e2e/008-dcgm-e2e-grafana-dashboard-screenshot.png b/gpu-telemetry/graphics/dcgm-e2e/008-dcgm-e2e-grafana-dashboard-screenshot.png deleted file mode 100644 index 9abb690e6..000000000 Binary files a/gpu-telemetry/graphics/dcgm-e2e/008-dcgm-e2e-grafana-dashboard-screenshot.png and /dev/null differ diff --git a/gpu-telemetry/graphics/dcgm-e2e/009-dcgm-e2e-deepstream-screenshot.png b/gpu-telemetry/graphics/dcgm-e2e/009-dcgm-e2e-deepstream-screenshot.png deleted file mode 100644 index bc2622f58..000000000 Binary files a/gpu-telemetry/graphics/dcgm-e2e/009-dcgm-e2e-deepstream-screenshot.png and /dev/null differ diff --git a/gpu-telemetry/graphics/dcgm-e2e/010-dcgm-e2e-deepstream-screenshot.png b/gpu-telemetry/graphics/dcgm-e2e/010-dcgm-e2e-deepstream-screenshot.png deleted file mode 100644 index 73d49e50f..000000000 Binary files a/gpu-telemetry/graphics/dcgm-e2e/010-dcgm-e2e-deepstream-screenshot.png and /dev/null differ diff --git a/gpu-telemetry/graphics/dcgm-e2e/011-dcgm-e2e-prom-dashboard-metrics-screenshot.png b/gpu-telemetry/graphics/dcgm-e2e/011-dcgm-e2e-prom-dashboard-metrics-screenshot.png deleted file mode 100644 index b2d856032..000000000 Binary files a/gpu-telemetry/graphics/dcgm-e2e/011-dcgm-e2e-prom-dashboard-metrics-screenshot.png and /dev/null differ diff --git a/gpu-telemetry/graphics/dcgm-exporter-bare-metal.png b/gpu-telemetry/graphics/dcgm-exporter-bare-metal.png deleted file mode 100644 index a60ea1a1c..000000000 Binary files a/gpu-telemetry/graphics/dcgm-exporter-bare-metal.png and /dev/null differ diff --git a/gpu-telemetry/graphics/dcgm-exporter-containers.png b/gpu-telemetry/graphics/dcgm-exporter-containers.png deleted file mode 100644 index 655576c8a..000000000 Binary files a/gpu-telemetry/graphics/dcgm-exporter-containers.png and /dev/null differ diff --git a/gpu-telemetry/graphics/dcgm-exporter_embedded.png b/gpu-telemetry/graphics/dcgm-exporter_embedded.png deleted file mode 100644 index 5263bfea7..000000000 Binary files a/gpu-telemetry/graphics/dcgm-exporter_embedded.png and /dev/null differ diff --git a/gpu-telemetry/graphics/dcgm_and_dcgm-exporter.png b/gpu-telemetry/graphics/dcgm_and_dcgm-exporter.png deleted file mode 100644 index f0a838e77..000000000 Binary files a/gpu-telemetry/graphics/dcgm_and_dcgm-exporter.png and /dev/null differ diff --git a/gpu-telemetry/index.rst b/gpu-telemetry/index.rst index 9fcc76f10..25ba1a1af 100644 --- a/gpu-telemetry/index.rst +++ b/gpu-telemetry/index.rst @@ -23,8 +23,6 @@ about-telemetry dcgm-exporter - integrating-telemetry-kubernetes - kube-prometheus .. include:: about-telemetry.rst diff --git a/gpu-telemetry/integrating-telemetry-kubernetes.rst b/gpu-telemetry/integrating-telemetry-kubernetes.rst deleted file mode 100644 index 45e25ce4b..000000000 --- a/gpu-telemetry/integrating-telemetry-kubernetes.rst +++ /dev/null @@ -1,262 +0,0 @@ -.. license-header - SPDX-FileCopyrightText: Copyright (c) 2023 NVIDIA CORPORATION & AFFILIATES. All rights reserved. - SPDX-License-Identifier: Apache-2.0 - - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. - -.. headings (h1/h2/h3/h4/h5) are # * = - - -######################################### -Integrating GPU Telemetry into Kubernetes -######################################### - -The purpose of this document is to enumerate an end-to-end (e2e) workflow -for setting up and using `DCGM `_ within a Kubernetes environment. - -For simplicity, the base environment being used in this guide is Ubuntu 18.04 LTS and -a native installation of the NVIDIA drivers on the GPU enabled nodes (i.e. neither -the `NVIDIA GPU Operator `_ nor containerized drivers are used -in this document). - - - -************** -NVIDIA Drivers -************** -This section provides a summary of the steps for installing the driver using the ``apt`` package manager on Ubuntu LTS. - -.. note:: - - For complete instructions on setting up NVIDIA drivers, visit the quickstart guide at https://docs.nvidia.com/datacenter/tesla/tesla-installation-notes/index.html. - The guide covers a number of pre-installation requirements and steps on supported Linux distributions for a successful install of the driver. - - -Install the kernel headers and development packages for the currently running kernel: - -.. code-block:: console - - $ sudo apt-get install linux-headers-$(uname -r) - -Setup the CUDA network repository and ensure packages on the CUDA network repository have priority over the Canonical repository: - -.. code-block:: console - - $ distribution=$(. /etc/os-release;echo $ID$VERSION_ID | sed -e 's/\.//g') \ - && wget https://developer.download.nvidia.com/compute/cuda/repos/$distribution/x86_64/cuda-$distribution.pin \ - && sudo mv cuda-$distribution.pin /etc/apt/preferences.d/cuda-repository-pin-600 - -Install the CUDA repository GPG key: - -.. code-block:: console - - $ sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/$distribution/x86_64/7fa2af80.pub \ - && echo "deb http://developer.download.nvidia.com/compute/cuda/repos/$distribution/x86_64 /" | sudo tee /etc/apt/sources.list.d/cuda.list - -Update the ``apt`` repository cache and install the driver using the ``cuda-drivers`` meta-package. Use the ``--no-install-recommends`` option for a lean driver install -without any dependencies on X packages. This is particularly useful for headless installations on cloud instances: - -.. code-block:: console - - $ sudo apt-get update \ - && sudo apt-get -y install cuda-drivers - -************** -Install Docker -************** - -Use the official Docker script to install the latest release of Docker: - -.. code-block:: console - - $ curl https://get.docker.com | sh - -.. code-block:: console - - $ sudo systemctl --now enable docker - -******************************** -Install NVIDIA Container Toolkit -******************************** - -To run GPU accelerated containers in Docker, NVIDIA Container Toolkit for Docker is required. - -Setup the ``stable`` repository and the GPG key: - -.. code-block:: console - - $ distribution=$(. /etc/os-release;echo $ID$VERSION_ID) \ - && curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add - \ - && curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list - -Install the NVIDIA runtime packages (and their dependencies) after updating the package listing: - -.. code-block:: console - - $ sudo apt-get update \ - && sudo apt-get install -y nvidia-docker2 - -Since Kubernetes does not support the ``--gpus`` option with Docker yet, the ``nvidia`` runtime should be setup as the -default container runtime for Docker on the GPU node. This can be done by adding the ``default-runtime`` line into the Docker daemon -config file, which is usually located on the system at ``/etc/docker/daemon.json``: - -.. code-block:: console - - { - "default-runtime": "nvidia", - "runtimes": { - "nvidia": { - "path": "/usr/bin/nvidia-container-runtime", - "runtimeArgs": [] - } - } - } - -Restart the Docker daemon to complete the installation after setting the default runtime: - -.. code-block:: console - - $ sudo systemctl restart docker - -At this point, a working setup can be tested by running a base CUDA container: - -.. code-block:: console - - $ sudo docker run --rm --gpus all nvidia/cuda:11.0-base nvidia-smi - -You should observe an output as shown below: - -.. code-block:: console - - +-----------------------------------------------------------------------------+ - | NVIDIA-SMI 450.51.06 Driver Version: 450.51.06 CUDA Version: 11.0 | - |-------------------------------+----------------------+----------------------+ - | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | - | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | - | | | MIG M. | - |===============================+======================+======================| - | 0 Tesla T4 On | 00000000:00:1E.0 Off | 0 | - | N/A 34C P8 9W / 70W | 0MiB / 15109MiB | 0% Default | - | | | N/A | - +-------------------------------+----------------------+----------------------+ - - +-----------------------------------------------------------------------------+ - | Processes: | - | GPU GI CI PID Type Process name GPU Memory | - | ID ID Usage | - |=============================================================================| - | No running processes found | - +-----------------------------------------------------------------------------+ - - -**************************** -Install NVIDIA Device Plugin -**************************** - -To use GPUs in Kubernetes, the `NVIDIA Device Plugin `_ is required. -The NVIDIA Device Plugin is a daemonset that automatically enumerates the number of GPUs on each node of the cluster -and allows pods to be run on GPUs. - -The preferred method to deploy the device plugin is as a daemonset using ``helm``. First, install Helm: - -.. code-block:: console - - $ curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 \ - && chmod 700 get_helm.sh \ - && ./get_helm.sh - -Add the ``nvidia-device-plugin`` ``helm`` repository: - -.. code-block:: console - - $ helm repo add nvdp https://nvidia.github.io/k8s-device-plugin \ - && helm repo update - -Deploy the device plugin: - -.. code-block:: console - - $ helm install --generate-name nvdp/nvidia-device-plugin - -For more user configurable options while deploying the daemonset, refer to the `documentation `_ - -At this point, all the pods should be deployed: - -.. code-block:: console - - $ kubectl get pods -A - -.. code-block:: console - - NAMESPACE NAME READY STATUS RESTARTS AGE - kube-system calico-kube-controllers-5fbfc9dfb6-2ttkk 1/1 Running 3 9d - kube-system calico-node-5vfcb 1/1 Running 3 9d - kube-system coredns-66bff467f8-jzblc 1/1 Running 4 9d - kube-system coredns-66bff467f8-l85sz 1/1 Running 3 9d - kube-system etcd-ip-172-31-81-185 1/1 Running 4 9d - kube-system kube-apiserver-ip-172-31-81-185 1/1 Running 3 9d - kube-system kube-controller-manager-ip-172-31-81-185 1/1 Running 3 9d - kube-system kube-proxy-86vlr 1/1 Running 3 9d - kube-system kube-scheduler-ip-172-31-81-185 1/1 Running 4 9d - kube-system nvidia-device-plugin-1595448322-42vgf 1/1 Running 2 9d - -To test whether CUDA jobs can be deployed, run a sample CUDA ``vectorAdd`` application: - -The pod spec is shown for reference below, which requests 1 GPU: - -.. code-block:: console - - apiVersion: v1 - kind: Pod - metadata: - name: gpu-operator-test - spec: - restartPolicy: OnFailure - containers: - - name: cuda-vector-add - image: "nvidia/samples:vectoradd-cuda10.2" - resources: - limits: - nvidia.com/gpu: 1 - - -Save this podspec as ``gpu-pod.yaml``. Now, deploy the application: - -.. code-block:: console - - $ kubectl apply -f gpu-pod.yaml - -Check the logs to ensure the app completed successfully: - -.. code-block:: console - - $ kubectl get pods gpu-operator-test - -.. code-block:: console - - NAME READY STATUS RESTARTS AGE - gpu-operator-test 0/1 Completed 0 9d - -And check the logs of the ``gpu-operator-test`` pod: - -.. code-block:: console - - $ kubectl logs gpu-operator-test - -.. code-block:: console - - [Vector addition of 50000 elements] - Copy input data from the host memory to the CUDA device - CUDA kernel launch with 196 blocks of 256 threads - Copy output data from the CUDA device to the host memory - Test PASSED - Done diff --git a/gpu-telemetry/kube-prometheus.rst b/gpu-telemetry/kube-prometheus.rst deleted file mode 100644 index 46cf29a19..000000000 --- a/gpu-telemetry/kube-prometheus.rst +++ /dev/null @@ -1,476 +0,0 @@ -.. Date: November 04 2020 -.. Author: pramarao - -##################### -Setting up Prometheus -##################### - - -*************************** -About Setting up Prometheus -*************************** - -Implementing a Prometheus stack can be complicated but can be managed by taking advantage of the ``Helm`` package manager and -the `Prometheus Operator `_ and `kube-prometheus `_ projects. -The Operator uses standard configurations and dashboards for Prometheus and Grafana and the Helm `prometheus-operator `_ -chart allows you to get a full cluster monitoring solution up and running by installing Prometheus Operator and the rest of the components listed above. - -First, add the ``helm`` repo: - -.. code-block:: console - - $ helm repo add prometheus-community \ - https://prometheus-community.github.io/helm-charts - - -Now, search for the available ``prometheus`` charts: - -.. code-block:: console - - $ helm search repo kube-prometheus - -After you locate the version of the chart to use, inspect the chart so we can modify the settings: - -.. code-block:: console - - $ helm inspect values prometheus-community/kube-prometheus-stack > /tmp/kube-prometheus-stack.values - - -Next, edit the values file to change the port at which the Prometheus server service is available. In the ``prometheus`` instance -section of the chart, change the service type from ``ClusterIP`` to ``NodePort``. This will allow the Prometheus server to be accessible at your -machine ip address at port 30090 as ``http://:30090/`` - -.. code-block:: console - - From: - ## Port to expose on each node - ## Only used if service.type is 'NodePort' - ## - nodePort: 30090 - - ## Loadbalancer IP - ## Only use if service.type is "loadbalancer" - loadBalancerIP: "" - loadBalancerSourceRanges: [] - ## Service type - ## - type: ClusterIP - - To: - ## Port to expose on each node - ## Only used if service.type is 'NodePort' - ## - nodePort: 30090 - - ## Loadbalancer IP - ## Only use if service.type is "loadbalancer" - loadBalancerIP: "" - loadBalancerSourceRanges: [] - ## Service type - ## - type: NodePort - -Also, modify the ``prometheusSpec.serviceMonitorSelectorNilUsesHelmValues`` settings to ``false`` below: - -.. code-block:: console - - ## If true, a nil or {} value for prometheus.prometheusSpec.serviceMonitorSelector will cause the - ## prometheus resource to be created with selectors based on values in the helm deployment, - ## which will also match the servicemonitors created - ## - serviceMonitorSelectorNilUsesHelmValues: false - -Add the following ``configMap`` to the section on ``additionalScrapeConfigs`` in the Helm chart. - -.. code-block:: console - - ## AdditionalScrapeConfigs allows specifying additional Prometheus scrape configurations. Scrape configurations - ## are appended to the configurations generated by the Prometheus Operator. Job configurations must have the form - ## as specified in the official Prometheus documentation: - ## https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config. As scrape configs are - ## appended, the user is responsible to make sure it is valid. Note that using this feature may expose the possibility - ## to break upgrades of Prometheus. It is advised to review Prometheus release notes to ensure that no incompatible - ## scrape configs are going to break Prometheus after the upgrade. - ## - ## The scrape configuration example below will find master nodes, provided they have the name .*mst.*, relabel the - ## port to 2379 and allow etcd scraping provided it is running on all Kubernetes master nodes - ## - additionalScrapeConfigs: - - job_name: gpu-metrics - scrape_interval: 1s - metrics_path: /metrics - scheme: http - kubernetes_sd_configs: - - role: endpoints - namespaces: - names: - - gpu-operator - relabel_configs: - - source_labels: [__meta_kubernetes_endpoints_name] - action: drop - regex: .*-node-feature-discovery-master - - source_labels: [__meta_kubernetes_pod_node_name] - action: replace - target_label: kubernetes_node - - -Finally, deploy the Prometheus and Grafana pods using the ``kube-prometheus-stack`` via Helm: - -.. code-block:: console - - $ helm install prometheus-community/kube-prometheus-stack \ - --create-namespace --namespace prometheus \ - --generate-name \ - --values /tmp/kube-prometheus-stack.values - -.. note:: - - You can also override values in the Prometheus chart directly on the Helm command line: - - .. code-block:: console - - $ helm install prometheus-community/kube-prometheus-stack \ - --create-namespace --namespace prometheus \ - --generate-name \ - --set prometheus.service.type=NodePort \ - --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false - -You should see a console output as below: - -.. code-block:: console - - NAME: kube-prometheus-stack-1637791640 - LAST DEPLOYED: Wed Nov 24 22:07:22 2021 - NAMESPACE: prometheus - STATUS: deployed - REVISION: 1 - NOTES: - kube-prometheus-stack has been installed. Check its status by running: - kubectl --namespace prometheus get pods -l "release=kube-prometheus-stack-1637791640" - - Visit https://github.com/prometheus-operator/kube-prometheus for instructions on how to create & configure Alertmanager and Prometheus instances using the Operator. - -Now you can see the Prometheus and Grafana pods: - -.. code-block:: console - - $ kubectl get pods -A - -.. code-block:: console - - NAMESPACE NAME READY STATUS RESTARTS AGE - kube-system calico-kube-controllers-8f59968d4-g28x8 1/1 Running 1 23m - kube-system calico-node-zfnfk 1/1 Running 1 23m - kube-system coredns-f9fd979d6-p7djj 1/1 Running 1 23m - kube-system coredns-f9fd979d6-qhhgq 1/1 Running 1 23m - kube-system etcd-ip-172-31-92-253 1/1 Running 1 23m - kube-system kube-apiserver-ip-172-31-92-253 1/1 Running 2 23m - kube-system kube-controller-manager-ip-172-31-92-253 1/1 Running 1 23m - kube-system kube-proxy-mh528 1/1 Running 1 23m - kube-system kube-scheduler-ip-172-31-92-253 1/1 Running 1 23m - kube-system nvidia-device-plugin-1603211071-7hlk6 1/1 Running 0 15m - prometheus alertmanager-kube-prometheus-stack-1603-alertmanager-0 2/2 Running 0 13m - prometheus kube-prometheus-stack-1603-operator-6b95bcdc79-wmbkn 2/2 Running 0 13m - prometheus kube-prometheus-stack-1603211794-grafana-67ff56c449-tlmxc 2/2 Running 0 13m - prometheus kube-prometheus-stack-1603211794-kube-state-metrics-877df67c49f 1/1 Running 0 13m - prometheus kube-prometheus-stack-1603211794-prometheus-node-exporter-b5fl9 1/1 Running 0 13m - prometheus prometheus-kube-prometheus-stack-1603-prometheus-0 3/3 Running 1 13m - - - -*************** -Setting up DCGM -*************** - -Now, we will deploy DCGM Exporter to gather GPU telemetry. First, lets setup the Helm repo: - -.. code-block:: console - - $ helm repo add gpu-helm-charts \ - https://nvidia.github.io/dcgm-exporter/helm-charts - -And then update the Helm repo: - -.. code-block:: console - - $ helm repo update - -************************************** -DCGM Exporter Helm Chart Customization -************************************** - -The DCGM-Exporter helm package includes several customization options for various use cases. - -arguments - Customize the command-line parameters passed to dcgm-exporter on startup. - - *Example: Set the metric collection interval to 1000 milliseconds.* - -:: - - arguments["-c", "1000"] - - -extraConfigMapVolumes - Attach ConfigMap volume containing the metrics to the be watched. - - *Example: Attach the 'exporter-metrics-config-map' volume to the pod.* - - .. code-block:: yaml - - extraConfigMapVolumes: - - name: exporter-metrics-volume - configMap: - name: exporter-metrics-config-map - -extraEnv - Customize environment variables, including and especially the DCGM-Exporter variables. - - *Example: Collect the metrics specified in the ConfigMap `exporter-metrics-volume`.* - - .. code-block:: yaml - - extraEnv: - - name: DCGM_EXPORTER_CONFIGMAP_DATA - value: "default:exporter-metrics-volume" - -Install the DCGM Exporter chart: - -.. code-block:: console - - $ helm install \ - --generate-name \ - gpu-helm-charts/dcgm-exporter - -Now, you can observe the DCGM Exporter pod: - -.. code-block:: console - - $ kubectl get pods -A - -.. code-block:: console - - NAMESPACE NAME READY STATUS RESTARTS AGE - default dcgm-exporter-2-1603213075-w27mx 1/1 Running 0 2m18s - kube-system calico-kube-controllers-8f59968d4-g28x8 1/1 Running 1 43m - kube-system calico-node-zfnfk 1/1 Running 1 43m - kube-system coredns-f9fd979d6-p7djj 1/1 Running 1 43m - kube-system coredns-f9fd979d6-qhhgq 1/1 Running 1 43m - kube-system etcd-ip-172-31-92-253 1/1 Running 1 43m - kube-system kube-apiserver-ip-172-31-92-253 1/1 Running 2 43m - kube-system kube-controller-manager-ip-172-31-92-253 1/1 Running 1 43m - kube-system kube-proxy-mh528 1/1 Running 1 43m - kube-system kube-scheduler-ip-172-31-92-253 1/1 Running 1 43m - kube-system nvidia-device-plugin-1603211071-7hlk6 1/1 Running 0 35m - prometheus alertmanager-kube-prometheus-stack-1603-alertmanager-0 2/2 Running 0 33m - prometheus kube-prometheus-stack-1603-operator-6b95bcdc79-wmbkn 2/2 Running 0 33m - prometheus kube-prometheus-stack-1603211794-grafana-67ff56c449-tlmxc 2/2 Running 0 33m - prometheus kube-prometheus-stack-1603211794-kube-state-metrics-877df67c49f 1/1 Running 0 33m - prometheus kube-prometheus-stack-1603211794-prometheus-node-exporter-b5fl9 1/1 Running 0 33m - prometheus prometheus-kube-prometheus-stack-1603-prometheus-0 3/3 Running 1 33m - - -You can view the services setup as part of the operator and DCGM Exporter: - -.. code-block:: console - - $ kubectl get svc -A - -.. code-block:: console - - NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE - default dcgm-exporter-2-1603213075 ClusterIP 10.104.40.255 9400/TCP 7m44s - default kubernetes ClusterIP 10.96.0.1 443/TCP 49m - kube-system kube-dns ClusterIP 10.96.0.10 53/UDP,53/TCP,9153/TCP 48m - kube-system kube-prometheus-stack-1603-coredns ClusterIP None 9153/TCP 28m - kube-system kube-prometheus-stack-1603-kube-controller-manager ClusterIP None 10252/TCP 28m - kube-system kube-prometheus-stack-1603-kube-etcd ClusterIP None 2379/TCP 28m - kube-system kube-prometheus-stack-1603-kube-proxy ClusterIP None 10249/TCP 28m - kube-system kube-prometheus-stack-1603-kube-scheduler ClusterIP None 10251/TCP 28m - kube-system kube-prometheus-stack-1603-kubelet ClusterIP None 10250/TCP,10255/TCP,4194/TCP 28m - prometheus alertmanager-operated ClusterIP None 9093/TCP,9094/TCP,9094/UDP 28m - prometheus kube-prometheus-stack-1603-alertmanager ClusterIP 10.100.20.237 9093/TCP 28m - prometheus kube-prometheus-stack-1603-operator ClusterIP 10.111.1.27 8080/TCP,443/TCP 28m - prometheus kube-prometheus-stack-1603-prometheus NodePort 10.99.188.46 9090:30090/TCP 28m - prometheus kube-prometheus-stack-1603211794-grafana ClusterIP 10.109.219.60 80/TCP 28m - prometheus kube-prometheus-stack-1603211794-kube-state-metrics ClusterIP 10.103.250.41 8080/TCP 28m - prometheus kube-prometheus-stack-1603211794-prometheus-node-exporter ClusterIP 10.108.225.36 9100/TCP 28m - prometheus prometheus-operated ClusterIP None 9090/TCP 28m - -You can observe that the Prometheus server is available at port 30090 on the node's IP address. Open your browser to ``http://:30090``. -It may take a few minutes for DCGM to start publishing the metrics to Prometheus. The metrics availability can be verified by typing ``DCGM_FI_DEV_GPU_UTIL`` -in the event bar to determine if the GPU metrics are visible: - -.. image:: graphics/dcgm-e2e/001-dcgm-e2e-prom-screenshot.png - :width: 800 - -************* -Using Grafana -************* - -You can also launch the Grafana tools for visualizing the GPU metrics. - -There are two mechanisms for dealing with the ports on which Grafana is available - the service can be patched or port-forwarding can be used to reach the home page. -Either option can be chosen based on preference. - -Patching the Grafana Service -============================ - -By default, Grafana uses a ``ClusterIP`` to expose the ports on which the service is accessible. This can be changed to a ``NodePort`` instead, so the page is accessible -from the browser, similar to the Prometheus dashboard. - -You can use `kubectl patch `_ to update the service API -object to expose a ``NodePort`` instead. - -First, modify the spec to change the service type: - -.. code-block:: console - - $ cat << EOF | tee grafana-patch.yaml - spec: - type: NodePort - nodePort: 32322 - EOF - -And now use ``kubectl patch``: - -.. code-block:: console - - $ kubectl patch svc kube-prometheus-stack-1603211794-grafana \ - -n prometheus \ - --patch "$(cat grafana-patch.yaml)" - -.. code-block:: console - - service/kube-prometheus-stack-1603211794-grafana patched - -You can verify that the service is now exposed at an externally accessible port: - -.. code-block:: console - - $ kubectl get svc -A - -.. code-block:: console - - NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE - - prometheus kube-prometheus-stack-1603211794-grafana NodePort 10.109.219.60 80:30759/TCP 32m - -Open your browser to ``http://:30759`` and view the Grafana login page. Access Grafana home using the ``admin`` username. -The password credentials for the login are available in the ``prometheus.values`` file we edited in the earlier section of the doc: - -.. code-block:: console - - ## Deploy default dashboards. - ## - defaultDashboardsEnabled: true - - adminPassword: prom-operator - -.. image:: graphics/dcgm-e2e/002-dcgm-e2e-grafana-screenshot.png - :width: 800 - -Port Forwarding -=============== - -Another method to access the Grafana page would be to use port forwarding. - -First, it can be observed that the Grafana service is available at port 80. We will need to port-forward the service from an arbitrary port - in this example, -we will forward from port 32322 on our local machine to port 80 on the service (which in turn will forward to port 3000 that the Grafana pod is listening at, as -shown below): - -.. code-block:: console - - $ kubectl port-forward svc/kube-prometheus-stack-1603211794-grafana -n prometheus 32322:80 - -.. code-block:: console - - Forwarding from 127.0.0.1:32322 -> 3000 - Forwarding from [::1]:32322 -> 3000 - Handling connection for 32322 - -If your cluster is setup on a cloud instance e.g. AWS EC2, you may have to setup an SSH tunnel between your local workstation and the instance using -port forwarding to view the Grafana tool in your local workstation's browser. For example, on Windows you can use PuTTY to open an SSH tunnel and specify the -source port as 32322 and destination as ``localhost:32322`` under the ``Tunnels`` sub-menu in the SSH menu. - -Open your browser and point to ``http://localhost:32322/`` to view the Grafana login page using the same credentials in the previous section. - - -************************* -DCGM Dashboard in Grafana -************************* - -To add a dashboard for DCGM, you can use a standard dashboard that NVIDIA has made available, which can also be customized. - -.. image:: graphics/dcgm-e2e/003-dcgm-e2e-grafana-home-screenshot.png - :width: 800 - -To access the dashboard, navigate from the Grafana home page to Dashboards -> Manage -> Import: - -.. image:: graphics/dcgm-e2e/004-dcgm-e2e-grafana-manage-screenshot.png - :width: 800 - -.. image:: graphics/dcgm-e2e/005-dcgm-e2e-grafana-import-screenshot.png - :width: 800 - -Import the NVIDIA dashboard from ``https://grafana.com/grafana/dashboards/12239`` -and choose *Prometheus* as the data source in the drop down: - -.. image:: graphics/dcgm-e2e/006-dcgm-e2e-grafana-import-screenshot.png - :width: 800 - -.. image:: graphics/dcgm-e2e/007-dcgm-e2e-grafana-import-screenshot.png - :width: 800 - -The GPU dashboard will now be available on Grafana for visualizing metrics: - -.. image:: graphics/dcgm-e2e/008-dcgm-e2e-grafana-dashboard-screenshot.png - :width: 800 - - - -**************************************** -Viewing Metrics for Running Applications -**************************************** - -In this section, let's run a more complicated application and view the GPU metrics on the NVIDIA dashboard. - -We can use the standard *DeepStream Intelligent Video Analytics* `Demo `_ available on the NGC registry. -For our example, let's use the Helm chart to use the WebUI: - -.. code-block:: console - - $ helm fetch https://helm.ngc.nvidia.com/nvidia/charts/video-analytics-demo-0.1.4.tgz && \ - helm install video-analytics-demo-0.1.4.tgz --generate-name - -.. code-block:: console - - NAME: video-analytics-demo-0-1596587131 - LAST DEPLOYED: Wed Aug 5 00:25:31 2020 - NAMESPACE: default - STATUS: deployed - REVISION: 1 - NOTES: - 1. Get the RTSP URL by running these commands: - export NODE_PORT=$(kubectl get --namespace default -o jsonpath="{.spec.ports[0].nodePort}" services video-analytics-demo-0-1596587131) - export NODE_IP=$(kubectl get nodes --namespace default -o jsonpath="{.items[0].status.addresses[0].address}") - echo rtsp://$NODE_IP:$NODE_PORT/ds-test - - 2.Get the WebUI URL by running these commands: - export ANT_NODE_PORT=$(kubectl get --namespace default -o jsonpath="{.spec.ports[0].nodePort}" services video-analytics-demo-0-1596587131-webui) - export NODE_IP=$(kubectl get nodes --namespace default -o jsonpath="{.items[0].status.addresses[0].address}") - echo http://$NODE_IP:$ANT_NODE_PORT/WebRTCApp/play.html?name=videoanalytics - Disclaimer: - Note: Due to the output from DeepStream being real-time via RTSP, you may experience occasional hiccups in the video stream depending on network conditions. - -The demo can be viewed in the browser by pointing to the address following the instructions above. - -The GPU metrics are also visible either in the Grafana dashboard or the Prometheus dashboard as can be seen in the following screenshots showing -GPU utilization, memory allocated as the application is running on the GPU: - -.. image:: graphics/dcgm-e2e/010-dcgm-e2e-deepstream-screenshot.png - :width: 800 - -.. image:: graphics/dcgm-e2e/011-dcgm-e2e-prom-dashboard-metrics-screenshot.png - :width: 800 - diff --git a/openshift/enable-gpu-monitoring-dashboard.rst b/openshift/enable-gpu-monitoring-dashboard.rst index 1d5be60a7..15b113548 100644 --- a/openshift/enable-gpu-monitoring-dashboard.rst +++ b/openshift/enable-gpu-monitoring-dashboard.rst @@ -31,6 +31,12 @@ Configuring the NVIDIA DCGM Exporter Dashboard $ curl -LfO https://github.com/NVIDIA/dcgm-exporter/raw/main/grafana/dcgm-exporter-dashboard.json + .. note:: + + The dashboard on the ``main`` branch queries the metric names that DCGM Exporter 4.9.0 and later use. + If your cluster runs an earlier version of DCGM Exporter, download the file from the tag that matches your version instead. + The GPU Operator recording rules that produce the ``accelerator_*`` series read both the new and the old metric names. + #. Create a config map from the downloaded file in the ``openshift-config-managed`` namespace: .. code-block:: console @@ -93,9 +99,9 @@ The following table provides a brief description of the graphs on the default da +--------------------------+------------------------------------------------------------+ | GPU SM Clocks | SM clock frequency in hertz. | +--------------------------+------------------------------------------------------------+ -| GPU Utilization | GPU utilization, percent. | +| GPU Utilization | GPU utilization. | +--------------------------+------------------------------------------------------------+ -| GPU Framebuffer Mem Used | Frame buffer memory used in MB. | +| GPU Framebuffer Mem Used | Frame buffer memory used. | +--------------------------+------------------------------------------------------------+ -| Tensor Core Utilization | Ratio of cycles the tensor (HMMA) pipe is active, percent. | +| Tensor Core Utilization | Ratio of cycles the tensor (HMMA) pipe is active. | +--------------------------+------------------------------------------------------------+ diff --git a/repo.toml b/repo.toml index 2752d317e..64fe59c08 100644 --- a/repo.toml +++ b/repo.toml @@ -228,6 +228,10 @@ project = "gpu-telemetry" name = "NVIDIA GPU Telemetry" version = "1.0.0" copyright_start = 2020 +redirects = [ + { path="integrating-telemetry-kubernetes.html", project="gpu-operator", target="getting-started.html" }, + { path="kube-prometheus.html", target="dcgm-exporter.html#dcgm-exporter-prometheus-scrape" }, +] [repo_docs.projects.gpu-telemetry.builds.linkcheck] build_by_default = false