[Bug]: default-kind management.nvidia.com/gpu incompatible with non-cdi strategies, making GPU pods unresolable

Describe the bug
The operator sets default-kind to management.nvidia.com/gpu, and that kind contains only the "all" device by design. Any device list that arrives as bare names therefore cannot resolve in CDI mode: it is qualified with a kind that is guaranteed not to contain it. That makes the volume-mounts device list strategy and ACCEPT_NVIDIA_VISIBLE_DEVICES_AS_VOLUME_MOUNTS incompatible with cdi.enabled.

To Reproduce

Deploy GPU Operator with cdi.enabled=true.
Set devicePlugin.env.DEVICE_LIST_STRATEGY=cdi-cri,cdi-annotations,volume-mounts and toolkit.env.ACCEPT_NVIDIA_VISIBLE_DEVICES_AS_VOLUME_MOUNTS=true in ClusterPolicy.
Run a pod requesting nvidia.com/gpu: 1 with runtimeClassName: nvidia.

  apiVersion: v1
  kind: Pod
  metadata:
    name: test-cdi-device-plugin
    namespace: cdi-test
  spec:
    tolerations:
    - key: nvidia.com/gpu
      value: "true"
      effect: NoSchedule
    restartPolicy: Never
    containers:
    - name: gpu-test
      image: nvcr.io/nvidia/cuda:13.0.0-base-ubuntu24.04
      command: ["sh", "-c", "nvidia-smi -L"]
      resources:
        limits:
          nvidia.com/gpu: 1

The container never starts with error -

  Normal   Pulling    12s   kubelet            spec.containers{gpu-test}: Pulling image "nvcr.io/nvidia/cuda:13.0.0-base-ubuntu24.04"
  Normal   Pulled     8s    kubelet            spec.containers{gpu-test}: Successfully pulled image "nvcr.io/nvidia/cuda:13.0.0-base-ubuntu24.04" in 4.301s (4.301s including waiting). Image size: 139357825 bytes.
  Normal   Created    8s    kubelet            spec.containers{gpu-test}: Container created
  Warning  Failed     8s    kubelet            spec.containers{gpu-test}: Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: could not apply required modification to OCI specification: error modifying OCI spec: failed to inject CDI devices: unresolvable CDI devices management.nvidia.com/gpu=GPU-67fa9265-2b87-d8ce-e708-13e93416062c

Expected behavior
(1) The management.nvidia.com/gpu as default shouldn't get applies on workload pods.

(2) It would help if the CDI page stated that when CDI is enabled, DEVICE_LIST_STRATEGY must contain only CDI strategies and ACCEPT_NVIDIA_VISIBLE_DEVICES_AS_VOLUME_MOUNTS must be unset in ClusterPolicy - https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/25.10/cdi.html#about-the-container-device-interfacehttps://docs.nvidia.com/datacenter/cloud-native/gpu-operator/25.10/cdi.html#about-the-container-device-interface

Environment (please provide the following information):
GPU operator: v25.10
nvidia-container-toolkit version: v1.18.1
NVIDIA Driver Version: 580.126.20
Host OS: Ubuntu 24.04.4 LTS
Kernel Version: 6.8.0-1046-nvidia
Container Runtime Version: bundled with v1.35.2+rke2r1
CPU Architecture x86_64 and arm64
GPU Model(s): B200, H100

@tariq1890

Moving the Slack conversation with @tariq1890 here for visibility -

@tariq1890

[@Aditi](https://dsx-ai-factory.slack.com/team/U0C5KK31Q8M) Thank you for the feedback.

We've received similar requests before. You are right that this issue is hard to debug as the user does not really get any feedback on the configuration they go with in their ClusterPolicy.

One approach is to implement some sort of validation in the ClusterPolicy itself to prevent users from setting a configuration combination that would never work.  While this seems straightforward, it's a slippery slope as it introduces a sort of scope creep in the gpu-operator and also makes assumptions on what versions of the devicePlugin may be deployed.

The other approach is to validate the configuration in the devicePlugin application and fail-fast when CDI is enabled and the DEVICE_LIST_STRATEGY isn't provided correctly.  I lean towards this for now. In this case, the invalid configuration would lead to a devicePlugin crash thus immediate feedback to the user.

@aditighag

Thank you @aditighag

I have raised this PR to address this issue - #3081

Reading through your bug report more, I see that PR #3081 does not fully resolve this issue.

@aditighag

Thanks @aditighag for filing this issue with detailed repro steps. I was able to reproduce this and have a better handle on what is going on here.

The exact sequence is as follows:

The kubelet makes an Allocate() call to the device-plugin
The device-plugin sets the following in its ContainerAllocateResponse:
i. Fully-qualified CDI device name(s) in the CDIDevices field
ii. Fully-qualified CDI device name(s) in the container Annotations
iii. One Mount per device id
The CDI device name(s) is passed to containerd via the CRI
containerd injects the CDI device(s)
i. Note, our CDI spec sets the NVIDIA_VISIBLE_DEVICES envvar to void to prevent the NVIDIA Container Runtime from also attempting to inject devices (in case it gets invoked).
The nvidia runtime is invoked
The nvidia runtime infers the list of devices to inject
i. It does NOT find any CDI devices specified through container annotations (this is not expected, more on this later)
ii. It gets the device list from volume mounts, since accept-nvidia-visible-devices-as-volume-mounts is true. It prefers volume mounts over the envvar.
iii. The nvidia runtime is running in cdi mode, so it constructs a fully-qualified CDI device name by prepending the default CDI kind (management.nvidia.com/gpu) to the device id found via the volume mount
iv. The nvidia runtime fails to resolve the CDI device

In my opinion, the "right" fix here is to add an additional guardrail to prevent duplicate device injection from the nvidia runtime when containerd (or cri-o) is already injecting the CDI device(s) for us. We have precedent for this as we set NVIDIA_VISIBLE_DEVICES=void in our CDI specs today for this exact purpose (for the envvar strategy). I have raised NVIDIA/nvidia-container-toolkit#2146 which provides a similar guardrail for when the volume-mounts strategy is configured.

While investigating this, I noticed that the CDI device annnotations (set by the device-plugin) were not making it to the resulting OCI runtime spec. I have raised a fix for this: #3088.

In my opinion, the "right" fix here is to add an additional guardrail to prevent duplicate device injection from the nvidia runtime when containerd (or cri-o) is already injecting the CDI device(s) for us.

Thanks for the detailed notes. The approach in NVIDIA/nvidia-container-toolkit#2146 looks right for the reported failure. I have a question about backward compatibility, so I'll follow it up on the PR.

There are two (related) cases that are not tackled:

CDI enabled with no CDI strategy. If DEVICE_LIST_STRATEGY is volume-mounts alone, containerd never injects a CDI device, so the void mount never arrives and the bare device ids still get qualified with default-kind. Can we add a guardrail for such cases?

Self-declared mounts. A pod that requests no GPU but mounts /dev/null at /var/run/nvidia-container-devices/all gets no CDI injection, hence no void mount, and all resolves against the management kind to every GPU on the node. Can GPU operator protect against such foot guns? I'm ok if we want to handle this in a separate follow-up.

CDI enabled with no CDI strategy. If DEVICE_LIST_STRATEGY is volume-mounts alone, containerd never injects a CDI device, so the void mount never arrives and the bare device ids still get qualified with default-kind. Can we add a guardrail for such cases?

Yes #3081 is the guardrail we are planning for this, to prevent users from configuring a DEVICE_LIST_STRATEGY envvar that is incompatible with the cdi.enabled setting in the gpu-operator.

Self-declared mounts. A pod that requests no GPU but mounts /dev/null at /var/run/nvidia-container-devices/all gets no CDI injection, hence no void mount, and all resolves against the management kind to every GPU on the node. Can GPU operator protect against such foot guns?

While this scenario is technically possible, I don't believe this is a significant problem in practice since any reasonable Kubernetes cluster disallows host volume mounts for unprivileged containers (this is the assumption that was made when we introduced the volume mounts strategy in the first place). I am happy to hear any ideas you may have on how to harden against this scenario, if possible.

Navigation Menu

Repository navigation

[Bug]: default-kind management.nvidia.com/gpu incompatible with non-cdi strategies, making GPU pods unresolable #3077

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Bug]: default-kind management.nvidia.com/gpu incompatible with non-cdi strategies, making GPU pods unresolable #3077

Description

Activity

aditighag commented on Oct 15, 2026

tariq1890 commented on Oct 15, 2026

tariq1890 commented on Oct 15, 2026

cdesiniotis commented on Oct 19, 2026

aditighag commented on Oct 20, 2026

cdesiniotis commented on Oct 21, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions