michael-balint · Oct 19, 2026 · Oct 13, 2026 · Oct 13, 2026 · Oct 14, 2026 · Oct 14, 2026
diff --git a/AGENTS.md b/AGENTS.md
@@ -92,6 +92,16 @@ model coverage. See `skills/manage-model-cache/` and
    variables in `config/group_vars/`; do not edit roles for site-specific
    values.
 
+## Optional: KAI Scheduler / Run:ai (off by default)
+
+`runai_mode: kai` (or the experimental `platform`) plus
+`playbooks/k8s-cluster/runai.yml` adds GPU-aware queues and fractional GPUs on top
+of the GPU Operator. The two modes exclude each other. Verify with
+`python3 scripts/validation/validate_runai.py --json` and require `"ok": true`.
+Memory-limited fractions (`runai_kai_gpu_fractioning`) need GPU Operator v26.7.1+
+and an r615+ driver the role will not install for you. See
+`skills/deploy-runai-kai/` and `docs/k8s-cluster/runai.md`.
+
 ## Gotchas that look like failures but are not
 
 - **`nvidia-smi` over SSH reports "No devices were found" on Slurm nodes.**

diff --git a/config.example/group_vars/k8s_cluster.yml b/config.example/group_vars/k8s_cluster.yml
@@ -94,3 +94,28 @@ kube_enable_rsyslog_client: true
 rsyslog_server_hostname: "{{ groups['kube_control_plane'][0] }}"
 rsyslog_client_tcp_host: "{{ rsyslog_server_hostname }}"
 rsyslog_client_group: "k8s_cluster"
+
+################################################################################
+# Run:ai / KAI Scheduler (optional, off by default)                            #
+################################################################################
+# Playbook: playbooks/k8s-cluster/runai.yml   Docs: docs/k8s-cluster/runai.md
+# Check:    python3 scripts/validation/validate_runai.py --json
+#
+#   off       do nothing (default)
+#   kai       KAI Scheduler: queues, gang scheduling, fractional GPU requests
+#   platform  Run:ai control plane + cluster, same Kubernetes cluster (experimental)
+# The modes exclude each other: the platform's cluster chart ships its own KAI.
+runai_mode: "off"
+
+# Hard GPU-memory limits for fractional pods (kai mode). Needs GPU Operator
+# v26.7.1+ and an r615+ driver, which you must set up first; DeepOps's default
+# driver is not changed. Leave false unless you need enforced memory limits.
+#runai_kai_gpu_fractioning: false
+
+# Platform mode only. Keep the admin login in an Ansible vault.
+#runai_platform_domain: "runai.example.com"
+#runai_platform_admin_user: "admin@example.com"
+#runai_platform_admin_password: "{{ vault_runai_admin_password }}"
+# Lab without DNS: let the role add the hosts entry and a cluster DNS rewrite.
+#runai_platform_manage_hosts_entry: true
+#runai_platform_coredns_rewrite: true
diff --git a/docs/deepops/validation.md b/docs/deepops/validation.md
@@ -88,6 +88,25 @@ namespace is kept on failure for debugging).
 
 For an exhaustive every-GPU job test, see `scripts/k8s/verify_gpu.sh`.
 
+## KAI Scheduler / Run:ai: `scripts/validation/validate_runai.py`
+
+Run anywhere `kubectl` reaches the cluster after `playbooks/k8s-cluster/runai.yml`
+(only if you enabled `runai_mode`; see [the guide](../k8s-cluster/runai.md)):
+
+```bash
+python3 scripts/validation/validate_runai.py --json
+python3 scripts/validation/validate_runai.py --json --memory-enforcement
+python3 scripts/validation/validate_runai.py --json --mode platform --platform-domain <fqdn>
+python3 scripts/validation/validate_runai.py --json --verify-uninstalled
+```
+
+It proves that a pod scheduled by KAI ran `nvidia-smi` and completed, that two
+half-GPU pods shared one GPU, optionally that a pod past its GPU-memory share is
+refused, and (platform) that the control plane answers and the cluster is
+Connected. `--verify-uninstalled` is read-only and passes only when no
+`kai-scheduler`/`runai*` namespace or release is left. The JSON carries
+`schema_version`, `ok`, one `*_ran`/`*_ok` pair per check, and `failures`.
+
 ## Testing the tools
 
 ```bash

diff --git a/docs/k8s-cluster/README.md b/docs/k8s-cluster/README.md
@@ -20,6 +20,7 @@ Instructions for deploying a GPU cluster with Kubernetes
     - [Load Balancer and Ingress](#load-balancer-and-ingress)
     - [Kubeflow](#kubeflow)
     - [NVIDIA Network Operator](#nvidia-network-operator)
+    - [Run:ai and KAI Scheduler](#runai-and-kai-scheduler)
   - [Cluster Maintenance](#cluster-maintenance)
     - [Adding Nodes](#adding-nodes)
     - [Removing Nodes](#removing-nodes)
@@ -264,6 +265,10 @@ For more information on Kubeflow, please refer to the [official documentation](h
 
 NVIDIA Network Operator leverages Kubernetes CRDs and Operator SDK to manage networking related components in Kuberenets cluster. High performance networking in Kuberentes requires many components, such as multus-CNI, device drivers and plugins to be installed correctly, NVIDIA network operator aims to manage all those necessary components automatically under one operator frame work to simply the deployment, operation and management of NVIDIA networking for Kubernetes. To deploy NVIDIA network operator, please refer to the [NVIDIA Network Operator Deployment Guide in DeepOps](nvidia-network-operator.md), for more information on NVIDIA network operator, please refer to its [github](https://github.com/Mellanox/network-operator) page and this [solution guide](https://docs.nvidia.com/networking/display/COKAN10/Network+Operator).
 
+### Run:ai and KAI Scheduler
+
+Optional GPU-aware scheduling (queues, gang scheduling, fractional GPUs) on top of the GPU Operator, off by default. See the [Run:ai and KAI Scheduler guide](runai.md).
+
 ## Cluster Maintenance
 
 DeepOps uses [Kubespray](https://github.com/kubernetes-sigs/kubespray) to deploy Kubernetes and therefore common cluster actions (such as adding nodes, removing them, draining and upgrading the cluster) should be performed with it. Kubespray is included as a submodule in the submodules/kubespray directory.

diff --git a/docs/k8s-cluster/runai.md b/docs/k8s-cluster/runai.md
@@ -0,0 +1,226 @@
+# Run:ai and KAI Scheduler (optional)
+
+DeepOps can install open-source GPU scheduling on top of the GPU Operator. It is
+**off by default** and changes nothing until you choose a mode.
+
+| `runai_mode` | What you get | Status in 26.11 |
+|---|---|---|
+| `off` (default) | Nothing installed. | |
+| `kai` | [KAI Scheduler](https://github.com/kai-scheduler/KAI-Scheduler): fair-share queues, gang scheduling, bin-packing, and pods that ask for a fraction of a GPU. | Supported. |
+| `platform` | The Run:ai control plane and cluster: the above plus projects and quotas, a web UI, CLI and REST API. | **Experimental.** Included only if it passes a live check before the release freeze. |
+
+The modes exclude each other. The Run:ai cluster chart ships its own KAI
+Scheduler, so the role refuses to install one beside the other. Switch by
+removing the first (`-e runai_state=absent`), then installing the second.
+
+Not covered: multi-cluster setups, Knative/KServe/Training Operator add-ons,
+and any DeepOps-owned UI. The web UI in `platform` mode is the upstream one.
+
+## Requirements
+
+- A DeepOps Kubernetes cluster with the GPU Operator (or the device plugin) and
+  at least one node that advertises `nvidia.com/gpu`. The role refuses to run
+  without one.
+- Kubernetes 1.35 to 1.37 (the Kubespray default in this release is inside that range).
+- `kai`: nothing else.
+- `platform`, which you must provide before running the role:
+  - a default StorageClass (`playbooks/k8s-cluster/nfs-client-provisioner.yml`)
+  - a load balancer for a `LoadBalancer` service ([Load Balancer and Ingress](ingress.md)),
+    or set `runai_platform_ingress_service_type: NodePort` (only NodePort was
+    live-tested in 26.11; `LoadBalancer` is untested in 26.11)
+  - `kube-prometheus-stack` 88.6.5 or newer (`scripts/k8s/deploy_monitoring.sh`
+    with `HELM_PROMETHEUS_CHART_VERSION` set)
+  - a DNS name for the control plane, and a TLS certificate for it
+  - room: the control plane alone wants about 10 CPU cores, 12 GB of memory and
+    110 GB of disk; the cluster services about 10 cores, 20 GB, 50 GB. See the
+    [Run:ai system requirements](https://docs.nvidia.com/run-ai/self-hosted/v3.0/documentation/installation-and-configuration/system-requirements.md)
+    for production sizing; DeepOps only tests a single large node.
+
+## Pinned versions
+
+| Component | Version |
+|---|---|
+| KAI Scheduler chart | v0.18.5 (v0.18 is a long-term-support line) |
+| gpu-fractioning (bundled in the KAI chart) | v0.1.7 |
+| Run:ai control-plane and cluster charts | 3.0.20 |
+| HAProxy ingress controller chart (platform) | 1.54.2 |
+
+All are Apache-2.0. The Run:ai images come from `nvcr.io` without an NGC key.
+Versions are variables (`runai_*_version`); pins are refreshed with each release.
+
+## Install KAI Scheduler
+
+1. Set the mode in `config/group_vars/k8s_cluster.yml`:
+
+   ```yaml
+   runai_mode: kai
+   ```
+
+2. Preview, then apply (runs on the first control-plane node):
+
+   ```bash
+   ansible-playbook -l kube_control_plane playbooks/k8s-cluster/runai.yml --check --diff
+   ansible-playbook -l kube_control_plane playbooks/k8s-cluster/runai.yml
+   ```
+
+3. Check it:
+
+   ```bash
+   python3 scripts/validation/validate_runai.py --json
+   ```
+
+   The validator creates a temporary namespace and proves, with real pods, that
+   (a) a pod using `schedulerName: kai-scheduler` in a queue ran `nvidia-smi`
+   and completed, and (b) two pods that each asked for half a GPU landed on the
+   same GPU and both completed. It requires `"ok": true`; if it is false, each
+   entry in `failures` names the problem. The temporary namespace is deleted on
+   success and kept on failure so you can look at it. The validator only ever
+   creates a new namespace (labelled `deepops.nvidia.com/runai-validate=owned`) and
+   deletes only what it created: if `--namespace` already exists it refuses and
+   touches nothing.
+
+Rerunning the playbook is safe; a converged run reports `changed=0`.
+
+### Submit a pod
+
+Pods need three things: the scheduler name, a queue label, and (for a fraction) an
+annotation. A fresh install creates `default-queue`.
+
+```yaml
+apiVersion: v1
+kind: Pod
+metadata:
+  name: half-gpu
+  labels:
+    kai.scheduler/queue: default-queue
+  annotations:
+    gpu-fraction: "0.5"
+spec:
+  schedulerName: kai-scheduler
+  containers:
+    - name: main
+      image: nvcr.io/nvidia/cuda:13.0.2-base-ubuntu24.04
+      command: ["nvidia-smi", "-L"]
+```
+
+Use a namespace of your own, not `kai-scheduler`. A pod cannot ask for both a
+fraction and a whole GPU. Without `gpu-memory.*` settings (below), a fraction is
+a scheduling promise, not a memory limit: nothing stops a pod from using more
+GPU memory than its share.
+
+To run the same vLLM quickstart pod through KAI instead of the default scheduler,
+add `schedulerName: kai-scheduler` and the queue label to its pod spec. The default
+scheduler stays the default.
+
+### Optional: enforced GPU-memory limits
+
+`runai_kai_gpu_fractioning: true` turns on the gpu-fractioning operator, which
+caps each pod's GPU memory. It is opt-in because it needs a GPU stack the
+DeepOps defaults do not use:
+
+- GPU Operator **v26.7.1 or newer** (DeepOps pins v26.7.0 by default), and
+- an NVIDIA driver **r615 or newer** on every GPU node. The GPU Operator does not
+  choose r615 by default; set `gpu_operator_driver_version` to an r615 release
+  and `gpu_operator_preinstalled_nvidia_software: false` (or install an r615 host
+  driver yourself).
+
+The role checks both and stops with an explanation if either is missing. It never
+changes your driver or GPU Operator. Then verify the limits actually hold:
+
+```bash
+python3 scripts/validation/validate_runai.py --json --memory-enforcement
+```
+
+This also proves a pod inside its share completes and a pod that allocates past
+its share is refused. In this mode pods ask for memory with the
+`nvidia.com/container.<name>.gpu-memory.request` and `.limit` annotations; see the
+[KAI GPU sharing guide](https://github.com/kai-scheduler/KAI-Scheduler/tree/v0.18.5/docs/gpu-sharing).
+
+## Install the platform (experimental)
+
+```yaml
+runai_mode: platform
+runai_platform_domain: "runai.example.com"
+runai_platform_admin_user: "admin@example.com"
+runai_platform_admin_password: "{{ vault_runai_admin_password }}"   # from an Ansible vault
+```
+
+```bash
+ansible-playbook -l kube_control_plane playbooks/k8s-cluster/runai.yml
+```
+
+The role then:
+
+1. checks the prerequisites above and stops with a pointer to the fix if one is missing;
+2. installs the HAProxy ingress controller the Run:ai documentation uses
+   (`ingress-nginx` has been retired upstream). Set
+   `runai_platform_install_ingress: false` if your site already runs a controller
+   and set `runai_platform_ingress_class` to match;
+3. creates the TLS secrets. With no certificate given it makes a local CA and a
+   certificate for the domain (lab use); for a real site set
+   `runai_platform_tls_cert_file`, `runai_platform_tls_key_file` and, if privately
+   signed, `runai_platform_ca_file` (paths on the control-plane node);
+4. installs the control plane, registers the cluster with it through its API, and
+   installs the cluster chart with the short-lived install key the API returns.
+
+**Names.** `runai_platform_domain` must resolve to the ingress address from the
+nodes and from wherever you browse. Without DNS (a lab), set
+`runai_platform_manage_hosts_entry: true` (hosts-file line on the control-plane
+node) and `runai_platform_coredns_rewrite: true` (in-cluster name). Kubespray owns
+the CoreDNS settings and overwrites the rewrite on its next run; for a lasting entry
+set Kubespray's `dns_etchosts` instead.
+
+Check it (the admin credentials go in the environment, never on the command line):
+
+```bash
+RUNAI_USERNAME=admin@example.com RUNAI_PASSWORD='...' \
+  python3 scripts/validation/validate_runai.py --json --mode platform \
+  --platform-domain runai.example.com --ca-file /opt/deepops/runai/ca.pem \
+  --create-project deepops-validate
+```
+
+In platform mode, pods must be submitted in a Run:ai project's namespace and use
+that project's queue. The validator does this itself: `--create-project deepops-validate`
+creates that project through the API (one GPU of quota in the first node pool), runs the
+whole-GPU and half-GPU checks with `runai-scheduler` in a uniquely named namespace the
+validator creates first and binds the project to, then deletes the project and that namespace (by UID). With that flag `"ok": true` also requires the project to
+have been created and deleted. GPU jobs run only after the namespace still has the UID the validator recorded and Run:ai's
+project label. The validator never reuses an existing namespace and never deletes one it did not create, so platform mode needs `--create-project NAME`
+(a name not in use) or `--skip-gpu-jobs` (control-plane and scheduler checks only). Beyond what the validator checks, DeepOps
+does not configure projects, users or quotas; use the Run:ai documentation.
+
+**Known limit (live test, 26.11):** on a one-machine test cluster with an A100, the cluster
+reported Connected, the whole-GPU pod ran under `runai-scheduler`, and the project was created
+and deleted through the API, but two half-GPU pods stayed Pending (the scheduler said only
+"Unschedulable"; the cause is not proven, GPU fractioning is off by default in this role
+because the chart's fractioning needs a newer GPU software stack). The half-GPU check is
+therefore not proven in platform mode. Pass `--skip-fractions` with `--create-project` to
+skip it; the result then shows `"fraction_job_skipped": true` so the skip is never silent.
+
+Run:ai 3.0 releases often. Keep `runai_chart_version` pinned and refresh it on
+purpose.
+
+## Remove it
+
+```bash
+ansible-playbook -l kube_control_plane playbooks/k8s-cluster/runai.yml -e runai_state=absent
+python3 scripts/validation/validate_runai.py --json --verify-uninstalled
+```
+
+The second command passes only if no `kai-scheduler`, `kai-resource-reservation`
+or `runai*` namespace or Helm release remains. The charts leave some namespaces
+and custom resource definitions behind on purpose; the role removes the
+namespaces. Add `-e runai_uninstall_remove_crds=true` to remove the definitions
+too (this deletes any queues and projects still defined).
+
+## Troubleshooting
+
+| Symptom | Likely cause and fix |
+|---|---|
+| Role stops: "No node advertises nvidia.com/gpu" | GPU stack not ready. Install the GPU Operator and wait for the node to report GPUs. |
+| Role stops: "ships its own KAI" | Both modes are installed. Remove one first. |
+| Role stops: GPU Operator or `r615` | Fractioning prerequisites missing; see above. |
+| Pod stays `Pending` with `schedulerName: kai-scheduler` | No queue label, or the queue does not exist: `kubectl get queues.scheduling.run.ai`. |
+| A half-GPU pod is rejected | It also asks for a whole `nvidia.com/gpu`, or the fraction is not between 0 and 1. |
+| Validator: scheduler deployments missing | The KAI operator is still starting. Wait a minute and rerun. |
+| Platform validator: control plane unreachable | The domain does not resolve to the ingress address, or the CA is not trusted: pass `--ca-file` and `--resolve-ip`. |
diff --git a/playbooks/k8s-cluster/runai.yml b/playbooks/k8s-cluster/runai.yml
@@ -0,0 +1,18 @@
+---
+# Optional: KAI Scheduler or the Run:ai platform on top of the GPU Operator.
+# Off by default. Choose a mode in config/group_vars/k8s_cluster.yml:
+#   runai_mode: kai        # KAI Scheduler (queues, gang scheduling, fractional GPUs)
+#   runai_mode: platform   # Run:ai control plane + cluster (experimental)
+# Remove with: -e runai_state=absent
+# Then check with: python3 scripts/validation/validate_runai.py --json
+# See docs/k8s-cluster/runai.md
+- hosts: kube_control_plane[0]
+  become: true
+  tasks:
+    - name: Install or remove Run:ai
+      include_role:
+        name: runai
+      run_once: true
+  environment: "{{ proxy_env if proxy_env is defined else {} }}"
+  tags:
+    - runai