# GPU on Kubernetes: what works together, September 2026

> A one-page compatibility snapshot for teams running GPU workloads on Kubernetes without a platform team, checked line by line against the NVIDIA, Kubernetes, cert-manager, Cilium, Calico and Helm official pages on 2026-09-04.

**In short.** As of 2026-09-04, run Kubernetes 1.35 or 1.36, pin the NVIDIA driver to R580 LTS or R595 production (which caps CUDA at 13.2), upgrade containerd to 2.0 before GPU Operator 26.7.0, keep the Container Toolkit at 1.17.8 or later, and replace the retired ingress-nginx with a Gateway API implementation.

Published 2026-09-04 · Language: English
HTML page: https://dixence.dev/en/articles/gpu-on-kubernetes-2026-09/ · Version française: https://dixence.dev/articles/gpu-on-kubernetes-2026-09/index.md
Publisher: Dixence, a brand of Idempotent Ltd (Switzerland) · contact@dixence.dev

---

Snapshot date: 2026-09-04. Every line below was checked against the NVIDIA, Kubernetes, cert-manager, Cilium, Calico and Helm official pages on that date. Versions and dates are quoted as printed there.

## The short version

- Run Kubernetes 1.35 or 1.36. 1.37 is out since 2026-08-26 but the GPU Operator, cert-manager, Cilium, Calico, Helm 4.2 and ingress-nginx don't support it yet. 1.34 reaches end of life on 2026-10-27, so don't start anything new on it.
- Pin the driver to R580 LTS (580.178.04, end of life June 2028) or R595 production (595.91.07, end of life March 2027). Both are CUDA 13 branches.
- That pin caps CUDA at 13.2. CUDA 13.3 needs driver 610.43.02, which lives on R610, a new feature branch NVIDIA gives no support length. Either cap your images at 13.2 or ship the forward compatibility package.
- Still on R535? It went end of life in June 2026 and can't run CUDA 13. Move now.
- ingress-nginx is retired. No build tested on Kubernetes 1.36 or 1.37 will ever exist, and no security fixes are coming. Pick a Gateway API implementation.
- GPU Operator 26.7.0 needs containerd 2.0. 26.3.x is deprecated but still accepts containerd 1.7. Upgrade containerd first, then the operator.
- Container Toolkit below 1.17.8 carries two published CVEs, one Critical. Current is 1.20.0. Check the version on every node this week.

## The matrix

| Component | Pin this today | Works with Kubernetes | Watch out |
|---|---|---|---|
| Kubernetes | 1.36.4 or 1.35.8 (both 2026-08-20) | n/a | 1.37 not on any NVIDIA support page; 1.34 EOL 2026-10-27 |
| NVIDIA GPU Operator | v26.7.0 (2026-08-21) | 1.33 to 1.36 | containerd 2.0 minimum; Docker not a listed runtime; 26.3.x deprecated |
| NVIDIA driver | R580 580.178.04 or R595 595.91.07 | via the operator | R535 off support; R610 has no support length |
| CUDA toolkit ceiling | 13.2 (Update 1 on 595.58.03 or later) | via the driver | 13.3 needs 610.43.02 or a forward compatibility package |
| NVIDIA Container Toolkit | 1.20.0 (2026-08-13) | ships with GPU Operator 26.7.0 | 1.17.8 is the CVE floor; CDI schema v0.7.0 since 1.19.1 needs containerd 1.7.16 or later |
| cert-manager | 1.21.1 (2026-07-29) | 1.33 to 1.36 | 1.21 dropped the default serviceaccounts/token RBAC; metrics port renamed to http-metrics |
| Ingress | a Gateway API implementation | n/a | ingress-nginx last builds 2026-03-19, top out at Kubernetes 1.35 |
| CNI (Cilium) | 1.20.1 (2026-08-18) | 1.33 to 1.36 | consecutive minors only, from the latest patch; Gateway API 1.6.1 or later |
| CNI (Calico) | 3.32.2 (2026-08-29) | 1.34 to 1.36 | Helm does not upgrade CRDs; cniVersion 1.0.0 needs containerd 1.6 or later |
| Helm | 4.2.4 (2026-08-13) | 1.33 to 1.36 (n-3 rule) | server-side apply by default; --atomic and --force renamed; Helm 3 bug fixes end 2026-09-09 |

## CUDA needs this driver

| CUDA | Minimum Linux driver |
|---|---|
| 13.3 (GA and Update 1) | 610.43.02 |
| 13.2 Update 1 | 595.58.03 |
| 13.2 GA | 595.45.04 |
| 13.1 Update 1 | 590.48.01 |
| 13.1 GA | 590.44.01 |
| 13.0 GA | 580.65.06 |
| 12.9 GA | 575.51.03 |
| 12.8 GA | 570.26 |

The floor is hard: every CUDA 13.x toolkit needs driver 580 or later, CUDA 12.x runs on 525 to 579, and minor version compatibility won't save PTX-compiled code or driver-spanning features on an older driver (cudaErrorCallRequiresNewerDriver).

## What actually breaks on upgrade

- Kubernetes 1.37: SELinuxMount is on by default and "may break existing workloads in clusters with SELinux enabled"; scheduling.k8s.io v1alpha2 is gone, so delete those objects from the kube-apiserver before you update.
- Kubernetes 1.36: the git-repo volume plugin is disabled with no way to re-enable it, and volume_operation_total_errors became volume_operation_errors_total, so dashboards and alerts stop matching.
- GPU Operator 26.7.0: DRA/GPUCluster mode needs Kubernetes 1.34.2 or later, driver 580 or later and a CDI-capable runtime; GPUCluster and ClusterPolicy can't coexist in one cluster.
- GPU Operator driver container: every worker node must run the same OS version; mixed OS only works with pre-installed drivers.
- Device plugin 0.19.0: default flags (gds, gdrcopy, mofed) changed and broke RDMA/NCCL workloads; fixed in 0.19.3 and GPU Operator 26.3.3.
- NVIDIA driver 570.124.06 to 570.158.01: MIG slices and full GPUs on the same node broke scheduling; fixed from 570.172.08 and 580.65.06.
- NVIDIA driver on Grace (GH200, GB200): 580.65.06 and later turn CDMM on by default, which is incompatible with MIG sharing and GPUDirect Storage.
- Container Toolkit 1.19.1 and later: CDI specs come out at schema v0.7.0 and older runtimes reject them; minimums are containerd 1.7.16, cri-o 1.30.0, docker 26.1.0, podman 5.1.0.
- cert-manager 1.21: Vault Kubernetes auth and Route53 setups that use the controller ServiceAccount lose serviceaccounts/token create; add your own RBAC or move to a dedicated ServiceAccount.
- Cilium 1.20: proxylib is removed, so strip kafka, l7 and l7proto policy sections before upgrading, and CiliumNodeConfig must be on cilium.io/v2.

## Dates that matter next

- 2026-09-09: Helm 3 bug fixes end.
- 2026-09-15: next Kubernetes patch set.
- 2026-10-27: Kubernetes 1.34 end of life.
- 2026-11 (planned): cert-manager 1.22 ships, which ends support for 1.20.
- 2027-02-10: Helm 3 security fixes end.
- 2027-02-28: Kubernetes 1.35 end of life.
- 2027-03: NVIDIA R595 production branch end of life.
- 2027-06-28: Kubernetes 1.36 end of life.
- 2028-06: NVIDIA R580 LTS branch end of life.

This is the list Dixence runs before touching any of the clusters it maintains. The October edition follows when the upstream pages change.