Docker with access to your GPU

Level BeginnerEstimated time 30 minRead first Install the NVIDIA driver and check your cardA word you don't know? Glossary

Last reviewed: 2026-10-22

Every model server in this series runs in a container: it installs cleanly, it uninstalls cleanly, and you can run two versions side by side. Two pieces, neither of them ours: Docker, and NVIDIA's Container Toolkit.

Five layers, one provider each. The CDI file describes the driver as it was: after a driver update, regenerate it.

A bit of history

Docker, released in 2013, made a whole application ship as one image that runs the same everywhere. Graphics cards came later: a container does not see the host's devices by default. NVIDIA's container toolkit bridges the gap, and the newer Container Device Interface (CDI) describes devices in a plain file that any container runtime can read, which is also why that file must stay in step with the driver.

1. Docker

Docker's own convenience script installs the latest version. Docker documents it for test and development machines; for a production server, use the repository method of their documentation.

curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh

2. NVIDIA Container Toolkit (Ubuntu, Debian)

From NVIDIA's installation guide:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
  | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
  | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
  | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Other distributions: follow the same guide on NVIDIA's site.

3. Test

sudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi

The same table as on the host, printed from inside a container: your containers can see the card.

The trap after a driver update

If you use CDI (the /etc/cdi/nvidia.yaml file), it describes the driver libraries as they were when it was generated. After a driver update it can point to files that no longer exist, and no GPU container starts. See the GPU container on the CPU for the boot-time fix. Regenerate it: sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml. We now regenerate it at every boot.

Sources

Found this useful? Tell us what to measure next.