Your GPU container runs on the CPU: three traps after an update or a reboot

Level IntermediateEstimated time 30 minRead first Docker with access to your GPUA word you don't know? Glossary

It worked yesterday. Today, after a driver update or a simple reboot, your model containers either refuse to start or, worse, start and run on the processor without a word. Both came from the same file, and both are fixed for good with one small service.

What you get. A two-minute check that tells you whether your card is really used, the fix for the two failures we hit, and a boot service that makes them never come back. Ours has run every boot since.

1. Is the card really used?

A model that turns healthy in seven seconds and answers at a few tokens per second is running on the processor. Check three things:

docker exec model nvidia-smi -L        # the card is visible...
docker logs model 2>&1 | grep -i -E "cuda|offloaded"   # ...but is CUDA alive?

If the second line says failed to initialize CUDA: unknown error, or offloaded 0/.. layers, the card is seen but not used. Seeing the card is not proof: nvidia-smi only needs the driver files, CUDA needs the device nodes too.

2. Trap one: the description is older than the driver

With CDI, Docker reads /etc/cdi/nvidia.yaml, a list of the driver's files and devices written when it was generated. Update the driver without regenerating it, and the list names files that no longer exist:

failed to fulfil mount request: open /usr/lib/libnvidia-egl-wayland.so.1.1.21: no such file or directory

Recreating the container changes nothing: the description is the source, not the container. Regenerate it:

sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml

3. Trap two: a device number that changes at every boot

The harder one. After a reboot, CUDA failed with "unknown error" while the host itself worked. We bisected the container's options one at a time:

--privileged: works;
only --device-cgroup-rule "c 235:* rwm": works;
seccomp, capabilities, /dev/dri, /sys: no effect.

The cause: the nvidia-uvm device gets its number at boot, dynamically. It was 234 when the description was written, 235 after the reboot. The device was mounted with the new number, but the container was only allowed the old one.

grep nvidia-uvm /proc/devices     # today's number

4. The permanent fix: regenerate at every boot

One small service, run before Docker starts, fixes both traps at once:

# /etc/systemd/system/nvidia-cdi-generate.service
[Unit]
Description=Regenerate the NVIDIA CDI spec (nvidia-uvm major changes each boot)
After=systemd-modules-load.service
Before=docker.service

[Service]
Type=oneshot
ExecStartPre=/usr/bin/nvidia-modprobe -u -c=0
ExecStart=/usr/bin/nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml

[Install]
WantedBy=multi-user.target

sudo systemctl enable nvidia-cdi-generate.service

nvidia-modprobe -u loads the nvidia-uvm module first, so the description is written with the number of this boot.

Traps

A driver update without a reboot hides trap one: running containers keep their old files open. It shows up days later, at the first real start. Regenerate right after the update.

When nothing makes sense, bisect: remove one option at a time and keep the one that changes the result.

Sources

Found this useful? Tell us what to measure next.