Several models, two graphics cards: a grid instead of a juggling act

Level AdvancedEstimated time 15 minRead first One gateway for all your models, and three ways a local model freezes your agentsA word you don't know? Glossary

Two 16 GB cards hold one big model each, or a few small ones, or one huge model across both. Which one serves depends on the hour: an agent coding, a long document to read, a game. Changing by hand means stopping containers, starting others, and forgetting one. We replaced that with a grid and one small program that owns it.

How these figures were obtained. Measured on our own machines (see the protocol), usually a single run, with speeds as reported by the server itself; we do not publish raw logs. Treat them as orders of magnitude on this hardware, not as guarantees on yours. Figures from outside are labelled with their source.

Each model exists once per place it can run: card 0, card 1, or both cards together. One cell is one container, created once. Swapping is stopping one cell and starting another; a game simply takes card 0 and gives it back.

1. One cell, one container

Each model exists once per place it can run: card 0, card 1, or both cards together. Each of those cells is its own container, created once with its exact command line (which card, how much context, which options). A container fixes its command line at creation: if one container served several cells, every swap would mean recreating it.

Our grid has about thirty cells. At any time, a few of them serve; the rest are stopped, ready.

2. Swap, and know what it costs

Swapping is stopping the cells that conflict and starting the one you want, then waiting until it really answers. We measured the load time on our cards: 0.48 s per GB spread over both cards, 0.75 s per GB on a single card. A 9 GB model is ready in about seven seconds; a model spread over both cards is ready faster than its size suggests. Knowing that number lets a scheduler decide whether a swap is worth it.

Over the period we counted, 425 swaps out of 463 succeeded. The 38 failures are why every swap ends with a check, never with an assumption.

3. Guests that give way

Some services are not models but still want a card: our voice service (Homelab 15) uses 3.9 GB. It is a guest: it runs when its card holds a compatible model, stops before a bigger one loads, and restarts after. A game is the strongest guest of all: it takes card 0 entirely, and the grid gives it back when the game ends (see Free your graphics cards for AI).

Traps we paid for

Never let a model fill its card. A 27B model that used 97.4 % of the VRAM loaded fine, then died at the first "hello": no room left for the computation buffers. Keep headroom, or the model is only healthy until someone talks to it.

Name each container after what it really runs (model, engine, card), and check it automatically: a container whose name lies sends every tool to the wrong place.

Anything started by hand outside the grid is invisible to it, and the grid will happily load a model onto a card that something else already fills. Either declare it, or let the scheduler report it as foreign.

Found this useful? Tell us what to measure next.