Strata: a 177-billion-parameter model on one gaming card

Flash-Next is a mixture-of-experts model of 177 billion parameters: for each token it only uses a small part of them. Strata, an open-source engine, is built around that fact: the busy part on the graphics card, the experts in system RAM, read as needed. The result is a model of server size on a gaming PC, with nothing leaving the house.

What you get. A large reasoning model on a single 16 GB card, next to a second card that keeps serving your agents. In our setup it was wired as the model for the "information" turns of the phone orchestrator, while a smaller model handles actions on the other card.

1. What it needs

• A recent NVIDIA (RTX 20 to 50) or AMD card with 12 GB or more.
• RAM: 32 GB minimum; 64 GB runs every size. On our machine the 2-bit version holds about 38 GB of experts in RAM.
• About 80 GB of disk, on an SSD if possible: the first start reads a lot.

Strata's authors publish their own speeds: on an RTX 5070 (12 GB), the 2-bit version writes 94 tokens per second and reads a prompt at 2,650. Measure on your own machine before believing any number, including theirs.

2. One more slot in the grid

We do not run Strata by hand: it is one more slot in our GPU grid, on the first card only. Choosing that slot unloads whatever was on the card and starts Strata in its container; choosing another slot puts things back.

docker run -d --name flashnext-strata-c0 --network host \
  --gpus device=GPU-xxxxxxxx \             # the card by its UUID, never by index
  -v strata-data:/data \
  -e FAMILY=qwen -e MODEL=Q2_0 -e CONTEXT=32768 -e HOST=127.0.0.1 -e PORT=8129 \
  strata:local

Then declare the slot in the model gateway and in the agents' configuration, or the grid and the agents disagree, which our drift check reports as an alert on the phone.

3. Three details that decide whether it starts

1. The card must really be free. Our first attempt failed with "cudaMalloc impossible": another model still held the memory. The grid unloads first, then starts.
2. Read its "ready" line the way it prints it. Strata announces ready: http://... on a line without the usual prefix. Our grid waited for a prefixed line and declared a timeout on a server that was up.
3. Bind it to the machine itself (HOST=127.0.0.1): the gateway is the only door, with its own access control.

Traps

When a role falls back to "the first model served", alphabetical order decides. Without its usual small model, our orchestrator sent its turns to the wrong one. Name the fallback explicitly.

Map the experts, never pin them: weights held resident in RAM cannot be freed, and the machine freezes (see the 177B on two cards).

Sources

• Strata, official repository

Found this useful? Tell us what to measure next.