One gateway for all your models, and three ways a local model freezes your agents

Once you run more than one model, every tool asks the same questions: which address, which name, what happens when that model is not loaded. Put a gateway in front of them, an OpenAI-compatible proxy such as LiteLLM, and the answers live in one place. Then learn the three settings that can make a perfectly healthy model look dead to your agents.

A bit of history

When OpenAI's chat API became the de facto standard in 2023, every local server started to imitate it: llama.cpp, Ollama, vLLM. A gateway such as LiteLLM, released that year, turned the idea around: one address that speaks the OpenAI and Anthropic formats to your tools, and routes each request to whichever model, local or hosted, sits behind a name.

1. One address

Agents, apps, the phone, Home Assistant: every client talks to the gateway, never to a model server directly. Keep the gateway on your private network (Homelab 6). When a model moves from one card to another, no client changes anything.

2. Names that never lie

We started with friendly role names ("writer", "reader") and fallback chains: if the preferred model was down, another one answered under the same name. It looked robust and made everything unreadable: a quality comparison between two "writers" compared two different models. Now:

• the model's name comes from one place, its weights file, and every other name is derived from it;
• the gateway's configuration is generated from the list of models the cards actually serve, never edited by hand;
• a name whose model is not loaded fails; it never quietly substitutes another one.

"The water changes with the flowers, but the vase is still called 9B." A name that encodes a role lies the day you change the model behind it.

3. Three ways a healthy model freezes your agents

One night our node was healthy, the model loaded, the container reported healthy, and a ten-byte request timed out after a minute. Three causes, stacked, all measured:

1. One slot for everyone. With --parallel 1, the server serves one request at a time. A coding agent asked for 32,000 tokens by default; at 12 tokens per second that is 35 minutes during which nobody else gets an answer. Cap max_tokens for agents, or give the server more slots.
2. Thinking on a tiny budget. The same question with reasoning enabled and a 16-token budget returned an empty answer; with reasoning off, "Ready" in 3 tokens. Turn thinking off for agents that call tools; keep it where you read the answer yourself.
3. A small micro-batch. -ub 128 made the model read prompts at 21.9 tokens per second. An agent's harness weighs about 8,000 tokens: six minutes before the first word. The agent timed out, retried, and every retry read the prompt again. Measure prompt speed, not only generation speed.

Read the server's own counters before blaming the network: llama.cpp's /slots showed the request holding the only slot, and /metrics the deferred ones.

Sources

• LiteLLM, official repository

Found this useful? Tell us what to measure next.