Your first local model with llama.cpp

Level BeginnerEstimated time 30 minRead first Docker with access to your GPUA word you don't know? Glossary

Last reviewed: 2026-10-22

llama.cpp is the open-source engine that runs most local models. Its server speaks the same API as the large cloud providers, so most tools can use it without changes.

A bit of history

In March 2023, a few days after Meta's LLaMA weights circulated, Georgi Gerganov published llama.cpp: plain C and C++ that ran a large language model on a laptop's processor. Quantisation, storing weights in 4 or 5 bits instead of 16, made it fit in ordinary memory. In August 2023 the project introduced the GGUF file format, now the common currency of local models.

1. Download a model

Qwen3 8B, compressed to 4 bits (Q4_K_M), 5.0 GB: it fits on an 8 GB card.

mkdir -p ~/models
curl -L -o ~/models/Qwen3-8B-Q4_K_M.gguf \
  https://huggingface.co/unsloth/Qwen3-8B-GGUF/resolve/main/Qwen3-8B-Q4_K_M.gguf

2. Serve it

docker run --rm --gpus all \
  -p 127.0.0.1:8080:8080 \
  -v ~/models:/models \
  ghcr.io/ggml-org/llama.cpp:server-cuda \
  -m /models/Qwen3-8B-Q4_K_M.gguf \
  -ngl 99 -c 8192 -fa on \
  --host 0.0.0.0 --port 8080

-ngl 99: put every layer on the GPU.
-c 8192: the context, how much text the model can keep in mind at once. More context, more VRAM.
-fa on: flash attention, faster and lighter on memory.
-p 127.0.0.1:8080:8080: reachable from this machine only. Do not open it to your network before you have put something in front of it.

One request, start to finish, without leaving the machine: the port only listens on 127.0.0.1.

3. Talk to it

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages": [{"role": "user", "content": "Say hello in one sentence."}]}'

Or open http://127.0.0.1:8080 in a browser: the server ships a small chat page.

A trap we measured

Same model, same card, one flag changed. Reading a 7,758-token prompt: 21 times slower with ‑ctv q4_0.

To save memory you can compress the context cache with -ctk q8_0 -ctv q8_0. Do not go further on the V side: with -ctv q4_0, flash attention silently switches off and reading a long prompt became 21 times slower on our card (62.7 instead of 1,330 tokens per second), while generation speed did not change. A test that only measures generation would never see it.

Sources

Found this useful? Tell us what to measure next.