Path: run a 7B model
Four tutorials, in this order. Tick each one when it works; your progress stays in your browser.
- 1Install the NVIDIA driver and check your cardThe one dependency everything else stands on, and how to know it really works.
Prove this step
Run this command on your machine and paste its output. Your browser checks it: nothing is sent.
nvidia-smiUnlocked tips
- Keep nvtop open while you experiment: you watch the graphics memory fill as a model loads.
- Write down the total memory shown: it decides the size of model you will be able to run.
Open-source apps for this step
- 2Docker with access to your GPUInstall Docker and the NVIDIA Container Toolkit, test it, and avoid the trap after a driver update.
Prove this step
Run this command on your machine and paste its output. Your browser checks it: nothing is sent.
sudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smiUnlocked tips
- Every GPU service you add will reuse these two options: --runtime=nvidia and --gpus all.
- lazydocker gives an overview of your containers in the terminal, logs included.
Open-source apps for this step
- 3Your first local model with llama.cppDownload an open model, serve it on your GPU, talk to it with the same API as the big clouds.
Prove this step · +120 XP
Run this command on your machine and paste its output. Your browser checks it: nothing is sent.
curl http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{"messages": [{"role": "user", "content": "Say hello in one sentence."}]}'Unlocked tips
- Your server speaks the same protocol as paid APIs: most tools accept a base address, give them http://127.0.0.1:8080/v1.
- Open WebUI puts a chat interface on top of this server, with no external account.
Open-source apps for this step
- 4Which local model for which job: a bench that measures real workBenchmarks favour the fast model; real work favoured the reliable one. A bench where code judges every answer, then each model works for real on a small repository. What it showed: small and well-framed beats big and vague, and four settings matter more than the model.
Prove this step · +140 XP
Run this command on your machine and paste its output. Your browser checks it: nothing is sent.
curl -s http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{"messages": [{"role": "user", "content": "Count from 1 to 50."}]}'Unlocked tips
- Measure again after every change (quantisation, context, driver): one number says nothing, two numbers tell a story.
- When you want to judge quality and not only speed, lm-evaluation-harness runs standard tests against your model.
Open-source apps for this step