Agentic AI networks module 1: understand and run a model
Fourteen sessions to stop treating a model as magic. You learn what it actually computes, what a token and a context window cost, why the same question gives two answers, how a 16 GB file becomes a 5 GB one, and you run two local runtimes on your own machine and measure them. Everything later in the path stands on this.
Commands are written for Linux or macOS; on Windows, use WSL or PowerShell equivalents. Every address in the examples is local (127.0.0.1) or reserved for documentation (192.0.2.0/24). Model names are examples current in November 2026: check the model page before you download, and pick a smaller one if your machine is short of memory.
Day 1 · 03/11 — What a language model is
Goal: explain in two sentences what a large language model (LLM) computes, and set up your notebook and repository.
Theory 1/2: a next-token predictor
An LLM reads a sequence of tokens (pieces of words) and outputs, for the next position, a probability for every token of its vocabulary. A program then picks one token, appends it, and asks again. A whole answer is that loop repeated, one token at a time. The model has no database of facts: it has billions of numbers (its weights, or parameters) tuned during training so that its guesses match a huge corpus of text.
The architecture almost every current model uses, the Transformer, was published by Google researchers in 2017 in "Attention Is All You Need". Its key idea, attention, lets each position weigh every earlier position when computing its guess.
What does a language model output at each step?
- A finished sentence
- A probability for each possible next token
- The address of a fact in its database
Answer
A probability distribution over its vocabulary. Picking a token from it is the job of the sampler, outside the model.
Which architecture, published in 2017, do almost all current LLMs use?
Answer
The Transformer ("Attention Is All You Need").
Theory 2/2: why it matters for agents
Two consequences follow you through the path. First, a model produces plausible text, not checked text: anything that must be right (a number, a file path, a command's result) is checked by code. Second, the model only knows what is in its weights and in the text you send it now. An agent is a program that feeds the model the right text, lets it ask for actions, and checks what it does.
A model states the version of a library with confidence. You…
- check it against the official documentation
- trust it: it was trained on the documentation
Answer
Check it. Plausible is not verified, and its training data stops at a cut-off date.
Practice (30 min)
- Create the notebook: one folder
ainet-notes/, one Markdown file per day (day-01.md…). - Create the project repository where your code will live all path long:
mkdir ainet-lab && cd ainet-lab && git init. - Install uv (Python project manager), then
uv initinainet-lab. - Watch the first part of Andrej Karpathy's "Intro to Large Language Models" (2023) and note three ideas in your own words.
- Read the path page section "Learning with an AI as a tutor" and write in
day-01.mdthe rule you will follow.
With your AI: "You are a patient tutor. I am starting a course on local LLMs and agents. Ask me five questions, one at a time, to check that I understand what a language model computes at each step. Do not give the answer before I reply; after each reply, tell me what is right, what is wrong, and one source I can check."
Recap (10 min)
Three lines: what the model outputs, what the sampler does, why an agent checks results with code. Then answer today's first exercise again without notes.
Day 2 · 04/11 — Tokens
Goal: count tokens, know why they are not words, and turn a token count into a cost.
Theory 1/2: subword tokens
Models do not read letters or words but tokens: frequent fragments learnt from the training text. Most tokenizers descend from byte-pair encoding (BPE), applied to neural translation by Sennrich and colleagues in 2015 (arXiv 1508.07909): start from characters (or bytes) and merge the most frequent pairs until the vocabulary reaches a fixed size. A common English word is often one token; a rare word, a French word with accents, code or a long number splits into several. Each model family has its own tokenizer, so the same text has a different count from one model to another.
The same paragraph is sent to two different model families. Its token count is…
- always identical
- often different: each family has its own tokenizer
Answer
Different tokenizers, different counts. Measure with the tokenizer of the model you use.
Theory 2/2: tokens are the unit of everything
Context size, speed (tokens per second) and hosted prices are all counted in tokens. Hosted APIs usually charge input and output tokens at different prices, quoted per million tokens. Locally you pay in time and electricity instead, but the count still decides what fits.
Write cost(tokensIn, tokensOut, priceIn, priceOut): prices are per million tokens; return the cost.
function cost(tokensIn, tokensOut, priceIn, priceOut) {
}
Answer
function cost(tokensIn, tokensOut, priceIn, priceOut) {
return (tokensIn * priceIn + tokensOut * priceOut) / 1000000;
}Which text usually costs more tokens per character?
- A long hexadecimal hash
- A plain English sentence
Answer
The hash: random characters match few learnt fragments, so it splits into many tokens.
Practice (30 min)
- Watch the first 20 minutes of Karpathy's "Let's build the GPT Tokenizer" (2024).
- Open the Hugging Face LLM course chapter on tokenizers and read the BPE section.
- In your notebook, predict whether each of these is one token or several:
the,anticonstitutionnellement,2026-11-04,def,4f3a9c1e. You will check them on day 8 with your own server's/tokenizeendpoint. - Compute with your
costfunction what 200 requests of 3,000 input and 500 output tokens would cost at a price of 1 per million in and 4 per million out.
With your AI: "Explain byte-pair encoding to a beginner with a worked example on the string 'low lower lowest', showing each merge step. Then give me three strings and ask me to guess which splits into the most tokens, and why. Say clearly that real counts depend on the tokenizer, and that I must check them with my model."
Recap (10 min)
Three lines: what a token is, why counts differ between models, what you pay per token. Then redo the cost exercise without notes.
Day 3 · 05/11 — The context window and the KV cache
Goal: know what fits in a context, why a long prompt is slow before the first word, and what the KV cache keeps.
Theory 1/2: the window
The context window is the maximum number of tokens the model can take into account at once: your instructions, the conversation, tool results and its own answer, all together. Beyond it, something has to be cut. A runtime also lets you choose a smaller context than the model's maximum (-c in llama.cpp), because memory grows with it. Everything an agent knows during a turn is in that window: there is no other memory unless code adds it.
What counts against the context window?
- Only your question
- Only the model's answer
- Instructions, history, tool results and the answer, together
Answer
Everything in the conversation, both directions.
Theory 2/2: two phases, one cache
Answering has two phases. Prompt processing (prefill) reads all input tokens at once, in parallel; generation produces output tokens one by one. To avoid recomputing the whole past at every new token, the runtime keeps intermediate results for each past token, the KV cache (keys and values of attention). It takes memory proportional to the context length. When the start of the next prompt is identical, a runtime can reuse the cached part and skip most of the prefill: that is why agents keep their long instructions at the very beginning and never change them mid-session. The site's article the KV cache trap shows a case where a single flag made answers 21 times slower.
An agent waits 40 seconds before the first word, then writes fast. The likely culprit is…
- slow generation
- a long prompt being processed, maybe without cache reuse
Answer
Prefill. Measure the time to first token separately from tokens per second (day 11).
To get the most cache reuse, the stable instructions of an agent go…
- at the end of each prompt
- at the very beginning, unchanged
Answer
At the start: reuse works on an identical prefix.
Practice (30 min)
- Read the KV cache trap and the context floor, which measured where an agent's tokens go.
- In your notebook, draw one agent turn as a bar: system instructions, tool descriptions, history, new message, answer. Estimate each part in tokens.
- Note the maximum context of two models you might use (from their Hugging Face model pages).
With your AI: "I am learning how LLM inference works. Explain the difference between prompt processing and token generation, and what the KV cache stores, in under 200 words. Then quiz me with three true-or-false statements about context windows, and correct me with reasons."
Recap (10 min)
Three lines: what fills the window, the two phases, why a stable prefix matters. Redo the second exercise without notes.
Day 4 · 06/11 — Sampling: temperature, top-p, min-p
Goal: predict what each sampling setting does, and choose settings for an agent.
Theory 1/2: from scores to a choice
The model outputs a score (logit) per token; a softmax turns scores into probabilities. The temperature divides scores before the softmax: below 1 the distribution sharpens (the favourite wins more often), above 1 it flattens. Temperature 0 is usually treated as "always take the most probable token" (greedy). Then filters cut the tail: top-k keeps the k best, top-p (nucleus sampling, Holtzman et al., 2019) keeps the smallest set whose probabilities add up to p, min-p drops tokens far less likely than the best one. The seed fixes the random draw so a run can be replayed. llama-server's defaults are temperature 0.8, top-p 0.95, min-p 0.05.
Write softmax(logits, t): divide each logit by the temperature t, exponentiate, normalise so the result sums to 1.
function softmax(logits, t) {
}
Answer
function softmax(logits, t) {
const scaled = logits.map(x => x / t);
const max = Math.max(...scaled);
const exps = scaled.map(x => Math.exp(x - max));
const sum = exps.reduce((a, b) => a + b, 0);
return exps.map(x => x / sum);
}Theory 2/2: settings for agents
An agent that must emit valid JSON or a precise command wants little randomness: low temperature, fixed seed while testing. Creative writing wants more. Model publishers often recommend settings on the model page: start from those. Low temperature does not make the model right; it makes it consistent, including consistently wrong.
Lowering the temperature…
- makes the most probable token win more often
- makes the model more accurate
- shortens the answer
Answer
It sharpens the distribution. Accuracy is a different matter.
Which setting lets you replay exactly the same random draw?
Answer
The seed (--seed in llama.cpp).
Practice (30 min)
- Run your
softmaxby hand on[2, 1, 0]at temperatures 0.5, 1 and 2; write the three distributions. - From the 1.0 distribution, apply top-p 0.9 on paper: which tokens remain?
- Look for recommended sampling settings on the model card of the model you chose (publishers often give them) and copy them to your notebook with the link, or note that there are none.
With your AI: "Give me five scenarios (JSON extraction, brainstorming, code generation, classification, story writing) and ask me which temperature range I would pick for each and why. Then tell me where you disagree, and remind me that the model card's recommended settings take precedence over your opinion."
Recap (10 min)
Three lines: what temperature, top-p and seed do. Redo the softmax exercise without notes.
Day 5 · 07/11 — Model files: weights, GGUF, licences
Goal: read a model page, estimate its memory need from its parameter count, and check its licence.
Theory 1/2: parameters and bytes
A "7B" model has about seven billion weights. Stored at 16 bits (2 bytes) each, that is about 14 GB before anything else. Models are published mostly on the Hugging Face Hub. For local runtimes, the common format is GGUF, introduced by the llama.cpp project in 2023: one file holding the weights, the tokenizer and metadata such as the chat template. Some models are mixture-of-experts (MoE), written like "35B-A3B": 35 billion weights in total, about 3 billion active for each token. All of them must be in memory, but each token computes with few, so it runs faster than a dense 35B.
Write sizeGB(paramsBillion, bitsPerWeight): the weights' size in gigabytes (109 bytes).
function sizeGB(paramsBillion, bitsPerWeight) {
}
Answer
function sizeGB(paramsBillion, bitsPerWeight) {
return paramsBillion * bitsPerWeight / 8;
}Theory 2/2: licences and provenance
"Open weights" does not mean open source, nor free for every use. Each model page states a licence: Apache 2.0 and MIT are permissive; others add conditions (use thresholds, attribution, acceptable use policies). Read it before you build on a model. Prefer files from the publisher or a well-known converter, and note the exact file you downloaded.
A model is "open weights". Can you use it commercially?
- Yes, always
- It depends on its licence: read it
Answer
The licence decides. Open weights only says you can download them.
A "35B-A3B" model…
- needs memory for 3 billion weights only
- needs memory for 35 billion weights but computes about 3 billion per token
Answer
All experts sit in memory; only a few are active for each token.
Practice (30 min)
- Open the page ggml-org/Qwen3.5-0.8B-GGUF and list its files and their sizes.
- Find the licence of the Qwen3.5 base model and note it with the link.
- With
sizeGB, estimate the 16-bit size of a 0.8B, a 4B and a 9B model. Compare with the real file sizes.
With your AI: "Here is the text of a model licence: [paste]. List, as bullet points, what it allows, what it forbids, and what it requires (attribution, thresholds). Quote the exact sentence for each point so I can check it. If something is unclear, say so instead of guessing."
Recap (10 min)
Three lines: parameters to bytes, what GGUF holds, why the licence matters. Redo sizeGB without notes.
Day 6 · 08/11 — Quantisation
Goal: choose a quantisation level knowing what you gain in memory and what you risk in quality.
Theory 1/2: fewer bits per weight
Quantisation stores weights with fewer bits: 8, 6, 5, 4, sometimes fewer, with a scale per small block so that the values stay close. llama.cpp names its formats like Q8_0, Q5_K_M, Q4_K_M (the "K" formats came in 2023). Its own documentation measures them on Llama 3.1 8B: 16.0 bits per weight and 14.96 GiB at F16, 8.50 bits and 7.95 GiB at Q8_0, 4.89 bits and 4.58 GiB at Q4_K_M. The figure is not exactly 4 bits because the block scales take space too.
According to llama.cpp's table, Q4_K_M on an 8B model weighs about…
- 15 GiB
- 8 GiB
- 4.6 GiB
Answer
4.58 GiB, against 14.96 at F16.
Theory 2/2: what you lose
Fewer bits mean a little more error on each weight. Around 4 to 5 bits, the loss is usually small for chat; below 4 bits it grows faster, and precise tasks (code, tool calls, arithmetic) suffer first. A larger model at Q4 often beats a smaller one at Q8 for the same memory, but only your own test (day 12) says so for your job.
Which task usually shows quantisation damage first?
- Casual chat
- Exact tool calls and code
Answer
Precise outputs: one wrong token breaks a JSON call or a command.
Write fits(modelGB, cacheGB, memoryGB): true if the model plus its cache fit in the memory.
function fits(modelGB, cacheGB, memoryGB) {
}
Answer
function fits(modelGB, cacheGB, memoryGB) {
return modelGB + cacheGB <= memoryGB;
}Practice (30 min)
- Read the quantisation tables in llama.cpp's quantize README.
- On unsloth/Qwen3.5-4B-GGUF, note the sizes of Q4_K_M and Q8_0.
- Write in your notebook which file you will download on day 8 for your memory, and why.
With your AI: "My machine has [X] GB of RAM and [Y] GB of VRAM (or none). I want a model for tool calling. Ask me three questions about my use, then propose two model sizes and quantisation levels, each with the estimated memory, and tell me which test I should run to choose between them. Do not claim one is better without a test."
Recap (10 min)
Three lines: what quantisation trades, the sizes from the table, how you will choose. Redo the first exercise without notes.
Day 7 · 09/11 — Hardware: VRAM, RAM, CPU
Goal: know what limits speed on your machine and how a runtime splits a model between GPU and CPU.
Theory 1/2: memory bandwidth rules generation
To generate each token, the runtime reads all active weights once. Generation speed is therefore limited mostly by memory bandwidth: a graphics card's memory (VRAM) is several times faster than system RAM, which is why the same model writes much faster on a GPU. Prompt processing is limited more by raw compute. A machine without GPU still runs small models on its processor: slower, but real.
Token generation speed depends mostly on…
- memory bandwidth
- disk speed
- screen resolution
Answer
Bandwidth: every active weight is read for every token.
Theory 2/2: offloading layers
A model is a stack of layers. When it does not fit in VRAM, llama.cpp puts some layers on the GPU and the rest on the CPU: the option is -ngl (--n-gpu-layers). Each layer left on the CPU slows the whole thing down. Remember the KV cache also needs memory. On NVIDIA cards, nvidia-smi shows the memory in use; the site's articles on Docker with GPU access and freeing your cards for AI go further.
Which llama.cpp option sets how many layers go to the GPU?
Answer
-ngl (--n-gpu-layers).
Practice (30 min)
- List your machine in the notebook: CPU model and cores (
lscpuor system info), RAM (free -h), GPU and VRAM (nvidia-smi, or your vendor's tool), free disk. - Decide your plan: GPU, CPU only, or mixed. Pick a model size that fits with a margin of 1 to 2 GB for the cache.
- If you have an NVIDIA card and want containers, follow Docker with access to your GPU; otherwise you will use the prebuilt binaries tomorrow.
With your AI: "Here is the output of lscpu, free -h and nvidia-smi on my machine: [paste]. Summarise what I have in a table (CPU cores, RAM, GPU, VRAM). Tell me which model sizes I can run fully on GPU, which need offloading, and which only on CPU, showing your arithmetic so I can check it."
Let an agent do it: now that you have read these outputs yourself, a coding agent with shell access (Claude Code, Kimi Code, OpenFox) can gather them: "Run lscpu, free -h, nvidia-smi and df -h, read-only, and write a Markdown table of this machine's resources into hardware.md." Check every figure against the raw outputs: an agent can misread a column.
Recap (10 min)
Three lines: what limits generation, what -ngl does, your machine's plan. Redo the first exercise without notes.
Day 8 · 10/11 — llama.cpp: your first server
Goal: serve a local model with llama.cpp, talk to it in the browser and with curl.
Theory 1/2: the project
llama.cpp was started by Georgi Gerganov in March 2023 to run Meta's LLaMA on a laptop in plain C/C++; it is now the engine under many local tools, under the MIT licence. Its llama-server exposes an HTTP API compatible with OpenAI's chat completions plus a web interface. The current documentation also offers a unified llama command (llama serve, llama cli). Install from the prebuilt releases, the official installer, Docker, or build from source (build guide).
llama-server's API is compatible with…
- nothing else, it is custom
- OpenAI's chat completions format
Answer
OpenAI-compatible, so most clients and libraries work with it.
Theory 2/2: the command line
The essential options: -m a local GGUF file, or -hf a Hugging Face repository with an optional :quant; -c the context size; -ngl layers on GPU; --host and --port. The server listens on 127.0.0.1 by default, reachable only from your machine: keep it that way until module 4. The current default port is 9931; we pass --port explicitly so the commands do not depend on the version.
Why keep --host 127.0.0.1 for now?
- Only your machine can reach the model
- It is faster
Answer
Nothing on the network can use or abuse it.
Practice (30 min)
- Install llama.cpp (release binary for your system, or the method in your first local model).
- Start a small model:
Addllama-server -hf ggml-org/Qwen3.5-0.8B-GGUF -c 8192 --host 127.0.0.1 --port 8080-ngl 99if you have a GPU. Use the 4B file chosen on day 6 if it fits. - Open
http://127.0.0.1:8080and chat. - Check health, then tokenize your day-2 strings:
Compare with your predictions.curl -s http://127.0.0.1:8080/health curl -s http://127.0.0.1:8080/tokenize -H 'Content-Type: application/json' -d '{"content": "anticonstitutionnellement"}'
With your AI: "Here is the startup log of llama-server: [paste]. Explain each line that mentions the model, the context size, the GPU layers and the chat template. Point out any warning and what I should check. Do not invent meanings for lines you do not recognise: say you are unsure."
Recap (10 min)
Three lines: the command you used and what each option did. Restart the server from memory.
Day 9 · 11/11 — Ollama: a second runtime
Goal: run the same family of model with Ollama, and compare it with llama.cpp.
Theory 1/2: what Ollama adds
Ollama wraps a llama.cpp-derived engine in a service with a model library: ollama pull, ollama run, models loaded and unloaded on demand. Its API listens on http://localhost:11434, with its own routes under /api and an OpenAI-compatible one under /v1 (documentation). Convenience has a price: fewer knobs, and defaults (such as context length) you must check.
On which port does Ollama's API listen by default?
Answer
11434.
Theory 2/2: one API, several runtimes
Because both speak the OpenAI format at /v1, your code can switch runtime by changing a base URL and a model name. That is the first brick of routing (module 3). Keep the base URL and the model in configuration, never in the code.
Your script must switch from llama.cpp to Ollama. What should change?
- The whole code
- The base URL and model name, read from configuration
Answer
Same API format: only configuration changes.
Practice (30 min)
- Install Ollama from its download page.
- Run
ollama run qwen3.5:0.8b(orqwen3.5:4b), ask a question, quit with/bye. - Call the OpenAI-compatible route:
curl -s http://localhost:11434/v1/chat/completions -H 'Content-Type: application/json' \ -d '{"model": "qwen3.5:0.8b", "messages": [{"role": "user", "content": "Say hello in French."}]}' - Write a table in the notebook: llama.cpp versus Ollama, install, port, how to choose the model, context setting, what you liked.
With your AI: "I run both llama.cpp's llama-server and Ollama. Make a comparison table of how each sets the context length, the number of GPU layers and the model to load, with the documentation link for each claim. Mark any row you are not sure of as 'to check'."
Recap (10 min)
Three lines: what Ollama adds, its port, why one API format matters. Redo the curl call from memory.
Day 10 · 12/11 — The chat API, and prompting 1
Goal: call the chat API from Python with roles, and write a prompt that can be checked.
Theory 1/3: messages and roles
A chat request is a list of messages, each with a role: system (standing instructions), user (the request), assistant (the model's earlier answers). The API is stateless: you resend the whole history at every call. Options such as temperature, max_tokens and stream travel with the request. The response carries the text and a usage block with token counts.
Does the server remember your previous request?
- Yes, it keeps the conversation
- No: you resend the history each time
Answer
Stateless. The history is your program's job.
Theory 2/3: prompting for this job, the anatomy
Weak prompt: "Tell me about my GPU." What is wrong: no role, no input (which GPU?), no task you can check, no format. Edited prompt:
Role: you are a hardware assistant for someone running local LLMs.
Context: here is the output of nvidia-smi: [paste].
Task: give the card's name, total memory, memory used, and driver version.
Constraints: use only values present in the output; write "absent" otherwise.
Format: a JSON object with keys name, total_mib, used_mib, driver.
Check: I will compare each value with the raw output.
Six parts: role and context, task, constraints, inputs, output format, how you will check. Add an example when the format is unusual.
Which prompt is better for an agent?
- "Summarise this log."
- "From the log below, list each ERROR line's time and message as JSON lines; ignore WARN; say 'none' if there is no error."
Answer
The second: input, task, constraint, format and a defined empty case, all checkable.
Theory 3/3: fix this prompt
"Write me a good script." What is the most important missing piece?
- Politeness
- A longer sentence
- The task itself: what the script does, on what input, in which language, and how you test it
Answer
Without a defined task and test, nothing can be checked.
Practice (30 min)
- In
ainet-lab:uv add openai. - Write
chat.py:import os from openai import OpenAI client = OpenAI(base_url=os.environ.get("LLM_BASE_URL", "http://127.0.0.1:8080/v1"), api_key=os.environ.get("LLM_API_KEY", "none")) reply = client.chat.completions.create( model=os.environ.get("LLM_MODEL", "local"), messages=[{"role": "system", "content": "Answer in one sentence."}, {"role": "user", "content": "What is a token?"}], temperature=0.2) print(reply.choices[0].message.content) print(reply.usage) - Run it against llama.cpp, then against Ollama by changing only the environment variables.
- Rewrite the "Tell me about my GPU" prompt with the six parts for your own machine and test it.
With your AI: "Here is a prompt I wrote: [paste]. Review it against six parts: role and context, task, constraints, inputs, output format, how I will check the answer. For each part, say whether it is present, missing or vague, and suggest one concrete fix. Do not rewrite the whole prompt for me."
Let an agent do it: having written chat.py yourself, ask a coding agent: "Add a --stream option to chat.py that prints tokens as they arrive, keeping the environment variables; add a test that runs the script with a fake server." Check by reading the diff and running it against your real server.
Recap (10 min)
Three lines: the three roles, why the API is stateless, the six parts of a prompt. Rewrite the GPU prompt from memory.
Day 11 · 13/11 — Measuring speed
Goal: measure prompt speed, generation speed and time to first token, and write them down properly.
Theory 1/2: three numbers
Report three numbers, never one: prompt processing in tokens per second (pp), text generation in tokens per second (tg), and the time to first token for a realistic prompt. llama.cpp ships llama-bench, which measures pp and tg with repetitions (5 by default) and prints a mean and a spread. Write down the model file, the quantisation, the options and the runtime version with every result: a number without its conditions cannot be compared.
Write tps(tokens, ms): tokens per second from a count and a duration in milliseconds.
function tps(tokens, ms) {
}
Answer
One line is enough:
function tps(tokens, ms) {
return tokens / (ms / 1000);
}Theory 2/2: measuring honestly
Warm up first (the first run loads the model), repeat, keep the spread, change one thing at a time. Close other heavy programs. Remember that a benchmark tool measures the machine as much as the model: the site's note which local model for which job shows why real work favoured a different model than raw speed.
You change both the quantisation and the number of GPU layers, and speed doubles. What can you conclude?
- The quantisation doubled the speed
- Nothing about either alone: change one thing at a time
Answer
Two changes, one result: you cannot attribute it.
Practice (30 min)
- Run
llama-bench -m <your model file>(add-ngl 99with a GPU). The file is in the llama.cpp cache when you used-hf; the server log prints its path. - Run it again with half the GPU layers, or a different quantisation. Only one change.
- Time a real request with
curl -w '%{time_starttransfer}\n'and"stream": trueto get the time to first token. - Write a results table in
bench/results.mdwith model, file, options, version, pp, tg, TTFT.
With your AI: "Here are my llama-bench results: [paste the table]. Explain what pp and tg mean in this table, which differences are within the spread and which are real, and what single next measurement would be most informative. Show your reasoning with the numbers."
Let an agent do it: once you have run the bench by hand, an agent can run the sweep: "Run llama-bench on this model with -ngl 0, 16 and 99, output JSON, and append a Markdown table to bench/results.md with the version from llama-server --version." Check that the conditions in the table match what actually ran.
Recap (10 min)
Three lines: the three numbers, why one change at a time, what goes with every result. Redo tps without notes.
Day 12 · 14/11 — A bench that measures your job
Goal: build a ten-question test where code, not you, decides if each answer is right.
Theory 1/2: public benchmarks are not your job
Public leaderboards measure general skills on public questions, which may have leaked into training data. What you need is whether a model does your tasks reliably: extract this field, choose this tool, follow this format. A small personal set, checked by code, run on every candidate model, beats any leaderboard for that decision.
To choose a model for extracting dates from invoices, the best evidence is…
- its rank on a general leaderboard
- its score on twenty of your invoices, checked by code
Answer
Your task, your data, an automatic check.
Theory 2/2: checks by code
Write each case as an input plus a check that a program can run: an exact value, a regular expression, valid JSON with given keys, a number within a range. Avoid checks that need your judgement: they do not scale and drift with your mood.
Write score(answers, expected): the fraction of answers equal to the expected value, ignoring case and surrounding spaces.
function score(answers, expected) {
}
Answer
function score(answers, expected) {
const norm = s => String(s).trim().toLowerCase();
let ok = 0;
answers.forEach((a, i) => { if (norm(a) === norm(expected[i])) ok++; });
return ok / expected.length;
}Practice (30 min)
- Write
bench/cases.jsonl: ten cases of a task you care about (e.g. "extract the amount from this sentence"), each withpromptandexpected. - Write
bench/run.pythat sends each prompt to the model (temperature 0), compares withexpectedby code, and prints the score and the failures. - Run it on two models or two quantisations; note scores and times in
bench/results.md.
With your AI: "Here are my ten benchmark cases: [paste]. For each, tell me whether the check can be done by code without ambiguity, and if not, propose a stricter expected answer. Then suggest three additional edge cases (empty input, two candidates, a trap), with their expected values."
Let an agent do it: now that your runner exists, an agent can extend it: "Add a --model option to bench/run.py, run it on these two model names, and write a comparison table; do not change cases.jsonl." Check that the cases are untouched (git diff bench/cases.jsonl must be empty) and rerun one case yourself.
Recap (10 min)
Three lines: why your own bench, what a good check is, your two scores. Redo score without notes.
Day 13 · 15/11 — Chat templates and thinking models
Goal: understand how messages become tokens, and handle models that "think" before answering.
Theory 1/2: the chat template
The model sees one long text, not a list of messages. A chat template, usually written in Jinja and stored in the GGUF, turns roles into the special markers the model was trained with. A wrong template gives odd answers or broken tool calls. llama-server applies the model's template with its Jinja engine (on by default in current builds; option --jinja), and you can override it with --chat-template-file. Its function calling documentation lists the formats it recognises.
A model answers well in its own app but breaks its tool calls in your runtime. A likely cause is…
- the model is too small
- a wrong or missing chat template
Answer
The template decides how tools and roles are written for the model.
Theory 2/2: reasoning models
Some models are trained to write a reasoning section before the answer (the Qwen3.5 family has a "thinking" mode, per its Ollama page). It often helps on hard tasks and costs tokens and time. Runtimes may return that part in a separate field. For an agent: keep it out of the history you resend unless the model's documentation says otherwise, and turn it off for simple decisions.
For a yes/no routing decision repeated a thousand times a day, a long thinking phase is…
- usually wasted time and tokens
- always necessary
Answer
Simple decisions do not need it; measure if unsure.
Practice (30 min)
- Call
curl -s http://127.0.0.1:8080/propsand find the chat template in the answer. - Identify the role markers in it and write them down.
- Ask the same tricky question with thinking on and off (see the model card for how), compare length, time and correctness, and note it.
With your AI: "Here is a Jinja chat template: [paste]. Walk me through what text it produces for one system message, one user message and one assistant message, writing the exact output. Then ask me to predict the output for a two-turn conversation and correct me."
Recap (10 min)
Three lines: what a template does, the symptom of a wrong one, when to turn thinking off. Redo the first exercise without notes.
Day 14 · 16/11 — Module 1 deliverable: your machine card
Goal: publish a documented, measured "machine card": what your machine runs, how fast, and how well on your task.
Theory 1/2: what a recruiter reads
A good technical deliverable shows conditions, numbers and limits. It says what was measured, how to reproduce it, and what it does not prove. Claims without a command to reproduce them are opinions.
Which line belongs in a machine card?
- "This model is the best."
- "Qwen3.5-4B Q4_K_M, -ngl 99, -c 8192: tg 42 t/s ±1 (llama-bench, 5 runs), 8/10 on bench/cases.jsonl."
Answer
Conditions, figure, spread, method. (The figures here are an example, not a measurement.)
Theory 2/2: review your own prompts
Look back at the prompts you wrote this fortnight. For each, ask the six-part question. A prompt you keep in the repository (prompts/) is versioned like code: you can see why it changed.
Where should the prompts of your bench live?
- Only in your head
- In a chat history
- In the repository, versioned with the code
Answer
Versioned, so a change can be explained and reverted.
Practice (30 min)
- Write
README.mdinainet-lab: machine (from day 7), runtimes and how you installed them, models tested with files and licences, the speed table, the bench score, how to rerun everything, and three limits. - Add a small diagram (text is fine): client → runtime → model file → hardware.
- Commit, and push to a public forge if you want to show it (remove anything personal first).
With your AI: "Here is my README for a local LLM machine card: [paste]. Act as a technical reviewer hiring a junior AI engineer. List what is unclear, what is not reproducible, and any claim without a measurement. Do not praise; give at most five concrete changes."
Let an agent do it: now you know each part, a coding agent can check reproducibility: "Follow README.md from a clean shell, run each command, and report any step that fails or is ambiguous, without changing the files." Read its report and fix the README yourself.
Recap (10 min)
Three lines in the notebook: what you can now do, what is still fuzzy, what you want from module 2. Answer the first exercise of day 4 again without notes.
Sources
- Vaswani et al. (2017), Attention Is All You Need
- Sennrich, Haddow, Birch (2015), Neural Machine Translation of Rare Words with Subword Units
- Holtzman et al. (2019), The Curious Case of Neural Text Degeneration (nucleus sampling)
- llama.cpp, its server README, quantize README and llama-bench README
- Ollama documentation
- Hugging Face — GGUF
- Hugging Face LLM course
- Andrej Karpathy, Intro to Large Language Models and Let's build the GPT Tokenizer
Found this useful? Tell us what to measure next.