Prompts module 4: project jobs, agents, evaluation and the final library

Day 44 · 16/12 — Project management: from meeting notes to actions

Goal: turn raw notes into decisions and actions without the model inventing owners or dates.

Theory 1/2: the weak prompt

Summarise this meeting. (paste of notes)

Diagnosis: a summary flattens what matters. After a meeting you need three lists: what was decided, who does what by when, and what is still open. A model tends to fill gaps: an action with no owner gets one, a vague "soon" becomes a date. Ask for the lists and for explicit gaps.

Theory 2/2: the edited prompt

From the notes below, produce three lists:
1) Decisions: what was decided, quoted from the notes.
2) Actions: | action | owner | due date | as stated in the notes;
   write [NONE] when the notes do not name an owner or a date.
3) Open questions: what was raised and not settled.
Then a 3-line status for people who were absent.
Use no name or date that is not in the notes.
<notes>…</notes>

Practice (30 min)

1. Write notes of an imaginary 20-minute project meeting (or take notes of a public meeting recording, such as an open-source project call), with at least one action without an owner.
2. Run the weak and edited prompts. In the edited output, check every name and date against the notes.
3. Card prompts/pm-meeting-actions.md.

With your AI: "Here are my meeting notes and the action list I made [paste]. Do not make your own list. Tell me which decisions or actions in the notes I missed, quoting the line each comes from."

Recap (10 min)

Three lines: three lists, [NONE] over guesses, the status for the absent. Then redo the exercises without notes.

Day 45 · 17/12 — Project management: user stories, risks and estimates

Goal: split a feature into stories a team can check, list risks, and treat model estimates for what they are.

Theory 1/2: stories with acceptance criteria

A user story names a user, a need and a benefit ("As a … I want … so that …"), and comes with acceptance criteria that say when it is done. A model is good at proposing a split and at drafting criteria; it does not know your users or your priorities. Give it the goal and the constraints, and ask for small, independent stories.

Feature: customers can cancel a booking online up to 24 h before.
Users: customers on phone; staff using the back office.
Task: split into at most 6 user stories, each small enough for one
developer in 2 days, each with 2 to 4 acceptance criteria in
Given/When/Then. Mark any story that depends on another.
List the questions I must answer before the stories are ready.

Theory 2/2: risks and estimates

Ask for risks with what would show them early (a signal) and what to do (a response), and rank them yourself. For estimates, a model has no knowledge of your team's speed or your codebase; its numbers are guesses with a confident tone. Use it to list the work items and their unknowns; estimate with the team.

Practice (30 min)

1. Pick a feature of one of your projects. Run the story prompt; edit the stories by hand until each criterion is checkable.
2. Ask for five risks with signal and response; rank them yourself and keep the top three.
3. Card prompts/pm-stories-risks.md.

With your AI: "Here are my six user stories [paste]. Check each against INVEST (independent, negotiable, valuable, estimable, small, testable) and tell me, for each letter that fails, which sentence of the story causes it."

Let an agent do it: once you have written stories by hand, an agent with access to your issue tracker can create the issues from an approved list: "Create one issue per story in stories.md, with the acceptance criteria in the body and the label 'ready'; do not assign anyone." You still check that each issue matches the approved text.

Recap (10 min)

Three lines: stories with checkable criteria, risks with signals, estimates belong to the team. Then redo the exercises without notes.

Day 46 · 18/12 — Technical writing: the right kind of document

Goal: ask for the kind of document the reader needs, not "some documentation".

Theory 1/2: four kinds of documentation

The Diátaxis framework, by Daniele Procida, separates four kinds: tutorials (learning by doing, for a beginner), how-to guides (steps to reach a goal, for someone who knows the basics), reference (exact description, to look things up) and explanation (why things are as they are). "Write the docs" mixes them; a good prompt names one.

Theory 2/2: the edited prompt

Weak:   Write documentation for my backup script.
Edited: Write a how-to guide: "Restore one file from last night's
        backup". Reader: a colleague who knows the shell but has
        never used this script. Use only the options in the --help
        output below; show each command in a code block and what
        the reader should see after it. End with "If it fails",
        listing the two most common errors from the script's
        source below and what they mean.
        <help>…</help> <source>…</source>

Practice (30 min)

1. Take a script or small tool you wrote during this path. Ask for a how-to guide and a reference with the edited pattern.
2. Follow the how-to guide literally in a fresh terminal. Note every place it is wrong or unclear.
3. Card prompts/doc-howto.md.

With your AI: "Here is a page of my documentation [paste]. Tell me which Diátaxis kind it is trying to be, which sentences belong to another kind, and where those sentences should go."

Recap (10 min)

Three lines: the four kinds, naming the reader, real inputs. Then redo the exercises without notes.

Day 47 · 19/12 — Technical writing: editing for the reader

Goal: use a model as an editor that proposes changes you accept one by one.

Theory 1/2: edit, do not rewrite

"Improve this text" returns a new text in the model's voice, often longer, sometimes with changed facts. An editing prompt sets the rules (Google's free technical writing courses give a good set: short sentences, active voice, one idea per sentence, define terms) and asks for a list of changes, each with its reason, that you accept or refuse.

Theory 2/2: the edited prompt

Edit the README section below for a reader new to the project.
Rules: sentences under 25 words; active voice; define each term
the first time; one instruction per numbered step; keep every
command and every fact exactly as is.
Output: a table | original sentence | proposed sentence | rule |.
Do not change anything inside code blocks.
<section>…</section>

Practice (30 min)

1. Take the README of your prompt library. Run the editing prompt.
2. Accept or refuse each line; for each refusal, write why in a word.
3. Compare the result with the original using git diff --word-diff; check no command changed.
4. Card prompts/doc-edit.md.

With your AI: "Quiz me on plain technical writing: show me one sentence at a time from a typical README, ask me to rewrite it under the rules (short, active, defined terms), then compare my version with yours and say which is clearer and why."

Recap (10 min)

Three lines: edit not rewrite, the rules, facts and code untouched. Then redo the exercises without notes.

Day 48 · 20/12 — Job search: a truthful CV for a self-taught learner

Goal: use a model to present real evidence well, never to invent experience.

Theory 1/2: the weak prompt

Write me a CV for a junior DevOps job.

Diagnosis: the model knows nothing about you, so it writes a generic CV with skills and results you do not have. A recruiter or an interview will find out. For a self-taught learner, the strength is the evidence: projects with a link, what you built, what you measured. The model's job is to phrase that evidence for a specific job offer.

Theory 2/2: the edited prompt

Context: self-taught, no IT diploma; I am applying for the job offer
below. My evidence: <projects>(each project: link, what I built, tools,
what I measured)</projects> <offer>…</offer>
Task: write the "Projects" section of my CV for this offer:
3 projects at most, the most relevant first, 2 bullet points each,
starting with a verb, using only facts from my evidence.
Then list the requirements of the offer my evidence does not cover,
so I can address them honestly in the cover letter.

Practice (30 min)

1. Write your evidence file: every project of this path and the others you followed, with link and facts.
2. Find a real public job offer that interests you. Run the edited prompt.
3. Check every bullet against your evidence; strike anything you cannot prove.
4. Card prompts/career-cv-projects.md (keep your personal evidence out of the public repository).

With your AI: "Here is my Projects section and the job offer [paste]. Act as a recruiter reading it for 30 seconds: what do you remember, what makes you doubt, and which bullet would you ask about first?"

Recap (10 min)

Three lines: evidence over claims, one offer at a time, the gaps list. Then redo the exercises without notes.

Day 49 · 21/12 — Job search: interview practice

Goal: rehearse an interview with a model that asks, follows up and gives honest feedback.

Theory 1/2: the weak prompt

Give me the top interview questions for a sysadmin job and the answers.

Diagnosis: memorised answers sound memorised, and the model's answers are not your experience. Practice works when the model plays the interviewer, asks one question at a time, follows up on your answer, and gives feedback at the end against criteria you set.

Theory 2/2: the edited prompt

Play the interviewer for the job offer below; I am a self-taught
candidate. Ask one question at a time and wait for my answer.
Mix: 3 technical questions on the offer's tools, 2 behavioural
questions ("tell me about a time…"), 1 question about a project
of mine from my evidence. Follow up once on each answer.
After the last one, give feedback per answer: was it specific,
did it follow Situation / Task / Action / Result, what was missing.
Do not answer the questions for me.
<offer>…</offer> <evidence>…</evidence>

Practice (30 min)

1. Run the mock interview with yesterday's offer. Answer out loud first, then type.
2. For every technical question, check the facts of your answer in the official documentation afterwards.
3. Rewrite your two weakest answers in STAR form and run those questions again.
4. Card prompts/career-mock-interview.md.

With your AI: "I am going to explain my final project of this path in two minutes. Interrupt me like a technical interviewer with the three hardest questions you can find, one at a time."

Recap (10 min)

Three lines: interviewer not answer sheet, follow-ups, STAR. Then redo the exercises without notes.

Day 50 · 22/12 — System prompts for agents

Goal: write the system prompt of a small agent: who it is, what it may do, when it stops.

Theory 1/2: an agent reads the prompt differently

An agent is a model in a loop: it reads its instructions, calls tools, reads the results, and goes on until it decides it is done. Its system prompt is read at every step, by the model, without you there to clarify. Anthropic's "Building effective agents" (2024) advises starting with the simplest system that works and adding autonomy only when needed. A system prompt for an agent states: its goal, its environment, the tools and when to use each, the rules it must never break, how to report, and when to stop and ask a human.

Theory 2/2: weak and edited

Weak:   You are a helpful DevOps agent. Fix any problems you find.

Edited: Goal: keep the disk usage of the lab server under 80%.
Environment: Debian 13 lab VM; you have the tools disk_usage,
list_large_files and delete_file.
Rules: delete only files under /var/tmp/lab older than 7 days;
never delete anything else, never use another directory.
Each run: 1) disk_usage; 2) if under 80%, report and stop;
3) list_large_files in /var/tmp/lab; 4) propose deletions as a
list and stop: a human approves before delete_file is used.
Report: usage before, files proposed, total size.
Stop and ask if: a tool returns an error twice, or usage is above
95%, or the files are not in /var/tmp/lab.

Practice (30 min)

1. Pick a small recurring task of your own (tidy a downloads folder, check that three sites answer). Write its agent system prompt with the six parts.
2. Run it in a coding agent restricted to a lab directory (the site's articles on OpenFox and Claude Code show how to set one up on local models). Watch every step.
3. Note where the agent hesitated or overstepped; fix one part of the prompt; run again.
4. Card prompts/agent-system-prompt.md.

With your AI: "Here is my agent's system prompt [paste]. Play a red team: list five situations in which an agent following it to the letter would still do something harmful or useless, and which line of the prompt allows it."

Recap (10 min)

Three lines: the six parts of an agent prompt, scope first, the stop conditions. Then redo the exercises without notes.

Day 51 · 23/12 — Project instruction files and the context an agent needs

Goal: write the file a coding agent reads in your repository, and keep it short and true.

Theory 1/2: the instruction file

Coding agents read a file at the root of the repository before working: AGENTS.md is an open format read by several agents; Claude Code reads CLAUDE.md. It holds what an agent cannot guess and needs every time: how to build and test, the conventions, what never to touch. Anthropic's "Effective context engineering for AI agents" describes the context as a limited budget: every line competes with the task for the model's attention.

Theory 2/2: weak and edited

Weak:   This is a great project. Please write high-quality code
        and follow best practices. Be careful.

Edited: # Build and test
        uv sync; uv run pytest -q (must pass before any commit)
        # Conventions
        Python 3.13, ruff format; English in code and commits.
        # Never
        Edit files under migrations/ by hand; commit secrets;
        change the public API in api/ without saying so in the PR.
        # When unsure
        Stop and ask in the PR description.

Practice (30 min)

1. Write AGENTS.md for your prompt library repository: how to run a prompt, how to run the tests (you will have them on day 56), the card template, what never to change.
2. Give a coding agent a small task in the repository (add a card from a description). Check whether it followed every line of the file.
3. Cut any line the agent did not need; add the one it lacked.

With your AI: "Here is my AGENTS.md [paste]. For each line, tell me whether an agent could check it is following it, and propose a checkable rewording for those it could not."

Let an agent do it: agents can update their own instruction file when they learn something about the project. Ask for the change as a proposal in a pull request, never a silent edit, and read it: a wrong line in this file misleads every later run.

Recap (10 min)

Three lines: what goes in the file, context as a budget, checkable lines. Then redo the exercises without notes.

Day 52 · 24/12 — Tool descriptions: the prompt inside the tool

Goal: write a tool description an agent uses at the right moment, with the right arguments.

Theory 1/2: why descriptions matter

When an agent has tools (functions, or servers speaking the Model Context Protocol, which Anthropic published in November 2024), it chooses which to call from their name, their description and their parameter schema, and nothing else. Anthropic's tool-use documentation calls detailed descriptions the most important factor in tool performance, and its article "Writing effective tools for agents" (2025) adds: clear names, descriptions that say when to use and when not to, parameters with formats and examples, and error messages that tell the agent what to do next.

Theory 2/2: weak and edited

Weak:
{"name": "check", "description": "Checks a site.",
 "input_schema": {"type": "object", "properties": {"s": {"type": "string"}}}}

Edited:
{"name": "check_https_site",
 "description": "Fetches https://HOST/ once and returns the HTTP status
  code, the response time in ms and the certificate expiry date. Use it
  to check that a public website answers. Do not use it for internal
  hosts or for anything other than the home page. On timeout it returns
  status 0: retry once, then report the site as down.",
 "input_schema": {"type": "object", "required": ["host"],
  "properties": {"host": {"type": "string",
   "description": "Hostname only, no scheme or path, e.g. www.example.com"}}}}

Practice (30 min)

1. Write the edited tool as a small script that does what its description says (Python urllib or curl), and its description in JSON.
2. Expose it to an agent that supports custom tools (an MCP server, or the tool-calling interface of your runtime). Ask the agent three requests: one that needs it, one that does not, one with a URL instead of a hostname.
3. Note each call and its arguments. Fix the description where the agent went wrong; rerun.
4. Card prompts/tool-description.md with the three requests as tests.

With your AI: "Here are three tool descriptions [paste]. For each, write two user requests where an agent would likely pick the wrong tool or pass a wrong argument because of the description, and say which words cause it."

Recap (10 min)

Three lines: name, description with when and when not, parameters with examples, errors with next steps. Then redo the exercises without notes.

Day 53 · 25/12 — Testing prompts: cases and assertions

Goal: turn your checks into test cases a program runs.

Theory 1/2: a test set for a prompt

Since day 8 you have checked prompts by reading. To compare versions and models, the checks must run alone. A prompt test set holds inputs, and for each, assertions on the output: contains this, does not contain that, is valid JSON, is under N words, matches a value. Anthropic's guide on building evaluations and OpenAI's evals guide give the same advice: write the tests from real tasks, include edge cases, prefer checks a program can grade, and keep the set when you change the prompt.

Theory 2/2: write the checker

function check(output, assertion) {
}

function check(output, assertion) {
  if (assertion.type === 'contains') return output.includes(assertion.value);
  if (assertion.type === 'not-contains') return !output.includes(assertion.value);
  if (assertion.type === 'max-words') {
    const words = output.trim().split(/\s+/).filter(Boolean);
    return words.length <= assertion.value;
  }
  return false;
}

Practice (30 min)

1. Take three cards of your library. For each, write five test cases in a file tests/<card>.json: input variables and a list of assertions.
2. Include for each card one "trap" case: the healthy log, the injection attempt, the ambiguous alert.
3. Run them by hand once, recording pass or fail per assertion. Those are tomorrow's baseline.

With your AI: "Here is a prompt card and my five test cases [paste]. Propose three more cases that would catch a regression my cases miss, and say for each which assertion would fail."

Recap (10 min)

Three lines: what a test case holds, assertions a program grades, the trap case. Then redo the exercises without notes.

Day 54 · 26/12 — Testing prompts: promptfoo, several models, judges

Goal: run a prompt's tests automatically on two models, and know the limits of a model as judge.

Theory 1/2: promptfoo

promptfoo is an open-source command-line tool: a YAML file lists prompts, providers (models, including local Ollama ones) and tests with assertions; promptfoo eval runs every combination and shows a pass/fail grid. Built-in assertion types include contains, icontains, not-contains, regex, is-json and javascript.

prompts:
  - file://prompts/ops-alert-routing.txt
providers:
  - ollama:chat:MODEL_A
  - ollama:chat:MODEL_B
tests:
  - vars: {alert: "disk /var 95% full on web1"}
    assert:
      - type: is-json
      - type: javascript
        value: JSON.parse(output).team === 'systems'
  - vars: {alert: "Coffee machine offline"}
    assert:
      - type: javascript
        value: JSON.parse(output).team === 'other'

Theory 2/2: a model as judge

Some qualities (tone, clarity, completeness against a rubric) need judgement. promptfoo's llm-rubric asks a model to grade the output against your rubric. It is useful and fallible: judges have biases (for instance toward longer answers, or toward their own style), so write a precise rubric, check a sample of the judge's verdicts by hand, and keep code assertions wherever they can do the job.

Practice (30 min)

1. With Node.js installed: npx promptfoo@latest init in your library, then write promptfooconfig.yaml for the alert-routing card with two local models.
2. npx promptfoo@latest eval, then npx promptfoo@latest view. Compare with yesterday's hand baseline.
3. Break the prompt on purpose (remove the "other" definition): the trap test must fail. If not, fix the test.

With your AI: "Here is my rubric for a support reply [paste]. Rewrite it so that two different people grading the same reply would almost always agree, and give one reply that passes and one that fails it."

Let an agent do it: now that you have written a config by hand, a coding agent can convert your remaining tests/*.json files into promptfoo tests and run the suite: "Convert every file in tests/ into promptfooconfig.yaml tests with the same assertions, run promptfoo eval on both local models, and give me the pass rate per card." You still check that each converted assertion means what the original did.

Recap (10 min)

Three lines: what promptfoo runs, why two models, how far to trust a judge. Then redo the exercises without notes.

Day 55 · 27/12 — Final project 1: choose and audit

Goal: choose the ten prompts of your final library and audit each against the card.

Theory 1/2: what the final project is

A public repository with ten prompts you actually use, at least five jobs covered, each as a full card (purpose, prompt, inputs, output, checks, history), each with automated tests run on two models, a README that explains how to use and test them, and a short report of three prompts you improved with measured results. It is the evidence on your CV (day 48).

Theory 2/2: the audit grid

For each candidate prompt, score 0 to 2: role and context, task, constraints, inputs fenced as data, output format, examples (if needed), success criteria, a way out ("not stated", [MISSING], other), injection risk handled if it reads outside data, nothing private in it.

Practice (30 min)

1. List all your cards; pick ten that cover at least five jobs and that you would really reuse.
2. Audit each with the grid; record the score in its card.
3. Open one issue per card scoring under 14 out of 20 in your repository, saying what to fix.

With your AI: "Here is one of my prompt cards [paste]. Score it on my audit grid, one line per criterion with a reason, and point to the single change that would raise the score most."

Recap (10 min)

Three lines: what the final project contains, the audit grid, the issues opened. Then redo the exercises without notes.

Day 56 · 28/12 — Final project 2: write the tests

Goal: every prompt of the library has an automated test suite with a trap case.

Theory 1/2: a good suite

Five to ten cases per prompt: typical inputs, edge inputs, at least one trap (injection, missing information, input outside the labels), and one case checking that the prompt says "I don't know" when it should. Assertions graded by code first; a rubric judge only for what code cannot grade.

Theory 2/2: check that a test can fail

For each prompt, break it once on purpose (remove the way out, remove the fence, change the format) and run the suite: at least one test must fail. A suite that stays green on a broken prompt protects nothing; it is the mutation idea of day 30, applied to prompts.

Practice (30 min)

1. Write the promptfoo tests of all ten prompts (the agent tip of day 54 can help, after you wrote the first ones).
2. Break each prompt once; confirm a test fails; restore it.
3. Commit with the message listing, per prompt, which test caught the mutation.

With your AI: "Here is a prompt and its test suite [paste]. Propose a mutation of the prompt that my suite would not catch, and the test that would."

Recap (10 min)

Three lines: what a good suite holds, the trap with no answer, the mutation check. Then redo the exercises without notes.

Day 57 · 29/12 — Final project 3: run, fail, fix

Goal: run the whole library on two models, fix the failures with the day 8 loop, and measure.

Theory 1/2: reading a results grid

A grid of prompts by models by tests shows three patterns: a test that fails on every model (the prompt or the test is wrong), a test that fails on one model only (the prompt leans on one model's habits), and a prompt that passes everywhere but barely (fragile). Fix the first kind first.

Theory 2/2: one change, measured

For each fix, the loop of day 8: name the cause, change one thing, rerun the full suite on both models, keep the change only if the pass rate rises and nothing else breaks. Record before and after in ## History: that table is the measured difference you will show.

Practice (30 min)

1. Run the full suite on both models; save the results (promptfoo eval -o results.json).
2. Fix the three worst prompts with one change each; rerun after each.
3. Write the before and after of those three in their cards.

With your AI: "Here are the failing outputs of one prompt and its test [paste]. Classify each failure (missing information, misunderstood task, ignored format, invention, test too strict) and propose one change, not more."

Let an agent do it: once you have fixed prompts by hand with the loop, an agent can run the loop on the remaining ones: "For each prompt under 80% pass rate, propose exactly one change, rerun the full suite on both models, and keep the change only if the pass rate rises and no other test fails; write the before and after in the card's History." You still read every kept change and its diff.

Recap (10 min)

Three lines: the three patterns, one change measured, the before and after. Then redo the exercises without notes.

Day 58 · 30/12 — Final project 4: document and publish

Goal: make the library usable by someone else in ten minutes, and publish it.

Theory 1/2: the README

The README answers, in this order: what the library is; how to run one prompt; how to run the tests and on which models; the card template; the list of prompts by job with their pass rate; the redaction rule (nothing private in prompts or tests); the licence. Apply day 47's editing rules to it.

Theory 2/2: the improvement report

A short page, REPORT.md: three prompts you improved. For each: the weak version, the diagnosis in the terms of this path, the edited version, the test results before and after on both models, and what you learnt. Facts only: if a change did not help, say so.

Practice (30 min)

1. Write the README and REPORT.md.
2. Search the repository for anything private: git grep -nE "password|token|@|BEGIN .*KEY" and read every match; check addresses are documentation ones.
3. Ask someone (or yourself on a fresh machine) to clone it and run one prompt and its tests from the README alone. Fix what blocked them.
4. Make the repository public. Module 4 deliverable: its link.

With your AI: "Here is my README [paste]. You are a developer who has never seen this repository. List, in order, the steps you would take to run one prompt and its tests, and stop at the first step where the README does not tell you what to do."

Recap (10 min)

Three lines: the README order, the report of three improvements, the privacy check. Then redo the exercises without notes.

Day 59 · 31/12 — Final review

Goal: go through the end-of-path checklist and plan what comes next.

Theory 1/2: the path in seven sentences

A model continues text: give it the facts it cannot guess. Data is never instructions: fence it, and put the real defences in code. A prompt is a task with checkable success criteria. Edit one thing at a time and measure. Structured output is checked by code. A model cannot verify itself: verification is external. A prompt you cannot test is a sentence you hope works.

Theory 2/2: mixed review

Practice (30 min)

1. Go through the checklist of the path page; tick what you can show, with the link to the proof.
2. For each box you cannot tick, write the session to redo and when.
3. Pick your next path on this site, and the first prompt card you will add to your library for it.

With your AI: "Quiz me on this whole path: twelve questions, one at a time, mixing the foundations (context, sampling, injection, structured output, self-checks) and the jobs (one question per job family). After each answer, tell me which day of the path to reread if I was wrong."

Recap (10 min)

Write the seven sentences from memory. Then redo every exercise of the path you got wrong at least once.