DevOps module 4: observability, incidents, security, and the final project

Day 43 · 15/12 — Observability: metrics, logs, traces

Goal: say what each signal answers, and pick what to watch with the golden signals.

Theory 1/2: monitoring and observability

Monitoring watches for problems you predicted (the disk is full); observability is being able to ask new questions of a running system from what it emits, including problems nobody predicted. Three kinds of signals: metrics (numbers over time, cheap: requests per second), logs (timestamped events, detailed: this request failed with this error), traces (the path of one request across services, with the time spent in each).

Theory 2/2: golden signals, USE, RED

Google's SRE book names four golden signals for a user-facing service: latency, traffic, errors, saturation. For resources (CPU, disk, network), Brendan Gregg's USE method: utilisation, saturation, errors. For request-driven services, Tom Wilkie's RED method: rate, errors, duration.

Practice (30 min)

1. Read the chapter Monitoring Distributed Systems of the SRE book, the part on the four golden signals.
2. For hello-ops, write in your notebook what each golden signal would be, in concrete terms (which number, from where).
3. Write three questions you would like to answer about your app in production, and which signal answers each.

With your AI: "For a small HTTP API with a Redis cache, propose the RED metrics and the USE metrics I should collect, as a table with metric name, type (counter, gauge, histogram) and the question it answers. Then quiz me with three 'which signal would you look at' cases."

Recap (10 min)

Three lines: monitoring versus observability, the three signals, the golden signals. Then recite RED and USE without notes.

Day 44 · 16/12 — Prometheus 1: scraping and PromQL

Goal: run Prometheus, scrape a machine's metrics and write your first queries.

Theory 1/2: the pull model

Prometheus started at SoundCloud in 2012, joined the CNCF in 2016 and was its second graduated project (2018); version 3.0 came out in November 2024. It pulls: every few seconds it scrapes an HTTP endpoint (/metrics) on each target and stores the samples as time series, named and labelled: http_requests_total{path="/", code="200"}. Exporters expose metrics for things that do not: node_exporter for a Linux machine (port 9100).

# prometheus.yml
global:
  scrape_interval: 15s
scrape_configs:
  - job_name: node
    static_configs:
      - targets: ["node-exporter:9100"]

Theory 2/2: PromQL

A counter only goes up (requests served); a gauge goes up and down (memory used); a histogram counts values in buckets (request durations). With a counter you look at its rate:

up                                             # 1 if the target answered the last scrape
rate(node_cpu_seconds_total{mode!="idle"}[5m]) # CPU used per second
sum by (code) (rate(http_requests_total[5m]))  # requests per second, per code
histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))

Practice (30 min)

1. In a new folder monitoring/, write a compose.yaml with two services: prometheus (image prom/prometheus, port 9090, your prometheus.yml mounted read-only at /etc/prometheus/prometheus.yml) and node-exporter (image prom/node-exporter). Check the tags on the projects' pages.
2. docker compose up -d; open http://127.0.0.1:9090, Status → Targets: both up.
3. Run in the query page: up, then the CPU rate, then node_memory_MemAvailable_bytes, in graph mode.
4. Generate load (yes > /dev/null & for a minute, then kill %1) and watch the CPU curve.

With your AI: "Teach me PromQL with node_exporter metrics: give me five questions of increasing difficulty ('available memory in percent', 'CPU per mode'…). I write the query, you check it and explain the error if any. Do not give the answer before I try."

Let an agent do it: an agent can write PromQL from a question in plain words. You still run every query in the Prometheus UI and check one value by hand (for example memory against free -b): a query that runs is not a query that is right.

Recap (10 min)

Three lines: pull and scrape, counter / gauge / histogram, rate(). Then rewrite the scrape configuration without notes.

Day 45 · 17/12 — Prometheus 2: instrumenting the app, alerting rules

Goal: make hello-ops expose its own metrics and fire an alert when it goes down.

Theory 1/2: instrumenting

The Python client library creates metrics and renders them in the text format Prometheus reads. Label values must stay few (path, code): a label holding a user id or a full URL creates one series per value and overloads Prometheus (high cardinality).

from prometheus_client import CONTENT_TYPE_LATEST, Counter, generate_latest

REQUESTS = Counter("http_requests_total", "HTTP requests", ["path", "code"])

# in do_GET, before routing:
if self.path == "/metrics":
    data = generate_latest()
    self.send_response(200)
    self.send_header("Content-Type", CONTENT_TYPE_LATEST)
    self.end_headers()
    self.wfile.write(data)
    return
# after routing:
REQUESTS.labels(path=self.path if code != 404 else "other", code=str(code)).inc()

Theory 2/2: alerting rules and Alertmanager

Prometheus evaluates alerting rules; a rule that stays true for its for: duration fires, and Alertmanager groups, silences and routes alerts to e-mail, chat or a pager.

groups:
  - name: hello-ops
    rules:
      - alert: HelloOpsDown
        expr: up{job="hello-ops"} == 0
        for: 1m
        labels:
          severity: page
        annotations:
          summary: "hello-ops is not answering scrapes"

Practice (30 min)

1. Add prometheus-client to a requirements.txt, the code above to app.py, and install the requirements in the Dockerfile's final stage. Rebuild; curl http://127.0.0.1:8080/metrics.
2. Add hello-ops to the monitoring compose project (same network) and a scrape job hello-ops targeting app:8080.
3. Write the rule in rules.yml, mount it, reference it with rule_files. Stop the app: after a minute the alert is "firing" in Prometheus' Alerts page.
4. Add a test to test_app.py that the counter increases after a request (use REQUESTS.labels(...)._value.get() or the registry's get_sample_value).

With your AI: "Review my metrics code for cardinality and naming against the Prometheus naming best practices: [paste the code]. List problems with the exact line, and tell me which page of prometheus.io I should read to check each point."

Let an agent do it: an agent can add a duration histogram to the handler. Instruction: "Add a Histogram http_request_duration_seconds labelled by path; keep labels bounded; add a test; run pytest." You still check /metrics by hand and that the label values cannot grow with user input.

Recap (10 min)

Three lines: instrumenting, cardinality, rule plus Alertmanager. Then rewrite the alerting rule from memory.

Day 46 · 18/12 — Grafana: dashboards

Goal: build a dashboard of the golden signals for hello-ops, and keep it as code.

Theory 1/2: data sources and panels

Grafana (Torkel Ödegaard, 2014) draws dashboards from data sources: Prometheus for metrics, Loki for logs, Jaeger or Tempo for traces. A panel is a query plus a visualisation; variables make one dashboard serve every instance. A good dashboard answers one question at a glance: is the service healthy, and if not, where.

Theory 2/2: dashboards as code

A dashboard clicked together is a pet. Grafana can provision data sources and dashboards from files at start-up (YAML for providers, JSON for dashboards): they live in Git, are reviewed, and survive a rebuilt container.

Practice (30 min)

1. Add grafana (image grafana/grafana, port 3000) to the monitoring stack, with a provisioning file declaring Prometheus at http://prometheus:9090.
2. Build a dashboard with four panels: requests per second, error ratio, p95 latency (if you added the histogram), and up.
3. Generate traffic with a loop of curl, some on /nothing for 404s; watch the panels.
4. Export the dashboard as JSON, save it in monitoring/dashboards/, provision it, recreate the container: it is still there.

With your AI: "Here are the metrics my app exposes: [paste /metrics]. Propose a four-panel dashboard of the golden signals: for each panel, the PromQL query, the visualisation type and the threshold that should turn it red. I will build it myself."

Let an agent do it: agents can generate dashboard JSON. You still load it, check each panel shows data matching a query you run yourself in Prometheus, and remove panels nobody would look at.

Recap (10 min)

Three lines: data sources and panels, one question per dashboard, dashboards as code. Then rewrite the error-ratio query from memory.

Day 47 · 19/12 — Logs: structured, collected, searched

Goal: make the app log one JSON line per request, collect the logs, and search them next to the metrics.

Theory 1/2: structured logs to stdout

The twelve-factor app (factor XI) treats logs as an event stream written to stdout; the platform (journald, Docker, Kubernetes) collects them. A structured log line is machine-readable, with stable field names, so you filter on fields rather than guessing with grep:

{"ts": "2026-12-19T10:42:07Z", "level": "info", "path": "/health", "code": 200, "ms": 1.8}

Never log secrets, tokens or personal data in clear: logs are copied to many places and kept a long time.

Theory 2/2: Loki and its collector

Loki (Grafana Labs, 2018) stores logs indexed only by a few labels (job, container), which keeps it cheap; you query with LogQL: {container="hello-ops"} | json | code >= 500. Logs are shipped by a collector: Promtail did it for years, but it reached end of life on 2 March 2026; Grafana now points to Grafana Alloy, and other collectors (Fluent Bit, the OpenTelemetry Collector) can also send to Loki.

Practice (30 min)

1. Override log_message (or log_request) in the handler to print one JSON line per request with json.dumps, to stderr. Rebuild; docker compose logs app shows JSON lines.
2. Add Loki and a collector to the monitoring stack, following Grafana's Loki quick start (it ships a Compose example with Alloy); point it at your containers' logs.
3. In Grafana, add Loki as a data source; in Explore, find all 404 lines of the last 15 minutes with LogQL.
4. Without the stack: docker compose logs app | grep '"code": 404' | wc -l, and compare the counts.

With your AI: "Here are ten of my JSON log lines: [paste]. Teach me LogQL on them: give me four questions (errors per path, slowest requests…), I write the query, you check it. Tell me which field names you would rename and why."

Let an agent do it: an agent can search logs for you across a time range and summarise. Ask it to give the exact LogQL or command it ran. You still rerun it and look at a few raw lines: summaries hide the odd line that matters.

Recap (10 min)

Three lines: logs to stdout, structured fields, Loki labels and its collector. Then rewrite the LogQL query from memory.

Day 48 · 20/12 — Traces: OpenTelemetry and Jaeger

Goal: emit a trace from the app and read it in Jaeger.

Theory 1/2: spans and context

A trace is the tree of spans of one request; a span is one operation with a start, a duration, attributes and a parent. Across services, the context (trace id, span id) travels in HTTP headers (the W3C traceparent header), so each service attaches its spans to the same trace. Google's Dapper paper (2010) described the idea; OpenTelemetry, born in 2019 from the merger of OpenTracing and OpenCensus and hosted by the CNCF, is today's standard API, SDK and wire protocol (OTLP) for traces, metrics and logs.

Theory 2/2: instrumenting by hand

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("hello-ops")

# in do_GET:
with tracer.start_as_current_span("GET " + self.path) as span:
    code, body = route(self.path)
    span.set_attribute("http.response.status_code", code)

The exporter reads its endpoint from OTEL_EXPORTER_OTLP_ENDPOINT and the service name from OTEL_SERVICE_NAME: configuration stays outside the code.

Practice (30 min)

1. Add opentelemetry-sdk and opentelemetry-exporter-otlp-proto-http to requirements.txt, and the code above to app.py.
2. Add Jaeger to the monitoring stack following its getting started page (all-in-one image: UI on 16686, OTLP over HTTP on 4318). Set OTEL_EXPORTER_OTLP_ENDPOINT=http://jaeger:4318 and OTEL_SERVICE_NAME=hello-ops on the app.
3. Send a few requests; open the Jaeger UI, pick hello-ops, open a trace, read its span and attribute.
4. Add a child span around a fake slow step (time.sleep(0.2) on / only) and find it in the trace. Remove it afterwards.

With your AI: "Explain, with my code below, what happens from the moment a request arrives to the moment its span appears in Jaeger: SDK, processor, exporter, OTLP, collector. One line per step. Then tell me what I would need to add for a second service to join the same trace: [paste the code]."

Let an agent do it: an agent can add spans around every function worth timing. You still read a trace and check the spans nest as the code does, and that no attribute carries personal data or a secret.

Recap (10 min)

Three lines: trace and span, context propagation, what OpenTelemetry is. Then rewrite the four set-up lines from memory.

Day 49 · 21/12 — SLIs, SLOs, error budgets, alerts that matter

Goal: define an SLO for hello-ops, compute its error budget, and alert on symptoms, not causes.

Theory 1/2: SLI, SLO, SLA

An SLI (service level indicator) is a measure of what users feel: the share of requests answered successfully in under 300 ms. An SLO (objective) is the target for it over a window: 99.5 % over 30 days. An SLA (agreement) is a contract with consequences if missed, set looser than the SLO. 100 % is the wrong target: users cannot tell 99.99 % from 100 %, and it would forbid any change.

Theory 2/2: the error budget

The error budget is what the SLO allows to fail: 100 % − SLO. At 99.9 % over 30 days (43,200 minutes) it is 0.1 %, about 43 minutes of full outage. While budget remains, the team ships; when it is spent, reliability work comes first. Alerts should page on symptoms users feel (the budget burns too fast), not on every cause (CPU at 80 %), or people stop reading them.

Practice (30 min)

1. Write the SLI of hello-ops as a PromQL ratio: successful (non-5xx) requests over all requests, over 30 days.
2. Choose an SLO, compute the budget in minutes and in failed requests for your traffic, write both in your notebook.
3. Write an alerting rule on the error ratio over 1 hour being above 14.4 times the budget rate (the SRE workbook's fast-burn example), and read the Alerting on SLOs chapter to understand the number.
4. Remove or downgrade any alert of yours that pages on a cause.

With your AI: "My service gets about 50,000 requests a day. Check my numbers for a 99.5 % monthly SLO: budget in failed requests and in minutes of full outage; and explain what a burn rate of 14.4 over one hour means in plain words. Show the calculation so I can redo it."

Let an agent do it: SLO tooling (for example Sloth or Pyrra) and agents can generate the multi-window burn-rate rules. You still recompute one threshold by hand and test the rule by injecting errors.

Recap (10 min)

Three lines: SLI / SLO / SLA, the error budget, symptoms versus causes. Then recompute the 99.9 % budget without notes.

Day 50 · 22/12 — Handling an incident, and prompting during one

Goal: run an incident with clear roles and a timeline, and use an AI during it without losing control.

Theory 1/3: roles and the first minutes

Incident practice borrows from the Incident Command System of American firefighters (1970s). Roles, even in a team of two: the incident commander coordinates and decides, does not type commands; operations investigates and acts; communications updates users and stakeholders. The first goal is to mitigate (stop the harm: roll back, fail over, switch a flag off), not to find the root cause. Keep a timestamped timeline of what was seen and done; it will feed the postmortem.

Theory 2/3: runbooks

A runbook is the written procedure for a known situation: symptom, checks, actions, how to verify, who to call. Every alert links to its runbook. Under stress nobody remembers; everybody can follow steps.

Theory 3/3: prompting during an incident

A weak prompt: "my site is down, what do I do?". What is wrong: no system, no symptom, no data, and it invites the model to suggest risky commands you will be tempted to paste under stress.

The edited prompt:

Role: you assist an on-call engineer; you propose, I decide and act.
Context: hello-ops on Kubernetes (kind), Deployment revision 7
deployed at 10:02; SLO alert fired at 10:09.
Task: rank the three most likely causes and the read-only check
that confirms or rules out each.
Constraints: read-only commands only (get, describe, logs); no
delete, no apply, no restart; say "unknown" rather than guess.
Inputs: kubectl get pods; kubectl describe of one failing Pod;
the last 30 log lines: [paste].
Output: a numbered list: cause, evidence in my inputs, check.
Check: I run each check myself and paste the result back.

Practice (30 min)

1. Write the runbook for the HelloOpsDown alert: symptom, checks (kubectl get pods, describe, logs, rollout history), actions (rollout undo), verification (/health, the SLI panel), escalation.
2. Drill: deploy the broken v3 image from day 39 while a curl loop runs. Act as commander and operator: open a timeline file, note each time and action, mitigate with the runbook.
3. Close the incident when the SLI recovers; keep the timeline for tomorrow.

With your AI: use the edited prompt above during the drill, with your real outputs pasted. Note in the timeline what it suggested, what you checked, and whether it was right.

Let an agent do it: an agent can be a scribe and an investigator in an incident: keep the timeline from the chat, run read-only checks and summarise. Give it read-only credentials. You still decide and run every action that changes production.

Recap (10 min)

Three lines: the three roles, mitigate first, what a runbook contains; and the incident prompt's constraints. Then rewrite the weak prompt properly from memory.

Day 51 · 23/12 — Blameless postmortems

Goal: write the postmortem of yesterday's drill, without blame and with action items that will be done.

Theory 1/2: why blameless

In May 2012 John Allspaw, then running operations at Etsy, published "Blameless PostMortems and a Just Culture": if people are punished for mistakes, they hide them, and the organisation stops learning. A blameless review asks what people knew, expected and did at each moment, and why it made sense to them; the person closest to the failure becomes the expert on preventing it. Google's SRE book has a chapter on postmortem culture.

Theory 2/2: the document

A postmortem holds: a summary; the impact (who, how long, how much budget); the timeline; how it was detected and mitigated; the contributing factors (rarely a single root cause; "five whys" helps but tends to follow one line); what went well and badly; action items with an owner and a date. The site's article When my AI agent opened a door in the wrong place is a real incident report written this way.

Practice (30 min)

1. Write postmortems/2026-12-22-broken-health-check.md from yesterday's timeline: summary, impact (minutes, failed requests, budget spent), timeline, detection, mitigation, contributing factors, what went well, action items.
2. Find at least one contributing factor in the system, not the person (for example: nothing in CI tested /health in the built image).
3. Turn one action item into a real change now (a CI step that runs the image and curls /health), and link the commit in the postmortem.

With your AI: "Review my postmortem as a facilitator trained in blameless reviews: [paste it]. Point out every sentence that blames a person, every action item without an owner or date, and any contributing factor I missed given the timeline. Quote the sentences."

Let an agent do it: an agent can draft the postmortem from the incident chat and the timeline. You still own the contributing factors and the action items: the agent was not there and does not know which fix the team will really do.

Recap (10 min)

Three lines: why blameless, the parts of a postmortem, action items. Then list the parts from memory.

Day 52 · 24/12 — Security in the pipeline 1: shift left

Goal: add code, dependency, secret and image scanning to the pipeline, and know what each finds.

Theory 1/2: DevSecOps and the kinds of scanners

DevSecOps puts security into the same pipeline as the rest, as early as possible ("shift left"): a flaw found in a pull request costs minutes, in production it costs an incident. Families of checks:

• SAST (static analysis of your code): Semgrep, CodeQL, Bandit for Python;
• SCA (your dependencies' known vulnerabilities): pip-audit, OSV-Scanner;
• secret scanning: Gitleaks (day 41);
• image and configuration scanning: Trivy, Grype; and misconfigurations in IaC;
• DAST (attacking the running app): ZAP.

The OWASP Top 10 lists the most common classes of web application risks.

Theory 2/2: findings, thresholds and noise

A scanner that fails on every low finding gets disabled within a week. Fail the build on high and critical findings that have a fix, report the rest, and record accepted risks with a reason and an expiry date. And remember March 2026: the scanner itself is a dependency (the Trivy action's tags were hijacked); pin it like any other action.

Practice (30 min)

1. Locally, run each tool once on hello-ops following its README: pip-audit -r requirements.txt, Bandit on app.py, Gitleaks on the history, Trivy (installed from its official documentation, checking the release is not one of the compromised versions listed in the advisory) on the image: trivy image hello-ops:dev.
2. Read each finding: real, false positive, or accepted? Note your decision.
3. Add the cheapest two (pip-audit and Gitleaks) to CI as a security job, with the threshold policy above.

With your AI: "Here is a Trivy report for my image: [paste the table]. For each high or critical finding, tell me whether a fixed version exists, whether it is in the base image or in my dependencies, and the smallest change that fixes it. Do not guess CVE details: say 'check the advisory' when unsure."

Let an agent do it: an agent can triage scanner output and open the upgrade pull requests. You still check that each upgrade's tests pass, read the changelog of any major bump, and never let it add an exception without a written reason.

Recap (10 min)

Three lines: shift left, SAST / SCA / secrets / image / DAST, the threshold policy. Then name the tool for each family without notes.

Day 53 · 25/12 — Security in the pipeline 2: the supply chain

Goal: produce an SBOM for the image, understand signing and provenance, and harden the pipeline itself.

Theory 1/2: attacks on the chain

The software supply chain is everything between a commit and what runs: dependencies, build tools, CI actions, registries. Attackers aim there because one compromise reaches thousands of users. SolarWinds (revealed December 2020): a build system compromised to ship a backdoored update. The xz utils backdoor (CVE-2024-3094, discovered in March 2024 by Andres Freund): a maintainer identity built up over years to slip malicious code into a compression library used by SSH on many distributions. The tj-actions (2025) and Trivy (2026) incidents hijacked CI actions.

Theory 2/2: SBOM, signing, provenance, SLSA

• An SBOM (software bill of materials) lists every component of an artifact, in a standard format (SPDX or CycloneDX); Syft generates one from an image.
• Signing: Sigstore's cosign signs images, also "keyless" with the CI's identity, so a cluster can refuse unsigned images.
• Provenance: a signed statement of how, where and from which commit an artifact was built. SLSA (Supply-chain Levels for Software Artifacts) defines levels of such guarantees.
• The pipeline itself: actions pinned by SHA, minimal token permissions, no secrets on pull requests from forks, protected branches with required reviews.

Practice (30 min)

1. Generate an SBOM: syft hello-ops:dev -o cyclonedx-json > sbom.json (Syft installed from its README); count the packages; find Python's version in it.
2. Scan the SBOM instead of the image (Trivy and Grype accept SBOMs) and compare with yesterday.
3. Audit your workflows against the checklist above: every action pinned, permissions minimal, a branch protection rule on main requiring the CI and one review.
4. Read Sigstore's cosign quickstart; note how keyless signing works in CI.

With your AI: "Assess my pipeline against SLSA build levels: here are my workflow files and my branch protection settings: [paste]. Tell me which level I plausibly meet, which requirement I miss for the next one, and link the slsa.dev section for each so I can check."

Let an agent do it: an agent can add SBOM generation and signing to the image job. You still verify the signature yourself with cosign verify and check the SBOM is attached to the image you deploy, not to another build.

Recap (10 min)

Three lines: what the supply chain is, three real attacks, SBOM / signing / provenance. Then retell the xz story in two sentences.

Day 54 · 26/12 — Final project 1: design and decisions

Goal: choose the final project's application and target, draw its pipeline, and record your decisions.

Theory 1/2: the brief

The final project takes a small application from commit to production through a documented pipeline. The application can be hello-ops grown a little (a second endpoint, a Redis counter) or a small app of your own: what matters is the chain, not the features. Two targets, both on your machine:

• A (recommended): GitOps on kind. CI tests, scans and pushes the image, then updates the image tag in the repository's production overlay; Argo CD in the cluster pulls and deploys. No credentials to your machine ever leave it.
• B: a VM. Ansible deploys the Compose stack on a second VM, run from a self-hosted runner on a private mirror of the repository (day 19).

Theory 2/2: architecture decision records

An ADR (architecture decision record, Michael Nygard, 2011) is a short file per significant decision: context, decision, consequences. Months later, it answers "why on earth did we do this?". Keep them in docs/adr/0001-….md.

Practice (30 min)

1. Create the final repository (for example hello-ops-prod, or grow hello-ops), with folders app/, k8s/ (or ansible/), monitoring/, docs/adr/, postmortems/.
2. Draw the pipeline (text or a diagram): push → lint → test → scan → build → push image → update tag → deploy → verify. For each step: the tool, what makes it fail, what it protects.
3. Write ADR 0001 (the target, A or B) and ADR 0002 (the tagging scheme).
4. Add a README.md with the project's purpose and the diagram; commit.

With your AI: "Here is my pipeline design and my two ADRs: [paste]. Act as a principal engineer in a design review: ask me the five hardest questions about failure modes (what happens if the registry is down, if Argo CD is down, if a bad image passes the tests…), one at a time, and wait for my answer to each."

Recap (10 min)

Three lines: the chain you will build, your target and why, what an ADR is. Then redraw the pipeline from memory.

Day 55 · 27/12 — Final project 2: continuous integration

Goal: a pipeline that lints, tests, scans and publishes an image tagged by commit, and protects main.

Theory 1/2: the order of the jobs

Cheap and likely failures first, expensive last; publish only what passed everything; one image built once and promoted, never rebuilt per environment (what you tested is what you ship).

lint ──┐
test ──┼──> image (build, scan, push :SHA) ──> deploy (day 56)
secrets┘

Theory 2/2: branch protection

The pipeline only protects main if main cannot be pushed to directly: a protected branch requires a pull request, the CI checks to pass, and (in a team) a review. Even alone, it stops you from pushing a quick fix that skips everything.

Practice (30 min)

1. Write the workflow: jobs lint (ruff, shellcheck, actionlint), test (pytest), secrets (gitleaks), then image (needs all three; build, scan with your threshold, push :SHA; on main only).
2. Add a smoke test to the image job: run the built image, curl /health, fail if not 200 (your day 51 action item).
3. Pin every action by SHA; set permissions: contents: read at the top.
4. Protect main: pull request required, checks required. Test it: a direct push is refused.

With your AI: "Here is my final ci.yml: [paste]. Check it against this list and answer as a table (item / met / line): jobs ordered cheap to expensive; image built once; publish only on main; smoke test; actions pinned by SHA; minimal permissions per job; no secret available to pull requests from forks."

Let an agent do it: you have written every piece of this pipeline by hand in module 2; an agent may now assemble it. You still review the full diff, run it on a pull request and on main, and check the published tag and digest yourself.

Recap (10 min)

Three lines: job order, build once and promote, branch protection. Then redraw the job graph from memory.

Day 56 · 28/12 — Final project 3: continuous deployment

Goal: every merge to main reaches production by itself, with a gate you control.

Theory 1/2: the deploy step for target A

After the image job, a deploy job writes the new tag into k8s/overlays/prod/kustomization.yaml (its images: … newTag: field) and commits it. Argo CD sees the commit and syncs the cluster. On GitHub, a commit pushed with the workflow's GITHUB_TOKEN does not trigger a new workflow run, which avoids an endless loop. A GitHub environment named production with a required reviewer turns this job into continuous delivery (a click) instead of deployment, if you want the gate.

Theory 2/2: verify after deploying

A deployment is finished when it is verified: the rollout completed, the health check answers, the SLI did not drop. Argo CD shows "Synced" and "Healthy"; the version endpoint shows the new SHA; the dashboard shows no error spike. If not: revert the tag commit in Git, and Argo CD rolls back.

Practice (30 min)

1. Install Argo CD in your kind cluster following its getting started page; create an Application pointing at k8s/overlays/prod of your public repository, with automated sync.
2. Add the deploy job (needs image, main only, permissions: contents: write): it replaces newTag with the commit SHA and pushes the commit.
3. Merge a visible change (the message on /): watch the CI, then Argo CD, then curl through a port-forward: the new version answers.
4. Roll back with git revert of the tag commit; check the old version returns. (Target B: the same with an Ansible play run by the self-hosted runner, and --check --diff on pull requests.)

With your AI: "Here is my deploy job and my Argo CD Application manifest: [paste]. Walk through what happens, step by step with the component responsible, from a merge to main until a Pod with the new image is ready. Then name two ways this chain can fail silently and how I would notice each."

Let an agent do it: an agent can watch a deployment end to end and report "deployed and verified" or "rolled back". You still define what "verified" means (which endpoint, which SLI threshold), and you look at the dashboard after the first few automated deployments.

Recap (10 min)

Three lines: how the tag reaches the cluster, synced versus healthy, rollback by revert. Then redraw the deploy chain from memory.

Day 57 · 29/12 — Final project 4: observability and SLO

Goal: the production app is measured, has a dashboard, an SLO and an alert with its runbook.

Theory 1/2: what "done" looks like

Production is observable when, for any alert, you can go from the page to the dashboard, from the dashboard to the logs and traces of the failing requests, and from there to the runbook. Each step is a link, not a search.

Theory 2/2: monitoring in the cluster

In Kubernetes, Prometheus discovers targets through the API (Pods with annotations, or ServiceMonitor objects with the Prometheus Operator). For this project, the simplest working setup is enough: the monitoring Compose stack from days 44–48 scraping the app through a NodePort or a port-forward, or Prometheus deployed in the cluster with a static scrape job. Simplicity you understand beats a chart you cannot debug.

Practice (30 min)

1. Point Prometheus at the production app; check the target is up.
2. Provision the golden-signals dashboard (day 46) and an SLO panel showing the remaining error budget.
3. Write the SLO and its burn-rate alert (day 49), with runbook_url and dashboard annotations.
4. Write docs/runbooks/slo-burn.md; check every link in the alert works.

With your AI: "Here is my SLO, my alert rule and my runbook: [paste]. Play the on-call engineer receiving this alert at 3 a.m. with no other context: tell me, step by step, what you would do with only these documents, and where you get stuck."

Let an agent do it: an agent can check that every alert has a runbook link that resolves and every dashboard panel has data. You still read the runbook as if half asleep: it must be followable without thinking.

Recap (10 min)

Three lines: alert → dashboard → logs and traces → runbook, the SLO, the burn-rate alert. Then write the alert's annotations from memory.

Day 58 · 30/12 — Final project 5: game day and postmortem

Goal: break production on purpose, detect, mitigate, and write the postmortem.

Theory 1/2: game days and chaos engineering

A game day is a planned exercise where a team injects a failure to test its detection, its runbooks and its reflexes. Netflix made the idea famous with Chaos Monkey (2011), which randomly terminates instances in production so that systems are built to survive it; the Principles of Chaos Engineering describe the method: state a hypothesis about steady state, inject a real-world failure, look for a difference, keep the blast radius small.

Theory 2/2: three scenarios

• Bad release: merge a change whose /health fails (does the smoke test catch it? If you bypass it, does the rollout stall and the alert fire?).
• Dead dependency: stop Redis (does the app degrade or crash? does the readiness probe react?).
• Resource exhaustion: lower the memory limit until OOMKilled (does the dashboard show it? does the alert fire?).

Practice (30 min)

1. Pick one scenario. Write the hypothesis (what the system, the alerts and you will do, within which time).
2. Run it with a curl loop measuring the impact; keep a timeline; follow your runbook.
3. Write postmortems/2026-12-30-game-day.md with the template of day 51, including where the hypothesis was wrong.
4. Fix one action item today and link it.

With your AI: "I am preparing a game day on my kind cluster. Scenario: [describe]. Challenge my hypothesis and my stop condition: list what could go differently from what I expect, and what I should measure to know. Do not propose more scenarios."

Let an agent do it: an agent can run the failure injection and the measurements on a schedule you set, and keep the timeline. You still run the response yourself: the point of the exercise is to train people, not the agent.

Recap (10 min)

Three lines: what a game day is, hypothesis and blast radius, what you learnt today. Then list the parts of your postmortem from memory.

Day 59 · 31/12 — Final project 6: documentation and portfolio

Goal: make the project readable by a stranger in five minutes, and close the path.

Theory 1/2: the README that gets you an interview

A recruiter or an engineer opens your repository for a few minutes. The top of the README answers: what it is (one sentence), the pipeline diagram, how to run it locally (copy-paste commands that work), what is tested and scanned, how it is deployed and rolled back, how it is observed (a dashboard screenshot), and links to the ADRs, the runbooks and the postmortems. Honest limits ("single node, no TLS") read better than claims.

Theory 2/2: path review

Practice (30 min)

1. Write the final README as described; add a dashboard screenshot (no internal names or addresses in it).
2. Follow your own README on a clean clone, command by command; fix what breaks.
3. Pin your four deliverables and the final project on your forge profile, with one sentence each.
4. Tick the end-of-path checklist on the path page; for each box that does not tick, write the session to redo.

With your AI: "Act as an interviewer for a junior DevOps or SRE role. Here is the README of my final project: [paste]. Ask me eight interview questions about it, one at a time, mixing 'why did you choose' and 'what happens if'. After each answer, tell me what a strong answer would have added."

Let an agent do it: from now on, let agents do the repetitive parts of this work, the way each session showed. Keep doing by hand, regularly, what you must be able to judge: reading a plan, a diff, a trace and a postmortem.

Recap (10 min)

Write three lines on what you can do now that you could not on 3 November. Then pick the next topic: Kubernetes in depth, a cloud provider, security, or SRE.