Production Engineering
What it looks like to actually operate a system — reading a symptom, running an investigation with real evidence, and knowing which failure mode produces which signature.
- The incident method — why fixing is step sevenDetect, isolate, mitigate, investigate, fix, prevent — the order production incidents actually get resolved in, and why jumping straight to a fix is the single most common mistake under pressure.
- Reading the symptoms — CPU, latency, and what each combination rules outLow CPU and high latency means waiting, not working — measured from a real service genuinely faulted six different ways, not asserted. The single fact this whole section is built around.
- Linux production debugging — top, free, ss, dmesg, and what they actually meanReal captured Linux command output from a genuinely loaded container — what every column in top means, why free's "used" number lies by omission, and the kernel's own record of an OOM kill in dmesg.
- Kubernetes production debugging — pod states, real and brokenImagePullBackOff, CrashLoopBackOff, Pending, OOMKilled — captured from four genuinely broken pods on a real kind cluster, with the exact kubectl describe events that explain each one.
- Distributed tracing — spans, context propagation, and where the time actually wentA trace is a tree of spans reconstructing one request across every service it touched. Drag one span's duration and watch why optimising the wrong one is arithmetically irrelevant.
- Cascading failures — how one slow dependency becomes a total outageA downstream 429 turned into a 5x amplification, captured for real — bulkheads, backpressure, and load shedding as the three mechanisms that stop one failure from becoming every failure.
- Capacity estimation — the back-of-the-envelope math that catches a bad plan earlyQPS, storage, bandwidth, and Little's Law worked as real arithmetic, not asserted — the estimate that tells you a plan won't work before you've built anything.
- AI incident catalogue — the failure modes unique to LLM systemsPrompt growth silently turning into a latency and cost regression, a retry loop between two agents that never terminates, a retrieval pipeline that went quiet — the incident shapes that don't show up in a normal backend's playbook.