The Orchestrator

6.4.1 You understand a system through failure

Resident training has to include failure on purpose, because failure is one of the most important ways anyone comes to understand a system. The engineering profession has always known this. Henry Petroski, who spent a career studying engineering failure, put the claim plainly: understanding the concept of failure was the key to understanding engineering. His books make the case through example after example — bridges, walkways, and, in the later volume, the Challenger and Columbia shuttle disasters — that engineering advances by learning from failures rather than from successes.⁷ Richard Feynman, serving on the Rogers Commission that investigated the Challenger accident, closed his appendix to the report with the sentence engineers still reach for: “For a successful technology, reality must take precedence over public relations, for nature cannot be fooled.”⁸ Failure, taken seriously, is how the profession corrects itself.

The author, on learning by breaking a very large website. One of the ways I learned to operate a very large website at scale was by breaking it. Not on purpose, at least at first — but over the years the outages I caused, the releases I got wrong, and the traffic I had failed to predict taught me more about how the system worked than any quiet stretch when everything went fine. The system I understood best was the one I had watched fail. That is ordinary among engineers who have operated something large, and it is the reason this chapter insists that residency rehearse failure rather than avoid it: the lesson does not transfer any other way.

As orchestration automates more of the routine work, catastrophic failure may become less and less available to the people who need it most — the failures that turn an operator into a seasoned engineer. A resident whose systems catch every mistake before it ships, whose fallbacks fire before anything degrades, may never acquire the scar tissue at all. That is a real cost of automation, and it is why the residency approach this chapter describes requires that professionals understand and experience failure in a controlled setting — planned outages, injected faults, staged incidents, the deliberate rehearsal of the worst day.

6.4.2 The scenario documents residents own

The concrete instrument for practicing failure is the scenario document: a written account of one way the system fails, and what happens next. The AI layer becomes unavailable. A provider comes under attack. A dependency goes dark. Data is corrupted. The deployment pipeline itself turns out to be the thing that broke. The resident — not the senior orchestrator, and not an agent — owns the document: models the risk, traces how far the damage spreads, and writes down who does what while the system is degraded.

You cannot write a credible account of how something fails without understanding how it works. The exercise forces the questions routine operation never raises: what breaks first, what the system was quietly assuming, how far the damage reaches, and what has to be true for recovery to work at all.

And the scenario does not stay on paper. It is the script for the drill: the planned outage, the injected fault, the worst-day rehearsal from the previous section. Written in training, the scenario documents become the organization’s contingency library — the same papers, kept current, that an experienced orchestrator reaches for when the real outage arrives.