18.6 The purpose put into the machine — Conclusion

The Introduction quoted Norbert Wiener’s warning that if we use a mechanical agency we cannot efficiently interfere with once started, “then we had better be quite sure that the purpose put into the machine is the purpose which we really desire.”⁷ Eighteen chapters later I can say what a purpose put into a machine looks like when it is done properly. It is not a sentence in a system prompt. It is four artifacts.

The objective is an outcome and its exclusions, written before the system runs — Chapters 10 and 13. The envelope is what the system may touch, held in a credential and a policy rather than in prose, so it does not move when the model changes its mind — Chapters 7 and 14. The evidence chain is the record of what was read, changed, tested, and left unverified, joined on an identifier a support lead and a payments engineer both recognize — Chapters 8 and 16. The stop is who can halt the system and which conditions halt it without asking — the part Chapter 9 shows is most often left for after the first failure. Pete’s system has all four at a scale you could draw on a napkin: the change he described, the list of what may change and what must come back to a person, the report it produced, and the fact that it prepared the deployment and did not perform it.

That is also where this book parts company with Asimov, whose Three Laws the Introduction read as a hierarchy of delegated authority. Asimov put the Laws inside the robot, and the stories were about what happened when the robot’s interpretation failed. This book puts them outside — in the harness, the credential, and the record — not because the model is malicious but because a model cannot be relied on to hold a rule under pressure from its own context. Chapter 14’s injection problem is the Second Law being rewritten by whoever gets text in front of the system; Chapter 8’s variation means the same instruction does not produce the same obedience twice. So the First Law becomes a deny-by-default policy the model cannot edit. The Second becomes a scoped credential with an expiration date. And the Third, self-preservation, is the one we do not grant at all: whether a system keeps running is a person’s decision, and a system arguing for its own continuation is exhibiting a failure mode, not a right. Asimov’s robots could argue about the Laws. Ours can too. That is precisely why the Laws must not live where the arguing happens.

Wiener made the deeper point a decade earlier, and the Introduction borrowed it as this book’s thesis: “The machine’s danger to society is not from the machine itself but from what man makes of it.” What we make of it is those four artifacts, or their absence — which is also a choice, and the more common one.

Wiener’s 1960 sentence has a clause the Introduction cut, and it is the part an orchestrator should keep: be sure the purpose put into the machine is the one we really desire, “and not merely a colorful imitation of it.” The colorful imitation is the demo — the prompt that says improve the returns flow and lets the model decide what improvement means. The four artifacts are the difference, and they fall on the orchestrator and nobody else: a purpose written where the system reads it and a person can review it, in a guidance file under version control rather than a chat window that closes; a stop decided before the run, because the moment to intervene arrives after the moment to have decided; and evidence that comes back whether or not anyone asked, because a purpose you cannot check against what the machine did is a purpose you only hoped it had. Wiener’s warning in the same chapter was that throwing the problem of one’s responsibility onto the machine is how responsibility finds its way back — amplified. The record is how it stays where it was put.⁸

Which is also why delegated intelligence turned out to be the term that did work. The Introduction made the case on direction: one phrase asks whether the machine is smart enough, the other whether the delegation was deliberate and who is accountable. By the end of the book the better argument is that every institution the book met that had to decide something real asked the second question, and none asked the first. The tribunal in Moffatt did not ask whether Air Canada’s chatbot understood bereavement fares; it asked whose website it was on. The AI Act sorts the people around a system into providers and deployers and never grades the system. Chapter 17’s payment protocols carry an allowance and no capability score.⁹ The term has a limit the book should own: delegated presumes a delegator who chose, and Chapter 2’s whole subject was delegation by default. So the phrase is less a description than a demand. If intelligence is being exercised on your organization’s behalf, somebody delegated it, whether or not they know it yet, and the book’s job has been to make that somebody findable.

Norbert Wiener, “Some Moral and Technical Consequences of Automation,” Science 131 (May 6, 1960): 1355–58, https://doi.org/10.1126/science.131.3410.1355; and The Human Use of Human Beings, rev. ed. (1954), chapter X. Quotations were checked against the 1960 article and the 1989 reprint of the revised book. The Introduction shortens the first quotation; the 1950 edition’s wording was not checked.↩︎
Wiener, The Human Use of Human Beings, rev. ed. (1954), chapter X, as above: “For the man who is not aware of this, to throw the problem of his responsibility on the machine, whether it can learn or not, is to cast his responsibility to the winds, and to find it coming back seated on the whirlwind.” Checked against the 1989 reprint of the 1954 text.↩︎
Moffatt v. Air Canada, 2024 BCCRT 149, is cited in full in Chapters 4, 8, and 17; Articles 25 and 26 of the AI Act as amended by Regulation (EU) 2026/1744, and the Agentic Commerce Protocol’s allowance object, are cited in Chapter 17. No new source is claimed; the reading of what each institution asked is mine.↩︎