2.1 Layers beneath the prompt — The Instructions Are Part of the System

2.1.1 Five things people keep calling a prompt

The simplest useful distinction is this:

Instructions — what the system is told.
Context — what the system is shown.
Capabilities — what the system can call or change.
State — what persists or moves between runs.
Enforcement — what happens even if the model does not comply.

Those layers touch one another constantly, and they are not interchangeable. A skill is not a tool, a tool is not a policy, and a plugin may package both. Memory can supply context without supplying an instruction. A hook can block an action without asking the model to reason about the rule at all. A system prompt may say “do not deploy” — and what makes that sentence real is the credential: the one that gives the system access to deploy, or does not. It is like telling someone not to drive your car. You can tell them, or you can take away the keys. With a large language model, predicting whether it will follow the rule is not just difficult — it cannot be guaranteed, because these systems are probabilistic by construction, and that is especially true for the rules you lay down in the system prompt. Deciding which kind of thing belongs in which layer is the orchestrator’s job.

The distinction matters because people mix the layers, and the mixing causes failures that look like model problems. The common assumption is that if information reaches the system in any form — a system prompt, a context file, a memory entry, a pasted document — the model will adhere to whatever guidance that information contains. Often, it does not. Three things break the assumption.

First, position matters. The system prompt is assembled first and is usually the most static part of the run, but where a piece of text sits in the context changes whether the model uses it at all. The pattern has been measured: the models and retrieval tasks tested in Lost in the Middle performed best when relevant information appeared at the beginning or end, and worse when it sat in the middle, including in models built for long context.⁷ That study tested information retrieval, not the enforcement of security instructions. It establishes that putting information in the window does not ensure the model will use it well. That is why practitioners put the actual request at the end, after the reference material, and why a rule in the system prompt cannot be assumed to hold just because it was written down first.

Second, the window overflows. A direct API request can be rejected for exceeding the context window. Agent clients may instead compact, summarize, or drop content before reaching it, and instructions are not exempt. What gets kept differs by product. Claude Code compacts automatically as the session approaches the window, summarizing the conversation so far; the summary keeps recent files and decisions, the user can run the compaction manually with instructions about what to preserve, and the whole operation is visible.⁸ OpenAI’s Responses API supports server-side compaction, triggered by a configured threshold or requested separately. It returns an encrypted context item that the client carries forward; that item is opaque to the client, rather than a readable summary.⁹

Third, memory drifts. Systems distill and rewrite memory over time, and a system that remembers the wrong thing will use it. ChatGPT made this visible in April 2025, when its memory began drawing on every past conversation to synthesize a running profile of the user — Simon Willison posted the dossier it had assembled about him, including confident inferences about his interests that read as established fact.¹⁰ And memory is not only corruptible but losable: when a system summarizes or distills what it remembers, entries can simply drop out, so a rule the system followed last week can be gone this week. A curated project memory behaves differently from an automatically synthesized one, and which kind you are using determines what “the system knows” actually means.

The layers also differ by model, not just by tool. Context windows differ by model and deployment. As of October 7, 2026, Claude Fable 5.1, Opus 5.5, Sonnet 5.5, and the new Haiku 5.5 have million-token windows; the older Haiku 4.5 has two hundred thousand. Gemini models sit in the million-token class, and Grok 4.7 offers half a million.¹¹ Switch models — which orchestrators do, often to control cost — and the same design hits the wall in a different place: a prompt strategy that fits everything into context on one model turns into a retrieval-and-summarization problem on another. Even the overflow behavior is a design decision: rather than only summarizing when the window fills, the Claude API can automatically clear old tool results at a configured threshold and warn the model to save important facts to file-based memory outside the window first.¹²

2.1.2 A prompt is not a security boundary

Drew Breunig has a clean name for what happens when teams keep fixing behavior by appending more English to the system prompt: prompt debt. Iteration slows, hotfixes regress earlier instructions, and the application quietly couples itself to one model’s quirks. His point is not that prompting is useless — for one-off tasks it is often exactly the right tool — but that mature engineering disciplines eventually automate what they once did by hand: assembly language gave way to compilers, hand-tuned database queries gave way to query planners, and prompt-writing is heading the same way.¹³ I agree. And the first behavior that has to move out of the prompt is security, because security directives do not belong in interpreted text at all.

Anything sent to a language model as a prompt or as context is an instruction the model may follow, not a guarantee that it will — and that includes the most direct sentence in your system prompt:

Never read production payment data. Never modify the payment repository’s release branch. Never send customer information outside the company.

Those are good instructions. They are not security controls. A language model processes them as part of its input, which means it may follow them, may misunderstand them, may encounter later context that conflicts with them, or may receive a tool result containing text engineered to override them. It may simply be confused about which repository it is in or which environment a command will touch. Careful wording reduces risk, and it cannot remove a capability.

So if an agent is working on a payment repository, the orchestrator enforces the boundary outside the model:

Requirement	Better control
Do not read production payment data	Do not provide production credentials; use masked fixtures or a separate test account
Do not modify the release branch	Give the agent a branch-only token and block direct writes to the protected branch
Do not deploy	Do not expose deployment credentials; require a separate approval and CI gate
Do not access unrelated repositories	Scope the token to one repository and run the agent in an isolated workspace
Do not send customer information elsewhere	Block the relevant network paths and apply data-loss controls outside the prompt
Do not change payment logic without review	Require reviewer approval, security checks, and a passing test suite before merge

The prompt still matters — it tells the agent what the organization expects, explains the task, supplies useful procedures, and helps it choose a sensible next action — but the prompt is guidance while the permission structure decides what is possible.

Two documented attacks make the risk concrete. In 2024, Anthropic showed that a long context window is itself an attack surface: fill the window with hundreds of fabricated exchanges in which an assistant answers harmful questions, and the model starts imitating the pattern instead of following its instructions — the attack only became possible when context windows grew large enough to hold hundreds of shots, and it gets more reliable the more shots are added.¹⁴ In June 2025, Microsoft patched CVE-2025-32711, a zero-click vulnerability in Microsoft 365 Copilot: a single crafted email — no click, no typed prompt — caused Copilot to pull internal documents and send their contents to a server the attacker controlled. Microsoft’s own advisory called it AI command injection.¹⁵ The industry name for this class of attack is prompt injection.

That changes the orchestrator’s job from writing the sternest possible instruction to asking one diagnostic question: what happens if the system ignores this sentence? If the answer is that the agent could still read the data, change the branch, or deploy the code, then the boundary has not been implemented yet. The design rule that follows is worth keeping somewhere visible:

Use instructions to guide behavior. Use permissions and isolation to limit behavior. Use deterministic checks to reject unsafe results.