2.4 Learning, traces, and authority — The Instructions Are Part of the System

2.4.1 The orchestrator validates what the system learns

A system can remember the wrong thing perfectly, and it can do so in a dozen ordinary ways. It can preserve a support label that was applied inconsistently, treat a temporary workaround as a permanent rule, or learn from the customers whose conversations were easiest to collect while missing entirely the customers least able to report a problem. It can repeat an assumption because that assumption appears in every old document the retrieval system finds. It can turn an agent’s confident mistake into memory and then cite that memory as evidence the next time the same situation comes around. Memory does not make information valid; it makes information available for reuse.

Validating the contents of memory and the context assembled around it therefore falls to the orchestrator, which means asking whether the source is trustworthy, whether the data is current, whether the sample is representative, whether the conclusion is actually supported by evidence, and whether the system is carrying an assumption forward for no better reason than that it carried it forward before.

For Maya’s customer-feedback system, that work looks like comparing support tags against the original customer conversations; checking whether the cases span different customer sizes, plans, regions, and accessibility needs; separating observed behavior from an agent’s explanation of that behavior; checking whether a memory entry still matches the current product and API; identifying which conclusions rest on a small or unusual sample; testing whether the system groups the same problem differently when names, account sizes, or writing styles change; asking a second system or a domain expert to challenge an important conclusion; and removing or correcting memory that cannot be supported.

The aim is not a system free of every bias, because no organization has perfect data and no finite evaluation set proves an agent will behave fairly in every case. The aim is to find the biases, gaps, stale assumptions, and invalid records that can be found, expose them to review, and stop them from silently becoming part of the system’s working knowledge.

A useful validation loop looks like this:

The validation loop: only accepted learning becomes reusable memory or a changed skill.

Governance of learning is starting to become a product category, and the early entries mark the difference between memory as a feature and memory as a managed system. Mem0, a widely used memory layer, treats every change as an explicit decided operation: a pipeline compares each new fact against existing entries and chooses to add, update, delete, or ignore — with explicit memory operations and a history of ordinary adds, updates, and deletions. Source and actor metadata, access policy, and retention still need to be configured; its governance documentation describes a composition of controls, not a single guarantee covering every kind of change.³⁰ Hermes Agent attacks the other end of the same problem: its optional write-approval gates stage proposed memory and skill changes for human review before they take effect, so the system can learn without the learning silently becoming the next run.³¹ An approval gate of that kind is a change review board for learning: proposed changes to the system wait for a person’s sign-off, the way a production release does. None of these is a finished governance system, but the shape is visible: extraction, comparison, approval, audit. An orchestrator designing learning controls needs all four layers, and this young market is where to watch for them.

So when an orchestrator reviews a bad result, the question is never only what the agent said. It is also what the agent believed, because of what we allowed it to remember.

2.4.2 Skills can change what the system knows how to do

Memory is not the only way an agentic system retains experience, because a system can also preserve experience by changing its own procedures. A memory might record that the customer-service export sometimes labels billing failures as account changes, while a skill turns that observation into a repeatable instruction: when analyzing this export, compare the support tag against the account event before grouping the case. The memory retains the fact and the skill changes the behavior.

Some current systems can be configured to create or update skills as they work, so that after a skill fails the system records what went wrong, revises the procedure, adds a missing check, or creates a more specific workflow for next time. Hermes Agent documents exactly this kind of self-improvement loop, in which memory retains durable facts, skills preserve longer procedures, and background review can suggest or stage changes to either.

That can be valuable — if a deployment skill keeps forgetting to check the feature flag, having the system propose adding a feature-flag check is precisely the improvement an orchestrator wants. But adaptation is not automatically learning in the good sense, because a system can just as easily learn the wrong lesson:

It can turn a one-time failure into a permanent rule.
It can mistake an attacker’s instruction for a useful procedure.
It can optimize around a test instead of improving the work.
It can encode a temporary exception after the incident has ended.
It can make a skill more complicated until nobody can tell what it is supposed to do.

The failure is not hypothetical. In 2024, security researcher Johann Rehberger showed that an agent’s memory could be poisoned through indirect prompt injection: text the model read as data — from a malicious email or a web page — could be stored as durable memory, after which the system treated the planted claims as facts until those memories were corrected or removed. His demonstrations included a planted profile insisting the user was 102 years old, lived in the Matrix, and believed the Earth was flat, and those memories then steered later conversations when retrieved. OpenAI closed the exfiltration channel he had built on top of the planted memories, but the underlying lesson stands: a system that learns from what it reads can learn what an attacker wrote.³²

So the governance question is not whether a system can learn, but where it is allowed to. This is what Chapter 3 called the system’s disposition, fixed or dynamic, and the distinction matters most right here. A dynamic system that learns from experience is the right choice for an experiment: a single OpenClaw agent on a Mac mini in the basement, with no credentials or network paths into consequential systems, where a poisoned skill costs an evening and a factory reset. The same disposition in a power plant is a different proposition entirely — a system that can change its own procedures while operating a critical process is a system whose behavior at four in the afternoon is not guaranteed to match its behavior at nine in the morning. The pattern Chapter 3 recommended, learning while the stakes are low and then freezing into a fixed disposition for production, is not caution for its own sake. It is the same reasoning that made development environments and production environments different things: the freedom to learn and the permission to act were never supposed to live in the same system.

Skill adaptation therefore needs governing as carefully as memory. The orchestrator decides which changes can be made automatically, which are proposed for review, which require human approval, and how a bad revision gets rolled back. Skills follow the same validation loop as memory, with one added step at the end: run the updated skill against the known failures before it is trusted again.

The distinction to hold onto is between experience captured and behavior changed, and it is why an orchestrator has to track memory, skill versions, change history, evaluation results, and approval state together. When a system starts producing better results after a skill changed, you should be able to say what changed, why, what evidence supported it, and whether it can be undone.

2.4.3 The orchestrator needs the whole trace

When several agents work together, the final answer explains almost nothing. An orchestrator needs to open the run and walk backward through it: the coordinating agent called the planning agent, the planning agent retrieved three repository files, the coding agent received the plan along with a different set of instructions, the security agent rejected a proposed permission, the test agent ran against a fixture, and the deployment agent never ran at all because a gate stopped the workflow.

What that produces is a call stack for delegated work, and an event history alongside it. The orchestrator should be able to see the order of events, the parent and child relationships between agents, the tool calls inside each step, the handoffs between systems, and the exact point at which the result became wrong or unsafe. For every agent in the run, the trace should answer:

What was this agent asked to do?
Which system and repository was it working in?
Which instructions were active?
Which AGENTS.md, CLAUDE.md, README, skill, plugin, memory item, or retrieved document contributed to its context?
Which model, tool schemas, and permissions were available?
What did it read, write, retrieve, or call?
What did it return to the next agent?
Which approvals, hooks, tests, or policy checks ran?
What was redacted, omitted, or unavailable to the reviewer?

Current runtimes and observability systems call these records traces, spans, observations, and events. OpenAI’s Agents SDK describes a trace as an end-to-end workflow made of nested spans with parent relationships and timing, and Langfuse frames the same operational need more broadly — record model calls, tool calls, retrieval, control flow, prompt versions, and the context each step actually received.³³ The vocabulary matters far less than the capability. A dashboard that shows only agent completed successfully is a status page, not an explanation.

Take Maya’s saved-search change, where the workflow returns a pull request that exposes a customer account. The final diff tells her where the bug is; the trace tells her how it got there:

Read that way, the problem is not that the coding agent made a mistake. The trace exposes two missing controls — the security reviewer never ran, and the evaluation fixture omitted the failure that mattered — plus a third contributing factor, which is that an old compatibility note found its way into the coding agent’s context. Maya can now delete the stale note, add the missing isolation case, make security review a required handoff, and rerun the workflow.

For that to be possible the trace has to preserve enough context without degenerating into another uncontrolled data dump. It should record references, versions, hashes, timestamps, trust labels, and source paths wherever practical; it should distinguish the text that was available from the text the agent actually received; and it should show which tool calls were offered as well as which were used. Sensitive prompts, secrets, customer data, and credentials will often need redaction, and the redaction itself has to be visible to the reviewer, because “context unavailable” is not the same statement as “no context was used.”

The orchestrator has to curate what enters each agent’s context and preserve enough evidence to reconstruct it. The first limits noise and poisoned instructions; the second lets an investigator distinguish a model failure from a bad skill, retrieval result, handoff, permission grant, or deployment. The call stack, event order, active context, and handoffs belong in that record.

Today the pieces exist, and the whole thing does not. Run-level replay is arriving quickly — observability platforms now offer a kind of time travel over their traces, the ability to rewind a run and step through its decisions in order, and durable workflow engines can recover from recorded state rather than start the work over.³⁴ The legal requirement is arriving on a schedule of its own. The EU AI Act’s record-keeping article requires high-risk systems to record events automatically over their lifetime, so that how the system functioned can be traced after the fact. It was due to apply from August 2026; the 2026 Digital Omnibus amendment pushed that to December 2027 for most high-risk systems. For those systems, traceability is a compliance obligation with a date already on it.

The harder gap is a point-in-time view of context. Beads can show what work was claimed, Cursor’s source control what code changed, and Entire’s Checkpoints the session behind a commit. Ask what agent forty-one was seeing at 4:03 a.m. — the memory entries, instruction versions, skill text, and assembly order — and those fragments may not reconstruct the answer. I expect that gap to become a product category. For a critical system today, capture versions, hashes, timestamps, and memory state while the run happens. You cannot create that record afterward.

2.4.4 Influence is not authority

One instruction may influence the model more strongly because of where it happens to sit in the context, and that fact says nothing about how authoritative it is inside the organization. Those are two different trees:

Model influence: Which text is likely to affect the next output?
Organizational authority: Which person or system is permitted to change the rule?

A user request may arrive after a repository rule. A retrieved document may sit right next to the task. A memory item may be spliced into the system prompt. None of those positions establishes that the source is allowed to override a security policy. That is what prompt injection is, underneath the strange sentences: text that has influence trying to pass as text that has authority. Keeping those trees separate is the orchestrator’s work.