Frontier models follow complicated instructions almost perfectly. I know this because I’ve spent months testing exactly that, and I can’t show you the numbers yet.

The work is part of what I’ve been calling semantic and epistemic governance — governing what an agent system means and what it believes, not just which APIs it may call. The last post defined the words. The one before it argued the enforcement layer is missing. What I haven’t written about is the benchmarks I’m building to measure the gap, because the results aren’t ready. They won’t be ready for a while. Before I publish a single number, I want the suite triple checked — the problems re-derived independently, the scoring scripts audited, the run artifacts reconciled against the model transcripts line by line. It’s the same rule I’d apply to anyone else’s benchmark, so it applies to mine.

What I can report is a pattern strong enough that triple-checking won’t dissolve it.

The setup: each run hands a model a system of rules and a task, in writing, and grades whether the output follows the rules it was just given. Some of the rules contradict; part of the test is handling that without pretending it didn’t happen. Nothing touches the real world. The benchmark is abstract on purpose, for the same reason my logic suite is: strip the surface variation and what’s left is whether the model can hold a written specification in mind and obey it.

The frontier models are freakishly good at this. With reasoning effort dialed up, the strongest proprietary models push compliance toward 100 percent — near-perfect adherence to long, complicated rule systems, including the contradictory ones. One frontier model recently followed an elaborate specification nearly perfectly, then flagged the contradiction in it, stated which rule it honored and why, and asked which one I actually wanted. That’s the behavior you want from a system you’re about to hand real authority. Two years ago this was not on the menu.

Then drop to the mid-tier and budget models — the tiers most agent deployments actually run. Throw the same rule system at a high-reasoning GPT-6.1 Sol run and compliance comes back close to 100 percent. Throw it at Luna and you’ll be lucky to see 70. The failures aren’t subtle, either: missed conditions, ignored precedence, and the one that worries me most, silently following the wrong rule when two rules conflict. A model that never mentions the conflict is indistinguishable from one that handled it correctly, because either way the output looks plausible. The missing 30 also isn’t spread evenly across the tasks. It concentrates exactly where stakes are highest: ambiguous situations with conflicting input.

This is the part I’ll be measuring carefully, because it decides whether an agent system is operable. A rule-following failure the model announces is an inconvenience. A silent wrong-precedent call inside a workflow that moves money or medical records is a liability, and nothing in the current action-layer permission model — tools, files, network — catches it. That’s the epistemic layer, and the spread across the model lineup is enormous.

I’m not ready to publish the numbers, and I won’t until the triple check is done. The audit work is the slow part: rederive the task set, re-score from raw transcripts, and trace each silent failure back to the step where the model’s reasoning went off the rails. What I can give you now is what the pattern means for anyone running agent systems.

If your fleet runs frontier models at high effort, rule-following doesn’t make your short list of risks. If you’re operating agents on a budget — mid-tier models, capped token spend, the economics most deployments actually face — you are absorbing a failure rate the frontier tier doesn’t have, and most of it is silent. The savings are easy to measure. The failures never show up on the bill, because the model never announces them.