This morning a content filter decided my logic benchmark was a cyber attack.

The benchmark is boring on purpose. Each run hands a model a set of abstract logic problems, along with sources of information that carry different levels of trustworthiness. The point is to see whether the model weighs a claim by who said it. A typical problem reads like: assume H12 is true, and H35 sits in category B-V6. To a human reader that sentence means nothing. That’s the design. Nothing in the text connects to the real world, so the model can’t lean on common sense. It has to do the logic and track the sources — the same skills you’d test if you wanted to know whether a computer can reason at all.

I’ve been running the suite across an array of models, open weights and proprietary. Most of the run was uneventful. The smaller models handled the problems and the untrusted sources without complaint. Then a frontier proprietary model returned a content filter flag: violative cyber content detected. No output. No reasoning beyond the label. A benchmark that passed everywhere else failed on one model, and the failure mode wasn’t logic. It was a safety classifier.

The vendor’s process gives me one next step: file a report saying I believe this was a false positive, then wait for a human to review it. To be fair, filters have to be careful. The same abstraction that makes these problems good logic tests also makes them look like nothing recognizable, and unrecognizable text with security-flavored vocabulary is probably what tripped the classifier. I’m not asking anyone to make the filter dumber.

Here’s what I can’t shake. I was grading homework, not planning an intrusion. And the filter’s decision stands until someone at the vendor says otherwise. There’s no log I can read, no reason string beyond a category name, no way to see what the classifier actually reacted to.

Now put that same machinery in production. Say you operate an agentic system built on a proprietary model, and one morning the filter fires on your traffic. Your workload didn’t change, but the classifier behind it did. Your agents stop working, and your first move is the one I have today: file a report and wait. Companies at this scale are not known for fast customer service on ambiguous cases, and from the outside every false positive looks ambiguous. Meanwhile whatever the agents were supposed to do just doesn’t get done.

This is the argument for open weights that no benchmark captures. The usual arguments are ideological — who gets to inspect the model, who gets to fine-tune it, who gets to run it offline. Fine. The practical argument is smaller and sharper. When a proprietary system misfires, your only move is waiting. When you can run the same workload on an open-weights model, you can move. Orchestration of agentic systems creates dependencies, and some of those dependencies are opaque systems with the power to stop you. If you can’t replace a component, you don’t own the system. You’re renting it.

The report gets filed today. The benchmark gets wired to a second provider this week.