One correct answer is encouraging. Ten judgments of the same problem reveal something else: how consistently the model meets the whole task.

The Basic Level 1 tests ask models to review rule changes and evidence under explicit requirements. I evaluated six hosted AI configurations on the same 48 rule-change problems and 96 evidence packets, with ten judgments planned for every input.

The reported amended dataset contains 8,640 responses. Every judgment started fresh, without earlier answers or checker feedback.

A complete answer has to hold together

Reaching the right final decision was insufficient. The complete answer also had to identify the relevant facts and consequences, supply valid supporting objects and, on the evidence track, choose an appropriate next check.

For example, a model might correctly deny a release while giving an invalid account of the evidence. That can be useful partial performance, but it is not a fully correct review.

Reported scores ignore a sole surrounding Markdown code fence: a harmless presentation wrapper around a machine-readable JSON answer. The object still has to satisfy the full task requirements.

The two tracks tell different stories

Performance on rule changes did not reliably predict performance on evidence review. Terra and Qwen handled the rule-change track much more successfully than the evidence track. Sonnet showed the opposite pattern.

The tracks differ in several ways, so this does not isolate a single cause. It does show why the two obligations deserve separate measurements.

An average can hide an unreliable run

Luna’s fence-normalized rule-change scores ranged from 56.25% to 85.42% across the ten runs. An average alone would conceal that spread. The paper therefore reports every run, variation across runs and how often each individual input succeeded out of ten judgments.

Sol’s near-perfect results set a boundary in the other direction: these particular tests are not a difficult frontier challenge for every configuration. A perfect empirical score on a finite set is also not a guarantee of perfect reliability elsewhere.

What the results establish

These are measurements of particular hosted configurations and settings. All requested medium reasoning and the same combined reasoning/output-token ceiling, but those labels do not establish equal realized computation across providers.

The inputs were already inspected development tests, with related cases built from twelve underlying groups or units in each track. Repeating them ten times does not create ten independent sets of new problems.

The results describe behavior on that fixed material, including the later follow-up timing. They do not establish performance on unseen tasks, authenticate real-world sources or prove that an autonomous agent can safely control a real system.

The useful outcome is a more detailed picture of success and failure: whether a model follows the format, identifies the consequences, uses evidence correctly and remains consistent when asked again.

The full paper includes all ten round scores, case-consistency results, separate uncertainty analyses and the preserved original incomplete experiment alongside the amended view. The public supplement supplies the tasks, response projections, scoring code and offline replay materials.