In September, someone from OpenAI contacted John Langford and asked where the idea behind one of his papers came from. Langford built one of the classic machine-learning tools of the last twenty years, runs a well-known research blog, and works at Microsoft Research. He had no idea what OpenAI was up to, and he said on X that the question itself was strange. It was about a rumor: that OpenAI’s newest flagship model uses a “full-bandwidth transformer”, the design described in a paper Langford co-authored with researchers from Johns Hopkins, Princeton, and Microsoft.

What was the call about? Langford did not say, and the honest answer is that nobody outside OpenAI knows. But there are only so many reasons a frontier lab contacts a researcher about a rumor concerning its own product. One is that the rumor is true and they are checking whether the idea leaked out — who knew what, and when. Another is that the rumor is false and they want it traced to its source. Either way, the call only happens if the idea is live somewhere inside that building. That is the signal worth reading: not the rumor itself, the interest in where it came from.

The idea is simple enough to explain with no math, and it points at something bigger than one architecture tweak.

What a model actually does when it writes

When ChatGPT writes you a paragraph, it does not compose whole sentences as you would. It produces one word at a time, and for each word it does a burst of work: the word passes through dozens of processing layers, and each layer refines what the previous one figured out. Think of it as scratch work on a pad of paper. At the top of the stack, the finished word comes out.

Then something odd happens. The word is kept. The scratch work is shredded.

To write the next word, the model starts over with almost nothing except the words already written. Everything it worked out while producing the previous word — the reasoning, the weighing, the half-formed conclusions — is thrown away. The only thing that survives from one word to the next is the word itself.

But doesn’t the model already remember everything?

It does, and this is the part most explanations skip. Every word you have ever typed into a chat gets converted, when it is processed, into two things that are filed away: a key, which says what the word is and where to find it, and a value, which carries what it actually means in context. Researchers call this filing system the KV cache, and it is why the model can refer back to something you said in paragraph two while writing paragraph twenty. Every card is still in the cabinet.

So why does anything get wasted? Because of how the cabinet can be used. A new word, working its way up the layers, can reach into the file and look up specific cards: what did that sentence say? What did the user ask earlier? That is the only access it gets — targeted retrieval of raw records. What it cannot do is inherit conclusions. The filed cards are the transcript, not the summary, and using them means re-doing the thinking every single time, from the bottom. The finished scratch work — the actual conclusions the model reached while writing the last word — sits at the top of the stack for one instant and then vanishes. The model can look up anything it ever said. It cannot carry forward anything it ever figured out.

Why the model thinks out loud

This explains something you have probably noticed. When an AI model “thinks,” it does it in text: “Let me check the base case first.” “Let me try a smaller example.” The model is not showing off. It is leaving itself notes, because spoken words are the only thing it gets to keep between steps. If the reasoning is not written down, it is shredded with everything else.

That habit has a price. Every word of thinking-out-loud costs time, and when you use these systems through an API, it costs money — the bill for an AI agent is largely the bill for a machine narrating its own work so it does not forget it.

The fix

The full-bandwidth transformer stops shredding the scratch work. At each step, the model’s finished thinking from the previous word — the whole top of the stack — is handed to the next step along with the new word, mixed together through one small valve. The model still speaks ordinary text, and nothing changes from the outside. What changes is that the conclusions from one word are still there while the model works on the next one, and they get a fresh full pass through every layer.

The part where the homework turned out to be unnecessary

There is a human story here. Langford had previously proposed a training rule called next-latent prediction: since the model forgets its scratch work, make it practice. During training, at every step, the model is forced to predict what its own internal notes will look like after reading the next word. A model drilled on that assignment builds compact, genuinely predictive notes, and Langford had a proof that they converge to the smallest summary of the past that still predicts the future.

The full-bandwidth paper grew out of that work. Then came two surprises, in his own words: you don’t need nextlat, and it worked easily. Once the pipe exists — once the previous step’s finished thinking automatically lands on the next step’s desk — the drills add nothing. You can delete them. The training rule told the model to keep good notes; the pipe makes the notes useful. Given a working channel, the model keeps its state because it pays to. No homework required.

The results, in plain terms: the researchers trained small language models this way and found they learned as much from a given amount of training material as ordinary models learned from about half again as much. On a set of math word problems, turning the new plumbing on at answer time — same model, same question — moved the score from 45.4% to 49.0%. And the models wrote shorter answers when no examples were handed to them: same accuracy, less thinking out loud, which is exactly what you would expect if the notes no longer had to be spoken to be kept.

The fine print: every result is on small models — a thousandth the size of a frontier release — and nobody has shown this works on the giant ones. One math benchmark did not improve. This is a promising research paper, not a product announcement.

Why this matters

Zoom out, because the interesting thing here is not the benchmark scores.

For a decade, the recipe for making a machine smarter was to make it talk. Chain-of-thought, reasoning steps, scratchpads — all of it rests on one assumption: that a model can only carry forward what it says. An entire industry of prompting technique, agent design, and pricing grew on top of that assumption. We built agents that narrate their work because their memory only accepted narration.

This paper is the first serious crack in that assumption. It says: maybe the narration was never the thinking. Maybe it was just the only format the pipe would carry. If machines can carry their thinking forward silently — directly, without paying the word tax — then the economics of reasoning change, and with them the economics of every product that runs an agent loop. Today the dominant cost of a capable agent is the words it burns deciding what to do. A design that halves that, or better, changes what is affordable to automate.

And there is a lesson in how the discovery happened. The field spent two years teaching models better note-taking habits. The breakthrough came from someone asking whether the note-taking was needed at all — whether the real problem was the plumbing. That pattern, fixing the system instead of training around it, is worth more than this one paper.

Nobody knows whether this becomes the next standard design or a discarded branch. But the phone call from OpenAI is a real signal, the paper is short, and the idea fits in one sentence: stop throwing away what the model already figured out.