Scientific superintelligence, what would it take to enhance model reasoning?

TL;DR. The reasoning gains of the last few years came from domains where a candidate answer is cheap to check. Math has a proof, code has a test suite, a game has a score. You can let a model propose many reasoning paths and keep the ones a verifier accepts, and search plus a cheap verifier is most of what reasoning has meant. Science breaks the recipe, because the verifier is the experiment, which is slow, expensive, and sometimes impossible at the moment of reasoning. So the path to models that do science centers on building verifiers for open-ended claims, which is the same problem as making an evaluation that still means something once a model optimizes against it. The reasoning engine is ready. The “is this true” oracle is the bottleneck.

Reasoning is propose-and-verify

Strip a modern reasoning system down and it is a search with a judge. The model proposes candidate steps or whole solutions, and a verifier scores them, so training and inference can both spend compute generating many attempts and keeping the ones that check out. Where that judge is cheap and reliable, the loop is extraordinary. A unit test passes or fails, a theorem prover closes the goal or leaves it open, and the model can be pushed hard against a signal that stays reliable. I take the last few years of reasoning progress as mostly a matter of where that reliable signal was cheap to get.

verifier: on candidates to check: pruned each step

Science removes the judge

Now take the loop to a scientific question. Does this mutation change the protein’s function? Is this the mechanism behind the disease? The verifier is an experiment, and it is slow, costly, and sometimes unavailable at the time the model is reasoning. Worse, the field’s usual stand-in for the experiment, a benchmark or a learned score, is the kind of judge that stops meaning what it claims the moment a search optimizes against it. A proxy that looked fine in bulk becomes the weakest scorer at the top of a ranking. A correctly computed number supports a wrong conclusion. A verifier inside an agent loop certifies a scorer that memorized the answers. Each of these is something I measured myself, and together they are why I now read an output score as a projection of the computation. When the judge is unreliable, more search amplifies the error, because the model optimizes the flaw.

This reframes what enhancing scientific reasoning means. It is tempting to picture a bigger model thinking harder. The binding constraint is upstream of that. Absent a trustworthy verifier, a more powerful reasoner is a more powerful optimizer of whatever proxy you handed it, and it will find the proxy’s shortcut before it finds the science. The capability still needed is cheap, reliable verification of open-ended claims.

If the judge is the bottleneck, then the research that matters is the research that builds better judges, and I think it looks like three things. First, evaluations designed to survive optimization. Scores need testing with trivial baselines and top-tail utility. They also need confound controls, held-out transfer, and selection-aware nulls, so passing them takes the science and closes the shortcut. Second, interpretability, reading the computation behind a claim, beyond the claim itself, because a model that can show its work in an inspectable way is one whose reasoning you can verify internally when verification against the world is out of reach. Third, a halting criterion. An autonomous scientific agent needs to know when to stop and when a result is sound, and that is the verifier problem in a scheduling form. We measure how well agents search. We barely measure whether they know when they are done. Of the three, the halting criterion is the one I think we measure least.

What I actually expect

I expect scientific superintelligence to arrive, if it does, as the judge gets cheaper and more trustworthy, one domain at a time, and the search that already works gets pointed at a signal that stays reliable. Bulk calibration alone is insufficient for protein design. It gets there when an in-silico score matches measurement at the top of the ranking. Mechanistic claims get there when we can test them with something independent of the representation that generated them. Until then, the most important thing a reasoning system can do about a scientific claim is the least glamorous. Say how it would be wrong, and how you would check. Reasoning was the easy part. Knowing when to believe it is the hard one.