A new paper by Gary King and Steven Worthington argues that, compared to very nicely compensated human reviews, “AI-generated article reviews are now better, by every measure (they) could construct.” Their setup was fairly straightforward; they had Claude Opus 4.8 on max reasoning and a clearly AI-generated prompt asking the model to generate a comprehensive review (see Appendix C). This was enough to create reviews that were preferred by authors themselves, a team of PhD students involved in the weird internal Harvard peer review procedure (rapidPeer), and, less surprisingly, GPT 5.5!
The paper has landed like a bomb on Twitter and Bluesky, and as usual, most takes miss the nuance of the moment we live in. On the one hand, the study shows something that anyone who has tried refine.ink or spent 1 hour setting up a peer-review agent already knows: AI is capable of giving tremendously helpful feedback; it is tremendously careful in assessing causal claims, small discrepancies in tables and figures, and incorrect citations to previous literature. Most scientific work kind of sucks, and it is very easy to use AI as a low-pass filter for bad scientific work. In an era of scarce peer review, this could improve science tremendously, especially as LLMs make scientists increasingly productive.
Yet King and Worthington’s framing oversimplifies the purpose of peer review. They write: “High quality scholarly reviews are intended to help (1) authors improve their papers and (2) journal editors make publication decisions.” The paper notes that its results speak only to (1), but it overlooks that peer review has more functions than those. Most importantly, peer review asks someone to make a subjective value judgment about what a research community finds important and worth publishing, and whether the claims made in a specific paper are justified to a degree the community finds important.
The correct yardstick for peer reviews is thus not the outcomes associated with a specific paper, but with research programs. One might invoke Lakatos here and argue that science should be evaluated by whether a line of inquiry becomes “progressive,” whether it keeps generating new insights about the world rather than producing papers that satisfy our current standards of rigor. The assessment of correctness has always been deeply entangled with these broader judgments, even if only through our unconscious biases. We as scientists largely tolerate imperfect things (from cute lab experiments to LLM judges) because we believe they advance a research program that is going somewhere.
Completely delegating this almost aesthetic judgment to AI is very likely a mistake. This is not only because AI models have a quite twisted sense of what matters, but, more importantly, because humans should determine this direction in the first place. We have no mechanism that can better reveal what a directions ought to be valued by society than asking scientists to exercise judgment about it. And peer review is, therefore, a way we have of nudging the consensus on what questions are interesting and which standards of evidence are appropriate.
But so it is misguided for the scientific community to ignore the wonders of AI peer reviewing. Nobody wants to check whether supplementary Table 3 is coherent with the inset plot of the second figure. Nobody does that! AI can make scrutiny incredibly cheap, and we should use this very useful signal to crop away the unambiguously bad science before it ever consumes a human reviewer’s time. But beware! Prompt Claude enough, and it will give you reasons to criticize any paper! Even those who have arguably spanned progressive research agendas! Sure, you invented CRISPR, but could the authors please explain the discrepancy in Figure S32?