Peer Review After the Gatekeepers

How AI is changing who can test ideas and what universities are for

I think an old bargain in the world of knowledge is coming apart. For much of modern history, serious intellectual scrutiny has been difficult to obtain. You needed access to people who understood your subject, had time to examine your work and were willing to take you seriously. Universities and scholarly journals organised much of that access. Their position gave them considerable influence over which ideas travelled and whose voices carried authority.

AI is changing the economics of that bargain. A person outside those institutions can now obtain sustained criticism of an argument, help examining evidence and explanations pitched at their own level of understanding. The quality varies, sometimes sharply. But the access itself is an extraordinary change.

My argument is that a model built around restricted access to intellectual scrutiny is becoming untenable. Peer review will evolve because many more people can now participate in the work it exists to do.

That claim needs some care. Universities have never owned intelligence, and publishing a paper has never made it true. Equally, a chatbot agreeing with you does not make you right. If we are going to challenge the gatekeepers of knowledge, we should bring at least as much discipline to the alternative.

How peer review became authoritative

Universities have concentrated expertise, libraries, equipment and communities of enquiry. That concentration has been enormously productive. Knowledge advances when someone who understands your work closely enough can identify the assumption you missed, the experiment you need or the conclusion your evidence cannot carry.

But the history of peer review is less tidy than the familiar story suggests. Scholarly societies, editors, commercial publishers and researchers beyond universities have all shaped it. The modern system did not arrive fully formed with the first scientific journals.

Henry Oldenburg launched Philosophical Transactions in 1665. One revealing episode followed in 1669: work submitted by George Sinclair was rejected after Robert Moray judged that much of it had already been reported by Robert Boyle. The Royal Society's archive records the incident. The complaint that a paper offers insufficient novelty has a surprisingly long history.

Yet historians Noah Moxham and Aileen Fyfe caution against projecting today's system backwards. Written refereeing developed through later institutional practices, and “peer review” itself became an established term only in the late twentieth century. What feels like an ancient, fixed arrangement has repeatedly changed.

Then there is Einstein. In 1936, he and Nathan Rosen submitted a paper arguing against the existence of gravitational waves. When Physical Review sent him an anonymous referee's critical report, Einstein objected and withdrew the paper. The referee was H. P. Robertson. The work eventually appeared elsewhere with its conclusion changed and an acknowledgement of Robertson's help in resolving the error. Daniel Kennefick's historical investigation reconstructs the episode.

Even Einstein benefited from someone prepared to challenge him. That is an excellent reason to defend rigorous scrutiny. It is also a reminder that the arrangements through which scrutiny happens can change without abandoning its purpose.

Publication and validation are already separating

We should distinguish making an idea available, examining it critically and establishing that it deserves confidence. A journal can help organise all three. They need not happen in that order or in the same place.

On 24 October 2026, physicist and Moonshots co-host Alex Wissner-Gross introduced Chain Theory through his own newsletter. He linked a preprint proposing a candidate for the fine-structure constant, which governs the strength of electromagnetism. He reports agreement with the measured value to ten significant digits.

The paper, dated 23 October, acknowledges what he calls a human–superintelligence collaboration. It distinguishes a mathematical calculation checked using the proof assistant Lean from the conjecture connecting that calculation to physical reality. A checked calculation does not, by itself, validate the physical theory. This is a research proposal, not an established breakthrough.

The publishing choice nevertheless matters. Readers can encounter the argument and its supporting material directly. Its scientific standing must now depend on what survives independent examination. The venue supplies neither proof nor disproof.

There are important precedents. Grigory Perelman posted his work on the Poincaré conjecture as preprints in 2002 and 2003. Mathematicians then undertook the demanding work of checking and explaining the proofs; the Clay Mathematics Institute describes the resulting detailed exposition. Public release opened the process. Expert scrutiny remained essential to establishing confidence.

Reform is also happening inside scholarly publishing. Under eLife's publishing model, papers selected for review appear as Reviewed Preprints, accompanied by public reviews and an assessment of their significance and evidential strength. There is no subsequent accept-or-reject decision after peer review. Editors still select which submissions receive review, so this is no claim that all gatekeeping has disappeared.

These examples make the direction clearer. AI enters a movement that already exists: wider access to research, more visible criticism and a clearer distinction between publication and validation.

The problem with waiting

The delays are real, although the claim that AI papers routinely arrive eighteen months out of date is too sweeping.

A 2017 study by Janine Huisman and Jeroen Smits analysed 3,500 author-reported review experiences. The first review round averaged 13 weeks, with substantial differences between disciplines. Those were self-reported experiences from an older sample, not a measurement of every journal today. They nevertheless show how a relatively small amount of specialist attention can be embedded in months of waiting.

A recent example is particularly apt. A study of AI feedback in peer review was received on 3 April 2025 and published on 23 February 2026. That is nearly eleven months from submission to publication, including revision and production as well as review. During that interval, Google's model-card record lists updates for Gemini 2.5 Pro, Gemini 3 Pro and Gemini 3.1 Pro alone.

Publication delay does not automatically make a finding obsolete. A sound insight about learning or research design can outlast the model used to investigate it. But claims about a particular model's capabilities need a date, a version and renewed testing.

AI research already uses preprints and conference proceedings to circulate work quickly. The remaining opportunity is to make scrutiny more timely and cumulative: release the work, attach substantive criticism, show the revisions and update the assessment as evidence changes. Speed has value when it shortens the route to correction.

I have used a related approach to keep our school digital strategy open to revision. The value lies partly in building a process that can respond when its evidence changes.

What AI can already contribute

There is evidence for AI's usefulness here beyond personal enthusiasm.

In a study subsequently published in NEJM AI, Weixin Liang and colleagues compared GPT-4 feedback with human reviews and asked 308 researchers about feedback on their own papers. Some 57.4% found the AI feedback helpful or very helpful; 82.4% considered it more beneficial than feedback from at least some human reviewers. That last qualification matters. The finding does not say that AI outperformed reviewers generally, and perceived helpfulness is not a direct measure of scientific correctness.

Stronger evidence for a specific application comes from the 2026 Nature Machine Intelligence study mentioned above. In a randomised trial involving more than 20,000 reviews at ICLR 2025, a system using several language models gave reviewers feedback on their draft reviews. Of those receiving feedback, 27% updated their reviews. A blinded evaluation found the revised reviews that received feedback more informative. Here, AI improved part of the human review process.

The capabilities are also becoming harder to dismiss. In July 2025, an advanced Gemini Deep Think system achieved a gold-medal score at the International Mathematical Olympiad, with solutions officially graded by IMO coordinators. This demonstrates exceptional performance on difficult mathematical problems. It cannot establish that AI is “more intelligent than the average lecturer”. Lecturers do many different things, and there is no single test that ranks the whole of that work.

We do not need that contest to justify change. AI already offers useful forms of explanation and criticism at a scale that individual experts cannot match. That is enough to alter who can get started, how often they can revise and how prepared they are when expert attention becomes available.

The university response deserves a fair account

In conversations with people in universities, I have encountered hostility towards AI and a reluctance to engage with what it can already do. Those encounters have made me wonder how much resistance concerns the technology and how much concerns a changing professional identity. They are personal observations, not a representative survey or evidence of anyone's motives.

There is evidence of an institutional gap. The HEPI and Kortext Student Generative AI Survey 2026, based on 1,054 full-time UK undergraduates, found that 95% reported using AI in some way. Only 48% felt their teaching staff were helping them develop AI skills for their future careers. That is a serious educational challenge.

There is also evidence against a picture of universal academic resistance. In Elsevier's 2025 survey of more than 3,200 academic and corporate researchers across 113 countries, 58% reported using AI in their work, up from 37% in 2024. Only 27% thought they had adequate training. Enthusiasm, uncertainty and inadequate support can coexist.

It would be unsurprising if some people felt threatened. A career built through years of difficult study deserves respect. When an unfamiliar system performs parts of that work in seconds, professional identity can feel unsettled. But concerns about fabricated evidence, weakened learning and unreliable outputs also deserve an answer. Dismissing them as status anxiety would make this argument less credible.

My challenge to universities is therefore practical. Students need guided experience of using AI well. Institutions should make that possible, while teaching them how to recognise its failures. A policy focused mainly on prohibition or misconduct cannot fulfil that educational responsibility.

Augmented Intelligence Review

I would call the emerging practice Augmented Intelligence Review: AIR. By that I mean a process combining AI criticism, inspectable evidence and human accountability. It is a proposed way of working, not a certificate that something is true.

It builds on the approach I described in In Praise of LLM Ping Pong: using different models to interrogate and improve an argument. For serious review, I would give that process five disciplines.

1. Make the claim testable. State precisely what you believe, what evidence supports it and what would change your mind. Separate the observations from the interpretation. Give reviewers the sources and methods, not only a polished conclusion.

2. Obtain separate critiques first. Ask two or more models to examine the same material without showing them one another's answers. Ask for missing evidence, alternative explanations and errors in reasoning. Then let them challenge the specific objections. This can reduce imitation between responses, although it cannot guarantee independent judgement.

3. Check against something outside the conversation. Open the cited sources. Recalculate the result or run the code where possible. Use formal verification for claims suited to it. For an empirical claim, examine the observations and the possibility of replication. A model's account of having checked something is insufficient.

4. Explain the disagreement. Ask for a clear account of where the reviewers differ, which assumptions drive that difference and what evidence could resolve it. Have the model explain unfamiliar concepts and then question you about them. You should finish better able to judge the argument yourself.

5. Keep an accountable record. Record model versions, material objections, checks performed, revisions and unresolved uncertainties. Seek independent specialists when the complexity or consequences warrant it. For public research, make appropriate parts of that record available. Use only tools authorised for confidential manuscripts or sensitive data.

The main trap is false consensus. An ICML 2025 study examining more than 350 models found substantial correlation in their errors, including across different providers. Three models can therefore agree for the wrong reason. More fluent objections are not necessarily better objections either; an AI can invent a flaw as readily as overlook one.

AIR should make the grounds for confidence more visible. It should never turn a majority vote between chatbots into a substitute for evidence. That is why the habits discussed in The Question Behind the Question matter: examining assumptions and asking questions that could genuinely change our minds.

More people can now join the conversation

The printing press expanded the circulation of ideas. The internet reduced the cost of finding and distributing them. AI adds the possibility of sustained intellectual assistance: a reader can ask for a difficult argument to be unpacked, its assumptions challenged and its implications explored.

That is the sense in which I see an intelligence revolution. Someone without access to a specialist seminar can begin an informed examination of an idea. A teacher can investigate a claim without waiting for a course. A student can ask the same question five ways without worrying about holding everyone else up. These are possibilities whose value depends on access, model quality and how people use the tools.

There is promising evidence for the teaching potential. A 2025 randomised study involving 194 Harvard physics students found that a carefully designed AI tutor produced greater immediate learning gains in less time than the comparison active-learning classes. It covered two lessons and used a structured tutor with prepared instructional material. We should take the promise seriously without assuming that an ordinary chatbot can replace a teacher or reproduce those results everywhere.

I see a related shift in professional feedback. Through Starlight, which I co-founded, we are trying to make private, regular feedback on lesson recordings available to teachers on their own terms. As I argued in Coaching, not compliance, access to reflection should be less dependent on whether someone else has time to observe you. Audio cannot reveal everything happening in a classroom, and useful coaching still demands judgement. The opportunity is to give teachers more evidence and more frequent support for thinking about their practice.

The benefit in both settings is greater agency. People can seek criticism when they need it and arrive at human conversations with more considered questions.

What universities can become

If analysis and explanation become easier to obtain, universities will need to demonstrate their value through what people can accomplish together.

One part of that is the campus as a community of enquiry. There is value in sustained attention from someone who knows you, in learning to disagree without retreating into separate feeds, and in working alongside people whose disciplines unsettle your assumptions. Physical proximity alone does not produce those experiences. Universities have to organise for them through seminars, studios, laboratories and shared projects.

Other responsibilities remain substantial. Universities maintain experimental facilities and archives. They train researchers to make sound judgements, support long investigations and create arrangements for ethical oversight and responsibility when things go wrong. AI can contribute to those activities; it does not automatically supply the institutions that sustain them.

The educational task should include showing what students understand independently and what they can accomplish with tools. The University of Sydney's assessment policy sets out a useful example: secure assessments to establish students' capabilities alongside open assessments designed for responsible use of available tools. That offers a concrete response to AI's presence in students' working lives.

There is a further public role. Broader access to AI could shift dependence from universities to a handful of technology companies. Access is not equal, sophisticated systems cost money, and their operation is often opaque. Universities can provide shared access, independently evaluate tools and preserve the freedom to challenge their suppliers. Democratising scrutiny will require deliberate choices.

This connects with the argument in Eating Soup with a Fork. The ability to produce an analysis does not settle whether it is useful in a particular place, for particular people. Judgement still has to be developed through experience and discussion. As competent analysis becomes more widely available, those settings for developing judgement may become more valuable.

I want universities to lead that transition. They can teach people how to question powerful systems, make research easier to inspect and welcome contributions from beyond their own walls.

Peer review's greatest contribution has been to expose ideas to informed challenge. AI gives us an opportunity to extend that benefit to many more people, much more often. Whether the next serious objection comes from a professor, an independent researcher or an AI-assisted teacher, the obligation is the same: examine the evidence and answer the objection.

That is a future universities can help build. Their authority will be stronger when it rests on the quality of that contribution.

Adam Sturdee is an Assistant Headteacher, AI adviser and co-founder of Starlight. He works with schools and trusts on responsible AI strategy, staff development and coaching. Explore his consultancy work.