AI as another critical reader
I have written about AI on this blog multiple times, and not long ago I got an invitation. Simon Spichak was writing an article for JMIR on whether AI can judge the quality of scientific work and asked whether he could interview me. I was one of several researchers he spoke to, and we had a very interesting conversation. My position was the cautious one: these tools can be genuinely useful in expert hands, and much less so without the expertise to judge what they produce. I greatly enjoyed the finished article and highly recommend it.
One of the specific points we talked about was that I had experienced that an AI tool had found a real gap in one of my manuscripts, triggering us to do additional experiments. I have written about this in my Paper Wizard review. What I could not mention at the time was what would happen when the very same manuscript reached three human reviewers. I did not have the answer; the reports were still out.
By now the paper is published and I have written about it here, so I can finally talk about what happened when the manuscript was reviewed for the journal it is now published in.
What the tools actually did
Before I submitted the paper to BMC Microbiology, it went through two rounds of critical reading that had nothing to do with the eventual reviewers.
The first was a genuine round of human peer review, at a different journal, which rejected it. I have written about that elsewhere, and the rejection was fair in substance: the editor wanted more mechanistic depth, which the paper lacked and which we had never intended to provide. But the reviews were not wasted. Rejection with useful comments almost never is.
The second was an AI pre-review, using a tool then called Paper Wizard. It flagged a genuine gap and suggested an additional experiment. I want to be precise about what happened next, because it is the whole point: I did not do what it said because it said it. I considered the suggestion on its scientific merits, decided it was worth doing, and did it – and the resulting data genuinely strengthened the paper. It was by no means the only suggestion it made. There were others: suggested experiments whose results would not have materially affected the conclusions I wanted to put forward. So the tool did not look through the arguments and come back with one razor-sharp comment. Instead, what it did was surface possibilities – slight discrepancies between what the experiments showed and what the draft claimed, the typical thing a tired author who has worked on a topic for years has stopped seeing. Still, one of the experiments suggested was a key experiment. We did the work, and it is included in the published version of the paper.
Editing example when the manuscript was re-worked for BMC Microbiology.
© RudolphLAB, 2026
There was also a second, stranger use, and it is the one I find, in retrospect, at least as interesting. Very early on, we had thought this work might turn into a critique of the replication run-out method itself. The evolving data forced us to abandon that stance, partially on the basis of the experiment suggested by Paper Wizard. As it turns out, the method largely holds. But an interpretive posture, once woven through a draft, is remarkably hard to remove cleanly. It does not sit in one paragraph you can delete. It is diffused through dozens of small phrasings, a word here, an implication there, and after a lot of work across various drafts my eye simply went blind to the residue – precisely because I had spent so long writing it.
So I used a general-purpose language model, quite deliberately and quite narrowly, to hunt for it. Not to rewrite anything for me, but to read the whole manuscript ruthlessly for any statement that could still be perceived as an attack on the method, and flag it for me to deal with. It was, in effect, a consistency auditor for tone. I made the changes myself, deciding in each case how to handle a flagged problem – but finding the things that needed changing, consistently, across a whole document, is a genuinely hard task that this kind of tool can be remarkably good at.
And then three humans read it
The manuscript then went to BMC Microbiology. The process took some time – a story for another day, and not one that reflects on the reviewers at all. But when the three reports finally arrived, the revisions they required were the lightest I have experienced in years.
I need to be precise here: the reviewers did not wave the paper through. Not by any stretch. They engaged with it seriously and raised real points. It is simply that the changes those points required amounted to a few relatively small modifications and one more substantially rewritten paragraph – a couple of hours' work, not the months and months of additional work that occur far too often these days. For a paper that had spent months in review, remarkably little needed doing.
The incredibly tempting conclusion writes itself: two rounds of critical reading beforehand – one human, one artificial – plus a very thorough tone audit, and the paper arrived at its final review in such a finished state that there was little left to catch. AI-assisted preparation, smooth passage. What a tidy little advertisement for the tools this would be.
Why I do not believe the tidy story
I have no doubt that the hard work which went into writing and re-writing this paper – the AI-based suggestions included – helped. But this is a single case. One paper, one outcome. There is no statistic here. I have experienced before that I worked through a long list of reviewers' comments for a manuscript, only to find that when the same paper went to different reviewers at a different journal, they raised completely different points. This was also always meant to be a smaller, more modest paper, which may itself have contributed to the lighter revisions regardless of what preceded them. And it had already survived a full round of review at another journal, so of course it was in better shape than a fresh submission – which has nothing to do with any tool.
I therefore cannot claim that AI-assisted preparation produced an easy review, and anyone who does is trying to sell you something.
And yet. Having done this many times, over many years, the paper felt different going into that final round. More complete. Not because the AI was clever, but because it had been read, hard, from several independent directions before the reviewers ever saw it – and the accumulation of that critical reading, human and artificial, seemed to leave very little uncaught.
That is not a result. It is a hypothesis, and I offer it as one worth others testing, not as something I have shown: that the value of these tools may lie less in any single gap they catch, and more in adding another independent pass of criticism to a manuscript before it meets the reviewers – and that enough independent passes, from different vantage points, may simply leave less for the final round to find. It is eminently testable, by people with more than one data point. I would be curious to see it tested.
There is a coda that cuts the other way, and I include it because a post that only flattered the tools would be dishonest. At the proofing stage, a set of small but real errors surfaced that I had not caught. By now I have developed a prompt that can be used in any general-purpose large language model. I have specifically tested it, and it would have found those small but real editorial errors. The tools helped where I used them and failed to help where I did not, which is precisely the point. They are not a safety net that operates whether or not you deploy it. They are instruments, and an instrument left in the drawer catches nothing.
Which brings me back to where I left the story with Simon, before I knew how it ended. AI can be another critical reader – one more independent pass, offered to a researcher who still has to make every judgement that matters. It cannot supply the judgement. On my evidence of one, that is as far as I can go. But I suspect the extra pass is worth more than we yet know how to measure.
Similar and related posts: