What should I do if I suspect one of the journal reviews I got is AI-generated?
I’ve submitted a paper to a reputable journal and received two reviews asking for a revision. One of the reviews is on-topic and helpful. The other, however, I suspect was generated by an AI based on the abstract only. I don’t want to debate the decision (the second reviewer is fair), but I wonder if I should mention my suspicions to the journal editor.
Should I? I doubt I can prove the use of AI. All I have is a strong suspicion. Plus, I don’t want to hurt my article. Yet, using AI for generating (not merely styling) reviews seems like such a bad practice that maybe it’s worth letting the editor know.
My suspicions are based on several things:
First, the style is off. The review consists only of very long questions like:
In terms of the paper's stated goal of contributing to the literature, how well does it provide [contribution mentioned in abstract] for [use-case mentioned in abstract]?
There are no suggestions, no feedback, and no statements. It’s only broad questions suspiciously rephrasing each line of the abstract (and heavily re-using entire phrases from it).
Second, the list of suggested articles is strange and includes existing papers (only names) from irrelevant fields.
Third, out of curiosity, I've run the text via several online AI-detection tools, and all of them conclude it’s AI (which I am aware is not a valid proof on its own).
3 Answers
I think you should contact the editor with basically the same content as your post here: explain why you think the review could be AI generated, including the limitations of each point of evidence. I think most editors see their job as letting reviewers guide their decision rather than reviewing the reviews, so I would start from the assumption that the editor would not have considered checking the reviews for suspicion of gen AI.
Explain why you think this is concerning, which I think at minimum would include the integrity of the peer review process (you and your colleagues are expecting papers to be reviewed by humans, not AI) and the protection of your intellectual property (having an AI review your paper involves inserting your work into a 3rd party tool which may incorporate it into the training data and is a definite breach of the confidentiality that reviewers are sworn to). I would check the journal's website to see if they have an explicit statement about use of AI in peer review; if so, you can refer to that as well. I think you can do this all in one sentence to not appear to lecture the editor.
Finally, make clear that you're not protesting the decision for a revision and plan to work on revising and resubmitting your paper. I'd include a note thanking the editor and the reviewers for their time.
From there, let the editor decide what should be done next; it's their call, your only duty here is to raise their awareness. It's possible they'll want to bring in another reviewer. It's also possible they'll decide it's okay. At that point you'll have to decide whether you're satisfied with their effort and whether that affects your opinion of the journal as an appropriate place to submit your work.
The Committee on Publication Ethics and related organizations are developing reporting and investigational best practices for this (COPE). It may be considered misconduct according to the policies of the journal, and if not disclosed, it most likely is and should be investigated by the editors. If they do not take your report seriously, I would question the integrity of the journal--see related material on predatory publishing.
In addition to reporting this to your editor for investigation, as @Bryan Krause stated above (+1 to everything he said), I would familiarize yourself with the work of the committee on this topic, as they are monitoring these issues closely and developing best practices and even formal guidelines on handling investigations.
COPE also provide significant resources to editors who may actually be less versed in dealing with these issues themselves, so it couldn't hurt to share that information with your editors if they need a bit of guidance.
COPE. 2023. Where next in Peer Review? Part II. https://publicationethics.org/news/where-next-peer-review-part-1.
I have been a journal editor for 20 years, and I would not pass on this review to the author. Definitely alert the journal.
Have a similar question?
Ask the community →