In 2023, an academic journal, the Annals of Operations Research, retracted an entire special isssue because the peer review process for it was compromised.
Author
- Gail Wilson
Adjunct Associate Professor, Office of the PVC (Academic Innovation), Southern Cross University
The case brought into sharp focus broader concerns about the peer review process in contemporary science. It showed that a process intended to catch problems with research before publication can itself go wrong.
And when it does, it creates large ripple effects that undermine the integrity of scientific research.
So how is the peer review meant to work? Why does it sometimes fail? And what can be done to improve it?
An evolving process
Peer review as we know it arose in the mid 20th century as the demand for specialised research grew following the end of the second world war. Contrast this with the 18th and 19th centuries, when peer review was undertaken mainly by editors of learned societies and university publishing presses.
Today, peer review is done largely by external peer reviewers who have been asked by a journal's editor to conduct a review of a manuscript focusing on the quality and value of the research.
They are selected from a pool of reviewers according to their discipline and their areas of expertise. Their task involves ensuring the paper is relevant to the aim and scope of the journal receiving the paper, reviewing the relevant literature, checking methodology, determining the importance of findings, highlighting areas that have been omitted in the paper, and suggesting changes to improve the paper overall.
Traditional forms of peer review occur before a paper is published. Both reviewers and authors remain anonymous.
Different disciplines take a slightly different approach to the review process. In the humanities, for example, double-blind peer review is favoured. This is where two external peer reviewers review the paper and send their reviews to the editor handling that paper. The author then responds to the editor's and reviewers' recommendations.
Based on editorial approval, the paper goes forward to publication.
Contrast this approach with open peer review which can occur both before and after the publication of an article. Supporters of this approach state that it promotes transparency and accountability.
Challenges with the current system
The example of the Annals of Operations Research retracting an entire special issue because of problems with the peer review process isn't isolated. Springer Nature retracted a total of 2,923 papers from their large journal portfolio in 2024, citing research and academic integrity issues.
A year earlier, the Journal of Electronic Imaging also retracted nearly 80 papers following an investigation into peer review fraud.
Actions like this highlight the many challenges to the current peer review system.
For example, heavy academic workloads and institutional pressures on academics to produce more and more publications reduces the time they can spend as external peer reviewers. It also prevents them from agreeing to be a peer reviewer in the first place.
This leads to what is called peer reviewer fatigue , meaning the reviewer simply doesn't have the capacity to do any more reviews at this time. Any journal editor can attest to this reason being given. Reviewers who produce quality manuscript reviews often also get more requests from journal editors than they are able to respond to, because of the time factor and their workload and institutional commitments mentioned above.
There's also the potential for manipulation of the peer review process. This can include the issue of a fake peer review - a process by which authors are asked to suggest reviewers and where fake email addresses and fake peer reviews are submitted. There are signs artificial intelligence is exacerbating this problem .
Predatory journals with dubious publishing practices such as charging authors a fee for publishing an article also publish low-quality articles that have not gone through a rigorous peer review process.
In a guest post for the academic integrity website Retraction Watch , educational researcher Richard Phelps blamed journal editors for not reviewing an article's literature review for accuracy. The post criticised dismissive claims from researchers about the absence of previous research on the topic, and low-quality literature reviews more broadly.
Strengthening the process
There are ways for journal editors to strengthen their journal's peer review process in relation to the quality of the reviewer pool and the quality of reviews received.
Journals can regularly review their current reviewer pool and broaden that pool by writing directly to authors of recently published papers. They can also make personal approaches to researchers in the field to undertake a review or be added to the reviewer pool list.
Journals can also review their current guidelines for reviewers to ensure there is a consistent set of criteria reviewers can use to rate the paper and explain the reasons for their ratings across key elements of the manuscript.
A "strength-based approach" to review can be encouraged. This is where feedback about the paper's strengths as well as the gaps in the paper makes the feedback more "developmental" and less focused on what's wrong with the paper.
From my experience as a journal editor, authors also find it helpful to receive the reviewers' comments together with an overall summary from the editor highlighting the key issues raised by the reviewers.
Gail Wilson does not work for, consult, own shares in or receive funding from any company or organisation that would benefit from this article, and has disclosed no relevant affiliations beyond their academic appointment.
