This article first appeared on Elephant in the Lab.
Double blind peer reviewing, compared to open reviewing, is a great system to ensure quality in research that focuses on the content rather than the person(s) behind the research. However, the current review system for many academic publications is flawed. It hinders publication of timely and excellent research for three main reasons. The examples provided for mediocre reviewer suggestions are from my own experience.
THREE PROBLEMS WITH THE CURRENT SYSTEM
First, while academic writing is already taught to undergraduates, academic reviewing is not even discussed in many PhD programs. Second, not having received best practises in their education, researchers do rarely get it from publishers either. Ferreira and her colleagues (2016) analyzing journals in Biology, argued that many journals neither provide guidance, let alone forms and defined criteria. From my personal experience, this holds true for other disciplines as well. Third, there are few incentives for academics to conduct thorough peer assessments, while the number of requests for reviews is simultaneously very high.
These reasons (and others) leave a significant portion of reviewers to deliver mediocre or bad assessments. About one third of reviews is considered useless or misleading, while another third is only OK (Prechelt et al. 2017). Sometimes coined “Reviewer 2” in memes and jokes, some reviewers ask authors to include research from their own study sub-field (sometimes suggesting quoting their very own publications). They ask researchers to include more literature and extend certain sections without explicitly mentioning why and what they are missing. Others are overly caustic and seem to be in an aggressive mood. Moreover, many reviewers ask to include additional analysis using their preferred method or ask authors to conduct additional experiments.
However, academia does not need to accept the current state. There are solutions both within the current system and beyond.
THREE SOLUTIONS FOR THE CURRENT SYSTEM
In terms of solving some of the problems within the prevailing setup, publishers should aim to both empower Associate Editors (AEs) and streamline the process, while universities should incorporate academic reviewing in their curricula.
First, some AEs do not use their power to mitigate the impact of bad reviewers. Other than simply concurring with the reviewers, they should encourage reviewers to be constructive and non-aggressive. When reviewers do not provide concrete guidance, AEs should single this out by asking the reviewers to either add information or by exempting the authors from having to respond to certain comments. Publishers should encourage their AEs to ensure such a behavior by setting clear rules and guidance.
Second, review processes take too long. Both authors and reviewers regularly have 8-12 weeks to complete a revision or review. This leads papers to remain behind review bars for well over a year and – in many cases – significantly longer. Yet, this is unnecessary. AEs could simply ask both authors and reviewers to conduct their work in 2-4 weeks or less. If matters that are more important come in the way, they can still ask for a vested extension. This would reduce waiting times, thereby cutting periods of vocational adjustments, and – more importantly – would allow others earlier access to cutting edge research.
Third, senior researchers and publishers should take the effort to educate reviewers more thoroughly on best practices for reviewing. These should find their way into PhD student education or even already in classes at Master level. As a study by Casnici et al. (2016) found that senior researchers are more likely to reject revision requests, junior researchers might realize through this education that they do not have to fully oblige to reviewers as some currently do.
Moving forward with post-publication
Yet, these are rather small improvements. Beyond the current system, there are various proposals to increase the quality of academic reviews more significantly. One rather prominent suggestion is post-publication peer review (PPPR). The principle is simple: Papers get published quickly after a light sanity check by an editor. Once published, referees conduct the same review process as with pre-publication review (i.e., the current system). Yet, their reviews get published right next to the article – possibly including the name and institute of the reviewer. Crucially, reviewers need not provide a recommendation to publish or withhold the paper (Hunter 2012). It is enough for them to point out problems and possible solutions. Authors can then either write rebuttals and try to solve problems. Reviewers could embark on solving these problems too. This in turn could increase the incentives to conduct reviews in the first place. Furthermore, PPPR could reduce the risk of reviewers to suggest unsubstantiated extensions of sections without providing concrete guidance (e.g., full citations).
Moreover, PPPR would require authors to share their data more openly in order to allow reviewers with all sorts of backgrounds to scrutinize their work. As both Psychology as well as Economics have recently suffered from a replication crisis, PPPR could contribute to reinstate trust.
In effect, authors would profit from better reviews and higher quality revisions, while reviewers could receive credit for their input more easily (da Silva & Dobranszki 2015). Some journals have taken up this idea and accept open peer review, where authors can ask their reviewers to publish their critique directly next to the article website. Publons collects peer reviews and evaluates them. Yet, these are only small changes.
USING TECHNOLOGICAL ADVANCES TO RECLAIM THE REVIEWING SYSTEM
New technologies offer the chance to improve the review system even further. For instance, Ferreira et al. (2016) argue that reviewing should be separated from the journals, towards a centralized platform that sets unilateral standards. They also argue that reviewing should be mandatory for all scientists and that monetary incentives should be provided to ensure this. Using distributed ledger technology (DLT), i.e. the blockchain, and building upon open access platforms, these proposals could be implemented in the real (academic) world. DLTs would be a powerful tool to support PPPR as they make it easy to trace the provenance and development of an article. The technology would even allow for anonymous, yet still trusted PPPR. Reviewers could also be compensated more easily, for instance, based on the readership of a paper (see Steam). DLT could support data sharing too. This would make the whole process more transparent and would encourage reviewers to be more constructive, while simultaneously gaining recognition for their reviews.
These changes to the system are bold, yet thankfully many researchers are open to change (Prechelt et al. 2017; The Cost of Knowledge). To some extent, there are also not in the interest of some of the major players in academia. Publishers disproportionately profit from free reviews and cheap articles provided for their journals, which in turn have fewer production costs and can still increase subscription prices for libraries and academic institutions. Predatory journals only make the situation wore (da Silva & Dobranszki 2015). Thus, changing the review system could contribute to changing the entire academic value chain for societal good. Thus, it might be worthwhile to revise our review system substantially.
PS: Reviewers interested in providing better reviews should avoid any of these formulations.
I would like to thank Benedikt Fecher for his valuable insights and suggestions that improved this article (in a streamlined, helpful process). This post is handled under a Creative Common License BY-SA 3.0.