The Golden Rule of Reviewing
Abstract
A major bottleneck in the time required to publish a scientific or scholarly paper is the speed with which reviews by peers are returned to journals. Peer review is a reciprocal altruistic system in which each individual may perform every task—editors, reviewers, and authors—at different times. Journals have no way to coerce reviewers to return their critiques faster. To greatly shorten the time to publication, all actors in this altruistic network should abide by the Golden Rule of Reviewing: review for others as you would have others review for you. Say yes to reviewing whenever your duties and schedule allow; provide a thorough, fair, and constructive critique of the work; and do it at your first opportunity regardless of the deadline.
I have always depended on the kindness of strangers. (Blanche DuBois)
We all want two things from the scientific journals where we send our papers for publication. First, we want a thorough, fair, and constructive critique of the manuscript. In general, the scientific peer‐review system ensures that this expectation is met; some problems will always exist, but we have great difficulty imagining a better and more cost‐effective system in practical terms for vetting the content of published works. Second, we all want the decisions about our papers to be made in as timely a fashion as possible. This second expectation frustrates authors, editors, and publishers alike.
We all know fields in which papers can appear in print within a month of submission, and we all know other fields in which papers may take years to wind their way through the bowels of review and editing to finally appear in print. Yet, papers in all peer‐reviewed journals pass through the same basic process from submission to publication. How can these tremendous discrepancies exist across fields that are all engaged in essentially the same activities? Moreover, how can the transit time in the slower fields and journals be reduced? These are questions with which every editor wrestles. All of these distill to a very simple question. How can we shorten the time required to make a decision about a manuscript? The answer to this question lies mainly with how scientists and scholars in these different fields respond to the peer‐review process.
The scientific peer‐review system is certainly an amazing cooperative network. We all know this, but we often forget and thus rarely stop to analyze how our own actions affect the entire system. Consider what happens when we submit a manuscript to a scientific journal (all our journals use some variant of this process but use somewhat different terms to identify the positions; we use American Naturalist terminology here). On submission, the paper enters a cooperative network of volunteers. An Editor reads the paper and then decides which of the Associate Editors on the Editorial Board will handle the actual review process. The Associate Editor then compiles a list of potential reviewers, those reviewers are asked whether they would be willing to review the paper in a timely fashion, and if they agree they are sent the paper. The reviewers then read the paper; consider the methods, data, analyses, and arguments; and write reviews containing their opinions about the paper and whether the paper should be published in that journal. The Associate Editor reads the reviews, usually two or more, often writes a third review, and makes a recommendation to the Editor. The Editor then writes a letter about the disposition of the paper.
The pace of work done by the professional staff at journals is never the time‐limiting feature of this chain. What determines how fast papers move through this system are the Editors, Associate Editors, and reviewers—the volunteer scientists and scholars.
Like Blanche DuBois, we all depend on the kindness of strangers to generously supply us with their time and expertise. Scientists and scholars do this because they expect others to reciprocate; the system works primarily because of the reciprocal altruism inherent in its operation. Scientists are willing to review papers because they know they will benefit in the future from similar contributions by others. As with any game involving altruists, cheaters are unwanted but unavoidable: some individuals flatly refuse to review papers or grants under any circumstances but expect others to review theirs. However, the vast majority of individuals feel that they have a responsibility to contribute their expertise as a reviewer whenever their schedule allows, and they take great pride in this work. Peer review succeeds only because the vast majority of scientists and scholars feel this way.
The reciprocal altruistic features inherent in the system influence not only whether the work gets done but also how fast. You are the reviewer for other people’s work, and they are the reviewers for your work. To provide a window into the review process, we have compiled the data on the reviewing time for every review submitted to the American Naturalist in 2008 as an example. Data for our other ecology, evolution, and behavior journals are similar. People responded very rapidly to the invitation to review, with 51% responding within 1 day and 78% responding within 4 days (fig. 1A). The primary bottleneck rested with the return time of reviews (fig. 1B). The American Naturalist requests that reviewers complete their work within 21 days (typical for our field) so that they have time to reflect on the manuscript and provide thorough and thoughtful analyses. We all also appreciate that reviewers are busy and may not be able to review a paper immediately. A small cadre of intrepid souls had rapid return times of 1–2 days, and the distribution is relatively flat for the rest of the first 2 weeks. However, only 41% of reviews were returned within 21 days. The modal return time was 22 days (the day when the journal office sends the first reminder that the review is late), and the distribution had a disappointingly long right tail (the maximum was 124 days, i.e., >4 months). In addition, given that multiple reviews are desired for a manuscript, the total time required to review a paper is defined by the longer return time. As a result, only 8% of the papers submitted to the American Naturalist in 2008 had external review completed within the desired 21 days (fig. 2), which is also a disappointing number.
Given that the entire process of peer review depends on the kindness of strangers, journals and granting agencies cannot force such distributions to shift left, and we have only a very tenuous ability to coax them to the left. Journals have absolutely no coercive power to compel their volunteer workforce to provide reviews on time or even to return late reviews: the volunteers will simply stop volunteering. Editors are also caught in a serious dilemma when a review is late. Can we coax the late reviewer to submit their review within a few days? What is the probability that the reviewer will submit the review at all? If we drop this reviewer and ask another, how long will it take the new reviewer to complete their task? Or should we base the decision about the manuscript on only one external review? Badgering reviewers is ineffective and ultimately detrimental; it is irritating and makes reviewers much less likely to review now and in the future. However, we cannot let this process drag on; authors deserve timely decisions.
The only way to significantly alter distributions like those in figures 1B and 2 into a more appealing form is for the entire community—Editors, Associate Editors, and reviewers alike—to continually remember the reciprocal altruistic nature of peer review. We should all abide by the Golden Rule of Reviewing:
Review for others as you would have others review for you.
So if you want your papers to appear faster in scientific journals, you can do something about it. Say yes to reviewing a manuscript or grant whenever your duties and schedule allow; provide a thorough, fair, and constructive critique of the work; and do it at your first opportunity regardless of the deadline. We all understand the dynamics of a system based on reciprocal altruism. Yes, the system must always support some level of cheaters, but the overall quality of the group depends on the frequency of altruists and the benefits that accrue to the group from the actions of each altruist. We all depend on the kindness of strangers; we are those strangers.