Writing highly effective reviews of a scientific manuscript
Peer review is an essential component of the scientific method. Here we offer advice on how to produce highly effective reviews. Our advice is presented in 2 parts in which we describe: 1) initial questions that a referee should address before agreeing to conduct a review and 2) formulaic components of a review. Our advice reflects our collective experiences over 40 y during which we have completed >200 reviews for at least 25 journals, and one of us (GJS) has 14 y of ongoing service as an associate editor with Freshwater Science and its predecessor, the Journal of the North American Benthological Society. We also have drawn advice from published and nontraditional accounts on how to complete effective reviews. We emphasize the role of referees and editorial staff in the peer review process. Our advice may be of greatest value to scientists who are in early phases of their careers.
Peer review is an essential component of the scientific method and effective reviews improve the quality of published papers. Highly effective reviews build credibility of journals, authors, and the scientific community. They also reduce the length of time between manuscript submission and publication by preventing time-consuming multiple rounds of reviews. Last, reviews provide learning opportunities for authors, referees, and editorial staff (Waser et al. 1992, McPeek et al. 2009, Research Information Network 2010, Baier and Baker 2013, Taylor 2016). The review process serves transparency in science and enables editors to make well-reasoned decisions on whether a manuscript should be accepted for publication (Nicholas and Gordon 2011, Baier and Baker 2013). Effective reviews also serve as one of several filters to reduce the frequency and magnitude of unethical practices, such as plagiarism (Baier and Baker 2013). Authors, referees, and editorial staff are committed to the peer review process. Authors expect quality reviews, and referees, editors, and associate editors (AEs) expect quality manuscripts. All participants expect a timely, transparent, and respectful process.
Despite its importance, a paucity of information is available on how to complete effective reviews. For instance, while journals provide guidance on reviews, the information often appears at a coarse level of resolution that precludes identifying some of the more subtle components required to ensure high-quality reviews. Traditionally, graduate students were seldom provided formal training on how to complete reviews (but see Anonymous 2011). The transition from a novice referee to a seasoned and highly competent referee is often challenging for recent graduates and editorial staff. However, the amount of information and formal training available to emerging scientists on how to complete reviews have increased over the last few years. This includes published accounts of the entire scientific publication process, the roles of all participants, and advice on how to complete effective reviews (e.g., Williams 2004, McPeek et al. 2009, Anonymous 2011, Nicholas and Gordon 2011, Baier and Baker 2013, Glen 2014, Alam and Patel 2015). This advice also includes an influx of nontraditional sources (e.g., blogs, editorials) that collectively provide a wealth of useful information (e.g., Zimmerman et al. 2011, Lucey 2013, Groom 2014, Alam and Patel 2015, Anonymous 2016a, b, Might 2016). The increase in the amount and availability of information on how to create effective reviews is important but still falls short of meeting increasing demands.
Here, we offer advice on how to produce highly effective reviews by identifying initial considerations and the fundamentals of creating highly effective reviews. We have drawn from several influential sources of information (McPeek et al. 2009, Nicholas and Gordon 2011, Baier and Baker 2013, Glen 2014, Alam and Patel 2015, Hames 2016) and combined these with our own experiences. Our advice may be of greatest value to scientists who are in the early phases of their careers (i.e., emerging scientists). We also highlight the role of referees and editorial staff (i.e., support staff, AEs, and the editor-in-chief [EIC]) in the peer review process (Hames 2016).
Am I a qualified referee?
Editors align areas of expertise and skill sets of a referee with the focus of the manuscript and, on occasion, will choose referees with different backgrounds (including area specialists and generalists) to provide different perspectives. A perfect alignment seldom occurs. Often, a referee who is in an early phase of her/his career will be paired with a more experienced referee. In most instances, an individual asked to complete a review should assume that he/she has been chosen for a good reason. If the invited referee is uncomfortable with the alignment of her/his areas of expertise with the focus of manuscript, he/she should ask, “Can I improve my knowledge sets and understanding of the general area being addressed in the manuscript quickly enough to provide an effective review?”. If the answer is “no”, then the invitation to review should be declined promptly (within 24 h if possible; McPeek et al. 2009).
Can I provide an unbiased review?
Good reviews present unbiased assessments of a manuscript (Might 2016). Potential biases can occur for a variety of reasons: referees know authors, they are colleagues or collaborators, or they are competitors. However, these relationships do not necessarily create a biased review. From a journal’s perspective, a biased review would occur when the referee’s relationships to the author(s) resulted in an overly positive or negative review. This, in turn, could influence the editor’s decision to accept or reject the manuscript. In many instances, the bias is underlain by a conflict of interest whereby a referee or his/her colleagues would gain a professional benefit from the review. Potential conflicts of interest can be identified if referees ask the following questions: 1) Do I have a predetermined view of the manuscript prior to reviewing it? and 2) Would I, or my close colleagues, benefit if the manuscript were accepted or rejected? If a referee’s answer to either question is yes, the referee should decline the request to complete the review.
Can I complete the review by the required time?
A review requires several hours to one or more days to complete and most referees complete reviews across several days or weeks. When soliciting a review, journals specify a time line for its completion. If a potential referee has insufficient time, he/she should promptly decline the request to complete the review. However, if the review could be completed within a few days or a week after the preferred timeline, the AE should be informed so that she/he can consider the alternative timeline. Editors do have some flexibility in terms of advancing manuscripts so a short delay may be acceptable. A referee who agrees to complete a review should ensure it is submitted on time.
Creating Highly Effective Reviews
Understand your role as a referee
The primary role of a referee is to assess the scientific rigor and relevance of a manuscript (Research Information Network 2010, Wilson 2012, Baier and Baker 2013). The referee’s perception of this task can influence the content and tone of a review. Effective reviews can be guided by recognizing the following: 1) all manuscripts have strengths and weaknesses, 2) editors are not striving for perfection, 3) an important role of a referee is to improve the manuscript, and 4) once published, the work reflects the opinions of the authors not the referees.
On occasion, a referee might not identify major problems with a manuscript but disagrees with the overall approach taken and for this reason recommends rejection or major revision. In these cases, the referee should consider publishing an article critiquing the approach with which he/she disagrees, rather than singling out the manuscript for criticism.
A systematic approach
Thoughtful and critical assessments of the strengths and weaknesses of a manuscript often mean that the referee will ask the author(s) to provide clarity, additional information, and evaluations of alternative approaches and interpretations. The review should address all major sections of the manuscript (Table 1). Many referees evaluate manuscripts by addressing key questions that can focus the review (Research Information Network 2010, Anonymous 2011, Baier and Baker 2013, Anonymous 2016b) and subsequently structure the review as comments on each of the major sections (Table 1).
|1||Is the focus and relevance of the manuscript clearly described?|
|2||Does it present the key findings and interpretations?|
|3||Is the strength of the conclusion(s) the same as that described in the Results and Discussion?|
|4||Are final points required to highlight future work, and if present, are they well described?|
|1||Has the context, relevance, and need for the research been correctly framed and supported with relevant published studies?|
|2||Is the research original and interesting?|
|3||Is the current state of the knowledge about the field accurately represented?|
|4||Have the scientific questions that are addressed been framed appropriately?|
|5||Are hypotheses and predictions required and are they appropriately described?|
|1||Are the methods described in sufficient detail so that the work could be replicated?|
|2||Is the study design clear (e.g., theoretical, experimental manipulation, natural experiment)?|
|3||Are the conceptual, analytical, or statistical models appropriately structured and have they been interpreted correctly?|
|4||Are the data-collection methods (and if relevant, are site locations appropriate) well described and are there clear linkages to the statistical models used to test hypotheses?|
|5||Are the methods under or over described?|
|6||If the study required the use of animals, is there clear evidence that ethics for the care and handling were reviewed and met?|
|1||Have all of the study questions been addressed?|
|2||Have the results for each study question been presented clearly and supported by statistical or analytical models?|
|3||Are the results convincing and are there additional analyses that could be completed to support or reject hypotheses and test predictions?|
|4||For journals that provide supplemental sources of information, can any of the Results be moved from the main body of the manuscript and presented as “Supplemental results”?|
|1||Do the authors need an opening paragraph that states the main objectives of the research and the key findings as a prelude?|
|2||Have the authors found the right balance between presenting results so that the broader context of them can be discussed rather than a rephrasing of the key results?|
|3||Has the discussion reverted into a rephrasing of the Results with little evaluation of the broader significance of these results?|
|4||Do the authors need to identify caveats to the major findings and interpretations?|
|5||Have the authors integrated the findings with existing knowledge?|
|6||Have the authors identified the need for future research efforts and is this information required?|
|1||Are all references cited in the body of the manuscript present in the reference section and are all citations in the reference section present in the body of the manuscript?|
|2||Have the references been correctly cited and formatted?|
|3||Have the most relevant works been cited and do they include recent publications?|
|4||Is the number of references appropriate and could the number be reduced?|
Referees must recognize that editorial staff from most journals, including Freshwater Science, assess manuscripts to ensure that they are: 1) within the scope of the journal, 2) of sufficient quality to warrant review, and 3) have not been published elsewhere. Consequently, in most cases, manuscripts that are sent for review should be of sufficient quality to warrant a referee’s time. Referees should always think long and hard about rejecting a manuscript because they think that it is outside of the scope of the journal or is of such low quality that it does not warrant review. However, screening processes are not perfect and if a referee continues to have serious reservations about the manuscript she/he should promptly contact the AE or EIC.
Effective reviews consist of a critical and thorough evaluation of the strengths and weaknesses of a submission that benefit the author, editor, and future readers. They present clear, helpful, courteous comments and, where appropriate, acknowledge positive points of a manuscript (Samet 1999, McPeek et al. 2009, Lucey 2013, Glen 2014, Hames 2016). A critical review should not be regarded as an opportunity to be cantankerous. As emphasized by Taylor (2016), tone matters. It influences the ability of the authors to address the issue rather than be preoccupied with a negative tone. Personal attacks or a tone (e.g., diatribes) that would prejudice the ability of the authors to respond have no place in a peer review (Samet 1999, McPeek et al 2009, Golash-Boza 2012, Glen 2014). Conversely, a pleasant and constructive tone should never compromise the ability of a referee to identify a serious flaw in a manuscript.
Constructing clear and concise comments
Questions and comments to authors must be clear and concise. Cross-referencing a comment to a specific location in a manuscript by identifying the line number is particularly useful to authors and editors. Referees often request that the author(s) provide additional details on the manuscript, especially study methods. Generic requests for additional information will often elicit generic responses. In many cases, however, referees are looking for specific information. For example, a request for an author(s) to provide additional information on their use of linear regression models is useful, but a more effective approach is to add another level of resolution by requesting that the authors present: 1) coefficient of determination, 2) p-values, 3) degrees of freedom, 4) processes used to reduce the suite of potential response variables to those in the final model (e.g., forward or backward selection), and 4) how assumptions of linear regression (e.g., normality, independence, homoscedasticity of residuals) were evaluated and met. Moreover, a request that authors consider an alternative statistical approach is reasonable but is more effective if the referee identifies the specific alternatives that he/she thinks should be considered and why this alternative approach might be beneficial. One-word comments from referees such as: “really!”, “unclear”, “awkward” or lengthy comments that are themselves unclear are not helpful. They also can detract from an otherwise effective review. However, we do encourage the use of 1-word positive comments such as “insightful” or “great”.
Dealing with poorly structured or written manuscripts
Low-quality manuscripts typically result from being poorly structured and poorly written. They compromise the ability of a referee to provide an effective review, and they pose substantial challenges for editorial staff. The editors of an increasing number of journals assess the quality of manuscripts before they are sent for review. If a referee thinks that the manuscript he/she agreed to review is of such low quality that it precludes review, the AE or the EIC should be contacted promptly so that the issue can be resolved. If a manuscript is of marginal quality, editorial staff would prefer that the referee complete the assignment.
In most cases, low-quality manuscripts are produced by inexperienced authors or authors for whom the published language of the journal is not the author’s first language. Many journals published in English offer support to authors where English is the author’s 2nd language, but they need to be asked to do so. Other common causes of low-quality manuscripts include: 1) failure by authors to proofread the final version of the manuscript before it is submitted, 2) failure of the authors to solicit ‘friendly’ reviews of a manuscript prior to submitting it for publication, 3) rejection by authors of sage advice provided during presubmission reviews, and 4) poor communication among authors on when the manuscript is of sufficient quality to be submitted for publication.
Referees are not expected to resolve poorly written manuscripts by completely rewriting them. However, they do have a responsibility to identify where problems exist, identify specific examples, and offer potential solutions. For example, a suggestion that the authors retain the services of a scientific editor or consider involving a more experienced author in the publication process would be reasonable if a referee concludes that a manuscript is very poorly written. However, an unreasonable response to poor quality would be a statement by the referee that the low quality is because the authors do not understand what they are doing.
Discriminating between major and minor concerns
Referees often identify concerns that vary in importance. Dividing these concerns into those that are of major or minor importance helps the authors and AEs understand what the referee views as being big or small problems. Doing so also allows the AE to compare major concerns identified by the different referees and to derive a recommendation to the editor on whether the manuscript is acceptable for publication in its current form. Points of major concern include potential problems related to: 1) inappropriate framing of the relevance and impact of the work, 2) use of theoretical or conceptual models that are inappropriate, 3) study designs and statistical tests that cannot be used to test the study hypotheses, 4) use of discredited methods or incorrect application of analytical methods, 5) substantive gaps in logic, 6) misinterpretation of key findings, 7) ignoring a process that is known to have a strong influence on the system under study, and 8) drawing conclusions that are fundamentally inconsistent with data presented by the author(s) (Nicholas and Gordon 2011). A need for extensive re-analysis of data is also a major point of concern. Major points of concern may be solvable but often require substantial investments of time by authors. Unresolved major points of concern typically result in rejection of a manuscript.
In contrast, minor concerns are those that probably can be resolved by the author(s). They include requests to: 1) rephrase or reduce the length of existing materials, 2) add new information, 3) complete additional analyses or present results in alternative formats, and 4) rectify spelling and grammatical errors. On occasion, minor revisions addressing requests for additional analyses can provide results that may not be consistent with those previously presented. This situation does occur, but it seldom results in rejection of a manuscript. Rather, it can change the interpretation of the results.
Discriminating between what constitutes a major vs a minor point of concern is not always simple, and opinions may differ among referees, AEs, and the EIC. Ultimately, the AE and EIC decide on what constitutes a major point of concern and its influence on acceptance of the manuscript.
Aligning a referee’s assessment with a decision on acceptability of a manuscript
Referees, especially those early in their careers, often struggle to align their recommendation on acceptance of a manuscript with the number and importance of major and minor concerns. Some are reluctant to reject manuscripts, even if their review clearly shows that this recommendation is a logical and defensible outcome. Assuming that a manuscript is novel and is an appreciable contribution to our knowledge, a referee probably would recommend: 1) acceptance if she/he did not identify major points and 2) rejection if several major points of concern (see above) were identified. Recommendations of: 3) reject with the possibility of resubmission or 4) accept with major revisions are possible if the referee thinks that the major concerns could be resolved. A recommendation of reject should not be based on the number of points of minor concern, even if the number is high.
Referees can reduce their angst by understanding that: 1) referees make recommendations to an AE, AEs make recommendations to the EIC, and the EIC makes the final decision; 2) the recommendation made by the AE is based on ≥2 independent reviews and often is accompanied by her/his own review; 3) an AE may solicit an additional review if a decision on acceptability is unclear, and 4) authors who disagree with the EIC’s decision can contact editorial staff to discuss this issue further. Although uncommonly done, decisions made by an EIC can be revised.
Discriminating between inappropriate approaches vs personal preferences
All authors and referees have preferences regarding what they consider to be ‘good science’ and how manuscripts should be constructed. These include: 1) how hypotheses are formulated and tested, 2) presentation and interpretation of results, and 3) the focus and role of the discussion. Referees must identify inappropriate approaches objectively and suggest they be replaced with acceptable approaches. Asking an author to consider an alternative approach or interpretation and explaining why are reasonable steps. However, referees should refrain from requesting that an author replace an acceptable approach with another acceptable approach that is simply the referee’s preferred way of doing things.
Identify potential problems and offer multiple and well-articulated solutions
Referees’ concerns often include manuscript length, the need to simplify overly complicated components of a manuscript, and the extent to which conclusions are supported by evidence presented in the manuscript. Highly effective reviews present clear descriptions of problems and a range of potential solutions. For example, referees often state that a section of a manuscript or the entire manuscript is needlessly long and reductions of, for example, 20 to 30% are required. The next, more helpful, step is to identify how the authors can achieve this reduction. Many referees also question how well the results support conclusions drawn by the authors. An exceptional contribution from a referee would be to identify additional analyses (possibly using data that are available to the authors) that could be completed to more fully support the authors’ conclusions. Clarity and an appropriate level of brevity are key attributes of an effective manuscript and an effective review.
Constructing confidential comments to the editor
The referee’s confidential comments to the editor must summarize the strengths and weaknesses of the manuscript, her/his conclusions on the acceptability of the manuscript, and the amount of work required before the manuscript may be suitable for publication. The referee’s comments to both the authors and editors also should outline criticism or support for the manuscript, be consistent in tone and content, and be clear. Last, the referee should inform the AE if she/he lacks the expertise to assess all components of a manuscript.
Learn from the other referees’ comments and those of the AE
Editorial staff of many journals share reviews among referees during the process of communicating their decision to the authors. These shared reviews are often highly informative to novice referees because they identify the AE’s decision on the acceptability of the manuscript, the changes that the AE may have identified to make it suitable for publication, and how the editors have interpreted the importance of the concerns raised by each referee. In many cases, editors are open to discussing reviews and the review process with referees. This practice provides important learning opportunities for referees.
The Review Elephant in The Room and A Call To Action
Effective reviews are the cornerstone of advancing scientific enquiry, and the ability to secure them is crucial to maintaining scientific credibility of the publication process. Conducting reviews is a professional obligation and a responsibility of researchers. The scientific community has entered a period of imbalance in the demand for and supply of referees. This imbalance is a result of a declining pool of seasoned referees (associated with the retirements of the baby-boomer generation) and production of increasing numbers of journal articles. Editors are finding it increasingly difficult to secure effective and timely reviews, authors are expressing concerns about the quality and timeliness of reviews, and referees are expressing concerns about the number of reviews that they are being asked to complete. This problem needs to be recognized and resolved—it is the elephant in the room. Our view is that we need to reduce our preoccupation with talking about the problem and to engage in solutions to resolve it. We need to fast-track the development of a new pool of qualified referees by: 1) increasing formal training of graduate students by academics and academic institutions (e.g., a component of graduate level courses) (Anonymous 2011), 2) providing mentoring opportunities, and 3) increasing the availability of materials like those presented in this 3-part editorial. Diverse mentoring opportunities exist. An experienced referee (e.g., a professor) could complete a review for a journal but, in an independent process, ask a graduate student to do the same. Some AEs are willing to critique draft reviews completed by inexperienced referees before formal submission of the review to the journal. Others complete independent reviews that subsequently are shared with the inexperienced referee. All 3 approaches provide useful learning opportunities by allowing inexperienced referees to compare the tone and content of reviews completed by referees with different levels of experience. Last, the staff of scientific journals and members of scientific societies (e.g., Baier and Baker 2013) can play a more central and active role in engaging and increasing the capacity of a new cohort of referees (Zimmerman et al. 2011).
We thank Brad Taylor for stimulating this editorial series. Brad Taylor, his research team, and John Bailey provided a suite of useful comments on a draft of this editorial that improved its tone, content, and breadth. Pamela Silver provided clarity on several aspects of the manuscript and shared some of her own views as a seasoned editor. We also thank Joseph Culp, Fred Wrona, David Rosenberg and Rosemary Mackay, and Bob Hudson for molding our views on how to complete effective reviews.
Alam, S., and J. Patel. 2015. Peer review: tips from field experts for junior reviewers. BioMed Central 13:269–270. Anonymous. 2011. Completing a peer review. University of Guelph, Integrative Biology Undergraduate Curriculum Committee 2011/2012. University of Guelph, Guelph, Ontario. (Available from: https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0ahUKEwjSzPK9or3OAhUDTCYKHS_zDpQQFggcMAA&url=https%3A%2F%2Fwww.uoguelph.ca%2Fib%2Fpdfs%2FCompleting%2520a%2520Peer%2520Review.pdf&usg=AFQjCNE-DXWSGCOSXpx-n_hHb6Pa4cyL-Q&sig2=cZi6VqDCwyYCrLAGlOYP1Q&cad=rja) Anonymous. 2016a. Writing a critical review. University of New South Wales, Sydney, Australia. (Available from: https://student.unsw.edu.au/writing-critical-review) Anonymous. 2016b. Writing a journal article review. Australian National University, Canberra, Australia. (Available from: https://academicskills.anu.edu.au/node/492) Baier, A., and L. Baker. 2013. A guide to peer review in ecology and evolution. British Ecological Society, London, UK. (Available from: http://www.britishecologicalsociety.org/wp-content/uploads/2016/04/Guide-to-Peer-Review.pdf) Glen, A. S. 2014. A new “golden rule” for peer review? Bulletin of the Ecological Society of America 95:431–434. Golash-Boza, T. 2012. How to write a peer review for an academic journal: six steps from start to finish. PhD2Published. (Available from: http://www.phd2published.com/2012/05/09/how-to-write-a-peer-review-for-an-academic-journal-six-steps-from-start-to-finish-by-tanya-golash-boza/) Groom, H. 2014. Peer review fundamentals and the future. WordPress.com. (Available from: https://elementofnature.wordpress.com/2014/05/14/peer-review-fundamentals-and-the-future) Hames, I. 2016. Peer Review golden rules and good practice checklist. Science Editing 3(1):36–42. Lucey, B. 2013. Peer review: how to get it right—10 tips. The Guardian, New York. (Available from: https://www.theguardian.com/higher-education-network/blog/2013/sep/27/peer-review-10-tips-research-paper) McPeek, M. A., D. L. DeAngelis, R. G. Shaw, A. J. Moore, M. D. Rausher, D. R. Strong, A. M. Ellison, L. Barrett, L. Rieseberg, M. D. Breed, J. Sullivan, C. M. Osenberg, M. Holyoak, and M. A. Elgar. 2009. The golden rule of reviewing. American Naturalist 173:E155–E158. Might, M. 2016. How to peer review. Matt.might.net. (Available from: http://matt.might.net/articles/how-to-peer-review/) Nicholas, K. A., and W. Gordon. 2011. A quick guide to writing a solid peer review. Eos 92:233–234. Research Information Network. 2010. Peer review—a guide for researchers. Research information network. (Available from: www.rin.ac.uk/peer-review-guide) Samet, J. M. 1999. Dear author—advice from a retiring editor. American Journal of Epidemiology 150:433–436. Taylor, B. W. 2016. Writing an effective response to a manuscript review. Freshwater Science 35:1082–1087. Waser, N. M., M. V. Price, and R. K. Grosberg. 1992. Writing an effective manuscript review. BioScience 42:621–623. Williams, H. C. 2004. How to reply to referees’ comments when submitting manuscripts for publication. Journal of the American Academy of Dermatology 51:79–83. Wilson, J. 2012. Peer review: the nuts and bolts. Standing up for Science 3. (Available from: http://www.senseaboutscience.org/data/files/resources/99/Peer-review_The-nuts-and-bolts.pdf) Zimmerman, N., R. Salguero-Gomez, and J. Ramos. 2011. The next generation of peer reviewing. Frontiers in Ecology 9(4). doi:10.1890/1540-9295-9.4.199.