Am I a qualified referee?

Editors align areas of expertise and skill sets of a referee with the focus of the manuscript and, on occasion, will choose referees with different backgrounds (including area specialists and generalists) to provide different perspectives. A perfect alignment seldom occurs. Often, a referee who is in an early phase of her/his career will be paired with a more experienced referee. In most instances, an individual asked to complete a review should assume that he/she has been chosen for a good reason. If the invited referee is uncomfortable with the alignment of her/his areas of expertise with the focus of manuscript, he/she should ask, “Can I improve my knowledge sets and understanding of the general area being addressed in the manuscript quickly enough to provide an effective review?”. If the answer is “no”, then the invitation to review should be declined promptly (within 24 h if possible; McPeek et al. 2009).

Can I provide an unbiased review?

Good reviews present unbiased assessments of a manuscript (Might 2016). Potential biases can occur for a variety of reasons: referees know authors, they are colleagues or collaborators, or they are competitors. However, these relationships do not necessarily create a biased review. From a journal’s perspective, a biased review would occur when the referee’s relationships to the author(s) resulted in an overly positive or negative review. This, in turn, could influence the editor’s decision to accept or reject the manuscript. In many instances, the bias is underlain by a conflict of interest whereby a referee or his/her colleagues would gain a professional benefit from the review. Potential conflicts of interest can be identified if referees ask the following questions: 1) Do I have a predetermined view of the manuscript prior to reviewing it? and 2) Would I, or my close colleagues, benefit if the manuscript were accepted or rejected? If a referee’s answer to either question is yes, the referee should decline the request to complete the review.

Can I complete the review by the required time?

A review requires several hours to one or more days to complete and most referees complete reviews across several days or weeks. When soliciting a review, journals specify a time line for its completion. If a potential referee has insufficient time, he/she should promptly decline the request to complete the review. However, if the review could be completed within a few days or a week after the preferred timeline, the AE should be informed so that she/he can consider the alternative timeline. Editors do have some flexibility in terms of advancing manuscripts so a short delay may be acceptable. A referee who agrees to complete a review should ensure it is submitted on time.

Understand your role as a referee

The primary role of a referee is to assess the scientific rigor and relevance of a manuscript (Research Information Network 2010, Wilson 2012, Baier and Baker 2013). The referee’s perception of this task can influence the content and tone of a review. Effective reviews can be guided by recognizing the following: 1) all manuscripts have strengths and weaknesses, 2) editors are not striving for perfection, 3) an important role of a referee is to improve the manuscript, and 4) once published, the work reflects the opinions of the authors not the referees.

On occasion, a referee might not identify major problems with a manuscript but disagrees with the overall approach taken and for this reason recommends rejection or major revision. In these cases, the referee should consider publishing an article critiquing the approach with which he/she disagrees, rather than singling out the manuscript for criticism.

A systematic approach

Thoughtful and critical assessments of the strengths and weaknesses of a manuscript often mean that the referee will ask the author(s) to provide clarity, additional information, and evaluations of alternative approaches and interpretations. The review should address all major sections of the manuscript (Table 1). Many referees evaluate manuscripts by addressing key questions that can focus the review (Research Information Network 2010, Anonymous 2011, Baier and Baker 2013, Anonymous 2016b) and subsequently structure the review as comments on each of the major sections (Table 1).

Referees must recognize that editorial staff from most journals, including Freshwater Science, assess manuscripts to ensure that they are: 1) within the scope of the journal, 2) of sufficient quality to warrant review, and 3) have not been published elsewhere. Consequently, in most cases, manuscripts that are sent for review should be of sufficient quality to warrant a referee’s time. Referees should always think long and hard about rejecting a manuscript because they think that it is outside of the scope of the journal or is of such low quality that it does not warrant review. However, screening processes are not perfect and if a referee continues to have serious reservations about the manuscript she/he should promptly contact the AE or EIC.

Tone matters

Effective reviews consist of a critical and thorough evaluation of the strengths and weaknesses of a submission that benefit the author, editor, and future readers. They present clear, helpful, courteous comments and, where appropriate, acknowledge positive points of a manuscript (Samet 1999, McPeek et al. 2009, Lucey 2013, Glen 2014, Hames 2016). A critical review should not be regarded as an opportunity to be cantankerous. As emphasized by Taylor (2016), tone matters. It influences the ability of the authors to address the issue rather than be preoccupied with a negative tone. Personal attacks or a tone (e.g., diatribes) that would prejudice the ability of the authors to respond have no place in a peer review (Samet 1999, McPeek et al 2009, Golash-Boza 2012, Glen 2014). Conversely, a pleasant and constructive tone should never compromise the ability of a referee to identify a serious flaw in a manuscript.

Constructing clear and concise comments

Questions and comments to authors must be clear and concise. Cross-referencing a comment to a specific location in a manuscript by identifying the line number is particularly useful to authors and editors. Referees often request that the author(s) provide additional details on the manuscript, especially study methods. Generic requests for additional information will often elicit generic responses. In many cases, however, referees are looking for specific information. For example, a request for an author(s) to provide additional information on their use of linear regression models is useful, but a more effective approach is to add another level of resolution by requesting that the authors present: 1) coefficient of determination, 2) p-values, 3) degrees of freedom, 4) processes used to reduce the suite of potential response variables to those in the final model (e.g., forward or backward selection), and 4) how assumptions of linear regression (e.g., normality, independence, homoscedasticity of residuals) were evaluated and met. Moreover, a request that authors consider an alternative statistical approach is reasonable but is more effective if the referee identifies the specific alternatives that he/she thinks should be considered and why this alternative approach might be beneficial. One-word comments from referees such as: “really!”, “unclear”, “awkward” or lengthy comments that are themselves unclear are not helpful. They also can detract from an otherwise effective review. However, we do encourage the use of 1-word positive comments such as “insightful” or “great”.

Dealing with poorly structured or written manuscripts

Low-quality manuscripts typically result from being poorly structured and poorly written. They compromise the ability of a referee to provide an effective review, and they pose substantial challenges for editorial staff. The editors of an increasing number of journals assess the quality of manuscripts before they are sent for review. If a referee thinks that the manuscript he/she agreed to review is of such low quality that it precludes review, the AE or the EIC should be contacted promptly so that the issue can be resolved. If a manuscript is of marginal quality, editorial staff would prefer that the referee complete the assignment.

In most cases, low-quality manuscripts are produced by inexperienced authors or authors for whom the published language of the journal is not the author’s first language. Many journals published in English offer support to authors where English is the author’s 2^nd language, but they need to be asked to do so. Other common causes of low-quality manuscripts include: 1) failure by authors to proofread the final version of the manuscript before it is submitted, 2) failure of the authors to solicit ‘friendly’ reviews of a manuscript prior to submitting it for publication, 3) rejection by authors of sage advice provided during presubmission reviews, and 4) poor communication among authors on when the manuscript is of sufficient quality to be submitted for publication.

Referees are not expected to resolve poorly written manuscripts by completely rewriting them. However, they do have a responsibility to identify where problems exist, identify specific examples, and offer potential solutions. For example, a suggestion that the authors retain the services of a scientific editor or consider involving a more experienced author in the publication process would be reasonable if a referee concludes that a manuscript is very poorly written. However, an unreasonable response to poor quality would be a statement by the referee that the low quality is because the authors do not understand what they are doing.

Discriminating between major and minor concerns

Referees often identify concerns that vary in importance. Dividing these concerns into those that are of major or minor importance helps the authors and AEs understand what the referee views as being big or small problems. Doing so also allows the AE to compare major concerns identified by the different referees and to derive a recommendation to the editor on whether the manuscript is acceptable for publication in its current form. Points of major concern include potential problems related to: 1) inappropriate framing of the relevance and impact of the work, 2) use of theoretical or conceptual models that are inappropriate, 3) study designs and statistical tests that cannot be used to test the study hypotheses, 4) use of discredited methods or incorrect application of analytical methods, 5) substantive gaps in logic, 6) misinterpretation of key findings, 7) ignoring a process that is known to have a strong influence on the system under study, and 8) drawing conclusions that are fundamentally inconsistent with data presented by the author(s) (Nicholas and Gordon 2011). A need for extensive re-analysis of data is also a major point of concern. Major points of concern may be solvable but often require substantial investments of time by authors. Unresolved major points of concern typically result in rejection of a manuscript.

In contrast, minor concerns are those that probably can be resolved by the author(s). They include requests to: 1) rephrase or reduce the length of existing materials, 2) add new information, 3) complete additional analyses or present results in alternative formats, and 4) rectify spelling and grammatical errors. On occasion, minor revisions addressing requests for additional analyses can provide results that may not be consistent with those previously presented. This situation does occur, but it seldom results in rejection of a manuscript. Rather, it can change the interpretation of the results.

Discriminating between what constitutes a major vs a minor point of concern is not always simple, and opinions may differ among referees, AEs, and the EIC. Ultimately, the AE and EIC decide on what constitutes a major point of concern and its influence on acceptance of the manuscript.

Aligning a referee’s assessment with a decision on acceptability of a manuscript

Referees, especially those early in their careers, often struggle to align their recommendation on acceptance of a manuscript with the number and importance of major and minor concerns. Some are reluctant to reject manuscripts, even if their review clearly shows that this recommendation is a logical and defensible outcome. Assuming that a manuscript is novel and is an appreciable contribution to our knowledge, a referee probably would recommend: 1) acceptance if she/he did not identify major points and 2) rejection if several major points of concern (see above) were identified. Recommendations of: 3) reject with the possibility of resubmission or 4) accept with major revisions are possible if the referee thinks that the major concerns could be resolved. A recommendation of reject should not be based on the number of points of minor concern, even if the number is high.

Referees can reduce their angst by understanding that: 1) referees make recommendations to an AE, AEs make recommendations to the EIC, and the EIC makes the final decision; 2) the recommendation made by the AE is based on ≥2 independent reviews and often is accompanied by her/his own review; 3) an AE may solicit an additional review if a decision on acceptability is unclear, and 4) authors who disagree with the EIC’s decision can contact editorial staff to discuss this issue further. Although uncommonly done, decisions made by an EIC can be revised.

Discriminating between inappropriate approaches vs personal preferences

All authors and referees have preferences regarding what they consider to be ‘good science’ and how manuscripts should be constructed. These include: 1) how hypotheses are formulated and tested, 2) presentation and interpretation of results, and 3) the focus and role of the discussion. Referees must identify inappropriate approaches objectively and suggest they be replaced with acceptable approaches. Asking an author to consider an alternative approach or interpretation and explaining why are reasonable steps. However, referees should refrain from requesting that an author replace an acceptable approach with another acceptable approach that is simply the referee’s preferred way of doing things.

Identify potential problems and offer multiple and well-articulated solutions

Referees’ concerns often include manuscript length, the need to simplify overly complicated components of a manuscript, and the extent to which conclusions are supported by evidence presented in the manuscript. Highly effective reviews present clear descriptions of problems and a range of potential solutions. For example, referees often state that a section of a manuscript or the entire manuscript is needlessly long and reductions of, for example, 20 to 30% are required. The next, more helpful, step is to identify how the authors can achieve this reduction. Many referees also question how well the results support conclusions drawn by the authors. An exceptional contribution from a referee would be to identify additional analyses (possibly using data that are available to the authors) that could be completed to more fully support the authors’ conclusions. Clarity and an appropriate level of brevity are key attributes of an effective manuscript and an effective review.

Constructing confidential comments to the editor

The referee’s confidential comments to the editor must summarize the strengths and weaknesses of the manuscript, her/his conclusions on the acceptability of the manuscript, and the amount of work required before the manuscript may be suitable for publication. The referee’s comments to both the authors and editors also should outline criticism or support for the manuscript, be consistent in tone and content, and be clear. Last, the referee should inform the AE if she/he lacks the expertise to assess all components of a manuscript.

Learn from the other referees’ comments and those of the AE

Editorial staff of many journals share reviews among referees during the process of communicating their decision to the authors. These shared reviews are often highly informative to novice referees because they identify the AE’s decision on the acceptability of the manuscript, the changes that the AE may have identified to make it suitable for publication, and how the editors have interpreted the importance of the concerns raised by each referee. In many cases, editors are open to discussing reviews and the review process with referees. This practice provides important learning opportunities for referees.