Peer Reviewing

Overview

This piece provides a few reflections and personal experiences of the peer review process. It was written in preparation for a workshop on peer reviewing, run by ‘Institute of Physics Publishing’ and the ‘Institute of Physics and Engineering in Medicine’.

The peer review process

Scientific journals are the main route for science communication.

Peer review is widely used to decide whether a manuscript should be published in a journal. The process consists of peers, i.e. other scientists in a similar field, reviewing a draft manuscript, providing feedback to the authors, and making a recommendation to the Editorial team on whether the manuscript should be published. Note that the peer review process is both decisive and constructive: it contributes to a decision on whether to publish a manuscript, and it also contributes to improving the manuscript.

The peer review process is often iterative: reviewers suggest areas for improvement, authors address these in an updated manuscript, and reviewers check whether their suggestions have been addressed (and if not can request further improvements).

Peer review contributes to upholding high standards in scientific publishing. However, it is not flawless. The COVID-19 pandemic helped us re-examine the role of peer review. Papers about COVID were often uploaded as preprints before peer review. On the one hand, results from preprints were discussed in the news, which could give the impression that they could be used to inform decision making without the need for peer review. On the other hand, news reporters were quick to note that results from preprints had not been subject to peer review, and therefore could not yet be fully trusted, potentially giving the impression that peer reviewed publications were flawless. Personally, I think the reality lies between these two extremes - peer review is a helpful process for enhancing scientific research and communication, but it does not guarantee flawless articles.

With this in mind, it’s helpful to consider what we are trying to achieve as peer reviewers. I primarily aim to make an informed recommendation of whether an article should be published, and secondly aim to suggest realistic improvements to the manuscript, and in some cases the study methodology, which could increase the value of the publication.

Why be a peer reviewer?

There are many reasons why one might be a peer reviewer. One benefit for me has been helping me to identify novel research directions and gaps in knowledge. In the late 2010s I reviewed several manuscripts for Physiological Measurement, and doing so helped broaden my interests. At the time I worked in a hospital, conducting research into using wearables to monitor the vital signs of acutely-ill hospital patients. Many of the manuscripts I reviewed focused on using wearables outside hospital, in the community. Furthermore, a hot topic appeared to be using wearables to detect atrial fibrillation, a common heart arrhythmia. This prompted me to consider conducting research in this area, and this is now my main area of research.

Models of peer review

Until relatively recently most of my peer reviewing was conducted using a ‘single-blind’ model of peer review, where the reviewers know the identity of the authors, but the authors do not know the identity of the reviewers. Recently, publishers such as IOP Publishing have begun using a ‘double-blind’ model, where the identities of both the authors and the reviewers are hidden. This ‘double-blind’ approach was found to reduce bias in the peer review process in a recent randomised trial (see here).

In retrospect, I certainly think I have been biased by seeing the names of well known researchers on the author list, and starting the review process with a prior assumption about the quality of the manuscript. One concern I had about the ‘double-blind’ process was the amount of work required by the authors to anonymise the manuscript. I recently co-ordinated a paper with 51 authors which underwent double-blind review. I was slightly apprehensive that the process for anonymising the paper would be overly burdensome, but to my surprise, I found it relatively easy to anonymise the paper to a reasonable extent.

There is also opportunity for peer review after publication. Here, I particularly like the model of writing letters to the editor. Some journals list alongside the original article any correspondence about it, which I think is healthy. I think when conducted in the right spirit it could be constructive. However, post-publication peer review is, in my field at least, seldom used, which highlights the importance of peer review prior to publication.

Deciding whether to accept an invitation to review

Here are a few of the more nuanced factors I consider when deciding whether to accept an invitation to review:

• How much of the scientific content of the manuscript falls within my expertise? Note that it’s acceptable to only be able to review some (and not all) of the scientific content). This is particularly true in multidisciplinary work, where it’s less likely that a single reviewer will have the range of expertise required to review all the scientific content of a manuscript. Here, it’s important to inform the journal about which parts you can and cannot review (perhaps through a ‘Comments to the Editor’ box on the review form).
• Can I provide an unbiased review? I find it relatively straightfoward to decide whether or not I have a conflict of interest (e.g. having worked with any of the co-authors previously). I find it more difficult to be aware of less obvious biases, such as biases towards scientific techniques which I am more familiar with, and I try to bear these in mind when conducting a review.
• Do I want to involve a colleague in the review process? IOP Publishing now offers ‘co-review’, in which two colleagues perform a peer review together, supporting each other through the process.
• Am I prepared to conduct this review to a high standard in a timely manner? If not, then it’s fine to decline the invitation, and it’s much better to decline at an early stage than to accept the invitation and later realise that you can’t conduct the review.

The contents of a review

When writing a review, it’s helpful to be aware of the journal’s scope, and any specific requirements for the journal. For instance, Physiological Measurement expects at least 30 subjects per group in studies on human subjects, and has specific expectation on the reporting of sex and gender (as detailed here).

Many publishers provide guidelines on what to include in a peer review report (e.g. see here for IOP Publishing’s guide). Generally, I would include the following in a review:

• An introductory paragraph summarising:
  • The study: i.e. what was done, and the key findings.
  • My assessment of how important the research question is (including the level of originality, and significance), how well it is addressed, and the clarify of the manuscript. (these are in-line with IOP’s suggestions)
  • Key strengths and weaknesses of the manuscript.
  • What needs to be done to make the manuscript ready for publication.
• Specific comments on potential improvements:
  • Numbered so that they can be easily referred to.
  • Separated into major comments (which I think should definitely be addressed) and minor comments (which I view as either easy to address, or optional).

Here are some examples of major and minor comments:

• Major Comments:
  • The study answered one particular question, but the title and abstract give the impression that it answers a bigger question.
  • A simple improvement to the data analysis would greatly increase the value of the results. “I think there are several areas in which the methodology could be improved to ensure the reliability of the results, and to ensure the results are as relevant as possible.” - this is particularly helpful in the context of data analysis, but less helpful if concerning data collection methodologies.
• Minor Comments:
  • Additional reference to add to discussion of prior art
  • Unclear wording, graphs, figures
  • Typos which affect the scientific content (e.g. mistakes in equations)
• Either category:
  • The number of subjects included in the analysis is not reported: this is easy to fix, but could affect the scientific value of the study.

I tend to put all my comments in the ‘Comments to Authors’ box, with two exceptions:

• I put any concerns about research integrity in the ‘Comments to Editors’ box.
• I also state any areas of the manuscript which I feel I am unable to review thoroughly in the ‘Comments to Editors’ box.

My approach

Personally, when reviewing a manuscript I consider two questions:

• does it ask an important research question; and
• does it answer it?

I note that I don’t place too much importance on how well cited I expect the article to be. This is because whilst I do think articles should address important research questions, I don’t think such articles will always be well cited. For instance, an article which addresses a less used physiological measurement technique may be important in that particular field, but may not be widely cited.

I try not to be too burdensome. There were times when I would give 20 or more comments on a manuscript. Having been on the receiving end of such reviews as an author, I now think I find it more helpful to receive the most important comments and to be given some flexibility about how to address them, rather than receiving an exhaustive list of demands from the reviewer.