Peer review publications remain a key stage in the quality assurance of new research, but some comments can be the stuff of nightmares, says Kingsley Purdam

Peer review prior to acceptance and publication can make a substantial contribution to the development of research and to the furthering of knowledge and understanding. Reviewer comments can help identify gaps in the literature cited, shortcomings in the methodology and interpretation of the data, limitations to the claims being made, and also help with communicating the research more clearly.

Peer review, alongside the important gatekeeper role played by editors, remains the best available approach for the quality assurance of new research. Yet this is peer review at its best, and it can often fall short of this ideal.

We are exploring the quality and usefulness of comments provided by peer reviewers. Although there are other ongoing efforts to highlight peer review comments, our approach is to focus more closely on the contribution the peer review process can make to the development of knowledge.

We began by collating examples of the perplexing comments some authors have received. For instance, some reviewers express opinions of simply liking or not liking a paper. This would seem an unhelpful subjectivity. Comments such as: “I have read this paper several times, and was prepared to like it…but I ultimately did not like it very much at all”, “the interpretation of findings is a bit irritating” or “this paper was ultimately a disappointment” and on a more positive note “I liked this paper”. Such throw away comments can leave authors unclear about what needs to be done to revise and improve their work.

The clarity of comments can also be an issue for authors: “I suggest the authors avoid talking about things they do not know enough” or “the authors are making a completely wrong argument” are particularly unhelpful, while some comments can be quite opaque or elusive including, for example,: “How about a passing glance at Bourdieu?”

The tone of comments can also be a challenge. For most authors the time taken to write and submit an article for review is considerable and there is little value in reviewers being dismissive or even rude. Comments such as “the authors are seemingly ignorant of much of the literature” or “the section of Ethical Issues is pointless” seem particularly unhelpful. Most damning of all one reviewer wrote: “Perhaps a different enterprise should have been embarked upon – a different question with different data”.

Comments on articles that have been revised and resubmitted can also cause consternation for authors. As one reviewer wrote: “The author has satisfactorily responded to the comments to the previous draft. […] It is not a very strong paper”. Another author reported how one reviewer declined to comment on a revised paper.

Peer review can also involve long delays and this can vary considerably between different journals. As Crook has argued, this can impact significantly on the career development of new academics. Journal editors and reviewers also need to be alert to articles that contain fraudulent material, while there have also been cases of authors anonymously reviewing their own articles. At a time when there are increasing evidence and truth claims it is more important than ever that academic research is accessible and transparent, and that it can be verified. Peer review is a constructive and robust part of this process. Indeed recent research in the UK, which involved a survey of civil servants, found that peer reviewed journal articles are still an influential source of evidence amongst policy makers.

We are not dismissive of the commitment and effort of academics who give up their time to contribute to the peer review process. Of course the quoted comments above are out of context and almost all of the reviews we have examined contained some useful suggestions. Some reviews provide a very detailed analysis of an article running to several pages. Moreover we recognise that it can take time for authors to appreciate the value of some reviewer comments once the cloud of disappointment of an article being rejected has cleared.

However, even if well intended, some comments do strike us as not being as direct and helpful as they could be and they can create frustration for authors. Some reviews can just be a couple of dismissive paragraphs. As a recent survey found, whilst most reviewers are motivated by making a contribution to the field, some do not feel confident to write reviews and many would like more training.

More constructive reviews will lead to more constructive responses. This could also help editors make more informed decisions about whether to accept an article for publication, including when reviewers hold differing opinions. It may also help editors communicate their decision to reject an article even when the reviewer comments are very positive. Editors of course have their own concerns including looking after the reputation and profile of their journal.

• If you are haunted by peer review comments and have examples which you have found particularly unhelpful to improving your article – or suggestions for the ways in which reviewer reports can be improved – please email them to peerreviewresearch@manchester.ac.uk.  All submissions will be treated anonymously.