Peer review is 'the least worst' system we have for verifying the ideas and conclusions in research publications. Peer review the nuts and bolts: a guide for early career researchers describes peer review as: "the system used to assess the quality of scientific research before it is published. Independent researchers in the same field scrutinise research papers for validity, significance and originality to help editors assess whether research papers should be published in their journal."
The Sense about Science project notes the purpose of reviewing is to:
- Comment on its validity – are the research results credible; are the design and methodology appropriate?
- Judge the significance - is it an important finding?
- Determine its originality - are the results new? Does the paper refer properly to work done by others?
- Give an opinion as to whether the paper should be published, improved or rejected (usually to be submitted elsewhere).
What is expected of reviewers?
The Proceedings of the National Academy of Sciences ask in their Peer Reviewer Instructions that their reviewers ensure the:
- Research is well designed and executed.
- Presentation of methods will permit replication.
- Data are unambiguous and properly analyzed.
- Conclusions are supported by data.
The CellPress Information for Reviewers states that the: Key features of a review include:
- an outline of the conceptual advance over previously published work,
- a specific recommendation,
- the reasons for that recommendation, and
- a summary of the specific strengths and weaknesses of the paper. In this regard, we encourage referees to comment on the quality and presentation of the figures as well as the validity of the statistical methods used to interpret them. (If necessary, the editors can obtain primary data from the authors for referees’ use in these more detailed evaluations.)
In just these two examples we can see there are considerably different expectations of reviewers depending on the publisher.
The Council of Science Editors have a White Paper on Publication Ethics which include 'Reviewer roles and responsibilities' (approved in 2012).
Tools that can help reviewers
The Peer Review at Science Publications page has comprehensive instructions and information for reviewers.
The Association of American Medical Colleges have comprehensive Reviewer Instructions based on the 'commonly accepted Glassick standards of educational scholarship' (Glassick CE, Huber MR, Maeroff GI. Scholarship Assessed: Evaluation of the Professoriate. 1997; San Francisco. CA: Jossey-Bass). These instructions relate to reviewing educational materials.
More reading on peer review issues
Samir Hachani (2015), Open Peer Review: Fast Forward for a New Science, in Anne Woodsworth , W. David Penniman (ed.) Current Issues in Libraries, Information Science and Related Fields (Advances in Librarianship, Volume 39) Emerald Group Publishing Limited, pp.115 - 141
Singal, Jesse, The Case of the Amazing Gay-Marriage Data: How a Graduate Student Reluctantly Uncovered a Huge Scientific Fraud, The Science of Us, 29 May 2015 - 'A discussion of the retraction of a Science paper in May 2015 because the supporting data was entirely fabricated. The paper got through peer review because it had a big name researcher on it. The final clue was getting hold of a closed data set that: 'wasn’t a publicly accessible dataset, but Kalla had figured out a way to download a copy'.
Cantor, Mauricio and Gero, Shane, Passing Review: how the R-index aims to improve the peer-review system by quantifying reviewer contributions LSE The Impact Blog, 20 May 2015 - '...giving citable recognition to reviewers can improve the system by encouraging more participation but also higher quality, constructive input, without the need for a loss of anonymity.'
Baldwin, Melinda, Is Peer Review Broken? Time Magazine, 29 April 2014 - 'If peer review is indeed broken, as some observers have claimed, an important part of fixing it may be adjusting our expectations of it'.
Hames, Irene, (2007) Peer Review and Manuscript Management in Scientific Journals: Guidelines for Good Practice, Blackwell Publishing, Malden, MA - This comprehensive yet concise book provides a thorough and complete guide to every aspect of managing the peer review process for scientific journals. Until now, little information has been readily available on how this important facet of the journal publishing process should be conducted properly. Peer Review and Manuscript Management in Scientific Journals fills this gap and provides clear guidance on all aspects of peer review, from manuscript submission to final decision.