A topic in research methodology
Peer review is the process by which articles (or book chapters, book manuscripts, book or journal proposals, funding applications, etc.) are scrutinised by experts in the field who comment on their strengths and weaknesses, recommend whether they should be published (or funded etc) and/or what changes might be required.
Peer review is a time-consuming process, when done well,
…Hooke had to study and comment on Newton's paper [on light and colour]. … He later admitted to spending no more than three hours on it."
Clegg, 2015
Usually peer review is (single) blind (the authors/proposers are not told who reviewed their work) or double blind (neither authors/proposer nor reviewers know the identify of the other).
Journal editors make decisions on accepted or rejecting article based on referee reports. Often referees recommend changes that are needed before an article can be published. Often revised/resubmitted manuscripts have to be tracked to highlight changes, and a detailed response is expected showing how referee comments have been addressed. Although the format of response is not usually specified, tabulating each reviewer comments with a response to show ow it has been addressed will suggest a systematic response that has given proper weight to all review comments, and helps editors and reviewers see how each point raised has been considered.
Sometimes authors are allowed to defend their original work against referee critique, but often they are expected to show that they have accommodated most, if not all, recommendations. Expectations vary from journal to journal. An editor is likely to expect authors to usually make revisions where referees see shortcomings in an empirical study. More latitude for genuine differences may be made in a more theoretical piece. Interdisciplinary journals may invite referees with a range of backgrounds, with the editor seeking to find a balance of views.
"The response to reviewers is of some use, but does not seem to have addressed most of the points raised previously. … It is disappointing that some of the issues raised in reviewing the original submission appear to have been ignored by the authors in preparing their review."
(Anonymous extract from peer review report seen on revised article submitted for publication)
Frustrations of peer review
Receiving peer review comments can be very frustrating as often an author feels that reviews may not fully understand the background to the work, or appreciate the nature of the methodology, or know enough about the specific topic of the study. Sometimes this may be a case of the author(s) not having sufficiently explained themselves, or made explicit assumptions that the authors are taking for granted (in writing, as in teaching, it is very easy to assume others take for granted what we do) ; sometimes it reflects the genuine challenge of finding suitable peer reviewers who have relevant expertise and who are available to carry out reviews (that is, prepared to put precious time and effort into reading and evaluating someone else's work rather than getting on with their own!)
The sense of frustration that authors may feel about peer review comments is reflected in the aside made here in describing how one (single) critic characterised Einstein's theories of relativity:
Delivering the keynote address [for "a conference explicitly devoted to the denunciation of Einstein's theory of relativity"], Weyland argued alternatively that the theory was false; possibly true although limited in its understanding; almost certainly false; largely true but clearly plagiarized from his own work; definitely false; and ultimately too incoherent for anyone to be able to tell one way or the other (all of which is actually quite a lot like academic peer-review today)
Dicken, 2018
That said, there is not really a sensible alternative to peer review. As has been said about democracy, peer review is the worst system there is apart form all the others that have been tried!
Quality of peer review
The quality of peer review depends both upon the systems set up by a journal and the ability to identify reviewers who are experts in the topic of the submission; who are prepared to do the work (usually only for the prestige of being a reviewer, which is only really valuable to relatively new academics) for that journal (and there is little prestige in working for a journal that is not well regarded in its field); and who are prepared to put aside sufficient time to do through reviewing.
Sadly many journals with impressive titles and claims to careful peer review procedures are either not willing or able to undertaken robust peer review.
Read 'A failure of peer review'
Work cited:
- Clegg, B. (2015) Light Years. The extraordinary story of mankind's fascination with light. London: Icon.
- Dicken, P. (2018). Getting Science Wrong. Why the philosophy of science matters. London: Bloomsbury.
My introduction to educational research:
Taber, K. S. (2013). Classroom-based Research and Evidence-based Practice: An introduction (2nd ed.). London: Sage.
