Reviewer, reviewer, on the wall: Who is the fairest of them all?
As peer review week has arrived, I’ve been thinking about this process with both my editor and researcher hats on. I’ve been lucky enough to serve as the editor of a peer-reviewed journal for almost 15 years, and this accumulated experience has given me lots of insight into different kinds of peer review comments and feedback received by our journal. The caveat: It’s important to remember that peer reviewers are almost always often full-time academics who are commenting on papers written by other colleagues as part of their normal day-to-day work; in my experience, this kind of ‘community service’ is very often not taken into account by universities as a component of job descriptions, and so peer review is usually undertaken by researchers in their ‘spare time’.
In my ‘Snow White’ role as a journal editor, I reckon I can identify seven very commonly encountered kinds of peer reviewers (the ‘Dwarfs’).
The ‘cursory’ reviewer. This first kind of peer reviewer is very common and is of no use at all to an editor because of their very short, cursory comments such as ‘this paper is great’, or ‘please publish as is’. This is either someone who has agreed to review a paper but does not have the time or inclination to do a decent job, or this could be a set up by the author. The latter is much more common that you would think: about one-quarter of all reviews.
The ‘aggressive’ reviewer. This kind of peer reviewer just hates the paper in question (or the author) and so tries to bury the study. Comments may be aggressive or personal in nature, and so must be balanced by an editor against more reasonable views. Again, this type of review is much more common than you’d think, and is frequently encountered by younger up-and-coming authors.
The ‘in-depth’ thoughtful reviewer. This is the best kind of peer reviewer, beloved of both editors and authors. Comments in this case are thoughtful, in-depth, and provide constructive remarks that enable the author to improve their work. These are not as common as you might hope for, but are something we should all aim for when asked to conduct reviews.
The rarely encountered ‘jealous’ reviewer. In this case, a reviewer can actually be jealous of the authors of a paper, or feel that he/she has made such a significant contribution with their comments that their name should be added to the study before publication. I’ve seen this a few times, including colleagues refusing to review the same paper for the second time (after ‘major revisions’) unless they were added as an author. Push-back required.
The ‘late’ reviewer. Often a friendly and ‘quick-to-email’ colleague who is ‘doing the review right now’ or ‘will get this back to you as soon as possible’. It's all talk, and comments are almost never actually returned in my experience; these kinds of reviewers are best avoided (or passed over) as they can hold up papers. Editors will send papers to a range of colleagues to avoid such situations, and will then proceed with the first two or three reviews that come back (see below).
The ‘angry non-reviewer’. In rare cases, reviewers who have been asked to look at papers are then passed over in the journal process if they either take too long or because other reviewers have returned their comments more promptly (see above). People sometimes do get angry about being asked to do something only to subsequently see that invitation cancelled; editors will take this path infrequently, however, and generally for the reasons mentioned above (such as when the reviewer is taking an unreasonable amount of time to provide comments).
The ‘non-existent’ reviewer. I process upwards of 400 manuscripts per year, and there are always one or two cases per month of papers for which we simply cannot find anyone to review. This is a nightmare for an editor (ScholarOne: 0 active selections; 30 invited; 0 agreed; 30 declined; 0 returned) and we try not to hang on to papers for too long if we just cannot solicit any comments; ‘I’ve asked 20 colleagues and no-one feels able to provide a review of your study’ is one email template I have ready to send out. Authors appreciate honesty and sometimes these situations are created simply by inappropriate journal selection; however, simply not being able to find reviewers for a paper that meets the journal's mandate is not a good enough reason for rejection. That’s life.
Apart from the last of these examples, I freely admit to falling into all of these reviewer types at one point or other in my career, which I realise is not ideal. I do feel strongly that, when agreeing to review someone else’s work, it’s important to try to be: (i) constructive; (ii) positive; (iii) thoughtful, and; (iv) on time. We’ll return to these points in the next article in this series.