Thoughts on Peer-Review...
Friday, February 11, 2011 at 10:21AM Peer-review is often touted as one the quintessential reasons for why scientific knowledge is more reliable than that obtained from other mediums: Experts in the field are required to go over submitted manuscripts and vet both the quality of the data and arguments being presented within. But how is peer review performed in practice? Essentially, as a member of a scientific field, you are expected to review manuscripts that are sent to you by editors of scientific publications, pro bono. All in all, I don't think that most scientists would argue that this is not a worthwhile endeavor for the greater good of the field.
This being said however, in some discussions I've had with fellow scientists, I think that many feel that there is a bit of a 'free ride' problem when it comes to peer-review. Though peer-review is designed to weed out those manuscripts whose work and interpretations do not fit the standards of legitimate science, one would expect that most manuscripts will be good enough shape by submission time that the task of the reviewer will involve judging only the content itself. Unfortunately, this is not always the case.
Various factors lead to incomplete manuscripts being submitted. A principle offender is likely the ability to list 'submitted' manuscripts as achievements in grant and scholarship applications. As the deadline for such applications loom, there's a strong pressure to get any 'nearly' complete manuscripts into the hands of a journal. Assuming that such manuscripts are not rejected outright, these greatly increase the amount of work for a reviewer, because in addition to evaluating the science itself, you may now be trying to parse poorly written sentences and incorrectly labeled figures.
Oddly perhaps, there are no 'classes' where you learn how to properly peer-review manuscripts. I suppose it's one of those things that you're supposed to learn from your various mentors - though in my experience, standards for acceptance/rejection vary widely between different people (which is not a bad thing if you value a diversity of opinions). However, I've never encountered a situation where someone thought it was okay to simply reject a manuscript outright because it was incomprehensible as written, thus requiring tedious reading of obscure references and external sources in order to understand what was being argued.
It is my impression that the number of papers being published is out pacing the number of new Ph.D.s actively working in academic scientific disciplines. Thus the pool of reviewers per capita is shrinking. I worry that at some point either we're going to have to work hard to make sure that manuscripts are of excellent presentational quality prior to submission, or the hammer is going to have to fall and more papers are going to get rejected outright for even minor 'preparational' offenses. Scientists are generally busy enough already.
Reader Comments (6)
Interesting idea. Given how much time and effort is required to feel like one is adequately evaluating a paper that is riddled with grammatical mistakes, typos, misspelling, and incomprehensible figure captions (and just bad writing in general), I can certainly see there could be a cut-off point for saying "no, I won't review this until you proofread it". Where that cut-off point falls is probably something that varies from reviewer to reviewer, and on the mood of the individual reviewer. I know I'd be more willing to dig through the bad writing to look for the good science if I was adequately dosed with coffee, whereas had I missed my morning coffee I'd be more likely to throw the thing across the room. Setting a minimum writing-quality point seems like the job of the journal editor, not a barista.
Also, if there really is an increasing trend in the ratio of manuscript submissions to working PhD-level scientists, do you think there is also an increasing trend in the number of peer-reviewed publications required for a PhD student to graduate? I have been told by various professors, including my current and past advisors and members of my advisory committees, how many papers I should publish before I complete my PhD. The number ranges from "3, no more than 3. Do 3 and you're done!" to "at least 6 or 7, but some of those can be as second author", to "as many as possible". I get the impression in these discussions that many professors have a somewhat vague feeling that minimum standards, strictly in terms of quantity have risen in the past decade or two and they worry that there may be a trade-off operating with a decline in average quality of PhD-student-authored papers.
Regarding the first point, I don't know if things actually need to change, but perhaps there could be a small staff that proofs papers for quality? I doubt it though, many of these things are run on shoestring budgets. As far as I know however, it's generally poor form for a reviewer to simply toss back an article and say, "I couldn't even figure out what you were trying to say."
On the second point, I think that it's HIGHLY field specific. If you do a Ph.D. in genetics, you're very unlikely to get more than one or two 1st authorship papers out of it. On the other hand, if you're in bioinformatics, it may be expected. I've had more than one talk with professors about how some fields are simply vastly different in terms of how much they want each paper to 'show'. I've seen situations where people demonstrated a very cool and useful result (e.g., that an area upstream of several genes bound a transcription factor responsible for regulating some important process), only to have their paper rejected because the reviewers wanted them to perform another year's-plus worth of work characterizing all of the details involved. The point was that the author of the paper really thought (and had me convinced) that the additional work required justified another manuscript. On the other hand, I've seen the flipside where papers were published in such a poor state of basic null-hypothesis testing that the result presented within are of dubious merit.
The problem with that last point isn't simply philosophical however. It's recently been shown that retractions to bad papers often go unnoticed, and retracted studies can go on being cited for years. So there's some fluctuating state of balance that has to be found!
I suspect you are quite right about the variation between disciplines. However, I think this variation does not preclude a broad trend operating across all or the large majority of disciplines of increasing quantity requirements without strong regard for quality.
On the other hand, I have been a recipient of one of those reviewers simply tossing back the article with a shrug and a frown. In that specific incident, there probably was a useful middle ground between "this paper is unreadable" and "this reviewer is incompetent". We had an excellent editor in that process, and she took it upon herself to review our manuscript (in record time, on a weekend!) thoroughly and critically, absent a full review from a reviewer. In the end, her comments, along with the comments of the other reviewer, were very useful in helping us re-write our certainly not-as-good-as-it-should-be manuscript. The only comment made by that first, giving-up reviewer, involved the structure of the introduction, and helped to convince us to add a (probably needed) initial section with a more broadly-interesting approach to our specific topic.
At this point, I'll make the aside that such a change in a manuscript, adding more deep-background, is something I would expect you to never need to do, from what I've seen of your publications. :)
Moving on, can we discuss the rewards of peer-review, to the reviewer? You say it's done "pro bono", but I have 2 questions.
1. Is peer-review of some quantity (e.g. 5 papers / year minimum) part of the in-writing job duties of some or all working scientists? Do professors in science departments need to list the number of manuscripts reviewed during evaluations? Do post-docs list the journals they have reviewed for on their C.V.s when applying for jobs? If it's part of job duties, then I suppose one could consider some fraction of annual salary to be remuneration for peer-review.
2. Is there a non-financial benefit to peer-review? It's supposed to be anonymous, but it's deemed OK by many I've talked to to discuss simple numbers and in some cases journals (e.g. "last year I reviewed 2 for Evolution, and 1 for the European Journal of Evolution, and the editor of Ecology is sending me a manuscript next week"). Does peer-reviewing papers contribute to a scientist's reputation among his or her peers? Does it help with other parts of academic life, such as getting access to shared resources within a department or institution, or applying for leave from teaching duties?
Sorry to hear about your outright rejection letter. That's frustrating, unless of course the reason was 'I couldn't understand what was written', which doesn't seem to be the case. Re: your discussion questions:
I don't think that there's any sort of minimums - I just think that everyone's expected to take on as many reviews as they can handle, unless of course they're not qualified to review a paper. I'm a little over a year over a postdoc, and so far I'm doing well over 5/year. I've had PIs, who claimed to receive 1-2 per week, which seems crazy.
I don't know what the details are for career scientists, but my PI strongly recommends that postdocs and graduate students list reviews on their CV (my CV above hasn't been updated, which is something I have to do, along with listing research interests). Again, it's anonymous, but you can list the years and journals for which you've done reviews. I assume that once you're PI yourself, it's probably more important to list editorial positions (though again, I believe my PI lists reviews on his CV).
It wasn't really a rejection letter (I've recieved those, too), it was an apathy letter - the reviewer went most of the way towards telling the editor he (?) was unable to review our manuscript, but his reason was mostly "I don't care about this topic" rather than "I am not sufficiently familiar with the field". He copped out, in other words.
Being an editor for a journal is another component of peer-review, I suppose. While senior PhD students seem to be the earliest career-scientists to be asked to do peer-review (disclaimer: I've never done it formally, but I've participated in exercises designed to train me for peer review), one needs to be well-established in a field and therefore usually of academic rank of at least assistant professor (or some rough equivalent in a non-university scientific institution) before becoming a journal editor. How does that work, anyways? Do the other editors, or the chief editor, or a representative of the publisher, approach a person and ask if they'd like to be an editor? Does one typically act as editor for a special issue of a journal, or as assistant to another editor, first?
Bringing it back to the original topic, do you think journals should offer writing assistance to some or all of their authors? I know many journals offer assistance to English-as-second-language authors, but to me that seems more like "help you effectively translate good writing in Japanese/German/Spanish/Mandarin/etc. into English" and less like "good writing help". This seems like the responsibility of instructors who appear much earlier in a scientist's career, well before it's time to submit a sole-author paper - I'm thinking this conversation is wandering into teaching good (scientific and/or general) writing skills to undergraduates. I still like your idea of reviewers being allowed to hand back a submission with the simple statement "reject, poor writing". It's pretty general and vague, but if many working scientists really are recieviing multiple manuscripts to review every month, a way to streamline the workflow and eliminate the very worst is necessary.
Have you discussed these ideas with your current PI, or others who review so many manuscripts?
I think that you're right in assuming that senior editors of journals contact potential editors about taking that position. I've been contacted myself, actually, but turned it down. It may not have been the best career choice - but I just couldn't imagine taking on an additional several hours of work per week and staying sane. Oh well.
I haven't discussed these ideas in any more detail than broad passing comments, unfortunately. Journals only offer ESL writing assistance for a fee, as far as I know. W.r.t. to teaching writing skills, see the new post (2011 Feb 22).