Tuesday, August 10, 2010

In Defense of Tardy Reviewers

Professor in Training has a cool post about how tardy reviewers are a pain in the butt of editors. She rightfully asks why someone would agree to referee a paper and then be several weeks late.

Being an author, I totally understand and agree. But, being a sometimes tardy reviewer myself, I feel I also need to speak in defense of my tardy brethren. We are very sorry to be late. We really mean it. We should certainly write an email to the editor, saying we need additional time. We actually say yes to a request to review because, upon cursory inspection of the paper, it appears interesting and within out field of expertise. We don't get much (if any) benefit from reviewing a massive number of papers every year, but we value journal review as an important service and are willing to put in the time. However, even with the best of intentions, life intervenes, and because paper review is a service activity with no recognition for the reviewer, it often gets demoted as people need to reprioritize. And boy, do we need to reprioritize!

Here's an example: I was recently on vacation and took a couple of papers to review with me. During the vacation, I got several more requests, of which I declined all but two. One was for a short paper which I reviewed promptly and one was for a longer paper, and I left it for after the vacation. When I came back from vacation, there was a major "fire" that needed putting out: I am a co-PI on a big, multi-PI grant that is up for renewal. There was a ton of work that had to be done over that very week on the grant proposal, because two big players were going to leave the country the following week, one of them for a whole year. So no paper review happened that week despite my intentions, but neither did some other important things, such as meeting with several of my students who got stuck with their projects while I was away.

Among comments to PiT's post, Dr. Girlfriend says it is just pain rude to say yes and then stress everyone else by not delivering in a timely manner. I agree, but...

Consider this instead — if people said yes to a review request only if they were 100% certain they could deliver by a given date, the editor would likely have to ask 20 people to get 2 to say yes and deliver; that alone would take a very large amount of time. Wouldn’t you rather have an occassional reviewer late? Remember that reviewing is an important service, but also a time-consuming activity that does not benefit the reviewer so yes it will get lower priority when more urgent stuff comes up. Ironically, when you are a reliable reviewer, editors tend to pile on you, so you end up being late. Personally, I would much rather my paper be reviewed by the editor's 1st or 2nd choice, even if the review is late, than the editor's 15th choice, because by that time we are likely moving far from the desired expertise of the reviewer. (See Odyssey's comment, who speaks from an editorial perspective.)

When I was a student I was always upset about late reviews of my papers; now that I review non-stop, I am no longer upset at people being late as I know what's going on. I am that reviewer who writes you a two-page report. Actually, I spend much more time on reports for unfavorably reviewed papers than the favorably reviewed ones, because it's important to give people something to work with to improve their papers. I try to complete my reviews in a timely fashion, but sometimes it ends up not being timely. Such is life. In fact, when I look at how long it takes me to get reviews back and how long I take to return them, I feel that the duration of the actual technical review correlates with the size of the paper (shorter papers review faster), time of year (high season for travel or not, grant proposal deadlines), and the quality of writing (if the paper is poorly written, my opinion is that it can delay the review, as the reviewer will likely try reading it several times, get pissed each time and just leave the manuscript for later. I do this, and so do several of my colleagues. So writing a nice readable paper will also increase your chances of getting the review back promptly.)
The review duration also correlates with the seniority of the reviewer (big shots get more review requests and are overall busier); to enhance the review speed, in some journals in my field, editors are encouraged to recruit more junior people (e.g. postdocs) as reviewers for precisely this reason — they are less busy and more likely to review in a timely fashion.

While I no longer wish all the worst to tardy reviewers, as I did as a student, I am still quite aware that publication speed makes a significant difference for the career prospects of students and postdocs. It is often favorable to choose a fast journal with prompt editorial attention over a sluggish one even at the expense of a couple of impact factor points. What does vary dramatically among journals is how long it takes the editor to make successful referrals and how prompt he/she is about making a decision once the reports are back; in my experience, editors who are practicing scientists (as opposed to career editors) as well as organized individuals have some of the best turnaround times I have seen. I am curious if this is universally true: dear readers, who do you think makes for an overall faster review process -- career editors or editors who are practicing scientists?


Dr.Girlfriend said...

I agree that reviewing a paper is of far more benefit to a postdoc or assistant professor than a more senior scientist. As a grad student I did not get nearly enough training in critically reading research papers. All those papers my boss stressed over could have doubled up as training for us students (the PI overseeing of course) if not for the overly strict confidentiality thing.

I believe the bigger problem is that it is a service rather than a profitable venture. Some journals offer discounts for reviewers, but this is not enough.

Tenure track professors are expected continue to fund their own salary by winning grants or teaching/performing administrative duties for the university. Many of these people got into the business because they wanted to do science, but so much of their time is taken up with teaching and administrative duties. If journals and/or institutions offered rewards to reviewers I suspect moral and productivity would increase across the board.

It is stressful for the person in a leadership position and relying on volunteers to help, who, because they are doing you and the community a favor, they feel less obligated to deliver. What is often forgotten is the person who is relying on you is often a volunteer too. The person who has taken on the leadership role cannot so easily walk away or re-prioritize when they realize they took on too much.

I think that practicing scientists can make efficient editors, but they will always be split between two jobs. A career editor has made the decision to stop practicing science and devote full-time to editing.

Dr. Postdoc said...

I'm pretty low on the career ladder at this point, so I think I've accepted the few opportunities I've had to review papers. When you decline to review is it possible to suggest other reviewers (i.e. a former student/junior colleague)?

GMP said...

Dr. G, thanks for your thoughts. It's true that it's hard to get stuff done when everyone is a volunteer.Editorial office of the American Physical Society (they publish several highly respected journals in all areas of physics) has started issuing awards for outstanding referees, which is a nice recognition.

Dr. Postdoc, it's definitely possible to suggest other reviewers, I think most editors welcome it. I certainly recommend my senior grad students and postdocs when their expertise is a good match. Some journals in my field allow for a joint review -- which is ideal for training students: the student and advisor write the review together, which enables the student to get the experience reviewing but it supervised in the process.

prodigal academic said...

I hear you on tardiness, GMP. I always mean to do my reviews in a timely manner, but sometimes something comes up, or it turns out the paper is less in my area than I thought from the title/abstract and it takes a much longer time to be thorough, or the paper is a total mess and takes a long time to review.

I agree with you that I spend way, way more time on mediocre papers than on good papers. Papers that fall below mediocre, I don't spend a huge amount of time on--I just give the major areas that need fixing before the paper can even reach acceptable and reject. I also have stopped editing for English grammatical and usage mistakes after I hit the first 3-5. Otherwise, it can take too long to review an otherwise good paper that just needs a pass through with a native English speaker (and I am not interested in doing that as an uninterested party).

When I submit a paper, I try to recommend postdocs or other young scientists, since I think they 1) get fewer requests (and might be more likely to be faster), 2) tend to be more thorough in reviewing, since they are new to the process, and 3) tend to be happy to be asked!

Like you, I've found that working scientists tend to be somewhat faster than pro editors, but not by a huge margin.

Nick said...

"(if the paper is poorly written, my opinion is that it can delay the review, as the reviewer will likely try reading it several times, get pissed each time and just leave the manuscript for later. I do this, and so do several of my colleagues. So writing a nice readable paper will also increase your chances of getting the review back promptly.)"

I could not agree more. I just went through this, in fact it was my first official review on my own (I've only reviewed papers with my advisors). I felt like a failure because I was so late, but it was so poorly written and referenced that I could not bear to pick it up. In the end I spent a good three days reviewing it, and thankfully 99% of the changes that I suggested were honored.

I have a question though... I was not given a chance to review the paper post changes. Is this normal? As an author I just assumed that my responses always went back to the reviewer for final approval, but I guess not? If the flaws are major enough, do I request to see the authors' response prior to final acceptance?

Alex said...

With poorly written papers, I have observed that if it is at the point where literally every sentence needs fixing then the science is pretty much always junk (or at least whatever I can decipher seems to be junk). I'm seriously thinking about having a rule to automatically reject a paper if the first 2 paragraphs need grammatical fixes in every single sentence.

My theory about this is that careful, conscientious people will find a way to make a paper readable. Even if English isn't their first language, they will use grammar-checking software, ask friends to review drafts, hire proof-readers, do whatever it takes. This may not result in a flawless paper, but it will result in a paper that can be read without constant aggravation. Generally, if the writing is competent but still has significant flaws, after a few paragraphs I stop noticing the "ESL voice" as my colleagues in the writing program call it, and I just notice the science.

Sloppy people, on the other hand, cannot be bothered to hire a proof-reader, consult with a friend, use a grammar check, or whatever.

Unbalanced Reaction said...

I think the recognition of reviewers is nice, but it still won't motivate all reviewers. I believe that for every reviewer out there like you, GMP, with a valid excuse for a tardy review, there is another who is just putting it off because they can. I think some reviewers forget how important a timely review is to a young faculty member on the tenure clock.

Female Computer Scientist said...

Interesting post(s). I've recently been in all three roles simultaneously and can completely empathize with the frustrations of each.

As an author, when I'm told by an editor that my reviews will be back January 15th and it's August 15th, I'm not feeling the love.

As a reviewer, I get the review invitation, check when the deadline is, then inevitably submit the review the day before it's due... when things are going well. When things are going poorly and fires are breaking out, then it's submitted late. I feel bad about it, but my own research gets priority. (And, frankly, I find at least what some journals do totally dishonest. They invite you to review and say, for example, "Your review is due 30 days from when you accept." BUT - if you wait one week and then go to the website to accept, your review is now due in 21 days. So really they mean "30 days from now." Totally nuts.)

As a PC member, I've had to delay notification because of reviewers who said, "I'll get the review back to you this weekend, I promise" and then never do. Or reviewers who accept an invitation to review and then fall of the planet. Meanwhile authors are emailing asking for status updates, publishers are asking for the list of accepted papers, etc. It's a horrible place to be in, and then you are up emailing every person you ever met trying to get them to review the paper. You have to pull in all these favors, and it's just rotten.

Not sure if career editors would help any of this muckity muck, at least in my subfield. Research in this area requires a lot of very specialized knowledge, and it's hard to imagine someone keeping on top of things who isn't a practicing scientist.

However, a career editor who could pre-screen/pre-edit the papers with horrible grammar, spelling, and inability to draw figures coherently would be extremely helpful for sure.

Anonymous said...

Wow, that's a cynical view, Unbalanced Reaction! While I'm sure there may be a FEW people who put off reviewing papers just because they can ("let's let them sweat for a few more weeks, Mwahahaha") I think that most reviewers have good intentions and just get sidetracked by one of the three hundred other things going on each week.

r.e. Nick, manuscripts usually don't go back to the reviewers, unless MAJOR revisions were required. That's why you take the time to write a nice letter to the editor describing your response to each of the reviewers' comments. It allows the editor to make an executive decision on the manuscript.

In my experience, journals with scientists as editors have a faster turnaround time, but I'd imagine it's field specific.

Unbalanced Reaction said...

I'm a pretty cynical prof, Anon.

While I'm with ya that many reviewers "get sidetracked by one of the three hundred other things going on," I don't believe that they all have good intentions. Except for you, of course.

Mizumi said...

Thanks for your perspective, GMP. I'm glad you're back!

Are there opinions on what is "fair" number of reviews to accept? Over the past few years as a PhD student and then postdoc I've maybe drawn 8-10 reviews per year. Does that mean 8-10 is a good number to take on, so at least I am not a net drain on reviewer resources?

GMP said...

Thanks everyone for the comments.

Nick and Mizumi asked a couple of important questions. I started responding but the responses got too long, so I decided on a follow-up post.