Thursday, August 12, 2010

Manuscript Review Questions

Among comments to the previous post, Nick and Mizumi asked two questions that warrant a longer response.

Nick asked: I was not given a chance to review the paper post changes. Is this normal? As an author I just assumed that my responses always went back to the reviewer for final approval, but I guess not? If the flaws are major enough, do I request to see the authors' response prior to final acceptance?

I am speaking here from an author's perspective, and some of the responses may be field specific (I am in a hard STEM field). Those of you who are editors or simply in a different field, please chime in and correct/add.

If all referees requested minor revisions, it is likely that the editor will just look over the resubmission letter and make a decision without a second round of review. If the review featured a mix of minor and major revision requests, the resubmission can go back to all reviewers or it can only go back to those who requested major revisions (whereas they are sometimes asked to make sure the authors considered all the suggestions in all the reports). Indeed, when you request mandatory revisions, the resubmitted paper will most likely come back to you for a second round of review.

However, it is possble for you, as a reviewer, to request mandatory revisions and still not get the paper back. The scenarios in which this happens are the following:

(a) the other referee convincingly argues that the paper is beyond salvation, so the editor rejects it flat out and does not encourage resubmission;

(b) the other referee requires major revisions or recommends rejection, the editor does encourage resubmission, but the authors decide the reviews are so negative that it is not worth their time to try to fight it and never resubmit. They may go to another journal, but if they are smart they will take your review into account before submitting anywhere else.

(c) same as (b), but where the authors decide to wait a couple of months to resubmit. Due to the delay, the paper is considered a new submission and assigned a new set of reviewers;

(d) your review was so negative that the authors argued you were needlessly hostile and/or not objective, and they requested that the paper not be sent to you any more but that another referee be chosen. The editor was convinced and complied, so you as a referee are supplanted at resubmission. (FSP has a post that describes this scenario.)

I would say that (a), (b), and (c) are fairly common. I don't know if (d) has ever happened to me as a reviewer, but I have been on a team where we were the authors in a (d) scenario; once or twice we actually did get another reviewer.

As for final acceptance, the decision is ultimately in the hands of the editor. You as a reviewer are really advisory to the editor, and editors usually follow the reviewers' recommendations, but the final decision is the editors'. I am sure, though, that if a decision has been made on the paper you reviwed, you can shoot the editor an email and ask for a status update. I might have done that once or twice, and the response each time was that the authors never resubmitted.

@ Mizumi: Are there opinions on what is "fair" number of reviews to accept? Over the past few years as a PhD student and then postdoc I've maybe drawn 8-10 reviews per year. Does that mean 8-10 is a good number to take on, so at least I am not a net drain on reviewer resources?

Mizumi, how many papers to review per year is really a question of how much time you have and how much you like reviewing papers. It's an activity that is fun to do every so often (after all, it's the fun science you get to read and help improve), but it does become a chore if you are under constant siege. One paper a month, which is approximately what you have been doing, is likely feasible even for very busy people. Would you be able to do one review every two weeks? How about one review per week? I think more than one a week (~ 50 per year) is difficult to accommodate. Unfortunately, the higher you are on the totem pole, the more requests you get per unit time -- easily multiple requests per week. My goal is to hover around 1/week, but I will usually end up having 2 or 3 on my desk at any point in time, and another one or two which I have delegated to my students or postdocs. For instance, I review papers from multiple journals, maybe 8-10 journals fairly regularly. So even if each journal sends you a paper once a month, that's still plenty.

As with most things in life, start on the lighter side -- 1 or 2 papers per month are certainly a good service to your community, and the load will likely not smother you.

I am curious to hear how much of a review load other people have (please state career stage) and, from our readers who are also editors, how much refereeing load they think is appropriate.


Nick said...

Thanks for taking the time to write a detailed response to my question GMP!

Anonymous said...

OMG - a paper a week? I'd go crazy. I do about 12-15 papers/large grant proposals per year. I'm coming up for tenure and promotion this year (SLAC).

I look at it this way - I shouldn't review more than 10X what I submit in a given year. However, if I were aiming to earn a position on an editorial board or a program officer term someday, then I would agree to do more.

Unbalanced Reaction said...

For a variety of reasons, I've completely switched research fields for my PUI position, so I've yet to publish in this "new world." Based on Anon's comment, my 0 publications x 10 = 0 reviews a year. Should I be concerned about this?

How important is it to be a reviewer, particularly at a predominantly undergraduate institution? Anyone have opinions on this?

Mizumi said...

Thank you for picking up the question!

My initial phrasing may have been unclear. When looking for a "fair" number to review, I was wondering if it's reasonable to take the average number of reviews *received* and making sure to "give" at least that number.

As you point out, GMP, obviously there is a certain number that's fun and not a burden. But (as has been well documented) as fresh-TT faculty you want to be careful with how you spend your time. So it would be nice to have a target of what's reasonable so I don't have to feel guilty for refusing. :-)

In that light, Anonymous --- reviewing 10X the number you receive seems like an incredibly large number!

GMP said...

Mizumi, I'd say aim for maybe 1 or 2 per month, and reject accordingly. What I usually do when I receive a review request is look if it is in my area, and, if so, if I have time and if it looks interesting. (I must admit that I sometimes take on refereeing even if I don't have time or it's slightly outside my expertise, but the paper looks really interesting.) I have a few favorite journals and I am more likely to accept a referee request from them as I am also more likely to submit my papers to them. I don't think 10x the number of submitted papers is feasible for me, it would be way way too many. Again, I would say 1-2 per month is an OK rate for junior people.

I have a colleague who boasts about never taking on referee requests unless it's to help a friend; on the other hand, he often comments how referees are late or stupid or evil; my approach to refereeing is to treat authors as you would like to be treated -- which means you have to review sometimes and do a good and thorough job. It's an important service to the community.

How much refereeing is enough? This touches upon the question of Unbalanced Reaction: How important is it to be a reviewer, particularly at a predominantly undergraduate institution?

Well, from the standpoint of a CV/tenure package, refereeing counts as service plus speaks indirectly of your establishment as an expert: showing you review regularly for prominent journals in your field is a way of showing that you are a good citizen and that editors consider you as an established expert.
(I have a colleague who will accept the first review request from a journal he'd never reviewed for before, so he can put it on his CV. From there on, he rejects requests unless they are in his narrow specialty.) From the standpoint of your tenure case external letter writers, it looks really good if you are a reviewer for GlamourMag or high-impact papers is your field.

Bottom line: refereeing speaks of your service + recognition as an expert, so it's good to show you review for a variety of good journals in your field, but there is no need to kill yourself doing it. I cannot give a PUI/SLAC specific view, because I don't have that experience, but I imagine it's viewed in a similar way.

Anonymous said...

I'm the SLAC prof who caps my reviewing activities at not more than 10 times what I submit. That is, if I submit one research paper per year for publication, then I can review up to 10 papers per year (about one per month).

Service is generally more important at a SLAC, but at mine, service to the university and dept. has more weight than professional service. Then again, overall, I've been told that service is almost irrelevant at tenure-time at my school, compared to research and teaching quality. It's best to ask around to find out how service is evaluated specifically at your school.

I think professional service keeps you out there in your discipline has indirect benefits. I actually think it's just good karma to do a good job reviewing, if you want the same in return for your own work.

Anonymous said...

One paper per week is way too much.

10X what one submits is also a lot (at least for someone who is very active in research). I figure ~2 or 3X is plenty. Look at it this way: each paper I submit has 2 or 3 reviewers. So if all my papers were single-authored I would be doing as much work as my own papers required. In practice I have co-authors who also do reviews so collectively we are doing more than our share.

I am midcareer (just promoted to full prof) so I can accept the argument that I should do more than my share. I claim I am, according to the argument above. I try to redirect a lot of my review requests to postdocs and junior scientists who are much less overwhelmed and may actually want the chance to do the reviews.

prodigal academic said...

I don't take on more than 1 review per month if I can avoid it. I am way too busy to spend a lot of time on reviews that aren't directly relevant to my current work, so I just turn those down. After I get tenure, I'll probably do more, just because I kind of like reviewing. It is fun to be out there on the cutting edge.

arezo said...

last month a journal gave me a chance to ba reviewr,i accepted ,just because i was so eager on it,unfortunatly these days i suddenly became more busy than normal for the past few weeks,so i think i should inform them and turn it down,but i don't know if it's possible or not,and if yes how should i write?

Anonymous said...

the peer-review system is flawed, and based on the buddy system or "prestige" system. friends sit on the "boards" and select their friends, or alienate their enemies, steal ideas, or play the delay game. academic "publishing" is mainly a farce. spend massive time, $$$, energy, etc...for what really? hamster wheeling.