Tuesday, December 6, 2011


Recently, I have been editing a special issue of a journal in my field. The special issue focuses on the topics in which I am an expert, so I know virtually all the contributors (at least the group leaders) fairly well. When soliciting reviews, I tried to contact postdocs and junior faculty about as often as I did senior people. This being a special issue means I have had to deal with a large number of papers in a fairly short time span and on a relatively small number of topics, which has helped make some interesting patterns in the review process obvious.

My initial guess was that younger people, postdocs and junior faculty, would be less busy and would be more likely to accept to review, would accept/decline to review more promptly (this journal provides an accept/decline option), and would overall be submitting their reports faster than the ultrabusy senior folks. Here's what I found:

Bar none, the fastest review (and a very detailed, to-the-point one) was received from a well-known, well-respected, and presumably very busy senior person in the field. In fact, several of the very busy, well-known people, whom I was reluctant to even ask for review because they likely have multiple review requests from different journals on their desk at any point in time, were very prompt in accepting to review, and were either on time or just a tiny bit late with their reports. These reports were always professional, detailed, and very useful.

So we can conclude "When you want something done, get a busy person to do it," right? Hold on, not so fast. Unfortunately, among the people who took the longest to even respond that they would indeed review, senior faculty are also the most numerous. Clearly, when it comes to plotting the distribution of referee responsiveness, the senior-folks distribution is tail-heavy.

Overall, I am quite puzzled by how many people take forever to accept/decline to review. I have never understood that. You read the abstract, and then decide whether or not to review, and click on a link. How hard is that?

I don't mind people declining to review, but I really appreciate it if they do it promptly. Unfortunately, postdocs and junior faculty have actually been much less likely to accept a referral for papers clearly within their expertise than I had hoped. I know that junior people are not idle, quite the contrary. My guess is that they don't want to review because they feel there is no immediate benefit to their careers from review, but I think that is short-sighted. Becoming a well-regarded reviewer improves your standing in the field and it keeps you at the forefront of your specialty. (FSP wrote about it. ) Junior people should have a light review load, but I think it's important that they review. It's part of training and part of being a good member of the scientific community. Being a sloppy or disinterested reviewer, while presumably expecting others to pull their weight on your behalf, is certainly not going to enhance your standing in the editor's eyes.

There is one person who submitted two papers to the special issue. Yet, I have not been able to get a single report out of him after multiple reminders. First he took forever to accept, and now now he's again taking forever to submit that one review. I would think that it's only fair that, if you want others to review your papers (note the plural), you should pitch in yourself, shouldn't you?

Another interesting issue, which was also discussed by FSP here, is whether to indicate that the paper you are reviewing is missing citations to your own work. In the case of a special issue, we are dealing with a focused set of topics, so this issue came up several times. I have found that most people have no qualms about requiring that their papers be cited (and generally rightly so), even though when the list of papers to cite contains only those from a single group it becomes quite obvious who the reviewer is (perhaps up to the uncertainty coming from a multiperson author list).

We all want our papers reviewed promptly and thoroughly -- I cannot imagine anyone wanting their paper reviewed slowly or sloppily -- but many people provide exceedingly terse reviews or are so late that the referral has to be withdrawn. It's an interesting feature of the human psyche -- we fail to treat others the way we would like to be treated ourselves.


Schlupp said...

Do "your" reviewers never ask for papers by someone else to be cited? I do so quite often, so I know for a fact that drawing conclusions from the suggested citations can go wrong.

feMOMhist said...

OMG I'm in the special issue editing hell right now and the reviewers have caused more confusion and delay than I ever would have thought possible, and yes the most senior people were the most reasonable and speedy, so the same holds true in the humanities it seems.

Anonymous said...

Is there any consequence to being a horribly late reviewer? It's not like editors "fire" reviewers, do they?

studyzone said...

In my field, many of the postdocs I know work for PIs who must approve any reviews written by the postdoc (and often insist on being co-authors). Thus, for some postdocs, the time to respond, and their final answer, are entirely dependent on their PI. Is this true for your field?

NonUS FSP said...

Did the senior people write the reviews themselves or did they subcontract parts of it to junior graduate students?
Here's what I do: I ask a grad student (usually on their first year or two) to carefully read the paper. (Almost always, the paper is highly related to the research topic of the student, so understanding it in detail is worthwhile for them.)
I also read the paper myself, concentrating on the "high-order bits". In my field this means that I won't check the line-by-line derivation of inequalities, or other very technical details of proofs.
Then, I will meet with the student to discuss our impressions and comments.
After that, I will ask the student to write down a detailed list of their comments, add my high-level, general comments, evaluation and recommendation.

Anyway, this clearly makes it easier to send the review on time.

GMP said...

Schlupp, in this special issue I have seen several instances where the referee requests that, like, 5 papers from their own group be cited and nothing else. I find that unusual -- when I ask that my work be cited, it's usually because the authors ignored a larger body of work so references to the work of others are missing as well, and I request citations to those too.

Anon at 6:23: Nothing terrible happens if you are a bad reviewer, but the editor can "fire" you -- withdraw outstanding referrals and not send you new ones. A senior colleague who is an editor of an IoP journal says their system purges consistently late referees from the database.

I like my senior students and postdocs to own their reviews. I generally don't have them review unless I feel they are ready and have a good grasp of their field, so probably not before their third year. If they did review a paper I had been sent, I will let the journal know that I forwarded the paper and ask that the referral be formally transferred to the student/postdoc. That way, they begin to acquire their own reviewer footprint. I go over their reviews with them the first couple of times, but then they are on their own. A few times I submitted a "joint review" where I listed that I and student/postdoc jointly reviewed. Generally, when I submit a review as myself, I am actually responsible for the whole review.

Thansk for the comments, everyone!

pyrope said...

I completely agree with your exasperation about failure to accept/decline reviews. I'm an ed for a Wiley journal and the system doesn't move on to the next reviewer request unless the previous one has been declined. So, a couple of times it has hung there for weeks waiting for some reviewer who never bothered to decline. Seriously, you opened the email, decide and click the damn link!
As far as consequences, I've had a couple of reviewers never turn in reviews and I gave up and made the decision with 1. The consequence is that I never again ask them to review...which is probably to their benefit? Kind of like my teenage cousin once told me - his strategy for getting out of chores is to break something so that he never gets asked again. Nice!

Alex said...

In my field, many of the postdocs I know work for PIs who must approve any reviews written by the postdoc (and often insist on being co-authors).

Wait, what? If the journal sent the review request to the postdoc, why would anybody else have to approve it? Or are the PIs just control freaks?

Also, the fact that you refer to the person in charge as the "PI" rather than "advisor", "professor", "supervisor", or some other word strongly suggests that you are in biomed. And now I have another insight into how screwed-up biomed is.

Anonymous said...

(same anon)

Oh that's good that some journals at least attend to who the slow reviewers are. My field is maddening. Some journals promise timely reviews and are actually pretty consistent. Others, no so much. I am seriously considering tracking this and using it in decisions of where to submit. If two comparable journals have a big difference in review time I may just go with the quicker one.

Anthea said...

I'm with Anonymous above and I now only submit articles to some journals and not others. One article of mine which I co-authored and submitted over 18 months ago to a journal sat "buried under some papers" of the editor for an unknown (but long) period of time. Let's just say that it's the last article that I'm submitting to this journal.

GMP said...

Anon, Anthea, I agree -- journal turnaround times are not an insignificant factor in publishing. In my field, the impact factor (IF) of a journal is the most important metric, but for journals with comparable IFs, I definitely favor the faster ones, even if I have to take a slight hit in terms of the IF. Like Anthea, there are journals where my article would sit for months without anyone touching it; needless to say, I will never ever submit to those journals again.

gasstationwithoutpumps said...

This year I have sometimes been very slow to respond to a review request. Partly this is because I'm switching fields, and though I feel I ought to still be reviewing papers for my old field, the titles and abstracts have seemed extremely boring, so I've dithered about making a decision to review or not review. I know that this is bad, but it has been difficult for me to just say no.