Thursday, February 23, 2012

Panelicious

My recent trip was to serve on an NSF panel. After 2 years away one tends to forget certain things, and by "forget" I mean "block out". Like how... interesting... serving on a panel can be.

The funding rates in my directorate are abysmally low. Think barely over 10%, which means that we were able to fund N proposals, where N is a teeny tiny integer, out of nearly 10N submitted proposals. I found it truly depressing.

How the discussions proceed varies from panel to panel, in my experience, and has to do with the program director a fair bit. Some are very hands-off, some pretty intrusive. The one on whose panel I just served was pretty hands-off and let the panel dynamics develop on its own...

Generally, panels run for two days. The first day is spent going over each individual proposal, with each panelist summarizing what they thought of the proposal and why (programs in my main division usually don't use external ad hoc reviewers; the panelists are the reviewers). Each panelist reviews a certain number of proposals, how many depends on the number of submissions and the number of panelists. Each proposal must have at least three reports, but often they have more.  For every proposal, one person who typically did not serve as a reviewer, serves as the panel scribe and takes notes on the panel discussion regarding said proposal. By the end of the first day, it is pretty clear who the top contenders for funding are and who has no chance, with a small number of proposals as "maybe". Because the funding rates have always been low ever since I became a professor, I do not recall ever having seen a proposal being promoted from a "maybe" to "highly recommended" (i.e. likely funded). The second day is spent on the final ranking -- deciding who among the top contenders actually gets funded -- and the clean-up of the panel summary statements.

The low funding rates have made the proposal review process seem stochastic. It is, but to a point. Namely, every time I have been on a panel, there has been one proposal that you could suspect would get funded just by flipping through it. It was just polished, everything was in its proper place, the figures were pretty, it was aesthetically quite pleasing... This outward attention to detail also manifested itself inside -- the proposal was meticulously crafted. In my experience, the most polished proposal always gets funded. When you run into it, you just recognize it -- it is the Great Polished One.

But then there are typically several more that have nearly everything: a great idea, preliminary results, a competent PI, a good plan, no major flaws in how the proposal was organized or written, no major qualms about anything technical... Of those, most of those will unfortunately not get funded...

Here is the stochastic part of the panel review process: the success of your proposal hinges on your panel. As you may know, the NSF panels are (completely or almost completely) different every time. In my directorate, it is unheard of to address the comments of a previous panel formally --  you can incorporate changes to your proposal, but nobody writes a summary of changes made in response to a previous panel.

This time, we could have easily recommended for funding 3N very good proposals. Unfortunately, only N will get funded, and the other 2N get tossed out and will try their luck next time, with the next panel. The worst destiny is that of the proposal ranked N+1 (I was in that situation not that long ago) -- close but no cigar. With the next panel, you may easily end up in the "not recommended" category entirely, the fact that you almost made it last time means very little (I suppose the program director could advocate for you, but I have never seen it).

So what decides who actually gets funded in this seemingly stochastic process? The panel composition and interpersonal dynamics.

1) You have absolutely zero chance of funding unless someone on the panel decides to champion your proposal. How do you ensure this happens? Obviously, your work should be interesting, relevant for the program you submitted to (talk to the program directors if you are unsure where to submit and what types of proposals they want), and well written. This is essentially all you can control. Other than that, you can suggest potential reviewers and those you would like excluded, but I am not sure how much program directors look at these suggestions. It probably varies from director to director, but my guess is they probably at least take a look, even if they don't actually invite the suggested researchers. It is even better if more than one person championa your proposal.

2) You have zero chance of funding unless the person championing your proposal is doing their task well. This means you have to provide them with enough ammunition to fight off the attacks of other panelists (who are championing other proposals). And this is under the assumption that your champion is knowledgeable enough to be able to fight for it with real arguments and that they have the tenacity and energy to persevere. Which brings me to...

3) You will not get funding if the person championing  your proposal is too nice/meek. The panel dynamics is always the same -- there is always one loudmouth who attempts to have the last say on everything and is supposedly an expert in everything. Too often have I seen nice people back down because someone boorish and aggressive has made them start to doubt a good proposal; the aggressor may not even have a point, but the aggression, combined with the human propensity to back down when the sliver of doubt emerges have been known to reduce the funding chances of perfectly decent projects. Also, your champion may be defeated in a sheer battle of wills -- some people simply don't like confrontation, and will back off because they don't feel it's worth their time and aggravation.

This is what has always aggravated me -- the proposal championed by the biggest a$$hole on the panel generally gets funded because most people are nice. We were down to N+1 proposals, and we needed to decide which one to drop. They were all perfectly nice proposals and any N of them should have received funding. I decided I would make sure the proposal I was backing got funding; you have to argue, you have to show some teeth. After my two-year break, I find that I can be as big of an a$$hole as anyone else.  This is a new development, I was not nearly as combative previously; I guess I am a big girl now. I must admit, and this is totally shallow of me, I kinda enjoyed the fighting and then getting my way.

4) You should not worry too much about points 1-3 as you have no control over them. Just make sure you write the best proposals you can and volunteer to sit on review panels to improve your grant-writing skills. Without a doubt, sitting on panels is the best way to learn how to write grants. It helps you see what people respond to, what others in your field propose to do, how good proposals are organized... You get to learn something new and you get to flex your debating muscles. It's good fun.









8 comments:

Anonymous said...

it is the Great Polished One.

It always surprises me how American scientists have become so comfortable with notion that her funding bodies (NSF/NIH/ect) main purpose is to determine whom among all distinguished scientists have the best spell checker and grammar editors and fund those.

Anonymous said...

After my two-year break, I find that I can be as big of an a$$hole as anyone else. This is a new development, I was not nearly as combative previously; I guess I am a big girl now. I must admit, and this is totally shallow of me, I kinda enjoyed the fighting and then getting my way.

Haha I have to admit I truly enjoyed reading this...I get a savage pleasure out of scaring 6-feet tall guys. Never without good reason, but if someone does deserve to be told off and they happen to be a tall guy, I get a bit of an immature pleasure from being a tiny girl that scared a super big dude lol. Yes, this is stereotypical and immature but don't worry - I never attack unless provoked.

GMP said...

Anon at 11:14 PM, it's the same (well-documented) psychological mechanism that leads us to subconsciously ascribe more technical competence to people who look well-put together. I don't think it's a US phenomenon.
Well-polished proposals tend to be well-written and easy to read, simply a signature that the PIs invested considerable time and thought into it. With the current funding rates, you have to worry about everything, it is really not just about the idea. Lots of smart people have lots of good ideas. But, if you threw the proposal together in one week (and trust me, it shows), you will not get funded, no matter what you propose. I suppose if people proposed something really earth-shattering, the form may be disregarded, but I suspect those cases are really, really few and far between.

Anonymous said...

GMP, thanks for writing about the panel process. As a newbie NSF proposal writer, I found your post very insightful.

Anonymous said...

"Polish" definitely doesn't come down to just spelling and grammar. I write very well---all the reviews I get on my papers say so---but even I can see that my proposals don't have the "polish" that the best ones do. I have not yet figured out the art that goes into proposal writing the way I have with paper writing.

I agree with most of what GMP says, but I would add one more thing (at least for my directorate): the proposals that the panel "highly recommends" are not necessarily the ones that will be funded. NSF has a broader mandate than just funding the best science, and sometimes a less highly-recommended proposal will be funded for other reasons. I got one funded with good-but-not-great ratings because I'm in an EPSCoR state. This stuff doesn't come up in the panel discussions though.

EliRabett said...

Very true, just two small additions:

First, as a panel member you have to decide if there is one proposal you really really really think should be done. Having done that you have no chance of pushing a second one and trying to do so will hurt the chances of the one you love.

Second, damn it proposers, read the goddamn call for proposals and PUNCH ALL THE BUTTONS, if I am going to champion your great idea, please don;t let someone else point out that you left out a key item like the postdoc mentoring plan or the data preservation part.

Third, N+! ain't totally awful, the program managers are good at scrounging in the Foundation for a few extra $$.

inBetween said...

First off, I am so impressed that you are sleeping well enough and had time to read all the proposals before tha panel! I served on one when I was about 5 months preg and it was really, really hard to have my wits about me. In my program we serve for 2-3 years and then rotate off, so there is a bit of continuity (which is good and bad).

I agree with si much of this -- the power of arbitrary things is kind of scary. Iam amazed that I haver ever gotten fundingnow that I've seen how it works from the other side.

Anonymous said...

It's more than just polished or spell-checked: the top proposals have an air of inevitability that says that this is the obvious next thing to do. I was struck by this on one panel I served on because we had proposed to do the same thing the year before and had been turned down. Even though their team faced the same technical challenges we did (and were in fact behind in technical development), their proposal better glossed over the difficulties to emphasize the essential need to address the work. I wish I knew how to bottle that.