My recent trip was to serve on an NSF panel. After 2 years away one tends to forget certain things, and by "forget" I mean "block out". Like how... interesting... serving on a panel can be.
The funding rates in my directorate are abysmally low. Think barely over 10%, which means that we were able to fund N proposals, where N is a teeny tiny integer, out of nearly 10N submitted proposals. I found it truly depressing.
How the discussions proceed varies from panel to panel, in my experience, and has to do with the program director a fair bit. Some are very hands-off, some pretty intrusive. The one on whose panel I just served was pretty hands-off and let the panel dynamics develop on its own...
Generally, panels run for two days. The first day is spent going over each individual proposal, with each panelist summarizing what they thought of the proposal and why (programs in my main division usually don't use external ad hoc reviewers; the panelists are the reviewers). Each panelist reviews a certain number of proposals, how many depends on the number of submissions and the number of panelists. Each proposal must have at least three reports, but often they have more. For every proposal, one person who typically did not serve as a reviewer, serves as the panel scribe and takes notes on the panel discussion regarding said proposal. By the end of the first day, it is pretty clear who the top contenders for funding are and who has no chance, with a small number of proposals as "maybe". Because the funding rates have always been low ever since I became a professor, I do not recall ever having seen a proposal being promoted from a "maybe" to "highly recommended" (i.e. likely funded). The second day is spent on the final ranking -- deciding who among the top contenders actually gets funded -- and the clean-up of the panel summary statements.
The low funding rates have made the proposal review process seem stochastic. It is, but to a point. Namely, every time I have been on a panel, there has been one proposal that you could suspect would get funded just by flipping through it. It was just polished, everything was in its proper place, the figures were pretty, it was aesthetically quite pleasing... This outward attention to detail also manifested itself inside -- the proposal was meticulously crafted. In my experience, the most polished proposal always gets funded. When you run into it, you just recognize it -- it is the Great Polished One.
But then there are typically several more that have nearly everything: a great idea, preliminary results, a competent PI, a good plan, no major flaws in how the proposal was organized or written, no major qualms about anything technical... Of those, most of those will unfortunately not get funded...
Here is the stochastic part of the panel review process: the success of your proposal hinges on your panel. As you may know, the NSF panels are (completely or almost completely) different every time. In my directorate, it is unheard of to address the comments of a previous panel formally -- you can incorporate changes to your proposal, but nobody writes a summary of changes made in response to a previous panel.
This time, we could have easily recommended for funding 3N very good proposals. Unfortunately, only N will get funded, and the other 2N get tossed out and will try their luck next time, with the next panel. The worst destiny is that of the proposal ranked N+1 (I was in that situation not that long ago) -- close but no cigar. With the next panel, you may easily end up in the "not recommended" category entirely, the fact that you almost made it last time means very little (I suppose the program director could advocate for you, but I have never seen it).
So what decides who actually gets funded in this seemingly stochastic process? The panel composition and interpersonal dynamics.
1) You have absolutely zero chance of funding unless someone on the panel decides to champion your proposal. How do you ensure this happens? Obviously, your work should be interesting, relevant for the program you submitted to (talk to the program directors if you are unsure where to submit and what types of proposals they want), and well written. This is essentially all you can control. Other than that, you can suggest potential reviewers and those you would like excluded, but I am not sure how much program directors look at these suggestions. It probably varies from director to director, but my guess is they probably at least take a look, even if they don't actually invite the suggested researchers. It is even better if more than one person championa your proposal.
2) You have zero chance of funding unless the person championing your proposal is doing their task well. This means you have to provide them with enough ammunition to fight off the attacks of other panelists (who are championing other proposals). And this is under the assumption that your champion is knowledgeable enough to be able to fight for it with real arguments and that they have the tenacity and energy to persevere. Which brings me to...
3) You will not get funding if the person championing your proposal is too nice/meek. The panel dynamics is always the same -- there is always one loudmouth who attempts to have the last say on everything and is supposedly an expert in everything. Too often have I seen nice people back down because someone boorish and aggressive has made them start to doubt a good proposal; the aggressor may not even have a point, but the aggression, combined with the human propensity to back down when the sliver of doubt emerges have been known to reduce the funding chances of perfectly decent projects. Also, your champion may be defeated in a sheer battle of wills -- some people simply don't like confrontation, and will back off because they don't feel it's worth their time and aggravation.
This is what has always aggravated me -- the proposal championed by the biggest a$$hole on the panel generally gets funded because most people are nice. We were down to N+1 proposals, and we needed to decide which one to drop. They were all perfectly nice proposals and any N of them should have received funding. I decided I would make sure the proposal I was backing got funding; you have to argue, you have to show some teeth. After my two-year break, I find that I can be as big of an a$$hole as anyone else. This is a new development, I was not nearly as combative previously; I guess I am a big girl now. I must admit, and this is totally shallow of me, I kinda enjoyed the fighting and then getting my way.
4) You should not worry too much about points 1-3 as you have no control over them. Just make sure you write the best proposals you can and volunteer to sit on review panels to improve your grant-writing skills. Without a doubt, sitting on panels is the best way to learn how to write grants. It helps you see what people respond to, what others in your field propose to do, how good proposals are organized... You get to learn something new and you get to flex your debating muscles. It's good fun.