5 ms·

Disclaimer: I have never been on a grant committee myself. I see two problems with grant committees. 1. Some of us are more theorists, others - engineers. Som

Disclaimer: I have never been on a grant committee myself. I see two problems with grant committees. 1. Some of us are more theorists, others - engineers. Some of us want to solve mysteries, others - practical problems. Which means that if the panel is composed primarily out of theorists then applications possibly yielding highly cited papers (solving general problems vs specific) are more likely to get funded. This also creates incentive to split the work up into as many papers as possible. Likewise, panel composed primarily out of engineers is likely to undervalue deep problem research without physical deliverables. I have absolutely zero idea how to adjust for both these cases, though. 2. Another problem is relevance/urgency. Some problems/proposals are always actual and can be polished and reapplied every year. Others can quickly get practically irrelevant or competing technology become de facto standard. E.g. analogue television (terrestrial), Magnetic Cassette (solve practical problems why those got phased out). Do we rank ones over the others or keep them in the same pool? I have no idea.

danieltillett 11y ago
I have actually been on grant committees before and the basic problem is you only have enough funding for 10% to 15% of the proposals. Almost all of the applications are really good and from good people. The reason for this is it is so much work (at least 6 to 8 weeks full time) to write a grant application that only the really good people put them in. On top of this the universities will “pre-review” all the grant applications internally before they go in to make sure that all the weak proposal are weeded out and all the obvious flaws removed. The end of this process is that almost every grant is really, really strong making it impossible to rank them consistently. The committee ends up ranking on trivia, or in the worse case on “old-boy” connections. We are asking these committees to do something that we know is impossible. If we can’t rank grant applications by quality then lets stop pretending we can. Just decide if they are strong and then put all the strong ones into a lottery and fund as many as we can.

danieltillett 11y ago

I have actually been on grant committees before and the basic problem is you only have enough funding for 10% to 15% of the proposals. Almost all of the applications are really good and from good people. The reason for this is it is so much work (at least 6 to 8 weeks full time) to write a grant application that only the really good people put them in. On top of this the universities will “pre-review” all the grant applications internally before they go in to make sure that all the weak proposal are weeded out and all the obvious flaws removed. The end of this process is that almost every grant is really, really strong making it impossible to rank them consistently. The committee ends up ranking on trivia, or in the worse case on “old-boy” connections. We are asking these committees to do something that we know is impossible. If we can’t rank grant applications by quality then lets stop pretending we can. Just decide if they are strong and then put all the strong ones into a lottery and fund as many as we can.