3 ms·
In the spirit of "betting is a tax on bullshit", this seems like an interesting opportunity to try it out. Why not find some way to make panelist rankings publ
by p4wnc6 11y ago
In the spirit of "betting is a tax on bullshit", this seems like an interesting opportunity to try it out.
Why not find some way to make panelist rankings public (or some function of panelist rankings)? If you serve as a panelist and you rank grant proposals highly that ultimately don't meet success criteria down the road (spawning high-profile publications, generating citations, coming in under-budget, whatever...) that should reflect badly upon you as a panelist, in a public way. And over time, your "score" as a panelist would modulate the extent to which your rankings are weighted in a panel discussion. Similar to how each major European football league gets a coefficient from UEFA, and based on the coefficients, the number of direct and playoff slots into the Champion's League tournament is decided.
Imagine if academics who want to serve on a panel must agree to also have such a 'coefficient' published about them. Now, their ranking is something they must openly bet their reputation upon, and over time those who place correct bets will be given more weight when panels select grants.
Obviously this isn't perfect. The coefficient scheme could be manipulated in the same way that network connections lead to manipulation of the rankings as it is now. If we don't trust the central body measuring and adjusting the coefficients, that would be a problem (cough... FIFA).
But still, wouldn't some version of this idea -- making academics pay a public reputation price in order to vote for their preferred funding recipients -- be better than letting people rank and vote without reputation effects? In theory, it should also mean that only those with a real stake in the decision will risk the reputation price to vote -- and you could imagine even beginning to open up grant funding decisions to much wider voting bodies. Instead of a small panel, just drop all of the proposals onto a site like arXiv, and allow absolutely anyone at all to vote -- so long as the weight of their vote corresponds to their public score, and that their future score will be affected by the success/failure of whatever they vote for. No more small / closed-off committees, just open speculative voting like a prediction market.
- nonbel 11y ago>"success criteria down the road (spawning high-profile publications, generating citations, coming in under-budget, whatever...)" I wouldn't handwave this aspect. Creating incentives to optimize the wrong thing can be worse than using an arbitrary filter. Your first two sound like they encourage hype, popularity contests, and quantity over quality. How about something combining reproducibility of analysis, availability/sharing of the data, precision of theoretical prediction, and consistency of quantitative estimates from independent replications?