4 ms·

This problem has been solved long ago. Screen all grant applications to weed out the flawed approaches (the bottom 25%) and then run a lottery on the remaining.

This problem has been solved long ago. Screen all grant applications to weed out the flawed approaches (the bottom 25%) and then run a lottery on the remaining. If there is nothing to game, the gaming stops.

bonoboTP 6y ago
But it goes way deeper than just grants. Who will get tenure for example? Citations and publication counts are used for many many things. Also, making your findings sexy is not just done to improve metrics for promotion and grants. Sexy research also draws attention among experts and lay media alike. They will be invited to give talks at conferences and seminars etc. more if their findings are exciting.
- danieltillett 6y ago
  The same approach can be applied to tenure. Set a basic threshold that someone needs to meet, then run a lottery to chose who gets tenure. Would end an amazing amount of unproductive gaming of the system.
  - bonoboTP 6y ago
    This would work if people are roughly equally good above that low bar. But there are vast differences. The current metrics aren't perfect but they do correlate with ability. And as I said, you can't randomize everything, like talk invitations or "followers", collaboration offers, research stay offers at prestigious places, media exposure etc.
    - danieltillett 6y ago
      Not very closely based on my personal experience. I am not sure that science needs more of the activities on your list.
      - bonoboTP 6y ago
        Ultimately, most people are driven by a striving for social status and prestige. This is very general and includes money, power, freedom, intellectual influence, access to interesting people to work with and discuss, being taken seriously by intelligent co-experts, being a valued and respected member of the community, whose insights people look out for etc. This is taboo to say explicitly of course, one must say that it's all internal passion, but people, whether they admit it or not, will have to strive for these things. If they don't, they fall behind. This goes way way deeper than grants. It's not even just human nature, but a fundamental property of life. The same is true in dating. You have to present yourself in the best possible way, and those who do have better success. It's not enough that you and your ideas are good, you have to be able to convince people that you are worth investing in. You can't forbid people to associate, to form cliques, to keep tabs on who is good and who isn't (and the same for departments and universities) etc, to invite each other to work together. A grant that led to a sexy result will be considered money better spent, than something that leads to no presentable conclusions. And beyond the gaming, there are genuine differences in how good a research vision some people have. Top, visionary people vs mediocre incrementalist copycats is a real distinction.
sokoloff 6y ago
Why wouldn’t the gaming morph to stuffing the lottery? Submit similar grants with different PIs, different combinations of researchers, slightly different focuses or plans.
- danieltillett 6y ago
  You can limit how many grants people are allowed to put in. This already exists with the current broken system.
dwohnitmok 6y ago
Apart from the issue of grant stuffing, I have a strong suspicion that 25% is an extremely low number and even among those grant applications that pass obvious flaws there is still an extremely large variance in quality. To put it another way, imagine if a strategy for funding software development was to take every piece of software that compiled and then randomly choose among them to fund. This seems like it's ripe for exploitaton and hard to fund meaningful software development (especially coherent ecosystems). Similar dynamics would probably be at play for research.
- danieltillett 6y ago
  Having personally read through a large number of grant applications I would say 25% is about accurate. It really doesn’t change anything if we were to raise the threshold to say the top 50%. The key to success is to remove the human element of trying to determine what is a top 5% grant from a top 10% grant. This has been proven to be impossible. You can stop the grant stuffing gaming by limiting the number of grants people can apply for every year.
  - evandijk70 6y ago
    Don't forget that the people writing the grants are heavily influenced by what is required to obtain a grant. If the aim of writing a grant becomes to not be in the bottom 25%, than the average quality of grants will go down. I think it has not been proven to be impossible to determine the difference between a top 5% and a top 10% grant, it has just been proven that the current system does not do that. In trainings for grant applications it is often mentioned that members of grant committees spend extremely little time per grant proposal, which I imagine is just sufficient to weed out the bottom 25% proposals (those that contain obvious flaws)
    - danieltillett 6y ago
      It doesn’t matter if the average quality of the grants goes down (it would actually be a good thing if it reflected less time being spent on grant writing), what matters if the average quality of the science goes up. Grants are a means to an end, not an end in themselves (administrators feel different about this I will admit). It has been proven that grant reviewers can’t agree amongst themselves what are the top 5% verses the top 10%. Having personally gone through the process as a reviewer I can identify the difference between a good and a bad grant, but I can’t with any reliability determine the difference between a top 5% and a top 10% grant. Once you get into the top 25% of grants the ranking gets very arbitrary. I might like one grant more than another purely because I like the area more, but my taste should not be the criteria on which grants are chosen. The crazy thing is all of this is known by everyone involved in the grant process yet everyone continues on as though the system is working.

bonoboTP 6y ago

But it goes way deeper than just grants. Who will get tenure for example? Citations and publication counts are used for many many things. Also, making your findings sexy is not just done to improve metrics for promotion and grants. Sexy research also draws attention among experts and lay media alike. They will be invited to give talks at conferences and seminars etc. more if their findings are exciting.

danieltillett 6y ago
The same approach can be applied to tenure. Set a basic threshold that someone needs to meet, then run a lottery to chose who gets tenure. Would end an amazing amount of unproductive gaming of the system.
- bonoboTP 6y ago
  This would work if people are roughly equally good above that low bar. But there are vast differences. The current metrics aren't perfect but they do correlate with ability. And as I said, you can't randomize everything, like talk invitations or "followers", collaboration offers, research stay offers at prestigious places, media exposure etc.
  - danieltillett 6y ago
    Not very closely based on my personal experience. I am not sure that science needs more of the activities on your list.
    - bonoboTP 6y ago
      Ultimately, most people are driven by a striving for social status and prestige. This is very general and includes money, power, freedom, intellectual influence, access to interesting people to work with and discuss, being taken seriously by intelligent co-experts, being a valued and respected member of the community, whose insights people look out for etc. This is taboo to say explicitly of course, one must say that it's all internal passion, but people, whether they admit it or not, will have to strive for these things. If they don't, they fall behind. This goes way way deeper than grants. It's not even just human nature, but a fundamental property of life. The same is true in dating. You have to present yourself in the best possible way, and those who do have better success. It's not enough that you and your ideas are good, you have to be able to convince people that you are worth investing in. You can't forbid people to associate, to form cliques, to keep tabs on who is good and who isn't (and the same for departments and universities) etc, to invite each other to work together. A grant that led to a sexy result will be considered money better spent, than something that leads to no presentable conclusions. And beyond the gaming, there are genuine differences in how good a research vision some people have. Top, visionary people vs mediocre incrementalist copycats is a real distinction.

sokoloff 6y ago

Why wouldn’t the gaming morph to stuffing the lottery? Submit similar grants with different PIs, different combinations of researchers, slightly different focuses or plans.

danieltillett 6y ago
You can limit how many grants people are allowed to put in. This already exists with the current broken system.

dwohnitmok 6y ago

Apart from the issue of grant stuffing, I have a strong suspicion that 25% is an extremely low number and even among those grant applications that pass obvious flaws there is still an extremely large variance in quality. To put it another way, imagine if a strategy for funding software development was to take every piece of software that compiled and then randomly choose among them to fund. This seems like it's ripe for exploitaton and hard to fund meaningful software development (especially coherent ecosystems). Similar dynamics would probably be at play for research.

danieltillett 6y ago
Having personally read through a large number of grant applications I would say 25% is about accurate. It really doesn’t change anything if we were to raise the threshold to say the top 50%. The key to success is to remove the human element of trying to determine what is a top 5% grant from a top 10% grant. This has been proven to be impossible. You can stop the grant stuffing gaming by limiting the number of grants people can apply for every year.
- evandijk70 6y ago
  Don't forget that the people writing the grants are heavily influenced by what is required to obtain a grant. If the aim of writing a grant becomes to not be in the bottom 25%, than the average quality of grants will go down. I think it has not been proven to be impossible to determine the difference between a top 5% and a top 10% grant, it has just been proven that the current system does not do that. In trainings for grant applications it is often mentioned that members of grant committees spend extremely little time per grant proposal, which I imagine is just sufficient to weed out the bottom 25% proposals (those that contain obvious flaws)
  - danieltillett 6y ago
    It doesn’t matter if the average quality of the grants goes down (it would actually be a good thing if it reflected less time being spent on grant writing), what matters if the average quality of the science goes up. Grants are a means to an end, not an end in themselves (administrators feel different about this I will admit). It has been proven that grant reviewers can’t agree amongst themselves what are the top 5% verses the top 10%. Having personally gone through the process as a reviewer I can identify the difference between a good and a bad grant, but I can’t with any reliability determine the difference between a top 5% and a top 10% grant. Once you get into the top 25% of grants the ranking gets very arbitrary. I might like one grant more than another purely because I like the area more, but my taste should not be the criteria on which grants are chosen. The crazy thing is all of this is known by everyone involved in the grant process yet everyone continues on as though the system is working.