3 ms·

It's difficult to systematically predict Karikos. Probably impossible with very low success rates. A possible way forward is to simply give out grants randomly

It's difficult to systematically predict Karikos. Probably impossible with very low success rates. A possible way forward is to simply give out grants randomly to all those that pass some minimum score. After enough filtering, all grants are essentially indistinguishable in quality.

tsumnia 5y ago
> those that pass some minimum score The issue though becomes that the score gradually stops being minimal as more and more requirements get added in over time
sxg 5y ago
I think this is an underrated idea in many domains. So many different things, including grant funding, university admissions, etc. should use a minimum threshold criteria with acceptances randomly granted to those meeting the criteria. The problem right now is that there are too many qualified people, leading to a rat race in which everyone's working for that extra 1% to set them apart from everyone else. But in reality, the standardized exams/scoring systems we use aren't sensitive enough to reliably differentiate people precisely. This leads to unneeded stress and wasted energy on trying to game the system to get that extra 1%. Definitely a controversial idea and would pose huge challenges to the system we have now. We'd have to confront the idea that a Harvard pedigree or an R01 NIH grant aren't quite the strong indicators of success that we think they are. We’d also have to consider that brilliance can be found in places we haven’t looked more often than we think.
- deleted 5y ago
  [deleted]
- ImaCake 5y ago
  I wonder if not randomly selecting after applying a quality filter is actually leading to worse outcomes? It is a well known in stock markets that it is often better to just randomly drop money in than to try and time the market. Maybe the politiking and bias of grant selection committees is picking the wrong half of the standard normal distribution?
zozbot234 5y ago
If you could systematically predict Karikó's, they would not be Karikó's. For all we know, any attempt to "solve" this problem might just be a fool's errand. At least in the absence of a meaningful argument to the contrary. > After enough filtering, all grants are essentially indistinguishable in quality. I don't believe this. You can always soften incentives (and thereby prevent wasteful gaming) by adding random noise/dithering to your scoring function, but a threshold-only approach seems quite blunt to me.
- derbOac 5y ago
  Actually, this model has been suggested by former heads of NIH and NSF, who have publicly expressed concern about the current funding paradigm. The problem is that impact by reasonable metrics is correlated about 0.30 with grant score, so it's difficult to predict what will be successful. (Note that impact itself is controversial, given self-fulfilling fad dynamics in research -- FOMO popularity spikes and so forth). One of the biggest predictors of grant success is having previously published with people on the review committees. The idea is that you randomly fund research because it's difficult to know what will be important, but try to balance that against grants that might be poorly conceived to begin with.
- cycomanic 5y ago
  > > After enough filtering, all grants are essentially indistinguishable in quality. > I don't believe this. You can always soften incentives (and thereby prevent wasteful gaming) by adding random noise/dithering to your scoring function, but a threshold-only approach seems quite blunt to me. These assessments come from statistically investigating grant scoring and success. What you find that 90%of grants are essentially scored the same, within the noise. In these situations what typically happens is that previous grant success will determine current success, i.e. the person previously got a grant so what they are doing must be good (the "safe" bet). The problem that creates that you create empires of people who get a lot of grants based essentially on one luck of the draw. The biggest issue is though not even that you miss some good people with good ideas, but the incentive structure you create. Because there is an expiry time on the benefit from previous success and it is research area specific, what happens is academics have to write grants constantly and you can't really change fields. Academics spend 30% of their time applying for and reviewing grants. None (hardly any) of the people who are very successful with getting grants are doing research themselves any more. I suspect this is also the reason why most Nobel winning research is done in people's 30s.more senior academics don't have time to do the research (and can't take the necessary risks).
Al-Khwarizmi 5y ago
I read this proposal of randomness often. It would work assuming researchers were robots, but I don't think it would really work with humans. It's just not aligned with basic human psychology and incentives. Why would a researcher strive to come up with a really good proposal if as long as it's somewhat decent, the outcome is random? Why would they strive to execute it well if it won't influence future outcomes? In fact, why would anyone want to be an academic researcher in the first place if funding depends on a lottery, and no matter what you do, you might find yourself without any funds to do anything meaningful? I think a system where grants are only given to a small minority is defective and wastes everyone's time, no matter how the minority is chosen. The solution has to be to give some reasonable funding to anyone employed as a researcher - which, depending on how much you value science, can be achieved by increasing the funding or by decreasing the number of researchers (I'm an academic myself so I believe in the former, but I might be biased).
- dariosalvi78 5y ago
  All researchers know very well that grants are assigned somewhat randomly. I mean bad science is usually filtered out, then there is good marketing, which shouldn't count but it does, and the rest basically depends on some random evaluation from some random person. It's a lottery. I support the idea of having a sort of 3 tiers: crap, excellent and good enough. The third one could be, at least partially, be assigned randomly.
  - Al-Khwarizmi 5y ago
    Yes, but psychologically speaking, the difference between "somewhat randomly" and "totally randomly" is huge. With the latter, you lose the incentives, because it doesn't matter what you do. In capitalism, there is a lot of randomness on who gets rich, much more than in grant reviewing, and while capitalism can be criticised from many angles, even the staunchest anti-capitalists wouldn't deny that it's great at incentivising people to try to make money... The variant you mention would be better than pure randomness, but at least for me, it wouldn't work either. First of all, in order to become "excellent" at research you typically have to start at "good enough", so the "excellent" category would also be hugely conditioned by the randomness - how do you build up an excellent CV if you repeatedly lose the grant lottery? Secondly, although incentives wouldn't be totally lost, for me it still wouldn't work psychologically. I can come to terms with the fact that there is a large random factor in grant reviewing, it's not something I like but that's just life and very difficult to avoid, but I wouldn't be able to accept seeing the researcher next door scoring a grant while I'm left without it due to a pure lottery, to be honest. I think I would go to industry. PS: regarding what you mention about marketing, something I would do would be to evaluate people (CVs) rather than project proposals. Proposals are indeed very, very biased by marketing and by how boldly researchers can make grandiose claims (which is likely to be correlated negatively with research integrity). And in CS they often don't even make sense... for example I work in NLP, large language models like BERT came in 2017-2018 and revolutionised the whole field, if I had followed the text of my ongoing grants to the letter I would have been doing obsolete research. (I can see how in other fields like physics, etc., this may not apply).
fuzzfactor 5y ago
Some people are just better at predicting the future than others.
kmonad 5y ago
Why even ask for a grant proposal? Require some evidence that the applicant is serious, and then let luck decide. You might be suggesting this already (but your last sentence read to me as if you're thinking of vetting proposals). Obviously not all funding should be allocated like this, but maybe a small fraction. Perhaps only to those without any grant support at the time. I think it could be a good hedging strategy for society.

tsumnia 5y ago

> those that pass some minimum score The issue though becomes that the score gradually stops being minimal as more and more requirements get added in over time

sxg 5y ago

I think this is an underrated idea in many domains. So many different things, including grant funding, university admissions, etc. should use a minimum threshold criteria with acceptances randomly granted to those meeting the criteria. The problem right now is that there are too many qualified people, leading to a rat race in which everyone's working for that extra 1% to set them apart from everyone else. But in reality, the standardized exams/scoring systems we use aren't sensitive enough to reliably differentiate people precisely. This leads to unneeded stress and wasted energy on trying to game the system to get that extra 1%. Definitely a controversial idea and would pose huge challenges to the system we have now. We'd have to confront the idea that a Harvard pedigree or an R01 NIH grant aren't quite the strong indicators of success that we think they are. We’d also have to consider that brilliance can be found in places we haven’t looked more often than we think.

deleted 5y ago
[deleted]
ImaCake 5y ago
I wonder if not randomly selecting after applying a quality filter is actually leading to worse outcomes? It is a well known in stock markets that it is often better to just randomly drop money in than to try and time the market. Maybe the politiking and bias of grant selection committees is picking the wrong half of the standard normal distribution?

zozbot234 5y ago

If you could systematically predict Karikó's, they would not be Karikó's. For all we know, any attempt to "solve" this problem might just be a fool's errand. At least in the absence of a meaningful argument to the contrary. > After enough filtering, all grants are essentially indistinguishable in quality. I don't believe this. You can always soften incentives (and thereby prevent wasteful gaming) by adding random noise/dithering to your scoring function, but a threshold-only approach seems quite blunt to me.

derbOac 5y ago
Actually, this model has been suggested by former heads of NIH and NSF, who have publicly expressed concern about the current funding paradigm. The problem is that impact by reasonable metrics is correlated about 0.30 with grant score, so it's difficult to predict what will be successful. (Note that impact itself is controversial, given self-fulfilling fad dynamics in research -- FOMO popularity spikes and so forth). One of the biggest predictors of grant success is having previously published with people on the review committees. The idea is that you randomly fund research because it's difficult to know what will be important, but try to balance that against grants that might be poorly conceived to begin with.
cycomanic 5y ago
> > After enough filtering, all grants are essentially indistinguishable in quality. > I don't believe this. You can always soften incentives (and thereby prevent wasteful gaming) by adding random noise/dithering to your scoring function, but a threshold-only approach seems quite blunt to me. These assessments come from statistically investigating grant scoring and success. What you find that 90%of grants are essentially scored the same, within the noise. In these situations what typically happens is that previous grant success will determine current success, i.e. the person previously got a grant so what they are doing must be good (the "safe" bet). The problem that creates that you create empires of people who get a lot of grants based essentially on one luck of the draw. The biggest issue is though not even that you miss some good people with good ideas, but the incentive structure you create. Because there is an expiry time on the benefit from previous success and it is research area specific, what happens is academics have to write grants constantly and you can't really change fields. Academics spend 30% of their time applying for and reviewing grants. None (hardly any) of the people who are very successful with getting grants are doing research themselves any more. I suspect this is also the reason why most Nobel winning research is done in people's 30s.more senior academics don't have time to do the research (and can't take the necessary risks).

Al-Khwarizmi 5y ago

I read this proposal of randomness often. It would work assuming researchers were robots, but I don't think it would really work with humans. It's just not aligned with basic human psychology and incentives. Why would a researcher strive to come up with a really good proposal if as long as it's somewhat decent, the outcome is random? Why would they strive to execute it well if it won't influence future outcomes? In fact, why would anyone want to be an academic researcher in the first place if funding depends on a lottery, and no matter what you do, you might find yourself without any funds to do anything meaningful? I think a system where grants are only given to a small minority is defective and wastes everyone's time, no matter how the minority is chosen. The solution has to be to give some reasonable funding to anyone employed as a researcher - which, depending on how much you value science, can be achieved by increasing the funding or by decreasing the number of researchers (I'm an academic myself so I believe in the former, but I might be biased).

dariosalvi78 5y ago
All researchers know very well that grants are assigned somewhat randomly. I mean bad science is usually filtered out, then there is good marketing, which shouldn't count but it does, and the rest basically depends on some random evaluation from some random person. It's a lottery. I support the idea of having a sort of 3 tiers: crap, excellent and good enough. The third one could be, at least partially, be assigned randomly.
- Al-Khwarizmi 5y ago
  Yes, but psychologically speaking, the difference between "somewhat randomly" and "totally randomly" is huge. With the latter, you lose the incentives, because it doesn't matter what you do. In capitalism, there is a lot of randomness on who gets rich, much more than in grant reviewing, and while capitalism can be criticised from many angles, even the staunchest anti-capitalists wouldn't deny that it's great at incentivising people to try to make money... The variant you mention would be better than pure randomness, but at least for me, it wouldn't work either. First of all, in order to become "excellent" at research you typically have to start at "good enough", so the "excellent" category would also be hugely conditioned by the randomness - how do you build up an excellent CV if you repeatedly lose the grant lottery? Secondly, although incentives wouldn't be totally lost, for me it still wouldn't work psychologically. I can come to terms with the fact that there is a large random factor in grant reviewing, it's not something I like but that's just life and very difficult to avoid, but I wouldn't be able to accept seeing the researcher next door scoring a grant while I'm left without it due to a pure lottery, to be honest. I think I would go to industry. PS: regarding what you mention about marketing, something I would do would be to evaluate people (CVs) rather than project proposals. Proposals are indeed very, very biased by marketing and by how boldly researchers can make grandiose claims (which is likely to be correlated negatively with research integrity). And in CS they often don't even make sense... for example I work in NLP, large language models like BERT came in 2017-2018 and revolutionised the whole field, if I had followed the text of my ongoing grants to the letter I would have been doing obsolete research. (I can see how in other fields like physics, etc., this may not apply).

fuzzfactor 5y ago

Some people are just better at predicting the future than others.

kmonad 5y ago

Why even ask for a grant proposal? Require some evidence that the applicant is serious, and then let luck decide. You might be suggesting this already (but your last sentence read to me as if you're thinking of vetting proposals). Obviously not all funding should be allocated like this, but maybe a small fraction. Perhaps only to those without any grant support at the time. I think it could be a good hedging strategy for society.