4 ms·

I think that, in a sense, a) would have been the right solution here. While I'm not sure this is the case in all fields, at least in mine, when you apply for

I think that, in a sense, a) would have been the right solution here. While I'm not sure this is the case in all fields, at least in mine, when you apply for grants, people too much about how famous you already are and what you have published. People don't care about how many grants you already have, or how much money you burned through to produce your previous publications. This is bad. First, it creates the illusion that there are more scientists out there than can be sustained by grant funding, when in reality the problem isn't lack of funding, but the fact that the funding gets concentrated in the hands of the highly successful instead of being evenly distributed to the highly and moderately successful. Thus, pursuing a career in science becomes risky because you might not be able to get a job (as GP stated), and so people decide they don't want to do science. Second, it's inefficient. A professor at my institution employs 50 people and produces about 1 paper per year. 15 moderately successful professors with 2 people working under each of them could do much better than that. But he has a Nobel prize and they don't. Finally, it's horrible for the people in these labs. This is where accountability comes in. A professor with a small lab is accountable for the success of each of his students/postdocs. With larger labs, professors can allow students to fail, because their success is no longer necessary for success of the labs. Moreover, professors run out of ideas and waste their trainees' time. (I have heard rumors that a different Nobel prize winning scientist hired multiple people to work on the same project, not as collaborators, but as competitors. The person who won got a high-impact publication. The person who lost got nothing.) There's no obvious solution here, since science can't be funded completely algorithmically, but to start I think that funding agencies should be requiring professors to disclose both the number of personnel they employ and their grant history when they apply for further grants, and take this into account during the review process.

bokonist 14y ago
Who exactly decides how to allocate these grants? Professional federal government bureaucrats? Fellow professors in the field? Do you have any insight on why they choose to allocate the grants in such an inefficient way? Is the classic "no one ever gets blamed for buying the name brand" dynamic (where the famous professor is the "name" brand)? Or is it more sinister insider dealing and back scratching?
- simonster 14y ago
  I've never participated in the grant process, so I don't have the full information here, but I do know that grants are scored by other professors in the field. Part of the problem is that the grants are numerically scored, and this scoring is primarily based on the merit of the proposal and the publication record of the lab. The lab budget isn't taken into account, and no one actually checks to make sure the project hasn't already been funded (see http://www.nytimes.com/2013/02/05/science/study-suggests-duplicate-financing-of-projects.html http://www.nytimes.com/2013/02/05/science/study-suggests-dup...). There is additional funding for young investigators, but otherwise, the funding agencies treat a process like they are funding a project, when in reality they are also funding a lab. The assessment is with regard to scientific impact, not scientific impact per dollar. The "no one ever gets blamed for buying the name brand" dynamic is probably a factor, but I doubt it's a very big one. If you win a Nobel, it seems that you're guaranteed grants for life, but otherwise, your research still gets defunded if you don't produce anything for a while, even if you were once at the forefront of your field. People are probably more likely to vote to give grants to their friends, but I don't think there's systematic manipulation.
  - bokonist 14y ago
    Interesting, thank you for sharing. I think it's a major problem that there is no external accountability. The grants are scored by peer professors, based on publications (which again requires approval by peers). This creates a bunch of problems 1) it's very easy for entire field or subfield to go down the wrong track, and have no way of getting out of that track because the professors controlling grants and publications are the ones doing the bad research. 2) there is a bias towards publishing research that shows some new finding, rather than research that shows that they could not re-produce some previous science. The entire idea of basing science off of peer review is erroneous IMO, science should advance via replication. In my ideal world, research would be funded in three, independent streams: 1) Prize based. The government would allocate a yearly budget of some very large amount of money - say $30 billion. It would create competitions and prizes in all sorts of fields. Example contests might include: self-driving car race, robot soccer olympics, longest battery life for $x of total materials used for making the battery, most yield of a plant strain per acre, etc. Teams would receive money for their research in return for placing or winning the contests. Research teams would also get money for achieving objective milestones: 2) Commission based. The government needs some new technology, maybe an upgrade of stealth tech for their fighter planes, so it pays a research institute to develop that tech. Unlink grant based funding, the commission is granted by the outsiders who need the tech, not by professors in the field giving to it to fellow professors. 3) Open Ended Institutional Funding. A certain number of reputable institutions should simply be given permanent annual funding. No grant process, no paperwork, no bureaucracy. The researchers would be overseen by an executive director, and the executive director would be overseen by a board of directors. The board of directors would consist of outside experts with a proven track record. For example, the board of a computer science research university would have people like Google's Urs Hölzle or Guido or Trevor Blackwell. The researchers would not have to apply for grants. They would just have to convince the board of directors that they were generally working on projects that had potential benefit to humanity, or else they would get fired. Every institution would be actually independent, governed by the personal judgement of that institutions board of directors.
vwinsyee 14y ago
A quick comment about point 2: A lot of academia is about quantity over quality these days. One high-impact, well-designed paper with strong conclusions could very well be worth more than 50 weaker papers. Unfortunately these days, many people, especially those outside of the field, may perceive the 50 weaker-paper lab(s) to be more "efficient" than the 1 strong-paper lab.
- simonster 14y ago
  > One high-impact, well-designed paper with strong conclusions could very well be worth more than 50 weaker papers. This is true. Assessing impact is difficult, and journal impact factor is not a great proxy. However, at least in my field, most Nature and Science papers aren't worth 50 second- or even third-tier papers.
  - XorNot 14y ago
    In my field (chemistry) plenty of people utterly loathe high-impact factor journal papers. They do something that looks impressive, but are a useless source of detail on _how_ it was actually accomplished, since more often then not they omit - not intentionally really - something which turns out to be very important to the process. Several simpler papers not trying to be as dramatic are going to be far more insightful since you get the descriptions of what was found _not_ to work about the method they were trying to replicate. Weaker papers - to me - are gold, since they're what you publish when you were trying to copy something and it just doesn't work as well as is being claimed.
  - grest 14y ago
    Yes, the risk-taking strategy of working towards a potentially high-impact publication is not worth the reward (as measured in citation rate comparing low-impact and high-impact articles )- see Foster et al Tradition and Innovation in Scientists Research Strategies http://arxiv.org/pdf/1302.6906.pdf http://arxiv.org/pdf/1302.6906.pdf

bokonist 14y ago

Who exactly decides how to allocate these grants? Professional federal government bureaucrats? Fellow professors in the field? Do you have any insight on why they choose to allocate the grants in such an inefficient way? Is the classic "no one ever gets blamed for buying the name brand" dynamic (where the famous professor is the "name" brand)? Or is it more sinister insider dealing and back scratching?

simonster 14y ago
I've never participated in the grant process, so I don't have the full information here, but I do know that grants are scored by other professors in the field. Part of the problem is that the grants are numerically scored, and this scoring is primarily based on the merit of the proposal and the publication record of the lab. The lab budget isn't taken into account, and no one actually checks to make sure the project hasn't already been funded (see http://www.nytimes.com/2013/02/05/science/study-suggests-duplicate-financing-of-projects.html http://www.nytimes.com/2013/02/05/science/study-suggests-dup...). There is additional funding for young investigators, but otherwise, the funding agencies treat a process like they are funding a project, when in reality they are also funding a lab. The assessment is with regard to scientific impact, not scientific impact per dollar. The "no one ever gets blamed for buying the name brand" dynamic is probably a factor, but I doubt it's a very big one. If you win a Nobel, it seems that you're guaranteed grants for life, but otherwise, your research still gets defunded if you don't produce anything for a while, even if you were once at the forefront of your field. People are probably more likely to vote to give grants to their friends, but I don't think there's systematic manipulation.
- bokonist 14y ago
  Interesting, thank you for sharing. I think it's a major problem that there is no external accountability. The grants are scored by peer professors, based on publications (which again requires approval by peers). This creates a bunch of problems 1) it's very easy for entire field or subfield to go down the wrong track, and have no way of getting out of that track because the professors controlling grants and publications are the ones doing the bad research. 2) there is a bias towards publishing research that shows some new finding, rather than research that shows that they could not re-produce some previous science. The entire idea of basing science off of peer review is erroneous IMO, science should advance via replication. In my ideal world, research would be funded in three, independent streams: 1) Prize based. The government would allocate a yearly budget of some very large amount of money - say $30 billion. It would create competitions and prizes in all sorts of fields. Example contests might include: self-driving car race, robot soccer olympics, longest battery life for $x of total materials used for making the battery, most yield of a plant strain per acre, etc. Teams would receive money for their research in return for placing or winning the contests. Research teams would also get money for achieving objective milestones: 2) Commission based. The government needs some new technology, maybe an upgrade of stealth tech for their fighter planes, so it pays a research institute to develop that tech. Unlink grant based funding, the commission is granted by the outsiders who need the tech, not by professors in the field giving to it to fellow professors. 3) Open Ended Institutional Funding. A certain number of reputable institutions should simply be given permanent annual funding. No grant process, no paperwork, no bureaucracy. The researchers would be overseen by an executive director, and the executive director would be overseen by a board of directors. The board of directors would consist of outside experts with a proven track record. For example, the board of a computer science research university would have people like Google's Urs Hölzle or Guido or Trevor Blackwell. The researchers would not have to apply for grants. They would just have to convince the board of directors that they were generally working on projects that had potential benefit to humanity, or else they would get fired. Every institution would be actually independent, governed by the personal judgement of that institutions board of directors.

vwinsyee 14y ago

A quick comment about point 2: A lot of academia is about quantity over quality these days. One high-impact, well-designed paper with strong conclusions could very well be worth more than 50 weaker papers. Unfortunately these days, many people, especially those outside of the field, may perceive the 50 weaker-paper lab(s) to be more "efficient" than the 1 strong-paper lab.

simonster 14y ago
> One high-impact, well-designed paper with strong conclusions could very well be worth more than 50 weaker papers. This is true. Assessing impact is difficult, and journal impact factor is not a great proxy. However, at least in my field, most Nature and Science papers aren't worth 50 second- or even third-tier papers.
- XorNot 14y ago
  In my field (chemistry) plenty of people utterly loathe high-impact factor journal papers. They do something that looks impressive, but are a useless source of detail on _how_ it was actually accomplished, since more often then not they omit - not intentionally really - something which turns out to be very important to the process. Several simpler papers not trying to be as dramatic are going to be far more insightful since you get the descriptions of what was found _not_ to work about the method they were trying to replicate. Weaker papers - to me - are gold, since they're what you publish when you were trying to copy something and it just doesn't work as well as is being claimed.
- grest 14y ago
  Yes, the risk-taking strategy of working towards a potentially high-impact publication is not worth the reward (as measured in citation rate comparing low-impact and high-impact articles )- see Foster et al Tradition and Innovation in Scientists Research Strategies http://arxiv.org/pdf/1302.6906.pdf http://arxiv.org/pdf/1302.6906.pdf