3 ms·

> Many breakthrough areas struggled along for years with little funding, while a ton of money went into dead ends. Eg immunotherapy in cancer, H. pylori as the

> Many breakthrough areas struggled along for years with little funding, while a ton of money went into dead ends. Eg immunotherapy in cancer, H. pylori as the major cause of gastric ulcers... This looks an awful lot like survivor bias. Tons of money was also put into well established research to move it along. And ideas that were stupid also didn't get any funding and struggled. > If your system encourages or rewards unethical behaviour that is a big problem. That's markets and that's why we have a justice system. A reason why the NIH now requires grant applications to be of publishable quality to help diminish completely turnaround in experimental design after grant have been issued. A part of the evolution that I speak of. > the idea of 'good scientists' scrutinising a proposal carefully and controlling for all these factors is a bit fantastical. I doubt you want to do away with peer review. I'll be the first to say that It's not perfect, but it's gotten us pretty far. "Good scientists" have their orthodoxy and "know" certain things to be to true and won't (most of the time) fund anything that goes against that orthodoxy. I.e. one funeral at a time. Well funded PIs have the ability to fund this type of science and many do. I think we're pretty much in agreement though. The system is slow, not great, but like another commenter suggested, the best we can do.

mattkrause 9y ago
> This looks an awful lot like survivor bias. I'm not sure it's survivorship bias. H. pylori, for example, actually is a major cause of gastric ulcers. If we were actually good at identifying good ideas, Marshall and Warren would have had a pretty steady rise to fame over the course of their careers since they had, in fact, figured it out. For what I've heard, they did not--they were treated pretty shabbily for a while, then the field did an about-face and they won a Nobel prize. It's not the case that anyone who (somehow) survived that long gets a Nobel! > I doubt you want to do away with peer review. I don't think anyone wants to completely do away with the idea of having others sanity-check their research. However, there is a huge gap between peer review as an abstract concept and peer review as currently practiced in NIH study sections. Personally, I think that there's a lot of room for improvement in the latter. For example, you could imagine giving program managers more room to actually manage a portfolio of research and (e.g.) hedge across multiple hypotheses.
- tertius 9y ago
  > I'm not sure it's survivorship bias. I'm referring to good ideas being underfunded for a time. Only good ideas survive. Bad ideas obviously don't get to a point where they turn into something accepted and widely applicable (I'm sure there's a countercase or two...) So only the good survive. Does that mean we need to fund across the board? That's just not feasible, so we have some good ideas that go unfunded, but a whole lot of bad ideas that do not. > manage a portfolio of research and (e.g.) hedge across multiple hypotheses. I see this happening in well funded labs. Taking a bit of money from here and some from there in things that are novel, but ethically related enough to be able to be justified. But the ethics involved makes this dangerous. You bring up good points.

mattkrause 9y ago

> This looks an awful lot like survivor bias. I'm not sure it's survivorship bias. H. pylori, for example, actually is a major cause of gastric ulcers. If we were actually good at identifying good ideas, Marshall and Warren would have had a pretty steady rise to fame over the course of their careers since they had, in fact, figured it out. For what I've heard, they did not--they were treated pretty shabbily for a while, then the field did an about-face and they won a Nobel prize. It's not the case that anyone who (somehow) survived that long gets a Nobel! > I doubt you want to do away with peer review. I don't think anyone wants to completely do away with the idea of having others sanity-check their research. However, there is a huge gap between peer review as an abstract concept and peer review as currently practiced in NIH study sections. Personally, I think that there's a lot of room for improvement in the latter. For example, you could imagine giving program managers more room to actually manage a portfolio of research and (e.g.) hedge across multiple hypotheses.

tertius 9y ago
> I'm not sure it's survivorship bias. I'm referring to good ideas being underfunded for a time. Only good ideas survive. Bad ideas obviously don't get to a point where they turn into something accepted and widely applicable (I'm sure there's a countercase or two...) So only the good survive. Does that mean we need to fund across the board? That's just not feasible, so we have some good ideas that go unfunded, but a whole lot of bad ideas that do not. > manage a portfolio of research and (e.g.) hedge across multiple hypotheses. I see this happening in well funded labs. Taking a bit of money from here and some from there in things that are novel, but ethically related enough to be able to be justified. But the ethics involved makes this dangerous. You bring up good points.