3 ms·

Hmmm, I'm not so sure about this. With things like microarrays and other tech, sure there were promises that were too big. But the underlying idea (gene express

Hmmm, I'm not so sure about this. With things like microarrays and other tech, sure there were promises that were too big. But the underlying idea (gene expression matters, there are lots of genes, we should measure them all) is correct. To wit, RNA-seq has transformed cancer biology. And Nanostring is now offering a microarray-like technology with extremely impressive signal to noise. And we still use lots of tools initially developed for microarrays (Limma, for example). So I would argue the investment in microarray technology has absolutely paid off. I think you're throwing the baby out with the bath water to say that, because a few bad eggs are effectively salesman and not scientists, that we wasted "hundreds of billions" of dollars (i.e., a large fraction of the NIH budget). The fact is, most research doesn't hold up not because the researchers weren't honest, or because they were salesmen, but fundamentally, because science is hard and it's really hard to be right for the right reason in science. Alternatively, if you have a formula to determine, in real time, which results are hype and won't stand up vs. which are real and will, well, that would be something too.

Fede_V 7y ago
I don't have issues with the technologies, I have issues with the early papers that came out with 'gene signatures' based on maybe 40 or 50 patients. I think in general it's actually quite easy to tell which results are hype and which are not. Do a power analysis and estimate what kind of sample size would be required to measure the effect the researchers estimate - if they are off by an order of magnitude, reject.
dfeliej 7y ago
In my experience, there's a disproportionate reward associated with fashions in science. That is, the people at the forefront of a trend get rewarded professionally (tenure, positions, promotion), even when behaving in a way that is irresponsible theoretically or empirically. The skeptics, on the other hand, who carefully lay out the problems with the new research, and eventually point out the obvious flaws, get treated as sticks-in-the-mud, etc., even though they are eventually proven correct. There are gains to be made through innovation, and I'm not arguing against that. Sometimes too, details are just minor details to work out. But sometimes the details matter and point to bigger problems. The problems with fads in science are that the focus in academics seems sometimes to be on popularity per se rather than verdicality, and denialism about scientific fads as a phenomenon. This is pointed out in the article, and is one of its better points. Someone else criticized the article for not presenting solutions, which is fair, but I think that's in part because effective solutions would probably require a huge shift in culture in academics. We'd need to return to more stable professional positions rather than the cannabilistic postdoc system in place now, with declining tenure; we'd need to focus more on groups of researchers and findings rather than celebrity academics ala TED talks; we'd need to put much more emphasis on a pattern of replications rather than single studies; etc. etc. Even then, I don't think the problem would entirely go away.

Fede_V 7y ago

I don't have issues with the technologies, I have issues with the early papers that came out with 'gene signatures' based on maybe 40 or 50 patients. I think in general it's actually quite easy to tell which results are hype and which are not. Do a power analysis and estimate what kind of sample size would be required to measure the effect the researchers estimate - if they are off by an order of magnitude, reject.

dfeliej 7y ago

In my experience, there's a disproportionate reward associated with fashions in science. That is, the people at the forefront of a trend get rewarded professionally (tenure, positions, promotion), even when behaving in a way that is irresponsible theoretically or empirically. The skeptics, on the other hand, who carefully lay out the problems with the new research, and eventually point out the obvious flaws, get treated as sticks-in-the-mud, etc., even though they are eventually proven correct. There are gains to be made through innovation, and I'm not arguing against that. Sometimes too, details are just minor details to work out. But sometimes the details matter and point to bigger problems. The problems with fads in science are that the focus in academics seems sometimes to be on popularity per se rather than verdicality, and denialism about scientific fads as a phenomenon. This is pointed out in the article, and is one of its better points. Someone else criticized the article for not presenting solutions, which is fair, but I think that's in part because effective solutions would probably require a huge shift in culture in academics. We'd need to return to more stable professional positions rather than the cannabilistic postdoc system in place now, with declining tenure; we'd need to focus more on groups of researchers and findings rather than celebrity academics ala TED talks; we'd need to put much more emphasis on a pattern of replications rather than single studies; etc. etc. Even then, I don't think the problem would entirely go away.