3 ms·

I am as anti-pomo/critical theory as it gets, but, sadly, this phenomena is very much present in the hard sciences as well. For example, about 10-15 years ago,

by Fede_V 7y ago

I am as anti-pomo/critical theory as it gets, but, sadly, this phenomena is very much present in the hard sciences as well. For example, about 10-15 years ago, the big craze in biology was microarrays, and the early papers made crazy promises about everything that they would deliver. With hindsight, it turned out almost all the early papers were fatally flawed and were drastically statistically underpowered: it turns out that living systems have a huge amount of noise, making clean measurements is difficult, and things are very complicated. The same exact thing played out with microRNAs, epigenetics (chipSeq data is exceedingly noisy), metagenomics, single cell genomics, etc. Every single time you'd get 'trailblazers' who'd come into the field, do shitty rushed science, hype their results like crazy, and by the time people figured out that all the initial results were garbage, they'd have moved onto the new hot thing. I guess the advantage that hard sciences have over the humanities is that eventually we'd figure it out and do things properly - but the amount of money that has been wasted in shitty research that didn't add anything to human knowledge is probably in the hundreds of billions in the last 15 years alone.

api 7y ago
> Every single time you'd get 'trailblazers' who'd come into the field, do shitty rushed science, hype their results like crazy, and by the time people figured out that all the initial results were garbage, they'd have moved onto the new hot thing. Same thing happens in business with the latest startup or investment fad. People are sheep.
dannykwells 7y ago
Hmmm, I'm not so sure about this. With things like microarrays and other tech, sure there were promises that were too big. But the underlying idea (gene expression matters, there are lots of genes, we should measure them all) is correct. To wit, RNA-seq has transformed cancer biology. And Nanostring is now offering a microarray-like technology with extremely impressive signal to noise. And we still use lots of tools initially developed for microarrays (Limma, for example). So I would argue the investment in microarray technology has absolutely paid off. I think you're throwing the baby out with the bath water to say that, because a few bad eggs are effectively salesman and not scientists, that we wasted "hundreds of billions" of dollars (i.e., a large fraction of the NIH budget). The fact is, most research doesn't hold up not because the researchers weren't honest, or because they were salesmen, but fundamentally, because science is hard and it's really hard to be right for the right reason in science. Alternatively, if you have a formula to determine, in real time, which results are hype and won't stand up vs. which are real and will, well, that would be something too.
- Fede_V 7y ago
  I don't have issues with the technologies, I have issues with the early papers that came out with 'gene signatures' based on maybe 40 or 50 patients. I think in general it's actually quite easy to tell which results are hype and which are not. Do a power analysis and estimate what kind of sample size would be required to measure the effect the researchers estimate - if they are off by an order of magnitude, reject.
- dfeliej 7y ago
  In my experience, there's a disproportionate reward associated with fashions in science. That is, the people at the forefront of a trend get rewarded professionally (tenure, positions, promotion), even when behaving in a way that is irresponsible theoretically or empirically. The skeptics, on the other hand, who carefully lay out the problems with the new research, and eventually point out the obvious flaws, get treated as sticks-in-the-mud, etc., even though they are eventually proven correct. There are gains to be made through innovation, and I'm not arguing against that. Sometimes too, details are just minor details to work out. But sometimes the details matter and point to bigger problems. The problems with fads in science are that the focus in academics seems sometimes to be on popularity per se rather than verdicality, and denialism about scientific fads as a phenomenon. This is pointed out in the article, and is one of its better points. Someone else criticized the article for not presenting solutions, which is fair, but I think that's in part because effective solutions would probably require a huge shift in culture in academics. We'd need to return to more stable professional positions rather than the cannabilistic postdoc system in place now, with declining tenure; we'd need to focus more on groups of researchers and findings rather than celebrity academics ala TED talks; we'd need to put much more emphasis on a pattern of replications rather than single studies; etc. etc. Even then, I don't think the problem would entirely go away.

by Fede_V 7y ago

api 7y ago

> Every single time you'd get 'trailblazers' who'd come into the field, do shitty rushed science, hype their results like crazy, and by the time people figured out that all the initial results were garbage, they'd have moved onto the new hot thing. Same thing happens in business with the latest startup or investment fad. People are sheep.

dannykwells 7y ago

Hmmm, I'm not so sure about this. With things like microarrays and other tech, sure there were promises that were too big. But the underlying idea (gene expression matters, there are lots of genes, we should measure them all) is correct. To wit, RNA-seq has transformed cancer biology. And Nanostring is now offering a microarray-like technology with extremely impressive signal to noise. And we still use lots of tools initially developed for microarrays (Limma, for example). So I would argue the investment in microarray technology has absolutely paid off. I think you're throwing the baby out with the bath water to say that, because a few bad eggs are effectively salesman and not scientists, that we wasted "hundreds of billions" of dollars (i.e., a large fraction of the NIH budget). The fact is, most research doesn't hold up not because the researchers weren't honest, or because they were salesmen, but fundamentally, because science is hard and it's really hard to be right for the right reason in science. Alternatively, if you have a formula to determine, in real time, which results are hype and won't stand up vs. which are real and will, well, that would be something too.

Fede_V 7y ago
I don't have issues with the technologies, I have issues with the early papers that came out with 'gene signatures' based on maybe 40 or 50 patients. I think in general it's actually quite easy to tell which results are hype and which are not. Do a power analysis and estimate what kind of sample size would be required to measure the effect the researchers estimate - if they are off by an order of magnitude, reject.
dfeliej 7y ago
In my experience, there's a disproportionate reward associated with fashions in science. That is, the people at the forefront of a trend get rewarded professionally (tenure, positions, promotion), even when behaving in a way that is irresponsible theoretically or empirically. The skeptics, on the other hand, who carefully lay out the problems with the new research, and eventually point out the obvious flaws, get treated as sticks-in-the-mud, etc., even though they are eventually proven correct. There are gains to be made through innovation, and I'm not arguing against that. Sometimes too, details are just minor details to work out. But sometimes the details matter and point to bigger problems. The problems with fads in science are that the focus in academics seems sometimes to be on popularity per se rather than verdicality, and denialism about scientific fads as a phenomenon. This is pointed out in the article, and is one of its better points. Someone else criticized the article for not presenting solutions, which is fair, but I think that's in part because effective solutions would probably require a huge shift in culture in academics. We'd need to return to more stable professional positions rather than the cannabilistic postdoc system in place now, with declining tenure; we'd need to focus more on groups of researchers and findings rather than celebrity academics ala TED talks; we'd need to put much more emphasis on a pattern of replications rather than single studies; etc. etc. Even then, I don't think the problem would entirely go away.