5 ms·
I've never encountered a field in science where fundamental results aren't sufficiently replicated by other labs. That nobody spent $500-$2000 to estimate leg
by epistasis 12y ago
I've never encountered a field in science where fundamental results aren't sufficiently replicated by other labs.
That nobody spent $500-$2000 to estimate leg hair drag probably speaks more about people not wanting to be caught measuring leg hair drag than about the entire field's tendency to replicate findings. It's also really weird for people to consider $500/hour tunnel time to be expensive for a sport with as much money as cycling.
Really, I thought it was weird, and probably inappropriate, to mix in so much of an outsider's amateur and unsopported opinion about science into an otherwise interesting story about leg hair drag.
If there's a serious story somewhere about results not getting replicated when they should in many fields of science, I'd like to hear about it. There's often stuff on the fringe, but any serious result will be replicated soon. Take, for example, the acid-bath stem cell retraction from not long ago. Where else are important results not replicated? The Higgs? It's a baffling perspective.
- thaumaturgy 12y ago> If there's a serious story somewhere about results not getting replicated when they should in many fields of science, I'd like to hear about it. Actually, there is! It's been referred to as "the decline effect", and it's fascinating. The New Yorker published an article about it a few years ago (http://www.newyorker.com/magazine/2010/12/13/the-truth-wears-off?currentPage=all http://www.newyorker.com/magazine/2010/12/13/the-truth-wears...). It has resurfaced in popularity a couple of times. It was written by disgraced journalist Jonah Lehrer, but later independent investigation failed to find any fault with the article he wrote (http://www.lastwordonnothing.com/2012/11/05/jonah-lehrer-nature-of-truth/ http://www.lastwordonnothing.com/2012/11/05/jonah-lehrer-nat... -- worth reading because it includes statements from scientists on Lehrer's article). So far, my favorite explanation is the direct, obvious one, which is that we're sometimes -- perhaps more often than we'd expect -- falling victim to statistics. Given the very large number of experiments and studies being conducted everywhere, and the selection for positive results, statistical anomalies are being accidentally selected for and then not being extensively enough re-evaluated much later. The article discusses that as a possible explanation, but the journalist does his job, igniting reader interest, by suggesting the effect somehow "defies the laws of statistics". I favor the simpler explanation, which is that our meat brains simply underestimate the vast number of ways in which statistics would like to bugger us. Some of the things that both you and the parent commenter mention are discussed in the article. For example: > Jennions, similarly, argues that the decline effect is largely a product of publication bias, or the tendency of scientists and scientific journals to prefer positive data over null results, which is what happens when no effect is found. and > Richard Palmer, a biologist at the University of Alberta, who has studied the problems surrounding fluctuating asymmetry, suspects that an equally significant issue is the selective reporting of results—the data that scientists choose to document in the first place. (aside: I'm not one of HN's vocal "science is junk" members. I love science, or, at least, I like to admire its butt (http://explosm.net/comics/3557/ http://explosm.net/comics/3557/). I don't believe there's something fundamentally wrong with the scientific method. But, I do think more emphasis needs to be placed on repeatability of experimental results over longer periods of time, and I hope that this will become more of a trend as the world economy continues to grow.)
- epistasis 12y agoThis is very fascinating stuff! And it's also familiar from GWAS where there's a fantastic number of statistical tests. However, it's all examples of things being replicated in the sense of (1) being tested again, and being "not replicated" in the sense of (2) being tested again and being found to not repeat as initial reports. I think the article's complaint was about (1), though I find this (2) far more interesting and will pursue these threads, thanks!
- sjackso 12y ago> Given the very large number of experiments and studies being conducted everywhere, and the selection for positive results, statistical anomalies are being accidentally selected for and then not being extensively enough re-evaluated much later. I once heard a natural language processing professor observe that a typical NLP conference might have twenty presenters, each giving results to a 95% confidence. "So even if everyone does all their work flawlessly, every conference will have on average one set of bad results."
- DanBC 12y agoWhy Most Published Research Findings Are False: http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124 http://journals.plos.org/plosmedicine/article?id=10.1371/jou... Everything We Know Is Wrong http://www.bbc.co.uk/programmes/b04f9r4k http://www.bbc.co.uk/programmes/b04f9r4k
- Malarkey73 12y agoIts a very common phenomenon in science that a finding is published in a glamour journal e.g. Nature/Science and then if not immediately controversial e.g. acid bath stem cells, it becomes very difficult to publish something contradictory. Journal referees will set the bar for such findings very very high...and you are quite likely to be refereed by the original author or their friends. I know this to my cost having gone to extraordinary lengths to publish something slightly contradictory in a far lower impact journal than the work deserved.
- learnstats2 12y agoIn general, if your results contradict existing science, your work for publication will be passed immediately to those scientists for "peer review". Those same scientists are motivated to protect their own existing and future work. The academic standards you are subjected to will be far higher than work supporting the status quo.
- rgejman 12y agoUnfortunately, they are usually also the best equipped to find flaws in the new work. True, this is not ideal, but it's also not ideal to send a paper to less expert reviewers. All of which goes to say that pre-print publication is the way to gooooo.
- learnstats2 12y agoResults which generate international headlines tend to get replicated in the hope of further headlines. This does not represent the experience of many millions of scientists. You should consider that all science is on the fringe - the purpose of academic science is to research novel ideas.
- TheCondor 12y agoI agree with your sentiment but there is one flaw, cycling is a dirt poor sport that has traditionally been very blue collar. There are racers in the Tour that make less than $30k per year, live at home with their parents in the off season and wrench as mechanics. There are only a small handful of them that make "pro athlete" money. That's a lot of money to most of them, especially when they can be testing fit and position which have well known advantages to both aero and power output AND they already shave.