18 ms·
We Should Not Accept Scientific Results That Have Not Been Repeated
- guaka 10y agoTotally agree. I'd go even further and make free licenses on scientific source and datasets mandatory. Research that is funded by public money should lead to public code and data.
- jeremysmyth 10y agoWhat about defense research?
- zdkl 10y agoIn that vein, what about {industrial|scientific|...} espionnage?
- amboar 10y agoWell, there's the GPL-styled approach: anyone with access to the results must also have access to the associated data. This doesn't mean it is mandatory to make it public, though you'd have to restrict the redistribution freedom.
- wodenokoto 10y agoI recently used a large dataset of tweets in a research project. As far as I know, I do not have the rights to distribute these. I also used a dataset consisting of newspaper articles. It cost me $1.000 to get access to, and I definitely do not have the rights to redistribute it.
- dagw 10y agoAs long as you provide a detailed enough description of the source of your dataset that I can reproduce it myself then that is fine. So in your first case tell me what criteria you used to select your tweets and in the second tell me where to send my $1000 and what to ask for.
- tnhh 10y agoUnfortunately not everyone reports this information. Here is a study that we did of over 500 papers using online social network data: http://tnhh.org/research/pubs/tetc2015.pdf http://tnhh.org/research/pubs/tetc2015.pdf While most authors would report high-level characteristics (e.g., which social network they measured), fewer authors reported how they sampled the network or collected data, and very few people reported on how they handled ethics, privacy and so forth.
- dalke 10y agoSo, $1 of public funding triggers public code release? Or is there a threshold? No one has been able to tell me why the need for reproducibility requires software freedom. Consider the program 'nauty'. It is available in source code form for anyone to review, but it cannot be used for military purposes. That's not free, certainly. But isn't that enough to call it good science? Similarly, consider the clause "only for use in verifying the result of paper X". That's also not free. But it serves the goal of letting others be able to verify X. Also, you haven't gone far enough. It's not only the license that matters, but access. You have to mandate that either anyone can get access to the code for no/low cost for some years (since I can sell my GPL'ed software for $30,000 or take down the download link once published), or link publication with a required submission to some repository with the mission of keeping all that source and data around, available to anyone, at no cost.
- eru 10y ago> So, $1 of public funding triggers public code release? Or is there a threshold? Yes, 1$, no threshold necessary. For projects where the cost of publication of data is not worth the dollar, people will just not accept the dollar. (That's where a 'natural threshold' comes in.)
- dalke 10y agoSo, if I have a summer intern who is paid by the government, and helps on my otherwise non-government-funded project, then is that enough to trigger the requirement? Or, if I use a government facility, like a supercomputing center, then that also trigger the release requirement? Even if no money changes hands? What about a government network? It sounds very tricky. If I get a government grant to work on a project, and that grant buys equipment X, which I also use for another project, then must both projects be subject to this form of release? Even if the second project is 10 years later? If in working on project X, but in the process of working on X I find some new knowledge Y, and publish it, even though that has nothing to do with the grant from the original X, does that count? (Think AT&T's observation of the cosmic microwave background, when their goal was to reduce noise in terrestrial microwave communications.) It seems like a very complicated scheme.
- return0 10y agoWe now have the tools to do it , and we should be doing it. The fate of scientific findings is not to publish papers, they belong to open and continuous scrutiny. And someone should build a github of scientific facts.
- ipstone2014 10y agolet's try it, if interested, send me a tweet at @isaacpei, seriously thinking about creating something for this
- return0 10y agoThere is no lack of efforts to get scientists discussing (shameless related plug http://sciboards.com http://sciboards.com), but unfortunately there is a disincentive for scientists to do so (politics).
- middleman90 10y agoI thought this was in the definition of "scientific"
- nxzero 10y agoPeer review is how most science is defined as science and peer review does not require reproduction of the work.
- seanmcdirmid 10y agoMuch of what we peer review is not real science, at least in its definition of applying the scientific method. For example, much of computer "science" is not. Math maybe, engineering probably, design sometimes, but "science" is rarely done. BUT the science envy is there, especially post 1990s, and it is as confusing as heck when multiple definitions of "science" collide in a conference culture. Yes I'm a researcher, no I'm not a scientist.
- aminorex 10y agoYou've illustrated the distinction between prescriptive and descriptive definition nicely.
- tnhh 10y agoPeer review is a relatively new aspect of science: the Philosophical Transactions of the Royal Society started systematic peer review in the mid-19th century https://arts.st-andrews.ac.uk/philosophicaltransactions/brief-history-of-phil-trans/ https://arts.st-andrews.ac.uk/philosophicaltransactions/brie...
- gnuvince 10y agoMaybe conferences should have a "reproducibility" track for that purpose? Also, I don't know about other fields, but I'm pretty sure that in CS, if you just took a paper and tried to reproduce the results, you'll get rejected on the ground that you offer no original contribution; no original contribution => no publication => no funding.
- nxzero 10y agoReal issue is that the way science is funded is broken.
- varjag 10y agoBroken assumes it was working well at some point. Nothing new here.
- mathgeek 10y agoIt worked pretty well back when scientific inquiry was funded by selling inventions or other work that the scientist did "as a day job." That was a long time ago, though.
- varjag 10y agoThis has worked and still works for technology, but is not how basic science research was done. There's never been "enough" money for sciences, nor they ever been allocated in a manner agreeable to all parties. For a long time, research was either a gentleman's hobby or a side project under auspices of unsuspecting donor. Lagrange has developed variational calculus while doing his artillery school tenure. Einstein's annus mirabilis happened at his stint in Swiss patent office.
- studentrob 10y agoFor CS, reproducing should be easy given the code and input data right? Other sciences' input isn't so easily shared
- undergroundOps 10y agoI'm a physician and I've been suggesting this to my colleagues for a few years, only to be met with alienated stares and labeled cynical. The doctors and doctors-in-training I work with have altruistic motives, but place too much stock in major medical studies. They also frequently apply single-study findings to patient care, even to patients that would've been excluded from that study (saw this a lot with the recent SPRINT blood pressure trial). And don't even get me started on the pulmonary embolism treatment studies. What a clinical mess that is. It's frustrating.
- appleflaxen 10y agobut in medicine, what is the practical alternative? how do you incorporate these findings? ignore them? if so, it's probably bad for your patients. the only thing worse than a single-study finding is a zero-study finding.
- mablap 10y agoNo. Step 1: Do no harm. In practice, this means staying up to date on the latest research (by law), but follow the guidelines. These are updated regularly, just not after each and every study that comes out. And it's a good thing a lot of doctors exercise caution.
- undergroundOps 10y agoI was simply suggesting what we all learn in medical school and residency: to appropriately evaluate clinical studies. Just don't think most doctors do. Let me give you an example of how I approach things. The guidelines for acute pancreatitis recommend using a fluid called LR instead of NS for volume resuscitation. This is based on an single study that included 10 patients and simply noted slightly better lab numbers; there was no difference in clinical outcome. Lots of problems with that study, right (small, underpowered, confounders, validity issues, etc)? However, there's no major disadvantage for using LR in those patients (unless hyperkalemia is a concern), so I use it since it might have a benefit. This is a very simple example. It gets much more complicated than that. "Probably" is one of favorite words in medicine, btw :).
- tudorw 10y agothis increasingly includes code that needs to run in the future, and citations within code, see this group working in that field https://www.force11.org/sites/default/files/shared-documents/software-citation-principles.pdf https://www.force11.org/sites/default/files/shared-documents...
- bontoJR 10y agoThe big issue right now is funding of replicated research, who wants to fund a research to prove someone else was right? Most of these funds are granted based on potential outcome of the new discovery like: potential business, patents, licenses, etc... not being the first one would probably wipe most of these benefits, cutting down to a small probably getting funded... Now, straight to the point, who's going to pay for the repeated research to prove the first one?
- avindroth 10y agoPerhaps those who are skeptical of the first research (and are losing money from it) should fund the replication research. Incentive for corrupting the data seems high, however.
- bontoJR 10y agoThis works for some "controversial" researches, which can be dismantled. What about successful (or apparently) ones which may lead to tons of $$$ in return? Is there any value to pay for a research which is going to prove someone else was right ending in making their wallets fatter? Sorry for the informal language, but makes things a little bit more salty.
- avindroth 10y agoIf someone is making money from it, someone else is (most likely) losing money from it as well. When the lightbulb was invented, Edison made lots of money, but I am sure candlemakers had plenty of incentive to fund research that hypothesized that lightbulbs emitted toxins.
- jjp 10y ago> Now, straight to the point, who's going to pay for the repeated research to prove the first one? Who is and who should are two different questions. The body who funded the original research should be best placed to fund the verification of the research. If the research isn't compelling enough to fund verification then why was it funded in the first place? And if the principle research group is requesting additional funding for more research that builds on the initial unverified research then that sounds like poor governance. I realise that this simplistic view gets messy and confused when research is really academic led innovation and incubation.
- grashalm 10y agoSometimes in cs if your research is embedded in a huge ecosystem, it can become quite expensive to reproduce results. I mean proper reproduction, not just rerunning the Benchmarks. If you are dealing with complicated stuff, the reproducer might also just not be able to do the same thing technically.
- osivertsson 10y agoMaybe, maybe not. Do you have something specific in mind here? I hope researchers and scientists don't considers others not capable enough, and therefore withhold info on how to reproduce. Even if the experiment is crazy expensive and complex right now it might be considered much more tractable in 10 years, or someone builds upon your work and invents a simpler method to show the same thing.
- grashalm 10y agoI am thinking of huge endeavors like building an asic or huge complex systems like virtual machines. Not always a comparable system for Repetition is available and must be built from scratch. Affording such rebuilds require huge sums. Of course nobody does consider others not capable enough. Its just that there are not so many people experienced enough to build certain systems in a decent amount of time.
- deleted 10y ago[deleted]
- eru 10y agoJust being able to rerun the benchmarks (or other data analysis for non-CS papers) would be an improvement on the current state of affairs, where people often don't publish code nor data.
- grashalm 10y agoAgreed. Some cs conferences verify artifacts these days, which is a good first step.
- ehnto 10y agoWe learned the importance of this in high school science and it baffles me that it's not already the case.
- wccrawford 10y agoWe have some kind of weird hero-worship of scientists where the general public just believes what they say, even if they never even attempt to replicate their results. They do an experiment (which may or may not be scientifically sound to start with) and then publish results, and the public eats it up. And then people have the nerve to say, "Last week chocolate was bad for me, now it's good? Make up you mind!" No, stop listening to un-replicated studies! Jeez.
- hudathun 10y agoGood point about the public involvement. The public and the news systems are part of the problem. I've lost count of how many 'battery breakthrough' articles I've come across, but they seem to pass the newsworthy test.
- eru 10y agoWasn't the problem with battery breakthroughs that they don't commercialise well, rather than that the science doesn't repeat?
- cs702 10y agoMy first reaction to this headline was "duh." Of course we should hold off on accepting scientific claims (i.e., predictions about the natural world) that to date have been verified only by the same person making those claims! My next reaction was, "wow, it's a sad state of affairs when a postdoctoral research fellow at Harvard Medical School feels he has to spell this out in a blog post." It implies that even at the prestigious institution in which he works, he is coming across people who treat science like religion.
- nxzero 10y agoOf course you as a reader of said claims confirm at the very least that they've been independently reproduced, right? (If so, this shouldn't be news.)
- lrem 10y agoUnfortunately, there is no easy way to do it. Confirmation studies are not easily accepted by impactful journals/conferences, thus nearly nobody bothers to do them. Even if there is one, it can be surprisingly hard to find it. As a point of anecdata: my wife's master thesis was a confirmation study of using LLDA for face recognition. I remember seeing it included in some book by the university press. I gave up Googling for it after 5 minutes.
- randall 10y agoThere needs to be better scientific protocol. More linking through data instead of annoying cites. I think anyway.
- BaronSamedi 10y agoMy reaction was the same. I would argue that the requirement is even stronger: the hypothesis must be verified by different scientists over time. One of the key roles of the scientific method is error correction for bugs in our mental machinery (i.e., cognitive biases, of which there are many ). I look at science as an essentially evolutionary process: a single study has a very low probability of being accurate, if the results survive additional testing over time, the probability of accuracy increases.
- aminorex 10y agoMore pragmatically, we should not accept scientific publications and conferences which do not publish negative results and disconfirmations.
- hudathun 10y agoLooking good is, sadly, better rewarded than doing good in many areas of life. It's doubly sad that this affects our body of scientific knowledge. Even claims that are reproduced can suffer from funding bias and confirmation bias. The truth hopefully comes out in the end, but I'm sad for the harm that's caused in the interim.
- dalke 10y agoI disagree. One part of science is observation. Including observations which cannot be, or at least have not been, repeated. For example, consider a rare event in astronomy which has only been detected once. Is that science? I say it is. But it's surely not repeatable. (Even if something like it is detected in the future, is it really a "repeat"?) Some experiments are immoral to repeat. For example, in a drug trial you may find that 95% survive with a given treatment, while only 5% survive with the placebo. (Think to the first uses of penicillin as as real-world example.) Who among you is going to argue that someone else needs to repeat that experiment before we regard it as a proper scientific result?
- humanrebar 10y agoThe point of science is to discover truth in a way that is objective and convincing. That's not to say there isn't other truth out there. I think people mixing 'science' up with 'truth about the universe' is causing some unnecessary cognitive dissonance.
- ebbv 10y ago> One part of science is observation. Including observations which cannot be, or at least have not been, repeated. For example, consider a rare event in astronomy which has only been detected once. Is that science? I say it is. But it's surely not repeatable. First off, you can accept the observation at face value as an observation, but conclusions drawn from the claims which have no other support or means of verification should not be accepted and would not be accepted. Fortunately, most of the time even if something is initially sparked by a very rare occurrence, it will have some kind of implications that are verifiable by some other means other than just waiting for something to happen in space. But even something that is rare and relies on observation, like gravitational waves, we have already been able to identify more than one occurrence. > Some experiments are immoral to repeat. For example, in a drug trial you may find that 95% survive with a given treatment, while only 5% survive with the placebo. What's more immoral, releasing a drug that's only had one test, even a striking one, on the public as a miracle cure that you have not truly verified or performing another test to actually be sure of your claims before you release it? > Who among you is going to argue that someone else needs to repeat that experiment before we regard it as a proper scientific result? That's how science works. If something is not independently repeatable and verifiable then science breaks down. Look at the recent EM drive. Most scientists in the field were skeptical of it, and once it was finally attempted to be independently verified the problems were found. Independent verification is the cornerstone of science and what makes it different from bogus claims by charlatans.
- Mendenhall 10y agoLook at research done on many political hot button topics. They love results that have not been repeated. I see all sorts of posts even on HN that reference such "science" as well. The root problem, people who are pushing an agenda.
- csydas 10y agoI think with the increased visibility of scientific research to the general public, it's less that science needs to stop accepting unrepeated results, but instead the paper process needs to be updated to reflect the new level of availability, and journal databases need better relationship views between papers and repeated tests. As an outsider looking in on the Scientific process, I am not really sure how applicable my opinions are, but I see these as useful changes. Basically, in reverse order, my suggestions for science to adopt are as follows: Papers in databases need to have fields related to reproduction studies, and it needs to start becoming a prideful part of the scientific process; just as there is a lot of pride and money, researchers should start to thump their chest based on the reproducibility of their work, actively seeking out contemporaries and requesting a reproduction study as part of the pubilshing process, and subsequently updating. The papers published themselves should take a moment (perhaps no more than a paragraph) to include a "for media" section that outlines the "do's and don't's" on reporting on the research. For example, cancer research should clearly state examples of acceptable understandings in lay person terms as a sort of catch for sloppy reporting. Something like "Do not write "cure for cancer found" or "Effective treatment", instead write "progress made, etc". Basically put a sucker punch to outlandish headlines and reporting right in the paper itself, and let journalists who want to be sensationalist embarrass themselves. This seems like two very simple acts that could raise the bar for science a bit.
- ebbv 10y agoThose are both good but the key here is the media needs to understand that scientific papers that have not been independently verified are in a "maybe" state. Of course, they probably do know this and just choose to ignore it because "Unverified Study that MIGHT Point to M&M's Being Good For You" won't get as many clicks as "M&M's Are Good For You Says New Study!"
- csydas 10y agoThis is sort of why I think having it stated explicitly within the paper, not just an aside but part of the actual process. It's to pit less scrupulous journalists against one another, in an "honor among thieves" sort of way I guess. If someone wants to go ahead and write clickbait, they can, but it leaves them open to someone else looking to discredit them going "well, did you even read the paper? they told you not to write that." it's not so much checking for the public purpose, it's for others.
- deleted 10y ago[deleted]
- Eerie 10y agoUGH. Scientists are not getting funds to repeat result.
- cube00 10y agoHaving wasted time trying to replicate someone else's results who 'lost' their code, I agree! Maybe repeating the experiment should be part of the peer review.
- closed 10y agoSo frustrating. Lost = "I didn't think anyone would hold me accountable for it."
- mydpy 10y agoI agree in principle. There are a few concerns: 1. How should we receive costly research that took special equipment and lots of time to develop and cultivate? I.e., CERN? 2. A lot of research is published, ignored, and then rediscovered. In this case, we may want to accept the research until it cannot be repeated (i.e., in another journal publication). 3. Reviewers of academic publications probably are not qualified or have the time to recreate all scientific research. 4. Isn't the academic system at its core kinda... broken?
- thefastlane 10y ago"Isn't the academic system at its core kinda... broken?" can your elaborate on what you mean?
- hdra 10y agoI gather its things like misaligned incentives. Like funding sources dictating the "desirable" result, reluctance to publish negative results, pursue of (vanity) metrics that leads to quota system by universities, etc.
- deleted 10y ago[deleted]
- misnome 10y agoDefine repetition. It's not as simple as that, for all sciences - once again an article on repeatability seems to have focused on medicinal drug research (it's usually that or psychology), and labelled the entire "Scientific community" as 'rampant' with " statistical, technical, and psychological biases". How about, Physics? The LHC has only been built once - it is the only accelerator we have that has seen the Higgs boson. The confirmation between ATLAS and CMS could be interpreted as merely internal cross-referencing - it is still using the same acceleration source. But everyone believes the results, and believes that they represent the Higgs. This isn't observed once in the experiment, it is observed many, many times, and very large amounts of scientists time are spent imagining, looking for, and measuring, any possible effect that could cause a distortion or bias to the data. When it costs billions to construct your experiment, sometimes reproducing the exact same thing can be hard. The same lengths are gone to in order to find alternate explanations or interpretations of the result data. If they don't, they know that some very hard questions are going to be asked - and there will be hard questions asked anyway, especially for extraordinary claims - look at e.g. DAMA/LIBRA which for years has observed what looks like indirect evidence for dark matter, but very few people actually believe it - the results remain unexplained whilst other experiments probe the same regions in different ways. Repetition is good, of course, but isn't a replacement for good science in the first place.
- welterde 10y agoATLAS, CMS, ALICE, LHCb, etc. are all different experiments that just happen to share the same accelerator. They are done using different detectors, designed, built, operated and the data is analyzed by different groups of people. If you follow your logic, that the different experiments using the same accelerator negates the whole thing, to the extreme then doing two experiments on the same planet/solar system/universe won't be enough.. And I don't buy your inverse argument that one (good) experiment is good enough either. It is very difficult to tell from outside the group if the experiment is actually good or not (although you probably can tell if it's bad). Screw-ups can happen no matter how many people look at the data if there is some flaw in the experimental setup - only way to make really sure is to use different experiments to measure the same thing.
- cm2187 10y agoHow do you that in the medical field? Studies are often based on a small number of patients affected by a particular condition.
- jbb555 10y agoWe shouldn't "accept" or "reject" results at all. It's not a binary option. One poor experiment might give us some evidence something is true. A single well reviewed experiment gives us more confidence. Repeating the results similarly does. As does the reputation of the person conducting the experiment and the way in which it was conducted. It's not a binary thing where we decide something is accepted or rejected, we gather evidence and treat it accordingly.
- James001 10y ago^ One of the only reasonable responses here.
- bbctol 10y agoSo many scientists I talk to don't have a basic understanding of philosophy of science. I don't necessarily blame them--I understand why "philosophy" as an academic field is seen as a soft, speculative, and pretentious field compared to the rigor of science, but as Daniel Dennett said, “There is no such thing as philosophy-free science; there is only science whose philosophical baggage is taken on board without examination." These days, if you ask a scientist "So how do we prove something is true using science?" they'll be able to recite Popper's falsificationism as if it's a fundamental truth, not a particular way of looking at the world. But the huge gap between the particular theory that people get taught in undergrad--that science can't actually prove anything true, just disprove things to approach better hypotheses--and the real-world process of running an experiment, analyzing data, and publishing a paper is unaddressed. The idea that there's a particular bar that must be passed before we accept something as true is exactly what got us into this mess in the first place! There's a naive implicit assumption in scientific publishing that a p-value < 0.05 means something is true, or at least likely true; this author is just suggesting that true things are those which yield a p-value under 0.05 twice! What's needed, in my opinion at least, is a more existential, practically-grounded view of science, in which we are more agnostic about the "truth" of our models with a closer eye to what we should actually do given the data. Instead of worrying about whether or not a particular model is "true" or "false," and thus whether we should "accept" or "reject" an experiment, focus on the predictions that can be made from the total data given, and the way we should actually live based on the datapoints collected. Instead, we have situations like the terrible state of debate on global warming, because any decent scientist knows they shouldn't say they're absolutely sure it's happening, or a replication crisis caused by experiments focused on propping up a larger model, instead of standing on their own.
- kayhi 10y agoIn the world of chemistry, biochemistry and microbiology a huge step forward would be for journals to require a complete list of products used. The publication should also include the certification of analysis for each item as they vary over time. For example, here are two product specifications for a dye called Sirius Red, the first by Sigma-Aldrich[1] and the second by Chem-Impex[2]. The Sigma-Aldrich product contains 25% dye while the Chem-Impex contains equal or greater than 21%. These two dyes could be quickly assessed with a spectrophotometer in order to determine an equivalency, however you need both dyes on hand which doesn't seems like a good use of funding. Also this touches on another problem in replication which is, what is in the other 75%+ of the bottle? [1] http://www.sigmaaldrich.com/Graphics/COfAInfo/SigmaSAPQM/SPEC/36/365548/365548-BULK_______SIAL_____.pdf http://www.sigmaaldrich.com/Graphics/COfAInfo/SigmaSAPQM/SPE... [2] http://www.chemimpex.com/MSDSDoc/22913.pdf http://www.chemimpex.com/MSDSDoc/22913.pdf
- Gatsky 10y agoLots of scientific results are repeated but not published. If it doesn't work then people just move on. The problem is journals. There is no way to publish your attempts to repeat an experiment, unless you put it into another paper. The other issue, especially in the life sciences, is inaquedate statistical input. If someone performs an underpowered, confounded experiment and gets a positive result, then someone else performs the same underpowered confounded experiment and gets a negative result, what have we learned except that the experiment is underpowered?
- dahart 10y agoIt's unfortunate that the suggestions at the end don't seem to offer a realistic attack vector. > First, scientists would need to be incentivized to perform replication studies, through recognition and career advancement. Second, a database of replication studies would need to be curated by the scientific community. Third, mathematical derivations of replication-based metrics would need to be developed and tested. Fourth, the new metrics would need to be integrated into the scientific process without disrupting its flow. Yes, absolutely those things need to happen, but the problem is how to get this funded, how to get people to not see reproducing results as career suicide, right? Items 2-4 will fall out as soon as item #1 happens. How do we make item #1 happen? What things could be done to make reproducing results actually an attractive activity to scientists?
- dragandj 10y agoThe problem is that, if you put mere reproduction as a goal, many scientists would see that as low hanging fruit to beef up the resume, so we'd get countless unnecessary "experiments". I'd say the goal that gets credited should not be merely reproducing the results, but finding errors in the previous research. That would count as novel, and is something that is presently recognized as contribution. The only problem is that journals or conferences treat it as unattractive, so good luck publishing something of the kind...
- dahart 10y ago> The problem is that, if you put mere reproduction as a goal, many scientists would see that as low hanging fruit to beef up the resume, so we'd get countless unnecessary "experiments". Only if you assume the incentives for the 2nd, 3rd, 4th, etc. reproduction experiments remain the same, right? I wouldn't assume that, both because the first reproduction is the most valuable, and for the reasons Ahmed discussed in the article - that scientists are motivated by their perceived ability to do something novel. So first reproduction might be novel, but the fifth would certainly be less valuable, so I wouldn't personally assume we'd get a flood of useless experiments. > I'd say the goal that gets credited should not be merely reproducing the results, but finding errors in the previous research Reproducing an experiment is meant to, without prejudice, either confirm or deny the previous research. It's not meant to confirm the previous results, it is meant to ask whether there could be errors in the research, but without assuming there are errors. It is novel to validate a result the first time, whether it's positive or negative, and for this incentive system to work, it has to appeal to people who might not find something dramatic or contradictory. It must be appealing to do the work, regardless of the outcome, or it's not an incentive at all.
- VikingCoder 10y agoThis is wrong-headed in the extreme. What we should demand is scientific results that have FAILED. When we see a p=0.05, but we don't know that this SAME EXACT EXPERIMENT has been run 20 times before, we're really screwing ourselves over. Relevant: https://xkcd.com/882/ https://xkcd.com/882/
- collyw 10y agoJust to play devils advocate, won't there be a self correcting mechanism? If results are genuinely useful, then people will want to build upon that work, and will have to repeat the science. On the other hand if it can't be repeated, then it will not get further work done and fade into obscurity. Curious what other peoples opinion on this are?
- deleted 10y ago[deleted]
- typhonic 10y agoI've always been amazed by how widely the Stanford Prison Experiment results are accepted when a) the experiment has not been repeated and b) the experiment didn't even get completed. It was stopped when the researchers had made up their minds about the results.
- framebit 10y agoThis problem, like many in modern day science, can in large part be traced back to unstable funding. On the Maslow's-style hierarchy of research lab needs, the need for funding is a lot lower on the scale than the aspiration for scientific purity, just as a human's need for food is lower on the scale than their desire for self-actualization. If competition for research dollars ceases to be so cutthroat, it will go a long way towards solving this and many other seemingly entrenched cultural problems.
- known 10y agoLottery < Statistics < Science
- leecarraher 10y agoIt's not just money that prevents people from repeating experiments, it's recognition. The general idea for research to be accepted is that it makes some novel, albeit small, impact on the field, acceptable for publication in a peer reviewed journal or proceeding. Repeating someone else's experiments wont get you that, so in general it wont help you graduate or move you toward a higher position at a university or in your profession, meaning there is very little motivation for researchers to pursue such endeavors. So instead of just throwing money at the problem, we may need to entirely revamp how we recognize the pursuits of researchers.
- dschiptsov 10y agoAccording to old-school philosophy of science truth could be discovered only by removing all the nonsense, as a remainder, not by pilling up nonsense on top of nonsense out of math and probabilities. Probabilities, for example, are not applicable to partially observed, guessed and modeled phenomena. It should be a type-error. As for math - existence of a concept as a mathematical abstraction does not imply its existence outside the realms of so-called collective consciousness. Projecting mathematical concepts onto physical phenomena which could not be observed is a way to create chimeras and to get lost in them. Read some Hegel to see how it works.)
- tedks 10y agoFrom the authors of "Why Science Needs Metaphysics" this rings a little hollow. Nautilus is just a slightly less vitriolic version of the Aeon-class anti-science postmodernist blog. Like Aeon, it's garbage.
- mmierz 10y agoI see a lot of people commenting here that there's no incentive to repeat previous research because it's not useful for getting grants, etc. This kind of true but I think it misses something important. At least in life sciences (can't comment on other fields), it's not that scientists don't repeat each other's results. After all, if you're going to invest a significant fraction of your tiny lab budget on a research project, you need to make sure that the basic premise is sound, so it's not uncommon that the first step is to confirm the previous published result before continuing. And if the replication fails, it's obviously not a wise idea to proceed with a project that relies on the prior result. But that work never makes it into a paper. If the replication succeeds, great! Proceed with the project. But it's time-consuming and expensive to make the reproduction publication worthy, so it will probably get buried in a data supplement if it's published at all. If the replication fails, it's even more time-consuming and expensive to convincingly demonstrate the negative result. Moreover, the work is being done by an ambitious student or postdoc who is staring down a horrible job market and needs novel results and interesting publications in order to have a future in science. Why would someone like that spend a year attacking the work of an established scientist over an uninteresting and possibly wrong negative result, and getting a crappy paper and an enemy out of it in the end, instead of planning for their own future? If enough people fail to replicate a result, it becomes "common knowledge" in the field that the result is wrong, and it kind of fades away. But it's not really in anyone's interest to write an explicit rebuttal, so it never happens.
- untilHellbanned 10y agoYes. Life scientist here. This is 100% on point. It seems that most people who comment that we need 100% published repetition aren't aware of how the process currently works as you describe it. Repetition happens regularly if not in the majority of published research. It's just that it often doesn't get written up in a way that one can easily see the connection with the previous work. Citations are supposed to help this problem but because citations aren't just links and are hidden behind paywalls (a whole different can of worms), its not easy to see what's been replicated vs. not.
- refurb 10y agoAs a former life scientist, I completely agree. If something truly revolutionary is published and it's relevant to your own work, it's very common to repeat the experiment yourself. If it works, great, continue on with your own work and reference the original paper when you publish new results. If it doesn't work, the outcome can range from: (1) mentally note and move onto an alternative if they are available to (2) spend a ton of time trying to "fix" it, including contacting the original author. The only time you see reproducibility experiments published is if the original paper made some serious errors, including outright fraud. An there is no way you'd be able to test the reproducibility of every paper because some papers aren't that important. The important ones get the attention and if you see multiple scientists referencing a work (and building off of it), you can have confidence it actually works.
- bane 10y agoI think a better way of thinking about what we want than "repetition" is "independent corroboration".
- VlijmenFileer 10y agoLike climate science, right? Let's set up a statistical meaningful set of equivalent earths, and start doing some serious peer review.
- ythl 10y ago> Nowadays there's a certain danger of the same thing happening (not repeating experiments), even in the famous field of physics. I was shocked to hear of an experiment done at the big accelerator at the National Accelerator Laboratory, where a person used deuterium. In order to compare his heavy hydrogen results to what might happen with light hydrogen, he had to use data from someone else's experiment on light, hydrogen, which was done on different apparatus. When asked why, he said it was because he couldn't get time on the program (because there's so little time and it's such expensive apparatus) to do the experiment with light hydrogen on this apparatus because there wouldn't be any new result. And so the men in charge of programs at NAL are so anxious for new results, in order to get more money to keep the thing going for public relations purposes, they are destroying--possibly--the value of the experiments themselves, which is the whole purpose of the thing. It is often hard for the experimenters there to complete their work as their scientific integrity demands. -- Richard Feynman, "Surely You're Joking, Mr. Feynman", pp. 225-226
- gohrt 10y agoCurrent top comment https://news.ycombinator.com/item?id=12186295 https://news.ycombinator.com/item?id=12186295 , specifically rebutted 30 years before the comment was written.
- throwaway729 10y agoThis isn't a rebuttal of the linked comment. The linked comment doesn't state that it would be a waste of time to replicate on a hypothetical LHC clone. Rather, the linked comment states that we can accept the Higgs result with reasonable confidence even though it's currently infeasible to replicate that experiment. Feynman's issue was also qualitiatively different -- the scientist was comparing results from two different instruments. The people in charge of one of the instruments wouldn't allow the scientist to run both experiments on a single instrument. In fact, from context, it's not even clear to me Feynmann would have insisted on re-running the original experiment if the scientist were not using a different accelerator for the second one. Anyways, in the Higgs case, there's no potential for a "comparing readings from instrument A to readings from instrument B" type bug. More to the point, and FWIW, I somehow doubt Feynman would insist on building a second LHC for the sole purpose of replicating the Higgs measurement. But I guess we have to leave that to pure speculation.
- jmilloy 10y agoObviously I agree that scientific results must be reproducible. But I also realize that it's simply infeasible to repeat the entirety of every study, and much less to also go to the effort to write and peer-review those repeated results. What I think is overlooked in this discussion as that a lot of confirmation work already happens. Most (all?) scientific results are incremental progress built on a heap of previous work. In the course of normal research, you reproduce existing results as necessary before altering conditions for your own study. If you can't confirm the results, well then perhaps you have a paper (though it can be politically challenging to get it published, and that's a separate problem). But if you do, then you don't waste time publishing that, you get on with the new stuff. Ultimately, I don't think scientists do accept results in their field that they have not repeated.
- fhood 10y agoMany people have mentioned that replicating an experiment can be expensive, but I don't think anybody has really brought up just how expensive this can be. Not all science is done in a lab. Replicating an experiment is obviously feasible for a short term psychology experiment, but in earth sciences (oceanography for instance.) it is far less often possible to reproduce an experiment for the following reasons. N.B. This is all from my personal experience of one field of science. 1.) Cost. If you got funding to take an ice-breaker to Antarctica to "do science" it required several million dollars to fund. It is difficult enough to secure funding for anything these days, none the less prohibitively expensive attempts to reproduce results. (honestly any serious research vessel will run costs into the millions, regardless of destination.) 2.) Time. Say you are on a research vessel taking measurements of the Amazon river basin. This is a trip that takes months to years to plan and execute. If you return to duplicate your experiment 2 years later, the ecology of the area you were taking measurements of may have changed completely. 3.) Politics. Earth sciences often require cooperation from foreign entities, many of which are not particularly stable, or whom may be engaging in political machinations that run counter to your nationality's presence in the country, or both. Iran and China are two good examples. Both are home to some excellent oceanographers, and both of which can be very difficult to Science in when your team includes non Iranian/Chinese nationalities.
- chucky_z 10y agoIn for instance, a bioscience lab, I don't believe that results should even be accepted unless they're repeated with similar reagents. Some reagents are so specific they only prove something.... for that one single thing, which could be unique on this planet.
- vonnik 10y agoIn that case, macro-economics is simply disqualified from being scientific. It's almost impossible to repeat large-scale events, controlling for all variables. Have to say I'm not particularly impressed with the quality of Nautilus's analysis.
- T-A 10y ago> In that case, macro-economics is simply disqualified from being scientific. Well, duh? (In other words: of course it's not a science.)
- Zenst 10y agoIndependently verified and repeated I would add. After all any scientific test that fails when somebody else repeats it is not science but the domain of magic and religion, so clearly not science.
- aficionado 10y agoThe solution is easy and it applies to most sciences: all research articles should include a pointer to download the dataset that was used and an annex with the details on how it was collected.
- munificent 10y agoMost disciplines where correctness is important seem to end up having some adversarial component. It is explicitly how the justice system in the US works [1]. Many software companies have separate QA departments that are deliberately kept at a remove from the engineers to encourage some rivalry between them. Security issues are almost always managed in an adversarial way (though here, you could argue that's because it reflects how the system itself is [mis-]used). Markets are intended to allow fair competition between producers to find an optimal price and product for consumers. Peer review is supposed to do this, but the fact that peer reviewers are often colleagues leads to collusion, whether intended or not. Maybe we need a separate body of scientists whose sole job—and whose entire prestige—derives from taking down and retracting bad science. [1]: https://en.wikipedia.org/wiki/Adversarial_system https://en.wikipedia.org/wiki/Adversarial_system
- teslaberry 10y agojust be honest, 90% of funded 'social' science is absolute garbage proposals producing expected absolute garbage results. plenty of 'environmental' and complex system science is just pseudoscience, this includes, especially , 'economics'. which is mostly a confidence game in jargon about complex systems which operate with stupendously unpredictable feedback loops of human behavior. and much of 'economics' is just financial dogma requiring obedience and faith in the current central banking regime as well as in the basic concept of top down control over your money by way of pure authority. you could defund 90% of these areas and you'd be no worse off, possibly better.
- jheriko 10y agowait! we should use the scientific method for science? its a radical suggestion. sad that it is, but true... ;) seriously though... you can't have falsifiable results if you don't constantly try to falsify them. then it just becomes a result, which means the conclusion you can draw is close to nothing.... not quite nothing, but exceptionally close. :)
- macspoofing 10y agoOr at least, the media shouldn't report on results until they have been repeated. This would cut down on the daily "X causes cancer / X doesn't cause cancer" media spam.
- nonbel 10y agoCue all the people justifying their pseudoscientific behavior. If it is too expensive to fund twice, it shouldn't be funded once. If that means the LHC and LIGO wouldn't get done, then we should have only funded one of them. We need to remain skeptical of those results until replicated by a new team. Even one replication is pretty weak... Independent replications of experiment (and the corresponding independent reports of observations) are a crucial part of the scientific method, no matter how much you wish it wasn't. Nature doesn't care if it is inconvenient for you to discover her secrets, or that it is more difficult for you to hype up your findings to the unsuspecting public.
- daveguy 10y agoYou do realize that scientists who work on the LHC have the highest repeatability standards of any science profession, right?
- nonbel 10y agoAs far as I know there is only one organization running a machine that can check the LHC results. That is the team that runs the LHC. On the other hand, there are many other experiments that are repeated billions+ times a day in order for consumer electronics to work, etc.
- jomamaxx 10y agoThe LHC experiment is not the issue here. There is a lot of transparency there, a lot of well meaning people with a lot of oversight. I suggest most would admit 'there could be a problem' there, but it's out in the open if there is. The problem of lack of repeatability I think has to do with subconscious bias on the part of the experimenters which will be less pronounced when there are 5000 people working on it.
- unabst 10y agoWith science, the profession and the product are distinctly different, and we are failing to hold the profession to the standards of the product. Science, the profession, is political, incentive driven, and circumstantial. Scientists need to get paid. Science, the product, is apolitical, existential, and universal. So those who love and believe in the products of science may wish upon themselves to be these things also. I know I do. Except, sometimes it just ins't practical, or even possible. But repeatability actually matters more professionally. Scientifically speaking, if the science is bad it just won't work when others try to make use of it. All bad science will be identified or corrected as we try and make use of it and convert it into new technology. Technology mandates repeatability. So those scientists who fail to produce repeatable science, regardless of how professionally successful they may be, will inevitably fail to produce any new technology or medicine, and vice versa.
- triangleman 10y agoIronically, one of the reasons Semmelweis's colleagues rejected his "hand-washing" hypothesis was that it did not have a good enough empirical/statistical basis. http://www.methodquarterly.com/2014/11/handwashing/ http://www.methodquarterly.com/2014/11/handwashing/ https://en.wikipedia.org/wiki/Contemporary_reaction_to_Ignaz_Semmelweis https://en.wikipedia.org/wiki/Contemporary_reaction_to_Ignaz...
- webosdude 10y agoDidn't John Oliver say the same thing few months ago in his episode on scientific studies, https://www.youtube.com/watch?v=0Rnq1NpHdmw https://www.youtube.com/watch?v=0Rnq1NpHdmw
- vorotato 10y agoThere is no science without repetition.
- mankash666 10y agoWhat we need is accountable statistics - something that cannot be manipulated. One idea is to enforce storing or indemnifying a time-stamped data base of the raw data on the block chain
- sevenless 10y agoWe also should not accept historical claims that have not been repeated :)
- pc2g4d 10y agoReplication isn't enough. It's also necessary to know how many non-replications have occurred but got swept under the rug. It's not the existence of replications that matter---it's the rate of replication relative to number of replication attempts. So I agree with the title "We Should Not Accept Scientific Results That Have Not Been Repeated". But I would add to it "We Should Not Accept Scientific Results from Studies That Weren't Preregistered". Registration of studies forces negative results to be made public, allowing for the positive result rate / replication rate to be calculated. Otherwise the existence of a "positive" result is more a function of the trendiness of a research area than it is of the properties of the underlying system being studied.
- yiyus 10y ago> The inconvenient truth is that scientists can achieve fame and advance their careers through accomplishments that do not prioritize the quality of their work An even more inconvenient truth is that scientists cannot even keep their jobs if they prioritize the quality of their work. The pressure to publish novel results is too strong and it is almost impossible to get any support for confirming previous ones.
- michaelbuddy 10y agoAgreed, which means 50% of social science at least is disqualified and should not be making into future publications or become part of curriculum.
- bshanks 10y agoI agree with the main point of this article but in terms of its analysis and prescriptions I think it gets two things backwards. (1) Most scientists seek fame as a means to the end of getting tenure and funding, not the other way around; if you gave them tenure (and the ability to move their tenure to somewhere else if they wanted to move) and perpetual funding and told them they could choose to be anonymous, I think many would choose that option. (2) Replication is not done/published enough because the incentive to do so (measured in: increase in probability of getting tenure per hour spent) is not high enough, not because people are overly willing to accept unreplicated work. In order for a lot more replication to get published, what would be needed would be for people who spent their careers replicating others' results (at the expense of not producing any important novel results of their own) to get tenure at top institutions (outcompeting others who had important novel results but not enough published replications).
- ramblenode 10y agoI have an alternative proposal: do a study right the first time. That means: A) Pre-registering the study design, including the statistical analysis. Otherwise, attaching a big label "Exploratory! Additional confirmation needed!" B) Properly powering the study. That means gathering a sample large enough that the chances of a false negative aren't just a coin flip. C) Making the data and analysis (scripts, etc.) publicly available where possible. It's truly astounding that this is not a best practice everywhere. D) Making the analysis reproducible without black magic. That includes C) as well as a more complete methods section and more automation of the analysis (one can call it automation but I see it more as reproducibility). Replication of the entire study is great, but it's also inefficient in the case of a perfect replication (the goal). Two identical and independent experiments will have both a higher false negative and false positive rate than a single experiment with twice the sample size. Additionally, it's unclear how to evaluate them in the case of conflicting results (unless one does a proper meta-analysis--but then why not just have a bigger single experiment?).
- physicalist 10y agoYour proposal is comparable to saying that checks and balances are not needed in a democracy, politicians just need to govern "right". This is about incentivising scientists to do the right thing instead of merely demanding it, like you do.
- ramblenode 10y agoHow is advocating for a new set of best practices any more "demanding" or wishful than a regime of obligatory replication? And how is this categorically different from current practices such as peer review, disclosing conflicts of interest, an IRB, etc.?
- colinprince 10y agoI searched for the word "tenure" in the article, but didn't find it. The drive to get tenure is a big reason that scientists publish so much, funny that was not mentioned.
- doug1001 10y ago"repeated" in this context is not incorrect, but i think "replicated" is perhaps a better choice. That aside, i think repeatability is a much more useful goal (rather than "has been repeated"). For one thing, meaningful replication must be done by someone else; for another, it's difficult and time consuming; the original investigator has no control over whether and when another in the community chooses to attempt replication of his result. What is within their control is an explanation of the methodology they relied on to produce their scientific result in sufficient detail to enable efficient repetition by the relevant community. To me that satisfies the competence threshold; good science isn't infallible science, and attempts to replicate it might fail, but some baseline frequency for ought to be acceptable.
- twoslide 10y agoOne problem is that there is no incentive to replicate. From the PhD onwards, academia creates incentives for original research. Replications, particularly those that confirmed existing research, would not benefit the researcher much.
- habitue 10y agoA big distinction here is that different fields have different levels of dependence on prior results. In fields like psychology etc, you don't need the previous results to work in order to run your own experiment. In other words, if you cite a well-known paper saying "people seem to work faster near the color red" and your paper runs an experiment to see if they work faster near the color yellow, if the red paper is later unreplicable, it doesn't change the outcome of your experiment in any way. In contrast, if you are in machine learning and you are extending an existing architecture you are very directly dependent on that original technique being useful. If it doesn't "replicate" the effectiveness of the original paper, you're going to find out quickly. Same for algorithms research. Some other comments here have mentioned life sciences being the same. So I think there's a qualitative difference between sciences where we understand things in a mostly statistical way (sociology, psychology, medical studies) where the mechanism is unknown (because it's very very complicated), but we use the process of science mechanistically to convince ourselves of effectiveness. e.g. I don't know why this color makes people work faster/ this drug increases rat longevity / complex human interactions adhere to this simple equation, but the p value is right, so we think it's true. Versus sciences where we have a good grasp of the underlying model and that model is backed up by many papers with evidence behind it, and we can make very specific predictions from that model and be confident of correctness.
- lutorm 10y agoDefine reproduced? Do we mean "conduct the same experiment multiple times so we can assess the variance on the outcome"? Or do we mean "conduct the same experiment multiple times to figure out if the first result is a screw-up"? Those two aren't the same, and I think far too many think that the point is the latter when, imho, it's actually the former. Pure screwups will likely get found out, just like glaring bugs are usually found. It's when your result actually has a huge variance but you're looking at only one (or a few) samples and draw conclusions from it that's insidious, like the fact that it's the bugs that just change the output by a tiny bit that are the hardest to notice.
- jernfrost 10y agoI don't get why this is not top on the agenda for the scientific community and the government. Huge amounts of research money is lost in repeating stuff that doesn't work. Huge amounts of money is lost chasing broken science. I blame this on the neo-liberal ideology. This intense focus on getting money's worth, on tying grants to specific goals, counting publications etc. Driving research exclusively on a very narrowly defined money incentive has driven us further into this sort of mess. The money grabbing journals which has prevented any significant innovation in how science is shared. I think what science needs is a model closer to that of open source. With open projects anybody can contribute to but where verification happens through personal forged relationships. The Linux kernel code quality is verified by a hierarchy of people trusting each other and knowing something about each others quality of work. Work should be shared like Linux source code in a transparent fashion and not behind some antiquated paywall. I don't think the grant system can entirely away, but perhaps it should be deemphasized and instead pay a higher minimum amount of money to scientists for doing what they want. Fundamental science breakthrough doesn't happen because people had a clear money incentive. Neither Einstein, Nils Bohr, Isaac Newton or Darwin pursued their scientific breakthroughs with an aim of getting rich. Few people become scientists to get rich. Why not try to tap into people's natural desire to discover?
- westurner 10y agoSo, we should have a structured way to represent that one study reproduces another? (e.g. that, with similar controls, the relation between the independent and dependent variables was sufficiently similar) - RDF is the best way to do this. RDF can be represented as RDFa (RDF in HTML) and as JSON-LD (JSON LinkedData). ... " #LinkedReproducibility " https://twitter.com/search?q=%23LinkedReproducibility https://twitter.com/search?q=%23LinkedReproducibility It isn't/wouldn't be sufficient to, with one triple, say (example.org/studyX, 'reproduces', example.org/studyY); there is a reified relation (an EdgeClass) containing metadata like who asserts that studyX reproduces studyY, when they assert that, and why (similar controls, similar outcome). Today, we have to compare PDFs of studies and dig through them for links to the actual datasets from which the summary statistics were derived; so specifying who is asserting that studyX reproduces studyY is very relevant. Ideally, it should be possible to publish a study with structured premises which lead to a conclusion (probably with formats like RDFa and JSON-LD, and a comprehensive schema for logical argumentation which does not yet exist). ("#StructuredPremises") Most simply, we should be able to say "the study control type URIs match", "the tabular column URIs match", "the samples were representative", and the identified relations were sufficiently within tolerances to say that studyX reproduces studyY. Doing so in prosaic, parenthetical two-column PDFs is wasteful and shortsighted. An individual researcher then, builds a set of beliefs about relations between factors in the world from a graph of studies ("#StudyGraph") with various quantitative and qualitative metadata attributes. As fields, we would then expect our aggregate #StudyGraphs to indicate which relations between dependent and independent variables are relevant to prediction and actionable decision making (e.g. policy, research funding).