64 ms·
Survey of scientists sheds light on reproducibility crisis (2016)
- tomlockwood 7y agoI wish more people would map their understanding of the reproducibility crisis to things like say, the sokal squared hoax. The current pressure to publish regularly in academia is very high, and it results in poor work and journals that don't do their due diligence, in every field.
- neutronman 7y agoThere is a startup called http://www.myire.com http://www.myire.com that is working on this issue. They have a platform that provides an A-Z approach to publishing research. Their marketing is probably the worst in class but the CEO is a passionate developer who has worked hard on this.
- vermilingua 7y agoReally doubt that this is an issue that can be solved by one startup. It's a procedural and cultural issue that affects the whole scientific world; not a problem that can be solved by business or technology.
- javajosh 7y agoOne factor that might increase reproducibility is to enhance negative reputational effects of publishing an un-reproducible study. That is, it should be so rare, so outré, to publish something unreproducible, your professional reputation is severely damaged, and if you do it again, you will leave academia. It sounds harsh, but severe self-regulation is what prevents the catastrophic failure of any institution that is really only answerable to itself. Professors aren't doing any favors by rubberstamping weak papers, or giving weak students PhD's (or BS's for that matter). EDIT: down-votes and no replies, as expected. Hey academics, sometimes to save the patient you have to amputate the limb. I care about science and want it to survive, so you guys continuing to be nice to each other even when you publish BS isn't serving that purpose.
- flukus 7y agoThe problem is not being reproducible is not necessarily a failure and does not necessarily indicate a bad actor, it may be bunk but it may show a worthwhile direction for others to investigate further. The problems come when an non-reproducible study gains some authority, citations in other studies or published in the media without proper vetting. It's probably worth punishing people that consistently publish UN-reproducible work, but it shouldn't a rare career destroying event like you are suggesting.
- javajosh 7y ago>not being reproducible is not necessarily a failure and does not necessarily indicate a bad actor I disagree with the first part and agree with the second. I would argue that failure to reproduce is perhaps the worst kind of scientific failure, because the activity cannot be called science anymore. And no, I don't mean anything personal when I say 'failure', you don't have to be malicious to do bad science -- after all 'doing bad science' is the default human condition. An analogy with classical music: A musician that fails to reproduce the score with his violin consistently, will lose his job quickly. It doesn't mean he's a bad person, but it does mean he's not good enough for the orchestra. The metaphor for the "reproducibility crisis" in an orchestra is what happens if they let bad players stay, the orchestra sounds worse and worse, and finally the audience stops coming. In the apocalyptic scenario, cultural forces cause ALL orchestras to stop honestly evaluating the skill of their players, and all orchestras simultaneously lose their ability to accurately play any but the simplest music. Standards are painful for those that can't meet them, but they make the world better, overall.
- inimino 7y agoYou can judge a musician in a 20-minute audition, but you can't tell if research will reproduce without doing the work. However, there are papers that are so vague and poorly written that they cannot possibly be reproducible, and this is becoming the norm in some fields, so I think you're right that standards would help there. Also, sometimes strange things just happen randomly (like the CERN faster-than-light neutrinos) and there's no shame in publishing results that aren't correct as long as you're honest about it. Some early-stage, low-power studies are going to randomly show results that don't reproduce, and that's also fine. Suppressing these results would hurt progress just as much as making the opposite error.
- MilnerRoute 7y agoThis is from 2016. Can we get one of the moderators to add (2016) to the headline?
- tlb 7y agoDone, thanks.
- macawfish 7y agoMy feeling is that this reproducibility crisis points to deeper stuff just waiting to be collectively understood about how to use math in experimental sciences. So many researchers rely on assumptions of linear dynamics. So many experiments and studies are designed without consideration of observer effects. Is it any wonder that a model might fit one day but not the next? There is so much to be learned from applied topology, dynamical systems theory and (quantum) information theory, but methods from these disciplines are only just barely starting to become more widely accessible.
- skybrian 7y agoMaking the model more complicated doesn't add external validity. It seems like the problem often comes from trying to make do with collecting a minimum amount of data, collected in the most convenient way. (For example, doing experiments on local college students.) It seems like a practice of at least collecting the data at two different colleges might help? But this would require more coordination.
- JorgeGT 7y agoThe whole, complex scientific field of metrology exists in order to make physical measurements anywhere on Earth comparable in a meaningful way. Trying to derive conclusions about the psychological traits of humans by observing n=200 of your local college students certainly seems... rather dubious.
- macawfish 7y agoJust because alternative methods aren't widespread, it doesn't necessarily mean they are "more complicated", just that rely on different assumptions. I'm really excited, for example, about "model-free" time series methods emerging from non-linear topological data analysis. Takens' embedding theorem shows that low dimensional "shadows" of high dimensional attractors can be constructed from a time series alone, and that they can reveal deep, useful facts about system a time series is part of, even if that system defies geometric modeling. These kinds of methods are radically different from conventional statistical methods. see here: https://www.youtube.com/watch?v=NrFdIz-D2yM https://www.youtube.com/watch?v=NrFdIz-D2yM Previously, researchers found themselves flailing around with geometry, trying to come up with models that accurately described the dynamical nature of the systems they were studying (e.g. fish populations). A model might work well for a few years, then stop working. These topological methods cut through and get straight to the underlying relationships that drive various observables in a complex system, without relying on fickle geometric assumptions about how exactly those relationships will be expressed.
- jokoon 7y agoI guess it's time to evaluate and grade scientific findings? It would be interesting to see how reproducibility varies by field, university, country, etc. Although I guess scientific reputation would already give a clue over the quality of scientific work?
- keldaris 7y agoAs a working scientist, I feel both sides of the problem every day. Most papers I come across turn out to be difficult if not impossible to reproduce, and I'm sure some of my own papers have fallen into that group at some points (although given that I'm a theorist, what "reproducible" means can be a bit fuzzy sometimes). At the same time, I'm confused every time I see people wondering about the scale of the problem or what to do about it. There is absolutely no mystery here whatsoever. Scientists are generally fairly smart people. Put smart people in a hypercompetitive environment and they will quickly identify the traits that are being rewarded and optimize for those. What does academia reward? Large numbers of publications with lots of citations, nothing else matters. So, naturally, scientists flock to hip buzzword-filled topics and churn out papers as quickly as they can. Not only is there absolutely no reward for easy reproducibility, it can actually harm you if someone manages to copy you quickly enough and then beat you to the next incremental publishable discovery in the neverending race. This is absurd and counterproductive to science in general, but such is the state of the academic job market. Anyone who purports to somehow address the situation without substantially changing the fundamental predicates for hiring and funding in the sciences is just wasting their breath. As long as the incentives are what they are, reproducibility will be a token most scientists pay tribute to in name only.
- haecceity 7y agoHow to fix?
- aalleavitch 7y agoScience shouldn't be funded this way
- keldaris 7y agoThe answer is simple, but it's also one that's completely useless for anyone reading this post. What needs to happen is a gradual realignment in both the funding and hiring criteria across all of academia. There needs to be less emphasis on quantitative metrics like citation counts, h-indices and other trivially gameable nonsense, and more emphasis on human judgment along with a set of basic criteria to provide a floor to research quality across the board (things that every good scientist should do, as opposed to things that you should relentlessly maximize). This would require the non-scientific managerial class to abrogate some of the power they now wield in the academia and relevant funding institutions, therefore is very unlikely to happen. And - probably, most annoyingly to the HN crowd - there is nothing meaningful that technological or business disruption can accomplish here. These are systemic social problems that have been continuously and systemically building up since (approximately) WW2. At this point the only entities large enough to make a difference are probably the NIH and NSF in the U.S. and the relevant EU funding agencies in Europe. In my judgment, these are precisely the agencies least likely to institute such change, so here we are.
- stewbrew 7y agoIMHO the simple problems with reproducibility (of statictical studies) can be solved with pre-registration, which should become a standard. The rest really is a system crisis. Imagine: Scientist A uses (a statistical) method M to assert X. People praise A until scientist B uses method M and cannot assert X (which BTW usually doesn't imply X is wrong). People wait for scientist C to use method M and re-check assertion X. People then take the majority vote (or meta-analysis) for true. That's the way much of science works today. It becomes a problem only in the following situations: 0. People hunt for statistically significant results. 1. X is mostly irrelevant to anyone who does not belong to the in-group of A, B, C, and peers. So nobody else really notices or cares about X. 2. There is no other way to observe any meaningful consequence of X other than by method M -- which eventually results in #1. 3. Method M is really expensive and complex, it's an almost impossible undertaking so that B or C most likely won't get a funding. 4. Everything is ok, actually, but the people who pay for the study don't care about A's methodological fineprint as long as the results play well with their other goals. The presented list of "factors that build reproducibility" focuses on #0, for which pre-registration is a simple and clean solution. IMHO the list is much to narrow and focused on academic practice, though.
- Gatsky 7y agoHmm... 40% of irreproducibility is due to fraud? Who in their right mind would put in all the effort that science requires if they think everyone else's work is commonly malicious bullshit? Obviously this survey is biased towards people with a certain outlook.
- fullshark 7y agoAcademics thinking little of other academics is not surprising.
- filleokus 7y agoKinda-OT but: During my studies I've read papers from everything to organisational theory (MBA stuff) to pretty "pure" CS (like compiler construction etc), and have in that process read so many papers in many different fields that just don't provide any insight or are barely comprehensible. Of course, sometimes it can probably be blamed on the reader (me) for not having enough insight, but many times the papers are just crap. My naive imagination of academic papers before university was that they were like those I now read in the __top__ journals / conferences in each field: concise understandable writing with actual insights and clear methodology that can be repeated. But papers like these are probably something like less than 1% of everything published. Many papers seem to be almost approaching levels of the Bogdanov thesis's [0]. To relate to the topic: I'm wondering if we'll see a shift in the academic culture and system, where the pressure to publish is lowered (somehow), and where focus is on quality and actually providing insight and reproducibility. The worst possible thing is that bad science gets influential on false grounds, which needs to be avoided. But increasing the signal-to-noise-ratio is probably not a bad idea either. And I suspect that they will both turn out to be improved by fixing the underlying incentive problems. [0]: https://en.wikipedia.org/wiki/Bogdanov_affair https://en.wikipedia.org/wiki/Bogdanov_affair
- jacquesm 7y agoIt's the same as with programming: many people now have a career in science because that's something that comes with prestige rather than that they are driven by their curiosity and need to understand how stuff works. Once established the treadmill needs to keep turning so you get tons of low value papers and the occasional gem. Publish or perish does nothing to improve the quality of what is published, rather to the contrary.
- allovernow 7y agoThe bar for publication (like much of our media) used to be far higher. Particularly with the push to get everyone through college, even at the cost of lowering standards, there's a sort of regression to the mean with respect to publication quality. It happens with any difficult, technical field that initially attracts the more capable in society. Hell, look at what the internet was back when it was still mostly needs with passion.
- ecmascript 7y agoMaybe because some of the "science" today is more about spreading some political view rather than anything else. For example, there are a lot of gender studies that are presented as a science in my country which I believe is complete bullshit and an ideology. People are pushing shit like that today and I believe we need more of the hard sciences and less of the soft sciences.
- ne0ncowb0y 7y agoWorth mentioning that scientific research, even if difficult to reproduce, is still better than, say, superstitious crap. Looking at you, chem trail antivax climate deniers.
- hyperdunc 7y agoIs it though? Some ideas are so obviously nonsense to most of us, whereas bad research can have a more robust veneer of legitimacy - which makes that research a potentially useful tool for bad actors. The internet is full of authoritative sounding tripe backed up by "science".
- asdff 7y agoBad research doesn't usually hold up. Even good research doesn't always hold up over time. Was Bohr a bastard for getting the atomic model wrong, or just offering his best interpretation given what was known at the time?
- pdonis 7y ago> Bad research doesn't usually hold up. Your faith in the reliability of the scientific process is touching, but naive. Bad research is driving public policy in many areas. > Even good research doesn't always hold up over time. In the sense that it gets superseded by better models (as Bohr's original model of the atom did), sure. That's to be expected in a healthy scientific field. But one of the things that's needed to keep a field healthy in that respect is the ability to do controlled experiments with high accuracy. That's how scientists figured out that Bohr's original model of the atom wasn't right--it couldn't match the results of the experiments as they got more accurate.
- SlowRobotAhead 7y agoVery strange you have a new account railing on “deniers” in regards to a topic that even what should be moderate complexity experiments are failing to replicate let alone the amazingly difficult models in climate science. It’s not helping whatever point you think you are making. Instead... consider the idea that WHAT IF climate models were failing reproducibility left and right along with the other sciences - should that mean a policy for a global tax model should be pushed as hard as it is? You don’t have to be a “denier” to be skeptical of the course that the “true believers” are taking. Some of us think the “climate religion” is a bad thing for a good cause. Long term counter productive.
- chrisbrandow 7y agoAs a chemistry PhD, this sure looks a lot like the reproducibility is directly proportional to the complexity of the underlying phenomena being studied.
- pochamago 7y agoIs there room for a genre of "failed to reproduce" journals? Contrarianism seems like it might be powerful enough to support a journal whose only purpose is to dispute major findings.
- oldgradstudent 7y agoThe problem is that's it's very easy to fall to reproduce a result. Maybe even easier than the p-hacking used to produce an irreproducible result.
- m3kw9 7y agoIs there an incentive to make certain experiments harder to reproduce.
- JorgeGT 7y agoOne explanation that I've heard (often in terms of global south v. global north) is to prevent more well-funded labs to quickly mirror your setup/method and, given their higher means/staff, deplete that line of research leaving your lab with no publications, patents, etc. and therefore no money. Science as a whole may benefit, but individual scientists still usually appreciate keeping their jobs.
- kzrdude 7y agoWhat you describe has a perfect analogy in how established actors squeeze out smaller actors in a race to the bottom in other areas of the economy. So it's certainly great that we don't have that in science, yet..
- asdff 7y agoBig labs don't work any faster than small labs, they just have more people to chase more projects; generally it's the same amount of people per project as a small lab. Chances are if you got scooped it was just because someone got started with the idea before you did. After all, the future directions at the end of the paper are usually active research by the time you are drafting and submitting for review. Getting 'scooped' isn't so bad in biology at least. If you were thinking about XYZ and someone publishes X and Z, great. Cite that paper, now you don't have to bother so much validating X and Z and can focus on strengthening your Y argument or adding argument W to your paper.
- JorgeGT 7y ago> Big labs don't work any faster than small labs Not true, at least in my field. If $BigLab can throw 100k CPU cores at a simulation it is going to finish way faster than $SmallLab 1k simulation. If $BigLab builds 10 dedicated test rigs they are going to finish that parametric study way waster than $SmallLab with 1 shared test rig. Getting scooped may not a big problem for papers (except when you start getting rejections because lack of novelty), but it is a big problem for patents, licenses, royalties and such things that allow small labs to survive when their governments have little to no money to spend.
- allovernow 7y agoI really think there's lost opportunity in the typical American master's program. Typically for a master of science you need to perform original research for your thesis, ostensibly to prepare you for the process of publication. But I think a master's could be a far more useful stepping stone between undergraduate academics and PHD or industry research if it were taken as an chance to have graduates reproduce important research. The grade will be based on the merits of the reproduction, which will be an excellent final introduction to research work, while also hugely benefiting the community by verifying previous research. Right now there simply isn't any incentive to reproduce anything but the biggest results.
- m0zg 7y agoAnother thing I'd like to ask from my fellow colleagues: please at least to some extent detail in your papers the things that you've tried that didn't work. I see this in my field (computer vision / deep learning) from time to time, but very rarely. There are typically ablation studies which aim to determine how much each of the _successful_ improvements contributes to the result, but there's almost never any mention of things that looked promising on paper but didn't pan out in practice, nor is there any discussion of the reasons why, even though the authors often have a good idea post-facto.
- mensetmanusman 7y agoThis is why I want to industry after MIT, science is really real when it manifests as a useful technology that someone will pay for. Also, I didn’t want to spend so much time writing grants...
- cft 7y agoNote a spike in Earth science at 30% confidence. Probably reflects a lot of vested interest funding politically beneficial results in this discipline.
- agumonkey 7y agothere were a few articles quoting the use of software like git, simple formats, Jupiter notebooks, etc etc is there anything of the sort that helped a bit ? very lightly funny to see this when nix and guix (and pure fp) poping at the same time in a very different context.
- TheRealNGenius 7y agoFrom my experience, math/computer-science type research is almost always exactly reproducible.
- asdff 7y agoIt would be surprising if it weren't. You are rerunning a list of instructions in a common syntax on common equipment shared among everyone. Every instruction and line is documented somewhere by someone. Imagine if programming languages were a thing produced by nature and no one knew any of the commands, and new commands were found by just testing them out or theorizing that they might exist. You'd be getting the nobel prize for proving ls can take arguments. Biology is not a clean science. Results are noisy. Results can be different depending on the manufacturers batch of the reagent.
- dragonwriter 7y ago> math/computer-science type research is almost always exactly reproducible. Math, including computer science, is almost entirely pure logic, not empirical science (despite the name of the latter), so it's not even in the same epistemological domain where reproducibility is conceptually a concern.
- naveen99 7y agoOn the other hand we have the government saying they don’t want other countries reproducing our results in AI, cryptography, genetics...
- JumpCrisscross 7y agoIs a 30% reproduction rate unusual? We know 19th-century science was productive. Was it similarly plagued by irreproducibility? I’m sceptical of this argument. But we need a baseline to render judgement.
- analog31 7y agoReproducibility is referred to as a "crisis," but I'd like to know if it's really a new thing. What if we tried to replicate the studies that surrounded great developments such as electromagnetism or thermodynamics. Did those advances emerge from an unbroken series of reproducible studies, or from a tangle of good and bad results? Before looking for root causes that invariably turn a cynical eye towards the motivations of scientists, let's make sure the effects don't precede the causes.
- javajosh 7y agoI really like this idea of having students (not just advanced ones) reproduce classic experiments (ideally before getting the theory, so the class can attempt to puzzle out what's happening). My sense is that early science was very hands-on and readers of such results were also doers. Plus, doing the experiment before theory puts them in the same spot as the scientist, and might give them a little more respect for an achievement that is really quite magical (and too often is taught in a kind of arrogant hindsight that implies the discovery is/was sneeringly obvious to any human that draws breath)
- sabas123 7y agoAs somebody only used to CS research, can any explain to me how much the costs of reproducing is considered into unreproducibility?
- lowdose 7y agoHow would the state of social science progress if every academic had all Facebooks data available to them?
- throwno 7y agoWeird how there can be a reproducibility crisis, but also climate change is 100% unquestionable science.