22 ms·

The Irreproducibility Crisis of Modern Science

270 points by vixen99 9y ago

JohnL4 9y ago
I think Planet Money did an episode on this a few years ago. Good stuff, and maybe easier than reading a paper. :) https://www.npr.org/sections/money/2018/03/07/591213302/episode-677-the-experiment-experiment https://www.npr.org/sections/money/2018/03/07/591213302/epis...
- Afforess 9y ago
  The report (not a paper) is actually written for a lay audience and very readable: https://www.nas.org/images/documents/irreproducibility_report/NAS_irreproducibilityReport.pdf https://www.nas.org/images/documents/irreproducibility_repor...
  - HarryHirsch 9y ago
    It's a position paper from a conservative think tank and needs to be treated as such.
    - ItsMe000001 9y ago
      > and needs to be treated as such What does that even mean? You have one point "it's a conservative think tank" - which does not mean anything - coupled with another empty statement. Now I sure don't mind criticism of that entity, but I find it ironic that it is done with such an empty statement that's just a not very cleverly disguised type of ad hominem attack. May I suggest you take some time actually reading the paper and then coming back with criticism of their actual points?
      - samfriedman 9y ago
        It is absolutely relevant to consider the political motivations of publishers of reports like this, when the publisher is a politically active organization as opposed to an independent research group. While the report itself may not push an obvious partisan agenda, a critical reader should consider the conservative agenda of the NAS in relation to why they might publish such a report. In the conservative wing's long-standing opposition to science, it's not far-fetched to view this report as being published tactically -- even if all of its points are well-founded -- as something to point to when future efforts attempt to undermine science-based policy on issues like climate change, stem-cell research, abortion, etc. Especially when the President of the NAS is a public "climate skeptic". https://www.chronicle.com/blogs/innovations/bottling-up-global-warming-skepticism/29754 https://www.chronicle.com/blogs/innovations/bottling-up-glob...
      - HarryHirsch 9y ago
        A position paper aims to make a political position appealing, a study about the sociology of science on the other hand would seek explanations for a position. Pointing out the purpose of something isn't a character attack.
    - Afforess 9y ago
      Ad hominem has never been a valid reason to disbelieve something. I quite enjoyed reading the report - it includes XKCD references and everything.
      - jnordwick 9y ago
        That you're being voted down shows that HN has really changed. (I blame all the overtly political articles that I believe have changed the tenor of the site, and politics has infected every article now. Thanks @dang!)
        sctb 9y ago
        If you won't stop posting this off-topic meta noise that we've asked you many times before to cut out, we'll ban the account. Post according to the guidelines or not at all. Fulminate elsewhere. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
    - kiziaru 9y ago
      What makes it a conservative think tank?
      - wRastel27 9y ago
        From the intro: The new, alternative view, was that college and universities should be places where fresh ideas untrammeled by hidden connections to the established structures of power in American society should have the chance to develop themselves. In practice this meant a hearty welcome to neo-Marxism, radical feminism, historicism, post-colonialism, deconstructionism, post-modernism, liberation theology, and a host of other ideologies. Also from the intro: The National Association of Scholars (NAS) has long been interested in the politicization of science.
Retric 9y ago
Being reproducible is only critically important if people treat individual studies as meaningful. That IMO is a far more dangerous stance. Any study can have hidden flaws, none should be trusted without some form of replication.
- mylons 9y ago
  not sure what you're getting at. are you saying meta analysis is more important than individual studies? but if a large % of your studies in the meta analysis can't be reproduced, what faith do you have in the meta analysis at all?
  - aleksei 9y ago
    Take into account that meta analysis should not only consider the results of its participant studies, but also whether there is enough information to reproduce the study and whether it has accounted for possible confounding factors.
- rplst8 9y ago
  How often is a study even repeated in the course of normal scientific research? I think most studies are focused on expanding the research and therefore knowledge about the science being studied. Accepting previous studies as fact is dangerous and could easily lead to a house of cards scenario. I think this is especially true in the very specific, niche studies that are common in today's highly competitive graduate student and publish or die research landscape.
  - lotsofpulp 9y ago
    That is why, as a (global) society, we should spend our resources in making sure the data we have is accurate by funding research and repeated studies, and that data should be open to all to verify and build upon.
    - jbob2000 9y ago
      I think we're passed the point where studies can be reproduced effectively. I actually think the scientific model is falling apart. We've discovered the easy things, and those studies were easy to reproduce. We built a model of science around the low hanging fruit. How are you going to reproduce a study about a cancer drug? Synthesizing the drug itself is a huge endeavor, never mind finding a group of participants with the exact cancer you need. How would you reproduce the Large Hadron Collider experiments? You'd need to rebuild one of the most technically challenging, most expensive scientific instruments ever made.
rplst8 9y ago
This is the exact reason I don't believe science should be used blindly to affect government policy. It flies in the face of democracy and is subject to falsehoods just like any other human endeavor.
- jboggan 9y ago
  Lysenkoism happens in the American and European scientific establishments as well. As an example I would suggest Ancel Keys and the cholesterol hypothesis of heart disease.
  - marcosdumay 9y ago
    That's a bad example. It's a much more complex scenario, where the drugs that fight the expected cause of the disease does in fact help protecting against it. Confusing causality is very hard to solve.
    - mattnewport 9y ago
      If you're talking about statins, Ancel Keys was promoting shoddy science on high fat diets to influence government policy before statins were discovered.
- tlb 9y ago
  Who is the strongest proponent of using science blindly to affect government policy? What are their main arguments, and how do you disagree?
  - AstralStorm 9y ago
    What do you mean by using science blindly? Three is NO way to do that. There are ways to misuse scientific results. Premature generalisation, assumptions of safety. Cherry picking to support preconceptions. Attributing strong support to tentative and/or unchallenged results. And more...
- lmm 9y ago
terminado 9y ago
I mean, this is still a good thing, since it charts a map of decidedly grey areas, where information may always be ambiguous, and useful information needs to be sussed out carefully. It's better than presumptively assuming that "Science" is an infallible always black-and-white.
quantumwoke 9y ago
Recently I have been beginning to question university education. It seems that today's education system has mismatched priorities which may have an impact on the reproducibility crisis. You see, to come up with a fantastic new alloy, you don't have to be a genius. Or even very good. You have to be persistent, and clever enough to realize when you've found something interesting. You will need to understand your field very well, but there are many, many fields. And each one is quite narrow, so while it is hard to understand your field well, many, many people can do this. So my less dystopian future goes like this. Train lots and lots of physicists, chemists, engineers, programmers, materials scientists, biologists, mathematicians. And (shock horror) through higher taxes employ them. Employ them on promising ideas, and on impossible ideas. Employ them on the problems of the day, and on arcane research that will almost certainly never bear fruit. And sure, right at the top in the Apples and Googles of the world, you'll have the best and brightest pulling together all of the discoveries made without all these reproducibility issues. Or we could continue down the dystopian path. The low tax path where there really are only meaningful jobs for the best and the brightest. Where we rely on Apple and Google to decide what ideas are worth pursuing. Where the skills and abilities of a great many are not used. And most of us fight for the crumbs from the oligarch's tables. And the oligarchs? For every Elon Musk, there will be 10 who just want the biggest yacht and the prettiest mistress. Let me say that again. In a low tax environment, the skills and abilities of a great many will not be used. And we will all be poorer for that. Genuinely poorer, because great discoveries will not be made. And it is happening now. And I'll go one step further. We are also wasting the talents of our best writers and artists. The BBC and other national broadcasters are squeezed. Squeezed so that the idea of something as original and amazing as Monty Python appearing now is laughable. And again, we are all the poorer for that. Not just the lucky ones who get to make amazing shows or works of art, but all of us who never get to revel in them. The low tax, small government world is a step backwards. If you wish to sum it up succinctly, you may say it is a world where, in order to placate the very rich, we take away opportunities from many, and impoverish the world as a whole.
- emodendroket 9y ago
  > Recently I have been beginning to question university education. Why are we bothering to educate anyone except the very, very best students in physics, maths and engineering? In fact, why aren't we identifying these students and sending them straight to Caltech? It seems intuitive that this would solve one half of the reproducibility crisis. That doesn't seem intuitive at all.
mistrial9 9y ago
Some of the comments miss an important distinction.. Science in the popular press, often refers to science versus completely non-science ways of forming an opinion or deciding policy. Meanwhile, science within a technical community, is subject to human error and manipulation, and relies on a reproducible result, as well as peer-review, to find answers to conflicting claims. There are certainly non-Science ways of forming an opinion and deciding policy, in many cases totally legitimate. But once science is claimed, then of course it has to be subject to science rigors.
- m-watson 9y ago
  Yes, thank you for saying this. I feel like, often, people use science as some ethereal being. It is a process that has to be used. If that process isn't happening it isn't science. That doesn't mean everything has to be science, but things that are science should be held to a certain standard but should also be taken with the extra weight if that standard is met.
  - Bograff 9y ago
    It's important to make the distinction between Pure Science and anything on the sliding scale from technical description to ELI5. There are levels to it, you and I know.
    - m-watson 9y ago
      Oh absolutely there is a sliding scale on both the actual practice of science as well as the reporting. From the practice of it, I know as simple anecdote I have participated in conducting research in hard science (physics), softer hard science (biology), social science (behavioral and economic), as well as straight policy analysis with a "scientific approach" through data science. The findings from those are reported differently and have stronger or weaker words and conclusions. I think that is where the practice sliding scale comes in, is how the researcher decided to write up and report their findings and appropriately identifying their methodology and short-comings. From the journalism side I think you end up with the problem the wide spread field of journalism has, you have to be read by your audience. That means something with a wider audience is probably going to be more on the ELI5 side which means both the reader and the journalist have to take any reported findings more lightly. The sliding scale, I believe, comes in with the rigor something is reported. That is both in the practice and the journalistic side. More details can give a more granular understanding showing limitations but some times broad strokes are necessary to get some information out to a wider audience or more quickly.
tonystubblebine 9y ago
We did an article trying to help people be better critical thinkers when they hear about psychology research. I thought the author did a good job getting into how these studies go wrong and also the types of magical results you should be suspicious of. https://medium.com/@jhreha/most-psychology-research-is-bs-73d4793b4dc6 https://medium.com/@jhreha/most-psychology-research-is-bs-73...
SubiculumCode 9y ago
We know that weak classifiers can be bagged to produce a strong classifier al la Adaboost. Each study is a weak classifier and would have a 'reproducibility crisis' if retested on new data. However after lots of studies of similar phenomena, a strong classifier emerges. In the field, we call this converging lines of evidence.
- scottfr 9y ago
  I'm not sure I really agree with this, because in science studies are biased towards aligning with the results of previous studies. This can be due to many factors: authors' expectations, journals like to publish positive over negative results, peer reviewers will look more critically (in an analytical sense) at work that disagrees with accepted research, etc.... It can be very hard for many reasons to stand up and say "this prior work is wrong".
  - soVeryTired 9y ago
    I know someone who quit her PhD in part because she couldn't replicate someone else's experimental work on rat behaviour. If the replication crisis had been as well-known back then, she might have fared better.
  - SubiculumCode 9y ago
    The argument is not that all is perfect in science. It is not. The argument is that strong replication for each study is not required for progress. Back to my analogy to the algorithms like Adaboost. Why ever aggregate over a a bunch of weak classifiers over a single strong one? Well, please correct me if I am wrong (machine learning is not my field), but the primary advantage is computational cost. Sometimes, for the same overall classification performance, training hundreds or thousands of weak classifiers requires fewer computations than training the strong classifier. To me it brings up an interesting point. If we view experiments as classifiers, then how would a machine learning expert set science policy and practice? Also viewed this way, it makes me wonder about p-hacking, which increases sensitivity while increasing false positives. Since negative results are not generally reported, I wonder whether p-hacking diminishes replicability at the study level for efficiency at the aggregate level. This is an empirical question, as ethically it is of course a dubious practice under current understanding.
timtadh 9y ago
In response to several threads here: it is important to distinguish when scientists are self critical vs. when non-scientists are critical of the scientific method. For instance, there is a long history of scientists criticizing how the scientific process is currently conducted for the purposes of improving the scientific endeavor. That work is sometimes used by non-scientists who question the overall scientific method. However, such use is invalid as the scientific self-criticism 1. assumes the validity of the scientific method 2. relies on the scientific method as its critical lens Whereas those who critique science as a whole: 1. assume that the scientific method does not work and does not arrive at "truth" 2. then use scientists being self critical to prove #1. Such a "proof" does not work as there is its uses the assumption "the scientific method arrives at truth" to derive the contradiction "the scientific method does not arrive at truth". See for instance comment: https://news.ycombinator.com/item?id=16859200 https://news.ycombinator.com/item?id=16859200 In reality, work on reproducibility is about improving the practice of science overall. It does not in itself show that science is inherently untrustworthy. What it does show is that scientific discovery is difficult and it takes a lot of effort and new findings should be treated critically. What does critically mean in this context? It means with in the boundaries of science analyzing the theoretical basis, hypothesis, method, and experimental results for potential flaws. It does not mean to be skeptical as a default because science "doesn't work."
- 323454 9y ago
  I agree with your overall point, but technically speaking it is logically valid to prove a hypothesis false by first assuming it and then deriving a contradiction, even when the contradiction is the negation of the original hypothesis (as it is in your example). What you should have said is that some critics start with the premise "the scientific method does not arrive at truth", and then use other people's arguments that depend on the premise "the scientific method arrives at truth" to support their claim, which is indeed logically invalid.
- buvanshak 9y ago
  >scientists criticizing how the scientific process is currently conducted for the purposes of improving the scientific endeavor. I think what happening here is a bit more serious. They are showing a widespread crisis. It is not just some minor feedback to improve the process. >It does not in itself show that science is inherently untrustworthy. I think when statistics is involved, the results are inherently untrustworthy. This is not really surprising because there is a whole bunch of ways these studies that involve statistics could go wrong. And we are still finding new ways on how this could go wrong. Then there are things like publication bias, that takes this to a whole new level. Things like that means that a biased body of journals can project any consensus that it favors just by selecting studies that fit its narrative. The inherent issues with statistics means that you can find studies that shows any possible outcome.
lisper 9y ago
The site is down so I can't read the original report, but I've read reports on this topic in the past so I'm going to chime in with some "usual suspects" caveats: 1. No result is 100% reproducible because you can never completely reproduce the conditions of any experiment. The best you can hope to do is to reproduce the conditions that matter, but enumerating those has to be part of the theory you are testing, and so you can never be 100% sure that you have a complete list. 2. Even a completely non-reproducible result can be scientifically significant. For example, celestial events are almost never reproducible. Our understanding of celestial mechanics nonetheless rests on solid science. 3. The end-product of science is not truth, it is explanations of observations. Those observations can (indeed must) include non-reproducible ones. Sometimes the explanation of non-reproducible results is "experimental error" or "delusion" or "we just don't know." But non-reproducible events are nonetheless within the purview of science. On the other hand... 4. The statistical tests currently in widespread use as a criterion for publication in peer-reviewed journals guarantee that at least one result in 20 will be due to chance and not because the hypothesis being tested is actually true. That, combined with the suppression of negative results, guarantees that the results published in scientific journals that adhere to those standards will be unreliable. But that doesn't really have anything to do with reproducibility per se, it has to do with the fact that the journals use a weak criterion for defining positive results. That, combined with our human predilection to value positive results over negative ones and the understandable desire of scientists to advance their careers, all but guarantees that journals will contain many defensible but false results. But this is not because of a lack of reproducibility per se, it's because of poor policy choices.
- Erlich_Bachman 9y ago
  4.) You mean that papers are published with P<0.05? When did this become a "standard"? Most papers I have seen in respectable journals and other sources would not strive for such a high Pvalue, it would be much lower like 0.001.
  - lisper 9y ago
    > You mean that papers are published with P<0.05? I'm guessing you meant to say that no papers are published with P<0.05. And of course they are, but that's irrelevant. What matters it that many journals, especially in "softer" sciences (i.e. not physics) use p<0.05 as their threshold, which guarantees that at least one paper in 20 will be "wrong". Then add self-suppression of negative results and the actual percentage of "wrong" papers becomes higher, potentially much higher if real positive results are rare. > When did this become a "standard"? I have no idea. You'll have a ask a historian.
gringoDan 9y ago
One of Slate Star Codex's top all-time articles discusses this very issue. Highly recommend: http://slatestarcodex.com/2014/04/28/the-control-group-is-out-of-control/ http://slatestarcodex.com/2014/04/28/the-control-group-is-ou...
- jnordwick 9y ago
  > On the meta-level, you’re studying some phenomenon and you get some positive findings. That doesn’t tell you much until you take some other researchers who are studying a phenomenon you know doesn’t exist – but which they themselves believe in – and see how many of them get positive findings. That number tells you how many studies will discover positive results whether the phenomenon is real or not. Unless studies of the real phenomenon do significantly better than studies of the placebo phenomenon, you haven’t found anything. This is such an astute observation that reproduction studies can be used to find a placebo control group for bad scuence. Parapsychology was one such group, but now we can find many others. Brilliant.
arafa 9y ago
What I'd like to know is if the "irreproducibility crisis" is really some combination of "sample size was too small" and "effect size was too small". When I went through a lot of these studies myself and saw the ones that don't reproduce, I saw this theme over and over. "P-hacking" is less of a concern to me when the effect is real and widespread. It's so bad now that for any article/study, I look at the sample size first. If it's too small (especially < 100) or they don't say I just ignore it. And if you don't publish or give some estimate of your effect size I just think about it directionally but don't give it much weight mentally.
- hnuser1234 9y ago
  I'm not sure this captures the whole problem. Sure, huge sample sizes are great, but look at the current top comment by lisper - we have gained a boatload (that's a scientific term) of insight from single events. LIGO is currently rocking a sample size of 7 but I doubt you ignored it.
  - arafa 9y ago
    Of course there are exceptions and what I said doesn't capture all cases (like with massive fixed costs or unique kinds of studies), you're right. I find it's a pretty good rule of thumb, though.
- amatic 9y ago
  Huge sample studies with high stat significance can have very low effect sizes, though. If an effect is shown on a small sample, it means it is a really big effect, and will likely be reproduced in a bigger study.
  - arafa 9y ago
    It's a combination of sample size and effect size. The larger one is, the more I'm willing to give on the other.
thedonkeycometh 9y ago
This is part and parcel of Scientific method. I'd be suspicious if all experiments were successful all the time. eyes pharma
danharaj 9y ago
Lets take a page from Marx. Science is many things, in particular a relationship between capital and labor. The scientific method is a wonderful idea, but it is subordinate to the economic forces that underlie scientific activity. Look at the conflicts and contradictions between those doing science (labor) and those deciding the science to be done (capital), and that is the ultimate source of these crises. The executive summary lists 40 points on how to improve the reproducibility of science. A bit over half of them are addressed to the sources of capital such as private organizations, universities, and governments. I think many of those points are good. However, I don't think the other points, the ones that recommend doing science in different ways, have a good punch. Even if you fix the problems there are today, so long as science is a rat race of trying to get grant money to stay afloat while burning out grad student after grad student, I think other pathological practices will creep in as a completely rational response on the parts of scientists to a hostile ecosystem. There's just a very big gap between how science should be done and what capital owners want from science.
- deleted 9y ago
  [deleted]
- TangoTrotFox 9y ago
  I think this is pretty self evident, but the issue is that let's say you have some system where economic forces are removed. Essentially a researcher basic income in one scenario. This would suddenly massively incentivize people towards this direction since it's basically a career path that guarantees a stable livelihood, which is something that's extremely rare today. Well you need to ensure there's nobody just completely gaming the system and so inextricably you'll end up with some sort of qualifier for results. And now suddenly you've done nothing but kick the can since this new qualifier is what's going to be gamed. Maybe the biggest problem is what you mention, but in another direction. For whatever reason there seems to be extremely little interest in the private direct funding of science. In times past the aristocracy would often fund scientific research on all sorts of topics. Today the practice seems to have all but disappeared, certainly if we measure relative ratios of the practice.
  - 9y ago
btilly 9y ago
The article may be good, but the source has a definite bias. See https://www.sourcewatch.org/index.php/National_Association_of_Scholars https://www.sourcewatch.org/index.php/National_Association_o... to see what that bias is. That said, they should be well-informed on the subject. So it probably does contain a lot of good information.
flamedoge 9y ago
There is the problem of conflating science with mathematics.
blueprint 9y ago
"In order to be a true scientist, you should be familiar with philosophy, first"
mathgenius 9y ago
I have met several "refugees" from particle physics, that left in part because of flimsy statistical methods. So let me ask: who is going to reproduce the experiment that found the Higgs boson?
rdlecler1 9y ago
Maybe papers need Yelp reviews on reproducibility. 1-Star couldn’t reproduce.
makecheck 9y ago
There has to be similar prestige/career-building/notoriety/funding for spending time on reproducing the experiments of others. Without that shift there will clearly be a greater tendency to just try something new. Also, when experiments depend on source code, etc. we need real engineering tools/principles applied. (Something like: “you can’t publish paper X if you aren’t including a public repository with build/run instructions”.) Unfortunately, there are all kinds of reasons why scripts/builds could fail just a few months or years later so they would have to be checked too. I think it would be cool if document-generation caught on in the publication of papers, i.e. the paper itself is generated by running actual experiment scripts and producing charts, etc. from plain text source.
- coldacid 9y ago
  So, Jupyter?
ajarmst 9y ago
Judging by the sheer amount of XKCD content in this report, Randall Munroe should get a co-author credit.
Thrymr 9y ago
To be clear, the "NAS" (nas.org) that published this study is the National Association of Scholars [0], a political group, not the National Academy of Sciences (nasonline.org) [1], a nongovernmental organization that consists of scientists elected by their peers to provide independent scientific advice to the US government. There was in fact a study published recently in PNAS, the Proceedings of the National Academy of Sciences, on this topic [2]. [0] https://en.wikipedia.org/wiki/National_Association_of_Scholars https://en.wikipedia.org/wiki/National_Association_of_Schola... [1] https://en.wikipedia.org/wiki/National_Academy_of_Sciences https://en.wikipedia.org/wiki/National_Academy_of_Sciences [2] http://www.pnas.org/content/early/2018/03/08/1802324115 http://www.pnas.org/content/early/2018/03/08/1802324115
- jrochkind1 9y ago
  Good catch thanks! Wikipedia says "The National Association of Scholars (NAS) is an American non-profit politically conservative advocacy group, with a particular interest in education" Thanks for the link to the PNAS -- looks like a whole special issue, not just an article, in fact. Great. I'm gonna ignore the fake-NAS one, and read the real-PNAS one. I wonder what National Association of Scholars' motivation is here exactly.
  - DanielBMarkham 9y ago
    Isn't the entire purpose of reproducability is to eliminate such questions from discussions about science?
    - da_chicken 9y ago
      The purpose of reproducibility is to be able to properly analyze the quality and accuracy of a study. The purpose is to support critical thinking and objective facts. However, this paper isn't a study. It's just an essay. And while, yes, the paper in question does highlight numerous issues in reasonable and rational ways, it also uses this as the breakout quote on page 36: > Scientists readily accept results that confirm liberal political arguments, and frequently reject contrary results out of hand. Now, there's a couple problems with this statement. The most obvious is that it implies blame on liberal politics. On the same page they detail that the percentage of self-identified liberal psychologists at a conference (~800 of 1,000) wildly outnumbers the number of self-identified conservative psychologists at the same conference (3 of 1,000). Well, this conference is the Society for Personality and Social Psychology, which is based in southern California. Would that bias attendance? However, even if we assume that distribution is correct, the quoted statement doesn't say anything about the conservative scientists. Would it also be correct that conservative scientists readily accept results that confirm their political arguments? We don't know. Maybe it would be more correct to say this: > Scientists readily accept results that confirm pre-existing personal political arguments, and frequently reject contrary results out of hand. Instead, this paper chooses to imply that scientists who are more liberal are somehow inherently more biased. If we assert that good scientists are those who don't allow political arguments to influence their acceptance of their results, then the natural conclusion to draw from that is that being liberal is incompatible with being a good scientist. Don't get me wrong, this paper makes a lot of legitimately good points with good examples, but the whole section on groupthink comes across a little bit biased itself. However, it's existence means I can't shake the feeling that maybe I'm missing something. So having show that there's a reproducibility problem, the paper concludes. Then very end of the paper, the afterword lists these points introduced seemingly out of the blue: > Symptoms of Pathological Science: > 1. The maximum effect that is observed is produced by a causative agent of barely detectable intensity, and the magnitude of the effect is substantially independent of the intensity of the cause. > 2. The effect is of a magnitude that remains close to the limit of detectability; or, many measurements are necessary because of the very low statistical significance of the results. > 3. Claims of great accuracy. > 4. Fantastic theories contrary to experience. > 5. Criticisms are met by ad hoc excuses thought up on the spur of the moment. > 6. Ratio of supporters to critics rises up to somewhere near 50% and then falls gradually to oblivion. These are from Irving Langmuir (https://en.wikipedia.org/wiki/Irving_Langmuir https://en.wikipedia.org/wiki/Irving_Langmuir) and his criticisms on research conducted with unconscious biases or subjective elements. He meant it as a criticism of research like cold fusion. Now, there's not any real one-for-one discussion of these items or how they relate to what came before. They're just presented basically as-is as the final portion of the essay with few notes that they correlate to what came before. You'll notice, however, they they all call into question the credibility of the scientist as well as the credibility of the scientific community, and not the quality of their research or data. However, let's say you're politically conservative, and you're reading this essay. Maybe you don't believe in climate change. You get to the end of this paper, and it tells you that in spite of the claims of accuracy of the data, in spite of the consensus of the community, in spite of all the evidence, if it's contrary to [your] experience or if the defenses to criticism feel invented on the spot or if you keep hearing claims that support is waning, and apparently especially if it's liberal, then you've found a legitimate case of bad science. I don't think that's necessarily the point they're trying to make here, but I really don't like the way this afterword is presented. It's introducing new concepts and arguments and then handwaves away having to justify them or explain their relevance.
air7 9y ago
A good solution would be to simply reduce the p-value threshold. The current "standard" p-value in many fields, of 5%, is arguably too high. consider throwing a double six in backgammon has probability of 3%. That means that, on some level, throwing 6-6 would be a valid scientific "proof" of ESP. (p-value<0.05) This of course is even before p-hacking. A high p-value is basically externalizing part of the research cost onto other scientists and in the process creates a lot of false-positive "noise".
mlvljr 9y ago
Misread as "circus", nodded.

270 points by vixen99 9y ago

I think Planet Money did an episode on this a few years ago. Good stuff, and maybe easier than reading a paper. :) https://www.npr.org/sections/money/2018/03/07/591213302/episode-677-the-experiment-experiment https://www.npr.org/sections/money/2018/03/07/591213302/epis...

Afforess 9y ago
The report (not a paper) is actually written for a lay audience and very readable: https://www.nas.org/images/documents/irreproducibility_report/NAS_irreproducibilityReport.pdf https://www.nas.org/images/documents/irreproducibility_repor...
- HarryHirsch 9y ago
  It's a position paper from a conservative think tank and needs to be treated as such.
  - ItsMe000001 9y ago
    > and needs to be treated as such What does that even mean? You have one point "it's a conservative think tank" - which does not mean anything - coupled with another empty statement. Now I sure don't mind criticism of that entity, but I find it ironic that it is done with such an empty statement that's just a not very cleverly disguised type of ad hominem attack. May I suggest you take some time actually reading the paper and then coming back with criticism of their actual points?
    - samfriedman 9y ago
      It is absolutely relevant to consider the political motivations of publishers of reports like this, when the publisher is a politically active organization as opposed to an independent research group. While the report itself may not push an obvious partisan agenda, a critical reader should consider the conservative agenda of the NAS in relation to why they might publish such a report. In the conservative wing's long-standing opposition to science, it's not far-fetched to view this report as being published tactically -- even if all of its points are well-founded -- as something to point to when future efforts attempt to undermine science-based policy on issues like climate change, stem-cell research, abortion, etc. Especially when the President of the NAS is a public "climate skeptic". https://www.chronicle.com/blogs/innovations/bottling-up-global-warming-skepticism/29754 https://www.chronicle.com/blogs/innovations/bottling-up-glob...
    - HarryHirsch 9y ago
      A position paper aims to make a political position appealing, a study about the sociology of science on the other hand would seek explanations for a position. Pointing out the purpose of something isn't a character attack.
  - Afforess 9y ago
    Ad hominem has never been a valid reason to disbelieve something. I quite enjoyed reading the report - it includes XKCD references and everything.
    - jnordwick 9y ago
      That you're being voted down shows that HN has really changed. (I blame all the overtly political articles that I believe have changed the tenor of the site, and politics has infected every article now. Thanks @dang!)
      - sctb 9y ago
        If you won't stop posting this off-topic meta noise that we've asked you many times before to cut out, we'll ban the account. Post according to the guidelines or not at all. Fulminate elsewhere. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
  - kiziaru 9y ago
    What makes it a conservative think tank?
    - wRastel27 9y ago
      From the intro: The new, alternative view, was that college and universities should be places where fresh ideas untrammeled by hidden connections to the established structures of power in American society should have the chance to develop themselves. In practice this meant a hearty welcome to neo-Marxism, radical feminism, historicism, post-colonialism, deconstructionism, post-modernism, liberation theology, and a host of other ideologies. Also from the intro: The National Association of Scholars (NAS) has long been interested in the politicization of science.

Retric 9y ago

Being reproducible is only critically important if people treat individual studies as meaningful. That IMO is a far more dangerous stance. Any study can have hidden flaws, none should be trusted without some form of replication.

mylons 9y ago
not sure what you're getting at. are you saying meta analysis is more important than individual studies? but if a large % of your studies in the meta analysis can't be reproduced, what faith do you have in the meta analysis at all?
- aleksei 9y ago
  Take into account that meta analysis should not only consider the results of its participant studies, but also whether there is enough information to reproduce the study and whether it has accounted for possible confounding factors.
rplst8 9y ago
How often is a study even repeated in the course of normal scientific research? I think most studies are focused on expanding the research and therefore knowledge about the science being studied. Accepting previous studies as fact is dangerous and could easily lead to a house of cards scenario. I think this is especially true in the very specific, niche studies that are common in today's highly competitive graduate student and publish or die research landscape.
- lotsofpulp 9y ago
  That is why, as a (global) society, we should spend our resources in making sure the data we have is accurate by funding research and repeated studies, and that data should be open to all to verify and build upon.
  - jbob2000 9y ago
    I think we're passed the point where studies can be reproduced effectively. I actually think the scientific model is falling apart. We've discovered the easy things, and those studies were easy to reproduce. We built a model of science around the low hanging fruit. How are you going to reproduce a study about a cancer drug? Synthesizing the drug itself is a huge endeavor, never mind finding a group of participants with the exact cancer you need. How would you reproduce the Large Hadron Collider experiments? You'd need to rebuild one of the most technically challenging, most expensive scientific instruments ever made.

rplst8 9y ago

This is the exact reason I don't believe science should be used blindly to affect government policy. It flies in the face of democracy and is subject to falsehoods just like any other human endeavor.

jboggan 9y ago
Lysenkoism happens in the American and European scientific establishments as well. As an example I would suggest Ancel Keys and the cholesterol hypothesis of heart disease.
- marcosdumay 9y ago
  That's a bad example. It's a much more complex scenario, where the drugs that fight the expected cause of the disease does in fact help protecting against it. Confusing causality is very hard to solve.
  - mattnewport 9y ago
    If you're talking about statins, Ancel Keys was promoting shoddy science on high fat diets to influence government policy before statins were discovered.
tlb 9y ago
Who is the strongest proponent of using science blindly to affect government policy? What are their main arguments, and how do you disagree?
- AstralStorm 9y ago
  What do you mean by using science blindly? Three is NO way to do that. There are ways to misuse scientific results. Premature generalisation, assumptions of safety. Cherry picking to support preconceptions. Attributing strong support to tentative and/or unchallenged results. And more...
lmm 9y ago

terminado 9y ago

I mean, this is still a good thing, since it charts a map of decidedly grey areas, where information may always be ambiguous, and useful information needs to be sussed out carefully. It's better than presumptively assuming that "Science" is an infallible always black-and-white.

quantumwoke 9y ago

Recently I have been beginning to question university education. It seems that today's education system has mismatched priorities which may have an impact on the reproducibility crisis. You see, to come up with a fantastic new alloy, you don't have to be a genius. Or even very good. You have to be persistent, and clever enough to realize when you've found something interesting. You will need to understand your field very well, but there are many, many fields. And each one is quite narrow, so while it is hard to understand your field well, many, many people can do this. So my less dystopian future goes like this. Train lots and lots of physicists, chemists, engineers, programmers, materials scientists, biologists, mathematicians. And (shock horror) through higher taxes employ them. Employ them on promising ideas, and on impossible ideas. Employ them on the problems of the day, and on arcane research that will almost certainly never bear fruit. And sure, right at the top in the Apples and Googles of the world, you'll have the best and brightest pulling together all of the discoveries made without all these reproducibility issues. Or we could continue down the dystopian path. The low tax path where there really are only meaningful jobs for the best and the brightest. Where we rely on Apple and Google to decide what ideas are worth pursuing. Where the skills and abilities of a great many are not used. And most of us fight for the crumbs from the oligarch's tables. And the oligarchs? For every Elon Musk, there will be 10 who just want the biggest yacht and the prettiest mistress. Let me say that again. In a low tax environment, the skills and abilities of a great many will not be used. And we will all be poorer for that. Genuinely poorer, because great discoveries will not be made. And it is happening now. And I'll go one step further. We are also wasting the talents of our best writers and artists. The BBC and other national broadcasters are squeezed. Squeezed so that the idea of something as original and amazing as Monty Python appearing now is laughable. And again, we are all the poorer for that. Not just the lucky ones who get to make amazing shows or works of art, but all of us who never get to revel in them. The low tax, small government world is a step backwards. If you wish to sum it up succinctly, you may say it is a world where, in order to placate the very rich, we take away opportunities from many, and impoverish the world as a whole.

emodendroket 9y ago
> Recently I have been beginning to question university education. Why are we bothering to educate anyone except the very, very best students in physics, maths and engineering? In fact, why aren't we identifying these students and sending them straight to Caltech? It seems intuitive that this would solve one half of the reproducibility crisis. That doesn't seem intuitive at all.

mistrial9 9y ago

Some of the comments miss an important distinction.. Science in the popular press, often refers to science versus completely non-science ways of forming an opinion or deciding policy. Meanwhile, science within a technical community, is subject to human error and manipulation, and relies on a reproducible result, as well as peer-review, to find answers to conflicting claims. There are certainly non-Science ways of forming an opinion and deciding policy, in many cases totally legitimate. But once science is claimed, then of course it has to be subject to science rigors.

m-watson 9y ago
Yes, thank you for saying this. I feel like, often, people use science as some ethereal being. It is a process that has to be used. If that process isn't happening it isn't science. That doesn't mean everything has to be science, but things that are science should be held to a certain standard but should also be taken with the extra weight if that standard is met.
- Bograff 9y ago
  It's important to make the distinction between Pure Science and anything on the sliding scale from technical description to ELI5. There are levels to it, you and I know.
  - m-watson 9y ago
    Oh absolutely there is a sliding scale on both the actual practice of science as well as the reporting. From the practice of it, I know as simple anecdote I have participated in conducting research in hard science (physics), softer hard science (biology), social science (behavioral and economic), as well as straight policy analysis with a "scientific approach" through data science. The findings from those are reported differently and have stronger or weaker words and conclusions. I think that is where the practice sliding scale comes in, is how the researcher decided to write up and report their findings and appropriately identifying their methodology and short-comings. From the journalism side I think you end up with the problem the wide spread field of journalism has, you have to be read by your audience. That means something with a wider audience is probably going to be more on the ELI5 side which means both the reader and the journalist have to take any reported findings more lightly. The sliding scale, I believe, comes in with the rigor something is reported. That is both in the practice and the journalistic side. More details can give a more granular understanding showing limitations but some times broad strokes are necessary to get some information out to a wider audience or more quickly.

tonystubblebine 9y ago

We did an article trying to help people be better critical thinkers when they hear about psychology research. I thought the author did a good job getting into how these studies go wrong and also the types of magical results you should be suspicious of. https://medium.com/@jhreha/most-psychology-research-is-bs-73d4793b4dc6 https://medium.com/@jhreha/most-psychology-research-is-bs-73...

SubiculumCode 9y ago

We know that weak classifiers can be bagged to produce a strong classifier al la Adaboost. Each study is a weak classifier and would have a 'reproducibility crisis' if retested on new data. However after lots of studies of similar phenomena, a strong classifier emerges. In the field, we call this converging lines of evidence.

scottfr 9y ago
I'm not sure I really agree with this, because in science studies are biased towards aligning with the results of previous studies. This can be due to many factors: authors' expectations, journals like to publish positive over negative results, peer reviewers will look more critically (in an analytical sense) at work that disagrees with accepted research, etc.... It can be very hard for many reasons to stand up and say "this prior work is wrong".
- soVeryTired 9y ago
  I know someone who quit her PhD in part because she couldn't replicate someone else's experimental work on rat behaviour. If the replication crisis had been as well-known back then, she might have fared better.
- SubiculumCode 9y ago
  The argument is not that all is perfect in science. It is not. The argument is that strong replication for each study is not required for progress. Back to my analogy to the algorithms like Adaboost. Why ever aggregate over a a bunch of weak classifiers over a single strong one? Well, please correct me if I am wrong (machine learning is not my field), but the primary advantage is computational cost. Sometimes, for the same overall classification performance, training hundreds or thousands of weak classifiers requires fewer computations than training the strong classifier. To me it brings up an interesting point. If we view experiments as classifiers, then how would a machine learning expert set science policy and practice? Also viewed this way, it makes me wonder about p-hacking, which increases sensitivity while increasing false positives. Since negative results are not generally reported, I wonder whether p-hacking diminishes replicability at the study level for efficiency at the aggregate level. This is an empirical question, as ethically it is of course a dubious practice under current understanding.

timtadh 9y ago

In response to several threads here: it is important to distinguish when scientists are self critical vs. when non-scientists are critical of the scientific method. For instance, there is a long history of scientists criticizing how the scientific process is currently conducted for the purposes of improving the scientific endeavor. That work is sometimes used by non-scientists who question the overall scientific method. However, such use is invalid as the scientific self-criticism 1. assumes the validity of the scientific method 2. relies on the scientific method as its critical lens Whereas those who critique science as a whole: 1. assume that the scientific method does not work and does not arrive at "truth" 2. then use scientists being self critical to prove #1. Such a "proof" does not work as there is its uses the assumption "the scientific method arrives at truth" to derive the contradiction "the scientific method does not arrive at truth". See for instance comment: https://news.ycombinator.com/item?id=16859200 https://news.ycombinator.com/item?id=16859200 In reality, work on reproducibility is about improving the practice of science overall. It does not in itself show that science is inherently untrustworthy. What it does show is that scientific discovery is difficult and it takes a lot of effort and new findings should be treated critically. What does critically mean in this context? It means with in the boundaries of science analyzing the theoretical basis, hypothesis, method, and experimental results for potential flaws. It does not mean to be skeptical as a default because science "doesn't work."

323454 9y ago
I agree with your overall point, but technically speaking it is logically valid to prove a hypothesis false by first assuming it and then deriving a contradiction, even when the contradiction is the negation of the original hypothesis (as it is in your example). What you should have said is that some critics start with the premise "the scientific method does not arrive at truth", and then use other people's arguments that depend on the premise "the scientific method arrives at truth" to support their claim, which is indeed logically invalid.
buvanshak 9y ago
>scientists criticizing how the scientific process is currently conducted for the purposes of improving the scientific endeavor. I think what happening here is a bit more serious. They are showing a widespread crisis. It is not just some minor feedback to improve the process. >It does not in itself show that science is inherently untrustworthy. I think when statistics is involved, the results are inherently untrustworthy. This is not really surprising because there is a whole bunch of ways these studies that involve statistics could go wrong. And we are still finding new ways on how this could go wrong. Then there are things like publication bias, that takes this to a whole new level. Things like that means that a biased body of journals can project any consensus that it favors just by selecting studies that fit its narrative. The inherent issues with statistics means that you can find studies that shows any possible outcome.

lisper 9y ago

The site is down so I can't read the original report, but I've read reports on this topic in the past so I'm going to chime in with some "usual suspects" caveats: 1. No result is 100% reproducible because you can never completely reproduce the conditions of any experiment. The best you can hope to do is to reproduce the conditions that matter, but enumerating those has to be part of the theory you are testing, and so you can never be 100% sure that you have a complete list. 2. Even a completely non-reproducible result can be scientifically significant. For example, celestial events are almost never reproducible. Our understanding of celestial mechanics nonetheless rests on solid science. 3. The end-product of science is not truth, it is explanations of observations. Those observations can (indeed must) include non-reproducible ones. Sometimes the explanation of non-reproducible results is "experimental error" or "delusion" or "we just don't know." But non-reproducible events are nonetheless within the purview of science. On the other hand... 4. The statistical tests currently in widespread use as a criterion for publication in peer-reviewed journals guarantee that at least one result in 20 will be due to chance and not because the hypothesis being tested is actually true. That, combined with the suppression of negative results, guarantees that the results published in scientific journals that adhere to those standards will be unreliable. But that doesn't really have anything to do with reproducibility per se, it has to do with the fact that the journals use a weak criterion for defining positive results. That, combined with our human predilection to value positive results over negative ones and the understandable desire of scientists to advance their careers, all but guarantees that journals will contain many defensible but false results. But this is not because of a lack of reproducibility per se, it's because of poor policy choices.

Erlich_Bachman 9y ago
4.) You mean that papers are published with P<0.05? When did this become a "standard"? Most papers I have seen in respectable journals and other sources would not strive for such a high Pvalue, it would be much lower like 0.001.
- lisper 9y ago
  > You mean that papers are published with P<0.05? I'm guessing you meant to say that no papers are published with P<0.05. And of course they are, but that's irrelevant. What matters it that many journals, especially in "softer" sciences (i.e. not physics) use p<0.05 as their threshold, which guarantees that at least one paper in 20 will be "wrong". Then add self-suppression of negative results and the actual percentage of "wrong" papers becomes higher, potentially much higher if real positive results are rare. > When did this become a "standard"? I have no idea. You'll have a ask a historian.

gringoDan 9y ago

One of Slate Star Codex's top all-time articles discusses this very issue. Highly recommend: http://slatestarcodex.com/2014/04/28/the-control-group-is-out-of-control/ http://slatestarcodex.com/2014/04/28/the-control-group-is-ou...

jnordwick 9y ago
> On the meta-level, you’re studying some phenomenon and you get some positive findings. That doesn’t tell you much until you take some other researchers who are studying a phenomenon you know doesn’t exist – but which they themselves believe in – and see how many of them get positive findings. That number tells you how many studies will discover positive results whether the phenomenon is real or not. Unless studies of the real phenomenon do significantly better than studies of the placebo phenomenon, you haven’t found anything. This is such an astute observation that reproduction studies can be used to find a placebo control group for bad scuence. Parapsychology was one such group, but now we can find many others. Brilliant.

arafa 9y ago

What I'd like to know is if the "irreproducibility crisis" is really some combination of "sample size was too small" and "effect size was too small". When I went through a lot of these studies myself and saw the ones that don't reproduce, I saw this theme over and over. "P-hacking" is less of a concern to me when the effect is real and widespread. It's so bad now that for any article/study, I look at the sample size first. If it's too small (especially < 100) or they don't say I just ignore it. And if you don't publish or give some estimate of your effect size I just think about it directionally but don't give it much weight mentally.

hnuser1234 9y ago
I'm not sure this captures the whole problem. Sure, huge sample sizes are great, but look at the current top comment by lisper - we have gained a boatload (that's a scientific term) of insight from single events. LIGO is currently rocking a sample size of 7 but I doubt you ignored it.
- arafa 9y ago
  Of course there are exceptions and what I said doesn't capture all cases (like with massive fixed costs or unique kinds of studies), you're right. I find it's a pretty good rule of thumb, though.
amatic 9y ago
Huge sample studies with high stat significance can have very low effect sizes, though. If an effect is shown on a small sample, it means it is a really big effect, and will likely be reproduced in a bigger study.
- arafa 9y ago
  It's a combination of sample size and effect size. The larger one is, the more I'm willing to give on the other.

thedonkeycometh 9y ago

This is part and parcel of Scientific method. I'd be suspicious if all experiments were successful all the time. eyes pharma

danharaj 9y ago

Lets take a page from Marx. Science is many things, in particular a relationship between capital and labor. The scientific method is a wonderful idea, but it is subordinate to the economic forces that underlie scientific activity. Look at the conflicts and contradictions between those doing science (labor) and those deciding the science to be done (capital), and that is the ultimate source of these crises. The executive summary lists 40 points on how to improve the reproducibility of science. A bit over half of them are addressed to the sources of capital such as private organizations, universities, and governments. I think many of those points are good. However, I don't think the other points, the ones that recommend doing science in different ways, have a good punch. Even if you fix the problems there are today, so long as science is a rat race of trying to get grant money to stay afloat while burning out grad student after grad student, I think other pathological practices will creep in as a completely rational response on the parts of scientists to a hostile ecosystem. There's just a very big gap between how science should be done and what capital owners want from science.

deleted 9y ago
[deleted]
TangoTrotFox 9y ago
I think this is pretty self evident, but the issue is that let's say you have some system where economic forces are removed. Essentially a researcher basic income in one scenario. This would suddenly massively incentivize people towards this direction since it's basically a career path that guarantees a stable livelihood, which is something that's extremely rare today. Well you need to ensure there's nobody just completely gaming the system and so inextricably you'll end up with some sort of qualifier for results. And now suddenly you've done nothing but kick the can since this new qualifier is what's going to be gamed. Maybe the biggest problem is what you mention, but in another direction. For whatever reason there seems to be extremely little interest in the private direct funding of science. In times past the aristocracy would often fund scientific research on all sorts of topics. Today the practice seems to have all but disappeared, certainly if we measure relative ratios of the practice.
- 9y ago

btilly 9y ago

The article may be good, but the source has a definite bias. See https://www.sourcewatch.org/index.php/National_Association_of_Scholars https://www.sourcewatch.org/index.php/National_Association_o... to see what that bias is. That said, they should be well-informed on the subject. So it probably does contain a lot of good information.

flamedoge 9y ago

There is the problem of conflating science with mathematics.

blueprint 9y ago

"In order to be a true scientist, you should be familiar with philosophy, first"

mathgenius 9y ago

I have met several "refugees" from particle physics, that left in part because of flimsy statistical methods. So let me ask: who is going to reproduce the experiment that found the Higgs boson?

rdlecler1 9y ago

Maybe papers need Yelp reviews on reproducibility. 1-Star couldn’t reproduce.

makecheck 9y ago

There has to be similar prestige/career-building/notoriety/funding for spending time on reproducing the experiments of others. Without that shift there will clearly be a greater tendency to just try something new. Also, when experiments depend on source code, etc. we need real engineering tools/principles applied. (Something like: “you can’t publish paper X if you aren’t including a public repository with build/run instructions”.) Unfortunately, there are all kinds of reasons why scripts/builds could fail just a few months or years later so they would have to be checked too. I think it would be cool if document-generation caught on in the publication of papers, i.e. the paper itself is generated by running actual experiment scripts and producing charts, etc. from plain text source.

coldacid 9y ago
So, Jupyter?

ajarmst 9y ago

Judging by the sheer amount of XKCD content in this report, Randall Munroe should get a co-author credit.

Thrymr 9y ago

To be clear, the "NAS" (nas.org) that published this study is the National Association of Scholars [0], a political group, not the National Academy of Sciences (nasonline.org) [1], a nongovernmental organization that consists of scientists elected by their peers to provide independent scientific advice to the US government. There was in fact a study published recently in PNAS, the Proceedings of the National Academy of Sciences, on this topic [2]. [0] https://en.wikipedia.org/wiki/National_Association_of_Scholars https://en.wikipedia.org/wiki/National_Association_of_Schola... [1] https://en.wikipedia.org/wiki/National_Academy_of_Sciences https://en.wikipedia.org/wiki/National_Academy_of_Sciences [2] http://www.pnas.org/content/early/2018/03/08/1802324115 http://www.pnas.org/content/early/2018/03/08/1802324115

jrochkind1 9y ago
Good catch thanks! Wikipedia says "The National Association of Scholars (NAS) is an American non-profit politically conservative advocacy group, with a particular interest in education" Thanks for the link to the PNAS -- looks like a whole special issue, not just an article, in fact. Great. I'm gonna ignore the fake-NAS one, and read the real-PNAS one. I wonder what National Association of Scholars' motivation is here exactly.
- DanielBMarkham 9y ago
  Isn't the entire purpose of reproducability is to eliminate such questions from discussions about science?
  - da_chicken 9y ago
    The purpose of reproducibility is to be able to properly analyze the quality and accuracy of a study. The purpose is to support critical thinking and objective facts. However, this paper isn't a study. It's just an essay. And while, yes, the paper in question does highlight numerous issues in reasonable and rational ways, it also uses this as the breakout quote on page 36: > Scientists readily accept results that confirm liberal political arguments, and frequently reject contrary results out of hand. Now, there's a couple problems with this statement. The most obvious is that it implies blame on liberal politics. On the same page they detail that the percentage of self-identified liberal psychologists at a conference (~800 of 1,000) wildly outnumbers the number of self-identified conservative psychologists at the same conference (3 of 1,000). Well, this conference is the Society for Personality and Social Psychology, which is based in southern California. Would that bias attendance? However, even if we assume that distribution is correct, the quoted statement doesn't say anything about the conservative scientists. Would it also be correct that conservative scientists readily accept results that confirm their political arguments? We don't know. Maybe it would be more correct to say this: > Scientists readily accept results that confirm pre-existing personal political arguments, and frequently reject contrary results out of hand. Instead, this paper chooses to imply that scientists who are more liberal are somehow inherently more biased. If we assert that good scientists are those who don't allow political arguments to influence their acceptance of their results, then the natural conclusion to draw from that is that being liberal is incompatible with being a good scientist. Don't get me wrong, this paper makes a lot of legitimately good points with good examples, but the whole section on groupthink comes across a little bit biased itself. However, it's existence means I can't shake the feeling that maybe I'm missing something. So having show that there's a reproducibility problem, the paper concludes. Then very end of the paper, the afterword lists these points introduced seemingly out of the blue: > Symptoms of Pathological Science: > 1. The maximum effect that is observed is produced by a causative agent of barely detectable intensity, and the magnitude of the effect is substantially independent of the intensity of the cause. > 2. The effect is of a magnitude that remains close to the limit of detectability; or, many measurements are necessary because of the very low statistical significance of the results. > 3. Claims of great accuracy. > 4. Fantastic theories contrary to experience. > 5. Criticisms are met by ad hoc excuses thought up on the spur of the moment. > 6. Ratio of supporters to critics rises up to somewhere near 50% and then falls gradually to oblivion. These are from Irving Langmuir (https://en.wikipedia.org/wiki/Irving_Langmuir https://en.wikipedia.org/wiki/Irving_Langmuir) and his criticisms on research conducted with unconscious biases or subjective elements. He meant it as a criticism of research like cold fusion. Now, there's not any real one-for-one discussion of these items or how they relate to what came before. They're just presented basically as-is as the final portion of the essay with few notes that they correlate to what came before. You'll notice, however, they they all call into question the credibility of the scientist as well as the credibility of the scientific community, and not the quality of their research or data. However, let's say you're politically conservative, and you're reading this essay. Maybe you don't believe in climate change. You get to the end of this paper, and it tells you that in spite of the claims of accuracy of the data, in spite of the consensus of the community, in spite of all the evidence, if it's contrary to [your] experience or if the defenses to criticism feel invented on the spot or if you keep hearing claims that support is waning, and apparently especially if it's liberal, then you've found a legitimate case of bad science. I don't think that's necessarily the point they're trying to make here, but I really don't like the way this afterword is presented. It's introducing new concepts and arguments and then handwaves away having to justify them or explain their relevance.

air7 9y ago

A good solution would be to simply reduce the p-value threshold. The current "standard" p-value in many fields, of 5%, is arguably too high. consider throwing a double six in backgammon has probability of 3%. That means that, on some level, throwing 6-6 would be a valid scientific "proof" of ESP. (p-value<0.05) This of course is even before p-hacking. A high p-value is basically externalizing part of the research cost onto other scientists and in the process creates a lot of false-positive "noise".

mlvljr 9y ago

Misread as "circus", nodded.