8 ms·
The site is down so I can't read the original report, but I've read reports on this topic in the past so I'm going to chime in with some "usual suspects" caveat
by lisper 9y ago
The site is down so I can't read the original report, but I've read reports on this topic in the past so I'm going to chime in with some "usual suspects" caveats:
1. No result is 100% reproducible because you can never completely reproduce the conditions of any experiment. The best you can hope to do is to reproduce the conditions that matter, but enumerating those has to be part of the theory you are testing, and so you can never be 100% sure that you have a complete list.
2. Even a completely non-reproducible result can be scientifically significant. For example, celestial events are almost never reproducible. Our understanding of celestial mechanics nonetheless rests on solid science.
3. The end-product of science is not truth, it is explanations of observations. Those observations can (indeed must) include non-reproducible ones. Sometimes the explanation of non-reproducible results is "experimental error" or "delusion" or "we just don't know." But non-reproducible events are nonetheless within the purview of science.
On the other hand...
4. The statistical tests currently in widespread use as a criterion for publication in peer-reviewed journals guarantee that at least one result in 20 will be due to chance and not because the hypothesis being tested is actually true. That, combined with the suppression of negative results, guarantees that the results published in scientific journals that adhere to those standards will be unreliable. But that doesn't really have anything to do with reproducibility per se, it has to do with the fact that the journals use a weak criterion for defining positive results. That, combined with our human predilection to value positive results over negative ones and the understandable desire of scientists to advance their careers, all but guarantees that journals will contain many defensible but false results. But this is not because of a lack of reproducibility per se, it's because of poor policy choices.
- Erlich_Bachman 9y ago4.) You mean that papers are published with P<0.05? When did this become a "standard"? Most papers I have seen in respectable journals and other sources would not strive for such a high Pvalue, it would be much lower like 0.001.
- lisper 9y ago> You mean that papers are published with P<0.05? I'm guessing you meant to say that no papers are published with P<0.05. And of course they are, but that's irrelevant. What matters it that many journals, especially in "softer" sciences (i.e. not physics) use p<0.05 as their threshold, which guarantees that at least one paper in 20 will be "wrong". Then add self-suppression of negative results and the actual percentage of "wrong" papers becomes higher, potentially much higher if real positive results are rare. > When did this become a "standard"? I have no idea. You'll have a ask a historian.
- hpcjoe 9y agoPart of the mistake here is in interpreting the p value as being a measure of anything really meaningful. P<0.05 doesn't mean 1 in 20 are wrong. It does mean that in the narrow confines of the definition of the statistical model used, it is possible that 1 in 20 would be explainable by the null hypothesis. It is not, in any way a measure of reality. It is a guess at how accurately you've modeled the system, how accurate your priors are, and so forth.
- lisper 9y ago> P<0.05 doesn't mean 1 in 20 are wrong > It does mean that ... 1 in 20 would be explainable by the null hypothesis. That seems like splitting a pretty fine hair to me, particularly since I haven't actually committed to any of the many possible interpretations of the word "wrong".
- haZard_OS 9y agop>0.05 has been a mainstay in entire fields of study for quite some time. Most notably, the psychological and biological science journals have unofficially endorsed 0.05 as a cutoff for "significant" results. While not everyone in a particular field has been a party to this myopic view of statistical significance, enough have been that the American Statistical Association released a statement on the topic not long ago. That may not seem like a big deal but, given the organization's historical reticence to officially weigh in on such issues, it really is. ASA statement on p-Values: https://www.amstat.org/asa/files/pdfs/P-ValueStatement.pdf https://www.amstat.org/asa/files/pdfs/P-ValueStatement.pdf
- Tenoke 9y ago>When did this become a "standard"? Most people would say in the 20s and point at Fischer's publications, though you can have arguments for later and earlier. Some of that is mentioned in the Wiki page, and you can see 0.05 all over the article [0], and the other related wiki pages[1]. You can find some key quotes there: "The significance level for a study is chosen before data collection, and typically set to 5% or much lower, depending on the field of study" "In 1925, Ronald Fisher advanced the idea of statistical hypothesis testing, which he called "tests of significance", in his publication Statistical Methods for Research Workers. Fisher suggested a probability of one in twenty (0.05) as a convenient cutoff level to reject the null hypothesis.In a 1933 paper, Jerzy Neyman and Egon Pearson called this cutoff the significance level, which they named α." If you mostly see lower p-values, that most likely means that you usually read non-social science papers. I guarantee that most social science (and related) fields use 0.05 regardless of the respectability of the journal. As an example of fields with lower p-values - 0.001 is common in medicine, and physics commonly has even lower values. 0. https://en.wikipedia.org/wiki/P-value#History https://en.wikipedia.org/wiki/P-value#History 1. https://en.wikipedia.org/wiki/Statistical_significance#History https://en.wikipedia.org/wiki/Statistical_significance#Histo...
- chrisfosterelli 9y ago> When did this become a "standard"? I think this depends on the field. In some areas of neuroscience for example, brain data can have extremely high variability and is very expensive to collect a lot of, so p<0.05 is not uncommon.
- toufka 9y agoI'd add a last point that a scientific paper (or a particular experimental result) is not sufficient for 'science'. Citing a paper as either a truth or discovery is a disingenuous. The endeavour of science gets at a kind of consensus by having lots of papers, lots of experiments, lots of additional layers of experiments build on each other. And yes it's annoying to realize a given paper is misleading, has errors, or is even wrong - but that annoyance and workarounds are part of the scientific process. It's actually fairly rare for an experiment or paper to be strong enough to stand completely on its own as a singular, thorough, and completed discovery. If you've been part of the scientific field you know that it yet takes years after even a good publication for the knowledge to accrete to the greater scientific body. Some of the best advice I got was to spend a lot more time than one might want to really come up with an experimental design that avoided a statistical outcome and instead probed at an either/or mechanism. It's not always possible, but when it is, it can be a powerful reply to fighting statistical battles (see the Michelson-Morely experiment for a classic example). Further careers, families, reputations, humans are also part of that scientific process, which together with the above require and admit some deviation from perfect efficiency. Which we as a society should demand be okay.
- chrisfosterelli 9y agoThank you for saying this! This is a common view I see outside academia, where people interpret a single paper to mean that something is definitively shown. I think news organizations make this worse. Whenever a new paper comes out they'll champion it as fact if doing so leads to clicks. Peer reviewers can only hope to validate that the science was done well, not that it is inarguably correct. It requires a body of work around a topic before it's fair to say at all that something is "for sure" proven.
- JackFr 9y ago> I think news organizations make this worse. University PR is far more guilty. Buzzfeed isn't scanning the current issue of "Physical Review Letters" for the latest scoop.
- wrp 9y ago
- Waterluvian 9y agoWe need a journal that exclusively publishes papers with results that were totally contrary to the hypothesis. So long as the researchers aren't outright inept or fraudulent, what's there to be ashamed about? Those would be some of the more interesting papers to read, in my opinion.
- lisper 9y ago> We need a journal that exclusively publishes papers with results that were totally contrary to the hypothesis. No, that won't help. You'd end up with the Journal of Creation Science and things like that. What we need is pre-registration: https://cos.io/prereg/ https://cos.io/prereg/ We also need to change the culture so that positive results are not disproportionately rewarded.
- 3JPLW 9y agoNegative results are really hard to do right, and even harder to do interestingly. There are so many possible confounders in experiment design and execution that they generally aren't worth evaluating. So we choose to spend our time reviewing and evaluating those experiments that do seem to have at least some chance of saying something interesting. Even then, though, depending on the discipline, if the researcher did everything perfectly, they're still saying that there's a 5% or some% chance that they just "got lucky" and there is no underlying effect at all.
- haZard_OS 9y agoThere are several. To offer but one: http://www.jnr-eeb.org/index.php/jnr/index http://www.jnr-eeb.org/index.php/jnr/index I would actually prefer that each and every journal devote at least half of their space to negative results. I have seen some sublime experimental setups that produced negative results. I can't begin to imagine how instructive it would be to see more such experiments.
- buvanshak 9y ago>For example, celestial events are almost never reproducible. Our understanding of celestial mechanics nonetheless rests on solid science. I am not sure this is correct. The rules that govern celestial bodies is same as the ones that govern objects on earth. So why are they not reproducible? It does not require to measure forces between celestial bodies to measure the value of G. Measuring the forces between two masses on earth is enough.
- lisper 9y agoYes, of course the rules are the same. Figuring this out was the event that launched the entire modern scientific endeavor. But nonetheless, the experimental data that went into this discovery was largely non-reproducible. The stars and planets go where they go and only very rarely does a given configuration repeat itself. In fact, no configuration ever really repeats itself in every detail. So, for example, we can predict with ridiculous accuracy where the next total solar eclipse is going to happen. But once it happens, we can't roll back the clock and make it happen again.
- buvanshak 9y ago> But nonetheless, the experimental data that went into this discovery was largely non-reproducible. I am finding a hard time to grasp what you are telling here. If you deduce a set of laws A from an event X, you don't need X to repeat to check the validity of A. You can check it against another event Y, where you predict its characteristics using A. If the laws are valid, the prediction ll match the actual observation..
- lisper 9y agoYes, all that is true. What does that have to do with reproducibility? You're talking about doing a lot of different experiments to test a single theory. Reproducibility is about doing the same experiment over again to see if you get the same results as the first time you did it, i.e. reproduce the results.
- 9y ago
- yellowstuff 9y ago> The statistical tests currently in widespread use as a criterion for publication in peer-reviewed journals guarantee that at least one result in 20 will be due to chance and not because the hypothesis being tested is actually true. I have a pedantic correction, but a relevant one. If a journal makes a rule that results must have p < .05, and then scientists go off and do a bunch of well-managed science they will get results with a range of p-values. .05 will be a ceiling, not a floor, on the possible p-values. More papers have a p-values right around .05 than they would if results were being produced fairly, so that's actually evidence of bad practices.
- sfink 9y agoIs it? You're capped at 0.05. You can't print "obvious" results. ("With a confidence of p=0.00002173, we find that people are more likely to be angry in the slapped-in-the-face-by-the-researcher group than the control group...") So there's a natural floor determined by what is worth doing research on. Now, it turns out that the floor is far lower than researchers believe and the field would be better off by slower incremental progress built atop "obvious" results, but still, I assert that publishability is an adequate explanation for high p values and obscures the evidence for bad practices. Not that I have any doubt that bad practices abound; I have an undergraduate degree in psychology and am well aware of how things are done. It's not even malicious; people don't understand statistics, don't understand significance, don't understand models, and are mostly just doing whatever they can to make themselves feel good about their own intuitions and biases.
- 3JPLW 9y ago> The statistical tests currently in widespread use as a criterion for publication in peer-reviewed journals guarantee that at least one result in 20 will be due to chance and not because the hypothesis being tested is actually true. That's wildly overly pessimistic. That would only be the case if scientists just went around, looking at the world, and came up with null-hypotheses willy nilly. That's generally not the case. There is often a _reason_ for conducting the test and a plausible cause of action. There are two possible reasons for a significant result: * The null hypothesis is correct but they got "unlucky" data (5% chance or p% chance) * There is a real effect (and the null hypothesis is actually wrong) This becomes more problematic in reproducibility tests, though, since that biases my prior towards a correct null hypothesis and now you must be very careful about pre-registration and the numbers of folks worldwide that are trying to reproduce a given experiment.
- PeterisP 9y ago"p-hacking" is also a thing, it's a reasonable possibility that they tried 20 false but plausible null hypothesis and published the one that got a significant result. If there's no pre-registration, my prior of a paper actually being true would be much lower than statistics would say simply because the incentives are aligned so that there's a motivation to exaggerate significance.
- lisper 9y ago> > at least one result in 20 will be due to chance > That's wildly overly pessimistic Why? You yourself said: > The null hypothesis is correct but they got "unlucky" data (5% chance or p% chance) How is that different from what I said? 5% is just another way of writing 1 in 20.
- 3JPLW 9y agoIt's different because you're ignoring my second bullet point. There's a very common misconception that p-value is the percent chance the data confirm that the effect is real. That's not at all how p-values work. They are showing the _likelihood_ of generating the data were there no "effect." You make a "null hypothesis" — that is, you assume there is no effect — and you construct a distribution of the results you might see in such a case. If your results are relatively unlikely, then you say the null hypothesis is rejected at a p-value level, leading support to the fact that there is such an effect. But there's another bullet in my comment. You could _also_ see a low p-value if the null hypothesis is wrong! That would _also_ lead to a low p-value, but for a completely different reasons — reasons that are typically elucidated in the entire rest of the paper. Thus, you should be most suspicious of papers that do not have a reasonable cause of action, as that should increase your prior that the null hypothesis is actually correct. If I were to edit that post, I'd add a third bullet — that they cheated or did bad science. My objection is to you stating that the statistical tests alone guarantee at least a 1 in 20 failure rate with a p-value of .05. There are indeed many other reasons to be suspect, but the statistical tests don't guarantee this.
- lordnacho 9y ago> 3. The end-product of science is not truth, it is explanations of observations. Those observations can (indeed must) include non-reproducible ones. Sometimes the explanation of non-reproducible results is "experimental error" or "delusion" or "we just don't know." But non-reproducible events are nonetheless within the purview of science. Yes and no. I think you have to have predictions in there somewhere. That's why your celestial mechanics is science: whatever the movement is, you were able to say what it was going to be beforehand, even if it only happened once. As for explanation, that is not necessarily so simple, either. Sure, a lot of things seem to make more sense with a theory, but not every theory makes sense. A lot of stuff in fundamental physics is predicted by the equations, but in what sense are the equations explanations? "Shut up and calculate" comes to mind.
- pas 9y agoShut up and calculate means that "sure, that's a nice toy model, but show that it reproduces the standard model", and also "sure, nobody really understands why the model is that way, but it works, so shut up and use it". And also, shut up and come up with something falsifiable (in our lifetime).
- eli_gottlieb 9y ago>The end-product of science is not truth, it is explanations of observations. That sounds a whole lot like truth to me, or at least, like any sane construal of truth.
- lisper 9y agoThat depends on what you mean by "truth". Newtonian mechanics says that gravity is pulling you down towards the surface of the earth. Is that "true"? It has tremendous explanatory power, but I would argue that it is not actually true. The truth is (as far as we can tell at the moment) that the surface of the earth is pushing you up and accelerating you through curved space-time.
- jcoffland 9y agoUltimately, it comes down to semantics. Does a vacuum suck or does external pressure push in on the vacuum? My highschool chem teacher said, "there is no suck," but, depending on your perspective, either explanation is valid. Reality is subjective and language is imperfect.
- eli_gottlieb 9y ago>The truth is (as far as we can tell at the moment) that the surface of the earth is pushing you up and accelerating you through curved space-time. And in fact, General Relativity is the best explanation science has given us at the moment, which seems to fit what I said.
- forapurpose 9y ago> The truth is (as far as we can tell at the moment) that the surface of the earth is pushing you up and accelerating you through curved space-time. Until this moment, I never suspected how insightful the following is: https://wiki.tfes.org/Universal_Acceleration https://wiki.tfes.org/Universal_Acceleration
- kerkeslager 9y ago> 1. No result is 100% reproducible because you can never completely reproduce the conditions of any experiment. It's worth noting that this is part of the goal of reproducing results. If two identical experiments produce the same result, you're only providing proof for the very narrow theory the experiment tests. Slightly different experiments allow you to demonstrate that the theory is applicable. Imagine if evolution only occurred in the Galapagos, or if gravity only worked in an apple orchard in Cambridge. The only reason we know that evolution and gravity aren't local oddities is that they have both been tested in a wide variety of locations, with a wide variety of experiments.
- lurquer 9y ago> 3. The end-product of science is not truth, it is explanations of observations. The term 'science' is used in this way. Perhaps it should not be. Perhaps semantics drive the issue. In the field in which I am familiar, we have 'expert witnesses' that will use reproducible scientific principles to analyze, for instance, blood scatter to determine the angle in which a shotgun was held when it discharged and blew off a particular person's head. The principles -- based on physics, chemistry, etc -- are sound and scientific. The problem is, we will invariably have TWO very scientific expert witnesses who BOTH use reproducible and accepted methods. Yet, using their scientific tools, they will come to two utterly different conclusions. It's left to a jury to decide which interpretation is correct. While the tools used by the experts are scientific, their opinions are not. While their scientific tools are reproducible (for instance, the speed at which coagulating blood and brain tissue will slide down a stucco wall given all the variables such as temperature, humidity, friction coefficients, etc.) their ultimate reconstruction of what happened is not reproducible, and it is left to lay people to draw their conclusions. There should be more of a distinction between Science and History. That is, science -- strictly defined -- is all about reproducibility. History -- by definition -- is not. Using scientific tools to reconstruct a past event does not result in a 'scientific' theory. Rather, it results in a historical theory arrived at with scientific tools. Whether it is blood scatter from a murder (or suicide, depending on whom you ask), the formation of the Grand Canyon, or the development of finch beaks on Galapagos, the ultimate theories are inherently non-scientific as they can not be tested. The methods and tools used to derive the theories can be... but not the conclusion itself. In short, perhaps the term 'science' is being used for things outside its domain. (Or, alternatively, if we wish to include such things inside the domain, we should broaden the strict definition of science. Like I said, there are some issues of semantics that may be driving some of the issues in the article... other issues, of course, pertain to sloppiness, errors, and the like.)
- khr 9y agoPower is also a concern that many researchers do not pay attention to (at least in psychology/neuroscience). If the original study was under-powered, the estimated effect size in that study will be inflated and any replication attempt that uses this inflated effect size estimate will be severely underpowered. Plus, two independently conducted studies that are both powered at 80% to detect a true effect will both be positive results only 64% of the time (assuming absolutely nothing fishy going on, e.g. p-hacking).
- godelski 9y ago> 1. No result is 100% reproducible because you can never completely reproduce the conditions of any experiment. This is why you include the error in your results. We don't care if your two experiments result in 100% the same results. We care about if your results line up within the error of your experiment. > 2. Even a completely non-reproducible result can be scientifically significant. For example, celestial events are almost never reproducible. Our understanding of celestial mechanics nonetheless rests on solid science. This is why we share data. That is in essence our observation. A celestial event might be caught by only one instrument, but several people can develop several different models. We wait and watch for similar events though, to check if models are consistent (within error). > 3. The end-product of science is not truth, it is explanations of observations. I just wanted to repeat this because it can never be stated enough. > 4. The statistical tests currently in widespread use as a criterion for publication in peer-reviewed journals guarantee that at least one result in 20 will be due to chance and not because the hypothesis being tested is actually true. While I don't like p values, because of hacking, (especially 0.05) that's not how stats work. Flipping 10 heads doesn't guarantee 5 tails, not even 1. I wouldn't use as strong as a word as guarantee. But this is also an argument FOR reproducing. If multiple experiments are consistent with one another (within error) than that strengthens the argument. TLDR: More brains that look at a problem helps solve the problem. I will add my own statements about the reproducability problem in science. One is because there is less funding for it. Reproducing an experiment isn't sexy. Another problem just stems from that data isn't always open. It is hard to review work if you don't know everything about the experiment. Data can even have simple mistakes that just weren't caught. But it is also embarrassing to share data.
- lisper 9y ago> Flipping 10 heads doesn't guarantee 5 tails I never said it did. But in the long run, flipping a fair coin will generate pretty close to 50% heads and 50% tails. That's what it means to be a fair coin. Likewise, in the long run, using a p threshold of 0.05 (which many journals do) will generate 5% false positive results (that's that the 0.05 means), i.e. on average 1 false positive result for every 20 experiments you do.
- 9y ago