4 ms·
> 1. No result is 100% reproducible because you can never completely reproduce the conditions of any experiment. This is why you include the error in your resu
by godelski 9y ago
> 1. No result is 100% reproducible because you can never completely reproduce the conditions of any experiment.
This is why you include the error in your results. We don't care if your two experiments result in 100% the same results. We care about if your results line up within the error of your experiment.
> 2. Even a completely non-reproducible result can be scientifically significant. For example, celestial events are almost never reproducible. Our understanding of celestial mechanics nonetheless rests on solid science.
This is why we share data. That is in essence our observation. A celestial event might be caught by only one instrument, but several people can develop several different models. We wait and watch for similar events though, to check if models are consistent (within error).
> 3. The end-product of science is not truth, it is explanations of observations.
I just wanted to repeat this because it can never be stated enough.
> 4. The statistical tests currently in widespread use as a criterion for publication in peer-reviewed journals guarantee that at least one result in 20 will be due to chance and not because the hypothesis being tested is actually true.
While I don't like p values, because of hacking, (especially 0.05) that's not how stats work. Flipping 10 heads doesn't guarantee 5 tails, not even 1. I wouldn't use as strong as a word as guarantee.
But this is also an argument FOR reproducing. If multiple experiments are consistent with one another (within error) than that strengthens the argument.
TLDR: More brains that look at a problem helps solve the problem.
I will add my own statements about the reproducability problem in science. One is because there is less funding for it. Reproducing an experiment isn't sexy. Another problem just stems from that data isn't always open. It is hard to review work if you don't know everything about the experiment. Data can even have simple mistakes that just weren't caught. But it is also embarrassing to share data.
- lisper 9y ago> Flipping 10 heads doesn't guarantee 5 tails I never said it did. But in the long run, flipping a fair coin will generate pretty close to 50% heads and 50% tails. That's what it means to be a fair coin. Likewise, in the long run, using a p threshold of 0.05 (which many journals do) will generate 5% false positive results (that's that the 0.05 means), i.e. on average 1 false positive result for every 20 experiments you do.
- logfromblammo 9y agoYour experiment shouldn't need to presume the coin is fair. Flipping an unfair coin twice in a row and discarding the HH and TT doubles will generate close to 50% HT pairs and 50% TH pairs. This only presumes that subsequent flips are independent events. It's still not going to be exactly 50%, because random events are random. You can get as small an error margin as you need by increasing the number of trials. And in some cases, this property of statistical analysis makes it easier to increase N than reduce the randomness and uncertainty in the experiment.
- Paradigma11 9y agoHow can u get a wrong positive result if you only test correct H1s?
- lisper 9y agoBecause there is a very long causal chain between raw physical phenomena and your perceptions. Your equipment could be faulty. You could be suffering from hallucinations. You could be a brain in a vat. Example: I want to test if evolution is true. I hypothesize that if evolution is not true, then God will give me some sort of sign, e.g. I pray to God to make a coin that I flip come up heads if evolution is false. I flip the coin and it comes up tails. I conclude that evolution is true. That conclusion would be wrong even if evolution is true.
- Paradigma11 9y agoYou are moving into metaphysics and including stuff that is not covered by Null Hypothesis Significance Testing. But on the original point "Likewise, in the long run, using a p threshold of 0.05 (which many journals do) will generate 5% false positive results" is just plain wrong. P(A|B) != P(A)
- lisper 9y ago> P(A|B) != P(A) It is if P(A) is 1, which is what you stipulated when you asked "How can u get a wrong positive result if you only test correct H1s?" > "... using a p threshold of 0.05 ... will generate 5% false positive results" is just plain wrong No, it's not wrong, it just makes a tacit assumption that most hypotheses that get experimentally tested are incorrect. Which is a reasonable assumption. If it were not true, making scientific progress would be a lot easier.