15 ms·
What is repeatability? 10% of the time? 50% of the time? 99% of the time? 99.9999% of the time? Even medications made in high volume processes have rejected un
by karthikb 8y ago
What is repeatability? 10% of the time? 50% of the time? 99% of the time? 99.9999% of the time?
Even medications made in high volume processes have rejected units (and sometimes, batches). Process drifted, machine failed, raw ingredients had some issue.
A scientific paper at the very early stage is meant to show that something is possible if the conditions are just right. It may be actually outside the capability of the lab to document (or measure, or even know) every variable that makes up these conditions. One of the reasons there is such a chasm between lab and practice is that as people dive in for deeper review and productization, they find the process is too finicky, the applicability too narrow, etc. But the scientific paper is the first step in this process of discovery.
That said, it's critical that authors document whatever they do know or have control over. Electronic publishing, tools like Github, etc, all make the barrier for disclosure much lower.
- drb91 8y agoPresumably it would be context sensitive like most variables. Just because something is hard does not mean it is not worth doing. Confirmation is inherently impossible, but we should be able to weed out bad papers with higher accuracy.
- karthikb 8y agoAbsolutely agreed
- stonemetal 8y agoGiven that P <= 0.05 is the usual standard, I would hazard that if it can't be reproduced at least 95% of the time then you miscalculated P.
- kgwgk 8y agoThis is no what a p-value means.
- stonemetal 8y agoIt isn't but it kind of is. It is a 95% chance that you should reject the null hypothesis. For example if we flip a coin a 5 times and get heads every time that gives you a p value .03 for rejecting the null hypothesis that the coin is fair. If I run the five coin flip experiment a 100 times and never once reproduce a significant result is it because gee science is hard, or is it because you hit one of those 3% of cases. The same goes for real science if you can't reproduce it more often than not then you might be close but you haven't figured it out yet.
- kgwgk 8y agoOk, so we can agree that if you got positive a result with p<0.05 and you repeat the experiment 100 times you expect to reproduce a positive result (i.e. getting a new p-value below 0.05) a) if the null hypothesis is true, 5 times b) if the null hypothesis is not true, anywhere between 0 and 100 times (depending on what is the true alternative) This is quite different from your previous assertion that the expected number of successful replications would be 95 out of 100.
- southern_cross 8y agoI once read about a scientist who, when others were having trouble replicating his results, with a straight face then claimed that his work was so bleeding-edge that it could only be reproduced by his own hands, in his own lab, using his own materials and equipment. Now, while there might have been an element of truth to this it sure sounded like total B.S. to me! I'm not sure than anyone had really challenged him on it, either.
- TeMPOraL 8y agoI think the key here is not if the paper was reproduced, but whether it includes enough information to make it possible. > Even medications made in high volume processes have rejected units (and sometimes, batches). Process drifted, machine failed, raw ingredients had some issue. Sure, but here the process is known in details, so the manufacturer can just rerun it. A scientific paper (with supplemental material) should be exactly such a detailed process description, so that researchers could run it if they want. Now, if the paper does not contain enough information to enable its reproduction, then there's really no way to tell whether it described real results, tampered results, or whether it was just made from whole cloth. There's no shame in being wrong. But there is shame in making sure no one can check if you're wrong.