3 ms·
> In fact, without open data ... it would be impossible for peer reviewers to validate any numbers Even this is not sufficient -- the fact is that this is a me
by andrewla 10y ago
> In fact, without open data ... it would be impossible for peer reviewers to validate any numbers
Even this is not sufficient -- the fact is that this is a methodological statistical error, not a mathematical statistical error.
The mistake is thinking that a dataset alone can yield data supporting a hypothesis derived from that dataset, rather than deriving a testable hypothesis from an existing dataset, and then gathering new data to validate.
I would say the only way that you could publish results derived from an existing data source is if you also published all the null results you got along the way when examining the data set; but this is not really a feasible thing to print in a publication; either the list would be very long and thus it would be clear that some sort of p-hacking was involved, or the list would be short and accidental or intentional omission would be suspected.
This dramatically reduces the amount of useful results that can be squeezed from a dataset, which is unfortunate, as many of them are hard to gather in the first place. It might be necessary to protect these datasets better -- to restrict access conditional on specifying the hypothesis being tested with the requirement that all results, including null results, be recorded, even if only in summary form if the results are not interesting enough to be accepted for publication or to be worth the effort of composing into a quality research paper.
- theossuary 10y agoSo what if you split the dataset to say 70-30, tried to find hypotheses in the the 70% dataset, then validate them with the 30% dataset. This is the process I use often with machine learning, would it also apply here?
- andrewla 10y agoCross-validation (and the bootstrap, a related tool) is a useful technique. It's most useful when deriving the parameters for the model, though, rather than for confirming the validity of the model. Basically, if you have a strong prior reason to believe that the model is basically valid, but you need to know the parameters that fit the data. The more hypotheses that are tested using the 30% "test" dataset, the more that dataset inadvertently becomes a "training" dataset, by progressively invalidating models. For applications with real-world significance, you usually have to accept that the "test" dataset will expire after some number of experiments into the "training" dataset, and gather a new test dataset to prevent overfitting. Hastie, Tibshirani, and Friedman[1], which I heartily recommend, cover this extensively in chapter 7. [1] https://statweb.stanford.edu/~tibs/ElemStatLearn/printings/ESLII_print10.pdf https://statweb.stanford.edu/~tibs/ElemStatLearn/printings/E...
- theossuary 10y agoThat's really good to know, I'm still just beginning to learn about data analytics, and the link you provided is already proving to be an interesting read, so thanks!
- danso 10y ago> Even this is not sufficient -- the fact is that this is a methodological statistical error, not a mathematical statistical error. That said, Wansink's papers had plenty of mathematical statistical errors that were evident without access to the original data, as documented in the mentioned "Statistical Heartburn" preprint: https://peerj.com/preprints/2748.pdf https://peerj.com/preprints/2748.pdf
- andrewla 10y agoVery much so -- I had only read the first part of the article when I wrote this reply, having read the whole thing now (and the linked pizza paper) I'm more than a little horrified at what can, in the most optimistic case, be described as extreme carelessness. I have a pre-existing bias against papers relating to nutrition science, and this looks worse even than I expect. I would love to see a treatment that looks at especially noteworthy research in the area that passes a rigorous methodological review, much less has been replicated with any reliability.