5 ms·

'Why did peer review not catch this? “Because peer review doesn’t do this,” Heathers told Ars. The point of peer review has always been for fellow scientists to

'Why did peer review not catch this? “Because peer review doesn’t do this,” Heathers told Ars. The point of peer review has always been for fellow scientists to judge whether a paper is of reasonable quality; reviewers aren't expected to perform an independent analysis of the data. ...In fact, without open data—something that’s historically been hit-or-miss—it would be impossible for peer reviewers to validate any numbers.' It doesn't seem unreasonable to think that the peer review process should include a statistician or someone who can review the statistics.

jobu 9y ago
This was my thought exactly. "Wansink added another note to his blog post, acknowledging the problems and announcing that a statistician would be redoing the analyses." With all of the issues lately around replication of studies it seems like having a statistician review the math should be step one of any peer-review process.
closed 9y ago
Interestingly, a few years ago Nature planned to include statisticians... > as consultants on the rare papers in which there are serious issues about the statistical techniques used, at the editor's discretion and based on the referees' suggestions. I think the crux is that editors already have enough trouble finding reviewers, and see getting a statistician as often unreasonable. It's too bad, because the nature of statistical issues is that people tend to be unaware of them until a statistician (used broadly) points them out.
- smnrchrds 9y ago
  I'm sure they won't have any touble hiring statisticians. The only unreasonable thing is that they expect people to work for free.
  - panglott 9y ago
    Yes. Perhaps it would be better to say "the review process" rather than "the peer review process".
andrewla 9y ago
> In fact, without open data ... it would be impossible for peer reviewers to validate any numbers Even this is not sufficient -- the fact is that this is a methodological statistical error, not a mathematical statistical error. The mistake is thinking that a dataset alone can yield data supporting a hypothesis derived from that dataset, rather than deriving a testable hypothesis from an existing dataset, and then gathering new data to validate. I would say the only way that you could publish results derived from an existing data source is if you also published all the null results you got along the way when examining the data set; but this is not really a feasible thing to print in a publication; either the list would be very long and thus it would be clear that some sort of p-hacking was involved, or the list would be short and accidental or intentional omission would be suspected. This dramatically reduces the amount of useful results that can be squeezed from a dataset, which is unfortunate, as many of them are hard to gather in the first place. It might be necessary to protect these datasets better -- to restrict access conditional on specifying the hypothesis being tested with the requirement that all results, including null results, be recorded, even if only in summary form if the results are not interesting enough to be accepted for publication or to be worth the effort of composing into a quality research paper.
- theossuary 9y ago
  So what if you split the dataset to say 70-30, tried to find hypotheses in the the 70% dataset, then validate them with the 30% dataset. This is the process I use often with machine learning, would it also apply here?
  - andrewla 9y ago
    Cross-validation (and the bootstrap, a related tool) is a useful technique. It's most useful when deriving the parameters for the model, though, rather than for confirming the validity of the model. Basically, if you have a strong prior reason to believe that the model is basically valid, but you need to know the parameters that fit the data. The more hypotheses that are tested using the 30% "test" dataset, the more that dataset inadvertently becomes a "training" dataset, by progressively invalidating models. For applications with real-world significance, you usually have to accept that the "test" dataset will expire after some number of experiments into the "training" dataset, and gather a new test dataset to prevent overfitting. Hastie, Tibshirani, and Friedman[1], which I heartily recommend, cover this extensively in chapter 7. [1] https://statweb.stanford.edu/~tibs/ElemStatLearn/printings/ESLII_print10.pdf https://statweb.stanford.edu/~tibs/ElemStatLearn/printings/E...

jobu 9y ago

This was my thought exactly. "Wansink added another note to his blog post, acknowledging the problems and announcing that a statistician would be redoing the analyses." With all of the issues lately around replication of studies it seems like having a statistician review the math should be step one of any peer-review process.

closed 9y ago

Interestingly, a few years ago Nature planned to include statisticians... > as consultants on the rare papers in which there are serious issues about the statistical techniques used, at the editor's discretion and based on the referees' suggestions. I think the crux is that editors already have enough trouble finding reviewers, and see getting a statistician as often unreasonable. It's too bad, because the nature of statistical issues is that people tend to be unaware of them until a statistician (used broadly) points them out.

smnrchrds 9y ago
I'm sure they won't have any touble hiring statisticians. The only unreasonable thing is that they expect people to work for free.
- panglott 9y ago
  Yes. Perhaps it would be better to say "the review process" rather than "the peer review process".

andrewla 9y ago

> In fact, without open data ... it would be impossible for peer reviewers to validate any numbers Even this is not sufficient -- the fact is that this is a methodological statistical error, not a mathematical statistical error. The mistake is thinking that a dataset alone can yield data supporting a hypothesis derived from that dataset, rather than deriving a testable hypothesis from an existing dataset, and then gathering new data to validate. I would say the only way that you could publish results derived from an existing data source is if you also published all the null results you got along the way when examining the data set; but this is not really a feasible thing to print in a publication; either the list would be very long and thus it would be clear that some sort of p-hacking was involved, or the list would be short and accidental or intentional omission would be suspected. This dramatically reduces the amount of useful results that can be squeezed from a dataset, which is unfortunate, as many of them are hard to gather in the first place. It might be necessary to protect these datasets better -- to restrict access conditional on specifying the hypothesis being tested with the requirement that all results, including null results, be recorded, even if only in summary form if the results are not interesting enough to be accepted for publication or to be worth the effort of composing into a quality research paper.

theossuary 9y ago
So what if you split the dataset to say 70-30, tried to find hypotheses in the the 70% dataset, then validate them with the 30% dataset. This is the process I use often with machine learning, would it also apply here?
- andrewla 9y ago
  Cross-validation (and the bootstrap, a related tool) is a useful technique. It's most useful when deriving the parameters for the model, though, rather than for confirming the validity of the model. Basically, if you have a strong prior reason to believe that the model is basically valid, but you need to know the parameters that fit the data. The more hypotheses that are tested using the 30% "test" dataset, the more that dataset inadvertently becomes a "training" dataset, by progressively invalidating models. For applications with real-world significance, you usually have to accept that the "test" dataset will expire after some number of experiments into the "training" dataset, and gather a new test dataset to prevent overfitting. Hastie, Tibshirani, and Friedman[1], which I heartily recommend, cover this extensively in chapter 7. [1] https://statweb.stanford.edu/~tibs/ElemStatLearn/printings/ESLII_print10.pdf https://statweb.stanford.edu/~tibs/ElemStatLearn/printings/E...