4 ms·

Abusing statistics: “Mr. Wansink’s fall from grace began with a 2016 blog post in which he blithely confessed to using improper research techniques known as p-

Abusing statistics: “Mr. Wansink’s fall from grace began with a 2016 blog post in which he blithely confessed to using improper research techniques known as p-hacking and HARKing. P-hacking involves running statistical analyses until they produce a statistically significant result; HARKing stands for “hypothesizing after the results are known.” The post prompted a small group of skeptics to take a hard look at Mr. Wansink’s past scholarship. Their analysis, published in January 2017, turned up an astonishing variety and quantity of errors in his statistical procedures and data.”

rossdavidh 8y ago
Oh, if only we had a system to pass articles that are submitted to scientific journals to some sort of expert committee, that could look at it and see if it was likely to be valid. We could call it, I don't know, "peer checkup", or "expert review", or...oh never mind.
- ble 8y ago
  Peer review can't fix broken incentives.
- sn41 8y ago
  The issue is that peer review never pays off directly. The time spent in reviews mostly goes down a black hole. Editorships are awarded based on publication record, and not quality of reviews. As a result, more and more, reviews will tend to be shallow and uncritical.
  - rossdavidh 8y ago
    Truth.
soperj 8y ago
>The post prompted a small group of skeptics to take a hard look at Mr. Wansink’s past scholarship It's funny that this is basically the peer-review version of harking.
- theshadowknows 8y ago
  You just wrecked my mind
- pcrh 8y ago
  It's also pretty close to the Baysian approach. The difference being that in the Baysian approach one is "permitted" to revise the hypothesis as new data is acquired.
  - antidesitter 8y ago
    Correction: Bayesian, not Baysian.
kradroy 8y ago
I've never heard the term "HARKing" before. This is exactly what my advisor had me do on our experiments that couldn't confirm our initial hypotheses. I knew it was wrong because we set up the experiments in a way to eliminate confounding variables. So any conclusions drawn on data for hypotheses outside of those would be suspect. The experiments we ran were costly in time to do because we were using feedback from students using our learning system over a semester. I quit working with her because of that, and also because she was a total asshole. I refuse to donate to my university until she's no longer on the faculty. I don't want her benefiting from my hard-earned money.
- MrEfficiency 8y ago
  This is why I believe in Engineering and not the modern "Academia Science". Totally corrupt, Ive seen it when I talked to professors about getting my PhD. You prove what they want proven. One is going to have millions of customers using it, the other is going to have a few biased professors review the data.
  - vkou 8y ago
    If you think this is any different from how industry works, you have not spent enough time in industry. After-the-fact-hypothesis is constantly used to retroactively justify business decisions.
    - MrEfficiency 8y ago
      I have 10 years in Industry, testing hypothesis is testing. Building a product that is bad because 'X person wanted it' is bad Capitalism and will be punished. Academia doesnt have this same customer focused delivery.
      - rossdavidh 8y ago
        Hmmm...I think it depends on how fast the feedback loop of bad sales is. There are lots of cases where somebody inside a corporation wants the answer to be "X", because that's what they had advocated. It may be worse for the corporation as a whole, but it's better for that guy's career, and if the hit to the corporation is not immediately visible.
nonbel 8y ago
He also shamed a post-doc who refused to do it: https://web.archive.org/web/20170312041524/http:/www.brianwansink.com/phd-advice/the-grad-student-who-never-said-no https://web.archive.org/web/20170312041524/http:/www.brianwa...
- sampo 8y ago
  "In the meantime, this Turkish woman's resume will always have the five papers below." Turns out it won't: 2 of the 5 papers have been retracted. https://www.frontiersin.org/articles/10.3389/fpsyg.2017.02140/full https://www.frontiersin.org/articles/10.3389/fpsyg.2017.0214... https://bmcnutr.biomedcentral.com/articles/10.1186/s40795-017-0195-6 https://bmcnutr.biomedcentral.com/articles/10.1186/s40795-01...

rossdavidh 8y ago

Oh, if only we had a system to pass articles that are submitted to scientific journals to some sort of expert committee, that could look at it and see if it was likely to be valid. We could call it, I don't know, "peer checkup", or "expert review", or...oh never mind.

ble 8y ago
Peer review can't fix broken incentives.
sn41 8y ago
The issue is that peer review never pays off directly. The time spent in reviews mostly goes down a black hole. Editorships are awarded based on publication record, and not quality of reviews. As a result, more and more, reviews will tend to be shallow and uncritical.
- rossdavidh 8y ago
  Truth.

soperj 8y ago

>The post prompted a small group of skeptics to take a hard look at Mr. Wansink’s past scholarship It's funny that this is basically the peer-review version of harking.

theshadowknows 8y ago
You just wrecked my mind
pcrh 8y ago
It's also pretty close to the Baysian approach. The difference being that in the Baysian approach one is "permitted" to revise the hypothesis as new data is acquired.
- antidesitter 8y ago
  Correction: Bayesian, not Baysian.

kradroy 8y ago

I've never heard the term "HARKing" before. This is exactly what my advisor had me do on our experiments that couldn't confirm our initial hypotheses. I knew it was wrong because we set up the experiments in a way to eliminate confounding variables. So any conclusions drawn on data for hypotheses outside of those would be suspect. The experiments we ran were costly in time to do because we were using feedback from students using our learning system over a semester. I quit working with her because of that, and also because she was a total asshole. I refuse to donate to my university until she's no longer on the faculty. I don't want her benefiting from my hard-earned money.

MrEfficiency 8y ago
This is why I believe in Engineering and not the modern "Academia Science". Totally corrupt, Ive seen it when I talked to professors about getting my PhD. You prove what they want proven. One is going to have millions of customers using it, the other is going to have a few biased professors review the data.
- vkou 8y ago
  If you think this is any different from how industry works, you have not spent enough time in industry. After-the-fact-hypothesis is constantly used to retroactively justify business decisions.
  - MrEfficiency 8y ago
    I have 10 years in Industry, testing hypothesis is testing. Building a product that is bad because 'X person wanted it' is bad Capitalism and will be punished. Academia doesnt have this same customer focused delivery.
    - rossdavidh 8y ago
      Hmmm...I think it depends on how fast the feedback loop of bad sales is. There are lots of cases where somebody inside a corporation wants the answer to be "X", because that's what they had advocated. It may be worse for the corporation as a whole, but it's better for that guy's career, and if the hit to the corporation is not immediately visible.

nonbel 8y ago

He also shamed a post-doc who refused to do it: https://web.archive.org/web/20170312041524/http:/www.brianwansink.com/phd-advice/the-grad-student-who-never-said-no https://web.archive.org/web/20170312041524/http:/www.brianwa...

sampo 8y ago
"In the meantime, this Turkish woman's resume will always have the five papers below." Turns out it won't: 2 of the 5 papers have been retracted. https://www.frontiersin.org/articles/10.3389/fpsyg.2017.02140/full https://www.frontiersin.org/articles/10.3389/fpsyg.2017.0214... https://bmcnutr.biomedcentral.com/articles/10.1186/s40795-017-0195-6 https://bmcnutr.biomedcentral.com/articles/10.1186/s40795-01...