11 ms·

A researcher's response to people pointing out 150 errors in four of his papers

127 points by frgtpsswrdlame 10y ago

tyingq 10y ago
I imagine Wansink does get it. Now that he's outed, he either trolling for fun, or hoping the blasé approach is the one that quiets all the fuss the fastest.
- nonbel 10y ago
  I wouldn't be certain. A lot of researchers really are clueless about anything to do with statistics. It takes much more work/time to do real research, so the funding system has actually been selecting for people who can remain ignorant (so it is not fraud) and just produce p-hacked "results". In many areas, this has been going on for multiple generations now and you are trained to do it as a grad student.
  - froindt 10y ago
    > I wouldn't be certain. A lot of researchers really are clueless about anything to do with statistics. This is definitely true. I once saw a book with a title similar to "Statistics for Dummies" in a professors office. He had plenty of access to staticians at the university too. Unfortunately if a given field involves many people with ignorance of statistics, these problems may not be called out during the peer review process.
    - nonbel 10y ago
      >"Unfortunately if a given field involves many people with ignorance of statistics, these problems may not be called out during the peer review process." As I said, poor understanding of statistics has been a beneficial trait in many areas of research for decades now. Universities and labs are rewarded with funding because they are able to (honestly, but incorrectly) pump out more "results" than if the researchers were properly trained. So the issue goes well beyond the problems "not being called out". The problems are institutionalized. Editors and reviewers will try to force you to commit the errors in order to publish (and hence have a career). That is what they have been trained to consider science.
    - pbhjpbhj 10y ago
      >I once saw a book with a title similar to "Statistics for Dummies" in a professors office. // Could easily have been a handy reference text for lending or showing to erring students/faculty.
      - froindt 10y ago
        I really hope so. It was a stat strong department (but not statistics).
        closed 10y ago
        I have all kinds of books like that, for teaching / perusing, but now am going to sandwich them with hardcore math books out of paranoia
    - watwut 10y ago
      > He had plenty of access to staticians at the university too. Assuming the book is for his own learning, as opposed just curiosity or for students etc, there is still nothing wrong with learning from book. Assuming he really dont know the statistics. And it definitely beats up people who don't know either, but prefer to stay ignorant just so they don't look like someone who reads beginners book.
  - tyingq 10y ago
    It's worse than just not knowing statistics. These errors were in a table in one of the published papers. Every single row is calculated wrong. > (43.67 - 44.55)/sqrt(18.5^2/43 + 14.3^2/52) should be 0.2551872, rounds to 0.26, he put 0.25 > (68.65 - 66.51)/sqrt(3.67^2/43 + 9.44^2/52) should be 1.503114, rounds to 1.5, he put 1.38 > (184.83 - 178.38)/sqrt(63.7^2/43 + 45.71^2/52) should be 0.556064, rounds to 0.56, he put 0.52 source: http://andrewgelman.com/2017/01/25/statistical-heartburn-attempt-digest-four-pizza-publications-cornell-food-brand-lab/ http://andrewgelman.com/2017/01/25/statistical-heartburn-att... And that's just basic math, nothing fancy. So, he knows it's bad. I assume it's difficult to get a doctorate at Stanford and then become a professor at Cornell, so there's got to be more to this than him just being stupid.
    - throwaway729 10y ago
      > a doctorate at Stanford and then become a professor at Cornell As an aside, academia is no meritocracy. A lot of what's required to get posts at schools like these amounts to self-promotion and luck (graduating in the right year & from the right lab). Of course hard work and technical competence is also required, so your point stands.
      - tyingq 10y ago
        Yeah...wasn't saying that path implies genius. It should, though, denote the ability to do basic math, or the common sense to have someone proofread your papers. This is just so odd to me, has me wondering if he's trolling.
    - nonbel 10y ago
      There are variants on the t-statistic calculation. I assume these were all checked? Either way this should be mentioned in the methods of the paper, along with the software used... I wonder if they used excel, in that case who knows what may have happened. I've seen excel change numbers/names to dates, and all sorts of wackiness.
- JorgeGT 10y ago
  Probably he gets it, but you know, “it is difficult to get a man to understand something, when his salary depends on his not understanding it” (Upton Sinclair).
Andys 10y ago
I still don't understand: Is there any good reason that raw data should not be published along with each paper?
- wfunction 10y ago
  > I still don't understand: Is there any good reason that raw data should not be published along with each paper? Confidentiality? What if correlations give away subjects' identities?
  - nonbel 10y ago
    Does a substantial portion of the subjects actually care about this?
    - wfunction 10y ago
      > Does a substantial portion of the subjects actually care about this? I don't think that's how laws or ethics rules work, for one. And I wasn't talking about a particular experiment, for two.
      - nonbel 10y ago
        I've found I'm exceptional in being at all concerned about privacy. I think that if you asked, most people would only want minimal effort put towards confidentiality, especially since the trade-off is that other researchers won't be able to double check the analysis.
        wfunction 10y ago
        > I've found I'm exceptional in being at all concerned about privacy. I think that if you asked, most people would only want minimal effort put towards confidentiality, especially since the trade-off is that other researchers won't be able to double check the analysis. Like I said, that's not how laws work.
        10y ago
valuearb 10y ago
"except that this sort of ridiculous hype is standard operating procedure among celebrated and leading researchers in psychology and economics." Thank god it's not SOP among actual sciences.
- KKKKkkkk1 10y ago
  I hope you're being sarcastic. I'm a bit hesitant to provide examples, but a lot of top results in computer science are sold as being much more far reaching than what they really are.
  - kem 10y ago
    Speaking from my own areas of experience, I've been struck by how much some celebrated methods and results in machine learning, broadly speaking, basically amount to tinkering without any hard generalizable proofs. In at least one case I'm aware of, there was a statistical proof later that made it clear the original researchers were close, but if they had a different dataset they would have come up with a different result. In fact, as I write this, I think I recall that different researchers did come up with different results, and it was in large part due to tinkering on different datasets. So yes, even in computer science you run into similar things. Maybe not the same, but similar processes leading to similar problems.
    - throwaway729 10y ago
      The problem is that universities have a strong incentive to optimize for juicy press releases. A theorem -- unless it's 100 years old or otherwise somehow easy to relate to the general public -- does not have this effect. Winning a competition or "outperforming humans" does. Aside from bumping up the quality of K-12 science education by at least an order of magnitude or two, I'm not sure there's a good solution to this problem. It's the human condition to be impressed by some things and not others, regardless of their relative actual importance.
    - thom_kaar 10y ago
      I am very interested in the paper with the mathematical proof. Would you link it or just mention its title?
MR4D 10y ago
It's weird. If conservatives don't believe scientists, it's because the conservatives are labelled "morons". Could it be that conservatives just think there's a lot more of this crappy "science" out there, and that a situation like this is the canary in the coal mine? When presented with examples like this, it's actually _irrational_ to think that there aren't other egregious papers out there. Possibly just a few, but possibly a large number of them. Thoughts?
- wbillingsley 10y ago
  Of course there are lots. There's a bucket-load of papers get published every year, and sometimes reviewers are in a hurry too and miss things. A miniscule number of papers ever replicated, and replication studies regularly find reproducability rates of less than 50%. Fortunately, critical thought does not stop at the point of publication. When researchers read each other's papers they (hopefully) do not blindly assume them to be bug-free.
  - comstock 10y ago
    I think there's a general popular misunderstanding of the process of science as it is generally practiced. A single paper isn't as meaningful as you might think. It's more like an interesting blog post. You might use it as a starting point for your own investigation. A single paper does not represent "the truth" as we currently understand it. For that you need to have a broad scientific consensus. You might be able to get a better understanding of that from review papers citing recent advances in the field. Or from attending conferences/reading a broad range of papers yourself.
- throwaway729 10y ago
  My thought: ridiculous false equivalency. Conservatives don't distrust science carte blanc. Many of them will chuck money at asinine bullshit as long as it's billed as a way to kill the other team. Check out some of the more ridiculous stuff to come out of DoD. They distrust climate science. And they don't distrust the actual science, just the conclusions. And increasingly not even the most basic conclusion, just the most obvious causal theory behind that conclusion. Let's not throw the baby out with the bath water.
cscurmudgeon 10y ago
A sidenote: Using a snappy XKCD is not a valid logical argument even though the poster might think it makes one look hip and cool. I wish researchers were not this lazy. There are at least a handful of valid arguments that one could use rather than an XKCD cartoon.
- darkkindness 10y ago
  The argument is that Wansink is responding to criticism (like the XKCD) with an air of indifference. Wansink's response to the XKCD is simply one of the examples; the XKCD itself is not being used to support the author's argument that the act of publishing non-rigorous work is too widespread in quantitative science today.
  - cscurmudgeon 10y ago
    My point was not about Wasink or the original article. My point its that the XKCD cartoon doesn't convey anything other than snark.
- sideshowb 10y ago
  That XKCD cartoon succinctly shows the problem with multiple testing. If every scientific paper were as clear, accurate and impactful as that we would have solved half the issues with science discussed on this comment thread.
it 10y ago
Unintentional irony: "The only new thing about it is the 150 errors—when does that every happen?"
sverige 10y ago
I went to high school with Brian Wansink. He is brilliant, funny, and affable. I could hear his voice speaking his responses to the objections raised, and snickered a little bit. I'm sure that Wansink cares about his research and its quality. He's that sort of person. I don't know what happened with these papers, but it seems likely that answering bloggers publicly would not serve the greater good. It seems to me that all the hand-wringing about these four papers misses the bigger picture: What can we do to improve the practice of science across the board? Why is there almost no field of study which isn't tainted by the politics of left vs. right? My suggestion is that it began when people began to believe the idea that all political and social questions are best served with some sort of applied scientific analysis. I counter that, in fact, scientists are some of the worst people to ask for solutions to political and social problems, since most of them have almost no understanding of the relevant fields of study, which mainly include history and philosophy, and perhaps psychology. Even fields like theology, linguistics, and literature and the arts have more to say to most political problems than, say, physics or chemistry or meteorology, because they speak to human behavior rather than the behavior of inanimate substances. Please understand that I say all this not to provoke an argument, necessarily. But as someone whose formal studies were mostly in political history, in general, listening to HN talk about politics is a lot like what I must imagine it is for most on HN to listen to people on TV news shows talk about tech in general and software in particular: they sort of get it, but not really, and certainly are not as expert as they imagine themselves to be.
- apathy 10y ago
  What does any of this have to do with politics? Wansink doesn't appear to be troubled by his prodigious contributions of unreliable results to the literature, nor does he appear inclined to correct them. All of science and discovery is a joke to him. Left or right, the troubled extremes of politics start where respect for evidence and truth ends. Scientists are, nominally, people who search for truths about our universe with some degree of rigor, some respect for evidence, some appetite for truth (and usually an insanely competitive nature, but I digress). Absent that, what's the point? This isn't about society, in my opinion. It's about having some respect for evidence and truth. Without these, science isn't science. And what wansink is doing isn't science. Cold fusion had more rigor. Yet he will probably be rewarded with grants of taxpayer money, taken from fellow citizens as taxes, under threat of force. Meanwhile pediatric bone marrow transplanters will shut down their labs for "lack of funding". Or, more accurately, lack of standards in science. Yes, it affects people when dishonesty is accepted.
jtraffic 10y ago
Wansink applied to the phd program I'm in for graduate school, and didn't get it, and then he applied for a professor position at my school and didn't get it. One of the faculty members told me she thought it was "a missed opportunity." One thing I have trouble figuring out: is this indicative of a long-term trend? Or did his success create an expectation he couldn't keep up with and so felt a need to rush (or cheat)? Incidentally, there is another professor at my school who has built a career out of exposing similar problems in research (especially p-hacking). It takes a brave person to inhabit that niche. It is a service to the research community, but I'll bet he sits alone at conferences.
yeukhon 10y ago
Are we sure this wasn't some early April Fools' work?
franciscop 10y ago
See this explain XKCD, it is really informative: https://explainxkcd.com/882/ https://explainxkcd.com/882/

127 points by frgtpsswrdlame 10y ago

I imagine Wansink does get it. Now that he's outed, he either trolling for fun, or hoping the blasé approach is the one that quiets all the fuss the fastest.

nonbel 10y ago
I wouldn't be certain. A lot of researchers really are clueless about anything to do with statistics. It takes much more work/time to do real research, so the funding system has actually been selecting for people who can remain ignorant (so it is not fraud) and just produce p-hacked "results". In many areas, this has been going on for multiple generations now and you are trained to do it as a grad student.
- froindt 10y ago
  > I wouldn't be certain. A lot of researchers really are clueless about anything to do with statistics. This is definitely true. I once saw a book with a title similar to "Statistics for Dummies" in a professors office. He had plenty of access to staticians at the university too. Unfortunately if a given field involves many people with ignorance of statistics, these problems may not be called out during the peer review process.
  - nonbel 10y ago
    >"Unfortunately if a given field involves many people with ignorance of statistics, these problems may not be called out during the peer review process." As I said, poor understanding of statistics has been a beneficial trait in many areas of research for decades now. Universities and labs are rewarded with funding because they are able to (honestly, but incorrectly) pump out more "results" than if the researchers were properly trained. So the issue goes well beyond the problems "not being called out". The problems are institutionalized. Editors and reviewers will try to force you to commit the errors in order to publish (and hence have a career). That is what they have been trained to consider science.
  - pbhjpbhj 10y ago
    >I once saw a book with a title similar to "Statistics for Dummies" in a professors office. // Could easily have been a handy reference text for lending or showing to erring students/faculty.
    - froindt 10y ago
      I really hope so. It was a stat strong department (but not statistics).
      - closed 10y ago
        I have all kinds of books like that, for teaching / perusing, but now am going to sandwich them with hardcore math books out of paranoia
  - watwut 10y ago
    > He had plenty of access to staticians at the university too. Assuming the book is for his own learning, as opposed just curiosity or for students etc, there is still nothing wrong with learning from book. Assuming he really dont know the statistics. And it definitely beats up people who don't know either, but prefer to stay ignorant just so they don't look like someone who reads beginners book.
- tyingq 10y ago
  It's worse than just not knowing statistics. These errors were in a table in one of the published papers. Every single row is calculated wrong. > (43.67 - 44.55)/sqrt(18.5^2/43 + 14.3^2/52) should be 0.2551872, rounds to 0.26, he put 0.25 > (68.65 - 66.51)/sqrt(3.67^2/43 + 9.44^2/52) should be 1.503114, rounds to 1.5, he put 1.38 > (184.83 - 178.38)/sqrt(63.7^2/43 + 45.71^2/52) should be 0.556064, rounds to 0.56, he put 0.52 source: http://andrewgelman.com/2017/01/25/statistical-heartburn-attempt-digest-four-pizza-publications-cornell-food-brand-lab/ http://andrewgelman.com/2017/01/25/statistical-heartburn-att... And that's just basic math, nothing fancy. So, he knows it's bad. I assume it's difficult to get a doctorate at Stanford and then become a professor at Cornell, so there's got to be more to this than him just being stupid.
  - throwaway729 10y ago
    > a doctorate at Stanford and then become a professor at Cornell As an aside, academia is no meritocracy. A lot of what's required to get posts at schools like these amounts to self-promotion and luck (graduating in the right year & from the right lab). Of course hard work and technical competence is also required, so your point stands.
    - tyingq 10y ago
      Yeah...wasn't saying that path implies genius. It should, though, denote the ability to do basic math, or the common sense to have someone proofread your papers. This is just so odd to me, has me wondering if he's trolling.
  - nonbel 10y ago
    There are variants on the t-statistic calculation. I assume these were all checked? Either way this should be mentioned in the methods of the paper, along with the software used... I wonder if they used excel, in that case who knows what may have happened. I've seen excel change numbers/names to dates, and all sorts of wackiness.
JorgeGT 10y ago
Probably he gets it, but you know, “it is difficult to get a man to understand something, when his salary depends on his not understanding it” (Upton Sinclair).

Andys 10y ago

I still don't understand: Is there any good reason that raw data should not be published along with each paper?

wfunction 10y ago
> I still don't understand: Is there any good reason that raw data should not be published along with each paper? Confidentiality? What if correlations give away subjects' identities?
- nonbel 10y ago
  Does a substantial portion of the subjects actually care about this?
  - wfunction 10y ago
    > Does a substantial portion of the subjects actually care about this? I don't think that's how laws or ethics rules work, for one. And I wasn't talking about a particular experiment, for two.
    - nonbel 10y ago
      I've found I'm exceptional in being at all concerned about privacy. I think that if you asked, most people would only want minimal effort put towards confidentiality, especially since the trade-off is that other researchers won't be able to double check the analysis.
      - wfunction 10y ago
        > I've found I'm exceptional in being at all concerned about privacy. I think that if you asked, most people would only want minimal effort put towards confidentiality, especially since the trade-off is that other researchers won't be able to double check the analysis. Like I said, that's not how laws work.
      - 10y ago

valuearb 10y ago

"except that this sort of ridiculous hype is standard operating procedure among celebrated and leading researchers in psychology and economics." Thank god it's not SOP among actual sciences.

KKKKkkkk1 10y ago
I hope you're being sarcastic. I'm a bit hesitant to provide examples, but a lot of top results in computer science are sold as being much more far reaching than what they really are.
- kem 10y ago
  Speaking from my own areas of experience, I've been struck by how much some celebrated methods and results in machine learning, broadly speaking, basically amount to tinkering without any hard generalizable proofs. In at least one case I'm aware of, there was a statistical proof later that made it clear the original researchers were close, but if they had a different dataset they would have come up with a different result. In fact, as I write this, I think I recall that different researchers did come up with different results, and it was in large part due to tinkering on different datasets. So yes, even in computer science you run into similar things. Maybe not the same, but similar processes leading to similar problems.
  - throwaway729 10y ago
    The problem is that universities have a strong incentive to optimize for juicy press releases. A theorem -- unless it's 100 years old or otherwise somehow easy to relate to the general public -- does not have this effect. Winning a competition or "outperforming humans" does. Aside from bumping up the quality of K-12 science education by at least an order of magnitude or two, I'm not sure there's a good solution to this problem. It's the human condition to be impressed by some things and not others, regardless of their relative actual importance.
  - thom_kaar 10y ago
    I am very interested in the paper with the mathematical proof. Would you link it or just mention its title?

MR4D 10y ago

It's weird. If conservatives don't believe scientists, it's because the conservatives are labelled "morons". Could it be that conservatives just think there's a lot more of this crappy "science" out there, and that a situation like this is the canary in the coal mine? When presented with examples like this, it's actually _irrational_ to think that there aren't other egregious papers out there. Possibly just a few, but possibly a large number of them. Thoughts?

wbillingsley 10y ago
Of course there are lots. There's a bucket-load of papers get published every year, and sometimes reviewers are in a hurry too and miss things. A miniscule number of papers ever replicated, and replication studies regularly find reproducability rates of less than 50%. Fortunately, critical thought does not stop at the point of publication. When researchers read each other's papers they (hopefully) do not blindly assume them to be bug-free.
- comstock 10y ago
  I think there's a general popular misunderstanding of the process of science as it is generally practiced. A single paper isn't as meaningful as you might think. It's more like an interesting blog post. You might use it as a starting point for your own investigation. A single paper does not represent "the truth" as we currently understand it. For that you need to have a broad scientific consensus. You might be able to get a better understanding of that from review papers citing recent advances in the field. Or from attending conferences/reading a broad range of papers yourself.
throwaway729 10y ago
My thought: ridiculous false equivalency. Conservatives don't distrust science carte blanc. Many of them will chuck money at asinine bullshit as long as it's billed as a way to kill the other team. Check out some of the more ridiculous stuff to come out of DoD. They distrust climate science. And they don't distrust the actual science, just the conclusions. And increasingly not even the most basic conclusion, just the most obvious causal theory behind that conclusion. Let's not throw the baby out with the bath water.

cscurmudgeon 10y ago

A sidenote: Using a snappy XKCD is not a valid logical argument even though the poster might think it makes one look hip and cool. I wish researchers were not this lazy. There are at least a handful of valid arguments that one could use rather than an XKCD cartoon.

darkkindness 10y ago
The argument is that Wansink is responding to criticism (like the XKCD) with an air of indifference. Wansink's response to the XKCD is simply one of the examples; the XKCD itself is not being used to support the author's argument that the act of publishing non-rigorous work is too widespread in quantitative science today.
- cscurmudgeon 10y ago
  My point was not about Wasink or the original article. My point its that the XKCD cartoon doesn't convey anything other than snark.
sideshowb 10y ago
That XKCD cartoon succinctly shows the problem with multiple testing. If every scientific paper were as clear, accurate and impactful as that we would have solved half the issues with science discussed on this comment thread.

it 10y ago

Unintentional irony: "The only new thing about it is the 150 errors—when does that every happen?"

sverige 10y ago

I went to high school with Brian Wansink. He is brilliant, funny, and affable. I could hear his voice speaking his responses to the objections raised, and snickered a little bit. I'm sure that Wansink cares about his research and its quality. He's that sort of person. I don't know what happened with these papers, but it seems likely that answering bloggers publicly would not serve the greater good. It seems to me that all the hand-wringing about these four papers misses the bigger picture: What can we do to improve the practice of science across the board? Why is there almost no field of study which isn't tainted by the politics of left vs. right? My suggestion is that it began when people began to believe the idea that all political and social questions are best served with some sort of applied scientific analysis. I counter that, in fact, scientists are some of the worst people to ask for solutions to political and social problems, since most of them have almost no understanding of the relevant fields of study, which mainly include history and philosophy, and perhaps psychology. Even fields like theology, linguistics, and literature and the arts have more to say to most political problems than, say, physics or chemistry or meteorology, because they speak to human behavior rather than the behavior of inanimate substances. Please understand that I say all this not to provoke an argument, necessarily. But as someone whose formal studies were mostly in political history, in general, listening to HN talk about politics is a lot like what I must imagine it is for most on HN to listen to people on TV news shows talk about tech in general and software in particular: they sort of get it, but not really, and certainly are not as expert as they imagine themselves to be.

apathy 10y ago
What does any of this have to do with politics? Wansink doesn't appear to be troubled by his prodigious contributions of unreliable results to the literature, nor does he appear inclined to correct them. All of science and discovery is a joke to him. Left or right, the troubled extremes of politics start where respect for evidence and truth ends. Scientists are, nominally, people who search for truths about our universe with some degree of rigor, some respect for evidence, some appetite for truth (and usually an insanely competitive nature, but I digress). Absent that, what's the point? This isn't about society, in my opinion. It's about having some respect for evidence and truth. Without these, science isn't science. And what wansink is doing isn't science. Cold fusion had more rigor. Yet he will probably be rewarded with grants of taxpayer money, taken from fellow citizens as taxes, under threat of force. Meanwhile pediatric bone marrow transplanters will shut down their labs for "lack of funding". Or, more accurately, lack of standards in science. Yes, it affects people when dishonesty is accepted.

jtraffic 10y ago

Wansink applied to the phd program I'm in for graduate school, and didn't get it, and then he applied for a professor position at my school and didn't get it. One of the faculty members told me she thought it was "a missed opportunity." One thing I have trouble figuring out: is this indicative of a long-term trend? Or did his success create an expectation he couldn't keep up with and so felt a need to rush (or cheat)? Incidentally, there is another professor at my school who has built a career out of exposing similar problems in research (especially p-hacking). It takes a brave person to inhabit that niche. It is a service to the research community, but I'll bet he sits alone at conferences.

yeukhon 10y ago

Are we sure this wasn't some early April Fools' work?

franciscop 10y ago

See this explain XKCD, it is really informative: https://explainxkcd.com/882/ https://explainxkcd.com/882/