9 ms·

A Big Study About Honesty Turns Out to Be Based on Fake Data

186 points by danso 5y ago

Borrible 5y ago
Nothing is true, everything is Excel...
darksaints 5y ago
I've read two of Dan Arielly's books and while occasionally entertaining and occasionally insightful, I could never stop feeling like he was a bullshit artist. His books read as if he read a paper about people liking counterintuitive headlines with sciency-sounding explanations, and he p-hacked his entire career to take advantage of it and become the next Malcolm Gladwell. I kinda expected to see this sort of rebuttal sooner, considering how much of a thorn he was in economists' sides, but not at all surprised to see that it eventually happened.
IceDane 5y ago
In my view, there are two possibilities: 1) The data is fabricated and some of the researchers were in on it. 2) The data is fabricated, and the researchers are extremely sloppy, irresponsible and should be ashamed of their poor work ethic. How can you not have done any kind of analysis on this data, even if only for curiosity's sake? No plots of the distributions? Nothing? Come on, it's stuff that take 3 mins to whip up in python. In this age of misinformation, we don't tolerate people spouting lies even if they claim to think it is the truth. I don't see how this is any different. They didn't even attempt to do basic verification.
jessaustin 5y ago
Subtitling his book "How We Lie to Everyone" could have been a sort of tell...
deleted 5y ago
[deleted]
warning26 5y ago
Ironic.
- jagged-chisel 5y ago
  Don’tcha think?
  - falcor84 5y ago
    A little too ironic
ridaj 5y ago
Previous discussion: https://news.ycombinator.com/item?id=28210642 https://news.ycombinator.com/item?id=28210642
efitz 5y ago
Science is a process, a way to find truth. It is not in itself truth. It is subject to all the same human flaws and venality as any other human endeavor. Just because something is published in a paper doesn’t make it true. Just because the paper was peer reviewed doesn’t make it true. Just because there is a “consensus” doesn’t make it true. I’m sick and tired of people saying “the science is settled” or “trust the science”. These statements indicate a fundamental misunderstanding of what science is and how it works (or fails) in the modern world. The modern scientific method is still, in my opinion, the most powerful way we have of learning how things work, but it is not without flaws. This is just another cautionary tale.
- jollybean 5y ago
  We need some new language over this because some science is 'settled' or at least 'strongly indicative'. I wonder if 'Social Sciences' and 'Psychology' - sine we know so little about it and don't necessarily have a foundation from which to work on ... if they should be called 'Social Philosophy' that happens to use applied scientific methods. And when the news talks about published papers and scientific results, we can agree on a language like simply the term 'Unverified' or 'Not Fully Verified' to effectively mean 'Not peer reviewed or duplicated'. That way, when the first Ivermectin trial comes out we can say 'Unverified Ivermectin Study' and that 'Verification' is in progress etc..
- fastaguy88 5y ago
  This seems like a generalization that needs to limited in scope. I think the science is settled about Newton’s laws, Maxwell’s equations, Avogadro’s number, and DNA as the genetic material in bacteria, archaea, and eukaryotes. For pretty much anything taught in a 100 level science course, the science is largely settled. There are controversies in science, and those controversies make for good popular science press, but there is a lot of science that is settled.
  - Sebb767 5y ago
    > I think the science is settled about Newton’s laws Except it isn't. These laws have been extended once by general relativity already and there's a non-zero chance that the dark matter problematic will prove them insufficient once again. I don't know enough about the other fields, but it's not unlikely that they have similar problems. The point is, things are really complex and surg edge cases. And if you make final statements like this to someone who is doubting the current consensus anyway, you'll only make yourself vulnerable without convincing them.
kenta_nagamine 5y ago
Oops
koboll 5y ago
"Now some are questioning whether the scientist himself is being dishonest" kinda undersells it. The debunking offers numerous pieces of stone cold proof that the paper is straight-up academic fraud: https://datacolada.org/98 https://datacolada.org/98 And since the post says "the fourth author has made it clear to us that he was the only author in touch with the insurance company", it seems clear that Ariely personally fabricated the data.
- unreal37 5y ago
  I don't think that's "clear" at all. Even the blog link doesn't come to that conclusion.
  - sodality2 5y ago
    I think "We ran 1 million simulations to determine how often this level of similarity could emerge just by chance. Under the most generous assumptions imaginable, it didn’t happen once." is pretty damning.
    - superjan 5y ago
      Oh, somebody made it up. It’s just not proven that he was the one who made it up. It could be someone at the insurance company, or some student paid to type it in. Less likely perhaps, but not impossible.
  - jsnell 5y ago
    The trail of the data goes cold at Ariely. He is refusing to tell who supposedly gave him the data, and can't keep his story straight. He is also marked as the creator of the Excel file in the metadata [0]. If he received the data in Excel format, he wouldn't be the creator. If he received it in some other format. It's obvious that Ariely is dirty, and I'd bet we'll hear of other similar cases soon enough from other studies that he was involved in. Cheaters don't cheat just once. [0] https://datacolada.org/98#footnote_13_6142 https://datacolada.org/98#footnote_13_6142
    - 5y ago
pram 5y ago
Immediately came to mind: https://youtu.be/bST8Xp8dtY0 https://youtu.be/bST8Xp8dtY0
krolchat 5y ago
>buzzfeed news What a quality source, I will absolutely believe in it's honesty
- simonw 5y ago
  Weirdly, Buzzfeed news have been a trustworthy source of high quality investigative reporting for quite a few years at this point. I think that's partly because they took advantage of the shrinking market for traditional print newspapers and snapped up some seriously high quality talent that had been laid off from other news organizations.
- sodality2 5y ago
  Buzzfeed investigative journalism is top-tier [1]. There is a clear difference between this type of article and the rest of the cesspool. [1]: https://www.buzzfeednews.com/investigations https://www.buzzfeednews.com/investigations
deleted 5y ago
[deleted]
DominikPeters 5y ago
New info from the Buzzfeed article: > “I can see why it is tempting to think that I had something to do with creating the data in a fraudulent way,” [Ariely] told BuzzFeed News. “I can see why it would be tempting to jump to that conclusion, but I didn’t.” He added, “If I knew that the data was fraudulent, I would have never posted it.” [..] he said that all his contacts at the insurer had left and that none of them remembered what happened, either. [..] Asked by BuzzFeed News when the experiment was conducted by the insurance company, he first replied, “I don’t remember if it was 2010 or ’11. One of those things.” [..] But Ariely discussed the study’s results in a July 2008 lecture at Google [..] did not have any emails from that time to review. Another quote from an article in The Economist (https://www.economist.com/graphic-detail/2021/08/20/a-study-on-dishonesty-was-based-on-fraudulent-data https://www.economist.com/graphic-detail/2021/08/20/a-study-...): > Mr Ariely has requested that the study be retracted, as have some of his co-authors. And he is steadfast that his mistake was honest. “I did not fabricate the data,” he insists. “I am willing to do a lie detection test on that.”
- noptd 5y ago
  >“I am willing to do a lie detection test on that.” For a scientist researching honesty, I'd say that referring to a polygraph as a lie detector test raises a few questions.
throwawaysleep 5y ago
The person doing the fraud realized that ironically, the main reason to be honest is if you are being checked on it. I would totally have believed this study intuitively as my honesty is dependent on my assessment of the risk of getting caught.
- HPsquared 5y ago
  This is why learning morals and ethics don't necessarily make a person more moral or ethical: the knowledge can be used to try and "beat the system".
  - perl4ever 5y ago
    The longer someone maintains a reputation for honesty and ethics, the more they will be trusted and the more they can get away with in the end. Therefore the most advantageous way to try to "beat the system" may be to defer any dishonest behavior for an indefinitely long time, until a "rainy day". Makes me think of: https://xkcd.com/810/ https://xkcd.com/810/
unreal37 5y ago
What's weird is that this blog post is really just expanding on the 2020 study by the original authors which says that the original data is unreplicatable. And provides the original data. The original authors probably should have retracted the 2012 paper in 2020. That was a mistake not to do that. Which leads to this. This blog article is just science at work. You have a study that says X. Then people try to replicate it, and can't. And in this case, the original authors all come out and say "it seems this conclusion was wrong". Kudos for figuring out why the data can't be replicated. But it was the second 2020 study that gave them the clues, not the 2012 one.
- DominikPeters 5y ago
  Not replicable is very different from “the data were made up using a random number generator”.
- thaumasiotes 5y ago
  > And in this case, the original authors all come out and say "it seems this conclusion was wrong". Well, not really. They published a paper with that conclusion. But they were happy to lie about it elsewhere. Compare the discussion at https://statmodeling.stat.columbia.edu/2021/08/19/a-scandal-in-tedhemia-noted-study-in-psychology-first-fails-to-replicate-but-is-still-promoted-by-npr-then-crumbles-with-striking-evidence-of-data-fraud/ https://statmodeling.stat.columbia.edu/2021/08/19/a-scandal-... : > Ariely is the author of the 2012 book, “The Honest Truth About Dishonesty: How We Lie to Everyone—Especially Ourselves.” A quick google search finds him featured in a recent Freakonomics radio show called, “Is Everybody Cheating These Days?”, and a 2020 NPR segment in which he says, “One of the frightening conclusions we have is that what separates honest people from not-honest people is not necessarily character, it’s opportunity . . . the surprising thing for a rational economist would be: why don’t we cheat more?” > But . . . wait a minute! The NPR segment, dated 17 Feb 2020, states: >> That’s why Ariely describes honesty as something of a state of mind. He thinks the IRS should have people sign a pledge committing to be honest when they start working on their taxes, not when they’re done. Setting the stage for honesty is more effective than asking someone after the fact whether or not they lied. > And that last sentence links directly to the 2012 paper—indeed, it links to a copy of the paper sitting at Ariely’s website. But the new paper with the failed replications, “Signing at the beginning versus at the end does not decrease dishonesty,” by Kristal Whillans, Bazerman, Gino, Shu, Mazar, and Ariely, is dated 31 Mar 2020, and it was sent to the journal in mid-2019 > Ariely, as a coauthor of this article, had to have known for at least half a year before the NPR story that this finding didn’t replicate. But in that NPR interview he wasn’t able to spare even a moment to share this information with the credulous reporter? This seems bad, even aside from any fraud. (emphasis original)
peterthehacker 5y ago
> In the first sign of something amiss, the 13,488 drivers in the study reported equally distributed levels of driving over the period of time covered in the study. In other words, just as many people racked up 500 miles as those who drove 10,000 miles as 40,000-milers. Also, not a single one went over 50,000. I have a hard time believing that Dan Ariely didn’t know about this. The uniform distribution of mileage makes no sense, so this should’ve been caught right away. Plotting a histogram of the mileage data would’ve been one of the first things Ariely’s team did with this data.
- wjnc 5y ago
  It’s not damning per se. That depends on the sampling (or otherwise it would be a tiny, tiny insurer). I’ve done studies where I sampled an insurance population to get equal group size on a few key parameters because we then did follow up questionnaires and I needed to account for non response. No point in random sampling then because all I probably would get was data on the largest groups in the population. As it turned out people loved the subject of our questionnaire (effect of preventive measures by home owners on incidence of a whole range of common claims) and we got about a 70% response rate (that’s crazy high for cold questionnaires to customers) so the study ended up quite overpowered. Not knowing the sampling, not documenting, not having the emails or at least a zip containing the work (over 4 authors)… that’s a different ballpark.
  - whimsicalism 5y ago
    Sorry, but you seem to be suggesting that non-response bias would be responsible for the uniform distribution seen in the Update mileage digits as well as the uniform miles driven distribution?
    - wjnc 5y ago
      Theoretically, yes. I’ve used stratified sampling in this kind of research, since the underlying portfolio is so skewed.
      - 5y ago
avalys 5y ago
Patrick Winston, professor and former director of MIT CSAIL, told a brief story in class back in 2008 or so. He said it was sort of a running joke in the graduate admissions committee that people were drawn to study and research the areas of AI that corresponded to weaknesses of their own. I.e., people with poor hearing study speech recognition. People with face blindness study computer vision. People with poor writing skills study natural language processing. And so forth. "And every so often", he said, turning to face the class with a small grin on his face, "we get a grad student who comes before the committee and says he's interested in all aspects of artificial intelligence." It was amusing at the time, but had an element of truth to it as well. It's not surprising to me that dishonest people would be drawn to a career studying honesty.
ad8e 5y ago
Another instance of suspicious behavior from Ariely: he has an experiment which shows that most people cheat a little, and very few cheat a lot. The experimental method involves prying off the teeth of shredders with a screwdriver. However, others could not replicate this method. http://fraudbytes.blogspot.com/2021/08/top-honesty-researcher-dan-ariely-has.html http://fraudbytes.blogspot.com/2021/08/top-honesty-researche... Someone has claimed that getting a shredder from Home Depot is suspicious [1], but I found it perfectly innocuous. Googling for "home depot shredder" brings up several options. [1] Aaron Charlton's response at https://twitter.com/sTeamTraen/status/1428275153155264520 https://twitter.com/sTeamTraen/status/1428275153155264520
- ryelee 5y ago
  Adding on to this: Ariely at one point in time bandied around this unfounded claim about how roughly half (!) of dentists willfully misinterpret medical images in order to fill cavities that don't exist, quoting Delta Dental. But: > “But according to Dr. Ariely, he was basing his statement on a conversation he said he had with someone at Delta Dental,” said Pyle. “But he cannot cite Delta Dental in making that claim because we don’t collect any data like that which would come to such a conclusion.” > So what happened? > Ariely said he got that 50 percent figure from a Delta source who told him about “some internal analysis they have done and they told me the results. But they didn’t give me the raw data. It’s just something they told me.” > Ariely did not provide the name of the Delta medical officer, whom Ariely said was not interested in talking with me." [1] https://www.wbur.org/npr/131079116/should-you-be-suspicious-of-your-dentist-or-nprs-source https://www.wbur.org/npr/131079116/should-you-be-suspicious-....
lolthishuman 5y ago
This is actually amazing when looked at another perspective. Data can only ever be naturally false. The only time it ever has truth is because you assigned it truth, either by your experience or a delegation to third party. Truth is convergence on consensus and ultimately a bias of experience in reality. This shows that how we use the concept of truth in directing society will need to be unpacked in a time where data and fact are abundant and not directly experienced. This leads to great potential but we must reevaluate our perspectives and weights regarding certainty, truth, non truth and so on.
azinman2 5y ago
> Looking at the data only when they were aggregated, anonymized, and sent to him, he said, freed him from the work of securing ethics approval from the university to perform research on human subjects. Note he was kicked out of MIT for doing work not approved by the IRB if I recall correctly.
darepublic 5y ago
It's like a set of code, you are liable to hear all kinds of claims about the code base, what it can do, what it can't, but the only way to be sure is to go and examine it for yourself. And every time I do that I find surprises. I imagine the world of scientific experimentation could be the similar in that respect. The synopsis brought to us by scientists can range from slightly misleading to downright fraudulent. It's valuable but unless you get into the weeds of it expect your knowledge to be flawed
ALittleLight 5y ago
People are being very generous to Ariely. His theory is that someone in the insurance company faked the data such that it would prove his hypothesis before giving it to him?

186 points by danso 5y ago

Nothing is true, everything is Excel...

I've read two of Dan Arielly's books and while occasionally entertaining and occasionally insightful, I could never stop feeling like he was a bullshit artist. His books read as if he read a paper about people liking counterintuitive headlines with sciency-sounding explanations, and he p-hacked his entire career to take advantage of it and become the next Malcolm Gladwell. I kinda expected to see this sort of rebuttal sooner, considering how much of a thorn he was in economists' sides, but not at all surprised to see that it eventually happened.

IceDane 5y ago

In my view, there are two possibilities: 1) The data is fabricated and some of the researchers were in on it. 2) The data is fabricated, and the researchers are extremely sloppy, irresponsible and should be ashamed of their poor work ethic. How can you not have done any kind of analysis on this data, even if only for curiosity's sake? No plots of the distributions? Nothing? Come on, it's stuff that take 3 mins to whip up in python. In this age of misinformation, we don't tolerate people spouting lies even if they claim to think it is the truth. I don't see how this is any different. They didn't even attempt to do basic verification.

jessaustin 5y ago

Subtitling his book "How We Lie to Everyone" could have been a sort of tell...

deleted 5y ago

[deleted]

warning26 5y ago

Ironic.

jagged-chisel 5y ago
Don’tcha think?
- falcor84 5y ago
  A little too ironic

ridaj 5y ago

Previous discussion: https://news.ycombinator.com/item?id=28210642 https://news.ycombinator.com/item?id=28210642

efitz 5y ago

Science is a process, a way to find truth. It is not in itself truth. It is subject to all the same human flaws and venality as any other human endeavor. Just because something is published in a paper doesn’t make it true. Just because the paper was peer reviewed doesn’t make it true. Just because there is a “consensus” doesn’t make it true. I’m sick and tired of people saying “the science is settled” or “trust the science”. These statements indicate a fundamental misunderstanding of what science is and how it works (or fails) in the modern world. The modern scientific method is still, in my opinion, the most powerful way we have of learning how things work, but it is not without flaws. This is just another cautionary tale.

jollybean 5y ago
We need some new language over this because some science is 'settled' or at least 'strongly indicative'. I wonder if 'Social Sciences' and 'Psychology' - sine we know so little about it and don't necessarily have a foundation from which to work on ... if they should be called 'Social Philosophy' that happens to use applied scientific methods. And when the news talks about published papers and scientific results, we can agree on a language like simply the term 'Unverified' or 'Not Fully Verified' to effectively mean 'Not peer reviewed or duplicated'. That way, when the first Ivermectin trial comes out we can say 'Unverified Ivermectin Study' and that 'Verification' is in progress etc..
fastaguy88 5y ago
This seems like a generalization that needs to limited in scope. I think the science is settled about Newton’s laws, Maxwell’s equations, Avogadro’s number, and DNA as the genetic material in bacteria, archaea, and eukaryotes. For pretty much anything taught in a 100 level science course, the science is largely settled. There are controversies in science, and those controversies make for good popular science press, but there is a lot of science that is settled.
- Sebb767 5y ago
  > I think the science is settled about Newton’s laws Except it isn't. These laws have been extended once by general relativity already and there's a non-zero chance that the dark matter problematic will prove them insufficient once again. I don't know enough about the other fields, but it's not unlikely that they have similar problems. The point is, things are really complex and surg edge cases. And if you make final statements like this to someone who is doubting the current consensus anyway, you'll only make yourself vulnerable without convincing them.

kenta_nagamine 5y ago

Oops

koboll 5y ago

"Now some are questioning whether the scientist himself is being dishonest" kinda undersells it. The debunking offers numerous pieces of stone cold proof that the paper is straight-up academic fraud: https://datacolada.org/98 https://datacolada.org/98 And since the post says "the fourth author has made it clear to us that he was the only author in touch with the insurance company", it seems clear that Ariely personally fabricated the data.

unreal37 5y ago
I don't think that's "clear" at all. Even the blog link doesn't come to that conclusion.
- sodality2 5y ago
  I think "We ran 1 million simulations to determine how often this level of similarity could emerge just by chance. Under the most generous assumptions imaginable, it didn’t happen once." is pretty damning.
  - superjan 5y ago
    Oh, somebody made it up. It’s just not proven that he was the one who made it up. It could be someone at the insurance company, or some student paid to type it in. Less likely perhaps, but not impossible.
- jsnell 5y ago
  The trail of the data goes cold at Ariely. He is refusing to tell who supposedly gave him the data, and can't keep his story straight. He is also marked as the creator of the Excel file in the metadata [0]. If he received the data in Excel format, he wouldn't be the creator. If he received it in some other format. It's obvious that Ariely is dirty, and I'd bet we'll hear of other similar cases soon enough from other studies that he was involved in. Cheaters don't cheat just once. [0] https://datacolada.org/98#footnote_13_6142 https://datacolada.org/98#footnote_13_6142
  - 5y ago

pram 5y ago

Immediately came to mind: https://youtu.be/bST8Xp8dtY0 https://youtu.be/bST8Xp8dtY0

krolchat 5y ago

>buzzfeed news What a quality source, I will absolutely believe in it's honesty

simonw 5y ago
Weirdly, Buzzfeed news have been a trustworthy source of high quality investigative reporting for quite a few years at this point. I think that's partly because they took advantage of the shrinking market for traditional print newspapers and snapped up some seriously high quality talent that had been laid off from other news organizations.
sodality2 5y ago
Buzzfeed investigative journalism is top-tier [1]. There is a clear difference between this type of article and the rest of the cesspool. [1]: https://www.buzzfeednews.com/investigations https://www.buzzfeednews.com/investigations

deleted 5y ago

[deleted]

DominikPeters 5y ago

New info from the Buzzfeed article: > “I can see why it is tempting to think that I had something to do with creating the data in a fraudulent way,” [Ariely] told BuzzFeed News. “I can see why it would be tempting to jump to that conclusion, but I didn’t.” He added, “If I knew that the data was fraudulent, I would have never posted it.” [..] he said that all his contacts at the insurer had left and that none of them remembered what happened, either. [..] Asked by BuzzFeed News when the experiment was conducted by the insurance company, he first replied, “I don’t remember if it was 2010 or ’11. One of those things.” [..] But Ariely discussed the study’s results in a July 2008 lecture at Google [..] did not have any emails from that time to review. Another quote from an article in The Economist (https://www.economist.com/graphic-detail/2021/08/20/a-study-on-dishonesty-was-based-on-fraudulent-data https://www.economist.com/graphic-detail/2021/08/20/a-study-...): > Mr Ariely has requested that the study be retracted, as have some of his co-authors. And he is steadfast that his mistake was honest. “I did not fabricate the data,” he insists. “I am willing to do a lie detection test on that.”

noptd 5y ago
>“I am willing to do a lie detection test on that.” For a scientist researching honesty, I'd say that referring to a polygraph as a lie detector test raises a few questions.

throwawaysleep 5y ago

The person doing the fraud realized that ironically, the main reason to be honest is if you are being checked on it. I would totally have believed this study intuitively as my honesty is dependent on my assessment of the risk of getting caught.

HPsquared 5y ago
This is why learning morals and ethics don't necessarily make a person more moral or ethical: the knowledge can be used to try and "beat the system".
- perl4ever 5y ago
  The longer someone maintains a reputation for honesty and ethics, the more they will be trusted and the more they can get away with in the end. Therefore the most advantageous way to try to "beat the system" may be to defer any dishonest behavior for an indefinitely long time, until a "rainy day". Makes me think of: https://xkcd.com/810/ https://xkcd.com/810/

unreal37 5y ago

What's weird is that this blog post is really just expanding on the 2020 study by the original authors which says that the original data is unreplicatable. And provides the original data. The original authors probably should have retracted the 2012 paper in 2020. That was a mistake not to do that. Which leads to this. This blog article is just science at work. You have a study that says X. Then people try to replicate it, and can't. And in this case, the original authors all come out and say "it seems this conclusion was wrong". Kudos for figuring out why the data can't be replicated. But it was the second 2020 study that gave them the clues, not the 2012 one.

DominikPeters 5y ago
Not replicable is very different from “the data were made up using a random number generator”.
thaumasiotes 5y ago
> And in this case, the original authors all come out and say "it seems this conclusion was wrong". Well, not really. They published a paper with that conclusion. But they were happy to lie about it elsewhere. Compare the discussion at https://statmodeling.stat.columbia.edu/2021/08/19/a-scandal-in-tedhemia-noted-study-in-psychology-first-fails-to-replicate-but-is-still-promoted-by-npr-then-crumbles-with-striking-evidence-of-data-fraud/ https://statmodeling.stat.columbia.edu/2021/08/19/a-scandal-... : > Ariely is the author of the 2012 book, “The Honest Truth About Dishonesty: How We Lie to Everyone—Especially Ourselves.” A quick google search finds him featured in a recent Freakonomics radio show called, “Is Everybody Cheating These Days?”, and a 2020 NPR segment in which he says, “One of the frightening conclusions we have is that what separates honest people from not-honest people is not necessarily character, it’s opportunity . . . the surprising thing for a rational economist would be: why don’t we cheat more?” > But . . . wait a minute! The NPR segment, dated 17 Feb 2020, states: >> That’s why Ariely describes honesty as something of a state of mind. He thinks the IRS should have people sign a pledge committing to be honest when they start working on their taxes, not when they’re done. Setting the stage for honesty is more effective than asking someone after the fact whether or not they lied. > And that last sentence links directly to the 2012 paper—indeed, it links to a copy of the paper sitting at Ariely’s website. But the new paper with the failed replications, “Signing at the beginning versus at the end does not decrease dishonesty,” by Kristal Whillans, Bazerman, Gino, Shu, Mazar, and Ariely, is dated 31 Mar 2020, and it was sent to the journal in mid-2019 > Ariely, as a coauthor of this article, had to have known for at least half a year before the NPR story that this finding didn’t replicate. But in that NPR interview he wasn’t able to spare even a moment to share this information with the credulous reporter? This seems bad, even aside from any fraud. (emphasis original)

peterthehacker 5y ago

> In the first sign of something amiss, the 13,488 drivers in the study reported equally distributed levels of driving over the period of time covered in the study. In other words, just as many people racked up 500 miles as those who drove 10,000 miles as 40,000-milers. Also, not a single one went over 50,000. I have a hard time believing that Dan Ariely didn’t know about this. The uniform distribution of mileage makes no sense, so this should’ve been caught right away. Plotting a histogram of the mileage data would’ve been one of the first things Ariely’s team did with this data.

wjnc 5y ago
It’s not damning per se. That depends on the sampling (or otherwise it would be a tiny, tiny insurer). I’ve done studies where I sampled an insurance population to get equal group size on a few key parameters because we then did follow up questionnaires and I needed to account for non response. No point in random sampling then because all I probably would get was data on the largest groups in the population. As it turned out people loved the subject of our questionnaire (effect of preventive measures by home owners on incidence of a whole range of common claims) and we got about a 70% response rate (that’s crazy high for cold questionnaires to customers) so the study ended up quite overpowered. Not knowing the sampling, not documenting, not having the emails or at least a zip containing the work (over 4 authors)… that’s a different ballpark.
- whimsicalism 5y ago
  Sorry, but you seem to be suggesting that non-response bias would be responsible for the uniform distribution seen in the Update mileage digits as well as the uniform miles driven distribution?
  - wjnc 5y ago
    Theoretically, yes. I’ve used stratified sampling in this kind of research, since the underlying portfolio is so skewed.
    - 5y ago

avalys 5y ago

Patrick Winston, professor and former director of MIT CSAIL, told a brief story in class back in 2008 or so. He said it was sort of a running joke in the graduate admissions committee that people were drawn to study and research the areas of AI that corresponded to weaknesses of their own. I.e., people with poor hearing study speech recognition. People with face blindness study computer vision. People with poor writing skills study natural language processing. And so forth. "And every so often", he said, turning to face the class with a small grin on his face, "we get a grad student who comes before the committee and says he's interested in all aspects of artificial intelligence." It was amusing at the time, but had an element of truth to it as well. It's not surprising to me that dishonest people would be drawn to a career studying honesty.

ad8e 5y ago

Another instance of suspicious behavior from Ariely: he has an experiment which shows that most people cheat a little, and very few cheat a lot. The experimental method involves prying off the teeth of shredders with a screwdriver. However, others could not replicate this method. http://fraudbytes.blogspot.com/2021/08/top-honesty-researcher-dan-ariely-has.html http://fraudbytes.blogspot.com/2021/08/top-honesty-researche... Someone has claimed that getting a shredder from Home Depot is suspicious [1], but I found it perfectly innocuous. Googling for "home depot shredder" brings up several options. [1] Aaron Charlton's response at https://twitter.com/sTeamTraen/status/1428275153155264520 https://twitter.com/sTeamTraen/status/1428275153155264520

ryelee 5y ago
Adding on to this: Ariely at one point in time bandied around this unfounded claim about how roughly half (!) of dentists willfully misinterpret medical images in order to fill cavities that don't exist, quoting Delta Dental. But: > “But according to Dr. Ariely, he was basing his statement on a conversation he said he had with someone at Delta Dental,” said Pyle. “But he cannot cite Delta Dental in making that claim because we don’t collect any data like that which would come to such a conclusion.” > So what happened? > Ariely said he got that 50 percent figure from a Delta source who told him about “some internal analysis they have done and they told me the results. But they didn’t give me the raw data. It’s just something they told me.” > Ariely did not provide the name of the Delta medical officer, whom Ariely said was not interested in talking with me." [1] https://www.wbur.org/npr/131079116/should-you-be-suspicious-of-your-dentist-or-nprs-source https://www.wbur.org/npr/131079116/should-you-be-suspicious-....

lolthishuman 5y ago

This is actually amazing when looked at another perspective. Data can only ever be naturally false. The only time it ever has truth is because you assigned it truth, either by your experience or a delegation to third party. Truth is convergence on consensus and ultimately a bias of experience in reality. This shows that how we use the concept of truth in directing society will need to be unpacked in a time where data and fact are abundant and not directly experienced. This leads to great potential but we must reevaluate our perspectives and weights regarding certainty, truth, non truth and so on.

azinman2 5y ago

> Looking at the data only when they were aggregated, anonymized, and sent to him, he said, freed him from the work of securing ethics approval from the university to perform research on human subjects. Note he was kicked out of MIT for doing work not approved by the IRB if I recall correctly.

darepublic 5y ago

It's like a set of code, you are liable to hear all kinds of claims about the code base, what it can do, what it can't, but the only way to be sure is to go and examine it for yourself. And every time I do that I find surprises. I imagine the world of scientific experimentation could be the similar in that respect. The synopsis brought to us by scientists can range from slightly misleading to downright fraudulent. It's valuable but unless you get into the weeds of it expect your knowledge to be flawed

ALittleLight 5y ago

People are being very generous to Ariely. His theory is that someone in the insurance company faked the data such that it would prove his hypothesis before giving it to him?