8 ms·

This is some damning info. Not only as evidence of outright fraud, but also of incompetence in that fraud. Uniformly generating something that's obviously not u

This is some damning info. Not only as evidence of outright fraud, but also of incompetence in that fraud. Uniformly generating something that's obviously not uniform, not even paying attention to fonts (!), and fabricating your result by adjusting not the collected "current" data, but rather the baseline data. Ariely's response[1] to this puts the fraud on the insurance company. He rightly notes that better data anomaly testing would have caught this, but I also wonder why the company "knew" which hypothesis to put their thumb on? And where did the impetus come from to double the number of records rather than leave N=6,744? [1] http://datacolada.org/storage_strong/DanBlogComment_Aug_16_2021_final.pdf http://datacolada.org/storage_strong/DanBlogComment_Aug_16_2...

frostburg 5y ago
I don't see how this could have happened without either collusion between the insurance company and the researchers or at least one of the researchers directly tampering with the data.
- function_seven 5y ago
  Right? The insurance company doesn't have a hypothesis they're married to. You'd think they would really just want to know an actual technique to improving mileage honesty among their policyholders. It's the researcher who has the motive to show a significant finding. But I can also see Ariely's contact at the firm wanting to "help him out" by providing "good"(!) data. I'm going through the responses from the other 3 study authors, and I'm seeing a pattern in their replies. They're all—as gently and politely as possible—laying it at Dan Ariely's feet. (He does so as well in his own response; but there's a tiny whiff of skepticism—at least in my reading between lines—that he was blameless.) Mr. Bazerman's response seems the strongest in this regard. From Francesca Gino[1]: > I start all my research collaborations from a place of trust and assume that all of my co-authors provide data collected with proper care and due diligence, and that they are presented with accuracy. In the case of Study 3, I was not involved in conversations with the insurance company that conducted the field experiment, nor in any of the steps of running the study or analyzing the data. From Max H. Bazerman[2]: > The first time I saw the combined three-study paper was on February 23, 2011. On this initial reading, I thought I saw a problem with implausible data in Study 3. I raised the issue with a coauthor and was assured the data was accurate. I continued to ask questions because I was not convinced by the initial responses. When I eventually met another coauthor responsible for this portion of the work at a conference, I was provided more plausible explanations and felt more confidence in the underlying data. I would note that this coauthor quickly showed me the data file on a laptop; I did not nor did I have others examine the data more carefully. From Nina Mazar[3]: > I want to make clear that I was not involved in conducting the field study, had no interactions with the insurance company, and don’t know when, how, or by whom exactly the data was collected and entered. I have no knowledge of who fabricated the data. and > This whole situation has reinforced the importance of having an explicit team contract, that clearly establishes roles, responsibilities, and processes [1] http://datacolada.org/storage_strong/Gino-memo-data-colada-August16.pdf http://datacolada.org/storage_strong/Gino-memo-data-colada-A... [2] http://datacolada.org/storage_strong/fraud.resonse.max_.8.13.21.pdf http://datacolada.org/storage_strong/fraud.resonse.max_.8.13... [3] http://datacolada.org/storage_strong/20210816_NM-Response2DataColada.pdf http://datacolada.org/storage_strong/20210816_NM-Response2Da...
  - duxup 5y ago
    Sounds like the classic case of a bunch of folks involved and the important thing is 'nobody's job' and so nobody does it. You see this in engineering failures when a bunch of companies or groups are involved to limited extents and everyone does their part, but nobody does something important because it wasn't defined who would do that.
  - rz2k 5y ago
    Could it be one executive wanting get credit for his brilliant initiative by scientifically "proving" how effective it is? Using the uniform distribution makes it seem like it wasn't one of the researchers or anyone at the insurance company who has studied actuarial science.
  - namelessoracle 5y ago
    > The insurance company doesn't have a hypothesis they're married to While the insurance company as a whole didn't. There may have been someone inside the insurance company who wanted to justify spending funds to do this research and wanted a laurel on their cap about how they improved the accuracy of self reporting by X or Y value.
- xondono 5y ago
  While I see the possibility of someone at the insurance company being the “motivated party” to fudge with the data, I agree that it’s difficult to see how one would do about it if one does not know what the researchers would look at. And if I was in charge of the study (or anyone with enough experience I think), that “test” should have been kept secret from the company. This to me implies at least blurry boundaries between the researcher (in this case Ariely) and someone at the company. I will also admit not to be very impartial to this, being to this day very unconvinced by Arielys research and especially his conclusions and the platform he has built around them.
frankster 5y ago
The most interesting thing about Dan Ariely's response is that the final line, containing his name, is written in a different font to the rest of his statement!

frostburg 5y ago

I don't see how this could have happened without either collusion between the insurance company and the researchers or at least one of the researchers directly tampering with the data.

function_seven 5y ago
Right? The insurance company doesn't have a hypothesis they're married to. You'd think they would really just want to know an actual technique to improving mileage honesty among their policyholders. It's the researcher who has the motive to show a significant finding. But I can also see Ariely's contact at the firm wanting to "help him out" by providing "good"(!) data. I'm going through the responses from the other 3 study authors, and I'm seeing a pattern in their replies. They're all—as gently and politely as possible—laying it at Dan Ariely's feet. (He does so as well in his own response; but there's a tiny whiff of skepticism—at least in my reading between lines—that he was blameless.) Mr. Bazerman's response seems the strongest in this regard. From Francesca Gino[1]: > I start all my research collaborations from a place of trust and assume that all of my co-authors provide data collected with proper care and due diligence, and that they are presented with accuracy. In the case of Study 3, I was not involved in conversations with the insurance company that conducted the field experiment, nor in any of the steps of running the study or analyzing the data. From Max H. Bazerman[2]: > The first time I saw the combined three-study paper was on February 23, 2011. On this initial reading, I thought I saw a problem with implausible data in Study 3. I raised the issue with a coauthor and was assured the data was accurate. I continued to ask questions because I was not convinced by the initial responses. When I eventually met another coauthor responsible for this portion of the work at a conference, I was provided more plausible explanations and felt more confidence in the underlying data. I would note that this coauthor quickly showed me the data file on a laptop; I did not nor did I have others examine the data more carefully. From Nina Mazar[3]: > I want to make clear that I was not involved in conducting the field study, had no interactions with the insurance company, and don’t know when, how, or by whom exactly the data was collected and entered. I have no knowledge of who fabricated the data. and > This whole situation has reinforced the importance of having an explicit team contract, that clearly establishes roles, responsibilities, and processes [1] http://datacolada.org/storage_strong/Gino-memo-data-colada-August16.pdf http://datacolada.org/storage_strong/Gino-memo-data-colada-A... [2] http://datacolada.org/storage_strong/fraud.resonse.max_.8.13.21.pdf http://datacolada.org/storage_strong/fraud.resonse.max_.8.13... [3] http://datacolada.org/storage_strong/20210816_NM-Response2DataColada.pdf http://datacolada.org/storage_strong/20210816_NM-Response2Da...
- duxup 5y ago
  Sounds like the classic case of a bunch of folks involved and the important thing is 'nobody's job' and so nobody does it. You see this in engineering failures when a bunch of companies or groups are involved to limited extents and everyone does their part, but nobody does something important because it wasn't defined who would do that.
- rz2k 5y ago
  Could it be one executive wanting get credit for his brilliant initiative by scientifically "proving" how effective it is? Using the uniform distribution makes it seem like it wasn't one of the researchers or anyone at the insurance company who has studied actuarial science.
- namelessoracle 5y ago
  > The insurance company doesn't have a hypothesis they're married to While the insurance company as a whole didn't. There may have been someone inside the insurance company who wanted to justify spending funds to do this research and wanted a laurel on their cap about how they improved the accuracy of self reporting by X or Y value.
xondono 5y ago
While I see the possibility of someone at the insurance company being the “motivated party” to fudge with the data, I agree that it’s difficult to see how one would do about it if one does not know what the researchers would look at. And if I was in charge of the study (or anyone with enough experience I think), that “test” should have been kept secret from the company. This to me implies at least blurry boundaries between the researcher (in this case Ariely) and someone at the company. I will also admit not to be very impartial to this, being to this day very unconvinced by Arielys research and especially his conclusions and the platform he has built around them.

frankster 5y ago

The most interesting thing about Dan Ariely's response is that the final line, containing his name, is written in a different font to the rest of his statement!