6 ms·

Fabricated data in research about honesty

107 points by danso 3y ago

jvanderbot 3y ago
This is the latest in my most frequently learned lesson: Nobody thinks they are doing anything wrong. They think they are shoring up a weakness in a fundamentally good argument or cause.
- AnimalMuppet 3y ago
  Ouch. That's not just scientific dishonesty. It's also politics, and the legal system, and advertising, and internet conversations, and...
- munchler 3y ago
  Yep. There are very few true villains in reality. Just a lot of people with different values and priorities. Even Vlad Putin probably thinks he’s on the side of the angels.
  - functoid 3y ago
    [flagged]
  - xdennis 3y ago
    Please don't call that guy Vlad. Vlad is short for Vladislav. You're looking vor Vova. It's like saying Ben is short for Bennett.
    - tomrod 3y ago
      Ben can be short for Bennett?
- godelski 3y ago
  While I generally go with this, it definitely isn't an absolutely true statement and we should even modify it. People know they are doing wrong, they just justify it with reasons why it is okay to do the wrong thing. I'm pretty sure that these people knew they were doing a big wrong, but I'm sure they justified it with "well everyone fudges the data a little." (this wasn't little)
  - bena 3y ago
    I'm going to go with both of you. They know what they're doing is wrong. But I don't think they did it because "everyone does it", they did it because they think they're ultimately right. They believe in their results, they know in their heart of hearts that their conclusions are true. It's just that not all of the data lines up. It's incomplete, inconclusive, etc. Yes, it is wrong to fabricate data, however, over time, results will bear out the truth of their statements and it won't matter. They are of the camp that process doesn't matter as long as you reach the right conclusion. Which is wrong. Process is kind of the only thing that matters.
- verisimi 3y ago
  "the greater good" "THE GREATER GOOD" https://www.youtube.com/watch?v=5u8vd_YNbTw https://www.youtube.com/watch?v=5u8vd_YNbTw
sproketboy 3y ago
[dead]
impalallama 3y ago
> The two worked together on a paper about how to "nudge" people to be more honest on things like forms or tax returns. Their trick: move the location where people attest that they have filled in a form honestly from the bottom of the form to the top. Another strike against the entire concept of "nudges". Its become clear that it was a bit of a liberal pipe dream saying you could accomplish huge changes in people or even society with imperceptible "nudges" that conveniently meant you didn't need to address systemic issues and stone wall any attempts at grander change. "Be more honest by changing a form." "Make people less fat by moving where the produce is in the store." Turns out things are more complicated.
- jstx1 3y ago
  > conveniently meant you didn't need to address systemic issues and stone wall any attempts at grander change Who is claiming this?
  - impalallama 3y ago
    https://en.wikipedia.org/wiki/Nudge_theory https://en.wikipedia.org/wiki/Nudge_theory
    - deleted 3y ago
      [deleted]
- gruez 3y ago
  >Its become clear that it was a bit of a liberal pipe dream saying [...] that conveniently meant you didn't need to address systemic issues and stone wall any attempts at grander change. Really? I thought liberals were all about addressing systemic issues (eg. systemic racism), and all about "grander change" (eg. defund the police/ICE, eat the rich, etc.).
  - lo_zamoyski 3y ago
    The word "liberal" is terribly overloaded to mean all sorts of things. The OP's intended meaning seems to be liberal in the sense of a lack of coercive measures, preferring incentives instead. Your meaning seems to mean "leftist". FWIW, I use it to describe that which is Lockean in substance (which entails a kind of radical, atomistic individualism that takes the individual to be the basic unit of society, and not the family), which basically means that most of the political spectrum is implicated.
boberoni 3y ago
The podcast “If Books Could Kill” has a great episode that breaks down and debunks the book “Nudge”. The podcast also factchecks various other pop-science and so called “airport books”.
- gloryjulio 3y ago
  I am glad that with internet I could cut out those word salad crap as much as possible, and stick with more reputable and updated sources. Basically we have a better package management system to reduce the supply chain attack
  - firewolf34 3y ago
    Now we just have to apply that concept to the watering-down which occurs when they try to take white-papers and disseminate them to the public through news articles and etc. It is important for the public to learn - and they will need assistance to understand findings that are outside of their usual realm of knowledge/experience. Unfortunately, the resources for this are shaky, profit-motivated, and arbitrary.
- verisimi 3y ago
  'debunks Nudge?' Someone should tell the politicians and PR consultants...
ldehaan 3y ago
[dead]
rahimnathwani 3y ago
The two linked source posts are excellent: http://datacolada.org/109 http://datacolada.org/109 http://datacolada.org/98 http://datacolada.org/98 This latest one has great sleuthing examples: http://datacolada.org/112 http://datacolada.org/112 Their analysis suggested that the data (originally collected using Qualtrics) had been modified, i.e. that the data in the data files would not match the original data (still stored on Qualtrics' servers). It turned out they were right: We have received confirmation, from outside of Harvard, that Harvard's investigators did look at the original Qualtrics data file and that the data had been modified.
apienx 3y ago
Recommended video on the matter: https://youtu.be/d2Tm3Yx4HWI https://youtu.be/d2Tm3Yx4HWI
cscheid 3y ago
I’m a former academic who did a bit of work that involved statistical analysis of data in a similar way as these studies. Fuck. These. People. We put so much work making sure the analysis is right. Sweat the design, sweat the data collection, the consent forms, and dropped the studies where the result wasn’t there, as well as reporting null results. I specifically remember having to have a terrible conversation with a phd student that, after doing a power analysis, setting up a pilot study, and collecting enough data, we got p=0.054 on the relevant test. “Sorry, we can’t add additional participants; we can’t try a new analysis. We did the right thing and we should report a non significant result. It sucks.” And these fuckers just go ahead and forge data to land TED talks? Yes, let’s also have a conversation about incentives and the system. But individually, I reserve my sympathy for better people. Salt the earth they walk on, and rake them over coils. This is beyond reprehensible. Fuck. Them.
- nyokodo 3y ago
  > This is beyond reprehensible. It is easy enough to make honest mistakes and spread misinformation before it can be properly replicated that we do still require scientists to be honest. Active deception should come with a high cost. There should be a scientific equivalent of disbarment.
  - cscheid 3y ago
    > Active deception should come with a high cost Yes, I wish so too. For what's worth, I'd buy "either of Dan Ariely and Francesca Gino write a NYT bestseller book in 5 years, go on a redemption tour, and nothing new happens" for $0.80 on a prediction market.
- naijaboiler 3y ago
  I think this is a misuse of p-value.
  - crackedbassoon 3y ago
    What do you mean?
dang 3y ago
Related. Others? Fraudulent data raise questions about superstar honesty researcher (2021) - https://news.ycombinator.com/item?id=36726485 https://news.ycombinator.com/item?id=36726485 - July 2023 (33 comments) UCLA professor refuses to cover for Dan Ariely in issue of data provenance - https://news.ycombinator.com/item?id=36684242 https://news.ycombinator.com/item?id=36684242 - July 2023 (131 comments) Harvard ethics professor allegedly fabricated multiple studies - https://news.ycombinator.com/item?id=36665247 https://news.ycombinator.com/item?id=36665247 - July 2023 (215 comments) Harvard dishonesty expert accused of dishonesty - https://news.ycombinator.com/item?id=36424090 https://news.ycombinator.com/item?id=36424090 - June 2023 (201 comments) Noted study in psychology fails to replicate, crumbles with evidence of fraud - https://news.ycombinator.com/item?id=28264097 https://news.ycombinator.com/item?id=28264097 - Aug 2021 (102 comments) A Big Study About Honesty Turns Out to Be Based on Fake Data - https://news.ycombinator.com/item?id=28257860 https://news.ycombinator.com/item?id=28257860 - Aug 2021 (90 comments) Evidence of fraud in an influential field experiment about dishonesty - https://news.ycombinator.com/item?id=28210642 https://news.ycombinator.com/item?id=28210642 - Aug 2021 (51 comments)
whynotmaybe 3y ago
Kinda ironic that my most remembered detail about Dan Ariely's book is that "everyone cheats a little". https://www.wired.com/2009/02/ted-1/ https://www.wired.com/2009/02/ted-1/
- vacuity 3y ago
  Wouldn't that be the opposite of irony?
  - whynotmaybe 3y ago
    I guess that Alanis Morissette tainted my perception of irony.
  - naijaboiler 3y ago
    Obviously he was studying himself

107 points by danso 3y ago

This is the latest in my most frequently learned lesson: Nobody thinks they are doing anything wrong. They think they are shoring up a weakness in a fundamentally good argument or cause.

AnimalMuppet 3y ago
Ouch. That's not just scientific dishonesty. It's also politics, and the legal system, and advertising, and internet conversations, and...
munchler 3y ago
Yep. There are very few true villains in reality. Just a lot of people with different values and priorities. Even Vlad Putin probably thinks he’s on the side of the angels.
- functoid 3y ago
  [flagged]
- xdennis 3y ago
  Please don't call that guy Vlad. Vlad is short for Vladislav. You're looking vor Vova. It's like saying Ben is short for Bennett.
  - tomrod 3y ago
    Ben can be short for Bennett?
godelski 3y ago
While I generally go with this, it definitely isn't an absolutely true statement and we should even modify it. People know they are doing wrong, they just justify it with reasons why it is okay to do the wrong thing. I'm pretty sure that these people knew they were doing a big wrong, but I'm sure they justified it with "well everyone fudges the data a little." (this wasn't little)
- bena 3y ago
  I'm going to go with both of you. They know what they're doing is wrong. But I don't think they did it because "everyone does it", they did it because they think they're ultimately right. They believe in their results, they know in their heart of hearts that their conclusions are true. It's just that not all of the data lines up. It's incomplete, inconclusive, etc. Yes, it is wrong to fabricate data, however, over time, results will bear out the truth of their statements and it won't matter. They are of the camp that process doesn't matter as long as you reach the right conclusion. Which is wrong. Process is kind of the only thing that matters.
verisimi 3y ago
"the greater good" "THE GREATER GOOD" https://www.youtube.com/watch?v=5u8vd_YNbTw https://www.youtube.com/watch?v=5u8vd_YNbTw

sproketboy 3y ago

[dead]

impalallama 3y ago

> The two worked together on a paper about how to "nudge" people to be more honest on things like forms or tax returns. Their trick: move the location where people attest that they have filled in a form honestly from the bottom of the form to the top. Another strike against the entire concept of "nudges". Its become clear that it was a bit of a liberal pipe dream saying you could accomplish huge changes in people or even society with imperceptible "nudges" that conveniently meant you didn't need to address systemic issues and stone wall any attempts at grander change. "Be more honest by changing a form." "Make people less fat by moving where the produce is in the store." Turns out things are more complicated.

jstx1 3y ago
> conveniently meant you didn't need to address systemic issues and stone wall any attempts at grander change Who is claiming this?
- impalallama 3y ago
  https://en.wikipedia.org/wiki/Nudge_theory https://en.wikipedia.org/wiki/Nudge_theory
  - deleted 3y ago
    [deleted]
gruez 3y ago
>Its become clear that it was a bit of a liberal pipe dream saying [...] that conveniently meant you didn't need to address systemic issues and stone wall any attempts at grander change. Really? I thought liberals were all about addressing systemic issues (eg. systemic racism), and all about "grander change" (eg. defund the police/ICE, eat the rich, etc.).
- lo_zamoyski 3y ago
  The word "liberal" is terribly overloaded to mean all sorts of things. The OP's intended meaning seems to be liberal in the sense of a lack of coercive measures, preferring incentives instead. Your meaning seems to mean "leftist". FWIW, I use it to describe that which is Lockean in substance (which entails a kind of radical, atomistic individualism that takes the individual to be the basic unit of society, and not the family), which basically means that most of the political spectrum is implicated.

boberoni 3y ago

The podcast “If Books Could Kill” has a great episode that breaks down and debunks the book “Nudge”. The podcast also factchecks various other pop-science and so called “airport books”.

gloryjulio 3y ago
I am glad that with internet I could cut out those word salad crap as much as possible, and stick with more reputable and updated sources. Basically we have a better package management system to reduce the supply chain attack
- firewolf34 3y ago
  Now we just have to apply that concept to the watering-down which occurs when they try to take white-papers and disseminate them to the public through news articles and etc. It is important for the public to learn - and they will need assistance to understand findings that are outside of their usual realm of knowledge/experience. Unfortunately, the resources for this are shaky, profit-motivated, and arbitrary.
verisimi 3y ago
'debunks Nudge?' Someone should tell the politicians and PR consultants...

ldehaan 3y ago

[dead]

rahimnathwani 3y ago

The two linked source posts are excellent: http://datacolada.org/109 http://datacolada.org/109 http://datacolada.org/98 http://datacolada.org/98 This latest one has great sleuthing examples: http://datacolada.org/112 http://datacolada.org/112 Their analysis suggested that the data (originally collected using Qualtrics) had been modified, i.e. that the data in the data files would not match the original data (still stored on Qualtrics' servers). It turned out they were right: We have received confirmation, from outside of Harvard, that Harvard's investigators did look at the original Qualtrics data file and that the data had been modified.

apienx 3y ago

Recommended video on the matter: https://youtu.be/d2Tm3Yx4HWI https://youtu.be/d2Tm3Yx4HWI

cscheid 3y ago

I’m a former academic who did a bit of work that involved statistical analysis of data in a similar way as these studies. Fuck. These. People. We put so much work making sure the analysis is right. Sweat the design, sweat the data collection, the consent forms, and dropped the studies where the result wasn’t there, as well as reporting null results. I specifically remember having to have a terrible conversation with a phd student that, after doing a power analysis, setting up a pilot study, and collecting enough data, we got p=0.054 on the relevant test. “Sorry, we can’t add additional participants; we can’t try a new analysis. We did the right thing and we should report a non significant result. It sucks.” And these fuckers just go ahead and forge data to land TED talks? Yes, let’s also have a conversation about incentives and the system. But individually, I reserve my sympathy for better people. Salt the earth they walk on, and rake them over coils. This is beyond reprehensible. Fuck. Them.

nyokodo 3y ago
> This is beyond reprehensible. It is easy enough to make honest mistakes and spread misinformation before it can be properly replicated that we do still require scientists to be honest. Active deception should come with a high cost. There should be a scientific equivalent of disbarment.
- cscheid 3y ago
  > Active deception should come with a high cost Yes, I wish so too. For what's worth, I'd buy "either of Dan Ariely and Francesca Gino write a NYT bestseller book in 5 years, go on a redemption tour, and nothing new happens" for $0.80 on a prediction market.
naijaboiler 3y ago
I think this is a misuse of p-value.
- crackedbassoon 3y ago
  What do you mean?

dang 3y ago

Related. Others? Fraudulent data raise questions about superstar honesty researcher (2021) - https://news.ycombinator.com/item?id=36726485 https://news.ycombinator.com/item?id=36726485 - July 2023 (33 comments) UCLA professor refuses to cover for Dan Ariely in issue of data provenance - https://news.ycombinator.com/item?id=36684242 https://news.ycombinator.com/item?id=36684242 - July 2023 (131 comments) Harvard ethics professor allegedly fabricated multiple studies - https://news.ycombinator.com/item?id=36665247 https://news.ycombinator.com/item?id=36665247 - July 2023 (215 comments) Harvard dishonesty expert accused of dishonesty - https://news.ycombinator.com/item?id=36424090 https://news.ycombinator.com/item?id=36424090 - June 2023 (201 comments) Noted study in psychology fails to replicate, crumbles with evidence of fraud - https://news.ycombinator.com/item?id=28264097 https://news.ycombinator.com/item?id=28264097 - Aug 2021 (102 comments) A Big Study About Honesty Turns Out to Be Based on Fake Data - https://news.ycombinator.com/item?id=28257860 https://news.ycombinator.com/item?id=28257860 - Aug 2021 (90 comments) Evidence of fraud in an influential field experiment about dishonesty - https://news.ycombinator.com/item?id=28210642 https://news.ycombinator.com/item?id=28210642 - Aug 2021 (51 comments)

whynotmaybe 3y ago

Kinda ironic that my most remembered detail about Dan Ariely's book is that "everyone cheats a little". https://www.wired.com/2009/02/ted-1/ https://www.wired.com/2009/02/ted-1/

vacuity 3y ago
Wouldn't that be the opposite of irony?
- whynotmaybe 3y ago
  I guess that Alanis Morissette tainted my perception of irony.
- naijaboiler 3y ago
  Obviously he was studying himself