20 ms·

UCLA professor refuses to cover for Dan Ariely in issue of data provenance

149 points by kblissett 3y ago

throwaway5959 3y ago
I had such high regard for academia while I was in school. Glad I didn’t end up going down that path, it seems like every day recently there have been stories like this posted to Hacker News. Behavioral economics especially seems hard hit.
- pocketsand 3y ago
  Seems like behavioral economics really just receives splash damage from what is the massive fraud at the heart of experimental social psychology. At this point I basically assume any catchy finding from the field is at best HARKed or p-hacked and at worst completely fraudulent.
  - chongli 3y ago
    Why aren't journals requiring pre-registration? It's an absolute catastrophe and no one seems to care!
    - pocketsand 3y ago
      Pre-registration protects against HARKing and p-hacking, but it unfortunately doesn't protect at all against outright fraud, which we appear to see from the likes of Ariely and Gino.
      - chongli 3y ago
        Right. Outright fraud needs to be dealt with much more severely. Total disgrace, loss of tenure, and potential prison time.
  - FreshStart 3y ago
    The field migrated to were the data is. Some communication dark departments within Meta and Google, sponsored by three letters. You can even do ethical virtual experiments with the data of 3 billion people. Just specify the situation, the person and what's to happen.. Voila. All things happen regularly at some point..
    - esafak 3y ago
      > The field migrated to were the data is. Some communication dark departments within Meta and Google, sponsored by three letters. You really need to clarify this.
      - cool_dude85 3y ago
        I think it's pretty clear what the post is saying. It doesn't need to be clarified, it needs to be supported, though the nature of the claim is such that it would be tough to provide direct evidence.
      - meepmorp 3y ago
        I’m going with: > The government is trying to control your mind through the internet
      - scrps 3y ago
        Same techniques (largely derived from psychology, statistics, sociology, etc) used in social media marketing, intent tracking, etc has uses in national security. Same thing that might let meta know you might be interested in batman pajamas can be used to sort out if you are being radicalized to action by some group. Interesting example is fractalrabbit. Group of people who both have the theoretical training and practical experience is probably pretty small so private/public crossover isn't unreasonable.
- passwordoops 3y ago
  Me too, then following my Ph.D at a relatively small school went off to a post-doc at a major research university where the grants are millions or you're worthless... That environment essentially forces you to commit fraud so I got out pretty quick
- cbsmith 3y ago
  Consider how many studies are being produced each day. That there is a story every day like this suggests that academia is making an effort to identify bad actors, but the frequency of discovery is actually quite low.
  - tnecniv 3y ago
    Well we don’t know how many people falsify data because, if it goes undiscovered, we don’t know it happened. It could easily be that the rate at which data is falsified is quite low and we are good at catching it, or it is high and we are bad at catching it. I would think instances of fabrication are quite low — it’s high-risk, medium reward. Even if it goes undiscovered, academia is a marketing game, and you can’t control what other people feel is impactful or not. It’s very easy to do bad science without falsifying data, which is probably the deadliest sin in academia.
    - cbsmith 3y ago
      > Well we don’t know how many people falsify data because, if it goes undiscovered, we don’t know it happened. It could easily be that the rate at which data is falsified is quite low and we are good at catching it, or it is high and we are bad at catching it. Yes, which is why I said the frequency of discovery is low. Seeing a story in the paper about fraud every day doesn't really tell us anything about the rate of fraudulence, which I was hoping would become transparent with my point.
- lolinder 3y ago
  To be fair, industry has a similar problem: every day there's a new company that gets in trouble for abusing customer data, losing customer data, abusing customer trust, making it hard to cancel a subscription, etc, etc. People in general tend to be dishonest if they think they can get away with it. I'm still glad I didn't go into academia for many other reasons, but I don't think I dodged an ethics bullet.
  - notahacker 3y ago
    yeah, this Reputation probably matters more for academia (people go into it because they think the research is important, not for the money, and they think they're standing on the shoulder of giants and not amiable bullshitters taking the credit for findings actually created by their overworked research assistants kludging the data), but if you think academia is full of lies and half truths relative to other professions, you haven't encountered many startups!
- usednet 3y ago
  Could somebody provide me some links to other recent scandals in academia/research? Very interested.
  - pocketsand 3y ago
    Check out the datacolada web site. https://datacolada.org/ https://datacolada.org/
  - tnecniv 3y ago
    I forget the guys name (he’s German iirc), but a very impactful one was at Bell Labs in the 00’s. STEM PhD programs now all have a research ethics component because of him. While you can’t teach someone to be ethical, it at least hammers home how many people you are defrauding by falsifying data, which includes everyone from your funding organizations to the lowly PhD students whose advisors are making them iterate on your study and don’t understand why, after three years, it still isn’t working.
  - dmvdoug 3y ago
    Retraction Watch, maybe? https://retractionwatch.com/ https://retractionwatch.com/
- tnecniv 3y ago
  While fields are small, academia is a big place. Most academics won’t falsify data — from a pragmatic perspective, that’s their career on the line, and from an emotional perspective, their academic work is tied to their self-image. It’s also arguably a lot harder to do in the hard sciences but there have still been examples. In the hard sciences, I feel like the solution is some method to increase how much people at different labs talk to one another (in addition to including all raw data and processing code from experiments). If an impactful article comes out, labs will try to build on it. Not every lab will have the equipment but those in the sub area might. If you’re trying to iterate on a study, and you can’t reproduce the base result before adding your modifications, you might just think you’re dumb or doing something dumb. There’s likely nobody outside maybe your research group who can help you. However, if it was easier to talk to other researchers in the field (like a web forum), you might find out that nobody has been able to get the original experiment working! In the social sciences, I’m not sure what you do besides running automated checks on submitted data. Humans are bad at generating things that look random (for example, we use more 3s and 7s than nature), and you can probably flag warnings. Either that or include paper trails like a lab notebook in chemistry — which is considered a legal document and thus must be kept in pen. That way you can establish a chain of custody for the data from collection to publication.
  - pevey 3y ago
    > Most academics won’t falsify data — from a pragmatic perspective, that’s their career on the line In my experience, this is absolutely not true. I mostly only have first-hand observations in the economics, finance, and marketing disciplines. They left me EXTREMELY cynical about all academics. I have seen: 1) A well-known professor at an Ivy-league B-school openly talking to his class about how the NAR funded his latest study on the real estate market, which (of course) found that the market was not overvalued. This was 2007. The co-author on the paper was the dean of the B-school. He is no longer the dean, but only because he seems to have moved on to more lucrative consulting roles. 2) An older professor, who was one of the first to be in the group labeled "behavioral economics," when asked why he wasn't writing a book focused on more mainstream readers like some other such big names were at the time, STRONGLY hinting if you filled in the blanks although not outright saying that their ethics were questionable, they were all forming hypotheses that were useful for for consulting dollars and dredging up data to support it, and he was too old school (and too old) to be interested in playing that game. 3) A full cohort of marketing grad students (6-7) talking openly about how the system is messed up, and to get a job, you absolutely have to have a good dissertation with a strong results, but to avoid p-hacking allegations, you have to shoot in the dark. So you're left in this situation where you commit financial and career suicide if you DON'T make up data to support your thesis. It is not rare. It is an open secret. No one likes it. They hate it. But they also feel like they are the stupid idiot if they try to fight against it alone. It is how the system works. Only a true saint would accept data that actually does NOT support their hypothesis because it makes it unlikely they will get a job in a very competitive market. Some people do report non-findings, but they are rare. IMO, academic departments should specifically seek out graduates who were willing to report that and stuck by it. But that is not the way the world works. As another poster pointed out, we all like stories, and no significance means no good story to tell.
- rayiner 3y ago
  Charlatans took advantage of the reputation of “science” in the 20th century. Thankfully, public perception is already adjusting.
- scrum-treats 3y ago
  Experimentation in moral psychology without robust, independently verified design is an incredibly slippery slope. This is why certain programs require interdisciplinary approaches (e.g., philosophy, psychology, linguistics, math, physics, neuroscience, biology, history). Otherwise people just make stuff up. And if there's no one to run independent replication studies, and no journals require replication studies prior to publication, then we get... this. Just wait for people to dive into the false equivalence of scales and measures used in political research. It's very much "make it up and see if it sticks." The goal appears to be Psychology Today, not scientific rigor. I wrote about this in graduate school, as part of my hefty statistics coursework, it bothered me so much.
passwordoops 3y ago
Wow to say the least. I have Predictably Irrational on my shelf and have highly recommend it in the past. Sad to see the whole field failing the irreproducibility task... I'm very happy to see this type of scrutiny come to headline research but I really wish it begins at the beginning, with funding agencies and journals. And at the very least the moment a university sends out a breathless PR, the media all scrutinize the data provenance... A boy can wish
- appleflaxen 3y ago
  "Failing irreproducibility" is a double negative that I don't think you intended.
  - neilv 3y ago
    Irregardless... :)
whyage 3y ago
For balance, here's Dan Arieli's response (Google-translated from the Hebrew source): It's ironic that you try to attribute unprofessionalism to me using unprofessional tools. The experiments conducted by Prof. Siniver are not reproductions of experiments I conducted, but rather experiments that are clearly different in many essential differences. A claim for refutation through a completely different experiment is a provocation without a professional basis. Researchers are required to keep data on studies for 5 years. I do not have data from studies that were carried out 15 years ago or more, and I have no way to find them either. This is not to criticize my work, and any attempt to do so is manipulation of the viewers. I don't do the research alone. Throughout 30 years of research I had 248 scientific writing partners, among them the most respected researchers in the field. I led 2 labs at MIT and another lab at Duke where hundreds of interns, students and research assistants worked alongside me. You understand that if the claim is that I fabricated studies, you accuse a huge group of academics from dozens of universities in the world, when the only "evidence" for this is that I failed to prove "that I don't have a sister", that is, to provide documentation that I no longer have. The experiment in which Prof. Amy Droley assisted took place over 15 years ago. I don't know why she remembers the things as she described to you. At first she didn't remember taking part in it at all, and only later did she remember. I and the other two professors who signed off on the study remember the details as they are described in the published article. In the shredder test we met all MIT conditions. There are many methods to maintain the anonymity of the participants and not just one way as you presented. There is a difference between academic articles, in which every word and data is carefully chosen, and between a free conversation in front of an audience, in which I convey the main points and do not go into all the details, for the important purpose of making science accessible. Your attempt to "catch" me with one word or another and take it out of context only reflects a desire to slander and nothing else. Regarding the dentists, these things were said to me and Dr. Janet Schwartz in a face-to-face meeting, by senior company officials, as Dr. Schwartz also confirmed to you. I repeated the things that were said to me in media interviews because they seem important to me. The blog of my institute at Duke University accidentally mixed in an incorrect quote and I asked for it to be corrected. We recently launched together with Harbor Capital various indices that try to improve the way companies treat their employees and also create value for investors. I forwarded to you the thorough research conducted by one of the best research units in the world, of the investment bank JP Morgan, on the data on which we were based, a study that reached the same conclusions as us. Prof. Omar Moab holds opinions about many people, but to my understanding he is not an expert in the behavioral sciences and it is not clear why he was specifically invited to give an opinion on my research. Only once was an error discovered in one of my studies, out of over 150 academic articles. My research partners and I realized that something was wrong with the data, we noticed it and shared the data transparently. I took responsibility for the incident, but I still don't know how it happened. In my more than 30 years as a scientist, I participated in hundreds of studies, wrote seven books and had the great privilege of helping many organizations improve people's lives. I am proud of my work and contribution.
faitswulff 3y ago
Can anyone give a summary or a source for the “Dan Ariely saga” mentioned in the article?
- readthenotes1 3y ago
  Yet another case where Scientists failed to do Science.
- renewiltord 3y ago
  I have found that Bing AI usually does well with questions such as yours, if you'd like a tool that can help. I don't mean that in a LMGTFY way, just in a "here's a tool that can help".
  - zdragnar 3y ago
    What more ironic suggestion could someone hope for on the topic of irreproducible results and fabricated data?
    - renewiltord 3y ago
      It's a good meme, sir, but the thing does work.
- hdesh 3y ago
  https://datacolada.org/109 https://datacolada.org/109
- stainablesteel 3y ago
  this relates to the headline that went around saying "person who studies dishonesty found to be dishonest"
beebmam 3y ago
Can someone summarize what's going on here? I am having a hard time following this post.
- rossdavidh 3y ago
  Essentially, Dan Ariely is a well known, oft-interviewed and oft-cited researcher, who has done research on how the environment influences a person to cheat or not cheat. One of his most commonly cited is that having students recall the Ten Commandments before a test reduced cheating. It has not proven to be reproducible, and it now appears that it may have all been a fabrication, which is both disappointing and also deliciously ironic.
carlossouza 3y ago
A) Our human brains are hardwired to trust others[1]. We are not programmed to question, probe, or doubt. We want to believe. B) Moreover, our brains are also hardwired to storytelling[2][3]. When presented with a nice narrative, we feel rewarded. C) Finally, to quote Charlie Munger, "Show me the incentive and I will show you the outcome". People in academia focus on the # of papers published. Put (A), (B), and (C) together and we can explain news such as these. [1] https://www.psychologytoday.com/us/blog/the-athletes-way/201508/the-neuroscience-trust https://www.psychologytoday.com/us/blog/the-athletes-way/201... (hopefully this study is legit :)) [2] https://www.tadigital.com/insights/perspectives/art-storytelling-how-humans-are-wired-story https://www.tadigital.com/insights/perspectives/art-storytel... [3] https://cnlm.uci.edu/2018/12/04/story/ https://cnlm.uci.edu/2018/12/04/story/
- woodruffw 3y ago
  Using social psychology research to explain why people falsify social psychology research is more than a little ironic!
- glutamate 3y ago
  > Our human brains are hardwired to trust others How could neuroscience possibly prove (or even provide any evidence in favour of) this? Did they use identify the gene for trust and produce a knock-out human?
renewiltord 3y ago
Interesting. He claims she collected the data. She claims she did not and could not have. At the least, it appears that it makes sense to have data provenance recorded and signed. The Latin phrases don't all appear to relate to what's going on. Don't know what's up with that, especially the one from Horace's Odes.
- cafard 3y ago
  It looked to me as if she had something like 'fortune' to pop random tags into the heading.
gsatic 3y ago
> journal of marketing research What are ppl even getting so excited about? I mean this is a field which teaches kids to recite - "half the money I spend is wasted; the trouble is I don't know which half".
dang 3y ago
Related. Others? Noted study in psychology fails to replicate, crumbles with evidence of fraud - https://news.ycombinator.com/item?id=28264097 https://news.ycombinator.com/item?id=28264097 - Aug 2021 (102 comments) A Big Study About Honesty Turns Out to Be Based on Fake Data - https://news.ycombinator.com/item?id=28257860 https://news.ycombinator.com/item?id=28257860 - Aug 2021 (90 comments) Evidence of fraud in an influential field experiment about dishonesty - https://news.ycombinator.com/item?id=28210642 https://news.ycombinator.com/item?id=28210642 - Aug 2021 (51 comments) Related but different (I think?): Harvard ethics professor allegedly fabricated multiple studies - https://news.ycombinator.com/item?id=36665247 https://news.ycombinator.com/item?id=36665247 - July 2023 (214 comments) Harvard dishonesty expert accused of dishonesty - https://news.ycombinator.com/item?id=36424090 https://news.ycombinator.com/item?id=36424090 - June 2023 (201 comments)
scrum-treats 3y ago
How refreshing, to be able to finally tell someone more senior and corrupt to F off. Cheers to Aimee Drolet.
noelwelsh 3y ago
Any field will have cheaters. Contrary to many posters I happy to see this. It shows that science is self-correcting, as it should be. I'm only disappointed that it's taken so long for many of these allegations to surface.
- tnecniv 3y ago
  Science is self-correcting. It just might ruin a number of grad students lives along the way.
  - noelwelsh 3y ago
    For sure. I'm not saying the current system is perfect, but recognising flaws is the first step to improvement.
xkcd-sucks 3y ago
It is so incredibly easy to fabricate data that looks real and which has provenance going back to "raw data files" or whatever - Just generate random values with the statistical properties of other published data on similar topics. A little more effort if there's cryptographic signing of files (put fake samples into the machine instead, no big deal that's what you do to calibrate the machine in the first place) or hand written questionnaires (actually this could be tricky without an army of complicit grad students). So it's shocking when successful high profile researchers are this sloppy and manage to get away with it for so long.
- rossdavidh 3y ago
  In fact, it strongly suggests that fabricating data is so common, that it doesn't seem dangerous or risky to them, and they take no great amount of care while doing it. Familiarity breeds contempt. I'm pretty sure the researchers implicated are smart enough to figure out how to fabricate data well, if they wanted to; that they did not bother to do so, suggests a cultural rot.
  - tnecniv 3y ago
    Familiarity for the researcher breeds contempt. If you succeed once and don’t get caught, it’s easier to do it the second time. It doesn’t require a culture of dishonestly for bad actors to get comfortable breaking rules.
    - rossdavidh 3y ago
      You're correct that I stated that badly. I didn't mean cultural rot as in "everyone does it", I meant cultural rot as in there isn't much perceived chance of being caught, i.e. not much of a culture of double-checking each others' work, so they do it again and again (until they get sloppy). But you're correct that it doesn't mean most people are doing it.
- juujian 3y ago
  It's a real problem. The whole issue complex has turned me off of using regression analysis in my work completely. If there is data, I just show the data as good as possible. But as long as regression analysis is a useful tool to advance your career (as opposed to research) this will continue.
deleted 3y ago
[deleted]
bobosha 3y ago
I took Prof Ariely's courses while he taught at MIT - a fantastic professor and well-thought material. However these are serious charges and I wish we exert bit more caution before trashing someone.
- JohnMakin 3y ago
  This has been going on for multiple years now. No one really disputes the evidence that he fabricated data. It's pretty damning.
- jstarfish 3y ago
  The irony here is that everything that surfaced makes him the very person you'd want to be learning about the subject from. He's not being dishonest in being an expert on dishonesty-- he is clearly familiar with the topic if he's being accused of it himself. If true, the man walks the walk, so we'd benefit from listening to him talk. This is one of those "trust but verify" situations. If it can't be verified, do not trust. Ariely will not be the last case of this in our lifetimes. Much more effective than endless pontificating and theorizing by people who have no idea what the fuck they're doing but haven't publicly fucked up yet so we trust them in reverence of their credentials. The amount of institutionalized deception that has permeated the field of psychology since its inception should scare anybody away from ever taking it seriously.
- jononomo 3y ago
  Once you've fabricated data, then your name should be dirt in academia forever. End of story.
justindomke 3y ago
I've long been suspicious of this study from Ariely from 2003. It suggests that if you "prime" students by having them write down the last 2 digits of their student ID, then students with numbers like "93" will then be willing to pay 3.3x as much for a bottle of wine as students with numbers like "04". At first, some follow-up research seemed to replicate this, but then later work found no effect whatsoever. Very odd. https://justindomke.wordpress.com/2021/10/14/is-anchoring-a-reliable-cognitive-bias/#:~:text=Irrelevant https://justindomke.wordpress.com/2021/10/14/is-anchoring-a-...
huitzitziltzin 3y ago
Summary for those who are not deep into academia… Dan Ariely is a fraud. To be specific: he has been credibly accused of fabricating data in at least four separate studies (Ten Commandments, delta dental, Hartford car insurance, and the study which he claimed Rossi ran for him which is discussed at the link). There are likely more. More broadly: There was a “golden age” for this kind of “behavioral economics”/social psychology work in the early 2000’s up to probably 2012-2015. Researchers would generally publish “results” which were either “delightfully counterintuitive” (ex: prime subjects to think about Florida and they walk slowly) OR featured extremely cheap interventions which showed large effects on outcomes (ex: signing an honest pledge at the top or the bottom of the page made a big difference in how honest people actually were on their tasks). In retrospect, there are two problems (at least!) with this literature. 1. There was no theory backing up any of these predictions, so you could rationalize any statistically significant result with a compelling story. Publish it and you’re on your way to tenure. 2. Related to (1.), you didn’t even need to bother running actual experiments at all! Just fabricate the data. Make it fit your cute story and you’re done. There are many tenured people in business schools (generally in “organizational behavior” or “social psychology” style departments rather than Econ) who were tenured on the basis of papers published under (1.) or (2.). This is a big shame because there are serious researchers who do difficult work in behavioral and experimental economics which is _not at all_ like this BS. (Some are personal friends of mine, so this bothers me.). There was no incentive for about 15 years to do the hard work though. And given that tenure is for life, it’ll be a long long time before these people are flushed out (barring more investigations, which should be encouraged!). What should you do? - be extremely, extremely skeptical of any results in the “priming” literature (eg the “think of Florida” example above). - be moderately skeptical of results in the “nudges” literature. The retirement savings stuff makes sense and has replicated but the honesty nudge above is BS and has not. - don’t get really excited about a researcher who says “economics bad - people dumb! - I behavioralist!” More than a few of these people have been revealed to be total frauds. - don’t click or share this stuff when you see it on Reddit, HN or anywhere else. That kind of engagement “feeds the beast” - Deans love it and Deans assume you haven’t fabricated your data. - maybe if you are specifically a Duke or Duke Fuqua alum send them a note asking why they aren’t taking the Ariely misconduct more seriously??? - (edit) throw away your copies of any of Ariely’s books. They are BS.
einpoklum 3y ago
Is this new information? It looks like this was written last year, although I couldn't say for sure. Specifically > Unfortunately for Ariely, the Israeli press won’t let the issue die and has been digging into this issue of provenance rather deeply. Haven't noticed any stories about this issue, this year (2023), in the Israeli press.
gnicholas 3y ago
When I studied Econ 20 years ago, behavioral economics wasn’t offered. A decade later, I’d read a bunch of books on the topic, including by Ariel’s. I lamented that this field wasn’t offered when I was in college. Now another decade later, I’m glad I wasn’t in school during the period when it was offered, since it seems to be holding up so poorly. I hope that it can have a more rigorous rebirth, and in the next decade produce some results that are more robust (even if they give rise to fewer TED talks).
seizethecheese 3y ago
Academic fraud is massively costly for society as it polluted downstream knowledge. Worse, it justifies doubt in all knowledge-generating efforts. It’s a shame that the worst that can happen to these frauds is, typically, job loss. The incentive to cheat is way higher than the cost times the likelihood of getting caught. There should be criminal penalties at least as severe as other white collar crimes.
apienx 3y ago
The average "half-life of knowledge" in psychology is already a dismal 7 years (and has been measured as low as 3.3 years). And that's excluding outright fraud... https://en.m.wikipedia.org/wiki/Half-life_of_knowledge https://en.m.wikipedia.org/wiki/Half-life_of_knowledge P.S. Ironically, the title of the paper is "The dishonesty of honest people" (it's 15 years old by now).
- crazygringo 3y ago
  You seem to be misinterpreting the concept of half-life of knowledge. Lower values mean the field is advancing quickly. Rapid scientific advancement is a good thing for knowledge generally. It's merely annoying for people in the field because they have to do more work to keep up-to-date. :) It's not a measure of things being proven wrong, but more about newer better-predictive theories making older theories obsolete.
deleted 3y ago
[deleted]
cobaltoxide 3y ago
Has _any_ of his work been successfully replicated?
neilv 3y ago
At least some of these allegations have been out there for awhile. Why haven't the connected universities and journals answered them definitively? I don't like to see trials by Twitter. I also don't like to see institutions of academia shirking their responsibilities. (Full disclosure: I met with Ariely a long time ago, and he seemed decent, but I know almost nothing about that field.)
salynchnew 3y ago
I think there is an insidious tendency to want to post-hoc rationalize things that are just random noise in the kinds of surveys that Ariely et al were doing. I can imagine over time that it must be tempting for these kinds of people to just start faking stuff. One of my undergrad psych research projects was focus on testing some of the Kahneman & Tversky anchoring/adjustment stuff. We were able to replicate some things related to the theory itself based around economic estimates, but also had some "control" questions in our surveys (one example: "estimate the average weight of a human brainL). Our experience was that some populations just sucked worse at the kinds of estimation tasks we were using. We had no good theory for gender-based differences in estimating the weight of a brain, and in retrospect it was probably just people confusing ounces with pounds as a half-remembered number for weight... but no one gets published by admitting that their experiments were probably just poorly designed.

149 points by kblissett 3y ago

I had such high regard for academia while I was in school. Glad I didn’t end up going down that path, it seems like every day recently there have been stories like this posted to Hacker News. Behavioral economics especially seems hard hit.

pocketsand 3y ago
Seems like behavioral economics really just receives splash damage from what is the massive fraud at the heart of experimental social psychology. At this point I basically assume any catchy finding from the field is at best HARKed or p-hacked and at worst completely fraudulent.
- chongli 3y ago
  Why aren't journals requiring pre-registration? It's an absolute catastrophe and no one seems to care!
  - pocketsand 3y ago
    Pre-registration protects against HARKing and p-hacking, but it unfortunately doesn't protect at all against outright fraud, which we appear to see from the likes of Ariely and Gino.
    - chongli 3y ago
      Right. Outright fraud needs to be dealt with much more severely. Total disgrace, loss of tenure, and potential prison time.
- FreshStart 3y ago
  The field migrated to were the data is. Some communication dark departments within Meta and Google, sponsored by three letters. You can even do ethical virtual experiments with the data of 3 billion people. Just specify the situation, the person and what's to happen.. Voila. All things happen regularly at some point..
  - esafak 3y ago
    > The field migrated to were the data is. Some communication dark departments within Meta and Google, sponsored by three letters. You really need to clarify this.
    - cool_dude85 3y ago
      I think it's pretty clear what the post is saying. It doesn't need to be clarified, it needs to be supported, though the nature of the claim is such that it would be tough to provide direct evidence.
    - meepmorp 3y ago
      I’m going with: > The government is trying to control your mind through the internet
    - scrps 3y ago
      Same techniques (largely derived from psychology, statistics, sociology, etc) used in social media marketing, intent tracking, etc has uses in national security. Same thing that might let meta know you might be interested in batman pajamas can be used to sort out if you are being radicalized to action by some group. Interesting example is fractalrabbit. Group of people who both have the theoretical training and practical experience is probably pretty small so private/public crossover isn't unreasonable.
passwordoops 3y ago
Me too, then following my Ph.D at a relatively small school went off to a post-doc at a major research university where the grants are millions or you're worthless... That environment essentially forces you to commit fraud so I got out pretty quick
cbsmith 3y ago
Consider how many studies are being produced each day. That there is a story every day like this suggests that academia is making an effort to identify bad actors, but the frequency of discovery is actually quite low.
- tnecniv 3y ago
  Well we don’t know how many people falsify data because, if it goes undiscovered, we don’t know it happened. It could easily be that the rate at which data is falsified is quite low and we are good at catching it, or it is high and we are bad at catching it. I would think instances of fabrication are quite low — it’s high-risk, medium reward. Even if it goes undiscovered, academia is a marketing game, and you can’t control what other people feel is impactful or not. It’s very easy to do bad science without falsifying data, which is probably the deadliest sin in academia.
  - cbsmith 3y ago
    > Well we don’t know how many people falsify data because, if it goes undiscovered, we don’t know it happened. It could easily be that the rate at which data is falsified is quite low and we are good at catching it, or it is high and we are bad at catching it. Yes, which is why I said the frequency of discovery is low. Seeing a story in the paper about fraud every day doesn't really tell us anything about the rate of fraudulence, which I was hoping would become transparent with my point.
lolinder 3y ago
To be fair, industry has a similar problem: every day there's a new company that gets in trouble for abusing customer data, losing customer data, abusing customer trust, making it hard to cancel a subscription, etc, etc. People in general tend to be dishonest if they think they can get away with it. I'm still glad I didn't go into academia for many other reasons, but I don't think I dodged an ethics bullet.
- notahacker 3y ago
  yeah, this Reputation probably matters more for academia (people go into it because they think the research is important, not for the money, and they think they're standing on the shoulder of giants and not amiable bullshitters taking the credit for findings actually created by their overworked research assistants kludging the data), but if you think academia is full of lies and half truths relative to other professions, you haven't encountered many startups!
usednet 3y ago
Could somebody provide me some links to other recent scandals in academia/research? Very interested.
- pocketsand 3y ago
  Check out the datacolada web site. https://datacolada.org/ https://datacolada.org/
- tnecniv 3y ago
  I forget the guys name (he’s German iirc), but a very impactful one was at Bell Labs in the 00’s. STEM PhD programs now all have a research ethics component because of him. While you can’t teach someone to be ethical, it at least hammers home how many people you are defrauding by falsifying data, which includes everyone from your funding organizations to the lowly PhD students whose advisors are making them iterate on your study and don’t understand why, after three years, it still isn’t working.
- dmvdoug 3y ago
  Retraction Watch, maybe? https://retractionwatch.com/ https://retractionwatch.com/
tnecniv 3y ago
While fields are small, academia is a big place. Most academics won’t falsify data — from a pragmatic perspective, that’s their career on the line, and from an emotional perspective, their academic work is tied to their self-image. It’s also arguably a lot harder to do in the hard sciences but there have still been examples. In the hard sciences, I feel like the solution is some method to increase how much people at different labs talk to one another (in addition to including all raw data and processing code from experiments). If an impactful article comes out, labs will try to build on it. Not every lab will have the equipment but those in the sub area might. If you’re trying to iterate on a study, and you can’t reproduce the base result before adding your modifications, you might just think you’re dumb or doing something dumb. There’s likely nobody outside maybe your research group who can help you. However, if it was easier to talk to other researchers in the field (like a web forum), you might find out that nobody has been able to get the original experiment working! In the social sciences, I’m not sure what you do besides running automated checks on submitted data. Humans are bad at generating things that look random (for example, we use more 3s and 7s than nature), and you can probably flag warnings. Either that or include paper trails like a lab notebook in chemistry — which is considered a legal document and thus must be kept in pen. That way you can establish a chain of custody for the data from collection to publication.
- pevey 3y ago
  > Most academics won’t falsify data — from a pragmatic perspective, that’s their career on the line In my experience, this is absolutely not true. I mostly only have first-hand observations in the economics, finance, and marketing disciplines. They left me EXTREMELY cynical about all academics. I have seen: 1) A well-known professor at an Ivy-league B-school openly talking to his class about how the NAR funded his latest study on the real estate market, which (of course) found that the market was not overvalued. This was 2007. The co-author on the paper was the dean of the B-school. He is no longer the dean, but only because he seems to have moved on to more lucrative consulting roles. 2) An older professor, who was one of the first to be in the group labeled "behavioral economics," when asked why he wasn't writing a book focused on more mainstream readers like some other such big names were at the time, STRONGLY hinting if you filled in the blanks although not outright saying that their ethics were questionable, they were all forming hypotheses that were useful for for consulting dollars and dredging up data to support it, and he was too old school (and too old) to be interested in playing that game. 3) A full cohort of marketing grad students (6-7) talking openly about how the system is messed up, and to get a job, you absolutely have to have a good dissertation with a strong results, but to avoid p-hacking allegations, you have to shoot in the dark. So you're left in this situation where you commit financial and career suicide if you DON'T make up data to support your thesis. It is not rare. It is an open secret. No one likes it. They hate it. But they also feel like they are the stupid idiot if they try to fight against it alone. It is how the system works. Only a true saint would accept data that actually does NOT support their hypothesis because it makes it unlikely they will get a job in a very competitive market. Some people do report non-findings, but they are rare. IMO, academic departments should specifically seek out graduates who were willing to report that and stuck by it. But that is not the way the world works. As another poster pointed out, we all like stories, and no significance means no good story to tell.
rayiner 3y ago
Charlatans took advantage of the reputation of “science” in the 20th century. Thankfully, public perception is already adjusting.
scrum-treats 3y ago
Experimentation in moral psychology without robust, independently verified design is an incredibly slippery slope. This is why certain programs require interdisciplinary approaches (e.g., philosophy, psychology, linguistics, math, physics, neuroscience, biology, history). Otherwise people just make stuff up. And if there's no one to run independent replication studies, and no journals require replication studies prior to publication, then we get... this. Just wait for people to dive into the false equivalence of scales and measures used in political research. It's very much "make it up and see if it sticks." The goal appears to be Psychology Today, not scientific rigor. I wrote about this in graduate school, as part of my hefty statistics coursework, it bothered me so much.

passwordoops 3y ago

Wow to say the least. I have Predictably Irrational on my shelf and have highly recommend it in the past. Sad to see the whole field failing the irreproducibility task... I'm very happy to see this type of scrutiny come to headline research but I really wish it begins at the beginning, with funding agencies and journals. And at the very least the moment a university sends out a breathless PR, the media all scrutinize the data provenance... A boy can wish

appleflaxen 3y ago
"Failing irreproducibility" is a double negative that I don't think you intended.
- neilv 3y ago
  Irregardless... :)

whyage 3y ago

For balance, here's Dan Arieli's response (Google-translated from the Hebrew source): It's ironic that you try to attribute unprofessionalism to me using unprofessional tools. The experiments conducted by Prof. Siniver are not reproductions of experiments I conducted, but rather experiments that are clearly different in many essential differences. A claim for refutation through a completely different experiment is a provocation without a professional basis. Researchers are required to keep data on studies for 5 years. I do not have data from studies that were carried out 15 years ago or more, and I have no way to find them either. This is not to criticize my work, and any attempt to do so is manipulation of the viewers. I don't do the research alone. Throughout 30 years of research I had 248 scientific writing partners, among them the most respected researchers in the field. I led 2 labs at MIT and another lab at Duke where hundreds of interns, students and research assistants worked alongside me. You understand that if the claim is that I fabricated studies, you accuse a huge group of academics from dozens of universities in the world, when the only "evidence" for this is that I failed to prove "that I don't have a sister", that is, to provide documentation that I no longer have. The experiment in which Prof. Amy Droley assisted took place over 15 years ago. I don't know why she remembers the things as she described to you. At first she didn't remember taking part in it at all, and only later did she remember. I and the other two professors who signed off on the study remember the details as they are described in the published article. In the shredder test we met all MIT conditions. There are many methods to maintain the anonymity of the participants and not just one way as you presented. There is a difference between academic articles, in which every word and data is carefully chosen, and between a free conversation in front of an audience, in which I convey the main points and do not go into all the details, for the important purpose of making science accessible. Your attempt to "catch" me with one word or another and take it out of context only reflects a desire to slander and nothing else. Regarding the dentists, these things were said to me and Dr. Janet Schwartz in a face-to-face meeting, by senior company officials, as Dr. Schwartz also confirmed to you. I repeated the things that were said to me in media interviews because they seem important to me. The blog of my institute at Duke University accidentally mixed in an incorrect quote and I asked for it to be corrected. We recently launched together with Harbor Capital various indices that try to improve the way companies treat their employees and also create value for investors. I forwarded to you the thorough research conducted by one of the best research units in the world, of the investment bank JP Morgan, on the data on which we were based, a study that reached the same conclusions as us. Prof. Omar Moab holds opinions about many people, but to my understanding he is not an expert in the behavioral sciences and it is not clear why he was specifically invited to give an opinion on my research. Only once was an error discovered in one of my studies, out of over 150 academic articles. My research partners and I realized that something was wrong with the data, we noticed it and shared the data transparently. I took responsibility for the incident, but I still don't know how it happened. In my more than 30 years as a scientist, I participated in hundreds of studies, wrote seven books and had the great privilege of helping many organizations improve people's lives. I am proud of my work and contribution.

faitswulff 3y ago

Can anyone give a summary or a source for the “Dan Ariely saga” mentioned in the article?

readthenotes1 3y ago
Yet another case where Scientists failed to do Science.
renewiltord 3y ago
I have found that Bing AI usually does well with questions such as yours, if you'd like a tool that can help. I don't mean that in a LMGTFY way, just in a "here's a tool that can help".
- zdragnar 3y ago
  What more ironic suggestion could someone hope for on the topic of irreproducible results and fabricated data?
  - renewiltord 3y ago
    It's a good meme, sir, but the thing does work.
hdesh 3y ago
https://datacolada.org/109 https://datacolada.org/109
stainablesteel 3y ago
this relates to the headline that went around saying "person who studies dishonesty found to be dishonest"

beebmam 3y ago

Can someone summarize what's going on here? I am having a hard time following this post.

rossdavidh 3y ago
Essentially, Dan Ariely is a well known, oft-interviewed and oft-cited researcher, who has done research on how the environment influences a person to cheat or not cheat. One of his most commonly cited is that having students recall the Ten Commandments before a test reduced cheating. It has not proven to be reproducible, and it now appears that it may have all been a fabrication, which is both disappointing and also deliciously ironic.

carlossouza 3y ago

A) Our human brains are hardwired to trust others[1]. We are not programmed to question, probe, or doubt. We want to believe. B) Moreover, our brains are also hardwired to storytelling[2][3]. When presented with a nice narrative, we feel rewarded. C) Finally, to quote Charlie Munger, "Show me the incentive and I will show you the outcome". People in academia focus on the # of papers published. Put (A), (B), and (C) together and we can explain news such as these. [1] https://www.psychologytoday.com/us/blog/the-athletes-way/201508/the-neuroscience-trust https://www.psychologytoday.com/us/blog/the-athletes-way/201... (hopefully this study is legit :)) [2] https://www.tadigital.com/insights/perspectives/art-storytelling-how-humans-are-wired-story https://www.tadigital.com/insights/perspectives/art-storytel... [3] https://cnlm.uci.edu/2018/12/04/story/ https://cnlm.uci.edu/2018/12/04/story/

woodruffw 3y ago
Using social psychology research to explain why people falsify social psychology research is more than a little ironic!
glutamate 3y ago
> Our human brains are hardwired to trust others How could neuroscience possibly prove (or even provide any evidence in favour of) this? Did they use identify the gene for trust and produce a knock-out human?

renewiltord 3y ago

Interesting. He claims she collected the data. She claims she did not and could not have. At the least, it appears that it makes sense to have data provenance recorded and signed. The Latin phrases don't all appear to relate to what's going on. Don't know what's up with that, especially the one from Horace's Odes.

cafard 3y ago
It looked to me as if she had something like 'fortune' to pop random tags into the heading.

gsatic 3y ago

> journal of marketing research What are ppl even getting so excited about? I mean this is a field which teaches kids to recite - "half the money I spend is wasted; the trouble is I don't know which half".

dang 3y ago

Related. Others? Noted study in psychology fails to replicate, crumbles with evidence of fraud - https://news.ycombinator.com/item?id=28264097 https://news.ycombinator.com/item?id=28264097 - Aug 2021 (102 comments) A Big Study About Honesty Turns Out to Be Based on Fake Data - https://news.ycombinator.com/item?id=28257860 https://news.ycombinator.com/item?id=28257860 - Aug 2021 (90 comments) Evidence of fraud in an influential field experiment about dishonesty - https://news.ycombinator.com/item?id=28210642 https://news.ycombinator.com/item?id=28210642 - Aug 2021 (51 comments) Related but different (I think?): Harvard ethics professor allegedly fabricated multiple studies - https://news.ycombinator.com/item?id=36665247 https://news.ycombinator.com/item?id=36665247 - July 2023 (214 comments) Harvard dishonesty expert accused of dishonesty - https://news.ycombinator.com/item?id=36424090 https://news.ycombinator.com/item?id=36424090 - June 2023 (201 comments)

scrum-treats 3y ago

How refreshing, to be able to finally tell someone more senior and corrupt to F off. Cheers to Aimee Drolet.

noelwelsh 3y ago

Any field will have cheaters. Contrary to many posters I happy to see this. It shows that science is self-correcting, as it should be. I'm only disappointed that it's taken so long for many of these allegations to surface.

tnecniv 3y ago
Science is self-correcting. It just might ruin a number of grad students lives along the way.
- noelwelsh 3y ago
  For sure. I'm not saying the current system is perfect, but recognising flaws is the first step to improvement.

xkcd-sucks 3y ago

It is so incredibly easy to fabricate data that looks real and which has provenance going back to "raw data files" or whatever - Just generate random values with the statistical properties of other published data on similar topics. A little more effort if there's cryptographic signing of files (put fake samples into the machine instead, no big deal that's what you do to calibrate the machine in the first place) or hand written questionnaires (actually this could be tricky without an army of complicit grad students). So it's shocking when successful high profile researchers are this sloppy and manage to get away with it for so long.

rossdavidh 3y ago
In fact, it strongly suggests that fabricating data is so common, that it doesn't seem dangerous or risky to them, and they take no great amount of care while doing it. Familiarity breeds contempt. I'm pretty sure the researchers implicated are smart enough to figure out how to fabricate data well, if they wanted to; that they did not bother to do so, suggests a cultural rot.
- tnecniv 3y ago
  Familiarity for the researcher breeds contempt. If you succeed once and don’t get caught, it’s easier to do it the second time. It doesn’t require a culture of dishonestly for bad actors to get comfortable breaking rules.
  - rossdavidh 3y ago
    You're correct that I stated that badly. I didn't mean cultural rot as in "everyone does it", I meant cultural rot as in there isn't much perceived chance of being caught, i.e. not much of a culture of double-checking each others' work, so they do it again and again (until they get sloppy). But you're correct that it doesn't mean most people are doing it.
juujian 3y ago
It's a real problem. The whole issue complex has turned me off of using regression analysis in my work completely. If there is data, I just show the data as good as possible. But as long as regression analysis is a useful tool to advance your career (as opposed to research) this will continue.

deleted 3y ago

[deleted]

bobosha 3y ago

I took Prof Ariely's courses while he taught at MIT - a fantastic professor and well-thought material. However these are serious charges and I wish we exert bit more caution before trashing someone.

JohnMakin 3y ago
This has been going on for multiple years now. No one really disputes the evidence that he fabricated data. It's pretty damning.
jstarfish 3y ago
The irony here is that everything that surfaced makes him the very person you'd want to be learning about the subject from. He's not being dishonest in being an expert on dishonesty-- he is clearly familiar with the topic if he's being accused of it himself. If true, the man walks the walk, so we'd benefit from listening to him talk. This is one of those "trust but verify" situations. If it can't be verified, do not trust. Ariely will not be the last case of this in our lifetimes. Much more effective than endless pontificating and theorizing by people who have no idea what the fuck they're doing but haven't publicly fucked up yet so we trust them in reverence of their credentials. The amount of institutionalized deception that has permeated the field of psychology since its inception should scare anybody away from ever taking it seriously.
jononomo 3y ago
Once you've fabricated data, then your name should be dirt in academia forever. End of story.

justindomke 3y ago

I've long been suspicious of this study from Ariely from 2003. It suggests that if you "prime" students by having them write down the last 2 digits of their student ID, then students with numbers like "93" will then be willing to pay 3.3x as much for a bottle of wine as students with numbers like "04". At first, some follow-up research seemed to replicate this, but then later work found no effect whatsoever. Very odd. https://justindomke.wordpress.com/2021/10/14/is-anchoring-a-reliable-cognitive-bias/#:~:text=Irrelevant https://justindomke.wordpress.com/2021/10/14/is-anchoring-a-...

huitzitziltzin 3y ago

Summary for those who are not deep into academia… Dan Ariely is a fraud. To be specific: he has been credibly accused of fabricating data in at least four separate studies (Ten Commandments, delta dental, Hartford car insurance, and the study which he claimed Rossi ran for him which is discussed at the link). There are likely more. More broadly: There was a “golden age” for this kind of “behavioral economics”/social psychology work in the early 2000’s up to probably 2012-2015. Researchers would generally publish “results” which were either “delightfully counterintuitive” (ex: prime subjects to think about Florida and they walk slowly) OR featured extremely cheap interventions which showed large effects on outcomes (ex: signing an honest pledge at the top or the bottom of the page made a big difference in how honest people actually were on their tasks). In retrospect, there are two problems (at least!) with this literature. 1. There was no theory backing up any of these predictions, so you could rationalize any statistically significant result with a compelling story. Publish it and you’re on your way to tenure. 2. Related to (1.), you didn’t even need to bother running actual experiments at all! Just fabricate the data. Make it fit your cute story and you’re done. There are many tenured people in business schools (generally in “organizational behavior” or “social psychology” style departments rather than Econ) who were tenured on the basis of papers published under (1.) or (2.). This is a big shame because there are serious researchers who do difficult work in behavioral and experimental economics which is _not at all_ like this BS. (Some are personal friends of mine, so this bothers me.). There was no incentive for about 15 years to do the hard work though. And given that tenure is for life, it’ll be a long long time before these people are flushed out (barring more investigations, which should be encouraged!). What should you do? - be extremely, extremely skeptical of any results in the “priming” literature (eg the “think of Florida” example above). - be moderately skeptical of results in the “nudges” literature. The retirement savings stuff makes sense and has replicated but the honesty nudge above is BS and has not. - don’t get really excited about a researcher who says “economics bad - people dumb! - I behavioralist!” More than a few of these people have been revealed to be total frauds. - don’t click or share this stuff when you see it on Reddit, HN or anywhere else. That kind of engagement “feeds the beast” - Deans love it and Deans assume you haven’t fabricated your data. - maybe if you are specifically a Duke or Duke Fuqua alum send them a note asking why they aren’t taking the Ariely misconduct more seriously??? - (edit) throw away your copies of any of Ariely’s books. They are BS.

einpoklum 3y ago

Is this new information? It looks like this was written last year, although I couldn't say for sure. Specifically > Unfortunately for Ariely, the Israeli press won’t let the issue die and has been digging into this issue of provenance rather deeply. Haven't noticed any stories about this issue, this year (2023), in the Israeli press.

gnicholas 3y ago

When I studied Econ 20 years ago, behavioral economics wasn’t offered. A decade later, I’d read a bunch of books on the topic, including by Ariel’s. I lamented that this field wasn’t offered when I was in college. Now another decade later, I’m glad I wasn’t in school during the period when it was offered, since it seems to be holding up so poorly. I hope that it can have a more rigorous rebirth, and in the next decade produce some results that are more robust (even if they give rise to fewer TED talks).

seizethecheese 3y ago

Academic fraud is massively costly for society as it polluted downstream knowledge. Worse, it justifies doubt in all knowledge-generating efforts. It’s a shame that the worst that can happen to these frauds is, typically, job loss. The incentive to cheat is way higher than the cost times the likelihood of getting caught. There should be criminal penalties at least as severe as other white collar crimes.

apienx 3y ago

The average "half-life of knowledge" in psychology is already a dismal 7 years (and has been measured as low as 3.3 years). And that's excluding outright fraud... https://en.m.wikipedia.org/wiki/Half-life_of_knowledge https://en.m.wikipedia.org/wiki/Half-life_of_knowledge P.S. Ironically, the title of the paper is "The dishonesty of honest people" (it's 15 years old by now).

crazygringo 3y ago
You seem to be misinterpreting the concept of half-life of knowledge. Lower values mean the field is advancing quickly. Rapid scientific advancement is a good thing for knowledge generally. It's merely annoying for people in the field because they have to do more work to keep up-to-date. :) It's not a measure of things being proven wrong, but more about newer better-predictive theories making older theories obsolete.

deleted 3y ago

[deleted]

cobaltoxide 3y ago

Has _any_ of his work been successfully replicated?

neilv 3y ago

At least some of these allegations have been out there for awhile. Why haven't the connected universities and journals answered them definitively? I don't like to see trials by Twitter. I also don't like to see institutions of academia shirking their responsibilities. (Full disclosure: I met with Ariely a long time ago, and he seemed decent, but I know almost nothing about that field.)

salynchnew 3y ago

I think there is an insidious tendency to want to post-hoc rationalize things that are just random noise in the kinds of surveys that Ariely et al were doing. I can imagine over time that it must be tempting for these kinds of people to just start faking stuff. One of my undergrad psych research projects was focus on testing some of the Kahneman & Tversky anchoring/adjustment stuff. We were able to replicate some things related to the theory itself based around economic estimates, but also had some "control" questions in our surveys (one example: "estimate the average weight of a human brainL). Our experience was that some populations just sucked worse at the kinds of estimation tasks we were using. We had no good theory for gender-based differences in estimating the weight of a brain, and in retrospect it was probably just people confusing ounces with pounds as a half-remembered number for weight... but no one gets published by admitting that their experiments were probably just poorly designed.