9 ms·

Noted study in psychology fails to replicate, crumbles with evidence of fraud

205 points by luu 5y ago

auganov 5y ago
If top scientists are willing to outright manufacture data, how many would be willing to intentionally follow unsound logic? As in dishonestly claiming some (legitimate) data implies something that it doesn't or intentionally avoiding alternative explanations. You'd have to assume there's even more of that and all other kinds of soft fraud going on. Ideally planning data production, execution of data production and analysis would all be done by different people and published separately. At the very least you'd want total transparency at each stage.
- throwthere 5y ago
  These are top social scientists and, unfortunately, their field is a total dumpster fire at the moment. See https://www.nature.com/articles/s41562-018-0399-z https://www.nature.com/articles/s41562-018-0399-z
  - auganov 5y ago
    If you're implying the replication crisis is limited to social sciences, that's too narrow. You'd expect to see these problems in any field that primarily relies on hard to gather data that is far removed from the underlying phenomenon. Many sciences that aren't strictly "soft" have "soft" fields. Say, Biology->drug research. Also not to argue semantics but when you hear the word "scientist" in common parlance it almost exclusively refers to people doing that kind of research. The "sciences" that don't have these problems tend to not be too big on using the word "science".
    - myWindoonn 5y ago
      Cosmology is the first field which comes to mind; it fits your criteria, but is far harder (as a science) than psychology and far more reliable. Psychologists can't define "mind" and change their entire profession every few decades, but "sun" has not changed much as a phenomenon for a long time. The reason why the social sciences (social psychology, criminology, economics, advertising/marketing) are hit hardest by the replication crisis is because they were not actually sciences! They are ways for our society to police its members' behavior (relationships, crime, money, attention) and they pretend to be science in order to gain prestige. There's also, finally, the matter of scale and relevance. When a chemist reports that a rare chemical reaction doesn't perform in the lab as expected, then this is a failed replication; however, chemistry labs regularly replicate entire textbooks' worth of material for entire classes of undergraduates. Meanwhile, some of the most important claims in psychology, like priming, are failing to replicate.
zhdc1 5y ago
I'm pretty sure this was discussed in an earlier HN post. However, I want to point out that "not replicating" doesn't necessarily equate to bad science (or, more accurately, bad practice on the part of the principal investigators irt accepted best practices in their field). There are a bunch of studies in the social sciences that failed to replicate simply because a p value was slightly below a threshold target, even though the coefficient direction/magnitude &c... agreed with the earlier study. That said, I'm not defending - at all - the paper discussed in the article.
- qnsi 5y ago
  There is a bigger accusation of fraud, not only failure to replicate Edit: Also [0] was on HN 5 days ago (discussion here: [1]). This is another blogpost that discusses [0] [0]https://datacolada.org/98 https://datacolada.org/98 [1]https://news.ycombinator.com/item?id=28210642 https://news.ycombinator.com/item?id=28210642
  - the-dude 5y ago
    And Buzzfeed : https://news.ycombinator.com/item?id=28257860 https://news.ycombinator.com/item?id=28257860 ( Yesterday, 86 comments )
- wanderingmind 5y ago
  Failure to replicate is the definition of bad science. Science is based on objective, repeatable experiments. If the experiments can't even be reproduced how can they be hypothesized and validated?
  - parliowl 5y ago
    This is so not true on many levels. Science is not about being right every time. Positive experimental findings can turn out to be wrong even if the study was excellent. This is called a type 1 error, and indeed they are guaranteed to happen with 5% probability assuming every study is conducted correctly and people don’t cheat. Also, I don’t think you understand that there is a difference between “replicate” and “reproduce” as those terms are typically used in social and biological science. Reproduction refers to the methods and is simply conducting the experiment again, regardless of the outcome. An experiment should be reproducible, of course—if it isn’t, the paper was poorly written (the experiment might still be ok though). Replication refers to the results: do we get the same answers as the original experiment? Bad science is one way an experiment might not replicate but other reasons are 1. Just a regular type 1 error in the original study, 2. A type 2 error in the new one, 3. an overlooked methodological difference or mistake somewhere. No. 3 doesn’t mean bad science, it could be just normal scientists making normal mistakes, as they sometimes do. Also, you don’t “hypothesize” an experiment. You come up with a hypothesis then test it with an experiment. (Sometimes scientists HARK but that’s a different discussion …) In this case, it sounds like there truly was some cheating going on of the worst kind (data fabrication).
aaron695 5y ago
The prediction market stuff it mentions is really interesting. https://polymarket.com/market/will-pnas-retract-dan-arielys-2012-paper-on-dishonesty-by-october-1-2021 https://polymarket.com/market/will-pnas-retract-dan-arielys-... There is also money to not have it retracted. But it is time limited. You just have to ensure it's not retracted before October. It's interesting to think where it would end up. A system that finally fixes incorrect papers but a battle ground over when.
willis936 5y ago
In private I have heard of science done that is suspected to be fraudulent. The results are so intriguing and promising as to send multiple grad students across multiple groups barreling into replicating and expanding the results. What happened? Years of time wasted with no publications. PIs and journals aren't interested in publishing negative results. The person who lied in their research is reaping career benefits of such a valuable contribution to research. Academia takes something as beautiful as the closed loop of science and opens it up only to benefit charlatans and punish the best and brightest.
- gus_massa 5y ago
  > The results are so intriguing and promising as to send multiple grad students across multiple groups barreling into replicating and expanding the results. One important task of the advisor is to filter bad projects. Does the paper pass the smell test? Are the authors reliable or they had published "dubious" results before? [I was recently discussing a paper in a virtual meeting. My conclusion is that Zoom does not have enough negative reactions.] Also, before assigning an experiment to a graduate student, try to have something like a plan: 1) This will surely work and is interesting enough for the dissertation and to be publish. 2) This part is dubious, because it's related to a too resent research paper or there may be some hidden technical problems that are difficult to see. 2') An underspecified alternative to 2, just in case. 3) Extra task for bonus points and perhaps a Nobel price.
- newsclues 5y ago
  Publishing negative results is important to solving this issue.
  - atoav 5y ago
    How about instead of applying to journals with a finished study, people would apply with a proposal of a study and then publish the results independent of how spectacular/boring they turn out? Journals would then have to select studies based on their design and based on the question and scientists could just do science.
knuthsat 5y ago
You don't even need the graphs, miles driven per year is a sum. Sums converge to gaussians yet here the deviation was half of mean which means they used a uniform variable generator for miles.
- throwthere 5y ago
  Interesting point. So the original paper (2012) didn't give a table or anything, but said: "Customers who signed at the beginning on average revealed higher use (M = 26,098.4, SD = 12,253.4) than those who signed at the end [M = 23,670.6, SD = 12,621.4; F(1, 13,485) = 128.63, P < 0.001]." Was M = 26,098.4, SD = 12,253.4 enough to infer a uniform distribution?
  - ImaCake 5y ago
    > Was M = 26,098.4, SD = 12,253.4 enough to infer a uniform distribution? No unfortunately it is not. It's perfectly plausible to have a normal distribution with mean = 26k and sd = 12k. Although those numbers do look kinda weird, but nothing you could verify. I am not sure what the F(*) means here, maybe an F statistic? But that seems wrong if you are comparing two normally distributed samples, you might expect a T statistic. To verify the distribution you would need a histogram or you could get fancy with a qqplot. You could also try a statistical test for a normal distribution but these fail on large sample sizes so the visual plots are your best bet.
- deleted 5y ago
  [deleted]
- voldacar 5y ago
  Taleb pointed this out on twitter iirc
- thaumasiotes 5y ago
  > yet here the deviation was half of mean which means they used a uniform variable What am I missing? The mean of a normal distribution can be any value. A normal with mean 2 and deviation 1 has deviation equal to half of the mean. Is this related to the fact that miles driven can't be negative (unlike a sample from a genuine normal)?
elevenoh 5y ago
I love polymarket. Prediction markets on topics like this are about the only way to discern truth from afar.
- ImaCake 5y ago
  Wouldn't polymarket be subject to incomplete information? Thus leading to bias. Polymarket would just reflect the market sentiment given a fairly robust analysis of the available information, admittedly a pretty useful idea. The other issue is that a very niche bet/topic might only have a small sample of analysts placing bets. This would mean you are subject to increased risk of bias from a single bad analyst. Or simply large variance due to small sample size.
qnsi 5y ago
I have several books of Ariely. I enjoy reading psychology, but I stopped reading pop psychology books and go for academic books or textbooks or individual articles. The amount of times I read about spectacular counter-intuitive study that "changes everything" and later find out it was fraudulent or failed to replicate makes me really doubt all of my knowledge I have from reading about psychology. Reproducibility crisis is real and it sucks for all the people that read individual books and don't know about reproducibility. I feel people that write those books should also put themself into higher standards and stop writing narrative books based on individual nonreplicated studies. You can read about reproducibility crisis here: https://www.theatlantic.com/science/archive/2018/11/psychologys-replication-crisis-real/576223/ https://www.theatlantic.com/science/archive/2018/11/psycholo...
- dalbasal 5y ago
  I think you need to draw a line between "not replicable" and fraudulent. This is about the latter. This is the kind of corruption that spreads. It's a very bad thing. It is related to the former problem of "not replicable," which is about application of the "scientific method." On this, my worldview has evolved a lot over the last decade. I think we've been applying overly Popperian methods, and formalisations of the scientific method where they don't work well. Say you are studying the effects of a breakfast program, 2 hr recess or push-up regiment on high schoolers. The study is well constructed, faithfully conducted, etc. You find interesting, potentially actionable ways of reducing obesity, improving psych health or something. Do/Should we even expect such a study to replicate in 1990s Tokyo, 2000s Miami, School A, School B, etc. Does that mean this study should not be conducted? Does it mean we can't generate knowledge and understanding of things this way? Is there any daylight between "anecdote" and a result that qualifies for publication in Physics? A lot of articles published in many fields feel like they are being hammered into a Popperian mold they don't fit into. Everyone knows it, but no one is willing to say "this method doesn't work here. we're actually doing a different thing." Instead, they sort of eye roll the article into the frame and let readers unscramble it for themselves.
  - rwbhn 5y ago
    If the results are not replicable, how can they be actionable?
mycodesucks 5y ago
My bet would be that a surprisingly large number of psychology studies are completely irreproducible. I'm even skeptical of a lot of fMRI experiments. Meta-analyses certainly can give a bit better insight, but have their own problems.
ghyiihdruh 5y ago
What's the purpose of this? It will just give ammo to anti-science Trump like people. It's harmful. This kind of stuff should be handled as quietly as possible, and definitely not in public. Science is already under attack, we need not undermine it from the inside too.
- technothrasher 5y ago
  Really? Covering up a messy truth because you perceive it as not politically expedient strikes me as the exact opposite of science. That road leads to worse knowledge, not better.
- AussieWog93 5y ago
  Anti-science people tend to be conspiracy-minded and believe that the establishment is a homogeneous group that is trying to mislead the public. Creating an actual conspiracy to mislead the public is not a good way to fight this perception.
- lyaa 5y ago
  I disagree. Constant re-evaluation of evidence and improvement of working theories are essential to science. Science needs transparency and honest critique. Acknowledging falsified results is not an attack on science, it is how science works. It shows that science is not a mysterious cult of knowledge conspiring to hide information and deceive others, but rather the collective effort of imperfect humans who want to understand the universe and make things better for us all.
- avnigo 5y ago
  What you just said goes against everything science stands for, otherwise it just becomes a belief system, and you're no better than what you warn against.
- throw_m239339 5y ago
  > What's the purpose of this? It will just give ammo to anti-science Trump like people. It's bizarre how today, in USA, for some it's all about "the other camp can't have a win, at any cost", and not what is right. Fake science hurts science the most, and covering it up IS WHAT is going to give ammo to whoever you do not like, calling a fraud a fraud doesn't do that.
observer20 5y ago
Psychology like psychiatry is very similar to pseudoscience and I mean identical to the snake oil salesmen. Both of the fields attempt to “help” the unfortunate, who truly need help and placebos can arguably help "overcome" life factors. I was naive young from wishful thinking/optimism and thus fell prey to the mentality of the fields being positive for society; nowadays I realize the opposite. The sad thing is that society has morally suffered tremendously from both fields' actions & thoughts throughout history. Persons having poor life outcomes from genetics, where their time is squandered by misguided trust from themselves or loved ones and where what they need isn't there while sometimes they have a lot worse because of the fields.
- weego 5y ago
  I think this is over-sold, but psychology as a field is absolutely closer to philosophy with a veneer of science put on top than the field would have you believe from the outside, and psychology as a job is absolutely not science based day-to-day.
- subroutine 5y ago
  Your assessment of psychology is flawed in several ways. The main concern of psychological research isn't to help someone. It's to understand how/why organisms think and behave under certain conditions. Psychological research can be conducted using the scientific method like any other science field. To say "Psychology", as a whole, is similar to snake oil, ignores thousands of perfectly sound findings. Fraud is certainly not unique to the field of psychology. Nor are the pressures to publish positive findings. Your final thoughts about 'life outcomes from genetics' is incoherent.
  - observer20 5y ago
    No, the sentence "persons having 'poor life outcomes' from genetics" isn't incoherent but maybe to a simpleton it happens to be.
- myWindoonn 5y ago
  It's worse than that; Pirsig reminds us that psychiatrists are literally thought police. From this perspective, psychology is a mechanism for justifying psychiatric abuses. After all, if psychology can show that minds exist, that there are normal and abnormal minds, that there are correct and incorrect modes of thought, then psychiatry has only to show that certain incorrect modes of thought form clustered disorders and syndromes. The resulting system can punish people for expressing anything which society doesn't like, from sexuality to politics to a desire not to be overworked.
jbullock35 5y ago
An important finding is buried in the linked blog post. It turns out that Ariely was hyping the fraudulent findings on NPR in February 2020. At the time, perhaps he didn't know that they were fraudulent. But he -did- know that there was great reason to doubt them. We can be confident of this point because, more than half a year earlier, he had submitted for publication a new article [1] that (a) failed to replicate the original findings, and (b) uncovered a massive problem with baseline measurements in the original study, suggesting that the randomization in that study might never even have taken place. Indeed, the whole point of the new article was to make these critical points about the older article! Even if Ariely had no role in the creation of the fake data, it seems hard to defend his behavior on NPR: hyping a study that he knew to be extremely problematic. [1] https://www.pnas.org/content/117/13/7103.short https://www.pnas.org/content/117/13/7103.short
- linuxftw 5y ago
  Let's not give NPR a pass here. They routinely present junk science as fact and never suffer any repercussions.
  - throwawaygh 5y ago
    As a general heuristic, social sciences research that tells a nice narrative and gets hyped to hell in TED talks/pop sci books/etc. is at best bullshit.
danvk 5y ago
The 2012 article states: > Specifically, we ran analyses examining whether the two conditions differed in the number of instances wherein reported odometer mileages ended with 0, 5, 00, 50, 000, or 500. Numbers that end with these digits indicate a higher likelihood that customers simply estimated their mileage. We detected no statistically significant differences between our two conditions in the instances in which these endings appeared (pooled measure: treatment, 19.9% vs. control, 20.8%; χ2 = 2.5, P = 0.12). 1. What does this mean? (“Pooled measure”) Wouldn't a mileage ending in 500 also end in 0 and 00? 2. If the numbers are uniformly randomly generated, shouldn’t the odds of ending in 0 be 10%, not 20%? Or is this an equal mix of fraudulent and real data in both arms? Just curious because this is the one of the only bits of data for the fraudulent study in the 2012 article.
mensetmanusman 5y ago
What if humans were generally so unique that there were no useful generalizations in psychology. (E.g every attempt at an experiment like this was just noise) How would that affect the field?
- sennight 5y ago
  It would probably focus on measuring the degree to which individuals differ, assigning explanatory variables, and figuring out the theoretical maximum (as brain chemistry sets a pretty hard limit). But that doesn't really matter, because we know that people can very easily be generalized - there is an entire industry devoted to it: advertising. Many years ago I had to design a process to pre-stage inventory based on expected demand - using historical data from a fairly inelastic customer base. It was spooky how accurately a Poisson distribution could predict long tailed events - but only after segmenting the customer base into something like 5 groups. This was for millions of people - 5 groups.
  - mensetmanusman 5y ago
    Makes sense for consumer behavior since there are a finite set of purchase choices. Things like ethics and behavior choices are fuzzy though…
    - sennight 5y ago
      > behavior choices I really wish this weren't the case, but "choice" isn't a thing - as far as anyone can tell. All the complexity on display in the world appears to be a result of feedback loops, it really doesn't take much to generate a continuous low interval stream of pseudo random feedback - predictable after enough observation. People who regularly purchase chunky peanut butter and cat litter have a significantly higher probability of driving a VW bug. That isn't the result of limited alternatives. https://en.wikipedia.org/wiki/Linear-feedback_shift_register https://en.wikipedia.org/wiki/Linear-feedback_shift_register
bawana 5y ago
Fake news in academia. They need to do it because their click bait does not drive ad revenue, rather it drives the awarding of grant money. Does the 'educational institution' where the researcher works then have to pay back the grant money? After all, everything in the lab is actually owned by the university and the grant is administered by the university and the university takes a hefty 'management fee'
revolvingocelot 5y ago
Hmm? Academic fakery to support some financial unreality isn't exactly new. Remember how some economists figured they wouldn't include important chunks of the first world in an Excel calculation, in order to justify the grinding austerity favoured by right-wingers in the early 2010s? [0] They kept the data, they just decided not to include it in the calculations. Amazingly, actually including the data shows that the effects of a high debt-to-GDP on growth are negligible at best, rather than overwhelmingly bad. Did they get anything more than a slap on the wrist? Are they still selling books based on the concept? The breathless commentary regarding how perfidious psychology is threatens to sprain the muscles that govern the rolling of my eyes. [0] https://en.wikipedia.org/wiki/Growth_in_a_Time_of_Debt https://en.wikipedia.org/wiki/Growth_in_a_Time_of_Debt
CrazyCatDog 5y ago
As a business academic, I can absolutely confirm that popular press attention corrupts us just as much as power corrupts political leaders. I can’t speak outside of b-schools, but suspect it’s no different. If anyone is privy to a platforms that would allow me to share proprietary data such that skeptics can run aggregated statistical analyses whilst limiting the ability to surface results with insufficient samples (to prevent scrapping the the proprietary data—if not proprietary then there is no excuse for not fully sharing) please comment! If not, YC hopefuls here is your chance to ACTUALLY make the world a better place!!! Happy to use my platform to recruit as many social scientists as you need to secure funding for a POC—seriously, ping me! Must haves: * Easily palatable settings for data obfuscation (non-tech corporate GC must be able to grok the level of obfuscation) * Trivial sliders to change level of obfuscation (eg min number of observation to run a “test”) * Messaging system to facilitate requests and permission granting between authors and skeptics * Enterprise-level data retention and documentation, such that changes to the “official” data set area reflected publicly * Cloud based R, Python, Stata functionality (else the solution would have to run locally in such a way that the granular data cannot be accessed directly on local machine—and this would likely be a non-starter for corporate GC’s). Ideally, one could reuse their local license for proprietary software (eg Stata), if not then said vendors will delight in an additional “attach” license to charge researchers even more (see below, there are beacoup bucks that will gladly be spent on this) Business model: researcher’s academic institution would (gladly) pay (if you get pushback, call the university’s academic HR department, find out who is in charge (Ie associate provost of research or HR) and email them asking they are prepared we’re to shine a spotlight on their own “Andrew Bird and Stephen Karolyi” like scholars). They will call you in 5 min. GitHub will acquire you post-POC upon enlisting 2-3 major universities (again, I know what higher-Ed procurement cycle is like, contracts will be signed record time). If GitHub fails, Google scholar will acquire you.
- anchpop 5y ago
  Hmm, I wonder if one could use differential privacy to make mathematically certain that no individual data point could leak. Although that may not be a strong enough guarantee
  - 5y ago
merricksb 5y ago
Yesterday: https://news.ycombinator.com/item?id=28257860 https://news.ycombinator.com/item?id=28257860
SNosTrAnDbLe 5y ago
I am wondering how much of such non replicated studies leaks into the decisions and recommendations by federal organizations like the CDC or ECDC or WHO. It creeps me out to think that some organizations adopted some of the ideas outlined in this person's book.

205 points by luu 5y ago

If top scientists are willing to outright manufacture data, how many would be willing to intentionally follow unsound logic? As in dishonestly claiming some (legitimate) data implies something that it doesn't or intentionally avoiding alternative explanations. You'd have to assume there's even more of that and all other kinds of soft fraud going on. Ideally planning data production, execution of data production and analysis would all be done by different people and published separately. At the very least you'd want total transparency at each stage.

throwthere 5y ago
These are top social scientists and, unfortunately, their field is a total dumpster fire at the moment. See https://www.nature.com/articles/s41562-018-0399-z https://www.nature.com/articles/s41562-018-0399-z
- auganov 5y ago
  If you're implying the replication crisis is limited to social sciences, that's too narrow. You'd expect to see these problems in any field that primarily relies on hard to gather data that is far removed from the underlying phenomenon. Many sciences that aren't strictly "soft" have "soft" fields. Say, Biology->drug research. Also not to argue semantics but when you hear the word "scientist" in common parlance it almost exclusively refers to people doing that kind of research. The "sciences" that don't have these problems tend to not be too big on using the word "science".
  - myWindoonn 5y ago
    Cosmology is the first field which comes to mind; it fits your criteria, but is far harder (as a science) than psychology and far more reliable. Psychologists can't define "mind" and change their entire profession every few decades, but "sun" has not changed much as a phenomenon for a long time. The reason why the social sciences (social psychology, criminology, economics, advertising/marketing) are hit hardest by the replication crisis is because they were not actually sciences! They are ways for our society to police its members' behavior (relationships, crime, money, attention) and they pretend to be science in order to gain prestige. There's also, finally, the matter of scale and relevance. When a chemist reports that a rare chemical reaction doesn't perform in the lab as expected, then this is a failed replication; however, chemistry labs regularly replicate entire textbooks' worth of material for entire classes of undergraduates. Meanwhile, some of the most important claims in psychology, like priming, are failing to replicate.

zhdc1 5y ago

I'm pretty sure this was discussed in an earlier HN post. However, I want to point out that "not replicating" doesn't necessarily equate to bad science (or, more accurately, bad practice on the part of the principal investigators irt accepted best practices in their field). There are a bunch of studies in the social sciences that failed to replicate simply because a p value was slightly below a threshold target, even though the coefficient direction/magnitude &c... agreed with the earlier study. That said, I'm not defending - at all - the paper discussed in the article.

qnsi 5y ago
There is a bigger accusation of fraud, not only failure to replicate Edit: Also [0] was on HN 5 days ago (discussion here: [1]). This is another blogpost that discusses [0] [0]https://datacolada.org/98 https://datacolada.org/98 [1]https://news.ycombinator.com/item?id=28210642 https://news.ycombinator.com/item?id=28210642
- the-dude 5y ago
  And Buzzfeed : https://news.ycombinator.com/item?id=28257860 https://news.ycombinator.com/item?id=28257860 ( Yesterday, 86 comments )
wanderingmind 5y ago
Failure to replicate is the definition of bad science. Science is based on objective, repeatable experiments. If the experiments can't even be reproduced how can they be hypothesized and validated?
- parliowl 5y ago
  This is so not true on many levels. Science is not about being right every time. Positive experimental findings can turn out to be wrong even if the study was excellent. This is called a type 1 error, and indeed they are guaranteed to happen with 5% probability assuming every study is conducted correctly and people don’t cheat. Also, I don’t think you understand that there is a difference between “replicate” and “reproduce” as those terms are typically used in social and biological science. Reproduction refers to the methods and is simply conducting the experiment again, regardless of the outcome. An experiment should be reproducible, of course—if it isn’t, the paper was poorly written (the experiment might still be ok though). Replication refers to the results: do we get the same answers as the original experiment? Bad science is one way an experiment might not replicate but other reasons are 1. Just a regular type 1 error in the original study, 2. A type 2 error in the new one, 3. an overlooked methodological difference or mistake somewhere. No. 3 doesn’t mean bad science, it could be just normal scientists making normal mistakes, as they sometimes do. Also, you don’t “hypothesize” an experiment. You come up with a hypothesis then test it with an experiment. (Sometimes scientists HARK but that’s a different discussion …) In this case, it sounds like there truly was some cheating going on of the worst kind (data fabrication).

aaron695 5y ago

The prediction market stuff it mentions is really interesting. https://polymarket.com/market/will-pnas-retract-dan-arielys-2012-paper-on-dishonesty-by-october-1-2021 https://polymarket.com/market/will-pnas-retract-dan-arielys-... There is also money to not have it retracted. But it is time limited. You just have to ensure it's not retracted before October. It's interesting to think where it would end up. A system that finally fixes incorrect papers but a battle ground over when.

willis936 5y ago

In private I have heard of science done that is suspected to be fraudulent. The results are so intriguing and promising as to send multiple grad students across multiple groups barreling into replicating and expanding the results. What happened? Years of time wasted with no publications. PIs and journals aren't interested in publishing negative results. The person who lied in their research is reaping career benefits of such a valuable contribution to research. Academia takes something as beautiful as the closed loop of science and opens it up only to benefit charlatans and punish the best and brightest.

gus_massa 5y ago
> The results are so intriguing and promising as to send multiple grad students across multiple groups barreling into replicating and expanding the results. One important task of the advisor is to filter bad projects. Does the paper pass the smell test? Are the authors reliable or they had published "dubious" results before? [I was recently discussing a paper in a virtual meeting. My conclusion is that Zoom does not have enough negative reactions.] Also, before assigning an experiment to a graduate student, try to have something like a plan: 1) This will surely work and is interesting enough for the dissertation and to be publish. 2) This part is dubious, because it's related to a too resent research paper or there may be some hidden technical problems that are difficult to see. 2') An underspecified alternative to 2, just in case. 3) Extra task for bonus points and perhaps a Nobel price.
newsclues 5y ago
Publishing negative results is important to solving this issue.
- atoav 5y ago
  How about instead of applying to journals with a finished study, people would apply with a proposal of a study and then publish the results independent of how spectacular/boring they turn out? Journals would then have to select studies based on their design and based on the question and scientists could just do science.

knuthsat 5y ago

You don't even need the graphs, miles driven per year is a sum. Sums converge to gaussians yet here the deviation was half of mean which means they used a uniform variable generator for miles.

throwthere 5y ago
Interesting point. So the original paper (2012) didn't give a table or anything, but said: "Customers who signed at the beginning on average revealed higher use (M = 26,098.4, SD = 12,253.4) than those who signed at the end [M = 23,670.6, SD = 12,621.4; F(1, 13,485) = 128.63, P < 0.001]." Was M = 26,098.4, SD = 12,253.4 enough to infer a uniform distribution?
- ImaCake 5y ago
  > Was M = 26,098.4, SD = 12,253.4 enough to infer a uniform distribution? No unfortunately it is not. It's perfectly plausible to have a normal distribution with mean = 26k and sd = 12k. Although those numbers do look kinda weird, but nothing you could verify. I am not sure what the F(*) means here, maybe an F statistic? But that seems wrong if you are comparing two normally distributed samples, you might expect a T statistic. To verify the distribution you would need a histogram or you could get fancy with a qqplot. You could also try a statistical test for a normal distribution but these fail on large sample sizes so the visual plots are your best bet.
deleted 5y ago
[deleted]
voldacar 5y ago
Taleb pointed this out on twitter iirc
thaumasiotes 5y ago
> yet here the deviation was half of mean which means they used a uniform variable What am I missing? The mean of a normal distribution can be any value. A normal with mean 2 and deviation 1 has deviation equal to half of the mean. Is this related to the fact that miles driven can't be negative (unlike a sample from a genuine normal)?

elevenoh 5y ago

I love polymarket. Prediction markets on topics like this are about the only way to discern truth from afar.

ImaCake 5y ago
Wouldn't polymarket be subject to incomplete information? Thus leading to bias. Polymarket would just reflect the market sentiment given a fairly robust analysis of the available information, admittedly a pretty useful idea. The other issue is that a very niche bet/topic might only have a small sample of analysts placing bets. This would mean you are subject to increased risk of bias from a single bad analyst. Or simply large variance due to small sample size.

qnsi 5y ago

I have several books of Ariely. I enjoy reading psychology, but I stopped reading pop psychology books and go for academic books or textbooks or individual articles. The amount of times I read about spectacular counter-intuitive study that "changes everything" and later find out it was fraudulent or failed to replicate makes me really doubt all of my knowledge I have from reading about psychology. Reproducibility crisis is real and it sucks for all the people that read individual books and don't know about reproducibility. I feel people that write those books should also put themself into higher standards and stop writing narrative books based on individual nonreplicated studies. You can read about reproducibility crisis here: https://www.theatlantic.com/science/archive/2018/11/psychologys-replication-crisis-real/576223/ https://www.theatlantic.com/science/archive/2018/11/psycholo...

dalbasal 5y ago
I think you need to draw a line between "not replicable" and fraudulent. This is about the latter. This is the kind of corruption that spreads. It's a very bad thing. It is related to the former problem of "not replicable," which is about application of the "scientific method." On this, my worldview has evolved a lot over the last decade. I think we've been applying overly Popperian methods, and formalisations of the scientific method where they don't work well. Say you are studying the effects of a breakfast program, 2 hr recess or push-up regiment on high schoolers. The study is well constructed, faithfully conducted, etc. You find interesting, potentially actionable ways of reducing obesity, improving psych health or something. Do/Should we even expect such a study to replicate in 1990s Tokyo, 2000s Miami, School A, School B, etc. Does that mean this study should not be conducted? Does it mean we can't generate knowledge and understanding of things this way? Is there any daylight between "anecdote" and a result that qualifies for publication in Physics? A lot of articles published in many fields feel like they are being hammered into a Popperian mold they don't fit into. Everyone knows it, but no one is willing to say "this method doesn't work here. we're actually doing a different thing." Instead, they sort of eye roll the article into the frame and let readers unscramble it for themselves.
- rwbhn 5y ago
  If the results are not replicable, how can they be actionable?

mycodesucks 5y ago

My bet would be that a surprisingly large number of psychology studies are completely irreproducible. I'm even skeptical of a lot of fMRI experiments. Meta-analyses certainly can give a bit better insight, but have their own problems.

ghyiihdruh 5y ago

What's the purpose of this? It will just give ammo to anti-science Trump like people. It's harmful. This kind of stuff should be handled as quietly as possible, and definitely not in public. Science is already under attack, we need not undermine it from the inside too.

technothrasher 5y ago
Really? Covering up a messy truth because you perceive it as not politically expedient strikes me as the exact opposite of science. That road leads to worse knowledge, not better.
AussieWog93 5y ago
Anti-science people tend to be conspiracy-minded and believe that the establishment is a homogeneous group that is trying to mislead the public. Creating an actual conspiracy to mislead the public is not a good way to fight this perception.
lyaa 5y ago
I disagree. Constant re-evaluation of evidence and improvement of working theories are essential to science. Science needs transparency and honest critique. Acknowledging falsified results is not an attack on science, it is how science works. It shows that science is not a mysterious cult of knowledge conspiring to hide information and deceive others, but rather the collective effort of imperfect humans who want to understand the universe and make things better for us all.
avnigo 5y ago
What you just said goes against everything science stands for, otherwise it just becomes a belief system, and you're no better than what you warn against.
throw_m239339 5y ago
> What's the purpose of this? It will just give ammo to anti-science Trump like people. It's bizarre how today, in USA, for some it's all about "the other camp can't have a win, at any cost", and not what is right. Fake science hurts science the most, and covering it up IS WHAT is going to give ammo to whoever you do not like, calling a fraud a fraud doesn't do that.

observer20 5y ago

Psychology like psychiatry is very similar to pseudoscience and I mean identical to the snake oil salesmen. Both of the fields attempt to “help” the unfortunate, who truly need help and placebos can arguably help "overcome" life factors. I was naive young from wishful thinking/optimism and thus fell prey to the mentality of the fields being positive for society; nowadays I realize the opposite. The sad thing is that society has morally suffered tremendously from both fields' actions & thoughts throughout history. Persons having poor life outcomes from genetics, where their time is squandered by misguided trust from themselves or loved ones and where what they need isn't there while sometimes they have a lot worse because of the fields.

weego 5y ago
I think this is over-sold, but psychology as a field is absolutely closer to philosophy with a veneer of science put on top than the field would have you believe from the outside, and psychology as a job is absolutely not science based day-to-day.
subroutine 5y ago
Your assessment of psychology is flawed in several ways. The main concern of psychological research isn't to help someone. It's to understand how/why organisms think and behave under certain conditions. Psychological research can be conducted using the scientific method like any other science field. To say "Psychology", as a whole, is similar to snake oil, ignores thousands of perfectly sound findings. Fraud is certainly not unique to the field of psychology. Nor are the pressures to publish positive findings. Your final thoughts about 'life outcomes from genetics' is incoherent.
- observer20 5y ago
  No, the sentence "persons having 'poor life outcomes' from genetics" isn't incoherent but maybe to a simpleton it happens to be.
myWindoonn 5y ago
It's worse than that; Pirsig reminds us that psychiatrists are literally thought police. From this perspective, psychology is a mechanism for justifying psychiatric abuses. After all, if psychology can show that minds exist, that there are normal and abnormal minds, that there are correct and incorrect modes of thought, then psychiatry has only to show that certain incorrect modes of thought form clustered disorders and syndromes. The resulting system can punish people for expressing anything which society doesn't like, from sexuality to politics to a desire not to be overworked.

jbullock35 5y ago

An important finding is buried in the linked blog post. It turns out that Ariely was hyping the fraudulent findings on NPR in February 2020. At the time, perhaps he didn't know that they were fraudulent. But he -did- know that there was great reason to doubt them. We can be confident of this point because, more than half a year earlier, he had submitted for publication a new article [1] that (a) failed to replicate the original findings, and (b) uncovered a massive problem with baseline measurements in the original study, suggesting that the randomization in that study might never even have taken place. Indeed, the whole point of the new article was to make these critical points about the older article! Even if Ariely had no role in the creation of the fake data, it seems hard to defend his behavior on NPR: hyping a study that he knew to be extremely problematic. [1] https://www.pnas.org/content/117/13/7103.short https://www.pnas.org/content/117/13/7103.short

linuxftw 5y ago
Let's not give NPR a pass here. They routinely present junk science as fact and never suffer any repercussions.
- throwawaygh 5y ago
  As a general heuristic, social sciences research that tells a nice narrative and gets hyped to hell in TED talks/pop sci books/etc. is at best bullshit.

danvk 5y ago

The 2012 article states: > Specifically, we ran analyses examining whether the two conditions differed in the number of instances wherein reported odometer mileages ended with 0, 5, 00, 50, 000, or 500. Numbers that end with these digits indicate a higher likelihood that customers simply estimated their mileage. We detected no statistically significant differences between our two conditions in the instances in which these endings appeared (pooled measure: treatment, 19.9% vs. control, 20.8%; χ2 = 2.5, P = 0.12). 1. What does this mean? (“Pooled measure”) Wouldn't a mileage ending in 500 also end in 0 and 00? 2. If the numbers are uniformly randomly generated, shouldn’t the odds of ending in 0 be 10%, not 20%? Or is this an equal mix of fraudulent and real data in both arms? Just curious because this is the one of the only bits of data for the fraudulent study in the 2012 article.

mensetmanusman 5y ago

What if humans were generally so unique that there were no useful generalizations in psychology. (E.g every attempt at an experiment like this was just noise) How would that affect the field?

sennight 5y ago
It would probably focus on measuring the degree to which individuals differ, assigning explanatory variables, and figuring out the theoretical maximum (as brain chemistry sets a pretty hard limit). But that doesn't really matter, because we know that people can very easily be generalized - there is an entire industry devoted to it: advertising. Many years ago I had to design a process to pre-stage inventory based on expected demand - using historical data from a fairly inelastic customer base. It was spooky how accurately a Poisson distribution could predict long tailed events - but only after segmenting the customer base into something like 5 groups. This was for millions of people - 5 groups.
- mensetmanusman 5y ago
  Makes sense for consumer behavior since there are a finite set of purchase choices. Things like ethics and behavior choices are fuzzy though…
  - sennight 5y ago
    > behavior choices I really wish this weren't the case, but "choice" isn't a thing - as far as anyone can tell. All the complexity on display in the world appears to be a result of feedback loops, it really doesn't take much to generate a continuous low interval stream of pseudo random feedback - predictable after enough observation. People who regularly purchase chunky peanut butter and cat litter have a significantly higher probability of driving a VW bug. That isn't the result of limited alternatives. https://en.wikipedia.org/wiki/Linear-feedback_shift_register https://en.wikipedia.org/wiki/Linear-feedback_shift_register

bawana 5y ago

Fake news in academia. They need to do it because their click bait does not drive ad revenue, rather it drives the awarding of grant money. Does the 'educational institution' where the researcher works then have to pay back the grant money? After all, everything in the lab is actually owned by the university and the grant is administered by the university and the university takes a hefty 'management fee'

revolvingocelot 5y ago

Hmm? Academic fakery to support some financial unreality isn't exactly new. Remember how some economists figured they wouldn't include important chunks of the first world in an Excel calculation, in order to justify the grinding austerity favoured by right-wingers in the early 2010s? [0] They kept the data, they just decided not to include it in the calculations. Amazingly, actually including the data shows that the effects of a high debt-to-GDP on growth are negligible at best, rather than overwhelmingly bad. Did they get anything more than a slap on the wrist? Are they still selling books based on the concept? The breathless commentary regarding how perfidious psychology is threatens to sprain the muscles that govern the rolling of my eyes. [0] https://en.wikipedia.org/wiki/Growth_in_a_Time_of_Debt https://en.wikipedia.org/wiki/Growth_in_a_Time_of_Debt

CrazyCatDog 5y ago

As a business academic, I can absolutely confirm that popular press attention corrupts us just as much as power corrupts political leaders. I can’t speak outside of b-schools, but suspect it’s no different. If anyone is privy to a platforms that would allow me to share proprietary data such that skeptics can run aggregated statistical analyses whilst limiting the ability to surface results with insufficient samples (to prevent scrapping the the proprietary data—if not proprietary then there is no excuse for not fully sharing) please comment! If not, YC hopefuls here is your chance to ACTUALLY make the world a better place!!! Happy to use my platform to recruit as many social scientists as you need to secure funding for a POC—seriously, ping me! Must haves: * Easily palatable settings for data obfuscation (non-tech corporate GC must be able to grok the level of obfuscation) * Trivial sliders to change level of obfuscation (eg min number of observation to run a “test”) * Messaging system to facilitate requests and permission granting between authors and skeptics * Enterprise-level data retention and documentation, such that changes to the “official” data set area reflected publicly * Cloud based R, Python, Stata functionality (else the solution would have to run locally in such a way that the granular data cannot be accessed directly on local machine—and this would likely be a non-starter for corporate GC’s). Ideally, one could reuse their local license for proprietary software (eg Stata), if not then said vendors will delight in an additional “attach” license to charge researchers even more (see below, there are beacoup bucks that will gladly be spent on this) Business model: researcher’s academic institution would (gladly) pay (if you get pushback, call the university’s academic HR department, find out who is in charge (Ie associate provost of research or HR) and email them asking they are prepared we’re to shine a spotlight on their own “Andrew Bird and Stephen Karolyi” like scholars). They will call you in 5 min. GitHub will acquire you post-POC upon enlisting 2-3 major universities (again, I know what higher-Ed procurement cycle is like, contracts will be signed record time). If GitHub fails, Google scholar will acquire you.

anchpop 5y ago
Hmm, I wonder if one could use differential privacy to make mathematically certain that no individual data point could leak. Although that may not be a strong enough guarantee
- 5y ago

merricksb 5y ago

Yesterday: https://news.ycombinator.com/item?id=28257860 https://news.ycombinator.com/item?id=28257860

SNosTrAnDbLe 5y ago

I am wondering how much of such non replicated studies leaks into the decisions and recommendations by federal organizations like the CDC or ECDC or WHO. It creeps me out to think that some organizations adopted some of the ideas outlined in this person's book.