14 ms·

An erroneous paper on religion and generosity is retracted

213 points by okfine 7y ago

neaden 7y ago
"The paper received a great deal of attention, and was covered by over 80 media outlets including The Economist, the Boston Globe, the Los Angeles Times, and Scientific American." And how many of these will cover the retraction? A dozen at most? And all those articles will be sitting out there, getting cited and read on occasion.
- bediger4000 7y ago
  The article said that the retraction was covered by 4 outlets.
  - efa 7y ago
    Boy how common and sad that is. Sensational findings on the front page. Correction saying sensational finding totally wrong buried on page ten or not published at all.
    - caseysoftware 7y ago
      Which furthers the "fake news" narrative because now even more people can point out the BS on the front page. Why don't reporters get angry about this and demand more and better retractions? This hurts their credibility more than anyone.
- charliesharding 7y ago
  Really sad to see and feels like it's becoming more common (maybe just because I'm paying closer attention). If it fits the narrative, accept first, retract later. It would be interesting to see view statistics on the original article vs the retraction.
  - astine 7y ago
    It's not just narrative fitting. There is also a strong bias towards publishing results that seem surprising because that gets more readers. Of course, that also biases toward wrong results because wrong results are likely to be surprising.
    - gumby 7y ago
      "wrong results are [more likely] to be surprising". This case is interesting because there's a large population who would find these (unproven as it turns out) results confirmatory rather than unexpected. In the end they were neither.
    - twy123456 7y ago
      Narrative fitting and surprise bias are largely orthogonal biases. Both are at work in this case. In the western press it is trendy to paint religion in a negative light, which is the narrative the paper reinforces. Imagine a bogus paper painting in an unfavorable light one of the western virtuous identities: not white, not religious, not male, not hetero. For example, "The Negative Association between Atheism and Children’s Altruism across the World". Or "The Negative Association between Homosexual Parents and Children’s Altruism across the World". Such papers will never be covered by over 80 media outlets without questioning, in spite of being surprising with respect of the narrative.
  - deleted 7y ago
    [deleted]
  - hos234 7y ago
    It's not sad, it's great. The history of science is full of drama where most issues took multiple generations to resolve. It's easy to forget, in those interim periods, people would build all kinds of castles on total BS all the time that cost society in so many ways. Today stuff gets resolved faster and that's a good thing. People, qualified or not, who get carried away by hype or bias look foolish much much faster. And thanks to how hard it is to erase mistakes from the internet good luck rebuilding lost cred.
- dukoid 7y ago
  I think there is no need to be concerned that the retraction won't be widely covered in the long run in this particular case though -- all the religious interest groups will make a big fuss about this...
  - TimTheTinker 7y ago
    That's almost like saying it's OK to falsely report about religion in the first place because "the religious interest groups will defend themselves adequately", etc. False reporting of any sort (including the failure to correct prior false reports, whether intended or not) always causes harm. Truth and principles matter. Furthermore, reducing ideas to power struggles between those who speak them is to commit both the genetic fallacy and the ad-hominem fallacy.
    - nurettin 7y ago
      Why would an idea exist other than to serve a purpose, and why would that purpose be distant from the mindset which benefits the most from it?
      - TimTheTinker 7y ago
        > Why would an idea exist other than to serve a purpose Because ideas are the substrate of thought, belief, motivation, desire, and human life itself. Without ideas we lack the ability to understand anything at all.
        nurettin 7y ago
        Not sure how that relates to my question. Why would an idea exist without someone to give it purpose?
        TimTheTinker 7y ago
        Because it constitutes belief and/or knowledge. Because it is true, or someone believes it to be so. Because it carries explanatory power - something we all strongly desire. You might call those reasons “purposes” - if so, I don’t disagree. But the truth or falsehood of an idea transcends any “purpose” someone may have for speaking it. This is the academic posture: to dispassionately evaluate truth claims without fearing the speaker.
- baddox 7y ago
  Apparently the journal published the original paper in 2015, published someone else's correction in 2016, and only published a retraction of the original paper just last month.
- yesenadam 7y ago
  I collected some headlines in 2015: Study Shows Non-Religious Kids Are More Altruistic and Generous Than Non-Religious Ones (TIME) Religious children more punitive, less likely to display altruism (Boing Boing) Study: Religious children less altruistic, more mean than nonreligious kids (Chicago Sun-Times) Religious Kids Aren’t as Good at Sharing, Study Finds (Yahoo Parenting) Study: Religion Makes Children Less Generous (Newser) Religion doesn’t make kids more generous or altruistic, study finds (LA Times) Religious children are meaner than their secular counterparts, study finds (The Guardian) Being religious makes you less generous, study finds (metro.co.uk) Kinder Without God: Kids Who Grow Up In A Religious Home Less Altruistic Than Those Without Religion (Medical Daily) Surprise! Science proves kids with religious upbringings are less generous — and so are adults (rawstory.com, the Medical Daily story reposted with a new headline)
- mackrevinack 7y ago
  there should be a law that makes them keep reposting the story every year until they reach the numbers from the original release
quotemstr 7y ago
My default presumption is that all results from the "social sciences" are false if they contradict my intuition. Whatever these fields are producing, it's not science in the Popperian or practical sense. A ton of policy has been built on the bad research and wishful thinking of the past few decades, and it's going to take a long time to unwind it all. The problem, really, has been accelerating for a while. Something really went off the rails after WWII.
- sebastos 7y ago
  My theory is that it's engineering, not science, that confirms our knowledge. You don't really know something about the world until you can use it to do things. WWII-era science was being incorporated into tools (well, weapons), so it had to work.
zdw 7y ago
As much as I like spreadsheets and other general purpose numerical analysis tools, I wish there was a restricted subset of their functionality that could be used that was more formally verifiable, to prevent issues like this and Reinhart–Rogoff (see: https://en.wikipedia.org/wiki/Growth_in_a_Time_of_Debt#Methodological_flaws https://en.wikipedia.org/wiki/Growth_in_a_Time_of_Debt#Metho... ) from being a common occurrence.
shriphani 7y ago
From the article: Although Decety’s paper had reported that they had controlled for country, they had accidentally not controlled for each country, but just treated it as a single continuous variable so that, for example “Canada” (coded as 2) was twice the “United States” (coded as 1). I mean I don't even understand how this seemed like a normal thing to do?
- UncleMeat 7y ago
  You've never had a bug in your code that seemed insane after the fact?
  - Fellshard 7y ago
    This seems like 101, 'another pair of experienced eyes would catch this' kind of malpractice.
    - smt88 7y ago
      Analyzing data in academia seems like a disaster. It's almost guaranteed to produce errors like this. You have: - people with no coding experience and, in some cases (especially in social sciences), a strong aversion to math - code that isn't unit tested, so answering the question, "Did it run correctly?" is often softened into, "Does this look plausible to me?" - a strong incentive to end up with certain results I dated a quantitative geneticist for a while, and her coding education was almost zero. She was writing code in R and essentially just changing lines until the output "looked right". It was insanely complicated math, so there was no way to make sure the output was good. The code had to be an exact match for the algorithm she had written out in mathematical notation, and there was essentially no chance of that. It got worse. She'd write the algorithm in R and then end up with batches that would take, in some cases, years to finish running. Obviously she ended up with even more dubious hacks. (For anyone curious, she's had a fairly decorated academic career under an acclaimed advisor who reviewed all of this code to some extent, and she's worked with most of the top genetics programs in the US.)
    - jvagner 7y ago
EGreg 7y ago
And even though it was corrected and the result retracted, it was already cited by many papers, the media didn’t report the retraction etc. How much of scientific reporting gets skewed by media interests (controversy and sensationalism) and funding and political interests? Furthermore, this whole field was called cargo cult science by Feynman, using correlations, p-hacking, data dredging, and more https://en.wikipedia.org/wiki/Multiple_comparisons_problem https://en.wikipedia.org/wiki/Multiple_comparisons_problem The best analysis I have seen of the corner we have painted ourselves into https://slatestarcodex.com/2014/04/28/the-control-group-is-out-of-control/ https://slatestarcodex.com/2014/04/28/the-control-group-is-o...
rahidz 7y ago
"But when they included their categorically-coded country (1 = US, 2 = Canada, and so on) in their models, it was entered not as fixed effects, with dummy variables for all of the countries except one, but as a continuous measure. This treats the variable as a measure of ‘country-ness’ (for example, Canada is twice as much a country as the US) instead of providing the fixed effects they explicitly intended" How did this not get caught immediately? If I did a study and found out that kids in Zambia are 47 more times as generous as American kids that'd make me instantly suspicious. Or maybe the reviewers were all Canadian /s
- TheCoelacanth 7y ago
  I don't think it's quite as obvious of an error as you are suggesting. They were trying to correct for scenarios like this: Hypothetically, Canadians are twice as generous as Americans and twice as religious, but religious Canadians are equally generous as non-religious Canadians and religious Americans are equally generous as non-religious Americans. On the surface, it appears that religious people are more generous, but really it's just that Canadians are more generous. Instead of treating the countries as discrete groupings, they treated them as points on a spectrum with each country being assigned an arbitrary place on the spectrum. If #3 happened to be China, they would be assuming that people in China should very similar to people in the US and Canada, because 1 vs. 3 on a scale that goes to 200 is hardly any difference at all, but really the numbers are just arbitrary identifiers.
  - tastygreenapple 7y ago
    I'm a data scientist and this is an incredibly embarrassing 'n00b' error to make. If these researchers were using anything other than deep learning, it's almost certain that each parameter of the model was manually selected. That the author made the mistake is bad, that no one caught the error is a disaster.
- im3w1l 7y ago
  What likely happens is that since the country variable is basically random, you will get a model where the country has negligible effect on the prediction.
ALittleLight 7y ago
Releasing your data should be a requirement for publication. If the original author had wanted to keep this a secret he could've withheld his data and nobody would've been able to correct him, there simply would've been discrepant studies.
- 0xb100db1ade 7y ago
  I see where you're coming from, but would the subjects be comfortable with all their data becoming public?
  - ALittleLight 7y ago
    I think you could, and should anyways, make the data anonymous. Just give every participant a GUID for a participant ID and add a step to purge personally identifiable information. Then you can share records without identity.
    - tobylane 7y ago
      That didn’t work for the AOL research several years ago. https://arstechnica.com/tech-policy/2009/09/your-secrets-live-online-in-databases-of-ruin/ https://arstechnica.com/tech-policy/2009/09/your-secrets-liv...
    - Fomite 7y ago
      Making things like medical records actually anonymous, especially in the face of bad actors, is an unsolved problem.
  - rossdavidh 7y ago
    Anonymizing data is, yes, a difficult problem, but in particular aggregated data can, and has been, reliably anonymized. For example, the problem with this dataset would have been visible in aggregated data (e.g. aggregated by nationality).
makomk 7y ago
Alas, these kinds of problems are not restricted to the social sciences. Case in point, this retraction from a couple of days ago: https://retractionwatch.com/2019/09/25/nature-paper-on-ocean-warming-retracted/ https://retractionwatch.com/2019/09/25/nature-paper-on-ocean... Very similar to this one really; the paper claimed to overturn our existing knowledge in a way that fitted a narrative people were inclined to believe (in that case: we're all doomed) and was immediately seized on by all the news sites because of it, except the statistics were mucked up and it couldn't show what it claimed to. The fact that it was so surprising should've been even more of a massive warning sign in that case though.
- justinclift 7y ago
  Interesting. Hadn't seen the Retraction Watch website before. Wonder if it'd be possible to automatically scan new papers added to (say) arxiv.org, for retracted papers in their references? eg to warn the authors, and maybe eventually automatically as part of the upload process for arxiv.org (and similar)
whymsicalburito 7y ago
There were even news sites that published articles about the original article AFTER the retraction was announced! The state of science reporting is very sad.
m0zg 7y ago
Another, more consequential paper that popped up on HN recently has been retracted as well. The one about ocean warming. The retraction notice is a masterclass on weasel language, worth reading in its own right. https://www.nature.com/articles/s41586-019-1585-5 https://www.nature.com/articles/s41586-019-1585-5
_edo 7y ago
Sometimes I feel weird coding zip codes as strings but this is a great example why. If my program ever treats a zip code like a number I would like it to throw an error. At least in this case the error looks like an accident. On topic, from yesterday: https://news.ycombinator.com/item?id=21067764 https://news.ycombinator.com/item?id=21067764 It's another social sciences paper but in this case a co-author has requested a retraction over his strong belief that the paper includes fabricated data. The retraction request has been denied. It differs from this paper in that the data anomalies look intentional.
- crooked-v 7y ago
  Non-US postal codes often include letters and spaces, and some areas don't use postal codes at all.
  - jedberg 7y ago
    All US postal codes have an optional - (hyphen) in them as well, and so you should always encode them as strings anyway.
- haasted 7y ago
  My rule of thumb is that anything that’s not part of a calculation will be a string. Any chance I ever want to multiply a zip code? Doubtful. String, it is.
- ars 7y ago
  > If my program ever treats a zip code like a number I would like it to throw an error. One interesting thing you can do though, is sort by zipcode. This sorts your mail from East to West in the US. You can use that as a rough estimate of shipping time.
  - LorenPechtel 7y ago
    So? You make sure the zip code is left zero filled and the strings sort fine. Admittedly, numbers would sort a bit faster.
mbell 7y ago
I think it's situations like this that result in society having such a hard time with 'science'. The term has been so heavily co-opted by fields that just don't have sufficient rigger for the term to hold weight. Yet, on various topics, we have this publicized attack of "your a science denier!". At the end of the day, there are two 'types' of science, one where I can take the results and make accurate predictions, and one where I can't. The later just amounts to 'our best guess' where the accuracy is entirely unknown. If we want the general populous to 'trust science', we need to stop calling the later science. In short, if you can't repeat and predict, stop calling it science.
- scarejunba 7y ago
  Perhaps the real problem is that reality does not punish being wrong and reward being right enough.
  - deleted 7y ago
    [deleted]
- learnstats2 7y ago
  That's not the problem here. A statistical error was made, published, and then corrected. That's possible in physics, chemistry, or biology just as well.
  - mbell 7y ago
    > That's possible in physics, chemistry, or biology just as well. Yet it seems to happen far less often, or at the very least, in less publicly impacting ways. Part of this is the media, but part of it is the participants themselves. If you discover something new in physics for example, either the public doesn't care, or they don't really know, they just get a faster iPhone processor a couple years down the line when the predictions hold. The folks in your field are even skeptical at first glance, "do you have a 6 sigma result?". "Ok, well, lets talk then, but I still wanna see it reproduced". Psychology on the other hand, some person does a half assed 'study' and uses it to claim knowledge of some important aspect of humanity. I'd really encourage looking at the Higgs discovery press conference as a perfect example. To my recollection, there was little to no mentioned of the Higgs, just cold hard facts, perhaps at the end there was a 'this is consistent with the Higgs'. Only months later were many of those involved even comfortable enough with their level of certainty to really say 'this is the Higgs'. They are searching out knowledge, and don't want to declare having found it unless they are certain. This is _good_, it's what we want science to be, the summation of our current, highly confident, view of the world. We don't want 'science' to encompass all untested and unproven hypothesis about the world. The difficult point which I have to concede is that terminology is important. Certainly many folks in physics that maybe haven't hit on a predictive result would like to be recognized as scientists, and rightly so if they are on the path toward this endeavor. But the populous is simple, they want clearly defined words, they want 'science' to be known fact. Outside introducing new words, I don't know how to resolve this. If we don't define 'science' as denoting that which is rigorous, then we can't use "science denier" as a term, regardless of the topic.
deleted 7y ago
[deleted]
realradicalwash 7y ago
so a categorical variable got mixed up as a numerical one and produced misleading results. to the credit of the authors, they released their data sets. -- however, i suspect that proper data exploration and visualisation would have prevented all this. visual inspection would have most likely revealed that there is no visible effect, or even an effect in the opposite direction, and once you see this, all alarm bells should go off if your model predicts otherwise. so i suspect that the authors skipped some basic steps and got carried away by results that promised a nice headline.
- rossdavidh 7y ago
  Agreed. However, another factor may be that, we tend to scrutinize more closely results that don't line up with what we expected. In fact, in this case it was another researcher who had gotten results that pointed in the opposite direction, who convinced the original researcher to release their data (which, to their credit, they did). Which, is one reason why having higher and higher percentages of academia and science researchers be from the same part of the political spectrum, is worrisome to me. If you have more diversity in ideology, there is more likely to be someone in each field to have the instinct to scrutinize closely a result which, when scrutinized, won't hold up.
Scapeghost 7y ago
I am from a third-world religious country and I'd just like to take this chance to say that religion should be classified as a mental disorder. I see the effects of it first hand, severely crippling the potential of individuals and entire cultures. There's nothing it offers that can't be taught by basic morals which don't require you to appease an imaginary entity or repress the human spirit. Even the leaders of religious countries know it's bullshit (see the UAE, where there are "No Displays of Affection" signs in upscale shopping malls but the streets are littered with cards advertising phone numbers for prostitution services), but maintain it as a convenient form of manipulation. If a handful of Dungeons & Dragons players got together and started dedicating all their actions to the Drow goddess Lolth they would be called loonies, but when millions of people worship an imaginary entity for centuries and erect monuments to it and start wars in its name, apparently that's something that should be respected. This is scary to even talk about. Not only would criticizing religion be frowned upon by first-worlders who seem to have a fetish for "exotic" cultures they have never lived in, it would put the critic at risk of physical harm in their own community.
- leereeves 7y ago
  While I'm an atheist and agree with you in part, I don't see how that's related to this story.
  - Scapeghost 7y ago
    There aren't many discussions about religion on HN but a topic related to it should prompt some, and I have no other outlet. Reddit isn't a very good place for it either.
    - Karunamon 7y ago
      You pretty much torpedoed any chance of a productive discussion on the subject when you opened with "religion should be classified as a mental disorder." That comment is weapons-grade flamebait on a discussion board and probably not something you'd say in mixed company in real life either.
cjohansson 7y ago
Psychology, sociology and theology (and more logys?) was never meant to be sciences. We can blame the enlightenment for that idea, let’s revert them back to renaissance activities
- daenz 7y ago
  There's too much power in them being considered sciences, which is why they'll remain that way.
- mr_overalls 7y ago
  Psychology and sociology can be made as rigorous as individuals & institutions care to. Theology, not so much.
  - high_5 7y ago
    Actually most of the (christian) theology (when discussed by theologists) within its confines is very rigorous.
    - mr_overalls 7y ago
      Theology can be rigorous in the sense of its models possessing internal validity (logical consistency, parsimony). However, because it's basically speculation about unobservable entities, theology completely lacks external validity. Social sciences can have both.
- blackbrokkoli 7y ago
  I hope you do realize the irony in just declaring that something was "never meant to be a science"... Science is about constantly evolving via falsification of theories. You might want to base your definition of what can be science on something different than an unsourced appeal to authority from 300 years ago?
pygy_ 7y ago
Conclusion of the new analysis: In sum, Decety et al. [1] have amassed a large and valuable dataset, but our reanalyses provide different interpretations of the authors’ initial conclusions. Most of the associations they observed with religious affiliation appear to be artifacts of between-country differences, driven primarily by low levels of generosity in Turkey and South Africa. However, children from highly religious households do appear slightly less generous than those from moderately religious ones. https://www.sciencedirect.com/science/article/pii/S0960982216306704 https://www.sciencedirect.com/science/article/pii/S096098221...
blackbrokkoli 7y ago
This is fundamentally one of the many disastrous outcomes of scientific publishing being controlled by greedy conglomerates. For the uninitiated: If you want to publish a scientific paper today, you basically sign up to sign over the rights to any publisher that's interested (please someone publish me). That publisher will then review that paper in a more often than not mostly undisclosed process and publish that paper. Everyone knows that to be a high-regarded publisher one must have a very own typographic formatting: Unreadable font, weird multi column layout to prevent accessibility and tables disregarding standards are a good start. Afterwards, the paper gets published on the publisher website, which again follows as little agreed upon standard as possible. Data is of course excluded, study itself is in PDF. Also put up a fat paywall, don't want those pesky poor people be scientifically literate. Give a few cents of your $70 fee to the authors, it is not an unethical business you're doing here! THIS is the systemic error people seem to be so happy to ignore because they're neck deep in social science not being real science memes. This prevents interesting new startups for fact-checking or meta analysis, which are e.g. happening in journalism because that field has a lot of the things science is sadly lacking. This creates a drift between extremely rich and rather poor countries/unis/humans in scientific ability. This generates a tar pit for scientific process. This wastes billions in funds because of people unaware of each other doing redundant studies (and not referencing/refuting/supporting each other in the process neither, of course) because they are literally better search engines to find Harry Potter fan fiction than there are for finding studies. And finally, this of course allows anything from honest statistical mistakes to snake oil sellers to slip through and doing generations worth of damage, because correcting, fact checking, re-researching, comparing, meta research, anything is slowed to a crawl. So please stop embracing scientific elitism and gatekeeping for this is exactly what brought us here in first place...
bshoemaker 7y ago
Fun to see all these software engineers criticizing scientists for a bug in their code
krilly 7y ago
Wait a second. If the issue with the analysis is as the article says, that is that some countries were weighted far higher, that means that in some countries the original conclusion does hold (probably for a small sample size). Perhaps this is because some religions promote generosity and some do not? Would be interesting to look in to.
jimmaswell 7y ago
Interesting, waitstaff on reddit always say religious people tip worse or not at all.
- CDSlice 7y ago
  Probably because most religious people don't tend to flaunt it everywhere they go, and in my experience the ones that do also tend to be jerks. It's also possible that wait staff tend to remember the ones that do more than the ones that don't because being religious and a jerk is more memorable than religious and not a jerk. Disclaimer: I am a religious person.
  - SauciestGNU 7y ago
    I worked in breakfast and brunch restaurants for a while, and my anecdotal experience is that the post-church rushes (we were across the street from a couple churches) had the worst overall tipping percentages that I would see all week.
    - mikestew 7y ago
      One could argue that those are the folks that aren’t religious enough to feel that they should not make others work on a sabbath as a result their personal commerce. The religious folk that make it rain come in on other days of the week.
peteretep 7y ago
> In fact, Decety’s paper has continued to be cited in media articles on religion. Just last month two such articles appeared (one on Buzzworthy and one on TruthTheory) citing Decety’s paper that religious children were less generous. “Media articles” is carrying a lot of weight there
yesenadam 7y ago
I wrote a webpage in 2015 about the Decety paper, what it was and how it was presented in the media, which might be of interest. The paper seemed highly suspect in various ways even at the time. I added an update in 2017 that "new analysis shows that the original study failed to adequately control for the children’s nationality". On the (unfinished) page I'm using the paper as an excuse to teach myself basic statistics and research methods. http://www.adamponting.com/decety/ http://www.adamponting.com/decety/
riffraff 7y ago
I wonder if a religious education resulting in some positive effects might be akin to the "any diet is effective" thing, i.e. it's not the specific upbringing as much as being exposed to _any_ moral thinking in general. I don't mean to imply that there aren't moral a-religious educators of course, just that it seems likely to have less discussion of ethics for any kid once the temple/church/mosque/whatever is removed.
d0100 7y ago
> Decety What an apt name. I wonder about the damage to public this unintentional deceit will bring... Go science! After all, a retraction is part of discovery.
imperio59 7y ago
Can we go back to basic principles and all agree that psychology and social studies are not true sciences anyways? Science is to me something where you extract natural laws that predict phenomena will occur 100% of the time given certain conditions. Physics, chemistry, computer science, most branches of medicine operate this way. A field like psychology that says "well sometimes people will..." or "we found in 60% of cases that..." is not science. It's a comment upon society maybe, but it does not produce broadly repeatable, predictable results. Not true science.
- DanBC 7y ago
  > most branches of medicine operate this way. No they don't.
- OnlineCourage 7y ago
  Those sciences not developed at Edinburgh are not developed by a true scottsman, either.
- ausbah 7y ago
  I think you're forgetting that the "replication crisis" also occurs in your so called "100% deterministic fields". CS has the issue of ML models, medicine drug studies, etc. The bottom line is science is hard to do, and even harder to do right. I think all fields do try to follow the scientific method to the best of their ability, but for some fields it is simply harder to do because of the sheer complexity of the subject matter being hard to model, experiment in, and understand.
otakucode 7y ago
If a research does not release the code used in their research, their papers should not be trusted. We wouldn't trust a paper where the researchers hid their experimental methods or analysis - it wouldn't even be allowed to be published. But if their experimental methods or analysis are done in code, those parts are allowed to be a black box and we're supposed to trust them. If we were willing to just trust them, there wouldn't even be a peer review process and it certainly wouldn't deserve the imprimatur of 'science'. This has been a problem for years, and I will personally not be the least bit surprised if we eventually see a death toll attached to it.

213 points by okfine 7y ago

"The paper received a great deal of attention, and was covered by over 80 media outlets including The Economist, the Boston Globe, the Los Angeles Times, and Scientific American." And how many of these will cover the retraction? A dozen at most? And all those articles will be sitting out there, getting cited and read on occasion.

bediger4000 7y ago
The article said that the retraction was covered by 4 outlets.
- efa 7y ago
  Boy how common and sad that is. Sensational findings on the front page. Correction saying sensational finding totally wrong buried on page ten or not published at all.
  - caseysoftware 7y ago
    Which furthers the "fake news" narrative because now even more people can point out the BS on the front page. Why don't reporters get angry about this and demand more and better retractions? This hurts their credibility more than anyone.
charliesharding 7y ago
Really sad to see and feels like it's becoming more common (maybe just because I'm paying closer attention). If it fits the narrative, accept first, retract later. It would be interesting to see view statistics on the original article vs the retraction.
- astine 7y ago
  It's not just narrative fitting. There is also a strong bias towards publishing results that seem surprising because that gets more readers. Of course, that also biases toward wrong results because wrong results are likely to be surprising.
  - gumby 7y ago
    "wrong results are [more likely] to be surprising". This case is interesting because there's a large population who would find these (unproven as it turns out) results confirmatory rather than unexpected. In the end they were neither.
  - twy123456 7y ago
    Narrative fitting and surprise bias are largely orthogonal biases. Both are at work in this case. In the western press it is trendy to paint religion in a negative light, which is the narrative the paper reinforces. Imagine a bogus paper painting in an unfavorable light one of the western virtuous identities: not white, not religious, not male, not hetero. For example, "The Negative Association between Atheism and Children’s Altruism across the World". Or "The Negative Association between Homosexual Parents and Children’s Altruism across the World". Such papers will never be covered by over 80 media outlets without questioning, in spite of being surprising with respect of the narrative.
- deleted 7y ago
  [deleted]
- hos234 7y ago
  It's not sad, it's great. The history of science is full of drama where most issues took multiple generations to resolve. It's easy to forget, in those interim periods, people would build all kinds of castles on total BS all the time that cost society in so many ways. Today stuff gets resolved faster and that's a good thing. People, qualified or not, who get carried away by hype or bias look foolish much much faster. And thanks to how hard it is to erase mistakes from the internet good luck rebuilding lost cred.
dukoid 7y ago
I think there is no need to be concerned that the retraction won't be widely covered in the long run in this particular case though -- all the religious interest groups will make a big fuss about this...
- TimTheTinker 7y ago
  That's almost like saying it's OK to falsely report about religion in the first place because "the religious interest groups will defend themselves adequately", etc. False reporting of any sort (including the failure to correct prior false reports, whether intended or not) always causes harm. Truth and principles matter. Furthermore, reducing ideas to power struggles between those who speak them is to commit both the genetic fallacy and the ad-hominem fallacy.
  - nurettin 7y ago
    Why would an idea exist other than to serve a purpose, and why would that purpose be distant from the mindset which benefits the most from it?
    - TimTheTinker 7y ago
      > Why would an idea exist other than to serve a purpose Because ideas are the substrate of thought, belief, motivation, desire, and human life itself. Without ideas we lack the ability to understand anything at all.
      - nurettin 7y ago
        Not sure how that relates to my question. Why would an idea exist without someone to give it purpose?
        TimTheTinker 7y ago
        Because it constitutes belief and/or knowledge. Because it is true, or someone believes it to be so. Because it carries explanatory power - something we all strongly desire. You might call those reasons “purposes” - if so, I don’t disagree. But the truth or falsehood of an idea transcends any “purpose” someone may have for speaking it. This is the academic posture: to dispassionately evaluate truth claims without fearing the speaker.
baddox 7y ago
Apparently the journal published the original paper in 2015, published someone else's correction in 2016, and only published a retraction of the original paper just last month.
yesenadam 7y ago
I collected some headlines in 2015: Study Shows Non-Religious Kids Are More Altruistic and Generous Than Non-Religious Ones (TIME) Religious children more punitive, less likely to display altruism (Boing Boing) Study: Religious children less altruistic, more mean than nonreligious kids (Chicago Sun-Times) Religious Kids Aren’t as Good at Sharing, Study Finds (Yahoo Parenting) Study: Religion Makes Children Less Generous (Newser) Religion doesn’t make kids more generous or altruistic, study finds (LA Times) Religious children are meaner than their secular counterparts, study finds (The Guardian) Being religious makes you less generous, study finds (metro.co.uk) Kinder Without God: Kids Who Grow Up In A Religious Home Less Altruistic Than Those Without Religion (Medical Daily) Surprise! Science proves kids with religious upbringings are less generous — and so are adults (rawstory.com, the Medical Daily story reposted with a new headline)
mackrevinack 7y ago
there should be a law that makes them keep reposting the story every year until they reach the numbers from the original release

quotemstr 7y ago

My default presumption is that all results from the "social sciences" are false if they contradict my intuition. Whatever these fields are producing, it's not science in the Popperian or practical sense. A ton of policy has been built on the bad research and wishful thinking of the past few decades, and it's going to take a long time to unwind it all. The problem, really, has been accelerating for a while. Something really went off the rails after WWII.

sebastos 7y ago
My theory is that it's engineering, not science, that confirms our knowledge. You don't really know something about the world until you can use it to do things. WWII-era science was being incorporated into tools (well, weapons), so it had to work.

zdw 7y ago

As much as I like spreadsheets and other general purpose numerical analysis tools, I wish there was a restricted subset of their functionality that could be used that was more formally verifiable, to prevent issues like this and Reinhart–Rogoff (see: https://en.wikipedia.org/wiki/Growth_in_a_Time_of_Debt#Methodological_flaws https://en.wikipedia.org/wiki/Growth_in_a_Time_of_Debt#Metho... ) from being a common occurrence.

shriphani 7y ago

From the article: Although Decety’s paper had reported that they had controlled for country, they had accidentally not controlled for each country, but just treated it as a single continuous variable so that, for example “Canada” (coded as 2) was twice the “United States” (coded as 1). I mean I don't even understand how this seemed like a normal thing to do?

UncleMeat 7y ago
You've never had a bug in your code that seemed insane after the fact?
- Fellshard 7y ago
  This seems like 101, 'another pair of experienced eyes would catch this' kind of malpractice.
  - smt88 7y ago
    Analyzing data in academia seems like a disaster. It's almost guaranteed to produce errors like this. You have: - people with no coding experience and, in some cases (especially in social sciences), a strong aversion to math - code that isn't unit tested, so answering the question, "Did it run correctly?" is often softened into, "Does this look plausible to me?" - a strong incentive to end up with certain results I dated a quantitative geneticist for a while, and her coding education was almost zero. She was writing code in R and essentially just changing lines until the output "looked right". It was insanely complicated math, so there was no way to make sure the output was good. The code had to be an exact match for the algorithm she had written out in mathematical notation, and there was essentially no chance of that. It got worse. She'd write the algorithm in R and then end up with batches that would take, in some cases, years to finish running. Obviously she ended up with even more dubious hacks. (For anyone curious, she's had a fairly decorated academic career under an acclaimed advisor who reviewed all of this code to some extent, and she's worked with most of the top genetics programs in the US.)
  - jvagner 7y ago

EGreg 7y ago

And even though it was corrected and the result retracted, it was already cited by many papers, the media didn’t report the retraction etc. How much of scientific reporting gets skewed by media interests (controversy and sensationalism) and funding and political interests? Furthermore, this whole field was called cargo cult science by Feynman, using correlations, p-hacking, data dredging, and more https://en.wikipedia.org/wiki/Multiple_comparisons_problem https://en.wikipedia.org/wiki/Multiple_comparisons_problem The best analysis I have seen of the corner we have painted ourselves into https://slatestarcodex.com/2014/04/28/the-control-group-is-out-of-control/ https://slatestarcodex.com/2014/04/28/the-control-group-is-o...

rahidz 7y ago

"But when they included their categorically-coded country (1 = US, 2 = Canada, and so on) in their models, it was entered not as fixed effects, with dummy variables for all of the countries except one, but as a continuous measure. This treats the variable as a measure of ‘country-ness’ (for example, Canada is twice as much a country as the US) instead of providing the fixed effects they explicitly intended" How did this not get caught immediately? If I did a study and found out that kids in Zambia are 47 more times as generous as American kids that'd make me instantly suspicious. Or maybe the reviewers were all Canadian /s

TheCoelacanth 7y ago
I don't think it's quite as obvious of an error as you are suggesting. They were trying to correct for scenarios like this: Hypothetically, Canadians are twice as generous as Americans and twice as religious, but religious Canadians are equally generous as non-religious Canadians and religious Americans are equally generous as non-religious Americans. On the surface, it appears that religious people are more generous, but really it's just that Canadians are more generous. Instead of treating the countries as discrete groupings, they treated them as points on a spectrum with each country being assigned an arbitrary place on the spectrum. If #3 happened to be China, they would be assuming that people in China should very similar to people in the US and Canada, because 1 vs. 3 on a scale that goes to 200 is hardly any difference at all, but really the numbers are just arbitrary identifiers.
- tastygreenapple 7y ago
  I'm a data scientist and this is an incredibly embarrassing 'n00b' error to make. If these researchers were using anything other than deep learning, it's almost certain that each parameter of the model was manually selected. That the author made the mistake is bad, that no one caught the error is a disaster.
im3w1l 7y ago
What likely happens is that since the country variable is basically random, you will get a model where the country has negligible effect on the prediction.

ALittleLight 7y ago

Releasing your data should be a requirement for publication. If the original author had wanted to keep this a secret he could've withheld his data and nobody would've been able to correct him, there simply would've been discrepant studies.

0xb100db1ade 7y ago
I see where you're coming from, but would the subjects be comfortable with all their data becoming public?
- ALittleLight 7y ago
  I think you could, and should anyways, make the data anonymous. Just give every participant a GUID for a participant ID and add a step to purge personally identifiable information. Then you can share records without identity.
  - tobylane 7y ago
    That didn’t work for the AOL research several years ago. https://arstechnica.com/tech-policy/2009/09/your-secrets-live-online-in-databases-of-ruin/ https://arstechnica.com/tech-policy/2009/09/your-secrets-liv...
  - Fomite 7y ago
    Making things like medical records actually anonymous, especially in the face of bad actors, is an unsolved problem.
- rossdavidh 7y ago
  Anonymizing data is, yes, a difficult problem, but in particular aggregated data can, and has been, reliably anonymized. For example, the problem with this dataset would have been visible in aggregated data (e.g. aggregated by nationality).

makomk 7y ago

Alas, these kinds of problems are not restricted to the social sciences. Case in point, this retraction from a couple of days ago: https://retractionwatch.com/2019/09/25/nature-paper-on-ocean-warming-retracted/ https://retractionwatch.com/2019/09/25/nature-paper-on-ocean... Very similar to this one really; the paper claimed to overturn our existing knowledge in a way that fitted a narrative people were inclined to believe (in that case: we're all doomed) and was immediately seized on by all the news sites because of it, except the statistics were mucked up and it couldn't show what it claimed to. The fact that it was so surprising should've been even more of a massive warning sign in that case though.

justinclift 7y ago
Interesting. Hadn't seen the Retraction Watch website before. Wonder if it'd be possible to automatically scan new papers added to (say) arxiv.org, for retracted papers in their references? eg to warn the authors, and maybe eventually automatically as part of the upload process for arxiv.org (and similar)

whymsicalburito 7y ago

There were even news sites that published articles about the original article AFTER the retraction was announced! The state of science reporting is very sad.

m0zg 7y ago

Another, more consequential paper that popped up on HN recently has been retracted as well. The one about ocean warming. The retraction notice is a masterclass on weasel language, worth reading in its own right. https://www.nature.com/articles/s41586-019-1585-5 https://www.nature.com/articles/s41586-019-1585-5

_edo 7y ago

Sometimes I feel weird coding zip codes as strings but this is a great example why. If my program ever treats a zip code like a number I would like it to throw an error. At least in this case the error looks like an accident. On topic, from yesterday: https://news.ycombinator.com/item?id=21067764 https://news.ycombinator.com/item?id=21067764 It's another social sciences paper but in this case a co-author has requested a retraction over his strong belief that the paper includes fabricated data. The retraction request has been denied. It differs from this paper in that the data anomalies look intentional.

crooked-v 7y ago
Non-US postal codes often include letters and spaces, and some areas don't use postal codes at all.
- jedberg 7y ago
  All US postal codes have an optional - (hyphen) in them as well, and so you should always encode them as strings anyway.
haasted 7y ago
My rule of thumb is that anything that’s not part of a calculation will be a string. Any chance I ever want to multiply a zip code? Doubtful. String, it is.
ars 7y ago
> If my program ever treats a zip code like a number I would like it to throw an error. One interesting thing you can do though, is sort by zipcode. This sorts your mail from East to West in the US. You can use that as a rough estimate of shipping time.
- LorenPechtel 7y ago
  So? You make sure the zip code is left zero filled and the strings sort fine. Admittedly, numbers would sort a bit faster.

mbell 7y ago

I think it's situations like this that result in society having such a hard time with 'science'. The term has been so heavily co-opted by fields that just don't have sufficient rigger for the term to hold weight. Yet, on various topics, we have this publicized attack of "your a science denier!". At the end of the day, there are two 'types' of science, one where I can take the results and make accurate predictions, and one where I can't. The later just amounts to 'our best guess' where the accuracy is entirely unknown. If we want the general populous to 'trust science', we need to stop calling the later science. In short, if you can't repeat and predict, stop calling it science.

scarejunba 7y ago
Perhaps the real problem is that reality does not punish being wrong and reward being right enough.
- deleted 7y ago
  [deleted]
learnstats2 7y ago
That's not the problem here. A statistical error was made, published, and then corrected. That's possible in physics, chemistry, or biology just as well.
- mbell 7y ago
  > That's possible in physics, chemistry, or biology just as well. Yet it seems to happen far less often, or at the very least, in less publicly impacting ways. Part of this is the media, but part of it is the participants themselves. If you discover something new in physics for example, either the public doesn't care, or they don't really know, they just get a faster iPhone processor a couple years down the line when the predictions hold. The folks in your field are even skeptical at first glance, "do you have a 6 sigma result?". "Ok, well, lets talk then, but I still wanna see it reproduced". Psychology on the other hand, some person does a half assed 'study' and uses it to claim knowledge of some important aspect of humanity. I'd really encourage looking at the Higgs discovery press conference as a perfect example. To my recollection, there was little to no mentioned of the Higgs, just cold hard facts, perhaps at the end there was a 'this is consistent with the Higgs'. Only months later were many of those involved even comfortable enough with their level of certainty to really say 'this is the Higgs'. They are searching out knowledge, and don't want to declare having found it unless they are certain. This is _good_, it's what we want science to be, the summation of our current, highly confident, view of the world. We don't want 'science' to encompass all untested and unproven hypothesis about the world. The difficult point which I have to concede is that terminology is important. Certainly many folks in physics that maybe haven't hit on a predictive result would like to be recognized as scientists, and rightly so if they are on the path toward this endeavor. But the populous is simple, they want clearly defined words, they want 'science' to be known fact. Outside introducing new words, I don't know how to resolve this. If we don't define 'science' as denoting that which is rigorous, then we can't use "science denier" as a term, regardless of the topic.

deleted 7y ago

[deleted]

realradicalwash 7y ago

so a categorical variable got mixed up as a numerical one and produced misleading results. to the credit of the authors, they released their data sets. -- however, i suspect that proper data exploration and visualisation would have prevented all this. visual inspection would have most likely revealed that there is no visible effect, or even an effect in the opposite direction, and once you see this, all alarm bells should go off if your model predicts otherwise. so i suspect that the authors skipped some basic steps and got carried away by results that promised a nice headline.

rossdavidh 7y ago
Agreed. However, another factor may be that, we tend to scrutinize more closely results that don't line up with what we expected. In fact, in this case it was another researcher who had gotten results that pointed in the opposite direction, who convinced the original researcher to release their data (which, to their credit, they did). Which, is one reason why having higher and higher percentages of academia and science researchers be from the same part of the political spectrum, is worrisome to me. If you have more diversity in ideology, there is more likely to be someone in each field to have the instinct to scrutinize closely a result which, when scrutinized, won't hold up.

Scapeghost 7y ago

I am from a third-world religious country and I'd just like to take this chance to say that religion should be classified as a mental disorder. I see the effects of it first hand, severely crippling the potential of individuals and entire cultures. There's nothing it offers that can't be taught by basic morals which don't require you to appease an imaginary entity or repress the human spirit. Even the leaders of religious countries know it's bullshit (see the UAE, where there are "No Displays of Affection" signs in upscale shopping malls but the streets are littered with cards advertising phone numbers for prostitution services), but maintain it as a convenient form of manipulation. If a handful of Dungeons & Dragons players got together and started dedicating all their actions to the Drow goddess Lolth they would be called loonies, but when millions of people worship an imaginary entity for centuries and erect monuments to it and start wars in its name, apparently that's something that should be respected. This is scary to even talk about. Not only would criticizing religion be frowned upon by first-worlders who seem to have a fetish for "exotic" cultures they have never lived in, it would put the critic at risk of physical harm in their own community.

leereeves 7y ago
While I'm an atheist and agree with you in part, I don't see how that's related to this story.
- Scapeghost 7y ago
  There aren't many discussions about religion on HN but a topic related to it should prompt some, and I have no other outlet. Reddit isn't a very good place for it either.
  - Karunamon 7y ago
    You pretty much torpedoed any chance of a productive discussion on the subject when you opened with "religion should be classified as a mental disorder." That comment is weapons-grade flamebait on a discussion board and probably not something you'd say in mixed company in real life either.

cjohansson 7y ago

Psychology, sociology and theology (and more logys?) was never meant to be sciences. We can blame the enlightenment for that idea, let’s revert them back to renaissance activities

daenz 7y ago
There's too much power in them being considered sciences, which is why they'll remain that way.
mr_overalls 7y ago
Psychology and sociology can be made as rigorous as individuals & institutions care to. Theology, not so much.
- high_5 7y ago
  Actually most of the (christian) theology (when discussed by theologists) within its confines is very rigorous.
  - mr_overalls 7y ago
    Theology can be rigorous in the sense of its models possessing internal validity (logical consistency, parsimony). However, because it's basically speculation about unobservable entities, theology completely lacks external validity. Social sciences can have both.
blackbrokkoli 7y ago
I hope you do realize the irony in just declaring that something was "never meant to be a science"... Science is about constantly evolving via falsification of theories. You might want to base your definition of what can be science on something different than an unsourced appeal to authority from 300 years ago?

pygy_ 7y ago

Conclusion of the new analysis: In sum, Decety et al. [1] have amassed a large and valuable dataset, but our reanalyses provide different interpretations of the authors’ initial conclusions. Most of the associations they observed with religious affiliation appear to be artifacts of between-country differences, driven primarily by low levels of generosity in Turkey and South Africa. However, children from highly religious households do appear slightly less generous than those from moderately religious ones. https://www.sciencedirect.com/science/article/pii/S0960982216306704 https://www.sciencedirect.com/science/article/pii/S096098221...

blackbrokkoli 7y ago

This is fundamentally one of the many disastrous outcomes of scientific publishing being controlled by greedy conglomerates. For the uninitiated: If you want to publish a scientific paper today, you basically sign up to sign over the rights to any publisher that's interested (please someone publish me). That publisher will then review that paper in a more often than not mostly undisclosed process and publish that paper. Everyone knows that to be a high-regarded publisher one must have a very own typographic formatting: Unreadable font, weird multi column layout to prevent accessibility and tables disregarding standards are a good start. Afterwards, the paper gets published on the publisher website, which again follows as little agreed upon standard as possible. Data is of course excluded, study itself is in PDF. Also put up a fat paywall, don't want those pesky poor people be scientifically literate. Give a few cents of your $70 fee to the authors, it is not an unethical business you're doing here! THIS is the systemic error people seem to be so happy to ignore because they're neck deep in social science not being real science memes. This prevents interesting new startups for fact-checking or meta analysis, which are e.g. happening in journalism because that field has a lot of the things science is sadly lacking. This creates a drift between extremely rich and rather poor countries/unis/humans in scientific ability. This generates a tar pit for scientific process. This wastes billions in funds because of people unaware of each other doing redundant studies (and not referencing/refuting/supporting each other in the process neither, of course) because they are literally better search engines to find Harry Potter fan fiction than there are for finding studies. And finally, this of course allows anything from honest statistical mistakes to snake oil sellers to slip through and doing generations worth of damage, because correcting, fact checking, re-researching, comparing, meta research, anything is slowed to a crawl. So please stop embracing scientific elitism and gatekeeping for this is exactly what brought us here in first place...

bshoemaker 7y ago

Fun to see all these software engineers criticizing scientists for a bug in their code

krilly 7y ago

Wait a second. If the issue with the analysis is as the article says, that is that some countries were weighted far higher, that means that in some countries the original conclusion does hold (probably for a small sample size). Perhaps this is because some religions promote generosity and some do not? Would be interesting to look in to.

jimmaswell 7y ago

Interesting, waitstaff on reddit always say religious people tip worse or not at all.

CDSlice 7y ago
Probably because most religious people don't tend to flaunt it everywhere they go, and in my experience the ones that do also tend to be jerks. It's also possible that wait staff tend to remember the ones that do more than the ones that don't because being religious and a jerk is more memorable than religious and not a jerk. Disclaimer: I am a religious person.
- SauciestGNU 7y ago
  I worked in breakfast and brunch restaurants for a while, and my anecdotal experience is that the post-church rushes (we were across the street from a couple churches) had the worst overall tipping percentages that I would see all week.
  - mikestew 7y ago
    One could argue that those are the folks that aren’t religious enough to feel that they should not make others work on a sabbath as a result their personal commerce. The religious folk that make it rain come in on other days of the week.

peteretep 7y ago

> In fact, Decety’s paper has continued to be cited in media articles on religion. Just last month two such articles appeared (one on Buzzworthy and one on TruthTheory) citing Decety’s paper that religious children were less generous. “Media articles” is carrying a lot of weight there

yesenadam 7y ago

I wrote a webpage in 2015 about the Decety paper, what it was and how it was presented in the media, which might be of interest. The paper seemed highly suspect in various ways even at the time. I added an update in 2017 that "new analysis shows that the original study failed to adequately control for the children’s nationality". On the (unfinished) page I'm using the paper as an excuse to teach myself basic statistics and research methods. http://www.adamponting.com/decety/ http://www.adamponting.com/decety/

riffraff 7y ago

I wonder if a religious education resulting in some positive effects might be akin to the "any diet is effective" thing, i.e. it's not the specific upbringing as much as being exposed to _any_ moral thinking in general. I don't mean to imply that there aren't moral a-religious educators of course, just that it seems likely to have less discussion of ethics for any kid once the temple/church/mosque/whatever is removed.

d0100 7y ago

> Decety What an apt name. I wonder about the damage to public this unintentional deceit will bring... Go science! After all, a retraction is part of discovery.

imperio59 7y ago

Can we go back to basic principles and all agree that psychology and social studies are not true sciences anyways? Science is to me something where you extract natural laws that predict phenomena will occur 100% of the time given certain conditions. Physics, chemistry, computer science, most branches of medicine operate this way. A field like psychology that says "well sometimes people will..." or "we found in 60% of cases that..." is not science. It's a comment upon society maybe, but it does not produce broadly repeatable, predictable results. Not true science.

DanBC 7y ago
> most branches of medicine operate this way. No they don't.
OnlineCourage 7y ago
Those sciences not developed at Edinburgh are not developed by a true scottsman, either.
ausbah 7y ago
I think you're forgetting that the "replication crisis" also occurs in your so called "100% deterministic fields". CS has the issue of ML models, medicine drug studies, etc. The bottom line is science is hard to do, and even harder to do right. I think all fields do try to follow the scientific method to the best of their ability, but for some fields it is simply harder to do because of the sheer complexity of the subject matter being hard to model, experiment in, and understand.

otakucode 7y ago

If a research does not release the code used in their research, their papers should not be trusted. We wouldn't trust a paper where the researchers hid their experimental methods or analysis - it wouldn't even be allowed to be published. But if their experimental methods or analysis are done in code, those parts are allowed to be a black box and we're supposed to trust them. If we were willing to just trust them, there wouldn't even be a peer review process and it certainly wouldn't deserve the imprimatur of 'science'. This has been a problem for years, and I will personally not be the least bit surprised if we eventually see a death toll attached to it.