4 ms·

> For an example, the replication crisis in the social sciences. These studies, containing flawed statistics, are carried out by social scientists with PhDs who

> For an example, the replication crisis in the social sciences. These studies, containing flawed statistics, are carried out by social scientists with PhDs who have been required to take courses in statistics designed for their discipline. > If they can’t get the statistics right, how can we expect it from teachers with far less education? This isn’t really a statistics education issue, much like Enron wasn’t an accounting education issue. Social scientists abuse statistics because the incentives are all aligned with abusing statistics. To get a good job you have to publish lots of papers. To publish you have to have statistically significant results. To get into top journals you need surprising results. You see other people in your field playing fast and loose with their data and getting rewarded for it. Why not exclude that one problematic subject from your data analysis? Without them the p-value (from a regression on a carefully chosen eight of your eleven measured variables) drops to .038, and you can publish... It’s not that nobody thought to teach the scientists about corrections for multiple comparisons or the dangers of picking observations to exclude as outliers based on what gives you the result you want. They learned all those things. But it’s so much harder to get results when you’re not willing to play games with the data, and they need results. The people who do try to play by the rules wash out, either by choice after getting fed up with the fraud they see all around them or by failing to publish N papers in top journals.

chongli 3y ago
Why aren't we cleaning house? If the social sciences are roughly comparable to Enron then shouldn't they suffer the same fate?
- CrazyStat 3y ago
  It’s like a country where bribery is endemic. Everybody’s used to it, it’s just the way things work. It’s very hard to stamp that out.
- verall 3y ago
  Enron left millions of people without power while poor social sciences research convinced millions of people that priming is real. They're both wrong but I'm not sure they're so comparable
- mcphage 3y ago
  Because if we did we’d have to admit just how little real information we know in so many fields. We want to think that (as a culture) we’ve got it all just about figured out.
- Mountain_Skies 3y ago
  Most study outcomes have a political cause connected to them, even if that wasn't the author's intent. Those movements want these study outcomes to give their cause legitimacy and the membership of such movements often have a large amount of overlap with those employed in social science academia. It's little different than petrol companies that either fund or amplify studies with outcomes that benefit their business.
- nonameiguess 3y ago
  Because all social sciences are not "roughly comparable to Enron." Hacker News is getting a bit ridiculous with the utter disdain shown for sciences and academia. The problem identified by the parent of this post is a problem of incentives that are misaligned with what we as a larger society want out of basic research. But every energy company had the same incentives as Enron. Not every energy company published fraudulent financial statements. Poor replicability of social sciences is multifactorial. To some extent, what they're studying is a moving target that is not all that amenable to scientific methods, which tend to assume static reductionist laws dictating system behavior. The dynamics of how fundamental forces dictate everything from gravity to covalent molecular bonding don't change from culture to culture as well as over time as trends. When you're studying human behaviors and preferences, I'm sure some of it is governed by more or less immutable eternal laws, but some of it is semi-random diffusion of learned trends and what is true today of one group of people may not be true of any other group or even the same group at some later point in time. Some of it is statistical illiteracy and not understanding the limitations of the techniques you're applying. Some of it is outright fraud. There is also interplay between those two because outright fraud is facilitated by peer reviewers not having the statistical maturity to be able to detect it.
  - CrazyStat 3y ago
    > Hacker News is getting a bit ridiculous with the utter disdain shown for sciences and academia. Just to be clear, nobody actually said the social sciences are roughly equivalent to Enron. I used Enron as a hyperbolic example to emphasize that there are reasons for poor behavior other than lack of education. I was not implying that academic fraud is at the same level as Enron fraud, and after rereading what I wrote I’m comfortable with how I worded it. > But every energy company had the same incentives as Enron. Not every energy company published fraudulent financial statements. Accountants and corporate executives have substantial disincentives against publishing fraudulent financial statements, like going to jail. Academics mostly do not have similar disincentives against abusing statistics. There have been a few high profile embarrassments, but for the most part even people who are widely known to have p-hacked their way to dozens of questionable publications are still sitting comfortably in their tenured professorships.
- Ancapistani 3y ago
  We are! While there are definitely still papers being published as a result of p-hacking (intentionally or not), psychology has been undergoing a renaissance in this area for a few years now. See the "Reproducibility Project: Psychology": https://osf.io/ezcuj/wiki/home/ https://osf.io/ezcuj/wiki/home/ In short, there is a movement to validate past results, starting with those most influential in the field. A lot of progress has been made in that area. Other fields are involved, as well: https://osf.io/collections/rpcb/discover https://osf.io/collections/rpcb/discover
abyssin 3y ago
My anecdotal experience in a psychology faculty makes me think there is an issue of statistics mastery in the social sciences. There's also an issue of corruption like p-hacking, but so far the statistics courses I've had were poor and the teachers didn't really understand what they were teaching.
- jampekka 3y ago
  Yes, but this is not necessarily (only) because of the teacher quality. Statistical testing is quite complicated and very easy to do wrong (violate assumptions). On average scientists aren't very technically or mathematically apt, so in the few short courses things have to be taught in a "cookbook style".
  - lotsofpulp 3y ago
    In the social “sciences”, problem 1, 2 and 3 is that there is no way to measure any of the properties they want to test. Yet, society wants to pretend like we can.
    - jampekka 3y ago
      Short term individual and small group behavior in artificial conditions can be somewhat reliably measured. How well and to what situations these genralize to is a harder question. Larger groups and timescales are (in practice) outside the domain of science (in the most rigorous form) and must be studied with other methods and epistemological criteria. This also leads to major abuse of statistics as the inference has to be done with unrealistic assumptions and unwieldly models. The "brand" of science is so strong that many fields, especially economics, want to appropriate it even though they don't and can't do science in the strict definition.
      - deleted 3y ago
        [deleted]
jampekka 3y ago
(Social) scientists don't necessarily know statistics even though they have had some courses. It's very common e.g. not to understand what p-value actually means. It's disturbingly common that non-linearities or heteroscedasticity are totally ignored in regression. Assumptions of statistical tests are typically not understood. It's largely a cargo cult. For example p-values are reported as t(N) = x, p < threshold but very few understand why and keep on doing it and even demanding it from others. (The why is because the p-values had to be read from tables before computers. It makes no sense nowadays.)
- begemotz 3y ago
  this is not the only reason 'why'. Another persistent reason is the function of the alpha value in typical null hypothesis testing leading to the (misguided) idea that a p value of .04 is functionally equivalent to a p value of 2.3e-10 since both are below threshold. So i would argue that this is more damning - essentially a misunderstanding of probability and what p values tell us at a fundamental level.
  - jampekka 3y ago
    Typically you see p < 0.01 etc even with 0.05 alpha. A lot of stats software gives only those inexact values. But yes, the interpretation of p-values and confidence levels are wildly misunderstood. p > alpha is often taken as "evidence of absence" of an effect, which is just wrong. Or when for some quantity p1 < alpha and other p2 > alpha, it's often intepreted that the quantities differ. It's a mess.

chongli 3y ago

Why aren't we cleaning house? If the social sciences are roughly comparable to Enron then shouldn't they suffer the same fate?

CrazyStat 3y ago
It’s like a country where bribery is endemic. Everybody’s used to it, it’s just the way things work. It’s very hard to stamp that out.
verall 3y ago
Enron left millions of people without power while poor social sciences research convinced millions of people that priming is real. They're both wrong but I'm not sure they're so comparable
mcphage 3y ago
Because if we did we’d have to admit just how little real information we know in so many fields. We want to think that (as a culture) we’ve got it all just about figured out.
Mountain_Skies 3y ago
Most study outcomes have a political cause connected to them, even if that wasn't the author's intent. Those movements want these study outcomes to give their cause legitimacy and the membership of such movements often have a large amount of overlap with those employed in social science academia. It's little different than petrol companies that either fund or amplify studies with outcomes that benefit their business.
nonameiguess 3y ago
Because all social sciences are not "roughly comparable to Enron." Hacker News is getting a bit ridiculous with the utter disdain shown for sciences and academia. The problem identified by the parent of this post is a problem of incentives that are misaligned with what we as a larger society want out of basic research. But every energy company had the same incentives as Enron. Not every energy company published fraudulent financial statements. Poor replicability of social sciences is multifactorial. To some extent, what they're studying is a moving target that is not all that amenable to scientific methods, which tend to assume static reductionist laws dictating system behavior. The dynamics of how fundamental forces dictate everything from gravity to covalent molecular bonding don't change from culture to culture as well as over time as trends. When you're studying human behaviors and preferences, I'm sure some of it is governed by more or less immutable eternal laws, but some of it is semi-random diffusion of learned trends and what is true today of one group of people may not be true of any other group or even the same group at some later point in time. Some of it is statistical illiteracy and not understanding the limitations of the techniques you're applying. Some of it is outright fraud. There is also interplay between those two because outright fraud is facilitated by peer reviewers not having the statistical maturity to be able to detect it.
- CrazyStat 3y ago
  > Hacker News is getting a bit ridiculous with the utter disdain shown for sciences and academia. Just to be clear, nobody actually said the social sciences are roughly equivalent to Enron. I used Enron as a hyperbolic example to emphasize that there are reasons for poor behavior other than lack of education. I was not implying that academic fraud is at the same level as Enron fraud, and after rereading what I wrote I’m comfortable with how I worded it. > But every energy company had the same incentives as Enron. Not every energy company published fraudulent financial statements. Accountants and corporate executives have substantial disincentives against publishing fraudulent financial statements, like going to jail. Academics mostly do not have similar disincentives against abusing statistics. There have been a few high profile embarrassments, but for the most part even people who are widely known to have p-hacked their way to dozens of questionable publications are still sitting comfortably in their tenured professorships.
Ancapistani 3y ago
We are! While there are definitely still papers being published as a result of p-hacking (intentionally or not), psychology has been undergoing a renaissance in this area for a few years now. See the "Reproducibility Project: Psychology": https://osf.io/ezcuj/wiki/home/ https://osf.io/ezcuj/wiki/home/ In short, there is a movement to validate past results, starting with those most influential in the field. A lot of progress has been made in that area. Other fields are involved, as well: https://osf.io/collections/rpcb/discover https://osf.io/collections/rpcb/discover

abyssin 3y ago

My anecdotal experience in a psychology faculty makes me think there is an issue of statistics mastery in the social sciences. There's also an issue of corruption like p-hacking, but so far the statistics courses I've had were poor and the teachers didn't really understand what they were teaching.

jampekka 3y ago
Yes, but this is not necessarily (only) because of the teacher quality. Statistical testing is quite complicated and very easy to do wrong (violate assumptions). On average scientists aren't very technically or mathematically apt, so in the few short courses things have to be taught in a "cookbook style".
- lotsofpulp 3y ago
  In the social “sciences”, problem 1, 2 and 3 is that there is no way to measure any of the properties they want to test. Yet, society wants to pretend like we can.
  - jampekka 3y ago
    Short term individual and small group behavior in artificial conditions can be somewhat reliably measured. How well and to what situations these genralize to is a harder question. Larger groups and timescales are (in practice) outside the domain of science (in the most rigorous form) and must be studied with other methods and epistemological criteria. This also leads to major abuse of statistics as the inference has to be done with unrealistic assumptions and unwieldly models. The "brand" of science is so strong that many fields, especially economics, want to appropriate it even though they don't and can't do science in the strict definition.
    - deleted 3y ago
      [deleted]

jampekka 3y ago

(Social) scientists don't necessarily know statistics even though they have had some courses. It's very common e.g. not to understand what p-value actually means. It's disturbingly common that non-linearities or heteroscedasticity are totally ignored in regression. Assumptions of statistical tests are typically not understood. It's largely a cargo cult. For example p-values are reported as t(N) = x, p < threshold but very few understand why and keep on doing it and even demanding it from others. (The why is because the p-values had to be read from tables before computers. It makes no sense nowadays.)

begemotz 3y ago
this is not the only reason 'why'. Another persistent reason is the function of the alpha value in typical null hypothesis testing leading to the (misguided) idea that a p value of .04 is functionally equivalent to a p value of 2.3e-10 since both are below threshold. So i would argue that this is more damning - essentially a misunderstanding of probability and what p values tell us at a fundamental level.
- jampekka 3y ago
  Typically you see p < 0.01 etc even with 0.05 alpha. A lot of stats software gives only those inexact values. But yes, the interpretation of p-values and confidence levels are wildly misunderstood. p > alpha is often taken as "evidence of absence" of an effect, which is just wrong. Or when for some quantity p1 < alpha and other p2 > alpha, it's often intepreted that the quantities differ. It's a mess.