5 ms·

In a similar vein, UPenn professor Angela Duckworth of "Grit" fame was caught misrepresenting statistics to embellish her claims a few years ago: https://www.np

by ipsum2 3y ago

In a similar vein, UPenn professor Angela Duckworth of "Grit" fame was caught misrepresenting statistics to embellish her claims a few years ago: https://www.npr.org/sections/ed/2016/05/25/479172868/angela-duckworth-responds-to-a-new-critique-of-grit https://www.npr.org/sections/ed/2016/05/25/479172868/angela-.... Who knew psychology had such questionable science?

marginalia_nu 3y ago
> Who knew psychology had such questionable science? Well given psychology is one of the two epicenters for the replication crisis, I didn't think this should be too unexpected.
- tgv 3y ago
  It is to the merit of psychology. Other branches of social sciences don't dare to propose such a thing. They know that their replication numbers are going to be worse than psychology's.
  - qsort 3y ago
    Most social sciences have numbers that are far better than psychology's. https://royalsocietypublishing.org/doi/10.1098/rsos.200566 https://royalsocietypublishing.org/doi/10.1098/rsos.200566 For fuck's sake, marketing and management has better numbers than psychology. It is to their merit they're admitting they have a problem, but it is 100% a problem of the field of study.
    - begemotz 3y ago
      You misrepresent the linked article which doesn't report replication rates but forecasted replication rates from participants "recruited through a number of mailing lists, Twitter and blog posts".
      - qsort 3y ago
        They used a prediction market, which at the time the article was published was still to resolve. This is standard practice and methodologically sound. They didn't go around asking people on Twitter.
        _Nat_ 3y ago
        Just skimmed it quickly, but from [their paper](https://royalsocietypublishing.org/doi/10.1098/rsos.200566 https://royalsocietypublishing.org/doi/10.1098/rsos.200566 ): > Participants were recruited through a number of mailing lists, Twitter and blog posts. [...] For all analyses, we use survey responses; for analysis (iii), we additionally use responses to a demographic survey that included a question on academic interests; and for analysis (iv), we additionally use the prediction market data. So it sounds like they only used prediction-market data "for analysis (iv)", which they describe as: > [...] (iv) whether survey-based aggregated forecasts and market-based forecasts are correlated. --- The last line of their paper: > If our forecasts hold up, it will be interesting to investigate if specific factors (such as different methodologies and policies) can be identified that influence replication rates. So, it doesn't sound like they're claiming that their forecasts should be assumed reliable.
        tgv 3y ago
        No, it isn't. A prediction market is not proof of anything.
        begemotz 3y ago
        I didn't say that the methodology was not standard practice or methodologically sound. I said you misrepresented the study, which you did. You said "Most social sciences have numbers that are far better than psychology's." in the context of a conversation about *replication numbers*. You did not say "in a prediction market, participants ("recruited through a number of mailing lists, Twitter and blog posts") have lower replication expectation rates for Psychology than other social sciences" -- if you had then, well, you wouldn't have misrepresented the study.
        3y ago
- implements 3y ago
  Not in the field, but this recent podcast covers it: https://podcasts.apple.com/gb/podcast/two-psychologists-four-beers/id1387529624?i=1000618107833 https://podcasts.apple.com/gb/podcast/two-psychologists-four... Harkening back to episode 73, Alexa and Yoel discuss recent evidence of fraud documented in the Data Colada blog post "Clusterfake." The post is the first in a series of four, which will collectively detail evidence of fraud in four papers co-authored by Harvard Business School Professor Francesca Gino. […] Finally, they consider what this means for a field still struggling to build a more trustworthy foundation.
- HDMI_Cable 3y ago
  What's the other epicentre?
  - marginalia_nu 3y ago
    They've had a bunch of trouble in medicine-adjacent fields. Nutrition and sports science has had an especially tough time, but it's been cropping up all over the field. I do think most of the problem is rooted in the fact that it's difficult to do this type of science, especially without serious ethical breaches. There are so many confounding variables, and self-reported observational data just isn't a very good tool. To truly eliminate confounders, you basically need to lock someone up for decades and control their entire lifestyle.
tarvaina 3y ago
Based on the NPR article you linked this sounds like a normal scientific debate, though? One researcher says that data supports their theory, another says that it doesn't if you look at the data more carefully. I don't see accusations of fraud here, just different interpretation of facts, and a valid challenge to an earlier theory. This is how science is supposed to work, isn't it? Gino, on the other hand, is accused of outright fraud. Data Colada claims she took real data and changed it to support her theory, or possibly created entirely fake data.
brmgb 3y ago
I mean, people don't like to say it because you get taxed of being snobbish at best or lacking culture and being close-minded at worst but everyone know that "soft" sciences are mostly bullshit. I started my eductation with a bachelor of maths and an engineering degree before doing a lot of economics while in business school. The field is full of interesting ideas but methodologically it mostly is garbage even you stick to the more mathematically oriented part of it. And that's economy which is supposed to be the most serious of the soft science.
- twobitshifter 3y ago
  Agreed, I have a similar background. Science is done in economics, but a lot of economics ends up being thought experiments turned into a graph eventually taken as fact. Once you get into the science of the shapes of everyones’ multi dimensional personal utility curves and rational behavior you are drifting from actual observed behavior. That’s the pivot to the other side of irrational behavior that Dan Ariely loved to push based on observations, but now we find that’s full of false claims.
brezelnbitte 3y ago
Its not just psychology. All science is affected. Stuart Ritchie’s Science Fictions describes cases from across many disciplines. The worst are cases of misconduct in medical research with maybe the worst case being Paolo Macchiarini who fooled the Karolinska Institute of Nobel fame into letting him experiment on patients with trachea replacement surgeries: https://news.ki.se/the-macchiarini-case-timeline https://news.ki.se/the-macchiarini-case-timeline

by ipsum2 3y ago

marginalia_nu 3y ago

> Who knew psychology had such questionable science? Well given psychology is one of the two epicenters for the replication crisis, I didn't think this should be too unexpected.

tgv 3y ago
It is to the merit of psychology. Other branches of social sciences don't dare to propose such a thing. They know that their replication numbers are going to be worse than psychology's.
- qsort 3y ago
  Most social sciences have numbers that are far better than psychology's. https://royalsocietypublishing.org/doi/10.1098/rsos.200566 https://royalsocietypublishing.org/doi/10.1098/rsos.200566 For fuck's sake, marketing and management has better numbers than psychology. It is to their merit they're admitting they have a problem, but it is 100% a problem of the field of study.
  - begemotz 3y ago
    You misrepresent the linked article which doesn't report replication rates but forecasted replication rates from participants "recruited through a number of mailing lists, Twitter and blog posts".
    - qsort 3y ago
      They used a prediction market, which at the time the article was published was still to resolve. This is standard practice and methodologically sound. They didn't go around asking people on Twitter.
      - _Nat_ 3y ago
        Just skimmed it quickly, but from [their paper](https://royalsocietypublishing.org/doi/10.1098/rsos.200566 https://royalsocietypublishing.org/doi/10.1098/rsos.200566 ): > Participants were recruited through a number of mailing lists, Twitter and blog posts. [...] For all analyses, we use survey responses; for analysis (iii), we additionally use responses to a demographic survey that included a question on academic interests; and for analysis (iv), we additionally use the prediction market data. So it sounds like they only used prediction-market data "for analysis (iv)", which they describe as: > [...] (iv) whether survey-based aggregated forecasts and market-based forecasts are correlated. --- The last line of their paper: > If our forecasts hold up, it will be interesting to investigate if specific factors (such as different methodologies and policies) can be identified that influence replication rates. So, it doesn't sound like they're claiming that their forecasts should be assumed reliable.
      - tgv 3y ago
        No, it isn't. A prediction market is not proof of anything.
      - begemotz 3y ago
        I didn't say that the methodology was not standard practice or methodologically sound. I said you misrepresented the study, which you did. You said "Most social sciences have numbers that are far better than psychology's." in the context of a conversation about *replication numbers*. You did not say "in a prediction market, participants ("recruited through a number of mailing lists, Twitter and blog posts") have lower replication expectation rates for Psychology than other social sciences" -- if you had then, well, you wouldn't have misrepresented the study.
        3y ago
implements 3y ago
Not in the field, but this recent podcast covers it: https://podcasts.apple.com/gb/podcast/two-psychologists-four-beers/id1387529624?i=1000618107833 https://podcasts.apple.com/gb/podcast/two-psychologists-four... Harkening back to episode 73, Alexa and Yoel discuss recent evidence of fraud documented in the Data Colada blog post "Clusterfake." The post is the first in a series of four, which will collectively detail evidence of fraud in four papers co-authored by Harvard Business School Professor Francesca Gino. […] Finally, they consider what this means for a field still struggling to build a more trustworthy foundation.
HDMI_Cable 3y ago
What's the other epicentre?
- marginalia_nu 3y ago
  They've had a bunch of trouble in medicine-adjacent fields. Nutrition and sports science has had an especially tough time, but it's been cropping up all over the field. I do think most of the problem is rooted in the fact that it's difficult to do this type of science, especially without serious ethical breaches. There are so many confounding variables, and self-reported observational data just isn't a very good tool. To truly eliminate confounders, you basically need to lock someone up for decades and control their entire lifestyle.

tarvaina 3y ago

Based on the NPR article you linked this sounds like a normal scientific debate, though? One researcher says that data supports their theory, another says that it doesn't if you look at the data more carefully. I don't see accusations of fraud here, just different interpretation of facts, and a valid challenge to an earlier theory. This is how science is supposed to work, isn't it? Gino, on the other hand, is accused of outright fraud. Data Colada claims she took real data and changed it to support her theory, or possibly created entirely fake data.

brmgb 3y ago

I mean, people don't like to say it because you get taxed of being snobbish at best or lacking culture and being close-minded at worst but everyone know that "soft" sciences are mostly bullshit. I started my eductation with a bachelor of maths and an engineering degree before doing a lot of economics while in business school. The field is full of interesting ideas but methodologically it mostly is garbage even you stick to the more mathematically oriented part of it. And that's economy which is supposed to be the most serious of the soft science.

twobitshifter 3y ago
Agreed, I have a similar background. Science is done in economics, but a lot of economics ends up being thought experiments turned into a graph eventually taken as fact. Once you get into the science of the shapes of everyones’ multi dimensional personal utility curves and rational behavior you are drifting from actual observed behavior. That’s the pivot to the other side of irrational behavior that Dan Ariely loved to push based on observations, but now we find that’s full of false claims.

brezelnbitte 3y ago

Its not just psychology. All science is affected. Stuart Ritchie’s Science Fictions describes cases from across many disciplines. The worst are cases of misconduct in medical research with maybe the worst case being Paolo Macchiarini who fooled the Karolinska Institute of Nobel fame into letting him experiment on patients with trachea replacement surgeries: https://news.ki.se/the-macchiarini-case-timeline https://news.ki.se/the-macchiarini-case-timeline