4 ms·

Summary: one unfalsifiable study, not replicated, with no theoretical basis or reference to other theories; a study unable to establish whether the measured tra

by lutusp 10y ago

Summary: one unfalsifiable study, not replicated, with no theoretical basis or reference to other theories; a study unable to establish whether the measured trait (intellectual humility) and the measured effect (more effective functioning) are in fact related as cause and effect, or the reverse, or are both related to some third unstudied property. In a recent ambitious and well-funded replication project, only 39% of published psychology studies could be replicated at all, and of the successful minority, the average measured effect size was half that of the original study, in some cases falling below generally accepted standards for statistical significance. The take-away from the replication project is that, when reading a random psychology study, the reader must remember that the study's probability of having any relation to reality is less than 50%. More on this topic: http://arachnoid.com/psychology_and_alchemy http://arachnoid.com/psychology_and_alchemy

kem 10y ago
The replication crisis is happening throughout science, including neuroscience, genetics, and oncology, to name a few: http://www.nature.com/nrn/journal/v18/n2/abs/nrn.2016.167.html http://www.nature.com/nrn/journal/v18/n2/abs/nrn.2016.167.ht... http://onlinelibrary.wiley.com/doi/10.1111/j.1360-0443.2009.02662.x/full http://onlinelibrary.wiley.com/doi/10.1111/j.1360-0443.2009.... http://www.slate.com/articles/health_and_science/future_tense/2016/04/biomedicine_facing_a_worse_replication_crisis_than_the_one_plaguing_psychology.html http://www.slate.com/articles/health_and_science/future_tens... In fact, some analyses suggest replication might be worse in the neurosciences than in psychology. The neurosciences are plagued by many of the same conceptual ambiguities in psychology, but with even smaller, expensive studies that are underpowered with more incentives to avoid sunk cost problems. The most accurate summary is that replicability problems affect the biomedical sciences the most in general (empirically speaking, psychology tends to be classified as a biomedical science, in topic analysis studies and so forth). Follow the money. You have lots of competition for limited resources, lots of money on the line, and soft or absent tenure protections. The link you provide amounts to nothing more than ignorant intellectual bigotry and killing the messenger. It's ironically sad, in fact, because psychologists and behavioral scientists are the ones exposing the replication crisis and doing the most to try to address the problem. The type of rhetoric in the linked essay amounts to a whitewashing of replicability problems in other areas, by denying the problem exists and is only true of "lesser sciences." The sort of defensive response represented by the linked essay allows scientists in other areas to avoid accountability and the threatening implications it has for other fields. Levels of analysis problems happen in all sorts of intellectual endeavors. Arguing that psychology will give way to the neurosciences is akin to arguing that nothing in the computer sciences is a legitimate topic of study unless it involves bare metal issues. Having said that, I also distrust this paper.
- lutusp 10y ago
  > It's ironically sad, in fact, because psychologists and behavioral scientists are the ones exposing the replication crisis and doing the most to try to address the problem. On the contrary, psychologists were shamed into addressing the replication problem by a Nobel Prizewinner's now-famous "train wreck looming" open letter[1]. They certainly wouldn't have gone there without being pushed. > Arguing that psychology will give way to the neurosciences is akin to arguing that nothing in the computer sciences is a legitimate topic of study unless it involves bare metal issues. Yes, unless the computer science theories under study are falsifiable and based on empirical evidence, in which case they constitute science as science is defined. That's what distinguishes computer science from psychology, or, for that matter, science from psychology. > The link you provide amounts to nothing more than ignorant intellectual bigotry and killing the messenger. I include this to show the essence and basic character of your argument. 1. http://www.chronicle.com/blogs/percolator/daniel-kahneman-sees-train-wreck-looming-for-social-psychology/31338 http://www.chronicle.com/blogs/percolator/daniel-kahneman-se...
  - WaxProlix 10y ago
    I hesitate to even respond to someone who's possibly just a highly energetic troll, but come on dude. The reason Kahneman wrote about psych is that it's the field he works in. He's not going to see these symptoms in ecological or environmental sciences because he doesn't read those papers, preside over those theses, vie for those grants, etc. > On the contrary, psychologists were shamed into addressing the replication problem It was raised by a professor of psychology, somebody IN THE COMMUNITY, and is being addressed, better than elsewhere. Your spin is absurd. A community takes some monumentally difficult steps to right itself in the face of bizarre pressures (mainly a broken funding model) and you take it as more evidence that they're inherently dishonest. The article you link here seems to go almost completely against your premise, by the way. If your goal is to make it obvious that some people have an absurd bias against the 'soft sciences', or to reinforce the stereotype of 'STEMlord' on HN, you're doing marvelously. Anything else and it's a flop.
- bobcostas55 10y ago
  I took a look at the genetics paper, and while it's true that the "candidate gene" approach led to a lot of false results, it should be noted that that paper is from 8 years ago and methodologies in genetics have changed _completely_ since then. The same can't be said for psychology.
rmxt 10y ago
It's clear upon clicking the link or viewing your HN profile, but in the interests of transparency, it might be useful to note that the link is to your own article and website.
xelxebar 10y ago
Kind of feel that replication study gets abused as a kind of "psychology is unreliable" argument. Instead, I think it makes a lot more sense to view it as an upper bound on the average reliability of our own personal beliefs about human behaviour. I mean, most of our day to day cognating about other people is fraught with unfalsifiable claims, dubious reasoning, and horrendous sample sizes. Takeaway: Be extra skeptical about psychology research claims and triply so about your own beliefs regarding people, including yourself.
- lutusp 10y ago
  > Kind of feel that replication study gets abused as a kind of "psychology is unreliable" argument. Since the study demonstrates that psychology is unreliable, I think it's something other than abuse. > I mean, most of our day to day cognating about other people is fraught with unfalsifiable claims, dubious reasoning, and horrendous sample sizes. Yes, but we don't call that science or open clinics to treat people who also behave that way. Psychologists do.
  - xelxebar 10y ago
    > Since the study demonstrates that psychology is unreliable, I think it's something other than abuse. Your comment is exactly my point though. I suspect that you see the number 30% and equate that with "bad". However, if 30% is more reliable on average than our own personal heuristics, then doesn't it seems shortsighted to write off every bit of psychology research as "bad science"?
coolsdude2282 10y ago
I evangelically bring this up constantly in day to day life, and people just outright refuse to accept that psychology isn't rigorous. This cannot be reiterated enough, if anyone reading this is unfamiliar with the current replication crisis, please stop whatever you are doing and read up on it. It changed the way I view all social sciences.
- coldtea 10y ago
  >It changed the way I view all social sciences. It should have changed the way you view all sciences, period, because the replication crisis concerns biology, medicine, physics, computer science etc equally. In fact some of the most powerful studies on the subject have been on the topic of biology.
  - therpe1 10y ago
    It's so striking how people who view the "soft sciences" this way really have no understanding of science at all. It's a very American thing and it really reveals the extent to which science has been socially constructed in America to be some kind of infallible truth machine. The hard-on for infallibility is so striking -- there's something about sociality, uncertainty, nuance, luck, and fallibility that really terrifies these people on a foundational level.
    - pottersbasilisk 10y ago
      Science has replaced religion for many people and so they expect something infallible or "word of god".
    - coldtea 10y ago
      >there's something about sociality, uncertainty, nuance, luck, and fallibility that really terrifies these people on a foundational level. It's the old protestant/puritan spirit, just moved from the certainty of God to another certainty (for all the lip service to the "scientific process", it's the certainty that's valued most in science).
dang 10y ago
Your hundreds of comments on this, all saying the same thing and replicating themselves like weeds in any remotely related context, have become so excessive that I need to ask you to stop using HN this way. We all respect your fabulous software work, but HN threads are for conversation. Nothing spoils good conversation like obsessive repetition. Taken as a whole, your comments on this matter are not conversation, they are a harangue—apparently an interminable one. Harangue is off-topic here, regardless of whether you're right or not.
- lutusp 10y ago
  > Your hundreds of comments on this ... You know, starting out with a wild exaggeration, an "alternative fact," undermines your otherwise meritorious point. > ... I need to ask you to stop using HN this way. In what way -- conversations about psychology? On what basis? HN regulars often post links to psychology articles -- people don't complain about that (and IMHO they shouldn't, because most are newsworthy). And by definition, comments about those articles are as topical as the articles themselves. Also, I never originate exchanges about psychology, I only respond to threads introduced/posted by others. If psychology were never an established topic here, I would never address it. This means you're objecting to a particular viewpoint on the topic, not the topic itself. > We all respect your fabulous software work, but HN threads are for conversation. Indeed they are. Case in point -- this conversation. Solely for balance you might consider objecting to the large number of psychology articles posted here. I'm not recommending this -- I only say this to try to get you to examine your position.
  - DanBC 10y ago
    > You know, starting out with a wild exaggeration, https://hn.algolia.com/?query=author:lutusp%20psychology&sort=byPopularity&prefix&page=0&dateRange=all&type=comment https://hn.algolia.com/?query=author:lutusp%20psychology&sor... Over 300 results, all saying pretty much the same thing.
    - lutusp 10y ago
      > all saying pretty much the same thing. The number is higher than I would have guessed, but they're certainly not saying "the same thing." In many cases they're included in the search results only because of the presence of the word "psychology" without any discussion of the topic at all.

by lutusp 10y ago

kem 10y ago

The replication crisis is happening throughout science, including neuroscience, genetics, and oncology, to name a few: http://www.nature.com/nrn/journal/v18/n2/abs/nrn.2016.167.html http://www.nature.com/nrn/journal/v18/n2/abs/nrn.2016.167.ht... http://onlinelibrary.wiley.com/doi/10.1111/j.1360-0443.2009.02662.x/full http://onlinelibrary.wiley.com/doi/10.1111/j.1360-0443.2009.... http://www.slate.com/articles/health_and_science/future_tense/2016/04/biomedicine_facing_a_worse_replication_crisis_than_the_one_plaguing_psychology.html http://www.slate.com/articles/health_and_science/future_tens... In fact, some analyses suggest replication might be worse in the neurosciences than in psychology. The neurosciences are plagued by many of the same conceptual ambiguities in psychology, but with even smaller, expensive studies that are underpowered with more incentives to avoid sunk cost problems. The most accurate summary is that replicability problems affect the biomedical sciences the most in general (empirically speaking, psychology tends to be classified as a biomedical science, in topic analysis studies and so forth). Follow the money. You have lots of competition for limited resources, lots of money on the line, and soft or absent tenure protections. The link you provide amounts to nothing more than ignorant intellectual bigotry and killing the messenger. It's ironically sad, in fact, because psychologists and behavioral scientists are the ones exposing the replication crisis and doing the most to try to address the problem. The type of rhetoric in the linked essay amounts to a whitewashing of replicability problems in other areas, by denying the problem exists and is only true of "lesser sciences." The sort of defensive response represented by the linked essay allows scientists in other areas to avoid accountability and the threatening implications it has for other fields. Levels of analysis problems happen in all sorts of intellectual endeavors. Arguing that psychology will give way to the neurosciences is akin to arguing that nothing in the computer sciences is a legitimate topic of study unless it involves bare metal issues. Having said that, I also distrust this paper.

lutusp 10y ago
> It's ironically sad, in fact, because psychologists and behavioral scientists are the ones exposing the replication crisis and doing the most to try to address the problem. On the contrary, psychologists were shamed into addressing the replication problem by a Nobel Prizewinner's now-famous "train wreck looming" open letter[1]. They certainly wouldn't have gone there without being pushed. > Arguing that psychology will give way to the neurosciences is akin to arguing that nothing in the computer sciences is a legitimate topic of study unless it involves bare metal issues. Yes, unless the computer science theories under study are falsifiable and based on empirical evidence, in which case they constitute science as science is defined. That's what distinguishes computer science from psychology, or, for that matter, science from psychology. > The link you provide amounts to nothing more than ignorant intellectual bigotry and killing the messenger. I include this to show the essence and basic character of your argument. 1. http://www.chronicle.com/blogs/percolator/daniel-kahneman-sees-train-wreck-looming-for-social-psychology/31338 http://www.chronicle.com/blogs/percolator/daniel-kahneman-se...
- WaxProlix 10y ago
  I hesitate to even respond to someone who's possibly just a highly energetic troll, but come on dude. The reason Kahneman wrote about psych is that it's the field he works in. He's not going to see these symptoms in ecological or environmental sciences because he doesn't read those papers, preside over those theses, vie for those grants, etc. > On the contrary, psychologists were shamed into addressing the replication problem It was raised by a professor of psychology, somebody IN THE COMMUNITY, and is being addressed, better than elsewhere. Your spin is absurd. A community takes some monumentally difficult steps to right itself in the face of bizarre pressures (mainly a broken funding model) and you take it as more evidence that they're inherently dishonest. The article you link here seems to go almost completely against your premise, by the way. If your goal is to make it obvious that some people have an absurd bias against the 'soft sciences', or to reinforce the stereotype of 'STEMlord' on HN, you're doing marvelously. Anything else and it's a flop.
bobcostas55 10y ago
I took a look at the genetics paper, and while it's true that the "candidate gene" approach led to a lot of false results, it should be noted that that paper is from 8 years ago and methodologies in genetics have changed _completely_ since then. The same can't be said for psychology.

rmxt 10y ago

It's clear upon clicking the link or viewing your HN profile, but in the interests of transparency, it might be useful to note that the link is to your own article and website.

xelxebar 10y ago

Kind of feel that replication study gets abused as a kind of "psychology is unreliable" argument. Instead, I think it makes a lot more sense to view it as an upper bound on the average reliability of our own personal beliefs about human behaviour. I mean, most of our day to day cognating about other people is fraught with unfalsifiable claims, dubious reasoning, and horrendous sample sizes. Takeaway: Be extra skeptical about psychology research claims and triply so about your own beliefs regarding people, including yourself.

lutusp 10y ago
> Kind of feel that replication study gets abused as a kind of "psychology is unreliable" argument. Since the study demonstrates that psychology is unreliable, I think it's something other than abuse. > I mean, most of our day to day cognating about other people is fraught with unfalsifiable claims, dubious reasoning, and horrendous sample sizes. Yes, but we don't call that science or open clinics to treat people who also behave that way. Psychologists do.
- xelxebar 10y ago
  > Since the study demonstrates that psychology is unreliable, I think it's something other than abuse. Your comment is exactly my point though. I suspect that you see the number 30% and equate that with "bad". However, if 30% is more reliable on average than our own personal heuristics, then doesn't it seems shortsighted to write off every bit of psychology research as "bad science"?

coolsdude2282 10y ago

I evangelically bring this up constantly in day to day life, and people just outright refuse to accept that psychology isn't rigorous. This cannot be reiterated enough, if anyone reading this is unfamiliar with the current replication crisis, please stop whatever you are doing and read up on it. It changed the way I view all social sciences.

coldtea 10y ago
>It changed the way I view all social sciences. It should have changed the way you view all sciences, period, because the replication crisis concerns biology, medicine, physics, computer science etc equally. In fact some of the most powerful studies on the subject have been on the topic of biology.
- therpe1 10y ago
  It's so striking how people who view the "soft sciences" this way really have no understanding of science at all. It's a very American thing and it really reveals the extent to which science has been socially constructed in America to be some kind of infallible truth machine. The hard-on for infallibility is so striking -- there's something about sociality, uncertainty, nuance, luck, and fallibility that really terrifies these people on a foundational level.
  - pottersbasilisk 10y ago
    Science has replaced religion for many people and so they expect something infallible or "word of god".
  - coldtea 10y ago
    >there's something about sociality, uncertainty, nuance, luck, and fallibility that really terrifies these people on a foundational level. It's the old protestant/puritan spirit, just moved from the certainty of God to another certainty (for all the lip service to the "scientific process", it's the certainty that's valued most in science).

dang 10y ago

Your hundreds of comments on this, all saying the same thing and replicating themselves like weeds in any remotely related context, have become so excessive that I need to ask you to stop using HN this way. We all respect your fabulous software work, but HN threads are for conversation. Nothing spoils good conversation like obsessive repetition. Taken as a whole, your comments on this matter are not conversation, they are a harangue—apparently an interminable one. Harangue is off-topic here, regardless of whether you're right or not.

lutusp 10y ago
> Your hundreds of comments on this ... You know, starting out with a wild exaggeration, an "alternative fact," undermines your otherwise meritorious point. > ... I need to ask you to stop using HN this way. In what way -- conversations about psychology? On what basis? HN regulars often post links to psychology articles -- people don't complain about that (and IMHO they shouldn't, because most are newsworthy). And by definition, comments about those articles are as topical as the articles themselves. Also, I never originate exchanges about psychology, I only respond to threads introduced/posted by others. If psychology were never an established topic here, I would never address it. This means you're objecting to a particular viewpoint on the topic, not the topic itself. > We all respect your fabulous software work, but HN threads are for conversation. Indeed they are. Case in point -- this conversation. Solely for balance you might consider objecting to the large number of psychology articles posted here. I'm not recommending this -- I only say this to try to get you to examine your position.
- DanBC 10y ago
  > You know, starting out with a wild exaggeration, https://hn.algolia.com/?query=author:lutusp%20psychology&sort=byPopularity&prefix&page=0&dateRange=all&type=comment https://hn.algolia.com/?query=author:lutusp%20psychology&sor... Over 300 results, all saying pretty much the same thing.
  - lutusp 10y ago
    > all saying pretty much the same thing. The number is higher than I would have guessed, but they're certainly not saying "the same thing." In many cases they're included in the search results only because of the presence of the word "psychology" without any discussion of the topic at all.