6 ms·

> This is the process working. This is all okay, and welcome No! This sort of thing casts enormous doubt on the field of psychology and many soft sciences as a

> This is the process working. This is all okay, and welcome No! This sort of thing casts enormous doubt on the field of psychology and many soft sciences as a whole. A lot of it is academic politics and people being afraid of calling things out in fear of hurting their career prospects. It's much easier to just stay silent, ignore it, and play along. Who knows who will sit on your next grant proposal's committee. Academia is full of this and it's a disgrace.

FredPret 5y ago
I, for one, will never trust another “finding” coming out of the field of psychology. Between this and the marshmallow study, I think psychology should be considered an art. We’re many centuries away from this progressing to being a science on par with physics. There’s nothing wrong with that, a wise therapist has much to contribute - but not to science.
- lotsofpulp 5y ago
  This xkcd is relevant: https://xkcd.com/435/ https://xkcd.com/435/ Unless the variables in the experiment are decently quantifiable. It is a garbage in, garbage out situation.
  - FredPret 5y ago
    The problem with psychology is: the subject matter is the most complex system we know of. In order to make definitive statements about said system, you need to start with the most basic behaviours and characteristics. As you work your way up, you’ll mountains of data to capture every permutation of human behaviour and emotion. We’re so, so far from any of that.
    - naasking 5y ago
      The messier the system, the more rigorous the methodology should be. Pre-registering trials to fight p-hacking, and open data should be the norm. That's decidedly not what we've gotten from this field, and some prominent figures have even been fighting such measures and consider the reproducibility crisis to not be a big deal. That's the problem.
  - ramblenode 5y ago
    More often than not the constructs in the garbage studies are quantifiable, just useless. The bigger problem is the "what" of what is being measured---whether that thing is representative, useful, or important. Part of the reason bad research has the veneer of authenticity is that the numbers, graphs, and statistics dress up the nonsense.
    - lotsofpulp 5y ago
      Yes, that is the distinction I was trying to make by writing "decently quantifiable".
- bee_rider 5y ago
  There's an unfortunate tendency to treat science as the only source of truth. Really, psychology (like engineering and medicine) is a profession. Professions aren't just based on the scientific method -- they draw from a semi-formal pile of experience known as "best practices." These are often just based on something that a clever person decided to try decades or centuries ago which seems to have not hurt anyone. Of course it is an iterative process so sometimes these things are overturned, but the process seems to work allright.
  - FredPret 5y ago
    Spot on. I’ve met many psychologists who struck me as kind, wise people with a helpful perspective. Society needs them. But to plug this fuzzy, professional knowledge into equations, call it science, and use it to make decisions in business and policy is at best naive and at worst fraud.
  - naasking 5y ago
    How could you possibly know that the "process seems to work alright" without scientific evidence that it's doing so? You actually have no idea in that case whether you're doing harm or helping, and if you're somehow helping, it could just be a very expensive placebo.
    - bee_rider 5y ago
      I mean, I don't work in mental health, but if somebody has mental health issues that they are dealing with, and they buy an "expensive placebo" in the form of talking about their problems with a professional, and it helps... that seems like a successful treatment, right? I'm actually not even sure how to define placebo in this context.
      - naasking 5y ago
        I think it's important to focus on "expensive" and not "placebo" in that statement. If a placebo is all that's needed (or all that we can do), then arguably it shouldn't be expensive.
- lrem 5y ago
  Marshmallow study? Is the finding about "wait 5 minutes to get two sweets - success in life" debunked, or are you referring to another thing?
  - quag 5y ago
    Yes, it has been debunked. They got the causal arrow around the wrong way. It turns out that kids who are in stable situations with trustworthy adults around them, tend to trust adults more when they say “wait 5 minutes and I’ll give you more”, and that kids in bad situations don’t trust adults. Turns out being in a bad situation as a kid does have a long term effect on your life. Nothing to do with the ability of people delay gratification.
- TameAntelope 5y ago
  If what you've read here concerns you to the point of throwing out the field, you're going to be immensely disappointed by every other scientific field, as well. [0] I'm not an expert, but my understanding is that this article could have been written about literally any field of science, with a depressingly few nouns changed. [1] This is what people mean when they say, "science is messy". This is how the sausage gets made, and if you're cool throwing out psychology as a field, you're probably also going to have to be cool throwing out a lot of other fields. [2] There are efforts being made to clean up how science is conducted, called "metascience", but it's a relatively new field and there's still a lot that hasn't been investigated yet. [3] Basically, if you've ever thought of yourself as "pro science", you've been supporting what happened in this article, whether you realized it or not. [0] https://www.nature.com/articles/533452a https://www.nature.com/articles/533452a [1] https://www.scientificamerican.com/video/is-there-a-reproducibility-crisis-in-science/ https://www.scientificamerican.com/video/is-there-a-reproduc... -- a slightly more accessible version of [0] [2] https://direct.mit.edu/posc/article/28/4/482/97500/Mess-in-Science-and-Wicked-Problems https://direct.mit.edu/posc/article/28/4/482/97500/Mess-in-S... [3] https://en.wikipedia.org/wiki/Metascience https://en.wikipedia.org/wiki/Metascience
  - fhood 5y ago
    I think we can maybe inject a little bit of extra nuance here. Not every field is the same, they vary massively in size, level of external influence, and complexity in subject matter. Psychology suffers more than most. It's a large field, so competition to get noticed is likely high. There is money to be made, as discussed in the article, so it struggles with influences outside of pure scientific inquiry, and they have no chance of describing their theories in terms of pure component mathematics. Contrast that with ocean and atmospheric sciences. It is a tiny and very insular field, so the rat race works very differently. This also protects them from outside influences, and a lot of the major underlying physical and biological processes involved can be modeled with mathematics that have been demonstrated to be predictive. It's just easier for some fields to reliably produce higher quality work. I think this is a case where blanket skepticism is just lazy. Every field suffers some level of a reproducibility issue, but some fields are better able to self select for a variety of reasons.
  - 5y ago
- lemmsjid 5y ago
  I won't argue with your mistrust of psychological 'findings'. But: If psychological research is not conducted with scientific rigor, and we treat it as an art, then where does that leave the phenomenology of mental illness, study of social dynamics, etc? Mentally ill people still need to be treated. And historically they have been treated in very, let us say, 'artistic' ways. If we abandon the scientific aspects of psychology, mental illness will still need to be dealt with, but without any framework of rigor or understanding. While landmark findings of psychology (such as the marshmallow experiment) can be questioned and debunked, a lot of what you'll learn when studying psychology as a science is how little we know about psychology. That's an important thing to learn! Does approach X work for people with debilitating anxiety? In all probability, we don't know, or it does work for some people and has the opposite effect on others. That's the day to day of psychological research. When something exciting and novel comes up (like the marshmallow thing, or the Stanford Prison Experiment), it represents an island of false certainty in a sea of uncertainty and suddenly all the laypeople know about it. Such understanding of our lack of understanding can form a bulwark against, at the very least, political, social and religious ideologies that make sweeping claims about behavior and dynamics. It's important to quantify what we don't know as well as what we know. Otherwise we'll be taken over by people making extraordinary and sweeping claims based on no evidence, rather than people at least publishing their evidence to be possibly later debunked. In the linked article's example, if psychology were purely art, there would only be the authors' book, not a paper with actual math to debunk and a journal to interact with. I am probably overstating my point, because when a specious claim is backed up by scientific trappings people may be more likely to believe it, but I am increasingly less certain of that. When it comes to critiques of 'soft sciences' I often come to this. Those fields often came into being because there's actual day to day issues that don't go away even if we have arguable theoretical foundations behind them. Absent said foundations, people still need to make sweeping decisions about economics and psychology that affect millions of lives. The scientific part arose from looking at the history of such interventions and wondering if things like exorcising people with visions actually worked any better than not exorcising them. For many of the people being exorcised, it was a 'landmark finding' that exorcism is not the most effective method of helping them with their issues.
  - creato 5y ago

FredPret 5y ago

I, for one, will never trust another “finding” coming out of the field of psychology. Between this and the marshmallow study, I think psychology should be considered an art. We’re many centuries away from this progressing to being a science on par with physics. There’s nothing wrong with that, a wise therapist has much to contribute - but not to science.

lotsofpulp 5y ago
This xkcd is relevant: https://xkcd.com/435/ https://xkcd.com/435/ Unless the variables in the experiment are decently quantifiable. It is a garbage in, garbage out situation.
- FredPret 5y ago
  The problem with psychology is: the subject matter is the most complex system we know of. In order to make definitive statements about said system, you need to start with the most basic behaviours and characteristics. As you work your way up, you’ll mountains of data to capture every permutation of human behaviour and emotion. We’re so, so far from any of that.
  - naasking 5y ago
    The messier the system, the more rigorous the methodology should be. Pre-registering trials to fight p-hacking, and open data should be the norm. That's decidedly not what we've gotten from this field, and some prominent figures have even been fighting such measures and consider the reproducibility crisis to not be a big deal. That's the problem.
- ramblenode 5y ago
  More often than not the constructs in the garbage studies are quantifiable, just useless. The bigger problem is the "what" of what is being measured---whether that thing is representative, useful, or important. Part of the reason bad research has the veneer of authenticity is that the numbers, graphs, and statistics dress up the nonsense.
  - lotsofpulp 5y ago
    Yes, that is the distinction I was trying to make by writing "decently quantifiable".
bee_rider 5y ago
There's an unfortunate tendency to treat science as the only source of truth. Really, psychology (like engineering and medicine) is a profession. Professions aren't just based on the scientific method -- they draw from a semi-formal pile of experience known as "best practices." These are often just based on something that a clever person decided to try decades or centuries ago which seems to have not hurt anyone. Of course it is an iterative process so sometimes these things are overturned, but the process seems to work allright.
- FredPret 5y ago
  Spot on. I’ve met many psychologists who struck me as kind, wise people with a helpful perspective. Society needs them. But to plug this fuzzy, professional knowledge into equations, call it science, and use it to make decisions in business and policy is at best naive and at worst fraud.
- naasking 5y ago
  How could you possibly know that the "process seems to work alright" without scientific evidence that it's doing so? You actually have no idea in that case whether you're doing harm or helping, and if you're somehow helping, it could just be a very expensive placebo.
  - bee_rider 5y ago
    I mean, I don't work in mental health, but if somebody has mental health issues that they are dealing with, and they buy an "expensive placebo" in the form of talking about their problems with a professional, and it helps... that seems like a successful treatment, right? I'm actually not even sure how to define placebo in this context.
    - naasking 5y ago
      I think it's important to focus on "expensive" and not "placebo" in that statement. If a placebo is all that's needed (or all that we can do), then arguably it shouldn't be expensive.
lrem 5y ago
Marshmallow study? Is the finding about "wait 5 minutes to get two sweets - success in life" debunked, or are you referring to another thing?
- quag 5y ago
  Yes, it has been debunked. They got the causal arrow around the wrong way. It turns out that kids who are in stable situations with trustworthy adults around them, tend to trust adults more when they say “wait 5 minutes and I’ll give you more”, and that kids in bad situations don’t trust adults. Turns out being in a bad situation as a kid does have a long term effect on your life. Nothing to do with the ability of people delay gratification.
TameAntelope 5y ago
If what you've read here concerns you to the point of throwing out the field, you're going to be immensely disappointed by every other scientific field, as well. [0] I'm not an expert, but my understanding is that this article could have been written about literally any field of science, with a depressingly few nouns changed. [1] This is what people mean when they say, "science is messy". This is how the sausage gets made, and if you're cool throwing out psychology as a field, you're probably also going to have to be cool throwing out a lot of other fields. [2] There are efforts being made to clean up how science is conducted, called "metascience", but it's a relatively new field and there's still a lot that hasn't been investigated yet. [3] Basically, if you've ever thought of yourself as "pro science", you've been supporting what happened in this article, whether you realized it or not. [0] https://www.nature.com/articles/533452a https://www.nature.com/articles/533452a [1] https://www.scientificamerican.com/video/is-there-a-reproducibility-crisis-in-science/ https://www.scientificamerican.com/video/is-there-a-reproduc... -- a slightly more accessible version of [0] [2] https://direct.mit.edu/posc/article/28/4/482/97500/Mess-in-Science-and-Wicked-Problems https://direct.mit.edu/posc/article/28/4/482/97500/Mess-in-S... [3] https://en.wikipedia.org/wiki/Metascience https://en.wikipedia.org/wiki/Metascience
- fhood 5y ago
  I think we can maybe inject a little bit of extra nuance here. Not every field is the same, they vary massively in size, level of external influence, and complexity in subject matter. Psychology suffers more than most. It's a large field, so competition to get noticed is likely high. There is money to be made, as discussed in the article, so it struggles with influences outside of pure scientific inquiry, and they have no chance of describing their theories in terms of pure component mathematics. Contrast that with ocean and atmospheric sciences. It is a tiny and very insular field, so the rat race works very differently. This also protects them from outside influences, and a lot of the major underlying physical and biological processes involved can be modeled with mathematics that have been demonstrated to be predictive. It's just easier for some fields to reliably produce higher quality work. I think this is a case where blanket skepticism is just lazy. Every field suffers some level of a reproducibility issue, but some fields are better able to self select for a variety of reasons.
- 5y ago
lemmsjid 5y ago
I won't argue with your mistrust of psychological 'findings'. But: If psychological research is not conducted with scientific rigor, and we treat it as an art, then where does that leave the phenomenology of mental illness, study of social dynamics, etc? Mentally ill people still need to be treated. And historically they have been treated in very, let us say, 'artistic' ways. If we abandon the scientific aspects of psychology, mental illness will still need to be dealt with, but without any framework of rigor or understanding. While landmark findings of psychology (such as the marshmallow experiment) can be questioned and debunked, a lot of what you'll learn when studying psychology as a science is how little we know about psychology. That's an important thing to learn! Does approach X work for people with debilitating anxiety? In all probability, we don't know, or it does work for some people and has the opposite effect on others. That's the day to day of psychological research. When something exciting and novel comes up (like the marshmallow thing, or the Stanford Prison Experiment), it represents an island of false certainty in a sea of uncertainty and suddenly all the laypeople know about it. Such understanding of our lack of understanding can form a bulwark against, at the very least, political, social and religious ideologies that make sweeping claims about behavior and dynamics. It's important to quantify what we don't know as well as what we know. Otherwise we'll be taken over by people making extraordinary and sweeping claims based on no evidence, rather than people at least publishing their evidence to be possibly later debunked. In the linked article's example, if psychology were purely art, there would only be the authors' book, not a paper with actual math to debunk and a journal to interact with. I am probably overstating my point, because when a specious claim is backed up by scientific trappings people may be more likely to believe it, but I am increasingly less certain of that. When it comes to critiques of 'soft sciences' I often come to this. Those fields often came into being because there's actual day to day issues that don't go away even if we have arguable theoretical foundations behind them. Absent said foundations, people still need to make sweeping decisions about economics and psychology that affect millions of lives. The scientific part arose from looking at the history of such interventions and wondering if things like exorcising people with visions actually worked any better than not exorcising them. For many of the people being exorcised, it was a 'landmark finding' that exorcism is not the most effective method of helping them with their issues.
- creato 5y ago