4 ms·

I won't argue with your mistrust of psychological 'findings'. But: If psychological research is not conducted with scientific rigor, and we treat it as an art

I won't argue with your mistrust of psychological 'findings'. But: If psychological research is not conducted with scientific rigor, and we treat it as an art, then where does that leave the phenomenology of mental illness, study of social dynamics, etc? Mentally ill people still need to be treated. And historically they have been treated in very, let us say, 'artistic' ways. If we abandon the scientific aspects of psychology, mental illness will still need to be dealt with, but without any framework of rigor or understanding. While landmark findings of psychology (such as the marshmallow experiment) can be questioned and debunked, a lot of what you'll learn when studying psychology as a science is how little we know about psychology. That's an important thing to learn! Does approach X work for people with debilitating anxiety? In all probability, we don't know, or it does work for some people and has the opposite effect on others. That's the day to day of psychological research. When something exciting and novel comes up (like the marshmallow thing, or the Stanford Prison Experiment), it represents an island of false certainty in a sea of uncertainty and suddenly all the laypeople know about it. Such understanding of our lack of understanding can form a bulwark against, at the very least, political, social and religious ideologies that make sweeping claims about behavior and dynamics. It's important to quantify what we don't know as well as what we know. Otherwise we'll be taken over by people making extraordinary and sweeping claims based on no evidence, rather than people at least publishing their evidence to be possibly later debunked. In the linked article's example, if psychology were purely art, there would only be the authors' book, not a paper with actual math to debunk and a journal to interact with. I am probably overstating my point, because when a specious claim is backed up by scientific trappings people may be more likely to believe it, but I am increasingly less certain of that. When it comes to critiques of 'soft sciences' I often come to this. Those fields often came into being because there's actual day to day issues that don't go away even if we have arguable theoretical foundations behind them. Absent said foundations, people still need to make sweeping decisions about economics and psychology that affect millions of lives. The scientific part arose from looking at the history of such interventions and wondering if things like exorcising people with visions actually worked any better than not exorcising them. For many of the people being exorcised, it was a 'landmark finding' that exorcism is not the most effective method of helping them with their issues.

creato 5y ago
> If we abandon the scientific aspects of psychology, mental illness will still need to be dealt with, but without any framework of rigor or understanding. I think false rigor is worse than no rigor at all. > In the linked article's example, if psychology were purely art, there would only be the authors' book, not a paper with actual math to debunk and a journal to interact with. I am probably overstating my point, because when a specious claim is backed up by scientific trappings people may be more likely to believe it, but I am increasingly less certain of that. I think the primary reason why people are less likely to "trust" scientific results now than in the past is precisely because of some scientists generating "scientific trappings" with the appearance of rigor to justify a bullshit result. Most of the time, no one cares, and those scientists go on to their academic careers as usual. Of course, it's the follow-up careers, books, headlines that lead to the erosion of trust, not the fake math itself (that's just necessary to gain the approval of other scientists). When that is attached to something that the average person can blatantly see is wrong, it erodes their confidence in the entire scientific process and establishment. They'll reject the entire concept of rigor before believing some of the things scientists claim to be true. How is a lay person supposed to separate the modeling that goes into e.g. climate change and "a 2.9 positivity ratio is the key to happiness"? To them they look pretty similar, a bunch of mathy looking garbage that tells them something that isn't intuitive. Personally, I've been forced to sit through a lot of corporate management trainings that are full of citations of titillating psychology results that I know are bullshit. I just don't have the patience or motivation that Nick Brown has to properly debunk them.
- lemmsjid 5y ago
  I quite agree with most of what you are saying, and you are saying it quite well. My main thrust is that we shouldn't throw the baby out with the bathwater by saying, 'Some people aren't rigorous so let's abandon rigor' (I don't think you're saying that, it's a strawman). Even in the paper critiqued in the original post, there was some rigor (in the midst of clear blindness and hubris), sufficient for the team in the article to take it apart on its own merits. There's real examples of cases where no rigor is applied at all, and some facsimile of rigor can improve things in a psychological context. For example, much ink has been spilled in Hacker News about traumatic interview practices adopted and popularized by the FANG companies, such as so-called 'brain teaser' questions. Included in these practices was the widespread notion that interviewees needed to be 'put on the spot', or 'think on their feet'. Interview practice has been studied by the field of organizational psychology for decades, and such practices were from day one counter to several findings (such as that putting the candidate off-balance or in a state of discomfort during the interview and having them successfully navigate that would somehow predict job performance). Eventually several large tech companies conducted internal studies and concluded that the practices had no bearing on job performance. I too have seen BS titillating psychology results, but the antidote is almost always to review the literature. For example you might often hear "conscientiousness on a Big 5 personality test correlates with job performance". Yes it does, but review the literature: it is quite weak and will not do a good job of explaining individual variance. Let's say I had heard about this magical ratio in the OP's article in a job context. My BS meter would have gone off, most certainly. Some actionable number that can explain a tremendously complex and poorly understood set of dynamics? Hmph! I would have reviewed the literature and found the paper in question. I would have seen the large number of citations. Ok, that gives it weight as an idea, but let's see if the idea has legs. Where are the meta studies showing multiple independent corroborations of the number in different contexts and performed by different researchers? Non existent. As someone who takes science and applies it, for me that puts it firmly in the area of 'very interesting finding, but not actionable'. Honestly I think that's probably what was quietly happening with that ratio even before the retractions. Such findings make for great media news, corporate power points, motivational speech meetings, annoying HR meetings, etc., but hopefully (I do say hopefully :) ) in almost every real world setting if someone told a senior manager, "Your team is going to be measured on this positivity ratio, because science!", that manager would make a stink. Of course, maybe not. I do believe an important skill that needs to be increasingly taught is the ability to parse and understand the landscape of papers on a subject.

creato 5y ago

> If we abandon the scientific aspects of psychology, mental illness will still need to be dealt with, but without any framework of rigor or understanding. I think false rigor is worse than no rigor at all. > In the linked article's example, if psychology were purely art, there would only be the authors' book, not a paper with actual math to debunk and a journal to interact with. I am probably overstating my point, because when a specious claim is backed up by scientific trappings people may be more likely to believe it, but I am increasingly less certain of that. I think the primary reason why people are less likely to "trust" scientific results now than in the past is precisely because of some scientists generating "scientific trappings" with the appearance of rigor to justify a bullshit result. Most of the time, no one cares, and those scientists go on to their academic careers as usual. Of course, it's the follow-up careers, books, headlines that lead to the erosion of trust, not the fake math itself (that's just necessary to gain the approval of other scientists). When that is attached to something that the average person can blatantly see is wrong, it erodes their confidence in the entire scientific process and establishment. They'll reject the entire concept of rigor before believing some of the things scientists claim to be true. How is a lay person supposed to separate the modeling that goes into e.g. climate change and "a 2.9 positivity ratio is the key to happiness"? To them they look pretty similar, a bunch of mathy looking garbage that tells them something that isn't intuitive. Personally, I've been forced to sit through a lot of corporate management trainings that are full of citations of titillating psychology results that I know are bullshit. I just don't have the patience or motivation that Nick Brown has to properly debunk them.

lemmsjid 5y ago
I quite agree with most of what you are saying, and you are saying it quite well. My main thrust is that we shouldn't throw the baby out with the bathwater by saying, 'Some people aren't rigorous so let's abandon rigor' (I don't think you're saying that, it's a strawman). Even in the paper critiqued in the original post, there was some rigor (in the midst of clear blindness and hubris), sufficient for the team in the article to take it apart on its own merits. There's real examples of cases where no rigor is applied at all, and some facsimile of rigor can improve things in a psychological context. For example, much ink has been spilled in Hacker News about traumatic interview practices adopted and popularized by the FANG companies, such as so-called 'brain teaser' questions. Included in these practices was the widespread notion that interviewees needed to be 'put on the spot', or 'think on their feet'. Interview practice has been studied by the field of organizational psychology for decades, and such practices were from day one counter to several findings (such as that putting the candidate off-balance or in a state of discomfort during the interview and having them successfully navigate that would somehow predict job performance). Eventually several large tech companies conducted internal studies and concluded that the practices had no bearing on job performance. I too have seen BS titillating psychology results, but the antidote is almost always to review the literature. For example you might often hear "conscientiousness on a Big 5 personality test correlates with job performance". Yes it does, but review the literature: it is quite weak and will not do a good job of explaining individual variance. Let's say I had heard about this magical ratio in the OP's article in a job context. My BS meter would have gone off, most certainly. Some actionable number that can explain a tremendously complex and poorly understood set of dynamics? Hmph! I would have reviewed the literature and found the paper in question. I would have seen the large number of citations. Ok, that gives it weight as an idea, but let's see if the idea has legs. Where are the meta studies showing multiple independent corroborations of the number in different contexts and performed by different researchers? Non existent. As someone who takes science and applies it, for me that puts it firmly in the area of 'very interesting finding, but not actionable'. Honestly I think that's probably what was quietly happening with that ratio even before the retractions. Such findings make for great media news, corporate power points, motivational speech meetings, annoying HR meetings, etc., but hopefully (I do say hopefully :) ) in almost every real world setting if someone told a senior manager, "Your team is going to be measured on this positivity ratio, because science!", that manager would make a stink. Of course, maybe not. I do believe an important skill that needs to be increasingly taught is the ability to parse and understand the landscape of papers on a subject.