3 ms·

As much as that’s an eye-catching headline, even the author admits it was a bad study that hasn’t been reproduced.

by solfox 2y ago

dustyventure 2y ago
If the author told you their psychology study was reproducible, not dismissing it would be the other category of error.
- godelski 2y ago
  Which I think is a critical aspect of the author's argument: the lack of replication. There's a lot of factors at play. As I mentioned in my main comment, the study naturally has a metric fuckton of variables and noise which makes even basic experiments extremely difficult, but there are other factors like willingness to close that gap (is it surprising few want to climb a treacherous mountain?), as well as the whole structure of academia and the way metrics are formed. How do you create a foundation when you're not only not incentivized to replicate but actively dis-incentivized? How do you explore that mountain which certainly has many pitfalls and uncertain paths when you must publish frequently? Those problems are not remotely isolated to psychology, but they have a huge and crazy difficult mountain to climb and should we be surprised that few try to climb it when attempting to do so is far more likely to lead to academic success rather than helping the field make their way up? Even if you don't fall off? It can take a long time to actually capitalize on those gains. I do think this is a conversation academia needs to have. Everything in place is logical and makes sense, they were done for good reasons, but I think we need to be honest that exploration is just a highly risky business. You'll never make it across the ocean, to the moon, or to other worlds if you are unwilling to lose a few people (researchers who never make any impact) or risk a have a few conmen (researchers who make shit up).Ironically, if you try too hard to prevent these from happening, you'll be doomed to only have them (you'll only explore just beyond the fence and you'll hear stories of what is imagined far beyond; either identical to just outside the fence or wild stories. But you'll never know until you go). We can inch our way out or we can be brave. I think unfortunately it is only explorers who can tell the brave from imaginative. It sucks, but is this not the nature of it? A story as old as stories are. But we wouldn't be where we are if we didn't engage in risky business.
  - dustyventure 2y ago
    It sounds to me like you are justifying a continuance of frauds that masquerade as scientific. The danger here is is not just direct effects but also what they displace. One can not actually do psychological studies beyond the test of chemical remedies anymore. The entire field is overrun with charlatans and 101 classes to make more of them because they will all be treated as valid enough to fund and they are all much more exciting than the scientific process.
    - godelski 2y ago
      Oh dear god no. You can check my comment history to see I rant about that frequently. I with in ML and dear god is there so much fraud. What I arguing for is honesty. It's perfectly okay to do work where there's a high level of uncertainty *UNDER THE CONDITION THAT YOU ARE UPFRONT WITH THAT UNCERTAINTY.* Short of that is at best deceptive. I'll encourage you to look at my recent comment history because I've said a lot about that. But what I'm trying to discuss in the previous comment is the systematic and structural components that encourage this deceptive and fraudulent behavior. Because if we don't recognize why it happens then we can't really solve it. Sure, we can hit a reset button and it'll go away, but it's we don't recognize the pressures that push the fields in these directions then it is just going to happen again. The reason I'm arguing that we have to take risks is because many of these pressures are created by naive metrics that attempt to measure things that are not realistically measurable (so of course the metrics get hacked. It's literally Goodhart's law in action). But there are so many issues at play. The lack of incentivization of replication means few people earnestly read the works of others. It means science doesn't actually happen because the whole fucking point is replication. People aren't even reading the papers of their peers in their own departments! Literally the person in the office next to them! The whole academic structure is screwy. You train someone for years to become an expert researcher (grad student) and then immediately put them in a manager role (professor)? The fuck were we doing that training for? I get managers, but with so many admins in universities why are professors doing less and less research? Advising grad students isn't research as every grad student will tell you their professor doesn't get details. I could go on. But my point is the system is fucked up and overrun by bureaucrats who are risk adverse. They'd rather rely on metrics they are unwilling to understand than acknowledge that they live in a world that is fuzzy and chaotic. That kind of blindness is for religion and conspiracies, not science.
scotty79 2y ago
There was another research that indicated that the therapeutical framework that the therapist uses has no influence on the probability of postive outcome. What mattered more is whether a therapist was able to form a meaningful connection with a patient.

As much as that’s an eye-catching headline, even the author admits it was a bad study that hasn’t been reproduced.

by solfox 2y ago

As much as that’s an eye-catching headline, even the author admits it was a bad study that hasn’t been reproduced.

dustyventure 2y ago

If the author told you their psychology study was reproducible, not dismissing it would be the other category of error.

godelski 2y ago
Which I think is a critical aspect of the author's argument: the lack of replication. There's a lot of factors at play. As I mentioned in my main comment, the study naturally has a metric fuckton of variables and noise which makes even basic experiments extremely difficult, but there are other factors like willingness to close that gap (is it surprising few want to climb a treacherous mountain?), as well as the whole structure of academia and the way metrics are formed. How do you create a foundation when you're not only not incentivized to replicate but actively dis-incentivized? How do you explore that mountain which certainly has many pitfalls and uncertain paths when you must publish frequently? Those problems are not remotely isolated to psychology, but they have a huge and crazy difficult mountain to climb and should we be surprised that few try to climb it when attempting to do so is far more likely to lead to academic success rather than helping the field make their way up? Even if you don't fall off? It can take a long time to actually capitalize on those gains. I do think this is a conversation academia needs to have. Everything in place is logical and makes sense, they were done for good reasons, but I think we need to be honest that exploration is just a highly risky business. You'll never make it across the ocean, to the moon, or to other worlds if you are unwilling to lose a few people (researchers who never make any impact) or risk a have a few conmen (researchers who make shit up).Ironically, if you try too hard to prevent these from happening, you'll be doomed to only have them (you'll only explore just beyond the fence and you'll hear stories of what is imagined far beyond; either identical to just outside the fence or wild stories. But you'll never know until you go). We can inch our way out or we can be brave. I think unfortunately it is only explorers who can tell the brave from imaginative. It sucks, but is this not the nature of it? A story as old as stories are. But we wouldn't be where we are if we didn't engage in risky business.
- dustyventure 2y ago
  It sounds to me like you are justifying a continuance of frauds that masquerade as scientific. The danger here is is not just direct effects but also what they displace. One can not actually do psychological studies beyond the test of chemical remedies anymore. The entire field is overrun with charlatans and 101 classes to make more of them because they will all be treated as valid enough to fund and they are all much more exciting than the scientific process.
  - godelski 2y ago
    Oh dear god no. You can check my comment history to see I rant about that frequently. I with in ML and dear god is there so much fraud. What I arguing for is honesty. It's perfectly okay to do work where there's a high level of uncertainty *UNDER THE CONDITION THAT YOU ARE UPFRONT WITH THAT UNCERTAINTY.* Short of that is at best deceptive. I'll encourage you to look at my recent comment history because I've said a lot about that. But what I'm trying to discuss in the previous comment is the systematic and structural components that encourage this deceptive and fraudulent behavior. Because if we don't recognize why it happens then we can't really solve it. Sure, we can hit a reset button and it'll go away, but it's we don't recognize the pressures that push the fields in these directions then it is just going to happen again. The reason I'm arguing that we have to take risks is because many of these pressures are created by naive metrics that attempt to measure things that are not realistically measurable (so of course the metrics get hacked. It's literally Goodhart's law in action). But there are so many issues at play. The lack of incentivization of replication means few people earnestly read the works of others. It means science doesn't actually happen because the whole fucking point is replication. People aren't even reading the papers of their peers in their own departments! Literally the person in the office next to them! The whole academic structure is screwy. You train someone for years to become an expert researcher (grad student) and then immediately put them in a manager role (professor)? The fuck were we doing that training for? I get managers, but with so many admins in universities why are professors doing less and less research? Advising grad students isn't research as every grad student will tell you their professor doesn't get details. I could go on. But my point is the system is fucked up and overrun by bureaucrats who are risk adverse. They'd rather rely on metrics they are unwilling to understand than acknowledge that they live in a world that is fuzzy and chaotic. That kind of blindness is for religion and conspiracies, not science.

scotty79 2y ago

There was another research that indicated that the therapeutical framework that the therapist uses has no influence on the probability of postive outcome. What mattered more is whether a therapist was able to form a meaningful connection with a patient.