21 ms·

I don't want to discount the replication crisis, but I'd like to ask you the same question again: how, specifically, do you design any psychological study that

by Timon3 3y ago

I don't want to discount the replication crisis, but I'd like to ask you the same question again: how, specifically, do you design any psychological study that you can't discard with "well, they could be faking it"?

yencabulator 3y ago
Measure changes in something the test subject isn't expecting and wouldn't realize they need to fake? E.g. if you had to sit in the waiting room for 5 extra minutes, your test score changes like this. Measure differences below conscious control, like reaction speed to positive/negative/etc words after exposure to certain inputs.
- ClumsyPilot 3y ago
  This is too vague. Hiw would you test this hypothesys, that this stidu is teating, by measuring such unrelated signals?
  - yencabulator 3y ago
    Perhaps this hypothesis simply is harder to test scientifically. It seems lots of claims in psychology are not verifiable.
- Timon3 3y ago
  > Measure changes in something the test subject isn't expecting and wouldn't realize they need to fake? E.g. if you had to sit in the waiting room for 5 extra minutes, your test score changes like this. I can discard this by saying "well, the subject might behave subconsciously differently when they know they are in any experiment". How would you refute that? > Measure differences below conscious control, like reaction speed to positive/negative/etc words after exposure to certain inputs. What if there are phenomena that don't correlate with the differences you know to measure?
deleted 3y ago
[deleted]
joebob42 3y ago
It being extremely difficult or impossible to design a convincing study doesn't make the other ones convincing.
- Timon3 3y ago
  I am not saying that any study is or is not convincing. I am literally just asking: how could you define a test for this? I am not stating that any individual study is good or bad or anything in between. I want to know: if I design a study, how do I design it in a way that makes it a good study in regards to this question?
  - bee_rider 3y ago
    I don’t think it is reasonable to expect for a person on in a sort of informal online conversation like this to be able to define a bulletproof experiment for the topic. Let’s apply a similar logic to something that is clearly possible to design an experiment around. How about detecting neutrinos? If we were commenting on an article about detecting neutrinos by dropping a piece of paper and seeing if it bounced around a bit on the way down, I think we’d all identify that as not a very good experiment. But, I could not design a very good experiment for detecting neutrinos. And, this doesn’t say much about whether or not neutrinos exist, I’m just not a physicist. So, if posed with that question, I would not have a good response. Despite this, I would still believe the dropped piece of paper experiment was not very good.
    - Timon3 3y ago
      > I don’t think it is reasonable to expect for a person on in a sort of informal online conversation like this to be able to define a bulletproof experiment for the topic. I am allowed to ask, right? It doesn't seem crazy to me that when someone criticizes a study, someone else will ask "how could you solve the problem you describe?". I would like to know if someone has any idea on how to falsify it, because if someone were to find one, it might convince me of the criticism. > Let’s apply a similar logic to something that is clearly possible to design an experiment around. How about detecting neutrinos? If we were commenting on an article about detecting neutrinos by dropping a piece of paper and seeing if it bounced around a bit on the way down, I think we’d all identify that as not a very good experiment. But, I could not design a very good experiment for detecting neutrinos. And, this doesn’t say much about whether or not neutrinos exist, I’m just not a physicist. I don't think that is a good comparison. A better one would be if there is an experiment, and then someone comments on the results by saying "well, it could have been dark matter particles interacting in some way". It might as well be true, but it's not definitive if they don't explain how to measure and isolate that. Someone else asking for how they would design the experiment to take this into account seems like a completely reasonable ask to me.
tsimionescu 3y ago
It might not be possible with current knowledge, putting psychology outside the realm of the sciences.
ineptech 3y ago
Was this not the whole point of the study in TFA comparing false memories of crimes to false memories of non-crimes? The assumption that people would be less willing to admit to a serious crime than to having been injured? > Intriguingly, the criminal false events seemed to be just as believable as the emotional ones. Students tended to provide the same number of details, and reported similar levels of confidence, vividness, and sensory detail for the two types of event.
YeGoblynQueenne 3y ago
Well, maybe you don't? Science gives you the tools to know when you're wrong. If you can't falsify your hypotheses then you're not generating new knowledge. So maybe stop trying and do something else. Which is not to say that we should stop psychological research- not at all. But maybe we should not treat psychology in the same way we treat science. Specifically, we shouldn't be trying to apply the scientific method in psychological research. Applying the tools of science in a domain where no scientific questions can be answered is only going to produce noise and confusion. I know that would really hurt psychologists' pride but, for example, philosophers are not considered scientists and yet their contributions to human knowledge are difficult to deny. Similar points have been made much more forcefully in "Cargo Cult Science", but I don't want to be mean myself. I'm just trying to say that I think there's far better ways to do psychology than trying to do it like it's physics or chemistry.

by Timon3 3y ago

yencabulator 3y ago

Measure changes in something the test subject isn't expecting and wouldn't realize they need to fake? E.g. if you had to sit in the waiting room for 5 extra minutes, your test score changes like this. Measure differences below conscious control, like reaction speed to positive/negative/etc words after exposure to certain inputs.

ClumsyPilot 3y ago
This is too vague. Hiw would you test this hypothesys, that this stidu is teating, by measuring such unrelated signals?
- yencabulator 3y ago
  Perhaps this hypothesis simply is harder to test scientifically. It seems lots of claims in psychology are not verifiable.
Timon3 3y ago
> Measure changes in something the test subject isn't expecting and wouldn't realize they need to fake? E.g. if you had to sit in the waiting room for 5 extra minutes, your test score changes like this. I can discard this by saying "well, the subject might behave subconsciously differently when they know they are in any experiment". How would you refute that? > Measure differences below conscious control, like reaction speed to positive/negative/etc words after exposure to certain inputs. What if there are phenomena that don't correlate with the differences you know to measure?

deleted 3y ago

[deleted]

joebob42 3y ago

It being extremely difficult or impossible to design a convincing study doesn't make the other ones convincing.

Timon3 3y ago
I am not saying that any study is or is not convincing. I am literally just asking: how could you define a test for this? I am not stating that any individual study is good or bad or anything in between. I want to know: if I design a study, how do I design it in a way that makes it a good study in regards to this question?
- bee_rider 3y ago
  I don’t think it is reasonable to expect for a person on in a sort of informal online conversation like this to be able to define a bulletproof experiment for the topic. Let’s apply a similar logic to something that is clearly possible to design an experiment around. How about detecting neutrinos? If we were commenting on an article about detecting neutrinos by dropping a piece of paper and seeing if it bounced around a bit on the way down, I think we’d all identify that as not a very good experiment. But, I could not design a very good experiment for detecting neutrinos. And, this doesn’t say much about whether or not neutrinos exist, I’m just not a physicist. So, if posed with that question, I would not have a good response. Despite this, I would still believe the dropped piece of paper experiment was not very good.
  - Timon3 3y ago
    > I don’t think it is reasonable to expect for a person on in a sort of informal online conversation like this to be able to define a bulletproof experiment for the topic. I am allowed to ask, right? It doesn't seem crazy to me that when someone criticizes a study, someone else will ask "how could you solve the problem you describe?". I would like to know if someone has any idea on how to falsify it, because if someone were to find one, it might convince me of the criticism. > Let’s apply a similar logic to something that is clearly possible to design an experiment around. How about detecting neutrinos? If we were commenting on an article about detecting neutrinos by dropping a piece of paper and seeing if it bounced around a bit on the way down, I think we’d all identify that as not a very good experiment. But, I could not design a very good experiment for detecting neutrinos. And, this doesn’t say much about whether or not neutrinos exist, I’m just not a physicist. I don't think that is a good comparison. A better one would be if there is an experiment, and then someone comments on the results by saying "well, it could have been dark matter particles interacting in some way". It might as well be true, but it's not definitive if they don't explain how to measure and isolate that. Someone else asking for how they would design the experiment to take this into account seems like a completely reasonable ask to me.

tsimionescu 3y ago

It might not be possible with current knowledge, putting psychology outside the realm of the sciences.

ineptech 3y ago

Was this not the whole point of the study in TFA comparing false memories of crimes to false memories of non-crimes? The assumption that people would be less willing to admit to a serious crime than to having been injured? > Intriguingly, the criminal false events seemed to be just as believable as the emotional ones. Students tended to provide the same number of details, and reported similar levels of confidence, vividness, and sensory detail for the two types of event.

YeGoblynQueenne 3y ago

Well, maybe you don't? Science gives you the tools to know when you're wrong. If you can't falsify your hypotheses then you're not generating new knowledge. So maybe stop trying and do something else. Which is not to say that we should stop psychological research- not at all. But maybe we should not treat psychology in the same way we treat science. Specifically, we shouldn't be trying to apply the scientific method in psychological research. Applying the tools of science in a domain where no scientific questions can be answered is only going to produce noise and confusion. I know that would really hurt psychologists' pride but, for example, philosophers are not considered scientists and yet their contributions to human knowledge are difficult to deny. Similar points have been made much more forcefully in "Cargo Cult Science", but I don't want to be mean myself. I'm just trying to say that I think there's far better ways to do psychology than trying to do it like it's physics or chemistry.