5 ms·

How would you propose we differentiate between students "playing along" and students really having false memories? Couldn't I discard any psychological study by

by Timon3 3y ago

How would you propose we differentiate between students "playing along" and students really having false memories? Couldn't I discard any psychological study by saying "well, the participants could have faked it", since it's by definition not falsifiable (any test could also be faked)?

canjobear 3y ago
You should indeed be extremely skeptical of studies whose only dependent variable is a voluntary participant response and where the participant has an incentive to tell the experimenter what they think they want to hear (for example, to make the experiment end faster).
- kenjackson 3y ago
  Except in this case for the follow up interviews there was no incentive to lie.
  - canjobear 3y ago
    They’re incentivized to tell the experimenters what they want to hear so they can get paid and go home faster.
    - somenameforme 3y ago
      Having participated in plenty of these (as it was required for class credit, and they also paid reasonably well for an in-between classes hour type thing to go do), the 'researchers' can also be completely obnoxious. One study I was involved in had me listen to some rant about how women do better on a math exam I was about to take, being left in a room for about 10 minutes alone, and then being given the exam - on pink paper. It was generic algebra 2 / SAT level test. After the exam the researchers just endlessly hounded me from a hundred different angles trying to get me to express doubt or uncertainty in how I performed on it. If I just wanted my $20 and credit, it would have been far easier to just give them what they wanted. But I've been ever the contrarian. They also said they'd email me my results in 8 weeks. They did not. I've always wondered if I ended up getting culled from their 'study' as an 'outlier.'
      - AlbertCory 3y ago
        Funny, but my one and only contact with PLATO when I was at the U of I was for a Psych 100 experiment like this.
      - kenjackson 3y ago
        That just sounds like bad research. In general if you know their angle that clearly then they’ve failed at how to do a research study of this type — obnoxious or not. It sounds like they don’t know how to do this type of research.
        somenameforme 3y ago
        I'm not sure that was the problem. Journals don't tend to publish negative results, and social science journals seem to have 0 issues with publishing studies that are overtly designed to confirm a hypothesis, rather than challenge it. Add in a bit of publish or perish, and this behavior is ubiquitous. What would you say is an example of good research in social psychology? As an example of bad science, you look to what's likely the single most well known experiment in social psychology - the Stanford Prisoner Experiment. It's not only failed to replicate, but was mostly entirely fabricated. [1] The famous clip of the guy screaming out 'I can’t stand another night! I just can’t take it anymore!' came from a guy who was intentionally faking a meltdown, because he needed to go study and they weren't allowing him access to his books, the guards were actively trained to be cruel, and more. Incidentally Zimbardo (the experimenter), far from being shunned, has received countless accolades for this experiment, some as recent as 2012. [1] - https://gen.medium.com/the-lifespan-of-a-lie-d869212b1f62 https://gen.medium.com/the-lifespan-of-a-lie-d869212b1f62
    - anigbrowl 3y ago
      In most psychological experiments, volunteers are told it will take an hour/an afternoon/whatever at the time they sign up to participate. The notion that all of them actually want to cut corners and just obtain the cash in return for the bare minimum of time and effort is an assumption about people in general, not a fact.
    - kenjackson 3y ago
      In the follow ups they are simply asked to recount and give their confidence of recounting. It’s only the original interview they are told to try to remember details even if they can’t. The premise of the study is that the first interview is used to plant the memories. The follow up interviews are to see if they stick. They specifically aren’t trying to intervene in those interviews.
- Timon3 3y ago
  How would you propose, in this specific study, they verify what you want? Please be specific regarding what questions you want asked, when they should be asked, and how these questions can't be discarded by saying "well, they could be faking it".
  - canjobear 3y ago
    I think this whole experimental paradigm is inherently unfixable.
    - Timon3 3y ago
      So we just stop studying psychology completely?
      - swayvil 3y ago
        You can't trust psychology because people sometimes lie. You can't trust physics because you might be hallucinating. What's left? Authoritarianism? Consensus? Tradition?
      - canjobear 3y ago
        No, but the focus for studies like this should be on nonvoluntary responses like reaction times and EEG in my opinion.
        Timon3 3y ago
        How do you know that the EEG monitoring doesn't influence people? Couldn't they subconsciously fake it due to being actively monitored?
      - cdogl 3y ago
        Or the people undertaking these studies (and more importantly, the people peer reviewing and publishing the work) push back on broad conclusions that aren't substantiated by the research. Having seen the nonsense my partner has to put up with in order to get a post-graduate qualification to practice as a professional psychologist I'm not holding my breath. From where I'm sitting the replication crisis in this field looks like a crisis of incompetence and apathy.
- darkerside 3y ago
  Is it that hard to imagine that teens and the mentally impaired would do the same, focusing on resolving the present conflict (locked in room) over a future hypothetical problem (locked in jail)?
somenameforme 3y ago
Yes, you could discard any study under that rationale among others. And the interesting thing is you'd likely be far more 'informed' for doing so. Social psychology has the lowest replication rate of any field. You're looking at top journals with replication rates in the twenties. [1] What this means is that if you were to take any give study in social psychology, and simply assume it was 'wrong' (either saying the opposite of the truth, or claiming a statistically significant effect when there really is none), then you'd, on average, be dramatically more 'informed' than somebody taking the studies at face value. [2] [1] - https://en.wikipedia.org/wiki/Replication_crisis#In_psychology https://en.wikipedia.org/wiki/Replication_crisis#In_psycholo... [2] - https://en.wikipedia.org/wiki/Pseudoscience https://en.wikipedia.org/wiki/Pseudoscience
- Timon3 3y ago
  I don't want to discount the replication crisis, but I'd like to ask you the same question again: how, specifically, do you design any psychological study that you can't discard with "well, they could be faking it"?
  - yencabulator 3y ago
    Measure changes in something the test subject isn't expecting and wouldn't realize they need to fake? E.g. if you had to sit in the waiting room for 5 extra minutes, your test score changes like this. Measure differences below conscious control, like reaction speed to positive/negative/etc words after exposure to certain inputs.
    - ClumsyPilot 3y ago
      This is too vague. Hiw would you test this hypothesys, that this stidu is teating, by measuring such unrelated signals?
      - yencabulator 3y ago
        Perhaps this hypothesis simply is harder to test scientifically. It seems lots of claims in psychology are not verifiable.
anonymouskimmer 3y ago
> How would you propose we differentiate between students "playing along" and students really having false memories? Remove the incentive for the student to fabulate. Have another, supposedly (to the students) 'independent' researcher ask them questions about it days or weeks later. A potential way of doing this would be to have a member of the Institutional Review Board (or supposed member of said board) following up with the students as a matter of 'quality control' and verification that the study followed IRB guidelines. Phrase the reason for the questioning as looking for non-punitive, but developmental, feedback for the researcher. For ethical reasons this would probably require blinding the original researcher to the identities of the students, but this could be done by having people other than the PI actually ask the questions. For a particular study on whether or not the student did something criminal, have the questioner truthfully assert to the student binding confidentiality as to their identity by dint of being a psychologist or someone like that who has the legal duty to not divulge.
- Timon3 3y ago
  Thank you for an actual proposal! It makes a lot of sense to me, but there is one reason it probably wouldn't be allowed: the study "implanted" false memories (in the sense that participants stated they believed false memories). I don't think the study would pass ethical review if you were only to follow up days or weeks later (after the supposed end of the study), since that would also be the earliest point you would be able to clear up the lie.
  - anonymouskimmer 3y ago
    It could probably be tailored to a shorter time period. A possible great way to do it is have the students originally questioned by grad students, and then the professor themself 'follow up' shortly afterward as a 'check' on their grad student's process. The problem with that is that some of the students may feel incentivized to make the grad student look good.

by Timon3 3y ago

canjobear 3y ago

You should indeed be extremely skeptical of studies whose only dependent variable is a voluntary participant response and where the participant has an incentive to tell the experimenter what they think they want to hear (for example, to make the experiment end faster).

kenjackson 3y ago
Except in this case for the follow up interviews there was no incentive to lie.
- canjobear 3y ago
  They’re incentivized to tell the experimenters what they want to hear so they can get paid and go home faster.
  - somenameforme 3y ago
    Having participated in plenty of these (as it was required for class credit, and they also paid reasonably well for an in-between classes hour type thing to go do), the 'researchers' can also be completely obnoxious. One study I was involved in had me listen to some rant about how women do better on a math exam I was about to take, being left in a room for about 10 minutes alone, and then being given the exam - on pink paper. It was generic algebra 2 / SAT level test. After the exam the researchers just endlessly hounded me from a hundred different angles trying to get me to express doubt or uncertainty in how I performed on it. If I just wanted my $20 and credit, it would have been far easier to just give them what they wanted. But I've been ever the contrarian. They also said they'd email me my results in 8 weeks. They did not. I've always wondered if I ended up getting culled from their 'study' as an 'outlier.'
    - AlbertCory 3y ago
      Funny, but my one and only contact with PLATO when I was at the U of I was for a Psych 100 experiment like this.
    - kenjackson 3y ago
      That just sounds like bad research. In general if you know their angle that clearly then they’ve failed at how to do a research study of this type — obnoxious or not. It sounds like they don’t know how to do this type of research.
      - somenameforme 3y ago
        I'm not sure that was the problem. Journals don't tend to publish negative results, and social science journals seem to have 0 issues with publishing studies that are overtly designed to confirm a hypothesis, rather than challenge it. Add in a bit of publish or perish, and this behavior is ubiquitous. What would you say is an example of good research in social psychology? As an example of bad science, you look to what's likely the single most well known experiment in social psychology - the Stanford Prisoner Experiment. It's not only failed to replicate, but was mostly entirely fabricated. [1] The famous clip of the guy screaming out 'I can’t stand another night! I just can’t take it anymore!' came from a guy who was intentionally faking a meltdown, because he needed to go study and they weren't allowing him access to his books, the guards were actively trained to be cruel, and more. Incidentally Zimbardo (the experimenter), far from being shunned, has received countless accolades for this experiment, some as recent as 2012. [1] - https://gen.medium.com/the-lifespan-of-a-lie-d869212b1f62 https://gen.medium.com/the-lifespan-of-a-lie-d869212b1f62
  - anigbrowl 3y ago
    In most psychological experiments, volunteers are told it will take an hour/an afternoon/whatever at the time they sign up to participate. The notion that all of them actually want to cut corners and just obtain the cash in return for the bare minimum of time and effort is an assumption about people in general, not a fact.
  - kenjackson 3y ago
    In the follow ups they are simply asked to recount and give their confidence of recounting. It’s only the original interview they are told to try to remember details even if they can’t. The premise of the study is that the first interview is used to plant the memories. The follow up interviews are to see if they stick. They specifically aren’t trying to intervene in those interviews.
Timon3 3y ago
How would you propose, in this specific study, they verify what you want? Please be specific regarding what questions you want asked, when they should be asked, and how these questions can't be discarded by saying "well, they could be faking it".
- canjobear 3y ago
  I think this whole experimental paradigm is inherently unfixable.
  - Timon3 3y ago
    So we just stop studying psychology completely?
    - swayvil 3y ago
      You can't trust psychology because people sometimes lie. You can't trust physics because you might be hallucinating. What's left? Authoritarianism? Consensus? Tradition?
    - canjobear 3y ago
      No, but the focus for studies like this should be on nonvoluntary responses like reaction times and EEG in my opinion.
      - Timon3 3y ago
        How do you know that the EEG monitoring doesn't influence people? Couldn't they subconsciously fake it due to being actively monitored?
    - cdogl 3y ago
      Or the people undertaking these studies (and more importantly, the people peer reviewing and publishing the work) push back on broad conclusions that aren't substantiated by the research. Having seen the nonsense my partner has to put up with in order to get a post-graduate qualification to practice as a professional psychologist I'm not holding my breath. From where I'm sitting the replication crisis in this field looks like a crisis of incompetence and apathy.
darkerside 3y ago
Is it that hard to imagine that teens and the mentally impaired would do the same, focusing on resolving the present conflict (locked in room) over a future hypothetical problem (locked in jail)?

somenameforme 3y ago

Yes, you could discard any study under that rationale among others. And the interesting thing is you'd likely be far more 'informed' for doing so. Social psychology has the lowest replication rate of any field. You're looking at top journals with replication rates in the twenties. [1] What this means is that if you were to take any give study in social psychology, and simply assume it was 'wrong' (either saying the opposite of the truth, or claiming a statistically significant effect when there really is none), then you'd, on average, be dramatically more 'informed' than somebody taking the studies at face value. [2] [1] - https://en.wikipedia.org/wiki/Replication_crisis#In_psychology https://en.wikipedia.org/wiki/Replication_crisis#In_psycholo... [2] - https://en.wikipedia.org/wiki/Pseudoscience https://en.wikipedia.org/wiki/Pseudoscience

Timon3 3y ago
I don't want to discount the replication crisis, but I'd like to ask you the same question again: how, specifically, do you design any psychological study that you can't discard with "well, they could be faking it"?
- yencabulator 3y ago
  Measure changes in something the test subject isn't expecting and wouldn't realize they need to fake? E.g. if you had to sit in the waiting room for 5 extra minutes, your test score changes like this. Measure differences below conscious control, like reaction speed to positive/negative/etc words after exposure to certain inputs.
  - ClumsyPilot 3y ago
    This is too vague. Hiw would you test this hypothesys, that this stidu is teating, by measuring such unrelated signals?
    - yencabulator 3y ago
      Perhaps this hypothesis simply is harder to test scientifically. It seems lots of claims in psychology are not verifiable.

anonymouskimmer 3y ago

> How would you propose we differentiate between students "playing along" and students really having false memories? Remove the incentive for the student to fabulate. Have another, supposedly (to the students) 'independent' researcher ask them questions about it days or weeks later. A potential way of doing this would be to have a member of the Institutional Review Board (or supposed member of said board) following up with the students as a matter of 'quality control' and verification that the study followed IRB guidelines. Phrase the reason for the questioning as looking for non-punitive, but developmental, feedback for the researcher. For ethical reasons this would probably require blinding the original researcher to the identities of the students, but this could be done by having people other than the PI actually ask the questions. For a particular study on whether or not the student did something criminal, have the questioner truthfully assert to the student binding confidentiality as to their identity by dint of being a psychologist or someone like that who has the legal duty to not divulge.

Timon3 3y ago
Thank you for an actual proposal! It makes a lot of sense to me, but there is one reason it probably wouldn't be allowed: the study "implanted" false memories (in the sense that participants stated they believed false memories). I don't think the study would pass ethical review if you were only to follow up days or weeks later (after the supposed end of the study), since that would also be the earliest point you would be able to clear up the lie.
- anonymouskimmer 3y ago
  It could probably be tailored to a shorter time period. A possible great way to do it is have the students originally questioned by grad students, and then the professor themself 'follow up' shortly afterward as a 'check' on their grad student's process. The problem with that is that some of the students may feel incentivized to make the grad student look good.