7 ms·

According to the "method" section, 138 out of 257 candidate (233 final) participants in the study were first-year psychology students - I can't help but wonder

According to the "method" section, 138 out of 257 candidate (233 final) participants in the study were first-year psychology students - I can't help but wonder if this might have biased the study's results. The investigators also classified graduate students as laypeople, which doesn't quite feel right to me.

jaclaz 6y ago
Besides, this is more or less an ode to "common sense" which is in my experience actually not so common. And definitely there is some form of age group bias, if the large majority were students (first year or otherwise) the sample must be dominated by 20-25 years people. Maybe (or maybe not) a sample with more aged people (i.e. with more life experience) would have reached more than 59%.
ithkuil 6y ago
As a layperson thus I can predict that this study won't replicate. (liar paradox)
- AstralStorm 6y ago
  You're only 50% accurate though.
  - ithkuil 6y ago
    I'm in a superposition state of being right and wrong
    - deleted 6y ago
      [deleted]
qznc 6y ago
Well, "laypeople" in this case necessarily refers to people who can actually read the actual study instead of some pop-sci article about it. (That excludes many of the journalists writing these pop sci articles and practically all of the editors writing the headlines. /sarcasm)
- deleted 6y ago
  [deleted]
- dash2 6y ago
  Actually not so: "The materials for each study included a short description of the research question, its operationalization, and the key finding. These descriptions were inspired by those provided in the SSRP and ML2, but were rephrased to be comprehensible by laypeople. In the description-only condition, solely these descriptive texts were provided; in the description-plus-evidence condition, the Bayes factor and its verbal interpretation (e.g., “moderate evidence”) were added to the description of each study."
jokethrowaway 6y ago
TIL: People are able to predict studies from people in their fields who probably a have a similar bias. I was surprised to read about the low reproducibility rate, how can you call it social "science" if you can't reproduce it?
- realradicalwash 6y ago
  > how can you call it social "science" if you can't reproduce it? NB: this is not specific to the social sciences. psychology, cognitive sciences, medicine(!), and other fields have a 'replication crisis' also. there are some really good papers on this. i am not aware of systematic queries into CS/ML/DL papers, but from my own experience: when you pick up a paper that claims 9x% accuracy, the chance that you are able to replicate that number is in line with replication % in other fields.
  - jokethrowaway 6y ago
    That's very interesting! I remember this happening a lot with sports / nutrition studies. I'm not surprised to see medicine in there as well. Overall, understanding the brain is hard, understanding our bodies is hard, it's normal that we can come up with a limited amount of consistently reproducible knowledge per year. From all my friends who pursued STEM, I assume this is happening because of the pressure on people pursuing higher education at all costs, not finding enough qualified jobs, getting stuck in academia and printing low quality papers to get a promotion / keep getting another public grant. If you add in to the mix results influenced by the current political climate, what your peers think, what your sponsors want to prove for financial gain, the situation gets bleak very fast. Irreproducible papers are the abandoned OSS projects of software engineers in private tech + corruption + public money, a recipe for disaster.
    - qchris 6y ago
      I don't think that's quite fair. Irreproducible papers have supposedly gone through a fairly rigorous review process by their peers, and are usually published by someone with a PhD (or who will soon have one) at an institution of higher learning. Many papers (which often end up being irreproducible) use language that says "This is the thing I found, and it is/is not true." OSS is often someone learning, or who has an itch to scratch, and put some code out there that may or may not solve your problem or work right. The "lack of warranty" clause in most open-source licensing is really important here, because it's largely designed to say "hey, I did a thing but make no guarantees so use at your own risk", whereas a published paper says "hey, a number of experts and people who should know all agree that this is a thoroughly-researched and well-thought-out position, and you can probably consider it to be true (or nearly so) and base some of your decisions on it." I think that change in context is really important to consider when making the analogy here.
Xelbair 6y ago
Well most social science studies are conducted over the same sample - it just proves that all their results are biased.
losvedir 6y ago
Not exactly the same, but there's this interesting quiz that presents a number of studies that appeared in Science and Nature and asks you to guess which failed to replicate: https://80000hours.org/psychology-replication-quiz/ https://80000hours.org/psychology-replication-quiz/ . After taking that the result in this paper doesn't surprise me. I'm educated, but not in Psychology, and I think most people can tell which studies are more likely to fail to replicate.
- IshKebab 6y ago
  Great quiz! I got 14/15 right. Pretty easy to tell I thought - it seems like all the bullshit ones are "people who do done activity then behave differently when doing some completely unrelated task". Do psychologists dream up interesting self-help style headlines and then try and prove them? "Spending 5 minutes a day thinking about sex could make you live 10 years longer!"
  - smogcutter 6y ago
    Yeah, they’re all priming studies. Afaik that was hot shit for a while but has been mostly discredited. See “power poses” for a dumb way it trickled into pop culture.
- ALittleLight 6y ago
  The last time I took this quiz I got them all right. I just used the heuristic "Would I expect this to be true regardless of whether this paper existed?"
  - gweinberg 6y ago
    Yeah, that works unless the quiz is cherry-picked to demonstrate the opposite.
  - srtjstjsj 6y ago
    So things they do you believe are common sense fail to replicate?
RandoHolmes 6y ago
Maybe this particular study, but I have to say I believe it in general. I remember working for a manufacturing company where I drove a forklift (during the summer). One of my most vivid memories is walking into the breakroom and seeing on the television a scrolling banner that declared homosexual couples were less likely to have children than heterosexual couples. And I remember everyone in that room laughing their asses off. And it's not as if no one in that room realized the reason for the study was so they could take into account things like adoption, etc. It's that the result was so obvious, even taking that into account, that it was amazing that someone was PAID to come to a conclusion that everyone knew without the money. And this is the crux of the problem with "science". It wants to be "interesting", so it will literally try to drum up something against what "everyone knows". So the idea that what everyone over generations "knows" is generally more applicable than "science" is not surprising at all.
- im3w1l 6y ago
  The qualitative factor is obvious. But the quantitative may hit that sweet spot. Studying just how many fewer children they have and how that interplays with free time, work and wealth. Or how the decreased contact with the younger generation changes your worldview and psychology.
- dominotw 6y ago
  > Maybe this particular study, but I have to say I believe it in general. recursion :D
- X6S1x6Okd1st 6y ago
  The scientific process should actually result in people verifying things that seem obvious all the time. Admitting that you might be wrong and checking it even though it seems obvious was a major factor in starting to really increase the density of correct knowledge.
  - RandoHolmes 6y ago
    How can we know that eggs are tasty unless a 3rd party tells us! I won't quote the science, I'll simply say that things are knowable outside of a system...
rvense 6y ago
This is extremely common in social science, for obvious reasons (imagine you're a social scientist and need to find 200 people to answer your questionnaire and you don't have any money...), and I wouldn't be surprised if it has some part in the "replication crisis". A personal favourite example is that meme darling, the Dunning-Kruger effect. In addition to often being an extremely simplified version of the original proposition, which is a lot more subtle than "dumb people think they're smart", it's often presented as if (social!) science has uncovered a universal trait in the human race. But really this effect is based on very few studies, and they were mostly done on American college students. I wonder how well it replicates...
- riffraff 6y ago
  For accidental reasons I've been around psychology students for a long time and rhere was a joke along the lines of "psychology is the study of people who study psychology". Students are expected to participate in a few studies now and then as they will be on the other side at some point.

jaclaz 6y ago

Besides, this is more or less an ode to "common sense" which is in my experience actually not so common. And definitely there is some form of age group bias, if the large majority were students (first year or otherwise) the sample must be dominated by 20-25 years people. Maybe (or maybe not) a sample with more aged people (i.e. with more life experience) would have reached more than 59%.

ithkuil 6y ago

As a layperson thus I can predict that this study won't replicate. (liar paradox)

AstralStorm 6y ago
You're only 50% accurate though.
- ithkuil 6y ago
  I'm in a superposition state of being right and wrong
  - deleted 6y ago
    [deleted]

qznc 6y ago

Well, "laypeople" in this case necessarily refers to people who can actually read the actual study instead of some pop-sci article about it. (That excludes many of the journalists writing these pop sci articles and practically all of the editors writing the headlines. /sarcasm)

deleted 6y ago
[deleted]
dash2 6y ago
Actually not so: "The materials for each study included a short description of the research question, its operationalization, and the key finding. These descriptions were inspired by those provided in the SSRP and ML2, but were rephrased to be comprehensible by laypeople. In the description-only condition, solely these descriptive texts were provided; in the description-plus-evidence condition, the Bayes factor and its verbal interpretation (e.g., “moderate evidence”) were added to the description of each study."

jokethrowaway 6y ago

TIL: People are able to predict studies from people in their fields who probably a have a similar bias. I was surprised to read about the low reproducibility rate, how can you call it social "science" if you can't reproduce it?

realradicalwash 6y ago
> how can you call it social "science" if you can't reproduce it? NB: this is not specific to the social sciences. psychology, cognitive sciences, medicine(!), and other fields have a 'replication crisis' also. there are some really good papers on this. i am not aware of systematic queries into CS/ML/DL papers, but from my own experience: when you pick up a paper that claims 9x% accuracy, the chance that you are able to replicate that number is in line with replication % in other fields.
- jokethrowaway 6y ago
  That's very interesting! I remember this happening a lot with sports / nutrition studies. I'm not surprised to see medicine in there as well. Overall, understanding the brain is hard, understanding our bodies is hard, it's normal that we can come up with a limited amount of consistently reproducible knowledge per year. From all my friends who pursued STEM, I assume this is happening because of the pressure on people pursuing higher education at all costs, not finding enough qualified jobs, getting stuck in academia and printing low quality papers to get a promotion / keep getting another public grant. If you add in to the mix results influenced by the current political climate, what your peers think, what your sponsors want to prove for financial gain, the situation gets bleak very fast. Irreproducible papers are the abandoned OSS projects of software engineers in private tech + corruption + public money, a recipe for disaster.
  - qchris 6y ago
    I don't think that's quite fair. Irreproducible papers have supposedly gone through a fairly rigorous review process by their peers, and are usually published by someone with a PhD (or who will soon have one) at an institution of higher learning. Many papers (which often end up being irreproducible) use language that says "This is the thing I found, and it is/is not true." OSS is often someone learning, or who has an itch to scratch, and put some code out there that may or may not solve your problem or work right. The "lack of warranty" clause in most open-source licensing is really important here, because it's largely designed to say "hey, I did a thing but make no guarantees so use at your own risk", whereas a published paper says "hey, a number of experts and people who should know all agree that this is a thoroughly-researched and well-thought-out position, and you can probably consider it to be true (or nearly so) and base some of your decisions on it." I think that change in context is really important to consider when making the analogy here.

Xelbair 6y ago

Well most social science studies are conducted over the same sample - it just proves that all their results are biased.

losvedir 6y ago

Not exactly the same, but there's this interesting quiz that presents a number of studies that appeared in Science and Nature and asks you to guess which failed to replicate: https://80000hours.org/psychology-replication-quiz/ https://80000hours.org/psychology-replication-quiz/ . After taking that the result in this paper doesn't surprise me. I'm educated, but not in Psychology, and I think most people can tell which studies are more likely to fail to replicate.

IshKebab 6y ago
Great quiz! I got 14/15 right. Pretty easy to tell I thought - it seems like all the bullshit ones are "people who do done activity then behave differently when doing some completely unrelated task". Do psychologists dream up interesting self-help style headlines and then try and prove them? "Spending 5 minutes a day thinking about sex could make you live 10 years longer!"
- smogcutter 6y ago
  Yeah, they’re all priming studies. Afaik that was hot shit for a while but has been mostly discredited. See “power poses” for a dumb way it trickled into pop culture.
ALittleLight 6y ago
The last time I took this quiz I got them all right. I just used the heuristic "Would I expect this to be true regardless of whether this paper existed?"
- gweinberg 6y ago
  Yeah, that works unless the quiz is cherry-picked to demonstrate the opposite.
- srtjstjsj 6y ago
  So things they do you believe are common sense fail to replicate?

RandoHolmes 6y ago

Maybe this particular study, but I have to say I believe it in general. I remember working for a manufacturing company where I drove a forklift (during the summer). One of my most vivid memories is walking into the breakroom and seeing on the television a scrolling banner that declared homosexual couples were less likely to have children than heterosexual couples. And I remember everyone in that room laughing their asses off. And it's not as if no one in that room realized the reason for the study was so they could take into account things like adoption, etc. It's that the result was so obvious, even taking that into account, that it was amazing that someone was PAID to come to a conclusion that everyone knew without the money. And this is the crux of the problem with "science". It wants to be "interesting", so it will literally try to drum up something against what "everyone knows". So the idea that what everyone over generations "knows" is generally more applicable than "science" is not surprising at all.

im3w1l 6y ago
The qualitative factor is obvious. But the quantitative may hit that sweet spot. Studying just how many fewer children they have and how that interplays with free time, work and wealth. Or how the decreased contact with the younger generation changes your worldview and psychology.
dominotw 6y ago
> Maybe this particular study, but I have to say I believe it in general. recursion :D
X6S1x6Okd1st 6y ago
The scientific process should actually result in people verifying things that seem obvious all the time. Admitting that you might be wrong and checking it even though it seems obvious was a major factor in starting to really increase the density of correct knowledge.
- RandoHolmes 6y ago
  How can we know that eggs are tasty unless a 3rd party tells us! I won't quote the science, I'll simply say that things are knowable outside of a system...

rvense 6y ago

This is extremely common in social science, for obvious reasons (imagine you're a social scientist and need to find 200 people to answer your questionnaire and you don't have any money...), and I wouldn't be surprised if it has some part in the "replication crisis". A personal favourite example is that meme darling, the Dunning-Kruger effect. In addition to often being an extremely simplified version of the original proposition, which is a lot more subtle than "dumb people think they're smart", it's often presented as if (social!) science has uncovered a universal trait in the human race. But really this effect is based on very few studies, and they were mostly done on American college students. I wonder how well it replicates...

riffraff 6y ago
For accidental reasons I've been around psychology students for a long time and rhere was a joke along the lines of "psychology is the study of people who study psychology". Students are expected to participate in a few studies now and then as they will be on the other side at some point.