7 ms·

I read the article. It barely scratches the surface of the issue, I was mad as a psychology student that this was the state of our knowledge when I read it in 2

I read the article. It barely scratches the surface of the issue, I was mad as a psychology student that this was the state of our knowledge when I read it in 2013 and didn't know about the publication crisis. The actual academic publication is really interesting and I'd recommend it as a read on whatever commute you're on. Here is the link: https://www.ssoar.info/ssoar/bitstream/handle/document/42104/ssoar-2010-henrich_et_al-The_weirdest_people_in_the.pdf?sequence=1 https://www.ssoar.info/ssoar/bitstream/handle/document/42104... Years after pondering this article, I felt I had figured it out: psychology is secretly the study of particular subset of 18 to 21 year old American women. The ones who study psychology because it's a female dominated study and all those psych students need their credit and 'participating' in research is part of it. Most of them are American because psychology is a bigger thing in the US than in Europe, or at least it seemed to be regarding well-known theories, so I presume most research happens there. There is another big group. A lot of dead mice (neuroscience).

simple_phrases 8y ago
> The ones who study psychology because it's a female dominated study and all those psych students need their credit and 'participating' in research is part of it. Wait, does participating in a study as a subject count for credit, or does working on a study count? Aren't studies explicit about the demographics of their participants? And don't studies that make generalized claims usually control for things like gender and age?
- mettamage 8y ago
  I'm not sure whether this is a rant, or informing or both. Let me know if it's a bit too much on the ranting side, I'll edit it. > Wait, does participating in a study as a subject count for credit, or does working on a study count? There was this course called "measurements and diagnostics" and for that you had to pass the theory/tutorials of the course and also 10 hours of being a participant in research. If you didn't do the 10 hours of participation, you could do them at any moment you want and if you completed it, you immediately passed the course. My university has about 400 psychology students, so that is 4000 hours worth of people participating per year. And I'm pretty sure that more universities are doing this. In The Netherlands alone I estimate there to be about 40000 psychology students per year and probably about 5 hours of compulsory 'participation' in experimental studies. So a tiny country like The Netherlands is skewing this particular bias about 20000 participant hours per year. Participating in experimental studies is quite fun (pro tip: go for the fMRI and EEG studies ASAP! ;-) ), but I didn't appreciate how I was forced to do it. I didn't learn too much from those experiments and I felt used. It furthermore is a way to be lazy regarding participant recruitment. I'm pretty sure that some startup could disrupt this particular problem. If you have ideas about it, email me, I'm willing to brainstorm and help you to look for ways to solve this problem. > Aren't studies explicit about the demographics of their participants? And don't studies that make generalized claims usually control for things like gender and age? Studies are explicit about the demographics, which is why the WEIRD article could be written. And yes they do control for things like gender and age and in general they are quite careful making those claims. The issue is though as a research field psychology isn't the study of the human mind and human behavior but of WEIRD people their mind and WEIRD people their behavior. Like a lot of societal issues that are in popular media today, this is a systemic issue. Though I could imagine psychology researchers being like: "Hmm, I could get a representative group or I could just get some psychology students and churn my next paper out twice as fast!"
  - jackvalentine 8y ago
    Would it be fair to say that by being forced to participate in a study you aren’t really giving your consent? Even though I presume you get to find/pick the studies you participate in.
    - saagarjha 8y ago
      Generally, there are "alternative" assignments that you can do instead, but they're usually somewhat more annoying.
      - mettamage 8y ago
        In my case, we had no alternative. If you chose to study psychology at my uni, you had to do this.
        iguy 8y ago
        Was there some choice of what studies? So that if you object to a particular one you could decline... if you object to all of them then I guess you should study something else.
    - ACow_Adonis 8y ago
      Its not just fair, i'd say its accurate. Here's another interesting phenomenon: a certain personality type (of which I may or may not count my self among) will deliberately supply false data which attempts to invalidate what we perceive as your study goal when forced to participate against our wills in this way. Its a small rebellion against an academic system that doesn't respect us or intellectual integrity, but is focused on pumping out shoddy research on unwilling participants. Turn academics into a game and thats how it'll be treated.
      - dwringer 8y ago
        I remember this practice of almost-compelled research participation even in the introductory undergraduate Psychology 101 course, and in that class (which included high school students and tons of non-majors) there was a lot of the attitude you describe.
- tingyue 8y ago
  I'm taking an intro neuroscience course this term. We can participate in department studies for extra credit, of up to a 3% bump for the entire course.
- kriro 8y ago
  It's common practice in psychology degrees to have some of your ungraded but required credit basically be "participate in the studies of others". I get why they do it this way (otherwise there wouldn't be enough people willing to participate in undergrad psychology experiments). Personally I find this practice very unethical.
  - Retric 8y ago
    While true, that’s misleading as this is generally for class assignments not published research. Some overlap does exist, but it’s significantly less than suggested by the requirement.
    - rjf72 8y ago
      What do you mean class assignments? It may vary by university but at my school the studies we participated in were indeed being published. This also was not only for psychology majors but anybody who took 'psych 101' -- participation in at least 3 studies was required to receive credit for the class. It was fun, but it was seriously disillusioning to see how absurd most psychology studies are. Two researchers talk to me for 5 minutes about women performing better than men on certain forms of math tests. They then sit me in a room alone for about 10 minutes presumably to try to make me nervous. Finally they give me a test, overtly handing me it on a pink sheet of paper instead of a white one. Then after the test they continually tried to get me to agree that I felt like I probably didn't do so well on the test. I refused - since I was certain I nailed it (it was a cookie cutter pre-sat math type test). They told me I'd receive my results by mail within 2 months. I never did. E-mails asking for such were ignored. The research got published and was of course some trite gender bias explains all difference in math paper. At the time I just found the whole thing quite absurd. In hindsight, I think it was even worse. That study was prepped and ready for p-hacking. They were measuring and possibly varying a large number of variables there. There was the paper color, the isolation, the 'chat(s)', different sections of the test, were men/women over/under estimating their performance, etc. And they were actively trying to push people into agreeing with what they wanted to find. You're going to be able to find some sort of statistically significant correlation there, even if none actually exists.
- brudgers 8y ago
  Characterizing the research domain as universal characteristics of human psychology has the convenient side effect of making demographics irrelevant.
- PurpleBoxDragon 8y ago
  Getting research subjects is hard. For any significant sort of test beyond a simple survey, it is really really hard. So colleges force their undergrads to be participants so that the graduates have enough fodder to write research papers off. Of course, the IRB views this as unethical, so all the undergraduates are given an alternative to being a participant in research, at which point it becomes a question of crafting an alternative that is painful enough all the undergrads choose being a research subject but not so painful the IRB considers it unethical. I'm use to the alternative being a research paper that can easily take 20 times longer to complete than being a test subject.
mpweiher 8y ago
> psychology is secretly the study of particular > subset of 18 to 21 year old American women. This is perfect! I had a similar epiphany about linguistics, much of which appears to be the study of example sentences that linguists come up with. Since you could invariably tell when something was an example sentence, this was obviously a different language from what people actually used. (Of course there are linguists who go out and study real language-use, but some actively dismiss this as mostly irrelevant "performance")
- ar-jan 8y ago
  Those example sentences linguists come up with are actually very reliable. Here's Sprouse & Almeida. 2012. Assessing the reliability of textbook data in syntax: Adger's Core Syntax. https://doi.org/10.1017/S0022226712000011 https://doi.org/10.1017/S0022226712000011 " [...] This suggests that even under the (likely unwarranted) assumption that the discrepant results are all false positives that have found their way into the syntactic literature due to the shortcomings of traditional methods, the minimum replication rate of these 469 data points is 98%." 98% replicability!! Compare that with psychology.
  - mpweiher 8y ago
    Interesting result, but seems to answer a different question: do naive native speakers find those examples acceptable? The first problem is that asking people how they speak or what is acceptable does not accurately represent how they actually speak when not observed/asked: https://en.wikipedia.org/wiki/Observer%27s_paradox https://en.wikipedia.org/wiki/Observer%27s_paradox The second problem is that, even if the first isn't an issue, the most this can demonstrate is that linguistic example sentences are a subset of actual language.
    - ar-jan 8y ago
      That's a fair point, but then the question is what should the question be? There are many subfields and disciplines of linguistics all of which are imo fascinating. In theoretical/generative linguistics the distinction between acceptability and performance is relevant, and acceptability seems to be a very robust measure.
bjourne 8y ago
> Years after pondering this article, I felt I had figured it out: psychology is secretly the study of particular subset of 18 to 21 year old American women. There are many analogs of that in Computer science too. For example in computer vision, CIFAR-10 is a de facto standard for measuring performance. A set of 60k 32x32 images. Good results on that data set doesn't necessarily translate well into real-world performance. But what can you do? Gathering huge data sets and having humans annotate them is incredibly expensive. Another ml example is music recognition. There are several state-of-the-art methods for detecting notes in music. So one Chinese researcher tried to apply them for Jingju music https://www.youtube.com/watch?v=NJsTl342RhI https://www.youtube.com/watch?v=NJsTl342RhI. The results were... less than stellar.
staamen 8y ago
> Most of them are American because psychology is a bigger thing in the US than in Europe, or at least it seemed to be regarding well-known theories, so I presume most research happens there. I'm not sure about the amount of research but the amount of psychology students here in the Netherlands is huge. And it seems to function in much the same way. Often first-years get some form of credit for mandatory participation in experiments for higher-year students and sometimes regular faculty members. I'm also aware of some researchers going out of their way to get a different sample from the population but in the end it's a lot easier to get participants when you make them.
nabnob 8y ago
Do you have any studies tracking the sex of study participants in psychology? I know that in clinical trials, historically the majority of participants are male: "After the tragedies caused by the use of thalidomide in pregnant women, the FDA issued “General Considerations for the Clinical Evaluation of Drugs” in 1977. This guidance document stated that women of child-bearing potential should be excluded from Phase 1 and early Phase 2 research, except if these studies were being conducted to test a drug for a life-threatening illness. If a drug appeared to have a favorable risk-benefit assessment, women could then be included in later Phase 2 and Phase 3 trials if animal teratogenicity and fertility studies were finished...In 1993, FDA reversed the 1977 guidance with another guidance document entitled Guidelines for the Study and Evaluation of Gender Differences in the Clinical Evaluation of Drugs." Also: "In a study that evaluated the inclusion and analysis of sex in the results of federally-funded randomized clinical trials in nine major medical journals in 2009, researchers found most studies that were not sex-specific had an average enrollment of 37% women." From "Women’s involvement in clinical trials: historical perspective and future implications" - https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4800017/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4800017/
- dleslie 8y ago
  Don't clinical trials often pay? They could be attractive to the desperately impoverished single adults, who are mostly men.
  - nabnob 8y ago
    Source? Everything I've read says that women are more likely to live in poverty. Stats on poverty by sex and age in the United States: https://www.statista.com/statistics/233154/us-poverty-rate-by-gender/ https://www.statista.com/statistics/233154/us-poverty-rate-b... Worldwide, the gap is even wider: https://www.washingtonpost.com/news/worldviews/wp/2018/02/14/women-poorer-and-hungrier-than-men-across-the-world-u-n-report-says/?noredirect=on&utm_term=.507ae58ee338 https://www.washingtonpost.com/news/worldviews/wp/2018/02/14...
    - ajscanlan 8y ago
mcguire 8y ago
I believe a significant chunk of genetics is similarly limited, to Icelanders, for example.
Balgair 8y ago
> There is another big group. A lot of dead mice (neuroscience). I'd add in Beagles, Capuchins, Zebrafish, and Mongolian Gerbils there too.
joe_the_user 8y ago
The thing with the reproducibility crisis is that a lot of psychology theories have the "looking under light post effect" (a drunk looses his glasses in a river and a police officer finds looking under a light post, "because at least I can see here). It's been well known for centuries that human motivations are complex, socially layered and contextual. But most psychology theories wind-up fairly simplistic and not framed by multiple contexts because ... these can be easily tested (except apparently they can't be easily reproduced, uh..). I don't know any easy way out here. Just because "folk psychology" can say formal psychology is too simple doesn't mean folk psychology and anecdote are more useful.

simple_phrases 8y ago

> The ones who study psychology because it's a female dominated study and all those psych students need their credit and 'participating' in research is part of it. Wait, does participating in a study as a subject count for credit, or does working on a study count? Aren't studies explicit about the demographics of their participants? And don't studies that make generalized claims usually control for things like gender and age?

mettamage 8y ago
I'm not sure whether this is a rant, or informing or both. Let me know if it's a bit too much on the ranting side, I'll edit it. > Wait, does participating in a study as a subject count for credit, or does working on a study count? There was this course called "measurements and diagnostics" and for that you had to pass the theory/tutorials of the course and also 10 hours of being a participant in research. If you didn't do the 10 hours of participation, you could do them at any moment you want and if you completed it, you immediately passed the course. My university has about 400 psychology students, so that is 4000 hours worth of people participating per year. And I'm pretty sure that more universities are doing this. In The Netherlands alone I estimate there to be about 40000 psychology students per year and probably about 5 hours of compulsory 'participation' in experimental studies. So a tiny country like The Netherlands is skewing this particular bias about 20000 participant hours per year. Participating in experimental studies is quite fun (pro tip: go for the fMRI and EEG studies ASAP! ;-) ), but I didn't appreciate how I was forced to do it. I didn't learn too much from those experiments and I felt used. It furthermore is a way to be lazy regarding participant recruitment. I'm pretty sure that some startup could disrupt this particular problem. If you have ideas about it, email me, I'm willing to brainstorm and help you to look for ways to solve this problem. > Aren't studies explicit about the demographics of their participants? And don't studies that make generalized claims usually control for things like gender and age? Studies are explicit about the demographics, which is why the WEIRD article could be written. And yes they do control for things like gender and age and in general they are quite careful making those claims. The issue is though as a research field psychology isn't the study of the human mind and human behavior but of WEIRD people their mind and WEIRD people their behavior. Like a lot of societal issues that are in popular media today, this is a systemic issue. Though I could imagine psychology researchers being like: "Hmm, I could get a representative group or I could just get some psychology students and churn my next paper out twice as fast!"
- jackvalentine 8y ago
  Would it be fair to say that by being forced to participate in a study you aren’t really giving your consent? Even though I presume you get to find/pick the studies you participate in.
  - saagarjha 8y ago
    Generally, there are "alternative" assignments that you can do instead, but they're usually somewhat more annoying.
    - mettamage 8y ago
      In my case, we had no alternative. If you chose to study psychology at my uni, you had to do this.
      - iguy 8y ago
        Was there some choice of what studies? So that if you object to a particular one you could decline... if you object to all of them then I guess you should study something else.
  - ACow_Adonis 8y ago
    Its not just fair, i'd say its accurate. Here's another interesting phenomenon: a certain personality type (of which I may or may not count my self among) will deliberately supply false data which attempts to invalidate what we perceive as your study goal when forced to participate against our wills in this way. Its a small rebellion against an academic system that doesn't respect us or intellectual integrity, but is focused on pumping out shoddy research on unwilling participants. Turn academics into a game and thats how it'll be treated.
    - dwringer 8y ago
      I remember this practice of almost-compelled research participation even in the introductory undergraduate Psychology 101 course, and in that class (which included high school students and tons of non-majors) there was a lot of the attitude you describe.
tingyue 8y ago
I'm taking an intro neuroscience course this term. We can participate in department studies for extra credit, of up to a 3% bump for the entire course.
kriro 8y ago
It's common practice in psychology degrees to have some of your ungraded but required credit basically be "participate in the studies of others". I get why they do it this way (otherwise there wouldn't be enough people willing to participate in undergrad psychology experiments). Personally I find this practice very unethical.
- Retric 8y ago
  While true, that’s misleading as this is generally for class assignments not published research. Some overlap does exist, but it’s significantly less than suggested by the requirement.
  - rjf72 8y ago
    What do you mean class assignments? It may vary by university but at my school the studies we participated in were indeed being published. This also was not only for psychology majors but anybody who took 'psych 101' -- participation in at least 3 studies was required to receive credit for the class. It was fun, but it was seriously disillusioning to see how absurd most psychology studies are. Two researchers talk to me for 5 minutes about women performing better than men on certain forms of math tests. They then sit me in a room alone for about 10 minutes presumably to try to make me nervous. Finally they give me a test, overtly handing me it on a pink sheet of paper instead of a white one. Then after the test they continually tried to get me to agree that I felt like I probably didn't do so well on the test. I refused - since I was certain I nailed it (it was a cookie cutter pre-sat math type test). They told me I'd receive my results by mail within 2 months. I never did. E-mails asking for such were ignored. The research got published and was of course some trite gender bias explains all difference in math paper. At the time I just found the whole thing quite absurd. In hindsight, I think it was even worse. That study was prepped and ready for p-hacking. They were measuring and possibly varying a large number of variables there. There was the paper color, the isolation, the 'chat(s)', different sections of the test, were men/women over/under estimating their performance, etc. And they were actively trying to push people into agreeing with what they wanted to find. You're going to be able to find some sort of statistically significant correlation there, even if none actually exists.
brudgers 8y ago
Characterizing the research domain as universal characteristics of human psychology has the convenient side effect of making demographics irrelevant.
PurpleBoxDragon 8y ago
Getting research subjects is hard. For any significant sort of test beyond a simple survey, it is really really hard. So colleges force their undergrads to be participants so that the graduates have enough fodder to write research papers off. Of course, the IRB views this as unethical, so all the undergraduates are given an alternative to being a participant in research, at which point it becomes a question of crafting an alternative that is painful enough all the undergrads choose being a research subject but not so painful the IRB considers it unethical. I'm use to the alternative being a research paper that can easily take 20 times longer to complete than being a test subject.

mpweiher 8y ago

> psychology is secretly the study of particular > subset of 18 to 21 year old American women. This is perfect! I had a similar epiphany about linguistics, much of which appears to be the study of example sentences that linguists come up with. Since you could invariably tell when something was an example sentence, this was obviously a different language from what people actually used. (Of course there are linguists who go out and study real language-use, but some actively dismiss this as mostly irrelevant "performance")

ar-jan 8y ago
Those example sentences linguists come up with are actually very reliable. Here's Sprouse & Almeida. 2012. Assessing the reliability of textbook data in syntax: Adger's Core Syntax. https://doi.org/10.1017/S0022226712000011 https://doi.org/10.1017/S0022226712000011 " [...] This suggests that even under the (likely unwarranted) assumption that the discrepant results are all false positives that have found their way into the syntactic literature due to the shortcomings of traditional methods, the minimum replication rate of these 469 data points is 98%." 98% replicability!! Compare that with psychology.
- mpweiher 8y ago
  Interesting result, but seems to answer a different question: do naive native speakers find those examples acceptable? The first problem is that asking people how they speak or what is acceptable does not accurately represent how they actually speak when not observed/asked: https://en.wikipedia.org/wiki/Observer%27s_paradox https://en.wikipedia.org/wiki/Observer%27s_paradox The second problem is that, even if the first isn't an issue, the most this can demonstrate is that linguistic example sentences are a subset of actual language.
  - ar-jan 8y ago
    That's a fair point, but then the question is what should the question be? There are many subfields and disciplines of linguistics all of which are imo fascinating. In theoretical/generative linguistics the distinction between acceptability and performance is relevant, and acceptability seems to be a very robust measure.

bjourne 8y ago

> Years after pondering this article, I felt I had figured it out: psychology is secretly the study of particular subset of 18 to 21 year old American women. There are many analogs of that in Computer science too. For example in computer vision, CIFAR-10 is a de facto standard for measuring performance. A set of 60k 32x32 images. Good results on that data set doesn't necessarily translate well into real-world performance. But what can you do? Gathering huge data sets and having humans annotate them is incredibly expensive. Another ml example is music recognition. There are several state-of-the-art methods for detecting notes in music. So one Chinese researcher tried to apply them for Jingju music https://www.youtube.com/watch?v=NJsTl342RhI https://www.youtube.com/watch?v=NJsTl342RhI. The results were... less than stellar.

staamen 8y ago

> Most of them are American because psychology is a bigger thing in the US than in Europe, or at least it seemed to be regarding well-known theories, so I presume most research happens there. I'm not sure about the amount of research but the amount of psychology students here in the Netherlands is huge. And it seems to function in much the same way. Often first-years get some form of credit for mandatory participation in experiments for higher-year students and sometimes regular faculty members. I'm also aware of some researchers going out of their way to get a different sample from the population but in the end it's a lot easier to get participants when you make them.

nabnob 8y ago

Do you have any studies tracking the sex of study participants in psychology? I know that in clinical trials, historically the majority of participants are male: "After the tragedies caused by the use of thalidomide in pregnant women, the FDA issued “General Considerations for the Clinical Evaluation of Drugs” in 1977. This guidance document stated that women of child-bearing potential should be excluded from Phase 1 and early Phase 2 research, except if these studies were being conducted to test a drug for a life-threatening illness. If a drug appeared to have a favorable risk-benefit assessment, women could then be included in later Phase 2 and Phase 3 trials if animal teratogenicity and fertility studies were finished...In 1993, FDA reversed the 1977 guidance with another guidance document entitled Guidelines for the Study and Evaluation of Gender Differences in the Clinical Evaluation of Drugs." Also: "In a study that evaluated the inclusion and analysis of sex in the results of federally-funded randomized clinical trials in nine major medical journals in 2009, researchers found most studies that were not sex-specific had an average enrollment of 37% women." From "Women’s involvement in clinical trials: historical perspective and future implications" - https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4800017/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4800017/

dleslie 8y ago
Don't clinical trials often pay? They could be attractive to the desperately impoverished single adults, who are mostly men.
- nabnob 8y ago
  Source? Everything I've read says that women are more likely to live in poverty. Stats on poverty by sex and age in the United States: https://www.statista.com/statistics/233154/us-poverty-rate-by-gender/ https://www.statista.com/statistics/233154/us-poverty-rate-b... Worldwide, the gap is even wider: https://www.washingtonpost.com/news/worldviews/wp/2018/02/14/women-poorer-and-hungrier-than-men-across-the-world-u-n-report-says/?noredirect=on&utm_term=.507ae58ee338 https://www.washingtonpost.com/news/worldviews/wp/2018/02/14...
  - ajscanlan 8y ago

mcguire 8y ago

I believe a significant chunk of genetics is similarly limited, to Icelanders, for example.

Balgair 8y ago

> There is another big group. A lot of dead mice (neuroscience). I'd add in Beagles, Capuchins, Zebrafish, and Mongolian Gerbils there too.

joe_the_user 8y ago

The thing with the reproducibility crisis is that a lot of psychology theories have the "looking under light post effect" (a drunk looses his glasses in a river and a police officer finds looking under a light post, "because at least I can see here). It's been well known for centuries that human motivations are complex, socially layered and contextual. But most psychology theories wind-up fairly simplistic and not framed by multiple contexts because ... these can be easily tested (except apparently they can't be easily reproduced, uh..). I don't know any easy way out here. Just because "folk psychology" can say formal psychology is too simple doesn't mean folk psychology and anecdote are more useful.