4 ms·

The paper that this article is discussing was published in The Journal of Personality and Social Psychology. That's interesting for two reasons. The first is th

by rjf72 7y ago

The paper that this article is discussing was published in The Journal of Personality and Social Psychology. That's interesting for two reasons. The first is that it's a rather well regarded journal with a very high impact rate. The other is because it's one of the journals that's driving the ongoing replication crisis plaguing psychology. [1] In particular the journal this was published in was found to have an overall replication rate of 23%. Another way of putting this is that if you assumed the exact opposite of everything you read that was published in that journal (e.g. - what they claim is statistically significant, is not actually significant) you'd be vastly more informed than a person who took what was published in the journal at face value. Of course that does not mean that this study is 'false', but you should take it with the same degree of credulity that you'd take something published in a venue when you know 77% of what is published is false. I mention this largely because I think there's a substantial degree of dissonance between the crisis in psychology and people continuing to take studies in psychology at face value. It's illogical. [1] - https://en.wikipedia.org/wiki/Replication_crisis#Psychology_replication_rates https://en.wikipedia.org/wiki/Replication_crisis#Psychology_...

nosianu 7y ago
> if you assumed the exact opposite of everything you read that was published in that journal ... you'd be vastly more informed I don't think the opposite of not knowing is knowing (the opposite fact) :-) That said, I agree with your main point on an even wider level. I read such very general headlines and conclusions - and when I read the actual paper there is a very specific and very limited experiment, with very specific people in a very specific context. I'm always left wondering how one gets from such a specific thing to such generalizations, especially when it's about such a complex subject as human behavior where tiny variations can produce radically different outcomes. --- PS: Animats says it's paywalled, but the PDF on the linked page is accessible and has the data and details? About the N=150,949 part: > EFL collected the data from small-business owners at the time they were applying for a loan at a microfinance institution (MFI) in Mexico from 2015 to 2017. As part of the loan application process, a prospective borrower had to complete a short psychometric assessment developed by EFL. That's just what I said, it's very specific. > The psychometric assessment that EFL developed consists of many modules, one of which is relevant for the purposes of the present investigation: the flashcard game. The flashcard game is a cognitive test on memory and executive functioning. In the flashcard game, participants are presented with an image; after pressing a key, they are shown a second image. Participants are then asked to indicate whether the second image matches the first. The flashcard game is scored based on whether or not the participant correctly identifies a match. and > At the end of the flashcard exercise, participants saw a question that asked them to estimate how well they think they did on this particular exercise relative to other applicants.... This question was added to the psychometric application for the purpose of this study I don't know about others, but I have a hard time seeing such tests justifying the headlines.
- DoreenMichele 7y ago
  I'm always left wondering how one gets from such a specific thing to such generalizations In this case, I will suggest that someone is simply marrying general "common knowledge" to the study specifics. Classism is very real. I've become very aware of its effects in recent years. My mother's mother came from a low level noble family that sold the title. I grew up in a nice suburb, daughter of a retired decorated soldier. I was one of the top people in my high school, arguably one of the top people in my state academically, based on placing 3rd in a statewide competition and being a National Merit Scholarship winner. In recent years, I've been quite poor and spent several years homeless. My social experience suggests that my recent poverty completely wipes out the value of my academic accomplishments etc in the eyes of a great many people. It's been a huge problem for trying to get my life back. In the eyes of many people, I'm too poor to be considered competent enough to hire for anything.
  - nosianu 7y ago
    Oh I don't want to doubt the result, I think it's probably true. I even posted this on my FB account (which I only use to bookmark and possibly spread "sciency" links I find interesting, not for anything personal) - but then deleted it again. Because I have to admit as much as I personally like the result because it confirms my personal believes, I find the foundation the headline is based on shaky ground given that it's supposed to be science and not opinion. I can't complain about others if I find my own bias influences whether I ignore how results are obtained and just spread the headline hoping (and knowing) that other people (following my news feed) won't read the details.
    - DoreenMichele 7y ago
      I'm pretty socially observant and my adult sons are rather aspie. In raising them, I had to really up my game for trying to find data and explanations for social things. I once had a conversation with my oldest where I tried to say something like "anger is like fire" as a metaphor and he leapt to some conclusion like "so, you rub two people together and they explode?" Um, no. Forget that metaphor. Let me get back to you. So we spend a lot of time talking about both social things and how to find actually good information and data because psych studies are so often terribly done and they are inherently hard to do well because most studies are de facto a form of manipulation, which itself pollutes the data.
Animats 7y ago
The paper looks more interesting, but it's paywalled.[1] The study involved over 150,000 people, which is impressive. Too many psych studies get subjects from undergrad students, or worse, Mechanical Turk. Here's the first author's web page.[2] He seems to specialize in this area. Unfortunately, none of his papers listed link to the full text. [1] https://psycnet.apa.org/doiLanding?doi=10.1037%2Fpspi0000187 https://psycnet.apa.org/doiLanding?doi=10.1037%2Fpspi0000187 [2] https://www.darden.virginia.edu/faculty-research/directory/peter-belmi/ https://www.darden.virginia.edu/faculty-research/directory/p...
- rjf72 7y ago
  https://sci-hub.tw/#about https://sci-hub.tw/#about for bypassing paywalls. But I would not expect to find any huge red flag. This journal, in spite of having an absolutely abysmal replication rate, is one of the leading journals in psychology. It's not like the 77% of studies that can't be replicated were just obviously broken in some way. It's simply that when people try to do the same thing, the results do not hold. Though, in this case at least a peer comment provided some more insight into the 150k.

by rjf72 7y ago

nosianu 7y ago

> if you assumed the exact opposite of everything you read that was published in that journal ... you'd be vastly more informed I don't think the opposite of not knowing is knowing (the opposite fact) :-) That said, I agree with your main point on an even wider level. I read such very general headlines and conclusions - and when I read the actual paper there is a very specific and very limited experiment, with very specific people in a very specific context. I'm always left wondering how one gets from such a specific thing to such generalizations, especially when it's about such a complex subject as human behavior where tiny variations can produce radically different outcomes. --- PS: Animats says it's paywalled, but the PDF on the linked page is accessible and has the data and details? About the N=150,949 part: > EFL collected the data from small-business owners at the time they were applying for a loan at a microfinance institution (MFI) in Mexico from 2015 to 2017. As part of the loan application process, a prospective borrower had to complete a short psychometric assessment developed by EFL. That's just what I said, it's very specific. > The psychometric assessment that EFL developed consists of many modules, one of which is relevant for the purposes of the present investigation: the flashcard game. The flashcard game is a cognitive test on memory and executive functioning. In the flashcard game, participants are presented with an image; after pressing a key, they are shown a second image. Participants are then asked to indicate whether the second image matches the first. The flashcard game is scored based on whether or not the participant correctly identifies a match. and > At the end of the flashcard exercise, participants saw a question that asked them to estimate how well they think they did on this particular exercise relative to other applicants.... This question was added to the psychometric application for the purpose of this study I don't know about others, but I have a hard time seeing such tests justifying the headlines.

DoreenMichele 7y ago
I'm always left wondering how one gets from such a specific thing to such generalizations In this case, I will suggest that someone is simply marrying general "common knowledge" to the study specifics. Classism is very real. I've become very aware of its effects in recent years. My mother's mother came from a low level noble family that sold the title. I grew up in a nice suburb, daughter of a retired decorated soldier. I was one of the top people in my high school, arguably one of the top people in my state academically, based on placing 3rd in a statewide competition and being a National Merit Scholarship winner. In recent years, I've been quite poor and spent several years homeless. My social experience suggests that my recent poverty completely wipes out the value of my academic accomplishments etc in the eyes of a great many people. It's been a huge problem for trying to get my life back. In the eyes of many people, I'm too poor to be considered competent enough to hire for anything.
- nosianu 7y ago
  Oh I don't want to doubt the result, I think it's probably true. I even posted this on my FB account (which I only use to bookmark and possibly spread "sciency" links I find interesting, not for anything personal) - but then deleted it again. Because I have to admit as much as I personally like the result because it confirms my personal believes, I find the foundation the headline is based on shaky ground given that it's supposed to be science and not opinion. I can't complain about others if I find my own bias influences whether I ignore how results are obtained and just spread the headline hoping (and knowing) that other people (following my news feed) won't read the details.
  - DoreenMichele 7y ago
    I'm pretty socially observant and my adult sons are rather aspie. In raising them, I had to really up my game for trying to find data and explanations for social things. I once had a conversation with my oldest where I tried to say something like "anger is like fire" as a metaphor and he leapt to some conclusion like "so, you rub two people together and they explode?" Um, no. Forget that metaphor. Let me get back to you. So we spend a lot of time talking about both social things and how to find actually good information and data because psych studies are so often terribly done and they are inherently hard to do well because most studies are de facto a form of manipulation, which itself pollutes the data.

Animats 7y ago

The paper looks more interesting, but it's paywalled.[1] The study involved over 150,000 people, which is impressive. Too many psych studies get subjects from undergrad students, or worse, Mechanical Turk. Here's the first author's web page.[2] He seems to specialize in this area. Unfortunately, none of his papers listed link to the full text. [1] https://psycnet.apa.org/doiLanding?doi=10.1037%2Fpspi0000187 https://psycnet.apa.org/doiLanding?doi=10.1037%2Fpspi0000187 [2] https://www.darden.virginia.edu/faculty-research/directory/peter-belmi/ https://www.darden.virginia.edu/faculty-research/directory/p...

rjf72 7y ago
https://sci-hub.tw/#about https://sci-hub.tw/#about for bypassing paywalls. But I would not expect to find any huge red flag. This journal, in spite of having an absolutely abysmal replication rate, is one of the leading journals in psychology. It's not like the 77% of studies that can't be replicated were just obviously broken in some way. It's simply that when people try to do the same thing, the results do not hold. Though, in this case at least a peer comment provided some more insight into the 150k.