3 ms·

Speaking as someone with a cognitive science background (and yes, probably too lazy to dig up specific references), I'll say that there exist a lot of poorly de

by pfooti 11y ago

Speaking as someone with a cognitive science background (and yes, probably too lazy to dig up specific references), I'll say that there exist a lot of poorly designed or poorly implemented tests of "intelligence". Even the concept of intelligence is poorly defined, and lots of different measures are just proxies. Even if the test is good and has some psychometric utility, often giving the test is a giant pain to do well. Even if the test is good, it's probably calibrated on WEIRD [0] subjects, so is really only good as another kind of gatekeeper - you can probably use old-school IQ tests as an excuse to reduce diversity. People who aren't WEIRD may not do as well on these tests (even though they're just as "intelligent" (parenthetically bracketing the ill-defined term for a moment)), so you have your reason that you only hire white dudes. But that's a cynical extension of logic there. Most people, when approaching a psychometric test like one of intelligence or personality make the first incorrect assumption that modern cognitive science can actually form a construct like "personality type" or "intelligence" that's stable across all cultures and norms, testable, and repeatable. That's just not really true, and by and large what makes it to public consumption is pseudo-science. Diverting just a little bit here, but MBTI [1] is another good example of bad use of psychometrics in business. There's about as much science supporting Myers-Briggs tests as there is astrology, yet people still use MBTI for actual decision making. Types aren't shown to be stable and they're not well-clustered (meaning you can be a mix of introvert and extrovert, or show signs of both depending on the day, context, mood). Zodiac Sign and MBTI Type should only be used as pick-up lines in a bar. "Hey, babe, I'm an INTJ, so I'm not going to say anything else." There are good uses of psychometrics, but they're rare enough that I take any reliance on any kind of psychometric during hiring as kind of a bad sign. 0: http://blogs.scientificamerican.com/primate-diaries/the-weird-evolution-of-human-psychology/ http://blogs.scientificamerican.com/primate-diaries/the-weir... 1: https://www.psychologytoday.com/blog/give-and-take/201309/goodbye-mbti-the-fad-won-t-die https://www.psychologytoday.com/blog/give-and-take/201309/go...

deleted 11y ago
[deleted]
microcolonel 11y ago
I find the "WEIRD" thing a bit tough to support, at least with regard to IQ. East Asian populations (Hong Kong, South Korea, and Japan right at the top) seem to consistently score higher on IQ tests than "WEIRD"os. Maybe the less pathetic and self-hating way to look at it would be "EIR" populations. Even then, it's hard to argue that spatial and linguistic reasoning are not universal values; even if you're uncomfortable that some populations have lower averages than others. I feel as though this must come from a lack of respect for the obvious personal value of Education, Industrialization, and Riches.
- pfooti 11y ago
  You're making two different arguments here and kind of tying them together. First: you're claiming that there actually is some kind of universal "spatial reasoning" skill. Usually people who claim that also believe that this skill is somewhat generic - it applies to lots of different contexts and transfers easily from (say) visualizing rotations in space to finding your way around a 3d environment and so on. There's actually a fair bit of research that questions this supposition (take a look at embodied cognition to challenge the theoretical model, or stereotype threat effects to challenge the assessment model). Secondly, in this context, the assumption is that not only does this generalized attribute exist in humanity and we can measure it, but this attribute causes success in $job. That's a further assumption that I'd call into question. Sure, plenty of people who do well on $test do well on $job, but those could easily be covariate or epiphenomenological relationships rather than causal ones. I don't doubt that for some values of $test, there's some discriminatory power predicting the presence of success, especially in more monocultural environments (affluent, often-white, often-male, often-young startups, to caricature). But I do believe that this kind of test would suffer from strong false negatives as many kinds of people who would otherwise be excellent programmers (or whatever other $job) are rejected for not doing well on $test. That's why I worry about anyone who puts real stock in these kinds of tests - it is (generally) indicative of a kind of science blind spot in the user. People are critical and skeptical of all kinds of other scientific claims, but psychological measures tend to get a pass. I imagine the fact that a lot of us tend to score well on IQ tests has a lot to contribute here. When I found out how many standard deviations above the mean my IQ score represented, I was pretty excited about how awesome I was. When I realized that the test was probably baloney, I had to figure out more interesting ways to shore up my self-esteem. Personally: I bake bread and fish for compliments on the quality of my sourdough.
YeGoblynQueenne 11y ago
Thank you! It's refreshing to hear that from anyone with a cognitive science background, as you put it. I had an MBTI test pretty much forced on me a while ago (senior engineer would be unhappy if I didn't take it). I argued very forcefully against it, saying it was like astrology, that it made me feel like the company was asking me to participate in some religious ritual when I was an atheist and so on. I'm not in that company anymore and one major reason I ran was that they allowed pop psychology like that to dictate decisions.
disgruntledphd2 11y ago
To be fair, I think that Raven's matrices probably get around this. If someone explains the process to you, you don't actually need to be able to read to do well on the test. https://en.wikipedia.org/wiki/Raven%27s_Progressive_Matrices https://en.wikipedia.org/wiki/Raven%27s_Progressive_Matrices I do find it surprising that Raven's appears to be more susceptable to the Flynn effect (people keep getting better at IQ tests, for an unknown reason). Completely agreed on the MBTI, but its surprisingly difficult to convince people that's its useful. FWIW, I agree with most of what you said above, I was just pointing out a counter-example. (Psychologist/psychometrician here).
- pfooti 11y ago
  Yes, I agree that Raven's matrices are pretty good at measuring something in a way that's not super-reliant on language. I'm not inherently opposed to the notion of generalized intelligence as a construct either, I just think measuring it is pretty tricky. The Flynn effect is another great example - these instruments are nuanced, and it's important to really understand them in order to interpret their results.
  - disgruntledphd2 11y ago
    Totally agree, I'm somewhat in sympathy with Shalizi's argument that g is just the result of iterated factor analysis. That being said, done well, these kinds of tests can provide useful data, especially if you're willing to randomly hire to calibrate performance (which no one ever is, sadly).

by pfooti 11y ago

deleted 11y ago

[deleted]

microcolonel 11y ago

I find the "WEIRD" thing a bit tough to support, at least with regard to IQ. East Asian populations (Hong Kong, South Korea, and Japan right at the top) seem to consistently score higher on IQ tests than "WEIRD"os. Maybe the less pathetic and self-hating way to look at it would be "EIR" populations. Even then, it's hard to argue that spatial and linguistic reasoning are not universal values; even if you're uncomfortable that some populations have lower averages than others. I feel as though this must come from a lack of respect for the obvious personal value of Education, Industrialization, and Riches.

pfooti 11y ago
You're making two different arguments here and kind of tying them together. First: you're claiming that there actually is some kind of universal "spatial reasoning" skill. Usually people who claim that also believe that this skill is somewhat generic - it applies to lots of different contexts and transfers easily from (say) visualizing rotations in space to finding your way around a 3d environment and so on. There's actually a fair bit of research that questions this supposition (take a look at embodied cognition to challenge the theoretical model, or stereotype threat effects to challenge the assessment model). Secondly, in this context, the assumption is that not only does this generalized attribute exist in humanity and we can measure it, but this attribute causes success in $job. That's a further assumption that I'd call into question. Sure, plenty of people who do well on $test do well on $job, but those could easily be covariate or epiphenomenological relationships rather than causal ones. I don't doubt that for some values of $test, there's some discriminatory power predicting the presence of success, especially in more monocultural environments (affluent, often-white, often-male, often-young startups, to caricature). But I do believe that this kind of test would suffer from strong false negatives as many kinds of people who would otherwise be excellent programmers (or whatever other $job) are rejected for not doing well on $test. That's why I worry about anyone who puts real stock in these kinds of tests - it is (generally) indicative of a kind of science blind spot in the user. People are critical and skeptical of all kinds of other scientific claims, but psychological measures tend to get a pass. I imagine the fact that a lot of us tend to score well on IQ tests has a lot to contribute here. When I found out how many standard deviations above the mean my IQ score represented, I was pretty excited about how awesome I was. When I realized that the test was probably baloney, I had to figure out more interesting ways to shore up my self-esteem. Personally: I bake bread and fish for compliments on the quality of my sourdough.

YeGoblynQueenne 11y ago

Thank you! It's refreshing to hear that from anyone with a cognitive science background, as you put it. I had an MBTI test pretty much forced on me a while ago (senior engineer would be unhappy if I didn't take it). I argued very forcefully against it, saying it was like astrology, that it made me feel like the company was asking me to participate in some religious ritual when I was an atheist and so on. I'm not in that company anymore and one major reason I ran was that they allowed pop psychology like that to dictate decisions.

disgruntledphd2 11y ago

To be fair, I think that Raven's matrices probably get around this. If someone explains the process to you, you don't actually need to be able to read to do well on the test. https://en.wikipedia.org/wiki/Raven%27s_Progressive_Matrices https://en.wikipedia.org/wiki/Raven%27s_Progressive_Matrices I do find it surprising that Raven's appears to be more susceptable to the Flynn effect (people keep getting better at IQ tests, for an unknown reason). Completely agreed on the MBTI, but its surprisingly difficult to convince people that's its useful. FWIW, I agree with most of what you said above, I was just pointing out a counter-example. (Psychologist/psychometrician here).

pfooti 11y ago
Yes, I agree that Raven's matrices are pretty good at measuring something in a way that's not super-reliant on language. I'm not inherently opposed to the notion of generalized intelligence as a construct either, I just think measuring it is pretty tricky. The Flynn effect is another great example - these instruments are nuanced, and it's important to really understand them in order to interpret their results.
- disgruntledphd2 11y ago
  Totally agree, I'm somewhat in sympathy with Shalizi's argument that g is just the result of iterated factor analysis. That being said, done well, these kinds of tests can provide useful data, especially if you're willing to randomly hire to calibrate performance (which no one ever is, sadly).