5 ms·

"IQ is Gaussian" – it was pointed out somewhere, and only then became obvious to me, that IQ is not Gaussian. The distribution is manufactured. If you have 100

"IQ is Gaussian" – it was pointed out somewhere, and only then became obvious to me, that IQ is not Gaussian. The distribution is manufactured. If you have 1000 possible IQ questions, you can ask a bunch of people those questions, and then pick out 100 questions that form a Gaussian distribution. This is how IQ tests are created. This is not unreasonable... if you picked out 100 super easy questions you wouldn't get much information, everyone would be in the "knows quite a lot" category. But you could try to create a uniform distribution, for instance, and still have a test that is usefully sensitive. But if you worry about the accuracy of the test then a Gaussian distribution is kind of convenient... there's this expectation that 50th percentile is not that different than 55th percentile, and people mostly care about that 5% difference only with 90th vs 95th. (But I don't think people care much about the difference between 10th percentile and 5th... which might imply an actual Pareto distribution, though I think it probably reflects more on societal attention) Anyway, kind of an aside, but also similar to what the article itself is talking about

EVa5I7bHFq9mnYK 2y ago
Sum of N independent similarly distributed variables (questions), will tend to be normally distributed, that the central limit theorem, no need to manufacture anything.
- mjburgess 2y ago
  They're not independent.
  - deleted 2y ago
    [deleted]
  - EVa5I7bHFq9mnYK 2y ago
    Yeah, if one answers question A correctly, they is more likely to answer question B correctly, right?
    - mjburgess 2y ago
      Indeed. The whole premise of the activity is that they are highly correlated. The imposition of a normal distribution is done ad-hoc at the population level. All it says is that if scores were normally distributed, then "people would be so-and-so comparable". Almost all assumptions of this method are false. Any time anyone mentions the central limit theorem in applied stats is a warning sign for pseudoscience. If reality existed at the end of the CLT, it would be in heat death.
CSMastermind 2y ago
IQ scores have proven highly correlated to educational achievement, occupational attainment, career advancement, lifetime earnings, brain volume, cortical thickness, health, longevity, and more. To the point of being accurate predictors of these things even when controlling for things like socioeconomic background. It's used because it works as a measuring tool, how the tests are constructed is largely irrelevant to the question of if the outcome of the test is an accurate predictor of things we care about. If you think you have a better measuring tool you should propose it and win several awards and accolades. No one has found one yet in spite of many smart people trying for decades.
- jprete 2y ago
  The GP is saying that IQ tests are deliberately calibrated and normalized to produce a Gaussian output, and that the input is not necessarily a Gaussian distribution of any particular quantity. This doesn't say anything in particular about whether it's useful, just that people should be careful interpreting the values directly.
  - lokar 2y ago
    Exactly. This is a criticism of the article where it says that HR has a good reason for assuming employee performance would be Gaussian, since IQ is Gaussian. IQ is defined a being Gaussian
    - ip26 2y ago
      If IQ is a good predictor of employee performance, then it does follow that employee performance would be Gaussian. It doesn’t matter that IQ was “made” to be Gaussian.
      - lokar 2y ago
        Are you assuming that employee performance is Gaussian?
      - Majromax 2y ago
torginus 2y ago
Yes, this has always bothered me. IQ doesn't easily correspond to any measurable real-world quality. For example, if we would postulate that height is gaussian, we could measure people's heights and just ordering them we could create a gaussian distribution. Then we could verify the hypothesis of height being gaussian by mapping the probability distribution function's parameter to a linear value (cm) and find that these approaches line up experimentally. We could do the same thing with any comparable quantity and make an order of them and try to map them to a gaussian distribution, but we would have no knowledge if what we were making actually corresponded to a linear quantity. This is a serious issue, as basically making any claim like 'group A scores 5 points higher than group B' is automatically, mathematically invalid.
zo1 2y ago
People may find that manufactured or "oh IQ is just made up and there is no measure of intelligence". But I find beauty in the way that IQ tests create and reconfigure a distribution across a multi-dimensional vector or dimensional space. It figures out what we need in the general case, and allows us to use and reason with it, without ever having to do the grunt work or arguably impossible task of finding out an actual measure of intelligence or some way to untangle the way a brain works.
- tptacek 2y ago
  That's a problem with it: its high legibility masks the complex (and deceptively muddy) math underneath it. Cosma Shalizi's "Statistical Myth" essay is a good dive into this; the "general factor" underneath all the different IQ tests is more or less a statistical inevitability, reproducing even with totally random tests.
seizethecheese 2y ago
I think your comment about an easy test having everyone in the “knows a lot” category hints that the reverse (a hard test) would be Pareto distributed.
SideQuark 2y ago
That’s not how IQ tests are made as can be found by reading how they’re actually made via Google scholar. And it would be spectacularly hard to do what you describe. How they’re actually made is a batch of questions thought to take some form of reasoning are curated, then ALL of those questions are used in the test. It is an empirical fact the percentages of decent sized groups of people will score a bell curve, in exactly the same way humans do on hard calc exams, on hard writing items, on chess problems, and across a bewildering amount of mental tasks, none of which are preselected and fidgeted with to fake a Gaussian. A simple example: see how many simple arithmetic problems people can do in fixed time. What do you find? Gaussian. No need to fiddle with removing pesky problem. Do reading. Do repeat this sequence for length. Just about any single class of questions has the same bell curve output in human mental ability. The curve may bend based on some inherent difficulty, say addition versus calculus, but there will be a bell curve. Now take plenty of types of questions to address various wobbles in people’s knowledge, upbringing, culture, etc, giving a host of bell curves per category (and those also correlated by individual). Then the sum of gaussians is gausdian. All IQ tests do is shift the mean score to be called 100 (normalized) and the std dev to match a preset amount of people so such tests can be compared over time. And the empirical evidence is these curves do strongly correlate over time, so scaling a test to align with this underlying g factor is well founded. This latter fact, that score on one form of intelligence seems to transfer well to others, forms the basis of modern intelligence research on the g factor. IQ tests correlate well with this g factor. And across all sorts of things the results are bell curves. For anyone wanting to hear all this and a ton more, Lex Fridman has an excellent interview with a state of the art intelligence researcher at https://www.youtube.com/watch?v=hppbxV9C63g https://www.youtube.com/watch?v=hppbxV9C63g. The researcher goes into great depth on what researchers do know, how they know it, what they don’t know, and what has been proven wrong. This is all there.
FredPret 2y ago
This is a subtle aspect of intelligence measurement that not many people think about. To go from an IQ of 100 to 130 might require an increase in brainpower of x, and from 130 to 170 might require 3x for example, and from 170-171 might be 9x compared to 100. We have to have a relative scale and contrive a Gaussian from the scores because we don’t have an absolute measure of intelligence. It would be a monumental achievement if computer science ever advances to the point where we have a mathematical way of determining the minimum absolute intelligence required to solve a given problem.
- groby_b 2y ago
  > It would be a monumental achievement if computer science ever advances to the point where we have a mathematical way of determining the minimum absolute intelligence required to solve a given problem. While that would be nice, it's likely a pipe dream :( There's a good chance "intelligence" is really a multi-dimensional thing influenced by a lot of different factors. We like pretending it's one-dimensional so we can sort folks (and money reinforces that one-dimensional thinking), but that means setting ourselves up for failure. It doesn't help that the tests we currently have (e.g. IQ) are deeply flawed and taint any thinking about the space. (Not least because folks who took a test and scored well are deeply invested in that test being right ;)
  - FredPret 2y ago
    It might be the hardest problem of them all, because you'd have to understand how all problems work. But on the other hand, maybe it all comes down to a Turing machine requiring a particular length of tape and runtime.
  - nextn 2y ago
    What is a flaw of the IQ test?
    - groby_b 2y ago
      There is no "the IQ test". The most prominent ones are Stanford-Binet and Wechsler. That, I think is the first problem. There isn't a single agreement what IQ is or how to measure it. There isn't a single one for good reasons, because they all measure slightly different things. But that means that fundamentally any single IQ scale is likely flawed. (Wechsler acknowledges this. SB sorta does as well, but hides it well) But if we're looking for a second at Stanford Binet : It's hard to administer. Scoring requires subjective judgment. It's sexist. It uses language and situations that don't apply to current times. It's highly verbal. The normative sample is questionable (though SB-V has gotten better) And because I've had this discussion before: I'm not saying IQ tests are completely meaningless. Yes, there's some signal there. But it's so deeply flawed signal that building rigorous science on top of it is just hard to impossible.
fnordlord 2y ago
I didn't know that about how IQ tests are formed. Would that mean that there could be some sliver of the population that could score in the top %'s on the 1000 question test but due to the selection of questions, scored average on the final IQ exam? If so, that'll be my excuse next time I have to take an IQ exam. I just got the wrong distribution.
marcosdumay 2y ago
It's worse, because every test is obviously bounded, and it's absurd to not expect some noise to be there. Join those two, and the test only becomes reasonable near the middle. But the middle is exactly where the pick of questions makes the most difference. All said, this means that IQ is kinda useful for sociological studies with large samples. But if you use it you are adding error, it's not reasonable to expect that error not to correlate with whatever you are looking at (since nobody understands it well), and it's not reasonable to expect the results to be stable. And it's really useless to make decisions based on small sample sizes.
jppope 2y ago
Correct. IQ isn't an effective measurement of intelligence as is typically stated. It is (at best) a measurement of learning disabilities.
- tptacek 2y ago
  I think this would be more accurate without the "at best"; I think IQ is widely considered to be a useful diagnostic measure, misapplied to prediction in generalized populations.
- smj-edison 2y ago
  I think IQ is useful in aggregate (for example, a finding that exposure to local toxins reduces a cities' performance on IQ by 10 points), but not useful an an individual level (e.g. you have an IQ of 130, so we can say with certainty you will earn $30,000 more per year). It's similar with MRI scans of ADHD: they find brain differences at a large scale, but you can't use a MRI to diagnose ADHD.
- liontwist 2y ago
  It’s a pretty good measurement of your ability to play logic games and fast pattern match. I’m sure we agree that doesn’t constitute “intelligence”, but it’s more than disability.
  - mjburgess 2y ago
    Individual test-retest variability is high. It's only a valid measure of anything much below 100. Consider a test of walking speed which each time you take it gives results of (2, 3, 6, 2, 3, 5, 7, 3) etc. -- does this measure some innate property of walking speed? No. Yet, if it were < 1, it would measure having a broken foot.
    - liontwist 2y ago
      Lots of research disagrees with you indicating it’s measurable and rigid throughout most of your life.
sapiogram 2y ago
> and then pick out 100 questions that form a Gaussian distribution. This is how IQ tests are created. You missed an extremely important final step. People's scores on those 100 questions still aren't going to form a Gaussion distribution. You have to rank-order everyone's scores, then you assign the final IQ scores based on each person's ranking, not their raw score.
- fwip 2y ago
  It would form a gaussian distribution if you pick the questions carefully enough. If you rank-order scores and fit to the distribution after the fact, the questions are nearly irrelevant, as long as you have a mix of easy, medium and hard questions.
  - sapiogram 2y ago
    > It would form a gaussian distribution if you pick the questions carefully enough. Why would that be the case? The Central Limit Theorem does not apply here, because the observations (questions) are correlated with each other.

EVa5I7bHFq9mnYK 2y ago

Sum of N independent similarly distributed variables (questions), will tend to be normally distributed, that the central limit theorem, no need to manufacture anything.

mjburgess 2y ago
They're not independent.
- deleted 2y ago
  [deleted]
- EVa5I7bHFq9mnYK 2y ago
  Yeah, if one answers question A correctly, they is more likely to answer question B correctly, right?
  - mjburgess 2y ago
    Indeed. The whole premise of the activity is that they are highly correlated. The imposition of a normal distribution is done ad-hoc at the population level. All it says is that if scores were normally distributed, then "people would be so-and-so comparable". Almost all assumptions of this method are false. Any time anyone mentions the central limit theorem in applied stats is a warning sign for pseudoscience. If reality existed at the end of the CLT, it would be in heat death.

CSMastermind 2y ago

IQ scores have proven highly correlated to educational achievement, occupational attainment, career advancement, lifetime earnings, brain volume, cortical thickness, health, longevity, and more. To the point of being accurate predictors of these things even when controlling for things like socioeconomic background. It's used because it works as a measuring tool, how the tests are constructed is largely irrelevant to the question of if the outcome of the test is an accurate predictor of things we care about. If you think you have a better measuring tool you should propose it and win several awards and accolades. No one has found one yet in spite of many smart people trying for decades.

jprete 2y ago
The GP is saying that IQ tests are deliberately calibrated and normalized to produce a Gaussian output, and that the input is not necessarily a Gaussian distribution of any particular quantity. This doesn't say anything in particular about whether it's useful, just that people should be careful interpreting the values directly.
- lokar 2y ago
  Exactly. This is a criticism of the article where it says that HR has a good reason for assuming employee performance would be Gaussian, since IQ is Gaussian. IQ is defined a being Gaussian
  - ip26 2y ago
    If IQ is a good predictor of employee performance, then it does follow that employee performance would be Gaussian. It doesn’t matter that IQ was “made” to be Gaussian.
    - lokar 2y ago
      Are you assuming that employee performance is Gaussian?
    - Majromax 2y ago

torginus 2y ago

Yes, this has always bothered me. IQ doesn't easily correspond to any measurable real-world quality. For example, if we would postulate that height is gaussian, we could measure people's heights and just ordering them we could create a gaussian distribution. Then we could verify the hypothesis of height being gaussian by mapping the probability distribution function's parameter to a linear value (cm) and find that these approaches line up experimentally. We could do the same thing with any comparable quantity and make an order of them and try to map them to a gaussian distribution, but we would have no knowledge if what we were making actually corresponded to a linear quantity. This is a serious issue, as basically making any claim like 'group A scores 5 points higher than group B' is automatically, mathematically invalid.

zo1 2y ago

People may find that manufactured or "oh IQ is just made up and there is no measure of intelligence". But I find beauty in the way that IQ tests create and reconfigure a distribution across a multi-dimensional vector or dimensional space. It figures out what we need in the general case, and allows us to use and reason with it, without ever having to do the grunt work or arguably impossible task of finding out an actual measure of intelligence or some way to untangle the way a brain works.

tptacek 2y ago
That's a problem with it: its high legibility masks the complex (and deceptively muddy) math underneath it. Cosma Shalizi's "Statistical Myth" essay is a good dive into this; the "general factor" underneath all the different IQ tests is more or less a statistical inevitability, reproducing even with totally random tests.

seizethecheese 2y ago

I think your comment about an easy test having everyone in the “knows a lot” category hints that the reverse (a hard test) would be Pareto distributed.

SideQuark 2y ago

That’s not how IQ tests are made as can be found by reading how they’re actually made via Google scholar. And it would be spectacularly hard to do what you describe. How they’re actually made is a batch of questions thought to take some form of reasoning are curated, then ALL of those questions are used in the test. It is an empirical fact the percentages of decent sized groups of people will score a bell curve, in exactly the same way humans do on hard calc exams, on hard writing items, on chess problems, and across a bewildering amount of mental tasks, none of which are preselected and fidgeted with to fake a Gaussian. A simple example: see how many simple arithmetic problems people can do in fixed time. What do you find? Gaussian. No need to fiddle with removing pesky problem. Do reading. Do repeat this sequence for length. Just about any single class of questions has the same bell curve output in human mental ability. The curve may bend based on some inherent difficulty, say addition versus calculus, but there will be a bell curve. Now take plenty of types of questions to address various wobbles in people’s knowledge, upbringing, culture, etc, giving a host of bell curves per category (and those also correlated by individual). Then the sum of gaussians is gausdian. All IQ tests do is shift the mean score to be called 100 (normalized) and the std dev to match a preset amount of people so such tests can be compared over time. And the empirical evidence is these curves do strongly correlate over time, so scaling a test to align with this underlying g factor is well founded. This latter fact, that score on one form of intelligence seems to transfer well to others, forms the basis of modern intelligence research on the g factor. IQ tests correlate well with this g factor. And across all sorts of things the results are bell curves. For anyone wanting to hear all this and a ton more, Lex Fridman has an excellent interview with a state of the art intelligence researcher at https://www.youtube.com/watch?v=hppbxV9C63g https://www.youtube.com/watch?v=hppbxV9C63g. The researcher goes into great depth on what researchers do know, how they know it, what they don’t know, and what has been proven wrong. This is all there.

FredPret 2y ago

This is a subtle aspect of intelligence measurement that not many people think about. To go from an IQ of 100 to 130 might require an increase in brainpower of x, and from 130 to 170 might require 3x for example, and from 170-171 might be 9x compared to 100. We have to have a relative scale and contrive a Gaussian from the scores because we don’t have an absolute measure of intelligence. It would be a monumental achievement if computer science ever advances to the point where we have a mathematical way of determining the minimum absolute intelligence required to solve a given problem.

groby_b 2y ago
> It would be a monumental achievement if computer science ever advances to the point where we have a mathematical way of determining the minimum absolute intelligence required to solve a given problem. While that would be nice, it's likely a pipe dream :( There's a good chance "intelligence" is really a multi-dimensional thing influenced by a lot of different factors. We like pretending it's one-dimensional so we can sort folks (and money reinforces that one-dimensional thinking), but that means setting ourselves up for failure. It doesn't help that the tests we currently have (e.g. IQ) are deeply flawed and taint any thinking about the space. (Not least because folks who took a test and scored well are deeply invested in that test being right ;)
- FredPret 2y ago
  It might be the hardest problem of them all, because you'd have to understand how all problems work. But on the other hand, maybe it all comes down to a Turing machine requiring a particular length of tape and runtime.
- nextn 2y ago
  What is a flaw of the IQ test?
  - groby_b 2y ago
    There is no "the IQ test". The most prominent ones are Stanford-Binet and Wechsler. That, I think is the first problem. There isn't a single agreement what IQ is or how to measure it. There isn't a single one for good reasons, because they all measure slightly different things. But that means that fundamentally any single IQ scale is likely flawed. (Wechsler acknowledges this. SB sorta does as well, but hides it well) But if we're looking for a second at Stanford Binet : It's hard to administer. Scoring requires subjective judgment. It's sexist. It uses language and situations that don't apply to current times. It's highly verbal. The normative sample is questionable (though SB-V has gotten better) And because I've had this discussion before: I'm not saying IQ tests are completely meaningless. Yes, there's some signal there. But it's so deeply flawed signal that building rigorous science on top of it is just hard to impossible.

fnordlord 2y ago

I didn't know that about how IQ tests are formed. Would that mean that there could be some sliver of the population that could score in the top %'s on the 1000 question test but due to the selection of questions, scored average on the final IQ exam? If so, that'll be my excuse next time I have to take an IQ exam. I just got the wrong distribution.

marcosdumay 2y ago

It's worse, because every test is obviously bounded, and it's absurd to not expect some noise to be there. Join those two, and the test only becomes reasonable near the middle. But the middle is exactly where the pick of questions makes the most difference. All said, this means that IQ is kinda useful for sociological studies with large samples. But if you use it you are adding error, it's not reasonable to expect that error not to correlate with whatever you are looking at (since nobody understands it well), and it's not reasonable to expect the results to be stable. And it's really useless to make decisions based on small sample sizes.

jppope 2y ago

Correct. IQ isn't an effective measurement of intelligence as is typically stated. It is (at best) a measurement of learning disabilities.

tptacek 2y ago
I think this would be more accurate without the "at best"; I think IQ is widely considered to be a useful diagnostic measure, misapplied to prediction in generalized populations.
smj-edison 2y ago
I think IQ is useful in aggregate (for example, a finding that exposure to local toxins reduces a cities' performance on IQ by 10 points), but not useful an an individual level (e.g. you have an IQ of 130, so we can say with certainty you will earn $30,000 more per year). It's similar with MRI scans of ADHD: they find brain differences at a large scale, but you can't use a MRI to diagnose ADHD.
liontwist 2y ago
It’s a pretty good measurement of your ability to play logic games and fast pattern match. I’m sure we agree that doesn’t constitute “intelligence”, but it’s more than disability.
- mjburgess 2y ago
  Individual test-retest variability is high. It's only a valid measure of anything much below 100. Consider a test of walking speed which each time you take it gives results of (2, 3, 6, 2, 3, 5, 7, 3) etc. -- does this measure some innate property of walking speed? No. Yet, if it were < 1, it would measure having a broken foot.
  - liontwist 2y ago
    Lots of research disagrees with you indicating it’s measurable and rigid throughout most of your life.

sapiogram 2y ago

> and then pick out 100 questions that form a Gaussian distribution. This is how IQ tests are created. You missed an extremely important final step. People's scores on those 100 questions still aren't going to form a Gaussion distribution. You have to rank-order everyone's scores, then you assign the final IQ scores based on each person's ranking, not their raw score.

fwip 2y ago
It would form a gaussian distribution if you pick the questions carefully enough. If you rank-order scores and fit to the distribution after the fact, the questions are nearly irrelevant, as long as you have a mix of easy, medium and hard questions.
- sapiogram 2y ago
  > It would form a gaussian distribution if you pick the questions carefully enough. Why would that be the case? The Central Limit Theorem does not apply here, because the observations (questions) are correlated with each other.