5 ms·

But we know that it’s true. That’s why we have been using objective metrics like test scores for millennia, across societies are different as China, India, and

But we know that it’s true. That’s why we have been using objective metrics like test scores for millennia, across societies are different as China, India, and Britain.

Latty 1y ago
Your best argument is "we've done it for a long time, so it can't be wrong"? Quite the contrary: there is a long history of "objective" tests being shown to be deeply flawed and biased towards certain factors (often cultural and class based), we explicitly know it isn't the case that test scores are purely about some innate intelligence characteristic: there is a reason the rich spend a lot of money to raise their children's scores. My secondary school claimed to have the best results for Business Studies A-levels in the country. They achieved this by taking the pre-released case study, writing every possible question they could think of about the study, writing model answers, and telling the students to memorise them. The idea that these scores represent some innate intelligence of the student is obviously nonsense if you interact with the system at all.
- rayiner 1y ago
  The notion that the British A levels have “cultural bias” is absurd, given that Asians outperform white British: https://www.ethnicity-facts-figures.service.gov.uk/education-skills-and-training/a-levels-apprenticeships-further-education/students-aged-16-to-18-achieving-3-a-grades-or-better-at-a-level/latest/ https://www.ethnicity-facts-figures.service.gov.uk/education.... In the U.S., research shows the SAT is highly predictive of college performance: https://freddiedeboer.substack.com/p/education-week-educational-assessments https://freddiedeboer.substack.com/p/education-week-educatio... (summarizing research).
  - cauch 1y ago
    It is strange to pretend that there is no cultural bias and then given an example that is usually explained because Asians seem to culturally value more education than white British. How will you explain that Asians outperform white British otherwise, knowing that the idea that Asians and white British are genetically different enough to explain this has been scientifically debunked, or that adopted Asians don't show the same pattern as not adopted Asians? (and, yes, of course SAT is highly predictive of college performance, isn't that the point: people who get better training get better college performance while not being "smarter", just "better trained")
    - rayiner 1y ago
      I’m talking about the supposed cultural bias of the test itself, not cultural differences among test takers. A culturally biased test is one that requires familiarity with a particular culture, generally that of the people who wrote the test. If Asians do better on a test developed by British people, that suggests that the test itself is not culturally biased.
      - cauch 1y ago
        Your argument would have been more convincing if the Asians were getting the same as British people. But as soon as the Asians do better, it means that the whole comparison is meaningless. It means that Asians and British scores differently. Maybe Asians normally should score 14 and British score 10, but they score 12 because the test is culturally biased. (sure, we expect that a small difference in "normal situation" should be relatively small, but the samples are defined as biased, so you cannot really rely on them unless making unproven assumptions)
        xyzzyz 1y ago
        There is no reason to expect that the test results would be the same across all demographic groups, and in fact, everything we know about psychometry (i.e. the science of mental testing) suggests that we should expect exactly opposite. See e.g. "Intelligence: Knowns and unknowns", which described the consensus position of the American Psychological Association as of 1995: > The cause of [test achievement] differential is not known; it is apparently not due to any simple form of bias in the content or administration of the tests themselves. https://www.mun.ca/biology/scarr/APA%201985%20Intelligence%20-%20Knowns%20and%20Unknowns.pdf https://www.mun.ca/biology/scarr/APA%201985%20Intelligence%2...
        cauch 1y ago
        Not sure what is your point, the "test achievement" mentioned in the document refers to totally different "test" that the ones we were talking about. Also, on just pure logic, I don't think the document shows what you think it shows. The document you provide (which is 30 years old, so with just this one, we should not assume it reflects today's consensus) explains that the difference is not understood, and that there is no _obvious_ answer, neither from biology, from group culture or from bias in the tests. In other words: the difference is due to something _not obvious_, for example (but not limited to, of course, it's just an example), _not obvious_ form of bias.
paganel 1y ago
You could fairly say that China’s pre-Opium Wars obsession with testing and meritocracy based on said testing is what brought them into all that mess, I’m pretty sure that the Portuguese that had gotten all the way from their small country all the way to Southern China using some stingy boats were not clerks nor great (potential) test-takers, and yet it was those Portuguese seafarers that were to change the fate of most of Asia forever, not the test-taking Chinese.
- nobodyandproud 1y ago
  Testing and meritocracy of what, though? Don’t forget that China chose Confucianism to put a halt to the perpetual, European style wars. Stagnation was by design, and caught up with them after 2000 years.
growse 1y ago
I got six A's at A level, over 20 years ago. Am i objectively smarter than every single other peer who only got 4 As? (I, for one, am confident I know the answer to this question).
- rayiner 1y ago
  Close. If you take a group of 50 people like you, who got six A’s, and a group of 50 people who only got 4 A’s, then the former group will be smarter.
  - growse 1y ago
    So people who've been taught more things are "smarter" than people who've not been taught as much? All babies are stupid, I therefore assume? What about the people who never get the chance to do any A levels? Are they all less smart than those who do?
- notahacker 1y ago
  I'm confident you're a better judge of the worth of A-levels than the people who've never even taken them furiously insisting they're objective indicators of merit, and not high school syllabus-recollection/essay-writing tests which are easily taught to, actively fudged by some schools and greatly variable in actual difficulty from one subject and exam board to another. Still, your grades (and mine) pale in comparison to all these youngsters with an opportunity to get A* grades...
ceejayoz 1y ago
We’ve done all sorts of dumb things for thousands of years. It’s one metric of many. We know that paying for a tutor can change test scores. We know that a shitty home life can, too. They’re just harder to measure.
- rayiner 1y ago
  If you’re suggesting that having “a shitty home life” can make people perform badly on tests, but not perform badly in real world tasks, we don’t “know” that. It’s something people want to be true, but there’s not much evidence for it. Meanwhile, there’s reams of evidence that standardized tests scores are highly predictive of performance.
  - ceejayoz 1y ago
    Most people don’t go to Oxford just to go right back home to said shitty home life.
    - rayiner 1y ago
      Okay, but where is the evidence that those people perform better in real world tasks than their test scores would indicate once they are no longer in that shitty home life?

But we know that it’s true. That’s why we have been using objective metrics like test scores for millennia, across societies are different as China, India, and Britain.

Latty 1y ago

Your best argument is "we've done it for a long time, so it can't be wrong"? Quite the contrary: there is a long history of "objective" tests being shown to be deeply flawed and biased towards certain factors (often cultural and class based), we explicitly know it isn't the case that test scores are purely about some innate intelligence characteristic: there is a reason the rich spend a lot of money to raise their children's scores. My secondary school claimed to have the best results for Business Studies A-levels in the country. They achieved this by taking the pre-released case study, writing every possible question they could think of about the study, writing model answers, and telling the students to memorise them. The idea that these scores represent some innate intelligence of the student is obviously nonsense if you interact with the system at all.

rayiner 1y ago
The notion that the British A levels have “cultural bias” is absurd, given that Asians outperform white British: https://www.ethnicity-facts-figures.service.gov.uk/education-skills-and-training/a-levels-apprenticeships-further-education/students-aged-16-to-18-achieving-3-a-grades-or-better-at-a-level/latest/ https://www.ethnicity-facts-figures.service.gov.uk/education.... In the U.S., research shows the SAT is highly predictive of college performance: https://freddiedeboer.substack.com/p/education-week-educational-assessments https://freddiedeboer.substack.com/p/education-week-educatio... (summarizing research).
- cauch 1y ago
  It is strange to pretend that there is no cultural bias and then given an example that is usually explained because Asians seem to culturally value more education than white British. How will you explain that Asians outperform white British otherwise, knowing that the idea that Asians and white British are genetically different enough to explain this has been scientifically debunked, or that adopted Asians don't show the same pattern as not adopted Asians? (and, yes, of course SAT is highly predictive of college performance, isn't that the point: people who get better training get better college performance while not being "smarter", just "better trained")
  - rayiner 1y ago
    I’m talking about the supposed cultural bias of the test itself, not cultural differences among test takers. A culturally biased test is one that requires familiarity with a particular culture, generally that of the people who wrote the test. If Asians do better on a test developed by British people, that suggests that the test itself is not culturally biased.
    - cauch 1y ago
      Your argument would have been more convincing if the Asians were getting the same as British people. But as soon as the Asians do better, it means that the whole comparison is meaningless. It means that Asians and British scores differently. Maybe Asians normally should score 14 and British score 10, but they score 12 because the test is culturally biased. (sure, we expect that a small difference in "normal situation" should be relatively small, but the samples are defined as biased, so you cannot really rely on them unless making unproven assumptions)
      - xyzzyz 1y ago
        There is no reason to expect that the test results would be the same across all demographic groups, and in fact, everything we know about psychometry (i.e. the science of mental testing) suggests that we should expect exactly opposite. See e.g. "Intelligence: Knowns and unknowns", which described the consensus position of the American Psychological Association as of 1995: > The cause of [test achievement] differential is not known; it is apparently not due to any simple form of bias in the content or administration of the tests themselves. https://www.mun.ca/biology/scarr/APA%201985%20Intelligence%20-%20Knowns%20and%20Unknowns.pdf https://www.mun.ca/biology/scarr/APA%201985%20Intelligence%2...
        cauch 1y ago
        Not sure what is your point, the "test achievement" mentioned in the document refers to totally different "test" that the ones we were talking about. Also, on just pure logic, I don't think the document shows what you think it shows. The document you provide (which is 30 years old, so with just this one, we should not assume it reflects today's consensus) explains that the difference is not understood, and that there is no _obvious_ answer, neither from biology, from group culture or from bias in the tests. In other words: the difference is due to something _not obvious_, for example (but not limited to, of course, it's just an example), _not obvious_ form of bias.

paganel 1y ago

You could fairly say that China’s pre-Opium Wars obsession with testing and meritocracy based on said testing is what brought them into all that mess, I’m pretty sure that the Portuguese that had gotten all the way from their small country all the way to Southern China using some stingy boats were not clerks nor great (potential) test-takers, and yet it was those Portuguese seafarers that were to change the fate of most of Asia forever, not the test-taking Chinese.

nobodyandproud 1y ago
Testing and meritocracy of what, though? Don’t forget that China chose Confucianism to put a halt to the perpetual, European style wars. Stagnation was by design, and caught up with them after 2000 years.

growse 1y ago

I got six A's at A level, over 20 years ago. Am i objectively smarter than every single other peer who only got 4 As? (I, for one, am confident I know the answer to this question).

rayiner 1y ago
Close. If you take a group of 50 people like you, who got six A’s, and a group of 50 people who only got 4 A’s, then the former group will be smarter.
- growse 1y ago
  So people who've been taught more things are "smarter" than people who've not been taught as much? All babies are stupid, I therefore assume? What about the people who never get the chance to do any A levels? Are they all less smart than those who do?
notahacker 1y ago
I'm confident you're a better judge of the worth of A-levels than the people who've never even taken them furiously insisting they're objective indicators of merit, and not high school syllabus-recollection/essay-writing tests which are easily taught to, actively fudged by some schools and greatly variable in actual difficulty from one subject and exam board to another. Still, your grades (and mine) pale in comparison to all these youngsters with an opportunity to get A* grades...

ceejayoz 1y ago

We’ve done all sorts of dumb things for thousands of years. It’s one metric of many. We know that paying for a tutor can change test scores. We know that a shitty home life can, too. They’re just harder to measure.

rayiner 1y ago
If you’re suggesting that having “a shitty home life” can make people perform badly on tests, but not perform badly in real world tasks, we don’t “know” that. It’s something people want to be true, but there’s not much evidence for it. Meanwhile, there’s reams of evidence that standardized tests scores are highly predictive of performance.
- ceejayoz 1y ago
  Most people don’t go to Oxford just to go right back home to said shitty home life.
  - rayiner 1y ago
    Okay, but where is the evidence that those people perform better in real world tasks than their test scores would indicate once they are no longer in that shitty home life?