5 ms·

The diversity thing is mostly a red-herring. Admissions offers are abandoning standardized tests because of Goodhart's Law. > for students who have the time an

The diversity thing is mostly a red-herring. Admissions offers are abandoning standardized tests because of Goodhart's Law. > for students who have the time and encouragement to study for them It's not just time and encouragement. SAT/ACT prep is serious industry. $100+/hr tutoring sessions. Owners of good SAT prep businesses can make six or seven figures in a few months. And good prep courses really do work. Tests, like coding interviews, are a game you can hack with enough time and effort. The signal is totally washed out; with enough money, you can learn how to take the test well enough.

SatvikBeri 8y ago
The research suggests SAT test prep has an effect of about one seventh of a standard deviation – about 50 points out of 2400[1]. That's significant but not huge. Most of the perception of large gains comes from the fact that people do better as they get older with or without coaching. (None of the research is great, but better than nothing.) [1]: https://www.jefftk.com/p/sat-coaching-what-effect-size https://www.jefftk.com/p/sat-coaching-what-effect-size
- learc83 8y ago
  The College Board released new results in 2017 showing a more than 50 point increase out of 1600 versus students who didn't practice. This was with just 20 hours of practice. Many high achieving students practice for 5+ hours per week. South Korean cram school style 2-3 hours per day is in a completely different league than a standard SAT prep course, but much of the research that shows no result counts them the same.
  - barry-cotter 8y ago
    This phenomenon is known as the practice effect. It has a ceiling in terms of IQ tests, just like it does in terms of physical tests. IQ tests are designed to be as close to ungameable as possible. This is like pointing out that I can make large gains in my 100m sprint time by practicing for half an hour a day for a month and predicting that I can make the Olympic team if I put in a thousand hours practice. Novice gains are real and significant but they peter out pretty quickly.
    - learc83 8y ago
      IQ tests are gameable to a degree. They are accurate across large populations largely because most people don't practice, not because they are actually ungameable. SATs are different in 2 respects. 1. There is a large incentive to practice. 2. It isn't an IQ test, it only correlates fairly strongly with scores on IQ tests. SATs are highly knowledge based--most of the math questions can be answered by memorizing algorithms, and the verbal section depends heavily on vocabulary. >This is like pointing out that I can make large gains in my 100m sprint time by practicing for half an hour a day for a month and predicting that I can make the Olympic team if I put in a thousand hours practice. Novice gains are real and significant but they peter out pretty quickly. It's unlikely you could become an Olympic sprinter, but with enough practice you could probably move into the top 1% of everyone in your age group because the vast majority aren't going to spend that much time practicing. The SAT equivalent (for the 2018 test) would be somewhere in the 1500-1550 range.
      - barry-cotter 8y ago
        The SAT is highly g loaded and it correlates with tests that are designed and marketed as IQ tests at levels sufficient that saying it isn’t one is quibbling. The lowest estimate I found for IQ SAT correlation was an r^2 of 0.76. SATs are more knowledge based than a test designed to be culture fair, like Raven’s Progressive Matrices but they’re not meant to be culture fair either, they’re for testing educated English speaking Americans. The test population are in high school but there’s nothing on the test that you wouldn’t have known by the end of middle school. As far as algorithms and vocabulary go, most people have limited ability to follow complex algorithms because that requires a lot of working memory, which correlates with... Intelligence!
        learc83 8y ago
        Vocabulary correlates with IQ about as strongly as SAT score, yet vocabulary can clearly be hugley improved with practice. Even if correlation with IQ somehow proved score durability, a correlation of 0.76 or even .8 is low enough to allow for huge swings away from IQ in a very large percentage of the population. A big chunk of students practicing and increasing their scores 200-300 points could easily be hiding in that 0.76 correlation. >As far as algorithms and vocabulary go, most people have limited ability to follow complex algorithms because that requires a lot of working memory, which correlates with... Intelligence! Most algorithms at that level are designed to be written out step by step explicitly to reduce working memory requirements. Part of SAT prep is learning algorithms for solving word problems that reduce working memory load by memorizing patterns in the questions and writing down the components in a standardized form. You said >As far as algorithms and vocabulary go And then never mentioned vocabulary. It has nothing to do with working memory and most definitely can be improved dramatically with practice. One more thing. The SAT was rewritten in 2016 to be far more content based, so the correlation with IQ will likely fall even further.
skookumchuck 8y ago
There are SAT prep books in the thrift store for a dollar. Nobody is shut out because of poverty.
gaius 8y ago
And good prep courses really do work. Tests, like coding interviews, are a game you can hack with enough time and effort. Years ago I took the GMAT and scored very highly, 98th percentile, and GMAT-taking is definitely a learnable skill in and of itself. Then again, maybe the test is really a proxy for “willing to do prep work upfront”. Having said that obviously it filters for those who have the opportunity to do so, but I am willing to give the benefit of the doubt and say the latter is not it’s deliberate intent.

SatvikBeri 8y ago

The research suggests SAT test prep has an effect of about one seventh of a standard deviation – about 50 points out of 2400[1]. That's significant but not huge. Most of the perception of large gains comes from the fact that people do better as they get older with or without coaching. (None of the research is great, but better than nothing.) [1]: https://www.jefftk.com/p/sat-coaching-what-effect-size https://www.jefftk.com/p/sat-coaching-what-effect-size

learc83 8y ago
The College Board released new results in 2017 showing a more than 50 point increase out of 1600 versus students who didn't practice. This was with just 20 hours of practice. Many high achieving students practice for 5+ hours per week. South Korean cram school style 2-3 hours per day is in a completely different league than a standard SAT prep course, but much of the research that shows no result counts them the same.
- barry-cotter 8y ago
  This phenomenon is known as the practice effect. It has a ceiling in terms of IQ tests, just like it does in terms of physical tests. IQ tests are designed to be as close to ungameable as possible. This is like pointing out that I can make large gains in my 100m sprint time by practicing for half an hour a day for a month and predicting that I can make the Olympic team if I put in a thousand hours practice. Novice gains are real and significant but they peter out pretty quickly.
  - learc83 8y ago
    IQ tests are gameable to a degree. They are accurate across large populations largely because most people don't practice, not because they are actually ungameable. SATs are different in 2 respects. 1. There is a large incentive to practice. 2. It isn't an IQ test, it only correlates fairly strongly with scores on IQ tests. SATs are highly knowledge based--most of the math questions can be answered by memorizing algorithms, and the verbal section depends heavily on vocabulary. >This is like pointing out that I can make large gains in my 100m sprint time by practicing for half an hour a day for a month and predicting that I can make the Olympic team if I put in a thousand hours practice. Novice gains are real and significant but they peter out pretty quickly. It's unlikely you could become an Olympic sprinter, but with enough practice you could probably move into the top 1% of everyone in your age group because the vast majority aren't going to spend that much time practicing. The SAT equivalent (for the 2018 test) would be somewhere in the 1500-1550 range.
    - barry-cotter 8y ago
      The SAT is highly g loaded and it correlates with tests that are designed and marketed as IQ tests at levels sufficient that saying it isn’t one is quibbling. The lowest estimate I found for IQ SAT correlation was an r^2 of 0.76. SATs are more knowledge based than a test designed to be culture fair, like Raven’s Progressive Matrices but they’re not meant to be culture fair either, they’re for testing educated English speaking Americans. The test population are in high school but there’s nothing on the test that you wouldn’t have known by the end of middle school. As far as algorithms and vocabulary go, most people have limited ability to follow complex algorithms because that requires a lot of working memory, which correlates with... Intelligence!
      - learc83 8y ago
        Vocabulary correlates with IQ about as strongly as SAT score, yet vocabulary can clearly be hugley improved with practice. Even if correlation with IQ somehow proved score durability, a correlation of 0.76 or even .8 is low enough to allow for huge swings away from IQ in a very large percentage of the population. A big chunk of students practicing and increasing their scores 200-300 points could easily be hiding in that 0.76 correlation. >As far as algorithms and vocabulary go, most people have limited ability to follow complex algorithms because that requires a lot of working memory, which correlates with... Intelligence! Most algorithms at that level are designed to be written out step by step explicitly to reduce working memory requirements. Part of SAT prep is learning algorithms for solving word problems that reduce working memory load by memorizing patterns in the questions and writing down the components in a standardized form. You said >As far as algorithms and vocabulary go And then never mentioned vocabulary. It has nothing to do with working memory and most definitely can be improved dramatically with practice. One more thing. The SAT was rewritten in 2016 to be far more content based, so the correlation with IQ will likely fall even further.

skookumchuck 8y ago

There are SAT prep books in the thrift store for a dollar. Nobody is shut out because of poverty.

gaius 8y ago

And good prep courses really do work. Tests, like coding interviews, are a game you can hack with enough time and effort. Years ago I took the GMAT and scored very highly, 98th percentile, and GMAT-taking is definitely a learnable skill in and of itself. Then again, maybe the test is really a proxy for “willing to do prep work upfront”. Having said that obviously it filters for those who have the opportunity to do so, but I am willing to give the benefit of the doubt and say the latter is not it’s deliberate intent.