3 ms·

The research suggests SAT test prep has an effect of about one seventh of a standard deviation – about 50 points out of 2400[1]. That's significant but not huge

The research suggests SAT test prep has an effect of about one seventh of a standard deviation – about 50 points out of 2400[1]. That's significant but not huge. Most of the perception of large gains comes from the fact that people do better as they get older with or without coaching. (None of the research is great, but better than nothing.) [1]: https://www.jefftk.com/p/sat-coaching-what-effect-size https://www.jefftk.com/p/sat-coaching-what-effect-size

learc83 8y ago
The College Board released new results in 2017 showing a more than 50 point increase out of 1600 versus students who didn't practice. This was with just 20 hours of practice. Many high achieving students practice for 5+ hours per week. South Korean cram school style 2-3 hours per day is in a completely different league than a standard SAT prep course, but much of the research that shows no result counts them the same.
- barry-cotter 8y ago
  This phenomenon is known as the practice effect. It has a ceiling in terms of IQ tests, just like it does in terms of physical tests. IQ tests are designed to be as close to ungameable as possible. This is like pointing out that I can make large gains in my 100m sprint time by practicing for half an hour a day for a month and predicting that I can make the Olympic team if I put in a thousand hours practice. Novice gains are real and significant but they peter out pretty quickly.
  - learc83 8y ago
    IQ tests are gameable to a degree. They are accurate across large populations largely because most people don't practice, not because they are actually ungameable. SATs are different in 2 respects. 1. There is a large incentive to practice. 2. It isn't an IQ test, it only correlates fairly strongly with scores on IQ tests. SATs are highly knowledge based--most of the math questions can be answered by memorizing algorithms, and the verbal section depends heavily on vocabulary. >This is like pointing out that I can make large gains in my 100m sprint time by practicing for half an hour a day for a month and predicting that I can make the Olympic team if I put in a thousand hours practice. Novice gains are real and significant but they peter out pretty quickly. It's unlikely you could become an Olympic sprinter, but with enough practice you could probably move into the top 1% of everyone in your age group because the vast majority aren't going to spend that much time practicing. The SAT equivalent (for the 2018 test) would be somewhere in the 1500-1550 range.
    - barry-cotter 8y ago
      The SAT is highly g loaded and it correlates with tests that are designed and marketed as IQ tests at levels sufficient that saying it isn’t one is quibbling. The lowest estimate I found for IQ SAT correlation was an r^2 of 0.76. SATs are more knowledge based than a test designed to be culture fair, like Raven’s Progressive Matrices but they’re not meant to be culture fair either, they’re for testing educated English speaking Americans. The test population are in high school but there’s nothing on the test that you wouldn’t have known by the end of middle school. As far as algorithms and vocabulary go, most people have limited ability to follow complex algorithms because that requires a lot of working memory, which correlates with... Intelligence!
      - learc83 8y ago
        Vocabulary correlates with IQ about as strongly as SAT score, yet vocabulary can clearly be hugley improved with practice. Even if correlation with IQ somehow proved score durability, a correlation of 0.76 or even .8 is low enough to allow for huge swings away from IQ in a very large percentage of the population. A big chunk of students practicing and increasing their scores 200-300 points could easily be hiding in that 0.76 correlation. >As far as algorithms and vocabulary go, most people have limited ability to follow complex algorithms because that requires a lot of working memory, which correlates with... Intelligence! Most algorithms at that level are designed to be written out step by step explicitly to reduce working memory requirements. Part of SAT prep is learning algorithms for solving word problems that reduce working memory load by memorizing patterns in the questions and writing down the components in a standardized form. You said >As far as algorithms and vocabulary go And then never mentioned vocabulary. It has nothing to do with working memory and most definitely can be improved dramatically with practice. One more thing. The SAT was rewritten in 2016 to be far more content based, so the correlation with IQ will likely fall even further.

learc83 8y ago

The College Board released new results in 2017 showing a more than 50 point increase out of 1600 versus students who didn't practice. This was with just 20 hours of practice. Many high achieving students practice for 5+ hours per week. South Korean cram school style 2-3 hours per day is in a completely different league than a standard SAT prep course, but much of the research that shows no result counts them the same.

barry-cotter 8y ago
This phenomenon is known as the practice effect. It has a ceiling in terms of IQ tests, just like it does in terms of physical tests. IQ tests are designed to be as close to ungameable as possible. This is like pointing out that I can make large gains in my 100m sprint time by practicing for half an hour a day for a month and predicting that I can make the Olympic team if I put in a thousand hours practice. Novice gains are real and significant but they peter out pretty quickly.
- learc83 8y ago
  IQ tests are gameable to a degree. They are accurate across large populations largely because most people don't practice, not because they are actually ungameable. SATs are different in 2 respects. 1. There is a large incentive to practice. 2. It isn't an IQ test, it only correlates fairly strongly with scores on IQ tests. SATs are highly knowledge based--most of the math questions can be answered by memorizing algorithms, and the verbal section depends heavily on vocabulary. >This is like pointing out that I can make large gains in my 100m sprint time by practicing for half an hour a day for a month and predicting that I can make the Olympic team if I put in a thousand hours practice. Novice gains are real and significant but they peter out pretty quickly. It's unlikely you could become an Olympic sprinter, but with enough practice you could probably move into the top 1% of everyone in your age group because the vast majority aren't going to spend that much time practicing. The SAT equivalent (for the 2018 test) would be somewhere in the 1500-1550 range.
  - barry-cotter 8y ago
    The SAT is highly g loaded and it correlates with tests that are designed and marketed as IQ tests at levels sufficient that saying it isn’t one is quibbling. The lowest estimate I found for IQ SAT correlation was an r^2 of 0.76. SATs are more knowledge based than a test designed to be culture fair, like Raven’s Progressive Matrices but they’re not meant to be culture fair either, they’re for testing educated English speaking Americans. The test population are in high school but there’s nothing on the test that you wouldn’t have known by the end of middle school. As far as algorithms and vocabulary go, most people have limited ability to follow complex algorithms because that requires a lot of working memory, which correlates with... Intelligence!
    - learc83 8y ago
      Vocabulary correlates with IQ about as strongly as SAT score, yet vocabulary can clearly be hugley improved with practice. Even if correlation with IQ somehow proved score durability, a correlation of 0.76 or even .8 is low enough to allow for huge swings away from IQ in a very large percentage of the population. A big chunk of students practicing and increasing their scores 200-300 points could easily be hiding in that 0.76 correlation. >As far as algorithms and vocabulary go, most people have limited ability to follow complex algorithms because that requires a lot of working memory, which correlates with... Intelligence! Most algorithms at that level are designed to be written out step by step explicitly to reduce working memory requirements. Part of SAT prep is learning algorithms for solving word problems that reduce working memory load by memorizing patterns in the questions and writing down the components in a standardized form. You said >As far as algorithms and vocabulary go And then never mentioned vocabulary. It has nothing to do with working memory and most definitely can be improved dramatically with practice. One more thing. The SAT was rewritten in 2016 to be far more content based, so the correlation with IQ will likely fall even further.