4 ms·

SAT prep courses advertise that they can increase scores by 100 points but generally they're lying in order to sell more test courses. But the real number is c

SAT prep courses advertise that they can increase scores by 100 points but generally they're lying in order to sell more test courses. But the real number is closer to "11 to 15 points on the math section and 6 to 9 points on the verbal."[1] [1]https://www.researchgate.net/publication/228337033_Using_Linear_Regression_and_Propensity_Score_Matching_to_Estimate_the_Effect_of_Coaching_on_the_SAT https://www.researchgate.net/publication/228337033_Using_Lin...

thaumasiotes 7y ago
Note also that the measurement error of a section score is 30 points, which dwarfs the effect of prep.
asdfman123 7y ago
As a former SAT tutor (and a guy who gamed the hell out of the SAT) I have trouble believing that. There are only a few classes of questions they ask on the test, and if you master them the whole test is a walk in the park. Maybe the problem is most students just don't give a damn about the classes their parents make them take? Can someone explain the findings in this paper?
- graeme 7y ago
  My impression as a former SAT instructor is a lot of students don’t spend that much time prepping. And if they have no good reading/math background it can take some effort to master things. That said, typically saw increases well above that in my classes. Even with zero instruction I’d expect a self studying student to improve more than that from familiarization. It did say upper class people improve more.
- closeparen 7y ago
  The SAT is designed to be an intelligence test, not a course final, and it's possible that the large team of education and psychometrics people who've been working on it for decades know what they're doing. Armchair speculation: there are probably some really bright kids out there with poor test-taking skills, and prep likely has a dramatic effect on scores for them, but most students are not that. Did you actually sit an SAT before you started gaming? What was your improvement?
  - asdfman123 7y ago
    > it's possible that the large team of education and psychometrics people who've been working on it for decades know what they're doing Don't assume that society is optimizing for what's socially optimal. Back when it was on a 1600 point scale, I got a 1300 on the first practice test for my SAT course. When I took the test for real, I got a 1570.
    - closeparen 7y ago
      I wouldn't assume it, but it's one of the plausible explanations for the test appearing to meet its design goals.
  - bermanoid 7y ago
    Not GP, but also a test prep teacher in a previous life. I always scored well on the SAT and test well in general, so I can't speak to personal improvement there, but a few points: First, the test prep companies heavily encourage (and may even require?) students to take the test multiple times. That change alone will boost most people's top score by a solid margin, because it's a noisy test. The first response they always gave to people complaining about lack of improvement was "take the test again, and if you still haven't improved, take the course again for free, and then take the test again". Second, psychometric expertise is great, but the goal of the SAT is not to be impossible to train for except as a secondary thing. It's really hard to do, especially when you are such a high value optimization target and have to build a test that doesn't rely on much specific knowledge and can be quickly scored. A lot of what the SAT courses do is just teach students to make slightly more accurate guesses on multiple-choice questions where someone unfamiliar with test strategy would leave things blank. That alone tends to boost scores, and some of the other strategies are fairly clever to help avoid common mistakes. Last, though I didn't have room to improve on the SAT, I also taught GRE classes and can speak to my improvement there. The math section is trivial to get an 800 on (it's easier than the SAT math, or at least it was when I taught 15+ years ago), but the verbal section is quite tough if you haven't studied, and in a lot of ways is a glorified vocabulary test, and the reading comprehension sections can be pretty tough as well. Before I trained to teach the courses, my verbal score was a 490 (on the real test, not practice), so I was considering not even trying to teach (there's some threshold you have to hit, maybe 700 at the time?), but the trainer encouraged all of us to try anyways because he said the content made such a difference. After just two weekends of intensive teacher training, I tested again and ended up with a 740. After teaching the course around a dozen times, I'm pretty confident I could have hit 800 without any difficulty, you basically just have to get used to the types of questions that they ask and get in the headspace of the question authors. Just one data point, but I definitely believe that this stuff is effective.
- SamReidHughes 7y ago
  Students naturally do better without test prep, too. They get older, their brains grow more, and they’re more intelligent. Much of the score increase people see is attributable to that. And most kids bright enough to exercise good test taking skills will have them without special training, I guess? 11 years of school will do that.
SkyBelow 7y ago
>Students were able to indicate if they had prepared through the use of school courses, commercial courses, tutoring, or a variety of preparatory materials. In what follows, a coached student is defined as one that reported participating in a commercial preparatory course. Is there any indication that they actually broke down any further, or is this just limited to the impact of average coaching and thus not a good representation of those who receive the top tier coaching?
knzhou 7y ago
Not only that, but these prep courses tend to assign practice SATs that are harder in the beginning and easier at the end, to give the impression of rapid progress.
skat20phys 7y ago
Those estimates are generally in line with what other independent investigators find, maybe a bit smaller. The elephant in the room in these kinds of discussions is the validity of the tests to begin with. MIT makes a statement that the general tests are predictive in combination with other criteria, but how predictive? Generally studies of these sorts of things find that they're moderately predictive of first-year GPA (like .4 correlation), and then trail off to zero as the interval between testing and outcomes increases. The effects are even smaller in studies in relatively unselected samples (due to court orders or legal decisions, for example). So one argument always goes that you should use whatever empirically-supported stuff you can to make fair decisions, but we treat them as if they're more than they are. Sure, you could have a lot of things that are significantly predictive but with small effect size, or where there's lots of noise, but why as a society to we pretend these are huge effects? The other thing about that paper is the hint that the practice effects are larger in higher-performing examinees, which also makes sense and is consistent with other studies. Why is this a problem? Because those are exactly the types of students for whom these issues are more applicable. A 12 point average gain with practice doesn't matter if you're including people who never had a chance at MIT anyway, but a 20 point gain might in a highly competitive group where small differences are being magnified tenfold. The issue in all of this isn't the people in the 99th percentile versus the 50th percentile, which is the bulk of what's going into these predictive models and effect sizes, it's the 99th versus the 95th. There's a ton of real-world noise, but society acts as if the noise is nonexistent. It's like we're idolizing the outcome of some kind of survivorship bias process.

thaumasiotes 7y ago

Note also that the measurement error of a section score is 30 points, which dwarfs the effect of prep.

asdfman123 7y ago

As a former SAT tutor (and a guy who gamed the hell out of the SAT) I have trouble believing that. There are only a few classes of questions they ask on the test, and if you master them the whole test is a walk in the park. Maybe the problem is most students just don't give a damn about the classes their parents make them take? Can someone explain the findings in this paper?

graeme 7y ago
My impression as a former SAT instructor is a lot of students don’t spend that much time prepping. And if they have no good reading/math background it can take some effort to master things. That said, typically saw increases well above that in my classes. Even with zero instruction I’d expect a self studying student to improve more than that from familiarization. It did say upper class people improve more.
closeparen 7y ago
The SAT is designed to be an intelligence test, not a course final, and it's possible that the large team of education and psychometrics people who've been working on it for decades know what they're doing. Armchair speculation: there are probably some really bright kids out there with poor test-taking skills, and prep likely has a dramatic effect on scores for them, but most students are not that. Did you actually sit an SAT before you started gaming? What was your improvement?
- asdfman123 7y ago
  > it's possible that the large team of education and psychometrics people who've been working on it for decades know what they're doing Don't assume that society is optimizing for what's socially optimal. Back when it was on a 1600 point scale, I got a 1300 on the first practice test for my SAT course. When I took the test for real, I got a 1570.
  - closeparen 7y ago
    I wouldn't assume it, but it's one of the plausible explanations for the test appearing to meet its design goals.
- bermanoid 7y ago
  Not GP, but also a test prep teacher in a previous life. I always scored well on the SAT and test well in general, so I can't speak to personal improvement there, but a few points: First, the test prep companies heavily encourage (and may even require?) students to take the test multiple times. That change alone will boost most people's top score by a solid margin, because it's a noisy test. The first response they always gave to people complaining about lack of improvement was "take the test again, and if you still haven't improved, take the course again for free, and then take the test again". Second, psychometric expertise is great, but the goal of the SAT is not to be impossible to train for except as a secondary thing. It's really hard to do, especially when you are such a high value optimization target and have to build a test that doesn't rely on much specific knowledge and can be quickly scored. A lot of what the SAT courses do is just teach students to make slightly more accurate guesses on multiple-choice questions where someone unfamiliar with test strategy would leave things blank. That alone tends to boost scores, and some of the other strategies are fairly clever to help avoid common mistakes. Last, though I didn't have room to improve on the SAT, I also taught GRE classes and can speak to my improvement there. The math section is trivial to get an 800 on (it's easier than the SAT math, or at least it was when I taught 15+ years ago), but the verbal section is quite tough if you haven't studied, and in a lot of ways is a glorified vocabulary test, and the reading comprehension sections can be pretty tough as well. Before I trained to teach the courses, my verbal score was a 490 (on the real test, not practice), so I was considering not even trying to teach (there's some threshold you have to hit, maybe 700 at the time?), but the trainer encouraged all of us to try anyways because he said the content made such a difference. After just two weekends of intensive teacher training, I tested again and ended up with a 740. After teaching the course around a dozen times, I'm pretty confident I could have hit 800 without any difficulty, you basically just have to get used to the types of questions that they ask and get in the headspace of the question authors. Just one data point, but I definitely believe that this stuff is effective.
SamReidHughes 7y ago
Students naturally do better without test prep, too. They get older, their brains grow more, and they’re more intelligent. Much of the score increase people see is attributable to that. And most kids bright enough to exercise good test taking skills will have them without special training, I guess? 11 years of school will do that.

SkyBelow 7y ago

>Students were able to indicate if they had prepared through the use of school courses, commercial courses, tutoring, or a variety of preparatory materials. In what follows, a coached student is defined as one that reported participating in a commercial preparatory course. Is there any indication that they actually broke down any further, or is this just limited to the impact of average coaching and thus not a good representation of those who receive the top tier coaching?

knzhou 7y ago

Not only that, but these prep courses tend to assign practice SATs that are harder in the beginning and easier at the end, to give the impression of rapid progress.

skat20phys 7y ago

Those estimates are generally in line with what other independent investigators find, maybe a bit smaller. The elephant in the room in these kinds of discussions is the validity of the tests to begin with. MIT makes a statement that the general tests are predictive in combination with other criteria, but how predictive? Generally studies of these sorts of things find that they're moderately predictive of first-year GPA (like .4 correlation), and then trail off to zero as the interval between testing and outcomes increases. The effects are even smaller in studies in relatively unselected samples (due to court orders or legal decisions, for example). So one argument always goes that you should use whatever empirically-supported stuff you can to make fair decisions, but we treat them as if they're more than they are. Sure, you could have a lot of things that are significantly predictive but with small effect size, or where there's lots of noise, but why as a society to we pretend these are huge effects? The other thing about that paper is the hint that the practice effects are larger in higher-performing examinees, which also makes sense and is consistent with other studies. Why is this a problem? Because those are exactly the types of students for whom these issues are more applicable. A 12 point average gain with practice doesn't matter if you're including people who never had a chance at MIT anyway, but a 20 point gain might in a highly competitive group where small differences are being magnified tenfold. The issue in all of this isn't the people in the 99th percentile versus the 50th percentile, which is the bulk of what's going into these predictive models and effect sizes, it's the 99th versus the 95th. There's a ton of real-world noise, but society acts as if the noise is nonexistent. It's like we're idolizing the outcome of some kind of survivorship bias process.