3 ms·

As a former SAT tutor (and a guy who gamed the hell out of the SAT) I have trouble believing that. There are only a few classes of questions they ask on the tes

As a former SAT tutor (and a guy who gamed the hell out of the SAT) I have trouble believing that. There are only a few classes of questions they ask on the test, and if you master them the whole test is a walk in the park. Maybe the problem is most students just don't give a damn about the classes their parents make them take? Can someone explain the findings in this paper?

graeme 7y ago
My impression as a former SAT instructor is a lot of students don’t spend that much time prepping. And if they have no good reading/math background it can take some effort to master things. That said, typically saw increases well above that in my classes. Even with zero instruction I’d expect a self studying student to improve more than that from familiarization. It did say upper class people improve more.
closeparen 7y ago
The SAT is designed to be an intelligence test, not a course final, and it's possible that the large team of education and psychometrics people who've been working on it for decades know what they're doing. Armchair speculation: there are probably some really bright kids out there with poor test-taking skills, and prep likely has a dramatic effect on scores for them, but most students are not that. Did you actually sit an SAT before you started gaming? What was your improvement?
- asdfman123 7y ago
  > it's possible that the large team of education and psychometrics people who've been working on it for decades know what they're doing Don't assume that society is optimizing for what's socially optimal. Back when it was on a 1600 point scale, I got a 1300 on the first practice test for my SAT course. When I took the test for real, I got a 1570.
  - closeparen 7y ago
    I wouldn't assume it, but it's one of the plausible explanations for the test appearing to meet its design goals.
- bermanoid 7y ago
  Not GP, but also a test prep teacher in a previous life. I always scored well on the SAT and test well in general, so I can't speak to personal improvement there, but a few points: First, the test prep companies heavily encourage (and may even require?) students to take the test multiple times. That change alone will boost most people's top score by a solid margin, because it's a noisy test. The first response they always gave to people complaining about lack of improvement was "take the test again, and if you still haven't improved, take the course again for free, and then take the test again". Second, psychometric expertise is great, but the goal of the SAT is not to be impossible to train for except as a secondary thing. It's really hard to do, especially when you are such a high value optimization target and have to build a test that doesn't rely on much specific knowledge and can be quickly scored. A lot of what the SAT courses do is just teach students to make slightly more accurate guesses on multiple-choice questions where someone unfamiliar with test strategy would leave things blank. That alone tends to boost scores, and some of the other strategies are fairly clever to help avoid common mistakes. Last, though I didn't have room to improve on the SAT, I also taught GRE classes and can speak to my improvement there. The math section is trivial to get an 800 on (it's easier than the SAT math, or at least it was when I taught 15+ years ago), but the verbal section is quite tough if you haven't studied, and in a lot of ways is a glorified vocabulary test, and the reading comprehension sections can be pretty tough as well. Before I trained to teach the courses, my verbal score was a 490 (on the real test, not practice), so I was considering not even trying to teach (there's some threshold you have to hit, maybe 700 at the time?), but the trainer encouraged all of us to try anyways because he said the content made such a difference. After just two weekends of intensive teacher training, I tested again and ended up with a 740. After teaching the course around a dozen times, I'm pretty confident I could have hit 800 without any difficulty, you basically just have to get used to the types of questions that they ask and get in the headspace of the question authors. Just one data point, but I definitely believe that this stuff is effective.
  - hackinthebochs 7y ago
    But for someone who did well on the SAT, you would be expected to do well on the GRE too. That you were able to prep your way into a good score doesn't indicate the GRE is measuring the wrong thing. You already previously showed a strong aptitude for g-loaded tests. What you needed was familiarity with the content and the kinds of questions asked. But that doesn't invalidate the test.
SamReidHughes 7y ago
Students naturally do better without test prep, too. They get older, their brains grow more, and they’re more intelligent. Much of the score increase people see is attributable to that. And most kids bright enough to exercise good test taking skills will have them without special training, I guess? 11 years of school will do that.

graeme 7y ago

My impression as a former SAT instructor is a lot of students don’t spend that much time prepping. And if they have no good reading/math background it can take some effort to master things. That said, typically saw increases well above that in my classes. Even with zero instruction I’d expect a self studying student to improve more than that from familiarization. It did say upper class people improve more.

closeparen 7y ago

The SAT is designed to be an intelligence test, not a course final, and it's possible that the large team of education and psychometrics people who've been working on it for decades know what they're doing. Armchair speculation: there are probably some really bright kids out there with poor test-taking skills, and prep likely has a dramatic effect on scores for them, but most students are not that. Did you actually sit an SAT before you started gaming? What was your improvement?

asdfman123 7y ago
> it's possible that the large team of education and psychometrics people who've been working on it for decades know what they're doing Don't assume that society is optimizing for what's socially optimal. Back when it was on a 1600 point scale, I got a 1300 on the first practice test for my SAT course. When I took the test for real, I got a 1570.
- closeparen 7y ago
  I wouldn't assume it, but it's one of the plausible explanations for the test appearing to meet its design goals.
bermanoid 7y ago
Not GP, but also a test prep teacher in a previous life. I always scored well on the SAT and test well in general, so I can't speak to personal improvement there, but a few points: First, the test prep companies heavily encourage (and may even require?) students to take the test multiple times. That change alone will boost most people's top score by a solid margin, because it's a noisy test. The first response they always gave to people complaining about lack of improvement was "take the test again, and if you still haven't improved, take the course again for free, and then take the test again". Second, psychometric expertise is great, but the goal of the SAT is not to be impossible to train for except as a secondary thing. It's really hard to do, especially when you are such a high value optimization target and have to build a test that doesn't rely on much specific knowledge and can be quickly scored. A lot of what the SAT courses do is just teach students to make slightly more accurate guesses on multiple-choice questions where someone unfamiliar with test strategy would leave things blank. That alone tends to boost scores, and some of the other strategies are fairly clever to help avoid common mistakes. Last, though I didn't have room to improve on the SAT, I also taught GRE classes and can speak to my improvement there. The math section is trivial to get an 800 on (it's easier than the SAT math, or at least it was when I taught 15+ years ago), but the verbal section is quite tough if you haven't studied, and in a lot of ways is a glorified vocabulary test, and the reading comprehension sections can be pretty tough as well. Before I trained to teach the courses, my verbal score was a 490 (on the real test, not practice), so I was considering not even trying to teach (there's some threshold you have to hit, maybe 700 at the time?), but the trainer encouraged all of us to try anyways because he said the content made such a difference. After just two weekends of intensive teacher training, I tested again and ended up with a 740. After teaching the course around a dozen times, I'm pretty confident I could have hit 800 without any difficulty, you basically just have to get used to the types of questions that they ask and get in the headspace of the question authors. Just one data point, but I definitely believe that this stuff is effective.
- hackinthebochs 7y ago
  But for someone who did well on the SAT, you would be expected to do well on the GRE too. That you were able to prep your way into a good score doesn't indicate the GRE is measuring the wrong thing. You already previously showed a strong aptitude for g-loaded tests. What you needed was familiarity with the content and the kinds of questions asked. But that doesn't invalidate the test.

SamReidHughes 7y ago

Students naturally do better without test prep, too. They get older, their brains grow more, and they’re more intelligent. Much of the score increase people see is attributable to that. And most kids bright enough to exercise good test taking skills will have them without special training, I guess? 11 years of school will do that.