5 ms·

I don't really have evidence for this, but it's always felt like SAT/ACT coaching doesn't improve scores so much as get rid of some of the "dumb mistakes" that

by paulmd 5y ago

I don't really have evidence for this, but it's always felt like SAT/ACT coaching doesn't improve scores so much as get rid of some of the "dumb mistakes" that cost you score. It's gonna help you when the test writer was playing some gotcha tricks with phrasing or whatever, but if you don't understand the material, even narrowing it down to a 50/50 isn't going to get you a good score, and if you truly don't understand the material you probably won't be able to eliminate half the answers anyway. And they are absolutely aware of the "answer b/c if you don't know" nugget, that's nothing special either. Also "adaptive difficulty" systems where the system throws harder questions at you after successfully answering the easier ones are basically the "elo rating" of academics. Everyone hates elo but... it slots you into a very statistically accurate ranking. If you score highly on Level 600 questions but you are failing on the Level 700 questions, odds are good you are somewhere between 600 and 700. My understanding is that's what SAT/ACT were moving towards about 10 years ago, that it would be computerized and "everyone's test is personalized" by the elo system probing your exact knowledge level from a bank of questions with difficulty scores dynamically based on how "similar ranked" students performed on that question.

sjg007 5y ago
Coaching and studying improves SAT scores. People learn the type of questions they do poorly on and can study to improve. The SAT is a test you can study for.
- likpok 5y ago
  The top link is a study that shows that the improvement is pretty marginal in practice. It does miss something: in specific ethnic enclaves SAT coaching is much more effective, perhaps because of a culture of out-of-school schoolwork and teaching beyond SAT prep. Those enclaves aren't particularly wealthy either (if I recall correctly it was a Korean enclave). Even then we're talking 70 points -- not nothing, but also not a radical transformation.
  - hhs 5y ago
    > Even then we're talking 70 points -- not nothing, but also not a radical transformation. That would depend on the baseline score. For instance, if it was a 1510 baseline and then went up +70, then it would be useful.
    - whimsicalism 5y ago
      I think it's usually not improvement at the highest levels.
  - chernevik 5y ago
    I wouldn't be surprised if those improvements were mostly moving below-average scores toward the average, by giving deprived students basic skills that their "education" didn't.
- boringg 5y ago
  I don't know many tests you can't study for especially tests that are run on an annual basis.
- pooper 5y ago
  In my personal experience, about seventeen years ago, retaking the test raised my score some 90 points (iirc) out of 1600, excluding writing section.
  - someguydave 5y ago
    the relevant question in that in that case is: what would your average score be with and without prep over say 10 tests
- ghaff 5y ago
  And there are books available to help with a lot of that. (And I actually agree with the point that doing some amount of test prep/sample tests is helpful. But it doesn't need to be super-expensive/time-consuming. I do understand that the playing field has probably upleveled over the decades but it's still probably as democratized as any such thing is.
- MengerSponge 5y ago
  The SAT is a test of your academic preparation, not Raven's progressive matrices. That you can study for it is not inherently a bad thing. Portions that are highly susceptible to coaching are bad, and that's why there aren't analogies any more. If students learn the vocabulary and practice the math to do better on the test, at some point it just becomes the Key and Peele Heist sketch: "That's called a job!" https://www.youtube.com/watch?v=jgYYOUC10aM https://www.youtube.com/watch?v=jgYYOUC10aM
  - GCA10 5y ago
    Agreed. I spent a bunch of time a few years ago doing home-coaching for our two teenagers as they rolled into SAT time. The way you get better at the math section is to genuinely fix whatever gaps might exist in your knowledge of algebra and geometry. As OP says, that "is not an inherently bad thing." There's probably another 20 points that can be picked up by learning to read the questions very carefully -- so that you don't race to show how quickly you can spin-up an off-task answer that precisely matches the wrong question. Getting that right also "is not an inherently bad thing." The verbal section is a bit more of a swamp, and there might be a larger element of gamesmanship there. But for schools like MIT, where math aptitude is the main event, I think keeping a math-focused role for the SAT can help a lot. It identifies not just the elite-school wizards with lots of AP and math SAT 800s -- but also the teens from humbler public schools that didn't have an AP track, but whose 790s on the (pre-calc focused) math SATs speak to their ability to play at a higher level. Apropos of analogies, I think the test-takers got rid of those because they can be ridiculously skewed to particular (affluent) cultures. For some people, it's obvious if yacht-to-dingy is akin to symphony-to-quartet. For people who grew up with less money, it is a total WTF moment.
    - WalterBright 5y ago
      I remember an analogy question that required knowledge of alcoholic drink formulations. I was way under drinking age, and had no idea what went into a martini.
      - MengerSponge 5y ago
- msdrigg 5y ago
  My own anecdotal experience confirms that sat and act tests are very studiable. Honestly even more than average tests just because there is so much material available to study with.
screye 5y ago
Yeah, I have found that it improves 3 things. (this is for GRE, which is similar) 1. Basics: If you don't know standard permutations and combinations then knowing those formulae off the top of your head is nice to have. The language portions in particular, take a lot of preparation for non-native speakers. 2. Speed: Giving a decent number of sample tests helps put you in game mode for the real thing. It also acquaints you to the manner in which questions are phrased and their intended meanings. (big deal for non-native speakers) Lastly, it helps ease anxiety. 3. Gotcha-proofing: Every examination has some familiar gotcha patterns. Some training helps in looking out for them helps.
- WalterBright 5y ago
  The gotchas are answers that match common student mistakes. What I'd do is solve the problem without looking at the answers so they wouldn't bias me. Then look for a match of mine with one of the answers.
TuringNYC 5y ago
>> I don't really have evidence for this, but it's always felt like SAT/ACT coaching doesn't improve scores so much as get rid of some of the "dumb mistakes" that cost you score. As i learned in college, the real "coaching" was rich parents getting rich kids more time on the SAT by getting psychiatric diagnostic classifications that give you more time. Time has been the real challenge on SATs for most people beyond a certain score threshold, and money can buy time.
- fhood 5y ago
  From my, albeit rather distant, recollection, if you desperately needed more time on the SAT you are probably already screwed.
  - TuringNYC 5y ago
    For students in the category of "exam easy. finished the exam w/o any time issues", the whole sub-thread is irrelevant. You're going to ace the exam rich, or poor. The sub-thread and discussion is about wealth bias for exam scores.
- WalterBright 5y ago
  I finished the SAT early, and used the extra time to go back to the beginning and verify each answer. More time wouldn't have done much. If you can answer the questions, there's enough time to complete the test.
  - mlyle 5y ago
    I understand your perspective-- my test memories were all breezing through tests with copious extra time... But as an educator I've noticed that there is a wide variation in the amount of time needed for a test between students. For some tasks it is nearly an order of magnitude. The students who are quick and on the competitive math team finish something in 6-7 minutes and some other students are doing correct work but not quite done in 45 minutes. More practice doesn't seem to make them much quicker, either. And this is in students without a formal diagnosis that allows them to spend extra time. [There was one time I crashed and burned on a test and ran out of time... where I didn't memorize enough of a big table of identities for a trig test and ended up having to derive everything from scratch]
scarecrw 5y ago
> My understanding is that's what SAT/ACT were moving towards about 10 years ago, that it would be computerized and "everyone's test is personalized" by the elo system probing your exact knowledge level from a bank of questions with difficulty scores dynamically based on how "similar ranked" students performed on that question. That is indeed where they're moving. They've recently announced that the SAT will be transitioning to a digital, adaptive test in the next 1-2 years. [1] Notably, the upcoming iteration of the test will only be semi-adaptive, adjusting the version of the second half of the test based on your performance on the first half, rather than adapting to your performance on a question-by-question basis. I suspect overall this will be an improvement in the accuracy of the results. As it stands, for students with a strong math background, a majority of the math questions on the current test are far too easy and cloud their results on the rest. With the recent removal of the CollegeBoard's math subject tests, high-level math students have very few opportunities to demonstrate their knowledge using a standardized metric. [1]: https://newsroom.collegeboard.org/digital-sat-brings-student-friendly-changes-test-experience https://newsroom.collegeboard.org/digital-sat-brings-student...
- caffeine 5y ago
  > high-level math students have very few opportunities to demonstrate their knowledge using a standardized metric. AP math and physics exams? Or IB? True that these are not accessible to everyone though..
- MaximumYComb 5y ago
  I think that is great. Once you get past a certain threshold, the test loses prediction value. Giving the kids in the top 5% a way to differentiate is great.
commandlinefan 5y ago
> SAT/ACT coaching doesn't improve scores so much as get rid of some of the "dumb mistakes" If so, that still doesn't imply that rich kids with access to private tutors will necessarily do better on these tests than poor kids - just that anybody with the motivation to read a test-prep book will.
stillsut 5y ago
I'd say one of the valuable things an SAT tutor could teach is an attitude: to take initiative, and reject resignment to failure. The biggest difference I noticed in how I would take a test versus other people I tried to coach is that I viewed the test as a fun game like a challenging video game level. And those who struggled on the test viewed it as dreadful judgement being rendered on them. It's like when you can tell someone is extremely self-conscious while dancing: Beyond teaching them any actual dance moves, you have to turn off the part of their brain which is blocking their natural mental resources for problem solving, and that's often the fear they are inadequate to the task, will disappoint their supporters, and that it will hurt their future prospects.
brewdad 5y ago
My kid didn't get to take an SAT/ACT due to Covid. They did end up with a substantially lower PSAT score due to one of those dumb mistakes. They missed one "easy" question on the math portion while nailing almost everything else. The difference between National Merit Scholar Finalist and a "really good score". They're doing great at a top 50 university now but it probably was the difference between waitlist and admit at two top 15 schools.

by paulmd 5y ago

sjg007 5y ago

Coaching and studying improves SAT scores. People learn the type of questions they do poorly on and can study to improve. The SAT is a test you can study for.

likpok 5y ago
The top link is a study that shows that the improvement is pretty marginal in practice. It does miss something: in specific ethnic enclaves SAT coaching is much more effective, perhaps because of a culture of out-of-school schoolwork and teaching beyond SAT prep. Those enclaves aren't particularly wealthy either (if I recall correctly it was a Korean enclave). Even then we're talking 70 points -- not nothing, but also not a radical transformation.
- hhs 5y ago
  > Even then we're talking 70 points -- not nothing, but also not a radical transformation. That would depend on the baseline score. For instance, if it was a 1510 baseline and then went up +70, then it would be useful.
  - whimsicalism 5y ago
    I think it's usually not improvement at the highest levels.
- chernevik 5y ago
  I wouldn't be surprised if those improvements were mostly moving below-average scores toward the average, by giving deprived students basic skills that their "education" didn't.
boringg 5y ago
I don't know many tests you can't study for especially tests that are run on an annual basis.
pooper 5y ago
In my personal experience, about seventeen years ago, retaking the test raised my score some 90 points (iirc) out of 1600, excluding writing section.
- someguydave 5y ago
  the relevant question in that in that case is: what would your average score be with and without prep over say 10 tests
ghaff 5y ago
And there are books available to help with a lot of that. (And I actually agree with the point that doing some amount of test prep/sample tests is helpful. But it doesn't need to be super-expensive/time-consuming. I do understand that the playing field has probably upleveled over the decades but it's still probably as democratized as any such thing is.
MengerSponge 5y ago
The SAT is a test of your academic preparation, not Raven's progressive matrices. That you can study for it is not inherently a bad thing. Portions that are highly susceptible to coaching are bad, and that's why there aren't analogies any more. If students learn the vocabulary and practice the math to do better on the test, at some point it just becomes the Key and Peele Heist sketch: "That's called a job!" https://www.youtube.com/watch?v=jgYYOUC10aM https://www.youtube.com/watch?v=jgYYOUC10aM
- GCA10 5y ago
  Agreed. I spent a bunch of time a few years ago doing home-coaching for our two teenagers as they rolled into SAT time. The way you get better at the math section is to genuinely fix whatever gaps might exist in your knowledge of algebra and geometry. As OP says, that "is not an inherently bad thing." There's probably another 20 points that can be picked up by learning to read the questions very carefully -- so that you don't race to show how quickly you can spin-up an off-task answer that precisely matches the wrong question. Getting that right also "is not an inherently bad thing." The verbal section is a bit more of a swamp, and there might be a larger element of gamesmanship there. But for schools like MIT, where math aptitude is the main event, I think keeping a math-focused role for the SAT can help a lot. It identifies not just the elite-school wizards with lots of AP and math SAT 800s -- but also the teens from humbler public schools that didn't have an AP track, but whose 790s on the (pre-calc focused) math SATs speak to their ability to play at a higher level. Apropos of analogies, I think the test-takers got rid of those because they can be ridiculously skewed to particular (affluent) cultures. For some people, it's obvious if yacht-to-dingy is akin to symphony-to-quartet. For people who grew up with less money, it is a total WTF moment.
  - WalterBright 5y ago
    I remember an analogy question that required knowledge of alcoholic drink formulations. I was way under drinking age, and had no idea what went into a martini.
    - MengerSponge 5y ago
msdrigg 5y ago
My own anecdotal experience confirms that sat and act tests are very studiable. Honestly even more than average tests just because there is so much material available to study with.

screye 5y ago

Yeah, I have found that it improves 3 things. (this is for GRE, which is similar) 1. Basics: If you don't know standard permutations and combinations then knowing those formulae off the top of your head is nice to have. The language portions in particular, take a lot of preparation for non-native speakers. 2. Speed: Giving a decent number of sample tests helps put you in game mode for the real thing. It also acquaints you to the manner in which questions are phrased and their intended meanings. (big deal for non-native speakers) Lastly, it helps ease anxiety. 3. Gotcha-proofing: Every examination has some familiar gotcha patterns. Some training helps in looking out for them helps.

WalterBright 5y ago
The gotchas are answers that match common student mistakes. What I'd do is solve the problem without looking at the answers so they wouldn't bias me. Then look for a match of mine with one of the answers.

TuringNYC 5y ago

>> I don't really have evidence for this, but it's always felt like SAT/ACT coaching doesn't improve scores so much as get rid of some of the "dumb mistakes" that cost you score. As i learned in college, the real "coaching" was rich parents getting rich kids more time on the SAT by getting psychiatric diagnostic classifications that give you more time. Time has been the real challenge on SATs for most people beyond a certain score threshold, and money can buy time.

fhood 5y ago
From my, albeit rather distant, recollection, if you desperately needed more time on the SAT you are probably already screwed.
- TuringNYC 5y ago
  For students in the category of "exam easy. finished the exam w/o any time issues", the whole sub-thread is irrelevant. You're going to ace the exam rich, or poor. The sub-thread and discussion is about wealth bias for exam scores.
WalterBright 5y ago
I finished the SAT early, and used the extra time to go back to the beginning and verify each answer. More time wouldn't have done much. If you can answer the questions, there's enough time to complete the test.
- mlyle 5y ago
  I understand your perspective-- my test memories were all breezing through tests with copious extra time... But as an educator I've noticed that there is a wide variation in the amount of time needed for a test between students. For some tasks it is nearly an order of magnitude. The students who are quick and on the competitive math team finish something in 6-7 minutes and some other students are doing correct work but not quite done in 45 minutes. More practice doesn't seem to make them much quicker, either. And this is in students without a formal diagnosis that allows them to spend extra time. [There was one time I crashed and burned on a test and ran out of time... where I didn't memorize enough of a big table of identities for a trig test and ended up having to derive everything from scratch]

scarecrw 5y ago

> My understanding is that's what SAT/ACT were moving towards about 10 years ago, that it would be computerized and "everyone's test is personalized" by the elo system probing your exact knowledge level from a bank of questions with difficulty scores dynamically based on how "similar ranked" students performed on that question. That is indeed where they're moving. They've recently announced that the SAT will be transitioning to a digital, adaptive test in the next 1-2 years. [1] Notably, the upcoming iteration of the test will only be semi-adaptive, adjusting the version of the second half of the test based on your performance on the first half, rather than adapting to your performance on a question-by-question basis. I suspect overall this will be an improvement in the accuracy of the results. As it stands, for students with a strong math background, a majority of the math questions on the current test are far too easy and cloud their results on the rest. With the recent removal of the CollegeBoard's math subject tests, high-level math students have very few opportunities to demonstrate their knowledge using a standardized metric. [1]: https://newsroom.collegeboard.org/digital-sat-brings-student-friendly-changes-test-experience https://newsroom.collegeboard.org/digital-sat-brings-student...

caffeine 5y ago
> high-level math students have very few opportunities to demonstrate their knowledge using a standardized metric. AP math and physics exams? Or IB? True that these are not accessible to everyone though..
MaximumYComb 5y ago
I think that is great. Once you get past a certain threshold, the test loses prediction value. Giving the kids in the top 5% a way to differentiate is great.

commandlinefan 5y ago

> SAT/ACT coaching doesn't improve scores so much as get rid of some of the "dumb mistakes" If so, that still doesn't imply that rich kids with access to private tutors will necessarily do better on these tests than poor kids - just that anybody with the motivation to read a test-prep book will.

stillsut 5y ago

I'd say one of the valuable things an SAT tutor could teach is an attitude: to take initiative, and reject resignment to failure. The biggest difference I noticed in how I would take a test versus other people I tried to coach is that I viewed the test as a fun game like a challenging video game level. And those who struggled on the test viewed it as dreadful judgement being rendered on them. It's like when you can tell someone is extremely self-conscious while dancing: Beyond teaching them any actual dance moves, you have to turn off the part of their brain which is blocking their natural mental resources for problem solving, and that's often the fear they are inadequate to the task, will disappoint their supporters, and that it will hurt their future prospects.

brewdad 5y ago

My kid didn't get to take an SAT/ACT due to Covid. They did end up with a substantially lower PSAT score due to one of those dumb mistakes. They missed one "easy" question on the math portion while nailing almost everything else. The difference between National Merit Scholar Finalist and a "really good score". They're doing great at a top 50 university now but it probably was the difference between waitlist and admit at two top 15 schools.