8 ms·

Not surprising. If only a small percentage of students take the test, it's only going to be the students on the top end of the bell curve. If everybody takes th

by swebs 7y ago

Not surprising. If only a small percentage of students take the test, it's only going to be the students on the top end of the bell curve. If everybody takes the test, you're going to get results that reflects the general population. http://archive.is/6GW9b http://archive.is/6GW9b

acdha 7y ago
This is the detail which needs to be added to almost all mainstream coverage of education stats — every year you'll see some hand-wringing about U.S. students falling behind internationally and the authors never mention that the effect is much weaker once you correct for the fact the U.S. has a much wider range of students taking most tests. Usually almost all of these metrics come back to the root cause of economic disparities along the spectrum from hard poverty to not being able to afford tutors and SAT prep courses. (This is also why the cool curriculum ideas or charter school concepts we hear about never go anywhere: the TED Talk sounds great but inevitably once it scales up it turns out that the effects were due to some sort of selection bias (quite possibly unintentional) or simply lucky small sample sizes which peters out at larger scale)
- _Nat_ 7y ago
  > from hard poverty to not being able to afford tutors and SAT prep courses. I went to a pretty fancy prep school with a lot of rich folks, but I don't think most did any special prep for the SAT. I think we had a class period where they talked about the sorts of questions that appear on the exam, told us about when/where it'd be offered, etc., along with ads for SAT-prep services, but I think a lot of folks ended up deciding it would be best to get a good-night's sleep before walking in on it. (We also debated stuff like if drinking coffee or/and having a chocolate bar beforehand would be helpful.) Which isn't to downplay the advantages that affluent students enjoy, just, I don't think that SAT prep courses are really the big thing. (Plus I suspect that students who spend a lot of time doing test-specific prep are losing out on their actual education that they'll actually need later on.)
  - gumby 7y ago
    Same: in 1981 our SAT "prep" was literally two sentences one morning: 1 - there will be little bubbles you need to fill in and you must use a pencil and 2 - some statistical strategy concerning guessing. Everybody took the test only once; I don't remember any efforts to game the system. But that in itself points out the advantage: the test was basically designed to reflect the level of instruction my school already gave. If you don't get that from your school (or believed you don't) you would have resorted to additional study and the like. Likewise back then there weren't "AP" classes and the like: if you wanted to take the AP test you could but there were no special arrangements at my school. I don't know how much things might have changed: my kid's SAT prep in 2017 was simply to purchase the book but not open it. On the other hand there's a whole apparatus of AP infrastructure with special "AP" classes, different scoring for those classes(!!) etc. AN enormous kabuki of infrastructure that as far as I can tell signifies nothing.
  - akhilcacharya 7y ago
    > I think we had a class period where they talked about the sorts of questions that appear on the exam, told us about when/where it'd be offered, etc., along with ads for SAT-prep services, but I think a lot of folks ended up deciding it would be best to get a good-night's sleep before walking in on it. That's actually non-trivial amount of prep, but I personally don't know anybody that took the SAT "cold". Everyone prepped quite a bit - I prepped for months and got mediocre scores consistently ¯\_(ツ)_/¯
    - _Nat_ 7y ago
      Yeah.. prep doesn't seem too effective. That's what they told us at prep school (heh). I mean, they basically just said that we ought to ensure that we have sharpened pencils; adequate batteries for our calculator; a good-night's sleep; and to show up early. I can appreciate how some folks want to do better, and are willing to work for better scores by investing in test-prep. And I can understand how test-prep providers would encourage them to do so, talking up their own services, as any business would. But, I don't think it's that big a thing in the end. If anything, I worry that students who spend a lot of time prepping are being cheated out of their time, money, and education. --- PS: To be clear, students would definitely do well to spend time studying for their future. Just, instead of trying to game the SAT, they can learn something useful, whether that's math, physics, computer programming, philosophy, creative writing, or whatever.
- gumby 7y ago
  > the U.S. has a much wider range of students taking most tests. Citation? All countries have a mix of wealthy, educated students, working class students, students whose parents are drug addicts, immigrants who don't speak the local language, etc. In my experience the US isn't that different from other wealthy countries I've lived in. SAT isn't really used for international comparison; a system like PISA does cross-sectional studies of countries, not elites. The results are not encouraging for the USA: https://en.wikipedia.org/wiki/Programme_for_International_Student_Assessment https://en.wikipedia.org/wiki/Programme_for_International_St... https://nces.ed.gov/surveys/pisa/ https://nces.ed.gov/surveys/pisa/ http://www.oecd.org/pisa/ http://www.oecd.org/pisa/
  - seanmcdirmid 7y ago
    Wasn’t Shanghai famous for getting really good PISA scores, that being the richest city in China?
    - gumby 7y ago
      The Shanghai results are quite controversial, not only in terms of accuracy but whether they might skew Chinese results significantly. Not being in the field (and not really caring much about it TBH) I don't know how to evaluate the controversy or even its legitimacy.
  - mieseratte 7y ago
    > Citation? All countries have a mix of wealthy, educated students, working class students, students whose parents are drug addicts, immigrants who don't speak the local language, etc. In my experience the US isn't that different from other wealthy countries I've lived in. Yes, but many European countries engage in so-called tracking of students, putting them in separate tracks for tradeswork, skilled labor, and university such that the "dumb" ones aren't taking university entrance exams. In the US this has become increasingly frowned upon by (perhaps) well-meaning administrators who feel that to do this is unconscionable. In my freshman year of high-school they had advanced academic and standard groupings in the same school, but removed this my sophmore year. They re-branded all the base-level courses as "College Prep," with "Honors" being the former advanced-track classes, AP and IB courses already being branded as such. Of course, if you were in the advanced group the counselors and teachers would all push you into AP and IB courses but they also encouraged everyone to take the SAT and ACT tests, to apply to university, and discouraged enrolling in the trades program that the county offered (half-day academic, half-day trades classes). This type of "everyone goes to college" mentality is why the SAT is a poor academic bellwether.
- losvedir 7y ago
  > Usually almost all of these metrics come back to the root cause of economic disparities along the spectrum from hard poverty to not being able to afford tutors and SAT prep courses. Do you have any resources for how much an SAT score can change from studying? In my experience offering free ACT tutoring at the local YMCA, I wasn't able to improve scores that much. Maybe I'm just a bad tutor. I did pretty well on the SAT when I took it, with little prep.
  - porknubbins 7y ago
    My experience with standardized testing is that there are some unprepared students who are not familiar with, or fail to use basic strategies like process of elimination, jumping between question and text, etc and these students will get a hugely significant boost, but beyond that progress from tutoring at best equal to self studying. I also did well with little prep but that was because of my highschool experience.
  - Loughla 7y ago
    My experience working with outreach programs taught me that we would see a 1 or 2% increase from focused efforts on content mastery in SAT/ACT prep classes. When we focused on specifically the skills needed to do well on 'generic' standardized tests, we saw a 7-12% increase in individual student score. These tests mostly test how well someone (a) responds to pressure and (b) takes a standardized test. They're not accurate measures of student learning.
    - ksdale 7y ago
      This was my experience. In my SAT prep class 15 years ago, the first thing the instructor did was talk about the statistics of guessing and how even if you don’t know any answers at all, you can improve your score dramatically just by eliminating the obviously wrong answers before you guess. Then we went about learning ways to find the obviously wrong answers based on the format of the question that was asked. We also took practice tests, but it was the generic test taking rules that made the difference for me. I did very, very well on the SAT and then later, I applied the same rules to the LSAT and did much less prep and did very well on that too.
- localhost 7y ago
  I have a 10th grader who is just starting out on his SAT/ACT journey. A combination of Khan Academy[1] and a $10 book is his prep. I'm interested in how much these resources and the deliberate practice that goes along with them will improve his baseline scores. [1] https://www.khanacademy.org/test-prep/sat https://www.khanacademy.org/test-prep/sat [2] https://www.amazon.com/gp/product/0525567658/ref=ppx_yo_dt_b_asin_title_o04_s01?ie=UTF8&psc=1 https://www.amazon.com/gp/product/0525567658/ref=ppx_yo_dt_b...
SkyPuncher 7y ago
I think I took the ACT instead of the SAT, as the ACT was the more popular exam of my day. The only people that took the SAT in addition to the ACT, were: 1. Applicants to "elite" schools that preferred the SAT. 2. Smart kids trying to get a leg up for large state schools. 3. Pretty smart kids trying to get a bump into a higher scholarship bracket. If your were anything less than an 80th percentile student, it'd be extremely unusual for you to take both the ACT and SAT. This was also at a Catholic school - most parents pushed their kids to do well academically to make tuition worthwhile.
Mountain_Skies 7y ago
A good example of this was when Mississippi test takers routinely outscored those in New York taking the test. There were two reasons. The first was that New York required all students to take the SAT while in Mississippi only students interested in attending college took the entrance exams. The other factor was that the ACT was much more commonly used by admissions offices in Mississippi and surrounding states. So out of college bound students, the SAT typically was only taken by a subset planning on attending schools further away, which were much more likely to be Ivy League or other types of schools that attract students much more broadly than the average state school.
deleted 7y ago
[deleted]

by swebs 7y ago

acdha 7y ago

This is the detail which needs to be added to almost all mainstream coverage of education stats — every year you'll see some hand-wringing about U.S. students falling behind internationally and the authors never mention that the effect is much weaker once you correct for the fact the U.S. has a much wider range of students taking most tests. Usually almost all of these metrics come back to the root cause of economic disparities along the spectrum from hard poverty to not being able to afford tutors and SAT prep courses. (This is also why the cool curriculum ideas or charter school concepts we hear about never go anywhere: the TED Talk sounds great but inevitably once it scales up it turns out that the effects were due to some sort of selection bias (quite possibly unintentional) or simply lucky small sample sizes which peters out at larger scale)

_Nat_ 7y ago
> from hard poverty to not being able to afford tutors and SAT prep courses. I went to a pretty fancy prep school with a lot of rich folks, but I don't think most did any special prep for the SAT. I think we had a class period where they talked about the sorts of questions that appear on the exam, told us about when/where it'd be offered, etc., along with ads for SAT-prep services, but I think a lot of folks ended up deciding it would be best to get a good-night's sleep before walking in on it. (We also debated stuff like if drinking coffee or/and having a chocolate bar beforehand would be helpful.) Which isn't to downplay the advantages that affluent students enjoy, just, I don't think that SAT prep courses are really the big thing. (Plus I suspect that students who spend a lot of time doing test-specific prep are losing out on their actual education that they'll actually need later on.)
- gumby 7y ago
  Same: in 1981 our SAT "prep" was literally two sentences one morning: 1 - there will be little bubbles you need to fill in and you must use a pencil and 2 - some statistical strategy concerning guessing. Everybody took the test only once; I don't remember any efforts to game the system. But that in itself points out the advantage: the test was basically designed to reflect the level of instruction my school already gave. If you don't get that from your school (or believed you don't) you would have resorted to additional study and the like. Likewise back then there weren't "AP" classes and the like: if you wanted to take the AP test you could but there were no special arrangements at my school. I don't know how much things might have changed: my kid's SAT prep in 2017 was simply to purchase the book but not open it. On the other hand there's a whole apparatus of AP infrastructure with special "AP" classes, different scoring for those classes(!!) etc. AN enormous kabuki of infrastructure that as far as I can tell signifies nothing.
- akhilcacharya 7y ago
  > I think we had a class period where they talked about the sorts of questions that appear on the exam, told us about when/where it'd be offered, etc., along with ads for SAT-prep services, but I think a lot of folks ended up deciding it would be best to get a good-night's sleep before walking in on it. That's actually non-trivial amount of prep, but I personally don't know anybody that took the SAT "cold". Everyone prepped quite a bit - I prepped for months and got mediocre scores consistently ¯\_(ツ)_/¯
  - _Nat_ 7y ago
    Yeah.. prep doesn't seem too effective. That's what they told us at prep school (heh). I mean, they basically just said that we ought to ensure that we have sharpened pencils; adequate batteries for our calculator; a good-night's sleep; and to show up early. I can appreciate how some folks want to do better, and are willing to work for better scores by investing in test-prep. And I can understand how test-prep providers would encourage them to do so, talking up their own services, as any business would. But, I don't think it's that big a thing in the end. If anything, I worry that students who spend a lot of time prepping are being cheated out of their time, money, and education. --- PS: To be clear, students would definitely do well to spend time studying for their future. Just, instead of trying to game the SAT, they can learn something useful, whether that's math, physics, computer programming, philosophy, creative writing, or whatever.
gumby 7y ago
> the U.S. has a much wider range of students taking most tests. Citation? All countries have a mix of wealthy, educated students, working class students, students whose parents are drug addicts, immigrants who don't speak the local language, etc. In my experience the US isn't that different from other wealthy countries I've lived in. SAT isn't really used for international comparison; a system like PISA does cross-sectional studies of countries, not elites. The results are not encouraging for the USA: https://en.wikipedia.org/wiki/Programme_for_International_Student_Assessment https://en.wikipedia.org/wiki/Programme_for_International_St... https://nces.ed.gov/surveys/pisa/ https://nces.ed.gov/surveys/pisa/ http://www.oecd.org/pisa/ http://www.oecd.org/pisa/
- seanmcdirmid 7y ago
  Wasn’t Shanghai famous for getting really good PISA scores, that being the richest city in China?
  - gumby 7y ago
    The Shanghai results are quite controversial, not only in terms of accuracy but whether they might skew Chinese results significantly. Not being in the field (and not really caring much about it TBH) I don't know how to evaluate the controversy or even its legitimacy.
- mieseratte 7y ago
  > Citation? All countries have a mix of wealthy, educated students, working class students, students whose parents are drug addicts, immigrants who don't speak the local language, etc. In my experience the US isn't that different from other wealthy countries I've lived in. Yes, but many European countries engage in so-called tracking of students, putting them in separate tracks for tradeswork, skilled labor, and university such that the "dumb" ones aren't taking university entrance exams. In the US this has become increasingly frowned upon by (perhaps) well-meaning administrators who feel that to do this is unconscionable. In my freshman year of high-school they had advanced academic and standard groupings in the same school, but removed this my sophmore year. They re-branded all the base-level courses as "College Prep," with "Honors" being the former advanced-track classes, AP and IB courses already being branded as such. Of course, if you were in the advanced group the counselors and teachers would all push you into AP and IB courses but they also encouraged everyone to take the SAT and ACT tests, to apply to university, and discouraged enrolling in the trades program that the county offered (half-day academic, half-day trades classes). This type of "everyone goes to college" mentality is why the SAT is a poor academic bellwether.
losvedir 7y ago
> Usually almost all of these metrics come back to the root cause of economic disparities along the spectrum from hard poverty to not being able to afford tutors and SAT prep courses. Do you have any resources for how much an SAT score can change from studying? In my experience offering free ACT tutoring at the local YMCA, I wasn't able to improve scores that much. Maybe I'm just a bad tutor. I did pretty well on the SAT when I took it, with little prep.
- porknubbins 7y ago
  My experience with standardized testing is that there are some unprepared students who are not familiar with, or fail to use basic strategies like process of elimination, jumping between question and text, etc and these students will get a hugely significant boost, but beyond that progress from tutoring at best equal to self studying. I also did well with little prep but that was because of my highschool experience.
- Loughla 7y ago
  My experience working with outreach programs taught me that we would see a 1 or 2% increase from focused efforts on content mastery in SAT/ACT prep classes. When we focused on specifically the skills needed to do well on 'generic' standardized tests, we saw a 7-12% increase in individual student score. These tests mostly test how well someone (a) responds to pressure and (b) takes a standardized test. They're not accurate measures of student learning.
  - ksdale 7y ago
    This was my experience. In my SAT prep class 15 years ago, the first thing the instructor did was talk about the statistics of guessing and how even if you don’t know any answers at all, you can improve your score dramatically just by eliminating the obviously wrong answers before you guess. Then we went about learning ways to find the obviously wrong answers based on the format of the question that was asked. We also took practice tests, but it was the generic test taking rules that made the difference for me. I did very, very well on the SAT and then later, I applied the same rules to the LSAT and did much less prep and did very well on that too.
localhost 7y ago
I have a 10th grader who is just starting out on his SAT/ACT journey. A combination of Khan Academy[1] and a $10 book is his prep. I'm interested in how much these resources and the deliberate practice that goes along with them will improve his baseline scores. [1] https://www.khanacademy.org/test-prep/sat https://www.khanacademy.org/test-prep/sat [2] https://www.amazon.com/gp/product/0525567658/ref=ppx_yo_dt_b_asin_title_o04_s01?ie=UTF8&psc=1 https://www.amazon.com/gp/product/0525567658/ref=ppx_yo_dt_b...

SkyPuncher 7y ago

I think I took the ACT instead of the SAT, as the ACT was the more popular exam of my day. The only people that took the SAT in addition to the ACT, were: 1. Applicants to "elite" schools that preferred the SAT. 2. Smart kids trying to get a leg up for large state schools. 3. Pretty smart kids trying to get a bump into a higher scholarship bracket. If your were anything less than an 80th percentile student, it'd be extremely unusual for you to take both the ACT and SAT. This was also at a Catholic school - most parents pushed their kids to do well academically to make tuition worthwhile.

Mountain_Skies 7y ago

A good example of this was when Mississippi test takers routinely outscored those in New York taking the test. There were two reasons. The first was that New York required all students to take the SAT while in Mississippi only students interested in attending college took the entrance exams. The other factor was that the ACT was much more commonly used by admissions offices in Mississippi and surrounding states. So out of college bound students, the SAT typically was only taken by a subset planning on attending schools further away, which were much more likely to be Ivy League or other types of schools that attract students much more broadly than the average state school.

deleted 7y ago

[deleted]