5 ms·

It is time we abandon SAT and ACT tests.

by epalmer 11y ago

threatofrain 11y ago
SAT II's are more predictive of college performance on their own than 4 years of high school grades, but standardized tests like these take a fraction of the time, as opposed to years of intimate teacher evaluations. The compromised operational security of these exams don't stop them from being more predictive than teachers. So if we want to retire these tests, then we need something better. What we should really be talking about is improving these tests and beefing up operational security.
- EliRivers 11y ago
  Serious question; on what basis is "how well they will do at college" the right metric to use for selecting who to accept?
  - danieltillett 11y ago
    If you run a college then selecting people who will do well should maximise your graduation rate. Of course if this is what should be optimised is an open question.
    - adrianN 11y ago
      Or you could keep graduation rate constant and make the taught material harder.
      - kuschku 11y ago
        Which will lead to a revolution if you have paid colleges. People apply, pay for a year or two, inevitably fail, but are now also 30k in debt. What do you think those people will do? Accepting anyone only works in countries without paid universities.
  - jasode 11y ago
    >on what basis is "how well they will do at college" the right metric use for selecting who to accept? Well, the underlying basis is the finite number of seats for students. If you have a limited number of slots, you must reject some applicants. Therefore, the question flipped around is, "what criteria to use to reject applicants?" That leads into what I guess your usage of "basis" is actually about: the philosophy of selecting on the "doing well" instead of some other criteria such as... "economic disadvantages?"... or "local student over out-of-state or foreign student?"..."athletic abilities?", etc. You'd have to expand on what alternatives you had in mind. I think the "doing well" criteria can be defended by filling up the limited seats with students who will get the most out of college. For example, a student taking "Physics I" needs to know what derivatives are (Calculus). Having some slots used up by students that should have been in remedial math courses (Algebra) instead of flunking out of PHYICS1 is unfair to the potential students who already have the baseline math skills required for solving physics equations. Multiply that example across the entire curriculum required for a degree and the "doing well" criteria seems justified. You want to reject potential students who are not able to succeed at college-level course work. Maybe tests like SAT are bad/flawed but so far, they appear to be the "least bad" tool to predict which students will do well in college. One can create institutions that don't care about the assessment of "doing well" as an entrance criteria as long as the students pay the fees. That signal is provided by many colleges[1] that don't require SAT/ACT scores. However, the prestige and respect of those institutions don't match the ones that do. That negative perception in the marketplace is not uncorrelated to their acceptance criteria. (E.g. University of Phoenix.) [1]http://www.collegexpress.com/lists/list/colleges-that-do-not-consider-standardized-test-scores/2083/ http://www.collegexpress.com/lists/list/colleges-that-do-not...
  - kaitai 11y ago
    An excellent question. In practice, that's not who we select. One college in the Midwest I know of selected 1) kids related to alumni, 2) people of a certain religion, 3) people who would add to the sports and music programs, 4) kids of color with good grades and high SAT scores, and last 5) people who didn't fit the above, especially from California, Texas, or Florida, who could pay full tuition. A major goal at some small colleges is balancing kids who will pay full price and kids who will do well at college, which creates a funny situation where you've got a well off heavy-majority-white party crowd with low test scores and a smaller and more diverse group of subsidized kids who bring many academic and non-academic talents (the music/sports/math/science/high school leadership crowd that may not have well-off parents).
  - jerf 11y ago
    Flip it around; how is it good to accept someone to college who does not have a good chance of doing well? How good is it to accept their money, probably put them in debt (even if not the catastrophic $100,000+ debt, even $20,000 debt can be quite a lot for a young person, and it's hard to get out of 4 years without at least that much), and claim very significant portions of some of the most important years of their life [1] if you already have good reason to believe they are unlikely to do well? (And remember we're judging lots of people so statistical bases are all we can use.) It may not be the only criterion of use, but "do we have good reason to believe they will succeed in college?" is at the very least a critical bar that ought to be passed. It's can be the difference between college as the empowering, education experience it's supposed to be, and college as the meat-grinder wealth-extraction tool it can be when you feed in students who shouldn't be there. [1]: Not necessarily "best", but being "earliest" in a world of compounding interest makes them quite valuable in their own way.
    - jtbigwoo 11y ago
      It's also true that the better less-selective (or non-selective) colleges provide more help than very selective colleges. I went to community college for a year and then went to a selective private college. The community college had far more help sessions, tutors, and study aids available. There was even a program at the community college that specifically sought out first-generation students to make sure they were getting extra support.
  - superuser2 11y ago
    I can try: universities differ in rigor and students differ in capability. A student gets the most out of education (and becomes more capable) when presented with a challenge just below his threshold. Put him in a program that's too hard, and he'll have a shitty time while learning nothing and wasting a lot of money before washing out. Put him in a program that's not hard enough, and he'll succeed within it but his growth will be suboptimal. Match students to the colleges that will be hard enough to get them to grow, but not so hard that they fail.
- labster 11y ago
  It might not matter what is on the exams. SAT could just be testing for test-taking ability. I'd assume test-taking ability would be a good predictor of college performance too, as exams are typically 40-90% of college grades in my experience. Is there any data out there which tries to measure test skills independently?
  - threatofrain 11y ago
    What do you propose are the factors behind test-taking ability between-subjects? I would propose that a major contributor is g, or general mental ability, and I propose it is for that reason that the SAT II's have a non-trivial relationship to college performance, more than 4 years of high school grades.
  - Spooky23 11y ago
    They don't test for test-taking ability, they test for wherewithal to enroll in test prep classes. I can't speak to the current state of affairs, but in the mid 90s, taking a 5 week Kaplan course in my peer group raised scores from the 1150-1300 range to the 1300-1500 range. That meant the difference between average state school and high probability of hitting a "reach" school. Many of the ultra achievers on these tests start prep at age 10 or younger.
    - threatofrain 11y ago
      That just means that we should think about accessibility issues, but for whatever reason, teachers as a whole, despite intimate access for 4 years to students, are still less predictive of future performance than a test developed from afar.
    - thedufer 11y ago
      On my phone so I don't have the paper handy, but all available research shows no more than 50 point gains on SAT from test prep.
      - seanmcdirmid 11y ago
- guard-of-terra 11y ago
  "standardized tests like these take a fraction of the time" Preparing for standardized tests ruins a fraction of life.
  - logicchains 11y ago
    Is this a criticism, or a positive via the implication that high school ruins a much large fraction?
    - deleted 11y ago
      [deleted]
- crispyambulance 11y ago
  SAT's are not used universally, nor are they the only factor in determining acceptance when they are used. Nor SHOULD they be. Whatever the case, it is not surprising that sketchy cram schools are cheating.
  - threatofrain 11y ago
    Whatever is the most performant set of predictive instruments should be used as a factor for acceptance for colleges seeking to optimize on whatever they feel are the "best" students. College essays? Sure, if the instrument is performant. SAT II's? Sure, if the instrument is performant. Grades? Sure, if the instrument is performant. Or extra-curriculars, or whatever.
    - crispyambulance 11y ago
      "Whatever" is certainly the case. College admissions at many of the better schools don't simply try to get "all the best" students that they can. They look to balance many factors, including admitting some fraction of students who have connections or can pay full tuition, including taking a risk on some students who would otherwise not be admitted but show some promise, including throttling-down the acceptance of over-represented groups. This is a fact. "predictive performant instruments" sure sounds very rigorous and objective, but you're talking about young people who are undergoing a lot of change. They're moving targets and predicting future performance based on one test only goes so far. A test like the SAT is always going to be a blunt instrument as is any singular factor. Admissions officers recognize this.
      - superuser2 11y ago
        >or can pay full tuition Can you name some "better schools" whose admissions are not need-blind? My understanding is that this is pretty much universal among prestigious schools.
        crispyambulance 11y ago
- jerf 11y ago
  "What we should really be talking about is improving these tests and beefing up operational security." We have a real-world instance of the DRM problem here. Once you've put a static exam physically into the hands of the students, it's compromised. The attackers have massively, massively more resources than the defenders. The attackers can hire a hundred students to each take one picture of one page, and put the test back together themselves. And there's no way that the proctors can prevent students sneaking one picture of one page nowadays. (Also, I'm not saying that's an optimal attack. I just put about 5 seconds' thought into it. But it's a sufficient attack already, so the existence of even further clever attacks only makes my point stronger.) Even presenting the problems one at a time on a screen (massively more expensive, btw, due to brand new hardware requirements, plus introduction of other attack vectors) would not be sufficient. Clever games played with contrast or whatever won't work because the test needs to be highly accessible, so you can't play games like the old copy protection booklets [1], either. Basically, if the test is static, it's game over. Not that students aren't fairly good at cheating with dynamic questions. But those really only work well with the math questions, and even the dumbed-down calculators they still sell today can be convinced to run programs sufficient to "crack" any dynamic problems that the test companies can get their hands on for the math portions. It's basically hopeless. The adversaries have gotten too darned powerful. They're armed with too much gear that would make old-school spies weep tears of joy. [1]: http://i.imgur.com/eU8Rvhh.jpg http://i.imgur.com/eU8Rvhh.jpg
- 1337biz 11y ago
  Might be true, still trying to measure the academic potential of a highschool student by a private company is absurd. I am all for participants doing their very best to "hack" the system. Kudos to those companies whelping them by take up the battle with the testing industry.
deleted 11y ago
[deleted]
VLM 11y ago
Wait for the higher ed crash. All things that can't go on forever, eventually stop. At that point the ACT/SAT problem will automatically solve itself.

It is time we abandon SAT and ACT tests.

by epalmer 11y ago

It is time we abandon SAT and ACT tests.

SAT II's are more predictive of college performance on their own than 4 years of high school grades, but standardized tests like these take a fraction of the time, as opposed to years of intimate teacher evaluations. The compromised operational security of these exams don't stop them from being more predictive than teachers. So if we want to retire these tests, then we need something better. What we should really be talking about is improving these tests and beefing up operational security.

EliRivers 11y ago
Serious question; on what basis is "how well they will do at college" the right metric to use for selecting who to accept?
- danieltillett 11y ago
  If you run a college then selecting people who will do well should maximise your graduation rate. Of course if this is what should be optimised is an open question.
  - adrianN 11y ago
    Or you could keep graduation rate constant and make the taught material harder.
    - kuschku 11y ago
      Which will lead to a revolution if you have paid colleges. People apply, pay for a year or two, inevitably fail, but are now also 30k in debt. What do you think those people will do? Accepting anyone only works in countries without paid universities.
- jasode 11y ago
  >on what basis is "how well they will do at college" the right metric use for selecting who to accept? Well, the underlying basis is the finite number of seats for students. If you have a limited number of slots, you must reject some applicants. Therefore, the question flipped around is, "what criteria to use to reject applicants?" That leads into what I guess your usage of "basis" is actually about: the philosophy of selecting on the "doing well" instead of some other criteria such as... "economic disadvantages?"... or "local student over out-of-state or foreign student?"..."athletic abilities?", etc. You'd have to expand on what alternatives you had in mind. I think the "doing well" criteria can be defended by filling up the limited seats with students who will get the most out of college. For example, a student taking "Physics I" needs to know what derivatives are (Calculus). Having some slots used up by students that should have been in remedial math courses (Algebra) instead of flunking out of PHYICS1 is unfair to the potential students who already have the baseline math skills required for solving physics equations. Multiply that example across the entire curriculum required for a degree and the "doing well" criteria seems justified. You want to reject potential students who are not able to succeed at college-level course work. Maybe tests like SAT are bad/flawed but so far, they appear to be the "least bad" tool to predict which students will do well in college. One can create institutions that don't care about the assessment of "doing well" as an entrance criteria as long as the students pay the fees. That signal is provided by many colleges[1] that don't require SAT/ACT scores. However, the prestige and respect of those institutions don't match the ones that do. That negative perception in the marketplace is not uncorrelated to their acceptance criteria. (E.g. University of Phoenix.) [1]http://www.collegexpress.com/lists/list/colleges-that-do-not-consider-standardized-test-scores/2083/ http://www.collegexpress.com/lists/list/colleges-that-do-not...
- kaitai 11y ago
  An excellent question. In practice, that's not who we select. One college in the Midwest I know of selected 1) kids related to alumni, 2) people of a certain religion, 3) people who would add to the sports and music programs, 4) kids of color with good grades and high SAT scores, and last 5) people who didn't fit the above, especially from California, Texas, or Florida, who could pay full tuition. A major goal at some small colleges is balancing kids who will pay full price and kids who will do well at college, which creates a funny situation where you've got a well off heavy-majority-white party crowd with low test scores and a smaller and more diverse group of subsidized kids who bring many academic and non-academic talents (the music/sports/math/science/high school leadership crowd that may not have well-off parents).
- jerf 11y ago
  Flip it around; how is it good to accept someone to college who does not have a good chance of doing well? How good is it to accept their money, probably put them in debt (even if not the catastrophic $100,000+ debt, even $20,000 debt can be quite a lot for a young person, and it's hard to get out of 4 years without at least that much), and claim very significant portions of some of the most important years of their life [1] if you already have good reason to believe they are unlikely to do well? (And remember we're judging lots of people so statistical bases are all we can use.) It may not be the only criterion of use, but "do we have good reason to believe they will succeed in college?" is at the very least a critical bar that ought to be passed. It's can be the difference between college as the empowering, education experience it's supposed to be, and college as the meat-grinder wealth-extraction tool it can be when you feed in students who shouldn't be there. [1]: Not necessarily "best", but being "earliest" in a world of compounding interest makes them quite valuable in their own way.
  - jtbigwoo 11y ago
    It's also true that the better less-selective (or non-selective) colleges provide more help than very selective colleges. I went to community college for a year and then went to a selective private college. The community college had far more help sessions, tutors, and study aids available. There was even a program at the community college that specifically sought out first-generation students to make sure they were getting extra support.
- superuser2 11y ago
  I can try: universities differ in rigor and students differ in capability. A student gets the most out of education (and becomes more capable) when presented with a challenge just below his threshold. Put him in a program that's too hard, and he'll have a shitty time while learning nothing and wasting a lot of money before washing out. Put him in a program that's not hard enough, and he'll succeed within it but his growth will be suboptimal. Match students to the colleges that will be hard enough to get them to grow, but not so hard that they fail.
labster 11y ago
It might not matter what is on the exams. SAT could just be testing for test-taking ability. I'd assume test-taking ability would be a good predictor of college performance too, as exams are typically 40-90% of college grades in my experience. Is there any data out there which tries to measure test skills independently?
- threatofrain 11y ago
  What do you propose are the factors behind test-taking ability between-subjects? I would propose that a major contributor is g, or general mental ability, and I propose it is for that reason that the SAT II's have a non-trivial relationship to college performance, more than 4 years of high school grades.
- Spooky23 11y ago
  They don't test for test-taking ability, they test for wherewithal to enroll in test prep classes. I can't speak to the current state of affairs, but in the mid 90s, taking a 5 week Kaplan course in my peer group raised scores from the 1150-1300 range to the 1300-1500 range. That meant the difference between average state school and high probability of hitting a "reach" school. Many of the ultra achievers on these tests start prep at age 10 or younger.
  - threatofrain 11y ago
    That just means that we should think about accessibility issues, but for whatever reason, teachers as a whole, despite intimate access for 4 years to students, are still less predictive of future performance than a test developed from afar.
  - thedufer 11y ago
    On my phone so I don't have the paper handy, but all available research shows no more than 50 point gains on SAT from test prep.
    - seanmcdirmid 11y ago
guard-of-terra 11y ago
"standardized tests like these take a fraction of the time" Preparing for standardized tests ruins a fraction of life.
- logicchains 11y ago
  Is this a criticism, or a positive via the implication that high school ruins a much large fraction?
  - deleted 11y ago
    [deleted]
crispyambulance 11y ago
SAT's are not used universally, nor are they the only factor in determining acceptance when they are used. Nor SHOULD they be. Whatever the case, it is not surprising that sketchy cram schools are cheating.
- threatofrain 11y ago
  Whatever is the most performant set of predictive instruments should be used as a factor for acceptance for colleges seeking to optimize on whatever they feel are the "best" students. College essays? Sure, if the instrument is performant. SAT II's? Sure, if the instrument is performant. Grades? Sure, if the instrument is performant. Or extra-curriculars, or whatever.
  - crispyambulance 11y ago
    "Whatever" is certainly the case. College admissions at many of the better schools don't simply try to get "all the best" students that they can. They look to balance many factors, including admitting some fraction of students who have connections or can pay full tuition, including taking a risk on some students who would otherwise not be admitted but show some promise, including throttling-down the acceptance of over-represented groups. This is a fact. "predictive performant instruments" sure sounds very rigorous and objective, but you're talking about young people who are undergoing a lot of change. They're moving targets and predicting future performance based on one test only goes so far. A test like the SAT is always going to be a blunt instrument as is any singular factor. Admissions officers recognize this.
    - superuser2 11y ago
      >or can pay full tuition Can you name some "better schools" whose admissions are not need-blind? My understanding is that this is pretty much universal among prestigious schools.
      - crispyambulance 11y ago
jerf 11y ago
"What we should really be talking about is improving these tests and beefing up operational security." We have a real-world instance of the DRM problem here. Once you've put a static exam physically into the hands of the students, it's compromised. The attackers have massively, massively more resources than the defenders. The attackers can hire a hundred students to each take one picture of one page, and put the test back together themselves. And there's no way that the proctors can prevent students sneaking one picture of one page nowadays. (Also, I'm not saying that's an optimal attack. I just put about 5 seconds' thought into it. But it's a sufficient attack already, so the existence of even further clever attacks only makes my point stronger.) Even presenting the problems one at a time on a screen (massively more expensive, btw, due to brand new hardware requirements, plus introduction of other attack vectors) would not be sufficient. Clever games played with contrast or whatever won't work because the test needs to be highly accessible, so you can't play games like the old copy protection booklets [1], either. Basically, if the test is static, it's game over. Not that students aren't fairly good at cheating with dynamic questions. But those really only work well with the math questions, and even the dumbed-down calculators they still sell today can be convinced to run programs sufficient to "crack" any dynamic problems that the test companies can get their hands on for the math portions. It's basically hopeless. The adversaries have gotten too darned powerful. They're armed with too much gear that would make old-school spies weep tears of joy. [1]: http://i.imgur.com/eU8Rvhh.jpg http://i.imgur.com/eU8Rvhh.jpg
1337biz 11y ago
Might be true, still trying to measure the academic potential of a highschool student by a private company is absurd. I am all for participants doing their very best to "hack" the system. Kudos to those companies whelping them by take up the battle with the testing industry.

deleted 11y ago

[deleted]

VLM 11y ago

Wait for the higher ed crash. All things that can't go on forever, eventually stop. At that point the ACT/SAT problem will automatically solve itself.