24 ms·

Test scores are not irrelevant

• KevinBenSmith 4y ago
  Great & useful article. We should have more of these for “popular papers”.
  • jeffhwang 4y ago
    Agreed! Although, I admittedly didn’t read the post carefully enough to evaluate how accurate its analysis of the original paper. But all else being equal, we’d live in a better world if papers that get a lot of popular attention are subjected to this scrutiny with an eye towards explaining what works/ doesn’t work for a general audience
• ajwinn 4y ago
  I worked in Admissions at a large state school from 2005-2007. Our method for determining admissibility reminded me of the heat map chart in this article - ie GPA on one axis, SAT/ACT score on the another axis, trace your finger along both paths to see if the applicant was in or out. I probably reviewed 20-30 applications a day. If high school transcripts were universally formatted, decisions could have been instantaneous but alas, we live in the real world and some human-in-the-loop normalizing had to be done over-and-over. With all that reviewing, patterns emerged, namely that SAT and ACT scores strongly correlate to GPA. Now, I’m the kind of person that roots for the genius to overcome his grades and emerge a genius on the SAT/ACT. But in two years, it probably happened only twice. Before calculating a normalized GPA, I could look at the test scores and predict “admit” or “deny”. While the author is correct to say “the irrelevance of test scores is greatly exaggerated”, in my experience, whether or not something is irrelevant has very little to do with what universities do. I’d recommend only using test scores. Or, only go with GPAs. Only test scores is more efficient. Only GPAs looks better on press releases.
  • insane_dreamer 4y ago
    SAT scores seem quite unfair as a single indicator of performance. GPA at least provides a view of a much wider range over which performance may naturally fluctuate.
  • cammikebrown 4y ago
    I had a 3.18 GPA and a 31 ACT, is that outlying enough?
    • ajwinn 4y ago
      From a state school’s Admissions perspective, where the unit test resolves to admit/deny, there is nothing particularly outlying about your scores from what I recall. I see 31 and think, “admit”. I see 3.18, I think “probably admit”. I check your math scores (which is where a lot of people trip up), and see you got mostly Bs- heck, you’re on Hacker Mews, probably As, and I think, “admit”. Now I do the normalizing math and yeah, Admit. Congratulations!
      • wincy 4y ago
        Ooh okay okay try me. I had a 30 on my ACT my sophomore year then dropped out of high school my junior year. I was getting F’s my junior year because I couldn’t stay awake and had maybe like a 2.3 GPA before that. Would have reduced drastically if I’d stayed in school. It was only that high because some teachers were always super annoyed with me, and a lot of them would try to bend the system to give me a better grade but some just were like “homework is the grade sorry” even though I’d ace tests. Also my stepdad was doing meth so that caused issues. So am I an actual rarity?
        • ajwinn 4y ago
          You might be an outlier! But it sounds like you did not graduate high school or apply to college. If that’s correct, I would not have the context to know since the baseline of inclusion into my experience is people who graduate high school in some form - GED included, AND who provide test scores. I’m really sorry to hear about your stepdad, wincy. I hope things are ok for you now.
          • wincy 4y ago
            Things are great for me now. Thanks! I’ve got a wife and two kids and just got promoted to a SWE team lead. Mostly self taught. College just wasn’t for me.
  • TotoHorner 4y ago
    Well, I had pretty good SAT scores (2350+) and a very poor GPA (3.0/4.0). Main reason was that I just spent all my time pursuing ECs and didn't have enough time to spend on grades. Think you'll lose a lot of smart kids who spend time on ECs that way.
    • ghaff 4y ago
      That was probably more or less my college story for grad school. Very good GREs but lousy GPA in college. (And only somewhat less my story in grad school before going to the next grad school :-))
  • yunyu 4y ago
    The edge cases are more common at elite institutions, which are the typical admissions mania targets. At the top ~10 university I attended, there were plenty of people (myself included) falling in the poor HS GPA bucket (mostly due to extracurriculars) with perfect or near perfect ACT/SAT scores, among other outliers.
  • rootusrootus 4y ago
    I can see the value of the GPA. When I was applying to Oregon State way back in the day (this would have been early 90s), my SAT score was really great. Probably top 10 in my school. My grades, on the other hand, were sub-2.00. I may have had a problem keeping myself interested in school. But I found tests extremely easy. They rightfully told me to go to community college instead, because my GPA was abysmal. I'd have failed right out of OSU. I went to that community college and failed right out of it ;-). Then I went in the USAF, and maybe a year into that experience something clicked and suddenly I felt like an adult. I felt like my priorities realigned and I knew what I wanted and how to get there, and I could stick with it. So when I left the military, I went back to the community college, got perfect grades, convinced the admissions gal at OSU to ignore all the HS & college grades from a few years earlier, and let me in. Graduated with my bachelor's in CS with excellent grades. Ultimately went on to get my masters, though that was years later. Anyway, all of that to say ... my SAT score would have said "admit him" but my GPA was a more accurate assessment of my grit. I think both scores are useful, but don't give up on GPA.
    • TimPC 4y ago
      GPA is problematic because not all high schools have access to the elite courses that push a GPA beyond 4.0 and many top schools look for an admission barrier around 4.5 (some with exceptions for valedictorians only).
      • empyrrhicist 4y ago
        Also, grade inflation is real and not uniformly distributed.
      • solveit 4y ago
        This points to the real problem with GPA. The institution assigning GPAs to students have a strong incentive to inflate GPAs. If colleges only looked at GPAs, high schools would inevitably give in to the temptation to give away perfect scores like candy, and students from schools retaining some shred of dignity would suffer for it. Standardized exams, for all their faults, are standardized, and don't have this problem.
  • mgkimsal 4y ago
    I think my HS GPA was... 2.9 or perhaps 3 (out of 4), yet I had a 29 on the ACT. The ACT score was in the 80s before some scoring model change. It's probably changed multiple times since then, but I'd learned there was a model change in 1990(?) which makes comparisons between them a bit less useful. My siblings had slightly lower scores years later, but because the model changed, there's not a lot of comparison we could make.
    • deleted 4y ago
      [deleted]
  • balderdash 4y ago
    Out of curiosity how did you normalize for say 3.0 GPA at a competitive admissions magnet school, and a 3.0 GPA at an underperforming high school? Did you correlate pervious students from that high school and their high school gpa and their graduating gpa?
    • pirocks 4y ago
      I have long wondered about this. Note that you also need to be able to normalize grades across students who went to the same school but had different teachers. I suspect the answer is that both normalizations are hard so admissions people don't bother.
      • solveit 4y ago
        They don't not bother. They just do it by gut feel without writing anything down about how they're doing it. Admissions counselors at elite high schools "network" with admissions offices at universities to benefit their students in this process. This is common knowledge in the industry to the point that foreign elite high schools, which have a harder time getting name-recognition, spend a pretty sum of money periodically sending their representatives on month-long US trips to go around, say, the top 30 US universities to market their brand.
        • bena 4y ago
          This is true. I took the ACT on a lark my sophomore year of high school. I scored a 29. I also attended one of the top high schools in the nation. When I was registering for classes in college, my advisor kept trying to put me in Trigonometry or Pre-calculus (I can't remember). Because my ACT score in math said that's what I needed. I took the ACT while I was taking Geometry and Algebra II, so yeah, my math was probably weak. I tried explaining to them that my high school transcript shows I've taken up to Calculus. They wouldn't hear it. I refused to register for any sort of math below Calculus. Eventually, they told me to go talk to the head of the math department. I told them fine. I went down there all prepared to make my case. As I started to make my case, he saw my transcript, said "Oh, <SCHOOL NAME REDACTED>. What class do you want to take?" He didn't care about my transcript in the opposite way. I probably could have registered for 300 or 400 level math courses.
  • twblalock 4y ago
    Entrance exams are a common option in other countries, and they are a good way for students who are late bloomers to get into top schools.
  • hedora 4y ago
    According to the University of California’s own internal study, there is a small group of students with terrible GPAs and good test scores. They tend to do well in college. There used to be a set percentage of spots for students in that situation, but the admissions people decided that “Standardized tests are racist!” is a good twitter sound byte and eliminated the spots. Of course, their own reports said that the “bad GPA, high test score” cohort were mostly from underprivileged areas (and disproportionately black or hispanic). That nuance didn’t make the 140 character cutoff, so that group is officially abandoned by UC now. In other news, they’re now promoting a big push to make the student body composition match the general population. However, there is only one over-represented ethnic group in the UC system. Guess which one? Hint 1: Whites are under-represented. Hint 2: They swore they’d never bring back UC’s old “asian quota”. I honestly think the people running this state’s school system are malicious racists. Their results speak for themselves: California schools are currently ranked 43rd in the nation. They used to be top ten.
    • lokar 4y ago
      You jump from the UC to primary/secondary schools. They are not run by the same people. Most of the decline is due to underfunding due to prop 13
      • WillPostForFood 4y ago
        California spends about twice as much per student today (inflation adjusted) as it did before Prop 13 passed, so it doesn't track that Prop 13, or underfunding, is the cause of the poor performance.
        • JamesianP 4y ago
          All the 200+ student classes would certainly hurt them a lot though since that's a big chunk of some rankings.
          • WillPostForFood 4y ago
  • scarface74 4y ago
    I just posted a comment about my step son who had stellar test scores after we spent $1200+ on a private tutor. Did he get more qualified for college than the kids who had parents who couldn’t afford private tutors?
    • ipaddr 4y ago
      Yes he is more ready to be a clog in the machine. He passed the first test. He studied enough leetcode and it doesn't matter how he got there. Will the tutor be there in college and will he be able to absorb all the information college gives are bigger questions.
      • scarface74 4y ago
        Doesn’t that argue that neither college test or the standard tech interviews are effective.
    • ajwinn 4y ago
      Excellent question. I’m of two minds here. As a high school student, I too struggled to get great test scores. I took the SAT four times, which finally resulted in a score that could get me scholarships. Did I become smarter? - I sincerely doubt it. Did I become less likely to fail out of college? …Seems unlikely, but I don’t know. As an admissions rep, I understood that high scores, whether SAT or GPA, are certainly both proxies for a prediction: If I admit this student, are they going to fail out? Unknown: It actually might be the case that students who get private tutoring are less likely to fail out because their parents can financially endure them struggling to repeat failed courses. I’d be interested to see a study on that.
      • hackinthebochs 4y ago
        Did your final score more accurately reflect your abilities? Probably. Can everyone increase their score by hundreds of points by repeated testing? Probably not.
        • 4y ago
  • club_tropical 4y ago
    Was there a secondary adjustment step to finalize admit/deny? Your heatmap-based method would almost certainly result in an unacceptable racial composition of the class.
    • ajwinn 4y ago
      I worked in the underrepresented student office, so I was keenly aware/curious about that possibility when I began. However, after reviewing thousands of applications, my experience, to my surprise, was that GPA to SAT/ACT scores remained highly correlated regardless of race. If memory serves (and it’s been awhile) the outliers who were more likely to have higher SAT/ACT than their GPA normally suggested, tended to come from predominantly non-white schools. I’ve seen so many reports to the contrary that this surprised me to no end, but that’s what I observed.
      • club_tropical 4y ago
        I see your point on the GPA/test correlation and that is good in that it reflects that grade inflation wasn’t a major problem while you worked there. But my question about final admit/deny stands. If you look at the racial distribution of the upper performers of any standardized exam (your “heatmap tracing”), it is unlikely to me that a college would be fine with that class composition. Either there must be a secondary layer where your admit/deny’s are adjusted or your acceptance rate must be very high.
  • dghlsakjg 4y ago
    I'm your outlier! My ACT was 34 and I can't recall my SAT score but it was also 99th percentile. My GPA on the other hand was abysmal. I can't recall, but I wouldn't be surprised if it was well below 2.5. Once I actually made it to college I was a pretty good student, though. Graduated with honors in 3.5 years.
  • vamsipk 4y ago
    Sure, we can use only GPAs as long as all the applicants are graded on the same common platform. But today the GPA from one college is not exactly equivalent to the GPA from another one. The grading system as I heard is also not standard across US (leave alone international) with some instructors using absolute while others relative grading. So an A student from college X is not necessarily as good as a B student from college Y.
  • Tade0 4y ago
    > With all that reviewing, patterns emerged, namely that SAT and ACT scores strongly correlate to GPA. My school experience was that it's easy to cheat during the year and much harder to do so in exams, so often people with generally good grades would do at best average at the end. I always had terrible grades(except for math), but they're not part of college admission scores in my country so I had no incentive to do anything about it.
  • evrydayhustling 4y ago
    Yes, nobody is dropping test scores because they are irrelevant -- although that overreach does exist in the press. They are dropping them because they add little after GPA is considered and may introduce costs, barriers and biases. The author identifies the steel man version early on: "standardized test scores are nearly useless (at least after you know GPAs)" but then mostly ignores that parenthetical. This part that addresses it seems deliberately obtuse: > If you care about the difference in accuracy between GPA and ACT, then you should care even more about the difference between (GPA only) and (GPA plus ACT). It’s incoherent to simultaneously claim that the GPA is better than the ACT and also that the ACT doesn’t add value. Again, the version of this that reasonable people argue isn't that the ACT doesn't add any kind of value, it's that the amount and kind of value it adds (incrementally more predictive accuracy) aren't worth the costs and biases are introduced. If you myopically value accuracy, it doesn't make folks who want to balance accuracy with other goals incoherent.
  • bena 4y ago
    I guess test scores are more efficient because GPAs are somewhat subjective and can be gamed to a degree. By both the school and the student. Two schools with two different grading tolerances can produce students with similar GPAs but different SAT/ACT scores. As the SAT/ACT is a national standard, not a regional one.
  • anthlax 4y ago
    What do you think about GPAs being non-standard? Test scores calibrate between schools - GPAs only kind of do. I remember many classes that were taught by two separate teachers: one teacher consistently gave students higher grades. The other teachers students consistently did better on the AP test - much more strict with homework and test scores, but it ended up positively impacting the students.
    • lupire 4y ago
      Why did't the other teachers correct their grades at the end of the term?
  • asimeqi 4y ago
    Short term one get can rid of test scores or GPA since they indeed seem to be correlated. But long term: 1. If we get rid of test scores, it would cause too much upward pressure on GPA which would make GPA less reliable. There are tens of thousands of high schools in the US. Even if the GPA distribution inside every high school is correct, how do you compare with GPAs from other high schools? And from experience I can predict that, once people realize that only the GPA is important, there will be outside influences (i.e. pay the teachers for good grades) that could cause even the distribution of GPA inside one school to not be correct. 2. If we get rid of GPA, students could just study for the test with obvious disastrous effects for their education. We need both GPA and test scores to keep both of them in check.
  • commandlinefan 4y ago
    > to overcome his grades and emerge a genius on the SAT/ACT Isn't it usually the opposite? Artificially high GPA from grade inflation and favoritism is "set straight" by a standardized test?
  • curiousllama 4y ago
    To be fair, some state schools do operate on just GPA. I recall my HS had that GPA/SAT scatterplot available for past applicants: UVA had a flat cutoff at ~4.1 weighted GPA, and the SAT literally didn't matter at all.
  • 2devnull 4y ago
    Grades and gpas are based on tests too, but on crappy tests without any psychometric validity, often correlated with race and broken in a bunch of ways which standardized tests are not. How many high school math teachers run irt or dif analysis on their tests? Meanwhile, standardized testing companies spend a lot of money and energy trying to remove race from their tests. It is as though smart people have been working really hard on this problem for many decades and found an optimal solution. The solution is standardized tests.
  • d0liver 4y ago
    My GPA was bottom 10ish at my high school. I graduated high school essentially because I already had most of the credits I needed to graduate from middle school. I got into college based on test scores alone (and I was surprised that this was possible). My GPA improved a bit early on in college and then improved much more around junior year once I started getting into harder classes. I ended up graduating with a math + CS double major in 4.5 years. So, sometimes people are just bored. And there are a bunch of other factors here too - perhaps the biggest one being that "graduated within six years" is not a great indicator of success. Linus Torvalds did not graduate within six years but did build the first version of Linux at university. GPA is a good indicator of how much work you're willing to put into what you're learning _now_, so it tracks well as a predictor of how much you care about GPA which is a pretty good predictor of whether you'll graduate in X years and probably not much else.
  • Jerrrry 4y ago
    28 on the ACT, no studying. 1.4 GPA. I was eligible for the maximum academic grants for my state, just to be denied due to GPA because the string of high schools I went to were utter dystopian nightmares. this tracks.
    • nemo44x 4y ago
      Another problem with relying on GPA is it is easily gamed. Parents and students can pressure teachers and administrations since so much is reliant on it. And a 4.0 at one school may not be the same at another school. I understand some people "test poorly" but the truth is, if you actually know the material well, you'll test just fine in most cases. We can also coach students on how to approach testing. Other ideas are to break the test into various sessions with different admins to limit cheating potential and to take an average of the scores (or some other method) for each subject. I guess my point is - make test results the metric to remove all bias. Change how we approach testing and preparation.
      • sokoloff 4y ago
        It doesn't remove all bias. It does remove a lot of bias and concentrates the remaining bias in a particular dimension, which probably does substantially improve the fairness of the system, but we need to be clear-eyed that it's not eradicating bias.
    • matai_kolila 4y ago
      Your refusal to take responsibility for your part in that 1.4 GPA also tracks, heh...
      • Jerrrry 4y ago
        of course, but that just illustrates that GPA is less about actual academic achievement, and more about playing a revolving game of teacher's pet at best, and at worst, being another tortured cog in a horrible system of petty social games called "high school" giving any credence to GPA is upholding this hellish set of norms. K-12 should be K-9. High school is glorified babysitting, with bullying and drugs being the only escape, for both parents, teachers, and students.
  • garyrob 4y ago
    My high-school GPA was mediocre because I simply was unable to pay attention in class, or to study at home, or to turn in homework that wasn't dinged over and over for "careless errors". I was born in 1956. Given today's knowledge of things like ADHD-Inattentive, I'm pretty confident that I would have been diagnosed. Some drug would probably have been prescribed, and it probably would have helped my grades (although I'm not convinced it wouldn't have hurt me in other ways). But for some reason, when I took my SATs, I found it to be fun. I got completely immersed for the period it took to take the test, and truly gave it all I had, with real focus. I got the best SAT scores in my school, which was shocking to some people. And it turned out that that was far more predictive of my future, including my college GPA performance, than my high school grades were.
  • tristor 4y ago
    Interestingly, I was one of those "geniuses" that had a perfect ACT score but a horrible GPA (long story). I got rejected by every college I applied to except for the local state school, so that's where I ended up going. In the end I dropped out in my final semester and just went to work, but that's besides the point. I was more than qualified to academically excel at any university I wanted to attend, and I maintained a 3.9 GPA in college up until I dropped out. The problem with using just GPA is that GPA can easily be manipulated by school authorities who dislike you, test scores cannot.
• xbar 4y ago
  My favorite point: "2. It might be that test scores don’t predict college graduation rates[...] is provably false."
• WBrentWilliams 4y ago
  Cute. Disclosure, I make the data (transcripts, test scores) flow to decision makers as part of my day job at a Big 10 University. We just went "test optional". I view "test optional" as a signaling exercise. In brief, how closely can you read the instructions and infer what you are not being told? The key is "test optional". If you sit for the test, you get more than just admissions, as merit-based financial awards are unlocked. The University Admissions does not require a test, but many of the donor contracts that fund scholarships do require those tests.
  • felideon 4y ago
    Interesting, and good to know! But wouldn't you say that students are better off not submitting their scores if they scored low on the tests? Only given a sufficiently high enough score does it become a "nothing to lose" situation. Unless the issue is students just don't want to take the test to begin with.
    • WBrentWilliams 4y ago
      I'm saying that you can be admitted without test scores. You leave money on the table when negotiating the bill when you do not take the test. Opting not to take the test could be an optimum strategy if you already know that the offered awards will not apply to you.
  • tqi 4y ago
    > I view "test optional" as a signaling exercise. In brief, how closely can you read the instructions and infer what you are not being told? Sorry, what does that mean - who is the "you" in that sentence, and what are they inferring? It sounds like what you mean is that that your school went test optional (ostensibly to help level the playing field), but those who are lucky or privileged enough to have the secret decoder ring can decipher the instructions and find the key to optimizing their chances of admission. If so (as someone who grew up with immigrant parents who were educated but definitely would not have had the awareness to navigate this), that seems really fucked up...
    • 4y ago
• olliej 4y ago
  I think there’s an issue on how these data are examined. There is a huge amount of data backing up the claim that the school attended influences the SAT, etc. There is a huge amount of evidence backing up the claim that SAT/ACT aren’t 100% predictive of later academic success. But “not being 100% predictive” does not mean “is not predictive”. The core problem the removal of SAT, etc is attempting (imo hamfistedly) to mitigate is the following chain: (1) Student A comes from affluent family, in an affluent region. (2) Student A goes to a private school with better teachers, etc, or because it’s an affluent region the public schools have better teachers, etc (3) Student A’s family can afford additional SAT, etc tutors Now imagine a somehow identical Student B, but from a poor family in a poor region. Before any other factors become involved, they don’t get the benefits of 2 or 3. We would expect that on average 2&3 result in higher scores, and so a higher acceptance rate, and a higher rate of scholarships for Student A vs Student B. Again this is on average. The TLDR is given two kids who have identical mental ability, the one from the richer family will, on average, have better scores. So schools say let’s drop the scores entirely, but that is also clearly nonsense: Even in this exact scenario Student A will do better in uni or what have you, because part of why they got a better score is that they have been taught and learned more - IE they absolutely do know more and are more academically proficient than Student B by this point. Moreover we know (because of yet more studies), Student A will tend to continue to do better than Student B once they’re at school, even once B “catches up”, because we know that people with fewer external stressors perform better at everything. So even if we discard the score based entry to unis, we’ll see a strong correlation between original score and subsequent academic success. So while I understand the desire to remove the “accident of birth” factor that benefits the lucky and penalizes the unlucky, simply removing the test requirements is fairly questionable as an approach. This is also before asking about Students C and D who are the peers of students A and B - surely the better scoring of A and C (or B and D) is who you should prefer, but now you’re discarding theoretically ideal and unbiased data in favour of…?
• steve76 4y ago
• faeriechangling 4y ago
  Test scores are accurate at predicting your ability to succeed at further testing, I don't know if this is all that controversial. What people are getting up in arms about is the validity of testing as a whole regarding real life outcomes, and this is an effective argument as it is damn hard to grade people by any metric that doesn't get gamified to death and resulting in people wasting a ton of time on playing the game.
  • nickelpro 4y ago
    A student who scores highly on a Calculus exam may be generally good at Calculus, or maybe they know a couple exam-specific algorithms to succeed and got a little lucky besides. A student who fails a Calculus exam is nearly guaranteed to be poor at Calculus. To say exams have no predictive power is to ignore the obvious.
    • thethethethe 4y ago
      Student A that passes a Calculus exam may be be more likely to know Calculus than student B that fails but so what? Passing a Calculus test doesn't necessarily predict if the student can apply their knowledge to production to generate new ideas and widgets which make our lives better, it just predicts that Student A can follow rules and learn something _when required by an authority_. I think this sentiment is similar to how people feel about Leet Code interview questions. So what if an engineer can invert a binary tree if they can't formulate new, useful ideas and convince others to work on them? Perhaps student A is more likely be create new, better widgets than student B but I think academicia/LC cannot test this directly, which is what people are getting at when they criticize tests.
      • AnimalMuppet 4y ago
        Again, though, if creating new, better widgets requires calculus, student A may not have what it takes, but student B almost certainly can't do it. Some jobs require a creativity, a spark, a something that you can't test for. But they also require some solid skills, and those skills are testable.
• rayiner 4y ago
  The implication of the “test scores don’t matter” argument is that we have the wrong people staffing virtually every credential-based position. There’s lots of kids with 4.0 GPAs from high school at directional state university getting education degrees. Test scores is what distinguishes those kids from the ones at Stanford. And Stanford is where everyone from Big Tech to Big Banks to Big Law to Big Consulting goes to hire. Who is working at Pfizer or Astra Zeneca doing drug development? It’s the people with high test scores. Every single Supreme Court justice in the last half century got there based on test scores. (The nominee whose test scores led her to a law school ranked outside the top 50, Harriet Miers, has to withdraw because of that.) Warren Buffett got into Wharton based on test scores. Charlie Munger got into Caltech based on his scores on the army’s intelligence test. Bill Gates got an 1590 on the SAT. Even the professors saying test scores don’t matter got where they are because of test scores. Maybe that’s all true. But it’s quite a remarkable claim that would turn the world as we know it upside down.
  • karpierz 4y ago
    Let's steelman that a bit. If you structured society such that only the top 0.5% of households by income could send their kids to college, you'd find that a lot of your successful entrepreneurs, judges, scientists, etc. went to college and had a high income growing up. But I don't know that proposing something else, like standardized testing, as an alternative to this would be quite a remarkable claim. You're just saying that the scoring function produces a lot of false negatives.
    • rayiner 4y ago
      In your hypothetical, institutions would quickly realize that a bunch of non-college graduates were just as smart as college graduates. Which is actually the public perception in places where jobs are filled based on social status and connections rather than objective metrics. What I’m saying that the current scoring function appears to match up pretty well with our intuition about who is smart enough to work on Wall Street or at Facebook. At least, these profit-seeking institutions don’t seem to think it would be better for them to shift their hiring from Stanford to kids with 4.1s and lower SATs who went to UCSB instead.
• jklm 4y ago
  Test scores have already been irrelevant for certain segments for over a decade, speaking as someone of East Asian ethnicity. Personal anecdote: 3.7 GPA in average high school with 8 APs (some of which were self-study), 2400 SAT, 36 ACT, lots of state-level top 3 finishes in multiple academic competitions, varsity track, musical instruments, first generation immigrant, etc. Ended up going to what I thought was my safety school and rejected/waitlisted at every reach school I applied to. I sometimes wish colleges mandated name/ethnicity-blind application reviews - not to sound ungrateful, but I’m still incredulous to this day that the best I could do was a public state school (still top 50 admittedly). It ended up being a great experience in that it was a forcing function and made me realize most of these rules and expectations around admissions were meaningless. But I probably could have saved years in high school exploring things that mattered rather than optimizing for a college application process that didn’t ultimately end up feeling very fair.
  • testfoobar 4y ago
    Glad it worked out for you. Asian college applicants have it rough these days. This is the President of Stanford speaking out of both sides of his mouth: 2022-10-12: "Stanford apologizes for admissions limits on Jewish students in the 1950s and pledges action on steps to enhance Jewish life on campus". Stanford President Marc Tessier-Lavigne apologized on behalf of the university and pledged action on recommendations in a task force report confirming Stanford limited the admission of Jewish students in the 1950s. https://news.stanford.edu/report/2022/10/12/task-force-report-jewish-admissions-and-jewish-life/ https://news.stanford.edu/report/2022/10/12/task-force-repor... 2022-08-02: Amicus brief filed in support of Harvard and University of North Carolina “A diverse student body enriches the academic experience for all Stanford students,” said Stanford President Marc Tessier-Lavigne. “Considering race as one part of holistic review helps us foster a diverse campus community, one in which all students have the opportunity to learn from each other’s experiences and to think critically about their own views and preconceptions. It also means that Stanford graduates from all backgrounds can go on to bring their unique insights to leadership roles in the government, business, and nonprofit sectors.” https://news.stanford.edu/report/2022/08/02/amicus-brief-filed-support-harvard-university-north-carolina/ https://news.stanford.edu/report/2022/08/02/amicus-brief-fil... Guess which race/ethnicity Stanford is limiting now?
• scarface74 4y ago
  Let me tell two stories: I graduated with the highest SAT scores in my school and the second highest in my city back in the early 90s. I’m not bragging, it was a relatively small town in the south with not great school systems and a total graduating class of 1500 across all of the schools. What are the chances that happen because I was so smart or because I had a mother who was not only a high school math teacher but also spent years volunteering to teach SAT prep classes? Second story: my step son didn’t have the ACT scores to get into the college he wanted to go to. Four months later and with the help of a private tutor who was a teacher (not in his school) that we paid $100/hour for 12 hours, he suddenly raised his scores more than enough. Did he gain aptitude in 12 weeks or did he learn test taking techniques?
  • rajin444 4y ago
    Don’t you think a mother able to teach those classes would be in general a better test taker than the average person, and so her children likely would be too? And nobody is denying that test scores can be raised. That’s likely a fixed amount so just amounts to constant noise. I think you’re as likely to find a way to consistently raise a 600 SAT to 1600 as you are trying to train any couch potato into an Olympic sprinter.
    • scarface74 4y ago
      So let’s say my mom had been a “good test taker”. But she didn’t spend anytime with me. Was always too busy to help me when I had a question, etc. Do you think I would have done as well? What are the chances that I would have chosen to be software developer coming from that same small town if my parents hadn’t bought me a fully tricked out Apple //e in 1986? Even when I was in college in 1992, most of the CS students didn’t have a computer home. By then I had a $4000 LCII set up with a $1000 laser printer and a $300 SoftPC application to run x86 compilers. My parents weren’t “rich”. My mom was a teacher and my dad was a factory worker.
  • hcurtiss 4y ago
    Yeah, I hear this, but in 1995 I just bought an SAT prep book for $12 and spent a couple months of Saturdays practicing and did very well. My family lived well below the poverty line. SAT prep is not confined to the privileged.
• faangiq 4y ago
  I’d gladly hire on test scores alone.
• ramraj07 4y ago
  So one of the best standardized tests I’ve ever taken is the Subject GRE. When I was applying for PhD programmes I and a few of my colleagues took it. It was perfect - it tested both knowledge and critical thinking skills and experimental design. It was obscure so no material was available for specific training. In the end the scores were so accurate in predicting eventual success - a decade later, the folks who got 96% plus are all either professors or deliberately chose not to, while the rest just took industry jobs (and in my opinion because they realized that’s best for them). Here’s the kicker - every institution explicitly said they will NOT consider these scores as part of admission process. None of the people who got in the top percentiles (4 I know) made it to a top 10 institution in the US, while a bunch of others did because they had paper authorship in their undergrad. But paper authorship in undergrad has NO correlation with your actual scientific skills! It just meant you were connected and or a hustler. It’s sad that such an awesome test is deliberately ignored by these institutions. The irony was that these tests were also not elitist. All you needed to do was thoroughly read Lodish and Lehninger and you’re good. We did study in a semi-premium institution in India, but by no means were we privileged by any special facilities or help (at least in the context of this test). The only barrier might be the test fee itself if anything.
  • thomasahle 4y ago
    > a decade later, the folks who got 96% plus are all either professors or deliberately chose not to, > All you needed to do was thoroughly read Lodish and Lehninger and you’re good. How can these be true at the same time? If you can just read a book and get a good score, why don't everyone just get the good score?
    • hawk_ 4y ago
      It's clear they mean that the barrier to entry was available to anyone. That doesn't mean that anyone has the capacity to understand and apply the contents of the text.
    • ramraj07 4y ago
      Lodish (a then popular cell biology book) and Lehninger (a 1500 page biochem tome) are substantial books. 101 courses would cover the first 5-9 chapters of 25. Reading back to back and understanding all of it puts you at the top of the best students in biology in the world. I was lucky, I genuinely loved reading them. I think I went through them both once every time I had a bad breakup lol.
• rajin444 4y ago
  Does the same hold true of medical professionals? ie MCAT scores correlate with med school and/or practice performance?
  • derbOac 4y ago
    It's been some time since I read this work, but at one point there was a situation where medical schools in a state were temporarily (like a year or two) prevented from using MCATs in admission decisions due to a judge's court order. Most applicants still had MCATs due to other schools, though. Some researchers collected data before, during, and after this time, and followed-up on the physicians later. My fuzzy memory (which might be incorrect) is that MCATs were predictive of GPA in the first two years of medical school like 0.20 on a Pearson correlation metric, and it dropped from there as you went out increasingly further in time and included things like ratings on clinical rotations. So it was predictive, but fairly weakly so and seemed less predictive as time went forward.
• cr4nberry 4y ago
  Why is it that the anti test score argument almost always tries to chalk it up to hiring tutors? They consistently try to make it sound like people "buy" higher test scores with tutoring in order to discredit tests like the SAT
  • pocketsand 4y ago
    There is research on this. I believe it was Thomas Espenshade, social scientist at Princeton, who had data on test prep access, modeled along with SES, etc. Test prep is worth about 40 points on the SAT. Not trivial, but not a game changer. It's true that people who are otherwise well off are more likely to have test prep, and those people's test scores are often several standard deviations higher than low-income students from bad schools without test prep. But the counterfactual of (same person) +/- test prep = about 40 SAT points.
• readthenotes1 4y ago
  Did I miss the correlation across school systems? Everything in the study was from Chicago public schools. My older brother switched from a B student in a private high school to the public high school and slept his ways to A's
• mradek 4y ago
  I feel like standardized tests are way more fair than GPA. I personally struggled a lot in school due to health and other issues. I also hated doing homework which is BS work, and I always had high scores on my midterms and finals which I think matter way more. Why does busywork play such a big role? It should be exempted. If I can get a 95+ on midterm and final I obviously understand the material. Why did I need to spend hours of my day doing BS exercises when I already understood the material? Nonsense. So I get a B at best overall because I get a F on homework? What a joke. I really think school is a waste of time when it’s done that way. Learn stuff, move on. I wish that when I have kids I can afford to send them somewhere they can actually grow and thrive, not just sit down and do a bunch of homework problems. (Just my $0.02)
  • Aliabid94 4y ago
    Most real world work is grinding through boring work as well, and it’s worthwhile to build the discipline for it.
    • TheHegemon 4y ago
      Completely agree with this. I used to skip most of my classes but would get As on all of my tests. Now I struggle immensely on going through the every day grind of adult life.
      • anthlax 4y ago
        Correlation or causation? Perhaps if you didn’t skip classes, you’d still struggle through adult life but also struggle through childhood. Perhaps you’d just be more numb.
    • mikebenfield 4y ago
      There are better ways to build discipline than spending a huge chunk of ages 5-18 doing otherwise pointless busy work.
      • barry-cotter 4y ago
• godelski 4y ago
  I think the author is missing some really key points here. Namely that GPA and test scores are highly correlated. I'd go so far as to say that there is a confounding variable that predicts both. I'd even say there are several upstream predictors, but I think we all know that (or I thought we did). Every single graph they show demonstrates this yet they keep missing it. The top comment right now (by ajwinn[0]) notes that they used a heatmap like the one in the article for determining admission. This is great because it builds multiple pathways to gain admission. That's honestly how we should treat variables like these: rely on either. As to explain the confusion in the models, well when you have confounding variables like this it is easy to reduce one variable's weight nearly to zero and place the rest in the other variable. So if I'm right (might not be) I think you could also show that the inverse is true, that grades aren't a good predictor and test scores are. The breakdown suggests that this is true too tbh. But I also disagree with some of the analysis and techniques here. It's not always easy to use models that have continuous(ish) variables (GPA/ACT/income) and discrete ones (race/gender). I bring this up because there's some hints that they aren't a stats expert. For example there's the line > The “likely-to-graduate” score becomes a probability after a sigmoid transformation. Which just isn't true. A Softmax function[1] creates a probability distribution but a Sigmoid[2] doesn't. A Sigmoid bounds the output to [0, 1] (or [-1,1]) but being bound on that range isn't the only requirement to a probability distribution. A sigmoid does not guarantee that the sum of the elements is 1 (cumulative density), but a softmax does. In fact, it almost never will. (I mention this in my ML class a lot so make sure you all don't make this mistake too! I'll be really annoying if you do) There's some more errors and novice approaches here and so I just want to note this because we shouldn't just take the results and conclusions as is when the modeling is bad. But I still think he makes some good (and bad) points and those are definitely up for discussion. [0] https://news.ycombinator.com/item?id=33338809 https://news.ycombinator.com/item?id=33338809 [1] https://en.wikipedia.org/wiki/Softmax_function https://en.wikipedia.org/wiki/Softmax_function [2] https://en.wikipedia.org/wiki/Sigmoid_function https://en.wikipedia.org/wiki/Sigmoid_function
• MarcoSanto 4y ago
  It'a sunken cost effect at paly, little more than that. Oh, and parents pressure, least we forget the role of family. You can test my claim empirically by asking yourself: "are all college graduates I met more intelligent than me?". Admission test measure how well you can prepare for admission tests, nothing more. There are too many confounding factors and co-factors that play in the career of a relatively fresh semi-adult aged 17-18, that reducing it to a single number is just moronic or bad-faith utilitaristic escape route. Universities only care that you graduate so they can make money, therefore they use the simplest KPI to measure that. Let's not forget Universities are staffed with people and are a business.
  • 411111111111111 4y ago
    It's unlikely to be a sunken cost fallacy. You're technically right with "Admission test measure how well you can prepare for admission tests, nothing more.". It's definitely not a silver bullet and doesn't assure success after admission. nonetheless, the ability to prepare for these tests is strongly correlated with success along this path. If they can't be arsed to prepare for them they're unlikely to bother with studying either, making their admissions pointless too. And if they attempted to prepare for the test but still failed... Then they're unlikely to be able to learn in the University setting either and should follow up on other avenues.
• tagami 4y ago
  “C’s get degrees”. Measuring college graduation rate is a vanity metric
• Ekaros 4y ago
  Test scores are the most objective metric available. And thus I really consider them fairest. Or we could just have entirely random lottery.
• vivegi 4y ago
  This whole concept of measuring SAT/ACT scores or GPAs and using them to "predict" graduation rates is bonkers. Sounds a lot like the precogs in Philip K. Dick's Minority Report. Several decades ago, I had a high-school History teacher. We had an elite private high-school in the same city as I lived (India has both private and public schools). They had an extremely rigorous selection process, including infamous interviews with parents for admission to kindergarten. Our history teacher once remarked, "They take horses and send them out as horses. We take horses, asses and everything else and send them out as horses". The SAT/ACT/GPA filter seems to be focused on selecting horses and proudly declaring all horses graduated.
  • club_tropical 4y ago
    > They take horses and send them out as horses. Correct. Although there are huge benefits of a gate-kept horse-only environment. Competing against each other, the horses become more excellent. This is why the parents are paying the big bucks and putting up with the process. It is about gaining access to a horse-only environment. > We take horses, asses and everything else and send them out as horses. This is just envy at the easier job of the private school teachers, not indicative of their actual transformational capabilities of the public school teacher. The best that these teachers can do (which is no small feat) is to manage the classroom such that the donkeys and the horses do not fight and the horses can stay motivated despite having fewer peers.
• dirtyid 4y ago
  If anything over indexing on solely on test scores will filter out a large % of psycho/socialpaths who gets by on personality scores.
• sakex 4y ago
  I think that when people say test scores are irrelevant, they don't mean irrelevant for studies but irrelevant as an indicator of job performance.
  • barry-cotter 4y ago
    > Meta-analyzing the relationship between grades and job performance. Roth, P. L., BeVier, C. A., Switzer, F. S. III, & Schippmann, J. S. (1996). Meta-analyzing the relationship between grades and job performance. Journal of Applied Psychology, 81(5), 548–556. https://doi.org/10.1037/0021-9010.81.5.548 https://doi.org/10.1037/0021-9010.81.5.548 Employers and academics have differing views on the value of grades for predicting job performance. Employers often believe grades are useful predictors, and they make hiring decisions that are based on them. Many academics believe that grades have little predictive validity. Past meta-analyses of the grades–performance relationship have suffered either from small sample sizes or the inability to correct observed correlations for research artifacts. This study demonstrated the observed correlation between grades and job performance was .16. Correction for research artifacts increased the correlation to the .30s. Several factors were found to moderate the relationship. The most powerful factors were the year of research publication and the time between graduation and performance measurement.
• qwerty456127 4y ago
  In my opinion discriminating against people given their past test scores is unfair. People can improve and having scored bad (which might well be a consequence of hard life circumstances at a specific moment of their life) in the past should not be allowed to ruin their future. So I believe nobody should be allowed to see one's previous test scores and judge them this way. Whenever it is legitimately necessary to assess a person's proficiency a new test should be taken.
• PaulHoule 4y ago
  My take is that standardized tests are a road to social mobility. People can bullshit their way to higher grades but there are some rich kids who can't test their way out of a paper bag and the one way we can stop them from laundering their parents accomplishments is standardized testing.
  • athorax 4y ago
    How does that stop them in any capacity from living off their parent's wealth?
    • ubercore 4y ago
      I don't think that's the point the parent post is making
  • ouid 4y ago
    Well, no, because schools have a profound interest in having those students parents pay them for a top notch education.
  • derbOac 4y ago
    Except you can buy your way into tutoring too, and pay for test accommodations, and so forth and so on. It might not amount to a huge effect, but if you're on the boundary area, it can make the difference. Someone might also argue that although bullshitting with grades makes a difference, it usually isn't going to turn a D into an A either (speaking as a university professor, with many colleagues at different universities, in those situations it's usually a matter of being pressured to turn an F into a D).
    • trentnix 4y ago
      Except you can buy your way into tutoring too, and pay for test accommodations, and so forth and so on. I don't buy this. It's like saying someone has a remarkable physique or remarkable fitness only because they used PEDs when it actually requires a significant amount of effort and discipline regardless of what's in the gas. Certainly targeted studying, tutoring, test accommodations, etc. can help. But the difference is at the margins.
• forgotmypw17 4y ago
  It is not that they are irrelevant, it is that they are ableist, selecting for a narrow skill and ability combination, and creating a barrier for anyone else who lacks it, for whatever reason.
• pwpw 4y ago
  I see zero discussion of test scores and people with learning disabilities. Those who cannot be properly accommodated will be unfairly represented and often have to put up with a lot more barriers (such as frequently messed up accommodations on test day leading to anxiety). Usually these accommodations help normalize the specific testing scenario and people with disabilities do not need as drastic accommodations in real job setups. The tests are designed for the average test taker but have lots of hiccups for those that deviate from the average. It’s a difficult problem to solve, but it’s unfair if a student gets easily distracted or anxious when taking a test around others in a time constrained setting. Those types of external factors generally shouldn’t be represented in a student’s test results yet often are. If tests are to be a good indicator for all students, I think accommodations have a ways to go. If tests are indeed a generally good indicator of a student’s success and are continued to be used in such a manner, I feel it’s important that the experience is fair for all types of test takers. Many US tests such as the CPA exam do a great job of handling accommodations. However, many others (like the PSAT) have a lot of room for improvement.
  • gumby 4y ago
    > I see zero discussion of test scores and people with learning disabilities. Those who cannot be properly accommodated will be unfairly represented… I have long wondered how these interact / can support people who are a couple of sigma out from what we have decided is the “norm”. I can see how additional test time can help someone with anxiety or distraction issues, which can help someone with other strengths shine. But how can this work outside the academy? A trial lawyer has only so much time to work on the case before it goes to trial or before some response to a filing is required. An engineer building a rocket still has to get something designed before assembly. Arent such time accommodations are hard to implement in the “real world”? There are nice counter examples to my question. For example dyslexic people struggle with all sorts of cases, yet technology can help many of them (e.g. new kinds of fonts), and not just current students but people already in the work force. I can also see the school accommodation helping someone who is separately working on their anxiety issues — though mental health cost support is quite poor, at least in the US. But at their root: do these accommodations help or do they provide an unrealistic hope to the student? NB: I want everyone to have a fair shake, and am a fan of ADA accommodation and affirmative action and related diversity efforts. I ask this question within that context.
• 3vidence 4y ago
  Maybe a hot take but I think school pedigree is irrelevant. I went to a top university in my country after transferring in after not getting in from highschool. Turns out myself and many others would have got a lot of value from going to a less prestigious university. Ultimately most of us end up working at the same select companies. But idk I am from Canada so maybe the intensity of selection is not the same. We put far too much value in prediction of success and not enough value in actual success. Seems like the academic version of keeping up with the Jones's
• bigbacaloa 4y ago
  Every metric has problem, but experience in a university suggests that, if one has to use a single metric to rank students, by far the best choice (the only choice really) is average grade.
• awkward 4y ago
  > The full dataset isn’t available, but since we have the number of students in each ACT / GPA bin above, we can create a “pseudo” dataset, with just a small loss of precision. Big oof. This is an example of terminal data science poisoning. The data in the paper already proved the point of the article! Using it to create some chart gore where you reproduce the same plot in uncanny valley levels of resolution doesn't add to the argument. This is using very sophisticated techniques to achieve what you could by simply blurring an image of the first chart.
• seibelj 4y ago
  SAT / ACT are basically IQ tests. Grades are far more fuzzy but generally show an ability to stick with something over the long term and execute on projects. I think both are important to being successful. Ignoring one for the other is probably bad. That said I prefer IQ over grades.
  • skizm 4y ago
    Very much disagree. Lazy genius is useless. Hard working, low IQ gets stuff done. Obviously hard working genius is ideal. But without the ability to focus on longer term goals, which grades is an indicator of more so than test scores, your high IQ means nothing.
    • candiodari 4y ago
      I would argue it very, very much matters what you're trying to do. If you're harvesting apples, then sure. Hard working beats IQ. Hell, IQ is a handicap, not a plus. If you're doing research, you have the other side of the spectrum, IQ beats everything, and hard work can actually prevent you from finding solutions.
      • prepend 4y ago
        Research is lots of hard work. There’s a minimum level of intelligence needed, but I’d rather have an acceptable intelligence+hard worker as a collaborator than a genius+slacker. There aren’t many Archimedes out there and most research is the result of lots of labor.
        • maximus-decimus 4y ago
          How do you define "acceptable intelligence"? By their ability to better a lazy genius? I don't really see a non-tautological way to define "acceptable intelligence"
          • prepend 4y ago
• SicSemperUranus 4y ago
  > Test scores are highly predictive Any study in this area that doesn't control for socio-economic factors is near useless, because it mainly predicts that children of successful (ie rich) people will likely grow up to be successful (ie rich). This is almost tautological.
  • xyzzyz 4y ago
    You seem to be completely unfamiliar with the research in the field. It is well known that controlling for socioeconomic factors does not substantially impact predictive validity of standardized tests. See for example https://journals.sagepub.com/doi/abs/10.1177/0956797612438732 https://journals.sagepub.com/doi/abs/10.1177/095679761243873... > Scores on the SAT (a test widely used in the admissions process in the United States), secondary school grades, college grades, and SES measures from 143,606 students at 110 colleges and universities were examined, and results of these analyses were compared with results obtained using a 41-school data set including scores from the prior version of the SAT and using University of California data from prior research on the role of SES. In all the data sets, the SAT showed incremental validity over secondary school grades in predicting subsequent academic performance, and this incremental relationship was not substantially affected by controlling for SES. Or this https://eric.ed.gov/?id=ED562860 https://eric.ed.gov/?id=ED562860 > (a) SES is related to SAT scores (r = 0.42 among the population of SAT takers), (b) SAT scores are predictive of freshman grades (r = 0.47 corrected for school-specific range restriction), and (c) statistically controlling for SES reduces the estimated SAT-grade correlation from r = 0.47 to r = 0.44. There are more studies on this, with rather consistent results. In short, poor students do about as well as rich students when matched on SAT score.
    • majormajor 4y ago
      Why would one expect socioeconomic factors to do one or both of: a) influence grades in primary and secondary school b) influence standardized test scores but not also do: c) influence college performance (esp. freshman year when you're least removed from those same effects in HS)?
      • 4y ago
• RobLach 4y ago
  Who's saying that test scores are uncorrelated with college graduation, which is largely dependent on passing further tests?
  • majormajor 4y ago
    There's certainly a strong whiff of "is this the right question" to the whole discussion. School and school-adjacent results predict school results. Whoo. Tests are just leetcode for students, after all.
  • fasthands9 4y ago
    Lots of schools are dropping the SAT requirement https://meet.nyu.edu/uncategorized/test-optional-at-nyu-what-it-is-and-isnt/ https://meet.nyu.edu/uncategorized/test-optional-at-nyu-what... https://www.latimes.com/california/story/2022-03-23/csu-drops-sat-admissions-requirement-in-landmark-move https://www.latimes.com/california/story/2022-03-23/csu-drop... https://wtop.com/maryland/2022/06/maryland-university-system-drops-sat-act-admissions-requirement/ https://wtop.com/maryland/2022/06/maryland-university-system... I don't know how this works in practice tbh. But it is curious.
    • alwayslikethis 4y ago
      It's a strategy to game the rankings. Factors for college rankings often include acceptance rate (lower is better), and average SAT/ACT scores. By not requiring SAT/ACT scores, you do two things at once: 1. you allow more people to apply, especially those who did not take the test or received a poor score, who you can simply reject to bring down your acceptance rate 2. by making it optional, only the applicants who think their score would be an advantage (i.e. those with high scores) would send them, which brings up your "average test scores".
    • WillPostForFood 4y ago
      I don't know how this works in practice tbh. We have some evidence from MIT that it is a bad idea to drop testing. https://mitadmissions.org/blogs/entry/we-are-reinstating-our-sat-act-requirement-for-future-admissions-cycles/ https://mitadmissions.org/blogs/entry/we-are-reinstating-our...