14 ms·
our ability to accurately predict student academic success at MIT 02 Our research shows this predictive validity holds even when you control for socioeconomic f
by paulpauper 5y ago
our ability to accurately predict student academic success at MIT 02 Our research shows this predictive validity holds even when you control for socioeconomic factors that correlate with testing. It also shows that good grades in high school do not themselves necessarily translate to academic success at MIT if you cannot account for testing. Of course, we can never be fully certain how any given applicant will do: we're predicting the development of people, not the movement of planets, and people always surprise you. However, our research does help us establish bands of confidence that hold true in the aggregate, while allowing us, as admissions officers, to exercise individual contextual discretion in each case. The word 'significantly' in this bullet point is accurate both statistically and idiomatically.is significantly improved by considering standardized testing — especially in mathematics — alongside other factors
So much for that common, popular notion that standardized tests do not predict anything of value.
- globuous 5y agoWhat's surprising though, is that APs and similar exams are not enough. In the UK, I though they essentially looked at A Level results, which are much more representative of what you'll actually study at uni. But I guess both SAT/ACTs & APs must be a better measure that just APs. I just remember fucking hating studying for the SATs though. So boring. SAT IIs were somewhat fun to study for though. In France for instance, they mostly just look at the baccalaureat to get into prep schools / first year at uni. Then exams to get into engr/business/vet schools are actually very interesting topics and very close to what you'll actually study. Same with exams at the end of the first year of med school (which you get into right after 12th grade, unlike in the US where it's after your bachelors). That being said, they seem to have backed up their numbers, and MIT knows how to count, so they must be right! I just always hoped SAT/ACTs weren't that conclusive so that we didn't have to go through them anymore and could focus on the funner AP/A Level stuff :)
- amalcon 5y agoThe boringness could actually be a big part of the effectiveness. Efficient study habits and ability to work through boredom certainly help with some undergrad classes. The test would have some predictive power even if it's just measuring those.
- baja_blast 5y agoAlso it's hard to standardize something that is not boring.
- paulmd 5y agotbh I thought APs were generally more difficult than actual classes at a high-level university (USAFA). And my high school's regular courses (granted a fantastic high school) were actually much more difficult than a state school's courses. That said, outside of admissions, I don't think I got academic value out of them. They were hard for the sake of being hard. I'd rather have taken the SAT or ACT any day. (also apropos of nothing but I don't think much of the writing section on the SAT either, which was a hot topic 15 or so years ago... a huge amount is dependent on the graders, and it's fundamentally a "blackboard programming" type scenario where the student is separated from basic resources like word processing and graded on the resulting product... that's not how you would actually work in an academic setting.)
- brimble 5y ago> And my high school's regular courses (granted a fantastic high school) were actually much more difficult than a state school's courses. I went to a barely-known state university and was very surprised when some of my intro-level gen ed requirement classes mostly covered material I'd already seen in, and with a similar level of rigor to, junior high school. And my junior high and high schools were nothing special at all—at the higher end of performance in the state (so far as those measures are helpful, anyway) but just regular public schools in a state with overall mediocre-bordering-on-poor schools. If I'd known that the first couple years of college weren't going to be harder than high school, and would have a lower total time commitment, hell, I'd have probably tried to go the drop out -> GED -> start college at 16 or 17 route. I wasn't gonna get into top-tier universities, anyway.
- adfgadfgaery 5y ago>What's surprising though, is that APs and similar exams are not enough. That isn't what they said. They said that access to those tests is not universal. Students from high schools that don't offer AP classes would have a hard time taking AP exams. This would exclude people from rural or impoverished areas. This is why the SAT and ACT are useful: they are meant to be aptitude tests. They are IQ tests in disguise. If properly designed, they will measure intelligence with minimal influence from education or cultural background. Theoretically something like these tests could be administered to elementary school students and still be useful for predicting success in college a decade later.
- floren 5y ago> They said that access to those tests is not universal. Students from high schools that don't offer AP classes would have a hard time taking AP exams. Yeah, I wish they'd just flat out told me "we expect AP courses" before I applied for MIT back in the day. Would have saved me a lot of hassle that just resulted in "sorry, we wanted AP credits" in the end.
- kragen 5y agoI passed the AP calc exam without a class. But that had a lot more to do with motivation and interest and a sense of entitlement than with aptitude. I wish everyone had my sense of entitlement, but they don't, and classes do seem to make a passable substitute.
- paulpauper 5y agoI practiced the math part several time to make sure I had it down pat, never the writing part though. Reading those long essays is a chore. I think the reading/verbal part is less coachable than the math part.
- JamesBarney 5y agoThere are a lot of highschools where AP classes aren't really available, or are taught with varying degrees of rigor.
- JustLurking2022 5y agoAP scores and SAT II's are highly subject to the quality of instruction. I had several teachers who treated it as a more advanced class than honors, but felt no need to teach to the rubric for the test specifically. I aced the SAT and ACT but had a decent number of mediocre AP scores because I was seeing the material for the first time when I opened the test. Got to college and after a single 45 minute lecture covering the gap material, I'm pretty sure I could have scored a 5. Ended up making for several easy A's freshman year.
- LewisVerstappen 5y agoIt's far easier for rich students to game GPA, college essays and extracurriculars than it is for them to game the SAT. For GPA, they can hire a private tutor in the subject they're struggling with. There are services out there that will basically write your english essay for you / do your math/science homework. For college essays, they can hire college counselors to help them draft a compelling essay. For extra curriculars, they can hire a private coach, etc. However, there is very limited evidence that SAT coaching actually increases your SAT score. The NACAC did a study on this and they found that average gains of test-prep students is ~30 points (this was when the SAT was 2400 points). A 30 point increase out of 2400 points is not material to college admissions. Out of the college essay, gpa, extracurriculars, etc. the SAT is the least influenced by your socioeconomic status. There obviously is an influence, but removing the SAT means more reliance on even more skewed factors. Here's the study -> https://files.eric.ed.gov/fulltext/ED505529.pdf https://files.eric.ed.gov/fulltext/ED505529.pdf
- thescriptkiddie 5y agoI wonder how much taking the SAT multiple times plays a role. Personally I took it at least three times, and my score improved each time (don't remember by how much).
- Loughla 5y agoWhich is also tied to socioeconomic status. If you can pay, your score will go up the more you take it, according to my experience, and the college board. If you can't pay, you take it the one time it's offered for free if your school offers that. Then you get what you get.
- whimsicalism 5y agoI think we should learn from the gaokao and only test once a year. The fact that you can pay for more tests (and pay more for "score choice") is the most unjust part of the whole thing.
- tharne 5y ago> So much for that common, popular notion that standardized tests do not predict anything of value. To be fair, I don't think the debate was ever about the quality or predictive value of the tests. There is a small, but well-organized and vocal subset of the population that hates the idea of excellence and differentiation. They want, and have been quite successful in, the replacement of standards of excellence with vaguely defined (defined by them, of course) buzzwords like "equity" and "diversity".
- gameswithgo 5y agoIs the current push back against coding tests in job hiring perhaps similar to this push back against the SAT?
- yeahwhatever10 5y agoI think that is more about the disconnect between coding tests and the actual day to day work and skills required to do the job. Example: I could have a high level of competency in software engineering and also not care how a mouse gets out of a bucket.
- alecbz 5y agoThose "mouse getting out of a blender" brain-teasers or whatever are pretty unheard of at this point I think. Most people complain about coding questions, generally leetcode-style questions I think.
- artful-hacker 5y agoIt's similar but it also brings in a challenging problem: coding tests costs candidates far more than it costs employers in terms of time. I am currently interviewing and two of the companies I am otherwise excited for sent take-home tests that just exhaust me, especially after a long day of otherwise productive work. I've got 12 years of experience under my belt but somehow great references and a killer resume aren't enough to convince them I can find a security vulnerability.
- smrtinsert 5y agoThe current discussion I thought focuses around the idea that higher scores should just about guarantee acceptance due some fantasy notion of objective merit, and here they pretty much say that's not the case here: > To be clear, performance on standardized tests is not the central focus of our holistic admissions process. We do not prefer people with perfect scores; indeed, despite what some people infer from our statistics, we do not consider an applicant’s scores at all beyond the point where preparedness has been established as part of a multifactor analysis
- JustLurking2022 5y agoSeems like semantics - maybe they don't explicitly prefer perfect scores but it sounds like they're filtering out using a minimum bar, and I'd bet it isn't too far off perfect.
- School-Cotton 5y ago> So much for that common, popular notion that standardized tests do not predict anything of value. It was always an example of people refusing to believe something because it would be nicer if it weren't true.
- hintymad 5y agoI wonder why we can't hire more teachers to grade word problems in tests like SAT, like Asian countries do. Those problems are much harder to game or cram.
- ghaff 5y agoMany years ago--business grad school--one of the professors (forecasting?) had done research into GMATs and various post-school success metrics. Like it or not, the standardized test scores were a better predictor (by far) than anything else. In terms of gaming scores on standardized tests generally, yes people with more money can take prep classes and the like, but there are also test prep books available--presumably even from the library--that probably get you a lot of the way there.
- screye 5y agoMIT and Caltech have always held a 'number-oriented universities' perception in my mind. This is in contrast to a 'prestige-oriented or identity-oriented' perception that I hold for Ivies. It is nice to see MIT do justice to those priors.
- didibus 5y agoI guess the issue in their analysis is that they are trying to predict MIT grade success, which is possibly another flawed metric. If say exams at MIT resembled SAT, it seems more logical that you'd find a correlation between SAT success and MIT academic success. What would be a more interesting measure of real value is to study for academic innovation and invention (valuable to society), as well as future success on the job market or at new business ventures (valuable to the student and economy). I'm making the assumption here that since we have limited educational resources, we'd want to provide the best education to those most likely to advance an academic field through new discovery or insight or invention, as well as those who'd best innovate or provide for existing business and services to society. And I'm curious if we've ever had any study looking into that? Or if this one did?
- sgustard 5y ago> common, popular notion that standardized tests do not predict anything of value It's backed up by research, for example: https://news.uchicago.edu/story/test-scores-dont-stack-gpas-predicting-college-success https://news.uchicago.edu/story/test-scores-dont-stack-gpas-... I don't doubt that MIT's study showed otherwise for their needs. But UChicago is also a top-tier school and is not requiring standardized tests, for educational not political reasons.
- WalterBright 5y agoThe University of California commissioned a study to determine the predictive value of SATs for college success. A strong positive correlation was found. The test requirement was dropped anyway.
- deleted 5y ago[deleted]
- fortran77 5y ago> So much for that common, popular notion that standardized tests do not predict anything of value. To people with a particular agenda -- that society will be improved if equal outcomes are mandated -- I suppose standardized tests aren't "valuable" to them. I am very grateful for the SAT. I wasn't a good student. I was unable to do any work outside of school because of a bad home (my brother did hours long "hand-clapping/stimming" and chanting rituals, my mother drank, and my father was violent). But I did well on the SAT -- enough to get a national merit scholarship and a scholarship from Hofstra University where I got my BA in Math in the early 80s. If it wasn't for the SAT I don't know what I would have done. Standardized tests are the only answer, and I was very upset when I read of schools getting rid of using them for admission. SATs are also a great predictor of a person's ability to complete college. That was one of their original uses. So without SATs, you'll get more people doing poorly, and more people with no degrees and a lot of debt. And if these people are members of what's considered an "under-represented minority" then there will be even more remedial action required elsewhere to fix the problem of the high failure rate (like giving them degrees anyway, etc).
- tptacek 5y agoIt's important to keep in mind that the context here is the SAT/ACT versus the currently available alternatives. They're not saying that the SAT/ACT is good, but simply that it's better than other options. Some of those options, like alternative standard tests, may be significantly better, but in the world as it is today, it doesn't matter if they are, because lower-income candidates don't have access to them.
- nkrisc 5y agoJust an interesting anecdote on the predictive power of standardized tests: When I took the ACT in 2006, I scored around a 24 or 26 for composite score. Not very good. I didn't prep nor study for it because I was pretty apathetic about school and did my best to coast on whatever natural talent I could muster. Since my score wasn't very good, I retook it several weeks later. About halfway through I realized I was given the exact same question set as the first time I took it. I had of course not studied nor prepped for the second time, being the apathetic teenager I was. However this time I score around 32 or 34 or so. I don't remember exactly. What was different? Why did my score go up about 8 points? Better mood that day? Which score was the "real" one that best represented my abilities? Did I eat a better breakfast beforehand? I recognized some of the questions but of course I never expected to see the same questions a second time so I didn't prep for that. I didn't prep for anything about it. I wonder how many kid's college admissions results are ultimately because of one bad or good day? I suppose the other lesson is if you get a bad score, try again. You only need one good score.
- hooande 5y agoThe problem with testing isn't that it's inaccurate, but that it's a poor tool for the problem that it's being used to solve. You can't express "likelihood of success in college" in one number. Looking back on my time in college, the idea that my success could be predicted by my knowledge of geometry and vocabulary words is laughable. If there was a test that could accurately predict someone's chances of "success", be it SAT or IQ test, it would be used by everyone for everything. Billion dollar companies would be giving CEOs a version of that test before hiring them. A near perfect SAT score is noteworthy personal trivia, but other than that it loses all meaning as soon as someone steps on campus. Schools with strong brands use tests because they have way more applicants than they can properly review. Standardized testing is made necessary by scale, not predictive accuracy. I'm sure that anyone who has worked in college admissions for years has a very clear picture of what a successful student looks like. But there doesn't seem to be a way to quantify and codify that knowledge. And there likely wouldn't be time to apply it to tens of thousands of people in a few months anyway. So, standardized testing is what we have until someone comes up with something better.
- toomim 5y ago> the idea that my success could be predicted by my knowledge of geometry and vocabulary words is laughable. You really should look into the g-factor research. This isn't about knowledge, but rather performance. It turns out that your performance on geometry and vocabulary tests is highly correlated with your performance on tests in such disparate fields as Classics and Music: https://en.wikipedia.org/wiki/G_factor_(psychometrics)#Cognitive_ability_testing https://en.wikipedia.org/wiki/G_factor_(psychometrics)#Cogni... The crazy result in general intelligence is that your performance in all these areas is highly correlated, and incredibly correlated with career success: > "Research indicates that tests of g are the best single predictors of job performance, with an average validity coefficient of .55 across several meta-analyses of studies based on supervisor ratings and job samples."
- macrolocal 5y agoDon't underestimate the value of the kind of conscientiousness it takes to do well on these tests! I found grit much more useful than cleverness when I was an academic researcher.
- deleted 5y ago[deleted]
- deleted 5y ago[deleted]