5 ms·

I'm specifically not talking about the SAT and ACT since they're widely used by elite universities. Those tests top out way too early and can't reliably disting

I'm specifically not talking about the SAT and ACT since they're widely used by elite universities. Those tests top out way too early and can't reliably distinguish between the top 1% and the top 0.1% (The fraction of perfect scorers is much smaller than 1% but the standard error of measurement is way too large). The exams I have in mind are those like the science olympiads and the Cambridge STEP. STEP in particular is basically used for this purpose (olympiads are a slightly different beast).

saagarjha 6y ago
In the US the appropriate exams would likely be the qualifiers for the International Olympiads.
TuringNYC 6y ago
It those were used for admissions, I wouldn’t be surprised if there was a huge market for answers (from somewhere in the production chain), stand-in test takers, etc.
godelski 6y ago
Are you trying to get students that can pass specific kinds of tests or students that will be successful? The two aren't strongly correlated. I'd argue successful builds more prestige for the university. And let's be real, everyone knows these tests are purposefully tricky. In the real world people don't give you multiple choices and try to trick you into picking the wrong answer and then laugh at you when you picked the one try tried to get you to pick (politics aside). In classes, unless your professor is an asshole, they aren't trying to trick you. Success depends on a lot of factors well beyond what these tests measure. This is why SAT and ACT don't measure success or even academic success (as in they don't strongly correlate [0][1] with success. This source should decent correlation with first year grades but not second). I'm not sure why one would even think an academic test could determine success. One of the biggest lies many of us have been told is that if you're just smart enough you'll be successful. Sorry, but that's not how it works. [0] https://edpolicyinca.org/publications/predicting-college-success-how-do-different-high-school-assessments-measure-2019 https://edpolicyinca.org/publications/predicting-college-suc... [1] https://www.latimes.com/california/story/2019-12-22/grades-vs-sat-scores-which-is-a-better-predictor-of-college-success https://www.latimes.com/california/story/2019-12-22/grades-v...
- deleted 6y ago
  [deleted]
- jeffe 6y ago
  At the extreme end of the spectrum with quant hedge funds this actually becomes somewhat true. They expect strong academic understanding of math and make a killing by hiring those who do.
krastanov 6y ago
I have some useful background here that leads me to strongly disagree with you. I did well at these international science olympiads in high school. To do that I studied for the particular shape of the exam questions. I became friends with many other people who were high performers at these olympiads. Later on I got a PhD from a prestigious institution and I am now a scientist at another one. Throughout this time, I was very involved in the pedagogical work at these institutions. One very important piece of insight I gained throughout that time is that after you use some exam to get the top 25% (ish) percent of students, the exam performance really does not matter: it does not correlate with scientific output or originality of work. If anything, I had to unlearn some of the skills that made me good at olympiads because they were severely limiting my creativity. At an olympiad you know there is a solution, while true research problems might be unsolvable and need to be approached differently. TL;DR: Exams (selective or not, hard or not) are great at giving you the top 20% of students, but they are inherently terrible at telling you who among the top 20% will be a productive scientist or engineer.
- MagnumOpus 6y ago
  Your anecdotal evidence is contradicted by wider evidence. In particular the famous Benbow study [1] that established that even among the top percentile of the population there are massive differences: the top quarter percentile (99.75-99.99) was 2-3 times more likely to have authored academic research later in life, and about 1.5x more likely to have attended postgrad education, to have gone for a STEM degree, to have gone for a PhD than the bottom quarter of the top percent (99.00-99.24). [1] https://www.gwern.net/docs/iq/smpy/1992-benbow.pdf https://www.gwern.net/docs/iq/smpy/1992-benbow.pdf
  - krastanov 6y ago
    This is interesting, thanks for sharing it! It would take me some time to read it, but given that you seem the have already looked into this, I was wondering whether you have suggestion for a wider set of studies, a meta-study maybe. I am asking, because if I have to choose between my empirical observations (anecdotal as they are, given it is n=1 observers), and a single piece of research without followups, I do feel justified to stick to what I have seen myself. But I am open to be convinced otherwise.

saagarjha 6y ago

In the US the appropriate exams would likely be the qualifiers for the International Olympiads.

TuringNYC 6y ago

It those were used for admissions, I wouldn’t be surprised if there was a huge market for answers (from somewhere in the production chain), stand-in test takers, etc.

godelski 6y ago

Are you trying to get students that can pass specific kinds of tests or students that will be successful? The two aren't strongly correlated. I'd argue successful builds more prestige for the university. And let's be real, everyone knows these tests are purposefully tricky. In the real world people don't give you multiple choices and try to trick you into picking the wrong answer and then laugh at you when you picked the one try tried to get you to pick (politics aside). In classes, unless your professor is an asshole, they aren't trying to trick you. Success depends on a lot of factors well beyond what these tests measure. This is why SAT and ACT don't measure success or even academic success (as in they don't strongly correlate [0][1] with success. This source should decent correlation with first year grades but not second). I'm not sure why one would even think an academic test could determine success. One of the biggest lies many of us have been told is that if you're just smart enough you'll be successful. Sorry, but that's not how it works. [0] https://edpolicyinca.org/publications/predicting-college-success-how-do-different-high-school-assessments-measure-2019 https://edpolicyinca.org/publications/predicting-college-suc... [1] https://www.latimes.com/california/story/2019-12-22/grades-vs-sat-scores-which-is-a-better-predictor-of-college-success https://www.latimes.com/california/story/2019-12-22/grades-v...

deleted 6y ago
[deleted]
jeffe 6y ago
At the extreme end of the spectrum with quant hedge funds this actually becomes somewhat true. They expect strong academic understanding of math and make a killing by hiring those who do.

krastanov 6y ago

I have some useful background here that leads me to strongly disagree with you. I did well at these international science olympiads in high school. To do that I studied for the particular shape of the exam questions. I became friends with many other people who were high performers at these olympiads. Later on I got a PhD from a prestigious institution and I am now a scientist at another one. Throughout this time, I was very involved in the pedagogical work at these institutions. One very important piece of insight I gained throughout that time is that after you use some exam to get the top 25% (ish) percent of students, the exam performance really does not matter: it does not correlate with scientific output or originality of work. If anything, I had to unlearn some of the skills that made me good at olympiads because they were severely limiting my creativity. At an olympiad you know there is a solution, while true research problems might be unsolvable and need to be approached differently. TL;DR: Exams (selective or not, hard or not) are great at giving you the top 20% of students, but they are inherently terrible at telling you who among the top 20% will be a productive scientist or engineer.

MagnumOpus 6y ago
Your anecdotal evidence is contradicted by wider evidence. In particular the famous Benbow study [1] that established that even among the top percentile of the population there are massive differences: the top quarter percentile (99.75-99.99) was 2-3 times more likely to have authored academic research later in life, and about 1.5x more likely to have attended postgrad education, to have gone for a STEM degree, to have gone for a PhD than the bottom quarter of the top percent (99.00-99.24). [1] https://www.gwern.net/docs/iq/smpy/1992-benbow.pdf https://www.gwern.net/docs/iq/smpy/1992-benbow.pdf
- krastanov 6y ago
  This is interesting, thanks for sharing it! It would take me some time to read it, but given that you seem the have already looked into this, I was wondering whether you have suggestion for a wider set of studies, a meta-study maybe. I am asking, because if I have to choose between my empirical observations (anecdotal as they are, given it is n=1 observers), and a single piece of research without followups, I do feel justified to stick to what I have seen myself. But I am open to be convinced otherwise.