6 ms·

I don't buy the notion that tests do not test relevant skills. In my long career I've noticed a strong correlation between SAT scores and academic performance

I don't buy the notion that tests do not test relevant skills. In my long career I've noticed a strong correlation between SAT scores and academic performance as well as job performance. > I don't think the comparison to flight school is relevant enough in this context because it's a too different of a world to traditional academia. My dad kept his flight school tests for flying all sorts of airplanes. They bear a lot of similarities with the SATs. There's a lot of math in there for things like fuel consumption, wind, maximum landing weight, glide distance, and so on. For example, one day he was cruising along in his F-86 when the engine failed. he radioed the tower, and they told him to bail out. But he calculated his speed, altitude, distance, wind, sink rate, air templeratur, etc., and figured he could make the field after configuring the airplane for maximum glide. He made a perfect landing, but still got reprimanded for risking his life bringing the airplane back. But he had worked the math and disagreed that it was more risky to bring it in than bail out.

hammock 4mo ago
> I don't buy the notion that tests do not test relevant skills. In my long career I've noticed a strong correlation between SAT scores and academic performance as well as job performance. SAT tests intelligence (aptitude), not skills. Which is why it correlates with job performance, where intelligence can (over some time) matter as much or more than a starting point of relevant skills.
- WalterBright 4mo ago
  Nothing's perfect, but the SAT tests do an adequate job.
- Newlaptop 4mo ago
  I just checked, and the SAT math section covers algebra, trigonometry and statistics. Look at this list: Quadratic equations and functions (vertex form, roots, discriminant) Polynomial operations and factoring Exponential functions and growth/decay Radical and rational expressions Function notation, composite and inverse functions Nonlinear graphs and their transformations A genius student who had never been taught those subjects wouldn't even know what the symbols meant. A mediocre student who had studied SAT-style questions for weeks leading up to the test would likely outperform a high IQ student who last solved those types of problems over a year prior. Standardized tests can be a great resource for assessing students, but they're not just testing for intelligence. Test-prep courses average increasing SAT scores by about 200 points. That's not because they're increasing the intelligence of the people taking them.
  - somenameforme 4mo ago
    Somebody who goes to take a test on something they know that they know nothing about could be called many things, but genius is not one, even moreso when they're paying for the privilege of taking that test. What is on the SAT is no secret, so people are free to prepare as little or as much as they might like. If somebody can't be assed to prepare for such a critical test, then they're probably going to be the sort of person who can't be assed to do much of anything in life. And the internet has also largely relegated the inequities in access to training quite obsolete. You can get free high quality training materials on everything for free.
    - sir0010010 4mo ago
      You've never met a smart high school student that didn't study? You've also never met someone who became more disciplined later in life?
      - somenameforme 4mo ago
        There's a difference between not studying and just hand-waving away the SAT, which plays a huge role in things like college admittance. And the SAT isn't particularly hard, even back in the days when it was apparently significantly more challenging, for a clever kid. My prep was picking up some old book on it at Half Price Books and leaving it in the bathroom. Worked excellently!
    - marcuskane2 4mo ago
      Or they're a student who has to work a part-time job after school. Or they have a long commute. Or they have hours of practice for a competitive sport or extracurricular. Or they have to take care of their siblings, grandparent or sick parent. Or they were told by their parents and teachers that because they're smart and do well in school that they don't need to worry about studying. Or they know their parents can't afford an elite school anyway, so they know they just have to score well enough to get into a school with in-state tuition and noncompetitive admissions. Two kids who both went to the same school, were told the same things about the importance of the SAT and dedicated equal time to study and preparation can probably say that the difference in their scores is indicative of differences in intelligence. But any two random students? There are so many factors that could cause students to underperform their hypothetical max score if they had perfect conditions in the months leading up to the test and day of.
- sir0010010 4mo ago
  I guess the question is: would you rather hire someone with poor SATs and god-tier Leetcoding skills or vice-versa?
  - sokoloff 4mo ago
    I’d for sure take the high SAT scores. I think the SAT has more correlation with/is more predictive of work success than leetcode.
nsagent 4mo ago
Do you also think LLM leaderboards accurately reflect the capabilities of the models being tested? If you do, then I can easily point you to numerous academic papers pointing out the various flaws in many leaderboards (from poorly designed benchmarks like bABI and the original SQuAD, to data contamination, and more). In that same way, any test, including the SAT and GRE have flaws. They can be gamed in ways similar to LLM leadeboards: test prep makes you better at them. That's one of the main reasons universities moved away from SAT; they were afraid that it disenfranchised lower socioeconomic status students (and it does to some degree). The issue is that the test is positively correlated with success in an undergraduate program, so they threw out the baby with the bathwster. The real issue is that the SAT is not able to distinguish the capabilities among students to the degree it purports to. And if you want an anecdote to match all yours, the first time I took a GRE practice test, I got a 3 on the writing. Not because I'm poor at writing, but because I didn't really know what they were looking for. After reading a test prep book, I got a 4.5 on my next practice test and a 5 on my final practice test. When I finally took the actual GRE, I got 6 on the analytical writing. Trust me, nothing changed in my writing ability over that time. In fact, I didn't even practice the skill except through those three practice tests. Clearly the test was not capable of determining my real ability to make an argument; it merely tested my ability to adapt my writing to what was supposedly being tested. Interestingly, the vast majority of universities that got rid of the GRE requirements for PhD programs are not going back on that. Turns out that the students with the highest GRE scores are the ones most likely to drop out of their STEM PhD. [1] [1]: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0206570 https://journals.plos.org/plosone/article?id=10.1371/journal...
- WalterBright 4mo ago
  I took the GREs, I don't recall a writing section. Anyhow, the questions were all about freshman engineering knowledge.
  - nsagent 4mo ago
    There are three major parts of the modern GRE: Verbal, Quantitative, and Analytical Writing. You could easily look that up, or ask if you didn't know. Responding off the cuff without any reflection on the comment you're responding to doesn't move the conversation forward in any meaningful way. It just comes across as disrespectful.
SkiFire13 4mo ago
> I don't buy the notion that tests do not test relevant skills. > In my long career I've noticed a strong correlation between SAT scores and academic performance as well as job performance. A test doesn't need to test the relevant skills for that, it just needs to test _something_ that correlates with academic performance and job success.
fmobus 4mo ago
In my country, we have public exams to get into Uni, with the ones for high demand majors being very competitive, but performance in that exam is not a good predictor of academic performance. The guy who got into my uni class as #2 in that exam dropped out after a few semesters because he couldn't beat calculus. The #4 took several extra semesters to graduate despite not working/not interning. Several others in the top third struggled through. We had _maybe_ 2 or 3 guys who straight-A'd the entire major. I myself got in as #17 and still failed a few courses. Thankfully no one cared throughout my professional career.
arw0n 4mo ago
> In my long career I've noticed a strong correlation between SAT scores and academic performance as well as job performance On average. However, I've also had the experience that some of the most competent people I've known had rather difficult teens and twenties. Hiring someone who flunked out of highschool, worked odd jobs for 10 years, then got a diploma and a degree is higher risk, higher reward. They are often times harder workers, unusual thinkers and more grateful for what they have.
danaris 4mo ago
My wife and I are perfect poster children for demonstrating the danger in making the assumption that tests are an accurate assessment of someone's mastery of material. My wife has anxiety, and the time-limited, high-stakes, pass-or-fail nature of a test makes it much, much harder for her to perform well. Outside that context, she's blazingly intelligent, performs very well under real-world high-pressure situations, is extremely diligent at getting any other assigned work done well on time, and has repeatedly written the guidelines for processes and procedures in her jobs. I'm the opposite in many ways. I generally do somewhat poorly in a classic classroom environment, primarily due to ADHD hampering my ability to get homework done regularly and on time. However, I test amazingly well. I consistently finish in 50-70% of the time of other students, with scores in the high 90% range. (In my jobs, I think I do pretty well.) Naturally the plural of "anecdote" is not "data", but I've known enough other people who fit both of our molds that I think it's fair to say that basing even a plurality of your assessment of someone purely on tests, especially standardised tests, is going to mislead you at least as often as not, in both directions.
koe123 4mo ago
Correlation isn’t causation. Students with test aptitude and ambition will score hire regardless of the test content.

hammock 4mo ago

> I don't buy the notion that tests do not test relevant skills. In my long career I've noticed a strong correlation between SAT scores and academic performance as well as job performance. SAT tests intelligence (aptitude), not skills. Which is why it correlates with job performance, where intelligence can (over some time) matter as much or more than a starting point of relevant skills.

WalterBright 4mo ago
Nothing's perfect, but the SAT tests do an adequate job.
Newlaptop 4mo ago
I just checked, and the SAT math section covers algebra, trigonometry and statistics. Look at this list: Quadratic equations and functions (vertex form, roots, discriminant) Polynomial operations and factoring Exponential functions and growth/decay Radical and rational expressions Function notation, composite and inverse functions Nonlinear graphs and their transformations A genius student who had never been taught those subjects wouldn't even know what the symbols meant. A mediocre student who had studied SAT-style questions for weeks leading up to the test would likely outperform a high IQ student who last solved those types of problems over a year prior. Standardized tests can be a great resource for assessing students, but they're not just testing for intelligence. Test-prep courses average increasing SAT scores by about 200 points. That's not because they're increasing the intelligence of the people taking them.
- somenameforme 4mo ago
  Somebody who goes to take a test on something they know that they know nothing about could be called many things, but genius is not one, even moreso when they're paying for the privilege of taking that test. What is on the SAT is no secret, so people are free to prepare as little or as much as they might like. If somebody can't be assed to prepare for such a critical test, then they're probably going to be the sort of person who can't be assed to do much of anything in life. And the internet has also largely relegated the inequities in access to training quite obsolete. You can get free high quality training materials on everything for free.
  - sir0010010 4mo ago
    You've never met a smart high school student that didn't study? You've also never met someone who became more disciplined later in life?
    - somenameforme 4mo ago
      There's a difference between not studying and just hand-waving away the SAT, which plays a huge role in things like college admittance. And the SAT isn't particularly hard, even back in the days when it was apparently significantly more challenging, for a clever kid. My prep was picking up some old book on it at Half Price Books and leaving it in the bathroom. Worked excellently!
  - marcuskane2 4mo ago
    Or they're a student who has to work a part-time job after school. Or they have a long commute. Or they have hours of practice for a competitive sport or extracurricular. Or they have to take care of their siblings, grandparent or sick parent. Or they were told by their parents and teachers that because they're smart and do well in school that they don't need to worry about studying. Or they know their parents can't afford an elite school anyway, so they know they just have to score well enough to get into a school with in-state tuition and noncompetitive admissions. Two kids who both went to the same school, were told the same things about the importance of the SAT and dedicated equal time to study and preparation can probably say that the difference in their scores is indicative of differences in intelligence. But any two random students? There are so many factors that could cause students to underperform their hypothetical max score if they had perfect conditions in the months leading up to the test and day of.
sir0010010 4mo ago
I guess the question is: would you rather hire someone with poor SATs and god-tier Leetcoding skills or vice-versa?
- sokoloff 4mo ago
  I’d for sure take the high SAT scores. I think the SAT has more correlation with/is more predictive of work success than leetcode.

nsagent 4mo ago

Do you also think LLM leaderboards accurately reflect the capabilities of the models being tested? If you do, then I can easily point you to numerous academic papers pointing out the various flaws in many leaderboards (from poorly designed benchmarks like bABI and the original SQuAD, to data contamination, and more). In that same way, any test, including the SAT and GRE have flaws. They can be gamed in ways similar to LLM leadeboards: test prep makes you better at them. That's one of the main reasons universities moved away from SAT; they were afraid that it disenfranchised lower socioeconomic status students (and it does to some degree). The issue is that the test is positively correlated with success in an undergraduate program, so they threw out the baby with the bathwster. The real issue is that the SAT is not able to distinguish the capabilities among students to the degree it purports to. And if you want an anecdote to match all yours, the first time I took a GRE practice test, I got a 3 on the writing. Not because I'm poor at writing, but because I didn't really know what they were looking for. After reading a test prep book, I got a 4.5 on my next practice test and a 5 on my final practice test. When I finally took the actual GRE, I got 6 on the analytical writing. Trust me, nothing changed in my writing ability over that time. In fact, I didn't even practice the skill except through those three practice tests. Clearly the test was not capable of determining my real ability to make an argument; it merely tested my ability to adapt my writing to what was supposedly being tested. Interestingly, the vast majority of universities that got rid of the GRE requirements for PhD programs are not going back on that. Turns out that the students with the highest GRE scores are the ones most likely to drop out of their STEM PhD. [1] [1]: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0206570 https://journals.plos.org/plosone/article?id=10.1371/journal...

WalterBright 4mo ago
I took the GREs, I don't recall a writing section. Anyhow, the questions were all about freshman engineering knowledge.
- nsagent 4mo ago
  There are three major parts of the modern GRE: Verbal, Quantitative, and Analytical Writing. You could easily look that up, or ask if you didn't know. Responding off the cuff without any reflection on the comment you're responding to doesn't move the conversation forward in any meaningful way. It just comes across as disrespectful.

SkiFire13 4mo ago

> I don't buy the notion that tests do not test relevant skills. > In my long career I've noticed a strong correlation between SAT scores and academic performance as well as job performance. A test doesn't need to test the relevant skills for that, it just needs to test _something_ that correlates with academic performance and job success.

fmobus 4mo ago

In my country, we have public exams to get into Uni, with the ones for high demand majors being very competitive, but performance in that exam is not a good predictor of academic performance. The guy who got into my uni class as #2 in that exam dropped out after a few semesters because he couldn't beat calculus. The #4 took several extra semesters to graduate despite not working/not interning. Several others in the top third struggled through. We had _maybe_ 2 or 3 guys who straight-A'd the entire major. I myself got in as #17 and still failed a few courses. Thankfully no one cared throughout my professional career.

arw0n 4mo ago

> In my long career I've noticed a strong correlation between SAT scores and academic performance as well as job performance On average. However, I've also had the experience that some of the most competent people I've known had rather difficult teens and twenties. Hiring someone who flunked out of highschool, worked odd jobs for 10 years, then got a diploma and a degree is higher risk, higher reward. They are often times harder workers, unusual thinkers and more grateful for what they have.

danaris 4mo ago

My wife and I are perfect poster children for demonstrating the danger in making the assumption that tests are an accurate assessment of someone's mastery of material. My wife has anxiety, and the time-limited, high-stakes, pass-or-fail nature of a test makes it much, much harder for her to perform well. Outside that context, she's blazingly intelligent, performs very well under real-world high-pressure situations, is extremely diligent at getting any other assigned work done well on time, and has repeatedly written the guidelines for processes and procedures in her jobs. I'm the opposite in many ways. I generally do somewhat poorly in a classic classroom environment, primarily due to ADHD hampering my ability to get homework done regularly and on time. However, I test amazingly well. I consistently finish in 50-70% of the time of other students, with scores in the high 90% range. (In my jobs, I think I do pretty well.) Naturally the plural of "anecdote" is not "data", but I've known enough other people who fit both of our molds that I think it's fair to say that basing even a plurality of your assessment of someone purely on tests, especially standardised tests, is going to mislead you at least as often as not, in both directions.

koe123 4mo ago

Correlation isn’t causation. Students with test aptitude and ambition will score hire regardless of the test content.