7 ms·

I always enjoy the advocates who claim that students have mastered their subjects, but "don't test well". Would you want a pilot on your flight who flunked fly

I always enjoy the advocates who claim that students have mastered their subjects, but "don't test well". Would you want a pilot on your flight who flunked flying school exams, but somehow "really knew how to fly!"?

Maxatar 4mo ago
Not really comparable... the overwhelming majority of flight tests involve flying an aircraft. There is no meaningful way someone can be excellent at flying an aircraft but can't pass a test which involves flying an aircraft. The same can't be said for many other tests. If the test involves the practical application of the very skill being tested, then that test has direct relevance to he competency of said skill. But many other tests are not like that. A teacher can be brilliant in the classroom yet stumble on a standardized certification exam full of pedagogical jargon. A chef can cook a variety of excellent dishes but fail a written culinary theory exam testing the French names of techniques they perform by instinct. And perhaps more relevant to this audience, a coding interview that relies on whiteboarding algorithms from memory can easily fail an excellent engineer who builds great software every day but doesn't recall the optimal solution to some puzzle on the spot.
- WalterBright 4mo ago
  They are never going to let you into a cockpit until you pass ground school, which involves a lot of math. > A teacher can be brilliant in the classroom yet stumble on a standardized certification exam full of pedagogical jargon. A teacher that cannot explain how calculus works cannot teach it to anybody. > a coding interview that relies on whiteboarding algorithms from memory can easily fail an excellent engineer who builds great software every day but doesn't recall the optimal solution to some puzzle on the spot. I've seen too many coders using bubble sort because they don't know enough to look for a better algorithm. In any case, the purpose of leet coding tests is to quickly filter out the utter frauds. I have a programmer friend who wanted a job at a major software corp. He knew he'd have to pass the leetcode in an early stage of the interviewing. He figured it would take 6 weeks or so to study that material. I suggested that, since he was applying for a $250K job, that would be the most productive studying he'd ever done. He agreed, did the 6 weeks of studying, aced the leetcode test, and got the $250K. So ya, there is a point to those tests, in filtering out the frauds and the ones who aren't willing to do what it takes to get those jobs.
  - Maxatar 4mo ago
    This has to be a joke... Ground school most certainly does not involve a lot of math, it's not like there's any calculus or algebra involved... it's basic arithmetic. Furthermore it's categorically false that you need to pass ground school before you're allowed to fly. Are you just making things up? >A teacher that cannot explain how calculus works cannot teach it to anybody. This is a strawman argument, I never made anything that could even remotely be interpreted as this. >I've seen too many coders using bubble sort because they don't know enough to look for a better algorithm. This is committing a very basic logical fallacy. The fact that someone who is incompetent likely can't pass a test is not the same claim as someone who can't pass a test is likely incompetent. Hopefully you are able to identify this logical mistake that you're committing and revise your position accordingly.
    - WalterBright 4mo ago
      > categorically false Google sez: "The U.S. Air Force strictly requires you to complete and pass formal academic ground training before you ever touch the controls of an aircraft" They're not going to risk an aircraft on an incompetent student. > A teacher can be brilliant in the classroom yet stumble on a standardized certification exam full of pedagogical jargon I stand by my statement. > logical fallacy A implies B meaning B implies A is indeed a logical fallacy. But that does not rule out B implies A. A and B can be strongly related to each other.
      - Maxatar 4mo ago
        So you are making things up... thanks for confirming that. While I appreciate that you reviewed what Google "sez"... you have misunderstood the relevant context which is that the U.S. Air Force also requires that you complete Initial Flight Training (IFT) before you start the Air Force's own formal training program (UPT). In IFT you will not be required to pass ground school before you get to fly. Furthermore, even if the Air Force did not require IFT before UPT (the Air Force's own training program), you've completely changed the nature of your argument. I have no dispute about whether the Air Force may or may not have stricter requirements for their pilots, but that wasn't your argument. >I stand by my statement. You've proudly planted your flag on a point nobody was contesting, which is a strange hill to celebrate on but you do you. >But that does not rule out B implies A. A and B can be strongly related to each other. Discussing a topic with someone who not only uses logically fallacies as justification for their argument but brazenly doubles down on said fallacy is a good sign that this is probably not a good discussion to continue spending time on. Like am I supposed to simply accept your logical fallacy and take on the burden of disproving every claim you can dream up simply because you've asserted it isn't logically impossible? The person making the claim carries the burden of supporting it, and "they're strongly related" is something you have to actually show, not something I'm obligated to refute on your behalf.
- BobbyJo 4mo ago
  The overwhelming majority of math tests involve doing math, so I'm not sure your critique is useful in this context.
  - Maxatar 4mo ago
    Given that I'm not responding to any claim about the efficacy of math tests... it's actually your statement which is wholly irrelevant to this discussion.
    - BobbyJo 4mo ago
      "You can't compare academic tests and flight tests because flight tests involve flying" "Academic tests involve academics" "I'm not talking about academics!" That about sum it up?
      - Maxatar 4mo ago
        The submission isn't about academic tests involving academics... it's about the relevance of using such academic tests for the purpose of job screening.
        lotsofpulp 4mo ago
        A business that wastes money on ineffective screening methods for employees will lose to a business hiring better employees.
        Maxatar 4mo ago
        Which is why almost no employers ask for SAT scores as a way of screening candidates, and why this article was written to showcase the specific reasons doing so is ineffective. Of course you'd have refrained from making such a vacuous reply if you bothered to read the article.
- jimbokun 4mo ago
  Is there a reason you left out the SAT and ACT? Because both have been shown to have predictive power for success in college.
  - Maxatar 4mo ago
    Because I'm not trying to make a universal claim about all tests.
  - halostatue 4mo ago
    For one semester. Maybe two.
- fn-mote 4mo ago
  The problem that colleges and the SWE profession in general face are identifying “bullshitters”. We need a filter and a fact based exam seems like a good place to start.
andrecarini 4mo ago
My understanding is you're equating `failing a test` to `lacking the relevant skills and knowledge to do a certain task competently`. The reality is sometimes tests in academia are just not very well made and don't really test what they are supposed to be testing, and that's usually due to multiple reasons like misaligned incentives, staffing shortages and maybe lack of resources / funding. I don't think the comparison to flight school is relevant enough in this context because it's a too different of a world to traditional academia.
- WalterBright 4mo ago
  I don't buy the notion that tests do not test relevant skills. In my long career I've noticed a strong correlation between SAT scores and academic performance as well as job performance. > I don't think the comparison to flight school is relevant enough in this context because it's a too different of a world to traditional academia. My dad kept his flight school tests for flying all sorts of airplanes. They bear a lot of similarities with the SATs. There's a lot of math in there for things like fuel consumption, wind, maximum landing weight, glide distance, and so on. For example, one day he was cruising along in his F-86 when the engine failed. he radioed the tower, and they told him to bail out. But he calculated his speed, altitude, distance, wind, sink rate, air templeratur, etc., and figured he could make the field after configuring the airplane for maximum glide. He made a perfect landing, but still got reprimanded for risking his life bringing the airplane back. But he had worked the math and disagreed that it was more risky to bring it in than bail out.
  - hammock 4mo ago
    > I don't buy the notion that tests do not test relevant skills. In my long career I've noticed a strong correlation between SAT scores and academic performance as well as job performance. SAT tests intelligence (aptitude), not skills. Which is why it correlates with job performance, where intelligence can (over some time) matter as much or more than a starting point of relevant skills.
    - WalterBright 4mo ago
hammock 4mo ago
> Would you want a pilot on your flight who flunked flying school exams, but somehow "really knew how to fly! Sure, just not in the cockpit
chasd00 4mo ago
I would be much more ok with someone who failed the test but knew how to pilot a plane vs someone who aced the test but can’t figure out how to get the engine started.
- WalterBright 4mo ago
  One who cannot start the engine cannot kill me. One who cannot calculate how much fuel he needs to cross the lake will kill me. Remember JFKjr? He killed himself, his fiance and her friend because he did not pay attention to the instruments. An acquaintance of mine died trying to fly through a thunderstorm. Another one didn't pay attention to the weather and nearly died from wing icing. Flying is no joke.
Morromist 4mo ago
I did very well with my SAT but to get to the city we had to take it was a 2 hour long drive through twisted roads that made me carsick. I lived in a small town far away from the city. By the time I got there my breakfast I had quickly eaten gave me a stomach ache, I had woken up far earlier than usual and not gotten my 7 hours too. I certainly would have done a lot better if I had lived in the big city. Another factor: if you wanted to pay the fee you could just take the test over and over again until you got a great score. So kids with poor parents obviously had a huge disadvantage. Also kids who had the time and money could study for it with prep books - I did, while some of my friends were flipping burgers while still in highschool. Its not surprising I got a higher score than them, but it said nothing about my intellegence or understanding compaired to theirs.
- apparent 4mo ago
  There are fee waivers for poor kids, and free test prep thanks to Khan Academy. These may not have existed decades ago, but they exist now.
  - Morromist 4mo ago
    Fee waivers are great. A lot of poor parents are too busy / don't have their shit together enough to learn about them & use them. A lot of poor parents won't even have their kids take the tests at all even if they are free, especially if it means taking an entire day to drive them to the test facility. They might not even know what an SAT test is.
    - apparent 4mo ago
      Hm that's true, but the alternative would also be flawed, and possibly worse. If you could only take the SAT once and you were stuck with that score, then people would complain that rich kids could take all sorts of prep courses to get ready and to figure out what they needed to work on before taking the test, whereas poor kids would be figuring this all out on test day. There is no ideal way to do this, and the current system may be better than anything else. Also, in a decent number of states, it's mandatory for all HS students to take (and I believe free to students as a result).
sir0010010 4mo ago
No, but it'd probably be okay if the pilot flunked their SATs.
FatherOfCurses 4mo ago
It's not a claim, there is psychological data supporting the negative impact stress can have on recall and that for some students a test is not the best way to identify their retained knowledge. Your single example identifies a situation where we would want someone resilient to stress to pass the testing process. However if you are evaluating someone's ability to identify characters in Shakespeare who are most closely representative of the Bard himself, a proctored exam may not be the only environment where that could be demonstrated. Anyone in software knows as well that for a test to be effective it has to be written properly. In the history of academic and standardized testing however there has been little rigor in the construction of tests, and those who pass are ones who give the "right" answers regardless of whether those answers are true proofs of knowledge.
- mangodrunk 4mo ago
  That still doesn’t seem like a reasonable reason to not do tests. In the software analogy you have a unit test that passes but the actual software fails when used in the real world. What is the point of “knowledge” that can’t be demonstrated. How will that person demonstrate their knowledge of Shakespeare?
- WalterBright 4mo ago
  I've been around tests all my life. I don't know anyone who bombed the test but mastered the material. And the ones who aced the tests - turns out that they actually had mastered the material. As for people who cannot perform under the stress of a test, how are they going to perform otherwise? Anyhow, the solution to test anxiety is to keep taking tests - the anxiety will recede. For example, the first time I tried public speaking I was paralyzed. But I kept trying it, and the anxiety went away, and now I do a fair amount of public speaking and enjoy it very much.
mangodrunk 4mo ago
I agree with you. The real world is stressful or less than ideal for many reasons and we have to apply our knowledge within that context. In addition to that, this claim is like the dragon in my garage thought experiment. It an unfalsifiable claim that they have “knowledge” but can’t demonstrate it.

Maxatar 4mo ago

Not really comparable... the overwhelming majority of flight tests involve flying an aircraft. There is no meaningful way someone can be excellent at flying an aircraft but can't pass a test which involves flying an aircraft. The same can't be said for many other tests. If the test involves the practical application of the very skill being tested, then that test has direct relevance to he competency of said skill. But many other tests are not like that. A teacher can be brilliant in the classroom yet stumble on a standardized certification exam full of pedagogical jargon. A chef can cook a variety of excellent dishes but fail a written culinary theory exam testing the French names of techniques they perform by instinct. And perhaps more relevant to this audience, a coding interview that relies on whiteboarding algorithms from memory can easily fail an excellent engineer who builds great software every day but doesn't recall the optimal solution to some puzzle on the spot.

WalterBright 4mo ago
They are never going to let you into a cockpit until you pass ground school, which involves a lot of math. > A teacher can be brilliant in the classroom yet stumble on a standardized certification exam full of pedagogical jargon. A teacher that cannot explain how calculus works cannot teach it to anybody. > a coding interview that relies on whiteboarding algorithms from memory can easily fail an excellent engineer who builds great software every day but doesn't recall the optimal solution to some puzzle on the spot. I've seen too many coders using bubble sort because they don't know enough to look for a better algorithm. In any case, the purpose of leet coding tests is to quickly filter out the utter frauds. I have a programmer friend who wanted a job at a major software corp. He knew he'd have to pass the leetcode in an early stage of the interviewing. He figured it would take 6 weeks or so to study that material. I suggested that, since he was applying for a $250K job, that would be the most productive studying he'd ever done. He agreed, did the 6 weeks of studying, aced the leetcode test, and got the $250K. So ya, there is a point to those tests, in filtering out the frauds and the ones who aren't willing to do what it takes to get those jobs.
- Maxatar 4mo ago
  This has to be a joke... Ground school most certainly does not involve a lot of math, it's not like there's any calculus or algebra involved... it's basic arithmetic. Furthermore it's categorically false that you need to pass ground school before you're allowed to fly. Are you just making things up? >A teacher that cannot explain how calculus works cannot teach it to anybody. This is a strawman argument, I never made anything that could even remotely be interpreted as this. >I've seen too many coders using bubble sort because they don't know enough to look for a better algorithm. This is committing a very basic logical fallacy. The fact that someone who is incompetent likely can't pass a test is not the same claim as someone who can't pass a test is likely incompetent. Hopefully you are able to identify this logical mistake that you're committing and revise your position accordingly.
  - WalterBright 4mo ago
    > categorically false Google sez: "The U.S. Air Force strictly requires you to complete and pass formal academic ground training before you ever touch the controls of an aircraft" They're not going to risk an aircraft on an incompetent student. > A teacher can be brilliant in the classroom yet stumble on a standardized certification exam full of pedagogical jargon I stand by my statement. > logical fallacy A implies B meaning B implies A is indeed a logical fallacy. But that does not rule out B implies A. A and B can be strongly related to each other.
    - Maxatar 4mo ago
      So you are making things up... thanks for confirming that. While I appreciate that you reviewed what Google "sez"... you have misunderstood the relevant context which is that the U.S. Air Force also requires that you complete Initial Flight Training (IFT) before you start the Air Force's own formal training program (UPT). In IFT you will not be required to pass ground school before you get to fly. Furthermore, even if the Air Force did not require IFT before UPT (the Air Force's own training program), you've completely changed the nature of your argument. I have no dispute about whether the Air Force may or may not have stricter requirements for their pilots, but that wasn't your argument. >I stand by my statement. You've proudly planted your flag on a point nobody was contesting, which is a strange hill to celebrate on but you do you. >But that does not rule out B implies A. A and B can be strongly related to each other. Discussing a topic with someone who not only uses logically fallacies as justification for their argument but brazenly doubles down on said fallacy is a good sign that this is probably not a good discussion to continue spending time on. Like am I supposed to simply accept your logical fallacy and take on the burden of disproving every claim you can dream up simply because you've asserted it isn't logically impossible? The person making the claim carries the burden of supporting it, and "they're strongly related" is something you have to actually show, not something I'm obligated to refute on your behalf.
BobbyJo 4mo ago
The overwhelming majority of math tests involve doing math, so I'm not sure your critique is useful in this context.
- Maxatar 4mo ago
  Given that I'm not responding to any claim about the efficacy of math tests... it's actually your statement which is wholly irrelevant to this discussion.
  - BobbyJo 4mo ago
    "You can't compare academic tests and flight tests because flight tests involve flying" "Academic tests involve academics" "I'm not talking about academics!" That about sum it up?
    - Maxatar 4mo ago
      The submission isn't about academic tests involving academics... it's about the relevance of using such academic tests for the purpose of job screening.
      - lotsofpulp 4mo ago
        A business that wastes money on ineffective screening methods for employees will lose to a business hiring better employees.
        Maxatar 4mo ago
        Which is why almost no employers ask for SAT scores as a way of screening candidates, and why this article was written to showcase the specific reasons doing so is ineffective. Of course you'd have refrained from making such a vacuous reply if you bothered to read the article.
jimbokun 4mo ago
Is there a reason you left out the SAT and ACT? Because both have been shown to have predictive power for success in college.
- Maxatar 4mo ago
  Because I'm not trying to make a universal claim about all tests.
- halostatue 4mo ago
  For one semester. Maybe two.
fn-mote 4mo ago
The problem that colleges and the SWE profession in general face are identifying “bullshitters”. We need a filter and a fact based exam seems like a good place to start.

andrecarini 4mo ago

My understanding is you're equating `failing a test` to `lacking the relevant skills and knowledge to do a certain task competently`. The reality is sometimes tests in academia are just not very well made and don't really test what they are supposed to be testing, and that's usually due to multiple reasons like misaligned incentives, staffing shortages and maybe lack of resources / funding. I don't think the comparison to flight school is relevant enough in this context because it's a too different of a world to traditional academia.

WalterBright 4mo ago
I don't buy the notion that tests do not test relevant skills. In my long career I've noticed a strong correlation between SAT scores and academic performance as well as job performance. > I don't think the comparison to flight school is relevant enough in this context because it's a too different of a world to traditional academia. My dad kept his flight school tests for flying all sorts of airplanes. They bear a lot of similarities with the SATs. There's a lot of math in there for things like fuel consumption, wind, maximum landing weight, glide distance, and so on. For example, one day he was cruising along in his F-86 when the engine failed. he radioed the tower, and they told him to bail out. But he calculated his speed, altitude, distance, wind, sink rate, air templeratur, etc., and figured he could make the field after configuring the airplane for maximum glide. He made a perfect landing, but still got reprimanded for risking his life bringing the airplane back. But he had worked the math and disagreed that it was more risky to bring it in than bail out.
- hammock 4mo ago
  > I don't buy the notion that tests do not test relevant skills. In my long career I've noticed a strong correlation between SAT scores and academic performance as well as job performance. SAT tests intelligence (aptitude), not skills. Which is why it correlates with job performance, where intelligence can (over some time) matter as much or more than a starting point of relevant skills.
  - WalterBright 4mo ago

hammock 4mo ago

> Would you want a pilot on your flight who flunked flying school exams, but somehow "really knew how to fly! Sure, just not in the cockpit

chasd00 4mo ago

I would be much more ok with someone who failed the test but knew how to pilot a plane vs someone who aced the test but can’t figure out how to get the engine started.

WalterBright 4mo ago
One who cannot start the engine cannot kill me. One who cannot calculate how much fuel he needs to cross the lake will kill me. Remember JFKjr? He killed himself, his fiance and her friend because he did not pay attention to the instruments. An acquaintance of mine died trying to fly through a thunderstorm. Another one didn't pay attention to the weather and nearly died from wing icing. Flying is no joke.

Morromist 4mo ago

I did very well with my SAT but to get to the city we had to take it was a 2 hour long drive through twisted roads that made me carsick. I lived in a small town far away from the city. By the time I got there my breakfast I had quickly eaten gave me a stomach ache, I had woken up far earlier than usual and not gotten my 7 hours too. I certainly would have done a lot better if I had lived in the big city. Another factor: if you wanted to pay the fee you could just take the test over and over again until you got a great score. So kids with poor parents obviously had a huge disadvantage. Also kids who had the time and money could study for it with prep books - I did, while some of my friends were flipping burgers while still in highschool. Its not surprising I got a higher score than them, but it said nothing about my intellegence or understanding compaired to theirs.

apparent 4mo ago
There are fee waivers for poor kids, and free test prep thanks to Khan Academy. These may not have existed decades ago, but they exist now.
- Morromist 4mo ago
  Fee waivers are great. A lot of poor parents are too busy / don't have their shit together enough to learn about them & use them. A lot of poor parents won't even have their kids take the tests at all even if they are free, especially if it means taking an entire day to drive them to the test facility. They might not even know what an SAT test is.
  - apparent 4mo ago
    Hm that's true, but the alternative would also be flawed, and possibly worse. If you could only take the SAT once and you were stuck with that score, then people would complain that rich kids could take all sorts of prep courses to get ready and to figure out what they needed to work on before taking the test, whereas poor kids would be figuring this all out on test day. There is no ideal way to do this, and the current system may be better than anything else. Also, in a decent number of states, it's mandatory for all HS students to take (and I believe free to students as a result).

sir0010010 4mo ago

No, but it'd probably be okay if the pilot flunked their SATs.

FatherOfCurses 4mo ago

It's not a claim, there is psychological data supporting the negative impact stress can have on recall and that for some students a test is not the best way to identify their retained knowledge. Your single example identifies a situation where we would want someone resilient to stress to pass the testing process. However if you are evaluating someone's ability to identify characters in Shakespeare who are most closely representative of the Bard himself, a proctored exam may not be the only environment where that could be demonstrated. Anyone in software knows as well that for a test to be effective it has to be written properly. In the history of academic and standardized testing however there has been little rigor in the construction of tests, and those who pass are ones who give the "right" answers regardless of whether those answers are true proofs of knowledge.

mangodrunk 4mo ago
That still doesn’t seem like a reasonable reason to not do tests. In the software analogy you have a unit test that passes but the actual software fails when used in the real world. What is the point of “knowledge” that can’t be demonstrated. How will that person demonstrate their knowledge of Shakespeare?
WalterBright 4mo ago
I've been around tests all my life. I don't know anyone who bombed the test but mastered the material. And the ones who aced the tests - turns out that they actually had mastered the material. As for people who cannot perform under the stress of a test, how are they going to perform otherwise? Anyhow, the solution to test anxiety is to keep taking tests - the anxiety will recede. For example, the first time I tried public speaking I was paralyzed. But I kept trying it, and the anxiety went away, and now I do a fair amount of public speaking and enjoy it very much.

mangodrunk 4mo ago

I agree with you. The real world is stressful or less than ideal for many reasons and we have to apply our knowledge within that context. In addition to that, this claim is like the dragon in my garage thought experiment. It an unfalsifiable claim that they have “knowledge” but can’t demonstrate it.