6 ms·

I think this is evidence against the usefulness of standardized tests rather than support for ChatGPT

rayiner 4y ago
Why? This section is literally just questions about a modestly complex reading passage. It’s the most real world part of the SAT.
- tshaddox 4y ago
  Also you’re not allowed to use any reference materials and certainly not the Internet when you’re taking these tests.
shaunsingh 4y ago
What I find more concerning is the 100% scores that ChatGPT has been able to achieve on AP CSA and APUSH, among others. It makes sense to teach 3rd graders 2+2, but if college board is creating "college level courses" with problems that can be easily solved through whats still a fairly new technology, we need to reevaluate what we're teaching in schools. So many AP courses are just memorization and identifying formulas. I'd bet the majority of AP CSA students wouldn't fare very well when tasked with detailing how they'd approach a project, whereas kids in IB schools etc. would find it straightforward.
- zamadatix 4y ago
  I don't find it at all concerning that e.g. a history course is primarily tested and easily passed by memorization. There's nothing wrong with teaching context instead of pure skills all the time IMO. I do agree it'd be good to see deeper computer science as an early option in the US but at the same time I'm not sure whether or not something like ChatGPT can pass is really a good indicator for why.
- fdr 4y ago
  The AP CSA pass surprises me (I've never taken it myself). I had this thought because I tried to use chatgpt for work programming stuff and it got stuff wrong very frequently. So frequently, and to a great enough degree, that I had to keep re-circumscribing what kind of problems were worth my time to throw at it, and now it's a tiny cross-section.
  - shaunsingh0207 4y ago
    Thats what surprises me as well. If I give chatGPT anything that requires actual thought or reasoning put into it, it just produces gibberish. It's great for getting a blueprint to start from. I recently had to write a gtk+rust client, and I asked chatGPT to give me some starter code which largely worked. All the chatGPT hype feels the same to me as the copilot hype from a while back. The people who got the most excited about it are the same people who its going to replace.
    - fdr 4y ago
      That does seem like a good application for it. If one thinks something is in the corpus many times (various GTK idioms), chatgpt is pretty good at a type of syntactic composition...e.g., "can you show it to me in rust"
- fifilura 4y ago
  There is one argument against this though that is somehow brewing in my brain. The end goal is to internalize the subject so that you can extrapolate your own insights. "This happened in WWII so if these conditions starts to appear it risks to happen again". Just to take history as an example. All other subjects have similar examples. And to reach that kind of insight you need to start filling the brain with information. That may be called memorization. The memorization is just one path of the road towards the black belt. If you skip it you will be lost.
  - tgv 4y ago
    Yes, but ... what this shows is that testing for "memorization" isn't good enough. We all know people who have learned information and can give answers with confidence, even though they are just winging it, to put it politely. ChatGPT could be considered their equal on such tasks. So schools need to test that students are capable of working with the acquired information in a way that befits the level of the course, just like they need to find a way to lose their addication to the lazy "write an essay" assignment. If not, education will lose its status.
    - fifilura 4y ago
      The question is if we should expect these test to test on the black belt level or at a lower level belt? (i am not native English speaking so i don't know what addication means, I couldn't quite understand that sentence)
      - throwaway8689 4y ago
        Seems to be a typo for Addiction
      - tgv 4y ago
        Sorry, that was a typo for addiction indeed. They should test at the level they claim to certify. So, an MSc. should not get questions that can be answered by rote, but it might be appropriate for a first-year student, when the topic is entirely new. And teachers should read essays for depth and understanding, instead of counting the words and skimming a few paragraphs. That'll draw complaints from the people that think every end goal and test of a course should be perfectly described and objective, but to them I say: move your pretty, empty heads elsewhere if you can't see you're parroting someone's failed attempt to operationalize a remote ideal.
- thaumasiotes 4y ago
  > if college board is creating "college level courses" with problems that can be easily solved through whats still a fairly new technology, we need to reevaluate what we're teaching in schools. I don't get it. AP calculus is problems that can be easily solved with what is now very old technology. The point of an AP test isn't that the problems are so difficult.
  - shaunsingh0207 4y ago
    These specific subjects shouldn't be solely about memorization, rather they should teach kids how to link that information together to solve problems (or in the case of history, avoid them in the future). Thats the issue. The point of an AP test shouldn't be that the problems are difficult. But currently thats what it's all about. They're just trying to test that kids are capable of surviving a "rigorous academic curriculum". [1] https://apcentral.collegeboard.org/about-ap/ap-a-glance https://apcentral.collegeboard.org/about-ap/ap-a-glance
kerpotgh 4y ago
[dead]
shreyshnaccount 4y ago
There is a leap in this logic- going from the fact that SAT is solvable by a LLM to the conclusion that the SAT is useless. Questions about addition can be solved using calculators, does that mean that teaching addition is useless? Also there is an underlying assumption that memorization is a useless skill, which I'd argue against, as in today's world most innovation is done by identifying patterns between disciplines, and memorization is an important part of recognising patterns
- ekianjo 4y ago
  not so much memorization as decomposition of numerous examples into logic based knowledge. As a simple example, knowing volumes of words is not as useful as knowing etymology since etymology brings you a deeper structure.
  - shreyshnaccount 4y ago
    I'd say that's subjective, depending on context and individuals. Sure etymology might be amazing to know for a lit major or author, but for someone who wants to just be able to write clearly, knowing a lot of words is useful.
- hgsgm 4y ago
  Well it shows that you can do well on SAT with basic word association without understanding and explaining the ideas of the passage. The "Joe Bloggs" theory has known this for ages.
  - shreyshnaccount 4y ago
    Yeah but that also says nothing about how useful or useless that is
- sublinear 4y ago
  You're correct that memorization and pattern recognition are useful, but they're not supposed to be the majority of the test! A good test should also include reasoning skills and comprehension. Having reasoning skills means not all necessary information can be derived from the test itself, the necessary information isn't always well defined, yet a correct answer is produced. Demonstrating comprehension requires the test taker to write out a model of what they think they know in their own words. Undoubtedly the answer will be at least slightly wrong which is why subjective expertise is required to evaluate the level of comprehension. To me it seems the quality of the tests has deteriorated due to cost cutting, not because AI has become so much more sophisticated. Subjectivity is usually dismissed these days as a roll of the dice, but it's vital to fitting the real world full of humans living in it. Humans must be the gatekeepers, not to fight off AI, but by the very definition of what it means to be convincing. Answers to real world questions solve many more constraints than just the concrete subject matter. The whole point to updating a test on a well understood topic should not merely be to prevent "replay attack" cheating, but to conform to the new subjectivity on the topic.
concordDance 4y ago
These particular tests maybe. I'd want to see how it does on the ones from a few decades ago.

Why? This section is literally just questions about a modestly complex reading passage. It’s the most real world part of the SAT.

tshaddox 4y ago
Also you’re not allowed to use any reference materials and certainly not the Internet when you’re taking these tests.

shaunsingh 4y ago

What I find more concerning is the 100% scores that ChatGPT has been able to achieve on AP CSA and APUSH, among others. It makes sense to teach 3rd graders 2+2, but if college board is creating "college level courses" with problems that can be easily solved through whats still a fairly new technology, we need to reevaluate what we're teaching in schools. So many AP courses are just memorization and identifying formulas. I'd bet the majority of AP CSA students wouldn't fare very well when tasked with detailing how they'd approach a project, whereas kids in IB schools etc. would find it straightforward.

zamadatix 4y ago
I don't find it at all concerning that e.g. a history course is primarily tested and easily passed by memorization. There's nothing wrong with teaching context instead of pure skills all the time IMO. I do agree it'd be good to see deeper computer science as an early option in the US but at the same time I'm not sure whether or not something like ChatGPT can pass is really a good indicator for why.
fdr 4y ago
The AP CSA pass surprises me (I've never taken it myself). I had this thought because I tried to use chatgpt for work programming stuff and it got stuff wrong very frequently. So frequently, and to a great enough degree, that I had to keep re-circumscribing what kind of problems were worth my time to throw at it, and now it's a tiny cross-section.
- shaunsingh0207 4y ago
  Thats what surprises me as well. If I give chatGPT anything that requires actual thought or reasoning put into it, it just produces gibberish. It's great for getting a blueprint to start from. I recently had to write a gtk+rust client, and I asked chatGPT to give me some starter code which largely worked. All the chatGPT hype feels the same to me as the copilot hype from a while back. The people who got the most excited about it are the same people who its going to replace.
  - fdr 4y ago
    That does seem like a good application for it. If one thinks something is in the corpus many times (various GTK idioms), chatgpt is pretty good at a type of syntactic composition...e.g., "can you show it to me in rust"
fifilura 4y ago
There is one argument against this though that is somehow brewing in my brain. The end goal is to internalize the subject so that you can extrapolate your own insights. "This happened in WWII so if these conditions starts to appear it risks to happen again". Just to take history as an example. All other subjects have similar examples. And to reach that kind of insight you need to start filling the brain with information. That may be called memorization. The memorization is just one path of the road towards the black belt. If you skip it you will be lost.
- tgv 4y ago
  Yes, but ... what this shows is that testing for "memorization" isn't good enough. We all know people who have learned information and can give answers with confidence, even though they are just winging it, to put it politely. ChatGPT could be considered their equal on such tasks. So schools need to test that students are capable of working with the acquired information in a way that befits the level of the course, just like they need to find a way to lose their addication to the lazy "write an essay" assignment. If not, education will lose its status.
  - fifilura 4y ago
    The question is if we should expect these test to test on the black belt level or at a lower level belt? (i am not native English speaking so i don't know what addication means, I couldn't quite understand that sentence)
    - throwaway8689 4y ago
      Seems to be a typo for Addiction
    - tgv 4y ago
      Sorry, that was a typo for addiction indeed. They should test at the level they claim to certify. So, an MSc. should not get questions that can be answered by rote, but it might be appropriate for a first-year student, when the topic is entirely new. And teachers should read essays for depth and understanding, instead of counting the words and skimming a few paragraphs. That'll draw complaints from the people that think every end goal and test of a course should be perfectly described and objective, but to them I say: move your pretty, empty heads elsewhere if you can't see you're parroting someone's failed attempt to operationalize a remote ideal.
thaumasiotes 4y ago
> if college board is creating "college level courses" with problems that can be easily solved through whats still a fairly new technology, we need to reevaluate what we're teaching in schools. I don't get it. AP calculus is problems that can be easily solved with what is now very old technology. The point of an AP test isn't that the problems are so difficult.
- shaunsingh0207 4y ago
  These specific subjects shouldn't be solely about memorization, rather they should teach kids how to link that information together to solve problems (or in the case of history, avoid them in the future). Thats the issue. The point of an AP test shouldn't be that the problems are difficult. But currently thats what it's all about. They're just trying to test that kids are capable of surviving a "rigorous academic curriculum". [1] https://apcentral.collegeboard.org/about-ap/ap-a-glance https://apcentral.collegeboard.org/about-ap/ap-a-glance

kerpotgh 4y ago

[dead]

shreyshnaccount 4y ago

There is a leap in this logic- going from the fact that SAT is solvable by a LLM to the conclusion that the SAT is useless. Questions about addition can be solved using calculators, does that mean that teaching addition is useless? Also there is an underlying assumption that memorization is a useless skill, which I'd argue against, as in today's world most innovation is done by identifying patterns between disciplines, and memorization is an important part of recognising patterns

ekianjo 4y ago
not so much memorization as decomposition of numerous examples into logic based knowledge. As a simple example, knowing volumes of words is not as useful as knowing etymology since etymology brings you a deeper structure.
- shreyshnaccount 4y ago
  I'd say that's subjective, depending on context and individuals. Sure etymology might be amazing to know for a lit major or author, but for someone who wants to just be able to write clearly, knowing a lot of words is useful.
hgsgm 4y ago
Well it shows that you can do well on SAT with basic word association without understanding and explaining the ideas of the passage. The "Joe Bloggs" theory has known this for ages.
- shreyshnaccount 4y ago
  Yeah but that also says nothing about how useful or useless that is
sublinear 4y ago
You're correct that memorization and pattern recognition are useful, but they're not supposed to be the majority of the test! A good test should also include reasoning skills and comprehension. Having reasoning skills means not all necessary information can be derived from the test itself, the necessary information isn't always well defined, yet a correct answer is produced. Demonstrating comprehension requires the test taker to write out a model of what they think they know in their own words. Undoubtedly the answer will be at least slightly wrong which is why subjective expertise is required to evaluate the level of comprehension. To me it seems the quality of the tests has deteriorated due to cost cutting, not because AI has become so much more sophisticated. Subjectivity is usually dismissed these days as a roll of the dice, but it's vital to fitting the real world full of humans living in it. Humans must be the gatekeepers, not to fight off AI, but by the very definition of what it means to be convincing. Answers to real world questions solve many more constraints than just the concrete subject matter. The whole point to updating a test on a well understood topic should not merely be to prevent "replay attack" cheating, but to conform to the new subjectivity on the topic.

concordDance 4y ago

These particular tests maybe. I'd want to see how it does on the ones from a few decades ago.