6 ms·

AI can beat real university students in exams, study suggests

20 points by fredley 2y ago

RecycledEle 2y ago
Yes, the current crop of world knowledge AIs are smarter than any human who ever lived. And big names are calling them useless. This is proof the human race is not generally capable to solving novel problems, so I hope people will stop expecting AIs to solve every novel problem.
squircle 2y ago
Ohno.
Yawrehto 2y ago
Lesson learned, don't go into psychology.
meristohm 2y ago
Can "educated machines" reproduce on their own yet? How do they fit into the food web, the carbon cycle, the nitrogen cycle, etc? Can they make meaning, towards a purpose in life? What role do they serve other than human ingenuity ego-stroking and for a few to further extract money from the many? Can AI love, yet? To what degree are we just avoiding dealing with existential threats by churning through resources to play god and make robots in our image? (Albeit a subset of humanity, and not without bias) I'm not yet convinced this AI work isn't a waste of time and other resources. I'd far rather we put our efforts into land/water stewardship and a "new" vision for human existence based on many of the old ways that got us this far, so that we might go another several hundred thousand years. In an unbroken oral tradition, what stories might those future people tell about this time?
- falcor84 2y ago
  >Can "educated machines" reproduce on their own yet? How do they fit into the food web, the carbon cycle, the nitrogen cycle, etc? Can they make meaning, towards a purpose in life? What role do they serve other than human ingenuity ego-stroking and for a few to further extract money from the many? Can AI love, yet? I'm sorry to say that to me all these questions sound as semantically meaningful as "can submarines swim?"
- daseiner1 2y ago
  Though I am admittedly an AI pessimist, as regards your penultimate paragraph, I think AI evangelists would argue that the aforementioned technology may offer a pathway to the new paradigm that you allude to. What consistently goes unremarked upon, however, is the massive political “realignment” (a polite term for revolution) that the proliferation of AI may lead to in the not-so distant future. Disruption and likely violence that may, at best, lead to such a “new vision” for human existence.
- fooker 2y ago
  I don't know what AI can or can not do, but it's getting good enough that I can't tell if this is written by ChatGPT. If you can't pass the Turing test, setting higher bars for AI doesn't make sense.
bee_rider 2y ago
Worrying that AI might make exams obsolete is kind of odd, I mean, it is a symptom I guess but only at the very end of a long stupid cascading failure. Students cheat because they want the degree but don’t care to learn the material. Or maybe they want to learn the material, but see employment at the end as requiring better grades than they can get naturally. Either one is the result of bullshit credentialism. (Bullshit credentialism probably comes in part as a result of bullshit jobs where work-product can’t be evaluated because it’s all useless). Hopefully students manage to cheat on so many tests that grades can become completely useless for employers. Then, they can become something useful for the students, a way to evaluate their progress and get feedback.
- prrrywinkle 2y ago
  [dead]
- deleted 2y ago
  [deleted]
- threatofrain 2y ago
  > Then, they can become something useful for the students, a way to evaluate their progress and get feedback. That's exactly why it'll become useful to employers. The degree to which student evaluations are frank assessments is the degree to which employers will value them even more. The more honest schools are the higher stakes they become. If you want to lower the stakes then increase the number of nice seats in the economy.
- nradov 2y ago
  Credentialism isn't really bullshit. For any decent entry-level job there will be an enormous surplus of applicants (elite overproduction). It isn't practical to evaluate them all in detail, the hiring managers simply don't have time. So they fall back on simple, somewhat arbitrary factors like educational degrees, school rankings, and GPA. It's imperfect but what else can work at scale?
  - deleted 2y ago
    [deleted]
quantum_state 2y ago
This says more about the exams than the AI or the students…
- shinycode 2y ago
  For a time teachers resisted calculators because they learned the hard way. But learning with a calculator doesn’t mean we suck at math, we can still learn what’s behind it. AI have to be a tool for learning better and faster and expand our reasoning abilities. It’s of no sense to be stuck in the past and resist it. We have to develop a more critical thinking, use AI in schools and criticize the output and make something better out of it. And we have to change the way knowledge is assessed. Teachers have to work hard to make better exams. I hope it will destroy the stupid tests where students have to learn like parrots for a set date then forget everything after. It’s so stupid.
- ziofill 2y ago
  I agree 100%. This AI wave hopefully if will force universities to rethink the way students are assessed. When I was faculty I had the luxury to teach small classes, and I could always be extremely flexible with assessments so I could assess students as the course progresses, or I could make it project-based, or a 1-on-1 oral exam. But I had colleagues that were teaching a class of 300 students, which meant most of the exams were looked at only by teaching assistants, or they were so heavily automated (e.g. multiple choice) that these days they could totally be fooled by an LLM. Something needs to change and it will (fingers crossed).
- tzs 2y ago
  Not really. The point of an exam is to assess whether or not the student has learned the material. That an LLM can answer all the questions in 2024 is not really all that much different from how it was circa 1980 when probably every integral on my calculus exam was in a table of integrals that I could buy at the college bookstore. It's just one more thing to add to the list of things that you need to prevent students from using during the exam. In 1980 that included that table of integrals on calculus exam. In 2024 it needs to include LLMs.
a_bonobo 2y ago
This fits well with what we know about AI and Bloom's Taxonomy of Learning, which goes from 'remembering' on the lowest step to 'creating' on the top step (remember, understand, apply, analyse, evaluate, create). Undergrad exams are usually somewhere around 'remembering', simple fact or definition regurgitation. Most of these facts should be in chatGPT's training data. As the degree proceeds things get harder and we move up the taxonomy, and that's where we know LLMs fail: there's nothing in there that can really 'understand', let alone 'create'.
threecheese 2y ago
This is just a way more reactionary way to communicate model benchmarks. “A computer algorithm performed better than humans on a task it was designed for” sounds like the last forty years in a nutshell.
- falcor84 2y ago
  The big question remains - "Is there any benchmark at all on which algorithms can't pass human performance given sufficient resources?". If I were a being man, I would put my money on 'no'
  - RoyalHenOil 2y ago
    "Given sufficient resources" is the key phrase here. Humans are ridiculously energy efficient. We are capable of extremely complex thought — as well as a range of physical movement! — while using less energy than most household appliances. I am skeptical that we will ever create non-biological AI that can compete with that.
paxys 2y ago
Next you will tell me that a calculator can beat students at addition and subtraction.

20 points by fredley 2y ago

Yes, the current crop of world knowledge AIs are smarter than any human who ever lived. And big names are calling them useless. This is proof the human race is not generally capable to solving novel problems, so I hope people will stop expecting AIs to solve every novel problem.

squircle 2y ago

Ohno.

Yawrehto 2y ago

Lesson learned, don't go into psychology.

meristohm 2y ago

Can "educated machines" reproduce on their own yet? How do they fit into the food web, the carbon cycle, the nitrogen cycle, etc? Can they make meaning, towards a purpose in life? What role do they serve other than human ingenuity ego-stroking and for a few to further extract money from the many? Can AI love, yet? To what degree are we just avoiding dealing with existential threats by churning through resources to play god and make robots in our image? (Albeit a subset of humanity, and not without bias) I'm not yet convinced this AI work isn't a waste of time and other resources. I'd far rather we put our efforts into land/water stewardship and a "new" vision for human existence based on many of the old ways that got us this far, so that we might go another several hundred thousand years. In an unbroken oral tradition, what stories might those future people tell about this time?

falcor84 2y ago
>Can "educated machines" reproduce on their own yet? How do they fit into the food web, the carbon cycle, the nitrogen cycle, etc? Can they make meaning, towards a purpose in life? What role do they serve other than human ingenuity ego-stroking and for a few to further extract money from the many? Can AI love, yet? I'm sorry to say that to me all these questions sound as semantically meaningful as "can submarines swim?"
daseiner1 2y ago
Though I am admittedly an AI pessimist, as regards your penultimate paragraph, I think AI evangelists would argue that the aforementioned technology may offer a pathway to the new paradigm that you allude to. What consistently goes unremarked upon, however, is the massive political “realignment” (a polite term for revolution) that the proliferation of AI may lead to in the not-so distant future. Disruption and likely violence that may, at best, lead to such a “new vision” for human existence.
fooker 2y ago
I don't know what AI can or can not do, but it's getting good enough that I can't tell if this is written by ChatGPT. If you can't pass the Turing test, setting higher bars for AI doesn't make sense.

bee_rider 2y ago

Worrying that AI might make exams obsolete is kind of odd, I mean, it is a symptom I guess but only at the very end of a long stupid cascading failure. Students cheat because they want the degree but don’t care to learn the material. Or maybe they want to learn the material, but see employment at the end as requiring better grades than they can get naturally. Either one is the result of bullshit credentialism. (Bullshit credentialism probably comes in part as a result of bullshit jobs where work-product can’t be evaluated because it’s all useless). Hopefully students manage to cheat on so many tests that grades can become completely useless for employers. Then, they can become something useful for the students, a way to evaluate their progress and get feedback.

prrrywinkle 2y ago
[dead]
deleted 2y ago
[deleted]
threatofrain 2y ago
> Then, they can become something useful for the students, a way to evaluate their progress and get feedback. That's exactly why it'll become useful to employers. The degree to which student evaluations are frank assessments is the degree to which employers will value them even more. The more honest schools are the higher stakes they become. If you want to lower the stakes then increase the number of nice seats in the economy.
nradov 2y ago
Credentialism isn't really bullshit. For any decent entry-level job there will be an enormous surplus of applicants (elite overproduction). It isn't practical to evaluate them all in detail, the hiring managers simply don't have time. So they fall back on simple, somewhat arbitrary factors like educational degrees, school rankings, and GPA. It's imperfect but what else can work at scale?
- deleted 2y ago
  [deleted]

quantum_state 2y ago

This says more about the exams than the AI or the students…

shinycode 2y ago
For a time teachers resisted calculators because they learned the hard way. But learning with a calculator doesn’t mean we suck at math, we can still learn what’s behind it. AI have to be a tool for learning better and faster and expand our reasoning abilities. It’s of no sense to be stuck in the past and resist it. We have to develop a more critical thinking, use AI in schools and criticize the output and make something better out of it. And we have to change the way knowledge is assessed. Teachers have to work hard to make better exams. I hope it will destroy the stupid tests where students have to learn like parrots for a set date then forget everything after. It’s so stupid.
ziofill 2y ago
I agree 100%. This AI wave hopefully if will force universities to rethink the way students are assessed. When I was faculty I had the luxury to teach small classes, and I could always be extremely flexible with assessments so I could assess students as the course progresses, or I could make it project-based, or a 1-on-1 oral exam. But I had colleagues that were teaching a class of 300 students, which meant most of the exams were looked at only by teaching assistants, or they were so heavily automated (e.g. multiple choice) that these days they could totally be fooled by an LLM. Something needs to change and it will (fingers crossed).
tzs 2y ago
Not really. The point of an exam is to assess whether or not the student has learned the material. That an LLM can answer all the questions in 2024 is not really all that much different from how it was circa 1980 when probably every integral on my calculus exam was in a table of integrals that I could buy at the college bookstore. It's just one more thing to add to the list of things that you need to prevent students from using during the exam. In 1980 that included that table of integrals on calculus exam. In 2024 it needs to include LLMs.

a_bonobo 2y ago

This fits well with what we know about AI and Bloom's Taxonomy of Learning, which goes from 'remembering' on the lowest step to 'creating' on the top step (remember, understand, apply, analyse, evaluate, create). Undergrad exams are usually somewhere around 'remembering', simple fact or definition regurgitation. Most of these facts should be in chatGPT's training data. As the degree proceeds things get harder and we move up the taxonomy, and that's where we know LLMs fail: there's nothing in there that can really 'understand', let alone 'create'.

threecheese 2y ago

This is just a way more reactionary way to communicate model benchmarks. “A computer algorithm performed better than humans on a task it was designed for” sounds like the last forty years in a nutshell.

falcor84 2y ago
The big question remains - "Is there any benchmark at all on which algorithms can't pass human performance given sufficient resources?". If I were a being man, I would put my money on 'no'
- RoyalHenOil 2y ago
  "Given sufficient resources" is the key phrase here. Humans are ridiculously energy efficient. We are capable of extremely complex thought — as well as a range of physical movement! — while using less energy than most household appliances. I am skeptical that we will ever create non-biological AI that can compete with that.

paxys 2y ago

Next you will tell me that a calculator can beat students at addition and subtraction.