4 ms·

> Individual humans can still easily beat it when it comes to knowledge of individual subjects. What does a phrase like "GPT-4 scores 90th percentile on the Un

> Individual humans can still easily beat it when it comes to knowledge of individual subjects. What does a phrase like "GPT-4 scores 90th percentile on the Uniform Bar Exam" mean to you, regarding whether humans can easily surpass its knowledge and reasoning? https://www.forbes.com/sites/johnkoetsier/2023/03/14/gpt-4-beats-90-of-lawyers-trying-to-pass-the-bar/ https://www.forbes.com/sites/johnkoetsier/2023/03/14/gpt-4-b...

dragonwriter 3y ago
> What does a phrase like "GPT-4 scores 90th percentile on the Uniform Bar Exam" mean to you, regarding whether humans can easily surpass its knowledge and reasoning? Absolutely nothing, because of construct validity. Those tests measure things that have shown to correlate with abilities of concern in humans, and so are, for their purposes, valid for humans. This hasn’t been demonstrated for LLMs, and the assumption that construct validity can be assumed without being established is begging the question: it is presuming not only that LLMs are general intelligences, but thaf they are general intelligences structurally similar to human intelligences such that the proxy measures for cognitive capacities work similarly.
- comp_throw7 3y ago
  Construct validity! I suppose, when GPT-4 writes correctly working code that does what you want on the first try, this says absolutely nothing about its cognitive capacity, because, after all, it's just a proxy measurement for the underlying generative process. (Yes, obviously the cognition is _different_ from what happens in humans. That does not mean that... it isn't intelligence?)
  - dragonwriter 3y ago
    > I suppose, when GPT-4 writes correctly working code that does what you want on the first try, this says absolutely nothing about its cognitive capacity It says something about its ability to write code. Beyond that... its impossible to say. We simply don’t have the information about generative AI models to be able to generalize from limited proxies about them; psychometry is not transferrable from humans to them — or at least, we have neither evidence nor a strong theoretical reason to think it should be.

dragonwriter 3y ago

> What does a phrase like "GPT-4 scores 90th percentile on the Uniform Bar Exam" mean to you, regarding whether humans can easily surpass its knowledge and reasoning? Absolutely nothing, because of construct validity. Those tests measure things that have shown to correlate with abilities of concern in humans, and so are, for their purposes, valid for humans. This hasn’t been demonstrated for LLMs, and the assumption that construct validity can be assumed without being established is begging the question: it is presuming not only that LLMs are general intelligences, but thaf they are general intelligences structurally similar to human intelligences such that the proxy measures for cognitive capacities work similarly.

comp_throw7 3y ago
Construct validity! I suppose, when GPT-4 writes correctly working code that does what you want on the first try, this says absolutely nothing about its cognitive capacity, because, after all, it's just a proxy measurement for the underlying generative process. (Yes, obviously the cognition is _different_ from what happens in humans. That does not mean that... it isn't intelligence?)
- dragonwriter 3y ago
  > I suppose, when GPT-4 writes correctly working code that does what you want on the first try, this says absolutely nothing about its cognitive capacity It says something about its ability to write code. Beyond that... its impossible to say. We simply don’t have the information about generative AI models to be able to generalize from limited proxies about them; psychometry is not transferrable from humans to them — or at least, we have neither evidence nor a strong theoretical reason to think it should be.