7 ms·

> The gap between an LLM and a "human with below average intelligence" is far more than 10x. In which direction? GPT-4 passes the bar exam with a top 10% scor

by p-e-w 3y ago

> The gap between an LLM and a "human with below average intelligence" is far more than 10x. In which direction? GPT-4 passes the bar exam with a top 10% score. How do you think a human with below average intelligence (or even with average intelligence) would fare? Copilot generates programming code that solves problems, and in most cases the code is correct. It outperforms many junior professional developers. Do you think a human with below average intelligence could do that?

intelVISA 3y ago
> Copilot generates programming code that solves problems, and in most cases the code is correct. It outperforms many junior professional developers. Do you think a human with below average intelligence could do that? Absolutely, it's why Javascript is so popular.
- p-e-w 3y ago
  A human with below average intelligence can outperform many junior professional software engineers because of JavaScript? Do you mean a programmer with below average intelligence (for a programmer)? Because I'm having a hard time believing that you actually believe what you wrote. BTW, JavaScript is actually a fairly difficult language to learn and use. Python, BASIC, and even Fortran are much simpler conceptually and have far fewer pitfalls. JavaScript is popular because it's the only language that every modern computer has an interpreter for, not because it's so easy that idiots can use it.
  - arcanemachiner 3y ago
    It's gotta be a low-effort joke.
  - intelVISA 3y ago
    It was a bit tongue in cheek I'll confess. :)
nemothekid 3y ago
>How do you think a human with below average intelligence (or even with average intelligence) would fare? I don't understand what rote memorization to pass a test has to do with intelligence. For the record Kim Kardashian passed the bar; I imagine anyone given the proper motivation and time to study could do it, it's not a hard test. If AGI is a computer passing a test, then AGI was achieved a long time ago. I don't think a human with below average intelligence could multiple 2 very large primes but I don't mistake my calculator for intelligence. A below average intelligence human can drive a car with a couple hours of training, an LLM can't do that (with a far lower power budget as well). I'm not saying AGI is impossible, but it's clear LLMs are not AGIs; it's not a matter of having a 100x more powerful LLM, just like making an Ape better at sign language won't make them better at abstract reasoning. An Ape's brain fundamentally lacks the mental machinery for higher level things that humans do. It's not a question of not being smart enough. LLMs are simply one component of the human mind.
- p-e-w 3y ago
  > If AGI is a computer passing a test, then AGI was achieved a long time ago. AIs couldn't even pass a third-grade reading comprehension exam until about 5 years ago. Computers being able to pass tests designed for humans is a very new thing. > it's clear LLMs are not AGIs And the main argument for that is that "it's clear". They're beating lawyers, doctors, and software engineers, but obviously, that's not real intelligence...
  - qup 3y ago
    > Computers being able to pass tests designed for humans is a very new thing. Yes! Captchas so effective!
    - comex 3y ago
      Yes, it would be more accurate to say “computers being able to pass tests designed for humans by humans”, as opposed to CAPTCHAs which are automatically generated tests.
  - 3y ago
satellite2 3y ago
Maybe we are simply culturally inclined to believe that the bar exam is more sophisticated than say, learn how to drive. A lot of the compute power required to drive is simply hidden from us because it's inconsious. While intellectual tasks are mostly done consciously. So it remains to be shown which task is actually more complex. And in the meantime it's clear than most people, independantly of their academic background can learn how to drive relatively easily.
dontupvoteme 3y ago
it's controversial but they're not allowed to sue us for it -- being a lawyer or doctor just requires a lot of rote memorization.
pmoriarty 3y ago
"GPT-4 passes the bar exam with a top 10% score." Turns out that may be as much marketing as truth. According to this paper[1]: "although GPT-4's UBE score nears the 90th percentile when examining approximate conversions from February administrations of the Illinois Bar Exam, these estimates are heavily skewed towards repeat test-takers who failed the July administration and score significantly lower than the general test-taking population. Second, data from a recent July administration of the same exam suggests GPT-4's overall UBE percentile was ~68th percentile, and ~48th percentile on essays. Third, examining official NCBE data and using several conservative statistical assumptions, GPT-4's performance against first-time test takers is estimated to be ~63rd percentile, including ~41st percentile on essays. Fourth, when examining only those who passed the exam (i.e. licensed or license-pending attorneys), GPT-4's performance is estimated to drop to ~48th percentile overall, and ~15th percentile on essays." [1] - https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4441311 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4441311
TimPC 3y ago
Sure but I can find examples of this in both directions. GPT-4 fails spectacularly at any task that doesn't have very precisely sanitized inputs presented to it in a certain format. It also fails at any task requiring any sort of interaction with the world. I think criticisms that it isn't a general intelligence are entirely fair even if it seems a bit more like general intelligence than most things that came before.
gentleman11 3y ago
> GPT-4 passes the bar exam with a top 10% score There was an article on HN yesterday debunking that. OpenAI is prone to exaggerating

by p-e-w 3y ago

intelVISA 3y ago

> Copilot generates programming code that solves problems, and in most cases the code is correct. It outperforms many junior professional developers. Do you think a human with below average intelligence could do that? Absolutely, it's why Javascript is so popular.

p-e-w 3y ago
A human with below average intelligence can outperform many junior professional software engineers because of JavaScript? Do you mean a programmer with below average intelligence (for a programmer)? Because I'm having a hard time believing that you actually believe what you wrote. BTW, JavaScript is actually a fairly difficult language to learn and use. Python, BASIC, and even Fortran are much simpler conceptually and have far fewer pitfalls. JavaScript is popular because it's the only language that every modern computer has an interpreter for, not because it's so easy that idiots can use it.
- arcanemachiner 3y ago
  It's gotta be a low-effort joke.
- intelVISA 3y ago
  It was a bit tongue in cheek I'll confess. :)

nemothekid 3y ago

>How do you think a human with below average intelligence (or even with average intelligence) would fare? I don't understand what rote memorization to pass a test has to do with intelligence. For the record Kim Kardashian passed the bar; I imagine anyone given the proper motivation and time to study could do it, it's not a hard test. If AGI is a computer passing a test, then AGI was achieved a long time ago. I don't think a human with below average intelligence could multiple 2 very large primes but I don't mistake my calculator for intelligence. A below average intelligence human can drive a car with a couple hours of training, an LLM can't do that (with a far lower power budget as well). I'm not saying AGI is impossible, but it's clear LLMs are not AGIs; it's not a matter of having a 100x more powerful LLM, just like making an Ape better at sign language won't make them better at abstract reasoning. An Ape's brain fundamentally lacks the mental machinery for higher level things that humans do. It's not a question of not being smart enough. LLMs are simply one component of the human mind.

p-e-w 3y ago
> If AGI is a computer passing a test, then AGI was achieved a long time ago. AIs couldn't even pass a third-grade reading comprehension exam until about 5 years ago. Computers being able to pass tests designed for humans is a very new thing. > it's clear LLMs are not AGIs And the main argument for that is that "it's clear". They're beating lawyers, doctors, and software engineers, but obviously, that's not real intelligence...
- qup 3y ago
  > Computers being able to pass tests designed for humans is a very new thing. Yes! Captchas so effective!
  - comex 3y ago
    Yes, it would be more accurate to say “computers being able to pass tests designed for humans by humans”, as opposed to CAPTCHAs which are automatically generated tests.
- 3y ago

satellite2 3y ago

Maybe we are simply culturally inclined to believe that the bar exam is more sophisticated than say, learn how to drive. A lot of the compute power required to drive is simply hidden from us because it's inconsious. While intellectual tasks are mostly done consciously. So it remains to be shown which task is actually more complex. And in the meantime it's clear than most people, independantly of their academic background can learn how to drive relatively easily.

dontupvoteme 3y ago

it's controversial but they're not allowed to sue us for it -- being a lawyer or doctor just requires a lot of rote memorization.

pmoriarty 3y ago

"GPT-4 passes the bar exam with a top 10% score." Turns out that may be as much marketing as truth. According to this paper[1]: "although GPT-4's UBE score nears the 90th percentile when examining approximate conversions from February administrations of the Illinois Bar Exam, these estimates are heavily skewed towards repeat test-takers who failed the July administration and score significantly lower than the general test-taking population. Second, data from a recent July administration of the same exam suggests GPT-4's overall UBE percentile was ~68th percentile, and ~48th percentile on essays. Third, examining official NCBE data and using several conservative statistical assumptions, GPT-4's performance against first-time test takers is estimated to be ~63rd percentile, including ~41st percentile on essays. Fourth, when examining only those who passed the exam (i.e. licensed or license-pending attorneys), GPT-4's performance is estimated to drop to ~48th percentile overall, and ~15th percentile on essays." [1] - https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4441311 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4441311

TimPC 3y ago

Sure but I can find examples of this in both directions. GPT-4 fails spectacularly at any task that doesn't have very precisely sanitized inputs presented to it in a certain format. It also fails at any task requiring any sort of interaction with the world. I think criticisms that it isn't a general intelligence are entirely fair even if it seems a bit more like general intelligence than most things that came before.

gentleman11 3y ago

> GPT-4 passes the bar exam with a top 10% score There was an article on HN yesterday debunking that. OpenAI is prone to exaggerating