4 ms·

Err, did you miss the part where I wrote that the post is not meant to imply that ChatGPT is not impressive?! Deep Blue beating Kasparov in chess showcased Dee

Err, did you miss the part where I wrote that the post is not meant to imply that ChatGPT is not impressive?! Deep Blue beating Kasparov in chess showcased Deep Blue's capabilities; Deep Blue beating Bryan Caplan would not have showcased these capabilities, despite the capabilities being there. The analogous hypothetical situation would be a Not-so-deep Blue that fails to beat not just Kasparov, but any random chess master with a nonzero Elo score, but manages to beat Bryan Caplan. After his defeat, Caplan would write an article claiming that he's "known to be very good at chess". This would definitely say more about Caplan's chess skill than the chess engine's capabilities. And none of this would imply that chess engines are in any way impossible or unlikely, or that the capabilities are not impressive. Indeed, a chess engine that beat Caplan would have been impressive in 1975, even though we were still very far from Deep Blue or Stockfish at that point.

p-e-w 3y ago
You did write that, but you first wrote that GPT-4's performance says more about the task than about its capabilities. Which is the exact same pattern that has been used to "deal with" advances in AI since forever. And Caplan is a university professor, not a random person. He's not an amateur at giving tests. It's literally (part of) his job. So what happened here is not at all analogous to your chess examples.
- palata 3y ago
  I would say that the OP is entitled to share their opinion. Which I tend to agree with. Also I would question if the job of a university professor is really to give tests, or rather to teach and conduct research. Tests are just an annoyance (that is deemed necessary). Because one passes the university tests does not mean they necessarily understand the topic very deeply, I think it's pretty obvious. That's actually an interesting point: university students can totally pass exams by "mimicking": not understanding the topic, but solving the exam problems in a way similar to the exercises they learned beforehand. I argue that the test is not perfect at discriminating those who understand from those who mimick impressively well.
  - ruszki 3y ago
    > the test is not perfect at discriminating those who understand from those who mimick impressively well. They discriminate all the time, just in the wrong direction. There are a lot of exams which are written in a way which counts on this mimicking, but in a way which makes tests way more difficult even if you understand the topic. It’s the same problem with remote learning: it’s way easier to cheat on them. In the past 3 years, (except the first semester when COVID broke out) remote exams were way more difficult than before to mitigate this. I had tests which were clearly made this cheating in mind. It was borderline impossible without cheating, they counted on it. At that point, I have no idea why they still tried to enforce any such rules. And still, I know many teachers who clearly don’t understand their topic completely, they just “mimic” it.
- posterboy 3y ago
  Shifting the goal post is a realistic measure on a slippery sloap. Chess isn't that different, inasmuch as memorizing openings and end games is part of the training.
- simonh 3y ago
  >And Caplan is a university professor, not a random person. He wasn't compared to random people, he was compared to other professors also setting tests, and how GPT-4 fared on their tests. You're blatantly misrepresenting the point being made.
  - p-e-w 3y ago
    I was referring to the chess analogy from the comment I responded to, not to the comparison from the top level comment.
roenxi 3y ago
> The analogous hypothetical situation... I'm not sure if that analogy holds any more given that neural nets scale with time and data. If that was happening with a neural net chess engine, the difference between beating Caplan and beating Kasparov is measured in maybe a couple of days. Google's Go AI only ever lost a game to Lee Sedol, then immediately moved a long way into superhuman territory with a bit more training. The gap between novice human and expert human looks quite small to a computer. If it can outperform a novice, it isn't far off beating an expert.
monjedetonsura 3y ago
The comments on the exam and the answers, which he claims to be correct and incorrect, make for interesting reading. https://www.thebigquestions.com/2023/04/05/gpt-4-fails-economics/ https://www.thebigquestions.com/2023/04/05/gpt-4-fails-econo...

p-e-w 3y ago

You did write that, but you first wrote that GPT-4's performance says more about the task than about its capabilities. Which is the exact same pattern that has been used to "deal with" advances in AI since forever. And Caplan is a university professor, not a random person. He's not an amateur at giving tests. It's literally (part of) his job. So what happened here is not at all analogous to your chess examples.

palata 3y ago
I would say that the OP is entitled to share their opinion. Which I tend to agree with. Also I would question if the job of a university professor is really to give tests, or rather to teach and conduct research. Tests are just an annoyance (that is deemed necessary). Because one passes the university tests does not mean they necessarily understand the topic very deeply, I think it's pretty obvious. That's actually an interesting point: university students can totally pass exams by "mimicking": not understanding the topic, but solving the exam problems in a way similar to the exercises they learned beforehand. I argue that the test is not perfect at discriminating those who understand from those who mimick impressively well.
- ruszki 3y ago
  > the test is not perfect at discriminating those who understand from those who mimick impressively well. They discriminate all the time, just in the wrong direction. There are a lot of exams which are written in a way which counts on this mimicking, but in a way which makes tests way more difficult even if you understand the topic. It’s the same problem with remote learning: it’s way easier to cheat on them. In the past 3 years, (except the first semester when COVID broke out) remote exams were way more difficult than before to mitigate this. I had tests which were clearly made this cheating in mind. It was borderline impossible without cheating, they counted on it. At that point, I have no idea why they still tried to enforce any such rules. And still, I know many teachers who clearly don’t understand their topic completely, they just “mimic” it.
posterboy 3y ago
Shifting the goal post is a realistic measure on a slippery sloap. Chess isn't that different, inasmuch as memorizing openings and end games is part of the training.
simonh 3y ago
>And Caplan is a university professor, not a random person. He wasn't compared to random people, he was compared to other professors also setting tests, and how GPT-4 fared on their tests. You're blatantly misrepresenting the point being made.
- p-e-w 3y ago
  I was referring to the chess analogy from the comment I responded to, not to the comparison from the top level comment.

roenxi 3y ago

> The analogous hypothetical situation... I'm not sure if that analogy holds any more given that neural nets scale with time and data. If that was happening with a neural net chess engine, the difference between beating Caplan and beating Kasparov is measured in maybe a couple of days. Google's Go AI only ever lost a game to Lee Sedol, then immediately moved a long way into superhuman territory with a bit more training. The gap between novice human and expert human looks quite small to a computer. If it can outperform a novice, it isn't far off beating an expert.

monjedetonsura 3y ago

The comments on the exam and the answers, which he claims to be correct and incorrect, make for interesting reading. https://www.thebigquestions.com/2023/04/05/gpt-4-fails-economics/ https://www.thebigquestions.com/2023/04/05/gpt-4-fails-econo...