5 ms·

This should really link directly to Landsburg’s post rather a Twitter promotion of it: https://www.thebigquestions.com/2023/04/05/gpt-4-fails-economics/ https:

by SilasX 4y ago

This should really link directly to Landsburg’s post rather a Twitter promotion of it: https://www.thebigquestions.com/2023/04/05/gpt-4-fails-economics/ https://www.thebigquestions.com/2023/04/05/gpt-4-fails-econo... Edit: Also, the post says GPT-4 scored 4/90, a little better than 0.

hgsgm 4y ago
Landsburg's test has intentionally misleading questions with ambiguous wording, including two he admits had mistakes per his own standard for himself. He then graded exceptionally harshly. I don't believe his students average 75% on the rubric he used for ChatGPT. I believe he made his mind up before he started the experiment.
- Our_Benefactors 4y ago
  This was my impression as well. When reading his nitpicky reasons a question was scored zero out of nine (the same as leaving it blank!) made me think that sitting this course would have me hating his guts by the end of it.
  - carlmr 4y ago
    It reminds me of one of my professors, actually also in economics. He would grade in a similar way. Basically he had some very extreme views, based on terribly simplified economic models, and would want you paraphrasing what he said, ideally quoting. The closer you could match his wording and extreme views the better you scored.
    - hgsgm 4y ago
      OP Lansburg is famous for his heterodox views on economics.
- usaar333 4y ago
  Ya, these questions seem too hard for an undergrad test or at least feel like they were only written in an age where you want to be adversarial to LLMs. Don't get me wrong - I can give simple tests that LLMs fail and a college kid could easily pass -- but I don't think I'd come up with them unless I wanted to be adversarial to LLMs. Example simple question that GPT4/Bard/etc. fail hard: > The moose is behind the cat. The dog is in front of the moose. The moose is in front of the rat. The rat is behind the cat. The elk is in front of the dog. What's a possible animal ordering?
  - Kokouane 4y ago
    Just gave this prompt to GPT 4 and it gave what appears to be a correct answer: > rat, moose, cat, dog, elk Is this wrong somehow?
    - usaar333 4y ago
      hmm, maybe you got lucky? My answer is on ChatGPT4: A possible animal ordering, from front to back, would be: Elk, Dog, Moose, Cat, Rat (Which violates the moose being behind the cat) Another answer I get in a new session is: A possible animal ordering based on the given information is: Rat, Cat, Moose, Dog, Elk (which is the same as above just revered)

by SilasX 4y ago

hgsgm 4y ago

Landsburg's test has intentionally misleading questions with ambiguous wording, including two he admits had mistakes per his own standard for himself. He then graded exceptionally harshly. I don't believe his students average 75% on the rubric he used for ChatGPT. I believe he made his mind up before he started the experiment.

Our_Benefactors 4y ago
This was my impression as well. When reading his nitpicky reasons a question was scored zero out of nine (the same as leaving it blank!) made me think that sitting this course would have me hating his guts by the end of it.
- carlmr 4y ago
  It reminds me of one of my professors, actually also in economics. He would grade in a similar way. Basically he had some very extreme views, based on terribly simplified economic models, and would want you paraphrasing what he said, ideally quoting. The closer you could match his wording and extreme views the better you scored.
  - hgsgm 4y ago
    OP Lansburg is famous for his heterodox views on economics.
usaar333 4y ago
Ya, these questions seem too hard for an undergrad test or at least feel like they were only written in an age where you want to be adversarial to LLMs. Don't get me wrong - I can give simple tests that LLMs fail and a college kid could easily pass -- but I don't think I'd come up with them unless I wanted to be adversarial to LLMs. Example simple question that GPT4/Bard/etc. fail hard: > The moose is behind the cat. The dog is in front of the moose. The moose is in front of the rat. The rat is behind the cat. The elk is in front of the dog. What's a possible animal ordering?
- Kokouane 4y ago
  Just gave this prompt to GPT 4 and it gave what appears to be a correct answer: > rat, moose, cat, dog, elk Is this wrong somehow?
  - usaar333 4y ago
    hmm, maybe you got lucky? My answer is on ChatGPT4: A possible animal ordering, from front to back, would be: Elk, Dog, Moose, Cat, Rat (Which violates the moose being behind the cat) Another answer I get in a new session is: A possible animal ordering based on the given information is: Rat, Cat, Moose, Dog, Elk (which is the same as above just revered)