11 ms·
GPT4 gets a 0 on Steven Landsbrug's undergrad econ exam
- hackrnusr 4y agoBecause it couldn't explain corporate executive compensation levels?
- eru 4y agoPlease consider reading the article next time.
- stuff4ben 4y agoI'd expect most economists couldn't do a good job either. It's black-box magic, suitable for wizards.
- phendrenad2 4y ago"Most economists couldn't pass an undergrad econ exam" is probably untrue, right?
- deleted 4y ago[deleted]
- jcparkyn 4y agoDoesn't seem too far fetched. I doubt many working engineers would pass all their undergrad exams without doing some revision first. And as other commenters have pointed out, this is clearly not a representative example of most exams.
- Ancalagon 4y agoHard to contrast this with the report of gpt-4 getting a B in quantum computing yesterday
- usaar333 4y agoAaronson's test seems to require less "fluid intelligence" to get right. Someone who has read the book and just pattern matches basically can do well on the test. A lot of tests work this way. Steve's test requires.. more fluid intelligence. Question 2 is simply not something you see on the internet in any common form, but the if you do lots of deduction of the implications -- ya, you get it. I've noticed similar weird fails with GPT-4. It seems like a genius recalling factual information and even (to some degree) translating common word problems into math. It starts failing hard though when you give it things way outside its domain (often children could outperform it since humans seem to not need so many shots of novel knowledge to understand things)
- TheCoelacanth 4y agoThat's putting it nicely. Aaronson's test was more focused on shamelessly parroting the professor's ideological views which were strongly hinted at in the question. GPT-4 indeed was better than the real students at that task since it lacks a sense of shame. This one seems more focused on actual application of economic principles.
- sverona 4y agoHonestly this doesn't surprise me. I don't have GPT-4 access, but I fed ChatGPT a simple calculus question as a sanity check. Question: Calculate the integral of x arctan x dx. Show your work. Answer: We can solve this integral using integration by parts: Let u = arctan x and dv = x dx, then we have: du/dx = 1/(1+x^2) v = (1/2)x^2 Using the integration by parts formula, we get: ∫ x arctan x dx = uv - ∫ v du/dx dx = (1/2)x^2 arctan x - (1/2)∫ x^2/(1+x^2) dx To solve the second integral, we can use substitution. Letting u = 1+x^2, then du/dx = 2x, and dx = du/(2x). Substituting into the integral, we get: ∫ x^2/(1+x^2) dx = (1/2) ∫ du/u = (1/2) ln|u| + C = (1/2) ln|1+x^2| + C Putting it all together, we get: ∫ x arctan x dx = (1/2)x^2 arctan x - (1/2) [(1/2) ln|1+x^2|] + C = (1/2)x^2 arctan x - (1/4) ln|1+x^2| + C Therefore, the integral of x arctan x dx is (1/2)x^2 arctan x - (1/4) ln|1+x^2| + C, where C is the constant of integration. --- This is wrong, although it gets pretty damn close (see if you can spot the mistake.) I was an exceptionally kind grader when I taught and Landsburg seems like an exceptionally picky one. But either way ChatGPT wouldn't pass calculus 2, either.
- MuffinFlavored 4y agoI think the real power will be when LLMs are trained to formulate the response "math detected, let me attach a low score to anything other than 'pipe user input into wolfram alpha and then use output'" Same for "let me call my XYZ plugin to do XYZ specialized task" to fill the gaps where LLMs fall short (most places when it comes to accuracy?). once ChatGPT can access the web/do basic curl/wget requests, then maybe visually inspect an image or a PDF, it's game on for "ok, AI is serious now".
- jasfi 4y agoExactly this. LLMs shouldn't attempt math, they should pass it to a solver.
- MuffinFlavored 4y agoi kind of wonder if in hindsight, not thinking in this pattern (pass math to a solver, etc.) was an oversight by... OpenAI? is that who/what is responsible for widespread popularity of LLMs? asking ChatGPT says LLMs got popular when GPT-2 was released by OpenAI in 2019
- SilasX 4y agoThis should really link directly to Landsburg’s post rather a Twitter promotion of it: https://www.thebigquestions.com/2023/04/05/gpt-4-fails-economics/ https://www.thebigquestions.com/2023/04/05/gpt-4-fails-econo... Edit: Also, the post says GPT-4 scored 4/90, a little better than 0.
- hgsgm 4y agoLandsburg's test has intentionally misleading questions with ambiguous wording, including two he admits had mistakes per his own standard for himself. He then graded exceptionally harshly. I don't believe his students average 75% on the rubric he used for ChatGPT. I believe he made his mind up before he started the experiment.
- Our_Benefactors 4y agoThis was my impression as well. When reading his nitpicky reasons a question was scored zero out of nine (the same as leaving it blank!) made me think that sitting this course would have me hating his guts by the end of it.
- carlmr 4y agoIt reminds me of one of my professors, actually also in economics. He would grade in a similar way. Basically he had some very extreme views, based on terribly simplified economic models, and would want you paraphrasing what he said, ideally quoting. The closer you could match his wording and extreme views the better you scored.
- hgsgm 4y agoOP Lansburg is famous for his heterodox views on economics.
- usaar333 4y agoYa, these questions seem too hard for an undergrad test or at least feel like they were only written in an age where you want to be adversarial to LLMs. Don't get me wrong - I can give simple tests that LLMs fail and a college kid could easily pass -- but I don't think I'd come up with them unless I wanted to be adversarial to LLMs. Example simple question that GPT4/Bard/etc. fail hard: > The moose is behind the cat. The dog is in front of the moose. The moose is in front of the rat. The rat is behind the cat. The elk is in front of the dog. What's a possible animal ordering?
- maister 4y agoAnyone had a look at those questions? > Question 2: In a country where everyone is identical, 100 people wait in line each day to buy raspberries at a controlled price. The government has decided to hand out free coffee to the people standing in line. The coffee costs the government $1 per cup, but the people in line value that coffee at only 75 cents per cup. What is the social cost of providing the coffee? > Grading Remarks: This answer completely misses the key fact that free coffee will cause the line to get longer (in fact it must cause the line to get longer, given the stated assumption that everyone is identical, hence initially indifferent between standing in line and not standing in line). In fact (unless one assumes a very small population), the line must grow until the extra waiting time completely dissipates the value of the free coffee; thus the social cost of providing the coffee is $100. Wait what? So the line must get longer, *given the stated assumption that everyone is identical*. Well what about the given stated assumption *that 100 people wait in line each day*? One of the stated assumptions can be dropped just like that, while the other is treated like a law of nature? Also where is the logic in "everyone is identical, hence initially indifferent between standing in line and not standing in line"? Am I missing something?
- mumrik 4y agoBefore the coffee, the equilibrium was for the line to be 100 persons long. After the introduction of the coffee, the equilibrium changes so that people are willing to wait in a longer line than 100 people, although the individuals themselves don't have any preferences (like A only standing in line if it's less than 20 people and so on). I think not much thought was given to explain the answers in depth since the main point was to show where GPTs reasoning was lacking.
- maister 4y ago> I think not much thought was given to explain the answers in depth since the main point was to show where GPTs reasoning was lacking. Funnily enough it seems as if the prof has also fallen for a reasoning error then. How can both be equally true? The line gets longer than 100 people and the social cost of the coffee is exactly 100 dollar (1 Dollar for every person).
- fullsend 4y ago[dead]
- SilverBirch 4y agoI'm still extremely skeptical of testing LLMs on any of type of these tests. The likelihood of the model being trained on the answers to the test is too high. For example, with the bar exam that people go on about - it seems to me that the internet is going to be absolutely chocked full of sample questions and answers for the bar exam. Anyone with any knowledge at all in AI knows that one of the worst things you can do is evaluate your AI using data in the training set. Of course it's going to do a good job! It's telling you what you told it! Why not just put down a SQL database and stick all the answers in it (I'm being facetious, but you get my point). We know from other research papers that people have managed to prompt the generative AI image generators to reproduce copyrighted pictures in a way that makes it very clear they were just very highly trained on that during training. It's the same with this, answers to standardized tests are always going to appear very commonly in your training data. The point of the standardized test works okay as a spot check for humans who have studied the course, but extremely badly for AI that hasn't studied the course but instead has just hoovered up every token in the course.
- hgsgm 4y agoHuman students also get to see those sample questions. Is the Bar exam a bad test? Maybe. But it's how we judge the merit of new lawyers.
- jfengel 4y agoHumans and LLMs solve the bar exam differently. A human can't memorize the complete set of solved examples, so they have to generate it at least somewhat from scratch. It's why we give the bar exam under controlled circumstances, so that they're not just plagiarizing from existing sources. And that's what you want to test them for: the ability to apply their knowledge to novel situations. LLMs don't have memory limitations, so they can produce equivalent answers using very different means. But that means that it's a poor proxy for those novel situations. The real challenge a lawyer faces is sitting down with a client and getting a stream of verbiage, some of which is relevant and some of which isn't. In that, it's much like a programmer's job. Extracting the actual requirements is usually harder than writing the code for them. As with programming, some days are easier than others. There's a middle ground of tasks that are harder than just filling in a form, and easier than inventing a new algorithm or legal theory. Perhaps it's the "no code" of law: good enough for users to solve a lot of problems on their own, and they need to call out to a professional for more things. The bar exam could potentially be a proxy for computers to prove that they're ready to take up that middle ground. But it hasn't been tested on that yet.