9 ms·
No, GPT4 Can’t Ace MIT
- underanalyzer 3y agoGreat analysis, props to these students for taking the time to challenge such a sensational headline. In the conclusion they mention my biggest problem with the paper which is that it appears gpt4 grades the answers as well (see section 2.6 "Automatic Grading"). In a way it makes perfect sense that gpt4 can score 100% on a test gpt4 also grades. To be clear the grading gpt4 has the answers so it does have more information but it still might overlook important subtleties in how the real answer differs from the generated answer due to it's own failure to understand the material.
- ghaff 3y agoI noticed that when I read the paper. I know it's hard to scale but I'd want to see competent TAs doing the grading. I also found the distribution of courses a bit odd. Some of it might be just individual samples but intro courses I'd expect to be pretty cookie cutter (for GPT) were fairly far down the list and things I'd expect to be really challenging had relatively good results.
- raunakchowdhuri 3y agoCan attest that the distribution is odd from the test set that we sampled. We've already run the compute to run the zero-shot GPT model on all of the datapoints in the provided test set. We're going through the process now of grading them manually (our whole fraternity is chipping in!) and should have the results out relatively soon. I can say that, so far, it's not looking good for that 90% correct zero-shot claim either.
- mquander 3y agoSince you are here, when I was reading the paper I wondered -- when they show the "zero-shot solve rates", does that mean that they are basically running the same experiment code, but without the prompts that call `few_shot_response` (i.e. they are still trying each question with every expert prefix, and every critique?) It wasn't clear to me at a glance.
- mquander 3y ago> In a way it makes perfect sense that gpt4 can score 100% on a test gpt4 also grades. Even this is overstating it, because for each question, GPT-4 is considered to get it "correct" if, across the (18?) trials with various prompts, it ever produces one single answer that GPT-4 then, for whatever reason, accepts. That's not getting "100%" on a test.
- deleted 3y ago[deleted]
- aeternum 3y agoIn the paper, they at least claimed to manually verify the correct answers.
- sanderjd 3y agoThen - having not read the paper - what is the point of the automated grading?
- mquander 3y agoI just looked again and I didn't see that claim, can you verify? https://arxiv.org/pdf/2306.08997.pdf https://arxiv.org/pdf/2306.08997.pdf If as per the linked critique, some of the questions in the test set were basically nonsense, then clearly they couldn't have manually verified all the answers or they would have noticed that.
- 3y ago
- code51 3y agoThis "GPT4 evaluating LLMs" problem is not limited to this case. I don't know why exactly but everyone seems to have accepted the evaluation of other LLM outputs using GPT4. GPT-4 at this point is being regarded as "ground-truth" with each passing day. Couple this with the reliance on crowd-sourcing to create evaluation datasets and heavy use of GPT3.5 and GPT4 by MTurk workers, you have a big fat feed-forward process benefiting only one party: OpenAI. The Internet we know is dead - this is a fact. I think OpenAI exactly knew how this would play out. Reddit, Twitter and the like are awakening just now - to find that they're basically powerless against this wave of distorted future standards. When sufficiently proven to pass every existing test on Earth, every institution would be so reliant on producing work with GPT that we won't have a "%100 handmade exam" anymore. No problem will be left for GPT to be tackled with.
- YeGoblynQueenne 3y ago>> I don't know why exactly but everyone seems to have accepted the evaluation of other LLM outputs using GPT4. GPT-4 at this point is being regarded as "ground-truth" with each passing day. Why? Because machine learning is not a scientific field. That means anyone can say and do whatever they like and there's no way to tell them that what they're doing is wrong. At this point, machine learning research is like the social sciences: a house of cards, unfalsifiable and unreproducible research built on top of other unfalsifiable and unreproducible research. People simply choose whatever approach they like, cite whatever result they like, because they like the result, not because there's any reason to trust it. Let me not bitch again about the complete lack of anything like objective measures of success in language modelling, in particular. There have been no good metrics, no meaningful benchmarks, for many decades now, in NLP as a whole, but in language generation even more so. This is taught at students in NLP courses (our tutors discussed it in my MSc course) there is scholarship on it, there is a constant chorus of "we have no idea what we're doing" but nothing changes. It's too much hard work to try and find good metrics, build good benchmarks. It's much easier to put a paper on arxiv that shows SOTA results (0.01 more than the best system compared to!). And so the house of cards rises ever towards the sky. Here's a recent paper that points out the sorry state of Natural Language Understanding (NLU) benchmarking: What Will it Take to Fix Benchmarking in Natural Language Understanding? https://aclanthology.org/2021.naacl-main.385/ https://aclanthology.org/2021.naacl-main.385/ There are many more, going back years. There are studies of how top-notch performance on NLU benchmarks is reduced to dust when the statistical regularities that models learn to overfit to in test datasets are removed. Nobody. fucking. cares. You can take your science and go home, we're making billion$$$ here!
- afro88 3y ago> but it still might overlook important subtleties If there's one thing we can be certain of, it's that LLMs often overlooks important subtleties. Can't believe they used GPT4 to also evaluate the results. I mean, we wouldn't trust a student to grade their own exam even when given the right answers to grade with.
- kurthr 3y agoIf people haven't seen it UT Prof Scott Aaronson had GPT4 take his Intro Quantum final exam and had his TA grade it. It made some mistakes, but did surprisingly well with a "B". He even had it argue for a better grade on a problem it did poorly on. Of course this was back in April when you could still get the pure unadulterated GPT4 and they hadn't cut it down with baby laxative for the noobs. https://scottaaronson.blog/?p=7209 https://scottaaronson.blog/?p=7209
- refulgentis 3y agoIt literally did not change. Not one bit. Please, if you're reading this, speak up when people say this. It's a fundamental misunderstanding, there's so much chatter around AI, not much info, and the SnR is getting worse
- kurthr 3y agoWell, I had hoped the sarcastic comparison to cut heroin would make it clear. No, I don't think there's much change at all to GPT-4 (at the API level) and probably not that much at the pre/post language detection and sanitation for apparently psychotic responses.
- jumploops 3y agoI attribute this to two things: 1. People have become more accustomed to the limits of GPT-4, similar to the Google effect. At first they were astounded, now they're starting to see it's limits 2. Enabling Plugins (or even small tweaks to the ChatGPT context like adding today's date) pollute the prompt, giving more directed/deterministic responses The API, as far as I can tell, is exactly the same as it was when I first had access (which has been confirmed by OpenAI folks on Twitter [0]) [0] https://twitter.com/jeffintime/status/1663759913678700544 https://twitter.com/jeffintime/status/1663759913678700544
- JieJie 3y agoIn my experience with Bing Chat, in addition to what you say, there is also some A/B testing going on as well.
- asylteltine 3y agoPlease pause AI development. It’s going to ruin all of our lives
- LouisSayers 3y agoThere's no need to even make such a complicated task. Recently I found that GPT4 can't even reliably create a list of german nouns with a given article (der / die / das). It will mess up a simple list - if you ask it to analyse it, it'll be able to tell you that it's wrong. Then you get it to correct the list and it may still be wrong. It can take several iterations to make the list correct. I would have thought this would be a super easy task for it, but apparently not.
- wudangmonk 3y agoI think this task is beyond the capabilities of what GPT4 can handle, this is simply asking too much out of it. For other languages I'm sure it has no problems. https://faculty.georgetown.edu/jod/texts/twain.german.html https://faculty.georgetown.edu/jod/texts/twain.german.html
- deleted 3y ago[deleted]
- freehorse 3y agoIn greek it is also really bad (and makes rather obvious mistakes), in french it seems much better but makes some very obvious mistakes too. To make it interesting, I emphasise that I want nouns that refer to objects only (else it just spits out profession names and stuff like that, which is not interesting). Also tbh, with all the hype of LLMs one would think that such a task would not be such a challenge.
- dontupvoteme 3y ago>Also tbh, with all the hype of LLMs one would think that such a task would not be such a challenge. The strange/sad thing is that despite being "large language models" they're often hypermyopic on English.. I've done some measurements comparing generation between various languages in the prompt and no matter what I do half the time i cannot get them to not include english text or comments in code unless the request is made in japanese, chinese, or a similar very-different-language.
- Spivak 3y ago[flagged]
- quadrifoliate 3y ago> peer review...Their null is GPT4 can ace MIT and they haven't provided any evidence to reject. I haven't been in research for a while, but I don't think that's how peer review works. You don't always have to assume the paper's claims (especially ones as novel as this one's) as a null hypothesis, and providing a compelling refutation. The detection of sloppy question framing, and answer feeding via the few-shot learning examples, and the problems with checking with GPT-4 itself reasonably show that there are serious flaws in multiple parts of the experiments described in the paper. > The real conclusion is "we think the paper's results might be weaker if repeated" and that's a good result on its own. No, they don't know enough to say that. The paper's results might be better with better experimentation! Or they might be totally false. The conclusion they provided is accurate precisely because it focuses on the methods and not the conclusions, like this: > One particularly worrying trend is the technique of evaluating a model’s accuracy using a language-based model like GPT-4. While a useful tool, its conclusions should never be overstated or treated as ground truth. ...and this: > Additionally, it is extremely important to reevaluate every data point and perform basic sanity checks before using data at all, whether for training, inference, benchmarking, or something else. Given the small size of the dataset in question, a simple manual validation would have been easily within the scope of the work.
- mquander 3y agoI think it's appropriate to be extremely critical. The paper is basically useless. The thing that they actually measured is "can GPT-4, when given a 'question' with lots of additional information and many tries with small permutations to produce an 'answer', at some point produce an 'answer' that GPT-4 will then claim is a 5 out of 5 answer, on a dataset of extremely messy 'questions' and 'answers' from MIT coursework." That's not an interesting thing to measure. The paper talks about it in terms that make it sound like it's a close proxy for whether GPT-4 "knows" how to do things in MIT coursework, by writing misleadingly about "fulfilling the graduation requirements" and having a "perfect solve rate." But in fact it's totally different. The result is that a bunch of people hear about this paper and get fooled into thinking that there is new interesting evidence about GPT-4's capabilities, unless they manage to read closely enough to see what actually happened. It's not a matter of whether the results would get weaker if repeated, it's a matter of the results being totally disconnected from any useful real-world information about what GPT-4 can do, or how it can do it.
- varjag 3y agotl;dr the dataset was nonsensical and the researchers used GPT-4 to rate its own answers in the tests.
- constantcrying 3y agoPretty damning. Certainly seems retraction worthy. Also somewhat upsetting that something so low quality actually is getting published. Seems entirely driven by hype and not intellectual rigor.
- ericpauley 3y agoIt should be noted that the original paper is a preprint, i.e., not peer-reviewed.
- constantcrying 3y agoI count 15 people as authors. Surely one of them was able to look over the data and methodology. "Not peer reviewed" is no excuse to publish non-information for the sake of headlines.
- rat9988 3y agoIt's not published though. That's what he is trying to tell you. There is no guarantee it would have been published.
- Sharlin 3y agoDifferent definitions of "published". Uploading a preprint to arxiv or whatever definitely counts as publishing it in the nontechnical sense of the word – to an audience comprising several billion eyeballs, no less!
- CameronNemo 3y agoStill, the question remains who published it. Some of the authors (perhaps the supervising ones) may have wished not to submit it to journals, and a zealous undergrad may have uploaded it to arxiv without removing the other authors.
- 3y ago
- freehorse 3y agoThis is one of the most embarrassing reviews I have ever read (for the paper reviewed). AI research needs urgently certain good practices to adhere to, but the current status is that it is really hard to take many of the results seriously due to the opaqueness that characterises it in many steps of the process. And such serious mistakes and bad practices certainly do not help the field to achieve any credibility.
- hn_throwaway_99 3y agoThis note from the article is important: > Several of the authors listed on the discussed paper are undergraduate researchers. Consequently, we believe it's inappropriate to hold these individuals accountable for any lapses present in the work. Instead, we believe the responsibility should lie with the supervising authors. They are the ones who are expected to ensure that the work meets the rigorous standards of public scholarship within their field.
- psyklic 3y agoEven research from OpenAI has attempted to use GPT-4 as quasi-ground truth (as a replacement for human evaluators). For example, their method in the recent paper "Language models can explain neurons in language models" [1] is: 1. Using GPT-4, generate a text explanation of a neuron's activations on sample input. 2. Using GPT-4 again, use the text explanation to simulate the neuron on some new text input. 3. Compare the result to the actual neuron's activations on the new text input. They justify this by saying human contractors do equally poorly at coming up with text descriptions. However, the procedure is such a black box that it is difficult to make scientific conclusions from the results. [1] https://openai.com/research/language-models-can-explain-neurons-in-language-models https://openai.com/research/language-models-can-explain-neur...
- EGreg 3y agoCan't we just have GPT-4 make the scientific conclusions from these results? /s
- malisper 3y ago> Even research from OpenAI has attempted to use GPT-4 as quasi-ground truth (as a replacement for human evaluators). The way OpenAI used GPT-4 is fundamentally different than how GPT-4 was used to score the answers to the MIT exam. In OpenAI's case, they had GPT-4 generate an explanation of when a neuron in GPT-3 would fire. They then gave that explanation back to GPT-4 and had GPT-4 predict when the specific neuron in GPT-3 would fire. The scoring was done by computing the correlation between when GPT-4 predicted the neuron would fire and when it actually fired. The scoring was not done by GPT-4 as was done for the MIT exam In addition OpenAI did have human evaluators score the explanations as well to make sure they were human interpretable[0] [0] https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html#sec-human-scoring https://openaipublic.blob.core.windows.net/neuron-explainer/...
- tehsauce 3y agoCorrect except they did this for gpt2 not gpt3
- 3y ago
- raunakchowdhuri 3y agoAm one of the authors of the linked article. Thanks for all of your kind comments! Let me know if you have any questions and I'll be happy to answer.
- dmazzoni 3y agoWhose idea was it? Was this something done for fun or was it suggested by a professor you're working with?
- raunakchowdhuri 3y agoNeil posted the paper in our fraternity's ML group chat (MIT things lol), and I expressed some skepticism at the results. Initially we started looking into it more for curiosity's sake, but as we started digging we kept finding more and more ridiculous stuff to the point where we decided to start working on blog post. Then David joined in on the action and helped a ton with the research and writeup. No professor was involved. The paper was released yesterday, so we just documented as we went along in the investigation process. It only took like 8 hours of work to compile the doc. We finished it last night and posted to Twitter in the morning.
- mensetmanusman 3y agoHow disruptive has this tech been to course 6 take home exams/homework/study axioms? I remember the very late nights at Burton Connor, maybe students will get more sleep now :)
- deleted 3y ago[deleted]
- muds 3y agoI'm not sure what to make of this post. There is always a degree of uncertainty with the experimental design and it's not surprising that there are a couple of buggy questions. Imagenet (one of the most famous CV datases) at this point is known to have many such buggy answers. What is surprising is the hearsay that plays out on social media that blows the proportion of the results out of the water and leads to opinion pieces like these targeting the authors instead. Most of the damning claims in the conclusion section (Obligatory: I haven't read the paper entirely, just skimmed it.) usually get ironed out in the final deadline run by the advisors anyway. I'm assuming this is a draft paper for the EMNLP deadline this coming Friday published on arxiv. So this paper hasn't even gone through the peer review process yet.
- iudqnolq 3y agoImageNet has five orders of magnitude more answers, which I would assume makes QA a completely different category of problem. The authors could probably have carefully review all ~300 of their questions. If they couldn't they could have just reduced their sample size to say 50.
- muds 3y agoI admit that Imagenet isn't the best analogy here. But I'm pretty confident that this data cleaning issue would be caught in peer review. The biggest issue which I still don't understand was the removal of the test set. That was bad practice on the authors' part.
- iudqnolq 3y agoIn general with evaluations LLVMs I keep seeing issues that would be caught by a human carefully looking at the results. To a nonexpert who occasionally peeks at things it seems like having a small dataset that's just slightly too big for you to manually review is a bad practice. It also seems like 100% accuracy should have raised red flags, especially if you know your dataset isn't perfectly cleaned.
- kgeist 3y agoWith LLMs hallucinating and being generally unreliable, I'm coming to the conclusion that LLMs should only be used to transform natural language to structured data (specific to your application), and knowledge about the world should be stored somewhere else (some sort of vector databases? tools?), and, fortunately, smaller models are already quite good at the former! Trying to extract world knowledge from LLMs can be a deadend... I don't see how the hallucination problem can be 100% fixed for LLMs themselves.
- tjr 3y agoI do not trust LLMs. They may very often be right, and may very often be useful, but I do not trust them. If you require correct results, their output should be checked somehow.
- aeternum 3y agoStrange take, do you consider world knowledge produced by humans to be anywhere near 100% accurate? Suppose you had an oracle that when asked a question gives you 5 answers, 1 of which is true. Or even 1 of which is true only 50% of the time. You would still generally be a fool to throw away that oracle.
- kk__ 3y agoIf I was able to use open book for every exam and had the whole thing digitized in advance with billions of notes and evaluations of the texts the questions were based on I'd probably ace it to.
- Solvency 3y agoThat's what separates you from GPT: it's willing to put in the work.
- iudqnolq 3y agoTangent: Question 42 seems bad to me. It asks to analyze the behavior of two processes concurrently calling a method on the same typescript class instance. Either in the browser or in node JavaScript does not have processes operating on shared memory. While I get what they're trying to say I think it's counterproductive to learning to call a task on the JavaScript event loop a process. Answering the question requires knowing that the runtime will only preempt at await statements, and calling it a process confuses that.
- renonce 3y agoHas the academia established clear standards on what is competent work and what is not? It occurs to me that while a small subset of papers stand out, many papers are struggling with conforming to basic practices like publishing runnable code and data, keeping them up to date with latest libraries and models within a year after the paper is published, and demonstration of performance in varied, potentially subjective ways, rather than picking a random benchmark and showing accuracy improvements by 0.01% and call it scientific. Is it just me, or if for the majority of papers, the effort required to understand and get value out of the paper is so much higher than the effort put in by the authors and reviewers to publish it?
- obscurette 3y agoThis trends of sensational but flawed papers is extremely worrying. I can literally see how it erodes the trust in science in general among tech people around me. And I understand it of course – if papers in your area are not trustworthy, what makes you think that other papers can be trusted?