13 ms·
Benchmarking GPT-4 Turbo – A Cautionary Tale
- Kuinox 3y agotldr: GPT-4 Turbo have worse score on synthetic benchmark of the first attempt because they speculate it's a smaller model, and isn't able to memorize as well the response.
- sunpazed 3y agoThis reflects my anecdotal findings in a very narrow domain: GPT-4 consistently resolves complex SQL from natural language, whereas GPT-4-Turbo is hit-and-miss, and similar to 3.5-Turbo in performance.
- Semaphor 3y agoBut that’s not what the article is saying at all?
- boxed 3y ago> We designed a test for this theory: we reran the benchmarks without showing the models the instructions to each exercise. Instead, we just told them that they were Exercism exercises, and gave them the exercise names and function stubs. This summarizes all my skepticism agains the AI field. Pretty clear that they aren't solving the problem, they have them memorized.
- mavhc 3y agoLLMs are lossy compression
- stoicfungi 3y agoAll models are, including humain brain.
- ChatGTP 3y agoThe human brain is a model?
- rocgf 3y agoIt models the world around it, so it's fairly similar to what GPT does, especially with the newly-added image capabilities and stuff.
- ChatGTP 3y agoBut the brain itself is not a model.
- FrustratedMonky 3y agoConsciousness itself is a model of the world. Our experience of the world is a model executing. Comparing the latest neuroscience to latest neural networks. They look and behave very similarly.
- famouswaffles 3y ago"They memorized all the problems" is not what was found here and still a wrong overcorrection.
- dartos 3y ago“Gpt-4 has more problems memorized than gpt-4 turbo” was exactly what was found here. That doesn’t mean it’s only able to solve problems in its training set (tho it’s much better at that obviously.)
- boxed 3y agoIf you are shown only the title of a coding problem and the site name where it's from, and you manage to solve it you are showing that you either cheated or knew the answer.
- famouswaffles 3y agoI mean sure, it memorized some of the answers. I'm not denying that. Clearly, it didn't memorize all of them.
- boxed 3y agoWhen people say "oh look how amazing, it can solve programming problems!" when in fact it has only seen the models CHEAT, is an enormous problem. For cases where finding the answer it's perfectly fine, but it's not fine for claims that it can code. There's a huge difference.
- FeepingCreature 3y agoI've seen it code on completely novel tasks, so I'm not sure what you're suggesting here. The model can unquestionably code.
- famouswaffles 3y agoOkay... Funny how forcing it to not CHEAT did not increase apparent ability. It can code and it has memorized some coding questions are not mutually exclusive.
- deleted 3y ago[deleted]
- DecayingOrganic 3y agoMemorization often gets a bad rap as the underachiever's shortcut. However, it's a fundamental component of any learning process! Our ability to reason, solve problems, and innovate is all built upon a foundation of memorized information. In fact, it's precisely the reason humans have thrived for so long; we were able to memorize and pass down knowledge culturally long before the written word, not because we were 100 times smarter than our nearest cousins. Without memorization, be it in our brains or AI algorithms, there's no foundation to build upon for higher reasoning.
- viraptor 3y agoIt's hard to decide for me without seeing the data. Even if you don't know the exact exercise, seeing the title and the function name/parameters is often enough for me to guess what the challenge is. I checked the public questions on exercism and almost all of those (that I spot checked) that contained the function name were extremely obvious. Knowing it's a programming challenge would also improve my guessing chances. For example the function stubs I can find are "value_of_card(<card>)" in exercise "Black Jack", or "generate_seat_letters(<number>)" in exercise "Plane Tickets". I think I could guess those without seeing the rest of the question.
- GaggiX 3y agoSo how can it solve novel problems? Internet does not have all combinations for every possible task with any random programming language, library or constraints. It can even solve problems with non-existing programming languages and libraries if you describe them, if that's just memorization then I don't know what it isn't.
- m3kw9 3y agoFrom a black box point of view and one angle, gpt is a web filter where it will try to find you the exact thing you are looking for but from memory. Vs google you have to distill all the info into what you need
- ren_engineer 3y agoeven if it's not true AI or even an architecture with the potential to become AI, LLMs are already good enough to provide real world value. Obviously "super autocomplete" isn't as sexy as true AI, but still very useful
- drcode 3y agoYou can call it whatever you want, all I know is I used to write programs in lines of code, then blocks of code at a time, spit out by LLMs Using GPT-4 Turbo yesterday, I feel like I'm moving to pages of code at a time now. Taking the ideas in my head and turning them into reality is so easy now
- blovescoffee 3y agoOk but you understand there's a body of literature that shows that LLMs don't "just" memorize
- gloosx 3y ago+100 to that. My biggest scepticism is people actually creating a new problem while thinking they are solving problem. Don't get me wrong, translating natural language ideas into code is fun and all, the truth it is also code, yet in ambiguous language format given to the machine. When did natural language became better for expressing development ideas than code? I know – when you don't know how to code in the first place. Then you will have to bet on all of the ambiguities of the language, cultural and meta-physical which words carry in order to hack your thing together instead of expressing yourself directly and explicitly. Finally what is beautiful about strict code format we are so used to - it is truly the fastest and shortest path to get your thing done, in case you possess the knowledge needed.
- ericrallen 3y agoThat sounds a lot like gatekeeping. These tools will empower folks who aren’t developers to build stuff and maybe learn a bit more about how programming works. They will enable folks who have ideas, but can’t express them, to actually be able to create what they are imagining. That’s awesome. Code isn’t beautiful (except for a few rare exceptions). Creating something with code is.
- gloosx 3y agoI agree it is a great tool for learning, but I don't believe anything more complex or of real use can be made AND maintained with it.
- ericrallen 3y agoI think we’re probably way to early in the AI lifecycle to really form any strongly held beliefs yet. In the 11 months since ChatGPT was released, things have come a long way. Who knows where we’ll be in another 11 months.
- gloosx 3y agoWhat I'm trying to say is that the problem is not approachable this way at all – efficiently generating code by describing what you want, since when you compress what you want into a prompt you lose the details, and in order to restore all of them you will need a much bigger prompt volume than code generated. Because it is code itself which compresses an idea but no idea can compress the code well enough. In another 11 months it will be exactly in the same spot - it will not be able to be more efficient at this task by the nature of it.
- caesil 3y agoIf that's your takeaway from this then you really missed the point. The implication here is that gpt-4 to gpt-4-turbo represents a leap away from memorization and toward better reasoning with a more complete world model.
- nathanfig 3y ago"memorize" implies they can only recite things verbatim and that's ignoring the massive leap in being able to synthesize disjoint "memories" in new ways.
- rldjbpin 3y agoif the benchmark means replicating the experience of taking technical interviews by most people, then this is a spot on approach and serves the potential user right.
- mccrory 3y agoGPT -4 Turbo is still in preview, maybe wait until it is fully released before judging?
- msp26 3y agoThe point of a preview phase is to test the model in real world use.
- seanhunter 3y agoThis isn't really real-world use any more than putting these same problems to people as a whiteboard coding exercise in an interview is real-world coding, yet seemingly a lot of people seem to be generalising from this tiny sample to all manner of overarching statements about performance of the model in general "it's faster but dumber", "this proves it only memorises" etc.
- deleted 3y ago[deleted]
- KaoruAoiShiho 3y agoWhy are all the comments here so negative... this is a good thing, turbo has less memorization but keeps the same reasoning ability. That's excellent and a relief.
- boxed 3y agoOr the programming quiz problems it tried to "solve" were in fact posted elsewhere also so it cheated on the ones it got right too.
- CamperBob2 3y agoPeople here spent a lot of time (and money) in school learning to do things that can now be automated. The whining is just beginning.
- deleted 3y ago[deleted]
- anotherpaulg 3y agoAider has had an Exercism benchmarking suite for quite some time. Interestingly, my benchmark results of GPT 4 Turbo show an opposite result: the new gpt-4-1106-preview did significantly better on the first try than the March and June models. https://aider.chat/docs/benchmarks-1106.html https://aider.chat/docs/benchmarks-1106.html Aider benchmarks against the 133 Exercism python exercises, not js exercises that mentat's benchmark uses. So this is not an apples-to-apples comparison, but there doesn't seem to be a strong reason to expect qualitatively different results. I also notice that the instructions prompt that mentat uses seems to be inspired by the aider benchmark? Glad to see others adopting similar benchmarking approaches. https://github.com/AbanteAI/mentat/blob/main/tests/benchmarks/exercism_practice.py#L115 https://github.com/AbanteAI/mentat/blob/main/tests/benchmark... https://github.com/paul-gauthier/aider/blob/main/benchmark/prompts.py https://github.com/paul-gauthier/aider/blob/main/benchmark/p... Edit: Not sure if the mentat authors are in this thread? After looking around a bit, there seems to be a bunch of aider code in your repo. Some attribution would be appreciated. It might even be required under aider's Apache 2.0 license?
- naiv 3y agoI am also noticing a massive improvement over the old model
- hanselot 3y agoIsn't it a good thing that of the benchmarks they ran, the newer model has fewer of the answers memorized (aka, its parroting less)? Wouldn't this actually be exactly proof that the model has improved over its predecessor by having to solve the problem itself rather than rely on memory? What use is a model that memorizes the answers to all the benchmarks (see the 7b models on open llm leaderboard for more info on that).
- derwiki 3y agoI’ve been using the new model with Aider since it was released, and my anecdata agrees—the “edits applied successfully “ failure rate is much lower than classic gpt4. Also THANK YOU for Aider! I talk it up to all my programmer friends; it really feels like a glimpse into the future of coding.
- mufasachan 3y ago> Although the author OCR’ed the SAT questions and believes that they weren’t in the training data I agree that the author of the tweet fairly underestimates the potential portion of OCR'ed contents in OpenAI's training data. In late August, Nougat[1] is released by Meta, this is an OCR model. Its performance are wild and the model is open source. I hardly believe that OpenAI does not spend effort on getting more training from OCR content. I also hardly believes that OpenAI waits for a Meta paper to have an internal performant OCR model. [1]: https://arxiv.org/abs/2308.13418 https://arxiv.org/abs/2308.13418
- msp26 3y agoI'm interested in more testing on the context side of things. For my NLP pipelines, I batch n-articles together to process (extract fields from) in one prompt (final output is something like this {"1":[{}], "2": [{},{}]...}) in one message. Compute-wise it's inefficient but OpenAI charges by the token so it doesn't matter. It's very reliable on gpt-4 8k. I was also pretty happy with the results on 4-turbo initially but it seems that once you go past 30k-ish tokens in context (needs way more testing), it shits itself. The indexes don't match anymore and n_final_output is different from n_articles. Still, great model and even if the limits are lower in practice I suspect I'll get good use out of it. Edit: With better prompting, it feels stable at n=42, ~42000 prompt tokens.
- phillipcarter 3y agoInteresting. I was skeptical about some of their claims regarding longer context, since it's been my experience that these models just get lost after enough of it.
- msp26 3y agoYeah, degraded performance on long contexts has been observed in plenty of other models [https://arxiv.org/abs/2307.03172 https://arxiv.org/abs/2307.03172] so I was cautious too. Unfortunately I don't have access to 4-32k. I would have liked to test that out too.
- FeepingCreature 3y agoI wonder how often a human could guess the exercise based on just the function stub.
- ConnorMooneyhan 3y agoyeah, some of the exercises are like the following: ``` function helloWorld() { return ""; } helloWorld() ``` but those sorts of obvious examples are mostly in the beginner exercises, so I wonder what the distribution of the correct answers was. If it was guessing based on function stubs, the prediction would be that correct answers would be clustered around the beginner exercises, and that as the exercises advanced in difficulty, there were fewer correct answers.
- famouswaffles 3y agoI think it's interesting that forcing models out of memorization don't always show a steep drop in ability. I've definitely had instances where 4 memorized a common puzzle and failed a subtly altered variant but then got the variant after changing variable names or otherwise making it look different from what it would have memorized.
- Havoc 3y agoIs it know what exactly OpenAI does in the background when they make these turbo editions? Seems like sacrificing some quality for large gains on speed and cost but anyone know more detail?
- thorax 3y agoDon't think so, but there were some guesses on 3.5-turbo-- i.e. training a much smaller model on quality questions/answers from GPT-4. Same tactic worked again and again for other LLMs. I'm definitely curious on the context window increase-- I'm having a hard time telling if it's 'real' vs a fast specially trained summarization prework step. That being said, it's been doing a rather solid job not losing info in that context window in my minor anecdotal use cases.
- maciejgryka 3y agoI have similar conclusions so far. We have a custom data set (basically visual Q&A about web apps) and `gpt4` gets roughly 90% correct, while `gpt-4-1106-preview` only 86%. It's a little noisy (I didn't yet check out the new seeds functionality), but roughly consistent. Since I created this dataset by hand, it can't really be memorized. I'm sure there's _similar_ data in the training set, but answering correctly still requires some reasoning-like capabilities.
- m3kw9 3y agoNow do a programming task that requires more than 32k of context and see who’s “better”. If you don’t bench mark that you cannot get an overall pic. GitHub copilot for example could benefit big from the increased context
- broast 3y agoObviously it's a drawback but the silver lining of the small context window is it forces me to decouple everything and have very sensible and strict api's where I just write the docs and it writes the code.
- biobootloader 3y agowe are working on creating "real world" benchmarks that require a lot of context, and will report when we have results!
- m3kw9 3y agoThat’s why is called 4 turbo, not “4.5”. But the context length is a bigger cargo space
- minihat 3y agoGPT-4 Turbo is dramatically worse at one task I often try: Read the following passage from [new ML article]. Identify their assumptions, and tell me which mathematical operations or procedures they use depend upon these assumptions. GPT-4: Usually correctly identifies the assumptions, and often quotes the correct mathematics in its reply. GPT-4 Turbo: Sometimes identifies the assumptions, and is guaranteed to stop trying at that point and then give me a Wikipedia-like summary about the assumptions rather than finish the task. Further prompting will not improve its result.
- thorax 3y agoDo you have a link or gist of an example run you tried? I'd be curious to try something similar.
- jphoward 3y agoThe problem is the discussed results are comparing proportions of a relatively small number - 67 questions. If you model this as a binomial distribution, then 62/67 which GPT4-turbo got gives a 95% confidence interval of the 'true' performance of 83.4% to 97.5%, ie it comfortably includes the proportion that GPT4 achieved (64/67=95.5%). I think the evidence from these tests are not strong enough to draw conclusions from.
- epups 3y agoYes. I see people make this mistake time and again when evaluating LLMs. For a proper comparison, it's not enough to simply throw less than a hundred questions at it and point to a single digit difference. Not to mention that LLMs have some inherent randomness, so even if you passed the exact same tasks to the same model you would expect some variance. I see a lot of room of improvement in how we apply statistics to understanding LLM performance.
- Racing0461 3y ago> I think the evidence from these tests are not strong enough to draw conclusions from. I've used gpt4 turbo for some coding problems yesterday. It was worse. That's enough to draw conclusions for me.
- Maxion 3y agoI’m not surprised, most people can’t even tell the median from the mean.
- iJohnDoe 3y agoA little bit off topic question. When people are talking about costs with GPT, like the following link. Does the cost concern only apply to the API? If you’re using the WebUI and have a Plus account, is it always just the flat $20 amount? https://news.ycombinator.com/item?id=38193978 https://news.ycombinator.com/item?id=38193978
- tosh 3y agousually, yes (either cost of the API or cost to serve for OpenAI)
- geraltofrivia 3y agoIn my day job we use GPT4 quite a bit and we shifted to GPT4 Turbo today. We got a 2-5% performance increase, and quite a bit of speed increase as well. Not to say that the parent post is incorrect, of course. Only that its not as cut and dry as a "GPT4 Turbo is distilled (read: watered down) GPT4".
- hu3 3y agoInteresting. What do you use it for?
- geraltofrivia 3y agoCurrently only for unstructured (OCR) text to structured text conversion. We're transitioning from a legacy codebase full of regexes and undocumented functions that are understood only by the developer and god. The developers left and I don't believe in god. We tried throwing the unstructed mess to GPT, alongwith a few examples and got surprisingly good results.
- attentive 3y ago> undocumented functions that are understood only by the developer and god oh the irony :)
- geraltofrivia 3y agoI don't follow . Ironical how?
- attentive 3y agoYou replaced it with the system that is even worse than "undocumented functions that are understood only by the developer and god" by design. It's not even deterministic.
- 3y ago
- Satam 3y agoVery interesting and basically confirms that GPT-4 turbo is a faster but dumber model. When a task doesn't rely on memorization of the training set, it reasons similarly well to GPT-4. Where memorization is helpful, it performs worse (due to quantization-induced "memory loss"). This also makes me look at GPT-4 as a "weak reasoner with a lot of knowledge". That really aligns with my experience where it is immensely helpful and has a superhuman knowledge base but still needs handholding to solve real problems.
- jessenaser 3y agoThe thing is why does the GPT-4 Turbo and the Updated GPT 3.5 Turbo have only an output of 4,096 tokens? Previous Model: gpt-3.5-turbo-16k, 16385 tokens context and completion (shared) New Model: gpt-3.5-turbo-1106, 16385 tokens context, 4096 tokens completion Previous Model: gpt-4, 8192 tokens context and completion (shared) New Model: gpt-4-1106-preview, 128000 tokens context, 4096 tokens completion Why would the same size of a 16K GPT-3.5 model now not allow larger completion sizes? Why would the new GPT-4 reduce the completion tokens as well, gpt-4 can do 8192 and gpt-4-32k can do 32768 completion tokens. Now the limit is 4096. So you would need to change the way you prompt (split the responses) to be able to get a longer response. --- So are these new models taking the old base models of 4K tokens context and completion and changing the context to 128000 but leaving the completion the same? If they could get gpt-4 to have gpt-4-8k and gpt-4-32k, why couldn't have it been 128000 context and 32768 completion?
- srdjanr 3y agoProbably because it's too expensive. Prompt can be processed quickly but output tokens are much slower (and that means more expensive). From my local test on a 13B model, output tokens are 20-30x more expensive than input tokens. So OpenAI's pricing structure is based on expectation that there's much more input than output tokens in an average response. It didn't matter too much if a small percentage of requests used all 4k tokens for output, but with 128k it's a different story.
- Racing0461 3y agoI believe openai wants to lower the time it takes for requests to finish to be able to accept more requests per server/gpu. ie money.
- qup 3y agoif i'm not mistaken, the model has to be trained for a specific context window
- refulgentis 3y agoMore or less, like there's stuff you can do to extend the window of an existing model fairly easily, i.e. LoRA type training budget, O($1000). But in practice, even when context_size max output token count was enabled, it simply couldn't make use of it, no matter how many prompt engineering tricks I threw at it.[1] And I've heard anecdotally that it's true for that LoRA-type technique. [1] TL;DR, about 1/5th the actual length: write 100 pages, 3 paragraphs each, number the pages as you go and write 1 page at a time until 100. Also write out "I have written page N and need to write 100 pages total" after each page. Inevitably it would "get tired" and be like "end page 23...now page 100"
- cultureswitch 3y agoEveryone who has any knowledge of machine learning knows that you don't evaluate your model by testing it on parts of its training data. The issue is, the training data for GPT-4 appears to be "everything".
- driverdan 3y agoThis is a big problem with independent LLM testing. You need to make sure your test set isn't included in the training set which isn't easy with closed source models. This makes me think of how hardware manufacturers optimize for benchmarks. Closed source LLMs can intentionally include likely test data in their training set to artificially inflate results. I'm not saying they are intentionally doing that now, but they could.
- Andrew018 3y ago[dead]
- Racing0461 3y agoOpenAI should be covered in lawsuits for nerfing a product people paid for and expect to not degrade over time while still paying the same amount.
- maxrmk 3y agoI've always been skeptical of benchmarking because of the memorization problem. I recently made up my own (simple) date reasoning benchmark to test this, and found that GPT-4 Turbo actually outperformed GPT-4: https://open.substack.com/pub/talcai/p/making-up-a-new-llm-benchmark?r=2xfcc5&utm_campaign=post&utm_medium=web https://open.substack.com/pub/talcai/p/making-up-a-new-llm-b...
- tornato7 3y agoI like the test but do you take multiple samples / runs of a result? IMO for a proper benchmark you should ask it the same question 10+ times and show a confidence interval, otherwise you don't know if it's just a fluke or a lucky guess.
- maxrmk 3y agoAhh good suggestion, I should clarify this in the article. I tried to compensate with volume -- I used a set of 200 questions for the testing. I was using temperature 0, so I'd get the same answer if I ran a single question multiple times.