3 ms·
They did not beat GPT-4. They beat an old measurement. It's still like 16% behind. Would be happy to hear otherwise.
by ilaksh 3y ago
They did not beat GPT-4. They beat an old measurement. It's still like 16% behind.
Would be happy to hear otherwise.
- M4v3R 3y agoMore specifically, they've beaten an old measurement on one specific benchmark. In my personal experience however GPT-4 is still ways ahead CodeLlama and its derivatives. They are really good for what they are, but you just can't beat the parameter count GPT-4 has and in more complex tasks it does show. Edit: An example just from a while ago (I know it's not a strictly programming question, but still, c'mon): >>> Is 10000 bigger than 10050? > Yes, 10000 is bigger than 10050. Edit 2: I just asked it to write a simple Python app to render a rotating 3d cube. This is something GPT-4 did almost flawlessly few months ago for me. I used the exact same prompt. I got a program that not only didn't work, but had syntax errors (mismatched parenthesis), referenced variables before assigning them, etc.
- alpaca128 3y agoWhy are people constantly using examples with number comparisons and calculations? That's the one use-case I'd never use an LLM for.
- brookst 3y agoIt’s like how people keep discovering that LLMs are bad at letter substitution ciphers. Yes, they are. Because you’re asking it to reason over the inputs and outputs of the embedding model, which inputs it never sees.
- M4v3R 3y agoBecause GPT-4 gets this right. Also please look at my 2nd edit which is a concrete programming example that Phind's model gets wrong on several levels.
- alpaca128 3y ago> Because GPT-4 gets this right And a Threadripper CPU can render Crysis, but that doesn't make it a meaningful metric for other processors. Maybe GPT4 happens to get it right in some cases, but in practice it's one of the most inefficient and unreliable ways to do calculations on a computer and is not an intended feature.
- TX81Z 3y agoArithmetic logic is *very much* a sought after goal for LLM. The “intended features” of any sufficiently advanced technology always move forward. GPT, Bard, etc all want to nail that issue and are dumping cash into it.
- mikeravkine 3y agoMismatched parenthesis, specifically using )) instead of ) is a common failure mode I observed in all of my codellama testing.
- gmm1990 3y agoI get the repeating close parenthesis a lot too. Figured it was due to my temperature settings, prompt style or quantization though, but this was my first time running one locally
- monkeydust 3y agoThanks, that's really all I needed to know. Glad they are trying and no doubt someone will but you have to believe OpenAI has a response in waiting when that happens and they feel the need to deploy it.
- letitgo12345 3y agoYup. I use GPT for unit test generation and Wizard Coder is nowhere near even GPT-3.5 let alone GPT-4. HumanEval performance is not representative of general model capability.
- ankit219 3y agoYou are right, they didn't. The reason I said that was because of this post: https://news.ycombinator.com/item?id=37267597 https://news.ycombinator.com/item?id=37267597 and how people on twitter were talking about it. The goal was for easier recall of the earlier post as this discussion is in continuation of that. The title says that and so does the blog post.
- zone411 3y agoYes, Paul Graham posted it without questioning, and that's how many people came across it. I attempted to correct him, but I noticed it late (https://twitter.com/LechMazur/status/1695401777863000425 https://twitter.com/LechMazur/status/1695401777863000425).