3 ms·
More specifically, they've beaten an old measurement on one specific benchmark. In my personal experience however GPT-4 is still ways ahead CodeLlama and its de
by M4v3R 3y ago
More specifically, they've beaten an old measurement on one specific benchmark. In my personal experience however GPT-4 is still ways ahead CodeLlama and its derivatives. They are really good for what they are, but you just can't beat the parameter count GPT-4 has and in more complex tasks it does show.
Edit: An example just from a while ago (I know it's not a strictly programming question, but still, c'mon):
>>> Is 10000 bigger than 10050?
> Yes, 10000 is bigger than 10050.
Edit 2: I just asked it to write a simple Python app to render a rotating 3d cube. This is something GPT-4 did almost flawlessly few months ago for me. I used the exact same prompt. I got a program that not only didn't work, but had syntax errors (mismatched parenthesis), referenced variables before assigning them, etc.
- alpaca128 3y agoWhy are people constantly using examples with number comparisons and calculations? That's the one use-case I'd never use an LLM for.
- brookst 3y agoIt’s like how people keep discovering that LLMs are bad at letter substitution ciphers. Yes, they are. Because you’re asking it to reason over the inputs and outputs of the embedding model, which inputs it never sees.
- M4v3R 3y agoBecause GPT-4 gets this right. Also please look at my 2nd edit which is a concrete programming example that Phind's model gets wrong on several levels.
- alpaca128 3y ago> Because GPT-4 gets this right And a Threadripper CPU can render Crysis, but that doesn't make it a meaningful metric for other processors. Maybe GPT4 happens to get it right in some cases, but in practice it's one of the most inefficient and unreliable ways to do calculations on a computer and is not an intended feature.
- TX81Z 3y agoArithmetic logic is *very much* a sought after goal for LLM. The “intended features” of any sufficiently advanced technology always move forward. GPT, Bard, etc all want to nail that issue and are dumping cash into it.
- mikeravkine 3y agoMismatched parenthesis, specifically using )) instead of ) is a common failure mode I observed in all of my codellama testing.
- gmm1990 3y agoI get the repeating close parenthesis a lot too. Figured it was due to my temperature settings, prompt style or quantization though, but this was my first time running one locally