5 ms·
The model is way too small, comparing it to Codex feels disingenous. Sure it's 77% smaller, it's also 77% worse. Although, it's a cool project nonetheless. For
by doodlesdev 3y ago
The model is way too small, comparing it to Codex feels disingenous. Sure it's 77% smaller, it's also 77% worse. Although, it's a cool project nonetheless.
For instance, even this simple snippet generates wrong inline completions:
// Only return even numbers bigger than 10 from the array
const arrayFilter = (array) =>
Replit-code-v1:
// Only return even numbers bigger than 10 from the array
const arrayFilter = (array) => {
return array.filter((item) => item > 10);
};
Gets it wrong, returns odd numbers.
Codeium:
// Only return even numbers bigger than 10 from the array
const arrayFilter = (array) => {
return array.filter((num) => num > 10 && num % 2 === 0);
};
ChatGPT (GPT-3.5 Turbo) - Code-only, without the rest of the completion since it's instruction-tuned:
const arrayFilter = (array) => {
return array.filter(num => num % 2 === 0 && num > 10);
}
Not comparable at all. For reference if anyone wants to test I ran this through the HuggingFace space using the default parameters, ChatGPT through chat.openai.com, and Codeium through the VSCodium extension on an empty JavaScript file.
- amasad 3y agoInteresting. This seems like a weakness of natural language understanding. If you rephrase your prompt slightly it would get it right. Try: // return even numbers that are also more than 10 const arrayFilter = (array) => It would do the right thing. The fine-tuned version gets your prompt right so maybe it benefited from natural language data. Will look more into it.
- doodlesdev 3y agoThat's really interesting, indeed I can reproduce this by changing the comment. I also managed to get correct output for this sample by renaming the function.
- eevilspock 3y agoclearly your original comment was unfair.
- int_19h 3y agoIs it, though? The major selling point of coding LLMs is that you can use natural language to describe what you want. If minor changes to wording - the ones that would not make any difference with a human - can result in drastically worse results, that feels problematic for real-world scenarios.
- throwaway675309 3y agoThe criticism stands if you have to continue to rewrite your "prompt" until you can coax out the correct desired output.
- SCLeo 3y agoI agree. Maybe it interpreted it as return the numbers that are more than 10 in the given array of even numbers. For example, if the instruction says "return person objects that are at least 20 years old", it might be more reasonable to generate: array.filter(item => item.age >= 20) as oppose to array.filter(item => (item instanceof Person) && (item.age >= 20))
- SheinhardtWigCo 3y agoIt seems like every week someone comes out with some version of "we can get results similar to OpenAI's API with our model that you can run on a Commodore 64!" And then you dig in, and it's always far behind in some important way. Not hating here, I love the pace of iteration, just not the hyperbole.
- barking_biscuit 3y ago>"we can get results similar to OpenAI's API with our model that you can run on a Commodore 64!" I have felt similar frustrations with statements that feel disingenuous too. Thanks for articulating this with such a beautifully hilarious metaphor.
- thewataccount 3y agoI need more time to compare it, the short 128 tokens in the demo is a bit rough but - On first look this seems to blow the current llama based models out of the water including the 30B ones. Pasting what you want + url + example json with no other context and it "knows" what the url and the json is for, without even telling it. I'm not even saying it's as good as chatGPT, but this is a tenth the size of the best llama models I've seen.
- johnfn 3y ago> Sure it's 77% smaller, it's also 77% worse. Hehe, yeah, imagine saying you made a new programming language with 77% less lines of code than Python.
- Zababa 3y agoFinally, an opportunity to share this https://nsl.com/papers/denial.html https://nsl.com/papers/denial.html
- deleted 3y ago[deleted]
- johnfn 3y agoI’m curious about the downvotes because I thought I was just agreeing with OP. Obviously lines of code in a programming language repo is no correlate at all to quality. It’s like the old adage about measuring aircraft quality by weight.
- barking_biscuit 3y agoI didn't get the punchline of this, so I asked GPT-4 to explain the punchline. Actually quite amusing.
- IshKebab 3y agoFrom the context it presumably finds the maximum of an array in K. Also quite a nice demonstration of why K is a bad language. For writing maintainable software at least; for code golf it's clearly amazing! (And maybe interactive calculator style usage?)
- moffkalast 3y agoYeah I tried the demo, it wrote some wrong code with comments in Chinese. I think I'll pass. It's a pretty well accepted fact now that bigger LLM = moar better without exceptions. I'm not sure why there's a race to the bottom of who'll make the most useless model that can run everywhere.
- thewataccount 3y ago> It's a pretty well accepted fact now that bigger LLM = moar better without exceptions. That's not true, the amount of training is a MAJOR factor. See the Chinchilla paper - https://arxiv.org/abs/2203.15556 https://arxiv.org/abs/2203.15556 tl;dr - a "fully" trained small model can outperform a "undertrained" larger model. If you have a fixed amount of compute (budget), then you need to optimize for the largest model that you can fully train, and not simply up the parameter count. EDIT: Also you can't necessarily compare the parameter count across model architectures* This thing seems to outperform the finetuned 30B llama models I've seen.
- moffkalast 3y agoWell if you're set on training something for a specific budget then it does of course make sense to pick the most optimal model size, true. But the problem is that these models don't exist in a vacuum, and have to go against slightly larger ones that are also compute optimal and use more data, which will definitely perform better. Then again maybe there is a sweet spot for a model that's small enough to run effortlessly on regular machines while only serving as a control node in an autoGPT style setting, where it fetches the context it can't possibly have from a curated online database to make up for its shortcomings.
- thewataccount 3y ago> But the problem is that these models don't exist in a vacuum, and have to go against slightly larger ones that are also compute optimal and use more data, which will definitely perform better. They don't have to go against those though. Most of these models are research models, either from academia or from companies experimenting to see what works. From my understanding, most of these are a - "We have X amount of USD for the next month or so, we'll try a few things, then whatever our best bet is we'll stick the time out on that". Very few companies have the resources to train big models with as much compute as Google/OpenAI/Microsoft/Facebook. These are also not being monetized as they're open source. Going from their 2.7B model to 10B would be ~10X the compute (FLOPS) required for an optimal model. And this is likely their first open model and not their last, since Replit likely doesn't have the budget that openai does it makes sense they didn't want to blow their entire year's budget on their first open model. 2.7B would also be a really nice if anyone can get it working because it's more likely to be able to run in the IDE at that point instead of needed a massively scaled cloud (which might be valuable for replit).