10 ms·
BigCode Project Releases StarCoder: A 15B Code LLM
- enum 3y agoStarCoder is a 15B LLM for code with 8k context and trained only on permissive data in 80+ programming languages. Release thread: https://twitter.com/BigCodeProject/status/1654174941976068119 https://twitter.com/BigCodeProject/status/165417494197606811... Model: https://huggingface.co/bigcode/starcoder https://huggingface.co/bigcode/starcoder Paper: https://drive.google.com/file/d/1cN-b9GnWtHzQRoE7M7gAEyivY0kl4BYs/view https://drive.google.com/file/d/1cN-b9GnWtHzQRoE7M7gAEyivY0k... Also available in HuggingChat: https://huggingface.co/chat https://huggingface.co/chat
- espadrine 3y agoA similar model from Replit was released this week: https://huggingface.co/replit/replit-code-v1-3b https://huggingface.co/replit/replit-code-v1-3b The performance of StarCoder seems superior on HumanEval pass@1: • replit-code-v1-3b: 21.9% • replit-finetuned-v1-3b: 30.5% • StarCoder: 33.6% • StarCoder prompted: 40.8%
- enum 3y agoAlso note that StarCoder itself is open (but not replit-finetuned).
- kartayyar 3y agoThe quality delta is so huge that it's in a different league. Replit's model seems to have focused on being cheap to train and run. StarCoder seems to be vastly better on quality.
- sillysaurusx 3y agoIf so, it’s a shame this post is getting comparatively little attention. Hopefully the quality will be enough for it to make a big splash somewhere.
- sauwan 3y agoI'm assuming higher is better? How does this compare to GPT-3.5/4?
- qwerty3344 3y agoI think it's still significantly behind GPT 3.5/4, both of which can get 67% on HumanEval, and 88% with Reflexion
- enum 3y agoKeep in mind that StarCoder(Base) is just a pretrained LM. The extra stuff that makes 3.5/4 like RLHF gets built on this.
- manojlds 3y agoAren't GPT-3 etc base LM and ChatGPT the instruction tuned? Or am I wrong?
- dpf 3y agocode-davinci-002 is a base LM, and the other 3.5 models (text-davinci-{002,003}, gpt-3.5-turbo, and ChatGPT) use instruction tuning and/or RLHF. Source: https://platform.openai.com/docs/model-index-for-researchers https://platform.openai.com/docs/model-index-for-researchers
- qwerty3344 3y agohttps://newatlas.com/technology/gpt-4-reflexion/ https://newatlas.com/technology/gpt-4-reflexion/
- djoldman 3y agoPaper: https://drive.google.com/file/d/1cN-b9GnWtHzQRoE7M7gAEyivY0kl4BYs/view https://drive.google.com/file/d/1cN-b9GnWtHzQRoE7M7gAEyivY0k... Evaluation results are impressive!
- anentropic 3y agoThe org page has a link to https://huggingface.co/bigcode/starencoder https://huggingface.co/bigcode/starencoder "Encoder model trained on TheStack." which is currently a 404 Just curious what that is vs the main https://huggingface.co/bigcode/starcoder https://huggingface.co/bigcode/starcoder model
- enum 3y agoThis was used for PII detection and will be public soon. It could have other uses, but its not what you want code generation. For code generation, use StarCoder or StarCoderBase.
- anentropic 3y agoI was wondering about maybe embeddings for code search
- captn3m0 3y agoLicense text: https://huggingface.co/spaces/bigcode/bigcode-model-license-agreement https://huggingface.co/spaces/bigcode/bigcode-model-license-.... Attachment A at the very end has the usage restrictions.
- hardwaresofton 3y agoVery interesting read. OK for some commercial uses but I wonder how many other models have restriction lists like this
- riedel 3y agoSome parts of ethical licensing seem to finally pick up momentum, seems like they are using a standard license: https://www.licenses.ai/blog/2022/8/26/bigscience-open-rail-m-license https://www.licenses.ai/blog/2022/8/26/bigscience-open-rail-... To me it would be even more responsible if they would define a default output license compatible to training input of the model. I guess they could simply refer to the input data set in there to fulfill attribution clauses of most permissive licenses. Or extract some really long list of copyright notices from the sources.
- hardwaresofton 3y agoWell rather than "ethical" licensing, I'd argue this is "moral" licensing, and as usual, morals differ. For example: > (m) To provide medical advice or medical results interpretation that is intended to be a substitute for professional medical advice, diagnosis, or treatment; This seems particularly overly broad -- there is a lot of benefit for people with no access to a doctor of any kind that is left on the ground by excluding that use case. The problem of bad advice is hard, but it's one that will likely become increasingly solved as the tech evolves, and this seems to exclude this model from that technological progression.
- captn3m0 3y agoDoes this work for second degree usecases? Since this a codegen model, can I output code that does something like fast-edge detection, put it under MIT, and then use it for medical analysis?
- anentropic 3y agohttps://huggingface.co/spaces/bigcode/bigcode-playground https://huggingface.co/spaces/bigcode/bigcode-playground Can someone explain to me the "Model Formats" section at the end of that page. It makes sense to me as a description of how to format the training data, but it also says "Use these templates to explore the model’s capacities" How would you make use of this prefix format in a prompt? <reponame>REPONAME<filename>FILENAME<gh_stars>STARS code<|endoftext|>
- enum 3y agoYou may (but do not have to) use <reponame>, <filename>, etc. as special tokens to prompt the model with extra metadata. These help you use the model go beyond just code completion. Page 30 of the TR has a few examples: https://drive.google.com/file/d/1cN-b9GnWtHzQRoE7M7gAEyivY0kl4BYs/view https://drive.google.com/file/d/1cN-b9GnWtHzQRoE7M7gAEyivY0k... Note that they were produced with StarCoderBase. E.g., here is one: Input: <commit_before>def fibonacci(n):<commit_msg>add type hints to function<commit_after>def Output: fibonacci(n: int) -> int: (If you try this in the playground, dial down the repetition penalty to say 1.0 in Advanced Settings.)
- espadrine 3y agoStarCoder is not just a code predictor, it is an assistant. It doesn’t just predict code; it can also help you review code and solve issues using metadata, thanks to being trained with special tokens. The <reponame> token specifies the name of the repository, and the same goes for the filename. A high gh_stars count can make it mimick popular open-source libraries. Admittedly, there is a bit of information that can be retrieved from those tokens, but not as much as the others. The more interesting ones are the commit tokens, which you can use to ask it as an assistant to implement a commit whose high-level description you give in plain English, and the issues token, to make it solve an issue. The Fill-in-the-middle tokens are most useful for autocompletion: it will allow it to gain information from the code that is present after your cursor, instead of just the code before your cursor. Of course, in practice, those tokens are meant for code editor plugin writers. Normal users won’t know about them.
- kateklink 3y agoI'm excited to see all the model's implementations that are yet to come. Refact.ai seems to be working on code transformation tools and chat inside Jetbrains and VSCode powered by StarCoder https://refact.ai/blog/2023/self-hosted-15b-code-model/ https://refact.ai/blog/2023/self-hosted-15b-code-model/
- eclark 3y agoThe training cluster is super impressive, any ideas on how to fine tune StarBase for a smaller language than python on something more attainable?
- syntaxing 3y agoSo many things are 404 on this announcement. The demo looks really amazing though, I’m pretty excited to try this out locally.
- tmaly 3y agoI might have missed it, what are the minimum hardware requirements needed to run this?
- enum 3y agoYou can just about load it on a 32GB GPU in 16bit mode. Quantized versions here: https://huggingface.co/mayank31398/starcoder-GPTQ https://huggingface.co/mayank31398/starcoder-GPTQ they will be benchmarked on humaneval and released soon—maybe tomorrow?
- SomeHacker44 3y agoi was not able to find the 80 programing languages it supports using my phone. you would think this would be front and center somewhere.
- enum 3y agoThere is a lot of subtlety here. The model is trained on 80+ languages, but the volume and quality of data varies significantly. We have results showing benchmark performance on 19 languages, which is a broader evaluation than most Code LLMs. But, I would hesitate to say that BigCode supports 80 PLs. Any LLM that claims to support 80 PLs is not presenting evidence that it does.
- EvgeniyZh 3y agoYou are correct that there is no evaluation of level of support, and it is hard to get evaluation set on 86 languages. On the other hand, we can try to extrapolate from what we have and try to guess in which languages we'd have reasonable performance. Note that out of 19 languages it was evaluated on only 17 were "officially" in the training set (language detection is not perfect and there may be some data for languages not included in training set) and it work reasonably well on the remaining two (Swift and D).
- kannangce 3y agoIs it possible to load the model for a particular language. Thus minimising the required resources.