10 ms·
StarCoder and StarCoderBase: 15.5B parameter models with 8K context length
- cs702 4y agoIt's great to see this! A big THANK YOU to everyone who made it possible. I'm looking forward to playing with it -- and also, eventually, inevitably, running a quantized, super-efficient version on my laptop.
- freeqaz 4y agoLooks like the model is on HuggingFace here, for anybody that is curious to play with it. https://huggingface.co/bigcode/starcoder https://huggingface.co/bigcode/starcoder
- CodeCompost 4y agoSorry I'm a bit new to this. How does this work? Trying to read the site but have a hard time understanding this.
- chillbill 4y agoWelcome to the crappy ml world, where everything is clunky and barely works one time on one machine if you don’t touch anything on the computer, where software loses its meaning and saving Jupiter notebooks as py files is the norm, where the majority of “data scientist” means glorified labeling machines.
- pbmonster 4y agoIn their defense, stuff is moving forward at half the speed of light, so everybody has better things to do right now. Also, investing serious time here on cleaner processes and better documentation of their specific implementation seems like a waste of time. I don't think much of any of this will still be in use next year, everybody will have moved on to more advanced projects. But I agree, once things stabilize, the most popular models should invest in clean up a fair bit.
- meghan_rain 4y agotldr how does it compare to codepilot/gpt4?
- bavell 4y agoFrom the summary: "We perform the most comprehensive evaluation of Code LLMs to date and show that StarCoderBase outperforms every open Code LLM that supports multiple programming languages and matches or outperforms the OpenAI code-cushman-001 model." So I'd assume not up to par with gpt4 or copilot. Can't wait to see it evolve from here!
- tulip4attoo 4y agoGPT4 is ways ahead. On HumanEval, it gets 67%, almost double this one.
- deleted 4y ago[deleted]
- nr2x 4y agoGiven some of my own open source code is no doubt in GPT and Bard, which feels wrong given the fees and limitations, I’m VERY VERY excited for this!
- speedgoose 4y agoIt’s perhaps in the training dataset but unless your code is extremely common and duplicated, it’s probably not in the final models. They aren’t that big.
- ChatGTP 4y agoHmm, I still don't like this argument. Whether there are actual bits of the code in the model or not, his code is still in there somewhere, even if it's just an approximation. I feel quite similar personally, I've worked hard on open source and I'll never have the same permissive license again after this.
- nr2x 4y agoI asked it to write a function with a given name and it came up with stuff that was in the general ballpark of the software. So yeah, it’s in there.
- jimlongton 4y ago(Possibly naive question) This is marketed as open source. Does that mean I can download the model and run it locally? If so, what kind of GPU would I need?
- pyrophane 4y agoHere is a good reference: https://huggingface.co/docs/transformers/perf_train_gpu_one https://huggingface.co/docs/transformers/perf_train_gpu_one
- joaogante 4y agoA 3090 (or any GPU with >=20GB VRAM) can run StarCoder with int8 quantization at about 12 tokens per second, 33 with assisted generation -- which will come out for StarCoder in the coming days. When 4-bit quantization comes out, I would expect a GPU with 12GB VRAM to be able to run it. Disclaimer: I work at Hugging Face
- fbodz 4y agoHas anyone figured out a way to fine tune this with 24gb of vram? I have tried with deepspeed etc but no luck. Seems to be just out of reach for fine tuning requiring 26gb.
- csdvrx 4y agoHave you tried quantization? It's often a cheap and simple way to reduce the VRAM requirements. What hardware are you using? (CPU,RAM,GPU,VRAM) Have you considered using llama.cpp for a mixed CPU+GPU use (if you have enough RAM)
- fbodz 4y agoYeah I am using the default training script with int8 quantisation. It uses peft with lora but this still requires 26gb
- freeqaz 4y agoIs it possible to offload some layers to CPU and still train in a reasonable amount of time?
- generalizations 4y agoThere’s also that pruning tool that was on hn in the last couple weeks. It seemed to work really well on the larger models, and could reduce size by 30-50%
- int_19h 4y agoI'm not sure about this model specifically, but training with 4-bit quantization has been a thing with LLaMA for a while now, although the setup involves manual hacks of various libraries.
- nl 4y agoYou probably don't want to fine-tune a quantized model. They are fine for inference but not great for training.
- simonw 4y agoThis is trained on The Stack, which is available here: https://huggingface.co/datasets/bigcode/the-stack/ https://huggingface.co/datasets/bigcode/the-stack/ Interesting to note that The Stack is 6TB - the whole of the RedPajama LLM training set (a lot more than just code) is only 2.6TB. To get an idea what that training data looks like, I grabbed the first 300MB SQL file from https://huggingface.co/datasets/bigcode/the-stack/tree/main/data/sql https://huggingface.co/datasets/bigcode/the-stack/tree/main/... and then dumped the first 1,000 rows from that into JSON and loaded it into Datasette Lite: https://lite.datasette.io/?json=https://gist.github.com/simonw/acc4e6973d38aa18c445e7279023dab5#/data/raw?_facet_array=max_stars_repo_licenses&_facet=ext https://lite.datasette.io/?json=https://gist.github.com/simo... Here's a query that shows a random row - hit the blue "Run SQL" button to see another one: https://lite.datasette.io/?json=https://gist.github.com/simonw/acc4e6973d38aa18c445e7279023dab5#/data?sql=select+max_stars_repo_path%2C+content+from+raw+order+by+random%28%29+limit+1 https://lite.datasette.io/?json=https://gist.github.com/simo...
- vlovich123 4y agoSomething tells me that I haven't trained on 6 TB of code and can meaningfully outperform any AI. That tells me that there's something still structurally missing about the training efficiency. I wonder if this replicates to things like chess/go - for a computer trained on the same number of games that a human is, is the computer still able to outperform a human?
- mysterydip 4y agoI wonder how curated the input data is. Just on the surface of it, there's a lot of spaghetti code out there that people may have shared. I once saw a codebase that used three different implementations of a date/time structure and overloaded operators to convert between them. Or people rolling their own crypto, sort, or random functions, reimplementing data structures, etc.
- RangerScience 4y agoIs this training just to understand code, or is training to understand code and language? (If we're comparing you to the model, is the model starting at "baby" or "teenager"?)
- bootloop 4y agoThe biggest interest I have in this, is that I would like to have the ability to ask questions about large code-bases. I think being able to generate small functions or explain single code sections is nice, but being able to ask bigger architectural questions would be really helpful for all kind of engineers (in particular in a large company). I have seen approaches with merging context across multiple levels. But that can only do so much. Is it viable to fine-train a model to a specific code-base so it has knowledge across all files? Does anyone have more info on this kind of problem space?
- bradleyjg 4y agoExactly. I’d love to be able to ask where and how would I go about adding some new feature to a code base.
- heliophobicdude 4y agoRefined training is usually updating the weights of usually what's called a foundational model with well structured and numerous data. It's can be expensive but most importantly disturb the usefulness of having all the generalizations baked in from training data [1]. While LLMs can generate code based on a wide range of inputs, they're not designed to retrieve specific pieces of information in the same way that a database or a search engine would. It's just very lossy. Perhaps it wouldn't be the best use for single code base fine tuning right now. Can you please share more about the merging context across levels? This sounds interesting! 1: "Language Models are Few-Shot Learners" Brown et al. https://arxiv.org/pdf/2005.14165.pdf https://arxiv.org/pdf/2005.14165.pdf
- zellyn 4y agoSteve Yegge's recent blog posts claim that SourceGraph are getting a pretty good result by using embeddings created from their knowledge graph of the code structure. That's still the usual [create embeddings, search against embedding of query, retrieve results and use them as prompt] schlep, so yeah, it isn't really understanding architecture well yet. I too have a job where almost every question is about structural understanding and improvement of a large existing codebase. I'd love to have AI help, but I think it's going to take another iteration or three of model architecture to get there.
- ftxbro 4y agoDo I need to make an account on huggingface to get the model? I would prefer not to do it, and just download a zip like you can on github.
- version_five 4y agoI thought you didn't need an account to download from HF anymore. You can just do git lfs pull, at least for the stuff I've downloaded. Personally I'm concerned about how model hosting has been concentrated in one company, and was previously very unhappy that they required accounts, but I think that's past. Let me know if it's still the case for some things.
- ftxbro 4y agoIt is suggesting that I have to make an account. When I go to https://huggingface.co/bigcode/starcoder https://huggingface.co/bigcode/starcoder it says "You need to agree to share your contact informations to access this model" and "[Log in] or [Sign Up] to review the conditions and access this model content."
- version_five 4y agoYeah you're right, that's super lame. It used to be like that with stable diffusion but they took the HF login requirement off at some point. It's enough of a deal breaker for me to not bother using the model. Especially when it's developed by a company that (I assume) wants to harvest your contact info - unless there's some other explanation for the login requirement. (I tried git lfs clone and got asked for hf login credentials)
- reneberlin 4y agoIt's just because you should check / accept and align to the license, how i understood it. I do not believe in data harvesting ideas by them.
- 4y ago
- daneel_w 4y ago>"The Stack, a large collection of permissively licensed GitHub repositories with inspection tools and an opt-out process." >"The Stack is a collection of source code from repositories with various licenses. Any use of all or part of the code gathered in The Stack must abide by the terms of the original licenses, including attribution clauses when relevant." Does it have a view of what licenses can mix, or is it simply disallowed from crossing that boundary and only offer answers sourced entirely within the confines of this or that specific license? The latter poses some interesting scenarios and questions.
- freeqaz 4y agoPermissively licensed would imply non-copyleft to me. That means only licenses like Apache or MIT would be allowed to be train on, but not licenses like GPL.
- chaxor 4y agoIt's my understanding that GPLv3 is perfectly fine to make businesses on, you just have to make your code open source. There isn't anything wrong with that, and in today's age, I would actually suggest that's an enormous positive for a business, as it allows people to trust the company much more.
- nl 4y agoDetails are here: https://huggingface.co/datasets/bigcode/the-stack https://huggingface.co/datasets/bigcode/the-stack There are 193 licenses in total. v1.0 of The Stack included MPL/EPL/LGPL whereas v1.1+ doesn't include them.
- Imnimo 4y ago>We inspected StarCoder-generated programs on these benchmarks and found that there were several cases where the model produces what are effectively empty solutions, e.g., pass or a comment Insert code here. We also observed this kind of failure in every model we evaluated. I'm not sure whether the AI learning that it can just write "#TODO" is a sign our jobs are safe or a sign our jobs are truly in danger.
- fleischhauf 4y agosounds more like lazyness, I think we might be ok actually.
- hutzlibu 4y agoSounds more like it was trained on too much incomplete code.
- bionhoward 4y agoCould be a sign the thing knows how to break work into multiple pieces. If it wasn’t just 1-pass and you give it a couple turns to document / test / deliver, it definitely can fill in placeholders from the initial generative step when it does refinement. Language chains, not instant zero shot perfection
- heliophobicdude 4y agoI think we need a different strategy to instruction tuning for coding LLMs. I don't think StarCoderBase is instruction-tuned off the bat but would serve as a good starting point for a new technique. RLHF is fine for things that are hard to measure and evaluate, but code is runnable and testable. I propose we try Reinforcement Learning Machine Feedback or RLMF. Prompts and responses are evaluated by how accurate the response evaluated to. We can then train a reward model to help refine StarCoder.
- grandmczeb 4y agoGood idea, but pretty sure this is already widely done. For example, Alex Gravely (architect behind copilot) mentioned on No Priors[1] they would generate the implementations for tests in random github projects and check if they passed as feedback. [1] https://open.spotify.com/episode/2a8Rtm4mhjzennOoAByFKx https://open.spotify.com/episode/2a8Rtm4mhjzennOoAByFKx around 15:10
- superkuh 4y agoThis didn't generate anything like actual perl code but the paper did say it wasn't good at perl (relatively) and in their defense my code it was completing was full of regex. What I did enjoy was how it picked up on my style of extremely long variable and subroutine names without spaces. It even named them with swear words like I do.
- DanielShir 4y agoNow I really want to read your code... Anything public? :)
- ofermend 4y agoAll the code generation tools, StarCoder included, still have hallucinations. In this context code that looks good, but doesn't work or has a subtle bug. How do we address that?
- arthurcolle 4y agoVerification systems that then feed back into the models and correct hallucinations. It is slow but I think that's the only real way forward
- DavidKarlas 4y agoSounds same as when I started codding, hallucinated some code, compiler told me it is non-sense, and eventually I understood most of the rules... I still mess it up...
- arthurcolle 4y agoIf you keep it up, eventually you get good
- theaiquestion 4y ago> All the code generation tools, StarCoder included, still have hallucinations. This also includes humans. We "hallucinate" in very similar ways. For example mistaking localhost:8080 for localhost:8008 in a large config file. Attempting to use methods that were deprecated and no longer exist, etc. IMO there's two ways to prevent this is - one is to make better performing models (architecture/training data/training amount/etc) The other is the exact same as humans. Compile time tools that let it know immediately if it hallucinated, types, linting, tests, etc. You just do it as a loop the exact same as a human. You write code, the compiler tells you that method doesn't exist, you adjust your code/consult the documents (also doable with agents).
- lee101 4y ago[dead]
- VadimPR 4y agoThis is great - we needed a model where we're sure it won't reproduce someone's code with an incompatible license.
- pabs3 4y agoPermissive licenses usually have attribution requirements, does using this mean you have to attribute all the projects from The Stack?
- pplanel 4y agoIt sucks at Rust fn convert_ogg_to_w av(input: Path) -> Result<
- ipsum2 4y agoI've been playing with StarCoder for the last week. It performs great once fine-tuned. Highly recommend people use it as a base model for anything, not just coding.
- buraksarica 4y agoIs Danish Contractor a real person? :D
- veselin 4y agoWhat speed should we expect from the model on consumer hardware? I tried a 8 bit quantized version on 4090 and got it to generate 100 tokens for 13 second, which seems a bit slow to me.
- pfd1986 4y agoCode LLMs and they didn't call it CLLaMs? :_(
- YeGoblynQueenne 4y agoAs usual, concrete results are poor, with best results obtained for Python, reported on HumanEval at 40.8% and 52.7% on MBPP, where previous best was 33.5 and 45.9 respectively (both by original copilot model). Results on DS-100 (simle data science programming tasks) are much more modest, at around 30%. And all this despite the "pass@k" evaluation metric, which is very misleading: it's clearly selected to make a code generating model look its absolute best. For example, the "pass@1" metric is _estimated_ not by choosing a single solution generated by a model, for a given programming task, and checking whether the solution completes the programming task correctly, but by generating a single solution multiple times (200 or 20, depending on model) and then averaging over them. So while it's called "pass-at-one" the "one" is actually a bunch of randomly drawn samples and not a single solution. Like I say, very misleading. See Section 6.1.1 in the paper.
- lma21 4y agoHow can I use StarCoder as a developer using vim or Visual Studio Code?
- joaogante 4y agovs code: https://marketplace.visualstudio.com/items?itemName=HuggingFace.huggingface-vscode https://marketplace.visualstudio.com/items?itemName=HuggingF... vim: https://github.com/huggingface/hfcc.nvim https://github.com/huggingface/hfcc.nvim
- kannangce 4y agoHas anyone tried if the model is capable of giving a structured output?