13 ms·
CodeAlpaca – Instruction following code generation model
- gigel82 4y agoThe results are pretty good; I wish they'd just publish the models so we can run the inference locally (not too many people have access to 8xA100 to train themselves, though I appreciate including the training data and instructions too).
- jerrygenser 4y agoLlama may not be licensed for people to share since you need to apply to get one from Facebook for non commercial use. I think it's more of a license issue Hopefully similar work can be done with LoRA so the fine-tuning is not as expensive
- sahil_chaudhary 4y agoYeah the license issue is the only reason why I didn't release the weights yet.
- doctoboggan 4y agoIf a couple of us get together and throw in some money we could train it on Lambda Labs hardware like the OP suggests. I would volunteer to do it myself but I don’t know enough about training models to guarantee I am not wasting money with a stupid mistake.
- menzoic 4y agoAnyone with a few hundred bucks to spare can do it by renting GPUs from a cloud provider. It only cost Stanford $600 to create Alpaca from LLAMA. $100 to generate instructions with GPT-3 and $500 to rent cloud GPUs. The license restriction is due to the use of GPT-3 output to train a model.
- MacsHeadroom 4y agoMore like $50 or even $5 or less for the cloud GPUs. Alpaca-7B's compute costs were close to $50 and that was before the 100x cost savings of using LoRA. A 4bit LoRA fine tune of this project would cost less than $5 to train even up to 30B/33B.
- nickthegreek 4y agoI’d love to see a crowdsourcing platform to donate to specific fine-tuning projects. I would gladly throw some money at someone to do the labor and release the models to the public.
- sahil_chaudhary 4y agoGiven how many people are asking for the weights, I'm working to make them available now. So you'll be able to run them locally as well.
- nico 4y ago> The code runs on a 8xA100 80GB, but can also run on 8xA10040GB or 4xA100 with lower batch size and gradient accumulation steps. To get the GPUs, I suggest using Lambda Labs, best pricing for the best hardware. I wonder how much it was total in $ for the fine-tuning. Also, does anyone have some sort of table/formula that relates MB/GB of training data to $ for fine-tuning?
- menzoic 4y agoStanford only spent $500 to fine-tune LLAMA for humam instruction with 52k instructions generated by GPT-3. This probably costs less. The use of GPT to generate the instruction data instead of humans is the massive cost reduction. The actual training for fine-tuning on GPUs is relatively cheap.
- IanCal 4y agoMost of that was getting the data, the training would cost something like $100 if memory serves.
- NhanH 4y agoProbably in the hundreds of dollar for 7B model, and may be a thousand or two for the 13B at worst
- MacsHeadroom 4y agoFar far less. Alpaca-7B's compute cost was around $60-$70 for Stanford and around $0.60 (yes 60 cents) for equivalent fine tunes using the Parameter Efficient Fine Tuning (PEFT) strategy of Low Rank Adapters (LoRA). The repo above can be replicated for similar costs. Easily less than $10 for up to 30B using LoRA (which requires only 24GB of VRAM for 30B/33B and smaller).
- Art9681 4y agoI am interested in this. What would be the cost for the best model possible by the public?
- underlines 4y ago"Model weights aren't part of the release for now, to respect OpenAI TOS and LLaMA license." I feel like the whole Open Source ML scene is slowed down by a strong chilling effect. Everyone seems to be afraid to release models. Meanwhile, other models are freely available up to alpaca 30b: https://github.com/underlines/awesome-marketing-datascience/blob/master/llama.md#resources https://github.com/underlines/awesome-marketing-datascience/...
- politician 4y agoI don’t understand why Facebook hasn’t taken down the original magnet links in its own repository.
- menzoic 4y agoWhy would they?
- detrites 4y agoTo avoid the embarrassment of having to serve themselves with a DMCA?
- dchest 4y agoThe PR is not in their repository, it's in the fork. Pull Request refers to "pull this from my repository, here's a link", GitHub just presents it in a convenient interface. (But you're right, those who send DMCAs are likely to just a send a link to the original repository :)
- saurik 4y agoOne can assuredly delete the pull request from your own list of open pull requests, right? It isn't that the information exists in theory on GitHub: it is that it is still listed right there when you go to check the health of the project's open issues/PRs.
- 4y ago
- wsxiaoys 4y agohttps://github.com/wsxiaoys/awesome-ai-coding https://github.com/wsxiaoys/awesome-ai-coding Great time to start an awesome AI coding list!
- sharemywin 4y agoI'm curious during training if all the weights change compared LLaMA?
- sahil_chaudhary 4y agoYeah the weights did change.
- sharemywin 4y agoI wonder if this https://techcrunch.com/2022/12/20/petals-is-creating-a-free-distributed-network-for-running-text-generating-ai/ https://techcrunch.com/2022/12/20/petals-is-creating-a-free-... could be modified to create a system that's trained.
- tarruda 4y agoDidn't explore much, but it seems alpaca-lora has better results for coding tasks. One example I've used was: "Implement quicksort in python.". This is the result with Code alpaca: def quicksort(arr): if len(arr) < 2: return arr else: pivot = arr[len(arr) / 2] left_index = 0 right_index = len(arr) - 1 while left_index <= right_index: while arr[left_index] < pivot: left_index += 1 while arr[right_index] > pivot: right_index -= 1 if left_index <= right_index: temp = arr[left_index] arr[left_index] = arr[right_index] arr[right_index] = temp left_index += 1 right_index -= 1 return quicksort(arr[:left_index]) + [arr[left_index]] + quicksort(arr[right_index + 1:]) Alpaca lora result: def quicksort(arr): if len(arr) <= 1: return arr pivot = arr[0] left = [arr[i] for i in range(1, len(arr)) if arr[i] < pivot] right = [arr[i] for i in range(1, len(arr)) if arr[i] > pivot] return quicksort(left) + [pivot] + quicksort(right) Shorter and much cleaner, not to mention it works (code alpaca version is broken). Also it matches what ChatGPT generates for me.
- sahil_chaudhary 4y agoInteresting, in my experiments alpaca-lora performed worse but definitely possible that isn't the case generally.
- grph123dot 4y agoThe Alpaca lora result is incorrect since, for example, if all elements in array are the same the result in just an array with one element.
- xiphias2 4y agoYou are correct also it's not quicksort as it's not in place array, but needs a lot of memory management
- bestcoder69 4y agoAnyone know of a 65B instruct model I can run on llama.cpp? Or if it’s possible to train one on a 64gb Mac Studio? Haven’t come across either yet.
- sahil_chaudhary 4y agoI'm training a 65B model right now, also I believe you can use lora-alpaca to train on this data on a much smaller machine.
- cfn 4y agoLlama has a 65B model and it runs (albeit slowly) on a 64gb Mac Studio with llama.cpp.
- bestcoder69 4y agoYea I have that running. Looking specifically for a 65B instruction-tuned model. I just found a way to get alpaca 30B going though.
- pharmakom 4y agoIf someone were to add noise to the llama weights and then retrain a little, would anyone be able to tell? Could that org then pass it off as their own training, MIT licensed for the good of humanity?
- intalentive 4y ago>The model is not finetuned to be safe and harmless, so be cautious. Ok I'll be careful, but if it says a bad word, I'm sending you my therapy bill.
- flangola7 4y agoTherapy bill?