4 ms·
How expensive is it? My understanding is that it's not reasonable to train an LLM from scratch by yourself, and that if you want one that isn't just very stupid
by ftxbro 4y ago
How expensive is it? My understanding is that it's not reasonable to train an LLM from scratch by yourself, and that if you want one that isn't just very stupid then you need to spend between hundreds of thousands and hundreds of millions of dollars. But if you don't want to train from scratch then you can fine-tune existing models for cheaper.
- kkielhofner 4y agoLooking at what they're doing here probably not as much as you think. As you note, with the plethora of open/open-ish LLMs today and LoRA + PEFT you can fine tune with low VRAM and pretty quickly so even a single A100 or whatever cloud GPUs are just fine. I've even seen people pull it off in reasonable time on super cheap T4s, A10s, etc. I doubt anyone reading a blog post is attempting to train a "true" multi-billion param LLM from scratch.
- ftxbro 4y ago[flagged]
- kkielhofner 4y agoOuch. Ok, I'll try again because I read the article and watched the demo video from Mosaic: - MosaicML claims to have some magic that efficiently autoscales with awareness of specific instance per hour pricing. - MosaicML claims to have auto-optimize magic that makes training 3-7x faster. - The incentives seem to be aligned because their value prop is likely "we'll markup 10% but save you 50%" (or whatever). It's too bad Replit didn't provide the costs for this post in the same way Stanford, etc have done with Alpaca, Vicuna, etc. They probably didn't for from-scratch training because the answer to your question is almost certainly "it depends" or "if you have to ask you can't afford it" (for now).
- ftxbro 4y agoThanks I understand better now. Sorry my other comment was rude. Probably I'm just frustrated that I will never be able to make a sentient bot because it will cost literally a billion dollars, and I just have to sit on sidelines and watch the billionaires play with their sentient bots.
- dkhudia 4y agoDisclaimer: I work for MosaicML (MosaicML is the creator of the training platform used by Replit). Training these models from scratch on your domain specific data is not as expensive as one might think. We have provided some cost estimates in our blogs. https://www.mosaicml.com/blog/mosaicbert https://www.mosaicml.com/blog/mosaicbert https://www.mosaicml.com/blog/training-stable-diffusion-from-scratch-costs-160k https://www.mosaicml.com/blog/training-stable-diffusion-from... https://www.mosaicml.com/blog/gpt-3-quality-for-500k https://www.mosaicml.com/blog/gpt-3-quality-for-500k
- ftxbro 4y agoThank you this is very interesting!
- moltar 4y agoDo you have any examples on how to train a model that can write code but in a specific domain? Eg I only want to train it on a specific set of code. Eg let’s say functional React components in TypeScript.
- dskhudia 4y agoWe recently released 1B parameter model trained on a mix of data.[1] If you got your domain-specific data, our platform can cover the rest. [1]: https://twitter.com/jefrankle/status/1649060478910357504?s=46&t=tQQDgg2lGjgTPfUQ2RRmlA https://twitter.com/jefrankle/status/1649060478910357504?s=4...
- moltar 4y agoBut do you have any examples of how to do this? I am a pretty seasoned dev, but never trained a model before :)