5 ms·
It supports local models.
by Cyphase 4y ago
It supports local models.
- Zetobal 4y agoI wonder how people that don't read properly before doing something will fare in a world of text interfaces/ai.
- Lockal 4y agoThey will answer "Please don't comment on whether someone read an article, please review https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html (actually, a bad excuse, but it is what it is).
- manojlds 4y agoBetter? Because the AI will help them?
- groby_b 4y agoDoes it matter who generated the text people don't bother to read? And how will lack of attention to detail affect prompt generation?
- psychphysic 4y agoAre there any good local models? Gpt-2 is pants once you've gotten used to 4.
- danielbln 4y agoLlama, Alpaca, Dolly, Vicuna
- psychphysic 4y agoI just tried llama (30B) and alpaca. I'd say they are closer to GOT2.5
- newswasboring 4y agoGPT2 is ancient news. There are now local running models which, allegedly, can reach the same performance as GPT-4. Look up llama.cpp[1] and all the various community generated models. [1] https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp
- napier 4y agoVery much allegedly. They’re good, but not great.
- holografix 4y agoDoes llama.cop not run on GPU?
- MacsHeadroom 4y agollama.cpp is CPU only but llama runs on GPU using the HuggingFace Transformers library. You can run the llama model which far outpaces GPT-3.5 on a couple of $200 Tesla P40 GPUs at faster speeds than GPT-3.5 Turbo, completely locally.
- LoganDark 4y agoIt doesn't, it's CPU-only. -Emily
- yreg 4y agoIncredibly difficult to measure it, but it seems llama is closest to GPT-3[0]. Which is pretty cool all things considered. Certainly better than most of us would have expected a short while ago, right? [0] https://arxiv.org/pdf/2302.13971.pdf https://arxiv.org/pdf/2302.13971.pdf
- zxexz 4y agoI don’t know about out of the box, but I’ve been having a great time training my own domain-specific models from scratch and utilizing them locally. It does seem that the more domain specific a task is, far fewer params are needed. Having fun right now trying to build a model in GDELT, not much luck so far, but I’ve pushed less than 5% of the data through so far. I’ve also been experimenting on fine-tuning Llama on my personal data archives, which seems promising but it’s pretty expensive to do so. Hoping someone will release a ~13B param model of Llama that they’ve trained with transfer learning from the 65B llama model and other data. FWIW even the 7B llama model running through llama.cpp after being quantized performs (subjectively, but substantially) better than GPT2. This has been one of my most expensive, but also most rewarding, hobbies thus far.
- worldsayshi 4y agoWhat is the ballpark cost of this hobby, if I may ask? Do you train exclusively in the cloud?
- zxexz 4y agoReally depends. I’ve pretty much almost burned through all the cloud credits I’ve been able to find at this point. Probably spent 20 grand over a year in credits? Never more than a couple hundred of my own money. Still trying to find a better solution. I have some 2080 TIs that I use locally. I tend to prototype architectures locally then rent out some A100s for a few hours to see how it goes, continue training when I can spare the cash. I do a lot of CPU and sharded training too. I really wish there were better options for hobbyists to play around with this stuff.
- swyx 4y agogot a quick guide as to available cloud credits for GPUs?
- LoganDark 4y agoTraining domain-specific models from scratch is both very difficult and also not possible in some cases. Sometimes we want to generate some niche content where there isn't enough training data to create a domain-specific model. We would have to somehow find a model that can produce the niche content through understanding what we're asking rather than through actually being trained on it. Producing "understanding" is not something you can do at home unless you have a bunch of A100s and have studied machine learning for years. -Emily