4 ms·
How much GPU memory do you have access to? If you can run it, Guanaco-65B is probably as close as you can get in terms of something publicly available. https://
by amilios 3y ago
How much GPU memory do you have access to? If you can run it, Guanaco-65B is probably as close as you can get in terms of something publicly available. https://github.com/artidoro/qlora https://github.com/artidoro/qlora. But as other comments mention, it's still noticeably worse in my experience.
- Taek 3y agoAnother vote from me for guanaco 65b. Here's a link to the model and weights: https://huggingface.co/TheBloke/guanaco-65B-GPTQ https://huggingface.co/TheBloke/guanaco-65B-GPTQ I use it all the time at home. It's decent at things like summaries and writing content. It follows instructions as well as GPT and isn't nearly as pretentious but it's still a bit pretentious. It's not very good at code.
- npteljes 3y agoI was wondering how much GPU memory does G-65B need, and from the docs, it's "48GB".
- amilios 3y agoIs this with the 4-bit quantization? The only issue with it is that inference is incredibly slow with it on right now, but that should be fixed up in the next few weeks I think.
- netsec_burn 3y agoI'm running it on Serge with 51 GB of RAM. I don't think it requires any GPU memory (llama.cpp runs on the CPU), I have a RTX 2060 in the system I'm running it on. Correct me if I'm wrong.
- forgingahead 3y agoHow many tokens per second are you getting with your setup?
- netsec_burn 3y agoAbout .5 tokens per second if I had to guess.