3 ms·
your statement sounds like it doesn't support GPU, llama supports GPU as well. However llama is just code to run models. OP needs to find a model that works
by segmondy 3y ago
your statement sounds like it doesn't support GPU, llama supports GPU as well. However llama is just code to run models. OP needs to find a model that works for them, there are lots of models out there.
- superkuh 3y agoIMO, the best out right now is https://huggingface.co/TheBloke/Wizard-Vicuna-30B-Uncensored-GGML https://huggingface.co/TheBloke/Wizard-Vicuna-30B-Uncensored... (in llama.cpp ggmlv3 format). A 30B llama fine-tuned on a mix of WizardLM and Vicuna training data. It does conversational interactions pretty well (the wizard) and handles instruction fine too (vicuna). But it is definitely no GPT3.5 or GPT4 replacement. It will not be good for getting "work" done or helping do tasks. It's for recreation. If you want a GPT3.5 level LLM, something akin to text-davinci-002, you'll need to do the SFT and RLHF fine tuning of the 65B llama yourself. And that's no small task. Neither is running a 65B model (even at 4 bits).