3 ms·
If you want/need to go cpu only then llama.cpp, and the assorted front ends people are building for it, is looking like a good project: https://github.com/ggerg
by samwillis 3y ago
If you want/need to go cpu only then llama.cpp, and the assorted front ends people are building for it, is looking like a good project: https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp
- grimgrin 3y agoI'll append this reference too, as it's related and interesting: https://justine.lol/mmap/ https://justine.lol/mmap/
- segmondy 3y agoyour statement sounds like it doesn't support GPU, llama supports GPU as well. However llama is just code to run models. OP needs to find a model that works for them, there are lots of models out there.
- superkuh 3y agoIMO, the best out right now is https://huggingface.co/TheBloke/Wizard-Vicuna-30B-Uncensored-GGML https://huggingface.co/TheBloke/Wizard-Vicuna-30B-Uncensored... (in llama.cpp ggmlv3 format). A 30B llama fine-tuned on a mix of WizardLM and Vicuna training data. It does conversational interactions pretty well (the wizard) and handles instruction fine too (vicuna). But it is definitely no GPT3.5 or GPT4 replacement. It will not be good for getting "work" done or helping do tasks. It's for recreation. If you want a GPT3.5 level LLM, something akin to text-davinci-002, you'll need to do the SFT and RLHF fine tuning of the 65B llama yourself. And that's no small task. Neither is running a 65B model (even at 4 bits).