3 ms·
Got stuck on the model same as you did. No idea what model to use, not interested in fighting with Python to convert the models. I was able to get llama.cpp it
by markdog12 3y ago
Got stuck on the model same as you did. No idea what model to use, not interested in fighting with Python to convert the models.
I was able to get llama.cpp itself to work, though, including image analysis.
- francisduvivier 3y agoI also had the same issue, in my case it was because I was trying to use a llama 2 model. When trying with codellama https://huggingface.co/TheBloke/CodeLlama-7B-GGUF/tree/main https://huggingface.co/TheBloke/CodeLlama-7B-GGUF/tree/main, which is based on the first llama, it works.
- francisduvivier 3y agoCorrection: looks like it has to do with the quantization rather: 8bit quantization works while less does not not seem to work. Other working model example (no conversion needed): https://huggingface.co/TheBloke//Yarn-Mistral-7B-64k-GGUF/yarn-mistral-7b-64k.Q8_0.gguf https://huggingface.co/TheBloke//Yarn-Mistral-7B-64k-GGUF/ya...