3 ms·

Use llama.cpp to run models locally.

by kardianos 3y ago

turnsout 3y ago
Can llama.cpp run this yet? That would be surprising
- loudmax 3y ago
  It would be very surprising. Mistral AI's github page has more information on their sliding window attention method to achieve this performance: https://github.com/mistralai/mistral-src https://github.com/mistralai/mistral-src If Mistral 7b lives up to the claims, I expect these techniques will make their way into llama.cpp. But I would be surprised if the required updates were quick or easy.
- daakus 3y ago
  It can! TheBloke is to thank for the incredibly quick turnaround. https://github.com/ggerganov/llama.cpp/pull/3362 https://github.com/ggerganov/llama.cpp/pull/3362 https://huggingface.co/TheBloke/Mistral-7B-v0.1-GGUF/tree/main https://huggingface.co/TheBloke/Mistral-7B-v0.1-GGUF/tree/ma...
  - turnsout 3y ago
    Wow, awesome!
  - aidenn0 3y ago
    I have yet to get any useful output out of the Q5_K_S version; haven't tried any others yet.
    - daakus 3y ago
      I used mistral-7b-v0.1.Q5_K_M.gguf and it responded to basic questions.
    - poser-boy 3y ago
      Linked is the base model. What you want is the instruct model (also on TheBloke's profile), which has been trained on following instructions.
  - moffkalast 3y ago
    Birds fly, sun shines, and TheBloke always delivers. Though I can't figure out that prompt and with LLama2's template it's... weird. Responds half in Korean and does unnecessary numbering of paragraphs. Just one big sigh towards those supposed efforts on prompt template standardization. Every single model just has to do something unique that breaks all compatibility but has never resulted in any performance gain.
    - daakus 3y ago
      I used the prompt included in llama.cpp and it worked for me in English (for fun GK type questions): MODEL=./models/mistral-7b-v0.1.Q5_K_M.gguf N_THREAD=16 ./examples/chat-13B.sh
- programd 3y ago
  I'm getting about 7 tokens per sec for Mistral with the Q6_K on a bog standard Intel i5-11400 desktop with 32G of memory and no discrete GPU (the CPU has Intel UHD Graphics 730 built in). So great performance on a cheap CPU from 2 years ago which costs, what $130 or so? I tried Llama.65B on the same hardware and it was way slower, but it worked fine. Took about 10 minutes to output some cooking recipe. I think people way overestimate the need for expensive GPUs to run these models at home. I haven't tried fine tuning, but I suspect instead of 30 hours on high end GPUs you can probably get away with fine tuning in what, about a week? two weeks? just on a comparable CPU. Has anybody actually run that experiment? Basically any kid with an old rig can roll their own customized model given a bit of time. So much for alignment.

Use llama.cpp to run models locally.

turnsout 3y ago

Can llama.cpp run this yet? That would be surprising

loudmax 3y ago
It would be very surprising. Mistral AI's github page has more information on their sliding window attention method to achieve this performance: https://github.com/mistralai/mistral-src https://github.com/mistralai/mistral-src If Mistral 7b lives up to the claims, I expect these techniques will make their way into llama.cpp. But I would be surprised if the required updates were quick or easy.
daakus 3y ago
It can! TheBloke is to thank for the incredibly quick turnaround. https://github.com/ggerganov/llama.cpp/pull/3362 https://github.com/ggerganov/llama.cpp/pull/3362 https://huggingface.co/TheBloke/Mistral-7B-v0.1-GGUF/tree/main https://huggingface.co/TheBloke/Mistral-7B-v0.1-GGUF/tree/ma...
- turnsout 3y ago
  Wow, awesome!
- aidenn0 3y ago
  I have yet to get any useful output out of the Q5_K_S version; haven't tried any others yet.
  - daakus 3y ago
    I used mistral-7b-v0.1.Q5_K_M.gguf and it responded to basic questions.
  - poser-boy 3y ago
    Linked is the base model. What you want is the instruct model (also on TheBloke's profile), which has been trained on following instructions.
- moffkalast 3y ago
  Birds fly, sun shines, and TheBloke always delivers. Though I can't figure out that prompt and with LLama2's template it's... weird. Responds half in Korean and does unnecessary numbering of paragraphs. Just one big sigh towards those supposed efforts on prompt template standardization. Every single model just has to do something unique that breaks all compatibility but has never resulted in any performance gain.
  - daakus 3y ago
    I used the prompt included in llama.cpp and it worked for me in English (for fun GK type questions): MODEL=./models/mistral-7b-v0.1.Q5_K_M.gguf N_THREAD=16 ./examples/chat-13B.sh
programd 3y ago
I'm getting about 7 tokens per sec for Mistral with the Q6_K on a bog standard Intel i5-11400 desktop with 32G of memory and no discrete GPU (the CPU has Intel UHD Graphics 730 built in). So great performance on a cheap CPU from 2 years ago which costs, what $130 or so? I tried Llama.65B on the same hardware and it was way slower, but it worked fine. Took about 10 minutes to output some cooking recipe. I think people way overestimate the need for expensive GPUs to run these models at home. I haven't tried fine tuning, but I suspect instead of 30 hours on high end GPUs you can probably get away with fine tuning in what, about a week? two weeks? just on a comparable CPU. Has anybody actually run that experiment? Basically any kid with an old rig can roll their own customized model given a bit of time. So much for alignment.