3 ms·
"Yes", but it's going to be unbearably slow on CPU without some form of GPU acceleration. You'll probably get more info/responses from https://www.reddit.com/r
by hnfong 3y ago
"Yes", but it's going to be unbearably slow on CPU without some form of GPU acceleration.
You'll probably get more info/responses from https://www.reddit.com/r/LocalLLaMA/ https://www.reddit.com/r/LocalLLaMA/
- rom16384 3y agoUsing llama.cpp on a 128 GB server running codellama-70b-python.Q4_K_M.gguf I get 1.3 tokens per second which is too slow. With Nous-Hermes-2-Mistral-7B-DPO.Q5_K_M.gguf I get 8.3 tokens per second which is usable.