4 ms·
From the blog article: > A few pip install X’s and you’re off to the races with Llama 2! Well, maybe you are, my dev machine doesn’t have the resources to resp
by thrdbndndn 3y ago
From the blog article:
> A few pip install X’s and you’re off to the races with Llama 2! Well, maybe you are, my dev machine doesn’t have the resources to respond on even the smallest model in less than an hour.
I never tried to run these LLMs on my own machine -- is it this bad?
I guess if I only have a moderate GPU, say a 4060TI, there is no chance I can play with it, then?
- Eisenstein 3y agoYou can load a 7B parameter model quantized at Q4_K_M as gguf. I don't know ollama, but you can load it in koboldcpp -- use cuBLAS and gpu layers 100 context 2048 and it should fit it all into 8GB of VRAM. For quantized models look at TheBloke on huggingface -- Mistral 7B is a good one to try.
- dizhn 3y agoIf I am not mistaken, layer offloading is a llama.cpp feature so a lot of frontends/loaders that use it also have it. I use it with koboldcpp and text-generation-webui.
- pitched 3y agoI would expect that 4060ti to get about 20-25 tokens per second on Mixtral. I can read at roughly 10-15 tokens per second so above that is where I see diminishing returns for a chatbot. Generating whole blog articles might have you sit waiting for a minute or so though.
- thrdbndndn 3y agoThanks, that sounds more than tolerable than "more than an hour"! I also have the 16GB version, which I assume would be a little bit better.
- cellis 3y agoIt depends on the context window, but my 3090 gets ~60/s on smaller windows.
- visarga 3y agoI get 50-60t/s on Mistral 7B on 2080 Ti
- jsjohnst 3y agoThe Apple M1 is very useable with ollama using 7B parameter models and is virtually as “fast” as ChatGPT in responding. Obviously not same quality, but still useful.
- jwr 3y agoOn an M3 MacBook Pro with 32GB of RAM, I can comfortably run 34B models like phind-codellama:34b-v2-q8_0. Unfortunately, having tried this and a bunch of other models, they are all toys compared to GPT-4.