3 ms·
Can you recommend hardware needed to run these?
by tomrod 2y ago
Can you recommend hardware needed to run these?
- simonw 2y agoI'm using an M2 64GB MacBook Pro. For the Llama 8B one I would expect 16GB to be enough. I don't have any experience running models on Windows or Linux, where your GPU VRAM becomes the most important factor.
- rane 2y agoWhy isn't GPU VRAM a factor on a Silicon mac?
- Nekit1234007 2y agoBecause there's no VRAM. The “regular” RAM on Apple Silicon devices is shared with the GPU virtually in its entirety.
- dragonwriter 2y agoOn Windows or Linux you can run from RAM or split layers between RAM and VRAM; running fully on GPU is faster than either of those, but the limit on what you can run at all isn’t VRAM.
- akhdanfadh 2y agoSo is it possible to load the ollama deepseek-r1 70b (43gb) model on my 24gb vram + 32gb ram machine? Does this depend on how I load the model, i.e., with ollama instead of other alternatives? Afaik, ollama is basically llama.cpp wrapper. I have tried to deploy one myself with openwebui+ollama but only for small LLM. Not sure about the bigger one, worried if that will crash my machine someway. Are there any docs? I am curious about this and how that works if any.
- yencabulator 2y agoollama runs deepseek-r1:7b on AMD 8945HS, CPU-only, at ~12 tokens/s. You can get started pretty easily in the ~7B model range, for learning purposes.