3 ms·
What hardware and software would be recommended for a "good quality" local inference with a LLM on: - a Windows or Linux machine - a Mac?
by lynguist 3y ago
What hardware and software would be recommended for a "good quality" local inference with a LLM on:
- a Windows or Linux machine
- a Mac?
- rootusrootus 3y agoI think it depends a lot on what you're wanting to do. If you're just playing, you can get surprisingly good results from a 13B model and run it on a fairly basic M1 Mac. And on an M2 Ultra with a lot of RAM you can run some seriously large models at a good speed. I compromised. Was going to build a PC with a nice graphics card just for LLM work, but in the end decided I didn't want the extra hardware and I could be happy with a powerful laptop that is my daily driver but also capable of running a good size LLM. Ended up with an M2 Max w/96GB of RAM. I can run 70B models (quantized at 4 bits) at a usable pace. Not 35 tokens/sec like this 4090 demo, but 5 tok/sec or so, which is usable for me.
- smoldesu 3y agoOn Windows and Linux, you can run 7B models on pretty much any CPU with AVX support. Most x86 CPUs that support SSE can get usable acceleration on Llama.cpp.