I think it depends a lot on what you're wanting to do. If you're just playing, you can get surprisingly good results from a 13B model and run it on a fairly ba

I think it depends a lot on what you're wanting to do. If you're just playing, you can get surprisingly good results from a 13B model and run it on a fairly basic M1 Mac. And on an M2 Ultra with a lot of RAM you can run some seriously large models at a good speed. I compromised. Was going to build a PC with a nice graphics card just for LLM work, but in the end decided I didn't want the extra hardware and I could be happy with a powerful laptop that is my daily driver but also capable of running a good size LLM. Ended up with an M2 Max w/96GB of RAM. I can run 70B models (quantized at 4 bits) at a usable pace. Not 35 tokens/sec like this 4090 demo, but 5 tok/sec or so, which is usable for me.