I can run a 30B parameter model on my M1 laptop at a pretty decent token rate. I can run two instances of the same model in parallel without affecting the speed

I can run a 30B parameter model on my M1 laptop at a pretty decent token rate. I can run two instances of the same model in parallel without affecting the speed. A maxed out M2 Mac Studio can easily run 70B models and very likely 137B models too. The key is quantization. So it’s all within reach for under $7k if you’re willing to accept a slightly lower quality (~5 bit quantization).