You need lots of memory. 16G will easily run 7B quantized models. Not the cheapest by far, but I recently bought a 32G internal memory M2 Pro Mac mini. I can r

You need lots of memory. 16G will easily run 7B quantized models. Not the cheapest by far, but I recently bought a 32G internal memory M2 Pro Mac mini. I can run about four 7B models concurrently. I was able to run a 30B quantized model without page faults, but I killed most user land processes. Also not what you are asking for, but I pay Google $10/month for Colab Pro and I can usually get an A100 whenever I request one. Between Colab and my 32G M2 box, I am very satisfied. Before I found good quantized models to run, I would rent a VPS by the hour from Lambda Labs, and that was a great experience, but I don’t need to do that now. EDIT: on the M2 Pro, I get 25 to 30 tokens per second. EDIT #2: I wrote a short blog yesterday on the best resources I have found so far for running on my Mac https://marklwatson.substack.com/p/running-open-llm-models-on-apple https://marklwatson.substack.com/p/running-open-llm-models-o...