> are about the same if I use huggingface transformers on the 3090 or llama.cpp on the M1 with the same quantization What is the difference when you use llama.

> are about the same if I use huggingface transformers on the 3090 or llama.cpp on the M1 with the same quantization What is the difference when you use llama.cpp or transformers for both? In my experience, the difference is pretty much far-and-away once you get the model loaded into RAM. My 3070 inferences about as fast as I've seen the M1 Ultra go (and my system is using a $500 chip, not a $5,000 one). For the money, I think you're just not getting the most bang for your buck with Apple Silicon. It's fine if you want a Mac and use AI to justify the bigger one, but... it would be such a sore experience to spend $7,000 to avoid CUDA. It's the wrong walled-garden to pick if you care more about AI. > (4 bit, I don't have enough RAM to run 8 bit on the M1). That does make a big difference in inferencing speed, though. Lower precision models are much faster, if you ran 4bit on both machines it would probably close the gap significantly.