Although pytorch is in python, some ML features in SoTA libraries are coded against CUDA and complains if you don't have a CUDA device. A lot of things work ou

Although pytorch is in python, some ML features in SoTA libraries are coded against CUDA and complains if you don't have a CUDA device. A lot of things work out of the box with pytorch on mps, but not everything, and it can be frustrating to figure what works and what doesn't while you have a bunch of other moving parts shifting under your feet due to how fast the whole thing is moving. That said, llama.cpp (aka "ggml") on M1/M2 Mac is very robust. If you just want to run inference (as opposed to training) of models and you're comfortable with a Mac, there's really no reason to go out and by an nVidia card. FWIW, some numbers I found on reddit: https://www.reddit.com/r/LocalLLaMA/comments/16o4ka8/running_ggufs_on_an_m1_ultra_is_an_interesting/ https://www.reddit.com/r/LocalLLaMA/comments/16o4ka8/running...