4 ms·
I don't think it is? But I would love to be wrong. Cheap mobile Nvidia cards outperform regular M2 chips on small learning tasks. M3 Max might be a contender,
by nextos 3y ago
I don't think it is? But I would love to be wrong. Cheap mobile Nvidia cards outperform regular M2 chips on small learning tasks.
M3 Max might be a contender, but Metal is far from CUDA yet? A laptop M3 Max looks like it can compete with a Nvidia 3070.
It's an unfair comparison as Nvidia is a huge desktop heat/energy hog and Apple M chips are really efficient. Let's see how desktop M3 chips fare.
- blep-arsh 3y agoConsumer nVidia cards are constrained by their RAM which is inconvenient considering the sizes of popular LLMs. An M2 Ultra basically gives you 100+ GB VRAM which is nice.
- hnfong 3y agoAlthough pytorch is in python, some ML features in SoTA libraries are coded against CUDA and complains if you don't have a CUDA device. A lot of things work out of the box with pytorch on mps, but not everything, and it can be frustrating to figure what works and what doesn't while you have a bunch of other moving parts shifting under your feet due to how fast the whole thing is moving. That said, llama.cpp (aka "ggml") on M1/M2 Mac is very robust. If you just want to run inference (as opposed to training) of models and you're comfortable with a Mac, there's really no reason to go out and by an nVidia card. FWIW, some numbers I found on reddit: https://www.reddit.com/r/LocalLLaMA/comments/16o4ka8/running_ggufs_on_an_m1_ultra_is_an_interesting/ https://www.reddit.com/r/LocalLLaMA/comments/16o4ka8/running...