4 ms·

> Additionally, support for up to 128GB of memory unlocks workflows previously not possible on a laptop, such as AI developers working with even larger transfor

> Additionally, support for up to 128GB of memory unlocks workflows previously not possible on a laptop, such as AI developers working with even larger transformer models with billions of parameters. The presentation explicitly highlighted this very randomly, without showing how AI development actually works. I know Apple has put more effort into Apple Silicon support for PyTorch but is it there yet?

wmf 3y ago
Are the cool kids using GGML?
- minimaxir 3y ago
  Also that.
  - hnfong 3y ago
    FWIW, the main developer of GGML apparently runs a M2 Ultra himself. Apple Silicon is a first class citizen on GGML and runs fast. See eg. https://www.reddit.com/r/LocalLLaMA/comments/16o4ka8/running_ggufs_on_an_m1_ultra_is_an_interesting/ https://www.reddit.com/r/LocalLLaMA/comments/16o4ka8/running...
- EVa5I7bHFq9mnYK 3y ago
  no, GGUF
mardifoufs 3y ago
I'm pretty sure pytorch supports it OOTB. In my experience it works even better than ROCM
nextos 3y ago
I don't think it is? But I would love to be wrong. Cheap mobile Nvidia cards outperform regular M2 chips on small learning tasks. M3 Max might be a contender, but Metal is far from CUDA yet? A laptop M3 Max looks like it can compete with a Nvidia 3070. It's an unfair comparison as Nvidia is a huge desktop heat/energy hog and Apple M chips are really efficient. Let's see how desktop M3 chips fare.
- blep-arsh 3y ago
  Consumer nVidia cards are constrained by their RAM which is inconvenient considering the sizes of popular LLMs. An M2 Ultra basically gives you 100+ GB VRAM which is nice.
- hnfong 3y ago
  Although pytorch is in python, some ML features in SoTA libraries are coded against CUDA and complains if you don't have a CUDA device. A lot of things work out of the box with pytorch on mps, but not everything, and it can be frustrating to figure what works and what doesn't while you have a bunch of other moving parts shifting under your feet due to how fast the whole thing is moving. That said, llama.cpp (aka "ggml") on M1/M2 Mac is very robust. If you just want to run inference (as opposed to training) of models and you're comfortable with a Mac, there's really no reason to go out and by an nVidia card. FWIW, some numbers I found on reddit: https://www.reddit.com/r/LocalLLaMA/comments/16o4ka8/running_ggufs_on_an_m1_ultra_is_an_interesting/ https://www.reddit.com/r/LocalLLaMA/comments/16o4ka8/running...

wmf 3y ago

Are the cool kids using GGML?

minimaxir 3y ago
Also that.
- hnfong 3y ago
  FWIW, the main developer of GGML apparently runs a M2 Ultra himself. Apple Silicon is a first class citizen on GGML and runs fast. See eg. https://www.reddit.com/r/LocalLLaMA/comments/16o4ka8/running_ggufs_on_an_m1_ultra_is_an_interesting/ https://www.reddit.com/r/LocalLLaMA/comments/16o4ka8/running...
EVa5I7bHFq9mnYK 3y ago
no, GGUF

mardifoufs 3y ago

I'm pretty sure pytorch supports it OOTB. In my experience it works even better than ROCM

nextos 3y ago

I don't think it is? But I would love to be wrong. Cheap mobile Nvidia cards outperform regular M2 chips on small learning tasks. M3 Max might be a contender, but Metal is far from CUDA yet? A laptop M3 Max looks like it can compete with a Nvidia 3070. It's an unfair comparison as Nvidia is a huge desktop heat/energy hog and Apple M chips are really efficient. Let's see how desktop M3 chips fare.

blep-arsh 3y ago
Consumer nVidia cards are constrained by their RAM which is inconvenient considering the sizes of popular LLMs. An M2 Ultra basically gives you 100+ GB VRAM which is nice.
hnfong 3y ago
Although pytorch is in python, some ML features in SoTA libraries are coded against CUDA and complains if you don't have a CUDA device. A lot of things work out of the box with pytorch on mps, but not everything, and it can be frustrating to figure what works and what doesn't while you have a bunch of other moving parts shifting under your feet due to how fast the whole thing is moving. That said, llama.cpp (aka "ggml") on M1/M2 Mac is very robust. If you just want to run inference (as opposed to training) of models and you're comfortable with a Mac, there's really no reason to go out and by an nVidia card. FWIW, some numbers I found on reddit: https://www.reddit.com/r/LocalLLaMA/comments/16o4ka8/running_ggufs_on_an_m1_ultra_is_an_interesting/ https://www.reddit.com/r/LocalLLaMA/comments/16o4ka8/running...