4 ms·

I am curious if anyone knows if the neural cores in apple silicon based machines are at all useful in training? I’ve been using the MLX framework but haven’t se

I am curious if anyone knows if the neural cores in apple silicon based machines are at all useful in training? I’ve been using the MLX framework but haven’t seen them mentioned anywhere so I’m just wondering if they are only useful for inference? I know whisper.cpp takes advantage of them in the inference context. Edit: I changed llama.cpp to whisper.cpp - I didn’t realize that llama.cpp doesn’t have a coreml option like whisper.cpp does.

lucasoshiro 1y ago
Well, the TensorFlow port to metal was written by Apple, and it doens't use ANE. If even they have chosen to use only GPU probably the ANE wouldn't help in training. I also heard that the ANE is way less powerful than Apple Silicon's GPU, but I don't have numbers
486sx33 1y ago
Maybe a quick side shift - What the heck are apples neural cores good for ? Used for ? Use cases ?
- giantrobot 1y ago
  They're used pretty extensively by stuff like the Photos app for various ML tasks. Not all AI is LLMs.
- kamranjon 1y ago
  They are definitely good for local inference as evidenced by the pretty amazing performance increase on Apple silicon when used with whisper.cpp, maybe other frameworks that utilize coreml? I think they’re sorta purpose built for doing matrix math.
- freehorse 1y ago
  In my understanding, they are intended to be utilized by various apps (and Apple intelligence) for performing different machine learning tasks (inference), in a manner that is unobtrusive to the user and the rest of the system. While a GPU would theoretically be more performant, it could potentially lead to excessive energy consumption, temperature rise, fan noise, and other factors that may not be desirable when performing basic tasks like OCR when viewing an image within an application. Currently, it is for example used through the "Vision Framework" eg for OCR tasks (for instance, when previewing an image in macOS, it performs OCR in the background using the ANE). Additionally, they are utilized by certain apple intelligence features that are executed locally (eg when I asked writing tools to rewrite this comment, I saw a spike in ANE usage). They can also be used for diffusion image models (through core ml, diffusers has a nice frontend for that) but my understanding is that they are primarily for "light" ML tasks within an application rather than running larger models (though that's also possible, but they are gonna probably run slower than in gpu).

lucasoshiro 1y ago

Well, the TensorFlow port to metal was written by Apple, and it doens't use ANE. If even they have chosen to use only GPU probably the ANE wouldn't help in training. I also heard that the ANE is way less powerful than Apple Silicon's GPU, but I don't have numbers

486sx33 1y ago

Maybe a quick side shift - What the heck are apples neural cores good for ? Used for ? Use cases ?

giantrobot 1y ago
They're used pretty extensively by stuff like the Photos app for various ML tasks. Not all AI is LLMs.
kamranjon 1y ago
They are definitely good for local inference as evidenced by the pretty amazing performance increase on Apple silicon when used with whisper.cpp, maybe other frameworks that utilize coreml? I think they’re sorta purpose built for doing matrix math.
freehorse 1y ago
In my understanding, they are intended to be utilized by various apps (and Apple intelligence) for performing different machine learning tasks (inference), in a manner that is unobtrusive to the user and the rest of the system. While a GPU would theoretically be more performant, it could potentially lead to excessive energy consumption, temperature rise, fan noise, and other factors that may not be desirable when performing basic tasks like OCR when viewing an image within an application. Currently, it is for example used through the "Vision Framework" eg for OCR tasks (for instance, when previewing an image in macOS, it performs OCR in the background using the ANE). Additionally, they are utilized by certain apple intelligence features that are executed locally (eg when I asked writing tools to rewrite this comment, I saw a spike in ANE usage). They can also be used for diffusion image models (through core ml, diffusers has a nice frontend for that) but my understanding is that they are primarily for "light" ML tasks within an application rather than running larger models (though that's also possible, but they are gonna probably run slower than in gpu).