4 ms·
Are the Apple neural engines even a practical target of LLMs? Maybe not strictly impossible, but ANE was designed with an earlier, pre-LLM style of ML. Running
by jonathaneunice 6mo ago
Are the Apple neural engines even a practical target of LLMs?
Maybe not strictly impossible, but ANE was designed with an earlier, pre-LLM style of ML. Running LLMs on ANE (e.g. via Core ML) possible in theory, but the substantial model conversion and custom hardware tuning required makes for a high hurdle IRL. The LLM ecosystem standardized around CPU/GPU execution, and to date at least seems unwilling to devote resources to ANE. Even Apple's MLX framework has no ANE support. There are models ANE runs well, but LLMs do not seem to be among them.
- Yukonv 6mo agoIt is possible but requires a very specific model design to utilize. As this reverse engineering effort has shown [0] "The ANE is not a GPU. It’s not a CPU. It’s a graph execution engine." To build one requires using a specific pipeline specifically for CoreML [1]. [0] https://maderix.substack.com/p/inside-the-m4-apple-neural-engine https://maderix.substack.com/p/inside-the-m4-apple-neural-en... [1] https://developer.apple.com/documentation/coreml https://developer.apple.com/documentation/coreml
- jonathaneunice 6mo agoThat's the best "what is ANE, really?" investigation / explanation I've seen. Directly lays out why LLMs aren't an ideal fit, its "convolution engine" architecture, the need for feeding ANE deep operation sequence plans / graphs (and the right data sizes) to get full performance, the fanciful nature of Apple's performance claims (~2x actually achievable, natch), and the (superior!) hard power gating... just _oodles_ of insight.
- krackers 6mo agoMore info on specific design choices needed to run models here [1]. I mean it is possible given that apple themselves did it in [2], but it's also not as general purpose or flexible as a GPU. [1] https://news.ycombinator.com/item?id=43881692 https://news.ycombinator.com/item?id=43881692 [2] https://machinelearning.apple.com/research/neural-engine-transformers https://machinelearning.apple.com/research/neural-engine-tra...
- dust42 6mo agoThere is a project on github named ANEMLL. Was discussed here a month ago, running LLMs on iPhone - https://news.ycombinator.com/item?id=47490070 https://news.ycombinator.com/item?id=47490070