4 ms·

I wonder if Apple ever followed up with this: https://github.com/apple/ml-ane-transformers https://github.com/apple/ml-ane-transformers They claim their ANE-op

I wonder if Apple ever followed up with this: https://github.com/apple/ml-ane-transformers https://github.com/apple/ml-ane-transformers They claim their ANE-optimized models achieve "up to 10 times faster and 14 times lower peak memory consumption compared to baseline implementations." AFAIK, neither MLX nor llama.cpp support ANE. Though llama.cpp started exploring this idea [0]. What's weird is that MLX is made by Apple and yet, they can't support ANE given its closed-source API! [1] [0]: https://github.com/ggml-org/llama.cpp/issues/10453 https://github.com/ggml-org/llama.cpp/issues/10453 [1]: https://github.com/ml-explore/mlx/issues/18#issuecomment-1846492294 https://github.com/ml-explore/mlx/issues/18#issuecomment-184...

kamranjon 2y ago
Whisper.cpp has a coreml option which gives 3x speed up over cpu only according to the docs: https://github.com/ggml-org/whisper.cpp?tab=readme-ov-file#core-ml-support https://github.com/ggml-org/whisper.cpp?tab=readme-ov-file#c...
- zozbot234 2y ago
  Some outdated information about bare-metal use of the ANE is available from the Whisper.cpp pull req: https://github.com/ggml-org/whisper.cpp/pull/1021 https://github.com/ggml-org/whisper.cpp/pull/1021 Even more outdated information at: https://github.com/eiln/ane/tree/33a61249d773f8f50c02ab0b9fe5b2db58346751/docs https://github.com/eiln/ane/tree/33a61249d773f8f50c02ab0b9fe... In short, the early (M1/M2) versions of ANE are unlikely to be useful for modern LLM inference due to their seemingly exclusive focus on statically scheduled FP16 and INT8 MADDs. More extensive information at https://github.com/tinygrad/tinygrad/tree/master/extra/accel/ane https://github.com/tinygrad/tinygrad/tree/master/extra/accel... (from the Tinygrad folks, note that this is also similarly outdated) seems to basically confirm the above. (The jury is still out for M3/M4 which currently have no Asahi support - thus, no current prospects for driving the ANE bare-metal. Note however that the M3/Pro/Max ANE reported performance numbers are quite close to the M2 version, so there may not be a real improvement there either. M3 Ultra and especially the M4 series may be a different story.)
  - kamranjon 2y ago
    I wouldn't say that they aren't useful for inference (there are pretty clear performance improvements even from the asahi effort you linked) - it's just that you have to convert the model ahead of time to be compatible with the ANE which is explained in the readme docs for whisper.cpp that I linked above. I would say though that this likely excludes them from being useful for training purposes.
    - zozbot234 2y ago
      Note that I was only commenting on modern quantized LLM's that basically avoid formats like FP16 or INT8, preferring lower precision wherever feasible. When in-memory model values must be padded to FP16/INT8 this slashes your effective use of memory bandwidth, which is what determines token generation speed. So the only feasible benefits are really in the prompt pre-processing phase, and even then only in lower power use compared to GPU, not really in higher speed.
      - kamranjon 2y ago
        That's really interesting! I didn't know that about the padding behavior here. I am interested to know which models this would include? I know Gemma 3 raw is bf16 - are you just talking about the quantized versions of these? Or are models being released purely as quantized versions these days? I know Google just released a QAT (Quantization Aware Training) model of Gemma 3 27b - but that base model was already released.
        zozbot234 2y ago
        Models may be released as unquantized (and even then they are gradually shifting towards lower precisions over time), but most people are going to be running them in a quantized version simply because that gives you the best bang for your buck (you can fit more interesting models on the same hardware). Of course this is strictly about local LLM inference, though one may reasonably assume that the big players are also doing something similar.
    - 2y ago
- jorvi 2y ago
  .. who is running LLMs on CPU instead of GPU or TPU/NPU
  - thot_experiment 2y ago
    [flagged]
    - kamranjon 2y ago
      Pretty sure they're using the 80 GPU cores available in that case.
      - echelon 2y ago
        And that still performs worse than entry-level Nvidia gaming cards. Apple isn't serious about AI and needs to figure their AI story out. Every other big tech company is doing something about it.
        voidspark 2y ago
        Not for inferencing. M3 Ultra runs big LLMs twice as fast as RTX 5090. https://creativestrategies.com/mac-studio-m3-ultra-ai-workstation-review/ https://creativestrategies.com/mac-studio-m3-ultra-ai-workst... RTX 5090 only has 32GB RAM. M3 Ultra has up to 512 GB with 819 GB/sec bandwidth. It can run models that will not fit on an RTX card. EDIT: Benchmark may not be properly utilizing the 5090. But the M3 Ultra is way more capable than an entry level RTX card at LLM inferencing.
        Spooky23 2y ago
        My little $599 Mac Mini does inference about 15-20% slower than a 5070 in my kids’ gaming rig. They cost about the same, and I got a free computer. Nvidia makes an incredible product, but apples different market segmentation strategy might make it a real player in the long run.
        balnazzar 2y ago
- echelon 2y ago
  > coreml option which gives 3x speed up over cpu Which is still painfully slow. CoreML is not a real ML platform.
ks2048 2y ago
One update from Apple Research since that one was "Deploying Attention-Based Vision Transformers to Apple Neural Engine" (but not clear the relationship. It doesn't build on ane_transformers, but maybe a sister project for vision?) blog: https://machinelearning.apple.com/research/vision-transformers https://machinelearning.apple.com/research/vision-transforme... github: https://github.com/apple/ml-vision-transformers-ane https://github.com/apple/ml-vision-transformers-ane
1W6MIC49CYX9GAP 2y ago
Based on the graphs "up to 10 times faster" compares before/after flash attention
noahbp 2y ago
Onnxruntime supports CoreML, though if my experience with converting an embedding model to CoreML using Apple's CoreML conversion tool is similar to the ORT maintainers', I can see why it would be unmaintained. It took multiple tries to get the model to convert at all to the mlpackage format, and then a lot of experimenting to get it to run on the ANE instead of the GPU, only to discover that constant reshaping was killing any performance benefit (either you have a fixed multiplication size or don't bother), and even at a fixed size and using the attention mask, its operations were slower than saturating the GPU with large batches. I discovered an issue where using the newer iOS 18 standard would cause the model conversion to break, and put an issue in on their GitHub, including an example repository for easy replication. I got a response quickly, but almost a year later, the bug is still unfixed. Even when George Hotz attempted to hack it to use it without Apple's really bad and unmaintained CoreML library, he gave up because it was impossible without breaking some pretty core OS features (certificate signing IIRC). The ANE/CoreML is just not serious at all about making their hardware usable at all. Even Apple's internal MLX team can't crack that nut.
- imtringued 2y ago
  ONNX is horrible for anything that has variable input shapes and that is why nobody uses it for LLMs. It fundamentally is poorly designed for anything that doesn't take a fixed size image.
  - zozbot234 2y ago
    ANE itself is also limited to fixed computation "shapes" so I'm not sure how much that would matter practically.
ljosifov 2y ago
This more than anything feels emblematic to me, that Apple executives are brain dead when it comes to software. AI seemingly being a step too far. (in the s/w direction.) While they could at some level grok classical s/w, NN-s are that Terra Incognita where Apple executives can not possibly follow in. It's just too strange and mysterious world for them to be able to effectively decide or execute on anything. Worked (20yrs ago) in an industrial R&D lab of a mostly h/w manufacturer for 3 years. It looked to me that the worlds of h/w and s/w, the mindsets, diverged pretty quickly on all important considerations of "what should happen next".
- anon373839 2y ago
  Huh? Apple ships heaps of deep learning-based products/features, and they’ve done so for years. But I’ll agree with you, they’re currently behind in generative AI.
- deleted 2y ago
  [deleted]
smpanaro 2y ago
Not a public follow-up but the iOS 17 speech-to-text model has a clever approach to KV caching that works within the ANE’s constraints (fixed size inputs). I wrote about it here[0] but the gist is you can have a fixed size cache and slide it in chunks with each inference. Not as efficient as a cache that grows by one each time of course. [0]: https://stephenpanaro.com/blog/inside-apples-2023-transformers https://stephenpanaro.com/blog/inside-apples-2023-transforme...
- kamranjon 2y ago
  Hey I just wanted to say that this is an amazing write up and I'm bookmarking your blog cause there isn't a ton of information out there about this stuff as it related to Apple hardware and you do a really great job of explaining many of the concepts that I'm wasn't already familiar with. Thank you!

kamranjon 2y ago

Whisper.cpp has a coreml option which gives 3x speed up over cpu only according to the docs: https://github.com/ggml-org/whisper.cpp?tab=readme-ov-file#core-ml-support https://github.com/ggml-org/whisper.cpp?tab=readme-ov-file#c...

zozbot234 2y ago
Some outdated information about bare-metal use of the ANE is available from the Whisper.cpp pull req: https://github.com/ggml-org/whisper.cpp/pull/1021 https://github.com/ggml-org/whisper.cpp/pull/1021 Even more outdated information at: https://github.com/eiln/ane/tree/33a61249d773f8f50c02ab0b9fe5b2db58346751/docs https://github.com/eiln/ane/tree/33a61249d773f8f50c02ab0b9fe... In short, the early (M1/M2) versions of ANE are unlikely to be useful for modern LLM inference due to their seemingly exclusive focus on statically scheduled FP16 and INT8 MADDs. More extensive information at https://github.com/tinygrad/tinygrad/tree/master/extra/accel/ane https://github.com/tinygrad/tinygrad/tree/master/extra/accel... (from the Tinygrad folks, note that this is also similarly outdated) seems to basically confirm the above. (The jury is still out for M3/M4 which currently have no Asahi support - thus, no current prospects for driving the ANE bare-metal. Note however that the M3/Pro/Max ANE reported performance numbers are quite close to the M2 version, so there may not be a real improvement there either. M3 Ultra and especially the M4 series may be a different story.)
- kamranjon 2y ago
  I wouldn't say that they aren't useful for inference (there are pretty clear performance improvements even from the asahi effort you linked) - it's just that you have to convert the model ahead of time to be compatible with the ANE which is explained in the readme docs for whisper.cpp that I linked above. I would say though that this likely excludes them from being useful for training purposes.
  - zozbot234 2y ago
    Note that I was only commenting on modern quantized LLM's that basically avoid formats like FP16 or INT8, preferring lower precision wherever feasible. When in-memory model values must be padded to FP16/INT8 this slashes your effective use of memory bandwidth, which is what determines token generation speed. So the only feasible benefits are really in the prompt pre-processing phase, and even then only in lower power use compared to GPU, not really in higher speed.
    - kamranjon 2y ago
      That's really interesting! I didn't know that about the padding behavior here. I am interested to know which models this would include? I know Gemma 3 raw is bf16 - are you just talking about the quantized versions of these? Or are models being released purely as quantized versions these days? I know Google just released a QAT (Quantization Aware Training) model of Gemma 3 27b - but that base model was already released.
      - zozbot234 2y ago
        Models may be released as unquantized (and even then they are gradually shifting towards lower precisions over time), but most people are going to be running them in a quantized version simply because that gives you the best bang for your buck (you can fit more interesting models on the same hardware). Of course this is strictly about local LLM inference, though one may reasonably assume that the big players are also doing something similar.
  - 2y ago
jorvi 2y ago
.. who is running LLMs on CPU instead of GPU or TPU/NPU
- thot_experiment 2y ago
  [flagged]
  - kamranjon 2y ago
    Pretty sure they're using the 80 GPU cores available in that case.
    - echelon 2y ago
      And that still performs worse than entry-level Nvidia gaming cards. Apple isn't serious about AI and needs to figure their AI story out. Every other big tech company is doing something about it.
      - voidspark 2y ago
        Not for inferencing. M3 Ultra runs big LLMs twice as fast as RTX 5090. https://creativestrategies.com/mac-studio-m3-ultra-ai-workstation-review/ https://creativestrategies.com/mac-studio-m3-ultra-ai-workst... RTX 5090 only has 32GB RAM. M3 Ultra has up to 512 GB with 819 GB/sec bandwidth. It can run models that will not fit on an RTX card. EDIT: Benchmark may not be properly utilizing the 5090. But the M3 Ultra is way more capable than an entry level RTX card at LLM inferencing.
        Spooky23 2y ago
        My little $599 Mac Mini does inference about 15-20% slower than a 5070 in my kids’ gaming rig. They cost about the same, and I got a free computer. Nvidia makes an incredible product, but apples different market segmentation strategy might make it a real player in the long run.
        balnazzar 2y ago
echelon 2y ago
> coreml option which gives 3x speed up over cpu Which is still painfully slow. CoreML is not a real ML platform.

ks2048 2y ago

One update from Apple Research since that one was "Deploying Attention-Based Vision Transformers to Apple Neural Engine" (but not clear the relationship. It doesn't build on ane_transformers, but maybe a sister project for vision?) blog: https://machinelearning.apple.com/research/vision-transformers https://machinelearning.apple.com/research/vision-transforme... github: https://github.com/apple/ml-vision-transformers-ane https://github.com/apple/ml-vision-transformers-ane

1W6MIC49CYX9GAP 2y ago

Based on the graphs "up to 10 times faster" compares before/after flash attention

noahbp 2y ago

Onnxruntime supports CoreML, though if my experience with converting an embedding model to CoreML using Apple's CoreML conversion tool is similar to the ORT maintainers', I can see why it would be unmaintained. It took multiple tries to get the model to convert at all to the mlpackage format, and then a lot of experimenting to get it to run on the ANE instead of the GPU, only to discover that constant reshaping was killing any performance benefit (either you have a fixed multiplication size or don't bother), and even at a fixed size and using the attention mask, its operations were slower than saturating the GPU with large batches. I discovered an issue where using the newer iOS 18 standard would cause the model conversion to break, and put an issue in on their GitHub, including an example repository for easy replication. I got a response quickly, but almost a year later, the bug is still unfixed. Even when George Hotz attempted to hack it to use it without Apple's really bad and unmaintained CoreML library, he gave up because it was impossible without breaking some pretty core OS features (certificate signing IIRC). The ANE/CoreML is just not serious at all about making their hardware usable at all. Even Apple's internal MLX team can't crack that nut.

imtringued 2y ago
ONNX is horrible for anything that has variable input shapes and that is why nobody uses it for LLMs. It fundamentally is poorly designed for anything that doesn't take a fixed size image.
- zozbot234 2y ago
  ANE itself is also limited to fixed computation "shapes" so I'm not sure how much that would matter practically.

ljosifov 2y ago

This more than anything feels emblematic to me, that Apple executives are brain dead when it comes to software. AI seemingly being a step too far. (in the s/w direction.) While they could at some level grok classical s/w, NN-s are that Terra Incognita where Apple executives can not possibly follow in. It's just too strange and mysterious world for them to be able to effectively decide or execute on anything. Worked (20yrs ago) in an industrial R&D lab of a mostly h/w manufacturer for 3 years. It looked to me that the worlds of h/w and s/w, the mindsets, diverged pretty quickly on all important considerations of "what should happen next".

anon373839 2y ago
Huh? Apple ships heaps of deep learning-based products/features, and they’ve done so for years. But I’ll agree with you, they’re currently behind in generative AI.
deleted 2y ago
[deleted]

smpanaro 2y ago

Not a public follow-up but the iOS 17 speech-to-text model has a clever approach to KV caching that works within the ANE’s constraints (fixed size inputs). I wrote about it here[0] but the gist is you can have a fixed size cache and slide it in chunks with each inference. Not as efficient as a cache that grows by one each time of course. [0]: https://stephenpanaro.com/blog/inside-apples-2023-transformers https://stephenpanaro.com/blog/inside-apples-2023-transforme...

kamranjon 2y ago
Hey I just wanted to say that this is an amazing write up and I'm bookmarking your blog cause there isn't a ton of information out there about this stuff as it related to Apple hardware and you do a really great job of explaining many of the concepts that I'm wasn't already familiar with. Thank you!