3 ms·

Apple Silicon: A little help from friends and co-processors

53 points by ingve 3y ago

uticus 3y ago
> The Apple Neural Engine (ANE) is a separate unit in M-series chips, and isn’t directly accessible to third parties. As a result, knowledge of it is limited...Access to ANE computation is only available through CoreML and related features in macOS. Even there, developers are given limited options as to what does get executed in the ANE, and in practice it appears little used even when running Machine Learning tasks. I had no idea ANE wasn't very accessible. Especially when the marketing materials [0] seem to tout is is available for use by devs, or when people are asking 'can I run LLM x on my Mac' [1] [0] "An enhanced 16-core Neural Engine accelerates popular machine learning models. Image processing tools like DeNoise AI and Gigapixel AI from Topaz get even faster." https://www.apple.com/macbook-pro/ https://www.apple.com/macbook-pro/ [1] https://news.ycombinator.com/item?id=38267869 https://news.ycombinator.com/item?id=38267869
- saagarjha 3y ago
  They are plenty accessible if you use the APIs Apple provides to talk to them, such as the one mentioned in the article: CoreML.
  - uticus 3y ago
    Well, that's my point though, it is already pointed out in the section I quoted: > Access to ANE computation is only available through CoreML and related features in macOS. Even there, developers are given limited options... The article makes it sound like it's not 'plenty accessible.'
AhtiK 3y ago
Regarding the use of the Apple Neural Engine (ANE) for ML tasks: yes, it is closed source for direct access. However, reports indicate that it supports only forward propagation (without backward propagation) and offers a relatively limited set of supported precisions [1]. Should the API ever be opened, the ANE's primary use case would likely shift towards inference rather than training. George Hotz made pretty good progress with the ANE access ca 3 years ago at one of his videos: https://www.youtube.com/watch?v=JAyw7OAcXDE https://www.youtube.com/watch?v=JAyw7OAcXDE [1] Comments from MLX (Apple-contributed Metal-only array compute library): https://github.com/ml-explore/mlx/issues/18 https://github.com/ml-explore/mlx/issues/18
neonsunset 3y ago
NEON is strictly not a vector processor for all intents and purposes in Apple cores. Execution units (ports) capable of dispatching SIMD/FP instructions are part of the core itself alongside regular ALU, and Load/Store units. https://dougallj.github.io/applecpu/firestorm.html https://dougallj.github.io/applecpu/firestorm.html

53 points by ingve 3y ago

> The Apple Neural Engine (ANE) is a separate unit in M-series chips, and isn’t directly accessible to third parties. As a result, knowledge of it is limited...Access to ANE computation is only available through CoreML and related features in macOS. Even there, developers are given limited options as to what does get executed in the ANE, and in practice it appears little used even when running Machine Learning tasks. I had no idea ANE wasn't very accessible. Especially when the marketing materials [0] seem to tout is is available for use by devs, or when people are asking 'can I run LLM x on my Mac' [1] [0] "An enhanced 16-core Neural Engine accelerates popular machine learning models. Image processing tools like DeNoise AI and Gigapixel AI from Topaz get even faster." https://www.apple.com/macbook-pro/ https://www.apple.com/macbook-pro/ [1] https://news.ycombinator.com/item?id=38267869 https://news.ycombinator.com/item?id=38267869

saagarjha 3y ago
They are plenty accessible if you use the APIs Apple provides to talk to them, such as the one mentioned in the article: CoreML.
- uticus 3y ago
  Well, that's my point though, it is already pointed out in the section I quoted: > Access to ANE computation is only available through CoreML and related features in macOS. Even there, developers are given limited options... The article makes it sound like it's not 'plenty accessible.'

AhtiK 3y ago

Regarding the use of the Apple Neural Engine (ANE) for ML tasks: yes, it is closed source for direct access. However, reports indicate that it supports only forward propagation (without backward propagation) and offers a relatively limited set of supported precisions [1]. Should the API ever be opened, the ANE's primary use case would likely shift towards inference rather than training. George Hotz made pretty good progress with the ANE access ca 3 years ago at one of his videos: https://www.youtube.com/watch?v=JAyw7OAcXDE https://www.youtube.com/watch?v=JAyw7OAcXDE [1] Comments from MLX (Apple-contributed Metal-only array compute library): https://github.com/ml-explore/mlx/issues/18 https://github.com/ml-explore/mlx/issues/18

neonsunset 3y ago

NEON is strictly not a vector processor for all intents and purposes in Apple cores. Execution units (ports) capable of dispatching SIMD/FP instructions are part of the core itself alongside regular ALU, and Load/Store units. https://dougallj.github.io/applecpu/firestorm.html https://dougallj.github.io/applecpu/firestorm.html