This article about the undocumented Apple AMX instructions is the missing piece of the puzzle for me. Various HN threads discussed SIMD [1] and ML/DL [2] on the

This article about the undocumented Apple AMX instructions is the missing piece of the puzzle for me. Various HN threads discussed SIMD [1] and ML/DL [2] on the Apple M1. Daniel Lemire's benchmark compared x64 SIMD (SSE, AVX) vs. ARM SIMD (Neon, SVE2 [next gen]). Apple's AMX is a coprocessor that does Matrix math in addition to Vector SIMD math. Maynard Handley clearly understood this in his comment on Lemire's blog [3]: > You (and other commenters) are aware of NEON, but apparently not of AMX. AMX may not work for the sorts of JSON parsing weirdness for which you use AVX256 (that’ll have to wait for SVE/2, probably next year) but it does solve the problem of “I want to execute dense linear algebra fast”. > You might want to run some comparisons of that for your M1 vs Intel MacBooks… The API’s to look at are in Accelerate() The Core ML [4] docs have a nice diagram that demonstrates the relationship with the lower level Accelerate [5] and Accelerate BNNS [6]: > Core ML is the foundation for domain-specific frameworks and functionality. Core ML supports Vision for analyzing images, Natural Language for processing text, Speech for converting audio to text, and Sound Analysis for identifying sounds in audio. Core ML itself builds on top of low-level primitives like Accelerate and BNNS, as well as Metal Performance Shaders. Apple's TensorFlow implementation for the M1 is analogous to Core ML. It probably invokes the appropriate Accelerate BNNS and/or Metal library calls. Maynard Handley was exactly right, SIMD/ML/DL code that targets Apple silicon should link against Apple's Accelerate or Core ML libraries. These libraries in turn invoke the appropriate AMX, Neural Engine, and GPU instructions for each target platform. [1] https://news.ycombinator.com/item?id=25408853 https://news.ycombinator.com/item?id=25408853 [2] https://news.ycombinator.com/item?id=25773109 https://news.ycombinator.com/item?id=25773109 [3] https://lemire.me/blog/2020/12/11/arm-macbook-vs-intel-macbook/#comment-561810 https://lemire.me/blog/2020/12/11/arm-macbook-vs-intel-macbo... [4] https://developer.apple.com/documentation/coreml https://developer.apple.com/documentation/coreml [5] https://developer.apple.com/documentation/accelerate https://developer.apple.com/documentation/accelerate [6] https://developer.apple.com/documentation/accelerate/bnns https://developer.apple.com/documentation/accelerate/bnns