3 ms·
Confusingly there are 2 mechanisms to do matrix operations on the new apple hardware - AMX (https://github.com/corsix/amx https://github.com/corsix/amx) - and t
by kamranjon 4y ago
Confusingly there are 2 mechanisms to do matrix operations on the new apple hardware - AMX (https://github.com/corsix/amx https://github.com/corsix/amx) - and the ANE (apple neural engine) - which is enabled by CoreML. This code does not run on the neural engine but the author has a branch for his whisper.cpp project which uses it here: https://github.com/ggerganov/whisper.cpp/pull/566 https://github.com/ggerganov/whisper.cpp/pull/566 - so it may not be long before we see it applied here as well. All of this is to say that it actually could get significantly faster if some of this work was able to be handed to the ANE with CoreML.
- lynguist 4y agoYou’re right! I wrote it too fast without thinking!
- raphlinus 4y agoThree. You can also do it in Metal, which as of recently has cooperative matrix multiplication in the form of the simd_matrix type (this is similar functionality as "tensor cores" in the Nvidia world). I have no idea what the software support is, but I have seen analysis suggesting that the raw tensor multiplication throughput is larger than ANE for the high-end GPUs.