The AMX units are really nice, especially because you can use them simply with standard sgemm through the Accelerate framework. However, in most applications wh

The AMX units are really nice, especially because you can use them simply with standard sgemm through the Accelerate framework. However, in most applications where latency is not an issue, you'll probably want to use Metal Performance shaders instead, not only are they much faster most applications, they can also be more energy efficient. For instance, we did benchmarks of spaCy (natural language processing) transformer models across various Apple Silicon SoCs and MPS was 1.9x (M1) to 5.5x faster (M1 Ultra) while providing far more performance per Watt. E.g. using MPS on an M2 MacBook Air used 4W less energy while being 2.7x faster than AMX. Full benchmarks are at the end of this post: https://explosion.ai/blog/metal-performance-shaders https://explosion.ai/blog/metal-performance-shaders