4 ms·
interesting write-up even outside of the risc-v context, bc most of the optimisations are the same for different platforms one interesting difference is that m
by taminka 3y ago
interesting write-up even outside of the risc-v context, bc most of the optimisations are the same for different platforms
one interesting difference is that memory prefetching/preloading has such a huge difference, whereas on arm/x86 it barely makes any difference (in my experience) that it's not even worth doing
also, doesn't allwinner d1-h have simd registers?
- adgjlsfhk1 3y agoThe chips you're used to have had decades of optimization to their automatic memory prefetching. Apples M1 etc do even better (they can speculate prefetch through a pointer which can make lots of data-structures a ton faster. Also for really high performance on x86, prefetching can often be a ~30% boost.
- taminka 3y agoyeah, that's probably it i'm using an m1 laptop, so my sgemm implementation relies on arm neon vector extension, and i saw literally no performance improvements when i added prefetching, which could either be a result of their speculative optimisations or an issue in my own implementation but i have noticed some ppl saying that adding prefetching on x86 has little performance improvement for sgemm, bc it's a scenario in which it's easier to predict the next memory address that's gonna be used
- ribit 3y agoProbably preaching to the choir here, but if you want high-performance sgemm on M-series, you should consider using the system-provided BLAS. It will invoke the dedicated matmul hardware on Apple CPUs, which has access to significantly higher memory and compute bandwidth than Neon. SIMD-based matmul on M1 only makes sense if your matrices are small and you need the result immediately (e.g. geometric transformations). I hope that Apple will implement SVE/SME one day which will allow us to leverage this hardware directly...
- taminka 3y agowait, are you suggesting apple has some hardware level GEMM instructions that are undocumented or smth? also, what do you mean by system-provided BLAS? their accelerate framework?
- adgjlsfhk1 3y agoyep. they have a neural engine that is separate from the CPU and GPU that does really fast matmuls https://github.com/hollance/neural-engine https://github.com/hollance/neural-engine. it's basically completely undocumented and for programs like blas it has more horsepower than all the cores combined. meteorlake and zen4 (8000) also both have npus which are similar. https://www.pcmag.com/news/the-meteor-lake-npu-meet-intels-dedicated-silicon-for-local-ai-processing https://www.pcmag.com/news/the-meteor-lake-npu-meet-intels-d...
- ribit 3y agoWhat I had in mind is not the neutral engine but the AMX (Apple's matrix/long vector coprocessor).
- ribit 3y agoI am talking about the matrix/vector coprocessor (AMX). You can find some reverse-engineered documentation here: https://github.com/corsix/amx https://github.com/corsix/amx On M3 a singe matrix block can achieve ~ 1TFLOP on DGEMM, I assume it will be closer to 4TFLOPS for SGEMM. The Max variants have two such blocks. Didn't do precise benchmarking myself, but switching Python/R matrix libraries to use Apple's BLAS result in 5-6x perf improvement on matrix heavy code for me.
- moonchild 3y ago> M1 etc do even better (they can speculate prefetch through a pointer Interesting. Do you have a reference or more information on this?
- adgjlsfhk1 3y agoI don't believe there is any official documentation on this, but https://github.com/JuliaLang/julia/pull/49430 https://github.com/JuliaLang/julia/pull/49430 for example added prefetching to the marking phase of a GC which saw speedups on x86, but not on M1. It should be pretty easy to find a MWE that shows the behavior. Something like summing an array of int* should do it.