3 ms·

The chips you're used to have had decades of optimization to their automatic memory prefetching. Apples M1 etc do even better (they can speculate prefetch throu

The chips you're used to have had decades of optimization to their automatic memory prefetching. Apples M1 etc do even better (they can speculate prefetch through a pointer which can make lots of data-structures a ton faster. Also for really high performance on x86, prefetching can often be a ~30% boost.

taminka 3y ago
yeah, that's probably it i'm using an m1 laptop, so my sgemm implementation relies on arm neon vector extension, and i saw literally no performance improvements when i added prefetching, which could either be a result of their speculative optimisations or an issue in my own implementation but i have noticed some ppl saying that adding prefetching on x86 has little performance improvement for sgemm, bc it's a scenario in which it's easier to predict the next memory address that's gonna be used
- ribit 3y ago
  Probably preaching to the choir here, but if you want high-performance sgemm on M-series, you should consider using the system-provided BLAS. It will invoke the dedicated matmul hardware on Apple CPUs, which has access to significantly higher memory and compute bandwidth than Neon. SIMD-based matmul on M1 only makes sense if your matrices are small and you need the result immediately (e.g. geometric transformations). I hope that Apple will implement SVE/SME one day which will allow us to leverage this hardware directly...
  - taminka 3y ago
    wait, are you suggesting apple has some hardware level GEMM instructions that are undocumented or smth? also, what do you mean by system-provided BLAS? their accelerate framework?
    - adgjlsfhk1 3y ago
      yep. they have a neural engine that is separate from the CPU and GPU that does really fast matmuls https://github.com/hollance/neural-engine https://github.com/hollance/neural-engine. it's basically completely undocumented and for programs like blas it has more horsepower than all the cores combined. meteorlake and zen4 (8000) also both have npus which are similar. https://www.pcmag.com/news/the-meteor-lake-npu-meet-intels-dedicated-silicon-for-local-ai-processing https://www.pcmag.com/news/the-meteor-lake-npu-meet-intels-d...
      - ribit 3y ago
        What I had in mind is not the neutral engine but the AMX (Apple's matrix/long vector coprocessor).
    - ribit 3y ago
      I am talking about the matrix/vector coprocessor (AMX). You can find some reverse-engineered documentation here: https://github.com/corsix/amx https://github.com/corsix/amx On M3 a singe matrix block can achieve ~ 1TFLOP on DGEMM, I assume it will be closer to 4TFLOPS for SGEMM. The Max variants have two such blocks. Didn't do precise benchmarking myself, but switching Python/R matrix libraries to use Apple's BLAS result in 5-6x perf improvement on matrix heavy code for me.
moonchild 3y ago
> M1 etc do even better (they can speculate prefetch through a pointer Interesting. Do you have a reference or more information on this?
- adgjlsfhk1 3y ago
  I don't believe there is any official documentation on this, but https://github.com/JuliaLang/julia/pull/49430 https://github.com/JuliaLang/julia/pull/49430 for example added prefetching to the marking phase of a GC which saw speedups on x86, but not on M1. It should be pretty easy to find a MWE that shows the behavior. Something like summing an array of int* should do it.

taminka 3y ago

yeah, that's probably it i'm using an m1 laptop, so my sgemm implementation relies on arm neon vector extension, and i saw literally no performance improvements when i added prefetching, which could either be a result of their speculative optimisations or an issue in my own implementation but i have noticed some ppl saying that adding prefetching on x86 has little performance improvement for sgemm, bc it's a scenario in which it's easier to predict the next memory address that's gonna be used

ribit 3y ago
Probably preaching to the choir here, but if you want high-performance sgemm on M-series, you should consider using the system-provided BLAS. It will invoke the dedicated matmul hardware on Apple CPUs, which has access to significantly higher memory and compute bandwidth than Neon. SIMD-based matmul on M1 only makes sense if your matrices are small and you need the result immediately (e.g. geometric transformations). I hope that Apple will implement SVE/SME one day which will allow us to leverage this hardware directly...
- taminka 3y ago
  wait, are you suggesting apple has some hardware level GEMM instructions that are undocumented or smth? also, what do you mean by system-provided BLAS? their accelerate framework?
  - adgjlsfhk1 3y ago
    yep. they have a neural engine that is separate from the CPU and GPU that does really fast matmuls https://github.com/hollance/neural-engine https://github.com/hollance/neural-engine. it's basically completely undocumented and for programs like blas it has more horsepower than all the cores combined. meteorlake and zen4 (8000) also both have npus which are similar. https://www.pcmag.com/news/the-meteor-lake-npu-meet-intels-dedicated-silicon-for-local-ai-processing https://www.pcmag.com/news/the-meteor-lake-npu-meet-intels-d...
    - ribit 3y ago
      What I had in mind is not the neutral engine but the AMX (Apple's matrix/long vector coprocessor).
  - ribit 3y ago
    I am talking about the matrix/vector coprocessor (AMX). You can find some reverse-engineered documentation here: https://github.com/corsix/amx https://github.com/corsix/amx On M3 a singe matrix block can achieve ~ 1TFLOP on DGEMM, I assume it will be closer to 4TFLOPS for SGEMM. The Max variants have two such blocks. Didn't do precise benchmarking myself, but switching Python/R matrix libraries to use Apple's BLAS result in 5-6x perf improvement on matrix heavy code for me.

moonchild 3y ago

> M1 etc do even better (they can speculate prefetch through a pointer Interesting. Do you have a reference or more information on this?

adgjlsfhk1 3y ago
I don't believe there is any official documentation on this, but https://github.com/JuliaLang/julia/pull/49430 https://github.com/JuliaLang/julia/pull/49430 for example added prefetching to the marking phase of a GC which saw speedups on x86, but not on M1. It should be pretty easy to find a MWE that shows the behavior. Something like summing an array of int* should do it.