3 ms·

AVX seems to me to be a band-aid to the instruction length problem inherent in x86 CISC systems vs. RISC chips like M1. Then again, maybe the fact that the mem

AVX seems to me to be a band-aid to the instruction length problem inherent in x86 CISC systems vs. RISC chips like M1. Then again, maybe the fact that the memory and GPU are all integrated into M1 also helps propel it over the top. How well would an AMD / Intel SoC perform if you threw a 5950X / 10900K, 16 - 128GB of LPDDR4X-4133, and a GTX 3090 into a blender and baked up an enormous die with a huge watercooled solution? I dunno. I want to believe its the architecture that's succeeding, not the SoC by virtue of being an SoC, but I don't have the knowledge or ability to judge that.

dragontamer 6y ago
> AVX seems to me to be a band-aid to the instruction length problem inherent in x86 CISC systems vs. RISC chips like M1. AVX is neither CISC nor RISC. Its SIMD. We're talking about 1970s Cray-systems or 1980s Connection Machine (CM-2 or CM-5). Its a completely different paradigm of compute. An entire history of computing that runs parallel to the CISC vs RISC wars. > Then again, maybe the fact that the memory and GPU are all integrated into M1 also helps propel it over the top. M1 has an awesomely wide 128-bit x4 pipeline SIMD execution backend. Apple is making sure its SIMD unit (for NEON ARM instructions) doesn't fall dramatically behind. M1 can perform 4x 128-bit Multiply-and-Accumulate instructions per clock tick. Its no slouch. AVX512 is better than ARM Neon on M1, but even Apple sees the importance of SIMD compute. The ARM world is talking about 512-bit SVE instructions. SIMD compute is obscure and niche: but its a parallel niche computation style with decades of history. Its the proper solution to a huge host of computational problems.
- cbozeman 6y ago
  Wow, fascinating! Thank you for the reply. :)
  - dragontamer 6y ago
    It should be noted that SIMD-on-CPU is not to be the fastest SIMD in the world... but instead the closest SIMD to your data. SIMD today largely exists in the realm of GPUs: the NVidia 3090 is a dedicated SIMD-architecture for example. But as an "external" device on a PCIe 4.0 port, your CPU can only stream 30GBps worth of data to it (the max-bandwidth of PCIe 4.0 x16) The AVX512-units inside of an Intel chip is the same GPU-SIMD architecture, but smaller. More importantly, it shares L1 cache, and therefore has TBps worth of data-sharing. As such, AVX512 becomes useful if you need a fast CPU to calculate some things, and then switch off to SIMD-paradigm for some reason. --------- What's a good example of this "switchoff" ?? Well, Stockfish-nnue (now default in Stockfish 12) is a great example. You might know that SIMD-compute, in particular GPUs, are the one of the best architectures for calculating neural nets. But GPUs are terrible with branchy-code (which happens often in chess: "if(bishop) then (calculate bishop moves)" is rather difficult to do efficiently on a GPU! Its hard to explain why, but its called "branch divergence". Look it up if you feel like it). Anyway, a CPU is best at the "if(bishop)" kind of computation. But GPUs are best at neural nets. Stockfish-nnue beats Leela-Zero, by using AVX512 on a small-and-lightweight neuralnet, but still uses the CPU for the fast if() statements. This is only made possible because AVX512 and the rest of the CPU shares the same L1 / L2 cache, meaning you can get TBps worth of data-sharing, instead of GBs if you went "off-chip" to another computer (like calling a GPU-to calculate a neural net). Furthermore, the NNUE neural net in Stockfish is composed of if-statements: only PART of the neural net updates, which means it can never be efficiently implemented on a GPU. Only something with fast if() statements can implement NNUE. (EDIT: Well... maybe not "never", but it'd be non-obvious to code up a solution. I think its possible, now that I think of it... but its not done in typical GPU code these days) GPUs / classical SIMD compute want all of its data to update uniformly, without if() statements mucking up the logic. CPUs have giant branch-prediction and hyperthreading and other such features to speed up if-statements. ----- A small SIMD for small SIMD-calculations, that combines with fast CPU-style branch-heavy code. A nightmare for GPUs to implement. Its something only possible with CPU + SIMD instruction assist. ------ GPUs are likely still best for large-scale SIMD. Like giant matrix multiplications (or many matrix-multiplications, like in video games). Where if-statements are less common.
    - ncmncm 6y ago
      What you say about performance on NV 3090 probably applies better to much older GPUs. NV has put a great deal of recent work into getting their system to execute regular multi-threaded code with the same sort of speedup as they have had with SIMD workloads. It might be better to think of today's NV GPU as a super-threadripper when the data to be processed is already in GPU memory. The coder's job becomes making sure enough threads are ready to run to use up all the compute units at all times.
      - dragontamer 6y ago
        > NV has put a great deal of recent work into getting their system to execute regular multi-threaded code with the same sort of speedup as they have had with SIMD workloads. I think you're talking about "Independent Thread Scheduling", which has been deployed since Volta. https://docs.nvidia.com/cuda/volta-tuning-guide/index.html#sm-independent-thread-scheduling https://docs.nvidia.com/cuda/volta-tuning-guide/index.html#s... https://images.nvidia.com/content/volta-architecture/pdf/volta-architecture-whitepaper.pdf https://images.nvidia.com/content/volta-architecture/pdf/vol... You're close... but wrong in a few key areas. What NVidia has done is make it POSSIBLE to run multithreaded code without deadlocks. There's still the thread-divergence performance problem. That is to say: you can now do more classical lock/unlock paradigms on NVidia hardware, but the code is serialized and becomes sequential. One 32-wide SIMD unit will execute only 1x thread at a time in that case. (That is: 1/32 utilization. The SM is designed to handle 32 SIMD threads but is only executing one of them at a time) So you still have the thread-divergence performance problem. On older GPUs, such divergence would have been a DEADLOCK instead. So you've gone from 0% performance utilization to 3% utilization, which means your code actually will finish. But it doesn't mean that your code actually has any degree of worthwhile performance. Indeed: CPUs have proven that time-and-time again, per-thread branch-prediction and prefetching is best for that kind of code. If you have a great deal of divergent if/else statements in your code, you'll want to run on CPUs. NVidia GPUs can now handle those situations better (ie: without deadlocking if those if/else statements contain thread-synchronization statements), but they aren't actually high-performance in those use cases.
        6y ago

dragontamer 6y ago

> AVX seems to me to be a band-aid to the instruction length problem inherent in x86 CISC systems vs. RISC chips like M1. AVX is neither CISC nor RISC. Its SIMD. We're talking about 1970s Cray-systems or 1980s Connection Machine (CM-2 or CM-5). Its a completely different paradigm of compute. An entire history of computing that runs parallel to the CISC vs RISC wars. > Then again, maybe the fact that the memory and GPU are all integrated into M1 also helps propel it over the top. M1 has an awesomely wide 128-bit x4 pipeline SIMD execution backend. Apple is making sure its SIMD unit (for NEON ARM instructions) doesn't fall dramatically behind. M1 can perform 4x 128-bit Multiply-and-Accumulate instructions per clock tick. Its no slouch. AVX512 is better than ARM Neon on M1, but even Apple sees the importance of SIMD compute. The ARM world is talking about 512-bit SVE instructions. SIMD compute is obscure and niche: but its a parallel niche computation style with decades of history. Its the proper solution to a huge host of computational problems.

cbozeman 6y ago
Wow, fascinating! Thank you for the reply. :)
- dragontamer 6y ago
  It should be noted that SIMD-on-CPU is not to be the fastest SIMD in the world... but instead the closest SIMD to your data. SIMD today largely exists in the realm of GPUs: the NVidia 3090 is a dedicated SIMD-architecture for example. But as an "external" device on a PCIe 4.0 port, your CPU can only stream 30GBps worth of data to it (the max-bandwidth of PCIe 4.0 x16) The AVX512-units inside of an Intel chip is the same GPU-SIMD architecture, but smaller. More importantly, it shares L1 cache, and therefore has TBps worth of data-sharing. As such, AVX512 becomes useful if you need a fast CPU to calculate some things, and then switch off to SIMD-paradigm for some reason. --------- What's a good example of this "switchoff" ?? Well, Stockfish-nnue (now default in Stockfish 12) is a great example. You might know that SIMD-compute, in particular GPUs, are the one of the best architectures for calculating neural nets. But GPUs are terrible with branchy-code (which happens often in chess: "if(bishop) then (calculate bishop moves)" is rather difficult to do efficiently on a GPU! Its hard to explain why, but its called "branch divergence". Look it up if you feel like it). Anyway, a CPU is best at the "if(bishop)" kind of computation. But GPUs are best at neural nets. Stockfish-nnue beats Leela-Zero, by using AVX512 on a small-and-lightweight neuralnet, but still uses the CPU for the fast if() statements. This is only made possible because AVX512 and the rest of the CPU shares the same L1 / L2 cache, meaning you can get TBps worth of data-sharing, instead of GBs if you went "off-chip" to another computer (like calling a GPU-to calculate a neural net). Furthermore, the NNUE neural net in Stockfish is composed of if-statements: only PART of the neural net updates, which means it can never be efficiently implemented on a GPU. Only something with fast if() statements can implement NNUE. (EDIT: Well... maybe not "never", but it'd be non-obvious to code up a solution. I think its possible, now that I think of it... but its not done in typical GPU code these days) GPUs / classical SIMD compute want all of its data to update uniformly, without if() statements mucking up the logic. CPUs have giant branch-prediction and hyperthreading and other such features to speed up if-statements. ----- A small SIMD for small SIMD-calculations, that combines with fast CPU-style branch-heavy code. A nightmare for GPUs to implement. Its something only possible with CPU + SIMD instruction assist. ------ GPUs are likely still best for large-scale SIMD. Like giant matrix multiplications (or many matrix-multiplications, like in video games). Where if-statements are less common.
  - ncmncm 6y ago
    What you say about performance on NV 3090 probably applies better to much older GPUs. NV has put a great deal of recent work into getting their system to execute regular multi-threaded code with the same sort of speedup as they have had with SIMD workloads. It might be better to think of today's NV GPU as a super-threadripper when the data to be processed is already in GPU memory. The coder's job becomes making sure enough threads are ready to run to use up all the compute units at all times.
    - dragontamer 6y ago
      > NV has put a great deal of recent work into getting their system to execute regular multi-threaded code with the same sort of speedup as they have had with SIMD workloads. I think you're talking about "Independent Thread Scheduling", which has been deployed since Volta. https://docs.nvidia.com/cuda/volta-tuning-guide/index.html#sm-independent-thread-scheduling https://docs.nvidia.com/cuda/volta-tuning-guide/index.html#s... https://images.nvidia.com/content/volta-architecture/pdf/volta-architecture-whitepaper.pdf https://images.nvidia.com/content/volta-architecture/pdf/vol... You're close... but wrong in a few key areas. What NVidia has done is make it POSSIBLE to run multithreaded code without deadlocks. There's still the thread-divergence performance problem. That is to say: you can now do more classical lock/unlock paradigms on NVidia hardware, but the code is serialized and becomes sequential. One 32-wide SIMD unit will execute only 1x thread at a time in that case. (That is: 1/32 utilization. The SM is designed to handle 32 SIMD threads but is only executing one of them at a time) So you still have the thread-divergence performance problem. On older GPUs, such divergence would have been a DEADLOCK instead. So you've gone from 0% performance utilization to 3% utilization, which means your code actually will finish. But it doesn't mean that your code actually has any degree of worthwhile performance. Indeed: CPUs have proven that time-and-time again, per-thread branch-prediction and prefetching is best for that kind of code. If you have a great deal of divergent if/else statements in your code, you'll want to run on CPUs. NVidia GPUs can now handle those situations better (ie: without deadlocking if those if/else statements contain thread-synchronization statements), but they aren't actually high-performance in those use cases.
      - 6y ago