6 ms·

There's not much use for AVX-512 in the kernel. But, from the people who are using it I've heard good things. It's an enormous collection of new instructions.

There's not much use for AVX-512 in the kernel. But, from the people who are using it I've heard good things. It's an enormous collection of new instructions. But, a lot of them came from the Larrabee team --which was probably the greatest case of high-perf software engineers having direct influence over the direction of a CPU. Otherwise, the history SSE and AVX has largely been software engineers requesting features and hardware engineers replying "Do not understand. How does that feature improve our SPECfp score? Rejected."

gameswithgo 6y ago
On the one hand AVX-512 is incredible, the performance gains from AVX-2 to AVX-512 are often more than the 2x you would hope for. On the other hand, when you can use AVX-512 you can often use a GPU instead, to greater benefit. So linus may be right, overall.
- smabie 6y ago
  Are you sure l? Gains between 10% and 50% are pretty common, but I've never heard of a 100% or greater improvement.
  - otherjason 6y ago
    Yes, these applications do exist. The speedups mainly come from the more flexible instruction set that AVX-512 provides (so in many cases you can replace several instructions with just one), plus the increased register set (32 architectural SIMD registers versus 16 on previous generations). If your algorithm isn't sensitive to either one of these, then yes, the speedups are often smaller, particularly if the limiting factor in performance is memory bandwidth and/or latency (AVX-512 doesn't help at all there), or if the speedups aren't enough to overcome clock throttling (which IMO is a bit overblown; most people cite the Cloudflare blog article from a few years ago that complained about throttling on Xeon Silver processors, but throttling is much less of an issue on higher-TDP Gold/Platinum CPUs). However, I have seen specific applications that are >100% faster with AVX-512.
  - TinkersW 6y ago
    If you write SoA code, know what are you doing, and aren't memory bound, 100% speedup isn't unreasonable. Current AVX512 implementations might not always get there, as I believe some instructions are dual issuing at 256bit under the hood.
- corysama 6y ago
  In my observations, GPUs are the way they are largely because Physics demands sacrifices if you want that kind of performance. Those same demands apply to CPUs, but Intel has been fighting against them because software engineers hate to change anything about how they code. Well... the fight was lost years ago and Intel is conceding to Physics by integrating architecture that was originally designed for a CPU-GPU hybrid. With AVX-512 and CUDA, Intel and Nvidia are trying to find the best developer experience they can to deal with the inevitable future of computing that is somewhere between those two experiences currently.
  - Const-me 6y ago
    > inevitable future of computing that is somewhere between those two experiences currently. Not sure that ever happens. The two are very different in their performance characteristics. CPUs have to stay focused on single-threaded performance. Single threaded performance = minimizing latency, that’s why in CPUs so many transistors and electricity is spent on cache hierarchy and coherency protocol, instructions reordering, ILP, microops fusion, prefetcher, branch prediction incl. indirect one, and so on. GPUs don’t care about single-threaded performance, all they care is throughput on massively parallel tasks. They ignore memory latency, switching threads until data arrives, makes caches much simpler. They don’t need high clock frequencies (this alone improves their flops/watt by a large factor), don’t do branch prediction, memory model is less strict, and so on. That’s how they can spend that many transistors on FPUs and ALUs. P.S. Intel tried many-slow-cores with Larrabee / knights landing / etc., all they got was a very expensive lesson, people didn’t want that stuff. I’ve heard a rumor initially Sony wanted to use another Cell CPU for the GPU in PS3, again, didn’t worked well enough and they negotiated with NVidia instead.
    - corysama 6y ago
      All true. Problem is that single-threaded latency minimization has been tapped out for a long time now. Physics says No. Memory is high-latency because of the speed of light. Larger caches have diminishing returns. Clocks can't go faster without melting the chips. There is no more prediction and pipelining to be dug out of a serial stream of instructions. We can still add more transistors. But, there is nowhere for them to contribute but in going wider. High latency, high throughput. High concurrency and careful management of the memory hierarchy are the only ways to pull that off. When you accept that fate, you start out with the Cell processor and you pass through Nvidia's Ampere chip --which has somewhat more thread independence and better automatic caching than past GPUs, but also finally gained manual async DMAs to local SRAM like the Cell had. Where we'll end up, I don't know.
- rumanator 6y ago
  > On the one hand AVX-512 is incredible, the performance gains from AVX-2 to AVX-512 are often more than the 2x you would hope for. Your assertion sounds like a gross exaggeration. Some benchmarks show only a few percent improvements (i.e.14% improvements on synthetic benchmarks), and results range between no improvements and almost 2x in some synthetic benchmarks. https://www.phoronix.com/scan.php?page=article&item=gcc9-skylake-avx512&num=1 https://www.phoronix.com/scan.php?page=article&item=gcc9-sky...
  - gameswithgo 6y ago
    Those look like benchmarks comparing just autovectorized compilation of some tools and measuring overall runtime. I am referring to cases of hand tuned use SIMD. The bSIMD project used to have a suite of benchmarks up and some where very very impressive for AVX-512. The page appears to be gone or moved now though. To be clear I am not suggest "avx-512 will cause programs to run twice as fast or more", just that specific functions that used to use AVX-2, may be more than twice as fast if you move them to AVX-512.
    - vardump 6y ago
      Some SIMD optimization cases can occasionally exceed scalar versions by more than lane width multiply. Often because of branch elision, but also less dependency chains stalls due to the larger register file. There are also some special instructions that can be very useful in SIMD case (packing, unpacking, permutation, etc.).
    - rumanator 6y ago
      > Those look like benchmarks comparing just autovectorized compilation of some tools and measuring overall runtime. I am referring to cases of hand tuned use SIMD. I presented data that showed that AVX-512 in reality only brings margins performance improvements. If you have any data that shows otherwise then feel free to present it. Still, hand-tuned assembly is hardly a relevant example, let alone selling point, as it creates hypothetical expectations that are never ever matched in reality.
- mvanveen 6y ago
  In many cases within regulated environments getting access to GPU in e.g. a customer's on-prem solution is prohibitive. In enterprise ML contexts on tabular data problems we found there are a lot of cases where even training can be greatly sped up by leveraging AVX instruction support in e.g. tensorflow builds. The gains from AVX instructions could boost training time by ~20% on the GAN use cases I profiled.
  - btilly 6y ago
    So you're saying that we need to solve organizational stupidity by complicating all chips for everyone so that bad organizations can get a performance boost on specialized tasks? What percentage of consumers with these chips installed do you think are getting a performance win? Do you think it might be as high as 1%? Do you think that the same resources devoted elsewhere might be worth more than 1% to that 99%? Whether that is in reduced cost, reduced bugs, or a boost for more widely used operations. Yes, if you target a specific use case for a specific set of people, you can give them a nice win. But you shouldn't lose sight of the fact that CPUs cover a lot of use cases for a lot of people. And simplifying then focusing on the core mission is better for everyone in the end.
    - anoncareer0212 6y ago
      Linus somewhat waved away the idea of tradeoffs, which is fine, he was speaking in generalities. Turning factual reports of where the tradeoff was helpful into a strawman insulting the reporter, and the users who benefit, is neither charitable nor illuminating.
- CoolGuySteve 6y ago
  I agree about GPUs. Intel used to have a low-end CPU with a 128MB L4 cache that was shared with the iGPU. It turns out having that large cache improved a lot of regular software too. I'd rather see die space used for that than AVX512. The only AVX512 instructions that actually seem essential imo are the VNNI extensions. There are a lot of use cases for a neural network that is trained on the GPU but where inference is done on a bunch of CPUs because it's cheaper than buying/renting inference hardware.
  - cogman10 6y ago
    I think you are thinking AVX takes up more diespace than it does. Only a tiny fraction of all CPU diespace is dedicated to the actual CPU. The rest goes primarily to cache and partially to the various controllers on the CPU (memory controller, PCI-Express controller, etc). There's a reason AMD is and Intel are able to cram full GPUs onto the same die as the CPU.
    - Dylan16807 6y ago
      https://imgur.com/r/intel/4SG36 https://imgur.com/r/intel/4SG36 Yeah, here's a good image of how much space it costs to add a 'second' AVX-512 unit to certain SKUs. Cutting AVX-512 support entirely would save a bit more space, and the total would be enough to make a difference, but it's not an extreme amount. https://software.intel.com/sites/default/files/managed/5f/99/Skylake_core.jpg https://software.intel.com/sites/default/files/managed/5f/99... https://twitter.com/GPUsAreMagic/status/1256866465577394181 https://twitter.com/GPUsAreMagic/status/1256866465577394181 these too (speculative)
    - jagger27 6y ago
      And the GPUs are usually the bigger component.
- solinent 6y ago
  Yes, in my opinion, the CPU model is dying as we fail to miniaturize we will be forced towards data parallel algorithms which benefit from speedups through SIMD. Ultimately AVX will be the future of Intel if they want to compute with GPUs and if everyone stays silicon.
  - cogman10 6y ago
    My view is that data parallel is here to stay and what's likely to happen is we'll see more power pumped into the instructions. Things like "If lane A > lane B add C to lane D" Eventually, I see us going basically full EPIC. That is "here's a batch of instructions that can happen in parallel, run it".
barkingcat 6y ago
I might suggest thinking along the lines that hardware engineers reject many features because of other constraints that impact the processor. The failure of the larrabee team to bring something competitive to the market a decade ago doesn't necessarily imply it will become competitive just because it was brought "on-chip" instead of as an additional card or a co-processor. If it worked all that well, it would be a product of its own, able to compete against Nvidia in GPU and GPGPU tasks.
- corysama 6y ago
  An example of what we are talking about is _mm_dp_ps (4-component dot product). It was heavily requested by game devs for SSE2 but didn't show up until 4.1. And, even then it wasn't much better than hand-rolling it with old instructions. The hardware engineers were "correct". It's a difficult instruction to make fast compared to completely independent mul-adds. Game devs "should just" re-architect their code to not deal with small situations individually and instead be more bulk-data-processing like SPECfp because that's what's friendly to the hardware. But, that's dismissing how the software actually worked --especially back then. 16 years later, many, but far from all of AAA game engines more closely resemble SPECfp. But, all along the way, and in tons of 3D graphics and gameplay code still today, it would have been nice to reliably have one of the most common and fundamental operations in games well-baked into the chips.
Dylan16807 6y ago
AVX-512 is two things though. It's new instructions for the vector units, and it's an extreme widening of those vector units. Your praise here is for the former, and Linus' complaints seem to generally be targeted at the latter.
- corysama 6y ago
  Yep. The latter is fallout from the hardware convergence I talk about in my reply to gameswithgo.
haecceity 6y ago
AVX-512 causes frequency throttling for thermal reasons. https://lemire.me/blog/2018/08/25/avx-512-throttling-heavy-instructions-are-maybe-not-so-dangerous/ https://lemire.me/blog/2018/08/25/avx-512-throttling-heavy-i... https://lemire.me/blog/2018/08/24/trying-harder-to-make-avx-512-look-bad-my-quantified-and-reproducible-results/ https://lemire.me/blog/2018/08/24/trying-harder-to-make-avx-... https://lemire.me/blog/2018/08/15/the-dangers-of-avx-512-throttling-a-3-impact/ https://lemire.me/blog/2018/08/15/the-dangers-of-avx-512-thr... Any performance gain from fewer cycle cost may be lost from the CPU being slower. Maybe there are situations where AVX-512 is faster but how is one to reason about performance when some instructions can slow the CPU.
- greggyb 6y ago
  > but how is one to reason about performance when some instructions can slow the CPU. The same way one should always do so on an advanced out-of-order CPU largely being programmed in modern high level languages: through thorough profiling and performance testing.
  - deleuze 6y ago
    Empiricism? In my software development? No way.
    - haecceity 6y ago
      I appreciate the humor but that's more time consuming than comparing instruction cycle counts and data dependencies. It's also workload dependent when it wouldn't be if they didn't come up with this throttling thing even if out of order execution makes it complicated.
      - greggyb 6y ago
        Most programmers are not regularly writing primarily assembly or reviewing the assembly generated by their compiler. These are the scenarios where instruction cycle counts become relevant - everything else is abstracted too far to rely on anything other than profiling and testing. If that's you, then I'm sorry for my response which didn't really take you into account.
serf 6y ago
Here's how I think of Intel chip features now after years of getting burned : 1)Has the feature been in the chip for one generation? True -> Skip this chip, (if I need to feature, that is) it'll have the feature disabled by microcode after someone realizes it may be a security or inaccuracy issue False -> Go ahead and buy it, probably at least a 50/50 coin flip that the feature will stay stable

gameswithgo 6y ago

On the one hand AVX-512 is incredible, the performance gains from AVX-2 to AVX-512 are often more than the 2x you would hope for. On the other hand, when you can use AVX-512 you can often use a GPU instead, to greater benefit. So linus may be right, overall.

smabie 6y ago
Are you sure l? Gains between 10% and 50% are pretty common, but I've never heard of a 100% or greater improvement.
- otherjason 6y ago
  Yes, these applications do exist. The speedups mainly come from the more flexible instruction set that AVX-512 provides (so in many cases you can replace several instructions with just one), plus the increased register set (32 architectural SIMD registers versus 16 on previous generations). If your algorithm isn't sensitive to either one of these, then yes, the speedups are often smaller, particularly if the limiting factor in performance is memory bandwidth and/or latency (AVX-512 doesn't help at all there), or if the speedups aren't enough to overcome clock throttling (which IMO is a bit overblown; most people cite the Cloudflare blog article from a few years ago that complained about throttling on Xeon Silver processors, but throttling is much less of an issue on higher-TDP Gold/Platinum CPUs). However, I have seen specific applications that are >100% faster with AVX-512.
- TinkersW 6y ago
  If you write SoA code, know what are you doing, and aren't memory bound, 100% speedup isn't unreasonable. Current AVX512 implementations might not always get there, as I believe some instructions are dual issuing at 256bit under the hood.
corysama 6y ago
In my observations, GPUs are the way they are largely because Physics demands sacrifices if you want that kind of performance. Those same demands apply to CPUs, but Intel has been fighting against them because software engineers hate to change anything about how they code. Well... the fight was lost years ago and Intel is conceding to Physics by integrating architecture that was originally designed for a CPU-GPU hybrid. With AVX-512 and CUDA, Intel and Nvidia are trying to find the best developer experience they can to deal with the inevitable future of computing that is somewhere between those two experiences currently.
- Const-me 6y ago
  > inevitable future of computing that is somewhere between those two experiences currently. Not sure that ever happens. The two are very different in their performance characteristics. CPUs have to stay focused on single-threaded performance. Single threaded performance = minimizing latency, that’s why in CPUs so many transistors and electricity is spent on cache hierarchy and coherency protocol, instructions reordering, ILP, microops fusion, prefetcher, branch prediction incl. indirect one, and so on. GPUs don’t care about single-threaded performance, all they care is throughput on massively parallel tasks. They ignore memory latency, switching threads until data arrives, makes caches much simpler. They don’t need high clock frequencies (this alone improves their flops/watt by a large factor), don’t do branch prediction, memory model is less strict, and so on. That’s how they can spend that many transistors on FPUs and ALUs. P.S. Intel tried many-slow-cores with Larrabee / knights landing / etc., all they got was a very expensive lesson, people didn’t want that stuff. I’ve heard a rumor initially Sony wanted to use another Cell CPU for the GPU in PS3, again, didn’t worked well enough and they negotiated with NVidia instead.
  - corysama 6y ago
    All true. Problem is that single-threaded latency minimization has been tapped out for a long time now. Physics says No. Memory is high-latency because of the speed of light. Larger caches have diminishing returns. Clocks can't go faster without melting the chips. There is no more prediction and pipelining to be dug out of a serial stream of instructions. We can still add more transistors. But, there is nowhere for them to contribute but in going wider. High latency, high throughput. High concurrency and careful management of the memory hierarchy are the only ways to pull that off. When you accept that fate, you start out with the Cell processor and you pass through Nvidia's Ampere chip --which has somewhat more thread independence and better automatic caching than past GPUs, but also finally gained manual async DMAs to local SRAM like the Cell had. Where we'll end up, I don't know.
rumanator 6y ago
> On the one hand AVX-512 is incredible, the performance gains from AVX-2 to AVX-512 are often more than the 2x you would hope for. Your assertion sounds like a gross exaggeration. Some benchmarks show only a few percent improvements (i.e.14% improvements on synthetic benchmarks), and results range between no improvements and almost 2x in some synthetic benchmarks. https://www.phoronix.com/scan.php?page=article&item=gcc9-skylake-avx512&num=1 https://www.phoronix.com/scan.php?page=article&item=gcc9-sky...
- gameswithgo 6y ago
  Those look like benchmarks comparing just autovectorized compilation of some tools and measuring overall runtime. I am referring to cases of hand tuned use SIMD. The bSIMD project used to have a suite of benchmarks up and some where very very impressive for AVX-512. The page appears to be gone or moved now though. To be clear I am not suggest "avx-512 will cause programs to run twice as fast or more", just that specific functions that used to use AVX-2, may be more than twice as fast if you move them to AVX-512.
  - vardump 6y ago
    Some SIMD optimization cases can occasionally exceed scalar versions by more than lane width multiply. Often because of branch elision, but also less dependency chains stalls due to the larger register file. There are also some special instructions that can be very useful in SIMD case (packing, unpacking, permutation, etc.).
  - rumanator 6y ago
    > Those look like benchmarks comparing just autovectorized compilation of some tools and measuring overall runtime. I am referring to cases of hand tuned use SIMD. I presented data that showed that AVX-512 in reality only brings margins performance improvements. If you have any data that shows otherwise then feel free to present it. Still, hand-tuned assembly is hardly a relevant example, let alone selling point, as it creates hypothetical expectations that are never ever matched in reality.
mvanveen 6y ago
In many cases within regulated environments getting access to GPU in e.g. a customer's on-prem solution is prohibitive. In enterprise ML contexts on tabular data problems we found there are a lot of cases where even training can be greatly sped up by leveraging AVX instruction support in e.g. tensorflow builds. The gains from AVX instructions could boost training time by ~20% on the GAN use cases I profiled.
- btilly 6y ago
  So you're saying that we need to solve organizational stupidity by complicating all chips for everyone so that bad organizations can get a performance boost on specialized tasks? What percentage of consumers with these chips installed do you think are getting a performance win? Do you think it might be as high as 1%? Do you think that the same resources devoted elsewhere might be worth more than 1% to that 99%? Whether that is in reduced cost, reduced bugs, or a boost for more widely used operations. Yes, if you target a specific use case for a specific set of people, you can give them a nice win. But you shouldn't lose sight of the fact that CPUs cover a lot of use cases for a lot of people. And simplifying then focusing on the core mission is better for everyone in the end.
  - anoncareer0212 6y ago
    Linus somewhat waved away the idea of tradeoffs, which is fine, he was speaking in generalities. Turning factual reports of where the tradeoff was helpful into a strawman insulting the reporter, and the users who benefit, is neither charitable nor illuminating.
CoolGuySteve 6y ago
I agree about GPUs. Intel used to have a low-end CPU with a 128MB L4 cache that was shared with the iGPU. It turns out having that large cache improved a lot of regular software too. I'd rather see die space used for that than AVX512. The only AVX512 instructions that actually seem essential imo are the VNNI extensions. There are a lot of use cases for a neural network that is trained on the GPU but where inference is done on a bunch of CPUs because it's cheaper than buying/renting inference hardware.
- cogman10 6y ago
  I think you are thinking AVX takes up more diespace than it does. Only a tiny fraction of all CPU diespace is dedicated to the actual CPU. The rest goes primarily to cache and partially to the various controllers on the CPU (memory controller, PCI-Express controller, etc). There's a reason AMD is and Intel are able to cram full GPUs onto the same die as the CPU.
  - Dylan16807 6y ago
    https://imgur.com/r/intel/4SG36 https://imgur.com/r/intel/4SG36 Yeah, here's a good image of how much space it costs to add a 'second' AVX-512 unit to certain SKUs. Cutting AVX-512 support entirely would save a bit more space, and the total would be enough to make a difference, but it's not an extreme amount. https://software.intel.com/sites/default/files/managed/5f/99/Skylake_core.jpg https://software.intel.com/sites/default/files/managed/5f/99... https://twitter.com/GPUsAreMagic/status/1256866465577394181 https://twitter.com/GPUsAreMagic/status/1256866465577394181 these too (speculative)
  - jagger27 6y ago
    And the GPUs are usually the bigger component.
solinent 6y ago
Yes, in my opinion, the CPU model is dying as we fail to miniaturize we will be forced towards data parallel algorithms which benefit from speedups through SIMD. Ultimately AVX will be the future of Intel if they want to compute with GPUs and if everyone stays silicon.
- cogman10 6y ago
  My view is that data parallel is here to stay and what's likely to happen is we'll see more power pumped into the instructions. Things like "If lane A > lane B add C to lane D" Eventually, I see us going basically full EPIC. That is "here's a batch of instructions that can happen in parallel, run it".

barkingcat 6y ago

I might suggest thinking along the lines that hardware engineers reject many features because of other constraints that impact the processor. The failure of the larrabee team to bring something competitive to the market a decade ago doesn't necessarily imply it will become competitive just because it was brought "on-chip" instead of as an additional card or a co-processor. If it worked all that well, it would be a product of its own, able to compete against Nvidia in GPU and GPGPU tasks.

corysama 6y ago
An example of what we are talking about is _mm_dp_ps (4-component dot product). It was heavily requested by game devs for SSE2 but didn't show up until 4.1. And, even then it wasn't much better than hand-rolling it with old instructions. The hardware engineers were "correct". It's a difficult instruction to make fast compared to completely independent mul-adds. Game devs "should just" re-architect their code to not deal with small situations individually and instead be more bulk-data-processing like SPECfp because that's what's friendly to the hardware. But, that's dismissing how the software actually worked --especially back then. 16 years later, many, but far from all of AAA game engines more closely resemble SPECfp. But, all along the way, and in tons of 3D graphics and gameplay code still today, it would have been nice to reliably have one of the most common and fundamental operations in games well-baked into the chips.

Dylan16807 6y ago

AVX-512 is two things though. It's new instructions for the vector units, and it's an extreme widening of those vector units. Your praise here is for the former, and Linus' complaints seem to generally be targeted at the latter.

corysama 6y ago
Yep. The latter is fallout from the hardware convergence I talk about in my reply to gameswithgo.

haecceity 6y ago

AVX-512 causes frequency throttling for thermal reasons. https://lemire.me/blog/2018/08/25/avx-512-throttling-heavy-instructions-are-maybe-not-so-dangerous/ https://lemire.me/blog/2018/08/25/avx-512-throttling-heavy-i... https://lemire.me/blog/2018/08/24/trying-harder-to-make-avx-512-look-bad-my-quantified-and-reproducible-results/ https://lemire.me/blog/2018/08/24/trying-harder-to-make-avx-... https://lemire.me/blog/2018/08/15/the-dangers-of-avx-512-throttling-a-3-impact/ https://lemire.me/blog/2018/08/15/the-dangers-of-avx-512-thr... Any performance gain from fewer cycle cost may be lost from the CPU being slower. Maybe there are situations where AVX-512 is faster but how is one to reason about performance when some instructions can slow the CPU.

greggyb 6y ago
> but how is one to reason about performance when some instructions can slow the CPU. The same way one should always do so on an advanced out-of-order CPU largely being programmed in modern high level languages: through thorough profiling and performance testing.
- deleuze 6y ago
  Empiricism? In my software development? No way.
  - haecceity 6y ago
    I appreciate the humor but that's more time consuming than comparing instruction cycle counts and data dependencies. It's also workload dependent when it wouldn't be if they didn't come up with this throttling thing even if out of order execution makes it complicated.
    - greggyb 6y ago
      Most programmers are not regularly writing primarily assembly or reviewing the assembly generated by their compiler. These are the scenarios where instruction cycle counts become relevant - everything else is abstracted too far to rely on anything other than profiling and testing. If that's you, then I'm sorry for my response which didn't really take you into account.

serf 6y ago

Here's how I think of Intel chip features now after years of getting burned : 1)Has the feature been in the chip for one generation? True -> Skip this chip, (if I need to feature, that is) it'll have the feature disabled by microcode after someone realizes it may be a security or inaccuracy issue False -> Go ahead and buy it, probably at least a 50/50 coin flip that the feature will stay stable