3 ms·

Yeah but AMD Zen4 has AVX512 enabled by default. And they're not too shabby with the performance either. AVX512 is the future, but maybe not Intel's future, ir

Yeah but AMD Zen4 has AVX512 enabled by default. And they're not too shabby with the performance either. AVX512 is the future, but maybe not Intel's future, ironically.

CamperBob2 4y ago
If I need to crunch small amounts of data in a hurry, existing instructions are fine for that. If I need to crunch large amounts of data in a hurry, I'll send it to a GPU. The CPU has no chance to compete with that. I honestly don't understand who/what AVX512 is really for, other than artificial benchmarks that are intentionally engineered to depend on AVX512 far more often than any real-world application would.
- nwiswell 4y ago
  What happens when you have a moderate amount of data and you can't wait for GPU latency?
  - CamperBob2 4y ago
    I can't think of a case like that, where the difference between AVX512 and other instruction sets would be humanly perceptible. Those situations usually end up constrained by memory bandwidth, not CPU or cache throughput. I'm sure those cases exist, but they don't justify such a large chunk of silicon. And they definitely don't justify slowing down the rest of the CPU.
    - dragontamer 4y ago
      Web Server. Say... this web page. Inspector says that this web page, we're talking on is 48kB in size. That's small enough to fit inside L1 cache. Every single connection to the Web Server needs its own AES-key for encryption sake. The 48kB of data (such as my posts above, or your posts) are hot in Cache, but if 100 visitors to this webpage need to get it, HTTPS needs to happen 100x different times with 100x different keys. So this 48kB text message (consisting of all the comments of this page) are going to have to be encrypted with different, random, AES keys to deliver the messages to you or me. AES operates on 16-bytes at a time, and AES-GCM is a newer algorithm that allows for all 48,000+ bytes to be processed in parallel. AVX-512 AES instructions are ideal for processing this data, are they not? And processing them 4x faster (since 4x instances of AES are occurring in parallel, since AVX512 can work on 64 bytes per tick / 16 bytes per AES instance), is a lot better than just doing it 16-bytes at at time with the legacy AES-NI instructions. ---------- Despite being a parallel problem, this will never be worthwhile to send to the GPU. First, GPUs don't have AES-ni instructions. But even if they did, it would take longer to talk to the GPU than for the AVX512-AES instructions to operate on the 48kB of data (again: ~40,000 clock ticks to just start talking with the GPU in practice). In that amount of time, you would have finished encrypting the payload and have sent it off.
      - Dylan16807 4y ago
        I've seen a lot about AVX-512 and didn't know those instructions existed until just now. They're not exactly generic vector instructions. And that's a nice improvement, but is AES-NI ever slow enough to matter? The numbers I found were inconsistent but all very fast. Probably more important, there's a 256 bit version of that instruction. You can get half of that extreme throughput without AVX-512.
        dragontamer 4y ago
        That Netflix guy who keeps optimizing their servers keeps coming back every year or so, talking about the latest optimizations he added. http://nabstreamingsummit.com/wp-content/uploads/2022/05/2022-Streaming-Summit-Netflix.pdf http://nabstreamingsummit.com/wp-content/uploads/2022/05/202... And a surprising amount of it was in TLS optimizations, in particular, offloading TLS to the hardware (Apparently Mellanox ConnectX ethernet adapters can do AES offload now, so the CPU doesn't have to worry about it). Since Mellanox ConnectX adapters are trying to solve the AES problem still, I have to imagine that its a significant portion of a lot of server's workloads. Intel / AMD are obviously interested in it enough to upgrade AES to 4x wide in the AVX512 instruction set. I can't say its particularly useful in any of _my_ workloads. But it seems to come up enough in those hyper-optimized web servers / presentations.
    - nwiswell 4y ago
      Something that comes to mind is real-time controls, like for high speed manufacturing, rockets and jets, medical robots, etc. These computations are often highly vectorized and are extremely latency-sensitive, for obvious reasons.
- dragontamer 4y ago
  > If I need to crunch large amounts of data in a hurry, I'll send it to a GPU. The CPU has no chance to compete with that. It takes literally 1 to 10 microseconds (10,000 nanoseconds) to talk to a GPU over PCIe. In the 40,000 clock cycles, you could have processed 2.5 MB of data with AVX512 instructions *BEFORE* the GPU even is aware that you're talking to it. Then you gotta start passing the data to the GPU, the GPU then has to process the data, and then it has to send it all back. All in all, SIMD instructions on CPU-side are worthwhile for anything less than 8MB for sure, maybe less than 16MB or 32MB, depending on various details. ---------- That's one core. If your one-core machine can talk to the other 32-cores or 128-cores of your computer (see 64-core EPYC dual-socket computers), you can communicate to another core in just 50-nanoseconds or so (~200 clock penalty), and those other cores can be processing AVX512 as well. So if you're able to use parallel programming on a CPU, its probably closer to 1GB+ of data before it actually is truly an 'obvious' choice to talk to the GPU, rather than just keeping it on CPU-only side. --------- Example of practical use: AES-GCM can be processed using AVX512 in parallel (each AES-GCM block is a parallel instance, the entire AES-GCM stream is in parallel), but no one will actually use GPUs to process this... because AES-instructions are single-clock tick (or faster!!) on modern CPUs like Intel / AMD Zen. That's just going to happen whenever you go to an TLS1.2 or HTTPS instance, which is pretty much all the time? Like, every single byte coming out of Youtube is over HTTPS these days and needs to be decrypted before further processing.
  - sitkack 4y ago
    This is why on-die crypto accelerators make so much sense.
    - janwas 4y ago
      Hopefully they're updated for the new post-quantum algorithms. Which would you rather have: some fixed-function unit shared between all cores (load balancing? what if you're suddenly doing crypto stuff on many cores?), or the general-purpose tools for running any algorithm on any core?
      - deleted 4y ago
- ilyt 4y ago
  If you look at what AVX512 is often used (as the article mentions), its less of math and more of just speeding up processing of deserializers and various other things that can do bunch of operation on few bytes at a time. Which does look like hilarious waste of silicon, it could cut all of the float/division transistors off and still be plenty useful
  - CamperBob2 4y ago
    Exactly. It's good at providing speed improvements which will never be noticed in the real world.
    - janwas 4y ago
      That's a bold claim which requires extraordinary evidence. Sufficient counterexamples, most already mentioned in this discussion: databases, NN-512, TLS(AES), JPEG XL decoding (1.5x speedup), ...
- sitkack 4y ago
  > If I need to crunch large amounts of data in a hurry, I'll send it to a GPU. The CPU has no chance to compete with that. PCIe is real bottleneck that affects bandwidth and latency. It only really works if your data is already resident on the GPU, your kernels are fixed and the amount of data returned is small. With HBM and VCache, we will see main memory bandwidth over 1TB/s for consumer high perf cpus in the near future, at those rates, GPUs won't make sense. GPUs are basically ASICs that can take advantage of hundreds of GB/s of memory bandwidth, when the CPU can do scans at that same rate, the necessity of a GPU is greatly reduced.
- bob1029 4y ago
  > If I need to crunch large amounts of data in a hurry, I'll send it to a GPU. The CPU has no chance to compete with that. Latency is a hell of a thing. Anything over a millisecond is an absolute eternity, and I know the GPU imposes more than that for most practical applications - especially gaming.

Yeah but AMD Zen4 has AVX512 enabled by default. And they're not too shabby with the performance either. AVX512 is the future, but maybe not Intel's future, ironically.

CamperBob2 4y ago

If I need to crunch small amounts of data in a hurry, existing instructions are fine for that. If I need to crunch large amounts of data in a hurry, I'll send it to a GPU. The CPU has no chance to compete with that. I honestly don't understand who/what AVX512 is really for, other than artificial benchmarks that are intentionally engineered to depend on AVX512 far more often than any real-world application would.

nwiswell 4y ago
What happens when you have a moderate amount of data and you can't wait for GPU latency?
- CamperBob2 4y ago
  I can't think of a case like that, where the difference between AVX512 and other instruction sets would be humanly perceptible. Those situations usually end up constrained by memory bandwidth, not CPU or cache throughput. I'm sure those cases exist, but they don't justify such a large chunk of silicon. And they definitely don't justify slowing down the rest of the CPU.
  - dragontamer 4y ago
    Web Server. Say... this web page. Inspector says that this web page, we're talking on is 48kB in size. That's small enough to fit inside L1 cache. Every single connection to the Web Server needs its own AES-key for encryption sake. The 48kB of data (such as my posts above, or your posts) are hot in Cache, but if 100 visitors to this webpage need to get it, HTTPS needs to happen 100x different times with 100x different keys. So this 48kB text message (consisting of all the comments of this page) are going to have to be encrypted with different, random, AES keys to deliver the messages to you or me. AES operates on 16-bytes at a time, and AES-GCM is a newer algorithm that allows for all 48,000+ bytes to be processed in parallel. AVX-512 AES instructions are ideal for processing this data, are they not? And processing them 4x faster (since 4x instances of AES are occurring in parallel, since AVX512 can work on 64 bytes per tick / 16 bytes per AES instance), is a lot better than just doing it 16-bytes at at time with the legacy AES-NI instructions. ---------- Despite being a parallel problem, this will never be worthwhile to send to the GPU. First, GPUs don't have AES-ni instructions. But even if they did, it would take longer to talk to the GPU than for the AVX512-AES instructions to operate on the 48kB of data (again: ~40,000 clock ticks to just start talking with the GPU in practice). In that amount of time, you would have finished encrypting the payload and have sent it off.
    - Dylan16807 4y ago
      I've seen a lot about AVX-512 and didn't know those instructions existed until just now. They're not exactly generic vector instructions. And that's a nice improvement, but is AES-NI ever slow enough to matter? The numbers I found were inconsistent but all very fast. Probably more important, there's a 256 bit version of that instruction. You can get half of that extreme throughput without AVX-512.
      - dragontamer 4y ago
        That Netflix guy who keeps optimizing their servers keeps coming back every year or so, talking about the latest optimizations he added. http://nabstreamingsummit.com/wp-content/uploads/2022/05/2022-Streaming-Summit-Netflix.pdf http://nabstreamingsummit.com/wp-content/uploads/2022/05/202... And a surprising amount of it was in TLS optimizations, in particular, offloading TLS to the hardware (Apparently Mellanox ConnectX ethernet adapters can do AES offload now, so the CPU doesn't have to worry about it). Since Mellanox ConnectX adapters are trying to solve the AES problem still, I have to imagine that its a significant portion of a lot of server's workloads. Intel / AMD are obviously interested in it enough to upgrade AES to 4x wide in the AVX512 instruction set. I can't say its particularly useful in any of _my_ workloads. But it seems to come up enough in those hyper-optimized web servers / presentations.
  - nwiswell 4y ago
    Something that comes to mind is real-time controls, like for high speed manufacturing, rockets and jets, medical robots, etc. These computations are often highly vectorized and are extremely latency-sensitive, for obvious reasons.
dragontamer 4y ago
> If I need to crunch large amounts of data in a hurry, I'll send it to a GPU. The CPU has no chance to compete with that. It takes literally 1 to 10 microseconds (10,000 nanoseconds) to talk to a GPU over PCIe. In the 40,000 clock cycles, you could have processed 2.5 MB of data with AVX512 instructions *BEFORE* the GPU even is aware that you're talking to it. Then you gotta start passing the data to the GPU, the GPU then has to process the data, and then it has to send it all back. All in all, SIMD instructions on CPU-side are worthwhile for anything less than 8MB for sure, maybe less than 16MB or 32MB, depending on various details. ---------- That's one core. If your one-core machine can talk to the other 32-cores or 128-cores of your computer (see 64-core EPYC dual-socket computers), you can communicate to another core in just 50-nanoseconds or so (~200 clock penalty), and those other cores can be processing AVX512 as well. So if you're able to use parallel programming on a CPU, its probably closer to 1GB+ of data before it actually is truly an 'obvious' choice to talk to the GPU, rather than just keeping it on CPU-only side. --------- Example of practical use: AES-GCM can be processed using AVX512 in parallel (each AES-GCM block is a parallel instance, the entire AES-GCM stream is in parallel), but no one will actually use GPUs to process this... because AES-instructions are single-clock tick (or faster!!) on modern CPUs like Intel / AMD Zen. That's just going to happen whenever you go to an TLS1.2 or HTTPS instance, which is pretty much all the time? Like, every single byte coming out of Youtube is over HTTPS these days and needs to be decrypted before further processing.
- sitkack 4y ago
  This is why on-die crypto accelerators make so much sense.
  - janwas 4y ago
    Hopefully they're updated for the new post-quantum algorithms. Which would you rather have: some fixed-function unit shared between all cores (load balancing? what if you're suddenly doing crypto stuff on many cores?), or the general-purpose tools for running any algorithm on any core?
    - deleted 4y ago
ilyt 4y ago
If you look at what AVX512 is often used (as the article mentions), its less of math and more of just speeding up processing of deserializers and various other things that can do bunch of operation on few bytes at a time. Which does look like hilarious waste of silicon, it could cut all of the float/division transistors off and still be plenty useful
- CamperBob2 4y ago
  Exactly. It's good at providing speed improvements which will never be noticed in the real world.
  - janwas 4y ago
    That's a bold claim which requires extraordinary evidence. Sufficient counterexamples, most already mentioned in this discussion: databases, NN-512, TLS(AES), JPEG XL decoding (1.5x speedup), ...
sitkack 4y ago
> If I need to crunch large amounts of data in a hurry, I'll send it to a GPU. The CPU has no chance to compete with that. PCIe is real bottleneck that affects bandwidth and latency. It only really works if your data is already resident on the GPU, your kernels are fixed and the amount of data returned is small. With HBM and VCache, we will see main memory bandwidth over 1TB/s for consumer high perf cpus in the near future, at those rates, GPUs won't make sense. GPUs are basically ASICs that can take advantage of hundreds of GB/s of memory bandwidth, when the CPU can do scans at that same rate, the necessity of a GPU is greatly reduced.
bob1029 4y ago
> If I need to crunch large amounts of data in a hurry, I'll send it to a GPU. The CPU has no chance to compete with that. Latency is a hell of a thing. Anything over a millisecond is an absolute eternity, and I know the GPU imposes more than that for most practical applications - especially gaming.