4 ms·

At some point I'm hoping to see AVX completely emulated in microcode and replaced with an embedded GPU core that can write to an L3 or L4 cache. This slow scal

At some point I'm hoping to see AVX completely emulated in microcode and replaced with an embedded GPU core that can write to an L3 or L4 cache. This slow scaling of 128->256->512 bits in the instruction set is more or less a solved problem in the GPU space with shader compilers and AVX would mostly be redundant with GPUs if it weren't for the memory bandwidth constraint. ie: When it comes to vector processing, go big or go home. AVX/SSE are a compromise from back when CPU die space and bandwidth was more precious. Now that we have 8-32 cores on a die with a good bus between them, it seems like duplicating those AVX units 8-32 times is less optimal.

dragontamer 8y ago
Please no. AVX's main advantage is that it is roughly 1-cycle away from your main registers, and 4-cycles away from L1 cache. Talking to and from L3 cache is on the order of 30 to 40 cycles, an order of magnitude slower. If your workload fits inside of 64kB, AVX is incredibly beneficial. If your workload fits within 8MB (L3 cache), you're starting to look at a point where maybe you should pipe that data to the GPU instead. A GPU call over PCIe is under 5 uS / 5000 nanoseconds, with a bandwidth of ~15GB/s. GPUs are certainly farther away than L3 cache, but if you're pushing L3 cache levels... you're getting close to the GPU anyway. --------- 128-bit SSE code is perfect for representing Complex Numbers (Two double-floats). 128-bit is also great for a x,y,z,w 32-bit vector. ------- GPUs are the "go big or go home" architecture. AVX's primary benefit is latency.
m0zg 8y ago
What you think of as "vector" processing is currently being used by compilers to speed up things you didn't think were vectorizable. This is possible only because these instructions are pretty cheap latency-wise. By introducing huge latency, you'd be ruining performance of autovectorization, which accounts for a lot of the performance gains in the past decade.
- deleted 8y ago
  [deleted]
- CoolGuySteve 8y ago
  After years of staring at HFT disassembly, I'm thoroughly disappointed by autovectorization. Here's the simplest godbolt example I could think of to illustrate, summing a string of fixed length and non-fixed length: https://godbolt.org/z/O_M5fU https://godbolt.org/z/O_M5fU You can see that the most recent GCC fails to use the AVX512 zmm registers even after being configured to do so (afaik) and also fails to use more than 4 registers. Clang does better, using zmm* and all the registers. But in both cases, the amount of code generated is quite large. If you compile with -Os instead of -O3, no vector instructions are used for some reason. So when you load this code, no matter what, you're loading a bunch of instruction cachelines, which will destroy most of your latency gain unless the input is very large. And even if your input is large, you'll fault that data anyways. So what's the point of doing this on the CPU again?
  - m0zg 8y ago
    Staring is not enough, you need to benchmark. You might be surprised. The results are often counterintuitive, because it's hard for a human to account for how the program _actually_ gets executed, which instructions go in which order, port utilization, data dependencies, CPU bugs, effects of alignment (or, often, lack thereof on Intel CPUs), micro-op caching, etc etc. For all you know GCC deliberately avoided the use of AVX512 registers because AVX512 causes the CPU to throttle the clock. I've also found that ye olde -O2 flag recommendation which says -O3 is likely to be slower in practice because it produces bulkier machine code, nearly never holds up anymore. That said, I concede that autovectorization is not perfect. I can almost always beat it if I really want to, sometimes by quite a margin. But what you're proposing is unlikely to help matters, especially for smaller inputs that you're concerned about.
    - CoolGuySteve 8y ago
      I have benchmarked similar code extensively, this is a toy version of a FIX checksum. It's not "counterintuitive", it's just broken, especially for infrequently executed code where the instruction fetch stalls will kill your latency. You are welcomed to load up the godbolt example and try and create better assembly using O2 or whatever else you can think of. You can build the code on your machine and benchmark it too. But at the end of the day, the compiler output sucks. It's clearly optimized for synthetic benchmarks that chug through large datasets using a tiny amount of code that can fit in L1 cache even after the compiler blows up its footprint. But my point is that these large datasets with tiny processing kernels (like a video codec for example) are much better suited to different kinds of processors.

dragontamer 8y ago

Please no. AVX's main advantage is that it is roughly 1-cycle away from your main registers, and 4-cycles away from L1 cache. Talking to and from L3 cache is on the order of 30 to 40 cycles, an order of magnitude slower. If your workload fits inside of 64kB, AVX is incredibly beneficial. If your workload fits within 8MB (L3 cache), you're starting to look at a point where maybe you should pipe that data to the GPU instead. A GPU call over PCIe is under 5 uS / 5000 nanoseconds, with a bandwidth of ~15GB/s. GPUs are certainly farther away than L3 cache, but if you're pushing L3 cache levels... you're getting close to the GPU anyway. --------- 128-bit SSE code is perfect for representing Complex Numbers (Two double-floats). 128-bit is also great for a x,y,z,w 32-bit vector. ------- GPUs are the "go big or go home" architecture. AVX's primary benefit is latency.

m0zg 8y ago

What you think of as "vector" processing is currently being used by compilers to speed up things you didn't think were vectorizable. This is possible only because these instructions are pretty cheap latency-wise. By introducing huge latency, you'd be ruining performance of autovectorization, which accounts for a lot of the performance gains in the past decade.

deleted 8y ago
[deleted]
CoolGuySteve 8y ago
After years of staring at HFT disassembly, I'm thoroughly disappointed by autovectorization. Here's the simplest godbolt example I could think of to illustrate, summing a string of fixed length and non-fixed length: https://godbolt.org/z/O_M5fU https://godbolt.org/z/O_M5fU You can see that the most recent GCC fails to use the AVX512 zmm registers even after being configured to do so (afaik) and also fails to use more than 4 registers. Clang does better, using zmm* and all the registers. But in both cases, the amount of code generated is quite large. If you compile with -Os instead of -O3, no vector instructions are used for some reason. So when you load this code, no matter what, you're loading a bunch of instruction cachelines, which will destroy most of your latency gain unless the input is very large. And even if your input is large, you'll fault that data anyways. So what's the point of doing this on the CPU again?
- m0zg 8y ago
  Staring is not enough, you need to benchmark. You might be surprised. The results are often counterintuitive, because it's hard for a human to account for how the program _actually_ gets executed, which instructions go in which order, port utilization, data dependencies, CPU bugs, effects of alignment (or, often, lack thereof on Intel CPUs), micro-op caching, etc etc. For all you know GCC deliberately avoided the use of AVX512 registers because AVX512 causes the CPU to throttle the clock. I've also found that ye olde -O2 flag recommendation which says -O3 is likely to be slower in practice because it produces bulkier machine code, nearly never holds up anymore. That said, I concede that autovectorization is not perfect. I can almost always beat it if I really want to, sometimes by quite a margin. But what you're proposing is unlikely to help matters, especially for smaller inputs that you're concerned about.
  - CoolGuySteve 8y ago
    I have benchmarked similar code extensively, this is a toy version of a FIX checksum. It's not "counterintuitive", it's just broken, especially for infrequently executed code where the instruction fetch stalls will kill your latency. You are welcomed to load up the godbolt example and try and create better assembly using O2 or whatever else you can think of. You can build the code on your machine and benchmark it too. But at the end of the day, the compiler output sucks. It's clearly optimized for synthetic benchmarks that chug through large datasets using a tiny amount of code that can fit in L1 cache even after the compiler blows up its footprint. But my point is that these large datasets with tiny processing kernels (like a video codec for example) are much better suited to different kinds of processors.