5 ms·

> except for byte-level processing, variable-length codecs, or mixed-precision numerics. That never works with autovectorization and can’t be solved with genera

by janwas 2y ago

> except for byte-level processing, variable-length codecs, or mixed-precision numerics. That never works with autovectorization and can’t be solved with general-purpose SIMD wrappers. Counterexamples: Chromium's byte-level HTML scanning, several var-len bit packing codecs, and Gemma.cpp's matmul is mixed-precision (fp8->bf16->fp32->fp64). All written with the Highway general-purpose SIMD wrapper. Please revise your post or expand upon the structure/dispatch concern.

ashvardanian 2y ago
Interesting references! I remember that Gemma.cpp used Highway, but I haven't checked the others much. Here is a puzzle, then. Let's say we are checking a single register of bytes for element-wise equality with another register of the same size. In AVX2, the output is another YMM register of 0xFF or 0x00 values. In AVX-512, for full ZMM-wide comparisons, it's a 64-bit mask in the K register. I struggle to see a good way to abstract such things, even for two consecutive SIMD generations on x86.
- dzaima 2y ago
  A simple enough abstraction is to have the mask type have inputs of both element size and count; so for a ≤256-bit product of those you do the homogeneous-bit elements, and for 512-bit you do the packed-bit values. Conversion methods on the mask can convert those to either an explicit full vector, or a packed bit integer, as you need. (and clang can optimize out unnecessary conversions between the two)
  - ashvardanian 2y ago
    Sure, that would make sense, but in AVX-512, there is also a comparison variant for 2x256-bit inputs that outputs a 32-bit mask and another one for 2x128 inputs and a 16-bit output mask. I would use those in different ways, occasionally mixing with the old AVX2 variant, depending on what I’m doing. I have tried to create a generalizable SIMD framework several times in the past, mostly in 2015-17, and still don’t have a good abstraction even between AVX2 and AVX-512.
    - dzaima 2y ago
      You'd just produce the ≤AVX2-style result; clang will switch to using the mask-returning comparison instrs if the usage ends up being a mask (gcc doesn't do such fancy things, so I guess getting the last bits of speedup from this strategy depends on how much control over compiler selection you choose to have). And if you really care about doing different things for ≤AVX2 vs AVX-512 masks, you're definitely in the "squeeze out the last 1.4x" camp and not the "simple ~8x speedup" one. And, realistically, you won't care about this on every single comparison, so where you really want to explicitly use mask-returning comparisons you could just switch to using intrinsics directly (or a different abstraction over them) temporarily. As a side-note, mask-returning comparisons have 2x less throughput than homogeneous-bit-returning ones for xmm/ymm as far as uops.info data goes[1] (and use port 5!), so this strategy is kinda just what you want really. (another strategy can be to just return the arch-specific result type, still preferring the ≤AVX2 style (with, if desired, a separate set of comparison ops for the AVX-512-mask output), and making mask-consuming ops polymorphic over the two) [1]: https://uops.info/table.html?search=cmpeqb%20ymm%29&cb_lat=on&cb_tp=on&cb_ports=on&cb_SKX=on&cb_TGL=on&cb_ADLP=on&cb_ZEN4=on&cb_measurements=on&cb_avx2=on&cb_avx512=on https://uops.info/table.html?search=cmpeqb%20ymm%29&cb_lat=o...
  - janwas 2y ago
    Yes indeed, this is what we do :) There is an opaque Mask type for which operations such as CountTrue, AllFalse etc are provided. If you really want the one or other representation, VecFromMask and BitsFromMask/StoreMaskBits convert as required. The former is a no-op on AVX2. As to the AVX2 comparison, for ICL I think we can do two of those per cycle, so no more than the throughput of an AVX-512 mask comparison. The question is what we do with it afterwards - for VQSort (one of the few applications where comparisons are really the bottleneck?) we are much happier to have the packed bits, because that can feed into vpcompress or a LUT implementing that.

by janwas 2y ago

ashvardanian 2y ago

Interesting references! I remember that Gemma.cpp used Highway, but I haven't checked the others much. Here is a puzzle, then. Let's say we are checking a single register of bytes for element-wise equality with another register of the same size. In AVX2, the output is another YMM register of 0xFF or 0x00 values. In AVX-512, for full ZMM-wide comparisons, it's a 64-bit mask in the K register. I struggle to see a good way to abstract such things, even for two consecutive SIMD generations on x86.

dzaima 2y ago
A simple enough abstraction is to have the mask type have inputs of both element size and count; so for a ≤256-bit product of those you do the homogeneous-bit elements, and for 512-bit you do the packed-bit values. Conversion methods on the mask can convert those to either an explicit full vector, or a packed bit integer, as you need. (and clang can optimize out unnecessary conversions between the two)
- ashvardanian 2y ago
  Sure, that would make sense, but in AVX-512, there is also a comparison variant for 2x256-bit inputs that outputs a 32-bit mask and another one for 2x128 inputs and a 16-bit output mask. I would use those in different ways, occasionally mixing with the old AVX2 variant, depending on what I’m doing. I have tried to create a generalizable SIMD framework several times in the past, mostly in 2015-17, and still don’t have a good abstraction even between AVX2 and AVX-512.
  - dzaima 2y ago
    You'd just produce the ≤AVX2-style result; clang will switch to using the mask-returning comparison instrs if the usage ends up being a mask (gcc doesn't do such fancy things, so I guess getting the last bits of speedup from this strategy depends on how much control over compiler selection you choose to have). And if you really care about doing different things for ≤AVX2 vs AVX-512 masks, you're definitely in the "squeeze out the last 1.4x" camp and not the "simple ~8x speedup" one. And, realistically, you won't care about this on every single comparison, so where you really want to explicitly use mask-returning comparisons you could just switch to using intrinsics directly (or a different abstraction over them) temporarily. As a side-note, mask-returning comparisons have 2x less throughput than homogeneous-bit-returning ones for xmm/ymm as far as uops.info data goes[1] (and use port 5!), so this strategy is kinda just what you want really. (another strategy can be to just return the arch-specific result type, still preferring the ≤AVX2 style (with, if desired, a separate set of comparison ops for the AVX-512-mask output), and making mask-consuming ops polymorphic over the two) [1]: https://uops.info/table.html?search=cmpeqb%20ymm%29&cb_lat=on&cb_tp=on&cb_ports=on&cb_SKX=on&cb_TGL=on&cb_ADLP=on&cb_ZEN4=on&cb_measurements=on&cb_avx2=on&cb_avx512=on https://uops.info/table.html?search=cmpeqb%20ymm%29&cb_lat=o...
- janwas 2y ago
  Yes indeed, this is what we do :) There is an opaque Mask type for which operations such as CountTrue, AllFalse etc are provided. If you really want the one or other representation, VecFromMask and BitsFromMask/StoreMaskBits convert as required. The former is a no-op on AVX2. As to the AVX2 comparison, for ICL I think we can do two of those per cycle, so no more than the throughput of an AVX-512 mask comparison. The question is what we do with it afterwards - for VQSort (one of the few applications where comparisons are really the bottleneck?) we are much happier to have the packed bits, because that can feed into vpcompress or a LUT implementing that.