4 ms·

I love how ever example of portable SIMD isn't portable. They specifies a constant SIMD width so it's non-portable. Well, not performance portable, but why are

I love how ever example of portable SIMD isn't portable. They specifies a constant SIMD width so it's non-portable. Well, not performance portable, but why are we using SIMD again?

tyho 2mo ago
Go's implementation is vector size independant https://pkg.go.dev/simd@master https://pkg.go.dev/simd@master
- IshKebab 2mo ago
  Sure but there's no real way to use that in a portable way, at least not a way that maximises performance on every CPU you run it on. That's pretty much impossible at the moment.
  - krapht 2mo ago
    Which is why I've never quite understood the appeal of portable SIMD libraries for performance-critical code. If I'm explicitly writing SIMD rather than relying on the auto-vectorizer, it's usually because I want access to the particular capabilities of the target ISA. For many problems, choosing the right instruction or instruction sequence makes a large difference. Portable SIMD abstractions necessarily expose some common semantic layer, but SIMD ISAs don't actually have equivalent capabilities. Instructions like pshufb, for example, enable algorithmic tricks that don't necessarily have an equally efficient analogue on another architecture. If maximum performance matters, I generally want intrinsics and architecture-specific implementations; if portability matters more, I'd rather move further up the abstraction stack and use something designed to target multiple architectures, such as ISPC. There are certainly cases where portable SIMD gets close enough to optimal, but I don't think there's a compiler or abstraction that can express every useful SIMD idiom and lower it equally efficiently across fundamentally different ISAs.
    - pjmlp 2mo ago
      Because usually they achieve a very good middle ground, they are useful for when autovectorization isn't good enough, and it is possible to give a little help to the compiler. There are many ways that performance matters without trying to win a F1 race. Go isn't alone, .NET, Java have similar portable libraries, and C++ is in the process of getting one.
    - kbolino 2mo ago
      Go doesn't have auto-vectorization in the first place, so its portable simd library is at least partly there to fill the gap.
MomsAVoxell 2mo ago
Why should it be portable? Honest question. SIMD seems to me, to be very platform specific. Maybe there are times one SIMD unit is not anothers' SIMD unit?
- camel-cdr 2mo ago
  The create is called portable_simd. There is no reason a portable_simd relu_dot implemention should need to specify the SIMD width. But the design and documentation of portable_simd makes the fixed size syntactically easy/the default and the width agnostic code harder.
  - dwattttt 2mo ago
    > There is no reason a portable_simd relu_dot implemention should need to specify the SIMD width. What should it choose then? I have a Zen 3 processor, and benchmarking some simd I did recently says 32 byte or 64 byte chunks was fastest. But I'm sure I'd get a different result on a different Zen, and different again on Intel's. How would the library decide what SIMD width I should use?
    - Groxx 2mo ago
      It'd need some kind of compile-time hardware-feature-detection, yea? That seems probably feasible since proc macros can do essentially anything they like (worryingly).
      - derefr 2mo ago
        Only if the end-user is the one compiling the software, on the same very system they'll be running it on. Which is true of GPU shader kernels, due to how GPU drivers work; but isn't generally true of CPU object code (unless you're on Gentoo.) What you'd actually want is a matrix of variant implementations burned into the binary, with runtime (or process-boot-time) hardware detection that swaps symbols out to point to the correct variant.
zamadatix 2mo ago
It should really be read/advertised as "portabler SIMD". It beats hoping the compiler autovectorizes everything well forever or writing architecture specific code manually again but is going to compromise on average performance vs platform specific SIMD.
- pjmlp 2mo ago
  .NET and Java have three levels of SIMD support, Go's ongoing efforts, and does the upcoming C++ standard. Autovectorization, depending on compiler's cleverness, really portable SIMD operations, and then the CPU specific SIMD ones. So this should be perfectly doable in crate that advertises as portable, while leaving the non portable stuff to another crate.
jandrewrogers 2mo ago
The capabilities of various SIMD ISAs don't have enough intersection to be portable outside of relatively trivial cases. Many of the somewhat unique capabilities are load-bearing, so you want to use them on architectures that support them. Taken in whole, someone who cares about performance would be using different data structures and algorithms depending on the specific SIMD architecture and that is nearly impossible to abstract in a library. Too many important but complex details are idiosyncratic to the implementation. Another way of looking at it is that our programming environments are not sufficiently powerful and expressive to create the necessary abstractions to make SIMD truly portable.
- simonask 2mo ago
  > The capabilities of various SIMD ISAs don't have enough intersection to be portable outside of relatively trivial cases. I would argue that the "trivial" cases (those relating to linear algebra in 3 dimensions) are also 95% of what people want SIMD for. If the API can achieve cross-platform and performant vector arithmetic, dot product, and matrix multiplication in the normal ways, that already covers a lot of what people actually need.
  - jandrewrogers 2mo ago
    Most use cases for SIMD are non-arithmetic in nature and don't assume tidy arrays of homogeneous types. I also use it for some computational geometry but that is the least interesting use case. SIMD is widely used throughout data infrastructure e.g. parsing data, complex constraint processing, parallel manipulation of heterogeneous data types, compression, etc. I even have an I/O scheduler written in AVX-512 that is many times faster than the scalar equivalent. The ability of SIMD to do complex manipulation of ordinary data structures several times faster than scalar code is under-rated. While linear algebra is the current thing, database engines have been using SIMD heavily for over a decade and arguably represent the frontier. It is for these use cases that SIMD is non-portable and data infrastructure isn't going away.
exDM69 2mo ago
> They specifies a constant SIMD width so it's non-portable. This is incorrect, you can use vectors wider than native SIMD width and the compiler will break them down to register size of the target cpu. In fact it's sometimes better to used wider than native width, in some applications I see 20% better throughput with f32x16 (512 bits) on an AVX2 CPU (256 bits). It is kinda like loop unrolling it.
- camel-cdr 2mo ago
  Except you can't use this in actual code, because either, as is the case in this example with f32x32, you run out of registers and spill all over the place. Or you aren't using your full vector register or could've gotten better performance by "unrolling" more often for the larger vectors. If you use f32x16 (the avx-512 wisth), SSE now effectively has 4 registers to work with and will spill when doing anything beyond the most simple stuff. The default should imo be relative to the native register width, so you can do 1x, 2x or sometimes 4x the native width, depensing on your register preasure.
  - exDM69 2mo ago
    I can and I do use this is "actual code" and I've got benchmarks to prove that it's got better throughput (for the particular use case, don't extrapolate from there) and the same applies to AVX2 and AVX512: twice the native vector width has ~20% better throughput (ie. using `f32x32` on AVX-512). I pass in the vector width as a generic parameter like this: fn do_simd_stuff<const N: usize>(x: Simd<f32, N>) { x.mul_add(x+x, x*x); } With this I can easily benchmark the same code for any vector width. I can also do some compile time heuristics to choose the vector width based on what's available on the compile target CPU. > you run out of registers and spill all over the place As usual when optimizing SIMD code, you should keep an eye on the generated disassembly and the benchmark results and watch for register pressure and the other usual things. I'm definitely NOT saying that you always get the best perf by using 2x SIMD width, but in this particular case it was so. This is much much easier to do with portable_simd than if you'd write the same with intrinsics, you can change the SIMD width without having to rewrite all your code (e.g. changing from SSE `_mm_add_ps` to AVX `_mm256_add_ps` etc). It's still a partial solution, you still need to drop down to intrinsics for some special instructions every now and then (which is easy), but in my projects this accounts for much less than 1% of the lines of code. Not applicable everywhere of course.

I love how ever example of portable SIMD isn't portable. They specifies a constant SIMD width so it's non-portable. Well, not performance portable, but why are we using SIMD again?

tyho 2mo ago

Go's implementation is vector size independant https://pkg.go.dev/simd@master https://pkg.go.dev/simd@master

IshKebab 2mo ago
Sure but there's no real way to use that in a portable way, at least not a way that maximises performance on every CPU you run it on. That's pretty much impossible at the moment.
- krapht 2mo ago
  Which is why I've never quite understood the appeal of portable SIMD libraries for performance-critical code. If I'm explicitly writing SIMD rather than relying on the auto-vectorizer, it's usually because I want access to the particular capabilities of the target ISA. For many problems, choosing the right instruction or instruction sequence makes a large difference. Portable SIMD abstractions necessarily expose some common semantic layer, but SIMD ISAs don't actually have equivalent capabilities. Instructions like pshufb, for example, enable algorithmic tricks that don't necessarily have an equally efficient analogue on another architecture. If maximum performance matters, I generally want intrinsics and architecture-specific implementations; if portability matters more, I'd rather move further up the abstraction stack and use something designed to target multiple architectures, such as ISPC. There are certainly cases where portable SIMD gets close enough to optimal, but I don't think there's a compiler or abstraction that can express every useful SIMD idiom and lower it equally efficiently across fundamentally different ISAs.
  - pjmlp 2mo ago
    Because usually they achieve a very good middle ground, they are useful for when autovectorization isn't good enough, and it is possible to give a little help to the compiler. There are many ways that performance matters without trying to win a F1 race. Go isn't alone, .NET, Java have similar portable libraries, and C++ is in the process of getting one.
  - kbolino 2mo ago
    Go doesn't have auto-vectorization in the first place, so its portable simd library is at least partly there to fill the gap.

MomsAVoxell 2mo ago

Why should it be portable? Honest question. SIMD seems to me, to be very platform specific. Maybe there are times one SIMD unit is not anothers' SIMD unit?

camel-cdr 2mo ago
The create is called portable_simd. There is no reason a portable_simd relu_dot implemention should need to specify the SIMD width. But the design and documentation of portable_simd makes the fixed size syntactically easy/the default and the width agnostic code harder.
- dwattttt 2mo ago
  > There is no reason a portable_simd relu_dot implemention should need to specify the SIMD width. What should it choose then? I have a Zen 3 processor, and benchmarking some simd I did recently says 32 byte or 64 byte chunks was fastest. But I'm sure I'd get a different result on a different Zen, and different again on Intel's. How would the library decide what SIMD width I should use?
  - Groxx 2mo ago
    It'd need some kind of compile-time hardware-feature-detection, yea? That seems probably feasible since proc macros can do essentially anything they like (worryingly).
    - derefr 2mo ago
      Only if the end-user is the one compiling the software, on the same very system they'll be running it on. Which is true of GPU shader kernels, due to how GPU drivers work; but isn't generally true of CPU object code (unless you're on Gentoo.) What you'd actually want is a matrix of variant implementations burned into the binary, with runtime (or process-boot-time) hardware detection that swaps symbols out to point to the correct variant.

zamadatix 2mo ago

It should really be read/advertised as "portabler SIMD". It beats hoping the compiler autovectorizes everything well forever or writing architecture specific code manually again but is going to compromise on average performance vs platform specific SIMD.

pjmlp 2mo ago
.NET and Java have three levels of SIMD support, Go's ongoing efforts, and does the upcoming C++ standard. Autovectorization, depending on compiler's cleverness, really portable SIMD operations, and then the CPU specific SIMD ones. So this should be perfectly doable in crate that advertises as portable, while leaving the non portable stuff to another crate.

jandrewrogers 2mo ago

The capabilities of various SIMD ISAs don't have enough intersection to be portable outside of relatively trivial cases. Many of the somewhat unique capabilities are load-bearing, so you want to use them on architectures that support them. Taken in whole, someone who cares about performance would be using different data structures and algorithms depending on the specific SIMD architecture and that is nearly impossible to abstract in a library. Too many important but complex details are idiosyncratic to the implementation. Another way of looking at it is that our programming environments are not sufficiently powerful and expressive to create the necessary abstractions to make SIMD truly portable.

simonask 2mo ago
> The capabilities of various SIMD ISAs don't have enough intersection to be portable outside of relatively trivial cases. I would argue that the "trivial" cases (those relating to linear algebra in 3 dimensions) are also 95% of what people want SIMD for. If the API can achieve cross-platform and performant vector arithmetic, dot product, and matrix multiplication in the normal ways, that already covers a lot of what people actually need.
- jandrewrogers 2mo ago
  Most use cases for SIMD are non-arithmetic in nature and don't assume tidy arrays of homogeneous types. I also use it for some computational geometry but that is the least interesting use case. SIMD is widely used throughout data infrastructure e.g. parsing data, complex constraint processing, parallel manipulation of heterogeneous data types, compression, etc. I even have an I/O scheduler written in AVX-512 that is many times faster than the scalar equivalent. The ability of SIMD to do complex manipulation of ordinary data structures several times faster than scalar code is under-rated. While linear algebra is the current thing, database engines have been using SIMD heavily for over a decade and arguably represent the frontier. It is for these use cases that SIMD is non-portable and data infrastructure isn't going away.

exDM69 2mo ago

> They specifies a constant SIMD width so it's non-portable. This is incorrect, you can use vectors wider than native SIMD width and the compiler will break them down to register size of the target cpu. In fact it's sometimes better to used wider than native width, in some applications I see 20% better throughput with f32x16 (512 bits) on an AVX2 CPU (256 bits). It is kinda like loop unrolling it.

camel-cdr 2mo ago
Except you can't use this in actual code, because either, as is the case in this example with f32x32, you run out of registers and spill all over the place. Or you aren't using your full vector register or could've gotten better performance by "unrolling" more often for the larger vectors. If you use f32x16 (the avx-512 wisth), SSE now effectively has 4 registers to work with and will spill when doing anything beyond the most simple stuff. The default should imo be relative to the native register width, so you can do 1x, 2x or sometimes 4x the native width, depensing on your register preasure.
- exDM69 2mo ago
  I can and I do use this is "actual code" and I've got benchmarks to prove that it's got better throughput (for the particular use case, don't extrapolate from there) and the same applies to AVX2 and AVX512: twice the native vector width has ~20% better throughput (ie. using `f32x32` on AVX-512). I pass in the vector width as a generic parameter like this: fn do_simd_stuff<const N: usize>(x: Simd<f32, N>) { x.mul_add(x+x, x*x); } With this I can easily benchmark the same code for any vector width. I can also do some compile time heuristics to choose the vector width based on what's available on the compile target CPU. > you run out of registers and spill all over the place As usual when optimizing SIMD code, you should keep an eye on the generated disassembly and the benchmark results and watch for register pressure and the other usual things. I'm definitely NOT saying that you always get the best perf by using 2x SIMD width, but in this particular case it was so. This is much much easier to do with portable_simd than if you'd write the same with intrinsics, you can change the SIMD width without having to rewrite all your code (e.g. changing from SSE `_mm_add_ps` to AVX `_mm256_add_ps` etc). It's still a partial solution, you still need to drop down to intrinsics for some special instructions every now and then (which is easy), but in my projects this accounts for much less than 1% of the lines of code. Not applicable everywhere of course.