3 ms·

If you compile the code by specifying the target as native you could get around that limitation no?

by almatabata 3y ago

cozzyd 3y ago
yes, but then if distributing binaries, you need a different binary for each SIMD width.
- vkazanov 3y ago
  You can actually autogenerate all reasonable variants of the code if necessary, there aren't that many architectures these days. Simd imstructions are usually very local, this shouldn't blow up the binary. The point is to not have to write repetitive source code many times.
- Pet_Ant 3y ago
  What is the real cost to just have those few methods be compiled in and then a branch? You don't need to ship a separate binary for each target, you can have dead code in it. I mean fat binaries take this idea to the extreme to support multiple architectures. https://en.wikipedia.org/wiki/Fat_binary https://en.wikipedia.org/wiki/Fat_binary
  - dzaima 3y ago
    That's 5 copies for SVE (had an error in first message - SVE allows up to 2048-bit vectors, not 1024), and 10 copies for RVV if you wanted to target all widths (though you'd probably be fine for a decade or a couple by targeting just 128 & 256-bit, and maybe 512-bit). Plus one more for a scalar fallback. And yes, it's not particularly large of a cost, other than it being an extremely pointless waste of space given that it is possible to have just one variant that covers them all. Though, it would become significantly more problematic if you wanted to target different extension groups too (which you would quite likely want to some extent) as those'd multiply with all the length targets - SVE vs SVE2 vs more future extensions, and on RVV there's just a lot (Zvfh & Zvfhmin for FP16, Zvbb for extra bitmanip stuff, many more here[1]; and potentially at some point there could be an extension that uses a wider encoding scheme to inline vsetvl fields & allow masking by registers other than v0, which could benefit everything) [1]: https://github.com/riscv/riscv-crypto/blob/c8ddeb7e64a3444dda0438316af1238aeed72041/doc/vector/riscv-crypto-spec-vector.adoc https://github.com/riscv/riscv-crypto/blob/c8ddeb7e64a3444dd...
    - anonymoushn 3y ago
      Prior to SVE2 string processing algorithms already had to ship a bunch of copies for SVE because it lacked a "shuffle the 16-byte lanes of A according to the shuffle in the 16-byte lanes of B" instruction :(
  - elabajaba 3y ago
    Not being able to inline and having to branch on every call to a simd function can sometimes make it slower than the basic scalar version.
    - Pet_Ant 3y ago
      Just a thought, but would it be possible to hot patch at the time of loading the binary? I realise it might require updates to the binary format, but it might be very well justified.
      - anonymoushn 3y ago
        This sounds similar to shipping all these routines in dynamic libraries and loading the right one at runtime. I'm not sure if the memory cost of having all these duplicate simd kernels is a big deal though.
    - dzaima 3y ago
      You sould branch at the level where inlining doesn't make sense, which would usually be some function wrapping the big loop, which should be rather free. Which is the same situation as on x86-64 if you want to target pre-AVX2/post-AVX2/AVX-512.
- almatabata 3y ago
  Ah makes sense if you have complete control over your hardware it could make sense but with open source projects and businesses with a wide customer base it might not make sense. Compiled languages like Rust, C++ and Zig cannot detect the hardware because they have no runtime right? Could a language like Go add the simd semantics and detect the support vector size?
  - dzaima 3y ago
    The problem isn't detecting the width (that's trivially possible at runtime with a single instruction, though both SVE and RVV have a way to write loops such that you don't even need to). The problem is that a "Simd<i32, 4>" will always have 4 elements, but you'd need a "Simd<i32, whatever the hardware has>" type, which has significant impact on what is possible to do with such a type.
    - almatabata 3y ago
      Ah thank you for clarifying so you would have to create an abstraction layer on top of the current simd implementation like for example simd_vector(type, size). That abstraction would have to dynamically detect the hardware and dispatch it to the hardware like the project you shared (https://github.com/google/highway https://github.com/google/highway). So technically it sounds feasible but all of the languages like Zig, C++ and Rust picked a simpler approach. Is it simply a first step to a more abstract approach?
      - dzaima 3y ago
        Not really - you don't need to dispatch anything, the idea is that the same code (and thus the same assembly/machine code) can operate on different sizes by itself. e.g. with RVV "vsetvli x0,x0,e32,m1,ta,ma; vadd.vv v0, v1, v2" on a CPU with 128-bit vectors will do 4 element additions, but on a CPU with 1024-bit vectors it'll do 32 additions. And some things you just can't really "generalize" to scalable vectors. e.g. you can store Simd<i32,4> in a struct or global variables, or initialize with, say, [3,2,1,0], but none of those things are possible with scalable vectors (globals/struct fields need a known size, and initializing with a hard-coded list of elements doesn't make much sense if you don't even know how many elements you'll need).

If you compile the code by specifying the target as native you could get around that limitation no?

cozzyd 3y ago

yes, but then if distributing binaries, you need a different binary for each SIMD width.

vkazanov 3y ago
You can actually autogenerate all reasonable variants of the code if necessary, there aren't that many architectures these days. Simd imstructions are usually very local, this shouldn't blow up the binary. The point is to not have to write repetitive source code many times.
Pet_Ant 3y ago
What is the real cost to just have those few methods be compiled in and then a branch? You don't need to ship a separate binary for each target, you can have dead code in it. I mean fat binaries take this idea to the extreme to support multiple architectures. https://en.wikipedia.org/wiki/Fat_binary https://en.wikipedia.org/wiki/Fat_binary
- dzaima 3y ago
  That's 5 copies for SVE (had an error in first message - SVE allows up to 2048-bit vectors, not 1024), and 10 copies for RVV if you wanted to target all widths (though you'd probably be fine for a decade or a couple by targeting just 128 & 256-bit, and maybe 512-bit). Plus one more for a scalar fallback. And yes, it's not particularly large of a cost, other than it being an extremely pointless waste of space given that it is possible to have just one variant that covers them all. Though, it would become significantly more problematic if you wanted to target different extension groups too (which you would quite likely want to some extent) as those'd multiply with all the length targets - SVE vs SVE2 vs more future extensions, and on RVV there's just a lot (Zvfh & Zvfhmin for FP16, Zvbb for extra bitmanip stuff, many more here[1]; and potentially at some point there could be an extension that uses a wider encoding scheme to inline vsetvl fields & allow masking by registers other than v0, which could benefit everything) [1]: https://github.com/riscv/riscv-crypto/blob/c8ddeb7e64a3444dda0438316af1238aeed72041/doc/vector/riscv-crypto-spec-vector.adoc https://github.com/riscv/riscv-crypto/blob/c8ddeb7e64a3444dd...
  - anonymoushn 3y ago
    Prior to SVE2 string processing algorithms already had to ship a bunch of copies for SVE because it lacked a "shuffle the 16-byte lanes of A according to the shuffle in the 16-byte lanes of B" instruction :(
- elabajaba 3y ago
  Not being able to inline and having to branch on every call to a simd function can sometimes make it slower than the basic scalar version.
  - Pet_Ant 3y ago
    Just a thought, but would it be possible to hot patch at the time of loading the binary? I realise it might require updates to the binary format, but it might be very well justified.
    - anonymoushn 3y ago
      This sounds similar to shipping all these routines in dynamic libraries and loading the right one at runtime. I'm not sure if the memory cost of having all these duplicate simd kernels is a big deal though.
  - dzaima 3y ago
    You sould branch at the level where inlining doesn't make sense, which would usually be some function wrapping the big loop, which should be rather free. Which is the same situation as on x86-64 if you want to target pre-AVX2/post-AVX2/AVX-512.
almatabata 3y ago
Ah makes sense if you have complete control over your hardware it could make sense but with open source projects and businesses with a wide customer base it might not make sense. Compiled languages like Rust, C++ and Zig cannot detect the hardware because they have no runtime right? Could a language like Go add the simd semantics and detect the support vector size?
- dzaima 3y ago
  The problem isn't detecting the width (that's trivially possible at runtime with a single instruction, though both SVE and RVV have a way to write loops such that you don't even need to). The problem is that a "Simd<i32, 4>" will always have 4 elements, but you'd need a "Simd<i32, whatever the hardware has>" type, which has significant impact on what is possible to do with such a type.
  - almatabata 3y ago
    Ah thank you for clarifying so you would have to create an abstraction layer on top of the current simd implementation like for example simd_vector(type, size). That abstraction would have to dynamically detect the hardware and dispatch it to the hardware like the project you shared (https://github.com/google/highway https://github.com/google/highway). So technically it sounds feasible but all of the languages like Zig, C++ and Rust picked a simpler approach. Is it simply a first step to a more abstract approach?
    - dzaima 3y ago
      Not really - you don't need to dispatch anything, the idea is that the same code (and thus the same assembly/machine code) can operate on different sizes by itself. e.g. with RVV "vsetvli x0,x0,e32,m1,ta,ma; vadd.vv v0, v1, v2" on a CPU with 128-bit vectors will do 4 element additions, but on a CPU with 1024-bit vectors it'll do 32 additions. And some things you just can't really "generalize" to scalable vectors. e.g. you can store Simd<i32,4> in a struct or global variables, or initialize with, say, [3,2,1,0], but none of those things are possible with scalable vectors (globals/struct fields need a known size, and initializing with a hard-coded list of elements doesn't make much sense if you don't even know how many elements you'll need).