7 ms·

SIMD Everywhere Optimization from ARM Neon to RISC-V Vector Extensions

107 points by camel-cdr 3y ago

KingLancelot 3y ago
We need to do better than ISA specific intrinsics. There should be a simd.h header in the C standard library that contains typedefs for vector types, and various functions to operate on them as well as Operators for them. Like my _Operator <symbol> <function name>; proposal, which requires no mangling.
- deleted 3y ago
  [deleted]
camel-cdr 3y ago
This doesn't seem to be upstreamed yet. I hope they have real hardware performance numbers for the rv summit talk.
- biocrusoe 3y ago
  SIMDe maintainer here, I would welcome a PR; yes!
almatabata 3y ago
Very neat, i hope this will get easier to do in the future once languages start including these SIMD semantics in the language itself like rust tries to do: https://doc.rust-lang.org/std/simd/struct.Simd.html https://doc.rust-lang.org/std/simd/struct.Simd.html Libraries implemented in languages without these semantics will greatly benefit from this.
- geertj 3y ago
  And C++: https://en.cppreference.com/w/cpp/experimental/simd https://en.cppreference.com/w/cpp/experimental/simd This proposal has been around for a while a but it recently got some new momentum and seems to be on track for c++26. Gcc ships a version today for those wanting to try it.
  - janwas 3y ago
    I'm curious what is the new momentum? It's difficult to understand the value proposition of std::experimental::simd. Standardization has taken many years, lost the "load/store" function naming which just about everyone everywhere(?) is using, and only resulted in ~30 ops [1] that are also mostly achievable with compiler builtins or perhaps even autovectorization. Four years ago, we stepped in with Highway to fill the gap; it now has about 250 ops [2] and an active community. I have not wanted to step on any toes, but wonder whether the ship has by now sailed on this? [1]: https://en.cppreference.com/w/cpp/header/experimental/simd https://en.cppreference.com/w/cpp/header/experimental/simd [2]: https://github.com/google/highway/blob/master/g3doc/quick_reference.md#operations https://github.com/google/highway/blob/master/g3doc/quick_re...
    - geertj 3y ago
      Late comment, just saw this. Not sure where the momentum is coming from. The proposal has a new maintainer now, new drafts have been released, and there's a stated goal to get this included in C++26. For me, the appeal for experimental/simd would be it being part of the stdlib. Have you considered proposing Highway for stdlib inclusion? From your description it appears to be a more complete implementation. I realize it's a long and arduous process, but I think it's the only way to make something truly the default implementation. (And thanks for the link. I was not aware that Highway existed.)
      - janwas 3y ago
        Interesting, thanks for sharing :) At the time we open-sourced Highway, the standardization process had already started and there were some discussions. I'm curious why stdlib is the only path you see to default? Compare the activity level of https://github.com/VcDevel/std-simd https://github.com/VcDevel/std-simd vs https://github.com/google/highway https://github.com/google/highway. As to open-source usage, after years of std::experimental, I see <200 search hits [1], vs >400 for Highway [2], even after excluding several library users. But that aside, I'm not convinced standardization is the best path for a SIMD library. We and external users extend Highway on a weekly basis as new use cases arise. What if we deferred those changes to 3-monthly meetings, or had to wait for one meeting per WD, CD, (FCD), DIS, (FDIS) stage before it's standardized? Standardization seems more useful for rarely-changing things. 1: https://sourcegraph.com/search?q=context:global+std::experimental::simd+-path:clang-tidy+count:10000+-path:sycl+-path:highway/+-repo:highway+-path:man/+-path:md%24+-path:rst%24+-path:sh%24+-path:txt%24+-path:testsuite/+-path:tests/+-path:svn-base%24&patternType=standard&sm=1&groupBy=repo https://sourcegraph.com/search?q=context:global+std::experim... 2: https://sourcegraph.com/search?q=context:global+HWY_NAMESPACE+-path:clang-tidy+count:10000+-path:sycl+-path:highway/+-repo:highway+-path:man/+-path:md%24+-path:rst%24+-path:sh%24+-path:txt%24+-path:testsuite/+-path:tests/+-path:svn-base%24+-path:libjxl+-path:jpeg-xl+-path:jxl+-path:jpegli&patternType=standard&sm=1&groupBy=repo https://sourcegraph.com/search?q=context:global+HWY_NAMESPAC...
- dzaima 3y ago
  A problem is that most such things (the rust thing, C++'s experimental/simd, Zig's SIMD types) have the vector size as a compile-time property, while ARM's SVE and RISC-V's RVV are designed such that it's possible to write portable code that can work for a range of implementation widths. Thus such a fixed-width SIMD library would be forced to target the minimum (128-bit) even if the hardware supports 256-bit, 512-bit, or more. (SVE supports up to 1024-bit, RVV - up to 65536-bit) There is Highway (https://github.com/google/highway https://github.com/google/highway) however, that does support dynamically-sized SIMD.
  - almatabata 3y ago
    If you compile the code by specifying the target as native you could get around that limitation no?
    - cozzyd 3y ago
      yes, but then if distributing binaries, you need a different binary for each SIMD width.
      - vkazanov 3y ago
        You can actually autogenerate all reasonable variants of the code if necessary, there aren't that many architectures these days. Simd imstructions are usually very local, this shouldn't blow up the binary. The point is to not have to write repetitive source code many times.
      - Pet_Ant 3y ago
        What is the real cost to just have those few methods be compiled in and then a branch? You don't need to ship a separate binary for each target, you can have dead code in it. I mean fat binaries take this idea to the extreme to support multiple architectures. https://en.wikipedia.org/wiki/Fat_binary https://en.wikipedia.org/wiki/Fat_binary
        dzaima 3y ago
adgjlsfhk1 3y ago
and hug of death
biocrusoe 3y ago
Archived copy: https://web.archive.org/web/20230929161438/https://arxiv.org/abs/2309.16509 https://web.archive.org/web/20230929161438/https://arxiv.org... Direct link to PDF: https://arxiv.org/pdf/2309.16509.pdf https://arxiv.org/pdf/2309.16509.pdf
atdt 3y ago
Highway (https://github.com/google/highway https://github.com/google/highway), Google's SIMD library, lets you write length-agnostic SIMD code. It has excellent support for a wide range of targets, including both RISC-V and Arm vector extensions.
- synergy20 3y ago
  second this
mgaunard 3y ago
There are so many SIMD libraries nowadays. I myself implemented one in the SSE4/Altivec days (later extended to AVX, AVX512 and NEON). There were only a few options then, but now everyone seems to be doing it.
snvzz 3y ago
RISC-V is rapidly building the strongest ecosystem.
Danidada 3y ago
Neat project! However, I'm pretty sure OpenCV has their "universal intrinsics" and RISC-V with scalable vector registers is supported in the latest OpenCV version Universal intrinsics (docs not updated): https://docs.opencv.org/4.x/d6/dd1/tutorial_univ_intrin.html https://docs.opencv.org/4.x/d6/dd1/tutorial_univ_intrin.html Scalable RVV support: https://github.com/opencv/opencv/pull/22179 https://github.com/opencv/opencv/pull/22179
kierank 3y ago
The paper suggests FFmpeg uses intrinsics which is not correct. There have been many SIMD abstraction layers created in the past but none of them will beat the raw speed of handwritten assembly. Try and implement something like vpternlogd in one of these abstraction layers.
- dist1ll 3y ago
  The main abstraction of intrinsics is register allocation, right? Is there anything else that can be gained by handwritten asm?
  - camel-cdr 3y ago
    For rvv specifically there are a few things that aren't possible using the intrinsics abstraction. E.g. in asm you can run the same instruction sequence with different vtype (element width and LMUL).
    - janwas 3y ago
      We are able to do the same with Highway's RVV :)
      - dzaima 3y ago
        I believe what camel-cdr is saying is being able to run the same code without duplication (say, a loop) which has no vsetvl-s inside, by conditionally choosing either an initial "vsetvli x0,x0,e32,m1" or "vsetvl x0,x0,e16,m1" or "vsetvl x0,x0,e32,m2" etc, which is just unachievable with the intrinsics as they hard-code vtype in each intrinsic. It's an extremely fun idea (primarily just for code size though), but thankfully (?) its usability is restricted by load/store instrs hard-coding the element type, so the main use of this would end up for switching LMUL, which has very limited usefulness. What Highway can support is generating multiple loops of different vtype from the same code, which effectively achieves the same thing, at the cost of machine code duplication.
        3y ago
bch 3y ago
This might be asking a lot, but what’s the Little Schemer for vector processing? A fun on-ramp.
- anonymoushn 3y ago
  https://en.algorithmica.org/hpc/ https://en.algorithmica.org/hpc/ and http://0x80.pl/ http://0x80.pl/ have some stuff about this, but the latter can be dense. I've had fun getting my hands dirty with some problems at https://highload.fun/ https://highload.fun/ but there's not much direction unless you go to the telegram chat and ask people questions.

107 points by camel-cdr 3y ago

We need to do better than ISA specific intrinsics. There should be a simd.h header in the C standard library that contains typedefs for vector types, and various functions to operate on them as well as Operators for them. Like my _Operator <symbol> <function name>; proposal, which requires no mangling.

deleted 3y ago
[deleted]

camel-cdr 3y ago

This doesn't seem to be upstreamed yet. I hope they have real hardware performance numbers for the rv summit talk.

biocrusoe 3y ago
SIMDe maintainer here, I would welcome a PR; yes!

almatabata 3y ago

Very neat, i hope this will get easier to do in the future once languages start including these SIMD semantics in the language itself like rust tries to do: https://doc.rust-lang.org/std/simd/struct.Simd.html https://doc.rust-lang.org/std/simd/struct.Simd.html Libraries implemented in languages without these semantics will greatly benefit from this.

geertj 3y ago
And C++: https://en.cppreference.com/w/cpp/experimental/simd https://en.cppreference.com/w/cpp/experimental/simd This proposal has been around for a while a but it recently got some new momentum and seems to be on track for c++26. Gcc ships a version today for those wanting to try it.
- janwas 3y ago
  I'm curious what is the new momentum? It's difficult to understand the value proposition of std::experimental::simd. Standardization has taken many years, lost the "load/store" function naming which just about everyone everywhere(?) is using, and only resulted in ~30 ops [1] that are also mostly achievable with compiler builtins or perhaps even autovectorization. Four years ago, we stepped in with Highway to fill the gap; it now has about 250 ops [2] and an active community. I have not wanted to step on any toes, but wonder whether the ship has by now sailed on this? [1]: https://en.cppreference.com/w/cpp/header/experimental/simd https://en.cppreference.com/w/cpp/header/experimental/simd [2]: https://github.com/google/highway/blob/master/g3doc/quick_reference.md#operations https://github.com/google/highway/blob/master/g3doc/quick_re...
  - geertj 3y ago
    Late comment, just saw this. Not sure where the momentum is coming from. The proposal has a new maintainer now, new drafts have been released, and there's a stated goal to get this included in C++26. For me, the appeal for experimental/simd would be it being part of the stdlib. Have you considered proposing Highway for stdlib inclusion? From your description it appears to be a more complete implementation. I realize it's a long and arduous process, but I think it's the only way to make something truly the default implementation. (And thanks for the link. I was not aware that Highway existed.)
    - janwas 3y ago
      Interesting, thanks for sharing :) At the time we open-sourced Highway, the standardization process had already started and there were some discussions. I'm curious why stdlib is the only path you see to default? Compare the activity level of https://github.com/VcDevel/std-simd https://github.com/VcDevel/std-simd vs https://github.com/google/highway https://github.com/google/highway. As to open-source usage, after years of std::experimental, I see <200 search hits [1], vs >400 for Highway [2], even after excluding several library users. But that aside, I'm not convinced standardization is the best path for a SIMD library. We and external users extend Highway on a weekly basis as new use cases arise. What if we deferred those changes to 3-monthly meetings, or had to wait for one meeting per WD, CD, (FCD), DIS, (FDIS) stage before it's standardized? Standardization seems more useful for rarely-changing things. 1: https://sourcegraph.com/search?q=context:global+std::experimental::simd+-path:clang-tidy+count:10000+-path:sycl+-path:highway/+-repo:highway+-path:man/+-path:md%24+-path:rst%24+-path:sh%24+-path:txt%24+-path:testsuite/+-path:tests/+-path:svn-base%24&patternType=standard&sm=1&groupBy=repo https://sourcegraph.com/search?q=context:global+std::experim... 2: https://sourcegraph.com/search?q=context:global+HWY_NAMESPACE+-path:clang-tidy+count:10000+-path:sycl+-path:highway/+-repo:highway+-path:man/+-path:md%24+-path:rst%24+-path:sh%24+-path:txt%24+-path:testsuite/+-path:tests/+-path:svn-base%24+-path:libjxl+-path:jpeg-xl+-path:jxl+-path:jpegli&patternType=standard&sm=1&groupBy=repo https://sourcegraph.com/search?q=context:global+HWY_NAMESPAC...
dzaima 3y ago
A problem is that most such things (the rust thing, C++'s experimental/simd, Zig's SIMD types) have the vector size as a compile-time property, while ARM's SVE and RISC-V's RVV are designed such that it's possible to write portable code that can work for a range of implementation widths. Thus such a fixed-width SIMD library would be forced to target the minimum (128-bit) even if the hardware supports 256-bit, 512-bit, or more. (SVE supports up to 1024-bit, RVV - up to 65536-bit) There is Highway (https://github.com/google/highway https://github.com/google/highway) however, that does support dynamically-sized SIMD.
- almatabata 3y ago
  If you compile the code by specifying the target as native you could get around that limitation no?
  - cozzyd 3y ago
    yes, but then if distributing binaries, you need a different binary for each SIMD width.
    - vkazanov 3y ago
      You can actually autogenerate all reasonable variants of the code if necessary, there aren't that many architectures these days. Simd imstructions are usually very local, this shouldn't blow up the binary. The point is to not have to write repetitive source code many times.
    - Pet_Ant 3y ago
      What is the real cost to just have those few methods be compiled in and then a branch? You don't need to ship a separate binary for each target, you can have dead code in it. I mean fat binaries take this idea to the extreme to support multiple architectures. https://en.wikipedia.org/wiki/Fat_binary https://en.wikipedia.org/wiki/Fat_binary
      - dzaima 3y ago

adgjlsfhk1 3y ago

and hug of death

biocrusoe 3y ago

Archived copy: https://web.archive.org/web/20230929161438/https://arxiv.org/abs/2309.16509 https://web.archive.org/web/20230929161438/https://arxiv.org... Direct link to PDF: https://arxiv.org/pdf/2309.16509.pdf https://arxiv.org/pdf/2309.16509.pdf

atdt 3y ago

Highway (https://github.com/google/highway https://github.com/google/highway), Google's SIMD library, lets you write length-agnostic SIMD code. It has excellent support for a wide range of targets, including both RISC-V and Arm vector extensions.

synergy20 3y ago
second this

mgaunard 3y ago

There are so many SIMD libraries nowadays. I myself implemented one in the SSE4/Altivec days (later extended to AVX, AVX512 and NEON). There were only a few options then, but now everyone seems to be doing it.

snvzz 3y ago

RISC-V is rapidly building the strongest ecosystem.

Danidada 3y ago

Neat project! However, I'm pretty sure OpenCV has their "universal intrinsics" and RISC-V with scalable vector registers is supported in the latest OpenCV version Universal intrinsics (docs not updated): https://docs.opencv.org/4.x/d6/dd1/tutorial_univ_intrin.html https://docs.opencv.org/4.x/d6/dd1/tutorial_univ_intrin.html Scalable RVV support: https://github.com/opencv/opencv/pull/22179 https://github.com/opencv/opencv/pull/22179

kierank 3y ago

The paper suggests FFmpeg uses intrinsics which is not correct. There have been many SIMD abstraction layers created in the past but none of them will beat the raw speed of handwritten assembly. Try and implement something like vpternlogd in one of these abstraction layers.

dist1ll 3y ago
The main abstraction of intrinsics is register allocation, right? Is there anything else that can be gained by handwritten asm?
- camel-cdr 3y ago
  For rvv specifically there are a few things that aren't possible using the intrinsics abstraction. E.g. in asm you can run the same instruction sequence with different vtype (element width and LMUL).
  - janwas 3y ago
    We are able to do the same with Highway's RVV :)
    - dzaima 3y ago
      I believe what camel-cdr is saying is being able to run the same code without duplication (say, a loop) which has no vsetvl-s inside, by conditionally choosing either an initial "vsetvli x0,x0,e32,m1" or "vsetvl x0,x0,e16,m1" or "vsetvl x0,x0,e32,m2" etc, which is just unachievable with the intrinsics as they hard-code vtype in each intrinsic. It's an extremely fun idea (primarily just for code size though), but thankfully (?) its usability is restricted by load/store instrs hard-coding the element type, so the main use of this would end up for switching LMUL, which has very limited usefulness. What Highway can support is generating multiple loops of different vtype from the same code, which effectively achieves the same thing, at the cost of machine code duplication.
      - 3y ago

bch 3y ago

This might be asking a lot, but what’s the Little Schemer for vector processing? A fun on-ramp.

anonymoushn 3y ago
https://en.algorithmica.org/hpc/ https://en.algorithmica.org/hpc/ and http://0x80.pl/ http://0x80.pl/ have some stuff about this, but the latter can be dense. I've had fun getting my hands dirty with some problems at https://highload.fun/ https://highload.fun/ but there's not much direction unless you go to the telegram chat and ask people questions.