10 ms·

I sometimes write code portable across SSE4 and NEON, and I'm not sure this is going to work fast enough for that. There're important unique features. SSE has

I sometimes write code portable across SSE4 and NEON, and I'm not sure this is going to work fast enough for that. There're important unique features. SSE has shuffles, pack/unpack, movemask, 64-bit doubles, testzero, float rounds, blends, integer averages, float square roots and dot product. NEON has interleaved RAM load/stores, vectors operators with scalar other argument, byte swap, rotate, bit scan and population count, and versions of all instructions processing 8-byte long vectors. That's enough differences that I have to adjust both algorithms and data structures to be portable between them. I'm not convinced it's possible to do automatically.

dragontamer 6y ago
Shuffle, pack/unpack, movemask, blends (SSE/AVX) and interleaved load/stores, byte swap (NEON) are "just" data-movement instructions. All of them can be implemented (with obvious slowdowns) with a conditional write to memory, then a conditional read from memory. Yeah, its inefficient to do it like this, but this "write then read" pattern really gives us an idea of what's really going on between the registers in a pack/pshufb/whatever instruction. On AMD GPUs, there's a fully arbitrary crossbar between SIMD-lanes allowing for arbitrary movement. The two instructions are just "permute" and "b-permute" (backwards permute), roughly correlating to gather and scatter respectively. On NVidia GPUs, perm and bperm are both implemented in PTX, but instead read/write to L1 or __shared__ memory. NVidia GPUs likely have a crossbar to L1 memory to make this instruction very fast. --------- The solution is to implement perm and bperm on AVX. Its already half-implemented: pshufb is equivalent to GPU-permute. CPUs are just missing the backwards permute. I'm pretty confident that pack/unpack, blends, interleaved load/stores, and more could all be implemented as pshufb and a hypothetical "backwards pshufb". Version 1.0 could be an NVidia-like "write to L1 cache" sort of implementation too, if full crossbars are too expensive at the hardware layer. ----------- So the question is: how should we write code today? CPUs of today do not implement this feature, but CPUs of the future might. I think specifying the memory-moves explicitly, and then working on a "pshufb compiler/optimizer" of sorts is what we need.
- Const-me 6y ago
  > the question is: how should we write code today? The only thing that matters today is how fast the code works on today’s hardware. To illustrate, see how I have emulated neon’s vst3q_s16 with SSE: https://github.com/Const-me/DtsDecoder/blob/master/Utils/store3-sse.hpp https://github.com/Const-me/DtsDecoder/blob/master/Utils/sto... However, not all of them can be emulated in a fast enough manner. > specifying the memory-moves explicitly, and then working on a "pshufb compiler/optimizer" of sorts is what we need. I’ll consider that approach when I’ll see that compiler/optimizer working, and producing output comparable to manually-written code. Until then I think that’s a “sufficiently smart compiler” class of problems. These are rarely ever solved, IMO. For example, we don’t have generally useful auto-vectorizers in C compilers despite two decades of R&D, even the best of them (clang, intel) are still very limited.
- floatboth 6y ago
  > how should we write code today? Write in a GPU-style "compute shader" language and let the compiler pick the fastest ways of doing things on each ISA? https://github.com/ispc/ispc https://github.com/ispc/ispc
  - dragontamer 6y ago
    Well, such a feature doesn't exist in ispc. I guess I'm proposing a hypothetical feature that doesn't exist yet in any compiler. But it'd be nice if it existed... For it to exist, we need a combination of new assembly instructions as well as a smart enough compiler.
gioele 6y ago
> I'm not sure this is going to work fast enough for that. From reading the SIMD Everywhere description, it seems to me that SIMDe is a way to allow code that targets only a platform to work on other platforms as well. As a nice byproduct, you get _some_ speed up if architecture targeted by the code is similar to the architecture that will run the code. Portability is the main focus, not speed. Obviously, once you have a good emulation of an architecture the first question is going to be: can I make it faster?
- mtgx 6y ago
  All other chip architectures should adopt Arm's SVE2 or something similar. https://community.arm.com/developer/ip-products/processors/b/processors-ip-blog/posts/new-technologies-for-the-arm-a-profile-architecture https://community.arm.com/developer/ip-products/processors/b...
  - fao_ 6y ago
    Hi there! It looks like you've been shadowbanned, I vouched a few of your comments because they didn't seem terrible, but I didn't scroll very far. It might be worth sending a mail to dang to ask him to take a look at your account or something!
    - chmod775 6y ago
      That account has 33k karma and is 8 years old. I just can't imagine a set of circumstances where a shadowban of all things could be justified if they transgressed in some way. It's probably a mistake?
      - fao_ 6y ago
        or they said something ridiculously heinous
        chmod775 6y ago
        In that case one should point that out to them and warn them. Shadowbans are quite a heinous punishment in themselves, and better used to prevent spammers etc. from creating new accounts. But people who make thousands upon thousands of comments are bound to make an emotional comment or an error of judgment at some point.
gameswithgo 6y ago
>I'm not convinced it's possible to do automatically. It isn't always, but sometimes it is.
Rebelgecko 6y ago
Are compilers smart enough to interleave load/stores? I've noticed when using NEON intrinsics that GCC occasionally did clever things (and of course sometimes I got huge speed improvements by making small changes, like adding a PLD a few instructions before a hunk of memory is used)
nemequ1729 6y ago
(Lead developer of SIMDe here.) It sounds like you actually know what you're doing, so in this case you're probably right, at least if all you do is compile your x86 code with SIMDe. That said, SIMDe also provides support for other architectures, notably WASM SIMD 128 and AltiVec/VSX, as well as portable implementations which work everywhere, including on CPUs I'd never heard of until people told me SIMDe was working well on them (I'm thinking of Kalray, which supports vectors but doesn't have an API and instead relies on compiler auto-vectorization support). One use case for SIMDe which may be interesting for you is that you can freely mix calls to different APIs. Say, for example, that you already have a bunch of x86 code written and want a NEON port. You can add SIMDe and you get a NEON port basically for free, then you can start adding some ifdefs to add optimizations for NEON without having to rewrite the whole thing. SIMDe doesn't in any way prevent you from optimizing your NEON (or whatever) port. The way I tend to look at it is that SIMDe never makes your code slower, only more portable.
- Const-me 6y ago
  I might give it a try next time I’ll need to do something for AMD64 + NEON. Not going to rewrite code I have already written, debugged and shipped. Also, about this > we have an extensive test suite to verify our implementations Don’t forget about MXCSR register in that suite, esp. the rounding bits of that. I avoid changing it as much as possible ‘coz the state is preserved across context switches and causes funny things in OpenMP and other thread pools, but not all people are aware of that. Also, there’s non-trivial amount of code written for SSE < 4.1 (the 4.1 introduced proper rounding instruction, roundps) where you sometimes forced to mess with MXCSR rounding bits because the alternatives are much slower.
  - nemequ1729 6y ago
    > Not going to rewrite code I have already written, debugged and shipped. I wouldn't, either. At least unless you want a WASM/AltiVec/etc. version. But if you already have good implementations SIMDe probably won't help. OTOH, if all you have is a x86 implementation and a portable fallback, the SIMDe version of the x86 implementation will probably be faster than your portable version. That's what happened with MMseqs2 (<https://github.com/soedinglab/MMseqs2> https://github.com/soedinglab/MMseqs2>). > Don’t forget about MXCSR register in that suite Yeah, rounding is definitely PITA. It's actually something I completely screwed up on in the beginning of the project and had to go back and correct :(. We do have some tests now which fiddle with the rounding mode to verify correctness, but could definitely use more, and obviously we can't always set a dedicated register to control behavior, so on some platforms `_mm_getcsr`/`_mm_setcsr`/`_MM_GET_ROUNDING_MODE`/`_MM_SET_ROUNDING_MODE` becomes `fegetround`/`fesetround`, which probably won't be a problem but still makes me uncomfortable. The other area where we could really use more tests is replicating behavior for NaNs. By default we try to replicate the behavior of the function we're trying to emulate, but we currently only test NaN handling on a few functions :(. If you use -ffast-math or -ffinite-math-only we disable that code (compilers define __FINITE_MATH_ONLY__), though, and just use the fastest implementation we can.

dragontamer 6y ago

Shuffle, pack/unpack, movemask, blends (SSE/AVX) and interleaved load/stores, byte swap (NEON) are "just" data-movement instructions. All of them can be implemented (with obvious slowdowns) with a conditional write to memory, then a conditional read from memory. Yeah, its inefficient to do it like this, but this "write then read" pattern really gives us an idea of what's really going on between the registers in a pack/pshufb/whatever instruction. On AMD GPUs, there's a fully arbitrary crossbar between SIMD-lanes allowing for arbitrary movement. The two instructions are just "permute" and "b-permute" (backwards permute), roughly correlating to gather and scatter respectively. On NVidia GPUs, perm and bperm are both implemented in PTX, but instead read/write to L1 or __shared__ memory. NVidia GPUs likely have a crossbar to L1 memory to make this instruction very fast. --------- The solution is to implement perm and bperm on AVX. Its already half-implemented: pshufb is equivalent to GPU-permute. CPUs are just missing the backwards permute. I'm pretty confident that pack/unpack, blends, interleaved load/stores, and more could all be implemented as pshufb and a hypothetical "backwards pshufb". Version 1.0 could be an NVidia-like "write to L1 cache" sort of implementation too, if full crossbars are too expensive at the hardware layer. ----------- So the question is: how should we write code today? CPUs of today do not implement this feature, but CPUs of the future might. I think specifying the memory-moves explicitly, and then working on a "pshufb compiler/optimizer" of sorts is what we need.

Const-me 6y ago
> the question is: how should we write code today? The only thing that matters today is how fast the code works on today’s hardware. To illustrate, see how I have emulated neon’s vst3q_s16 with SSE: https://github.com/Const-me/DtsDecoder/blob/master/Utils/store3-sse.hpp https://github.com/Const-me/DtsDecoder/blob/master/Utils/sto... However, not all of them can be emulated in a fast enough manner. > specifying the memory-moves explicitly, and then working on a "pshufb compiler/optimizer" of sorts is what we need. I’ll consider that approach when I’ll see that compiler/optimizer working, and producing output comparable to manually-written code. Until then I think that’s a “sufficiently smart compiler” class of problems. These are rarely ever solved, IMO. For example, we don’t have generally useful auto-vectorizers in C compilers despite two decades of R&D, even the best of them (clang, intel) are still very limited.
floatboth 6y ago
> how should we write code today? Write in a GPU-style "compute shader" language and let the compiler pick the fastest ways of doing things on each ISA? https://github.com/ispc/ispc https://github.com/ispc/ispc
- dragontamer 6y ago
  Well, such a feature doesn't exist in ispc. I guess I'm proposing a hypothetical feature that doesn't exist yet in any compiler. But it'd be nice if it existed... For it to exist, we need a combination of new assembly instructions as well as a smart enough compiler.

gioele 6y ago

> I'm not sure this is going to work fast enough for that. From reading the SIMD Everywhere description, it seems to me that SIMDe is a way to allow code that targets only a platform to work on other platforms as well. As a nice byproduct, you get _some_ speed up if architecture targeted by the code is similar to the architecture that will run the code. Portability is the main focus, not speed. Obviously, once you have a good emulation of an architecture the first question is going to be: can I make it faster?

mtgx 6y ago
All other chip architectures should adopt Arm's SVE2 or something similar. https://community.arm.com/developer/ip-products/processors/b/processors-ip-blog/posts/new-technologies-for-the-arm-a-profile-architecture https://community.arm.com/developer/ip-products/processors/b...
- fao_ 6y ago
  Hi there! It looks like you've been shadowbanned, I vouched a few of your comments because they didn't seem terrible, but I didn't scroll very far. It might be worth sending a mail to dang to ask him to take a look at your account or something!
  - chmod775 6y ago
    That account has 33k karma and is 8 years old. I just can't imagine a set of circumstances where a shadowban of all things could be justified if they transgressed in some way. It's probably a mistake?
    - fao_ 6y ago
      or they said something ridiculously heinous
      - chmod775 6y ago
        In that case one should point that out to them and warn them. Shadowbans are quite a heinous punishment in themselves, and better used to prevent spammers etc. from creating new accounts. But people who make thousands upon thousands of comments are bound to make an emotional comment or an error of judgment at some point.

gameswithgo 6y ago

>I'm not convinced it's possible to do automatically. It isn't always, but sometimes it is.

Rebelgecko 6y ago

Are compilers smart enough to interleave load/stores? I've noticed when using NEON intrinsics that GCC occasionally did clever things (and of course sometimes I got huge speed improvements by making small changes, like adding a PLD a few instructions before a hunk of memory is used)

nemequ1729 6y ago

(Lead developer of SIMDe here.) It sounds like you actually know what you're doing, so in this case you're probably right, at least if all you do is compile your x86 code with SIMDe. That said, SIMDe also provides support for other architectures, notably WASM SIMD 128 and AltiVec/VSX, as well as portable implementations which work everywhere, including on CPUs I'd never heard of until people told me SIMDe was working well on them (I'm thinking of Kalray, which supports vectors but doesn't have an API and instead relies on compiler auto-vectorization support). One use case for SIMDe which may be interesting for you is that you can freely mix calls to different APIs. Say, for example, that you already have a bunch of x86 code written and want a NEON port. You can add SIMDe and you get a NEON port basically for free, then you can start adding some ifdefs to add optimizations for NEON without having to rewrite the whole thing. SIMDe doesn't in any way prevent you from optimizing your NEON (or whatever) port. The way I tend to look at it is that SIMDe never makes your code slower, only more portable.

Const-me 6y ago
I might give it a try next time I’ll need to do something for AMD64 + NEON. Not going to rewrite code I have already written, debugged and shipped. Also, about this > we have an extensive test suite to verify our implementations Don’t forget about MXCSR register in that suite, esp. the rounding bits of that. I avoid changing it as much as possible ‘coz the state is preserved across context switches and causes funny things in OpenMP and other thread pools, but not all people are aware of that. Also, there’s non-trivial amount of code written for SSE < 4.1 (the 4.1 introduced proper rounding instruction, roundps) where you sometimes forced to mess with MXCSR rounding bits because the alternatives are much slower.
- nemequ1729 6y ago
  > Not going to rewrite code I have already written, debugged and shipped. I wouldn't, either. At least unless you want a WASM/AltiVec/etc. version. But if you already have good implementations SIMDe probably won't help. OTOH, if all you have is a x86 implementation and a portable fallback, the SIMDe version of the x86 implementation will probably be faster than your portable version. That's what happened with MMseqs2 (<https://github.com/soedinglab/MMseqs2> https://github.com/soedinglab/MMseqs2>). > Don’t forget about MXCSR register in that suite Yeah, rounding is definitely PITA. It's actually something I completely screwed up on in the beginning of the project and had to go back and correct :(. We do have some tests now which fiddle with the rounding mode to verify correctness, but could definitely use more, and obviously we can't always set a dedicated register to control behavior, so on some platforms `_mm_getcsr`/`_mm_setcsr`/`_MM_GET_ROUNDING_MODE`/`_MM_SET_ROUNDING_MODE` becomes `fegetround`/`fesetround`, which probably won't be a problem but still makes me uncomfortable. The other area where we could really use more tests is replicating behavior for NaNs. By default we try to replicate the behavior of the function we're trying to emulate, but we currently only test NaN handling on a few functions :(. If you use -ffast-math or -ffinite-math-only we disable that code (compilers define __FINITE_MATH_ONLY__), though, and just use the fastest implementation we can.