4 ms·

I just do gcc -O3 and get SIMD without having to learn it

by qurren 3mo ago

forrestthewoods 3mo ago
auto-vectorization is not nearly as good as you would hope it to be. The best SIMD optimizations likely require changing your data format from AoS to SoA.
- raegis 3mo ago
  What are AoS and SoA?
  - nylonstrung 3mo ago
    Array of Structs and Struct of Arrays
  - Georgelemental 3mo ago
    Array of Structs and Struct of Arrays https://en.wikipedia.org/wiki/AoS_and_SoA https://en.wikipedia.org/wiki/AoS_and_SoA
    - Rendello 3mo ago
      A good introduction to SoA (for anyone curious) are the two most famous Data-Oriented Design talks by Mike Acton (game engine dev) [1] and Andrew Kelley (Zig lead dev) [2] respectively. I read a book book about DoD [3] really which confused me at first with all its talk about database table design (in a book about a high-performance C++ game engine?), but when it finally clicked it was amazing. The point is that you want to think hard about your access patterns and what could constitute good "primary keys", then model it accordingly. SoA ends up being useful a lot of the time, because having your data in homogeneous arrays/vectors is great for cache locality and branch elimination. Even without SIMD you can get huge speedups from that, but that's also where your compiler (or you as a programmer) can get incredible SIMD gains. SoA is not a silver bullet as it may not align well with your access patterns, but it can great to add to your toolkit. --- Mike Acton: Data-Oriented Design and C++: https://www.youtube.com/watch?v=rX0ItVEVjHc https://www.youtube.com/watch?v=rX0ItVEVjHc Andrew Kelley: A Practical Guide to Applying Data Oriented Design: https://www.youtube.com/watch?v=IroPQ150F6c https://www.youtube.com/watch?v=IroPQ150F6c Richard Fabian: Data-Oriented Design: https://www.dataorienteddesign.com/dodbook/ https://www.dataorienteddesign.com/dodbook/
- formerly_proven 3mo ago
  And -march=native or at least -march=x86-64-v3 or similar, alternatively identifying relevant functions and manually invoking FMV and uarch specialization via target_clones. Plus non-integer code can generally not be autovectorized in normal-math mode since FP is non-commutative.
- nylonstrung 3mo ago
  The one feature in Jonathan Blow's Jai language I really envy is a a single keyword to switch AoS to SoA and visa-versa at comptime
  - mbStavola 3mo ago
    Didn't he drop this feature years ago?
- Joker_vD 3mo ago
  Well, then I just prompt Claude and get SIMD without having to learn it /s
- ethin 3mo ago
  Either this or you have to do special tricks like pairwise tree reductions and hand-unroll certain portions of loops.
- pjmlp 3mo ago
  While C++ may be reaching levels of Algol 68, PL/I complexity, with C++26 reflection you can do automatically. See https://github.com/cern-nextgen/reflmempp https://github.com/cern-nextgen/reflmempp
  - forrestthewoods 3mo ago
    That’s a research project. So no.
- exDM69 3mo ago
  > The best SIMD optimizations likely require changing your data format from AoS to SoA. We do have gather load instructions in SIMD instruction sets these days (AVX2 and newer), so AoS vs SoA is not nearly as important as it was once. Scatter stores are also available but only in newer CPUs.
  - inigyou 3mo ago
    It remains extremely important. Gather loads are much more expensive than sequential loads, for obvious reasons.
ashton314 3mo ago
In the article, Mitchel mentions how this doesn’t always work. In fact, as someone who’s worked in compiler development, I can say it’s a small miracle when it does work.
- mitchellh 3mo ago
  Case-in-point, the example in my own post doesn't auto-vectorize with LLVM or GCC at highest optimization levels. Basically, compilers will never auto-vectorize loops with an early loop break afaik.
  - yunnpp 3mo ago
    You need to let the compiler know that there are at least 4 or 8 elements to process. This may require padding data and/or having a second loop after the main one that processes the remainder <4 or <8 elements. You start the post with: > There is an opportunity to use SIMD. SIMD turns those into this: > > for (8 byte chunk in bytes) { /* ... */ } If you actually wrote that loop, there is a good chance the compiler (gcc specifically) will auto-vectorize. In any case, the more manual SIMD optimizations I have seen require reworking the data altogether, not just processing N elements at a time. For example, instead of packing two 4-vectors into two registers to do a dot product, pack the XXXXs, YYYYs, etc. into 4 vectors and compute 4 dot products for the price of one. That not only requires having 4 vectors to process, but also thinking how exactly they are packed in registers. I don't know why qurren is downvoted. You really should see if you can get the compiler to auto-vectorize first (possibly padding data structures and loops) before you write anything by hand.
    - spider-mario 3mo ago
      > You really should see if you can get the compiler to auto-vectorize first (possibly padding data structures and loops) before you write anything by hand. Counterpoint: https://pharr.org/matt/blog/2018/04/18/ispc-origins https://pharr.org/matt/blog/2018/04/18/ispc-origins > I think that the fatal flaw with the approach the compiler team was trying to make work was best diagnosed by T. Foley, who’s full of great insights about this stuff: auto-vectorization is not a programming model. > The problem with an auto-vectorizer is that as long as vectorization can fail (and it will), then if you’re a programmer who actually cares about what code the compiler generates for your program, you must come to deeply understand the auto-vectorizer. Then, when it fails to vectorize code you want to be vectorized, you can either poke it in the right ways or change your program in the right ways so that it works for you again. This is a horrible way to program; it’s all alchemy and guesswork and you need to become deeply specialized about the nuances of a single compiler’s implementation—something you wouldn’t otherwise need to care about one bit. > And God help you when they release a new version of the compiler with changes to the auto-vectorizer’s implementation. > With a proper programming model, then the programmer learns the model (which is hopefully fairly clean), one or more compilers implement it, the generated code is predictable (no performance cliffs), and everyone’s happy.
jandrewrogers 3mo ago
Most scalar-to-SIMD conversion requires changing the design of data structures and algorithms to be effective. Compilers are required to exactly reproduce the specified data structures in a deterministic way for obvious reasons. Even if compilers were clever enough to transform your data structures and algorithms for SIMD (they're not), the data structures are a contract that can't be unilaterally modified.
llm_nerd 3mo ago
I have no idea why you're being downvoted. HN has a fetish for SIMD, but if you are hand-rolling SIMD and you aren't writing an explicit acceleration library, you're doing it wrong. Like, 100% of the time. Every modern language has a vectorization optimizing compiler, and through some fairly straightforward techniques this is automagic. And contrary to the various replies, unless you screwed something up compilers are really good at vectorizing on whatever hardware you're targeting, including SVE.
- saagarjha 3mo ago
  Compilers are really good but really good is not actually that useful in cases where you need SIMD
  - llm_nerd 3mo ago
    I mean, utter bullshit. If you "need SIMD" you know exactly the programming pattern to guarantee SIMD from the compiler. And the single and only people who "need SIMD" know these rules. It is only the hobbyist "SIMD is neat" community that upvotes these ridiculous articles.
    - saagarjha 3mo ago
      When I need SIMD I rarely use the compiler to help
- spider-mario 3mo ago
  > HN has a fetish for SIMD, but if you are hand-rolling SIMD and you aren't writing an explicit acceleration library, you're doing it wrong. Like, 100% of the time. What if the “explicit acceleration library” for what you need to do doesn’t exist?
  - Archit3ch 3mo ago
    Or it exists and is not optimal for your use case?

I just do gcc -O3 and get SIMD without having to learn it

by qurren 3mo ago

I just do gcc -O3 and get SIMD without having to learn it

auto-vectorization is not nearly as good as you would hope it to be. The best SIMD optimizations likely require changing your data format from AoS to SoA.

raegis 3mo ago
What are AoS and SoA?
- nylonstrung 3mo ago
  Array of Structs and Struct of Arrays
- Georgelemental 3mo ago
  Array of Structs and Struct of Arrays https://en.wikipedia.org/wiki/AoS_and_SoA https://en.wikipedia.org/wiki/AoS_and_SoA
  - Rendello 3mo ago
    A good introduction to SoA (for anyone curious) are the two most famous Data-Oriented Design talks by Mike Acton (game engine dev) [1] and Andrew Kelley (Zig lead dev) [2] respectively. I read a book book about DoD [3] really which confused me at first with all its talk about database table design (in a book about a high-performance C++ game engine?), but when it finally clicked it was amazing. The point is that you want to think hard about your access patterns and what could constitute good "primary keys", then model it accordingly. SoA ends up being useful a lot of the time, because having your data in homogeneous arrays/vectors is great for cache locality and branch elimination. Even without SIMD you can get huge speedups from that, but that's also where your compiler (or you as a programmer) can get incredible SIMD gains. SoA is not a silver bullet as it may not align well with your access patterns, but it can great to add to your toolkit. --- Mike Acton: Data-Oriented Design and C++: https://www.youtube.com/watch?v=rX0ItVEVjHc https://www.youtube.com/watch?v=rX0ItVEVjHc Andrew Kelley: A Practical Guide to Applying Data Oriented Design: https://www.youtube.com/watch?v=IroPQ150F6c https://www.youtube.com/watch?v=IroPQ150F6c Richard Fabian: Data-Oriented Design: https://www.dataorienteddesign.com/dodbook/ https://www.dataorienteddesign.com/dodbook/
formerly_proven 3mo ago
And -march=native or at least -march=x86-64-v3 or similar, alternatively identifying relevant functions and manually invoking FMV and uarch specialization via target_clones. Plus non-integer code can generally not be autovectorized in normal-math mode since FP is non-commutative.
nylonstrung 3mo ago
The one feature in Jonathan Blow's Jai language I really envy is a a single keyword to switch AoS to SoA and visa-versa at comptime
- mbStavola 3mo ago
  Didn't he drop this feature years ago?
Joker_vD 3mo ago
Well, then I just prompt Claude and get SIMD without having to learn it /s
ethin 3mo ago
Either this or you have to do special tricks like pairwise tree reductions and hand-unroll certain portions of loops.
pjmlp 3mo ago
While C++ may be reaching levels of Algol 68, PL/I complexity, with C++26 reflection you can do automatically. See https://github.com/cern-nextgen/reflmempp https://github.com/cern-nextgen/reflmempp
- forrestthewoods 3mo ago
  That’s a research project. So no.
exDM69 3mo ago
> The best SIMD optimizations likely require changing your data format from AoS to SoA. We do have gather load instructions in SIMD instruction sets these days (AVX2 and newer), so AoS vs SoA is not nearly as important as it was once. Scatter stores are also available but only in newer CPUs.
- inigyou 3mo ago
  It remains extremely important. Gather loads are much more expensive than sequential loads, for obvious reasons.

ashton314 3mo ago

In the article, Mitchel mentions how this doesn’t always work. In fact, as someone who’s worked in compiler development, I can say it’s a small miracle when it does work.

mitchellh 3mo ago
Case-in-point, the example in my own post doesn't auto-vectorize with LLVM or GCC at highest optimization levels. Basically, compilers will never auto-vectorize loops with an early loop break afaik.
- yunnpp 3mo ago
  You need to let the compiler know that there are at least 4 or 8 elements to process. This may require padding data and/or having a second loop after the main one that processes the remainder <4 or <8 elements. You start the post with: > There is an opportunity to use SIMD. SIMD turns those into this: > > for (8 byte chunk in bytes) { /* ... */ } If you actually wrote that loop, there is a good chance the compiler (gcc specifically) will auto-vectorize. In any case, the more manual SIMD optimizations I have seen require reworking the data altogether, not just processing N elements at a time. For example, instead of packing two 4-vectors into two registers to do a dot product, pack the XXXXs, YYYYs, etc. into 4 vectors and compute 4 dot products for the price of one. That not only requires having 4 vectors to process, but also thinking how exactly they are packed in registers. I don't know why qurren is downvoted. You really should see if you can get the compiler to auto-vectorize first (possibly padding data structures and loops) before you write anything by hand.
  - spider-mario 3mo ago
    > You really should see if you can get the compiler to auto-vectorize first (possibly padding data structures and loops) before you write anything by hand. Counterpoint: https://pharr.org/matt/blog/2018/04/18/ispc-origins https://pharr.org/matt/blog/2018/04/18/ispc-origins > I think that the fatal flaw with the approach the compiler team was trying to make work was best diagnosed by T. Foley, who’s full of great insights about this stuff: auto-vectorization is not a programming model. > The problem with an auto-vectorizer is that as long as vectorization can fail (and it will), then if you’re a programmer who actually cares about what code the compiler generates for your program, you must come to deeply understand the auto-vectorizer. Then, when it fails to vectorize code you want to be vectorized, you can either poke it in the right ways or change your program in the right ways so that it works for you again. This is a horrible way to program; it’s all alchemy and guesswork and you need to become deeply specialized about the nuances of a single compiler’s implementation—something you wouldn’t otherwise need to care about one bit. > And God help you when they release a new version of the compiler with changes to the auto-vectorizer’s implementation. > With a proper programming model, then the programmer learns the model (which is hopefully fairly clean), one or more compilers implement it, the generated code is predictable (no performance cliffs), and everyone’s happy.

jandrewrogers 3mo ago

Most scalar-to-SIMD conversion requires changing the design of data structures and algorithms to be effective. Compilers are required to exactly reproduce the specified data structures in a deterministic way for obvious reasons. Even if compilers were clever enough to transform your data structures and algorithms for SIMD (they're not), the data structures are a contract that can't be unilaterally modified.

llm_nerd 3mo ago

I have no idea why you're being downvoted. HN has a fetish for SIMD, but if you are hand-rolling SIMD and you aren't writing an explicit acceleration library, you're doing it wrong. Like, 100% of the time. Every modern language has a vectorization optimizing compiler, and through some fairly straightforward techniques this is automagic. And contrary to the various replies, unless you screwed something up compilers are really good at vectorizing on whatever hardware you're targeting, including SVE.

saagarjha 3mo ago
Compilers are really good but really good is not actually that useful in cases where you need SIMD
- llm_nerd 3mo ago
  I mean, utter bullshit. If you "need SIMD" you know exactly the programming pattern to guarantee SIMD from the compiler. And the single and only people who "need SIMD" know these rules. It is only the hobbyist "SIMD is neat" community that upvotes these ridiculous articles.
  - saagarjha 3mo ago
    When I need SIMD I rarely use the compiler to help
spider-mario 3mo ago
> HN has a fetish for SIMD, but if you are hand-rolling SIMD and you aren't writing an explicit acceleration library, you're doing it wrong. Like, 100% of the time. What if the “explicit acceleration library” for what you need to do doesn’t exist?
- Archit3ch 3mo ago
  Or it exists and is not optimal for your use case?