10 ms·

I’ve said it before and I’ll say it again: Rust feels like a Python developer’s idea of a high-performance computing language. It’s a great language for many ki

I’ve said it before and I’ll say it again: Rust feels like a Python developer’s idea of a high-performance computing language. It’s a great language for many kinds of applications — just not when you need to squeeze out every bit of performance from advanced hardware. Even before getting into SIMD, try using Rust for concurrent, succinct, or external-memory data structures. It quickly becomes clear where the friction is. Cargo is fantastic — clean, ergonomic, and a joy compared to many toolchains. But it’s much easier to keep things simple when you don’t have to support dozens of AVX-512 variants, AMX, SME, different CUDA generations, ROCm, or any of the other modern hardware capabilities. Standardising SIMD in the standard library — in Rust or C++ — has always been a questionable idea. Most of these APIs cater to operations that compilers already auto-vectorize reasonably well, and they barely touch the recent capabilities of SIMD. Just consider how hard it is to build any meaningful abstraction over the predicate/register models across AVX-512, SVE, and RVV. RVV aside, this should illustrate the point: https://www.modular.com/blog/understanding-simd-infinite-complexity-of-trivial-problems https://www.modular.com/blog/understanding-simd-infinite-com...

lifthrasiir 2y ago
> Just consider how hard it is to build any meaningful abstraction over the predicate/register models across AVX-512, SVE, and RVV. Note that Highway mentioned in the post does take care of this, which is no easy feat but also a proof that it is doable.
the__alchemist 2y ago
I guess it comes down to application. If you don't attempt to find the most general solution, you can dodge those pitfalls. Case in point, abstracting over AVX-512, SVE, and RVV may be tough, but picking one is fine (On nightly only for now), can with the right abstractions can be almost as straightforward as using normal scalar values. I don't have a solution on the CUDA variants either; have been hard-coding that as well... (Cudarc lib with CUDA-version feature gates and GPU-series-specific code). Haven't hit a brick wall yet, but might... or might not.
dzaima 2y ago
I don't think Rust is particularly problematic here. As long as you don't want to do funky things like use immutable argument memory as temporary scratch space (with you restoring the values afterwards of course), all it means is some `unsafe`ing at worst, compared to C/C++. And there are some safe abstractions you can make over loads/stores (everything else being safe, even if not yet marked as such). Do agree that a standard SIMD type is rather pointless, if not immediately, then in like 5 years. (and, seemingly, both Rust and C++ are like over 10 years behind on SIMD, so they're already out-of-date) Maybe somewhat useful if you just want the simple ~8x speedup, and not squeeze out the last 1.4x or whatever, but autovectorization should be capable of covering a significant amount of such.
- ashvardanian 2y ago
  … except for byte-level processing, variable-length codecs, or mixed-precision numerics. That never works with autovectorization and can’t be solved with general-purpose SIMD wrappers. For me the solution was to implement those manually, and even at a scale of just 2 libraries I’ve eneded up with somewhat different project layouts & dispatch mechanisms: https://github.com/ashvardanian/SimSIMD https://github.com/ashvardanian/SimSIMD , https://github.com/ashvardanian/StringZilla https://github.com/ashvardanian/StringZilla One big family not covered there, is sparse data-strictures and related algorithms. I’ve only started integrating scatter/gather in AVX-512 and SVE, and on synthetic benchmarks both look promising: https://github.com/ashvardanian/less_slow.cpp/releases/tag/v0.3.0 https://github.com/ashvardanian/less_slow.cpp/releases/tag/v... Those should probably unlock a much wider set of applications for SIMD, but designing libraries for those may benefit from yet another project structure.
  - dzaima 2y ago
    Ah yeah, gather/scatter are indeed a rather problematic thing for autovectorization. That said, with no-alias info (which Rust has a lot of) it's possible: https://rust.godbolt.org/z/zTfo9nxhd https://rust.godbolt.org/z/zTfo9nxhd. Unfortunately it doesn't get autovectorized without the unsafes, but theoretically it should be possible-ish for bounds checking to be autovectorized (most problematic aspect being that it might be hard to annoying-to-impossible to ensure that in the case of multiple panic/UB sources the proper one happens first). I'd imagine in any non-trivial situation you'd want a custom layer of abstractions over whatever the language provides for all languages. For that a portable-simd thing is actually a rather good base, on which you could add custom arch-specific abstractions/ops as desired. Not sure what's problematic with mixed-precision (I know SVE is rather weird for mixed-width elements, but that's about it?), though I primarily don't care about float stuff generally. Also no clue what's problematic with byte-level stuff. Indeed there are still a bunch of things that you want proper manual SIMD for (hell the SIMDful project I work on has an entire DSL for doing nice SIMD), but autovectorization still covers a good amount.
creata 2y ago
> Rust feels like a Python developer’s idea of a high-performance computing language. I might be wrong, but I think it sounds more like Rust doesn't move as far away from the C or C++ way of doing things as you want it to. At the very least, Rust is no worse than C or C++ at any of the things you mentioned.
- ashvardanian 2y ago
  In my (biased) experience, Rust is much harder to use for advanced projects, than C and C++. On the bright side, it’s also harder to misuse :)
  - saagarjha 2y ago
    I don’t think Rust is worse at SIMD code, though.
  - vlovich123 2y ago
    I've found it much easier to use from writing a web service to writing a high performance DB that outperforms RocksDB and clearly people are using it for things like writing operating systems as well as game engines. I'm not sure what in your mind falls under "advanced projects" but I suspect it's something like number crunching (although you link StringZilla so not sure). I'm still not seeing any description of specific challenges you feel are harder in Rust than in C/C++. In my mind Rust is completely equivalent in being able to accomplish the same tasks.
    - porridgeraisin 2y ago
      They wrote 2 whole paragraphs about SIMD.
      - vlovich123 2y ago
        And to me SIMD does not seem significantly harder in Rust than in C++ but I've only done a little Rust SIMD so I'm willing to be wrong on that (although others have said it's not much harder than in C++ so not sure).
pclmulqdq 2y ago
Rust is easy to understand as "a language by browser writers for writing browsers." That statement alone gives you most of Rust's design choices: * Safety over everything else * Very good single threaded performance * Javascript-like syntax and a Javascript-like package manager I have been working on some software in Rust recently that needs bit and byte manipulation, and we have "unsafe" everywhere and hugely complicated spaghetti compared to the equivalent code in C.
- 1932812267 2y ago
  I love your username, btw :)
- pjmlp 2y ago
  Most likely because many keep forgetting that bit and byte manipulation in C is a mix of implementation defined and UB, depending on how it is coded.
  - motorest 2y ago
    What's the problem of using toolchain-specific features instead of behavior defined by the standard? Isn't this the bread and butter of embedded development and the reason why some behavior is purposely left undefined in the standard?
    - eru 2y ago
      > [...] and the reason why some behavior is purposely left undefined in the standard? It's one of the saner reasons why there's UB in C and C++. But there's lots of crazier reasons.
      - motorest 2y ago
        > It's one of the saner reasons why there's UB in C and C++. It is the reason why the standard purposely leaves some areas undefined. C and C++ detractors talk a lot about UB but they always show their understanding on the subject is at best superficial. The parrot UB as if it was this major gotcha, when it is literally behavior the standardization committees make a call to not define behavior to purposely leave it open, so that any implementation can still be conforming even if they decide to implement behavior not defined in the standard. It is that simple, but somehow people parrot UB behavior as if it was this major gotcha. Baffling.
1932812267 2y ago
I've written a fair bit of SIMD code in Rust, and it definitely had lots of sore spots. The main advantage was that, because Rust doesn't use TBAA, it's completely legal (and safe, if you use bytemuck) to freely cast pointers and values around. TBAA in C++ makes it much easier to hit undefined behavior. But also, because of various miscompilations, Rust refuses (or at least refused) to pass SIMD arguments in registers, so every non-inlined function call passes arguments via the stack. There were also miscompilations if you enabled a target_feature for just one function, so we ended up just passing `-C target-cpu=...` globally, and if we wanted to support a different microarchitecture, we just recompiled the whole program. On top of that, there's no good way to check to see what microarchitecture you're compiling for, so we had to resort to specifying the target cpu in multiple places, with comments reminding us to keep the places in sync.
pcwalton 2y ago
> Even before getting into SIMD, try using Rust for concurrent, succinct, or external-memory data structures. It quickly becomes clear where the friction is. It's the exact opposite for me: I use concurrent data structures more often in Rust than I do in C++ because I don't have to worry about dumb data race bugs. If one of my Bevy systems is slow, I slap par_iter() on the query and if it compiles it probably works, or at least fails for a not-stupid reason.
- SleepyMyroslav 2y ago
  What do you think about 'par_iter' having to wait for work imbalance to return execution back? With 'when_all' like primitive one can continue execution on any thread without losing one for waiting. ps as someone who does not have rust job i would like to see an example how rust deals with task based systems if you have public one at hand ofc.
  - pornel 2y ago
    The rayon library uses work stealing for this. Its parallel iterators offer some control of splitting strategies and granularity, so you can tune the trade-off between full utilization and cost of moving things between threads. Additionally, in Bevy, independent queries (systems) are executed in parallel, so there's always something else to do, except your one worst loop.
- pclmulqdq 2y ago
  Concurrent data structures are rarely faster than "lock + non-concurrent data structure" and if you're putting constructs like par_iter() in a lot of places, there's a good chance you would be better off with the "dumb" pattern than the concurrent data structure. The same goes with Arc - if you're using it a lot there's a good chance that code with a lot of Arcs is slower than equivalent GC-ed code.
  - 1932812267 2y ago
    While it's true that par_iter() uses a concurrent data structure under the hood, it's specifically designed to use work-stealing to avoid needing threads to communicate. Why would putting a lock over a global workqueue be faster than per-thread workqueues that don't require inter-thread communication (except in the case where work-stealing is required)?
jandrewrogers 2y ago
One of the things I find interesting about SIMD is that a lot of behavior that is “undefined” for scalar types in C-derived languages is explicitly fully defined when you use SIMD intrinsics with the same integral types. UB exists to hide the fact that major CPU architectures give different results for basic ALU operations in some cases. SIMD makes no such pretense of abstraction. If I am using AVX-512 I explicitly get the full Intel architecture experience, the implementation details are not hidden behind UB. Same with ARM, etc. For example, shift overflows are masked on x86, zero-filled on ARM, and undefined in C/C++. In SIMD-land, none of this is hidden and so you design your code to leverage the reality that those instructions behave differently, whereas in C/C++ only the behavior they have in common is “defined”. The vector ISAs are sufficiently different from each other (and normal CPUs) that it is like trying to build a compiler that can automagically produce optimized code for both CPUs and GPUs from the same source tree. I am not optimistic that this will happen anytime soon. AVX-512 essentially started life as a GPU ISA, which probably explains the interesting fact that a modern x86 CPU core has more AVX-512 registers than x86 registers.
- grandempire 2y ago
  Of course. Many things are UB because dictating a policy for all machines doesn’t make sense. Whereas AVX is a specification for a specific hardware capability.
athrowaway3z 2y ago
https://xkcd.com/378/ https://xkcd.com/378/
imtringued 2y ago
Python doesn't have anything remotely similar to Cargo that isn't written in Rust (uv is). It's exactly backwards. Python is benefitting from Rust philosophy.
fulafel 2y ago
> Rust feels like a Python developer’s idea of a high-performance computing language. I'd say the language complexity (and it keeps piling more things) means it's pretty far from Python's ideas. Julia is closer.
- andrepd 2y ago
  Rust is an extraordinary simple language. Compare it to the complexity of C++.
  - fulafel 2y ago
    We can say that Mars is more similar to Earth than Jupiter even though Sagittarius A* exists.
    - andrepd 2y ago
      Unsure which celestial body is supposed to be which in this comparison :p
  - vacuity 2y ago
    If you only compare against C++, maybe. Otherwise, no. C++ and Rust are in a class of their own when it comes to compile times and complexity.
    - andrepd 2y ago
      Well C++ is the only major language (other than C itself) that occupies the same space. So it's natural that they are compared. And Rust is an order of magnitude simpler language than C++ almost comical web of intricacy.
beng-nl 2y ago
I’m seriously interested; what is the best language/franework/method that will squeeze out every bit of performance from advanced hardware that you know of?

lifthrasiir 2y ago

> Just consider how hard it is to build any meaningful abstraction over the predicate/register models across AVX-512, SVE, and RVV. Note that Highway mentioned in the post does take care of this, which is no easy feat but also a proof that it is doable.

the__alchemist 2y ago

I guess it comes down to application. If you don't attempt to find the most general solution, you can dodge those pitfalls. Case in point, abstracting over AVX-512, SVE, and RVV may be tough, but picking one is fine (On nightly only for now), can with the right abstractions can be almost as straightforward as using normal scalar values. I don't have a solution on the CUDA variants either; have been hard-coding that as well... (Cudarc lib with CUDA-version feature gates and GPU-series-specific code). Haven't hit a brick wall yet, but might... or might not.

dzaima 2y ago

I don't think Rust is particularly problematic here. As long as you don't want to do funky things like use immutable argument memory as temporary scratch space (with you restoring the values afterwards of course), all it means is some `unsafe`ing at worst, compared to C/C++. And there are some safe abstractions you can make over loads/stores (everything else being safe, even if not yet marked as such). Do agree that a standard SIMD type is rather pointless, if not immediately, then in like 5 years. (and, seemingly, both Rust and C++ are like over 10 years behind on SIMD, so they're already out-of-date) Maybe somewhat useful if you just want the simple ~8x speedup, and not squeeze out the last 1.4x or whatever, but autovectorization should be capable of covering a significant amount of such.

ashvardanian 2y ago
… except for byte-level processing, variable-length codecs, or mixed-precision numerics. That never works with autovectorization and can’t be solved with general-purpose SIMD wrappers. For me the solution was to implement those manually, and even at a scale of just 2 libraries I’ve eneded up with somewhat different project layouts & dispatch mechanisms: https://github.com/ashvardanian/SimSIMD https://github.com/ashvardanian/SimSIMD , https://github.com/ashvardanian/StringZilla https://github.com/ashvardanian/StringZilla One big family not covered there, is sparse data-strictures and related algorithms. I’ve only started integrating scatter/gather in AVX-512 and SVE, and on synthetic benchmarks both look promising: https://github.com/ashvardanian/less_slow.cpp/releases/tag/v0.3.0 https://github.com/ashvardanian/less_slow.cpp/releases/tag/v... Those should probably unlock a much wider set of applications for SIMD, but designing libraries for those may benefit from yet another project structure.
- dzaima 2y ago
  Ah yeah, gather/scatter are indeed a rather problematic thing for autovectorization. That said, with no-alias info (which Rust has a lot of) it's possible: https://rust.godbolt.org/z/zTfo9nxhd https://rust.godbolt.org/z/zTfo9nxhd. Unfortunately it doesn't get autovectorized without the unsafes, but theoretically it should be possible-ish for bounds checking to be autovectorized (most problematic aspect being that it might be hard to annoying-to-impossible to ensure that in the case of multiple panic/UB sources the proper one happens first). I'd imagine in any non-trivial situation you'd want a custom layer of abstractions over whatever the language provides for all languages. For that a portable-simd thing is actually a rather good base, on which you could add custom arch-specific abstractions/ops as desired. Not sure what's problematic with mixed-precision (I know SVE is rather weird for mixed-width elements, but that's about it?), though I primarily don't care about float stuff generally. Also no clue what's problematic with byte-level stuff. Indeed there are still a bunch of things that you want proper manual SIMD for (hell the SIMDful project I work on has an entire DSL for doing nice SIMD), but autovectorization still covers a good amount.

creata 2y ago

> Rust feels like a Python developer’s idea of a high-performance computing language. I might be wrong, but I think it sounds more like Rust doesn't move as far away from the C or C++ way of doing things as you want it to. At the very least, Rust is no worse than C or C++ at any of the things you mentioned.

ashvardanian 2y ago
In my (biased) experience, Rust is much harder to use for advanced projects, than C and C++. On the bright side, it’s also harder to misuse :)
- saagarjha 2y ago
  I don’t think Rust is worse at SIMD code, though.
- vlovich123 2y ago
  I've found it much easier to use from writing a web service to writing a high performance DB that outperforms RocksDB and clearly people are using it for things like writing operating systems as well as game engines. I'm not sure what in your mind falls under "advanced projects" but I suspect it's something like number crunching (although you link StringZilla so not sure). I'm still not seeing any description of specific challenges you feel are harder in Rust than in C/C++. In my mind Rust is completely equivalent in being able to accomplish the same tasks.
  - porridgeraisin 2y ago
    They wrote 2 whole paragraphs about SIMD.
    - vlovich123 2y ago
      And to me SIMD does not seem significantly harder in Rust than in C++ but I've only done a little Rust SIMD so I'm willing to be wrong on that (although others have said it's not much harder than in C++ so not sure).

pclmulqdq 2y ago

Rust is easy to understand as "a language by browser writers for writing browsers." That statement alone gives you most of Rust's design choices: * Safety over everything else * Very good single threaded performance * Javascript-like syntax and a Javascript-like package manager I have been working on some software in Rust recently that needs bit and byte manipulation, and we have "unsafe" everywhere and hugely complicated spaghetti compared to the equivalent code in C.

1932812267 2y ago
I love your username, btw :)
pjmlp 2y ago
Most likely because many keep forgetting that bit and byte manipulation in C is a mix of implementation defined and UB, depending on how it is coded.
- motorest 2y ago
  What's the problem of using toolchain-specific features instead of behavior defined by the standard? Isn't this the bread and butter of embedded development and the reason why some behavior is purposely left undefined in the standard?
  - eru 2y ago
    > [...] and the reason why some behavior is purposely left undefined in the standard? It's one of the saner reasons why there's UB in C and C++. But there's lots of crazier reasons.
    - motorest 2y ago
      > It's one of the saner reasons why there's UB in C and C++. It is the reason why the standard purposely leaves some areas undefined. C and C++ detractors talk a lot about UB but they always show their understanding on the subject is at best superficial. The parrot UB as if it was this major gotcha, when it is literally behavior the standardization committees make a call to not define behavior to purposely leave it open, so that any implementation can still be conforming even if they decide to implement behavior not defined in the standard. It is that simple, but somehow people parrot UB behavior as if it was this major gotcha. Baffling.

1932812267 2y ago

I've written a fair bit of SIMD code in Rust, and it definitely had lots of sore spots. The main advantage was that, because Rust doesn't use TBAA, it's completely legal (and safe, if you use bytemuck) to freely cast pointers and values around. TBAA in C++ makes it much easier to hit undefined behavior. But also, because of various miscompilations, Rust refuses (or at least refused) to pass SIMD arguments in registers, so every non-inlined function call passes arguments via the stack. There were also miscompilations if you enabled a target_feature for just one function, so we ended up just passing `-C target-cpu=...` globally, and if we wanted to support a different microarchitecture, we just recompiled the whole program. On top of that, there's no good way to check to see what microarchitecture you're compiling for, so we had to resort to specifying the target cpu in multiple places, with comments reminding us to keep the places in sync.

pcwalton 2y ago

> Even before getting into SIMD, try using Rust for concurrent, succinct, or external-memory data structures. It quickly becomes clear where the friction is. It's the exact opposite for me: I use concurrent data structures more often in Rust than I do in C++ because I don't have to worry about dumb data race bugs. If one of my Bevy systems is slow, I slap par_iter() on the query and if it compiles it probably works, or at least fails for a not-stupid reason.

SleepyMyroslav 2y ago
What do you think about 'par_iter' having to wait for work imbalance to return execution back? With 'when_all' like primitive one can continue execution on any thread without losing one for waiting. ps as someone who does not have rust job i would like to see an example how rust deals with task based systems if you have public one at hand ofc.
- pornel 2y ago
  The rayon library uses work stealing for this. Its parallel iterators offer some control of splitting strategies and granularity, so you can tune the trade-off between full utilization and cost of moving things between threads. Additionally, in Bevy, independent queries (systems) are executed in parallel, so there's always something else to do, except your one worst loop.
pclmulqdq 2y ago
Concurrent data structures are rarely faster than "lock + non-concurrent data structure" and if you're putting constructs like par_iter() in a lot of places, there's a good chance you would be better off with the "dumb" pattern than the concurrent data structure. The same goes with Arc - if you're using it a lot there's a good chance that code with a lot of Arcs is slower than equivalent GC-ed code.
- 1932812267 2y ago
  While it's true that par_iter() uses a concurrent data structure under the hood, it's specifically designed to use work-stealing to avoid needing threads to communicate. Why would putting a lock over a global workqueue be faster than per-thread workqueues that don't require inter-thread communication (except in the case where work-stealing is required)?

jandrewrogers 2y ago

One of the things I find interesting about SIMD is that a lot of behavior that is “undefined” for scalar types in C-derived languages is explicitly fully defined when you use SIMD intrinsics with the same integral types. UB exists to hide the fact that major CPU architectures give different results for basic ALU operations in some cases. SIMD makes no such pretense of abstraction. If I am using AVX-512 I explicitly get the full Intel architecture experience, the implementation details are not hidden behind UB. Same with ARM, etc. For example, shift overflows are masked on x86, zero-filled on ARM, and undefined in C/C++. In SIMD-land, none of this is hidden and so you design your code to leverage the reality that those instructions behave differently, whereas in C/C++ only the behavior they have in common is “defined”. The vector ISAs are sufficiently different from each other (and normal CPUs) that it is like trying to build a compiler that can automagically produce optimized code for both CPUs and GPUs from the same source tree. I am not optimistic that this will happen anytime soon. AVX-512 essentially started life as a GPU ISA, which probably explains the interesting fact that a modern x86 CPU core has more AVX-512 registers than x86 registers.

grandempire 2y ago
Of course. Many things are UB because dictating a policy for all machines doesn’t make sense. Whereas AVX is a specification for a specific hardware capability.

athrowaway3z 2y ago

https://xkcd.com/378/ https://xkcd.com/378/

imtringued 2y ago

Python doesn't have anything remotely similar to Cargo that isn't written in Rust (uv is). It's exactly backwards. Python is benefitting from Rust philosophy.

fulafel 2y ago

> Rust feels like a Python developer’s idea of a high-performance computing language. I'd say the language complexity (and it keeps piling more things) means it's pretty far from Python's ideas. Julia is closer.

andrepd 2y ago
Rust is an extraordinary simple language. Compare it to the complexity of C++.
- fulafel 2y ago
  We can say that Mars is more similar to Earth than Jupiter even though Sagittarius A* exists.
  - andrepd 2y ago
    Unsure which celestial body is supposed to be which in this comparison :p
- vacuity 2y ago
  If you only compare against C++, maybe. Otherwise, no. C++ and Rust are in a class of their own when it comes to compile times and complexity.
  - andrepd 2y ago
    Well C++ is the only major language (other than C itself) that occupies the same space. So it's natural that they are compared. And Rust is an order of magnitude simpler language than C++ almost comical web of intricacy.

beng-nl 2y ago

I’m seriously interested; what is the best language/franework/method that will squeeze out every bit of performance from advanced hardware that you know of?