3 ms·
A really powerful outworking of it is seen in the Rayon crate, where you can change a sequential iterator into a parallel iterator just by adding the crate, imp
by chrismorgan 9y ago
A really powerful outworking of it is seen in the Rayon crate, where you can change a sequential iterator into a parallel iterator just by adding the crate, importing the trait and changing `iter` to `par_iter`. If it’s not thread-safe to do, then it won’t compile. (That’s the big difference from C++.) If it is, it will, and it’ll be smart about how it runs, spreading the load across all available cores pretty much optimally, or not bothering with multiple threads if it’s not going to be worth it (e.g. single-threaded, or only one item in the iterator). And all that with close enough to no overhead.
Basically, Rayon makes data parallelism really easy in a way that few if any other languages do. I’d love to have an equivalent in Python or Node, but it’s just not possible to achieve such a thing in most languages—even if you ignore the thread safety aspect.
Parallelism hasn’t seen a great deal of use until it’s urgently needed, because it’s hard to get right in most environments, and you normally need to substantially refactor code to make it happen. My hope is that with the likes of Rayon, parallelism can be a much more natural thing that people that care even a little about performance will just do, because it’s so easy to do.
https://crates.io/crates/rayon https://crates.io/crates/rayon
- pjmlp 9y ago> Parallelism hasn’t seen a great deal of use until it’s urgently needed This is the main reason why, initially, Windows Store APIs were all async. Microsoft learned that when developers can choose between both models, by default most chose synch models.
- ethagnawl 9y agoFWIW, you can also achieve similar behavior in Clojure via pmap, Reducers, etc.
- floatboth 9y agoAnd in Haskell https://www.stackage.org/haddock/lts-10.6/parallel-3.2.1.1/Control-Parallel-Strategies.html https://www.stackage.org/haddock/lts-10.6/parallel-3.2.1.1/C... And in .NET https://docs.microsoft.com/en-us/dotnet/api/system.threading.tasks.parallel?view=netframework-4.7.1 https://docs.microsoft.com/en-us/dotnet/api/system.threading... And in Java https://blog.oio.de/2016/01/22/parallel-stream-processing-in-java-8-performance-of-sequential-vs-parallel-stream-processing/ https://blog.oio.de/2016/01/22/parallel-stream-processing-in... Rayon is notable because Rust is a native-compiled, no-GC language.
- brazzy 9y agoTo me, the "If it’s not thread-safe to do, then it won’t compile" part sounds rather more notable.
- coolsunglasses 9y agoThat's true of Haskell for the parallelism faculties. It's also true for STM. You can still get stuck or have race conditions with the other concurrency abstractions but it's less common than I encountered elsewhere. I didn't have problems resulting from mutable shared state that was aliased across threads and wasn't supposed to be, it's always explicit.
- mitchtbaum 9y agoAnother good crate to look at for this is Actix, Rust actors framework: https://github.com/actix/actix https://github.com/actix/actix
- romwell 9y ago>Basically, Rayon makes data parallelism really easy in a way that few if any other languages do. Syntax-wise, there's OpenMP which can turn a for-loop into a parallelized for-loop (independently scheduled iterations) with just some syntactic sugar on top of the loop. OpenMP has support for at least C++ and Fortran, and is not hard to use. I wonder how Rayon compares to OpenMP.