7 ms·

Saved by the compiler: Parallelizing a loop with Rust and rayon

179 points by GolDDranks 10y ago

ekidd 10y ago
Author here. I'm happy to answer questions! I don't have any performance numbers for this loop because it was already very fast. This was just a warm-up exercise before trying to parallelize more complicated loops. My experience with Rust has been interesting. On the one hand, if you compare Rust to Ruby or JavaScript, there really is a "Rust tax"—you have to think about ownership and references and types. But Rust pays me back by making refactoring very easy, and by catching some surprisingly subtle errors at compile time. Plus, I really like Cargo and the crates.io ecosystem. Rust might not be a good match for every project or every programmer, but I'm really enjoying it for low-level, high-performance code.
- dkastner 10y ago
  A huge payoff that Rust provides vs Ruby or JavaScript is when writing development tools. It saves so much time and trouble to install a binary than to try and get someone to download the right version of Ruby, gem install, etc. Rust, while having its tax, is really great about telling you why your code won't compile and keeping you from footgunning yourself.
  - chrismorgan 10y ago
    Trying to get ack or ag working properly on Windows is a bit of a pain. Now there’s an equivalent in Rust, rg, and I can get something that’s mostly superior to these others with `cargo install rg`. (OK, so I had to have installed Rust first. But I had done that.)
    - eridius 10y ago
      There does not appear to be a crate called "rg". Are you perhasp referring to https://crates.io/crates/ripgrep https://crates.io/crates/ripgrep?
      - GolDDranks 10y ago
        He definitely is, it's just that "rg" is the binary that compiling ripgrep produces.
      - chrismorgan 10y ago
        Whoops, yep. `cargo install ripgrep`.
    - bluejekyll 10y ago
      You always have to install something or it must exist first. Node, FCC, clang, Ruby, Python... you just need to think of rustup as the the equivalent of getting you rust compiler environment setup.
    - maccard 10y ago
      What's difficult about it? Just use chocolatey.
    - burntsushi 10y ago
      You can also get a Windows binary that I think should Just Work :-) https://github.com/BurntSushi/ripgrep/releases https://github.com/BurntSushi/ripgrep/releases
      - chrismorgan 10y ago
        Hooray for standalone programs!
- munificent 10y ago
  This was a great article. I love step-by-step journeys through what programming actually looks like like this.
jackmott 10y ago
A while back I did a whirlwind tour trying out different parallel looping abstractions in different languages, comparing their performance and ease of use. Rust+Rayon was one of the best. Simple code, fast runtime: https://jackmott.github.io/programming/2016/08/30/think-before-you-parallelize.html https://jackmott.github.io/programming/2016/08/30/think-befo...
- masklinn 10y ago
  The first phrase is objectionable, the Pentium D was a (pretty badly implemented) reaction to AMD's announcement that they were going to bring their workstation and server dual-cores to the desktop. And while Intel technically did manage to take the crown the D was a pretty bad dual-core compared to the A64X2.
  - oldmanjay 10y ago
    From an editorial perspective that digression wouldn't have added a whole lot to the point of the page. What would you be hoping to communicate?
    - jackmott 10y ago
      It might be slightly good to at least mention that AMD and Intel both worked on this at about the same time. I may make this edit next time I push an update for something else. I welcome pedantic critiques because, well..I too am pedantic. :D
      - masklinn 10y ago
        Alternatively, remove that mention and just note that over the last 10 years pretty much all computers have become multi-core, even expanding to multi-core phones?
    - masklinn 10y ago
      > From an editorial perspective that digression wouldn't have added a whole lot to the point of the page. Then remove it altogether and only talk in generalities? The rest of the paragraph doesn't namedrop specific brands.
jsingleton 10y ago
Your homepage crashed my browser. Not sure what you've got going on there. :/ Nice name but how does it relate to your product? I assume it refers to Michael Faraday. I have a bit of history about him and how he compares to Tesla, along with a tour of monuments to him in London here: https://unop.uk/faraday-tour https://unop.uk/faraday-tour
- lmm 10y ago
  Maybe you should switch to a browser written in Rust.
  - jsingleton 10y ago
    Ironically, it is that browser.
    - zaf 10y ago
      IE6? Dude...
  - chrismorgan 10y ago
    Servo isn’t exactly hard to crash at present. (“But hang on, it wasn’t supposed to crash!” I hear you saying. These are controlled, deliberate terminations due to detected logic errors, rather than crashes owing to memory unsafety issues like null pointer dereferencing.)
- ekidd 10y ago
  > Your homepage crashed my browser. Ouch. Sorry to hear that. What browser and version were you running? I'll file an issue. I asked our CEO where the name came from, and he said that Faraday is famous for being an experimentalist, which ties into our predictive modeling. (We also have lots of customers in the residential solar industry.)
  - jsingleton 10y ago
    I wouldn't worry, it's heavily customised. If you tested in all the main browsers you're probably fine. I suspect it was olark, as I can't see anything else obvious. That screenshot image is pretty heavyweight too, I can see it downloading.
leshow 10y ago
Rayon is a great library. It's perfect for spots where you think parallelism might be advantageous but you really want to base that decision on available resources at runtime
acdimalev 10y ago
Result has a FromIterator impl for converting e.g. from a `Vec<Result<a, b>>` to a `Result<Vec<a>, b>`. Is there a reason to prefer the reduce_with method here over FromIterator?

179 points by GolDDranks 10y ago

ekidd 10y ago

Author here. I'm happy to answer questions! I don't have any performance numbers for this loop because it was already very fast. This was just a warm-up exercise before trying to parallelize more complicated loops. My experience with Rust has been interesting. On the one hand, if you compare Rust to Ruby or JavaScript, there really is a "Rust tax"—you have to think about ownership and references and types. But Rust pays me back by making refactoring very easy, and by catching some surprisingly subtle errors at compile time. Plus, I really like Cargo and the crates.io ecosystem. Rust might not be a good match for every project or every programmer, but I'm really enjoying it for low-level, high-performance code.

dkastner 10y ago
A huge payoff that Rust provides vs Ruby or JavaScript is when writing development tools. It saves so much time and trouble to install a binary than to try and get someone to download the right version of Ruby, gem install, etc. Rust, while having its tax, is really great about telling you why your code won't compile and keeping you from footgunning yourself.
- chrismorgan 10y ago
  Trying to get ack or ag working properly on Windows is a bit of a pain. Now there’s an equivalent in Rust, rg, and I can get something that’s mostly superior to these others with `cargo install rg`. (OK, so I had to have installed Rust first. But I had done that.)
  - eridius 10y ago
    There does not appear to be a crate called "rg". Are you perhasp referring to https://crates.io/crates/ripgrep https://crates.io/crates/ripgrep?
    - GolDDranks 10y ago
      He definitely is, it's just that "rg" is the binary that compiling ripgrep produces.
    - chrismorgan 10y ago
      Whoops, yep. `cargo install ripgrep`.
  - bluejekyll 10y ago
    You always have to install something or it must exist first. Node, FCC, clang, Ruby, Python... you just need to think of rustup as the the equivalent of getting you rust compiler environment setup.
  - maccard 10y ago
    What's difficult about it? Just use chocolatey.
  - burntsushi 10y ago
    You can also get a Windows binary that I think should Just Work :-) https://github.com/BurntSushi/ripgrep/releases https://github.com/BurntSushi/ripgrep/releases
    - chrismorgan 10y ago
      Hooray for standalone programs!
munificent 10y ago
This was a great article. I love step-by-step journeys through what programming actually looks like like this.

jackmott 10y ago

A while back I did a whirlwind tour trying out different parallel looping abstractions in different languages, comparing their performance and ease of use. Rust+Rayon was one of the best. Simple code, fast runtime: https://jackmott.github.io/programming/2016/08/30/think-before-you-parallelize.html https://jackmott.github.io/programming/2016/08/30/think-befo...

masklinn 10y ago
The first phrase is objectionable, the Pentium D was a (pretty badly implemented) reaction to AMD's announcement that they were going to bring their workstation and server dual-cores to the desktop. And while Intel technically did manage to take the crown the D was a pretty bad dual-core compared to the A64X2.
- oldmanjay 10y ago
  From an editorial perspective that digression wouldn't have added a whole lot to the point of the page. What would you be hoping to communicate?
  - jackmott 10y ago
    It might be slightly good to at least mention that AMD and Intel both worked on this at about the same time. I may make this edit next time I push an update for something else. I welcome pedantic critiques because, well..I too am pedantic. :D
    - masklinn 10y ago
      Alternatively, remove that mention and just note that over the last 10 years pretty much all computers have become multi-core, even expanding to multi-core phones?
  - masklinn 10y ago
    > From an editorial perspective that digression wouldn't have added a whole lot to the point of the page. Then remove it altogether and only talk in generalities? The rest of the paragraph doesn't namedrop specific brands.

jsingleton 10y ago

Your homepage crashed my browser. Not sure what you've got going on there. :/ Nice name but how does it relate to your product? I assume it refers to Michael Faraday. I have a bit of history about him and how he compares to Tesla, along with a tour of monuments to him in London here: https://unop.uk/faraday-tour https://unop.uk/faraday-tour

lmm 10y ago
Maybe you should switch to a browser written in Rust.
- jsingleton 10y ago
  Ironically, it is that browser.
  - zaf 10y ago
    IE6? Dude...
- chrismorgan 10y ago
  Servo isn’t exactly hard to crash at present. (“But hang on, it wasn’t supposed to crash!” I hear you saying. These are controlled, deliberate terminations due to detected logic errors, rather than crashes owing to memory unsafety issues like null pointer dereferencing.)
ekidd 10y ago
> Your homepage crashed my browser. Ouch. Sorry to hear that. What browser and version were you running? I'll file an issue. I asked our CEO where the name came from, and he said that Faraday is famous for being an experimentalist, which ties into our predictive modeling. (We also have lots of customers in the residential solar industry.)
- jsingleton 10y ago
  I wouldn't worry, it's heavily customised. If you tested in all the main browsers you're probably fine. I suspect it was olark, as I can't see anything else obvious. That screenshot image is pretty heavyweight too, I can see it downloading.

leshow 10y ago

Rayon is a great library. It's perfect for spots where you think parallelism might be advantageous but you really want to base that decision on available resources at runtime

acdimalev 10y ago

Result has a FromIterator impl for converting e.g. from a `Vec<Result<a, b>>` to a `Result<Vec<a>, b>`. Is there a reason to prefer the reduce_with method here over FromIterator?