6 ms·

> But rendering in separate threads turned out to be (unsurprisingly) harder than the way I would do it in C++...It was a bit frustrating to figure out how to a

> But rendering in separate threads turned out to be (unsurprisingly) harder than the way I would do it in C++...It was a bit frustrating to figure out how to accomplish this. Googling yielded a few stack overflow posts with similar questions, and were answered by people basically saying use my crate! Based on some discussion in r/rust (https://www.reddit.com/r/rust/comments/c7t5za/writing_a_small_ray_tracer_in_rust_and_zig/ https://www.reddit.com/r/rust/comments/c7t5za/writing_a_smal...) I went ahead and added a Rayon-based answer to that SO question (https://stackoverflow.com/a/56840441/823869 https://stackoverflow.com/a/56840441/823869). That's been the de facto standard for data parallelism in Rust for the last few years. But the article highlights that discovering the de facto standards is still a challenge for new Rust users -- does anyone know of a well-maintained list of the 10-20 most critical crates that new users should familiarize themselves with after reading The Book? Things like Rayon and lazy_static. The ranked search results at https://crates.io/crates?sort=recent-downloads https://crates.io/crates?sort=recent-downloads are almost good enough, but they include a lot of transitive dependencies that new users shouldn't care about. (I.e. `regex` is a very important crate, but `aho-corasick` is usually only downloaded as a dependency of `regex`.)

steveklabnik 7y ago
https://rust-lang-nursery.github.io/rust-cookbook/ https://rust-lang-nursery.github.io/rust-cookbook/ is sorta kinda this, sorta
- oconnor663 7y ago
  Oh nice.
- OskarS 7y ago
  I will say, the one example they have there which is sort-of analogous to "render each pixel of this image in parallel" is the "draw a julia set" one [0], and it's a very bad way of convincing a C/C++ programmer that Rust is good at this sort of thing. Even if the "loop over all rows in the main thread, adding to the pool a lambda that loops over each column" is somehow optimized in a good data-parallel way (I doubt it compares favorably performance-wise with "#pragma omp parallel for"), the lambdas then push each finished pixel into a channel along with their coordinates. The main thread then has to literally loop through every pixel and read from the channel for each and every one. The natural way to do that in C/C++ is to just write the pixel to the bitmap in each thread. There are no race conditions here (everything is embarassingly parallel), just write the resulting pixel to the bitmap and be done with it. The only reason to have that channel with all that overhead (and that final synchronization on the main thread) is to satisfy the borrow checker, which is just silly in this case. It adds a tremendous amount of overhead just to make it idiomatic Rust. It's true that you can do it the "C++ way" in Rust using unsafe and raw pointers (and there's probably crates that can do the "parallel for" in a way that compares well with OpenMP), but as a graphics programmer who's done a lot of this sort of thing, that piece of code made a very bad first impression of Rust as a high-performance language. EDIT: also, the description is wrong. It says: "ThreadPool::execute receives each pixel as a separate job". No it doesn't, it receives each scanline as a separate job. It might be better if the pool had each pixel as a separate job, but that's not what the code is doing. [0]: https://rust-lang-nursery.github.io/rust-cookbook/concurrency/threads.html#draw-fractal-dispatching-work-to-a-thread-pool https://rust-lang-nursery.github.io/rust-cookbook/concurrenc...
  - gmueckl 7y ago
    To me, this somehow says that most truely interesting problems can only be solved efficiently in Rust using unsafe. Granted, my definition of interesting may be a bit limited, but my initial takeaway here is that Rusts safety story is much more limited than its proponents claim.
    - steveklabnik 7y ago
      To reply to both you and your parent at once, yes, the reason this is in the nursery is that it's not a complete resource yet. I agree that this particular example is bad; it's showing the right pattern for the wrong example. The right way is to use rayon, which is similar to OpenMP in this sense. Each thread would get a pointer to the chunk of the buffer it's operating on. You don't need to write any unsafe code to do this.
      - OskarS 7y ago
        It's good to hear that there's an idiomatic Rust way of doing this that doesn't require unsafe but is still performant. I figured there was, but to an outsider it was not obvious how. This is not actually the first time I've seen this example code. I was wondering how Rust solved this issue (parallel rendering into a single framebuffer) in a way that satisfied the borrow checker, and came across this example. It was a bit disheartening. Rust seems like an excellent language with boundless potential, but as I said, this particular example makes for a very bad first impression. It is also not obvious at all that this is part of the "nursery" and is not an official Rust document (the "Rust Cookbook" makes it seem more polished than maybe it is).
        steveklabnik 7y ago
        > It's good to hear that there's an idiomatic Rust way of doing this that doesn't require unsafe but is still performant. I figured there was, but to an outsider it was not obvious how. Yeah, I mean, the reddit thread way up this comment chain talks about how Rayon should have been used, but the discoverability issue is real. > It is also not obvious at all that this is part of the "nursery" and is not an official Rust document (the "Rust Cookbook" makes it seem more polished than maybe it is). Yep; it's in the URL but not everyone will see that. Like most things, this is basically an accident of history; there was a push to clean this project up and make it an official resource, but before that work was finished, all the contributors had life stuff happen and had to go do something else. So it's in this weird quasi-state where there's a ton of good in there, but also some bad. We'll get to it...
xiphias2 7y ago
I still don't understand why Rayon is not part of the Rust standard library. Rust was created to have easy and safe multithreading on the CPU, and Rayon is the clear winner in this space, for me it feels like something that should be part of Rust.
- 0815test 7y ago
  The Rust stdlib is specifically intended to be as lightweight as possible (while still providing those idiomatic abstractions that might be needed throughout the ecosystem, e.g. std.future) in order to avoid the Python "dead batteries" problem. What the Rust ecosystem is still lacking is a quasi-standard "Rust Platform" of best-practice library components where the community can freely deprecate something when a clearly better replacement comes along. There are a few "community" websites that try to provide guidance wrt. some parts of the Rust ecosystem, but nothing that feels even close to official or consensus-driven.
  - oblio 7y ago
    Rust needs something like Rust dev blessed "extension packs", in my opinion. Sort of like VS Code has extension packs. Basically have some metapackages for common development areas, that pull in community vetted, high quality libraries for certain areas.
    - empath75 7y ago
      why do they need to be dev blessed? You could do something like conda that's basically a distribution.
    - steveklabnik 7y ago
      We have tried to suggest this, but the community has overwhelmingly rejected it, both as an explicit plan, and when people provided such a thing.
      - davemp 7y ago
        That's unfortunate. Whenever I talk about rust with my coworkers, getting crates through IT comes up as a major deal breaker. I imagine cooperate approved "packs" would make it much easier for us rust hobbyists to get some projects started at work in rust. See python: my workplace (and I'm sure many others) is stuck with what's in anaconda for better or worse. The situation is certainly better than pure std.
gbmor 7y ago
> But the article highlights that discovering the de facto standards is still a challenge for new Rust users -- does anyone know of a well-maintained list of the 10-20 most critical crates that new users should familiarize themselves with after reading The Book? I came across this exact thing recently: https://github.com/brson/stdx https://github.com/brson/stdx
- ChrisSD 7y ago
  That seems old, especially as error-chain is no longer maintained so I wouldn't recommend it. [0]: https://users.rust-lang.org/t/error-chain-is-no-longer-maintained/27561 https://users.rust-lang.org/t/error-chain-is-no-longer-maint...
  - gbmor 7y ago
    I'm new to Rust, so I wasn't aware. Thanks!
Anonymous4C54D6 7y ago
Also, https://www.arewewebyet.org/ https://www.arewewebyet.org/ is somehow trying to provide a little bit of this but fails to actually recommend the best option.
- arve0 7y ago
  What is the best option, in your opinion?
arve0 7y ago
I find lib.rs mentioned in the article a good tool. Here on concurrency: https://lib.rs/concurrency https://lib.rs/concurrency

steveklabnik 7y ago

https://rust-lang-nursery.github.io/rust-cookbook/ https://rust-lang-nursery.github.io/rust-cookbook/ is sorta kinda this, sorta

oconnor663 7y ago
Oh nice.
OskarS 7y ago
I will say, the one example they have there which is sort-of analogous to "render each pixel of this image in parallel" is the "draw a julia set" one [0], and it's a very bad way of convincing a C/C++ programmer that Rust is good at this sort of thing. Even if the "loop over all rows in the main thread, adding to the pool a lambda that loops over each column" is somehow optimized in a good data-parallel way (I doubt it compares favorably performance-wise with "#pragma omp parallel for"), the lambdas then push each finished pixel into a channel along with their coordinates. The main thread then has to literally loop through every pixel and read from the channel for each and every one. The natural way to do that in C/C++ is to just write the pixel to the bitmap in each thread. There are no race conditions here (everything is embarassingly parallel), just write the resulting pixel to the bitmap and be done with it. The only reason to have that channel with all that overhead (and that final synchronization on the main thread) is to satisfy the borrow checker, which is just silly in this case. It adds a tremendous amount of overhead just to make it idiomatic Rust. It's true that you can do it the "C++ way" in Rust using unsafe and raw pointers (and there's probably crates that can do the "parallel for" in a way that compares well with OpenMP), but as a graphics programmer who's done a lot of this sort of thing, that piece of code made a very bad first impression of Rust as a high-performance language. EDIT: also, the description is wrong. It says: "ThreadPool::execute receives each pixel as a separate job". No it doesn't, it receives each scanline as a separate job. It might be better if the pool had each pixel as a separate job, but that's not what the code is doing. [0]: https://rust-lang-nursery.github.io/rust-cookbook/concurrency/threads.html#draw-fractal-dispatching-work-to-a-thread-pool https://rust-lang-nursery.github.io/rust-cookbook/concurrenc...
- gmueckl 7y ago
  To me, this somehow says that most truely interesting problems can only be solved efficiently in Rust using unsafe. Granted, my definition of interesting may be a bit limited, but my initial takeaway here is that Rusts safety story is much more limited than its proponents claim.
  - steveklabnik 7y ago
    To reply to both you and your parent at once, yes, the reason this is in the nursery is that it's not a complete resource yet. I agree that this particular example is bad; it's showing the right pattern for the wrong example. The right way is to use rayon, which is similar to OpenMP in this sense. Each thread would get a pointer to the chunk of the buffer it's operating on. You don't need to write any unsafe code to do this.
    - OskarS 7y ago
      It's good to hear that there's an idiomatic Rust way of doing this that doesn't require unsafe but is still performant. I figured there was, but to an outsider it was not obvious how. This is not actually the first time I've seen this example code. I was wondering how Rust solved this issue (parallel rendering into a single framebuffer) in a way that satisfied the borrow checker, and came across this example. It was a bit disheartening. Rust seems like an excellent language with boundless potential, but as I said, this particular example makes for a very bad first impression. It is also not obvious at all that this is part of the "nursery" and is not an official Rust document (the "Rust Cookbook" makes it seem more polished than maybe it is).
      - steveklabnik 7y ago
        > It's good to hear that there's an idiomatic Rust way of doing this that doesn't require unsafe but is still performant. I figured there was, but to an outsider it was not obvious how. Yeah, I mean, the reddit thread way up this comment chain talks about how Rayon should have been used, but the discoverability issue is real. > It is also not obvious at all that this is part of the "nursery" and is not an official Rust document (the "Rust Cookbook" makes it seem more polished than maybe it is). Yep; it's in the URL but not everyone will see that. Like most things, this is basically an accident of history; there was a push to clean this project up and make it an official resource, but before that work was finished, all the contributors had life stuff happen and had to go do something else. So it's in this weird quasi-state where there's a ton of good in there, but also some bad. We'll get to it...

xiphias2 7y ago

I still don't understand why Rayon is not part of the Rust standard library. Rust was created to have easy and safe multithreading on the CPU, and Rayon is the clear winner in this space, for me it feels like something that should be part of Rust.

0815test 7y ago
The Rust stdlib is specifically intended to be as lightweight as possible (while still providing those idiomatic abstractions that might be needed throughout the ecosystem, e.g. std.future) in order to avoid the Python "dead batteries" problem. What the Rust ecosystem is still lacking is a quasi-standard "Rust Platform" of best-practice library components where the community can freely deprecate something when a clearly better replacement comes along. There are a few "community" websites that try to provide guidance wrt. some parts of the Rust ecosystem, but nothing that feels even close to official or consensus-driven.
- oblio 7y ago
  Rust needs something like Rust dev blessed "extension packs", in my opinion. Sort of like VS Code has extension packs. Basically have some metapackages for common development areas, that pull in community vetted, high quality libraries for certain areas.
  - empath75 7y ago
    why do they need to be dev blessed? You could do something like conda that's basically a distribution.
  - steveklabnik 7y ago
    We have tried to suggest this, but the community has overwhelmingly rejected it, both as an explicit plan, and when people provided such a thing.
    - davemp 7y ago
      That's unfortunate. Whenever I talk about rust with my coworkers, getting crates through IT comes up as a major deal breaker. I imagine cooperate approved "packs" would make it much easier for us rust hobbyists to get some projects started at work in rust. See python: my workplace (and I'm sure many others) is stuck with what's in anaconda for better or worse. The situation is certainly better than pure std.

gbmor 7y ago

> But the article highlights that discovering the de facto standards is still a challenge for new Rust users -- does anyone know of a well-maintained list of the 10-20 most critical crates that new users should familiarize themselves with after reading The Book? I came across this exact thing recently: https://github.com/brson/stdx https://github.com/brson/stdx

ChrisSD 7y ago
That seems old, especially as error-chain is no longer maintained so I wouldn't recommend it. [0]: https://users.rust-lang.org/t/error-chain-is-no-longer-maintained/27561 https://users.rust-lang.org/t/error-chain-is-no-longer-maint...
- gbmor 7y ago
  I'm new to Rust, so I wasn't aware. Thanks!

Anonymous4C54D6 7y ago

Also, https://www.arewewebyet.org/ https://www.arewewebyet.org/ is somehow trying to provide a little bit of this but fails to actually recommend the best option.

arve0 7y ago
What is the best option, in your opinion?

arve0 7y ago

I find lib.rs mentioned in the article a good tool. Here on concurrency: https://lib.rs/concurrency https://lib.rs/concurrency