7 ms·

,,I ended up not using the fancy iterator systems (yet?) but the ThreadPool is perfect.'' Please give Rayon iterators another try instead of using thread pool

,,I ended up not using the fancy iterator systems (yet?) but the ThreadPool is perfect.'' Please give Rayon iterators another try instead of using thread pool + message passing, it's one of the best features of Rust.

amelius 8y ago
> Please give Rayon iterators another try (...) it's one of the best features of Rust Isn't it possible to implement these iterators in any language? Why are they a feature of specifically Rust?
- carlosdp 8y ago
  Sure, but when you use a parallel iterator like Rayon in Rust, memory and data-race safety is guaranteed at compile time. And using Rayon is as simple as changing “.iter()” to “.par_iter()” in your code thanks to the trait system. Not that you lose that safety doing the thread pools and such manually, it’s just unnecessary.
  - jandrese 8y ago
    Is there any reason to use .iter() over .par_iter()? Does the latter have higher constant costs that only amortize on lists larger than some value? Or could you just replace .iter() with .par_iter() under the hood and reap the benefits for free? I guess if you are iterating over something declared unsafe? But the compiler should be able to recognize that and fall back to the linear version automatically.
    - steveklabnik 8y ago
      iter is in the standard library, and par_iter isn't, for one. I'm not quite sure about the constant cost thing, but I would imagine that it's true, as it's inherently doing more work.
    - pornel 8y ago
      Rayon has relatively small overhead, but that doesn't mean you can find'n'replace iter() with par_iter(): - not every collection/combination of iterators supports parallel work. For some it doesn't make sense, for some it's just not implemented. - while Rust prevents data races, it can't prevent deadlocks, so you need to be mindful of locks. - .iter() is allowed to mutate state outside of the iterator and use non-thread-safe objects, but .par_iter() isn't (without explicit synchronization primitives). So par_iter() will force you to use more expensive atomic and mutex-locked objects. For data-bound problems most of the time it just works great. I've ran into problems with some networking libraries having global state, which prevented use of par_iter() (but no crashes - caught at compile time!)
    - TheCoelacanth 8y ago
      There are a lot of cases where using par_iter wouldn't be correct even without unsafe. If you are doing anything with side-effects par_iter would execute the side effects out of order.
  - anderspitman 8y ago
    The first time I added rayon to a project and switched my main `iter` to `par_iter` and it compiled with no other changes was decidedly creepy. That particular code didn't run any faster, but the fact that it was still guaranteed to be safe just felt so counter to everything the last 10 years of programming had taught me. Rust is not a perfect language (whatever that means), but more so than any language I've ever used it gives me these strange sort of mind-blowing moments.
- steveklabnik 8y ago
  Rust’s concurrency guarantees make it much easier to not introduce problems. The API can be similar in other languages, but not copied directly, as those guarantees are encoded in the type signature of said API.
  - amelius 8y ago
    Ok, but if one function you call uses unsafe Rust, then it might break down. Which isn't a problem, except if you don't know that it happens. Would Rust warn about that? Just curious.
    - steveklabnik 8y ago
      It would not warn; that's not how unsafe works in Rust. Or rather, if you are calling something unsafe, you have to write "unsafe { }". That's a sort of "warning" in a sense. But if that unsafe code is wrapped in a safe interface, it's that code's responsibility to be correct. Bugs do happen!
      - amelius 8y ago
        Ok, I guess it would be nice then if code which is unsafe but still thread-safe can be marked as such, and Rust should give a warning if used incorrectly. That way, you can parallelize any piece of code without having to think about it.
        steveklabnik 8y ago
        Almost all code ends up going through unsafe, as calling into the OS to do any I/O ends up being unsafe. A warning like this would warn on almost all useful programs, making the warning useless. You already don't need to think about it. If you're not calling unsafe yourself, then it's not your job.
        amelius 8y ago
        What I meant is that sometimes, when you have to write unsafe code, it's easier to make it not thread-safe. Then later on, someone uses that piece of code and tries to parallelize it, and runs into problems. This can be prevented with language support.
- rurban 8y ago
  It would be better to use a real concurrency-safe language like pony or distributed pascal for the fast ones, or e.g. clojure or perl6, ..., and not rust which requires a mutex on all stateful objects. But hype-driven worse is better always succeeds.
  - steveklabnik 8y ago
    Rust does not require a mutex on all stateful objects.
    - rurban 8y ago
      either a mutex or a rwlock. both are a sign of bad concurrency design and dead locks. why dont you fix your docs finally regarding your wrong claim of "fearless" concurrency. they are still many people believing your hype.
      - GolDDranks 8y ago
        Rust also supports atomics, plus plain access of immutable data.
      - steveklabnik 8y ago
        Not even that! It requires that you prevent data races. Scopes threads, for example, can have no synchronization primitives at all! When you do need one, there are far more options than Mutex or RwLock, you have atomics, for example. Someone made a monitor the other day, I think. Really, anything works.
        rurban 8y ago
        That's what I meant, totally concurrency unsafe. A proper and "fearless" concurrency safe language is thread-safe without manual mutex/locks/atomics/semaphores.

amelius 8y ago

> Please give Rayon iterators another try (...) it's one of the best features of Rust Isn't it possible to implement these iterators in any language? Why are they a feature of specifically Rust?

carlosdp 8y ago
Sure, but when you use a parallel iterator like Rayon in Rust, memory and data-race safety is guaranteed at compile time. And using Rayon is as simple as changing “.iter()” to “.par_iter()” in your code thanks to the trait system. Not that you lose that safety doing the thread pools and such manually, it’s just unnecessary.
- jandrese 8y ago
  Is there any reason to use .iter() over .par_iter()? Does the latter have higher constant costs that only amortize on lists larger than some value? Or could you just replace .iter() with .par_iter() under the hood and reap the benefits for free? I guess if you are iterating over something declared unsafe? But the compiler should be able to recognize that and fall back to the linear version automatically.
  - steveklabnik 8y ago
    iter is in the standard library, and par_iter isn't, for one. I'm not quite sure about the constant cost thing, but I would imagine that it's true, as it's inherently doing more work.
  - pornel 8y ago
    Rayon has relatively small overhead, but that doesn't mean you can find'n'replace iter() with par_iter(): - not every collection/combination of iterators supports parallel work. For some it doesn't make sense, for some it's just not implemented. - while Rust prevents data races, it can't prevent deadlocks, so you need to be mindful of locks. - .iter() is allowed to mutate state outside of the iterator and use non-thread-safe objects, but .par_iter() isn't (without explicit synchronization primitives). So par_iter() will force you to use more expensive atomic and mutex-locked objects. For data-bound problems most of the time it just works great. I've ran into problems with some networking libraries having global state, which prevented use of par_iter() (but no crashes - caught at compile time!)
  - TheCoelacanth 8y ago
    There are a lot of cases where using par_iter wouldn't be correct even without unsafe. If you are doing anything with side-effects par_iter would execute the side effects out of order.
- anderspitman 8y ago
  The first time I added rayon to a project and switched my main `iter` to `par_iter` and it compiled with no other changes was decidedly creepy. That particular code didn't run any faster, but the fact that it was still guaranteed to be safe just felt so counter to everything the last 10 years of programming had taught me. Rust is not a perfect language (whatever that means), but more so than any language I've ever used it gives me these strange sort of mind-blowing moments.
steveklabnik 8y ago
Rust’s concurrency guarantees make it much easier to not introduce problems. The API can be similar in other languages, but not copied directly, as those guarantees are encoded in the type signature of said API.
- amelius 8y ago
  Ok, but if one function you call uses unsafe Rust, then it might break down. Which isn't a problem, except if you don't know that it happens. Would Rust warn about that? Just curious.
  - steveklabnik 8y ago
    It would not warn; that's not how unsafe works in Rust. Or rather, if you are calling something unsafe, you have to write "unsafe { }". That's a sort of "warning" in a sense. But if that unsafe code is wrapped in a safe interface, it's that code's responsibility to be correct. Bugs do happen!
    - amelius 8y ago
      Ok, I guess it would be nice then if code which is unsafe but still thread-safe can be marked as such, and Rust should give a warning if used incorrectly. That way, you can parallelize any piece of code without having to think about it.
      - steveklabnik 8y ago
        Almost all code ends up going through unsafe, as calling into the OS to do any I/O ends up being unsafe. A warning like this would warn on almost all useful programs, making the warning useless. You already don't need to think about it. If you're not calling unsafe yourself, then it's not your job.
        amelius 8y ago
        What I meant is that sometimes, when you have to write unsafe code, it's easier to make it not thread-safe. Then later on, someone uses that piece of code and tries to parallelize it, and runs into problems. This can be prevented with language support.
rurban 8y ago
It would be better to use a real concurrency-safe language like pony or distributed pascal for the fast ones, or e.g. clojure or perl6, ..., and not rust which requires a mutex on all stateful objects. But hype-driven worse is better always succeeds.
- steveklabnik 8y ago
  Rust does not require a mutex on all stateful objects.
  - rurban 8y ago
    either a mutex or a rwlock. both are a sign of bad concurrency design and dead locks. why dont you fix your docs finally regarding your wrong claim of "fearless" concurrency. they are still many people believing your hype.
    - GolDDranks 8y ago
      Rust also supports atomics, plus plain access of immutable data.
    - steveklabnik 8y ago
      Not even that! It requires that you prevent data races. Scopes threads, for example, can have no synchronization primitives at all! When you do need one, there are far more options than Mutex or RwLock, you have atomics, for example. Someone made a monitor the other day, I think. Really, anything works.
      - rurban 8y ago
        That's what I meant, totally concurrency unsafe. A proper and "fearless" concurrency safe language is thread-safe without manual mutex/locks/atomics/semaphores.