6 ms·

This is pretty cool :) Not sure if author is here, but I had some questions: - How do exceptions work in parallel streaming? How might async complicate things?

This is pretty cool :) Not sure if author is here, but I had some questions: - How do exceptions work in parallel streaming? How might async complicate things? - What kind of logging / instrumentation is available? Can you use trace? - Is there a result set generated at the end like Trio nurseries, or are results meant to be handled by another process? I know nothing about Rust or low-level systems programming, but I should go and learn it (and a bunch of other stuff too).

capableweb 7y ago
author here on HN is yoshuaw (great guy btw!), someone should ping him on Twitter that his project hit HN frontpage and I'm sure he'll find you :) https://twitter.com/yoshuawuyts https://twitter.com/yoshuawuyts
- yoshuaw 7y ago
  Thanks for the kind words; I just got a ping on Twitter and found my way here, almost certainly because of your comment. Thank you (:
leshow 7y ago
- Rust doesn't really have exceptions is the short answer. Errors are represented by the Result type. - You can trace inside of async code, there are libraries like tracing and tracing-futures. - Not sure how to answer your last question. From my understanding of this library is uses the async-std runtime, which comes with a threadpool capable of executing many tasks over a set of processes. Each of the async blocks you write inside a combinator here (.map, .for_each, etc) could hypothetically run on any thread, the idea is each task is running in parallel. That's what I'm gathering anyway
yoshuaw 7y ago
Hi, author here. Glad you've enjoyed the post! To answer your questions: As leshow pointed out, Rust uses the Result type to propagate errors. The way we'll integrate this in parallel-stream is by adding `try_for_each`, `try_fold`, and `try_map` methods. These methods will short-circuit when an error is returned, and have counterparts in `std` and `async-std` already. One tricky part is to get cancellation right, but we already have a design for that, it's just a matter of implementing it now. `async-std` comes with support for the `log` crate built-in, which provides information when a new `Task` (lightweight thread) is spawned. `parallel-stream` doesn't perform any logs on top of that, but it probably should! I'll open an issue. Result sets can be generated through the `collect` method which calls the `FromParallelStream` trait. This can be implemented for various types, but common examples include: `Vec<T>`(generic array), `HashMap`, and `Result<Vec<T>>`. If you're interested in how this exactly works this post by jcoglan goes into incredible detail: https://blog.jcoglan.com/2019/04/22/generic-returns-in-rust https://blog.jcoglan.com/2019/04/22/generic-returns-in-rust I hope this answers your questions!
- yazaddaruvala 7y ago
  Given only the thread would die, what about panics? For example, `.parallel_stream().map(n => 1 / n)` What happens to the `n=0` case? Does parallel_stream propagate the panic, or replace the value with a Result::Err?
  - yoshuaw 7y ago
    It relies on async-std's behavior, which will propagate the panic.

capableweb 7y ago

author here on HN is yoshuaw (great guy btw!), someone should ping him on Twitter that his project hit HN frontpage and I'm sure he'll find you :) https://twitter.com/yoshuawuyts https://twitter.com/yoshuawuyts

yoshuaw 7y ago
Thanks for the kind words; I just got a ping on Twitter and found my way here, almost certainly because of your comment. Thank you (:

leshow 7y ago

- Rust doesn't really have exceptions is the short answer. Errors are represented by the Result type. - You can trace inside of async code, there are libraries like tracing and tracing-futures. - Not sure how to answer your last question. From my understanding of this library is uses the async-std runtime, which comes with a threadpool capable of executing many tasks over a set of processes. Each of the async blocks you write inside a combinator here (.map, .for_each, etc) could hypothetically run on any thread, the idea is each task is running in parallel. That's what I'm gathering anyway

yoshuaw 7y ago

Hi, author here. Glad you've enjoyed the post! To answer your questions: As leshow pointed out, Rust uses the Result type to propagate errors. The way we'll integrate this in parallel-stream is by adding `try_for_each`, `try_fold`, and `try_map` methods. These methods will short-circuit when an error is returned, and have counterparts in `std` and `async-std` already. One tricky part is to get cancellation right, but we already have a design for that, it's just a matter of implementing it now. `async-std` comes with support for the `log` crate built-in, which provides information when a new `Task` (lightweight thread) is spawned. `parallel-stream` doesn't perform any logs on top of that, but it probably should! I'll open an issue. Result sets can be generated through the `collect` method which calls the `FromParallelStream` trait. This can be implemented for various types, but common examples include: `Vec<T>`(generic array), `HashMap`, and `Result<Vec<T>>`. If you're interested in how this exactly works this post by jcoglan goes into incredible detail: https://blog.jcoglan.com/2019/04/22/generic-returns-in-rust https://blog.jcoglan.com/2019/04/22/generic-returns-in-rust I hope this answers your questions!

yazaddaruvala 7y ago
Given only the thread would die, what about panics? For example, `.parallel_stream().map(n => 1 / n)` What happens to the `n=0` case? Does parallel_stream propagate the panic, or replace the value with a Result::Err?
- yoshuaw 7y ago
  It relies on async-std's behavior, which will propagate the panic.