4 ms·

Parallel-Stream

80 points by lukastyrychtr 7y ago

yingw787 7y ago
This is pretty cool :) Not sure if author is here, but I had some questions: - How do exceptions work in parallel streaming? How might async complicate things? - What kind of logging / instrumentation is available? Can you use trace? - Is there a result set generated at the end like Trio nurseries, or are results meant to be handled by another process? I know nothing about Rust or low-level systems programming, but I should go and learn it (and a bunch of other stuff too).
- capableweb 7y ago
  author here on HN is yoshuaw (great guy btw!), someone should ping him on Twitter that his project hit HN frontpage and I'm sure he'll find you :) https://twitter.com/yoshuawuyts https://twitter.com/yoshuawuyts
  - yoshuaw 7y ago
    Thanks for the kind words; I just got a ping on Twitter and found my way here, almost certainly because of your comment. Thank you (:
- leshow 7y ago
  - Rust doesn't really have exceptions is the short answer. Errors are represented by the Result type. - You can trace inside of async code, there are libraries like tracing and tracing-futures. - Not sure how to answer your last question. From my understanding of this library is uses the async-std runtime, which comes with a threadpool capable of executing many tasks over a set of processes. Each of the async blocks you write inside a combinator here (.map, .for_each, etc) could hypothetically run on any thread, the idea is each task is running in parallel. That's what I'm gathering anyway
- yoshuaw 7y ago
  Hi, author here. Glad you've enjoyed the post! To answer your questions: As leshow pointed out, Rust uses the Result type to propagate errors. The way we'll integrate this in parallel-stream is by adding `try_for_each`, `try_fold`, and `try_map` methods. These methods will short-circuit when an error is returned, and have counterparts in `std` and `async-std` already. One tricky part is to get cancellation right, but we already have a design for that, it's just a matter of implementing it now. `async-std` comes with support for the `log` crate built-in, which provides information when a new `Task` (lightweight thread) is spawned. `parallel-stream` doesn't perform any logs on top of that, but it probably should! I'll open an issue. Result sets can be generated through the `collect` method which calls the `FromParallelStream` trait. This can be implemented for various types, but common examples include: `Vec<T>`(generic array), `HashMap`, and `Result<Vec<T>>`. If you're interested in how this exactly works this post by jcoglan goes into incredible detail: https://blog.jcoglan.com/2019/04/22/generic-returns-in-rust https://blog.jcoglan.com/2019/04/22/generic-returns-in-rust I hope this answers your questions!
  - yazaddaruvala 7y ago
    Given only the thread would die, what about panics? For example, `.parallel_stream().map(n => 1 / n)` What happens to the `n=0` case? Does parallel_stream propagate the panic, or replace the value with a Result::Err?
    - yoshuaw 7y ago
      It relies on async-std's behavior, which will propagate the panic.
dnpp123 7y ago
This looks cool. Something I really never get with async is the executor: - if I use this do I have do use tokio or async-std ? none ? is the code assuming an executor ? - e.g can I use this library with, let's say, hyper ? tide ?
- yoshuaw 7y ago
  This library assumes `async-std`; the way it parallelizes work is by spawning tasks on an executor, so by design it needs to have access to one. Without that we couldn't provide the API we do. I haven't tested it with `hyper`, but it should soon be compatible with `tide`(assuming `async-h1/async-std` is used as the backend).

80 points by lukastyrychtr 7y ago

This is pretty cool :) Not sure if author is here, but I had some questions: - How do exceptions work in parallel streaming? How might async complicate things? - What kind of logging / instrumentation is available? Can you use trace? - Is there a result set generated at the end like Trio nurseries, or are results meant to be handled by another process? I know nothing about Rust or low-level systems programming, but I should go and learn it (and a bunch of other stuff too).

capableweb 7y ago
author here on HN is yoshuaw (great guy btw!), someone should ping him on Twitter that his project hit HN frontpage and I'm sure he'll find you :) https://twitter.com/yoshuawuyts https://twitter.com/yoshuawuyts
- yoshuaw 7y ago
  Thanks for the kind words; I just got a ping on Twitter and found my way here, almost certainly because of your comment. Thank you (:
leshow 7y ago
- Rust doesn't really have exceptions is the short answer. Errors are represented by the Result type. - You can trace inside of async code, there are libraries like tracing and tracing-futures. - Not sure how to answer your last question. From my understanding of this library is uses the async-std runtime, which comes with a threadpool capable of executing many tasks over a set of processes. Each of the async blocks you write inside a combinator here (.map, .for_each, etc) could hypothetically run on any thread, the idea is each task is running in parallel. That's what I'm gathering anyway
yoshuaw 7y ago
Hi, author here. Glad you've enjoyed the post! To answer your questions: As leshow pointed out, Rust uses the Result type to propagate errors. The way we'll integrate this in parallel-stream is by adding `try_for_each`, `try_fold`, and `try_map` methods. These methods will short-circuit when an error is returned, and have counterparts in `std` and `async-std` already. One tricky part is to get cancellation right, but we already have a design for that, it's just a matter of implementing it now. `async-std` comes with support for the `log` crate built-in, which provides information when a new `Task` (lightweight thread) is spawned. `parallel-stream` doesn't perform any logs on top of that, but it probably should! I'll open an issue. Result sets can be generated through the `collect` method which calls the `FromParallelStream` trait. This can be implemented for various types, but common examples include: `Vec<T>`(generic array), `HashMap`, and `Result<Vec<T>>`. If you're interested in how this exactly works this post by jcoglan goes into incredible detail: https://blog.jcoglan.com/2019/04/22/generic-returns-in-rust https://blog.jcoglan.com/2019/04/22/generic-returns-in-rust I hope this answers your questions!
- yazaddaruvala 7y ago
  Given only the thread would die, what about panics? For example, `.parallel_stream().map(n => 1 / n)` What happens to the `n=0` case? Does parallel_stream propagate the panic, or replace the value with a Result::Err?
  - yoshuaw 7y ago
    It relies on async-std's behavior, which will propagate the panic.

dnpp123 7y ago

This looks cool. Something I really never get with async is the executor: - if I use this do I have do use tokio or async-std ? none ? is the code assuming an executor ? - e.g can I use this library with, let's say, hyper ? tide ?

yoshuaw 7y ago
This library assumes `async-std`; the way it parallelizes work is by spawning tasks on an executor, so by design it needs to have access to one. Without that we couldn't provide the API we do. I haven't tested it with `hyper`, but it should soon be compatible with `tide`(assuming `async-h1/async-std` is used as the backend).