3 ms·

Question for spooneybarger or anyone else at Walaroo Labs. How do you see this comparing to something like Dask? Would it compete with Dask, or be able to some

Question for spooneybarger or anyone else at Walaroo Labs. How do you see this comparing to something like Dask? Would it compete with Dask, or be able to somehow work together with it? Dask seems to let you write idiomatic Python code and not even think about splitting, joining, etc... and it builds the pipeline automagically by introspecting the AST.

spooneybarger 9y ago
We are actively looking at how we can better handle use cases like Dask handles. A couple folks we are working with use Dask heavily and are looking to switch. They are far more expert in Dask than we are so I don't want to shoot my mouth of. The issues they've general discussed with us are around performance and scaling problems they have had. We are still actively learning from them and hope to have a first version to start addressing those use cases around the end of this year. Ideally middle of Decemeber. Dask is very "batch" oriented, which as I said above, is something we are in the process of adding to Wallaroo. Wallaroo is very stream processing oriented. Wallaroo's strength are working with stateful, event-by-event applications. If you were to take word count as an example. Dask would be great if you had a body of text in files or whatnot that you needed to count. There's a beginning and end to that task. Count the words in this text. Wallaroo would shine if you had a stream of never ending text, like twitter's trending topics. That's a very coarse outline of a couple of differences. While we are working with clients to help them move off of Dask by adding that functionality to Dask, I also think that if you wanted to, you could use Wallaroo along side a more batch oriented system like Dask. Stream processing and batch processing are complementary. A number of technologies (us included) are looking to unify them. Why? Well, there's a lot of operational overhead to running a batch system and a streaming system. A lot of folks would like to run a single system that works well for both. I hope that answers your question.
- ericfrederich 9y ago
  Thanks, I think it does. I did a quick search to see if or how Dask handles streams. It seems trivial to add queues to dask. https://www.youtube.com/watch?v=hfmwXSeM8pk https://www.youtube.com/watch?v=hfmwXSeM8pk I've never actually played with Dask at all for even batch processing let alone stream. If I ever get some free time I'll try to implement Word Count in Dask and see how the two codes compare.
- happy-go-lucky 9y ago
  > Stream processing and batch processing are complimentary Must be complementary, I think.
  - spooneybarger 9y ago
    Indeed. I regularly mix the two up. Thanks. Editing to fix.

spooneybarger 9y ago

We are actively looking at how we can better handle use cases like Dask handles. A couple folks we are working with use Dask heavily and are looking to switch. They are far more expert in Dask than we are so I don't want to shoot my mouth of. The issues they've general discussed with us are around performance and scaling problems they have had. We are still actively learning from them and hope to have a first version to start addressing those use cases around the end of this year. Ideally middle of Decemeber. Dask is very "batch" oriented, which as I said above, is something we are in the process of adding to Wallaroo. Wallaroo is very stream processing oriented. Wallaroo's strength are working with stateful, event-by-event applications. If you were to take word count as an example. Dask would be great if you had a body of text in files or whatnot that you needed to count. There's a beginning and end to that task. Count the words in this text. Wallaroo would shine if you had a stream of never ending text, like twitter's trending topics. That's a very coarse outline of a couple of differences. While we are working with clients to help them move off of Dask by adding that functionality to Dask, I also think that if you wanted to, you could use Wallaroo along side a more batch oriented system like Dask. Stream processing and batch processing are complementary. A number of technologies (us included) are looking to unify them. Why? Well, there's a lot of operational overhead to running a batch system and a streaming system. A lot of folks would like to run a single system that works well for both. I hope that answers your question.

ericfrederich 9y ago
Thanks, I think it does. I did a quick search to see if or how Dask handles streams. It seems trivial to add queues to dask. https://www.youtube.com/watch?v=hfmwXSeM8pk https://www.youtube.com/watch?v=hfmwXSeM8pk I've never actually played with Dask at all for even batch processing let alone stream. If I ever get some free time I'll try to implement Word Count in Dask and see how the two codes compare.
happy-go-lucky 9y ago
> Stream processing and batch processing are complimentary Must be complementary, I think.
- spooneybarger 9y ago
  Indeed. I regularly mix the two up. Thanks. Editing to fix.