5 ms·

Stateful Multi-Stream Processing in Python with Wallaroo

59 points by jtfmumm 9y ago

jtfmumm 9y ago
I'm the author of this post and I'm happy to answer questions here.
- TuringNYC 9y ago
  Thank you for the informative blog post, this is the type of well-written detailed technical post with solid use case (rather than vague sales-ey ones) that pushes me to give a chance to new software. I had never heard of Wallaroo before your post, but quite impressed. If this works, this can solve a lot of problems. I like the price tiering and great to see a NYC company amidst a sea of Valley ones.
  - jtfmumm 9y ago
    Thanks! That's great to hear.
alextheparrot 9y ago
Couple questions: 1. How does this compare to other distributed streaming systems in terms of latency, throughput, fault tolerance, and message processing guarantees? 2. Why Pony? I’m not super familiar with the language, does it give you any major gains? 3. How does this handle shared aggregations - i.e. I have one stream updating a table constantly and another that reads from the table. This is useful as it decouples the two event streams, meaning my reading job can have better latency guarantees with the tradeoff of possible having stale data. I didn’t see how I can link together two independently deployed streaming states, unless the idea is any stream using some state will be declared in one place. 4. Does this support transactional event processing, wherein my originals HTTP call is returned with a success or failure, or does it just ack the event was received, even if the event fails downstream of the source? 5. Are there any major, novel differences in this stream processing system versus the next that I’m not aware of? I didn’t see anything too unique in this post.
- strmpnk 9y ago
  While I'm not affiliated with Wallaroo Labs, I've been following their work. One of the best things to check out is their nice collection of blog articles. 1: https://blog.wallaroolabs.com/2017/03/hello-wallaroo/ https://blog.wallaroolabs.com/2017/03/hello-wallaroo/ has some figures around latencies their working with and https://blog.wallaroolabs.com/2017/10/measuring-correctness-of-state-in-a-distributed-system/ https://blog.wallaroolabs.com/2017/10/measuring-correctness-... talks about how they test their system under faults 2: https://blog.wallaroolabs.com/2017/10/why-we-used-pony-to-write-wallaroo/ https://blog.wallaroolabs.com/2017/10/why-we-used-pony-to-wr... For the rest, I better leave it to them to give the most up to date answers on those as the project is moving fast.
- spooneybarger 9y ago
  hi! I work at Wallaroo Labs. strmpnk had good content for answers for 1 and 2. thank you for that! re: 3. The one stream updates/one stream reads is detailed in the "MarketSpread" example in the post. I suspect the key to your question is "two independently deployed streaming states". Could you elaborate on what you mean by that? I think if I understood that better, I could give a good answer to #3. 4. Currently all pipelines have a source and a sink. In your HTTP call example, that would most likely be a combination source/sink where we want to have both the input and some output result flow back out. That currently isn't supported in Wallaroo although we have had discussions about what that might mean. You could do transactional processing where you receive an event and then reply but at the moment, the reply would need to go out over the sink which is a different channel than the source. I'd be really interested in working with folks who could benefit from a combination source/sink approach. We prefer to add additional functionality after working through use cases with interested parties. We find that leads to better results than our designing in a vacuum. 5. Everyone has a different definition of novel. In my view, most streaming systems have left the handling of application state in the hands of the programmer. You can keep it in memory for speed but then you have to manage the resilience of that in-memory state. Wallaroo makes that in-memory state a first class citizen. It's not detailed in this post, but Wallaroo can provide resilience for you state so that if there is a failure we can bring that state back. Currently that involves using a write ahead log that is stored on the local filesystem. We are working on replicating that log to other nodes in the cluster to provide additional resilience. If i was to hightlight one feature to answer your question about novelty, that would be it. We have a lot more content on the blog that might answer your question better. The post in question is very much a "how do I do X" sort of post rather than a "Why Wallaroo?" (of which there are a couple on the blog).
stocktech 9y ago
As someone just getting into stream processing, does anyone have any resources for comparing frameworks/systems? When Apache has multiple projects that sound like they do the same thing, I don't even know where to start.
- spooneybarger 9y ago
  It's a very confusing space to start getting into. I'd be happy to step outside of my role as a principal at Wallaroo Labs and have an email conversation to discuss what your use cases are and what tools you should consider looking at. sean@wallaroolabs.com
- TheCowboy 9y ago
  I've been trying to learn about all of this stuff over the last couple of weeks, and agree that it isn't obvious. Designing Data-Intensive Applications by Kleppmann has been helpful. It doesn't cover every framework, but I think it helps explain where a lot of the pieces fit together and when you might want to use some of them. I've also found it useful to find podcasts that explain specific projects, such as Apache Kafka, and listen to them when I'm running.

59 points by jtfmumm 9y ago

jtfmumm 9y ago

I'm the author of this post and I'm happy to answer questions here.

TuringNYC 9y ago
Thank you for the informative blog post, this is the type of well-written detailed technical post with solid use case (rather than vague sales-ey ones) that pushes me to give a chance to new software. I had never heard of Wallaroo before your post, but quite impressed. If this works, this can solve a lot of problems. I like the price tiering and great to see a NYC company amidst a sea of Valley ones.
- jtfmumm 9y ago
  Thanks! That's great to hear.

alextheparrot 9y ago

Couple questions: 1. How does this compare to other distributed streaming systems in terms of latency, throughput, fault tolerance, and message processing guarantees? 2. Why Pony? I’m not super familiar with the language, does it give you any major gains? 3. How does this handle shared aggregations - i.e. I have one stream updating a table constantly and another that reads from the table. This is useful as it decouples the two event streams, meaning my reading job can have better latency guarantees with the tradeoff of possible having stale data. I didn’t see how I can link together two independently deployed streaming states, unless the idea is any stream using some state will be declared in one place. 4. Does this support transactional event processing, wherein my originals HTTP call is returned with a success or failure, or does it just ack the event was received, even if the event fails downstream of the source? 5. Are there any major, novel differences in this stream processing system versus the next that I’m not aware of? I didn’t see anything too unique in this post.

strmpnk 9y ago
While I'm not affiliated with Wallaroo Labs, I've been following their work. One of the best things to check out is their nice collection of blog articles. 1: https://blog.wallaroolabs.com/2017/03/hello-wallaroo/ https://blog.wallaroolabs.com/2017/03/hello-wallaroo/ has some figures around latencies their working with and https://blog.wallaroolabs.com/2017/10/measuring-correctness-of-state-in-a-distributed-system/ https://blog.wallaroolabs.com/2017/10/measuring-correctness-... talks about how they test their system under faults 2: https://blog.wallaroolabs.com/2017/10/why-we-used-pony-to-write-wallaroo/ https://blog.wallaroolabs.com/2017/10/why-we-used-pony-to-wr... For the rest, I better leave it to them to give the most up to date answers on those as the project is moving fast.
spooneybarger 9y ago
hi! I work at Wallaroo Labs. strmpnk had good content for answers for 1 and 2. thank you for that! re: 3. The one stream updates/one stream reads is detailed in the "MarketSpread" example in the post. I suspect the key to your question is "two independently deployed streaming states". Could you elaborate on what you mean by that? I think if I understood that better, I could give a good answer to #3. 4. Currently all pipelines have a source and a sink. In your HTTP call example, that would most likely be a combination source/sink where we want to have both the input and some output result flow back out. That currently isn't supported in Wallaroo although we have had discussions about what that might mean. You could do transactional processing where you receive an event and then reply but at the moment, the reply would need to go out over the sink which is a different channel than the source. I'd be really interested in working with folks who could benefit from a combination source/sink approach. We prefer to add additional functionality after working through use cases with interested parties. We find that leads to better results than our designing in a vacuum. 5. Everyone has a different definition of novel. In my view, most streaming systems have left the handling of application state in the hands of the programmer. You can keep it in memory for speed but then you have to manage the resilience of that in-memory state. Wallaroo makes that in-memory state a first class citizen. It's not detailed in this post, but Wallaroo can provide resilience for you state so that if there is a failure we can bring that state back. Currently that involves using a write ahead log that is stored on the local filesystem. We are working on replicating that log to other nodes in the cluster to provide additional resilience. If i was to hightlight one feature to answer your question about novelty, that would be it. We have a lot more content on the blog that might answer your question better. The post in question is very much a "how do I do X" sort of post rather than a "Why Wallaroo?" (of which there are a couple on the blog).

stocktech 9y ago

As someone just getting into stream processing, does anyone have any resources for comparing frameworks/systems? When Apache has multiple projects that sound like they do the same thing, I don't even know where to start.

spooneybarger 9y ago
It's a very confusing space to start getting into. I'd be happy to step outside of my role as a principal at Wallaroo Labs and have an email conversation to discuss what your use cases are and what tools you should consider looking at. sean@wallaroolabs.com
TheCowboy 9y ago
I've been trying to learn about all of this stuff over the last couple of weeks, and agree that it isn't obvious. Designing Data-Intensive Applications by Kleppmann has been helpful. It doesn't cover every framework, but I think it helps explain where a lot of the pieces fit together and when you might want to use some of them. I've also found it useful to find podcasts that explain specific projects, such as Apache Kafka, and listen to them when I'm running.