4 ms·

Hi everyone, I'm the VP of Engineering over at Wallaroo Labs and the author of this post. I'll be keeping an eye on the comments here and I'm happy to answer y

Hi everyone, I'm the VP of Engineering over at Wallaroo Labs and the author of this post. I'll be keeping an eye on the comments here and I'm happy to answer your questions as best I can. -Sean-

haolez 8y ago
How does this compare with Apache Kafka? Genuine interest :)
- spooneybarger 8y ago
  There are a couple aspects of Kafka. The primary aspect that most people use it for is as a distributed log. If you are familiar with message buses like RabbitMQ, a distributed log can serve the same "make data easily available to a variety of consumers role" while also having some interesting properties about how long the data lives in the log etc. Wallaroo applications are often consumers of data that is read off of Kafka. Wallaroo applications can also get data from a variety of other sources including message queues, databases, files etc. Wallaroo is more like Kafka streams than Kafka. Wallaroo is a processing engine, taking data/events an event at time and doing running computations against it. Those computations could be creating aggregations, monitoring as part of an alerting system or a variety of other cases. You can imagine Wallaroo as being in the same space as Spark, Flink, Storm etc. A big difference is that Wallaroo API targets Python users. Wallaroo has an embedded Python interpreter that allows the developer to use any of the standard Python libraries: Pandas, Numpy, whatever. Another key diffence is in how we do clustering. Wallaroo was designed to make it easy to create ad-hoc processing clusters. We have a use case where based on incoming data arriving (large csv file over http), we can spin up a Wallaroo cluster (powered by Pulumi!) in AWS and have a number of Wallaroo workers that process the data. That sort of "on the fly" clustering is something that many tools have a hard time with. You can see it in action as part of our "Scale Independence" video at https://vimeo.com/270509076 https://vimeo.com/270509076. Not sure if that answers you question. Hopefully it does at least a little bit.

Hi everyone, I'm the VP of Engineering over at Wallaroo Labs and the author of this post. I'll be keeping an eye on the comments here and I'm happy to answer your questions as best I can. -Sean-

haolez 8y ago

How does this compare with Apache Kafka? Genuine interest :)

spooneybarger 8y ago
There are a couple aspects of Kafka. The primary aspect that most people use it for is as a distributed log. If you are familiar with message buses like RabbitMQ, a distributed log can serve the same "make data easily available to a variety of consumers role" while also having some interesting properties about how long the data lives in the log etc. Wallaroo applications are often consumers of data that is read off of Kafka. Wallaroo applications can also get data from a variety of other sources including message queues, databases, files etc. Wallaroo is more like Kafka streams than Kafka. Wallaroo is a processing engine, taking data/events an event at time and doing running computations against it. Those computations could be creating aggregations, monitoring as part of an alerting system or a variety of other cases. You can imagine Wallaroo as being in the same space as Spark, Flink, Storm etc. A big difference is that Wallaroo API targets Python users. Wallaroo has an embedded Python interpreter that allows the developer to use any of the standard Python libraries: Pandas, Numpy, whatever. Another key diffence is in how we do clustering. Wallaroo was designed to make it easy to create ad-hoc processing clusters. We have a use case where based on incoming data arriving (large csv file over http), we can spin up a Wallaroo cluster (powered by Pulumi!) in AWS and have a number of Wallaroo workers that process the data. That sort of "on the fly" clustering is something that many tools have a hard time with. You can see it in action as part of our "Scale Independence" video at https://vimeo.com/270509076 https://vimeo.com/270509076. Not sure if that answers you question. Hopefully it does at least a little bit.