4 ms·

Wallaroo goes full Apache 2.0

69 points by spooneybarger 8y ago

spooneybarger 8y ago
Hi everyone, I'm the VP of Engineering over at Wallaroo Labs and the author of this post. I'll be keeping an eye on the comments here and I'm happy to answer your questions as best I can. -Sean-
- haolez 8y ago
  How does this compare with Apache Kafka? Genuine interest :)
  - spooneybarger 8y ago
    There are a couple aspects of Kafka. The primary aspect that most people use it for is as a distributed log. If you are familiar with message buses like RabbitMQ, a distributed log can serve the same "make data easily available to a variety of consumers role" while also having some interesting properties about how long the data lives in the log etc. Wallaroo applications are often consumers of data that is read off of Kafka. Wallaroo applications can also get data from a variety of other sources including message queues, databases, files etc. Wallaroo is more like Kafka streams than Kafka. Wallaroo is a processing engine, taking data/events an event at time and doing running computations against it. Those computations could be creating aggregations, monitoring as part of an alerting system or a variety of other cases. You can imagine Wallaroo as being in the same space as Spark, Flink, Storm etc. A big difference is that Wallaroo API targets Python users. Wallaroo has an embedded Python interpreter that allows the developer to use any of the standard Python libraries: Pandas, Numpy, whatever. Another key diffence is in how we do clustering. Wallaroo was designed to make it easy to create ad-hoc processing clusters. We have a use case where based on incoming data arriving (large csv file over http), we can spin up a Wallaroo cluster (powered by Pulumi!) in AWS and have a number of Wallaroo workers that process the data. That sort of "on the fly" clustering is something that many tools have a hard time with. You can see it in action as part of our "Scale Independence" video at https://vimeo.com/270509076 https://vimeo.com/270509076. Not sure if that answers you question. Hopefully it does at least a little bit.
le-mark 8y ago
The features we kept under our enterprise license were scaling related features. If you wanted to use Wallaroo on more than 24 CPUs, you needed to pay us money. Upon reflection, that was a mistake. Wallaroo is about making scaling easy. We locked the core value proposition of the product behind a commercial license and tried to build an open source community. I'm curious about this analysis; was the dual license nature really the problem, or is the "market" not as large as you may have thought, in reality? What is the business model now? I've really benefited from your technical blog the past few years, I personally am not a wallaroo user though. Good luck!
- spooneybarger 8y ago
  Anything I can do to get you to become a Wallaroo user? <I at least have to ask> Also, thank you for the kind words about our blog. It means a lot. We put a lot of effort into it and we hope that it both drives Wallaroo usage but is also valuable as a general technical resource. > was the dual license nature really the problem, or is the "market" not as large as you may have thought, in reality? The dual license was definitely a problem if the goal is to get usage. Would the people who balked at the license have paid us money? Maybe not but, they could have been valuable as members of the community. Could have helped us learn about new and interesting Wallaroo use cases. Could have acted as social proof for Wallaroo. We get none of those things if they are scared away the license. And I personally heard from a lot of folks who balked at the license. I was talking to an engineer from very large company who wanted to use Wallaroo and would have made great press for us but, they couldn't without getting legal involved and the project died there. That was the day that I personally started thinking seriously about the license and if we had done the right thing. It's hard to be out there telling people the real value of the project is in the ease of scaling but making payment required for that. Ended a lot of conversations. > is the "market" not as large as you may have thought Well, we went into this with "there's a small market now, but we see it growing". So, I would say no. We knew the Python streaming/event-driven application market was small compared to the Java one right now. But, we also see it growing at a steady rate (with signs of a big big jump at some point). > What is the business model now? Things we've found that people are happy to pay for - Support/license that involves us giving them the same assurances of "someone being there" that they get with proprietary software - "Managed service", we'll install and run your Wallaroo applications for you inside your AWS/GCP/Azure accounts. - Enterprise version. We plan on creating products that live around Wallaroo that aren't of interest to the open source community but would be valuable to enterprise users. In the end, what's most important to us right now is getting closer to more users and using what we learn from them to accelerate our improvement of Wallaroo. The more users we have, the more chances we have learn, and the more chances we have to continue to improve our product market fit.
haney 8y ago
Very excited to see this, I've been intrigued by it over the last few months but have avoided it because I was concerned if I built a successful project with it I'd eventually reach a hard cliff in terms of pricing. It's especially difficult to weigh cost benefit of an architecture with a new framework that has specific resource based pricing when you don't have a way of estimating resource usage because you've never used the framework.
- spooneybarger 8y ago
  You aren't alone in that concern. We heard that from a lot of folks. And I agree with you. When I stopped thinking as "VP of Engineering at Wallaroo" and started thinking about how I would evaluate Wallaroo if I wasn't with the company, I ended up with the problem you are describing. Basically: "why would I spend time evaluating this thing when there's that cliff that I don't know when I'll hit it".

69 points by spooneybarger 8y ago

Hi everyone, I'm the VP of Engineering over at Wallaroo Labs and the author of this post. I'll be keeping an eye on the comments here and I'm happy to answer your questions as best I can. -Sean-

haolez 8y ago
How does this compare with Apache Kafka? Genuine interest :)
- spooneybarger 8y ago
  There are a couple aspects of Kafka. The primary aspect that most people use it for is as a distributed log. If you are familiar with message buses like RabbitMQ, a distributed log can serve the same "make data easily available to a variety of consumers role" while also having some interesting properties about how long the data lives in the log etc. Wallaroo applications are often consumers of data that is read off of Kafka. Wallaroo applications can also get data from a variety of other sources including message queues, databases, files etc. Wallaroo is more like Kafka streams than Kafka. Wallaroo is a processing engine, taking data/events an event at time and doing running computations against it. Those computations could be creating aggregations, monitoring as part of an alerting system or a variety of other cases. You can imagine Wallaroo as being in the same space as Spark, Flink, Storm etc. A big difference is that Wallaroo API targets Python users. Wallaroo has an embedded Python interpreter that allows the developer to use any of the standard Python libraries: Pandas, Numpy, whatever. Another key diffence is in how we do clustering. Wallaroo was designed to make it easy to create ad-hoc processing clusters. We have a use case where based on incoming data arriving (large csv file over http), we can spin up a Wallaroo cluster (powered by Pulumi!) in AWS and have a number of Wallaroo workers that process the data. That sort of "on the fly" clustering is something that many tools have a hard time with. You can see it in action as part of our "Scale Independence" video at https://vimeo.com/270509076 https://vimeo.com/270509076. Not sure if that answers you question. Hopefully it does at least a little bit.

le-mark 8y ago

The features we kept under our enterprise license were scaling related features. If you wanted to use Wallaroo on more than 24 CPUs, you needed to pay us money. Upon reflection, that was a mistake. Wallaroo is about making scaling easy. We locked the core value proposition of the product behind a commercial license and tried to build an open source community. I'm curious about this analysis; was the dual license nature really the problem, or is the "market" not as large as you may have thought, in reality? What is the business model now? I've really benefited from your technical blog the past few years, I personally am not a wallaroo user though. Good luck!

spooneybarger 8y ago
Anything I can do to get you to become a Wallaroo user? <I at least have to ask> Also, thank you for the kind words about our blog. It means a lot. We put a lot of effort into it and we hope that it both drives Wallaroo usage but is also valuable as a general technical resource. > was the dual license nature really the problem, or is the "market" not as large as you may have thought, in reality? The dual license was definitely a problem if the goal is to get usage. Would the people who balked at the license have paid us money? Maybe not but, they could have been valuable as members of the community. Could have helped us learn about new and interesting Wallaroo use cases. Could have acted as social proof for Wallaroo. We get none of those things if they are scared away the license. And I personally heard from a lot of folks who balked at the license. I was talking to an engineer from very large company who wanted to use Wallaroo and would have made great press for us but, they couldn't without getting legal involved and the project died there. That was the day that I personally started thinking seriously about the license and if we had done the right thing. It's hard to be out there telling people the real value of the project is in the ease of scaling but making payment required for that. Ended a lot of conversations. > is the "market" not as large as you may have thought Well, we went into this with "there's a small market now, but we see it growing". So, I would say no. We knew the Python streaming/event-driven application market was small compared to the Java one right now. But, we also see it growing at a steady rate (with signs of a big big jump at some point). > What is the business model now? Things we've found that people are happy to pay for - Support/license that involves us giving them the same assurances of "someone being there" that they get with proprietary software - "Managed service", we'll install and run your Wallaroo applications for you inside your AWS/GCP/Azure accounts. - Enterprise version. We plan on creating products that live around Wallaroo that aren't of interest to the open source community but would be valuable to enterprise users. In the end, what's most important to us right now is getting closer to more users and using what we learn from them to accelerate our improvement of Wallaroo. The more users we have, the more chances we have learn, and the more chances we have to continue to improve our product market fit.

haney 8y ago

Very excited to see this, I've been intrigued by it over the last few months but have avoided it because I was concerned if I built a successful project with it I'd eventually reach a hard cliff in terms of pricing. It's especially difficult to weigh cost benefit of an architecture with a new framework that has specific resource based pricing when you don't have a way of estimating resource usage because you've never used the framework.

spooneybarger 8y ago
You aren't alone in that concern. We heard that from a lot of folks. And I agree with you. When I stopped thinking as "VP of Engineering at Wallaroo" and started thinking about how I would evaluate Wallaroo if I wasn't with the company, I ended up with the problem you are describing. Basically: "why would I spend time evaluating this thing when there's that cliff that I don't know when I'll hit it".