3 ms·

We currently do event-by-event data processing. We are working on adding microbatching features. Wallaroo provides the infrastructure plumbing to write resilie

We currently do event-by-event data processing. We are working on adding microbatching features. Wallaroo provides the infrastructure plumbing to write resilient, scalable applications. You write your code to our scale-independent APIs (currently Python 2.7 is supported. Go and Python 3 coming later this year). Wallaroo hands laying out your data and computations across X number of processes which can change as the application is running. We have built in message replay, are working on completing exactly-once message processing (at-least-once delivery with deduplication). Wallaroo handles the infrastructure concerns so application programmers can worry about the business logic. "the machine learning scenario" is rather broad so I can't answer that specifically. We just starting working with a company that is interested in the ability to take the Python their data scientists are writing on their laptops and move it into production with no logic changes. Currently they end up rewriting in a JVM based language and often have to switch machine learning libraries. There's a 15 minute video I did about scale-independence that includes code and a demo that you might be interested in: https://vimeo.com/234753585 https://vimeo.com/234753585 The vision here is to make it much easier to build and scale distributed data applications in general, not just event-by-event. We are in the process of putting together functionality we need to support microbatch type workloads and jobs that have a beginning and end. re: Too general/too specific. my personal email is in my profile. i'd love to hear from you on what was too general and what was too specific. We did our best with explaining but we knew that we would still end up being off the mark in some cases. If you could spare the time to drop me some notes, I'd appreciate it. Sorry if that answer is a little rambling. We had a number of long hours while cleaning everything up for the first open source release and I'm exhausted.

dozzie 9y ago
1. What does it mean "scalable"? When would Wallaroo not be "scalable"? 2. How does Wallaroo differ from just running bunch of AMQP clients for data processing? And I ask this as somebody who's in the middle of writing his own stream processing engine. And a few comments: - Installation procedure is so unnecessarily long. It would be much shorter if you provided source package and not require anything from outside of the distribution. - Installation procedure involves downloading Pony compiler from non-canonical source. It's a warning of sorts. I don't know if the compiler source was changed from the upstream, so I'm wary about Wallaroo's compilation with official Pony compiler. - Installation procedure only ever mentions Ubuntu. You do know that people run other distributions, right? Does Wallaroo even compile under Debian or CentOS, or was it written for Ubuntu specifically? - Requiring Docker to run examples is bad. Your software should be easy to run without garbage from containers.
- spooneybarger 9y ago
  A lot of how Wallaroo handles state mirrors what Pat Helland wrote about in "Beyond Distributed Transactions". It ends up having a number of nice features for stateful applications. You can see it in action in the 15 minute "get to know scale-independence" video that I did: https://vimeo.com/234753585 https://vimeo.com/234753585 Thanks for your comments. - We are pairing down the installation procedure. We couldn't do everything we wanted before we did our first release. - The compiler was changed from upstream. We are in the process of getting back onto mainline ponyc. That should be happening in the next couple weeks at which point, we will be taking the first step in pairing down the instructions. When we were first developing Wallaroo, we wanted to be able to rapidly evolve the Pony runtime as well as Wallaroo itself. To do that, we needed to fork the project. We've gotten almost all of our changes back into the mainline ponyc. - It works on other distributions. We currently only provide instructions for Trusty and Xenial because as you said, it's a somewhat involved process at the moment that we are working on simplifying. - We went back and forth on if we should use docker for the UI. Its written in Elixir and requires that Erlang be installed. After working with some clients, we decided it was easier to provide a docker container.
- mcguire 9y ago
  BTW, several of the core Pony people are working for Wallaroo labs. The fork sounds temporary, and this is probably a good thing for those following Pony.
makmanalp 9y ago
So are we talking about something like a compile-to-native-binary storm / kafka?
- spooneybarger 9y ago
  The C++ API that we currently aren't supporting and the Go language bindings that we are working on create native binaries; yes. The Python version embeds a Python 2.7 interpreter inside a running application. We have C code that acts as an interface between our Pony runtime and the developer's Python code, calling the required framework functions at the correct time. The event-by-event features in Wallaroo are somewhat analagous to Storm but we had management of in-memory state that allows you to grow the size of the cluster and have the application state redistributed across the cluster. This can be done while an applicaiton is running. In the case of Storm, this would be like: 1- keeping your application state in objects in your bolts (this helps with performance compared to an external store) 2- having a means to add more workers to the running storm topology and have that in bolt state redistributed across the new number of machines. The 15-minute "scale independence" video shows this in action: https://vimeo.com/234753585 https://vimeo.com/234753585 In many ways, Wallaroo looks a lot like the systems that Pat Helland describes in "Beyond Distributed Transactions": http://queue.acm.org/detail.cfm?id=3025012 http://queue.acm.org/detail.cfm?id=3025012 One of our engineers is giving a talk at Velocity conf NY next week. He's going to be discussing Pat's paper and how Wallaroo exhibits a number of characteristics he describes.
  - makmanalp 9y ago
    Yes, that paper reference was exactly what I wanted. Thank you!
    - spooneybarger 9y ago
      If you are into papers and talks, we have a few more on the website that we love to turn people on to: http://www.wallaroolabs.com/community/talks http://www.wallaroolabs.com/community/talks
nomass 9y ago
I just watched the video and here is what I think could be approved. As someone who knows nothing about your project the first thing I’d like to know how it runs. So I think you could get out more from your demonstration and place it before the API usage tutorial part. Right now the demonstration feels a bit hasty and nervous. It would be nice to have more detail on whats going on in the demo.
- spooneybarger 9y ago
  That's a paired down version from 30 minutes. The decision we faced was: Do we think asking people to watch a 30 minute intro video will cause far more people to drop off than a 15 minute video? If yes, can we convey a lot of the value of Wallaroo in 15 minutes and get them interested in learning more? We decided the answer to both questions was yes. I'm planning on doing a couple longer intro videos for those with an appetite to learn more at a slightly more relaxed and less hurried pace. My question to folks would be, what's the longest you'd be interested in watching? I have enough to cover that I could do: 30 minute, 60 minute or 90 minute videos, each providing more depth. How much time are folks generally interested in investing in videos of this sort?

dozzie 9y ago

1. What does it mean "scalable"? When would Wallaroo not be "scalable"? 2. How does Wallaroo differ from just running bunch of AMQP clients for data processing? And I ask this as somebody who's in the middle of writing his own stream processing engine. And a few comments: - Installation procedure is so unnecessarily long. It would be much shorter if you provided source package and not require anything from outside of the distribution. - Installation procedure involves downloading Pony compiler from non-canonical source. It's a warning of sorts. I don't know if the compiler source was changed from the upstream, so I'm wary about Wallaroo's compilation with official Pony compiler. - Installation procedure only ever mentions Ubuntu. You do know that people run other distributions, right? Does Wallaroo even compile under Debian or CentOS, or was it written for Ubuntu specifically? - Requiring Docker to run examples is bad. Your software should be easy to run without garbage from containers.

spooneybarger 9y ago
A lot of how Wallaroo handles state mirrors what Pat Helland wrote about in "Beyond Distributed Transactions". It ends up having a number of nice features for stateful applications. You can see it in action in the 15 minute "get to know scale-independence" video that I did: https://vimeo.com/234753585 https://vimeo.com/234753585 Thanks for your comments. - We are pairing down the installation procedure. We couldn't do everything we wanted before we did our first release. - The compiler was changed from upstream. We are in the process of getting back onto mainline ponyc. That should be happening in the next couple weeks at which point, we will be taking the first step in pairing down the instructions. When we were first developing Wallaroo, we wanted to be able to rapidly evolve the Pony runtime as well as Wallaroo itself. To do that, we needed to fork the project. We've gotten almost all of our changes back into the mainline ponyc. - It works on other distributions. We currently only provide instructions for Trusty and Xenial because as you said, it's a somewhat involved process at the moment that we are working on simplifying. - We went back and forth on if we should use docker for the UI. Its written in Elixir and requires that Erlang be installed. After working with some clients, we decided it was easier to provide a docker container.
mcguire 9y ago
BTW, several of the core Pony people are working for Wallaroo labs. The fork sounds temporary, and this is probably a good thing for those following Pony.

makmanalp 9y ago

So are we talking about something like a compile-to-native-binary storm / kafka?

spooneybarger 9y ago
The C++ API that we currently aren't supporting and the Go language bindings that we are working on create native binaries; yes. The Python version embeds a Python 2.7 interpreter inside a running application. We have C code that acts as an interface between our Pony runtime and the developer's Python code, calling the required framework functions at the correct time. The event-by-event features in Wallaroo are somewhat analagous to Storm but we had management of in-memory state that allows you to grow the size of the cluster and have the application state redistributed across the cluster. This can be done while an applicaiton is running. In the case of Storm, this would be like: 1- keeping your application state in objects in your bolts (this helps with performance compared to an external store) 2- having a means to add more workers to the running storm topology and have that in bolt state redistributed across the new number of machines. The 15-minute "scale independence" video shows this in action: https://vimeo.com/234753585 https://vimeo.com/234753585 In many ways, Wallaroo looks a lot like the systems that Pat Helland describes in "Beyond Distributed Transactions": http://queue.acm.org/detail.cfm?id=3025012 http://queue.acm.org/detail.cfm?id=3025012 One of our engineers is giving a talk at Velocity conf NY next week. He's going to be discussing Pat's paper and how Wallaroo exhibits a number of characteristics he describes.
- makmanalp 9y ago
  Yes, that paper reference was exactly what I wanted. Thank you!
  - spooneybarger 9y ago
    If you are into papers and talks, we have a few more on the website that we love to turn people on to: http://www.wallaroolabs.com/community/talks http://www.wallaroolabs.com/community/talks

nomass 9y ago

I just watched the video and here is what I think could be approved. As someone who knows nothing about your project the first thing I’d like to know how it runs. So I think you could get out more from your demonstration and place it before the API usage tutorial part. Right now the demonstration feels a bit hasty and nervous. It would be nice to have more detail on whats going on in the demo.

spooneybarger 9y ago
That's a paired down version from 30 minutes. The decision we faced was: Do we think asking people to watch a 30 minute intro video will cause far more people to drop off than a 15 minute video? If yes, can we convey a lot of the value of Wallaroo in 15 minutes and get them interested in learning more? We decided the answer to both questions was yes. I'm planning on doing a couple longer intro videos for those with an appetite to learn more at a slightly more relaxed and less hurried pace. My question to folks would be, what's the longest you'd be interested in watching? I have enough to cover that I could do: 30 minute, 60 minute or 90 minute videos, each providing more depth. How much time are folks generally interested in investing in videos of this sort?