6 ms·

I could really do with a brief rundown of a what a data processing engine does? Have briefly scanned https://www.wallaroolabs.com/ https://www.wallaroolabs.com/

I could really do with a brief rundown of a what a data processing engine does? Have briefly scanned https://www.wallaroolabs.com/ https://www.wallaroolabs.com/ but the descriptions are both too general and too specific. Let's take the machine learning scenario: what does Wallaroo do for me here specifically..?

spooneybarger 9y ago
We currently do event-by-event data processing. We are working on adding microbatching features. Wallaroo provides the infrastructure plumbing to write resilient, scalable applications. You write your code to our scale-independent APIs (currently Python 2.7 is supported. Go and Python 3 coming later this year). Wallaroo hands laying out your data and computations across X number of processes which can change as the application is running. We have built in message replay, are working on completing exactly-once message processing (at-least-once delivery with deduplication). Wallaroo handles the infrastructure concerns so application programmers can worry about the business logic. "the machine learning scenario" is rather broad so I can't answer that specifically. We just starting working with a company that is interested in the ability to take the Python their data scientists are writing on their laptops and move it into production with no logic changes. Currently they end up rewriting in a JVM based language and often have to switch machine learning libraries. There's a 15 minute video I did about scale-independence that includes code and a demo that you might be interested in: https://vimeo.com/234753585 https://vimeo.com/234753585 The vision here is to make it much easier to build and scale distributed data applications in general, not just event-by-event. We are in the process of putting together functionality we need to support microbatch type workloads and jobs that have a beginning and end. re: Too general/too specific. my personal email is in my profile. i'd love to hear from you on what was too general and what was too specific. We did our best with explaining but we knew that we would still end up being off the mark in some cases. If you could spare the time to drop me some notes, I'd appreciate it. Sorry if that answer is a little rambling. We had a number of long hours while cleaning everything up for the first open source release and I'm exhausted.
- dozzie 9y ago
  1. What does it mean "scalable"? When would Wallaroo not be "scalable"? 2. How does Wallaroo differ from just running bunch of AMQP clients for data processing? And I ask this as somebody who's in the middle of writing his own stream processing engine. And a few comments: - Installation procedure is so unnecessarily long. It would be much shorter if you provided source package and not require anything from outside of the distribution. - Installation procedure involves downloading Pony compiler from non-canonical source. It's a warning of sorts. I don't know if the compiler source was changed from the upstream, so I'm wary about Wallaroo's compilation with official Pony compiler. - Installation procedure only ever mentions Ubuntu. You do know that people run other distributions, right? Does Wallaroo even compile under Debian or CentOS, or was it written for Ubuntu specifically? - Requiring Docker to run examples is bad. Your software should be easy to run without garbage from containers.
  - spooneybarger 9y ago
    A lot of how Wallaroo handles state mirrors what Pat Helland wrote about in "Beyond Distributed Transactions". It ends up having a number of nice features for stateful applications. You can see it in action in the 15 minute "get to know scale-independence" video that I did: https://vimeo.com/234753585 https://vimeo.com/234753585 Thanks for your comments. - We are pairing down the installation procedure. We couldn't do everything we wanted before we did our first release. - The compiler was changed from upstream. We are in the process of getting back onto mainline ponyc. That should be happening in the next couple weeks at which point, we will be taking the first step in pairing down the instructions. When we were first developing Wallaroo, we wanted to be able to rapidly evolve the Pony runtime as well as Wallaroo itself. To do that, we needed to fork the project. We've gotten almost all of our changes back into the mainline ponyc. - It works on other distributions. We currently only provide instructions for Trusty and Xenial because as you said, it's a somewhat involved process at the moment that we are working on simplifying. - We went back and forth on if we should use docker for the UI. Its written in Elixir and requires that Erlang be installed. After working with some clients, we decided it was easier to provide a docker container.
  - mcguire 9y ago

spooneybarger 9y ago

We currently do event-by-event data processing. We are working on adding microbatching features. Wallaroo provides the infrastructure plumbing to write resilient, scalable applications. You write your code to our scale-independent APIs (currently Python 2.7 is supported. Go and Python 3 coming later this year). Wallaroo hands laying out your data and computations across X number of processes which can change as the application is running. We have built in message replay, are working on completing exactly-once message processing (at-least-once delivery with deduplication). Wallaroo handles the infrastructure concerns so application programmers can worry about the business logic. "the machine learning scenario" is rather broad so I can't answer that specifically. We just starting working with a company that is interested in the ability to take the Python their data scientists are writing on their laptops and move it into production with no logic changes. Currently they end up rewriting in a JVM based language and often have to switch machine learning libraries. There's a 15 minute video I did about scale-independence that includes code and a demo that you might be interested in: https://vimeo.com/234753585 https://vimeo.com/234753585 The vision here is to make it much easier to build and scale distributed data applications in general, not just event-by-event. We are in the process of putting together functionality we need to support microbatch type workloads and jobs that have a beginning and end. re: Too general/too specific. my personal email is in my profile. i'd love to hear from you on what was too general and what was too specific. We did our best with explaining but we knew that we would still end up being off the mark in some cases. If you could spare the time to drop me some notes, I'd appreciate it. Sorry if that answer is a little rambling. We had a number of long hours while cleaning everything up for the first open source release and I'm exhausted.

dozzie 9y ago
1. What does it mean "scalable"? When would Wallaroo not be "scalable"? 2. How does Wallaroo differ from just running bunch of AMQP clients for data processing? And I ask this as somebody who's in the middle of writing his own stream processing engine. And a few comments: - Installation procedure is so unnecessarily long. It would be much shorter if you provided source package and not require anything from outside of the distribution. - Installation procedure involves downloading Pony compiler from non-canonical source. It's a warning of sorts. I don't know if the compiler source was changed from the upstream, so I'm wary about Wallaroo's compilation with official Pony compiler. - Installation procedure only ever mentions Ubuntu. You do know that people run other distributions, right? Does Wallaroo even compile under Debian or CentOS, or was it written for Ubuntu specifically? - Requiring Docker to run examples is bad. Your software should be easy to run without garbage from containers.
- spooneybarger 9y ago
  A lot of how Wallaroo handles state mirrors what Pat Helland wrote about in "Beyond Distributed Transactions". It ends up having a number of nice features for stateful applications. You can see it in action in the 15 minute "get to know scale-independence" video that I did: https://vimeo.com/234753585 https://vimeo.com/234753585 Thanks for your comments. - We are pairing down the installation procedure. We couldn't do everything we wanted before we did our first release. - The compiler was changed from upstream. We are in the process of getting back onto mainline ponyc. That should be happening in the next couple weeks at which point, we will be taking the first step in pairing down the instructions. When we were first developing Wallaroo, we wanted to be able to rapidly evolve the Pony runtime as well as Wallaroo itself. To do that, we needed to fork the project. We've gotten almost all of our changes back into the mainline ponyc. - It works on other distributions. We currently only provide instructions for Trusty and Xenial because as you said, it's a somewhat involved process at the moment that we are working on simplifying. - We went back and forth on if we should use docker for the UI. Its written in Elixir and requires that Erlang be installed. After working with some clients, we decided it was easier to provide a docker container.
- mcguire 9y ago