4 ms·

How does the wallaroo roadmap compare to other ambitious frameworks such as Flink and Beam ? Haven't had much time to look at the API, are there tools to reaso

How does the wallaroo roadmap compare to other ambitious frameworks such as Flink and Beam ? Haven't had much time to look at the API, are there tools to reason about both processing / event time ? Is it possible to save state and restart processes with updated code ?

spooneybarger 9y ago
Hi Gepsens, - How does the wallaroo roadmap compare to other ambitious frameworks such as Flink and Beam ? I'm not familiar with the roadmaps for Flink and Beam so I can't comment on that. I can however, speak to our roadmap. For Go in particular, I'm going to be starting on performance improvements very soon. Beyond that, we are waiting for feedback from folks to determine what the best course of action for the Go API is. When we released our Python API, we got a lot of excellent feedback that lead to us switching from a class based API to one that emphasized functions and decorators. We expect that similar evolution will happen with the Go API. For Wallaroo in general, we're working on improvements to our documentation, our installation experience, Python API improvements, state object migrations (see below), as well as an expanded API to allow you to express more types of dataflows with Wallaroo. There's more beyond that, but you get the idea. Wallaroo is definitely a younger product than Flink and Beam. We open sourced it in September of 2017 so we have a lot of work to do. We'd love to hear from folks who are interested in being able to do stream processing in Go or Python without needing to become experts in running the JVM in production. Talking with folks helps us prioritize our work and deliver a product that can be valuable to some folks now and still more in the future as we add features. - Tools to reason about processing/event time: Wallaroo has a metrics UI that allows you to see throughput and within Wallaroo latency on a per-pipeline basis as well as per computation. It can be very helpful in spotting bottlenecks. I'm interested in hearing what other tools you think would be useful. We have a few ideas but are always looking for more things we can build to make the product better. - Is it possible to save state and restart processes with updated code? Yes, so long as your data objects don't change. If your data objects change (for example, add a field, remove a field) then not currently. However, we started initial discussion on our approach to that this week and starting work on it in the near future. The current solution if your data objects change is to get the state of objects out of the system before shutdown and then stream in those new values when you startup with the code that involves a schema change to your data objects. It's far from perfect but works for now until, the schema migration feature is added.

spooneybarger 9y ago

Hi Gepsens, - How does the wallaroo roadmap compare to other ambitious frameworks such as Flink and Beam ? I'm not familiar with the roadmaps for Flink and Beam so I can't comment on that. I can however, speak to our roadmap. For Go in particular, I'm going to be starting on performance improvements very soon. Beyond that, we are waiting for feedback from folks to determine what the best course of action for the Go API is. When we released our Python API, we got a lot of excellent feedback that lead to us switching from a class based API to one that emphasized functions and decorators. We expect that similar evolution will happen with the Go API. For Wallaroo in general, we're working on improvements to our documentation, our installation experience, Python API improvements, state object migrations (see below), as well as an expanded API to allow you to express more types of dataflows with Wallaroo. There's more beyond that, but you get the idea. Wallaroo is definitely a younger product than Flink and Beam. We open sourced it in September of 2017 so we have a lot of work to do. We'd love to hear from folks who are interested in being able to do stream processing in Go or Python without needing to become experts in running the JVM in production. Talking with folks helps us prioritize our work and deliver a product that can be valuable to some folks now and still more in the future as we add features. - Tools to reason about processing/event time: Wallaroo has a metrics UI that allows you to see throughput and within Wallaroo latency on a per-pipeline basis as well as per computation. It can be very helpful in spotting bottlenecks. I'm interested in hearing what other tools you think would be useful. We have a few ideas but are always looking for more things we can build to make the product better. - Is it possible to save state and restart processes with updated code? Yes, so long as your data objects don't change. If your data objects change (for example, add a field, remove a field) then not currently. However, we started initial discussion on our approach to that this week and starting work on it in the near future. The current solution if your data objects change is to get the state of objects out of the system before shutdown and then stream in those new values when you startup with the code that involves a schema change to your data objects. It's far from perfect but works for now until, the schema migration feature is added.