4 ms·

Does this support advanced windowing and watermarking strategies like are found in Flink?

spooneybarger 9y ago
You can do event based windowing now, but you need to build it up from primitives. Time based windows aren't supported yet, they are however on our roadmap. We plan on adding windowing APIs to make it very easy to do a variety of windowing. You can learn more about the existing event based windowing approach that is available now here: https://blog.wallaroolabs.com/2017/11/non-native-event-driven-windowing-in-wallaroo/ https://blog.wallaroolabs.com/2017/11/non-native-event-drive... --- Can you elaborate on what specifically you are referring to as "watermarking strategies" in Flink? We are familiar with Flink's use of the Chandy Lamport algorithm but I'm not sure if that is what you are referring to.
- joshlemer 9y ago
  So in flink you can use a few different bundled in strategies for emitting watermarks, or you can define your own. Here's the documentation on that https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/event_timestamps_watermarks.html https://ci.apache.org/projects/flink/flink-docs-release-1.4/... For example, you can use the bundled MaxOutOfOrdernessTimestampExtractor to emit watermarks that lag behind the maximum timestamp seen so far by some constant amount (say, 50 ms). You could customize this behaviour by implementing the required interface, to emit a watermark, say, whenever we pass an other calendar day. You can also assign strictly-ascending watermarks which you could say is equivalent to maxOutOfOrderness = 0. You can also choose to emit watermarks based on the content of the stream (see Punctuated Watermarks https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/event_timestamps_watermarks.html#with-punctuated-watermarks https://ci.apache.org/projects/flink/flink-docs-release-1.4/...)
  - spooneybarger 9y ago
    Our time based windowing has been implemented yet, so the answer would be: not at this time. It is however on our roadmap.

You can do event based windowing now, but you need to build it up from primitives. Time based windows aren't supported yet, they are however on our roadmap. We plan on adding windowing APIs to make it very easy to do a variety of windowing. You can learn more about the existing event based windowing approach that is available now here: https://blog.wallaroolabs.com/2017/11/non-native-event-driven-windowing-in-wallaroo/ https://blog.wallaroolabs.com/2017/11/non-native-event-drive... --- Can you elaborate on what specifically you are referring to as "watermarking strategies" in Flink? We are familiar with Flink's use of the Chandy Lamport algorithm but I'm not sure if that is what you are referring to.

joshlemer 9y ago
So in flink you can use a few different bundled in strategies for emitting watermarks, or you can define your own. Here's the documentation on that https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/event_timestamps_watermarks.html https://ci.apache.org/projects/flink/flink-docs-release-1.4/... For example, you can use the bundled MaxOutOfOrdernessTimestampExtractor to emit watermarks that lag behind the maximum timestamp seen so far by some constant amount (say, 50 ms). You could customize this behaviour by implementing the required interface, to emit a watermark, say, whenever we pass an other calendar day. You can also assign strictly-ascending watermarks which you could say is equivalent to maxOutOfOrderness = 0. You can also choose to emit watermarks based on the content of the stream (see Punctuated Watermarks https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/event_timestamps_watermarks.html#with-punctuated-watermarks https://ci.apache.org/projects/flink/flink-docs-release-1.4/...)
- spooneybarger 9y ago
  Our time based windowing has been implemented yet, so the answer would be: not at this time. It is however on our roadmap.