2 ms·
In SQL we currently support specifying watermarks based on SQL expressions, so it can be a bit more expressive than just the basic fixed-delay watermark: https:
by necubi 3y ago
In SQL we currently support specifying watermarks based on SQL expressions, so it can be a bit more expressive than just the basic fixed-delay watermark: https://doc.arroyo.dev/sql/ddl#options https://doc.arroyo.dev/sql/ddl#options. For watermark-based systems, I think the ideal is something that allows users to express either max latencies or statistical completeness (like, wait up to 1 minute or until an estimated 99.5% of the data is there).
Beyond that, there are systems that are more integrated end-to-end that can update as late arriving data comes in (like Materialize), and think those have there place. However for many uses of stream processing what's important is taking action once the data is complete enough, and watermarks a useful and pretty straightforward mechanism for that.