4 ms·
Unbounded streams, but with watermarks (which right now seem fixed length?): https://doc.arroyo.dev/concepts#watermarks https://doc.arroyo.dev/concepts#waterma
by thom 3y ago
Unbounded streams, but with watermarks (which right now seem fixed length?):
https://doc.arroyo.dev/concepts#watermarks https://doc.arroyo.dev/concepts#watermarks
Also works based on fixed, pre-built pipelines. This is all very much in the style of most stream processing platforms today but I hope we’ll continue to move closer as an industry to having our cake and eating it: ingest everything in real-time, while serving any query (with joins) over the full dataset (either incrementally or ad-hoc).
- necubi 3y agoIn SQL we currently support specifying watermarks based on SQL expressions, so it can be a bit more expressive than just the basic fixed-delay watermark: https://doc.arroyo.dev/sql/ddl#options https://doc.arroyo.dev/sql/ddl#options. For watermark-based systems, I think the ideal is something that allows users to express either max latencies or statistical completeness (like, wait up to 1 minute or until an estimated 99.5% of the data is there). Beyond that, there are systems that are more integrated end-to-end that can update as late arriving data comes in (like Materialize), and think those have there place. However for many uses of stream processing what's important is taking action once the data is complete enough, and watermarks a useful and pretty straightforward mechanism for that.