3 ms·
Flink uses highwatermarks like google's dataflow and is based on (I think) the original Millwheel paper. The alternative to all this nonsense is to just throw
by caust1c 6y ago
Flink uses highwatermarks like google's dataflow and is based on (I think) the original Millwheel paper.
The alternative to all this nonsense is to just throw everything into clickhouse and build materialized views! The drawback is you can't do complex joins, but for 90% of use-cases, clickhouse materialized views work swimmingly.
- thom 6y agoAs I understand it, ClickHouse still doesn't support window functions either, so it gets hard to do complex sequence-based logic in calculations. Materialise actually supports lateral joins so you can do some powerful things, but I am sure a better developer experience is possible.
- hodgesrm 6y agoClickHouse does this using arrays. There's a rich set of functions to pull grouped values into arrays, process them with lambdas (e.g, map, sort, etc.), and explode them back out into tabular result. Here's a slide deck with examples from a recent presentation: https://altinity.com/presentations/introduction-to-high-velocity-analytics-using-clickhouse-arrays-2 https://altinity.com/presentations/introduction-to-high-velo... ClickHouse sneaks a functional programming model with lambda expressions into SQL. It's not standard SQL but has enormous flexibility.