4 ms·

Doesn't Apache Flink already solve these problems? E.g. stateful, fault tolerant, scalable stream processing.

caust1c 6y ago
Flink uses highwatermarks like google's dataflow and is based on (I think) the original Millwheel paper. The alternative to all this nonsense is to just throw everything into clickhouse and build materialized views! The drawback is you can't do complex joins, but for 90% of use-cases, clickhouse materialized views work swimmingly.
- thom 6y ago
  As I understand it, ClickHouse still doesn't support window functions either, so it gets hard to do complex sequence-based logic in calculations. Materialise actually supports lateral joins so you can do some powerful things, but I am sure a better developer experience is possible.
  - hodgesrm 6y ago
    ClickHouse does this using arrays. There's a rich set of functions to pull grouped values into arrays, process them with lambdas (e.g, map, sort, etc.), and explode them back out into tabular result. Here's a slide deck with examples from a recent presentation: https://altinity.com/presentations/introduction-to-high-velocity-analytics-using-clickhouse-arrays-2 https://altinity.com/presentations/introduction-to-high-velo... ClickHouse sneaks a functional programming model with lambda expressions into SQL. It's not standard SQL but has enormous flexibility.
necubi 6y ago
Sort of. Let's say you're counting the number of unique users visiting your site in a 5 minute fixed window. This is pretty simple: you have a set, and each time a request comes in you add it to the set. At the end of the 5 minute window you output the set cardinality to your downstream consumer and throw out the raw data. But what if this is a mobile app, and requests may be delayed by minutes or even hours due to gaps in cellular connectivity? In that case, we need a strategy to handle "late arriving" data. Flink has several approaches for this, but they're all based around a "high-watermark." This is basically an estimate at time T that all data from time T-D has arrived. Once the watermark has passed the end of our window we consider it closed and can compute the final value for it. (How you compute the watermark is up to you; typically you use a fixed value but this can be made more accurate if you have out-of-band information). In addition to the default behavior (where late-arriving data is dropped) you can customize this by specifying a trigger that runs when late-arriving date comes, which can be used to e.g., update an external datastore. However at that point the reconciliation is outside of the scope of Flink. There are various other ways to organize this within flink, i.e., you can keep the windows open indefinitely, update the internal state of your flink job, and serve directly from the flink state, or you can use a periodic trigger to periodically update your downstream from that state. Obviously this will require a larger state size (in disk or in memory depending on your state backend) since you won't be able to close out the window and have to keep all of the data around. Much more about all this in the Flink docs (https://ci.apache.org/projects/flink/flink-docs-release-1.11/concepts/timely-stream-processing.html https://ci.apache.org/projects/flink/flink-docs-release-1.11...). Also a lot of the theory behind this comes from the Google Dataflow paper (https://research.google/pubs/pub43864/ https://research.google/pubs/pub43864/) which is also just a great read.
- kohlerm 6y ago
  Yes I'm aware that Flink has these mechanisms. But I'm not sure I understand how Differential Dataflow handles this in a better way. For simple windowing functions there are certainly shortcuts possible, but for more complex ones such as session windows it gets complicated. You might have to merge sessions as late data arrives. This is what Flink and others (Google Dataflow) do.

Flink uses highwatermarks like google's dataflow and is based on (I think) the original Millwheel paper. The alternative to all this nonsense is to just throw everything into clickhouse and build materialized views! The drawback is you can't do complex joins, but for 90% of use-cases, clickhouse materialized views work swimmingly.

thom 6y ago
As I understand it, ClickHouse still doesn't support window functions either, so it gets hard to do complex sequence-based logic in calculations. Materialise actually supports lateral joins so you can do some powerful things, but I am sure a better developer experience is possible.
- hodgesrm 6y ago
  ClickHouse does this using arrays. There's a rich set of functions to pull grouped values into arrays, process them with lambdas (e.g, map, sort, etc.), and explode them back out into tabular result. Here's a slide deck with examples from a recent presentation: https://altinity.com/presentations/introduction-to-high-velocity-analytics-using-clickhouse-arrays-2 https://altinity.com/presentations/introduction-to-high-velo... ClickHouse sneaks a functional programming model with lambda expressions into SQL. It's not standard SQL but has enormous flexibility.

necubi 6y ago

Sort of. Let's say you're counting the number of unique users visiting your site in a 5 minute fixed window. This is pretty simple: you have a set, and each time a request comes in you add it to the set. At the end of the 5 minute window you output the set cardinality to your downstream consumer and throw out the raw data. But what if this is a mobile app, and requests may be delayed by minutes or even hours due to gaps in cellular connectivity? In that case, we need a strategy to handle "late arriving" data. Flink has several approaches for this, but they're all based around a "high-watermark." This is basically an estimate at time T that all data from time T-D has arrived. Once the watermark has passed the end of our window we consider it closed and can compute the final value for it. (How you compute the watermark is up to you; typically you use a fixed value but this can be made more accurate if you have out-of-band information). In addition to the default behavior (where late-arriving data is dropped) you can customize this by specifying a trigger that runs when late-arriving date comes, which can be used to e.g., update an external datastore. However at that point the reconciliation is outside of the scope of Flink. There are various other ways to organize this within flink, i.e., you can keep the windows open indefinitely, update the internal state of your flink job, and serve directly from the flink state, or you can use a periodic trigger to periodically update your downstream from that state. Obviously this will require a larger state size (in disk or in memory depending on your state backend) since you won't be able to close out the window and have to keep all of the data around. Much more about all this in the Flink docs (https://ci.apache.org/projects/flink/flink-docs-release-1.11/concepts/timely-stream-processing.html https://ci.apache.org/projects/flink/flink-docs-release-1.11...). Also a lot of the theory behind this comes from the Google Dataflow paper (https://research.google/pubs/pub43864/ https://research.google/pubs/pub43864/) which is also just a great read.

kohlerm 6y ago
Yes I'm aware that Flink has these mechanisms. But I'm not sure I understand how Differential Dataflow handles this in a better way. For simple windowing functions there are certainly shortcuts possible, but for more complex ones such as session windows it gets complicated. You might have to merge sessions as late data arrives. This is what Flink and others (Google Dataflow) do.