4 ms·

Stream Processing with Apache Flink

43 points by brakmic 11y ago

vonnik 11y ago
Really impressed with Flink and the Data Artisans team. If I had to bet, I'd say they're the future of streaming.
LgWoodenBadger 11y ago
Curious that there's no comparison with the other stream processing platform of choice, Apache Storm.
- brakmic 11y ago
  Hi, Some time ago I wrote an article about Apache Spark: http://blog.brakmic.com/data-science-for-losers-part-3-scala-apache-spark/ http://blog.brakmic.com/data-science-for-losers-part-3-scala... However, I don't think that writing a comparison of such complex technologies within an introductory article would be of much value to readers. Of course, it would be easy to create some tabular structure containing "pro/contra" about Flink and Spark, but this would be way too simplistic, imho. To be honest, I don't consider myself an expert on Flink and/or Spark so it would be unfair towards my readers to embed such a comparison within the article. But a properly written comparison could be of value for those who have to decide what kind of processing (batch/streaming/machine learning etc.) they need to fulfill some specific (that is, "business") tasks. Kind Regards, Harris
jamesblonde 11y ago
Here's a good intro on Flink streaming: https://dato.com/videos/data-science-summit-seif-haridi.html https://dato.com/videos/data-science-summit-seif-haridi.html Summary: advantages are real-time (10s of milliseconds) processing latency, exactly-once semantics, failure-recovery model based on Chandy Lamport, backpressure support.
DannoHung 11y ago
Hey, I know you're saying you're not an expert on Flink or Spark, but do you know if Flink and/or Spark allow you to merge a stream in real-time into a data set that you can run queries against? Generating analytics directly off a stream has never seemed that useful to me. Being able to take all the data in the stream and merge it with previously collected data, then query that merged data is the sort of thing I need to be able to do.
- brakmic 11y ago
  Hi DannoHung, Maybe this article could be of some value to you: http://flink.apache.org/news/2015/02/09/streaming-example.html http://flink.apache.org/news/2015/02/09/streaming-example.ht... Btw. handling DataStreams and DataSets in Flink is done via almost similar API calls. In DataStreams you only have to additionally insert the Time-Window parameters. Everything else (transformations like filter, map, flatMap etc.) remain the same. Kind regards, Harris
- capkutay 11y ago
  "Being able to take all the data in the stream and merge it with previously collected data, then query that merged data is the sort of thing I need to be able to do." Sorry for the shameless plug, but Striim does that with a SQL-like interface. Joining streams and previously collected, cached data is as easy as writing a join. You can try it out here: http://www.striim.com http://www.striim.com
  - brakmic 11y ago
    If I'm not mistaken, "striim" neither is an open-source project nor an open-sourced product.
- samstokes 11y ago
  I don't know about Flink or Spark, but streaming joins is a use case Apache Samza was explicitly designed to handle. It uses node-local caching to make the join efficient, and persists the cache updates in Kafka to make the cache persist across node failures.

43 points by brakmic 11y ago

vonnik 11y ago

Really impressed with Flink and the Data Artisans team. If I had to bet, I'd say they're the future of streaming.

LgWoodenBadger 11y ago

Curious that there's no comparison with the other stream processing platform of choice, Apache Storm.

brakmic 11y ago
Hi, Some time ago I wrote an article about Apache Spark: http://blog.brakmic.com/data-science-for-losers-part-3-scala-apache-spark/ http://blog.brakmic.com/data-science-for-losers-part-3-scala... However, I don't think that writing a comparison of such complex technologies within an introductory article would be of much value to readers. Of course, it would be easy to create some tabular structure containing "pro/contra" about Flink and Spark, but this would be way too simplistic, imho. To be honest, I don't consider myself an expert on Flink and/or Spark so it would be unfair towards my readers to embed such a comparison within the article. But a properly written comparison could be of value for those who have to decide what kind of processing (batch/streaming/machine learning etc.) they need to fulfill some specific (that is, "business") tasks. Kind Regards, Harris

jamesblonde 11y ago

Here's a good intro on Flink streaming: https://dato.com/videos/data-science-summit-seif-haridi.html https://dato.com/videos/data-science-summit-seif-haridi.html Summary: advantages are real-time (10s of milliseconds) processing latency, exactly-once semantics, failure-recovery model based on Chandy Lamport, backpressure support.

DannoHung 11y ago

Hey, I know you're saying you're not an expert on Flink or Spark, but do you know if Flink and/or Spark allow you to merge a stream in real-time into a data set that you can run queries against? Generating analytics directly off a stream has never seemed that useful to me. Being able to take all the data in the stream and merge it with previously collected data, then query that merged data is the sort of thing I need to be able to do.

brakmic 11y ago
Hi DannoHung, Maybe this article could be of some value to you: http://flink.apache.org/news/2015/02/09/streaming-example.html http://flink.apache.org/news/2015/02/09/streaming-example.ht... Btw. handling DataStreams and DataSets in Flink is done via almost similar API calls. In DataStreams you only have to additionally insert the Time-Window parameters. Everything else (transformations like filter, map, flatMap etc.) remain the same. Kind regards, Harris
capkutay 11y ago
"Being able to take all the data in the stream and merge it with previously collected data, then query that merged data is the sort of thing I need to be able to do." Sorry for the shameless plug, but Striim does that with a SQL-like interface. Joining streams and previously collected, cached data is as easy as writing a join. You can try it out here: http://www.striim.com http://www.striim.com
- brakmic 11y ago
  If I'm not mistaken, "striim" neither is an open-source project nor an open-sourced product.
samstokes 11y ago
I don't know about Flink or Spark, but streaming joins is a use case Apache Samza was explicitly designed to handle. It uses node-local caching to make the join efficient, and persists the cache updates in Kafka to make the cache persist across node failures.