4 ms·
Hey, I know you're saying you're not an expert on Flink or Spark, but do you know if Flink and/or Spark allow you to merge a stream in real-time into a data set
by DannoHung 11y ago
Hey, I know you're saying you're not an expert on Flink or Spark, but do you know if Flink and/or Spark allow you to merge a stream in real-time into a data set that you can run queries against?
Generating analytics directly off a stream has never seemed that useful to me. Being able to take all the data in the stream and merge it with previously collected data, then query that merged data is the sort of thing I need to be able to do.
- brakmic 11y agoHi DannoHung, Maybe this article could be of some value to you: http://flink.apache.org/news/2015/02/09/streaming-example.html http://flink.apache.org/news/2015/02/09/streaming-example.ht... Btw. handling DataStreams and DataSets in Flink is done via almost similar API calls. In DataStreams you only have to additionally insert the Time-Window parameters. Everything else (transformations like filter, map, flatMap etc.) remain the same. Kind regards, Harris
- capkutay 11y ago"Being able to take all the data in the stream and merge it with previously collected data, then query that merged data is the sort of thing I need to be able to do." Sorry for the shameless plug, but Striim does that with a SQL-like interface. Joining streams and previously collected, cached data is as easy as writing a join. You can try it out here: http://www.striim.com http://www.striim.com
- brakmic 11y agoIf I'm not mistaken, "striim" neither is an open-source project nor an open-sourced product.
- samstokes 11y agoI don't know about Flink or Spark, but streaming joins is a use case Apache Samza was explicitly designed to handle. It uses node-local caching to make the join efficient, and persists the cache updates in Kafka to make the cache persist across node failures.