5 ms·
Announcing Apache Flink 1.0.0
- easytiger 11y ago> High throughput and low latency stream processing It should be illegal to claim nonsense like that without some comparative example vs a c+= zmq/tcp pipeline doing the same thing
- faizshah 11y agoHere: https://yahooeng.tumblr.com/post/135321837876/benchmarking-streaming-computation-engines-at https://yahooeng.tumblr.com/post/135321837876/benchmarking-s... http://data-artisans.com/extending-the-yahoo-streaming-benchmark/ http://data-artisans.com/extending-the-yahoo-streaming-bench...
- bsg75 11y agoDoes anyone have a good comparison of Flink and Spark, especially from a use case perspective? Most I have found are light in actual contrast detail.
- sewen 11y agoThe differences are mainly around batch-centric vs. streaming-centric model and executions. Here is for example a video walkthrough by MapR: https://www.mapr.com/blog/apache-spark-vs-apache-flink-whiteboard-walkthrough https://www.mapr.com/blog/apache-spark-vs-apache-flink-white...
- faizshah 11y agoAnother major difference is that Flink's scheduler doesn't have a notion of data locality like Spark's. If you want to use data local to the node you have to query whatever you're storing your stuff in (HDFS) and filter those items that aren't on the node.
- sewen 11y agoThat works a bit differently in Flink and Spark. Inside a data flow program, the scheduler tries to schedule as local as possible. For the inputs to a streaming program (for example Kafka partitions), there is currently no locality consideration, but the locality there changes throughout a program lifetime anyways (brokers change leadership and rebalance) Flink's DataSet API does assign data to tasks after the scheduling. That assignment respects locality, actually. That lazy assignment makes it possible to handle large numbers of small files, for example.
- faizshah 11y agoHmm, I didn't know that about the DataSet API. When I looked at the scheduler's code it didn't seem to have any notion of data locality except for colocation of vertices. I'll take a look at the DataSet API's code though, thanks!
- mtrn 11y agoFlink comes from a more database oriented background. It grew out of research project at TU Berlin[1]. I believe, Flink tried to focus on query language and optimization, as you probably would in a database settings. In contrast, Spark is sometimes described as a batch processing system, which provides a real-time experience by intelligently partitioning the work. [1] http://stratosphere.eu/ http://stratosphere.eu/
- sewen 11y agoThat was the research project where many contributors came from. Most of the streaming tech and developments in Flink are very disconnected from that and have little to do with database tech any more, actually.
- doctorcroc 11y agoThe benchmarking study done by Yahoo was fairly comprehensive, and quantitatively assesses the different stream frameworks (including Flink and SS) https://yahooeng.tumblr.com/post/135321837876/benchmarking-streaming-computation-engines-at https://yahooeng.tumblr.com/post/135321837876/benchmarking-s... We also did a podcast about it if you're interested in digging deeper -- http://softwareengineeringdaily.com/2016/02/03/benchmarking-stream-processing-frameworks-with-bobby-evans/ http://softwareengineeringdaily.com/2016/02/03/benchmarking-...
- abeppu 11y agoThanks for sharing that. I recently was looking at a different comparison of the _batch_ capabilities of flink and SS [1], which found that flink was faster at terasort than SS. I'm curious to understand why it is that it looks like SS can get higher throughput than flink in the streaming case, but less in the batch case. [1] http://www.slideshare.net/ssuser6bb12d/a-comparative-performance-evaluation-of-apache-flink http://www.slideshare.net/ssuser6bb12d/a-comparative-perform...
- sewen 11y agoI don't think SS gets higher streaming throughput than Flink. That was an assumption written in the Yahoo! streaming benchmark without an actual experiment.
- ecesena 11y agoIf you're interested in Apache Beam (dataflow), then Flink seems to me the best candidate to become sn open source runner. Spark 2.0 may change things though.
- pcx 11y agoThis is awesome, congrats to the Flink team! For our current project we evaluated Spark and Storm, and have gone the Storm way. Personally I spent a little time on Flink, but could not convince the management for more serious consideration. Mainly because of the name recognition, Spark and Storm are so popular already. But I am confident Flink will become a great option, especially with hitting 1.0. I felt the community was very active and documentation pretty good.
- mring33621 11y agoFlink has a Storm compatibility layer, which may aid the transition, should your management change its mind.
- ptgoetz 11y agoThe Storm compatibility layer may work for word count, but not for anything non-trivial. Here's the fine print (no fault tolerance, etc.): https://github.com/apache/flink/blob/master/flink-contrib/flink-storm/README.md https://github.com/apache/flink/blob/master/flink-contrib/fl...
- pcx 11y agoI don't think it's going to be easy to move from Storm to Flink. They are very different systems underneath. Any substantially complex application is going to end up having compatibility problems. My bets are on Apache Beam (Google Dataflow) to create that compatibility layer. Haven't used it yet, hoping it works out though!
- anc84 11y agoOk, but what IS it?
- ecesena 11y agoA real time processing engine. You can think of an alternative to Storm or Spark Streaming.
- CaptSpify 11y agoI always hate when announcements (especially for a 1.0 release) don't start off with "This is what this is!" http://flink.apache.org/index.html http://flink.apache.org/index.html
- zeisss 11y agoThe info graphic on the right does not show Hadoop, but the getting started guide suggests that Hadoop is a requirement.
- falcolas 11y ago> High throughput and low latency stream processing with exactly-once guarantees. Precisely how is this possible in a real-world distributed system? At-least-once and at-most-once are possible with different distribution methods, but "exactly-once" is, well, I have yet to see it actually implemented (even theoretically, see the Two Generals thought experiment). Even their checkpointing algorithm has an assumption that the underlying transport is "exactly once" and "FIFO ordered". If I had to guess, I'd say that what they're really guaranteeing is "at most once" message delivery. Unfortunately, that has a very different meaning from "exactly once". http://bravenewgeek.com/you-cannot-have-exactly-once-delivery/ http://bravenewgeek.com/you-cannot-have-exactly-once-deliver...
- faizshah 11y agoI'm pretty sure that this is the algorithm flink uses: http://arxiv.org/pdf/1506.08603.pdf http://arxiv.org/pdf/1506.08603.pdf EDIT: fyi sewen is one of the authors of the paper and a founder of flink
- sewen 11y agoYou do not need physical exactly once. If you can "roll back" changes, than physical "at least once" is enough to realize logical "exactly once".
- falcolas 11y agoRolling back changes means that at some point in time, you have "more than once". It also means you can only ever implement logic which can be rolled back. In a pipeline scenario, that is challenging to do without some sort of external locking system.
- sewen 11y agoTrue. In most streaming systems, exactly-once refers to the state that is maintained and controlled by the streaming system. For that, Flink can guarantee such semantics. That state can also be exposed externally, see for example here: http://data-artisans.com/extending-the-yahoo-streaming-benchmark/ http://data-artisans.com/extending-the-yahoo-streaming-bench... For interactions (writes) with external systems, you need transactional integration. There is WIP to do this for transactional databases, for non transactional systems, there are cases where it cannot be strictly solved.
- bfrog 11y agoIt seems like there's a multitude of these data analytics frameworks with slightly different features and goals, but generally speaking, do similiar things. Is there some significant differences between the 3 that maybe aren't as clear from a quick overview?
- jahitr 11y agoStorm is pretty much dead. Spark, from my point of view, is the current forerunner. It support mostly all use cases. Batch jobs. Streaming jobs. Work is being done to reduce the batch time to ms. Comercial support is very good from the guys at Cloudera and Databricks. My 2 cents. I hope It helps someone.
- pcx 11y agoI love Spark, it has great documentations and Scala! But how is Storm dead?! The documentation is not good enough yet, but there is a strong userbase and it is being actively used in a lot of companies.
- cbsmith 11y agoStorm is anything but dead. Spark streaming is definitely not all there yet.
- jonstewart 11y agoAs luck would have it, I'm going to write a POC next month with Spark Streaming and Flink. Should I also throw Storm into the mix? What do you think about Flink? I generally can get by with Batch, but streaming makes a lot of our logic easier, and might allow us to handle some new situations. Flink looks pretty good on paper.
- cbsmith 11y agoFlink looks really interesting to me. If it can get community momentum I might switch to it. I'd definitely recommend including Storm for a POC. For any POC I'd warn that most of these streaming solutions are operationally a bit of a PITA, which tends to distort POC results.
- jonstewart 11y agoWe will be half-to-single rack scale for a while, so I'm not too too worried about operations--we also need to target solo workstations, which is the real challenge with any of these systems. If things go haywire, we can just restart processing later.
- tristanz 11y ago"The Evolution of Massive-Scale Data Processing" presentation from Google provides a good comparison of Flink with other tools: https://docs.google.com/presentation/d/10vs2PnjynYMtDpwFsqmSePtMnfJirCkXcHZ1SkwDg-s/present?slide=id.gd44d9957e_0_243 https://docs.google.com/presentation/d/10vs2PnjynYMtDpwFsqmS...
- ozten 11y agoThe favicon is... unfortunate. The squirrel mascot is cute, but doesn't reduce well.