4 ms·
From the little I know, Spark was built as a Batch job solution. Flink was built as a Streaming solution. Since Spark needed to adapt, they leveraged micro-bat
by yazaddaruvala 6y ago
From the little I know,
Spark was built as a Batch job solution. Flink was built as a Streaming solution. Since Spark needed to adapt, they leveraged micro-batching to operate as a streaming solution.
They are very similar today. Maybe some remnants of the original design make Spark "slightly less appropriate" for pure streaming usecases, or possibly less able to iterate on future features/optimizations, but so-far that line of reasoning has been only speculation.
- faizshah 6y agoSo Flink differentiates in stream processing in two major ways: - Flink guarantees exactly once stream processing through a barrier checkpointing system. - Flink has very detailed APIs for handling state and doing stateful stream processing. Additionally, Flink has one of the most active communities in the Apache Software Foundation: https://blogs.apache.org/foundation/entry/the-apache-software-foundation-announces55 https://blogs.apache.org/foundation/entry/the-apache-softwar... In situations where you want to create a streaming application with persistent state you might choose Flink over spark streaming.