4 ms·

If you are familiar with Flink, would you mind sharing your use cases? why Flink and not Kafka ? I have experience with Kafka but I have not had a reason to inv

If you are familiar with Flink, would you mind sharing your use cases? why Flink and not Kafka ? I have experience with Kafka but I have not had a reason to investigate Flink for my use cases.

PhoenixReborn 5y ago
> why Flink and not Kafka ? I assume you mean "Kafka Streams" when you say this, as Kafka is just a event bus and Flink can be used to read messages from it. The biggest advantage of Flink (IMO) is that you can write the Flink logic once, and reuse it for both batch and stream processing. So if I write a Flink job that consumes a Kafka stream and produces some aggregated outputs, that same job can be run against my data lake in S3/GCS/Azure Blob Storage, etc. Kafka Streams does not support batch processing, or working on top of anything other than Kafka. Flink supports building on top of other message buses like Pulsar as well: https://flink.apache.org/news/2019/11/25/query-pulsar-streams-using-apache-flink.html https://flink.apache.org/news/2019/11/25/query-pulsar-stream...
- dikei 5y ago
  Actually, unified batch and stream processing in Flink is a bit of false advertisement. In Flink, stream and batch have different API (DataStream vs DataSet) so reusing logic is not really practical, unless you abstract out everything, in which case you might as well use Spark which is faster than DataSet API. Flink's developers is trying to get rid of DataSet API and move to DataStream and Table API for everything, but it's not done yet.
dikei 5y ago
Advantage of Flink over Kafka Stream * Flink work on other message queue tech other than Kafka like Amazon SQS, Pulsar, etc.. * Back when I last read about Kafka Stream, Flink has better support for stateful processing, the entire state of execution can be safely snapshot into storage and resume at any time. * Kafka Stream shuffle data is slower, because it has to send data to a new topic in the broker, instead of sending it directly between compute node. Advantage of Kafka Stream over Flink * Kafka Stream deployment is simple, just start the jar like any other Java program, scaling up by running the jar multiple time. On the other hand, Flink need a mature orchestration framework like YARN, Meso or K8S, trying to manage a Flink deployment without them is very painful. * Flink require a central, persistent storage like HDFS or S3 for its checkpoint mechanism; Kafka stream doesn't.

If you are familiar with Flink, would you mind sharing your use cases? why Flink and not Kafka ? I have experience with Kafka but I have not had a reason to investigate Flink for my use cases.

PhoenixReborn 5y ago

> why Flink and not Kafka ? I assume you mean "Kafka Streams" when you say this, as Kafka is just a event bus and Flink can be used to read messages from it. The biggest advantage of Flink (IMO) is that you can write the Flink logic once, and reuse it for both batch and stream processing. So if I write a Flink job that consumes a Kafka stream and produces some aggregated outputs, that same job can be run against my data lake in S3/GCS/Azure Blob Storage, etc. Kafka Streams does not support batch processing, or working on top of anything other than Kafka. Flink supports building on top of other message buses like Pulsar as well: https://flink.apache.org/news/2019/11/25/query-pulsar-streams-using-apache-flink.html https://flink.apache.org/news/2019/11/25/query-pulsar-stream...

dikei 5y ago
Actually, unified batch and stream processing in Flink is a bit of false advertisement. In Flink, stream and batch have different API (DataStream vs DataSet) so reusing logic is not really practical, unless you abstract out everything, in which case you might as well use Spark which is faster than DataSet API. Flink's developers is trying to get rid of DataSet API and move to DataStream and Table API for everything, but it's not done yet.

dikei 5y ago

Advantage of Flink over Kafka Stream * Flink work on other message queue tech other than Kafka like Amazon SQS, Pulsar, etc.. * Back when I last read about Kafka Stream, Flink has better support for stateful processing, the entire state of execution can be safely snapshot into storage and resume at any time. * Kafka Stream shuffle data is slower, because it has to send data to a new topic in the broker, instead of sending it directly between compute node. Advantage of Kafka Stream over Flink * Kafka Stream deployment is simple, just start the jar like any other Java program, scaling up by running the jar multiple time. On the other hand, Flink need a mature orchestration framework like YARN, Meso or K8S, trying to manage a Flink deployment without them is very painful. * Flink require a central, persistent storage like HDFS or S3 for its checkpoint mechanism; Kafka stream doesn't.