3 ms·

The differences are mainly around batch-centric vs. streaming-centric model and executions. Here is for example a video walkthrough by MapR: https://www.mapr.c

• faizshah 11y ago
  Another major difference is that Flink's scheduler doesn't have a notion of data locality like Spark's. If you want to use data local to the node you have to query whatever you're storing your stuff in (HDFS) and filter those items that aren't on the node.
  • sewen 11y ago
    That works a bit differently in Flink and Spark. Inside a data flow program, the scheduler tries to schedule as local as possible. For the inputs to a streaming program (for example Kafka partitions), there is currently no locality consideration, but the locality there changes throughout a program lifetime anyways (brokers change leadership and rebalance) Flink's DataSet API does assign data to tasks after the scheduling. That assignment respects locality, actually. That lazy assignment makes it possible to handle large numbers of small files, for example.
    • faizshah 11y ago
      Hmm, I didn't know that about the DataSet API. When I looked at the scheduler's code it didn't seem to have any notion of data locality except for colocation of vertices. I'll take a look at the DataSet API's code though, thanks!