4 ms·
That works a bit differently in Flink and Spark. Inside a data flow program, the scheduler tries to schedule as local as possible. For the inputs to a streami
by sewen 11y ago
That works a bit differently in Flink and Spark.
Inside a data flow program, the scheduler tries to schedule as local as possible.
For the inputs to a streaming program (for example Kafka partitions), there is currently no locality consideration, but the locality there changes throughout a program lifetime anyways (brokers change leadership and rebalance)
Flink's DataSet API does assign data to tasks after the scheduling. That assignment respects locality, actually. That lazy assignment makes it possible to handle large numbers of small files, for example.
- faizshah 11y agoHmm, I didn't know that about the DataSet API. When I looked at the scheduler's code it didn't seem to have any notion of data locality except for colocation of vertices. I'll take a look at the DataSet API's code though, thanks!