4 ms·
Workloads have to use the network to do the following : 1. Shift data onto HDFS (obviously), this is rare though 2. Write results after reduce (or you can't re
by sgt101 13y ago
Workloads have to use the network to do the following :
1. Shift data onto HDFS (obviously), this is rare though
2. Write results after reduce (or you can't read them) when you are doing ETL this can be significant; you are sorting out an image for a production system and this is non-trivial abouts of data (100's GB at least)
3. Shuffle; this is the worry (for me) MR jobs can shuffle a lot of data around pre reduce.