Imagine you have a rack of servers. Around 40 of them fit in a rack, you equip each of them with 10Gb ethernet, they're running HDFS. You plug these 40 ethernet

Imagine you have a rack of servers. Around 40 of them fit in a rack, you equip each of them with 10Gb ethernet, they're running HDFS. You plug these 40 ethernet cables into one switch, so they can all talk to eachother, serving files to its clients, shuffling segments around etc. Now you want to connect that rack to another rack of 40 servers, on another floor of your building. Doing that with 1 10Gb cable, and you do not have full bandwidth between all your 80 servers. Even bonding a handful of 40Gb links will not be enough - and it'll be a lot of cables you have to manage. Doing it with 1 Tb link and you start having full bandwidth. Or just imagine connecting 2 buildings. Fewer cables. Higher bandwidth. The clue is you don't feed all this into 1 server running a commodity CPU and OS, you feed it into blinking boxes that will distribute the data to/from a high speed link onto many lower speed links to reach ordinary servers. Switches/routers can do that at line-rate if you implement this properly in hardware.