3 ms·

Can anyone explain how these speeds, if possible, are even useful? For example, the bandwidth of L2 cache on a core i7 is about 70GB/s, which doesn't come close

by ender7 14y ago

Can anyone explain how these speeds, if possible, are even useful? For example, the bandwidth of L2 cache on a core i7 is about 70GB/s, which doesn't come close to Tbit's theoretical 125GB/s.

SoftwareMaven 14y ago
These networks are the ones sitting inside massive datacenters connecting thousands of machines hosting dozens (hundreds by then) of cores. Of course, a single machine isn't going to saturate it, but 5000 running a massive map/reduce will.
iyulaev 14y ago
High-end routers and switches use dedicated h/w (ASICs or FPGAs) to move data, rather than shuffling all of it through a processor. Other, similar limitations exist, but through an FPGA you can likely push (roughly) an order of magnitude more data than through a processor due to a wider internal interconnect.
Danieru 14y ago
L2 caches are optimized for latency and thus make a poor metric for comparing bandwidth. As well these standards have considerable lead time and start off as math and physics. They need to be ready for when hardware is fast enough.
Andys 14y ago
Oh, you only have one machine in a datacentre?
noselasd 14y ago
Imagine you have a rack of servers. Around 40 of them fit in a rack, you equip each of them with 10Gb ethernet, they're running HDFS. You plug these 40 ethernet cables into one switch, so they can all talk to eachother, serving files to its clients, shuffling segments around etc. Now you want to connect that rack to another rack of 40 servers, on another floor of your building. Doing that with 1 10Gb cable, and you do not have full bandwidth between all your 80 servers. Even bonding a handful of 40Gb links will not be enough - and it'll be a lot of cables you have to manage. Doing it with 1 Tb link and you start having full bandwidth. Or just imagine connecting 2 buildings. Fewer cables. Higher bandwidth. The clue is you don't feed all this into 1 server running a commodity CPU and OS, you feed it into blinking boxes that will distribute the data to/from a high speed link onto many lower speed links to reach ordinary servers. Switches/routers can do that at line-rate if you implement this properly in hardware.
- mbubb 14y ago
  I can imagine it as I am building out a 30 node hdfs cluster with 10g BNT switches. How do you design 40 nodes to a rack? Just curious...
  - noselasd 14y ago
    Racks come in all heights, we have 3 42u racks here populted with 1u boxes. There's also many designs around smaller half depth servers, enabling you to fill servers from the front and back.

by ender7 14y ago

Can anyone explain how these speeds, if possible, are even useful? For example, the bandwidth of L2 cache on a core i7 is about 70GB/s, which doesn't come close to Tbit's theoretical 125GB/s.

SoftwareMaven 14y ago

These networks are the ones sitting inside massive datacenters connecting thousands of machines hosting dozens (hundreds by then) of cores. Of course, a single machine isn't going to saturate it, but 5000 running a massive map/reduce will.

iyulaev 14y ago

High-end routers and switches use dedicated h/w (ASICs or FPGAs) to move data, rather than shuffling all of it through a processor. Other, similar limitations exist, but through an FPGA you can likely push (roughly) an order of magnitude more data than through a processor due to a wider internal interconnect.

Danieru 14y ago

L2 caches are optimized for latency and thus make a poor metric for comparing bandwidth. As well these standards have considerable lead time and start off as math and physics. They need to be ready for when hardware is fast enough.

Andys 14y ago

Oh, you only have one machine in a datacentre?

noselasd 14y ago

Imagine you have a rack of servers. Around 40 of them fit in a rack, you equip each of them with 10Gb ethernet, they're running HDFS. You plug these 40 ethernet cables into one switch, so they can all talk to eachother, serving files to its clients, shuffling segments around etc. Now you want to connect that rack to another rack of 40 servers, on another floor of your building. Doing that with 1 10Gb cable, and you do not have full bandwidth between all your 80 servers. Even bonding a handful of 40Gb links will not be enough - and it'll be a lot of cables you have to manage. Doing it with 1 Tb link and you start having full bandwidth. Or just imagine connecting 2 buildings. Fewer cables. Higher bandwidth. The clue is you don't feed all this into 1 server running a commodity CPU and OS, you feed it into blinking boxes that will distribute the data to/from a high speed link onto many lower speed links to reach ordinary servers. Switches/routers can do that at line-rate if you implement this properly in hardware.

mbubb 14y ago
I can imagine it as I am building out a 30 node hdfs cluster with 10g BNT switches. How do you design 40 nodes to a rack? Just curious...
- noselasd 14y ago
  Racks come in all heights, we have 3 42u racks here populted with 1u boxes. There's also many designs around smaller half depth servers, enabling you to fill servers from the front and back.