4 ms·

I've been cabling servers, kind of for a hobby, for a couple of years now. Even a two-server-per-U cabling situation (with redundant ports on each server) is a

by kabdib 12y ago

I've been cabling servers, kind of for a hobby, for a couple of years now. Even a two-server-per-U cabling situation (with redundant ports on each server) is a nightmare. Once you're talking hundreds of wires in a rack, it's no longer fun. I can't imagine going to top-of-rack directly from (say) 512 or 1024 Pis in a rack. So you need intermediate switches, probably a small one every couple of U. From there to top of rack at 10G (could get away with 1G if you know your network bandwidth over your couple U of Pis won't get saturated). Top of rack switch will need to be optical, probably redundant, at 40G aggregate or better. Did I mention that those first-layer switches probably take up a U themselves? Per Pi, we might be spending as much on each switch port as we are on the Pi itself, maybe more (a 48-port switch that we use is about $2k delivered, or about $40/port, and that's just the first switching layer). You can probably buy cheaper switches than the ones we use; I don't know if there is drop in reliability. Haven't figured the cost of the optical links, either, but they can get spendy as well. I think that box of Pis needs its own switching fabric, so that 1G link never leaves the chassis the Pi is in. The switch doesn't need to be fancy, but it looks pretty custom and you'll have to amortize its cost over a big build. I really hate wires :-)

chetanahuja 12y ago
hmm... yeah that's what I thought. And yes, the best case might be have hundreds of these "systems on a chip" on a custom board with it's internal bus (PCI-whatever) with avirtual eth0's visible on each internal system. This whole rigmarole could be connected to the rest of the network via a couple of 10G networks. Then some flavor of SDN would work to divide the bandwidth in a fair manner among the boxes. But doing all that custom electronics does take the fun out of the idea of "just a bunch of cheap raspberry pi's doing their thing". So maybe not.
astrodust 12y ago
It is possible that future servers might be connected to the network backplane, power, drives, and pretty much everything else by multiple USB-3.1 type connections. The bandwidth is there. Just imagine racking in machines like that which take N USB connections, where that's 1, 2, 4, 8, 16 or whatever is necessary.
- kabdib 12y ago
  Hmm. USB fan-out is pretty cheap (the cabling is not fussy). Schedule maybe 70% of the raw bandwidth and you should have enough to drive 6-7 servers at 1GBit from each root hub. If you know your servers are less chatty you can get away with less guaranteed bandwidth. Can also play games with isoch. You can even add redundancy by connecting each Pi server to more than one root hub. Writing the USB-based switch for this would be fun (probably someone has done this, though).

by kabdib 12y ago

chetanahuja 12y ago

hmm... yeah that's what I thought. And yes, the best case might be have hundreds of these "systems on a chip" on a custom board with it's internal bus (PCI-whatever) with avirtual eth0's visible on each internal system. This whole rigmarole could be connected to the rest of the network via a couple of 10G networks. Then some flavor of SDN would work to divide the bandwidth in a fair manner among the boxes. But doing all that custom electronics does take the fun out of the idea of "just a bunch of cheap raspberry pi's doing their thing". So maybe not.

astrodust 12y ago

It is possible that future servers might be connected to the network backplane, power, drives, and pretty much everything else by multiple USB-3.1 type connections. The bandwidth is there. Just imagine racking in machines like that which take N USB connections, where that's 1, 2, 4, 8, 16 or whatever is necessary.

kabdib 12y ago
Hmm. USB fan-out is pretty cheap (the cabling is not fussy). Schedule maybe 70% of the raw bandwidth and you should have enough to drive 6-7 servers at 1GBit from each root hub. If you know your servers are less chatty you can get away with less guaranteed bandwidth. Can also play games with isoch. You can even add redundancy by connecting each Pi server to more than one root hub. Writing the USB-based switch for this would be fun (probably someone has done this, though).