3 ms·

> Any reason on 60 drives instead of 70 or more? One step at a time. :-) We think we can put a few more drives in the motherboard half of the POD without man

> Any reason on 60 drives instead of 70 or more? One step at a time. :-) We think we can put a few more drives in the motherboard half of the POD without many other changes, and connect them to already existing SATA connectors on the motherboard. To get all the way to 75 drives we will need another row of port multipliers, which then means we need even more SATA cards, etc. > No 25/40gig networking? We're currently not able to saturate the 10 Gbit so for us it is wasted money to go faster. Remember that we run these in "vaults of 20" so we have 20 pods EACH with 10 Gbits so the vault hoovers in data at 200 Gbits/sec if you can thread the application. > How tall are your racks? Most of our racks are 45U tall where the PDU takes 4U, so we can fit (not necessarily power) 10 pods per rack and still fit a network switch in the 1U left at the top. In the past (with 45 drive pods) we ran 2 circuits of 30A 208V power (three phase). The two circuits are redundant - our power strips can flip over from one to the other if there is a loss of power on the currently used power. But I think they only put 8 or 9 of the 45 drive pods in a cabinet because it slightly exceeds what the datacenter likes. And now that we're moving up to all 60 drive pods, we have to rejigger the whole thing. It is also worth mentioning that when we bring a 20 pod vault online, we put each of the 20 pods in different racks in different parts of the datacenter. This helps keep the vault completely online if any one component fails. For example, if the power strip feeding the pods in 1 rack power dies, it only takes 1 pod offline out of several different vaults, so ALL the vaults stay online both reading and writing data. (I hope that made sense.)

scurvy 10y ago
Makes sense. Thanks for the reply! As to spreading the vaults around the DC, it sounds like that's more to prepare for ToR switch failure -- its sounds like an ATS handles your PDU and power circuit failover. Yeah it sounds like you're mostly CPU bound calculating EC if you're going to rebalance and replace failed pods so 25 gbps ethernet probably isn't interesting (yet). That said, it's only marginally more expensive than 10gbps twinax. Interesting that you're running 10gig baseT. Interesting from a power and latency perspective, though I'd imagine you're not worried too much about latency.

scurvy 10y ago

Makes sense. Thanks for the reply! As to spreading the vaults around the DC, it sounds like that's more to prepare for ToR switch failure -- its sounds like an ATS handles your PDU and power circuit failover. Yeah it sounds like you're mostly CPU bound calculating EC if you're going to rebalance and replace failed pods so 25 gbps ethernet probably isn't interesting (yet). That said, it's only marginally more expensive than 10gbps twinax. Interesting that you're running 10gig baseT. Interesting from a power and latency perspective, though I'd imagine you're not worried too much about latency.