11 ms·
The article alludes to the technique for dealing with failures inside a sealed underwater container: you never actually replace hardware but instead work around
by gorgoiler 2y ago
The article alludes to the technique for dealing with failures inside a sealed underwater container: you never actually replace hardware but instead work around failures by reprovisioning services onto different nodes. Over time more and more nodes will fail until the container becomes exhausted and needs replacing.
That therefore begs the question: what are the narrowest points of failure in a system like this? For example it would be naive to have 140 fungible compute nodes if they all went through a single networking switch. Node workloads can be moved but a switch failure makes the entire container useless.
This works physically as well. Marine vessels have sealed bulkheads such that a hull breach in one end doesn’t cause flooding in the other end.
How do these vessels fan out all the other points of failure and what are the error rates in those components?
- wongarsu 2y agoIt's worth noting that redundant networking gear and redundant power are the norm even in normal data centers. Having an entire row of racks go offline because of a router failure is already considered unacceptable and well-mitigated. Their biggest point of failure is probably the land connection. Looks like it's one big cable bundle. Those get damaged all the time from sea life, fishing, people throwing anchors where they shouldn't etc. Land-based datacenters have the same issue with excavators damaging power or network lines, but it's easy to run multiple lines far apart from each other when you are on land.