Hrm. My speculative presumption was going off the sole datapoint of "we have not completely restarted the index subsystem or the placement subsystem in our lar

Hrm. My speculative presumption was going off the sole datapoint of "we have not completely restarted the index subsystem or the placement subsystem in our larger regions for many years". I'm not quite sure how to interpret "restart" in this context, mostly due to lack of exposure or experience. The report also says "Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. The servers that were inadvertently removed supported two other S3 subsystems." So you're right, it looks like multiple servers were supporting these systems, which does make sense (especially considering the load they would have seen). Okay. I guess I didn't quite think through the load requirements and thought these were single machines - which is certainly ludicrous thinking :) - and that's where I got the SPOF reasoning from. You're very right though, these consensus systems must be built as bottlenecks in order to see everything. And there aren't really any alternatives: "build extra indexers and placement systems!" just gives you "but what if _all_ of them get taken offline?" and "it can't leave the datacenter, it sees 100GB/s of throughput" (number taken out of thin air). Good points.