4 ms·

> (...) [W]e have not completely restarted the index subsystem or the placement subsystem in our larger regions for many years. S3 has experienced massive growt

by i336_ 10y ago

> (...) [W]e have not completely restarted the index subsystem or the placement subsystem in our larger regions for many years. S3 has experienced massive growth over the last several years and the process of restarting these services and running the necessary safety checks to validate the integrity of the metadata took longer than expected. All those tweets saying "turn it off and back on again"? "We accidentally turned it off, but it hasn't been turned it off for so long it took us hours to figure out how to turn it back on." Poorly-presented jokes aside, this is rather concerning. The indexer and placement systems are SPOFs!! I mean, I'd presume these subsystems had ultra-low-latency hot failover, but this says they never restarted, and I wonder if AWS didn't simply invest a ton of magic pixie dust in making Absolutely Totally Sure™ the subsystems physically, literally never crashed in years. Impressive engineering but also very scary. At least they've restarted it now. And I'm guessing the current hires now know a lot about the indexer and placer, which won't do any harm to the sharding effort (I presume this'll be being sharded quicksmart). I wonder if all the approval guys just photocopied their signatures onto a run of blank forms, heheh.

richardwhiuk 10y ago
I don't think you don't understand the architecture of the system, if you are describing the indexer as a SPOF. The system is a collection of shards. If you replicate it to create a second shard, then you'll just have a large a system, which is still a single point of failure. The index, by necessity, has to be able to answer the question 'this object exists' or 'this object doesn't exit' - so it needs to have consensus.
- i336_ 10y ago
  Hrm. My speculative presumption was going off the sole datapoint of "we have not completely restarted the index subsystem or the placement subsystem in our larger regions for many years". I'm not quite sure how to interpret "restart" in this context, mostly due to lack of exposure or experience. The report also says "Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. The servers that were inadvertently removed supported two other S3 subsystems." So you're right, it looks like multiple servers were supporting these systems, which does make sense (especially considering the load they would have seen). Okay. I guess I didn't quite think through the load requirements and thought these were single machines - which is certainly ludicrous thinking :) - and that's where I got the SPOF reasoning from. You're very right though, these consensus systems must be built as bottlenecks in order to see everything. And there aren't really any alternatives: "build extra indexers and placement systems!" just gives you "but what if _all_ of them get taken offline?" and "it can't leave the datacenter, it sees 100GB/s of throughput" (number taken out of thin air). Good points.

by i336_ 10y ago

richardwhiuk 10y ago

I don't think you don't understand the architecture of the system, if you are describing the indexer as a SPOF. The system is a collection of shards. If you replicate it to create a second shard, then you'll just have a large a system, which is still a single point of failure. The index, by necessity, has to be able to answer the question 'this object exists' or 'this object doesn't exit' - so it needs to have consensus.

i336_ 10y ago
Hrm. My speculative presumption was going off the sole datapoint of "we have not completely restarted the index subsystem or the placement subsystem in our larger regions for many years". I'm not quite sure how to interpret "restart" in this context, mostly due to lack of exposure or experience. The report also says "Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. The servers that were inadvertently removed supported two other S3 subsystems." So you're right, it looks like multiple servers were supporting these systems, which does make sense (especially considering the load they would have seen). Okay. I guess I didn't quite think through the load requirements and thought these were single machines - which is certainly ludicrous thinking :) - and that's where I got the SPOF reasoning from. You're very right though, these consensus systems must be built as bottlenecks in order to see everything. And there aren't really any alternatives: "build extra indexers and placement systems!" just gives you "but what if _all_ of them get taken offline?" and "it can't leave the datacenter, it sees 100GB/s of throughput" (number taken out of thin air). Good points.