4 ms·

Not piling on AWS. These things happen and I’m sure everyone involved is working to improve things. Yes, everyone should deploy in multiple availability zones.

Not piling on AWS. These things happen and I’m sure everyone involved is working to improve things. Yes, everyone should deploy in multiple availability zones. My 2 cents. Outages happen. Network glitches happen. Bad configs and bad updates happen. However, power issues should not really happen. One of the primary cost saving areas of going to the cloud is not having to do on-prem power, such as UPS, generators, maintenance etc. Not having to do on-prem cooling is another thing. These should be solved from the customer’s perspective when going into a professional data center and are the things you don’t want to worry about anymore.

kazen44 5y ago
Getting power right in a DC takes a ton of preventative maintenance. (doing black building tests, make sure there are working load tests which occur frequently, managing UPS battery health etc).
- spenczar5 5y ago
  Yes, it's hard work... but it's work AWS should be great at. Like, really great at. I used to work at AWS. The most worked-up I ever saw Charlie Bell (the de facto head of AWS engineering) was in the weekly operations meeting, going over a postmortem which described a near-miss as several generators failed to start properly in a power interruption. In this meeting - with nearly 1,000 participants! - Charlie got increasingly irate about the fact that this had happened. For the next few weeks, details of some sort of flushing subsystem for backup generators was an every-time topic. Sadly, Charlie left AWS and now works at Microsoft. I can't help but wonder what that operations meeting is like now without him.
- Justin_K 5y ago
  This is what we've been paying for and they've failed us this time.
ksec 5y ago
Yes. I expect software and network glitches because they are complicated and error prone. But I fully expect hardware and power redundancy. Even if they were down it shouldn't "kill" them. At least not on AWS, the biggest and arguably best DC running in the world. But it did. This suggest to me DataCenter, as an infrastructure and building on itself still have plenty of room for improvements.

kazen44 5y ago

Getting power right in a DC takes a ton of preventative maintenance. (doing black building tests, make sure there are working load tests which occur frequently, managing UPS battery health etc).

spenczar5 5y ago
Yes, it's hard work... but it's work AWS should be great at. Like, really great at. I used to work at AWS. The most worked-up I ever saw Charlie Bell (the de facto head of AWS engineering) was in the weekly operations meeting, going over a postmortem which described a near-miss as several generators failed to start properly in a power interruption. In this meeting - with nearly 1,000 participants! - Charlie got increasingly irate about the fact that this had happened. For the next few weeks, details of some sort of flushing subsystem for backup generators was an every-time topic. Sadly, Charlie left AWS and now works at Microsoft. I can't help but wonder what that operations meeting is like now without him.
Justin_K 5y ago
This is what we've been paying for and they've failed us this time.

ksec 5y ago

Yes. I expect software and network glitches because they are complicated and error prone. But I fully expect hardware and power redundancy. Even if they were down it shouldn't "kill" them. At least not on AWS, the biggest and arguably best DC running in the world. But it did. This suggest to me DataCenter, as an infrastructure and building on itself still have plenty of room for improvements.