3 ms·

Why the AWS Crash is a Good Thing.

45 points by shanecox1 15y ago

The cloud computing world is in crisis and its a good thing for everyone in the long term.

Animus7 15y ago
100% uptime is possible, and it's possible today. Nobody complains that their torrents are ever "down". So to correct: 100% uptime is not possible... with a central point of failure such as a datacenter.
- mchusma 15y ago
  Fair enough. I should clarify that it is not impossible, only impossible for now with our existing structures. I think we will get there.
  - sausagefeet 15y ago
    Except Netflix weathered this well... those sites just didn't build their infrastructure out to handle this.
    - mchusma 15y ago
      Cross provider scaling is not easy, but necessary for companies of a different size. I expect Quora (and others) to make some changes after this.
    - tednaleid 15y ago
      Netflix has downtime just not full outages, it didn't work for my family and we've had other downtime with them in the recent past. They even sent us a credit a month or two ago because of the outage. Their infrastructure is such that it doesn't go down for everyone, but it does go down from time to time for a subset of their subscribers.
- aisalwaysa 15y ago
  I think "feasible" is more of the discussion. Interesting how we learned during the 08 financial crisis that everything is correlated in a crash... I think there are similarities to the cloud, Black Swan fat tails if you will.
  - mchusma 15y ago
    I thought about including some mention of this. I'm sure Amazon though this specific crisis as a six-sigma event. Taleb points out in the book Black Swan, the sum of all possible six-sigma events happens a lot more often.
- samatman 15y ago
  Ironically enough, I have an... interesting service provider at this location who aggressively throttles peer connections. I've been using torrific as a workaround, which, in turn, uses AWS. So yes. My torrents are "down".
  - Animus7 15y ago
    Your torrents aren't "down". In the same sense that having a theoretical crappy ISP that blocks GMail doesn't mean your GMail is "down". When we talk about downtime we usually mean a global outage, for most everyone. That's what happened at Amazon. This never happens for an active torrent. And it's all by design.
blhack 15y ago
Twitter goes down all the time, sure, facebook has dropped off, yeah... But neither of these things have ever happened for a full day. If twitter goes out, refreshing a few times (this was true even a couple of years ago when the downtime was really bad) usually lets you get through. The worst I remember seeing was a few hours down. Same goes for facebook. The worst I can recall right now is part of the oauth system bailing out for a few hours. AWS going out like this is an entirely different disaster. Not only was it down for an entire day, but there were a ton of eggs in the basket.
- shanecox1 15y ago
  I expect large companies will rethink scalability and not put all eggs in one basket.
bradgessler 15y ago
While I agree this is a good thing, the central issue in my mind is that AWS lead a lot of folks to believe that spanning their application across multiple availability zones was a way to protect against AWS's own failures. They spoke at YC a few months back and recommended this as a primary strategy for dealing with failure. None of their engineers or staff recommended spanning applications across different physical locations. When all of the dust settles on this mess, thats the one question I'll be demanding from AWS. Also, how will they prevent multiple AZ's from going down simultaneously in the future?
- mchusma 15y ago
  Some of this is evolving, but if it turns out AWS mislead people you are right.
eof 15y ago
s/Why/How/
gcb 15y ago
> Why the AWS Crash is a Good Thing because it didn't happen with me.
dhughes 15y ago
How much do organizations such as Amazon, Facebook, Twitter, Hacker News etc. plan ahead for holidays such as Christmas, Easter, Thanksgiving, Mother's Day, or the last day of school, first day of University? It seems most outages could at least be predicted somewhat just by looking at "cultural patterns", for lack of a better phrase, from my perspective a lot of outages seem to occur during the events I mentioned. Add to that most people and resources may not be available as much or as quickly during the days when it's busier making it even more likely something will happen.

45 points by shanecox1 15y ago

The cloud computing world is in crisis and its a good thing for everyone in the long term.

100% uptime is possible, and it's possible today. Nobody complains that their torrents are ever "down". So to correct: 100% uptime is not possible... with a central point of failure such as a datacenter.

mchusma 15y ago
Fair enough. I should clarify that it is not impossible, only impossible for now with our existing structures. I think we will get there.
- sausagefeet 15y ago
  Except Netflix weathered this well... those sites just didn't build their infrastructure out to handle this.
  - mchusma 15y ago
    Cross provider scaling is not easy, but necessary for companies of a different size. I expect Quora (and others) to make some changes after this.
  - tednaleid 15y ago
    Netflix has downtime just not full outages, it didn't work for my family and we've had other downtime with them in the recent past. They even sent us a credit a month or two ago because of the outage. Their infrastructure is such that it doesn't go down for everyone, but it does go down from time to time for a subset of their subscribers.
aisalwaysa 15y ago
I think "feasible" is more of the discussion. Interesting how we learned during the 08 financial crisis that everything is correlated in a crash... I think there are similarities to the cloud, Black Swan fat tails if you will.
- mchusma 15y ago
  I thought about including some mention of this. I'm sure Amazon though this specific crisis as a six-sigma event. Taleb points out in the book Black Swan, the sum of all possible six-sigma events happens a lot more often.
samatman 15y ago
Ironically enough, I have an... interesting service provider at this location who aggressively throttles peer connections. I've been using torrific as a workaround, which, in turn, uses AWS. So yes. My torrents are "down".
- Animus7 15y ago
  Your torrents aren't "down". In the same sense that having a theoretical crappy ISP that blocks GMail doesn't mean your GMail is "down". When we talk about downtime we usually mean a global outage, for most everyone. That's what happened at Amazon. This never happens for an active torrent. And it's all by design.

blhack 15y ago

Twitter goes down all the time, sure, facebook has dropped off, yeah... But neither of these things have ever happened for a full day. If twitter goes out, refreshing a few times (this was true even a couple of years ago when the downtime was really bad) usually lets you get through. The worst I remember seeing was a few hours down. Same goes for facebook. The worst I can recall right now is part of the oauth system bailing out for a few hours. AWS going out like this is an entirely different disaster. Not only was it down for an entire day, but there were a ton of eggs in the basket.

shanecox1 15y ago
I expect large companies will rethink scalability and not put all eggs in one basket.

bradgessler 15y ago

While I agree this is a good thing, the central issue in my mind is that AWS lead a lot of folks to believe that spanning their application across multiple availability zones was a way to protect against AWS's own failures. They spoke at YC a few months back and recommended this as a primary strategy for dealing with failure. None of their engineers or staff recommended spanning applications across different physical locations. When all of the dust settles on this mess, thats the one question I'll be demanding from AWS. Also, how will they prevent multiple AZ's from going down simultaneously in the future?

mchusma 15y ago
Some of this is evolving, but if it turns out AWS mislead people you are right.

eof 15y ago

s/Why/How/

gcb 15y ago

> Why the AWS Crash is a Good Thing because it didn't happen with me.

dhughes 15y ago

How much do organizations such as Amazon, Facebook, Twitter, Hacker News etc. plan ahead for holidays such as Christmas, Easter, Thanksgiving, Mother's Day, or the last day of school, first day of University? It seems most outages could at least be predicted somewhat just by looking at "cultural patterns", for lack of a better phrase, from my perspective a lot of outages seem to occur during the events I mentioned. Add to that most people and resources may not be available as much or as quickly during the days when it's busier making it even more likely something will happen.