4 ms·

Everyone should take a page from the book of Netflix right now. It's pretty embarrassing to be anyone that's entirely down and can't do a thing about it due to

by zeeg 14y ago

Everyone should take a page from the book of Netflix right now. It's pretty embarrassing to be anyone that's entirely down and can't do a thing about it due to an EC2 outage. How do you explain to your customers/users/etc that you were down and have absolutely no control of when you will be back online? How can you explain it to yourself?

bcjordan 14y ago
Netflix's "Lessons Learned" from the April 2011 AWS outage: http://techblog.netflix.com/2011/04/lessons-netflix-learned-from-aws-outage.html http://techblog.netflix.com/2011/04/lessons-netflix-learned-... Though they're not 100% degradation proof today either: http://i.imgur.com/MJfqj.png http://i.imgur.com/MJfqj.png
harryh 14y ago
Because this shit is really hard especially when you're trying to build a product at the same time. It's not like you can just wake up one day and say "I'm gonna go build a fully fault tolerant distributed system that works across multiple data centers!" and then you're done by the time you go to sleep. Go actually talk to some Netflix engineers. They'll tell you the same thing.
- zeeg 14y ago
  That's my point exactly. Everyone relies entirely (almost) on EC2 for mission critical business, and then they're left there with no outs as soon as a big outage occurs.
- flyt 14y ago
  this. stop talking about how great a MVP is and then complain when people haven't build multi region distributed services that are fault tolerant to major platform outages.
- arohner 14y ago
  Yes, you're absolutely right. However, Netflix is distributed across multiple AZ, while Heroku has spent the last two years after their $212MM acquisition in the same AZ. That makes it sound like Netflix has a more reliable platform than the PaaS company.
Cushman 14y ago
"We're sorry about the current downtime. We know some of you are frustrated, so we thought we'd take a moment to explain why this happened. "Running a web server is very expensive. After we've built the site, if we want to keep it running, someone needs to be on-call 24 hours a day. That means at least one full-time staff member who does nothing else-- more if we want them to stay sane. "To save us and you some money, we've contracted maintenance of our servers out to a third-party service. This is great for us, since they run it more reliably than we could, and it's great for you, because it costs a lot less. But the downside is that things still break sometimes, and when they do, it's completely out of our hands. We're left waiting for things to get fixed just like you are. "So we understand your frustration; we're frustrated too. But unfortunately, downtimes do happen. Guaranteeing our service 100% of the time would cost hundreds of thousands of extra dollars per year, and for most of our users, that's simply not worth the cost. Our provider guarantees 99.[nines]% reliability for much less money, and this is the 0.01%. "If you have something that absolutely must get done, shoot us an email right now and we'll take care of it for you as soon as the site is active again. "Although this is technically out of our hands, we aren't trying to shift the blame; we made this decision with open eyes, and we stand by that decision. Again, we sincerely apologize for the temporary inconvenience. We hope we can make it up to you with some new features we'll be rolling out this month :)" It doesn't have to be that hard.
- geoka9 14y ago
  Yes, it's actually easier to apologize to your customers when you have AWS to blame for the outage. After all, you're in good company ("even Heroku is down - what do you want from us?")
  - DeepDuh 14y ago
    You're lucky if your average customer knows what AWS is, let alone Heroku.
    - geoka9 14y ago
      I'm sure your average customer has heard of Amazon :)
mnutt 14y ago
Maybe you tell them that they won't be able to trade cat pictures for some time. And that the outage happened because you didn't overinvest in infrastructure so that you could keep prices down and roll out some new features faster. Even if you aren't a cat picture site, many startups can deal with some downtime occasionally and it's the right tradeoff to make.

by zeeg 14y ago

bcjordan 14y ago

Netflix's "Lessons Learned" from the April 2011 AWS outage: http://techblog.netflix.com/2011/04/lessons-netflix-learned-from-aws-outage.html http://techblog.netflix.com/2011/04/lessons-netflix-learned-... Though they're not 100% degradation proof today either: http://i.imgur.com/MJfqj.png http://i.imgur.com/MJfqj.png

harryh 14y ago

Because this shit is really hard especially when you're trying to build a product at the same time. It's not like you can just wake up one day and say "I'm gonna go build a fully fault tolerant distributed system that works across multiple data centers!" and then you're done by the time you go to sleep. Go actually talk to some Netflix engineers. They'll tell you the same thing.

zeeg 14y ago
That's my point exactly. Everyone relies entirely (almost) on EC2 for mission critical business, and then they're left there with no outs as soon as a big outage occurs.
flyt 14y ago
this. stop talking about how great a MVP is and then complain when people haven't build multi region distributed services that are fault tolerant to major platform outages.
arohner 14y ago
Yes, you're absolutely right. However, Netflix is distributed across multiple AZ, while Heroku has spent the last two years after their $212MM acquisition in the same AZ. That makes it sound like Netflix has a more reliable platform than the PaaS company.

Cushman 14y ago

"We're sorry about the current downtime. We know some of you are frustrated, so we thought we'd take a moment to explain why this happened. "Running a web server is very expensive. After we've built the site, if we want to keep it running, someone needs to be on-call 24 hours a day. That means at least one full-time staff member who does nothing else-- more if we want them to stay sane. "To save us and you some money, we've contracted maintenance of our servers out to a third-party service. This is great for us, since they run it more reliably than we could, and it's great for you, because it costs a lot less. But the downside is that things still break sometimes, and when they do, it's completely out of our hands. We're left waiting for things to get fixed just like you are. "So we understand your frustration; we're frustrated too. But unfortunately, downtimes do happen. Guaranteeing our service 100% of the time would cost hundreds of thousands of extra dollars per year, and for most of our users, that's simply not worth the cost. Our provider guarantees 99.[nines]% reliability for much less money, and this is the 0.01%. "If you have something that absolutely must get done, shoot us an email right now and we'll take care of it for you as soon as the site is active again. "Although this is technically out of our hands, we aren't trying to shift the blame; we made this decision with open eyes, and we stand by that decision. Again, we sincerely apologize for the temporary inconvenience. We hope we can make it up to you with some new features we'll be rolling out this month :)" It doesn't have to be that hard.

geoka9 14y ago
Yes, it's actually easier to apologize to your customers when you have AWS to blame for the outage. After all, you're in good company ("even Heroku is down - what do you want from us?")
- DeepDuh 14y ago
  You're lucky if your average customer knows what AWS is, let alone Heroku.
  - geoka9 14y ago
    I'm sure your average customer has heard of Amazon :)

mnutt 14y ago

Maybe you tell them that they won't be able to trade cat pictures for some time. And that the outage happened because you didn't overinvest in infrastructure so that you could keep prices down and roll out some new features faster. Even if you aren't a cat picture site, many startups can deal with some downtime occasionally and it's the right tradeoff to make.