4 ms·

Lessons We’ve Learned Using AWS (2010)

80 points by denzil_correa 10y ago

mtdewcmu 10y ago
I worked on a web store once that depended on a couple of 3rd party web services and set up subscription billing. The project turned out to be like an iceberg, where 10% of the work went into features, and 90% of the work went into heading off all of the various possible failure modes. When the system failed, people's credit cards got billed the wrong amount and either the store didn't get paid or the customer got overcharged. So I can relate to all of the effort they put into fail-safety.
SFJulie 10y ago
http://www.networkworld.com/article/3037428/cloud-computing/netflix-is-not-really-all-in-on-amazon-s-cloud.html http://www.networkworld.com/article/3037428/cloud-computing/... Well they also learned to not put their most important asset on AWS.
- amaks 10y ago
  Kind of makes sense given the recent outage. At least it looks like it's a good idea to diversify the cloud technologies across several vendors.
- jlgaddis 10y ago
  They'd have to jack up subscription prices 500% to just cover the AWS bandwidth charges!
- nickpsecurity 10y ago
  As in, the mission-critical, high-bandwidth stuff is on distributed servers they manage themselves like the traditional companies do. Unsurprising to we skeptics of going all-in on clouds.
HNLogInShit 10y ago
AWS is a massive pain in the ass. Its documentation is abysmal. Its API is a mess. Its admin tools are cumbersome at best. A company like Netflix will get its own liaison from Amazon, which mitigates the glaring problems with the service. After dicking around with AWS for a couple of months, we set up our own VPS at Linode and never looked back. What a refreshing change, which allowed us to develop a product instead of wade through a not-ready-for-customers back end.
venkasub 10y ago
Architecting applications for the cloud is a non-trivial problem. Lot of forethought needs to go in place while choosing a particular infrastructure component. And the ideas of resiliency, robustness (what can be bucketed into 'graceful-degradation') etc along with the most important Monitoring needs to be put in place right from the initial days of the app, so that rearchitecting is does NOT end up being a costly affair.
- Johnny555 10y ago
  Architecting for a physical data center is also a non-trivial problem, and not all that different than architecting for the cloud.
  - bluejekyll 10y ago
    Couldn't agree more. The cloud just makes it clear where you didn't architect for resiliency properly, in your own DC you work around this by doing things that you can't do in the cloud. But the biggest difference is automation. Because in the cloud you have a full suite of APIs, you can literally automate ever piece of your infrastructure. In your own DC you have to design this stuff to be automated from the ground up, which just making a poor layer2 choice can make that task astronomically more difficult.
skyisblue 10y ago
What happens when multiple regions on AWS are down? Would it be a good idea to also have a complete backup infrastructure on another cloud service?
- edaemon 10y ago
  I think it's best to keep your critical infrastructure portable. It usually doesn't make financial sense to have a hot backup on another cloud service but having a plan to quickly switch it over in case of emergency is a good idea. For example, using something like Terraform could spin up infrastructure on AWS, but in a pinch you could make your templates work for Google, Azure, etc. That said, if your infrastructure is at all complex it can be very difficult to keep it portable.
  - Erwin 10y ago
    Do you know of any known public example of someone who has actually done that? This seems advise similar to eating 6 portions vegetables a day: everyone says it's critical, nobody is doing it. In all the cases of Amazon and Google Cloud outages I've seen that were beyond a single zone, it all hit the high-valued companies hard. I've never seen anyone declare that they decided to activate their alternative provider where they have a continuously updated copy of all the data so the data loss is minimal. Everyone is just assuming Amazon or Google will fix it and that any data loss from activating a non-hot standby is not worth it.
    - samhamilton 10y ago
      I believe wix.com does this, a mix of AWS and Google Cloud. Can't find the blog post where they listed out how they achieved it
    - nickpsecurity 10y ago
      There's products on the market dedicated to abstracting clouds for using more than one or switching easily. I'd be shocked if there weren't companies using them for the former. Those most concerned with availability usually just avoid clouds, though.

80 points by denzil_correa 10y ago

I worked on a web store once that depended on a couple of 3rd party web services and set up subscription billing. The project turned out to be like an iceberg, where 10% of the work went into features, and 90% of the work went into heading off all of the various possible failure modes. When the system failed, people's credit cards got billed the wrong amount and either the store didn't get paid or the customer got overcharged. So I can relate to all of the effort they put into fail-safety.

SFJulie 10y ago

http://www.networkworld.com/article/3037428/cloud-computing/netflix-is-not-really-all-in-on-amazon-s-cloud.html http://www.networkworld.com/article/3037428/cloud-computing/... Well they also learned to not put their most important asset on AWS.

amaks 10y ago
Kind of makes sense given the recent outage. At least it looks like it's a good idea to diversify the cloud technologies across several vendors.
jlgaddis 10y ago
They'd have to jack up subscription prices 500% to just cover the AWS bandwidth charges!
nickpsecurity 10y ago
As in, the mission-critical, high-bandwidth stuff is on distributed servers they manage themselves like the traditional companies do. Unsurprising to we skeptics of going all-in on clouds.

HNLogInShit 10y ago

AWS is a massive pain in the ass. Its documentation is abysmal. Its API is a mess. Its admin tools are cumbersome at best. A company like Netflix will get its own liaison from Amazon, which mitigates the glaring problems with the service. After dicking around with AWS for a couple of months, we set up our own VPS at Linode and never looked back. What a refreshing change, which allowed us to develop a product instead of wade through a not-ready-for-customers back end.

venkasub 10y ago

Architecting applications for the cloud is a non-trivial problem. Lot of forethought needs to go in place while choosing a particular infrastructure component. And the ideas of resiliency, robustness (what can be bucketed into 'graceful-degradation') etc along with the most important Monitoring needs to be put in place right from the initial days of the app, so that rearchitecting is does NOT end up being a costly affair.

Johnny555 10y ago
Architecting for a physical data center is also a non-trivial problem, and not all that different than architecting for the cloud.
- bluejekyll 10y ago
  Couldn't agree more. The cloud just makes it clear where you didn't architect for resiliency properly, in your own DC you work around this by doing things that you can't do in the cloud. But the biggest difference is automation. Because in the cloud you have a full suite of APIs, you can literally automate ever piece of your infrastructure. In your own DC you have to design this stuff to be automated from the ground up, which just making a poor layer2 choice can make that task astronomically more difficult.

skyisblue 10y ago

What happens when multiple regions on AWS are down? Would it be a good idea to also have a complete backup infrastructure on another cloud service?

edaemon 10y ago
I think it's best to keep your critical infrastructure portable. It usually doesn't make financial sense to have a hot backup on another cloud service but having a plan to quickly switch it over in case of emergency is a good idea. For example, using something like Terraform could spin up infrastructure on AWS, but in a pinch you could make your templates work for Google, Azure, etc. That said, if your infrastructure is at all complex it can be very difficult to keep it portable.
- Erwin 10y ago
  Do you know of any known public example of someone who has actually done that? This seems advise similar to eating 6 portions vegetables a day: everyone says it's critical, nobody is doing it. In all the cases of Amazon and Google Cloud outages I've seen that were beyond a single zone, it all hit the high-valued companies hard. I've never seen anyone declare that they decided to activate their alternative provider where they have a continuously updated copy of all the data so the data loss is minimal. Everyone is just assuming Amazon or Google will fix it and that any data loss from activating a non-hot standby is not worth it.
  - samhamilton 10y ago
    I believe wix.com does this, a mix of AWS and Google Cloud. Can't find the blog post where they listed out how they achieved it
  - nickpsecurity 10y ago
    There's products on the market dedicated to abstracting clouds for using more than one or switching easily. I'd be shocked if there weren't companies using them for the former. Those most concerned with availability usually just avoid clouds, though.