4 ms·

Is this reasonable? I'm sure a lot of amazon hosted companies are thinking similar thoughts. But being on multiple Availability Zones was supposed to be bullet

Is this reasonable? I'm sure a lot of amazon hosted companies are thinking similar thoughts. But being on multiple Availability Zones was supposed to be bulletproof (according to Amazon). Now that we know that wasn't the case, is being hosted on multiple regions going to provide the necessary level of protection? Is it an over-reaction to say that relying completely on Amazon could now be seen as irresponsible to your users, given the magnitude of this event?

PakG1 15y ago
The one problem I have with such concerns is this: what other viable options are there? Google AppSpot? Windows Azure? Perhaps. But AWS is flexible, very few stack limitations. The only other alternative I think is to go back to the pre-cloud era, when hosting was much more expensive, and outages were still possible, especially when you couldn't keep up with big traffic spikes. Honestly, I would prefer this kind of mass outage than the alternative. It's cheaper, easier, and I bet you there's still better uptime overall.
- tmeasday 15y ago
  Agreed. On both counts. However obviously it's good PR, and we all appreciate the Mea Cupla from Heroku, the fact is, they are proposing to migrate to a situation where they are still completely reliant on AWS for their hosting. I'm just not sure you can really say "We don’t want to ever put our customers through something like this again and we’re working as hard as we can on making sure that we won’t ever have to.", when at the end of the day, you are again relying on a company that has failed you in the past. Not trying to attack Amazon or Heroku, I'm honestly intrigued by this issue; not to mention the fact that we are facing the exact same decision at work.
  - Pahalial 15y ago
    Interesting that this is essentially all stemming from yet again a communication failure from AWS. Once they have a post-mortem and can explain the multi-AZ issue, we may have a better idea of whether multi-region spread is sufficient redundancy. Or they could completely fail to communicate enough information, and adequately wary customers will be left with no choice but to assume that regions are not sufficiently independent. Wait and see it is...
  - hallmark 15y ago
    Regarding Heroku's plan to continue relying on a company, Amazon, that failed them before: If Heroku evolves to an architecture in which they utilize multiple AWS regions (as they mention in lesson #1 of their post-mortem) and if each region has a distinctly partitioned API "control plane," this should result in a materially improved availability situation for Heroku. EC2 Availability Zones guard against machine, power, and building failures. EC2 Regions should theoretically guard against API infrastructure and AWS software code failures. Heroku need not necessarily ditch their current single-IaaS-provider architecture in order to achieve significantly better control over their service's uptime. On the other hand, when downtime does occur, the ability for Heroku to prioritize their incident response manpower to first handle paying customers has its limits based on their downstream dependencies. If all the broken bits are within Amazon's black box, Heroku doesn't have much control over prioritization (Amazon fixes your stuff whenever it gets around to fixing your stuff). If Heroku operated over multiple cloud providers, even with the added complexity of such an approach, at least Heroku would have control over choosing which of their most important customers to migrate first to a working cloud, away from a broken and black box cloud. In the end, I certainly don't see these considerations as simple. It's easy to cry when things go wrong, but I think the level of scalability and availability that has been achieved up to the present is quite noteworthy.
- richardw 15y ago
  I'm currently trying pretty hard to get one support answer out of Google to help prevent 6 hours of downtime next week. Go with the smaller companies who will at least respond when you need it.
  - PakG1 15y ago
    Human contact is not exactly one of Google's strengths in the first place though. :)
  - upinsmoke 15y ago
    Go HRD, much better.
    - richardw 15y ago
      Agreed. I'm trying to, but need them to link my old app to the new one for SSL API purposes, but that means a mystical form-filling-waiting-might-happen-might-not. I'll focus on the things that I can change while I wait!
- ry0ohki 15y ago
  The other option would of course to be old school and go "non-cloud" and buy their own hardware that they manage, but this is an old man talking.
- upinsmoke 15y ago
  You are comparing IaaS to PaaS. They are not the same.
  - PakG1 15y ago
    Fair enough, point well taken. But I don't think acknowledging that difference changes the crux of the Q&A here. What IaaS does it better than AWS? Seriously. It's not a surprise that Heroku, Engineyard, and various other PaaS options are built on AWS.
crb 15y ago
There is a case for considering "regions" as analogous to "different clouds". Amazon's regions are another cloud, with some tools for migrating content between them, and the same API provided at both [1]. You have only shared public Internet connectivity between them. AWS could say "sorry, we're shutting down, effective 10pm tonight." What do you do then? Pull your data out of the cloud, and put it somewhere else. Overall, the industry deem the risk of this happening as very low. Even though their regions are completely separate, they presumably run the same code, and a bug in their API could see all the instances shutting down at once. Again, reasonably low risk. Up until last week, the industry, based on Amazon's word and their track record, deemed the risk of an entire region failing as very low also. How can you mitigate this? Run across two clouds - either two regions, or two completely separate providers. Does it double your cost? Potentially. You decide how much of the risk you are prepared to accept, and you build that level of redundancy into your architecture. [1] Within reason: new features tend to come in at us-east-1, and then make their way outwards after that.
seandougall 15y ago
We could just as easily say that relying completely on Heroku is irresponsible to our own users. Just as Heroku took responsibility for the unexpected weaknesses their reliance on a single region created, I believe their customers should take responsibility for the unexpected weaknesses our reliance on a single hosting provider has created. Heroku still has the value of added resiliency, even if it's not 110% bulletproof. Ultimately, we're responsible for the architecture design of our own sites.

PakG1 15y ago

The one problem I have with such concerns is this: what other viable options are there? Google AppSpot? Windows Azure? Perhaps. But AWS is flexible, very few stack limitations. The only other alternative I think is to go back to the pre-cloud era, when hosting was much more expensive, and outages were still possible, especially when you couldn't keep up with big traffic spikes. Honestly, I would prefer this kind of mass outage than the alternative. It's cheaper, easier, and I bet you there's still better uptime overall.

tmeasday 15y ago
Agreed. On both counts. However obviously it's good PR, and we all appreciate the Mea Cupla from Heroku, the fact is, they are proposing to migrate to a situation where they are still completely reliant on AWS for their hosting. I'm just not sure you can really say "We don’t want to ever put our customers through something like this again and we’re working as hard as we can on making sure that we won’t ever have to.", when at the end of the day, you are again relying on a company that has failed you in the past. Not trying to attack Amazon or Heroku, I'm honestly intrigued by this issue; not to mention the fact that we are facing the exact same decision at work.
- Pahalial 15y ago
  Interesting that this is essentially all stemming from yet again a communication failure from AWS. Once they have a post-mortem and can explain the multi-AZ issue, we may have a better idea of whether multi-region spread is sufficient redundancy. Or they could completely fail to communicate enough information, and adequately wary customers will be left with no choice but to assume that regions are not sufficiently independent. Wait and see it is...
- hallmark 15y ago
  Regarding Heroku's plan to continue relying on a company, Amazon, that failed them before: If Heroku evolves to an architecture in which they utilize multiple AWS regions (as they mention in lesson #1 of their post-mortem) and if each region has a distinctly partitioned API "control plane," this should result in a materially improved availability situation for Heroku. EC2 Availability Zones guard against machine, power, and building failures. EC2 Regions should theoretically guard against API infrastructure and AWS software code failures. Heroku need not necessarily ditch their current single-IaaS-provider architecture in order to achieve significantly better control over their service's uptime. On the other hand, when downtime does occur, the ability for Heroku to prioritize their incident response manpower to first handle paying customers has its limits based on their downstream dependencies. If all the broken bits are within Amazon's black box, Heroku doesn't have much control over prioritization (Amazon fixes your stuff whenever it gets around to fixing your stuff). If Heroku operated over multiple cloud providers, even with the added complexity of such an approach, at least Heroku would have control over choosing which of their most important customers to migrate first to a working cloud, away from a broken and black box cloud. In the end, I certainly don't see these considerations as simple. It's easy to cry when things go wrong, but I think the level of scalability and availability that has been achieved up to the present is quite noteworthy.
richardw 15y ago
I'm currently trying pretty hard to get one support answer out of Google to help prevent 6 hours of downtime next week. Go with the smaller companies who will at least respond when you need it.
- PakG1 15y ago
  Human contact is not exactly one of Google's strengths in the first place though. :)
- upinsmoke 15y ago
  Go HRD, much better.
  - richardw 15y ago
    Agreed. I'm trying to, but need them to link my old app to the new one for SSL API purposes, but that means a mystical form-filling-waiting-might-happen-might-not. I'll focus on the things that I can change while I wait!
ry0ohki 15y ago
The other option would of course to be old school and go "non-cloud" and buy their own hardware that they manage, but this is an old man talking.
upinsmoke 15y ago
You are comparing IaaS to PaaS. They are not the same.
- PakG1 15y ago
  Fair enough, point well taken. But I don't think acknowledging that difference changes the crux of the Q&A here. What IaaS does it better than AWS? Seriously. It's not a surprise that Heroku, Engineyard, and various other PaaS options are built on AWS.

crb 15y ago

There is a case for considering "regions" as analogous to "different clouds". Amazon's regions are another cloud, with some tools for migrating content between them, and the same API provided at both [1]. You have only shared public Internet connectivity between them. AWS could say "sorry, we're shutting down, effective 10pm tonight." What do you do then? Pull your data out of the cloud, and put it somewhere else. Overall, the industry deem the risk of this happening as very low. Even though their regions are completely separate, they presumably run the same code, and a bug in their API could see all the instances shutting down at once. Again, reasonably low risk. Up until last week, the industry, based on Amazon's word and their track record, deemed the risk of an entire region failing as very low also. How can you mitigate this? Run across two clouds - either two regions, or two completely separate providers. Does it double your cost? Potentially. You decide how much of the risk you are prepared to accept, and you build that level of redundancy into your architecture. [1] Within reason: new features tend to come in at us-east-1, and then make their way outwards after that.

seandougall 15y ago

We could just as easily say that relying completely on Heroku is irresponsible to our own users. Just as Heroku took responsibility for the unexpected weaknesses their reliance on a single region created, I believe their customers should take responsibility for the unexpected weaknesses our reliance on a single hosting provider has created. Heroku still has the value of added resiliency, even if it's not 110% bulletproof. Ultimately, we're responsible for the architecture design of our own sites.