4 ms·

I'm truly shocked that there was not one person in their crew with the capacity to realize that the very existence of an "availability zone" indicates that you

I'm truly shocked that there was not one person in their crew with the capacity to realize that the very existence of an "availability zone" indicates that you should spread your resources across several of them. How many other "lessons" are waiting to pounce on them? They really need to cultivate their sense of paranoia if they plan to deliver consistent value to their clients.

nethergoat 17y ago
Agreed, that's a whole lot of eggs in one basket. I'd be interested to hear their reason for not spreading across multiple availability zones. My first thought was that m2.2xlarge nodes, since they're relatively new, might still be restricted to a single AZ; on further investigation, however, this does not appear to be the case.
jgilliam 17y ago
My guess is the reason heroku hasn't done it yet is because of latency issues between availability zones. Ping times can be 6x between availability zones, vs. within it. http://orensol.com/2009/05/24/network-latency-inside-and-across-amazon-ec2-availability-zones/ http://orensol.com/2009/05/24/network-latency-inside-and-acr... More and more cloud services are running on AWS deliberately so there is very low latency. I can host a SOLR index on one service, a MongoDB on another service, and still get reasonable performance.
- houseabsolute 17y ago
  Yeah, it's expected that you're going to get microsecond pings within the same building and millisecond pings to buildings across the country. I guess I can admit that they may have had their reasons, but they can't be good enough to justify this lapse. If it's as inexpensive to run this setup as the article says, they should be able to afford two completely separate stacks of servers, one in each availability zone. It's not going to be easy to figure out how to sync the data across each one, but the fact remains that you have to exist at n+1 at least if you're going to deliver reliability.
aminuit 17y ago
You run a hosted application environment. You experience an outage that affects all of your customers. Do you a) admit that you encountered a problem that sysadmins have known about since the dawn of the Internet, or b) spin it as a novel web 2.0 problem ("unknown unknown", seriously?), solvable only by your crack team of cloud computing experts? For those of us that still host our own servers, it's a little bit frustrating to see this spun as something new. It's not. If you design services that require high uptime, you incorporate the notion of losing a datacenter, which is effectively what happened here.

nethergoat 17y ago

Agreed, that's a whole lot of eggs in one basket. I'd be interested to hear their reason for not spreading across multiple availability zones. My first thought was that m2.2xlarge nodes, since they're relatively new, might still be restricted to a single AZ; on further investigation, however, this does not appear to be the case.

jgilliam 17y ago

My guess is the reason heroku hasn't done it yet is because of latency issues between availability zones. Ping times can be 6x between availability zones, vs. within it. http://orensol.com/2009/05/24/network-latency-inside-and-across-amazon-ec2-availability-zones/ http://orensol.com/2009/05/24/network-latency-inside-and-acr... More and more cloud services are running on AWS deliberately so there is very low latency. I can host a SOLR index on one service, a MongoDB on another service, and still get reasonable performance.

houseabsolute 17y ago
Yeah, it's expected that you're going to get microsecond pings within the same building and millisecond pings to buildings across the country. I guess I can admit that they may have had their reasons, but they can't be good enough to justify this lapse. If it's as inexpensive to run this setup as the article says, they should be able to afford two completely separate stacks of servers, one in each availability zone. It's not going to be easy to figure out how to sync the data across each one, but the fact remains that you have to exist at n+1 at least if you're going to deliver reliability.

aminuit 17y ago

You run a hosted application environment. You experience an outage that affects all of your customers. Do you a) admit that you encountered a problem that sysadmins have known about since the dawn of the Internet, or b) spin it as a novel web 2.0 problem ("unknown unknown", seriously?), solvable only by your crack team of cloud computing experts? For those of us that still host our own servers, it's a little bit frustrating to see this spun as something new. It's not. If you design services that require high uptime, you incorporate the notion of losing a datacenter, which is effectively what happened here.