7 ms·

Before the doom and gloomers come out, this is the first time since leaving beta I can remember it happening. We left AWS about 18 months ago after one of the

Before the doom and gloomers come out, this is the first time since leaving beta I can remember it happening. We left AWS about 18 months ago after one of the outages and switched to GAE. I've counted 3-4 big downtimes for AWS compared to this one on GAE. That's still a good decision (for now)....

lyddonb 14y ago
Yep. We've never seen anything like this before.
namank 14y ago
Same. Minor updates and issues pop up with GAE but I've never experienced an outage such as this in my two years of use. Wonder what's happening.
tedroden 14y ago
It's one of the first times the whole service has been down, but parts of the service go down at least once a week. memcache, task queues are "elevated" with regularity and urlfetch is frequently down totally. ("elevated" generally means unusable). Of course master/slave even has scheduled downtime.
mikegioia 14y ago
I have yet to experience downtime with RackspaceCloud and I've been using them for like 3 years.
- jmsduran 14y ago
  Me too.
- dchest 14y ago
  I had network issues with my two virtual machines on RackspaceCloud about a year or so ago (something was wrong with their routing). They were resolved quickly, though.
acdha 14y ago
One thing to remember: this took down all of app engine for at least an hour. AWS has had only 17 minutes of downtime affecting all of us-east this year (that network glitch a couple days after PyCon) - the rest of it has been a subset of the service amplified by people rediscovering that they weren't as redundant as they thought. The correct less to draw is that any one point of infrastructure is a risk, so you need to scale wide. This is possible to do with AWS regions, or other providers - even internal bare iron if you're so inclined, but impossible to do with GAE because you're committed to a single-vendor API as well as their infrastructure.
- stickfigure 14y ago
  GAE applications are distributed across multiple data centers[1], so in theory you get "scale wide" automatically. Unfortunately it looks like there was some sort of flaw in the architecture. I believe this is the first systemwide failure of the HRD. The real question is: Can you and your ops team build a "scale wide" system better than Google?[2] How much effort are you willing to put into it, when those development resources could be put into making features instead? [1] For apps which use the high-replication datastore. Old (deprecated) master/slave apps are served out of a single datacenter. [2] This is a reasonably serious question to ask. In GAE, you're one app among many, so a you-specific scaling solution might be easier and more robust than the generalized one Google builds. But not necessarily.
  - acdha 14y ago
    I completely agree that Google is likely to do better than many teams, modulo your second point (which I completely agree with - generic is much harder than specific). For me it really just comes down to the lock-in aspect: with GAE if you decide that Google isn't taking the platform in the right direction for your business you're looking at something close to rewriting your application. This is far from the most likely outcome - although pricing can be interesting - but it's the kind of thing you really want to consciously acknowledge, similar to the way Netflix has accepted the risks of AWS by investing heavily in failure management tools.

lyddonb 14y ago

Yep. We've never seen anything like this before.

namank 14y ago

Same. Minor updates and issues pop up with GAE but I've never experienced an outage such as this in my two years of use. Wonder what's happening.

tedroden 14y ago

It's one of the first times the whole service has been down, but parts of the service go down at least once a week. memcache, task queues are "elevated" with regularity and urlfetch is frequently down totally. ("elevated" generally means unusable). Of course master/slave even has scheduled downtime.

mikegioia 14y ago

I have yet to experience downtime with RackspaceCloud and I've been using them for like 3 years.

jmsduran 14y ago
Me too.
dchest 14y ago
I had network issues with my two virtual machines on RackspaceCloud about a year or so ago (something was wrong with their routing). They were resolved quickly, though.

acdha 14y ago

One thing to remember: this took down all of app engine for at least an hour. AWS has had only 17 minutes of downtime affecting all of us-east this year (that network glitch a couple days after PyCon) - the rest of it has been a subset of the service amplified by people rediscovering that they weren't as redundant as they thought. The correct less to draw is that any one point of infrastructure is a risk, so you need to scale wide. This is possible to do with AWS regions, or other providers - even internal bare iron if you're so inclined, but impossible to do with GAE because you're committed to a single-vendor API as well as their infrastructure.

stickfigure 14y ago
GAE applications are distributed across multiple data centers[1], so in theory you get "scale wide" automatically. Unfortunately it looks like there was some sort of flaw in the architecture. I believe this is the first systemwide failure of the HRD. The real question is: Can you and your ops team build a "scale wide" system better than Google?[2] How much effort are you willing to put into it, when those development resources could be put into making features instead? [1] For apps which use the high-replication datastore. Old (deprecated) master/slave apps are served out of a single datacenter. [2] This is a reasonably serious question to ask. In GAE, you're one app among many, so a you-specific scaling solution might be easier and more robust than the generalized one Google builds. But not necessarily.
- acdha 14y ago
  I completely agree that Google is likely to do better than many teams, modulo your second point (which I completely agree with - generic is much harder than specific). For me it really just comes down to the lock-in aspect: with GAE if you decide that Google isn't taking the platform in the right direction for your business you're looking at something close to rewriting your application. This is far from the most likely outcome - although pricing can be interesting - but it's the kind of thing you really want to consciously acknowledge, similar to the way Netflix has accepted the risks of AWS by investing heavily in failure management tools.