7 ms·

App Engine down

152 points by soofaloofa 14y ago

debacle 14y ago
I'm really happy I don't host in the cloud. How quickly are the cost savings of cloud computing obliterated by PR, customer service, and system administration time when an outage like this occurs?
- tomgallard 14y ago
  Surely hosting yourself exposes you to just as much, if not more risk? Problem in the datacentre where you're co-lo'd, or one of your servers blows up? I think people not trusting the cloud is similar to how people feel safer driving their cars then taking a plane. The stats say the plane's safer, but people prefer being in control. People like the idea of being in control of their servers, even if that means there's hundreds of extra things that can go wrong compared to a cloud provider. We also get a lot more publicity when a cloud provider has an outage as LOTS of sites go down at once. Hardly anyone notices when service X who self-host go down for a few hours...
  - Alan01252 14y ago
    I agree with you. From my past experience any data center is subject to risks. I've witnessed: Power failures. Cross site links being cut due to engineering works. Over heating due to air conditioning failures. Flooding And I've experienced all the above from a very large, very well known, very expensive data center company based in London.
    - acdha 14y ago
      This is true of every data center I've worked with. Also network providers: everyone has downtime and sometimes you learn the hard way that despite being written into your contract someone took the cheap way and ran your “redundant” fiber through the same conduit which a backhoe just tore up.
  - peterwwillis 14y ago
    The "safety" of the cloud is about two things: 1. trusting your service provider, and 2. redundancy. You have to trust your cloud provider. They control everything you do. If their security isn't bulletproof, you're screwed. If their SAN's firmware isn't upgraded properly to deal with some performance issue, you're screwed. If their developers fuck up the API and you can't modify your instances, you're screwed. You have to put complete faith in a secret infrastructure run for hundreds of thousands of clients so there's no customer relationship to speak of. That's just the "trust" issue. Then there's the issue of actual redundancy. It's completely possible to have a network-wide outage for a cloud provider. There will be no redundancy, because their entire system is built to be in unison; one change affects everything. Running it yourself means you know how secure it is, how robust the procedures are, and you can build in real redundancy and real disaster recovery. Do people build themselves bulletproof services like this? Usually not. But if you cared to, you could.
  - debacle 14y ago
    We're coloed across three datacenters spanning the US (one might be in TO I think) and if a datacenter were to go down, we have a hot backup that's no more than 12 hours stale. The only real manual maintenance that we've got is a rolling reimaging of servers based on whatever's in version control, which usually takes a few hours twice a year, but we'd probably do that if we were in the cloud anyway. When you can script away 90% of your system administration tasks, hosting in the cloud doesn't really make a ton of sense.
    - tomgallard 14y ago
      But a DNS based failover is still going to take an hour or so to propagate right (given that a lot of browsers/proxies/DNS servers don't respect TTL very well at all)? And then you end up with a system with stale data, and the mess of trying to reconcile it when your other system comes back up. I'd take an hour long Appengine outage once a year over that anytime!
      - peterwwillis 14y ago
        Your name server or stub resolver is what respects DNS TTL, not your browser or proxy. Everyone - including people hosting on AWS - needs to be able to fail over DNS, if the AWS IP you're using is in a zone that just went down, for example. Any time you have an outage you need to contact your service provider to get an estimate of downtime. If they can't give you one, assume it'll take forever and cut the DNS over. The worst case is some of your users will start to come back online slowly. If you don't cut over, the worst case is all your users are down until whenever the service provider fixes it, and you get to tell your users "we're waiting for someone else to deal with it", which won't make them very happy. 12 hour stale data sounds kind of long to me. 4 hours sounds more reasonable.
        codeka 14y ago
        I've seen plenty of crappy ISP DNS servers ignore TTL values and cache DNS entries for many hours longer than they're supposed to. Unfortunately, it's all too common.
  - modoc 14y ago
    It really depends on how many servers and how good your sys admins are. If you have 1 server in a cheap colo, then yes, the cloud is probably better. If you have a GOOD hosting provider, and build out a redundant at every tier cluster, you can easily beat the major cloud providers' uptime. We run about 11 client clusters across ~250 servers across 3 data centers in the US and Europe. Each of our client's uptime is very very close to 100%, and we've NEVER lost everything, even for 1 second.
- hartleybrody 14y ago
  I don't think you move to the cloud for cost savings. More like time and headache savings.
- bobmyers 14y ago
  Service outages happen to everyone. Thats why everyone advertises 99.99% uptime four nines or five nines. No one can achieve 100%. The good thing about the cloud is you can say, this one isn't my fault.
- untog 14y ago
  I think there is an interesting debate to be had here. When your site goes down at the same time as Tumblr/Reddit/one of the big guys, the damage might not actually be that high. In the minds of many, "the internet" is probably quite broken.
  - debacle 14y ago
    Depends on the site. Most of the traffic we cater to is not referrer driven.
- bsaul 14y ago
  It's funny, i've got the exact opposite reasoning : it's those moments where i can really appreciate the fact that i'm using the cloud : 1/ I don't have to spend the night debugging or replacing broken hardware 2/ It doesn't cost me any time, any additional resource, any support upgrade, any hardware. 3/ No one can blame me or anybody in my team for the fact that it's not working. I don't feel like i'm lacking control, i feel like somebody else is taking care of that really annoying shit that happens all the time, no matter how well you design your system.
  - lurker14 14y ago
    You customers will not be mollified if you tell them that the service you sold is down due to a subcontractor failure.
  - modoc 14y ago
    If you have paying clients, they WILL blame you and your team for the fact it's not working. They don't care who/what the underlying infrastructure is. Also a good hosting company will handle identifying/fixing/replacing bad hardware for you.
    - calinet6 14y ago
      Not to mention sufficient redundancy will ensure that you never see many effects from those hardware failures/power outages/floods/fires/anything else with any reasonable probability. "The cloud is great because I can blame someone else" is obviously a tenuous argument.
- ceejayoz 14y ago
  My colocation provider (Frontier, telecom in 27 states) went down for an hour and a half last night. It's hardly unique to the cloud.
bsaul 14y ago
I would love it so much to see people at google showing all the internal tools they're using to detect and solve this kind of issues. I can only imagine a war room with screens all over the place showing gigantic amount of red flashing lines :) Hope it doesn't last for long though, i was just praising what a good choice app engine has been so far 10 minutes ago...
notreadbyhumans 14y ago
It's a bit nuts that they're hosting the status pages on the same infrastructure.
- harshadura 14y ago
  jz only 1-2 hr down-age ;)
chrisfarms 14y ago
> "App Engine is currently experiencing serving issues. The team is actively working on restoring the service to full strength. Please follow this thread for updates." -- Max Ross (Google) maxr@google.com via googlegroups.com https://groups.google.com/forum/?fromgroups=#!topic/google-appengine-downtime-notify/SMd2pDJsCPo https://groups.google.com/forum/?fromgroups=#!topic/google-a...
davis_m 14y ago
I think this is larger than just GAE. http://internettrafficreport.com/namerica.htm http://internettrafficreport.com/namerica.htm It seems like large portions of the internet are down.
- untog 14y ago
  Yes, Tumblr is experiencing difficulties and AFAIK they don't run anything on App Engine.
- sparkinson 14y ago
  Now those are some interesting graphs.
- FiloSottile 14y ago
  The internet is burning :O Well, seriously what could be a root cause that affects so many nodes?
- bsaul 14y ago
  only 51% of north american internet is working ?? Am i reading this correctly ?
  - seldo 14y ago
    As Ewantoo mentioned, ITR is pretty unreliable. We have found it works as a relative measure -- when their packet loss rises, we do indeed see consumer traffic fall, though usually not by the amount their graphs would indicate.
  - EwanToo 14y ago
    You're reading it correctly, it's just telling you the wrong thing. 51% of a very small self-selecting set of machines are down, mostly small businesses or home users which have obviously been shutdown or renamed. If internettrafficreport automatically removed all devices which returned no results for 7 days, they'd be a slightly more useful resource.
- cfontes 14y ago
tomnewton 14y ago
My Google contact said that 'SRE are all over it. Hope to have more details soon.' but that was about 30 minutes ago. Does tumblr.com use app engine? They're down...
- tszming 14y ago
  @tumblr (https://twitter.com/tumblr/status/261840787350896640 https://twitter.com/tumblr/status/261840787350896640) Tumblr is experiencing network problems following an issue with one of our uplink providers. We will return to full service shortly.
cyberpanther 14y ago
I'm seeing a bunch of Google properties also. Maybe they are running on app engine? Like https://developers.google.com/ https://developers.google.com/
- libria 14y ago
  Also, http://dartlang.org http://dartlang.org http://golang.org http://golang.org, http://code.google.com/codejam/ http://code.google.com/codejam/ http://www.chromeexperiments.com/ http://www.chromeexperiments.com/ Gotta give 'em props for dogfooding.
  - moepstar 14y ago
    Props, really? That's what i _expect_ them to do - otherwise i can't see anyone trusting/using them if even they themselves avoid their own product(s)..
- Sami_Lehtinen 14y ago
  http://sites.google.com/ http://sites.google.com/ was down too. I got monitoring report, but I first assumed it's false. Because false down reports are much more common than Google services being actually down.
aidos 14y ago
I'm surprised that their status system seems to run off the main infrastructure too. Everyone else gets slated for making that mistake.
foolery 14y ago
Yup, all of the HRD apps are down. But the M/S apps are working.
- jis 14y ago
  Well, my one M/S app is also down.
  - aderaynal 14y ago
    I have 2 M/S apps and 3 HRD. They are all down :(
    - cfontes 14y ago
      Mine is also down... :( any one got any position about it ?
  - vitorarins 14y ago
    my one M/S app is up and running.
    - sebbi 14y ago
      same here ... M/S up and running ... all recently migrated HRD apps down :/
davidjgraph 14y ago
Before the doom and gloomers come out, this is the first time since leaving beta I can remember it happening. We left AWS about 18 months ago after one of the outages and switched to GAE. I've counted 3-4 big downtimes for AWS compared to this one on GAE. That's still a good decision (for now)....
- lyddonb 14y ago
  Yep. We've never seen anything like this before.
- namank 14y ago
  Same. Minor updates and issues pop up with GAE but I've never experienced an outage such as this in my two years of use. Wonder what's happening.
- tedroden 14y ago
  It's one of the first times the whole service has been down, but parts of the service go down at least once a week. memcache, task queues are "elevated" with regularity and urlfetch is frequently down totally. ("elevated" generally means unusable). Of course master/slave even has scheduled downtime.
- mikegioia 14y ago
  I have yet to experience downtime with RackspaceCloud and I've been using them for like 3 years.
  - jmsduran 14y ago
    Me too.
  - dchest 14y ago
    I had network issues with my two virtual machines on RackspaceCloud about a year or so ago (something was wrong with their routing). They were resolved quickly, though.
- acdha 14y ago
libria 14y ago
Hm, bad week for the Cloud. Can't even get to the status page; hopefully it's not hosted on App Engine. So going forward, what's the best way to protect against cloud downtime? Have a hot/standby failover with a different provider? Prepare customers' expectations for the possibility of server outages? Do a ton of research, pay $$$ for lots of nines uptime, and lambast the host when they don't deliver?
- bad_user 14y ago
  Downtimes happen regardless, unless you have a lot of money and talent to spend on your own infrastructure and even then it's hard to beat cloud providers like Amazon, or Google, who have more resources and knowledge than you do. The greatest thing about cloud-hosting is that you can just sit by and let them fix it. It usually takes about half an hour, or a couple of hours if the outage is severe, but usually less than the time it takes for an update of DNS records (unless you've got some proxy in front of your IPs, which would be another point of failure). And then, even with these severe outages, the overall monthly uptime is still better than %99.9 and it's really hard to beat that, so just relax and let them fix it.
- davidjgraph 14y ago
  I'd say the answer depends on how fast GAE recovers. If you're building redundancy over multiple clouds, if there's a lot of data: 1) It's very complex and expensive 2) You're looking at DNS to hot failover, in most cases. If GAE can recover in less than 30 minutes and sticks to, say, one outage a year, you just can't justify the kind of cost you're looking at with 2 (seriously, it's a lot of cash).
- josh2600 14y ago
  Build redundancy into your software to deal with single provider failure.
  - Achshar 14y ago
    That is not always a feasible option, specially for young projects with limited capital.
hugofierro 14y ago
I hope it's not due to DiRT Exercises (SRE Disaster Recovery Test). Looking forward to reading the post-mortem report!
ams6110 14y ago
passpack.com seems to be affected.
jsdalton 14y ago
It's really quite remarkable (to be honest, inexcusable is probably a better word) that their status page is failing as well. My expectations for a company with Google's resources and infrastructure are a lot higher than that. Nothing on their Twitter account either: https://twitter.com/app_engine https://twitter.com/app_engine A poor handling of a systems failure in my opinion.
- rdwallis 14y ago
  If you subscribe to: https://groups.google.com/forum/?hl=en&fromgroups#!forum/google-appengine-downtime-notify https://groups.google.com/forum/?hl=en&fromgroups#!forum... They'll email you when issues occur and info becomes available. It took about 30 minutes after the crash for me to receive an email which seems very reasonable.
  - IheartApplesDix 14y ago
    If you find that reasonable, I have a slice in Brooklyn to sell you.
deleted 14y ago
[deleted]
kjhughes 14y ago
What's the earliest sign of trouble you've had? Pingdom reports my GAE-hosted site has been down since 2012-10-26 10:37:38 EST, a bit over an hour now. UPDATE: My site is back. Delayed report from Pingdom says site came back online after 50 minutes. Performance is sketchy still. We're probably not in the clear yet. At least we can now get to the status dash: http://code.google.com/status/appengine http://code.google.com/status/appengine
- warrentr 14y ago
  I think the first tweet i saw was 7:35 PST
deleted 14y ago
[deleted]
- peterwwillis 14y ago
  I can not bear with a wall of text about a note-taking app. Could you pastebin that?
- whalesalad 14y ago
  Whoa did not expect to receive such negative feedback on this post. Was really looking for feedback from the community on a replacement. I'll post this elsewhere I guess.
  - tomgallard 14y ago
    I think the best place would be in an 'Ask HN' thread. I think the community felt your post was off topic- and that's why it was downvoted
abhijitr 14y ago
Meanwhile... Gmail etc are working quite fine. So the claim that if you build on GAE you "take advantage of the same infrastructure used for Google services!!" starts to ring a bit hollow.
- rytis 14y ago
  I wonder if anyone ever believed this claim to be true...
  - wildmXranat 14y ago
    Or if the definition of the word 'same' is somewhat fluid enough to get away with it
- hugoc 14y ago
  Well http://developer.android.com/ http://developer.android.com/ is down, so google does use GAE for some of its properties?
- adolph 14y ago
  Maybe they mean == instead of ===.
- Ironlink 14y ago
  Or perhaps they are older than the affected infrastructure and just haven't been rewritten. Or perhaps they happen to not use the affected resource. In fact, affected services include developers.google.com and code.google.com, and to a lesser extent youtube.com.
- Tobu 14y ago
  I know the claim of running Amazon.com on AWS was initially bogus, but did anyone notice trouble on that site with the recent AWS outages? It would be instructive to know how much they have integrated it since.
cfontes 14y ago
things are starting to run again... My site is back up :D SLOW but up.
jeremi23 14y ago
Now even the Google AppEngine status page is down.
aidos 14y ago
Dropbox [0] is showing a 500 for me to. I've very confused as to what has just happened to the internet... [0] https://www.dropbox.com/ https://www.dropbox.com/
singingwolfboy 14y ago
www.howfuckedisappengine.com
fidotron 14y ago
It's time we remembered the whole strength of the internet was that it was distributed and we avoided introducing single points of failure. We have ended up using vast amounts of infrastructure for no reason other than developer convenience (often with respect to security), when having local direct connections is often more suitable than shooting everything into the cloud.
- lurker14 14y ago
  Which is better? Having a day of downtime each year, or not launching at all?
  - peterwwillis 14y ago
    Which is better? Using a fallacious comparison to suggest cloud computing is the only viable option, or comparing the pros and cons of different computing models to choose the best one for you?
    - davidkatz 14y ago
      While the argument was perhaps coming on a bit too strong, it's hard to deny the ease of deployment on cloud services. It's probably a safe bet to say that for most early stage startups the cloud is a good move.
      - peterwwillis 14y ago
        I'm at a loss in these discussions. I don't understand this developer-point-of-view. Can you specifically give me examples of why using a cloud provider is better for a startup than, for example, using a couple desktops in your garage? You can't say it's because of backups because the cloud doesn't provide a backup (unless you purchase an extra data backup solution with your cloud provider?). And correct me if i'm wrong, but you still have to set up your development environment on your local computer to write the code, install libraries to test with, etc. What exactly are the steps involved in "deploying" that you couldn't do on your laptop, or a VPS?
harshadura 14y ago
i think now its back again live. atleast for me.
sebbi 14y ago
Seems to be online again ...
deleted 14y ago
[deleted]
Yoms 14y ago
Latest update: "At approximately 7:30am Pacific time this morning, Google began experiencing slow performance and dropped connections from one of the components of App Engine. The symptoms that service users would experience include slow response and an inability to connect to services. We currently show that a majority of App Engine users and services are affected. Google engineering teams are investigating a number of options for restoring service as quickly as possible, and we will provide another update as information changes, or within 60 minutes." https://groups.google.com/forum/?fromgroups=#!topic/google-appengine-downtime-notify/SMd2pDJsCPo https://groups.google.com/forum/?fromgroups=#!topic/google-a...
albumedia 14y ago
GAE has been very good since I started using it. They entire internet seems to be slow...even HN
peterwwillis 14y ago
Here's a zen koan for you: If every website on the internet is hosted in the cloud, and the cloud goes down, is there an internet?
vanwilder77 14y ago
The Google app engine page down as well!
cilvento 14y ago
At about 7:30am US/Pacific time this morning, Google began experiencing slow performance and dropped connections from one of the components of App Engine. Many App Engine applications are experiencing slow responses and an inability to connect to services. We currently show that a majority of App Engine users and services are affected. We are actively working on restoring service as quickly as possible. We are posting regular updates to our downtime-notify list here: https://groups.google.com/forum/?fromgroups=#!topic/google-appengine-downtime-notify/SMd2pDJsCPo https://groups.google.com/forum/?fromgroups=#!topic/google-a... Thanks, Christina, Google App Engine Product Manager
brutuscat 14y ago
Status reports from the mailing list https://groups.google.com/d/topic/google-appengine-downtime-notify/SMd2pDJsCPo/discussion https://groups.google.com/d/topic/google-appengine-downtime-...
daave 14y ago
And they've sent the all-clear: At this point, we have stabilized service to App Engine applications. App Engine is now successfully serving at our normal daily traffic level, and we are closely monitoring the situation and working to prevent recurrence of this incident. This morning around 7:30AM US/Pacific time, a large percentage of App Engine’s load balancing infrastructure began failing. As the system recovered, individual jobs became overloaded with backed-up traffic, resulting in cascading failures. Affected applications experienced increased latencies and error rates. Once we confirmed this cycle, we temporarily shut down all traffic and then slowly ramped it back up to avoid overloading the load balancing infrastructure as it recovered. This restored normal serving behavior for all applications. We’ll be posting a more detailed analysis of this incident once we have fully investigated and analyzed the root cause. Regards, Christina Ilvento on behalf of the Google App Engine Team https://groups.google.com/forum/#!topic/google-appengine-downtime-notify/SMd2pDJsCPo/discussion https://groups.google.com/forum/#!topic/google-appengine-dow...
deleted 14y ago
[deleted]

152 points by soofaloofa 14y ago

I'm really happy I don't host in the cloud. How quickly are the cost savings of cloud computing obliterated by PR, customer service, and system administration time when an outage like this occurs?

tomgallard 14y ago
Surely hosting yourself exposes you to just as much, if not more risk? Problem in the datacentre where you're co-lo'd, or one of your servers blows up? I think people not trusting the cloud is similar to how people feel safer driving their cars then taking a plane. The stats say the plane's safer, but people prefer being in control. People like the idea of being in control of their servers, even if that means there's hundreds of extra things that can go wrong compared to a cloud provider. We also get a lot more publicity when a cloud provider has an outage as LOTS of sites go down at once. Hardly anyone notices when service X who self-host go down for a few hours...
- Alan01252 14y ago
  I agree with you. From my past experience any data center is subject to risks. I've witnessed: Power failures. Cross site links being cut due to engineering works. Over heating due to air conditioning failures. Flooding And I've experienced all the above from a very large, very well known, very expensive data center company based in London.
  - acdha 14y ago
    This is true of every data center I've worked with. Also network providers: everyone has downtime and sometimes you learn the hard way that despite being written into your contract someone took the cheap way and ran your “redundant” fiber through the same conduit which a backhoe just tore up.
- peterwwillis 14y ago
  The "safety" of the cloud is about two things: 1. trusting your service provider, and 2. redundancy. You have to trust your cloud provider. They control everything you do. If their security isn't bulletproof, you're screwed. If their SAN's firmware isn't upgraded properly to deal with some performance issue, you're screwed. If their developers fuck up the API and you can't modify your instances, you're screwed. You have to put complete faith in a secret infrastructure run for hundreds of thousands of clients so there's no customer relationship to speak of. That's just the "trust" issue. Then there's the issue of actual redundancy. It's completely possible to have a network-wide outage for a cloud provider. There will be no redundancy, because their entire system is built to be in unison; one change affects everything. Running it yourself means you know how secure it is, how robust the procedures are, and you can build in real redundancy and real disaster recovery. Do people build themselves bulletproof services like this? Usually not. But if you cared to, you could.
- debacle 14y ago
  We're coloed across three datacenters spanning the US (one might be in TO I think) and if a datacenter were to go down, we have a hot backup that's no more than 12 hours stale. The only real manual maintenance that we've got is a rolling reimaging of servers based on whatever's in version control, which usually takes a few hours twice a year, but we'd probably do that if we were in the cloud anyway. When you can script away 90% of your system administration tasks, hosting in the cloud doesn't really make a ton of sense.
  - tomgallard 14y ago
    But a DNS based failover is still going to take an hour or so to propagate right (given that a lot of browsers/proxies/DNS servers don't respect TTL very well at all)? And then you end up with a system with stale data, and the mess of trying to reconcile it when your other system comes back up. I'd take an hour long Appengine outage once a year over that anytime!
    - peterwwillis 14y ago
      Your name server or stub resolver is what respects DNS TTL, not your browser or proxy. Everyone - including people hosting on AWS - needs to be able to fail over DNS, if the AWS IP you're using is in a zone that just went down, for example. Any time you have an outage you need to contact your service provider to get an estimate of downtime. If they can't give you one, assume it'll take forever and cut the DNS over. The worst case is some of your users will start to come back online slowly. If you don't cut over, the worst case is all your users are down until whenever the service provider fixes it, and you get to tell your users "we're waiting for someone else to deal with it", which won't make them very happy. 12 hour stale data sounds kind of long to me. 4 hours sounds more reasonable.
      - codeka 14y ago
        I've seen plenty of crappy ISP DNS servers ignore TTL values and cache DNS entries for many hours longer than they're supposed to. Unfortunately, it's all too common.
- modoc 14y ago
  It really depends on how many servers and how good your sys admins are. If you have 1 server in a cheap colo, then yes, the cloud is probably better. If you have a GOOD hosting provider, and build out a redundant at every tier cluster, you can easily beat the major cloud providers' uptime. We run about 11 client clusters across ~250 servers across 3 data centers in the US and Europe. Each of our client's uptime is very very close to 100%, and we've NEVER lost everything, even for 1 second.
hartleybrody 14y ago
I don't think you move to the cloud for cost savings. More like time and headache savings.
bobmyers 14y ago
Service outages happen to everyone. Thats why everyone advertises 99.99% uptime four nines or five nines. No one can achieve 100%. The good thing about the cloud is you can say, this one isn't my fault.
untog 14y ago
I think there is an interesting debate to be had here. When your site goes down at the same time as Tumblr/Reddit/one of the big guys, the damage might not actually be that high. In the minds of many, "the internet" is probably quite broken.
- debacle 14y ago
  Depends on the site. Most of the traffic we cater to is not referrer driven.
bsaul 14y ago
It's funny, i've got the exact opposite reasoning : it's those moments where i can really appreciate the fact that i'm using the cloud : 1/ I don't have to spend the night debugging or replacing broken hardware 2/ It doesn't cost me any time, any additional resource, any support upgrade, any hardware. 3/ No one can blame me or anybody in my team for the fact that it's not working. I don't feel like i'm lacking control, i feel like somebody else is taking care of that really annoying shit that happens all the time, no matter how well you design your system.
- lurker14 14y ago
  You customers will not be mollified if you tell them that the service you sold is down due to a subcontractor failure.
- modoc 14y ago
  If you have paying clients, they WILL blame you and your team for the fact it's not working. They don't care who/what the underlying infrastructure is. Also a good hosting company will handle identifying/fixing/replacing bad hardware for you.
  - calinet6 14y ago
    Not to mention sufficient redundancy will ensure that you never see many effects from those hardware failures/power outages/floods/fires/anything else with any reasonable probability. "The cloud is great because I can blame someone else" is obviously a tenuous argument.
ceejayoz 14y ago
My colocation provider (Frontier, telecom in 27 states) went down for an hour and a half last night. It's hardly unique to the cloud.

bsaul 14y ago

I would love it so much to see people at google showing all the internal tools they're using to detect and solve this kind of issues. I can only imagine a war room with screens all over the place showing gigantic amount of red flashing lines :) Hope it doesn't last for long though, i was just praising what a good choice app engine has been so far 10 minutes ago...

notreadbyhumans 14y ago

It's a bit nuts that they're hosting the status pages on the same infrastructure.

harshadura 14y ago
jz only 1-2 hr down-age ;)

chrisfarms 14y ago

> "App Engine is currently experiencing serving issues. The team is actively working on restoring the service to full strength. Please follow this thread for updates." -- Max Ross (Google) maxr@google.com via googlegroups.com https://groups.google.com/forum/?fromgroups=#!topic/google-appengine-downtime-notify/SMd2pDJsCPo https://groups.google.com/forum/?fromgroups=#!topic/google-a...

davis_m 14y ago

I think this is larger than just GAE. http://internettrafficreport.com/namerica.htm http://internettrafficreport.com/namerica.htm It seems like large portions of the internet are down.

untog 14y ago
Yes, Tumblr is experiencing difficulties and AFAIK they don't run anything on App Engine.
sparkinson 14y ago
Now those are some interesting graphs.
FiloSottile 14y ago
The internet is burning :O Well, seriously what could be a root cause that affects so many nodes?
bsaul 14y ago
only 51% of north american internet is working ?? Am i reading this correctly ?
- seldo 14y ago
  As Ewantoo mentioned, ITR is pretty unreliable. We have found it works as a relative measure -- when their packet loss rises, we do indeed see consumer traffic fall, though usually not by the amount their graphs would indicate.
- EwanToo 14y ago
  You're reading it correctly, it's just telling you the wrong thing. 51% of a very small self-selecting set of machines are down, mostly small businesses or home users which have obviously been shutdown or renamed. If internettrafficreport automatically removed all devices which returned no results for 7 days, they'd be a slightly more useful resource.
cfontes 14y ago

tomnewton 14y ago

My Google contact said that 'SRE are all over it. Hope to have more details soon.' but that was about 30 minutes ago. Does tumblr.com use app engine? They're down...

tszming 14y ago
@tumblr (https://twitter.com/tumblr/status/261840787350896640 https://twitter.com/tumblr/status/261840787350896640) Tumblr is experiencing network problems following an issue with one of our uplink providers. We will return to full service shortly.

cyberpanther 14y ago

I'm seeing a bunch of Google properties also. Maybe they are running on app engine? Like https://developers.google.com/ https://developers.google.com/

libria 14y ago
Also, http://dartlang.org http://dartlang.org http://golang.org http://golang.org, http://code.google.com/codejam/ http://code.google.com/codejam/ http://www.chromeexperiments.com/ http://www.chromeexperiments.com/ Gotta give 'em props for dogfooding.
- moepstar 14y ago
  Props, really? That's what i _expect_ them to do - otherwise i can't see anyone trusting/using them if even they themselves avoid their own product(s)..
Sami_Lehtinen 14y ago
http://sites.google.com/ http://sites.google.com/ was down too. I got monitoring report, but I first assumed it's false. Because false down reports are much more common than Google services being actually down.

aidos 14y ago

I'm surprised that their status system seems to run off the main infrastructure too. Everyone else gets slated for making that mistake.

foolery 14y ago

Yup, all of the HRD apps are down. But the M/S apps are working.

jis 14y ago
Well, my one M/S app is also down.
- aderaynal 14y ago
  I have 2 M/S apps and 3 HRD. They are all down :(
  - cfontes 14y ago
    Mine is also down... :( any one got any position about it ?
- vitorarins 14y ago
  my one M/S app is up and running.
  - sebbi 14y ago
    same here ... M/S up and running ... all recently migrated HRD apps down :/

davidjgraph 14y ago

Before the doom and gloomers come out, this is the first time since leaving beta I can remember it happening. We left AWS about 18 months ago after one of the outages and switched to GAE. I've counted 3-4 big downtimes for AWS compared to this one on GAE. That's still a good decision (for now)....

lyddonb 14y ago
Yep. We've never seen anything like this before.
namank 14y ago
Same. Minor updates and issues pop up with GAE but I've never experienced an outage such as this in my two years of use. Wonder what's happening.
tedroden 14y ago
It's one of the first times the whole service has been down, but parts of the service go down at least once a week. memcache, task queues are "elevated" with regularity and urlfetch is frequently down totally. ("elevated" generally means unusable). Of course master/slave even has scheduled downtime.
mikegioia 14y ago
I have yet to experience downtime with RackspaceCloud and I've been using them for like 3 years.
- jmsduran 14y ago
  Me too.
- dchest 14y ago
  I had network issues with my two virtual machines on RackspaceCloud about a year or so ago (something was wrong with their routing). They were resolved quickly, though.
acdha 14y ago

libria 14y ago

Hm, bad week for the Cloud. Can't even get to the status page; hopefully it's not hosted on App Engine. So going forward, what's the best way to protect against cloud downtime? Have a hot/standby failover with a different provider? Prepare customers' expectations for the possibility of server outages? Do a ton of research, pay $$$ for lots of nines uptime, and lambast the host when they don't deliver?

bad_user 14y ago
Downtimes happen regardless, unless you have a lot of money and talent to spend on your own infrastructure and even then it's hard to beat cloud providers like Amazon, or Google, who have more resources and knowledge than you do. The greatest thing about cloud-hosting is that you can just sit by and let them fix it. It usually takes about half an hour, or a couple of hours if the outage is severe, but usually less than the time it takes for an update of DNS records (unless you've got some proxy in front of your IPs, which would be another point of failure). And then, even with these severe outages, the overall monthly uptime is still better than %99.9 and it's really hard to beat that, so just relax and let them fix it.
davidjgraph 14y ago
I'd say the answer depends on how fast GAE recovers. If you're building redundancy over multiple clouds, if there's a lot of data: 1) It's very complex and expensive 2) You're looking at DNS to hot failover, in most cases. If GAE can recover in less than 30 minutes and sticks to, say, one outage a year, you just can't justify the kind of cost you're looking at with 2 (seriously, it's a lot of cash).
josh2600 14y ago
Build redundancy into your software to deal with single provider failure.
- Achshar 14y ago
  That is not always a feasible option, specially for young projects with limited capital.

hugofierro 14y ago

I hope it's not due to DiRT Exercises (SRE Disaster Recovery Test). Looking forward to reading the post-mortem report!

ams6110 14y ago

passpack.com seems to be affected.

jsdalton 14y ago

It's really quite remarkable (to be honest, inexcusable is probably a better word) that their status page is failing as well. My expectations for a company with Google's resources and infrastructure are a lot higher than that. Nothing on their Twitter account either: https://twitter.com/app_engine https://twitter.com/app_engine A poor handling of a systems failure in my opinion.

rdwallis 14y ago
If you subscribe to: https://groups.google.com/forum/?hl=en&fromgroups#!forum/google-appengine-downtime-notify https://groups.google.com/forum/?hl=en&fromgroups#!forum... They'll email you when issues occur and info becomes available. It took about 30 minutes after the crash for me to receive an email which seems very reasonable.
- IheartApplesDix 14y ago
  If you find that reasonable, I have a slice in Brooklyn to sell you.

deleted 14y ago

[deleted]

kjhughes 14y ago

What's the earliest sign of trouble you've had? Pingdom reports my GAE-hosted site has been down since 2012-10-26 10:37:38 EST, a bit over an hour now. UPDATE: My site is back. Delayed report from Pingdom says site came back online after 50 minutes. Performance is sketchy still. We're probably not in the clear yet. At least we can now get to the status dash: http://code.google.com/status/appengine http://code.google.com/status/appengine

warrentr 14y ago
I think the first tweet i saw was 7:35 PST

deleted 14y ago

[deleted]

peterwwillis 14y ago
I can not bear with a wall of text about a note-taking app. Could you pastebin that?
whalesalad 14y ago
Whoa did not expect to receive such negative feedback on this post. Was really looking for feedback from the community on a replacement. I'll post this elsewhere I guess.
- tomgallard 14y ago
  I think the best place would be in an 'Ask HN' thread. I think the community felt your post was off topic- and that's why it was downvoted

abhijitr 14y ago

Meanwhile... Gmail etc are working quite fine. So the claim that if you build on GAE you "take advantage of the same infrastructure used for Google services!!" starts to ring a bit hollow.

rytis 14y ago
I wonder if anyone ever believed this claim to be true...
- wildmXranat 14y ago
  Or if the definition of the word 'same' is somewhat fluid enough to get away with it
hugoc 14y ago
Well http://developer.android.com/ http://developer.android.com/ is down, so google does use GAE for some of its properties?
adolph 14y ago
Maybe they mean == instead of ===.
Ironlink 14y ago
Or perhaps they are older than the affected infrastructure and just haven't been rewritten. Or perhaps they happen to not use the affected resource. In fact, affected services include developers.google.com and code.google.com, and to a lesser extent youtube.com.
Tobu 14y ago
I know the claim of running Amazon.com on AWS was initially bogus, but did anyone notice trouble on that site with the recent AWS outages? It would be instructive to know how much they have integrated it since.

cfontes 14y ago

things are starting to run again... My site is back up :D SLOW but up.

jeremi23 14y ago

Now even the Google AppEngine status page is down.

aidos 14y ago

Dropbox [0] is showing a 500 for me to. I've very confused as to what has just happened to the internet... [0] https://www.dropbox.com/ https://www.dropbox.com/

singingwolfboy 14y ago

www.howfuckedisappengine.com

fidotron 14y ago

It's time we remembered the whole strength of the internet was that it was distributed and we avoided introducing single points of failure. We have ended up using vast amounts of infrastructure for no reason other than developer convenience (often with respect to security), when having local direct connections is often more suitable than shooting everything into the cloud.

lurker14 14y ago
Which is better? Having a day of downtime each year, or not launching at all?
- peterwwillis 14y ago
  Which is better? Using a fallacious comparison to suggest cloud computing is the only viable option, or comparing the pros and cons of different computing models to choose the best one for you?
  - davidkatz 14y ago
    While the argument was perhaps coming on a bit too strong, it's hard to deny the ease of deployment on cloud services. It's probably a safe bet to say that for most early stage startups the cloud is a good move.
    - peterwwillis 14y ago
      I'm at a loss in these discussions. I don't understand this developer-point-of-view. Can you specifically give me examples of why using a cloud provider is better for a startup than, for example, using a couple desktops in your garage? You can't say it's because of backups because the cloud doesn't provide a backup (unless you purchase an extra data backup solution with your cloud provider?). And correct me if i'm wrong, but you still have to set up your development environment on your local computer to write the code, install libraries to test with, etc. What exactly are the steps involved in "deploying" that you couldn't do on your laptop, or a VPS?

harshadura 14y ago

i think now its back again live. atleast for me.

sebbi 14y ago

Seems to be online again ...

deleted 14y ago

[deleted]

Yoms 14y ago

Latest update: "At approximately 7:30am Pacific time this morning, Google began experiencing slow performance and dropped connections from one of the components of App Engine. The symptoms that service users would experience include slow response and an inability to connect to services. We currently show that a majority of App Engine users and services are affected. Google engineering teams are investigating a number of options for restoring service as quickly as possible, and we will provide another update as information changes, or within 60 minutes." https://groups.google.com/forum/?fromgroups=#!topic/google-appengine-downtime-notify/SMd2pDJsCPo https://groups.google.com/forum/?fromgroups=#!topic/google-a...

albumedia 14y ago

GAE has been very good since I started using it. They entire internet seems to be slow...even HN

peterwwillis 14y ago

Here's a zen koan for you: If every website on the internet is hosted in the cloud, and the cloud goes down, is there an internet?

vanwilder77 14y ago

The Google app engine page down as well!

cilvento 14y ago

At about 7:30am US/Pacific time this morning, Google began experiencing slow performance and dropped connections from one of the components of App Engine. Many App Engine applications are experiencing slow responses and an inability to connect to services. We currently show that a majority of App Engine users and services are affected. We are actively working on restoring service as quickly as possible. We are posting regular updates to our downtime-notify list here: https://groups.google.com/forum/?fromgroups=#!topic/google-appengine-downtime-notify/SMd2pDJsCPo https://groups.google.com/forum/?fromgroups=#!topic/google-a... Thanks, Christina, Google App Engine Product Manager

brutuscat 14y ago

Status reports from the mailing list https://groups.google.com/d/topic/google-appengine-downtime-notify/SMd2pDJsCPo/discussion https://groups.google.com/d/topic/google-appengine-downtime-...

daave 14y ago

And they've sent the all-clear: At this point, we have stabilized service to App Engine applications. App Engine is now successfully serving at our normal daily traffic level, and we are closely monitoring the situation and working to prevent recurrence of this incident. This morning around 7:30AM US/Pacific time, a large percentage of App Engine’s load balancing infrastructure began failing. As the system recovered, individual jobs became overloaded with backed-up traffic, resulting in cascading failures. Affected applications experienced increased latencies and error rates. Once we confirmed this cycle, we temporarily shut down all traffic and then slowly ramped it back up to avoid overloading the load balancing infrastructure as it recovered. This restored normal serving behavior for all applications. We’ll be posting a more detailed analysis of this incident once we have fully investigated and analyzed the root cause. Regards, Christina Ilvento on behalf of the Google App Engine Team https://groups.google.com/forum/#!topic/google-appengine-downtime-notify/SMd2pDJsCPo/discussion https://groups.google.com/forum/#!topic/google-appengine-dow...

deleted 14y ago

[deleted]