12 ms·
GCE down in all regions
- deleted 11y ago[deleted]
- avolcano 11y agoSpotify is down due to this, which is, uh, pretty hilarious https://news.spotify.com/us/2016/02/23/announcing-spotify-infrastructures-googley-future/ https://news.spotify.com/us/2016/02/23/announcing-spotify-in...
- outside1234 11y agoI bet they and Google will have a positive post written up about it tomorrow though. :)
- draw_down 11y agoI remember people speculating that Spotify most likely received a big discount for doing this. Guess you get what you pay for ;)
- trhway 11y agoGoogle provides 99.95% - https://cloud.google.com/appengine/sla https://cloud.google.com/appengine/sla That is like 17 of such 15 minutes breaks per year, i.e. an allowance for one small (or a large fixed quickly) screwdup/month :)
- ec109685 11y agoThis allows unlimited small outages: "Downtime Period" means, for an Application, a period of five consecutive minutes of Downtime. Intermittent Downtime for a period of less than five minutes will not be counted towards any Downtime Periods.
- jpatokal 11y ago99.95% is correct, but the Compute Engine SLA is actually here: https://cloud.google.com/compute/sla https://cloud.google.com/compute/sla Today's incident did not impact App Engine at all. (Disclaimer: I work in Google Cloud Support.)
- r0fls 11y agoIt seems to be working fine, I just tested.
- BinaryIdiot 11y agoOutage lasted about 16 minutes
- Mafana0 11y agoI see your point. Based on @SpotifyStatus [0], it wasn't uncommon for Spotify to have service disruptions before they did the move though. [0]: https://twitter.com/SpotifyStatus https://twitter.com/SpotifyStatus
- eva1984 11y agoTo be fair, AWS has some significant down time in the second half of last year.
- cavisne 11y agoNot across multiple regions though. It's not trivial to make an application cross region but at least there is a way to engineer around an outage, unlike this outage
- jasonjei 11y agoWas Google services (search, email, drives, apps) impacted at all?
- asadlionpk 11y agoNo they don't use gcloud to host their own apps. Yes that's ridiculous.
- morgante 11y agoWhich is exactly why I won't use GCE. If Google isn't confident enough to use it for themselves, neither will I. The fact that Amazon dogfoods AWS is a major advantage for them.
- brown9-2 11y agoEven if Amazon.com uses AWS (the extent of which seems it may be a mixture of marketing hype and urban legend), there are many ways AWS could fail that affects customers but leaves Amazon.com unaffected.
- tempestn 11y agoGoogle Custom Search was down for a similar time period. (Outage lasted longer than GCE, but seems likely related.)
- kahwooi 11y agoIt would be hilarious if they use to host their services on AWS or Azure.
- chris_wot 11y agoAnd they won't be down again in a long, long time.
- ryankupyn 11y agoIf Spotify wanted to be really sneaky, some amount of downtime might be good for them financially. The bulk of their revenue comes from customers who subscribe on a per-month basis, while they pay out royalties on a per-song-played basis. This outage is reducing the amount they have to pay, and if the outage-elasticity-of-demand is low enough they could (hypothetically) come out ahead!
- flashman 11y ago> while they pay out royalties on a per-song-played basis I believe this is inaccurate. They pay out royalties on a share-of-all-plays basis, don't they?[0] So an outage wouldn't reduce the payout amount, it would just slightly alter the balance of payments for individual rightsholders. [0]http://www.spotifyartists.com/spotify-explained/#royalties-in-detail http://www.spotifyartists.com/spotify-explained/#royalties-i...: "That 70% is split amongst the rights holders in accordance with the popularity of their music on the service. The label or publisher then divides these royalties and accounts to each artist depending on their individual deals... Spotify does not calculate royalties based upon a fixed “per play” rate."
- itsnotvalid 11y agoRather, as they don't have SLA to end-users, they get credits from Google, so they still earn money.
- kahwooi 11y agoBut the reputation is damage.
- trevor-e 11y agoI was getting really frustrated at the gym when the Spotify app wouldn't work. Didn't expect to find the answer here.
- sundvor 11y agoHappy to say I didn't notice; I am using a 517 song offlined playlist ("EVE Online" by Michael Andrew) for my programming work.
- bonestamp2 11y agoThat sounds awesome, care to share?
- sundvor 11y agoSorry late - link is https://open.spotify.com/user/1231239981/playlist/3ka1SYnv2bm3HgMi5x248d https://open.spotify.com/user/1231239981/playlist/3ka1SYnv2b... . Hope you see this post and enjoy it. I see from the down votes that my reply must have been seen as kind of off topic to the GCE issue, however since Spotify came up as a "victim", I did feel it prudent to mention that Spotify Premium has offline playlists to allow users to weather network issues of any kind. Also for me personally, big playlists of quality music like this one is fantastic for my work.
- NetStrikeForce 11y agoI think it's a great idea to diversify not just regions, but providers. The future is for per-application virtual networks that are agnostic to the underlying hosting provider. These networks work as an overlay, which means that your applications can be moved through providers without changing their architecture at all. You could even shut it down in provider A and start it in provider B without any changes at all. At Wormhole[1] we have identified this problem and solved it. [1]: https://wormhole.network https://wormhole.network
- mrdrozdov 11y agoI do not envy the current on-call rotation.
- spyspy 11y agoI don't envy anyone with an on-call job.
- jethro_tell 11y agoWhat do you recommend? I figure that if you're working on something without oncall no one probably cares about it any way. I prefer to have a good rotation than no rotation.
- kps 11y agoThere are still a few things in the world that aren't internet services.
- distrill 11y agoOnly a few though.
- simula67 11y agoGames, mobile apps, desktop apps like Photoshop, Office, Intellij etc and some shrink wrapped server side apps. But you are right, some of these products are starting to have an online component as well.
- lucian1900 11y agoInterestingly, nowadays all but the last one tend to require a service to be available.
- deleted 11y ago[deleted]
- silverlight 11y agoThat was nuts. Interested to read the post-mortem on this one. Our site went down as well. What could cause a sudden all-region meltdown like that? Aren't regions supposed to be more isolated to prevent this type of thing? Seems to have only been down for about 10 minutes, so I'm thinking some sort of mis-configuration that got deployed everywhere...they were working to fix a VPN issue in a specific region right before it went down...
- taf2 11y agoUnless a bug was in all regions it's why it's good to consider multiple cloud services for failover
- tracker1 11y agoAnd where do you put your system that directs traffic to one cloud and/or another... and what happens when that goes down?
- jon-wood 11y agoYou put it in all your clouds, with low TTL DNS entries pointing at all those instances (or the closest one geographically maybe). Then if you're really paranoid you use redundant DNS providers as well.
- tracker1 11y agoThe point was that adding another point for potential failure still won't reduce the chance of failure... it's just something else that can and will break. In any case, failures happen, and most systems are better off being as simple as possible and accepting the unforeseen failures than trying to add complexity to overcome them.
- packetslave 11y agoAnd then you discover that there are a LOT of craptastic DNS resolvers, middle boxes, AND ISP DNS servers out there that happily ignore or rewrite TTLs. With a high-volume web service, you can have a 1 minute TTL, change your A records, and still see a lovely long tail of traffic hitting the old IP for HOURS.
- jpatokal 11y agoBack up as of 19:27 US/Pacific: https://status.cloud.google.com/incident/compute/16007 https://status.cloud.google.com/incident/compute/16007
- ikeboy 11y agoGreat timing for that new book :)
- deleted 11y ago[deleted]
- awinter-py 11y agofive nines = 45?
- fixermark 11y agoNo; 999.99% uptime.
- awinter-py 11y ago1000% = 10 days per year
- alecbaldwinlol 11y agoGuys, I've got it- instead of locking ourselves in with one vendor's platform and being subject to their mismanagement and arbitrary rules, why don't we buy our own hardware and do it ourselves? We can free up our OpEx budget too! My sales rep sent me a TCO that shows it is way cheaper to run a data center than to pay a cloud subscription! I'm calling the CFO!
- paulsutter 11y agoAsk HN: Is anyone using different cloud providers for failover and what's your DNS configuration? Do any cloud providers allow announcing routes for anycast DNS?
- matt_wulfeck 11y agoIt's very uncommon. In my experience the database becomes the issue.
- tracker1 11y agoIf you're using something like Cassandra (C*), it's pretty easy to have replicated data to multiple zones in multiple clouds. RethinkDB has a similar replication system... there are many others as well. Not all data models fit into a Non-SQL database though, and may work better in a more relational.. caching and read-only partially up, can be another approach. Designing your data around this may be impractical for some applications, and on the small scale, likely more costly than necessary. Most systems can tolerate 15-30 minutes of downtime every couple years because of an upstream provider.
- anthony_franco 11y agoYes, I'm using two different providers and sync them up using master-master replication. For DNS I use DNSMadeEasy's DNS Failover feature that automatically fails over to a different IP address when it's unable to ping the server.
- retrogradeorbit 11y agoI, too, would be interested in info on any cloud providers that support anycast.
- Artemis2 11y agoI'm no networking expert but packet.net has a page on this: https://www.packet.net/bare-metal/network/anycast/ https://www.packet.net/bare-metal/network/anycast/
- qaq 11y agoI know this will get downvoted but clouds suck and this is just one more manifestation of why they suck. Unless you have very spiky workload save yourself long term pain and don't go this route (applies if your monthly AWS/GCE/Azure bill is over few K)
- deleted 11y ago[deleted]
- virmundi 11y agoI'm not seeing why not. Your data center could go down for a myriad of reasons (ISP goes down, HDs, tripping on power cable, etc). If that happens you're pretty much screwed. You could compensate by having multiple data centers with different infrastructure providers. If you do, you're probably spending more than the few K you referenced in your post. Yes, it's bad that apparently all of the regions failed. Google will hear about it. People will get in trouble. But a screw up at this level is rare. If you use cloud, or even a VPS provider like Linode, you get auto-fail over and someone that is contractually obligated to deal with failures.
- qaq 11y agoYou are paying penalty in complexity, latency and poor tenant isolation when running on "cloud infrastructure" and when things blow up you have no recourse.
- Aoreias 11y agoDo you have any examples of poor tenant isolation in AWS, GCE, or Azure? Cloud complexity is also lower because you don't have to worry about power, cooling, upstream connectivity, capacity budgeting, etc. If 99.9-99.95% availability is fine for your application then you probably don't have to worry about your provider either.
- qaq 11y agoAWS Netflix consumes enough resources that if they spike 40-50% everyone is screwed. The software required to run the cloud like AWS is orders of magnitude more complex then what avg project would need and results in major screwups. Both major AWS outages were due control plane issues second case was result of massive Netflix migration that triggered throttles for everyone in affected AZs. The throttles in the first place were put in due to the major outage that lasted for many hours.
- gtaylor 11y agoI've been overall very impressed with the direction of Google Cloud over the last year. I feel like their container strategy is much better than Amazon's ECS in that the core is built on open source technology. This can wipe away a lot of goodwill, though. A worldwide outage is catastrophic and embarrassing. AWS has had some pretty spectacular failures in us-east (which has a huge chunk of the web running within it), but I'm not sure that I can recall a global outage. To my understanding, these systems are built specifically not to let failures spill over to other regions. Ah well. Godspeed to anyone affected by this, including the SREs over at Google!
- sandGorgon 11y agoCan you talk about this? I have been spectacularly unsuccessful at using ECS (and currently run my VMs on a vanilla Debian ec2 instance)
- jordanthoms 11y agoSwitching from ECS to GKE (Google Container Engine) currently. Both seem overcomplex for the simpler cases of deploying apps (and provide a lot of flexibility in return), but I have found the performance of GKE (e.g. time for configuration changes to be applied, new containers booted, etc) to be vastly superior. The networking is also much better, GKE has overlay networking so your containers can talk to each other and the outside world pretty smoothly. GKE has good commandline tools but the web interface is even more limited than ECS's is - I assume at some point they'll integrate the Kubernetes webui into the GCP console. GKE is still pretty immature though, more so than I realized when I started working with it. The deployments API (which is a huge improvement) has only just landed, and the integration with load balancing and SSL etc is still very green. ECS is also pretty immature though.
- huslage 11y agoI'm admittedly biased, but have you checked out Docker Cloud? http://cloud.docker.com http://cloud.docker.com
- HorizonXP 11y agoSo in all seriousness, how do folks deal with this? In this case, it ended up being a multi-region failure, so your only real solution is to spread it across providers, not just regions. But I imagine it's a similar issue to scaling across regions, even within a provider. We can spin up machines in each region to provide fault tolerance, but we're at the mercy of our Postgres database. What do others do?
- _delirium 11y agoDepends entirely on your business, but what I do is just tolerate the occasional 15-minute outage. There's increasing cost to getting more 9's on your uptime, and for me, engineering a system that has better uptime than Google Cloud does, by doing multi-cloud failover, is way out of favorable cost/benefit territory.
- kbenson 11y agoThat's the only sane thing to do. It is impossible to ensure 100% uptime, and it gets increasingly harder to approach that as you put more separate entities between yourself and the client. The thing is, you'll be blamed for problems that aren't in your control and aren't really related to your service, but to the customer. For example, local connectivity, phone data service, misbehaving DNS resolvers, packet mangling routers, mis-configured networks, mis-advertised network routes, etc. Every single one of those examples can happen before the customer traffic even gets to the internet, much less where you have your servers housed. All you can do is accept that there will be problems that are attributed to your service, rightly so or not, and work to mitigate and reduce the possibility the problems you can, and learn it's not the end of the world.
- pjlegato 11y agoMost people just deal with it and accept that their site will go down for 20 minutes every 3-4 years or so, even when hosting on a major cloud, because: 1) the cost of mitigating that risk is much higher than the cost of just eating the outage, and 2) their high traffic production site is routinely down for that long anyway, for unrelated reasons. If you really, really can't bear the business costs of an entire provider ever going down, even that rarely (e.g. you're doing life support, military systems, big finance), then you just pay a lot of money to rework your entire system into a fully redundant infrastructure that runs on multiple providers simultaneously. There really aren't any other options besides these two.
- Scarbutt 11y agoSo what Google services went down with this? looks like they are not eating their own dog food?
- trhway 11y agomay be the things like gmail have multicloud failover, say to AWS?
- chipperyman573 11y agoGoogle is self-hosted, but they might not use the same hardware GCE uses.
- ocdtrekkie 11y agoA lot of Google's own services predate these public offerings. So my guess is even if they're on the same or similar technologies, they may be separate systems.
- firloop 11y agoHas Amazon.com ever been affected by a AWS outage? edit: at least it's impacted some of their ancillary services before http://www.theregister.co.uk/2015/09/20/aws_database_outage/ http://www.theregister.co.uk/2015/09/20/aws_database_outage/
- discodave 11y agoAmazon.com depends on AWS but not all the services and not necessarily in the critical path of serving up pages on the main website. E.g. All the servers run on EC2 but they don't really use ELB or EBS. S3 is used heavily throughout the company but some of the newer services not so much.
- chris_wot 11y agoHmmm... would this have affected Netflix?
- rajathagasthya 11y agoNetflix runs on AWS.
- TheDong 11y agoIt's fairly well known that netflix runs primarily on AWS. So probably no.
- tszming 11y ago(rant) Yes, traffic to Netflix increased significantly as Spotify was down.
- cellularmitosis 11y agoTime Warner Cable's DNS server went down at roughly the same time (in Austin). I'm hoping that's just a coincidence.
- Artemis2 11y agoSeems kinda misleading from Google to claim repeatedly that they are hosted just on the same infrastructure of GCP, and not go down with it. EDIT: Switched from "dishonest" to "misleading"; while it's abundantly clear that Google doesn't run on GCP, GCP feels like a second-class citizen to Google because you just cannot get Google uptime with it.
- dsymonds 11y agoGoogle and GCP run on the same infrastructure, but this was a GCP problem, not a problem with that common infrastructure.
- remosi 11y ago(I'm a Google SRE, I'm on the team that dealt with this outage) This did impact common infrastructure. Some (non-cloud) Google services were impacted. We've spent years working on making sure gigantic outages are not externally visible for our services, but if you looked very closely at latency to some services you might have been able to see a spike during this outage. My colleagues managed to resolve this before it stressed the non-cloud Google services to the point that the outage was "revealed". If this was not mitigated, the scope of the outage would have increased to include non-cloud Google services.
- ryanobjc 11y agothere's a lot of infrastructure at Google. And claim is correct - GCP and Google is on top of the same servers, same backend systems. Are they on the exactly same servers? No, of course not -- there's a few servers out there :-)
- NetStrikeForce 11y agoThis was a global network outage, we can't talk about the "exact same servers". There seems to be an implication that Google runs on GCP and a global network outage can't affect all GCP's customers but one.
- avs733 11y agoThere has to be a certain level of karma/schadenfreude of this happening in the week where they are pushing their SRE book...did they handle it well? It seems so, but a lot of their book is an ounce of prevention over a pound of on-call pagers going off.
- packetslave 11y agoPrevention is a big part of SRE, but an equally big part is formalizing a process to learn from the inevitable outages that come with running a large, complex, distributed system built by fallible humans. You figure out what went wrong and fix it, of course, but more importantly, you figure out where your existing systems and processes (failover, monitoring, incident response, etc.) did and didn't work, and you improve them for the next time.
- tempestn 11y agoGoogle Custom Search also seems to have gone down globally today. Likely related, although GCE is back up, CSE is still out, leaving many sites without an international search feature.
- max_ 11y agoso much for the SRE they publishd last week
- kyrra 11y ago16-17 minutes of down time isn't all that bad if you consider the SLA for GCE is 99.95%: https://cloud.google.com/compute/sla https://cloud.google.com/compute/sla So they can have 262 minutes of down time a year and still be within their SLA.
- salilpa 11y agoi migrated from aws to google yesterday. fml
- kahwooi 11y agoI did not choose because GCE does not have server at Asia Pacific, Microsoft Azure, DigitalOcean and AWS has one. Sorry, correction, What I mean is South East Asia.
- gresrun 11y agoThey have three zones in Asia Pacific[0]. [0] https://cloud.google.com/compute/docs/zones#available https://cloud.google.com/compute/docs/zones#available
- simonebrunozzi 11y agoAWS had several major outages in the past, especially between 2009 and 2012. In some cases, it was not only downtime, but also data loss, which is the hardest part. 8,760 hours in a year, if you are down for a total of less than 8.7 hours, you're in the >99.9% uptime category (also called "three 9s"). Four 9s (99.99%) is considered a nice plus. Very few businesses really need that. However, uptime is one thing. Data loss is another different beast. AFAIK, this one for Google is only downtime. Being able to maintain GCE up for most of the year, except a few hours, means more than 99.9% availability, which is what most customers need. Operational excellence, or the ability to have your cloud up and running, comes only with a large customer set; Google is now gaining a lot of significant customers (Spotify, Snapchat, Apple, etc), and therefore I expect them to learn what's needed over the coming months. 2016 will be ok. 2017, in my view, will be near perfect. If Google wants to differentiate them from AWS, they should offer an SLA on data integrity (at a premium, obviously). Here's how you can get thousands of enterprise customers. Shameless plug: I've also extensively written about AWS, Azure and GCE here: https://medium.com/simone-brunozzi/the-cloud-wars-of-2016-3f87e0a03d18 https://medium.com/simone-brunozzi/the-cloud-wars-of-2016-3f...
- trufflepiggames 11y agoMaybe it was just that bastard of game developer from Germany, I know him all too well. http://www.bridezilla-game.com http://www.bridezilla-game.com
- novaleaf 11y agoCorrect me if i'm wrong, but it looks like Cloud VPN went down, not all of GCE. FYI I run about 15 servers in Asia, USAEast, and Europe on GCE with external monitoring and didn't get a peep from my error checking emails during that timeframe.