14 ms·
Disclosure: I work on Google Cloud (but disclaimer, I'm on vacation and so not much use to you!). We're having what appears to be a serious networking outage.
by boulos 7y ago
Disclosure: I work on Google Cloud (but disclaimer, I'm on vacation and so not much use to you!).
We're having what appears to be a serious networking outage. It's disrupting everything, including unfortunately the tooling we usually use to communicate across the company about outages.
There are backup plans, of course, but I wanted to at least come here to say: you're not crazy, nothing is lost (to those concerns downthread), but there is serious packet loss at the least. You'll have to wait for someone actually involved in the incident to say more.
- odiroot 7y ago> including unfortunately the tooling we usually use to communicate across the company about outages. There's some irony in that.
- boulos 7y agoEdit: and I agree! I’m not in SRE so I don’t bother with all the backup modes (direct IRC channel, phone lines, “pagers” with backup numbers). I don’t think the networking SRE folks are as impacted in their direct communication, but they are (obviously) not able to get the word out as easily. Still, it seems reasonable to me to use tooling for most outages that relies on “the network is fine overall”, to optimize for the common case. Note: the status dashboard now correctly highlights (Edit: with a banner at the top) that multiple things are impacted because Networking. The Networking outage is the root cause.
- marksomnian 7y ago> the status dashboard now correctly highlights that multiple things are impacted because Networking. this column of green checkmarks begs to differ: https://i.imgur.com/2TPD9e9.png https://i.imgur.com/2TPD9e9.png
- boulos 7y agoThe banner at the top. Sorry if that wasn’t clear.
- pm90 7y agoThis is a person who's trying to help out while on vacation...can we try being more thankful, and not nitpick everything they say?
- deleted 7y ago[deleted]
- boulos 7y agoThanks! I’ll leave this here as evidence that I should rightfully reduce my days off by 1 :).
- seltzered_ 7y agoWhile not exactly google cloud, G suite dashboard seems accurate: https://www.google.com/appsstatus#hl=en&v=status https://www.google.com/appsstatus#hl=en&v=status
- TimothyBJacobs 7y agoFor me, at least, that was showing as all green for at least 30 minutes.
- Twirrim 7y agoAWS experienced a major outage a few years ago that couldn't be communicated to customers because it took out all the components central to update the status board. One of those obvious-in-hindsight situations. Not long after that incident, they migrated it to something that couldn't be affected by any outage. I imagine Google will probably do the same thing after this :)
- captn3m0 7y agoWas just reading it, they made their status page multi-region.
- techslave 7y ago> Not long after that incident, they migrated it to something that couldn't be affected by any outage. Like the black box on an airplane, if it has 100% uptime why don’t they build the whole thing out of that? ;)
- flurdy 7y agoThe status page is the kind of thing you expect to be hosted on a competitor network. It is not dogfooding but it is sensible. Reminds me of when I was working with a telecoms company. It was a large multinational company and the second largest network in the country I was in at the time. I was surprised when I noticed all the senior execs were carrying two phones, of which the second was a mobile number on the main competitor (ie the largest network). After a while, I realised that it made sense, as when the shit really hit the fan they could still be reached even when our network had a total outage.
- k_bx 7y agoEven more irony: Google+ shown as working fine: https://i.imgur.com/52ACuiY.png https://i.imgur.com/52ACuiY.png
- chupasaurus 7y agoG+ is alive and well for G Suite subscribers, not the general users.
- fredthomsen 7y agothey still have general users? hehe yes I am trolling
- SmokeGS 7y ago>nothing is lost except time
- sdan 7y agoand a nice Sunday afternoon
- toomuchtodo 7y agoAnd the illusion of superiority over non cloud offerings.
- k__ 7y agoHow come?
- toomuchtodo 7y agoThere are some whole argue that the resiliency of cloud providers beats on prem or self hosted, and yet they’re down just as much or more (GCP, Azure, and AWS all the same). Don’t take my word for it; search HN for “$provider is down” and observe the frequency of occurrences. You want velocity for your dev team? You get that. You want better uptime? Your expectations are gonna have a bad time. No need for rapid dev or bursty workloads? You’re lighting money on fire. Disclaimer: I get paid to move clients to or from the cloud, everyone’s money is green. Opinion above is my own.
- TeMPOraL 7y agoSolutions based on third-party butts have essentially two modes: the usual, where everything is smooth, and the bad one, where nothing works and you're shit out of luck - you can't get to your data anymore, because it's in my butt, accessible only through that butt, and arguably not even your data. With on-prem solutions, you can at least access the physical servers and get your data out to carry on with your day while the infrastructure gets fixed.
- 7y ago
- 123jay7 7y agoThis happened to Amazon S3 as well once. The "X" image they use to indicate a service outage was served by... yup, S3, which was down obviously.
- hinkley 7y agoOne of the projects I worked on was using data URIs for critical images, and I wouldn’t trust that particular team to babysit my goldfish. Sounds like Google and Amazon are hiring way too many optimists. I kinda blame the war on QA for part of this, but damn that’s some Pollyanna bullshit.
- boulos 7y agoTo clarify something: this outage doesn’t appear to be global, but it is hitting us particularly hard in parts of the US. So for the folks with working VMs in Mumbai, you’re not crazy. But for everyone with sadness in us-central1, the team is on it.
- digaozao 7y agoIt seems global to me. This is really strange compared to AWS. I don't remember an outage there (other than s3) impacting instances or networking globally.
- murat124 7y agoYou obviously don't recall the early years of AWS. Half of internet would go down for hours.
- ljm 7y agoAlways in Virginia, because US-east has always been cheaper.
- mooreds 7y agoI know a consultant who calls that region us-tirefire-1.
- autotune 7y agoI and some previous coworkers call it the YOLO region.
- Scoundreller 7y agoAnd for those of us in GST/HST/VAT land, hosting in USA saves us some tax expenditures.
- BillinghamJ 7y ago
- bufferoverflow 7y agoWell, lots of revenue is lost, that's for sure.
- ChuckMcM 7y agoI'm guessing this will be part of the next DiRT exercise :-) (DiRT being the disaster recovery exercises that Google runs internally to prepare for this sort of thing)
- foobarbazetc 7y agoSeems to be the private network. The public network looks fine to us from all over the world?
- nodesocket 7y agoNot on my end. Public access in us-west2 (Los Angeles) is down for me.
- foobarbazetc 7y agoHmmm... why is our monitoring network not showing that? Edit: ah, looks like the LB is sending LA traffic to Oregon.
- teamspirit 7y agoOur Oregon VMs are up.
- ohazi 7y ago> including unfortunately the tooling we usually use to communicate across the company about outages. So memegen is down?
- dosy 7y agoYou're brave to jump on here when on holiday! Shouldn't that outage system be aware when service heartbeats stop? Could this be a solar flare?
- gingabriska 7y agoWe are also hosted on GCP bit nothing is down for us. We are using 3 regions in US and 2 in EU.
- ikiris 7y agogo/stopleaks :)
- foota 7y agoHm, isn't releasing go links publicly also verboten? :)
- breadandcrumbel 7y agoWhat can be the reason for the outage? Can it be a cyber attack to your servers?
- Double_a_92 7y ago> but there is serious packet loss at the least. Can confirm with Gmail in Europe. Everything works but it's sluggish (i.e. no immediate reaction on button clicks).