6 ms·
Yesterday's Google outage was a pool of routing servers crashing
- heartbeats 6y agoAnyone know what routing servers? BGP?
- mcpherrinm 6y agoI don't know the details of Google's networks, but I assume something like their Maglev load balancers: https://research.google/pubs/pub44824/ https://research.google/pubs/pub44824/
- MyelinatedT 6y agoSounds more like an application load balancer issue ("routing requests" seems to imply L7) than network routing, but I might be misunderstanding.
- kyrra 6y ago(Googler, opinion is my own, I know nothing about this specific outage). Google has LOTS of internal routing systems. BGP is about anouncing what IPs a given network can handle, which is not the case here. Before hitting application level routing, I believe you hit the Maglev[0]. Seems unlikely this was the cause, as it would likely take down all services. One of the first application layers balancers you hit that is well known is the GFE[1][2]. This is similar to an HTTP reverse proxy, but Google made. I could definitely see this as the cause. [0] https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/44824.pdf https://static.googleusercontent.com/media/research.google.c... [1] https://cloud.google.com/security/infrastructure/design#google_front_end_service https://cloud.google.com/security/infrastructure/design#goog... [2] https://landing.google.com/sre/workbook/chapters/managing-load/ https://landing.google.com/sre/workbook/chapters/managing-lo...
- tyingq 6y agoDoes that match the list of reported stuff that was down? It appeared to hit a wide range of services. Gmail, Analytics, GKE, Google Keep, Meet, YouTube, GCE buckets, Sheets, Docs, Calendar, Stadia, Firebase, Voice, Music, Nest. From the thread: https://news.ycombinator.com/item?id=24585478 https://news.ycombinator.com/item?id=24585478
- the-rc 6y agoNeither Maglev nor GFE are usually tied to a specific service nowadays, so it could still be either of them. Way back when, some teams or services such as Checkout had to run their own private pool of GFEs. Given Urs' mention of backends, I am slightly inclined toward GFE.
- enneff 6y agoTraffic entering Google's network hits a bunch of front ends that route traffic to the relevenat back ends. I'd guess it's those application-level front ends that were having trouble, rather than anything network-level like BGP.
- tyingq 6y agoIn a follow up tweet, he mentions a post mortem is coming. Where would that get posted?
- kyrra 6y agoIt tends to be linked from the outage itself, probably here: https://status.cloud.google.com/incident/zall/20010 https://status.cloud.google.com/incident/zall/20010 For example, PM was posted on this previous outage: https://status.cloud.google.com/incident/cloud-networking/19009 https://status.cloud.google.com/incident/cloud-networking/19...
- advisedwang 6y agoAs this impacted G Suite much more than GCP, it's possible it will be posted on https://www.google.com/appsstatus https://www.google.com/appsstatus instead.
- skim_milk 6y agoThere's a huge """secret""" Google data center in Council Bluffs, Iowa that appears to be in the finishing phases of completion. I talked yesterday to a union worker who is moving to Des Moines to work on a new Microsoft data center there tonight, it appears that work is drying up at this data center here and a lot of the travelling blue collar folk are leaving this area. I wonder if this data center coming apparently partially online is a part of the problem? Also, after this he is likely to work on an Amazon fulfillment center next year - impressed by all the (albeit temporary) blue collar jobs created by FAANG at the moment!
- throwawayinfo 6y agoSpoiler alert: it's not.
- SteveNuts 6y agoI think by now Google has lots of experience with bringing data centers online, so this seems incredibly far fetched to me.
- jeffbee 6y agoLOL you mean this one that has a big sign out front that's been there for years and years? https://www.google.com/maps/@41.2197694,-95.8658016,3a,89.3y,307.91h,90.59t/data=!3m6!1e1!3m4!1sNSMqEK0LvfGn4FY_ijLw1Q!2e0!7i16384!8i8192 https://www.google.com/maps/@41.2197694,-95.8658016,3a,89.3y...
- joshuamorton 6y agoAlso like: https://www.google.com/about/datacenters/locations/ https://www.google.com/about/datacenters/locations/
- tidepod12 6y agoAren't there GCP regions in Los Angeles and Salt Lake City? Interesting that those DCs don't seem to be on this public list.
- 6y ago
- Kaknut 6y agoDependencies of these big providers like Google, Microsoft, Cloudflare are increasing which results to failure on a wide scale even if one fails. Distribution is the key.
- auganov 6y agoWell for the vast majority of simple apps you're better off failing when everybody else is. People will blame it on you less. When your alternative solution fails and everything else seems to be up the blame will fall on you.
- Kaknut 6y agoI always prefer to have a backup solution which could at least crawl during these situation if not able to walk. I see many SaaS relying only on google/twitter/fb auth but they need to understand that having own system too won't harm them much.
- tyingq 6y agoGoogle could probably do a better job here and not put so many services on the same pool of L7 devices. Separate pools with smaller groupings would reduce the blast radius.