3 ms·

Anyone know what routing servers? BGP?

mcpherrinm 6y ago
I don't know the details of Google's networks, but I assume something like their Maglev load balancers: https://research.google/pubs/pub44824/ https://research.google/pubs/pub44824/
MyelinatedT 6y ago
Sounds more like an application load balancer issue ("routing requests" seems to imply L7) than network routing, but I might be misunderstanding.
kyrra 6y ago
(Googler, opinion is my own, I know nothing about this specific outage). Google has LOTS of internal routing systems. BGP is about anouncing what IPs a given network can handle, which is not the case here. Before hitting application level routing, I believe you hit the Maglev[0]. Seems unlikely this was the cause, as it would likely take down all services. One of the first application layers balancers you hit that is well known is the GFE[1][2]. This is similar to an HTTP reverse proxy, but Google made. I could definitely see this as the cause. [0] https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/44824.pdf https://static.googleusercontent.com/media/research.google.c... [1] https://cloud.google.com/security/infrastructure/design#google_front_end_service https://cloud.google.com/security/infrastructure/design#goog... [2] https://landing.google.com/sre/workbook/chapters/managing-load/ https://landing.google.com/sre/workbook/chapters/managing-lo...
- tyingq 6y ago
  Does that match the list of reported stuff that was down? It appeared to hit a wide range of services. Gmail, Analytics, GKE, Google Keep, Meet, YouTube, GCE buckets, Sheets, Docs, Calendar, Stadia, Firebase, Voice, Music, Nest. From the thread: https://news.ycombinator.com/item?id=24585478 https://news.ycombinator.com/item?id=24585478
  - the-rc 6y ago
    Neither Maglev nor GFE are usually tied to a specific service nowadays, so it could still be either of them. Way back when, some teams or services such as Checkout had to run their own private pool of GFEs. Given Urs' mention of backends, I am slightly inclined toward GFE.
enneff 6y ago
Traffic entering Google's network hits a bunch of front ends that route traffic to the relevenat back ends. I'd guess it's those application-level front ends that were having trouble, rather than anything network-level like BGP.

I don't know the details of Google's networks, but I assume something like their Maglev load balancers: https://research.google/pubs/pub44824/ https://research.google/pubs/pub44824/

MyelinatedT 6y ago

Sounds more like an application load balancer issue ("routing requests" seems to imply L7) than network routing, but I might be misunderstanding.

kyrra 6y ago

(Googler, opinion is my own, I know nothing about this specific outage). Google has LOTS of internal routing systems. BGP is about anouncing what IPs a given network can handle, which is not the case here. Before hitting application level routing, I believe you hit the Maglev[0]. Seems unlikely this was the cause, as it would likely take down all services. One of the first application layers balancers you hit that is well known is the GFE[1][2]. This is similar to an HTTP reverse proxy, but Google made. I could definitely see this as the cause. [0] https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/44824.pdf https://static.googleusercontent.com/media/research.google.c... [1] https://cloud.google.com/security/infrastructure/design#google_front_end_service https://cloud.google.com/security/infrastructure/design#goog... [2] https://landing.google.com/sre/workbook/chapters/managing-load/ https://landing.google.com/sre/workbook/chapters/managing-lo...

tyingq 6y ago
Does that match the list of reported stuff that was down? It appeared to hit a wide range of services. Gmail, Analytics, GKE, Google Keep, Meet, YouTube, GCE buckets, Sheets, Docs, Calendar, Stadia, Firebase, Voice, Music, Nest. From the thread: https://news.ycombinator.com/item?id=24585478 https://news.ycombinator.com/item?id=24585478
- the-rc 6y ago
  Neither Maglev nor GFE are usually tied to a specific service nowadays, so it could still be either of them. Way back when, some teams or services such as Checkout had to run their own private pool of GFEs. Given Urs' mention of backends, I am slightly inclined toward GFE.

enneff 6y ago

Traffic entering Google's network hits a bunch of front ends that route traffic to the relevenat back ends. I'd guess it's those application-level front ends that were having trouble, rather than anything network-level like BGP.