3 ms·

Place those bets, was it BGP? My money is always on BGP somehow.

by iou 4y ago

mmaunder 4y ago
BGP problems will usually manifest as something being unreachable. This seems to be 500 errors which suggest code breaking somewhere.
- assttoasstmgr 4y ago
  I was getting mostly 502's indicating their reverse proxies couldn't reach the back end so it's very possible an internal routing issue was at fault.
  - throwdbaaway 4y ago
    Misconfiguration at GFE control plane?
inferiorhuman 4y ago
Wonder if it had anything to do with char broiling a few of their electricians this morning.
- mmaunder 4y ago
  Say what??
  - lamontcg 4y ago
    https://www.msn.com/en-us/news/us/three-people-in-critical-condition-after-google-data-center-electrical-incident/ar-AA10sb68 https://www.msn.com/en-us/news/us/three-people-in-critical-c...
  - jimmygrapes 4y ago
    Probably referring to this: https://www.msn.com/en-us/news/us/three-people-in-critical-condition-after-google-data-center-electrical-incident/ar-AA10sb68 https://www.msn.com/en-us/news/us/three-people-in-critical-c... (sorry for msn link, archiving on phone is rough)
    - inferiorhuman 4y ago
      Yes, that. People downvote anything these days.
      - herendin 4y ago
        >People downvote anything these days. Maybe because you said something crudely insensitive in a community that might know the victims, and definitely includes people who do similar jobs? I hope none of your friends or family ever get char broiled.
        inferiorhuman 4y ago
        Yeah, Hacker News that bastion of tolerance and civility is aghast at gallows humor. I'm shocked! Shocked I say! Oh well, I guess shitting all over women and minorities in tech is a more comfortable target for this demographic.
abhishekjha 4y ago
5xx responses mostly come from their own server implying that we are able to reach them. Unlikely to be a routing error.
41b696ef1113 4y ago
Maybe someone tried to resurrect the Search Timer code [0]. [0]: https://news.ycombinator.com/item?id=32295674 https://news.ycombinator.com/item?id=32295674
Titan2189 4y ago
My bet: It’s not DNS There is a no way it’s DNS It was DNS
- utopcell 4y ago
  it was not DNS
qmarchi 4y ago
I'll place bets on that. A) Googler B) it wasn't DNS or BGP

Place those bets, was it BGP? My money is always on BGP somehow.

by iou 4y ago

Place those bets, was it BGP? My money is always on BGP somehow.

BGP problems will usually manifest as something being unreachable. This seems to be 500 errors which suggest code breaking somewhere.

assttoasstmgr 4y ago
I was getting mostly 502's indicating their reverse proxies couldn't reach the back end so it's very possible an internal routing issue was at fault.
- throwdbaaway 4y ago
  Misconfiguration at GFE control plane?

inferiorhuman 4y ago

Wonder if it had anything to do with char broiling a few of their electricians this morning.

mmaunder 4y ago
Say what??
- lamontcg 4y ago
  https://www.msn.com/en-us/news/us/three-people-in-critical-condition-after-google-data-center-electrical-incident/ar-AA10sb68 https://www.msn.com/en-us/news/us/three-people-in-critical-c...
- jimmygrapes 4y ago
  Probably referring to this: https://www.msn.com/en-us/news/us/three-people-in-critical-condition-after-google-data-center-electrical-incident/ar-AA10sb68 https://www.msn.com/en-us/news/us/three-people-in-critical-c... (sorry for msn link, archiving on phone is rough)
  - inferiorhuman 4y ago
    Yes, that. People downvote anything these days.
    - herendin 4y ago
      >People downvote anything these days. Maybe because you said something crudely insensitive in a community that might know the victims, and definitely includes people who do similar jobs? I hope none of your friends or family ever get char broiled.
      - inferiorhuman 4y ago
        Yeah, Hacker News that bastion of tolerance and civility is aghast at gallows humor. I'm shocked! Shocked I say! Oh well, I guess shitting all over women and minorities in tech is a more comfortable target for this demographic.

abhishekjha 4y ago

5xx responses mostly come from their own server implying that we are able to reach them. Unlikely to be a routing error.

41b696ef1113 4y ago

Maybe someone tried to resurrect the Search Timer code [0]. [0]: https://news.ycombinator.com/item?id=32295674 https://news.ycombinator.com/item?id=32295674

Titan2189 4y ago

My bet: It’s not DNS There is a no way it’s DNS It was DNS

utopcell 4y ago
it was not DNS

qmarchi 4y ago

I'll place bets on that. A) Googler B) it wasn't DNS or BGP