4 ms·

The author (Tom Paseka) wrote near the conclusion that himself addressed the Google's issue, by contacting a Moratel's engineer. Do you have the same feeling wh

by ybaumes 14y ago

The author (Tom Paseka) wrote near the conclusion that himself addressed the Google's issue, by contacting a Moratel's engineer. Do you have the same feeling when reading the article? It sounds weird that Google did not triggered a recovery procedure on its own. Maybe I see bad things everywhere and you may call paranoïd, but could it be some sort of ("false") advertising on the side of cloudfare?

archangel_one 14y ago
I'm not a network engineer, but it seems like the kind of thing that might be very hard to detect when you're already inside or near to the google.com domain. Or maybe CloudFlare just got there first. I don't think it's necessary to call BS on Cloudflare without any kind of evidence at all.
- emilw 14y ago
  Stop excusing yourself for not being something, either read more and then comment or trust your gut feeling. (not meant to sound harsh)
  - JohnLBevan 14y ago
    I think it's good to qualify your opinion with your level of expertise. There's no rule that says HN should only be for discussion by experts (hopefully there never will be), and if you don't know something for sure it's best to say so that others don't take your word as gospel. That said, I'm no expert ;).
- amalcon 14y ago
  This is basically correct. BGP is weird. The addresses for one of Google's many datacenters were routed incorrectly for packets coming from some subset of IP space. Unless Google is running active ping tests to that subset of IP space, the way they would normally detect it is for someone to call and complain. In this case, the author decided to take a shortcut and call the owner of the "problem peer" directly.
  - veidr 14y ago
    Although only a vanishingly small percentage of Google users can call and complain. Blog or tweet or post to HN and hope Matt Cutts sees it and notifies the right team, maybe.
    - mh- 14y ago
      >Blog or tweet or post to HN and hope Matt Cutts sees it and notifies the right team, maybe. It seems that's more or less the quasi-official support channel even for paid services from Google.
    - Matt_Cutts 14y ago
      A team of Googlers could have been working on this in parallel to Tom. I'm guessing that a sudden drop of queries like that would cause people at Google to start digging into what happened. I don't know either way, because network ops and BGP is pretty far from my area (search quality).
  - jauer 14y ago
    A common way to notice things like this is to subscribe to a service like Renesys or Cyclops (http://cyclops.cs.ucla.edu/ http://cyclops.cs.ucla.edu/) that will alert you if it sees your subnets being announced by a different AS.
precisioncoder 14y ago
The key quote seems to be "Looking at peering maps, I'd estimate the outage impacted around 3–5% of the Internet's population." so if it didn't affect google directly it would have to go through customer server -> network technicians which would probably take more than the 26 min that Google was down for those customers for. I'm sure they would have been right on it if it hadn't have been fixed so fast.
michaelt 14y ago
It sounds weird that Google did not triggered a recovery procedure on its own. It's possible they didn't have the personal contact details for the engineer capable of fixing the problem. We all know how hard it can be to contact a competent person at a big corporation when you have a problem [1]. Would Google find it easier than every other human being? [1] http://xkcd.com/806/ http://xkcd.com/806/
notatoad 14y ago
It sounds like cloudflare simply got there first. Unless cloudflare is outright lying (highly unlikely) they saw a problem they could fix, and fixed it. What's the false advertising there?
- dsl 14y ago
  They didn't fix anything. Multiple people noticed, all of them contacted the network in question, then they took credit publicly when another network fixed its mistake.

by ybaumes 14y ago

archangel_one 14y ago

I'm not a network engineer, but it seems like the kind of thing that might be very hard to detect when you're already inside or near to the google.com domain. Or maybe CloudFlare just got there first. I don't think it's necessary to call BS on Cloudflare without any kind of evidence at all.

emilw 14y ago
Stop excusing yourself for not being something, either read more and then comment or trust your gut feeling. (not meant to sound harsh)
- JohnLBevan 14y ago
  I think it's good to qualify your opinion with your level of expertise. There's no rule that says HN should only be for discussion by experts (hopefully there never will be), and if you don't know something for sure it's best to say so that others don't take your word as gospel. That said, I'm no expert ;).
amalcon 14y ago
This is basically correct. BGP is weird. The addresses for one of Google's many datacenters were routed incorrectly for packets coming from some subset of IP space. Unless Google is running active ping tests to that subset of IP space, the way they would normally detect it is for someone to call and complain. In this case, the author decided to take a shortcut and call the owner of the "problem peer" directly.
- veidr 14y ago
  Although only a vanishingly small percentage of Google users can call and complain. Blog or tweet or post to HN and hope Matt Cutts sees it and notifies the right team, maybe.
  - mh- 14y ago
    >Blog or tweet or post to HN and hope Matt Cutts sees it and notifies the right team, maybe. It seems that's more or less the quasi-official support channel even for paid services from Google.
  - Matt_Cutts 14y ago
    A team of Googlers could have been working on this in parallel to Tom. I'm guessing that a sudden drop of queries like that would cause people at Google to start digging into what happened. I don't know either way, because network ops and BGP is pretty far from my area (search quality).
- jauer 14y ago
  A common way to notice things like this is to subscribe to a service like Renesys or Cyclops (http://cyclops.cs.ucla.edu/ http://cyclops.cs.ucla.edu/) that will alert you if it sees your subnets being announced by a different AS.

precisioncoder 14y ago

The key quote seems to be "Looking at peering maps, I'd estimate the outage impacted around 3–5% of the Internet's population." so if it didn't affect google directly it would have to go through customer server -> network technicians which would probably take more than the 26 min that Google was down for those customers for. I'm sure they would have been right on it if it hadn't have been fixed so fast.

michaelt 14y ago

It sounds weird that Google did not triggered a recovery procedure on its own. It's possible they didn't have the personal contact details for the engineer capable of fixing the problem. We all know how hard it can be to contact a competent person at a big corporation when you have a problem [1]. Would Google find it easier than every other human being? [1] http://xkcd.com/806/ http://xkcd.com/806/

notatoad 14y ago

It sounds like cloudflare simply got there first. Unless cloudflare is outright lying (highly unlikely) they saw a problem they could fix, and fixed it. What's the false advertising there?

dsl 14y ago
They didn't fix anything. Multiple people noticed, all of them contacted the network in question, then they took credit publicly when another network fixed its mistake.