4 ms·

Hi all - Seth from Google here. Our team is aware and we are working on mitigation. In short, a third party telco provider is advertising on one of our IP block

Hi all - Seth from Google here. Our team is aware and we are working on mitigation. In short, a third party telco provider is advertising on one of our IP blocks. Unfortunately that's all the information I can share at this time.

fxdoublecute 8y ago
thanks for the update! FWIW we started noticing the connectivity problems around 2018-11-12 21:17 UTC
- sethvargo 8y ago
  Thanks for the info. We have reports showing it started a bit earlier than that, but every piece of information is helping in managing an incident. I'll make sure the team is aware.
konschubert 8y ago
EDIT: This is a general statement, I am not complaining to google here. This kind of thing should not be possible. Are there any protocol proposals or other kind of upgrades to the routing protocols that would prevent these kind of mistakes/attacks?
- cm2187 8y ago
  I am surprised how fragile is the internet given how our society is increasingly becoming critically reliant on it.
  - QML 8y ago
    The internet was not really built with security in mind. Look at DNSSEC or BGPSEC.
  - zzzcpan 8y ago
    On the other hand it's not that fragile everywhere and for everyone. When ISP markets are not monopolized and the service doesn't rely on a big cloud - much fewer users will get rerouted through random countries and the service itself can failover to properly working datacenters, tolerating all those BGP misdesigns. It's if the internet doesn't like all that centralization with all that market domination. It's naturally resilient only when there is a lot of competition.
- jldugger 8y ago
  > This kind of thing should not be possible. It sounds like you're asking google to solve https://en.wikipedia.org/wiki/BGP_hijacking https://en.wikipedia.org/wiki/BGP_hijacking ?
  - konschubert 8y ago
    Sorry, I didn't intend it to be directed at google.
    - timdierks 8y ago
      The Internet is assembled out of duct tape. We apologize for the design.
- viraptor 8y ago
  Check out BGPSec and RPKI - they should prevent issues like this one. They're not widely implemented/enforced. Maybe it's going to change though now that it looks like we've got a "misconfiguration" somewhere every month or so.
- neuromantik8086 8y ago
  Resource Public Key Infrastructure, but ISPs are too cheap to actually implement it.
- raesene9 8y ago
  yeah there's been proposals on improving BGP security for at least 14 years that I've been aware of :) Getting the big ISPs/Telcos to adopt them... that's another matter
  - almost_usual 8y ago
    I'm thinking 1998 and Peiter Zatko
- draw_down 8y ago
  > This kind of thing should not be possible. When reality conflicts with what you believe to be possible, it's time to reexamine your assumptions.
- tinus_hn 8y ago
  There are a lot of proposals but the problem is quite difficult to begin with and also involves centralizing policy. And then the chosen protocol has to be implemented by parties that tend to move at a glacial pace.
- red0point 8y ago
  Yes there is an approach out there, solving many problems of the internet at once. It‘s called SCION and is being used in production at large swiss banks today. https://www.scion-architecture.net/ https://www.scion-architecture.net/
sethvargo 8y ago
We've updated our status page with as much information as we can provide at this time: https://status.cloud.google.com/incident/cloud-networking/18018 https://status.cloud.google.com/incident/cloud-networking/18... Our teams are continuing to work with upstream and downstream service providers to remedy the issue.
amingilani 8y ago
Reminds me of the time Pakistan knocked YouTube offline by hijacking their IPs globally[0] Edit: Didn't someone recently share a tool to monitor BGP hijack attempts? [0]: https://www.cnet.com/news/how-pakistan-knocked-youtube-offline-and-how-to-make-sure-it-never-happens-again/ https://www.cnet.com/news/how-pakistan-knocked-youtube-offli...
garysahota93 8y ago
I love that Google monitors this site. I really appreciate you reaching out and letting us know the current status!

fxdoublecute 8y ago

thanks for the update! FWIW we started noticing the connectivity problems around 2018-11-12 21:17 UTC

sethvargo 8y ago
Thanks for the info. We have reports showing it started a bit earlier than that, but every piece of information is helping in managing an incident. I'll make sure the team is aware.

konschubert 8y ago

EDIT: This is a general statement, I am not complaining to google here. This kind of thing should not be possible. Are there any protocol proposals or other kind of upgrades to the routing protocols that would prevent these kind of mistakes/attacks?

cm2187 8y ago
I am surprised how fragile is the internet given how our society is increasingly becoming critically reliant on it.
- QML 8y ago
  The internet was not really built with security in mind. Look at DNSSEC or BGPSEC.
- zzzcpan 8y ago
  On the other hand it's not that fragile everywhere and for everyone. When ISP markets are not monopolized and the service doesn't rely on a big cloud - much fewer users will get rerouted through random countries and the service itself can failover to properly working datacenters, tolerating all those BGP misdesigns. It's if the internet doesn't like all that centralization with all that market domination. It's naturally resilient only when there is a lot of competition.
jldugger 8y ago
> This kind of thing should not be possible. It sounds like you're asking google to solve https://en.wikipedia.org/wiki/BGP_hijacking https://en.wikipedia.org/wiki/BGP_hijacking ?
- konschubert 8y ago
  Sorry, I didn't intend it to be directed at google.
  - timdierks 8y ago
    The Internet is assembled out of duct tape. We apologize for the design.
viraptor 8y ago
Check out BGPSec and RPKI - they should prevent issues like this one. They're not widely implemented/enforced. Maybe it's going to change though now that it looks like we've got a "misconfiguration" somewhere every month or so.
neuromantik8086 8y ago
Resource Public Key Infrastructure, but ISPs are too cheap to actually implement it.
raesene9 8y ago
yeah there's been proposals on improving BGP security for at least 14 years that I've been aware of :) Getting the big ISPs/Telcos to adopt them... that's another matter
- almost_usual 8y ago
  I'm thinking 1998 and Peiter Zatko
draw_down 8y ago
> This kind of thing should not be possible. When reality conflicts with what you believe to be possible, it's time to reexamine your assumptions.
tinus_hn 8y ago
There are a lot of proposals but the problem is quite difficult to begin with and also involves centralizing policy. And then the chosen protocol has to be implemented by parties that tend to move at a glacial pace.
red0point 8y ago
Yes there is an approach out there, solving many problems of the internet at once. It‘s called SCION and is being used in production at large swiss banks today. https://www.scion-architecture.net/ https://www.scion-architecture.net/

sethvargo 8y ago

We've updated our status page with as much information as we can provide at this time: https://status.cloud.google.com/incident/cloud-networking/18018 https://status.cloud.google.com/incident/cloud-networking/18... Our teams are continuing to work with upstream and downstream service providers to remedy the issue.

amingilani 8y ago

Reminds me of the time Pakistan knocked YouTube offline by hijacking their IPs globally[0] Edit: Didn't someone recently share a tool to monitor BGP hijack attempts? [0]: https://www.cnet.com/news/how-pakistan-knocked-youtube-offline-and-how-to-make-sure-it-never-happens-again/ https://www.cnet.com/news/how-pakistan-knocked-youtube-offli...

garysahota93 8y ago

I love that Google monitors this site. I really appreciate you reaching out and letting us know the current status!