7 ms·

Serious question - what could cause this? I don't know much about network configuration or DNS. Could it be a coordinated attack on Comcast's infrastructure som

Serious question - what could cause this? I don't know much about network configuration or DNS. Could it be a coordinated attack on Comcast's infrastructure somewhere? Seems a little crazy that they'd be experiencing problem nation-wide with no redundancy baked into their system. I tried manually flipping over to Google's DNS servers with the same result.

_rpd 9y ago
> what could cause this? It could be ... The Backhoe, The Internet's Natural Enemy (2006) https://it.slashdot.org/story/06/01/19/1643215/the-backhoe-the-internets-natural-enemy https://it.slashdot.org/story/06/01/19/1643215/the-backhoe-t...
Shelnutt2 9y ago
Since we are resolving dns, we know its not a dns issue. This appears to be pack loss / routing issue. The most likely cause of this is someone has leaked internal routes, or leaked bgp. I don't see anything from bgpmon (https://twitter.com/bgpmon https://twitter.com/bgpmon) or bgpstream (https://twitter.com/bgpstream https://twitter.com/bgpstream) for North America though. There have been multiple bgp leaks today but nothing that I can see that effects comcast/rnc. It does appear at least some site I'm seeing are being routed through Europe. traceroute kayak.com traceroute to kayak.com (185.6.169.69), 64 hops max, 52 byte packets 2 96.120.68.17 (96.120.68.17) 7.840 ms 8.995 ms 8.354 ms 3 96.108.154.229 (96.108.154.229) 8.136 ms 9.174 ms 8.325 ms 4 be-23-sur03.westroxbury.ma.boston.comcast.net (69.139.221.226) 9.879 ms 7.478 ms 7.667 ms 5 be-20-ar01.needham.ma.boston.comcast.net (68.85.106.21) 9.527 ms 12.000 ms 8.524 ms 6 * * * 7 ae-1-16.bar1.zurich1.level3.net (4.69.142.129) 118.578 ms 118.947 ms 118.229 ms 8 ae-1-16.bar1.zurich1.level3.net (4.69.142.129) 127.931 ms 116.424 ms 117.852 ms 9 nexellent-a.bar1.zurich1.level3.net (213.242.82.110) 118.340 ms 117.555 ms 115.950 ms
- Eyght 9y ago
  Seems like the UK and Germany are under relatively heavy load right now. https://www.akamai.com/uk/en/solutions/intelligent-platform/visualizing-akamai/real-time-web-monitor.jsp https://www.akamai.com/uk/en/solutions/intelligent-platform/...
- funwithjustin 9y ago
  Thanks for the info. How on earth could Comcast have a single point of failure for their entire network nation-wide? Would this suggest that there's possibly something like a MITM attack going on that's hijacking their traffic somewhere?
  - tclancy 9y ago
    >Thanks for the info. How on earth could Comcast have a single point of failure for their entire network nation-wide? I guess I've been a Comcast customer too long given I don't even understand your question. It would seem more shocking if it were not something like this.
quwert95 9y ago
I'm in the SF Bay Area and after looking at traceroutes and packet loss it looks like something at PAIX has failed between Comcast, Hurricane Electric, NTT, and others. A coordinated attack could be a cause of something like this, but more often than not it's a hardware or power failure at a critical exchange like PAO, DFW, IAD, etc. Also possible, human error in turning down traffic to certain servers, networks, and IP ranges.

_rpd 9y ago

> what could cause this? It could be ... The Backhoe, The Internet's Natural Enemy (2006) https://it.slashdot.org/story/06/01/19/1643215/the-backhoe-the-internets-natural-enemy https://it.slashdot.org/story/06/01/19/1643215/the-backhoe-t...

Shelnutt2 9y ago

Since we are resolving dns, we know its not a dns issue. This appears to be pack loss / routing issue. The most likely cause of this is someone has leaked internal routes, or leaked bgp. I don't see anything from bgpmon (https://twitter.com/bgpmon https://twitter.com/bgpmon) or bgpstream (https://twitter.com/bgpstream https://twitter.com/bgpstream) for North America though. There have been multiple bgp leaks today but nothing that I can see that effects comcast/rnc. It does appear at least some site I'm seeing are being routed through Europe. traceroute kayak.com traceroute to kayak.com (185.6.169.69), 64 hops max, 52 byte packets 2 96.120.68.17 (96.120.68.17) 7.840 ms 8.995 ms 8.354 ms 3 96.108.154.229 (96.108.154.229) 8.136 ms 9.174 ms 8.325 ms 4 be-23-sur03.westroxbury.ma.boston.comcast.net (69.139.221.226) 9.879 ms 7.478 ms 7.667 ms 5 be-20-ar01.needham.ma.boston.comcast.net (68.85.106.21) 9.527 ms 12.000 ms 8.524 ms 6 * * * 7 ae-1-16.bar1.zurich1.level3.net (4.69.142.129) 118.578 ms 118.947 ms 118.229 ms 8 ae-1-16.bar1.zurich1.level3.net (4.69.142.129) 127.931 ms 116.424 ms 117.852 ms 9 nexellent-a.bar1.zurich1.level3.net (213.242.82.110) 118.340 ms 117.555 ms 115.950 ms

Eyght 9y ago
Seems like the UK and Germany are under relatively heavy load right now. https://www.akamai.com/uk/en/solutions/intelligent-platform/visualizing-akamai/real-time-web-monitor.jsp https://www.akamai.com/uk/en/solutions/intelligent-platform/...
funwithjustin 9y ago
Thanks for the info. How on earth could Comcast have a single point of failure for their entire network nation-wide? Would this suggest that there's possibly something like a MITM attack going on that's hijacking their traffic somewhere?
- tclancy 9y ago
  >Thanks for the info. How on earth could Comcast have a single point of failure for their entire network nation-wide? I guess I've been a Comcast customer too long given I don't even understand your question. It would seem more shocking if it were not something like this.

quwert95 9y ago

I'm in the SF Bay Area and after looking at traceroutes and packet loss it looks like something at PAIX has failed between Comcast, Hurricane Electric, NTT, and others. A coordinated attack could be a cause of something like this, but more often than not it's a hardware or power failure at a critical exchange like PAO, DFW, IAD, etc. Also possible, human error in turning down traffic to certain servers, networks, and IP ranges.