6 ms·

Massive network issues across Europe

110 points by abritishguy 12y ago

Well I'm not sure exactly what it is but a twitter search has confirmed it is not just me. Cloudflare sites (including reddit) are being very very slow and most requests end with a gateway timeout. Akamai CDN is the same (facebook images, twitter images) See traffic map here: http://www.akamai.com/html/technology/dataviz1.html 21:42 BST - 4/6 European cloudflare datacentres are currently offline. 21:49 BST - It has been reported that Telias Transatlantic cable is down 21:51 BST - Cloudflare confirm that Telias cable has been cut 21:56 BST - Traffic now being rerouted

rmoriz 12y ago
Germany: Dropbox unreachable, Tumblr very slow
deleted 12y ago
[deleted]
Strom 12y ago
I can pretty much only reach sites which are routed through the netherlands for me (like Cloudflare proxied HN, or Google). Everything else doesn't connect at all or times out after downloading 10% of the HTML. I'm connecting from Estonia. The Digital Ocean status page [1] claims that "Preliminary investigation indicates that Telias Transatlantic cable are down." which would make sense, because my ISP is Telia and I can't connect to most of the internet. Edit: Well, this outage lasted a bit over an hour. Now at 20:55 UTC I can access everything once again. [1] http://www.digitaloceanstatus.com/ http://www.digitaloceanstatus.com/
mcintyre1994 12y ago
> "One of our upstreams (Telia) appears to have a trans-Atlantic cable cut. We're routing around in many EU locations now." @cloudflare https://twitter.com/CloudFlare/status/468494276691640320 https://twitter.com/CloudFlare/status/468494276691640320
Ihmahr 12y ago
Transatlantic cable down. Are they busy installing a prism?
- rbanffy 12y ago
  Where is the USS Jimmy Carter?
  - politician 12y ago
    For everyone else who didn't catch the reference, apparently this Seawolf-class submarine can tap undersea cables. http://www.nytimes.com/2005/02/20/politics/20submarine.html http://www.nytimes.com/2005/02/20/politics/20submarine.html http://cryptome.org/eyeball/mmp/jimmy-carter.htm http://cryptome.org/eyeball/mmp/jimmy-carter.htm
- kabdib 12y ago
  They just want to redirect some traffic through the US for a while.
  - zokier 12y ago
    Yeah, that is kinda the point of having transatlantic cables in the first place :)
- JetSpiegel 12y ago
  So this is the famous "losing hearts and minds".
- deleted 12y ago
  [deleted]
crucialfelix 12y ago
mostly unreachable from berlin: rackspace.com, amazon.com but I see the traceroutes failing after they make it to the US.
- jaekwon 12y ago
  Weird, why would it fail after reaching the US?
  - bajsejohannes 12y ago
    It could be misleading. During the outage, I saw traceroutes like: 1 someserver.in.europe 10ms 15ms 20ms 2 someother.in.europe 50ms 60ms 50ms 3 * * * 4 * * * 5 * * * 6 * * * 7 * * * 8 * * * 9 someserver.in.us * 500ms * 10 * * * 11 otherserver.in.us 900ms * * So, it looks like for 10 and 11 hops we are losing a lot of traffic in the US, but that might as well be traffic that's lost in hop #3. The traceroute only tells you at some point between the start and hop #11 we lost 66% of the packets. It doesn't tell you where.
    - caw 12y ago
      You can also get * on traceroutes (step 3-8) because the devices don't respond to ICMP. Network admins like to disable ICMP on their switches and routers so you can't trace the network topology. Your only definite packet loss is on 9 and 11 because you only got 1 of the 3 packets returned.
zokier 12y ago
> It was a scheduled job that went a bit wrong, says Bo Djurberg, manager at Telia from http://www.aftonbladet.se/nyheter/article18915221.ab http://www.aftonbladet.se/nyheter/article18915221.ab (via google translate)
- sandstrom 12y ago
  The text [now updated] says it was solved at around 23:00 CET, after an hours outage.
mjcohen 12y ago
Russia starting a cyber-war?
- WellKnown 12y ago
  Actually recent news show that this is US who has the most control over the internet. But this outage could be natual disaster :-)
justincormack 12y ago
Clearly a test for who can withstand network partition...
jpdus 12y ago
Ah, thanks for the info. Connections to our US broker failed at 10pm (dst) on different servers (based in Germany) and i had no idea why, I already wanted to blame our hosting provider.
karnei 12y ago
Its amazing how well traffic gets rerouted now days
jaekwon 12y ago
Hi guys, I'm very curious about the effects of this. What is the network like? Can you characterize it in terms of latency, dropped packets, bandwidth? Say if you wanted to upload/download a megabyte of data to a datacenter in the US using TCP with retries, would it work eventually with high probability?
cordite 12y ago
It was really interesting how I could still communicate to some of the servers via connections that went through Level3. Yet once I got back in the server (behind that indirection), the server couldn't even ping google, and its DNS was totally down. Would've done a traceroute for fun if OVH's mirrors were reachable.

110 points by abritishguy 12y ago

rmoriz 12y ago

Germany: Dropbox unreachable, Tumblr very slow

deleted 12y ago

[deleted]

Strom 12y ago

I can pretty much only reach sites which are routed through the netherlands for me (like Cloudflare proxied HN, or Google). Everything else doesn't connect at all or times out after downloading 10% of the HTML. I'm connecting from Estonia. The Digital Ocean status page [1] claims that "Preliminary investigation indicates that Telias Transatlantic cable are down." which would make sense, because my ISP is Telia and I can't connect to most of the internet. Edit: Well, this outage lasted a bit over an hour. Now at 20:55 UTC I can access everything once again. [1] http://www.digitaloceanstatus.com/ http://www.digitaloceanstatus.com/

mcintyre1994 12y ago

> "One of our upstreams (Telia) appears to have a trans-Atlantic cable cut. We're routing around in many EU locations now." @cloudflare https://twitter.com/CloudFlare/status/468494276691640320 https://twitter.com/CloudFlare/status/468494276691640320

Ihmahr 12y ago

Transatlantic cable down. Are they busy installing a prism?

rbanffy 12y ago
Where is the USS Jimmy Carter?
- politician 12y ago
  For everyone else who didn't catch the reference, apparently this Seawolf-class submarine can tap undersea cables. http://www.nytimes.com/2005/02/20/politics/20submarine.html http://www.nytimes.com/2005/02/20/politics/20submarine.html http://cryptome.org/eyeball/mmp/jimmy-carter.htm http://cryptome.org/eyeball/mmp/jimmy-carter.htm
kabdib 12y ago
They just want to redirect some traffic through the US for a while.
- zokier 12y ago
  Yeah, that is kinda the point of having transatlantic cables in the first place :)
JetSpiegel 12y ago
So this is the famous "losing hearts and minds".
deleted 12y ago
[deleted]

crucialfelix 12y ago

mostly unreachable from berlin: rackspace.com, amazon.com but I see the traceroutes failing after they make it to the US.

jaekwon 12y ago
Weird, why would it fail after reaching the US?
- bajsejohannes 12y ago
  It could be misleading. During the outage, I saw traceroutes like: 1 someserver.in.europe 10ms 15ms 20ms 2 someother.in.europe 50ms 60ms 50ms 3 * * * 4 * * * 5 * * * 6 * * * 7 * * * 8 * * * 9 someserver.in.us * 500ms * 10 * * * 11 otherserver.in.us 900ms * * So, it looks like for 10 and 11 hops we are losing a lot of traffic in the US, but that might as well be traffic that's lost in hop #3. The traceroute only tells you at some point between the start and hop #11 we lost 66% of the packets. It doesn't tell you where.
  - caw 12y ago
    You can also get * on traceroutes (step 3-8) because the devices don't respond to ICMP. Network admins like to disable ICMP on their switches and routers so you can't trace the network topology. Your only definite packet loss is on 9 and 11 because you only got 1 of the 3 packets returned.

zokier 12y ago

> It was a scheduled job that went a bit wrong, says Bo Djurberg, manager at Telia from http://www.aftonbladet.se/nyheter/article18915221.ab http://www.aftonbladet.se/nyheter/article18915221.ab (via google translate)

sandstrom 12y ago
The text [now updated] says it was solved at around 23:00 CET, after an hours outage.

mjcohen 12y ago

Russia starting a cyber-war?

WellKnown 12y ago
Actually recent news show that this is US who has the most control over the internet. But this outage could be natual disaster :-)

justincormack 12y ago

Clearly a test for who can withstand network partition...

jpdus 12y ago

Ah, thanks for the info. Connections to our US broker failed at 10pm (dst) on different servers (based in Germany) and i had no idea why, I already wanted to blame our hosting provider.

karnei 12y ago

Its amazing how well traffic gets rerouted now days

jaekwon 12y ago

Hi guys, I'm very curious about the effects of this. What is the network like? Can you characterize it in terms of latency, dropped packets, bandwidth? Say if you wanted to upload/download a megabyte of data to a datacenter in the US using TCP with retries, would it work eventually with high probability?

cordite 12y ago

It was really interesting how I could still communicate to some of the servers via connections that went through Level3. Yet once I got back in the server (behind that indirection), the server couldn't even ping google, and its DNS was totally down. Would've done a traceroute for fun if OVH's mirrors were reachable.