5 ms·

Major routing issues on the Internet

118 points by whyleyc 9y ago

_eric 9y ago
OT, but that yellow/white combination of background and font color is terrible.
leesalminen 9y ago
I just saw this as well. Got a downtime alert, but app seems to be working from my location.
- juliand 9y ago
  Same here. Not sure if Pingdom is acting up or they legitimately can't reach my site due to the problems reported by Cloudflare or something.
gregschlom 9y ago
Reddit is down
- AznHisoka 9y ago
  no, it's not.
  - sccxy 9y ago
    It was. Or at least they showed me error message that their CDNs were down. Haven't seen that kind of error before in Reddit
  - apeacox 9y ago
    Maybe DNS issues
- mrspeaker 9y ago
  Status confirms something goin on! http://www.redditstatus.com/ http://www.redditstatus.com/ (http://imgur.com/a/WWUeo http://imgur.com/a/WWUeo)
goblin89 9y ago
Experienced uncharacteristic connection issues in the last ~15 minutes, was primarily SSHing into AWS EC2 at the time though so can’t comment on the true scale.
akhilcacharya 9y ago
Well that's...specific
prashnts 9y ago
From cloudflare status [1] posted on HN [2]: > Network Performance Issues in multiple locations > The issue is related to a specific transit provider and we are working on temporarily disabling this provider to route around the issue [1] https://www.cloudflarestatus.com/ https://www.cloudflarestatus.com/? [2] https://news.ycombinator.com/item?id=14246888 https://news.ycombinator.com/item?id=14246888
- semi-extrinsic 9y ago
  That cloudflare status is very strange: it says (said) Angola, Dubai, Philadelphia (US), some Australian locations and some in South America were affected. Anyone have a good explanation of such a random distribution?
  - p49k 9y ago
    I think they're just listing the POPs that were unreachable from their main office. It was definitely more widespread than that.
  - austinheap 9y ago
    Telia, the ISP that cause this issue, is very very large -- one of few Tier 1 ISPs. That said, when Tier 1 ISPs experience issues they're incredibly wide spread and behave oddly. The POPs you mention are the Cloudflare POPs that experienced degraded or dropped throughput due to Telia being (one of?) Cloudflare's preferred routes for that POP.
wiredfool 9y ago
HN has been sketchy for me for the last half hour or so.
Avenger42 9y ago
Is it possible to determine the transit provider at fault based on Cloudflare's list of affected areas?
- austinheap 9y ago
  I don't believe that would be possible... (Caveat: I am not the World's Smartest & Bestest Network Engineer Ever™, would be great if other's chimed in!) 1. Each Cloudflare POP has multiple bandwidth providers. 2. Each Cloudflare POP can and does change it's preferred BGP routes. 3. Telia's outage wasn't 100% of their network, so there are plenty of Telia links -- including ones to Cloudflare POPs -- that were not impacted.
simtel20 9y ago
Looks like telia was down and has fixed a mistake in their configs.
VA3FXP 9y ago
It looks like Telia is apparently the issue. Our CDN's are having problems all over the place. No indications of what shit the bed, but this is more then Cloudflare or Level3.
- rbjorklin 9y ago
  Do you have a source? (Would like to pass it on internally)
  - austinheap 9y ago
    Multiple ISPs I work with have explicitly stated to me that it was Telia. I'm not aware of any public sources confirming this yet.
  - VA3FXP 9y ago
    No, I have no public sources I can share. Just relaying what our CDN's have shared with me. I also can't provide anything specific. Sorry, I know it's unreliable and rumor but it's all I can provide.
leesalminen 9y ago
Cloudflare is reporting that the issue has been resolved. https://www.cloudflarestatus.com/incidents/51q3xhq8w7t8 https://www.cloudflarestatus.com/incidents/51q3xhq8w7t8
deleted 9y ago
[deleted]
tedmiston 9y ago
I'm curious what Pingdom uses to detect their own outages.
leke 9y ago
I noticed reddit went down briefly.
revmoo 9y ago
I've had Pingdom for about three months now. They have failed to alert on every outage we've had since I started INCLUDING the first "test outage" I created on day 1 to verify the monitoring was working. I am not impressed at all by their service and will probably be asking for a full refund depending on the outcome of my current trouble ticket with them. Also they've been deleting history of our outages after the fact; i.e. an outage occurs, they fail to alert but they do log it, and then several days later the outage is removed from their history. Pingdom is run by absolute clowns.
NKCSS 9y ago
A BGP problem again? Or something else this time? It amazes me that there has not been an alternative to handle routing except full trust for everyone.
redm 9y ago
We dropped Telia as a carrier about 2 months ago. I've never seen a network with such stability issues; outages/slowdowns occurred on a weekly basis for us and tech support was useless. The only good thing about Telia is they did a great job with service credits. According to VP's there they have been going through major transitions in the last year but given their Tier-1 status, you would expect more stability.

118 points by whyleyc 9y ago

_eric 9y ago

OT, but that yellow/white combination of background and font color is terrible.

leesalminen 9y ago

I just saw this as well. Got a downtime alert, but app seems to be working from my location.

juliand 9y ago
Same here. Not sure if Pingdom is acting up or they legitimately can't reach my site due to the problems reported by Cloudflare or something.

gregschlom 9y ago

Reddit is down

AznHisoka 9y ago
no, it's not.
- sccxy 9y ago
  It was. Or at least they showed me error message that their CDNs were down. Haven't seen that kind of error before in Reddit
- apeacox 9y ago
  Maybe DNS issues
mrspeaker 9y ago
Status confirms something goin on! http://www.redditstatus.com/ http://www.redditstatus.com/ (http://imgur.com/a/WWUeo http://imgur.com/a/WWUeo)

goblin89 9y ago

Experienced uncharacteristic connection issues in the last ~15 minutes, was primarily SSHing into AWS EC2 at the time though so can’t comment on the true scale.

akhilcacharya 9y ago

Well that's...specific

prashnts 9y ago

From cloudflare status [1] posted on HN [2]: > Network Performance Issues in multiple locations > The issue is related to a specific transit provider and we are working on temporarily disabling this provider to route around the issue [1] https://www.cloudflarestatus.com/ https://www.cloudflarestatus.com/? [2] https://news.ycombinator.com/item?id=14246888 https://news.ycombinator.com/item?id=14246888

semi-extrinsic 9y ago
That cloudflare status is very strange: it says (said) Angola, Dubai, Philadelphia (US), some Australian locations and some in South America were affected. Anyone have a good explanation of such a random distribution?
- p49k 9y ago
  I think they're just listing the POPs that were unreachable from their main office. It was definitely more widespread than that.
- austinheap 9y ago
  Telia, the ISP that cause this issue, is very very large -- one of few Tier 1 ISPs. That said, when Tier 1 ISPs experience issues they're incredibly wide spread and behave oddly. The POPs you mention are the Cloudflare POPs that experienced degraded or dropped throughput due to Telia being (one of?) Cloudflare's preferred routes for that POP.

wiredfool 9y ago

HN has been sketchy for me for the last half hour or so.

Avenger42 9y ago

Is it possible to determine the transit provider at fault based on Cloudflare's list of affected areas?

austinheap 9y ago
I don't believe that would be possible... (Caveat: I am not the World's Smartest & Bestest Network Engineer Ever™, would be great if other's chimed in!) 1. Each Cloudflare POP has multiple bandwidth providers. 2. Each Cloudflare POP can and does change it's preferred BGP routes. 3. Telia's outage wasn't 100% of their network, so there are plenty of Telia links -- including ones to Cloudflare POPs -- that were not impacted.

simtel20 9y ago

Looks like telia was down and has fixed a mistake in their configs.

VA3FXP 9y ago

It looks like Telia is apparently the issue. Our CDN's are having problems all over the place. No indications of what shit the bed, but this is more then Cloudflare or Level3.

rbjorklin 9y ago
Do you have a source? (Would like to pass it on internally)
- austinheap 9y ago
  Multiple ISPs I work with have explicitly stated to me that it was Telia. I'm not aware of any public sources confirming this yet.
- VA3FXP 9y ago
  No, I have no public sources I can share. Just relaying what our CDN's have shared with me. I also can't provide anything specific. Sorry, I know it's unreliable and rumor but it's all I can provide.

leesalminen 9y ago

Cloudflare is reporting that the issue has been resolved. https://www.cloudflarestatus.com/incidents/51q3xhq8w7t8 https://www.cloudflarestatus.com/incidents/51q3xhq8w7t8

deleted 9y ago

[deleted]

tedmiston 9y ago

I'm curious what Pingdom uses to detect their own outages.

leke 9y ago

I noticed reddit went down briefly.

revmoo 9y ago

I've had Pingdom for about three months now. They have failed to alert on every outage we've had since I started INCLUDING the first "test outage" I created on day 1 to verify the monitoring was working. I am not impressed at all by their service and will probably be asking for a full refund depending on the outcome of my current trouble ticket with them. Also they've been deleting history of our outages after the fact; i.e. an outage occurs, they fail to alert but they do log it, and then several days later the outage is removed from their history. Pingdom is run by absolute clowns.

NKCSS 9y ago

A BGP problem again? Or something else this time? It amazes me that there has not been an alternative to handle routing except full trust for everyone.

redm 9y ago

We dropped Telia as a carrier about 2 months ago. I've never seen a network with such stability issues; outages/slowdowns occurred on a weekly basis for us and tech support was useless. The only good thing about Telia is they did a great job with service credits. According to VP's there they have been going through major transitions in the last year but given their Tier-1 status, you would expect more stability.