6 ms·

Network Performance Issues in multiple locations

83 points by ryanlol 10y ago

mef 10y ago
pingdom also reports "something is wrong on the internet" https://status.pingdom.com/ https://status.pingdom.com/
- nannal 10y ago
  Bright yellow on white background?
jgrahamc 10y ago
Telia had problems hitting many service providers. Not a Cloudflare problem per se. Plenty of non-Cloudflare stuff affected like Reddit, AWS, Fastly, ... Check out the dip in requests to Reddit: http://www.redditstatus.com/ http://www.redditstatus.com/
- goshx 10y ago
  Did Reddit stop using Cloudflare?
  - profmonocle 10y ago
    Reddit moved to Fastly at some point.
  - MeltedLux 10y ago
    They switched from Cloudflare to Fastly a few months back.
- goshx 10y ago
  That dip looks normal for that time of the day, btw. Click on the "Week" tab.
  - stoolpigeon 10y ago
    I can get to subreddits - but not reddit.com for the last hour or so.
  - jgrahamc 10y ago
    That's not the dip I'm talking about. Look again.
    - goshx 10y ago
      Ok, I see it. Same time as the error rates went high.
- gtaylor 10y ago
  Yes, we were affected by this: https://status.fastly.com/incidents/3j0pnly3gvqb https://status.fastly.com/incidents/3j0pnly3gvqb Fastly was on top of it and routed around the issue quickly.
iou 10y ago
They said they were scheduling some dashboard maintainence for today. Perhaps this had some unforseen side effect?
- jgrahamc 10y ago
  This was not a Cloudflare-specific problem (see my comment above). Maintenance was yesterday not today and was completed.
ShakataGaNai 10y ago
Anyone have more concrete information on where the routing issues are and/or who's affected?
- Thaxll 10y ago
  Lot of routing in Europe is done through Telia.
  - VA3FXP 10y ago
    It looks like Telia is apparently the issue. Our CDN's are having problems all over the place. No indications of what shit the bed, but this is more then Cloudflare
Sir_Cmpwn 10y ago
Idea: let's proxy half the internet through a private, proprietary service! We can get people to give us valid SSL certificates for their sites, too, and let's fuck up Tor while we're at it. We can totally handle it, right? Oh, and we need to pay for it somehow, so let's go the venture capital approach and just pretend we won't eventually hit a growth cap and ruin our company Twitter-style when we get there. The crazy thing is people actually bought it.
- ryanlol 10y ago
  While I entirely agree with everything you said, it's worth pointing out that Cloudflare isn't really at fault for this particular outage.
  - Sir_Cmpwn 10y ago
    Everyone is to blame here. Equal parts Cloudflare and everyone who uses their service.
- Afforess 10y ago
  I love a good pitchfork and torch session as much as the next rabble-rouser, but let's remember Cloudflare got popular because they _solved*_ a hard problem: how to deal with a DDOS, as a small or medium size website. Cloudflare's essentially a for-profit insurance pool for bandwidth. No individual site has enough bandwidth to handle a DDOS, nor can afford it, but pooled together, many sites can afford a service that can handle individual DDOS attacks. Even if you want to solve the problem without a profit-motive, you still end up with a solution that is going to look very similar to Cloudflare.
  - Sir_Cmpwn 10y ago
    >solved "worked around" is more appropriate, and introduced huge problems with their workaround. The correct solution is to punish ISPs that permit this behavior to continue unchecked. We need offense, not defense. Any ISP that doesn't detect and kill DDoS participants needs to be severely throttled by other ISPs. Organizations like the FCC should be tackling this and levying fines against US-based ISPs for non-compliance and lobbying for foreign policies that punish foreign ISPs.
uwu 10y ago
the title made me think of cloudflare watch (http://www.crimeflare.com/ http://www.crimeflare.com/)
rckrd 10y ago
Why is this on the front page? It seems like whatever happened had nothing to do with cloudflare. Can a mod remove this, as a status page is not news?
- bdcravens 10y ago
  I'm inclined to agree, but if it's being voted up it's "news". It's as much news as "Github Down" when the submitted link times out.
ComputerGuru 10y ago
Obligatory comment that I (and others) make every single time: can we please for <insert diety name>'s sake stop centralizing everything? We are literally throwing away all the benefits of a mostly-decentralized internet for the sake convenience.
- noplay 10y ago
  It's not a cloudflare issue. They report an incident on a major transit provider. This impact a log of site not just cloudflare.
- phailhaus 10y ago
  Do you have a viable alternative to protecting small websites from DDOS attacks?
  - AstralStorm 10y ago
    Freenet style distributed cache and name resolution. But that would require an Internet v3.
  - cremp 10y ago
    You have to realize that DDoS mitigaters are in a position to not stop attacks. They get paid more money when attacks happen; so any company whose sole purpose is mitigation, has a major conflict of interest. A small site can easily be hosted on AWS, which has their own protection which is transparent. Any other cloud provider should offer it transparently anyway. I absolutely hate people who claim Cloudflare is their only solution for mitigation/protection, because it simply isn't true, and Cloudfare does some rather shady stuff.
    - kossae 10y ago
      I feel like saying DDoS mitigators are in a position to not stop attacks is akin to saying car insurance companies are in a position to not stop car accidents. I think the value prop is the quality of the service WHEN the attacks happen, and when they aren't happening it is effectively an insurance-like business. However if I get DDoS'd and my mitigator does nothing, one would think they would eventually be overtaken by a more competent competitor.
DocG 10y ago
More general question, as similar situation has happened multiple times where we are not sure: 1) did we break our client server 2) did our internet provider die 3) did the service die What are recommended ways of finding out fast and reliably in these cases where the fault is.
- pyvpx 10y ago
  uhh, monitoring those things? you can monitor your internet provider a number of ways. something like smokeping allows for an exceedingly simplistic test: ping stuff on the internet. logs will tell you if a client server or individual service has died. that are literally hundreds of solutions for these.
- corobo 10y ago
  Pro-active monitoring rather than reactive diagnosis. Zabbix is an example piece of software, probably overkill for most but I haven't used anything else in the last 5 or so years so don't have any better suggestions.
- jankedeen 10y ago
  External monitoring from AWS or a colo. Simple icmp checks and tcp connects + possible up to app layer checks allowed from these failsafes. Obfuscate as needed.
- zhan_eg 10y ago
  Some of my experience and solution to those issues 1) UptimeRobot [0] - use to monitor various client websites. The free plan checks every 5 minutes, which should be enough. Notifications can be sent to email, slack, sms and many others. If you think there may be a problem only from some locations make a fast check with [1]. If you suspect DNS issues [2] or [3]. 2) Again use UptimeRobot for monitoring device publicly accessible from your network. Moreover, if you are in control of your office network, using pfSense [4] notifications when a network gateway goes down works well (still, that works only if you have 2 or more ISPs). Or use a dedicated monitoring device/service like Zabbix. 3) Using to Twitter to Slack notification, subscribe for updates from both services that you use and major services responsible for Internet backbone. An example is, that using GitLab, comes with multiple time when the service dies (even that they are improving) - seeing the message in Slack that something is WIP currently by all team members (in a dedicated channel), helps to skip unnecessary debugging [5] :) Not affiliate with any of the service. Still - met the UptimeRobot guys some ago - they are a small startup based in Malta, are very cool and have very stable service :) [0] https://uptimerobot.com/ https://uptimerobot.com/ [1] http://www.super-ping.com/ http://www.super-ping.com/ [2] https://www.whatsmydns.net/ https://www.whatsmydns.net/ [3] https://dnschecker.org/ https://dnschecker.org/ [4] https://doc.pfsense.org/index.php/Gateway_Settings#Gateway_Settings https://doc.pfsense.org/index.php/Gateway_Settings#Gateway_S... [5] https://twitter.com/gitlabstatus https://twitter.com/gitlabstatus
infogulch 10y ago
I like to think of Cloudflare like insurance. Any single website may need it rarely if ever, but if it happens to you, you have little to no recourse that doesn't involve large sums of money. Instead, you pay Cloudflare a regular, small amount of money† to reduce the risk of having to pay a large sum of money in case you're targeted. This sounds almost exactly like insurance to me. † Sometimes the marginal cost is actually $0!

83 points by ryanlol 10y ago

mef 10y ago

pingdom also reports "something is wrong on the internet" https://status.pingdom.com/ https://status.pingdom.com/

nannal 10y ago
Bright yellow on white background?

jgrahamc 10y ago

Telia had problems hitting many service providers. Not a Cloudflare problem per se. Plenty of non-Cloudflare stuff affected like Reddit, AWS, Fastly, ... Check out the dip in requests to Reddit: http://www.redditstatus.com/ http://www.redditstatus.com/

goshx 10y ago
Did Reddit stop using Cloudflare?
- profmonocle 10y ago
  Reddit moved to Fastly at some point.
- MeltedLux 10y ago
  They switched from Cloudflare to Fastly a few months back.
goshx 10y ago
That dip looks normal for that time of the day, btw. Click on the "Week" tab.
- stoolpigeon 10y ago
  I can get to subreddits - but not reddit.com for the last hour or so.
- jgrahamc 10y ago
  That's not the dip I'm talking about. Look again.
  - goshx 10y ago
    Ok, I see it. Same time as the error rates went high.
gtaylor 10y ago
Yes, we were affected by this: https://status.fastly.com/incidents/3j0pnly3gvqb https://status.fastly.com/incidents/3j0pnly3gvqb Fastly was on top of it and routed around the issue quickly.

iou 10y ago

They said they were scheduling some dashboard maintainence for today. Perhaps this had some unforseen side effect?

jgrahamc 10y ago
This was not a Cloudflare-specific problem (see my comment above). Maintenance was yesterday not today and was completed.

ShakataGaNai 10y ago

Anyone have more concrete information on where the routing issues are and/or who's affected?

Thaxll 10y ago
Lot of routing in Europe is done through Telia.
- VA3FXP 10y ago
  It looks like Telia is apparently the issue. Our CDN's are having problems all over the place. No indications of what shit the bed, but this is more then Cloudflare

Sir_Cmpwn 10y ago

Idea: let's proxy half the internet through a private, proprietary service! We can get people to give us valid SSL certificates for their sites, too, and let's fuck up Tor while we're at it. We can totally handle it, right? Oh, and we need to pay for it somehow, so let's go the venture capital approach and just pretend we won't eventually hit a growth cap and ruin our company Twitter-style when we get there. The crazy thing is people actually bought it.

ryanlol 10y ago
While I entirely agree with everything you said, it's worth pointing out that Cloudflare isn't really at fault for this particular outage.
- Sir_Cmpwn 10y ago
  Everyone is to blame here. Equal parts Cloudflare and everyone who uses their service.
Afforess 10y ago
I love a good pitchfork and torch session as much as the next rabble-rouser, but let's remember Cloudflare got popular because they _solved*_ a hard problem: how to deal with a DDOS, as a small or medium size website. Cloudflare's essentially a for-profit insurance pool for bandwidth. No individual site has enough bandwidth to handle a DDOS, nor can afford it, but pooled together, many sites can afford a service that can handle individual DDOS attacks. Even if you want to solve the problem without a profit-motive, you still end up with a solution that is going to look very similar to Cloudflare.
- Sir_Cmpwn 10y ago
  >solved "worked around" is more appropriate, and introduced huge problems with their workaround. The correct solution is to punish ISPs that permit this behavior to continue unchecked. We need offense, not defense. Any ISP that doesn't detect and kill DDoS participants needs to be severely throttled by other ISPs. Organizations like the FCC should be tackling this and levying fines against US-based ISPs for non-compliance and lobbying for foreign policies that punish foreign ISPs.

uwu 10y ago

the title made me think of cloudflare watch (http://www.crimeflare.com/ http://www.crimeflare.com/)

rckrd 10y ago

Why is this on the front page? It seems like whatever happened had nothing to do with cloudflare. Can a mod remove this, as a status page is not news?

bdcravens 10y ago
I'm inclined to agree, but if it's being voted up it's "news". It's as much news as "Github Down" when the submitted link times out.

ComputerGuru 10y ago

Obligatory comment that I (and others) make every single time: can we please for <insert diety name>'s sake stop centralizing everything? We are literally throwing away all the benefits of a mostly-decentralized internet for the sake convenience.

noplay 10y ago
It's not a cloudflare issue. They report an incident on a major transit provider. This impact a log of site not just cloudflare.
phailhaus 10y ago
Do you have a viable alternative to protecting small websites from DDOS attacks?
- AstralStorm 10y ago
  Freenet style distributed cache and name resolution. But that would require an Internet v3.
- cremp 10y ago
  You have to realize that DDoS mitigaters are in a position to not stop attacks. They get paid more money when attacks happen; so any company whose sole purpose is mitigation, has a major conflict of interest. A small site can easily be hosted on AWS, which has their own protection which is transparent. Any other cloud provider should offer it transparently anyway. I absolutely hate people who claim Cloudflare is their only solution for mitigation/protection, because it simply isn't true, and Cloudfare does some rather shady stuff.
  - kossae 10y ago
    I feel like saying DDoS mitigators are in a position to not stop attacks is akin to saying car insurance companies are in a position to not stop car accidents. I think the value prop is the quality of the service WHEN the attacks happen, and when they aren't happening it is effectively an insurance-like business. However if I get DDoS'd and my mitigator does nothing, one would think they would eventually be overtaken by a more competent competitor.

DocG 10y ago

More general question, as similar situation has happened multiple times where we are not sure: 1) did we break our client server 2) did our internet provider die 3) did the service die What are recommended ways of finding out fast and reliably in these cases where the fault is.

pyvpx 10y ago
uhh, monitoring those things? you can monitor your internet provider a number of ways. something like smokeping allows for an exceedingly simplistic test: ping stuff on the internet. logs will tell you if a client server or individual service has died. that are literally hundreds of solutions for these.
corobo 10y ago
Pro-active monitoring rather than reactive diagnosis. Zabbix is an example piece of software, probably overkill for most but I haven't used anything else in the last 5 or so years so don't have any better suggestions.
jankedeen 10y ago
External monitoring from AWS or a colo. Simple icmp checks and tcp connects + possible up to app layer checks allowed from these failsafes. Obfuscate as needed.
zhan_eg 10y ago
Some of my experience and solution to those issues 1) UptimeRobot [0] - use to monitor various client websites. The free plan checks every 5 minutes, which should be enough. Notifications can be sent to email, slack, sms and many others. If you think there may be a problem only from some locations make a fast check with [1]. If you suspect DNS issues [2] or [3]. 2) Again use UptimeRobot for monitoring device publicly accessible from your network. Moreover, if you are in control of your office network, using pfSense [4] notifications when a network gateway goes down works well (still, that works only if you have 2 or more ISPs). Or use a dedicated monitoring device/service like Zabbix. 3) Using to Twitter to Slack notification, subscribe for updates from both services that you use and major services responsible for Internet backbone. An example is, that using GitLab, comes with multiple time when the service dies (even that they are improving) - seeing the message in Slack that something is WIP currently by all team members (in a dedicated channel), helps to skip unnecessary debugging [5] :) Not affiliate with any of the service. Still - met the UptimeRobot guys some ago - they are a small startup based in Malta, are very cool and have very stable service :) [0] https://uptimerobot.com/ https://uptimerobot.com/ [1] http://www.super-ping.com/ http://www.super-ping.com/ [2] https://www.whatsmydns.net/ https://www.whatsmydns.net/ [3] https://dnschecker.org/ https://dnschecker.org/ [4] https://doc.pfsense.org/index.php/Gateway_Settings#Gateway_Settings https://doc.pfsense.org/index.php/Gateway_Settings#Gateway_S... [5] https://twitter.com/gitlabstatus https://twitter.com/gitlabstatus

infogulch 10y ago

I like to think of Cloudflare like insurance. Any single website may need it rarely if ever, but if it happens to you, you have little to no recourse that doesn't involve large sums of money. Instead, you pay Cloudflare a regular, small amount of money† to reduce the risk of having to pay a large sum of money in case you're targeted. This sounds almost exactly like insurance to me. † Sometimes the marginal cost is actually $0!