5 ms·

Cloudflare DNS problem

60 points by sinak 11y ago

sinak 11y ago
More details here on Cloudflare's status page here: http://www.cloudflarestatus.com/incidents/m8hf8q8vs81d http://www.cloudflarestatus.com/incidents/m8hf8q8vs81d
packetized 11y ago
Dyn also appears to be having a real, real bad day. Glue records vanished like a fart in the wind. $ dig @8.8.8.8 ns2.p29.dynect.net ; <<>> DiG 9.7.3-P3-RedHat-9.7.3-8.P3.el6 <<>> @8.8.8.8 ns2.p29.dynect.net ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 2690 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0 ;; QUESTION SECTION: ;ns2.p29.dynect.net. IN A ;; AUTHORITY SECTION: dynect.net. 600 IN SOA ns0.dynamicnetworkservices.net. hostmaster.dyndns.com. 2015070600 10800 1800 604800 1800 ;; Query time: 2 msec ;; SERVER: 8.8.8.8#53(8.8.8.8) ;; WHEN: Mon Jul 6 21:42:48 2015 ;; MSG SIZE rcvd: 120
- mattbeckman 11y ago
  Does CloudFlare use Dyn or are these unrelated issues?
  - rdl 11y ago
    The affected zones are those cnamed from CloudFlare to Dyn.
    - sajal83 11y ago
      Any example hostname thats configured on Dyn and is not resolving?
      - deleted 11y ago
        [deleted]
kelseydh 11y ago
I love how even https://support.cloudflare.com/ https://support.cloudflare.com/ is down by this.
- jonny_eh 11y ago
  Makes me feel good as an impacted user. I know that the problem won't be ignored!
- colinbartlett 11y ago
  They do at least use a different DNS provider for their status page: http://www.cloudflarestatus.com http://www.cloudflarestatus.com
  - Someone1234 11y ago
    I do love how everything is operational... I see the big blurb about DNS at the top. But it still feels like the table is misleading. When they're having a major DNS outage and every site is "operational" it feels misleading. Although if this was Amazon's AWS console we wouldn't even have the blurb at the top...
    - colinbartlett 11y ago
      It's frustrating for me, too. Because my side project https://statusgator.io https://statusgator.io pulls in this status data and sends alerts to subscribers when services go down. But since CloudFlare has not indicated any downtime, everything still appears up.
- xxdesmus 11y ago
  ZenDesk uses Dyn. Hence, it was down. Sorry about that.
rdl 11y ago
Sorry -- our ops and engineering teams are looking into this and will update/resolve as quickly as possible. https://www.cloudflarestatus.com/incidents/m8hf8q8vs81d https://www.cloudflarestatus.com/incidents/m8hf8q8vs81d
- iaw 11y ago
  Good luck, this is really rough. I encountered a 101 on a random site before I saw this and was a bit baffled.
kelseydh 11y ago
Lately my number one bloody cause of downtime has been DNS lookup failures. I switched to Cloudflare hoping to avoid this. If I can't trust Cloudflare, then who the heck can I trust? Also, is there any way to actually resolve this issue quickly while Cloudflare deals with their fuckup? I don't like being at the mercy of their support team while my site has literal downtime.
- rdl 11y ago
  We should have it fixed very shortly; 99.9% on the cause.
- colinbartlett 11y ago
  I use DNSimple and they have a feature that lets you sync to another provider. I have not used it but, as I understand it, the feature helps you setup slave servers which respond to Notify messages from DNSimple. That way, if they go down, your slave provider will keep things up.
- laumars 11y ago
  You could always run your own DNS server. Of course, you'd then lose the other services Cloudflare offer like basic DDoS protection, but that's the trade off you'd have to consider.
- elmin 11y ago
  I've never had any trouble with Amazon Route53.
  - zrail 11y ago
    That's because Route53 is basically bullet proof, such that their SLA guarantees 100% availability. I have switched everything I touch to Route53 and have had zero issues.
- rdl 11y ago
  In this case you would have been unaffected if you used CloudFlare for DNS, rather than using Dyn for DNS and CNAME from CloudFlare to it. Using CloudFlare for DNS is our preferred configuration.
kelseydh 11y ago
If you're on Heroku, one immediate way around this problem is to guide your users towards your heroku domain. I've been doing this for clients contacting me who absolutely need access right this moment. E.g. https://myappname.herokuapp.com https://myappname.herokuapp.com
- Someone1234 11y ago
  That's a nice tip. Can I ask a question while we wait for Cloudflare to fix this... Heroku looks nice. But also quite expensive after you add on dyno, Postgre/Redis, and "addons." What about Heroku do you like/appreciate that justifies the slightly higher costs? Or do you just like that all of the devops-style stuff is handled by them (as opposed to AWS/Azure/Linode, etc where you'd have to do it yourself). I am genuinely asking, I have no reason to dislike Heroku, and I know it is quite popular. Just trying to understand what the "value add" is over other cloud services (or why they're dissimilar). Please don't feel obliged to answer, but if you have the time it would be appreciated/interesting.
  - kelseydh 11y ago
    No problem, here's my two (relatively unorganized) two cents: With Heroku (frankly also thanks to Ruby on Rails) I can be a sys-admin, developer, database administrator and QA all at once; far easier than before. For the price of $100k I have seen developers build successful products that used to cost $1M+ to develop. On Heroku you definitely pay for convenience and ease of use, knowing that you get a benefit in doing so by being able to focus on the development of your app, the part that matters. The costs grow if you get traffic, but through careful management (such as packing multiple processes into one dyno), you can reduce costs. Even if many find Heroku's prices high, they are still much lower than what you see in enterprise land, or even on AWS if you aren't careful. Heroku's documentation is also incredibly good, and this helps you a lot to move fast. Almost by accident, simply by following Heroku documentation I have found that developers end up following good cloud server practices almost by default (i.e. because Heroku forces you to build your app to follow certain conventions, conventions needed for making your app horizontally scaleable; such as delegating clock processes to a separate process, and not allowing dynos to manipulate or rely on unique file memory specific only to them). If I architectured or configured my own servers it would have taken 10x as long, exposed me to far more security issues, and I would have probably screwed it up any way. The lesson from this is that I think as developers we need far less freedom to configure than we think we need. So in conclusion, if you're a small to medium-sized app that does not have complex server or bandwidth needs (i.e. you are not Imgur or Pied Piper), then I highly recommend it. I'm sure many Oracle salesman will be unhappy with this opinion, but take my word for it that the vast majority of applications do not need complex, overengineered server systems to be successful. (After all as a startup, you should not be wasting months planning on how you will scale to support 10 million concurrent users. I've seen that hubris distract failing startups before.)
jgrahamc 11y ago
This appears to not be a problem with CloudFlare, but a problem with Dyn: https://www.dynstatus.com/ https://www.dynstatus.com/. Of course, Dyn and CloudFlare have mutual customers and so if a Dyn has a problem it's possible for a mutual customer to have a problem.
- hosay123 11y ago
  Looks like your support site might need a little.. support http://i.imgur.com/X76QdaC.png http://i.imgur.com/X76QdaC.png You might want to try putting it behind a CDN to cope with the load
  - jgrahamc 11y ago
    We use Zendesk. Zendesk uses Dyn. Boom!
- stockliasteroid 11y ago
  Strange, we are using CNAME apex resolution with cloudflare to point to Heroku apps, so I'm not aware that Dyn is used anywhere in that chain, but resolution is still down for those apps. edit: resolved now.
  - jgrahamc 11y ago
    $ dig +short NS heroku.com ns1.p19.dynect.net. ns4.p19.dynect.net. ns3.p19.dynect.net. ns2.p19.dynect.net.
    - stockliasteroid 11y ago
      Thanks!
rdl 11y ago
Sorry about this: just posted a blog post. Appears to be a Dyn issue, but affects a lot of our customers (including support.cloudflare.com which is Zendesk) https://blog.cloudflare.com/the-internet-is-a-cooperative-system-dns-outage-of-6-july-2015/ https://blog.cloudflare.com/the-internet-is-a-cooperative-sy...
deleted 11y ago
[deleted]

60 points by sinak 11y ago

sinak 11y ago

More details here on Cloudflare's status page here: http://www.cloudflarestatus.com/incidents/m8hf8q8vs81d http://www.cloudflarestatus.com/incidents/m8hf8q8vs81d

packetized 11y ago

Dyn also appears to be having a real, real bad day. Glue records vanished like a fart in the wind. $ dig @8.8.8.8 ns2.p29.dynect.net ; <<>> DiG 9.7.3-P3-RedHat-9.7.3-8.P3.el6 <<>> @8.8.8.8 ns2.p29.dynect.net ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 2690 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0 ;; QUESTION SECTION: ;ns2.p29.dynect.net. IN A ;; AUTHORITY SECTION: dynect.net. 600 IN SOA ns0.dynamicnetworkservices.net. hostmaster.dyndns.com. 2015070600 10800 1800 604800 1800 ;; Query time: 2 msec ;; SERVER: 8.8.8.8#53(8.8.8.8) ;; WHEN: Mon Jul 6 21:42:48 2015 ;; MSG SIZE rcvd: 120

mattbeckman 11y ago
Does CloudFlare use Dyn or are these unrelated issues?
- rdl 11y ago
  The affected zones are those cnamed from CloudFlare to Dyn.
  - sajal83 11y ago
    Any example hostname thats configured on Dyn and is not resolving?
    - deleted 11y ago
      [deleted]

kelseydh 11y ago

I love how even https://support.cloudflare.com/ https://support.cloudflare.com/ is down by this.

jonny_eh 11y ago
Makes me feel good as an impacted user. I know that the problem won't be ignored!
colinbartlett 11y ago
They do at least use a different DNS provider for their status page: http://www.cloudflarestatus.com http://www.cloudflarestatus.com
- Someone1234 11y ago
  I do love how everything is operational... I see the big blurb about DNS at the top. But it still feels like the table is misleading. When they're having a major DNS outage and every site is "operational" it feels misleading. Although if this was Amazon's AWS console we wouldn't even have the blurb at the top...
  - colinbartlett 11y ago
    It's frustrating for me, too. Because my side project https://statusgator.io https://statusgator.io pulls in this status data and sends alerts to subscribers when services go down. But since CloudFlare has not indicated any downtime, everything still appears up.
xxdesmus 11y ago
ZenDesk uses Dyn. Hence, it was down. Sorry about that.

rdl 11y ago

Sorry -- our ops and engineering teams are looking into this and will update/resolve as quickly as possible. https://www.cloudflarestatus.com/incidents/m8hf8q8vs81d https://www.cloudflarestatus.com/incidents/m8hf8q8vs81d

iaw 11y ago
Good luck, this is really rough. I encountered a 101 on a random site before I saw this and was a bit baffled.

kelseydh 11y ago

Lately my number one bloody cause of downtime has been DNS lookup failures. I switched to Cloudflare hoping to avoid this. If I can't trust Cloudflare, then who the heck can I trust? Also, is there any way to actually resolve this issue quickly while Cloudflare deals with their fuckup? I don't like being at the mercy of their support team while my site has literal downtime.

rdl 11y ago
We should have it fixed very shortly; 99.9% on the cause.
colinbartlett 11y ago
I use DNSimple and they have a feature that lets you sync to another provider. I have not used it but, as I understand it, the feature helps you setup slave servers which respond to Notify messages from DNSimple. That way, if they go down, your slave provider will keep things up.
laumars 11y ago
You could always run your own DNS server. Of course, you'd then lose the other services Cloudflare offer like basic DDoS protection, but that's the trade off you'd have to consider.
elmin 11y ago
I've never had any trouble with Amazon Route53.
- zrail 11y ago
  That's because Route53 is basically bullet proof, such that their SLA guarantees 100% availability. I have switched everything I touch to Route53 and have had zero issues.
rdl 11y ago
In this case you would have been unaffected if you used CloudFlare for DNS, rather than using Dyn for DNS and CNAME from CloudFlare to it. Using CloudFlare for DNS is our preferred configuration.

kelseydh 11y ago

If you're on Heroku, one immediate way around this problem is to guide your users towards your heroku domain. I've been doing this for clients contacting me who absolutely need access right this moment. E.g. https://myappname.herokuapp.com https://myappname.herokuapp.com

Someone1234 11y ago
That's a nice tip. Can I ask a question while we wait for Cloudflare to fix this... Heroku looks nice. But also quite expensive after you add on dyno, Postgre/Redis, and "addons." What about Heroku do you like/appreciate that justifies the slightly higher costs? Or do you just like that all of the devops-style stuff is handled by them (as opposed to AWS/Azure/Linode, etc where you'd have to do it yourself). I am genuinely asking, I have no reason to dislike Heroku, and I know it is quite popular. Just trying to understand what the "value add" is over other cloud services (or why they're dissimilar). Please don't feel obliged to answer, but if you have the time it would be appreciated/interesting.
- kelseydh 11y ago
  No problem, here's my two (relatively unorganized) two cents: With Heroku (frankly also thanks to Ruby on Rails) I can be a sys-admin, developer, database administrator and QA all at once; far easier than before. For the price of $100k I have seen developers build successful products that used to cost $1M+ to develop. On Heroku you definitely pay for convenience and ease of use, knowing that you get a benefit in doing so by being able to focus on the development of your app, the part that matters. The costs grow if you get traffic, but through careful management (such as packing multiple processes into one dyno), you can reduce costs. Even if many find Heroku's prices high, they are still much lower than what you see in enterprise land, or even on AWS if you aren't careful. Heroku's documentation is also incredibly good, and this helps you a lot to move fast. Almost by accident, simply by following Heroku documentation I have found that developers end up following good cloud server practices almost by default (i.e. because Heroku forces you to build your app to follow certain conventions, conventions needed for making your app horizontally scaleable; such as delegating clock processes to a separate process, and not allowing dynos to manipulate or rely on unique file memory specific only to them). If I architectured or configured my own servers it would have taken 10x as long, exposed me to far more security issues, and I would have probably screwed it up any way. The lesson from this is that I think as developers we need far less freedom to configure than we think we need. So in conclusion, if you're a small to medium-sized app that does not have complex server or bandwidth needs (i.e. you are not Imgur or Pied Piper), then I highly recommend it. I'm sure many Oracle salesman will be unhappy with this opinion, but take my word for it that the vast majority of applications do not need complex, overengineered server systems to be successful. (After all as a startup, you should not be wasting months planning on how you will scale to support 10 million concurrent users. I've seen that hubris distract failing startups before.)

jgrahamc 11y ago

This appears to not be a problem with CloudFlare, but a problem with Dyn: https://www.dynstatus.com/ https://www.dynstatus.com/. Of course, Dyn and CloudFlare have mutual customers and so if a Dyn has a problem it's possible for a mutual customer to have a problem.

hosay123 11y ago
Looks like your support site might need a little.. support http://i.imgur.com/X76QdaC.png http://i.imgur.com/X76QdaC.png You might want to try putting it behind a CDN to cope with the load
- jgrahamc 11y ago
  We use Zendesk. Zendesk uses Dyn. Boom!
stockliasteroid 11y ago
Strange, we are using CNAME apex resolution with cloudflare to point to Heroku apps, so I'm not aware that Dyn is used anywhere in that chain, but resolution is still down for those apps. edit: resolved now.
- jgrahamc 11y ago
  $ dig +short NS heroku.com ns1.p19.dynect.net. ns4.p19.dynect.net. ns3.p19.dynect.net. ns2.p19.dynect.net.
  - stockliasteroid 11y ago
    Thanks!

rdl 11y ago

Sorry about this: just posted a blog post. Appears to be a Dyn issue, but affects a lot of our customers (including support.cloudflare.com which is Zendesk) https://blog.cloudflare.com/the-internet-is-a-cooperative-system-dns-outage-of-6-july-2015/ https://blog.cloudflare.com/the-internet-is-a-cooperative-sy...

deleted 11y ago

[deleted]