7 ms·

That's great, but as a nasty surprise we found when using it on servers, is that it's rate-limited at 10req/s which is easy to hit even on a development server.

That's great, but as a nasty surprise we found when using it on servers, is that it's rate-limited at 10req/s which is easy to hit even on a development server. Compared to googles 8.8.8.8 and 8.8.4.4 that has a very high 1500 QPS limit [aednichols][0]. I find it weird when they have so much linux tooling for encrypted DNS..

simlevesque 3y ago
> which is easy to hit even on a development server That's actually a good thing.
- tmikaeld 3y ago
  Not a good thing when doing load-testing and DNS resolution rate-limiting silently fails.
  - growse 3y ago
    What are you load testing and hooking up to a public DNS resolver? Would a local caching resolver not be much more suitable?
    - someplaceguy 3y ago
      I'm not the OP, but I'm in the same boat. On some servers, I can use a local caching resolver. However, on others, I'm forced to configure the caching resolver to forward queries to an upstream public DNS server like 1.1.1.1 (via DNS-over-TLS) because of network issues. These can arise from DNS (or perhaps UDP?) rate limiting by the server hosting provider itself, at the network level. In another instance, I noticed that the hosting provider outright drops all fragmented IP packets. I'm concerned this could lead to DNS query failures, especially if the authoritative DNS server is not available over TCP. When contacted, they told me they do this for security reasons and would not be able to disable this filter for me. Like the OP, on these systems I often encounter the 1.1.1.1 rate limit. This is particularly the case since I have DNSSEC verification enabled, so I'm considering switching over to 8.8.8.8.
      - willcipriano 3y ago
        This is why we need rate limits. People will send a billion requests per second rather than fix the problem on their end.
        someplaceguy 3y ago
        What do you mean by "fix the problem on their end"? As I explained, I am not able to fix the problem on my end, if that's what you mean. In fact, this is the exact reason why public DNS servers are so useful: they are much more reliable than the available alternatives, in many cases. Sure, if everything was perfect and all ISP/hosting provider DNS servers were modern and well-configured and had great availability and didn't censor or hijack results, then they wouldn't be necessary. But alas, unfortunately we have to live in the real world. I am not against rate limits, because as you said, sending a billion requests per second (due to misconfiguration, or a bug, or for malicious reasons) is not reasonable. But 10 requests/second is way, way too strict, especially if you use DNSSEC and/or have multiple client machines sharing the same IP address without a common DNS caching resolver in-between (which may not be possible to have for several reasons).
        willcipriano 3y ago
        > which may not be possible to have for several reasons Try me. What's the reason? Money can't be one of them if you expect someone else to scale up and handle your load for free as that would be very rude of you.
        someplaceguy 3y ago
        Sure, here are three off the top of my head that have affected me personally (not to mention the million other possible different scenarios that might exist): 1. In one case, I have multiple machines on the same network but simply speaking, none of them are turned on 24/7 (except for the router), so none of them can be configured to be a common caching resolver. The router is a proprietary Unifi gateway device, for which it is not possible to configure it to use DNS-over-TLS, neither on the local side nor when forwarding to the public DNS server. 2. The other common case for me are my mobile devices (laptops and smartphones). I simply cannot configure them to use a common local caching DNS server because there isn't one, as these devices frequently connect over multiple networks (such as 5G networks, hotel Wi-Fi, etc) and you cannot just use the network-provided DNS server without being vulnerable to man-in-the-middle attacks, censored and hijacked results, etc. 3. Another issue is public Wi-Fi networks. In large hotels, stadiums, etc, there might be hundreds or even thousands of devices behind the public same IP address(es), and as a user yourself, you have no control over them or the network. And obviously, you cannot just use their provided DNS server on these Wi-Fi networks, for multiple reasons: 1) why would you trust the DNS server of a random Wi-Fi network in the first place?, 2) even if you trusted the provider, it is not possible to authenticate these networks, so you could be talking to a man-in-the-middle attacker without realizing, and 3) even if there isn't a man-in-the-middle attacker and the provider is trustworthy, often these DNS servers are extremely poor, as they often censor and hijack results, pollute the DNS cache, don't support DNSSEC queries, etc. And I'm not even mentioning the privacy issues. To be clear, I always use local caching resolvers. But on these systems, I am forced to configure them to forward queries directly to public DNS servers, which means that they don't have a common caching resolver except for the public DNS server itself, hence the rate limit issue when they are all behind the same IP address.
  - simlevesque 3y ago
    It's a good thing that if fails in development and not just in prod. The rate-limiting itself isn't the best scenario but it is to be expected with a free product.
aednichols 3y ago
8.8.8.8 has documented rate limiting [0], but it may be higher than you're hitting. [0] https://developers.google.com/speed/public-dns/docs/isp https://developers.google.com/speed/public-dns/docs/isp
- tmikaeld 3y ago
  Yes, the 1500 QPS is very hard to hit compared to 10 QPS ;-)
pkulak 3y ago
I wonder if that's why I lost DNS yesterday? I had 1.1.1.1 set up as my home network upstream DNS. Lots of caching, but maybe I still hit the limit? Went back to 8.8.8.8 and 8.8.4.4 and all is good again.
- kderbyma 3y ago
  Same....DNS crashed this past week multiple times...I use 1.1.1.1
  - pkulak 3y ago
    I've decided to put it back in, but round-robin with the Google ones. It's so darn fast, I can't quit it quite yet.
    - Terretta 3y ago
      Super fast except for the times it just doesn't. Which is more frustrating than always mildly slower.
kijin 3y ago
I wonder how many of the 1.3T daily requests are made by misconfigured cloud servers with no local DNS cache, hitting popular endpoints like <service>.<region>.amazonaws.com or fcm.googleapis.com over and over again. Consumer devices generally don't have this problem because 1) they respect the TTL, and 2) so do most ISP resolvers. Ubuntu ships systemd-resolved by default these days, which is bare-bones compared to dnsmasq but still helps reduce unnecessary DNS traffic -- not to mention precious round-trip time. You might still hit the rate limit if your app connects to more than a dozen different FQDNs within a second of starting up, but that's not a very common situation.
- hnbear 3y ago
  I have a bunch of IoT stuff that hit my Pi-hole with the same DNS query a lot, like every second in some cases. Definitely plenty of badly designed consumer devices out there doing horrible things with phoning home.
c7DJTLrn 3y ago
Woah, did not know that. Definitely wouldn't want to set it as an upstream DNS server on an office router then.
- theshrike79 3y ago
  Presumably a) the office router will cache the DNS and b) not every piece of software will hit DNS for every single query
- Vosporos 3y ago
  Well no, use a local DNS resolver.
sgi 3y ago
The rate limit isn't static and 10qps is pretty rare. Email me: sergi at cloudflare w details and I can look into why you are seeing that.
- tmikaeld 3y ago
  It's been about 1.5 years since we used it, so this may have changed since then. I was vocal about it on the CF community, but no one seemed to care and no fix was possible so we setup a local DNS.
  - rolph 3y ago
    same vein here, i use local DNS first, and if no strikes then, remote DNS service and vetting the link for inclusion to local services
  - sgi 3y ago
    resolver at cloudflare hits the whole team (and is durable should I disappear) if you see it again in the future.
  - riffic 3y ago
    > so we setup a local DNS that is actually the appropriate (ahem) resolution.
riffic 3y ago
on servers you should definitely have a cache nearby
itsTyrion 3y ago
no caching, even very short-lived, with over 10 QPS?

simlevesque 3y ago

> which is easy to hit even on a development server That's actually a good thing.

tmikaeld 3y ago
Not a good thing when doing load-testing and DNS resolution rate-limiting silently fails.
- growse 3y ago
  What are you load testing and hooking up to a public DNS resolver? Would a local caching resolver not be much more suitable?
  - someplaceguy 3y ago
    I'm not the OP, but I'm in the same boat. On some servers, I can use a local caching resolver. However, on others, I'm forced to configure the caching resolver to forward queries to an upstream public DNS server like 1.1.1.1 (via DNS-over-TLS) because of network issues. These can arise from DNS (or perhaps UDP?) rate limiting by the server hosting provider itself, at the network level. In another instance, I noticed that the hosting provider outright drops all fragmented IP packets. I'm concerned this could lead to DNS query failures, especially if the authoritative DNS server is not available over TCP. When contacted, they told me they do this for security reasons and would not be able to disable this filter for me. Like the OP, on these systems I often encounter the 1.1.1.1 rate limit. This is particularly the case since I have DNSSEC verification enabled, so I'm considering switching over to 8.8.8.8.
    - willcipriano 3y ago
      This is why we need rate limits. People will send a billion requests per second rather than fix the problem on their end.
      - someplaceguy 3y ago
        What do you mean by "fix the problem on their end"? As I explained, I am not able to fix the problem on my end, if that's what you mean. In fact, this is the exact reason why public DNS servers are so useful: they are much more reliable than the available alternatives, in many cases. Sure, if everything was perfect and all ISP/hosting provider DNS servers were modern and well-configured and had great availability and didn't censor or hijack results, then they wouldn't be necessary. But alas, unfortunately we have to live in the real world. I am not against rate limits, because as you said, sending a billion requests per second (due to misconfiguration, or a bug, or for malicious reasons) is not reasonable. But 10 requests/second is way, way too strict, especially if you use DNSSEC and/or have multiple client machines sharing the same IP address without a common DNS caching resolver in-between (which may not be possible to have for several reasons).
        willcipriano 3y ago
        > which may not be possible to have for several reasons Try me. What's the reason? Money can't be one of them if you expect someone else to scale up and handle your load for free as that would be very rude of you.
        someplaceguy 3y ago
        Sure, here are three off the top of my head that have affected me personally (not to mention the million other possible different scenarios that might exist): 1. In one case, I have multiple machines on the same network but simply speaking, none of them are turned on 24/7 (except for the router), so none of them can be configured to be a common caching resolver. The router is a proprietary Unifi gateway device, for which it is not possible to configure it to use DNS-over-TLS, neither on the local side nor when forwarding to the public DNS server. 2. The other common case for me are my mobile devices (laptops and smartphones). I simply cannot configure them to use a common local caching DNS server because there isn't one, as these devices frequently connect over multiple networks (such as 5G networks, hotel Wi-Fi, etc) and you cannot just use the network-provided DNS server without being vulnerable to man-in-the-middle attacks, censored and hijacked results, etc. 3. Another issue is public Wi-Fi networks. In large hotels, stadiums, etc, there might be hundreds or even thousands of devices behind the public same IP address(es), and as a user yourself, you have no control over them or the network. And obviously, you cannot just use their provided DNS server on these Wi-Fi networks, for multiple reasons: 1) why would you trust the DNS server of a random Wi-Fi network in the first place?, 2) even if you trusted the provider, it is not possible to authenticate these networks, so you could be talking to a man-in-the-middle attacker without realizing, and 3) even if there isn't a man-in-the-middle attacker and the provider is trustworthy, often these DNS servers are extremely poor, as they often censor and hijack results, pollute the DNS cache, don't support DNSSEC queries, etc. And I'm not even mentioning the privacy issues. To be clear, I always use local caching resolvers. But on these systems, I am forced to configure them to forward queries directly to public DNS servers, which means that they don't have a common caching resolver except for the public DNS server itself, hence the rate limit issue when they are all behind the same IP address.
- simlevesque 3y ago
  It's a good thing that if fails in development and not just in prod. The rate-limiting itself isn't the best scenario but it is to be expected with a free product.

aednichols 3y ago

8.8.8.8 has documented rate limiting [0], but it may be higher than you're hitting. [0] https://developers.google.com/speed/public-dns/docs/isp https://developers.google.com/speed/public-dns/docs/isp

tmikaeld 3y ago
Yes, the 1500 QPS is very hard to hit compared to 10 QPS ;-)

pkulak 3y ago

I wonder if that's why I lost DNS yesterday? I had 1.1.1.1 set up as my home network upstream DNS. Lots of caching, but maybe I still hit the limit? Went back to 8.8.8.8 and 8.8.4.4 and all is good again.

kderbyma 3y ago
Same....DNS crashed this past week multiple times...I use 1.1.1.1
- pkulak 3y ago
  I've decided to put it back in, but round-robin with the Google ones. It's so darn fast, I can't quit it quite yet.
  - Terretta 3y ago
    Super fast except for the times it just doesn't. Which is more frustrating than always mildly slower.

kijin 3y ago

I wonder how many of the 1.3T daily requests are made by misconfigured cloud servers with no local DNS cache, hitting popular endpoints like <service>.<region>.amazonaws.com or fcm.googleapis.com over and over again. Consumer devices generally don't have this problem because 1) they respect the TTL, and 2) so do most ISP resolvers. Ubuntu ships systemd-resolved by default these days, which is bare-bones compared to dnsmasq but still helps reduce unnecessary DNS traffic -- not to mention precious round-trip time. You might still hit the rate limit if your app connects to more than a dozen different FQDNs within a second of starting up, but that's not a very common situation.

hnbear 3y ago
I have a bunch of IoT stuff that hit my Pi-hole with the same DNS query a lot, like every second in some cases. Definitely plenty of badly designed consumer devices out there doing horrible things with phoning home.

c7DJTLrn 3y ago

Woah, did not know that. Definitely wouldn't want to set it as an upstream DNS server on an office router then.

theshrike79 3y ago
Presumably a) the office router will cache the DNS and b) not every piece of software will hit DNS for every single query
Vosporos 3y ago
Well no, use a local DNS resolver.

sgi 3y ago

The rate limit isn't static and 10qps is pretty rare. Email me: sergi at cloudflare w details and I can look into why you are seeing that.

tmikaeld 3y ago
It's been about 1.5 years since we used it, so this may have changed since then. I was vocal about it on the CF community, but no one seemed to care and no fix was possible so we setup a local DNS.
- rolph 3y ago
  same vein here, i use local DNS first, and if no strikes then, remote DNS service and vetting the link for inclusion to local services
- sgi 3y ago
  resolver at cloudflare hits the whole team (and is durable should I disappear) if you see it again in the future.
- riffic 3y ago
  > so we setup a local DNS that is actually the appropriate (ahem) resolution.

riffic 3y ago

on servers you should definitely have a cache nearby

itsTyrion 3y ago

no caching, even very short-lived, with over 10 QPS?