7 ms·

This is a great write up. It's also why the DNS root servers have a policy of surviving DDoS through massively over-provisioned, multi-org, anycasted redundancy

This is a great write up. It's also why the DNS root servers have a policy of surviving DDoS through massively over-provisioned, multi-org, anycasted redundancy rather this sort of smart DDoS mitigation that drops traffic: DNS is so critical that any risk of dropping real traffic is unacceptable. (obviously, such a scale is impractical for 99% of services) A good takeaway from this outage for the average user would be to make sure that your fallback DNS resolvers are operated by totally separate providers. (eg, configure 1.1.1.1 with 8.8.8.8 as a fallback, rather than 1.1.1.1 and 1.0.0.1) (Edit: fixed cloudflare's secondary address)

Dylan16807 8y ago
(the cloudflare secondary is 1.0.0.1)
- cmurf 8y ago
  Cloudfare makes it weirdly difficult to find this. 1.1.1.1 is plastered over many pages but not concomitant with the secondary.
  - bobf 8y ago
    That's not necessarily a bad thing, since it's probably better to use another source for a secondary if that's your primary.
    - tracker1 8y ago
      I'm using cloudflare first, and google as secondary in my router. For a while I was running my own, but it became less necessary when I stopped doing as much dev at home.
  - chdefrene 8y ago
    That is not my experience at all. By following the `install`-instructions on http://1.1.1.1 http://1.1.1.1, all available DNS addresses are listed for both ipv4 and ipv6.
    - Mononokay 8y ago
      FYI, https://1.1.1.1 https://1.1.1.1 works over SSL, so there's no need to format it http://1.1.1.1 http://1.1.1.1.
berti 8y ago
Or 9.9.9.9 if you're not as comfortable with Google services.
- spondyl 8y ago
  For anyone wondering, 9.9.9.9 is https://www.quad9.net/ https://www.quad9.net/ who claim to not only resolve requests but also check them against IBM X-Force's threat intelligence database.
  - nerdponx 8y ago
    For the curious, they also have an extremely detailed privacy policy: https://www.quad9.net/policy/ https://www.quad9.net/policy/ Seems like a fair deal to me. I get free DNS service, and companies sponsoring this program get metrics on threats and general Internet usage. I'm a little skeptical of their claims that individuals can't be identified from their anonymized data. E.g. I probably only get one or two hits on my personal website every week, so it might not be hard for a malicious employee to deanonymize visitors to my site. Some highlights from the policy: Many nations classify IP addresses as Personally-Identifiable Information (PII), and we take a conservative approach in treating IP addresses as PII in all jurisdictions in which our systems reside. Our normal course of data management does not have any IP address information or other PII logged to disk or transmitted out of the location in which the query was received. We may aggregate certain counters to larger network block levels for statistical collection purposes, but those counters do not maintain specific IP address data nor is the format or model of data stored capable of being reverse-engineered to ascertain what specific IP addresses made what queries. There are exceptions to this storage model: In the event of events or observed behaviors which we deem malicious or anomalous, we may utilize more detailed logging to collect more specific IP address data in the process of normal network defense and mitigation. This collection and transmission off-site will be limited to IP addresses that we determine are involved in the event. ... We do not correlate or combine information from our logs with any personal information that you have provided Quad9 for other services, or with your specific IP address. ... Quad9 DNS Services generate and share high level anonymized aggregate statistics including threat metrics on threat type, geolocation, and if available, sector, as well as other vertical metrics including performance metrics on the Quad9 DNS Services (i.e. number of threats blocked, infrastructure uptime) when available with the Quad9 threat intelligence (TI) partners, academic researchers, or the public. Quad9 DNS Services share anonymized data on specific domains queried (records such as domain, timestamp, geolocation, number of hits, first seen, last seen) with its threat intelligence partners. Quad9 DNS Services also builds, stores, and may share certain DNS data streams which store high level information about domain resolved, query types, result codes, and timestamp. These streams do not contain IP address information of requestor and cannot be correlated to IP address or other PII. ... Quad9 does not track visitors over time and across third-party websites, and therefore does not respond to Do Not Track signaling.
h1d 8y ago
Not if you're not comfortable giving up so many of your internet activity to Google but as others have pointed out there are alternatives to spread.
bluedino 8y ago
I’m amazed at how many people abandoned other providers and blindly switched to 1.1.1.1 I can’t even use that address with the ISP Alestra in Mexico
- toomuchtodo 8y ago
  The lack of operational knowledge is real, and there is no easy fix other than resilient defaults by a benevolent dictator (in this case, resolvers across netblock and provider demarcations) or spending the time to educate yourself first on the “why” then the “how”.
crtasm 8y ago
I read on the Pi-Hole forums that 'fallback' is a misleading term because clients don't work that way - they will happily spread requests between two functioning DNS servers. Can anyone confirm this or provide further insight?
- mjcl 8y ago
  Different OSes handle it differently. Windows tries the primary, waits 1 second and then starts trying secondaries.
  - dharmab 8y ago
    glibc tries in order. musl (Alpine Linux) tries all in parallel and returns thebl first resolution (If any).
- TravelTechGuy 8y ago
  Can confirm. If I have a secondary DNS specified in my router, other than my pi-hole, it becomes useless. My guess is the router is either measuring response time, and goes with the most efficient, or otherwise round-robining the requests. Either way, requests bypass the pi-hole in such quantities that it became useless. PS: someone here mentioned that this behavior is OS-dependent - nope, this happens on the router level, and all devices in my apartment suffer.
  - detaro 8y ago
    Then your devices are likely using your router as a DNS resolver, which in turn talks to your pihole and the external one. And thus it depends on your router's OS what it does.
  - jschwartzi 8y ago
    It depends on how your router's DHCP server is configured. If you configure your router to pass its own IP address out as the DNS server for the local subnet then the router's behavior dictates how DNS works. If your router is passing out an external DNS in the DHCP configuration, then you'll get OS-dependent behavior. My router uses a DNS resolver internally, and it will spread-cast to multiple DNS servers and use the quickest response it can get. It also caches using the TTL in the DNS response, and so it will serve up cached records transparently.

Dylan16807 8y ago

(the cloudflare secondary is 1.0.0.1)

cmurf 8y ago
Cloudfare makes it weirdly difficult to find this. 1.1.1.1 is plastered over many pages but not concomitant with the secondary.
- bobf 8y ago
  That's not necessarily a bad thing, since it's probably better to use another source for a secondary if that's your primary.
  - tracker1 8y ago
    I'm using cloudflare first, and google as secondary in my router. For a while I was running my own, but it became less necessary when I stopped doing as much dev at home.
- chdefrene 8y ago
  That is not my experience at all. By following the `install`-instructions on http://1.1.1.1 http://1.1.1.1, all available DNS addresses are listed for both ipv4 and ipv6.
  - Mononokay 8y ago
    FYI, https://1.1.1.1 https://1.1.1.1 works over SSL, so there's no need to format it http://1.1.1.1 http://1.1.1.1.

berti 8y ago

Or 9.9.9.9 if you're not as comfortable with Google services.

spondyl 8y ago
For anyone wondering, 9.9.9.9 is https://www.quad9.net/ https://www.quad9.net/ who claim to not only resolve requests but also check them against IBM X-Force's threat intelligence database.
- nerdponx 8y ago
  For the curious, they also have an extremely detailed privacy policy: https://www.quad9.net/policy/ https://www.quad9.net/policy/ Seems like a fair deal to me. I get free DNS service, and companies sponsoring this program get metrics on threats and general Internet usage. I'm a little skeptical of their claims that individuals can't be identified from their anonymized data. E.g. I probably only get one or two hits on my personal website every week, so it might not be hard for a malicious employee to deanonymize visitors to my site. Some highlights from the policy: Many nations classify IP addresses as Personally-Identifiable Information (PII), and we take a conservative approach in treating IP addresses as PII in all jurisdictions in which our systems reside. Our normal course of data management does not have any IP address information or other PII logged to disk or transmitted out of the location in which the query was received. We may aggregate certain counters to larger network block levels for statistical collection purposes, but those counters do not maintain specific IP address data nor is the format or model of data stored capable of being reverse-engineered to ascertain what specific IP addresses made what queries. There are exceptions to this storage model: In the event of events or observed behaviors which we deem malicious or anomalous, we may utilize more detailed logging to collect more specific IP address data in the process of normal network defense and mitigation. This collection and transmission off-site will be limited to IP addresses that we determine are involved in the event. ... We do not correlate or combine information from our logs with any personal information that you have provided Quad9 for other services, or with your specific IP address. ... Quad9 DNS Services generate and share high level anonymized aggregate statistics including threat metrics on threat type, geolocation, and if available, sector, as well as other vertical metrics including performance metrics on the Quad9 DNS Services (i.e. number of threats blocked, infrastructure uptime) when available with the Quad9 threat intelligence (TI) partners, academic researchers, or the public. Quad9 DNS Services share anonymized data on specific domains queried (records such as domain, timestamp, geolocation, number of hits, first seen, last seen) with its threat intelligence partners. Quad9 DNS Services also builds, stores, and may share certain DNS data streams which store high level information about domain resolved, query types, result codes, and timestamp. These streams do not contain IP address information of requestor and cannot be correlated to IP address or other PII. ... Quad9 does not track visitors over time and across third-party websites, and therefore does not respond to Do Not Track signaling.

h1d 8y ago

Not if you're not comfortable giving up so many of your internet activity to Google but as others have pointed out there are alternatives to spread.

bluedino 8y ago

I’m amazed at how many people abandoned other providers and blindly switched to 1.1.1.1 I can’t even use that address with the ISP Alestra in Mexico

toomuchtodo 8y ago
The lack of operational knowledge is real, and there is no easy fix other than resilient defaults by a benevolent dictator (in this case, resolvers across netblock and provider demarcations) or spending the time to educate yourself first on the “why” then the “how”.

crtasm 8y ago

I read on the Pi-Hole forums that 'fallback' is a misleading term because clients don't work that way - they will happily spread requests between two functioning DNS servers. Can anyone confirm this or provide further insight?

mjcl 8y ago
Different OSes handle it differently. Windows tries the primary, waits 1 second and then starts trying secondaries.
- dharmab 8y ago
  glibc tries in order. musl (Alpine Linux) tries all in parallel and returns thebl first resolution (If any).
TravelTechGuy 8y ago
Can confirm. If I have a secondary DNS specified in my router, other than my pi-hole, it becomes useless. My guess is the router is either measuring response time, and goes with the most efficient, or otherwise round-robining the requests. Either way, requests bypass the pi-hole in such quantities that it became useless. PS: someone here mentioned that this behavior is OS-dependent - nope, this happens on the router level, and all devices in my apartment suffer.
- detaro 8y ago
  Then your devices are likely using your router as a DNS resolver, which in turn talks to your pihole and the external one. And thus it depends on your router's OS what it does.
- jschwartzi 8y ago
  It depends on how your router's DHCP server is configured. If you configure your router to pass its own IP address out as the DNS server for the local subnet then the router's behavior dictates how DNS works. If your router is passing out an external DNS in the DHCP configuration, then you'll get OS-dependent behavior. My router uses a DNS resolver internally, and it will spread-cast to multiple DNS servers and use the quickest response it can get. It also caches using the TTL in the DNS response, and so it will serve up cached records transparently.