21 ms·
I’m responsible for a bunch of IoT hardware, and every firmware spec I write includes a note on not using the DNS servers provided via DHCP. While sure there ar
by jon-wood 6y ago
I’m responsible for a bunch of IoT hardware, and every firmware spec I write includes a note on not using the DNS servers provided via DHCP. While sure there are companies explicitly doing this to avoid filtering, at least in my case it’s because a significant proportion of DHCP servers are configured to send DNS to your ISP, and ISP provided DNS is almost universally terrible. They’ll ignore TTLs, rewrite NXDOMAIN responses into the IP address of their ad-laced web search, and occasionally just highjack every single query to send you to a page saying you’re approaching your bandwidth quota. In the face of that sort of behaviour you inevitably end up with technical support having to field angry customers who just don’t get that it’s not our fault the hardware isn’t working, and the response to that is hardcoding a set of known good DNS servers which we can rely on.
Please, if you have a Pihole, redirect all DNS through it as described in the article. Just be aware DNS over HTTPS is a thing now, and while the devices I’m responsible for aren’t going to try and evade your redirects, the companies that are trying to make sure ads get delivered will absolutely switch to DoH which will be much more difficult to work around.
- foxtrottbravo 6y agoWow, thanks for breaking established standards then...
- Avamander 6y agoGo beat your ISP first.
- intricatedetail 6y agoThere is no justification for bypassing users network settings, and especially doing it for your own convenience. At very least it should be opt in. If you want to violate customer privacy to save money on support, then I don't know why this is worth even discussing.
- rightbyte 6y agoWhat DNS do you hardcode? Google's? Or do you advice the use to set it up himself? I am very suspicious of the push for https and the like. I feel it is mainly about hiding the payload from me not any third party.
- EvanAnderson 6y agoYou're right to be suspicious. The DNS-over-HTTPS model favors those who run the servers (because they get exclusive access to monetizable end user name resolution data) and those who control the resolvers. You might control the resolver on your personal computer (for now). You probably don't control it on your phone. You most likely won't control it on your embedded devices.
- t0astbread 6y agoThe root evil here is that you can't change the root certificates in such devices. Even if you controlled its DNS, the device could still just be programmed to fail if it doesn't reach its analytics/ad/whatever server.
- GekkePrutser 6y agoThe Amazon Fire TV does this already :( If it doesn't see internet it just blocks itself and goes to a screen "Oops I have no internet". So you can forget about watching movies from your local server using the VLC app as well. Ridiculous.
- moonbug 6y agomakes it utterly useless in an IPv6+NAT64 environment, incidentally.
- tguvot 6y agoYou can go to settings > installed application and lunch VLC or anything else directly from there
- GekkePrutser 6y agoI'm pretty sure the last time my internet was down I couldn't get into anything but the wifi settings. But I'll try it next time, thanks for the tip!
- MayeulC 6y agoHow do you deal with NAT64? It's a real thing with some consumer ISPs. I can see myself using it as well, maybe even just to see which devices don't use DNS. Here's another idea: generate a unique IPv6 address per DNS request, route them to the correct destination, filter other IPs. Not really scalable, but usable on small networks/VLANs.
- jon-wood 6y agoCurrently, we don’t. The devices don’t support IPv6, so it’s not a problem we’ve had to deal with so far.
- moonbug 6y agoGreat work there.
- dingaling 6y agoSo you'll only start working on the problem when IPv6-aware devices start failing? Why wait?
- zinekeller 6y agoNot the OP, but you should also be reminded that many of these limitations are brought out by cost-cutting measures that we didn't have control of and could be used to fire us immediately, and even when IPv6 is implemented most IoT backend servers only operate over IPv4 anyway. Another one that you might be angry about is only including 2.4GHz-band WiFi even though 5GHz WiFi is already ubiquitous (although only including 5GHz WiFi is much more questionable ;).
- jon-wood 6y agoExactly this, we’re a small team and have to pick our battles. In practice very few people have IPv6 only connections, and even fewer of the people who buy our products.
- jlgaddis 6y ago> ... and the response to that is hardcoding a set of known good DNS servers which we can rely on. Netgear, at one point, also decided that hard-coding an IP address (for NTP, not DNS) was the best solution [0]. -- [0]: https://en.wikipedia.org/wiki/NTP_server_misuse_and_abuse#Netgear_and_the_University_of_Wisconsin%E2%80%93Madison https://en.wikipedia.org/wiki/NTP_server_misuse_and_abuse#Ne...
- Rychard 6y agoAs opposed to? I thought you had to specify your DNS server as an IP address. How would it resolve itself?
- jlgaddis 6y ago> As opposed to? As opposed to the IP addresses that your DHCP server told you about.
- bsagdiyev 6y agoSony did the same thing with a Blu-Ray player for NTP, effectively DDoSed that particular box because they also didn’t think to stagger the polling so like clockwork millions of consumer devices would check in at the same time nightly. From what I gathered from the electronics guys a few floors down they eventually resolved that in later revisions, or so I was told.
- lozaning 6y agoI worked on an IoT platform for a while that struggled with this. Our workaround was to tell support to tell people to not set automations to run on the quarter, half, 3 quarters or hour mark. Something set to run at 7:14 or 7:16 PM would run every night, something set to run at 7:15 was a crapshoot.
- black3r 6y agoand then there are providers/networks who straight up drop outgoing DNS queries, like the university I went to (including dormitory network).
- znpy 6y ago> Just be aware DNS over HTTPS is a thing now, and while the devices I’m responsible for aren’t going to try and evade your redirects, the companies that are trying to make sure ads get delivered will absolutely switch to DoH which will be much more difficult to work around. I wonder who's the first sponsor for that thing, dns-over-https...
- the8472 6y ago> and ISP provided DNS is almost universally terrible. They’ll ignore TTLs, rewrite NXDOMAIN responses into the IP address of their ad-laced web search, and occasionally just highjack every single query to send you to a page saying you’re approaching your bandwidth quota Universally? Perhaps in the US? But IoT devices are sold worldwide. While I do run my own recursive resolver I checked my ISP's and they're behaving fairly reasonable and do none of the above and I have a direct (contractual) relationship with them and we reside in the same jurisdiction so at least in principle I could apply pressure to them if they do something shady. The same can't be said about google or cloudflare. > Just be aware DNS over HTTPS is a thing now, and while the devices I’m responsible for aren’t going to try and evade your redirects, the companies that are trying to make sure ads get delivered will absolutely switch to DoH which will be much more difficult to work around. I'm sure google had only our best interests in mind when unleashing that on us.
- acdha 6y agoIt’s not just the U.S. A problem report once lead me to discover that the Spanish company Telefónica cached DNS results for years longer than the TTL. My error monitoring also picked up content injection (prior to HTTPS) and DNS hijacking around the world. Before going with the conspiratorial take note that Mozilla shipped DoH early and both Microsoft and Apple implemented it. Untrustworthy ISPs are a real problem even if Google deservedly gets suspicion about their motives. This isn’t another AMP.
- syshum 6y agoAt this point Mozilla is just the Non-profit Arm of Google, They do what google tells them.... and DoH is absolutely designed to get around network based security and filtering, both for Ad's and other reasons.
- acdha 6y ago> At this point Mozilla is just the Non-profit Arm of Google, They do what google tells them You can just say you don’t follow this closely. Mozilla is not perfect but they do push for privacy, with an increasingly limited amount of negotiating power. > DoH is absolutely designed to get around network based security and filtering, both for Ad's and other reasons. This is similarly reflecting a poor understanding of the situation. DoH can’t get around network filtering - if you block packets, there’s no magic trick to bypass it. It’s great for preventing ISPs from tampering with traffic or monitoring activity (this will also require eSNI to complete) but it’s not giving an attacker any capability they didn’t already have. If you’re concerned about security you’re fooling yourself if you don’t have endpoint management and some level of network segmentation and egress control. Attackers have hard-coded DNS servers, C&C endpoints, etc. for decades.
- maxerickson 6y agoHow come you don't specify that it be configurable?
- jon-wood 6y agoBecause no one has asked for it. The people who care are redirecting port 53, which the devices cope fine with, because if you’re doing that you probably have same resolvers upstream. The people who don’t care are never going to decide they’d like to change the DNS resolvers.
- rixrax 6y agoOut of curiosity, how do you handle situation where outbound tcp/udp port 53 is blocked? Do you fall back to dhcp supplied dns server?
- JMTQp8lwXL 6y agoThis sounds like a game of cat-and-mouse. Lists of 'known' DoH DNS servers will be collected, and PiHole-esque blocking will take shape via more advanced filtering.
- NelsonMinar 6y agoWhat a terrible state of affairs we're in now that you can't trust the network's own DNS. Please consider making it an advanced option to switch back on the DHCP provided address.
- o-__-o 6y ago> Please consider When you are contracted to build something to spec and you don’t build to spec, you don’t make any money. If you push back on the requested spec, the client will leave and go to another company thus leaving you without any money. What I’m trying to say unless you are writing the spec, you usually have no chance to change anything
- dvfjsdhgfv 6y ago> and occasionally just highjack every single query to send you to a page saying you’re approaching your bandwidth quota It boggles my mind it's still a problem in some countries. Last time I saw this kind of notice it was in the nineties, and on a web server, definitely not on a client endpoint. People have had unlimited bandwidth for at least a decade now.
- deleted 6y ago[deleted]
- quineoa 6y agoThis has regressed. In the NetZero days you would be limited to some hours per month depending on your plan. Then came unlimited cable. And now we are back: my residential AT&T DSL connection is limited to 1TB per month in the Bay Area
- iforgotpassword 6y agoWhoa. That would give you one day of laundry washing per month in the IoT age. https://old.reddit.com/r/homeautomation/comments/k72lzq/sowhy_would_my_lg_thinq_washer_have_a_need_to/ https://old.reddit.com/r/homeautomation/comments/k72lzq/sowh...
- Codazoa 6y agoComcast in my area has a 1TB/month limit. When I used their service I hit that limit nearly every month.
- Datenstrom 6y agoDoH will just be the start of another arms race. If you would like to control DoH also user LinuxBender responded to a question I asked about DoH blocklists with these repos in another post. I haven't got around to trying it out yet though. [1]: https://github.com/bambenek/block-doh https://github.com/bambenek/block-doh [2]: https://github.com/Sekhan/TheGreatWall https://github.com/Sekhan/TheGreatWall Edit: I am mostly curious to see what devices/applications it will break.
- mdoms 6y ago> In the face of that sort of behaviour you inevitably end up with technical support having to field angry customers who just don’t get that it’s not our fault the hardware isn’t working If your hardware isn't resilient to network failures then it absolutely is your fault.
- heavyset_go 6y ago> Just be aware DNS over HTTPS is a thing now, and while the devices I’m responsible for aren’t going to try and evade your redirects, the companies that are trying to make sure ads get delivered will absolutely switch to DoH which will be much more difficult to work around. I've been running PiHole-like software on my network for a few years now. A couple of years ago, it would block over 40% of traffic consistently. I never saw ads, and it was nice. In the last year, blocked traffic has dropped to about 15%, and I'm increasingly getting ads on my phone and Chromecast despite tunneling my traffic through my ad-blocked network and blocking Google's DNS at the network level.
- tamrix 6y agoSearch up some more blocklists online which are still being updated. It can use the same block lists browsers ad blocks use. Run 'update your block lists' on your Pihole to make sure the lists are being updated correctly.
- bosswipe 6y ago> DNS over HTTPS is a thing now I'm one of those "idiots" whose been whistling in the wind against encryption of everything and all kinds of security lockdowns and it's because of this sort of thing. The theoretical threat of someone sniffing my traffic is just not a concern to me compared to the very real and increasing threat of handing all control of my computing to centralized user-hostile powers.
- protomyth 6y agoI’m responsible for a bunch of IoT hardware, and every firmware spec I write includes a note on not using the DNS servers provided via DHCP. And that along with DoH is contributing to making my life a pain in the butt. How exactly do you folks who avoid our DHCP's DNS expect us to comply with legal filtering requirements? Also, what happens when your hard coded DNS servers are shutdown?
- mrmanner 6y ago> How exactly do you folks who avoid our DHCP's DNS expect us to comply with legal filtering requirements? I would argue that this is more a problem with the legal requirements than with the equipment - the law(maker) has expectations you can’t reasonably fulfill. Doesn’t make your situation any better of course, the law is the law even when it’s impossible
- asclepi 6y agoOh, we can fulfill them. We'll resort to TLS inspection and force you to trust our CA on your device if you want to continue accessing our corporate network. And now we get to see (almost) everything again, like in the "good old days," not just your DNS queries. Clear text DNS is the ultimate compromise, a gentleman's agreement if you want, that benefits everyone. We can see just enough to filter what we are required to by law on a best-effort basis, but we never see what you are actually doing thanks to the prevalence of TLS. DoH just broke that agreement. It's a sad example of how a privacy solution like DoH will eventually result in less privacy, at least in some environments. And I'm not even considering how DoH will be the excuse for totalitarian regimes to up their surveillance antics.
- protomyth 6y agoYeah. The pre-DoH world was good for both. I could say its all filtered for the kids on the locked down machines and the adults who knew something about technology could get on with their lives. Now, we are entering a world where we are going to end up locking down everyone. Good job. I'm damn sure once I have to do the trusted CA path that someone is going to sell a deep packet inspection solution and present it at some conference where someone in charge will hear about it and then it will be off to the races.
- deleted 6y ago[deleted]
- cannabis_sam 6y ago> ISP provided DNS is almost universally terrible Yet that is what internet actually is for a lot of people...
- mkhalil 6y agoI can attest to this. Story: Many years ago, a old friend of mine purhcased a new Panasonic Smart TV. It was when "Smart TVs" were just becoming a thing. I hooked her TV up for her; wiring it into the ATT uVerse modem directly. Other devices worked, but this one did not. After resetting the modem, factory resetting TV, and making sure the ip address on the TV's menu were displaying properly and matched the router's config (they were), as a young naive tech nerd, I just said: "Looks like they sold you a dud. Thankfully you kept the receipt!. Either way, you still have warranty to get it replaced." My friend replied: "Shouldn't I call them first before taking it back?" I said I didn't think it would help, but go ahead. About 30 mins laters, she was talking to Panasonic tech support and they asked her to manually enter the DNS entries [I believe it was 75.75.. so Comcrap's], and voila, the TV was online again. We were very happy it was an easy fix; but that day I deftinely did a LOT of reading on DNS servers. Till this day, DNS entries are something I always check over when troubleshooting (as well as setting my router to Cloudfare's).
- y1zhou 6y agoI've recently setup AdGuard Home on my Raspberry Pi, and noted that requests made from an Android phone weren't going through the DNS server at all. Turns out it uses a default IPv6 DNS first (which I found no method to modify) and then falls back to the manually entered IPv4 DNS server.
- rconti 6y agoWhere do you want them to send it to?
- zxcvasdf 6y agoIt's not perfect, but I find blocking all traffic in/out to global dns servers with pfBlockerNG to be an okay way to limit the DoH bypass of simple port 53 blocking. There are still ways around it, but it has caught a lot of other interesting traffic (snmp, ntp, etc.) leaving my network and that a simple port 53 block misses. This is the list I use: https://public-dns.info/nameservers-all.txt https://public-dns.info/nameservers-all.txt
- morpheuskafka 6y agoI agree, if it is an embedded device specifically designed to connect to a limited set of services, there is no reason to expect it to follow DNS provided by DHCP. Conceivably, there's no need to it to even use "real" DNS at all, you could just run a server that responds to queries like "updateserver.ecorp" and save the hassle of even announcing these servers to the public DNS at all. Also, DoH or not, there are plenty of other ways to ensure that ads get through a DNS filter. For example, a local hosts file could be included in firmware updates and they would just need to make extra effort to ensure that the server IPs didn't change (an elastic IP or load balancer in AWS would be all you need, then it can persist even if the VM has to be deleted).
- rndgermandude 6y agoI'd think most of IoT interfaces have some form of user-definable configuration. So most of the time it should be possible to make DNS user-configurable (hidden in some "advanced settings" area). Of course, it might lead to some more support calls eventually and requires a little more money (as in paid work) to implement, but you'd be playing nice and have a balanced compromise of default-hardcoded-DNS with an option to appease power users and/or orgs+corps, and you'd also avoid angry-me calling your support hotline and ranting at your people :P
- nuker 6y ago> Just be aware DNS over HTTPS is a thing now As is DoT :)
- topranks 6y agoI think that’s a gross overstatement about ISP DNS. Certainly in Europe I’ve not seen anything even close to that. I’d love to see a good country-by-country survey on ISP DNS to see just how common this manipulation is.
- lazyjones 6y agoThat's not really a consumer-friendly solution. It's also borderline illegal in the EU IMHO (if you hardcode DNS servers in the USA, for example). The toxic ISP issue isn't so dramatic where I live, it used to be worse 15-20 years ago. But the solution has always been: if the ISP is messing with you, you just buy your own router and configure it for your network, with a VPN tunnel if necessary. Ignoring DHCP makes this unnecessarily harder.
- Diskutant 6y agoWell then just use a hardcoded one by default but allow the user to set one manually and then use that.