7 ms·

I only know enough about networking to be dangerous but I am convinced Comcast is doing shady shit with my modem when I change the DNS settings to use non-Comca

I only know enough about networking to be dangerous but I am convinced Comcast is doing shady shit with my modem when I change the DNS settings to use non-Comcast servers. Every once in a while I’ll attempt to use Wireshark to try to make sense of what’s happening but I’m pretty clueless and don’t really know what I’m looking at/for. If anyone knows any good resources to learn about the ISP nuts and bolts that make internet magic happen between my modem and everyone else’s servers I would be most appreciative.

bewuethr 5y ago
I found The UNIX and Linux System Administration Handbook (5th Edition), chapters about networking and DNS very instructive, and they list a ton of additional references if you want to dig deeper.
gruez 5y ago
>I am convinced Comcast is doing shady shit with my modem when I change the DNS settings to use non-Comcast servers Well that's vague. What are the symptoms? How would comcast even know that you changed DNS settings? It's possible to infer that from DNS queries to their servers dropping off and traffic to 1.1.1.1 or 8.8.4.4 increasing, but I doubt comcast is competent enough to build that sort of detection system.
- yaur 5y ago
  On my home network I just run a transparent proxy and direct all outbound traffic bound to port 53 to my local dns server, it’s not hard.
  - xnyan 5y ago
    Interestingly enough, this is almost exactly how ISPs do it when they really want to get your attention. A couple years ago I forgot to update an expired credit card that I used to pay my spectrum cable bill. One morning every DNS request resolved to their "your account is about to be closed due to nonpayment" page. As I also use my own DNS sever I was surprised by this, and sure enough everything going out of my network on 53 was being grabbed up by their CGNAT and sent to their DNS server.
  - addingnumbers 5y ago
    I just block all outbound port 53 traffic, any device or app that doesn't honor my DHCP-provided DNS resolver can suck it. Looking at you, Chromecast that tries 8.8.8.8 40 times an hour even though you know perfectly damn well that 10.10.10.1 is working
    - robocat 5y ago
      The DNS provided by many ISPs is not to be trusted, as per this thread, so how else can your Chromecast act to find a trustworthy DNS? And with newer decides that use DoH, you can no longer prevent devices from contacting their own DNS provider without totally firewalling them (or perhaps using some IP blacklist or whitelist, if available?) https://en.wikipedia.org/wiki/DNS_over_HTTPS https://en.wikipedia.org/wiki/DNS_over_HTTPS
      - addingnumbers 5y ago
        When I said blocking all outbound 53 I meant no exceptions, my local forwarder already uses DoH to an outside resolver. Everything that I don't have complete visibility into the network stack of goes on a VLAN that does not forward traffic to the internet, it advertises a proxy via WPAD and DHCP option 252. I have a whitelist of hostnames that each device is allowed to make CONNECT requests to, so far there is only one. If it's not a plain unencrypted HTTP request to my proxy, or a CONNECT request involving a server/device pair I've decided to trust, it's not going anywhere. This breaks a lot of things that I would just as soon rather do without. I can't change my universal remote hub settings from the vendor portal, boo-hoo. I can't view my cameras from the hardened VLAN or from the internet (unless I VPN in first since the only copy of the recordings is on my local NAS)... good.
- roelschroeven 5y ago
  I should think they just detect traffic from your IP address on port 53 to any IP address that's not one of their nameservers.
- xnyan 5y ago
  I agree with you that comcast is incompetent, but everything becomes cheaper and easier over time and network hardware/software products that perform "deep" packet inspection at line rate as well as provide analytics on that returned data are now trivial and pretty much table stakes for Cisco, Juniper, Palo Alto et al. Specifically for detecting if a user is not using their DNS, yes you could correlate a user's http requests (unless you are using ESNI the requested domain is in plaintext by design) with traffic logs on their DNS server and observe that there was no DNS request to the ISP DNS server before a request was made, I don't think that would be necessary. Most users use the ISP default DNS - that's your baseline. If most customers hit your DNS X times per Mb of web traffic, then someone using a custom DNS is going to stand out like a sore thumb. Again, 100% agree that ISPs are not very technically competent (to put it mildly), but as time marches on the ability to both capture and more importantly analyze and report on that data is becoming cheaper and easier. ISPs want to get value from (sell) your data and vendors want to sell ISPs subscriptions to analytics and other platforms that bring them reoccurring revenue. Data from customer DNS is one of the most valuable sources of information an ISP has and I would be surprised if there was not at least an attempt to know how many customers did not use it.
- elliekelly 5y ago
  Sorry, I didn't get into details because I wasn't intending to ask HN to troubleshoot for me. But, since you asked... I have an "XFi Gateway" combination modem/router provided by Comcast (perhaps my first mistake) so the DNS settings are restricted and cannot be changed. I have the Comcast modem/router set to bridge mode and connected my own router where I can control the DNS settings. My understanding is the DNS settings closer to the client control. So in addition to having set my router to Cloudflare's DNS I also set my devices as well. One day, maybe a year ago or so, I'm on HN and I click an archive.is link, read the article, and go to the discussion thread only to see several comments about how archive.is is blocked by Cloudflare DNS. I checked the DNS settings on my MacBook and router and I was indeed using Cloudflare DNS but for some reason I was able to access the "blocked" address. So I went to the terminal, cleared the cache, and checked nslookup archive.is and it responded correctly. Then I checked a nonsense DNS server: nslookup archive.is 5.9.3.7 or something and it still responded correctly. I tried the same with different websites and got the same result. So I searched "see my DNS server" or something and found a few websites but they all showed Cloudflare. Very odd. When I logged in with my VPN, Mullvad, and changed the DNS settings on the router and my laptop to Mullvad's and repeated the experiment it finally returned NXDOMAIN. Then I disconnected from the VPN but left Mullvad's DNS settings, repeated the experiment again with the same results - even when I was using a totally bogus DNS server it was returning the correct IP address. That's when I installed Wireshark and, lo and behold, I could see the requests that should have been going to 1.1.1.1 or 5.9.3.7 going to 75.75.75.75. Comcast. A call to Comcast was, as expected, a complete waste of time. First they told me it was using their DNS settings because of "their firewall" and then they told me that if I used their built-in router rather than mine + bridge mode I wouldn't have the issue at all. Messing around in Wireshark I eventually determined the issue had something to do with one specific port that was making the requests (I can't recall how but I think because I could see Mullvad VPN was using a different port for DNS?) so I fiddled around and forced (or maybe redirected?) my router to use that port too and that finally worked in avoiding the Comcast servers. But, knowing just enough to be dangerous and not entirely sure what I was doing, I didn't keep the forced port and decided I'd have to get my own modem and use my VPN in the meantime. Before I had gotten around to buying a new modem (this was somewhat early in the pandemic) I saw a post on HN about NextDNS and decided I'd see if I ran into the same issue. I didn't, as far as I could tell at least. When I run Wireshark now (I still use NextDNS) I don't see any contact with 75.75.75.75 or 75.75.76.76. I think this is because NextDNS uses DoH? But who knows. Like I said, I only know enough to be dangerous so perhaps I just had something configured in an odd way that made the Comcast servers step in as a fail-safe and there's a totally innocent explanation. But based on my experience as a Comcast customer I don't really think they're deserving of the benefit of the doubt so I've definitely got a bit of a tin foil hat when it comes to them secretly messing around with my traffic through the leased modem.
Proven 5y ago
in other words, you've clue what's going on with your clients, but it must be Comcast because you're knowledgeable enough to know it's not you. right.
jaywalk 5y ago
Comcast isn't doing anything to your DNS. They're the largest ISP in the country, there'd be a huge uproar if they were doing something like that. There are plenty of experts who are subscribers who'd be able to figure out exactly what's going on.
- wizzwizz4 5y ago
  There's already a huge uproar around Comcast. But Comcast isn't losing any customers, because they have monopolies.
  - jaywalk 5y ago
    Yeah, uproar around prices and speeds and stuff like that. Nothing like screwing with third-party DNS requests.
    - wizzwizz4 5y ago
      Actually, I have heard that claim more than once, about various providers (up to, and including, “when I changed my DNS settings and the traffic slowed down, the tech got me to change them back, and the traffic sped up”). It's less common, I think, because more people know how to check their speed than change their DNS.
      - jaywalk 5y ago
        The only way I can think of that working is if the provider is intercepting DNS requests for popular speed tests to redirect to an internally-hosted version that would be faster. Otherwise, I can't think of any realistic way DNS settings can affect actual throughput.
        wizzwizz4 5y ago
        If the ISP is checking for DNS lookup of speed test websites, then allocating higher bandwidth to the connection for a brief period of time? Or, somehow more cynically, the ISP makes money from selling the data collected from DNS, so punishes people who use a different DNS provider. (DNS is plaintext-by-default, so I don't quite see how this would work, but it's possible.) Or perhaps the system uses DNS lookups as a proxy for “is a human browsing the web”; if there aren't enough, it's clearly some kind of automated computer program that doesn't deserve internet access.
_jal 5y ago
The recommendation of _The UNIX and Linux System Administration Handbook_ is a good one. As far as Comcast, I'm stuck with them, too. At least in my experience, they don't monkey with DNS - I run and use my own DNS servers, and have never seen interference. They do run deep packet inspection, and if they detect you, for instance, torrenting commercial media, they'll inject scary messages in port 80 traffic. Given that nearly all web traffic is encrypted now, the main effect of this is to break things like automated `apt-get update`s. One thing you can do to detect transparent DNS hijacking is to ask a nonexistent server a question. Something like `dig @13.14.15.16 news.ycombinator.com` should not give you an answer. If it does, someone's spying on and/or gaslighting you.
- mikro2nd 5y ago
  Curious: how can they detect whether you're torrenting commercial media if you've enabled Bittorrent protocol encryption? Surely all they can see then is the outer (envelope) of the packets...?
  - Shank 5y ago
    This is a bit of a misconception. Copyright holders have always gone after seeders based on people connecting to swarms, tracker info, and crawling DHT. There’s no reason to use DPI when the list of uploaders is just given out by trackers and DHT for free. See: https://www.usenix.org/legacy/event/woot10/tech/full_papers/Wolchok.pdf https://www.usenix.org/legacy/event/woot10/tech/full_papers/...
    - tgragnato 5y ago
      Exactly Bittorrent protocol encryption is only useful to protest against the use of DPI for bandwidth shaping, it has no influence on privacy. Even with (the weak) encryption, connections to trackers and DHT nodes are easily identified
      - mikro2nd 5y ago
        Thank you for clarifying this!
Cicero22 5y ago
I'm getting this second hand, but I've heard that some isps will redirect dns traffic to their servers based on it going to port 53.

bewuethr 5y ago

I found The UNIX and Linux System Administration Handbook (5th Edition), chapters about networking and DNS very instructive, and they list a ton of additional references if you want to dig deeper.

gruez 5y ago

>I am convinced Comcast is doing shady shit with my modem when I change the DNS settings to use non-Comcast servers Well that's vague. What are the symptoms? How would comcast even know that you changed DNS settings? It's possible to infer that from DNS queries to their servers dropping off and traffic to 1.1.1.1 or 8.8.4.4 increasing, but I doubt comcast is competent enough to build that sort of detection system.

yaur 5y ago
On my home network I just run a transparent proxy and direct all outbound traffic bound to port 53 to my local dns server, it’s not hard.
- xnyan 5y ago
  Interestingly enough, this is almost exactly how ISPs do it when they really want to get your attention. A couple years ago I forgot to update an expired credit card that I used to pay my spectrum cable bill. One morning every DNS request resolved to their "your account is about to be closed due to nonpayment" page. As I also use my own DNS sever I was surprised by this, and sure enough everything going out of my network on 53 was being grabbed up by their CGNAT and sent to their DNS server.
- addingnumbers 5y ago
  I just block all outbound port 53 traffic, any device or app that doesn't honor my DHCP-provided DNS resolver can suck it. Looking at you, Chromecast that tries 8.8.8.8 40 times an hour even though you know perfectly damn well that 10.10.10.1 is working
  - robocat 5y ago
    The DNS provided by many ISPs is not to be trusted, as per this thread, so how else can your Chromecast act to find a trustworthy DNS? And with newer decides that use DoH, you can no longer prevent devices from contacting their own DNS provider without totally firewalling them (or perhaps using some IP blacklist or whitelist, if available?) https://en.wikipedia.org/wiki/DNS_over_HTTPS https://en.wikipedia.org/wiki/DNS_over_HTTPS
    - addingnumbers 5y ago
      When I said blocking all outbound 53 I meant no exceptions, my local forwarder already uses DoH to an outside resolver. Everything that I don't have complete visibility into the network stack of goes on a VLAN that does not forward traffic to the internet, it advertises a proxy via WPAD and DHCP option 252. I have a whitelist of hostnames that each device is allowed to make CONNECT requests to, so far there is only one. If it's not a plain unencrypted HTTP request to my proxy, or a CONNECT request involving a server/device pair I've decided to trust, it's not going anywhere. This breaks a lot of things that I would just as soon rather do without. I can't change my universal remote hub settings from the vendor portal, boo-hoo. I can't view my cameras from the hardened VLAN or from the internet (unless I VPN in first since the only copy of the recordings is on my local NAS)... good.
roelschroeven 5y ago
I should think they just detect traffic from your IP address on port 53 to any IP address that's not one of their nameservers.
xnyan 5y ago
I agree with you that comcast is incompetent, but everything becomes cheaper and easier over time and network hardware/software products that perform "deep" packet inspection at line rate as well as provide analytics on that returned data are now trivial and pretty much table stakes for Cisco, Juniper, Palo Alto et al. Specifically for detecting if a user is not using their DNS, yes you could correlate a user's http requests (unless you are using ESNI the requested domain is in plaintext by design) with traffic logs on their DNS server and observe that there was no DNS request to the ISP DNS server before a request was made, I don't think that would be necessary. Most users use the ISP default DNS - that's your baseline. If most customers hit your DNS X times per Mb of web traffic, then someone using a custom DNS is going to stand out like a sore thumb. Again, 100% agree that ISPs are not very technically competent (to put it mildly), but as time marches on the ability to both capture and more importantly analyze and report on that data is becoming cheaper and easier. ISPs want to get value from (sell) your data and vendors want to sell ISPs subscriptions to analytics and other platforms that bring them reoccurring revenue. Data from customer DNS is one of the most valuable sources of information an ISP has and I would be surprised if there was not at least an attempt to know how many customers did not use it.
elliekelly 5y ago
Sorry, I didn't get into details because I wasn't intending to ask HN to troubleshoot for me. But, since you asked... I have an "XFi Gateway" combination modem/router provided by Comcast (perhaps my first mistake) so the DNS settings are restricted and cannot be changed. I have the Comcast modem/router set to bridge mode and connected my own router where I can control the DNS settings. My understanding is the DNS settings closer to the client control. So in addition to having set my router to Cloudflare's DNS I also set my devices as well. One day, maybe a year ago or so, I'm on HN and I click an archive.is link, read the article, and go to the discussion thread only to see several comments about how archive.is is blocked by Cloudflare DNS. I checked the DNS settings on my MacBook and router and I was indeed using Cloudflare DNS but for some reason I was able to access the "blocked" address. So I went to the terminal, cleared the cache, and checked nslookup archive.is and it responded correctly. Then I checked a nonsense DNS server: nslookup archive.is 5.9.3.7 or something and it still responded correctly. I tried the same with different websites and got the same result. So I searched "see my DNS server" or something and found a few websites but they all showed Cloudflare. Very odd. When I logged in with my VPN, Mullvad, and changed the DNS settings on the router and my laptop to Mullvad's and repeated the experiment it finally returned NXDOMAIN. Then I disconnected from the VPN but left Mullvad's DNS settings, repeated the experiment again with the same results - even when I was using a totally bogus DNS server it was returning the correct IP address. That's when I installed Wireshark and, lo and behold, I could see the requests that should have been going to 1.1.1.1 or 5.9.3.7 going to 75.75.75.75. Comcast. A call to Comcast was, as expected, a complete waste of time. First they told me it was using their DNS settings because of "their firewall" and then they told me that if I used their built-in router rather than mine + bridge mode I wouldn't have the issue at all. Messing around in Wireshark I eventually determined the issue had something to do with one specific port that was making the requests (I can't recall how but I think because I could see Mullvad VPN was using a different port for DNS?) so I fiddled around and forced (or maybe redirected?) my router to use that port too and that finally worked in avoiding the Comcast servers. But, knowing just enough to be dangerous and not entirely sure what I was doing, I didn't keep the forced port and decided I'd have to get my own modem and use my VPN in the meantime. Before I had gotten around to buying a new modem (this was somewhat early in the pandemic) I saw a post on HN about NextDNS and decided I'd see if I ran into the same issue. I didn't, as far as I could tell at least. When I run Wireshark now (I still use NextDNS) I don't see any contact with 75.75.75.75 or 75.75.76.76. I think this is because NextDNS uses DoH? But who knows. Like I said, I only know enough to be dangerous so perhaps I just had something configured in an odd way that made the Comcast servers step in as a fail-safe and there's a totally innocent explanation. But based on my experience as a Comcast customer I don't really think they're deserving of the benefit of the doubt so I've definitely got a bit of a tin foil hat when it comes to them secretly messing around with my traffic through the leased modem.

Proven 5y ago

in other words, you've clue what's going on with your clients, but it must be Comcast because you're knowledgeable enough to know it's not you. right.

jaywalk 5y ago

Comcast isn't doing anything to your DNS. They're the largest ISP in the country, there'd be a huge uproar if they were doing something like that. There are plenty of experts who are subscribers who'd be able to figure out exactly what's going on.

wizzwizz4 5y ago
There's already a huge uproar around Comcast. But Comcast isn't losing any customers, because they have monopolies.
- jaywalk 5y ago
  Yeah, uproar around prices and speeds and stuff like that. Nothing like screwing with third-party DNS requests.
  - wizzwizz4 5y ago
    Actually, I have heard that claim more than once, about various providers (up to, and including, “when I changed my DNS settings and the traffic slowed down, the tech got me to change them back, and the traffic sped up”). It's less common, I think, because more people know how to check their speed than change their DNS.
    - jaywalk 5y ago
      The only way I can think of that working is if the provider is intercepting DNS requests for popular speed tests to redirect to an internally-hosted version that would be faster. Otherwise, I can't think of any realistic way DNS settings can affect actual throughput.
      - wizzwizz4 5y ago
        If the ISP is checking for DNS lookup of speed test websites, then allocating higher bandwidth to the connection for a brief period of time? Or, somehow more cynically, the ISP makes money from selling the data collected from DNS, so punishes people who use a different DNS provider. (DNS is plaintext-by-default, so I don't quite see how this would work, but it's possible.) Or perhaps the system uses DNS lookups as a proxy for “is a human browsing the web”; if there aren't enough, it's clearly some kind of automated computer program that doesn't deserve internet access.

_jal 5y ago

The recommendation of _The UNIX and Linux System Administration Handbook_ is a good one. As far as Comcast, I'm stuck with them, too. At least in my experience, they don't monkey with DNS - I run and use my own DNS servers, and have never seen interference. They do run deep packet inspection, and if they detect you, for instance, torrenting commercial media, they'll inject scary messages in port 80 traffic. Given that nearly all web traffic is encrypted now, the main effect of this is to break things like automated `apt-get update`s. One thing you can do to detect transparent DNS hijacking is to ask a nonexistent server a question. Something like `dig @13.14.15.16 news.ycombinator.com` should not give you an answer. If it does, someone's spying on and/or gaslighting you.

mikro2nd 5y ago
Curious: how can they detect whether you're torrenting commercial media if you've enabled Bittorrent protocol encryption? Surely all they can see then is the outer (envelope) of the packets...?
- Shank 5y ago
  This is a bit of a misconception. Copyright holders have always gone after seeders based on people connecting to swarms, tracker info, and crawling DHT. There’s no reason to use DPI when the list of uploaders is just given out by trackers and DHT for free. See: https://www.usenix.org/legacy/event/woot10/tech/full_papers/Wolchok.pdf https://www.usenix.org/legacy/event/woot10/tech/full_papers/...
  - tgragnato 5y ago
    Exactly Bittorrent protocol encryption is only useful to protest against the use of DPI for bandwidth shaping, it has no influence on privacy. Even with (the weak) encryption, connections to trackers and DHT nodes are easily identified
    - mikro2nd 5y ago
      Thank you for clarifying this!

Cicero22 5y ago

I'm getting this second hand, but I've heard that some isps will redirect dns traffic to their servers based on it going to port 53.