7 ms·

DoH Privacy Enhancement: Do Not Set the User-Agent Header for DoH Requests

116 points by generalpass 7y ago

Someone1234 7y ago
I'm glad this happened. I feel like it is one of those "if you don't do it now, you won't be able to do it later." As more DoH implementations spring up, at least one of them will require the User-Agent for no reason. Plus it is useless, may allow hacky discrimination, and wastes bytes.
- desc 7y ago
  I wonder if the best approach might be to specify the precise request format required for DoH and actively deny requests with extra information. 'Thou shalt provide exactly this, no more and no less' Depends on whether we want to consider later 'improvements' to be silent extensions or an entirely new protocol version. Given how easily abused HTTP headers have been in the past, might be worth the inertia to limit extensibility by design in order to limit MITM tracking. But MITM middleboxes will of course start to rely on that... I suppose that if TLS is broken for this part of the system you're rather screwed anyway, but defence in depth is always worth considering.
  - anoncake 7y ago
    What's the point of using HTTP for DNS if you aren't going to use its abilities?
    - mantap 7y ago
      The point, iirc is to run it on port 443 so that it doesn't get blocked.
      - anoncake 7y ago
        That would explain DNS over TLS on port 443, no HTTP needed.
        anticensor 7y ago
        Yes but initial hanshakes would look foreign compared to regular port 443 traffic.
    - CiPHPerCoder 7y ago
      To stop people from blocking DNS over HTTPS.
    - Someone1234 7y ago
      Its raison d'être is largely to mitigate the issues that limited DoT's (DNS over TLS) widespread adoption, specifically "Middlebox Bypass." Meaning if you connect to a public WiFi network, many will only allow captive DNS and block other protocols including DoT. DoH is difficult to distinguish from other HTTPS traffic and is more likely to successfully resolve (since they cannot block HTTPS and still be useful). Therefore you've created a secure, flexible, and reliable DNS replacement with only mild technical downsides on modern hardware (a lot of the complaints are ideological or relate to specific incompatibilities with existing solutions like the lack of HOSTS file support in the initial offerings). So I guess, everything I just said is "the point." Rather than trying to use every single possible combination of HTTP headers.
      - ocdtrekkie 7y ago
        DoH as a protocol has no issues with HOSTS files. The issue is browsers implementing DoH instead of operating systems. If browsers stayed in their lane, and let operating systems implement DoH, compliant with corporate IT design and systems like the HOSTS file, everything would be fine. Microsoft has already committed to implement DoH in Windows itself. Browsers just need to stop trying to be their own DNS clients.
        PenguinCoder 7y ago
        Spot on. This needs to be supported and implemented at the OS layer, not the application
        baroffoos 7y ago
        OSs didn't implement it though. And they never would unless browsers forced it on them. The right approach is for browsers to bring it in and then when the OS supports it, default to the OS version. But right now I don't know of any OS other than I think android that supports it.
        PenguinCoder 7y ago
    - em-bee 7y ago
      the user-agent header was a mistake from the very beginning. it was meant to help servers cater to different browser capabilities, but it ended up being misused to segregate the net and drive non-standard features. without the user-agent header, browsers would have been forced to standardize earlier.
      - pg_is_a_butt 7y ago
        > without the user-agent header, browsers would have been forced to standardize earlier. Or, we'd all still be using WorldWideWeb 1.0
- nicolaslem 7y ago
  If it comes to wasted bytes, there is also Server response header which is high on my list of things to disable whenever I get the chance.
ghostpepper 7y ago
To save everyone else from googling, DoH is DNS over HTTPS.
- burundi_coffee 7y ago
  Not to be confused with DNS over TLS (DoT)
badrabbit 7y ago
Just curious, would your employers let you use DoH? I was told to stop using it.
- treve 7y ago
  Employers are typically not afraid of the protocol, but afraid of not being able to control it. Companies can run their own filtered DoH just like they can with DNS.
3xblah 7y ago
The Cloudflare TRR endpoint is mapped to a subdomain that indicates the user agent, i.e., browser, was distributed by Mozilla: mozilla.cloudflare-dns.com. The IP address for the subdomain is the same as for cloudflare-dns.com. Would this subdomain be leaked in SNI.^1 Presumably it serves to enable someone, maybe Cloudflare, to track queries as coming from Firefox. Will the endpoints for other TRRs have special subdomains. Perhaps another "privacy enhancement" would be to avoid using a browser-specific subdomain. 1. ESNI is still experimental and if I am not mistaken it is disabled by default in Firefox.
- a012 7y ago
  Encrypted SNI is still experimental and Disabled by default in Firefox, but it requires Cloudflare to work.
- M2Ys4U 7y ago
  That's still better than sending the entire UA, though, which includes OS and browser version numbers.
g82918 7y ago
I feel like User-Agent in general is not well suited to the modern web. Most browsers pretend to be some version of Mozilla or Netscape. And most things that scrape sites like Apple's messaging app pretend to be something equally outdated. Deprecating User-Agent seems like the best course of action in the long run.
- judge2020 7y ago
  Relevant https://webaim.org/blog/user-agent-string-history/ https://webaim.org/blog/user-agent-string-history/
- rictic 7y ago
  I'd be happy for the User-Agent string to die (or be frozen in amber), but I want the ability to have a rough sense of the capabilities of a browser in order to determine what JS I should send down to them. Like, if you're on IE11 I can send down code that's compiled to ES5 and has all the polyfills you need, and if you're on almost any other browser I just need some signal that you've got a reasonably up to date version and I can send down a much smaller and more optimal JS file. Maybe runtime browser-side feature detection can be fast enough. I haven't benchmarked. The other tricky thing that User-Agent strings are good for is working around browser bugs. Every once in a while a browser will ship with a totally broken language feature that's essentially impossible to detect at runtime. A particularly bad example was from an old version of Edge, which had a JIT bug where class constructors could – in very particular circumstances – return the class object itself rather than an instance of the class. In that situation, if you want your site to work you have to detect that version of Edge and treat it like IE: send it the bigger, more compatible blob rather than code for modern browsers.
- tzs 7y ago
  I'd be OK with getting rid of User-Agent if some kind of "capabilities supported" header were added. Something like Agent-Capabilities: 1; 97464bde5d94a54f6e199309489a4b60 where the 97464bde5d94a54f6e199309489a4b60 is a set of bit flags in hex, each representing some feature, and the 1 before the semicolon is a version number for the bit definitions. The assignment of these flags would have to be standardized. When new versions of HTML, CSS, or JavaScript are standardized, that would include assigning bits for their new features, and bumping the bit definition version number. You might say that it is better to do run-time checks on the client for features you want, and have the client then load workarounds if necessary for missing features, and that indeed is a good way to go in many cases (maybe even most cases). That method, though, only works going forward. Sometimes I want to know what capabilities past visitors to my site have had. For example, I'm going to be redoing some pages at work and want to know if I can require certain CSS and JavaScript and HTML features. I could make a list of the features I want to use, add some JavaScript to those pages to test for those, and send the results back to the server...and then I'd have to wait weeks or months or longer to get a good idea of what the consequences would be of requiring those features. With User-Agent strings, all I had to do was pull up the site logs. Our shopping cart logs the User-Agent string, and page flow through the cart. It was an easy matter to pull up the last 18 months of those logs, and write a script to find all the successful orders and make a table showing how many there were for each browser. (I limited it to successful orders, because those would almost certainly all be real people in regular browsers that are reasonably honest about the User-Agents). I could then look up the features I was interested in on https://caniuse.com/ https://caniuse.com/ and see which of the features were not on browsers that we still get significant traffic through. Dropping User-Agent without adding something to allow that kind of analysis, such as Agent-Capabilities, would be very annoying.
cocoa19 7y ago
> Instead of setting it to an arbitrary string it would be great if the UA header was not set at all. Didn't someone post his experience getting more expensive flights because he used a Mac? Yes! Please, take UA out.
- kalmi10 7y ago
  The sites still get your UA. This was about your DNS server getting your UA.
- robgibbons 7y ago
  Destination websites/servers can still require that you have a UA. DoH moving away from requiring a UA wouldn't change that.
- jrockway 7y ago
  Are we sure this wasn't a session cookie that just happened to pick experiment A or experiment B? The user agent isn't very important. What we should start being worried about is that ISPs are now offering API access that maps your IP address to your phone number and demographic information. (Verizon does this right now, but you can opt out in your account settings.) Maybe all those VPN providers have a point. Delete the user agent, share an IP with thousands of people, and maybe you can browse in peace.
  - DownGoat 7y ago
    Then you get stuck in ReCaptcha hell because you are not "human".
- mrspuratic 7y ago
  Then there's this: https://isc.sans.edu/forums/diary/Is+it+Possible+to+Identify+DNS+over+HTTPs+Without+Decrypting+TLS/25616/ https://isc.sans.edu/forums/diary/Is+it+Possible+to+Identify... detection based on payload size alone. Any implementations with a random padding ballast header?
deleted 7y ago
[deleted]
deleted 7y ago
[deleted]
rasvj 7y ago
What if there's a bug in Mozilla's implementation at some point and DoH servers have to return a slightly different response for certain versions of Firefox? How will they achieve that?
- Kuinox 7y ago
  Then Mozilla need to fix their implementation. If the client is bugged you don't fix the bug in the server.
  - rasvj 7y ago
    Welcome to the real world.
- maxbond 7y ago
  That seems less like a fix than a bandaid applied at the wrong level of the stack. It's perfectly normal for websites to serve slightly different pages to different browser versions, but I would submit that is not a desirable property for a system to have but rather a symptom of our inability to properly standardize & implement something as complex as a browser.
DyslexicAtheist 7y ago
I don't see a benefit of the built in DoH in firefox since it is shifting the trust from my existing DNS provider to cloudflare[1] (based in the US). Today I discovered that I couldn't just kill the disqus.com requests with my usual method by sink-holing them in /etc/hosts because for some reason firefox ignored the /etc/hosts file. While Chrome seemed to work fine and local dig requests also returned disqus.com. 0 IN A 0.0.0.0 I was left dumbstruck. I was able to override this in the about:config as described in [1]. I really don't like how mozilla ignors what is defined in /etc/nsswitsch.conf because this breaks all the blacklisting rules a user had set up themselves. At least they should have produced a huge warning telling users that this ignores any local blacklists that are in place. [1] https://ungleich.ch/en-us/cms/blog/2019/09/11/turn-off-doh-firefox/ https://ungleich.ch/en-us/cms/blog/2019/09/11/turn-off-doh-f...
- ceejayoz 7y ago
  Yeah, this was a surprise to me as well. Spent a while trying to figure out why Firefox wouldn't show a local development instance despite curling it working fine.
- cf141q5325 7y ago
  >At least they should have produced a huge warning telling users that this ignores any local blacklists that are in place. Its not just here, its unfortunately common with firefox. I still cant believe that firefox just silently disables addons if they are no longer supported instead of giving a giant colorful warning.
- mbreese 7y ago
  If you are using DoH, wouldn’t you need to ignore the local resolv/nsswitch settings? Otherwise your request would get sent out to whatever the configured DNS server is. I’m sure there is some smarter way to do this, but when you’re handling network configuration at the application level, you’re going to miss local changes (including blacklists). Maybe he best way to handle this would be to change the system level resolver to use DoH?
  - DyslexicAtheist 7y ago
nif2ee 7y ago
It's all bout money encapsulated by "security" marketing buzzword. Yes, DoH is an improvement over old plaintext UDP DNS. But why should I used a nobody service over Cloudflare or Google that update their databases within seconds all over the world and rarely if ever faced a downtime?
- generalpass 7y ago
  > It's all bout money encapsulated by "security" marketing buzzword. Yes, DoH is an improvement over old plaintext UDP DNS. But why should I used a nobody service over Cloudflare or Google that update their databases within seconds all over the world and rarely if ever faced a downtime? It is not so straightforward as always being an improvement. For example, a user can be uniquely identified through session resumption session ID whereas with plaintext UDP behind a firewall with many others doing lookups unique identification of a user can be close to impossible.
  - nif2ee 7y ago
    Yes, that's a one more reason to trust Cloudflare and Google over a nobody service since their motives with their free DNS services (including DoH and DNS over TLS) relate more to having up-to-date synchronization, load balancing, etc.. with the servers they host themselves. Spying on users via DNS queries by Cloudflare or Google is kind of a joke that can bait only the ignorant. Cloudflare terminates TLS connections and even issues TLS certs and can "theoretically" see every sensitive data that can compromise both businesses and users and yet they are trusted by countless public companies and startups alike. The same thing if not more for Google for their own ervices or their GCP business.

116 points by generalpass 7y ago

I'm glad this happened. I feel like it is one of those "if you don't do it now, you won't be able to do it later." As more DoH implementations spring up, at least one of them will require the User-Agent for no reason. Plus it is useless, may allow hacky discrimination, and wastes bytes.

desc 7y ago
I wonder if the best approach might be to specify the precise request format required for DoH and actively deny requests with extra information. 'Thou shalt provide exactly this, no more and no less' Depends on whether we want to consider later 'improvements' to be silent extensions or an entirely new protocol version. Given how easily abused HTTP headers have been in the past, might be worth the inertia to limit extensibility by design in order to limit MITM tracking. But MITM middleboxes will of course start to rely on that... I suppose that if TLS is broken for this part of the system you're rather screwed anyway, but defence in depth is always worth considering.
- anoncake 7y ago
  What's the point of using HTTP for DNS if you aren't going to use its abilities?
  - mantap 7y ago
    The point, iirc is to run it on port 443 so that it doesn't get blocked.
    - anoncake 7y ago
      That would explain DNS over TLS on port 443, no HTTP needed.
      - anticensor 7y ago
        Yes but initial hanshakes would look foreign compared to regular port 443 traffic.
  - CiPHPerCoder 7y ago
    To stop people from blocking DNS over HTTPS.
  - Someone1234 7y ago
    Its raison d'être is largely to mitigate the issues that limited DoT's (DNS over TLS) widespread adoption, specifically "Middlebox Bypass." Meaning if you connect to a public WiFi network, many will only allow captive DNS and block other protocols including DoT. DoH is difficult to distinguish from other HTTPS traffic and is more likely to successfully resolve (since they cannot block HTTPS and still be useful). Therefore you've created a secure, flexible, and reliable DNS replacement with only mild technical downsides on modern hardware (a lot of the complaints are ideological or relate to specific incompatibilities with existing solutions like the lack of HOSTS file support in the initial offerings). So I guess, everything I just said is "the point." Rather than trying to use every single possible combination of HTTP headers.
    - ocdtrekkie 7y ago
      DoH as a protocol has no issues with HOSTS files. The issue is browsers implementing DoH instead of operating systems. If browsers stayed in their lane, and let operating systems implement DoH, compliant with corporate IT design and systems like the HOSTS file, everything would be fine. Microsoft has already committed to implement DoH in Windows itself. Browsers just need to stop trying to be their own DNS clients.
      - PenguinCoder 7y ago
        Spot on. This needs to be supported and implemented at the OS layer, not the application
        baroffoos 7y ago
        OSs didn't implement it though. And they never would unless browsers forced it on them. The right approach is for browsers to bring it in and then when the OS supports it, default to the OS version. But right now I don't know of any OS other than I think android that supports it.
        PenguinCoder 7y ago
  - em-bee 7y ago
    the user-agent header was a mistake from the very beginning. it was meant to help servers cater to different browser capabilities, but it ended up being misused to segregate the net and drive non-standard features. without the user-agent header, browsers would have been forced to standardize earlier.
    - pg_is_a_butt 7y ago
      > without the user-agent header, browsers would have been forced to standardize earlier. Or, we'd all still be using WorldWideWeb 1.0
nicolaslem 7y ago
If it comes to wasted bytes, there is also Server response header which is high on my list of things to disable whenever I get the chance.

ghostpepper 7y ago

To save everyone else from googling, DoH is DNS over HTTPS.

burundi_coffee 7y ago
Not to be confused with DNS over TLS (DoT)

badrabbit 7y ago

Just curious, would your employers let you use DoH? I was told to stop using it.

treve 7y ago
Employers are typically not afraid of the protocol, but afraid of not being able to control it. Companies can run their own filtered DoH just like they can with DNS.

3xblah 7y ago

The Cloudflare TRR endpoint is mapped to a subdomain that indicates the user agent, i.e., browser, was distributed by Mozilla: mozilla.cloudflare-dns.com. The IP address for the subdomain is the same as for cloudflare-dns.com. Would this subdomain be leaked in SNI.^1 Presumably it serves to enable someone, maybe Cloudflare, to track queries as coming from Firefox. Will the endpoints for other TRRs have special subdomains. Perhaps another "privacy enhancement" would be to avoid using a browser-specific subdomain. 1. ESNI is still experimental and if I am not mistaken it is disabled by default in Firefox.

a012 7y ago
Encrypted SNI is still experimental and Disabled by default in Firefox, but it requires Cloudflare to work.
M2Ys4U 7y ago
That's still better than sending the entire UA, though, which includes OS and browser version numbers.

g82918 7y ago

I feel like User-Agent in general is not well suited to the modern web. Most browsers pretend to be some version of Mozilla or Netscape. And most things that scrape sites like Apple's messaging app pretend to be something equally outdated. Deprecating User-Agent seems like the best course of action in the long run.

judge2020 7y ago
Relevant https://webaim.org/blog/user-agent-string-history/ https://webaim.org/blog/user-agent-string-history/
rictic 7y ago
I'd be happy for the User-Agent string to die (or be frozen in amber), but I want the ability to have a rough sense of the capabilities of a browser in order to determine what JS I should send down to them. Like, if you're on IE11 I can send down code that's compiled to ES5 and has all the polyfills you need, and if you're on almost any other browser I just need some signal that you've got a reasonably up to date version and I can send down a much smaller and more optimal JS file. Maybe runtime browser-side feature detection can be fast enough. I haven't benchmarked. The other tricky thing that User-Agent strings are good for is working around browser bugs. Every once in a while a browser will ship with a totally broken language feature that's essentially impossible to detect at runtime. A particularly bad example was from an old version of Edge, which had a JIT bug where class constructors could – in very particular circumstances – return the class object itself rather than an instance of the class. In that situation, if you want your site to work you have to detect that version of Edge and treat it like IE: send it the bigger, more compatible blob rather than code for modern browsers.
tzs 7y ago
I'd be OK with getting rid of User-Agent if some kind of "capabilities supported" header were added. Something like Agent-Capabilities: 1; 97464bde5d94a54f6e199309489a4b60 where the 97464bde5d94a54f6e199309489a4b60 is a set of bit flags in hex, each representing some feature, and the 1 before the semicolon is a version number for the bit definitions. The assignment of these flags would have to be standardized. When new versions of HTML, CSS, or JavaScript are standardized, that would include assigning bits for their new features, and bumping the bit definition version number. You might say that it is better to do run-time checks on the client for features you want, and have the client then load workarounds if necessary for missing features, and that indeed is a good way to go in many cases (maybe even most cases). That method, though, only works going forward. Sometimes I want to know what capabilities past visitors to my site have had. For example, I'm going to be redoing some pages at work and want to know if I can require certain CSS and JavaScript and HTML features. I could make a list of the features I want to use, add some JavaScript to those pages to test for those, and send the results back to the server...and then I'd have to wait weeks or months or longer to get a good idea of what the consequences would be of requiring those features. With User-Agent strings, all I had to do was pull up the site logs. Our shopping cart logs the User-Agent string, and page flow through the cart. It was an easy matter to pull up the last 18 months of those logs, and write a script to find all the successful orders and make a table showing how many there were for each browser. (I limited it to successful orders, because those would almost certainly all be real people in regular browsers that are reasonably honest about the User-Agents). I could then look up the features I was interested in on https://caniuse.com/ https://caniuse.com/ and see which of the features were not on browsers that we still get significant traffic through. Dropping User-Agent without adding something to allow that kind of analysis, such as Agent-Capabilities, would be very annoying.

cocoa19 7y ago

> Instead of setting it to an arbitrary string it would be great if the UA header was not set at all. Didn't someone post his experience getting more expensive flights because he used a Mac? Yes! Please, take UA out.

kalmi10 7y ago
The sites still get your UA. This was about your DNS server getting your UA.
robgibbons 7y ago
Destination websites/servers can still require that you have a UA. DoH moving away from requiring a UA wouldn't change that.
jrockway 7y ago
Are we sure this wasn't a session cookie that just happened to pick experiment A or experiment B? The user agent isn't very important. What we should start being worried about is that ISPs are now offering API access that maps your IP address to your phone number and demographic information. (Verizon does this right now, but you can opt out in your account settings.) Maybe all those VPN providers have a point. Delete the user agent, share an IP with thousands of people, and maybe you can browse in peace.
- DownGoat 7y ago
  Then you get stuck in ReCaptcha hell because you are not "human".
mrspuratic 7y ago
Then there's this: https://isc.sans.edu/forums/diary/Is+it+Possible+to+Identify+DNS+over+HTTPs+Without+Decrypting+TLS/25616/ https://isc.sans.edu/forums/diary/Is+it+Possible+to+Identify... detection based on payload size alone. Any implementations with a random padding ballast header?

deleted 7y ago

[deleted]

deleted 7y ago

[deleted]

rasvj 7y ago

What if there's a bug in Mozilla's implementation at some point and DoH servers have to return a slightly different response for certain versions of Firefox? How will they achieve that?

Kuinox 7y ago
Then Mozilla need to fix their implementation. If the client is bugged you don't fix the bug in the server.
- rasvj 7y ago
  Welcome to the real world.
maxbond 7y ago
That seems less like a fix than a bandaid applied at the wrong level of the stack. It's perfectly normal for websites to serve slightly different pages to different browser versions, but I would submit that is not a desirable property for a system to have but rather a symptom of our inability to properly standardize & implement something as complex as a browser.

DyslexicAtheist 7y ago

I don't see a benefit of the built in DoH in firefox since it is shifting the trust from my existing DNS provider to cloudflare[1] (based in the US). Today I discovered that I couldn't just kill the disqus.com requests with my usual method by sink-holing them in /etc/hosts because for some reason firefox ignored the /etc/hosts file. While Chrome seemed to work fine and local dig requests also returned disqus.com. 0 IN A 0.0.0.0 I was left dumbstruck. I was able to override this in the about:config as described in [1]. I really don't like how mozilla ignors what is defined in /etc/nsswitsch.conf because this breaks all the blacklisting rules a user had set up themselves. At least they should have produced a huge warning telling users that this ignores any local blacklists that are in place. [1] https://ungleich.ch/en-us/cms/blog/2019/09/11/turn-off-doh-firefox/ https://ungleich.ch/en-us/cms/blog/2019/09/11/turn-off-doh-f...

ceejayoz 7y ago
Yeah, this was a surprise to me as well. Spent a while trying to figure out why Firefox wouldn't show a local development instance despite curling it working fine.
cf141q5325 7y ago
>At least they should have produced a huge warning telling users that this ignores any local blacklists that are in place. Its not just here, its unfortunately common with firefox. I still cant believe that firefox just silently disables addons if they are no longer supported instead of giving a giant colorful warning.
mbreese 7y ago
If you are using DoH, wouldn’t you need to ignore the local resolv/nsswitch settings? Otherwise your request would get sent out to whatever the configured DNS server is. I’m sure there is some smarter way to do this, but when you’re handling network configuration at the application level, you’re going to miss local changes (including blacklists). Maybe he best way to handle this would be to change the system level resolver to use DoH?
- DyslexicAtheist 7y ago

nif2ee 7y ago

It's all bout money encapsulated by "security" marketing buzzword. Yes, DoH is an improvement over old plaintext UDP DNS. But why should I used a nobody service over Cloudflare or Google that update their databases within seconds all over the world and rarely if ever faced a downtime?

generalpass 7y ago
> It's all bout money encapsulated by "security" marketing buzzword. Yes, DoH is an improvement over old plaintext UDP DNS. But why should I used a nobody service over Cloudflare or Google that update their databases within seconds all over the world and rarely if ever faced a downtime? It is not so straightforward as always being an improvement. For example, a user can be uniquely identified through session resumption session ID whereas with plaintext UDP behind a firewall with many others doing lookups unique identification of a user can be close to impossible.
- nif2ee 7y ago
  Yes, that's a one more reason to trust Cloudflare and Google over a nobody service since their motives with their free DNS services (including DoH and DNS over TLS) relate more to having up-to-date synchronization, load balancing, etc.. with the servers they host themselves. Spying on users via DNS queries by Cloudflare or Google is kind of a joke that can bait only the ignorant. Cloudflare terminates TLS connections and even issues TLS certs and can "theoretically" see every sensitive data that can compromise both businesses and users and yet they are trusted by countless public companies and startups alike. The same thing if not more for Google for their own ervices or their GCP business.