4 ms·

The HTTP User-Agent header was a mistake from the beginning. There is no legitimate need for the server to know what software the client is (or claims to be) ru

by nradov 4mo ago

The HTTP User-Agent header was a mistake from the beginning. There is no legitimate need for the server to know what software the client is (or claims to be) running.

parasense 4mo ago
It certainly wasn't a mistake in the beginning, but it's certainly a mistake now.
- thaumasiotes 4mo ago
  It was a mistake in the beginning too. If you want to serve different content to different people, you use two different URLs. There's never a valid case for doing it while hiding it from the user.
  - sroussey 4mo ago
    No, that misses the whole point of content negotiation (for language, et al). In the age of search engines though, different urls are often perferable.
    - thaumasiotes 4mo ago
      > No, that misses the whole point of content negotiation (for language, et al). There's been some pushing for browsers to automatically decide what language you want to view a webpage in, and that is even more insane than the User-Agent header is. The correct solution for content that exists in multiple languages is to make each language available at its own URL. Just like I already said. Why in the world would you want to lose the option to select the language you want?
esrauch 4mo ago
I feel like this is with 2026 view where browsers are so mutually compatible. In the bad old days there were so many differences between html, css and js behaviors that if you wanted your site to be nice you had to change it for the browser. The way css padding worked wasn't even the same. Feature detection was rarely viable for any of this. No user agent would probably have only entrenched IE6 dominance even more by blocking you from deliberately making a site that works at all on other browsers (including IE7 for that matter)
- nradov 4mo ago
  I'm aware of that history and the User-Agent header was a mistake even back then. It took the pressure off of browser vendors and gave them an excuse to not fix their bugs.
  - tyami94 4mo ago
    road to hell is paved with good intentions. one could argue it would've made browser vendors fix their bugs, but with the side-effect that any behavior that isn't bug-compatible with IE becomes a bug. this would've ironically entrenched IE permanently.
    - smaudet 4mo ago
      The mistake was not claiming features rather browser client versions... JS devs were kinda able to patch around the nonsense because they were able to feature-detect - part of the reason this stuck around was because no legitimate user or dev cared (or should care). But the header was mostly (useless) noise, and the people spoofing were dealing with the couple bad apples of the time. Of course, defining features is easier said than done, and a standards body is a challenging environment to define these in... I get why people are fingerprinting bots and others are working around it, but neither are "legitimate" applications - if your content is public, it's public, end of story. And working around these controls to sell botnet access to sites is equally illegitimate - nobody has a right to resell content they do not own...
      - esrauch 4mo ago
        I just can't see how it could make sense to define features when a ton of the the behavior wasn't even intentional but just tons of bugs. I recall debugging issues that only reproduced in "IE7 compatibility mode of IE8" which didn't reproduce in either IE7 or IE8. And that was already after the standards were taken at all seriously
- 14113 4mo ago
  > I feel like this is with 2026 view where browsers are so mutually compatible. I wish this was the case. Unfortunately, companies that work on non-Chromium browsers need to employ dedicated web compatibility teams to either a) help website users fix non-standard (i.e. Chrome only) HTML/CSS/JS, or b) replicate Chromium-like behaviour for specific (very popular) websites so that they work "correctly". There's also the websites that deliberately block certain browsers which is what tools like "chrome-mask"[1] are built to solve. [1] https://addons.mozilla.org/en-GB/firefox/addon/chrome-mask/ https://addons.mozilla.org/en-GB/firefox/addon/chrome-mask/
Ferret7446 4mo ago
There is actually/obviously. It lets you identify bugs in well meaning clients. e.g. you can identify if IE v1.2.3 isn't handling form submissions correctly. It lets you serve content that works for clients with known issues or limitations.
1vuio0pswjnm7 4mo ago
"The HTTP User-Agent header was a mistake form the beginning." User-Agent spoofing was added to NCSA Mosaic in 1996. The public www was three years old and text-only clients were still in widespread use https://raw.githubusercontent.com/alandipert/ncsa-mosaic/master/mosaic-spoof-agents https://raw.githubusercontent.com/alandipert/ncsa-mosaic/mas... NCSA Mosaic is the early graphical browser that begat Netscape Navigator that begat Firefox. Later came Internet Explorer, Safari, Chrome and so on What was the point of spoofing in 1996 Maybe it was just for fun, judging by the examples in the "mosaic-spoof-agents" file As a matter of practice, by default I do not send a user-agent header. I only send the minumum headers required. For me, that's almost always 1-2 for GET and 4 for POST For the vast majority of websites I have accessed,^1 this header minimisation has zero effect on the success of the HTTP request 1. For example, I have used a database of sites submitted to HN as way to test if header minimisation affects HTTP request success Generally I do not use a web browser to make HTTP requests. I perform text processing on the HTML, JSON or whatever is returned, using custom utilities. I store information in SQLite. I read this information as plain text, preferably 7-bit ASCII. I dislike UTF-8 Onlilne debates about "browser fingerprinting" always seem to focus on trying to "blend in", e.g., via "spoofing" As such, because browser continue to get more bloated with "features", online commenters argue in favor of sending more and more data points to servers that can be used to create a fingerprint instead of reducing the amount of data sent Because, according to their reasoning (or lack thereof), sending less data would "stand out" True, but it's generally easier to "spoof" a client that sends less data than one that sends more. And the number of sites that require a user-agent header is still smaller than the number that don't
- Chu4eeno 4mo ago
  I take it you either have a godly IP reputation, or you don't hit 90% of the current internet which is behind cloudflare, google, akamai, etc? They're pretty brutal in blocking non-browsers, I tend to have to add a lot of useless headers.
  - 1vuio0pswjnm7 4mo ago
    According to their /clientrep-lookup database the IP address I am using has no reputation Blocking "non-browsers" and blocking IP addresses are two different things, so to speak The first is based on dumb heuristics and (incorrect) assumptions about behavior based on what software someone is (mistakenly) presumed to be using The second is based on past behaviour I send the minimum HTTP headers I only request what I want; I do not send requests for ads, tracking or telemetry I accept text formats, e.g., HTML, JSON, etc. No need for images, CSS, Javascript, etc.
chrismorgan 4mo ago
When distributing software, it’s very useful to know which OS and which architecture the user is on in order to present them with the appropriate file.
- NetMageSCW 4mo ago
  Or you could let the user decide since you have no idea what system they want to execute the software on, just what system they are using to browse your site.
  - chrismorgan 4mo ago
    The overwhelming majority of the time (typically far higher than 99% of the time), if they hit a download button, they want it for their current platform. And detecting their platform doesn’t stop you from presenting supplemental links for all platforms. In fact I’d say that’s completely normal.

by nradov 4mo ago

The HTTP User-Agent header was a mistake from the beginning. There is no legitimate need for the server to know what software the client is (or claims to be) running.

parasense 4mo ago

It certainly wasn't a mistake in the beginning, but it's certainly a mistake now.

thaumasiotes 4mo ago
It was a mistake in the beginning too. If you want to serve different content to different people, you use two different URLs. There's never a valid case for doing it while hiding it from the user.
- sroussey 4mo ago
  No, that misses the whole point of content negotiation (for language, et al). In the age of search engines though, different urls are often perferable.
  - thaumasiotes 4mo ago
    > No, that misses the whole point of content negotiation (for language, et al). There's been some pushing for browsers to automatically decide what language you want to view a webpage in, and that is even more insane than the User-Agent header is. The correct solution for content that exists in multiple languages is to make each language available at its own URL. Just like I already said. Why in the world would you want to lose the option to select the language you want?

esrauch 4mo ago

I feel like this is with 2026 view where browsers are so mutually compatible. In the bad old days there were so many differences between html, css and js behaviors that if you wanted your site to be nice you had to change it for the browser. The way css padding worked wasn't even the same. Feature detection was rarely viable for any of this. No user agent would probably have only entrenched IE6 dominance even more by blocking you from deliberately making a site that works at all on other browsers (including IE7 for that matter)

nradov 4mo ago
I'm aware of that history and the User-Agent header was a mistake even back then. It took the pressure off of browser vendors and gave them an excuse to not fix their bugs.
- tyami94 4mo ago
  road to hell is paved with good intentions. one could argue it would've made browser vendors fix their bugs, but with the side-effect that any behavior that isn't bug-compatible with IE becomes a bug. this would've ironically entrenched IE permanently.
  - smaudet 4mo ago
    The mistake was not claiming features rather browser client versions... JS devs were kinda able to patch around the nonsense because they were able to feature-detect - part of the reason this stuck around was because no legitimate user or dev cared (or should care). But the header was mostly (useless) noise, and the people spoofing were dealing with the couple bad apples of the time. Of course, defining features is easier said than done, and a standards body is a challenging environment to define these in... I get why people are fingerprinting bots and others are working around it, but neither are "legitimate" applications - if your content is public, it's public, end of story. And working around these controls to sell botnet access to sites is equally illegitimate - nobody has a right to resell content they do not own...
    - esrauch 4mo ago
      I just can't see how it could make sense to define features when a ton of the the behavior wasn't even intentional but just tons of bugs. I recall debugging issues that only reproduced in "IE7 compatibility mode of IE8" which didn't reproduce in either IE7 or IE8. And that was already after the standards were taken at all seriously
14113 4mo ago
> I feel like this is with 2026 view where browsers are so mutually compatible. I wish this was the case. Unfortunately, companies that work on non-Chromium browsers need to employ dedicated web compatibility teams to either a) help website users fix non-standard (i.e. Chrome only) HTML/CSS/JS, or b) replicate Chromium-like behaviour for specific (very popular) websites so that they work "correctly". There's also the websites that deliberately block certain browsers which is what tools like "chrome-mask"[1] are built to solve. [1] https://addons.mozilla.org/en-GB/firefox/addon/chrome-mask/ https://addons.mozilla.org/en-GB/firefox/addon/chrome-mask/

Ferret7446 4mo ago

There is actually/obviously. It lets you identify bugs in well meaning clients. e.g. you can identify if IE v1.2.3 isn't handling form submissions correctly. It lets you serve content that works for clients with known issues or limitations.

1vuio0pswjnm7 4mo ago

"The HTTP User-Agent header was a mistake form the beginning." User-Agent spoofing was added to NCSA Mosaic in 1996. The public www was three years old and text-only clients were still in widespread use https://raw.githubusercontent.com/alandipert/ncsa-mosaic/master/mosaic-spoof-agents https://raw.githubusercontent.com/alandipert/ncsa-mosaic/mas... NCSA Mosaic is the early graphical browser that begat Netscape Navigator that begat Firefox. Later came Internet Explorer, Safari, Chrome and so on What was the point of spoofing in 1996 Maybe it was just for fun, judging by the examples in the "mosaic-spoof-agents" file As a matter of practice, by default I do not send a user-agent header. I only send the minumum headers required. For me, that's almost always 1-2 for GET and 4 for POST For the vast majority of websites I have accessed,^1 this header minimisation has zero effect on the success of the HTTP request 1. For example, I have used a database of sites submitted to HN as way to test if header minimisation affects HTTP request success Generally I do not use a web browser to make HTTP requests. I perform text processing on the HTML, JSON or whatever is returned, using custom utilities. I store information in SQLite. I read this information as plain text, preferably 7-bit ASCII. I dislike UTF-8 Onlilne debates about "browser fingerprinting" always seem to focus on trying to "blend in", e.g., via "spoofing" As such, because browser continue to get more bloated with "features", online commenters argue in favor of sending more and more data points to servers that can be used to create a fingerprint instead of reducing the amount of data sent Because, according to their reasoning (or lack thereof), sending less data would "stand out" True, but it's generally easier to "spoof" a client that sends less data than one that sends more. And the number of sites that require a user-agent header is still smaller than the number that don't

Chu4eeno 4mo ago
I take it you either have a godly IP reputation, or you don't hit 90% of the current internet which is behind cloudflare, google, akamai, etc? They're pretty brutal in blocking non-browsers, I tend to have to add a lot of useless headers.
- 1vuio0pswjnm7 4mo ago
  According to their /clientrep-lookup database the IP address I am using has no reputation Blocking "non-browsers" and blocking IP addresses are two different things, so to speak The first is based on dumb heuristics and (incorrect) assumptions about behavior based on what software someone is (mistakenly) presumed to be using The second is based on past behaviour I send the minimum HTTP headers I only request what I want; I do not send requests for ads, tracking or telemetry I accept text formats, e.g., HTML, JSON, etc. No need for images, CSS, Javascript, etc.

chrismorgan 4mo ago

When distributing software, it’s very useful to know which OS and which architecture the user is on in order to present them with the appropriate file.

NetMageSCW 4mo ago
Or you could let the user decide since you have no idea what system they want to execute the software on, just what system they are using to browse your site.
- chrismorgan 4mo ago
  The overwhelming majority of the time (typically far higher than 99% of the time), if they hit a download button, they want it for their current platform. And detecting their platform doesn’t stop you from presenting supplemental links for all platforms. In fact I’d say that’s completely normal.