7 ms·

The long and sordid history of the user-agent HTTP header field

195 points by damian2000 14y ago

JBiserkov 14y ago
Educational and fun! All this could have been avoided if Webmasters used <noframes>, but I'm not sure when it was added to HTML.
- yuhong 14y ago
  In fact frames was introduced in Netscape 2, released after IE2.
w0utert 14y ago
That was interesting and fun to read, never thought about the User-Agent string and why it's so messed up.
y0ghur7_xxx 14y ago
...and everyone pretended to be everyone else, and confusion abounded. And then JavaScript driven feature detection came to be, and everyone thought it was a good idea. And the people wrung their hands and wept
- tomjen3 14y ago
  Not really, because it won't allow you to serve an MS compitable css page.
  - deleted 14y ago
    [deleted]
  - ksherlock 14y ago
    IE conditional comments solve that problem.
- TazeTSchnitzel 14y ago
  What's wrong with it? Unlike user agent sniffing, it works properly.
  - nfm 14y ago
    I think the GP was referring to weeping with joy.
    - TazeTSchnitzel 14y ago
      Oh, in that case, carry on!
    - bluesnowmonkey 14y ago
      People don't wring their hands with joy.
      - yahelc 14y ago
        They wrung their hands of user agent sniffing and wept for joy.
  - sesqu 14y ago
    Oh? So how do I use it to detect which box model to use in my CSS? How do I use it to detect whether I can use gradients? And if you're just thinking about javascript, I've ran into a few text manipulation functions that always returned "not implemented". But hey, sniffer says they're there! edit: Oh, and there's still plenty of html fun, too. How does your browser support <video>? <thead>? <svg>? I'm guessing the answer starts with "javascript".
decklin 14y ago
I wonder why browsers with a modern automatic-update process don't set their user-agent to something that discards all this madness ("Chrome/23.4.5678 (Windows)", or similar) for the cutting-edge/nightly builds only (or even betas, if they wanted to discourage casual users from switching to them, but I don't think that's the case at this point). Surely their users have signed up for a little breakage in exchange for the latest features? And if they actually get website operators to stop or at least fix their sniffing, the whole prisoners-dilemma situation would disappear. (I guess this assumes that the huge user-agent that my Chrome is currently sending is necessarily bad, and in the real world maybe no one really cares...)
- phillmv 14y ago
  1. I think the most interesting thing about that blog post is that it illustrates how the incentives in standards building get warped. I like to describe this sort of thing as "the effect of economics on programming" - not because there is money involved, but because of the nature of the incentives. 2. Graceful degradation. We've sniffed UA's from the minute they were invented. Any change whatsoever would create untold problems for untold millions of people. The UA is just an arbitrary string so… who cares? Very few people (you and I are amongst these "very few") have to be concerned with this compared to the people such a change would affect. It's because of 1 and 2 (my second point is really an instance of the first) that we're stuck with Javascript. No one in their right mind thinks it's a good language, but getting all the different browser vendors to adopt a good bytecode would be nightmarish (and not necessarily in the interest of every browser vendor).
  - potch 14y ago
    I think JS is a good language, but I am probably not in my right mind, so touché.
- MichaelGG 14y ago
  Imagine the problems that would cause! You have Chrome 58 Beta, and stuff works one way. Then they say it's good and release Chrome 58 final, and all of a sudden, stuff changes all over the web. UA string is just one example of unfortunate hacks that evolved in the web protocols. Compared to probably everything else in HTML it's probably just not even worth it to consider fixing it. We'll always need the old string for compatibility, so it's really only to save a few lines of parsing. Compared to the nightmare of parsing rules for HTTP and HTML, it's not even relevant.
mnutt 14y ago
Not only the user agent, either. Try javascript `navigator.appName` in any browser, and you'll get "netscape". `navigator.appCodeName` in most browsers returns "mozilla". Mike Taylor gave a talk about this and more at yesterday's GothamJS conference: http://miketaylr.com/pres/gothamjs/shower/ http://miketaylr.com/pres/gothamjs/shower/
benatkin 14y ago
I wanted to try making an HTTP request from Telnet the other day. I tried Wikipedia, using the Host header. I got a 403 for not including a user agent, so I tried again with User-Agent: Telnet and it worked! It's one of the most important headers for clients, since if you don't include it you might not get a 200.
- JoshTriplett 14y ago
  In the particular case of Wikipedia, I think they check User-Agent to prevent people from unthinkingly wasting gigabytes of bandwidth scraping Wikipedia via tools like wget. In Wikipedia's case, better ways exist to download large quantities of their content in a more usable form.
  - icebraining 14y ago
    They may do that ('though requesting a single article works fine), but it's not very smart. Throttling heavy users - possibly returning 429 with a link to the download pages - would make much more sense. It's not like wget users can't change their UA.
  - cubicle 14y ago
    ? bbot@magnesium:~> wget http://en.wikipedia.org/wiki/Japanese_yen --2012-07-15 13:54:29-- http://en.wikipedia.org/wiki/Japanese_yen Resolving en.wikipedia.org... 208.80.154.225, 2620:0:861:ed1a::1 Connecting to en.wikipedia.org|208.80.154.225|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 203481 (199K) [text/html]
    - verroq 14y ago
      wget uses the "wget/version" useragent.
      - cubicle 14y ago
        Yes, I am aware. The point of my comment is that Wikipedia obviously does not block wget.
cvursache 14y ago
i just read your article after having some austrian wine and absinth73 and it was very funny
- cvursache 14y ago
  actually meant the article was really cool, but that's what i get - downvotes- for using my slim english skills while tipsy (:
AshleysBrain 14y ago
The user agent string is too loaded with backwards compatibility to remove or change. So the next best thing to do is supersede it - add a new agent-id or some such which is mandated by standard to be in the form "BrowserName/Version", e.g. "Chrome/22.0" or "Firefox/15.0.1", while keeping the old user-agent. Problem is I guess it's not really worth it - it doesn't expose any new information not already in the user agent, and it doesn't stop site authors relying on specific agent-ids. So I guess the way forward is try to ignore the user agent completely and just use feature detection.
- riffraff 14y ago
  Aren't there many uses of user agent strings outside of javascript/css usage? E.g. rate limiting, visitor stats etc
  - pserwylo 14y ago
    I enjoy sites which provide me with the correct download link for software, based on the fact Firefox places "Linux x86_64" in my User-Agent. Having said that, the flip side is just as bad, when you get completely rejected from sites because "We haven't tested this site for your browser or operating system". It's a website for crying out loud. I don't get too mad at sites which implement this as long as they provide me a way to continue "at my own risk". However it's the final straw when they flat out refuse to serve me anything other than a page telling me they haven't tested the website for my browser/OS combination... sigh Edit: This might give some food for thought for AshleysBrain. I like your suggestion, but am curious if we can find a way to send OS and architecture to sites so that they can give me nice download links...
    - UnoriginalGuy 14y ago
      A lot of that functionality can be implemented using JS rather than the user-agent string.
  - eli 14y ago
    Pretty much just visitor stats (not that those aren't important!) It doesn't make sense to use User Agent string for anything that someone might have a reason to game since it's so trivial to change them.
FuzzyDunlop 14y ago
The problem with the user agent is that you can't fix it without repeating the same cycle. All you'd do is make it easier. It's a good depiction of the issues you have with trying to write code once, and have it work the same in many different environments, though. It's just with browsers, rather than operating systems or hardware. Such is the evolution of the internet.
joshtynjala 14y ago
This ugly quagmire makes me wary of compatibility fixes where mimicking another browser is somehow involved. When i heard about non-WebKit browsers adopting -webkit CSS vendor prefixes, the user agent string mess was the first thing that came to mind.
buro9 14y ago
The user-agent has two implied pieces of functionality: 1) Describe the device that the agent is coming from (operating system) 2) Describe the capabilities of the agent (this browser, those plugins) One of the things I loathe about the user agent header is the lack of reasonable maximum length, and the inconsistent way in which developers have overloaded the value. Parsing it is difficult (especially given that the length means there is a lot of scope for bad input). I would love to see user agent be a virtual header comprised of other headers. The other headers would not be mandatory, but as most browsers would provide them you could reasonably use them in most cases. These other headers may be things like: os: Windows os-version: 7 client: Gecko client-version: 16 plugins: [{'flash':11}] Basically... same info but more structure with known acceptable types for certain values. Headers taking uncompressed space it would also be helpful if shorthand names were accepted: c-v for client-version, etc. This is me thinking aloud, and perhaps it's an idea that has been thought of before and rejected... but by offering User-Agent as a virtual header that is comprised of all of the other headers you maintain some background compatibility whilst providing something easier to parse, use and trust for developers.
- UnoriginalGuy 14y ago
  The problem with "fixing" the user-agent string is that making it easier to pass/use only means web-developers will find it easier to continue to abuse it. In an ideal world web-developers should be testing if individual pieces of functionality exist rather than inferring what is supported based on the browser. I think JS does a fairly good job of allowing developers to test for functionality, unfortunately CSS does not. I am well aware that it is meant to "fail gracefully" but to a lot of developers they want to supply alternative looks where functionality isn't available and CSS doesn't lend itself to that. So you wind up inferring CSS support from JS support which is just as broken as inferring JS support from the browser's version/name/platform.

195 points by damian2000 14y ago

Educational and fun! All this could have been avoided if Webmasters used <noframes>, but I'm not sure when it was added to HTML.

yuhong 14y ago
In fact frames was introduced in Netscape 2, released after IE2.

w0utert 14y ago

That was interesting and fun to read, never thought about the User-Agent string and why it's so messed up.

y0ghur7_xxx 14y ago

...and everyone pretended to be everyone else, and confusion abounded. And then JavaScript driven feature detection came to be, and everyone thought it was a good idea. And the people wrung their hands and wept

tomjen3 14y ago
Not really, because it won't allow you to serve an MS compitable css page.
- deleted 14y ago
  [deleted]
- ksherlock 14y ago
  IE conditional comments solve that problem.
TazeTSchnitzel 14y ago
What's wrong with it? Unlike user agent sniffing, it works properly.
- nfm 14y ago
  I think the GP was referring to weeping with joy.
  - TazeTSchnitzel 14y ago
    Oh, in that case, carry on!
  - bluesnowmonkey 14y ago
    People don't wring their hands with joy.
    - yahelc 14y ago
      They wrung their hands of user agent sniffing and wept for joy.
- sesqu 14y ago
  Oh? So how do I use it to detect which box model to use in my CSS? How do I use it to detect whether I can use gradients? And if you're just thinking about javascript, I've ran into a few text manipulation functions that always returned "not implemented". But hey, sniffer says they're there! edit: Oh, and there's still plenty of html fun, too. How does your browser support <video>? <thead>? <svg>? I'm guessing the answer starts with "javascript".

decklin 14y ago

I wonder why browsers with a modern automatic-update process don't set their user-agent to something that discards all this madness ("Chrome/23.4.5678 (Windows)", or similar) for the cutting-edge/nightly builds only (or even betas, if they wanted to discourage casual users from switching to them, but I don't think that's the case at this point). Surely their users have signed up for a little breakage in exchange for the latest features? And if they actually get website operators to stop or at least fix their sniffing, the whole prisoners-dilemma situation would disappear. (I guess this assumes that the huge user-agent that my Chrome is currently sending is necessarily bad, and in the real world maybe no one really cares...)

phillmv 14y ago
1. I think the most interesting thing about that blog post is that it illustrates how the incentives in standards building get warped. I like to describe this sort of thing as "the effect of economics on programming" - not because there is money involved, but because of the nature of the incentives. 2. Graceful degradation. We've sniffed UA's from the minute they were invented. Any change whatsoever would create untold problems for untold millions of people. The UA is just an arbitrary string so… who cares? Very few people (you and I are amongst these "very few") have to be concerned with this compared to the people such a change would affect. It's because of 1 and 2 (my second point is really an instance of the first) that we're stuck with Javascript. No one in their right mind thinks it's a good language, but getting all the different browser vendors to adopt a good bytecode would be nightmarish (and not necessarily in the interest of every browser vendor).
- potch 14y ago
  I think JS is a good language, but I am probably not in my right mind, so touché.
MichaelGG 14y ago
Imagine the problems that would cause! You have Chrome 58 Beta, and stuff works one way. Then they say it's good and release Chrome 58 final, and all of a sudden, stuff changes all over the web. UA string is just one example of unfortunate hacks that evolved in the web protocols. Compared to probably everything else in HTML it's probably just not even worth it to consider fixing it. We'll always need the old string for compatibility, so it's really only to save a few lines of parsing. Compared to the nightmare of parsing rules for HTTP and HTML, it's not even relevant.

mnutt 14y ago

Not only the user agent, either. Try javascript `navigator.appName` in any browser, and you'll get "netscape". `navigator.appCodeName` in most browsers returns "mozilla". Mike Taylor gave a talk about this and more at yesterday's GothamJS conference: http://miketaylr.com/pres/gothamjs/shower/ http://miketaylr.com/pres/gothamjs/shower/

benatkin 14y ago

I wanted to try making an HTTP request from Telnet the other day. I tried Wikipedia, using the Host header. I got a 403 for not including a user agent, so I tried again with User-Agent: Telnet and it worked! It's one of the most important headers for clients, since if you don't include it you might not get a 200.

JoshTriplett 14y ago
In the particular case of Wikipedia, I think they check User-Agent to prevent people from unthinkingly wasting gigabytes of bandwidth scraping Wikipedia via tools like wget. In Wikipedia's case, better ways exist to download large quantities of their content in a more usable form.
- icebraining 14y ago
  They may do that ('though requesting a single article works fine), but it's not very smart. Throttling heavy users - possibly returning 429 with a link to the download pages - would make much more sense. It's not like wget users can't change their UA.
- cubicle 14y ago
  ? bbot@magnesium:~> wget http://en.wikipedia.org/wiki/Japanese_yen --2012-07-15 13:54:29-- http://en.wikipedia.org/wiki/Japanese_yen Resolving en.wikipedia.org... 208.80.154.225, 2620:0:861:ed1a::1 Connecting to en.wikipedia.org|208.80.154.225|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 203481 (199K) [text/html]
  - verroq 14y ago
    wget uses the "wget/version" useragent.
    - cubicle 14y ago
      Yes, I am aware. The point of my comment is that Wikipedia obviously does not block wget.

cvursache 14y ago

i just read your article after having some austrian wine and absinth73 and it was very funny

cvursache 14y ago
actually meant the article was really cool, but that's what i get - downvotes- for using my slim english skills while tipsy (:

AshleysBrain 14y ago

The user agent string is too loaded with backwards compatibility to remove or change. So the next best thing to do is supersede it - add a new agent-id or some such which is mandated by standard to be in the form "BrowserName/Version", e.g. "Chrome/22.0" or "Firefox/15.0.1", while keeping the old user-agent. Problem is I guess it's not really worth it - it doesn't expose any new information not already in the user agent, and it doesn't stop site authors relying on specific agent-ids. So I guess the way forward is try to ignore the user agent completely and just use feature detection.

riffraff 14y ago
Aren't there many uses of user agent strings outside of javascript/css usage? E.g. rate limiting, visitor stats etc
- pserwylo 14y ago
  I enjoy sites which provide me with the correct download link for software, based on the fact Firefox places "Linux x86_64" in my User-Agent. Having said that, the flip side is just as bad, when you get completely rejected from sites because "We haven't tested this site for your browser or operating system". It's a website for crying out loud. I don't get too mad at sites which implement this as long as they provide me a way to continue "at my own risk". However it's the final straw when they flat out refuse to serve me anything other than a page telling me they haven't tested the website for my browser/OS combination... sigh Edit: This might give some food for thought for AshleysBrain. I like your suggestion, but am curious if we can find a way to send OS and architecture to sites so that they can give me nice download links...
  - UnoriginalGuy 14y ago
    A lot of that functionality can be implemented using JS rather than the user-agent string.
- eli 14y ago
  Pretty much just visitor stats (not that those aren't important!) It doesn't make sense to use User Agent string for anything that someone might have a reason to game since it's so trivial to change them.

FuzzyDunlop 14y ago

The problem with the user agent is that you can't fix it without repeating the same cycle. All you'd do is make it easier. It's a good depiction of the issues you have with trying to write code once, and have it work the same in many different environments, though. It's just with browsers, rather than operating systems or hardware. Such is the evolution of the internet.

joshtynjala 14y ago

This ugly quagmire makes me wary of compatibility fixes where mimicking another browser is somehow involved. When i heard about non-WebKit browsers adopting -webkit CSS vendor prefixes, the user agent string mess was the first thing that came to mind.

buro9 14y ago

The user-agent has two implied pieces of functionality: 1) Describe the device that the agent is coming from (operating system) 2) Describe the capabilities of the agent (this browser, those plugins) One of the things I loathe about the user agent header is the lack of reasonable maximum length, and the inconsistent way in which developers have overloaded the value. Parsing it is difficult (especially given that the length means there is a lot of scope for bad input). I would love to see user agent be a virtual header comprised of other headers. The other headers would not be mandatory, but as most browsers would provide them you could reasonably use them in most cases. These other headers may be things like: os: Windows os-version: 7 client: Gecko client-version: 16 plugins: [{'flash':11}] Basically... same info but more structure with known acceptable types for certain values. Headers taking uncompressed space it would also be helpful if shorthand names were accepted: c-v for client-version, etc. This is me thinking aloud, and perhaps it's an idea that has been thought of before and rejected... but by offering User-Agent as a virtual header that is comprised of all of the other headers you maintain some background compatibility whilst providing something easier to parse, use and trust for developers.

UnoriginalGuy 14y ago
The problem with "fixing" the user-agent string is that making it easier to pass/use only means web-developers will find it easier to continue to abuse it. In an ideal world web-developers should be testing if individual pieces of functionality exist rather than inferring what is supported based on the browser. I think JS does a fairly good job of allowing developers to test for functionality, unfortunately CSS does not. I am well aware that it is meant to "fail gracefully" but to a lot of developers they want to supply alternative looks where functionality isn't available and CSS doesn't lend itself to that. So you wind up inferring CSS support from JS support which is just as broken as inferring JS support from the browser's version/name/platform.