6 ms·

UA strings need to die a horrible horrible death. "Everyone" should just agree on a standard string and completely stop updating them with anything else (and ev

UA strings need to die a horrible horrible death. "Everyone" should just agree on a standard string and completely stop updating them with anything else (and eventually remove the standard string, in theory). Feature detection is a major step forward HOWEVER sometimes a feature is "supported" but has different incompatible behaviour between browser A and browser B. I feel like that is less common today than it was even four years ago thanks in part to a better working relationship between browser vendors but also due to more comprehensive test suites and people spending time testing new functionality. If they absolutely insist on keeping UA strings then let's wrap them in a Javascript structure. Parsing them is insanely convoluted at this point (which is why many just use .Contains() which has its own issues). Just have e.g. UA.Agent (.Name, .Version, .OS) and UA.CompatibleWith[] (.Name, .Version, .OS).

toast0 12y ago
UA strings are increasingly atrocious, but they're still very useful for targeting platforms and reducing the amount of content sent. Eg, if an iPhone browser comes to my site, the FAQ shouldn't show s40 content by default. And when an s40 user comes, the download section shouldn't link to iTunes (and their browser probably doesn't support JavaScript in a meaningful way) It's also helpful to put a splash page up when a specific browser release breaks your site, and users need to down or upgrade. (users who tweak their UA may be wrong targeted in this case, but whatever)
untog 12y ago
Agreed. I'd love to use feature detection for everything, but I just can't - particularly on mobile. For instance, the iPhone has a weird caching bug with audio files - there's no feature detection for it, my only option is to sniff what browser we're on and change behaviour. Edge cases like this aren't going to disappear any time soon.
userbinator 12y ago
Absolutely. I use a filtering proxy which among other things enables modifying request headers, and I've experimented with a ton of different UA headers over the years; and it's astounding how many sites will fail in mysterious ways if your UA doesn't match one of the ones they're expecting (perhaps this has decreased a bit now with all the diversity in browsers.) In particular, I've noticed that having absolutely no UA header produces the most horrible results: "500 Internal Server Error"s, "we think you are a bot so you have been denied access" (403s are common), and even IP bans. UA strings were intended only as a form of branding/advisory, but I think they're now being used in an unfairly discriminatory way. Denying access because of a "foreign" UA is almost like racial profiling.
- sciurus 12y ago
  I don't think your analogy works. Racial profiling is when law enforcement decides to target someone someone based on their ethnicity (an innate characteristic) instead of their behavior. Whether or not an HTTP client sends a user-agent header is a part of its behavior when making a request. Its fair to observe that along with other aspects of the clients behavior when deciding how to handle it. You're probably being blocked by sites running WAFs like ModSecurity. There's a lot of crap out there, and I don't blame people for setting a low threshold for blocking unusual requests. Here's the rules which would catch a missing user-agent: https://github.com/SpiderLabs/owasp-modsecurity-crs/blob/master/base_rules/modsecurity_crs_21_protocol_anomalies.conf https://github.com/SpiderLabs/owasp-modsecurity-crs/blob/mas...
tacostakohashi 12y ago
So, all the people that agree on the new "standard string" update their browsers to send that that, and find themselves unable to use all the websites that are still checking for some variant of "Mozilla/..." because their administrators haven't updated them yet (i.e., the entire internet). Nice idea.
- UnoriginalGuy 12y ago
  That would be the standard string. It would just be an ugly legacy string that tries to be all classic UA strings. You just stop updating it past that. The main goal isn't to break backwards compatibility, it is to make the point that there won't be FORWARDS compatibility. That UA string is static forever.
  - tacostakohashi 12y ago
    That's what "Mozilla/..." already is and always has been, and it obviously doesn't work. All you need is for one browser manufacturer to break the rule and advertise their new feature via something new at the end, and then everyone else who continues to play by the "standard string" rules suffers. Don't get me wrong, I am not advocating UA strings - but to get rid of them, there needs to be an approach where there isn't a punishment for following the rules.
    - cpeterso 12y ago
      Even removing"Mozilla/" could break UA detection that uses hardcoded string indexes to version numberz. :(
      - anonymfus 12y ago
        Opera since 9.0 up to 12.x had user-agent without Mozilla. I still use Opera 12.17 and mostly only Google's sites are broken.
        cpeterso 12y ago
101914 12y ago
"UA strings need to die... just agree on a standard string... and eventually remove the standard string"
lambda 12y ago
So, the one main use case that can't be handled by feature detection or a JavaScript API is returning a mobile or desktop site without extra round-trips. If a server can detect that you have a mobile user-agent, it can directly deliver you an optimized mobile site, that avoids sending lots of extra data that is only relevant for the desktop version, and avoids extra slow round-trips to fetch data later once JavaScript based detection has had a chance to run. UA strings are awful, but I haven't seen any other good solution for this problem.
- TazeTSchnitzel 12y ago
  Why are desktop and mobile sites so different? Why do they need to be different? Can't we just make responsive designs? Having separate desktop and mobile sites is an anti-pattern.
  - qq66 12y ago
    A phone-optimized site will have less content, usually, so why send the extra data to phone that don't need it, especially when those phones are running at lower bandwidth and are often billed per gigabyte?
    - TazeTSchnitzel 12y ago
      I usually find it frustrating when a phone site doesn't let me see what the desktop site does. If there's content only for one form-factor, load it with AJAX.
  - dubcanada 12y ago
    I disagree, there are instances where having a completely different website for a smaller form factor is beneficial to the user.
    - mcovey 12y ago
      It is! And I wish more sites would make it accessible like it used to be by just going to m.domain.tld rather than user-agent sniffing, because I often prefer the simpler mobile version I get when I spoof my user agent from the desktop.
cbr 12y ago
I work on PageSpeed, which makes heavy use of UserAgent string parsing. I would love to stop using the UA and just leave everything up to the browser, but you can't do that without slowing things down. They key problem is we need to know on the server at request time what features the browser supports. For example, only some browsers support WebP images. We'd like to send WebP to browsers that support it, and JPEG to the rest. If you wanted to do it in the browser you'd have something like: <img src=""> <script> if (SupportsWeb()) { img.src = "/image.webp"; } else { img.src = "/image.jpg"; } </script> But this is slow. Why? Because the lookahead parser can't find the image url, so the download can't start until the regular parser gets there and has run the javascript. Instead we want to issue a simple: <img src="/image.webp"> to supporting browsers and <img src="/image.jpg"> otherwise. You would think this would be what the "Accept:" header would be for. Browsers that support WebP would send "Accept: image/webp". And this almost, almost works. But there are problems: Opera started sending the header when it supported lossy WebPs but before it added support for WebP lossless, Chrome v36 on iOS sent the header but didn't support WebP in data urls [1], etc. WebP support is just an example; this applies to many of the optimizations we want to make that are dependent on browser support. If we want to be fast we need to maintain a mapping from UserAgent to supported features. Doesn't mean I like it, and yes it's ugly, but we do need to keep doing this. (The gory details: https://code.google.com/p/modpagespeed/source/browse/trunk/src/pagespeed/kernel/http/user_agent_matcher.cc https://code.google.com/p/modpagespeed/source/browse/trunk/s...) [1] http://crbug.com/402514 http://crbug.com/402514
- notatoad 12y ago
  The problem seems to be that everybody wants to avoid breaking anything. You should just serve webp to browsers that send an Accept:webp, and if those browsers get broken images that's their problem, and not the rest of the internet's problem. Website operators keep compensating for the brokenness of browsers, and then the browser vendors have no motivation to fix their shit.
  - cbr 12y ago
    People screw up. The people working on Chrome for iOS didn't mean to break WebP support, and when they discovered this they fixed it in the next revision. They're already plenty motivated and on top of things. It doesn't make sense to punish visitors for the mistakes of browser makers, even if browser makers need more pressure, which they don't. (And the situation with Opera is even trickier. They did support basic WebP and no one was even doing Accept-based detection yet. More communication would have been helpful between Chrome and Opera to coordinate what "Accept: webp" would mean, but it's not a situation punishment would improve.)
Bahamut 12y ago
The real problem is that IE & Safari doesn't update frequently enough, and so sometimes we have to create hacks that target certain browsers - it's unfortunate, but those browser vendors need to do a better job moving to a continually updating browser model.
- hirsin 12y ago
  IE updates every patch Tuesday. I know that in F12 tools we've shipped new features on an arbitrary Tuesday.
crabasa 12y ago
I agree with cbr, there are too many cases where feature detection either isn't possible or hinders the performance of the application. Does anyone know if there is an authoritative database of user agent strings, their associated browsers and the features of those browsers? If this resource existed, it would be trivial to wrap it in an API that web developers could use in their apps.

toast0 12y ago

UA strings are increasingly atrocious, but they're still very useful for targeting platforms and reducing the amount of content sent. Eg, if an iPhone browser comes to my site, the FAQ shouldn't show s40 content by default. And when an s40 user comes, the download section shouldn't link to iTunes (and their browser probably doesn't support JavaScript in a meaningful way) It's also helpful to put a splash page up when a specific browser release breaks your site, and users need to down or upgrade. (users who tweak their UA may be wrong targeted in this case, but whatever)

untog 12y ago

Agreed. I'd love to use feature detection for everything, but I just can't - particularly on mobile. For instance, the iPhone has a weird caching bug with audio files - there's no feature detection for it, my only option is to sniff what browser we're on and change behaviour. Edge cases like this aren't going to disappear any time soon.

userbinator 12y ago

Absolutely. I use a filtering proxy which among other things enables modifying request headers, and I've experimented with a ton of different UA headers over the years; and it's astounding how many sites will fail in mysterious ways if your UA doesn't match one of the ones they're expecting (perhaps this has decreased a bit now with all the diversity in browsers.) In particular, I've noticed that having absolutely no UA header produces the most horrible results: "500 Internal Server Error"s, "we think you are a bot so you have been denied access" (403s are common), and even IP bans. UA strings were intended only as a form of branding/advisory, but I think they're now being used in an unfairly discriminatory way. Denying access because of a "foreign" UA is almost like racial profiling.

sciurus 12y ago
I don't think your analogy works. Racial profiling is when law enforcement decides to target someone someone based on their ethnicity (an innate characteristic) instead of their behavior. Whether or not an HTTP client sends a user-agent header is a part of its behavior when making a request. Its fair to observe that along with other aspects of the clients behavior when deciding how to handle it. You're probably being blocked by sites running WAFs like ModSecurity. There's a lot of crap out there, and I don't blame people for setting a low threshold for blocking unusual requests. Here's the rules which would catch a missing user-agent: https://github.com/SpiderLabs/owasp-modsecurity-crs/blob/master/base_rules/modsecurity_crs_21_protocol_anomalies.conf https://github.com/SpiderLabs/owasp-modsecurity-crs/blob/mas...

tacostakohashi 12y ago

So, all the people that agree on the new "standard string" update their browsers to send that that, and find themselves unable to use all the websites that are still checking for some variant of "Mozilla/..." because their administrators haven't updated them yet (i.e., the entire internet). Nice idea.

UnoriginalGuy 12y ago
That would be the standard string. It would just be an ugly legacy string that tries to be all classic UA strings. You just stop updating it past that. The main goal isn't to break backwards compatibility, it is to make the point that there won't be FORWARDS compatibility. That UA string is static forever.
- tacostakohashi 12y ago
  That's what "Mozilla/..." already is and always has been, and it obviously doesn't work. All you need is for one browser manufacturer to break the rule and advertise their new feature via something new at the end, and then everyone else who continues to play by the "standard string" rules suffers. Don't get me wrong, I am not advocating UA strings - but to get rid of them, there needs to be an approach where there isn't a punishment for following the rules.
  - cpeterso 12y ago
    Even removing"Mozilla/" could break UA detection that uses hardcoded string indexes to version numberz. :(
    - anonymfus 12y ago
      Opera since 9.0 up to 12.x had user-agent without Mozilla. I still use Opera 12.17 and mostly only Google's sites are broken.
      - cpeterso 12y ago

101914 12y ago

"UA strings need to die... just agree on a standard string... and eventually remove the standard string"

lambda 12y ago

So, the one main use case that can't be handled by feature detection or a JavaScript API is returning a mobile or desktop site without extra round-trips. If a server can detect that you have a mobile user-agent, it can directly deliver you an optimized mobile site, that avoids sending lots of extra data that is only relevant for the desktop version, and avoids extra slow round-trips to fetch data later once JavaScript based detection has had a chance to run. UA strings are awful, but I haven't seen any other good solution for this problem.

TazeTSchnitzel 12y ago
Why are desktop and mobile sites so different? Why do they need to be different? Can't we just make responsive designs? Having separate desktop and mobile sites is an anti-pattern.
- qq66 12y ago
  A phone-optimized site will have less content, usually, so why send the extra data to phone that don't need it, especially when those phones are running at lower bandwidth and are often billed per gigabyte?
  - TazeTSchnitzel 12y ago
    I usually find it frustrating when a phone site doesn't let me see what the desktop site does. If there's content only for one form-factor, load it with AJAX.
- dubcanada 12y ago
  I disagree, there are instances where having a completely different website for a smaller form factor is beneficial to the user.
  - mcovey 12y ago
    It is! And I wish more sites would make it accessible like it used to be by just going to m.domain.tld rather than user-agent sniffing, because I often prefer the simpler mobile version I get when I spoof my user agent from the desktop.

cbr 12y ago

I work on PageSpeed, which makes heavy use of UserAgent string parsing. I would love to stop using the UA and just leave everything up to the browser, but you can't do that without slowing things down. They key problem is we need to know on the server at request time what features the browser supports. For example, only some browsers support WebP images. We'd like to send WebP to browsers that support it, and JPEG to the rest. If you wanted to do it in the browser you'd have something like: <img src=""> <script> if (SupportsWeb()) { img.src = "/image.webp"; } else { img.src = "/image.jpg"; } </script> But this is slow. Why? Because the lookahead parser can't find the image url, so the download can't start until the regular parser gets there and has run the javascript. Instead we want to issue a simple: <img src="/image.webp"> to supporting browsers and <img src="/image.jpg"> otherwise. You would think this would be what the "Accept:" header would be for. Browsers that support WebP would send "Accept: image/webp". And this almost, almost works. But there are problems: Opera started sending the header when it supported lossy WebPs but before it added support for WebP lossless, Chrome v36 on iOS sent the header but didn't support WebP in data urls [1], etc. WebP support is just an example; this applies to many of the optimizations we want to make that are dependent on browser support. If we want to be fast we need to maintain a mapping from UserAgent to supported features. Doesn't mean I like it, and yes it's ugly, but we do need to keep doing this. (The gory details: https://code.google.com/p/modpagespeed/source/browse/trunk/src/pagespeed/kernel/http/user_agent_matcher.cc https://code.google.com/p/modpagespeed/source/browse/trunk/s...) [1] http://crbug.com/402514 http://crbug.com/402514

notatoad 12y ago
The problem seems to be that everybody wants to avoid breaking anything. You should just serve webp to browsers that send an Accept:webp, and if those browsers get broken images that's their problem, and not the rest of the internet's problem. Website operators keep compensating for the brokenness of browsers, and then the browser vendors have no motivation to fix their shit.
- cbr 12y ago
  People screw up. The people working on Chrome for iOS didn't mean to break WebP support, and when they discovered this they fixed it in the next revision. They're already plenty motivated and on top of things. It doesn't make sense to punish visitors for the mistakes of browser makers, even if browser makers need more pressure, which they don't. (And the situation with Opera is even trickier. They did support basic WebP and no one was even doing Accept-based detection yet. More communication would have been helpful between Chrome and Opera to coordinate what "Accept: webp" would mean, but it's not a situation punishment would improve.)

Bahamut 12y ago

The real problem is that IE & Safari doesn't update frequently enough, and so sometimes we have to create hacks that target certain browsers - it's unfortunate, but those browser vendors need to do a better job moving to a continually updating browser model.

hirsin 12y ago
IE updates every patch Tuesday. I know that in F12 tools we've shipped new features on an arbitrary Tuesday.

crabasa 12y ago

I agree with cbr, there are too many cases where feature detection either isn't possible or hinders the performance of the application. Does anyone know if there is an authoritative database of user agent strings, their associated browsers and the features of those browsers? If this resource existed, it would be trivial to wrap it in an API that web developers could use in their apps.