4 ms·

I wonder what the (legitimate?) reason is for them to spoof. Seems intentionally shady. Maybe there's a legit reason we're missing?

by rapind 6y ago

detaro 6y ago
Possibly trying to avoid people sending them a different version of the page than users would see (of course they could change the page after the initial caching of a preview, but Twitter might refresh/check them later). Also, you often need an impressive amount of the stuff thats in a normal UA string for random sites to not break/send you the "unsupported browser, please use Netscape 4 or newer!!!" page/..., although you normally can fit an identifier of what you really are at the end. (As an example, here's an iOS Safari user agent: "Mozilla/5.0 (iPhone; CPU iPhone OS 13_3_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.5 Mobile/15E148 Safari/604.1" - it's not Mozilla, it's not Gecko, but it has those keywords and patterns because sites expect to see that)
- rapind 6y ago
  That makes sense. I couldn't come up with a shady reason why they would do it to be honest, but I was curious.
- yjftsjthsd-h 6y ago
  Yeah, I once tried to tell my browser to send... I forget; either no UA, or a blank UA string, or nonsense or just "Firefox" or something. I figured, "hey, some sites might break, but it can't be that important!" It broke everything. IIRC, the breaking point was that my own server refused to talk to me. Now, I still think this is insane, but apparently this really is how it is right now.
  - bzb3 6y ago
    You removed a vital part of the http protocol and you're surprised when things break?
    - zucker42 6y ago
      Seems like it follows the spec to me: https://tools.ietf.org/html/rfc7231#section-5.5.3 https://tools.ietf.org/html/rfc7231#section-5.5.3
  - hombre_fatal 6y ago
    Should make you realize just how much abuse there is on the internet that it's worth it to just filter traffic with no UA. Usually people get stuck on the fact that we can't have nice things, so X sucks for not letting us have nice things, yet I seem to never see people acknowledge why we can't have nice things. Then I'd see a lot more "ugh, bad actors suck!" and less "ugh, websites are just trying to make life miserable for me >:("
    - licebmi__at__ 6y ago
      Well, if filtering for UA really makes things difficult for bad actors, they do suck but more in as a technical opinion than an moral statement.
    - lkrubner 6y ago
      Is there an official error message for that? Because filtering no UA would trip me up every time I use wget. Most of the time, if I'm casually using wget for something, I don't bother with a UA. If sites started rejecting that, I'd like to get a clear error message, so I would not go crazy trying to figure out what the problem was. If I got a clear message "send a UA" then I would probably started wrapping my wget requests in a short bash script with some extra stuff thrown in to keep everyone happy. But I'd have to know what it is that is needed to keep everyone happy.
      - auscompgeek 6y ago
        wget has a default User-Agent string.
  - laughinghan 6y ago
    That's amazing. Any idea what piece of middleware on your own server was doing that?
    - yjftsjthsd-h 6y ago
      I don't remember, but what really got me was that I wasn't running anything that I expected to do fancy filtering; I think this was just Apache httpd running on CentOS. But there was no web application firewall, no load balancers, pretty sure fail2ban was only set up for sshd. It at least appeared that just the apache stock config was in play.
- xg15 6y ago
  Obligatory link: https://webaim.org/blog/user-agent-string-history/ https://webaim.org/blog/user-agent-string-history/
technion 6y ago
When you know your target runs Internet Explorer, you serve phishing pages to IE users and some boring content to other users, at the server level. We've had this keep our user tests out of Google Safebrowse and so on. I'm sure similar tricks end up applied to Facebook's UA and "legitimate" marketing sites.

I wonder what the (legitimate?) reason is for them to spoof. Seems intentionally shady. Maybe there's a legit reason we're missing?

by rapind 6y ago

I wonder what the (legitimate?) reason is for them to spoof. Seems intentionally shady. Maybe there's a legit reason we're missing?

detaro 6y ago

Possibly trying to avoid people sending them a different version of the page than users would see (of course they could change the page after the initial caching of a preview, but Twitter might refresh/check them later). Also, you often need an impressive amount of the stuff thats in a normal UA string for random sites to not break/send you the "unsupported browser, please use Netscape 4 or newer!!!" page/..., although you normally can fit an identifier of what you really are at the end. (As an example, here's an iOS Safari user agent: "Mozilla/5.0 (iPhone; CPU iPhone OS 13_3_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.5 Mobile/15E148 Safari/604.1" - it's not Mozilla, it's not Gecko, but it has those keywords and patterns because sites expect to see that)

rapind 6y ago
That makes sense. I couldn't come up with a shady reason why they would do it to be honest, but I was curious.
yjftsjthsd-h 6y ago
Yeah, I once tried to tell my browser to send... I forget; either no UA, or a blank UA string, or nonsense or just "Firefox" or something. I figured, "hey, some sites might break, but it can't be that important!" It broke everything. IIRC, the breaking point was that my own server refused to talk to me. Now, I still think this is insane, but apparently this really is how it is right now.
- bzb3 6y ago
  You removed a vital part of the http protocol and you're surprised when things break?
  - zucker42 6y ago
    Seems like it follows the spec to me: https://tools.ietf.org/html/rfc7231#section-5.5.3 https://tools.ietf.org/html/rfc7231#section-5.5.3
- hombre_fatal 6y ago
  Should make you realize just how much abuse there is on the internet that it's worth it to just filter traffic with no UA. Usually people get stuck on the fact that we can't have nice things, so X sucks for not letting us have nice things, yet I seem to never see people acknowledge why we can't have nice things. Then I'd see a lot more "ugh, bad actors suck!" and less "ugh, websites are just trying to make life miserable for me >:("
  - licebmi__at__ 6y ago
    Well, if filtering for UA really makes things difficult for bad actors, they do suck but more in as a technical opinion than an moral statement.
  - lkrubner 6y ago
    Is there an official error message for that? Because filtering no UA would trip me up every time I use wget. Most of the time, if I'm casually using wget for something, I don't bother with a UA. If sites started rejecting that, I'd like to get a clear error message, so I would not go crazy trying to figure out what the problem was. If I got a clear message "send a UA" then I would probably started wrapping my wget requests in a short bash script with some extra stuff thrown in to keep everyone happy. But I'd have to know what it is that is needed to keep everyone happy.
    - auscompgeek 6y ago
      wget has a default User-Agent string.
- laughinghan 6y ago
  That's amazing. Any idea what piece of middleware on your own server was doing that?
  - yjftsjthsd-h 6y ago
    I don't remember, but what really got me was that I wasn't running anything that I expected to do fancy filtering; I think this was just Apache httpd running on CentOS. But there was no web application firewall, no load balancers, pretty sure fail2ban was only set up for sshd. It at least appeared that just the apache stock config was in play.
xg15 6y ago
Obligatory link: https://webaim.org/blog/user-agent-string-history/ https://webaim.org/blog/user-agent-string-history/

technion 6y ago

When you know your target runs Internet Explorer, you serve phishing pages to IE users and some boring content to other users, at the server level. We've had this keep our user tests out of Google Safebrowse and so on. I'm sure similar tricks end up applied to Facebook's UA and "legitimate" marketing sites.