3 ms·

If you're lying in the requests you send, to trick my server into returning the content you want, instead of what I would want to return to webscrapers, that's

If you're lying in the requests you send, to trick my server into returning the content you want, instead of what I would want to return to webscrapers, that's non-consensual. You don't need my permission to send a GET request, I completely agree. In fact, by having a publicly accessible webserver, there's implied consent that I'm willing to accept reasonable, and valid GET requests. But I have configured my server to spend server resources the way I want, you don't like how my server works, so your configure your bot to lie. If you get what you want only because you're willing to lie, where's the implied consent?

batch12 1y ago
Browser user agents have a history of being lies from the earliest days of usage. Official browsers lied about what they were- and still do.
- jraph 1y ago
  Lies in user agent strings where for bypassing bugs, poor workarounds and assumptions that became wrong, they are nothing like what we are talking about.
  - gkbrk 1y ago
    A server returning HTML for Chrome but not cURL seems like a bug, no? This is why there are so many libraries to make requests that look like they came from browser, to work around buggy servers or server operators with wrong assumptions.
    - grayhatter 1y ago
      > A server returning HTML for Chrome but not cURL seems like a bug, no? tell me you've never heard of https://wttr.in/ https://wttr.in/ without telling me. :P It would absolutely be a bug iff this site returned html to curl. > This is why there are so many libraries to make requests that look like they came from browser, to work around buggy servers or server operators with wrong assumptions. This is a shallow take, the best counter example is how googlebot has no problem identifying it itself both in and out of thue user agent. Do note user agent packing, is distinctly different from a fake user agent selected randomly from the list of most common. The existence of many libraries with the intent to help conceal the truth about a request doesn't feel like proof that's what everyone should be doing. It feels more like proof that most people only want to serve traffic to browsers and real users. And it's the bots and scripts that are the fuckups.
      - batch12 1y ago
        Googlebot has no problem identifying itself because Google knows that you want it to index your site if you want visitors. It doesn't identify itself to give you the option to block it. It identifies itself so you don't.
        grayhatter 1y ago
        I care much less about being indexed by Google as much as you might think. Google bot doesn't get blocked from my server primarily because it's a *very* well behaved bot. It sends a lot of requests, but it's very kind, and has never acted in a way that could overload my server. It respects robots.txt, and identifies itself multiple times. Google bot doesn't get blocked, because it's a well behaved bot that eagerly follows the rules. I wouldn't underestimate how far that goes towards the reason it doesn't get blocked. Much more than the power gained by being google search.
  - batch12 1y ago
    Yes, the client wanted the server to deliver content it had intended for a different client, regardless of what the service operator wanted, so it lied using its user agent. Exact same thing we are talking about. The difference is that people don't want companies to profit off of their content. That's fair. In this case, they should maybe consider some form of real authentication, or if the bot is abusive, some kind of rate limiting control.
    - grayhatter 1y ago
      > Yes, the client wanted the server to deliver content it had intended for a different client, regardless of what the service operator wanted, so it lied using its user agent. I would actually argue, it's not nearly the same type of misconfiguration. The reason scripts, which have never been a browser, who omit their real identity, are doing it, is to evade bot detection. The reason browsers pack their UA with so much legacy data, is because of misconfigured servers. The server owner wants to send data to users and their browsers, but through incompetence, they've made a mistake. Browsers adapted by including extra strings in the UA to account for the expectations of incorrectly configured servers. Extra strings being the critical part, Google bot's UA is an example of this being done correctly.
    - jraph 1y ago
      Add "assumptions that became wrong" to "intended" and the perspective radically changes, to the point that omitting this part from my comment changes everything. I would even add: > the client wanted the server to deliver content it had intended for a different client In most cases, the webmaster intended their work to look good, not really to send different content to different clients. That later part is a technical means, a workaround. The intent of bringing the ok version to the end user was respected… even better with the user agent lies! > The difference is that people don't want companies to profit off of their content. Indeed¹, and also they don't want terrible bot to bring down their servers. 1: well, my open source work explicitly allows people to profit off of it - as long as the license is respected (attribution, copyleft, etc)
- grayhatter 1y ago
  Can you give a single example of a browser with a user agent that lies about it's real origin? The best I can come up with is the TOR browser, which will reduce the number of bits of information it will return, but I dont consider that to be misleading. It's a custom build of firefox, that discloses it is firefox, and otherwise behaves exactly as I would expect firefox to behave.
wqaatwt 1y ago
Somebody concealing or obfuscating various information a browser would send by standard for privacy or other reasons is also “lying” by that standard? Or someone using a VPN?
- grayhatter 1y ago
  Someone using a VPN is not lying. The intent of a user agent is to identify the software sending the request. The IP address isn't sent by the browser, and isn't part of the HTTP request. It's part of the routing information required to deliver the packet back to the client. If a client sent it's "real" IP address as an HTTP header, and I tried to respond to that IP instead of the IP address from the TCP packet. It would never arrive. There's a difference between sending no data, and sending false data. I don't block requests without http referrers for that very reason.
  - wqaatwt 1y ago
    IIRC Firefox (and I assume other browsers) when using privacy/no tracking mode does send fake data..
    - grayhatter 1y ago
      You're incorrect. I've never seen any browser, on it's own lie about it's user agent. (I can set a custom string and lie with it, but that's not the agent doing it) Do you have a specific / concrete example in mind? Or are you mistaking a feature from something other than a mainstream browser?
      - gkbrk 1y ago
        Firefox sends an incorrect version and operating system on its User-Agent when the privacy settings are turned on. IIRC it defaults to a Windows user agent even when you use it on other operating systems.
        grayhatter 1y ago

batch12 1y ago

Browser user agents have a history of being lies from the earliest days of usage. Official browsers lied about what they were- and still do.

jraph 1y ago
Lies in user agent strings where for bypassing bugs, poor workarounds and assumptions that became wrong, they are nothing like what we are talking about.
- gkbrk 1y ago
  A server returning HTML for Chrome but not cURL seems like a bug, no? This is why there are so many libraries to make requests that look like they came from browser, to work around buggy servers or server operators with wrong assumptions.
  - grayhatter 1y ago
    > A server returning HTML for Chrome but not cURL seems like a bug, no? tell me you've never heard of https://wttr.in/ https://wttr.in/ without telling me. :P It would absolutely be a bug iff this site returned html to curl. > This is why there are so many libraries to make requests that look like they came from browser, to work around buggy servers or server operators with wrong assumptions. This is a shallow take, the best counter example is how googlebot has no problem identifying it itself both in and out of thue user agent. Do note user agent packing, is distinctly different from a fake user agent selected randomly from the list of most common. The existence of many libraries with the intent to help conceal the truth about a request doesn't feel like proof that's what everyone should be doing. It feels more like proof that most people only want to serve traffic to browsers and real users. And it's the bots and scripts that are the fuckups.
    - batch12 1y ago
      Googlebot has no problem identifying itself because Google knows that you want it to index your site if you want visitors. It doesn't identify itself to give you the option to block it. It identifies itself so you don't.
      - grayhatter 1y ago
        I care much less about being indexed by Google as much as you might think. Google bot doesn't get blocked from my server primarily because it's a *very* well behaved bot. It sends a lot of requests, but it's very kind, and has never acted in a way that could overload my server. It respects robots.txt, and identifies itself multiple times. Google bot doesn't get blocked, because it's a well behaved bot that eagerly follows the rules. I wouldn't underestimate how far that goes towards the reason it doesn't get blocked. Much more than the power gained by being google search.
- batch12 1y ago
  Yes, the client wanted the server to deliver content it had intended for a different client, regardless of what the service operator wanted, so it lied using its user agent. Exact same thing we are talking about. The difference is that people don't want companies to profit off of their content. That's fair. In this case, they should maybe consider some form of real authentication, or if the bot is abusive, some kind of rate limiting control.
  - grayhatter 1y ago
    > Yes, the client wanted the server to deliver content it had intended for a different client, regardless of what the service operator wanted, so it lied using its user agent. I would actually argue, it's not nearly the same type of misconfiguration. The reason scripts, which have never been a browser, who omit their real identity, are doing it, is to evade bot detection. The reason browsers pack their UA with so much legacy data, is because of misconfigured servers. The server owner wants to send data to users and their browsers, but through incompetence, they've made a mistake. Browsers adapted by including extra strings in the UA to account for the expectations of incorrectly configured servers. Extra strings being the critical part, Google bot's UA is an example of this being done correctly.
  - jraph 1y ago
    Add "assumptions that became wrong" to "intended" and the perspective radically changes, to the point that omitting this part from my comment changes everything. I would even add: > the client wanted the server to deliver content it had intended for a different client In most cases, the webmaster intended their work to look good, not really to send different content to different clients. That later part is a technical means, a workaround. The intent of bringing the ok version to the end user was respected… even better with the user agent lies! > The difference is that people don't want companies to profit off of their content. Indeed¹, and also they don't want terrible bot to bring down their servers. 1: well, my open source work explicitly allows people to profit off of it - as long as the license is respected (attribution, copyleft, etc)
grayhatter 1y ago
Can you give a single example of a browser with a user agent that lies about it's real origin? The best I can come up with is the TOR browser, which will reduce the number of bits of information it will return, but I dont consider that to be misleading. It's a custom build of firefox, that discloses it is firefox, and otherwise behaves exactly as I would expect firefox to behave.

wqaatwt 1y ago

Somebody concealing or obfuscating various information a browser would send by standard for privacy or other reasons is also “lying” by that standard? Or someone using a VPN?

grayhatter 1y ago
Someone using a VPN is not lying. The intent of a user agent is to identify the software sending the request. The IP address isn't sent by the browser, and isn't part of the HTTP request. It's part of the routing information required to deliver the packet back to the client. If a client sent it's "real" IP address as an HTTP header, and I tried to respond to that IP instead of the IP address from the TCP packet. It would never arrive. There's a difference between sending no data, and sending false data. I don't block requests without http referrers for that very reason.
- wqaatwt 1y ago
  IIRC Firefox (and I assume other browsers) when using privacy/no tracking mode does send fake data..
  - grayhatter 1y ago
    You're incorrect. I've never seen any browser, on it's own lie about it's user agent. (I can set a custom string and lie with it, but that's not the agent doing it) Do you have a specific / concrete example in mind? Or are you mistaking a feature from something other than a mainstream browser?
    - gkbrk 1y ago
      Firefox sends an incorrect version and operating system on its User-Agent when the privacy settings are turned on. IIRC it defaults to a Windows user agent even when you use it on other operating systems.
      - grayhatter 1y ago