7 ms·
> Most users aren't spoofing their user agent headers to be a different operating system. The people behind the LLM behind this blog post are. They're trying t
by jeroenhd 3mo ago
> Most users aren't spoofing their user agent headers to be a different operating system.
The people behind the LLM behind this blog post are. They're trying to pretend their robots are people to sell other websites' data to their customer. It's easier to pass bot detection gates if you pretend to be a physical machine running Windows or macOS than if you honestly admit you're using Linux on a VM.
- reactordev 3mo agoThe Internet is a cesspool of scams now
- d1ss0nanz 3mo agoAlways was.
- pocksuppet 3mo agoscraping, however, is not intrinsically a scam.
- mplewis 3mo agoIt is when you're doing it like the LLM companies are: at scale, to the degree that you're taking down my site, without my consent by masking your user-agent, for the purpose of stealing data I didn't authorize you to have.
- lsaferite 3mo agoUnfortunately the response to that right now is to nuke everything that isn't within a strict set of constraints. That helps with the bad bulk-scrapers, but hits everyone else as well as collateral damage.
- VorpalWay 3mo agoSomething like Anubis in front of the server to protect it might be an option. It sucks that we have to resort to that yes, but it seems the least bad option currently (better than the entire internet going through Cloudflare at least).
- flexagoon 3mo agoYou can just use one line of JS instead https://fxgn.dev/blog/anubis https://fxgn.dev/blog/anubis
- userbinator 3mo agoThe idiotic "stealing" argument again? At least use "piracy" if you want to be correct... The moment you openly publish information on the Internet, you have already given consent. There are other solutions to bandwidth usage. User-agent discrimination should be illegal. All it does is further the control that Big Tech has, and help authoritarian governments with their control too.
- pdpi 3mo agoAs with anything IP-related, the data is licensed under whatever terms you choose to impose. That’s the extent of the consent you’ve provided.
- Asraelite 3mo agoPermission to access is not the same as permission to use.
- NetMageSCW 3mo agoReading is using.
- Asraelite 3mo agoNo it's not. There is a legal distinction in IP law between the act of observing information and the act of using that information for some kind of personal gain.
- matheusmoreira 3mo ago> User-agent discrimination should be illegal. Couldn't agree more. Nonsense like remote attestation too. If this discrimination persists, it will lead to the destruction of what little computing freedom we still have.
- Bender 3mo agoI documented some crude methods that can stop most of that without a CDN. [1] There will be some false positives so I guess it depends on ones priorities which methods if any to implement or test on a throw-away test site. Not perfect, nothing is. I am watching hundreds of bots sending SYN's and the daemons are oblivious to them. The only method I have not played around with yet is #7 ssl fingerprinting. There are additional methods I chose not to document such as limiting access to logged in accounts that require double-opting-in to acceptable use policies and terms of use, not that most scrapers would give a toss. That it too much whack-a-mole for me personally. That method requires progressively adding friction to account creation and that comes with some pros and cons. [1] - https://nochan.net/b/Internet-Crap/20260606-How-To-Block-Some-Of-The-Bots/ https://nochan.net/b/Internet-Crap/20260606-How-To-Block-Som...
- nradov 3mo agoWhat data is being stolen? Are you referring to copyright violation or something else? If you don't want LLM companies to scrape a site then just restrict access to authorized users. Simple.
- deleted 3mo ago[deleted]
- kdheiwns 3mo agoThis is the same mindset of criminal in a low trust society btw "If you didn't want me to do this, you should had a fence/cameras/security guards. You shouldn't have dressed like that. You shouldn't have put your phone in that pocket." Excusing trillion dollar corporations like low level criminals is embarrassing. Society shouldn't have to lock itself up because bad actors are spreading everywhere. The bad actors should just be removed.
- nradov 3mo agoRelax buddy, it's just HTTP requests. They're not stealing your phone.
- reactordev 3mo agoyou never heard of HTTP request smuggling or XSS then... Let alone being able to fingerprint and dump all your browser history and session data, cookies, etc.
- cindyllm 3mo ago[dead]
- jeroenhd 3mo agoThat's what's being done, isn't it? Fingerprinting technologies and proof of work are trying to authorize people and block bots. Unfortunately, companies like these are trying to bypass the authorization controls. If these scraping companies would respect some kind of X-Scraping-Permitted header, this whole process would be a lot better, but that's not going to happen in an industry that has already normalized using botnets.
- parasense 3mo agoI think the key word was "intrinsically". It's like public photography, it's intrinsically legal, except when it's a Flock camera and then it's suddenly an invasion of privacy.
- nativeit 3mo agoA Flock camera isn’t an invasion of privacy. Fleets of hundreds or thousands of Flock cameras feeding into a massive dataset is definitely an invasion of privacy, and hardly sudden.
- tadfisher 3mo agoMore precisely: pinpointing you at a given place and time does not need a warrant. Collecting many such pinpoints, allowing authorities to reconstruct your exact movements at some point in the future (without needing to flag you or issue a BOLO or whatever), does.
- matheusmoreira 3mo ago> at scale, to the degree that you're taking down my site Fair. Scrapers should be polite and do their utmost to consume the smallest possible amount of resources. > without my consent by masking your user-agent Your consent is not required. It's my user agent. I set it to whatever I want. > for the purpose of stealing data I didn't authorize you to have Data can't be "stolen", only copied. You set up an HTTP server that literally sends people the data when they request it. Don't do that if you don't want people to have the data. Secrecy is the only possible defense.
- lixtra 3mo ago> You set up an HTTP server that literally sends people the data when they request it. Don't do that if you don't want people to have the data. By the same argument I could say: If I send you an exploit and you execute it, don’t complain that your setup fell for it. Just don’t download and run random data from the internet. In reality there’s a consent and expectation beyond the pure technicals.
- matheusmoreira 3mo ago> Just don’t download and run random data from the internet. I don't. I go out of my way to filter everything. Scraping is but one of the tools I use to do it. I want just the data that I actually care about, not people's javascripted hot mess websites full of malware-vectoring ads, fingerprinting and tracking. I don't let my computers talk to strangers either. My servers don't respond to just anyone, they only reply to me, and only after I've cryptographically authenticated. When others try to talk to them it's like they're not even there. But people want their computers to talk to strangers, don't they? They want to serve pages and pages of ads to massive audiences. Unlike your exploitation example, nobody's actively invading their computers and exfiltrating data. Breaking into someone else's computers and dumping their private databases is one thing. We're just requesting the exact same data that they're more than happy to send out to literally anyone who shows up with a browser, through the exact same channels even. So I really have no sympathy.
- pocksuppet 3mo ago
- pocksuppet 3mo agoThat isn't the type of scraping being discussed here.
- duped 3mo agoNo it's just intrinsically antisocial behavior
- GaryBluto 3mo agoScraping has never "intrinsically" been anti-social behaviour, you just don't like it.
- Mistletoe 3mo agoHas the internet made the average human’s life better? As someone that lived in both eras I have to say no. If you want to improve humanity greatly and prevent currently 8.3 billion people’s misery and counting, you build a time machine and prevent Tim Berners-Lee from inventing it in the first place. But it wouldn’t help, because someone else would still invent this torture nexus of hyperconnected human minds.
- matheusmoreira 3mo agoThe internet was amazing before the big techs started centralizing everything, before governments started fragmenting it into censored regional networks...
- NetMageSCW 3mo agoAsk those who had no medical assistance before the Internet. Or ask Ukraine.
- acters 3mo agoIt's sometimes easier to lie than to tell the truth, and being on Linux telling the truth gets me more scrutiny than those pretending to be legit.
- genxy 3mo agoSame as sites the block "wget" as a user agent, but then you can pass in a -U "Cheetos/10" and sail right in. I'd be fine if they throttled the connection, or a contact page. Every little bit helps build a high trust society.
- baby_souffle 3mo agoSome scrape activity happens because I can't obtain the data any other way. I would be thrilled if certain retailers had price and availability data as an API so I could not bother with the bulk of scrape and process.
- greggsy 3mo agoAs much as I would like pricing APIs as well, the economics don’t add up for retail, and it would effectively lead to a scalper’s market. A regular Joe consumer shouldn’t have to learn python to get thier foot in the door because some arbitrage purchaser has automated a ‘best deal’ purchasing bot based on real time pricing APIs.
- nradov 3mo agoThe HTTP User-Agent header was a mistake from the beginning. There is no legitimate need for the server to know what software the client is (or claims to be) running.
- parasense 3mo agoIt certainly wasn't a mistake in the beginning, but it's certainly a mistake now.
- thaumasiotes 3mo agoIt was a mistake in the beginning too. If you want to serve different content to different people, you use two different URLs. There's never a valid case for doing it while hiding it from the user.
- sroussey 3mo agoNo, that misses the whole point of content negotiation (for language, et al). In the age of search engines though, different urls are often perferable.
- thaumasiotes 3mo ago> No, that misses the whole point of content negotiation (for language, et al). There's been some pushing for browsers to automatically decide what language you want to view a webpage in, and that is even more insane than the User-Agent header is. The correct solution for content that exists in multiple languages is to make each language available at its own URL. Just like I already said. Why in the world would you want to lose the option to select the language you want?
- esrauch 3mo agoI feel like this is with 2026 view where browsers are so mutually compatible. In the bad old days there were so many differences between html, css and js behaviors that if you wanted your site to be nice you had to change it for the browser. The way css padding worked wasn't even the same. Feature detection was rarely viable for any of this. No user agent would probably have only entrenched IE6 dominance even more by blocking you from deliberately making a site that works at all on other browsers (including IE7 for that matter)
- vlovich123 3mo agoThis tanh approach won’t tell you if you’re in a VM - it’ll report the guest OS
- zx8080 3mo agoInfinite captcha welcomes those who are using FF and linux on stackoverflow. Easier to skip its links in search results than waste time solving it (captcha never finishes).
- preg_match 3mo agoI have a user-agent switching extension, and I use it because some websites like to claim up and down "ohhh we don't work on Firefox!" but then you switch to chrome's user-agent and - surprise! - no bugs at all. I mean, I get it, saying you support something is technically a commitment. But my goal is to basically never open a chromium browser, so, I'm gonna do what I have to do. And you would be shocked, in the past 5 years I have only encountered ONE website which legitimately required a chromium browser. grapheneos.com, for the web usb installer.