4 ms·
> I am now of the opinion that every form of web-scraping should be considered abusive behaviour and web servers should block all of them. If you think your web
by zahlman 2y ago
> I am now of the opinion that every form of web-scraping should be considered abusive behaviour and web servers should block all of them. If you think your web-scraping is acceptable behaviour, you can thank these shady companies and the “AI” hype for moving you to the bad corner.
I imagine that e.g. Youtube would be happy to agree with this. Not that it would turn them against AI generally.
- BlueTemplar 2y agoYeah, also this means the death of archival efforts like the Internet Archive.
- jeroenhd 2y agoWelcome scrapers (IA, maybe Google and Bing) can publish their IP addresses and get whitelisted. Websites that want to prevent being on the Internet Archive can pretty much just ask for their website to be excluded (even retroactively). [Cloudflare](https://developers.cloudflare.com/cache/troubleshooting/always-online/ https://developers.cloudflare.com/cache/troubleshooting/alwa...) tags the internet archive as operating from 207.241.224.0/20 and 208.70.24.0/21 so disabling the bot-prevention framework on connections from there should be enough.
- trinsic2 2y agoThis sounds like it would be a good idea. Create a whitelist of IPs and block the rest.
- realusername 2y agoThat's basically asking to close the market in favor of the current actors. New actors have the right to emerge.
- 0dayz 2y agoNo they don't. There's no rule that you have to let anyone in who claims to be a web crawler.
- areyourllySorry 2y agowhich is why they will stop claiming to be one.
- chii 2y agoso what happened to competition fostering a better outcome for all then?
- realusername 2y agoSo who decides that you can be one? Right now it's Cloudflare, a litteral monopoly... The truth is that I sympathize with the people trying to use mobile connections to bypass such a cartel. What Cloudflare is doing now is worse than the web crawlers themselves and the legality of blocking crawlers with a monopoly is dubious at best.
- jeroenhd 2y agoThey have the right to try to convince me to let them scrape me. Most of the time they're thinly veiled data traders. I haven't seen any new company try to scrape my stuff since maybe Kagi. Kagi is welcome to scrape from their IP addresses. Other bots that behave are fine too (Huawei and various other Chinese bots don't and I've had to put an IP block on those).
- areyourllySorry 2y agoa large chunk of internet archive's snapshots are from archiveteam, where "warriors" bring their own ips (and they crawl respectfully!). save page now is important too, but you don't realise what is useful until you lose it.
- Centigonal 2y agoyeah, but you can't, that's the problem. Plenty of service operators would like to block every scraper that doesn't obey their robots.txt, but there's no good way to do that without blocking human traffic too (Anubis et al are okay, but they are half-measures). On a separate note, I believe open web scraping has been a massive benefit to the internet on net, and almost entirely positive pre-2021. Web scraping & crawling enables search engines, services like Internet Archive, walled-garden-busting (like Invidious, yt-dlp, and Nitter), mashups (Spotube, IFTT, and Plaid would have been impossible to bootstrap without web scraping), and all kinds of interesting data science projects (e.g. scraping COVID-19 stats from local health departments to patch together a picture of viral spread for epidemiologists).
- udev4096 2y agoWe should have a way to verify the user-agents of the valid and useful scrapers such as Internet Archive by having some kind of cryptographic signature of their user-agents and being able to validate it with any reverse proxy seems like a good start
- lelanthran 2y ago> Plenty of service operators would like to block every scraper that doesn't obey their robots.txt, but there's no good way to do that without blocking human traffic too (Anubis et al are okay, but they are half-measures) Why is Anubis-type mitigations a half-measure?
- 2y ago