4 ms·
As if scraping the entire internet doesn't break thousands of TOSes?
by crote 3mo ago
As if scraping the entire internet doesn't break thousands of TOSes?
- sroussey 3mo agoYes, indeed! I have a site where 99% of the traffic is Claude. I could ban them but I don’t bother.
- SXX 3mo agoBetter to feed them poisoned results...
- Twirrim 3mo agoI poison their requests with crap, particularly because they were the worst offenders for sheer request rate. The site has maybe a hundred pages, and I'd see them hit at a sustained 20 rps for weeks on end, same content being pulled over and over and over again. Easily handled by my small server as it's static content, but the sheer arrogance (inadvertent or otherwise) pissed me off. I'm not sure if they caught on, or just finally started behaving better, but after months of being served crap in my never ending labyrinth of auto-generated junk, they have stopped.
- sroussey 3mo agoMy stuff is on Vercel, so it’s more of an issue as it uses up my limits. I’m moving to Cloudflare and I want to see how changes I make affect their bot, if at all.
- addedGone 3mo agoScrapping internet realistically does involve Botnets of all kind, the scrapping itself might be legal, but it's certain a ton of infected computers are routing traffic and being used as Proxy.