3 ms·
It is direct financial damage if my servers not on an unmetered connection — after years of bills coming in around $3/mo I got a surprise >$800 bill on a site n
by razingeden 6mo ago
It is direct financial damage if my servers not on an unmetered connection — after years of bills coming in around $3/mo I got a surprise >$800 bill on a site nobody on earth appears to care about besides AI scrapers.
It hasn’t even been updated in years so hell if I know why it needs to be fetched constantly and aggressively, - but fuck every single one of these companies now whining about bots scraping and victimizing them, here’s my violin.
- gzread 6mo agoIf you can identify the scraper you should have a valid legal case to recover damages.
- thisislife2 6mo agoOnly if they had a robots.txt for their site.
- razingeden 6mo agoI hadn’t even considered that. Don’t know why that comment is greyed out or downvoted. It’s a static site that hasn’t been updated since 2016—- so it’s .. since been moved to cloudflare r2 where it’s getting a $0.00 bill, and it now has a disallow / directive. I’m not sure if it’s being obeyed because the cf dash still says it’s getting 700-1300 hits a day even with all the anti bot, “cf managed robots” stuff for ai crawlers in there. The content is so dry and irrelevant I just can’t even fathom 1/100th of that being legitimate human interest but I thought these things just vacuumed up and stole everyone’s content instead of nailing their pages constantly?
- gzread 6mo agoNo, it's still illegal to DDoS sites that don't have robots.txt.
- thisislife2 6mo agoYou are right, I hadn't considered that aspect.