3 ms·
Copyright-ignoring AI scraper bots laugh at robots.txt
- PeterStuer 1y agoHonestly, some sites are so ridiculously malconfigured in their anti-bot zeal that it becomes a Heisenberg like dilemma. E.g. I want to pull in the rss. It is there specifically for m2m. If I dare get the robots.txt, i'm flagged as a bot, and denied the whole site. including not just the rss but even the parts that are not denied per the robots.txt
- maniacwhat 1y agoThe ai companies have shown they don't care at all about the preferences of site owners by ignoring them. I don't see why a new language to express preferences would make any difference here.