3 ms·
This is factually false. There's ample documentation of crawlers straight-up ignoring robots.txt. It's not a legal control, but a technical one - and a volunt
by throw10920 1y ago
This is factually false.
There's ample documentation of crawlers straight-up ignoring robots.txt.
It's not a legal control, but a technical one - and a voluntary one, which means that it's trivial to ignore.
And there's obviously nowhere to put a robots.txt for a book that you've published.
- jart 1y agoThe biggest, best, most reputable organizations e.g. Google, Bing, Yahoo, Yandex, Baidu, DuckDuckGo, OpenAI, and Anthropic have all publicly promised to respect your robots.txt file. You can make them hurt if they lie. So you know they're telling the truth. There's some people out there who don't respect robots.txt like Archive Team. However they're more likely to be treated as folk heroes here on Hacker News than trigger AI training fears.