> On the other hand, blocking training on published information doesn’t make sense: If you don’t want your stuff to be read, don’t publish it! > This tradeoff

> On the other hand, blocking training on published information doesn’t make sense: If you don’t want your stuff to be read, don’t publish it! > This tradeoff has basically nothing to do with recent advances in AI though. I am surprised someone on HN would think this, especially considering the recent examples of DDoS via LLM crawlers compared to how websites are glad to be crawled by search engines. For the first part : why do you think that robots.txt even exists ? Or why, say, YouTube constantly tries (and fails) to prevent you to use the 'wrong' kind of software to download their videos ?