4 ms·

That also means his blog won't be indexed and won't be found in search results

• tareqak 4y ago
  That is a good point in the case the blogger still wants their blog to be indexed and found in search results, but does not want their content to be used as ML training data. However, I still think that a program that crawls the web to collect data should respect robots.txt irrespective of what it does with the collected data both during the crawl and afterwards. I would strongly assume that someone who doesn’t want their website to be indexed and found in search results would also likely not want their content to be processed as ML training data since it too reads the content, but I might be wrong here.
• capableweb 4y ago
  You can create a robots.txt that blocks everything but known bots (by name, like googlebot).