3 ms·

Exactly. robots.txt is useless and those that think its useful for preventing unwanted crawling are clueless

• bdcravens 2y ago
  It's a "Keep off the grass" sign. The polite will obey, the ones who are truly the problem will not.
  • immibis 2y ago
    So your site ends up not being findable by Google, and missing from the Wayback Machine once it's dead, but it does nothing for LLMs.