2 ms·

Because - according to the developer - respecting robots.txt is unethical. His contention is that denying content to AI tools deprives people of their right to

• kordlessagain 4y ago
  It’s a straw man argument, which gives you a good idea inside the psyche of the dev. If anything picks up a URL and uses it later, that is definitely a web crawler.