Such as OpenAI, who will ignore robots.txt and change their user agent to evade blocks, apparently[1] 1: https://www.reddit.com/r/selfhosted/comments/1i154h7/o

Such as OpenAI, who will ignore robots.txt and change their user agent to evade blocks, apparently[1] 1: https://www.reddit.com/r/selfhosted/comments/1i154h7/openai_not_respecting_robotstxt_and_being_sneaky/ https://www.reddit.com/r/selfhosted/comments/1i154h7/openai_...