3 ms·
> Those restrictions are set up through the Robots Exclusion Protocol If anyone would like to join in, there's an actively maintained robots.txt here: https:/
by jsheard 2y ago
> Those restrictions are set up through the Robots Exclusion Protocol
If anyone would like to join in, there's an actively maintained robots.txt here:
https://github.com/ai-robots-txt/ai.robots.txt https://github.com/ai-robots-txt/ai.robots.txt
Yes, I know this isn't legally binding and scrapers can ignore it if they want to.