3 ms·
It won't help with the more egregious scrapers, but this list is handy for telling the ones that do respect robots.txt to kindly fuck off: https://github.com/a
by jsheard 2y ago
It won't help with the more egregious scrapers, but this list is handy for telling the ones that do respect robots.txt to kindly fuck off:
https://github.com/ai-robots-txt/ai.robots.txt https://github.com/ai-robots-txt/ai.robots.txt