3 ms·
There's also popular repository that maintains a comprehensive list of LLM and AI related bots to aid in blocking these abusive strip miners. https://github.co
by vunderba 2y ago
There's also popular repository that maintains a comprehensive list of LLM and AI related bots to aid in blocking these abusive strip miners.
https://github.com/ai-robots-txt/ai.robots.txt https://github.com/ai-robots-txt/ai.robots.txt
- Pooge 2y agoI didn't know about this. Thank you! After some digging, I also found a great way to surprise bots that don't respect robots.txt[1] :) [1]: https://melkat.blog/p/unsafe-pricing https://melkat.blog/p/unsafe-pricing