3 ms·
Is there an agreed upon best effort robots.txt that I can reference to cut out crawlers? Or a list of IP addresses or user agents or other identifying info kno
by pton_xd 2y ago
Is there an agreed upon best effort robots.txt that I can reference to cut out crawlers?
Or a list of IP addresses or user agents or other identifying info known to be used by AI crawlers?
Ultimately pointless but just curious.
- input_sh 2y agohttps://github.com/ai-robots-txt/ai.robots.txt https://github.com/ai-robots-txt/ai.robots.txt
- alentred 2y ago> Ultimately pointless I wonder if there are crowd-sourced iptables instead. Similar to how ad filters are maintained.
- mewpmewp2 2y agoNone the less dark actors are still going to get through and sell this data later. There are probably tons of those crawlers and with intent to later launder that data, probably using LLMs intended to change the content just enough to be outside of copyright zone.