4 ms·
Despite broadcasting their user agents properly, the AI companies ignore robots.txt and still waste my server resources. So yeah, the dishonest botnets will hav
by MillironX 1y ago
Despite broadcasting their user agents properly, the AI companies ignore robots.txt and still waste my server resources. So yeah, the dishonest botnets will have an advantage, but I don't give swindlers a pass just because they rob me to my face. I'm okay with defaults that punish all bots.
- goku12 1y agoYou can have a bot allow list. I think it's also being planned as a subscription service (not sure about this part).