2 ms·
My experience disagrees with the 'Respects robots.txt' column for most of the bots listed. Would love to see more details of how they determine that metric.
by jedahan 1y ago
My experience disagrees with the 'Respects robots.txt' column for most of the bots listed. Would love to see more details of how they determine that metric.
- o11c 1y agoAre you verifying the IP, or just blindly trusting the user agent?
- jedahan 1y agoGood question - I am just putting up robots.txt, and seeing little to no decrease in traffic. I have not tried verifying that server logs user agent corresponds to specific IP addresses. Do you have resources where all the AI bots post their list of IP addresses? Would be easier to just ban by IP completely. From what I've read these bots rotate and use residential blocks so I am not sure I can even see all of them.