3 ms·
Author here. I wish I could, but I bet most would just ignore robots.txt.
by gavinhoward 3y ago
Author here.
I wish I could, but I bet most would just ignore robots.txt.
- catboybotnet 3y agoSeconding. robots.txt is just a way of "asking nicely." If somebody wants to scrape, they're spoofing their UA and ignoring it. Can't do anything other than monitor the logs and ban IPs one by one.