4 ms·
easy guess that length breaks some legacy stuff but every robots.txt should have a auto-ban trap line ie. crawl it and die basically a script that puts the r
by ck2 2y ago
easy guess that length breaks some legacy stuff
but every robots.txt should have a auto-ban trap line
ie. crawl it and die
basically a script that puts the requesting IP into firewall
of course it's possible to abuse that so it has to be monitored
- okdood64 2y agoHow do you discern a crawler agent and a human? Is it easily as the fact that they might cover something like 80%+ of the site in one visit fairly quickly?
- SoftTalker 2y agoCrawlers/archivers will be hitting your site much faster than a human user.
- johneth 2y agoI thought about doing something like that, but then I realised: what if someone linked to the trap URL it from another site and a crawler followed that link to the trap? You might end up penalising Googlebot or Bingbot. If anyone knew what that trap URL did, and felt malicious, this could happen.
- CodesInChaos 2y agoA crawler could easily avoid that by fetching the target domain's robots.txt before fetching the link target. However a website could also embed the honeypot link in an <img> tag and get the user banned when their browser attempts to load the image.