2 ms·
>I'm fine with robots.txt violations being considered hacking. Okay. Start with something simple then - how would you define a "bot" and thus subject to your r
by alasdair_ 9y ago
>I'm fine with robots.txt violations being considered hacking.
Okay. Start with something simple then - how would you define a "bot" and thus subject to your robots.txt rule?
Is my web-browser a bot? What about a proxy? What about a deaf persons screen reader?
If my web-browser pre-fetches links near my mouse pointer, is that a bot? What if it downloads the whole of an article split over, say, ten pages?
I think of robots.txt similar to posting a "No Trespassing" sign. For a private residence, it's almost not even required, yet for something like a shopping mall during opening hours, the default assumption is that anyone is allowed to be there without a specific invitation, until they are expressly asked to leave and not come back.
Trying to nail down the exact line is a tough issue.