3 ms·
> You would need "bots allowed" information in the HTTP handshake itself to prevent bots from accidentally hitting pages they shouldn't. Humans could also hit
by TazeTSchnitzel 12y ago
> You would need "bots allowed" information in the HTTP handshake itself to prevent bots from accidentally hitting pages they shouldn't.
Humans could also hit such pages. If your GET requests change state, there's no helping you.
- wareya 12y agoThe whole point of robots.txt is that there are pages which people may hit that bots can't. What are you on?