I have clearly communicated the constraints of use on my site using robots.txt, which is the established standard for communicating things like these. You are e

I have clearly communicated the constraints of use on my site using robots.txt, which is the established standard for communicating things like these. You are expected to fetch it and figure it out. It is not necessary to build walls and lock doors when a no trespassing sign is clearly visible. Seriously, though, making robots.txt have the weight of law would be a massive improvement for the ecosystem in general.