This is a hypothetical so give me a little rope here, but what if robots.txt wasn't a suggestion? What if it were binding (leaving aside for a moment how one wo

This is a hypothetical so give me a little rope here, but what if robots.txt wasn't a suggestion? What if it were binding (leaving aside for a moment how one would enforce / mandate / guarantee that)? Would that solve the whole problem? Folks who ran webservers declared what they consent to, and that happens? I think it's useful to just see if there's a consensus on that: actually making that happen is a whole can of worms itself, but it's strictly simpler than devising a good outcome without the consensus. (And such things are not impossible, merely difficult, we have other systems ranging from BGP to the TLD mechanism that get honored in real life).