Crawling a website in my opinion is akin to someone snooping inside your house and making a index of rooms and stuff you have. Robots.txt protocol exists but i

Crawling a website in my opinion is akin to someone snooping inside your house and making a index of rooms and stuff you have. Robots.txt protocol exists but it is not even official internet standard under IETF. You can break robots.txt rules and there are no consequences at all. If robots.txt protocol was made law then it would be whole another story.