https://en.wikipedia.org/wiki/Robots_exclusion_standard#Security https://en.wikipedia.org/wiki/Robots_exclusion_standard#Secu... robots.txt was proposed after

https://en.wikipedia.org/wiki/Robots_exclusion_standard#Security https://en.wikipedia.org/wiki/Robots_exclusion_standard#Secu... robots.txt was proposed after a badly behaved bot DoSed a web server 20+ years ago, those were different times. With the robots.txt standed now those who wants to play nice can do so without asking anything, for the badly behaved ones it's still up to the admin to put forward the appropriate measures.