Wow has it really been more than 20 years!?? I feel old now... I do get what you're saying but if you have to implement "appropriate measures" anyway then the

Wow has it really been more than 20 years!?? I feel old now... I do get what you're saying but if you have to implement "appropriate measures" anyway then the robots.txt file becomes completely redundant.