11 ms·Yes, and obviously: bots crushing servers in strict contravention of the robots.txt rules.by boothby 4mo agoYes, and obviously: bots crushing servers in strict contravention of the robots.txt rules.