2 ms·You assumed people don't give a crap about robots.txt, otherwise you wouldn't have been surprised.by eesmith 4y agoYou assumed people don't give a crap about robots.txt, otherwise you wouldn't have been surprised.