3 ms·maybe to see what firms ignore robot.txt files?by drhodes 17y agomaybe to see what firms ignore robot.txt files?