4 ms·Most robots don't honour robots.txt anyways...by nubinetwork 2y agoMost robots don't honour robots.txt anyways...