2 ms·At this point honoring robots.txt will ensure you have a terrible searching experience...by bearjaws 2y agoAt this point honoring robots.txt will ensure you have a terrible searching experience...