2 ms·It honors robots.txt, even retroactively across changes of owners of domain names.by at_a_remove 6y agoIt honors robots.txt, even retroactively across changes of owners of domain names.