3 ms·My advice is: Obey robots.txt and provide a real user agent. End of story. Seriously.by chinathrow 11y agoMy advice is: Obey robots.txt and provide a real user agent. End of story. Seriously.