2 ms·

Not honoring http://www.robotstxt.org/faq/prevent.html http://www.robotstxt.org/faq/prevent.html can be a felony;