3 ms·

Thanks...On the legal front, what all should I consider before crawling a website, apart from honoring robots.txt?