3 ms·

Haven't they heard of robots.txt ?