3 ms·It seems they haven't heard of robots.txt.by gasull 17y agoIt seems they haven't heard of robots.txt.