2 ms·What is funny is that using robots.txt would have been enough.by yuhong 12y agoWhat is funny is that using robots.txt would have been enough.