2 ms·Isn't that basically like robots.txt? But robots.txt is often ignored.by JohnFen 4y agoIsn't that basically like robots.txt? But robots.txt is often ignored.