3 ms·Check out robots.txt -- most companies use it (in this case) because it's easier than dmca takedowns.by greglindahl 12y agoCheck out robots.txt -- most companies use it (in this case) because it's easier than dmca takedowns.