2 ms·As someone who has seen way too many robots.txt files that's exactly how you do it.by kabacha 6y agoAs someone who has seen way too many robots.txt files that's exactly how you do it.