2 ms·Someone who doesn't care about polluting our corpus is not going to care about your robots.txtby rvbissell 2y agoSomeone who doesn't care about polluting our corpus is not going to care about your robots.txt