3 ms·One would assume legit spiders obey robots.txt.by xyzal 7mo agoOne would assume legit spiders obey robots.txt.lolc 7mo agoThis, to me, is the strongest argument to offer these slop generators. It provides an incentive to follow the robots.txt.xyzal 7mo agoExactly. You disobey robots file => we'll make your crawl gain a net negative.heyethan 7mo ago[flagged]