3 ms·How about forcing Bytespider to observe robots.txt while you're at it?by bediger4000 3y agoHow about forcing Bytespider to observe robots.txt while you're at it?