2 ms·If robots.txt said to jump off a bridge, would your spider do it?by elwell 13y agoIf robots.txt said to jump off a bridge, would your spider do it?