4 ms·Ethical at least requires respecting robots.txt.by andreasvc 12y agoEthical at least requires respecting robots.txt.elwell 12y agoIf robots.txt said to jump off a bridge, would your spider do it?