Because of the way 80legs handles crawls, users don't need to worry about loop detection. There are really two issues here: 1. For each user crawl, we only al

Because of the way 80legs handles crawls, users don't need to worry about loop detection. There are really two issues here: 1. For each user crawl, we only allow the same url once, so any simple loops that involve the same urls are eliminated by this process no matter how large the loop 2. For more sophisticated "spider traps" that work with different urls and domains, they can have a limited effect on your crawls. Because of our per-domain rate throttling, the worst these traps can do is add a few cents per day to someone's crawl.