3 ms·
..:: Loop Detection?? ::.. Are you supposed to implement your own cycle detection? If not, how deep is the cycle detection that 80legs offers? -- There are
by tezza 17y ago
..:: Loop Detection?? ::..
Are you supposed to implement your own cycle detection?
If not, how deep is the cycle detection that 80legs offers?
--
There are plenty of HoneyTrap[1] OSS projects which will quickly rack up lots of $$$ if the 80legs spider is backlisted.
For those who don't know, these projects create deeply linked pages, and sometimes create infinite cycles. They are trying to hinder spammers, but may hinder 80legs too.
--
[1] http://www.davidnaylor.co.uk/stopping-bad-robots-with-honeytraps.html http://www.davidnaylor.co.uk/stopping-bad-robots-with-honeyt...
- jdrock 17y agoBecause of the way 80legs handles crawls, users don't need to worry about loop detection. There are really two issues here: 1. For each user crawl, we only allow the same url once, so any simple loops that involve the same urls are eliminated by this process no matter how large the loop 2. For more sophisticated "spider traps" that work with different urls and domains, they can have a limited effect on your crawls. Because of our per-domain rate throttling, the worst these traps can do is add a few cents per day to someone's crawl.