3 ms·
Something that has always puzzled me about scrapers is how they avoid scraper traps. For example, what if you have a web site that generates a thousand random
by t0mbstone 8y ago
Something that has always puzzled me about scrapers is how they avoid scraper traps.
For example, what if you have a web site that generates a thousand random links on a page, which all load pages that generate another thousand random links, to infinity?
- astonex 8y agoJust have a maximum traversal depth
- vidarh 8y agoA well-behaved crawler will be self-limiting in this regard in that they seek to avoid requesting URLs from the same domain too often, and then the better you are at spreading requests over as many domains at possible, the less you'll be affected by such traps. You can do a lot of other things too, but the above already takes most of the sting out of such traps unless you use a huge number of domains.
- lfowles 8y agoI used to have what was basically a spider trap on my website while learning Django. The truth is that every bot that found it hammered my site with tens of thousands of requests per day (even Googlebot). I don't think it stopped until I blocked the page in robots.txt
- iso1337 8y agoYou can use the AOPIC algorithm that has a credit and taxation system to penalize these types of websites. https://stackoverflow.com/questions/5834808/designing-a-web-crawler https://stackoverflow.com/questions/5834808/designing-a-web-...