3 ms·
Well, do you have something like this?: http://setiathome.berkeley.edu/top_hosts.php http://setiathome.berkeley.edu/top_hosts.php BTW, how do you ensure someb
by vicaya 17y ago
Well, do you have something like this?:
http://setiathome.berkeley.edu/top_hosts.php http://setiathome.berkeley.edu/top_hosts.php
BTW, how do you ensure somebody is not submitting a malware (intentional or not) that can DDoS someone by crawling the same host at pretty much the same time?
- jdrock 17y agoI know Plura uses IP information to prevent bandwidth overages, but I don't think they have a list of top hosts. 80legs enforces throttles and rate limits on the sites that are crawled. We try to make a best guess at a rate limit for each site. Sometimes that best guess is too fast, so we take corrective action when we're notified by the webmaster. Users can't override the crawling rates.