4 ms·
The 50,000+ computers come from a distributed grid computing system that's similar to a commercial opt-in SETI@home. Because of that, we don't know exact specs
by jdrock 17y ago
The 50,000+ computers come from a distributed grid computing system that's similar to a commercial opt-in SETI@home.
Because of that, we don't know exact specs on the computers, but here are the limitations that 80Apps have: http://80legs.pbworks.com/80Apps#LimitationsonYourCode http://80legs.pbworks.com/80Apps#LimitationsonYourCode
- vicaya 17y agoWell, do you have something like this?: http://setiathome.berkeley.edu/top_hosts.php http://setiathome.berkeley.edu/top_hosts.php BTW, how do you ensure somebody is not submitting a malware (intentional or not) that can DDoS someone by crawling the same host at pretty much the same time?
- jdrock 17y agoI know Plura uses IP information to prevent bandwidth overages, but I don't think they have a list of top hosts. 80legs enforces throttles and rate limits on the sites that are crawled. We try to make a best guess at a rate limit for each site. Sometimes that best guess is too fast, so we take corrective action when we're notified by the webmaster. Users can't override the crawling rates.