You could crawl a lot of different sites one page at a time. When I wrote a large distributed download system, I would use pycurl's bandwidth throttle and also

You could crawl a lot of different sites one page at a time. When I wrote a large distributed download system, I would use pycurl's bandwidth throttle and also store a 5 minute average of bandwidth per domain that would prevent other downloaders from saturating a domain.