I would expect that you would use both async requests & multithreading. I might restrict each thread to operate on an exclusive set of domains to throttle the

I would expect that you would use both async requests & multithreading. I might restrict each thread to operate on an exclusive set of domains to throttle the request rate to any given domain at a point in time