I've read wget, and I am a cURL contributor as well. I know the capabilities of each very well. But when you have 1M+ urls to download and you care about proper

I've read wget, and I am a cURL contributor as well. I know the capabilities of each very well. But when you have 1M+ urls to download and you care about proper error handling at a large scale, those tools will fall short. Not all errors need to be handled in the same way. Some need retries some don't depending on http response codes for example. Another problem you'll hit is how to make sure that the machines are saturated. How many jobs should be running at the current time depends on what's being downloaded and how much room you have to run additional downloads. Again, it all depends on what you need from the system and how much leeway you have.