A while ago I wrote a shell script to open like 100 instances on AWS, running a script on each that would be in charge of downloading a part of the files. usin

A while ago I wrote a shell script to open like 100 instances on AWS, running a script on each that would be in charge of downloading a part of the files. using curl or wget could probably be a bit faster than python, you should also run multiple requests concurrently, because it's unlikely that the sites can fill your whole bandwidth You could probably do the same with some serverless/cloud functions but it could be a bit expensive