4 ms·
Cool, I was just looking for something like this! It's surprisingly annoying in built-in python to do something like this. The most recent thing I was trying t
by JackC 2y ago
Cool, I was just looking for something like this!
It's surprisingly annoying in built-in python to do something like this. The most recent thing I was trying to do was:
- load URLs from a file
- hand them out to one subprocess per cpu
- download them concurrently in threads or async within each subprocess
- pull the results back into a single process for formatting and storing
Getting this to work and handle queues, ctrl-c, exceptions etc. is just a whole mess involving python builtins created at different times with different interfaces; I hacked until I kind of got it working, but didn't love it. Bundling it all in a single tested package would be great.
- zenapollo 2y agoI stumbled on grequests for this use case and it just works.