3 ms·
How to fetch URLs in parallel using Python
- riobard 16y agoWon't using multiple curl processes consume more resources than threads?
- coderdude 16y agoYes. However, I think how well this works depends on how many cores you have and how evenly you spread those processes across each core.
- riobard 16y agoIf the goal is to fetch as many URLs as possible, the best bet seems to be using async epoll to simultaneously fetch tens of thousands links. Processes/threads consumes too much memory to be viable.
- deleted 16y ago[deleted]
- mumrah 16y agoSeems like it would be easier to use xargs or GNU Parallel.