If the goal is to fetch as many URLs as possible, the best bet seems to be using async epoll to simultaneously fetch tens of thousands links. Processes/threads

If the goal is to fetch as many URLs as possible, the best bet seems to be using async epoll to simultaneously fetch tens of thousands links. Processes/threads consumes too much memory to be viable.