2 ms·
This is great. I was running threads on a current crawl job but the real bottleneck is BeautifulSoup and not the network. So splitting the project into threads(
by breck 18y ago
This is great. I was running threads on a current crawl job but the real bottleneck is BeautifulSoup and not the network. So splitting the project into threads(while it helped about 10%) wasn't really necessary and Twisted probably would have done the trick.