4 ms·
If you're crawling different origins in parallel then you could gain a boost in speed by threading. It's usually bound by the network/processing on the receivin
by tokenizerrr 11y ago
If you're crawling different origins in parallel then you could gain a boost in speed by threading. It's usually bound by the network/processing on the receiving side. Your server can probably handle several outgoing connections at once.
- sebcat 11y agoYou can have several outgoing connections in one single thread, at the same time. Threading is for CPU bound problems that requires a shared memory space. I/O multiplexing and/or cooperative multitasking means you can have thousands concurrent connections per thread.
- tokenizerrr 11y agoSure. The parent comment referred to one download per thread so I did as well.