4 ms·
I think it's a mistake to think of virtual threads as "threads, but magically better". They have real drawbacks, and aren't suitable in every scenario. A craw
by marginalia_nu 1y ago
I think it's a mistake to think of virtual threads as "threads, but magically better". They have real drawbacks, and aren't suitable in every scenario.
A crawler is a textbook case where you get more out of a bounded (but large) threadpool than virtual threads, because the boundedness is a feature. Even with infinite RAM, crawling from as many connections as possible leads to a ton of network congestion, and ironically, will slow down your crawling speed.