Turns out crawlers spend a lot more (wall clock) time waiting for a complete response than they do requesting it. However, scheduling is a much (much, much, muc

Turns out crawlers spend a lot more (wall clock) time waiting for a complete response than they do requesting it. However, scheduling is a much (much, much, much) harder problem to deal with than async i/o but it's not what many people here need to worry about.