2 ms·
Crank up the ulimit. And what is this 1 req/second nonsense? 1 req/sec/domain maybe. I have to agree, my first thought is "why this is not a single node in Go?"
by sethammons 3y ago
Crank up the ulimit. And what is this 1 req/second nonsense? 1 req/sec/domain maybe. I have to agree, my first thought is "why this is not a single node in Go?"
- marginalia_nu 3y agoOh yeah, I mean 1 req/sec per domain of course. It's very easy to end up with tens of thousands of not-quite-closed connections while crawling, even with SO_LINGER(0), proper closure, tweaking the TCP settings, and doing all things by the book. It's a different situation from e.g. having a bunch of incoming connections to a node, the traffic patterns are not at all similar.