4 ms·
With async in any language of choice this should need just 1 server. 250M/40hours=1.7K requests/second. You can probably do it on a single core in a low level
by ddorian43 3y ago
With async in any language of choice this should need just 1 server. 250M/40hours=1.7K requests/second.
You can probably do it on a single core in a low level language like Rust/Java.
- marginalia_nu 3y agoThreads isn't the only bottleneck in crawling. Assuming you're crawling at a civilized rate of about 1 request/second, you're only so many network hiccups away from consuming the entire ephemeral port range with connections in TIME_WAIT or CLOSE_WAIT.
- sethammons 3y agoCrank up the ulimit. And what is this 1 req/second nonsense? 1 req/sec/domain maybe. I have to agree, my first thought is "why this is not a single node in Go?"
- marginalia_nu 3y agoOh yeah, I mean 1 req/sec per domain of course. It's very easy to end up with tens of thousands of not-quite-closed connections while crawling, even with SO_LINGER(0), proper closure, tweaking the TCP settings, and doing all things by the book. It's a different situation from e.g. having a bunch of incoming connections to a node, the traffic patterns are not at all similar.