You can have quite a number of concurrent connections in 1GB of RAM, 10k at least? That seems allright for a single node, and scaling horizontally after that.

You can have quite a number of concurrent connections in 1GB of RAM, 10k at least? That seems allright for a single node, and scaling horizontally after that. For something like crawling, which has a lot of time spent on CPU intensive parsing and I with unpredictable times (hitting external service) I would throw that in a background worker from the start.