3 ms·
250000000 / (3600 * 40) / 20 = 86 86 pages per machine is not very performant at all, just very simple parallelism will do.
by ospider 8y ago
250000000 / (3600 * 40) / 20 = 86
86 pages per machine is not very performant at all, just very simple parallelism will do.
- deleted 8y ago[deleted]
- natdempk 8y agoThe units of your equation are pages per second per machine, but I agree. Reading briefly it seems like he only used 141 threads per machine to do the actual crawling. This likely could be pushed an order of magnitude further (or even more!), especially by using green threads. It does seem like he was running up against CPU constraints and network issues soon after that, but also this was written in 2012.
- vidarh 8y agoCa. 2007 I was running a RSS feed fetcher that was running 400 processes (not threads) in parallel on dual CPU Xeon's. The only reason we didn't push it higher was that 400 in parallel was more than enough for our use at the time. Of course of those 400 some were always waiting on IO. I never measured how many feeds we did on average every second.