That's why these kinds of discussions are difficult. Not just the usual "it depends" but multiple levels of "it depends". As you say, if you're job mix is keepi

That's why these kinds of discussions are difficult. Not just the usual "it depends" but multiple levels of "it depends". As you say, if you're job mix is keeping a pipe saturated then raw latency is probably not your critical problem. Though, of course, if, for example, you have one pipe for bulk transfer and one for control/meta-data.... :-) The comment about handling data locally was trying to get to the issue of how much data needs to be going back and for across the cluster rather than just living on the individual nodes. For example, we crawl millions of sites so there's an initial lump of intake data to each of the crawlers. On each crawler, that lump of data will get processed, redacted, expanded, and what not and only the resulting indexes and segments are ever transfered off the crawlers.