4 ms·

There seems to be a fundamental trade-off between latency and throughput, with stream processors optimizing for latency and batch processors optimizing for thro

by ntoshev 12y ago

There seems to be a fundamental trade-off between latency and throughput, with stream processors optimizing for latency and batch processors optimizing for throughput. The post says stream processors are actually used for reprocessing in production at LinkedIn, and simplifying things might well worth it being slower, but do they know how much slower this is than using Hadoop?

boredandroid 12y ago
This is an excellent point, there IS a fundamental tradeoff between latency and throughput. Computers are much better a processing larger chunks of data linearly rather than small bits of data. However the question to ask is how much data do you have to batch together to get good throughput? There are diminishing returns and you stop getting much benefit after about 1MB in my experience. So this line of reasoning will not get you to additional latency of more than a few seconds, it definitely won't get you to a 24 hour batch data cycle. Actually because we think this tradeoff is important we allow consumers from Kafka to specify how much data they want accumulated on the server before the server should complete their fetch request. The default is 1 byte, which means respond as soon as there is anything new for me, which optimizes latency. However setting this to a higher number will reduce round-trips and optimize throughput by avoiding lots of small fetches. This allows you to trade a small amount of latency for throughput. (In either case you can bound the waiting with a timeout so the latency is never worse than some maximum delay even if sufficient data hasn't arrived). In any case when doing re-processing, as described in this article, there will always be lots of data accumulated and you will always fetch chunks of the maximum size you have configured (say 1MB). So in the reprocessing case you always get the "batch"-like throughput.
mythbuster 12y ago
FWIW most of LinkedIn's batch processing still runs on Hadoop. Stream versus batch is not just about throughput versus latency. It is about what you do with the extra latency budget to increase your throughput. e.g. If you reorganize your data to make your batch process go 10x faster because you used a better join algorithm, then its worth doing things in batch. Otherwise there isn't much difference.

by ntoshev 12y ago

boredandroid 12y ago

This is an excellent point, there IS a fundamental tradeoff between latency and throughput. Computers are much better a processing larger chunks of data linearly rather than small bits of data. However the question to ask is how much data do you have to batch together to get good throughput? There are diminishing returns and you stop getting much benefit after about 1MB in my experience. So this line of reasoning will not get you to additional latency of more than a few seconds, it definitely won't get you to a 24 hour batch data cycle. Actually because we think this tradeoff is important we allow consumers from Kafka to specify how much data they want accumulated on the server before the server should complete their fetch request. The default is 1 byte, which means respond as soon as there is anything new for me, which optimizes latency. However setting this to a higher number will reduce round-trips and optimize throughput by avoiding lots of small fetches. This allows you to trade a small amount of latency for throughput. (In either case you can bound the waiting with a timeout so the latency is never worse than some maximum delay even if sufficient data hasn't arrived). In any case when doing re-processing, as described in this article, there will always be lots of data accumulated and you will always fetch chunks of the maximum size you have configured (say 1MB). So in the reprocessing case you always get the "batch"-like throughput.

mythbuster 12y ago

FWIW most of LinkedIn's batch processing still runs on Hadoop. Stream versus batch is not just about throughput versus latency. It is about what you do with the extra latency budget to increase your throughput. e.g. If you reorganize your data to make your batch process go 10x faster because you used a better join algorithm, then its worth doing things in batch. Otherwise there isn't much difference.