4 ms·
> when do I need this Probably never. Spark, flink and a few proprietary tools can operate in both batch and streaming modes which means you can share your co
by estefan 11y ago
> when do I need this
Probably never.
Spark, flink and a few proprietary tools can operate in both batch and streaming modes which means you can share your codebase between both tasks if you do them. Before these tools it used to be hadoop for batch and storm for streaming, but I suspect those days are dying now (which goes some way to explaining spark's success) except if you require exceptional throughput in which case do some research & benchmarks.
- tromobne8vb 11y agoStorm is a few years late. It has largely been eclipsed by the projects you mentioned and others. In theory, Storm can be set up to run with much lower latency than the others, but it is rare that an app will need that kind of response time.
- fludlight 11y agoWhat's the range of latencies here? What types of projects really need the lowest latencies?
- anonymousDan 11y agoFinancial trading algos where low latency is important were the initial big market for stream processing/cep. As another example, imagine you're trying to do real-time fraud detection based on click stream analysis. If your latency is low enough you can potentially prevent suspicious transactions ever happening instead of having to allow them and then recover them somehow later.
- d3fun 11y agoStorm is actually few years earlier. Its the first real-time processing framework before other projects showed up.
- tromobne8vb 11y agoTrue, it showed up first. And then it stagnated for several years. Now it is behind the other alternatives.
- jwn 11y agoI don't see why this was downvoted. The points may be debatable, but isn't that the point of a discussion?
- aktau 11y agoAbsolutely, I find Google's "The Dataflow Model" paper about this to be a good read: http://research.google.com/pubs/pub43864.html http://research.google.com/pubs/pub43864.html (it talks about generalizing streaming vs. batch vs. micro-batch frameworks so it becomes an easy cost-based decision).
- abc_lisper 11y agoI disagree. If low latency is a priority, Storm has little competition. Checkout the benchmarks done by Yahoo engineers. https://yahooeng.tumblr.com/post/135321837876/benchmarking-streaming-computation-engines-at https://yahooeng.tumblr.com/post/135321837876/benchmarking-s...
- anonetal 11y agoThanks for that link. However, they appear to conclude Flink and Storm are quite similar in performance. If you really need low latencies, it is quite likely that none of these will work for you anyway and you would have to build a specialized CEP-style system.
- abc_lisper 11y agoYes, this release is all about performance. Now Storm must thus be far ahead.