7 ms·

Apache Storm 1.0 released

160 points by wener 11y ago

ausjke 11y ago
so a newbie question, when do I need this? the only thing I understand is that hadoop is for batch process while storm is for realtime big-data processing.
- narsil 11y ago
  The home page has a 'Why use Storm' section that you might find helpful: https://storm.apache.org/index.html https://storm.apache.org/index.html For example, Twitter used to use Storm to process analytics on tweet data. They now use Heron. [1] Storm is valuable when your data is constantly being generated as a stream, on which you would like to perform processing immediately. I've previously also used Storm's distributed RPC cluster, where I fire off a task that I would like processed in the background and wait for the response to arrive. [1] https://blog.twitter.com/2015/flying-faster-with-twitter-heron https://blog.twitter.com/2015/flying-faster-with-twitter-her...
- pixelmonkey 11y ago
  If you're a Python user, I gave a talk at PyData called "Beating the GIL with Python" that covers all the tooling for parallel computing in Python, from the simplest (e.g. multiprocessing) to the more complex (Storm/Spark), with many tools in between, and what the trade-offs are. That may help you out. https://www.youtube.com/watch?v=gVBLF0ohcrE https://www.youtube.com/watch?v=gVBLF0ohcrE The rough summary is, if you need to do distributed multi-node computation with very low latency -- that is, end-to-end data processing time measured in millis or seconds -- then Storm will help you out. The "competitive" product in open source right now is pyspark + Spark Streaming. The benefit of Storm is that it is an older (more mature) project with many production deployments. My team at Parse.ly wrote the open source Python + Storm integration libraries, pystorm and streamparse, which help Python programmers make use of Storm for large-scale stream processing: https://github.com/Parsely/streamparse https://github.com/Parsely/streamparse. We also wrote pykafka, which is a Kafka client for Python: https://github.com/Parsely/pykafka https://github.com/Parsely/pykafka -- these projects are related since Kafka is often the streaming data source used as the input to a Storm topology.
  - RBerenguel 11y ago
    After the recent comparisons on real time processing between Flink and Spark, isn't Flink the actual "competitive" product? In the sense that it's real "real time", not the way Spark streaming (not dismissing it, it's what I actually use) works.
    - pixelmonkey 11y ago
      That's an interesting question. Unfortunately I haven't had a chance to evaluate Flink in-depth, it being relatively new.
      - gshulegaard 11y ago
        That's the gist. I am a Flink fan, but I get the impression it's still in the research and development phase...although rapidly approaching a potentially production-level quality. That said, Storm is Event-streaming processing which is usually the focal point of most comparisons between Flink and Spark Streaming.
  - narsil 11y ago
    I prefer Celery for Python. Far more light-weight, and intended for distributed task processing. http://celery.readthedocs.org/ http://celery.readthedocs.org/
    - brianwawok 11y ago
      Are those really competing though? Celery rocks for some background image processing. Not for multinode analytics of huge data sets.
      - throwaway-anon 11y ago
        Actually that's not true. Have a look at the airflow project by AirBnB. https://github.com/airbnb/airflow https://github.com/airbnb/airflow
- estefan 11y ago
  > when do I need this Probably never. Spark, flink and a few proprietary tools can operate in both batch and streaming modes which means you can share your codebase between both tasks if you do them. Before these tools it used to be hadoop for batch and storm for streaming, but I suspect those days are dying now (which goes some way to explaining spark's success) except if you require exceptional throughput in which case do some research & benchmarks.
  - tromobne8vb 11y ago
    Storm is a few years late. It has largely been eclipsed by the projects you mentioned and others. In theory, Storm can be set up to run with much lower latency than the others, but it is rare that an app will need that kind of response time.
    - fludlight 11y ago
      What's the range of latencies here? What types of projects really need the lowest latencies?
      - anonymousDan 11y ago
        Financial trading algos where low latency is important were the initial big market for stream processing/cep. As another example, imagine you're trying to do real-time fraud detection based on click stream analysis. If your latency is low enough you can potentially prevent suspicious transactions ever happening instead of having to allow them and then recover them somehow later.
    - d3fun 11y ago
      Storm is actually few years earlier. Its the first real-time processing framework before other projects showed up.
      - tromobne8vb 11y ago
        True, it showed up first. And then it stagnated for several years. Now it is behind the other alternatives.
- cprayingmantis 11y ago
  As with most tools it's depending on what you want to do with it. Storm is meant to handle streaming data. It's wonderful for low latency big data applications. You can check out the benchmarks for it at Yahoo Developer Tumblr https://yahooeng.tumblr.com/post/135321837876/benchmarking-streaming-computation-engines-at https://yahooeng.tumblr.com/post/135321837876/benchmarking-s...
therealdrag0 11y ago
Wow. Good work Storm people! Some nice features here. Anyone privy to what changes they made to yield the performance improvements?
- lovelearning 11y ago
  From the issue log : "Profiling a simple speed of light topology, shows that a good chunk of time of the SpoutOutputCollector.emit() is spent in the clojure reduce() function.. which is part of the ACK-ing logic. Re-implementing this reduce() logic in java gives a big performance boost in both in the Spout.nextTuple() and Bolt.execute()" [1]: https://issues.apache.org/jira/browse/STORM-1539 https://issues.apache.org/jira/browse/STORM-1539
- d3fun 11y ago
  More coming in the roadmap. We are also working performance improvements around internal queuing.
- kabhwan 11y ago
  Apache Storm 1.0.0 introduces batching internal disruptor queue which adds latencies slightly (but with timeout set to 1ms per each queue so it doesn't hurt) but gives huge throughput improvement. https://github.com/apache/storm/pull/765 https://github.com/apache/storm/pull/765
rar_ram 11y ago
Excellent work guys! I had developed a custom caching solution for storm in v0.9.6. Its great that distributed caching is being supported now. I would love to see some more documentation and examples on it.
pknerd 11y ago
Spark || Storm? Your take, guys?

160 points by wener 11y ago

so a newbie question, when do I need this? the only thing I understand is that hadoop is for batch process while storm is for realtime big-data processing.

narsil 11y ago
The home page has a 'Why use Storm' section that you might find helpful: https://storm.apache.org/index.html https://storm.apache.org/index.html For example, Twitter used to use Storm to process analytics on tweet data. They now use Heron. [1] Storm is valuable when your data is constantly being generated as a stream, on which you would like to perform processing immediately. I've previously also used Storm's distributed RPC cluster, where I fire off a task that I would like processed in the background and wait for the response to arrive. [1] https://blog.twitter.com/2015/flying-faster-with-twitter-heron https://blog.twitter.com/2015/flying-faster-with-twitter-her...
pixelmonkey 11y ago
If you're a Python user, I gave a talk at PyData called "Beating the GIL with Python" that covers all the tooling for parallel computing in Python, from the simplest (e.g. multiprocessing) to the more complex (Storm/Spark), with many tools in between, and what the trade-offs are. That may help you out. https://www.youtube.com/watch?v=gVBLF0ohcrE https://www.youtube.com/watch?v=gVBLF0ohcrE The rough summary is, if you need to do distributed multi-node computation with very low latency -- that is, end-to-end data processing time measured in millis or seconds -- then Storm will help you out. The "competitive" product in open source right now is pyspark + Spark Streaming. The benefit of Storm is that it is an older (more mature) project with many production deployments. My team at Parse.ly wrote the open source Python + Storm integration libraries, pystorm and streamparse, which help Python programmers make use of Storm for large-scale stream processing: https://github.com/Parsely/streamparse https://github.com/Parsely/streamparse. We also wrote pykafka, which is a Kafka client for Python: https://github.com/Parsely/pykafka https://github.com/Parsely/pykafka -- these projects are related since Kafka is often the streaming data source used as the input to a Storm topology.
- RBerenguel 11y ago
  After the recent comparisons on real time processing between Flink and Spark, isn't Flink the actual "competitive" product? In the sense that it's real "real time", not the way Spark streaming (not dismissing it, it's what I actually use) works.
  - pixelmonkey 11y ago
    That's an interesting question. Unfortunately I haven't had a chance to evaluate Flink in-depth, it being relatively new.
    - gshulegaard 11y ago
      That's the gist. I am a Flink fan, but I get the impression it's still in the research and development phase...although rapidly approaching a potentially production-level quality. That said, Storm is Event-streaming processing which is usually the focal point of most comparisons between Flink and Spark Streaming.
- narsil 11y ago
  I prefer Celery for Python. Far more light-weight, and intended for distributed task processing. http://celery.readthedocs.org/ http://celery.readthedocs.org/
  - brianwawok 11y ago
    Are those really competing though? Celery rocks for some background image processing. Not for multinode analytics of huge data sets.
    - throwaway-anon 11y ago
      Actually that's not true. Have a look at the airflow project by AirBnB. https://github.com/airbnb/airflow https://github.com/airbnb/airflow
estefan 11y ago
> when do I need this Probably never. Spark, flink and a few proprietary tools can operate in both batch and streaming modes which means you can share your codebase between both tasks if you do them. Before these tools it used to be hadoop for batch and storm for streaming, but I suspect those days are dying now (which goes some way to explaining spark's success) except if you require exceptional throughput in which case do some research & benchmarks.
- tromobne8vb 11y ago
  Storm is a few years late. It has largely been eclipsed by the projects you mentioned and others. In theory, Storm can be set up to run with much lower latency than the others, but it is rare that an app will need that kind of response time.
  - fludlight 11y ago
    What's the range of latencies here? What types of projects really need the lowest latencies?
    - anonymousDan 11y ago
      Financial trading algos where low latency is important were the initial big market for stream processing/cep. As another example, imagine you're trying to do real-time fraud detection based on click stream analysis. If your latency is low enough you can potentially prevent suspicious transactions ever happening instead of having to allow them and then recover them somehow later.
  - d3fun 11y ago
    Storm is actually few years earlier. Its the first real-time processing framework before other projects showed up.
    - tromobne8vb 11y ago
      True, it showed up first. And then it stagnated for several years. Now it is behind the other alternatives.
cprayingmantis 11y ago
As with most tools it's depending on what you want to do with it. Storm is meant to handle streaming data. It's wonderful for low latency big data applications. You can check out the benchmarks for it at Yahoo Developer Tumblr https://yahooeng.tumblr.com/post/135321837876/benchmarking-streaming-computation-engines-at https://yahooeng.tumblr.com/post/135321837876/benchmarking-s...

therealdrag0 11y ago

Wow. Good work Storm people! Some nice features here. Anyone privy to what changes they made to yield the performance improvements?

lovelearning 11y ago
From the issue log : "Profiling a simple speed of light topology, shows that a good chunk of time of the SpoutOutputCollector.emit() is spent in the clojure reduce() function.. which is part of the ACK-ing logic. Re-implementing this reduce() logic in java gives a big performance boost in both in the Spout.nextTuple() and Bolt.execute()" [1]: https://issues.apache.org/jira/browse/STORM-1539 https://issues.apache.org/jira/browse/STORM-1539
d3fun 11y ago
More coming in the roadmap. We are also working performance improvements around internal queuing.
kabhwan 11y ago
Apache Storm 1.0.0 introduces batching internal disruptor queue which adds latencies slightly (but with timeout set to 1ms per each queue so it doesn't hurt) but gives huge throughput improvement. https://github.com/apache/storm/pull/765 https://github.com/apache/storm/pull/765

rar_ram 11y ago

Excellent work guys! I had developed a custom caching solution for storm in v0.9.6. Its great that distributed caching is being supported now. I would love to see some more documentation and examples on it.

pknerd 11y ago

Spark || Storm? Your take, guys?