9 ms·

History of Apache Storm and Lessons Learned

174 points by plinkplonk 12y ago

discardorama 12y ago
Great read. An aside: when Nathan started work on Storm, there was a competing streaming platform being worked on inside Yahoo, called "S4". It had many similarities to Storm, and some key differences. It is still in the incubator: http://incubator.apache.org/s4/ http://incubator.apache.org/s4/ (I have no connection with S4, just know some people who worked on it)
- dimon 12y ago
  The key advantage Storm brought to the table was, as briefly mentioned in the article, the at-least once processing guarantee Storm offers thanks to its efficient tracking of tuples and their descendants (see https://storm.incubator.apache.org/documentation/Guaranteeing-message-processing.html https://storm.incubator.apache.org/documentation/Guaranteein...). To my knowledge S4 offers no such guarantees. I think this made Storm more attractive for many use-cases, it did for me.
clubhi 12y ago
I've been using Storm for a few years in production. Thanks for all the hard work Nathan. You and Kyle Kingsbury are two "younger" guys that I always enjoy reading whatever you write.
Terr_ 12y ago
I've seen Samza's documentation comparing itself to Storm, I wonder what the reverse-perspective is...
- wink 12y ago
  As I understood it Samza ties in with Hadoop/HBase, so if you already have that, it might be easier (or even possible at all) to integrate, whereas Storm has nothing to do with with that and only does stream processing.
  - bhangi 12y ago
    Samza can run on a Hadoop 2 cluster using YARN. As far as I know, there is no other connection to Hadoop or HBase.
skybrian 12y ago
How does it compare to Apache Kafka? (I haven't used either.)
- akbar501 12y ago
  Kafka is a high performance, persistent message queue. Storm is distributed stream processing. The two work together very well. Often, a server will publish messages to Kafka (or another queue), which are then read and processed by Storm.
  - throwaway1979 12y ago
    Interesting question and response. I haven't used either but read about both systems independently. A bit more googling and I found: http://www.slideshare.net/gschmutz/kafka-andstromeventprocessinginrealtime http://www.slideshare.net/gschmutz/kafka-andstromeventproces... Slides 42 and 43 describe an architecture with all three ... hadoop, storm and kafka. Seems like a beast and I don't have applications but pretty cool combo of technologies.
    - erichmond 12y ago
      http://manning.com/marz/ http://manning.com/marz/ - this is often the trifecta used in the "lambda architecture", coined by nathan marz. This is also the system they built @ backtype to do their real-time analytics system.
      - phunge 12y ago
        Also worth checking out http://radar.oreilly.com/2014/07/questioning-the-lambda-architecture.html http://radar.oreilly.com/2014/07/questioning-the-lambda-arch... from Jay Kreps, who's part of the Kafka braintrust.
- pixelmonkey 12y ago
  Complementary technologies, often used together. In fact, in Storm 0.9.x, Storm actually bundles a JVM implementation of a Storm Spout that plugs directly into Kafka: https://github.com/apache/storm/tree/master/external/storm-kafka https://github.com/apache/storm/tree/master/external/storm-k...
dmritard96 12y ago
Nice read. At my previous job we used storm for all sorts of cool things. It worked great and it was a pleasure to be the one at the company that got to really cozy up with it. Deployment (with or without storm-deploy) was also well thought out.
jtheory 12y ago
I think I get the concept (and it's interesting); but I'm a bit baffled by the terminology: > That led me to the idea of "spouts" and "bolts" – a spout produces brand new streams, and a bolt takes in streams as input and produces streams as output. Spout I can grasp; why "bolt"? Why these two, together? I'd have really appreciated an extra sentence or two here as to why he chose these terms -- naming is so crucial, and the fact that both "spouts" and "bolts" sound like sources/producers to me (one of water, another of electricity...?) made it harder to grasp how Storm works.
- nemothekid 12y ago
  Bolt here probably references thunderbolt, and they both references (rain)storms.
  - jtheory 12y ago
    That was my assumption; but if you're going to name parts of your system after natural phenomena, you have to choose ones that make some sense, not just names out of a hat that are vaguely in the same domain. "Spout" makes sense as something that's a source of a stream. I haven't been able to figure out the metaphor for "bolt" at all, particularly as something that processes water (stream/spout...) in some way.
- siddboots 12y ago
  I always assumed that "bolt" refers to the threaded fasteners, not electrical bolts. The analogy they are going for is that they are the components responsible for "bolting" spouts together (although, the mixing of metaphors bothers me a little bit... why are we using bolts to fasten fluids together?) To be super clear about it, a "topology" in Storm is a directed acyclical graph. The nodes are called "bolts" and the edges are called "spouts".
  - dwenzek 12y ago
    > a "topology" in Storm is a directed acyclical graph. The nodes are called "bolts" and the edges are called spouts" Not really. Edges are streams and nodes are either spouts or bolts. Spouts have no predecessors and inject external data into the system. Bolts gather one or more input streams and produce zero or more output streams. These names are a bit confusing, but only during the very first steps of a project. Quickly every developers grasp the concepts. I have nevertheless a concern regarding spouts and bolts. In practice, when a topology is refactored, bolts have frequently to be transformed into spouts. This occurs when a topology is split in two to ease deployment; and when a persistent queue is added along a stream. This highlights that the distinction between spouts and bolts is bit artificial.
schultz9999 12y ago
Looks like we overloaded the site...

174 points by plinkplonk 12y ago

Great read. An aside: when Nathan started work on Storm, there was a competing streaming platform being worked on inside Yahoo, called "S4". It had many similarities to Storm, and some key differences. It is still in the incubator: http://incubator.apache.org/s4/ http://incubator.apache.org/s4/ (I have no connection with S4, just know some people who worked on it)

dimon 12y ago
The key advantage Storm brought to the table was, as briefly mentioned in the article, the at-least once processing guarantee Storm offers thanks to its efficient tracking of tuples and their descendants (see https://storm.incubator.apache.org/documentation/Guaranteeing-message-processing.html https://storm.incubator.apache.org/documentation/Guaranteein...). To my knowledge S4 offers no such guarantees. I think this made Storm more attractive for many use-cases, it did for me.

clubhi 12y ago

I've been using Storm for a few years in production. Thanks for all the hard work Nathan. You and Kyle Kingsbury are two "younger" guys that I always enjoy reading whatever you write.

Terr_ 12y ago

I've seen Samza's documentation comparing itself to Storm, I wonder what the reverse-perspective is...

wink 12y ago
As I understood it Samza ties in with Hadoop/HBase, so if you already have that, it might be easier (or even possible at all) to integrate, whereas Storm has nothing to do with with that and only does stream processing.
- bhangi 12y ago
  Samza can run on a Hadoop 2 cluster using YARN. As far as I know, there is no other connection to Hadoop or HBase.

skybrian 12y ago

How does it compare to Apache Kafka? (I haven't used either.)

akbar501 12y ago
Kafka is a high performance, persistent message queue. Storm is distributed stream processing. The two work together very well. Often, a server will publish messages to Kafka (or another queue), which are then read and processed by Storm.
- throwaway1979 12y ago
  Interesting question and response. I haven't used either but read about both systems independently. A bit more googling and I found: http://www.slideshare.net/gschmutz/kafka-andstromeventprocessinginrealtime http://www.slideshare.net/gschmutz/kafka-andstromeventproces... Slides 42 and 43 describe an architecture with all three ... hadoop, storm and kafka. Seems like a beast and I don't have applications but pretty cool combo of technologies.
  - erichmond 12y ago
    http://manning.com/marz/ http://manning.com/marz/ - this is often the trifecta used in the "lambda architecture", coined by nathan marz. This is also the system they built @ backtype to do their real-time analytics system.
    - phunge 12y ago
      Also worth checking out http://radar.oreilly.com/2014/07/questioning-the-lambda-architecture.html http://radar.oreilly.com/2014/07/questioning-the-lambda-arch... from Jay Kreps, who's part of the Kafka braintrust.
pixelmonkey 12y ago
Complementary technologies, often used together. In fact, in Storm 0.9.x, Storm actually bundles a JVM implementation of a Storm Spout that plugs directly into Kafka: https://github.com/apache/storm/tree/master/external/storm-kafka https://github.com/apache/storm/tree/master/external/storm-k...

dmritard96 12y ago

Nice read. At my previous job we used storm for all sorts of cool things. It worked great and it was a pleasure to be the one at the company that got to really cozy up with it. Deployment (with or without storm-deploy) was also well thought out.

jtheory 12y ago

I think I get the concept (and it's interesting); but I'm a bit baffled by the terminology: > That led me to the idea of "spouts" and "bolts" – a spout produces brand new streams, and a bolt takes in streams as input and produces streams as output. Spout I can grasp; why "bolt"? Why these two, together? I'd have really appreciated an extra sentence or two here as to why he chose these terms -- naming is so crucial, and the fact that both "spouts" and "bolts" sound like sources/producers to me (one of water, another of electricity...?) made it harder to grasp how Storm works.

nemothekid 12y ago
Bolt here probably references thunderbolt, and they both references (rain)storms.
- jtheory 12y ago
  That was my assumption; but if you're going to name parts of your system after natural phenomena, you have to choose ones that make some sense, not just names out of a hat that are vaguely in the same domain. "Spout" makes sense as something that's a source of a stream. I haven't been able to figure out the metaphor for "bolt" at all, particularly as something that processes water (stream/spout...) in some way.
siddboots 12y ago
I always assumed that "bolt" refers to the threaded fasteners, not electrical bolts. The analogy they are going for is that they are the components responsible for "bolting" spouts together (although, the mixing of metaphors bothers me a little bit... why are we using bolts to fasten fluids together?) To be super clear about it, a "topology" in Storm is a directed acyclical graph. The nodes are called "bolts" and the edges are called "spouts".
- dwenzek 12y ago
  > a "topology" in Storm is a directed acyclical graph. The nodes are called "bolts" and the edges are called spouts" Not really. Edges are streams and nodes are either spouts or bolts. Spouts have no predecessors and inject external data into the system. Bolts gather one or more input streams and produce zero or more output streams. These names are a bit confusing, but only during the very first steps of a project. Quickly every developers grasp the concepts. I have nevertheless a concern regarding spouts and bolts. In practice, when a topology is refactored, bolts have frequently to be transformed into spouts. This occurs when a topology is split in two to ease deployment; and when a persistent queue is added along a stream. This highlights that the distinction between spouts and bolts is bit artificial.

schultz9999 12y ago

Looks like we overloaded the site...