I'll preface this with I'm one of the authors of Storm Applied. A book about Storm from Manning. There's a number of performance tips, best practices and what

I'll preface this with I'm one of the authors of Storm Applied. A book about Storm from Manning. There's a number of performance tips, best practices and what not in Storm Applied. One of the things we did when building Wallaroo was to build such best practices and tips into Wallaroo itself so that application programmers didn't have to think about them. Many of the systems I've built using Storm, needed to keep state in memory in order to achieve the performance we needed, trying to rapidly calculate new results and save those results to an external store (for recovery in case of failure) was problematic: 1) performance was almost always bounded by the speed of the external data store 2) We had to roll our own idempotence mechanisms (which invariably had bugs) Wallaroo features integrated state management. Rather than creating fields inside of bolts and managing state that way, Wallaroo's APIs allow you to setup state objects that Wallaroo manages. You define the objects, the basic partitioning scheme and how messages get routed to them and Wallaroo manages them for you. When you have a message that needs to operate on state, your computation is given both the message and the data to operate on. This allows a number of interesting features: - You can scale state across a running cluster, adding more servers or removing some and Wallaroo can redistribute your state for you. Our 15-minute demo video shows this in action: https://vimeo.com/234753585 https://vimeo.com/234753585 - You can much more tightly integrate failure handling and resilience and avoid a number of potentially nasty edge cases. - (Coming soon, not yet in Wallaroo). You can create new state objects and change partitioning on the fly. For example, in Storm if you are doing word count, you use a fields grouping. You set up a number of tasks for counting words (fixed) and each bolt has a map of words to count that it handles. If you created 128 tasks then that is the limit of what you can parallelize to while running. We are working on "dynamic partition keys" in Wallaroo. What this means is that I can have a state object per word in Wallaroo. Rather than having a fixed number of buckets to store word counts in (which limits parallelization), you can have 1 for every word you encounter. In addition to allowing for parallelization that can grow with the shape of your data, it also allows for some interesting characteristics for simplying business logic (which I'm not going to cover now as its a bit involved to explain without a lot of code samples). Lastly, this is going to allow us to create replicas of data across a cluster of Wallaroo workers. Which should end up allowing for a number of nice characteristics. Wallaroo's approach to state is very much in line with Pat Helland's paper "Beyond Distributed Transactions". You can find a link to it (and other papers/talks that we hope more people are exposed to) at https://www.wallaroolabs.com/community/talks https://www.wallaroolabs.com/community/talks --- Storm is JVM based, if you want to use a non-jvm language, you need to use its multi-language support which involves marshalling data between the jvm and an external process via Thrift. There are pros and cons to this approach. Its great for making it easier to support new languages. It has performance and operational overhead. If you use a JVM language with Storm (Java, Scala, Clojure) there is no such overhead as thrift isn't involved. Wallaroo is not JVM based. Our cost of adding support for new languages is higher but we have better performance and operational stories. In the case of a language like C++ or Go, your application code is compiled together with the Wallaroo framework code into a single binary that you distribute. Your application is that binary. You set up a cluster by having the different running instance communicate with one another. This means, that all state is and messages remain in a single process (except when moving from one cluster member to another), which would provide less overhead if you wanted to do stream processing with a language like Go or C++ that isn't JVM native. When considering how our Python implementation works, its similiar except, we embed a Python interpreter along with the Wallaroo framework code. It works in the same fashion as the means described above with C++ and Go except, you need to distribute that runner application (Wallaroo + Python), your Python code and any required Python libraries together. It's slightly less self contained than the single binary you get with a compiled language like C++. --- Storm generally provides minimum latencies that are measured in milliseconds. Wallaroo applications can be microseconds. --- Storm provides at-least-once message processing or at-most-once message processing. You can do microbatching with Trident to get "exactly-once" processing but the perform is quite a lot worse than you get with Storm's event-by-event APIs. Wallaroo provides exactly-once processing while doing event-by-event processing. There's a small amount of latency overhead and at the moment and a distinct drop in throughput. Our reference application that we use that has minimal business logic (allowing us to more easily measure the overhead that Wallaroo is added) has a 10x drop in throughput when running with all our resilience features on. It should be noted however that even with that drop, we are processing hundreds of thousands of events per second with tail latencies measured in single digit milliseconds. And we have plenty more tuning to do there an expect it to get better. --- There are other differences but those are a couple I'd like to highlight. My basic statement to Storm users would be. If you: - use a JVM based language - are generally happy with Storm then Wallaroo isn't for you. But if that doesn't describe you, then you should keep an eye on Wallaroo. We are actively developing and will be releasing a number of improvements over the coming months. For some use-cases, Wallaroo is usable now. For others, it isn't yet but we are actively working to expand the types of applications that Wallaroo is good for. We are happy to chat over email, IRC etc with folks to learn more about their use case. One of the big reasons that we released Wallaroo now rather than waiting is we wanted to expand the pool of people whose use cases would be presented to us. We want to make building data processing applications much easier than it currently is. A big part of that is hearing the types of applications you are building, the types you can't currently build because its too hard, the problems you are having with existing tools. Hope that answer helped you some.