5 ms·

It would be even closer to the Unix philosophy if a competitive distributed graph processing framework such as these could became available on something other t

It would be even closer to the Unix philosophy if a competitive distributed graph processing framework such as these could became available on something other than the JVM / Java ecosystem. I personally am not using either for this exact reason and hope that Golang's goroutines will get their own distributed processing analogue, or that Mirage OS will take off. I am amazed that these systems, Storm / Samza / Spark / even the new Flink are all Java / Scala based.

teacup50 11y ago
Why do you find this surprising? What other language and runtime platform provides: 1) Productive, high-level programming languages. 2) Portability to any OS. 3) High runtime performance for the server-side usage model. 4) Real shared-state SMP (necessary for memory and core efficiency). 5) Standalone, single-directory distribution of the applications and all library dependencies. 6) A broad, stable, mature base of well-documented, well-tested libraries. The CLR has some advantages over the JVM, but it also shares most of the JVM's attributes.
- vegabook 11y ago
  A mature, reliable, performant, but unadventurous platform which attracts corporates. I am not knocking it - I think it's obvious that the JVM is well suited, but isn't there room for at least one or two competitors on other platforms? Do any exist that you know of? I tend to be a bit more adventurous in my coding projects and given that embarking on projects requires one to be enthusiastic about one's tools, I cannot use this as Java leaves me cold. This sounds flippant but really I don't think it is. I love the idea of DAG workflows but I don't want to do it in Java, even with all its strengths.
  - dm3 11y ago
    The other benefit of the JVM is the multiple languages that run on it - did you try Clojure or JRuby already? Or do you mean Java as in JVM and you don't want that in your stack for some reason?
    - vegabook 11y ago
      I am not a fan of any compatibility layer technology as I am a bit of a bare-metal purist, and I believe Linux does this already. The original post waxes lyrical about the Unix philosophy, but part of the Unix philosophy is supreme efficiency and this is violated by the JVM. I know I know - performance is comparable but then coding is a bit like art - you have to love what you are doing. As for using other languages - I played with Python on Storm but I very quickly found that I had to basically use Java because the entire community and documentation was java-centric (last I checked - a few months ago). I may be being obtuse here - in fact I know I am - but my point is that there is a large community out there which is not Java friendly (logically or not) and so I lament the fact that all the advancement in this new, very exciting, field, is JVM-based. My dislike for Java was sealed by the Bloomberg terminal API, some of which's basic functionality is 5 to 6 dots deep. Seriously I had 80+ character function calls. This.that.that.this.this.finally()! Perhaps we'll just have to kick in, leave our biases behind and start running with the JVM. FWIW my use case is streaming fixed income (bond pricing) data analysis. Python/Numpy is running out of steam fast for me and we're using quite a bit of C. I basically come from the scientific computing set. We're applying advanced statistical analysis to pricing (not HFT - we're operating in the 5 minute thru 1-week horizon - not 5ns).
      - pixelmonkey 11y ago
        If you're interested in Python + Storm, we are making it happen via our open source project streamparse. See the project on Github[1] and also my PyCon US 2015 talk[2]. Your point about documentation is well-taken. We are trying to document streamparse + Storm usage from a Pythonista standpoint via our online documentation[3], e.g. here is our detailed Python API documentation[4]. [1]: https://github.com/Parsely/streamparse https://github.com/Parsely/streamparse [2]: https://www.youtube.com/watch?v=ja4Qj9-l6WQ https://www.youtube.com/watch?v=ja4Qj9-l6WQ [3]: http://streamparse.readthedocs.org/en/stable/ http://streamparse.readthedocs.org/en/stable/ [4]: http://streamparse.readthedocs.org/en/stable/api.html http://streamparse.readthedocs.org/en/stable/api.html
      - teacup50 11y ago
        In one breath "I am a bit of a bare-metal purist" and in the next "As for using other languages - I played with Python on Storm". So which one is it? :-)
  - teacup50 11y ago
    That's why I highlighted the platform strengths -- what others platforms can reasonably compete? That said, you can use these tools as complete applications without actually dipping your toes into Java (or even the JVM, other than running it). If you do want to write code directly against them, the choices (of which I'm sure you'll well aware) range from Clojure to Scala to Java to JRuby. Scala is one ugly duck, but personally I find it to be a sufficient language for writing JVM-based code without losing my lunch.
  - squarecog 11y ago
    Well, first, a lot of this stuff is actually written in Scala, which is like Java's younger and more fun cousin. Still runs on the JVM but has many fun language aspects I won't get into because there's already a billion posts online about that. But second, not quite a "platform" but I've been enjoying Frank McSherry's blog posts in which he builds up a complete distributed dataflow engine in Rust: http://www.frankmcsherry.org/ http://www.frankmcsherry.org/
  - a8da6b0c91d 11y ago
    I'm sure I'm being stupid and underinformed, but it doesn't seem like these systems do anything you can't replicate easily enough with htcondor/gridengine and so-forth alongside cluster computing filesystems. It almost seems like Java people decided they wanted Java only in their stack, so they ignored all the existing unix cluster computing stuff. Somebody's downvoting me but I doubt they know anything about streaming and short-job support in gridengine.
  - threeseed 11y ago
    Scala/Python using Spark Streaming.
- srean 11y ago
  C++11 perhaps.
  - teacup50 11y ago
    Precluded by requirement #1 (which I say as someone who has written an awful lot of C++).
    - herewego 11y ago
      That's subjective. I'm quite productive writing system software in C++11/14.
- dap 11y ago
  I disagree that the JVM is the only platform on which you could (or would) build such a system. I know because we built a similar system at Joyent called Manta, which provides a distributed, Unix-based, map-reduce environment. It's built almost entirely in Node.js. We use nginx for serving raw data from the storage tier and PostgreSQL for storing and querying metadata (as well as compute job state), but job supervision and actual execution are implemented in Node.js.
  - dap 11y ago
    Sorry for the self-reply, but to clarify: the _implementation_ is Node.js. Users can run their actual workloads using any program they want.
pixelmonkey 11y ago
One of the nice things about Storm, in particular, is that the creator of the project thought about multiple language support from the start. We use Storm via our open source project, streamparse[1], and this enables usage of normal Python programs running under a Storm topology. In our case, we achieve multi-process parallelism and fault tolerance using Storm's simple multi-lang protocol, which is abstracted by our library. We also use Kafka natively from Python via the other module we released publicly, pykafka[2]. It provides support for nearly all of the Kafka protocol, and we even have an optimized C extension module in the works that may even be faster than the JVM consumer. My view on this is that infrastructure has to be implemented in some language. I don't lament the fact that Postgres was written in C or that Cassandra was written in Java -- I just use these technologies as infrastructure. Both have very full-featured Python bindings. The only thing missing on the Storm/Kafka/streams side are the full-featured Python bindings. But my team at Parse.ly has been hard at work building those in open source. Help us! [1]: https://github.com/Parsely/streamparse https://github.com/Parsely/streamparse [2]: https://github.com/Parsely/pykafka https://github.com/Parsely/pykafka
- infinite8s 11y ago
  How do you find Storm's scalability and resource efficiency? Its interesting that Twitter themselves have moved away from Storm (and have even published a paper about it - http://blog.acolyer.org/2015/06/15/twitter-heron-stream-processing-at-scale/ http://blog.acolyer.org/2015/06/15/twitter-heron-stream-proc...)
  - pixelmonkey 11y ago
    I find it scales well and uses resources well, within the limitations of its design. For example, Storm does not attempt to handle concerns like resource rebalancing, contention management, or auto-scaling. It has a simple REBALANCE command, but this amounts to killing all your workers and randomly redistributing them in your cluster. It isn't a smart rebalancer based on load or throughput, or anything. My understanding is that Twitter's rewrite of Storm (Heron) was mainly to make it work with Mesos as a resource manager. This is probably wise since Mesos handles a lot of the concerns I described above. But Mesos didn't really exist when Storm was written. They chose to re-implement 100% of Storm's API in doing so, which perhaps shows that the high level concept of the framework has staying power even though you might benefit from a more advanced resource manager at 1,000-node scale. I wish they had reimplemented it as a competing implementation and actually released it as open source. But no, they decided to keep it 100% proprietary. So it goes. I guess once companies become a certain size, they turn their back on open source if it's not directly in their interest any longer. We run it with 10-15 Storm nodes with 32 cores each and find it to be immensely helpful in this context, keeping each node in the cluster lit up to ~60% CPU utilization and plowing through 10K events/second on a Kafka topic, despite the fact that we are using Python and there isn't a line of code using threads or process management. People are often shocked we pull this off, since, in theory, CPython's GIL means you can't even run on more than one core at a time. But we write simple Python programs that run on Storm and utilize hundreds of cores at once, across multiple machines. And we get the Erlang-style "let it crash" / "fail fast" process supervision for free. I have a more cynical view of the Heron paper overall -- if you're curious about that, reach out to me directly (@amontalenti on Twitter).

teacup50 11y ago

Why do you find this surprising? What other language and runtime platform provides: 1) Productive, high-level programming languages. 2) Portability to any OS. 3) High runtime performance for the server-side usage model. 4) Real shared-state SMP (necessary for memory and core efficiency). 5) Standalone, single-directory distribution of the applications and all library dependencies. 6) A broad, stable, mature base of well-documented, well-tested libraries. The CLR has some advantages over the JVM, but it also shares most of the JVM's attributes.

vegabook 11y ago
A mature, reliable, performant, but unadventurous platform which attracts corporates. I am not knocking it - I think it's obvious that the JVM is well suited, but isn't there room for at least one or two competitors on other platforms? Do any exist that you know of? I tend to be a bit more adventurous in my coding projects and given that embarking on projects requires one to be enthusiastic about one's tools, I cannot use this as Java leaves me cold. This sounds flippant but really I don't think it is. I love the idea of DAG workflows but I don't want to do it in Java, even with all its strengths.
- dm3 11y ago
  The other benefit of the JVM is the multiple languages that run on it - did you try Clojure or JRuby already? Or do you mean Java as in JVM and you don't want that in your stack for some reason?
  - vegabook 11y ago
    I am not a fan of any compatibility layer technology as I am a bit of a bare-metal purist, and I believe Linux does this already. The original post waxes lyrical about the Unix philosophy, but part of the Unix philosophy is supreme efficiency and this is violated by the JVM. I know I know - performance is comparable but then coding is a bit like art - you have to love what you are doing. As for using other languages - I played with Python on Storm but I very quickly found that I had to basically use Java because the entire community and documentation was java-centric (last I checked - a few months ago). I may be being obtuse here - in fact I know I am - but my point is that there is a large community out there which is not Java friendly (logically or not) and so I lament the fact that all the advancement in this new, very exciting, field, is JVM-based. My dislike for Java was sealed by the Bloomberg terminal API, some of which's basic functionality is 5 to 6 dots deep. Seriously I had 80+ character function calls. This.that.that.this.this.finally()! Perhaps we'll just have to kick in, leave our biases behind and start running with the JVM. FWIW my use case is streaming fixed income (bond pricing) data analysis. Python/Numpy is running out of steam fast for me and we're using quite a bit of C. I basically come from the scientific computing set. We're applying advanced statistical analysis to pricing (not HFT - we're operating in the 5 minute thru 1-week horizon - not 5ns).
    - pixelmonkey 11y ago
      If you're interested in Python + Storm, we are making it happen via our open source project streamparse. See the project on Github[1] and also my PyCon US 2015 talk[2]. Your point about documentation is well-taken. We are trying to document streamparse + Storm usage from a Pythonista standpoint via our online documentation[3], e.g. here is our detailed Python API documentation[4]. [1]: https://github.com/Parsely/streamparse https://github.com/Parsely/streamparse [2]: https://www.youtube.com/watch?v=ja4Qj9-l6WQ https://www.youtube.com/watch?v=ja4Qj9-l6WQ [3]: http://streamparse.readthedocs.org/en/stable/ http://streamparse.readthedocs.org/en/stable/ [4]: http://streamparse.readthedocs.org/en/stable/api.html http://streamparse.readthedocs.org/en/stable/api.html
    - teacup50 11y ago
      In one breath "I am a bit of a bare-metal purist" and in the next "As for using other languages - I played with Python on Storm". So which one is it? :-)
- teacup50 11y ago
  That's why I highlighted the platform strengths -- what others platforms can reasonably compete? That said, you can use these tools as complete applications without actually dipping your toes into Java (or even the JVM, other than running it). If you do want to write code directly against them, the choices (of which I'm sure you'll well aware) range from Clojure to Scala to Java to JRuby. Scala is one ugly duck, but personally I find it to be a sufficient language for writing JVM-based code without losing my lunch.
- squarecog 11y ago
  Well, first, a lot of this stuff is actually written in Scala, which is like Java's younger and more fun cousin. Still runs on the JVM but has many fun language aspects I won't get into because there's already a billion posts online about that. But second, not quite a "platform" but I've been enjoying Frank McSherry's blog posts in which he builds up a complete distributed dataflow engine in Rust: http://www.frankmcsherry.org/ http://www.frankmcsherry.org/
- a8da6b0c91d 11y ago
  I'm sure I'm being stupid and underinformed, but it doesn't seem like these systems do anything you can't replicate easily enough with htcondor/gridengine and so-forth alongside cluster computing filesystems. It almost seems like Java people decided they wanted Java only in their stack, so they ignored all the existing unix cluster computing stuff. Somebody's downvoting me but I doubt they know anything about streaming and short-job support in gridengine.
- threeseed 11y ago
  Scala/Python using Spark Streaming.
srean 11y ago
C++11 perhaps.
- teacup50 11y ago
  Precluded by requirement #1 (which I say as someone who has written an awful lot of C++).
  - herewego 11y ago
    That's subjective. I'm quite productive writing system software in C++11/14.
dap 11y ago
I disagree that the JVM is the only platform on which you could (or would) build such a system. I know because we built a similar system at Joyent called Manta, which provides a distributed, Unix-based, map-reduce environment. It's built almost entirely in Node.js. We use nginx for serving raw data from the storage tier and PostgreSQL for storing and querying metadata (as well as compute job state), but job supervision and actual execution are implemented in Node.js.
- dap 11y ago
  Sorry for the self-reply, but to clarify: the _implementation_ is Node.js. Users can run their actual workloads using any program they want.

pixelmonkey 11y ago

One of the nice things about Storm, in particular, is that the creator of the project thought about multiple language support from the start. We use Storm via our open source project, streamparse[1], and this enables usage of normal Python programs running under a Storm topology. In our case, we achieve multi-process parallelism and fault tolerance using Storm's simple multi-lang protocol, which is abstracted by our library. We also use Kafka natively from Python via the other module we released publicly, pykafka[2]. It provides support for nearly all of the Kafka protocol, and we even have an optimized C extension module in the works that may even be faster than the JVM consumer. My view on this is that infrastructure has to be implemented in some language. I don't lament the fact that Postgres was written in C or that Cassandra was written in Java -- I just use these technologies as infrastructure. Both have very full-featured Python bindings. The only thing missing on the Storm/Kafka/streams side are the full-featured Python bindings. But my team at Parse.ly has been hard at work building those in open source. Help us! [1]: https://github.com/Parsely/streamparse https://github.com/Parsely/streamparse [2]: https://github.com/Parsely/pykafka https://github.com/Parsely/pykafka

infinite8s 11y ago
How do you find Storm's scalability and resource efficiency? Its interesting that Twitter themselves have moved away from Storm (and have even published a paper about it - http://blog.acolyer.org/2015/06/15/twitter-heron-stream-processing-at-scale/ http://blog.acolyer.org/2015/06/15/twitter-heron-stream-proc...)
- pixelmonkey 11y ago
  I find it scales well and uses resources well, within the limitations of its design. For example, Storm does not attempt to handle concerns like resource rebalancing, contention management, or auto-scaling. It has a simple REBALANCE command, but this amounts to killing all your workers and randomly redistributing them in your cluster. It isn't a smart rebalancer based on load or throughput, or anything. My understanding is that Twitter's rewrite of Storm (Heron) was mainly to make it work with Mesos as a resource manager. This is probably wise since Mesos handles a lot of the concerns I described above. But Mesos didn't really exist when Storm was written. They chose to re-implement 100% of Storm's API in doing so, which perhaps shows that the high level concept of the framework has staying power even though you might benefit from a more advanced resource manager at 1,000-node scale. I wish they had reimplemented it as a competing implementation and actually released it as open source. But no, they decided to keep it 100% proprietary. So it goes. I guess once companies become a certain size, they turn their back on open source if it's not directly in their interest any longer. We run it with 10-15 Storm nodes with 32 cores each and find it to be immensely helpful in this context, keeping each node in the cluster lit up to ~60% CPU utilization and plowing through 10K events/second on a Kafka topic, despite the fact that we are using Python and there isn't a line of code using threads or process management. People are often shocked we pull this off, since, in theory, CPython's GIL means you can't even run on more than one core at a time. But we write simple Python programs that run on Storm and utilize hundreds of cores at once, across multiple machines. And we get the Erlang-style "let it crash" / "fail fast" process supervision for free. I have a more cynical view of the Heron paper overall -- if you're curious about that, reach out to me directly (@amontalenti on Twitter).