6 ms·

Why do you find this surprising? What other language and runtime platform provides: 1) Productive, high-level programming languages. 2) Portability to any OS

Why do you find this surprising? What other language and runtime platform provides: 1) Productive, high-level programming languages. 2) Portability to any OS. 3) High runtime performance for the server-side usage model. 4) Real shared-state SMP (necessary for memory and core efficiency). 5) Standalone, single-directory distribution of the applications and all library dependencies. 6) A broad, stable, mature base of well-documented, well-tested libraries. The CLR has some advantages over the JVM, but it also shares most of the JVM's attributes.

vegabook 11y ago
A mature, reliable, performant, but unadventurous platform which attracts corporates. I am not knocking it - I think it's obvious that the JVM is well suited, but isn't there room for at least one or two competitors on other platforms? Do any exist that you know of? I tend to be a bit more adventurous in my coding projects and given that embarking on projects requires one to be enthusiastic about one's tools, I cannot use this as Java leaves me cold. This sounds flippant but really I don't think it is. I love the idea of DAG workflows but I don't want to do it in Java, even with all its strengths.
- dm3 11y ago
  The other benefit of the JVM is the multiple languages that run on it - did you try Clojure or JRuby already? Or do you mean Java as in JVM and you don't want that in your stack for some reason?
  - vegabook 11y ago
    I am not a fan of any compatibility layer technology as I am a bit of a bare-metal purist, and I believe Linux does this already. The original post waxes lyrical about the Unix philosophy, but part of the Unix philosophy is supreme efficiency and this is violated by the JVM. I know I know - performance is comparable but then coding is a bit like art - you have to love what you are doing. As for using other languages - I played with Python on Storm but I very quickly found that I had to basically use Java because the entire community and documentation was java-centric (last I checked - a few months ago). I may be being obtuse here - in fact I know I am - but my point is that there is a large community out there which is not Java friendly (logically or not) and so I lament the fact that all the advancement in this new, very exciting, field, is JVM-based. My dislike for Java was sealed by the Bloomberg terminal API, some of which's basic functionality is 5 to 6 dots deep. Seriously I had 80+ character function calls. This.that.that.this.this.finally()! Perhaps we'll just have to kick in, leave our biases behind and start running with the JVM. FWIW my use case is streaming fixed income (bond pricing) data analysis. Python/Numpy is running out of steam fast for me and we're using quite a bit of C. I basically come from the scientific computing set. We're applying advanced statistical analysis to pricing (not HFT - we're operating in the 5 minute thru 1-week horizon - not 5ns).
    - pixelmonkey 11y ago
      If you're interested in Python + Storm, we are making it happen via our open source project streamparse. See the project on Github[1] and also my PyCon US 2015 talk[2]. Your point about documentation is well-taken. We are trying to document streamparse + Storm usage from a Pythonista standpoint via our online documentation[3], e.g. here is our detailed Python API documentation[4]. [1]: https://github.com/Parsely/streamparse https://github.com/Parsely/streamparse [2]: https://www.youtube.com/watch?v=ja4Qj9-l6WQ https://www.youtube.com/watch?v=ja4Qj9-l6WQ [3]: http://streamparse.readthedocs.org/en/stable/ http://streamparse.readthedocs.org/en/stable/ [4]: http://streamparse.readthedocs.org/en/stable/api.html http://streamparse.readthedocs.org/en/stable/api.html
    - teacup50 11y ago
      In one breath "I am a bit of a bare-metal purist" and in the next "As for using other languages - I played with Python on Storm". So which one is it? :-)
      - srean 11y ago
        I am not the OP but it is not necessarily a contradiction. The underlying goal in Pythonic data-science is that one pushes as much of the computation into pre-packaged loops that have already been written in a close to the metal language (C, C++, Fortran). Well, that's as far as the intent goes, practice deviates from it by degrees. This does not work quite as well in Java (or JVM) because JNI is just supremely god-awful. JVM semantics are overly strict, this over-specification kills optimization opportunities. Is heavy on memory, I would rather use the memory for loading more data than fill it with overhead. Finally there is this OOP culture that gets in the way. That said, JVM is one of the most mature, and well engineered VMs out there (Java is another story), but not very well suited for number crunching, because there is more to number crunching than calling BLAS APIs.
      - vegabook 11y ago
        last time I checked it was Java Scala or Python. None is bare metal. We use Python as glue. Any heavy lifting is done using Numpy ecosystem or C.
- teacup50 11y ago
  That's why I highlighted the platform strengths -- what others platforms can reasonably compete? That said, you can use these tools as complete applications without actually dipping your toes into Java (or even the JVM, other than running it). If you do want to write code directly against them, the choices (of which I'm sure you'll well aware) range from Clojure to Scala to Java to JRuby. Scala is one ugly duck, but personally I find it to be a sufficient language for writing JVM-based code without losing my lunch.
- squarecog 11y ago
  Well, first, a lot of this stuff is actually written in Scala, which is like Java's younger and more fun cousin. Still runs on the JVM but has many fun language aspects I won't get into because there's already a billion posts online about that. But second, not quite a "platform" but I've been enjoying Frank McSherry's blog posts in which he builds up a complete distributed dataflow engine in Rust: http://www.frankmcsherry.org/ http://www.frankmcsherry.org/
- a8da6b0c91d 11y ago
  I'm sure I'm being stupid and underinformed, but it doesn't seem like these systems do anything you can't replicate easily enough with htcondor/gridengine and so-forth alongside cluster computing filesystems. It almost seems like Java people decided they wanted Java only in their stack, so they ignored all the existing unix cluster computing stuff. Somebody's downvoting me but I doubt they know anything about streaming and short-job support in gridengine.
- threeseed 11y ago
  Scala/Python using Spark Streaming.
srean 11y ago
C++11 perhaps.
- teacup50 11y ago
  Precluded by requirement #1 (which I say as someone who has written an awful lot of C++).
  - herewego 11y ago
    That's subjective. I'm quite productive writing system software in C++11/14.
dap 11y ago
I disagree that the JVM is the only platform on which you could (or would) build such a system. I know because we built a similar system at Joyent called Manta, which provides a distributed, Unix-based, map-reduce environment. It's built almost entirely in Node.js. We use nginx for serving raw data from the storage tier and PostgreSQL for storing and querying metadata (as well as compute job state), but job supervision and actual execution are implemented in Node.js.
- dap 11y ago
  Sorry for the self-reply, but to clarify: the _implementation_ is Node.js. Users can run their actual workloads using any program they want.

vegabook 11y ago

A mature, reliable, performant, but unadventurous platform which attracts corporates. I am not knocking it - I think it's obvious that the JVM is well suited, but isn't there room for at least one or two competitors on other platforms? Do any exist that you know of? I tend to be a bit more adventurous in my coding projects and given that embarking on projects requires one to be enthusiastic about one's tools, I cannot use this as Java leaves me cold. This sounds flippant but really I don't think it is. I love the idea of DAG workflows but I don't want to do it in Java, even with all its strengths.

dm3 11y ago
The other benefit of the JVM is the multiple languages that run on it - did you try Clojure or JRuby already? Or do you mean Java as in JVM and you don't want that in your stack for some reason?
- vegabook 11y ago
  I am not a fan of any compatibility layer technology as I am a bit of a bare-metal purist, and I believe Linux does this already. The original post waxes lyrical about the Unix philosophy, but part of the Unix philosophy is supreme efficiency and this is violated by the JVM. I know I know - performance is comparable but then coding is a bit like art - you have to love what you are doing. As for using other languages - I played with Python on Storm but I very quickly found that I had to basically use Java because the entire community and documentation was java-centric (last I checked - a few months ago). I may be being obtuse here - in fact I know I am - but my point is that there is a large community out there which is not Java friendly (logically or not) and so I lament the fact that all the advancement in this new, very exciting, field, is JVM-based. My dislike for Java was sealed by the Bloomberg terminal API, some of which's basic functionality is 5 to 6 dots deep. Seriously I had 80+ character function calls. This.that.that.this.this.finally()! Perhaps we'll just have to kick in, leave our biases behind and start running with the JVM. FWIW my use case is streaming fixed income (bond pricing) data analysis. Python/Numpy is running out of steam fast for me and we're using quite a bit of C. I basically come from the scientific computing set. We're applying advanced statistical analysis to pricing (not HFT - we're operating in the 5 minute thru 1-week horizon - not 5ns).
  - pixelmonkey 11y ago
    If you're interested in Python + Storm, we are making it happen via our open source project streamparse. See the project on Github[1] and also my PyCon US 2015 talk[2]. Your point about documentation is well-taken. We are trying to document streamparse + Storm usage from a Pythonista standpoint via our online documentation[3], e.g. here is our detailed Python API documentation[4]. [1]: https://github.com/Parsely/streamparse https://github.com/Parsely/streamparse [2]: https://www.youtube.com/watch?v=ja4Qj9-l6WQ https://www.youtube.com/watch?v=ja4Qj9-l6WQ [3]: http://streamparse.readthedocs.org/en/stable/ http://streamparse.readthedocs.org/en/stable/ [4]: http://streamparse.readthedocs.org/en/stable/api.html http://streamparse.readthedocs.org/en/stable/api.html
  - teacup50 11y ago
    In one breath "I am a bit of a bare-metal purist" and in the next "As for using other languages - I played with Python on Storm". So which one is it? :-)
    - srean 11y ago
      I am not the OP but it is not necessarily a contradiction. The underlying goal in Pythonic data-science is that one pushes as much of the computation into pre-packaged loops that have already been written in a close to the metal language (C, C++, Fortran). Well, that's as far as the intent goes, practice deviates from it by degrees. This does not work quite as well in Java (or JVM) because JNI is just supremely god-awful. JVM semantics are overly strict, this over-specification kills optimization opportunities. Is heavy on memory, I would rather use the memory for loading more data than fill it with overhead. Finally there is this OOP culture that gets in the way. That said, JVM is one of the most mature, and well engineered VMs out there (Java is another story), but not very well suited for number crunching, because there is more to number crunching than calling BLAS APIs.
    - vegabook 11y ago
      last time I checked it was Java Scala or Python. None is bare metal. We use Python as glue. Any heavy lifting is done using Numpy ecosystem or C.
teacup50 11y ago
That's why I highlighted the platform strengths -- what others platforms can reasonably compete? That said, you can use these tools as complete applications without actually dipping your toes into Java (or even the JVM, other than running it). If you do want to write code directly against them, the choices (of which I'm sure you'll well aware) range from Clojure to Scala to Java to JRuby. Scala is one ugly duck, but personally I find it to be a sufficient language for writing JVM-based code without losing my lunch.
squarecog 11y ago
Well, first, a lot of this stuff is actually written in Scala, which is like Java's younger and more fun cousin. Still runs on the JVM but has many fun language aspects I won't get into because there's already a billion posts online about that. But second, not quite a "platform" but I've been enjoying Frank McSherry's blog posts in which he builds up a complete distributed dataflow engine in Rust: http://www.frankmcsherry.org/ http://www.frankmcsherry.org/
a8da6b0c91d 11y ago
I'm sure I'm being stupid and underinformed, but it doesn't seem like these systems do anything you can't replicate easily enough with htcondor/gridengine and so-forth alongside cluster computing filesystems. It almost seems like Java people decided they wanted Java only in their stack, so they ignored all the existing unix cluster computing stuff. Somebody's downvoting me but I doubt they know anything about streaming and short-job support in gridengine.
threeseed 11y ago
Scala/Python using Spark Streaming.

srean 11y ago

C++11 perhaps.

teacup50 11y ago
Precluded by requirement #1 (which I say as someone who has written an awful lot of C++).
- herewego 11y ago
  That's subjective. I'm quite productive writing system software in C++11/14.

dap 11y ago

I disagree that the JVM is the only platform on which you could (or would) build such a system. I know because we built a similar system at Joyent called Manta, which provides a distributed, Unix-based, map-reduce environment. It's built almost entirely in Node.js. We use nginx for serving raw data from the storage tier and PostgreSQL for storing and querying metadata (as well as compute job state), but job supervision and actual execution are implemented in Node.js.

dap 11y ago
Sorry for the self-reply, but to clarify: the _implementation_ is Node.js. Users can run their actual workloads using any program they want.