3 ms·

Not sure the JVM world is really even targeting MPI. As an author of a competing lib to this with traction in the big data world (we are mainly used on large sp

Not sure the JVM world is really even targeting MPI. As an author of a competing lib to this with traction in the big data world (we are mainly used on large spark/flink gpu mixed cpu/gpu clusters), I'd actually do the reverse pitch on you here and say "why should we care about HPC?" Usually the folks using MPI and the like either know python or c++ and are fine with that ecosystem where it's already well served often directly from vendors like nvidia and intel.

neutronicus 9y ago
Ha. Myopia on my part, I guess. I am, in fact, a Python / C++ user who probably isn't switching to Clojure any time soon (unless I switch jobs). I still think that these posts are a little light on what exactly the library's cluster story is. Your library, for instance, says "Distributed" right there on the landing page, which is already super helpful for me to know what I'm looking at at a glance.
- agibsonccc 9y ago
  Yeah! Totally get it. The "distributed" part is an overloaded term though which is why I called out the "JVM version" which is more focused on commodity boxes. I should note that ours also isn't "GPU first" but more "GPUs as an addon to your cluster for Deep learning" which is how the resource managers typically used in JVM land like mesos and yarn are treating it.
fulafel 9y ago
Maybe not the MPI protocol specifically, but there are probably more users and implementors of HPC style applications in the business world using Java than in the scientific computing world. It's become mainstream with all the ready to run data-crunching-in-cluster stuff available on AWS. Stuff like: http://spark.apache.org/news/spark-wins-daytona-gray-sort-100tb-benchmark.html http://spark.apache.org/news/spark-wins-daytona-gray-sort-10...
- agibsonccc 9y ago
  Right that's what I was talking about though. I mentioned spark and flink in my original comment. Spark makes different assumptions about the computers than what MPI does. "HPC" is 99% of the time some variant of MPI with some sort of special hardware interconnect like infiniband. It's just in general a different software stack with different assumptions. Like I said- I work on gpu stuff for the JVM as well, the assumption is different with how the cluster is expected to be setup though.

neutronicus 9y ago

Ha. Myopia on my part, I guess. I am, in fact, a Python / C++ user who probably isn't switching to Clojure any time soon (unless I switch jobs). I still think that these posts are a little light on what exactly the library's cluster story is. Your library, for instance, says "Distributed" right there on the landing page, which is already super helpful for me to know what I'm looking at at a glance.

agibsonccc 9y ago
Yeah! Totally get it. The "distributed" part is an overloaded term though which is why I called out the "JVM version" which is more focused on commodity boxes. I should note that ours also isn't "GPU first" but more "GPUs as an addon to your cluster for Deep learning" which is how the resource managers typically used in JVM land like mesos and yarn are treating it.

fulafel 9y ago

Maybe not the MPI protocol specifically, but there are probably more users and implementors of HPC style applications in the business world using Java than in the scientific computing world. It's become mainstream with all the ready to run data-crunching-in-cluster stuff available on AWS. Stuff like: http://spark.apache.org/news/spark-wins-daytona-gray-sort-100tb-benchmark.html http://spark.apache.org/news/spark-wins-daytona-gray-sort-10...

agibsonccc 9y ago
Right that's what I was talking about though. I mentioned spark and flink in my original comment. Spark makes different assumptions about the computers than what MPI does. "HPC" is 99% of the time some variant of MPI with some sort of special hardware interconnect like infiniband. It's just in general a different software stack with different assumptions. Like I said- I work on gpu stuff for the JVM as well, the assumption is different with how the cluster is expected to be setup though.