4 ms·

> Pachyderm will eventually be a complete replacement for Hadoop, built on top of a modern toolchain instead of the JVM. Is the JVM legitimately not considered

by tiles 12y ago

> Pachyderm will eventually be a complete replacement for Hadoop, built on top of a modern toolchain instead of the JVM. Is the JVM legitimately not considered a modern toolchain? I understand the benefits of leveraging Docker, but this comparison seems short-sighted.

jdoliner 12y ago
This is a fair criticism and I think we used unclear phrasing here. It's not that we don't consider the JVM to be a modern toolchain in general. Rather we think it's not a modern toolchain for implementing MapReduce because it forces users to work in a JVM compatible way. Docker is a better solution because it gives users the freedom to use what they're already using.
- valarauca1 12y ago
  Doesn't this approach make users send 1GB+ Docker images instead of small 5-10MB .jar files?
  - jdoliner 12y ago
    Useful Docker images can actually be made quite a bit smaller than 1GB+ by using small distros like busybox and following best practices such as cleaning up temporary data between commits. That being said this is a legitimate concern because it affects how tight a developers loop is so it's something we'll probably have to address longterm.
  - coffeemug 12y ago
    That sounds relatively easy to fix. I'd imagine a diff between two Docker images is relatively small compared to the size of the image. I'd imagine a deduplication scheme based on widely available tools (e.g. rsync) would solve this problems fairly easily. (Though I have no idea how hard it would be to implement; it doesn't sound too hard off the top of my head)
    - jdoliner 12y ago
      This is a very good point. In addition Docker itself tries to address this by using btrfs's diffing. Like a lot of things in Docker it's not perfect but it's getting there.
- teacup50 12y ago
  How is schlepping around non-portable Linux-specific binaries packaged into what is essentially a VM a "more modern toolchain"?
  - shykes 12y ago
    (just my personal opinion below, I'm not affiliated with the grand-parent) The tooling around the JVM is much more mature the Docker ecosystem in a lot of ways. However, one way to justify the "more modern" label is that Docker can be used to package and manage every component of a Linux-based application stack, whereas JVM tools can only run the JVM-backed pieces. The modern application architecture is decisively service-oriented and polyglot, which makes a JVM-only toolbox less relevant.
alrs 12y ago
The bummer about the JVM is that the first piece of advice you get when things are going wrong is "you're using Oracle Java, not OpenJDK, right?" There are situations now where the JVM is unavoidable. It's an increasingly poor choice for greenfield development.
- jdoliner 12y ago
  Yeah it can be a big pain. Especially for companies who aren't already using the JVM and are forced in to it so they can leverage the Hadoop ecosystem. I've had some very painful experiences trying to figure out what's going wrong on my JVM that keeps dying somewhere over the course of MapReduce job and not leaving behind any log files.
- necubi 12y ago
  Since Java 7, Oracle Java and OpenJDK are nearly identical [0]. There's little reason to use the former instead of the latter. Previous to 7 OpenJDK had a much worse performance than the Sun/Oracle JDK, but that hasn't been true for a while now. It remains that the JVM is incredibly mature and performant, and almost any library you could want has been implemented on it. It also remains a fruitful ground for new language development, although LLVM is an increasingly popular alternative. [0] http://stackoverflow.com/questions/22358071/differences-between-oracle-jdk-and-open-jdk-and-garbage-collection http://stackoverflow.com/questions/22358071/differences-betw...
- teacup50 12y ago
  One of the highest performing VMs with some of the most advanced JIT and GC technology is a poor choice for greenfield development? In what universe? The issue with OpenJDK arose when open source folks "backported" the unstable Java 7 OpenJDK source drops to create OpenJDK6, which was ostensibly compatible with (but not identical to) Java 6. As of OpenJDK7/OpenJDK8, none of this hackery is required, and we're seeing convergence of the implementations.

by tiles 12y ago

jdoliner 12y ago

This is a fair criticism and I think we used unclear phrasing here. It's not that we don't consider the JVM to be a modern toolchain in general. Rather we think it's not a modern toolchain for implementing MapReduce because it forces users to work in a JVM compatible way. Docker is a better solution because it gives users the freedom to use what they're already using.

valarauca1 12y ago
Doesn't this approach make users send 1GB+ Docker images instead of small 5-10MB .jar files?
- jdoliner 12y ago
  Useful Docker images can actually be made quite a bit smaller than 1GB+ by using small distros like busybox and following best practices such as cleaning up temporary data between commits. That being said this is a legitimate concern because it affects how tight a developers loop is so it's something we'll probably have to address longterm.
- coffeemug 12y ago
  That sounds relatively easy to fix. I'd imagine a diff between two Docker images is relatively small compared to the size of the image. I'd imagine a deduplication scheme based on widely available tools (e.g. rsync) would solve this problems fairly easily. (Though I have no idea how hard it would be to implement; it doesn't sound too hard off the top of my head)
  - jdoliner 12y ago
    This is a very good point. In addition Docker itself tries to address this by using btrfs's diffing. Like a lot of things in Docker it's not perfect but it's getting there.
teacup50 12y ago
How is schlepping around non-portable Linux-specific binaries packaged into what is essentially a VM a "more modern toolchain"?
- shykes 12y ago
  (just my personal opinion below, I'm not affiliated with the grand-parent) The tooling around the JVM is much more mature the Docker ecosystem in a lot of ways. However, one way to justify the "more modern" label is that Docker can be used to package and manage every component of a Linux-based application stack, whereas JVM tools can only run the JVM-backed pieces. The modern application architecture is decisively service-oriented and polyglot, which makes a JVM-only toolbox less relevant.

alrs 12y ago

The bummer about the JVM is that the first piece of advice you get when things are going wrong is "you're using Oracle Java, not OpenJDK, right?" There are situations now where the JVM is unavoidable. It's an increasingly poor choice for greenfield development.

jdoliner 12y ago
Yeah it can be a big pain. Especially for companies who aren't already using the JVM and are forced in to it so they can leverage the Hadoop ecosystem. I've had some very painful experiences trying to figure out what's going wrong on my JVM that keeps dying somewhere over the course of MapReduce job and not leaving behind any log files.
necubi 12y ago
Since Java 7, Oracle Java and OpenJDK are nearly identical [0]. There's little reason to use the former instead of the latter. Previous to 7 OpenJDK had a much worse performance than the Sun/Oracle JDK, but that hasn't been true for a while now. It remains that the JVM is incredibly mature and performant, and almost any library you could want has been implemented on it. It also remains a fruitful ground for new language development, although LLVM is an increasingly popular alternative. [0] http://stackoverflow.com/questions/22358071/differences-between-oracle-jdk-and-open-jdk-and-garbage-collection http://stackoverflow.com/questions/22358071/differences-betw...
teacup50 12y ago
One of the highest performing VMs with some of the most advanced JIT and GC technology is a poor choice for greenfield development? In what universe? The issue with OpenJDK arose when open source folks "backported" the unstable Java 7 OpenJDK source drops to create OpenJDK6, which was ostensibly compatible with (but not identical to) Java 6. As of OpenJDK7/OpenJDK8, none of this hackery is required, and we're seeing convergence of the implementations.