4 ms·

Dremio is JVM based but with compilation down to LLVM and they contributed Gandiva to the Apache Arrow project. Dremio seems pretty impressive but I haven't per

Dremio is JVM based but with compilation down to LLVM and they contributed Gandiva to the Apache Arrow project. Dremio seems pretty impressive but I haven't personally used it. https://www.dremio.com/announcing-gandiva-initiative-for-apache-arrow/ https://www.dremio.com/announcing-gandiva-initiative-for-apa...

atombender 7y ago
Interesting, thanks. I know the entire industry is built around Hadoop and the JVM right now (with some Python), but I'm hoping the pendulum will swing in a different direction soon.
- andygrove 7y ago
  Me too. I have a JVM background for > 20 years and have been using Apache Spark for several years and while I have great respect for the engineering that has gone into Spark, I feel that they just started out with the wrong language. GC-based languages are not ideal for large scale distributed data processing, IMHO.
  - pjmlp 7y ago
    Only when the said languages don't support value types, it was an error from Java, not to offer what Wirth inspired languages already had in the 90's (Oberon variants, Modula-3, Eiffel). The first EA release for value types support is out now. I firmly believe in the end tracing GC with value types and local ownership, like D, Swift and C# are pursuing will win, at least in the realm of userspace programming.
    - voodootrucker 7y ago
      Do you know the status of value types for Java? I see they were slated to be in v10 of the JVM, and I can find discussion [0] up to a year ago about them making it into the language, but that's ass far as my googling has taken me. If Java can have struct access into native memory (without the presently complicated API [1]) that would open a lot of doors and get it to near feature parity with the CLR. [1] https://docs.oracle.com/javase/6/docs/api/java/nio/ByteBuffer.html https://docs.oracle.com/javase/6/docs/api/java/nio/ByteBuffe... [0] https://news.ycombinator.com/item?id=14583530 https://news.ycombinator.com/item?id=14583530
      - pjmlp 7y ago
        Yes, have a look at https://mail.openjdk.java.net/pipermail/valhalla-dev/2019-July/006094.html https://mail.openjdk.java.net/pipermail/valhalla-dev/2019-Ju... And follow the links from there, including the Wiki page describing the EA release, unfortunately my submission did not gather too much uptake. It is taking all this this time because the team wants pre-value types world jars to keep running unmodified on the new world, not an easy engineering task.
- geezerjay 7y ago
  Why are you hoping that the proverbial pendulum swings away fron Hadoop?
  - atombender 7y ago
    I find the Hadoop platform to be heavyweight and difficult to manage. Its design came from the original GFS and MapReduce papers, and was perhaps not great to begin with (name nodes failover, etc.). I would like to see the industry move away from the JVM to less memory-hungry, AOT-compiled languages. Too much of the Hadoop world simply assumes that you'll be using a JVM language, and don't provide non-JVM APIs at all. It's been a while since I looked, but I think Apex and Flink were basically useless if you didn't use a JVM language or Python. Of all the Apache projects related to Hadoop, I believe Beam is the only one that has Go support (for implementing processing logic), for example.

atombender 7y ago

Interesting, thanks. I know the entire industry is built around Hadoop and the JVM right now (with some Python), but I'm hoping the pendulum will swing in a different direction soon.

andygrove 7y ago
Me too. I have a JVM background for > 20 years and have been using Apache Spark for several years and while I have great respect for the engineering that has gone into Spark, I feel that they just started out with the wrong language. GC-based languages are not ideal for large scale distributed data processing, IMHO.
- pjmlp 7y ago
  Only when the said languages don't support value types, it was an error from Java, not to offer what Wirth inspired languages already had in the 90's (Oberon variants, Modula-3, Eiffel). The first EA release for value types support is out now. I firmly believe in the end tracing GC with value types and local ownership, like D, Swift and C# are pursuing will win, at least in the realm of userspace programming.
  - voodootrucker 7y ago
    Do you know the status of value types for Java? I see they were slated to be in v10 of the JVM, and I can find discussion [0] up to a year ago about them making it into the language, but that's ass far as my googling has taken me. If Java can have struct access into native memory (without the presently complicated API [1]) that would open a lot of doors and get it to near feature parity with the CLR. [1] https://docs.oracle.com/javase/6/docs/api/java/nio/ByteBuffer.html https://docs.oracle.com/javase/6/docs/api/java/nio/ByteBuffe... [0] https://news.ycombinator.com/item?id=14583530 https://news.ycombinator.com/item?id=14583530
    - pjmlp 7y ago
      Yes, have a look at https://mail.openjdk.java.net/pipermail/valhalla-dev/2019-July/006094.html https://mail.openjdk.java.net/pipermail/valhalla-dev/2019-Ju... And follow the links from there, including the Wiki page describing the EA release, unfortunately my submission did not gather too much uptake. It is taking all this this time because the team wants pre-value types world jars to keep running unmodified on the new world, not an easy engineering task.
geezerjay 7y ago
Why are you hoping that the proverbial pendulum swings away fron Hadoop?
- atombender 7y ago
  I find the Hadoop platform to be heavyweight and difficult to manage. Its design came from the original GFS and MapReduce papers, and was perhaps not great to begin with (name nodes failover, etc.). I would like to see the industry move away from the JVM to less memory-hungry, AOT-compiled languages. Too much of the Hadoop world simply assumes that you'll be using a JVM language, and don't provide non-JVM APIs at all. It's been a while since I looked, but I think Apex and Flink were basically useless if you didn't use a JVM language or Python. Of all the Apache projects related to Hadoop, I believe Beam is the only one that has Go support (for implementing processing logic), for example.