5 ms·

How does this compare to dremio, that also uses Apache Arrow? Is this a competitor?

andygrove 7y ago
No. Dremio is a mature product backed by a company. This is a two week old PoC right now. There could be overlap and collaboration for sure via the Apache Arrow project since Dremio are well represented there and I am an Arrow committer too.
- andygrove 7y ago
  My goal is to kick start some things around the Rust language for data science / data analytics and projects like this help to showcase what Rust can do. See my original blog post for more context: https://andygrove.io/2018/01/rust-is-for-big-data/ https://andygrove.io/2018/01/rust-is-for-big-data/
  - Keyframe 7y ago
    It will be tough, and next to improbable, to match python's ecosystem. Maybe a way into it would be to marry the two.
  - kentm 7y ago
    Interestingly I’ve had the same observations that you’ve had about the work the Spark project has done to work around the JVM. And is also been idly wondering what a native-first distributes data processing framework would look like and how it will perform. I’ll be really interested in seeing where your project goes.
atombender 7y ago
What languages does Dremio support?
- polskibus 7y ago
  SQL
- andygrove 7y ago
  Dremio is JVM based but with compilation down to LLVM and they contributed Gandiva to the Apache Arrow project. Dremio seems pretty impressive but I haven't personally used it. https://www.dremio.com/announcing-gandiva-initiative-for-apache-arrow/ https://www.dremio.com/announcing-gandiva-initiative-for-apa...
  - atombender 7y ago
    Interesting, thanks. I know the entire industry is built around Hadoop and the JVM right now (with some Python), but I'm hoping the pendulum will swing in a different direction soon.
    - andygrove 7y ago
      Me too. I have a JVM background for > 20 years and have been using Apache Spark for several years and while I have great respect for the engineering that has gone into Spark, I feel that they just started out with the wrong language. GC-based languages are not ideal for large scale distributed data processing, IMHO.
      - pjmlp 7y ago
        Only when the said languages don't support value types, it was an error from Java, not to offer what Wirth inspired languages already had in the 90's (Oberon variants, Modula-3, Eiffel). The first EA release for value types support is out now. I firmly believe in the end tracing GC with value types and local ownership, like D, Swift and C# are pursuing will win, at least in the realm of userspace programming.
        voodootrucker 7y ago
        Do you know the status of value types for Java? I see they were slated to be in v10 of the JVM, and I can find discussion [0] up to a year ago about them making it into the language, but that's ass far as my googling has taken me. If Java can have struct access into native memory (without the presently complicated API [1]) that would open a lot of doors and get it to near feature parity with the CLR. [1] https://docs.oracle.com/javase/6/docs/api/java/nio/ByteBuffer.html https://docs.oracle.com/javase/6/docs/api/java/nio/ByteBuffe... [0] https://news.ycombinator.com/item?id=14583530 https://news.ycombinator.com/item?id=14583530

No. Dremio is a mature product backed by a company. This is a two week old PoC right now. There could be overlap and collaboration for sure via the Apache Arrow project since Dremio are well represented there and I am an Arrow committer too.

andygrove 7y ago
My goal is to kick start some things around the Rust language for data science / data analytics and projects like this help to showcase what Rust can do. See my original blog post for more context: https://andygrove.io/2018/01/rust-is-for-big-data/ https://andygrove.io/2018/01/rust-is-for-big-data/
- Keyframe 7y ago
  It will be tough, and next to improbable, to match python's ecosystem. Maybe a way into it would be to marry the two.
- kentm 7y ago
  Interestingly I’ve had the same observations that you’ve had about the work the Spark project has done to work around the JVM. And is also been idly wondering what a native-first distributes data processing framework would look like and how it will perform. I’ll be really interested in seeing where your project goes.

atombender 7y ago

What languages does Dremio support?

polskibus 7y ago
SQL
andygrove 7y ago
Dremio is JVM based but with compilation down to LLVM and they contributed Gandiva to the Apache Arrow project. Dremio seems pretty impressive but I haven't personally used it. https://www.dremio.com/announcing-gandiva-initiative-for-apache-arrow/ https://www.dremio.com/announcing-gandiva-initiative-for-apa...
- atombender 7y ago
  Interesting, thanks. I know the entire industry is built around Hadoop and the JVM right now (with some Python), but I'm hoping the pendulum will swing in a different direction soon.
  - andygrove 7y ago
    Me too. I have a JVM background for > 20 years and have been using Apache Spark for several years and while I have great respect for the engineering that has gone into Spark, I feel that they just started out with the wrong language. GC-based languages are not ideal for large scale distributed data processing, IMHO.
    - pjmlp 7y ago
      Only when the said languages don't support value types, it was an error from Java, not to offer what Wirth inspired languages already had in the 90's (Oberon variants, Modula-3, Eiffel). The first EA release for value types support is out now. I firmly believe in the end tracing GC with value types and local ownership, like D, Swift and C# are pursuing will win, at least in the realm of userspace programming.
      - voodootrucker 7y ago
        Do you know the status of value types for Java? I see they were slated to be in v10 of the JVM, and I can find discussion [0] up to a year ago about them making it into the language, but that's ass far as my googling has taken me. If Java can have struct access into native memory (without the presently complicated API [1]) that would open a lot of doors and get it to near feature parity with the CLR. [1] https://docs.oracle.com/javase/6/docs/api/java/nio/ByteBuffer.html https://docs.oracle.com/javase/6/docs/api/java/nio/ByteBuffe... [0] https://news.ycombinator.com/item?id=14583530 https://news.ycombinator.com/item?id=14583530