9 ms·

Ballista: Distributed compute platform implemented in Rust using Apache Arrow

223 points by Kinrany 6y ago

secondcoming 6y ago
Hopefully this is the beginning of the end for JVM use in data-centric applications like this. I'm not particularly bothered if it's Rust or C++
- joshlk 6y ago
  I still don't understand why lots of new databases and data applications (Neo4J, Cassandra, Hadoop ...) are written in Java when it is well known that Java is not well suited to these types to tasks: GC freezes, bad memory model for large data, not fully compiled as so much slower when compared to C++, C or Rust. Why choose Java for these applications?
  - pjmlp 6y ago
    Productivity, unmatched tooling for cluster monitoring, memory safe by default, value types are coming. And even without value types, there are off heap allocations, also if "Python" libraries can be actually written in C, so can Java ones, without loosing the plus of the ecosystem.
    - joshlk 6y ago
      I think those are ok trade offs for a lot of projects but not for high performance and low latency applications such as databases. It seems to me that those project have shot themselves in the foot before even starting.
      - pjmlp 6y ago
        I doubt Hadoop would have been as successful if written in C or C++. While the database engine of most RDMS servers is written in C and C++, anc it won't change given the history behind the code, the bulk of the code is written in some form of managed SQL, with IBM, Oracle and Microsoft also allowing for Java and .NET code. Finally, https://www.efinancialcareers.co.uk/news/2020/11/low-latency-java-trading-systems https://www.efinancialcareers.co.uk/news/2020/11/low-latency... Once upon a time, anyone knew that high performance systems were naturally only viable in Assembly unless proven otherwise. Or how C++ wasn't ever to be a thing in game development, C was the king of console SDKs, after years trying to take Assembly's place on 16 bit platforms.
        secondcoming 6y ago
        Well, to quote TFA: > Essentially, we use a contrived form of Java that avoids all the Java constructs that make things go slow. We only use the constructs that are fast and efficient, and we avoid all the garbage > The only problem with low latency Java is that most experience Java programmers struggle with the new paradigm. "A lot of people who program in Java are used to working in an environment where latency isn't a criteria," says Lawrey. So, the best Java developers would be former C/C++ developers. That's hardly a ringing endorsement for the language. Look at LMAX's Disruptor, for example. It's hardly Java since it gets its performance from use of sun.misc.UnSafe. Java does give you quite good IDEs though. That's about it.
        pjmlp 6y ago
        35 years ago, So, the best C developers would be former Assembly developers. That's hardly a ringing endorsement for the language. Look at XYZ game, for example. It's hardly C since it gets its performance from use of inline Assembly. C does give you quite good shell utilities though. That's about it.
  - secondcoming 6y ago
    We switched from Cassandra (Java) to Scylla (C++) and saw immediate improvements in both query latency and machine count.
- pjmlp 6y ago
  Except value types will put an end to the anti-JVM dream. C++ is never going to be safe by default, unless ISO is willing to do a Python 3. Rust still needs to improve its usability story against everything that is available on the JVM, and compile times, oh boy.
speps 6y ago
> using Apache Arrow MEMORY MODEL Probably got cut because of maximum title length but important nonetheless.
- superbcarrot 6y ago
  It has Rust in the title, that will be enough.
  - DSingularity 6y ago
    lol
- davesque 6y ago
  Not sure what that omitted portion would have clarified. Apache Arrow, at its core, is a memory model spec. Also, it appears that this project is using the official Rust library which is developed in the same repo as all the other language implementations. So the simple statement that they're using Apache Arrow seems adequate.
  - eb0la 6y ago
    Author is the same guy that wrote Arrow Rust library ;-)
    - davesque 6y ago
      So he is, hah. Well there you go :).
habitue 6y ago
It saddens me a little bit that a nascent protocol like arrow flight is using grpc + protobufs behind the scenes with some hacks on top to skip the cost of deserializing protobufs. It seems like a really common belief that protobufs have so much engineering time behind them and are cross-language that it's a no brainer to implement your new protocol on top of them. In reality, all the engineering and optimization time is behind the implementations for the google internal languages, and even the python protobuf implementation is pretty bad. Protobuf makes some stunningly bad decisions like using varints, etc that you shouldn't make the immediate assumption "google has tons of great engineers, google uses protobuf for everything internally, therefore, protobuf is a good foundation to build my new thing on top of" In reality, path dependence and the (amazing) internal tooling ecosystem at google both play a huge part of why they use protobuf so extensively. (Grpc is a little overly complicated to be a universal recommendation, but I could believe it's a good choice for Arrow Flight. But it seems like they didn't do grpc + arrow or grpc + flatbuffer + arrow in the hopes that "dumb" grpc + protobuf implementations would be able to still benefit. In my opinion, grpc implementations are so coupled, there's no reason to make this unnecessary concession to protobufs)
- jfim 6y ago
  I'm not sure I'd call varints a stunningly bad decision. It seems more like the kind of tradeoff one would make if storage and network transfer costs are considered to be more important than the serialization/deserialization speed. That being said, the fact that that particular tradeoff was considered to be good for Google doesn't mean it actually is, or that it's applicable to one's application.
  - xiphias2 6y ago
    I guess it was good before compression. Nowdays working with protobufs is painful inside Google as well, but at least it's supported by everything. Most of what the CPUs at Google are doing is just copying fields from one protobuffer to another.
    - cogman10 6y ago
eb0la 6y ago
Blog post that started the project. Worth reading. https://andygrove.io/2019/07/announcing-ballista/ https://andygrove.io/2019/07/announcing-ballista/
- andygrove 6y ago
  There is also a more recent blog post which perhaps led to the project being posted here (I am guessing). https://andygrove.io/2021/01/ballista-2021/ https://andygrove.io/2021/01/ballista-2021/
  - Kinrany 6y ago
    I saw the project in github.com/trending
lumost 6y ago
I think it's telling on the state of rust in 2021 that this project can't compile fully for the latest rust versions. Maintaining these types of frameworks in the early days is an intensive and often thankless job, having your language leave you behind is a near guaranteed way to kill off your project, not too mention introduce the obvious "I tried using library X and hit compilation issue Y type issues". I'm curious to see how this evolves as there are a number of motivated folks working on similar efforts such as Vega. I for one would welcome a mature rust based distributed compute platform.
- andygrove 6y ago
  The project uses stable Rust. Which version are you trying to compile with?
  - frankmcsherry 6y ago
    I think maybe they were confused by this text (which I agree has nothing to do with Rust itself breaking): > With the release of Apache Arrow 3.0.0 there are many breaking changes in the Rust implementation and as a result it has been necessary to comment out much of the code in this repository and gradually get each Rust module working again with the 3.0.0 release.
    - andygrove 6y ago
      Ah, yes, that makes sense. I can see how this could have been misread.
      - lumost 6y ago
        aye - it ultimately was my misreading of the commit history. Agree that this wasn't a rust specific change.
- jasonpbecker 6y ago
  > With the release of Apache Arrow 3.0.0 there are many breaking changes in the Rust implementation and as a result it has been necessary to comment out much of the code in this repository and gradually get each Rust module working again with the 3.0.0 release. This appears to be an issue with Arrows implementation hitting a new major version and the Rust libraries not yet being compatible with the newest versions of Arrow. That's not something specific to the Rust ecosystem. It's not like a new version of Rust broke this project. But even if it had, maintenance is always hard and the health of a project is better measured by how long it takes to be working with new, major, stable versions after widespread community adoption of those new, major, stable versions. I don't know if Arrow 3.0 is the most commonly used implementation-- it may not have even reached that milestone.
georgewfraser 6y ago
This kind of data infrastructure is a great use case for Rust. A lot of data infrastructure is memory-bound, so saving the memory overhead of GC is a huge win. The use of Arrow to support multiple programming languages is also a great concept. Other distributed computing engines have ended up tied to the JVM (Spark, Presto, Kafka) as a way of avoiding serialization/deserialization costs when you go across a language boundary. Arrow is a really elegant solution, as long as you're willing to batch up operations.
- MrPowers 6y ago
  Databricks recently rebuilt Spark in C++ in a project called "Delta Engine" to overcome the JVM limitations you're pointing out. You're right, Rust is a great way to sidestep the dreaded JVM GC pauses.
  - sitkack 6y ago
    At the same time the JVM is getting better memory tracking analysis and incremental pauseless collectors (C4, ZGZ, Shenandoah, G1 improvements) https://blogs.oracle.com/javamagazine/understanding-the-jdks-new-superfast-garbage-collectors https://blogs.oracle.com/javamagazine/understanding-the-jdks...
    - georgewfraser 6y ago
      These new GCs are amazing technology, but they primarily target pause time, whereas in data processing the primary concern is the “headroom” of extra space in your heap to allow the GC to work efficiently.
      - sitkack 6y ago
        For those cases large off heap structures of arrays can make hundreds of GB of data invisible to the GC. One can do both.
  - kenhwang 6y ago
    Our experience with Delta Engine has been that it's way more resource hungry than the JVM code it replaced. It doesn't handle resource exhaustion well at all; lots of crashing and deadlock when nearing full resource utilization. I would love to have something more resource efficient than Spark on JVM, but Delta Engine isn't there yet.
alisaus6 6y ago
Hottie hangout pics with nude babes - https://adultlove.life https://adultlove.life
vasi26ro 6y ago
I have less then an year as software tester and I have a startup in web. I am wining around 300 EUR a month but the level of knowledge that I won is amazing. Even a freelancer business is trying to sign contract with me to establish them self as authority in the startup world. Don't worry just try
pjmlp 6y ago
> The choice of Rust as the main execution language means that memory usage is deterministic and avoids the overhead of GC pauses. While such projects are welcomed, when value types finally land, the argumentation would need to be upgraded. And as someone that has to track down memory issues in C and C++ projects, not having a GC doesn't mean the memory usage is deterministic.
mkevac 6y ago
What is a distributed compute platform? It's the one thing they are not explaining :-(

223 points by Kinrany 6y ago

Hopefully this is the beginning of the end for JVM use in data-centric applications like this. I'm not particularly bothered if it's Rust or C++

joshlk 6y ago
I still don't understand why lots of new databases and data applications (Neo4J, Cassandra, Hadoop ...) are written in Java when it is well known that Java is not well suited to these types to tasks: GC freezes, bad memory model for large data, not fully compiled as so much slower when compared to C++, C or Rust. Why choose Java for these applications?
- pjmlp 6y ago
  Productivity, unmatched tooling for cluster monitoring, memory safe by default, value types are coming. And even without value types, there are off heap allocations, also if "Python" libraries can be actually written in C, so can Java ones, without loosing the plus of the ecosystem.
  - joshlk 6y ago
    I think those are ok trade offs for a lot of projects but not for high performance and low latency applications such as databases. It seems to me that those project have shot themselves in the foot before even starting.
    - pjmlp 6y ago
      I doubt Hadoop would have been as successful if written in C or C++. While the database engine of most RDMS servers is written in C and C++, anc it won't change given the history behind the code, the bulk of the code is written in some form of managed SQL, with IBM, Oracle and Microsoft also allowing for Java and .NET code. Finally, https://www.efinancialcareers.co.uk/news/2020/11/low-latency-java-trading-systems https://www.efinancialcareers.co.uk/news/2020/11/low-latency... Once upon a time, anyone knew that high performance systems were naturally only viable in Assembly unless proven otherwise. Or how C++ wasn't ever to be a thing in game development, C was the king of console SDKs, after years trying to take Assembly's place on 16 bit platforms.
      - secondcoming 6y ago
        Well, to quote TFA: > Essentially, we use a contrived form of Java that avoids all the Java constructs that make things go slow. We only use the constructs that are fast and efficient, and we avoid all the garbage > The only problem with low latency Java is that most experience Java programmers struggle with the new paradigm. "A lot of people who program in Java are used to working in an environment where latency isn't a criteria," says Lawrey. So, the best Java developers would be former C/C++ developers. That's hardly a ringing endorsement for the language. Look at LMAX's Disruptor, for example. It's hardly Java since it gets its performance from use of sun.misc.UnSafe. Java does give you quite good IDEs though. That's about it.
        pjmlp 6y ago
        35 years ago, So, the best C developers would be former Assembly developers. That's hardly a ringing endorsement for the language. Look at XYZ game, for example. It's hardly C since it gets its performance from use of inline Assembly. C does give you quite good shell utilities though. That's about it.
- secondcoming 6y ago
  We switched from Cassandra (Java) to Scylla (C++) and saw immediate improvements in both query latency and machine count.
pjmlp 6y ago
Except value types will put an end to the anti-JVM dream. C++ is never going to be safe by default, unless ISO is willing to do a Python 3. Rust still needs to improve its usability story against everything that is available on the JVM, and compile times, oh boy.

speps 6y ago

> using Apache Arrow MEMORY MODEL Probably got cut because of maximum title length but important nonetheless.

superbcarrot 6y ago
It has Rust in the title, that will be enough.
- DSingularity 6y ago
  lol
davesque 6y ago
Not sure what that omitted portion would have clarified. Apache Arrow, at its core, is a memory model spec. Also, it appears that this project is using the official Rust library which is developed in the same repo as all the other language implementations. So the simple statement that they're using Apache Arrow seems adequate.
- eb0la 6y ago
  Author is the same guy that wrote Arrow Rust library ;-)
  - davesque 6y ago
    So he is, hah. Well there you go :).

habitue 6y ago

It saddens me a little bit that a nascent protocol like arrow flight is using grpc + protobufs behind the scenes with some hacks on top to skip the cost of deserializing protobufs. It seems like a really common belief that protobufs have so much engineering time behind them and are cross-language that it's a no brainer to implement your new protocol on top of them. In reality, all the engineering and optimization time is behind the implementations for the google internal languages, and even the python protobuf implementation is pretty bad. Protobuf makes some stunningly bad decisions like using varints, etc that you shouldn't make the immediate assumption "google has tons of great engineers, google uses protobuf for everything internally, therefore, protobuf is a good foundation to build my new thing on top of" In reality, path dependence and the (amazing) internal tooling ecosystem at google both play a huge part of why they use protobuf so extensively. (Grpc is a little overly complicated to be a universal recommendation, but I could believe it's a good choice for Arrow Flight. But it seems like they didn't do grpc + arrow or grpc + flatbuffer + arrow in the hopes that "dumb" grpc + protobuf implementations would be able to still benefit. In my opinion, grpc implementations are so coupled, there's no reason to make this unnecessary concession to protobufs)

jfim 6y ago
I'm not sure I'd call varints a stunningly bad decision. It seems more like the kind of tradeoff one would make if storage and network transfer costs are considered to be more important than the serialization/deserialization speed. That being said, the fact that that particular tradeoff was considered to be good for Google doesn't mean it actually is, or that it's applicable to one's application.
- xiphias2 6y ago
  I guess it was good before compression. Nowdays working with protobufs is painful inside Google as well, but at least it's supported by everything. Most of what the CPUs at Google are doing is just copying fields from one protobuffer to another.
  - cogman10 6y ago

eb0la 6y ago

Blog post that started the project. Worth reading. https://andygrove.io/2019/07/announcing-ballista/ https://andygrove.io/2019/07/announcing-ballista/

andygrove 6y ago
There is also a more recent blog post which perhaps led to the project being posted here (I am guessing). https://andygrove.io/2021/01/ballista-2021/ https://andygrove.io/2021/01/ballista-2021/
- Kinrany 6y ago
  I saw the project in github.com/trending

lumost 6y ago

I think it's telling on the state of rust in 2021 that this project can't compile fully for the latest rust versions. Maintaining these types of frameworks in the early days is an intensive and often thankless job, having your language leave you behind is a near guaranteed way to kill off your project, not too mention introduce the obvious "I tried using library X and hit compilation issue Y type issues". I'm curious to see how this evolves as there are a number of motivated folks working on similar efforts such as Vega. I for one would welcome a mature rust based distributed compute platform.

andygrove 6y ago
The project uses stable Rust. Which version are you trying to compile with?
- frankmcsherry 6y ago
  I think maybe they were confused by this text (which I agree has nothing to do with Rust itself breaking): > With the release of Apache Arrow 3.0.0 there are many breaking changes in the Rust implementation and as a result it has been necessary to comment out much of the code in this repository and gradually get each Rust module working again with the 3.0.0 release.
  - andygrove 6y ago
    Ah, yes, that makes sense. I can see how this could have been misread.
    - lumost 6y ago
      aye - it ultimately was my misreading of the commit history. Agree that this wasn't a rust specific change.
jasonpbecker 6y ago
> With the release of Apache Arrow 3.0.0 there are many breaking changes in the Rust implementation and as a result it has been necessary to comment out much of the code in this repository and gradually get each Rust module working again with the 3.0.0 release. This appears to be an issue with Arrows implementation hitting a new major version and the Rust libraries not yet being compatible with the newest versions of Arrow. That's not something specific to the Rust ecosystem. It's not like a new version of Rust broke this project. But even if it had, maintenance is always hard and the health of a project is better measured by how long it takes to be working with new, major, stable versions after widespread community adoption of those new, major, stable versions. I don't know if Arrow 3.0 is the most commonly used implementation-- it may not have even reached that milestone.

georgewfraser 6y ago

This kind of data infrastructure is a great use case for Rust. A lot of data infrastructure is memory-bound, so saving the memory overhead of GC is a huge win. The use of Arrow to support multiple programming languages is also a great concept. Other distributed computing engines have ended up tied to the JVM (Spark, Presto, Kafka) as a way of avoiding serialization/deserialization costs when you go across a language boundary. Arrow is a really elegant solution, as long as you're willing to batch up operations.

MrPowers 6y ago
Databricks recently rebuilt Spark in C++ in a project called "Delta Engine" to overcome the JVM limitations you're pointing out. You're right, Rust is a great way to sidestep the dreaded JVM GC pauses.
- sitkack 6y ago
  At the same time the JVM is getting better memory tracking analysis and incremental pauseless collectors (C4, ZGZ, Shenandoah, G1 improvements) https://blogs.oracle.com/javamagazine/understanding-the-jdks-new-superfast-garbage-collectors https://blogs.oracle.com/javamagazine/understanding-the-jdks...
  - georgewfraser 6y ago
    These new GCs are amazing technology, but they primarily target pause time, whereas in data processing the primary concern is the “headroom” of extra space in your heap to allow the GC to work efficiently.
    - sitkack 6y ago
      For those cases large off heap structures of arrays can make hundreds of GB of data invisible to the GC. One can do both.
- kenhwang 6y ago
  Our experience with Delta Engine has been that it's way more resource hungry than the JVM code it replaced. It doesn't handle resource exhaustion well at all; lots of crashing and deadlock when nearing full resource utilization. I would love to have something more resource efficient than Spark on JVM, but Delta Engine isn't there yet.

alisaus6 6y ago

Hottie hangout pics with nude babes - https://adultlove.life https://adultlove.life

vasi26ro 6y ago

I have less then an year as software tester and I have a startup in web. I am wining around 300 EUR a month but the level of knowledge that I won is amazing. Even a freelancer business is trying to sign contract with me to establish them self as authority in the startup world. Don't worry just try

pjmlp 6y ago

> The choice of Rust as the main execution language means that memory usage is deterministic and avoids the overhead of GC pauses. While such projects are welcomed, when value types finally land, the argumentation would need to be upgraded. And as someone that has to track down memory issues in C and C++ projects, not having a GC doesn't mean the memory usage is deterministic.

mkevac 6y ago

What is a distributed compute platform? It's the one thing they are not explaining :-(