11 ms·

Ballista: Distributed Compute with Rust, Apache Arrow, and Kubernetes

194 points by andygrove 7y ago

polskibus 7y ago
How does this compare to dremio, that also uses Apache Arrow? Is this a competitor?
- andygrove 7y ago
  No. Dremio is a mature product backed by a company. This is a two week old PoC right now. There could be overlap and collaboration for sure via the Apache Arrow project since Dremio are well represented there and I am an Arrow committer too.
  - andygrove 7y ago
    My goal is to kick start some things around the Rust language for data science / data analytics and projects like this help to showcase what Rust can do. See my original blog post for more context: https://andygrove.io/2018/01/rust-is-for-big-data/ https://andygrove.io/2018/01/rust-is-for-big-data/
    - Keyframe 7y ago
      It will be tough, and next to improbable, to match python's ecosystem. Maybe a way into it would be to marry the two.
    - kentm 7y ago
      Interestingly I’ve had the same observations that you’ve had about the work the Spark project has done to work around the JVM. And is also been idly wondering what a native-first distributes data processing framework would look like and how it will perform. I’ll be really interested in seeing where your project goes.
- atombender 7y ago
  What languages does Dremio support?
  - polskibus 7y ago
    SQL
  - andygrove 7y ago
    Dremio is JVM based but with compilation down to LLVM and they contributed Gandiva to the Apache Arrow project. Dremio seems pretty impressive but I haven't personally used it. https://www.dremio.com/announcing-gandiva-initiative-for-apache-arrow/ https://www.dremio.com/announcing-gandiva-initiative-for-apa...
    - atombender 7y ago
      Interesting, thanks. I know the entire industry is built around Hadoop and the JVM right now (with some Python), but I'm hoping the pendulum will swing in a different direction soon.
      - andygrove 7y ago
        Me too. I have a JVM background for > 20 years and have been using Apache Spark for several years and while I have great respect for the engineering that has gone into Spark, I feel that they just started out with the wrong language. GC-based languages are not ideal for large scale distributed data processing, IMHO.
        pjmlp 7y ago
        Only when the said languages don't support value types, it was an error from Java, not to offer what Wirth inspired languages already had in the 90's (Oberon variants, Modula-3, Eiffel). The first EA release for value types support is out now. I firmly believe in the end tracing GC with value types and local ownership, like D, Swift and C# are pursuing will win, at least in the realm of userspace programming.
sandGorgon 7y ago
How about Dask - which is fairly production grade and has experimental Arrow integration. https://github.com/apache/arrow/blob/master/integration/dask/Dockerfile https://github.com/apache/arrow/blob/master/integration/dask... Dask deploys pretty well on k8s - https://kubernetes.dask.org/en/latest/ https://kubernetes.dask.org/en/latest/
- andygrove 7y ago
  Although I know and work with some of the contributors from this project, I have no real world experience with Dask or Python data science tools in general. Thanks for the link. I will read about their Kubernetes support.
- wesm 7y ago
  Dask does not have "experimental Arrow integration". It supports using Arrow to read Parquet files but no Arrow-based computational functionality.
  - arijun 7y ago
    Thanks for the clarification, Wes! Semi-related question: How do you expect Arrow to be integrated to the larger data science landscape? Will it mostly be used as a go between format? Will new libraries using it internally and old libraries just reading and translating it to a native format? Do you think established libraries will change their back-end to arrow? Is that even feasible with e.g. Pandas (or are you too far from their governance now to say)?
    - wesm 7y ago
      Too big of a discussion for Hacker News! Come on dev@arrow.apache.org if you want to talk about it
      - sandGorgon 7y ago
        thanks for correcting me. i was not aware of this nuance. would you be open to posting quick thoughts here for the rest of us ?
StreamBright 7y ago
Could this be used without Kubernetes?
- andygrove 7y ago
  With further development, yes. It's just Docker containers. However, the current CLI is specific to Kubernetes. If you weren't using Kubernetes, what orchestrator would you be interested in?
  - pondidum 7y ago
    I'd be interested in a version I could use with Nomad at least.
  - StreamBright 7y ago
    Thanks Andy! To be clear, I really appreciate your effort to create a better platform for big data. I have spent 10 years on trying to make Hadoop & Spark financially and technically scalable for companies with more than 1PB data and I totally agree with you that we need a better system. I am just not ready to trade Hadoop or Spark problems to Kubernetes problems. The question is what orchestration do we need? What model do you want to implement? Could we build a better Kubernetes?
    - andygrove 7y ago
      I've only been using Kubernetes for a couple months so far and am still learning, but I am very impressed so far. I love the way it facilitates dev and devops collaborating and the fact that it is cloud agnostic (I can even run a Kubernetes cluster on my desktop for local testing). My opinion so far is that the distributed cluster part is really solved by Kubernetes.
      - StreamBright 7y ago
        Yeah it is absolutely amazing for exactly that and yes it has a good approach to the distributed cluster problem. Once you put it in production there a very different picture. https://github.com/hjacobs/kubernetes-failure-stories https://github.com/hjacobs/kubernetes-failure-stories I haven't had a single Kubernetes user who had it in production and did not have stability or performance issues with it. This is why I mentioned that I am not ready to trade my Hadoop outages and performance issues for Kubernetes ones.
m0zg 7y ago
Serious, non-facetious question: who is this for?
- andygrove 7y ago
  This is for people who want to live in a future where we use efficient and safe system level languages for massively scalable distributed data processing. IMHO, Python and Java are not ideal language choices for these purposes. Rust offers much lower TCO compared to current industry best practices in this area.
  - geezerjay 7y ago
    But claiming that "X software is written in Y language/framework" says nothing about efficiency or safety. It's just meaningless marketting piggy-backing on popular buzzwords. And claims about "the future" are simply absurd. Oddly enough, this link appears right next to another story on how Cobol powers the world's economy. Frankly, I'm surprised blockchain wasn't shoved somewhere in the announcement.
    - andygrove 7y ago
      Check out my previous benchmarks on Rust-based DataFusion vs JVM-based Apache Spark workloads. It isn't just about raw performance numbers but also about resource requirements (especially RAM requirements when using JVM or similar GC-based languages). There are order-of-magnitude differences.
      - geezerjay 7y ago
        If you have tangible results then present your benchmarks. If you limit your marketing to empty claims regarding "the future" and vague assertions on performance then you're actually actively working to lower your credibility.
        andygrove 7y ago
        This is a personal open source project. I'm not sure I'm "marketing" it since I make zero dollars from this work. For benchmarks, check out the past 18 months of posts on my blog. Here is the most recent: https://andygrove.io/2019/04/datafusion-0.13.0-benchmarks/ https://andygrove.io/2019/04/datafusion-0.13.0-benchmarks/
ohnoesjmr 7y ago
I congratulate the effort, as I always thought that Spark is great, but the fact it was written in Java hinders it quite badly (GC, tons of memory required for the runtime, jar hell (want to use proto3 in your spark job? Good luck)). I do however worry that rust has a high bar of entry.
- andygrove 7y ago
  I agree. Rust has a very steep learning curve compared to JVM languages. My hope in building this platform is that it can provide value to other languages (especially JVM) by taking query plans and executing them efficiently.
  - 0815test 7y ago
    > compared to JVM languages Apache Spark is written in Scala, and I wouldn't describe that as having an 'easy' learning curve, even compared to Rust! If you want something 'easy' on the JVM, Kotlin (and perhaps Eta or Frege) might be more appropriate.
    - pjmlp 7y ago
      Scala is way easier to use, given it uses a tracing GC, has three good IDEs, and more mature ecosystem.
    - twic 7y ago
      Coming from Java, i found Rust easier to learn than Scala.
      - Recurecur 7y ago
        I expect you were exposed using a heavily functional programming approach. That's a whole 'nother (larger) area to pick up along with the different language. If Scala is used as a "better Java" to start with, and then the developers explore FP at their own pace, I think there's a better outcome. Granted Java 8+ has done some to close the gap with Scala as well. I wonder if Scala-Native will absorb some Rustish features as other languages have or are attempting to... (What I said above echoes Odersky's ideas about Scala developer levels: https://www.scala-lang.org/old/node/8610.html https://www.scala-lang.org/old/node/8610.html)
wiradikusuma 7y ago
So Spark is bloated bcoz of JVM. Does Graal make the point moot?
- voodootrucker 7y ago
  Not really, no. Even though Graal compiles to native code, these JVM based languages are all based on two ideas: 1. there will be a GC to manage memory 2. memory will be managed The GC slows things down, so spark tries to work around this by accessing "off heap" memory (which really just means off the JVM's heap and on the OS'). So you end up getting OOM errors if you give to little to the JVM or if you give too little to the OS. It's a hacky balancing act to get native access to memory, which comes for free with rust.
s_Hogg 7y ago
Hang in there mate :) I really don't think you deserve a lot of the crap you've been given in this thread. Someone has to try something new.
- eb0la 7y ago
  The fact people opposed to your idea / work means it is valuable enough for people to say something against and not ignore it. I must confess I miss native execution of (big)Data jobs. I know moving jvm bytecode between nodes to be run is portable, but nowadays nobody has mixed architecture (intel/mips/sparc/arm) servers, so... why do we need a bytecode execution layer? Maybe I am too critic, but bare metal - hypervisor - jvm - app looks too much layers to me.
  - s_Hogg 7y ago
    Totally, I think there are a lot of cases where all that machinery is actually completely superfluous anyway because thinking about the problem you're trying to solve could lead you to a way of doing it that doesn't need all that compute. There's a blog post about this I read a while back where someone did a simple graph search using spark and then did it in a slightly smarter way on a commodity laptop and it outran spark on a large-ish amount of data. Wish I could find it.
    - buckminster 7y ago
      This one? http://www.frankmcsherry.org/graph/scalability/cost/2015/01/15/COST.html http://www.frankmcsherry.org/graph/scalability/cost/2015/01/...
      - s_Hogg 7y ago
        Yep, that's the one!
    - eb0la 7y ago
      I remember 4-5 years ago I tried to slice a big CSV file with Spark and took soo loong it didn't finished after some hours. Later on the day I realized it could be done from the command line with 'cut' and 'grep'. Took less than 15 minutes.
eb0la 7y ago
This project needs a "how to help" section urgently
- andygrove 7y ago
  I will write up some guidance in the next few days for those looking to contribute!
deleted 7y ago
[deleted]
snicker7 7y ago
Would this system support custom aggregates? How would I, for example, create a routine that defines a covariance matrix and have Ballista deal with the necessary map-reduce logic?
fspear 7y ago
Are you looking for contributors? I don't have any rust, arrow or k8s experience but been looking to learn all 3, I've also been looking to contribute to os projects so I'm happy to pick up any low hanging fruits if you are interested. I do have a few years of experience with Spark and hadoop if that's worth anything.
- andygrove 7y ago
  Yes contributors are welcome. I will write up some guidance in the next few days for those looking to contribute!
dswalter 7y ago
I'm actually excited about the possibilities. I've watched DataFusion from afar, and I have spent a decent amount of time wishing the Big Data ecosystem had arrived during a time when something like Rust was a viable option, both for memory and for parallel computing. I use Presto all the time, I love how fully-featured it is, but garbage collection is a non-trivial component of time-to-execute for my queries.
kyllo 7y ago
This is really cool! What do you see as ideally the primary API for something like this? SQL is great for relational algebra expressions to transform tables but its limited support for variables and control flow constructs make it less than ideal for complex, multi-step data analysis scripts. And when it comes to running statistical tests, regressions, training ML models, it's wholly inappropriate. Rust is a very expressive systems programming language, but it's unclear at this point how good of a fit it can be for data analysis and statistical programming tasks. It doesn't have much in the way of data science libraries yet. Would you potentially add e.g. a Python interpreter on top of such a framework, or would you focus on building out a more fully-featured Rust API for data analysis and even go so far as to suggest that data scientists start to learn and use Rust? (There is some precedence for this with Scala and Spark)
- andygrove 7y ago
  These are great questions and topics I plan on addressing in a future blog post. SQL is a great convenience for simple analytical queries and it can be nice to be able to mix and match SQL and other access patterns (this is one thing I like with the Apache Spark DataFrame approach). It's true that the Rust ecosystem for data science is not really there yet and I am trying to inspire people to start changing that. I think Rust does have some good potential here. In the nearer term though I am exploring options around support user code in distributed query execution and I don't want to limit it to Rust.
  - kyllo 7y ago
    I like it--I'm a data scientist by day, and I've been following Rust with interest but I haven't found a good use case for it in my job yet. A Rust-based Spark competitor sounds like it could be exactly that excuse to use Rust at work that I've been looking for!
    - andygrove 7y ago
      This is exactly the situation I am in. We have workloads that I think we could deliver with much lower TCO using Rust/DataFusion/Ballista. We could probably even use a single node for some of our workflows just using DataFusion directly with same performance as a distributed Spark job.
cozos 7y ago
Most "big data" distributed compute frameworks that come to mind are written in a JVM language, so the focus on Rust is interesting. So then, would Rust be better than a JVM language for a distributed compute framework like Apache Spark? Based on what others said in this thread, these are the primary arguments for Rust: 1. JVM GC overhead 2. JVM GC pauses 3. JVM memory overhead. 4. Native code (i.e. Rust) has better raw performance than a JVM language My take on it: (1) I believe Spark basically wrote its own memory management layer with Unsafe that let's it bypass the GC [0], so for Dataframe/SQL we might be ok here. Hopefully value types are coming to Java/Scala soon. (2) Majority of Apache Spark use-cases are batch right? In this case who cares about a little stop-the-world pause here and there, as long as we're optimizing the GC for throughput. I recognize that streaming is also a thing, so maybe a non-GC language like Rust is better suited for latency sensitive streaming workloads. Perhaps the Shenandoah GC would be of help here. (3) What's the memory overhead of a JVM process, 100-200 MB? That doesn't seem too bad to me when clusters these days have terabytes of memory. (4) I wonder how much of an impact performance improvements from Rust will have over Spark's optimized code generation [1], which basically converts your code into array loops that utilize cache locality, loop unrolling, and simd. I imagine that most of the gains to be had from a Rust rewrite would come from these "bare metal' techniques, so it might the case that Spark already has that going for it... Having said that, I can't think of any reasons why a compute engine on Rust is a bad idea. Developer productivity and ecosystem perhaps? [0] https://databricks.com/blog/2015/04/28/project-tungsten-bringing-spark-closer-to-bare-metal.html https://databricks.com/blog/2015/04/28/project-tungsten-brin... [1] https://databricks.com/blog/2016/05/23/apache-spark-as-a-compiler-joining-a-billion-rows-per-second-on-a-laptop.html https://databricks.com/blog/2016/05/23/apache-spark-as-a-com...
- andygrove 7y ago
  Some good points. Some incredible engineering has gone into Spark to work around the fact that it runs on the JVM. Memory overhead of Spark particularly (not just JVM) is very high. In some cases close to 100x more memory than equivalent query execution with DataFusion [0]. Also you might be interested to see my original blog post with some of my thoughts on this [1]. [0] https://andygrove.io/2019/04/datafusion-0.13.0-benchmarks/ https://andygrove.io/2019/04/datafusion-0.13.0-benchmarks/ [1] https://andygrove.io/2018/01/rust-is-for-big-data/ https://andygrove.io/2018/01/rust-is-for-big-data/
senderista 7y ago
If you’re looking for an approachable distributed query planner, https://github.com/uwescience/raco https://github.com/uwescience/raco might be a good place to start.
blittable 7y ago
Super cool. Perhaps naive, but how does distributing computation with serialization square with Arrow's in-memory design?
- andygrove 7y ago
  Each executor within the cluster would use Arrow in-memory design. If you have enough cores and memory on a single node then potentially you wouldn't need a cluster.

194 points by andygrove 7y ago

polskibus 7y ago

How does this compare to dremio, that also uses Apache Arrow? Is this a competitor?

andygrove 7y ago
No. Dremio is a mature product backed by a company. This is a two week old PoC right now. There could be overlap and collaboration for sure via the Apache Arrow project since Dremio are well represented there and I am an Arrow committer too.
- andygrove 7y ago
  My goal is to kick start some things around the Rust language for data science / data analytics and projects like this help to showcase what Rust can do. See my original blog post for more context: https://andygrove.io/2018/01/rust-is-for-big-data/ https://andygrove.io/2018/01/rust-is-for-big-data/
  - Keyframe 7y ago
    It will be tough, and next to improbable, to match python's ecosystem. Maybe a way into it would be to marry the two.
  - kentm 7y ago
    Interestingly I’ve had the same observations that you’ve had about the work the Spark project has done to work around the JVM. And is also been idly wondering what a native-first distributes data processing framework would look like and how it will perform. I’ll be really interested in seeing where your project goes.
atombender 7y ago
What languages does Dremio support?
- polskibus 7y ago
  SQL
- andygrove 7y ago
  Dremio is JVM based but with compilation down to LLVM and they contributed Gandiva to the Apache Arrow project. Dremio seems pretty impressive but I haven't personally used it. https://www.dremio.com/announcing-gandiva-initiative-for-apache-arrow/ https://www.dremio.com/announcing-gandiva-initiative-for-apa...
  - atombender 7y ago
    Interesting, thanks. I know the entire industry is built around Hadoop and the JVM right now (with some Python), but I'm hoping the pendulum will swing in a different direction soon.
    - andygrove 7y ago
      Me too. I have a JVM background for > 20 years and have been using Apache Spark for several years and while I have great respect for the engineering that has gone into Spark, I feel that they just started out with the wrong language. GC-based languages are not ideal for large scale distributed data processing, IMHO.
      - pjmlp 7y ago
        Only when the said languages don't support value types, it was an error from Java, not to offer what Wirth inspired languages already had in the 90's (Oberon variants, Modula-3, Eiffel). The first EA release for value types support is out now. I firmly believe in the end tracing GC with value types and local ownership, like D, Swift and C# are pursuing will win, at least in the realm of userspace programming.

sandGorgon 7y ago

How about Dask - which is fairly production grade and has experimental Arrow integration. https://github.com/apache/arrow/blob/master/integration/dask/Dockerfile https://github.com/apache/arrow/blob/master/integration/dask... Dask deploys pretty well on k8s - https://kubernetes.dask.org/en/latest/ https://kubernetes.dask.org/en/latest/

andygrove 7y ago
Although I know and work with some of the contributors from this project, I have no real world experience with Dask or Python data science tools in general. Thanks for the link. I will read about their Kubernetes support.
wesm 7y ago
Dask does not have "experimental Arrow integration". It supports using Arrow to read Parquet files but no Arrow-based computational functionality.
- arijun 7y ago
  Thanks for the clarification, Wes! Semi-related question: How do you expect Arrow to be integrated to the larger data science landscape? Will it mostly be used as a go between format? Will new libraries using it internally and old libraries just reading and translating it to a native format? Do you think established libraries will change their back-end to arrow? Is that even feasible with e.g. Pandas (or are you too far from their governance now to say)?
  - wesm 7y ago
    Too big of a discussion for Hacker News! Come on dev@arrow.apache.org if you want to talk about it
    - sandGorgon 7y ago
      thanks for correcting me. i was not aware of this nuance. would you be open to posting quick thoughts here for the rest of us ?

StreamBright 7y ago

Could this be used without Kubernetes?

andygrove 7y ago
With further development, yes. It's just Docker containers. However, the current CLI is specific to Kubernetes. If you weren't using Kubernetes, what orchestrator would you be interested in?
- pondidum 7y ago
  I'd be interested in a version I could use with Nomad at least.
- StreamBright 7y ago
  Thanks Andy! To be clear, I really appreciate your effort to create a better platform for big data. I have spent 10 years on trying to make Hadoop & Spark financially and technically scalable for companies with more than 1PB data and I totally agree with you that we need a better system. I am just not ready to trade Hadoop or Spark problems to Kubernetes problems. The question is what orchestration do we need? What model do you want to implement? Could we build a better Kubernetes?
  - andygrove 7y ago
    I've only been using Kubernetes for a couple months so far and am still learning, but I am very impressed so far. I love the way it facilitates dev and devops collaborating and the fact that it is cloud agnostic (I can even run a Kubernetes cluster on my desktop for local testing). My opinion so far is that the distributed cluster part is really solved by Kubernetes.
    - StreamBright 7y ago
      Yeah it is absolutely amazing for exactly that and yes it has a good approach to the distributed cluster problem. Once you put it in production there a very different picture. https://github.com/hjacobs/kubernetes-failure-stories https://github.com/hjacobs/kubernetes-failure-stories I haven't had a single Kubernetes user who had it in production and did not have stability or performance issues with it. This is why I mentioned that I am not ready to trade my Hadoop outages and performance issues for Kubernetes ones.

m0zg 7y ago

Serious, non-facetious question: who is this for?

andygrove 7y ago
This is for people who want to live in a future where we use efficient and safe system level languages for massively scalable distributed data processing. IMHO, Python and Java are not ideal language choices for these purposes. Rust offers much lower TCO compared to current industry best practices in this area.
- geezerjay 7y ago
  But claiming that "X software is written in Y language/framework" says nothing about efficiency or safety. It's just meaningless marketting piggy-backing on popular buzzwords. And claims about "the future" are simply absurd. Oddly enough, this link appears right next to another story on how Cobol powers the world's economy. Frankly, I'm surprised blockchain wasn't shoved somewhere in the announcement.
  - andygrove 7y ago
    Check out my previous benchmarks on Rust-based DataFusion vs JVM-based Apache Spark workloads. It isn't just about raw performance numbers but also about resource requirements (especially RAM requirements when using JVM or similar GC-based languages). There are order-of-magnitude differences.
    - geezerjay 7y ago
      If you have tangible results then present your benchmarks. If you limit your marketing to empty claims regarding "the future" and vague assertions on performance then you're actually actively working to lower your credibility.
      - andygrove 7y ago
        This is a personal open source project. I'm not sure I'm "marketing" it since I make zero dollars from this work. For benchmarks, check out the past 18 months of posts on my blog. Here is the most recent: https://andygrove.io/2019/04/datafusion-0.13.0-benchmarks/ https://andygrove.io/2019/04/datafusion-0.13.0-benchmarks/

ohnoesjmr 7y ago

I congratulate the effort, as I always thought that Spark is great, but the fact it was written in Java hinders it quite badly (GC, tons of memory required for the runtime, jar hell (want to use proto3 in your spark job? Good luck)). I do however worry that rust has a high bar of entry.

andygrove 7y ago
I agree. Rust has a very steep learning curve compared to JVM languages. My hope in building this platform is that it can provide value to other languages (especially JVM) by taking query plans and executing them efficiently.
- 0815test 7y ago
  > compared to JVM languages Apache Spark is written in Scala, and I wouldn't describe that as having an 'easy' learning curve, even compared to Rust! If you want something 'easy' on the JVM, Kotlin (and perhaps Eta or Frege) might be more appropriate.
  - pjmlp 7y ago
    Scala is way easier to use, given it uses a tracing GC, has three good IDEs, and more mature ecosystem.
  - twic 7y ago
    Coming from Java, i found Rust easier to learn than Scala.
    - Recurecur 7y ago
      I expect you were exposed using a heavily functional programming approach. That's a whole 'nother (larger) area to pick up along with the different language. If Scala is used as a "better Java" to start with, and then the developers explore FP at their own pace, I think there's a better outcome. Granted Java 8+ has done some to close the gap with Scala as well. I wonder if Scala-Native will absorb some Rustish features as other languages have or are attempting to... (What I said above echoes Odersky's ideas about Scala developer levels: https://www.scala-lang.org/old/node/8610.html https://www.scala-lang.org/old/node/8610.html)

wiradikusuma 7y ago

So Spark is bloated bcoz of JVM. Does Graal make the point moot?

voodootrucker 7y ago
Not really, no. Even though Graal compiles to native code, these JVM based languages are all based on two ideas: 1. there will be a GC to manage memory 2. memory will be managed The GC slows things down, so spark tries to work around this by accessing "off heap" memory (which really just means off the JVM's heap and on the OS'). So you end up getting OOM errors if you give to little to the JVM or if you give too little to the OS. It's a hacky balancing act to get native access to memory, which comes for free with rust.

s_Hogg 7y ago

Hang in there mate :) I really don't think you deserve a lot of the crap you've been given in this thread. Someone has to try something new.

eb0la 7y ago
The fact people opposed to your idea / work means it is valuable enough for people to say something against and not ignore it. I must confess I miss native execution of (big)Data jobs. I know moving jvm bytecode between nodes to be run is portable, but nowadays nobody has mixed architecture (intel/mips/sparc/arm) servers, so... why do we need a bytecode execution layer? Maybe I am too critic, but bare metal - hypervisor - jvm - app looks too much layers to me.
- s_Hogg 7y ago
  Totally, I think there are a lot of cases where all that machinery is actually completely superfluous anyway because thinking about the problem you're trying to solve could lead you to a way of doing it that doesn't need all that compute. There's a blog post about this I read a while back where someone did a simple graph search using spark and then did it in a slightly smarter way on a commodity laptop and it outran spark on a large-ish amount of data. Wish I could find it.
  - buckminster 7y ago
    This one? http://www.frankmcsherry.org/graph/scalability/cost/2015/01/15/COST.html http://www.frankmcsherry.org/graph/scalability/cost/2015/01/...
    - s_Hogg 7y ago
      Yep, that's the one!
  - eb0la 7y ago
    I remember 4-5 years ago I tried to slice a big CSV file with Spark and took soo loong it didn't finished after some hours. Later on the day I realized it could be done from the command line with 'cut' and 'grep'. Took less than 15 minutes.

eb0la 7y ago

This project needs a "how to help" section urgently

andygrove 7y ago
I will write up some guidance in the next few days for those looking to contribute!

deleted 7y ago

[deleted]

snicker7 7y ago

Would this system support custom aggregates? How would I, for example, create a routine that defines a covariance matrix and have Ballista deal with the necessary map-reduce logic?

fspear 7y ago

Are you looking for contributors? I don't have any rust, arrow or k8s experience but been looking to learn all 3, I've also been looking to contribute to os projects so I'm happy to pick up any low hanging fruits if you are interested. I do have a few years of experience with Spark and hadoop if that's worth anything.

andygrove 7y ago
Yes contributors are welcome. I will write up some guidance in the next few days for those looking to contribute!

dswalter 7y ago

I'm actually excited about the possibilities. I've watched DataFusion from afar, and I have spent a decent amount of time wishing the Big Data ecosystem had arrived during a time when something like Rust was a viable option, both for memory and for parallel computing. I use Presto all the time, I love how fully-featured it is, but garbage collection is a non-trivial component of time-to-execute for my queries.

kyllo 7y ago

This is really cool! What do you see as ideally the primary API for something like this? SQL is great for relational algebra expressions to transform tables but its limited support for variables and control flow constructs make it less than ideal for complex, multi-step data analysis scripts. And when it comes to running statistical tests, regressions, training ML models, it's wholly inappropriate. Rust is a very expressive systems programming language, but it's unclear at this point how good of a fit it can be for data analysis and statistical programming tasks. It doesn't have much in the way of data science libraries yet. Would you potentially add e.g. a Python interpreter on top of such a framework, or would you focus on building out a more fully-featured Rust API for data analysis and even go so far as to suggest that data scientists start to learn and use Rust? (There is some precedence for this with Scala and Spark)

andygrove 7y ago
These are great questions and topics I plan on addressing in a future blog post. SQL is a great convenience for simple analytical queries and it can be nice to be able to mix and match SQL and other access patterns (this is one thing I like with the Apache Spark DataFrame approach). It's true that the Rust ecosystem for data science is not really there yet and I am trying to inspire people to start changing that. I think Rust does have some good potential here. In the nearer term though I am exploring options around support user code in distributed query execution and I don't want to limit it to Rust.
- kyllo 7y ago
  I like it--I'm a data scientist by day, and I've been following Rust with interest but I haven't found a good use case for it in my job yet. A Rust-based Spark competitor sounds like it could be exactly that excuse to use Rust at work that I've been looking for!
  - andygrove 7y ago
    This is exactly the situation I am in. We have workloads that I think we could deliver with much lower TCO using Rust/DataFusion/Ballista. We could probably even use a single node for some of our workflows just using DataFusion directly with same performance as a distributed Spark job.

cozos 7y ago

Most "big data" distributed compute frameworks that come to mind are written in a JVM language, so the focus on Rust is interesting. So then, would Rust be better than a JVM language for a distributed compute framework like Apache Spark? Based on what others said in this thread, these are the primary arguments for Rust: 1. JVM GC overhead 2. JVM GC pauses 3. JVM memory overhead. 4. Native code (i.e. Rust) has better raw performance than a JVM language My take on it: (1) I believe Spark basically wrote its own memory management layer with Unsafe that let's it bypass the GC [0], so for Dataframe/SQL we might be ok here. Hopefully value types are coming to Java/Scala soon. (2) Majority of Apache Spark use-cases are batch right? In this case who cares about a little stop-the-world pause here and there, as long as we're optimizing the GC for throughput. I recognize that streaming is also a thing, so maybe a non-GC language like Rust is better suited for latency sensitive streaming workloads. Perhaps the Shenandoah GC would be of help here. (3) What's the memory overhead of a JVM process, 100-200 MB? That doesn't seem too bad to me when clusters these days have terabytes of memory. (4) I wonder how much of an impact performance improvements from Rust will have over Spark's optimized code generation [1], which basically converts your code into array loops that utilize cache locality, loop unrolling, and simd. I imagine that most of the gains to be had from a Rust rewrite would come from these "bare metal' techniques, so it might the case that Spark already has that going for it... Having said that, I can't think of any reasons why a compute engine on Rust is a bad idea. Developer productivity and ecosystem perhaps? [0] https://databricks.com/blog/2015/04/28/project-tungsten-bringing-spark-closer-to-bare-metal.html https://databricks.com/blog/2015/04/28/project-tungsten-brin... [1] https://databricks.com/blog/2016/05/23/apache-spark-as-a-compiler-joining-a-billion-rows-per-second-on-a-laptop.html https://databricks.com/blog/2016/05/23/apache-spark-as-a-com...

andygrove 7y ago
Some good points. Some incredible engineering has gone into Spark to work around the fact that it runs on the JVM. Memory overhead of Spark particularly (not just JVM) is very high. In some cases close to 100x more memory than equivalent query execution with DataFusion [0]. Also you might be interested to see my original blog post with some of my thoughts on this [1]. [0] https://andygrove.io/2019/04/datafusion-0.13.0-benchmarks/ https://andygrove.io/2019/04/datafusion-0.13.0-benchmarks/ [1] https://andygrove.io/2018/01/rust-is-for-big-data/ https://andygrove.io/2018/01/rust-is-for-big-data/

senderista 7y ago

If you’re looking for an approachable distributed query planner, https://github.com/uwescience/raco https://github.com/uwescience/raco might be a good place to start.

blittable 7y ago

Super cool. Perhaps naive, but how does distributing computation with serialization square with Arrow's in-memory design?

andygrove 7y ago
Each executor within the cluster would use Arrow in-memory design. If you have enough cores and memory on a single node then potentially you wouldn't need a cluster.